The Twitter connector will crawl content from any twitter account.
The Twitter connector is a crawler developed using the Twitter Developer Platform for tweets discovery, but relies on the Aspire 3 Connector Framework to handle connections and distributed crawls.On this page
Features
Some of the features of the Twitter connector include:
- Authentication using twitter user, consumer key and consumer secret key
- Incremental crawl
- Full crawl
Content Retrieved
The Twitter connector retrieves all tweets related to the twitter user specified. Listed below are some examples content information of different tweet types that can be retrieved by this crawler.
- Text tweet
- URL links
- Geo location
- Hashtags
- User mentions entities
- Media entities
- Retweet count
This crawler will retrieve any document found linked in the HTML Markup as links (such as PDFs, MS Word, MS PowerPoint, etc).
Limitations
Due to the design implementation, Twitter connector has the following limitations:
- To retrieve the tweets from Twitter the implementation is using a application only authentication using the getUserTimeline method to extract user tweets the approach is returning at maximum 3200 tweets for reference see https://developer.twitter.com/en/docs/tweets/timelines/api-reference/get-statuses-user_timeline.html
Anything we should add? Please let us know.
Overview
Content Tools