Actor - Twitter Search Scraper

🐦 Twitter Search Scraper

The Twitter Search Scraper provides powerful tools to extract data from Twitter's search functionality, allowing you to find tweets based on keywords, phrases, and advanced search parameters.

🚀 Features and Functionality

Extremely detailed tweet information for data mining.

Multi-language support on Twitter. Globally working Twitter search actor. No restrictions!

Geo-target the tweets with advanced search capabilities. Integrate into your system with ease.

Lightning speed. Extreme efficiency. Suitable for integration into any platform.

Query Wizard: Easy-to-use Input for non-technical users. Create Twitter Advanced Search effortlessly.

A list of filters that you can easily use. Target your tweet search with precision!

• Custom Start URLs: Begin scraping from search query URLs.

  • Examples:
    • https://x.com/search?q=real%20madrid&src=typed_query
    • https://x.com/search?q=from%3AKMbappe&src=typed_query

• Search Queries: Provide a list of keywords or phrases to find tweets that match specific topics, such as "barca" or "atletico madrid"

• Authentication Support: Use Twitter authentication tokens for improved reliability and access to more data.

• Search Modes:

  • Choose between:
    • : Most recent tweets
    • : Most popular tweets

• Date Filters:

  • Define specific date ranges for scraping:
    • Tweets newer than a specific date (YYYY-MM-DD)
    • Tweets older than a specific date (YYYY-MM-DD)

• Content Filters:

  • Customize the type of tweets to scrape:
    • Only tweets with media (images, videos, or GIFs)
    • Only tweets from verified users
    • Exclude replies, retweets, or links

• Tweet Stats Filters: Filter tweets based on minimum likes, replies, or retweets.

• Account Filters:

  • Scrape tweets:
    • From a specific account.
    • To a specific account.
    • Mentioning a specific account.

• Language and Location: Limit results to specific languages or find tweets near a defined location.

• Proxy Configuration: Use advanced proxy settings to ensure anonymous and efficient scraping.

• Limitations: Set limits on the number of tweets or pages to scrape for optimized performance.

These features allow for a highly customizable and efficient search process, suitable for various data collection needs.

😎 Who needs this?

Twitter Search Scraper is designed to provide enriched search data with high speed in a cost-effective manner. It serves a wide range of users:

• Cryptocurrency Analysts and Investors - can use Twitter Search Scraper to track sentiment, key influencers' opinions, and market-moving announcements related to cryptocurrencies. Catch the panic buy or panic sell right away.
• Market Research Analysts can use Twitter Search Scraper to gather public opinion and trends on different products, services, or topics. By analyzing tweets, they can identify consumer behavior patterns, preferences, and market trends, helping companies make informed decisions.
• Academic Researchers can use Twitter Search Scraper to analyze social behaviors, communication patterns, and public opinion on various topics. This can aid in understanding societal trends, public reactions to events, and the spread of information or misinformation.
• Human Resources and Recruitment Specialists can use Twitter Search Scraper to monitor industry trends, understand job market sentiments, and identify potential candidates for recruitment, enhancing their talent acquisition strategies.

🔧 Bugs, fixes, updates, and changelog

This scraper is under active development. If you have any feature requests you can create an issue from here.

⚙️ Input and Definitions

Field	Type	Description
startUrls	array	Twitter (X) search URLs.
searchTerms	array	Search terms or queries you want to search from Twitter (X). Check this page for further reference.
sort	String	Sort order for search tweets. It can be either "Latest" or "Top".
authTokens	array	Authentication tokens for Twitter (X) to improve reliability and access more data.
maxItems	number	Maximum number of items that you want to receive as output
tweetLanguage	string	Restricts tweets to the given language, given by an ISO 639-1 code.
onlyVerifiedUsers	boolean	Returns only tweets by users who are verified.
onlyTwitterBlue	boolean	Returns only tweets by users who are Twitter Blue subscribers.
onlyImage	boolean	Returns only tweets that contain images.
onlyVideo	boolean	Returns only tweets that contain videos.
onlyQuote	boolean	Returns only tweets that are quotes.
author	String	Returns tweets sent by the given user. It should be a Twitter (X) Handle.
inReplyTo	String	Returns tweets that are replies to the given user. It should be a Twitter (X) Handle.
mentioning	String	Returns tweets mentioning the given user. It should be a Twitter (X) Handle.
geotaggedNear	String	Returns tweets sent near the given location.
withinRadius	String	Returns tweets sent within the given radius of the given location.
geocode	String	Returns tweets sent by users located within a given radius of the given latitude/longitude.
placeObjectId	String	Returns tweets tagged with the given place.
minimumRetweets	number	Returns tweets with at least the given number of retweets.
minimumFavorites	number	Returns tweets with at least the given number of favorites.
minimumReplies	number	Returns tweets with at least the given number of replies.
start	String	Returns tweets sent after the given date.
end	String	Returns tweets sent before the given date.
includeSearchTerms	boolean	If selected, a field will be added to each tweet about the search term that was used to find it.
customMapFunction	string	Function that takes each of the objects as argument and returns data that will be mapped by the function itself.
proxy	object	Select proxies to be used by your crawler.

This solution requires the use of Proxy servers, either your own proxy servers or you can use Apify Proxy.

Tip

Important: You must provide your own Twitter/X authentication tokens in the authTokens array for the scraper to work. These can be obtained from your Twitter accounts.

When you want to scrape over a specific search URL, just copy and paste the link as one of the startUrl.

If you would like to scrape only the first page of search results then put the link for the page and have the endPage as 1.

With the last approach that is explained above you can also fetch any interval of pages. If you provide the 5th page of a search and define the endPage parameter as 6 then you'll have the 5th and 6th pages only.

Compute Unit Consumption

The actor is optimized to run blazing fast and scrape many tweets as possible. Therefore, it forefronts all tweet detail requests. If the actor doesn't block very often it'll scrape 100 tweets in 20 seconds with ~0.02-0.025 compute units.

Twitter Search Scraper Input example

1{
2    "proxy": {
3        "useApifyProxy": true
4    },
5    "startUrls": [
6        "https://x.com/KMbappe",
7        "https://x.com/search?q=real%20madrid&src=typed_query",
8        "https://x.com/realmadrid/status/1877491794000412879"
9    ],
10    "searchQueries": [
11        "barcelona",
12        "atletico madrid"
13    ],
14    "searchMode": "Latest",
15    "maxItems": 100,
16    "endPage": 5,
17    "authTokens": [
18        "auth_token_example_1",
19        "auth_token_example_2"
20    ],
21}

During the Run

During the run, the actor will output messages letting you know what is going on. Each message always contains a short label specifying which page from the provided list is currently specified. When items are loaded from the page, you should see a message about this event with a loaded item count and total item count for each page.

If you provide incorrect input to the actor, it will immediately stop with a failure state and output an explanation of what is wrong.

Tweet Export

During the run, the actor stores results into a dataset. Each tweet is a separate item in the dataset.

You can manage the results in any language (Python, PHP, Node JS/NPM). See the FAQ or our API reference to learn more about getting results from this Twitter Search actor.

Scraped Twitter Properties

The structure of each tweet looks like this:

Tweet Detail

1{
2	"id": "1877102356627673443",
3	"url": "https://x.com/10Ronaldinho/status/1877102356627673443",
4	"text": "Hermosos recuerdos en Barcelona... Sempre Barça!!!!! https://t.co/1ljiuvNHf9",
5	"retweetCount": 4078,
6	"replyCount": 357,
7	"likeCount": 58982,
8	"quoteCount": 147,
9	"createdAt": "Wed Jan 08 21:17:31 +0000 2025",
10	"lang": "es",
11	"protected": false,
12	"bookmarkCount": 421,
13	"isReply": false,
14	"entities": {
15		"hashtags": [],
16		"media": [
17			{
18				"display_url": "pic.x.com/1ljiuvNHf9",
19				"expanded_url": "https://x.com/10Ronaldinho/status/1877102356627673443/photo/1",
20				"id_str": "1877102283202162688",
21				"indices": [
22					53,
23					76
24				],
25				"media_key": "3_1877102283202162688",
26				"media_url_https": "https://pbs.twimg.com/media/GgzOlM1W8AAXri7.jpg",
27				"type": "photo",
28				"url": "https://t.co/1ljiuvNHf9",
29				"ext_media_availability": {
30					"status": "Available"
31				},
32				"features": {
33					"all": {
34						"tags": [
35							{
36								"user_id": "96951800",
37								"name": "FC Barcelona",
38								"screen_name": "FCBarcelona",
39								"type": "user"
40							}
41						]
42					},
43					"large": {
44						"faces": []
45					},
46					"medium": {
47						"faces": []
48					},
49					"small": {
50						"faces": []
51					},
52					"orig": {
53						"faces": []
54					}
55				},
56				"sizes": {
57					"large": {
58						"h": 1280,
59						"w": 1280,
60						"resize": "fit"
61					},
62					"medium": {
63						"h": 1200,
64						"w": 1200,
65						"resize": "fit"
66					},
67					"small": {
68						"h": 680,
69						"w": 680,
70						"resize": "fit"
71					},
72					"thumb": {
73						"h": 150,
74						"w": 150,
75						"resize": "crop"
76					}
77				},
78				"original_info": {
79					"height": 1280,
80					"width": 1280,
81					"focus_rects": [
82						{
83							"x": 0,
84							"y": 0,
85							"w": 1280,
86							"h": 717
87						},
88						{
89							"x": 0,
90							"y": 0,
91							"w": 1280,
92							"h": 1280
93						},
94						{
95							"x": 111,
96							"y": 0,
97							"w": 1123,
98							"h": 1280
99						},
100						{
101							"x": 352,
102							"y": 0,
103							"w": 640,
104							"h": 1280
105						},
106						{
107							"x": 0,
108							"y": 0,
109							"w": 1280,
110							"h": 1280
111						}
112					]
113				},
114				"media_results": {
115					"result": {
116						"media_key": "3_1877102283202162688"
117					}
118				}
119			}
120		],
121		"symbols": [],
122		"timestamps": [],
123		"urls": [],
124		"user_mentions": []
125	},
126	"extentedEntities": {
127		"media": [
128			{
129				"display_url": "pic.x.com/1ljiuvNHf9",
130				"expanded_url": "https://x.com/10Ronaldinho/status/1877102356627673443/photo/1",
131				"id_str": "1877102283202162688",
132				"indices": [
133					53,
134					76
135				],
136				"media_key": "3_1877102283202162688",
137				"media_url_https": "https://pbs.twimg.com/media/GgzOlM1W8AAXri7.jpg",
138				"type": "photo",
139				"url": "https://t.co/1ljiuvNHf9",
140				"ext_media_availability": {
141					"status": "Available"
142				},
143				"features": {
144					"all": {
145						"tags": [
146							{
147								"user_id": "96951800",
148								"name": "FC Barcelona",
149								"screen_name": "FCBarcelona",
150								"type": "user"
151							}
152						]
153					},
154					"large": {
155						"faces": []
156					},
157					"medium": {
158						"faces": []
159					},
160					"small": {
161						"faces": []
162					},
163					"orig": {
164						"faces": []
165					}
166				},
167				"sizes": {
168					"large": {
169						"h": 1280,
170						"w": 1280,
171						"resize": "fit"
172					},
173					"medium": {
174						"h": 1200,
175						"w": 1200,
176						"resize": "fit"
177					},
178					"small": {
179						"h": 680,
180						"w": 680,
181						"resize": "fit"
182					},
183					"thumb": {
184						"h": 150,
185						"w": 150,
186						"resize": "crop"
187					}
188				},
189				"original_info": {
190					"height": 1280,
191					"width": 1280,
192					"focus_rects": [
193						{
194							"x": 0,
195							"y": 0,
196							"w": 1280,
197							"h": 717
198						},
199						{
200							"x": 0,
201							"y": 0,
202							"w": 1280,
203							"h": 1280
204						},
205						{
206							"x": 111,
207							"y": 0,
208							"w": 1123,
209							"h": 1280
210						},
211						{
212							"x": 352,
213							"y": 0,
214							"w": 640,
215							"h": 1280
216						},
217						{
218							"x": 0,
219							"y": 0,
220							"w": 1280,
221							"h": 1280
222						}
223					]
224				},
225				"media_results": {
226					"result": {
227						"media_key": "3_1877102283202162688"
228					}
229				}
230			}
231		]
232	},
233	"author": {
234		"username": "10Ronaldinho",
235		"url": "https://x.com/10Ronaldinho",
236		"id": "29479000",
237		"name": "Ronaldinho",
238		"isVerified": false,
239		"isBlueVerified": true,
240		"profilePicture": "https://pbs.twimg.com/profile_images/1537593562619060225/AmmxIwWi_normal.jpg",
241		"coverPicture": "https://pbs.twimg.com/profile_banners/29479000/1650911561",
242		"description": "Twitter oficial de Ronaldinho Gaúcho. Duas vezes eleito o melhor do mundo pela FIFA.\n\n@BetifyOfficial embaixador\nhttps://t.co/jctU4nkSvp",
243		"location": "",
244		"followers": 21437149,
245		"following": 152,
246		"canDm": false,
247		"canMediaTag": false,
248		"createdAt": "Tue Apr 07 16:11:50 +0000 2009",
249		"entities": {
250			"description": {
251				"urls": [
252					{
253						"display_url": "bit.ly/RonniexBetify",
254						"expanded_url": "https://bit.ly/RonniexBetify",
255						"url": "https://t.co/jctU4nkSvp",
256						"indices": [
257							113,
258							136
259						]
260					}
261				]
262			},
263			"url": {
264				"urls": [
265					{
266						"display_url": "nooke.ag",
267						"expanded_url": "http://www.nooke.ag",
268						"url": "https://t.co/f6O6lqd6yM",
269						"indices": [
270							0,
271							23
272						]
273					}
274				]
275			}
276		},
277		"fastFollowersCount": 0,
278		"favouritesCount": 1552,
279		"listedCount": 26192,
280		"hasCustomTimelines": false,
281		"isTranslator": false,
282		"mediaCount": 3449,
283		"statusesCount": 7101,
284		"withheldInCountries": [],
285		"affiliatesHighlightedLabel": {}
286	}
287}

Contact

Please visit us through epctex.com to see all the products that are available for you. If you are looking for any custom integration or so, please reach out to us through the chat box in epctex.com. In need of support? business@epctex.com is at your service.