Twitter (X) Comment Scraper

This Apify actor is designed to scrape comments from a specific tweet.

Features

This actor provides comprehensive data collection capabilities from Twitter comments. It scrapes comments from a specific tweet and extracts detailed information including user profiles, engagement metrics, and media content. The collected data encompasses user details such as username, follower counts, and verification status, along with tweet-specific information including content, timestamps, and engagement statistics. For media-rich tweets, it captures various media types including images, videos, and GIFs, along with their associated metadata. The actor offers flexible comment sorting options, allowing users to sort comments by relevance, newest, or most liked to better suit their data collection needs.

The scraper is designed to handle Twitter's modern interface and can collect data such as:

• Comprehensive user profile information including Twitter Blue verification status
• Complete tweet content with engagement metrics (likes, retweets, replies, quotes)
• Advanced statistics like bookmark counts and view counts
• Detailed media information including video quality and duration
• Tweet metadata such as creation time, language, and source application
• Flexible comment sorting options (relevance, newest, most liked)

Obtaining Cookie Information

You can use the Cookie Editor extension to obtain Twitter cookie information. Follow these steps:

1. Add the Cookie Editor extension to your Chrome browser.
2. Log in to Twitter.
3. Click on the Cookie Editor icon in the top right corner of your browser.
4. Click the "Export" button to export your cookie information in JSON format.
5. Use this JSON string as the cookieJson input.

Note: You can validate with a tool like JSONLint to make sure your JSON input is correct.

Note: If you encounter errors or if 0 comments are processed, ensure that your account is functioning normally and log in again to obtain new cookies to use as input.

Usage

1. Run this actor in the Apify console.
2. Provide the desired inputs:

• tweetUrl: The URL of the tweet you want to scrape comments from.
• cookieJson: A JSON string containing your Twitter account's cookie information.
• maxDepth: Maximum comment depth (0 = unlimited).
• sortBy: Sort comments by relevance (default), newest, or most liked.
• onlyDirectReplies: Show only replies to the tweet owner (default: true).

Output

The scraped comments are saved to the Apify dataset. The output data includes:

User Information

• user.name: Username
• user.screen_name: User tag
• user.followers_count: Number of followers
• user.following_count: Number of following
• user.profile_image: Profile image URL
• user.is_blue_verified: Twitter Blue verification status

Tweet Information

• text: Tweet content
• reply_to_username: Username being replied to
• favorite_count: Number of likes
• reply_count: Number of replies
• retweet_count: Number of retweets
• quote_count: Number of quotes
• bookmark_count: Number of bookmarks
• views_count: Number of views
• created_at: Creation timestamp
• lang: Tweet language
• source: Tweet source

Media Information

• media: Media content (images, videos, GIFs)
  • type: Media type
  • thumb_url: Thumbnail URL
  • url: Media URL
  • video_url: Video URL (if applicable)
  • duration_ms: Video duration (if applicable)

Example Output

1{
2  "user": {
3    "name": "Baboon Forest Entertainment",
4    "screen_name": "BaboonForestEnt",
5    "followers_count": 27494,
6    "following_count": 4377,
7    "profile_image": "https://pbs.twimg.com/profile_images/1524636589703401472/b9s9kNcE_normal.jpg",
8    "verified": false,
9    "is_blue_verified": true
10  },
11  "text": "@elonmusk @realDonaldTrump 📢HUUUGEE NEWS !!🔉 SOUND ON 🔉\n\nMillions day! 🔉 Sound ON! 🔉 https://t.co/5wh38S1YVs",
12  "reply_to_username": "elonmusk",
13  "favorite_count": 2156,
14  "reply_count": 43,
15  "retweet_count": 305,
16  "quote_count": 0,
17  "bookmark_count": 1,
18  "views_count": "3835",
19  "created_at": "Fri Nov 01 16:21:51 +0000 2024",
20  "lang": "en",
21  "source": "Twitter Web App",
22  "media": [
23    {
24      "type": "video",
25      "thumb_url": "https://pbs.twimg.com/ext_tw_video_thumb/1852385552315592705/pu/img/kgy8ADvi8YeEV46U.jpg",
26      "url": "https://x.com/BaboonForestEnt/status/1852385575937986709/video/1",
27      "video_url": "https://video.twimg.com/ext_tw_video/1852385552315592705/pu/vid/avc1/1280x720/KSshZIvP3JZ6EjZv.mp4?tag=12",
28      "duration_ms": 25900
29    }
30  ]
31}

This example output shows the structure of a single comment. The actual output will be a list of similar objects for all scraped comments.

Notes

• It is designed to scrape comments visible on the tweet screen. Therefore, it cannot process nested comments and can retrieve a maximum of 200 comments due to limitations.
• The collected data is stored in Apify’s default data store.