Reddit Subreddit Scraper with Advanced Flair Filtering

Overview

Given Reddit's high API costs, this subreddit scraper provides an alternative to scraping subreddits. It is designed to streamline data collection from Reddit, empowering users to extract detailed post information while focusing on the content most relevant to their needs. With advanced flair-based filtering and automated scrolling for dynamic post loading, the scraper is perfect for tasks that require precision and efficiency.

Key Features and Benefits

1. Dynamic Post Loading

   • What it does: Automatically scrolls through the subreddit to trigger Reddit's infinite scrolling, loading more posts.
   • How it helps: Saves time by automating the tedious process of manually scrolling and ensures a comprehensive scrape of all available posts.

2. Detailed Post Information Extraction

   • What it does: Collects detailed data, including full post body, flairs, author information, timestamps, and more.
   • How it helps: Enables a deep understanding of subreddit trends and insights for research, marketing, or data analysis.

3. Advanced Flair Filtering

   • What it does: Filters posts based on specific flairs, such as "hiring" or "for sale."
   • How it helps: Cuts through noise to focus only on posts that meet your criteria, enhancing productivity and relevance for job boards, marketplaces, or community management.

Use Cases

1. Recruitment and Talent Sourcing

   • Identify hiring posts in subreddits like r/forhire or r/jobs to find qualified candidates or job opportunities.

2. Market Research and Trend Analysis

   • Extract posts from niche subreddits to monitor trends, customer feedback, and emerging market demands.

3. Competitive Analysis

   • Monitor posts in competitor-focused subreddits to understand their activities and customer sentiment.

4. Product and Service Leads

   • Collect posts advertising needs or requests for services and reach out with tailored offers.

5. Event and Campaign Monitoring

   • Filter and collect posts tagged as "event" or "campaign" to track community interest and participation.

Input Parameters

1. Subreddit URL

   • Description: The URL of the subreddit you want to scrape.
   • Requirements: Ensure it is the URL of a subreddit, not a post or comment thread. Example: https://www.reddit.com/r/example/.

2. Flairs

   • Description: Specific flairs to filter posts. Leave empty to gather all posts.
   • Requirements: Copy-paste flairs exactly as they appear in the subreddit. Inconsistent formatting will result in no matches.

3. Proxy

   • Description: A proxy to route your requests.
   • Recommendation: Use a residential proxy for reliable scraping. Without a proxy, Reddit may block your activity due to high request volumes.

4. MaxNumberOfPosts

   • Description: The maximum number of posts to scrape.
   • Considerations:
     • Setting this too high may increase runtime and usage costs.
     • Choose a reasonable number based on your data needs and time constraints.

FAQ

Is there a limit on a monthly subscription?

Apify does not impose a limit on monthly subscriptions. Any limit that occurs is probably due to the hardware limitations, reddit restrictions, or code inefficiencies. For example, you probably can't make 100 trillion trillion calls per month.

How do I find the correct flairs for filtering?

Check the subreddit and copy-paste the flairs directly from the posts or flair list to ensure accuracy.

What happens if I don’t use a proxy?

Without a proxy, Reddit is likely to block your activity after detecting unusual traffic patterns. Use a residential proxy for reliability.

How long does the scraper take to run?

The runtime depends on the number of posts requested and the subreddit’s loading speed. A high maxnumberofposts will take longer.

Can I export results in a specific format?

Yes, the scraper supports multiple formats such as JSON and CSV. Modify the configuration to specify the desired output.

Is it legal to scrape Reddit?

Ensure compliance with Reddit's Terms of Service and local regulations when using this scraper. Be sure to collect only public data. When in doubt, check https://help.apify.com/en/articles/6980905-is-web-scraping-legal and consult professional advice.

Example Output

1[{
2  "title": "Turkish Drama Content Writer Needed for News and Reviews\n",
3  "body": "We are seeking a talented writer who is passionate about Turkish dramas to create engaging articles covering the latest news related to Turkish drama celebrities, reviews of popular shows, and episode recaps. The ideal candidate should have a deep understanding of the Turkish drama landscape and be able to present content that resonates with fans. If you have experience in entertainment writing and a knack for storytelling, we want to hear from you!\nAt the beginning (the first month or so), Two or three articles a week are expected. We are offering 7c/w or 15$ an hour at the moment since we are at the beginning of this operation. This is negotiable. For the first year, all revenue will be invested back into the project, the writer's salary will increase to a monthly wage and daily articles will be expected.\nPlease DM for anyone interested.",
4  "author": "Emotional_Sherbert21",
5  "subreddit": "r/HireaWriter",
6  "flair": "Hiring (Entry Level)",
7  "post-type": "text",
8  "number-of-upvotes": "0",
9  "number-of-comments": "2",
10  "author-pfp": "https://styles.redditmedia.com/t5_4d9a9w/styles/profileIcon_snoo-nftv2_bmZ0X2VpcDE1NToxMzdfNTA2NmMwOTM0NjMyYmNjMjkwMmQxMzlkN2M4NzVjYmQyOTU0MjlmOF80NDc5_rare_de1004e3-dace-4756-991e-942b54a401f0-headshot.png?width=64&height=64&frame=1&auto=webp&crop=64:64,smart&s=0799fc16298ab85ae5b74b75d4cb6b1081da7209",
11  "moderation-verdict": "",
12  "is-awardable": "",
13  "user-id": "",
14  "subreddit-id": "t5_2zlvj",
15  "post-id": "t3_1gta128",
16  "score": "0",
17  "timestamp": "2024-11-17T09:32:48.377000+0000",
18  "post-link": "https://www.reddit.com/r/HireaWriter/comments/1gta128/turkish_drama_content_writer_needed_for_news_and/"
19},
20{
21  "title": "LF: character backstory writer",
22  "body": "Base pay per project/ piece: $40-$100 l\nBase pay per hour: $15-$25\nLooking for a character backstory writer, willing to pay by project or by hour, the characters basis is already written such as : Name, gender identity, ethnicity, nationality, height, age, birthday, eye color, occupation and weight.\nI will express the character more once I get into contact with you.\nWhat I want form the writer/ written backstory for the character:\nchildhood backstory\nAdolescence backstory\ncore personality written and shown through out both backstory’s\nhow the character met current relationships",
23  "author": "sugarpixie208",
24  "subreddit": "r/HireaWriter",
25  "flair": "Hiring (Entry Level)",
26  "post-type": "text",
27  "number-of-upvotes": "23",
28  "number-of-comments": "27",
29  "author-pfp": "https://styles.redditmedia.com/t5_6s1epe/styles/profileIcon_yr2qxi2p221e1.png?width=64&height=64&frame=1&auto=webp&crop=64:64,smart&s=9d47d9e046b933ef085fb399edecc103573c7f5d",
30  "moderation-verdict": "",
31  "is-awardable": "",
32  "user-id": "",
33  "subreddit-id": "t5_2zlvj",
34  "post-id": "t3_1gs11jm",
35  "score": "23",
36  "timestamp": "2024-11-15T17:02:05.283000+0000",
37  "post-link": "https://www.reddit.com/r/HireaWriter/comments/1gs11jm/lf_character_backstory_writer/"
38}]