Scrape Reddit posts and comments without the official API
Last updated
You can scrape Reddit without the API by running the Datapika Reddit actor on Apify. It reads public Reddit pages, so it needs no developer key, OAuth token, or login, and it never depended on the unauthenticated .json endpoints that began returning 403 in late May 2026. Pick one of six actions (scrape_subreddit, search_posts, search_comments, search_subreddits, fetch_post, or reddit_answers), set a limit up to 500, and export JSON or CSV. Pricing is per result: $2 per 1,000 posts, $3 per 1,000 with full comment trees, and the actor has 6,931 runs with a 4.9 rating (Apify Store, August 2026).
Why did scraping Reddit with .json URLs stop working?
Since late May 2026, appending .json to a Reddit URL returns 403 Forbidden for requests without a login (Crawlora, 2026; FetchLayer, 2026). Reddit had already closed self-service OAuth registration in November 2025, so the .json trick was the last free path left, and removing it broke most self-hosted Reddit scrapers, monitoring scripts, and RSS bridges in a single week. Some pipelines failed loudly with 403s; others kept returning HTTP 200 with empty JSON, so the break went unnoticed until dashboards ran dry. Logged-in sessions and the official OAuth Data API still work, but both require an approved app and a Reddit account.
Datapika's Reddit actor never depended on those endpoints. It requests the same public pages a browser loads, with browser-grade TLS fingerprinting and 8 rotating user agents, and falls back to old.reddit.com when the main site refuses a request. The Apify Store page showed 100 percent of runs succeeding as of August 2026.
If your pipeline started returning 403s or empty JSON this summer, the fix is to change the data source, not to add retries.
- Cutoff: late May 2026; unauthenticated .json requests now return 403 Forbidden (Crawlora, 2026)
- Self-service OAuth app registration closed in November 2025, so new keys need Reddit approval (FetchLayer, 2026)
- Reddit has flagged RSS as the next surface it may close (Crawlora, 2026)
- Datapika reads public HTML pages, not .json endpoints, so the cutoff did not change its data source
- Store-reported success rate: 100 percent of runs succeeded, as of August 2026
- No Reddit account, developer key, or OAuth registration is involved at any step
What can you scrape from Reddit without an API key?
The actor bundles six actions behind one input schema, so you choose an action and pass the matching parameters. scrape_subreddit pulls a subreddit's feed sorted by hot, new, top, rising, or controversial, with a time filter from past hour to all time on top and controversial sorts and an includeComments switch. search_posts runs a keyword search across all of Reddit or inside one subreddit, sorted by relevance, new, top, or most comments.
search_comments finds public comments matching a phrase, optionally scoped to one subreddit, which is the quickest way to catch brand mentions buried deep in threads. search_subreddits takes a topic and returns communities with mention counts and sample posts, useful when you do not yet know where a niche audience posts. fetch_post takes a permalink and returns that thread with its complete comment tree.
The sixth action, reddit_answers, queries Reddit's own AI answer engine and returns a markdown answer with follow-up questions, source post IDs, and source subreddits. The actor's listing describes it as the only Apify actor exposing that feature.
- scrape_subreddit: any public subreddit, sorted by hot, new, top, rising, or controversial, time filters from hour to all
- search_posts: keyword search across Reddit or scoped to one subreddit, sorted by relevance, new, top, or comments
- search_comments: public comments matching a phrase, with parent IDs for thread context
- search_subreddits: communities ranked by mention count, each with sample posts
- fetch_post: one permalink in, full thread with nested comments out
- reddit_answers: Reddit AI Answers as markdown plus follow_ups and source lists
What fields does each Reddit post and comment include?
Every post row shares one flat schema whichever action produced it, so you can union a subreddit sweep and a keyword search without remapping columns. The core fields are post_id, permalink, subreddit_name, author_name, title, body, media URLs, num_comments, num_upvotes, and an ISO 8601 post_timestamp.
When includeComments is on for scrape_subreddit, each row carries a comments array holding author_name, body, media, and parent_id for every comment, so you can rebuild the nested thread by joining parent_id to post and comment IDs. Comment trees are fetched by a configurable pool of 1 to 20 parallel workers (default 10), which keeps a 100-post run with comments in the range of minutes rather than hours. A post with 500 comments still means 500 page requests, so budget time accordingly.
AI Answer rows use a different shape: markdown, follow_ups, source_posts, and source_subreddits. Whatever the action, output lands in an Apify dataset you can download as JSON, CSV, or Excel, or read through the dataset API.
- Post identity: post_id, permalink, subreddit_name, author_name
- Content: title, full body text, and an array of media URLs (images, video links)
- Engagement: num_upvotes and num_comments at scrape time
- Timing: post_timestamp in ISO 8601 UTC, for example 2026-04-01T12:00:00Z
- Comments: author_name, body, media, parent_id per comment, nested to full depth
- Export: JSON, CSV, Excel, or programmatic access through the Apify dataset API
How much does it cost to scrape Reddit without the API?
Pricing is per result delivered, with no subscription and no minimum, and you are billed only for rows actually returned. Posts without comments are $2 per 1,000, posts with full comment trees are $3 per 1,000, search results and discovered subreddits are $2 per 1,000, single post fetches are $3 per 1,000, and AI Answers are $10 per 1,000 queries (Apify Store, August 2026). A daily brand sweep pulling 500 search results and 50 threads with comments therefore costs about $1.15, before proxy bandwidth.
For comparison, Trudax's Reddit Scraper Lite, one of the most-used Reddit actors on Apify, lists from $3.40 per 1,000 results with a 4.57 rating and 40,040 users as of August 29, 2026. Reddit's own commercial Data API is quoted at roughly $0.24 per 1,000 requests, but each request needs an approved OAuth app and is metered per call, not per row (Crawlora, 2026).
Datapika's actor carries a 4.9 rating from 8 reviews and 6,931 runs as of August 2026.
- Posts without comments: $2 per 1,000 (Apify Store, August 2026)
- Posts with full comment trees: $3 per 1,000
- Search results and subreddits found: $2 per 1,000 each
- Single post fetch: $3 per 1,000; Reddit AI Answers: $10 per 1,000 queries
- Worked example: 500 search results plus 50 threads with comments is about $1.15
- Trudax Reddit Scraper Lite for comparison: from $3.40 per 1,000, 4.57 rating, as of August 29, 2026
How do you scrape Reddit at scale without getting blocked?
Reddit rate-limits aggressively, and a naive script hits 429 errors within a few hundred requests. The actor handles this with exponential backoff and jitter, so a 429 delays the next request instead of ending the run. Every request carries a browser-grade TLS fingerprint and one of 8 rotating Windows, macOS, or Linux user agents.
When www.reddit.com refuses a page, the actor retries against old.reddit.com, and as a last resort an archive layer can recover some deleted posts and comments. The default proxy configuration already uses Apify's residential proxy group, billed at $8 per GB through your Apify account; the README recommends keeping it for anything beyond test runs, since datacenter IPs draw far more Reddit blocks.
Start with limit set to 10 to 50 to validate a query, then raise it toward the 500 cap. Turn includeComments off when you only need titles and scores, since that speeds the run and cuts the price from $3 to $2 per 1,000 posts. Private and restricted subreddits are out of scope because they require a login.
- Exponential backoff with jitter on 429 responses, so runs slow down rather than fail
- 8 rotating browser user agents across Windows, macOS, and Linux plus TLS fingerprinting
- Fallback chain: www.reddit.com, then old.reddit.com, then an archive layer for deleted content
- Residential proxies on by default at $8 per GB; the README cites millions of posts per day with them
- Comment fetching uses 1 to 20 parallel workers with random jitter between requests
- Test with limit 10 to 50, then scale to 500; includeComments off drops cost to $2 per 1,000
| Action | Input | Each row returns | Price per 1,000 |
|---|---|---|---|
| scrape_subreddit | subreddit, sort, timeFilter, limit | Post metadata: title, body, author, upvotes, comment count, media URLs, timestamp | $2 |
| scrape_subreddit with includeComments | same plus includeComments: true | Post metadata plus full nested comment tree | $3 |
| search_posts | query, optional subreddit, sort, limit | Matching posts in the standard post schema | $2 |
| search_comments | query, optional subreddit, limit | Matching comments with author, body, media, parent_id | $2 (search-result rate) |
| search_subreddits | query, limit | Subreddit name, mention count, sample posts | $2 |
| fetch_post | postUrl | One thread with its complete comment tree | $3 |
| reddit_answers | natural-language question | Markdown answer, follow_ups, source_posts, source_subreddits | $10 |
How to do it
- 1.Open the Datapika Reddit actor at https://apify.com/openclawai/reddit-scraper, or attach it to an agent through https://mcp.apify.com/?tools=fetch-actor-details,openclawai/reddit-scraper, and choose an action: scrape_subreddit, search_posts, search_comments, search_subreddits, fetch_post, or reddit_answers.
- 2.Fill the inputs for that action, for example subreddit technology, sort top, timeFilter week, limit 100, and leave includeComments off for a first pass. Keep the default residential proxy group if you plan to go past a few hundred rows.
- 3.Start the run. Rows stream into the dataset as they are fetched, so check the first 10 to 50 to confirm the fields you need, then raise the limit or add includeComments for full threads.
- 4.Download the dataset as JSON, CSV, or Excel, or read it through the Apify dataset API. Save the input and schedule it daily to turn a one-off scrape into a running Reddit feed.
Questions, answered
Does Reddit still have an official API, and what happened to the .json endpoints?
Reddit's official Data API still exists but requires an approved OAuth app; commercial use is quoted near $0.24 per 1,000 requests (Crawlora, 2026). The unauthenticated .json endpoints, which were the free workaround, stopped serving anonymous requests in late May 2026 and now return 403 Forbidden without a login. Datapika does not use either path, so the change did not alter its data source.
Do I need a Reddit account, developer key, or my own proxies to run this?
No account, developer key, or OAuth registration is needed, because the actor reads public Reddit pages rather than the API. You do not need your own proxies either: the default run configuration uses Apify's residential proxy group, billed at $8 per GB through your Apify account rather than through the per-result price. You can switch to datacenter proxies for small tests, at a higher risk of Reddit blocks.
Can it recover deleted Reddit posts or comments?
Partially. The fallback chain ends at an archive layer that can return some deleted posts and comments when Reddit itself no longer serves them. Coverage depends on what that archive captured before deletion, so treat recovered content as best effort rather than complete. Recovered rows are billed at the same per-result rate as live rows.
Does it work on private, restricted, or quarantined subreddits?
No. Private and restricted subreddits require a logged-in Reddit account with access, and the actor deliberately runs without authentication. Any public subreddit, public post, public comment, or public search result is in scope. If a subreddit went private after you scraped it, historical rows remain in your dataset but new runs against it will return nothing.
Can an AI agent call this Reddit scraper through MCP or the API?
Yes. The actor is exposed over the Apify MCP server, so a Claude, GPT, or custom agent can call it as a tool at https://mcp.apify.com/?tools=fetch-actor-details,openclawai/reddit-scraper, pass an action and query, and read the dataset back. The same input works through the Apify REST API for scheduled jobs. Billing stays per result, so an agent pulling 200 search results spends about $0.40.
Is scraping public Reddit data legal?
The actor only accesses content Reddit serves to any anonymous visitor and does not bypass authentication or touch private data. Public availability does not remove your own obligations: Reddit's user agreement, copyright in user posts, and data-protection laws such as GDPR still apply to how you store and use the rows, especially author names. Review those rules for your jurisdiction before running at scale.