Crawl4AI

🔥🕷️ Crawl4AI: Crawl Smarter, Faster, Freely. For AI.

Crawl4AI is an Apify Actor that wraps the powerful Crawl4AI library, providing you with a feature-packed web crawler and scraper with additional functionalities like link-following and automatic retries for failed requests.

The Actor can:

• Crawl and scrape websites with precision using CSS, XPath, or LLM-based extraction methods.
• Generate clean Markdown output, suitable for RAG pipelines or direct ingestion into large language models.
• Automatically follow links to explore websites further without manual intervention.
• Retry failed requests to ensure maximum data collection with minimal effort.

Usage

Scraping with Crawl4AI is straightforward. Just follow these steps to get your data quickly:

1. Input your target URLs.
2. Set your extraction method (optional - CSS, XPath, or LLM-based).
3. Configure advanced options like proxies or session settings (optional).
4. Run the Actor to start crawling, link-following, and retrying failed requests automatically.
5. Retrieve your data in structured Markdown format for further use in your projects.

How much will it cost?

Apify provides $5 free usage credits every month on the Apify Free plan. With Crawl4AI, you can enjoy a certain number of results per month for free.

For larger data needs, consider upgrading to the $49/month Starter plan for increased monthly results volume. Or opt for the Scale plan for even higher result limits.

Results

Here is an example of the data that the Actor produces:

1[{
2  "url": "https://docs.crawl4ai.com/",
3  "markdown": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242424242.md",
4  "html": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242424242.html",
5  "metadata": {
6    "title": "Home - Crawl4AI Documentation (v0.5.x)",
7    "description": "🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper"
8  }
9},
10{
11  "url": "https://docs.crawl4ai.com/advanced/ssl-certificate/",
12  "markdown": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242424242.md",
13  "html": "https://api.apify.com/v2/key-value-stores/m1Sqnke1KWM0AI8co/records/content_4242424242.html",
14  "metadata": {
15    "title": "SSL Certificate - Crawl4AI Documentation (v0.5.x)",
16    "description": "🚀🤖 Crawl4AI, Open-source LLM-Friendly Web Crawler & Scraper"
17  }
18},
19// ...
20]