crawler

🔥 The Web Data API for AI - Power AI agents with clean web data

Scrapy, a fast high-level web crawling & scraping framework for Python.

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

Python scraper based on AI

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥

Web crawler and scraper for Rust

An ergonomic Python HTTP Client with TLS fingerprint

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

❤️ Fredy - [F]ind [R]eal [E]state [D]amn Eas[y] - Fredy keeps searching for new apartments, houses, and flats in Germany on platforms like ImmoScout24, Immowelt, Immonet, eBay Kleinanzeigen, and WG-Gesucht and instantly delivers the results to you via Slack, Telegram, Email, Discord or ntfy, so you can focus on the more important things in life ;)

Library for Rapid (Web) Crawler and Scraper Development

The unix-way web crawler

SEO/GEO toolkit to analyse, crawl, parse and optimise websites & logs (Nginx & Apache)