The AI search stack.
A short list of the APIs and tools we point members to for building agentic and search-driven work, with what each is best for. 8 tools, curated by Tom and Jeremy.
Web Scrapers & Crawlers
Compare themFirecrawl
AI-first web data infrastructure designed to convert websites into clean, LLM-ready structured data formats like Markdown and JSON.
Best for
Converting complex websites directly into clean, LLM-ready markdown.
- AI Data Extraction via Prompts
- Intelligent Crawling & Mapping
- Bypass anti-bot & CAPTCHAs
- Output in Markdown/JSON
firecrawl.devPricing
Freemium / Pay per page
Apify
A full-stack cloud platform for web scraping and browser automation, featuring a massive marketplace of pre-built 'Actors'.
Best for
Large-scale scraping workflows and pre-built integrations.
- Serverless Actors
- Proxy rotation & Anti-Bot
- Zapier/LangChain Integrations
- JavaScript Rendering
apify.comPricing
Freemium / Usage based
Browserbase
Cloud infrastructure for headless browser automation, functioning as an essential bridge between AI agents and the web.
Best for
Running autonomous AI agents in secure headless browsers.
- Stagehand AI Framework
- Session recording & replays
- Proxy management
- Secure isolated containers
browserbase.comPricing
Usage based
Jina Reader API
An API that converts complex web pages, PDFs, and search results into LLM-ready text, simplifying data ingestion.
Best for
Instant extraction of clean markdown from articles, PDFs, and search results.
- Search Grounding (s.jina.ai)
- PDF & Webpage parsing (r.jina.ai)
- Noise reduction (removes ads/nav)
- Clean Markdown output
jina.aiPricing
Freemium
Zyte
Premium web scraping API and proxies to get data from any website without blocks, using advanced AI parsing.
Best for
Enterprise-grade proxy rotation and unblocking highly-secured sites.
- Smart proxy rotation
- Headless browsers
- AI-powered data parsing
- Automatic unblocking
zyte.comPricing
Usage based
Prompt Trackers
Compare themLangfuse
Open source LLM engineering platform. Tracing, evals, prompt management and metrics.
Best for
Open-source observability and end-to-end prompt management.
- Tracing
- Prompt Management
- Evaluations
- Metrics
langfuse.comPricing
Open Source / Cloud Hosted
Braintrust
The enterprise-grade platform for evaluating and improving LLM applications.
Best for
Rigorous enterprise LLM evaluations and custom dataset building.
- Evaluations
- Playgrounds
- Prompt Management
- Datasets
braintrust.devPricing
Enterprise / SaaS
Helicone
Purpose-built observability for generative AI. Monitor, optimize, and evaluate LLMs.
Best for
Purpose-built latency caching, cost tracking, and rate limiting.
- Analytics
- Caching
- Custom Properties
- Playground
helicone.aiPricing
Freemium
Using something that belongs here? Tell us at tom@searchleaders.ai.
See what's insideNot ready to join? Get the newsletter.
AI Search Leaders on Substack: Tom Critchlow on AI search and the people working it out, by email. Free.
Don't get left behind.
Join 150+ senior search people working out AI search together.