The AI search stack.

A short list of the APIs and tools we point members to for building agentic and search-driven work, with what each is best for. 8 tools, curated by Tom and Jeremy.

Web Scrapers & Crawlers

Compare them
  • Firecrawl

    AI-first web data infrastructure designed to convert websites into clean, LLM-ready structured data formats like Markdown and JSON.

    Best for

    Converting complex websites directly into clean, LLM-ready markdown.

    • AI Data Extraction via Prompts
    • Intelligent Crawling & Mapping
    • Bypass anti-bot & CAPTCHAs
    • Output in Markdown/JSON

    Pricing

    Freemium / Pay per page

    firecrawl.dev
  • Apify

    A full-stack cloud platform for web scraping and browser automation, featuring a massive marketplace of pre-built 'Actors'.

    Best for

    Large-scale scraping workflows and pre-built integrations.

    • Serverless Actors
    • Proxy rotation & Anti-Bot
    • Zapier/LangChain Integrations
    • JavaScript Rendering

    Pricing

    Freemium / Usage based

    apify.com
  • Browserbase

    Cloud infrastructure for headless browser automation, functioning as an essential bridge between AI agents and the web.

    Best for

    Running autonomous AI agents in secure headless browsers.

    • Stagehand AI Framework
    • Session recording & replays
    • Proxy management
    • Secure isolated containers

    Pricing

    Usage based

    browserbase.com
  • Jina Reader API

    An API that converts complex web pages, PDFs, and search results into LLM-ready text, simplifying data ingestion.

    Best for

    Instant extraction of clean markdown from articles, PDFs, and search results.

    • Search Grounding (s.jina.ai)
    • PDF & Webpage parsing (r.jina.ai)
    • Noise reduction (removes ads/nav)
    • Clean Markdown output

    Pricing

    Freemium

    jina.ai
  • Zyte

    Premium web scraping API and proxies to get data from any website without blocks, using advanced AI parsing.

    Best for

    Enterprise-grade proxy rotation and unblocking highly-secured sites.

    • Smart proxy rotation
    • Headless browsers
    • AI-powered data parsing
    • Automatic unblocking

    Pricing

    Usage based

    zyte.com

Prompt Trackers

Compare them
  • Langfuse

    Open source LLM engineering platform. Tracing, evals, prompt management and metrics.

    Best for

    Open-source observability and end-to-end prompt management.

    • Tracing
    • Prompt Management
    • Evaluations
    • Metrics

    Pricing

    Open Source / Cloud Hosted

    langfuse.com
  • Braintrust

    The enterprise-grade platform for evaluating and improving LLM applications.

    Best for

    Rigorous enterprise LLM evaluations and custom dataset building.

    • Evaluations
    • Playgrounds
    • Prompt Management
    • Datasets

    Pricing

    Enterprise / SaaS

    braintrust.dev
  • Helicone

    Purpose-built observability for generative AI. Monitor, optimize, and evaluate LLMs.

    Best for

    Purpose-built latency caching, cost tracking, and rate limiting.

    • Analytics
    • Caching
    • Custom Properties
    • Playground

    Pricing

    Freemium

    helicone.ai

Using something that belongs here? Tell us at tom@searchleaders.ai.

See what's inside

Not ready to join? Get the newsletter.

AI Search Leaders on Substack: Tom Critchlow on AI search and the people working it out, by email. Free.

Subscribe free (opens Substack)

Don't get left behind.

Join 150+ senior search people working out AI search together.