Three fallback layers, one seamless user experience.
Why bother?
Give a user a text box and they will paste a URL into it. The question is what happens next.
Mirror Ai accepts URLs from users. The user submits a link, and the AI reads it, understands it, and answers questions about it. Simple on the surface. Complex underneath, because the web does not make itself easy to read.
Some pages load in milliseconds and serve clean HTML. Others require JavaScript rendering, while others are PDFs disguised as web pages. A single approach fails for too many of them.
We built a three-layer fallback stack that handles this.
Layer 1: Trafilatura
The first attempt uses Trafilatura — a lightweight, fast Python library purpose-built for extracting readable content from HTML pages. It handles roughly 90% of the URLs users throw at it. It is fast enough to return results in under a second, and it requires no external dependencies beyond the library itself.
Trafilatura works well on news articles, blog posts, documentation pages, and most standard web content. It is the default for a reason: it is fast, it is free, and it works.
Layer 2: Firecrawl via OpenRouter
For pages that Trafilatura cannot handle — JavaScript-heavy Single Page Applications, pages behind Cloudflare, dynamically rendered content — we fall back to Firecrawl, accessed through the OpenRouter API.
Firecrawl renders the page in a headless browser first, then extracts the text content after JavaScript execution has completed. This catches the pages that look empty to a simple HTML parser because their actual content is generated by client-side JavaScript.
OpenRouter acts as the gateway, providing access to the Firecrawl API without requiring a separate subscription. If we already have OpenRouter credits, the Firecrawl calls go through the same billing pipeline.
Layer 3: Plain HTTP with Readability
If Firecrawl is unavailable or rate-limited, the final fallback is a plain HTTP fetch with Python's readability-lxml extractor. This is the least sophisticated approach — it downloads the raw HTML and applies heuristics to find the main content — but it works when the other two options are not available.
Why Three Layers?
One tool cannot handle all of the web. Trafilatura is fast but fails on JavaScript-rendered pages. Firecrawl handles JavaScript but costs per request and can be rate-limited. The plain HTTP fallback is free but produces inconsistent quality.
Three layers means the user never sees the complexity. They paste a URL. The AI reads it. The answer arrives. The fallback happens automatically, silently, and in under a few seconds.
This is the same philosophy that runs through the rest of Mirror Ai: the complexity lives in the infrastructure, not in the user experience.
References
Trafilatura: https://trafilatura.readthedocs.io/
Firecrawl: https://www.firecrawl.dev/
OpenRouter: https://openrouter.ai/
readability-lxml: https://github.com/buriy/python-readability
