# IDEA.md – a crawler of your own for technical SEO This is a self-contained brief. Give it in full to Claude Code (or a similar tool) to build an implementation of your own. It draws on what was learned building SEO Spider, a desktop app for Windows and macOS written in Rust with Claude Code. ## 1. What to build A desktop crawler for technical SEO analysis modelled on Screaming Frog SEO Spider – the same scope and terminology, but considerably faster and lighter on memory, and without a subscription. Plus a CLI that runs the same engine without a window. The app must be able to: - **crawl a site** from a start URL respecting robots.txt (including crawl-delay), scope (internal/external, include/exclude), depth and count limits, and concurrency; pause and resume, even after a restart - show the result in **tabs with filters** using Screaming Frog's names: Internal, External, Response Codes, Page Titles, Meta Description, H1, Content, Images, Canonicals, Pagination & AMP, Hreflang, Accessibility, Cookies, Structured Data, Sitemaps, JavaScript, Directives and more – for example titles that are missing, duplicated, over 60 or below 30 characters - a **virtualised table** with sorting, free-text search and export, and a **detail panel** per URL: details, inlinks, outlinks, duplicates, hreflang, headers, cookies, structured data, accessibility - **link graph and link score** (PageRank over internal pages), orphan pages - **XML sitemaps**: read, index, gzip, compare with the crawl, generate - **JavaScript rendering** in headless Chrome via CDP, with a raw/rendered comparison - **structured data** (JSON-LD against Google's rich result requirements), **hreflang** (return links, self-reference, x-default, codes against ISO; from ``, the `Link:` header and sitemaps), **pagination and AMP** - **custom extraction** (CSS selectors and regular expressions) and **custom search** in the source, in the same parsing pass - **list mode**, **near duplicates** (MinHash with LSH), **crawl comparison**, **accessibility** (static rules or axe-core in the rendered page), **cookies** with their flags, **authentication** (HTTP Basic and custom headers) - **visualisations** (crawl tree, directory tree, force-directed graph, word cloud), **scheduling** and a **crawl library** - **recalculate** a saved crawl without fetching the site again, when the raw HTML was stored - Search Console, Analytics and PageSpeed Insights through Google's APIs **Not** included: XPath in custom extraction (needs a document tree), form login with a session across the crawl, and cloud services. ## 2. Stack and architecture What **matters** is marked; the rest can be swapped. - **A Rust workspace**: `parse` (HTML extraction, pure CPU, no I/O), `core` (the crawl engine: HTTP, robots, scope, queue, link graph, sitemaps, rendering, structured data, MinHash, Google clients), `store` (storage and queries), `cli`. On top of it a **Tauri 2 app** with React and Vite. - **Important: streaming extraction with `lol_html`, no DOM tree.** A single tokenising pass picks everything up on the way – no tree allocation. That's the whole performance point. All new work goes into the same pass, not into an extra scan of the document. - **Important: a coordinator without locks.** The queue and the set of seen URLs are owned by a single task. The workers only do I/O and parsing and send finished results back over a channel. No `Mutex` in the hot path. - **Storage in redb and queries in Polars.** Crawl data is written to redb during the run and materialised into Arrow batches that Polars queries, so sorting millions of rows is a column read. - **The link graph outside the records**, in two tables keyed `from‖to` and `to‖from`. The inlinks panel becomes a sequential read instead of a scan of the whole graph. - The interface: a virtualised grid (only visible rows are drawn), tabs and filters as data, light and dark themes. All user-facing text in English, with Screaming Frog's terms. ## 3. Decisions that turned out to matter **Store exactly what was sent and received** – all response headers and request headers – so a report months later shows what happened. Read header values as bytes, not only as visible ASCII. Capture `Content-Encoding` and `Content-Length` before the HTTP client decompresses the response, or they disappear along with the true size. **But mask what the user added.** Headers from authentication and custom headers are stored with the name kept and the value replaced. The headers the crawler sets itself (`accept`, `user-agent` …) are stored as they are. Test both directions: the header must still be sent and must not reach the file. **`X-Robots-Tag` matters as much as the robots meta tag.** Read it before parsing – it applies to PDFs and other responses without HTML too. **External URLs are only checked.** Fetch the status code and headers, but not the body. Follow sitemaps from robots.txt only for internal hosts. Otherwise a single external link can pull in other sites' sitemaps, with new external hosts and new robots.txt files – a chain reaction. **Optional storage of the raw HTML**, compressed in the crawl file. New checks can then be run against an old crawl without fetching the site again. **A persistent queue**, so a paused crawl can be saved, reopened and resumed where it stopped. **Each renderer gets its own Chrome profile** in a temporary directory, and old profiles are cleaned up. Otherwise two rendering crawls at the same time kill each other. **Save a crawl through the database's own handle**, record by record. Never copy the open file behind the database's back. **Terminology like Screaming Frog** – `Response Codes`, `Indexability Status`, `Orphan URLs`, `Link Score` – so results can be compared between the tools. **Developer help from day one**: startup arguments like `--url` and `--crawl`, a CLI with the same engine and the same reports, and a script that finds the app window, clicks, types and takes a screenshot of just that window. ## 4. Pitfalls - **File locks behave differently.** On Windows a lock is binding for all other handles, even reading ones; on Linux `flock` is only advisory. A bug that only exists on Windows never shows when trying it on Linux. - **Tauri must be built with `--features custom-protocol`** for a distributed binary. Without it the app loads the interface from the dev server on localhost, and the bug shows only for someone starting the package. Check afterwards that the frontend's files are in the binary. - **Don't build the frontend into `dist/`** if `dist/` holds the distribution packages – Vite empties the directory. - **Don't build in release to try behaviour.** With `lto` and `codegen-units = 1` the release build takes about 20 times as long as dev; functional checks give the same answer. - **Under WSL: put `target/` on ext4**, not on the Windows drive through 9p, where cargo's many small writes are very slow. Don't set a shared `CARGO_TARGET_DIR` for every project. - **Under WSLg** the window must be started with `GDK_BACKEND=x11` for X tools to see it, and WebKitGTK needs `WEBKIT_DISABLE_COMPOSITING_MODE=1`. Check that the screenshot tool picks the right window. - **Don't measure crawl throughput against `python3 -m http.server`** – it collapses at many concurrent connections and measures the server, not the crawler. - **Don't trust comments.** "The values are never written to the crawl file" stood in the code while they were. Test what the comment promises. ## 5. Requirements and measurements - Extraction on one thread: around 34,000 small pages (1.6 KB) per second and around 1,900 realistic pages (62 KB, 180 links, 120 images) per second. With many cores, parsing never becomes the bottleneck. - Sorting and filtering hundreds of thousands of rows without waiting. - The Rust side with over 350 tests, the frontend with its own tests of the pure functions, type checking and lint without warnings. ## 6. How it's verified - `cargo test --workspace`, `cargo clippy --workspace --all-targets -- -D warnings`, the frontend's tests, type checking and lint. - Integration tests against a real local server: that the crawl reports exactly the headers it sends, redirects, robots, sitemaps, `X-Robots-Tag`. - The rendering test with an installed Chrome, separately. - The Windows part with `cargo xwin` from WSL, and the running app on each platform. - **In the running interface**: crawling, switching tabs, filters, free-text search, scrolling past the row window, the detail panel and the settings – and that the numbers in the interface match the CLI's. ## 7. Way of working - **Research first**: Screaming Frog and other crawlers, what each tab and filter means and how Google interprets the directives. Mark what was checked against a source and what is judgement. - **One task at a time**, with the measurement in the closing comment. - **Try it against real sites.** Several of the most important bugs only showed there: the queue that exploded, the headers that were missing compared with the browser's network panel. - **What needs a real account** – Google's APIs, signing – becomes a verification task of its own with a checklist. ## 8. The prompt > Build a desktop crawler for technical SEO for [Windows and macOS] in [Rust with Tauri 2 and React], modelled on Screaming Frog SEO Spider: the same tabs, filters and terminology, but faster and lighter on memory. It should crawl a site respecting robots.txt and scope, show the result in a virtualised table with filters and a detail panel, and handle JavaScript rendering, sitemaps, hreflang, structured data, custom extraction and list mode. A CLI should run the same engine. > > Read all of IDEA.md below first. It describes the features, the architecture, the decisions that turned out to matter and the pitfalls to avoid. > > Work like this: > 1. Research and write a plan split into tasks. Ask me about the decisions that are mine. > 2. Extract everything in a single streaming parsing pass, without a DOM tree. > 3. Let a single task own the queue, with no locks in the hot path. > 4. Never store passwords or keys in the crawl file. > 5. Try it in the running interface and compare the numbers with the CLI. > > [paste IDEA.md here]