SysPad › Examples › Web crawler
Web crawler
URL Frontier → HTML Downloader (+ DNS Resolver) → Content Storage + Parser → content dedup (checksum) and URL dedup (Bloom-filter role via Redis). A scheduler re-enqueues known pages for freshness.
A crawl pipeline with dedup stages explicit, and the discovery/recrawl feedback loop modelled honestly as a scheduled re-feed rather than a cycle the simulator can’t traverse.
- Throughput: 400 req/sec
- First to saturate: Nothing at this load
- Estimated cost: $8,174/mo
How it works
- URL Frontier
- The priority and politeness queue. New links discovered by the parser and pages due for recrawl both land here; the downloader pulls at a rate that respects per-host politeness.
- DNS Resolver
- Resolving hostnames is a real bottleneck at crawl scale, so results are cached hard - only a fraction of fetches (the edge multiplier) actually hit the resolver.
- Content Dedup (checksums)
- Stores a checksum per page so re-fetched or mirror content isn’t stored or parsed twice.
- URL Dedup (Bloom filter)
- Stands in for a Bloom filter (a data structure, not a deployable box): a fast set-membership check that keeps already-seen URLs from re-entering the frontier.
Request flows
Crawl a page
- Seed URLs
- URL Frontier
- HTML Downloader
- DNS Resolver
- Content Storage
- Parser + URL Extractor
- Content Dedup (checksums)
- URL Filter (robots.txt)
- URL Dedup (Bloom filter)
Scheduled recrawl
- Recrawl Schedule
- URL Frontier