SysPad › Examples › Ad click event aggregation
Ad click event aggregation
Click events are deduplicated once, then split into a real-time stream aggregator (fast, approximate) and a raw-event archive feeding a periodic batch reconciliation job (slow, accurate).
The real-time-vs-batch accuracy trade-off, sharing one dedup gate.
- Load: 30,000 req/sec
- First to saturate: Batch Reconciliation
- Estimated cost: $95k/mo
How it works
- Dedup Cache
- The single gate both paths pass through: a click id seen before is dropped once, here, so neither the real-time count nor the archive ever double-counts a retried event.
- Stream Aggregator
- The fast, approximate path: windowed counts land in the OLAP store within seconds, good enough for live dashboards but not guaranteed exact.
- Raw Event Archive
- Every deduplicated event, kept verbatim - the source of truth the nightly batch replays to correct whatever the real-time path got approximately wrong.
- Batch Reconciliation
- The slow, exact path: it re-aggregates the full archive on a schedule and overwrites the approximate numbers in the warehouse with reconciled ones.
Request flows
Count a click
- Ad Clients
- Click Event Stream
- Dedup Cache
- Stream Aggregator
- Real-time OLAP
- Raw Event Archive
Nightly reconciliation
- Raw Event Archive
- Batch Reconciliation
- Ad Warehouse
Lessons for this architecture
- Modelling users and traffic entry points
- How Amazon MSK works
- How Amazon ElastiCache works
- How AWS Lambda works
- How OpenSearch Service works
- How S3 works
- How Amazon EMR works
- How Redshift works
Worked step by step
The same interview question, built up one twist at a time, with a simulation at each stage.