SysPad › Examples › Ad click event aggregation
Ad click event aggregation
Click events are deduplicated once, then split into a real-time stream aggregator (fast, approximate) and a raw-event archive feeding a periodic batch reconciliation job (slow, accurate).
The real-time-vs-batch accuracy trade-off, sharing one dedup gate.
- Throughput: 30,000 req/sec
- First to saturate: Batch Reconciliation
- Estimated cost: $87k/mo
How it works
- Dedup Cache
- The single gate both paths pass through: a click id seen before is dropped once, here, so neither the real-time count nor the archive ever double-counts a retried event.
- Stream Aggregator
- The fast, approximate path: windowed counts land in the OLAP store within seconds, good enough for live dashboards but not guaranteed exact.
- Raw Event Archive
- Every deduplicated event, kept verbatim - the source of truth the nightly batch replays to correct whatever the real-time path got approximately wrong.
- Batch Reconciliation
- The slow, exact path: it re-aggregates the full archive on a schedule and overwrites the approximate numbers in the warehouse with reconciled ones.
Request flows
Count a click
- Ad Clients
- Click Event Stream
- Dedup Cache
- Stream Aggregator
- Real-time OLAP
- Raw Event Archive
Nightly reconciliation
- Raw Event Archive
- Batch Reconciliation
- Ad Warehouse