SysPad › Examples › Modern data stack
Modern data stack
A Databricks + Snowflake lakehouse: CDC events land in S3, Databricks does the ETL/ML transform, Snowflake serves governed BI queries. Contrast with the AWS-native Glue/EMR/Redshift pipeline.
Two very different philosophies for the same job - compare the cost shape against the AWS-native analytics pipeline elsewhere in this gallery.
- Throughput: 50 req/sec
- First to saturate: Nothing at this load
- Estimated cost: $4,868/mo
How it works
- CDC Exporter
- Change data capture, not user traffic. The rate here is database changes per second landing in the lake, which is why this example is measured in tens rather than thousands.
- Landing Zone
- Raw events land here first and stay. Storage is cheap enough that fixing a bad transform means rerunning it over these same files rather than asking the source system again.
- Databricks
- Does the transform and the ML work against the lake directly. Billed by cluster hours, so what moves the bill is how long jobs run, not how many rows they read.
- Snowflake
- The governed warehouse the BI team queries. Keeping it separate from the transform layer is what stops one analyst’s heavy query from delaying the pipeline.