SysPad › Examples › Enterprise data & ML orchestration
Enterprise data & ML orchestration
EventBridge Scheduler triggers an MWAA ETL DAG (extract → S3 lake → Redshift), which pings a legacy on-prem Dkron scheduler mid-migration and hands off to a separate Dagster ML asset pipeline once the warehouse load lands.
Four orchestration tools with four distinct, non-overlapping jobs - AWS-native cron, the core ETL DAG, a legacy-system bridge, and a different team’s downstream asset pipeline - not four ways to do the same thing.
- Throughput: 3 req/sec
- First to saturate: ETL DAG (MWAA)
- Estimated cost: $365k/mo
How it works
- Nightly Trigger
- Just the clock. It answers "when", and nothing else in this diagram has to know what time it is.
- ETL DAG (MWAA)
- Owns the dependency order of the twelve extract and load tasks, and the retries when one fails. This is the piece that knows the warehouse load cannot start until the lake write finished.
- Legacy Sync (Dkron)
- The scheduler the old reporting system already trusts, kept alive during the migration. Rewriting those jobs is a project; calling them from the DAG is an afternoon.
- ML Asset Pipeline (Dagster)
- A different team, a different tool, and deliberately so: they think in assets ("this feature table is fresh"), not in tasks, and they start where the warehouse load ends.