SysPad › Lessons › analytics lessons › Databricks

How Databricks works

Databricks lakehouse - analytics and ML compute, usually the sink of a data pipeline. Billed by DBUs consumed while clusters run.

For years teams kept two data systems: a cheap, flexible data lake (files in S3) and a reliable but rigid warehouse, copying data between them and fighting drift. Databricks’ big idea is the lakehouse: bring warehouse reliability directly to the lake’s files, so one system, powered by Spark, serves ETL, SQL analytics, and machine learning.

What you will learn

What problem does the "lakehouse" solve that a lake + warehouse created?
How does Delta Lake add reliability to plain files?
What does Spark do, and why distributed compute?
When is Databricks a fit versus a pure data warehouse?

The lakehouse idea

A data lake stores raw files cheaply in object storage but offers no transactions or schema guarantees. A warehouse is reliable and fast for SQL but rigid and separate. Keeping both means duplicating data and reconciling two worlds.

The lakehouse collapses them: warehouse-grade reliability applied directly to the lake’s files, so you keep one copy of data and one platform.

Delta Lake: reliability on files

Delta Lake is the storage layer that makes it work. It adds ACID transactions, schema enforcement, and time travel (versioned data) on top of ordinary Parquet files in object storage. So files behave like reliable tables, no more half-written, inconsistent data.

⚓ Big-data ETL, streaming + batch unification, BI on the lake, and ML feature pipelines.

Spark: distributed compute

Databricks runs on Apache Spark: a driver splits a job into tasks across a cluster of workers that process partitions of the data in parallel. That’s how it crunches datasets far too big for one machine, for ETL, SQL, and ML alike.

Databricks vs a pure warehouse

Choose Databricks when you need data engineering at scale, streaming, machine learning, and flexibility over raw/semi-structured data, all on one copy. Choose a pure warehouse (Snowflake/BigQuery/Redshift) when the job is mostly SQL analytics on structured data and you want maximum simplicity.

Flashcards

What is a lakehouse, and what pain does it remove?: Warehouse reliability applied to data-lake files, one system instead of a separate lake and warehouse with duplicated, drifting data.
What does Delta Lake add to plain Parquet files?: ACID transactions, schema enforcement, and time travel, so files behave like reliable tables.
How does Databricks process datasets too big for one machine?: Apache Spark: a driver splits work into tasks across a cluster of workers processing data partitions in parallel.
What workloads run on the one lakehouse copy of data?: ETL, SQL analytics, and machine learning, unified on a single dataset rather than copies across systems.
Databricks vs a pure warehouse, when each?: Databricks for data-engineering/streaming/ML over raw/semi-structured data. Pure warehouse for mostly-SQL analytics on structured data with max simplicity.

Sources

Open the SysPad canvas · Official Databricks documentation