SysPad › Lessons › analytics lessons › AWS Glue

How AWS Glue works

Serverless Spark ETL. Capacity is provisioned in DPUs (~4 vCPU / 16 GiB each); throughput scales with DPU parallelism and per-record time. Billed per DPU-hour - good for batch/streaming transforms, not OLTP.

A data lake is just a pile of files until something knows what is in them and can reshape them. AWS Glue is that something: it catalogues your data so every query engine agrees on its schema, and it runs the serverless ETL jobs that clean and move it, the connective tissue of the whole lake.

What you will learn

Why does a data lake need a central catalog before queries are useful?
How do crawlers discover schema and fill the Glue Data Catalog?
What is a Glue ETL job, and how is it billed?
How does Glue differ from a query engine like Athena?

One catalog every engine trusts

The Glue Data Catalog is a central metadata store: table names, columns, types, and where the data lives. Crucially it is shared, Athena, Redshift Spectrum, and EMR all read it, so they agree on one definition of each dataset.

Crawlers discover the schema

You rarely type schemas by hand. A crawler scans files in S3 (or other sources), infers their columns and types, detects partitions, and writes or updates the table definitions in the catalog automatically.

⚓ Building a queryable data lake, onboarding new datasets whose schema you do not yet know, and keeping catalog tables in step as new partitions land.

Serverless ETL jobs

Glue also runs ETL: extract, transform, load. Jobs run on serverless Apache Spark (or a lighter Python-shell option for small tasks), and you can author them visually in Glue Studio or in code. Billing is per DPU-hour (a DPU is a unit of processing capacity).

Glue prepares; Athena queries

They are partners, not rivals. Glue catalogues and transforms the data; Athena queries it with SQL. A typical lake uses Glue crawlers to build the catalog and Glue jobs to convert raw data to partitioned Parquet, then Athena to ask questions of the result.

Flashcards

What is the Glue Data Catalog and who uses it?: Central metadata (tables, columns, locations) shared by Athena, Redshift Spectrum, and EMR so they agree on each dataset.
What does a Glue crawler do?: Scans data sources, infers schema and partitions, and populates/updates the catalog automatically.
How do Glue ETL jobs run and bill?: On serverless Spark (or a Python-shell option), billed per DPU-hour; no cluster to manage.
Glue vs Athena, who does what?: Glue catalogues and transforms data; Athena queries it with SQL. They are used together.
How can you author Glue jobs without writing all the code?: With Glue Studio, a visual job builder; you can still drop into code when needed.

Sources

Open the SysPad canvas · Official AWS Glue documentation