SysPad › Lessons › analytics lessons › Amazon EMR

How Amazon EMR works

Managed Spark/Hadoop/Hive cluster on EC2 - a primary node plus core nodes. Throughput scales with total cluster vCPUs; cost is the EC2 price of every node plus the EMR per-instance uplift. Size nodes + count below.

Some jobs are too big and too custom for serverless SQL: a 10-terabyte Spark pipeline with your own libraries and tuning. Amazon EMR gives you a managed cluster running the real open-source big-data engines, Spark, Hadoop, Hive, so you get full control of the framework without hand-building the cluster.

What you will learn

What does EMR manage for you, and what do you still control?
How is an EMR cluster structured, and what runs on it?
How is EMR billed, and how do spot and transient clusters cut cost?
When do you choose EMR over Athena or Glue?

Managed clusters for real frameworks

EMR (Elastic MapReduce) provisions and manages clusters running open-source big-data frameworks: Spark, Hadoop, Hive, Presto, HBase, and more. AWS handles the cluster plumbing; you keep full control of framework versions, libraries, and tuning.

Primary, core, and task nodes

A cluster has a primary node that coordinates, core nodes that run tasks and store data (HDFS), and optional task nodes that add compute only. It can run on EC2, on EKS, or as EMR Serverless when you do not want to manage instances at all.

⚓ Large-scale ETL, machine-learning feature pipelines, genomics, clickstream and log processing at terabyte-to-petabyte scale.

Cost: EC2 plus an EMR uplift

On EC2, the bill is the instance cost plus a per-instance EMR uplift. Because it is EC2 underneath, you can run spot instances for the task fleet and use transient clusters that spin up, run a job, and terminate, so you pay only while work is happening.

EMR vs Athena vs Glue

All three touch big data, differently. Athena is serverless ad-hoc SQL. Glue is serverless ETL plus the catalog. EMR is for heavy, custom, framework-level processing where you need Spark/Hadoop control, big sustained jobs, or specific libraries.

Flashcards

What does EMR give you that Athena does not?: Managed clusters running open-source frameworks (Spark, Hadoop, Hive) with full version, library, and tuning control.
What are the three EMR node roles?: Primary (coordinates), core (compute + HDFS storage), and task (extra compute, no data).
How is EMR on EC2 billed, and how do you cut cost?: EC2 cost + a per-instance EMR uplift. Cut it with spot instances and transient (auto-terminating) clusters.
Besides EC2, where else can EMR run?: On EKS, or as EMR Serverless when you do not want to manage instances.
When EMR over Athena/Glue?: For heavy, custom framework-level jobs (big Spark/Hadoop, specific libraries, deep tuning); otherwise serverless Athena/Glue is simpler.

Sources

Open the SysPad canvas · Official Amazon EMR documentation