How Storage Node (EC2 instance store) works

A node that owns its disks

A storage node is an EC2 instance that keeps its data on instance store. These disks are inside the host, not on the network like EBS. Cassandra rings, HDFS data tiers and home-built object stores run on this shape.

The d3, d3en and h1 families have hard disks. The i4i and i8g families have NVMe SSDs. For each size, AWS lists the disk count and the disk size. It lists a disk speed for all but h1, where SysPad derives one from the family figure.

A hard disk pays for every seek

A hard disk has one arm. Each random read moves the arm first. That seek costs about 8.3 ms, so one drive does about 120 random reads a second.

On d3en.12xlarge, each of the 24 drives streams about 258 MiB/s. A 4 KiB read spends 0.02 ms on transfer and 8.3 ms on the seek. A 1 MiB read spends 3.9 ms on transfer.

Why does an NVMe node have no seek time on SysPad?

An SSD has no arm. AWS publishes NVMe IOPS at 4 KiB, for example 400,000 reads and 220,000 writes a second on i4i.4xlarge. SysPad counts a large read as many 4 KiB I/Os. AWS publishes no NVMe MiB/s, so that count is a floor, not an exact figure.

Copies are load, and bytes

With replicas, each write lands on N nodes and each read touches R copies. With erasure coding k + m, each write lands as k + m shards, each one k-th of the value. A read then reads k shards.

3 copies store 3 bytes for each byte of data. RS 6+3, the HDFS default policy, stores 1.5 bytes. On 10 d3en.12xlarge nodes, 1,100,000 GiB at 3 copies fills 106% of each node. At RS 6+3 it fills 53%.

⚓ On the SysPad node, set Redundancy and the copy or shard counts on the card itself. The edge carries client operations only. Do not multiply the edge by the copies, or the load is counted twice.

Which wall binds first

The network is a wall too. i4i.4xlarge has 400,000 read IOPS at 4 KiB. Its 9.375 Gbps baseline sends only about 286,000 reads of 4 KiB a second. So the network binds before the disk.

Bytes held are a state, not a rate. A full node is marked saturated at any request rate. Its read and write ceiling does not change.

⚓ On the SysPad node, set Data held to check bytes. Set Measured ops per node from a test on the same instance type. Both are 0 by default, which means not checked.

Why not use the 25 Gbps burst figure?

Burst bandwidth runs on network credits and lasts 5 to 60 minutes. A storage node serves its load all day, so SysPad uses the baseline in each direction. The warning names the burst figure so you can see the headroom.