SysPad › Lessons › database lessons › OpenSearch Service

How OpenSearch Service works

Managed search and log-analytics engine (OpenSearch / Elasticsearch). Data nodes hold primary and replica shards - replicas multiply search throughput and add resilience. Tune node type, count, and replica copies.

Searching for a word by scanning every document is hopeless at scale, the same problem a book solves with an index at the back. OpenSearch builds that index for your data: it flips "documents contain words" into "words point to documents", which is why it can full-text search and aggregate across billions of log lines in milliseconds.

What you will learn

What’s an inverted index, and why does it make full-text search instant?
How do shards and replicas spread search across a cluster and survive failures?
Why is OpenSearch "near real-time" rather than instantly consistent?
When is it a search/analytics layer, and not your primary database?

The inverted index

OpenSearch stores documents (JSON) and builds an inverted index: for every term, it keeps the list of documents that contain it. Searching a word is then a direct lookup, not a scan.

It also analyses text (lowercasing, stemming, tokenising), so a search for "running" can match "run", which is what makes it feel like real search rather than exact-match.

Shards and replicas

An index is divided into shards, each a self-contained search index living on a data node. A query runs across all shards in parallel and merges the results, so search scales horizontally.

Each shard can have replica shards on other nodes. Replicas serve reads (more throughput) and provide failover if a node dies.

⚓ The engine behind log analytics, observability dashboards, and site/product search.

What do the dedicated "manager" (master) nodes do?

A production cluster uses dedicated manager nodes to track cluster state (which shards live where, node health) while data nodes hold shards and do the searching. Separating these roles keeps the cluster stable under heavy query load.

Near real-time, by design

Newly indexed documents become searchable after a short refresh (about a second by default), not the instant they’re written. OpenSearch batches changes into the index for efficiency.

A search layer, not your source of truth

OpenSearch is superb at search and aggregation but isn’t built for transactions, relational integrity, or being the authoritative store. The common pattern is to keep the system of record elsewhere and stream data into OpenSearch for searching.

Flashcards

What data structure lets OpenSearch find a word across millions of docs instantly?: An inverted index: it maps each term to the list of documents containing it, so search is a lookup, not a scan.
Why can a search for "running" match a document that says "run"?: Text analysis (tokenising, lowercasing, stemming) normalises words at index and query time, so related forms match.
How does OpenSearch scale a search across a large dataset?: It splits the index into shards across data nodes and queries them in parallel, merging results. Replica shards add read throughput and failover.
You index a log line, then query for it 200 ms later and it’s missing. Bug?: No. OpenSearch is near real-time; documents become searchable after a short refresh (~1s by default), not instantly.
Can OpenSearch safely be the only copy of critical transactional data?: No. It’s a search/analytics layer, not a transactional system of record. Keep the authoritative copy elsewhere and index into OpenSearch.

Sources

Open the SysPad canvas · Official OpenSearch Service documentation