Elasticsearch Roadmap

Elasticsearch and ES Core

Elasticsearch, often shortened to ES, is a distributed search and analytics engine built on the Apache Lucene library. You send it JSON documents over HTTP, and it indexes them so that you can run full-text searches, filters and aggregations in milliseconds, even over hundreds of millions of documents.

It is used for site search, log and metrics analysis, product catalogues, security analytics and, more recently, vector and hybrid search for AI features. It is not a replacement for a transactional database. It trades strict consistency and joins for fast, flexible search.

A fork called OpenSearch exists, created after a licence change, and most of the basics here apply to it as well. Check which one your target employer runs.

Practice. Write two sentences on a feature in an app you use that must be search-engine backed, not database-backed.

Search Engines vs. Relational Databases Core

A relational database is built for correct updates across related tables: transactions, constraints and joins. A search engine is built to find documents by their content and rank them by relevance, which a LIKE '%word%' query cannot do well or fast.

Elasticsearch denormalises data. Instead of joining orders to customers at query time, you store the customer name inside each order document. Updates are near-real-time, not instantaneous, and there are no multi-document transactions.

The usual architecture keeps the relational database as the source of truth and copies data to Elasticsearch for search. Your job is then to keep the two in step, through change events, a periodic sync or dual writes with care. See SQL Roadmap for the relational side.

Practice. List three queries that suit a search engine and three that suit a relational database.

Elasticsearch Usecases Core

Common use cases fall into a few groups. Application search covers site search, autocomplete, product and document search with typo tolerance and ranking. Observability covers logs, metrics and traces, kept in time-based indices. Security analytics scans events for threats. Analytics and dashboards aggregate business data. Newer uses are semantic and vector search for retrieval-augmented generation.

Each shapes design decisions. Search workloads care about analyzers and relevance. Log workloads care about ingest rate, retention and cost, which leads to data tiers and lifecycle policies.

In an interview, name the workload first. "Log analytics at 200 GB a day" and "product search for 5 million items" lead to different answers on shards, mappings and hardware.

Practice. Pick one use case and note which later stages of this roadmap matter most for it.

The ELK Stack Recommended

ELK stands for Elasticsearch, Logstash and Kibana. Logstash is a pipeline tool that reads data, transforms it and writes to Elasticsearch. Kibana is the web interface for exploring data, building dashboards and sending requests through its console. Lightweight shippers called Beats, such as Filebeat, collect logs on servers, and the combination is now often called the Elastic Stack.

In many modern setups, Elastic Agent and ingest pipelines inside Elasticsearch replace Logstash, or Logstash stays only for heavy transformations. Know the roles of each part, since interviewers still say ELK.

Practice. Draw the data flow from an application log file to a Kibana dashboard.

Running Elasticsearch with Docker Core

Docker is the fastest way to a local single-node cluster. Version 8 enables security by default, so the first start prints a generated password for the elastic user and uses HTTPS with a self-signed certificate.

docker run -d --name es -p 9200:9200   -e "discovery.type=single-node"   -e "xpack.security.enabled=false"   -e "ES_JAVA_OPTS=-Xms1g -Xmx1g"   docker.elastic.co/elasticsearch/elasticsearch:8.15.0
curl localhost:9200

Disabling security is acceptable on a laptop only. Check the current version tag before use. Give the container enough memory, and on Linux raise vm.max_map_count to 262144, or the node will not start. Add a Kibana container to get the console.

Practice. Start a single-node cluster with Docker and call the root endpoint to read the version and cluster name.

REST API Basics and the Kibana Console Core

Everything in Elasticsearch is an HTTP call with JSON bodies. GET /_cluster/health reports health, PUT /products creates an index, POST /products/_doc indexes a document and GET /products/_search searches.

Kibana Console (Dev Tools) is the place to experiment. It takes a compact form with no host and no curl:

GET /_cat/indices?v
PUT /products
GET /products/_search
{ "query": { "match_all": {} } }

Responses carry status codes, and search responses include took, hits.total and hits.hits, each with an _id, a _score and _source. Official clients exist for many languages; they are thin wrappers over these calls.

Practice. In Kibana Console, create an index, add two documents and search them.

Core Vocabulary: Cluster, Node, Index, Document and ID Core

A Cluster (System) is a group of nodes that share data and appear as one service. A Node (Instance) is a single running Elasticsearch process. An Index (Database) is a named collection of documents with a shared mapping. A Document (Row) is a JSON object stored in an index. The ID (Primary Key) is the document's unique key within the index, chosen by you or generated.

The analogies in brackets help, but they are loose. An index is closer to a table than to a database, and since version 7 each index has a single type, _doc. Within a document, a field maps to a column.

Be ready to explain this vocabulary quickly, along with shards, which split an index across nodes.

Practice. Map the terms table, row, column and primary key to Elasticsearch terms in a four-line table.

JSON for Elasticsearch Core

Documents, mappings, queries and responses are all JSON. Know objects, arrays, strings, numbers, booleans and null, and how they map to field types. Arrays have no special type: a field simply holds one or many values of the same type, and "tags": ["a", "b"] is searched as two values.

Objects can nest, and Elasticsearch flattens them to dotted paths such as address.city by default, which affects how arrays of objects behave (see the nested type later). Dates are strings or numbers interpreted by the mapping. Invalid JSON, a trailing comma for example, is the most common beginner error and returns a parse_exception.

Practice. Write a JSON document for a book with nested author and an array of tags, and index it.

The Inverted Index Core

The inverted index is the structure that makes full-text search fast. Instead of storing for each document the words it contains, it stores for each distinct term the list of documents that contain it, along with positions and frequencies.

For two documents, "red fox" and "red car", the index holds red -> [1, 2], fox -> [1], car -> [2]. A search for red reads one list; a search for red fox intersects two. Lookups cost the same however many documents exist.

Lucene writes this data into immutable files called segments. Immutability allows caching and fast reads, and is the reason updates are really delete-and-reindex operations.

Practice. Build an inverted index by hand for four short sentences and answer two queries from it.

Analyzers and the Standard Analyzer Core

An analyzer turns the text of a field into terms, in three steps: character filters clean the raw text, a tokenizer splits it into tokens and token filters change them (lowercase, remove stop words, stem, add synonyms).

The Standard Analyzer is the default for text fields. It splits on word boundaries using Unicode rules and lowercases, and it removes no stop words in current versions. So "The Quick-Brown Fox's" becomes the, quick, brown, fox's.

The same analysis runs when indexing and, by default, when searching a text field, so "FOX" finds "fox". Fields of type keyword skip analysis and keep the exact value.

Practice. Predict the tokens of a sentence with punctuation and numbers under the standard analyzer, then check them.

The Analyze API Core

The Analyze API shows what an analyzer does to a string, so use it whenever search results surprise you.

POST /_analyze
{ "analyzer": "standard", "text": "Quick Brown Foxes!" }

GET /products/_analyze
{ "field": "name", "text": "Quick Brown Foxes!" }

The first form tests a built-in analyzer. The second uses the analyzer configured on a field in an index. The response lists tokens with positions and offsets. You can also test a custom combination by listing tokenizer and filter directly in the request, which speeds up design work.

Practice. Find out how the English analyzer treats "running shoes" and compare with standard.

Custom Analyzers and the Search Analyzer Recommended

When built-in analyzers do not fit, define a custom one in index settings by combining a tokenizer with filters:

PUT /articles
{ "settings": { "analysis": { "analyzer": { "my_english": {
    "type": "custom", "tokenizer": "standard",
    "filter": ["lowercase", "asciifolding", "english_stemmer"] } },
  "filter": { "english_stemmer": { "type": "stemmer", "language": "english" } } } } }

A field can use a different Search Analyzer from the index-time one via search_analyzer in the mapping. A classic case is autocomplete: index with an edge n-gram analyzer so "ela" is stored for "elasticsearch", and search with the standard analyzer so the user's query is not split into grams.

You cannot change an analyzer on an existing field; create a new index and reindex.

Practice. Create an autocomplete field using edge n-grams and test it with partial words.

How Search Works Across Shards Core

An index is split into primary shards, and each shard is a complete Lucene index. A search runs in two phases. In the query phase the coordinating node sends the request to one copy of every shard; each returns its top document ids and scores. In the fetch phase the coordinating node asks the shards holding the winning documents for their _source, merges them and replies.

Because scoring happens per shard, relevance statistics are local to each shard, which can skew scores on small data sets. The search type dfs_query_then_fetch collects global statistics first, at extra cost.

Deep pagination is expensive for this reason: asking for page 10,000 makes every shard return that many entries to be merged.

Practice. Run a search with `?explain=true` and `?search_type=dfs_query_then_fetch` and compare scores on a tiny index.

Mappings: Dynamic and Explicit Core

A mapping defines the fields of an index and the type of each. With Dynamic Mappings, Elasticsearch guesses types from the first value it sees: a string becomes text with a keyword sub-field, a whole number becomes long, a date-like string becomes date. This is handy and risky, because the guess may be wrong and cannot be changed afterwards.

Explicit Mappings are declared up front and are what production systems should use:

PUT /products
{ "mappings": { "dynamic": "strict", "properties": {
    "name":  { "type": "text" },
    "sku":   { "type": "keyword" },
    "price": { "type": "scaled_float", "scaling_factor": 100 } } } }

dynamic: strict rejects unknown fields, false keeps them unindexed. Existing fields cannot change type: add a new field or reindex.

Practice. Index a document with dynamic mapping, read the generated mapping and rewrite it as an explicit one.

Data Types Overview Core

Every field has a type that decides how it is indexed and which queries work on it. The main families are text and keyword for strings, numeric types for numbers, date, boolean, object and nested for structure, geo_point and geo_shape for locations, ip, range types and dense_vector for embeddings.

Choosing correctly matters because search behaviour follows from the type. A text field is analysed and used for full-text search; a keyword field is exact and used for filters, sorting and aggregations. Multi-fields let one source value be indexed in more than one way, for example name as text and name.raw as keyword.

Practice. Create a mapping with at least one field from each family and index a sample document.

Text vs Keyword Data Types Core

Use the Text Data Type for content you search by words: titles, descriptions, comments. It is analysed into terms, supports scoring and phrase queries, but cannot be sorted or aggregated on in the normal way.

Use the Keyword Data Type for identifiers, enums, tags, email addresses, status codes and anything you filter, sort or aggregate on exactly. Keyword values are not analysed, so Berlin and berlin differ unless you add a normaliser.

A common bug is a term query on a text field: the indexed tokens are lowercase pieces, so term: {"name": "Quick Fox"} finds nothing. Query the keyword sub-field, or use match on the text field.

Practice. Show a term query that fails on a text field and succeed with the keyword sub-field.

Numeric, Boolean and Date Types Core

Numeric Data Types include integer, long, short, byte, double, float, half_float and scaled_float. Pick the smallest that fits. Use scaled_float for prices. Use keyword for numeric identifiers you never do range queries on, such as phone numbers, since it makes term lookups faster.

The Boolean Data Type accepts true and false, and strings of those values.

Dates are stored as milliseconds since the epoch (or nanoseconds with date_nanos) and accept formats you declare, for example "format": "strict_date_optional_time||epoch_millis". Always store timestamps in UTC, and use range queries and date histograms for time analysis.

Practice. Define a mapping for an order with a price, paid flag and created date, and query the last 7 days.

Object, Nested and Flattened Types Core

By default an array of objects is flattened, losing the link between fields of the same object. If comments holds {user: "alice", stars: 5} and {user: "bob", stars: 1}, a query for user alice with stars 1 wrongly matches.

The Object Data Type is the default for JSON objects. The Nested Data Type stores each array element as a hidden separate document so that queries with nested match fields within one element. It costs memory and indexing time, and has a default limit on nested objects per document.

The Flattened Data Type indexes a whole object as one keyword field. It protects you from a mapping explosion with arbitrary keys, at the cost of only simple term-style queries.

Practice. Reproduce the false match with an object field and fix it with a nested field and a nested query.

Geo Points Recommended

A geo_point field stores a latitude and longitude. Provide it as an object, a string "41.12,-71.34", an array [lon, lat] (note the order) or a geohash. Queries include geo_distance for points within a radius, geo_bounding_box for a rectangle and geo_polygon, and you can sort by distance or aggregate into geo grids.

GET /shops/_search
{ "query": { "bool": { "filter": { "geo_distance": {
   "distance": "2km", "location": { "lat": 52.52, "lon": 13.40 } } } } } }

For shapes such as regions use geo_shape. The array order [lon, lat] is a frequent source of mistakes.

Practice. Index five shops and find those within 3 km of a point, sorted by distance.

Doc Values and Fielddata Recommended

Sorting and aggregations need to look up values by document, which an inverted index does badly. Doc Values are an on-disk column-oriented structure built at index time for most types, including keyword, numeric and date, and they power sorting and aggregations without using heap memory.

Text fields have no doc values. Fielddata builds the structure in heap memory at query time and is disabled on text fields by default because it can exhaust memory. Do not enable it. Use a keyword sub-field instead. You can turn doc values off for fields you never sort or aggregate on to save disk.

Practice. Try sorting by a text field, read the error and then sort by its keyword sub-field.

Mapping Explosion Recommended

A mapping explosion happens when an index collects too many unique field names, for example if you index user-defined JSON keys, or one key per ID. Each field costs memory in the cluster state, and eventually the cluster slows or nodes run out of memory. A default limit of 1,000 fields (index.mapping.total_fields.limit) stops it with an error.

Prevent it by using explicit mappings, dynamic: strict or false, the flattened type for free-form objects, and a key-value structure like [{"key": "colour", "value": "red"}] in place of dynamic keys. Raising the limit is a last resort.

Practice. Write a script that sends documents with random keys and observe the limit error.

Create Index and Delete Index Core

PUT /orders creates an index, with optional settings (shard and replica counts, analysers) and mappings. DELETE /orders removes it with all its data, immediately and permanently, so be careful with wildcards; the setting action.destructive_requires_name can block DELETE /*.

PUT /orders
{ "settings": { "number_of_shards": 1, "number_of_replicas": 1 } }
DELETE /orders

The number of primary shards is fixed at creation, apart from split and shrink operations, while replicas can change at any time. Use aliases (POST /_aliases) so applications talk to orders while the real index is orders-v2; that makes reindexing without downtime possible.

Practice. Create an index with one shard, add an alias and move the alias to a second index.

Index, Get, Update and Delete Documents Core

To Index Document, send PUT /orders/_doc/1 with your own ID, or POST /orders/_doc and let Elasticsearch generate one. Indexing an existing ID replaces the document and increments its _version. PUT /orders/_create/1 fails if the ID exists.

To Get Document, use GET /orders/_doc/1, which is real-time. Update Document with POST /orders/_update/1 and a partial doc or a script:

POST /orders/_update/1
{ "doc": { "status": "shipped" } }

Remember that an update fetches the document, merges and reindexes the whole thing. Delete Documents with DELETE /orders/_doc/1. New documents become searchable only after the next refresh, about one second by default, so a search right after indexing can miss them unless you pass refresh=wait_for.

Practice. Index a document, update one field, read it back, delete it and watch _version and result values.

Update by Query and Delete by Query Recommended

When many documents change at once, use _update_by_query and _delete_by_query. Both take a query and apply a script or deletion to every match, working in batches on a snapshot of the index taken at the start.

POST /orders/_update_by_query
{ "query": { "term": { "status": "pending" } },
  "script": { "source": "ctx._source.status = 'expired'" } }

POST /logs/_delete_by_query
{ "query": { "range": { "@timestamp": { "lt": "now-30d" } } } }

They can take a long time; run with wait_for_completion=false to receive a task ID and check it through the Tasks API. Deleting whole time-based indices is far cheaper than delete by query, since deleted documents only disappear when segments merge.

Practice. Mark all pending orders older than one day as expired using update by query and verify the count.

Bulk Indexing and Optimizing Bulk Indexing Core

The _bulk API sends many operations in one request, as newline-delimited JSON: an action line, then a source line.

POST /_bulk
{ "index": { "_index": "orders", "_id": "1" } }
{ "total": 30, "status": "new" }
{ "delete": { "_index": "orders", "_id": "7" } }

The response reports each item separately, so check the errors flag and retry failures. Optimizing Bulk Indexing: aim for 5 to 15 MB per request, test to find the best size, use several parallel workers, and back off on 429 responses. For one-off big loads set refresh_interval to -1 and replicas to 0, then restore both afterwards. Let Elasticsearch generate IDs when you can, since checking for existing IDs costs time. Use the official bulk helpers in your language.

Practice. Load 100,000 generated documents using bulk, trying three batch sizes, and record the time for each.

Reindex API Core

_reindex copies documents from one index to another, optionally with a query to filter and a script to change them. It is the standard way to change a mapping, analyser or shard count, since those cannot change in place.

POST /_reindex
{ "source": { "index": "products-v1" },
  "dest": { "index": "products-v2" } }

The recipe for zero downtime: create products-v2 with the new mapping, reindex, switch the alias from v1 to v2 in one atomic call and delete v1 later. Writes during the copy will be missed unless you handle them, so either pause writes, dual-write, or run a second catch-up pass with a query on a timestamp. Reindex from remote clusters is possible and is also used for migrations.

Practice. Change a field from text to keyword by reindexing into a new index and swapping an alias.

Query DSL and Leaf vs. Compound Queries Core

The Query DSL is the JSON language for searches, sent in the query element of a _search request. Queries come in two kinds. Leaf queries look for a value in a field: match, term, range, exists. Compound queries wrap other queries and combine them: bool, boosting, function_score, dis_max.

GET /products/_search
{ "query": { "match": { "name": "wireless headphones" } } }

Think of a query as a tree. Learn to read the nesting from the outside in. Compare Query context, which asks "how well does it match?" and produces a score, with filter context, which asks "does it match?" and does not score.

Practice. Write a bool query nesting a match, a term and a range, and draw it as a tree.

Match and Match Phrase Queries Core

The Match Query is the standard full-text query. It analyses the input and looks for any of the resulting terms, scoring documents that contain more or rarer terms higher. With "operator": "and", all terms must be present, and fuzziness: "AUTO" tolerates typos.

{ "match": { "title": { "query": "quick fox", "operator": "and", "fuzziness": "AUTO" } } }

The Match Phrase Query requires the terms in order and next to each other: "quick fox" does not match "quick brown fox" unless you add "slop": 1. Use multi_match to search several fields at once with boosts, such as ["title^3", "body"], and match_phrase_prefix for search-as-you-type.

Practice. Compare result sets for match with OR, match with AND and match_phrase on the same three sentences.

Term, Prefix and Wildcard Queries Core

The Term Query looks for an exact value, without analysis. Use it on keyword, numeric, boolean and date fields: { "term": { "status": "shipped" } }. terms takes a list of values.

The Prefix Query matches terms starting with a string, and the Wildcard Query supports * and ?. These examine many terms, so are slower. A leading wildcard such as *phone is especially costly because it cannot use the sorted term list efficiently. Avoid it on large indices, or use the wildcard field type, which is designed for it, or an n-gram analyser.

Also available are regexp and fuzzy queries. Never use a term query on an analysed text field, since the indexed form is different from what you type.

Practice. Query a keyword field with term, prefix and wildcard, and compare timings on 1 million documents.

Range, Exists and ID Queries Core

The Range Query filters numbers, dates and strings with gt, gte, lt and lte. For dates, use date maths such as now-7d/d.

{ "range": { "created": { "gte": "now-7d/d", "lt": "now/d" } } }

The Exists Query matches documents where a field has a non-null value, which is the way to find documents with a field, and combined in must_not finds those without it. An empty string or an empty array counts as present or absent in subtle ways; test your data.

The ID Query fetches documents by a list of _id values: { "ids": { "values": ["1", "2"] } }. For simple lookups by known ID, the Get API is faster.

Practice. Find documents missing a field, then documents created in the last 24 hours.

Bool Queries: Must, Should, Must_Not and Filter Context Core

The bool query combines clauses. Must Queries are required and contribute to the score. Should Queries are optional and raise the score when matched; if there is no must or filter, at least one should clause must match (see minimum_should_match). Must_Not Queries exclude matches and never score. The Bool Query Filter Context clause is required but skips scoring, and results are cacheable.

{ "bool": {
   "must":     [ { "match": { "name": "laptop" } } ],
   "filter":   [ { "term": { "in_stock": true } }, { "range": { "price": { "lte": 1500 } } } ],
   "must_not": [ { "term": { "brand": "unknown" } } ],
   "should":   [ { "term": { "featured": true } } ] } }

Put exact conditions in filter for speed and put the text the user typed in must.

Practice. Write a bool query for in-stock laptops under a price limit, preferring featured ones and excluding one brand.

Boosting and Function Score Queries Recommended

Sometimes text relevance is not enough. A Boosting Query lowers the score of documents matching a negative query without excluding them: positive, negative and negative_boost: 0.5. A per-clause boost raises one field's weight, as in title^3.

Function Score Query changes the score with functions: field_value_factor (use popularity or rating), gauss or exp decay (prefer recent or nearby), random_score and scripts.

{ "function_score": { "query": { "match": { "name": "phone" } },
  "field_value_factor": { "field": "sales", "modifier": "log1p" },
  "boost_mode": "multiply" } }

Use a logarithm modifier so that a few runaway values do not swamp the text score, and tune with real queries and a judged test set.

Practice. Boost recent articles with a decay function and compare the ranking before and after.

Lucene Query Syntax and KQL Optional

Lucene Query Syntax is a compact string form used through the query_string and simple_query_string queries: title:fox AND (status:new OR status:open) NOT draft. It supports fields, boolean operators, wildcards, ranges [1 TO 5], phrases and boosts. It is powerful and lets users cause errors or slow queries, so prefer simple_query_string for user input.

KQL, the Kibana Query Language, is the search bar syntax in Kibana: status:error and not host:"web-01". It is translated by Kibana to Query DSL and is easier than Lucene syntax, but only works inside Kibana.

Practice. Write the same filter in Query DSL, Lucene syntax and KQL.

Controlling Search Results: Pagination, Sorting and Source Filtering Core

from and size give simple Pagination, but from + size cannot exceed 10,000 by default, and deep pages cost more. For deeper paging use search_after with a sort that includes a unique tiebreaker, and a point in time (PIT) to keep a consistent view. The old scroll API is meant for bulk export, not user paging.

Sorting uses sort on keyword, numeric or date fields, and _score by default. Sorting on a text field is not allowed.

Source Filtering trims what is returned: "_source": ["name", "price"] or "_source": false. With fields you can return values normalised by the mapping. Smaller responses mean faster searches.

Practice. Page through 50,000 documents using search_after and a PIT.

Highlighting Recommended

Highlighting returns fragments of text with matching terms wrapped in tags, so a results page can show why a document matched.

GET /articles/_search
{ "query": { "match": { "body": "inverted index" } },
  "highlight": { "fields": { "body": { "fragment_size": 120, "number_of_fragments": 2 } },
                 "pre_tags": ["<mark>"], "post_tags": ["</mark>"] } }

Highlights appear in a separate highlight section of each hit. Escape the text before inserting it into HTML. Highlighting can be costly for long fields; for those, use the unified highlighter with offsets in the index or limit the fragment count.

Practice. Show highlighted snippets in a small page and escape the HTML safely.

BM25 Algorithm and Understanding Similarity Core

BM25 is the default relevance function. For each query term it multiplies an inverse document frequency (rare terms matter more) by a term frequency component that saturates, so the tenth repeat adds little, and then normalises by field length so that matches in short fields count for more.

Understanding Similarity means knowing these three ideas and the two knobs: k1 controls term-frequency saturation and b controls length normalisation. They are set per field through a custom similarity in the mapping, and most users never change them.

GET /idx/_search?explain=true shows how each score was computed. Older versions used TF-IDF, and you may still hear the term.

Practice. Use explain on two results and write down which factor made one score higher.

Vector Search Recommended

Vector search finds documents whose embedding is close to a query embedding, which captures meaning instead of exact words. Store vectors in a dense_vector field, and search with a knn clause or query.

PUT /docs
{ "mappings": { "properties": { "embedding": { "type": "dense_vector",
   "dims": 384, "index": true, "similarity": "cosine" } } } }
GET /docs/_search
{ "knn": { "field": "embedding", "query_vector": [0.1, 0.2], "k": 10, "num_candidates": 100 } }

Elasticsearch uses an approximate nearest neighbour index, an HNSW graph, so recall trades against speed through num_candidates. Quantisation options reduce memory. You create embeddings with a model outside, or inside using inference endpoints.

Practice. Embed twenty sentences with any embedding model, index them and run a knn search.

Semantic Search Recommended

Semantic search retrieves by meaning, so "cheap places to sleep" can match "budget hotel". In Elasticsearch it is built from vector search plus embeddings produced by a model, with the semantic_text field type simplifying setup by creating embeddings at index and query time through an inference endpoint. Sparse models such as ELSER give semantic matching with an inverted-index-style representation.

Semantic search can miss exact tokens such as part numbers and names, which is one reason many systems combine it with keyword search. Check the current documentation for field types and models, as this area changes quickly.

Practice. Compare keyword and semantic results for five natural-language questions on a small document set.

Hybrid Search Recommended

Hybrid Search runs a lexical query (BM25) and a vector query, then merges the two ranked lists. Lexical search is strong on exact terms, codes and rare names. Vector search is strong on paraphrase and intent. Together they usually beat either alone.

Merging uses Reciprocal Rank Fusion, which adds 1 / (rank + k) for each list, so no score normalisation is needed, or a weighted sum of normalised scores. In Elasticsearch you express this with the retriever syntax or by putting query and knn in one request, optionally with a reranking step.

Evaluate with real queries and relevance judgements, since the best mix depends on the data.

Practice. Implement RRF in plain code over two result lists and compare it with the built-in retriever.

Metric Aggregations: Avg, Sum, Min, Max, Stats, Value Count and Cardinality Core

Metric aggregations compute a number from field values. Avg, Sum, Min and Max Aggregations each return one value. Stats and Extended Stats Aggregations return count, min, max, avg and sum in one call, and extended stats adds variance and standard deviation. The Value Count Aggregation counts values, and the Cardinality Aggregation estimates distinct values using HyperLogLog++, which is approximate and cheap.

GET /orders/_search
{ "size": 0, "aggs": {
   "revenue": { "sum": { "field": "total" } },
   "customers": { "cardinality": { "field": "customer_id" } } } }

Use "size": 0 to skip the hits when you only want the summary. Cardinality is not exact above the precision threshold, and percentiles are also approximate.

Practice. Compute revenue, average order value and distinct customers in one request.

Bucket Aggregations: Terms, Histogram and Range Core

Bucket aggregations group documents. The Terms Aggregation makes one bucket per distinct value of a keyword field and returns the top ones, such as top categories. Counts across shards are approximate when there are many unique terms; raise shard_size for accuracy.

The Histogram Aggregation makes fixed-width numeric buckets, and date_histogram makes time buckets by calendar or fixed intervals, central to dashboards. The Range/Date Range Aggregations make buckets from ranges you define, such as price bands.

{ "aggs": { "per_day": { "date_histogram": { "field": "@timestamp", "calendar_interval": "day" } } } }

Buckets can hold sub-aggregations, which is where the real power comes from.

Practice. Build a daily revenue histogram and a price-band breakdown.

Filter and Nested Aggregations Recommended

The Filter Aggregations create a single bucket from documents matching a query, such as "errors only", which lets you compute different metrics on different subsets in one request. filters makes several named buckets.

Nested Aggregations come in two meanings. Nesting means placing aggregations inside buckets, as in revenue per country per month. The nested aggregation is the special form needed to aggregate inside fields of the nested type, followed by reverse_nested to go back up to parent documents.

{ "aggs": { "by_country": { "terms": { "field": "country" },
    "aggs": { "avg_total": { "avg": { "field": "total" } } } } } }

Practice. Compute average order total per country, then only for orders with a refund.

Pipeline Aggregations Optional

Pipeline aggregations take the output of other aggregations as input, instead of document fields. Examples are derivative (rate of change between buckets), cumulative_sum, moving_avg style smoothing through moving_fn, bucket_sort, bucket_selector and avg_bucket.

{ "aggs": { "per_day": { "date_histogram": { "field": "date", "calendar_interval": "day" },
    "aggs": { "sales": { "sum": { "field": "total" } },
              "growth": { "derivative": { "buckets_path": "sales" } } } } } }

buckets_path points at the metric to use. They suit reports with running totals and growth, but move work to the coordinating node, so keep bucket counts moderate.

Practice. Show cumulative revenue per day using a cumulative_sum pipeline aggregation.

Node Roles: Master-Eligible, Data and Coordinating Nodes Core

Nodes have roles. Master-Eligible Nodes can be elected master, which keeps cluster state: index metadata, mappings and shard locations. Data Nodes hold shards and do the indexing and search work; they come in tiers such as hot, warm, cold and frozen. Coordinating Nodes (any node, or dedicated ones with no other roles) accept requests, fan them out and merge results. Others include ingest nodes, machine learning nodes and transform nodes.

Small clusters give every node several roles. Larger ones use three dedicated master-eligible nodes for stability, so that heavy queries never starve the master. Know which node role fails in which scenario.

Practice. Sketch a 9-node cluster with dedicated roles and say what happens if one master-eligible node dies.

Primary and Replica Shards Core

An index is split into Primary Shards, each holding a part of the data, chosen by hashing the document ID (or routing value). Replica Shards are copies of primaries on other nodes. They give failover, since a replica is promoted if its primary is lost, and extra read capacity.

The primary count is fixed at index creation, so plan it. Too many small shards waste heap and slow the cluster; too few large ones are slow to recover. A common guide is to keep shards between 10 and 50 GB and to avoid thousands of shards per node.

number_of_replicas: 1 doubles storage. Replicas are never on the same node as their primary, which is why a single-node cluster shows yellow status.

Practice. Create an index with 3 primaries and 1 replica on a 3-node cluster and list shard placement with _cat/shards.

The Split Brain Problem Recommended

Split brain is when a network partition leaves two groups of nodes that each elect their own master and accept different writes, so the data diverges. Elasticsearch before version 7 required careful configuration of minimum_master_nodes to prevent it.

Since version 7, the cluster coordination layer manages voting itself. It requires a majority of master-eligible nodes to elect a master, tracked in a voting configuration that adjusts as nodes join or leave. This is why you should run three master-eligible nodes, not two: with two, losing one means no majority. With three you tolerate one failure.

Practice. Explain why an even number of master-eligible nodes gives no extra safety.

Segment Merging Recommended

Each refresh writes a new small segment, and many segments slow searches. Background merging combines small segments into larger ones and drops deleted documents at the same time. Merges use disk I/O and CPU, and are throttled automatically.

This is why deleted and updated documents free their space late, and why disk usage can briefly rise during a merge. For an index that will not change again, the _forcemerge API with max_num_segments=1 makes searches a little faster; never use it on an index still receiving writes.

Practice. Index many small batches, watch segment counts with _cat/segments and run a force merge on a read-only index.

CAT API and Cluster Monitoring Core

The CAT API returns compact, human-readable tables: _cat/health, _cat/nodes?v, _cat/indices?v, _cat/shards?v, _cat/allocation?v and _cat/thread_pool. Add ?v for headers, and ?h= to choose columns.

For Cluster Monitoring, GET _cluster/health returns a status: green (all shards allocated), yellow (primaries yes, some replicas missing) or red (a primary is missing, so some data is unavailable). GET _cluster/allocation/explain tells you why a shard is unassigned, and GET _nodes/stats shows heap, GC, CPU and disk.

Watch heap use, disk watermarks (at which allocation stops), rejected thread pool tasks, search and indexing latency, and pending tasks. Kibana's Stack Monitoring or Prometheus exporters provide dashboards.

Practice. Cause a yellow state by adding a replica on a single-node cluster, then explain it with allocation explain.

Performance Tuning Basics Recommended

Start with the data model and the queries, not hardware. Use filters for exact conditions, avoid leading wildcards, avoid deep pagination and script queries, and keep mappings tight. Give the JVM heap at most half of RAM, up to roughly 31 GB, so compressed pointers still work, and leave the rest to the filesystem cache, which Lucene relies on.

For indexing speed, use bulk, longer refresh intervals and fewer replicas during loads. For search speed, use fewer, larger shards, preferring routing when queries target a single tenant, and use SSDs. Measure with the slow logs (index.search.slowlog.threshold.query.warn) and the profile API, "profile": true.

Practice. Run a query with profile enabled and find the most expensive component.

Index Lifecycle Management, Rollover Policies and Data Tiers Core

Time-based data such as logs should be managed automatically. Index Lifecycle Management (ILM) policies define phases: hot (actively written), warm, cold, frozen and delete. Rollover Policies create a new index when the current one reaches a size, document count or age, for example 50 GB or 7 days, so that shards stay a healthy size.

PUT _ilm/policy/logs
{ "policy": { "phases": {
  "hot":    { "actions": { "rollover": { "max_primary_shard_size": "50gb", "max_age": "7d" } } },
  "delete": { "min_age": "30d", "actions": { "delete": {} } } } } }

Data Tiers map phases to hardware: fast storage for hot, cheaper disks for warm and cold, and searchable snapshots for frozen. Data streams combine a write alias, rollover and backing indices, and are the standard for logs and metrics.

Practice. Create a data stream with a policy that rolls over at 1 GB and deletes after 7 days.

Snapshots, Restores and SLM Core

Replicas protect against node failure, not against deleting data by mistake or losing the cluster. Snapshots do. Register a repository (an S3 bucket, shared file system or similar), then PUT _snapshot/my_repo/snap1 takes a snapshot. Snapshots are incremental: only new segments are copied.

Restores use POST _snapshot/my_repo/snap1/_restore, and can rename indices to restore side by side. SLM (snapshot lifecycle management) schedules snapshots and retention, such as a nightly one kept for 30 days.

Test restores. A backup that never has been restored is a guess.

Practice. Snapshot an index to a local repository, delete it and restore it under a new name.

Cross-Cluster Replication and Autoscaling Optional

Cross-Cluster Replication (CCR) copies indices from a leader cluster to a follower cluster in another location, for disaster recovery or reading data closer to users. It is a licensed feature. Cross-cluster search queries several clusters in one request without copying.

Autoscaling lets Elastic Cloud and orchestration setups on Kubernetes adjust capacity to need, using deciders based on storage or memory, and recommending sizes for tiers. It does not solve poor shard design; it only adds hardware.

Practice. Describe a disaster recovery design using CCR for a search cluster in two regions.

Elastic Cloud Recommended

Elastic Cloud is the managed service from Elastic, running deployments on major cloud providers. It handles provisioning, upgrades, snapshots, security defaults and scaling, and offers serverless projects where you do not size nodes. Cloud providers and other vendors offer managed Elasticsearch or OpenSearch as well.

Managed services remove most operations work, so the interview focus shifts to data modelling, shard and tier choices, cost, and relevance. Compare self-hosted and managed costs including engineer time.

Practice. Compare the monthly cost of a self-run three-node cluster and a managed deployment of equal size.

Authentication, Roles and Users, and API Keys Core

Since 8.0 security is on by default, with TLS between nodes and for HTTP. Authentication verifies who the caller is: native users, LDAP, SAML, OpenID Connect or tokens. Roles and Users assign permissions: a role lists cluster privileges and index privileges, optionally limited to certain fields (field-level security) or documents (document-level security through a query).

POST /_security/role/orders_reader
{ "indices": [ { "names": ["orders-*"], "privileges": ["read"] } ] }

API Keys let applications authenticate without passwords. Create them with limited roles and an expiry, and revoke them without touching users. Apply least privilege, so an application that only searches cannot delete indices.

Practice. Create a role that can read only one index and an API key bound to it, then test a forbidden call.

Elasticsearch SQL, ES|QL and Event Query Language (EQL) Recommended

Elasticsearch SQL translates SQL queries to Query DSL, helpful for analysts and BI tools via JDBC: POST _sql?format=txt with SELECT name FROM products LIMIT 5. It supports a subset, with no general joins.

ES|QL is a newer piped query language, with a flow such as FROM logs | WHERE status >= 500 | STATS count = COUNT(*) BY host | SORT count DESC. It runs on its own execution engine and is built for analytics and exploration.

The Event Query Language (EQL) targets event sequences, mostly for security: sequence by host.id [process where name == "cmd.exe"] [network where true] finds events in order. Learn Query DSL first, then these as conveniences.

Practice. Write the same top-hosts-by-errors question in SQL, ES|QL and Query DSL.

Transform API: Pivot and Latest Transformations Optional

Transforms convert an index into a summarised entity-centric index that updates continuously or in batches. The Transform API supports two types. A Pivot Transformation groups by one or more fields and applies aggregations, for example one document per customer with total spend and order count. A Latest Transformation keeps only the most recent document per key, for example the last known status per device.

PUT _transform/customer_totals
{ "source": { "index": "orders" }, "dest": { "index": "customers" },
  "pivot": { "group_by": { "customer": { "terms": { "field": "customer_id" } } },
             "aggregations": { "spend": { "sum": { "field": "total" } } } } }

Use them to precompute heavy aggregations for dashboards.

Practice. Build a pivot transform that outputs one document per customer with spend and order count.