Category: Databases
Managed Elasticsearch and OpenSearch: search and logs without disk or memory outages
Your search or logging cluster with memory, shards and disk under control, backups that restore and versions kept current, in your account.
- Where it runs
- In your account, in your name
- Backups
- With tested restores
- Updates
- Planned and documented
- Licence
- OpenSearch: Apache 2.0. Elasticsearch: AGPLv3, Elastic License 2.0 or SSPL, your choice
In short
Elasticsearch and OpenSearch are search and analytics engines that index documents and logs so they can be queried in milliseconds. Managing them means keeping the cluster from turning red for lack of disk or memory. We tune memory and shards, automate the index lifecycle, take snapshots that we restore in tests and plan upgrades, always in your account.
What do we look after?
A search or logging cluster can run fine for months and turn red one morning because a node passed the disk watermark or the JVM ran out of memory. Our job is to make sure growth is anticipated and every index has its lifecycle from day one.
Nodes and roles
Master-eligible nodes, data nodes and, if needed, coordinating nodes, spread across separate machines.
Memory
JVM heap tuned and free memory left for the file system cache.
Shards
Shard size and count per index based on volume and queries.
Lifecycle
ILM or ISM to roll over, move to cheaper nodes and delete what is no longer queried.
Alerts
Disk, memory, cluster health, rejected writes and slow queries.
Security
TLS, roles per application and nothing exposed to the internet.
Which architecture do we recommend?
For production, three master-eligible nodes on separate machines, so that the cluster keeps quorum if one goes down, and data nodes according to volume. In small clusters the three nodes play both roles. With logs that are mostly queried for the last few days, we separate hot nodes with fast disk from warm nodes with more capacity, and the lifecycle moves indices from one to the other.
| Situation | What we set up | Why |
|---|---|---|
| Search for an application | Three nodes with all roles and one replica per index | Simple and survives losing a node |
| High-volume logs | Hot and warm nodes with ILM or ISM | Fast disk only for what is queried daily |
| Your cloud has the managed service | Amazon OpenSearch Service, Elastic Cloud or the DigitalOcean or OVHcloud one | Less maintenance; we tune shards, lifecycle and alerts |
As a guide, we aim for shards of a few tens of gigabytes and avoid thousands of small shards: each one uses memory even if nobody queries it.
Which problems do we come across most often?
- Disk above the watermarks: the cluster stops allocating shards and ends up with indices in read-only mode.
- Heap set wrongly: too small and the JVM pauses constantly; too large and there is no memory left for the cache.
- Too many shards: a daily index with five shards out of habit, multiplied by months of retention.
- No lifecycle: logs from two years ago take up the same expensive disk as yesterday’s.
- Snapshots never restored: configured once and never tested.
- Cluster open to the internet: with security switched off, one of the most frequent causes of data leaks.
How do we test that the backup works?
We configure a snapshot repository on S3, or on compatible storage in another account or with another provider, and a policy that takes them daily and deletes old ones. Every so often we restore indices to a test cluster, check that queries return what is expected and note how long it takes. Mappings, templates and policies live in the repository, so a new cluster can be rebuilt from code and filled from the latest snapshot.
How do we carry out a version upgrade?
We review the release notes, the deprecated APIs your clients use and plugin compatibility. We rehearse on a copy and then upgrade node by node: disable shard allocation, upgrade, wait for the cluster to return to green and move on to the next. Between major versions we first check that old indices can be read, or we reindex them.
How do we migrate from Elasticsearch to OpenSearch, or the other way round?
We set up the target cluster in parallel, reindex from the old one or ingest the data again, and switch the application over once queries give the same results. We review clients, plugins and dashboards, because Kibana and OpenSearch Dashboards are no longer interchangeable. If the reason is the cost of the managed service, we compare it against your bills in cloud migrations and costs.
What does day-to-day support look like?
The cluster’s metrics go to Prometheus and Grafana, as we describe in monitoring with Grafana and Prometheus. Automatic monitoring is continuous; the human response comes the next business day, during business hours, as part of SRE as a service. Every month you receive a report with status, disk growth, the latest restore and changes.
- IngestionApplication, Logstash or agents
- Coordinating nodesOptional, for heavy queries
- Data nodesHot and warm
- Master-eligible nodesThree, for quorum
- SnapshotsS3 or compatible, outside the cluster
Where do we set it up?
In your account, with the provider you already use or the one that fits best. If the data has to stay in the EU, Hetzner and OVHcloud are the European options.
Providers we set it up on
- Hetzner
- AWS
- Google Cloud
- Azure
- DigitalOcean
- OVH
- Own server
| Provider | How we set it up | Things to bear in mind |
|---|---|---|
| Hetzner | Dedicated or cloud servers with Elasticsearch or OpenSearch, local NVMe disks and snapshots in Object Storage. | No managed service: we provide the nodes, the backups and the upgrades. Lots of disk for little cost. |
| AWS | Amazon OpenSearch Service, or Elastic Cloud on AWS; if they do not fit, EC2 or EKS with the corresponding operator. | On Amazon OpenSearch Service we review instance types, shards, ISM and manual snapshots as well as the automatic ones. |
| Google Cloud | Elastic Cloud on Google Cloud, or Compute Engine and GKE with Elasticsearch or OpenSearch. | Google Cloud has no Elasticsearch or OpenSearch service of its own. |
| Azure | Elastic Cloud integrated into Azure, or virtual machines and AKS with Elasticsearch or OpenSearch. | Azure has no managed OpenSearch of its own. |
| DigitalOcean | DigitalOcean managed OpenSearch, or Droplets with the cluster set up by us. | |
| OVH | Managed OpenSearch from OVHcloud Public Cloud Databases, or your own instances. | |
| Own server | Elasticsearch or OpenSearch on your servers, with snapshots to external S3 storage. | Three nodes on a single physical machine will not survive it going down: quorum needs separate machines. |
What is included
- A cluster with three master-eligible nodes and data nodes sized for your volume.
- JVM memory tuned, with headroom left for the file system cache.
- Shards and replicas per index calculated from size and queries, not left at the defaults.
- Index lifecycle with ILM (Elasticsearch) or ISM (OpenSearch): rollover, hot and warm tiers, and deletion.
- Snapshots to S3 or compatible storage, with a documented test restore.
- Alerts for disk before the watermarks, memory, cluster health and slow queries.
- TLS, users and roles per application, and the cluster with no access from the internet.
- Upgrades and migration between Elasticsearch and OpenSearch rehearsed first on a copy.
Frequently asked questions
Elasticsearch or OpenSearch?
If you already use Elasticsearch with Elastic-specific features, such as some of its security or machine learning functions, staying on Elasticsearch is usually sensible. If you want an Apache 2.0 licence with no caveats or your cloud offers managed OpenSearch, OpenSearch is a good option. For ordinary logging and search, both work well.
Can we migrate from Elasticsearch to OpenSearch?
Yes, but not always by restoring a snapshot: it depends on the source version. From older versions, a snapshot may work; from recent ones, the usual route is to reindex from the old cluster or ingest the data again. We review plugins, mappings and clients first, because the APIs have been drifting apart.
What about the Elasticsearch licence?
Since 2024 Elasticsearch can be used under the AGPLv3, as well as the Elastic License 2.0 and the SSPL. Using it within your own company is usually not a problem. We always run it in your account and for your use, never as a service shared between clients.
Why does the cluster turn red?
Almost always because of disk or memory. Once the disk watermarks are passed, the cluster stops allocating shards and ends up blocking writes; with too little memory, JVM pauses knock nodes out. Both can be seen coming with alerts and properly set retention.
Tell us about your case
In a free 20-minute call we look at where you are and where to start. No commitment.
Official documentation consulted
The technical details on this page are based on:
Reviewed on