Category: Observability
Monitoring with Grafana and Prometheus, deployed and operated
Metrics, logs and alerts that warn you about what matters to your users, not every CPU spike, on a platform that lives in your account.
- Where it runs
- In your account, in your name
- Backups
- With tested restores
- Updates
- Planned and documented
In short
Grafana and Prometheus are open source tools for collecting metrics, visualising them and generating alerts; Loki does the same with logs. We deploy them in your account with Alloy or OpenTelemetry as the collector, define service level objectives (SLOs), configure Alertmanager so that alerts reach the right person without noise, build per-service dashboards and keep an eye on retention and cost.
Which component does what?
A monitoring platform is not a pretty dashboard: it is a chain that collects data, stores it, decides whether something is wrong and alerts the right person. If one link fails, you find out from a customer.
Prometheus
Collects and stores metrics, and evaluates the alerting rules every few seconds.
Loki
Stores logs indexing only their labels, which makes it cheap to store.
Alloy or OpenTelemetry
Collectors on each server or cluster that send metrics, logs and traces.
Alertmanager
Groups, routes, inhibits and silences alerts before sending them to anyone.
Grafana
Per-service dashboards and exploration of metrics and logs in one place.
External check
A probe from outside that alerts you even if the monitoring itself goes down.
How do we design alerts so they are not noisy?
Most of the alerts we come across warn about causes, such as high CPU or memory at 80%, rather than symptoms. The result is that they go off often, nobody looks at them and the one that matters gets lost. We approach it the other way round:
Service level objectives
We define with you what working well means: for example, that 99.5% of requests are answered without error. That is an SLO.
Alerts on error budget burn
You are alerted when the error budget is being used up too quickly, not for every isolated spike.
Routes in Alertmanager
Each alert goes to the right team and channel; urgent ones go to on-call if there is one, and the rest to a work queue.
Grouping and inhibition
If a server goes down, you get one alert, not forty. If the data centre is not responding, the alerts for its services are inhibited.
Every alert carries a link to its dashboard and a note on what to check first. An alert with no possible action is removed.
What problems do we find in existing monitoring?
- Hundreds of alerts nobody responds to: the alerts channel has been muted for months.
- Labels with too many values: user or request IDs used as labels blow up Prometheus memory or the Loki index.
- Logs with no retention: the disk fills up with logs nobody reads.
- Hand-made dashboards: they only exist in the interface and are lost on reinstall.
- Monitoring runs on the same server it watches: if it goes down, nobody is alerted.
- Nothing measures what the user sees: everything is green and the website will not load.
How do we keep retention and cost under control?
With pay-as-you-go services, the cost grows with every host and every gigabyte of logs. In your own stack, what grows is storage, and that can be controlled: short retention at full detail, summarised data for history, object storage for older data and filters in the collector so that logs that add nothing are not stored. We review it every month along with the rest of the report.
What do we monitor for each application?
Each application has its own signals. In PostgreSQL, replication lag and WAL archiving; in Kafka, consumer lag; in Kubernetes on Hetzner, the control plane and the nodes; in GitLab, the runners and the registry; in Keycloak, failed sign-ins. If what you want is someone to answer for all of this every month, that is covered by SRE as a service.
What about Graylog?
Graylog indexes the full text of logs on top of OpenSearch: it searches very well, but needs more disk and memory than Loki, which only indexes labels, and for most teams Loki alongside Grafana is enough. If you often search the content of your logs or already have it, we explain it in log centralisation with Graylog or OpenSearch.
Where do we set it up?
In your account, with the provider you already use or the one that fits best. If the data has to stay in the EU, Hetzner and OVHcloud are the European options.
Providers we set it up on
- Hetzner
- AWS
- Google Cloud
- Azure
- DigitalOcean
- OVH
- Own server
| Provider | How we set it up | Things to bear in mind |
|---|---|---|
| Hetzner | Prometheus, Loki and Grafana on cloud servers or in your cluster, with the Loki data and the metrics history in Hetzner Object Storage. | No managed monitoring service: we operate it all ourselves. Very cheap on storage. |
| AWS | The stack on EC2 or EKS with S3 as storage, or Amazon Managed Service for Prometheus and Amazon Managed Grafana. | CloudWatch is still needed for some AWS services; we integrate it as a data source. |
| Google Cloud | The stack on GKE with Cloud Storage, or Google Cloud Managed Service for Prometheus. | Cloud Monitoring can be queried from Grafana. |
| Azure | The stack on AKS with Blob Storage, or Azure Monitor managed service for Prometheus with Azure Managed Grafana. | Azure Monitor is added as a data source in Grafana. |
| DigitalOcean | The stack on Droplets or DOKS, with Spaces as object storage. | No equivalent managed service: we set it up and operate it ourselves. |
| OVH | The stack on Public Cloud instances or Managed Kubernetes, with S3-compatible Object Storage. | |
| Own server | Prometheus, Loki and Grafana on an office server or a virtual machine. | The watcher must not go down with what it watches: we add an external check from another location. |
What is included
- Prometheus or a compatible store for metrics, with defined retention and long-term storage.
- Loki for logs, with labels designed so that searches are fast and cheap.
- Grafana Alloy or OpenTelemetry Collector on servers, containers and Kubernetes.
- Service level objectives (SLOs) for what your users notice: availability, latency and errors.
- Alertmanager with routes, grouping, inhibitions and silences, connected to email, chat or your on-call tool.
- Per-service dashboards in Grafana, stored as code in your repository.
- Sign-in to Grafana with your identity provider.
- Monthly review of alerts: those that serve no purpose are removed or adjusted.
Frequently asked questions
Are Grafana and Prometheus an alternative to Datadog?
For metrics, logs, dashboards and alerts, yes. You lose the convenience of a service you can install in an afternoon and gain control over the data and the cost, which does not grow per host or per gigabyte ingested. What you need is someone to operate the stack; that is our job.
Who handles alerts at night?
Automatic monitoring is continuous: alerts fire and are recorded at any hour. Human response is the next business day, during business hours, with response times agreed in writing.
How long are metrics and logs kept?
Whatever we decide based on usage and cost. The usual approach is a short period at full detail and a longer one with summarised data. Logs containing personal data also have a maximum period that we set for data protection reasons.
Can you add monitoring to what we already have?
Yes. If you already have a half-finished Prometheus or Grafana, we start from there: we review what is collected, remove the alerts nobody responds to and add what is missing.
Tell us about your case
In a free 20-minute call we look at where you are and where to start. No commitment.
Official documentation consulted
The technical details on this page are based on:
Reviewed on