Category: Messaging
Managed RabbitMQ in your account: reliable queues, clustering and upgrades
Your message broker in a cluster, with queues that survive the loss of a node, alarms you can see coming and upgrades without any drama, in your account and as code.
- Where it runs
- In your account, in your name
- Backups
- With tested restores
- Updates
- Planned and documented
- Licence
- Open source, MPL 2.0
In short
RabbitMQ is an open source message broker that distributes work between applications through queues. Managing it means answering for no message being lost and no queue getting stuck. We set up a three-node cluster with quorum queues, monitor memory, disk and queue backlogs, and plan the upgrades, on virtual machines or on Kubernetes with its operator.
What do we take care of?
RabbitMQ tends to cause trouble at two moments: when a consumer stops working and the queues grow until they trigger the memory alarm, and when it is time to upgrade a version that has not been touched for years. We make sure the first gives you plenty of warning and the second is a rehearsed procedure.
Cluster
An odd number of nodes: quorum queues need a majority, and with three the cluster survives the loss of one.
Queues
Delivery limit and a dead letter queue, so that a message that always fails is not retried forever.
Alarms
By default the memory alarm fires at 40% of RAM and blocks every publisher: we tune it to the actual node.
Metrics
Delivered and unacknowledged messages per consumer: the first thing we look at when a queue grows.
Access
Never the guest user: one user and one vhost per application, with permissions only on its own queues.
Upgrades
All feature flags enabled before changing version; if one is missing, the new node will not start.
What architecture do we recommend?
For production, a three-node cluster, because quorum queues need a majority to keep working: with three, it survives the loss of one. Applications connect through a load balancer or a list of nodes, publish with publisher confirms and consume with manual acknowledgement, so that a message is not considered delivered until it has been processed. On Kubernetes we use the official RabbitMQ operator, which declares the cluster, users and queues as resources in the repository.
| Situation | What we set up | Why |
|---|---|---|
| You already use Kubernetes and run it well | RabbitMQ operator with three replicas and persistent volumes | The cluster and its topology in GitOps |
| No Kubernetes | Three virtual machines with RabbitMQ, deployed with Ansible or Terraform | Few moving parts and easy to reason about |
| You are on AWS and it is enough for you | Amazon MQ for RabbitMQ in cluster mode | Less maintenance; we monitor queues, alerts and versions |
For network partitions, we choose the strategy the cluster follows and test it by cutting the network between nodes in a test environment, so it does not come as a surprise in production.
What problems do we come across most often?
- Classic mirrored queues: set up years ago, they prevent the move to RabbitMQ 4.
- A single node: if it goes down, every application that depends on it stops.
- Queues that grow forever: failed or slow consumers, with no length limit or dead letter queue.
- Memory alarm triggered: publishers are blocked and the application seems to hang.
- Lost messages: publishing without confirms or consuming with automatic acknowledgement.
- An exposed management UI: with the guest user or a password shared between applications.
How do we protect messages and configuration?
In RabbitMQ, message protection comes from replicated quorum queues, not a nightly backup: messages are short-lived and yesterday’s backup is no use for recovering them. What we do keep is the cluster definition: users, vhosts, queues, exchanges and policies, in the repository and exported regularly. To check that it works, we bring up a test cluster from those definitions and shut down a node under load to see that nothing is lost.
How do we carry out a version upgrade?
We read the release notes, check the supported upgrade path and make sure all feature flags are enabled before jumping to a new series, because RabbitMQ requires it. We rehearse in testing and then upgrade one node at a time, waiting for the quorum queues to recover their replicas before moving on. When the jump is large or the queue type has to change, we build a new cluster in parallel and move the applications across gradually, in what RabbitMQ calls a blue-green deployment.
How do we migrate from another RabbitMQ or from Amazon MQ?
With the new cluster up and the definitions imported, we use Shovel or Federation to move the pending messages while we switch over the consumers first and then the publishers. We take the opportunity to move anything still on classic queues to quorum queues. If you are thinking of changing provider for cost reasons, we compare it against your bills in cloud migrations and costs.
What does day-to-day support look like?
The metrics go to Prometheus and Grafana, as we describe in monitoring with Grafana and Prometheus, with alerts on pending messages, failed consumers, alarms and nodes out of the cluster. Automatic monitoring is continuous; human response is the next business day, during business hours, as part of SRE as a service. Every month you receive a report with versions, incidents and changes made.
- PublishersWith publisher confirms
- Load balancerSingle entry point to the cluster
- RabbitMQThree nodes, quorum queues
- ConsumersWith manual acknowledgement
- PrometheusMetrics and alerts
Where do we set it up?
In your account, with the provider you already use or the one that fits best. If the data has to stay in the EU, Hetzner and OVHcloud are the European options.
Providers we set it up on
- Hetzner
- AWS
- Google Cloud
- Azure
- DigitalOcean
- OVH
- Own server
| Provider | How we set it up | Things to bear in mind |
|---|---|---|
| Hetzner | Three cloud servers on a private network with clustered RabbitMQ, or in your Kubernetes with the RabbitMQ operator. | Hetzner does not offer managed RabbitMQ: we provide the cluster and the upgrades. |
| AWS | Amazon MQ for RabbitMQ when it fits; otherwise, EC2 or EKS with the RabbitMQ operator across three zones. | With Amazon MQ we review the instance size, queue types, alerts and maintenance windows. |
| Google Cloud | Compute Engine or GKE with RabbitMQ and its operator. | Google Cloud has no managed RabbitMQ of its own; Pub/Sub is a different service with a different model. |
| Azure | Virtual machines or AKS with RabbitMQ and its operator. | Azure has no managed RabbitMQ of its own; Service Bus is an alternative, not a drop-in replacement. |
| DigitalOcean | Droplets or DOKS with clustered RabbitMQ. | No managed RabbitMQ: we set it up and maintain it ourselves. |
| OVH | Public Cloud instances or Managed Kubernetes with RabbitMQ. | |
| Own server | Clustered RabbitMQ on your servers or virtual machines, or in your Kubernetes. | The three nodes must be on separate physical machines with a reliable network between them. |
What is included
- Three-node cluster on separate machines or zones, as code with Terraform, Ansible or the Kubernetes operator.
- Quorum queues for data that cannot be lost, and streams when messages need to be re-read.
- Migration away from the old classic mirrored queues, which RabbitMQ 4.0 no longer supports.
- Memory and disk limits tuned, with alerts before the alarms fire and block the publishers.
- Prometheus plugin with Grafana dashboards: pending messages, consumers, acknowledgements and connections.
- A network partition handling strategy chosen and tested, so you know what the cluster does if the network is cut.
- Per-application users, vhosts and permissions, TLS and a management UI with no access from the internet.
- Upgrades with feature flags reviewed or through a blue-green deployment, rehearsed first.
Frequently asked questions
Quorum queues or classic queues?
For messages that cannot be lost, quorum queues: they are replicated across several nodes with a consensus protocol and survive the loss of one. Classic queues are still useful for transient data where speed matters more. Classic mirrored queues were removed in RabbitMQ 4.0, so if you still use them they have to be migrated before upgrading.
RabbitMQ or Kafka?
RabbitMQ fits when you distribute tasks between workers and each message is consumed and deleted. Kafka fits when many consumers read the same event stream or you need to keep it for days. RabbitMQ streams cover part of that gap; if you are unsure, we look at it with your real use case.
Amazon MQ or our own RabbitMQ?
If you are on AWS and your workload is normal, Amazon MQ takes broker maintenance off your hands. We run RabbitMQ ourselves when your provider does not offer it, when you need plugins or configuration the service does not allow, or when you want the broker inside your Kubernetes.
Why do the publishers get blocked?
Because of the memory or disk alarms: when a node goes over the limit, RabbitMQ stops accepting messages to protect itself. It is usually a symptom of slow or failed consumers and queues that keep growing. We prevent it with alerts on pending messages before the alarm is reached.
Tell us about your case
In a free 20-minute call we look at where you are and where to start. No commitment.
Official documentation consulted
The technical details on this page are based on:
Reviewed on