Skip to content
Menu

Category: Messaging

Managed Apache Kafka in your own account: cluster, retention and upgrades without scares

Your Kafka cluster with replicas that really protect the data, retention that does not fill the disk and node-by-node upgrades, in your account and described as code.

Where it runs
In your account, in your name
Backups
With tested restores
Updates
Planned and documented
Licence
Open source, Apache 2.0 (Confluent components have other licences)

In short

Apache Kafka is an open source platform for moving and storing event streams between applications. Managing it means being answerable for no messages being lost, consumers not falling behind and the disk not filling up. We set up and run the cluster in your account, on virtual machines or on Kubernetes with Strimzi, with everything described in the repository.

Book a 20-minute callinfo@podsre.com

What do we look after?

Kafka usually works quietly until a broker runs out of disk, a consumer group builds up hours of lag or a postponed upgrade leaves the cluster out of support. Our job is to make sure those warnings arrive before the problem does and that every change is in the repository.

  • Brokers and KRaft

    Controller quorum, brokers spread across different zones or machines and configuration as code.

  • Replicas

    Replication factor 3 and min.insync.replicas 2 on topics that cannot lose data.

  • Retention and disk

    Time-based and size-based retention in line with the disk, with headroom so that a failed broker does not overwhelm the rest.

  • Lag and metrics

    Lag of each consumer group, under-replicated partitions and request latency.

  • Access

    TLS between clients and brokers, SASL or mTLS authentication and ACLs per application.

  • Upgrades

    Kafka and Java versions kept current, broker by broker and without losing availability.

Which architecture do we recommend?

For production, at least three brokers on different machines or zones and a quorum of three KRaft controllers, which in small clusters can share machines with the brokers and in large ones run separately. Important topics have replication factor 3 and min.insync.replicas 2, and producers write with acks=all: that way a broker can fail without losing an acknowledged message and without writes stopping.

Situation What we set up Why
You already use Kubernetes and run it well Strimzi with the cluster, topics and users declared in YAML Everything in GitOps and upgrades driven by the operator
No Kubernetes, steady load Virtual machines with Kafka in KRaft, deployed with Ansible or Terraform Fewer layers and fast local disks
Your cloud has managed Kafka and it is enough for you Amazon MSK, the Google Cloud service, DigitalOcean or OVHcloud Less maintenance; we monitor topics, lag, retention and cost

We calculate partitions according to how many consumers you need in parallel and how much the volume will grow. Increasing them later is possible, but it changes which partition each key goes to, and reducing them is not possible: that is why they are thought through beforehand.

Which problems do we come across most often?

  • Replication factor 1 or min.insync.replicas 1: the cluster looks redundant, but an outage loses acknowledged data.
  • Full disk: time-based retention with no size limit, or a topic that grows more than expected.
  • Lag nobody looks at: a consumer has been hours behind and it is noticed when the data has already expired through retention.
  • Too many partitions: thousands of small partitions that lengthen start-ups and leader elections.
  • ZooKeeper still in use: the cluster cannot move to Kafka 4 without first migrating to KRaft.
  • Topics created by hand: automatic creation enabled and nobody knows what configuration each one has.

How do we protect the data and test recovery?

Kafka has no classic backup: the protection is the replicas inside the cluster and, if the data justifies it, a copy in another cluster or region with MirrorMaker 2, or the topics dumped to object storage. What we do test is recovery: we shut down a broker and check that the leaders move, that nobody loses messages and how long it takes to catch up when it comes back. The definitions of topics, ACLs and quotas live in the repository, so a new cluster can be rebuilt from code.

How do we carry out a version upgrade?

We read the release notes, check client and Java compatibility, and rehearse in a test environment. Then we upgrade broker by broker: we wait until there are no under-replicated partitions before moving on to the next, and we raise the metadata version only when the whole cluster is on the new release and has been running for a while. With Strimzi, the operator follows that same order.

How do we migrate from another cluster or from Confluent Cloud?

With MirrorMaker 2 we replicate the topics, their configuration and the consumer group offsets to the new cluster in your account. We switch the consumers first, which carry on where they left off, and then the producers, topic by topic and with a rollback. Anything that is specific to Confluent rather than Kafka, such as Schema Registry or some connectors, we replace with open alternatives before the cutover. If the reason is cost, we compare it with your bills in cloud migrations and costs.

What does day-to-day support look like?

Kafka’s metrics go to Prometheus and Grafana, as we explain in monitoring with Grafana and Prometheus, with alerts for lag, under-replicated partitions, active controller and disk. Automated monitoring is continuous; human response is next business day, during business hours, as part of SRE as a service. Every month you receive a report with versions, incidents, disk growth and changes made.

How the pieces fit together
  1. Producersacks=all and idempotence
  2. Kafka brokersThree or more, in different zones
  3. KRaft controllersQuorum of three for metadata
  4. ConsumersGroups with monitored lag
  5. PrometheusMetrics and alerts

Where do we set it up?

In your account, with the provider you already use or the one that fits best. If the data has to stay in the EU, Hetzner and OVHcloud are the European options.

Providers we set it up on

  • Hetzner
  • AWS
  • Google Cloud
  • Azure
  • DigitalOcean
  • OVH
  • Own server
ProviderHow we set it upThings to bear in mind
HetznerCloud or dedicated servers with Kafka in KRaft set up by us, local disks or volumes and a private network between brokers.Hetzner does not offer managed Kafka: the cluster, the replicas and the upgrades are on us.
AWSAmazon MSK when it fits (provisioned or serverless); otherwise, EC2 or EKS with Strimzi and brokers spread across three zones.With MSK we still review topics, replicas, retention, lag and the cost of cross-zone traffic.
Google CloudGoogle Cloud Managed Service for Apache Kafka, or GKE with Strimzi.We choose between managed and Strimzi depending on version, size and which parameters you need to change.
AzureAKS with Strimzi or virtual machines with Kafka. Event Hubs accepts the Kafka protocol, but it is not Kafka.Event Hubs works for many simple producers and consumers; with Kafka Streams, Connect or transactions it is worth testing first.
DigitalOceanDigitalOcean managed Kafka, or Droplets and DOKS with Strimzi.A simple option for small clusters; we review configuration limits before choosing it.
OVHManaged Kafka from OVHcloud Public Cloud Databases, or Public Cloud instances with your own Kafka.
Own serverKafka in KRaft on your servers or virtual machines, or on your Kubernetes with Strimzi.Three brokers on the same physical machine are not high availability: you need at least three machines.

What is included

  • Kafka cluster in KRaft mode, without ZooKeeper, with controllers separate from the brokers when the size calls for it.
  • Replication factor, min.insync.replicas and acks reviewed per topic so that an outage does not lose acknowledged messages.
  • Partitions sized to the real parallelism of the consumers, not out of habit.
  • Time-based and size-based retention matched to the disk, with alerts before the limit is reached.
  • Broker and consumer metrics: lag, under-replicated partitions, requests and disk, with alerts and runbooks.
  • Topics, ACLs and quotas declared as code (Strimzi or Terraform), not created by hand.
  • Version upgrades broker by broker, rehearsed first in a test environment.
  • Migration from another cluster or from Confluent Cloud with MirrorMaker 2.

Frequently asked questions

Is Amazon MSK or Confluent Cloud better than our own Kafka?

It depends on your team and the volume. A managed service takes broker maintenance off your hands, but not topic design, retention, lag or the cost of traffic. We set up your own Kafka when the provider does not offer it, when the managed service restricts parameters you need or when its cost grows faster than the load.

Kafka on Kubernetes or on virtual machines?

If you already have a well-run Kubernetes, Strimzi makes it very convenient to declare the cluster, the topics and the users as code. If you do not have Kubernetes, a few virtual machines with Ansible or Terraform are simpler to reason about. In both cases Kafka needs fast persistent disks and brokers spread across different machines.

What happens to our cluster if it still uses ZooKeeper?

Since Kafka 4.0, ZooKeeper is no longer supported. To upgrade, the metadata first has to be migrated to KRaft on a 3.x version that allows it, and then the version can be raised. We plan it with a rehearsal beforehand and a rollback ready.

Tell us about your case

In a free 20-minute call we look at where you are and where to start. No commitment.

Book a 20-minute call

Official documentation consulted

The technical details on this page are based on:

Reviewed on

WhatsApp