Category: Virtualisation
Managed Proxmox VE: virtualisation clusters with high availability
Your virtual machines and containers in a cluster that fails over on its own if a node goes down, with the right storage choice and upgrades one node at a time.
- Where it runs
- In your account, in your name
- Backups
- With tested restores
- Updates
- Planned and documented
- Licence
- Open source, AGPLv3 (optional subscription)
In short
Proxmox VE is an open source virtualisation platform that brings together KVM virtual machines, LXC containers, storage and high availability in a single interface. We build and operate clusters on your servers or on a provider's dedicated servers: networking, ZFS or Ceph, high availability, backups, monitoring and upgrades. We also migrate machines from VMware.
What do we take care of?
Proxmox VE installs from an image in no time and can run for a long while without anyone touching it. The problems arrive with the first node lost without quorum, a ZFS pool at ninety per cent or a major upgrade done in a hurry. This is what we keep under control:
Nodes
Installation, networking, firmware and packages kept up to date on every server in the cluster.
Cluster and HA
Corosync with quorum, high availability groups and tested failover.
Storage
ZFS with replication or Ceph, space monitored and disks checked with SMART.
Backups
Proxmox Backup Server or vzdump to another site, with test restores.
Metrics
Cluster status, per-machine usage, ZFS or Ceph and useful alerts.
Access
Interface only over VPN, two-factor authentication, named users and group permissions.
What architecture do we recommend?
High availability in Proxmox VE depends on quorum: more than half of the cluster’s votes have to agree for one node to start another node’s machines. That is why we start from three nodes, or two plus an external quorum device. Corosync traffic runs on its own network, because if it gets saturated by backups or Ceph, the nodes think they are isolated and reboot.
| Situation | What we set up | Why |
|---|---|---|
| A single server | Mirrored ZFS and backups to an off-site Proxmox Backup Server | No high availability, but tested recovery |
| Two or three nodes with local disks | ZFS on each node with scheduled replication and HA | Simple; on failover you lose the minutes since the last replication |
| Three or more nodes with a 10 Gbit/s or faster network | Ceph with several disks per node | Shared storage, failover with no data loss |
| An existing storage array | Shared iSCSI or NFS | Makes use of what you have; the array becomes the single point of failure |
ZFS replication or Ceph redundancy is no substitute for a backup: if someone deletes a machine or ransomware encrypts its disks, the damage is replicated. Backups always go to another machine, and ideally to another site.
What problems do we come across most often?
- Two nodes without a quorum device: high availability does not kick in, or the surviving node gets stuck.
- An unmonitored ZFS pool: it fills up and the machines can no longer write.
- Corosync sharing a network with the backups: nodes rebooting in the middle of the night for no apparent reason.
- Backups in the same cluster: the backup storage is a disk on one of the nodes.
- Web interface open to the internet with the root user and no two-factor authentication.
- An unprepared major upgrade: the check script is skipped and a machine never boots again.
How do we carry out upgrades?
We apply minor upgrades one node at a time: we live-migrate the machines to another node, upgrade, reboot if needed and move them back. The cluster keeps serving throughout. For a major version, we read the notes, run the check tool that Proxmox publishes for each version jump, fix what it flags and take a recent backup of every machine before starting. If you use Ceph, its upgrade follows its own order and we do it before or after as the guide indicates.
Who is it for?
- A company running VMware that is looking for an alternative after the licensing changes and wants to keep its virtual machines.
- A team with its own servers in the office or in a data centre that needs high availability without paying per-processor licences.
- Anyone who wants virtual machines on dedicated servers, for cost or control, with someone answering for the cluster.
We usually send this cluster’s backups to a managed Proxmox Backup Server, and its metrics to monitoring with Grafana and Prometheus.
What does a migration from VMware look like?
We start with an inventory: machines, operating systems, disks, networks and the dependencies between them. We build the Proxmox VE cluster in parallel and use its import wizard, which connects to the ESXi host and brings each machine across with its configuration. On Windows machines we install the VirtIO drivers first; on all of them we check network cards, UEFI or BIOS boot and the guest agent. We migrate less critical machines first, then the important ones in agreed windows, and the VMware environment stays on until everything works. If the migration is part of a bigger change of provider, we plan it within cloud migrations and costs.
- Remote accessVPN and two-factor authentication
- Proxmox VEThree-node cluster
- High availabilityQuorum with Corosync
- StorageZFS with replication or Ceph
- Proxmox Backup ServerBackups outside the cluster
Where do we set it up?
In your account, with the provider you already use or the one that fits best. If the data has to stay in the EU, Hetzner and OVHcloud are the European options.
Providers we set it up on
- Hetzner
- OVH
- Own server
| Provider | How we set it up | Things to bear in mind |
|---|---|---|
| Hetzner | Dedicated servers running Proxmox VE, a private network between them with vSwitch, and additional IPs or a subnet for the virtual machines. | On a dedicated server's public network, not every MAC configuration is allowed; we set up routing or NAT depending on what you have ordered. |
| OVH | Dedicated servers running Proxmox VE, a private network between nodes and additional IPs for the virtual machines. | |
| Own server | Proxmox VE on your physical servers, in your office or in a data centre, managed over a VPN. | Real high availability needs a third vote: a third node or a quorum device on another machine. |
What is included
- Proxmox VE cluster with three nodes, or two plus a quorum device, on your hardware or on dedicated servers.
- Separate networks for management, cluster traffic, storage and virtual machines, with VLANs or bridges as appropriate.
- Storage with ZFS and replication between nodes, or Ceph when there are three or more nodes and a fast network.
- High availability configured for the critical machines, with failover tested by shutting down a node.
- Backups with Proxmox Backup Server or with vzdump to an external target, and a regular test restore.
- Metrics for the cluster, ZFS or Ceph and the disks, with alerts on quorum, space, SMART and failed backups.
- Minor and major upgrades one node at a time, live-migrating the machines first.
- Web interface not exposed to the internet, two-factor authentication, named users and migration from VMware ESXi.
Frequently asked questions
Can we migrate our machines from VMware ESXi?
Yes. Proxmox VE includes an import wizard that connects to an ESXi host as if it were a storage and brings the machines across with their configuration; Proxmox states that it has been tested from ESXi 6.5 to 8.0. For cases the wizard does not cover, we use OVF import. On Windows we check the VirtIO drivers and the disk type, and on all of them the network and boot settings.
Do we have to pay for the Proxmox subscription?
It is not mandatory. Proxmox VE is open source and can also be updated from the no-subscription repository. The subscription gives access to the enterprise repository and to vendor support. For production we usually recommend it; the decision is yours and we explain it to you before we start.
ZFS or Ceph?
ZFS with replication between nodes is simpler and enough for many small clusters, at the cost of losing the minutes of data since the last replication if a node goes down. Ceph spreads the disks across all the nodes and loses no data on failover, but it needs at least three nodes, several disks per node and a fast dedicated network.
Does it work with two servers?
Not with two nodes on their own: if one goes down, the other loses quorum and will not start anything by itself. The fix is an external quorum device, which can be a small machine or a virtual server in another location, or a third node.
How do you manage it if the servers are in our office?
Over a VPN that only opens management access. If the hardware allows it, we also configure the server's out-of-band console so we can get in even if the system will not boot.
Tell us about your case
In a free 20-minute call we look at where you are and where to start. No commitment.
Official documentation consulted
The technical details on this page are based on:
- Proxmox VE Administration Guide: Cluster Manager
- Proxmox VE Wiki: Migrate to Proxmox VE
- Proxmox VE Administration Guide: Deploy Hyper-Converged Ceph Cluster
- Hetzner Docs: Error FAQ (MAC errors)
Reviewed on