A homelab primer on designing, maintaining and scaling up a Proxmox cluster
Posted on: 2025-04-20
As part of my role in IT, I've had to design many virtualization environments. I also had to maintain and scale these environments. Spinning up a computer that you can use as a server to host various applications isn't hard. But if you want to do it properly, including making it easier on you when it comes to maintaining software updates, deploying more applications, VMs and containers, handling security, backups, and so on.. that's when things start becoming a lot more complex.
In this article I'm going to try and share some of my experience in the form of tips and tricks you can use so you'll be able to design an environment that will not only work for a homelab, but that can also be scaled up to handle a full production workload. I'll focus mostly on Proxmox because I believe it's the best option currently available these days, not only for homelab users but also for the enterprise.
Why use Proxmox?
Proxmox Virtual Environment (Proxmox VE) is a free and open source virtualization platform focused on hyper-converged infrastructure (HCI). This means it can virtualize not only Virtual Machines (VMs) and containers, but also Software Defined Networking (SDN) and storage. It provides a myriad of features including replication, live migration, high availability and an advanced backup server. It was first released in 2005 and has been in active development ever since, with enterprise support available from various companies.
If you start small and only wish to host a single or a small number of applications, it might be tempting to simply install an operating system like Linux or Windows Server directly on a single node. But even for small setups, I urge you to consider virtualization. By running your workloads on Promox, you can easily deploy, backup, restore, snapshot and destroy your entire workload with a single click. Thanks to templates and automation, everything from the moment of installation can be made much faster and more efficient. And should you need to scale up, you can build a cluster and use high availability features to handle a higher workload.
Proxmox runs on a modified Debian Linux kernel so it supports most hardware platforms. You can download it for free from the website and install it using a USB key. Once installed, you can access the web UI from any system on the network by using the IP address assigned to the machine.
Hardware selection
Most PC systems will run Proxmox, but there are some considerations to be taken if you want a scalable virtualization platform. First, you need to decide on the hardware to use. In the homelab community, using old enterprise hardware is a popular option, but this should be done mostly for fun. The reality is that this gear tends to be underpowered, very loud and uses a lot of power. Instead, an option that's become more popular lately are mini-PCs like the HP EliteDesk G800, Dell OptiPlex Micro and Lenovo ThinkCenter M710. These can be acquired for around $50 to $250 on eBay and will allow you to upgrade the RAM and run dozens of VMs or containers.
For virtualization, the main resource type to look at is RAM memory. The more, the better, because each of your workloads will require its own reserved amount any time it's running. Fortunately, RAM tends to be fairly inexpensive, so upgrading a system to 32GB, 64GB or more is not rare. CPU is obviously also important if you want to run many processes at the same time, and finally disk storage.
Even if you max out your single-node virtualization system, you can expand it by using clustering. Proxmox supports clusters out of the box, and you can expand from 1 node to 3, 4, 5 and many more. One important note is that a 2-node cluster is not supported. While there are hacks around that, it's best to avoid and go straight to 3 nodes. If you have 2 nodes and one of them fails, the remaining node will kill all of its VMs and containers and refuse to restart them, due to a lack of quorum.
Storage overview
The first item we'll take a look at when designing our cluster is storage. Proxmox supports a number of storage options, and I would argue that it's one of the most important decision you need to make before you start deploying the environment. It's going to be a lot easier if you design it properly beforehand, rather than try and retrofit a different option.
Here is a table showing the different storage options:
| Feature | LVM | ZFS | Ceph | SAN/NAS |
|---|---|---|---|---|
| Type of storage | Local storage | Local storage | Local storage | Network storage |
| Minimum number of nodes | 1 | 3 | 5 | 1 |
| Redundancy | None | With replication | Automatic | Depends on setup |
| Scalability | Low | Medium | High | High |
| High availability | No | Yes | Yes | Depends on setup |
| Complexity | Low | Medium | High | Depends (Medium/High) |
| Resource usage | Low | Medium | High | Low |
Logical Volume Manager (LVM) is the default and easiest type of storage you can use in Proxmox. By default, when you finish your installation, this is the type of storage that will be used to hold VM and container disks on local hardware. In a simple setup, you could just keep using this without worry. However, as soon as you want something more robust, I suggest you look at alternatives.
ZFS is a robust, well proven filesystem with error correction, supported out of the box by Proxmox. While it does require a setup process, and has a higher RAM requirement (typically 1 GB of memory for each 1 TB of storage), it's well worth it. Not only is it a more robust option that could prevent data loss, but it also enables the use of replication and high availability in a cluster. For example, if you have a 3-node cluster, you could have a ZFS disk in each node, then setup replication every 5 minutes with high availability. This way if one system suffers a hardware failure, the workloads running on that node will automatically be restarted on another node, with the replicated disks.
Ceph is a distributed object storage meant for high end environments. It turns your local storage throughout the cluster into a single distributed object storage, allowing your workloads to be instantly brought back up if one node fails. It's the best option if you want the best redundancy and scalability, however it's much more complex to setup, manage and requires at least 5 nodes with a high speed network connection between them, at 10Gbps minimum to be usable.
Finally, a Network Attached Storage (NAS) or Storage Area Network (SAN) simply refers to a dedicated storage system used to host the data for your workloads. With this, the cluster only acts as the compute nodes, and you defer storage to another platform. This also requires a high speed network, but the complexity, scalability and redundancy will depend on your specific setup. It also frees all the resources from the cluster nodes to be used solely for your workloads.
Virtual Machines and Containers
Proxmox supports VMs and LXC containers out of the box. For example, you can upload an ISO file to install a Linux or Windows virtual machine. You can also use a template to start a container. Proxmox comes with a large array of templates from the TurnKey project that makes it very easy to deploy popular software like Nginx, Apache HTTP, Dokuwiki, Drupal, Redis, PostgreSQL and much more.
Here are a few differences between using VMs and LXC containers:
- VMs provide full OS isolation, while containers use the node's OS, which may complicate things if you need specific hardware drivers.
- Containers have slightly better performance and a lower storage requirement than VMs since it uses shared resources.
- Containers start faster than VMs, since it doesn't need to boot an OS.
- Hardware passthrough may not be available for containers, depending on the hardware.
- Containers can only run Linux workloads, while you can install Windows and other OSes on VMs.
- VMs can be live migrated, but containers must be killed and restarted.
It's important to note that Proxmox does not support Docker containers by default. However, you can easily install Docker inside of a VM, or even an orchestrator like Kubernetes or Portainer.
Backups and snapshots
The last topic I'll cover are backups and snapshots. Even if you have RAID or replication, don't forget to have actual backups. Proxmox supports backups out of the box, and you can set full backups on a schedule for specific VMs and containers, or all of them at once. You can also configure how many past backups you want to keep. Note however that if you want support for incremental backups, which is really useful in a production environment, you should look into the Proxmox Backup Server (PBS). This is a separate package that you can download and install on a separate physical machine, with a lot of different backup options for your Proxmox environment
Snapshots are also a very useful feature. You can take a manual snapshot of any VM or container, and then revert to this snapshot as a point in time. This is very useful to do before you do software upgrades, install a new package, or anything that has a risk of messing up your workloads. I highly encourage you to get in the habit of taking snapshots on a regular basis.
Learning and using a virtualization platform like Proxmox can take some time and appear challenging, but it's incredibly useful to run anything from personal apps to large, scalable enterprise ones. This also teaches you a lot of important concepts in IT. Hopefully these tips give you some idea of how to properly get started down this path.