This is the main physical layout page for the lab. It covers the internet connections, routing and switching path, compute hardware, storage hardware, rack layout, power layout, UPS coverage, and generator hookup. The other platform pages should reference this page instead of repeating the same physical topology.
At a high level, the lab is built around a compact Proxmox and Docker Swarm environment backed by a larger Unraid storage server. The current design favors several efficient systems over older high-power enterprise servers. The MS-01 nodes provide the primary clustered compute and Ceph storage tier, the Lenovo Tiny/Neo systems add lighter worker capacity, and the NAS handles bulk storage, media, backups, and long-term retained data.
Internet service enters through two WAN paths: ATT Fiber as the primary connection and Starlink as the secondary connection. Both feed a single UDM Pro, which is currently the main routing point for the lab. From there, the network path runs to a UniFi Pro Max 16, then across a 10 Gb link to the outbuilding UniFi Pro Max 16 PoE. The high-speed aggregation/core side also includes a MikroTik CRS317 for SFP/SFP+ connectivity and server uplinks.
That means the main physical path is:
ATT Fiber + Starlink
-> UDM Pro
-> UniFi Pro Max 16
-> 10 Gb outbuilding uplink
-> UniFi Pro Max 16 PoE
-> rack, servers, access devices, and PoE devices
The network is not fully redundant yet because the UDM Pro is still a single routing point. The design does have WAN diversity through ATT Fiber and Starlink, and the switching/uplink side is being kept clean enough that future router or switch redundancy can be added without rewiring the whole lab.
| Host group | Count | Role | Notes |
|---|---|---|---|
| MS-01 series | 3 | Primary Proxmox, Ceph, and Swarm manager VM hosts | Main performance tier. These hosts also form the private Thunderbolt/USB4 ring used for storage traffic. |
| M900 series | 4 inventory entries, 2 active in the current inventory | Proxmox worker host tier | Lightweight utility compute and Swarm worker VM placement. |
| Lenovo Neo series | 2 | Proxmox worker host tier | Newer utility compute and Swarm worker VM placement. |
| Unraid NAS | 1 | Bulk storage and backup landing zone | Exports media, games, backup, cloud, and ISO shares to Swarm nodes. |
The MS-01 tier is the center of the current lab. Those nodes are small, power-efficient, and still strong enough to run the core platform. They host the main Proxmox layer, contribute NVMe storage to Ceph, and back the Swarm manager VMs. They also carry Intel GPU capability that can be used for media workloads where the Docker node labels and placement rules allow it.
The Lenovo M900 and Neo systems are treated as practical worker capacity. They are useful for lighter workloads, extra Swarm placement options, and tasks that do not need the primary MS-01 storage/compute tier. The NAS is treated differently: it is capacity-oriented rather than compute-oriented, and it should be protected carefully because it is the home for bulk media, backups, game data, and retained files.
| Inventory group | Hosts |
|---|---|
proxmox_managers |
ms-01, ms-02, ms-03 |
proxmox_tb_ring |
ms-01, ms-02, ms-03 |
proxmox_workers |
m900-1, m900-2, m900-3, m900-4, neo-1, neo-2 |
unraid |
Tower |
proxmox_intel_gpu |
M900, MS-01, and Neo nodes with Intel GPU capability |
~/HomeLab/ansible/hosts.ini is the source of truth for what Ansible will touch. Some hosts can remain in the file but commented out when they are not active. This page should describe the intended physical design, while the inventory controls automation behavior.
| Layer | Hardware/source | Used for |
|---|---|---|
| MS-01 NVMe storage | 2 Samsung NVMe drives per MS-01 node | Ceph-backed active storage. |
| Ceph RBD | MS-01 Ceph cluster | VM disks and block-style workloads. |
| CephFS | MS-01 Ceph cluster | Shared app configuration and service data mounted at /mnt/cephfs. |
| Unraid array | Tower at 10.0.30.116 |
Bulk media, game data, backups, cloud data, and ISOs. |
Ceph is the active clustered storage layer for the lab. Proxmox and Swarm-related workloads can use RBD or CephFS depending on the need. CephFS is mounted on Swarm nodes at /mnt/cephfs, and many app configs follow the pattern /mnt/cephfs/docker/<stack>/<service>.
Unraid is the larger capacity layer. It exports NFS shares for media, games, backups, cloud data, and ISOs. The Swarm nodes mount those shares through host-level automounts under /srv/shared_volumes, then expose them to Docker as canonical bind-backed volumes such as unraid-data, unraid-games, unraid-backup, unraid-cloud, and unraid-isos.
The three MS-01 systems use a routed Thunderbolt/USB4 ring for private storage traffic. This keeps Ceph traffic off the normal client/public switching path and gives the storage tier a high-speed private route between the nodes.
| Host | Loopback | Ring interfaces |
|---|---|---|
ms-01 |
172.31.255.1/32 |
thunderbolt0 to ms-03, thunderbolt1 to ms-02 |
ms-02 |
172.31.255.2/32 |
thunderbolt0 to ms-01, thunderbolt1 to ms-03 |
ms-03 |
172.31.255.3/32 |
thunderbolt0 to ms-01, thunderbolt1 to ms-02 |
FRR and OSPF are configured by HomeLab/ansible/playbooks/proxmox/configure-thunderbolt-ring.yml. The role uses fast OSPF timers, BFD, jumbo MTU, and loopback addresses so the private storage network can recover cleanly if one path has a problem.
The rack is organized around serviceability first: switching and patching near the top, compute in the middle, and the heavy NAS/UPS equipment lower in the rack. The pull-out KVM and patching area should remain easy to reach because they are what make local troubleshooting practical when remote access is unavailable.
Any physical move should be reflected in this page at the same time as cable labels are updated. Rack docs are only useful if they match what someone sees standing in front of the rack.
Important rack rules:
The lab is fed by dual 20A circuits. The current rack power design uses dual UPS units and one PDU. Critical equipment is placed on UPS-backed power so short outages or power dips do not immediately drop routing, switching, storage, and core compute.
The NAS is treated as a high-priority load because it holds bulk storage and backup targets. Network gear is also high priority because routing, switching, DNS, and access to management interfaces need to stay up long enough to understand what happened during an outage. Noncritical workloads can come back after power returns.
Current power model:
| Item | Current role |
|---|---|
| Dual 20A circuits | Main electrical feed capacity for the rack/lab side. |
| Dual UPS units | Battery-backed power for critical equipment. |
| One PDU | Current rack distribution point. |
| NAS UPS feed | Protects bulk storage and backup target. |
| Network UPS feed | Keeps routing and switching alive through short events. |
The next planned upgrade is an ATS fed by both UPS units. That would let selected loads survive a UPS-side maintenance event or failure more cleanly. The goal is not to run everything forever on battery; the goal is to keep the important pieces stable long enough for either a short outage to pass or for a controlled shutdown/recovery decision.
The generator hookup belongs with the physical power design because it is part of outage handling, not application runtime. During an extended outage, the generator path is the next layer after the UPS units. UPS units cover the immediate transition and short interruptions; the generator is for longer continuity when utility power is out long enough that battery runtime is no longer the plan.
Operationally, the important part is to know what must stay online before the generator is connected and what can stay off. Routing/switching, NAS safety, and core management access are the first concerns. Media, lab experiments, and nonessential compute can stay down until power is stable.
Near-term hardware and power improvements are focused on resilience and serviceability:
| Hardware concern | Repo/path |
|---|---|
| Proxmox host tuning | ~/HomeLab/ansible/playbooks/proxmox |
| Thunderbolt ring | ~/HomeLab/ansible/roles/frr_ospf |
| Swarm VM creation | ~/HomeLab/ansible/playbooks/vm/create-swarm-host-vms.yml |
| CephFS client mounts | ~/HomeLab/ansible/roles/cephfs_mount |
| Unraid-backed Docker volumes | ~/HomeLab/ansible/roles/docker_shared_volumes |
| Keepalived for Swarm manager VIP | ~/HomeLab/ansible/roles/keepalived |