Dear selfhosters!
I come to you in the hope of help for avoiding some rookie mistakes.
I plan to migrate my very diverse hard- and software environment to a single machine.
Current mode of operation
I operate several RaspberryPis, a hardware firewall running on OpenWRT and a NUC like mini PC.
The RaspberryPis more or less are there for a single function; one runs Nextcloudpi, two run PiHoles, another one runs iSpy.
The mini PC is for the tasks that are heavier on CPU, RAM or storage space.
Maintaing this has become somwehat cumbersome and a replacement is dearly needed. My plan is to move all to a Proxmox sever.
I do have a general idea how to set up things, but as I’m brand new to Proxmox, I fear that there’s a lot of mistakes to be made. I haven’t read all documentation, but enough to know that it’s no easy task to set up and operate Proxmox properly.
I’m aware that not having server hardware (e.g. no ECC RAM) is not the best setup, but AFAIU at least having a data centre SSD and lots of RAM is a good start.
Hardware
In the future all services are meant to run on this machine:
Case/Mainboard: AsRock Deskmeet X300
CPU: AMD Ryzen 5 5600GT
RAM: 64 GB
Storage:
- 480 GB SSD (Intel DC S4500 Series)
- 4 TB SSD (Team Group MP44)
- 16 TB HDD (Seagate Exos, yeah, I know, but realized too late…)
OS: Proxmox 8.3.1
Future mode of operation
Here’s a high-level scheme of what I plan to do:
- Install Proxmox on the Intel SSD
- Use the 4 TB SSD as storage drive for the machines
- Use the 16 TB HDD as storage drive for backups and additional storage (for files that mainly get read like media) for the machines
- Migrate each physical device to a virtual machine (or create a new one to replicate the service)
- Repurpose the mini PC as Proxmox backup server
Help!
The areas where I think reading documents can’t beat experience are:
- Do I use BTRFS or ZFS? I tend to use ZFS because of its advantages when making backups. What would you do?
- Do I use QEMU/KVM virtual machines or LXC/LXD cointainers? Performance wise QEMU emulating the host architecture should be the way to go, right?
- I shy away from running all services as Docker on the same machine for backup/restore purposes and rather have VMs per service. Is there anything wrong with this approach?
- I’d love to keep NextcloudPi (because it’d make it easy to migrate settings and files) and there’s an LXD container for it. Would you recommend doing a switch to Nextcloud AIO instead?
- I’ve equipped the Deskmeet X300 with a WiFi card and antennas. AFAIU trying to use WLAN instead of LAN will create some trouble. Has anyone running Proxmox on a machine with WLAN insteal of LAN access successfully?
- I’m aware that Proxmox comes with a firewall, but I don’t feel very confortable using a software firewall running on the same machine that hosts the virtual machines. Is this just me being paranoid or would you recommend putting a hardware firewall between the internet access and the Proxmox server?
- What else should I think of, but haven’t talked about/asked yet?
Thank you very much for your time and your suggestions in advance!
Hopping in here to mention Proxmox Helper Scripts . They have many scripts that help you set up LXCs with software you may be using, including the full aar stack.
I tend to test things in a dedicated new VM, to get a feel for it, make sure I need to add it to my permanent services. If it does, I try to find a way to run it via LXC, and if that is too complicated/won’t work, I have a dedicated docker VM I throw it on. Everyone will answer the “LXC/VM/Docker” question differently, and they will all be correct. What is easiest for you is the right way.
I run a VM with opnsense as my network firewall. Moved it from a hardware install. I don’t see any issues, and there are loads of times it’s saved my ass having it backed up as a VM.
Slam as much ram as you can afford/fit inside the computer too. Every time I think I have enough, I always find I have need/use for more.
Hopping in here to mention Proxmox Helper Scripts . They have many scripts that help you set up LXCs with software you may be using, including the full aar stack.
I got made aware of these scripts by @Krik@lemmy.dbzer0.com already, but thank you for pointing me to this very helpful resource!
I tend to test things in a dedicated new VM, to get a feel for it, make sure I need to add it to my permanent services. If it does, I try to find a way to run it via LXC, and if that is too complicated/won’t work, I have a dedicated docker VM I throw it on. Everyone will answer the “LXC/VM/Docker” question differently, and they will all be correct. What is easiest for you is the right way.
I suppose I will go that road for new things I’m about to try out if it’s as easy as spinning up another VM or LXC.
Replicating services provided by the RaspberryPis and the mini PC I think I will try the LXC way and see how far I get.
This is leaning heavily on the experience of @Krik@lemmy.dbzer0.com regarding performance advantages of LXC over VM.I run a VM with opnsense as my network firewall. Moved it from a hardware install. I don’t see any issues, and there are loads of times it’s saved my ass having it backed up as a VM.
Not having to deal with a dedicated piece of hardware/configuration is for sure in favour of a virtual firewall.
Then again the configuration of the firewall is pretty static, unless I plan on adding services in the firewall zone that need to reach the rest of the local network. I need to mull over this some more.Slam as much ram as you can afford/fit inside the computer too. Every time I think I have enough, I always find I have need/use for more.
64 GB has pretty much reached the limit, if I don’t want to throw the 4 DIMMs away and purchase a new set. Let me find out how far that carries me.
Not answering every single point but generally: I’d set up proxmox, test everything, make notes, reach a state/config that you like, and then start over doing it “properly” from start.
Personally: ZFS yes, quemu/lxc depends on use case
I shy away from running all services as Docker on the same machine for backup/restore purposes and rather have VMs per service. Is there anything wrong with this approach?
No, but you’ll have much more overhead. I have a VM that hosts all Docker deployments which don’t need much disk space (most of them)
No, but you’ll have much more overhead. I have a VM that hosts all Docker deployments which don’t need much disk space (most of them)
This is a big point. One of the key advantages of docker is the layering and the fact that you can build up a pretty sizeable stack of isolated services based on the same set of core OS layers, which means significant disk space savings.
Sure, 200-700MB for a stack of core layers seems small but multiply that by a lot of containers and it adds up.
Thanks! I was planning on taking my time. I won’t decommission my current setup before I’m confident the new one is doing all right.
And I’m damn sure I will need several iterations of setting all up, testing, finding obstacles and starting over.
I just tried to get a head start instead of beginning with the simplest mistakes :)
Some concerns:
- get multiple smaller harddrives and put them into some kind of RAID / zpool with redundancy. the drives will fail.
- there is absolutely zero reason to have a VM per service when a container will do. There are no advantages. But VMs will take significantly more resources and be harder to right-size. There is no restore/backup advantage using VMs.
- for that reason there is also no reason to use proxmox in the first place, unless you want to learn proxmox. Truenas scale for example comes with pre-installed k3s.
- I would get a separate hardware firewall because it makes easier to expand the network later.
Thank you for advising me of your concerns!
- get multiple smaller harddrives and put them into some kind of RAID / zpool with redundancy. the drives will fail.
That’d require moving all to a different hardware platform. I hope to get the risks associated with failing drives mitigated by the Proxmox backup server
- there is absolutely zero reason to have a VM per service when a container will do. There are no advantages. But VMs will take significantly more resources and be harder to right-size. There is no restore/backup advantage using VMs.
That’s good advice! It seems I need to get comfortable with automatic backups of Docker containers and data volumes
- for that reason there is also no reason to use proxmox in the first place, unless you want to learn proxmox. Truenas scale for example comes with pre-installed k3s.
Getting familiar with Proxmox is indeed one of the reasons I switched to that route. My initial plan was to replace the mini PC with the X300 and move all to docker. Then one consideration lead to another. Maybe I need to re-evaluate whether going the Proxmox route is worth the trouble.
- I would get a separate hardware firewall because it makes easier to expand the network later.
I agree that’s another reason for having a hardware firewall besides the security aspects of having one.
The containers will store their data in volumes, and ideally those volumes are individual ZFS datasets. The containers themselves are stateless, and you can just boot them up with the volume to “restore” them.
However if you want to learn proxmox anyway this is a moot point anyway.
You write that you need help. From the questions it sound like you need experience. You may not like my answer.
BTRFS and ZFS are very similar, try them out figuring out.
Try LXC if it doesn’t work for you, use standard VM.
There is nothing wrong with have all in one machine, one VM per service or a hybrid.
There is nothing wrong with using a WLAN as long as you accept the consequences of that.
I think you should try it out and get some experience, what is the worse that can happen? You learn something and try something different.
Yeah, I’m aware that I’m going to deal with a broad subject and being new to Proxmox you’re spot-on that I lack experience.
So I tried to get a head start instead of beginning with the simplest mistakes.
Thank you for your recommendations!
Do I use BTRFS or ZFS? I tend to use ZFS because of its advantages when making backups. What would you do?
Usually VMs are usually I/O starved therefore I would try to go as lightweight as possible and chose Ext4 or XFS (depending on what the VM is used for). The VMs can be backed up whole by Proxmox. You have more than enough space to do that and it’s considerably easier to set up. And honestly how big could the containers and VMs be? I guess the containers are 50-200 MB and a VM a few GBs. That’s almost nothing.
Do I use QEMU/KVM virtual machines or LXC/LXD cointainers? Performance wise QEMU emulating the host architecture should be the way to go, right?
LXC containers are way more lightweight than VMs. I depends on what you want to do. Docker and a file server work better in a VM so far but Pi-hole and Jellyfin run perfectly in a container.
I shy away from running all services as Docker on the same machine for backup/restore purposes and rather have VMs per service. Is there anything wrong with this approach?
I would go for LXC first. If that isn’t possible or too cumbersome I would try docker (in a VM) next and one-VM-per-service last as they need the most resources.
I’d love to keep NextcloudPi (because it’d make it easy to migrate settings and files) and there’s an LXD container for it. Would you recommend doing a switch to Nextcloud AIO instead?
Sorry, no idea.
I’ve equipped the Deskmeet X300 with a WiFi card and antennas. AFAIU trying to use WLAN instead of LAN will create some trouble. Has anyone running Proxmox on a machine with WLAN insteal of LAN access successfully?
I would always try to connect it to LAN.
I’m aware that Proxmox comes with a firewall, but I don’t feel very confortable using a software firewall running on the same machine that hosts the virtual machines. Is this just me being paranoid or would you recommend putting a hardware firewall between the internet access and the Proxmox server?
No idea. I wouldn’t mind a firewall container. If something breaks through you are fucked one way or the other. The firewall in your router isn’t much different than any other.
You should always go for Wireguard or another VPN to access your network from the outside.What else should I think of, but haven’t talked about/asked yet?
Helper scripts for beginners: https://community-scripts.github.io/ProxmoxVE/
Just give them a look.And it seems you are ignoring Proxmox’ LXC. They are one of main reasons to pick that software.
Edit: As an additional note: I ran about 6 or 7 VMs on a mini PC (Intel N100) with 16 GB RAM. RAM was almost used up and the cpu was at ~15 %.
I then switched mostly to LXC and only one VM. The cpu was now at ~1% and RAM usage went down to 3 GB while still providing the same services as before.
The power of containers, baby! :DThanks a million for the extensive feedback, especially because it’s enriched by your own experience!
Usually VMs are usually I/O starved therefore I would try to go as lightweight as possible and chose Ext4 or XFS (depending on what the VM is used for). The VMs can be backed up whole by Proxmox. You have more than enough space to do that and it’s considerably easier to set up. And honestly how big could the containers and VMs be? I guess the containers are 50-200 MB and a VM a few GBs. That’s almost nothing.
I suppose your expectations about VM size are appropriate. The RaspberryPis have 8 GB SD cards and there’s quite some space left on them. I don’t know why the space requirement should be very different on a VM. Going from Raspbian/Armbian to Debian shouldn’t play that much of a role size wise.
Wouldn’t pick ETX4 oder ZFS make replicating data to the Proxmox backup server way less efficient?LXC containers are way more lightweight than VMs. I depends on what you want to do. Docker and a file server work better in a VM so far but Pi-hole and Jellyfin run perfectly in a container.
I would go for LXC first. If that isn’t possible or too cumbersome I would try docker (in a VM) next and one-VM-per-service last as they need the most resources.
I will try LXC before VM then!
I would always try to connect it to LAN.
That will make the physical placement harder, but I was afraid that’s the way to go: connect it to LAN…
No idea. I wouldn’t mind a firewall container. If something breaks through you are fucked one way or the other. The firewall in your router isn’t much different than any other.
You should always go for Wireguard or another VPN to access your network from the outside.Some ports need to be forwarded in order for e.g. Nextcloud to work. Right now they are forwarded to my firewall and all that’s reachable from outside is behind that firewall. The main purpose of the firewall is to protect the rest of the network from a compromised device within the firewall zone. So if something breaks through a bug in Nextcloud now, it will hopefully have a hard time breaking through the firewall.
Having a bug in Nextcloud running in an LXC or VM may allow additional attack vectors, if there’s no hardware firewall (and only the built-in firewall functions or a firewall container) between them and the rest of the network.
Connection from outside to my home network is via Wireguard tunnel.Helper scripts for beginners: https://community-scripts.github.io/ProxmoxVE/
Just give them a look.I was reading up on Proxmox setup both by consulting official documentation and forum entries, but I haven’t stumbled upon that link so far.
It looks awesome!
And I’m damn sure it will save me plenty of time :)
I found tteck’s helper-scripts (https://tteck.github.io/Proxmox/), but the collection linked by you looks more tidy.And it seems you are ignoring Proxmox’ LXC. They are one of main reasons to pick that software.
I fear that’s because I hadn’t understood the benefits of LXC over VM, which you made clear very plainly:
As an additional note: I ran about 6 or 7 VMs on a mini PC (Intel N100) with 16 GB RAM. RAM was almost used up and the cpu was at ~15 %.
I then switched mostly to LXC and only one VM. The cpu was now at ~1% and RAM usage went down to 3 GB while still providing the same services as before.
The power of containers, baby! :DIt’s about time to get Proxmox set up and dirty my hands!
