@Falzo do you use ZFS on your Proxmox installs?
I’ve always stayed as far away from ZFS on Proxmox. One thing you might want to do is remove deduplication unless you really need it.
no, not really. I use ZFS on one node, but only for a secondary set of harddrives using an ssd as caching device. I think that was more of an experiment 
anyway, so far no troubles with that.
I usually use just plain sw raid 1 or 10 with sparse disk images, not even thin lvm…
1 Like
Same here, last I remember ZFS was using 16 GB of RAM at boot just for being ZFS.
1 Like
Not yet, I need to migrate 2 VMs before breaking the server.
This is for one
I’d suggest getting away from ZFS, as a trial.
If you need some temporary hosting space, let me know.
3 Likes
Moved 2/5 
LOL, you will get there. I’ve moved VMs many-a-time.
1 Like
Correct ZFS by default uses 50% of your system RAM.
Rule of thumb is you need 1GB RAM for every 1TB storage.
You can set limits on how much ZFS is allowed to use by visiting:
https://pve.proxmox.com/wiki/ZFS_on_Linux
Find on the page the section defined as: Limit ZFS Memory Usage
3 Likes
As for swap space, swapfiles running on ZFS doesn’t play too nicely. What you should do instead is when you install Proxmox is use the advanced setup, on that tab at the bottom (forgot what its called) but the bottom box allows you to set some free space at the end of your drive. Add however much space you want reserved for swap then continue installing the system as normal. After the machine is online, just create swap on that unused freespace like you would usually do (gdisk, mkswap etc). This will give you the best performance and stability.
You can read more on what I said here and search for SWAP on ZFS:
https://pve.proxmox.com/pve-docs/local-zfs-plain.html
4 Likes
Tune the ZFS ARC. I use 4-6GB on 32GB hosts.
zfsonlinux uses 50% of host RAM by default.
Next, MOVE/disable all swap on zfs zvols. (the default installer option I think)
ANY system(hypervisor) swap ‘should’ be on non ZFS volume for safety reasons. (explained later)
The newer proxmox installers provide a feature to ‘leave’ some unallocated space after the ZFS partition marker. Make a linux SWAP partion in that space and use that instead for the hypervisor linux system.
Having swap on ZFS during a critical OOM situation can cause catastrophic data loss.
I think this was the issue that alerted me to it https://github.com/openzfs/zfs/issues/7734
It’s been quite a while, there may have been movement on the bug.
But I doubt it due to the nature of ZFS design. Almost every operation in ZFS involves some memory allocation ops. So, an OOM is very bad thing for swap-out.
None of this applies to Guests. SWAP or no-SWAP is up to you there.
Better to just give enough MIN and big MAX ram and enable Balloon driver like Falzo mentioned.
Edit: Also https://github.com/openzfs/zfs/issues/342
2 Likes
Is it safe to put options zfs zfs_arc_max=8589934592 and run update-initramfs -u while the RAM usage is at 80%?
Depends what your RAM is for one, 80% of 1TB is a lot of ram left 
So remember thats 8GB and you only need that if you got 8TB of storage? do you?
Also keep in mind with the change you need to reboot as well.
2 Likes
I forgot to change the number from the docs. 1073741824 it is.
Currently using 23 of 32GB. That looks like a great amount of free RAM.
If you got around 1TB of storage then 1GB RAM is what you ideally need set as max which is what that number reflects. And tbh you prob find you are really using more then that right now for ZFS as it would be using 16GB (max) on that system.
2 Likes
The way I understand it is, 1GB would be the bare minimum for ARC_size per terabyte of addressable storage.
Additional ARC size functions as a hypervisor wide disk page cache. I get ssd-cached-like performance in my lxc containers (backed by zfs datasets by default in proxmox . KVMs are backed by ZVOLS; slightly different internally)
I say 4GB ARC_size and a minimal 256MB swap inside guests.
You want guests to take advantage of ZFS storage magic.
3 Likes
Found a citation in the new Proxmox 6 Admin guide as well.
Section 3.8.7
https://pve.proxmox.com/pve-docs/pve-admin-guide.html#chapter_zfs
3.8.7. SWAP on ZFS
Swap-space created on a zvol may generate some troubles, like blocking the server or generating a high IO load, often seen when starting a Backup to an external Storage.
We strongly recommend to use enough memory, so that you normally do not run into low memory situations. Should you need or want to add swap, it is preferred to create a partition on a physical disk and use it as swapdevice. You can leave some space free for this purpose in the advanced options of the installer.
… and thats reason why I said adjust the partition when your installing (a few posts up) and leave some space at the end of the drive for “real swap” if you need it rather then doing on ZFS.
1 Like
Hi!
I moved the VMs I could move to another server and there is a lot of free RAM. I still have disk IO problems here. Some examples:
- YABS run on host or guests, takes down the whole node.
- Inside a cPanel VM: I need to restore an account backup and the VM becomes unresponsive.
Anyone would like to take a look? PM an estimated price please, I don’t know where to look anymore. You must be a well-known member from HostBalls.