Fixing the "Too many open files" Problem on My lumine VM, and Notes on Moving It to an LXC

1. Overview of the Problem#

On the Proxmox VM lumine (ID 100), the Too many open files in system error kept popping up during filesystem operations such as ls /mnt/data. As a result, the services running in Docker inside that VM, such as Jellyfin, Stash and Copyparty, stopped working properly.

The problem showed up right after I changed how the storage (an 8 TB HDD) was shared: from passing it straight through to the VM, to sharing it via the Proxmox host with multiple guests (the lumine VM and the nahida LXC).

2. Investigation Log (How My Thinking and My Commands Evolved)#

1. Initial hypothesis: Are the applications in the VM (Jellyfin, Netdata) leaking file descriptors?

   • Investigation: Ran lsof inside the lumine VM.

     1
     sudo lsof | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 10

   • Result: jellyfin and netdata turned out to have over 10,000 files open. They rose to the top of the suspect list.

2. Revising the hypothesis: The per-process limit (ulimit -n) is a very high 1,048,576, so 10,000 or so wouldn’t hit it. The system-wide count (cat /proc/sys/fs/file-nr) was also a low value of around 2400, which contradicted the lsof results.

   • Reasoning: The error message appears inside the VM, but the real cause of the resource exhaustion might be outside the VM. I decided that VirtIO-FS, the mechanism used to share storage with the VM, looked suspicious.

3. Identifying the real culprit: On the Proxmox host (pve1), I investigated the virtiofsd process running for the lumine VM.

   • Investigation: On pve1, I found the PID of the virtiofsd process and checked how many files it had open.

     1
     ps aux | grep virtiofsd
     2
     # PID 454363 identified as the suspect
     3
     ls -l /proc/454363/fd | wc -l

   • Smoking gun: The virtiofsd process was consuming about 1 million file descriptors, and had hit its process limit.

3. Conclusion on the Root Cause#

When an application that scans the filesystem broadly, like Jellyfin or Netdata, runs inside the VM, its huge volume of access requests concentrates on the VirtIO-FS relay daemon (virtiofsd). This virtiofsd couldn’t keep up with the requests and exhausted its resources (file descriptors), and as a result it returned Too many open files errors to the VM side.

I concluded that this is a fundamental incompatibility between broad-scanning applications and the VirtIO-FS technology itself.

1. Basic Policy#

To stop using VirtIO-FS entirely, I move the role of lumine from a VM to an LXC container. With LXC, the host’s filesystem can be shared via a bind mount, a fast and stable kernel-level mechanism, so a bottleneck like virtiofsd doesn’t occur.

To keep the current docker-compose.yml workflow, I chose what I’d call the “realistic, practical best move”: build a Docker environment inside a single LXC and migrate the existing setup over wholesale.

2. Migration Procedure Summary#

1
# Bind-mount the HDD for media
2
mp0: /mnt/storage-8tb,mp=/mnt/data
3

4
# Intel QSV (iGPU) passthrough
5
lxc.cgroup2.devices.allow: c 226:* rwm
6
lxc.mount.entry: /dev/dri dev/dri none bind,optional,create=dir

1
# Run on the lumine-new LXC side
2
mkdir -p /home/myuser/docker/main
3
rsync -avz --progress myuser@192.168.x.250:~/docker/main/ /home/myuser/docker/main/

Problem 1: AdGuard Home Port Conflict#

• Symptom: When running docker-compose up -d, adguard failed to start with an address already in use error.
• Investigation: sudo lsof -i :53 showed that the host OS’s systemd-resolved was using port 53.
• Fix: Stopped and disabled systemd-resolved.

  1
  sudo systemctl stop systemd-resolved
  2
  sudo systemctl disable systemd-resolved

• Additional problem: During AdGuard Home’s initial setup, I had set the admin port to 80, which conflicted with NPM.
• Fix: Edited AdGuardHome.yaml directly and changed http.address to :3000.

Problem 2: The Tailscale Daemon Failed to Start#

• Symptom: sudo tailscale up failed. The tailscaled service crashed with failed (Result: exit-code).
• Investigation: Checked the detailed logs with sudo journalctl -u tailscaled.service. I found this decisive error:

  1
  CreateTUN("tailscale0") failed; /dev/net/tun does not exist

• Cause: The tun kernel module wasn’t loaded on the Proxmox host, so /dev/net/tun couldn’t be created inside the LXC.
• Fix: A two-step fix.
  1. On the pve1 host, load the tun module and make it persistent.

     1
     modprobe tun
     2
     echo tun > /etc/modules-load.d/tun.conf

  2. Also on the pve1 host, append the device permission and mount settings to the LXC’s config file (/etc/pve/lxc/103.conf).

     1
     lxc.cgroup2.devices.allow: c 10:200 rwm
     2
     lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

Problem 3: Access via *.lumine Failed#

• Symptom: Access via IP:PORT worked, but access by hostname such as memos.lumine didn’t.
• Investigation:
  1. ping lumine on the client PC returned the old Tailscale IP, showing that name resolution was off.
  2. The AdGuard Home query log showed no record at all of queries from the client PC.
  3. tcpdump also showed that packets from the client PC were not reaching the LXC.
• Cause: “Override DNS servers” in Tailscale’s DNS settings was enabled, forcibly overwriting the client PC’s DNS settings. Because of that, the AdGuard Home on the LAN wasn’t being used, and name resolution failed.
• Fix: In the Tailscale Admin Console > DNS settings, I did the following.
  1. Turn “Override DNS servers” off.
  2. Set AdGuard Home’s LAN IP (192.168.x.201) as a “Global nameserver”.

1. Final System Configuration#

The lumine services now run stably under Docker Compose inside a lightweight LXC container on Proxmox. Storage is shared quickly via a bind mount from the host, and the file descriptor problem is completely gone. On the network side, AdGuard Home and Tailscale work together, so every service can be reached by a consistent hostname from both inside and outside the LAN.

2. Lessons Learned#

1. The limits of VirtIO-FS: VirtIO-FS is convenient, but it is a poor fit for workloads like Jellyfin that scan the entire filesystem at high frequency. It can become a resource-exhaustion bottleneck.
2. The advantages of LXC: For uses like a Docker host, LXC has less overhead than a VM and is better for performance (especially I/O). Bind mounts are very powerful.
3. The importance of systematic troubleshooting: By working through the layers in order, lsof → journalctl → tcpdump → query log, you can eventually pin down the cause even of a complicated problem. Don’t fix things on guesswork; act on facts.
4. Always suspect DNS: With network problems, especially “I can’t reach it by name”, the shortcut to a solution is isolating which layer of DNS is at fault (client cache, local DNS, Tailscale MagicDNS).

3. Remaining Tasks#

• [Done] Back up and delete the old lumine-vm.
• [Done] Remove the systemd settings that are no longer needed.
• [Done] Change the new lumine LXC’s internal hostname to lumine.
• [TODO] Set up a periodic backup job in Proxmox for the new lumine LXC.

That completes all the work on this matter.

↑Made with nano-banana. So cute.