Hi, I need help to understand what I am doing wrong with my setup.
I am running a proxmox node (pve) where I have mounted my nfs storage (containing backups from my old server) on the host and assigning them to containers using pct set command.
On Host:I am setting the user permissions to the mounted folder so that the user from lxc can rw to it and the sub-folders(or atleast I thought it would be possible).
On Unpreviledged LXC:The mount is recognized and is accessible with the correct user permissions to rw
The docker container created inside the lxc is unable to read/write to this storage even though they are assigned to the correct user id 1000. The docker setup is logging errors and won’t start up.
Appreciate the help!
I’ll be honest man, just don’t do it. I tried, I really did, to make this exact scenario to work. You can get it to work - but it will be extremely brittle. You’re essentially hacking around LXC to do things it wasn’t built to do, and most of it is disabling security that’s there for a reason. At the end of the day you are essentially running docker directly on the host anyway, the passthrough lxc becomes less and less “there” vs passing stuff through. Then, every update to proxmox became anxiety riddled because every update would change or break something on my setup.
If you want to continue, more power to you, but I hope you heed my warnings. This is a path you will spend a lot of time on and experience a lot of frustration. Spin up a tiny debian VM and run the containers there, the overhead of the VM has been negligible, and any speedup I might have had has been made up 10x by cutting the amount of time I’ve had to hack proxmox to make it work.
Sage advice.
I went through the same thing and eventually reached the conclusion that a VM really is the best method. I did get a working LXC Docker setup going, but I could just not get it to be as stable as a VM long-term.
I might have to heed to your warnings. The nfs share is actually a mirror RAID hosted on an OMV VM on the same host. It would be just easier to spin up bookstack image on that VM itself. Thanks
The issue is that the docker container will still be running as the LXC’s root user even if you specify another user to run as in the docker compose file or run command, and if root doesn’t have access to the dir the container will always fail.
The solution to this is to remap the unprivileged LXC’s root user to a user on the Proxmox host that has access to the dir using the LXC’s config file, mount the container’s filesystem using pct mount, and then chown everything in the container owned by the default root mapped user (100000).
These are the commands I use for this:
find /var/lib/lxc/xxx/rootfs -user 100000 -type f -exec chown username {} +; find /var/lib/lxc/xxx/rootfs -user 100000 -type d -exec chown username {} +; find /var/lib/lxc/xxx/rootfs -user 100000 -type l -exec chown -h username {} +; find /var/lib/lxc/xxx/rootfs -group 100000 -type f -exec chown :username {} +; find /var/lib/lxc/xxx/rootfs -group 100000 -type d -exec chown :username {} +; find /var/lib/lxc/xxx/rootfs -group 100000 -type l -exec chown -h :username {} +(Replace xxx with the LXC number and username with the host user/UID)
If group permissions are involved you’ll also have to map those groups in the LXC config, create them in the LXC with the corresponding GIDs, add them as supplementary groups to the root user in the LXC, and then add them to the docker compose yaml using group_add.
It’s super confusing and annoying but this is the workflow I’m using now to avoid having to have any resources tied up in VMs unnecessarily.
I’m not a docker expert- i tend to just run everything in an LXC. But, doesn’t docker typically run as root? It might be that you gave your lxc user UID proper permissions, but not the lxc root UID.
Alternatively, you are aware that LXC UID 1000 != Host UID 1000, yes?
FWIW, permissions in proxmox/LXC are really clear and predictable… once you understand the way the map in the config files.
the default is root, but most containers will specify a UID/GID to run under. That makes it even harder with docker on lxc on proxmox.
This is the case with bookstack image. I specify the uid and gid as environment variables and it is expected that the ids remain compatible with the lxc and the host. But I am having trouble writing to the mount even from the lxc with the uid 1000 which is mapped to 101000 on the host.
So I don’t use docker. But. Could you set up an lxc container running samba and then bind mount the shares to that container. Then mount those shares inside docker running on the other lxc container?
That sounds a bit too convoluted for my needs. Even if it works, I am not sure if I will be able to maintain such a setup. But thanks for your inputs.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters LXC Linux Containers NFS Network File System, a Unix-based file-sharing protocol known for performance and efficiency RAID Redundant Array of Independent Disks for mass storage
[Thread #882 for this sub, first seen 24th Jul 2024, 09:45] [FAQ] [Full list] [Contact] [Source code]
It is not 1000 it 10000.
You can try with setting g+w on pve.
It is 100000 actually and even with 777 rights it still fails.
In incus, I had the same setup of an LCX container with a Docker container inside of it. I passed 1000/1000 to the LXC container but the LXC container’s default root user has a an ID set of 0/0. So I had to pass 0/0 to the Docker container, not 1000/1000 to get the read/write permissions working.
That may fix your issue as it’s basically the same tech, just different automated things implementing the LXC container!
I run dozens of proxmox LXCs, most with docker, but can confirm that while proxmox is fantastic to host NFS shares, it is near impossible to mount external ones to LXCs without doing some weirdness on the host. best practice would be to turn the NFS into samba or something and configure the docker vms and LXCs to use that (45drives has awesome repos on GitHub that work really well; that’s how I worked around that issue) the only downside is you need to usually offload everything as a backup first before the switch.
