Re: [LSF/MM/BPF TOPIC] VFS idmappings support in NFS

Kohei Sugihara
2026-02-19T08:53:40Z

On Thu, Feb 19, 2026 at 9:58 AM NeilBrown <neilb@ownmail.net> wrote:
>
> On Thu, 19 Feb 2026, Jeff Layton wrote:
> > On Wed, 2026-02-18 at 15:36 +0100, Alexander Mikhalitsyn wrote:
> > > Am Mi., 18. Feb. 2026 um 14:49 Uhr schrieb Jeff Layton <jlayton@kernel.org>:
> > > >
> > > > On Wed, 2026-02-18 at 13:44 +0100, Alexander Mikhalitsyn wrote:
> > > > > Dear friends,
> > > > >
> > > > > I would like to propose "VFS idmappings support in NFS" as a topic for discussion at the LSF/MM/BPF Summit.
> > > > >
> > > > > Previously, I worked on VFS idmap support for FUSE/virtiofs [2] and cephfs [1] with support/guidance
> > > > > from Christian.
> > > > >
> > > > > This experience with Cephfs & FUSE has shown that VFS idmap semantics, while being very elegant and
> > > > > intuitive for local filesystems, can be quite challenging to combine with network/network-like (e.g. FUSE)
> > > > > FSes. In case of Cephfs we had to modify its protocol (!) (see [2]) as a part of our agreement with
> > > > > ceph folks about the right way to support idmaps.
> > > > >
> > > > > One obstacle here was that cephfs has some features that are not very Linux-wayish, I would say.
> > > > > In particular, system administrator can configure path-based UID/GID restrictions on a *server*-side (Ceph MDS).
> > > > > Basically, you can say "I expect UID 1000 and GID 2000 for all files under /stuff directory".
> > > > > The problem here is that these UID/GIDs are taken from a syscall-caller's creds (not from (struct file *)->f_cred)
> > > > > which makes cephfs FDs not very transferable through unix sockets. [3]
> > > > >
> > > > > These path-based UID/GID restrictions mean that server expects client to send UID/GID with every single request,
> > > > > not only for those OPs where UID/GID needs to be written to the disk (mknod, mkdir, symlink, etc).
> > > > > VFS idmaps API is designed to prevent filesystems developers from making a mistakes when supporting FS_ALLOW_IDMAP.
> > > > > For example, (struct mnt_idmap *) is not passed to every single i_op, but instead to only those where it can be
> > > > > used legitimately. Particularly, readlink/listxattr or rmdir are not expected to use idmapping information anyhow.
> > > > >
> > > > > We've seen very similar challenges with FUSE. Not a long time ago on Linux Containers project forum, there
> > > > > was a discussion about mergerfs (a popular FUSE-based filesystem) & VFS idmaps [5]. And I see that this problem
> > > > > of "caller UID/GID are needed everywhere" still blocks VFS idmaps adoption in some usecases.
> > > > > Antonio Musumeci (mergerfs maintainer) claimed that in many cases filesystems behind mergerfs may not be fully
> > > > > POSIX and basically, when mergerfs does IO on the underlying FSes it needs to do UID/GID switch to caller's UID/GID
> > > > > (taken from FUSE request header).
> > > > >
> > > > > We don't expect NFS to be any simpler :-) I would say that supporting NFS is a final boss. It would be great
> > > > > to have a deep technical discussion with VFS/FSes maintainers and developers about all these challenges and
> > > > > make some conclusions and identify a right direction/approach to these problems. From my side, I'm going
> > > > > to get more familiar with high-level part of NFS (or even make PoC if time permits), identify challenges,
> > > > > summarize everything and prepare some slides to navigate/plan discussion.
> > > > >
> > > > > [1] cephfs https://lore.kernel.org/linux-fsdevel/20230807132626.182101-1-aleksandr.mikhalitsyn@canonical.com
> > > > > [2] cephfs protocol changes https://github.com/ceph/ceph/pull/52575
> > > > > [3] cephfs & f_cred https://lore.kernel.org/lkml/CAEivzxeZ6fDgYMnjk21qXYz13tHqZa8rP-cZ2jdxkY0eX+dOjw@mail.gmail.com/
> > > > > [4] fuse/virtiofs https://lore.kernel.org/linux-fsdevel/20240903151626.264609-1-aleksandr.mikhalitsyn@canonical.com/
> > > > > [5]
> > > > > mergerfshttps://discuss.linuxcontainers.org/t/is-it-the-case-that-you-cannot-use-shift-true-for-disk-devices-where-the-source-is-a-mergerfs-mount-is-there-a-workaround/25336/11?u=amikhalitsyn
> > > > >
> > > > > Kind regards,
> > > > > Alexander Mikhalitsyn @ futurfusion.io
> > > >
> > >
> > > Hi Jeff,
> > >
> > > thanks for such a fast reply! ;)
> > >
> > > >
> > > > IIUC, people mostly use vfs-layer idmappings because they want to remap
> > > > the uid/gid values of files that get stored on the backing store (disk,
> > > > ceph MDS, or whatever).
> > >
> > > yes, precisely.
> > >
> > > >
> > > > I've never used idmappings myself much in practice. Could you lay out
> > > > an example of how you would use them with NFS in a real environment so
> > > > I understand the problem better? I'd start by assuming a simple setup
> > > > with AUTH_SYS and no NFSv4 idmapping involved, since that case should
> > > > be fairly straightforward.
> > >
> > > For me, from the point of LXC/Incus project, idmapped mounts are used as
> > > a way to "delegate" filesystems (or subtrees) to the containers:
> > > 1. We, of course, assume that container enables user namespaces and
> > > user can't mount a filesystem
> > > inside because it has no FS_USERNS_MOUNT flag set (like in case of Cephfs, NFS,
> > > CIFS and many others).
> > > 2. At the same time host's system administrator wants to avoid
> > > remapping between container's user ns and
> > > sb->s_user_ns (which is init_user_ns for those filesystems). [
> > > motivation here is that in many
> > > cases you may want to have the same subtree to be shared with other
> > > containers and even host users too and
> > > you want UIDs to be "compatible", i.e UID 1000 in one container and
> > > UID 1000 in another container should
> > > land as UID 1000 on the filesystem's inode ]
> > >
> > > For this usecase, when we bind-mount filesystem to container, we apply
> > > VFS idmap equal to container's
> > > user namespace. This makes a behavior I described.
> > >
> >
> > Ok: so you have a process running in a userns as UID 2000 and you want
> > to use vfs layer idmapping so that when you create a file as that user
> > that it ends up being owned by UID 1000. Is that basically correct?
> >
> > Typically, the RPC credentials used in an OPEN or CREATE call is what
> > determines its ownership (at least until a SETATTR comes in). With
> > AUTH_SYS, the credential is just a uid and set of gids.
> >
> > So in this case, it sounds like you would need just do that conversion
> > (maybe at the RPC client layer?) when issuing an RPC. You don't really
> > need a protocol extension for that case.
>
> You also need to consider the conversion when receiving an RPC.
>
> If you use krb5 and NFSv3 then you really want the mapping between krb5
> identity and uid to be the same on client and server, so then when an
> application creates a file and the stats it, it sees that it owns it.
>
> If I use a krb5 identity in an idmapped NFS filesystem I'll want the
> server to map the identity to the "underlying" uid (was would be stored
> in a local filesystem) and then when the client gets a GETATTR reply,
> the VFS maps back to the uid seen by the application.

Thank you Alex for the proposal and quick follow-ups. We're really
interested in this feature and we'd like to share our use case.

> > > But this is just one use case. I'm pretty sure there are some more
> > > around here :)
> > > I know that folks from Preferred Networks (preferred.jp) are also
> > > interested in VFS idmap support in NFS,
> > > probably they can share some ideas/use cases too.

Our use case is running multi-tenant Kubernetes clusters with
Kubernetes User Namespaces [1]. Basically we need to share a single
storage endpoint among multiple pods using ReadWriteMany (RWX) access
mode. Implementations that support both RWX and ID-mapped mount are
limited [2].

NFS is operationally common, so I am interested in supporting NFS for
ID-mapping, but NFS is complex due to its variety of mount options and
security features as Trond mentioned. We'd like to share our use case
and define the minimum goal. Our goal is here:

- 1: Mount the same NFS export as a persistent volume from multiple
Kubernetes Pods running on different compute nodes. Each tenant has
its own exports.
- 2: UID/GID in a container in the pod can be configurable to an
arbitrary value by runAsUser/runAsGroup (e.g. runAsUser/Group is set
to 1000).
- 3: We can access the export from the container as 1000:1000. At
minimum, ownership should be consistent from the container view (i.e.
stat shows 1000:1000 for files that the container creates). Today,
ID-mapped mount does not support NFS. The NFS client ends up using the
host-mapped uid/gid (e.g. container 1000 becomes host 11000), so the
container view becomes inconsistent across nodes.

There are (at least) two possible models here:
a) the NFS client sends 1000:1000 on the wire and the server stores
1000:1000 (so server-side ownership matches the container uid/gid), or
b) the server stores the host uid/gid (e.g. 11000:11000) and the
client/VFS maps it so that the container still sees 1000:1000.
My intuition is that (a) is simpler for a multi-node RWX setup, but it
may have security / policy implications depending on how the server
does authorization (especially with sec=sys). I think it’s worth
discussing what the safe and reasonable minimum should be.

In this case, UID/GID in the host node is not deterministic for the
process in the container due to user_namespaces(7), so we need to do
ID-mapping to unify UID/GID between container and file system. Also,
we likely need to consider both request and reply paths (e.g. GETATTR)
to keep the view consistent.

> Mixing in AUTH_GSS and real idmapping will be where things get harder,
> so let's not worry about those cases for now.
I totally agree with Jeff. We can start a minimum PoC with AUTH_SYS.

> With NFSv4 and the idmapper you wouldn't need (or want) the kernel
> idmapping to be used at all. You would want the idmapper deamon to run
> in the user-namespace and map from on-the-wire names to the appropriate
> app-level uids.
> This would mean that a given NFS mount would need to be an a given user
> namespace. Maybe that isn't desired.
Neil, thank you for your comment. We initially expected it to be in
NFSv4. I totally agree with you and exactly our concern is how do we
make it consistent with idmapd(8). In the Kubernetes case, we cannot
pass CAP_SYS_ADMIN to allow pods to mount NFS directly, so mount will
be done on the host. As you mentioned, we think we can share a single
NFS export from multiple hosts and pods, so I think introducing
ID-mapping into the VFS layer (with referencing local id-mapping
table) is appropriate.

We can start by picking a small case. My concern was whether this
could violate NFS protocol or not, whether things can be done on the
client side or not, and this topic is suitable for dealing with this
as the VFS community. If things can be done on the client side, we can
cover existing NFS server implementations (e.g. OpenZFS, proprietary
appliances). I believe this can be applied to recent containerized
runtime environments, even this small working set.

Adding more context, Kubernetes and the container community actively
work on host isolation using the Linux user namespace feature.
Recently they experienced RCE vulnerabilities on container runtime but
it could be mitigated by host isolation using the user namespace
isolation [3]. Along with migrating the runtime environment to user
namespace, extending file system support will be worth discussing.

Kind regards,
Kohei

[1] https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/
[2] https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes
[3] https://lpc.events/event/19/contributions/2065/