Re: Maximum Number of ACL on NFSv4

[Trond Myklebust <trondmy@hammerspace.com>]

On Wed, 2019-08-28 at 15:29 -0400, J. Bruce Fields wrote:
> On Wed, Aug 28, 2019 at 03:09:03PM -0400, Olga Kornievskaia wrote:
> > On Wed, Aug 28, 2019 at 2:06 PM J. Bruce Fields <
> > bfields@fieldses.org> wrote:
> > > On Mon, Aug 26, 2019 at 11:28:21PM +0000, de Vandiere, Louis
> > > wrote:
> > > > Thank you Olga! Somehow, I failed to look into this file
> > > > although I looked in fs/nfs/ without success and I understand
> > > > why now.
> > > > 
> > > > I'd like to see it increased and be scalable like XFS is, but I
> > > > understand it might impact multiple libraries. Should I open a
> > > > bug/feature request somewhere?
> > > 
> > > I wonder if it'd be OK to remove the limit completely (and then
> > > leave
> > > it to the filesystem to reject if if it wants).
> > > 
> > > It does mean we're passing an arbitrary client-supplied value to
> > > kmalloc.  Is it OK to do that and just leave it to the allocator
> > > to
> > > reject excessive requests, or do we risk pushing it into making
> > > heroic
> > > efforts to satisfy a possibly malicious or broken client?
> > > 
> > > I wonder if there's also a risk in passing down posix ACLs larger
> > > than
> > > could have been created with the setxattr system call.
> > > 
> > > Assuming it's still safest to have a limit....
> > > 
> > > XATTR_LIST_MAX is a global limit on the size of xattrs.  We could
> > > try to
> > > estimate how big the converted posix ACL will be and work out a
> > > maximum
> > > based on that.
> > 
> > I agree there should be a limit and using XATTR_LIST_MAX sounds
> > reasonable to me.
> 
> OK.  And, whoops, I meant to say XATTR_SIZE_MAX.  (Both are currently
> 64K, though.)
> 

Umm... Don't forget that NFSv4 ACL aces are typically much larger than
POSIX ACL aces because the user/group names are encoded as strings, not
binary uids and gids.

IOW: The size of the RPC message is likely to be a lot larger than the
resulting POSIX ACL...

> --b.
> 
> > > --b.
> > > 
> > > > Best,
> > > > Louis de Vandière
> > > > 
> > > > -----Original Message-----
> > > > From: Olga Kornievskaia <aglo@umich.edu>
> > > > Sent: Monday, August 26, 2019 2:31 PM
> > > > To: de Vandiere, Louis <louis.devandiere@atos.net>
> > > > Cc: linux-nfs@vger.kernel.org
> > > > Subject: Re: Maximum Number of ACL on NFSv4
> > > > 
> > > > From fs/nfsd/acl.h
> > > > /*
> > > >  * Maximum ACL we'll accept from a client; chosen (somewhat
> > > >  * arbitrarily) so that kmalloc'ing the ACL shouldn't require a
> > > >  * high-order allocation.  This allows 204 ACEs on x86_64:
> > > >  */
> > > > #define NFS4_ACL_MAX ((PAGE_SIZE - sizeof(struct nfs4_acl)) \
> > > >                         / sizeof(struct nfs4_ace))
> > > > 
> > > > I don't know how Bruce feels about increasing that limit.
> > > > Perhaps he'd be opened to a patch that increases that.
> > > > 
> > > > On Mon, Aug 26, 2019 at 2:30 PM de Vandiere, Louis <
> > > > louis.devandiere@atos.net> wrote:
> > > > > Thanks Niels, I tried your suggestion. According to the
> > > > > documentation (
> > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinux.die.net%2Fman%2F8%2Fmkfs.xfs&amp;data=02%7C01%7Clouis.devandiere%40atos.net%7Ce185f99cb3ad476fd39308d72a5bf6d5%7C33440fc6b7c7412cbb730e70b0198d5a%7C0%7C0%7C637024446785324973&amp;sdata=HZbnVSzTKKCXpEv5JLgZKeEgQx5BPKeEs4SYZqRhhbk%3D&amp;reserved=0
> > > > > ), the maximum size for the inode is 2048 byte. So I set it
> > > > > to this value, and faced the exact same limitation. On the
> > > > > other hand, when I used setfacl -m on the XFS mounted disk, I
> > > > > did not face any limitation and I was able to set thousands
> > > > > of ACLs on a single file.
> > > > > 
> > > > > When I do a strace, I see two different types of ACL used
> > > > > when the system calls setxattr: system.posix_acl_default and
> > > > > system.nfsv4_acl. I tried to look for hardcoded limits
> > > > > associated with system.nfsv4_acl but I don't have much
> > > > > experience with C and linux kernel.
> > > > > 
> > > > > Thank you for your help.
> > > > > Best,
> > > > > Louis de Vandière
> > > > > 
> > > > > -----Original Message-----
> > > > > From: Niels de Vos <ndevos@redhat.com>
> > > > > Sent: Monday, August 26, 2019 11:46 AM
> > > > > To: de Vandiere, Louis <louis.devandiere@atos.net>
> > > > > Cc: linux-nfs@vger.kernel.org
> > > > > Subject: Re: Maximum Number of ACL on NFSv4
> > > > > 
> > > > > On Mon, Aug 26, 2019 at 02:53:05PM +0000, de Vandiere, Louis
> > > > > wrote:
> > > > > > Yes, I assume it's not very frequent to have hundreds of
> > > > > > NFSv4 ACLs. For compliance and organizational issue, we
> > > > > > cannot use groups efficiently to manage access to the
> > > > > > shares, so it's user-based and case by case.
> > > > > > 
> > > > > > My real goal is to be able to replicate some files to a new
> > > > > > NFSv4 server while preserving the ACLs. By using "cp -R --
> > > > > > preserve=all acl-folder/", I'm able to preserve the ACLs
> > > > > > when their number does not exceed 200, above it, I see the
> > > > > > "File too large" error while rsync does not work at all
> > > > > > (even in version 3.1.3). That's why I'm digging into this
> > > > > > and checking what possibly could go wrong.
> > > > > 
> > > > > You might be hitting a limit in the filesystem on the NFS
> > > > > server. The
> > > > > ACLs are stored in extended attributes. Depending on the
> > > > > filesystem,
> > > > > you may be able to configure larger inode sizes (or other
> > > > > storage for
> > > > > xattrs). With XFS this can be done with 'mkfs -t xfs -i
> > > > > size=.. ...',
> > > > > 
> > > > > HTH,
> > > > > Niels
> > > > > 
> > > > > 
> > > > > > Thank you.
> > > > > > Best,
> > > > > > Louis de Vandière
> > > > > > 
> > > > > > 
> > > > > > -----Original Message-----
> > > > > > From: Goetz, Patrick G <pgoetz@math.utexas.edu>
> > > > > > Sent: Monday, August 26, 2019 8:44 AM
> > > > > > To: de Vandiere, Louis <louis.devandiere@atos.net>;
> > > > > > linux-nfs@vger.kernel.org
> > > > > > Subject: Re: Maximum Number of ACL on NFSv4
> > > > > > 
> > > > > > I'm dying to know what the use case is for this, and why
> > > > > > you can't just do this with group permissions (unless
> > > > > > you're talking about hundreds of group ACLs).
> > > > > > 
> > > > > > On 8/23/19 5:31 PM, de Vandiere, Louis wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > > > I'm currently trying to apply hundreds of ACLs on file
> > > > > > > hosted on a NFSv4 server (nfs-utils-1.3.0-0.61.el7.x86_64 
> > > > > > > and nfs4-acl-tools.0.3.3-19.el7.x86_64). It appears that
> > > > > > > the limit I can apply is 207. After the limit is reached,
> > > > > > > the command "nfs4_setfacl -a" returned the error "Failed
> > > > > > > setxattr operation: File too large". The same problem
> > > > > > > happens if I use an ACL with more than 200 line in it. I
> > > > > > > did a little debugging session but I was not able to come
> > > > > > > up with an explanation on why I'm facing such an issue.
> > > > > > > 
> > > > > > > On the other hand, I can apply hundreds of ACLs on XFS
> > > > > > > without issue. Do you know if it could be a bug with the
> > > > > > > nfs4-acl-tools package?
> > > > > > > Thank you for your support.
> > > > > > > Best,
> > > > > > > Louis de Vandière
> > > > > > > > > This message is from an external sender. Learn more
> > > > > > > > > about why this <<
> > > > > > > > > matters at 
> > > > > > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinks.utexas.edu%2Frtyclf&amp;data=02%7C01%7Clouis.devandiere%40atos.net%7Ce185f99cb3ad476fd39308d72a5bf6d5%7C33440fc6b7c7412cbb730e70b0198d5a%7C0%7C0%7C637024446785324973&amp;sdata=r345rqWN4GKT0mBmQmMTnaC%2FFEyUTidjBlGeAMRdEpA%3D&amp;reserved=0
> > > > > > > > > .                        <<
-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com