Issues using new idmapper in large sites

Issues using new idmapper in large sites

[Brian De Wolf]

Hello,

I've been having some problems after upgrading to 3.4.44 that seem to
stem from the new idmapper.  We've got a site with ~36k users and our
interactive login servers pretty quickly started identifying users as
nfsnobody (-2).  Looking at /proc/key-users, we had exhausted the
available space for "keys".  After tuning these variables up to large
values, though, it still fails to cache more than ~500 users.

I made a directory with a file owned by every user and started testing
with /proc/sys/kernel/key values.  To test, my script
prints /proc/key-users, times an "ls -ln", and checks the output for
wrong uids.

Before tweaking values:

3.4.44-gentoo
    0:   620 619/514 615/1000 19995/20000
real    0m52.758s
user    0m0.370s
sys     0m7.020s
Missing users: 35784
    0:   620 619/514 615/1000 19995/20000

After tweaking values (and with a hot cache):

3.4.44-gentoo
    0:   620 619/514 615/1000000 19995/536870912
real    0m17.198s
user    0m0.410s
sys     0m5.020s
Missing users: 35784
    0: 72188 72187/514 72183/1000000 1964565/536870912

It's fast but...it also missed most of my users (it only has 503
cached, there are 36287 total).  The refcount and number of keys
skyrocket even further on repeated runs but the number of missing
users remains the same.

After testing with 3.9.6, I'm really wondering about the number of keys
instantiated being so low.  It seems to hit the same ~500 limit but does
something so that it can keep working:

3.9.6-gentoo
    0:    13 12/12 8/1000000 239/536870912
real    12m3.462s
user    0m0.440s
sys     0m10.720s
Missing users: 0
    0:   519 518/518 513/1000000 17276/536870912

The key numbers settle at ~500 and refuse to settle any higher, even on
repeated runs (although if I watch /proc/key-users while it runs, it
sometimes jumps to ~700 and goes back to ~500.  Aggressive GC?).  It
would be nice to be able to give it a bit more room to cache.

Is there anything else I should test?  Is there a tunable I missed?  It
looks like idmapping in 3.4.44 is problematic with several hundred
users and slow in 3.9.6.  Solaris performs the same test in 1 minute (1
second with a hot cache, though the cache quickly dissipates).

Thanks,
Brian

Re: Issues using new idmapper in large sites

[Myklebust, Trond]

On Tue, 2013-06-18 at 20:12 -0700, Brian De Wolf wrote:
> Hello,
> 
> I've been having some problems after upgrading to 3.4.44 that seem to
> stem from the new idmapper.  We've got a site with ~36k users and our
> interactive login servers pretty quickly started identifying users as
> nfsnobody (-2).  Looking at /proc/key-users, we had exhausted the
> available space for "keys".  After tuning these variables up to large
> values, though, it still fails to cache more than ~500 users.
> 
> I made a directory with a file owned by every user and started testing
> with /proc/sys/kernel/key values.  To test, my script
> prints /proc/key-users, times an "ls -ln", and checks the output for
> wrong uids.
> 
> Before tweaking values:
> 
> 3.4.44-gentoo
>     0:   620 619/514 615/1000 19995/20000
> real    0m52.758s
> user    0m0.370s
> sys     0m7.020s
> Missing users: 35784
>     0:   620 619/514 615/1000 19995/20000
> 
> After tweaking values (and with a hot cache):
> 
> 3.4.44-gentoo
>     0:   620 619/514 615/1000000 19995/536870912
> real    0m17.198s
> user    0m0.410s
> sys     0m5.020s
> Missing users: 35784
>     0: 72188 72187/514 72183/1000000 1964565/536870912
> 
> It's fast but...it also missed most of my users (it only has 503
> cached, there are 36287 total).  The refcount and number of keys
> skyrocket even further on repeated runs but the number of missing
> users remains the same.
> 
> After testing with 3.9.6, I'm really wondering about the number of keys
> instantiated being so low.  It seems to hit the same ~500 limit but does
> something so that it can keep working:
> 
> 3.9.6-gentoo
>     0:    13 12/12 8/1000000 239/536870912
> real    12m3.462s
> user    0m0.440s
> sys     0m10.720s
> Missing users: 0
>     0:   519 518/518 513/1000000 17276/536870912
> 
> The key numbers settle at ~500 and refuse to settle any higher, even on
> repeated runs (although if I watch /proc/key-users while it runs, it
> sometimes jumps to ~700 and goes back to ~500.  Aggressive GC?).  It
> would be nice to be able to give it a bit more room to cache.
> 
> Is there anything else I should test?  Is there a tunable I missed?  It
> looks like idmapping in 3.4.44 is problematic with several hundred
> users and slow in 3.9.6.  Solaris performs the same test in 1 minute (1
> second with a hot cache, though the cache quickly dissipates).
> 
> Thanks,
> Brian

Hi David,

What happened to your keyring patchset that was supposed to fix the
problem that Brian is reporting?

Cheers
  Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: Issues using new idmapper in large sites

[David Howells]

Myklebust, Trond <Trond.Myklebust@netapp.com> wrote:

> > It's fast but...it also missed most of my users (it only has 503
> > cached, there are 36287 total).  The refcount and number of keys
> > skyrocket even further on repeated runs but the number of missing
> > users remains the same.

Okay...  I need to improve keyring capacity.

Current max keyring capacity is

	(PAGE_SIZE - header) / sizeof(key *)

in other words, 500-ish on a 64-bit machine.

Let me have a play - in theory, it shouldn't be too hard; the main trick is
retaining RCU-based reading and, ideally, tree size.  Plus if I can sort it
too...

David