rpc.mountd & manage-gids behaviour change?

rpc.mountd & manage-gids behaviour change?

[Daire Byrne]

Hi,

We have lots of Linux storage servers running combinations of RHEL7,
RHEL8 and more recently RHEL9. We also use "manage-gids" and have lots
of groups of users and apply permissions to directories on the
exported filesystems.

We also use sssd and AD/LDAP on these storage servers to resolve the
groups and do the user lookups. This setup has worked great for our
needs for many years but we have noticed a change in RHEL9 which
results in many more uid/gid lookups hitting our LDAP servers.

It seems like with RHEL7 & 8 era kernels and nfs-utils, sssd/nss would
receive a single request from rpc.mountd whereas with RHEL9 we now get
duplicated requests for each rpc.mountd thread (8 by default) even for
a single client mount. So 8 uid/gid requests hit sssd at the same
time, and because it's not in cache, all those 8 requests go out over
the wire to our AD server.

So for lookups not in the cache, we have 8 times more requests hitting
our LDAP servers. Not to mention that sssd sometimes crashes or loses
connectivity with the LDAP server with this increased load.

I had a look through the changes for linux-nfs but nothing jumped out
at me in that time frame (other than code to make exportd
multi-threaded). Does anyone have any ideas where this change of
behaviour might be coming from?

RHEL9: nfs-utils-2.5.4
RHEL8: nfs-utils-2.3.3

Cheers,

Daire

Re: rpc.mountd & manage-gids behaviour change?

[Daire Byrne]

For the benefit of anyone who finds this thread via a search, the
issue turns out to be this change to SSSD which introduced parallel
requests:

https://sssd.io/release-notes/sssd-2.7.3.html

But then the rpc.mountd "client" threads would send 8 requests
(rpc.mountd threads) to SSSD which then all get sent over the wire to
your LDAP/AD server whereas before it was a single request.

New fixes from Redhat:

https://bugzilla.redhat.com/show_bug.cgi?id=2234829
https://github.com/SSSD/sssd/issues/6911

Daire

On Thu, 24 Aug 2023 at 18:39, Daire Byrne <daire@dneg.com> wrote:
>
> Hi,
>
> We have lots of Linux storage servers running combinations of RHEL7,
> RHEL8 and more recently RHEL9. We also use "manage-gids" and have lots
> of groups of users and apply permissions to directories on the
> exported filesystems.
>
> We also use sssd and AD/LDAP on these storage servers to resolve the
> groups and do the user lookups. This setup has worked great for our
> needs for many years but we have noticed a change in RHEL9 which
> results in many more uid/gid lookups hitting our LDAP servers.
>
> It seems like with RHEL7 & 8 era kernels and nfs-utils, sssd/nss would
> receive a single request from rpc.mountd whereas with RHEL9 we now get
> duplicated requests for each rpc.mountd thread (8 by default) even for
> a single client mount. So 8 uid/gid requests hit sssd at the same
> time, and because it's not in cache, all those 8 requests go out over
> the wire to our AD server.
>
> So for lookups not in the cache, we have 8 times more requests hitting
> our LDAP servers. Not to mention that sssd sometimes crashes or loses
> connectivity with the LDAP server with this increased load.
>
> I had a look through the changes for linux-nfs but nothing jumped out
> at me in that time frame (other than code to make exportd
> multi-threaded). Does anyone have any ideas where this change of
> behaviour might be coming from?
>
> RHEL9: nfs-utils-2.5.4
> RHEL8: nfs-utils-2.3.3
>
> Cheers,
>
> Daire