Limits to number of files opened by remote hosts over NFSv4?

Limits to number of files opened by remote hosts over NFSv4?

[Brian Cowan]

I am responsible for supporting an application that opens LOTS of
files over NFS from a given host, and potentially a few files/host
from a LOT of clients. We've run into some "interesting" limitations
from other OS's when it comes to NFSv4...

Solaris, for example, "only" allows 10K or so files per NFS export to
be opened over NFSv4. When you have 2500+ client hosts opening files
over NFSv4, the Solaris NFS server stops responding to "open" requests
until an entry in its state table is freed up by a file close. Which
causes single threaded client processes trying to open said files to
hang... Luckily we convinced that customer to move the clients back to
using NFSv3 since they didn't need the additional features of V4.

We're also seeing a potential issue with NetApp filers where opening
too many files from a single host seems to have issues. We're being
told that DataOnTAP has a per-client-host limit on the number of files
in the Openstate pool (and not being told what that limit is...) I say
"potential" since the only report is from things falling apart after
moving from AIX 7.2 to 7.3 (meaning there is a non-zero chance that
this is actually an AIX NFS issue). In this case, NFSv3 is not an
option since NFSv4 ACLs are required...

Anyway, as a result, I'm trying to find out if the Linux NFSv4 server
has a limit on either total number of files, total number of files per
export, or total files per host.

Re: Limits to number of files opened by remote hosts over NFSv4?

[Jeff Layton]

On Tue, 2024-07-23 at 10:15 -0400, Brian Cowan wrote:
> I am responsible for supporting an application that opens LOTS of
> files over NFS from a given host, and potentially a few files/host
> from a LOT of clients. We've run into some "interesting" limitations
> from other OS's when it comes to NFSv4...
> 
> Solaris, for example, "only" allows 10K or so files per NFS export to
> be opened over NFSv4. When you have 2500+ client hosts opening files
> over NFSv4, the Solaris NFS server stops responding to "open" requests
> until an entry in its state table is freed up by a file close. Which
> causes single threaded client processes trying to open said files to
> hang... Luckily we convinced that customer to move the clients back to
> using NFSv3 since they didn't need the additional features of V4.
> 
> We're also seeing a potential issue with NetApp filers where opening
> too many files from a single host seems to have issues. We're being
> told that DataOnTAP has a per-client-host limit on the number of files
> in the Openstate pool (and not being told what that limit is...) I say
> "potential" since the only report is from things falling apart after
> moving from AIX 7.2 to 7.3 (meaning there is a non-zero chance that
> this is actually an AIX NFS issue). In this case, NFSv3 is not an
> option since NFSv4 ACLs are required...
> 
> Anyway, as a result, I'm trying to find out if the Linux NFSv4 server
> has a limit on either total number of files, total number of files per
> export, or total files per host.
> 

An interesting question. There is a practical limit:

Each on-the-wire OPEN takes a stateid, and when we allocate a new one,
we generate its id with an idr hash. That's limited to 2^32 values, so
once you hit that many active stateids you won't be able to create
more.

Note that that includes all stateful objects (open, lock, delegation
and layout stateids), so you may not be able to do that many OPENs
depending on what else is out there.

In principle, we might be able to extend that too, but we'd need some
evidence that someone's workload was hitting that limit.

Beyond that, I can't think of any other limits we might have on being
able to create new stateful objects. I'll chime in again though if I
think of others though.
-- 
Jeff Layton <jlayton@kernel.org>

RE: Limits to number of files opened by remote hosts over NFSv4?

[Frank Filz]

Any server is likely to have some limit based on the memory use for the open state.

We recently introduced a configuration to limit the number of opens per clientid in nfs-ganesha for comparison, prior to that it was not specifically limited (and the configuration defaults to no limit) other than we would eventually run out of memory.

It's probably good to have a limit, but your case suggests a value in that limit being configurable.

Frank

> -----Original Message-----
> From: Brian Cowan [mailto:brian.cowan@hcl-software.com]
> Sent: Tuesday, July 23, 2024 7:16 AM
> To: linux-nfs@vger.kernel.org
> Subject: Limits to number of files opened by remote hosts over NFSv4?
> 
> I am responsible for supporting an application that opens LOTS of files over NFS
> from a given host, and potentially a few files/host from a LOT of clients. We've
> run into some "interesting" limitations from other OS's when it comes to
> NFSv4...
> 
> Solaris, for example, "only" allows 10K or so files per NFS export to be opened
> over NFSv4. When you have 2500+ client hosts opening files over NFSv4, the
> Solaris NFS server stops responding to "open" requests until an entry in its state
> table is freed up by a file close. Which causes single threaded client processes
> trying to open said files to hang... Luckily we convinced that customer to move
> the clients back to using NFSv3 since they didn't need the additional features of
> V4.
> 
> We're also seeing a potential issue with NetApp filers where opening too many
> files from a single host seems to have issues. We're being told that DataOnTAP
> has a per-client-host limit on the number of files in the Openstate pool (and not
> being told what that limit is...) I say "potential" since the only report is from
> things falling apart after moving from AIX 7.2 to 7.3 (meaning there is a non-
> zero chance that this is actually an AIX NFS issue). In this case, NFSv3 is not an
> option since NFSv4 ACLs are required...
> 
> Anyway, as a result, I'm trying to find out if the Linux NFSv4 server has a limit on
> either total number of files, total number of files per export, or total files per
> host.

Re: Limits to number of files opened by remote hosts over NFSv4?

[Brian Cowan]

Yes, a DOCUMENTED and configurable limit may be useful... If there is
some way to monitor it, that would help too. Because "hidden" limits
are a pain... If I'm about to hit a brick wall, I'd like to at least
see it coming...

Regards,

Brian Cowan

ClearCase/VersionVault SWAT



Mob: +1 (978) 907-2334



hcltechsw.com



On Tue, Jul 23, 2024 at 1:22 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
>
> Any server is likely to have some limit based on the memory use for the open state.
>
> We recently introduced a configuration to limit the number of opens per clientid in nfs-ganesha for comparison, prior to that it was not specifically limited (and the configuration defaults to no limit) other than we would eventually run out of memory.
>
> It's probably good to have a limit, but your case suggests a value in that limit being configurable.
>
> Frank
>
> > -----Original Message-----
> > From: Brian Cowan [mailto:brian.cowan@hcl-software.com]
> > Sent: Tuesday, July 23, 2024 7:16 AM
> > To: linux-nfs@vger.kernel.org
> > Subject: Limits to number of files opened by remote hosts over NFSv4?
> >
> > I am responsible for supporting an application that opens LOTS of files over NFS
> > from a given host, and potentially a few files/host from a LOT of clients. We've
> > run into some "interesting" limitations from other OS's when it comes to
> > NFSv4...
> >
> > Solaris, for example, "only" allows 10K or so files per NFS export to be opened
> > over NFSv4. When you have 2500+ client hosts opening files over NFSv4, the
> > Solaris NFS server stops responding to "open" requests until an entry in its state
> > table is freed up by a file close. Which causes single threaded client processes
> > trying to open said files to hang... Luckily we convinced that customer to move
> > the clients back to using NFSv3 since they didn't need the additional features of
> > V4.
> >
> > We're also seeing a potential issue with NetApp filers where opening too many
> > files from a single host seems to have issues. We're being told that DataOnTAP
> > has a per-client-host limit on the number of files in the Openstate pool (and not
> > being told what that limit is...) I say "potential" since the only report is from
> > things falling apart after moving from AIX 7.2 to 7.3 (meaning there is a non-
> > zero chance that this is actually an AIX NFS issue). In this case, NFSv3 is not an
> > option since NFSv4 ACLs are required...
> >
> > Anyway, as a result, I'm trying to find out if the Linux NFSv4 server has a limit on
> > either total number of files, total number of files per export, or total files per
> > host.
>