Re: bad things when too many negative dentries in a directory

[Matthew Wilcox <willy@infradead.org>]

On Mon, Apr 14, 2025 at 10:07:09AM -0400, James Bottomley wrote:
> On Fri, 2025-04-11 at 17:01 +0100, Matthew Wilcox wrote:
> > On Fri, Apr 11, 2025 at 05:40:08PM +0200, Miklos Szeredi wrote:
> > > However, hundreds of millions of negative dentries can be created
> > > rather efficiently without unlink, though this one probably doesn't
> > > happen under normal circumstances.
> > 
> > Depends on your userspace.  Since we don't have union directories,
> > consider the not uncommon case of having a search path A:B:C. 
> > Application looks for D in directory A, doesn't find it, creates a
> > negative dentry. Application looks for D in directory B, creates a
> > negative dentry. Application looks for D in directory C, doesn't find
> > it, so it creates it. Now we have two negative dentries and one
> > positive dentry.
> 
> If an application does an A:B:C directory search pattern it's usually
> because it doesn't directly own the file location and hence suggests
> that other applications would also be looking for it, which would seem
> to indicate, if the search pattern gets repeated, that the two negative
> dentries do serve a purpose.

Not in this case.  It's doing something like looking in /etc/app.d
/usr/share/app/defaults/ and then /var/run/app/ .  Don't quote me on the
exact paths, or suggest alternatives based on these names; it's been a
few years since I last looked.  But I can assure you no other app is
looking at these dentries; they're looked up exactly once.

> > And for some applications, the name "D" is going to be unique, so the
> > negative dentries have _no_ further use.  The application isn't even
> > going to open C/D again.  If there's no memory pressure, we can build
> > up billions of dentries.  I believe the customer is currently echoing
> > 2 to /proc/sys/vm/drop-caches every hour.
> 
> So this is an application that's the sole owner of D (i.e. sole
> controller of the entire path) yet it still does a search for it, why
> is that (if it's something like to update the location, it would be
> better served by first looking in the default location before searching
> others)?  The problem is the pattern exactly matches the shared file
> one above so there doesn't seem to be a heuristic way to distinguish
> them.

Everything works fine when there's memory pressure.  The problem is that
negative dentry growth is only constrained by available memory; there's
no reclaim of negative dentries which haven't been looked at in seconds
or minutes.