Re: [PATCH RFC] vfs: Introduce a new open flag to imply dentry deletion on file removal

[Matthew Wilcox <willy@infradead.org>]

On Thu, Sep 12, 2024 at 01:36:45PM +0200, Mateusz Guzik wrote:
> I have to note what to do with a dentry after unlink is merely a subset
> of the general problem of what to do about negative entries.  I had a
> look at it $elsewhere some years back and as one might suspect userspace
> likes to do counterproductive shit. For example it is going to stat a
> non-existent path 2-3 times and then open(..., O_CREAT) on it.
> 
> I don't have numbers handy and someone(tm) will need to re-evaluate, but
> crux of the findings was as follows:
> - there is a small subset of negative entries which keep getting tons of
>   hits
> - a sizeable count literally does not get any hits after being created
>   (aka wastes memory)
> - some negative entries get 2-3 hits and get converted into a positive
>   entry afterwards (see that stat shitter)
> - some flip flop with deletion/creation
> 
> So whatever magic mechanism, if it wants to mostly not get in the way in
> terms of performance, will have to account for the above.
> 
> I ended up with a kludge where negative entries hang out on some number
> of LRU lists and get promoted to a hot list if they manage to get some
> number of hits. The hot list is merely a FIFO and entries there no
> longer count any hits. Removal from the cold LRU also demotes an entry
> from the hot list.

This all reminds me of that paper you pointed me at.

https://arxiv.org/pdf/1512.00727

Summary for the impatient: Use 10% of the memory for a "not yet proven
to be useful" entries, and use multiple Bloom filters to decide which
ones are sufficiently useful to be added to the "more permanent" cache.

I don't think it solves every problem our current dcache implementation
has, but I'm 90% sure it'll be a huge improvement.