Re: [PATCH v4 0/6] fs/dcache: Limit # of negative dentries

[Waiman Long <longman@redhat.com>]

On 09/18/2017 02:20 PM, Waiman Long wrote:
>  
> A rogue application can potentially create a large number of negative
> dentries in the system consuming most of the memory available even if
> memory controller is enabled to limit memory usage. This can impact
> performance of other applications running on the system.
>
> We have customers seeing soft lockup and unresponsive system when
> tearing down a container because of the large number of negative
> dentries accumulated during its up time that had to be cleaned up at
> exit time when the container's filesystem was unmounted. So we need
> to do something about it.
>
> This patchset introduces changes to the dcache subsystem to limit
> the number of negative dentries allowed to be created thus limiting
> the amount of memory that can be consumed by negative dentries.
>
> Patch 1 just relocates the postion of the dentry_kill() function.
>
> Patch 2 tracks the number of negative dentries present in the LRU
> lists and reports it in /proc/sys/fs/dentry-state.
>
> Patch 3 sets a limit on the number of negative dentries allowable as a
> small percentage (2%) of total system memory. So the larger the system,
> the more negative dentries can be allowed. Once the limit is reached,
> new negative dentries will be killed after use.
>
> Patch 4 enables automatic pruning of least recently used negative
> dentries when it is close to the limit so that we won't end up killing
> recently used negative dentries.
>
> Patch 5 shows the number of forced negative dentry killings in
> /proc/sys/fs/dentry-state.
>
> Patch 6 enables auto-tuneup of free pool negative dentry count to
> no more than the maximum number of positive dentries ever used.
>
> With a 4.13 based kernel, the positive & negative dentries lookup rates
> (lookups per second) after initial boot on a 36-core 50GB memory VM
> with and without the patch were as follows:
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Positive dentry lookup      840269           845762
>   Negative dentry lookup     1903405          1962514
>   Negative dentry creation   6817957          6928768
>
> The last row refers to the creation rate of 1 millions negative
> dentries. With 50GB of memory, 1 millions negative dentries can be
> created with the patched kernel without any pruning or dentry killing.
>
> Ignoring some inherent noise in the test results, there wasn't any
> noticeable difference in term of lookup and negative dentry creation
> performance with or without this patch.
>
> By creating 10 millions negative dentries, however, the performance
> differed.
>
>   Metric                    w/o patch        with patch
>   ------                    ---------        ----------
>   Negative dentry creation   651663            190105
>
>
> For the patched kernel, the corresponding dentry-state was:
>
>   1608833	1590416	45	0	1579878	8286952
>
> This was expected as negative dentry creation throttling with forced
> dentry deletion happened in this case.
>
> Running the AIM7 high-systime workload on the same VM, the baseline
> performance was 186770 jobs/min. By running a single-thread rogue
> negative dentry creation program in the background until the patched
> kernel with 2% limit started throttling, the performance was 183746
> jobs/min. On an unpatched kernel with memory almost exhausted and
> memory shrinker was kicked in, the performance was 148997 jobs/min.
>
> So the patch does protect the system from suffering significant
> performance degradation in case a negative dentry creation rogue
> program is runninig in the background.
>
> Waiman Long (6):
>   fs/dcache: Relocate dentry_kill() after lock_parent()
>   fs/dcache: Track & report number of negative dentries
>   fs/dcache: Limit numbers of negative dentries
>   fs/dcache: Enable automatic pruning of negative dentries
>   fs/dcache: Track count of negative dentries forcibly killed
>   fs/dcache: Autotuning of negative dentry limit
>
>  fs/dcache.c              | 462 +++++++++++++++++++++++++++++++++++++++++++----
>  include/linux/dcache.h   |   8 +-
>  include/linux/list_lru.h |   1 +
>  mm/list_lru.c            |   4 +-
>  4 files changed, 439 insertions(+), 36 deletions(-)
>
Ping!

Any comments on this v4 patchset. We have customers seeing problems
because of the unlimited growth of negative dentries. I am hoping that
at least part of this patchset, in some form, can be merged eventually.

Thanks,
Longman