A few million negative dentries causes CPU regression for open() sys calls

A few million negative dentries causes CPU regression for open() sys calls

[Ilker Yaz]

Hi all,
I've been investigating why system (kernel-mode) CPU usage of our
hosts keeps climbing up over time (over several weeks).
I realized it can be remedied by dropping cache w:
  $ echo 2 > /proc/sys/vm/drop_caches
which significantly reduces the system CPU usage like below w/o any
other change:
https://photos.app.goo.gl/aYQmi5v6qnThbt8S7

I was able to repro the same by running:
  $ stress-ng -o 10
to generate open() sys calls, and later by opening non-existent files
to increase negative dentry count to ~8.6 million:
  $ cat /proc/sys/fs/dentry-state
  11039972    10987061    45    0    8615033    0

Before stress-ng was consuming 20% system CPU (negative dentry count
was around 2 million) and after it started to consume +40% (w/ 8.6
million).
https://photos.app.goo.gl/xc3m5fwZdbqgbdaM7


I tried the same on another host by increasing regular dentry count to
~80 million by iterating over existing files. There was no regression.
Why would negative dentries have such a profound impact on open() sys
call performance?

Linux version:
  $ uname -r
  5.14.0-284.11.1.el9_2.x86_64

Thanks!

P.S. I've perf top / perf record / flamegraphs if needed.