Re: Question on the xfs inode slab memory

[Dave Chinner <david@fromorbit.com>]

On Wed, May 31, 2023 at 02:29:52PM -0700, Jianan Wang wrote:
> Hi all,
> 
> I have a question regarding the xfs slab memory usage when operating a
> filesystem with 1-2 billion inodes (raid 0 with 6 disks, totally
> 18TB). On this partition, whenever there is a high disk io operation,
> like removing millions of small files, the slab kernel memory usage
> will increase a lot, leading to many OOM issues happening for the
> services running on this node. You could check some of the stats as
> the following (only includes the xfs related):

You didn't include all the XFS related slabs. At minimum, the inode
log item slab needs to be shown (xfs_ili) because that tells us how
many of the inodes in the cache have been dirtied.

As it is, I'm betting the problem is the disk subsystem can't write
back dirty inodes fast enough to keep up with memory demand and so
reclaim is declaring OOM faster than your disks can clean inodes to
enable them to be reclaimed.

> #########################################################################
> Active / Total Objects (% used):  281803052 / 317485764 (88.8%)
> Active / Total Slabs (% used): 13033144 / 13033144 (100.0%)
> Active / Total Caches (% used): 126 / 180 (70.0%)
> Active / Total Size (% used): 114671057.99K / 127265108.19K (90.1%)
> Minium / Average / Maximum Object : 0.01K / 0.40K / 16.75K
> 
> OBJS               ACTIVE      USE     OBJ SIZE     SLABS
> OBJ/SLAB    CACHE SIZE    NAME
> 78207920      70947541      0%       1.00K           7731010
>  32            247392320K     xfs_inode
> 59945928      46548798      0%       0.19K           1433102
>  42              11464816K     dentry
> 25051296      25051282      0%       0.38K           599680
>   42            9594880K         xfs_buf

Ok, that's from slabtop. Please don't autowrap stuff you've pasted
in - it makes it really hard to read. (reformatted so I can read
it).

OBJS           ACTIVE      USE     OBJ SIZE     SLABS OBJ/SLAB    CACHE SIZE    NAME
78207920      70947541      0%       1.00K     7731010   32       247392320K     xfs_inode
59945928      46548798      0%       0.19K     1433102   42        11464816K     dentry
25051296      25051282      0%       0.38K      599680   42         9594880K         xfs_buf

So, 70 million cached inodes, with a cache size of 240GB. There are
7.7 million slabs, 32 objects per slab, and that's roughly 240GB.

But why does the slab report only 78 million objects in the slab
when at 240GB there should be 240 million objects in the slab?

It looks like theres some kind of accounting problem here, likely in
the slabtop program. I have always found slabtop to be unreliable
like this....

Can you attach the output of 'cat /proc/slabinfo' and 'cat
/proc/meminfo' when you have a large slab cache in memory?

> #########################################################################
> 
> The peak slab memory usage could spike all the way to 100GB+.

Is that all? :)

> We are using Ubuntu 18.04 and the xfs version is 4.9, kernel version is 5.4

Ah, I don't think there's anything upstream can do for you. We
rewrote large portions of the XFS inode reclaim in 5.9 (3 years ago)
to address the issues with memory reclaim getting stuck on dirty XFS
inodes, so inode reclaim behaviour in modern kernels is completely
different to old kernels.

I'd suggest that you need to upgrade your systems to run a more
modern kernel and see if that fixes the issues you are seeing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com