Re: XFS / xfs_repair - problem reading very large sparse files on very large filesystem

[Dave Chinner <david@fromorbit.com>]

On Thu, Nov 04, 2021 at 10:09:15AM +0100, Nikola Ciprich wrote:
> Hello fellow XFS users and developers,
> 
> we've stumbled upon strange problem which I think might be somewhere
> in XFS code.
> 
> we have very large ceph-based storage on top which there is 1.5PiB volume
> with XFS filesystem. This contains very large (ie 500TB) sparse files,
> partially filled with data.
> 
> problem is, trying to read those files leads to processes blocked in D
> state showing very very bad performance - ~200KiB/s, 50IOPS.

It's been told it to go slow... :/

> I tried running xfs_repair on the volume, but this seems to behave in
> very similar way - very quickly it gets into almost stalled state, without
> almost any progress..
> 
> [root@spbstdnas ~]# xfs_repair -P -t 60 -v -v -v -v /dev/sdk

.... because "-P" turns off prefetching and all the IO optimisation
that comes along with the prefetching mechanisms. In effect, "-P"
means "go really slowly".

Try:

# xfs_repair -o bhash_size=101371 -o ag_stride=100 /dev/sdk

To get a good sized buffer cache and a decent (but not excessive)
amount of concurrency in the scanning processing. It still may end
up being slow if it has to single thread walk a huge btree
(essentially pointer chasing on disk), but at least that won't hold
up all the other scanning that isn't dependent on that huge btree..

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com