Re: Suggested XFS setup/options for 10TB file system w/ 18-20M files.

["R. Jason Adams" <rjasonadams@gmail.com>]

> 
> With 4k directory block size and your write heavy workload, you
> could get away with just 10 directories. However, it'd probably be
> better to use a single level 100-directory wide hash to bring to
> down to less than 200k files per directory….


Moved over to single level with 100 directories. 


> Small files should be a single extent, so there's heaps of room for
> a 200 byte xattr in the inode. using 512 byte inodes will half
> memory demand for caching inode buffers….

Moved to 512 byte inodes.

> In general, use the defaults and don't add anything extra unless you
> know it solves a specific problem you've witnessed in testing…

Moved to the defaults.

> 
> Most likely going to be metadata writeback of inode buffers
> requiring RMW based on experience with gluster and ceph having
> exactly the same problems.  Use blktrace to identify what the reads
> are, and see if those same blocks are written later on. An io marked
> a "M" is a metadata IO. Post the blktrace output of the bits you
> find relevant.

Reformated the drive and it's refilling. With the changes suggested (100 dir, 512 nodes, defaults) it already seems better. We’re currently at 6% full and the reads are quite a bit less than they were before at similar fullness. One thing I’m noticing in Grafana, the read request/s seem to keep increasing (up to ~8/s) for around an 15 minutes, then they drop down 1/s for 10-15 minutes.. then over the next 15 minutes they build back up.. etc

> FWIW, how much RAM do you have in the system, and what does 'echo
> 200 > /proc/sys/fs/xfs/xfssyncd_centisecs' do to the behaviour?

System has 24G of ram. I’m guessing a move to 96 or 192G would help a lot.. in the end the system will have 36 of these 10TB drives.

I want to thank you and Eric for the time you’ve taken to help. Feels good to make some progress on this issue.

-R. Jason Adams