Re: Suggested XFS setup/options for 10TB file system w/ 18-20M files.

[Dave Chinner <david@fromorbit.com>]

On Tue, Oct 03, 2017 at 02:10:57PM -0400, R. Jason Adams wrote:
> Reformated the drive and it's refilling. With the changes
> suggested (100 dir, 512 nodes, defaults) it already seems better.
> We’re currently at 6% full and the reads are quite a bit less
> than they were before at similar fullness.

Ok, that's good - we're making progress :)

> One thing I’m
> noticing in Grafana, the read request/s seem to keep increasing
> (up to ~8/s) for around an 15 minutes, then they drop down 1/s for
> 10-15 minutes.. then over the next 15 minutes they build back up..
> etc

I'm still curious as to what the reads are - can you run a blktrace
to try to capture some of them? That will give us a better idea of
what might be done to further reduce this.

It might be worth checking what is happening with the xfs inode
cache and overall memory usage and see if this change in read
behaviour correlates with memory reclaim being active?

> > FWIW, how much RAM do you have in the system, and what does
> > 'echo 200 > /proc/sys/fs/xfs/xfssyncd_centisecs' do to the
> > behaviour?
> 
> System has 24G of ram. I’m guessing a move to 96 or 192G would
> help a lot.. in the end the system will have 36 of these 10TB
> drives.

24G should be enough if you're not walking the entire data set
periodically. The working set of inodes should be pretty small,
so the amount of memory being used to cache the working set
shouldn't be huge nor cause much trouble in terms of turnover.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com