Re: xfsaild in D state seems to be blocking all other i/o sporadically

[Dave Chinner <david@fromorbit.com>]

On Thu, Apr 20, 2017 at 09:11:22AM +0200, Michael Weissenbacher wrote:
> On 20.04.2017 01:48, Dave Chinner wrote:
> > On Wed, Apr 19, 2017 at 11:47:43PM +0200, Michael Weissenbacher wrote:
> >> OK, do i understand you correctly that the xfsaild does all the actual
> >> work of updating the inodes? And it's doing that single-threaded,
> >> reading both the log and the indoes themselves?
> > 
> > The problem is that the backing buffers that are used for flushing
> > inodes have been reclaimed due to memory pressure, but the inodes in
> > cache are still dirty. Hence to write the dirty inodes, we first
> > have to read the inode buffer back into memory.
> > 
> Interesting find. Is there a way to prevent those buffers from getting
> reclaimed?

Not really. It's simply a side effect of memory reclaim not being
able to reclaim inodes or the page cache because they are dirty, and
so it goes and puts lots more pressure on clean caches. The working
set in those other caches gets trashed, and this it's a downward
spiral because it means dirty inodes and pages take longer are
require blocking IO to refill on demand...

> Would adjusting vfs_cache_pressure help?

Unlikely.

> Or adding more
> memory to the system?

Unlikely - that'll just lead to bigger stalls.

> In fact the best thing would be to disable file
> content caching completely. Because of the use-case (backup server) it's
> worthless to cache file content.
> My primary objective is to avoid those stalls and reduce latency, at the
> expense of throughput.

Set up dirty page cache writeback thresholds to be low (a couple of
hundred MB instead of 10/20% of memory) so that data writeback
starts early and throttles dirty pages to a small amount of memory.
This will help keep the page cache clean and immediately
reclaimable, hence it shouldn't put as much pressure on other caches
when memory reclaim is required.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com