Re: XFS Syncd

[Dave Chinner <david@fromorbit.com>]

On Tue, Jun 02, 2015 at 11:43:30AM -0700, Shrinand Javadekar wrote:
> Sorry, I dropped the ball on this one. We found some other problems
> and I was busy fixing them.
> 
> So, the xfsaild thread/s that kick in every 30 seconds are hitting us
> pretty badly. Here's a graph with the latest tests I ran. We get great
> throughput for ~18 seconds but then the world pretty much stops for
> the next ~12 seconds or so making the final numbers look pretty bad.
> This particular graph was plotted when the disk had ~150GB of data
> (total capacity of 3TB).
> 
> I am using a 3.16.0-38-generic kernel (upgraded since the time I wrote
> the first email on this thread).
> 
> I know fs.xfs.xfssyncd_centisecs controls this interval of 30 seconds.
> What other options can I tune for making this work better?
> 
> We have 8 disks. And unfortunately, all 8 disks are brought to a halt
> every 30 seconds. Does XFS have options to only work on a subset of
> disks at a time?
> 
> Also, what does XFS exactly do every 30 seconds? If I understand it
> right, metadata can be 3 locations:
> 
> 1. Memory
> 2. Log buffer on disk
> 3. Final location on disk.
> 
> Every 30 seconds, from where to where is this metadata being copied?
> Are there ways to just disable this to avoid the stop-of-the-world
> pauses (at the cost of lower but sustained performance)?

I can't use this information to help you as you haven't presented
any of the data I've asked for.  We need to restart here and base
everything on data and observation. i.e. first principles.

Can you provide all of the information here:

http://xfs.org/index.php/XFS_FAQ#Q:_What_information_should_I_include_when_reporting_a_problem.3F

and most especially the iostat and vmstat outputs while the problem
is occurring. The workload description is not what is going wrong
or what you think is happening, but a description of the application
you are running that causes the problem.

This will give me a baseline of your hardware, the software, the
behaviour and the application you are running, and hence give me
something to start with.

I'd also like to see the output from perf top while the problem is
occurring, so we might be able to see what is generating the IO...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs