Re: gather write metrics on multiple files

[Stan Hoeppner <stan@hardwarefreak.com>]

On 10/08/2014 11:49 PM, Joe Landman wrote:
> On 10/09/2014 12:40 AM, Stan Hoeppner wrote:
>> Does anyone know of a utility that can track writes to files in an
>> XFS directory tree, or filesystem wide for that matter, and gather
>> filesystem blocks written per second data, or simply KiB/s, etc?  I
>> need to analyze an application's actual IO behavior to see if it
>> matches what I'm being told the application is supposed to be doing.
>>
> 
> We've written a few for this purpose (local IO probing).
> 
> Start with collectl (looks at /proc/diskstats), and others.  Our tools go to /proc/diskstats, and use this to compute BW and IOPs per device.
> 
> If you need to log it for a long time, set up a time series database (we use influxdb and the graphite plugin).  Then grab your favorite metrics tool that talks to graphite/influxdb (I like https://github.com/joelandman/sios-metrics for obvious reasons), and start collecting data.

I'm told we have 800 threads writing to nearly as many files concurrently on a single XFS on a 12+2 spindle RAID6 LUN.  Achieved data rate is currently ~300 MiB/s.  Some of these are files are supposedly being written at a rate of only 32KiB every 2-3 seconds, while some (two) are ~50 MiB/s.  I need to determine how many bytes we're writing to each of the low rate files, and how many files, to figure out RMW mitigation strategies.  Out of the apparent 800 streams 700 are these low data rate suckers, one stream writing per file.  

Nary a stock RAID controller is going to be able to assemble full stripes out of these small slow writes.  With a 768 KiB stripe that's what, 24 seconds to fill it at 2 seconds per 32 KiB IO?  I've been playing with bcache for a few days but it actually drops throughput by about 30% no matter how I turn its knobs.  Unless I can get Kent to respond to some of my questions bcache will be a dead end.  I had high hopes for it, thinking it would turn these small random IOs into larger sequential writes.  It may actually be doing so, but it's doing something else too, and badly.  IO times go through the roof once bcache starts gobbling IOs, and throughput to the LUNs drops significantly even though bcache is writing 50-100 MIB/s to the SSD.  Not sure what's causing that.


>> Thanks, Stan
>>
>> _______________________________________________ xfs mailing list
>> xfs@oss.sgi.com http://oss.sgi.com/mailman/listinfo/xfs
>>
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs