Re: agsize and performance

[Stan Hoeppner <stan@hardwarefreak.com>]

On 10/30/2013 9:46 AM, K T wrote:
> I meant sync not fsync(O_SYNC flag).
> 
> My main question is why there is better throughput when I make the agsize
> smaller?

The transfer rate of a spinning rust drive is greatest on the outer
cylinders and least on the inner cylinders.

A default mkfs.xfs with 4 AGs causes the first directory created to be
placed in AG0 on the outer cylinders.  The 4th dir will be placed in AG3
on the inner cylinders.  Thus writing to the 4th directory will be
significantly slower, 4x or more, than to the 1st directory.

With 3726 allocation groups, the first few hundred directories you
create will be on the outer cylinders of the drive and writes to these
will be about the same speed, and much greater than to AG3 in the
default case.

You omitted data showing which AGs your tests were writing to in each
case, as best I could tell.  But given the above, it's most likely that
this is the cause of the behavior you are seeing.



> On Wed, Oct 30, 2013 at 5:59 AM, Matthias Schniedermeyer <ms@citd.de> wrote:
> 
>> On 29.10.2013 18:10, K T wrote:
>>> Hi,
>>>
>>> I have a 1 TB SATA disk(WD1003FBYX) with XFS. In my tests, I preallocate
>> a
>>> bunch of 10GB files and write data to the files one at a time. I have
>>> observed that the default mkfs setting(4 AGs) gives very low throughput.
>>> When I reformat the disk with a agsize of 256mb(agcount=3726), I see
>> better
>>> throughput. I thought with a bigger agsize, the files will be made of
>> fewer
>>> extents and hence perform better(due to lesser entries in the extent map
>>> getting updated). But, according to my tests, the opposite seems to be
>>> true. Can you please explain why this the case? Am I missing something?
>>>
>>> My test parameters:
>>>
>>> mkfs.xfs -f /dev/sdbf1
>>> mount  -o inode64 /dev/sdbf1 /mnt/test
>>> fallocate -l 10G fname
>>> dd if=/dev/zero of=fname bs=2M count=64 oflag=direct,sync conv=notrunc
>> seek=0
>>
>> I get the same bad performance with your dd statement.
>>
>> fallocate -l 10G fname
>> time dd if=/dev/zero of=fname bs=2M count=64 oflag=direct,sync
>> conv=notrunc seek=0
>> 64+0 records in
>> 64+0 records out
>> 134217728 bytes (134 MB) copied, 4,24088 s, 31,6 MB/s
>>
>> After pondering the really hard to read dd-man-page.
>> Sync is for 'synchronized' I/O. aka REALLY BAD PERFORMANCE. And i assume
>> you don't really that.
>>
>> I think what you meant is fsync. (a.k.a. File (and Metadata) has hit
>> stable-storage before dd exits).
>> That is: conv=fsync
>>
>> So:
>> time dd if=/dev/zero of=fname bs=2M count=64 oflag=direct
>> conv=notrunc,fsync seek=0
>> 64+0 records in
>> 64+0 records out
>> 134217728 bytes (134 MB) copied, 1,44088 s, 93,2 MB/s
>>
>> That gets much better performance, and in my case it can't get any
>> better because the HDD (and encryption) just can't go any faster.
>>
>>
>>
>>
>> --
>>
>> Matthias
>>
> 
> 
> 
> _______________________________________________
> xfs mailing list
> xfs@oss.sgi.com
> http://oss.sgi.com/mailman/listinfo/xfs
> 

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs