Re: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6.27 => 2.6.32 results in 6 times slowdown]

[Michael Tokarev <mjt@tls.msk.ru>]

09.06.2010 11:47, Dave Chinner wrote:
> On Wed, Jun 09, 2010 at 10:43:37AM +0400, Michael Tokarev wrote:
>> 09.06.2010 03:18, Dave Chinner wrote:
>>> On Wed, Jun 09, 2010 at 12:34:00AM +0400, Michael Tokarev wrote:
>> []
>>>> Simple test doing random reads or writes of 4k blocks in a 1Gb
>>>> file located on an xfs filesystem, Mb/sec:
>>>>
>>>>                       sync  direct
>>>>               read   write   write
>>>> 2.6.27 xfs   1.17    3.69    3.80
>>>> 2.6.32 xfs   1.26    0.52    5.10
>>>>                      ^^^^
>>>> 2.6.32 ext3  1.19    4.91    5.02
>
> Out of curiousity, what does 2.6.34 get on this workload?

2.6.34 works quite well:
      2.6.34 xfs    1.14   4.75    5.00

The same is with -o osyncisosync (in .34).  Actually,
osyncis[od]sync mount options does not change anything, not
in .32 nor in .34.

> Also, what happens if you test with noop or deadline scheduler,
> rather than cfq (or whichever one you are using)? i.e. is this a
> scheduler regression rather than a filesystem issue?

Using deadline.  Switching to noop makes no difference whatsoever.

> Also, a block trace of the sync write workload on both .27 and .32
> would be interesting to see what the difference in IO patterns is...

I see.  Will try to collect them.  With the limited timeframe I have
to do any testing.

[]
> Well, I normally just create a raid0 lun per disk in those cases,
> hence the luns present the storage to linux as a JBOD....

That's, um, somewhat ugly :)

>> I also experimented with both O_SYNC|O_DIRECT: it is as slow as
>> without O_DIRECT, i.e. O_SYNC makes whole thing slow regardless
>> of other options.
>
> So it's the inode writeback that is causing the slowdown. We've
> recently changed O_SYNC semantics to be real O_SYNC, not O_DSYNC
> as .27 is. I can't remember if that was in 2.6.32 or not, but
> there's definitely a recent change to O_SYNC behaviouri that would
> cause this...

But there are two mount options that seems to control this behavour:
osyncisosync and osyncisdsync.  Neither of which - seemingly - makes
no difference.

>> related to block devices or usage of barriers.  For XFS it always
>> mounts like this:
>>
>>   SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
>>   SGI XFS Quota Management subsystem
>>   XFS mounting filesystem sda6
>
> So barriers are being issued.

They _are_ being issued, I knew it from the start.  What I asked
several times is if there's a way to know if they're _hitting_ the
actual low-level device (disk or raid controller).  This is entirely
different story... ;)

>> and for the device in question, it is always like
>>
>>   Adaptec aacraid driver 1.1-5[2456]-ms
>>   aacraid 0000:03:01.0: PCI INT A ->  GSI 24 (level, low) ->  IRQ 24
>>   AAC0: kernel 5.1-0[8832] Feb  1 2006
>
> Old firmware. An update might help.

Well, it worked just fine in .27.  So far I see some problem in kernel,
not in controller [firmware]... ;)

Thank you !

/mjt