Re: xfs, 2.6.27=>.32 sync write 10 times slowdown [was: xfs, aacraid 2.6.27 => 2.6.32 results in 6 times slowdown]

[Michael Tokarev <mjt@tls.msk.ru>]

09.06.2010 03:18, Dave Chinner wrote:
> On Wed, Jun 09, 2010 at 12:34:00AM +0400, Michael Tokarev wrote:
[]
>> Simple test doing random reads or writes of 4k blocks in a 1Gb
>> file located on an xfs filesystem, Mb/sec:
>>
>>                       sync  direct
>>               read   write   write
>> 2.6.27 xfs   1.17    3.69    3.80
>> 2.6.32 xfs   1.26    0.52    5.10
>>                      ^^^^
>> 2.6.32 ext3  1.19    4.91    5.02
>>
>> Note the 10 times difference between O_SYNC and O_DIRECT writes
>> in 2.6.32.  This is, well, huge difference, and this is where
>> the original slowdown comes from, apparently.
>
> Are you running on the raw block device, or on top of LVM/DM/MD to
> split up the space on the RAID drive? DM+MD have grown barrier
> support since 2.6.27, so it may be that barriers are now being
> passed down to the raid hardware on 2.6.32 and they never were on
> 2.6.27. Can you paste the output of dmesg when the XFS filesystem in

That's why I asked how to tell if barriers are actually hitting the
device in question.

No, this is the only machine where DM/MD is _not_ used.  On all other
machines we use MD software raid, this machine comes with an onboard
raid controller that does not work in JBOD mode so I weren't able to
use linux software raid.  This is XFS on top of Adaptec RAID card,
nothing in-between.

Also, as I mentioned in the previous email, remounting with nobarrier
makes no difference whatsoever.

(Another side note here - I discovered that unknown options are
silently ignored in "remount mode" while correctly rejected in
"plain mount" mode, -- it looks like a kernel bug actually, but
it's entirely different issue).

> question is mounted on both 2.6.27 and 2.6.32 so we can see if
> there is a difference in the use of barriers?
>
> Also, remember that O_DIRECT does not imply O_SYNC. O_DIRECT writes
> only write data, while O_SYNC will also write metadata and/or the
> log.

I know this.  I also found osyncisosync and osyncisdsync mount
options, and when I try to use the latter, kernel tells it's the
default and hence deprecated.  I don't need metadata updates, but
it _looks_ like the system is doing such updates (with barriers
or flushes?) anyway even when mounted with -o osyncisdsync it behaves
the same: very slow.

I also experimented with both O_SYNC|O_DIRECT: it is as slow as
without O_DIRECT, i.e. O_SYNC makes whole thing slow regardless
of other options.

I looked at the dmesg outputs, and there's no relevant differences
related to block devices or usage of barriers.  For XFS it always
mounts like this:

  SGI XFS with ACLs, security attributes, large block/inode numbers, no debug enabled
  SGI XFS Quota Management subsystem
  XFS mounting filesystem sda6

and for the device in question, it is always like

  Adaptec aacraid driver 1.1-5[2456]-ms
  aacraid 0000:03:01.0: PCI INT A -> GSI 24 (level, low) -> IRQ 24
  AAC0: kernel 5.1-0[8832] Feb  1 2006
  AAC0: monitor 5.1-0[8832]
  AAC0: bios 5.1-0[8832]
  AAC0: serial 267BE0
  AAC0: Non-DASD support enabled.
  AAC0: 64bit support enabled.
  AAC0: 64 Bit DAC enabled
  scsi0 : aacraid
  scsi 0:0:0:0: Direct-Access     Adaptec  f0500            V1.0 PQ: 0 ANSI: 2
  sd 0:0:0:0: [sda] 286715904 512-byte hardware sectors (146799 MB)
  sd 0:0:0:0: [sda] Write Protect is off
  sd 0:0:0:0: [sda] Mode Sense: 06 00 10 00
  sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
   sda: sda1 sda2 sda3 < sda5 sda6 >

There are tons of other differences, but that is to be expected (like
format of CPU topology printing which is changed between .27 and .32).

Thanks!

/mjt