After reboot fs with barrier faster deletes then fs with nobarrier

After reboot fs with barrier faster deletes then fs with nobarrier

[Szabolcs Illes]

Hi,

I am using XFS on my laptop, I have realized that nobarrier mount options  
sometimes slows down deleting large number of small files, like the kernel  
source tree. I made four tests, deleting the kernel source right after  
unpack and after reboot, with both barrier and nobarrier options:

mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync &&  
reboot

After reboot:
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m28.127s
user    0m0.044s
sys     0m2.924s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m14.872s
user    0m0.044s
sys     0m2.872s

-------------------------------------------------------------------
mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync &&  
reboot
After reboot:

illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    1m12.738s
user    0m0.032s
sys     0m2.548s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m7.884s
user    0m0.028s
sys     0m2.008s

It looks like with barrier it's faster deleting files after reboot. ( 28  
sec vs 72 sec !!! ).
I thought it was supposed to be faster using the nobarrier mount options,  
but is not, very strange.

However deleting right after the unpack ( so everything in mem cache )  is  
faster with nobarrier ( 15 sec vs 9 sec ), not to much surprise here.

I repeated this test several times, same results. I made sure nothing was  
running while I was doing the tests, cpu was idle, hdd led was not on, etc.

I have found nothing in the logs regarding to barrier is not working.

Can anyone explain this?
Is it normal?
Is there any point using barrier (expect it's sometimes faster :) ) on a  
laptop?

Cheers,
Szabolcs


Some info:
Test system: opensuse10.2
xfsprogs-2.8.11_1-11

sunset:~ # uname -a
Linux sunset 2.6.18.8-0.3-default #1 SMP Tue Apr 17 08:42:35 UTC 2007 i686  
i686 i386 GNU/Linux

sunset:~ # xfs_info:

meta-data=/dev/hda3              isize=256    agcount=8, agsize=946078 blks
          =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=7568623, imaxpct=25
          =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=3695, version=2
          =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

sunset:~ # hdparm /dev/hda

/dev/hda:
  multcount    = 16 (on)
  IO_support   =  1 (32-bit)
  unmaskirq    =  0 (off)
  using_dma    =  1 (on)
  keepsettings =  0 (off)
  readonly     =  0 (off)
  readahead    = 256 (on)
  geometry     = 16383/255/63, sectors = 78140160, start = 0

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Chris Wedgwood]

On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:

> I repeated this test several times, same results. I made sure
> nothing was running while I was doing the tests, cpu was idle, hdd
> led was not on, etc.

instead of doing a reboot can you try something like:

    echo 1 > /proc/sys/vm/drop_caches

or
    echo 3 > /proc/sys/vm/drop_caches

(the value is a bit mask, bit 0 will drop the page cache, bit 1 will
drop the slab)

does that give you more or less the same results as rebooting?




another thing to try, before the delete, also try:

    find path/to/whatever -noleaf >/dev/null

and see if that helps (i expect it should greatly)

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Szabolcs Illes]

On Wed, 27 Jun 2007 22:45:06 +0100, Chris Wedgwood <cw@f00f.org> wrote:

> On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
>
>> I repeated this test several times, same results. I made sure
>> nothing was running while I was doing the tests, cpu was idle, hdd
>> led was not on, etc.
>
> instead of doing a reboot can you try something like:
>
>     echo 1 > /proc/sys/vm/drop_caches
>
> or
>     echo 3 > /proc/sys/vm/drop_caches
>
> (the value is a bit mask, bit 0 will drop the page cache, bit 1 will
> drop the slab)
>
> does that give you more or less the same results as rebooting?
yes it does.
>
>
>
>
> another thing to try, before the delete, also try:
>
>     find path/to/whatever -noleaf >/dev/null
>
> and see if that helps (i expect it should greatly)
It doesn't help to much, see the updated tests:

mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync &&  
reboot

After reboot:
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m28.127s
user    0m0.044s
sys     0m2.924s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
real    0m14.872s
user    0m0.044s
sys     0m2.872s


illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 1 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m28.257s
user    0m0.036s
sys     0m2.732s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 3 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m28.155s
user    0m0.048s
sys     0m2.772s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 3 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> find  linux-2.6.21.5/ -noleaf >/dev/null
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m25.702s
user    0m0.064s
sys     0m2.664s

-------------------------------------------------------------------
mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync &&  
reboot
After reboot:

illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    1m12.738s
user    0m0.032s
sys     0m2.548s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    0m7.884s
user    0m0.028s
sys     0m2.008s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 1 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    1m15.367s
user    0m0.048s
sys     0m2.264s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 3 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    1m16.043s
user    0m0.060s
sys     0m2.448s

illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
sunset:~ # echo 3 > /proc/sys/vm/drop_caches
illes@sunset:~/tmp> find  linux-2.6.21.5/ -noleaf >/dev/null
illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/

real    1m7.856s
user    0m0.044s
sys     0m2.020s


Cheers,
Szabolcs

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[David Chinner]

On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
> Hi,
> 
> I am using XFS on my laptop, I have realized that nobarrier mount options  
> sometimes slows down deleting large number of small files, like the kernel  
> source tree. I made four tests, deleting the kernel source right after  
> unpack and after reboot, with both barrier and nobarrier options:
> 
> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
> 
> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync &&  
> reboot
> 
> After reboot:
> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
> 
> real    0m28.127s
> user    0m0.044s
> sys     0m2.924s
> 
> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync
> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
> 
> real    0m14.872s
> user    0m0.044s
> sys     0m2.872s

Of course the second run will be faster here - the inodes are already in
cache and so there's no reading from disk needed to find the files
to delete....

That's because run time after reboot is determined by how fast you
can traverse the directory structure (i.e. how many seeks are
involved). Barriers will have little impact on the uncached rm -rf
results, but as the cached rm time will be log-I/O bound, nobarrier
will be far faster (as you've found out).

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Timothy Shimmin]

David Chinner wrote:
> On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
>> Hi,
>>
>> I am using XFS on my laptop, I have realized that nobarrier mount options  
>> sometimes slows down deleting large number of small files, like the kernel  
>> source tree. I made four tests, deleting the kernel source right after  
>> unpack and after reboot, with both barrier and nobarrier options:
>>
 >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
 >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && reboot
 >> After reboot:
 >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
 >> real    0m28.127s
 >> user    0m0.044s
 >> sys     0m2.924s
 >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier
 >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && reboot
 >> After reboot:
 >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
 >> real    1m12.738s
 >> user    0m0.032s
 >> sys     0m2.548s
 >> It looks like with barrier it's faster deleting files after reboot.
 >> ( 28 sec vs 72 sec !!! ).
> 
> Of course the second run will be faster here - the inodes are already in
> cache and so there's no reading from disk needed to find the files
> to delete....
> 
> That's because run time after reboot is determined by how fast you
> can traverse the directory structure (i.e. how many seeks are
> involved). 
> Barriers will have little impact on the uncached rm -rf
> results, 

But it looks like barriers _are_ having impact on the uncached rm -rf
results.

--Tim

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Szabolcs Illes]

On Thu, 28 Jun 2007 06:00:46 +0100, Timothy Shimmin <tes@sgi.com> wrote:

> David Chinner wrote:
>> On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
>>> Hi,
>>>
>>> I am using XFS on my laptop, I have realized that nobarrier mount  
>>> options  sometimes slows down deleting large number of small files,  
>>> like the kernel  source tree. I made four tests, deleting the kernel  
>>> source right after  unpack and after reboot, with both barrier and  
>>> nobarrier options:
>>>
>  >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
>  >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 &&  
> sync && reboot
>  >> After reboot:
>  >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
>  >> real    0m28.127s
>  >> user    0m0.044s
>  >> sys     0m2.924s
>  >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier
>  >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 &&  
> sync && reboot
>  >> After reboot:
>  >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
>  >> real    1m12.738s
>  >> user    0m0.032s
>  >> sys     0m2.548s
>  >> It looks like with barrier it's faster deleting files after reboot.
>  >> ( 28 sec vs 72 sec !!! ).
>>  Of course the second run will be faster here - the inodes are already  
>> in
>> cache and so there's no reading from disk needed to find the files
>> to delete....
>>  That's because run time after reboot is determined by how fast you
>> can traverse the directory structure (i.e. how many seeks are
>> involved). Barriers will have little impact on the uncached rm -rf
>> results,
>
> But it looks like barriers _are_ having impact on the uncached rm -rf
> results.
and the same happens on my desktop pc which is much more faster, 64 bit  
dual core P4, sata drive, etc.

nobarrier:

+ echo 3 > /proc/sys/vm/drop_caches
+ rm -rf /root/fs/linux-2.6.21.5 /root/fs/x.tar

real    0m12.526s
user    0m0.016s
sys     0m1.244s

barrier:

+ echo 3 > /proc/sys/vm/drop_caches
+ rm -rf /root/fs/linux-2.6.21.5 /root/fs/x.tar

real    0m6.784s
user    0m0.032s
sys     0m1.244s

Szabolcs

>
> --Tim

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[David Chinner]

On Thu, Jun 28, 2007 at 03:00:46PM +1000, Timothy Shimmin wrote:
> David Chinner wrote:
> >On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
> >>Hi,
> >>
> >>I am using XFS on my laptop, I have realized that nobarrier mount options 
> >>sometimes slows down deleting large number of small files, like the 
> >>kernel  source tree. I made four tests, deleting the kernel source right 
> >>after  unpack and after reboot, with both barrier and nobarrier options:
> >>
> >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
> >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && 
> reboot
> >> After reboot:
> >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
> >> real    0m28.127s
> >> user    0m0.044s
> >> sys     0m2.924s
> >> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier
> >> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && 
> reboot
> >> After reboot:
> >> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
> >> real    1m12.738s
> >> user    0m0.032s
> >> sys     0m2.548s
> >> It looks like with barrier it's faster deleting files after reboot.
> >> ( 28 sec vs 72 sec !!! ).
> >
> >Of course the second run will be faster here - the inodes are already in
> >cache and so there's no reading from disk needed to find the files
> >to delete....
> >
> >That's because run time after reboot is determined by how fast you
> >can traverse the directory structure (i.e. how many seeks are
> >involved). 
> >Barriers will have little impact on the uncached rm -rf
> >results, 
> 
> But it looks like barriers _are_ having impact on the uncached rm -rf
> results.

Tim, please be care with what you quote - you've quoted a different
set of results wot what I did and commented on and that takes my
comments way out of context.

In hindsight, I should have phrased it as "barriers _should_ have
little impact on uncached rm -rf results."

We've seen little impact in the past, and it's always been a
decrease in performance, so what we need to find out is how they are
having an impact. I suspect that it's to do with drive cache control
algorithms and barriers substantially reducing the amount of dirty
data being cached and hence read caching is working efficiently as a
side effect.

I guess the only way to confirm this is blktrace output to see what
I/Os are taking longer to execute when barriers are disabled.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[David Chinner]

On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
> Hi,
> 
> I am using XFS on my laptop, I have realized that nobarrier mount options  
> sometimes slows down deleting large number of small files, like the kernel  
> source tree. I made four tests, deleting the kernel source right after  
> unpack and after reboot, with both barrier and nobarrier options:
> 
> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2

FWIW, I bet these mount options have something to do with the
issue.

Here's the disk I'm testing against - 36GB 10krpm u160 SCSI:

<5>[   25.427907] sd 0:0:2:0: [sdb] 71687372 512-byte hardware sectors (36704 MB)
<5>[   25.440393] sd 0:0:2:0: [sdb] Write Protect is off
<7>[   25.441276] sd 0:0:2:0: [sdb] Mode Sense: ab 00 10 08
<5>[   25.442662] sd 0:0:2:0: [sdb] Write cache: disabled, read cache: enabled, supports DPO and FUA
<6>[   25.446992]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb7 sdb8 sdb9

Note - read cache is enabled, write cache is disabled, so barriers
cause a FUA only.  i.e. the only bubble in the I/O pipeline that
barriers cause are in teh elevator and the scsi command queue.

The disk is capable of about 30MB/s on the inner edge.

Mount options are default (so logbsize=32k,logbufs=8), mkfs
options are default, 4GB partition on inner (slow) edge of disk.
Kernel is 2.6.22-rc4 with all debug and tracing options turned on
on ia64.

For this config, I see:

		barrier		nobarrier
hot cache	22s		14s
cold cache 	21s		20s

In this case, barriers have little impact on cold cache behaviour,
and the difference on the hot cache behaviour will probably be
because of FUA being used on barrier writes (i.e. no combining
of sequential log I/Os in the elevator).

The difference in I/O behaviour b/t hot cache and cold cache during
the rm -rf  is that there are zero read I/Os on a hot cache and
50-100 read I/Os per second on a cold cache which is easily
within the capability of this drive.

After turning on the write cache with:

# sdparm -s WCE -S /dev/sdb
# reboot

[   25.717942] sd 0:0:2:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA

I get:
					barrier		nobarrier
logbsize=32k,logbufs=8: hot cache	24s		11s
logbsize=32k,logbufs=8: cold cache 	33s		16s
logbsize=256k,logbufs=8: hot cache	10s		10s
logbsize=256k,logbufs=8: cold cache 	16s		16s
logbsize=256k,logbufs=2: hot cache	11s		9s
logbsize=256k,logbufs=2: cold cache 	17s		13s

Out of the box, barriers are 50% slower with WCE=1 than with WCE=0
on the cold cache test, but are almost as fast with larger
log buffer size (i.e. less barrier writes being issued).

Worth noting is that at 10-11s runtime, the disk is bandwidth
bound (i.e. we're doing 30MB/s), so that's the fastest time
rm -rf will do on this filesystem.

So, clearly we have differing performance depending on
mount options and at best barriers give equal performance.

I just ran the same tests on an x86_64 box with 7.2krpm 500GB SATA
disks with WCE (2.6.18 kernel) using a 30GB partition on the outer
edge:

					barrier		nobarrier
logbsize=32k,logbufs=8: hot cache	29s		29s
logbsize=32k,logbufs=8: cold cache 	33s		30s
logbsize=256k,logbufs=8: hot cache	8s		8s
logbsize=256k,logbufs=8: cold cache 	11s		11s
logbsize=256k,logbufs=2: hot cache	8s		8s
logbsize=256k,logbufs=2: cold cache 	11s		11s

Barriers make little to zero difference here.

> Can anyone explain this?

Right now I'm unable to reproduce your results even on 2.6.18 so I
suspect a drive level issue here.

Can I suggest that you try the same tests with write caching turned
off on the drive(s)? (hdparm -W 0 <dev>, IIRC).

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Timothy Shimmin]

Hi Dave,

David Chinner wrote:
> On Thu, Jun 28, 2007 at 03:00:46PM +1000, Timothy Shimmin wrote:
>> David Chinner wrote:
>>> On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
>>>> Hi,
>>>>
>>>> I am using XFS on my laptop, I have realized that nobarrier mount options 
>>>> sometimes slows down deleting large number of small files, like the 
>>>> kernel  source tree. I made four tests, deleting the kernel source right 
>>>> after  unpack and after reboot, with both barrier and nobarrier options:
>>>>
>>>> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
>>>> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && 
>> reboot
>>>> After reboot:
>>>> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
>>>> real    0m28.127s
>>>> user    0m0.044s
>>>> sys     0m2.924s
>>>> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2,nobarrier
>>>> illes@sunset:~/tmp> tar xjf ~/Download/linux-2.6.21.5.tar.bz2 && sync && 
>> reboot
>>>> After reboot:
>>>> illes@sunset:~/tmp> time rm -rf linux-2.6.21.5/
>>>> real    1m12.738s
>>>> user    0m0.032s
>>>> sys     0m2.548s
>>>> It looks like with barrier it's faster deleting files after reboot.
>>>> ( 28 sec vs 72 sec !!! ).
>>> Of course the second run will be faster here - the inodes are already in
>>> cache and so there's no reading from disk needed to find the files
>>> to delete....
>>>
>>> That's because run time after reboot is determined by how fast you
>>> can traverse the directory structure (i.e. how many seeks are
>>> involved). 
>>> Barriers will have little impact on the uncached rm -rf
>>> results, 
>> But it looks like barriers _are_ having impact on the uncached rm -rf
>> results.
> 
> Tim, please be care with what you quote - you've quoted a different
> set of results wot what I did and commented on and that takes my
> comments way out of context.

Sorry for rearranging the quote (haven't touched it this time ;-).
My aim was just to highlight the uncached results which I thought were a
bit surprising. (The other results not being surprising)
I was wondering what your take on that was.

> 
> In hindsight, I should have phrased it as "barriers _should_ have
> little impact on uncached rm -rf results."
> 
> We've seen little impact in the past, and it's always been a
> decrease in performance, so what we need to find out is how they are
> having an impact. I suspect that it's to do with drive cache control
> algorithms and barriers substantially reducing the amount of dirty
> data being cached and hence read caching is working efficiently as a
> side effect.
> 
> I guess the only way to confirm this is blktrace output to see what
> I/Os are taking longer to execute when barriers are disabled.
> 
Yep.

--Tim

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[Szabolcs Illes]

On Fri, 29 Jun 2007 01:16:48 +0100, David Chinner <dgc@sgi.com> wrote:

> On Wed, Jun 27, 2007 at 06:58:29PM +0100, Szabolcs Illes wrote:
>> Hi,
>>
>> I am using XFS on my laptop, I have realized that nobarrier mount  
>> options
>> sometimes slows down deleting large number of small files, like the  
>> kernel
>> source tree. I made four tests, deleting the kernel source right after
>> unpack and after reboot, with both barrier and nobarrier options:
>>
>> mount opts: rw,noatime,nodiratime,logbsize=256k,logbufs=2
>
> FWIW, I bet these mount options have something to do with the
> issue.
>
> Here's the disk I'm testing against - 36GB 10krpm u160 SCSI:
>
> <5>[   25.427907] sd 0:0:2:0: [sdb] 71687372 512-byte hardware sectors  
> (36704 MB)
> <5>[   25.440393] sd 0:0:2:0: [sdb] Write Protect is off
> <7>[   25.441276] sd 0:0:2:0: [sdb] Mode Sense: ab 00 10 08
> <5>[   25.442662] sd 0:0:2:0: [sdb] Write cache: disabled, read cache:  
> enabled, supports DPO and FUA
> <6>[   25.446992]  sdb: sdb1 sdb2 sdb3 sdb4 sdb5 sdb6 sdb7 sdb8 sdb9
>
> Note - read cache is enabled, write cache is disabled, so barriers
> cause a FUA only.  i.e. the only bubble in the I/O pipeline that
> barriers cause are in teh elevator and the scsi command queue.
>
> The disk is capable of about 30MB/s on the inner edge.
>
> Mount options are default (so logbsize=32k,logbufs=8), mkfs
> options are default, 4GB partition on inner (slow) edge of disk.
> Kernel is 2.6.22-rc4 with all debug and tracing options turned on
> on ia64.
>
> For this config, I see:
>
> 		barrier		nobarrier
> hot cache	22s		14s
> cold cache 	21s		20s
>
> In this case, barriers have little impact on cold cache behaviour,
> and the difference on the hot cache behaviour will probably be
> because of FUA being used on barrier writes (i.e. no combining
> of sequential log I/Os in the elevator).
>
> The difference in I/O behaviour b/t hot cache and cold cache during
> the rm -rf  is that there are zero read I/Os on a hot cache and
> 50-100 read I/Os per second on a cold cache which is easily
> within the capability of this drive.
>
> After turning on the write cache with:
>
> # sdparm -s WCE -S /dev/sdb
> # reboot
>
> [   25.717942] sd 0:0:2:0: [sdb] Write cache: enabled, read cache:  
> enabled, supports DPO and FUA
>
> I get:
> 					barrier		nobarrier
> logbsize=32k,logbufs=8: hot cache	24s		11s
> logbsize=32k,logbufs=8: cold cache 	33s		16s
> logbsize=256k,logbufs=8: hot cache	10s		10s
> logbsize=256k,logbufs=8: cold cache 	16s		16s
> logbsize=256k,logbufs=2: hot cache	11s		9s
> logbsize=256k,logbufs=2: cold cache 	17s		13s
>
> Out of the box, barriers are 50% slower with WCE=1 than with WCE=0
> on the cold cache test, but are almost as fast with larger
> log buffer size (i.e. less barrier writes being issued).
>
> Worth noting is that at 10-11s runtime, the disk is bandwidth
> bound (i.e. we're doing 30MB/s), so that's the fastest time
> rm -rf will do on this filesystem.
>
> So, clearly we have differing performance depending on
> mount options and at best barriers give equal performance.
>
> I just ran the same tests on an x86_64 box with 7.2krpm 500GB SATA
> disks with WCE (2.6.18 kernel) using a 30GB partition on the outer
> edge:
>
> 					barrier		nobarrier
> logbsize=32k,logbufs=8: hot cache	29s		29s
> logbsize=32k,logbufs=8: cold cache 	33s		30s
> logbsize=256k,logbufs=8: hot cache	8s		8s
> logbsize=256k,logbufs=8: cold cache 	11s		11s
> logbsize=256k,logbufs=2: hot cache	8s		8s
> logbsize=256k,logbufs=2: cold cache 	11s		11s
>
> Barriers make little to zero difference here.
>
>> Can anyone explain this?
>
> Right now I'm unable to reproduce your results even on 2.6.18 so I
> suspect a drive level issue here.
>
> Can I suggest that you try the same tests with write caching turned
> off on the drive(s)? (hdparm -W 0 <dev>, IIRC).

on my laptop I could not set W 0:

sunset:~ # hdparm -W0 /dev/hda

/dev/hda:
  setting drive write-caching to 0 (off)
      HDIO_SET_WCACHE(wcache) failed: Success


on my desktop pc:

WCE=1
                                         barrier                 nobarrier
logbsize=256k,logbufs=4: hot cache      6.3s/6.3s/6.5s           
10.8s/1.9s/2s
logbsize=256k,logbufs=4: cold cache     11.1s/10.9s/10.7         
4.8s/5.8s/7.3s
logbsize=256k,logbufs=4: after reboot   11.9s/10.3s             52.2s/47.2s

WCE=0

logbsize=256k,logbufs=4: hot cache      5.7s/5.6s/5.6s           
8.3s/5.6s/5.6s
logbsize=256k,logbufs=4: cold cache     9.5s/9/9s/9.9s           
9.5s/9.9s/9.8s
logbsize=256k,logbufs=4: after reboot   9.9s                    48.0s


for cold cache I used: echo 3 > /proc/sys/vm/drop_caches
it looks like this machine is only affected after reboot, maybe the hdd  
has more cache, then the hdd in my 3 years old laptop.
on my laptop it was enought to clear the kernel cache.

How did you do your "cold" tests? reboot or drop_caches?

Cheers,
Szabolcs


>
> Cheers,
>
> Dave.

Re: After reboot fs with barrier faster deletes then fs with nobarrier

[David Chinner]

On Fri, Jun 29, 2007 at 01:01:36PM +0100, Szabolcs Illes wrote:
> on my desktop pc:
> 
> WCE=1

hot-cache/cold cache nobarrier is faster. same results as I got.
After reboot, nobarrier is way slow.

> WCE=0

hot-cache/cold-cache is pretty much identical. Same results I got.
After reboot, nobarrier is way slow.

> for cold cache I used: echo 3 > /proc/sys/vm/drop_caches

Same here.

> it looks like this machine is only affected after reboot, maybe the hdd  
> has more cache, then the hdd in my 3 years old laptop.
> on my laptop it was enought to clear the kernel cache.
> 
> How did you do your "cold" tests? reboot or drop_caches?

drop_caches.

So there definitely appears to be something about a reboot that is
causing an issue here. I'll see if I can reproduce it here.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group