Speed of rm compared to reiserfs (slow)

Speed of rm compared to reiserfs (slow)

[Török Edwin]

Hi,

I am happily using xfs for /var, /usr and /, and I am very pleased with
the read speed.
I've just recommended xfs to a friend, and he complained about the speed
of rm.

I did a test on my box, and indeed the speed of rm is order of magnitude
slower compared to reiserfs.
I already use lazy-count, and noatime/nodiratime. Write barriers are off
because I run on raid10.

Is there anything else I can tune to get faster rm speed?

Test:
1. svn co svn://gcc.gnu.org/svn/gcc/trunk gcc
2.
# sync; echo 3 >/proc/sys/vm/drop_caches; sync
3. time rm -rf gcc

Results for XFS:
real 1m30.100s
user 0m0.188s
sys 0m11.401s

Results for Reiserfs:
real 0m10.937s
user 0m0.140s
sys 0m6.616


# xfs_info /var
meta-data=/dev/mapper/vg--all-lv--var isize=256    agcount=4,
agsize=85196800 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=340787200, imaxpct=5
         =                       sunit=16     swidth=96 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=16 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

# mount | grep var
/dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)

# mount | grep reiserfs
/dev/mapper/vg--all-lv--home on /home type reiserfs (rw,noatime,notail)

# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md3              9.2G  5.1G  3.7G  58% /
tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
udev                   10M  188K  9.9M   2% /dev
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/mapper/vg--all-lv--usr
                      100G  5.3G   95G   6% /usr
/dev/mapper/vg--all-lv--var
                      1.3T  230G  1.1T  18% /var
/dev/mapper/vg--all-lv--home
                      600G   41G  560G   7% /home
tmpfs                 2.0G  1.9M  2.0G   1% /tmp

[   23.861497] Filesystem "dm-0": Disabling barriers, not supported by
the underlying device
[   23.863915] XFS mounting filesystem dm-0
[   24.050223] Ending clean XFS mount for filesystem: dm-0
[   24.056295] Filesystem "dm-1": Disabling barriers, not supported by
the underlying device
[   24.056295] XFS mounting filesystem dm-1
[   24.248025] Ending clean XFS mount for filesystem: dm-1
[   24.335968] ReiserFS: dm-2: found reiserfs format "3.6" with standard
journal
[   24.335968] ReiserFS: dm-2: using ordered data mode
[   24.359968] ReiserFS: dm-2: journal params: device dm-2, size 8192,
journal first block 18,
max trans len 1024, max batch 900, max commit age 30, max trans age 30
[   24.359968] ReiserFS: dm-2: checking transaction log (dm-2)
[   24.426700] ReiserFS: dm-2: Using r5 hash to sort names


Best regards,
--Edwin

Re: Speed of rm compared to reiserfs (slow)

[Dave Chinner]

On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
> Hi,
> 
> I am happily using xfs for /var, /usr and /, and I am very pleased with
> the read speed.
> I've just recommended xfs to a friend, and he complained about the speed
> of rm.
> 
> I did a test on my box, and indeed the speed of rm is order of magnitude
> slower compared to reiserfs.
> I already use lazy-count, and noatime/nodiratime. Write barriers are off
> because I run on raid10.
> 
> Is there anything else I can tune to get faster rm speed?

mount -o logbsize=262144 <dev> <mtpt>

> # mount | grep var
> /dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)

BTW, noatime implies nodiratime - you don't ned to specify both.

> tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
> udev                   10M  188K  9.9M   2% /dev
> tmpfs                 2.0G     0  2.0G   0% /dev/shm
> /dev/mapper/vg--all-lv--usr
>                       100G  5.3G   95G   6% /usr
> /dev/mapper/vg--all-lv--var
>                       1.3T  230G  1.1T  18% /var

At 1.1T, you probably want to use inode64 for /var. The different
allocation strategy of inode32 can be substantially slower than
inode64.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Speed of rm compared to reiserfs (slow)

[Török Edwin]

On 2008-09-25 03:27, Dave Chinner wrote:
> On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
>   
>> Hi,
>>
>> I am happily using xfs for /var, /usr and /, and I am very pleased with
>> the read speed.
>> I've just recommended xfs to a friend, and he complained about the speed
>> of rm.
>>
>> I did a test on my box, and indeed the speed of rm is order of magnitude
>> slower compared to reiserfs.
>> I already use lazy-count, and noatime/nodiratime. Write barriers are off
>> because I run on raid10.
>>
>> Is there anything else I can tune to get faster rm speed?
>>     
>
> mount -o logbsize=262144 <dev> <mtpt>
>   

I've added it to my mount options, also tried logbufs=8 (but that didn't
make much difference).

>   
>> # mount | grep var
>> /dev/mapper/vg--all-lv--var on /var type xfs (rw,noatime,nodiratime)
>>     
>
> BTW, noatime implies nodiratime - you don't ned to specify both.
>
>   
>> tmpfs                 2.0G   12K  2.0G   1% /lib/init/rw
>> udev                   10M  188K  9.9M   2% /dev
>> tmpfs                 2.0G     0  2.0G   0% /dev/shm
>> /dev/mapper/vg--all-lv--usr
>>                       100G  5.3G   95G   6% /usr
>> /dev/mapper/vg--all-lv--var
>>                       1.3T  230G  1.1T  18% /var
>>     
>
> At 1.1T, you probably want to use inode64 for /var. The different
> allocation strategy of inode32 can be substantially slower than
> inode64.
>
>   

Thanks for the suggestions, the time for rm has improved a bit, but is
still slower than reiserfs:

time rm -rf gcc

real    1m18.818s
user    0m0.156s
sys     0m11.777s

Is there anything else I can try to make it faster?

Best regards,
--Edwin

Re: Speed of rm compared to reiserfs (slow)

[gus3]

--- On Thu, 9/25/08, Török Edwin <edwintorok@gmail.com> wrote:

> >> I did a test on my box, and indeed the speed of rm
> is order of magnitude
> >> slower compared to reiserfs.
> >> I already use lazy-count, and noatime/nodiratime.
> Write barriers are off
> >> because I run on raid10.
> >>
> >> Is there anything else I can tune to get faster rm
> speed?

Others' suggestions stand, but I have found the best way to speed up a journalled filesystem (of any kind) is with an external journal on a separate controller channel. If your XFS journal is internal, the RAID10 layer creates a longer data path when actions are recorded in the journal, then read from the journal, and finally committed to the main filesystem.

You may also investigate the speed of different elevator algorithms. XFS does best with "noop" or "deadline" in my experience. Most Linux distributions ship with "cfq" as their default elevator, which can clash badly against XFS.

Re: Speed of rm compared to reiserfs (slow)

[Dave Chinner]

On Thu, Sep 25, 2008 at 11:16:35AM +0300, Török Edwin wrote:
> On 2008-09-25 03:27, Dave Chinner wrote:
> > On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
> Thanks for the suggestions, the time for rm has improved a bit, but is
> still slower than reiserfs:
> 
> time rm -rf gcc
> 
> real    1m18.818s
> user    0m0.156s
> sys     0m11.777s
> 
> Is there anything else I can try to make it faster?

Buy more disks. ;)

XFS is not really optimised for single disk, metadata intensive,
small file workloads. It scales by being able to keep lots of disks
busy at the same time. Those algorithms don't map to single disk
configs as efficiently as a filesystem that was specifically
designed for optimal performance for these workloads (like
reiserfs). We're working on making it better, but that takes time....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: Speed of rm compared to reiserfs (slow)

[Török Edwin]

On 2008-09-26 02:54, Dave Chinner wrote:
> On Thu, Sep 25, 2008 at 11:16:35AM +0300, Török Edwin wrote:
>   
>> On 2008-09-25 03:27, Dave Chinner wrote:
>>     
>>> On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
>>>       
>> Thanks for the suggestions, the time for rm has improved a bit, but is
>> still slower than reiserfs:
>>
>> time rm -rf gcc
>>
>> real    1m18.818s
>> user    0m0.156s
>> sys     0m11.777s
>>
>> Is there anything else I can try to make it faster?
>>     
>
> Buy more disks. ;)
>
> XFS is not really optimised for single disk, metadata intensive,
> small file workloads.

I have 6 disks, in raid10 :)

md4 : active raid10 sda3[0] sdf3[5] sdc3[4] sde3[3] sdd3[2] sdb3[1]
      2159617728 blocks 64K chunks 2 near-copies [6/6] [UUUUUU]

--- Logical volume ---
  LV Name                /dev/vg-all/lv-var
  VG Name                vg-all
  LV UUID                CQHPts-K3OE-9kWV-hg7q-328i-RP0i-Dew94c
  LV Write Access        read/write
  LV Status              available
  # open                 1
  LV Size                1.27 TB
  Current LE             332800
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     256
  Block device           253:1

  --- Segments ---
  Logical extent 0 to 332799:
    Type                linear
    Physical volume     /dev/md4
    Physical extents    25600 to 358399

>  It scales by being able to keep lots of disks
> busy at the same time. Those algorithms don't map to single disk
> configs as efficiently as a filesystem that was specifically
> designed for optimal performance for these workloads (like
> reiserfs). We're working on making it better, but that takes time....

I see.
Well the read performance is very good as I said in my initial email ;)

Thanks,
--Edwin

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

[Török Edwin]

On 2008-09-26 10:41, Török Edwin wrote:
> On 2008-09-26 02:54, Dave Chinner wrote:
>   
>> On Thu, Sep 25, 2008 at 11:16:35AM +0300, Török Edwin wrote:
>>   
>>     
>>> On 2008-09-25 03:27, Dave Chinner wrote:
>>>     
>>>       
>>>> On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
>>>>       
>>>>         
>>> Thanks for the suggestions, the time for rm has improved a bit, but is
>>> still slower than reiserfs:
>>>
>>> time rm -rf gcc
>>>
>>> real    1m18.818s
>>> user    0m0.156s
>>> sys     0m11.777s
>>>
>>> Is there anything else I can try to make it faster?
>>>     
>>>       
>> Buy more disks. ;)
>>
>> XFS is not really optimised for single disk, metadata intensive,
>> small file workloads.
>>     
>
> I have 6 disks, in raid10 :)
>   


I moved the logdevice to another disk (not part of RAID array), and now
the speed of rm is down to 30 seconds!

I didn't find any documentation on how to convert an XFS fs with
internal log device to use external, so I did the following:
# xfs_db -x /dev/mapper/vg--all-lv--opt
xfs_db>sb
xfs_db> write logstart 0
xfs_db> quit

The zero the logdevice, and mount using logdev=. This worked.

However if I now use xfs_repair on this filesystem it restores the
internal log. Is there a way to make this change permanent? (i.e. switch
permanently to using external log?)

Best regards,
--Edwin

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

[Eric Sandeen]

Török Edwin wrote:
> On 2008-09-26 10:41, Török Edwin wrote:
>> On 2008-09-26 02:54, Dave Chinner wrote:
>>   
>>> On Thu, Sep 25, 2008 at 11:16:35AM +0300, Török Edwin wrote:
>>>   
>>>     
>>>> On 2008-09-25 03:27, Dave Chinner wrote:
>>>>     
>>>>       
>>>>> On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
>>>>>       
>>>>>         
>>>> Thanks for the suggestions, the time for rm has improved a bit, but is
>>>> still slower than reiserfs:
>>>>
>>>> time rm -rf gcc
>>>>
>>>> real    1m18.818s
>>>> user    0m0.156s
>>>> sys     0m11.777s
>>>>
>>>> Is there anything else I can try to make it faster?
>>>>     
>>>>       
>>> Buy more disks. ;)
>>>
>>> XFS is not really optimised for single disk, metadata intensive,
>>> small file workloads.
>>>     
>> I have 6 disks, in raid10 :)
>>   
> 
> 
> I moved the logdevice to another disk (not part of RAID array), and now
> the speed of rm is down to 30 seconds!
> 
> I didn't find any documentation on how to convert an XFS fs with
> internal log device to use external, so I did the following:
> # xfs_db -x /dev/mapper/vg--all-lv--opt
> xfs_db>sb
> xfs_db> write logstart 0
> xfs_db> quit
> 
> The zero the logdevice, and mount using logdev=. This worked.
> 
> However if I now use xfs_repair on this filesystem it restores the
> internal log. Is there a way to make this change permanent? (i.e. switch
> permanently to using external log?)

Odds are you need to also set the logstart in the other superblocks:

xfs_db> sb 0
xfs_db> write logstart 0
xfs_db> sb 1
xfs_db> write logstart 0

...

you could script this with xfs_db -c

-Eric


> Best regards,
> --Edwin
> 
> 

Re: Speed of rm compared to reiserfs (slow) - and switching logdevices

[Török Edwin]

On 2008-09-28 21:25, Eric Sandeen wrote:
> Török Edwin wrote:
>   
>> On 2008-09-26 10:41, Török Edwin wrote:
>>     
>>> On 2008-09-26 02:54, Dave Chinner wrote:
>>>   
>>>       
>>>> On Thu, Sep 25, 2008 at 11:16:35AM +0300, Török Edwin wrote:
>>>>   
>>>>     
>>>>         
>>>>> On 2008-09-25 03:27, Dave Chinner wrote:
>>>>>     
>>>>>       
>>>>>           
>>>>>> On Wed, Sep 24, 2008 at 11:43:13AM +0300, Török Edwin wrote:
>>>>>>       
>>>>>>         
>>>>>>             
>>>>> Thanks for the suggestions, the time for rm has improved a bit, but is
>>>>> still slower than reiserfs:
>>>>>
>>>>> time rm -rf gcc
>>>>>
>>>>> real    1m18.818s
>>>>> user    0m0.156s
>>>>> sys     0m11.777s
>>>>>
>>>>> Is there anything else I can try to make it faster?
>>>>>     
>>>>>       
>>>>>           
>>>> Buy more disks. ;)
>>>>
>>>> XFS is not really optimised for single disk, metadata intensive,
>>>> small file workloads.
>>>>     
>>>>         
>>> I have 6 disks, in raid10 :)
>>>   
>>>       
>> I moved the logdevice to another disk (not part of RAID array), and now
>> the speed of rm is down to 30 seconds!
>>
>> I didn't find any documentation on how to convert an XFS fs with
>> internal log device to use external, so I did the following:
>> # xfs_db -x /dev/mapper/vg--all-lv--opt
>> xfs_db>sb
>> xfs_db> write logstart 0
>> xfs_db> quit
>>
>> The zero the logdevice, and mount using logdev=. This worked.
>>
>> However if I now use xfs_repair on this filesystem it restores the
>> internal log. Is there a way to make this change permanent? (i.e. switch
>> permanently to using external log?)
>>     
>
> Odds are you need to also set the logstart in the other superblocks:
>
> xfs_db> sb 0
> xfs_db> write logstart 0
> xfs_db> sb 1
> xfs_db> write logstart 0
>
> ...
>
> you could script this with xfs_db -c
>   

Thank you, xfs_repair now no longer changes the logdevice.

Best regards,
--Edwin