XFS for lots of small files

XFS for lots of small files

[Leszek Dubiel]

Hello!

I consider moving server from reiserfs to xfs. In all benchmarks I have 
read both file systems have had comparable results.

But I've made a test:

1. formated /dev/hda2 with reiserfs with default options and made 10.000 
files
2. formated /dev/hda2 with xfs with default options and made 10.000

Reiserfs created those files in 2 (two) seconds, and xfs created them in 
35 (thirty five) seconds.

Is that normal? What I am doing wrong?

My system is Debian, current stable version. Below is a log of operation.


Thanks in advance.



debian:/mnt/hdc2# mkfs.xfs /dev/hdc2 -f; mount /dev/hdc2 /mnt/hdc2; cd /mnt/hdc2; mount | egrep hdc2
meta-data=/dev/hdc2              isize=256    agcount=16, agsize=1220688 blks
         =                       sectsz=512   attr=0
data     =                       bsize=4096   blocks=19531008, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=9536, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0
/dev/hdc2 on /mnt/hdc2 type xfs (rw)



debian:/mnt/hdc2# time for f in `seq 9999`; do echo $f > $f; done

real	0m35.558s
user	0m0.256s
sys	0m1.080s

debian:/mnt/hdc2# time cat * | wc -l 
9999

real	0m0.239s
user	0m0.020s
sys	0m0.172s



debian:/mnt/hdc2# cd /; umount /dev/hdc2; mkreiserfs /dev/hdc2 -f; mount /dev/hdc2 /mnt/hdc2; cd /mnt/hdc2; mount | egrep hdc2
mkreiserfs 3.6.19 (2003 www.namesys.com)
[...]
ReiserFS is successfully created on /dev/hdc2.
/dev/hdc2 on /mnt/hdc2 type reiserfs (rw)



debian:/mnt/hdc2# time for f in `seq 9999`; do echo $f > $f; done

real	0m1.075s
user	0m0.232s
sys	0m0.844s

debian:/mnt/hdc2# time cat * | wc -l
9999

real	0m0.257s
user	0m0.028s
sys	0m0.256s



debian:/mnt/hdc2# exit

Re: XFS for lots of small files

[Nicolas KOWALSKI]

On Tue, 6 May 2008, Leszek Dubiel wrote:

> Hello!

Hello,

> Is that normal? What I am doing wrong?

For metadata intensive work, you should consider a little bit of tuning. 

See this usefull advice:
http://thread.gmane.org/gmane.comp.file-systems.xfs.general/21239/focus=21240

-- 
Nicolas

Re: XFS for lots of small files

[Martin Steigerwald]

Am Dienstag 06 Mai 2008 schrieb Leszek Dubiel:
> Hello!

Hi Leszek,

> I consider moving server from reiserfs to xfs. In all benchmarks I have
> read both file systems have had comparable results.
>
> But I've made a test:
>
> 1. formated /dev/hda2 with reiserfs with default options and made
> 10.000 files
> 2. formated /dev/hda2 with xfs with default options and made 10.000
>
> Reiserfs created those files in 2 (two) seconds, and xfs created them
> in 35 (thirty five) seconds.
>
> Is that normal? What I am doing wrong?
>
> My system is Debian, current stable version. Below is a log of
> operation.
>
>
> Thanks in advance.

[...]

> debian:/mnt/hdc2# time for f in `seq 9999`; do echo $f > $f; done
>
> real	0m35.558s
> user	0m0.256s
> sys	0m1.080s
>
> debian:/mnt/hdc2# time cat * | wc -l
> 9999
>
> real	0m0.239s
> user	0m0.020s
> sys	0m0.172s

I get

martin@shambala:~/Zeit/filetest -> rm *; sync ; time for ((I=1; I<=10000; 
I=I+1)); do echo $I > $I; done

real    0m10.642s
user    0m0.907s
sys     0m1.713s
martin@shambala:~/Zeit/filetest -> sync ; time cat * >/dev/null

real    0m0.238s
user    0m0.087s
sys     0m0.153s
martin@shambala:~/Zeit/filetest -> sync ; time cat * | wc -l
10000

real    0m0.375s
user    0m0.120s
sys     0m0.247s
martin@shambala:~/Zeit/filetest -> sync ; time rm *

real    0m7.600s
user    0m0.113s
sys     0m1.377s


for XFS with optimized settings...

shambala> xfs_info /home
meta-data=/dev/sda5              isize=256    agcount=6, agsize=4883256 
blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=29299536, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096
log      =internal               bsize=4096   blocks=32768, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

agcount is two more than would be optimal cause I growed the partition 
once.

shambala> mount | grep home
/dev/sda5 on /home type xfs (rw,relatime,logbsize=256k,logbufs=8)

This is on a ThinkPad T42 internal laptop 160 GB harddisk drive with I 
think 5400rpm.

Partition I tested on was not empty at that time and is heavily used.

shambala> LANG=EN df -h /home
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5             112G   84G   29G  75% /home

And there is quite some fragmentation on it:

xfs_db> frag
actual 653519, ideal 587066, fragmentation factor 10.17%

I do not have free space in my playground LVM to test against ext3 and 
reiserfs at the moment.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS for lots of small files

[Linda Walsh]

Martin Steigerwald wrote:
> And there is quite some fragmentation on it:
> 
> xfs_db> frag
> actual 653519, ideal 587066, fragmentation factor 10.17%
----
	Not to disturb you, but are  you running xfs_fsr on a
regular basis?

I have a shell script that runs in my 'cron.daily' that has two
lines:
----
#!/bin/bash
ionice -c 3 /usr/sbin/xfs_fsr &
-------

I've never noticed fragmentation above 1% on my disks...but your
mileage may vary ....

Re: XFS for lots of small files

[Martin Steigerwald]

Hi Linda,

Am Freitag 23 Mai 2008 schrieb Linda Walsh:
> Martin Steigerwald wrote:
> > And there is quite some fragmentation on it:
> >
> > xfs_db> frag
> > actual 653519, ideal 587066, fragmentation factor 10.17%
>
> ----
> 	Not to disturb you, but are  you running xfs_fsr on a
> regular basis?

No, not yet. I had those bad experiences with xfs_fsr on a ck patchset 
patched kernel and after this I didn't try again. Its pretty sure that my 
file content corruption issues was related to the ck patchset as I was 
not able to reproduce it with a CFS or mainline kernel. But still I have 
been reluctant. I think, I give it a try after I made my backups.

> I have a shell script that runs in my 'cron.daily' that has two
> lines:
> ----
> #!/bin/bash
> ionice -c 3 /usr/sbin/xfs_fsr &
> -------
>
> I've never noticed fragmentation above 1% on my disks...but your
> mileage may vary ....

Lets see ;-)

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS for lots of small files

[Eric Sandeen]

Martin Steigerwald wrote:

> And there is quite some fragmentation on it:
> 
> xfs_db> frag
> actual 653519, ideal 587066, fragmentation factor 10.17%


No, there's not.

You have 653519 extents out of an "ideal" 587066.

That is 653519/587066 = 1.113 extents per file.

It is not "quite some" fragmentation, it is near perfect (although this
is subjective, and also depends on the size of your files... if they are
all 8k then 1.113 extents per file might be a bit high; if they average
1G then 1.113 extents on average is pretty darned good.)

-Eric

Re: XFS for lots of small files

[Martin Steigerwald]

Am Sonntag 25 Mai 2008 schrieb Eric Sandeen:
> Martin Steigerwald wrote:
> > And there is quite some fragmentation on it:
> >
> > xfs_db> frag
> > actual 653519, ideal 587066, fragmentation factor 10.17%
>
> No, there's not.

OK, so there is or better was (see below) *some* fragmentation.

> You have 653519 extents out of an "ideal" 587066.
>
> That is 653519/587066 = 1.113 extents per file.
>
> It is not "quite some" fragmentation, it is near perfect (although this
> is subjective, and also depends on the size of your files... if they
> are all 8k then 1.113 extents per file might be a bit high; if they
> average 1G then 1.113 extents on average is pretty darned good.)

They vary a lot. From KMail ~/Mail directory with hundred of thousands of 
mails in maildir format to a picture and movie collection from various 
digicams with 150KB over 2-4 MB to 50-200 MB in size and a music 
collection and kernel sources and and and... would need to run a tool on 
them to gather some statistic.

Anyway, nothing that can't be optimized:

shambala> xfs_db -r /dev/sda5
xfs_db> frag
actual 683648, ideal 617593, fragmentation factor 9.66%
xfs_db> quit

shambala> xfs_fsr /dev/sda5
/home start inode=0

shambala> xfs_db -r /dev/sda5
xfs_db> frag
actual 620316, ideal 617584, fragmentation factor 0.44%
xfs_db> quit

xfs_fsr copied over several gigabytes and the free space of the partition 
temporarily more than once was 4 GB less than the 20 GB of free space it 
had before and after invoking xfs_fsr ;)

Not that I noticed a difference up to now however.

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS for lots of small files

[Eric Sandeen]

Martin Steigerwald wrote:
> Am Sonntag 25 Mai 2008 schrieb Eric Sandeen:
>> Martin Steigerwald wrote:
>>> And there is quite some fragmentation on it:
>>>
>>> xfs_db> frag
>>> actual 653519, ideal 587066, fragmentation factor 10.17%
>> No, there's not.
> 
> OK, so there is or better was (see below) *some* fragmentation.
> 
>> You have 653519 extents out of an "ideal" 587066.
>>
>> That is 653519/587066 = 1.113 extents per file.
>>
>> It is not "quite some" fragmentation, it is near perfect (although this
>> is subjective, and also depends on the size of your files... if they
>> are all 8k then 1.113 extents per file might be a bit high; if they
>> average 1G then 1.113 extents on average is pretty darned good.)
> 
> They vary a lot. From KMail ~/Mail directory with hundred of thousands of 
> mails in maildir format to a picture and movie collection from various 
> digicams with 150KB over 2-4 MB to 50-200 MB in size and a music 
> collection and kernel sources and and and... would need to run a tool on 
> them to gather some statistic.
> 
> Anyway, nothing that can't be optimized:

Sure... I'd just argue that it's diminishing returns :)

> shambala> xfs_db -r /dev/sda5
> xfs_db> frag
> actual 683648, ideal 617593, fragmentation factor 9.66%
> xfs_db> quit
> 
> shambala> xfs_fsr /dev/sda5
> /home start inode=0
> 
> shambala> xfs_db -r /dev/sda5
> xfs_db> frag
> actual 620316, ideal 617584, fragmentation factor 0.44%
> xfs_db> quit
> 
> xfs_fsr copied over several gigabytes and the free space of the partition 
> temporarily more than once was 4 GB less than the 20 GB of free space it 
> had before and after invoking xfs_fsr ;)

fsr needs to preallocate space to "defragment into" so this is expected,
temporarily.

> Not that I noticed a difference up to now however.

right, my original reply was meant to imply that fragmentation is not
really a problem for you.  :)

And in the larger picture, I just wanted to point out that the
"fragmentation factor" can be pretty misleading.  It reports as (actual
- ideal) / actual.

Imagine a filesystem full of 8GB dvd iso images, each with 4 2GB
extents.  The fragmentation factor would be reported as (4X - 1X) / 4X =
75%.  Which looks "bad," but really isn't.

-Eric