xfs for free hosting on linux ? performance questions

xfs for free hosting on linux ? performance questions

[Rekrutacja]

hello, i'm going to create xfs on RAID 5 array consiting of 5 drives, 
each 500GB, it will be software raid made with mdadm on linux. i run 
free hosting, and the array is only for users files (system and such are 
on ext3).

what options do you recommend while creating and while mounting? i had 
performance problems while testing this setup with bonnie++ and postmark 
(postmark especially).

data will be in many directories, users files are in 
/var/www/users/username and i have more than 100 000 accounts. also a 
policy to make max 3 MB files, so most of my files are around 50 KB i think.

so knowing this all, what would you advice me if i may ask? my current 
idea was like this:

mkfs.xfs -l size=128m,lazy-count=1
mount -o nobarrier,noatime,logbufs=8,logbsize=256k

this will have full backup, so i'm more after performance, not 
stability, so if you can give me some performance tips... (postmark run 
on array made from 4 disks, not 5, durning test, with parameters - 
number of files 10k, transactions 10k, subdirs 20k, was giving very very 
slow reads, around 200KB/s, while reiser4/btrfs were like 2MB/s reads, 
but even ext3 was faster - any idea why? i tested on 2.6.24.3 and 
2.6.25-rc4 )

i keep seeing tests like this: http://www.t2-project.org/zine/1/
also, i have real life example, that 3 disks RAID-0 array with tweaked 
xfs is as fast as just one disk with reiser4 (my free hosting with 
millions of files)

i know reiser4 is not stable (i also experienced this), so i'm asking if 
  you know any additional tips to tweak xfs, or maybe you know why it 
has performance problems in some situations.

thanks in advance

Re: xfs for free hosting on linux ? performance questions

[Peter Grandi]

[ ... 

> data will be in many directories, users files are in
> /var/www/users/username and i have more than 100 000
> accounts. also a policy to make max 3 MB files, so most of my
> files are around 50 KB i think.

This is a much better situation than the frequent inane one
where someone asks about using a filesystem as a database for
small records.

But a ''source files'' filesystem is still a bit of challenge.

> [ ... ] mount -o nobarrier,noatime,logbufs=8,logbsize=256k [
> ... ]

Even with backups, 'nobarrier' means begging for trouble.

> [ ... ] number of files 10k, transactions 10k, subdirs 20k,
> was giving very very slow reads, around 200KB/s, while
> reiser4/btrfs were like 2MB/s reads, but even ext3 was faster
> - any idea why? i tested on 2.6.24.3 and 2.6.25-rc4 ) [ ... ] 
> that 3 disks RAID-0 array with tweaked xfs is as fast as just
> one disk with reiser4 (my free hosting with millions of files)
> [ ... ]

Just use the by now very well tested ReiserFS version 3 for
that. Just as you would use XFS for bulk streaming of large
files.

Recently on the same 4x(1+1) RAID10 f2 I did some (single
threaded, single file reading and writing) tests involving
various file systems with both large files (2-12 2GB ones) and a
directory tree contain lots of small Java source files (6.5GB in
50k directories containing 150k files).

While most file systems could reach something like 250MB/s
writing and 430MB/s reading with the small set of large files
(with XFS a few dozen MB/s ahead of most of the others and would
have been more so on multithreaded), on the directory tree
ReiserFS did around 57MB/s, JFS around 36MB/s, and XFS around
20-25MB/s.

There are several reasons why ReiserFS is better suited for
large directory tree with lots of small files; one of them is
that it was designed for that :-).

Re: xfs for free hosting on linux ? performance questions

[Rekrutacja119]

>
> There are several reasons why ReiserFS is better suited for
> large directory tree with lots of small files; one of them is
> that it was designed for that :-).



but reiserfs is slow for me too, maybe i have some other problems? i made
array like this mdadm --create /dev/md0 --verbose --level=5 --raid-devices=5
/dev/sd{b,c,d,e,f}1 --assume-clean

then just mkfs.reiserfs /dev/md0 and then mount /dev/md0 to /array and run
postmark with these:
set numbers 20000
set transactions 10000
set subdirectories 20000
set location /array/

results i got:

Time:
    287 seconds total
    118 seconds of transactions (84 per second)

Files:
    25014 created (87 per second)
        Creation alone: 20000 files (168 per second)
        Mixed with transactions: 5014 files (42 per second)
    5026 read (42 per second)
    4971 appended (42 per second)
    25014 deleted (87 per second)
        Deletion alone: 20028 files (400 per second)
        Mixed with transactions: 4986 files (42 per second)

Data:
    26.48 megabytes read (94.47 kilobytes per second)
    136.18 megabytes written (485.89 kilobytes per second)


is there something wrong with my system?? LA is 0.50 right now (i'm testing
it at night), hdparm is showing that every HD from the array is doing
100MB/s at least, so why these numbers?


[[HTML alternate version deleted]]

Re: xfs for free hosting on linux ? performance questions

[Peter Grandi]

> but reiserfs is slow for me too, maybe i have some other
> problems? i made array like this mdadm --create /dev/md0
> --verbose --level=5 --raid-devices=5 /dev/sd{b,c,d,e,f}1
> --assume-clean

Uhhhhh, '--assume-clean' on a RAID5 is not necessarily a good
idea. Just don't do it, and wait a few hours for the initial
sync to happen.

> [ ... ]
>     26.48 megabytes read (94.47 kilobytes per second)
>     136.18 megabytes written (485.89 kilobytes per second)

> is there something wrong with my system??

The speed above is indeed terrible. Perhaps the array is
rebuilding frantically as it finds all stripes have the wrong
parity because of '--assume-clean'.

> [ ... ] LA is 0.50 right now (i'm testing it at night),

Why is LA relevant? What is running on that system now? Are
those physical disks being used by some applications?

> hdparm is showing that every HD from the array is doing
> 100MB/s at least, so why these numbers?

Perhaps after rebuilding the array without '--assume-clean' you
might try to do such simple tests on the '/dev/md0' device
itself, just to be sure that it performs more or less adequately.

After several reports that is helps, try this before the test:

  blockdev --setra 1024 /dev/md0

but even without it you should be getting at least 40-50MB/s.

And always check the array status with 'mdadm --detail' to see
if it is resyncing or not before testing.

Using 'watch iostat -k sd{b,c,d,e,f} 1 2' (and checking out the
_second_ set of figures) is also nice to see the actual transfer
rates of each drive in the array and verify that if you are
reading there is no writing going on etc.