XFS mkfs/mount options (w/ better results this time)

XFS mkfs/mount options (w/ better results this time)

[Mark]

I have found that Dbench highlights the differences of various filesystems' throughput, and the results are surprising. (Summary: XFS is the clear winner, under pretty much all circumstances.)

The basic test:

# dbench -D /tester -t 60 {threads}

The primary variable is the filesystem (ext3, XFS, and JFS), with CPU speed and disk elevator being secondary variables. The journal partition is on the first SATA controller, and the main data partition is on the second SATA controller, to exploit SMP capabilities. I'm running on an AMD64-X2 2.5GHz with 2G RAM.

My kernel is custom-compiled for 1000Hz task scheduler and forced kernel preemption, including preempting the Big Kernel Lock. Per Martin Steigerwald's earlier advice, I also ran irqbalance (but more on this later). All tests were run in runlevel 1, which on Slackware means 6 VT's but no services (net, mouse, etc.).

Aside: 60 seconds may not be a long time to run a disk benchmark, but I found that after about 15 seconds, the disk throughput had leveled off dramatically on my system. In the context of the final numbers, the actual intra-test variation became insignificant. I have kept all test output for review.

I created and mounted filesystems with the following options:

ext3: no options to mke2fs, so sparse_super,filetype,resize_inode,dir_index,ext_attr were all enabled
mounted with "data=writeback" and noatime

JFS: mounted with "noatime"

XFS: mkfs.xfs -l size=64m,logdev=/dev/sda2,lazy-count=1 -d agcount=8 -f /dev/sdb5
mount -o noatime,logdev=/dev/sda2,logbufs=8

I ran a round of tests using 5 threads, to resemble 1 runnable and 1 waiting on each CPU, plus 1 more waiting. In other words, lightly overloaded. XFS was the clear winner, with 378 MB/sec using the "noop" scheduler. The "deadline" scheduler was a close second, with 371 MB/sec.

Here was the first twist: The completely fair queueing (CFQ) scheduler seriously impeded XFS performance, so badly that even "noop" out-performed it when the CPU was running at 40% clock.

After I un-mounted the XFS volume, I noticed I was not actually running irqbalance. Oops! It had been running under runlevel 4, but exited when I ran "telinit 1". The IRQs were left in the last "balanced" state, but there was no dynamic re-adjustment for the different conditions. If irqbalance could help the other two filesystems reach 378 MB/sec throughput, why not? So I launched it and continued.

Running the same test on JFS gave a comparatively poor result, with 147 MB/sec using the "deadline" scheduler. ext3 was somewhat better, with 279 MB/sec also with "deadline." They didn't get nearly enough help from irqbalance to make a difference in rank.

I re-ran all tests with 20 threads, to simulate severe process I/O overloading. Even on my 2-CPU system, XFS scaled somewhat, achieving 403 MB/sec with "deadline" and 401 MB/sec with "anticipatory." CFQ didn't hurt the throughput as much this time, but it still came in last (263 MB/sec).

JFS suffered horribly under 20 threads. All throughput fell between 41 and 46 MB/sec, no matter the elevator or the CPU speed.

ext3 fared somewhat better, with 256 MB/sec on the CFQ scheduler. Finally, a filesystem that CFQ can help. But it's still not as good as XFS on "deadline."

For a final twist, I re-ran the 5-thread tests, but adding "sync" to the mount options. I hand-tuned the elevator according to the best for each filesystem. I ran dbench once, to warm disk buffers, then re-ran and captured the results. Once more, XFS is the hands-down winner:

XFS+noop: 65 MB/sec
JFS+deadline: 40 MB/sec
ext3+CFQ: 37 MB/sec

I am very impressed.

If you want to see the test script and results for yourself, I can put a tarball on my website and send you the link. My testing may not be perfect, but I find these numbers very difficult to explain by test shortcomings.

-- 
Mark

"What better place to find oneself than
 on the streets of one's home village?"
      --Capt. Jean-Luc Picard, "Family"

Re: XFS mkfs/mount options (w/ better results this time)

[Martin Steigerwald]

Am Montag 23 Juni 2008 schrieb Mark:

> I ran a round of tests using 5 threads, to resemble 1 runnable and 1
> waiting on each CPU, plus 1 more waiting. In other words, lightly
> overloaded. XFS was the clear winner, with 378 MB/sec using the "noop"
> scheduler. The "deadline" scheduler was a close second, with 371
> MB/sec.
>
> Here was the first twist: The completely fair queueing (CFQ) scheduler
> seriously impeded XFS performance, so badly that even "noop"
> out-performed it when the CPU was running at 40% clock.


[...]

> I re-ran all tests with 20 threads, to simulate severe process I/O
> overloading. Even on my 2-CPU system, XFS scaled somewhat, achieving
> 403 MB/sec with "deadline" and 401 MB/sec with "anticipatory." CFQ
> didn't hurt the throughput as much this time, but it still came in last
> (263 MB/sec).

Thats interesting. I was curious and thus switched from cfq to deadline 
scheduler during parallel I/O workload on my ThinkPad T42 (aptitude 
upgrade / kmail receiving mails from POP3 account).

It subjectively feeled way faster with deadline. I always wondered about 
the slowness of my ThinkPad T42 at massive parallel I/O. Now it feels a 
lot more responsive. Its as if I bought a new super-seek harddisk or what 
(compared to before).

I think I will try deadline for some days at least, also on my ThinkPad 
T23 and on my workstation at work.

No objective performance measurements yet. ;) And not much time for them 
either.

Have I/O schedulers been tested against different filesystems before? 
Maybe the default I/O scheduler cfq isn't the best one for XFS, but only 
for ext3?

Ciao,
-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7

Re: XFS mkfs/mount options (w/ better results this time)

[Martin Steigerwald]

Am Montag 23 Juni 2008 schrieb Martin Steigerwald:
> Am Montag 23 Juni 2008 schrieb Mark:
> > I ran a round of tests using 5 threads, to resemble 1 runnable and 1
> > waiting on each CPU, plus 1 more waiting. In other words, lightly
> > overloaded. XFS was the clear winner, with 378 MB/sec using the
> > "noop" scheduler. The "deadline" scheduler was a close second, with
> > 371 MB/sec.
> >
> > Here was the first twist: The completely fair queueing (CFQ)
> > scheduler seriously impeded XFS performance, so badly that even
> > "noop" out-performed it when the CPU was running at 40% clock.
>
> [...]
>
> > I re-ran all tests with 20 threads, to simulate severe process I/O
> > overloading. Even on my 2-CPU system, XFS scaled somewhat, achieving
> > 403 MB/sec with "deadline" and 401 MB/sec with "anticipatory." CFQ
> > didn't hurt the throughput as much this time, but it still came in
> > last (263 MB/sec).
>
> Thats interesting. I was curious and thus switched from cfq to deadline
> scheduler during parallel I/O workload on my ThinkPad T42 (aptitude
> upgrade / kmail receiving mails from POP3 account).
>
> It subjectively feeled way faster with deadline. I always wondered
> about the slowness of my ThinkPad T42 at massive parallel I/O. Now it
> feels a lot more responsive. Its as if I bought a new super-seek
> harddisk or what (compared to before).

It feels like I have a completely different system. Not only on massive 
parallel I/O. Starting OpenOffice... starting KDE apps... deadline seems 
to outperform cfq regarding subjectively perceived desktop performance to 
no end. That difference is absolutely astonishing for me. My ThinkPad 
*flies* compared to before.

-- 
Martin 'Helios' Steigerwald - http://www.Lichtvoll.de
GPG: 03B0 0D6C 0040 0710 4AFA  B82F 991B EAAC A599 84C7