[parisc-linux] ext2 vs ext3 on hppa

[parisc-linux] ext2 vs ext3 on hppa

[Grant Grundler]

Hi all,
ext3 perf isn't so good on parisc.
I've posted profile data for kernel builds:

http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html

But not for "tar". I can't find the data I collect before. :^(
ISTR the culprit in ext3 was the bitmap search using atomic ops.
On parisc, atomic ops are terrible since it requires a spinlock/op/unlock
sequence.

I was asked about tar perf again and decided to post some results.
If someone tells me which wiki page to add this, I'll pound parts
of the data below into it.

In a nut shell, I timed unpacking the linux source tree:

FS 	real		user		sys
ext2	2m10.814s	0m53.424s	0m6.084s
ext3	3m46.577s	0m54.619s	1m34.553s
xfs	1m22.205s	0m53.952s	0m9.739s
md0+xfs	1m27.740s	0m54.550s	0m12.885s

User time is predominantly bzip2 running.

I didn't realize xfs can schedule work better than ext2 on a single disk.
Adding md striping adds ~6% overhead...that's not so good.
It's possible I need bigger chunks and that would go away.

I don't know if defaults are comparable between xfs and ext3.
Is one implementing more or better features by default than the other?

I'll be looking at mdadm+xfs for future file systems to see
if I can break it. Or I'll have to live without journaling on
"stable" build machines. The ext3 "tar" performance is just
unacceptable.

BTW, I get segfaults on the ext3 G

"raw" data follows.

grant

grundler@riot:~$ uname -a
Linux riot 2.6.11-rc3-pa3 #6 SMP Mon Feb 7 04:25:07 UTC 2005 parisc64 GNU/Linux

grundler@riot:~$ cat /proc/cpuinfo 
processor       : 0
cpu family      : PA-RISC 2.0
cpu             : PA8700 (PCX-W2)
cpu MHz         : 650.000000
model           : 9000/800/A500-6X
...
processor       : 1
cpu family      : PA-RISC 2.0
cpu             : PA8700 (PCX-W2)
cpu MHz         : 650.000000
model           : 9000/800/A500-6X
model name      : Crescendo 650 W2
...


riot:/# lsscsi
[1:0:15:0]   disk    SEAGATE  ST336704LC       HP01  /dev/sda
[4:0:0:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdb
[4:0:1:0]    disk    SEAGATE  ST318451LC       HP01  /dev/sdc
[4:0:2:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdd
[4:0:3:0]    disk    SEAGATE  ST318451LC       HP01  /dev/sde
[4:0:8:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdf
[4:0:9:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdg
[4:0:10:0]   disk    SEAGATE  ST318451LC       HP00  /dev/sdh
[4:0:12:0]   disk    SEAGATE  ST318451LC       HP01  /dev/sdi
[4:0:13:0]   disk    SEAGATE  ST318451LC       HP01  /dev/sdj
[4:0:14:0]   enclosu HP       A5272A           HP04  -       
[5:0:0:0]    disk    SEAGATE  ST318451LC       HP01  /dev/sdk
[5:0:1:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdl
[5:0:2:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdm
[5:0:3:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdn
[5:0:8:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdo
[5:0:9:0]    disk    SEAGATE  ST318451LC       HP00  /dev/sdp
[5:0:10:0]   disk    SEAGATE  ST318451LC       HP00  /dev/sdq
[5:0:11:0]   disk    SEAGATE  ST318451LC       HP00  /dev/sdr
[5:0:12:0]   disk    SEAGATE  ST318451LC       HP01  /dev/sds
[5:0:13:0]   disk    SEAGATE  ST318451LC       HP01  /dev/sdt
[5:0:14:0]   enclosu HP       A5272A           HP05  -       
riot:/# lspci -vt
-+-[30]-+-02.0  LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter
 |      \-02.1  LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter
 +-[20]---00.0  LSI Logic / Symbios Logic 53c1010 66MHz  Ultra3 SCSI Adapter
 +-[10]---00.0  Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet
 \-[00]-+-00.0  Digital Equipment Corporation DECchip 21142/43
        +-01.0  LSI Logic / Symbios Logic 53C896/897
        +-01.1  LSI Logic / Symbios Logic 53C896/897
        +-02.0  LSI Logic / Symbios Logic 53c875
        +-02.1  LSI Logic / Symbios Logic 53c875
        +-04.0  Hewlett-Packard Company: Unknown device 128d
        \-04.1  Hewlett-Packard Company Diva Serial [GSP] Multiport UART
...

# mkfs /dev/sdk
mke2fs 1.35 (28-Feb-2004)
/dev/sdk is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
2223872 inodes, 4445810 blocks
222290 blocks (5.00%) reserved for the super user
First data block=0
136 block groups
32768 blocks per group, 32768 fragments per group
16352 inodes per group
...
riot:/mnt# time tar xjf linux-2.6.11-rc3-pa3.tar.bz2 

real    2m10.814s
user    0m53.424s
sys     0m6.084s
riot:/mnt#
...

riot:/# mke2fs -j /dev/sdk
mke2fs 1.35 (28-Feb-2004)
/dev/sdk is entire device, not just one partition!
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
2223872 inodes, 4445810 blocks
222290 blocks (5.00%) reserved for the super user
First data block=0
136 block groups
32768 blocks per group, 32768 fragments per group
16352 inodes per group
...
riot:/# mount /dev/sdk /mnt
riot:/# cd /mnt
riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2 

real    3m46.577s
user    0m54.619s
sys     1m34.553s
riot:/mnt# 

riot:/# mkfs -t xfs /dev/sdk
meta-data=/dev/sdk               isize=256    agcount=16, agsize=277863 blks
         =                       sectsz=512  
data     =                       bsize=4096   blocks=4445808, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=2560, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0

riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2 

real    1m22.205s
user    0m53.952s
sys     0m9.739s


riot:/# mdadm -C /dev/md0 -l 0 -n 6 /dev/sdb /dev/sdk /dev/sdc /dev/sdl /dev/sdd /dev/sdm
...
riot:/# cat /proc/mdstat 
Personalities : [linear] [raid0] [raid1] 
md0 : active raid0 sdm[5] sdd[4] sdl[3] sdc[2] sdk[1] sdb[0]
      106699008 blocks 64k chunks
...
riot:/# mkfs -t xfs /dev/md0
meta-data=/dev/md0               isize=256    agcount=16, agsize=1667184 blks
         =                       sectsz=512  
data     =                       bsize=4096   blocks=26674752, imaxpct=25
         =                       sunit=16     swidth=96 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal log           bsize=4096   blocks=13024, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=393216 blocks=0, rtextents=0
riot:/# mount /dev/md0 /mnt
riot:/# cd /mnt
riot:/mnt# time tar xjf ~/linux-2.6.11-rc3-pa3.tar.bz2 

real    1m27.740s
user    0m54.550s
sys     0m12.885s

[ after running some kernel builds on the xfs file system ]
grundler@riot:/mnt$ time tar xjf /root/linux-2.6.11-rc3-pa3.tar.bz2 

real    2m35.000s
user    0m55.749s
sys     0m21.332s

_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Corné Beerse]

Grant Grundler wrote:

> Hi all,
> ext3 perf isn't so good on parisc.

Well, From what I know on the linux filesystem types, I can imagine that ext3 is 
not the fastest among them.

ext2 is a (the?) linux normal filesystem: data storage is straigt-forward and such.
ext3 is basically ext2 with added journalling. For reading, it should be just as 
fast as ext2. For writing it is definitly slower than ext2 since it does ext2 
and journalling. As far as I know, the journalling is added to get additional 
data-safety; the filesystem-check is definitly faster/better than for ext2. I 
expect this is at the cost of writing performance.

I don't know xfs, I expect it to be comparable with reiserfs in that it is a 
journalling filesystem, designed from scratch. For these filesystems, I expect 
better performance than ext3 and, depending on several parameters, even better 
performance than ext2.

Between ext2 and xfs/reiser I expect ext2 to be better for large files and the 
others better for lots of small files.

> I've posted profile data for kernel builds:
> 
> http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html
> 
> But not for "tar". I can't find the data I collect before. :^(
> ISTR the culprit in ext3 was the bitmap search using atomic ops.
> On parisc, atomic ops are terrible since it requires a spinlock/op/unlock
> sequence.
> 
> I was asked about tar perf again and decided to post some results.
> If someone tells me which wiki page to add this, I'll pound parts
> of the data below into it.
> 
> In a nut shell, I timed unpacking the linux source tree:
> 
> FS 	real		user		sys
> ext2	2m10.814s	0m53.424s	0m6.084s
> ext3	3m46.577s	0m54.619s	1m34.553s
> xfs	1m22.205s	0m53.952s	0m9.739s
> md0+xfs	1m27.740s	0m54.550s	0m12.885s

This clearly shows, that the filesystems do have some differences in performance 
and all are in system-time, the user-time is comparable.

It would be nice if you can give similar stats for different architectures, 
either within pa-risc or even on other platforms.

> 
> User time is predominantly bzip2 running.

There are several things in comparing filesystems. Some (ext2 I expect) 
filesystems are good in hanldling large files. Some (I expect the journaling 
ones) are good in handling many small files.

With bzip2 running, I expect you use packing or unpacking the kernel-source for 
a test. To get raw figures, better use `tar`, `cp -R` or `dd`. And be sure to 
work from one disk to an other.

> 
> I didn't realize xfs can schedule work better than ext2 on a single disk.
> Adding md striping adds ~6% overhead...that's not so good.
> It's possible I need bigger chunks and that would go away.

I don't know what you mean by mdstripping, If it is something like raid-0 or 
raid-1 (or the combination) then striping is only faster on reading, not on writing.

> 
> I don't know if defaults are comparable between xfs and ext3.
> Is one implementing more or better features by default than the other?

As above: ext3 is ext2 with added journalling. xfs is designed from the ground 
up. I'm surprised to see that ext3 takes 15 times more system-time!

> 
> I'll be looking at mdadm+xfs for future file systems to see
> if I can break it. Or I'll have to live without journaling on
> "stable" build machines. The ext3 "tar" performance is just
> unacceptable.

You'd definitly also peek at reiserfs, Its the default fs used in SuSE linux.

If you intend to re-do your tests, think about the next: To test read speed, `cp 
-r` to /dev/null. To test write speed of small files, tar from the fastest 
read-system. (for large files, just copy from /dev/zero).

> 
> BTW, I get segfaults on the ext3 G

Thats a bad sign...



my 2 cents

CBee
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Grant Grundler]

On Fri, Feb 11, 2005 at 02:08:12PM +0100, Corn? Beerse wrote:
...
> >In a nut shell, I timed unpacking the linux source tree:
> >
> >FS 	real		user		sys
> >ext2	2m10.814s	0m53.424s	0m6.084s
> >ext3	3m46.577s	0m54.619s	1m34.553s
> >xfs	1m22.205s	0m53.952s	0m9.739s
> >md0+xfs	1m27.740s	0m54.550s	0m12.885s
...
> It would be nice if you can give similar stats for different architectures, 
> either within pa-risc or even on other platforms.

Sorry, who ever is interested in that comparison can provide that data. :^)

> I don't know what you mean by mdstripping, If it is something like raid-0 
> or raid-1 (or the combination) then striping is only faster on reading, not 
> on writing.

Yes, I meant raid0.

> You'd definitly also peek at reiserfs, Its the default fs used in SuSE 
> linux.

That sounds like an excercise for someone else.

> If you intend to re-do your tests, think about the next: To test read 
> speed, `cp -r` to /dev/null. To test write speed of small files, tar from 
> the fastest read-system. (for large files, just copy from /dev/zero).

ok. My guess was with 4GB of RAM, I was mostly operating from buffer cache.
Maybe next time I'll use ram disks instead.

> >BTW, I get segfaults on the ext3.
> 
> Thats a bad sign...

My bad. I was running two instances of "make" in the same source tree.
I had forgotten about the "while :" loop that was doing builds.
/me hides

thanks,
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Joel Soete]

[...]
> 
> My bad. I was running two instances of "make" in the same source tree.
> I had forgotten about the "while :" loop that was doing builds.
> /me hides
> 
mmm when I did (sorry not any more able to do it :_( ) similar test it seems that segfault to only occured with option -j2 (or 
greater) and only on smp system (on such b2k the same test _never_ segfault also on ext3)

hth,
	Joel
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Matthew Wilcox]

On Fri, Feb 11, 2005 at 12:16:29AM -0700, Grant Grundler wrote:
> Hi all,
> ext3 perf isn't so good on parisc.
> I've posted profile data for kernel builds:
> 
> http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025196.html
> 
> But not for "tar". I can't find the data I collect before. :^(
> ISTR the culprit in ext3 was the bitmap search using atomic ops.
> On parisc, atomic ops are terrible since it requires a spinlock/op/unlock
> sequence.

I switched the ext2/3 bitops to use the non-atomic ops.  Tried it on nicol
(dual 550MHz PA8600) ...

$ time tar xjf ~/linux-2.6.11-rc3.tar.bz2 

real    25m4.557s
user    1m7.262s
sys     24m59.092s

ouch.  time to do some profiling, i guess, instead of assuming I saw
the problem and the fix ...

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Grant Grundler]

On Fri, Feb 11, 2005 at 12:16:29AM -0700, Grant Grundler wrote:
> But not for "tar". I can't find the data I collect before. :^(

I found them:
-rw-r--r--  1 root root  33517 Jan  9 21:27 prof-2.6.10-pa6-01
-rw-r--r--  1 root root  23770 Jan 16 19:46 prof-2.6.10-pa6-02

and posted both on
	http://www.parisc-linux.org/~grundler/prof-a500-65/

> ISTR the culprit in ext3 was the bitmap search using atomic ops.
> On parisc, atomic ops are terrible since it requires a spinlock/op/unlock
> sequence.

Yeah, this seems to be right.
I gather this is only an issue for SMP kernels.

grundler <506>sort -rnk 3 prof-2.6.10-pa6-01
 55063 _spin_lock                               1376.5750
 42958 ext3_test_allocatable                     97.6318
  5974 rb_next                                   57.4423
  5541 flush_kernel_icache_page                  55.4100
   800 fdsync                                    40.0000
  1279 flush_user_icache_range_asm               35.5278
 35341 bitmap_search_next_usable_block           30.6780
  1328 _spin_lock_irqsave                        27.6667
  1470 machine_restart                           22.9688
   747 flush_user_dcache_range_asm               20.7500
  1043 cpu_idle                                  14.4861
   135 fisync                                     8.4375
   647 __clear_user_page_asm                      5.7768
...


Sorry - I didn't note exactly which workload the above represented.

And I suspect -02 data is with willy's vma_prev patch applied
that he proposed here:
http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.html

grundler <509>sort -rnk 3 prof-2.6.10-pa6-02
 61105 _spin_lock                               1527.6250
 43040 ext3_test_allocatable                     97.8182
  4902 rb_next                                   47.1346
 35646 bitmap_search_next_usable_block           30.9427
  2217 flush_kernel_icache_page                  22.1700
   969 _spin_lock_irqsave                        20.1875
  1055 machine_restart                           16.4844
   764 cpu_idle                                  10.6111
   157 fdsync                                     7.8500
   270 flush_user_icache_range_asm                7.5000
   209 flush_user_dcache_range_asm                5.8056
...

hth, 
grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Matthew Wilcox]

On Fri, Feb 11, 2005 at 09:53:22AM -0700, Grant Grundler wrote:
> And I suspect -02 data is with willy's vma_prev patch applied
> that he proposed here:
> http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.html

Probably not -- it crashes the machine ;-)

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux

Re: [parisc-linux] ext2 vs ext3 on hppa

[Grant Grundler]

On Fri, Feb 11, 2005 at 04:54:15PM +0000, Matthew Wilcox wrote:
> On Fri, Feb 11, 2005 at 09:53:22AM -0700, Grant Grundler wrote:
> > And I suspect -02 data is with willy's vma_prev patch applied
> > that he proposed here:
> > http://lists.parisc-linux.org/pipermail/parisc-linux/2004-November/025210.html
> 
> Probably not -- it crashes the machine ;-)

Ok :^(

We called flush_kernel_icache_page half as many times (5541 vs 2217).
And flush_user_icache_range_asm 1/5th a often (1279 vs 270).
Both are SMP kernels.

I have no idea what else might result in such a major difference.

grant
_______________________________________________
parisc-linux mailing list
parisc-linux@lists.parisc-linux.org
http://lists.parisc-linux.org/mailman/listinfo/parisc-linux