File system performance, hardware performance, ext3, 3ware RAID1, etc.

File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Timothy Miller]

I'm attempting to get some idea of how much overhead ext3 causes me on 
my workstation at home.  Furthermore, I'm trying to determine what sort 
of advantage I'm really getting from my 3ware RAID controller (model 
7000-2, configured for RAID1) over single disks.

Using iozone, I'm finding an upper bound for disk reads at about 40 
megs/sec, which is okay, but no better than single disk.  That's 
probably expected, since sequential reads can't come off the disk any 
faster than the disk spins.  RAID1 would show its greatest benefit with 
RANDOM reads.

To determine the highest upper bound for sequential read throughput, I 
timed a dd of the first gigabyte from /dev/sga (the raw device) to 
/dev/null.  This showed a throughput of 47meg/sec.  This shows that ext3 
  isn't hurting reads.

For writes, iozone found an upper bound of about 10megs/sec, which is 
abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
than reads, because once the write is sent, you can forget about it. 
You don't have to wait around for something to come back, and that 
latency for reads can hurt performance.  The OS can also buffer writes 
and reorder them in order to improve efficiency.

The 3ware has this write cache that you can turn on or off.  With it 
off, it ensures that writes make it to the disks in order.  With it on, 
it will reorder writes more efficiently.  However, I noticed that the 
performance only went up to about 16meg/sec with the cache ON.

For comparison, I would like to estimate the maximum WRITE throughout 
for the raw device.  But I'm not ready to dump zeros to my working 
partitions.  I was thinking that I could do this with the SWAP 
partition.  I could turn off swap and then dd TO the swap partition. 
Being on the inner tracks, it won't perform as well as the max for the 
drive, but it'll give me a lower bound for raw write throughput to 
compare against.

IMPORTANT QUESTION:  Is there any metadata anywhere in the swap 
partition (when it's not in use) that I need to save before I fill it 
with zeros?

Also, what do I use as a source for zeros when writing with dd? 
"/dev/zero"?

What's the command?  How about this:
     time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024

Will that do it?  Should I use an offset to avoid any kind of header or 
metadata?

If anyone has numbers for what they get with WD1200JB drives, I'd love 
to compare.

Thanks!

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Willy Tarreau]

On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:
 
> For writes, iozone found an upper bound of about 10megs/sec, which is 
> abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
> than reads, because once the write is sent, you can forget about it. 
> You don't have to wait around for something to come back, and that 
> latency for reads can hurt performance.  The OS can also buffer writes 
> and reorder them in order to improve efficiency.

It depends on the disk too. Lots of disks (specially IDE) are far slower
on writes than they are on reads.

> The 3ware has this write cache that you can turn on or off.  With it 
> off, it ensures that writes make it to the disks in order.  With it on, 
> it will reorder writes more efficiently.  However, I noticed that the 
> performance only went up to about 16meg/sec with the cache ON.

I don't think that the FS type has much importance once the cache is ON.

> IMPORTANT QUESTION:  Is there any metadata anywhere in the swap 
> partition (when it's not in use) that I need to save before I fill it 
> with zeros?

No, there's nothing to save, but you'll have to rebuild the swap at the
end of your tests : mkswap /dev/sga3
I too often use the swap partition for testing purpose or for temporary
storage ; it's easy to do a swapoff; mke2fs on it :-)

> Also, what do I use as a source for zeros when writing with dd? 
> "/dev/zero"?

yes. but don't try to copy any file from the disk onto this partition,
because head seeks will slow things down.

> What's the command?  How about this:
>     time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024

do it like this, but use higher values, particularly for bs which is only
1kB here. Using something like bs=65536 and count=4096 will give you a 256 MB
file.

> Will that do it?  Should I use an offset to avoid any kind of header or 
> metadata?

not needed. Just ensure that you write to the right partition, and better
check twice.

Regards
Willy

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Timothy Miller]

Thanks for your suggestions.  They have aided me in my tests... read on.

Willy Tarreau wrote:
> On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:
>  
> 
>>For writes, iozone found an upper bound of about 10megs/sec, which is 
>>abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
>>than reads, because once the write is sent, you can forget about it. 
>>You don't have to wait around for something to come back, and that 
>>latency for reads can hurt performance.  The OS can also buffer writes 
>>and reorder them in order to improve efficiency.
> 
> 
> It depends on the disk too. Lots of disks (specially IDE) are far slower
> on writes than they are on reads.

I wonder why that is.  What adds extra overhead to the writes?  Verify? 
    Given the lousy quality of today's hard drives, I can see why that 
might be necessary.

> 
> 
>>The 3ware has this write cache that you can turn on or off.  With it 
>>off, it ensures that writes make it to the disks in order.  With it on, 
>>it will reorder writes more efficiently.  However, I noticed that the 
>>performance only went up to about 16meg/sec with the cache ON.
> 
> 
> I don't think that the FS type has much importance once the cache is ON.

It might, but from my tests (see below), you're right.

> 
> 
>>What's the command?  How about this:
>>    time dd if=/dev/zero of=/dev/sga3 bs=1024 count=1024
> 
> 
> do it like this, but use higher values, particularly for bs which is only
> 1kB here. Using something like bs=65536 and count=4096 will give you a 256 MB
> file.
> 

I mistyped.  I did "bs=1024k count=1024".

> 
>>Will that do it?  Should I use an offset to avoid any kind of header or 
>>metadata?
> 
> 
> not needed. Just ensure that you write to the right partition, and better
> check twice.
> 

Done.  Here are my results:

As I had mentioned, I did raw read performance by timing a dd from 
/dev/sda to /dev/null.  That demonstrated a throughput of 47 megs/sec 
which is pretty close to the benchmarks that I find in reviews.

I was able to perform the write performance test last night.  The swap 
partition was actually /dev/sda2.  I think / and /boot are sda1 and 
sda3, but I forget the order.  Either way, the swap partition is not 
worst case.  In my test, I wrote 1 gigabyte in 1 meg blocks in 73.522 
seconds which translates into 13.92 megs/sec.  That's terrible -- worse 
because the 3ware's write cache is ON.

According to Tom's Hardware, raw write throughput for the WD1200JB 
varies from 39500 k/sec at the outer tracks to 14200 k/sec at the inner 
tracks.  Something is seriously bottle-necking the performance through 
the 3ware.

For comparison, my wife's computer has the same model of drive as its 
primary IDE.  This is a single drive, and the box runs Windows XP Pro 
with 1GB of RAM.  The disk was defragmented this morning at 5AM, and I 
ran the test this morning at about 9AM.  I ran SiSoft SANDRA, and these 
are the results:

Buffered Read     73 mb/s
Seqential Read    37 mb/s
Random read        6 mb/s
Buffered write    60 mb/s
Sequential write  39 mb/s
Random write      12 mb/s

Assuming that the "buffered" speeds are being buffered by the OS, we'll 
ignore those.  I am therefore observing that the writes to a single 
drive are 3 to 4 times faster than they are through the RAID controller, 
even with the 3ware write cache ON.

Does that make any sense?


[Kernel version:  Most recent Red Hat update, 2.4.20-something.
  3ware 7000-2 with two 120GB WD drives in RAID 1 array.]

Thanks.

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Willy Tarreau]

On Fri, Feb 13, 2004 at 02:19:59PM -0500, Timothy Miller wrote:
 
> Assuming that the "buffered" speeds are being buffered by the OS, we'll 
> ignore those.  I am therefore observing that the writes to a single 
> drive are 3 to 4 times faster than they are through the RAID controller, 
> even with the 3ware write cache ON.
> 
> Does that make any sense?

Well, it reminds me a disk I had problems with several years ago. It had
a few defects in the FAT area, which were relocated at the end. Performance
was terrible since the head had to move constantly. It took ages to install
Win31 on it, so it finally was returned to the vendor. But in your case it
seems a little bit different since you experience slow writes anywhere on
the medium. Would it be possible that your controller does something like
read-modify-write because of too big chunk size ?

Regards,
Willy

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Timothy Miller]

Willy Tarreau wrote:
> On Fri, Feb 13, 2004 at 02:19:59PM -0500, Timothy Miller wrote:
>  
> 
>>Assuming that the "buffered" speeds are being buffered by the OS, we'll 
>>ignore those.  I am therefore observing that the writes to a single 
>>drive are 3 to 4 times faster than they are through the RAID controller, 
>>even with the 3ware write cache ON.
>>
>>Does that make any sense?
> 
> 
> Well, it reminds me a disk I had problems with several years ago. It had
> a few defects in the FAT area, which were relocated at the end. Performance
> was terrible since the head had to move constantly. It took ages to install
> Win31 on it, so it finally was returned to the vendor. But in your case it
> seems a little bit different since you experience slow writes anywhere on
> the medium. Would it be possible that your controller does something like
> read-modify-write because of too big chunk size ?
> 

I'm getting 10-15 meg/sec even with the largest block sizes.  With dd, I 
  set the block size to 1 megabyte, so there's no chance that the block 
being written is too small or that the disk block is too big.

Also, it's a RAID1.

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Eric D. Mudama]

On Fri, Feb 13 at  6:53, Willy Tarreau wrote:
>It depends on the disk too. Lots of disks (specially IDE) are far slower
>on writes than they are on reads.

This may be a function of the operating system or the filesystem, but
it isn't necessarilly an artifact of the drives themselves.  With both
read and write caching enabled, random writes will always be faster
than random reads from the drive perspective.

Even with queueing enabled (legacy TCQ or Native-SATA "NCQ"), your
ability to reorder reads is limited in ATA to 32 tags, while the
ability to reorder cached writes is limited only by buffer size and
cache granularity...  the absolute worst-case write performance should
be the same as read performance.

--eric


-- 
Eric D. Mudama
edmudama@mail.bounceswoosh.org

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Linus Torvalds]

On Fri, 13 Feb 2004, Eric D. Mudama wrote:
> 
> This may be a function of the operating system or the filesystem, but
> it isn't necessarilly an artifact of the drives themselves.  With both
> read and write caching enabled, random writes will always be faster
> than random reads from the drive perspective.

Well.. Yes and no.

Writes are fundamentally more schedulable, and that's a huge advantage for 
throughput, since latency really doesn't matter. Which means that you'll 
find a lot of loads that can much more easily get to platter speeds for 
writes.

On the other hand, reads are inherently easier on a pure hardware level, 
since read-ahead and track buffers are trivial ways to get to true platter 
speeds for a lot of reasonable loads.

And the software-visible 512-byte blocking factor just has to be
_incredibly_ painful on a hardware level, and I'd be surprised if there
aren't disks out there already where the actual real physical block-size
is bigger. Which means that I would expect a lot of drives to internally
do read-modify-write cycles for small writes.

And especially in a market where density is often more important than pure
speed, I'd expect hw manufacturers to have a _huge_ bias towards big
blocks on the platter, in order to avoid having to have lots of
inter-sector gaps etc.

So in that kind of schenario, random writes would be clearly slower than
random reads.

> the absolute worst-case write performance should be the same as read
> performance.

That is only true if the disk block-size is smaller than the IO blocksize. 
Can somebody fill me in on what modern disks do, especially the 
high-density ones?

			Linus

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[John Bradford]

> And the software-visible 512-byte blocking factor just has to be
> _incredibly_ painful on a hardware level, and I'd be surprised if there
> aren't disks out there already where the actual real physical block-size
> is bigger. Which means that I would expect a lot of drives to internally
> do read-modify-write cycles for small writes.
> 
> And especially in a market where density is often more important than pure
> speed, I'd expect hw manufacturers to have a _huge_ bias towards big
> blocks on the platter, in order to avoid having to have lots of
> inter-sector gaps etc.

Since drives don't generally do defect re-allocation on read, only on
write, large blocks at the hardware level cause difficulties with
defect management, so hardware 512 byte sectors are still common :-(.

John.

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Willy Tarreau]

On Fri, Feb 13, 2004 at 11:55:16AM -0800, Linus Torvalds wrote:
 
> > the absolute worst-case write performance should be the same as read
> > performance.
> 
> That is only true if the disk block-size is smaller than the IO blocksize. 
> Can somebody fill me in on what modern disks do, especially the 
> high-density ones?

This is purely hypothetical, but perhaps write-precompensation or the signal
intensity or shape in the head implies to write at "safer" frequencies ? we're
speaking about hundreds of megahertz, and I've always wondered what the signal
looks like when it reaches the head.

Cheers,
Willy

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Daniel Blueman]

Willy Tarreau <willy@w.ods.org> wrote in message
news:<1oEGw-2ex-1@gated-at.bofh.it>...
> On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:
>  
> > For writes, iozone found an upper bound of about 10megs/sec, which is 
> > abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
> > than reads, because once the write is sent, you can forget about it. 
> > You don't have to wait around for something to come back, and that 
> > latency for reads can hurt performance.  The OS can also buffer writes 
> > and reorder them in order to improve efficiency.
> 
> It depends on the disk too. Lots of disks (specially IDE) are far slower
> on writes than they are on reads.

No. Have you verified this? If you 'dd' your swap partition from /dev/zero
on IDE, you'll see write performance closely matches read performance, for
drives old and new.

In the case of small transfers, the drive can hand them off to the on-drive
write cache (2/8MB usually). The only case where IDE disks will be 'slow' for
write performance is where you have no disk I/O scheduling and lots of small
reads/writes - this case wins on SCSI, but many modern IDE disks and
controllers also have tagged command queuing, so it is even more of a corner case.

Dan

-- 
Daniel J Blueman

GMX ProMail (250 MB Mailbox, 50 FreeSMS, Virenschutz, 2,99 EUR/Monat...)
jetzt 3 Monate GRATIS + 3x DER SPIEGEL +++ http://www.gmx.net/derspiegel +++

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Jamie Lokier]

Daniel Blueman wrote:
> many modern IDE disks and
> controllers also have tagged command queuing, so it is even more of a corner case.

Linux doesn't use tagged command queueing, though - the code has been
disabled for some time.  I thought the TCQ stuff was disabled because
only very few disks supported it and the code wasn't reliable.

Yet you say many modern disks support it?

-- Jamie

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Daniel Blueman]

It is a small loss that the TCQ-support implementation hasn't been stable on
linux so far. I guess mostly due to the great I/O scheduling Linux has (ie
anticipatory scheduler).

Yes, a growing number of IDE disk manufacturers are using native S-ATA
controllers in their disks, and TCQ is one of the core features (ie unique selling
point), as well as being implemented on the more lavish P-ATA controllers.

It's just a shame that there are still issues saturating a bunch of IDE
disks with software RAID in the 2.6 series, but I guess that'll be ironed out in
time...

> Daniel Blueman wrote:
> > many modern IDE disks and
> > controllers also have tagged command queuing, so it is even more of a
> corner case.
> 
> Linux doesn't use tagged command queueing, though - the code has been
> disabled for some time.  I thought the TCQ stuff was disabled because
> only very few disks supported it and the code wasn't reliable.
> 
> Yet you say many modern disks support it?
> 
> -- Jamie
> 

-- 
Daniel J Blueman

GMX ProMail (250 MB Mailbox, 50 FreeSMS, Virenschutz, 2,99 EUR/Monat...)
jetzt 3 Monate GRATIS + 3x DER SPIEGEL +++ http://www.gmx.net/derspiegel +++

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Bartlomiej Zolnierkiewicz]

On Friday 13 of February 2004 15:44, Daniel Blueman wrote:
> It is a small loss that the TCQ-support implementation hasn't been stable
> on linux so far. I guess mostly due to the great I/O scheduling Linux has
> (ie anticipatory scheduler).
>
> Yes, a growing number of IDE disk manufacturers are using native S-ATA
> controllers in their disks, and TCQ is one of the core features (ie unique
> selling point), as well as being implemented on the more lavish P-ATA
> controllers.

Ugh, you are messing PATA drive TCQ, PATA host TCQ and SATA TCQ.

> It's just a shame that there are still issues saturating a bunch of IDE
> disks with software RAID in the 2.6 series, but I guess that'll be ironed
> out in time...
>
> > Daniel Blueman wrote:
> > > many modern IDE disks and
> > > controllers also have tagged command queuing, so it is even more of a
> >
> > corner case.
> >
> > Linux doesn't use tagged command queueing, though - the code has been
> > disabled for some time.  I thought the TCQ stuff was disabled because
> > only very few disks supported it and the code wasn't reliable.

You can re-enable it quite simply 8).

--- Kconfig.orig	2004-02-10 16:29:19.000000000 +0100
+++ Kconfig	2004-02-13 17:06:57.628617096 +0100
@@ -441,7 +441,7 @@
 # TCQ is disabled for now
 config BLK_DEV_IDE_TCQ
 	bool "ATA tagged command queueing (EXPERIMENTAL)"
-	depends on EXPERIMENTAL && n
+	depends on EXPERIMENTAL
 	help
 	  Support for tagged command queueing on ATA disk drives. This enables
 	  the IDE layer to have multiple in-flight requests on hardware that

> > Yet you say many modern disks support it?

No, only IBM/Hitachi and some WD disks support IDE TCQ.

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Timothy Miller]

Daniel Blueman wrote:
> Willy Tarreau <willy@w.ods.org> wrote in message
> news:<1oEGw-2ex-1@gated-at.bofh.it>...
> 
>>On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:
>> 
>>
>>>For writes, iozone found an upper bound of about 10megs/sec, which is 
>>>abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
>>>than reads, because once the write is sent, you can forget about it. 
>>>You don't have to wait around for something to come back, and that 
>>>latency for reads can hurt performance.  The OS can also buffer writes 
>>>and reorder them in order to improve efficiency.
>>
>>It depends on the disk too. Lots of disks (specially IDE) are far slower
>>on writes than they are on reads.
> 
> 
> No. Have you verified this? If you 'dd' your swap partition from /dev/zero
> on IDE, you'll see write performance closely matches read performance, for
> drives old and new.
> 

And this sort of things is what I find with raw writes to the model of 
drive I'm using.  However, it seems that there must be some issue with 
the 3ware 7000-2 which is killing performance, or the way the Linux 
kernel is dealing with this sorta-SCSI device.

The WD1200JB should get like 30-40 megs/sec, but when being accessed 
through the 3ware, I get 10-16 megs/sec.

What could the 3ware be doing?

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Walt H]

On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:

> For writes, iozone found an upper bound of about 10megs/sec, which is 
> abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
> than reads, because once the write is sent, you can forget about it. 
> You don't have to wait around for something to come back, and that 
> latency for reads can hurt performance.  The OS can also buffer writes 
> and reorder them in order to improve efficiency.

I'm joining this thread rather late, so perhaps I missed something. What 
hardware is the test being ran on? I have an AMD MPX based setup which 
suffers from a chipset bug which effectively limits writes to ~25MB/sec to 
devices connected via the 768 southbridge. Maybe something similar with your 
hardware?

-Walt

Re: File system performance, hardware performance, ext3, 3ware RAID1, etc.

[Timothy Miller]

Walt H wrote:
> On Thu, Feb 12, 2004 at 06:32:31PM -0500, Timothy Miller wrote:
> 
> 
>>For writes, iozone found an upper bound of about 10megs/sec, which is 
>>abysmal.  Typically, I'd expect writes to be faster (on a single drive) 
>>than reads, because once the write is sent, you can forget about it. 
>>You don't have to wait around for something to come back, and that 
>>latency for reads can hurt performance.  The OS can also buffer writes 
>>and reorder them in order to improve efficiency.
> 
> 
> I'm joining this thread rather late, so perhaps I missed something. What 
> hardware is the test being ran on? I have an AMD MPX based setup which 
> suffers from a chipset bug which effectively limits writes to ~25MB/sec to 
> devices connected via the 768 southbridge. Maybe something similar with your 
> hardware?

Athlon 2800+
Via KT400 chipset (ABIT KD7, I believe)
WD1200JB (Western Digital 120gb)
3ware 7000-2 as RAID1
Latest Red Hat 2.4.20-?? kernel


My wife's computer has the ABIT board with the KT333 and the same drive, 
but not RAID.  SiSoft SANDRA reports 39 megs/sec for writes to an NTFS 
volume.


Right now, I'm thinking the bottleneck may be the 3ware.