Tiobench results LOWER with more threads

Tiobench results LOWER with more threads

[Vladimir Milovanovic]

OK, just joined the list and rad the faq, and something caught my eye. 
Tiobench results are apparently supposed to INCREASE when there are more 
threads.

What I have is Tiobench results decreasing with more threads. This is my 
setup:

Celeron 633
196 MB PC 133
Adaptec 29160 SCSI controller (PCI)
5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec 
controller
Red Hat 7.3 Linux (2.4.18-3)

Experimenting with different RAID configurations, I have found that I 
can not get more than 32 MB/s from this array with 4 disks, one spare. I 
have actually found out that the disks set the SCSI bus at 40 MB/s 
(since the disks are old) and that in RAID 0 it scales well, the speed 
doubles for two disks, and then the third disk brings in a little more 
performance, and then things topp off at 32 MB/s with four disks. Adding 
the fifth disk gains no extra performance.

Apparently VIA chipsets have problems with PCI bursting, so that is why 
I can't see the full 40 MB/s. That's fine.

But, my tests with tiobench also show that performance decreases as 
extra threads are added. I am testing with a file of 800 MB (approx. 4x 
size of RAM, to get meaningful results) and the decrease with threads, 
while READING only, is consistent in all RAID levels. The write 
performance will sometimes increase, sometimes stay the same.

WHY is this happening? Is it something I have not set up right , or 
what? I am not so much interested in getting more speed out of these old 
disks, they are gonna be replaced soon anyway, but I REALLY want to know 
WHY is this??

Can anyone give me any pointers?
Thanks,
Vlad.

Re: Tiobench results LOWER with more threads

[Jakob Oestergaard]

On Tue, Oct 15, 2002 at 10:20:55AM +0200, Vladimir Milovanovic wrote:
> OK, just joined the list and rad the faq, and something caught my eye. 
> Tiobench results are apparently supposed to INCREASE when there are more 
> threads.

No, what gave you that idea?

It is so much easier for the kernel to handle one sequential stream of
I/O, instead of many streams.

If you have more than one stream, you need to seek. Seeking is bad. One
sequential I/O is almost always (with the notable exception of RAID-1
reads) faster in total sustained throughput, and always (as in really
always) faster in per-thread sustained throughput.

> 
> What I have is Tiobench results decreasing with more threads. This is my 
> setup:

Good, tiobench works  :)

> 
> Celeron 633
> 196 MB PC 133
> Adaptec 29160 SCSI controller (PCI)
> 5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec 
> controller
> Red Hat 7.3 Linux (2.4.18-3)
> 
> Experimenting with different RAID configurations, I have found that I 
> can not get more than 32 MB/s from this array with 4 disks, one spare. I 
> have actually found out that the disks set the SCSI bus at 40 MB/s 
> (since the disks are old) and that in RAID 0 it scales well, the speed 
> doubles for two disks, and then the third disk brings in a little more 
> performance, and then things topp off at 32 MB/s with four disks. Adding 
> the fifth disk gains no extra performance.
> 
> Apparently VIA chipsets have problems with PCI bursting, so that is why 
> I can't see the full 40 MB/s. That's fine.

There's some SCSI overhead as well.  And probably you have some RAM
bandwidth limitation also - although that is probably not very important
at the speed you're seeing.  But it all adds up.

> But, my tests with tiobench also show that performance decreases as 
> extra threads are added. I am testing with a file of 800 MB (approx. 4x 
> size of RAM, to get meaningful results) and the decrease with threads, 
> while READING only, is consistent in all RAID levels. The write 
> performance will sometimes increase, sometimes stay the same.
> 
> WHY is this happening? Is it something I have not set up right , or 
> what? I am not so much interested in getting more speed out of these old 
> disks, they are gonna be replaced soon anyway, but I REALLY want to know 
> WHY is this??

If you used SDRAM instead of disks with actual spindles, you should see
almost the same total sustained I/O going from 1 to some handfull or two
of threads.

But you use real disks. Those have heads that need to move when seeking.
The average seek time for your disks is probably around 7 ms. Let's say
you can do 10 MB/sec sequential reading from a disk (low number
probably), then *one* seek costs you the equivalent of 72 kB (low number
again) of transfer that you do not get while seeking.

The kernel will have to do a lot of seeks to satisfy your multiple
readers. That's many times 72kB.  And that's why you're losing
performance using more readers (or writers).

The more you add, the more you lose  :)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: Tiobench results LOWER with more threads

[Gregory Leblanc]

On Thu, Oct 17, 2002 at 11:50:14PM +0200, Jakob Oestergaard wrote:
> On Tue, Oct 15, 2002 at 10:20:55AM +0200, Vladimir Milovanovic wrote:
> > OK, just joined the list and rad the faq, and something caught my eye. 
> > Tiobench results are apparently supposed to INCREASE when there are more 
> > threads.
> 
> No, what gave you that idea?
> 
> It is so much easier for the kernel to handle one sequential stream of
> I/O, instead of many streams.
> 
> If you have more than one stream, you need to seek. Seeking is bad. One
> sequential I/O is almost always (with the notable exception of RAID-1
> reads) faster in total sustained throughput, and always (as in really
> always) faster in per-thread sustained throughput.

Err, so are you saying that a single sequential I/O is slower on RAID
1 when compared with a single disk?  That doesn't make a lot of
sense.  There -are- instances where parallel I/O is required.  In
these cases, any RAID-1 should be much faster than a single disk, as
should RAID 10.  I'm not sure that RAID 5 should give a similar
benefit, but given the cost of disks, I don't care about RAID 5.

> > Celeron 633
> > 196 MB PC 133
> > Adaptec 29160 SCSI controller (PCI)
> > 5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec 
> > controller
> > Red Hat 7.3 Linux (2.4.18-3)
> > 
> > Experimenting with different RAID configurations, I have found that I 
> > can not get more than 32 MB/s from this array with 4 disks, one spare. I 
> > have actually found out that the disks set the SCSI bus at 40 MB/s 
> > (since the disks are old) and that in RAID 0 it scales well, the speed 
> > doubles for two disks, and then the third disk brings in a little more 
> > performance, and then things topp off at 32 MB/s with four disks. Adding 
> > the fifth disk gains no extra performance.

Interesting.  How fast are you getting single-threaded reads from a
single disk?  And what are the real specs on the drives (RPM, cache,
SCSI version support (SCSI 3 is a family, not a version)).

> > Apparently VIA chipsets have problems with PCI bursting, so that is why 
> > I can't see the full 40 MB/s. That's fine.
> 
> There's some SCSI overhead as well.  And probably you have some RAM
> bandwidth limitation also - although that is probably not very important
> at the speed you're seeing.  But it all adds up.

Yeah, if this is a 40MB/sec bus, you shouldn't expect to get more than
35MB/sec out of the bus.  It'd be slick to get more, but overhead eats
the rest.
    Greg

Re: Tiobench results LOWER with more threads

[Maurice Hilarius]

With regards to your message at 08:31 PM 10/17/02, Gregory Leblanc. Where 
you stated:

>Err, so are you saying that a single sequential I/O is slower on RAID
>1 when compared with a single disk?  That doesn't make a lot of
>sense.  There -are- instances where parallel I/O is required.  In
>these cases, any RAID-1 should be much faster than a single disk, as
>should RAID 10.  I'm not sure that RAID 5 should give a similar
>benefit, but given the cost of disks, I don't care about RAID 5.

Of course RAID1 should be slower.
We are doing more WORK!
We have to generate two writes, and manage them and the RAID.
With a single disk we are doing much less, and there is _much_ less overhead.
In an ideal world RAID10 will gain us back the loss because of the RAID0 
element which DOES gain us some performance back.
But even then the CPU load will be much more, as we are still writing more 
data.


With our best regards,

Maurice W. Hilarius       Telephone: 01-780-456-9771
Hard Data Ltd.               FAX:       01-780-456-9772
11060 - 166 Avenue        mailto:maurice@harddata.com
Edmonton, AB, Canada      http://www.harddata.com/
    T5X 1Y3

Ask me about NAS and near-line storage

Re: Tiobench results LOWER with more threads

[Vladimir Milovanovic]

Gregory Leblanc wrote:

>
>
>Interesting.  How fast are you getting single-threaded reads from a
>single disk?  And what are the real specs on the drives (RPM, cache,
>SCSI version support (SCSI 3 is a family, not a version)).
>
I am getting about 12 MB/s for sequential reads single threaded per 
disk. The disks are 18GB, 7200 rpm, ultra fast/wide scsi disks. I am 
really not sure about the cache.

So what is the consensus now? Let's just stick to RAID 0 - should I see 
more when going multithreaded or should I not be worried that the 
transfer rates are dropping off? Note that the rates drop off even if I 
am only running a 2 disk RAID  0 i.e. my bus is not being maxed out then.

Cheers, Vlad.

Re: Tiobench results LOWER with more threads

[Jakob Oestergaard]

On Thu, Oct 17, 2002 at 07:31:37PM -0700, Gregory Leblanc wrote:
> On Thu, Oct 17, 2002 at 11:50:14PM +0200, Jakob Oestergaard wrote:
> > On Tue, Oct 15, 2002 at 10:20:55AM +0200, Vladimir Milovanovic wrote:
> > > OK, just joined the list and rad the faq, and something caught my eye. 
> > > Tiobench results are apparently supposed to INCREASE when there are more 
> > > threads.
> > 
> > No, what gave you that idea?
> > 
> > It is so much easier for the kernel to handle one sequential stream of
> > I/O, instead of many streams.
> > 
> > If you have more than one stream, you need to seek. Seeking is bad. One
> > sequential I/O is almost always (with the notable exception of RAID-1
> > reads) faster in total sustained throughput, and always (as in really
> > always) faster in per-thread sustained throughput.
> 
> Err, so are you saying that a single sequential I/O is slower on RAID
> 1 when compared with a single disk?  That doesn't make a lot of
> sense.  There -are- instances where parallel I/O is required.  In
> these cases, any RAID-1 should be much faster than a single disk, as
> should RAID 10.  I'm not sure that RAID 5 should give a similar
> benefit, but given the cost of disks, I don't care about RAID 5.

The exception I made was:

"One sequential I/O is almost always (with the notable exception of
RAID-1 reads) faster in total sustained throughput,..."

What I wanted to say was: One sequential reader is usually faster than N
readers are in total.  Except on an N-disk RAID-1, where up to N readers
will be faster in total than one reader.

An N-disk RAID-1 will scale with up to N readers.

There's some fuzz on these measurements, because a sequential read on a
filesystem is not a sequential read on the disk (because of fs
metadata), so therefore often a single threaded read on an N-disk RAID-1
will be faster than on a single disk - and maybe RAID-1 will not scale
up to N readers, but only N-1, but that all depends on the fs - and
let's not go too deep into that for now  :)

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html