new bottleneck section in wiki

new bottleneck section in wiki

[Keld Jørn Simonsen]

I should have done something else this afternoon, but anyway, I was
inspired to write up this text for the wiki. Comments welcome.

Keld

Bottlenecks

There can be a number of bottlenecks other than the disk subsystem that
hinders you in getting full performance out of your disks.

One is the PCI bus. Older PCI bus has a 33 MHz cycle and a 32 bit width,
giving a maximum bandwidth of about 1 Gbit/s, or 133 MB/s. This will
easily cause trouble with newer SATA disks which easily gives 70-90 MB/s
each. So do not put your SATA controllers on a 33 MHz PCI bus.

The 66 MHz 64-bit PCI bus is capable of handling about 4 Gbit/s, or
about 500 MB/s. This can also be a bottleneck with bigger arrays, eg a 6
drive array will be able to deliver about 500 MB/s, and maybe you want
also to feed a gigabyte ethernet card - 125 MB/s, totalling potentially
625 MB/s on the PCI bus.

The PCI-Express bus v1.1 has a limit of 250 MB/s per lane per dirction,
and that limit can easily be hit eg by a 4-drive array.

Many SATA controllers are on-board and do not use the PCI bus. Anyway
bandwidth is limited, but it is probably different from motherboard to
motherboard. On board disk controllers most likely have a bigger
bandwidth than IO controllers on a 32-bit PCI 33 MHz, 64-bit PCI 66 MHz,
or PCI-E x1 bus.

Having a RAID connected over the LAN can be a bottleneck, if the LAN
speed is only 1 Gbit/s - this limits the speed of the IO system to 125
MB/s by itself.

Classical bottlenecks are PATA drives placed on the same DMA channel, or
the same PATA cable. This will of cause limit performance, but it should
work, given you have no other means of connecting your disks by. Also
placing more than one element of an array on the same disk hurts
performace seriously, and also gives redundancy problems.

A classical problem is also not to have enabled DMA transfer, or having
lost this setting due to some problem, including not well connected
cables, or setting the transfer speed to less than optimal.

RAM sppec may be a bottleneck. Using 32 bit RAM - or using a 32 bit
operating system may double time spent reading and writing RAM.

CPU usage may be a bottleneck, also combined with slow RAM or only using
RAM in 32-bit mode.

BIOS settings may also impede your performance. 

Re: new bottleneck section in wiki

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1345 bytes --]



On Wed, 2 Jul 2008, Keld Jørn Simonsen wrote:

> The 66 MHz 64-bit PCI bus is capable of handling about 4 Gbit/s, or
> about 500 MB/s. This can also be a bottleneck with bigger arrays, eg a 6
> drive array will be able to deliver about 500 MB/s, and maybe you want
> also to feed a gigabyte ethernet card - 125 MB/s, totalling potentially
> 625 MB/s on the PCI bus.
>
> The PCI-Express bus v1.1 has a limit of 250 MB/s per lane per dirction,
> and that limit can easily be hit eg by a 4-drive array.
Correction, can easily be hit with two veliciraptors.
One veliciraptor for much of the read/write throughout the entire disk
is 120MiB/s and it slows down toward the middle obviously but FYI.
So two disks saturate it, I get 120MiB/s on a PCI-e x1 card but when I run two
disks on it at the same time (saturing them I only get 80-90MiB/s) on each of
them for an aggregate bandwidth of ~160MiB/s, it could be the card too.

03:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
04:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)
05:00.0 RAID bus controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

(Not using RAID, they also act as a normal SATA controller)

Otherwise, very good write-up!

Justin.

RE: new bottleneck section in wiki

[David Lethe]

-----Original Message-----
From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
Sent: Wednesday, July 02, 2008 10:56 AM
To: linux-raid@vger.kernel.org
Subject: new bottleneck section in wiki

I should have done something else this afternoon, but anyway, I was
inspired to write up this text for the wiki. Comments welcome.

Keld

Bottlenecks

There can be a number of bottlenecks other than the disk subsystem that
hinders you in getting full performance out of your disks.

One is the PCI bus. Older PCI bus has a 33 MHz cycle and a 32 bit width,
giving a maximum bandwidth of about 1 Gbit/s, or 133 MB/s. This will
easily cause trouble with newer SATA disks which easily gives 70-90 MB/s
each. So do not put your SATA controllers on a 33 MHz PCI bus.

The 66 MHz 64-bit PCI bus is capable of handling about 4 Gbit/s, or
about 500 MB/s. This can also be a bottleneck with bigger arrays, eg a 6
drive array will be able to deliver about 500 MB/s, and maybe you want
also to feed a gigabyte ethernet card - 125 MB/s, totalling potentially
625 MB/s on the PCI bus.

The PCI-Express bus v1.1 has a limit of 250 MB/s per lane per dirction,
and that limit can easily be hit eg by a 4-drive array.

Many SATA controllers are on-board and do not use the PCI bus. Anyway
bandwidth is limited, but it is probably different from motherboard to
motherboard. On board disk controllers most likely have a bigger
bandwidth than IO controllers on a 32-bit PCI 33 MHz, 64-bit PCI 66 MHz,
or PCI-E x1 bus.

Having a RAID connected over the LAN can be a bottleneck, if the LAN
speed is only 1 Gbit/s - this limits the speed of the IO system to 125
MB/s by itself.

Classical bottlenecks are PATA drives placed on the same DMA channel, or
the same PATA cable. This will of cause limit performance, but it should
work, given you have no other means of connecting your disks by. Also
placing more than one element of an array on the same disk hurts
performace seriously, and also gives redundancy problems.

A classical problem is also not to have enabled DMA transfer, or having
lost this setting due to some problem, including not well connected
cables, or setting the transfer speed to less than optimal.

RAM sppec may be a bottleneck. Using 32 bit RAM - or using a 32 bit
operating system may double time spent reading and writing RAM.

CPU usage may be a bottleneck, also combined with slow RAM or only using
RAM in 32-bit mode.

BIOS settings may also impede your performance. 
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
=================================================================

I would add -
The PCI (and PCI-X) bus is shared bandwidth, and operates at lowest common denominator.  Put a 33Mhz card in the PCI bus, and not only does everything operate at 33Mhz, but all of the cards compete.  Grossly simplified, if you have a 133Mhz card and a 33Mhz card in the same PCI bus, then that card will operate at 16Mhz. Your motherboard's embedded Ethernet chip and disk controllers are "on" the PCI bus, so even if you have a single PCI controller card, and a multiple-bus motherboard, then it does make a difference what slot you put the controller in.

If this isn't bad enough, then consider the consequences of arbitration.  All of the PCI devices have to constantly negotiate between themselves to get a chance to compete against all of the other devices attached to other PCI busses to get a chance to talk to the CPU and RAM.  As such, every packet your Ethernet card picks up could temporarily suspend disk I/O if you don't configure things wisely.

- David Lethe






--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Keld Jørn Simonsen]

On Wed, Jul 02, 2008 at 12:43:40PM -0400, Justin Piszcz wrote:
> 
> 
> On Wed, 2 Jul 2008, Keld Jørn Simonsen wrote:
> 
> >The 66 MHz 64-bit PCI bus is capable of handling about 4 Gbit/s, or
> >about 500 MB/s. This can also be a bottleneck with bigger arrays, eg a 6
> >drive array will be able to deliver about 500 MB/s, and maybe you want
> >also to feed a gigabyte ethernet card - 125 MB/s, totalling potentially
> >625 MB/s on the PCI bus.
> >
> >The PCI-Express bus v1.1 has a limit of 250 MB/s per lane per dirction,
> >and that limit can easily be hit eg by a 4-drive array.

> Correction, can easily be hit with two veliciraptors.

OK, I added that information. 

Best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Iustin Pop]

On Wed, Jul 02, 2008 at 05:56:03PM +0200, Keld Jørn Simonsen wrote:
> I should have done something else this afternoon, but anyway, I was
> inspired to write up this text for the wiki. Comments welcome.
> 
> Keld
> [...]
> Many SATA controllers are on-board and do not use the PCI bus. Anyway
> bandwidth is limited, but it is probably different from motherboard to
> motherboard. On board disk controllers most likely have a bigger
> bandwidth than IO controllers on a 32-bit PCI 33 MHz, 64-bit PCI 66 MHz,
> or PCI-E x1 bus.

Sorry to ask... are you sure the on-board controllers are *not* on the
PCI bus?

They are not physically over the PCI bus, but still connected via the
same upstream-controller, I think, and still limited.

I'm not familiar in this area, but the mainboard diagrams (the ones with
pretty pictures, not electric ones) show that the on-board controllers
'hang' on the same PCI or PCI-E bus as the physical slots.

> RAM sppec may be a bottleneck. Using 32 bit RAM - or using a 32 bit
> operating system may double time spent reading and writing RAM.
> 
> CPU usage may be a bottleneck, also combined with slow RAM or only using
> RAM in 32-bit mode.

Sorry, what is "32 bit RAM"? I never heard of this. Do you mean
dual-channel versus single-channel?

thanks!
iustin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Keld Jørn Simonsen]

On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
> Sent: Wednesday, July 02, 2008 10:56 AM
> To: linux-raid@vger.kernel.org
> Subject: new bottleneck section in wiki
> 
> I should have done something else this afternoon, but anyway, I was
> inspired to write up this text for the wiki. Comments welcome.
 ....

> 
> I would add -
> The PCI (and PCI-X) bus is shared bandwidth, and operates at lowest common denominator.  Put a 33Mhz card in the PCI bus, and not only does everything operate at 33Mhz, but all of the cards compete.  Grossly simplified, if you have a 133Mhz card and a 33Mhz card in the same PCI bus, then that card will operate at 16Mhz. Your motherboard's embedded Ethernet chip and disk controllers are "on" the PCI bus, so even if you have a single PCI controller card, and a multiple-bus motherboard, then it does make a difference what slot you put the controller in.
> 
> If this isn't bad enough, then consider the consequences of arbitration.  All of the PCI devices have to constantly negotiate between themselves to get a chance to compete against all of the other devices attached to other PCI busses to get a chance to talk to the CPU and RAM.  As such, every packet your Ethernet card picks up could temporarily suspend disk I/O if you don't configure things wisely.

Thanks, I added this text, modifed a little. And also I would like to
note that I was inspired by some emailing with you, when writing the
text.

Current motherboards with onboard disk controlles normally do not have
the disk IO connected via the PCI or PCI-E busses, but rather directly via the
southbridge. What are typical transfer rates between the southbridge
and the northbridge? Could this potentionally be a bottleneck?

And also the disk controllers, could these be bottlenecks? They typically
operate at 300 MB/s nominally, per disk channel, and presumably they
then have a connection to the southbridge that is capable of handling
this speed. So for a 4-disk SATA-II controller this would be at least
1200 MB/s or about 10 gigabit. 

best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: new bottleneck section in wiki

[David Lethe]

-----Original Message-----
From: Keld Jørn Simonsen [mailto:keld@dkuug.dk] 
Sent: Wednesday, July 02, 2008 12:51 PM
To: David Lethe
Cc: linux-raid@vger.kernel.org
Subject: Re: new bottleneck section in wiki

On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
> Sent: Wednesday, July 02, 2008 10:56 AM
> To: linux-raid@vger.kernel.org
> Subject: new bottleneck section in wiki
> 
> I should have done something else this afternoon, but anyway, I was
> inspired to write up this text for the wiki. Comments welcome.
 ....

> 
> I would add -
> The PCI (and PCI-X) bus is shared bandwidth, and operates at lowest common denominator.  Put a 33Mhz card in the PCI bus, and not only does everything operate at 33Mhz, but all of the cards compete.  Grossly simplified, if you have a 133Mhz card and a 33Mhz card in the same PCI bus, then that card will operate at 16Mhz. Your motherboard's embedded Ethernet chip and disk controllers are "on" the PCI bus, so even if you have a single PCI controller card, and a multiple-bus motherboard, then it does make a difference what slot you put the controller in.
> 
> If this isn't bad enough, then consider the consequences of arbitration.  All of the PCI devices have to constantly negotiate between themselves to get a chance to compete against all of the other devices attached to other PCI busses to get a chance to talk to the CPU and RAM.  As such, every packet your Ethernet card picks up could temporarily suspend disk I/O if you don't configure things wisely.

Thanks, I added this text, modifed a little. And also I would like to
note that I was inspired by some emailing with you, when writing the
text.

Current motherboards with onboard disk controlles normally do not have
the disk IO connected via the PCI or PCI-E busses, but rather directly via the
southbridge. What are typical transfer rates between the southbridge
and the northbridge? Could this potentionally be a bottleneck?

And also the disk controllers, could these be bottlenecks? They typically
operate at 300 MB/s nominally, per disk channel, and presumably they
then have a connection to the southbridge that is capable of handling
this speed. So for a 4-disk SATA-II controller this would be at least
1200 MB/s or about 10 gigabit. 

best regards
keld
-------------------
It is much more complicated than just saying what the transfer rates are, especially in the world of blocking, arbitration, and unbalanced I/O.  

Everything is a potential bottleneck.  As I am under NDA with most of the controller vendors, then I can not provide specifics, but suffice to say that certain cards with certain chipsets will max out at well under published speeds.  Heck, you could attach solid-state disks with random I/O access time in the nanosecond range and still only get 150MB/sec out of certain controllers, even on a PCIe X 16 bus. 

BTW, there isn't a SATA-II controller in the planet that will deliver 1200 MB/sec with 4 disk drives. 

David



--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Keld Jørn Simonsen]

On Wed, Jul 02, 2008 at 07:33:31PM +0200, Iustin Pop wrote:
> On Wed, Jul 02, 2008 at 05:56:03PM +0200, Keld Jørn Simonsen wrote:
> > I should have done something else this afternoon, but anyway, I was
> > inspired to write up this text for the wiki. Comments welcome.
> > 
> > Keld
> > [...]
> > Many SATA controllers are on-board and do not use the PCI bus. Anyway
> > bandwidth is limited, but it is probably different from motherboard to
> > motherboard. On board disk controllers most likely have a bigger
> > bandwidth than IO controllers on a 32-bit PCI 33 MHz, 64-bit PCI 66 MHz,
> > or PCI-E x1 bus.
> 
> Sorry to ask... are you sure the on-board controllers are *not* on the
> PCI bus?

My understanding is that they are not on the PCI or PCI-E bus, but on
the southbridge. The southbridge then handles the PCI and PCI-E busses,
the ISA bus, and other busses, and also the IO controller.
Then the southbridge is connected to the northbridge.

> They are not physically over the PCI bus, but still connected via the
> same upstream-controller, I think, and still limited.

Yes, and I do not know what speeds that are typical here.
I would like to know more, if somebody can enlighten me.

> I'm not familiar in this area, but the mainboard diagrams (the ones with
> pretty pictures, not electric ones) show that the on-board controllers
> 'hang' on the same PCI or PCI-E bus as the physical slots.

This has been the case previously, but my understanding is that
this is not the case any longer for current motherboards, even for the
cheapest current motherboards.

> > RAM sppec may be a bottleneck. Using 32 bit RAM - or using a 32 bit
> > operating system may double time spent reading and writing RAM.
> > 
> > CPU usage may be a bottleneck, also combined with slow RAM or only using
> > RAM in 32-bit mode.
> 
> Sorry, what is "32 bit RAM"? I never heard of this. Do you mean
> dual-channel versus single-channel?

My terminology may be wrong. But I understand that if you
are running an OS as a 32 bit OS, then your memcpy() function
operate on the memory in 32 bit quantities, taking double the time to
move a memory area as if it were done with 64-bit operations. 
I may be wrong there, and I would be glad if anybody could clarify.
Maybe the 32-bit OS and C library does this with 64-bit loads and
stores, or maybe there is an instruction in use to move a whole string
in one instruction.

A number of people run a 32-bit OS on 64 bit machine, either because a 64
bit version of the OS is not available, or that some parts of the
functionality, eg. codecs are only available in a 32-bit version, or for
reasons of ignorance, they did not know that a 64 bit OS would be
better. So this is my concern that there may be room for improvement in
these cases.

best regards
keld

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Keld Jørn Simonsen]

On Wed, Jul 02, 2008 at 01:08:04PM -0500, David Lethe wrote:
> 
> 
> And also the disk controllers, could these be bottlenecks? They typically
> operate at 300 MB/s nominally, per disk channel, and presumably they
> then have a connection to the southbridge that is capable of handling
> this speed. So for a 4-disk SATA-II controller this would be at least
> 1200 MB/s or about 10 gigabit. 
> 
> best regards
> keld
> -------------------
> It is much more complicated than just saying what the transfer rates are, especially in the world of blocking, arbitration, and unbalanced I/O.  

Yes, that is understood, but I am only listing some potential
bottlenecs, of cause there may be more.

> Everything is a potential bottleneck.  As I am under NDA with most of the controller vendors, then I can not provide specifics, but suffice to say that certain cards with certain chipsets will max out at well under published speeds.  Heck, you could attach solid-state disks with random I/O access time in the nanosecond range and still only get 150MB/sec out of certain controllers, even on a PCIe X 16 bus. 
> 
> BTW, there isn't a SATA-II controller in the planet that will deliver 1200 MB/sec with 4 disk drives. 

Yes, but I think this is normally due to that the max transfer speed per
disk is in the ballpark of 80-120 MB/s - which is less than half the
SATA-II max speed. And I think much of this slowdown comes from head movement,
track-to-track, disk latency etc. I was of the impression, that when the
transfer between the disk and the controller is going on, then the
transfer speed would be not far from the 300 MB/s max speed, eg for
90 MB/s 1 TB disks that I bougth recently, or the faster 15000 RPM
disks, which give something like 120 MB/s.

best regards
keld

Re: new bottleneck section in wiki

[Matt Garman]

On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> The PCI (and PCI-X) bus is shared bandwidth, and operates at
> lowest common denominator.  Put a 33Mhz card in the PCI bus, and
> not only does everything operate at 33Mhz, but all of the cards
> compete.  Grossly simplified, if you have a 133Mhz card and a
> 33Mhz card in the same PCI bus, then that card will operate at
> 16Mhz. Your motherboard's embedded Ethernet chip and disk
> controllers are "on" the PCI bus, so even if you have a single PCI
> controller card, and a multiple-bus motherboard, then it does make
> a difference what slot you put the controller in.

Is that true for all PCI-X implementations?  What's the point, then,
of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
card (32 bit/33 MHz)?

A lot of "server" motherboards offer PCI-X and some simple graphics
chip.  If you read the motherboard specs, that simple graphics is
usually attached to the PCI bus [1].  So what's the point of having
PCI-X slots if everything is automatically downgraded to PCI speeds
due to the embedded graphics?

I read some of the high-level info on the Intel 6702 PHX PCI-X hub
[2].  If I understand correctly, that controller is actually
attached to the PCI express bus.  So to me, it seems possible that
PCI and PCI-X could be independant, and that PCI-X will compete with
PCI-E for bandwidth.

[1] The ASUS M2N-LR has PCI-X (via the Intel 6702PHX) and an
embedded ATI ES1000 video card.  The ES1000's specs say it has a PCI
bus interface.
ES1000: http://ati.amd.com/products/server/es1000/index.html

[2] http://www.intel.com/design/chipsets/datashts/303633.htm

Re: new bottleneck section in wiki

[Jon Nelson]

On Wed, Jul 2, 2008 at 2:03 PM, Matt Garman <matthew.garman@gmail.com> wrote:
> On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
>> The PCI (and PCI-X) bus is shared bandwidth, and operates at
>> lowest common denominator.  Put a 33Mhz card in the PCI bus, and
>> not only does everything operate at 33Mhz, but all of the cards
>> compete.  Grossly simplified, if you have a 133Mhz card and a
>> 33Mhz card in the same PCI bus, then that card will operate at
>> 16Mhz. Your motherboard's embedded Ethernet chip and disk
>> controllers are "on" the PCI bus, so even if you have a single PCI
>> controller card, and a multiple-bus motherboard, then it does make
>> a difference what slot you put the controller in.
>
> Is that true for all PCI-X implementations?  What's the point, then,
> of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
> card (32 bit/33 MHz)?

This motherboard (EPoX MF570SLI) uses PCI-E.
It has a plain old PCI video card in it:
Trident Microsystems TGUI 9660/938x/968x
and yet I appear to be able to sustain plenty of disk bandwidth to 4 drives:
(dd if=/dev/sd[b,c,d,e] of=/dev/null bs=64k)
vmstat 1 reports:
290000 to 310000 "blocks in", hovering around 300000.

4x70 would be more like 280, 4x75 is 300. Clearly the system is not
bandwidth challenged.
(This is with 4500 context switches/second, BTW.)

-- 
Jon

Re: new bottleneck section in wiki

[Robin Hill]

[-- Attachment #1: Type: text/plain, Size: 2486 bytes --]

On Wed Jul 02, 2008 at 02:03:37PM -0500, Matt Garman wrote:

> On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> > The PCI (and PCI-X) bus is shared bandwidth, and operates at
> > lowest common denominator.  Put a 33Mhz card in the PCI bus, and
> > not only does everything operate at 33Mhz, but all of the cards
> > compete.  Grossly simplified, if you have a 133Mhz card and a
> > 33Mhz card in the same PCI bus, then that card will operate at
> > 16Mhz. Your motherboard's embedded Ethernet chip and disk
> > controllers are "on" the PCI bus, so even if you have a single PCI
> > controller card, and a multiple-bus motherboard, then it does make
> > a difference what slot you put the controller in.
> 
> Is that true for all PCI-X implementations?  What's the point, then,
> of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
> card (32 bit/33 MHz)?
> 
> A lot of "server" motherboards offer PCI-X and some simple graphics
> chip.  If you read the motherboard specs, that simple graphics is
> usually attached to the PCI bus [1].  So what's the point of having
> PCI-X slots if everything is automatically downgraded to PCI speeds
> due to the embedded graphics?
> 
Server class boards have multiple PCI buses (2 or 3 usually), so the
PCI-X slots are on a different bus.

> I read some of the high-level info on the Intel 6702 PHX PCI-X hub
> [2].  If I understand correctly, that controller is actually
> attached to the PCI express bus.  So to me, it seems possible that
> PCI and PCI-X could be independant, and that PCI-X will compete with
> PCI-E for bandwidth.
> 
Newer workstation (and probably server boards) have the PCI-X slots on a
PCI-E "bus" (hence you can actually get boards with PCI-X slots for a
reasonable price nowadays).  In actual fact PCI-E doesn't have a bus as
such, all the connections are point-to-point so there's no bandwidth
contention at that stage.  You may get contention afterwards though,
depending on how the PCI-E lanes are connected to the processor.  AMDs
HyperTransport is supposed to be very good for this, whereas Intel
systems tend to suffer more (or did - I've not looked at this recently
so they may have moved on now).

Cheers,
    Robin
-- 
     ___        
    ( ' }     |       Robin Hill        <robin@robinhill.me.uk> |
   / / )      | Little Jim says ....                            |
  // !!       |      "He fallen in de water !!"                 |

[-- Attachment #2: Type: application/pgp-signature, Size: 197 bytes --]

Re: new bottleneck section in wiki

[Keld J?rn Simonsen]

On Wed, Jul 02, 2008 at 02:10:21PM -0500, Jon Nelson wrote:
> On Wed, Jul 2, 2008 at 2:03 PM, Matt Garman <matthew.garman@gmail.com> wrote:
> > On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> >> The PCI (and PCI-X) bus is shared bandwidth, and operates at
> >> lowest common denominator.  Put a 33Mhz card in the PCI bus, and
> >> not only does everything operate at 33Mhz, but all of the cards
> >> compete.  Grossly simplified, if you have a 133Mhz card and a
> >> 33Mhz card in the same PCI bus, then that card will operate at
> >> 16Mhz. Your motherboard's embedded Ethernet chip and disk
> >> controllers are "on" the PCI bus, so even if you have a single PCI
> >> controller card, and a multiple-bus motherboard, then it does make
> >> a difference what slot you put the controller in.
> >
> > Is that true for all PCI-X implementations?  What's the point, then,
> > of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
> > card (32 bit/33 MHz)?
> 
> This motherboard (EPoX MF570SLI) uses PCI-E.

PCI-E is quite different architecturally from PCI-X.

> It has a plain old PCI video card in it:
> Trident Microsystems TGUI 9660/938x/968x
> and yet I appear to be able to sustain plenty of disk bandwidth to 4 drives:
> (dd if=/dev/sd[b,c,d,e] of=/dev/null bs=64k)
> vmstat 1 reports:
> 290000 to 310000 "blocks in", hovering around 300000.
> 
> 4x70 would be more like 280, 4x75 is 300. Clearly the system is not
> bandwidth challenged.
> (This is with 4500 context switches/second, BTW.)

Possibly you are using an on-board disk controller, and then it most
likely does not use the PCI-E bus for disk IO.

best regards
keld

Re: new bottleneck section in wiki

[Jon Nelson]

>> This motherboard (EPoX MF570SLI) uses PCI-E.
>
> PCI-E is quite different architecturally from PCI-X.
>
>> It has a plain old PCI video card in it:
>> Trident Microsystems TGUI 9660/938x/968x
>> and yet I appear to be able to sustain plenty of disk bandwidth to 4 drives:
>> (dd if=/dev/sd[b,c,d,e] of=/dev/null bs=64k)
>> vmstat 1 reports:
>> 290000 to 310000 "blocks in", hovering around 300000.
>>
>> 4x70 would be more like 280, 4x75 is 300. Clearly the system is not
>> bandwidth challenged.
>> (This is with 4500 context switches/second, BTW.)
>
> Possibly you are using an on-board disk controller, and then it most
> likely does not use the PCI-E bus for disk IO.

I only point it out to show how this setup scales. If there were
bottlenecks in the chipset, they'd have shown up in the test.

-- 
Jon

Re: new bottleneck section in wiki

[Keld J?rn Simonsen]

On Wed, Jul 02, 2008 at 02:03:37PM -0500, Matt Garman wrote:
> On Wed, Jul 02, 2008 at 12:04:11PM -0500, David Lethe wrote:
> > The PCI (and PCI-X) bus is shared bandwidth, and operates at
> > lowest common denominator.  Put a 33Mhz card in the PCI bus, and
> > not only does everything operate at 33Mhz, but all of the cards
> > compete.  Grossly simplified, if you have a 133Mhz card and a
> > 33Mhz card in the same PCI bus, then that card will operate at
> > 16Mhz. Your motherboard's embedded Ethernet chip and disk
> > controllers are "on" the PCI bus, so even if you have a single PCI
> > controller card, and a multiple-bus motherboard, then it does make
> > a difference what slot you put the controller in.
> 
> Is that true for all PCI-X implementations?  What's the point, then,
> of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
> card (32 bit/33 MHz)?

My understanding is that this is not true for all PCI-X busses, only for
some.

> A lot of "server" motherboards offer PCI-X and some simple graphics
> chip.  If you read the motherboard specs, that simple graphics is
> usually attached to the PCI bus [1].  So what's the point of having
> PCI-X slots if everything is automatically downgraded to PCI speeds
> due to the embedded graphics?

I think there are some mobos that have both PCI-X and PCI busses.

> I read some of the high-level info on the Intel 6702 PHX PCI-X hub
> [2].  If I understand correctly, that controller is actually
> attached to the PCI express bus.  So to me, it seems possible that
> PCI and PCI-X could be independant, and that PCI-X will compete with
> PCI-E for bandwidth.

Yes, that is possible.

> [1] The ASUS M2N-LR has PCI-X (via the Intel 6702PHX) and an
> embedded ATI ES1000 video card.  The ES1000's specs say it has a PCI
> bus interface.
> ES1000: http://ati.amd.com/products/server/es1000/index.html
> 
> [2] http://www.intel.com/design/chipsets/datashts/303633.htm

best regards
keld

Re: new bottleneck section in wiki

[Matt Garman]

On Wed, Jul 02, 2008 at 01:08:04PM -0500, David Lethe wrote:
> Everything is a potential bottleneck.  As I am under NDA with most
> of the controller vendors, then I can not provide specifics, but
> suffice to say that certain cards with certain chipsets will max
> out at well under published speeds.  Heck, you could attach
> solid-state disks with random I/O access time in the nanosecond
> range and still only get 150MB/sec out of certain controllers,
> even on a PCIe X 16 bus. 

Short of signing an NDA, how would one go about determining which
chipsets are least likely to be a bottleneck?  I'm interested in
building an NFS server with a Linux software RAID-5 data store.  To
me, that means my I/O subsystem should be as fast and capable as
possible.

For example, looking at the block diagram [1] of Intel's P45/ICH10
chipset [2], it appears that the link between the north and south
bridges is only 2 GB/s.  I would think that any raid level that
requires the CPU (e.g. parity calculations) would clog that link
fairly quickly, at least if large block transfers are taking place.
And then I wonder what impact that has on the performance of the
NIC(s) (I don't know how much a NIC has to talk to the CPU).

Then there are chips like nvidia's 8200 [3] that don't separate the
north and south bridges.  I can't find a block diagram on their
website, but I found a few via google "nvidia 8200 block diagram".
[4]  It looks like the 8200 MCP is connected via HT3 (don't know
exactly what that means/implies), but everything is connected to
this one chip (PCI, SATA, PCI-Express, NIC, etc).

Anyone have any suggestions on getting a motherboard with the most
I/O bandwidth?

[1] http://www.intel.com/products/chipsets/P45/pix/p45_blockdiagram.gif

[2] http://www.intel.com/products/chipsets/P45/index.htm

[3] http://www.nvidia.com/object/geforce_8200mgpu.html
    http://www.nvidia.com/object/mobo_gpu_tech_specs.html

[4] http://www.hardwarezone.com.sg/img/data/articles/2008/2453/GeForce_8200_Block_Diagram.jpg
    http://www.pcper.com/images/reviews/507/GeForce_8200_Block_Diagram.jpg
    http://media.arstechnica.com/news.media/GeForce8200.jpg

Re: new bottleneck section in wiki

[Keld J?rn Simonsen]

On Wed, Jul 02, 2008 at 02:45:46PM -0500, Matt Garman wrote:
> On Wed, Jul 02, 2008 at 01:08:04PM -0500, David Lethe wrote:
> > Everything is a potential bottleneck.  As I am under NDA with most
> > of the controller vendors, then I can not provide specifics, but
> > suffice to say that certain cards with certain chipsets will max
> > out at well under published speeds.  Heck, you could attach
> > solid-state disks with random I/O access time in the nanosecond
> > range and still only get 150MB/sec out of certain controllers,
> > even on a PCIe X 16 bus. 
> 
> Short of signing an NDA, how would one go about determining which
> chipsets are least likely to be a bottleneck?  I'm interested in
> building an NFS server with a Linux software RAID-5 data store.  To
> me, that means my I/O subsystem should be as fast and capable as
> possible.
> 
> For example, looking at the block diagram [1] of Intel's P45/ICH10
> chipset [2], it appears that the link between the north and south
> bridges is only 2 GB/s.  I would think that any raid level that
> requires the CPU (e.g. parity calculations) would clog that link
> fairly quickly, at least if large block transfers are taking place.
> And then I wonder what impact that has on the performance of the
> NIC(s) (I don't know how much a NIC has to talk to the CPU).

2 GB/s seems adequate. Not many raids are in this class. Theoretically
you would need something like more than 20 disks to get this bandwidth,
and normally you only have 4 to 8 disks attachable to IO controllers on
the mobo. 

Parity calculations are done by the cpu on the ram, and should not touch
the northbridge - southbridge internal bus in vanilla motherboards.

best regards
keld

Re: new bottleneck section in wiki

[Richard Scobie]

Matt Garman wrote:

> Anyone have any suggestions on getting a motherboard with the most
> I/O bandwidth?

Ideally you want to be avoiding the south bridge altogether.

A board like the Tyan S5396 has an MCH with 2 x 16 lane PCIe slots, so a 
high bandwidth storage controller can go in one slot and quad GigE or 
10GE in the other and both have direct access to CPUs/RAM.

These slots are also PCIe V2.0 capable, although I am not aware of any 
card that take advantage of this.

Regards,

Richard

Re: new bottleneck section in wiki

[Roger Heflin]

David Lethe wrote:
> -----Original Message-----
> From: linux-raid-owner@vger.kernel.org [mailto:linux-raid-owner@vger.kernel.org] On Behalf Of Keld Jørn Simonsen
> Sent: Wednesday, July 02, 2008 10:56 AM
> To: linux-raid@vger.kernel.org
> Subject: new bottleneck section in wiki
> 
> I should have done something else this afternoon, but anyway, I was
> inspired to write up this text for the wiki. Comments welcome.
> 
> Keld
> 
> Bottlenecks
> 
> There can be a number of bottlenecks other than the disk subsystem that
> hinders you in getting full performance out of your disks.
> 
> One is the PCI bus. Older PCI bus has a 33 MHz cycle and a 32 bit width,
> giving a maximum bandwidth of about 1 Gbit/s, or 133 MB/s. This will
> easily cause trouble with newer SATA disks which easily gives 70-90 MB/s
> each. So do not put your SATA controllers on a 33 MHz PCI bus.
> 
> The 66 MHz 64-bit PCI bus is capable of handling about 4 Gbit/s, or
> about 500 MB/s. This can also be a bottleneck with bigger arrays, eg a 6
> drive array will be able to deliver about 500 MB/s, and maybe you want
> also to feed a gigabyte ethernet card - 125 MB/s, totalling potentially
> 625 MB/s on the PCI bus.
> 
> The PCI-Express bus v1.1 has a limit of 250 MB/s per lane per dirction,
> and that limit can easily be hit eg by a 4-drive array.
> 
> Many SATA controllers are on-board and do not use the PCI bus. Anyway
> bandwidth is limited, but it is probably different from motherboard to
> motherboard. On board disk controllers most likely have a bigger
> bandwidth than IO controllers on a 32-bit PCI 33 MHz, 64-bit PCI 66 MHz,
> or PCI-E x1 bus.
> 
> Having a RAID connected over the LAN can be a bottleneck, if the LAN
> speed is only 1 Gbit/s - this limits the speed of the IO system to 125
> MB/s by itself.
> 
> Classical bottlenecks are PATA drives placed on the same DMA channel, or
> the same PATA cable. This will of cause limit performance, but it should
> work, given you have no other means of connecting your disks by. Also
> placing more than one element of an array on the same disk hurts
> performace seriously, and also gives redundancy problems.
> 
> A classical problem is also not to have enabled DMA transfer, or having
> lost this setting due to some problem, including not well connected
> cables, or setting the transfer speed to less than optimal.
> 
> RAM sppec may be a bottleneck. Using 32 bit RAM - or using a 32 bit
> operating system may double time spent reading and writing RAM.
> 
> CPU usage may be a bottleneck, also combined with slow RAM or only using
> RAM in 32-bit mode.
> 
> BIOS settings may also impede your performance. 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> =================================================================
> 
> I would add -
> The PCI (and PCI-X) bus is shared bandwidth, and operates at 
 > lowest common denominator.  Put a 33Mhz card in the PCI bus,
> and not only does everything operate at 33Mhz, but all of 
 > the cards compete.  Grossly simplified, if you have a 133Mhz
 > card and a 33Mhz card in the same PCI bus, then that card
 > will operate at 16Mhz. Your motherboard's embedded Ethernet
 > chip and disk controllers are "on" the PCI bus, so even if
 > you have a single PCI controller card, and a multiple-bus
 > motherboard, then it does make a difference what slot
 > you put the controller in.

Add in on the higher end MB's (with PCI-X, and PCIe, and
stuff on the built-in to the motherboard) there is often a nice
block diagram  that indicates which resources are sharing bandwidth,
and often how much bandwidth they are sharing, so if one is
careful one can carefully put different things on unshared parts,
and take careful note of what other MB things they are being
shared with.  With desktop motherboards this does not generally
matter at all as there is typically only one PCI (32bit) bus and it
is all shared.   And often the stuff on the MB is only connected
slightly better that a 32-bit/33mhz PCI bus, so one has to be
careful and take note of the reality of their MB.

> 
> If this isn't bad enough, then consider the consequences of
 > arbitration.  All of the PCI devices have to constantly
> negotiate between themselves to get a chance to compete 
> against all of the other devices attached to other PCI 
> busses to get a chance to talk to the CPU and RAM.  As 
> such, every packet your Ethernet card picks up could 
> temporarily suspend disk I/O if you don't configure things wisely.

And note that in my experience if you are going to find a "bug" in the MB design 
this sharing/arbitration under high loads is where you will find it, and it can 
result in everything from silent corruption to the entire machine crashing when
put under heavy load.

                              Roger
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: new bottleneck section in wiki

[Roger Heflin]

Keld Jørn Simonsen wrote:
> On Wed, Jul 02, 2008 at 01:08:04PM -0500, David Lethe wrote:
>>
>> And also the disk controllers, could these be bottlenecks? They typically
>> operate at 300 MB/s nominally, per disk channel, and presumably they
>> then have a connection to the southbridge that is capable of handling
>> this speed. So for a 4-disk SATA-II controller this would be at least
>> 1200 MB/s or about 10 gigabit. 
>>
>> best regards
>> keld
>> -------------------
>> It is much more complicated than just saying what the transfer rates are, especially in the world of blocking, arbitration, and unbalanced I/O.  
> 
> Yes, that is understood, but I am only listing some potential
> bottlenecs, of cause there may be more.
> 
>> Everything is a potential bottleneck.  As I am under NDA with most of the controller vendors, then I can not provide specifics, but suffice to say that certain cards with certain chipsets will max out at well under published speeds.  Heck, you could attach solid-state disks with random I/O access time in the nanosecond range and still only get 150MB/sec out of certain controllers, even on a PCIe X 16 bus. 
>>
>> BTW, there isn't a SATA-II controller in the planet that will deliver 1200 MB/sec with 4 disk drives. 
> 
> Yes, but I think this is normally due to that the max transfer speed per
> disk is in the ballpark of 80-120 MB/s - which is less than half the
> SATA-II max speed. And I think much of this slowdown comes from head movement,
> track-to-track, disk latency etc. I was of the impression, that when the
> transfer between the disk and the controller is going on, then the
> transfer speed would be not far from the 300 MB/s max speed, eg for
> 90 MB/s 1 TB disks that I bougth recently, or the faster 15000 RPM
> disks, which give something like 120 MB/s.

Actually enough sectors are only passing under the head at that 70-120MB/second 
rate, so when you add in the seeks and such things are a bit lower, but you 
aren't ever going to be able to exceed that given bit rate, and the bit rate 
changes from the inside of the disk to the outside of the disk (the outside has 
more sectors on it than the inside), in the disk data sheet's there is usually a 
range of bit rates listed showing this.

                             Roger

                                 Roger
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: new bottleneck section in wiki

[David Lethe]

-----Original Message-----
From: jnelson@jamponi.net [mailto:jnelson@jamponi.net] On Behalf Of Jon
Nelson
Sent: Wednesday, July 02, 2008 2:38 PM
To: Keld J?rn Simonsen
Cc: Matt Garman; David Lethe; linux-raid@vger.kernel.org
Subject: Re: new bottleneck section in wiki

>> This motherboard (EPoX MF570SLI) uses PCI-E.
>
> PCI-E is quite different architecturally from PCI-X.
>
>> It has a plain old PCI video card in it:
>> Trident Microsystems TGUI 9660/938x/968x
>> and yet I appear to be able to sustain plenty of disk bandwidth to 4
drives:
>> (dd if=/dev/sd[b,c,d,e] of=/dev/null bs=64k)
>> vmstat 1 reports:
>> 290000 to 310000 "blocks in", hovering around 300000.
>>
>> 4x70 would be more like 280, 4x75 is 300. Clearly the system is not
>> bandwidth challenged.
>> (This is with 4500 context switches/second, BTW.)
>
> Possibly you are using an on-board disk controller, and then it most
> likely does not use the PCI-E bus for disk IO.

I only point it out to show how this setup scales. If there were
bottlenecks in the chipset, they'd have shown up in the test.

-- 
Jon
===========================
Not true, Jon.  The test above was limited by the amount of data that a
single drive head can push to the controller with 100% sequential reads
of 64KB.  You can't say anything about chipset bottlenecks, because you
haven't created a condition where the chipset could even be a
bottleneck.  I/O is constrained by the disk drives.

Now, if you attached industrial class SSDs that operate at media speeds
with access time in the NS range, then you could be in a position to
benchmark chipset performance.  

David

Re: new bottleneck section in wiki

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 1952 bytes --]

On Wed, 2008-07-02 at 14:03 -0500, Matt Garman wrote:
> Is that true for all PCI-X implementations?  What's the point, then,
> of having PCI-X (64 bit/66 MHz or greater) if you have even one PCI
> card (32 bit/33 MHz)?

A 32bit/33MHz card only slows down the particular bus it's plugged into.
A lot of the server class machines that have had PCI-X in the past made
a specific point of using multiple PCI busses for the slots so that
plugging in that 32bit/33MHz card wouldn't effect the PCI-X card next to
it because they were on physically distinct busses.

> A lot of "server" motherboards offer PCI-X and some simple graphics
> chip.  If you read the motherboard specs, that simple graphics is
> usually attached to the PCI bus [1].  So what's the point of having
> PCI-X slots if everything is automatically downgraded to PCI speeds
> due to the embedded graphics?

Again, different physical busses.

> I read some of the high-level info on the Intel 6702 PHX PCI-X hub
> [2].  If I understand correctly, that controller is actually
> attached to the PCI express bus.  So to me, it seems possible that
> PCI and PCI-X could be independant, and that PCI-X will compete with
> PCI-E for bandwidth.
> 
> [1] The ASUS M2N-LR has PCI-X (via the Intel 6702PHX) and an
> embedded ATI ES1000 video card.  The ES1000's specs say it has a PCI
> bus interface.
> ES1000: http://ati.amd.com/products/server/es1000/index.html
> 
> [2] http://www.intel.com/design/chipsets/datashts/303633.htm
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
Doug Ledford <dledford@redhat.com>
              GPG KeyID: CFBFF194
              http://people.redhat.com/dledford

Infiniband specific RPMs available at
              http://people.redhat.com/dledford/Infiniband


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: new bottleneck section in wiki

[Jon Nelson]

> Not true, Jon.  The test above was limited by the amount of data that a
> single drive head can push to the controller with 100% sequential reads
> of 64KB.  You can't say anything about chipset bottlenecks, because you
> haven't created a condition where the chipset could even be a
> bottleneck.  I/O is constrained by the disk drives.
>
> Now, if you attached industrial class SSDs that operate at media speeds
> with access time in the NS range, then you could be in a position to
> benchmark chipset performance.

Well, that's true. However, perhaps I mis-spoke. What I was intending
on doing is showing that the chipset was /not/ the limiting factor
here, not checking it's maximum speed, but rather given what I know
about the drives individually, and then what the rates are with all
drives cookin', then I can say, "Well, at least the chipset or bus
aren't getting in the way." Note that I also added "or bus" to my
statement. Would you say that this statement is more accurate or am I
getting the terminology wrong, again?


-- 
Jon

Re: new bottleneck section in wiki

[Justin Piszcz]

On Thu, 3 Jul 2008, Jon Nelson wrote:

>> Not true, Jon.  The test above was limited by the amount of data that a
>> single drive head can push to the controller with 100% sequential reads
>> of 64KB.  You can't say anything about chipset bottlenecks, because you
>> haven't created a condition where the chipset could even be a
>> bottleneck.  I/O is constrained by the disk drives.
>>
>> Now, if you attached industrial class SSDs that operate at media speeds
>> with access time in the NS range, then you could be in a position to
>> benchmark chipset performance.
>
> Well, that's true. However, perhaps I mis-spoke. What I was intending
> on doing is showing that the chipset was /not/ the limiting factor
> here, not checking it's maximum speed, but rather given what I know
> about the drives individually, and then what the rates are with all
> drives cookin', then I can say, "Well, at least the chipset or bus
> aren't getting in the way." Note that I also added "or bus" to my
> statement. Would you say that this statement is more accurate or am I
> getting the terminology wrong, again?
Also check my thread regarding the Veliciraptors, when I read 
concurrently off of two of them, I do not get that great of performance.
Whereas the ports on the motherboard are excellent!