How to determine performance bottleneck?

How to determine performance bottleneck?

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1339 bytes --]

I'm playing around with performance tuning my disk access, but there is
something limiting my bandwidth to the disks. I was hoping you could
help me determine what.

My setup is a sil3132 controller, connected to a PMP with five disks
behind it. I'm using iostat to measure disk traffic and I'm using reads
via dd for testing.

When I'm accessing a single disk, the bandwidth is 70 - 80 MiB/s. When
I access a second disk, the bandwidth is about 50 MiB/s/disk, and all
five results in 25 MiB/s/disk. In other words, something is limiting
things to about 100 MiB/s. Now the question is what that limiting
factor is.

The PCIe bus can sustain 250 MiB/s, so even with overhead that should
be plenty. The SATA links are in theory 300 MiB/s, so that can't be it
either. The remaining factors are the controller and the multiplier
chip, and/or the way we access them.

Tejun, what kind of throughput have you seen when you have been testing
the sil3132 and multipliers?

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to determine performance bottleneck?

[Alan Cox]

> five results in 25 MiB/s/disk. In other words, something is limiting
> things to about 100 MiB/s. Now the question is what that limiting
> factor is.

Whats the PCIe bridge chip and what is the memory bandwidth of the
system ?

Re: How to determine performance bottleneck?

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 927 bytes --]

On Mon, 13 Oct 2008 21:12:02 +0100
Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:

> > five results in 25 MiB/s/disk. In other words, something is limiting
> > things to about 100 MiB/s. Now the question is what that limiting
> > factor is.
> 
> Whats the PCIe bridge chip and what is the memory bandwidth of the
> system ?
> 

No idea. The board uses a AMD 690G/SB600 chipset (unsure which part
houses the PCIe bridge), and the memory is two PC6400 modules. The
processor is a Athlon 4050e (did AMD ever put the dual lane capabilty
in the non-opteron models?).

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to determine performance bottleneck?

[Tejun Heo]

Pierre Ossman wrote:
> I'm playing around with performance tuning my disk access, but there is
> something limiting my bandwidth to the disks. I was hoping you could
> help me determine what.
> 
> My setup is a sil3132 controller, connected to a PMP with five disks
> behind it. I'm using iostat to measure disk traffic and I'm using reads
> via dd for testing.
> 
> When I'm accessing a single disk, the bandwidth is 70 - 80 MiB/s. When
> I access a second disk, the bandwidth is about 50 MiB/s/disk, and all
> five results in 25 MiB/s/disk. In other words, something is limiting
> things to about 100 MiB/s. Now the question is what that limiting
> factor is.
> 
> The PCIe bus can sustain 250 MiB/s, so even with overhead that should
> be plenty. The SATA links are in theory 300 MiB/s, so that can't be it
> either. The remaining factors are the controller and the multiplier
> chip, and/or the way we access them.
> 
> Tejun, what kind of throughput have you seen when you have been testing
> the sil3132 and multipliers?

That's the limit of sil3124/3132.  Dunno why but you can't go over that.
 There's 3132-2 chip on the market.  Maybe it's worth a try?

-- 
tejun

Re: How to determine performance bottleneck?

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1215 bytes --]

On Tue, 14 Oct 2008 08:25:22 +0900
Tejun Heo <tj@kernel.org> wrote:

> Pierre Ossman wrote:
> > When I'm accessing a single disk, the bandwidth is 70 - 80 MiB/s. When
> > I access a second disk, the bandwidth is about 50 MiB/s/disk, and all
> > five results in 25 MiB/s/disk. In other words, something is limiting
> > things to about 100 MiB/s. Now the question is what that limiting
> > factor is.
> > 
> 
> That's the limit of sil3124/3132.  Dunno why but you can't go over that.

Bummer. Is this per port, or is it the PCIe side of the chip that is
limited to this speed?

What do the silicon image guys have to say about it?

>  There's 3132-2 chip on the market.  Maybe it's worth a try?
> 

Now that model number was not something that was easily found on
google, nor on silicon image's website. Know of any boards with that
chip?

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to determine performance bottleneck?

[Tejun Heo]

Pierre Ossman wrote:
> On Tue, 14 Oct 2008 08:25:22 +0900
> Tejun Heo <tj@kernel.org> wrote:
> 
>> Pierre Ossman wrote:
>>> When I'm accessing a single disk, the bandwidth is 70 - 80 MiB/s. When
>>> I access a second disk, the bandwidth is about 50 MiB/s/disk, and all
>>> five results in 25 MiB/s/disk. In other words, something is limiting
>>> things to about 100 MiB/s. Now the question is what that limiting
>>> factor is.
>>>
>> That's the limit of sil3124/3132.  Dunno why but you can't go over that.
> 
> Bummer. Is this per port, or is it the PCIe side of the chip that is
> limited to this speed?

I think it's on the host bus side.

> What do the silicon image guys have to say about it?

ISTR they confirmed the problem but I don't remember talking too much
about it.

>>  There's 3132-2 chip on the market.  Maybe it's worth a try?
>
> Now that model number was not something that was easily found on
> google, nor on silicon image's website. Know of any boards with that
> chip?

Somebody wrote me an email about it.  Looking up... eh.. can't find.  Sorry.

-- 
tejun

Re: How to determine performance bottleneck?

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

On Tue, 14 Oct 2008 14:37:41 +0900
Tejun Heo <tj@kernel.org> wrote:

> Pierre Ossman wrote:
> > 
> > Bummer. Is this per port, or is it the PCIe side of the chip that is
> > limited to this speed?
> 
> I think it's on the host bus side.
> 
> > What do the silicon image guys have to say about it?
> 
> ISTR they confirmed the problem but I don't remember talking too much
> about it.
> 

The optimist in me was hoping that this could be a problem that can be
sidestepped with a clever enough implementation. :)

> >>  There's 3132-2 chip on the market.  Maybe it's worth a try?
> >
> > Now that model number was not something that was easily found on
> > google, nor on silicon image's website. Know of any boards with that
> > chip?
> 
> Somebody wrote me an email about it.  Looking up... eh.. can't find.  Sorry.
> 

Do you remember if that's a new chip designation, or 3132 (rev 02)?
Just so I know what to keep an eye out for.

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to determine performance bottleneck?

[Tejun Heo]

Pierre Ossman wrote:
> On Tue, 14 Oct 2008 14:37:41 +0900
> Tejun Heo <tj@kernel.org> wrote:
> 
>> Pierre Ossman wrote:
>>> Bummer. Is this per port, or is it the PCIe side of the chip that is
>>> limited to this speed?
>> I think it's on the host bus side.
>>
>>> What do the silicon image guys have to say about it?
>> ISTR they confirmed the problem but I don't remember talking too much
>> about it.
>>
> 
> The optimist in me was hoping that this could be a problem that can be
> sidestepped with a clever enough implementation. :)
> 
>>>>  There's 3132-2 chip on the market.  Maybe it's worth a try?
>>> Now that model number was not something that was easily found on
>>> google, nor on silicon image's website. Know of any boards with that
>>> chip?
>> Somebody wrote me an email about it.  Looking up... eh.. can't find.  Sorry.
>>
> 
> Do you remember if that's a new chip designation, or 3132 (rev 02)?
> Just so I know what to keep an eye out for.

Yeah, it had a new designation.  Something like 3132-2 (don't confuse it
with 3132 2 ports or 3132 2 ports SATA 2, yeah a lot of 2s there).

-- 
tejun

Re: How to determine performance bottleneck?

[Pierre Ossman]

[-- Attachment #1: Type: text/plain, Size: 913 bytes --]

On Tue, 14 Oct 2008 15:12:20 +0900
Tejun Heo <tj@kernel.org> wrote:

> Pierre Ossman wrote:
> > 
> > Do you remember if that's a new chip designation, or 3132 (rev 02)?
> > Just so I know what to keep an eye out for.
> 
> Yeah, it had a new designation.  Something like 3132-2 (don't confuse it
> with 3132 2 ports or 3132 2 ports SATA 2, yeah a lot of 2s there).
> 

Ok. Thanks for all the info. :)

Any objections to putting this on the wiki, so the next person having
less than expected performance doesn't have to ask on the mailing list?

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  rdesktop, core developer          http://www.rdesktop.org

  WARNING: This correspondence is being monitored by the
  Swedish government. Make sure your server uses encryption
  for SMTP traffic and consider using PGP for end-to-end
  encryption.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: How to determine performance bottleneck?

[Tejun Heo]

Pierre Ossman wrote:
> On Tue, 14 Oct 2008 15:12:20 +0900
> Tejun Heo <tj@kernel.org> wrote:
> 
>> Pierre Ossman wrote:
>>> Do you remember if that's a new chip designation, or 3132 (rev 02)?
>>> Just so I know what to keep an eye out for.
>> Yeah, it had a new designation.  Something like 3132-2 (don't confuse it
>> with 3132 2 ports or 3132 2 ports SATA 2, yeah a lot of 2s there).
>>
> 
> Ok. Thanks for all the info. :)
> 
> Any objections to putting this on the wiki, so the next person having
> less than expected performance doesn't have to ask on the mailing list?

Hmmm... Let me check this with SIMG first so that I have more concrete
idea about which ones have what kind of limit and whether there are any
which don't have such limitations.

Thanks.

-- 
tejun

Re: How to determine performance bottleneck?

[Grant Grundler]

On Mon, Oct 13, 2008 at 4:25 PM, Tejun Heo <tj@kernel.org> wrote:
...
>> Tejun, what kind of throughput have you seen when you have been testing
>> the sil3132 and multipliers?
>
> That's the limit of sil3124/3132.  Dunno why but you can't go over that.

To be clear, I understood "that" ==  ~100 MB/s

>  There's 3132-2 chip on the market.  Maybe it's worth a try?

Are you possibly thinking of 3124-2 ? (4-port PCI-X device, not PCI-e)

I didn't see mention of 3132-2 on
http://www.siliconimage.com/products/product.aspx?id=32

thanks,
grant

Re: How to determine performance bottleneck?

[Tejun Heo]

Grant Grundler wrote:
> On Mon, Oct 13, 2008 at 4:25 PM, Tejun Heo <tj@kernel.org> wrote:
> ...
>>> Tejun, what kind of throughput have you seen when you have been testing
>>> the sil3132 and multipliers?
>> That's the limit of sil3124/3132.  Dunno why but you can't go over that.
> 
> To be clear, I understood "that" ==  ~100 MB/s

It's somewhere around that.  I don't remember the exact number at the
moment.

>>  There's 3132-2 chip on the market.  Maybe it's worth a try?
> 
> Are you possibly thinking of 3124-2 ? (4-port PCI-X device, not PCI-e)
> 
> I didn't see mention of 3132-2 on
> http://www.siliconimage.com/products/product.aspx?id=32

I asked SIMG about the issue and whether they have newer revisions which
solve the issue.  When I know more, I'll post it on the ata wiki page.

Thanks.

-- 
tejun

Re: How to determine performance bottleneck?

[Grant Grundler]

On Tue, Oct 14, 2008 at 10:19 AM, Tejun Heo <tj@kernel.org> wrote:
> Grant Grundler wrote:
>> On Mon, Oct 13, 2008 at 4:25 PM, Tejun Heo <tj@kernel.org> wrote:
>> ...
>>>> Tejun, what kind of throughput have you seen when you have been testing
>>>> the sil3132 and multipliers?
>>> That's the limit of sil3124/3132.  Dunno why but you can't go over that.
>>
>> To be clear, I understood "that" ==  ~100 MB/s
>
> It's somewhere around that.  I don't remember the exact number at the
> moment.

I know a number but I'm also burdened with an NDA.

If someone could post the output of this command:
fio-1.21 --runtime=60 --time_based --bwavgtime=5000 \
        --thread --numjobs=4 \
        -rw=read --direct=1 --iodepth=8 --ioengine=sync \
        --ioscheduler=noop \
        --bs=256k --size=256k \
        --name=port1 --filename=/dev/sdb \
        --name=port2 --filename=/dev/sdc

Fix up the --filename to be the correct /dev names.
Re-run the above with "--rw=write" and make sure the disks can be clobbered!

This will "exercise" the disk controller cache so it doesn't matter
which disk is connected.

For fun, use 4k instead of 256k to measure CPU utilization and
transaction rates.
In this case, the disk models vary alot.

...
> I asked SIMG about the issue and whether they have newer revisions which
> solve the issue.  When I know more, I'll post it on the ata wiki page.

thanks!
grant