Re: Celistica with AMD chip-set

Re: Celistica with AMD chip-set

[Andi Kleen]

Alan Cox <alan@lxorguk.ukuu.org.uk> writes:

> On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
>> Hello all,
>> 
>> The Celistica server with the AMD chip-set has very poor
>> PCI performance with Linux (and probably W$ too).
>> 
>> The problem was traced to incorrect bridge configuration
>> in the HyperTransport(tm) chips that connect up pairs
>> of slots.

>    while((pdev = pci_find_device(0x1022, 0x7450, pdev)) != NULL)
>        pci_write_config_dword(pdev,  0x4c, 0x2ec1);


> Can you get Celestica to mail me their PCI subvendor
> id/devid's for the problem configuration or DMI strings
> and then we can do a PCI quirk properly for this.

Celistica is just the contract manufacturer for the AMD
reference design, they're really AMD designs. 0122/7450 
is the AMD 8131 PCI-X bridge.

I doubt it is a good idea to blindly change the configuration of the
bridge like this in the kernel.  It is probably better to wait for an
BIOS update or if you really need an urgent fix to do it from user
space. Any fix should probably read/change bit/write the register, not
blindly overwrite.

-Andi

Re: Celistica with AMD chip-set

[Richard B. Johnson]

On Mon, 30 Aug 2004, Andi Kleen wrote:

> Alan Cox <alan@lxorguk.ukuu.org.uk> writes:
>
> > On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
> >> Hello all,
> >>
> >> The Celistica server with the AMD chip-set has very poor
> >> PCI performance with Linux (and probably W$ too).
> >>
> >> The problem was traced to incorrect bridge configuration
> >> in the HyperTransport(tm) chips that connect up pairs
> >> of slots.
>
> >    while((pdev = pci_find_device(0x1022, 0x7450, pdev)) != NULL)
> >        pci_write_config_dword(pdev,  0x4c, 0x2ec1);
>
>
> > Can you get Celestica to mail me their PCI subvendor
> > id/devid's for the problem configuration or DMI strings
> > and then we can do a PCI quirk properly for this.
>
> Celistica is just the contract manufacturer for the AMD
> reference design, they're really AMD designs. 0122/7450
> is the AMD 8131 PCI-X bridge.
>
> I doubt it is a good idea to blindly change the configuration of the
> bridge like this in the kernel.  It is probably better to wait for an
> BIOS update or if you really need an urgent fix to do it from user
> space. Any fix should probably read/change bit/write the register, not
> blindly overwrite.
>
> -Andi
>
It is NOT "blindly" overwritten. We know what was in that register
(0x2c01).

Also, it is a "fix" for those who are getting crappy performance
from their boxes. Hopefully a new BIOS will be out with more
permanent fixes.


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Celistica with AMD chip-set

[Andi Kleen]

> It is NOT "blindly" overwritten. We know what was in that register
> (0x2c01).

You don't, it depends on the BIOS version and the hardware revision.

-Andi

Celistica with AMD chip-set

[Richard B. Johnson]

Hello all,

The Celistica server with the AMD chip-set has very poor
PCI performance with Linux (and probably W$ too).

The problem was traced to incorrect bridge configuration
in the HyperTransport(tm) chips that connect up pairs
of slots.

I don't know how to make a generic chip-set bug-fix, and
probably a new BIOS will come out with the correct values
in the registers. But right now, the following initialization
code will fix the box.



//-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
//
//  Some kind of problem, here. If the HyperTransport(tm) bridge
//  is found, one of the register values needs to be changed to
//  fix the bus performance. Need to turn back on prefetch.
//

    while((pdev = pci_find_device(0x1022, 0x7450, pdev)) != NULL)
        pci_write_config_dword(pdev,  0x4c, 0x2ec1);



Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
          "DMA used to run at 180 Mb/s. With the new hardware it's only
           30 Mb/s. Must be a software problem......." -actual complaint.
            Note 96.31% of all statistics are fiction.

Re: Celistica with AMD chip-set

[Alan Cox]

On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
> Hello all,
> 
> The Celistica server with the AMD chip-set has very poor
> PCI performance with Linux (and probably W$ too).
> 
> The problem was traced to incorrect bridge configuration
> in the HyperTransport(tm) chips that connect up pairs
> of slots.

Can you get Celestica to mail me their PCI subvendor
id/devid's for the problem configuration or DMI strings
and then we can do a PCI quirk properly for this.

Alan

Re: Celistica with AMD chip-set

[Arthur Perry]

Hello Alan and Richard,

I have to advise caution here, as it is currently unconfirmed whether or
not the PCI bridge configuration is "incorrect", and that it has "very
poor PCI performance".
Unless everyone in the whole wide world is setting this value and we are
the only ones who are not, I find it hard to believe that this statement
is not overspeculative.

The proper place for this should be in the BIOS, if it is indeed a true
optimization point.
But until that is positively identified, we should not assume that
applying this globally for everyone is the right thing to do.
As in any assumed optimization for a simgle case, it could potentially
cause performance degradation in somebody else's HBA.

This is a cache optimization.

Have you considered the possibility of this "optimization" causing a
performance hit with Mellanox's PCI implementation?

What about people who have already tailored their device driver to work
well in on this chipset and currently use "read multiple" rather than
"read cacheline". This optimization could potentially cause a slight
degradation of performance for them.

I just propose that we test this change with various card vendors and see
what the real impact is before we jump to the conclusion that this is a
serious performance problem for everybody.

Secondly, if it is the case, then the correct place to put this change is
in the system's BIOS, and having a software workaround is a last resort.

If you want, I can write a userspace utility to package with your existing
tools that can be installed and launched from init to provide this
optimization feature to the 8131 PCI bridge that your card resides on, to
ensure that your card gets this necessary optimization.
Or, you can easily put this capability into your existing device driver.

I would just rather not assume too much when dealing with something that
can potentially have a large reprocussion.






On Mon, 30 Aug 2004, Alan Cox wrote:

> On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
> > Hello all,
> >
> > The Celistica server with the AMD chip-set has very poor
> > PCI performance with Linux (and probably W$ too).
> >
> > The problem was traced to incorrect bridge configuration
> > in the HyperTransport(tm) chips that connect up pairs
> > of slots.
>
> Can you get Celestica to mail me their PCI subvendor
> id/devid's for the problem configuration or DMI strings
> and then we can do a PCI quirk properly for this.
>
> Alan
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>





Arthur Perry
Linux Systems/Software Architect
Lead Linux Engineer
CSU Validation Group
Celestica, Salem, NH
aperry@celestica.com

Re: Celistica with AMD chip-set

[Arthur Perry]

I believe that in the card's own PCI config space, there should be a
bit (bit 3) in the BAR configuration register which will tell "somebody" how to set up the device.
Bit 3 tells "somebody" whether or not it is prefetchable.
The bridge then be configured properly for this device.

I am not sure yet just who this should be, but my guess is that there should be a routine
in BIOS that does this.

So the first question I think is to verify that the card does indeed set this bit proper
for the BARs used.

If it is then not being set in the bridge, I would suspect a fault in BIOS.

Am I missing something here?
Please let me know if I am totally off track.


Thanks!




On Mon, 30 Aug 2004, Arthur Perry wrote:

> Hello Alan and Richard,
>
> I have to advise caution here, as it is currently unconfirmed whether or
> not the PCI bridge configuration is "incorrect", and that it has "very
> poor PCI performance".
> Unless everyone in the whole wide world is setting this value and we are
> the only ones who are not, I find it hard to believe that this statement
> is not overspeculative.
>
> The proper place for this should be in the BIOS, if it is indeed a true
> optimization point.
> But until that is positively identified, we should not assume that
> applying this globally for everyone is the right thing to do.
> As in any assumed optimization for a simgle case, it could potentially
> cause performance degradation in somebody else's HBA.
>
> This is a cache optimization.
>
> Have you considered the possibility of this "optimization" causing a
> performance hit with Mellanox's PCI implementation?
>
> What about people who have already tailored their device driver to work
> well in on this chipset and currently use "read multiple" rather than
> "read cacheline". This optimization could potentially cause a slight
> degradation of performance for them.
>
> I just propose that we test this change with various card vendors and see
> what the real impact is before we jump to the conclusion that this is a
> serious performance problem for everybody.
>
> Secondly, if it is the case, then the correct place to put this change is
> in the system's BIOS, and having a software workaround is a last resort.
>
> If you want, I can write a userspace utility to package with your existing
> tools that can be installed and launched from init to provide this
> optimization feature to the 8131 PCI bridge that your card resides on, to
> ensure that your card gets this necessary optimization.
> Or, you can easily put this capability into your existing device driver.
>
> I would just rather not assume too much when dealing with something that
> can potentially have a large reprocussion.
>
>
>
>
>
>
> On Mon, 30 Aug 2004, Alan Cox wrote:
>
> > On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
> > > Hello all,
> > >
> > > The Celistica server with the AMD chip-set has very poor
> > > PCI performance with Linux (and probably W$ too).
> > >
> > > The problem was traced to incorrect bridge configuration
> > > in the HyperTransport(tm) chips that connect up pairs
> > > of slots.
> >
> > Can you get Celestica to mail me their PCI subvendor
> > id/devid's for the problem configuration or DMI strings
> > and then we can do a PCI quirk properly for this.
> >
> > Alan
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>
>
>
>
> Arthur Perry
> Linux Systems/Software Architect
> Lead Linux Engineer
> CSU Validation Group
> Celestica, Salem, NH
> aperry@celestica.com
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


Arthur Perry
Linux Systems/Software Architect
Lead Linux Engineer
CSU Validation Group
Celestica, Salem, NH
aperry@celestica.com

Re: Celistica with AMD chip-set

[Richard B. Johnson]

On Mon, 30 Aug 2004, Arthur Perry wrote:

> I believe that in the card's own PCI config space, there should be a
> bit (bit 3) in the BAR configuration register which will tell
> "somebody" how to set up the device.
> Bit 3 tells "somebody" whether or not it is prefetchable.
> The bridge then be configured properly for this device.
>

No. The prefetch bit in the BAR is for the device, not for the
system RAM to which a DMA may be accomplished. Since DMA
occurs from a fixed address on the device, to a incrementing
address in RAM, the prefetch bit must NOT be set for this
BAR or else a cache-line is fetched and one one DWORD is
used, the rest thrown away.

> I am not sure yet just who this should be, but my guess is that there
> should be a routine
> in BIOS that does this.
>

The BIOS should configure bridges to be correct.

> So the first question I think is to verify that the card does indeed
> set this bit proper
> for the BARs used.
>

The card correctly sets its BARs.

> If it is then not being set in the bridge, I would suspect a fault in BIOS.
>
> Am I missing something here?
> Please let me know if I am totally off track.
>
>
> Thanks!
>
>
>
>
> On Mon, 30 Aug 2004, Arthur Perry wrote:
>
> > Hello Alan and Richard,
> >
> > I have to advise caution here, as it is currently unconfirmed whether or
> > not the PCI bridge configuration is "incorrect", and that it has "very
> > poor PCI performance".
> > Unless everyone in the whole wide world is setting this value and we are
> > the only ones who are not, I find it hard to believe that this statement
> > is not overspeculative.
> >
> > The proper place for this should be in the BIOS, if it is indeed a true
> > optimization point.
> > But until that is positively identified, we should not assume that
> > applying this globally for everyone is the right thing to do.
> > As in any assumed optimization for a simgle case, it could potentially
> > cause performance degradation in somebody else's HBA.
> >
> > This is a cache optimization.
> >
> > Have you considered the possibility of this "optimization" causing a
> > performance hit with Mellanox's PCI implementation?
> >
> > What about people who have already tailored their device driver to work
> > well in on this chipset and currently use "read multiple" rather than
> > "read cacheline". This optimization could potentially cause a slight
> > degradation of performance for them.
> >
> > I just propose that we test this change with various card vendors and see
> > what the real impact is before we jump to the conclusion that this is a
> > serious performance problem for everybody.
> >
> > Secondly, if it is the case, then the correct place to put this change is
> > in the system's BIOS, and having a software workaround is a last resort.
> >
> > If you want, I can write a userspace utility to package with your existing
> > tools that can be installed and launched from init to provide this
> > optimization feature to the 8131 PCI bridge that your card resides on, to
> > ensure that your card gets this necessary optimization.
> > Or, you can easily put this capability into your existing device driver.
> >
> > I would just rather not assume too much when dealing with something that
> > can potentially have a large reprocussion.
> >
> >
> > On Mon, 30 Aug 2004, Alan Cox wrote:
> >
> > > On Llu, 2004-08-30 at 15:02, Richard B. Johnson wrote:
> > > > Hello all,
> > > >
> > > > The Celistica server with the AMD chip-set has very poor
> > > > PCI performance with Linux (and probably W$ too).
> > > >
> > > > The problem was traced to incorrect bridge configuration
> > > > in the HyperTransport(tm) chips that connect up pairs
> > > > of slots.
> > >
> > > Can you get Celestica to mail me their PCI subvendor
> > > id/devid's for the problem configuration or DMI strings
> > > and then we can do a PCI quirk properly for this.
> > >
> > > Alan
> > >
> > > -
> > > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > > the body of a message to majordomo@vger.kernel.org
> > > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > > Please read the FAQ at  http://www.tux.org/lkml/
> > >
> >
> >
> >
> >
> >
> > Arthur Perry
> > Linux Systems/Software Architect
> > Lead Linux Engineer
> > CSU Validation Group
> > Celestica, Salem, NH
> > aperry@celestica.com
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
>
>
> Arthur Perry
> Linux Systems/Software Architect
> Lead Linux Engineer
> CSU Validation Group
> Celestica, Salem, NH
> aperry@celestica.com
>

Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Celistica with AMD chip-set

[Richard B. Johnson]

On Mon, 30 Aug 2004, Arthur Perry wrote:

> Hello Alan and Richard,
>
> I have to advise caution here, as it is currently unconfirmed whether or
> not the PCI bridge configuration is "incorrect", and that it has "very
> poor PCI performance".

The crapiest 33MHz, 32-bit PCI/Bus in lowly '486 machines hanging
around the lab will beat the Celistia hands-down.

> Unless everyone in the whole wide world is setting this value and we are
> the only ones who are not, I find it hard to believe that this statement
> is not overspeculative.
>

Really?  Well somebody from Salem New Hampshire wrote email to
one of my zillions of managers claiming that this was the "fix".

I was forced to make this "fix" and it "fixed" it. Further,
every "^!&@*$%^!_*(@" so called software, hardware, and whatever
"Engineer......" If that's the correct word, required (demanded)
to see the source code because they were "sure" that " had screwed
up.

To date, there has been no such finding of screw-ups on my part.
FYI, it's really difficult to screw up a "(@#%^P_*!@&#" DMA!

Other machines are able to DMA at over 130 megabytes/second. The
boxes in question run at only 50 megabytes/second.

> The proper place for this should be in the BIOS, if it is indeed a true
> optimization point.
> But until that is positively identified, we should not assume that
> applying this globally for everyone is the right thing to do.
> As in any assumed optimization for a simgle case, it could potentially
> cause performance degradation in somebody else's HBA.
>

Not so.

> This is a cache optimization.
>
> Have you considered the possibility of this "optimization" causing a
> performance hit with Mellanox's PCI implementation?
>

I published a "fix" for the abysimal PCI performance on that
piece of crap. If you don't like it then so what. Fix the
damn box.

> What about people who have already tailored their device driver to work
> well in on this chipset and currently use "read multiple" rather than
> "read cacheline". This optimization could potentially cause a slight
> degradation of performance for them.
>

I don't give a damn.  The box has no DMA capability as it is.
One might as well just use a wet string top communicate with
the PCI boards. The "fix" forced upon me by you guys is now
somehow incorrect?

Go to hell.

>
> Arthur Perry
> Linux Systems/Software Architect
> Lead Linux Engineer
> CSU Validation Group
> Celestica, Salem, NH
> aperry@celestica.com
>


Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: Celistica with AMD chip-set

[Arthur Perry]

Richard,

Look, I think you are taking this the wrong way.
I am not trying to banter you, I was just advising caution in
the manner which this "fix" was being implemented.
Perhaps I did not understand exactly how you were trying to implement it.

I apologise if I had belittled you in any way, as this was not my
intention. I know I sound somewhat caustic at times, but that is just my
writing style. I am not out to publicly humiliate you.
I do think however, that your response was rather unprofessional.

My argument was simply that I don't believe that we should have the kernel
set this prefetch register up, just because it is an 8131, or just because
it is a "Celestica" server.
Just because it works better for your card, does not mean it won't have
any kind of performance hit for other people's cards.
What would you do if this change was put into the kernel, assuming this
should be set, then down the line someone discovers that they have to set
it the other way to negate your change. I would think that would be rather
embarrasing.

I am just trying to take the more conservative approach and with testing,
determine if this is indeed the correct fix.

In the meantime, a simple 5-10 line shell script launched at boot time
could even provide the immediate "fix" that everyone is looking for.

And I do think you took that email from Celestica too personally.
I believe the intention of getting the source code was not to see if you
screwed up, but rather get the source needed to compile for different
platforms. I think we were only supplied with a 32 bit Fedora OS
from you guys, and whatever came on it, to test with.

Now if you don't mind, I think it would be in both of our best interest to
take the rest of this offline, seeing that things are getting as heated,
and there may be little value in this to the rest of the Linux
community at this phase.


Thanks.

Best Regards,
Art Perry




On Mon, 30 Aug 2004, Richard B. Johnson wrote:

> On Mon, 30 Aug 2004, Arthur Perry wrote:
>
> > Hello Alan and Richard,
> >
> > I have to advise caution here, as it is currently unconfirmed whether or
> > not the PCI bridge configuration is "incorrect", and that it has "very
> > poor PCI performance".
>
> The crapiest 33MHz, 32-bit PCI/Bus in lowly '486 machines hanging
> around the lab will beat the Celistia hands-down.
>
> > Unless everyone in the whole wide world is setting this value and we are
> > the only ones who are not, I find it hard to believe that this statement
> > is not overspeculative.
> >
>
> Really?  Well somebody from Salem New Hampshire wrote email to
> one of my zillions of managers claiming that this was the "fix".
>
> I was forced to make this "fix" and it "fixed" it. Further,
> every "^!&@*$%^!_*(@" so called software, hardware, and whatever
> "Engineer......" If that's the correct word, required (demanded)
> to see the source code because they were "sure" that " had screwed
> up.
>
> To date, there has been no such finding of screw-ups on my part.
> FYI, it's really difficult to screw up a "(@#%^P_*!@&#" DMA!
>
> Other machines are able to DMA at over 130 megabytes/second. The
> boxes in question run at only 50 megabytes/second.
>
> > The proper place for this should be in the BIOS, if it is indeed a true
> > optimization point.
> > But until that is positively identified, we should not assume that
> > applying this globally for everyone is the right thing to do.
> > As in any assumed optimization for a simgle case, it could potentially
> > cause performance degradation in somebody else's HBA.
> >
>
> Not so.
>
> > This is a cache optimization.
> >
> > Have you considered the possibility of this "optimization" causing a
> > performance hit with Mellanox's PCI implementation?
> >
>
> I published a "fix" for the abysimal PCI performance on that
> piece of crap. If you don't like it then so what. Fix the
> damn box.
>
> > What about people who have already tailored their device driver to work
> > well in on this chipset and currently use "read multiple" rather than
> > "read cacheline". This optimization could potentially cause a slight
> > degradation of performance for them.
> >
>
> I don't give a damn.  The box has no DMA capability as it is.
> One might as well just use a wet string top communicate with
> the PCI boards. The "fix" forced upon me by you guys is now
> somehow incorrect?
>
> Go to hell.
>
> >
> > Arthur Perry
> > Linux Systems/Software Architect
> > Lead Linux Engineer
> > CSU Validation Group
> > Celestica, Salem, NH
> > aperry@celestica.com
> >
>
>
> Cheers,
> Dick Johnson
> Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
>             Note 96.31% of all statistics are fiction.
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>