pci cacheline size / latency oddness.

pci cacheline size / latency oddness.

[Dave Jones]

During boot of todays -git, I noticed this..

PCI: Setting latency timer of device 0000:00:1d.7 to 64

after boot, lspci shows..

00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) (prog-if 20 [EHCI])
Subsystem: Dell: Unknown device 0169
Flags: bus master, medium devsel, latency 0, IRQ 201
                                          ^^						

It also complains about..

PCI: cache line size of 128 is not supported by device 0000:00:1d.7

x86-64 doesn't have an arch override for pci_cache_line_size, so
it ends up at L1_CACHE_BYTES >> 2, which is 128 if you build
x86-64 kernels with CONFIG_GENERIC_CPU or CONFIG_MPSC
This means we will do the wrong thing on AMD machines which have
64 byte cachelines.   I saw this problem however on an em64t box.
Would it make sense to shift >> once more if it fails, and retry
with a smaller size perhaps ?

		Dave

Re: pci cacheline size / latency oddness.

[Jeff Garzik]

Dave Jones wrote:
> During boot of todays -git, I noticed this..
> 
> PCI: Setting latency timer of device 0000:00:1d.7 to 64
> 
> after boot, lspci shows..
> 
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) (prog-if 20 [EHCI])
> Subsystem: Dell: Unknown device 0169
> Flags: bus master, medium devsel, latency 0, IRQ 201
>                                           ^^						

Probably the hardware doesn't want you to set it, similar to what I 
describe in the following:


> It also complains about..
> 
> PCI: cache line size of 128 is not supported by device 0000:00:1d.7

This message means that it couldn't set the cacheline size at all.  Most 
likely it is either zero, or hardcoded in the silicon.  Has very little 
to do with the platform, and more to do with the device.


> x86-64 doesn't have an arch override for pci_cache_line_size, so
> it ends up at L1_CACHE_BYTES >> 2, which is 128 if you build
> x86-64 kernels with CONFIG_GENERIC_CPU or CONFIG_MPSC
> This means we will do the wrong thing on AMD machines which have
> 64 byte cachelines.   I saw this problem however on an em64t box.
> Would it make sense to shift >> once more if it fails, and retry
> with a smaller size perhaps ?

Too big is far better than too small.

	Jeff

Re: pci cacheline size / latency oddness.

[Parag Warudkar]

On Mon, 2005-08-01 at 19:35 -0400, Dave Jones wrote:
> This means we will do the wrong thing on AMD machines which have
> 64 byte cachelines.

pcibios_init (in i386/pci/common.c, which is linked in by X86_64 PCI
code) seems to do this 
 
if (c->x86 >= 6 && c->x86_vendor == X86_VENDOR_AMD)
                pci_cache_line_size = 64 >> 2;  /* K7 & K8 */

Is it correct to expect all AMD k7/8 machines to have 16 as cache line
size - I thought 64 was more appropriate?

On my Athlon64 laptop, all PCI devices end up having 0 latency. 

> x86-64 doesn't have an arch override for pci_cache_line_size

I am trying to fix it up - What's the right way to override it in x86_64
code?  Just initialize it to 64 may be?

Parag

Re: pci cacheline size / latency oddness.

[Andi Kleen]

On Mon, Aug 01, 2005 at 07:35:17PM -0400, Dave Jones wrote:
> During boot of todays -git, I noticed this..
> 
> PCI: Setting latency timer of device 0000:00:1d.7 to 64
> 
> after boot, lspci shows..
> 
> 00:1d.7 USB Controller: Intel Corporation 82801EB/ER (ICH5/ICH5R) USB2 EHCI Controller (rev 02) (prog-if 20 [EHCI])
> Subsystem: Dell: Unknown device 0169
> Flags: bus master, medium devsel, latency 0, IRQ 201
>                                           ^^						
> 
> It also complains about..
> 
> PCI: cache line size of 128 is not supported by device 0000:00:1d.7
> 
> x86-64 doesn't have an arch override for pci_cache_line_size, so
> it ends up at L1_CACHE_BYTES >> 2, which is 128 if you build
> x86-64 kernels with CONFIG_GENERIC_CPU or CONFIG_MPSC
> This means we will do the wrong thing on AMD machines which have
> 64 byte cachelines.   I saw this problem however on an em64t box.

We should be running arch/i386/pci/common.c:pcibios_init
As far as I can see that should do the right thing on x86-64 too.

> Would it make sense to shift >> once more if it fails, and retry
> with a smaller size perhaps ?

Not sure how much sense it makes to configure a PCI device
to a smaller cache line size than true. Best probably to leave
it alone in this case.

-Andi

Re: pci cacheline size / latency oddness.

[Andi Kleen]

On Mon, Aug 01, 2005 at 08:41:49PM -0400, Parag Warudkar wrote:
> On Mon, 2005-08-01 at 19:35 -0400, Dave Jones wrote:
> > This means we will do the wrong thing on AMD machines which have
> > 64 byte cachelines.
> 
> pcibios_init (in i386/pci/common.c, which is linked in by X86_64 PCI
> code) seems to do this 
>  
> if (c->x86 >= 6 && c->x86_vendor == X86_VENDOR_AMD)
>                 pci_cache_line_size = 64 >> 2;  /* K7 & K8 */
> 
> Is it correct to expect all AMD k7/8 machines to have 16 as cache line
> size - I thought 64 was more appropriate?

iirc the pci cache line register takes a value shifted left by 2 bits.

And yes all K7/K8 machines have 64byte cache lines.

> On my Athlon64 laptop, all PCI devices end up having 0 latency. 

That has nothing to do with the cache line size.

> 
> > x86-64 doesn't have an arch override for pci_cache_line_size
> 
> I am trying to fix it up - What's the right way to override it in x86_64
> code?  Just initialize it to 64 may be?

I don't think there is anything to fix.

-Andi