PCIe issue with NIC card that has 64bit BARs

PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

Bjorn,

I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
recent SoC (K2G) that re-uses the same driver and hardware, I have issues
bringing up PCIe.

The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
controller and that on K2G with rtk8168. Is there way we can get it assigned
32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.

Here are the logs.

K2E log with Marvel controller (Good working case).
===================================================
[    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
[    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
[    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
[    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
[    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
[    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
[    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
[    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
[    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
[    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
[    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]

K2G log with Tealtek NIC card
=============================
[    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.337567] PCI: bus0: Fast back to back transfers disabled
[    2.361159] PCI: bus1: Fast back to back transfers disabled
[    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
[    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
[    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
[    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Bjorn Helgaas]

Hi Murali,

On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:
> Bjorn,
> 
> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> bringing up PCIe.
> 
> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> controller and that on K2G with rtk8168. Is there way we can get it assigned
> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> 
> Here are the logs.
> 
> K2E log with Marvel controller (Good working case).
> ===================================================
> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> 
> K2G log with Tealtek NIC card
> =============================
> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]

There's no "(bus address ...)" annotation here, which means these
windows map CPU addresses to identical bus addresses.  The
"[mem 0x50000000-0x5fffffff]" window is a 32-bit window.

> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]

I assume these BARs (2 and 4) are the ones in question.  They are
64-bit BARs, but the addresses currently assigned to them fit in 32
bits, so the space is already all below 4GB.

The BAR type (32-bit or 64-bit) is built into the device; Linux
doesn't have any influence on that.  The type is encoded in the
low-order four bits of the BAR, which are read-only.

> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286

Looks like this is printed by rtl_init_one().  I see that we read
zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
Could it be that some EEPROM on the card hasn't been programmed yet?
Does the card work if you put it in a different machine?

> [    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

Hi Bjorn,

Thanks for your quick response!
See below for some follow up question.

On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:
> Hi Murali,
> 
> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:
>> Bjorn,
>>
>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>> bringing up PCIe.
>>
>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>
>> Here are the logs.
>>
>> K2E log with Marvel controller (Good working case).
>> ===================================================
>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>
>> K2G log with Tealtek NIC card
>> =============================
>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
> 
> There's no "(bus address ...)" annotation here, which means these
> windows map CPU addresses to identical bus addresses.  The
> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> 
>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
> 
> I assume these BARs (2 and 4) are the ones in question.  They are
> 64-bit BARs, but the addresses currently assigned to them fit in 32
> bits, so the space is already all below 4GB.
> 
> The BAR type (32-bit or 64-bit) is built into the device; Linux
> doesn't have any influence on that.  The type is encoded in the
> low-order four bits of the BAR, which are read-only.
> 
>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
> 
> Looks like this is printed by rtl_init_one().  I see that we read
> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
So if device specific configuration registers for this device are mapped
to the window to have 64bit access, they are to be accessed with proper
address to get the lower 32 bits of the data, right? So is there a chance
it is making access to upper 4 bytes and hence reading all zeros?
 
> Could it be that some EEPROM on the card hasn't been programmed yet?
> Does the card work if you put it in a different machine?
> 
Probably. I will ask this to the owner of r8169 driver. He seems to think
the access is not right.  I need to find a PC that has PCIe slot to do this
test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
driver in Linux? 

Thanks 

Murali

>> [    2.478169] r8169 0000:01:00.0 eth0: jumbo features [frames: 7152 bytes, tx checksumming: ok]


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Alex Williamson]

On Mon, 9 May 2016 17:02:23 -0400
Murali Karicheri <m-karicheri2@ti.com> wrote:

> Hi Bjorn,
> 
> Thanks for your quick response!
> See below for some follow up question.
> 
> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:
> > Hi Murali,
> > 
> > On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:  
> >> Bjorn,
> >>
> >> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> >> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> >> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> >> bringing up PCIe.
> >>
> >> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> >> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> >> controller and that on K2G with rtk8168. Is there way we can get it assigned
> >> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> >>
> >> Here are the logs.
> >>
> >> K2E log with Marvel controller (Good working case).
> >> ===================================================
> >> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> >> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> >> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> >> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> >> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> >> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> >> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> >> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> >> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> >> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> >> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> >> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> >>
> >> K2G log with Tealtek NIC card
> >> =============================
> >> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> >> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> >> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> >> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]  
> > 
> > There's no "(bus address ...)" annotation here, which means these
> > windows map CPU addresses to identical bus addresses.  The
> > "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> >   
> >> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> >> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> >> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> >> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> >> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> >> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> >> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]  
> > 
> > I assume these BARs (2 and 4) are the ones in question.  They are
> > 64-bit BARs, but the addresses currently assigned to them fit in 32
> > bits, so the space is already all below 4GB.
> > 
> > The BAR type (32-bit or 64-bit) is built into the device; Linux
> > doesn't have any influence on that.  The type is encoded in the
> > low-order four bits of the BAR, which are read-only.
> >   
> >> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> >> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> >> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> >> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> >> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> >> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> >> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> >> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> >> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286  
> > 
> > Looks like this is printed by rtl_init_one().  I see that we read
> > zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.  
> So if device specific configuration registers for this device are mapped
> to the window to have 64bit access, they are to be accessed with proper
> address to get the lower 32 bits of the data, right? So is there a chance
> it is making access to upper 4 bytes and hence reading all zeros?
>  
> > Could it be that some EEPROM on the card hasn't been programmed yet?
> > Does the card work if you put it in a different machine?
> >   
> Probably. I will ask this to the owner of r8169 driver. He seems to think
> the access is not right.  I need to find a PC that has PCIe slot to do this
> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
> driver in Linux? 

I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
Adapter) that only has 32bit BARs:

02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e800 [size=32]
02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
	Region 2: I/O ports at e400 [size=32]

Thanks,
Alex

Re: PCIe issue with NIC card that has 64bit BARs

[Alex Williamson]

On Mon, 9 May 2016 15:23:51 -0600
Alex Williamson <alex.williamson@redhat.com> wrote:

> On Mon, 9 May 2016 17:02:23 -0400
> Murali Karicheri <m-karicheri2@ti.com> wrote:
> 
> > Hi Bjorn,
> > 
> > Thanks for your quick response!
> > See below for some follow up question.
> > 
> > On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
> > > Hi Murali,
> > > 
> > > On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
> > >> Bjorn,
> > >>
> > >> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> > >> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> > >> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> > >> bringing up PCIe.
> > >>
> > >> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> > >> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> > >> controller and that on K2G with rtk8168. Is there way we can get it assigned
> > >> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> > >>
> > >> Here are the logs.
> > >>
> > >> K2E log with Marvel controller (Good working case).
> > >> ===================================================
> > >> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> > >> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> > >> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> > >> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> > >> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> > >> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> > >> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> > >> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> > >> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> > >> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> > >> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> > >> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> > >> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> > >> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> > >>
> > >> K2G log with Tealtek NIC card
> > >> =============================
> > >> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> > >> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> > >> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> > >> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
> > > 
> > > There's no "(bus address ...)" annotation here, which means these
> > > windows map CPU addresses to identical bus addresses.  The
> > > "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> > >     
> > >> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> > >> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> > >> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> > >> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> > >> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> > >> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> > >> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> > >> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
> > > 
> > > I assume these BARs (2 and 4) are the ones in question.  They are
> > > 64-bit BARs, but the addresses currently assigned to them fit in 32
> > > bits, so the space is already all below 4GB.
> > > 
> > > The BAR type (32-bit or 64-bit) is built into the device; Linux
> > > doesn't have any influence on that.  The type is encoded in the
> > > low-order four bits of the BAR, which are read-only.
> > >     
> > >> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> > >> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> > >> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> > >> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> > >> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> > >> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> > >> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> > >> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> > >> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> > >> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
> > > 
> > > Looks like this is printed by rtl_init_one().  I see that we read
> > > zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
> > So if device specific configuration registers for this device are mapped
> > to the window to have 64bit access, they are to be accessed with proper
> > address to get the lower 32 bits of the data, right? So is there a chance
> > it is making access to upper 4 bytes and hence reading all zeros?
> >    
> > > Could it be that some EEPROM on the card hasn't been programmed yet?
> > > Does the card work if you put it in a different machine?
> > >     
> > Probably. I will ask this to the owner of r8169 driver. He seems to think
> > the access is not right.  I need to find a PC that has PCIe slot to do this
> > test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
> > driver in Linux?   
> 
> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
> Adapter) that only has 32bit BARs:
> 
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 2: I/O ports at e800 [size=32]
> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
> 	Region 2: I/O ports at e400 [size=32]

Intel has lots apparently, 82574L, 92579LM, I218-LM and even an 82576.
IOW grab a random Intel 1G NIC and it probably has 32bit BARs.

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

On 05/09/2016 05:23 PM, Alex Williamson wrote:
> On Mon, 9 May 2016 17:02:23 -0400
> Murali Karicheri <m-karicheri2@ti.com> wrote:
> 
>> Hi Bjorn,
>>
>> Thanks for your quick response!
>> See below for some follow up question.
>>
>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:
>>> Hi Murali,
>>>
>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:  
>>>> Bjorn,
>>>>
>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>>>> bringing up PCIe.
>>>>
>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>>>
>>>> Here are the logs.
>>>>
>>>> K2E log with Marvel controller (Good working case).
>>>> ===================================================
>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>>>
>>>> K2G log with Tealtek NIC card
>>>> =============================
>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]  
>>>
>>> There's no "(bus address ...)" annotation here, which means these
>>> windows map CPU addresses to identical bus addresses.  The
>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
>>>   
>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]  
>>>
>>> I assume these BARs (2 and 4) are the ones in question.  They are
>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
>>> bits, so the space is already all below 4GB.
>>>
>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
>>> doesn't have any influence on that.  The type is encoded in the
>>> low-order four bits of the BAR, which are read-only.
>>>   
>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286  
>>>
>>> Looks like this is printed by rtl_init_one().  I see that we read
>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.  
>> So if device specific configuration registers for this device are mapped
>> to the window to have 64bit access, they are to be accessed with proper
>> address to get the lower 32 bits of the data, right? So is there a chance
>> it is making access to upper 4 bytes and hence reading all zeros?
>>  
>>> Could it be that some EEPROM on the card hasn't been programmed yet?
>>> Does the card work if you put it in a different machine?
>>>   
>> Probably. I will ask this to the owner of r8169 driver. He seems to think
>> the access is not right.  I need to find a PC that has PCIe slot to do this
>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
>> driver in Linux? 
> 
> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
> Adapter) that only has 32bit BARs:
> 
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 2: I/O ports at e800 [size=32]
> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
> 	Region 2: I/O ports at e400 [size=32]
Alex,

Thanks for the response. I see following mini pci card which match with your
description.

http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt

Is this same thing? I assume e1000e/82571.c is the driver for this.
Could you please confirm so that I can procure the same?

Murali
> 
> Thanks,
> Alex
> 


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Alex Williamson]

On Mon, 9 May 2016 17:38:27 -0400
Murali Karicheri <m-karicheri2@ti.com> wrote:

> On 05/09/2016 05:23 PM, Alex Williamson wrote:
> > On Mon, 9 May 2016 17:02:23 -0400
> > Murali Karicheri <m-karicheri2@ti.com> wrote:
> >   
> >> Hi Bjorn,
> >>
> >> Thanks for your quick response!
> >> See below for some follow up question.
> >>
> >> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
> >>> Hi Murali,
> >>>
> >>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
> >>>> Bjorn,
> >>>>
> >>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> >>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> >>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> >>>> bringing up PCIe.
> >>>>
> >>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> >>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> >>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
> >>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> >>>>
> >>>> Here are the logs.
> >>>>
> >>>> K2E log with Marvel controller (Good working case).
> >>>> ===================================================
> >>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> >>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> >>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> >>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> >>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> >>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> >>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> >>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> >>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> >>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> >>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> >>>>
> >>>> K2G log with Tealtek NIC card
> >>>> =============================
> >>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> >>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> >>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> >>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
> >>>
> >>> There's no "(bus address ...)" annotation here, which means these
> >>> windows map CPU addresses to identical bus addresses.  The
> >>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> >>>     
> >>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> >>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> >>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> >>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> >>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> >>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> >>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
> >>>
> >>> I assume these BARs (2 and 4) are the ones in question.  They are
> >>> 64-bit BARs, but the addresses currently assigned to them fit in 32
> >>> bits, so the space is already all below 4GB.
> >>>
> >>> The BAR type (32-bit or 64-bit) is built into the device; Linux
> >>> doesn't have any influence on that.  The type is encoded in the
> >>> low-order four bits of the BAR, which are read-only.
> >>>     
> >>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> >>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> >>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> >>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> >>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> >>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> >>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> >>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
> >>>
> >>> Looks like this is printed by rtl_init_one().  I see that we read
> >>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
> >> So if device specific configuration registers for this device are mapped
> >> to the window to have 64bit access, they are to be accessed with proper
> >> address to get the lower 32 bits of the data, right? So is there a chance
> >> it is making access to upper 4 bytes and hence reading all zeros?
> >>    
> >>> Could it be that some EEPROM on the card hasn't been programmed yet?
> >>> Does the card work if you put it in a different machine?
> >>>     
> >> Probably. I will ask this to the owner of r8169 driver. He seems to think
> >> the access is not right.  I need to find a PC that has PCIe slot to do this
> >> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
> >> driver in Linux?   
> > 
> > I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
> > Adapter) that only has 32bit BARs:
> > 
> > 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> > 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
> > 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
> > 	Region 2: I/O ports at e800 [size=32]
> > 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> > 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
> > 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
> > 	Region 2: I/O ports at e400 [size=32]  
> Alex,
> 
> Thanks for the response. I see following mini pci card which match with your
> description.
> 
> http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
> 
> Is this same thing? I assume e1000e/82571.c is the driver for this.
> Could you please confirm so that I can procure the same?

I would venture to say yes, but I offer no guarantees ;)

Ark seems to indicate all 1000 PT dual port cards use that chip:
http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

On 05/09/2016 07:12 PM, Alex Williamson wrote:
> On Mon, 9 May 2016 17:38:27 -0400
> Murali Karicheri <m-karicheri2@ti.com> wrote:
> 
>> On 05/09/2016 05:23 PM, Alex Williamson wrote:
>>> On Mon, 9 May 2016 17:02:23 -0400
>>> Murali Karicheri <m-karicheri2@ti.com> wrote:
>>>   
>>>> Hi Bjorn,
>>>>
>>>> Thanks for your quick response!
>>>> See below for some follow up question.
>>>>
>>>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
>>>>> Hi Murali,
>>>>>
>>>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
>>>>>> Bjorn,
>>>>>>
>>>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>>>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>>>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>>>>>> bringing up PCIe.
>>>>>>
>>>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>>>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>>>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>>>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>>>>>
>>>>>> Here are the logs.
>>>>>>
>>>>>> K2E log with Marvel controller (Good working case).
>>>>>> ===================================================
>>>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>>>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>>>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>>>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>>>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>>>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>>>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>>>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>>>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>>>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>>>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>>>>>
>>>>>> K2G log with Tealtek NIC card
>>>>>> =============================
>>>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>>>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
>>>>>
>>>>> There's no "(bus address ...)" annotation here, which means these
>>>>> windows map CPU addresses to identical bus addresses.  The
>>>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
>>>>>     
>>>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>>>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>>>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>>>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>>>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>>>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
>>>>>
>>>>> I assume these BARs (2 and 4) are the ones in question.  They are
>>>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
>>>>> bits, so the space is already all below 4GB.
>>>>>
>>>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
>>>>> doesn't have any influence on that.  The type is encoded in the
>>>>> low-order four bits of the BAR, which are read-only.
>>>>>     
>>>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>>>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>>>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>>>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
>>>>>
>>>>> Looks like this is printed by rtl_init_one().  I see that we read
>>>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
>>>> So if device specific configuration registers for this device are mapped
>>>> to the window to have 64bit access, they are to be accessed with proper
>>>> address to get the lower 32 bits of the data, right? So is there a chance
>>>> it is making access to upper 4 bytes and hence reading all zeros?
>>>>    
>>>>> Could it be that some EEPROM on the card hasn't been programmed yet?
>>>>> Does the card work if you put it in a different machine?
>>>>>     
>>>> Probably. I will ask this to the owner of r8169 driver. He seems to think
>>>> the access is not right.  I need to find a PC that has PCIe slot to do this
>>>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
>>>> driver in Linux?   
>>>
>>> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
>>> Adapter) that only has 32bit BARs:
>>>
>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 2: I/O ports at e800 [size=32]
>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 2: I/O ports at e400 [size=32]  
>> Alex,
>>
>> Thanks for the response. I see following mini pci card which match with your
>> description.
>>
>> http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
>>
>> Is this same thing? I assume e1000e/82571.c is the driver for this.
>> Could you please confirm so that I can procure the same?
> 
> I would venture to say yes, but I offer no guarantees ;)
> 
> Ark seems to indicate all 1000 PT dual port cards use that chip:
> http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
> 
Thanks Alex!

Murali

-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

Bjorn, Alex,

On 05/09/2016 07:12 PM, Alex Williamson wrote:
> On Mon, 9 May 2016 17:38:27 -0400
> Murali Karicheri <m-karicheri2@ti.com> wrote:
> 
>> On 05/09/2016 05:23 PM, Alex Williamson wrote:
>>> On Mon, 9 May 2016 17:02:23 -0400
>>> Murali Karicheri <m-karicheri2@ti.com> wrote:
>>>   
>>>> Hi Bjorn,
>>>>
>>>> Thanks for your quick response!
>>>> See below for some follow up question.
>>>>
>>>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
>>>>> Hi Murali,
>>>>>
>>>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
>>>>>> Bjorn,
>>>>>>
>>>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>>>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>>>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>>>>>> bringing up PCIe.
>>>>>>
>>>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>>>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>>>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>>>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>>>>>
>>>>>> Here are the logs.
>>>>>>
>>>>>> K2E log with Marvel controller (Good working case).
>>>>>> ===================================================
>>>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>>>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>>>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>>>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>>>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>>>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>>>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>>>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>>>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>>>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>>>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>>>>>
>>>>>> K2G log with Tealtek NIC card
>>>>>> =============================
>>>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>>>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
>>>>>
>>>>> There's no "(bus address ...)" annotation here, which means these
>>>>> windows map CPU addresses to identical bus addresses.  The
>>>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
>>>>>     
>>>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>>>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>>>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>>>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>>>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>>>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
>>>>>
>>>>> I assume these BARs (2 and 4) are the ones in question.  They are
>>>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
>>>>> bits, so the space is already all below 4GB.
>>>>>
>>>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
>>>>> doesn't have any influence on that.  The type is encoded in the
>>>>> low-order four bits of the BAR, which are read-only.
>>>>>     
>>>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>>>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>>>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>>>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
>>>>>
>>>>> Looks like this is printed by rtl_init_one().  I see that we read
>>>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
>>>> So if device specific configuration registers for this device are mapped
>>>> to the window to have 64bit access, they are to be accessed with proper
>>>> address to get the lower 32 bits of the data, right? So is there a chance
>>>> it is making access to upper 4 bytes and hence reading all zeros?
>>>>    
>>>>> Could it be that some EEPROM on the card hasn't been programmed yet?
>>>>> Does the card work if you put it in a different machine?
>>>>>     
>>>> Probably. I will ask this to the owner of r8169 driver. He seems to think
>>>> the access is not right.  I need to find a PC that has PCIe slot to do this
>>>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
>>>> driver in Linux?   
>>>
>>> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
>>> Adapter) that only has 32bit BARs:
>>>
>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 2: I/O ports at e800 [size=32]
>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
>>> 	Region 2: I/O ports at e400 [size=32]  
>> Alex,
>>
>> Thanks for the response. I see following mini pci card which match with your
>> description.
>>
>> http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
>>
>> Is this same thing? I assume e1000e/82571.c is the driver for this.
>> Could you please confirm so that I can procure the same?
> 
> I would venture to say yes, but I offer no guarantees ;)
> 
> Ark seems to indicate all 1000 PT dual port cards use that chip:
> http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
> 
I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC
and I am getting the following error:- This one has a 32bit bar, but similar
issue as with r8169 and tg3. The PCI access over the window seems to return zeros.
Per suggestion, tried dumping the memory window and got all zeros. See below
for additional debug info.

root@k2g-evm:/# insmod e1000e.ko debug=16
[  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
[  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
[  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
[  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
[  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5

root@k2g-evm:/# lspci -vvv
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 287
        Region 0: Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 50000000-500fffff
        Prefetchable memory behind bridge: 50100000-501fffff
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
        BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
                Address: 0000000021800054  Data: 0000
        Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 256 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
                        ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
                RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
                RootCap: CRSVisible-
                RootSta: PME ReqID 0000, PMEStatus- PMEPending-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
        Subsystem: Intel Corporation PRO/1000 PT Server Adapter
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 319
        Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
        Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
        Region 2: I/O ports at 1000 [disabled] [size=32]
        [virtual] Expansion ROM at 50100000 [disabled] [size=128K]
        Capabilities: [c8] Power Management version 2
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000021800054  Data: 0001
        Capabilities: [e0] Express (v1) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2

Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with
Broadcom tg3 and 

root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
50000000:00000000 00000000 00000000 00000000
50000010:00000000 00000000 00000000 00000000
50000020:00000000 00000000 00000000 00000000
50000030:00000000 00000000 00000000 00000000
50000040:00000000 00000000 00000000 00000000
50000050:00000000 00000000 00000000 00000000
50000060:00000000 00000000 00000000 00000000
50000070:00000000 00000000 00000000 00000000
50000080:00000000 00000000 00000000 00000000
50000090:00000000 00000000 00000000 00000000
500000A0:00000000 00000000 00000000 00000000
500000B0:00000000 00000000 00000000 00000000
500000C0:00000000 00000000 00000000 00000000
500000D0:00000000 00000000 00000000 00000000
500000E0:00000000 00000000 00000000 00000000
500000F0:00000000 00000000 00000000 00000000

root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
50020000:00000000 00000000 00000000 00000000
50020010:00000000 00000000 00000000 00000000
50020020:00000000 00000000 00000000 00000000
50020030:00000000 00000000 00000000 00000000
50020040:00000000 00000000 00000000 00000000
50020050:00000000 00000000 00000000 00000000
50020060:00000000 00000000 00000000 00000000
50020070:00000000 00000000 00000000 00000000
50020080:00000000 00000000 00000000 00000000
50020090:00000000 00000000 00000000 00000000
500200A0:00000000 00000000 00000000 00000000
500200B0:00000000 00000000 00000000 00000000
500200C0:00000000 00000000 00000000 00000000
500200D0:00000000 00000000 00000000 00000000
500200E0:00000000 00000000 00000000 00000000
500200F0:00000000 00000000 00000000 00000000

Corresponding PCI boot log

[    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
[    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
[    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
[    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
[    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
[    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
[    2.361483] PCI: bus0: Fast back to back transfers disabled
[    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
[    2.378560] PCI: bus1: Fast back to back transfers disabled
[    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
[    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
[    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
[    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
[    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
[    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
[    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
[    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt


Any idea where to start to debug this issue? Suggestion?

Thanks
-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Bjorn Helgaas]

On Wed, May 11, 2016 at 05:46:49PM -0400, Murali Karicheri wrote:
> Bjorn, Alex,
> 
> On 05/09/2016 07:12 PM, Alex Williamson wrote:
> > On Mon, 9 May 2016 17:38:27 -0400
> > Murali Karicheri <m-karicheri2@ti.com> wrote:
> > 
> >> On 05/09/2016 05:23 PM, Alex Williamson wrote:
> >>> On Mon, 9 May 2016 17:02:23 -0400
> >>> Murali Karicheri <m-karicheri2@ti.com> wrote:
> >>>   
> >>>> Hi Bjorn,
> >>>>
> >>>> Thanks for your quick response!
> >>>> See below for some follow up question.
> >>>>
> >>>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
> >>>>> Hi Murali,
> >>>>>
> >>>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
> >>>>>> Bjorn,
> >>>>>>
> >>>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> >>>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> >>>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> >>>>>> bringing up PCIe.
> >>>>>>
> >>>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> >>>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> >>>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
> >>>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> >>>>>>
> >>>>>> Here are the logs.
> >>>>>>
> >>>>>> K2E log with Marvel controller (Good working case).
> >>>>>> ===================================================
> >>>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> >>>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> >>>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >>>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> >>>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> >>>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> >>>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> >>>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> >>>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> >>>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> >>>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >>>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> >>>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> >>>>>>
> >>>>>> K2G log with Tealtek NIC card
> >>>>>> =============================
> >>>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> >>>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> >>>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> >>>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
> >>>>>
> >>>>> There's no "(bus address ...)" annotation here, which means these
> >>>>> windows map CPU addresses to identical bus addresses.  The
> >>>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> >>>>>     
> >>>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> >>>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> >>>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> >>>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> >>>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >>>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> >>>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> >>>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
> >>>>>
> >>>>> I assume these BARs (2 and 4) are the ones in question.  They are
> >>>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
> >>>>> bits, so the space is already all below 4GB.
> >>>>>
> >>>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
> >>>>> doesn't have any influence on that.  The type is encoded in the
> >>>>> low-order four bits of the BAR, which are read-only.
> >>>>>     
> >>>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> >>>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> >>>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >>>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> >>>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> >>>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> >>>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> >>>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> >>>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> >>>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
> >>>>>
> >>>>> Looks like this is printed by rtl_init_one().  I see that we read
> >>>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
> >>>> So if device specific configuration registers for this device are mapped
> >>>> to the window to have 64bit access, they are to be accessed with proper
> >>>> address to get the lower 32 bits of the data, right? So is there a chance
> >>>> it is making access to upper 4 bytes and hence reading all zeros?
> >>>>    
> >>>>> Could it be that some EEPROM on the card hasn't been programmed yet?
> >>>>> Does the card work if you put it in a different machine?
> >>>>>     
> >>>> Probably. I will ask this to the owner of r8169 driver. He seems to think
> >>>> the access is not right.  I need to find a PC that has PCIe slot to do this
> >>>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
> >>>> driver in Linux?   
> >>>
> >>> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
> >>> Adapter) that only has 32bit BARs:
> >>>
> >>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> >>> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
> >>> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
> >>> 	Region 2: I/O ports at e800 [size=32]
> >>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
> >>> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
> >>> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
> >>> 	Region 2: I/O ports at e400 [size=32]  
> >> Alex,
> >>
> >> Thanks for the response. I see following mini pci card which match with your
> >> description.
> >>
> >> http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
> >>
> >> Is this same thing? I assume e1000e/82571.c is the driver for this.
> >> Could you please confirm so that I can procure the same?
> > 
> > I would venture to say yes, but I offer no guarantees ;)
> > 
> > Ark seems to indicate all 1000 PT dual port cards use that chip:
> > http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
> > 
> I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC
> and I am getting the following error:- This one has a 32bit bar, but similar
> issue as with r8169 and tg3. The PCI access over the window seems to return zeros.
> Per suggestion, tried dumping the memory window and got all zeros. See below
> for additional debug info.
> 
> root@k2g-evm:/# insmod e1000e.ko debug=16
> [  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
> [  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
> [  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
> [  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
> [  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5
> 
> root@k2g-evm:/# lspci -vvv
> 00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Latency: 0, Cache Line Size: 64 bytes
>         Interrupt: pin A routed to IRQ 287
>         Region 0: Memory at <ignored> (32-bit, non-prefetchable)
>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>         I/O behind bridge: 00001000-00001fff
>         Memory behind bridge: 50000000-500fffff
>         Prefetchable memory behind bridge: 50100000-501fffff
>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>         BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>         Capabilities: [40] Power Management version 3
>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
>                 Address: 0000000021800054  Data: 0000
>         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0
>                         ExtTag- RBE+
>                 DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                         MaxPayload 128 bytes, MaxReadReq 256 bytes
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
>                 LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
>                         ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
>                 LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
>                 RootCap: CRSVisible-
>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>                          Compliance De-emphasis: -6dB
>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>         Capabilities: [100 v1] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>         Kernel driver in use: pcieport
> lspci: Unable to load libkmod resources: error -12
> 
> 01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
>         Subsystem: Intel Corporation PRO/1000 PT Server Adapter
>         Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>         Interrupt: pin A routed to IRQ 319
>         Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
>         Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
>         Region 2: I/O ports at 1000 [disabled] [size=32]
>         [virtual] Expansion ROM at 50100000 [disabled] [size=128K]
>         Capabilities: [c8] Power Management version 2
>                 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>         Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>                 Address: 0000000021800054  Data: 0001
>         Capabilities: [e0] Express (v1) Endpoint, MSI 00
>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
>                 DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>                 LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
>                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>         Capabilities: [100 v1] Advanced Error Reporting
>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>                 UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>                 AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>         Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2
> 
> Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with
> Broadcom tg3 and 
> 
> root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
> 50000000:00000000 00000000 00000000 00000000
> 50000010:00000000 00000000 00000000 00000000
> 50000020:00000000 00000000 00000000 00000000
> 50000030:00000000 00000000 00000000 00000000
> 50000040:00000000 00000000 00000000 00000000
> 50000050:00000000 00000000 00000000 00000000
> 50000060:00000000 00000000 00000000 00000000
> 50000070:00000000 00000000 00000000 00000000
> 50000080:00000000 00000000 00000000 00000000
> 50000090:00000000 00000000 00000000 00000000
> 500000A0:00000000 00000000 00000000 00000000
> 500000B0:00000000 00000000 00000000 00000000
> 500000C0:00000000 00000000 00000000 00000000
> 500000D0:00000000 00000000 00000000 00000000
> 500000E0:00000000 00000000 00000000 00000000
> 500000F0:00000000 00000000 00000000 00000000
> 
> root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
> 50020000:00000000 00000000 00000000 00000000
> 50020010:00000000 00000000 00000000 00000000
> 50020020:00000000 00000000 00000000 00000000
> 50020030:00000000 00000000 00000000 00000000
> 50020040:00000000 00000000 00000000 00000000
> 50020050:00000000 00000000 00000000 00000000
> 50020060:00000000 00000000 00000000 00000000
> 50020070:00000000 00000000 00000000 00000000
> 50020080:00000000 00000000 00000000 00000000
> 50020090:00000000 00000000 00000000 00000000
> 500200A0:00000000 00000000 00000000 00000000
> 500200B0:00000000 00000000 00000000 00000000
> 500200C0:00000000 00000000 00000000 00000000
> 500200D0:00000000 00000000 00000000 00000000
> 500200E0:00000000 00000000 00000000 00000000
> 500200F0:00000000 00000000 00000000 00000000
> 
> Corresponding PCI boot log
> 
> [    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
> [    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
> [    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
> [    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
> [    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> [    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> [    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
> [    2.361483] PCI: bus0: Fast back to back transfers disabled
> [    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
> [    2.378560] PCI: bus1: Fast back to back transfers disabled
> [    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> [    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> [    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> [    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
> [    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
> [    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> [    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
> [    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> [    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> [    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> [    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> [    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> 
> 
> Any idea where to start to debug this issue? Suggestion?

If I understand correctly, K2G is the platform where these NICs don't
work.  Do *any* PCI devices work correctly on that platform?  Maybe
there's some host controller configuration problem related to the MMIO
aperture?

If you have an analyzer, I guess you could look for your Read Requests
and Completions, but I doubt you'd learn anything.  I assume the NICs
are sending data over the link correctly, and something is happening
to the data between the device and the CPU.

I would suspect something in the host controller.  Most controllers
would give you 0xffffffff back instead of zeros if there's a problem,
but maybe yours is special.

Bjorn

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

Bjorn,

On 05/11/2016 06:55 PM, Bjorn Helgaas wrote:
> On Wed, May 11, 2016 at 05:46:49PM -0400, Murali Karicheri wrote:
>> Bjorn, Alex,

>> 500200D0:00000000 00000000 00000000 00000000
>> 500200E0:00000000 00000000 00000000 00000000
>> 500200F0:00000000 00000000 00000000 00000000
>>
>> Corresponding PCI boot log
>>
>> [    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
>> [    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
>> [    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
>> [    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
>> [    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>> [    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>> [    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
>> [    2.361483] PCI: bus0: Fast back to back transfers disabled
>> [    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
>> [    2.378560] PCI: bus1: Fast back to back transfers disabled
>> [    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>> [    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>> [    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> [    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
>> [    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
>> [    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>> [    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
>> [    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> [    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>> [    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>> [    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>> [    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>
>>
>> Any idea where to start to debug this issue? Suggestion?
> 
> If I understand correctly, K2G is the platform where these NICs don't
> work. 

Yes.

> Do *any* PCI devices work correctly on that platform? 

No. I am trying to test a SATA controller and see if that works.
Marvel SATA controller works fine using Keystone PCIe controller, but this
is on K2E EVM and this shares the same PCIe controller hardware (designware).

> Maybe
> there's some host controller configuration problem related to the MMIO
> aperture?
> 
> If you have an analyzer, I guess you could look for your Read Requests
> and Completions, but I doubt you'd learn anything.  I assume the NICs
> are sending data over the link correctly, and something is happening
> to the data between the device and the CPU.
> 
> I would suspect something in the host controller.  Most controllers
> would give you 0xffffffff back instead of zeros if there's a problem,
> but maybe yours is special.
I will work with our hardware folks to see if they can suggest something
to debug.

Thanks for you response.

Murali
> 
> Bjorn
> 


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

On 05/11/2016 06:55 PM, Bjorn Helgaas wrote:
> On Wed, May 11, 2016 at 05:46:49PM -0400, Murali Karicheri wrote:
>> Bjorn, Alex,
>>
>> On 05/09/2016 07:12 PM, Alex Williamson wrote:
>>> On Mon, 9 May 2016 17:38:27 -0400
>>> Murali Karicheri <m-karicheri2@ti.com> wrote:
>>>
>>>> On 05/09/2016 05:23 PM, Alex Williamson wrote:
>>>>> On Mon, 9 May 2016 17:02:23 -0400
>>>>> Murali Karicheri <m-karicheri2@ti.com> wrote:
>>>>>   
>>>>>> Hi Bjorn,
>>>>>>
>>>>>> Thanks for your quick response!
>>>>>> See below for some follow up question.
>>>>>>
>>>>>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:  
>>>>>>> Hi Murali,
>>>>>>>
>>>>>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:    
>>>>>>>> Bjorn,
>>>>>>>>
>>>>>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>>>>>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>>>>>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>>>>>>>> bringing up PCIe.
>>>>>>>>
>>>>>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>>>>>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>>>>>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>>>>>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>>>>>>>
>>>>>>>> Here are the logs.
>>>>>>>>
>>>>>>>> K2E log with Marvel controller (Good working case).
>>>>>>>> ===================================================
>>>>>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>>>>>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>>>>>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>>>>>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>>>>>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>>>>>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>>>>>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>>>>>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>>>>>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>>>>>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>>>>>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>>>>>>>
>>>>>>>> K2G log with Tealtek NIC card
>>>>>>>> =============================
>>>>>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>>>>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>>>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>>>>>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]    
>>>>>>>
>>>>>>> There's no "(bus address ...)" annotation here, which means these
>>>>>>> windows map CPU addresses to identical bus addresses.  The
>>>>>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
>>>>>>>     
>>>>>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>>>>>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>>>>>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>>>>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>>>>>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>>>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>>>>>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>>>>>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]    
>>>>>>>
>>>>>>> I assume these BARs (2 and 4) are the ones in question.  They are
>>>>>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
>>>>>>> bits, so the space is already all below 4GB.
>>>>>>>
>>>>>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
>>>>>>> doesn't have any influence on that.  The type is encoded in the
>>>>>>> low-order four bits of the BAR, which are read-only.
>>>>>>>     
>>>>>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>>>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>>>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>>>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>>>>>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>>>>>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>>>>>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>>>>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>>>>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>>>>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286    
>>>>>>>
>>>>>>> Looks like this is printed by rtl_init_one().  I see that we read
>>>>>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.    
>>>>>> So if device specific configuration registers for this device are mapped
>>>>>> to the window to have 64bit access, they are to be accessed with proper
>>>>>> address to get the lower 32 bits of the data, right? So is there a chance
>>>>>> it is making access to upper 4 bytes and hence reading all zeros?
>>>>>>    
>>>>>>> Could it be that some EEPROM on the card hasn't been programmed yet?
>>>>>>> Does the card work if you put it in a different machine?
>>>>>>>     
>>>>>> Probably. I will ask this to the owner of r8169 driver. He seems to think
>>>>>> the access is not right.  I need to find a PC that has PCIe slot to do this
>>>>>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
>>>>>> driver in Linux?   
>>>>>
>>>>> I happen to have an Intel 82571EB (PRO/1000 PT Dual Port Server
>>>>> Adapter) that only has 32bit BARs:
>>>>>
>>>>> 02:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>>>> 	Region 0: Memory at fe7e0000 (32-bit, non-prefetchable) [size=128K]
>>>>> 	Region 1: Memory at fe7c0000 (32-bit, non-prefetchable) [size=128K]
>>>>> 	Region 2: I/O ports at e800 [size=32]
>>>>> 02:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:105e] (rev 06)
>>>>> 	Region 0: Memory at fe7a0000 (32-bit, non-prefetchable) [size=128K]
>>>>> 	Region 1: Memory at fe780000 (32-bit, non-prefetchable) [size=128K]
>>>>> 	Region 2: I/O ports at e400 [size=32]  
>>>> Alex,
>>>>
>>>> Thanks for the response. I see following mini pci card which match with your
>>>> description.
>>>>
>>>> http://www.amazon.com/Intel-1000-Dual-Server-Adapter/dp/B000BMZHX2/ref=sr_1_1?ie=UTF8&qid=1462829688&sr=8-1&keywords=intel+expi9402pt
>>>>
>>>> Is this same thing? I assume e1000e/82571.c is the driver for this.
>>>> Could you please confirm so that I can procure the same?
>>>
>>> I would venture to say yes, but I offer no guarantees ;)
>>>
>>> Ark seems to indicate all 1000 PT dual port cards use that chip:
>>> http://ark.intel.com/products/50494/Intel-PRO1000-PT-Dual-Port-Server-Adapter
>>>
>> I got hold of a Intel pro server adapter card and tried it with Keystone PCIe RC
>> and I am getting the following error:- This one has a 32bit bar, but similar
>> issue as with r8169 and tg3. The PCI access over the window seems to return zeros.
>> Per suggestion, tried dumping the memory window and got all zeros. See below
>> for additional debug info.
>>
>> root@k2g-evm:/# insmod e1000e.ko debug=16
>> [  604.596552] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k
>> [  604.602505] e1000e: Copyright(c) 1999 - 2015 Intel Corporation.
>> [  604.608774] e1000e 0000:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode
>> [  606.462763] e1000e 0000:01:00.0: The NVM Checksum Is Not Valid
>> [  606.477439] e1000e: probe of 0000:01:00.0 failed with error -5
>>
>> root@k2g-evm:/# lspci -vvv
>> 00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal decode])
>>         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
>>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>         Latency: 0, Cache Line Size: 64 bytes
>>         Interrupt: pin A routed to IRQ 287
>>         Region 0: Memory at <ignored> (32-bit, non-prefetchable)
>>         Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
>>         I/O behind bridge: 00001000-00001fff
>>         Memory behind bridge: 50000000-500fffff
>>         Prefetchable memory behind bridge: 50100000-501fffff
>>         Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
>>         BridgeCtl: Parity+ SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
>>                 PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
>>         Capabilities: [40] Power Management version 3
>>                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
>>         Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
>>                 Address: 0000000021800054  Data: 0000
>>         Capabilities: [70] Express (v2) Root Port (Slot-), MSI 00
>>                 DevCap: MaxPayload 256 bytes, PhantFunc 0
>>                         ExtTag- RBE+
>>                 DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                         MaxPayload 128 bytes, MaxReadReq 256 bytes
>>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
>>                 LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s <2us, L1 <64us
>>                         ClockPM- Surprise- LLActRep+ BwNot- ASPMOptComp-
>>                 LnkCtl: ASPM Disabled; RCB 128 bytes Disabled- CommClk-
>>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive+ BWMgmt- ABWMgmt-
>>                 RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna+ CRSVisible-
>>                 RootCap: CRSVisible-
>>                 RootSta: PME ReqID 0000, PMEStatus- PMEPending-
>>                 DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported ARIFwd-
>>                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
>>                 LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
>>                          Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
>>                          Compliance De-emphasis: -6dB
>>                 LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
>>                          EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
>>         Capabilities: [100 v1] Advanced Error Reporting
>>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
>>                 AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
>>         Kernel driver in use: pcieport
>> lspci: Unable to load libkmod resources: error -12
>>
>> 01:00.0 Ethernet controller: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) (rev 06)
>>         Subsystem: Intel Corporation PRO/1000 PT Server Adapter
>>         Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
>>         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
>>         Interrupt: pin A routed to IRQ 319
>>         Region 0: Memory at 50000000 (32-bit, non-prefetchable) [size=128K]
>>         Region 1: Memory at 50020000 (32-bit, non-prefetchable) [size=128K]
>>         Region 2: I/O ports at 1000 [disabled] [size=32]
>>         [virtual] Expansion ROM at 50100000 [disabled] [size=128K]
>>         Capabilities: [c8] Power Management version 2
>>                 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
>>                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=1 PME-
>>         Capabilities: [d0] MSI: Enable- Count=1/1 Maskable- 64bit+
>>                 Address: 0000000021800054  Data: 0001
>>         Capabilities: [e0] Express (v1) Endpoint, MSI 00
>>                 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
>>                         ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
>>                 DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
>>                         RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                         MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                 DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
>>                 LnkCap: Port #4, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
>>                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
>>                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
>>                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
>>                 LnkSta: Speed 2.5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
>>         Capabilities: [100 v1] Advanced Error Reporting
>>                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
>>                 UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
>>                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>>                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
>>                 AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
>>         Capabilities: [140 v1] Device Serial Number 00-15-17-ff-ff-90-0f-b2
>>
>> Tried dumping the window 50000000 and 50020000 and See all zeros. Similar observation with
>> Broadcom tg3 and 
>>
>> root@k2g-evm:/# ./rdwrmem -m -s 0x50000000 -b 4 -l 0x100
>> 50000000:00000000 00000000 00000000 00000000
>> 50000010:00000000 00000000 00000000 00000000
>> 50000020:00000000 00000000 00000000 00000000
>> 50000030:00000000 00000000 00000000 00000000
>> 50000040:00000000 00000000 00000000 00000000
>> 50000050:00000000 00000000 00000000 00000000
>> 50000060:00000000 00000000 00000000 00000000
>> 50000070:00000000 00000000 00000000 00000000
>> 50000080:00000000 00000000 00000000 00000000
>> 50000090:00000000 00000000 00000000 00000000
>> 500000A0:00000000 00000000 00000000 00000000
>> 500000B0:00000000 00000000 00000000 00000000
>> 500000C0:00000000 00000000 00000000 00000000
>> 500000D0:00000000 00000000 00000000 00000000
>> 500000E0:00000000 00000000 00000000 00000000
>> 500000F0:00000000 00000000 00000000 00000000
>>
>> root@k2g-evm:/# ./rdwrmem -m -s 0x50020000 -b 4 -l 0x100                                                                                    
>> 50020000:00000000 00000000 00000000 00000000
>> 50020010:00000000 00000000 00000000 00000000
>> 50020020:00000000 00000000 00000000 00000000
>> 50020030:00000000 00000000 00000000 00000000
>> 50020040:00000000 00000000 00000000 00000000
>> 50020050:00000000 00000000 00000000 00000000
>> 50020060:00000000 00000000 00000000 00000000
>> 50020070:00000000 00000000 00000000 00000000
>> 50020080:00000000 00000000 00000000 00000000
>> 50020090:00000000 00000000 00000000 00000000
>> 500200A0:00000000 00000000 00000000 00000000
>> 500200B0:00000000 00000000 00000000 00000000
>> 500200C0:00000000 00000000 00000000 00000000
>> 500200D0:00000000 00000000 00000000 00000000
>> 500200E0:00000000 00000000 00000000 00000000
>> 500200F0:00000000 00000000 00000000 00000000
>>
>> Corresponding PCI boot log
>>
>> [    2.310465] PCI host bridge /soc/pcie@21800000 ranges:
>> [    2.316146]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
>> [    2.323302]    IO 0x23250000..0x23253fff -> 0x00000000
>> [    2.328455]   MEM 0x50000000..0x5fffffff -> 0x50000000
>> [    2.335406] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>> [    2.342206] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [    2.347691] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>> [    2.353937] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
>> [    2.361483] PCI: bus0: Fast back to back transfers disabled
>> [    2.368357] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can enable it with 'pcie_aspm=force'
>> [    2.378560] PCI: bus1: Fast back to back transfers disabled
>> [    2.384366] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>> [    2.391314] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>> [    2.398534] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> [    2.404699] pci 0000:01:00.0: BAR 0: assigned [mem 0x50000000-0x5001ffff]
>> [    2.411555] pci 0000:01:00.0: BAR 1: assigned [mem 0x50020000-0x5003ffff]
>> [    2.418348] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>> [    2.425591] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
>> [    2.431712] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    2.436675] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> [    2.442788] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>> [    2.449572] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>> [    2.457313] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>> [    2.464412] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>
>>
>> Any idea where to start to debug this issue? Suggestion?
> 
> If I understand correctly, K2G is the platform where these NICs don't
> work.  Do *any* PCI devices work correctly on that platform?  Maybe
> there's some host controller configuration problem related to the MMIO
> aperture?
> 
> If you have an analyzer, I guess you could look for your Read Requests
> and Completions, but I doubt you'd learn anything.  I assume the NICs
> are sending data over the link correctly, and something is happening
> to the data between the device and the CPU.
> 
> I would suspect something in the host controller.  Most controllers
> would give you 0xffffffff back instead of zeros if there's a problem,
> but maybe yours is special.
> 
Bjorn,

I am able to bring up PCIe last week. There was a PCIe memory address 
incorrect in the UG that was pointing to a reserved area. So was reading
all zeros. With that fixed, PCIe devices such as e1000e and tg3 worked
fine and I could pass network traffic over the PCIe interface without
issues. Also a USB device worked fine. But I have some issues with the
Marvell 88SE9120 SATA 6Gb/s SATA (link never comes up) and also one
of the NIC card RTL8168c/8111c (no data over the network interface)

root@k2g-evm:/# insmod r8169.ko
[   43.980877] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   43.986609] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[   43.993132] r8169 0000:01:00.0: limiting MRRS to 256
[   44.004152] r8169 0000:01:00.0 eth1: RTL8168c/8111c at 0xf0f56000,
00:e0:4c:21:16:2c, XID 1c4000c0 IRQ 288
[   44.024167] r8169 0000:01:00.0 eth1: jumbo features [frames: 6128 bytes, tx
checksumming: ko]
root@k2g-evm:/# ifup eth1
[   51.782650] r8169 0000:01:00.0 eth1: link down
[   51.787121] r8169 0000:01:00.0 eth1: link down
[   51.792348] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
udhcpc (v1.23.1) started
Sending discover...
[   54.158011] r8169 0000:01:00.0 eth1: link up
[   54.162344] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sending discover...
Sending discover...
No lease, forking to background
root@k2g-evm:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 50:72:24:D6:DE:ED
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 00:E0:4C:21:16:2C
          inet6 addr: fe80::2e0:4cff:fe21:162c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


root@k2g-evm:/# insmod ahci.ko
[   79.283747] ahci 0000:01:00.0: limiting MRRS to 256
[   79.288878] ahci 0000:01:00.0: AHCI 0001.0000 32 slots 2 ports 6 Gbps 0x3
impl SATA mode
[   79.297868] ahci 0000:01:00.0: flags: 64bit ncq sntf led only pmp fbs pio
slum part sxs
[   79.327629] scsi host0: ahci
[   79.344756] scsi host1: ahci
[   79.350808] ata1: SATA max UDMA/133 abar m2048@0x70000000 port 0x70000100 irq
288
[   79.358973] ata2: SATA max UDMA/133 abar m2048@0x70000000 port 0x70000180 irq
288
[   79.711079] ata2: SATA link down (SStatus 0 SControl 330)
[   79.716555] ata1: SATA link down (SStatus 0 SControl 330)


Will need to debug the above, but for now we have few NIC devices that works
and USB. I would like to try the Marvel 918x as well which is known to work on
K2E EVM.

Thanks once again for the right pointers that helped me debug this issue.


Murali
> Bjorn
> 


-- 
Murali Karicheri
Linux Kernel, Keystone

Re: PCIe issue with NIC card that has 64bit BARs

[Bjorn Helgaas]

On Mon, May 09, 2016 at 05:02:23PM -0400, Murali Karicheri wrote:
> Hi Bjorn,
> 
> Thanks for your quick response!
> See below for some follow up question.
> 
> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:
> > Hi Murali,
> > 
> > On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:
> >> Bjorn,
> >>
> >> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
> >> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
> >> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
> >> bringing up PCIe.
> >>
> >> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
> >> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
> >> controller and that on K2G with rtk8168. Is there way we can get it assigned
> >> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
> >>
> >> Here are the logs.
> >>
> >> K2E log with Marvel controller (Good working case).
> >> ===================================================
> >> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
> >> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
> >> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
> >> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
> >> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
> >> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
> >> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
> >> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
> >> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
> >> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
> >> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
> >> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
> >>
> >> K2G log with Tealtek NIC card
> >> =============================
> >> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
> >> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
> >> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
> >> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
> > 
> > There's no "(bus address ...)" annotation here, which means these
> > windows map CPU addresses to identical bus addresses.  The
> > "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
> > 
> >> [    2.337567] PCI: bus0: Fast back to back transfers disabled
> >> [    2.361159] PCI: bus1: Fast back to back transfers disabled
> >> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
> >> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
> >> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> >> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
> >> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
> >> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
> > 
> > I assume these BARs (2 and 4) are the ones in question.  They are
> > 64-bit BARs, but the addresses currently assigned to them fit in 32
> > bits, so the space is already all below 4GB.
> > 
> > The BAR type (32-bit or 64-bit) is built into the device; Linux
> > doesn't have any influence on that.  The type is encoded in the
> > low-order four bits of the BAR, which are read-only.
> > 
> >> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
> >> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
> >> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> >> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
> >> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
> >> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> >> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> >> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
> >> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
> >> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
> > 
> > Looks like this is printed by rtl_init_one().  I see that we read
> > zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
> So if device specific configuration registers for this device are mapped
> to the window to have 64bit access, they are to be accessed with proper
> address to get the lower 32 bits of the data, right?

I'm not sure I understand the question, so correct me if I'm going
astray.

r8169.c assumes device registers are mapped by BAR 1 or 2 (see
rtl_cfg_infos []).  Your device has no BAR 1, so the registers must be
mapped by BAR 2.

"BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]" means the *BAR* is
64 bits wide.  This has nothing to do with the access size for reads &
writes to the area mapped by the BAR.  The BAR currently contains
0x50000000, which is only 32 bits, but it *could* contain any 64-bit
address.

This BAR is 4KB in size, so it maps 4KB of register space.  The BAR
places it at bus address 0x50000000 (which in this case is also at CPU
physical address 0x50000000, since there's no host bridge address
translation).

The driver ioremaps that space, and it apparently got 0xf0d6a000, so
it can access the registers by reading and writing at virtual
addresses 0xf0d6a000-0xf0d6afff.

You should be able to manually read the register space with a program
like http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c, which uses
/dev/mem.  You would use the CPU physical addresses, e.g., 0x50000000.

> So is there a chance
> it is making access to upper 4 bytes and hence reading all zeros?

If you read the entire 64-bit BAR itself, you should see zeros in the
upper 32 bits, e.g.,

  $ setpci -sBB:DD.F BASE_ADDRESS_2 BASE_ADDRESS_3
  50000000
  00000000

since the address fits in the low 32 bits.  But accesses to the
registers themselves would use 32-bit addresses starting at
0x50000000.

> > Could it be that some EEPROM on the card hasn't been programmed yet?
> > Does the card work if you put it in a different machine?
> > 
> Probably. I will ask this to the owner of r8169 driver. He seems to think
> the access is not right.  I need to find a PC that has PCIe slot to do this
> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
> driver in Linux? 

Nope, sorry, I don't pay much attention to specific devices.

Bjorn

Re: PCIe issue with NIC card that has 64bit BARs

[Murali Karicheri]

Bjorn,

On 05/09/2016 05:33 PM, Bjorn Helgaas wrote:
> On Mon, May 09, 2016 at 05:02:23PM -0400, Murali Karicheri wrote:
>> Hi Bjorn,
>>
>> Thanks for your quick response!
>> See below for some follow up question.
>>
>> On 05/09/2016 04:34 PM, Bjorn Helgaas wrote:
>>> Hi Murali,
>>>
>>> On Mon, May 09, 2016 at 03:32:42PM -0400, Murali Karicheri wrote:
>>>> Bjorn,
>>>>
>>>> I am running into an issue with using a rtk8168 GiB NIC card with Keystone PCIe.
>>>> It works for 32bit BARs such as the Marvel SATA controller on K2E. On another
>>>> recent SoC (K2G) that re-uses the same driver and hardware, I have issues
>>>> bringing up PCIe.
>>>>
>>>> The rtk8168 NIC gets detected, but all read values are zeros. Based on the boot
>>>> log, it appears to be getting assigned 64bit BARs. See the log on K2E with Marvell
>>>> controller and that on K2G with rtk8168. Is there way we can get it assigned
>>>> 32Bit BAR and get it functional? Keystone is a 32bit ARM A15 SoC.
>>>>
>>>> Here are the logs.
>>>>
>>>> K2E log with Marvel controller (Good working case).
>>>> ===================================================
>>>> [    0.236353] pci 0000:00:00.0: BAR 8: assigned [mem 0x60000000-0x600fffff]
>>>> [    0.236364] pci 0000:00:00.0: BAR 9: assigned [mem 0x60100000-0x601fffff pref]
>>>> [    0.236373] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>> [    0.236385] pci 0000:01:00.0: BAR 6: assigned [mem 0x60100000-0x6010ffff pref]
>>>> [    0.236394] pci 0000:01:00.0: BAR 5: assigned [mem 0x60000000-0x600001ff]
>>>> [    0.236406] pci 0000:01:00.0: BAR 4: assigned [io  0x1000-0x100f]
>>>> [    0.236418] pci 0000:01:00.0: BAR 0: assigned [io  0x1010-0x1017]
>>>> [    0.236429] pci 0000:01:00.0: BAR 2: assigned [io  0x1018-0x101f]
>>>> [    0.236441] pci 0000:01:00.0: BAR 1: assigned [io  0x1020-0x1023]
>>>> [    0.236452] pci 0000:01:00.0: BAR 3: assigned [io  0x1024-0x1027]
>>>> [    0.236464] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> [    0.236472] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>> [    0.236481] pci 0000:00:00.0:   bridge window [mem 0x60000000-0x600fffff]
>>>> [    0.236490] pci 0000:00:00.0:   bridge window [mem 0x60100000-0x601fffff pref]
>>>>
>>>> K2G log with Tealtek NIC card
>>>> =============================
>>>> [    2.311572] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>>>> [    2.318188] pci_bus 0000:00: root bus resource [bus 00-ff]
>>>> [    2.323844] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>>>> [    2.330023] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
>>>
>>> There's no "(bus address ...)" annotation here, which means these
>>> windows map CPU addresses to identical bus addresses.  The
>>> "[mem 0x50000000-0x5fffffff]" window is a 32-bit window.
>>>
>>>> [    2.337567] PCI: bus0: Fast back to back transfers disabled
>>>> [    2.361159] PCI: bus1: Fast back to back transfers disabled
>>>> [    2.366889] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>>>> [    2.373841] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>>>> [    2.381061] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>>>> [    2.387225] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>>>> [    2.394505] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>>>> [    2.402288] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
>>>
>>> I assume these BARs (2 and 4) are the ones in question.  They are
>>> 64-bit BARs, but the addresses currently assigned to them fit in 32
>>> bits, so the space is already all below 4GB.
>>>
>>> The BAR type (32-bit or 64-bit) is built into the device; Linux
>>> doesn't have any influence on that.  The type is encoded in the
>>> low-order four bits of the BAR, which are read-only.
>>>
>>>> [    2.409610] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>>>> [    2.415729] pci 0000:00:00.0: PCI bridge to [bus 01]
>>>> [    2.420693] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>>>> [    2.426806] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>>>> [    2.433610] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>>>> [    2.441365] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>>>> [    2.448323] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>>>> [    2.455567] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>>>> [    2.461332] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>>>> [    2.468789] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
>>>
>>> Looks like this is printed by rtl_init_one().  I see that we read
>>> zeroes from MAC0 and TxConfig, but I don't see any PCI problem here.
>> So if device specific configuration registers for this device are mapped
>> to the window to have 64bit access, they are to be accessed with proper
>> address to get the lower 32 bits of the data, right?
> 
> I'm not sure I understand the question, so correct me if I'm going
> astray.
> 
> r8169.c assumes device registers are mapped by BAR 1 or 2 (see
> rtl_cfg_infos []).  Your device has no BAR 1, so the registers must be
> mapped by BAR 2.
> 
Ok. I will review and get back. Thanks for your response!

Regards,

Murali

> "BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]" means the *BAR* is
> 64 bits wide.  This has nothing to do with the access size for reads &
> writes to the area mapped by the BAR.  The BAR currently contains
> 0x50000000, which is only 32 bits, but it *could* contain any 64-bit
> address.
> 
> This BAR is 4KB in size, so it maps 4KB of register space.  The BAR
> places it at bus address 0x50000000 (which in this case is also at CPU
> physical address 0x50000000, since there's no host bridge address
> translation).
> 
> The driver ioremaps that space, and it apparently got 0xf0d6a000, so
> it can access the registers by reading and writing at virtual
> addresses 0xf0d6a000-0xf0d6afff.
> 
> You should be able to manually read the register space with a program
> like http://cmp.felk.cvut.cz/~pisa/linux/rdwrmem.c, which uses
> /dev/mem.  You would use the CPU physical addresses, e.g., 0x50000000.
> 
>> So is there a chance
>> it is making access to upper 4 bytes and hence reading all zeros?
> 
> If you read the entire 64-bit BAR itself, you should see zeros in the
> upper 32 bits, e.g.,
> 
>   $ setpci -sBB:DD.F BASE_ADDRESS_2 BASE_ADDRESS_3
>   50000000
>   00000000
> 
> since the address fits in the low 32 bits.  But accesses to the
> registers themselves would use 32-bit addresses starting at
> 0x50000000.
> 
>>> Could it be that some EEPROM on the card hasn't been programmed yet?
>>> Does the card work if you put it in a different machine?
>>>
>> Probably. I will ask this to the owner of r8169 driver. He seems to think
>> the access is not right.  I need to find a PC that has PCIe slot to do this
>> test. Meanwhile do you know of any 1GiB NIC card that has 32bit BARs and
>> driver in Linux? 
> 
> Nope, sorry, I don't pay much attention to specific devices.
> 
> Bjorn
> 


-- 
Murali Karicheri
Linux Kernel, Keystone