Re: rtk8168 driver help needed

[Murali Karicheri <m-karicheri2@ti.com>]

On 05/07/2016 04:35 AM, Francois Romieu wrote:
> Murali Karicheri <m-karicheri2@ti.com> :
> [...]
>> I am trying to integrate the rtl8168 PCIe card to have Ethernet functional
>> on my Keystone EVM.
> 
> Which (EVM) one ?
> 
>> I purchased the rtl8111c Gib card from Amazon. The Card is detected
>> by the RC and I can see it is enumerated and show up when doing lspci command.
> 
> What does "the RC" mean ? A different PC ?
> 
>> However I can't get the Ethernet port functional.
> 
>> Does this need MSI interrupt ?
> 
> No.
> 
>> I can't see it has requested any.
> 
> Yes, something went really, really wrong. See below.
> 
I have figured out why I was getting all zeros over the PCIe bus. The PCIe 
address range used by the CPU to access the PCI space is wrong. Once that
is fixed, I was able to test few PCIe EP devices such as Intel e1000e, Broadcom
tg3, Marvel SATA etc. But the rtk8168 driver still don't work. I have following
log now.

[    2.372877] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.378362] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.384609] pci_bus 0000:00: root bus resource [mem 0x70000000-0x7fffffff]
[    2.392155] PCI: bus0: Fast back to back transfers disabled
[    2.411222] PCI: bus1: Fast back to back transfers disabled
[    2.416964] pci 0000:00:00.0: BAR 8: assigned [mem 0x70000000-0x700fffff]
[    2.423927] pci 0000:00:00.0: BAR 9: assigned [mem 0x70100000-0x701fffff
pref]
[    2.431217] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    2.437315] pci 0000:01:00.0: BAR 6: assigned [mem 0x70100000-0x7011ffff
pref]
[    2.444597] pci 0000:01:00.0: BAR 4: assigned [mem 0x70120000-0x7012ffff
64bit pref]
[    2.452383] pci 0000:01:00.0: BAR 2: assigned [mem 0x70000000-0x70000fff
64bit]
[    2.459705] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
[    2.465825] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.470788] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    2.476905] pci 0000:00:00.0:   bridge window [mem 0x70000000-0x700fffff]
[    2.483711] pci 0000:00:00.0:   bridge window [mem 0x70100000-0x701fffff
pref]


And then 

root@k2g-evm:~# lspci -v
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal
decode])
        Flags: bus master, fast devsel, latency 0, IRQ 287
        Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 00001000-00001fff
        Memory behind bridge: 70000000-700fffff
        Prefetchable memory behind bridge: 70100000-701fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
PCI Express Gigabit Ethernet Controller (rev 02)
        Subsystem: Realtek Semiconductor Co., Ltd. Device 0123
        Flags: fast devsel, IRQ 319
        I/O ports at 1000 [disabled] [size=256]
        Memory at 70000000 (64-bit, non-prefetchable) [disabled] [size=4K]
        Memory at 70120000 (64-bit, prefetchable) [disabled] [size=64K]
        [virtual] Expansion ROM at 70100000 [disabled] [size=128K]
        Capabilities: [40] Power Management version 0
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Endpoint, MSI 01
        Capabilities: [b0] MSI-X: Enable- Count=2 Masked-
        Capabilities: [d0] Vital Product Data
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [140] Virtual Channel
        Capabilities: [160] Device Serial Number 01-00-00-00-68-4c-e0-00


And then

root@k2g-evm:/# insmod r8169.ko
[   43.980877] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
[   43.986609] r8169 0000:01:00.0: enabling device (0140 -> 0143)
[   43.993132] r8169 0000:01:00.0: limiting MRRS to 256
[   44.004152] r8169 0000:01:00.0 eth1: RTL8168c/8111c at 0xf0f56000,
00:e0:4c:21:16:2c, XID 1c4000c0 IRQ 288
[   44.024167] r8169 0000:01:00.0 eth1: jumbo features [frames: 6128 bytes, tx
checksumming: ko]
root@k2g-evm:/# ifup eth1
[   51.782650] r8169 0000:01:00.0 eth1: link down
[   51.787121] r8169 0000:01:00.0 eth1: link down
[   51.792348] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
udhcpc (v1.23.1) started
Sending discover...
[   54.158011] r8169 0000:01:00.0 eth1: link up
[   54.162344] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sending discover...
Sending discover...
No lease, forking to background
root@k2g-evm:/# ifconfig
eth0      Link encap:Ethernet  HWaddr 50:72:24:D6:DE:ED
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

eth1      Link encap:Ethernet  HWaddr 00:E0:4C:21:16:2C
          inet6 addr: fe80::2e0:4cff:fe21:162c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)


Looks like there is nothing transmitted.


For Broadcom tg3, I get

[    2.353188] pci_bus 0000:00: root bus resource [bus 00-ff]
[    2.358673] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
[    2.364924] pci_bus 0000:00: root bus resource [mem 0x70000000-0x7fffffff]
[    2.372468] PCI: bus0: Fast back to back transfers disabled
[    2.378612] pci 0000:01:00.0: reg 0x10: can't handle BAR above 4GB (bus
address 0x2400460302000000)
[    2.388393] pci 0000:01:00.0: disabling ASPM on pre-1.1 PCIe device.  You can
enable it with 'pcie_aspm=force'
[    2.398565] PCI: bus1: Fast back to back transfers disabled
[    2.404437] pci 0000:00:00.0: BAR 8: assigned [mem 0x70000000-0x700fffff]
[    2.411322] pci 0000:01:00.0: BAR 0: assigned [mem 0x70000000-0x7000ffff
64bit]
[    2.418648] pci 0000:00:00.0: PCI bridge to [bus 01]
[    2.423682] pci 0000:00:00.0:   bridge window [mem 0x70000000-0x700fffff]
[    2.430978] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    2.438106] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
[    2.445532] keystone-navigator-qmss soc:qmss@4020000: qmgr start queue 0,
number of queues 128
[    2.454489] keystone-navigator-qmss soc:qmss@4020000: added qmgr start queue

And then

root@k2g-evm:~# lspci -v
00:00.0 PCI bridge: Texas Instruments Device b00b (rev 01) (prog-if 00 [Normal
decode])
        Flags: bus master, fast devsel, latency 0, IRQ 287
        Memory at <ignored> (32-bit, non-prefetchable)
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        Memory behind bridge: 70000000-700fffff
        Capabilities: [40] Power Management version 3
        Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [70] Express Root Port (Slot-), MSI 00
        Capabilities: [100] Advanced Error Reporting
        Kernel driver in use: pcieport
lspci: Unable to load libkmod resources: error -12

01:00.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5721 Gigabit
Ethernet PCI Express (rev 21)
        Subsystem: Broadcom Corporation NetXtreme BCM5721 Gigabit Ethernet PCI
Express
        Flags: fast devsel, IRQ 319
        Memory at 70000000 (64-bit, non-prefetchable) [disabled] [size=64K]
        Capabilities: [48] Power Management version 2
        Capabilities: [50] Vital Product Data
        Capabilities: [58] MSI: Enable- Count=1/8 Maskable- 64bit+
        Capabilities: [d0] Express Endpoint, MSI 00
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [13c] Virtual Channel
        Capabilities: [160] Device Serial Number 00-10-18-ff-fe-27-cc-c0
        Capabilities: [16c] Power Budgeting <?>


And then

root@k2g-evm:/# insmod tg3.ko
[   49.615358] tg3.c:v3.137 (May 11, 2014)
[   49.619223] tg3 0000:01:00.0: enabling device (0140 -> 0142)
[   49.625590] tg3 0000:01:00.0: limiting MRRS to 256
[   49.654559] tg3 0000:01:00.0 eth1: Tigon3 [partno(BCM95721A211F) rev 4201]
(PCI Express) MAC address 00:10:18:27:cc:c0
[   49.675314] tg3 0000:01:00.0 eth1: attached PHY is 5750 (10/100/1000Base-T
Ethernet) (WireSpeed[1], EEE[0])
[   49.698863] tg3 0000:01:00.0 eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0]
TSOcap[1]
[   49.715708] tg3 0000:01:00.0 eth1: dma_rwctrl[76180000] dma_mask[64-bit]
root@k2g-evm:/#
root@k2g-evm:/#
root@k2g-evm:/#
root@k2g-evm:/# ifp eth1
-sh: ifp: command not found
root@k2g-evm:/# ifup eth1
[   59.207581] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
udhcpc (v1.23.1) started
Sending discover...
Sending discover...
[   62.361817] tg3 0000:01:00.0 eth1: Link is up at 1000 Mbps, full duplex
[   62.368436] tg3 0000:01:00.0 eth1: Flow control is on for TX and on for RX
[   62.376029] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
Sending select for 192.168.0.27...
Lease of 192.168.0.27 obtained, lease time 864000
/etc/udhcpc.d/50default: Adding DNS 192.168.0.2
/etc/udhcpc.d/50default: Adding DNS 192.168.0.10
root@k2g-evm:/#
root@k2g-evm:/#
root@k2g-evm:/#
root@k2g-evm:/# ping 192.168.0.21
PING 192.168.0.21 (192.168.0.21): 56 data bytes
64 bytes from 192.168.0.21: seq=0 ttl=64 time=0.752 ms
64 bytes from 192.168.0.21: seq=1 ttl=64 time=0.384 ms
64 bytes from 192.168.0.21: seq=2 ttl=64 time=0.341 ms
64 bytes from 192.168.0.21: seq=3 ttl=64 time=0.277 ms

Do you what could be wrong with rtk8168?

Murali


> [...]
>> [    2.303965] PCI host bridge /soc/pcie@21800000 ranges:
>> [    2.309108]   No bus range found for /soc/pcie@21800000, using [bus 00-ff]
>> [    2.316269]    IO 0x23250000..0x23253fff -> 0x00000000
>> [    2.321499]   MEM 0x50000000..0x5fffffff -> 0x50000000
>> [    2.331666] keystone-pcie 21801000.pcie: PCI host bridge to bus 0000:00
>> [    2.338283] pci_bus 0000:00: root bus resource [bus 00-ff]
>> [    2.343937] pci_bus 0000:00: root bus resource [io  0x0000-0x3fff]
>> [    2.350114] pci_bus 0000:00: root bus resource [mem 0x50000000-0x5fffffff]
>> [    2.357095] pci 0000:00:00.0: [104c:b00b] type 01 class 0x060400
>> [    2.357665] PCI: bus0: Fast back to back transfers disabled
>> [    2.363717] pci 0000:01:00.0: [10ec:8168] type 00 class 0x020000
>> [    2.363809] pci 0000:01:00.0: reg 0x10: [io  0x0000-0x00ff]
>> [    2.363867] pci 0000:01:00.0: reg 0x18: [mem 0x00000000-0x00000fff 64bit]
>> [    2.363909] pci 0000:01:00.0: reg 0x20: [mem 0x00000000-0x0000ffff 64bit pref]
>> [    2.363939] pci 0000:01:00.0: reg 0x30: [mem 0x00000000-0x0001ffff pref]
>> [    2.364099] pci 0000:01:00.0: supports D1 D2
>> [    2.364116] pci 0000:01:00.0: PME# supported from D0 D1 D2 D3hot D3cold
>> [    2.381251] PCI: bus1: Fast back to back transfers disabled
>> [    2.386989] pci 0000:00:00.0: BAR 8: assigned [mem 0x50000000-0x500fffff]
>> [    2.393937] pci 0000:00:00.0: BAR 9: assigned [mem 0x50100000-0x501fffff pref]
>> [    2.401221] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
>> [    2.407320] pci 0000:01:00.0: BAR 6: assigned [mem 0x50100000-0x5011ffff pref]
>> [    2.414597] pci 0000:01:00.0: BAR 4: assigned [mem 0x50120000-0x5012ffff 64bit pref]
>> [    2.422380] pci 0000:01:00.0: BAR 2: assigned [mem 0x50000000-0x50000fff 64bit]
>> [    2.429702] pci 0000:01:00.0: BAR 0: assigned [io  0x1000-0x10ff]
>> [    2.435821] pci 0000:00:00.0: PCI bridge to [bus 01]
>> [    2.440783] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
>> [    2.446896] pci 0000:00:00.0:   bridge window [mem 0x50000000-0x500fffff]
>> [    2.453699] pci 0000:00:00.0:   bridge window [mem 0x50100000-0x501fffff pref]
>> [    2.461453] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
>> [    2.468411] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
>> [    2.475075] pcie_pme 0000:00:00.0:pcie01: service driver pcie_pme loaded
>> [    2.475392] aer 0000:00:00.0:pcie02: service driver aer loaded
>> [    2.475652] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded
>> [    2.481419] r8169 0000:01:00.0: enabling device (0140 -> 0143)
>> [    2.488865] r8169 0000:01:00.0 eth0: RTL8169 at 0xf0d6a000, 00:00:00:00:00:00, XID 00000000 IRQ 286
> 
> No need to go further, there is a serious problem here.
> 
> Most of the XID bits are read from a mapped register. They're definitely not
> expected to be null as the driver requires it to identify a proper chipset
> (the common PCI configuration registers only tell that you aren't dealing
> with a coffee machine).
> 
> Any read returning zeroes would not surprize me.
> 
> [...]
>> Can someone help me figure out what is missing ?
> 
> Hardly at this point. I can only suggest to switch power off, plug
> the 8168 device in a x86 (32 or 64 bits) system and check how it behaves.
> 


-- 
Murali Karicheri
Linux Kernel, Keystone