* pcie_get_minimum_link returns 0 width [not found] <20130825100157.GA16500@lb-tlvb-yuvalmin.il.broadcom.com> @ 2013-08-25 11:21 ` Yuval Mintz 2013-08-26 18:27 ` Keller, Jacob E 2013-08-26 22:05 ` Bjorn Helgaas 0 siblings, 2 replies; 8+ messages in thread From: Yuval Mintz @ 2013-08-25 11:21 UTC (permalink / raw) To: jacob.e.keller@intel.com Cc: bhelgaas@google.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org Hi, I tried adding support for the newly added 'pcie_get_minimum_link' into the bnx2x driver, but found out the some of my devices started showing width x0. By adding debug prints, I've found out there were devices up the chain that Showed 0 when their PCI_EXP_LNKSTA was read by said function. However, when I tried looking via lspci the output claimed the width was x4. lspci -vt output: [0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller Hub +-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]-- +-01.0-[0e-10]--+-00.0 Broadcom Corporation NetXtreme II BCM57710 10-Gigabit PCIe [Everest] Where: 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port 2 (rev 93) 09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream Port (rev 01) 0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Downstream Port E2 (rev 01) 0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57710 10-Gigabit PCIe [Everest] (rev 01) The output for "lspci -vvvv | grep LnkSta for all four is: LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- But added prints inside the function's loop show: LnkSta 1041 [000e:00.00] LnkSta 0000 [000a:01.00] LnkSta 0000 [0009:00.00] LnkSta 3041 [0000:02.00] (PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC) Thanks, Yuval ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width 2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz @ 2013-08-26 18:27 ` Keller, Jacob E 2013-08-26 22:05 ` Bjorn Helgaas 1 sibling, 0 replies; 8+ messages in thread From: Keller, Jacob E @ 2013-08-26 18:27 UTC (permalink / raw) To: Yuval Mintz Cc: bhelgaas@google.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org > -----Original Message----- > From: Yuval Mintz [mailto:yuvalmin@broadcom.com] > Sent: Sunday, August 25, 2013 4:22 AM > To: Keller, Jacob E > Cc: bhelgaas@google.com; linux-pci@vger.kernel.org; > netdev@vger.kernel.org > Subject: pcie_get_minimum_link returns 0 width > > Hi, > > I tried adding support for the newly added 'pcie_get_minimum_link' into > the > bnx2x driver, but found out the some of my devices started showing > width x0. > > By adding debug prints, I've found out there were devices up the chain > that > Showed 0 when their PCI_EXP_LNKSTA was read by said function. > However, when I tried looking via lspci the output claimed the width was > x4. > > lspci -vt output: > [0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller > Hub > +-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]-- > +-01.0-[0e-10]--+-00.0 > Broadcom > Corporation NetXtreme II > BCM57710 > 10-Gigabit PCIe [Everest] > > Where: > 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 > Port > 2 (rev 93) > 09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express > Upstream > Port (rev 01) > 0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express > Downstream Port E2 (rev 01) > 0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II > BCM57710 > 10-Gigabit PCIe [Everest] (rev 01) > > The output for "lspci -vvvv | grep LnkSta for all four is: > LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- > BWMgmt- > ABWMgmt- > > But added prints inside the function's loop show: > LnkSta 1041 [000e:00.00] > LnkSta 0000 [000a:01.00] > LnkSta 0000 [0009:00.00] > LnkSta 3041 [0000:02.00] > (PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC) > > Thanks, > Yuval Interesting... It looks like the entire LnkSta read failed for the two in the middle.. I don't know how much I can help on this issue because I don't have a machine that exhibits this symptom.. Any suggestions? Regards, Jake ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width 2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz 2013-08-26 18:27 ` Keller, Jacob E @ 2013-08-26 22:05 ` Bjorn Helgaas 2013-08-26 23:36 ` Yuval Mintz 1 sibling, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2013-08-26 22:05 UTC (permalink / raw) To: Yuval Mintz Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org On Sun, Aug 25, 2013 at 5:21 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote: > Hi, > > I tried adding support for the newly added 'pcie_get_minimum_link' into the > bnx2x driver, but found out the some of my devices started showing width x0. > > By adding debug prints, I've found out there were devices up the chain that > Showed 0 when their PCI_EXP_LNKSTA was read by said function. > However, when I tried looking via lspci the output claimed the width was x4. I don't see a 'pcie_get_minimum_link()' function in my current tree (git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next" branch). Maybe seeing your patch (with the debug prints) would give me a hook for somewhere to start looking. Can you figure out why lspci shows the correct value and this kernel interface does not? The current lspci source is in git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git and the relevant code is probably in cap_express_link() here: http://git.kernel.org/cgit/utils/pciutils/pciutils.git/tree/ls-caps.c#n750 > lspci -vt output: > [0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller Hub > +-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]-- > +-01.0-[0e-10]--+-00.0 Broadcom > Corporation NetXtreme II BCM57710 > 10-Gigabit PCIe [Everest] > > Where: > 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port > 2 (rev 93) > 09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream > Port (rev 01) > 0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express > Downstream Port E2 (rev 01) > 0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57710 > 10-Gigabit PCIe [Everest] (rev 01) > > The output for "lspci -vvvv | grep LnkSta for all four is: > LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- > ABWMgmt- > > But added prints inside the function's loop show: > LnkSta 1041 [000e:00.00] > LnkSta 0000 [000a:01.00] > LnkSta 0000 [0009:00.00] > LnkSta 3041 [0000:02.00] > (PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC) > > Thanks, > Yuval > ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width 2013-08-26 22:05 ` Bjorn Helgaas @ 2013-08-26 23:36 ` Yuval Mintz 2013-08-26 23:57 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Yuval Mintz @ 2013-08-26 23:36 UTC (permalink / raw) To: Bjorn Helgaas Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org > > Hi, > > > > I tried adding support for the newly added 'pcie_get_minimum_link' into > the > > bnx2x driver, but found out the some of my devices started showing width > x0. > > > > By adding debug prints, I've found out there were devices up the chain that > > Showed 0 when their PCI_EXP_LNKSTA was read by said function. > > However, when I tried looking via lspci the output claimed the width was > x4. > > I don't see a 'pcie_get_minimum_link()' function in my current tree > (git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next" > branch). Maybe seeing your patch (with the debug prints) would give > me a hook for somewhere to start looking. You've acked the patch and it was applied in net-next; Obviously it has yet to migrate to your tree. > > Can you figure out why lspci shows the correct value and this kernel > interface does not? The current lspci source is in > git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git and the > relevant code is probably in cap_express_link() here: > http://git.kernel.org/cgit/utils/pciutils/pciutils.git/tree/ls-caps.c#n750 Traced the read as going via the pci-sysfs. Where should I look for the bus's read ops? Thanks, Yuval ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width 2013-08-26 23:36 ` Yuval Mintz @ 2013-08-26 23:57 ` Bjorn Helgaas 2013-08-27 7:40 ` Yuval Mintz 0 siblings, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2013-08-26 23:57 UTC (permalink / raw) To: Yuval Mintz Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com> wrote: >> > Hi, >> > >> > I tried adding support for the newly added 'pcie_get_minimum_link' into >> the >> > bnx2x driver, but found out the some of my devices started showing width >> x0. >> > >> > By adding debug prints, I've found out there were devices up the chain that >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function. >> > However, when I tried looking via lspci the output claimed the width was >> x4. >> >> I don't see a 'pcie_get_minimum_link()' function in my current tree >> (git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next" >> branch). Maybe seeing your patch (with the debug prints) would give >> me a hook for somewhere to start looking. > > You've acked the patch and it was applied in net-next; Obviously it has yet > to migrate to your tree. Oh, yeah, that's right. I knew it sounded familiar, but I didn't remember where it went. Looking at its implementation, one obvious difference is that pcie_get_minimum_link() traverses up the hierarchy and keeps track of the minimum values it finds. lspci, on the other hand, just reads PCI_EXP_LNKSTA from a single device and decodes it. You said lspci reports x4 for every device from the Root Port all the way down to your NIC, so I would think the minimum from pcie_get_minimum_link() would be x4. But apparently it's not. Maybe there's a bug in it. Can you post the complete "lspci -vv" output along with your debug patch and output? Bjorn ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width 2013-08-26 23:57 ` Bjorn Helgaas @ 2013-08-27 7:40 ` Yuval Mintz 2013-08-27 13:57 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Yuval Mintz @ 2013-08-27 7:40 UTC (permalink / raw) To: Bjorn Helgaas Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org > On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com> > wrote: > >> > Hi, > >> > > >> > I tried adding support for the newly added 'pcie_get_minimum_link' into > >> the > >> > bnx2x driver, but found out the some of my devices started showing > width > >> x0. > >> > > >> > By adding debug prints, I've found out there were devices up the chain > that > >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function. > >> > However, when I tried looking via lspci the output claimed the width was > >> x4. > Looking at its implementation, one obvious difference is that > pcie_get_minimum_link() traverses up the hierarchy and keeps track of > the minimum values it finds. lspci, on the other hand, just reads > PCI_EXP_LNKSTA from a single device and decodes it. > > You said lspci reports x4 for every device from the Root Port all the > way down to your NIC, so I would think the minimum from > pcie_get_minimum_link() would be x4. But apparently it's not. Maybe > there's a bug in it. Can you post the complete "lspci -vv" output > along with your debug patch and output? > > Bjorn Here's the patch: --- drivers/pci/pci.c | 8 +++++++- 1 files changed, 7 insertions(+), 1 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index c71e78c..72cb87b 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed, enum pcie_link_width next_width; ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); - if (ret) + if (ret) { + printk(KERN_ERR "Failed to Read LNKSTA\n"); return ret; + } next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS]; next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >> PCI_EXP_LNKSTA_NLW_SHIFT; + printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n", + lnksta, dev->bus->number, PCI_SLOT(dev->devfn), + PCI_FUNC(dev->devfn)); + if (next_speed < *speed) *speed = next_speed; -- 1.7.1 And here are the lscpi results for 09:00.0 and 0a:01.0 (The two devices for which the loop returns 0 width): Sysfs do read [Pos 0, len 64] 09:00.0 Class 0604: Device 8086:3500 (rev 01) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=09, secondary=0a, subordinate=11, sec-latency=0 Memory behind bridge: fb800000-fd7fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Sysfs do read [Pos 68, len 4] [44] Express (v1) Upstream Port, MSI 00 Sysfs do read [Pos 72, len 16] DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: Sysfs do read [Pos 112, len 4] [70] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Sysfs do read [Pos 116, len 4] Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: Sysfs do read [Pos 128, len 4] [80] Sysfs do read [Pos 132, len 4] Subsystem: Device 0000:0000 Sysfs do read [Pos 256, len 4] Capabilities: [100 v1] Advanced Error Reporting Sysfs do read [Pos 260, len 24] UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr- AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport Kernel modules: shpchp Sysfs do read [Pos 0, len 64] 0a:01.0 Class 0604: Device 8086:3514 (rev 01) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Bus: primary=0a, secondary=0e, subordinate=10, sec-latency=0 Memory behind bridge: fb800000-fd7fffff Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: Sysfs do read [Pos 68, len 4] [44] Express (v1) Downstream Port (Slot-), MSI 00 Sysfs do read [Pos 72, len 16] DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag- RBE- FLReset- DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- Capabilities: Sysfs do read [Pos 96, len 4] [60] MSI: Enable- Count=1/1 Maskable- 64bit+ Sysfs do read [Pos 100, len 10] Address: 0000000000000000 Data: 0000 Capabilities: Sysfs do read [Pos 112, len 4] [70] Power Management version 2 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Sysfs do read [Pos 116, len 4] Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: Sysfs do read [Pos 128, len 4] [80] Sysfs do read [Pos 132, len 4] Subsystem: Device 0000:0000 Sysfs do read [Pos 256, len 4] Capabilities: [100 v1] Advanced Error Reporting Sysfs do read [Pos 260, len 24] UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr- AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: pcieport Kernel modules: shpchp Thanks, Yuval ^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width 2013-08-27 7:40 ` Yuval Mintz @ 2013-08-27 13:57 ` Bjorn Helgaas 2013-08-27 16:13 ` Bjorn Helgaas 0 siblings, 1 reply; 8+ messages in thread From: Bjorn Helgaas @ 2013-08-27 13:57 UTC (permalink / raw) To: Yuval Mintz Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org, Jiang Liu [+cc Jiang] On Tue, Aug 27, 2013 at 1:40 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote: >> On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com> >> wrote: >> >> > Hi, >> >> > >> >> > I tried adding support for the newly added 'pcie_get_minimum_link' into >> >> the >> >> > bnx2x driver, but found out the some of my devices started showing >> width >> >> x0. >> >> > >> >> > By adding debug prints, I've found out there were devices up the chain >> that >> >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function. >> >> > However, when I tried looking via lspci the output claimed the width was >> >> x4. >> Looking at its implementation, one obvious difference is that >> pcie_get_minimum_link() traverses up the hierarchy and keeps track of >> the minimum values it finds. lspci, on the other hand, just reads >> PCI_EXP_LNKSTA from a single device and decodes it. >> >> You said lspci reports x4 for every device from the Root Port all the >> way down to your NIC, so I would think the minimum from >> pcie_get_minimum_link() would be x4. But apparently it's not. Maybe >> there's a bug in it. Can you post the complete "lspci -vv" output >> along with your debug patch and output? >> >> Bjorn > > Here's the patch: > --- > drivers/pci/pci.c | 8 +++++++- > 1 files changed, 7 insertions(+), 1 deletions(-) > > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c > index c71e78c..72cb87b 100644 > --- a/drivers/pci/pci.c > +++ b/drivers/pci/pci.c > @@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed, > enum pcie_link_width next_width; > > ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); > - if (ret) > + if (ret) { > + printk(KERN_ERR "Failed to Read LNKSTA\n"); > return ret; > + } > > next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS]; > next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >> > PCI_EXP_LNKSTA_NLW_SHIFT; > > + printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n", > + lnksta, dev->bus->number, PCI_SLOT(dev->devfn), > + PCI_FUNC(dev->devfn)); I think pcie_cap_has_lnkctl() is incorrect: it currently thinks only Root Ports, Endpoints, and Legacy Endpoints have link registers. But I think switch ports (Upstream Ports and Downstream Ports) also have link registers (PCIe spec r3.0, sec 7.8). Jiang? > if (next_speed < *speed) > *speed = next_speed; > > -- > 1.7.1 > > And here are the lscpi results for 09:00.0 and 0a:01.0 > (The two devices for which the loop returns 0 width): > > Sysfs do read [Pos 0, len 64] > 09:00.0 Class 0604: Device 8086:3500 (rev 01) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Bus: primary=09, secondary=0a, subordinate=11, sec-latency=0 > Memory behind bridge: fb800000-fd7fffff > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: Sysfs do read [Pos 68, len 4] > [44] Express (v1) Upstream Port, MSI 00 > Sysfs do read [Pos 72, len 16] > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us > ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-SlotPowerLimit 0.000W > DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 256 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend- > LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > Capabilities: Sysfs do read [Pos 112, len 4] > [70] Power Management version 2 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Sysfs do read [Pos 116, len 4] > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: Sysfs do read [Pos 128, len 4] > [80] Sysfs do read [Pos 132, len 4] > Subsystem: Device 0000:0000 > Sysfs do read [Pos 256, len 4] > Capabilities: [100 v1] Advanced Error Reporting > Sysfs do read [Pos 260, len 24] > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr- > AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn- > Kernel driver in use: pcieport > Kernel modules: shpchp > > Sysfs do read [Pos 0, len 64] > 0a:01.0 Class 0604: Device 8086:3514 (rev 01) > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- > Latency: 0, Cache Line Size: 64 bytes > Bus: primary=0a, secondary=0e, subordinate=10, sec-latency=0 > Memory behind bridge: fb800000-fd7fffff > Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- > BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B- > PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- > Capabilities: Sysfs do read [Pos 68, len 4] > [44] Express (v1) Downstream Port (Slot-), MSI 00 > Sysfs do read [Pos 72, len 16] > DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us > ExtTag- RBE- FLReset- > DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- > RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- > MaxPayload 256 bytes, MaxReadReq 4096 bytes > DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- > LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited > ClockPM- Surprise- LLActRep- BwNot- > LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk- > ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- > LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- > Capabilities: Sysfs do read [Pos 96, len 4] > [60] MSI: Enable- Count=1/1 Maskable- 64bit+ > Sysfs do read [Pos 100, len 10] > Address: 0000000000000000 Data: 0000 > Capabilities: Sysfs do read [Pos 112, len 4] > [70] Power Management version 2 > Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) > Sysfs do read [Pos 116, len 4] > Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- > Capabilities: Sysfs do read [Pos 128, len 4] > [80] Sysfs do read [Pos 132, len 4] > Subsystem: Device 0000:0000 > Sysfs do read [Pos 256, len 4] > Capabilities: [100 v1] Advanced Error Reporting > Sysfs do read [Pos 260, len 24] > UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- > UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- > UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol- > CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- > CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr- > AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- > Kernel driver in use: pcieport > Kernel modules: shpchp > > Thanks, > Yuval > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width 2013-08-27 13:57 ` Bjorn Helgaas @ 2013-08-27 16:13 ` Bjorn Helgaas 0 siblings, 0 replies; 8+ messages in thread From: Bjorn Helgaas @ 2013-08-27 16:13 UTC (permalink / raw) To: Yuval Mintz Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org, netdev@vger.kernel.org, Jiang Liu On Tue, Aug 27, 2013 at 07:57:20AM -0600, Bjorn Helgaas wrote: > [+cc Jiang] > > On Tue, Aug 27, 2013 at 1:40 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote: > >> On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com> > >> wrote: > >> >> > Hi, > >> >> > > >> >> > I tried adding support for the newly added 'pcie_get_minimum_link' into > >> >> the > >> >> > bnx2x driver, but found out the some of my devices started showing > >> width > >> >> x0. > >> >> > > >> >> > By adding debug prints, I've found out there were devices up the chain > >> that > >> >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function. > >> >> > However, when I tried looking via lspci the output claimed the width was > >> >> x4. > >> Looking at its implementation, one obvious difference is that > >> pcie_get_minimum_link() traverses up the hierarchy and keeps track of > >> the minimum values it finds. lspci, on the other hand, just reads > >> PCI_EXP_LNKSTA from a single device and decodes it. > >> > >> You said lspci reports x4 for every device from the Root Port all the > >> way down to your NIC, so I would think the minimum from > >> pcie_get_minimum_link() would be x4. But apparently it's not. Maybe > >> there's a bug in it. Can you post the complete "lspci -vv" output > >> along with your debug patch and output? > >> > >> Bjorn > > > > Here's the patch: > > --- > > drivers/pci/pci.c | 8 +++++++- > > 1 files changed, 7 insertions(+), 1 deletions(-) > > > > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c > > index c71e78c..72cb87b 100644 > > --- a/drivers/pci/pci.c > > +++ b/drivers/pci/pci.c > > @@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed, > > enum pcie_link_width next_width; > > > > ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta); > > - if (ret) > > + if (ret) { > > + printk(KERN_ERR "Failed to Read LNKSTA\n"); > > return ret; > > + } > > > > next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS]; > > next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >> > > PCI_EXP_LNKSTA_NLW_SHIFT; > > > > + printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n", > > + lnksta, dev->bus->number, PCI_SLOT(dev->devfn), > > + PCI_FUNC(dev->devfn)); > > I think pcie_cap_has_lnkctl() is incorrect: it currently thinks only > Root Ports, Endpoints, and Legacy Endpoints have link registers. But > I think switch ports (Upstream Ports and Downstream Ports) also have > link registers (PCIe spec r3.0, sec 7.8). Jiang? Yuval, can you try the patch below? PCI: Allow access to link-related registers for switch and bridge ports From: Bjorn Helgaas <bhelgaas@google.com> Every PCIe device has a link, except Root Complex Integrated Endpoints and Root Complex Event Collectors. Previously we didn't give access to PCIe capability link-related registers for Upstream Ports, Downstream Ports, and bridges, so attempts to read PCI_EXP_LNKCTL returned zero. See PCIe spec r3.0, sec 7.8 and 1.3.2.3. Reported-by: Yuval Mintz <yuvalmin@broadcom.com> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com> --- drivers/pci/access.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/pci/access.c b/drivers/pci/access.c index 1cc2366..46dd5ad 100644 --- a/drivers/pci/access.c +++ b/drivers/pci/access.c @@ -485,9 +485,8 @@ static inline bool pcie_cap_has_lnkctl(const struct pci_dev *dev) int type = pci_pcie_type(dev); return pcie_cap_version(dev) > 1 || - type == PCI_EXP_TYPE_ROOT_PORT || - type == PCI_EXP_TYPE_ENDPOINT || - type == PCI_EXP_TYPE_LEG_END; + !(type == PCI_EXP_TYPE_RC_END || + type == PCI_EXP_TYPE_RC_EC); } static inline bool pcie_cap_has_sltctl(const struct pci_dev *dev) ^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-08-27 16:13 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20130825100157.GA16500@lb-tlvb-yuvalmin.il.broadcom.com>
2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz
2013-08-26 18:27 ` Keller, Jacob E
2013-08-26 22:05 ` Bjorn Helgaas
2013-08-26 23:36 ` Yuval Mintz
2013-08-26 23:57 ` Bjorn Helgaas
2013-08-27 7:40 ` Yuval Mintz
2013-08-27 13:57 ` Bjorn Helgaas
2013-08-27 16:13 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).