* pcie_get_minimum_link returns 0 width
[not found] <20130825100157.GA16500@lb-tlvb-yuvalmin.il.broadcom.com>
@ 2013-08-25 11:21 ` Yuval Mintz
2013-08-26 18:27 ` Keller, Jacob E
2013-08-26 22:05 ` Bjorn Helgaas
0 siblings, 2 replies; 8+ messages in thread
From: Yuval Mintz @ 2013-08-25 11:21 UTC (permalink / raw)
To: jacob.e.keller@intel.com
Cc: bhelgaas@google.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
Hi,
I tried adding support for the newly added 'pcie_get_minimum_link' into the
bnx2x driver, but found out the some of my devices started showing width x0.
By adding debug prints, I've found out there were devices up the chain that
Showed 0 when their PCI_EXP_LNKSTA was read by said function.
However, when I tried looking via lspci the output claimed the width was x4.
lspci -vt output:
[0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller Hub
+-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]--
+-01.0-[0e-10]--+-00.0 Broadcom
Corporation NetXtreme II BCM57710
10-Gigabit PCIe [Everest]
Where:
00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port
2 (rev 93)
09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream
Port (rev 01)
0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
Downstream Port E2 (rev 01)
0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57710
10-Gigabit PCIe [Everest] (rev 01)
The output for "lspci -vvvv | grep LnkSta for all four is:
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt-
ABWMgmt-
But added prints inside the function's loop show:
LnkSta 1041 [000e:00.00]
LnkSta 0000 [000a:01.00]
LnkSta 0000 [0009:00.00]
LnkSta 3041 [0000:02.00]
(PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC)
Thanks,
Yuval
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width
2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz
@ 2013-08-26 18:27 ` Keller, Jacob E
2013-08-26 22:05 ` Bjorn Helgaas
1 sibling, 0 replies; 8+ messages in thread
From: Keller, Jacob E @ 2013-08-26 18:27 UTC (permalink / raw)
To: Yuval Mintz
Cc: bhelgaas@google.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
> -----Original Message-----
> From: Yuval Mintz [mailto:yuvalmin@broadcom.com]
> Sent: Sunday, August 25, 2013 4:22 AM
> To: Keller, Jacob E
> Cc: bhelgaas@google.com; linux-pci@vger.kernel.org;
> netdev@vger.kernel.org
> Subject: pcie_get_minimum_link returns 0 width
>
> Hi,
>
> I tried adding support for the newly added 'pcie_get_minimum_link' into
> the
> bnx2x driver, but found out the some of my devices started showing
> width x0.
>
> By adding debug prints, I've found out there were devices up the chain
> that
> Showed 0 when their PCI_EXP_LNKSTA was read by said function.
> However, when I tried looking via lspci the output claimed the width was
> x4.
>
> lspci -vt output:
> [0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller
> Hub
> +-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]--
> +-01.0-[0e-10]--+-00.0
> Broadcom
> Corporation NetXtreme II
> BCM57710
> 10-Gigabit PCIe [Everest]
>
> Where:
> 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4
> Port
> 2 (rev 93)
> 09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Upstream
> Port (rev 01)
> 0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E2 (rev 01)
> 0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II
> BCM57710
> 10-Gigabit PCIe [Everest] (rev 01)
>
> The output for "lspci -vvvv | grep LnkSta for all four is:
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive-
> BWMgmt-
> ABWMgmt-
>
> But added prints inside the function's loop show:
> LnkSta 1041 [000e:00.00]
> LnkSta 0000 [000a:01.00]
> LnkSta 0000 [0009:00.00]
> LnkSta 3041 [0000:02.00]
> (PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC)
>
> Thanks,
> Yuval
Interesting... It looks like the entire LnkSta read failed for the two in the middle.. I don't know how much I can help on this issue because I don't have a machine that exhibits this symptom.. Any suggestions?
Regards,
Jake
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width
2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz
2013-08-26 18:27 ` Keller, Jacob E
@ 2013-08-26 22:05 ` Bjorn Helgaas
2013-08-26 23:36 ` Yuval Mintz
1 sibling, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2013-08-26 22:05 UTC (permalink / raw)
To: Yuval Mintz
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
On Sun, Aug 25, 2013 at 5:21 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote:
> Hi,
>
> I tried adding support for the newly added 'pcie_get_minimum_link' into the
> bnx2x driver, but found out the some of my devices started showing width x0.
>
> By adding debug prints, I've found out there were devices up the chain that
> Showed 0 when their PCI_EXP_LNKSTA was read by said function.
> However, when I tried looking via lspci the output claimed the width was x4.
I don't see a 'pcie_get_minimum_link()' function in my current tree
(git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next"
branch). Maybe seeing your patch (with the debug prints) would give
me a hook for somewhere to start looking.
Can you figure out why lspci shows the correct value and this kernel
interface does not? The current lspci source is in
git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git and the
relevant code is probably in cap_express_link() here:
http://git.kernel.org/cgit/utils/pciutils/pciutils.git/tree/ls-caps.c#n750
> lspci -vt output:
> [0000:00]-+-00.0 Intel Corporation 5000P Chipset Memory Controller Hub
> +-02.0-[09-12]--+-00.0-[0a-11]--+-00.0-[0b-0d]--
> +-01.0-[0e-10]--+-00.0 Broadcom
> Corporation NetXtreme II BCM57710
> 10-Gigabit PCIe [Everest]
>
> Where:
> 00:02.0 PCI bridge: Intel Corporation 5000 Series Chipset PCI Express x4 Port
> 2 (rev 93)
> 09:00.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express Upstream
> Port (rev 01)
> 0a:01.0 PCI bridge: Intel Corporation 6311ESB/6321ESB PCI Express
> Downstream Port E2 (rev 01)
> 0e:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57710
> 10-Gigabit PCIe [Everest] (rev 01)
>
> The output for "lspci -vvvv | grep LnkSta for all four is:
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt-
> ABWMgmt-
>
> But added prints inside the function's loop show:
> LnkSta 1041 [000e:00.00]
> LnkSta 0000 [000a:01.00]
> LnkSta 0000 [0009:00.00]
> LnkSta 3041 [0000:02.00]
> (PCI_EXP_LNKSTA value, bus->number, PCI_SLOT, PCI_FUNC)
>
> Thanks,
> Yuval
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width
2013-08-26 22:05 ` Bjorn Helgaas
@ 2013-08-26 23:36 ` Yuval Mintz
2013-08-26 23:57 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Yuval Mintz @ 2013-08-26 23:36 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
> > Hi,
> >
> > I tried adding support for the newly added 'pcie_get_minimum_link' into
> the
> > bnx2x driver, but found out the some of my devices started showing width
> x0.
> >
> > By adding debug prints, I've found out there were devices up the chain that
> > Showed 0 when their PCI_EXP_LNKSTA was read by said function.
> > However, when I tried looking via lspci the output claimed the width was
> x4.
>
> I don't see a 'pcie_get_minimum_link()' function in my current tree
> (git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next"
> branch). Maybe seeing your patch (with the debug prints) would give
> me a hook for somewhere to start looking.
You've acked the patch and it was applied in net-next; Obviously it has yet
to migrate to your tree.
>
> Can you figure out why lspci shows the correct value and this kernel
> interface does not? The current lspci source is in
> git://git.kernel.org/pub/scm/utils/pciutils/pciutils.git and the
> relevant code is probably in cap_express_link() here:
> http://git.kernel.org/cgit/utils/pciutils/pciutils.git/tree/ls-caps.c#n750
Traced the read as going via the pci-sysfs. Where should I look for the bus's
read ops?
Thanks,
Yuval
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width
2013-08-26 23:36 ` Yuval Mintz
@ 2013-08-26 23:57 ` Bjorn Helgaas
2013-08-27 7:40 ` Yuval Mintz
0 siblings, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2013-08-26 23:57 UTC (permalink / raw)
To: Yuval Mintz
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com> wrote:
>> > Hi,
>> >
>> > I tried adding support for the newly added 'pcie_get_minimum_link' into
>> the
>> > bnx2x driver, but found out the some of my devices started showing width
>> x0.
>> >
>> > By adding debug prints, I've found out there were devices up the chain that
>> > Showed 0 when their PCI_EXP_LNKSTA was read by said function.
>> > However, when I tried looking via lspci the output claimed the width was
>> x4.
>>
>> I don't see a 'pcie_get_minimum_link()' function in my current tree
>> (git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci.git "next"
>> branch). Maybe seeing your patch (with the debug prints) would give
>> me a hook for somewhere to start looking.
>
> You've acked the patch and it was applied in net-next; Obviously it has yet
> to migrate to your tree.
Oh, yeah, that's right. I knew it sounded familiar, but I didn't
remember where it went.
Looking at its implementation, one obvious difference is that
pcie_get_minimum_link() traverses up the hierarchy and keeps track of
the minimum values it finds. lspci, on the other hand, just reads
PCI_EXP_LNKSTA from a single device and decodes it.
You said lspci reports x4 for every device from the Root Port all the
way down to your NIC, so I would think the minimum from
pcie_get_minimum_link() would be x4. But apparently it's not. Maybe
there's a bug in it. Can you post the complete "lspci -vv" output
along with your debug patch and output?
Bjorn
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: pcie_get_minimum_link returns 0 width
2013-08-26 23:57 ` Bjorn Helgaas
@ 2013-08-27 7:40 ` Yuval Mintz
2013-08-27 13:57 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Yuval Mintz @ 2013-08-27 7:40 UTC (permalink / raw)
To: Bjorn Helgaas
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org
> On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com>
> wrote:
> >> > Hi,
> >> >
> >> > I tried adding support for the newly added 'pcie_get_minimum_link' into
> >> the
> >> > bnx2x driver, but found out the some of my devices started showing
> width
> >> x0.
> >> >
> >> > By adding debug prints, I've found out there were devices up the chain
> that
> >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function.
> >> > However, when I tried looking via lspci the output claimed the width was
> >> x4.
> Looking at its implementation, one obvious difference is that
> pcie_get_minimum_link() traverses up the hierarchy and keeps track of
> the minimum values it finds. lspci, on the other hand, just reads
> PCI_EXP_LNKSTA from a single device and decodes it.
>
> You said lspci reports x4 for every device from the Root Port all the
> way down to your NIC, so I would think the minimum from
> pcie_get_minimum_link() would be x4. But apparently it's not. Maybe
> there's a bug in it. Can you post the complete "lspci -vv" output
> along with your debug patch and output?
>
> Bjorn
Here's the patch:
---
drivers/pci/pci.c | 8 +++++++-
1 files changed, 7 insertions(+), 1 deletions(-)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index c71e78c..72cb87b 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
enum pcie_link_width next_width;
ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
- if (ret)
+ if (ret) {
+ printk(KERN_ERR "Failed to Read LNKSTA\n");
return ret;
+ }
next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS];
next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >>
PCI_EXP_LNKSTA_NLW_SHIFT;
+ printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n",
+ lnksta, dev->bus->number, PCI_SLOT(dev->devfn),
+ PCI_FUNC(dev->devfn));
+
if (next_speed < *speed)
*speed = next_speed;
--
1.7.1
And here are the lscpi results for 09:00.0 and 0a:01.0
(The two devices for which the loop returns 0 width):
Sysfs do read [Pos 0, len 64]
09:00.0 Class 0604: Device 8086:3500 (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=09, secondary=0a, subordinate=11, sec-latency=0
Memory behind bridge: fb800000-fd7fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: Sysfs do read [Pos 68, len 4]
[44] Express (v1) Upstream Port, MSI 00
Sysfs do read [Pos 72, len 16]
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-SlotPowerLimit 0.000W
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: Sysfs do read [Pos 112, len 4]
[70] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Sysfs do read [Pos 116, len 4]
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: Sysfs do read [Pos 128, len 4]
[80] Sysfs do read [Pos 132, len 4]
Subsystem: Device 0000:0000
Sysfs do read [Pos 256, len 4]
Capabilities: [100 v1] Advanced Error Reporting
Sysfs do read [Pos 260, len 24]
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: pcieport
Kernel modules: shpchp
Sysfs do read [Pos 0, len 64]
0a:01.0 Class 0604: Device 8086:3514 (rev 01)
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Bus: primary=0a, secondary=0e, subordinate=10, sec-latency=0
Memory behind bridge: fb800000-fd7fffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: Sysfs do read [Pos 68, len 4]
[44] Express (v1) Downstream Port (Slot-), MSI 00
Sysfs do read [Pos 72, len 16]
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
Capabilities: Sysfs do read [Pos 96, len 4]
[60] MSI: Enable- Count=1/1 Maskable- 64bit+
Sysfs do read [Pos 100, len 10]
Address: 0000000000000000 Data: 0000
Capabilities: Sysfs do read [Pos 112, len 4]
[70] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Sysfs do read [Pos 116, len 4]
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: Sysfs do read [Pos 128, len 4]
[80] Sysfs do read [Pos 132, len 4]
Subsystem: Device 0000:0000
Sysfs do read [Pos 256, len 4]
Capabilities: [100 v1] Advanced Error Reporting
Sysfs do read [Pos 260, len 24]
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
Kernel driver in use: pcieport
Kernel modules: shpchp
Thanks,
Yuval
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width
2013-08-27 7:40 ` Yuval Mintz
@ 2013-08-27 13:57 ` Bjorn Helgaas
2013-08-27 16:13 ` Bjorn Helgaas
0 siblings, 1 reply; 8+ messages in thread
From: Bjorn Helgaas @ 2013-08-27 13:57 UTC (permalink / raw)
To: Yuval Mintz
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org, Jiang Liu
[+cc Jiang]
On Tue, Aug 27, 2013 at 1:40 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote:
>> On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com>
>> wrote:
>> >> > Hi,
>> >> >
>> >> > I tried adding support for the newly added 'pcie_get_minimum_link' into
>> >> the
>> >> > bnx2x driver, but found out the some of my devices started showing
>> width
>> >> x0.
>> >> >
>> >> > By adding debug prints, I've found out there were devices up the chain
>> that
>> >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function.
>> >> > However, when I tried looking via lspci the output claimed the width was
>> >> x4.
>> Looking at its implementation, one obvious difference is that
>> pcie_get_minimum_link() traverses up the hierarchy and keeps track of
>> the minimum values it finds. lspci, on the other hand, just reads
>> PCI_EXP_LNKSTA from a single device and decodes it.
>>
>> You said lspci reports x4 for every device from the Root Port all the
>> way down to your NIC, so I would think the minimum from
>> pcie_get_minimum_link() would be x4. But apparently it's not. Maybe
>> there's a bug in it. Can you post the complete "lspci -vv" output
>> along with your debug patch and output?
>>
>> Bjorn
>
> Here's the patch:
> ---
> drivers/pci/pci.c | 8 +++++++-
> 1 files changed, 7 insertions(+), 1 deletions(-)
>
> diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> index c71e78c..72cb87b 100644
> --- a/drivers/pci/pci.c
> +++ b/drivers/pci/pci.c
> @@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
> enum pcie_link_width next_width;
>
> ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
> - if (ret)
> + if (ret) {
> + printk(KERN_ERR "Failed to Read LNKSTA\n");
> return ret;
> + }
>
> next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS];
> next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >>
> PCI_EXP_LNKSTA_NLW_SHIFT;
>
> + printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n",
> + lnksta, dev->bus->number, PCI_SLOT(dev->devfn),
> + PCI_FUNC(dev->devfn));
I think pcie_cap_has_lnkctl() is incorrect: it currently thinks only
Root Ports, Endpoints, and Legacy Endpoints have link registers. But
I think switch ports (Upstream Ports and Downstream Ports) also have
link registers (PCIe spec r3.0, sec 7.8). Jiang?
> if (next_speed < *speed)
> *speed = next_speed;
>
> --
> 1.7.1
>
> And here are the lscpi results for 09:00.0 and 0a:01.0
> (The two devices for which the loop returns 0 width):
>
> Sysfs do read [Pos 0, len 64]
> 09:00.0 Class 0604: Device 8086:3500 (rev 01)
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Bus: primary=09, secondary=0a, subordinate=11, sec-latency=0
> Memory behind bridge: fb800000-fd7fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: Sysfs do read [Pos 68, len 4]
> [44] Express (v1) Upstream Port, MSI 00
> Sysfs do read [Pos 72, len 16]
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-SlotPowerLimit 0.000W
> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 256 bytes, MaxReadReq 4096 bytes
> DevSta: CorrErr- UncorrErr- FatalErr+ UnsuppReq+ AuxPwr- TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 unlimited, L1 unlimited
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> Capabilities: Sysfs do read [Pos 112, len 4]
> [70] Power Management version 2
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Sysfs do read [Pos 116, len 4]
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: Sysfs do read [Pos 128, len 4]
> [80] Sysfs do read [Pos 132, len 4]
> Subsystem: Device 0000:0000
> Sysfs do read [Pos 256, len 4]
> Capabilities: [100 v1] Advanced Error Reporting
> Sysfs do read [Pos 260, len 24]
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
> UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
> AERCap: First Error Pointer: 14, GenCap- CGenEn- ChkCap- ChkEn-
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> Sysfs do read [Pos 0, len 64]
> 0a:01.0 Class 0604: Device 8086:3514 (rev 01)
> Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0, Cache Line Size: 64 bytes
> Bus: primary=0a, secondary=0e, subordinate=10, sec-latency=0
> Memory behind bridge: fb800000-fd7fffff
> Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
> BridgeCtl: Parity+ SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
> PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
> Capabilities: Sysfs do read [Pos 68, len 4]
> [44] Express (v1) Downstream Port (Slot-), MSI 00
> Sysfs do read [Pos 72, len 16]
> DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
> ExtTag- RBE- FLReset-
> DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
> MaxPayload 256 bytes, MaxReadReq 4096 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
> LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L0s, Latency L0 unlimited, L1 unlimited
> ClockPM- Surprise- LLActRep- BwNot-
> LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
> Capabilities: Sysfs do read [Pos 96, len 4]
> [60] MSI: Enable- Count=1/1 Maskable- 64bit+
> Sysfs do read [Pos 100, len 10]
> Address: 0000000000000000 Data: 0000
> Capabilities: Sysfs do read [Pos 112, len 4]
> [70] Power Management version 2
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
> Sysfs do read [Pos 116, len 4]
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: Sysfs do read [Pos 128, len 4]
> [80] Sysfs do read [Pos 132, len 4]
> Subsystem: Device 0000:0000
> Sysfs do read [Pos 256, len 4]
> Capabilities: [100 v1] Advanced Error Reporting
> Sysfs do read [Pos 260, len 24]
> UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
> UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
> UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC- UnsupReq+ ACSViol-
> CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
> CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr-
> AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
> Kernel driver in use: pcieport
> Kernel modules: shpchp
>
> Thanks,
> Yuval
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pcie_get_minimum_link returns 0 width
2013-08-27 13:57 ` Bjorn Helgaas
@ 2013-08-27 16:13 ` Bjorn Helgaas
0 siblings, 0 replies; 8+ messages in thread
From: Bjorn Helgaas @ 2013-08-27 16:13 UTC (permalink / raw)
To: Yuval Mintz
Cc: jacob.e.keller@intel.com, linux-pci@vger.kernel.org,
netdev@vger.kernel.org, Jiang Liu
On Tue, Aug 27, 2013 at 07:57:20AM -0600, Bjorn Helgaas wrote:
> [+cc Jiang]
>
> On Tue, Aug 27, 2013 at 1:40 AM, Yuval Mintz <yuvalmin@broadcom.com> wrote:
> >> On Mon, Aug 26, 2013 at 5:36 PM, Yuval Mintz <yuvalmin@broadcom.com>
> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I tried adding support for the newly added 'pcie_get_minimum_link' into
> >> >> the
> >> >> > bnx2x driver, but found out the some of my devices started showing
> >> width
> >> >> x0.
> >> >> >
> >> >> > By adding debug prints, I've found out there were devices up the chain
> >> that
> >> >> > Showed 0 when their PCI_EXP_LNKSTA was read by said function.
> >> >> > However, when I tried looking via lspci the output claimed the width was
> >> >> x4.
> >> Looking at its implementation, one obvious difference is that
> >> pcie_get_minimum_link() traverses up the hierarchy and keeps track of
> >> the minimum values it finds. lspci, on the other hand, just reads
> >> PCI_EXP_LNKSTA from a single device and decodes it.
> >>
> >> You said lspci reports x4 for every device from the Root Port all the
> >> way down to your NIC, so I would think the minimum from
> >> pcie_get_minimum_link() would be x4. But apparently it's not. Maybe
> >> there's a bug in it. Can you post the complete "lspci -vv" output
> >> along with your debug patch and output?
> >>
> >> Bjorn
> >
> > Here's the patch:
> > ---
> > drivers/pci/pci.c | 8 +++++++-
> > 1 files changed, 7 insertions(+), 1 deletions(-)
> >
> > diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
> > index c71e78c..72cb87b 100644
> > --- a/drivers/pci/pci.c
> > +++ b/drivers/pci/pci.c
> > @@ -3601,13 +3601,19 @@ int pcie_get_minimum_link(struct pci_dev *dev, enum pci_bus_speed *speed,
> > enum pcie_link_width next_width;
> >
> > ret = pcie_capability_read_word(dev, PCI_EXP_LNKSTA, &lnksta);
> > - if (ret)
> > + if (ret) {
> > + printk(KERN_ERR "Failed to Read LNKSTA\n");
> > return ret;
> > + }
> >
> > next_speed = pcie_link_speed[lnksta & PCI_EXP_LNKSTA_CLS];
> > next_width = (lnksta & PCI_EXP_LNKSTA_NLW) >>
> > PCI_EXP_LNKSTA_NLW_SHIFT;
> >
> > + printk(KERN_ERR "LnkSta %04x [%04x:%02x.%02x]\n",
> > + lnksta, dev->bus->number, PCI_SLOT(dev->devfn),
> > + PCI_FUNC(dev->devfn));
>
> I think pcie_cap_has_lnkctl() is incorrect: it currently thinks only
> Root Ports, Endpoints, and Legacy Endpoints have link registers. But
> I think switch ports (Upstream Ports and Downstream Ports) also have
> link registers (PCIe spec r3.0, sec 7.8). Jiang?
Yuval, can you try the patch below?
PCI: Allow access to link-related registers for switch and bridge ports
From: Bjorn Helgaas <bhelgaas@google.com>
Every PCIe device has a link, except Root Complex Integrated Endpoints
and Root Complex Event Collectors. Previously we didn't give access
to PCIe capability link-related registers for Upstream Ports, Downstream
Ports, and bridges, so attempts to read PCI_EXP_LNKCTL returned zero.
See PCIe spec r3.0, sec 7.8 and 1.3.2.3.
Reported-by: Yuval Mintz <yuvalmin@broadcom.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
---
drivers/pci/access.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/drivers/pci/access.c b/drivers/pci/access.c
index 1cc2366..46dd5ad 100644
--- a/drivers/pci/access.c
+++ b/drivers/pci/access.c
@@ -485,9 +485,8 @@ static inline bool pcie_cap_has_lnkctl(const struct pci_dev *dev)
int type = pci_pcie_type(dev);
return pcie_cap_version(dev) > 1 ||
- type == PCI_EXP_TYPE_ROOT_PORT ||
- type == PCI_EXP_TYPE_ENDPOINT ||
- type == PCI_EXP_TYPE_LEG_END;
+ !(type == PCI_EXP_TYPE_RC_END ||
+ type == PCI_EXP_TYPE_RC_EC);
}
static inline bool pcie_cap_has_sltctl(const struct pci_dev *dev)
^ permalink raw reply related [flat|nested] 8+ messages in thread
end of thread, other threads:[~2013-08-27 16:13 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20130825100157.GA16500@lb-tlvb-yuvalmin.il.broadcom.com>
2013-08-25 11:21 ` pcie_get_minimum_link returns 0 width Yuval Mintz
2013-08-26 18:27 ` Keller, Jacob E
2013-08-26 22:05 ` Bjorn Helgaas
2013-08-26 23:36 ` Yuval Mintz
2013-08-26 23:57 ` Bjorn Helgaas
2013-08-27 7:40 ` Yuval Mintz
2013-08-27 13:57 ` Bjorn Helgaas
2013-08-27 16:13 ` Bjorn Helgaas
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).