From: Gianni Tedesco <gianni.tedesco@citrix.com>
To: Sander Eikelenboom <linux@eikelenboom.it>
Cc: Xen Devel <xen-devel@lists.xensource.com>,
Stefano Stabellini <Stefano.Stabellini@eu.citrix.com>,
Bruce Edge <bruce.edge@gmail.com>,
Keir Fraser <Keir.Fraser@eu.citrix.com>
Subject: Re: MSI badness in xen-unstable
Date: Mon, 18 Oct 2010 18:16:39 +0100 [thread overview]
Message-ID: <1287422199.12843.820.camel@qabil.uk.xensource.com> (raw)
In-Reply-To: <895776665.20101016192503@eikelenboom.it>
On Sat, 2010-10-16 at 18:25 +0100, Sander Eikelenboom wrote:
> Probably there are more problems, you could also try a xen-unstable
> from before the commit that changed this code (msi.c)
> Another thing that could make it eassier to debug would be to put some
> printk's around the WARN_ON's in msi.c at the linenumbers that gave
> the warnings, showing but parts of the equation in the WARN_ON
Yes, I am still getting WARN's after the inverted masks patch too.
Bruces patch was line-wrap mangled but I instrumented the WARN that I'm
hitting based on that. The device in question is a broadcom netXtreme II
- there are two installed in the box but only one of them is brought up.
The WARN's happen when the interface is brought up for DHCP.
(XEN) =================================================
(XEN) msi->table_base != read_pci_mem_bar(bus, slot, func, bir)
(XEN) msi->table_base = da000000
(XEN) read_pci_mem_bar = 0
(XEN) bus=2, slot=0, func=0, bir=0
(XEN) =================================================
(XEN) =================================================
(XEN) No pba_addr: bus=2, slot=0, func=0, bir=0
(XEN) =================================================
The problem appears to be as simple as read_pci_mem_bar() returning
zero. This can only happen for a few possible reasons and in my case
what I got was:
pci_conf_read8(bus, slot, func, PCI_HEADER_TYPE) is not one of:
PCI_HEADER_TYPE_NORMAL
PCI_HEADER_TYPE_BRIDGE
PCI_HEADER_TYPE_CARDBUS
Thereby bailing in the switch statement. It seems that the problem here
is that the multi-function bit (0x80) was not being masked out. Does the
following patch work for you guys?
diff -r fc2242ac90e1 xen/arch/x86/msi.c
--- a/xen/arch/x86/msi.c Mon Oct 18 11:31:47 2010 +0100
+++ b/xen/arch/x86/msi.c Mon Oct 18 18:14:22 2010 +0100
@@ -527,7 +527,7 @@ static u64 read_pci_mem_bar(u8 bus, u8 s
u8 limit;
u32 addr;
- switch ( pci_conf_read8(bus, slot, func, PCI_HEADER_TYPE) )
+ switch ( pci_conf_read8(bus, slot, func, PCI_HEADER_TYPE) & 0x7f )
{
case PCI_HEADER_TYPE_NORMAL:
limit = 6;
FYI: This is function 0 of my multi-function bnx2 NIC. I notice your
affected devices were also multi-function
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
Subsystem: Dell Device 02a3
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 16
Region 0: Memory at da000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: Broadcom NetXtreme II Ethernet Controller
Read-only fields:
[PN] Part number: BCM95716C1
[EC] Engineering changes: 220197-3
[SN] Serial number: 0123456789
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: 5.0.13
[RV] Reserved: checksum good, 22 byte(s) reserved
End
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable+ Count=9 Masked-
Vector table: BAR=0 offset=0000c000
PBA: BAR=0 offset=0000e000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [100 v1] Device Serial Number a4-ba-db-ff-fe-4d-11-0b
Capabilities: [110 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: bnx2
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet (rev 20)
Subsystem: Dell Device 02a3
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 17
Region 0: Memory at dc000000 (64-bit, non-prefetchable) [size=32M]
Capabilities: [48] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
Capabilities: [50] Vital Product Data
Product Name: Broadcom NetXtreme II Ethernet Controller
Read-only fields:
[PN] Part number: BCM95716C1
[EC] Engineering changes: 220197-3
[SN] Serial number: 0123456789
[MN] Manufacture ID: 31 30 32 38
[V0] Vendor specific: 5.0.13
[RV] Reserved: checksum good, 22 byte(s) reserved
End
Capabilities: [58] MSI: Enable- Count=1/16 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [a0] MSI-X: Enable- Count=9 Masked-
Vector table: BAR=0 offset=0000c000
PBA: BAR=0 offset=0000e000
Capabilities: [ac] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr+ NoSnoop+
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L0s L1, Latency L0 <2us, L1 <2us
ClockPM- Surprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Range ABCD, TimeoutDis+
DevCtl2: Completion Timeout: 65ms to 210ms, TimeoutDis-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB
Capabilities: [100 v1] Device Serial Number a4-ba-db-ff-fe-4d-11-0c
Capabilities: [110 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
CEMsk: RxErr- BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+
AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
Capabilities: [150 v1] Power Budgeting <?>
Capabilities: [160 v1] Virtual Channel
Caps: LPEVC=0 RefClk=100ns PATEntryBits=1
Arb: Fixed- WRR32- WRR64- WRR128-
Ctrl: ArbSelect=Fixed
Status: InProgress-
VC0: Caps: PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
Arb: Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
Ctrl: Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
Status: NegoPending- InProgress-
Kernel driver in use: bnx2
next prev parent reply other threads:[~2010-10-18 17:16 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-10-08 9:33 MSI badness in xen-unstable Gianni Tedesco
2010-10-08 10:01 ` Gianni Tedesco
2010-10-08 10:03 ` Gianni Tedesco
2010-10-11 17:12 ` Gianni Tedesco
2010-10-11 21:05 ` Bruce Edge
2010-10-16 16:14 ` Bruce Edge
2010-10-16 16:29 ` Sander Eikelenboom
2010-10-16 17:14 ` Bruce Edge
2010-10-16 17:25 ` Sander Eikelenboom
2010-10-18 17:16 ` Gianni Tedesco [this message]
2010-10-18 17:29 ` Bruce Edge
2010-10-19 7:18 ` Jan Beulich
2010-10-16 17:25 ` Sander Eikelenboom
2010-10-16 17:26 ` Sander Eikelenboom
2010-10-17 20:19 ` Bruce Edge
2010-10-17 22:33 ` Bruce Edge
2010-10-18 8:24 ` Jan Beulich
2010-10-18 16:32 ` Bruce Edge
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1287422199.12843.820.camel@qabil.uk.xensource.com \
--to=gianni.tedesco@citrix.com \
--cc=Keir.Fraser@eu.citrix.com \
--cc=Stefano.Stabellini@eu.citrix.com \
--cc=bruce.edge@gmail.com \
--cc=linux@eikelenboom.it \
--cc=xen-devel@lists.xensource.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.