[PATCH] PCI: Mark broken INTx masking for BENET devices

[PATCH] PCI: Mark broken INTx masking for BENET devices

[Gavin Shan]

Similar to commit 11e4253 ("PCI: Assume all Mellanox devices have
broken INTx masking"), when passing through following PCI device
using VFIO infrastructure, interrupt storm are reported. After
marking its INTx masking is broken, the interrupt storm isn't
raised again:

 # lspci -s 0000::.
 0000:01:00.0 Ethernet controller: Emulex Corporation \
              OneConnect 10Gb NIC (be3) (rev 02)
 0000:01:00.1 Ethernet controller: Emulex Corporation \
              OneConnect 10Gb NIC (be3) (rev 02)
 # lspci -n -s 0000::.
 0000:01:00.0 0200: 19a2:0710 (rev 02)
 0000:01:00.1 0200: 19a2:0710 (rev 02)

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 drivers/pci/quirks.c    | 2 ++
 include/linux/pci_ids.h | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index ed6f89b..e823ac0 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -3027,6 +3027,8 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_REALTEK, 0x8169,
 			 quirk_broken_intx_masking);
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID,
 			 quirk_broken_intx_masking);
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BE, PCI_ANY_ID,
+			 quirk_broken_intx_masking);
 
 #ifdef CONFIG_ACPI
 /*
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index e63c02a..df70b76 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -2481,6 +2481,8 @@
 #define PCI_DEVICE_ID_KORENIX_JETCARDF2	0x1700
 #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
 
+#define PCI_VENDOR_ID_BE		0x19a2
+
 #define PCI_VENDOR_ID_QMI		0x1a32
 
 #define PCI_VENDOR_ID_AZWAVE		0x1a3b
-- 
1.8.3.2

RE: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Venkat Duvvuru]

Can someone please explain, why interrupt storm issue doesn't occur in non-VFIO scenario?

> -----Original Message-----
> From: linux-pci-owner@vger.kernel.org [mailto:linux-pci-
> owner@vger.kernel.org] On Behalf Of Gavin Shan
> Sent: Monday, January 05, 2015 7:48 AM
> To: linux-pci@vger.kernel.org
> Cc: Ajit Kumar Khaparde; bhelgaas@google.com; Gavin Shan
> Subject: [PATCH] PCI: Mark broken INTx masking for BENET devices
> 
> Similar to commit 11e4253 ("PCI: Assume all Mellanox devices have
> broken INTx masking"), when passing through following PCI device
> using VFIO infrastructure, interrupt storm are reported. After
> marking its INTx masking is broken, the interrupt storm isn't
> raised again:
> 
>  # lspci -s 0000::.
>  0000:01:00.0 Ethernet controller: Emulex Corporation \
>               OneConnect 10Gb NIC (be3) (rev 02)
>  0000:01:00.1 Ethernet controller: Emulex Corporation \
>               OneConnect 10Gb NIC (be3) (rev 02)
>  # lspci -n -s 0000::.
>  0000:01:00.0 0200: 19a2:0710 (rev 02)
>  0000:01:00.1 0200: 19a2:0710 (rev 02)
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  drivers/pci/quirks.c    | 2 ++
>  include/linux/pci_ids.h | 2 ++
>  2 files changed, 4 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index ed6f89b..e823ac0 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -3027,6 +3027,8 @@
> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_REALTEK, 0x8169,
>  			 quirk_broken_intx_masking);
>  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID,
>  			 quirk_broken_intx_masking);
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BE, PCI_ANY_ID,
> +			 quirk_broken_intx_masking);
> 
>  #ifdef CONFIG_ACPI
>  /*
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index e63c02a..df70b76 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2481,6 +2481,8 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF2	0x1700
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
> 
> +#define PCI_VENDOR_ID_BE		0x19a2
> +
>  #define PCI_VENDOR_ID_QMI		0x1a32
> 
>  #define PCI_VENDOR_ID_AZWAVE		0x1a3b
> --
> 1.8.3.2
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Alex Williamson]

On Wed, 2015-01-07 at 04:29 +0000, Venkat Duvvuru wrote:
> Can someone please explain, why interrupt storm issue doesn't occur in non-VFIO scenario?

In a host driver scenario, the driver hopefully knows how to handle the
device in a way that interrupt masking is not needed.  If you're
comparing to legacy KVM device assignment, the answer is likely that
that code supports emulation of INTx using MSI.  This is a fairly kludgy
mode of operation that I'm not all the keen on implementing in VFIO.
The pci-assign driver can be forced to not use this mode, as it's been
found not to work universally, with the option prefer_msi=off.  Thanks,

Alex

> > -----Original Message-----
> > From: linux-pci-owner@vger.kernel.org [mailto:linux-pci-
> > owner@vger.kernel.org] On Behalf Of Gavin Shan
> > Sent: Monday, January 05, 2015 7:48 AM
> > To: linux-pci@vger.kernel.org
> > Cc: Ajit Kumar Khaparde; bhelgaas@google.com; Gavin Shan
> > Subject: [PATCH] PCI: Mark broken INTx masking for BENET devices
> > 
> > Similar to commit 11e4253 ("PCI: Assume all Mellanox devices have
> > broken INTx masking"), when passing through following PCI device
> > using VFIO infrastructure, interrupt storm are reported. After
> > marking its INTx masking is broken, the interrupt storm isn't
> > raised again:
> > 
> >  # lspci -s 0000::.
> >  0000:01:00.0 Ethernet controller: Emulex Corporation \
> >               OneConnect 10Gb NIC (be3) (rev 02)
> >  0000:01:00.1 Ethernet controller: Emulex Corporation \
> >               OneConnect 10Gb NIC (be3) (rev 02)
> >  # lspci -n -s 0000::.
> >  0000:01:00.0 0200: 19a2:0710 (rev 02)
> >  0000:01:00.1 0200: 19a2:0710 (rev 02)
> > 
> > Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> > ---
> >  drivers/pci/quirks.c    | 2 ++
> >  include/linux/pci_ids.h | 2 ++
> >  2 files changed, 4 insertions(+)
> > 
> > diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> > index ed6f89b..e823ac0 100644
> > --- a/drivers/pci/quirks.c
> > +++ b/drivers/pci/quirks.c
> > @@ -3027,6 +3027,8 @@
> > DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_REALTEK, 0x8169,
> >  			 quirk_broken_intx_masking);
> >  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID,
> >  			 quirk_broken_intx_masking);
> > +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BE, PCI_ANY_ID,
> > +			 quirk_broken_intx_masking);
> > 
> >  #ifdef CONFIG_ACPI
> >  /*
> > diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> > index e63c02a..df70b76 100644
> > --- a/include/linux/pci_ids.h
> > +++ b/include/linux/pci_ids.h
> > @@ -2481,6 +2481,8 @@
> >  #define PCI_DEVICE_ID_KORENIX_JETCARDF2	0x1700
> >  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
> > 
> > +#define PCI_VENDOR_ID_BE		0x19a2
> > +
> >  #define PCI_VENDOR_ID_QMI		0x1a32
> > 
> >  #define PCI_VENDOR_ID_AZWAVE		0x1a3b
> > --
> > 1.8.3.2
> > 
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Venkat Duvvuru]

Hi Gavin,
We tried to reproduce this in our lab but our observation is that we don't see "interrupt storm" in our system.
Could you please give us the details of the repro scenario?

A few more details will help us understand the problem better as well.
1. be2net driver version
2. be2net firmware version
3. lspci -vvv output of the card (lspci -d 19a2: -vvv)
4. Is SR-IOV enabled in your card?

Thanks,
Venkat.


> -----Original Message-----
> From: linux-pci-owner@vger.kernel.org [mailto:linux-pci-
> owner@vger.kernel.org] On Behalf Of Gavin Shan
> Sent: Monday, January 05, 2015 7:48 AM
> To: linux-pci@vger.kernel.org
> Cc: Ajit Kumar Khaparde; bhelgaas@google.com; Gavin Shan
> Subject: [PATCH] PCI: Mark broken INTx masking for BENET devices
> 
> Similar to commit 11e4253 ("PCI: Assume all Mellanox devices have
> broken INTx masking"), when passing through following PCI device
> using VFIO infrastructure, interrupt storm are reported. After
> marking its INTx masking is broken, the interrupt storm isn't
> raised again:
> 
>  # lspci -s 0000::.
>  0000:01:00.0 Ethernet controller: Emulex Corporation \
>               OneConnect 10Gb NIC (be3) (rev 02)
>  0000:01:00.1 Ethernet controller: Emulex Corporation \
>               OneConnect 10Gb NIC (be3) (rev 02)
>  # lspci -n -s 0000::.
>  0000:01:00.0 0200: 19a2:0710 (rev 02)
>  0000:01:00.1 0200: 19a2:0710 (rev 02)
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  drivers/pci/quirks.c    | 2 ++
>  include/linux/pci_ids.h | 2 ++
>  2 files changed, 4 insertions(+)
> 
> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
> index ed6f89b..e823ac0 100644
> --- a/drivers/pci/quirks.c
> +++ b/drivers/pci/quirks.c
> @@ -3027,6 +3027,8 @@
> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_REALTEK, 0x8169,
>  			 quirk_broken_intx_masking);
>  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID,
>  			 quirk_broken_intx_masking);
> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BE, PCI_ANY_ID,
> +			 quirk_broken_intx_masking);
> 
>  #ifdef CONFIG_ACPI
>  /*
> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
> index e63c02a..df70b76 100644
> --- a/include/linux/pci_ids.h
> +++ b/include/linux/pci_ids.h
> @@ -2481,6 +2481,8 @@
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF2	0x1700
>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
> 
> +#define PCI_VENDOR_ID_BE		0x19a2
> +
>  #define PCI_VENDOR_ID_QMI		0x1a32
> 
>  #define PCI_VENDOR_ID_AZWAVE		0x1a3b
> --
> 1.8.3.2
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Gavin Shan]

On Thu, Jan 08, 2015 at 11:33:46AM +0000, Venkat Duvvuru wrote:
>Hi Gavin,
>We tried to reproduce this in our lab but our observation is that we don't see "interrupt storm" in our system.
>Could you please give us the details of the repro scenario?
>
>A few more details will help us understand the problem better as well.
>1. be2net driver version
>2. be2net firmware version
>3. lspci -vvv output of the card (lspci -d 19a2: -vvv)
>4. Is SR-IOV enabled in your card?
>

Venkat, On IBM's Power7 box, I passed through following adpater to guest with following
QEMU command line:

Information from host
=====================

[root@ltcfbl8eb ~]# uname -a
Linux 3.19.0-rc2+ #290 SMP Fri Jan 9 11:20:20 EST 2015 ppc64 ppc64 ppc64 GNU/Linux

[root@ltcfbl8eb ~]# lspci -vvv -s 0000:01:00.0
0000:01:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
	Subsystem: Emulex Corporation Device e733
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at 3ce080100000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at 3ce080080000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at 3ce0800a0000 (64-bit, non-prefetchable) [size=128K]
	[virtual] Expansion ROM at 3ce080000000 [disabled] [size=256K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <16us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis+ BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis+
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance+ ComplianceSOS+
			 Compliance De-emphasis: -3.5dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b8] Vital Product Data
		Product Name: IBM Flex System EN4054 4-port 10Gb Ethernet Adapter, NIC PF
		Read-only fields:
			[PN] Part number: 81Y3124
			[SN] Serial number: 11S81Y3126Y651HY2B801A
			[V0] Vendor specific: FC24495774
			[FC] Unknown: 31 37 36 32
			[EC] Engineering changes:  N28285V
			[VI] Vendor specific: 001
			[VJ] Vendor specific: 001A
			[VL] Vendor specific: 81Y3125
			[VM] Vendor specific: 3
			[VN] Vendor specific: Y651HY2B801A
			[V1] Vendor specific: Emulex 81Y3126 10Gb Ethernet Adapter (Fabric Mezz)
			[V2] Vendor specific: 81Y3126
			[V4] Vendor specific: 3
			[V5] Vendor specific: OCm11104-N-P
			[RV] Reserved: checksum good, 124 byte(s) reserved
		End
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 0, stride: 1, Device ID: 0710
		Supported Page Size: 00000557, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [160 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [168 v1] Device Serial Number 00-90-fa-ff-fe-11-fb-b4
	Capabilities: [12c v1] Transaction Processing Hints
		No steering table available
	Kernel driver in use: vfio-pci

Command line to start the guest
===============================

echo 0000:01:00.0 > /sys/bus/pci/drivers/be2net/unbind
echo 0000:01:00.1 > /sys/bus/pci/drivers/be2net/unbind
echo 19a2 0710 > /sys/bus/pci/drivers/vfio-pci/new_id

/home/gavin/qemu/ppc64-softmmu/qemu-system-ppc64 \
-M pseries -m 2048 -enable-kvm -nographic -vga none \
-device spapr-pci-vfio-host-bridge,id=Emulex,iommu=0,index=16 \
-device vfio-pci,host=0000:01:00.0,addr=1.0 \
-device vfio-pci,host=0000:01:00.1,addr=2.0 \
-boot c -hda /home/gavin/images/fc20.img

Information from guest
======================

[root@localhost ~]# cat /etc/issue.net
Fedora release 20 (Heisenbug)
Kernel \r on an \m (\l)
[root@localhost ~]# 
[root@localhost ~]# uname -a
Linux localhost.localdomain 3.11.10-301.fc20.ppc64p7 #1 SMP Tue Dec 10 00:35:14 MST 2013 ppc64 ppc64 ppc64 GNU/Linux
[root@localhost ~]# 
[root@localhost ~]# lspci
0001:00:01.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
0001:00:02.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
[root@localhost ~]# lspci -vvv -s 0001:00:01.0
0001:00:01.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02)
	Subsystem: Emulex Corporation Device e733
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 17
	Region 0: Memory at 200b0000000 (64-bit, non-prefetchable) [size=16K]
	Region 2: Memory at 200b0020000 (64-bit, non-prefetchable) [size=128K]
	Region 4: Memory at 200b0040000 (64-bit, non-prefetchable) [size=128K]
	Expansion ROM at 200b0080000 [disabled] [size=256K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI+ D1- D2- AuxCurrent=375mA PME(D0-,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI-X: Enable+ Count=32 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [c0] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <16us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
		DevCtl:	Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported+
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr+ TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis+ BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis+, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b8] Vital Product Data
		Product Name: IBM Flex System EN4054 4-port 10Gb Ethernet Adapter, NIC PF
		Read-only fields:
			[PN] Part number: 81Y3124
			[SN] Serial number: 11S81Y3126Y651HY2B801A
			[V0] Vendor specific: FC24495774
			[FC] Unknown: 31 37 36 32
			[EC] Engineering changes:  N28285V
			[VI] Vendor specific: 001
			[VJ] Vendor specific: 001A
			[VL] Vendor specific: 81Y3125
			[VM] Vendor specific: 3
			[VN] Vendor specific: Y651HY2B801A
			[V1] Vendor specific: Emulex 81Y3126 10Gb Ethernet Adapter (Fabric Mezz)
			[V2] Vendor specific: 81Y3126
			[V4] Vendor specific: 3
			[V5] Vendor specific: OCm11104-N-P
			[RV] Reserved: checksum good, 124 byte(s) reserved
		End
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn+ ChkCap+ ChkEn+
	Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
		IOVCap:	Migration-, Interrupt Message Number: 000
		IOVCtl:	Enable- Migration- Interrupt- MSE- ARIHierarchy-
		IOVSta:	Migration-
		Initial VFs: 0, Total VFs: 0, Number of VFs: 0, Function Dependency Link: 00
		VF offset: 0, stride: 1, Device ID: 0710
		Supported Page Size: 00000557, System Page Size: 00000001
		Region 0: Memory at 0000000000000000 (64-bit, non-prefetchable)
		VF Migration: offset: 00000000, BIR: 0
	Capabilities: [160 v1] Alternative Routing-ID Interpretation (ARI)
		ARICap:	MFVC- ACS-, Next Function: 1
		ARICtl:	MFVC- ACS-, Function Group: 0
	Capabilities: [168 v1] Device Serial Number 00-90-fa-ff-fe-11-fb-b4
	Capabilities: [12c v1] Transaction Processing Hints
		No steering table available
	Kernel driver in use: be2net

[root@localhost ~]# ethtool -i enP1p0s1
driver: be2net
version: 4.6.62.0u
firmware-version: 4.1.422.0
bus-info: 0001:00:01.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Steps to recreate the issue
===========================

1. Configure the NIC and ping it from external.
2. Inject EEH error by running following command in host side. After about
   20 seconds, I got following message from host side. With the patch applied
   to host kernel, I didn't see the warning messages:

   host# echo 1:0:0:0:0:0 > /sys/kernel/debug/powerpc/PCI0000/err_injct

   :
[  419.083164] P7IOC PHB#0 Diag-data (Version: 1)
[  419.083229] brdgCtl:     00000002
[  419.083300] UtlSts:      00000000 40000000 00000000
[  419.083347] RootSts:     0000004f 016003c0 a0820008 40100147 00004000
[  419.083404] RootErrSts:  00000024 00000000 00000000
[  419.083451] RootErrLog1: 00010000 0000000000000000 0000000000000000
[  419.083508] PhbSts:      0000001c00000000 0000001c00000000
[  419.083555] Lem:         0200000000000004 1249a1147f500f2c 0000000000000000
[  419.083613] PhbErr:      0000000020000000 0000000020000000 0000000000000000 000002b0f8000000
[  419.083692] OutErr:      0200000000000000 0200000000000000 1204ce7000443ce0 800a012000000000
[  419.083772] PE[  1] A/B: 8720003d01000000 8000000000000000
[  419.249183] irq 17: nobody cared (try booting with the "irqpoll" option)
[  419.249252] CPU: 44 PID: 2488 Comm: qemu-system-ppc Not tainted 3.19.0-rc2+ #290
[  419.249259] Call Trace:
[  419.249268] [c000000f2ee63ae0] [c0000000008360d0] .dump_stack+0x88/0xa8 (unreliable)
[  419.249280] [c000000f2ee63b60] [c0000000000d7e80] .__report_bad_irq+0x44/0x104
[  419.249288] [c000000f2ee63c10] [c0000000000d848c] .note_interrupt+0x24c/0x304
[  419.249359] [c000000f2ee63cc0] [c0000000000d56d4] .handle_irq_event_percpu+0x1c4/0x220
[  419.249439] [c000000f2ee63d90] [c0000000000d579c] .handle_irq_event+0x6c/0x98
[  419.249509] [c000000f2ee63e10] [c0000000000d9418] .handle_fasteoi_irq+0xc4/0x184
[  419.249589] [c000000f2ee63e90] [c0000000000d4ad4] .generic_handle_irq+0x48/0x60
[  419.249671] [c000000f2ee63f10] [c00000000000eeac] .__do_irq+0xcc/0x168
[  419.249741] [c000000f2ee63f90] [c00000000001c7d8] .call_do_irq+0x14/0x24
[  419.249811] [c0000007f35ef2f0] [c00000000000efd4] .do_IRQ+0x8c/0xc8
[  419.249881] [c0000007f35ef3a0] [c0000000000022b0] hardware_interrupt_common+0x130/0x180
[  419.249964] --- interrupt: 501 at .__kvmppc_vcore_entry+0x13c/0x1a0
    LR = kvmppc_call_hv_entry+0x8/0x118
[  419.250056] [c0000007f35ef690] [c00000000007080c] .__kvmppc_vcore_entry+0x13c/0x1a0 (unreliable)
[  419.250149] [c0000007f35ef860] [c00000000006fd30] .kvmppc_vcpu_run_hv+0xa68/0x1408
[  419.250230] [c0000007f35ef9f0] [c0000000000682e0] .kvmppc_vcpu_run+0x44/0x58
[  419.250300] [c0000007f35efa70] [c0000000000662cc] .kvm_arch_vcpu_ioctl_run+0xfc/0x130
[  419.250382] [c0000007f35efb00] [c00000000005fb18] .kvm_vcpu_ioctl+0x1a8/0x5f4
[  419.250452] [c0000007f35efcb0] [c0000000001e8e98] .do_vfs_ioctl+0x618/0x69c
[  419.250521] [c0000007f35efd90] [c0000000001e8f8c] .SyS_ioctl+0x70/0x9c
[  419.250592] [c0000007f35efe30] [c000000000009198] syscall_exit+0x0/0x98
[  419.250661] handlers:
[  419.250687] [<c000000001586560>] .vfio_intx_handler
[  419.250744] Disabling IRQ #17
[  419.422125] irq 18: nobody cared (try booting with the "irqpoll" option)
[  419.422208] CPU: 24 PID: 2488 Comm: qemu-system-ppc Not tainted 3.19.0-rc2+ #290
[  419.422215] Call Trace:
[  419.422221] [c000000f2ef03ae0] [c0000000008360d0] .dump_stack+0x88/0xa8 (unreliable)
[  419.422243] [c000000f2ef03b60] [c0000000000d7e80] .__report_bad_irq+0x44/0x104
[  419.422324] [c000000f2ef03c10] [c0000000000d848c] .note_interrupt+0x24c/0x304
[  419.422393] [c000000f2ef03cc0] [c0000000000d56d4] .handle_irq_event_percpu+0x1c4/0x220
[  419.422475] [c000000f2ef03d90] [c0000000000d579c] .handle_irq_event+0x6c/0x98
[  419.422545] [c000000f2ef03e10] [c0000000000d9418] .handle_fasteoi_irq+0xc4/0x184
[  419.422628] [c000000f2ef03e90] [c0000000000d4ad4] .generic_handle_irq+0x48/0x60
[  419.422710] [c000000f2ef03f10] [c00000000000eeac] .__do_irq+0xcc/0x168
[  419.422780] [c000000f2ef03f90] [c00000000001c7d8] .call_do_irq+0x14/0x24
[  419.422849] [c0000007f35ef450] [c00000000000efd4] .do_IRQ+0x8c/0xc8
[  419.422919] [c0000007f35ef500] [c0000000000022b0] hardware_interrupt_common+0x130/0x180
[  419.423000] --- interrupt: 501 at .arch_local_irq_restore+0x54/0x78
    LR = .arch_local_irq_restore+0x54/0x78
[  419.423105] [c0000007f35ef7f0] [c0000007f35ef890] 0xc0000007f35ef890 (unreliable)
[  419.423186] [c0000007f35ef860] [c00000000006fce4] .kvmppc_vcpu_run_hv+0xa1c/0x1408
[  419.423267] [c0000007f35ef9f0] [c0000000000682e0] .kvmppc_vcpu_run+0x44/0x58
[  419.423336] [c0000007f35efa70] [c0000000000662cc] .kvm_arch_vcpu_ioctl_run+0xfc/0x130
[  419.423416] [c0000007f35efb00] [c00000000005fb18] .kvm_vcpu_ioctl+0x1a8/0x5f4
[  419.423486] [c0000007f35efcb0] [c0000000001e8e98] .do_vfs_ioctl+0x618/0x69c
[  419.423555] [c0000007f35efd90] [c0000000001e8f8c] .SyS_ioctl+0x70/0x9c
[  419.423625] [c0000007f35efe30] [c000000000009198] syscall_exit+0x0/0x98
[  419.423693] handlers:
[  419.423720] [<c000000001586560>] .vfio_intx_handler
[  419.423778] Disabling IRQ #18

Thanks,
Gavin 

>> -----Original Message-----
>> From: linux-pci-owner@vger.kernel.org [mailto:linux-pci-
>> owner@vger.kernel.org] On Behalf Of Gavin Shan
>> Sent: Monday, January 05, 2015 7:48 AM
>> To: linux-pci@vger.kernel.org
>> Cc: Ajit Kumar Khaparde; bhelgaas@google.com; Gavin Shan
>> Subject: [PATCH] PCI: Mark broken INTx masking for BENET devices
>> 
>> Similar to commit 11e4253 ("PCI: Assume all Mellanox devices have
>> broken INTx masking"), when passing through following PCI device
>> using VFIO infrastructure, interrupt storm are reported. After
>> marking its INTx masking is broken, the interrupt storm isn't
>> raised again:
>> 
>>  # lspci -s 0000::.
>>  0000:01:00.0 Ethernet controller: Emulex Corporation \
>>               OneConnect 10Gb NIC (be3) (rev 02)
>>  0000:01:00.1 Ethernet controller: Emulex Corporation \
>>               OneConnect 10Gb NIC (be3) (rev 02)
>>  # lspci -n -s 0000::.
>>  0000:01:00.0 0200: 19a2:0710 (rev 02)
>>  0000:01:00.1 0200: 19a2:0710 (rev 02)
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  drivers/pci/quirks.c    | 2 ++
>>  include/linux/pci_ids.h | 2 ++
>>  2 files changed, 4 insertions(+)
>> 
>> diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
>> index ed6f89b..e823ac0 100644
>> --- a/drivers/pci/quirks.c
>> +++ b/drivers/pci/quirks.c
>> @@ -3027,6 +3027,8 @@
>> DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_REALTEK, 0x8169,
>>  			 quirk_broken_intx_masking);
>>  DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_MELLANOX, PCI_ANY_ID,
>>  			 quirk_broken_intx_masking);
>> +DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BE, PCI_ANY_ID,
>> +			 quirk_broken_intx_masking);
>> 
>>  #ifdef CONFIG_ACPI
>>  /*
>> diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
>> index e63c02a..df70b76 100644
>> --- a/include/linux/pci_ids.h
>> +++ b/include/linux/pci_ids.h
>> @@ -2481,6 +2481,8 @@
>>  #define PCI_DEVICE_ID_KORENIX_JETCARDF2	0x1700
>>  #define PCI_DEVICE_ID_KORENIX_JETCARDF3	0x17ff
>> 
>> +#define PCI_VENDOR_ID_BE		0x19a2
>> +
>>  #define PCI_VENDOR_ID_QMI		0x1a32
>> 
>>  #define PCI_VENDOR_ID_AZWAVE		0x1a3b
>> --
>> 1.8.3.2
>> 
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-pci" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Alex Williamson]

----- Original Message -----
> On Thu, Jan 08, 2015 at 11:33:46AM +0000, Venkat Duvvuru wrote:
> >Hi Gavin,
> >We tried to reproduce this in our lab but our observation is that we don't
> >see "interrupt storm" in our system.
> >Could you please give us the details of the repro scenario?
> >
> >A few more details will help us understand the problem better as well.
> >1. be2net driver version
> >2. be2net firmware version
> >3. lspci -vvv output of the card (lspci -d 19a2: -vvv)
> >4. Is SR-IOV enabled in your card?
> >
> 
> Venkat, On IBM's Power7 box, I passed through following adpater to guest with
> following
...
> 
> Steps to recreate the issue
> ===========================
> 
> 1. Configure the NIC and ping it from external.
> 2. Inject EEH error by running following command in host side. After about
>    20 seconds, I got following message from host side. With the patch applied
>    to host kernel, I didn't see the warning messages:

This seems really dubious and I don't see any justification at all for declaring DisINTx broken for all devices for the vendor.  Typically to call DisINTx broken for a given device, you can just boot the guest with pci=nomsi to force INTx to be used.  If that works, then DisINTx masking works.  If you require EEH injection to to trigger this, then the problem is more likely some containment issue during EEH recovery.  Thanks,

Alex

RE: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Venkat Duvvuru]

PiA+DQo+ID4gVmVua2F0LCBPbiBJQk0ncyBQb3dlcjcgYm94LCBJIHBhc3NlZCB0aHJvdWdoIGZv
bGxvd2luZyBhZHBhdGVyIHRvIGd1ZXN0DQo+IHdpdGgNCj4gPiBmb2xsb3dpbmcNCj4gLi4uDQo+
ID4NCj4gPiBTdGVwcyB0byByZWNyZWF0ZSB0aGUgaXNzdWUNCj4gPiA9PT09PT09PT09PT09PT09
PT09PT09PT09PT0NCj4gPg0KPiA+IDEuIENvbmZpZ3VyZSB0aGUgTklDIGFuZCBwaW5nIGl0IGZy
b20gZXh0ZXJuYWwuDQo+ID4gMi4gSW5qZWN0IEVFSCBlcnJvciBieSBydW5uaW5nIGZvbGxvd2lu
ZyBjb21tYW5kIGluIGhvc3Qgc2lkZS4gQWZ0ZXIgYWJvdXQNCj4gPiAgICAyMCBzZWNvbmRzLCBJ
IGdvdCBmb2xsb3dpbmcgbWVzc2FnZSBmcm9tIGhvc3Qgc2lkZS4gV2l0aCB0aGUgcGF0Y2gNCj4g
YXBwbGllZA0KPiA+ICAgIHRvIGhvc3Qga2VybmVsLCBJIGRpZG4ndCBzZWUgdGhlIHdhcm5pbmcg
bWVzc2FnZXM6DQo+IA0KPiBUaGlzIHNlZW1zIHJlYWxseSBkdWJpb3VzIGFuZCBJIGRvbid0IHNl
ZSBhbnkganVzdGlmaWNhdGlvbiBhdCBhbGwgZm9yIGRlY2xhcmluZw0KPiBEaXNJTlR4IGJyb2tl
biBmb3IgYWxsIGRldmljZXMgZm9yIHRoZSB2ZW5kb3IuICBUeXBpY2FsbHkgdG8gY2FsbCBEaXNJ
TlR4IGJyb2tlbg0KPiBmb3IgYSBnaXZlbiBkZXZpY2UsIHlvdSBjYW4ganVzdCBib290IHRoZSBn
dWVzdCB3aXRoIHBjaT1ub21zaSB0byBmb3JjZSBJTlR4DQo+IHRvIGJlIHVzZWQuICBJZiB0aGF0
IHdvcmtzLCB0aGVuIERpc0lOVHggbWFza2luZyB3b3Jrcy4gIElmIHlvdSByZXF1aXJlIEVFSA0K
PiBpbmplY3Rpb24gdG8gdG8gdHJpZ2dlciB0aGlzLCB0aGVuIHRoZSBwcm9ibGVtIGlzIG1vcmUg
bGlrZWx5IHNvbWUgY29udGFpbm1lbnQNCj4gaXNzdWUgZHVyaW5nIEVFSCByZWNvdmVyeS4gIFRo
YW5rcywNCj4gDQo+IEFsZXgNClllcyBJIGFncmVlIHdpdGggQWxleC4gDQpwY2k9bm9tc2kgaXMg
d2hhdCBleGFjdGx5IHdlIGRpZCB0byBzZWUgaWYgRGlzSU5UeCBpcyBicm9rZW4gb3Igbm90IGFu
ZCBEaXNJTlR4IHdvcmtzIGZpbmUgaW4gb3VyIHNldHVwLg0KSSB0aGluaywgd2UgbmVlZCB0byBh
bmFseXplIHRoaXMgRUVIIGluamVjdGlvbiBzY2VuYXJpbyBmdXJ0aGVyLCB0byByb290IGNhdXNl
IHRoZSBwcm9ibGVtLg0KV2Ugd2lsbCBwb3N0IHlvdSB1cGRhdGVzIG9uIG91ciBmaW5kaW5ncyBh
ZnRlciB0aGUgYW5hbHlzaXMuDQo=

Re: [PATCH] PCI: Mark broken INTx masking for BENET devices

[Gavin Shan]

On Fri, Jan 09, 2015 at 07:40:08AM +0000, Venkat Duvvuru wrote:
>> >
>> > Venkat, On IBM's Power7 box, I passed through following adpater to guest
>> with
>> > following
>> ...
>> >
>> > Steps to recreate the issue
>> > ===========================
>> >
>> > 1. Configure the NIC and ping it from external.
>> > 2. Inject EEH error by running following command in host side. After about
>> >    20 seconds, I got following message from host side. With the patch
>> applied
>> >    to host kernel, I didn't see the warning messages:
>> 
>> This seems really dubious and I don't see any justification at all for declaring
>> DisINTx broken for all devices for the vendor.  Typically to call DisINTx broken
>> for a given device, you can just boot the guest with pci=nomsi to force INTx
>> to be used.  If that works, then DisINTx masking works.  If you require EEH
>> injection to to trigger this, then the problem is more likely some containment
>> issue during EEH recovery.  Thanks,
>> 
>> Alex
>Yes I agree with Alex. 
>pci=nomsi is what exactly we did to see if DisINTx is broken or not and DisINTx works fine in our setup.
>I think, we need to analyze this EEH injection scenario further, to root cause the problem.
>We will post you updates on our findings after the analysis.

Thanks for suggestions. I'll do more experiments to locate the EEH. With the older
QEMU version, I didn't find this issue, so I guess it would be introduced by recent
QEMU VFIO changes.

Note: pci=nomsi didn't give me usable console from the guest side. I don't know why
yet. Need some time to investigate.

Thanks,
Gavin