From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: MIME-Version: 1.0 In-Reply-To: References: <20180213184435.GE75542@bhelgaas-glaptop.roam.corp.google.com> From: Kallol Biswas Date: Tue, 20 Feb 2018 22:23:25 -0800 Message-ID: Subject: Re: enabling multiple MSI vectors for a DS port To: Bjorn Helgaas Cc: linux-pci@vger.kernel.org, Bjorn Helgaas Content-Type: multipart/alternative; boundary="001a114d37bafe121f0565b2f6bb" List-ID: --001a114d37bafe121f0565b2f6bb Content-Type: text/plain; charset="UTF-8" Hello Bjorn, We have worked around the issue by ack'ing dpc interrupt in irq handler than in worker thread. Regarding sharing the output of lspci -vv, we are not authorized to send customer system's detail, otherwise we would be happy to do so. Anyway, is there a good document on linux PCIe hotplug? --------------------------------------------- static irqreturn_t dpc_irq(int irq, void *context) { struct dpc_dev *dpc = (struct dpc_dev *)context; struct pci_dev *pdev = dpc->dev->port; u16 status, source; pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status); pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID, &source); if (!status) return IRQ_NONE; dev_info(&dpc->dev->device, "DPC containment event, status:%#06x source:%#06x\n", status, source); if (status & PCI_EXP_DPC_STATUS_TRIGGER) { u16 reason = (status >> 1) & 0x3; pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, PCI_EXP_DPC_STATUS_INTERRUPT); dev_warn(&dpc->dev->device, "DPC %s triggered, remove downstream devices\n", (reason == 0) ? "unmasked uncorrectable error" : (reason == 1) ? "ERR_NONFATAL" : (reason == 2) ? "ERR_FATAL" : "extended error"); schedule_work(&dpc->work); } return IRQ_HANDLED; On Tue, Feb 20, 2018 at 10:20 PM, Kallol Biswas wrote: > Hello Bjorn, > We have worked around the issue by ack'ing dpc interrupt > in irq handler than in worker thread. > > Regarding sharing the output of lspci -vv, we are not authorized to send > customer system's detail, otherwise we would be happy to do so. > > > Anyway, is there a good document on linux PCIe hotplug? > > > > --------------------------------------------- > static irqreturn_t dpc_irq(int irq, void *context) > { > > struct dpc_dev *dpc = (struct dpc_dev *)context; > > struct pci_dev *pdev = dpc->dev->port; > > u16 status, source; > > > pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status); > > pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID, > > &source); > > if (!status) > > return IRQ_NONE; > > > dev_info(&dpc->dev->device, "DPC containment event, status:%#06x > source:%#06x\n", > > status, source); > > > if (status & PCI_EXP_DPC_STATUS_TRIGGER) { > > u16 reason = (status >> 1) & 0x3; > > pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, > > PCI_EXP_DPC_STATUS_INTERRUPT); > > > > dev_warn(&dpc->dev->device, "DPC %s triggered, remove downstream > devices\n", > > (reason == 0) ? "unmasked uncorrectable error" : > > (reason == 1) ? "ERR_NONFATAL" : > > (reason == 2) ? "ERR_FATAL" : "extended error"); > > schedule_work(&dpc->work); > > } > > return IRQ_HANDLED; > > On Tue, Feb 13, 2018 at 10:44 AM, Bjorn Helgaas > wrote: > >> Hi Kallol, >> >> On Tue, Feb 13, 2018 at 10:21:04AM -0800, Kallol Biswas wrote: >> > Hi Bjorn, >> > I am facing an interrupt generation issue when single MSI >> > is shared by both dpc and hotplug interrupt. >> > >> > The DS port is multi-MSI capable. What I have noticed that if I move >> > the switch (with DS port) and OS disk to a different system multiple >> > MSI vectors are allocated and dpc isr and hotplug isr are on different >> > vectors. >> > >> > It seems that BIOS controls allocation of MSI vectors. Is there anyway >> > to force linux kernel to allocate multiple vectors when port is >> > capable? >> >> BIOS doesn't control the allocation of MSI vectors. That's done by >> Linux in pcie_init_service_irqs(), and it depends on many things: what >> services are supported by Linux (hotplug, PME, AER, DPC), which ones >> Linux is allowed to use (based on the ACPI _OSC method, so this part >> can be influenced by the BIOS), etc. >> >> If you can share the "lspci -vv" and dmesg output for these two >> systems, we can probably work out what's going on. >> >> Things should *work* even if DPC and hotplug need to share an >> interrupt. If that doesn't work, that's a Linux bug that should be >> fixed. >> >> Bjorn >> > > --001a114d37bafe121f0565b2f6bb Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Bjorn,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 We have worked around the issue by ack'ing dpc= interrupt in irq handler than in worker thread.

Regardi= ng sharing the output of lspci -vv, we are not authorized to send customer = system's detail, otherwise we would be happy to do so.


Anyway, is there a good document on linux PCIe hotp= lug?=C2=A0



--------------------------------------------= -
static irqreturn_t dpc_irq(int irq, void *context)
{

struct = dpc_dev *dpc =3D (struct dpc_dev *)context;

struct pci_dev *pdev =3D= dpc->dev->port;

u16 status, source;


pci_read_confi= g_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status);

pci= _read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID,

&am= p;source);

if (!status)

return IRQ_NONE;


dev_info(= &dpc->dev->device, "DPC containment event, status:%#06x sour= ce:%#06x\n",

status, source);


if (status & PCI_E= XP_DPC_STATUS_TRIGGER) {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 u16 reas= on =3D (status >> 1) & 0x3;

=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0pci_write_config_word(pdev, dpc->cap_pos += PCI_EXP_DPC_STATUS,

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0PCI_EXP_DPC_STATUS_INTERRUPT);

<= br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0dev_warn(&dpc->dev->devi= ce, "DPC %s triggered, remove downstream devices\n",

=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(reason =3D=3D 0) ? "unmasked uncorrect= able error" :

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(reason =3D=3D = 1) ? "ERR_NONFATAL" :

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(r= eason =3D=3D 2) ? "ERR_FATAL" : "extended error");
<= br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 schedule_work(&dpc->work);
=
}

return IRQ_HANDLED;


On Tue, Feb 20, 2018 at 10:20 PM, Kallol Biswas= <ndprkm@gmail.com> wrote:
=
Hello Bjorn,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 We have worked around the issue by ack'ing dpc interrupt = in irq handler than in worker thread.

Regarding sharing = the output of lspci -vv, we are not authorized to send customer system'= s detail, otherwise we would be happy to do so.

Anyway, is there a good document on linux PCIe hotplug?=C2=A0<= br>


---------------------------------------------
static= irqreturn_t dpc_irq(int irq, void *context)
{

struct dpc_dev *dp= c =3D (struct dpc_dev *)context;

struct pci_dev *pdev =3D dpc->de= v->port;

u16 status, source;


pci_read_config_word(pdev= , dpc->cap_pos + PCI_EXP_DPC_STATUS, &status);

pci_read_confi= g_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID,

&source);<= br>
if (!status)

return IRQ_NONE;


dev_info(&dpc-&g= t;dev->device, "DPC containment event, status:%#06x source:%#06x\n&= quot;,

status, source);


if (status & PCI_EXP_DPC_STAT= US_TRIGGER) {

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 u16 reason =3D (sta= tus >> 1) & 0x3;

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_= STATUS,

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0PCI_EXP_DPC_STATUS_INTERRUPT);



=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0dev_warn(&dpc->dev->device, "DPC = %s triggered, remove downstream devices\n",

=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0(reason =3D=3D 0) ? "unmasked uncorrectable error&quo= t; :

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(reason =3D=3D 1) ? "ERR= _NONFATAL" :

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(reason =3D=3D 2= ) ? "ERR_FATAL" : "extended error");

=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 schedule_work(&dpc->work);

}

= return IRQ_HANDLED;
=

On Tue, Feb 13, 2018 at 10:44 AM, Bjorn Helgaas <h= elgaas@kernel.org> wrote:
H= i Kallol,

On Tue, Feb 13, 2018 at 10:21:04AM -0800, Kallol Biswas wrote:
> Hi Bjorn,
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0I am facing an interrup= t generation issue when single MSI
> is shared by both dpc and hotplug interrupt.
>
> The DS port is multi-MSI capable. What I have noticed that if I move > the switch (with DS port) and OS disk to a different system multiple > MSI vectors are allocated and dpc isr and hotplug isr are on different=
> vectors.
>
> It seems that BIOS controls allocation of MSI vectors. Is there anyway=
> to force linux kernel to allocate multiple vectors when port is
> capable?

BIOS doesn't control the allocation of MSI vectors.=C2=A0 That&#= 39;s done by
Linux in pcie_init_service_irqs(), and it depends on many things: what
services are supported by Linux (hotplug, PME, AER, DPC), which ones
Linux is allowed to use (based on the ACPI _OSC method, so this part
can be influenced by the BIOS), etc.

If you can share the "lspci -vv" and dmesg output for these two systems, we can probably work out what's going on.

Things should *work* even if DPC and hotplug need to share an
interrupt.=C2=A0 If that doesn't work, that's a Linux bug that shou= ld be
fixed.

Bjorn


--001a114d37bafe121f0565b2f6bb--