enabling multiple MSI vectors for a DS port

enabling multiple MSI vectors for a DS port

[Kallol Biswas]

Hi Bjorn,
            I am facing an interrupt generation issue when single MSI
is shared by both dpc and hotplug interrupt.

The DS port is multi-MSI capable. What I have noticed that if I move
the switch (with DS port) and OS disk to a different system multiple
MSI vectors are allocated and dpc isr and hotplug isr are on different
vectors.

It seems that BIOS controls allocation of MSI vectors. Is there anyway
to force linux kernel to allocate multiple vectors when port is
capable?

Regards,
Kallol

Re: enabling multiple MSI vectors for a DS port

[Bjorn Helgaas]

Hi Kallol,

On Tue, Feb 13, 2018 at 10:21:04AM -0800, Kallol Biswas wrote:
> Hi Bjorn,
>             I am facing an interrupt generation issue when single MSI
> is shared by both dpc and hotplug interrupt.
> 
> The DS port is multi-MSI capable. What I have noticed that if I move
> the switch (with DS port) and OS disk to a different system multiple
> MSI vectors are allocated and dpc isr and hotplug isr are on different
> vectors.
> 
> It seems that BIOS controls allocation of MSI vectors. Is there anyway
> to force linux kernel to allocate multiple vectors when port is
> capable?

BIOS doesn't control the allocation of MSI vectors.  That's done by
Linux in pcie_init_service_irqs(), and it depends on many things: what
services are supported by Linux (hotplug, PME, AER, DPC), which ones
Linux is allowed to use (based on the ACPI _OSC method, so this part
can be influenced by the BIOS), etc.

If you can share the "lspci -vv" and dmesg output for these two
systems, we can probably work out what's going on.

Things should *work* even if DPC and hotplug need to share an
interrupt.  If that doesn't work, that's a Linux bug that should be
fixed.

Bjorn

Re: enabling multiple MSI vectors for a DS port

[Kallol Biswas]

[-- Attachment #1: Type: text/plain, Size: 2641 bytes --]

Hello Bjorn,
                We have worked around the issue by ack'ing dpc interrupt in
irq handler than in worker thread.

Regarding sharing the output of lspci -vv, we are not authorized to send
customer system's detail, otherwise we would be happy to do so.


Anyway, is there a good document on linux PCIe hotplug?



---------------------------------------------
static irqreturn_t dpc_irq(int irq, void *context)
{

struct dpc_dev *dpc = (struct dpc_dev *)context;

struct pci_dev *pdev = dpc->dev->port;

u16 status, source;


pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status);

pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID,

&source);

if (!status)

return IRQ_NONE;


dev_info(&dpc->dev->device, "DPC containment event, status:%#06x
source:%#06x\n",

status, source);


if (status & PCI_EXP_DPC_STATUS_TRIGGER) {

          u16 reason = (status >> 1) & 0x3;

         pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS,

                   PCI_EXP_DPC_STATUS_INTERRUPT);



         dev_warn(&dpc->dev->device, "DPC %s triggered, remove downstream
devices\n",

         (reason == 0) ? "unmasked uncorrectable error" :

         (reason == 1) ? "ERR_NONFATAL" :

         (reason == 2) ? "ERR_FATAL" : "extended error");

          schedule_work(&dpc->work);

}

return IRQ_HANDLED;

On Tue, Feb 13, 2018 at 10:44 AM, Bjorn Helgaas <helgaas@kernel.org> wrote:

> Hi Kallol,
>
> On Tue, Feb 13, 2018 at 10:21:04AM -0800, Kallol Biswas wrote:
> > Hi Bjorn,
> >             I am facing an interrupt generation issue when single MSI
> > is shared by both dpc and hotplug interrupt.
> >
> > The DS port is multi-MSI capable. What I have noticed that if I move
> > the switch (with DS port) and OS disk to a different system multiple
> > MSI vectors are allocated and dpc isr and hotplug isr are on different
> > vectors.
> >
> > It seems that BIOS controls allocation of MSI vectors. Is there anyway
> > to force linux kernel to allocate multiple vectors when port is
> > capable?
>
> BIOS doesn't control the allocation of MSI vectors.  That's done by
> Linux in pcie_init_service_irqs(), and it depends on many things: what
> services are supported by Linux (hotplug, PME, AER, DPC), which ones
> Linux is allowed to use (based on the ACPI _OSC method, so this part
> can be influenced by the BIOS), etc.
>
> If you can share the "lspci -vv" and dmesg output for these two
> systems, we can probably work out what's going on.
>
> Things should *work* even if DPC and hotplug need to share an
> interrupt.  If that doesn't work, that's a Linux bug that should be
> fixed.
>
> Bjorn
>

[-- Attachment #2: Type: text/html, Size: 3621 bytes --]

Re: enabling multiple MSI vectors for a DS port

[Kallol Biswas]

[-- Attachment #1: Type: text/plain, Size: 4200 bytes --]

Hello Bjorn,
                We have worked around the issue by ack'ing dpc interrupt in
irq handler than in worker thread.

Regarding sharing the output of lspci -vv, we are not authorized to send
customer system's detail, otherwise we would be happy to do so.


Anyway, is there a good document on linux PCIe hotplug?



---------------------------------------------
static irqreturn_t dpc_irq(int irq, void *context)
{

struct dpc_dev *dpc = (struct dpc_dev *)context;

struct pci_dev *pdev = dpc->dev->port;

u16 status, source;


pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status);

pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID,

&source);

if (!status)

return IRQ_NONE;


dev_info(&dpc->dev->device, "DPC containment event, status:%#06x
source:%#06x\n",

status, source);


if (status & PCI_EXP_DPC_STATUS_TRIGGER) {

          u16 reason = (status >> 1) & 0x3;

         pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS,

                   PCI_EXP_DPC_STATUS_INTERRUPT);



         dev_warn(&dpc->dev->device, "DPC %s triggered, remove downstream
devices\n",

         (reason == 0) ? "unmasked uncorrectable error" :

         (reason == 1) ? "ERR_NONFATAL" :

         (reason == 2) ? "ERR_FATAL" : "extended error");

          schedule_work(&dpc->work);

}

return IRQ_HANDLED;



On Tue, Feb 20, 2018 at 10:20 PM, Kallol Biswas <ndprkm@gmail.com> wrote:

> Hello Bjorn,
>                 We have worked around the issue by ack'ing dpc interrupt
> in irq handler than in worker thread.
>
> Regarding sharing the output of lspci -vv, we are not authorized to send
> customer system's detail, otherwise we would be happy to do so.
>
>
> Anyway, is there a good document on linux PCIe hotplug?
>
>
>
> ---------------------------------------------
> static irqreturn_t dpc_irq(int irq, void *context)
> {
>
> struct dpc_dev *dpc = (struct dpc_dev *)context;
>
> struct pci_dev *pdev = dpc->dev->port;
>
> u16 status, source;
>
>
> pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS, &status);
>
> pci_read_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_SOURCE_ID,
>
> &source);
>
> if (!status)
>
> return IRQ_NONE;
>
>
> dev_info(&dpc->dev->device, "DPC containment event, status:%#06x
> source:%#06x\n",
>
> status, source);
>
>
> if (status & PCI_EXP_DPC_STATUS_TRIGGER) {
>
>           u16 reason = (status >> 1) & 0x3;
>
>          pci_write_config_word(pdev, dpc->cap_pos + PCI_EXP_DPC_STATUS,
>
>                    PCI_EXP_DPC_STATUS_INTERRUPT);
>
>
>
>          dev_warn(&dpc->dev->device, "DPC %s triggered, remove downstream
> devices\n",
>
>          (reason == 0) ? "unmasked uncorrectable error" :
>
>          (reason == 1) ? "ERR_NONFATAL" :
>
>          (reason == 2) ? "ERR_FATAL" : "extended error");
>
>           schedule_work(&dpc->work);
>
> }
>
> return IRQ_HANDLED;
>
> On Tue, Feb 13, 2018 at 10:44 AM, Bjorn Helgaas <helgaas@kernel.org>
> wrote:
>
>> Hi Kallol,
>>
>> On Tue, Feb 13, 2018 at 10:21:04AM -0800, Kallol Biswas wrote:
>> > Hi Bjorn,
>> >             I am facing an interrupt generation issue when single MSI
>> > is shared by both dpc and hotplug interrupt.
>> >
>> > The DS port is multi-MSI capable. What I have noticed that if I move
>> > the switch (with DS port) and OS disk to a different system multiple
>> > MSI vectors are allocated and dpc isr and hotplug isr are on different
>> > vectors.
>> >
>> > It seems that BIOS controls allocation of MSI vectors. Is there anyway
>> > to force linux kernel to allocate multiple vectors when port is
>> > capable?
>>
>> BIOS doesn't control the allocation of MSI vectors.  That's done by
>> Linux in pcie_init_service_irqs(), and it depends on many things: what
>> services are supported by Linux (hotplug, PME, AER, DPC), which ones
>> Linux is allowed to use (based on the ACPI _OSC method, so this part
>> can be influenced by the BIOS), etc.
>>
>> If you can share the "lspci -vv" and dmesg output for these two
>> systems, we can probably work out what's going on.
>>
>> Things should *work* even if DPC and hotplug need to share an
>> interrupt.  If that doesn't work, that's a Linux bug that should be
>> fixed.
>>
>> Bjorn
>>
>
>

[-- Attachment #2: Type: text/html, Size: 6822 bytes --]