Re: [PATCH 00/10] igb_uio patches

Re: [PATCH 00/10] igb_uio patches

[Carew, Alan]

-----Original Message-----
From: Stephen Hemminger [mailto:stephen-OTpzqLSitTUnbdJkjeBofR2eb7JE58TQ@public.gmane.org] 
Sent: Friday, June 06, 2014 10:27 PM
To: Carew, Alan
Cc: dev-g7QkDGwwQmw@public.gmane.org
Subject: [PATCH 00/10] igb_uio patches

These apply to the current DPDK tree, and are an alternative to
the patches from Alan. It provides indication of IRQ mode via
sysfs attribute. They include cleanups and fixes to allow use
of MSI and do proper fallback for the case where MSI-X vectors
are exhausted.

Hi Stephen,

Good catches and fixes in this set, would it be possible to refactor the "enum igbuio_intr_mode" to a common header files as with http://dpdk.org/ml/archives/dev/2014-June/003127.html
With that infrastructure in place, I could resubmit my set {2,3,4} to work cleanly with yours, maintaining the same interface across user/kernel, FreeBSD and fixing Virtio also.

Regards,
Alan

Re: [PATCH 00/10] igb_uio patches

[Burakov, Anatoly]

> These apply to the current DPDK tree, and are an alternative to the patches
> from Alan. It provides indication of IRQ mode via sysfs attribute. They include
> cleanups and fixes to allow use of MSI and do proper fallback for the case
> where MSI-X vectors are exhausted.
> 
> Hi Stephen,
> 
> Good catches and fixes in this set, would it be possible to refactor the "enum
> igbuio_intr_mode" to a common header files as with
> http://dpdk.org/ml/archives/dev/2014-June/003127.html
> With that infrastructure in place, I could resubmit my set {2,3,4} to work
> cleanly with yours, maintaining the same interface across user/kernel,
> FreeBSD and fixing Virtio also.

...or better yet, with

http://dpdk.org/ml/archives/dev/2014-June/003221.html

:-)

Best regards,
Anatoly Burakov
DPDK SW Engineer

Re: [PATCH 00/10] igb_uio patches

[Carew, Alan]

-----Original Message-----
From: Stephen Hemminger [mailto:stephen-OTpzqLSitTUnbdJkjeBofR2eb7JE58TQ@public.gmane.org] 
Sent: Tuesday, June 10, 2014 6:41 PM
To: Carew, Alan
Cc: dev-g7QkDGwwQmw@public.gmane.org
Subject: Re: [PATCH 00/10] igb_uio patches

On Tue, 10 Jun 2014 15:33:31 +0000
"Carew, Alan" <alan.carew-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> Good catches and fixes in this set, would it be possible to refactor the "enum igbuio_intr_mode" to a common header files as with http://dpdk.org/ml/archives/dev/2014-June/003127.html

Actually, thought that was ugly. The userspace shouldn't have to see all these
definitions.

Not a big fan of doing this and it would go against goal of making a version
that could be merged upstream.

Stephen,

One alternative for us is then to take Neil Horman's idea of scanning /sys/bus/pci/devices/<D:B:D.F>/msi_irqs/
This gives us at least (MSI | MSI-X) | INT-X, however the Virtio specification mentions MSI-X explicitly for determining the correct offset and might not address the underlying problem my patch set was addressing, I have not tried Virtio with MSI.

A question then, how does uio_pci_generic handle interrupt modes in terms of userspace interface when determining the configured interrupt mode?

In the short term while we still work with igb_uio we may have to rely on SYSFS, but there is nothing stopping us from having device specific code paths that does not exclude us from ignoring igb_uio. The shared headers also gives us the benefit of having an internal interface(rte_pci_device: rte_intr_mode ) already in place when  swapping out kernel drivers, Anatoly's VFIO implementation already uses such an interface.

Thanks,
Alan

Re: [PATCH 00/10] igb_uio patches

[Stephen Hemminger]

On Tue, 10 Jun 2014 19:26:08 +0000
"Carew, Alan" <alan.carew-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> One alternative for us is then to take Neil Horman's idea of scanning /sys/bus/pci/devices/<D:B:D.F>/msi_irqs/
> This gives us at least (MSI | MSI-X) | INT-X, however the Virtio specification mentions MSI-X explicitly for determining the correct offset and might not address the underlying problem my patch set was addressing, I have not tried Virtio with MSI.

MSI doesn't work with virtio. pci_enable_msi() returns error. The device
does not report MSI capability.

Re: [PATCH 00/10] igb_uio patches

[Neil Horman]

On Tue, Jun 10, 2014 at 01:47:09PM -0700, Stephen Hemminger wrote:
> On Tue, 10 Jun 2014 19:26:08 +0000
> "Carew, Alan" <alan.carew-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:
> 
> > One alternative for us is then to take Neil Horman's idea of scanning /sys/bus/pci/devices/<D:B:D.F>/msi_irqs/
> > This gives us at least (MSI | MSI-X) | INT-X, however the Virtio specification mentions MSI-X explicitly for determining the correct offset and might not address the underlying problem my patch set was addressing, I have not tried Virtio with MSI.
> 
> MSI doesn't work with virtio. pci_enable_msi() returns error. The device
> does not report MSI capability.
> 
/sys/bus/pci/devices/<device>/irq

also indicates which legacy irq is assigned to a given device

Neil

Re: [PATCH 00/10] igb_uio patches

[Stephen Hemminger]

On Tue, 10 Jun 2014 19:26:08 +0000
"Carew, Alan" <alan.carew-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> A question then, how does uio_pci_generic handle interrupt modes in terms of userspace interface when determining the configured interrupt mode?

uio_pci_generic only handles INTX disable, and nothing else. It doesn't do MSI
or allow IRQ control.

Also, it doesn't do iomap of resources, so theoretically if IOMMU is enabled
userspace would still not be able to access the resources.