[Qemu-devel] removing on-demand msix vector allocation

[Qemu-devel] removing on-demand msix vector allocation

[Michael S. Tsirkin]

I've been looking at handling of msix masking
in qemu. It looks like all of virtio,vfio and
device assignment implemented their own
similar but slightly different thing.
So I am inclined to move this handling to common
code in msix.c, adding irqfd support right there.

While doing this rework, one of the more painful
bits of code to change is the code that dynamically
allocates msix table entries as we inject msi.
If this actually triggers it's going to be
painfully slow as route changes are rcu
write side in kernel.
Since recent kernels support direct injection,
do we care anymore? I think if you run out of
vectors, it's better to simply disable irqchip
than try to limp along changing routes all the time.

Comments?

-- 
MST

Re: [Qemu-devel] removing on-demand msix vector allocation

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

On 2012-12-06 08:59, Michael S. Tsirkin wrote:
> I've been looking at handling of msix masking
> in qemu. It looks like all of virtio,vfio and
> device assignment implemented their own
> similar but slightly different thing.
> So I am inclined to move this handling to common
> code in msix.c, adding irqfd support right there.
> 
> While doing this rework, one of the more painful
> bits of code to change is the code that dynamically
> allocates msix table entries as we inject msi.
> If this actually triggers it's going to be
> painfully slow as route changes are rcu
> write side in kernel.
> Since recent kernels support direct injection,
> do we care anymore? I think if you run out of
> vectors, it's better to simply disable irqchip
> than try to limp along changing routes all the time.

But how would the logic without dynamic allocation look like? Always
configure a route in the PCI layer if an MSI/MSI-X entry is enabled?
That would also affect emulated devices that don't use irqfd, thus you
would waste routing entries. OTOH, if don't set up such routes, you
cannot support MSI/-X on older x86 kernels without direct vector injection.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [Qemu-devel] removing on-demand msix vector allocation

[Michael S. Tsirkin]

On Fri, Dec 07, 2012 at 08:37:22AM +0100, Jan Kiszka wrote:
> On 2012-12-06 08:59, Michael S. Tsirkin wrote:
> > I've been looking at handling of msix masking
> > in qemu. It looks like all of virtio,vfio and
> > device assignment implemented their own
> > similar but slightly different thing.
> > So I am inclined to move this handling to common
> > code in msix.c, adding irqfd support right there.
> > 
> > While doing this rework, one of the more painful
> > bits of code to change is the code that dynamically
> > allocates msix table entries as we inject msi.
> > If this actually triggers it's going to be
> > painfully slow as route changes are rcu
> > write side in kernel.
> > Since recent kernels support direct injection,
> > do we care anymore? I think if you run out of
> > vectors, it's better to simply disable irqchip
> > than try to limp along changing routes all the time.
> 
> But how would the logic without dynamic allocation look like? Always
> configure a route in the PCI layer if an MSI/MSI-X entry is enabled?
> That would also affect emulated devices that don't use irqfd, thus you
> would waste routing entries.

Yes. 
So we can fail during initialization and ask user to
disable irqchip: at the moment, at least in my testing,
dynamic swap out of MSI entries performs very badly
anyway.

> OTOH, if don't set up such routes, you
> cannot support MSI/-X on older x86 kernels without direct vector injection.
> 
> Jan
> 
> 

Re: [Qemu-devel] removing on-demand msix vector allocation

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1872 bytes --]

On 2012-12-10 10:36, Michael S. Tsirkin wrote:
> On Fri, Dec 07, 2012 at 08:37:22AM +0100, Jan Kiszka wrote:
>> On 2012-12-06 08:59, Michael S. Tsirkin wrote:
>>> I've been looking at handling of msix masking
>>> in qemu. It looks like all of virtio,vfio and
>>> device assignment implemented their own
>>> similar but slightly different thing.
>>> So I am inclined to move this handling to common
>>> code in msix.c, adding irqfd support right there.
>>>
>>> While doing this rework, one of the more painful
>>> bits of code to change is the code that dynamically
>>> allocates msix table entries as we inject msi.
>>> If this actually triggers it's going to be
>>> painfully slow as route changes are rcu
>>> write side in kernel.
>>> Since recent kernels support direct injection,
>>> do we care anymore? I think if you run out of
>>> vectors, it's better to simply disable irqchip
>>> than try to limp along changing routes all the time.
>>
>> But how would the logic without dynamic allocation look like? Always
>> configure a route in the PCI layer if an MSI/MSI-X entry is enabled?
>> That would also affect emulated devices that don't use irqfd, thus you
>> would waste routing entries.
> 
> Yes. 
> So we can fail during initialization and ask user to
> disable irqchip: at the moment, at least in my testing,
> dynamic swap out of MSI entries performs very badly
> anyway.

That would be a poor approach as it regresses needlessly even over
latest kernels. We only allocate/flush dynamically over older kernels
without direct MSI injections.

What we need is a flag, set e.g. on msi[x]_init, to give the core a hint
if it should allocate static routes for irqfd or if it will be able to
use direct injection later on. Then we can simply do static allocation
unconditionally on kernels without direct injection.

Jan



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 261 bytes --]

Re: [Qemu-devel] removing on-demand msix vector allocation

[Michael S. Tsirkin]

On Mon, Dec 10, 2012 at 10:39:39AM +0100, Jan Kiszka wrote:
> On 2012-12-10 10:36, Michael S. Tsirkin wrote:
> > On Fri, Dec 07, 2012 at 08:37:22AM +0100, Jan Kiszka wrote:
> >> On 2012-12-06 08:59, Michael S. Tsirkin wrote:
> >>> I've been looking at handling of msix masking
> >>> in qemu. It looks like all of virtio,vfio and
> >>> device assignment implemented their own
> >>> similar but slightly different thing.
> >>> So I am inclined to move this handling to common
> >>> code in msix.c, adding irqfd support right there.
> >>>
> >>> While doing this rework, one of the more painful
> >>> bits of code to change is the code that dynamically
> >>> allocates msix table entries as we inject msi.
> >>> If this actually triggers it's going to be
> >>> painfully slow as route changes are rcu
> >>> write side in kernel.
> >>> Since recent kernels support direct injection,
> >>> do we care anymore? I think if you run out of
> >>> vectors, it's better to simply disable irqchip
> >>> than try to limp along changing routes all the time.
> >>
> >> But how would the logic without dynamic allocation look like? Always
> >> configure a route in the PCI layer if an MSI/MSI-X entry is enabled?
> >> That would also affect emulated devices that don't use irqfd, thus you
> >> would waste routing entries.
> > 
> > Yes. 
> > So we can fail during initialization and ask user to
> > disable irqchip: at the moment, at least in my testing,
> > dynamic swap out of MSI entries performs very badly
> > anyway.
> 
> That would be a poor approach as it regresses needlessly even over
> latest kernels.
> We only allocate/flush dynamically over older kernels
> without direct MSI injections.
> 
> What we need is a flag, set e.g. on msi[x]_init, to give the core a hint
> if it should allocate static routes for irqfd or if it will be able to
> use direct injection later on. Then we can simply do static allocation
> unconditionally on kernels without direct injection.
> 
> Jan
> 
> 

Makes sense.

-- 
MST