Re: [Xen-devel] [PATCH] xen: reuse the same pirq allocated when driver load first time

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, May 22, 2013 at 10:37:39AM +0100, Jan Beulich wrote:
> >>> On 22.05.13 at 00:41, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> > On Tue, May 21, 2013 at 10:50:09PM +0100, Stefano Stabellini wrote:
> >> We have to be careful about this: the point of PHYSDEVOP_get_free_pirq is
> >> that Linux can know for sure the pirq that is going to be used to map the
> >> MSI by QEMU. If you modify is_free_pirq that way, suddenly the pirq
> >> could be allocated for something else after Linux called
> >> PHYSDEVOP_get_free_pirq and before QEMU called xc_physdev_map_pirq_msi.
> > 
> > Yes. And I think the 'is_free_pirq' modification above is incorrect.
> > 
> > I think the fix should be in the unmap_pirq code (hypervisor) to check
> > if the arch.irq is either zero or PIRQ_ALLOCATED. Right now it only
> > checks for zero.
> 
> Which check are you talking about? Looking at physdev_unmap_pirq()

Sorry about being so haphazard here. I am still digging in the code
and trying to get a sense of how QEMU and hypervisor are suppose to
dance together.

The check was on the PHYSDEVOP_get_free_pirq, which calls get_free_pirq
and uses the is_free_pirq check. After the get_free_pirq call, the logic
in PHYSDEVOP_get_free_pirq sets info->arch.pirq = PIRQ_ALLOCATED to
protect itself from giving the same PIRQ twice.

The physdev_unmap_pirq (from PHYSDEVOP_unmap_pirq), only has this
check:
 if (domain_pirq_to_emuirq(d, pirq) != IRQ_UNBOUND)

and since the arch.hvm.emuirq is IRQ_UNBOUND (-1), it does not
call unmap_domain_pirq_emuirq. It probably shouldn't, but it should
at least remove the info->arch.pirq = PIRQ_ALLOCATED as we are
telling the hypervisor: "hey, I am done with this, return to the
pool." But since that is not cleared, the PHYSDEVOP_get_free_pirq
will skip this pirq as arch.pirq is still set to PIRQ_ALLOCATED.

> I see none at all, unmap_domain_pirq() has a <= 0 check, and
> unmap_domain_pirq_emuirq() again doesn't appear to have any.

The 'unmap_domain_pirq' path would be if dom0 (so QEMU) did the
unmap for the guest. That is via the PHYSDEVOP_unmap_pirq. And
I think if that path was taken (as Stefano suggests QEMU should
do when a MSI or MSI-X driver is unloaded and zero is writen as
an PIRQ), we would end up calling clear_domain_irq_pirq, which
would set arch.pirq = 0.

Or to a negative value as you pointed out later. Which then
means we won't be ever able to re-use the PIRQ (as
PHYSDEVOP_get_free_pirq or rather get_free_pirq would skip over it
as arch.pirq != 0).
> 
> If you're talking about unmap_domain_pirq(), then you'll need to
> be careful: A negative value here doesn't necessarily mean
> PIRQ_ALLOCATED, but could also come from another run that
> found it necessary to force the unbind. Hence the definition of
> PIRQ_ALLOCATED would then collide with the (unlikely?) case of
> IRQ1 having got assigned to a guest. To be on the safe side, we
> should therefore redefine PIRQ_ALLOCATED to say INT_MIN.

You are right about being cautious - this is a bit of complex
code interaction between Xen, QEMU, and Linux kernel.