Re: [PATCH v2] X86: Hook apic vector allocation domain only when interrupt routing are set to ignore

[Ingo Molnar <mingo@kernel.org>]

* Oren Twaig <oren@scalemp.com> wrote:

> On 4/25/2014 11:01 AM, Ingo Molnar wrote:
> >
> > * Oren Twaig <oren@scalemp.com> wrote:
> >
> >> vSMP Foundation provides locality based interrupt routing which needed
> >> vector_allocation_domain to allow all online cpus can handle all
> possible
> >> vectors.
> >>
> >> Enforcing Interrupt Routing Comply (IRC) mode requires us to
> unplug this hook as
> >> otherwise the IOAPIC, MSI and MSIX destination selectors to
> always select the
> >> lowest online cpu as the destination. I.e affinity of HW
> interrupts cannot be
> >> controled by kernel and/or userspace code.
> >>
> >> The purpose of the patch is to fix the code to set override
> vector allocation
> >> domain only when IRC is set to ignore to allow the kernel and
> userspace to
> >> effectively control the destination of the HW interrupts.
> >>
> >> Signed-off-by: Oren Twaig <oren@scalemp.com>
> >> Acked-by: Shai Fultheim <shai@scalemp.com>
> >
> > So what was the behavior before the change - certain IRQs did not get
> > routed, they just ended up on CPU0 or on some other undesirable CPU?
> > Or was IRQ distribution random? It's not clear from the changelog.
> 
> It all depends on the IRC flag. When set to "ignore" by the linux 
> kernel, vSMP Foundation knew that it can deliver the IRQ to the CPU 
> which would result in less virtualization overhead. For example, we 
> could deliver the HW interrupt to the CPU which got it or any other 
> CPU in the system. We couldn't have done it without the kernel 
> making sure that each vector can be passed to all CPUs. This is why 
> we override the verctor allocation domain to signal all CPUs.
> 
> But, when the IRC is set to "comply" we, before this patch, still 
> efected the allocation domains alltough it wasn't needed. It wasn't 
> needed because when in "comply" mode, we always pass the HW 
> interrupt to the CPU the kernel requested (by setting the IOAPIC 
> entry, MSI/X entry or IR entry)

I still don't see a clear explanation of what the _user_ saw and sees 
before and after the change. What is the effect of the patch: correct 
IRQ routing (i.e. before the change IRQs would end up on the wrong 
CPU), lower overhead IRQ routing (i.e. before the change IRQ routing 
overhead was more expensive), or something else?

You don't spell this out clearly and it's a crucial piece of 
information that comes before every other explanation.

Thanks,

	Ingo