Re: IRQ affinity on Linux guest

[Paolo Bonzini <pbonzini@redhat.com>]

On 21/08/2015 05:33, Mihai Neagu wrote:
> Radim,
> 
> Thanks for your answer. Indeed setting IRQ affinity to a specific core
> seems to be respected.
> 
> However, on software emulation and on the real machine, while IRQ
> affinity defaults to 3, all interrupts go on CPU0, while on KVM they go
> on CPU1.

Are you sure that KVM doesn't do some kind of round robin?

Also, it's probably not _all_ interrupts but most of them.  For example
my laptop has:

                                                            smp_affinity
        cpu0      cpu1     cpu2      cpu3                           ||||
1:     32824     1764       151       130   IO-APIC-edge  i8042     0123
12:  1587436   203990     16666     17064   IO-APIC-edge  i8042     0123
24:    63361     5160    212545      1276   PCI-MSI-edge  ahci      ..2.
25:        2        0      1082     10710   PCI-MSI-edge  xhci_hcd  ..23
29:  1867844    72169        61     57514   PCI-MSI-edge  iwlwifi   0...

I added a column at the end with the SMP affinity of the interrupts.
You can see that:

* real hardware also shows traces of interrupts delivered before the
affinity was set (especially vectors 24 and 29)

* real hardware also distributes interrupts across multiple CPUs when
the affinity covers multiple processors (see vectors 1, 12 and 25)

> I wonder why KVM would act differently than both the real machine and
> the software emulation in this particular aspect.
> 
> Is there a machine or processor that I can specify at KVM command line
> to make it behave like the real x86_64 processor which defaults
> interrupts to CPU0?

The behavior of interrupt arbitration is not specified in the processor
documentation and can change across processors and implementations.
Presumably it is either patented or an Intel trade secret.

Older processors were documented to pick a processor that was not
running another interrupt service routine.  This can explain why
software emulation shows that all interrupts go on CPU0.

At least some newer processors are not doing it anymore, and the SDM has
some high-level explanation of their behavior and its effects:

   [...] the chipset bus controller accepts messages from the I/O APIC
   agents in the system and directs interrupts to the processors on the
   system bus. When using the lowest priority delivery mode, the
   chipset chooses a target processor to receive the interrupt out of
   the set of possible targets.

   In operating systems that use the lowest priority delivery mode but
   do not update the TPR, the TPR information saved in the chipset will
   potentially cause the interrupt to be always delivered to the same
   processor from the logical set. This behavior is functionally
   backward compatible with the P6 family processor but may result in
   unexpected performance implications.

Other processors are doing so called "vector hashing" (i.e. pick a CPU
number based on the affinity mask and vector number, and use it most of
the time), which would also explain why the interrupts all go on CPU0.

Paolo