IRQ affinity on Linux guest

IRQ affinity on Linux guest

[Mihai Neagu]

Hello,

I'm trying to assign some IRQ affinities to core 0 by setting 
smp_affinity to 1.
This is on a dual-core embedded Linux virtual machine ran with KVM.
However, ISRs continue to run on both cores.
The same technique works well with QEMU with full software emulation.

Here is the output of the following:

host> uname -a
Linux rtcon2 3.19.0-22-generic #22~14.04.1-Ubuntu SMP Wed Jun 17
10:03:13 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

guest> uname -a
Linux NI-cRIO-Linux-525400123456 3.14.40-rt37-nilrt #1 SMP PREEMPT RT
Thu Aug 6 20:35:41 EEST 2015 x86_64 GNU/Linux

host> kvm --version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.15), Copyright
(c) 2003-2008 Fabrice Bellard

Here is how IRQ affinity is configured on guest at startup, in an init.d
script:

echo 1 > /proc/irq/default_smp_affinity
for x in /proc/irq/*/smp_affinity;
do
   echo 1 > $x
done 2> /dev/null

The command line for starting the hardware accelerated VM:
kvm -kernel bzImage -hda rootfs.ext2 -append "root=/dev/sda console=ttyS0  \
rw" -nographic -cpu qemu64 -snapshot -smp 2 -m 2048

... which actually runs this:
qemu-system-x86-64 -enable-kvm -kernel bzImage -hda rootfs.ext2 -append    \
"root=/dev/sda console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2   \
-m 2048

On the hardware accelerated guest, 'cat /proc/interrupts' shows:
            CPU0       CPU1
   0:         26          0   IO-APIC-edge      timer
   1:          7          4   IO-APIC-edge      i8042
   4:       1137        523   IO-APIC-edge      serial
   8:          0          1   IO-APIC-edge      rtc0
   9:          0          0   IO-APIC-fasteoi   acpi
  11:       4971          4   IO-APIC-fasteoi   eth0
  12:         66         64   IO-APIC-edge      i8042
  14:       1958        714   IO-APIC-edge      ata_piix
  15:       4512         63   IO-APIC-edge      ata_piix
...
Interrupts are serviced on both cores, even though affinity is set to 1.

The command line for starting full software emulation VM is:
qemu-system-x86_64 -kernel bzImage -hda rootfs.ext2 -append "root=/dev/sda \
console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2 -m 2048

On the full software emulation guest, 'cat /proc/interrupts' shows:
            CPU0       CPU1
   0:         36          0   IO-APIC-edge      timer
   1:         10          0   IO-APIC-edge      i8042
   4:       1775          0   IO-APIC-edge      serial
   8:          1          0   IO-APIC-edge      rtc0
   9:          0          0   IO-APIC-fasteoi   acpi
  11:        345          0   IO-APIC-fasteoi   eth0
  12:        125          0   IO-APIC-edge      i8042
  14:       1720          0   IO-APIC-edge      ata_piix
  15:        481          0   IO-APIC-edge      ata_piix
...
Interrupts are serviced only on CPU0, therefore setting IRQ affinity worked.

Do you have any idea why KVM doesn't respect IRQ affinity on the guest?
What can I do to get it working with IRQ affinity set to 1?

Thanks,
Mihai

Re: IRQ affinity on Linux guest

[Radim Krčmář]

2015-08-20 17:16+0300, Mihai Neagu:
> Here is how IRQ affinity is configured on guest at startup, in an init.d
> script:
> 
> echo 1 > /proc/irq/default_smp_affinity
> for x in /proc/irq/*/smp_affinity;
> do
>   echo 1 > $x
> done 2> /dev/null
> 
> The command line for starting the hardware accelerated VM:
> qemu-system-x86-64 -enable-kvm -kernel bzImage -hda rootfs.ext2 -append    \
> "root=/dev/sda console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2   \
> -m 2048
> 
> On the hardware accelerated guest, 'cat /proc/interrupts' shows:
>            CPU0       CPU1
>   0:         26          0   IO-APIC-edge      timer
>   1:          7          4   IO-APIC-edge      i8042
>   4:       1137        523   IO-APIC-edge      serial
>   8:          0          1   IO-APIC-edge      rtc0
>   9:          0          0   IO-APIC-fasteoi   acpi
>  11:       4971          4   IO-APIC-fasteoi   eth0
>  12:         66         64   IO-APIC-edge      i8042
>  14:       1958        714   IO-APIC-edge      ata_piix
>  15:       4512         63   IO-APIC-edge      ata_piix
> ...
> Interrupts are serviced on both cores, even though affinity is set to 1.

KVM's APIC balances interrupts -- until you set the affinity (probably
near the end of boot process), both CPUs are going to receive roughly
the same amount but after directing subsequent interrupts to CPU0, CPU1
shouldn't receive more.

Please verify that CPU0 is not receiving all interrupts by doing a
difference between two `cat /proc/interrupts` after the affinity was
set.  (CPU0 has higher numbers in your excerpt, which makes me suspect
that it works as expected.)

Thanks.

Re: IRQ affinity on Linux guest

[Mihai Neagu]

Radim,

Thanks for your answer. Indeed setting IRQ affinity to a specific core 
seems to be respected.

However, on software emulation and on the real machine, while IRQ 
affinity defaults to 3, all interrupts go on CPU0, while on KVM they go 
on CPU1.

I wonder why KVM would act differently than both the real machine and 
the software emulation in this particular aspect.

Is there a machine or processor that I can specify at KVM command line 
to make it behave like the real x86_64 processor which defaults 
interrupts to CPU0?

Thanks,
Mihai

On 08/21/2015 02:20 AM, Radim Krčmář wrote:
> 2015-08-20 17:16+0300, Mihai Neagu:
>> Here is how IRQ affinity is configured on guest at startup, in an init.d
>> script:
>>
>> echo 1 > /proc/irq/default_smp_affinity
>> for x in /proc/irq/*/smp_affinity;
>> do
>>    echo 1 > $x
>> done 2> /dev/null
>>
>> The command line for starting the hardware accelerated VM:
>> qemu-system-x86-64 -enable-kvm -kernel bzImage -hda rootfs.ext2 -append    \
>> "root=/dev/sda console=ttyS0 rw" -nographic -cpu qemu64 -snapshot -smp 2   \
>> -m 2048
>>
>> On the hardware accelerated guest, 'cat /proc/interrupts' shows:
>>             CPU0       CPU1
>>    0:         26          0   IO-APIC-edge      timer
>>    1:          7          4   IO-APIC-edge      i8042
>>    4:       1137        523   IO-APIC-edge      serial
>>    8:          0          1   IO-APIC-edge      rtc0
>>    9:          0          0   IO-APIC-fasteoi   acpi
>>   11:       4971          4   IO-APIC-fasteoi   eth0
>>   12:         66         64   IO-APIC-edge      i8042
>>   14:       1958        714   IO-APIC-edge      ata_piix
>>   15:       4512         63   IO-APIC-edge      ata_piix
>> ...
>> Interrupts are serviced on both cores, even though affinity is set to 1.
> KVM's APIC balances interrupts -- until you set the affinity (probably
> near the end of boot process), both CPUs are going to receive roughly
> the same amount but after directing subsequent interrupts to CPU0, CPU1
> shouldn't receive more.
>
> Please verify that CPU0 is not receiving all interrupts by doing a
> difference between two `cat /proc/interrupts` after the affinity was
> set.  (CPU0 has higher numbers in your excerpt, which makes me suspect
> that it works as expected.)
>
> Thanks.
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IRQ affinity on Linux guest

[Paolo Bonzini]

On 21/08/2015 05:33, Mihai Neagu wrote:
> Radim,
> 
> Thanks for your answer. Indeed setting IRQ affinity to a specific core
> seems to be respected.
> 
> However, on software emulation and on the real machine, while IRQ
> affinity defaults to 3, all interrupts go on CPU0, while on KVM they go
> on CPU1.

Are you sure that KVM doesn't do some kind of round robin?

Also, it's probably not _all_ interrupts but most of them.  For example
my laptop has:

                                                            smp_affinity
        cpu0      cpu1     cpu2      cpu3                           ||||
1:     32824     1764       151       130   IO-APIC-edge  i8042     0123
12:  1587436   203990     16666     17064   IO-APIC-edge  i8042     0123
24:    63361     5160    212545      1276   PCI-MSI-edge  ahci      ..2.
25:        2        0      1082     10710   PCI-MSI-edge  xhci_hcd  ..23
29:  1867844    72169        61     57514   PCI-MSI-edge  iwlwifi   0...

I added a column at the end with the SMP affinity of the interrupts.
You can see that:

* real hardware also shows traces of interrupts delivered before the
affinity was set (especially vectors 24 and 29)

* real hardware also distributes interrupts across multiple CPUs when
the affinity covers multiple processors (see vectors 1, 12 and 25)

> I wonder why KVM would act differently than both the real machine and
> the software emulation in this particular aspect.
> 
> Is there a machine or processor that I can specify at KVM command line
> to make it behave like the real x86_64 processor which defaults
> interrupts to CPU0?

The behavior of interrupt arbitration is not specified in the processor
documentation and can change across processors and implementations.
Presumably it is either patented or an Intel trade secret.

Older processors were documented to pick a processor that was not
running another interrupt service routine.  This can explain why
software emulation shows that all interrupts go on CPU0.

At least some newer processors are not doing it anymore, and the SDM has
some high-level explanation of their behavior and its effects:

   [...] the chipset bus controller accepts messages from the I/O APIC
   agents in the system and directs interrupts to the processors on the
   system bus. When using the lowest priority delivery mode, the
   chipset chooses a target processor to receive the interrupt out of
   the set of possible targets.

   In operating systems that use the lowest priority delivery mode but
   do not update the TPR, the TPR information saved in the chipset will
   potentially cause the interrupt to be always delivered to the same
   processor from the logical set. This behavior is functionally
   backward compatible with the P6 family processor but may result in
   unexpected performance implications.

Other processors are doing so called "vector hashing" (i.e. pick a CPU
number based on the affinity mask and vector number, and use it most of
the time), which would also explain why the interrupts all go on CPU0.

Paolo