INIT ipi under Xen

INIT ipi under Xen

[Rafal Wojtczuk]

Hello,
    Does anyone know whether Xen-4.0.1, after it has booted and created dom0, does 
anything special to prevent interprocessor interrupt with delivery mode INIT from 
affecting processors ? Judging by Intel Software Developer Manual there is no 
documented way to do this; but maybe something less official ?

    I use the following piece of code (in hypervisor context) to trigger IPI:
unsigned long send_init_ipi()
{
        unsigned long ret;
        int i;
        rdmsrl(IA32_APIC_BASE, ret);
        xen_printk("IA32_APIC_BASE=0x%x\n", ret);
        if (ret & (1 << 8)) {   // I am BSP
                *(unsigned int *) (APIC_BASE + 0x300) = 0xc4500; // load ICR1 
                asm("wbinvd"); // just in case APIC_BASE is cached WB
        }
        return ret;
}

This code does not seem to cause any effect (even when
IA32_APIC_BASE=0xfee00900 is logged) - namely, I can still observe both CPUs
running: subsequent triggering of this code sometimes results in
IA32_APIC_BASE=0xfee00900 logged, sometimes 0xfee00800.

If I try sending "ordinary" interrupts (by
eg writing 0x4169 do ICR1) I can see the interrupt count in /proc/cpuinfo
increase. Moreover, if I run the above code when a HVM is running, then it
is killed with note in the Xen log that unexpected exit_reason=3 (meaning,
EXIT_REASON_INIT) has beed observed. So, everything indicates that indeed
the above code generates INIT, but somehow it is ignored by the destination.
Even writing 0x84500 (send INIT to every CPU) causes no effect.

The similar code run on bare metal Linux behaves more sanely.
Can anyone offer a clue ? This is on Intel Q45 board, E8400 CPU, 64bit
Xen-4.0.1.

RW

PS
Similarly, sending SIPI causes nothing; but I would expect that SIPI is
reacted upon only in wait-for-SIPI state. However, this is not documented
anywhere - can someone confirm this assumption more authoritatively ?

Re: INIT ipi under Xen

[Keir Fraser]

On 14/01/2011 14:54, "Rafal Wojtczuk" <rafal@invisiblethingslab.com> wrote:

> Hello,
>     Does anyone know whether Xen-4.0.1, after it has booted and created dom0,
> does 
> anything special to prevent interprocessor interrupt with delivery mode INIT
> from 
> affecting processors ? Judging by Intel Software Developer Manual there is no
> documented way to do this; but maybe something less official ?

Intel SDM Vol.3B, 19.8 ("Restrictions on VMX Operation"), final bullet
point: "The INIT signal is blocked whenever a logical processor is in VMX
root operation."

Also, your code won't work if the APIC happens to be in x2APIC mode. You
should write to the ICR with apic_icr_write(), which handles both xAPIC and
x2APIC modes correctly.

 -- Keir

>     I use the following piece of code (in hypervisor context) to trigger IPI:
> unsigned long send_init_ipi()
> {
>         unsigned long ret;
>         int i;
>         rdmsrl(IA32_APIC_BASE, ret);
>         xen_printk("IA32_APIC_BASE=0x%x\n", ret);
>         if (ret & (1 << 8)) {   // I am BSP
>                 *(unsigned int *) (APIC_BASE + 0x300) = 0xc4500; // load ICR1
>                 asm("wbinvd"); // just in case APIC_BASE is cached WB
>         }
>         return ret;
> }
> 
> This code does not seem to cause any effect (even when
> IA32_APIC_BASE=0xfee00900 is logged) - namely, I can still observe both CPUs
> running: subsequent triggering of this code sometimes results in
> IA32_APIC_BASE=0xfee00900 logged, sometimes 0xfee00800.
> 
> If I try sending "ordinary" interrupts (by
> eg writing 0x4169 do ICR1) I can see the interrupt count in /proc/cpuinfo
> increase. Moreover, if I run the above code when a HVM is running, then it
> is killed with note in the Xen log that unexpected exit_reason=3 (meaning,
> EXIT_REASON_INIT) has beed observed. So, everything indicates that indeed
> the above code generates INIT, but somehow it is ignored by the destination.
> Even writing 0x84500 (send INIT to every CPU) causes no effect.
> 
> The similar code run on bare metal Linux behaves more sanely.
> Can anyone offer a clue ? This is on Intel Q45 board, E8400 CPU, 64bit
> Xen-4.0.1.
> 
> RW
> 
> PS
> Similarly, sending SIPI causes nothing; but I would expect that SIPI is
> reacted upon only in wait-for-SIPI state. However, this is not documented
> anywhere - can someone confirm this assumption more authoritatively ?
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: INIT ipi under Xen

[Keir Fraser]

On 14/01/2011 15:44, "Keir Fraser" <keir@xen.org> wrote:

> On 14/01/2011 14:54, "Rafal Wojtczuk" <rafal@invisiblethingslab.com> wrote:
> 
>> Hello,
>>     Does anyone know whether Xen-4.0.1, after it has booted and created dom0,
>> does 
>> anything special to prevent interprocessor interrupt with delivery mode INIT
>> from 
>> affecting processors ? Judging by Intel Software Developer Manual there is no
>> documented way to do this; but maybe something less official ?
> 
> Intel SDM Vol.3B, 19.8 ("Restrictions on VMX Operation"), final bullet
> point: "The INIT signal is blocked whenever a logical processor is in VMX
> root operation."

However, it might be remembered and acted on when the CPU executes VMXOFF. I
tried a variant on your code, and as expected it had no immediate effect,
but my system hung when trying to offline CPUs for ACPI S5 (power off). I
didn't dig to confirm, but one reason could be that the secondary CPU
processes the INIT as soon as it executes VMXOFF and hence stops executing
before it has ack'ed to the rest of Xen that it is going offline. Hence Xen
would hang waiting for an ack that never comes. Just a theory.

 -- Keir

> Also, your code won't work if the APIC happens to be in x2APIC mode. You
> should write to the ICR with apic_icr_write(), which handles both xAPIC and
> x2APIC modes correctly.
> 
>  -- Keir
> 
>>     I use the following piece of code (in hypervisor context) to trigger IPI:
>> unsigned long send_init_ipi()
>> {
>>         unsigned long ret;
>>         int i;
>>         rdmsrl(IA32_APIC_BASE, ret);
>>         xen_printk("IA32_APIC_BASE=0x%x\n", ret);
>>         if (ret & (1 << 8)) {   // I am BSP
>>                 *(unsigned int *) (APIC_BASE + 0x300) = 0xc4500; // load ICR1
>>                 asm("wbinvd"); // just in case APIC_BASE is cached WB
>>         }
>>         return ret;
>> }
>> 
>> This code does not seem to cause any effect (even when
>> IA32_APIC_BASE=0xfee00900 is logged) - namely, I can still observe both CPUs
>> running: subsequent triggering of this code sometimes results in
>> IA32_APIC_BASE=0xfee00900 logged, sometimes 0xfee00800.
>> 
>> If I try sending "ordinary" interrupts (by
>> eg writing 0x4169 do ICR1) I can see the interrupt count in /proc/cpuinfo
>> increase. Moreover, if I run the above code when a HVM is running, then it
>> is killed with note in the Xen log that unexpected exit_reason=3 (meaning,
>> EXIT_REASON_INIT) has beed observed. So, everything indicates that indeed
>> the above code generates INIT, but somehow it is ignored by the destination.
>> Even writing 0x84500 (send INIT to every CPU) causes no effect.
>> 
>> The similar code run on bare metal Linux behaves more sanely.
>> Can anyone offer a clue ? This is on Intel Q45 board, E8400 CPU, 64bit
>> Xen-4.0.1.
>> 
>> RW
>> 
>> PS
>> Similarly, sending SIPI causes nothing; but I would expect that SIPI is
>> reacted upon only in wait-for-SIPI state. However, this is not documented
>> anywhere - can someone confirm this assumption more authoritatively ?
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
>