APIC MSRs query

APIC MSRs query

[Andrew Cooper]

Hello,

I am currently cleaning up the APIC code for the sake of 
shutdown/reboot/crashdump and have a query about the (modified for 
brevity) snippet of code:

uint64_t msr_content;
rdmsrl(MSR_IA32_APICBASE, msr_content);
msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
msr_content = (uint32_t)msr_content;
wrmsrl(MSR_IA32_APICBASE, msr_content);

which is added into apic.c in changeset b622e411eef8, and has propagated 
elsewhere in the codebase during subsequent cleanups etc.

The MP spec and x2apic spec states that bits [35:12] of 
MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why 
the code (almost always) clears the top 4 bits, or is it just an 
overlooked mistake?

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: APIC MSRs query

[Jan Beulich]

>>> On 17.05.11 at 15:25, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> Hello,
> 
> I am currently cleaning up the APIC code for the sake of 
> shutdown/reboot/crashdump and have a query about the (modified for 
> brevity) snippet of code:
> 
> uint64_t msr_content;
> rdmsrl(MSR_IA32_APICBASE, msr_content);
> msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
> msr_content = (uint32_t)msr_content;
> wrmsrl(MSR_IA32_APICBASE, msr_content);
> 
> which is added into apic.c in changeset b622e411eef8, and has propagated 
> elsewhere in the codebase during subsequent cleanups etc.
> 
> The MP spec and x2apic spec states that bits [35:12] of 
> MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why 
> the code (almost always) clears the top 4 bits, or is it just an 
> overlooked mistake?

I think this is a benign mistake. Benign because I don't think there is
a meaningful (to Xen at least) number of systems that would not
have their LAPIC at the default address (which fits in 32 bits).

Jan

Re: APIC MSRs query

[Andrew Cooper]

On 17/05/11 14:43, Jan Beulich wrote:
>>>> On 17.05.11 at 15:25, Andrew Cooper<andrew.cooper3@citrix.com>  wrote:
>> Hello,
>>
>> I am currently cleaning up the APIC code for the sake of
>> shutdown/reboot/crashdump and have a query about the (modified for
>> brevity) snippet of code:
>>
>> uint64_t msr_content;
>> rdmsrl(MSR_IA32_APICBASE, msr_content);
>> msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
>> msr_content = (uint32_t)msr_content;
>> wrmsrl(MSR_IA32_APICBASE, msr_content);
>>
>> which is added into apic.c in changeset b622e411eef8, and has propagated
>> elsewhere in the codebase during subsequent cleanups etc.
>>
>> The MP spec and x2apic spec states that bits [35:12] of
>> MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why
>> the code (almost always) clears the top 4 bits, or is it just an
>> overlooked mistake?
> I think this is a benign mistake. Benign because I don't think there is
> a meaningful (to Xen at least) number of systems that would not
> have their LAPIC at the default address (which fits in 32 bits).
>
> Jan
>
Ok - I will fix this up in my cleanup.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: APIC MSRs query

[Ian Campbell]

On Tue, 2011-05-17 at 14:43 +0100, Jan Beulich wrote:
> >>> On 17.05.11 at 15:25, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
> > Hello,
> > 
> > I am currently cleaning up the APIC code for the sake of 
> > shutdown/reboot/crashdump and have a query about the (modified for 
> > brevity) snippet of code:
> > 
> > uint64_t msr_content;
> > rdmsrl(MSR_IA32_APICBASE, msr_content);
> > msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
> > msr_content = (uint32_t)msr_content;
> > wrmsrl(MSR_IA32_APICBASE, msr_content);
> > 
> > which is added into apic.c in changeset b622e411eef8, and has propagated 
> > elsewhere in the codebase during subsequent cleanups etc.
> > 
> > The MP spec and x2apic spec states that bits [35:12] of 
> > MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why 
> > the code (almost always) clears the top 4 bits, or is it just an 
> > overlooked mistake?
> 
> I think this is a benign mistake. Benign because I don't think there is
> a meaningful (to Xen at least) number of systems that would not
> have their LAPIC at the default address (which fits in 32 bits).

That "msr_content = (uint32_t)msr_content;" seems to be pretty
deliberate, what else would it be trying to do? 

FWIW enable_x2apic in Linux seems to have a similar construct which
throws away the top half of the MSR:

void enable_x2apic(void)
{
       int msr, msr2;

       rdmsr(MSR_IA32_APICBASE, msr, msr2);
       if (!(msr & X2APIC_ENABLE)) {
               printk("Enabling x2apic\n");
               wrmsr(MSR_IA32_APICBASE, msr | X2APIC_ENABLE, 0);
       }
}

(FWIW the original Xen code in 17545:9fd00ff95068 looked a lot like this
too, b622e411eef8 just switched to wrmsrl and preserved the clearing
behaviour).

Perhaps there is some errata? Google didn't find one, but ...

Ian.

Re: APIC MSRs query

[Andrew Cooper]

On 17/05/11 14:59, Ian Campbell wrote:
> On Tue, 2011-05-17 at 14:43 +0100, Jan Beulich wrote:
>>>>> On 17.05.11 at 15:25, Andrew Cooper<andrew.cooper3@citrix.com>  wrote:
>>> Hello,
>>>
>>> I am currently cleaning up the APIC code for the sake of
>>> shutdown/reboot/crashdump and have a query about the (modified for
>>> brevity) snippet of code:
>>>
>>> uint64_t msr_content;
>>> rdmsrl(MSR_IA32_APICBASE, msr_content);
>>> msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
>>> msr_content = (uint32_t)msr_content;
>>> wrmsrl(MSR_IA32_APICBASE, msr_content);
>>>
>>> which is added into apic.c in changeset b622e411eef8, and has propagated
>>> elsewhere in the codebase during subsequent cleanups etc.
>>>
>>> The MP spec and x2apic spec states that bits [35:12] of
>>> MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why
>>> the code (almost always) clears the top 4 bits, or is it just an
>>> overlooked mistake?
>> I think this is a benign mistake. Benign because I don't think there is
>> a meaningful (to Xen at least) number of systems that would not
>> have their LAPIC at the default address (which fits in 32 bits).
> That "msr_content = (uint32_t)msr_content;" seems to be pretty
> deliberate, what else would it be trying to do?
>
> FWIW enable_x2apic in Linux seems to have a similar construct which
> throws away the top half of the MSR:
>
> void enable_x2apic(void)
> {
>         int msr, msr2;
>
>         rdmsr(MSR_IA32_APICBASE, msr, msr2);
>         if (!(msr&  X2APIC_ENABLE)) {
>                 printk("Enabling x2apic\n");
>                 wrmsr(MSR_IA32_APICBASE, msr | X2APIC_ENABLE, 0);
>         }
> }
>
> (FWIW the original Xen code in 17545:9fd00ff95068 looked a lot like this
> too, b622e411eef8 just switched to wrmsrl and preserved the clearing
> behaviour).
>
> Perhaps there is some errata? Google didn't find one, but ...
>
> Ian.
>
I couldn't find any errata which is why I asked here.  Bits [63:36] are 
reserved so should be WriteAsZero - it is possible that whoever put it 
into Linux just wanted to zero the top bits and either missed the top 4 
bits or decided that they would never be set.

-- 
Andrew Cooper - Dom0 Kernel Engineer, Citrix XenServer
T: +44 (0)1223 225 900, http://www.citrix.com

Re: APIC MSRs query

[Jan Beulich]

>>> On 17.05.11 at 15:59, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Tue, 2011-05-17 at 14:43 +0100, Jan Beulich wrote:
>> >>> On 17.05.11 at 15:25, Andrew Cooper <andrew.cooper3@citrix.com> wrote:
>> > Hello,
>> > 
>> > I am currently cleaning up the APIC code for the sake of 
>> > shutdown/reboot/crashdump and have a query about the (modified for 
>> > brevity) snippet of code:
>> > 
>> > uint64_t msr_content;
>> > rdmsrl(MSR_IA32_APICBASE, msr_content);
>> > msr_content |= MSR_IA32_APICBASE_ENABLE | MSR_IA32_APICBASE_EXTD;
>> > msr_content = (uint32_t)msr_content;
>> > wrmsrl(MSR_IA32_APICBASE, msr_content);
>> > 
>> > which is added into apic.c in changeset b622e411eef8, and has propagated 
>> > elsewhere in the codebase during subsequent cleanups etc.
>> > 
>> > The MP spec and x2apic spec states that bits [35:12] of 
>> > MSR_IA32_APICBASE is the base APIC MMIO address.  Is there reason why 
>> > the code (almost always) clears the top 4 bits, or is it just an 
>> > overlooked mistake?
>> 
>> I think this is a benign mistake. Benign because I don't think there is
>> a meaningful (to Xen at least) number of systems that would not
>> have their LAPIC at the default address (which fits in 32 bits).
> 
> That "msr_content = (uint32_t)msr_content;" seems to be pretty
> deliberate, what else would it be trying to do? 
> 
> FWIW enable_x2apic in Linux seems to have a similar construct which
> throws away the top half of the MSR:

Surely the Xen code got cloned from the Linux one.

> void enable_x2apic(void)
> {
>        int msr, msr2;
> 
>        rdmsr(MSR_IA32_APICBASE, msr, msr2);
>        if (!(msr & X2APIC_ENABLE)) {
>                printk("Enabling x2apic\n");
>                wrmsr(MSR_IA32_APICBASE, msr | X2APIC_ENABLE, 0);
>        }
> }
> 
> (FWIW the original Xen code in 17545:9fd00ff95068 looked a lot like this
> too, b622e411eef8 just switched to wrmsrl and preserved the clearing
> behaviour).

This is what I assumed, and I'm sure it's just for simplicity that 0 gets
passed here. I don't think it's correct, the more that the actual base
address doesn't matter while in x2apic mode (but would matter when
switching back to legacy mode e.g. for kexec).

Jan

Re: APIC MSRs query

[Keir Fraser]

On 17/05/2011 15:08, "Jan Beulich" <JBeulich@novell.com> wrote:

>>> I think this is a benign mistake. Benign because I don't think there is
>>> a meaningful (to Xen at least) number of systems that would not
>>> have their LAPIC at the default address (which fits in 32 bits).
>> 
>> That "msr_content = (uint32_t)msr_content;" seems to be pretty
>> deliberate, what else would it be trying to do?
>> 
>> FWIW enable_x2apic in Linux seems to have a similar construct which
>> throws away the top half of the MSR:
> 
> Surely the Xen code got cloned from the Linux one.

If you actually look at the change in 21669:b622e411, you can see that the
above line is deliberate. The old code did wrmsr(msr, lo, 0), and clearing
upper 32 bits before calling wrmsrl() is the equivalent of that.

So this should not be 'cleaned up' without understanding that old code
(which almost certainly itself comes from Linux).

And I should also warn that I will simply ignore gratuitous cleanups in the
APIC code. The code is mostly Linux sourced, I'd rather we synced again than
diverge further.

 -- Keir