Re: [PATCH v2 0/4] KVM: APIC improvements (with bonus mixed mode)

["Radim Krčmář" <rkrcmar@redhat.com>]

2015-02-25 22:55-0300, Marcelo Tosatti:
> >  1) when we have no x2APIC ID 0xff000000, but send x2APIC message there
> 
> 10.7 SYSTEM AND APIC BUS ARBITRATION
> 
> Note that except for the SIPI IPI (see Section 10.6.1, “Interrupt
> Command Register (ICR)”), all bus messages that fail to be delivered
> to their specified destination or destinations are automatically
> retried. Software should avoid situations in which IPIs are sent to
> disabled or nonexistent local APICs, causing the messages to be resent
> repeatedly.

Perfect, thanks.
(I thought that the restriction only applies to lowest priority.)

> > or after something that isn't forbidden in SDM most likely because they
> > didn't think that anyone would ever consider it
> >  2) we have x2APIC ID 0xff000000 and reset other x2APICs into xAPIC mode
> 
> Or just x2APIC initialization in non lockstep:
> 
> 
> 	vcpu0				vcpu1
> T0) 	xapic mode			xapic mode
> T1)	x2apic enable
> T2)					broadcast IPI

Good point, there is no reason why the BIOS couldn't do that.
(If we have APIC ID > 255, OS gets CPUs in x2APIC mode, or with high IDs
 disabled; 10.12.8.1 Consistency of APIC IDs and CPUID)

> > No volunteer came to clear this up, so I hacked Linux to have one x2APIC
> > core between xAPIC ones.  Physical IPI to 0xff000000 got delivered only
> > to CPU0, like most other destinations, Logical IPI to 0xff000000 got
> > dropped and only 0xffffffff worked as a broadcast in both modes.
> > 
> > I think it is because Intel never considered mixed mode to be valid, and
> > seen delivery might be an emergent feature of QPI routing.
> 
> > Luckily, broadcast from xAPIC isn't delivered to x2APIC.
> 
> In real hardware?

Yes, it was ivy bridge, iirc.

> > Real exploration would require greater modifications to Linux (ideally
> > writing a custom kernel), so this series implements something that makes
> > some sense and isn't too far from reality.
> 
> Ok, so the problem is that broadcast (ICR.destination == 0xff000000)
> from xAPIC CPU is not delivered to x2APIC CPUs ?

Mixed mode doesn't work much without this series -- we internally shift
all destinations to 0x0-0xff range and use a map that can't handle
duplicate entries, so the broadcast IPI is just a detail.

The biggest problem that for me is that we allow x2APIC without
interrupt remapping, so all devices use (IO)xAPIC and broadcast is
inconsistent with the design -- interpreted as a 0xff x2APIC message.

> I can't find any restriction against (cpu0==x2APIC, cpu1==xAPIC) in 
> the documentation.

Me neither ... the most I found is that boot restriction mentioned
earlier. (10.12.8.1)

> Anyway, emulation should match physical hardware.

(I think that we have created a de facto standard by rejecting SDM,
 10.12.7 Initialization by System Software, and no OS that enabled
 x2APIC under KVM should do things that do not work.  The main benefit
 of this series is making the code better.)

>                                                   From your message above,
> it is not clear what is the behaviour there:
> 
> "No volunteer came to clear this up, so I hacked Linux to have one x2APIC
> core between xAPIC ones.  Physical IPI to 0xff000000 got delivered only
> to CPU0, like most other destinations, Logical IPI to 0xff000000 got
> dropped and only 0xffffffff worked as a broadcast in both modes.
> 
> Luckily, broadcast from xAPIC isn't delivered to x2APIC."
> 
> From the x2APIC CPU or the xAPIC ones?

The first paragraph was about x2APIC IPIs and their results on xAPICs.

The second paragraph was a broadcast from xAPIC. (I actually did the
reverse for that one -- one xAPIC besides x2APIC CPUs.)

> It should be easy to write kvm-unit-test testcases that match physical
> hardware behaviour (in general, i am having a hard time figure out
> in what way "mixed mode" is supposed to behave, please describe it
> clearly).

I think that mixed mode is undefined in real hardware: delivering x2APIC
0xff112233 to xAPIC 0 makes not sense.
I already wrote all that I know about it, sorry.