Re: [PATCH v2] x86, vt-d: enable x2apic opt out

[Alex Williamson <alex.williamson@redhat.com>]

On Wed, 2011-05-18 at 23:58 +0100, Woodhouse, David wrote:
> On Mon, 2011-05-16 at 21:32 +0100, Alex Williamson wrote:
> > 
> > > Is this just a workaround for a crappy BIOS? What is the *actual* reason
> > > for wanting to disable x2apic?
> > 
> > Just a guess, but the OEM probably hasn't updated their SMI handlers to
> > understand x2apic yet and won't before the product ships because some
> > other OS doesn't bother to use x2apic. 
> 
> But I think we'll still use x2apic if interrupt remapping isn't enabled
> (for example if VT-d is disabled in the BIOS and the whole DMAR table is
> absent), or if we're booted with 'iommu=off' or indeed if the
> distribution vendor has hacked the kernel so that iommu=off is the
> *default*, because they've seen so many broken BIOSes.
> 
> AFAICT although CONFIG_X86_X2APIC depends on CONFIG_INTR_REMAP, we can
> still enable x2apic at runtime if interrupt remapping is not operating?
> We end up hitting this code in enable_IR_x2apic():
> 
> 	if (!ret) {
> 		/* IR is required if there is APIC ID > 255 even when running
> 		 * under KVM
> 		 */
> 		if (max_physical_apicid > 255 ||
> 		    !hypervisor_x2apic_available())
> 			goto nox2apic;
> 		/*
> 		 * without IR all CPUs can be addressed by IOAPIC/MSI
> 		 * only in physical mode
> 		 */
> 		x2apic_force_phys();
> 	}
> 
> So if that *is* the reason, this doesn't seem like a viable solution.

I've always been under the impression that interrupt remapping is
required to enable x2apic because it enables IOxAPICs to work in that
mode.  Particularly this excerpt from the x2apic spec (sec 1.2):

        Similarly no modifications are required to the IOxAPIC. The
        routing of interrupts from these devices in x2APIC mode
        leverages the interrupt remapping architecture specified in the
        Intel Virtualization Technology for Directed I/O, Rev 1.1
        specification.

KVM wants to enable x2apic because it's easier to virtualize and
provides better performance than xapic.  We won't be taking the above
x2apic physical mode path on real hardware though, so I'm not sure that
code snippet is relevant here.

AIUI, platforms that require x2apic (>255 APIC IDs) probably aren't
going to have an option to disable VT-d in the BIOS.  If such a platform
hands off in xapic mode with iommu=off, you stay in xapic mode and don't
bring all the processors online.  If it hands off in x2apic mode with
iommu=off, hopefully we can switch back to xapic mode and lose
processors, but ISTR the x2apic->xapic transition isn't particularly
easy, so you may just be screwed.  Thanks,

Alex