Re: ACPI-Tables corrupted?

[Keir Fraser <keir.fraser@eu.citrix.com>]

On 29/07/2010 08:37, "Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:

> Sorry that I didn't notice it is for crash kernel. In fact, I tried kexec
> before and never succed to bring it up.
> 
> What do you mean of "stab at disabling x2apic"? You mean we need disable
> x2apic before transfer control to crash kernel, right?
> 
> Per my understanding, with kexec, when system crash, it will jump directly to
> new kernel's entry point, no guest destroy (i.e. clean-up), not reset signal
> to cpu/chipset, right?
> If yes, another issue need be considered is VT-d. I didn't find the vt-d
> disable code in xen's kexec_crash code, if the new kernel has no idea of vt-d
> (thus does not reset the vt-d engine), it may have trouble. Or, will the kexec
> kernel not use device assigned to guest?
> 
> Of course it is ok if crash kernel support vt-d too.

A crash kernel needs to be carefully configured anyway. For example it may
not be started up on the boot processor: it gets started on whichever cpu
initiated the crash.

 K.

> Thanks
> --jyh
> 
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Thursday, July 29, 2010 2:48 PM
>> To: Jiang, Yunhong; Juergen Gross
>> Cc: Jeremy Fitzhardinge; xen-devel@lists.xensource.com; Han, Weidong
>> Subject: Re: [Xen-devel] ACPI-Tables corrupted?
>> 
>> Strictly speaking we should have a stab at disabling x2apic as well, but
>> that's harder. And not necessary for newer Linux kernels, which is what we
>> will usually be kexec'ing to anyway.
>> 
>> -- Keir
>> 
>> On 29/07/2010 07:40, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
>> 
>>> It's not a dom0, it's a kexec'ed crash kernel. We should be reinstating DMAR
>>> before jumping into a native kernel. I will sort out a fix.
>>> 
>>>  -- Keir
>>> 
>>> On 29/07/2010 07:31, "Jiang, Yunhong" <yunhong.jiang@intel.com> wrote:
>>> 
>>>> A bit curios, why enable_IR_x2apic() will be called in dom0? IMO, dom0 will
>>>> not control interrupt controller, either xapic, or x2apic. Shouldn't this
>>>> be
>>>> commented out in pvops dom0?
>>>> 
>>>> What's the panic point in your environment? Is it the following code? If
>>>> yes,
>>>> that means you enable x2apic in BIOS and you can workaround this issue by
>>>> disable x2apic in BIOS.
>>>> 
>>>>         if (x2apic_preenabled)
>>>>                  panic("x2apic: enabled by BIOS but kernel init failed.");
>>>> 
>>>> Thanks
>>>> --jyh
>>>> 
>>>>> -----Original Message-----
>>>>> From: xen-devel-bounces@lists.xensource.com
>>>>> [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Juergen Gross
>>>>> Sent: Wednesday, July 28, 2010 8:14 PM
>>>>> To: Keir Fraser
>>>>> Cc: xen-devel@lists.xensource.com
>>>>> Subject: Re: [Xen-devel] ACPI-Tables corrupted?
>>>>> 
>>>>> On 07/28/2010 01:39 PM, Keir Fraser wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On 28/07/2010 12:26, "Juergen Gross"<juergen.gross@ts.fujitsu.com>
>> wrote:
>>>>>> 
>>>>>>> On 07/28/2010 12:03 PM, Keir Fraser wrote:
>>>>>>>> On 28/07/2010 10:38, "Juergen Gross"<juergen.gross@ts.fujitsu.com>
>>>>> wrote:
>>>>>>>> 
>>>>>>>>> As you can see, the DMAR eye-catcher is replaced by blanks!
>>>>>>>>> This leads to a programmed panic in the crash kernel later in case of
>>>>>>>>> a
>>>>>>>>> panic in dom0...
>>>>>>>>> 
>>>>>>>>> Any ideas?
>>>>>>>>> BTW: seen in unstable AND 4.0
>>>>>>>> 
>>>>>>>> Look at the tail of xen/drivers/passthrough/vtd/dmar.c: Xen *always*
>>>>>>>> *unconditionally* trashes the DMAR so that dom0 will not parse it.
>>>>>>>> Presumably bad stuff would happen if it did.
>>>>>>> 
>>>>>>> As Dom0 is a pv-kernel, it should be able to ignore this entry.
>>>>>>> The crash kernel OTOH should not panic due to the trashed entry!
>>>>>>> What is the correct solution here?
>>>>>> 
>>>>>> Could provide a cmdline option to not nobble the DMAR?
>>>>> 
>>>>> That's a possibility.
>>>>> I wonder whether it wouldn't be better to let dom0 decide not to use it if
>>>>> running under xen. This would remove the requirement for zapping the ACPI
>>>>> table. IMO it's always a bad idea to change data of a deeper layer...
>>>>> 
>>>>>> 
>>>>>>> The crash kernel expects a valid DMAR entry, as following code in
>>>>>>> enable_IR_x2apic() suggests:
>>>>>> 
>>>>>> I don't know what that function does, nor how the error path below
>>>>>> depends
>>>>>> on DMAR. DMAR isn't mentioned in the below code.
>>>>> 
>>>>> Sorry, here a larger fragment (source arch/x86/kernel/apic/apic.c):
>>>>> 
>>>>> void __init enable_IR_x2apic(void)
>>>>> {
>>>>>         unsigned long flags;
>>>>>         struct IO_APIC_route_entry **ioapic_entries = NULL;
>>>>>         int ret, x2apic_enabled = 0;
>>>>>         int dmar_table_init_ret = 0;
>>>>> 
>>>>> #ifdef CONFIG_INTR_REMAP
>>>>>         dmar_table_init_ret = dmar_table_init();
>>>>>         if (dmar_table_init_ret)
>>>>>                 pr_debug("dmar_table_init() failed with %d:\n",
>>>>>                                 dmar_table_init_ret);
>>>>> #endif
>>>>> 
>>>>>         ioapic_entries = alloc_ioapic_entries();
>>>>>         if (!ioapic_entries) {
>>>>>                 pr_err("Allocate ioapic_entries failed\n");
>>>>>                 goto out;
>>>>>         }
>>>>> 
>>>>>         ret = save_IO_APIC_setup(ioapic_entries);
>>>>>         if (ret) {
>>>>>                 pr_info("Saving IO-APIC state failed: %d\n", ret);
>>>>>                 goto out;
>>>>>         }
>>>>> 
>>>>>         local_irq_save(flags);
>>>>>         mask_8259A();
>>>>>         mask_IO_APIC_setup(ioapic_entries);
>>>>> 
>>>>>         if (dmar_table_init_ret)
>>>>>                 ret = 0;
>>>>>         else
>>>>>                 ret = enable_IR();
>>>>> 
>>>>>         if (!ret) {
>>>>>                 /* IR is required if there is APIC ID > 255 even when
>>>>> running
>>>>>                  * under KVM
>>>>>                  */
>>>>>                 if (max_physical_apicid > 255 || !kvm_para_available())
>>>>>                         goto nox2apic;
>>>>>                 /*
>>>>>                  * without IR all CPUs can be addressed by IOAPIC/MSI
>>>>>                  * only in physical mode
>>>>>                  */
>>>>>                 x2apic_force_phys();
>>>>>         }
>>>>> 
>>>>>         x2apic_enabled = 1;
>>>>> 
>>>>>         if (x2apic_supported() && !x2apic_mode) {
>>>>>                 x2apic_mode = 1;
>>>>>                 enable_x2apic();
>>>>>                 pr_info("Enabled x2apic\n");
>>>>>         }
>>>>> 
>>>>> nox2apic:
>>>>>         if (!ret) /* IR enabling failed */
>>>>>                 restore_IO_APIC_setup(ioapic_entries);
>>>>>         unmask_8259A();
>>>>>         local_irq_restore(flags);
>>>>> 
>>>>> out:
>>>>>         if (ioapic_entries)
>>>>>                 free_ioapic_entries(ioapic_entries);
>>>>> 
>>>>>         if (x2apic_enabled)
>>>>>                 return;
>>>>> 
>>>>>         if (x2apic_preenabled)
>>>>>                 panic("x2apic: enabled by BIOS but kernel init failed.");
>>>>>         else if (cpu_has_x2apic)
>>>>>                 pr_info("Not enabling x2apic, Intr-remapping init
>>>>> failed.\n");
>>>>> }
>>>>> 
>>>>> 
>>>>> dmar_table_init() will return -ENODEV if no DMAR record is found.
>>>>> 
>>>>> 
>>>>> Juergen
>>>>> 
>>>>> --
>>>>> Juergen Gross                 Principal Developer Operating Systems
>>>>> TSP ES&S SWE OS6                       Telephone: +49 (0) 89 3222 2967
>>>>> Fujitsu Technology Solutions              e-mail:
>>>>> juergen.gross@ts.fujitsu.com
>>>>> Domagkstr. 28                           Internet: ts.fujitsu.com
>>>>> D-80807 Muenchen                 Company details:
>>>>> ts.fujitsu.com/imprint.html
>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@lists.xensource.com
>>>>> http://lists.xensource.com/xen-devel
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>> 
>