Re: [PATCH][RFC] x86, hotplug: Use zero page for monitor when resuming from hibernation

[Chen Yu <yu.c.chen@intel.com>]

Hi Pavel,

On 2016年06月07日 16:03, Pavel Machek wrote:
> On Mon 2016-06-06 22:19:09, Chen Yu wrote:
>> Stress test from Varun Koyyalagunta reports that, the
>> nonboot CPU would hang occasionally, when resuming from
>> hibernation. Further investigation shows that, the precise
>> phase when nonboot CPU hangs, is the time when the nonboot
>> CPU been woken up incorrectly, and tries to monitor the
>> mwait_ptr for the second time, then an exception is
>> triggered due to illegal vaddr access, say, something like,
>> 'Unable to handler kernel address of 0xffff8800ba800010...'
>>
>> One of the possible scenarios for this issue is illustrated below,
>> when the boot CPU tries to resume from hibernation:
>> 1. puts the nonboot CPUs offline, so the nonboot CPUs are monitoring
>>     at the address of the task_struct.flags.
>> 2. boot CPU copies pages to their original address, which includes
>>     task_struct.flags, thus wakes up one of the nonboot CPUs.
>> 3. nonboot CPU tries to monitor the task_struct.flags again, but since
>>     the page table for task_struct.flags has been overwritten by
>>     boot CPU, and there is probably a changed across hibernation
>>     (because of inconsistence of e820 memory map), an exception is
>>     triggered.
> If memory map changes between suspend and resume, there'll be fun. If
> that's suspected, should we attach md5 sum of e820 to the hibernation
> image?
Actually what I described  in the  scenario might be not so accurate,
it might not be related to inconsistence of e820 map,
because there is no guarantee that boot kernel and resume kernel
have the same memory layout(page table).

I've re-checked the logs from reporter, it seems that, the fault
access is caused by accessing a page without PRESENT flag,
and the pte entry for this vaddr is zero:

    // DATA ADDRESS TRANSLATION, virtual address = 0xFFFF8800BA803E88
    //   DATA TABLE WALK, virtual address = 0xFFFF8800BA803E88
    pml4read   0x0001C12880    0x01FD2067; // pgd
    pdptread   0x0001FD2010    0x9BEBF063; // pud
    pderead    0x009BEBFEA0    0x2D9D9063; // pmd
    pteread    0x002D9D9018    0x00000000; // pte

The last line above is a pte entry located at physical address 0x2d9d9018,
with value zero, thus access to this vaddr results in a page-not-present exception.


Since some of the pud/pde/pte are allocated dynamically during bootup
(kernel_physical_mapping_init),
it is possible that, when the boot cpu writes to this vaddr,
the page table(especially for pud/pmd/pte) for this vaddr
are not the same as it is before hibernation,  thus an exception would be
triggered due to incorrect page table, even e820 is consistent.

I'm doing more test to verify this.

thanks,
Yu