Re: questions of vm save/restore on arm64

[Chenxiao Zhao <chenxiao.zhao@gmail.com>]

On 6/6/2016 7:58 PM, Stefano Stabellini wrote:
> On Fri, 3 Jun 2016, Chenxiao Zhao wrote:
>> On 6/3/2016 4:02 AM, Julien Grall wrote:
>>> Hello,
>>>
>>> First thing, the time in the mail headers seems to be wrong. Maybe
>>> because of a wrong timezone?
>>>
>>> I got: 04/06/16 02:32 however we are still the 3rd in my timezone.
>>>
>>> On 04/06/16 02:32, Chenxiao Zhao wrote:
>>>>
>>>>
>>>> On 6/3/2016 3:16 AM, Julien Grall wrote:
>>>>> Hello,
>>>>>
>>>>> On 03/06/16 18:05, Chenxiao Zhao wrote:
>>>>>> I finally found out that the problem is that the toolstack did not get
>>>>>> corret p2m_size while sending all pages on save(always be zero).
>>>>>> After I
>>>>>> fixed that, the guest could be restored but guest kernel caught
>>>>>> handle_mm_fault().
>>>>>>
>>>>>> where do you think I'm going to investigate, guest kernel hibernation
>>>>>> restore or xen?
>>>>>
>>>>> The hibernation support for ARM64 has only been merged recently in the
>>>>> kernel. Which kernel are you using?
>>>>
>>>> Hi Julien,
>>>>
>>>> I'm using a linaro ported Linux kernel 4.1 for hikey from this link.
>>>>
>>>> https://github.com/96boards/linux/tree/android-hikey-linaro-4.1
>>>>
>>>> I also applied following patches to make the kernel support hibernation.
>>>
>>> This looks the wrong way to do it as this series may requires some
>>> patches which have been upstreamed before hand.
>>>
>>> Linux upstream seems support to the hikey board [1]. Any reason to not
>>> using it?
>>
>> I tried a newer version of kernel 4.4, but got no luck to start dom0 with xen.
>> so I decide to stay in 4.1 for now.
>>
>>>
>>>> [1] http://www.spinics.net/lists/arm-kernel/msg477769.html
>>>> [2] http://lists.xen.org/archives/html/xen-devel/2015-12/msg01068.html
>>>>
>>>>>
>>>>> Also, what are the modifications you have made to support Xen
>>>>> suspend/resume for ARM64?
>>>>
>>>> I believe I have posted my modifications on xen in the first mail of
>>>> this thread.
>>>
>>> I mean in Linux. The patch from Ian Campbell does not have any kind of
>>> support for ARM64.
>>>
>>> For instance arch/arm/xen/suspend.c needs to be built for ARM64. So I am
>>> wondering if your kernel has support of hibernation...
>>
>> Oh, yes, I most forgot I added this file in arch/arm64/xen/Makefile to let it
>> build for arm64.
>>>
>>>>
>>>>  From my understanding, a kernel hibernation will cause kernel to save
>>>> memories to disk(swap partition). But on guest save progress, the
>>>> hibernation for domU does not make the guest save memories to disk. it's
>>>> more like suspend all processes in guest, and memors actually depends on
>>>> xen toolstack to save the pages to file. Am I correct?
>>>
>>> You are using an older tree with a patch series based on a newer tree.
>>>
>>> So I would recommend you to move to a newer tree. If it is not possible,
>>> please test that hibernation works on baremetal.
>>
>> I think the suspend/resume in guest is working, cause I can use pause/unpause
>> command in toolstack to suspend/resume guest without problem. I can also see
>> the suspend/resume kernel messages from guest's console. The only problem is
>> it's can not resume from restore.
>
> But can you still connect to the guest after resume, maybe over the network?
> If you cannot, then something is likely wrong.

Hi Stefano,

I can connect to the guest after resume from xen console. It responds by 
'return' key, but I can not run any other commands, e.g. ls or ps. I 
think the guest is not 'fully' restored.

>
>
>> One thing that confused me is that the kernel's hibernation means the guest
>> kernel will save the memory state to disk and power off VM at last. The guest
>> will also take care of the memory restore itself. But I do not see the
>> save/restore on xen works that way. So my question is why it requires
>> hibernation (aka. suspend to disk) instead of the real suspend (aka. suspend
>> to RAM and standby)?
>
> Xen suspend/resume has nothing to do with guest suspend to RAM or guest
> hibernation.
>
> Xen suspend/resume is a way for the hypervisor to save to file the
> entire state of the VM, including RAM and the state of any devices.
> Guest suspend to RAM and guest hibernation are two guest driven
> technologies to save the state of the operating system to RAM or to
> disk. The only link between Xen suspend and guest suspend is that when
> Xen issues a domain suspend, it notifies the guest of it so that it can
> ease the process.  The code in Linux to support Xen suspend/resume is:
>
> drivers/xen/manage.c:do_suspend
>
> and makes use of some of the Linux internal hooks provided for
> hibernations (see CONFIG_HIBERNATE_CALLBACKS). But that's just for
> better integration with the rest of Linux: hibernation is NOT what is
> happening.
>
> I hope that this clarifies things a bit, I realize that it is confusing.
>

Thanks for your explanation, It clear enough and just as my 
understanding from the code. I think the problem might caused by 
incompatible of arm p2m and xen save/restore mechanism. I'll try a 
core-dump and compare the memory after save and restore. I suppose this 
two dumps should be identical but there already pages are different. 
I'll let you know if I got some progress.

Regards.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel