[PATCH 0/4] HVM Virtual S3

[PATCH 0/4] HVM Virtual S3

[Yu, Ke]

[PATCH 0/4] HVM Virtual S3

These set of patches are our prototype for HVM virtual ACPI S3 support:
 - patch 1/4: Xen interface for HVM S3
 - patch 2/4: QEMU interface for HVM S3
 - patch 3/4: rombios interface for HVM S3
 - patch 4/4: xend interface for HVM S3

The main idea is:
- emulate ACPI PM1A control resiger in QEMU to capture guest S3 request
- when QEMU capture guest S3 request, it call hypercall to trap to Xen
- Xen will suspend HVM by reseting all vcpus
- on resume, "xm resume" will call hypercall to trap to Xen
- Xen will resume HVM by setting HVM vcpu EIP to VMXASSIST BASE, which
will start rombios post code
- rombios post code will start s3 resume by jumping to wakeup vector set
by guest OS.

How to use it:
- apply this patch to changeset 15050:05c128b0188a
- create and boot HVM domain 
- In HVM guest, enter S3 state
  * for Linux, "echo mem >/sys/power/state"
  * for Windows, shutdown windows by Standby
- to resume HVM domain, "xm resume <domid>"

Note: FC6 guest can successfully S3 suspend/resume in both text mode and
X window. However, Windows guest can not suspend, because the windows
driver for QEMU VGA and monitor has no power management support, and
windows refuse to enter S3 sleep state. 

Kevin/Ke

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 16/5/07 17:48, "Yu, Ke" <ke.yu@intel.com> wrote:

> The main idea is:
> - emulate ACPI PM1A control resiger in QEMU to capture guest S3 request
> - when QEMU capture guest S3 request, it call hypercall to trap to Xen

So why emulate this register in QEMU at all, rather than directly in Xen?
Xen already knows the address of the pm1a block of ports because it emulates
the pmtimer.

 -- Keir

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 16/5/07 17:48, "Yu, Ke" <ke.yu@intel.com> wrote:

> - apply this patch to changeset 15050:05c128b0188a
> - create and boot HVM domain
> - In HVM guest, enter S3 state
>   * for Linux, "echo mem >/sys/power/state"
>   * for Windows, shutdown windows by Standby
> - to resume HVM domain, "xm resume <domid>"

Noone's seriously going to want to use S3-S5 in this way though, are they.
Keeping domains hanging around in sleep states taking up memory doesn't make
sense -- users will want to save the domain off to disc and restore later.
So this only makes sense at all if integrated with existing HVM
save/restore.

As I understand it, the only difference between S3 and S5 is that RAM
contents are maintained when the machine awakens. In this case S3 is like a
simpler version of current save/restore where we do not necessarily need to
save device state (but we might decide to if it makes the implementation
easier).

I think actually this isn't far off from being what we want. If the
s3_suspend hypercall put the domain into suspend state, this would then
neatly trigger xc_save to do its stuff. Then you can strip out s3_resume
because you should automatically start executing BIOS code when xc_resume
restores VCPU state and kicks the domain to start running again. The
sleep_state domain variable you added shouldn't be necessary imo.

 -- Keir

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 16/5/07 22:59, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:

>> The main idea is:
>> - emulate ACPI PM1A control resiger in QEMU to capture guest S3 request
>> - when QEMU capture guest S3 request, it call hypercall to trap to Xen
> 
> So why emulate this register in QEMU at all, rather than directly in Xen?
> Xen already knows the address of the pm1a block of ports because it emulates
> the pmtimer.

Oh, I see there is device state to be reset in QEMU. Emulating the port
access in QEMU makes sense then, but I wonder if rather than adding an extra
hypercall command we could emulate it in both Xen and QEMU: Xen emulates the
instruction, resets state and triggers domain shutdown, then passes the port
access up to QEMU so it does its thing also. We do this emulate-in-both for
other things already (e.g., the CMOS index register I believe).

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser  wrote on 6:34:
> On 16/5/07 22:59, "Keir Fraser" <Keir.Fraser@cl.cam.ac.uk> wrote:
> 
>>> The main idea is:
>>> - emulate ACPI PM1A control resiger in QEMU to capture guest S3
>>> request 
>>> - when QEMU capture guest S3 request, it call hypercall to trap to
>>> Xen 
>> 
>> So why emulate this register in QEMU at all, rather than directly in
>> Xen? Xen already knows the address of the pm1a block of ports
>> because it emulates the pmtimer.
> 
> Oh, I see there is device state to be reset in QEMU. Emulating the
> port access in QEMU makes sense then, but I wonder if rather than
> adding an extra hypercall command we could emulate it in both Xen and
> QEMU: Xen emulates the instruction, resets state and triggers domain
> shutdown, then passes the port access up to QEMU so it does its thing
> also. We do this emulate-in-both for other things already (e.g., the
> CMOS index register I believe). 
> 
>  -- Keir

Yes. that's a good idea. The only disadvantage is that it scatter the
ACPI register logic from QEMU, and make it a little bit harder for
maintaince. But it is a acceptable tradeoff. 

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser  wrote on 2007年5月17日 6:30:
> On 16/5/07 17:48, "Yu, Ke" <ke.yu@intel.com> wrote:
> 
>> - apply this patch to changeset 15050:05c128b0188a
>> - create and boot HVM domain
>> - In HVM guest, enter S3 state
>>   * for Linux, "echo mem >/sys/power/state"
>>   * for Windows, shutdown windows by Standby
>> - to resume HVM domain, "xm resume <domid>"
> 
> Noone's seriously going to want to use S3-S5 in this way though, are
> they. Keeping domains hanging around in sleep states taking up memory
> doesn't make sense -- users will want to save the domain off to disc
> and restore later. So this only makes sense at all if integrated with
> existing HVM save/restore.
> 
> As I understand it, the only difference between S3 and S5 is that RAM
> contents are maintained when the machine awakens. In this case S3 is
> like a simpler version of current save/restore where we do not
> necessarily need to save device state (but we might decide to if it
> makes the implementation easier).
> 
> I think actually this isn't far off from being what we want. If the
> s3_suspend hypercall put the domain into suspend state, this would
> then neatly trigger xc_save to do its stuff. Then you can strip out
> s3_resume because you should automatically start executing BIOS code
> when xc_resume restores VCPU state and kicks the domain to start
> running again. The sleep_state domain variable you added shouldn't be
> necessary imo. 
> 
>  -- Keir

My concern here is that: save/restore is a heavy operation just like S4 (hibernate), while the purpose of S3 is quick suspend and quick resume comapred to S4. if we implement S3 like save/restore, I don't see the value here, because HVM save/resotre or HVM S4 is just enough. How do you think?

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Kaushik Barde]

Can you please shed some more light on Windows guest s3 behavior?

Is it due to inadequate ACPI support in rombios?

-Kaushik
 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yu, Ke
Sent: Wednesday, May 16, 2007 9:48 AM
To: xen-devel@lists.xensource.com
Subject: [Xen-devel] [PATCH 0/4] HVM Virtual S3

[PATCH 0/4] HVM Virtual S3

These set of patches are our prototype for HVM virtual ACPI S3 support:
 - patch 1/4: Xen interface for HVM S3
 - patch 2/4: QEMU interface for HVM S3
 - patch 3/4: rombios interface for HVM S3
 - patch 4/4: xend interface for HVM S3

The main idea is:
- emulate ACPI PM1A control resiger in QEMU to capture guest S3 request
- when QEMU capture guest S3 request, it call hypercall to trap to Xen
- Xen will suspend HVM by reseting all vcpus
- on resume, "xm resume" will call hypercall to trap to Xen
- Xen will resume HVM by setting HVM vcpu EIP to VMXASSIST BASE, which
will start rombios post code
- rombios post code will start s3 resume by jumping to wakeup vector set
by guest OS.

How to use it:
- apply this patch to changeset 15050:05c128b0188a
- create and boot HVM domain 
- In HVM guest, enter S3 state
  * for Linux, "echo mem >/sys/power/state"
  * for Windows, shutdown windows by Standby
- to resume HVM domain, "xm resume <domid>"

Note: FC6 guest can successfully S3 suspend/resume in both text mode and
X window. However, Windows guest can not suspend, because the windows
driver for QEMU VGA and monitor has no power management support, and
windows refuse to enter S3 sleep state. 

Kevin/Ke

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

The symptom is that: if use stdvga in config file, the windows guest
will not show "standby" in shutdown menu, which means S3 is disabled.
The following link explains why.
http://support.microsoft.com/?scid=kb%3Ben-us%3B257562&x=16&y=12 

When use cirrus_vga, the "standby" appears in shutdown menu. However,
after "standby" is pressed, windows will pop up a dialog box, saying
default monitor driver has no power  management support, so windows
refuse to go further. 

I think Windows has strict checking on standby. it won't do standby
until the required device driver has suspend/resume support. so it is
due to qemu device emulation, not ACPI in rombios.

Best Regards
Ke

Kaushik Barde  wrote on 11:14:
> Can you please shed some more light on Windows guest s3 behavior?
> 
> Is it due to inadequate ACPI support in rombios?
> 
> -Kaushik
> 

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 17/5/07 03:32, "Yu, Ke" <ke.yu@intel.com> wrote:

> My concern here is that: save/restore is a heavy operation just like S4
> (hibernate), while the purpose of S3 is quick suspend and quick resume
> comapred to S4. if we implement S3 like save/restore, I don't see the value
> here, because HVM save/resotre or HVM S4 is just enough. How do you think?

I do not think that pure virtual S3 by itself makes sense. Unless the whole
machine is going into S3, what really is the difference between a HVM guest
in S3 and an HVM guest that simply is idle and so has all its VCPUs HLTed
99.9% of the time? Both are tying up memory resource, neither is burning
significant CPU resource or I/O resource.

Virtual S3 *does* make sense with HVM save/restore because it makes even an
HVM guest without PV drivers aware of the save/restore event so it can do
things like resync its clock. That is why we are interested in virtual S3:
only as an adjunct to save/restore.

Another application would be as part of host S3. Given that those patches
currently save/restore all domains (which actually I think is stupid, but
that's for a separate email) you should automatically improve host S3 by
integrating with save/restore. However, again, the benefit is not the
effects of the state the guest finds itself in --- since S3 is not
enormously meaningful in a virtualised environment --- but because of the
side effects we enjoy when the guest awakens and resyncs its world.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> The symptom is that: if use stdvga in config file, the windows guest
> will not show "standby" in shutdown menu, which means S3 is disabled.
> The following link explains why.
> http://support.microsoft.com/?scid=kb%3Ben-us%3B257562&x=16&y=12
> 
> When use cirrus_vga, the "standby" appears in shutdown menu. However,
> after "standby" is pressed, windows will pop up a dialog box, saying
> default monitor driver has no power  management support, so windows
> refuse to go further.

Is the stdvga driver what's used if you use qemu-dm 0.9.0's VESA FB
emulation or does windows have a more advanced driver for VESA mode,
hopefully one that supports S3?

Ian

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser  wrote on 2007年5月17日 15:41:
> On 17/5/07 03:32, "Yu, Ke" <ke.yu@intel.com> wrote:
> 
>> My concern here is that: save/restore is a heavy operation just like
>> S4 (hibernate), while the purpose of S3 is quick suspend and quick
>> resume comapred to S4. if we implement S3 like save/restore, I don't
>> see the value here, because HVM save/resotre or HVM S4 is just
>> enough. How do you think? 
> 
> I do not think that pure virtual S3 by itself makes sense. Unless the
> whole machine is going into S3, what really is the difference between
> a HVM guest in S3 and an HVM guest that simply is idle and so has all
> its VCPUs HLTed 
> 99.9% of the time? Both are tying up memory resource, neither is
> burning significant CPU resource or I/O resource.

Agree.Virtual S3 itself don't have practical value.

> 
> Virtual S3 *does* make sense with HVM save/restore because it makes
> even an HVM guest without PV drivers aware of the save/restore event
> so it can do things like resync its clock. That is why we are
> interested in virtual S3: only as an adjunct to save/restore.

I see your point. Virtual S3 can be used  to assist HVM save/resotor. It is fine. 

> 
> Another application would be as part of host S3. Given that those
> patches currently save/restore all domains (which actually I think is
> stupid, but that's for a separate email) you should automatically
> improve host S3 by integrating with save/restore. However, again, the
> benefit is not the effects of the state the guest finds itself in ---
> since S3 is not enormously meaningful in a virtualised environment
> --- but because of the side effects we enjoy when the guest awakens
> and resyncs its world. 
> 
>  -- Keir

Virtual S3 is also an important part of host S3, when HVM domain is assigned with physical device, e.g. with VT-d introduced. When host S3 begins, HVM domain should suspend its physical device,  otherwise the physical device may not work after resume. This is the reason why we develop the vritaul S3 feature: help VT-d HVM domain to do power management. In this case, integrating save/resotre with virtual S3 is not necessary,  and it actually brings extra overhead. 

To summary, virtual S3 can have two functionality
- assist HVM to do save/restore
- assist HVM to do vt-d device power management
The first item need to call save/restore while the second item do not.

So is it possible that we have two code path, and use flags to tell virtual S3 which path it should go?

BTW, I did not see current host S3 patches save/restore all domains. Could you please elabroate more?

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Zhang, Jingke]

Hi Yuke,
> BTW, I did not see current host S3 patches save/restore all domains. Could you please elabroate more?
    I have tested Kevin's patches about Dom0-S3. I found the host can resume the HVM after its waking. That is to say host sleep-wake course does not impact the HVM guest on it.
    My test steps are:
    1. Create HVM guest
    2. echo "mem" > /sys/power/state on host
    3. Wake up the host by Wake-On-Lan
    4. Checking "xm lis" and "xm console <Domain_id>" is OK.




-----Original Message-----
From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Yu, Ke
Sent: 2007年5月17日 20:40
To: Keir Fraser; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] [PATCH 0/4] HVM Virtual S3

Keir Fraser  wrote on 2007年5月17日 15:41:
> On 17/5/07 03:32, "Yu, Ke" <ke.yu@intel.com> wrote:
> 
>> My concern here is that: save/restore is a heavy operation just like
>> S4 (hibernate), while the purpose of S3 is quick suspend and quick
>> resume comapred to S4. if we implement S3 like save/restore, I don't
>> see the value here, because HVM save/resotre or HVM S4 is just
>> enough. How do you think? 
> 
> I do not think that pure virtual S3 by itself makes sense. Unless the
> whole machine is going into S3, what really is the difference between
> a HVM guest in S3 and an HVM guest that simply is idle and so has all
> its VCPUs HLTed 
> 99.9% of the time? Both are tying up memory resource, neither is
> burning significant CPU resource or I/O resource.

Agree.Virtual S3 itself don't have practical value.

> 
> Virtual S3 *does* make sense with HVM save/restore because it makes
> even an HVM guest without PV drivers aware of the save/restore event
> so it can do things like resync its clock. That is why we are
> interested in virtual S3: only as an adjunct to save/restore.

I see your point. Virtual S3 can be used  to assist HVM save/resotor. It is fine. 

> 
> Another application would be as part of host S3. Given that those
> patches currently save/restore all domains (which actually I think is
> stupid, but that's for a separate email) you should automatically
> improve host S3 by integrating with save/restore. However, again, the
> benefit is not the effects of the state the guest finds itself in ---
> since S3 is not enormously meaningful in a virtualised environment
> --- but because of the side effects we enjoy when the guest awakens
> and resyncs its world. 
> 
>  -- Keir

Virtual S3 is also an important part of host S3, when HVM domain is assigned with physical device, e.g. with VT-d introduced. When host S3 begins, HVM domain should suspend its physical device,  otherwise the physical device may not work after resume. This is the reason why we develop the vritaul S3 feature: help VT-d HVM domain to do power management. In this case, integrating save/resotre with virtual S3 is not necessary,  and it actually brings extra overhead. 

To summary, virtual S3 can have two functionality
- assist HVM to do save/restore
- assist HVM to do vt-d device power management
The first item need to call save/restore while the second item do not.

So is it possible that we have two code path, and use flags to tell virtual S3 which path it should go?

BTW, I did not see current host S3 patches save/restore all domains. Could you please elabroate more?

Best Regards
Ke

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 17/5/07 13:39, "Yu, Ke" <ke.yu@intel.com> wrote:

> Virtual S3 is also an important part of host S3, when HVM domain is assigned
> with physical device, e.g. with VT-d introduced. When host S3 begins, HVM
> domain should suspend its physical device,  otherwise the physical device may
> not work after resume. This is the reason why we develop the vritaul S3
> feature: help VT-d HVM domain to do power management. In this case,
> integrating save/resotre with virtual S3 is not necessary,  and it actually
> brings extra overhead.
> 
> To summary, virtual S3 can have two functionality
> - assist HVM to do save/restore
> - assist HVM to do vt-d device power management
> The first item need to call save/restore while the second item do not.

Yes I absolutely agree with all the above. I don't think we want full
save/restore for host S3 at all. We just want to give the domains a little
kick so that they at least resync their clocks, maybe regain DHCP leases,
etc, when the system comes back up.

> So is it possible that we have two code path, and use flags to tell virtual S3
> which path it should go?

I think we'll end up with something like this in the tools, yes. But let's
get the save/restore one plumbed in first. The second case needs integration
with VT-d patches and host S3 patches, which aren't yet in the tree.

> BTW, I did not see current host S3 patches save/restore all domains. Could you
> please elabroate more?

They were posted by your Intel colleague Kevin Tian just a few days ago.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser  wrote on 2007年5月17日 21:07:
> 
> I think we'll end up with something like this in the tools, yes. But
> let's get the save/restore one plumbed in first. The second case
> needs integration with VT-d patches and host S3 patches, which aren't
> yet in the tree. 

Fine to me.

> 
>> BTW, I did not see current host S3 patches save/restore all domains.
>> Could you please elabroate more?
> 
> They were posted by your Intel colleague Kevin Tian just a few days
> ago. 

Yes, I see the patches. But as far as I can see, these patches does not save/restore all domains. It only reuse some driver domain suspend/resume logic to suspend/resume the physical devices that driver domain owns. 

- Ke

> 
>  -- Keir

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 17/5/07 14:16, "Yu, Ke" <ke.yu@intel.com> wrote:

>> They were posted by your Intel colleague Kevin Tian just a few days
>> ago. 
> 
> Yes, I see the patches. But as far as I can see, these patches does not
> save/restore all domains. It only reuse some driver domain suspend/resume
> logic to suspend/resume the physical devices that driver domain owns.

Ah, okay, it looks like I misunderstood.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Ian Pratt  wrote on 17:54:
> 
> Is the stdvga driver what's used if you use qemu-dm 0.9.0's VESA FB
> emulation or does windows have a more advanced driver for VESA mode,
> hopefully one that supports S3?
> 
> Ian

In VGA card side, the VESA FB emulation has not been tried yet, but the
cirrus_vga works fine with Windows. Windows has cirrus VGA driver, which
supports S3, So this issue is resolved.

Current issue is that Windows is not happy with the standard monitor
driver. And unfortunately, seems there is no monitor emulation code in
QEMU, so there may be no way to change the QEMU monitor type. It would
be better if someone can tell how to change that.

Best Regards
Ke 

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> In VGA card side, the VESA FB emulation has not been tried yet, but
the
> cirrus_vga works fine with Windows. Windows has cirrus VGA driver,
which
> supports S3, So this issue is resolved.
> 
> Current issue is that Windows is not happy with the standard monitor
> driver. And unfortunately, seems there is no monitor emulation code in
> QEMU, so there may be no way to change the QEMU monitor type. It would
> be better if someone can tell how to change that.

But presumably the monitor driver changes if you use VESA FB as the DDC
information is available via VESA calls? Hopefully this supports S3.


Ian

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Ian Pratt  wrote on 2007年5月17日 22:15:
>> In VGA card side, the VESA FB emulation has not been tried yet, but
>> the cirrus_vga works fine with Windows. Windows has cirrus VGA
>> driver, which supports S3, So this issue is resolved.
>> 
>> Current issue is that Windows is not happy with the standard monitor
>> driver. And unfortunately, seems there is no monitor emulation code
>> in QEMU, so there may be no way to change the QEMU monitor type. It
>> would be better if someone can tell how to change that.
> 
> But presumably the monitor driver changes if you use VESA FB as the
> DDC information is available via VESA calls? Hopefully this supports
> S3. 
> 
> 
> Ian

Oh, I will try that. Thanks for the info.

- Ke

RE: [PATCH 0/4] HVM Virtual S3

[Kaushik Barde]

I don't believe so.. We intend to use this support (with some additional
ACPI enhancements) as elaborated by Ke.


-Kaushik 

-----Original Message-----
From: xen-devel-bounces@lists.xensource.com
[mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of Keir Fraser
Sent: Wednesday, May 16, 2007 3:30 PM
To: Yu, Ke; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] [PATCH 0/4] HVM Virtual S3

On 16/5/07 17:48, "Yu, Ke" <ke.yu@intel.com> wrote:

> - apply this patch to changeset 15050:05c128b0188a
> - create and boot HVM domain
> - In HVM guest, enter S3 state
>   * for Linux, "echo mem >/sys/power/state"
>   * for Windows, shutdown windows by Standby
> - to resume HVM domain, "xm resume <domid>"

Noone's seriously going to want to use S3-S5 in this way though, are
they.
Keeping domains hanging around in sleep states taking up memory doesn't
make
sense -- users will want to save the domain off to disc and restore
later.
So this only makes sense at all if integrated with existing HVM
save/restore.

As I understand it, the only difference between S3 and S5 is that RAM
contents are maintained when the machine awakens. In this case S3 is
like a
simpler version of current save/restore where we do not necessarily need
to
save device state (but we might decide to if it makes the implementation
easier).

I think actually this isn't far off from being what we want. If the
s3_suspend hypercall put the domain into suspend state, this would then
neatly trigger xc_save to do its stuff. Then you can strip out s3_resume
because you should automatically start executing BIOS code when
xc_resume
restores VCPU state and kicks the domain to start running again. The
sleep_state domain variable you added shouldn't be necessary imo.

 -- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Kaushik Barde  wrote on 4:25:
> I don't believe so.. We intend to use this support (with some
> additional ACPI enhancements) as elaborated by Ke.
> 
> 
> -Kaushik
> 

Nice to know virtual S3 has other usage model. Do you want to use that
for S3 validation?

- Ke

RE: [PATCH 0/4] HVM Virtual S3

[Tian, Kevin]

>From: Keir Fraser
>Sent: 2007年5月17日 21:34
>
>On 17/5/07 14:16, "Yu, Ke" <ke.yu@intel.com> wrote:
>
>>> They were posted by your Intel colleague Kevin Tian just a few days
>>> ago.
>>
>> Yes, I see the patches. But as far as I can see, these patches does not
>> save/restore all domains. It only reuse some driver domain
>suspend/resume
>> logic to suspend/resume the physical devices that driver domain owns.
>
>Ah, okay, it looks like I misunderstood.
>
> -- Keir
>

Great that people are looking at host S3 side. :-)

The host S3 patch only tackles dom0/xen, and leave user level to notify 
all para-domUs and HVM domains before requesting real S3 switch. So 
basically to make a complete eco-system:
	- Need to trigger virtual S3 to HVM domain
	- Need light weight suspend/resume to para-domU (without 
domain destroy) for fast resume. Actually I sent out months ago and 
you suggested to re-send after basic host S3 is ready

Currently host S3 only imposes a last level means to force all domains 
paused in case they're not doing well.

Thanks,
Kevin

RE: [PATCH 0/4] HVM Virtual S3

[Tian, Kevin]

>From: Keir Fraser
>Sent: 2007年5月17日 6:34
>
>Oh, I see there is device state to be reset in QEMU. Emulating the port
>access in QEMU makes sense then, but I wonder if rather than adding
>an extra
>hypercall command we could emulate it in both Xen and QEMU: Xen
>emulates the
>instruction, resets state and triggers domain shutdown, then passes the
>port
>access up to QEMU so it does its thing also. We do this emulate-in-both
>for
>other things already (e.g., the CMOS index register I believe).
>
> -- Keir
>

I doubt whether this will bring some tricky issues if Qemu happens 
to inject some event after Xen's reset but before Qemu's reset. If 
that event results a stale pending interrupt after Xen resets vPIC or 
vIOAPIC, will that result a spurious interrupt after HVM is resumed? 
By current style, Qemu first reset all virtual devices and thus stop 
interrupt sources. Then it's safe to let Xen do a full reset subsequently.

Also normally the reason to put Qemu logic into Xen is for performance. For virtual S3 interception, it doesn't matter since only resume time is 
the concern.

Just doubt whether it's worthy of such split anyway. :-)

Of course above concern is only for one path per your on-going 
discussion. For the other path on top of save/restore, it's always 
OK since domain is re-created.

Thanks,
Kevin

RE: [PATCH 0/4] HVM Virtual S3

[Kaushik Barde]

Absolutely.. Is this (Virtual S3) patch independent of  Xen i.e. can we
use it with 3.0.4?

Thanks.

-Kaushik

-----Original Message-----
From: Yu, Ke [mailto:ke.yu@intel.com] 
Sent: Thursday, May 17, 2007 7:06 PM
To: Kaushik Barde; Keir Fraser; xen-devel@lists.xensource.com
Subject: RE: [Xen-devel] [PATCH 0/4] HVM Virtual S3

Kaushik Barde  wrote on 4:25:
> I don't believe so.. We intend to use this support (with some
> additional ACPI enhancements) as elaborated by Ke.
> 
> 
> -Kaushik
> 

Nice to know virtual S3 has other usage model. Do you want to use that
for S3 validation?

- Ke

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Kaushik Barde  wrote on 12:22:
> Absolutely.. Is this (Virtual S3) patch independent of  Xen i.e. can
> we use it with 3.0.4?
> 
> Thanks.
> 
> -Kaushik
> 

It should be able to work in Xen 3.0.4. You need rebase and the rebase
effort is small. 

- Ke

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 18/5/07 03:38, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> I doubt whether this will bring some tricky issues if Qemu happens
> to inject some event after Xen's reset but before Qemu's reset. If
> that event results a stale pending interrupt after Xen resets vPIC or
> vIOAPIC, will that result a spurious interrupt after HVM is resumed?
> By current style, Qemu first reset all virtual devices and thus stop
> interrupt sources. Then it's safe to let Xen do a full reset subsequently.
> 
> Also normally the reason to put Qemu logic into Xen is for performance. For
> virtual S3 interception, it doesn't matter since only resume time is
> the concern.
> 
> Just doubt whether it's worthy of such split anyway. :-)

Okay, let's rename this s3_suspend hypercall to system_reset instead, and
leave it that.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser  wrote on 2007年5月17日 21:07:
> 
> I think we'll end up with something like this in the tools, yes. But
> let's get the save/restore one plumbed in first. The second case
> needs integration with VT-d patches and host S3 patches, which aren't
> yet in the tree. 
> 
>  -- Keir

Hi Keir,

I just discuss with Edwin and Kevin on the HVM save/restore integration with virtual S3. Basically, the following things need to be done:

1. For HVM without PV, QEMU call xc_save, qemu_savevm,... at the virtual S3 final stage. 
2. For HVM with PV, need also add PV driver suspend/resume interface to disconnect/reconnect xenbus connection. 

Since I am not familiar with the HVM save/restore, how about the following working model:
- I implement the basic Virtual S3 code and leave interface for save/restore
- You or the PV maintainer add the above save/restore logic upon the  basic virtual S3 code.

Best Regards
Ke

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 21/5/07 04:10, "Yu, Ke" <ke.yu@intel.com> wrote:

> 1. For HVM without PV, QEMU call xc_save, qemu_savevm,... at the virtual S3
> final stage.

In this case S3 should somehow be an option to 'xm save', so an alternative
to doing an immediate remote shutdown on the HVM guest. Instead we would
inject a S3 event and when the guest enters S3 qemu would do the remote
shutdown at that point. This has to be an option to 'xm save', or perhaps a
domain config option, because not all guests will respond to PM events to
enter S3 state.

> 2. For HVM with PV, need also add PV driver suspend/resume interface to
> disconnect/reconnect xenbus connection.

Possibly we can leave the S3 kickoff to the PV drivers. So xend notifies PV
drivers to suspend, they do their thing, kick S3 code in a guest-OS-specific
way, guest enters S3, qemu does remote shutdown. Then on wakeup the S3
restore code runs before PV-driver restore.

 -- Keir

> - I implement the basic Virtual S3 code and leave interface for save/restore
> - You or the PV maintainer add the above save/restore logic upon the  basic
> virtual S3 code.

Re: [PATCH 0/4] HVM Virtual S3

[Tim Deegan]

[-- Attachment #1: Type: text/plain, Size: 1736 bytes --]

At 22:17 +0800 on 17 May (1179440232), Yu, Ke wrote:
> Ian Pratt  wrote on 2007??5??17?? 22:15:
> >> In VGA card side, the VESA FB emulation has not been tried yet, but
> >> the cirrus_vga works fine with Windows. Windows has cirrus VGA
> >> driver, which supports S3, So this issue is resolved.
> >> 
> >> Current issue is that Windows is not happy with the standard monitor
> >> driver. And unfortunately, seems there is no monitor emulation code
> >> in QEMU, so there may be no way to change the QEMU monitor type. It
> >> would be better if someone can tell how to change that.

Sorry for joining this thread quite late - I've been away for a few
weeks.

I found that the attached patch to the Cirrus VGA BIOS helps Windows
along (but not quite enough).  It adds some VBE power-management calls
to the BIOS, telling Windows that it's capable of putting the screen to
sleep.  Unfortunately, this just changes the error from the monitor
driver refusing to allow S3 to the Cirrus driver refusing to allow
it. :|

Tracing the BIOS and I/O activity of Windows, the next most likely
culprit is when the Cirrus driver tries to talk DDC2B to the monitor at
start of day (by frobbing the line state manually via Cirrus control
registers) and gets no response.  Since it can't read the monitor's
descriptor tables it can't be sure that it's possible to put it to sleep
properly.

Since implementing a DDC2B state machine and faking out the tables
sounds nasty, I agree with Ian that the VESA/VBE device model is
probably the way forward, in the hope of getting Windows to use BIOS
calls for all this.

Cheers,

Tim.

-- 
Tim Deegan <Tim.Deegan@xensource.com>, XenSource UK Limited
Registered office c/o EC2Y 5EB, UK; company number 05334508

[-- Attachment #2: vbe-pm --]
[-- Type: text/plain, Size: 2726 bytes --]

diff -r dcf5e2a0d370 tools/firmware/vgabios/clext.c
--- a/tools/firmware/vgabios/clext.c	Thu Apr 26 11:48:41 2007 +0100
+++ b/tools/firmware/vgabios/clext.c	Thu Apr 26 11:49:02 2007 +0100
@@ -26,6 +26,7 @@
 #define PM_BIOSMEM_CURRENT_MODE 0x449
 #define PM_BIOSMEM_CRTC_ADDRESS 0x463
 #define PM_BIOSMEM_VBE_MODE 0x4BA
+#define PM_BIOSMEM_VBE_POWER 0x4BC 
 
 typedef struct
 {
@@ -491,7 +492,7 @@ cirrus_vesa:
 #ifdef CIRRUS_DEBUG
   call cirrus_debug_dump
 #endif
-  cmp al, #0x0F
+  cmp al, #0x10
   ja cirrus_vesa_not_handled
   push bx
   xor bx, bx
@@ -682,7 +683,7 @@ c80h_2:
   ret
 
 cirrus_extbios_81h:
-  mov ax, #0x100 ;; XXX
+  mov ax, #0x103 ;; XXX
   ret
 cirrus_extbios_82h:
   push dx
@@ -1175,6 +1176,52 @@ cirrus_vesa_07h_2:
   mov  cx, ax
   pop  dx
   mov  ax, #0x004f
+  ret
+
+cirrus_vesa_10h: ;; Power management functions
+  ;; Set up DS to read stored power info from RAM
+  push ds
+#ifdef CIRRUS_VESA3_PMINFO
+ db 0x2e ;; cs:
+  mov ax, [cirrus_vesa_sel0000_data]
+#else
+  xor ax, ax
+#endif
+  mov  ds, ax
+  ;; Now choose the right function
+  cmp  bl, #0x00
+  ja   cirrus_vesa_10h_01
+  ;;
+  ;; Function 00h: Get capabilities
+  ;;
+  mov  bx, #0x0720 ;; 07: standby/suspend/off, 20: VBE/PM 2.0
+  mov  ax, #0x004f
+  jmp cirrus_vesa_10h_done
+cirrus_vesa_10h_01:
+  cmp  bl, #0x01
+  ja   cirrus_vesa_10h_02
+  ;;
+  ;; Function 01h: Set power state
+  ;; 
+  mov  ax, bx
+  mov  bx, # PM_BIOSMEM_VBE_POWER
+  mov  [bx], ah
+  mov  ax, #0x004f 
+  jmp cirrus_vesa_10h_done
+cirrus_vesa_10h_02:
+  cmp  bl, #0x02
+  ja   cirrus_vesa_10h_unimplemented
+  ;;
+  ;; Function 02h: Get power state
+  ;; 
+  mov  bx, # PM_BIOSMEM_VBE_POWER
+  mov  bh, [bx]
+  mov  ax, #0x004f 
+  jmp cirrus_vesa_10h_done
+cirrus_vesa_10h_unimplemented:
+  mov  ax, #0x014F ;; not implemented
+cirrus_vesa_10h_done:
+  pop ds
   ret
 
 cirrus_vesa_unimplemented:
@@ -1573,9 +1620,9 @@ cirrus_vesa_handlers:
   dw cirrus_vesa_00h
   dw cirrus_vesa_01h
   dw cirrus_vesa_02h
-  dw cirrus_vesa_03h
+  dw cirrus_vesa_unimplemented
   ;; 04h
-  dw cirrus_vesa_unimplemented
+  dw cirrus_vesa_04h
   dw cirrus_vesa_05h
   dw cirrus_vesa_06h
   dw cirrus_vesa_07h
@@ -1589,7 +1636,8 @@ cirrus_vesa_handlers:
   dw cirrus_vesa_unimplemented
   dw cirrus_vesa_unimplemented
   dw cirrus_vesa_unimplemented
-
+  ;; 10h
+  dw cirrus_vesa_10h
 
 
 ASM_END
diff -r dcf5e2a0d370 tools/firmware/vgabios/vgatables.h
--- a/tools/firmware/vgabios/vgatables.h	Thu Apr 26 11:48:41 2007 +0100
+++ b/tools/firmware/vgabios/vgatables.h	Thu Apr 26 11:48:42 2007 +0100
@@ -25,6 +25,7 @@
 #define BIOSMEM_VS_POINTER    0xA8
 #define BIOSMEM_VBE_FLAG      0xB9
 #define BIOSMEM_VBE_MODE      0xBA
+#define BIOSMEM_VBE_POWER     0xBC
 
 
 /*

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

[PATCH 0/4] HVM Virtual S3

[Ke, Liping]

[PATCH 0/4] HVM Virtual S3

These set of patches are our prototype for HVM virtual 
ACPI S3 support:
 - patch 1/4: Xen interface for HVM S3
 - patch 2/4: QEMU interface for HVM S3
 - patch 3/4: rombios interface for HVM S3
 - patch 4/4: xend interface for HVM S3

The main idea is:
- emulate ACPI PM1A control resiger in QEMU to capture guest S3 request
- when QEMU capture guest S3 request, it call hypercall to trap to Xen
- Xen will suspend HVM by reseting all vcpus
- on resume, "xm resume" will call hypercall to trap to Xen
- Xen will resume HVM by setting HVM vcpu EIP to 0xfff0, cs base to 0xf0000, 
  and also set other related registers/msr to the correct value/attributes 
  in realmode environment which will start from rombios post Entry code 
  in realmode directly when resuming.
- rombios post code will start s3 resume by jumping to wakeup vector set
  by guest OS.

How to use it:
- apply this patch to changeset 17589:9a6ad687ec20
- create and boot HVM domain 
- In HVM guest, enter S3 state
  * for Linux, "echo mem >/sys/power/state"
  * for Windows, shutdown windows by Standby
- to resume HVM domain, "xm resume <domid>"

Note: FC6/FC5 guest can successfully S3 suspend/resume in both text mode and
X window. Also, with vtd-enabled, network card assigned to HVM, it works
too. However, Windows guest can not suspend, because the windows
driver for QEMU VGA and monitor has no power management support, and
windows refuse to enter S3 sleep state. 

Kevin/Ke/Liping

RE: [PATCH 0/4] HVM Virtual S3

[James Harper]

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of Ke, Liping
> Sent: Thursday, 8 May 2008 15:13
> To: xen-devel@lists.xensource.com
> Cc: Tian, Kevin; Jiang,Yunhong; Yu, Ke; Ke, Liping
> Subject: [Xen-devel] [PATCH 0/4] HVM Virtual S3
> 
> [PATCH 0/4] HVM Virtual S3
> 
> These set of patches are our prototype for HVM virtual
> ACPI S3 support:
>  - patch 1/4: Xen interface for HVM S3
>  - patch 2/4: QEMU interface for HVM S3
>  - patch 3/4: rombios interface for HVM S3
>  - patch 4/4: xend interface for HVM S3
> 

How does this interact with the PV drivers? Linux, but also how should I
interact with it in the GPL PV drivers for Windows?

> However, Windows guest can not suspend, because the windows
> driver for QEMU VGA and monitor has no power management support, and
> windows refuse to enter S3 sleep state.

I have successfully suspended Windows before using the GPL PV drivers,
but I haven't gotten resume working yet...

James

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

James Harper wrote:
>> 
>> [PATCH 0/4] HVM Virtual S3
>> 
>> These set of patches are our prototype for HVM virtual
>> ACPI S3 support:
>>  - patch 1/4: Xen interface for HVM S3
>>  - patch 2/4: QEMU interface for HVM S3
>>  - patch 3/4: rombios interface for HVM S3
>>  - patch 4/4: xend interface for HVM S3
>> 
> 
> How does this interact with the PV drivers? Linux, but also how
> should I interact with it in the GPL PV drivers for Windows?

By design, PV drivers should have suspend/resume routine, and it will be
called by guest OS S3 sleep/resume logic. We have test the linux pv
driver, it works fine. Due to current windows virtual S3 resume issue,
we have no chance to test your windows PV driver. But by desing, as long
as PV drvier has right suspeend/resume routine, it should work just
fine.

> 
>> However, Windows guest can not suspend, because the windows
>> driver for QEMU VGA and monitor has no power management support, and
>> windows refuse to enter S3 sleep state.
> 
> I have successfully suspended Windows before using the GPL PV drivers,
> but I haven't gotten resume working yet...
> 
> James

Can you elaborate how you manage to suspend guest windows? We are very
interested in this. In our environment, windows XP refuse to suspend due
to QEMU VGA

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

We plan to send the patches today, other four patches will come out soon.
And please see below answers.

Thanks& Regards
Criping

James Harper wrote:
>> -----Original Message-----
>> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
>> bounces@lists.xensource.com] On Behalf Of Ke, Liping
>> Sent: Thursday, 8 May 2008 15:13
>> To: xen-devel@lists.xensource.com
>> Cc: Tian, Kevin; Jiang,Yunhong; Yu, Ke; Ke, Liping
>> Subject: [Xen-devel] [PATCH 0/4] HVM Virtual S3
>> 
>> [PATCH 0/4] HVM Virtual S3
>> 
>> These set of patches are our prototype for HVM virtual
>> ACPI S3 support:
>>  - patch 1/4: Xen interface for HVM S3
>>  - patch 2/4: QEMU interface for HVM S3
>>  - patch 3/4: rombios interface for HVM S3
>>  - patch 4/4: xend interface for HVM S3
>> 
> 
> How does this interact with the PV drivers? Linux, but also how
> should I interact with it in the GPL PV drivers for Windows?

I have tested the patch with PV driver Linux HVM, FC5-32e, 
it could resume back sucessfully. 
Both in Txt mode and X windows. Since after S3, only cpu reset,
all pv drivers or (device drivers such as 
network with vtd-enabled) information still in memory, 
it should work fine after resume. 
But I did not test windows yet because of below problems.

> 
>> However, Windows guest can not suspend, because the windows
>> driver for QEMU VGA and monitor has no power management support, and
>> windows refuse to enter S3 sleep state.
> 
> I have successfully suspended Windows before using the GPL PV drivers,
> but I haven't gotten resume working yet...
And could you help to tell us how to let windows hvm successfully sleep down?

We used to successfully do this on va project, but never succeed on upstream 
because of qemu VGA drivers seems does not support pnp S3 request. 
I have tried both std-vga and cirrus vga on xp/server2003/vista hvm, yet all failed
.  Std-vga, even s3 button is disabed.
. Cirrus-vga, s3 is enabled, when you click, it will say "default monitor prevent S3 operation...".
And thanks for the information.

> 
> James

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> The main idea is:
> - emulate ACPI PM1A control resiger in QEMU to capture guest S3
request
> - when QEMU capture guest S3 request, it call hypercall to trap to Xen
> - Xen will suspend HVM by reseting all vcpus
> - on resume, "xm resume" will call hypercall to trap to Xen
> - Xen will resume HVM by setting HVM vcpu EIP to 0xfff0, cs base to
> 0xf0000,
>   and also set other related registers/msr to the correct
> value/attributes
>   in realmode environment which will start from rombios post Entry
code
>   in realmode directly when resuming.
> - rombios post code will start s3 resume by jumping to wakeup vector
> set
>   by guest OS.

Have you checked save/restore works when the guest is in S3 state? This
is essential for some of the server (as opposed to client) use cases for
guest S3.

[One thing to watch out for is that if the guest is in S3 it should be
possible to skip the usual synchronization with PV drivers as they will
already be shutdown.]


Further, do we actually need the s3 resume hypercall? Couldn't we just
put the VCPUs and xen-emulated devices into the right state for resume
at the end of s3 sleep and just leave the domain paused (i.e.
unscheduled)?


Also, do you have plans to implement an "xm send-power-event" type thing
that enables us to inform the guest of actions such as lid switch
open/close, power cable in/out, power button pressed etc ?

Many Thanks,
Ian

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

Ian Pratt wrote:
>> The main idea is:
>> - emulate ACPI PM1A control resiger in QEMU to capture guest S3
>> request 
>> - when QEMU capture guest S3 request, it call hypercall to trap to
>> Xen 
>> - Xen will suspend HVM by reseting all vcpus
>> - on resume, "xm resume" will call hypercall to trap to Xen
>> - Xen will resume HVM by setting HVM vcpu EIP to 0xfff0, cs base to
>>   0xf0000, and also set other related registers/msr to the correct
>>   value/attributes in realmode environment which will start from
>>   rombios post Entry code in realmode directly when resuming.
>> - rombios post code will start s3 resume by jumping to wakeup vector
>>   set by guest OS.
> 
> Have you checked save/restore works when the guest is in S3 state?
> This is essential for some of the server (as opposed to client) use
> cases for guest S3.
Yes, save/restore should work when guest is in S3 state. When S3_sleep,
domain is still alive, just only that VCPU is blocked. We'll do further test
on it these days.
> 
> [One thing to watch out for is that if the guest is in S3 it should be
> possible to skip the usual synchronization with PV drivers as they
> will already be shutdown.]
Yes, when guest is in S3, PV drivers should not deal with any synchronization
Jobs. When resume back, it returns back to work again.
> 
> 
> Further, do we actually need the s3 resume hypercall? Couldn't we just
> put the VCPUs and xen-emulated devices into the right state for resume
> at the end of s3 sleep and just leave the domain paused (i.e.
> unscheduled)?
As I can understand, when S3, HVM will perform like native-machine S3, 
all execution context are in memory When resume back, jump to the CS/EIP,
those vcpus resume execution normally.
Ke& Kevin, any coments here?

> 
> 
> Also, do you have plans to implement an "xm send-power-event" type
> thing that enables us to inform the guest of actions such as lid
> switch open/close, power cable in/out, power button pressed etc ?
Since now host S3 and HVM S3 is not integrated yet, I am not sure
whether those lid/cable/power button are appropriate to be involved with
Guest notification. Especially if we have multiple guest. Yet I think we must
consider it carefully as part of VMM PM solution :)
If both host and HVM is S3 done, when lid open,
Xen is back, host is back and then HVM is back. It is a possible senario?

> 
> Many Thanks,
> Ian

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 8/5/08 15:57, "Ke, Liping" <liping.ke@intel.com> wrote:

>> Further, do we actually need the s3 resume hypercall? Couldn't we just
>> put the VCPUs and xen-emulated devices into the right state for resume
>> at the end of s3 sleep and just leave the domain paused (i.e.
>> unscheduled)?
> As I can understand, when S3, HVM will perform like native-machine S3,
> all execution context are in memory When resume back, jump to the CS/EIP,
> those vcpus resume execution normally.
> Ke& Kevin, any coments here?

I think part of what Ian means is that you may as well have a single
interface into each device model in Xen -- e.g., hpet_reset(), pit_reset(),
and so on. Rather than have separate poweroff/poweron functions. Apart from
the greater simplicity, some models already have a reset interface, and I'm
not sure the implemented poweroff/poweron splits are all correct.

And then, once the poweron calls are all gone, what else does s3_resume
actually have to do?

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Ke, Liping wrote:
> Ian Pratt wrote:
>>> The main idea is:
>>> - emulate ACPI PM1A control resiger in QEMU to capture guest S3
>>> request 
>>> - when QEMU capture guest S3 request, it call hypercall to trap to
>>> Xen 
>>> - Xen will suspend HVM by reseting all vcpus
>>> - on resume, "xm resume" will call hypercall to trap to Xen
>>> - Xen will resume HVM by setting HVM vcpu EIP to 0xfff0, cs base to
>>>   0xf0000, and also set other related registers/msr to the correct
>>>   value/attributes in realmode environment which will start from
>>>   rombios post Entry code in realmode directly when resuming.
>>> - rombios post code will start s3 resume by jumping to wakeup vector
>>>   set by guest OS.
>> 
>> Have you checked save/restore works when the guest is in S3 state?
>> This is essential for some of the server (as opposed to client) use
>> cases for guest S3.
> Yes, save/restore should work when guest is in S3 state. When
> S3_sleep, domain is still alive, just only that VCPU is blocked.
> We'll do further test on it these days.

We did not try yet. Since our original purose is for PCI passthrouhg
device. And surely we will test it. Thanks for reminder.

>> 
>> [One thing to watch out for is that if the guest is in S3 it should
>> be possible to skip the usual synchronization with PV drivers as they
>> will already be shutdown.]
> Yes, when guest is in S3, PV drivers should not deal with any
> synchronization Jobs. When resume back, it returns back to work again.

Oh, right, we will look more on this. Basic test shows vritual S3 works
fine with Linux PV drivers.

>> 
>> 
>> Further, do we actually need the s3 resume hypercall? Couldn't we
>> just put the VCPUs and xen-emulated devices into the right state for
>> resume at the end of s3 sleep and just leave the domain paused (i.e.
>> unscheduled)?
> As I can understand, when S3, HVM will perform like native-machine S3,
> all execution context are in memory When resume back, jump to the
> CS/EIP, those vcpus resume execution normally.
> Ke& Kevin, any coments here?

That is good idea. Basically we will combine suspend/resume into one
logic. And use "xm unpause" for resume. We will look into more.

> 
>> 
>> 
>> Also, do you have plans to implement an "xm send-power-event" type
>> thing that enables us to inform the guest of actions such as lid
>> switch open/close, power cable in/out, power button pressed etc ?
> Since now host S3 and HVM S3 is not integrated yet, I am not sure
> whether those lid/cable/power button are appropriate to be involved
> with Guest notification. Especially if we have multiple guest. Yet I
> think we must consider it carefully as part of VMM PM solution :)
> If both host and HVM is S3 done, when lid open,
> Xen is back, host is back and then HVM is back. It is a possible
> senario? 

We are also thinking about this. This is basically qemu work, and we see
qemu upstream already has some logic like GPE event. We will analysis
the gap and propose task plan for next step. 

> 
>> 
>> Many Thanks,
>> Ian

Thanks for the valuable comments :)

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> > Also, do you have plans to implement an "xm send-power-event" type
> > thing that enables us to inform the guest of actions such as lid
> > switch open/close, power cable in/out, power button pressed etc ?
> Since now host S3 and HVM S3 is not integrated yet, I am not sure
> whether those lid/cable/power button are appropriate to be involved
> with Guest notification. Especially if we have multiple guest. Yet I
think
> we must consider it carefully as part of VMM PM solution :)
> If both host and HVM is S3 done, when lid open,
> Xen is back, host is back and then HVM is back. It is a possible
> senario?

I don't think we need any integration with host S3 as part of the xend
-- that's a separate issue, maybe an example script that runs (early) as
part of a host S3 that puts the guests into S3 too.

I think what we need in xm/xend is a command for informing the guest of
ACPI events such as lid switch, power cable, power button etc.

Ian

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 8/5/08 16:11, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

> I don't think we need any integration with host S3 as part of the xend
> -- that's a separate issue, maybe an example script that runs (early) as
> part of a host S3 that puts the guests into S3 too.
> 
> I think what we need in xm/xend is a command for informing the guest of
> ACPI events such as lid switch, power cable, power button etc.

We can actually extend the 'xm trigger' command for this with some more
events. 'xm trigger {lid-closed,lid-open}'?

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Yu, Ke]

Keir Fraser wrote:
> On 8/5/08 16:11, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:
> 
>> I don't think we need any integration with host S3 as part of the
>> xend -- that's a separate issue, maybe an example script that runs
>> (early) as part of a host S3 that puts the guests into S3 too.
>> 
>> I think what we need in xm/xend is a command for informing the guest
>> of ACPI events such as lid switch, power cable, power button etc.
> 
> We can actually extend the 'xm trigger' command for this with some
> more events. 'xm trigger {lid-closed,lid-open}'?
> 
>  -- Keir
> 
> 

Yes. And another thing we may need think about is the inter-process
communication with qemu, to notify qemu to trigger the ACPI event.

Best Regards
Ke

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

>>> 
>>> Further, do we actually need the s3 resume hypercall? Couldn't we
>>> just put the VCPUs and xen-emulated devices into the right state for
>>> resume at the end of s3 sleep and just leave the domain paused (i.e.
>>> unscheduled)?
>> As I can understand, when S3, HVM will perform like native-machine
>> S3, all execution context are in memory When resume back, jump to the
>> CS/EIP, those vcpus resume execution normally.
>> Ke& Kevin, any coments here?
> 
> That is good idea. Basically we will combine suspend/resume into one
> logic. And use "xm unpause" for resume. We will look into more. 
> 
Hi, Ke and Ian
Today I am trying this idea, I found combining S3
sus/resume+domain_pause, and then
use domain_unpause letting domain back works fine.

Yet when I am trying xm save after pausing domain, I found if a domain
not running, 
Xm save will reject the save request.
So I want to know whether domain running is a must  for xm save
operation?

Thanks& Regards,
Criping

>>> 
>>> Many Thanks,
>>> Ian
> 
> Thanks for the valuable comments :)
> 
> Best Regards
> Ke

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

'xm [un]pause' already has a meaning, and I don't think the meaning should
be overloaded for HVM guests. That's asking for trouble. I'd rather we went
with the 'xm trigger' approach, making it look like a lid opening and
closing.

Apart from that, this approach of doing most of the work on the suspend side
is definitely the right thing to do. Then 'xm trigger lid-open' (or whatever
you call it) will be really not much more complicated than domain_unpause().

 -- Keir

On 9/5/08 10:58, "Ke, Liping" <liping.ke@intel.com> wrote:

>> That is good idea. Basically we will combine suspend/resume into one
>> logic. And use "xm unpause" for resume. We will look into more.
>> 
> Hi, Ke and Ian
> Today I am trying this idea, I found combining S3
> sus/resume+domain_pause, and then
> use domain_unpause letting domain back works fine.
> 
> Yet when I am trying xm save after pausing domain, I found if a domain
> not running, 
> Xm save will reject the save request.
> So I want to know whether domain running is a must  for xm save
> operation?

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> Today I am trying this idea, I found combining S3
> sus/resume+domain_pause, and then
> use domain_unpause letting domain back works fine.
> 
> Yet when I am trying xm save after pausing domain, I found if a domain
> not running,
> Xm save will reject the save request.
> So I want to know whether domain running is a must  for xm save
> operation?

Yes, you'll certainly need a few tweaks to the save/restore code.

Normally save/restore needs to send a notification to the guest such
that the PV drivers get themselves into a 'safe' state. If the guest is
in S3, the drivers will already be shutdown (hence safe).

I think you'll need to add an explicit new state that a VM can be in (S3
sleep), and then gate out some of the save code based on that state.
You'll also need to make sure that the state is saved as part of the
image written to disk to make sure that when it restored it remains in
S3 sleep until triggered back to life. [We may want to add flags to the
xm save/restore commands to incorporate S3 sleep and wake up behaviour
as part of save restore, but this is just CLI sugar.]

Ian

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

Hi, Ian and Keir
Thanks for the detailed explanation.
I have tried below solution according to all feedbacks:
1. merged s3suspend+s3resume+domain_pause phase into one s3_suspend phase 
2. s3 resume phase is now very light. Do something like unpause domain
3. add xm trigger <domid> s3resume interface to for s3_resume operation
4. add new state s3suspended plus (paused, blocked, etc)
All works just fine.

Yet I have met below problems:
1. if hvm in s3suspended, could we allow pause/unpause/suspend/resume operation?
    Need we add condition guard in python code?
2. when we add a new flag is_s3_suspended in dominfo structure, it will affect other
Trace/statistic tools in xen, seems this is not a very small change? 
   The above two changes is ok for you or not?

Thanks a lot for your help!
Regards,
Criping


Ian Pratt wrote:
>> Today I am trying this idea, I found combining S3
>> sus/resume+domain_pause, and then
>> use domain_unpause letting domain back works fine.
>> 
>> Yet when I am trying xm save after pausing domain, I found if a
>> domain not running, Xm save will reject the save request.
>> So I want to know whether domain running is a must  for xm save
>> operation?
> 
> Yes, you'll certainly need a few tweaks to the save/restore code.
> 
> Normally save/restore needs to send a notification to the guest such
> that the PV drivers get themselves into a 'safe' state. If the guest
> is in S3, the drivers will already be shutdown (hence safe).
> 
> I think you'll need to add an explicit new state that a VM can be in
> (S3 sleep), and then gate out some of the save code based on that
> state. You'll also need to make sure that the state is saved as part
> of the image written to disk to make sure that when it restored it
> remains in S3 sleep until triggered back to life. [We may want to add
> flags to the xm save/restore commands to incorporate S3 sleep and
> wake up behaviour as part of save restore, but this is just CLI
> sugar.] 
> 
> Ian

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 14/5/08 10:50, "Ke, Liping" <liping.ke@intel.com> wrote:

> 1. if hvm in s3suspended, could we allow pause/unpause/suspend/resume
> operation?
>     Need we add condition guard in python code?

I think pause/unpause still makes sense. Obviously a paused and s3suspended
guest will not run again until both s3 resumed and unpaused. As for
suspend/resume, that should also just work I think? Isn't the point to be
able to s3-suspend then save then restore then s3-resume?

> 2. when we add a new flag is_s3_suspended in dominfo structure, it will affect
> other
> Trace/statistic tools in xen, seems this is not a very small change?
>    The above two changes is ok for you or not?

I don't really care to have is_s3_suspended visible via dominfo. Leave it
hidden if you like. Xend should be able to remember whether it s3-suspended
a domain, if it needs to. Or we can expose it via an interface other than
dominfo. We could, for example, make it an HVM_PARAM. Then you can
enter/leave/interrogate s3suspend state via the existing hvm_op param
interfaces.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

Keir Fraser wrote:
> On 14/5/08 10:50, "Ke, Liping" <liping.ke@intel.com> wrote:
> 
>> 1. if hvm in s3suspended, could we allow
>>     pause/unpause/suspend/resume operation? Need we add condition
>> guard in python code? 
> 
> I think pause/unpause still makes sense. Obviously a paused and
> s3suspended guest will not run again until both s3 resumed and
> unpaused. As for suspend/resume, that should also just work I think?
> Isn't the point to be able to s3-suspend then save then restore then
> s3-resume? 
> 
Yes, now S3 could works with save/restore. Since suspend/resume has 
Similar mechanism with save/restore, it works. I just thought s3_suspend/resume 
is a kind of suspend/resume already?
As for pause/unpause, since after s3_supend is now already in pause state,
I just thought is it a little interleaving? If you like, I will make it work firstly.

>> 2. when we add a new flag is_s3_suspended in dominfo structure, it
>> will affect other Trace/statistic tools in xen, seems this is not a
>>    very small change? The above two changes is ok for you or not?
> 
> I don't really care to have is_s3_suspended visible via dominfo.
> Leave it hidden if you like. Xend should be able to remember whether
> it s3-suspended a domain, if it needs to. Or we can expose it via an
> interface other than dominfo. We could, for example, make it an
> HVM_PARAM. Then you can enter/leave/interrogate s3suspend state via
> the existing hvm_op param interfaces.
Ok. I will use the first solution and do not revise trace related code.
If we want to expose this state in the furture, first solution is easier to modify.
> 
>  -- Keir

Thanks a lot for your help:)
Regards,
Criping

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 15/5/08 02:31, "Ke, Liping" <liping.ke@intel.com> wrote:

>> I think pause/unpause still makes sense. Obviously a paused and
>> s3suspended guest will not run again until both s3 resumed and
>> unpaused. As for suspend/resume, that should also just work I think?
>> Isn't the point to be able to s3-suspend then save then restore then
>> s3-resume? 
>> 
> Yes, now S3 could works with save/restore. Since suspend/resume has
> Similar mechanism with save/restore, it works. I just thought
> s3_suspend/resume
> is a kind of suspend/resume already?
> As for pause/unpause, since after s3_supend is now already in pause state,
> I just thought is it a little interleaving? If you like, I will make it work
> firstly.

The 'pause' state visible to tools is a specific type of domain_pause(), and
*has* to be entered via the pause domctl hypercall. So don't worry -- your
new type of pause should stack on top of 'xm pause/unpause' no problem.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> The 'pause' state visible to tools is a specific type of
> domain_pause(), and
> *has* to be entered via the pause domctl hypercall. So don't worry --
> your new type of pause should stack on top of 'xm pause/unpause' no
problem.

In fact, does the S3 suspended guest need to go into a pause state at
all? Presumably we could just let it block?
Xend does need to be able to know when the S3 suspend has actually
happened -- there should probably be a VIRQ for this.

Ian

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 15/5/08 12:45, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

>> The 'pause' state visible to tools is a specific type of
>> domain_pause(), and
>> *has* to be entered via the pause domctl hypercall. So don't worry --
>> your new type of pause should stack on top of 'xm pause/unpause' no
> problem.
> 
> In fact, does the S3 suspended guest need to go into a pause state at
> all? Presumably we could just let it block?
> Xend does need to be able to know when the S3 suspend has actually
> happened -- there should probably be a VIRQ for this.

S3 suspend is enacted by an I/O port write from the guest, trapped by qemu,
so the final reset of machine state is synchronously triggered by the tools
anyway.

As for pause-vs-block I think this is just a matter of terminology. At the
end of the day we want the domain to not run until s3-resume is triggered.
The natural way to do this within the hypervisor is to maintain a per-domain
is_s3_suspended flag and domain_[un]pause() when this flag toggles.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Ian Pratt]

> > In fact, does the S3 suspended guest need to go into a pause state
at
> > all? Presumably we could just let it block?
> > Xend does need to be able to know when the S3 suspend has actually
> > happened -- there should probably be a VIRQ for this.
> 
> S3 suspend is enacted by an I/O port write from the guest, trapped by
> qemu,
> so the final reset of machine state is synchronously triggered by the
> tools
> anyway.

OK, as long as qemu-dm communicates this via xenstore.
 
> As for pause-vs-block I think this is just a matter of terminology. At
> the end of the day we want the domain to not run until s3-resume is
> triggered.
> The natural way to do this within the hypervisor is to maintain a per-
> domain is_s3_suspended flag and domain_[un]pause() when this flag
toggles.

We wouldn't need to do the pause if the code after the magic port write
did "cli;hlt". Does it do this or loop?
[On real hardware the effect of the port write wouldn't be immediate, so
it must do something]

Ian

Re: [PATCH 0/4] HVM Virtual S3

[Keir Fraser]

On 15/5/08 13:42, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

>> As for pause-vs-block I think this is just a matter of terminology. At
>> the end of the day we want the domain to not run until s3-resume is
>> triggered.
>> The natural way to do this within the hypervisor is to maintain a per-
>> domain is_s3_suspended flag and domain_[un]pause() when this flag
> toggles.
> 
> We wouldn't need to do the pause if the code after the magic port write
> did "cli;hlt". Does it do this or loop?
> [On real hardware the effect of the port write wouldn't be immediate, so
> it must do something]

It spins. But to take advantage of that we have to restructure
s3-suspend/resume so that the reset of machine state happens in the resume
hypercall. At that point suspend, from the point of view of Xen, becomes a
no-op. I suppose we could do that... It seems a bit odd though and I don;t
know that it's guaranteed that all operating systems will happily spin
forever waiting for S3 oblivion. Some may time out.

 -- Keir

RE: [PATCH 0/4] HVM Virtual S3

[Ke, Liping]

Ok. So we will keep the current implementation now.
After we finish the patch, we'll send it for a quick review first.

Keir Fraser wrote:
> On 15/5/08 13:42, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:
> 
>>> As for pause-vs-block I think this is just a matter of terminology.
>>> At the end of the day we want the domain to not run until s3-resume
>>> is triggered. The natural way to do this within the hypervisor is
>>> to maintain a per- domain is_s3_suspended flag and
>>> domain_[un]pause() when this flag toggles. 
>> 
>> We wouldn't need to do the pause if the code after the magic port
>> write did "cli;hlt". Does it do this or loop?
>> [On real hardware the effect of the port write wouldn't be
>> immediate, so it must do something]
> 
> It spins. But to take advantage of that we have to restructure
> s3-suspend/resume so that the reset of machine state happens in the
> resume hypercall. At that point suspend, from the point of view of
> Xen, becomes a no-op. I suppose we could do that... It seems a bit
> odd though and I don;t know that it's guaranteed that all operating
> systems will happily spin forever waiting for S3 oblivion. Some may
> time out. 
> 
>  -- Keir