[RFC QEMU PATCH v9 0/1] S3 support

[RFC QEMU PATCH v9 0/1] S3 support

[Jiqian Chen]

Hi all,
This is the v9 patch to support S3.
v9 makes below changes:
* patch#1 no changes
* patch#2 remove unnecessary parentheses.
          add some comments to remind we may need to consider SUSPEND bit in future.
          change the commit message to describe which virtio device was tested.
          keep No_Soft_Reset bit false by default for safety.


Best regards,
Jiqian Chen


v8 makes below changes:
* Add a new patch#1 to fix a problem import by 27ce0f3afc9dd25d21b43bbce505157afd93d111,
  the right action is that only the state of PM_CTRL can be clear when resetting.
* patch#2 is the original patch to implement No_Soft_Reset bit, and in this version, I
  rename function and change some condition sequence.


v7 makes below changes:
* Tested this patch with Qemu on Xen hypervisor. Depending on kernel
  patch (virtio: Add support for no-reset virtio PCI PM:
  https://lore.kernel.org/lkml/20231208070754.3132339-1-stevensd@chromium.org/)
* Changed the default value of flag VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT to false
* Fixed coding style violation
* Modified the content of the comments.
* Removed useless flag PCI_PM_CTRL_DATA_SCALE_MASK.


V6:
In current code, when guest does S3, virtio devices are reset during that process, that
causes the display resources of virtio-gpu are destroyed, then the display can't come
back after resuming.
This v6 patch implement the No_Soft_Reset bit of PCI_PM_CTRL register, when this bit is
set, the resetting will not be done, so that the display can work after resuming.
This version abandons all previous version implementations and is a new different
solution according to the outcome of the discussion and suggestions in the mailing
thread of virtio-spec.
(https://lists.oasis-open.org/archives/virtio-comment/202401/msg00077.html)


V5:
v5 makes below changes:
* Since this series patches add a new mechanism that let virtgpu and Qemu can negotiate
  their reset behavior, and other guys hope me can improve this mechanism to virtio pci
  level, so that other virtio devices can also benefit from it. So instead of adding
  new feature flag VIRTIO_GPU_F_FREEZE_S3 only serves for virtgpu, v5 add a new parameter
  named freeze_mode to struct VirtIODevice, when guest begin suspending, set freeze_mode
  to VIRTIO_PCI_FREEZE_MODE_FREEZE_S3, and then all virtio devices can get this status,
  and notice that guest is suspending, then they can change their reset behavior . See
  the new commit "virtio-pci: Add freeze_mode case for virtio pci"
* The second commit is just for virtgpu, when freeze_mode is VIRTIO_PCI_FREEZE_MODE_FREEZE_S3,
  prevent Qemu destroying render resources, so that the display can come back after resuming.
V5 of kernel patch:
https://lore.kernel.org/lkml/20230919104607.2282248-1-Jiqian.Chen@amd.com/T/#t
The link to trace this issue:
https://gitlab.com/qemu-project/qemu/-/issues/1860


v4:
Thanks for Gerd Hoffmann's advice. V4 makes below changes:
* Use enum for freeze mode, so this can be extended with more
  modes in the future.
* Rename functions and paratemers with "_S3" postfix.
And no functional changes.
Link:
https://lore.kernel.org/qemu-devel/20230720120816.8751-1-Jiqian.Chen@amd.com/
No v4 patch on kernel side.


v3:
Thanks for Michael S. Tsirkin's advice. V3 makes below changes:
* Remove changes in file include/standard-headers/linux/virtio_gpu.h
  I am not supposed to edit this file and it will be imported after
  the patches of linux kernel was merged.
Link:
https://lore.kernel.org/qemu-devel/20230719074726.1613088-1-Jiqian.Chen@amd.com/T/#t
V3 of kernel patch:
https://lore.kernel.org/lkml/20230720115805.8206-1-Jiqian.Chen@amd.com/T/#t


v2:
makes below changes:
* Change VIRTIO_CPU_CMD_STATUS_FREEZING to 0x0400 (<0x1000)
* Add virtio_gpu_device_unrealize to destroy resources to solve
  potential memory leak problem. This also needs hot-plug support.
* Add a new feature flag VIRTIO_GPU_F_FREEZING, so that guest and
  host can negotiate whenever freezing is supported or not.
Link:
https://lore.kernel.org/qemu-devel/20230630070016.841459-1-Jiqian.Chen@amd.com/T/#t
V2 of kernel patch:
https://lore.kernel.org/lkml/20230630073448.842767-1-Jiqian.Chen@amd.com/T/#t


v1:
Hi all,
I am working to implement virtgpu S3 function on Xen.

Currently on Xen, if we start a guest who enables virtgpu, and then run
"echo mem > /sys/power/state" to suspend guest. And run "sudo xl trigger <guest id> s3resume"
to resume guest. We can find that the guest kernel comes back, but the display doesn't.
It just shown a black screen.

Through reading codes, I founded that when guest was during suspending, it called into Qemu
to call virtio_gpu_gl_reset. In virtio_gpu_gl_reset, it destroyed all resources and reset
renderer. This made the display gone after guest resumed.

I think we should keep resources or prevent they being destroyed when guest is suspending.
So, I add a new status named freezing to virtgpu, and add a new ctrl message
VIRTIO_GPU_CMD_STATUS_FREEZING to get notification from guest. If freezing is set to true,
and then Qemu will realize that guest is suspending, it will not destroy resources and will
not reset renderer. If freezing is set to false, Qemu will do its origin actions, and has no
other impaction.

And now, display can come back and applications can continue their status after guest resumes.
Link:
https://lore.kernel.org/qemu-devel/20230608025655.1674357-1-Jiqian.Chen@amd.com/
V1 of kernel patch:
https://lore.kernel.org/lkml/20230608063857.1677973-1-Jiqian.Chen@amd.com/

Jiqian Chen (2):
  virtio-pci: only reset pm state during resetting
  virtio-pci: implement No_Soft_Reset bit

 hw/virtio/virtio-pci.c         | 45 +++++++++++++++++++++++++++++++++-
 include/hw/virtio/virtio-pci.h |  5 ++++
 2 files changed, 49 insertions(+), 1 deletion(-)

-- 
2.34.1

[RFC QEMU PATCH v9 1/2] virtio-pci: only reset pm state during resetting

[Jiqian Chen]

Fix bug imported by 27ce0f3afc9dd25d21b43bbce505157afd93d111
(fix Power Management Control Register for PCI Express virtio devices)

Only state of PM_CTRL is writable.
Only when flag VIRTIO_PCI_FLAG_INIT_PM is set, need to reset state.

Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
---
 hw/virtio/virtio-pci.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index cb159fd0785c..a1b61308e7a0 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -2300,10 +2300,16 @@ static void virtio_pci_bus_reset_hold(Object *obj)
     virtio_pci_reset(qdev);
 
     if (pci_is_express(dev)) {
+        VirtIOPCIProxy *proxy = VIRTIO_PCI(dev);
+
         pcie_cap_deverr_reset(dev);
         pcie_cap_lnkctl_reset(dev);
 
-        pci_set_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL, 0);
+        if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
+            pci_word_test_and_clear_mask(
+                dev->config + dev->exp.pm_cap + PCI_PM_CTRL,
+                PCI_PM_CTRL_STATE_MASK);
+        }
     }
 }
 
-- 
2.34.1

[RFC QEMU PATCH v9 2/2] virtio-pci: implement No_Soft_Reset bit

[Jiqian Chen]

In current code, when guest does S3, virtio-gpu are reset due to the
bit No_Soft_Reset is not set. After resetting, the display resources
of virtio-gpu are destroyed, then the display can't come back and only
show blank after resuming.

Implement No_Soft_Reset bit of PCI_PM_CTRL register, then guest can check
this bit, if this bit is set, the devices resetting will not be done, and
then the display can work after resuming.

No_Soft_Reset bit is implemented for all virtio devices, and was tested
only on virtio-gpu device. Set it false by default for safety.

Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
---
 hw/virtio/virtio-pci.c         | 37 ++++++++++++++++++++++++++++++++++
 include/hw/virtio/virtio-pci.h |  5 +++++
 2 files changed, 42 insertions(+)

diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index a1b61308e7a0..82fa4defe5cd 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -2230,6 +2230,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
             pcie_cap_lnkctl_init(pci_dev);
         }
 
+        if (proxy->flags & VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET) {
+            pci_set_word(pci_dev->config + pos + PCI_PM_CTRL,
+                         PCI_PM_CTRL_NO_SOFT_RESET);
+        }
+
         if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
             /* Init Power Management Control Register */
             pci_set_word(pci_dev->wmask + pos + PCI_PM_CTRL,
@@ -2292,11 +2297,37 @@ static void virtio_pci_reset(DeviceState *qdev)
     }
 }
 
+static bool virtio_pci_no_soft_reset(PCIDevice *dev)
+{
+    uint16_t pmcsr;
+
+    if (!pci_is_express(dev) || !dev->exp.pm_cap) {
+        return false;
+    }
+
+    pmcsr = pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL);
+
+    /*
+     * When No_Soft_Reset bit is set and the device
+     * is in D3hot state, don't reset device
+     */
+    return (pmcsr & PCI_PM_CTRL_NO_SOFT_RESET) &&
+           (pmcsr & PCI_PM_CTRL_STATE_MASK) == 3;
+}
+
 static void virtio_pci_bus_reset_hold(Object *obj)
 {
     PCIDevice *dev = PCI_DEVICE(obj);
     DeviceState *qdev = DEVICE(obj);
 
+    /*
+     * Note that: a proposal to add SUSPEND bit is being discussed,
+     * may need to consider the state of SUSPEND bit in future
+     */
+    if (virtio_pci_no_soft_reset(dev)) {
+        return;
+    }
+
     virtio_pci_reset(qdev);
 
     if (pci_is_express(dev)) {
@@ -2336,6 +2367,12 @@ static Property virtio_pci_properties[] = {
                     VIRTIO_PCI_FLAG_INIT_LNKCTL_BIT, true),
     DEFINE_PROP_BIT("x-pcie-pm-init", VirtIOPCIProxy, flags,
                     VIRTIO_PCI_FLAG_INIT_PM_BIT, true),
+    /*
+     * for safety, set this false by default, if change it to true,
+     * need to consider compatible for old machine
+     */
+    DEFINE_PROP_BIT("pcie-pm-no-soft-reset", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT, false),
     DEFINE_PROP_BIT("x-pcie-flr-init", VirtIOPCIProxy, flags,
                     VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
     DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
index 59d88018c16a..9e67ba38c748 100644
--- a/include/hw/virtio/virtio-pci.h
+++ b/include/hw/virtio/virtio-pci.h
@@ -43,6 +43,7 @@ enum {
     VIRTIO_PCI_FLAG_INIT_FLR_BIT,
     VIRTIO_PCI_FLAG_AER_BIT,
     VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED_BIT,
+    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT,
 };
 
 /* Need to activate work-arounds for buggy guests at vmstate load. */
@@ -79,6 +80,10 @@ enum {
 /* Init Power Management */
 #define VIRTIO_PCI_FLAG_INIT_PM (1 << VIRTIO_PCI_FLAG_INIT_PM_BIT)
 
+/* Init The No_Soft_Reset bit of Power Management */
+#define VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET \
+  (1 << VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT)
+
 /* Init Function Level Reset capability */
 #define VIRTIO_PCI_FLAG_INIT_FLR (1 << VIRTIO_PCI_FLAG_INIT_FLR_BIT)
 
-- 
2.34.1

Re: [RFC QEMU PATCH v9 2/2] virtio-pci: implement No_Soft_Reset bit

[Igor Mammedov]

On Tue, 16 Apr 2024 15:01:27 +0800
Jiqian Chen <Jiqian.Chen@amd.com> wrote:

> In current code, when guest does S3, virtio-gpu are reset due to the
> bit No_Soft_Reset is not set. After resetting, the display resources
> of virtio-gpu are destroyed, then the display can't come back and only
> show blank after resuming.

Just a high level question.
Typically when system goes into S3 all devices (modulo RAM) loose context
(aka powered off), and then it's upto device driver to recover whatever
was lost.
So why should we implement hw(qemu) workaround for a driver problem?

> 
> Implement No_Soft_Reset bit of PCI_PM_CTRL register, then guest can check
> this bit, if this bit is set, the devices resetting will not be done, and
> then the display can work after resuming.
> 
> No_Soft_Reset bit is implemented for all virtio devices, and was tested
> only on virtio-gpu device. Set it false by default for safety.
> 
> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
> ---
>  hw/virtio/virtio-pci.c         | 37 ++++++++++++++++++++++++++++++++++
>  include/hw/virtio/virtio-pci.h |  5 +++++
>  2 files changed, 42 insertions(+)
> 
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index a1b61308e7a0..82fa4defe5cd 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -2230,6 +2230,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>              pcie_cap_lnkctl_init(pci_dev);
>          }
>  
> +        if (proxy->flags & VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET) {
> +            pci_set_word(pci_dev->config + pos + PCI_PM_CTRL,
> +                         PCI_PM_CTRL_NO_SOFT_RESET);
> +        }
> +
>          if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
>              /* Init Power Management Control Register */
>              pci_set_word(pci_dev->wmask + pos + PCI_PM_CTRL,
> @@ -2292,11 +2297,37 @@ static void virtio_pci_reset(DeviceState *qdev)
>      }
>  }
>  
> +static bool virtio_pci_no_soft_reset(PCIDevice *dev)
> +{
> +    uint16_t pmcsr;
> +
> +    if (!pci_is_express(dev) || !dev->exp.pm_cap) {
> +        return false;
> +    }
> +
> +    pmcsr = pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL);
> +
> +    /*
> +     * When No_Soft_Reset bit is set and the device
> +     * is in D3hot state, don't reset device
> +     */
> +    return (pmcsr & PCI_PM_CTRL_NO_SOFT_RESET) &&
> +           (pmcsr & PCI_PM_CTRL_STATE_MASK) == 3;
> +}
> +
>  static void virtio_pci_bus_reset_hold(Object *obj)
>  {
>      PCIDevice *dev = PCI_DEVICE(obj);
>      DeviceState *qdev = DEVICE(obj);
>  
> +    /*
> +     * Note that: a proposal to add SUSPEND bit is being discussed,
> +     * may need to consider the state of SUSPEND bit in future
> +     */
> +    if (virtio_pci_no_soft_reset(dev)) {
> +        return;
> +    }
> +
>      virtio_pci_reset(qdev);
>  
>      if (pci_is_express(dev)) {
> @@ -2336,6 +2367,12 @@ static Property virtio_pci_properties[] = {
>                      VIRTIO_PCI_FLAG_INIT_LNKCTL_BIT, true),
>      DEFINE_PROP_BIT("x-pcie-pm-init", VirtIOPCIProxy, flags,
>                      VIRTIO_PCI_FLAG_INIT_PM_BIT, true),
> +    /*
> +     * for safety, set this false by default, if change it to true,
> +     * need to consider compatible for old machine
> +     */
> +    DEFINE_PROP_BIT("pcie-pm-no-soft-reset", VirtIOPCIProxy, flags,
> +                    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT, false),
>      DEFINE_PROP_BIT("x-pcie-flr-init", VirtIOPCIProxy, flags,
>                      VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
>      DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
> index 59d88018c16a..9e67ba38c748 100644
> --- a/include/hw/virtio/virtio-pci.h
> +++ b/include/hw/virtio/virtio-pci.h
> @@ -43,6 +43,7 @@ enum {
>      VIRTIO_PCI_FLAG_INIT_FLR_BIT,
>      VIRTIO_PCI_FLAG_AER_BIT,
>      VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED_BIT,
> +    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT,
>  };
>  
>  /* Need to activate work-arounds for buggy guests at vmstate load. */
> @@ -79,6 +80,10 @@ enum {
>  /* Init Power Management */
>  #define VIRTIO_PCI_FLAG_INIT_PM (1 << VIRTIO_PCI_FLAG_INIT_PM_BIT)
>  
> +/* Init The No_Soft_Reset bit of Power Management */
> +#define VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET \
> +  (1 << VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT)
> +
>  /* Init Function Level Reset capability */
>  #define VIRTIO_PCI_FLAG_INIT_FLR (1 << VIRTIO_PCI_FLAG_INIT_FLR_BIT)
>  

Re: [RFC QEMU PATCH v9 2/2] virtio-pci: implement No_Soft_Reset bit

[Chen, Jiqian]

On 2024/4/16 23:45, Igor Mammedov wrote:
> On Tue, 16 Apr 2024 15:01:27 +0800
> Jiqian Chen <Jiqian.Chen@amd.com> wrote:
> 
>> In current code, when guest does S3, virtio-gpu are reset due to the
>> bit No_Soft_Reset is not set. After resetting, the display resources
>> of virtio-gpu are destroyed, then the display can't come back and only
>> show blank after resuming.
> 
> Just a high level question.
> Typically when system goes into S3 all devices (modulo RAM) loose context
> (aka powered off), and then it's upto device driver to recover whatever
> was lost.
No, what you said is just one situation that when system goes into S3 devices loose context and fully reinitialized and be D0-uninitialized state.
The other situation is the context must be maintained so that the devices can quickly be D0-active state after resuming.
There are some descriptions in PCIe specification in Chapter 5.3.1.4 D3 state:
" Functional context is required to be maintained by Functions in the D3Hot state if the No_Soft_Reset field in the PMCSR is
Set. In this case, System Software is not required to re-initialize the Function after a transition from D3Hot to D0 (the
Function will be in the D0active state). If the No_Soft_Reset bit is Clear, functional context is not required to be
maintained by the Function in the D3Hot state, however it is not guaranteed that functional context will be cleared and
software must not depend on such behavior. As a result, in this case System Software is required to fully re-initialize the
Function after a transition to D0 as the Function will be in the D0uninitialized state."

What's more, not all virtio devices' context can be recovered by driver after resuming, like virtio-gpu, we have not enough information to re-create all
display resources, that is discussed in my previous version email thread.

> So why should we implement hw(qemu) workaround for a driver problem?
I think this is not a workaround, No_Soft_Reset bit is also described in PCIe specification, it can be set or not.
And we can set this bit for the specific device which we want to maintain context during S3.

> 
>>
>> Implement No_Soft_Reset bit of PCI_PM_CTRL register, then guest can check
>> this bit, if this bit is set, the devices resetting will not be done, and
>> then the display can work after resuming.
>>
>> No_Soft_Reset bit is implemented for all virtio devices, and was tested
>> only on virtio-gpu device. Set it false by default for safety.
>>
>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
>> ---
>>  hw/virtio/virtio-pci.c         | 37 ++++++++++++++++++++++++++++++++++
>>  include/hw/virtio/virtio-pci.h |  5 +++++
>>  2 files changed, 42 insertions(+)
>>
>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>> index a1b61308e7a0..82fa4defe5cd 100644
>> --- a/hw/virtio/virtio-pci.c
>> +++ b/hw/virtio/virtio-pci.c
>> @@ -2230,6 +2230,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>>              pcie_cap_lnkctl_init(pci_dev);
>>          }
>>  
>> +        if (proxy->flags & VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET) {
>> +            pci_set_word(pci_dev->config + pos + PCI_PM_CTRL,
>> +                         PCI_PM_CTRL_NO_SOFT_RESET);
>> +        }
>> +
>>          if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
>>              /* Init Power Management Control Register */
>>              pci_set_word(pci_dev->wmask + pos + PCI_PM_CTRL,
>> @@ -2292,11 +2297,37 @@ static void virtio_pci_reset(DeviceState *qdev)
>>      }
>>  }
>>  
>> +static bool virtio_pci_no_soft_reset(PCIDevice *dev)
>> +{
>> +    uint16_t pmcsr;
>> +
>> +    if (!pci_is_express(dev) || !dev->exp.pm_cap) {
>> +        return false;
>> +    }
>> +
>> +    pmcsr = pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL);
>> +
>> +    /*
>> +     * When No_Soft_Reset bit is set and the device
>> +     * is in D3hot state, don't reset device
>> +     */
>> +    return (pmcsr & PCI_PM_CTRL_NO_SOFT_RESET) &&
>> +           (pmcsr & PCI_PM_CTRL_STATE_MASK) == 3;
>> +}
>> +
>>  static void virtio_pci_bus_reset_hold(Object *obj)
>>  {
>>      PCIDevice *dev = PCI_DEVICE(obj);
>>      DeviceState *qdev = DEVICE(obj);
>>  
>> +    /*
>> +     * Note that: a proposal to add SUSPEND bit is being discussed,
>> +     * may need to consider the state of SUSPEND bit in future
>> +     */
>> +    if (virtio_pci_no_soft_reset(dev)) {
>> +        return;
>> +    }
>> +
>>      virtio_pci_reset(qdev);
>>  
>>      if (pci_is_express(dev)) {
>> @@ -2336,6 +2367,12 @@ static Property virtio_pci_properties[] = {
>>                      VIRTIO_PCI_FLAG_INIT_LNKCTL_BIT, true),
>>      DEFINE_PROP_BIT("x-pcie-pm-init", VirtIOPCIProxy, flags,
>>                      VIRTIO_PCI_FLAG_INIT_PM_BIT, true),
>> +    /*
>> +     * for safety, set this false by default, if change it to true,
>> +     * need to consider compatible for old machine
>> +     */
>> +    DEFINE_PROP_BIT("pcie-pm-no-soft-reset", VirtIOPCIProxy, flags,
>> +                    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT, false),
>>      DEFINE_PROP_BIT("x-pcie-flr-init", VirtIOPCIProxy, flags,
>>                      VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
>>      DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
>> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
>> index 59d88018c16a..9e67ba38c748 100644
>> --- a/include/hw/virtio/virtio-pci.h
>> +++ b/include/hw/virtio/virtio-pci.h
>> @@ -43,6 +43,7 @@ enum {
>>      VIRTIO_PCI_FLAG_INIT_FLR_BIT,
>>      VIRTIO_PCI_FLAG_AER_BIT,
>>      VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED_BIT,
>> +    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT,
>>  };
>>  
>>  /* Need to activate work-arounds for buggy guests at vmstate load. */
>> @@ -79,6 +80,10 @@ enum {
>>  /* Init Power Management */
>>  #define VIRTIO_PCI_FLAG_INIT_PM (1 << VIRTIO_PCI_FLAG_INIT_PM_BIT)
>>  
>> +/* Init The No_Soft_Reset bit of Power Management */
>> +#define VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET \
>> +  (1 << VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT)
>> +
>>  /* Init Function Level Reset capability */
>>  #define VIRTIO_PCI_FLAG_INIT_FLR (1 << VIRTIO_PCI_FLAG_INIT_FLR_BIT)
>>  
> 

-- 
Best regards,
Jiqian Chen.

Re: [RFC QEMU PATCH v9 2/2] virtio-pci: implement No_Soft_Reset bit

[Yan Vugenfirer]

On Wed, Apr 17, 2024 at 6:13 AM Chen, Jiqian <Jiqian.Chen@amd.com> wrote:
>
> On 2024/4/16 23:45, Igor Mammedov wrote:
> > On Tue, 16 Apr 2024 15:01:27 +0800
> > Jiqian Chen <Jiqian.Chen@amd.com> wrote:
> >
> >> In current code, when guest does S3, virtio-gpu are reset due to the
> >> bit No_Soft_Reset is not set. After resetting, the display resources
> >> of virtio-gpu are destroyed, then the display can't come back and only
> >> show blank after resuming.
> >
> > Just a high level question.
> > Typically when system goes into S3 all devices (modulo RAM) loose context
> > (aka powered off), and then it's upto device driver to recover whatever
> > was lost.
> No, what you said is just one situation that when system goes into S3 devices loose context and fully reinitialized and be D0-uninitialized state.
> The other situation is the context must be maintained so that the devices can quickly be D0-active state after resuming.
> There are some descriptions in PCIe specification in Chapter 5.3.1.4 D3 state:
> " Functional context is required to be maintained by Functions in the D3Hot state if the No_Soft_Reset field in the PMCSR is
> Set. In this case, System Software is not required to re-initialize the Function after a transition from D3Hot to D0 (the
> Function will be in the D0active state). If the No_Soft_Reset bit is Clear, functional context is not required to be
> maintained by the Function in the D3Hot state, however it is not guaranteed that functional context will be cleared and
> software must not depend on such behavior. As a result, in this case System Software is required to fully re-initialize the
> Function after a transition to D0 as the Function will be in the D0uninitialized state."
>
BTW: according to Windows documentation D3 Hot state implementation is
a must for real HW devices.
https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/device-sleeping-states
"Device power states D1, D2, and D3 are the device low-power states.
Starting with Windows 8, D3 is divided into two substates, D3hot and
D3cold.

D1 and D2 are intermediate low-power states. Many classes of devices
do not define these states. All devices must define D3hot."

Best regards,
Yan.


> What's more, not all virtio devices' context can be recovered by driver after resuming, like virtio-gpu, we have not enough information to re-create all
> display resources, that is discussed in my previous version email thread.
>
> > So why should we implement hw(qemu) workaround for a driver problem?
> I think this is not a workaround, No_Soft_Reset bit is also described in PCIe specification, it can be set or not.
> And we can set this bit for the specific device which we want to maintain context during S3.
>
> >
> >>
> >> Implement No_Soft_Reset bit of PCI_PM_CTRL register, then guest can check
> >> this bit, if this bit is set, the devices resetting will not be done, and
> >> then the display can work after resuming.
> >>
> >> No_Soft_Reset bit is implemented for all virtio devices, and was tested
> >> only on virtio-gpu device. Set it false by default for safety.
> >>
> >> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
> >> ---
> >>  hw/virtio/virtio-pci.c         | 37 ++++++++++++++++++++++++++++++++++
> >>  include/hw/virtio/virtio-pci.h |  5 +++++
> >>  2 files changed, 42 insertions(+)
> >>
> >> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> >> index a1b61308e7a0..82fa4defe5cd 100644
> >> --- a/hw/virtio/virtio-pci.c
> >> +++ b/hw/virtio/virtio-pci.c
> >> @@ -2230,6 +2230,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
> >>              pcie_cap_lnkctl_init(pci_dev);
> >>          }
> >>
> >> +        if (proxy->flags & VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET) {
> >> +            pci_set_word(pci_dev->config + pos + PCI_PM_CTRL,
> >> +                         PCI_PM_CTRL_NO_SOFT_RESET);
> >> +        }
> >> +
> >>          if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
> >>              /* Init Power Management Control Register */
> >>              pci_set_word(pci_dev->wmask + pos + PCI_PM_CTRL,
> >> @@ -2292,11 +2297,37 @@ static void virtio_pci_reset(DeviceState *qdev)
> >>      }
> >>  }
> >>
> >> +static bool virtio_pci_no_soft_reset(PCIDevice *dev)
> >> +{
> >> +    uint16_t pmcsr;
> >> +
> >> +    if (!pci_is_express(dev) || !dev->exp.pm_cap) {
> >> +        return false;
> >> +    }
> >> +
> >> +    pmcsr = pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL);
> >> +
> >> +    /*
> >> +     * When No_Soft_Reset bit is set and the device
> >> +     * is in D3hot state, don't reset device
> >> +     */
> >> +    return (pmcsr & PCI_PM_CTRL_NO_SOFT_RESET) &&
> >> +           (pmcsr & PCI_PM_CTRL_STATE_MASK) == 3;
> >> +}
> >> +
> >>  static void virtio_pci_bus_reset_hold(Object *obj)
> >>  {
> >>      PCIDevice *dev = PCI_DEVICE(obj);
> >>      DeviceState *qdev = DEVICE(obj);
> >>
> >> +    /*
> >> +     * Note that: a proposal to add SUSPEND bit is being discussed,
> >> +     * may need to consider the state of SUSPEND bit in future
> >> +     */
> >> +    if (virtio_pci_no_soft_reset(dev)) {
> >> +        return;
> >> +    }
> >> +
> >>      virtio_pci_reset(qdev);
> >>
> >>      if (pci_is_express(dev)) {
> >> @@ -2336,6 +2367,12 @@ static Property virtio_pci_properties[] = {
> >>                      VIRTIO_PCI_FLAG_INIT_LNKCTL_BIT, true),
> >>      DEFINE_PROP_BIT("x-pcie-pm-init", VirtIOPCIProxy, flags,
> >>                      VIRTIO_PCI_FLAG_INIT_PM_BIT, true),
> >> +    /*
> >> +     * for safety, set this false by default, if change it to true,
> >> +     * need to consider compatible for old machine
> >> +     */
> >> +    DEFINE_PROP_BIT("pcie-pm-no-soft-reset", VirtIOPCIProxy, flags,
> >> +                    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT, false),
> >>      DEFINE_PROP_BIT("x-pcie-flr-init", VirtIOPCIProxy, flags,
> >>                      VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
> >>      DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
> >> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
> >> index 59d88018c16a..9e67ba38c748 100644
> >> --- a/include/hw/virtio/virtio-pci.h
> >> +++ b/include/hw/virtio/virtio-pci.h
> >> @@ -43,6 +43,7 @@ enum {
> >>      VIRTIO_PCI_FLAG_INIT_FLR_BIT,
> >>      VIRTIO_PCI_FLAG_AER_BIT,
> >>      VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED_BIT,
> >> +    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT,
> >>  };
> >>
> >>  /* Need to activate work-arounds for buggy guests at vmstate load. */
> >> @@ -79,6 +80,10 @@ enum {
> >>  /* Init Power Management */
> >>  #define VIRTIO_PCI_FLAG_INIT_PM (1 << VIRTIO_PCI_FLAG_INIT_PM_BIT)
> >>
> >> +/* Init The No_Soft_Reset bit of Power Management */
> >> +#define VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET \
> >> +  (1 << VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT)
> >> +
> >>  /* Init Function Level Reset capability */
> >>  #define VIRTIO_PCI_FLAG_INIT_FLR (1 << VIRTIO_PCI_FLAG_INIT_FLR_BIT)
> >>
> >
>
> --
> Best regards,
> Jiqian Chen.

Re: [RFC QEMU PATCH v9 2/2] virtio-pci: implement No_Soft_Reset bit

[Chen, Jiqian]

On 2024/4/17 13:33, Yan Vugenfirer wrote:
> On Wed, Apr 17, 2024 at 6:13 AM Chen, Jiqian <Jiqian.Chen@amd.com> wrote:
>>
>> On 2024/4/16 23:45, Igor Mammedov wrote:
>>> On Tue, 16 Apr 2024 15:01:27 +0800
>>> Jiqian Chen <Jiqian.Chen@amd.com> wrote:
>>>
>>>> In current code, when guest does S3, virtio-gpu are reset due to the
>>>> bit No_Soft_Reset is not set. After resetting, the display resources
>>>> of virtio-gpu are destroyed, then the display can't come back and only
>>>> show blank after resuming.
>>>
>>> Just a high level question.
>>> Typically when system goes into S3 all devices (modulo RAM) loose context
>>> (aka powered off), and then it's upto device driver to recover whatever
>>> was lost.
>> No, what you said is just one situation that when system goes into S3 devices loose context and fully reinitialized and be D0-uninitialized state.
>> The other situation is the context must be maintained so that the devices can quickly be D0-active state after resuming.
>> There are some descriptions in PCIe specification in Chapter 5.3.1.4 D3 state:
>> " Functional context is required to be maintained by Functions in the D3Hot state if the No_Soft_Reset field in the PMCSR is
>> Set. In this case, System Software is not required to re-initialize the Function after a transition from D3Hot to D0 (the
>> Function will be in the D0active state). If the No_Soft_Reset bit is Clear, functional context is not required to be
>> maintained by the Function in the D3Hot state, however it is not guaranteed that functional context will be cleared and
>> software must not depend on such behavior. As a result, in this case System Software is required to fully re-initialize the
>> Function after a transition to D0 as the Function will be in the D0uninitialized state."
>>
> BTW: according to Windows documentation D3 Hot state implementation is
> a must for real HW devices.
> https://learn.microsoft.com/en-us/windows-hardware/drivers/kernel/device-sleeping-states
> "Device power states D1, D2, and D3 are the device low-power states.
> Starting with Windows 8, D3 is divided into two substates, D3hot and
> D3cold.
> 
> D1 and D2 are intermediate low-power states. Many classes of devices
> do not define these states. All devices must define D3hot."
Yes, I know, it is also described in PCIe specification. But this is not a conflict.
When set No_Soft_Reset didn’t mean the device can't enter D3hot state. It means when device enter D3hot, the context of device must be maintained.

> 
> Best regards,
> Yan.
> 
> 
>> What's more, not all virtio devices' context can be recovered by driver after resuming, like virtio-gpu, we have not enough information to re-create all
>> display resources, that is discussed in my previous version email thread.
>>
>>> So why should we implement hw(qemu) workaround for a driver problem?
>> I think this is not a workaround, No_Soft_Reset bit is also described in PCIe specification, it can be set or not.
>> And we can set this bit for the specific device which we want to maintain context during S3.
>>
>>>
>>>>
>>>> Implement No_Soft_Reset bit of PCI_PM_CTRL register, then guest can check
>>>> this bit, if this bit is set, the devices resetting will not be done, and
>>>> then the display can work after resuming.
>>>>
>>>> No_Soft_Reset bit is implemented for all virtio devices, and was tested
>>>> only on virtio-gpu device. Set it false by default for safety.
>>>>
>>>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
>>>> ---
>>>>  hw/virtio/virtio-pci.c         | 37 ++++++++++++++++++++++++++++++++++
>>>>  include/hw/virtio/virtio-pci.h |  5 +++++
>>>>  2 files changed, 42 insertions(+)
>>>>
>>>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>>>> index a1b61308e7a0..82fa4defe5cd 100644
>>>> --- a/hw/virtio/virtio-pci.c
>>>> +++ b/hw/virtio/virtio-pci.c
>>>> @@ -2230,6 +2230,11 @@ static void virtio_pci_realize(PCIDevice *pci_dev, Error **errp)
>>>>              pcie_cap_lnkctl_init(pci_dev);
>>>>          }
>>>>
>>>> +        if (proxy->flags & VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET) {
>>>> +            pci_set_word(pci_dev->config + pos + PCI_PM_CTRL,
>>>> +                         PCI_PM_CTRL_NO_SOFT_RESET);
>>>> +        }
>>>> +
>>>>          if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
>>>>              /* Init Power Management Control Register */
>>>>              pci_set_word(pci_dev->wmask + pos + PCI_PM_CTRL,
>>>> @@ -2292,11 +2297,37 @@ static void virtio_pci_reset(DeviceState *qdev)
>>>>      }
>>>>  }
>>>>
>>>> +static bool virtio_pci_no_soft_reset(PCIDevice *dev)
>>>> +{
>>>> +    uint16_t pmcsr;
>>>> +
>>>> +    if (!pci_is_express(dev) || !dev->exp.pm_cap) {
>>>> +        return false;
>>>> +    }
>>>> +
>>>> +    pmcsr = pci_get_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL);
>>>> +
>>>> +    /*
>>>> +     * When No_Soft_Reset bit is set and the device
>>>> +     * is in D3hot state, don't reset device
>>>> +     */
>>>> +    return (pmcsr & PCI_PM_CTRL_NO_SOFT_RESET) &&
>>>> +           (pmcsr & PCI_PM_CTRL_STATE_MASK) == 3;
>>>> +}
>>>> +
>>>>  static void virtio_pci_bus_reset_hold(Object *obj)
>>>>  {
>>>>      PCIDevice *dev = PCI_DEVICE(obj);
>>>>      DeviceState *qdev = DEVICE(obj);
>>>>
>>>> +    /*
>>>> +     * Note that: a proposal to add SUSPEND bit is being discussed,
>>>> +     * may need to consider the state of SUSPEND bit in future
>>>> +     */
>>>> +    if (virtio_pci_no_soft_reset(dev)) {
>>>> +        return;
>>>> +    }
>>>> +
>>>>      virtio_pci_reset(qdev);
>>>>
>>>>      if (pci_is_express(dev)) {
>>>> @@ -2336,6 +2367,12 @@ static Property virtio_pci_properties[] = {
>>>>                      VIRTIO_PCI_FLAG_INIT_LNKCTL_BIT, true),
>>>>      DEFINE_PROP_BIT("x-pcie-pm-init", VirtIOPCIProxy, flags,
>>>>                      VIRTIO_PCI_FLAG_INIT_PM_BIT, true),
>>>> +    /*
>>>> +     * for safety, set this false by default, if change it to true,
>>>> +     * need to consider compatible for old machine
>>>> +     */
>>>> +    DEFINE_PROP_BIT("pcie-pm-no-soft-reset", VirtIOPCIProxy, flags,
>>>> +                    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT, false),
>>>>      DEFINE_PROP_BIT("x-pcie-flr-init", VirtIOPCIProxy, flags,
>>>>                      VIRTIO_PCI_FLAG_INIT_FLR_BIT, true),
>>>>      DEFINE_PROP_BIT("aer", VirtIOPCIProxy, flags,
>>>> diff --git a/include/hw/virtio/virtio-pci.h b/include/hw/virtio/virtio-pci.h
>>>> index 59d88018c16a..9e67ba38c748 100644
>>>> --- a/include/hw/virtio/virtio-pci.h
>>>> +++ b/include/hw/virtio/virtio-pci.h
>>>> @@ -43,6 +43,7 @@ enum {
>>>>      VIRTIO_PCI_FLAG_INIT_FLR_BIT,
>>>>      VIRTIO_PCI_FLAG_AER_BIT,
>>>>      VIRTIO_PCI_FLAG_ATS_PAGE_ALIGNED_BIT,
>>>> +    VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT,
>>>>  };
>>>>
>>>>  /* Need to activate work-arounds for buggy guests at vmstate load. */
>>>> @@ -79,6 +80,10 @@ enum {
>>>>  /* Init Power Management */
>>>>  #define VIRTIO_PCI_FLAG_INIT_PM (1 << VIRTIO_PCI_FLAG_INIT_PM_BIT)
>>>>
>>>> +/* Init The No_Soft_Reset bit of Power Management */
>>>> +#define VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET \
>>>> +  (1 << VIRTIO_PCI_FLAG_PM_NO_SOFT_RESET_BIT)
>>>> +
>>>>  /* Init Function Level Reset capability */
>>>>  #define VIRTIO_PCI_FLAG_INIT_FLR (1 << VIRTIO_PCI_FLAG_INIT_FLR_BIT)
>>>>
>>>
>>
>> --
>> Best regards,
>> Jiqian Chen.
> 

-- 
Best regards,
Jiqian Chen.

Re: [RFC QEMU PATCH v9 1/2] virtio-pci: only reset pm state during resetting

[Chen, Jiqian]

Hi,

On 2024/4/16 15:01, Jiqian Chen wrote:
> Fix bug imported by 27ce0f3afc9dd25d21b43bbce505157afd93d111
> (fix Power Management Control Register for PCI Express virtio devices)
> 
> Only state of PM_CTRL is writable.
> Only when flag VIRTIO_PCI_FLAG_INIT_PM is set, need to reset state.
> 
> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
> ---
>  hw/virtio/virtio-pci.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index cb159fd0785c..a1b61308e7a0 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -2300,10 +2300,16 @@ static void virtio_pci_bus_reset_hold(Object *obj)
>      virtio_pci_reset(qdev);
>  
>      if (pci_is_express(dev)) {
> +        VirtIOPCIProxy *proxy = VIRTIO_PCI(dev);
> +
>          pcie_cap_deverr_reset(dev);
>          pcie_cap_lnkctl_reset(dev);
>  
> -        pci_set_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL, 0);
> +        if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
> +            pci_word_test_and_clear_mask(
> +                dev->config + dev->exp.pm_cap + PCI_PM_CTRL,
> +                PCI_PM_CTRL_STATE_MASK);
> +        }
>      }
>  }
>  

Do you have any other doubts about this patch?
And another patch of this series?

-- 
Best regards,
Jiqian Chen.

Re: [RFC QEMU PATCH v9 1/2] virtio-pci: only reset pm state during resetting

[Michael S. Tsirkin]

On Tue, Apr 16, 2024 at 03:01:26PM +0800, Jiqian Chen wrote:
> Fix bug imported by 27ce0f3afc9dd25d21b43bbce505157afd93d111
> (fix Power Management Control Register for PCI Express virtio devices)


should be:

27ce0f3afc9dd ("fix Power Management Control Register for PCI Express virtio devices"

Pls include a bit more info about the bug: after this change, we observe ...... .

> Only state of PM_CTRL is writable.
> Only when flag VIRTIO_PCI_FLAG_INIT_PM is set, need to reset state.



and here, add:

Fixes: 27ce0f3afc9dd ("fix Power Management Control Register for PCI Express virtio devices"

> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
> ---
>  hw/virtio/virtio-pci.c | 8 +++++++-
>  1 file changed, 7 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> index cb159fd0785c..a1b61308e7a0 100644
> --- a/hw/virtio/virtio-pci.c
> +++ b/hw/virtio/virtio-pci.c
> @@ -2300,10 +2300,16 @@ static void virtio_pci_bus_reset_hold(Object *obj)
>      virtio_pci_reset(qdev);
>  
>      if (pci_is_express(dev)) {
> +        VirtIOPCIProxy *proxy = VIRTIO_PCI(dev);
> +
>          pcie_cap_deverr_reset(dev);
>          pcie_cap_lnkctl_reset(dev);
>  
> -        pci_set_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL, 0);
> +        if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
> +            pci_word_test_and_clear_mask(
> +                dev->config + dev->exp.pm_cap + PCI_PM_CTRL,
> +                PCI_PM_CTRL_STATE_MASK);
> +        }
>      }
>  }
>  
> -- 
> 2.34.1

Re: [RFC QEMU PATCH v9 1/2] virtio-pci: only reset pm state during resetting

[Chen, Jiqian]

On 2024/5/10 18:18, Michael S. Tsirkin wrote:
> On Tue, Apr 16, 2024 at 03:01:26PM +0800, Jiqian Chen wrote:
>> Fix bug imported by 27ce0f3afc9dd25d21b43bbce505157afd93d111
>> (fix Power Management Control Register for PCI Express virtio devices)
> 
> 
> should be:
> 
> 27ce0f3afc9dd ("fix Power Management Control Register for PCI Express virtio devices"
> 
> Pls include a bit more info about the bug: after this change, we observe ...... .
OK, I will modify in next version.
How about the other patch of this series?
Do you have any doubts?

> 
>> Only state of PM_CTRL is writable.
>> Only when flag VIRTIO_PCI_FLAG_INIT_PM is set, need to reset state.
> 
> 
> 
> and here, add:
> 
> Fixes: 27ce0f3afc9dd ("fix Power Management Control Register for PCI Express virtio devices"
> 
>> Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com>
>> ---
>>  hw/virtio/virtio-pci.c | 8 +++++++-
>>  1 file changed, 7 insertions(+), 1 deletion(-)
>>
>> diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
>> index cb159fd0785c..a1b61308e7a0 100644
>> --- a/hw/virtio/virtio-pci.c
>> +++ b/hw/virtio/virtio-pci.c
>> @@ -2300,10 +2300,16 @@ static void virtio_pci_bus_reset_hold(Object *obj)
>>      virtio_pci_reset(qdev);
>>  
>>      if (pci_is_express(dev)) {
>> +        VirtIOPCIProxy *proxy = VIRTIO_PCI(dev);
>> +
>>          pcie_cap_deverr_reset(dev);
>>          pcie_cap_lnkctl_reset(dev);
>>  
>> -        pci_set_word(dev->config + dev->exp.pm_cap + PCI_PM_CTRL, 0);
>> +        if (proxy->flags & VIRTIO_PCI_FLAG_INIT_PM) {
>> +            pci_word_test_and_clear_mask(
>> +                dev->config + dev->exp.pm_cap + PCI_PM_CTRL,
>> +                PCI_PM_CTRL_STATE_MASK);
>> +        }
>>      }
>>  }
>>  
>> -- 
>> 2.34.1
> 

-- 
Best regards,
Jiqian Chen.