[virtio-dev] [PATCH v3 0/1] Define low power mode for devices

[virtio-dev] [PATCH v3 0/1] Define low power mode for devices

[David Stevens]

The virtio spec currently does not include the concept of device power
management. The lack means that there is no good action drivers can take
when they are requested to put the device into a low power state (e.g.
when a guest is entering a system-wide low power state like S0ix/S3).
Stateless devices can be handled - albeit inefficiently - by resetting
and reinitialzing the device. However, stateful devices cannot support
this situation. This patch defines a low power mode for devices that can
be used in this situation.

Low power mode is mostly defined at the transport layer, and all
device-side power optimizations are optional. This avoids the need for
invasive device-by-device definitions. It also pushes the requirement
onto the device side, to simplify what driver side changes are
necessary to just [1].

I believe this patch may address the virtio-gpu issue which [2] is
trying to address by avoiding the reset altogether when the guest enters
S3.

[1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
[2] https://lore.kernel.org/lkml/20230919114242.2283646-1-Jiqian.Chen@amd.com/

v2 -> v3:
 - Use different words for some concepts to avoid conflicts with other
   parts of the spec.
 - Rewrite various sentences to improve clarity.
v1 -> v2:
 - Define virtio-pci support on top of PCI power management.
 - Add more conformance requirements.

David Stevens (1):
  Define a low power mode for devices

 content.tex       | 61 +++++++++++++++++++++++++++++++++++++++++++++++
 transport-pci.tex |  9 +++++++
 2 files changed, 70 insertions(+)

-- 
2.43.0.rc2.451.g8631bc7472-goog


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v3 1/1] Define a low power mode for devices

[David Stevens]

Define a low power mode for virtio devices where the devices are
expected to maintain their state. This gives drivers an option for power
management besides simply resetting their device. In the virtualization
use case, this allows the guest to be suspended even with stateful
virtio devices like gpu and fs.

Low power mode is primarily defined at the transport layer. The only
part that depends on device-type specific details is whether a given
virtqueue is device driven or driver driven.

This change only defines the transport-specific implementation for
Virtio over PCI.
---
 content.tex       | 61 +++++++++++++++++++++++++++++++++++++++++++++++
 transport-pci.tex |  9 +++++++
 2 files changed, 70 insertions(+)

diff --git a/content.tex b/content.tex
index 0a62dce5f65f..c1ff542bbe2c 100644
--- a/content.tex
+++ b/content.tex
@@ -502,6 +502,67 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Low Power Mode}\label{sec:Basic Facilities of a Virtio Device / Low Power Mode}
+
+Devices and drivers can implement a low power mode: in this mode,
+device state is maintained but the driver does not access the device,
+allowing the device to reduce its power consumption. While in low
+power mode, after a device sends a notification, it generates a wakeup
+signal to inform the driver of the pending notification. While the
+device is in low power mode, the driver only needs to check for
+notifications after receiving a wakeup signal. How a device is put
+into and taken out of low power mode is transport specific, as is how
+wakeup signals are implemented.
+
+To give the driver a chance to respond to external input, the device
+is expected to send a wakeup signal after sending a notification
+resulting from such input. To achieve this, the device is expected to
+send wakeup signals after sending configuration change notifications.
+For used buffer notifications, device queues can be split into two
+categories: device led queues and driver led queues. Device led queues
+are queues where the device holds onto buffers that have been made
+available by the driver for an indeterminate period of time, until
+some device-side event occurs (e.g. event queues, rx queues). Driver
+led queues are queues where the device processes available buffers and
+adds them to used queue within a bounded time (e.g. command queues, tx
+queues). The device is expected to send wakeup signals after sending
+used buffer notifications for device led queues, but not for driver
+led queues.
+
+\devicenormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
+Device / Low Power Mode}
+
+A device in low power mode MUST maintain its state such that all
+driver visible state after exiting low power mode exactly matches
+driver visible state before entering low power mode.
+
+A device in low power mode MUST continue to operate device led queues.
+After sending a used buffer notification for such a queue, the device
+MUST generate a wakeup signal.
+
+A device in low power mode MUST continue to send configuration change
+notifications. After sending a configuration change notification, the
+device MUST generate a wakeup signal.
+
+A device in low power mode SHOULD NOT generate wakeup signal for
+driver driven queues. A device SHOULD stop sending used buffer
+notifications for such queues until after exiting the low power state.
+
+A device in low power mode SHOULD minimize its resource usage,
+although what steps to take are specific to a particular device
+implementation.
+
+\drivernormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
+Device / Low Power Mode}
+
+A driver MUST not interact with a device in low power mode except
+to take the device out of low power mode or to recieve and
+acknowledge a wakeup signal generated by the device, all of which are
+done in a transport-specific way.
+
+A driver MAY use wakeup signals as a trigger to take the device out of
+low power mode, but it MAY also ignore wakeup signals.
+
 \input{admin.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
diff --git a/transport-pci.tex b/transport-pci.tex
index a5c6719ea871..932f3299dbab 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1212,3 +1212,12 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
         re-examine the configuration space to see what changed.
     \end{itemize}
 \end{itemize}
+
+\subsubsection{Low Power Mode}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Low Power Mode}
+
+Devices can implement support for low power mode (see \ref{sec:Basic
+Facilities of a Virtio Device / Low Power Mode}).  Low power mode
+corresponds to the device power management state
+D3\textsubscript{Hot}. A device advertises support for low power mode
+by presenting the PCI Power Management Capability. Wakeup events are
+implemented as PCI Power Management Events (PMEs).
-- 
2.43.0.rc2.451.g8631bc7472-goog


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[Jason Wang]

On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
>
> Define a low power mode for virtio devices where the devices are
> expected to maintain their state. This gives drivers an option for power
> management besides simply resetting their device. In the virtualization
> use case, this allows the guest to be suspended even with stateful
> virtio devices like gpu and fs.
>
> Low power mode is primarily defined at the transport layer. The only
> part that depends on device-type specific details is whether a given
> virtqueue is device driven or driver driven.
>
> This change only defines the transport-specific implementation for
> Virtio over PCI.

A dumb question, if this is only for PCI, can the device just
implement no_soft_reset via PMC?

Thanks


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[David Stevens]

On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> >
> > Define a low power mode for virtio devices where the devices are
> > expected to maintain their state. This gives drivers an option for power
> > management besides simply resetting their device. In the virtualization
> > use case, this allows the guest to be suspended even with stateful
> > virtio devices like gpu and fs.
> >
> > Low power mode is primarily defined at the transport layer. The only
> > part that depends on device-type specific details is whether a given
> > virtqueue is device driven or driver driven.
> >
> > This change only defines the transport-specific implementation for
> > Virtio over PCI.
>
> A dumb question, if this is only for PCI, can the device just
> implement no_soft_reset via PMC?

This is basically No_Soft_Reset, yes. If a change similar to [1] would
be acceptable based only on the No_Soft_Reset bit even with no concept
of power management in the virtio spec, then I personally don't have
any problems with that.

[1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/

-David

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[Jason Wang]

On Tue, Dec 5, 2023 at 6:58 PM David Stevens <stevensd@chromium.org> wrote:
>
> On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> > >
> > > Define a low power mode for virtio devices where the devices are
> > > expected to maintain their state. This gives drivers an option for power
> > > management besides simply resetting their device. In the virtualization
> > > use case, this allows the guest to be suspended even with stateful
> > > virtio devices like gpu and fs.
> > >
> > > Low power mode is primarily defined at the transport layer. The only
> > > part that depends on device-type specific details is whether a given
> > > virtqueue is device driven or driver driven.
> > >
> > > This change only defines the transport-specific implementation for
> > > Virtio over PCI.
> >
> > A dumb question, if this is only for PCI, can the device just
> > implement no_soft_reset via PMC?
>
> This is basically No_Soft_Reset, yes. If a change similar to [1] would
> be acceptable based only on the No_Soft_Reset bit even with no concept
> of power management in the virtio spec, then I personally don't have
> any problems with that.
>
> [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/

So if I read the code correctly, the current Qemu advertises PM but
without no_soft_reset.

So this patch seems to break e.g virtio-net and doesn't fix virtio-GPU.

Thanks

>
> -David
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[Michael S. Tsirkin]

On Wed, Dec 06, 2023 at 05:16:04PM +0800, Jason Wang wrote:
> On Tue, Dec 5, 2023 at 6:58 PM David Stevens <stevensd@chromium.org> wrote:
> >
> > On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
> > >
> > > On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> > > >
> > > > Define a low power mode for virtio devices where the devices are
> > > > expected to maintain their state. This gives drivers an option for power
> > > > management besides simply resetting their device. In the virtualization
> > > > use case, this allows the guest to be suspended even with stateful
> > > > virtio devices like gpu and fs.
> > > >
> > > > Low power mode is primarily defined at the transport layer. The only
> > > > part that depends on device-type specific details is whether a given
> > > > virtqueue is device driven or driver driven.
> > > >
> > > > This change only defines the transport-specific implementation for
> > > > Virtio over PCI.
> > >
> > > A dumb question, if this is only for PCI, can the device just
> > > implement no_soft_reset via PMC?
> >
> > This is basically No_Soft_Reset, yes. If a change similar to [1] would
> > be acceptable based only on the No_Soft_Reset bit even with no concept
> > of power management in the virtio spec, then I personally don't have
> > any problems with that.
> >
> > [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> 
> So if I read the code correctly, the current Qemu advertises PM but
> without no_soft_reset.
> 
> So this patch seems to break e.g virtio-net and doesn't fix virtio-GPU.
> 
> Thanks
> 
> >
> > -David
> >

what is the breakage exactly? Maybe we need a new feature bit?

-- 
MST


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[Jason Wang]

On Wed, Dec 6, 2023 at 6:17 PM Michael S. Tsirkin <mst@redhat.com> wrote:
>
> On Wed, Dec 06, 2023 at 05:16:04PM +0800, Jason Wang wrote:
> > On Tue, Dec 5, 2023 at 6:58 PM David Stevens <stevensd@chromium.org> wrote:
> > >
> > > On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
> > > >
> > > > On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> > > > >
> > > > > Define a low power mode for virtio devices where the devices are
> > > > > expected to maintain their state. This gives drivers an option for power
> > > > > management besides simply resetting their device. In the virtualization
> > > > > use case, this allows the guest to be suspended even with stateful
> > > > > virtio devices like gpu and fs.
> > > > >
> > > > > Low power mode is primarily defined at the transport layer. The only
> > > > > part that depends on device-type specific details is whether a given
> > > > > virtqueue is device driven or driver driven.
> > > > >
> > > > > This change only defines the transport-specific implementation for
> > > > > Virtio over PCI.
> > > >
> > > > A dumb question, if this is only for PCI, can the device just
> > > > implement no_soft_reset via PMC?
> > >
> > > This is basically No_Soft_Reset, yes. If a change similar to [1] would
> > > be acceptable based only on the No_Soft_Reset bit even with no concept
> > > of power management in the virtio spec, then I personally don't have
> > > any problems with that.
> > >
> > > [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> >
> > So if I read the code correctly, the current Qemu advertises PM but
> > without no_soft_reset.
> >
> > So this patch seems to break e.g virtio-net and doesn't fix virtio-GPU.
> >
> > Thanks
> >
> > >
> > > -David
> > >
>
> what is the breakage exactly?

It looks to me we need:

1) Device side, if device can preserve state on d3hot, it may
advertise no_soft_reset
2) Driver side, if driver sees no_soft_reset, it doesn't need to
manually suspend and resume. But in the above patch, it only checks
the existence of PM, it looks to me it needs to check no_soft_reset
instead.

> Maybe we need a new feature bit?

Probably not.

Thanks

>
> --
> MST
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[David Stevens]

On Thu, Dec 7, 2023 at 1:16 PM Jason Wang <jasowang@redhat.com> wrote:
>
> On Wed, Dec 6, 2023 at 6:17 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Wed, Dec 06, 2023 at 05:16:04PM +0800, Jason Wang wrote:
> > > On Tue, Dec 5, 2023 at 6:58 PM David Stevens <stevensd@chromium.org> wrote:
> > > >
> > > > On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
> > > > >
> > > > > On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> > > > > >
> > > > > > Define a low power mode for virtio devices where the devices are
> > > > > > expected to maintain their state. This gives drivers an option for power
> > > > > > management besides simply resetting their device. In the virtualization
> > > > > > use case, this allows the guest to be suspended even with stateful
> > > > > > virtio devices like gpu and fs.
> > > > > >
> > > > > > Low power mode is primarily defined at the transport layer. The only
> > > > > > part that depends on device-type specific details is whether a given
> > > > > > virtqueue is device driven or driver driven.
> > > > > >
> > > > > > This change only defines the transport-specific implementation for
> > > > > > Virtio over PCI.
> > > > >
> > > > > A dumb question, if this is only for PCI, can the device just
> > > > > implement no_soft_reset via PMC?
> > > >
> > > > This is basically No_Soft_Reset, yes. If a change similar to [1] would
> > > > be acceptable based only on the No_Soft_Reset bit even with no concept
> > > > of power management in the virtio spec, then I personally don't have
> > > > any problems with that.
> > > >
> > > > [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> > >
> > > So if I read the code correctly, the current Qemu advertises PM but
> > > without no_soft_reset.
> > >
> > > So this patch seems to break e.g virtio-net and doesn't fix virtio-GPU.
> > >
> > > Thanks
> > >
> > > >
> > > > -David
> > > >
> >
> > what is the breakage exactly?
>
> It looks to me we need:
>
> 1) Device side, if device can preserve state on d3hot, it may
> advertise no_soft_reset
> 2) Driver side, if driver sees no_soft_reset, it doesn't need to
> manually suspend and resume. But in the above patch, it only checks
> the existence of PM, it looks to me it needs to check no_soft_reset
> instead.

Right, my original reply was needlessly vague. Similar to the linked
patch, but based on checking No_Soft_Reset instead of just pm_cap.

That also raises the point that the proposed spec changes are also
insufficient - the PCI section needs to say that setting No_Soft_Reset
is required for low power mode.

-David

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v3 1/1] Define a low power mode for devices

[Jason Wang]

On Thu, Dec 7, 2023 at 12:55 PM David Stevens <stevensd@chromium.org> wrote:
>
> On Thu, Dec 7, 2023 at 1:16 PM Jason Wang <jasowang@redhat.com> wrote:
> >
> > On Wed, Dec 6, 2023 at 6:17 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> > >
> > > On Wed, Dec 06, 2023 at 05:16:04PM +0800, Jason Wang wrote:
> > > > On Tue, Dec 5, 2023 at 6:58 PM David Stevens <stevensd@chromium.org> wrote:
> > > > >
> > > > > On Tue, Dec 5, 2023 at 1:18 PM Jason Wang <jasowang@redhat.com> wrote:
> > > > > >
> > > > > > On Mon, Dec 4, 2023 at 5:41 PM David Stevens <stevensd@chromium.org> wrote:
> > > > > > >
> > > > > > > Define a low power mode for virtio devices where the devices are
> > > > > > > expected to maintain their state. This gives drivers an option for power
> > > > > > > management besides simply resetting their device. In the virtualization
> > > > > > > use case, this allows the guest to be suspended even with stateful
> > > > > > > virtio devices like gpu and fs.
> > > > > > >
> > > > > > > Low power mode is primarily defined at the transport layer. The only
> > > > > > > part that depends on device-type specific details is whether a given
> > > > > > > virtqueue is device driven or driver driven.
> > > > > > >
> > > > > > > This change only defines the transport-specific implementation for
> > > > > > > Virtio over PCI.
> > > > > >
> > > > > > A dumb question, if this is only for PCI, can the device just
> > > > > > implement no_soft_reset via PMC?
> > > > >
> > > > > This is basically No_Soft_Reset, yes. If a change similar to [1] would
> > > > > be acceptable based only on the No_Soft_Reset bit even with no concept
> > > > > of power management in the virtio spec, then I personally don't have
> > > > > any problems with that.
> > > > >
> > > > > [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> > > >
> > > > So if I read the code correctly, the current Qemu advertises PM but
> > > > without no_soft_reset.
> > > >
> > > > So this patch seems to break e.g virtio-net and doesn't fix virtio-GPU.
> > > >
> > > > Thanks
> > > >
> > > > >
> > > > > -David
> > > > >
> > >
> > > what is the breakage exactly?
> >
> > It looks to me we need:
> >
> > 1) Device side, if device can preserve state on d3hot, it may
> > advertise no_soft_reset
> > 2) Driver side, if driver sees no_soft_reset, it doesn't need to
> > manually suspend and resume. But in the above patch, it only checks
> > the existence of PM, it looks to me it needs to check no_soft_reset
> > instead.
>
> Right, my original reply was needlessly vague. Similar to the linked
> patch, but based on checking No_Soft_Reset instead of just pm_cap.

Right, so I think I'm fine if it works like this.

>
> That also raises the point that the proposed spec changes are also
> insufficient - the PCI section needs to say that setting No_Soft_Reset
> is required for low power mode.

Yes, but I think we probably don't need the spec patch.

Thanks

>
> -David
>


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org