[virtio-dev] [PATCH v2 0/1] Define low power mode for devices

[virtio-dev] [PATCH v2 0/1] Define low power mode for devices

[David Stevens]

The virtio spec currently does not include the concept of device power
management. The lack means that there is no good action drivers can take
when they are requested to put the device into a low power state (e.g.
when a guest is entering a system-wide low power state like S0ix/S3).
Stateless devices can be handled - albeit inefficiently - by resetting
and reinitialzing the device. However, stateful devices cannot support
this situation. This patch defines a low power mode for devices that can
be used in this situation.

Low power mode is mostly defined at the transport layer, and all
device-side power optimizations are optional. This avoids the need for
invasive device-by-device definitions. It also pushes the requirement
onto the device side, to simplify what driver side changes are
necessary to just [1].

I believe this patch may address the virtio-gpu issue which [2] is
trying to address by avoiding the reset altogether when the guest enters
S3.

[1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
[2] https://lore.kernel.org/lkml/20230919114242.2283646-1-Jiqian.Chen@amd.com/

v1 -> v2:
 - Define virtio-pci support on top of PCI power management.
 - Add more conformance requirements.

David Stevens (1):
  Define a low power mode for devices

 content.tex       | 45 +++++++++++++++++++++++++++++++++++++++++++++
 transport-pci.tex |  7 +++++++
 2 files changed, 52 insertions(+)

-- 
2.42.0.869.gea05f2083d-goog


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] [PATCH v2 1/1] Define a low power mode for devices

[David Stevens]

Define a low power mode for virtio devices where the devices are
expected to maintain their state. This gives drivers an option for power
management besides simply resetting their device. In the virtualization
use case, this allows the guest to be suspended even with stateful
virtio devices like gpu and fs.

Low power mode is primarily defined at the transport layer. The only
part that depends on device-type specific details is whether a given
virtqueue is device driven or driver driven.

This change only defines the transport-specific implementation for
Virtio over PCI.
---
 content.tex       | 45 +++++++++++++++++++++++++++++++++++++++++++++
 transport-pci.tex |  7 +++++++
 2 files changed, 52 insertions(+)

diff --git a/content.tex b/content.tex
index 0a62dce5f65f..7016778c38d6 100644
--- a/content.tex
+++ b/content.tex
@@ -502,6 +502,51 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
 types. It is RECOMMENDED that devices generate version 4
 UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
 
+\section{Low Power Mode}\label{sec:Basic Facilities of a Virtio Device / Low Power Mode}
+
+Low power mode is a state a driver can put its device into to try to
+reduce the resource consumption of the device. While in low power
+mode, a device can generate wakeup events to inform its driver about
+device driven events. How a device is put into and taken out of low
+power mode is transport specific, as is how wakeup events are
+implemented.
+
+\devicenormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
+Device / Low Power Mode}
+
+A device in low power mode MUST maintain its state such that all
+driver visible state after exiting low power mode exactly matches
+driver visible state before entering low power mode.
+
+A device in low power mode MUST continue to operate device driven
+queues. Device driven queues are queues where the driver provides
+buffers to the device that the device holds onto for an indeterminate
+time until some device-side event occurs (e.g. event queues, rx
+queues). When sending a used buffer notification for such a queue, the
+device MUST generate a wakeup event.
+
+A device in low power mode MUST continue to send configuration change
+notifications. When sending a configuration change notification, the
+device MUST generate a wakeup event.
+
+A device in low power mode SHOULD NOT generate wakeup events for
+driver driven queues. A device SHOULD stop sending used buffer
+notifications for such queues until after exiting the low power state.
+
+A device in low power mode SHOULD minimize its resource usage,
+although what steps to take are specific to a particular device
+implementation.
+
+\drivernormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
+Device / Low Power Mode}
+
+A driver MUST not interact with a device in low power mode except
+to take the device out of low power mode or to handle wakeup events
+generated by the device.
+
+A driver MAY use wakeup events as a trigger to take the device out of
+low power mode, but it MAY also ignore wakeup events.
+
 \input{admin.tex}
 
 \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
diff --git a/transport-pci.tex b/transport-pci.tex
index a5c6719ea871..6694e0f72d46 100644
--- a/transport-pci.tex
+++ b/transport-pci.tex
@@ -1212,3 +1212,10 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
         re-examine the configuration space to see what changed.
     \end{itemize}
 \end{itemize}
+
+\subsubsection{Low Power Mode}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Low Power Mode}
+
+Low power mode corresponds to the device power management state
+D3\textsubscript{Hot}. A device advertises support for low power mode
+by presenting the PCI Power Management Capability. Wakeup events are
+implemented as PCI Power Management Events (PMEs).
-- 
2.42.0.869.gea05f2083d-goog


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [PATCH v2 0/1] Define low power mode for devices

[David Stevens]

On Mon, Nov 13, 2023 at 3:20 PM David Stevens <stevensd@chromium.org> wrote:
>
> The virtio spec currently does not include the concept of device power
> management. The lack means that there is no good action drivers can take
> when they are requested to put the device into a low power state (e.g.
> when a guest is entering a system-wide low power state like S0ix/S3).
> Stateless devices can be handled - albeit inefficiently - by resetting
> and reinitialzing the device. However, stateful devices cannot support
> this situation. This patch defines a low power mode for devices that can
> be used in this situation.
>
> Low power mode is mostly defined at the transport layer, and all
> device-side power optimizations are optional. This avoids the need for
> invasive device-by-device definitions. It also pushes the requirement
> onto the device side, to simplify what driver side changes are
> necessary to just [1].
>
> I believe this patch may address the virtio-gpu issue which [2] is
> trying to address by avoiding the reset altogether when the guest enters
> S3.
>
> [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> [2] https://lore.kernel.org/lkml/20230919114242.2283646-1-Jiqian.Chen@amd.com/
>
> v1 -> v2:
>  - Define virtio-pci support on top of PCI power management.
>  - Add more conformance requirements.

Friendly ping, and directly CC'ing reviewers.

-David

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v2 1/1] Define a low power mode for devices

[Michael S. Tsirkin]

On Mon, Nov 13, 2023 at 03:19:50PM +0900, David Stevens wrote:
> Define a low power mode for virtio devices where the devices are
> expected to maintain their state. This gives drivers an option for power
> management besides simply resetting their device. In the virtualization
> use case, this allows the guest to be suspended even with stateful
> virtio devices like gpu and fs.
> 
> Low power mode is primarily defined at the transport layer. The only
> part that depends on device-type specific details is whether a given
> virtqueue is device driven or driver driven.
> 
> This change only defines the transport-specific implementation for
> Virtio over PCI.
> ---
>  content.tex       | 45 +++++++++++++++++++++++++++++++++++++++++++++
>  transport-pci.tex |  7 +++++++
>  2 files changed, 52 insertions(+)
> 
> diff --git a/content.tex b/content.tex
> index 0a62dce5f65f..7016778c38d6 100644
> --- a/content.tex
> +++ b/content.tex
> @@ -502,6 +502,51 @@ \section{Exporting Objects}\label{sec:Basic Facilities of a Virtio Device / Expo
>  types. It is RECOMMENDED that devices generate version 4
>  UUIDs as specified by \hyperref[intro:rfc4122]{[RFC4122]}.
>  
> +\section{Low Power Mode}\label{sec:Basic Facilities of a Virtio Device / Low Power Mode}




> +
> +Low power mode is a state a driver can put its device into to try to
> +reduce the resource consumption of the device.

This sounds somewhat vague and the grammar is convoluted. E.g. who
tries what?

How about something like:
	Devices and drivers can implement a low power mode: in this mode device
	state is maintained but driver does not access the device, allowing the
	device to reduce the power consumption.


> While in low power
> +mode, a device can generate wakeup events to inform its driver about
> +device driven events.

using the term events here is confusing because it already
has a meaning in the spec solely related to ring (discussed
in the context of device event suppression).
And "driven" is too close to "driver" and
confuses me.

I think these are
really vq notifications and config change events right?
Maybe just say so instead of coming up with new terms.


> How a device is put into and taken out of low
> +power mode is transport specific, as is how wakeup events are
> +implemented.
> +

I would move the definition of device driven queues here and
add the definition of driver driven queues.

And add a high level explanation of how device and driver
interact (repeating a bit what normative sections say,
this duplication is normal).


> +\devicenormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
> +Device / Low Power Mode}
> +
> +A device in low power mode MUST maintain its state such that all
> +driver visible state after exiting low power mode exactly matches
> +driver visible state before entering low power mode.
> +
> +A device in low power mode MUST continue to operate device driven
> +queues. Device driven queues are queues where the driver provides

makes available is the term we use

> +buffers to the device that the device holds onto for an indeterminate
> +time until some device-side event occurs (e.g. event queues, rx
> +queues). When sending a used buffer notification for such a queue, the
> +device MUST generate a wakeup event.

is it only when sending notification? not when buffers are used?
also what does "when" mean here? after sending a notification?

> +
> +A device in low power mode MUST continue to send configuration change
> +notifications. When sending a configuration change notification, the
> +device MUST generate a wakeup event.
> +
> +A device in low power mode SHOULD NOT generate wakeup events for
> +driver driven queues. A device SHOULD stop sending used buffer
> +notifications for such queues until after exiting the low power state.
> +
> +A device in low power mode SHOULD minimize its resource usage,
> +although what steps to take are specific to a particular device
> +implementation.
> +
> +\drivernormative{\subsection}{Low Power Mode}{Basic Facilities of a Virtio
> +Device / Low Power Mode}
> +
> +A driver MUST not interact with a device in low power mode except
> +to take the device out of low power mode

i get this
in a transport specific manner, yes?

> or to handle wakeup events
> +generated by the device.

i don't get this. what does "handle" mean here?


> +
> +A driver MAY use wakeup events as a trigger to take the device out of
> +low power mode, but it MAY also ignore wakeup events.
> +
>  \input{admin.tex}
>  
>  \chapter{General Initialization And Device Operation}\label{sec:General Initialization And Device Operation}
> diff --git a/transport-pci.tex b/transport-pci.tex
> index a5c6719ea871..6694e0f72d46 100644
> --- a/transport-pci.tex
> +++ b/transport-pci.tex
> @@ -1212,3 +1212,10 @@ \subsubsection{Driver Handling Interrupts}\label{sec:Virtio Transport Options /
>          re-examine the configuration space to see what changed.
>      \end{itemize}
>  \end{itemize}
> +
> +\subsubsection{Low Power Mode}\label{sec:Virtio Transport Options / Virtio Over PCI Bus / PCI-specific Initialization And Device Operation / Low Power Mode}
> +

a bit of introduction here can't hurt too - this is a cmpletely separate
section.
e.g. devices can implement support for low power mode, see
(link to generic description).

> +Low power mode corresponds to the device power management state
> +D3\textsubscript{Hot}. A device advertises support for low power mode
> +by presenting the PCI Power Management Capability. Wakeup events are
> +implemented as PCI Power Management Events (PMEs).
> -- 
> 2.42.0.869.gea05f2083d-goog
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org

[virtio-dev] Re: [virtio-comment] [PATCH v2 0/1] Define low power mode for devices

[Michael S. Tsirkin]

On Mon, Nov 13, 2023 at 03:19:49PM +0900, David Stevens wrote:
> The virtio spec currently does not include the concept of device power
> management. The lack means that there is no good action drivers can take
> when they are requested to put the device into a low power state (e.g.
> when a guest is entering a system-wide low power state like S0ix/S3).
> Stateless devices can be handled - albeit inefficiently - by resetting
> and reinitialzing the device. However, stateful devices cannot support
> this situation. This patch defines a low power mode for devices that can
> be used in this situation.
> 
> Low power mode is mostly defined at the transport layer, and all
> device-side power optimizations are optional. This avoids the need for
> invasive device-by-device definitions. It also pushes the requirement
> onto the device side, to simplify what driver side changes are
> necessary to just [1].

Great idea.
This couldn't have come at a better time, too.
Please do remember to CC reviewers directly in the future, though.


> I believe this patch may address the virtio-gpu issue which [2] is
> trying to address by avoiding the reset altogether when the guest enters
> S3.
> 
> [1] https://lore.kernel.org/lkml/20231113055138.117392-1-stevensd@chromium.org/
> [2] https://lore.kernel.org/lkml/20230919114242.2283646-1-Jiqian.Chen@amd.com/
> 
> v1 -> v2:
>  - Define virtio-pci support on top of PCI power management.
>  - Add more conformance requirements.
> 
> David Stevens (1):
>   Define a low power mode for devices
> 
>  content.tex       | 45 +++++++++++++++++++++++++++++++++++++++++++++
>  transport-pci.tex |  7 +++++++
>  2 files changed, 52 insertions(+)
> 
> -- 
> 2.42.0.869.gea05f2083d-goog
> 
> 
> This publicly archived list offers a means to provide input to the
> OASIS Virtual I/O Device (VIRTIO) TC.
> 
> In order to verify user consent to the Feedback License terms and
> to minimize spam in the list archive, subscription is required
> before posting.
> 
> Subscribe: virtio-comment-subscribe@lists.oasis-open.org
> Unsubscribe: virtio-comment-unsubscribe@lists.oasis-open.org
> List help: virtio-comment-help@lists.oasis-open.org
> List archive: https://lists.oasis-open.org/archives/virtio-comment/
> Feedback License: https://www.oasis-open.org/who/ipr/feedback_license.pdf
> List Guidelines: https://www.oasis-open.org/policies-guidelines/mailing-lists
> Committee: https://www.oasis-open.org/committees/virtio/
> Join OASIS: https://www.oasis-open.org/join/
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org