* [PATCH v10 0/1] Virtio video device specification @ 2026-02-13 8:53 Alexander Gordeev 2026-02-13 8:53 ` [PATCH v10 1/1] virtio-video: Add virtio " Alexander Gordeev 0 siblings, 1 reply; 9+ messages in thread From: Alexander Gordeev @ 2026-02-13 8:53 UTC (permalink / raw) To: virtio-comment Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media Hi, This is the 10th version of virtio-video device specification patch. There are some major changes based on internal review. I think it is good enough conceptually, but may need few minor changes, when the driver gets fully updated. It took a long time. We were very busy working on the implementation. I hope to release the updated driver soon. Please review. Changelog v9 -> v10: 1. Combine both encoder and decoder in a single device. There is HW out there that can do both encoding and decoding at the same time (e.g. Qualcomm). Having both functions in a single device can save resources. 2. New device ID. Point 1 required the new device ID. I've already sent the patch to reserve the new ID and asked for a vote. I took the liberty to already put this new ID here. I'll send a new patch if any change would be necessary. 3. Reinvented the internal queues, added per stream main queue. This is based on internal review. Having too many virtqueues may use more HW resources. Still there is a change in the scheme. Now each stream has three internal queues: mainqX, inputqX, outputqX. 4. Now all stream commands return the results as async responses over eventq and nothing is returned on the commandq anymore. This way the race handling both commandq's and eventq's used queues is completely avoided. Now the eventq is the single source of truth about the device's state. 5. SET_PARAMS returns only changed parameters, GET_CONFIG returns everything. 6. SET_PARAMS/dynamic parameter change now blocks outputqX to allow driver finish output format negotiation and reallocate buffers if necessary. This fixes a potential race in the implementation. This is somewhat similar to what V4L2_DEC_CMD_START is for. 7. Added VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER feature flag for compatibility with V4L2's V4L2_BUF_FLAG_LAST. 8. The spec size went down from 19 to 18 pages. Alexander Gordeev (1): virtio-video: Add virtio video device specification conformance.tex | 12 +- content.tex | 1 + device-types/video/description.tex | 1592 +++++++++++++++++++++ device-types/video/device-conformance.tex | 22 + device-types/video/driver-conformance.tex | 20 + introduction.tex | 21 + 6 files changed, 1664 insertions(+), 4 deletions(-) create mode 100644 device-types/video/description.tex create mode 100644 device-types/video/device-conformance.tex create mode 100644 device-types/video/driver-conformance.tex -- 2.43.0 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-13 8:53 [PATCH v10 0/1] Virtio video device specification Alexander Gordeev @ 2026-02-13 8:53 ` Alexander Gordeev 2026-02-15 10:49 ` Parav Pandit 0 siblings, 1 reply; 9+ messages in thread From: Alexander Gordeev @ 2026-02-13 8:53 UTC (permalink / raw) To: virtio-comment Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media From: Alexander Gordeev <Alexander.Gordeev@opensynergy.com> Add the specification of the video decoder and encoder device, which can be used to provide host-accelerated video operations to the guest. Signed-off-by: Alexander Gordeev <alexander.gordeev@opensynergy.com> --- conformance.tex | 12 +- content.tex | 1 + device-types/video/description.tex | 1592 +++++++++++++++++++++ device-types/video/device-conformance.tex | 22 + device-types/video/driver-conformance.tex | 20 + introduction.tex | 21 + 6 files changed, 1664 insertions(+), 4 deletions(-) create mode 100644 device-types/video/description.tex create mode 100644 device-types/video/device-conformance.tex create mode 100644 device-types/video/driver-conformance.tex diff --git a/conformance.tex b/conformance.tex index 9af31e2..f34e600 100644 --- a/conformance.tex +++ b/conformance.tex @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize} @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. \end{itemize} @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/spi/driver-conformance.tex} \input{device-types/media/driver-conformance.tex} \input{device-types/rtc/driver-conformance.tex} +\input{device-types/video/driver-conformance.tex} \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} \input{device-types/spi/device-conformance.tex} \input{device-types/media/device-conformance.tex} \input{device-types/rtc/device-conformance.tex} +\input{device-types/video/device-conformance.tex} \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance} A conformant implementation MUST be either transitional or diff --git a/content.tex b/content.tex index 5de811f..0c13f68 100644 --- a/content.tex +++ b/content.tex @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} \input{device-types/spi/description.tex} \input{device-types/media/description.tex} \input{device-types/rtc/description.tex} +\input{device-types/video/description.tex} \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} diff --git a/device-types/video/description.tex b/device-types/video/description.tex new file mode 100644 index 0000000..8945e26 --- /dev/null +++ b/device-types/video/description.tex @@ -0,0 +1,1592 @@ +\section{Video Device} +\label{sec:Device Types / Video Device} + +The virtio video device provides support for host-accelerated video encoding +and decoding. + +\subsection{Device ID} +\label{sec:Device Types / Video Device / Device ID} + +50 + +\subsection{Virtqueues} +\label{sec:Device Types / Video Device / Virtqueues} + +\begin{description} + \item[0] + commandq - driver commands + \item[1] + eventq - device async responses to commands and standalone device events +\end{description} + +\subsection{Feature bits} +\label{sec:Device Types / Video Device / Feature bits} + +\begin{description} + \item[VIRTIO_VIDEO_F_ENCODER (0)] + The device can encode video. + \item[VIRTIO_VIDEO_F_DECODER (1)] + The device can decode video. + % Use-case: the device can support both encoding and decoding, so having both + % here can save resources. + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] + Guest pages can be used as the backing memory of resources. + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] + The device can use non-contiguous guest memory as the backing memory of + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also + set. + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] + Objects exported by another virtio device can be used as the backing memory + of resources. + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] + The device releases an extra empty output buffer after a drain or DPC so that + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. +\end{description} + +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} + +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. + +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would +mean that no memory can be used at all for resources. + +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. + +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} + +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and +VIRTIO_VIDEO_F_DECODER features. + +The driver MUST negotiate at least one of the +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT +features. + +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically +contiguous memory for all the buffers it allocates. + +\subsection{Device configuration layout} +\label{sec:Device Types / Video Device / Device configuration layout} + +The video device configuration space uses the following layout: + +\begin{lstlisting} +struct virtio_video_config { + le32 max_streams; + le32 caps_length; +}; +\end{lstlisting} + +\begin{description} + \item[\field{max_streams}] + is the maximum number of concurrent streams the device supports. + \item[\field{caps_length}] + is the minimum length in bytes that a device-writable buffer must have + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. +\end{description} + +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} + +\field{max_streams} MUST be positive. + +\field{caps_length} MUST be set to the response size of +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. + +\subsection{Device Initialization} +\label{sec:Device Types / Video Device / Device Initialization} +\begin{enumerate} + \item + The driver reads the feature bits and negotiates the features it needs. + \item + The driver sets up the commandq and the eventq. + \item + The driver reads the \field{caps_length} field of the configuration + space, prepares a buffer of at least that size and sends the buffer on the + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. + \item + The device sends a response over commandq to + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the + command. + \item + The driver receives the response from the device, and parses its capabilities. +\end{enumerate} + +\subsection{Device Operation} +\label{sec:Device Types / Video Device / Device Operation} + +The device supports opening and operating a number of parallel streams up to +\field{max_streams}. Each stream has three internal device queues: mainqX, +inputqX and outputqX, where X is the stream id. Each stream command has a +field that dispatches the command to the specific internal queue. + +% Use-case: there might be different real-time requirements for different +% streams, so more virtqueues can be added in the future if necessary. +% The internal queues don't change, the data formats don't change, only the +% mapping of streams/internal queues to particular virtqueues changes. + +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for +format negotiation. + +The inputqX and outputqX are used to queue input or output resources using +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete +processing of all queued input resources and make the resulting output +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. + +All the stream commands start async operations, return the results +using async responses over eventq. +The eventq is used by the device to send the device's async responses to +stream commands and the device's standalone events. + +% This way eventq becomes the single source of truth about the device's state, +% the driver doesn't have to tediously synchronize commandq's and eventq's +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF +% and DQEVENT). One more benefit is that commandq is processed fast and +% strictly in order, so commandq descriptors exhaustion should never happen in +% practice. + +Parameters allow the driver to configure the stream including setting up the +resources. Available parameters depend on the device type, see +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. + +A resource is a set of memory buffers that contain a unit of data that +the device can process or produce. Most resources have only one buffer, +raw frames using a multi-planar format can have several. +Input resources are filled by the driver with compressed (coded) video data +for a decoder and raw frames for an encoder, output resources are filled by +the device as the result of processing the input resources with decoded raw +frames for a decoder and compressed (encoded) data for an encoder. +Resources from inputqX and outputqX are consumed independently, not in pairs. +One input resource can result in zero to many produced output resources. +A decoder device dequeues the output decoded frames in presentation order. +An encoder device dequeues the output decoded frames in decoding order. +The driver can reuse a queued resource after receiving a corresponding async +response. Dequeued output resources can still be used by the device as +reference frames, so the driver can't write to them. + +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued +% output resource is not used by the device anymore and therefore becomes +% writeable? + +The device can detect standalone stream-related events: errors and dynamic +parameters changes that require intervention from the driver (e.g. +reallocating backing memory of output resources to fit the new parameters). +The events are signalled on the eventq, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. + +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} + +The device MUST set to zero all unused, disabled or padding bits in its +responses. + +\subsubsection{Device Operation: Command Virtqueue} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} + +This section lists the commands that can be sent by the driver to commandq. + +Different structures are used for each command and response. A command +structure starts with the requested command code, defined as follows: + +\begin{lstlisting} +/* Device */ +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 + +/* Stream */ +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 +\end{lstlisting} + +Stream commands start with a header: + +\begin{lstlisting} +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 + +struct virtio_video_stream_cmd_header { + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ + le32 stream_id; + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ + le32 async_response_cookie; +}; +\end{lstlisting} + +\begin{description} + \item[\field{async_response_cookie}] + is an async response cookie provided by the driver, that allows + to relate an async response to the previously submitted command. +\end{description} + +\subsubsection{Device Operation: Event Virtqueue} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} + +The eventq is used by the device to send async responses to commands queued +by the driver on commandq and standalone events. Stream errors and dynamic +parameters changes are caused by changes in the device's state, not by +commands, still they are delivered as responses to implicit +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, +respectively. + +Events start with a header: + +\begin{lstlisting} +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) + +struct virtio_video_event_header { + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ + le32 stream_id; + le32 async_response_cookie; + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{event_type}] + is the type of the event. + \item[\field{stream_id}] + is the ID of a valid stream. + \item[\field{async_response_cookie}] + is an async response cookie provided by the driver, that allows + to relate the event to a previously submitted command. + \item[\field{event_flags}] + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags + + \begin{description} + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] + is set if the command finished with an error due to an + invalid argument or for other reasons. + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] + is set for standalone events, see + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] + is set if the command has been canceled by another + command, that has higher priority. Doesn't make sense + for standalone events. + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] + is set if the command triggered a block on the + outputqX to allow output format negotiation. + When the negotiation is finished the block has to be + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK + command, see + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. + \end{description} +\end{description} + +The particular data structure representing the event is selected according to +the \field{event_type}. + +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} + +The driver MUST at any time have at least one descriptor with a used +buffer large enough to contain a \field{struct virtio_video_event} +queued on the eventq. + +The driver MUST NOT put device-readable descriptors into the eventq. + +The driver MUST account for the fact that the async responses to commands +might come out-of-order (i.e. after other commands sent to the device), +and that some of them can be cancelled. + +The driver SHOULD wait for an async response of command A, that caused +cancellation of command B, before queueing the command B again. + +\subsubsection{Device Operation: TLV format} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} + +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS +commands represent device capabilities and corresponding device parameters +in the form of TLV (Type-Length-Value): + +\begin{lstlisting} +struct virtio_video_tlv { + le32 type; + le32 length; + u8 value[length]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{type}] + specifies the type of data in \field{value}. + \item[\field{length}] + specifies the \field{value} size in bytes aligned to 4 bytes. + \item[\field{value}] + contains the data according to the type. +\end{description} + +The following TLV types are defined: + +\begin{lstlisting} +#define VIRTIO_VIDEO_TLV_CODED_SET 1 +#define VIRTIO_VIDEO_TLV_RAW_SET 2 +#define VIRTIO_VIDEO_TLV_LINK 3 +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 +#define VIRTIO_VIDEO_TLV_CROP 10 +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 +\end{lstlisting} + +Some TLVs are used only as containers for sequences of nested TLVs: + +\begin{description} + \item[\field{VIRTIO_VIDEO_TLV_CODED_SET}] + groups various capabilities or parameters related to a particular coded + format. + \item[\field{VIRTIO_VIDEO_TLV_RAW_SET}] + groups various capabilities or parameters related to a particular raw + format. + \item[\field{VIRTIO_VIDEO_TLV_V4L2_CONTROLS}] + contains V4L2 controls represented in the TLV format. Within this container + only selected V4L2 control identifiers (V4L2_CID_*) are allowed to be used + in the TLV \field{type} field, see + \ref{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS}. +\end{description} + +TLV with type VIRTIO_VIDEO_TLV_LINK is a special one used to define relations +between sets of capabilities, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. + +For each of the remaining TLV types two different contained data formats are +defined: one for the capabilities and one for the specific parameter values. + +\field{struct virtio_video_range} is used to represent a range of values in +some TLVs: + +\begin{lstlisting} +struct virtio_video_range { + le32 min; + le32 max; + le32 step; + u8 padding[4]; +}; +\end{lstlisting} + +An integer \(x\) is within the range \field{r} if +\(\field{r.min} \le x \le \field{r.max}\) holds and \(x\) equals to +\((\field{min} + \field{step} * n)\) for some integer \(n\). + +\devicenormative{\paragraph}{Device Operation: TLV format}{Device Types / Video Device / Device Operation / Device Operation: TLV format} + +\field{min}, \field{step} and \field{max} MUST be positive. + +\field{min} MUST be less then or equal to \field{max} within the same range. + +\subsubsection{Device Operation: Device Commands} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands} + +This command allows retrieving the device capabilities. + +\paragraph{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} + +Retrieve device capabilities for all available stream parameters (for example, +the range of values). + +The driver sends this command with +\field{struct virtio_video_device_query_caps}: + +\begin{lstlisting} +struct virtio_video_device_query_caps { + le32 type; /* VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS */ +}; +\end{lstlisting} + +The device responds with +\field{struct virtio_video_device_query_caps_resp}: + +\begin{lstlisting} + +#define VIRTIO_VIDEO_RESULT_OK 0 +#define VIRTIO_VIDEO_RESULT_ERROR 1 + +struct virtio_video_device_query_caps_resp { + le32 result; /* VIRTIO_VIDEO_RESULT_* */ + u8 padding[4]; + /** + * Followed by a sequence of TLVs up to caps_length + * counted in bytes from the beginning of the struct. + */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{result}] + is + + \begin{description} + \item[VIRTIO_VIDEO_RESULT_OK] + if the command succeeded, + \item[VIRTIO_VIDEO_RESULT_ERROR] + if the descriptor was smaller than the defined \field{caps_length} in + the video device configuration. + \end{description} +\end{description} + +The sequence of TLVs consists of TLV containers VIRTIO_VIDEO_TLV_CODED_SET +and VIRTIO_VIDEO_TLV_RAW_SET defining sets of possible coded, or respectively +raw formats, including the corresponding parameters (e.g. profiles, levels +or format modifiers, resolutions), that are supported by the device. For the +details see +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. +The last TLVs in the sequence with type VIRTIO_VIDEO_TLV_LINK establish +relations between the two groups. If there is a link, then the device supports +decoding from the specified coded set to the specified raw set, or encoding in +the opposite direction. The value format is defined as follows: + +\begin{lstlisting} +#define VIRTIO_VIDEO_STREAM_TYPE_DECODER 0 +#define VIRTIO_VIDEO_STREAM_TYPE_ENCODER 1 + +struct virtio_video_tlv_link { + le32 stream_type; /* One of VIRTIO_VIDEO_STREAM_TYPE_* */ + u8 padding[4]; + le64 links[n_coded * (n_raw + 63) / 64]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{stream_type}] + is the type of the stream that is supported and negotiated using + the respective feature flags. + \item[\field{n_coded}] + is the number of VIRTIO_VIDEO_TLV_CODED_SET containers. + \item[\field{n_raw}] + is the number of VIRTIO_VIDEO_TLV_RAW_SET containers. + \item[\field{links}] + is a bitset establishing links between coded and raw sets. For \field{i}-th + coded and \field{j}-th raw sets counted from zero bit \field{(j \% 64)} in + \field{links[i * (n_raw + 63) / 64 + j / 64]} defines the link if it is set. +\end{description} + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS}{Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} + +Response to the command MUST be written by the device in the first +device-writable descriptor of the descriptor chain from which the command +came. + +The device MUST include into the response at least one +VIRTIO_VIDEO_TLV_CODED_SET and at least one VIRTIO_VIDEO_TLV_RAW_SET TLV +containers, and one VIRTIO_VIDEO_TLV_LINK TLV defining links between the coded +and raw sets for each VIRTIO_VIDEO_F_ENCODER and VIRTIO_VIDEO_F_DECODER +feature flag that is negotiated. The VIRTIO_VIDEO_TLV_LINK TLVs MUST be the +last in the sequence. + +Each VIRTIO_VIDEO_TLV_CODED_SET and VIRTIO_VIDEO_TLV_RAW_SET MUST take part +in at least one defined link in one of the VIRTIO_VIDEO_TLV_LINK TLVs. + +Each VIRTIO_VIDEO_TLV_CODED_SET MUST contain exactly one +VIRTIO_VIDEO_TLV_CODED_FORMAT TLV, exactly one +VIRTIO_VIDEO_TLV_CODED_RESOURCES TLV and at most one TLV of other types. + +Each VIRTIO_VIDEO_TLV_RAW_SET MUST contain exactly one +VIRTIO_VIDEO_TLV_RAW_FORMAT TLV, exactly one VIRTIO_VIDEO_TLV_RAW_RESOURCES +TLV and at most one TLV of other types. + +VIRTIO_VIDEO_TLV_RAW_SET containers SHOULD be ordered according to raw format +preferences of the device from preferred to not preferred ones. + +The total size of the response MUST be equal to \field{caps_length} +bytes, as reported by the device configuration. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS}{Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} + +Descriptor chains sent to the commandq by the driver MUST include at least one +device-writable descriptor. The combined length of all such descriptors MUST +be at least \field{caps_length} bytes. + +\subsubsection{Device Operation: Stream commands} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands} + +Stream commands allow to open, close and control the flow of a stream. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_OPEN} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} + +Open a video stream. + +The driver sends this command with \field{struct virtio_video_stream_open}: + +\begin{lstlisting} +struct virtio_video_stream_open { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_OPEN -> mainqX */ + le32 stream_type; /* One of VIRTIO_VIDEO_STREAM_TYPE_* */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to be opened. + \item[\field{stream_type}] + is the type of the stream to be opened. +\end{description} + +The device begins an async OPEN operation. When the operation is completed the +device sends the VIRTIO_VIDEO_CMD_STREAM_OPEN async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_OPEN}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} + +The device MUST ensure that the \field{stream_id} is within limits and that +the corresponding stream is not open. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_OPEN}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} + +VIRTIO_VIDEO_CMD_STREAM_OPEN MUST be sent to mainqX. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_CLOSE} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} + +% Use-case: the guest user-space app closes the file descriptor. Cleanup fast. + +Close a video stream. Any activity on the stream is halted and all resources +are released by the time the async response is received by the driver. + +The driver sends this command with +\field{struct virtio_video_stream_close}: + +\begin{lstlisting} +struct virtio_video_stream_close { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_CLOSE -> mainqX */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{stream_id}] + is the ID of the stream to be closed. +\end{description} + +The device begins an async CLOSE operation, that consists of RESET operations +on inputqX and outputqX, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET}, +detaching all the resources, releasing the internal stream queues and any +corresponding internal resources. When the CLOSE operation is completed the +device sends the VIRTIO_VIDEO_CMD_STREAM_CLOSE async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. +The same async response can also come after an unrecoverable stream error, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Error Event}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_CLOSE}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} + +After VIRTIO_VIDEO_CMD_STREAM_CLOSE is queued, the device MUST send an async +response with VIRTIO_VIDEO_EVENT_FLAG_ERROR flag set to any subsequently +queued command with this stream ID except VIRTIO_VIDEO_CMD_STREAM_OPEN. + +The CLOSE operation MUST NOT be canceled. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_CLOSE}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} + +\field{stream_id} MUST be set to a valid stream ID of an open stream. + +VIRTIO_VIDEO_CMD_STREAM_CLOSE MUST be sent to mainqX. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} + +Set selected parameters of inputqX or outputqX of a given stream and receive +back the actual values of the set parameters. Coded parameters within a +VIRTIO_VIDEO_TLV_CODED_SET container belong to inputqX of a decoder stream and +to outputqX of an encoder stream. Raw parameters within a +VIRTIO_VIDEO_TLV_RAW_SET container belong to outputqX of a decoder stream and +to inputqX of an encoder stream. + +The driver sends this command with +\field{struct virtio_video_stream_set_params}: + +\begin{lstlisting} +struct virtio_video_stream_get_set_params { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS */ + /** + * Followed by a single VIRTIO_VIDEO_TLV_CODED_SET or + * VIRTIO_VIDEO_TLV_RAW_SET TLV container with the parameters. + */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to set parameters for. +\end{description} + +The command can be queued to: + +\begin{itemize} + \item[\field{inputqX}] + The parameters are set after all commands previously queued on the + inputqX are processed. + \item[\field{mainqX}] + The parameters are set without waiting for other commands queued + on inputqX to be processed. +\end{itemize} + +% Use-case: for the decoder, resolution can be set manually by the driver +% (useful for codecs that do not embed this information, like MPEG-2). +% The processing sequence should look similar to the dynamic parameters +% change case. +% Use-case: V4L2's request API. + +Any changes to formats, decreasing resource numbers or changes to resources +in use trigger the following sequence: +\begin{enumerate} + \item + an implicit DRAIN operation, see + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN}; + \item + only for changes to outputqX parameters: outputqX is blocked until + a VIRTIO_VIDEO_CMD_STREAM_UNBLOCK command is received, see + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. + When the block is active the decoder or encoder can't get anything + from outputqX. +\end{enumerate} + +% Use-case: dynamic resolution changes on inter frames in VP9. +% See https://lore.kernel.org/linux-media/20240314153226.197445-1-benjamin.gaignard@collabora.com/ +% Because of this buffers can't be detached/deallocated/reset the way it was +% specified in the past when a dynamic resolution change happens. Therefore +% the access to the outputqX is blocked until the format negotiation is +% finished. Otherwise there is a race condition: device sends a DPC event, the +% driver meantime keeps queueing buffers, the device can't figure out if these +% buffers should be used after the DPC or not. This is somewhat similar to +% V4L2_DEC_CMD_START command except that here it only covers the parameter +% changes. + +Only the parameters returned in one of the corresponding sets in the device +capabilities can be set, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. +The device checks and applies the parameter changes and sends the +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. +The async response can also come in case of a dynamic parameters change, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Dynamic Parameters Change Event}. + +The command-specific async response +\field{struct virtio_video_stream_params_async_resp} is defined +as follows: + +\begin{lstlisting} +struct virtio_video_stream_get_set_params_async_resp { + struct virtio_video_event_header hdr; + /** + * Followed by a single VIRTIO_VIDEO_TLV_CODED_SET or + * VIRTIO_VIDEO_TLV_RAW_SET TLV container with the parameters. + */ +}; +\end{lstlisting} + +The TLV container in the response is of the same type as in the request and it +contains the actual values of the set parameters supported by the +device. The values set by the device can differ from the requested values +depending on the device's capabilities. If the TLV container in the request is +empty, the response is also empty. \textbf{Note:} lengths of the +VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES and +VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT TLVs are always 8 in the device's +response in order to save memory and make eventq descriptor size more +predictable, i.e. they include only the \field{resource_id} fields. Missing +TLV for a resource means that it is not attached. + +The backing memory for resources can only be attached when there is no chance +for it to be simultaneously used by the device. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} + +The device MUST initialize each parameter to a valid default value. + +The device MUST allow each parameter to be read even without the driver +explicitly setting a value for them beforehand. + +The device MAY adjust any received parameter to a closest supported +value if the received one is not supported with the current settings. + +The parameters received and returned by the device MUST fit together into a +pair of linked sets returned in +\field{struct virtio_video_device_query_caps_resp}. + +The parameters MUST be applied in the order of appearance in the TLV +container. +The device MUST send an async response with an error flag set as soon as it +encounters an invalid not correctable input and stop processing the TLVs +afterwards. + +The device MUST process parameters changes, that are embedded in the input +stream, in the same way as if there is a VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS +command queued into inputqX changing the outputqX parameters. A standalone DPC +event MUST be sent in place of the command's async response in this case. + +The device MUST return the changed inputqX or outputqX parameters in the +async response. + +If the command is interrupted with a RESET operation on inputqX or a CLOSE +operation, the device MUST send the async response with +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} + +\field{stream_id} MUST be set to a valid stream ID of an open stream. + +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS MUST be sent to mainqX or inputqX or +outputqX where X equals \field{stream_id}. + +The driver MUST put exactly one TLV container to the request with type +selected according to the queue type. + +The driver MUST check the actual values of the parameters as set by the +device and work with these values, or try to set different ones if it +cannot, or fail properly. + +After creating a new stream, the initial value of all parameters is +undefined to the driver. Thus, the driver MUST NOT assume the default +value of any parameter and MAY use VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS +in order to get the values of the parameters it needs. + +If some of the resources were detached as a result of this command the driver +SHOULD reattach the backing memories of these resources and queue them again +to resume the device operation. + +The same type of backing memories (either guest pages, or virtio objects) +MUST be used for all resources within a queue. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / GET PARAMS} + +Get the current values of all parameters supported by the device for inputqX +or outputqX of a given stream as reported by +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. +The command is very similar to VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, except that +the TLV set in the command is always empty and all supported parameters are +returned by the device in the async response. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} + +Unblock the access to outputqX after output format and resource parameters +negotiation between the device and the driver is finished. + +The driver sends this command with +\field{struct virtio_video_stream_unblock}: + +\begin{lstlisting} +struct virtio_video_stream_unblock { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_UNBLOCK -> mainqX */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to continue. +\end{description} + +When the outputqX is unblocked the device sends the +VIRTIO_VIDEO_CMD_STREAM_UNBLOCK async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} + +The device MUST set the VIRTIO_VIDEO_EVENT_FLAG_ERROR flag if the outputqX is +not blocked. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} + +VIRTIO_VIDEO_CMD_STREAM_UNBLOCK MUST be sent to mainqX. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_DRAIN} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} + +Complete processing of all input resources queued before this command +and make the resulting output resources available to the driver. + +The driver sends this command with +\field{struct virtio_video_stream_drain}: + +\begin{lstlisting} +struct virtio_video_stream_drain { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_DRAIN -> inputqX */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to drain. +\end{description} + +The device begins the async DRAIN operation. When the operation is completed +the device sends the VIRTIO_VIDEO_CMD_STREAM_DRAIN async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_DRAIN}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} + +Before the device sends the response, it MUST process and respond to all +the commands on the inputqX that were sent before the drain command, and make +all the corresponding output resources available to the driver with async +responses to their VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE commands. + +The device MUST be able to accept commands to inputqX while a DRAIN operation +is ongoing, but any resulting async responses MUST NOT be sent before +the async response to the command, that started the DRAIN operation. + +If the command is interrupted with a RESET operation on inputqX or a CLOSE +operation, the device MUST send the async response with +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. + +The device MUST retain certain state after a DRAIN e.g. VPS, SPS, PPS. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_DRAIN}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} + +VIRTIO_VIDEO_CMD_STREAM_DRAIN MUST be sent to inputqX. + +The driver MUST keep queueing output resources until it gets the +async response to this command or cancels it using +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET or +VIRTIO_VIDEO_CMD_STREAM_CLOSE. Failure to do so may result in the +device stalling as it waits for output resources to write into. + +The driver MUST send a VIRTIO_VIDEO_CMD_STREAM_DRAIN command when it does not +have any further input to ensure it receives all the output. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} + +Immediately cancel all queued resources in inputqX or outputqX without +processing them and discard any processing results, that are not yet dequeued. +This command is useful for decoders that need to quickly jump to another point +in the stream (i.e. for seeking), or in order to clear the queue as quickly as +possible. + +The driver sends this command with +\field{struct virtio_video_queue_reset}: + +\begin{lstlisting} +struct virtio_video_queue_reset { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET -> mainqX */ + le32 reset_queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_{INPUT|OUTPUT} */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to reset. + \item[\field{reset_queue_type}] + is the queue type to reset. +\end{description} + +The device begins the async RESET operation. When the async RESET operation is +completed the device sends the VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET async +response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} + +The device MUST send async responses with VIRTIO_VIDEO_EVENT_FLAG_CANCELED +flag set for all active or pending commands in the selected queue before +sending the async response to this command. + +The device MUST interrupt operation as quickly as possible. Doing a RESET of +inputqX MUST NOT depend on output resources being queued by the driver. + +% If the device must accept more input after the beginning of the RESET +% like it was required in the previous versions of the specification, then +% some more measures are necessary because these are different queues now. +% For example, adding a "generation" field into the commands. At the moment +% this doesn't look like a problem because this is not supported in V4L2. + +The device MUST retain certain state after a RESET e.g. VPS, SPS, PPS. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} + +\field{stream_id} MUST be set to a valid stream ID of an open stream. + +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET MUST be sent to mainqX. + +\paragraph{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} + +Provide an input or output resource to the device for processing. + +\begin{lstlisting} +#define VIRTIO_VIDEO_MAX_PLANES 8 + +#define VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME (1 << 0) +#define VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME (1 << 1) +#define VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME (1 << 2) +/* On dequeue only */ +#define VIRTIO_VIDEO_QUEUE_FLAG_V4L2_DPC_LAST (1 << 3) +#define VIRTIO_VIDEO_QUEUE_FLAG_V4L2_DRAIN_LAST (1 << 4) + +struct virtio_video_resource_queue { + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE */ + le32 resource_id; + le32 flags; /* Bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* */ + le64 timestamp; + le32 offsets[VIRTIO_VIDEO_MAX_PLANES]; + le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{hdr.stream_id}] + is the ID of the stream to provide the resource to. + \item[\field{hdr.queue_type}] + is either inpuqX, or outputqX. + \item[\field{resource_id}] + is the ID of the resource to be queued. + \item[\field{flags}] + is a bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* values. + + \begin{description} + \item[VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME] + can be set on decoder input resources when the + resource contains an encoded key frame. + \item[VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME] + can be set on decoder input resources when the + resource contains only differences to preceding frames. + \item[VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME] + can be set on decoder input resources when the + resource contains the differences between the current + frame and both the preceding and following frames. + \end{description} + \item[\field{timestamp}] + is an abstract sequence counter that can be used on the inputqX for + synchronization. Resources produced on the output queue will carry the + \field{timestamp} of the first input resource they have been produced + from. + \item[\field{offsets}] + is the starting offset for the data in the buffer for each plane. + The number of planes depends on the format. Set by the driver for input + resources. + \item[\field{data_sizes}] + is number of data bytes used for each plane. Set by the driver for input + resources. +\end{description} + +The device uses the resource in the video processing. When the processing of +the resource is completed the device sends the +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE async response, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. + +The command-specific async response +\field{struct virtio_video_resource_queue_async_resp} is defined +as follows: + +\begin{lstlisting} +struct virtio_video_resource_queue_async_resp { + struct virtio_video_event_header hdr; + le32 flags; + u8 padding[4]; + le64 timestamp; + le32 offsets[VIRTIO_VIDEO_MAX_PLANES]; + le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{flags}] + is a bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* flags. + + \begin{description} + \item[VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME] + is set on encoder output resources when the + resource contains an encoded key frame. + \item[VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME] + is set on encoder output resources when the + resource contains only differences to preceding frames. + \item[VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME] + is set on encoder output resources when the + resource contains the differences between the current + frame and both the preceding and following frames. + \end{description} + \item[\field{timestamp}] + is set on output resources to the \field{timestamp} value of the first input + resource that produced the resource. + \item[\field{offsets}] + is set on output resources to the starting offset for the data in the + buffer for each plane. + \item[\field{data_sizes}] + is set on output resources to the amount of data written by the device, + for each plane. +\end{description} + +VIRTIO_VIDEO_EVENT_FLAG_ERROR is set in event_flags of the async response on +resources when a non-fatal processing error has happened and the data +contained in the resource is likely to be corrupted. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} + +The device MUST set VIRTIO_VIDEO_EVENT_FLAG_ERROR in the async response if the +resource has not been attached prior to queueing it, or for an attempt to queue +a resources that is still processed asynchronously, or for resources that might +contain corrupted content due to an error. + +For output resources, the device MUST copy the \field{timestamp} +parameter of the first input resource that produced it into the async +response. +When many output resources are produced from a single input resource, the +device MUST copy the timestamp of the input resource to all of the output +resources. + +In case of encoder, the device MUST mark each output resource with one of +VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME, VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME, or +VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME. + +If the processing of a resource was canceled due to a stream event, a +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, or a VIRTIO_VIDEO_CMD_STREAM_CLOSE, +the device MUST send the corresponding async response with +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. + +When starting or resuming processing after a RESET or a DRAIN operation, the +device MAY skip input data until it finds a point that allows it to resume +operation properly (e.g. until a keyframe is found in the input stream of a +decoder). + +The device MUST properly handle the case when a dequeued but still referenced +resource is queued again. + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE}{Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} + +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE MUST be sent to inputqX or outputqX. + +\field{resource_id} MUST be an ID of a resource, that is both allocated and +attached for the queue. + +The driver MUST be able to handle the output resources in decoding order in +encoder case, i.e. with timestamps out of order. + +\subsubsection{Device Operation: Standalone Events} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events} + +These events are caused by state changes in the device, not as an async +response to any command. + +\paragraph{Error Event} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Error Event} + +The error event is sent by the device when an unrecoverable error occurs +during processing a stream. The device operation is the same as when +it receives a VIRTIO_VIDEO_CMD_STREAM_CLOSE command, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} +except that it also sets the VIRTIO_VIDEO_EVENT_FLAG_ERROR flag in the +response. Note that this is different from other async responses carrying the +VIRTIO_VIDEO_EVENT_FLAG_ERROR flag. The latter indicates that e.g. the +particular output frame might be corrupted, but the stream still exists +and can recover. + +\paragraph{Dynamic Parameters Change Event} +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Dynamic Parameters Change Event} + +A Dynamic Parameters Change (or DPC) event is sent by a decoder device when it +detects that the parameters of the stream being decoded have changed. +The device operation is the same as if it receives a +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS command in the inputqX at the exact same +point in the stream, that changes outputqX parameters, see +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS}. +This includes activating the outputqX block if necessary. + +% TODO add QoS events and overall think about quotas. Codecs are normally +% limited by bandwidth/macroblocks per second. How can we accommodate this? + +\subsection{Device capabilities and parameters} +\label{sec:Device Types / Video Device / Device capabilities and parameters} + +\subsubsection{VIRTIO_VIDEO_TLV_CODED_SET} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED} + +These parameters are defined for the coded parameter sets. + +\paragraph{VIRTIO_VIDEO_TLV_CODED_FORMAT} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / FORMAT} + +The following coded formats are defined: + +\begin{lstlisting} +#define VIRTIO_VIDEO_CODED_FORMAT_MPEG2 1 /* MPEG-2 Part 2 (V4L2_PIX_FMT_MPEG2) */ +#define VIRTIO_VIDEO_CODED_FORMAT_MPEG4 2 /* MPEG-4 Part 2 (V4L2_PIX_FMT_MPEG4) */ +#define VIRTIO_VIDEO_CODED_FORMAT_H264 3 /* H.264 (V4L2_PIX_FMT_H264) */ +#define VIRTIO_VIDEO_CODED_FORMAT_HEVC 4 /* HEVC aka H.265 (V4L2_PIX_FMT_HEVC) */ +#define VIRTIO_VIDEO_CODED_FORMAT_VP8 5 /* VP8 (V4L2_PIX_FMT_VP8) */ +#define VIRTIO_VIDEO_CODED_FORMAT_VP9 6 /* VP9 (V4L2_PIX_FMT_VP9) */ +#define VIRTIO_VIDEO_CODED_FORMAT_FWHT 7 /* FWHT (V4L2_PIX_FMT_FWHT) */ +\end{lstlisting} + +The coded formats and the expected data units per buffer are documented in +\hyperref[intro:V4L2]{V4L2 header} and +\hyperref[intro:V4L2 compressed]{V4L2 compressed formats documentation}. + +\field{struct virtio_video_tlv_coded_format} represents both the coded format +in a coded set of capabilities and the specific parameter values: + +\begin{lstlisting} +struct virtio_video_tlv_coded_format { + le32 format; /* VIRTIO_VIDEO_CODED_FORMAT_* */ +}; +\end{lstlisting} + +\paragraph{VIRTIO_VIDEO_TLV_CODED_RESOURCES} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCES} + +Setup common coded resources parameters. + +\field{struct virtio_video_tlv_coded_resources_caps} represents capabilities: + +\begin{lstlisting} +struct virtio_video_tlv_coded_resources_caps { + struct virtio_video_range num_resources_range; + struct virtio_video_range resource_size_range; +}; +\end{lstlisting} + +\begin{description} + \item[\field{num_resources_range}] + is the supported range of resources number of a particular coded set. + \item[\field{resource_size_range}] + is the supported range of resource sizes. +\end{description} + +\field{struct virtio_video_tlv_resources_val} represents the parameter values: + +\begin{lstlisting} +struct virtio_video_tlv_resources_val { + le32 num_resources; +}; +\end{lstlisting} + +\begin{description} + \item[\field{num_resources}] + is the number of resources that can be addressed for the queue, numbered + from \(0\) to \(num\_resources - 1\). Setting this parameter to zero is + allowed even when \field{num_resources_range.min} is positive, this results + in detaching all the resources. +\end{description} + +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCE GUEST PAGES} + +This TLV is defined in the same way as for the VIRTIO_VIDEO_TLV_RAW_SET, see +\ref{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES}. +Note, that a coded resource can only have a single buffer, so only the first +element of \field{num_entries} is not zero. + +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCE VIRTIO OBJECT} + +This TLV is defined in the same way as for the VIRTIO_VIDEO_TLV_RAW_SET, see +\ref{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT}. +Note, that a coded resource can only have a single buffer, so +\field{num_objects} is always 1. + +\paragraph{VIRTIO_VIDEO_TLV_V4L2_CONTROLS} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS} + +Inside this TLV container only selected V4L2 control IDs are allowed to be +used as TLV types. The subset is listed in subsections below. Values of the +V4L2 controls can be converted to TLVs by taking their representation as +\hyperref[intro:V4L2 ext ctrls]{struct v4l2_ext_control} and replacing the +pointers with values. Capabilities of the V4L2 controls can't be converted to +TLVs as easily, so they are described below. This is mostly useful for +encoders. Most relevant V4L2 controls are codec-specific. All definitions +related to V4L2 controls can be found in +\hyperref[intro:V4L2 controls]{V4L2 controls header}, their descriptions +can be found in \hyperref[intro:V4L2 codec controls]{V4L2 documentation}. + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_V4L2_CONTROLS}{Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS} + +The device MUST NOT advertise codec-specific parameters not corresponding to +the coded format of the particular coded set. + +\subparagraph{V4L2 controls: 32 bit integers} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: 32 bit integers} + +Integer V4L2 controls are signed by default, but this specification doesn't +define any signed integer types, see \ref{sec:Structure Specifications}, so +not every integer V4L2 control could be used directly. Still for many of them +negative values don't make sense, so these controls are allowed in the range +from 0 to INT32_MAX: + +\begin{itemize} + \item V4L2_CID_MPEG_VIDEO_BITRATE: bitrate in bits per second +\end{itemize} + +For capabilities the TLV value is defined as follows: + +\begin{lstlisting} +struct virtio_video_tlv_v4l2_int_caps { + struct virtio_video_range range; +}; +\end{lstlisting} + +\begin{description} + \item[\field{range}] + is a range of possible values of the control. +\end{description} + +For the control values the TLV value is defined as follows: + +\begin{lstlisting} +struct virtio_video_tlv_v4l2_int_val { + le32 value; +}; +\end{lstlisting} + +\begin{description} + \item[\field{value}] + is one of the supported control values. +\end{description} + +\drivernormative{\subparagraph}{V4L2 controls: 32 bit integers}{Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: 32 bit integers} + +The integer V4L2 control values MUST be in the range from 0 to INT32_MAX. + +\subparagraph{V4L2 controls: Enumerations} +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: Enumerations} + +The following V4L2 controls with values defined as enums are allowed: +\begin{itemize} + \item V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE + % enum v4l2_mpeg_video_mpeg2_profile: V4L2_MPEG_VIDEO_MPEG2_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL + % enum v4l2_mpeg_video_mpeg2_level: V4L2_MPEG_VIDEO_MPEG2_LEVEL_* + \item V4L2_CID_MPEG_VIDEO_MPEG4_PROFILE + % enum v4l2_mpeg_video_mpeg4_profile: V4L2_MPEG_VIDEO_MPEG4_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL + % enum v4l2_mpeg_video_mpeg4_level: V4L2_MPEG_VIDEO_MPEG4_LEVEL_* + \item V4L2_CID_MPEG_VIDEO_H264_PROFILE + % enum v4l2_mpeg_video_h264_profile: V4L2_MPEG_VIDEO_H264_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_H264_LEVEL + % enum v4l2_mpeg_video_h264_level: V4L2_MPEG_VIDEO_H264_LEVEL_* + \item V4L2_CID_MPEG_VIDEO_HEVC_PROFILE + % enum v4l2_mpeg_video_hevc_profile: V4L2_MPEG_VIDEO_HEVC_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_HEVC_LEVEL + % enum v4l2_mpeg_video_hevc_level: V4L2_MPEG_VIDEO_HEVC_LEVEL_* + \item V4L2_CID_MPEG_VIDEO_VP8_PROFILE + % enum v4l2_mpeg_video_vp8_profile: V4L2_MPEG_VIDEO_VP8_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_VP9_PROFILE + % enum v4l2_mpeg_video_vp9_profile: V4L2_MPEG_VIDEO_VP9_PROFILE_* + \item V4L2_CID_MPEG_VIDEO_VP9_LEVEL + % enum v4l2_mpeg_video_vp9_level: V4L2_MPEG_VIDEO_VP9_LEVEL_* +\end{itemize} + +For capabilities the TLV value is defined as follows: + +\begin{lstlisting} +#define MASK(x) (1 << (x)) + +struct virtio_video_tlv_v4l2_enum_caps { + le32 bitmask; /* Bitmask of MASK(<enum value>) */ +}; +\end{lstlisting} + +\begin{description} + \item[\field{bitmask}] + is a bitmask of supported enum values used as bit numbers, see + \hyperref[intro:V4L2 controls]{V4L2 controls header}. +\end{description} + +For the control values the TLV value is defined as follows: + +\begin{lstlisting} +struct virtio_video_tlv_v4l2_enum_val { + u8 value; /* <enum value> */ + u8 padding[3]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{value}] + is one of the supported enum values, see + \hyperref[intro:V4L2 controls]{V4L2 controls header}. +\end{description} + +\subsubsection{VIRTIO_VIDEO_TLV_RAW_SET} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW} + +These parameters are defined for the raw parameter sets. + +\paragraph{VIRTIO_VIDEO_TLV_RAW_FORMAT} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / FORMAT} + +DRM fourcc format definitions and DRM format modifiers are used to represent +raw formats capabilities and values. The layouts of raw formats are documented +in \hyperref[intro:DRM formats]{DRM} and \hyperref[intro:V4L2]{V4L2} headers, +as well as in \hyperref[intro:V4L2 RGB]{V4L2 RGB} and +\hyperref[intro:V4L2 YUV]{planar YUV} formats documentation. + +% Some DRM and V4L2 formats can be matched with this table: +% DRM_FORMAT_ARGB8888 = V4L2_PIX_FMT_ABGR32 +% DRM_FORMAT_BGRA8888 = V4L2_PIX_FMT_ARGB32 +% DRM_FORMAT_RGBA8888 = V4L2_PIX_FMT_BGRA32 +% DRM_FORMAT_NV12 = V4L2_PIX_FMT_NV12 +% DRM_FORMAT_YUV420 = V4L2_PIX_FMT_YUV420 +% DRM_FORMAT_YVU420 = V4L2_PIX_FMT_YVU420 +% DRM_FORMAT_YUYV = V4L2_PIX_FMT_YUYV + +\field{struct virtio_video_tlv_raw_format_caps} is used to describe the +capabilities: + +\begin{lstlisting} +enum virtio_video_planes_layout { + VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1, + VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS = 2, +}; + +struct virtio_video_tlv_raw_format_caps { + le32 planes_layout_mask; /* Bitmask of VIRTIO_VIDEO_PLANES_LAYOUT_* */ + le32 fourcc; /* DRM_FORMAT_* */ + le64 modifier; /* DRM_FORMAT_MOD_* */ + struct virtio_video_range width_range; + struct virtio_video_range height_range; + le32 stride_align_mask; + le32 height_align_mask; + le32 plane_align_mask; +}; +\end{lstlisting} + +\begin{description} + \item[\field{planes_layout_mask}] + is a bitmask of supported planes layout types according to + \field{enum virtio_video_planes_layout}. + \item[\field{fourcc}] + specifies the raw format, to which these capabilities apply. + \item[\field{modifier}] + specifies the raw format modifier. + \item[\field{width_range}] + is a range of widths in pixels. + \item[\field{height_range}] + is a range of heights in pixels. + \item[\field{stride_align_mask}] + is a bitmask of all supported power of two alignments of the distance in + bytes between two lines of data (stride). + \item[\field{height_align_mask}] + is a bitmask of all supported power of two height alignments in pixels (scanlines). + \item[\field{plane_align_mask}] + is a bitmask of all supported power of two alignments in bytes of planes + within a buffer. This field is valid only if \field{planes_layout_mask} has + the \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit set. +\end{description} + +\field{struct virtio_video_tlv_raw_format_val} is used to describe the +values: + +\begin{lstlisting} +struct virtio_video_tlv_raw_format_val { + le32 planes_layout; /* VIRTIO_VIDEO_PLANES_LAYOUT_* */ + le32 fourcc; /* DRM_FORMAT_* */ + le64 modifier; /* DRM_FORMAT_MOD_* */ + le32 width; + le32 height; + le32 stride_align; + le32 height_align; + le32 plane_align; +}; +\end{lstlisting} + +\begin{description} + \item[\field{planes_layout}] + is the actual layout of the planes. + \item[\field{fourcc}] + specifies the raw format. + \item[\field{modifier}] + specifies the raw format modifier. + \item[\field{width}] + is the width in pixels of the stream frames. + \item[\field{height}] + is the height in pixels of the stream frames. + \item[\field{stride_align}] + is the power of two stride alignment in bytes. + \item[\field{height_align}] + is the power of two height alignment in pixels (scanlines). + \item[\field{plane_align}] + is the power of two alignment in bytes of planes within a buffer. This field + is valid only if \field{planes_layout} has the + \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit set. +\end{description} + +% TODO: add colorimetry + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_RAW_FORMAT}{Device Types / Video Device / Device capabilities and parameters / RAW / FORMAT} + +The device MUST set \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit in +\field{planes_layout_mask} if the plane layout with planes of a frame laid out +one after another in the same buffer is supported. + +The device MUST set \field{VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS} bit in +\field{planes_layout_mask} if the plane layout with planes of a frame laid out +in separate buffers is supported. + +% TODO: not sure if !VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG should be compatible +% with VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS or not. + +\paragraph{VIRTIO_VIDEO_TLV_RAW_RESOURCES} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCES} + +\field{struct virtio_video_tlv_raw_resources_caps} represents capabilities: + +\begin{lstlisting} +struct virtio_video_tlv_raw_resources_caps { + struct virtio_video_range num_resources_range; +}; +\end{lstlisting} + +\begin{description} + \item[\field{num_resources_range}] + is the supported range of resources number of a particular raw set. +\end{description} + +\field{struct virtio_video_tlv_resources_val} represents the parameter values, +see +\ref{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCES}. + +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES} + +Setup guest pages as backing memory of a resource. + +The parameter capabilities are empty. The empty TLV with zero length indicates +the support for attaching guest pages to resources. + +\field{struct virtio_video_tlv_resource_guest_pages} represents the parameter +values: + +\begin{lstlisting} +struct virtio_video_resource_sg_entry { + le64 addr; + le32 length; + u8 padding[4]; +}; + +struct virtio_video_tlv_resource_guest_pages { + le32 resource_id; + u8 padding[4]; + le32 num_entries[VIRTIO_VIDEO_MAX_PLANES]; + struct virtio_video_resource_sg_entry entries[]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{resource_id}] + is the ID of the resource. + \item[\field{num_entries}] + is the number of scatter-gather list entries in each of the separate buffers + forming together the resource according to currently set format. Unused + array elements are set to 0. Sum of the array is the length of the + \field{entries} array. + \item[\field{entries}] + is an array of the scatter-gather list entries: + + \begin{description} + \item[\field{addr}] + is a guest physical address of the start of the SG entry aligned to + the physical guest pages size. + \item[\field{length}] + is the length of the SG entry in bytes aligned to the physical guest + pages size. + \end{description} +\end{description} + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES}{Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES} + +\field{resource_id} MUST be an integer within the range of resource IDs +currently allocated for the queue. + +The memory regions identified by the elements of the \field{entries} array +MUST NOT overlap. + +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT} + +Setup virtio objects as backing memory to a resource. + +The parameter capabilities are empty. The empty TLV with zero length indicates +the support for attaching virtio objects to resources. + +\field{struct virtio_video_tlv_resource_virtio_object} represents the parameter +values: + +\begin{lstlisting} +struct virtio_video_resource_object { + u8 uuid[16]; +}; + +struct virtio_video_tlv_resource_virtio_object { + le32 resource_id; + le32 num_objects; /* Up to VIRTIO_VIDEO_MAX_PLANES */ + struct virtio_video_resource_object objects[num_objects]; +}; +\end{lstlisting} + +\begin{description} + \item[\field{resource_id}] + is the ID of the resource. + \item[\field{num_objects}] + is the length of the \field{objects} array according to currently set + format. + \item[\field{object}] + is an array of objects exported from another virtio device, see + \ref{sec:Basic Facilities of a Virtio Device / Exporting Objects}. + + \begin{description} + \item[uuid] + is a version 4 UUID specified by \hyperref[intro:rfc4122]{[RFC4122]}. + \end{description} +\end{description} + +\drivernormative{\subparagraph}{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT}{Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT} + +\field{resource_id} MUST be an integer within the range of resource IDs +currently allocated for the queue. + +\paragraph{VIRTIO_VIDEO_TLV_CROP} +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / CROP} + +% TODO There is no reason in doing crop if it doesn't affect the output. +% So the output frames have to be smaller, than the full size. In the decoder +% case this means, that the buffers can't be used as a reference. So when crop +% is enabled, the decoder probably has to have some intermediate buffers. +% Is it reasonable to reallocate output buffers then? It could be. So it has +% to be decided (probably in the same way it is usually done in V4L2). In the +% encoder case this is not a problem. +% Is setting compose rectangles useful at all? + +This parameter sets a rectangle covering the visible area of the frame. + +The parameter capabilities are empty. The empty TLV with zero length indicates +the support for cropping. + +The parameter value is defined as follows: + +\begin{lstlisting} +struct virtio_video_tlv_crop_val { + le32 left; + le32 top; + le32 width; + le32 height; +}; +\end{lstlisting} + +\begin{description} + \item[\field{left, top}] + are coordinates of top left corner of the crop rectangle in pixels. + \item[\field{width, height}] + are dimensions of the crop rectangle in pixels. +\end{description} + +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_CROP}{Device Types / Video Device / Device capabilities and parameters / RAW / CROP} + +The crop rectangle MUST be reset to full frame size on every resolution +change. diff --git a/device-types/video/device-conformance.tex b/device-types/video/device-conformance.tex new file mode 100644 index 0000000..64960ac --- /dev/null +++ b/device-types/video/device-conformance.tex @@ -0,0 +1,22 @@ +\conformance{\subsection}{Video Device Conformance} +\label{sec:Conformance / Device Conformance / Video Device Conformance} + +A video device MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{devicenormative:Device Types / Video Device / Feature bits} +\item \ref{devicenormative:Device Types / Video Device / Device configuration layout} +\item \ref{devicenormative:Device Types / Video Device / Device Operation} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: TLV format} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS} +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / RAW / FORMAT} +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / RAW / CROP} +\end{itemize} diff --git a/device-types/video/driver-conformance.tex b/device-types/video/driver-conformance.tex new file mode 100644 index 0000000..ca59885 --- /dev/null +++ b/device-types/video/driver-conformance.tex @@ -0,0 +1,20 @@ +\conformance{\subsection}{Video Driver Conformance} +\label{sec:Conformance / Driver Conformance / Video Driver Conformance} + +A video driver MUST conform to the following normative statements: + +\begin{itemize} +\item \ref{drivernormative:Device Types / Video Device / Feature bits} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: 32 bit integers} +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES} +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT} +\end{itemize} diff --git a/introduction.tex b/introduction.tex index 9a9cbde..52efa16 100644 --- a/introduction.tex +++ b/introduction.tex @@ -113,6 +113,15 @@ \section{Normative References}\label{sec:Normative References} \phantomsection\label{intro:SEC1}\textbf{[SEC1]} & Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000. \newline\url{https://www.secg.org/sec1-v2.pdf}\\ + \phantomsection\label{intro:V4L2}\textbf{[V4L2]} & + Linux V4L2 interface. + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/videodev2.h}\\ + \phantomsection\label{intro:V4L2 controls}\textbf{[V4L2 Controls]} & + Linux V4L2 controls definitions. + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/v4l2-controls.h}\\ + \phantomsection\label{intro:DRM formats}\textbf{[DRM Formats]} & + Linux DRM format definitions. + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h}\\ \phantomsection\label{intro:rfc2784}\textbf{[RFC2784]} & Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol. @@ -194,6 +203,18 @@ \section{Non-Normative References} \phantomsection\label{intro:Virtio PCI Draft}\textbf{[Virtio PCI Draft]} & Virtio PCI Draft Specification \newline\url{http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf}\\ + \phantomsection\label{intro:V4L2 compressed}\textbf{[V4L2 compressed formats]} & + Detailed descriptions of V4L2 compressed formats + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst}\\ + \phantomsection\label{intro:V4L2 RGB}\textbf{[V4L2 RGB formats]} & + Detailed descriptions of V4L2 RGB formats + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/pixfmt-rgb.rst}\\ + \phantomsection\label{intro:V4L2 YUV}\textbf{[V4L2 planar YUV formats]} & + Detailed descriptions of V4L2 planar YUV formats + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/pixfmt-yuv-planar.rst}\\ + \phantomsection\label{intro:V4L2 codec controls}\textbf{[V4L2 codec controls]} & + Detailed descriptions of V4L2 controls + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/ext-ctrls-codec.rst}\\ \end{longtable} \section{Terminology}\label{Terminology} -- 2.43.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* RE: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-13 8:53 ` [PATCH v10 1/1] virtio-video: Add virtio " Alexander Gordeev @ 2026-02-15 10:49 ` Parav Pandit 2026-02-15 15:11 ` Michael S. Tsirkin 2026-02-24 20:02 ` Alexander Gordeev 0 siblings, 2 replies; 9+ messages in thread From: Parav Pandit @ 2026-02-15 10:49 UTC (permalink / raw) To: Alexander Gordeev, virtio-comment@lists.linux.dev Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org > From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> > Sent: 13 February 2026 02:23 PM > > From: Alexander Gordeev <Alexander.Gordeev@opensynergy.com> > > Add the specification of the video decoder and encoder device, which > can be used to provide host-accelerated video operations to the guest. > > Signed-off-by: Alexander Gordeev <alexander.gordeev@opensynergy.com> > --- > conformance.tex | 12 +- > content.tex | 1 + > device-types/video/description.tex | 1592 +++++++++++++++++++++ > device-types/video/device-conformance.tex | 22 + > device-types/video/driver-conformance.tex | 20 + > introduction.tex | 21 + > 6 files changed, 1664 insertions(+), 4 deletions(-) > create mode 100644 device-types/video/description.tex > create mode 100644 device-types/video/device-conformance.tex > create mode 100644 device-types/video/driver-conformance.tex > > diff --git a/conformance.tex b/conformance.tex > index 9af31e2..f34e600 100644 > --- a/conformance.tex > +++ b/conformance.tex > @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, > \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, > \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, > -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or > -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. > +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, > +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or > +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. > > \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. > \end{itemize} > @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, > \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, > \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, > -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or > -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. > +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, > +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or > +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. > > \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. > \end{itemize} > @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \input{device-types/spi/driver-conformance.tex} > \input{device-types/media/driver-conformance.tex} > \input{device-types/rtc/driver-conformance.tex} > +\input{device-types/video/driver-conformance.tex} > > \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} > > @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > \input{device-types/spi/device-conformance.tex} > \input{device-types/media/device-conformance.tex} > \input{device-types/rtc/device-conformance.tex} > +\input{device-types/video/device-conformance.tex} > > \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy > Interface: Transitional Device and Transitional Driver Conformance} > A conformant implementation MUST be either transitional or > diff --git a/content.tex b/content.tex > index 5de811f..0c13f68 100644 > --- a/content.tex > +++ b/content.tex > @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} > \input{device-types/spi/description.tex} > \input{device-types/media/description.tex} > \input{device-types/rtc/description.tex} > +\input{device-types/video/description.tex} > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > diff --git a/device-types/video/description.tex b/device-types/video/description.tex > new file mode 100644 > index 0000000..8945e26 > --- /dev/null > +++ b/device-types/video/description.tex > @@ -0,0 +1,1592 @@ > +\section{Video Device} > +\label{sec:Device Types / Video Device} > + > +The virtio video device provides support for host-accelerated video encoding > +and decoding. > + > +\subsection{Device ID} > +\label{sec:Device Types / Video Device / Device ID} > + > +50 > + > +\subsection{Virtqueues} > +\label{sec:Device Types / Video Device / Virtqueues} > + > +\begin{description} > + \item[0] > + commandq - driver commands > + \item[1] > + eventq - device async responses to commands and standalone device events > +\end{description} > + > +\subsection{Feature bits} > +\label{sec:Device Types / Video Device / Feature bits} > + > +\begin{description} > + \item[VIRTIO_VIDEO_F_ENCODER (0)] > + The device can encode video. > + \item[VIRTIO_VIDEO_F_DECODER (1)] > + The device can decode video. > + % Use-case: the device can support both encoding and decoding, so having both > + % here can save resources. > + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] > + Guest pages can be used as the backing memory of resources. > + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] > + The device can use non-contiguous guest memory as the backing memory of > + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also > + set. > + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] > + Objects exported by another virtio device can be used as the backing memory > + of resources. > + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] > + The device releases an extra empty output buffer after a drain or DPC so that > + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. This does not seem relavant to the video device. It should not be attached to any V4L2 implementation. Can you please craft it differently? > +\end{description} > + > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > + > +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. > + > +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or > +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would > +mean that no memory can be used at all for resources. > + > +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets > +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. > + > +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > + > +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and > +VIRTIO_VIDEO_F_DECODER features. > + > +The driver MUST negotiate at least one of the > +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT > +features. > + You wouldn't need virtio_object feature bit. It should be covered using the capability. > +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not > +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically > +contiguous memory for all the buffers it allocates. > + > +\subsection{Device configuration layout} > +\label{sec:Device Types / Video Device / Device configuration layout} > + > +The video device configuration space uses the following layout: > + > +\begin{lstlisting} > +struct virtio_video_config { > + le32 max_streams; > + le32 caps_length; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{max_streams}] > + is the maximum number of concurrent streams the device supports. > + \item[\field{caps_length}] > + is the minimum length in bytes that a device-writable buffer must have > + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > +\end{description} > + > +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} > + > +\field{max_streams} MUST be positive. > + > +\field{caps_length} MUST be set to the response size of > +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. > + This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. Please rework the patch to use the existing basic facility. This can also possibly eliminate plumbing device specific command q. More below. > +\subsection{Device Initialization} > +\label{sec:Device Types / Video Device / Device Initialization} > +\begin{enumerate} > + \item > + The driver reads the feature bits and negotiates the features it needs. > + \item > + The driver sets up the commandq and the eventq. > + \item > + The driver reads the \field{caps_length} field of the configuration > + space, prepares a buffer of at least that size and sends the buffer on the > + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. > + \item > + The device sends a response over commandq to > + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the > + command. > + \item > + The driver receives the response from the device, and parses its capabilities. > +\end{enumerate} > + > +\subsection{Device Operation} > +\label{sec:Device Types / Video Device / Device Operation} > + > +The device supports opening and operating a number of parallel streams up to > +\field{max_streams}. Each stream has three internal device queues: mainqX, > +inputqX and outputqX, where X is the stream id. Each stream command has a > +field that dispatches the command to the specific internal queue. > + If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, Than keyword 'internal' should be dropped and rephased. > +% Use-case: there might be different real-time requirements for different > +% streams, so more virtqueues can be added in the future if necessary. > +% The internal queues don't change, the data formats don't change, only the > +% mapping of streams/internal queues to particular virtqueues changes. > + > +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, > +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX > +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out > +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock > +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for > +format negotiation. > + > +The inputqX and outputqX are used to queue input or output resources using > +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and > +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete > +processing of all queued input resources and make the resulting output > +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. > + > +All the stream commands start async operations, return the results > +using async responses over eventq. > +The eventq is used by the device to send the device's async responses to > +stream commands and the device's standalone events. > + > +% This way eventq becomes the single source of truth about the device's state, > +% the driver doesn't have to tediously synchronize commandq's and eventq's > +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF > +% and DQEVENT). One more benefit is that commandq is processed fast and > +% strictly in order, so commandq descriptors exhaustion should never happen in > +% practice. > + Video device should be implementable without any V4L2 binding/description etc. > +Parameters allow the driver to configure the stream including setting up the > +resources. Available parameters depend on the device type, see > +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. > + > +A resource is a set of memory buffers that contain a unit of data that > +the device can process or produce. Most resources have only one buffer, > +raw frames using a multi-planar format can have several. > +Input resources are filled by the driver with compressed (coded) video data > +for a decoder and raw frames for an encoder, output resources are filled by > +the device as the result of processing the input resources with decoded raw > +frames for a decoder and compressed (encoded) data for an encoder. > +Resources from inputqX and outputqX are consumed independently, not in pairs. > +One input resource can result in zero to many produced output resources. > +A decoder device dequeues the output decoded frames in presentation order. > +An encoder device dequeues the output decoded frames in decoding order. > +The driver can reuse a queued resource after receiving a corresponding async > +response. Dequeued output resources can still be used by the device as > +reference frames, so the driver can't write to them. > + > +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued > +% output resource is not used by the device anymore and therefore becomes > +% writeable? > + > +The device can detect standalone stream-related events: errors and dynamic > +parameters changes that require intervention from the driver (e.g. > +reallocating backing memory of output resources to fit the new parameters). > +The events are signalled on the eventq, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > + > +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} > + > +The device MUST set to zero all unused, disabled or padding bits in its > +responses. > + > +\subsubsection{Device Operation: Command Virtqueue} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} > + > +This section lists the commands that can be sent by the driver to commandq. > + > +Different structures are used for each command and response. A command > +structure starts with the requested command code, defined as follows: > + > +\begin{lstlisting} > +/* Device */ > +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 > + > +/* Stream */ > +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 > +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 You can craft the stream using a existing basic facility of resource object. Where each stream is just a resource object, that be queried or modified. > +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 > +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 > +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 > +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 > +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 > +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 > +\end{lstlisting} > + > +Stream commands start with a header: > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 > +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 > +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 > + > +struct virtio_video_stream_cmd_header { > + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ > + le32 stream_id; > + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ > + le32 async_response_cookie; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{async_response_cookie}] > + is an async response cookie provided by the driver, that allows > + to relate an async response to the previously submitted command. > +\end{description} > + > +\subsubsection{Device Operation: Event Virtqueue} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} > + > +The eventq is used by the device to send async responses to commands queued > +by the driver on commandq and standalone events. Stream errors and dynamic > +parameters changes are caused by changes in the device's state, not by > +commands, still they are delivered as responses to implicit > +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, > +respectively. > + > +Events start with a header: > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) > +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) > +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) > +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) > + > +struct virtio_video_event_header { > + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ > + le32 stream_id; > + le32 async_response_cookie; > + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{event_type}] > + is the type of the event. > + \item[\field{stream_id}] > + is the ID of a valid stream. > + \item[\field{async_response_cookie}] > + is an async response cookie provided by the driver, that allows > + to relate the event to a previously submitted command. > + \item[\field{event_flags}] > + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags > + > + \begin{description} > + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] > + is set if the command finished with an error due to an > + invalid argument or for other reasons. > + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] > + is set for standalone events, see > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] > + is set if the command has been canceled by another > + command, that has higher priority. Doesn't make sense > + for standalone events. > + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] > + is set if the command triggered a block on the > + outputqX to allow output format negotiation. > + When the negotiation is finished the block has to be > + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK > + command, see > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. > + \end{description} > +\end{description} > + > +The particular data structure representing the event is selected according to > +the \field{event_type}. > + > +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: > Event Virtqueue} > + > +The driver MUST at any time have at least one descriptor with a used > +buffer large enough to contain a \field{struct virtio_video_event} > +queued on the eventq. > + > +The driver MUST NOT put device-readable descriptors into the eventq. > + > +The driver MUST account for the fact that the async responses to commands > +might come out-of-order (i.e. after other commands sent to the device), > +and that some of them can be cancelled. > + > +The driver SHOULD wait for an async response of command A, that caused > +cancellation of command B, before queueing the command B again. > + > +\subsubsection{Device Operation: TLV format} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} > + > +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ > +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS > +commands represent device capabilities and corresponding device parameters > +in the form of TLV (Type-Length-Value): > + > +\begin{lstlisting} > +struct virtio_video_tlv { > + le32 type; > + le32 length; > + u8 value[length]; > +}; > +\end{lstlisting} > + All the caps can be crafted using existing capabilities infra. Please restructure the patch to use it. > +\begin{description} > + \item[\field{type}] > + specifies the type of data in \field{value}. > + \item[\field{length}] > + specifies the \field{value} size in bytes aligned to 4 bytes. > + \item[\field{value}] > + contains the data according to the type. > +\end{description} > + > +The following TLV types are defined: > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_TLV_CODED_SET 1 > +#define VIRTIO_VIDEO_TLV_RAW_SET 2 > +#define VIRTIO_VIDEO_TLV_LINK 3 > +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 > +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 > +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 > +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 > +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 > +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 > +#define VIRTIO_VIDEO_TLV_CROP 10 > +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 > +\end{lstlisting} > + You can use the 'flow filter' example of network device to see how to frame the individual or group of capabilities. > +Some TLVs are used only as containers for sequences of nested TLVs: > + > +\begin{description} > + \item[\field{VIRTIO_VIDEO_TLV_CODED_SET}] > + groups various capabilities or parameters related to a particular coded > + format. > + \item[\field{VIRTIO_VIDEO_TLV_RAW_SET}] > + groups various capabilities or parameters related to a particular raw > + format. > + \item[\field{VIRTIO_VIDEO_TLV_V4L2_CONTROLS}] > + contains V4L2 controls represented in the TLV format. Within this container > + only selected V4L2 control identifiers (V4L2_CID_*) are allowed to be used > + in the TLV \field{type} field, see > + \ref{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS}. > +\end{description} > + > +TLV with type VIRTIO_VIDEO_TLV_LINK is a special one used to define relations > +between sets of capabilities, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > + > +For each of the remaining TLV types two different contained data formats are > +defined: one for the capabilities and one for the specific parameter values. > + > +\field{struct virtio_video_range} is used to represent a range of values in > +some TLVs: > + > +\begin{lstlisting} > +struct virtio_video_range { > + le32 min; > + le32 max; > + le32 step; > + u8 padding[4]; > +}; > +\end{lstlisting} > + > +An integer \(x\) is within the range \field{r} if > +\(\field{r.min} \le x \le \field{r.max}\) holds and \(x\) equals to > +\((\field{min} + \field{step} * n)\) for some integer \(n\). > + > +\devicenormative{\paragraph}{Device Operation: TLV format}{Device Types / Video Device / Device Operation / Device Operation: TLV > format} > + > +\field{min}, \field{step} and \field{max} MUST be positive. > + > +\field{min} MUST be less then or equal to \field{max} within the same range. > + > +\subsubsection{Device Operation: Device Commands} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands} > + > +This command allows retrieving the device capabilities. > + > +\paragraph{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} > + > +Retrieve device capabilities for all available stream parameters (for example, > +the range of values). > + > +The driver sends this command with > +\field{struct virtio_video_device_query_caps}: > + > +\begin{lstlisting} > +struct virtio_video_device_query_caps { > + le32 type; /* VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS */ > +}; > +\end{lstlisting} > + > +The device responds with > +\field{struct virtio_video_device_query_caps_resp}: > + > +\begin{lstlisting} > + > +#define VIRTIO_VIDEO_RESULT_OK 0 > +#define VIRTIO_VIDEO_RESULT_ERROR 1 > + > +struct virtio_video_device_query_caps_resp { > + le32 result; /* VIRTIO_VIDEO_RESULT_* */ > + u8 padding[4]; > + /** > + * Followed by a sequence of TLVs up to caps_length > + * counted in bytes from the beginning of the struct. > + */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{result}] > + is > + > + \begin{description} > + \item[VIRTIO_VIDEO_RESULT_OK] > + if the command succeeded, > + \item[VIRTIO_VIDEO_RESULT_ERROR] > + if the descriptor was smaller than the defined \field{caps_length} in > + the video device configuration. > + \end{description} > +\end{description} > + > +The sequence of TLVs consists of TLV containers VIRTIO_VIDEO_TLV_CODED_SET > +and VIRTIO_VIDEO_TLV_RAW_SET defining sets of possible coded, or respectively > +raw formats, including the corresponding parameters (e.g. profiles, levels > +or format modifiers, resolutions), that are supported by the device. For the > +details see > +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. > +The last TLVs in the sequence with type VIRTIO_VIDEO_TLV_LINK establish > +relations between the two groups. If there is a link, then the device supports > +decoding from the specified coded set to the specified raw set, or encoding in > +the opposite direction. The value format is defined as follows: > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_STREAM_TYPE_DECODER 0 > +#define VIRTIO_VIDEO_STREAM_TYPE_ENCODER 1 > + > +struct virtio_video_tlv_link { > + le32 stream_type; /* One of VIRTIO_VIDEO_STREAM_TYPE_* */ > + u8 padding[4]; > + le64 links[n_coded * (n_raw + 63) / 64]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{stream_type}] > + is the type of the stream that is supported and negotiated using > + the respective feature flags. > + \item[\field{n_coded}] > + is the number of VIRTIO_VIDEO_TLV_CODED_SET containers. > + \item[\field{n_raw}] > + is the number of VIRTIO_VIDEO_TLV_RAW_SET containers. > + \item[\field{links}] > + is a bitset establishing links between coded and raw sets. For \field{i}-th > + coded and \field{j}-th raw sets counted from zero bit \field{(j \% 64)} in > + \field{links[i * (n_raw + 63) / 64 + j / 64]} defines the link if it is set. > +\end{description} > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS}{Device Types / Video Device / Device Operation / Device > Operation: Device Commands / QUERY CAPS} > + > +Response to the command MUST be written by the device in the first > +device-writable descriptor of the descriptor chain from which the command > +came. > + > +The device MUST include into the response at least one > +VIRTIO_VIDEO_TLV_CODED_SET and at least one VIRTIO_VIDEO_TLV_RAW_SET TLV > +containers, and one VIRTIO_VIDEO_TLV_LINK TLV defining links between the coded > +and raw sets for each VIRTIO_VIDEO_F_ENCODER and VIRTIO_VIDEO_F_DECODER > +feature flag that is negotiated. The VIRTIO_VIDEO_TLV_LINK TLVs MUST be the > +last in the sequence. > + > +Each VIRTIO_VIDEO_TLV_CODED_SET and VIRTIO_VIDEO_TLV_RAW_SET MUST take part > +in at least one defined link in one of the VIRTIO_VIDEO_TLV_LINK TLVs. > + > +Each VIRTIO_VIDEO_TLV_CODED_SET MUST contain exactly one > +VIRTIO_VIDEO_TLV_CODED_FORMAT TLV, exactly one > +VIRTIO_VIDEO_TLV_CODED_RESOURCES TLV and at most one TLV of other types. > + > +Each VIRTIO_VIDEO_TLV_RAW_SET MUST contain exactly one > +VIRTIO_VIDEO_TLV_RAW_FORMAT TLV, exactly one VIRTIO_VIDEO_TLV_RAW_RESOURCES > +TLV and at most one TLV of other types. > + > +VIRTIO_VIDEO_TLV_RAW_SET containers SHOULD be ordered according to raw format > +preferences of the device from preferred to not preferred ones. > + > +The total size of the response MUST be equal to \field{caps_length} > +bytes, as reported by the device configuration. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS}{Device Types / Video Device / Device Operation / Device > Operation: Device Commands / QUERY CAPS} > + > +Descriptor chains sent to the commandq by the driver MUST include at least one > +device-writable descriptor. The combined length of all such descriptors MUST > +be at least \field{caps_length} bytes. > + > +\subsubsection{Device Operation: Stream commands} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands} > + > +Stream commands allow to open, close and control the flow of a stream. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_OPEN} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} > + > +Open a video stream. > + > +The driver sends this command with \field{struct virtio_video_stream_open}: > + > +\begin{lstlisting} > +struct virtio_video_stream_open { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_OPEN -> mainqX */ > + le32 stream_type; /* One of VIRTIO_VIDEO_STREAM_TYPE_* */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to be opened. > + \item[\field{stream_type}] > + is the type of the stream to be opened. > +\end{description} > + > +The device begins an async OPEN operation. When the operation is completed the > +device sends the VIRTIO_VIDEO_CMD_STREAM_OPEN async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_OPEN}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / OPEN} > + > +The device MUST ensure that the \field{stream_id} is within limits and that > +the corresponding stream is not open. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_OPEN}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / OPEN} > + > +VIRTIO_VIDEO_CMD_STREAM_OPEN MUST be sent to mainqX. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_CLOSE} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} > + > +% Use-case: the guest user-space app closes the file descriptor. Cleanup fast. > + > +Close a video stream. Any activity on the stream is halted and all resources > +are released by the time the async response is received by the driver. > + > +The driver sends this command with > +\field{struct virtio_video_stream_close}: > + > +\begin{lstlisting} > +struct virtio_video_stream_close { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_CLOSE -> mainqX */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{stream_id}] > + is the ID of the stream to be closed. > +\end{description} > + > +The device begins an async CLOSE operation, that consists of RESET operations > +on inputqX and outputqX, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET}, > +detaching all the resources, releasing the internal stream queues and any > +corresponding internal resources. When the CLOSE operation is completed the > +device sends the VIRTIO_VIDEO_CMD_STREAM_CLOSE async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > +The same async response can also come after an unrecoverable stream error, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Error Event}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_CLOSE}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / CLOSE} > + > +After VIRTIO_VIDEO_CMD_STREAM_CLOSE is queued, the device MUST send an async > +response with VIRTIO_VIDEO_EVENT_FLAG_ERROR flag set to any subsequently > +queued command with this stream ID except VIRTIO_VIDEO_CMD_STREAM_OPEN. > + > +The CLOSE operation MUST NOT be canceled. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_CLOSE}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / CLOSE} > + > +\field{stream_id} MUST be set to a valid stream ID of an open stream. > + > +VIRTIO_VIDEO_CMD_STREAM_CLOSE MUST be sent to mainqX. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} > + > +Set selected parameters of inputqX or outputqX of a given stream and receive > +back the actual values of the set parameters. Coded parameters within a > +VIRTIO_VIDEO_TLV_CODED_SET container belong to inputqX of a decoder stream and > +to outputqX of an encoder stream. Raw parameters within a > +VIRTIO_VIDEO_TLV_RAW_SET container belong to outputqX of a decoder stream and > +to inputqX of an encoder stream. > + > +The driver sends this command with > +\field{struct virtio_video_stream_set_params}: > + > +\begin{lstlisting} > +struct virtio_video_stream_get_set_params { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS */ > + /** > + * Followed by a single VIRTIO_VIDEO_TLV_CODED_SET or > + * VIRTIO_VIDEO_TLV_RAW_SET TLV container with the parameters. > + */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to set parameters for. > +\end{description} > + > +The command can be queued to: > + > +\begin{itemize} > + \item[\field{inputqX}] > + The parameters are set after all commands previously queued on the > + inputqX are processed. > + \item[\field{mainqX}] > + The parameters are set without waiting for other commands queued > + on inputqX to be processed. > +\end{itemize} > + > +% Use-case: for the decoder, resolution can be set manually by the driver > +% (useful for codecs that do not embed this information, like MPEG-2). > +% The processing sequence should look similar to the dynamic parameters > +% change case. > +% Use-case: V4L2's request API. > + > +Any changes to formats, decreasing resource numbers or changes to resources > +in use trigger the following sequence: > +\begin{enumerate} > + \item > + an implicit DRAIN operation, see > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN}; > + \item > + only for changes to outputqX parameters: outputqX is blocked until > + a VIRTIO_VIDEO_CMD_STREAM_UNBLOCK command is received, see > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. > + When the block is active the decoder or encoder can't get anything > + from outputqX. > +\end{enumerate} > + > +% Use-case: dynamic resolution changes on inter frames in VP9. > +% See https://lore.kernel.org/linux-media/20240314153226.197445-1-benjamin.gaignard@collabora.com/ > +% Because of this buffers can't be detached/deallocated/reset the way it was > +% specified in the past when a dynamic resolution change happens. Therefore > +% the access to the outputqX is blocked until the format negotiation is > +% finished. Otherwise there is a race condition: device sends a DPC event, the > +% driver meantime keeps queueing buffers, the device can't figure out if these > +% buffers should be used after the DPC or not. This is somewhat similar to > +% V4L2_DEC_CMD_START command except that here it only covers the parameter > +% changes. > + > +Only the parameters returned in one of the corresponding sets in the device > +capabilities can be set, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > +The device checks and applies the parameter changes and sends the > +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > +The async response can also come in case of a dynamic parameters change, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Dynamic Parameters Change Event}. > + > +The command-specific async response > +\field{struct virtio_video_stream_params_async_resp} is defined > +as follows: > + > +\begin{lstlisting} > +struct virtio_video_stream_get_set_params_async_resp { > + struct virtio_video_event_header hdr; > + /** > + * Followed by a single VIRTIO_VIDEO_TLV_CODED_SET or > + * VIRTIO_VIDEO_TLV_RAW_SET TLV container with the parameters. > + */ > +}; > +\end{lstlisting} > + > +The TLV container in the response is of the same type as in the request and it > +contains the actual values of the set parameters supported by the > +device. The values set by the device can differ from the requested values > +depending on the device's capabilities. If the TLV container in the request is > +empty, the response is also empty. \textbf{Note:} lengths of the > +VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES and > +VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT TLVs are always 8 in the device's > +response in order to save memory and make eventq descriptor size more > +predictable, i.e. they include only the \field{resource_id} fields. Missing > +TLV for a resource means that it is not attached. > + > +The backing memory for resources can only be attached when there is no chance > +for it to be simultaneously used by the device. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / SET PARAMS} > + > +The device MUST initialize each parameter to a valid default value. > + > +The device MUST allow each parameter to be read even without the driver > +explicitly setting a value for them beforehand. > + > +The device MAY adjust any received parameter to a closest supported > +value if the received one is not supported with the current settings. > + > +The parameters received and returned by the device MUST fit together into a > +pair of linked sets returned in > +\field{struct virtio_video_device_query_caps_resp}. > + > +The parameters MUST be applied in the order of appearance in the TLV > +container. > +The device MUST send an async response with an error flag set as soon as it > +encounters an invalid not correctable input and stop processing the TLVs > +afterwards. > + > +The device MUST process parameters changes, that are embedded in the input > +stream, in the same way as if there is a VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS > +command queued into inputqX changing the outputqX parameters. A standalone DPC > +event MUST be sent in place of the command's async response in this case. > + > +The device MUST return the changed inputqX or outputqX parameters in the > +async response. > + > +If the command is interrupted with a RESET operation on inputqX or a CLOSE > +operation, the device MUST send the async response with > +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / SET PARAMS} > + > +\field{stream_id} MUST be set to a valid stream ID of an open stream. > + > +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS MUST be sent to mainqX or inputqX or > +outputqX where X equals \field{stream_id}. > + > +The driver MUST put exactly one TLV container to the request with type > +selected according to the queue type. > + > +The driver MUST check the actual values of the parameters as set by the > +device and work with these values, or try to set different ones if it > +cannot, or fail properly. > + > +After creating a new stream, the initial value of all parameters is > +undefined to the driver. Thus, the driver MUST NOT assume the default > +value of any parameter and MAY use VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS > +in order to get the values of the parameters it needs. > + > +If some of the resources were detached as a result of this command the driver > +SHOULD reattach the backing memories of these resources and queue them again > +to resume the device operation. > + > +The same type of backing memories (either guest pages, or virtio objects) > +MUST be used for all resources within a queue. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / GET PARAMS} > + > +Get the current values of all parameters supported by the device for inputqX > +or outputqX of a given stream as reported by > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > +The command is very similar to VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, except that > +the TLV set in the command is always empty and all supported parameters are > +returned by the device in the async response. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} > + > +Unblock the access to outputqX after output format and resource parameters > +negotiation between the device and the driver is finished. > + > +The driver sends this command with > +\field{struct virtio_video_stream_unblock}: > + > +\begin{lstlisting} > +struct virtio_video_stream_unblock { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_UNBLOCK -> mainqX */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to continue. > +\end{description} > + > +When the outputqX is unblocked the device sends the > +VIRTIO_VIDEO_CMD_STREAM_UNBLOCK async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / UNBLOCK} > + > +The device MUST set the VIRTIO_VIDEO_EVENT_FLAG_ERROR flag if the outputqX is > +not blocked. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_UNBLOCK}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / UNBLOCK} > + > +VIRTIO_VIDEO_CMD_STREAM_UNBLOCK MUST be sent to mainqX. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_DRAIN} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} > + > +Complete processing of all input resources queued before this command > +and make the resulting output resources available to the driver. > + > +The driver sends this command with > +\field{struct virtio_video_stream_drain}: > + > +\begin{lstlisting} > +struct virtio_video_stream_drain { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_DRAIN -> inputqX */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to drain. > +\end{description} > + > +The device begins the async DRAIN operation. When the operation is completed > +the device sends the VIRTIO_VIDEO_CMD_STREAM_DRAIN async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_DRAIN}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / DRAIN} > + > +Before the device sends the response, it MUST process and respond to all > +the commands on the inputqX that were sent before the drain command, and make > +all the corresponding output resources available to the driver with async > +responses to their VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE commands. > + > +The device MUST be able to accept commands to inputqX while a DRAIN operation > +is ongoing, but any resulting async responses MUST NOT be sent before > +the async response to the command, that started the DRAIN operation. > + > +If the command is interrupted with a RESET operation on inputqX or a CLOSE > +operation, the device MUST send the async response with > +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. > + > +The device MUST retain certain state after a DRAIN e.g. VPS, SPS, PPS. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_DRAIN}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / DRAIN} > + > +VIRTIO_VIDEO_CMD_STREAM_DRAIN MUST be sent to inputqX. > + > +The driver MUST keep queueing output resources until it gets the > +async response to this command or cancels it using > +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET or > +VIRTIO_VIDEO_CMD_STREAM_CLOSE. Failure to do so may result in the > +device stalling as it waits for output resources to write into. > + > +The driver MUST send a VIRTIO_VIDEO_CMD_STREAM_DRAIN command when it does not > +have any further input to ensure it receives all the output. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} > + > +Immediately cancel all queued resources in inputqX or outputqX without > +processing them and discard any processing results, that are not yet dequeued. > +This command is useful for decoders that need to quickly jump to another point > +in the stream (i.e. for seeking), or in order to clear the queue as quickly as > +possible. > + > +The driver sends this command with > +\field{struct virtio_video_queue_reset}: > + > +\begin{lstlisting} > +struct virtio_video_queue_reset { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET -> mainqX */ > + le32 reset_queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_{INPUT|OUTPUT} */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to reset. > + \item[\field{reset_queue_type}] > + is the queue type to reset. > +\end{description} > + > +The device begins the async RESET operation. When the async RESET operation is > +completed the device sends the VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET async > +response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / QUEUE RESET} > + > +The device MUST send async responses with VIRTIO_VIDEO_EVENT_FLAG_CANCELED > +flag set for all active or pending commands in the selected queue before > +sending the async response to this command. > + > +The device MUST interrupt operation as quickly as possible. Doing a RESET of > +inputqX MUST NOT depend on output resources being queued by the driver. > + > +% If the device must accept more input after the beginning of the RESET > +% like it was required in the previous versions of the specification, then > +% some more measures are necessary because these are different queues now. > +% For example, adding a "generation" field into the commands. At the moment > +% this doesn't look like a problem because this is not supported in V4L2. > + > +The device MUST retain certain state after a RESET e.g. VPS, SPS, PPS. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET}{Device Types / Video Device / Device Operation / Device > Operation: Stream commands / QUEUE RESET} > + > +\field{stream_id} MUST be set to a valid stream ID of an open stream. > + > +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET MUST be sent to mainqX. > + > +\paragraph{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} > + > +Provide an input or output resource to the device for processing. > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_MAX_PLANES 8 > + > +#define VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME (1 << 0) > +#define VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME (1 << 1) > +#define VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME (1 << 2) > +/* On dequeue only */ > +#define VIRTIO_VIDEO_QUEUE_FLAG_V4L2_DPC_LAST (1 << 3) > +#define VIRTIO_VIDEO_QUEUE_FLAG_V4L2_DRAIN_LAST (1 << 4) > + > +struct virtio_video_resource_queue { > + struct virtio_video_stream_cmd_header hdr; /* VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE */ > + le32 resource_id; > + le32 flags; /* Bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* */ > + le64 timestamp; > + le32 offsets[VIRTIO_VIDEO_MAX_PLANES]; > + le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{hdr.stream_id}] > + is the ID of the stream to provide the resource to. > + \item[\field{hdr.queue_type}] > + is either inpuqX, or outputqX. > + \item[\field{resource_id}] > + is the ID of the resource to be queued. > + \item[\field{flags}] > + is a bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* values. > + > + \begin{description} > + \item[VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME] > + can be set on decoder input resources when the > + resource contains an encoded key frame. > + \item[VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME] > + can be set on decoder input resources when the > + resource contains only differences to preceding frames. > + \item[VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME] > + can be set on decoder input resources when the > + resource contains the differences between the current > + frame and both the preceding and following frames. > + \end{description} > + \item[\field{timestamp}] > + is an abstract sequence counter that can be used on the inputqX for > + synchronization. Resources produced on the output queue will carry the > + \field{timestamp} of the first input resource they have been produced > + from. > + \item[\field{offsets}] > + is the starting offset for the data in the buffer for each plane. > + The number of planes depends on the format. Set by the driver for input > + resources. > + \item[\field{data_sizes}] > + is number of data bytes used for each plane. Set by the driver for input > + resources. > +\end{description} > + > +The device uses the resource in the video processing. When the processing of > +the resource is completed the device sends the > +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE async response, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue}. > + > +The command-specific async response > +\field{struct virtio_video_resource_queue_async_resp} is defined > +as follows: > + > +\begin{lstlisting} > +struct virtio_video_resource_queue_async_resp { > + struct virtio_video_event_header hdr; > + le32 flags; > + u8 padding[4]; > + le64 timestamp; > + le32 offsets[VIRTIO_VIDEO_MAX_PLANES]; > + le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{flags}] > + is a bitmask of VIRTIO_VIDEO_QUEUE_FLAG_* flags. > + > + \begin{description} > + \item[VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME] > + is set on encoder output resources when the > + resource contains an encoded key frame. > + \item[VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME] > + is set on encoder output resources when the > + resource contains only differences to preceding frames. > + \item[VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME] > + is set on encoder output resources when the > + resource contains the differences between the current > + frame and both the preceding and following frames. > + \end{description} > + \item[\field{timestamp}] > + is set on output resources to the \field{timestamp} value of the first input > + resource that produced the resource. > + \item[\field{offsets}] > + is set on output resources to the starting offset for the data in the > + buffer for each plane. > + \item[\field{data_sizes}] > + is set on output resources to the amount of data written by the device, > + for each plane. > +\end{description} > + > +VIRTIO_VIDEO_EVENT_FLAG_ERROR is set in event_flags of the async response on > +resources when a non-fatal processing error has happened and the data > +contained in the resource is likely to be corrupted. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE}{Device Types / Video Device / Device Operation / > Device Operation: Stream commands / RESOURCE QUEUE} > + > +The device MUST set VIRTIO_VIDEO_EVENT_FLAG_ERROR in the async response if the > +resource has not been attached prior to queueing it, or for an attempt to queue > +a resources that is still processed asynchronously, or for resources that might > +contain corrupted content due to an error. > + > +For output resources, the device MUST copy the \field{timestamp} > +parameter of the first input resource that produced it into the async > +response. > +When many output resources are produced from a single input resource, the > +device MUST copy the timestamp of the input resource to all of the output > +resources. > + > +In case of encoder, the device MUST mark each output resource with one of > +VIRTIO_VIDEO_QUEUE_FLAG_KEY_FRAME, VIRTIO_VIDEO_QUEUE_FLAG_P_FRAME, or > +VIRTIO_VIDEO_QUEUE_FLAG_B_FRAME. > + > +If the processing of a resource was canceled due to a stream event, a > +VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, or a VIRTIO_VIDEO_CMD_STREAM_CLOSE, > +the device MUST send the corresponding async response with > +VIRTIO_VIDEO_EVENT_FLAG_CANCELED flag set. > + > +When starting or resuming processing after a RESET or a DRAIN operation, the > +device MAY skip input data until it finds a point that allows it to resume > +operation properly (e.g. until a keyframe is found in the input stream of a > +decoder). > + > +The device MUST properly handle the case when a dequeued but still referenced > +resource is queued again. > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE}{Device Types / Video Device / Device Operation / > Device Operation: Stream commands / RESOURCE QUEUE} > + > +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE MUST be sent to inputqX or outputqX. > + > +\field{resource_id} MUST be an ID of a resource, that is both allocated and > +attached for the queue. > + > +The driver MUST be able to handle the output resources in decoding order in > +encoder case, i.e. with timestamps out of order. > + > +\subsubsection{Device Operation: Standalone Events} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events} > + > +These events are caused by state changes in the device, not as an async > +response to any command. > + > +\paragraph{Error Event} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Error Event} > + > +The error event is sent by the device when an unrecoverable error occurs > +during processing a stream. The device operation is the same as when > +it receives a VIRTIO_VIDEO_CMD_STREAM_CLOSE command, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} > +except that it also sets the VIRTIO_VIDEO_EVENT_FLAG_ERROR flag in the > +response. Note that this is different from other async responses carrying the > +VIRTIO_VIDEO_EVENT_FLAG_ERROR flag. The latter indicates that e.g. the > +particular output frame might be corrupted, but the stream still exists > +and can recover. > + > +\paragraph{Dynamic Parameters Change Event} > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events / Dynamic Parameters Change Event} > + > +A Dynamic Parameters Change (or DPC) event is sent by a decoder device when it > +detects that the parameters of the stream being decoded have changed. > +The device operation is the same as if it receives a > +VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS command in the inputqX at the exact same > +point in the stream, that changes outputqX parameters, see > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS}. > +This includes activating the outputqX block if necessary. > + > +% TODO add QoS events and overall think about quotas. Codecs are normally > +% limited by bandwidth/macroblocks per second. How can we accommodate this? > + > +\subsection{Device capabilities and parameters} > +\label{sec:Device Types / Video Device / Device capabilities and parameters} > + > +\subsubsection{VIRTIO_VIDEO_TLV_CODED_SET} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED} > + > +These parameters are defined for the coded parameter sets. > + > +\paragraph{VIRTIO_VIDEO_TLV_CODED_FORMAT} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / FORMAT} > + > +The following coded formats are defined: > + > +\begin{lstlisting} > +#define VIRTIO_VIDEO_CODED_FORMAT_MPEG2 1 /* MPEG-2 Part 2 (V4L2_PIX_FMT_MPEG2) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_MPEG4 2 /* MPEG-4 Part 2 (V4L2_PIX_FMT_MPEG4) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_H264 3 /* H.264 (V4L2_PIX_FMT_H264) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_HEVC 4 /* HEVC aka H.265 (V4L2_PIX_FMT_HEVC) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_VP8 5 /* VP8 (V4L2_PIX_FMT_VP8) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_VP9 6 /* VP9 (V4L2_PIX_FMT_VP9) */ > +#define VIRTIO_VIDEO_CODED_FORMAT_FWHT 7 /* FWHT (V4L2_PIX_FMT_FWHT) */ > +\end{lstlisting} > + > +The coded formats and the expected data units per buffer are documented in > +\hyperref[intro:V4L2]{V4L2 header} and > +\hyperref[intro:V4L2 compressed]{V4L2 compressed formats documentation}. > + > +\field{struct virtio_video_tlv_coded_format} represents both the coded format > +in a coded set of capabilities and the specific parameter values: > + > +\begin{lstlisting} > +struct virtio_video_tlv_coded_format { > + le32 format; /* VIRTIO_VIDEO_CODED_FORMAT_* */ > +}; > +\end{lstlisting} > + > +\paragraph{VIRTIO_VIDEO_TLV_CODED_RESOURCES} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCES} > + > +Setup common coded resources parameters. > + > +\field{struct virtio_video_tlv_coded_resources_caps} represents capabilities: > + > +\begin{lstlisting} > +struct virtio_video_tlv_coded_resources_caps { > + struct virtio_video_range num_resources_range; > + struct virtio_video_range resource_size_range; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{num_resources_range}] > + is the supported range of resources number of a particular coded set. > + \item[\field{resource_size_range}] > + is the supported range of resource sizes. > +\end{description} > + > +\field{struct virtio_video_tlv_resources_val} represents the parameter values: > + > +\begin{lstlisting} > +struct virtio_video_tlv_resources_val { > + le32 num_resources; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{num_resources}] > + is the number of resources that can be addressed for the queue, numbered > + from \(0\) to \(num\_resources - 1\). Setting this parameter to zero is > + allowed even when \field{num_resources_range.min} is positive, this results > + in detaching all the resources. > +\end{description} > + > +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCE GUEST PAGES} > + > +This TLV is defined in the same way as for the VIRTIO_VIDEO_TLV_RAW_SET, see > +\ref{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES}. > +Note, that a coded resource can only have a single buffer, so only the first > +element of \field{num_entries} is not zero. > + > +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCE VIRTIO OBJECT} > + > +This TLV is defined in the same way as for the VIRTIO_VIDEO_TLV_RAW_SET, see > +\ref{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT}. > +Note, that a coded resource can only have a single buffer, so > +\field{num_objects} is always 1. > + > +\paragraph{VIRTIO_VIDEO_TLV_V4L2_CONTROLS} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS} > + > +Inside this TLV container only selected V4L2 control IDs are allowed to be > +used as TLV types. The subset is listed in subsections below. Values of the > +V4L2 controls can be converted to TLVs by taking their representation as > +\hyperref[intro:V4L2 ext ctrls]{struct v4l2_ext_control} and replacing the > +pointers with values. Capabilities of the V4L2 controls can't be converted to > +TLVs as easily, so they are described below. This is mostly useful for > +encoders. Most relevant V4L2 controls are codec-specific. All definitions > +related to V4L2 controls can be found in > +\hyperref[intro:V4L2 controls]{V4L2 controls header}, their descriptions > +can be found in \hyperref[intro:V4L2 codec controls]{V4L2 documentation}. > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_V4L2_CONTROLS}{Device Types / Video Device / Device capabilities and parameters > / CODED / V4L2 CONTROLS} > + > +The device MUST NOT advertise codec-specific parameters not corresponding to > +the coded format of the particular coded set. > + > +\subparagraph{V4L2 controls: 32 bit integers} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: 32 bit integers} > + > +Integer V4L2 controls are signed by default, but this specification doesn't > +define any signed integer types, see \ref{sec:Structure Specifications}, so > +not every integer V4L2 control could be used directly. Still for many of them > +negative values don't make sense, so these controls are allowed in the range > +from 0 to INT32_MAX: > + > +\begin{itemize} > + \item V4L2_CID_MPEG_VIDEO_BITRATE: bitrate in bits per second > +\end{itemize} > + > +For capabilities the TLV value is defined as follows: > + > +\begin{lstlisting} > +struct virtio_video_tlv_v4l2_int_caps { > + struct virtio_video_range range; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{range}] > + is a range of possible values of the control. > +\end{description} > + > +For the control values the TLV value is defined as follows: > + > +\begin{lstlisting} > +struct virtio_video_tlv_v4l2_int_val { > + le32 value; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{value}] > + is one of the supported control values. > +\end{description} > + > +\drivernormative{\subparagraph}{V4L2 controls: 32 bit integers}{Device Types / Video Device / Device capabilities and parameters / CODED > / V4L2 CONTROLS / V4L2 controls: 32 bit integers} > + > +The integer V4L2 control values MUST be in the range from 0 to INT32_MAX. > + > +\subparagraph{V4L2 controls: Enumerations} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: Enumerations} > + > +The following V4L2 controls with values defined as enums are allowed: > +\begin{itemize} > + \item V4L2_CID_MPEG_VIDEO_MPEG2_PROFILE > + % enum v4l2_mpeg_video_mpeg2_profile: V4L2_MPEG_VIDEO_MPEG2_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_MPEG2_LEVEL > + % enum v4l2_mpeg_video_mpeg2_level: V4L2_MPEG_VIDEO_MPEG2_LEVEL_* > + \item V4L2_CID_MPEG_VIDEO_MPEG4_PROFILE > + % enum v4l2_mpeg_video_mpeg4_profile: V4L2_MPEG_VIDEO_MPEG4_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_MPEG4_LEVEL > + % enum v4l2_mpeg_video_mpeg4_level: V4L2_MPEG_VIDEO_MPEG4_LEVEL_* > + \item V4L2_CID_MPEG_VIDEO_H264_PROFILE > + % enum v4l2_mpeg_video_h264_profile: V4L2_MPEG_VIDEO_H264_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_H264_LEVEL > + % enum v4l2_mpeg_video_h264_level: V4L2_MPEG_VIDEO_H264_LEVEL_* > + \item V4L2_CID_MPEG_VIDEO_HEVC_PROFILE > + % enum v4l2_mpeg_video_hevc_profile: V4L2_MPEG_VIDEO_HEVC_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_HEVC_LEVEL > + % enum v4l2_mpeg_video_hevc_level: V4L2_MPEG_VIDEO_HEVC_LEVEL_* > + \item V4L2_CID_MPEG_VIDEO_VP8_PROFILE > + % enum v4l2_mpeg_video_vp8_profile: V4L2_MPEG_VIDEO_VP8_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_VP9_PROFILE > + % enum v4l2_mpeg_video_vp9_profile: V4L2_MPEG_VIDEO_VP9_PROFILE_* > + \item V4L2_CID_MPEG_VIDEO_VP9_LEVEL > + % enum v4l2_mpeg_video_vp9_level: V4L2_MPEG_VIDEO_VP9_LEVEL_* > +\end{itemize} > + > +For capabilities the TLV value is defined as follows: > + > +\begin{lstlisting} > +#define MASK(x) (1 << (x)) > + > +struct virtio_video_tlv_v4l2_enum_caps { > + le32 bitmask; /* Bitmask of MASK(<enum value>) */ > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{bitmask}] > + is a bitmask of supported enum values used as bit numbers, see > + \hyperref[intro:V4L2 controls]{V4L2 controls header}. > +\end{description} > + > +For the control values the TLV value is defined as follows: > + > +\begin{lstlisting} > +struct virtio_video_tlv_v4l2_enum_val { > + u8 value; /* <enum value> */ > + u8 padding[3]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{value}] > + is one of the supported enum values, see > + \hyperref[intro:V4L2 controls]{V4L2 controls header}. > +\end{description} > + > +\subsubsection{VIRTIO_VIDEO_TLV_RAW_SET} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW} > + > +These parameters are defined for the raw parameter sets. > + > +\paragraph{VIRTIO_VIDEO_TLV_RAW_FORMAT} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / FORMAT} > + > +DRM fourcc format definitions and DRM format modifiers are used to represent > +raw formats capabilities and values. The layouts of raw formats are documented > +in \hyperref[intro:DRM formats]{DRM} and \hyperref[intro:V4L2]{V4L2} headers, > +as well as in \hyperref[intro:V4L2 RGB]{V4L2 RGB} and > +\hyperref[intro:V4L2 YUV]{planar YUV} formats documentation. > + > +% Some DRM and V4L2 formats can be matched with this table: > +% DRM_FORMAT_ARGB8888 = V4L2_PIX_FMT_ABGR32 > +% DRM_FORMAT_BGRA8888 = V4L2_PIX_FMT_ARGB32 > +% DRM_FORMAT_RGBA8888 = V4L2_PIX_FMT_BGRA32 > +% DRM_FORMAT_NV12 = V4L2_PIX_FMT_NV12 > +% DRM_FORMAT_YUV420 = V4L2_PIX_FMT_YUV420 > +% DRM_FORMAT_YVU420 = V4L2_PIX_FMT_YVU420 > +% DRM_FORMAT_YUYV = V4L2_PIX_FMT_YUYV > + > +\field{struct virtio_video_tlv_raw_format_caps} is used to describe the > +capabilities: > + > +\begin{lstlisting} > +enum virtio_video_planes_layout { > + VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1, > + VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS = 2, > +}; > + > +struct virtio_video_tlv_raw_format_caps { > + le32 planes_layout_mask; /* Bitmask of VIRTIO_VIDEO_PLANES_LAYOUT_* */ > + le32 fourcc; /* DRM_FORMAT_* */ > + le64 modifier; /* DRM_FORMAT_MOD_* */ > + struct virtio_video_range width_range; > + struct virtio_video_range height_range; > + le32 stride_align_mask; > + le32 height_align_mask; > + le32 plane_align_mask; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{planes_layout_mask}] > + is a bitmask of supported planes layout types according to > + \field{enum virtio_video_planes_layout}. > + \item[\field{fourcc}] > + specifies the raw format, to which these capabilities apply. > + \item[\field{modifier}] > + specifies the raw format modifier. > + \item[\field{width_range}] > + is a range of widths in pixels. > + \item[\field{height_range}] > + is a range of heights in pixels. > + \item[\field{stride_align_mask}] > + is a bitmask of all supported power of two alignments of the distance in > + bytes between two lines of data (stride). > + \item[\field{height_align_mask}] > + is a bitmask of all supported power of two height alignments in pixels (scanlines). > + \item[\field{plane_align_mask}] > + is a bitmask of all supported power of two alignments in bytes of planes > + within a buffer. This field is valid only if \field{planes_layout_mask} has > + the \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit set. > +\end{description} > + > +\field{struct virtio_video_tlv_raw_format_val} is used to describe the > +values: > + > +\begin{lstlisting} > +struct virtio_video_tlv_raw_format_val { > + le32 planes_layout; /* VIRTIO_VIDEO_PLANES_LAYOUT_* */ > + le32 fourcc; /* DRM_FORMAT_* */ > + le64 modifier; /* DRM_FORMAT_MOD_* */ > + le32 width; > + le32 height; > + le32 stride_align; > + le32 height_align; > + le32 plane_align; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{planes_layout}] > + is the actual layout of the planes. > + \item[\field{fourcc}] > + specifies the raw format. > + \item[\field{modifier}] > + specifies the raw format modifier. > + \item[\field{width}] > + is the width in pixels of the stream frames. > + \item[\field{height}] > + is the height in pixels of the stream frames. > + \item[\field{stride_align}] > + is the power of two stride alignment in bytes. > + \item[\field{height_align}] > + is the power of two height alignment in pixels (scanlines). > + \item[\field{plane_align}] > + is the power of two alignment in bytes of planes within a buffer. This field > + is valid only if \field{planes_layout} has the > + \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit set. > +\end{description} > + > +% TODO: add colorimetry > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_RAW_FORMAT}{Device Types / Video Device / Device capabilities and parameters / > RAW / FORMAT} > + > +The device MUST set \field{VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER} bit in > +\field{planes_layout_mask} if the plane layout with planes of a frame laid out > +one after another in the same buffer is supported. > + > +The device MUST set \field{VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS} bit in > +\field{planes_layout_mask} if the plane layout with planes of a frame laid out > +in separate buffers is supported. > + > +% TODO: not sure if !VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG should be compatible > +% with VIRTIO_VIDEO_PLANES_LAYOUT_MULTI_BUFFERS or not. > + > +\paragraph{VIRTIO_VIDEO_TLV_RAW_RESOURCES} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCES} > + > +\field{struct virtio_video_tlv_raw_resources_caps} represents capabilities: > + > +\begin{lstlisting} > +struct virtio_video_tlv_raw_resources_caps { > + struct virtio_video_range num_resources_range; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{num_resources_range}] > + is the supported range of resources number of a particular raw set. > +\end{description} > + > +\field{struct virtio_video_tlv_resources_val} represents the parameter values, > +see > +\ref{sec:Device Types / Video Device / Device capabilities and parameters / CODED / RESOURCES}. > + > +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES} > + > +Setup guest pages as backing memory of a resource. > + > +The parameter capabilities are empty. The empty TLV with zero length indicates > +the support for attaching guest pages to resources. > + > +\field{struct virtio_video_tlv_resource_guest_pages} represents the parameter > +values: > + > +\begin{lstlisting} > +struct virtio_video_resource_sg_entry { > + le64 addr; > + le32 length; > + u8 padding[4]; > +}; > + > +struct virtio_video_tlv_resource_guest_pages { > + le32 resource_id; > + u8 padding[4]; > + le32 num_entries[VIRTIO_VIDEO_MAX_PLANES]; > + struct virtio_video_resource_sg_entry entries[]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{resource_id}] > + is the ID of the resource. > + \item[\field{num_entries}] > + is the number of scatter-gather list entries in each of the separate buffers > + forming together the resource according to currently set format. Unused > + array elements are set to 0. Sum of the array is the length of the > + \field{entries} array. > + \item[\field{entries}] > + is an array of the scatter-gather list entries: > + > + \begin{description} > + \item[\field{addr}] > + is a guest physical address of the start of the SG entry aligned to > + the physical guest pages size. > + \item[\field{length}] > + is the length of the SG entry in bytes aligned to the physical guest > + pages size. > + \end{description} > +\end{description} > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES}{Device Types / Video Device / Device capabilities and > parameters / RAW / RESOURCE GUEST PAGES} > + > +\field{resource_id} MUST be an integer within the range of resource IDs > +currently allocated for the queue. > + > +The memory regions identified by the elements of the \field{entries} array > +MUST NOT overlap. > + > +\paragraph{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT} > + > +Setup virtio objects as backing memory to a resource. > + > +The parameter capabilities are empty. The empty TLV with zero length indicates > +the support for attaching virtio objects to resources. > + > +\field{struct virtio_video_tlv_resource_virtio_object} represents the parameter > +values: > + > +\begin{lstlisting} > +struct virtio_video_resource_object { > + u8 uuid[16]; > +}; > + > +struct virtio_video_tlv_resource_virtio_object { > + le32 resource_id; > + le32 num_objects; /* Up to VIRTIO_VIDEO_MAX_PLANES */ > + struct virtio_video_resource_object objects[num_objects]; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{resource_id}] > + is the ID of the resource. > + \item[\field{num_objects}] > + is the length of the \field{objects} array according to currently set > + format. > + \item[\field{object}] > + is an array of objects exported from another virtio device, see > + \ref{sec:Basic Facilities of a Virtio Device / Exporting Objects}. > + > + \begin{description} > + \item[uuid] > + is a version 4 UUID specified by \hyperref[intro:rfc4122]{[RFC4122]}. > + \end{description} > +\end{description} > + > +\drivernormative{\subparagraph}{VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT}{Device Types / Video Device / Device capabilities and > parameters / RAW / RESOURCE VIRTIO OBJECT} > + > +\field{resource_id} MUST be an integer within the range of resource IDs > +currently allocated for the queue. > + > +\paragraph{VIRTIO_VIDEO_TLV_CROP} > +\label{sec:Device Types / Video Device / Device capabilities and parameters / RAW / CROP} > + > +% TODO There is no reason in doing crop if it doesn't affect the output. > +% So the output frames have to be smaller, than the full size. In the decoder > +% case this means, that the buffers can't be used as a reference. So when crop > +% is enabled, the decoder probably has to have some intermediate buffers. > +% Is it reasonable to reallocate output buffers then? It could be. So it has > +% to be decided (probably in the same way it is usually done in V4L2). In the > +% encoder case this is not a problem. > +% Is setting compose rectangles useful at all? > + > +This parameter sets a rectangle covering the visible area of the frame. > + > +The parameter capabilities are empty. The empty TLV with zero length indicates > +the support for cropping. > + > +The parameter value is defined as follows: > + > +\begin{lstlisting} > +struct virtio_video_tlv_crop_val { > + le32 left; > + le32 top; > + le32 width; > + le32 height; > +}; > +\end{lstlisting} > + > +\begin{description} > + \item[\field{left, top}] > + are coordinates of top left corner of the crop rectangle in pixels. > + \item[\field{width, height}] > + are dimensions of the crop rectangle in pixels. > +\end{description} > + > +\devicenormative{\subparagraph}{VIRTIO_VIDEO_TLV_CROP}{Device Types / Video Device / Device capabilities and parameters / RAW / > CROP} > + > +The crop rectangle MUST be reset to full frame size on every resolution > +change. > diff --git a/device-types/video/device-conformance.tex b/device-types/video/device-conformance.tex > new file mode 100644 > index 0000000..64960ac > --- /dev/null > +++ b/device-types/video/device-conformance.tex > @@ -0,0 +1,22 @@ > +\conformance{\subsection}{Video Device Conformance} > +\label{sec:Conformance / Device Conformance / Video Device Conformance} > + > +A video device MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{devicenormative:Device Types / Video Device / Feature bits} > +\item \ref{devicenormative:Device Types / Video Device / Device configuration layout} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: TLV format} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} > +\item \ref{devicenormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} > +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS} > +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / RAW / FORMAT} > +\item \ref{devicenormative:Device Types / Video Device / Device capabilities and parameters / RAW / CROP} > +\end{itemize} > diff --git a/device-types/video/driver-conformance.tex b/device-types/video/driver-conformance.tex > new file mode 100644 > index 0000000..ca59885 > --- /dev/null > +++ b/device-types/video/driver-conformance.tex > @@ -0,0 +1,20 @@ > +\conformance{\subsection}{Video Driver Conformance} > +\label{sec:Conformance / Driver Conformance / Video Driver Conformance} > + > +A video driver MUST conform to the following normative statements: > + > +\begin{itemize} > +\item \ref{drivernormative:Device Types / Video Device / Feature bits} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / OPEN} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / CLOSE} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / SET PARAMS} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / DRAIN} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / QUEUE RESET} > +\item \ref{drivernormative:Device Types / Video Device / Device Operation / Device Operation: Stream commands / RESOURCE QUEUE} > +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / CODED / V4L2 CONTROLS / V4L2 controls: 32 > bit integers} > +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE GUEST PAGES} > +\item \ref{drivernormative:Device Types / Video Device / Device capabilities and parameters / RAW / RESOURCE VIRTIO OBJECT} > +\end{itemize} > diff --git a/introduction.tex b/introduction.tex > index 9a9cbde..52efa16 100644 > --- a/introduction.tex > +++ b/introduction.tex > @@ -113,6 +113,15 @@ \section{Normative References}\label{sec:Normative References} > \phantomsection\label{intro:SEC1}\textbf{[SEC1]} & > Standards for Efficient Cryptography Group(SECG), ``SEC1: Elliptic Cureve Cryptography'', Version 1.0, September 2000. > \newline\url{https://www.secg.org/sec1-v2.pdf}\\ > + \phantomsection\label{intro:V4L2}\textbf{[V4L2]} & > + Linux V4L2 interface. > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/videodev2.h}\\ > + \phantomsection\label{intro:V4L2 controls}\textbf{[V4L2 Controls]} & > + Linux V4L2 controls definitions. > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/v4l2-controls.h}\\ > + \phantomsection\label{intro:DRM formats}\textbf{[DRM Formats]} & > + Linux DRM format definitions. > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/drm/drm_fourcc.h}\\ > > \phantomsection\label{intro:rfc2784}\textbf{[RFC2784]} & > Generic Routing Encapsulation. This protocol is only specified for IPv4 and used as either the payload or delivery protocol. > @@ -194,6 +203,18 @@ \section{Non-Normative References} > \phantomsection\label{intro:Virtio PCI Draft}\textbf{[Virtio PCI Draft]} & > Virtio PCI Draft Specification > \newline\url{http://ozlabs.org/~rusty/virtio-spec/virtio-0.9.5.pdf}\\ > + \phantomsection\label{intro:V4L2 compressed}\textbf{[V4L2 compressed formats]} & > + Detailed descriptions of V4L2 compressed formats > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace- > api/media/v4l/pixfmt-compressed.rst}\\ > + \phantomsection\label{intro:V4L2 RGB}\textbf{[V4L2 RGB formats]} & > + Detailed descriptions of V4L2 RGB formats > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace- > api/media/v4l/pixfmt-rgb.rst}\\ > + \phantomsection\label{intro:V4L2 YUV}\textbf{[V4L2 planar YUV formats]} & > + Detailed descriptions of V4L2 planar YUV formats > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace- > api/media/v4l/pixfmt-yuv-planar.rst}\\ > + \phantomsection\label{intro:V4L2 codec controls}\textbf{[V4L2 codec controls]} & > + Detailed descriptions of V4L2 controls > + \newline\url{https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/ext- > ctrls-codec.rst}\\ > \end{longtable} > > \section{Terminology}\label{Terminology} > -- > 2.43.0 > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-15 10:49 ` Parav Pandit @ 2026-02-15 15:11 ` Michael S. Tsirkin 2026-02-25 14:21 ` Alexander Gordeev 2026-02-24 20:02 ` Alexander Gordeev 1 sibling, 1 reply; 9+ messages in thread From: Michael S. Tsirkin @ 2026-02-15 15:11 UTC (permalink / raw) To: Parav Pandit Cc: Alexander Gordeev, virtio-comment@lists.linux.dev, Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org On Sun, Feb 15, 2026 at 10:49:22AM +0000, Parav Pandit wrote: > > > From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> > > Sent: 13 February 2026 02:23 PM > > > > From: Alexander Gordeev <Alexander.Gordeev@opensynergy.com> > > > > Add the specification of the video decoder and encoder device, which > > can be used to provide host-accelerated video operations to the guest. > > > > Signed-off-by: Alexander Gordeev <alexander.gordeev@opensynergy.com> > > --- > > conformance.tex | 12 +- > > content.tex | 1 + > > device-types/video/description.tex | 1592 +++++++++++++++++++++ > > device-types/video/device-conformance.tex | 22 + > > device-types/video/driver-conformance.tex | 20 + > > introduction.tex | 21 + > > 6 files changed, 1664 insertions(+), 4 deletions(-) > > create mode 100644 device-types/video/description.tex > > create mode 100644 device-types/video/device-conformance.tex > > create mode 100644 device-types/video/driver-conformance.tex > > > > diff --git a/conformance.tex b/conformance.tex > > index 9af31e2..f34e600 100644 > > --- a/conformance.tex > > +++ b/conformance.tex > > @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > > \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, > > \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, > > \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, > > -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or > > -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. > > +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, > > +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or > > +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. > > > > \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. > > \end{itemize} > > @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > > \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, > > \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, > > \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, > > -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or > > -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. > > +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, > > +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or > > +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. > > > > \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. > > \end{itemize} > > @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > > \input{device-types/spi/driver-conformance.tex} > > \input{device-types/media/driver-conformance.tex} > > \input{device-types/rtc/driver-conformance.tex} > > +\input{device-types/video/driver-conformance.tex} > > > > \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} > > > > @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} > > \input{device-types/spi/device-conformance.tex} > > \input{device-types/media/device-conformance.tex} > > \input{device-types/rtc/device-conformance.tex} > > +\input{device-types/video/device-conformance.tex} > > > > \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy > > Interface: Transitional Device and Transitional Driver Conformance} > > A conformant implementation MUST be either transitional or > > diff --git a/content.tex b/content.tex > > index 5de811f..0c13f68 100644 > > --- a/content.tex > > +++ b/content.tex > > @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} > > \input{device-types/spi/description.tex} > > \input{device-types/media/description.tex} > > \input{device-types/rtc/description.tex} > > +\input{device-types/video/description.tex} > > > > \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} > > > > diff --git a/device-types/video/description.tex b/device-types/video/description.tex > > new file mode 100644 > > index 0000000..8945e26 > > --- /dev/null > > +++ b/device-types/video/description.tex > > @@ -0,0 +1,1592 @@ > > +\section{Video Device} > > +\label{sec:Device Types / Video Device} > > + > > +The virtio video device provides support for host-accelerated video encoding > > +and decoding. > > + > > +\subsection{Device ID} > > +\label{sec:Device Types / Video Device / Device ID} > > + > > +50 > > + > > +\subsection{Virtqueues} > > +\label{sec:Device Types / Video Device / Virtqueues} > > + > > +\begin{description} > > + \item[0] > > + commandq - driver commands > > + \item[1] > > + eventq - device async responses to commands and standalone device events > > +\end{description} > > + > > +\subsection{Feature bits} > > +\label{sec:Device Types / Video Device / Feature bits} > > + > > +\begin{description} > > + \item[VIRTIO_VIDEO_F_ENCODER (0)] > > + The device can encode video. > > + \item[VIRTIO_VIDEO_F_DECODER (1)] > > + The device can decode video. > > + % Use-case: the device can support both encoding and decoding, so having both > > + % here can save resources. > > + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] > > + Guest pages can be used as the backing memory of resources. > > + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] > > + The device can use non-contiguous guest memory as the backing memory of > > + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also > > + set. > > + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] > > + Objects exported by another virtio device can be used as the backing memory > > + of resources. > > + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] > > + The device releases an extra empty output buffer after a drain or DPC so that > > + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. > This does not seem relavant to the video device. > It should not be attached to any V4L2 implementation. > Can you please craft it differently? My understanding is that this is a V4L2 device just like virtio input makes the linux input layer into a device. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > > + > > +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. > > + > > +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or > > +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would > > +mean that no memory can be used at all for resources. > > + > > +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets > > +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. > > + > > +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > > + > > +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and > > +VIRTIO_VIDEO_F_DECODER features. > > + > > +The driver MUST negotiate at least one of the > > +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT > > +features. > > + > You wouldn't need virtio_object feature bit. It should be covered using the capability. To be frank I'm pretty split on this. Capabilities make sense if there are envisioned to be lots of these. I'll let the contributor decide if that is the case. > > +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not > > +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically > > +contiguous memory for all the buffers it allocates. > > + > > +\subsection{Device configuration layout} > > +\label{sec:Device Types / Video Device / Device configuration layout} > > + > > +The video device configuration space uses the following layout: > > + > > +\begin{lstlisting} > > +struct virtio_video_config { > > + le32 max_streams; > > + le32 caps_length; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > + \item[\field{max_streams}] > > + is the maximum number of concurrent streams the device supports. > > + \item[\field{caps_length}] > > + is the minimum length in bytes that a device-writable buffer must have > > + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see > > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > > +\end{description} > > + > > +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} > > + > > +\field{max_streams} MUST be positive. > > + > > +\field{caps_length} MUST be set to the response size of > > +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. > > + > This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. > Please rework the patch to use the existing basic facility. > This can also possibly eliminate plumbing device specific command q. > More below. admin commands are kind of heavyweight, they are designed for control path. Question is, is this a data path or a control path thing? If data path it is not appropriate, if control path it is. > > +\subsection{Device Initialization} > > +\label{sec:Device Types / Video Device / Device Initialization} > > +\begin{enumerate} > > + \item > > + The driver reads the feature bits and negotiates the features it needs. > > + \item > > + The driver sets up the commandq and the eventq. > > + \item > > + The driver reads the \field{caps_length} field of the configuration > > + space, prepares a buffer of at least that size and sends the buffer on the > > + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. > > + \item > > + The device sends a response over commandq to > > + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the > > + command. > > + \item > > + The driver receives the response from the device, and parses its capabilities. > > +\end{enumerate} > > + > > +\subsection{Device Operation} > > +\label{sec:Device Types / Video Device / Device Operation} > > + > > +The device supports opening and operating a number of parallel streams up to > > +\field{max_streams}. Each stream has three internal device queues: mainqX, > > +inputqX and outputqX, where X is the stream id. Each stream command has a > > +field that dispatches the command to the specific internal queue. > > + > If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, > Than keyword 'internal' should be dropped and rephased. I think the implication is that within each queue commands are consumed in order? This isn't terribly clear. > > +% Use-case: there might be different real-time requirements for different > > +% streams, so more virtqueues can be added in the future if necessary. > > +% The internal queues don't change, the data formats don't change, only the > > +% mapping of streams/internal queues to particular virtqueues changes. > > + > > +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, > > +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX > > +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out > > +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock > > +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for > > +format negotiation. > > + > > +The inputqX and outputqX are used to queue input or output resources using > > +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and > > +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete > > +processing of all queued input resources and make the resulting output > > +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. > > + > > +All the stream commands start async operations, return the results > > +using async responses over eventq. > > +The eventq is used by the device to send the device's async responses to > > +stream commands and the device's standalone events. > > + > > +% This way eventq becomes the single source of truth about the device's state, > > +% the driver doesn't have to tediously synchronize commandq's and eventq's > > +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF > > +% and DQEVENT). One more benefit is that commandq is processed fast and > > +% strictly in order, so commandq descriptors exhaustion should never happen in > > +% practice. > > + > Video device should be implementable without any V4L2 binding/description etc. > > > +Parameters allow the driver to configure the stream including setting up the > > +resources. Available parameters depend on the device type, see > > +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. > > + > > +A resource is a set of memory buffers that contain a unit of data that > > +the device can process or produce. Most resources have only one buffer, > > +raw frames using a multi-planar format can have several. > > +Input resources are filled by the driver with compressed (coded) video data > > +for a decoder and raw frames for an encoder, output resources are filled by > > +the device as the result of processing the input resources with decoded raw > > +frames for a decoder and compressed (encoded) data for an encoder. > > +Resources from inputqX and outputqX are consumed independently, not in pairs. > > +One input resource can result in zero to many produced output resources. > > +A decoder device dequeues the output decoded frames in presentation order. > > +An encoder device dequeues the output decoded frames in decoding order. > > +The driver can reuse a queued resource after receiving a corresponding async > > +response. Dequeued output resources can still be used by the device as > > +reference frames, so the driver can't write to them. > > + > > +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued > > +% output resource is not used by the device anymore and therefore becomes > > +% writeable? > > + > > +The device can detect standalone stream-related events: errors and dynamic > > +parameters changes that require intervention from the driver (e.g. > > +reallocating backing memory of output resources to fit the new parameters). > > +The events are signalled on the eventq, see > > +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > > + > > +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} > > + > > +The device MUST set to zero all unused, disabled or padding bits in its > > +responses. > > + > > +\subsubsection{Device Operation: Command Virtqueue} > > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} > > + > > +This section lists the commands that can be sent by the driver to commandq. > > + > > +Different structures are used for each command and response. A command > > +structure starts with the requested command code, defined as follows: > > + > > +\begin{lstlisting} > > +/* Device */ > > +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 > > + > > +/* Stream */ > > +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 > > +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 > You can craft the stream using a existing basic facility of resource object. > Where each stream is just a resource object, that be queried or modified. > > > +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 > > +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 > > +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 > > +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 > > +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 > > +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 > > +\end{lstlisting} > > + > > +Stream commands start with a header: > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 > > +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 > > +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 > > + > > +struct virtio_video_stream_cmd_header { > > + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ > > + le32 stream_id; > > + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ > > + le32 async_response_cookie; > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > + \item[\field{async_response_cookie}] > > + is an async response cookie provided by the driver, that allows > > + to relate an async response to the previously submitted command. > > +\end{description} > > + > > +\subsubsection{Device Operation: Event Virtqueue} > > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} > > + > > +The eventq is used by the device to send async responses to commands queued > > +by the driver on commandq and standalone events. Stream errors and dynamic > > +parameters changes are caused by changes in the device's state, not by > > +commands, still they are delivered as responses to implicit > > +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, > > +respectively. > > + > > +Events start with a header: > > + > > +\begin{lstlisting} > > +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) > > +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) > > +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) > > +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) > > + > > +struct virtio_video_event_header { > > + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ > > + le32 stream_id; > > + le32 async_response_cookie; > > + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ > > +}; > > +\end{lstlisting} > > + > > +\begin{description} > > + \item[\field{event_type}] > > + is the type of the event. > > + \item[\field{stream_id}] > > + is the ID of a valid stream. > > + \item[\field{async_response_cookie}] > > + is an async response cookie provided by the driver, that allows > > + to relate the event to a previously submitted command. > > + \item[\field{event_flags}] > > + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags > > + > > + \begin{description} > > + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] > > + is set if the command finished with an error due to an > > + invalid argument or for other reasons. > > + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] > > + is set for standalone events, see > > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > > + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] > > + is set if the command has been canceled by another > > + command, that has higher priority. Doesn't make sense > > + for standalone events. > > + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] > > + is set if the command triggered a block on the > > + outputqX to allow output format negotiation. > > + When the negotiation is finished the block has to be > > + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK > > + command, see > > + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. > > + \end{description} > > +\end{description} > > + > > +The particular data structure representing the event is selected according to > > +the \field{event_type}. > > + > > +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: > > Event Virtqueue} > > + > > +The driver MUST at any time have at least one descriptor with a used > > +buffer large enough to contain a \field{struct virtio_video_event} > > +queued on the eventq. > > + > > +The driver MUST NOT put device-readable descriptors into the eventq. > > + > > +The driver MUST account for the fact that the async responses to commands > > +might come out-of-order (i.e. after other commands sent to the device), > > +and that some of them can be cancelled. > > + > > +The driver SHOULD wait for an async response of command A, that caused > > +cancellation of command B, before queueing the command B again. > > + > > +\subsubsection{Device Operation: TLV format} > > +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} > > + > > +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ > > +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS > > +commands represent device capabilities and corresponding device parameters > > +in the form of TLV (Type-Length-Value): > > + > > +\begin{lstlisting} > > +struct virtio_video_tlv { > > + le32 type; > > + le32 length; > > + u8 value[length]; > > +}; > > +\end{lstlisting} will cause all kind of padding mischief if length is not a multiple of 4. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-15 15:11 ` Michael S. Tsirkin @ 2026-02-25 14:21 ` Alexander Gordeev 0 siblings, 0 replies; 9+ messages in thread From: Alexander Gordeev @ 2026-02-25 14:21 UTC (permalink / raw) To: Michael S. Tsirkin, Parav Pandit Cc: virtio-comment@lists.linux.dev, Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org On 15/02/2026 16:11, Michael S. Tsirkin wrote: > On Sun, Feb 15, 2026 at 10:49:22AM +0000, Parav Pandit wrote: >> >>> From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> >>> Sent: 13 February 2026 02:23 PM >>> >>> From: Alexander Gordeev <Alexander.Gordeev@opensynergy.com> >>> >>> Add the specification of the video decoder and encoder device, which >>> can be used to provide host-accelerated video operations to the guest. >>> >>> Signed-off-by: Alexander Gordeev <alexander.gordeev@opensynergy.com> >>> --- >>> conformance.tex | 12 +- >>> content.tex | 1 + >>> device-types/video/description.tex | 1592 +++++++++++++++++++++ >>> device-types/video/device-conformance.tex | 22 + >>> device-types/video/driver-conformance.tex | 20 + >>> introduction.tex | 21 + >>> 6 files changed, 1664 insertions(+), 4 deletions(-) >>> create mode 100644 device-types/video/description.tex >>> create mode 100644 device-types/video/device-conformance.tex >>> create mode 100644 device-types/video/driver-conformance.tex >>> >>> diff --git a/conformance.tex b/conformance.tex >>> index 9af31e2..f34e600 100644 >>> --- a/conformance.tex >>> +++ b/conformance.tex >>> @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, >>> \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, >>> \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, >>> -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or >>> -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. >>> +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, >>> +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or >>> +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. >>> >>> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >>> \end{itemize} >>> @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, >>> \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, >>> \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, >>> -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or >>> -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. >>> +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, >>> +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or >>> +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. >>> >>> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >>> \end{itemize} >>> @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \input{device-types/spi/driver-conformance.tex} >>> \input{device-types/media/driver-conformance.tex} >>> \input{device-types/rtc/driver-conformance.tex} >>> +\input{device-types/video/driver-conformance.tex} >>> >>> \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} >>> >>> @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \input{device-types/spi/device-conformance.tex} >>> \input{device-types/media/device-conformance.tex} >>> \input{device-types/rtc/device-conformance.tex} >>> +\input{device-types/video/device-conformance.tex} >>> >>> \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy >>> Interface: Transitional Device and Transitional Driver Conformance} >>> A conformant implementation MUST be either transitional or >>> diff --git a/content.tex b/content.tex >>> index 5de811f..0c13f68 100644 >>> --- a/content.tex >>> +++ b/content.tex >>> @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} >>> \input{device-types/spi/description.tex} >>> \input{device-types/media/description.tex} >>> \input{device-types/rtc/description.tex} >>> +\input{device-types/video/description.tex} >>> >>> \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} >>> >>> diff --git a/device-types/video/description.tex b/device-types/video/description.tex >>> new file mode 100644 >>> index 0000000..8945e26 >>> --- /dev/null >>> +++ b/device-types/video/description.tex >>> @@ -0,0 +1,1592 @@ >>> +\section{Video Device} >>> +\label{sec:Device Types / Video Device} >>> + >>> +The virtio video device provides support for host-accelerated video encoding >>> +and decoding. >>> + >>> +\subsection{Device ID} >>> +\label{sec:Device Types / Video Device / Device ID} >>> + >>> +50 >>> + >>> +\subsection{Virtqueues} >>> +\label{sec:Device Types / Video Device / Virtqueues} >>> + >>> +\begin{description} >>> + \item[0] >>> + commandq - driver commands >>> + \item[1] >>> + eventq - device async responses to commands and standalone device events >>> +\end{description} >>> + >>> +\subsection{Feature bits} >>> +\label{sec:Device Types / Video Device / Feature bits} >>> + >>> +\begin{description} >>> + \item[VIRTIO_VIDEO_F_ENCODER (0)] >>> + The device can encode video. >>> + \item[VIRTIO_VIDEO_F_DECODER (1)] >>> + The device can decode video. >>> + % Use-case: the device can support both encoding and decoding, so having both >>> + % here can save resources. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] >>> + Guest pages can be used as the backing memory of resources. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] >>> + The device can use non-contiguous guest memory as the backing memory of >>> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also >>> + set. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] >>> + Objects exported by another virtio device can be used as the backing memory >>> + of resources. >>> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] >>> + The device releases an extra empty output buffer after a drain or DPC so that >>> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. >> This does not seem relavant to the video device. >> It should not be attached to any V4L2 implementation. >> Can you please craft it differently? > > My understanding is that this is a V4L2 device just like virtio input makes > the linux input layer into a device. Well, the device isn't really V4L2 and doesn't even use V4L2, but indeed the V4L2 driver is our main target for now. There is already virtio-media for exposing V4L2 devices from the host. There are other drivers to come though, so we're going to have setups with no V4L2 at all. >>> +\end{description} >>> + >>> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>> + >>> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. >>> + >>> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or >>> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would >>> +mean that no memory can be used at all for resources. >>> + >>> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets >>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. >>> + >>> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>> + >>> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and >>> +VIRTIO_VIDEO_F_DECODER features. >>> + >>> +The driver MUST negotiate at least one of the >>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT >>> +features. >>> + >> You wouldn't need virtio_object feature bit. It should be covered using the capability. > > To be frank I'm pretty split on this. Capabilities make sense > if there are envisioned to be lots of these. > I'll let the contributor decide if that is the case. I'd rather not change this. We won't have many flags I think. >>> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not >>> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically >>> +contiguous memory for all the buffers it allocates. >>> + >>> +\subsection{Device configuration layout} >>> +\label{sec:Device Types / Video Device / Device configuration layout} >>> + >>> +The video device configuration space uses the following layout: >>> + >>> +\begin{lstlisting} >>> +struct virtio_video_config { >>> + le32 max_streams; >>> + le32 caps_length; >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{max_streams}] >>> + is the maximum number of concurrent streams the device supports. >>> + \item[\field{caps_length}] >>> + is the minimum length in bytes that a device-writable buffer must have >>> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. >>> +\end{description} >>> + >>> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} >>> + >>> +\field{max_streams} MUST be positive. >>> + >>> +\field{caps_length} MUST be set to the response size of >>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. >>> + >> This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. >> Please rework the patch to use the existing basic facility. >> This can also possibly eliminate plumbing device specific command q. >> More below. > > admin commands are kind of heavyweight, they are designed for control > path. Question is, is this a data path or a control path thing? > If data path it is not appropriate, if control path it is. > > >>> +\subsection{Device Initialization} >>> +\label{sec:Device Types / Video Device / Device Initialization} >>> +\begin{enumerate} >>> + \item >>> + The driver reads the feature bits and negotiates the features it needs. >>> + \item >>> + The driver sets up the commandq and the eventq. >>> + \item >>> + The driver reads the \field{caps_length} field of the configuration >>> + space, prepares a buffer of at least that size and sends the buffer on the >>> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. >>> + \item >>> + The device sends a response over commandq to >>> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the >>> + command. >>> + \item >>> + The driver receives the response from the device, and parses its capabilities. >>> +\end{enumerate} >>> + >>> +\subsection{Device Operation} >>> +\label{sec:Device Types / Video Device / Device Operation} >>> + >>> +The device supports opening and operating a number of parallel streams up to >>> +\field{max_streams}. Each stream has three internal device queues: mainqX, >>> +inputqX and outputqX, where X is the stream id. Each stream command has a >>> +field that dispatches the command to the specific internal queue. >>> + >> If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, >> Than keyword 'internal' should be dropped and rephased. > > I think the implication is that within each queue commands > are consumed in order? > > This isn't terribly clear. Noted, I'll make sure to improve the description in the next draft. >>> +% Use-case: there might be different real-time requirements for different >>> +% streams, so more virtqueues can be added in the future if necessary. >>> +% The internal queues don't change, the data formats don't change, only the >>> +% mapping of streams/internal queues to particular virtqueues changes. >>> + >>> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, >>> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX >>> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out >>> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock >>> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for >>> +format negotiation. >>> + >>> +The inputqX and outputqX are used to queue input or output resources using >>> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and >>> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete >>> +processing of all queued input resources and make the resulting output >>> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. >>> + >>> +All the stream commands start async operations, return the results >>> +using async responses over eventq. >>> +The eventq is used by the device to send the device's async responses to >>> +stream commands and the device's standalone events. >>> + >>> +% This way eventq becomes the single source of truth about the device's state, >>> +% the driver doesn't have to tediously synchronize commandq's and eventq's >>> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF >>> +% and DQEVENT). One more benefit is that commandq is processed fast and >>> +% strictly in order, so commandq descriptors exhaustion should never happen in >>> +% practice. >>> + >> Video device should be implementable without any V4L2 binding/description etc. >> >>> +Parameters allow the driver to configure the stream including setting up the >>> +resources. Available parameters depend on the device type, see >>> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. >>> + >>> +A resource is a set of memory buffers that contain a unit of data that >>> +the device can process or produce. Most resources have only one buffer, >>> +raw frames using a multi-planar format can have several. >>> +Input resources are filled by the driver with compressed (coded) video data >>> +for a decoder and raw frames for an encoder, output resources are filled by >>> +the device as the result of processing the input resources with decoded raw >>> +frames for a decoder and compressed (encoded) data for an encoder. >>> +Resources from inputqX and outputqX are consumed independently, not in pairs. >>> +One input resource can result in zero to many produced output resources. >>> +A decoder device dequeues the output decoded frames in presentation order. >>> +An encoder device dequeues the output decoded frames in decoding order. >>> +The driver can reuse a queued resource after receiving a corresponding async >>> +response. Dequeued output resources can still be used by the device as >>> +reference frames, so the driver can't write to them. >>> + >>> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued >>> +% output resource is not used by the device anymore and therefore becomes >>> +% writeable? >>> + >>> +The device can detect standalone stream-related events: errors and dynamic >>> +parameters changes that require intervention from the driver (e.g. >>> +reallocating backing memory of output resources to fit the new parameters). >>> +The events are signalled on the eventq, see >>> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>> + >>> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} >>> + >>> +The device MUST set to zero all unused, disabled or padding bits in its >>> +responses. >>> + >>> +\subsubsection{Device Operation: Command Virtqueue} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} >>> + >>> +This section lists the commands that can be sent by the driver to commandq. >>> + >>> +Different structures are used for each command and response. A command >>> +structure starts with the requested command code, defined as follows: >>> + >>> +\begin{lstlisting} >>> +/* Device */ >>> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 >>> + >>> +/* Stream */ >>> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 >>> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 >> You can craft the stream using a existing basic facility of resource object. >> Where each stream is just a resource object, that be queried or modified. >> >>> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 >>> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 >>> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 >>> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 >>> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 >>> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 >>> +\end{lstlisting} >>> + >>> +Stream commands start with a header: >>> + >>> +\begin{lstlisting} >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 >>> + >>> +struct virtio_video_stream_cmd_header { >>> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ >>> + le32 stream_id; >>> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ >>> + le32 async_response_cookie; >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{async_response_cookie}] >>> + is an async response cookie provided by the driver, that allows >>> + to relate an async response to the previously submitted command. >>> +\end{description} >>> + >>> +\subsubsection{Device Operation: Event Virtqueue} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} >>> + >>> +The eventq is used by the device to send async responses to commands queued >>> +by the driver on commandq and standalone events. Stream errors and dynamic >>> +parameters changes are caused by changes in the device's state, not by >>> +commands, still they are delivered as responses to implicit >>> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, >>> +respectively. >>> + >>> +Events start with a header: >>> + >>> +\begin{lstlisting} >>> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) >>> + >>> +struct virtio_video_event_header { >>> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ >>> + le32 stream_id; >>> + le32 async_response_cookie; >>> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{event_type}] >>> + is the type of the event. >>> + \item[\field{stream_id}] >>> + is the ID of a valid stream. >>> + \item[\field{async_response_cookie}] >>> + is an async response cookie provided by the driver, that allows >>> + to relate the event to a previously submitted command. >>> + \item[\field{event_flags}] >>> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags >>> + >>> + \begin{description} >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] >>> + is set if the command finished with an error due to an >>> + invalid argument or for other reasons. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] >>> + is set for standalone events, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] >>> + is set if the command has been canceled by another >>> + command, that has higher priority. Doesn't make sense >>> + for standalone events. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] >>> + is set if the command triggered a block on the >>> + outputqX to allow output format negotiation. >>> + When the negotiation is finished the block has to be >>> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK >>> + command, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. >>> + \end{description} >>> +\end{description} >>> + >>> +The particular data structure representing the event is selected according to >>> +the \field{event_type}. >>> + >>> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: >>> Event Virtqueue} >>> + >>> +The driver MUST at any time have at least one descriptor with a used >>> +buffer large enough to contain a \field{struct virtio_video_event} >>> +queued on the eventq. >>> + >>> +The driver MUST NOT put device-readable descriptors into the eventq. >>> + >>> +The driver MUST account for the fact that the async responses to commands >>> +might come out-of-order (i.e. after other commands sent to the device), >>> +and that some of them can be cancelled. >>> + >>> +The driver SHOULD wait for an async response of command A, that caused >>> +cancellation of command B, before queueing the command B again. >>> + >>> +\subsubsection{Device Operation: TLV format} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} >>> + >>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ >>> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS >>> +commands represent device capabilities and corresponding device parameters >>> +in the form of TLV (Type-Length-Value): >>> + >>> +\begin{lstlisting} >>> +struct virtio_video_tlv { >>> + le32 type; >>> + le32 length; >>> + u8 value[length]; >>> +}; >>> +\end{lstlisting} > > will cause all kind of padding mischief if length is not a multiple of > 4. Yes, true. It is indeed mentioned further: + \item[\field{length}] + specifies the \field{value} size in bytes aligned to 4 bytes. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-15 10:49 ` Parav Pandit 2026-02-15 15:11 ` Michael S. Tsirkin @ 2026-02-24 20:02 ` Alexander Gordeev 2026-02-25 4:59 ` Parav Pandit 1 sibling, 1 reply; 9+ messages in thread From: Alexander Gordeev @ 2026-02-24 20:02 UTC (permalink / raw) To: Parav Pandit, virtio-comment@lists.linux.dev Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org On 15/02/2026 11:49, Parav Pandit wrote: > >> From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> >> Sent: 13 February 2026 02:23 PM >> >> From: Alexander Gordeev <Alexander.Gordeev@opensynergy.com> >> >> Add the specification of the video decoder and encoder device, which >> can be used to provide host-accelerated video operations to the guest. >> >> Signed-off-by: Alexander Gordeev <alexander.gordeev@opensynergy.com> >> --- >> conformance.tex | 12 +- >> content.tex | 1 + >> device-types/video/description.tex | 1592 +++++++++++++++++++++ >> device-types/video/device-conformance.tex | 22 + >> device-types/video/driver-conformance.tex | 20 + >> introduction.tex | 21 + >> 6 files changed, 1664 insertions(+), 4 deletions(-) >> create mode 100644 device-types/video/description.tex >> create mode 100644 device-types/video/device-conformance.tex >> create mode 100644 device-types/video/driver-conformance.tex >> >> diff --git a/conformance.tex b/conformance.tex >> index 9af31e2..f34e600 100644 >> --- a/conformance.tex >> +++ b/conformance.tex >> @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >> \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, >> \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, >> \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, >> -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or >> -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. >> +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, >> +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or >> +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. >> >> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >> \end{itemize} >> @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >> \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, >> \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, >> \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, >> -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or >> -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. >> +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, >> +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or >> +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. >> >> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >> \end{itemize} >> @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >> \input{device-types/spi/driver-conformance.tex} >> \input{device-types/media/driver-conformance.tex} >> \input{device-types/rtc/driver-conformance.tex} >> +\input{device-types/video/driver-conformance.tex} >> >> \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} >> >> @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >> \input{device-types/spi/device-conformance.tex} >> \input{device-types/media/device-conformance.tex} >> \input{device-types/rtc/device-conformance.tex} >> +\input{device-types/video/device-conformance.tex} >> >> \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy >> Interface: Transitional Device and Transitional Driver Conformance} >> A conformant implementation MUST be either transitional or >> diff --git a/content.tex b/content.tex >> index 5de811f..0c13f68 100644 >> --- a/content.tex >> +++ b/content.tex >> @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} >> \input{device-types/spi/description.tex} >> \input{device-types/media/description.tex} >> \input{device-types/rtc/description.tex} >> +\input{device-types/video/description.tex} >> >> \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} >> >> diff --git a/device-types/video/description.tex b/device-types/video/description.tex >> new file mode 100644 >> index 0000000..8945e26 >> --- /dev/null >> +++ b/device-types/video/description.tex >> @@ -0,0 +1,1592 @@ >> +\section{Video Device} >> +\label{sec:Device Types / Video Device} >> + >> +The virtio video device provides support for host-accelerated video encoding >> +and decoding. >> + >> +\subsection{Device ID} >> +\label{sec:Device Types / Video Device / Device ID} >> + >> +50 >> + >> +\subsection{Virtqueues} >> +\label{sec:Device Types / Video Device / Virtqueues} >> + >> +\begin{description} >> + \item[0] >> + commandq - driver commands >> + \item[1] >> + eventq - device async responses to commands and standalone device events >> +\end{description} >> + >> +\subsection{Feature bits} >> +\label{sec:Device Types / Video Device / Feature bits} >> + >> +\begin{description} >> + \item[VIRTIO_VIDEO_F_ENCODER (0)] >> + The device can encode video. >> + \item[VIRTIO_VIDEO_F_DECODER (1)] >> + The device can decode video. >> + % Use-case: the device can support both encoding and decoding, so having both >> + % here can save resources. >> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] >> + Guest pages can be used as the backing memory of resources. >> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] >> + The device can use non-contiguous guest memory as the backing memory of >> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also >> + set. >> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] >> + Objects exported by another virtio device can be used as the backing memory >> + of resources. >> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] >> + The device releases an extra empty output buffer after a drain or DPC so that >> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. > This does not seem relavant to the video device. > It should not be attached to any V4L2 implementation. > Can you please craft it differently? Hmm, do you mean renaming the feature? The thing is that V4L2 driver is sitting on top in many cases and V4L2 needs an extra output buffer to pass the DPC/EOS events. But this may be not necessary in other implementations (e.g. Windows). So this feature helps here. V4L2 is mentioned in many places here and also some definitions are borrowed/referenced from it. It was agreed during the draft v6 review that this is ok in general. So I'm not sure what do you want me to do here. >> +\end{description} >> + >> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >> + >> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. >> + >> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or >> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would >> +mean that no memory can be used at all for resources. >> + >> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets >> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. >> + >> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >> + >> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and >> +VIRTIO_VIDEO_F_DECODER features. >> + >> +The driver MUST negotiate at least one of the >> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT >> +features. >> + > You wouldn't need virtio_object feature bit. It should be covered using the capability. Well, Michael says it is ok >> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not >> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically >> +contiguous memory for all the buffers it allocates. >> + >> +\subsection{Device configuration layout} >> +\label{sec:Device Types / Video Device / Device configuration layout} >> + >> +The video device configuration space uses the following layout: >> + >> +\begin{lstlisting} >> +struct virtio_video_config { >> + le32 max_streams; >> + le32 caps_length; >> +}; >> +\end{lstlisting} >> + >> +\begin{description} >> + \item[\field{max_streams}] >> + is the maximum number of concurrent streams the device supports. >> + \item[\field{caps_length}] >> + is the minimum length in bytes that a device-writable buffer must have >> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. >> +\end{description} >> + >> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} >> + >> +\field{max_streams} MUST be positive. >> + >> +\field{caps_length} MUST be set to the response size of >> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. >> + > This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. > Please rework the patch to use the existing basic facility. > This can also possibly eliminate plumbing device specific command q. > More below. I'm still learning about capabilities and admin commands, so please correct me if I'm wrong. Here are some considerations: 1. AFAIU capabilities and admin commands need an admin virtqueue. The latter depends on the VIRTIO_F_ADMIN_VQ feature. What if the device doesn't support it? Also it looks like the admin virtqueues are only defined and implemented in Virtio Over PCI at the moment, "reserved for future use" in Virtio Over MMIO and in Virtio Over Channel I/O and also not available in vhost-user. For us this makes admin queues unusable for now. 2. The virtio capabilities that you mention seem to be about enabling and disabling certain functionality based on a bitmap, right? (Still looking into the flow filter example, which seems to be more complex.) The virtio-video caps are more like an attempt to represent a graph of dependencies between different coded and raw formats, resolutions, etc. For example there will normally be multiple TLVs with the same types (coded and raw sets) "linked" to each other. I'm not sure this maps well into the existing virtio capabilities. 3. If this is really targeted towards SR-IOV maybe it is better to keep it this way. I mean we may need admin commands to create a virtio-video device per guest VM, so maybe using the same admin commands + some device-specific commands inside each device could be confusing. The nesting actually continues here, because virtio-video streams have shared buffers called "resources" (probably these should be renamed). 4. Also I'm not sure how does it actually improve the spec/implementation. >> +\subsection{Device Initialization} >> +\label{sec:Device Types / Video Device / Device Initialization} >> +\begin{enumerate} >> + \item >> + The driver reads the feature bits and negotiates the features it needs. >> + \item >> + The driver sets up the commandq and the eventq. >> + \item >> + The driver reads the \field{caps_length} field of the configuration >> + space, prepares a buffer of at least that size and sends the buffer on the >> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. >> + \item >> + The device sends a response over commandq to >> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the >> + command. >> + \item >> + The driver receives the response from the device, and parses its capabilities. >> +\end{enumerate} >> + >> +\subsection{Device Operation} >> +\label{sec:Device Types / Video Device / Device Operation} >> + >> +The device supports opening and operating a number of parallel streams up to >> +\field{max_streams}. Each stream has three internal device queues: mainqX, >> +inputqX and outputqX, where X is the stream id. Each stream command has a >> +field that dispatches the command to the specific internal queue. >> + > If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, > Than keyword 'internal' should be dropped and rephased. As Michael already assumed the implication is that within each queue commands are consumed in order. On top of this there are queue priorities: mainq has higher priority than inputq and outputq. This enables resetting inputq and outputq or setting parameters out of band. >> +% Use-case: there might be different real-time requirements for different >> +% streams, so more virtqueues can be added in the future if necessary. >> +% The internal queues don't change, the data formats don't change, only the >> +% mapping of streams/internal queues to particular virtqueues changes. >> + >> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, >> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX >> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out >> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock >> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for >> +format negotiation. >> + >> +The inputqX and outputqX are used to queue input or output resources using >> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and >> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete >> +processing of all queued input resources and make the resulting output >> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. >> + >> +All the stream commands start async operations, return the results >> +using async responses over eventq. >> +The eventq is used by the device to send the device's async responses to >> +stream commands and the device's standalone events. >> + >> +% This way eventq becomes the single source of truth about the device's state, >> +% the driver doesn't have to tediously synchronize commandq's and eventq's >> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF >> +% and DQEVENT). One more benefit is that commandq is processed fast and >> +% strictly in order, so commandq descriptors exhaustion should never happen in >> +% practice. >> + > Video device should be implementable without any V4L2 binding/description etc. Well, as I wrote above some V4L2 definitions are used anyway. It was agreed that this is fine and that we can have V4L2 headers as normative references during the v6 review. In this draft I decided to add some comments describing the reasons behind some design decisions because I keep forgetting them myself. I can edit this particular comment though of course. >> +Parameters allow the driver to configure the stream including setting up the >> +resources. Available parameters depend on the device type, see >> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. >> + >> +A resource is a set of memory buffers that contain a unit of data that >> +the device can process or produce. Most resources have only one buffer, >> +raw frames using a multi-planar format can have several. >> +Input resources are filled by the driver with compressed (coded) video data >> +for a decoder and raw frames for an encoder, output resources are filled by >> +the device as the result of processing the input resources with decoded raw >> +frames for a decoder and compressed (encoded) data for an encoder. >> +Resources from inputqX and outputqX are consumed independently, not in pairs. >> +One input resource can result in zero to many produced output resources. >> +A decoder device dequeues the output decoded frames in presentation order. >> +An encoder device dequeues the output decoded frames in decoding order. >> +The driver can reuse a queued resource after receiving a corresponding async >> +response. Dequeued output resources can still be used by the device as >> +reference frames, so the driver can't write to them. >> + >> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued >> +% output resource is not used by the device anymore and therefore becomes >> +% writeable? >> + >> +The device can detect standalone stream-related events: errors and dynamic >> +parameters changes that require intervention from the driver (e.g. >> +reallocating backing memory of output resources to fit the new parameters). >> +The events are signalled on the eventq, see >> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >> + >> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} >> + >> +The device MUST set to zero all unused, disabled or padding bits in its >> +responses. >> + >> +\subsubsection{Device Operation: Command Virtqueue} >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} >> + >> +This section lists the commands that can be sent by the driver to commandq. >> + >> +Different structures are used for each command and response. A command >> +structure starts with the requested command code, defined as follows: >> + >> +\begin{lstlisting} >> +/* Device */ >> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 >> + >> +/* Stream */ >> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 >> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 > You can craft the stream using a existing basic facility of resource object. > Where each stream is just a resource object, that be queried or modified. Replied about the admin commands above. >> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 >> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 >> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 >> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 >> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 >> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 >> +\end{lstlisting} >> + >> +Stream commands start with a header: >> + >> +\begin{lstlisting} >> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 >> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 >> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 >> + >> +struct virtio_video_stream_cmd_header { >> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ >> + le32 stream_id; >> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ >> + le32 async_response_cookie; >> +}; >> +\end{lstlisting} >> + >> +\begin{description} >> + \item[\field{async_response_cookie}] >> + is an async response cookie provided by the driver, that allows >> + to relate an async response to the previously submitted command. >> +\end{description} >> + >> +\subsubsection{Device Operation: Event Virtqueue} >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} >> + >> +The eventq is used by the device to send async responses to commands queued >> +by the driver on commandq and standalone events. Stream errors and dynamic >> +parameters changes are caused by changes in the device's state, not by >> +commands, still they are delivered as responses to implicit >> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, >> +respectively. >> + >> +Events start with a header: >> + >> +\begin{lstlisting} >> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) >> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) >> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) >> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) >> + >> +struct virtio_video_event_header { >> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ >> + le32 stream_id; >> + le32 async_response_cookie; >> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ >> +}; >> +\end{lstlisting} >> + >> +\begin{description} >> + \item[\field{event_type}] >> + is the type of the event. >> + \item[\field{stream_id}] >> + is the ID of a valid stream. >> + \item[\field{async_response_cookie}] >> + is an async response cookie provided by the driver, that allows >> + to relate the event to a previously submitted command. >> + \item[\field{event_flags}] >> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags >> + >> + \begin{description} >> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] >> + is set if the command finished with an error due to an >> + invalid argument or for other reasons. >> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] >> + is set for standalone events, see >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] >> + is set if the command has been canceled by another >> + command, that has higher priority. Doesn't make sense >> + for standalone events. >> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] >> + is set if the command triggered a block on the >> + outputqX to allow output format negotiation. >> + When the negotiation is finished the block has to be >> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK >> + command, see >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. >> + \end{description} >> +\end{description} >> + >> +The particular data structure representing the event is selected according to >> +the \field{event_type}. >> + >> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: >> Event Virtqueue} >> + >> +The driver MUST at any time have at least one descriptor with a used >> +buffer large enough to contain a \field{struct virtio_video_event} >> +queued on the eventq. >> + >> +The driver MUST NOT put device-readable descriptors into the eventq. >> + >> +The driver MUST account for the fact that the async responses to commands >> +might come out-of-order (i.e. after other commands sent to the device), >> +and that some of them can be cancelled. >> + >> +The driver SHOULD wait for an async response of command A, that caused >> +cancellation of command B, before queueing the command B again. >> + >> +\subsubsection{Device Operation: TLV format} >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} >> + >> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ >> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS >> +commands represent device capabilities and corresponding device parameters >> +in the form of TLV (Type-Length-Value): >> + >> +\begin{lstlisting} >> +struct virtio_video_tlv { >> + le32 type; >> + le32 length; >> + u8 value[length]; >> +}; >> +\end{lstlisting} >> + > All the caps can be crafted using existing capabilities infra. Please restructure the patch to use it. Replied above. >> +\begin{description} >> + \item[\field{type}] >> + specifies the type of data in \field{value}. >> + \item[\field{length}] >> + specifies the \field{value} size in bytes aligned to 4 bytes. >> + \item[\field{value}] >> + contains the data according to the type. >> +\end{description} >> + >> +The following TLV types are defined: >> + >> +\begin{lstlisting} >> +#define VIRTIO_VIDEO_TLV_CODED_SET 1 >> +#define VIRTIO_VIDEO_TLV_RAW_SET 2 >> +#define VIRTIO_VIDEO_TLV_LINK 3 >> +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 >> +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 >> +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 >> +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 >> +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 >> +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 >> +#define VIRTIO_VIDEO_TLV_CROP 10 >> +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 >> +\end{lstlisting} >> + > You can use the 'flow filter' example of network device to see how to frame the individual or group of capabilities. Checking it, thanks. Kind regards, Alexander ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-24 20:02 ` Alexander Gordeev @ 2026-02-25 4:59 ` Parav Pandit 2026-02-25 14:04 ` Alexander Gordeev 0 siblings, 1 reply; 9+ messages in thread From: Parav Pandit @ 2026-02-25 4:59 UTC (permalink / raw) To: Alexander Gordeev, virtio-comment@lists.linux.dev Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org > From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> > Sent: 25 February 2026 01:33 AM > > On 15/02/2026 11:49, Parav Pandit wrote: [..] > >> + > >> +\begin{description} > >> + \item[VIRTIO_VIDEO_F_ENCODER (0)] > >> + The device can encode video. > >> + \item[VIRTIO_VIDEO_F_DECODER (1)] > >> + The device can decode video. > >> + % Use-case: the device can support both encoding and decoding, so having both > >> + % here can save resources. > >> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] > >> + Guest pages can be used as the backing memory of resources. > >> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] > >> + The device can use non-contiguous guest memory as the backing memory of > >> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also > >> + set. > >> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] > >> + Objects exported by another virtio device can be used as the backing memory > >> + of resources. > >> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] > >> + The device releases an extra empty output buffer after a drain or DPC so that > >> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. > > This does not seem relavant to the video device. > > It should not be attached to any V4L2 implementation. > > Can you please craft it differently? > > Hmm, do you mean renaming the feature? No. virtio spec already has Media device that exposes V4L2 devices. So I am not sure why a dedicated video device is needed. If a dedicated video device is desired, it should not be linked to V4L2 at all. > The thing is that V4L2 driver is sitting on top in many cases and V4L2 needs an extra output buffer to pass the DPC/EOS events. But this may > be not necessary in other implementations (e.g. Windows). So this feature helps here. V4L2 is mentioned in many places here and also some > definitions are borrowed/referenced from it. It was agreed during the draft v6 review that this is ok in general. So I'm not sure what do you > want me to do here. > Can you please explain why v4l2 Media device (device id 48) is not sufficient for your use case? I expect video device to be not linked to V4L2 at all. If it needs to be, please explain the reason in the commit log cover letter too. > >> +\end{description} > >> + > >> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > >> + > >> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. > >> + > >> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or > >> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would > >> +mean that no memory can be used at all for resources. > >> + > >> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets > >> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. > >> + > >> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > >> + > >> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and > >> +VIRTIO_VIDEO_F_DECODER features. > >> + > >> +The driver MUST negotiate at least one of the > >> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT > >> +features. > >> + > > You wouldn't need virtio_object feature bit. It should be covered using the capability. > > Well, Michael says it is ok > I am not sure. Once you structure using capability unnecessary feature bits goes away. Feature bits offer simplicity but are only useful when one needs them before DRIVER_OK phase. So it is unclear why/what driver needs to do with this bit before DRIVER_OK stage. > >> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not > >> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically > >> +contiguous memory for all the buffers it allocates. > >> + > >> +\subsection{Device configuration layout} > >> +\label{sec:Device Types / Video Device / Device configuration layout} > >> + > >> +The video device configuration space uses the following layout: > >> + > >> +\begin{lstlisting} > >> +struct virtio_video_config { > >> + le32 max_streams; > >> + le32 caps_length; > >> +}; > >> +\end{lstlisting} > >> + > >> +\begin{description} > >> + \item[\field{max_streams}] > >> + is the maximum number of concurrent streams the device supports. > >> + \item[\field{caps_length}] > >> + is the minimum length in bytes that a device-writable buffer must have > >> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see > >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > >> +\end{description} > >> + > >> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} > >> + > >> +\field{max_streams} MUST be positive. > >> + > >> +\field{caps_length} MUST be set to the response size of > >> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. > >> + > > This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. > > Please rework the patch to use the existing basic facility. > > This can also possibly eliminate plumbing device specific command q. > > More below. > > I'm still learning about capabilities and admin commands, so please correct me if I'm wrong. > Here are some considerations: > 1. AFAIU capabilities and admin commands need an admin virtqueue. The latter depends on the VIRTIO_F_ADMIN_VQ feature. What if the > device doesn't support it? It isn’t the right question. It is a question similar to: what if the virtio-net device does not support control VQ, can it still enable multi-queue? No, it cannot. Hence virtio-net to implement cvq to support MQ. What if the virtio-blk device does not support AQ, can it will support device parts basic facility? No. it cannot. Does it mean these devices should invent new queue, instead of admin vq? No. The devices are encouraged to use the existing basic facilities of chapter 2 that is used by various features and devices such as virtio-net, block, crypto. > Also it looks like the admin virtqueues are only defined and implemented in Virtio Over PCI at the moment, > "reserved for future use" in Virtio Over MMIO and in Virtio Over Channel I/O and also not available in vhost-user. For us this makes admin > queues unusable for now. This is not a limitation. AQ support can be added for MMIO transport or channel transport too. Admin VQ is not any different beast than newly proposed command queue. > 2. The virtio capabilities that you mention seem to be about enabling and disabling certain functionality based on a bitmap, right? (Still looking > into the flow filter example, which seems to be more complex.) The virtio-video caps are more like an attempt to represent a graph of > dependencies between different coded and raw formats, resolutions, etc. For example there will normally be multiple TLVs with the same > types (coded and raw sets) "linked" to each other. I'm not sure this maps well into the existing virtio capabilities. Sure it can map. Capabilities can be define them. > 3. If this is really targeted towards SR-IOV maybe it is better to keep it this way. No. it is not targeted towards SR-IOV. It is basic facilities uses for SR-IOV, flow filters, crypto across 3 different device types. > I mean we may need admin commands to create a virtio- > video device per guest VM, so maybe using the same admin commands + some device-specific commands inside each device could be > confusing. The nesting actually continues here, because virtio-video streams have shared buffers called "resources" (probably these should be > renamed). > 4. Also I'm not sure how does it actually improve the spec/implementation. > It improves the spec by reusing the basic facilities of the spec and avoids creating new objects. This reduces the spec burden. It also reuses the existing implementation of the device and driver. > >> +\subsection{Device Initialization} > >> +\label{sec:Device Types / Video Device / Device Initialization} > >> +\begin{enumerate} > >> + \item > >> + The driver reads the feature bits and negotiates the features it needs. > >> + \item > >> + The driver sets up the commandq and the eventq. > >> + \item > >> + The driver reads the \field{caps_length} field of the configuration > >> + space, prepares a buffer of at least that size and sends the buffer on the > >> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. > >> + \item > >> + The device sends a response over commandq to > >> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the > >> + command. > >> + \item > >> + The driver receives the response from the device, and parses its capabilities. > >> +\end{enumerate} > >> + > >> +\subsection{Device Operation} > >> +\label{sec:Device Types / Video Device / Device Operation} > >> + > >> +The device supports opening and operating a number of parallel streams up to > >> +\field{max_streams}. Each stream has three internal device queues: mainqX, > >> +inputqX and outputqX, where X is the stream id. Each stream command has a > >> +field that dispatches the command to the specific internal queue. > >> + > > If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, > > Than keyword 'internal' should be dropped and rephased. > > As Michael already assumed the implication is that within each queue commands are consumed in order. Admin virtqueue can be extended to support out of order. One can always add a new first admin command to operate in out of order mode. Or even a feature bit to have AQ_OUT_OF_ORDER. And again this is optimization. In other thread we discussed that optimization may come later. But if you think it is essential to have the good out of order performance now (and how much it improves - rough estimates), it is worth to extend AQ for out of order. Even SR-IOV can benefit of this feature right away for reading and writing device parts of unrelated VFs. So there is already more than one user of the out of order feature you suggest. 😊 > On top of this there are queue > priorities: mainq has higher priority than inputq and outputq. This enables resetting inputq and outputq or setting parameters out of band. > So multiple admin queues are also supported. Before starting these queues, their priority can be set using a new command. > >> +% Use-case: there might be different real-time requirements for different > >> +% streams, so more virtqueues can be added in the future if necessary. > >> +% The internal queues don't change, the data formats don't change, only the > >> +% mapping of streams/internal queues to particular virtqueues changes. > >> + > >> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, > >> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX > >> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out > >> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock > >> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for > >> +format negotiation. > >> + > >> +The inputqX and outputqX are used to queue input or output resources using > >> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and > >> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete > >> +processing of all queued input resources and make the resulting output > >> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. > >> + > >> +All the stream commands start async operations, return the results > >> +using async responses over eventq. > >> +The eventq is used by the device to send the device's async responses to > >> +stream commands and the device's standalone events. > >> + > >> +% This way eventq becomes the single source of truth about the device's state, > >> +% the driver doesn't have to tediously synchronize commandq's and eventq's > >> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF > >> +% and DQEVENT). One more benefit is that commandq is processed fast and > >> +% strictly in order, so commandq descriptors exhaustion should never happen in > >> +% practice. > >> + > > Video device should be implementable without any V4L2 binding/description etc. > > Well, as I wrote above some V4L2 definitions are used anyway. It was agreed that this is fine and that we can have V4L2 headers as normative > references during the v6 review. > In this draft I decided to add some comments describing the reasons behind some design decisions because I keep forgetting them myself. I > can edit this particular comment though of course. > Lets first understand why existing media device is not enough. > >> +Parameters allow the driver to configure the stream including setting up the > >> +resources. Available parameters depend on the device type, see > >> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. > >> + > >> +A resource is a set of memory buffers that contain a unit of data that > >> +the device can process or produce. Most resources have only one buffer, > >> +raw frames using a multi-planar format can have several. > >> +Input resources are filled by the driver with compressed (coded) video data > >> +for a decoder and raw frames for an encoder, output resources are filled by > >> +the device as the result of processing the input resources with decoded raw > >> +frames for a decoder and compressed (encoded) data for an encoder. > >> +Resources from inputqX and outputqX are consumed independently, not in pairs. > >> +One input resource can result in zero to many produced output resources. > >> +A decoder device dequeues the output decoded frames in presentation order. > >> +An encoder device dequeues the output decoded frames in decoding order. > >> +The driver can reuse a queued resource after receiving a corresponding async > >> +response. Dequeued output resources can still be used by the device as > >> +reference frames, so the driver can't write to them. > >> + > >> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued > >> +% output resource is not used by the device anymore and therefore becomes > >> +% writeable? > >> + > >> +The device can detect standalone stream-related events: errors and dynamic > >> +parameters changes that require intervention from the driver (e.g. > >> +reallocating backing memory of output resources to fit the new parameters). > >> +The events are signalled on the eventq, see > >> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > >> + > >> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} > >> + > >> +The device MUST set to zero all unused, disabled or padding bits in its > >> +responses. > >> + > >> +\subsubsection{Device Operation: Command Virtqueue} > >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} > >> + > >> +This section lists the commands that can be sent by the driver to commandq. > >> + > >> +Different structures are used for each command and response. A command > >> +structure starts with the requested command code, defined as follows: > >> + > >> +\begin{lstlisting} > >> +/* Device */ > >> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 > >> + > >> +/* Stream */ > >> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 > >> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 > > You can craft the stream using a existing basic facility of resource object. > > Where each stream is just a resource object, that be queried or modified. > > Replied about the admin commands above. > Comments above. This can be done using admin commands easily. > >> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 > >> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 > >> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 > >> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 > >> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 > >> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 > >> +\end{lstlisting} > >> + > >> +Stream commands start with a header: > >> + > >> +\begin{lstlisting} > >> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 > >> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 > >> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 > >> + > >> +struct virtio_video_stream_cmd_header { > >> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ > >> + le32 stream_id; > >> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ > >> + le32 async_response_cookie; > >> +}; > >> +\end{lstlisting} > >> + > >> +\begin{description} > >> + \item[\field{async_response_cookie}] > >> + is an async response cookie provided by the driver, that allows > >> + to relate an async response to the previously submitted command. > >> +\end{description} > >> + > >> +\subsubsection{Device Operation: Event Virtqueue} > >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} > >> + > >> +The eventq is used by the device to send async responses to commands queued > >> +by the driver on commandq and standalone events. Stream errors and dynamic > >> +parameters changes are caused by changes in the device's state, not by > >> +commands, still they are delivered as responses to implicit > >> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, > >> +respectively. > >> + > >> +Events start with a header: > >> + > >> +\begin{lstlisting} > >> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) > >> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) > >> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) > >> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) > >> + > >> +struct virtio_video_event_header { > >> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ > >> + le32 stream_id; > >> + le32 async_response_cookie; > >> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ > >> +}; > >> +\end{lstlisting} > >> + > >> +\begin{description} > >> + \item[\field{event_type}] > >> + is the type of the event. > >> + \item[\field{stream_id}] > >> + is the ID of a valid stream. > >> + \item[\field{async_response_cookie}] > >> + is an async response cookie provided by the driver, that allows > >> + to relate the event to a previously submitted command. > >> + \item[\field{event_flags}] > >> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags > >> + > >> + \begin{description} > >> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] > >> + is set if the command finished with an error due to an > >> + invalid argument or for other reasons. > >> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] > >> + is set for standalone events, see > >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > >> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] > >> + is set if the command has been canceled by another > >> + command, that has higher priority. Doesn't make sense > >> + for standalone events. > >> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] > >> + is set if the command triggered a block on the > >> + outputqX to allow output format negotiation. > >> + When the negotiation is finished the block has to be > >> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK > >> + command, see > >> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. > >> + \end{description} > >> +\end{description} > >> + > >> +The particular data structure representing the event is selected according to > >> +the \field{event_type}. > >> + > >> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: > >> Event Virtqueue} > >> + > >> +The driver MUST at any time have at least one descriptor with a used > >> +buffer large enough to contain a \field{struct virtio_video_event} > >> +queued on the eventq. > >> + > >> +The driver MUST NOT put device-readable descriptors into the eventq. > >> + > >> +The driver MUST account for the fact that the async responses to commands > >> +might come out-of-order (i.e. after other commands sent to the device), > >> +and that some of them can be cancelled. > >> + > >> +The driver SHOULD wait for an async response of command A, that caused > >> +cancellation of command B, before queueing the command B again. > >> + > >> +\subsubsection{Device Operation: TLV format} > >> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} > >> + > >> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ > >> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS > >> +commands represent device capabilities and corresponding device parameters > >> +in the form of TLV (Type-Length-Value): > >> + > >> +\begin{lstlisting} > >> +struct virtio_video_tlv { > >> + le32 type; > >> + le32 length; > >> + u8 value[length]; > >> +}; > >> +\end{lstlisting} > >> + > > All the caps can be crafted using existing capabilities infra. Please restructure the patch to use it. > > Replied above. > Same as above. > >> +\begin{description} > >> + \item[\field{type}] > >> + specifies the type of data in \field{value}. > >> + \item[\field{length}] > >> + specifies the \field{value} size in bytes aligned to 4 bytes. > >> + \item[\field{value}] > >> + contains the data according to the type. > >> +\end{description} > >> + > >> +The following TLV types are defined: > >> + > >> +\begin{lstlisting} > >> +#define VIRTIO_VIDEO_TLV_CODED_SET 1 > >> +#define VIRTIO_VIDEO_TLV_RAW_SET 2 > >> +#define VIRTIO_VIDEO_TLV_LINK 3 > >> +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 > >> +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 > >> +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 > >> +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 > >> +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 > >> +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 > >> +#define VIRTIO_VIDEO_TLV_CROP 10 > >> +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 > >> +\end{lstlisting} > >> + > > You can use the 'flow filter' example of network device to see how to frame the individual or group of capabilities. > > Checking it, thanks. > > Kind regards, > Alexander ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-25 4:59 ` Parav Pandit @ 2026-02-25 14:04 ` Alexander Gordeev 2026-02-25 14:38 ` Parav Pandit 0 siblings, 1 reply; 9+ messages in thread From: Alexander Gordeev @ 2026-02-25 14:04 UTC (permalink / raw) To: Parav Pandit, virtio-comment@lists.linux.dev Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org On 25/02/2026 05:59, Parav Pandit wrote: > >> From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> >> Sent: 25 February 2026 01:33 AM >> >> On 15/02/2026 11:49, Parav Pandit wrote: > > [..] >>>> + >>>> +\begin{description} >>>> + \item[VIRTIO_VIDEO_F_ENCODER (0)] >>>> + The device can encode video. >>>> + \item[VIRTIO_VIDEO_F_DECODER (1)] >>>> + The device can decode video. >>>> + % Use-case: the device can support both encoding and decoding, so having both >>>> + % here can save resources. >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] >>>> + Guest pages can be used as the backing memory of resources. >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] >>>> + The device can use non-contiguous guest memory as the backing memory of >>>> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also >>>> + set. >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] >>>> + Objects exported by another virtio device can be used as the backing memory >>>> + of resources. >>>> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] >>>> + The device releases an extra empty output buffer after a drain or DPC so that >>>> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. >>> This does not seem relavant to the video device. >>> It should not be attached to any V4L2 implementation. >>> Can you please craft it differently? >> >> Hmm, do you mean renaming the feature? > No. virtio spec already has Media device that exposes V4L2 devices. > So I am not sure why a dedicated video device is needed. > If a dedicated video device is desired, it should not be linked to V4L2 at all. > >> The thing is that V4L2 driver is sitting on top in many cases and V4L2 needs an extra output buffer to pass the DPC/EOS events. But this may >> be not necessary in other implementations (e.g. Windows). So this feature helps here. V4L2 is mentioned in many places here and also some >> definitions are borrowed/referenced from it. It was agreed during the draft v6 review that this is ok in general. So I'm not sure what do you >> want me to do here. >> > Can you please explain why v4l2 Media device (device id 48) is not sufficient for your use case? There were multiple discussions already. I'd prefer to avoid starting the discussion again. Here are some references: https://lore.kernel.org/virtio-dev/b5c9d69a-fbec-8046-225f-bdb5850bbd54@opensynergy.com/ https://lore.kernel.org/virtio-comment/92964190-81d0-4419-833a-65e254abc0df@opensynergy.com/ https://lore.kernel.org/virtio-comment/f6dc2d4d-89d9-4721-9c67-a214025cb3c3@gmail.com/ > I expect video device to be not linked to V4L2 at all. > If it needs to be, please explain the reason in the commit log cover letter too. I checked the spec again. Here are the types of V4L2 references in it: 1. Comments about use-cases. These are for illustrative purposes, they don't go into the text. Is this really a problem? 2. Compatibility feature flag and buffer dequeue flags. This is a way to do things simpler if the driver is not V4L2, but retain compatibility if the driver is V4L2. This is actually an attempt to be not linked to V4L2 and make things simpler if we don't have V4L2. 3. Compressed formats and controls. Virtio-video references the descriptions of compressed formats from https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst This way we can avoid including lengthy descriptions for formats like this: - 'H264' - H264 Access Unit. The decoder expects one Access Unit per buffer. The encoder generates one Access Unit per buffer. If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM`` then the decoder has no requirements since it can parse all the information from the raw bytestream. For the controls this is also mostly about avoiding including all the control descriptions and enum values into the spec like this: enum v4l2_mpeg_video_h264_profile { V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE = 0, V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE = 1, V4L2_MPEG_VIDEO_H264_PROFILE_MAIN = 2, V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED = 3, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH = 4, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_10 = 5, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_422 = 6, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_444_PREDICTIVE = 7, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_10_INTRA = 8, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_422_INTRA = 9, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_444_INTRA = 10, V4L2_MPEG_VIDEO_H264_PROFILE_CAVLC_444_INTRA = 11, V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_BASELINE = 12, V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_HIGH = 13, V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_HIGH_INTRA = 14, V4L2_MPEG_VIDEO_H264_PROFILE_STEREO_HIGH = 15, V4L2_MPEG_VIDEO_H264_PROFILE_MULTIVIEW_HIGH = 16, V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_HIGH = 17, }; I don't think you want all these lengthy enums in the spec. Right now the virtio-video spec takes 18 pages. With all the enums it will be much more. And I think it is really pointless to copy them. This is it. So no, this device doesn't just expose V4L2 as virtio-media. The V4L2 references are very limited. During draft v6 discussions the spec length was brought as one of the issues. Therefore the references to V4L2 formats and controls. I believe the current state is really the right amount of referencing, also it was discussed in the past, so I don't like to change this. Of course it is possible to rephrase something on case by case basis if you insist. >>>> +\end{description} >>>> + >>>> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>>> + >>>> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. >>>> + >>>> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or >>>> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would >>>> +mean that no memory can be used at all for resources. >>>> + >>>> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets >>>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. >>>> + >>>> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>>> + >>>> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and >>>> +VIRTIO_VIDEO_F_DECODER features. >>>> + >>>> +The driver MUST negotiate at least one of the >>>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT >>>> +features. >>>> + >>> You wouldn't need virtio_object feature bit. It should be covered using the capability. >> >> Well, Michael says it is ok >> > I am not sure. Once you structure using capability unnecessary feature bits goes away. > Feature bits offer simplicity but are only useful when one needs them before DRIVER_OK phase. > So it is unclear why/what driver needs to do with this bit before DRIVER_OK stage. > >>>> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not >>>> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically >>>> +contiguous memory for all the buffers it allocates. >>>> + >>>> +\subsection{Device configuration layout} >>>> +\label{sec:Device Types / Video Device / Device configuration layout} >>>> + >>>> +The video device configuration space uses the following layout: >>>> + >>>> +\begin{lstlisting} >>>> +struct virtio_video_config { >>>> + le32 max_streams; >>>> + le32 caps_length; >>>> +}; >>>> +\end{lstlisting} >>>> + >>>> +\begin{description} >>>> + \item[\field{max_streams}] >>>> + is the maximum number of concurrent streams the device supports. >>>> + \item[\field{caps_length}] >>>> + is the minimum length in bytes that a device-writable buffer must have >>>> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. >>>> +\end{description} >>>> + >>>> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} >>>> + >>>> +\field{max_streams} MUST be positive. >>>> + >>>> +\field{caps_length} MUST be set to the response size of >>>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. >>>> + >>> This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. >>> Please rework the patch to use the existing basic facility. >>> This can also possibly eliminate plumbing device specific command q. >>> More below. >> >> I'm still learning about capabilities and admin commands, so please correct me if I'm wrong. >> Here are some considerations: >> 1. AFAIU capabilities and admin commands need an admin virtqueue. The latter depends on the VIRTIO_F_ADMIN_VQ feature. What if the >> device doesn't support it? > It isn’t the right question. > It is a question similar to: what if the virtio-net device does not support control VQ, can it still enable multi-queue? > No, it cannot. Hence virtio-net to implement cvq to support MQ. > What if the virtio-blk device does not support AQ, can it will support device parts basic facility? > No. it cannot. > > Does it mean these devices should invent new queue, instead of admin vq? > No. > The devices are encouraged to use the existing basic facilities of chapter 2 that is used by various features and devices such as virtio-net, block, crypto. > >> Also it looks like the admin virtqueues are only defined and implemented in Virtio Over PCI at the moment, >> "reserved for future use" in Virtio Over MMIO and in Virtio Over Channel I/O and also not available in vhost-user. For us this makes admin >> queues unusable for now. > This is not a limitation. AQ support can be added for MMIO transport or channel transport too. > Admin VQ is not any different beast than newly proposed command queue. This has to be done by someone first. This is definitely outside the scope of this patch. For now there are no admin queues in MMIO transport, this is a problem for us. How about agreeing to do as follows: 1. For now virtio-video doesn't require an admin queue, they can be enabled later with a feature flag when at least MMIO transport is ready (possibly together with media, gpu, etc). 2. Maybe you can suggest certain specific things that can be already done in virtio-video to make the future transition to admin queues simpler. Like maybe moving the command type IDs into a certain range (0x8000 - 0xFFFF ??? but it doesn't say this is reserved for device-specific commands though) and maybe changing some types. Capabilities don't fit for now because they depend on the admin queue AFAIU. Please note that we'd like to avoid creating any unnecessary extra virtqueues so that we can save limited HW resources. Therefore it should be either a commandq, or an admin queue with both admin commands and device-specific commands mixed and consumed in strict order. >> 2. The virtio capabilities that you mention seem to be about enabling and disabling certain functionality based on a bitmap, right? (Still looking >> into the flow filter example, which seems to be more complex.) The virtio-video caps are more like an attempt to represent a graph of >> dependencies between different coded and raw formats, resolutions, etc. For example there will normally be multiple TLVs with the same >> types (coded and raw sets) "linked" to each other. I'm not sure this maps well into the existing virtio capabilities. > Sure it can map. Capabilities can be define them. > >> 3. If this is really targeted towards SR-IOV maybe it is better to keep it this way. > No. it is not targeted towards SR-IOV. > It is basic facilities uses for SR-IOV, flow filters, crypto across 3 different device types. > >> I mean we may need admin commands to create a virtio- >> video device per guest VM, so maybe using the same admin commands + some device-specific commands inside each device could be >> confusing. The nesting actually continues here, because virtio-video streams have shared buffers called "resources" (probably these should be >> renamed). >> 4. Also I'm not sure how does it actually improve the spec/implementation. >> > It improves the spec by reusing the basic facilities of the spec and avoids creating new objects. > This reduces the spec burden. > It also reuses the existing implementation of the device and driver. OK, got it. Replied above. >>>> +\subsection{Device Initialization} >>>> +\label{sec:Device Types / Video Device / Device Initialization} >>>> +\begin{enumerate} >>>> + \item >>>> + The driver reads the feature bits and negotiates the features it needs. >>>> + \item >>>> + The driver sets up the commandq and the eventq. >>>> + \item >>>> + The driver reads the \field{caps_length} field of the configuration >>>> + space, prepares a buffer of at least that size and sends the buffer on the >>>> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. >>>> + \item >>>> + The device sends a response over commandq to >>>> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the >>>> + command. >>>> + \item >>>> + The driver receives the response from the device, and parses its capabilities. >>>> +\end{enumerate} >>>> + >>>> +\subsection{Device Operation} >>>> +\label{sec:Device Types / Video Device / Device Operation} >>>> + >>>> +The device supports opening and operating a number of parallel streams up to >>>> +\field{max_streams}. Each stream has three internal device queues: mainqX, >>>> +inputqX and outputqX, where X is the stream id. Each stream command has a >>>> +field that dispatches the command to the specific internal queue. >>>> + >>> If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, >>> Than keyword 'internal' should be dropped and rephased. >> >> As Michael already assumed the implication is that within each queue commands are consumed in order. > Admin virtqueue can be extended to support out of order. > One can always add a new first admin command to operate in out of order mode. > Or even a feature bit to have AQ_OUT_OF_ORDER. > And again this is optimization. > In other thread we discussed that optimization may come later. > But if you think it is essential to have the good out of order performance now (and how much it improves - rough estimates), it is worth to extend AQ for out of order. > Even SR-IOV can benefit of this feature right away for reading and writing device parts of unrelated VFs. > So there is already more than one user of the out of order feature you suggest. 😊 I think there is some misunderstanding. We actually want all the queues including the internal ones to be consumed in order. Otherwise the decoding just falls apart. >> On top of this there are queue >> priorities: mainq has higher priority than inputq and outputq. This enables resetting inputq and outputq or setting parameters out of band. >> > So multiple admin queues are also supported. > Before starting these queues, their priority can be set using a new command. As I wrote above we'd like to avoid creating extra virtqueues unless necessary. This is a requirement from our HW engineers. If every stream always has 3 real virtqueues, this is definitely a problem. So this spec has a scheme to multiplex several stream queues into the single commandq. I don't see any generic multiplexing scheme in the admin queues for now, so this is also an issue if we consider them managing the internal queues. >>>> +% Use-case: there might be different real-time requirements for different >>>> +% streams, so more virtqueues can be added in the future if necessary. >>>> +% The internal queues don't change, the data formats don't change, only the >>>> +% mapping of streams/internal queues to particular virtqueues changes. >>>> + >>>> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, >>>> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX >>>> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out >>>> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock >>>> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for >>>> +format negotiation. >>>> + >>>> +The inputqX and outputqX are used to queue input or output resources using >>>> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and >>>> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete >>>> +processing of all queued input resources and make the resulting output >>>> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. >>>> + >>>> +All the stream commands start async operations, return the results >>>> +using async responses over eventq. >>>> +The eventq is used by the device to send the device's async responses to >>>> +stream commands and the device's standalone events. >>>> + >>>> +% This way eventq becomes the single source of truth about the device's state, >>>> +% the driver doesn't have to tediously synchronize commandq's and eventq's >>>> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF >>>> +% and DQEVENT). One more benefit is that commandq is processed fast and >>>> +% strictly in order, so commandq descriptors exhaustion should never happen in >>>> +% practice. >>>> + >>> Video device should be implementable without any V4L2 binding/description etc. >> >> Well, as I wrote above some V4L2 definitions are used anyway. It was agreed that this is fine and that we can have V4L2 headers as normative >> references during the v6 review. >> In this draft I decided to add some comments describing the reasons behind some design decisions because I keep forgetting them myself. I >> can edit this particular comment though of course. >> > Lets first understand why existing media device is not enough. > >>>> +Parameters allow the driver to configure the stream including setting up the >>>> +resources. Available parameters depend on the device type, see >>>> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. >>>> + >>>> +A resource is a set of memory buffers that contain a unit of data that >>>> +the device can process or produce. Most resources have only one buffer, >>>> +raw frames using a multi-planar format can have several. >>>> +Input resources are filled by the driver with compressed (coded) video data >>>> +for a decoder and raw frames for an encoder, output resources are filled by >>>> +the device as the result of processing the input resources with decoded raw >>>> +frames for a decoder and compressed (encoded) data for an encoder. >>>> +Resources from inputqX and outputqX are consumed independently, not in pairs. >>>> +One input resource can result in zero to many produced output resources. >>>> +A decoder device dequeues the output decoded frames in presentation order. >>>> +An encoder device dequeues the output decoded frames in decoding order. >>>> +The driver can reuse a queued resource after receiving a corresponding async >>>> +response. Dequeued output resources can still be used by the device as >>>> +reference frames, so the driver can't write to them. >>>> + >>>> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued >>>> +% output resource is not used by the device anymore and therefore becomes >>>> +% writeable? >>>> + >>>> +The device can detect standalone stream-related events: errors and dynamic >>>> +parameters changes that require intervention from the driver (e.g. >>>> +reallocating backing memory of output resources to fit the new parameters). >>>> +The events are signalled on the eventq, see >>>> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>>> + >>>> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} >>>> + >>>> +The device MUST set to zero all unused, disabled or padding bits in its >>>> +responses. >>>> + >>>> +\subsubsection{Device Operation: Command Virtqueue} >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} >>>> + >>>> +This section lists the commands that can be sent by the driver to commandq. >>>> + >>>> +Different structures are used for each command and response. A command >>>> +structure starts with the requested command code, defined as follows: >>>> + >>>> +\begin{lstlisting} >>>> +/* Device */ >>>> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 >>>> + >>>> +/* Stream */ >>>> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 >>> You can craft the stream using a existing basic facility of resource object. >>> Where each stream is just a resource object, that be queried or modified. >> >> Replied about the admin commands above. >> > Comments above. > This can be done using admin commands easily. > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 >>>> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 >>>> +\end{lstlisting} >>>> + >>>> +Stream commands start with a header: >>>> + >>>> +\begin{lstlisting} >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 >>>> + >>>> +struct virtio_video_stream_cmd_header { >>>> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ >>>> + le32 stream_id; >>>> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ >>>> + le32 async_response_cookie; >>>> +}; >>>> +\end{lstlisting} >>>> + >>>> +\begin{description} >>>> + \item[\field{async_response_cookie}] >>>> + is an async response cookie provided by the driver, that allows >>>> + to relate an async response to the previously submitted command. >>>> +\end{description} >>>> + >>>> +\subsubsection{Device Operation: Event Virtqueue} >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} >>>> + >>>> +The eventq is used by the device to send async responses to commands queued >>>> +by the driver on commandq and standalone events. Stream errors and dynamic >>>> +parameters changes are caused by changes in the device's state, not by >>>> +commands, still they are delivered as responses to implicit >>>> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, >>>> +respectively. >>>> + >>>> +Events start with a header: >>>> + >>>> +\begin{lstlisting} >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) >>>> + >>>> +struct virtio_video_event_header { >>>> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ >>>> + le32 stream_id; >>>> + le32 async_response_cookie; >>>> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ >>>> +}; >>>> +\end{lstlisting} >>>> + >>>> +\begin{description} >>>> + \item[\field{event_type}] >>>> + is the type of the event. >>>> + \item[\field{stream_id}] >>>> + is the ID of a valid stream. >>>> + \item[\field{async_response_cookie}] >>>> + is an async response cookie provided by the driver, that allows >>>> + to relate the event to a previously submitted command. >>>> + \item[\field{event_flags}] >>>> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags >>>> + >>>> + \begin{description} >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] >>>> + is set if the command finished with an error due to an >>>> + invalid argument or for other reasons. >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] >>>> + is set for standalone events, see >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] >>>> + is set if the command has been canceled by another >>>> + command, that has higher priority. Doesn't make sense >>>> + for standalone events. >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] >>>> + is set if the command triggered a block on the >>>> + outputqX to allow output format negotiation. >>>> + When the negotiation is finished the block has to be >>>> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK >>>> + command, see >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. >>>> + \end{description} >>>> +\end{description} >>>> + >>>> +The particular data structure representing the event is selected according to >>>> +the \field{event_type}. >>>> + >>>> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: >>>> Event Virtqueue} >>>> + >>>> +The driver MUST at any time have at least one descriptor with a used >>>> +buffer large enough to contain a \field{struct virtio_video_event} >>>> +queued on the eventq. >>>> + >>>> +The driver MUST NOT put device-readable descriptors into the eventq. >>>> + >>>> +The driver MUST account for the fact that the async responses to commands >>>> +might come out-of-order (i.e. after other commands sent to the device), >>>> +and that some of them can be cancelled. >>>> + >>>> +The driver SHOULD wait for an async response of command A, that caused >>>> +cancellation of command B, before queueing the command B again. >>>> + >>>> +\subsubsection{Device Operation: TLV format} >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} >>>> + >>>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ >>>> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS >>>> +commands represent device capabilities and corresponding device parameters >>>> +in the form of TLV (Type-Length-Value): >>>> + >>>> +\begin{lstlisting} >>>> +struct virtio_video_tlv { >>>> + le32 type; >>>> + le32 length; >>>> + u8 value[length]; >>>> +}; >>>> +\end{lstlisting} >>>> + >>> All the caps can be crafted using existing capabilities infra. Please restructure the patch to use it. >> >> Replied above. >> > Same as above. > >>>> +\begin{description} >>>> + \item[\field{type}] >>>> + specifies the type of data in \field{value}. >>>> + \item[\field{length}] >>>> + specifies the \field{value} size in bytes aligned to 4 bytes. >>>> + \item[\field{value}] >>>> + contains the data according to the type. >>>> +\end{description} >>>> + >>>> +The following TLV types are defined: >>>> + >>>> +\begin{lstlisting} >>>> +#define VIRTIO_VIDEO_TLV_CODED_SET 1 >>>> +#define VIRTIO_VIDEO_TLV_RAW_SET 2 >>>> +#define VIRTIO_VIDEO_TLV_LINK 3 >>>> +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 >>>> +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 >>>> +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 >>>> +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 >>>> +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 >>>> +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 >>>> +#define VIRTIO_VIDEO_TLV_CROP 10 >>>> +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 >>>> +\end{lstlisting} >>>> + >>> You can use the 'flow filter' example of network device to see how to frame the individual or group of capabilities. >> >> Checking it, thanks. ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [PATCH v10 1/1] virtio-video: Add virtio video device specification 2026-02-25 14:04 ` Alexander Gordeev @ 2026-02-25 14:38 ` Parav Pandit 0 siblings, 0 replies; 9+ messages in thread From: Parav Pandit @ 2026-02-25 14:38 UTC (permalink / raw) To: Alexander Gordeev, virtio-comment@lists.linux.dev Cc: Albert Esteve, Alex Bennée, Cornelia Huck, Daniel Almeida, Nicolas Dufresne, Enric Balletbo i Serra, Kieran Bingham, Laurent Pinchart, Michael S . Tsirkin, Peter Griffin, Demi Marie Obenour, Manos Pitsidianakis, Matias Ezequiel Vara Larsen, Trilok Soni, Matti Moell, linux-media@vger.kernel.org > From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> > Sent: 25 February 2026 07:34 PM > > On 25/02/2026 05:59, Parav Pandit wrote: > > > >> From: Alexander Gordeev <alexander.gordeev@oss.qualcomm.com> > >> Sent: 25 February 2026 01:33 AM > >> > >> On 15/02/2026 11:49, Parav Pandit wrote: > > > > [..] > >>>> + > >>>> +\begin{description} > >>>> + \item[VIRTIO_VIDEO_F_ENCODER (0)] > >>>> + The device can encode video. > >>>> + \item[VIRTIO_VIDEO_F_DECODER (1)] > >>>> + The device can decode video. > >>>> + % Use-case: the device can support both encoding and decoding, so having both > >>>> + % here can save resources. > >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] > >>>> + Guest pages can be used as the backing memory of resources. > >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] > >>>> + The device can use non-contiguous guest memory as the backing memory of > >>>> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also > >>>> + set. > >>>> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] > >>>> + Objects exported by another virtio device can be used as the backing memory > >>>> + of resources. > >>>> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] > >>>> + The device releases an extra empty output buffer after a drain or DPC so that > >>>> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. > >>> This does not seem relavant to the video device. > >>> It should not be attached to any V4L2 implementation. > >>> Can you please craft it differently? > >> > >> Hmm, do you mean renaming the feature? > > No. virtio spec already has Media device that exposes V4L2 devices. > > So I am not sure why a dedicated video device is needed. > > If a dedicated video device is desired, it should not be linked to V4L2 at all. > > > >> The thing is that V4L2 driver is sitting on top in many cases and V4L2 needs an extra output buffer to pass the DPC/EOS events. But this > may > >> be not necessary in other implementations (e.g. Windows). So this feature helps here. V4L2 is mentioned in many places here and also > some > >> definitions are borrowed/referenced from it. It was agreed during the draft v6 review that this is ok in general. So I'm not sure what do you > >> want me to do here. > >> > > Can you please explain why v4l2 Media device (device id 48) is not sufficient for your use case? > > There were multiple discussions already. I'd prefer to avoid starting the discussion again. > Here are some references: > https://lore.kernel.org/virtio-dev/b5c9d69a-fbec-8046-225f-bdb5850bbd54@opensynergy.com/ > https://lore.kernel.org/virtio-comment/92964190-81d0-4419-833a-65e254abc0df@opensynergy.com/ > https://lore.kernel.org/virtio-comment/f6dc2d4d-89d9-4721-9c67-a214025cb3c3@gmail.com/ > I will read it up later this part. > > I expect video device to be not linked to V4L2 at all. > > If it needs to be, please explain the reason in the commit log cover letter too. > > I checked the spec again. Here are the types of V4L2 references in it: > 1. Comments about use-cases. > These are for illustrative purposes, they don't go into the text. Is this really a problem? > 2. Compatibility feature flag and buffer dequeue flags. > This is a way to do things simpler if the driver is not V4L2, but retain compatibility if the driver is V4L2. > This is actually an attempt to be not linked to V4L2 and make things simpler if we don't have V4L2. > 3. Compressed formats and controls. > Virtio-video references the descriptions of compressed formats from > https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/userspace-api/media/v4l/pixfmt-compressed.rst > This way we can avoid including lengthy descriptions for formats like this: > - 'H264' > - H264 Access Unit. > The decoder expects one Access Unit per buffer. > The encoder generates one Access Unit per buffer. > If :ref:`VIDIOC_ENUM_FMT` reports ``V4L2_FMT_FLAG_CONTINUOUS_BYTESTREAM`` > then the decoder has no requirements since it can parse all the > information from the raw bytestream. > For the controls this is also mostly about avoiding including all the control descriptions and enum values into the spec like this: > > enum v4l2_mpeg_video_h264_profile { > V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE = 0, > V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_BASELINE = 1, > V4L2_MPEG_VIDEO_H264_PROFILE_MAIN = 2, > V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED = 3, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH = 4, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_10 = 5, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_422 = 6, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_444_PREDICTIVE = 7, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_10_INTRA = 8, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_422_INTRA = 9, > V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_444_INTRA = 10, > V4L2_MPEG_VIDEO_H264_PROFILE_CAVLC_444_INTRA = 11, > V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_BASELINE = 12, > V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_HIGH = 13, > V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_HIGH_INTRA = 14, > V4L2_MPEG_VIDEO_H264_PROFILE_STEREO_HIGH = 15, > V4L2_MPEG_VIDEO_H264_PROFILE_MULTIVIEW_HIGH = 16, > V4L2_MPEG_VIDEO_H264_PROFILE_CONSTRAINED_HIGH = 17, > }; > > I don't think you want all these lengthy enums in the spec. Right now the virtio-video spec takes 18 pages. > With all the enums it will be much more. And I think it is really pointless to copy them. > > This is it. So no, this device doesn't just expose V4L2 as virtio-media. The V4L2 references are very limited. > During draft v6 discussions the spec length was brought as one of the issues. Therefore the references to V4L2 formats and controls. > I believe the current state is really the right amount of referencing, also it was discussed in the past, so I don't like to change this. > Of course it is possible to rephrase something on case by case basis if you insist. > I need to understand why the V4l2 is not enough. So let me read the links you provided. > >>>> +\end{description} > >>>> + > >>>> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > >>>> + > >>>> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. > >>>> + > >>>> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or > >>>> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would > >>>> +mean that no memory can be used at all for resources. > >>>> + > >>>> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets > >>>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. > >>>> + > >>>> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} > >>>> + > >>>> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and > >>>> +VIRTIO_VIDEO_F_DECODER features. > >>>> + > >>>> +The driver MUST negotiate at least one of the > >>>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT > >>>> +features. > >>>> + > >>> You wouldn't need virtio_object feature bit. It should be covered using the capability. > >> > >> Well, Michael says it is ok > >> > > I am not sure. Once you structure using capability unnecessary feature bits goes away. > > Feature bits offer simplicity but are only useful when one needs them before DRIVER_OK phase. > > So it is unclear why/what driver needs to do with this bit before DRIVER_OK stage. > > > >>>> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not > >>>> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically > >>>> +contiguous memory for all the buffers it allocates. > >>>> + > >>>> +\subsection{Device configuration layout} > >>>> +\label{sec:Device Types / Video Device / Device configuration layout} > >>>> + > >>>> +The video device configuration space uses the following layout: > >>>> + > >>>> +\begin{lstlisting} > >>>> +struct virtio_video_config { > >>>> + le32 max_streams; > >>>> + le32 caps_length; > >>>> +}; > >>>> +\end{lstlisting} > >>>> + > >>>> +\begin{description} > >>>> + \item[\field{max_streams}] > >>>> + is the maximum number of concurrent streams the device supports. > >>>> + \item[\field{caps_length}] > >>>> + is the minimum length in bytes that a device-writable buffer must have > >>>> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see > >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. > >>>> +\end{description} > >>>> + > >>>> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} > >>>> + > >>>> +\field{max_streams} MUST be positive. > >>>> + > >>>> +\field{caps_length} MUST be set to the response size of > >>>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. > >>>> + > >>> This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. > >>> Please rework the patch to use the existing basic facility. > >>> This can also possibly eliminate plumbing device specific command q. > >>> More below. > >> > >> I'm still learning about capabilities and admin commands, so please correct me if I'm wrong. > >> Here are some considerations: > >> 1. AFAIU capabilities and admin commands need an admin virtqueue. The latter depends on the VIRTIO_F_ADMIN_VQ feature. What if the > >> device doesn't support it? > > It isn’t the right question. > > It is a question similar to: what if the virtio-net device does not support control VQ, can it still enable multi-queue? > > No, it cannot. Hence virtio-net to implement cvq to support MQ. > > What if the virtio-blk device does not support AQ, can it will support device parts basic facility? > > No. it cannot. > > > > Does it mean these devices should invent new queue, instead of admin vq? > > No. > > The devices are encouraged to use the existing basic facilities of chapter 2 that is used by various features and devices such as virtio-net, > block, crypto. > > > >> Also it looks like the admin virtqueues are only defined and implemented in Virtio Over PCI at the moment, > >> "reserved for future use" in Virtio Over MMIO and in Virtio Over Channel I/O and also not available in vhost-user. For us this makes admin > >> queues unusable for now. > > This is not a limitation. AQ support can be added for MMIO transport or channel transport too. > > Admin VQ is not any different beast than newly proposed command queue. > > This has to be done by someone first. This is definitely outside the scope of this patch. For now there are no admin queues in MMIO transport, > this is a problem for us. Ok. Please have a pre-patch to this series. That should be fine. It cannot be the reason to not do it. In virtio community, basic facilities are built so that wider use can happen. Since you mentioned about hw implementation, it would be interesting to see MMIO hw implementation. You can see that virtio-msg patches are build by community for multiple users. They are not creating FFA or Xen specific shortcuts. > How about agreeing to do as follows: > 1. For now virtio-video doesn't require an admin queue, they can be enabled later with a feature flag when at least MMIO transport is ready > (possibly together with media, gpu, etc). I don’t think this will ever happen. 😊 Better to do it now specially when new queue is being introduced. > 2. Maybe you can suggest certain specific things that can be already done in virtio-video to make the future transition to admin queues > simpler. Like maybe moving the command type IDs into a certain range (0x8000 - 0xFFFF ??? but it doesn't say this is reserved for device- > specific commands though) and maybe changing some types. Capabilities don't fit for now because they depend on the admin queue AFAIU. > Please note that we'd like to avoid creating any unnecessary extra virtqueues so that we can save limited HW resources. Therefore it should > be either a commandq, or an admin queue with both admin commands and device-specific commands mixed and consumed in strict order. I am confused. You mentioned below that you prefer them in-order. What did I miss? > > >> 2. The virtio capabilities that you mention seem to be about enabling and disabling certain functionality based on a bitmap, right? (Still > looking > >> into the flow filter example, which seems to be more complex.) The virtio-video caps are more like an attempt to represent a graph of > >> dependencies between different coded and raw formats, resolutions, etc. For example there will normally be multiple TLVs with the same > >> types (coded and raw sets) "linked" to each other. I'm not sure this maps well into the existing virtio capabilities. > > Sure it can map. Capabilities can be define them. > > > >> 3. If this is really targeted towards SR-IOV maybe it is better to keep it this way. > > No. it is not targeted towards SR-IOV. > > It is basic facilities uses for SR-IOV, flow filters, crypto across 3 different device types. > > > >> I mean we may need admin commands to create a virtio- > >> video device per guest VM, so maybe using the same admin commands + some device-specific commands inside each device could be > >> confusing. The nesting actually continues here, because virtio-video streams have shared buffers called "resources" (probably these should > be > >> renamed). > >> 4. Also I'm not sure how does it actually improve the spec/implementation. > >> > > It improves the spec by reusing the basic facilities of the spec and avoids creating new objects. > > This reduces the spec burden. > > It also reuses the existing implementation of the device and driver. > > OK, got it. Replied above. > > >>>> +\subsection{Device Initialization} > >>>> +\label{sec:Device Types / Video Device / Device Initialization} > >>>> +\begin{enumerate} > >>>> + \item > >>>> + The driver reads the feature bits and negotiates the features it needs. > >>>> + \item > >>>> + The driver sets up the commandq and the eventq. > >>>> + \item > >>>> + The driver reads the \field{caps_length} field of the configuration > >>>> + space, prepares a buffer of at least that size and sends the buffer on the > >>>> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. > >>>> + \item > >>>> + The device sends a response over commandq to > >>>> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the > >>>> + command. > >>>> + \item > >>>> + The driver receives the response from the device, and parses its capabilities. > >>>> +\end{enumerate} > >>>> + > >>>> +\subsection{Device Operation} > >>>> +\label{sec:Device Types / Video Device / Device Operation} > >>>> + > >>>> +The device supports opening and operating a number of parallel streams up to > >>>> +\field{max_streams}. Each stream has three internal device queues: mainqX, > >>>> +inputqX and outputqX, where X is the stream id. Each stream command has a > >>>> +field that dispatches the command to the specific internal queue. > >>>> + > >>> If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, > >>> Than keyword 'internal' should be dropped and rephased. > >> > >> As Michael already assumed the implication is that within each queue commands are consumed in order. > > Admin virtqueue can be extended to support out of order. > > One can always add a new first admin command to operate in out of order mode. > > Or even a feature bit to have AQ_OUT_OF_ORDER. > > And again this is optimization. > > In other thread we discussed that optimization may come later. > > But if you think it is essential to have the good out of order performance now (and how much it improves - rough estimates), it is worth to > extend AQ for out of order. > > Even SR-IOV can benefit of this feature right away for reading and writing device parts of unrelated VFs. > > So there is already more than one user of the out of order feature you suggest. 😊 > > I think there is some misunderstanding. We actually want all the queues including the internal ones to be consumed in order. > Otherwise the decoding just falls apart. > Ok. So in-order is already supported using VIRTIO_F_IN_ORDER. > >> On top of this there are queue > >> priorities: mainq has higher priority than inputq and outputq. This enables resetting inputq and outputq or setting parameters out of band. > >> > > So multiple admin queues are also supported. > > Before starting these queues, their priority can be set using a new command. > > As I wrote above we'd like to avoid creating extra virtqueues unless necessary. This is a requirement from our HW engineers. > If every stream always has 3 real virtqueues, this is definitely a problem. So this spec has a scheme to multiplex several stream queues into > the single commandq. > I don't see any generic multiplexing scheme in the admin queues for now, so this is also an issue if we consider them managing the internal > queues. Admin queue can carry device specific commands for each stream_id. Stream_id is the object id. Later on Friday, I will try to find a mapping of commands you proposed to admin q based commands, (in case if you have not figured out by then). > > >>>> +% Use-case: there might be different real-time requirements for different > >>>> +% streams, so more virtqueues can be added in the future if necessary. > >>>> +% The internal queues don't change, the data formats don't change, only the > >>>> +% mapping of streams/internal queues to particular virtqueues changes. > >>>> + > >>>> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, > >>>> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX > >>>> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out > >>>> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock > >>>> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for > >>>> +format negotiation. > >>>> + > >>>> +The inputqX and outputqX are used to queue input or output resources using > >>>> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and > >>>> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete > >>>> +processing of all queued input resources and make the resulting output > >>>> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. > >>>> + > >>>> +All the stream commands start async operations, return the results > >>>> +using async responses over eventq. > >>>> +The eventq is used by the device to send the device's async responses to > >>>> +stream commands and the device's standalone events. > >>>> + > >>>> +% This way eventq becomes the single source of truth about the device's state, > >>>> +% the driver doesn't have to tediously synchronize commandq's and eventq's > >>>> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF > >>>> +% and DQEVENT). One more benefit is that commandq is processed fast and > >>>> +% strictly in order, so commandq descriptors exhaustion should never happen in > >>>> +% practice. > >>>> + > >>> Video device should be implementable without any V4L2 binding/description etc. > >> > >> Well, as I wrote above some V4L2 definitions are used anyway. It was agreed that this is fine and that we can have V4L2 headers as > normative > >> references during the v6 review. > >> In this draft I decided to add some comments describing the reasons behind some design decisions because I keep forgetting them myself. I > >> can edit this particular comment though of course. > >> > > Lets first understand why existing media device is not enough. > > > >>>> +Parameters allow the driver to configure the stream including setting up the > >>>> +resources. Available parameters depend on the device type, see > >>>> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. > >>>> + > >>>> +A resource is a set of memory buffers that contain a unit of data that > >>>> +the device can process or produce. Most resources have only one buffer, > >>>> +raw frames using a multi-planar format can have several. > >>>> +Input resources are filled by the driver with compressed (coded) video data > >>>> +for a decoder and raw frames for an encoder, output resources are filled by > >>>> +the device as the result of processing the input resources with decoded raw > >>>> +frames for a decoder and compressed (encoded) data for an encoder. > >>>> +Resources from inputqX and outputqX are consumed independently, not in pairs. > >>>> +One input resource can result in zero to many produced output resources. > >>>> +A decoder device dequeues the output decoded frames in presentation order. > >>>> +An encoder device dequeues the output decoded frames in decoding order. > >>>> +The driver can reuse a queued resource after receiving a corresponding async > >>>> +response. Dequeued output resources can still be used by the device as > >>>> +reference frames, so the driver can't write to them. > >>>> + > >>>> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued > >>>> +% output resource is not used by the device anymore and therefore becomes > >>>> +% writeable? > >>>> + > >>>> +The device can detect standalone stream-related events: errors and dynamic > >>>> +parameters changes that require intervention from the driver (e.g. > >>>> +reallocating backing memory of output resources to fit the new parameters). > >>>> +The events are signalled on the eventq, see > >>>> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > >>>> + > >>>> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} > >>>> + > >>>> +The device MUST set to zero all unused, disabled or padding bits in its > >>>> +responses. > >>>> + > >>>> +\subsubsection{Device Operation: Command Virtqueue} > >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} > >>>> + > >>>> +This section lists the commands that can be sent by the driver to commandq. > >>>> + > >>>> +Different structures are used for each command and response. A command > >>>> +structure starts with the requested command code, defined as follows: > >>>> + > >>>> +\begin{lstlisting} > >>>> +/* Device */ > >>>> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 > >>>> + > >>>> +/* Stream */ > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 > >>> You can craft the stream using a existing basic facility of resource object. > >>> Where each stream is just a resource object, that be queried or modified. > >> > >> Replied about the admin commands above. > >> > > Comments above. > > This can be done using admin commands easily. > > > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 > >>>> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 > >>>> +\end{lstlisting} > >>>> + > >>>> +Stream commands start with a header: > >>>> + > >>>> +\begin{lstlisting} > >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 > >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 > >>>> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 > >>>> + > >>>> +struct virtio_video_stream_cmd_header { > >>>> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ > >>>> + le32 stream_id; > >>>> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ > >>>> + le32 async_response_cookie; > >>>> +}; > >>>> +\end{lstlisting} > >>>> + > >>>> +\begin{description} > >>>> + \item[\field{async_response_cookie}] > >>>> + is an async response cookie provided by the driver, that allows > >>>> + to relate an async response to the previously submitted command. > >>>> +\end{description} > >>>> + > >>>> +\subsubsection{Device Operation: Event Virtqueue} > >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} > >>>> + > >>>> +The eventq is used by the device to send async responses to commands queued > >>>> +by the driver on commandq and standalone events. Stream errors and dynamic > >>>> +parameters changes are caused by changes in the device's state, not by > >>>> +commands, still they are delivered as responses to implicit > >>>> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, > >>>> +respectively. > >>>> + > >>>> +Events start with a header: > >>>> + > >>>> +\begin{lstlisting} > >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) > >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) > >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) > >>>> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) > >>>> + > >>>> +struct virtio_video_event_header { > >>>> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ > >>>> + le32 stream_id; > >>>> + le32 async_response_cookie; > >>>> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ > >>>> +}; > >>>> +\end{lstlisting} > >>>> + > >>>> +\begin{description} > >>>> + \item[\field{event_type}] > >>>> + is the type of the event. > >>>> + \item[\field{stream_id}] > >>>> + is the ID of a valid stream. > >>>> + \item[\field{async_response_cookie}] > >>>> + is an async response cookie provided by the driver, that allows > >>>> + to relate the event to a previously submitted command. > >>>> + \item[\field{event_flags}] > >>>> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags > >>>> + > >>>> + \begin{description} > >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] > >>>> + is set if the command finished with an error due to an > >>>> + invalid argument or for other reasons. > >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] > >>>> + is set for standalone events, see > >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. > >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] > >>>> + is set if the command has been canceled by another > >>>> + command, that has higher priority. Doesn't make sense > >>>> + for standalone events. > >>>> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] > >>>> + is set if the command triggered a block on the > >>>> + outputqX to allow output format negotiation. > >>>> + When the negotiation is finished the block has to be > >>>> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK > >>>> + command, see > >>>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. > >>>> + \end{description} > >>>> +\end{description} > >>>> + > >>>> +The particular data structure representing the event is selected according to > >>>> +the \field{event_type}. > >>>> + > >>>> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: > >>>> Event Virtqueue} > >>>> + > >>>> +The driver MUST at any time have at least one descriptor with a used > >>>> +buffer large enough to contain a \field{struct virtio_video_event} > >>>> +queued on the eventq. > >>>> + > >>>> +The driver MUST NOT put device-readable descriptors into the eventq. > >>>> + > >>>> +The driver MUST account for the fact that the async responses to commands > >>>> +might come out-of-order (i.e. after other commands sent to the device), > >>>> +and that some of them can be cancelled. > >>>> + > >>>> +The driver SHOULD wait for an async response of command A, that caused > >>>> +cancellation of command B, before queueing the command B again. > >>>> + > >>>> +\subsubsection{Device Operation: TLV format} > >>>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} > >>>> + > >>>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ > >>>> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS > >>>> +commands represent device capabilities and corresponding device parameters > >>>> +in the form of TLV (Type-Length-Value): > >>>> + > >>>> +\begin{lstlisting} > >>>> +struct virtio_video_tlv { > >>>> + le32 type; > >>>> + le32 length; > >>>> + u8 value[length]; > >>>> +}; > >>>> +\end{lstlisting} > >>>> + > >>> All the caps can be crafted using existing capabilities infra. Please restructure the patch to use it. > >> > >> Replied above. > >> > > Same as above. > > > >>>> +\begin{description} > >>>> + \item[\field{type}] > >>>> + specifies the type of data in \field{value}. > >>>> + \item[\field{length}] > >>>> + specifies the \field{value} size in bytes aligned to 4 bytes. > >>>> + \item[\field{value}] > >>>> + contains the data according to the type. > >>>> +\end{description} > >>>> + > >>>> +The following TLV types are defined: > >>>> + > >>>> +\begin{lstlisting} > >>>> +#define VIRTIO_VIDEO_TLV_CODED_SET 1 > >>>> +#define VIRTIO_VIDEO_TLV_RAW_SET 2 > >>>> +#define VIRTIO_VIDEO_TLV_LINK 3 > >>>> +#define VIRTIO_VIDEO_TLV_CODED_FORMAT 4 > >>>> +#define VIRTIO_VIDEO_TLV_RAW_FORMAT 5 > >>>> +#define VIRTIO_VIDEO_TLV_CODED_RESOURCES 6 > >>>> +#define VIRTIO_VIDEO_TLV_RAW_RESOURCES 7 > >>>> +#define VIRTIO_VIDEO_TLV_RESOURCE_GUEST_PAGES 8 > >>>> +#define VIRTIO_VIDEO_TLV_RESOURCE_VIRTIO_OBJECT 9 > >>>> +#define VIRTIO_VIDEO_TLV_CROP 10 > >>>> +#define VIRTIO_VIDEO_TLV_V4L2_CONTROLS 11 > >>>> +\end{lstlisting} > >>>> + > >>> You can use the 'flow filter' example of network device to see how to frame the individual or group of capabilities. > >> > >> Checking it, thanks. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-02-25 14:38 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-02-13 8:53 [PATCH v10 0/1] Virtio video device specification Alexander Gordeev 2026-02-13 8:53 ` [PATCH v10 1/1] virtio-video: Add virtio " Alexander Gordeev 2026-02-15 10:49 ` Parav Pandit 2026-02-15 15:11 ` Michael S. Tsirkin 2026-02-25 14:21 ` Alexander Gordeev 2026-02-24 20:02 ` Alexander Gordeev 2026-02-25 4:59 ` Parav Pandit 2026-02-25 14:04 ` Alexander Gordeev 2026-02-25 14:38 ` Parav Pandit
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox