[virtio-dev] Re: [RFC PATCH v6] virtio-video: Add virtio video device specification

[Alexander Gordeev <alexander.gordeev@opensynergy.com>]

Hello,

Sorry for the delay. I'm still gathering data in the old emails...

On 08.12.22 08:23, Alexandre Courbot wrote:
> Add the specification of the video decoder and encoder devices, which
> can be used to provide host-accelerated video operations to the guest.
>
> Signed-off-by: Keiichi Watanabe <keiichiw@chromium.org>
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> --
> Here is the long-overdue new revision of the virtio-video RFC. This
> version reorganizes the specification quite a bit and tries to simplify
> the protocol further. Nonetheless, it still results in a rather long (17
> pages) specification for just these devices, even though the spec is not
> fully complete (I want to rethink the formats descriptions, and some
> parameters need to be added for the encoder device).
>
> I would like to get some high-level feedback on this version and maybe
> propose to do things a bit differently before people invest too much
> time reviewing this in depth. While rewriting this document, it became
> more and more obvious that this is just a different, and maybe a bit
> simpler, reimplementation of the V4L2 stateless decoder protocol [1]. I
> am now wondering whether it would not make more sense to rewrite this
> specification as just a way to transport V4L2 requests over virtio,
> similarly to how virtio-fs does with the FUSE protocol [2].
>
> At the time we started writing this implementation, the V4L2 protocols
> for decoders and encoders were not set in stone yet, but now that they
> are it might make sense to reconsider. Switching to this solution would
> greatly shorten the virtio-video device spec, and also provide a way to
> support other kind of V4L2 devices like cameras or image processors at
> no extra cost.
>
> Note that doing so would not require that either the host or guest uses
> V4L2 - the virtio video device would just emulate a V4L2 device over
> virtio. A few adaptations would need to be done regarding how memory
> types work, but otherwise I believe most of V4L2 could be used as-is.
>
> Please share your thoughts about this, and I will either explore this
> idea further with a prototype, or keep moving the present spec forward,
> hopefully at a faster pace.
>
> Due to the RFC state of this patch I have refrained from referencing the
> normative statements in conformance.tex - I will do that as a final step
> once the spec is mostly agreed on.
>
> [1] https://ddec1-0-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdocs.kernel.org%2fuserspace%2dapi%2fmedia%2fv4l%2fdev%2dstateless%2ddecoder.html&umid=8ec8d8c9-b83c-40de-9337-a377056fe2af&auth=53c7c7de28b92dfd96e93d9dd61a23e634d2fbec-e98508782bc1c9aa6b2e4a9df9d4dd170f9a5ffa
> [2] https://github.com/oasis-tcs/virtio-spec/blob/master/virtio-fs.tex
>
> Full PDF:
> https://ddec1-0-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdrive.google.com%2ffile%2fd%2f1HRVDiDdo50%2dS9X5tWgzmT90FJRHoB1dN%2fview%3fusp%3dsharing&umid=8ec8d8c9-b83c-40de-9337-a377056fe2af&auth=53c7c7de28b92dfd96e93d9dd61a23e634d2fbec-e315af79a067165e908bf1d803441eb181e2f375
>
> PDF of video section only:
> https://ddec1-0-en-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fdrive.google.com%2ffile%2fd%2f1Sm6LSqvKqQiwYmDE9BXZ0po3XTKnKYlD%2fview%3fusp%3dsharing&umid=8ec8d8c9-b83c-40de-9337-a377056fe2af&auth=53c7c7de28b92dfd96e93d9dd61a23e634d2fbec-b5c45bb4b6ccc5b73ea2a54e26f151d61722d0df
> ---
>   content.tex      |    1 +
>   virtio-video.tex | 1420 ++++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 1421 insertions(+)
>   create mode 100644 virtio-video.tex
>
> ...snip...
>
> +\subsection{Supported formats}\label{sec:Device Types / Video Device / Supported formats}
> +
> +Bitstream and image formats are identified by their fourcc code, which
> +is a four-bytes ASCII sequence uniquely identifying the format and its
> +properties.
> +
> +\subsubsection{Bitstream formats}\label{sec:Device Types / Video Device / Supported formats / Bitstream formats}
> +
> +The fourcc code of each supported bitstream format is given, as well as
> +the unit of data requested in each input resource for the decoder, or
> +produced in each output resource for the encoder.
> +
> +\begin{description}
> +\item[\field{MPG2}]
> +MPEG2 encoded stream. One Access Unit per resource.
> +\item[\field{H264}]
> +H.264 encoded stream. One NAL unit per resource.
> +\item[\field{HEVC}]
> +HEVC encoded stream. One NAL unit per resource.
> +\item[\field{VP80}]
> +VP8 encoded stream. One frame per resource.
> +\item[\field{VP90}]
> +VP9 encoded stream. One frame per resource.
> +\end{description}
> +

Actually I'm not sure where do these fourcc codes for bitstream formats
come from.

In one of the old comments I found a reference to
https://www.rfc-editor.org/rfc/rfc2361

But it doesn't define MPG2, HEVC, VP80, VP90.

Also there is this comment to virtio-video v1 from Tomasz Figa and the
related discussion:
https://markmail.org/message/gc6h25acct22niut#query:+page:1+mid:et4l3ni7qjqhiygo+state:results

He wrote that it is not worth it because there are so many conflicting
sets of fourcc codes.

I can see that fourcc codes were not used in virtio-video spec draft
versions 1 to 5. So now it looks quite weird to see them here. Probably
this is because you'd like to mimic v4l2 interface more closely?

> +\subsubsection{Image formats}\label{sec:Device Types / Video Device / Supported formats / Image formats}
> +
> +The fourcc code of each supported image format is given, as well as its
> +number of planes, physical buffers, and eventual subsampling.
> +
> +\begin{description}
> +\item[\field{RGB3}]
> +one RGB plane where each component takes one byte, i.e.~3 bytes per
> +pixel.
> +\item[\field{NV12}]
> +one Y plane followed by interleaved U and V data, in a single buffer.
> +4:2:0 subsampling.
> +\item[\field{NV12}]
> +same as \field{NV12} but using two separate buffers for the Y and UV
> +planes.
> +\item[\field{YU12}]
> +one Y plane followed by one Cb plane, followed by one Cr plane, in a
> +single buffer. 4:2:0 subsampling.
> +\item[\field{YM12}]
> +same as \field{YU12} but using three separate buffers for the Y, U and V
> +planes.
> +\end{description}

--
Alexander Gordeev
Senior Software Engineer

OpenSynergy GmbH
Rotherstr. 20, 10245 Berlin

Phone: +49 30 60 98 54 0 - 88
Fax: +49 (30) 60 98 54 0 - 99
EMail: alexander.gordeev@opensynergy.com

www.opensynergy.com

Handelsregister/Commercial Registry: Amtsgericht Charlottenburg, HRB 108616B
Geschäftsführer/Managing Director: Régis Adjamah


Please mind our privacy notice<https://www.opensynergy.com/datenschutzerklaerung/privacy-notice-for-business-partners-pursuant-to-article-13-of-the-general-data-protection-regulation-gdpr/> pursuant to Art. 13 GDPR. // Unsere Hinweise zum Datenschutz gem. Art. 13 DSGVO finden Sie hier.<https://www.opensynergy.com/de/datenschutzerklaerung/datenschutzhinweise-fuer-geschaeftspartner-gem-art-13-dsgvo/>

---------------------------------------------------------------------
To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org