[PATCH 0/8] virtio: Add vhost-user based Video decode

[PATCH 0/8] virtio: Add vhost-user based Video decode

[Peter Griffin]

This series adds support for virtio-video decoder devices in Qemu
and also provides a vhost-user-video vmm implementation.

The vhost-user-video vmm currently parses virtio-vido v3 protocol
(as that is what the Linux frontend driver implements).
It then converts that to a v4l2 mem2mem stateful decoder device.
Currently this has been tested using v4l2 vicodec test driver in Linux
[1] but it is intended to be used with Arm SoCs which often implement
v4l2 stateful decoders/encoders drivers for their video accelerators.

The primary goal so far has been to allow continuing development
of virtio-video Linux frontend driver and testing with Qemu. Using
vicodec on the host allows a purely virtual dev env, and allows for
ci integration in the future by kernelci etc.

This series also adds the virtio_video.h header and adds the
FWHT format which is used by vicodec driver.

I have tested this VMM using v4l2-ctl from v4l2 utils in the guest
to do a video decode to a file. This can then be validated using ffplay
v4l2-compliance tool in the guest has also been run which stresses the
interface and issues lots of syscall level tests

See the README.md for example commands on how to configure guest kernel
and do a video decode using Qemu, vicodec using this VMM.

Linux virtio-video frontend driver code:
https://github.com/petegriffin/linux/commits/v5.10-virtio-video-latest

Qemu vmm code:
https://github.com/petegriffin/qemu/tree/vhost-virtio-video-master-v1

This is part of a wider initiative by Linaro called
"project Stratos" for which you can find information here:

  https://collaborate.linaro.org/display/STR/Stratos+Home

Applies cleanly to git://git.qemu.org/qemu.git master(a3607def89).

Thanks,

Peter.

[1] https://lwn.net/Articles/760650/

Peter Griffin (8):
  vhost-user-video: Add a README.md with cheat sheet of commands
  MAINTAINERS: Add virtio-video section
  vhost-user-video: boiler plate code for vhost-user-video device
  vhost-user-video: add meson subdir build logic
  standard-headers: Add virtio_video.h
  virtio_video: Add Fast Walsh-Hadamard Transform format
  hw/display: add vhost-user-video-pci
  tools/vhost-user-video: Add initial vhost-user-video vmm

 MAINTAINERS                                   |    8 +
 hw/display/Kconfig                            |    5 +
 hw/display/meson.build                        |    3 +
 hw/display/vhost-user-video-pci.c             |   82 +
 hw/display/vhost-user-video.c                 |  386 ++++
 include/hw/virtio/vhost-user-video.h          |   41 +
 include/standard-headers/linux/virtio_video.h |  484 +++++
 tools/meson.build                             |    9 +
 tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
 tools/vhost-user-video/README.md              |   98 +
 tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
 tools/vhost-user-video/meson.build            |   10 +
 tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
 tools/vhost-user-video/v4l2_backend.h         |   99 +
 tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
 tools/vhost-user-video/virtio_video_helpers.h |  166 ++
 tools/vhost-user-video/vuvideo.h              |   43 +
 17 files changed, 5358 insertions(+)
 create mode 100644 hw/display/vhost-user-video-pci.c
 create mode 100644 hw/display/vhost-user-video.c
 create mode 100644 include/hw/virtio/vhost-user-video.h
 create mode 100644 include/standard-headers/linux/virtio_video.h
 create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
 create mode 100644 tools/vhost-user-video/README.md
 create mode 100644 tools/vhost-user-video/main.c
 create mode 100644 tools/vhost-user-video/meson.build
 create mode 100644 tools/vhost-user-video/v4l2_backend.c
 create mode 100644 tools/vhost-user-video/v4l2_backend.h
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
 create mode 100644 tools/vhost-user-video/vuvideo.h

-- 
2.25.1

[PATCH 1/8] vhost-user-video: Add a README.md with cheat sheet of commands

[Peter Griffin]

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 tools/vhost-user-video/README.md | 98 ++++++++++++++++++++++++++++++++
 1 file changed, 98 insertions(+)
 create mode 100644 tools/vhost-user-video/README.md

diff --git a/tools/vhost-user-video/README.md b/tools/vhost-user-video/README.md
new file mode 100644
index 0000000000..c55e0a7b68
--- /dev/null
+++ b/tools/vhost-user-video/README.md
@@ -0,0 +1,98 @@
+# Overview vhost-user-video
+
+This vmm translates from virtio-video v3 protocol and writes
+to a v4l2 mem2mem stateful decoder/encoder device [1]. v3 was
+chosen as that is what the virtio-video Linux frontend driver
+currently implements.
+
+The primary goal so far is to enable development of virtio-video
+frontend driver using purely open source software. Using vicodec
+v4l2 stateful decoder on the host for testing then allows a pure
+virtual environment for development and testing.
+
+Currently the vmm only supports v4l2 stateful devices, and the
+intention is it will be used with Arm SoCs that implement stateful
+decode/encode devices such as Qcom Venus, RPi, MediaTek etc.
+
+A Qemu + vicodec setup for virtio-video should also allow for
+CI systems like kernelci, lkft to test the virtio-video interface
+easily.
+
+Currently support for VAAPI or decoding via libavcodec or similar
+libraries is not implemented, but this could be added in the future.
+
+Some example commands are provided below on how to run the daemon
+and achieve a video decode using vicodec and a link to some test
+content.
+
+[1] https://www.kernel.org/doc/html/latest/userspace-api/media/
+    v4l/dev-decoder.html
+
+[2] https://lwn.net/Articles/760650/
+
+# Guest Linux kernel modules
+CONFIG_MEDIA_SUPPORT=y
+CONFIG_MEDIA_TEST_SUPPORT=y
+CONFIG_V4L_TEST_DRIVERS=y
+CONFIG_VIRTIO_VIDEO=y
+CONFIG_GDB_SCRIPTS=y
+CONFIG_DRM_VIRTIO_GPU=y
+
+# Host kernel modules
+CONFIG_MEDIA_SUPPORT=y
+CONFIG_MEDIA_TEST_SUPPORT=y
+CONFIG_V4L_TEST_DRIVERS=y
+CONFIG_VIDEO_VICODEC=y
+
+# Run vhost-user-video daemon with vicodec
+# (video3 typically is the stateful video)
+vhost-user-video --socket-path=/tmp/video.sock --v4l2-device=/dev/video3
+
+# Qemu command for virtio-video device
+
+-device vhost-user-video-pci,chardev=video,id=video
+-chardev socket,path=/tmp//video.sock,id=video
+
+# Example v4l2-ctl decode command
+wget https://people.linaro.org/~peter.griffin/jelly_640_480-420P.fwht
+
+v4l2-ctl -d0 -x width=640,height=480 -v width=640,height=480,pixelformat=YU12
+--stream-mmap --stream-out-mmap --stream-from jelly_640_480-420P.fwht
+--stream-to out-jelly-640-480.YU12
+
+# Play the raw decoded video with ffplay or mplayer
+ffplay -loglevel warning -v info -f rawvideo -pixel_format  yuv420p
+  -video_size "640x480" ./out-jelly-640-480.YU12
+
+mplayer -demuxer rawvideo -rawvideo
+  format=i420:w=640:h=480:fps=25 out-jelly-640-480.YU12
+
+# Enable v4l2 debug in virtio-video frontend driver
+echo 0x1f > /sys/class/video4linux/video0/dev_debug
+
+# Enable v4l2 debug in vicodec backend driver
+echo 0x1f > /sys/class/video4linux/video3/dev_debug
+
+# optee-build system qemu virtio-video command
+make QEMU_VIRTFS_ENABLE=y QEMU_USERNET_ENABLE=y CFG_TA_ASLR=n
+    QEMU_VHOSTUSER_MEM=y QEMU_VIRTVIDEO_ENABLE=y SSH_PORT_FW=y run-only
+
+Current status
+* Tested with v4l2-ctl from v4l2-utils and vicodec stateful decoder driver
+* v4l2-compliance - reports
+Total: 43, Succeeded: 37, Failed: 6, Warnings: 0
+
+Known Issues
+* 6 v4l2-compliance failures remaining
+* v4l2-ctl 0fps misleading output
+* v4l2-ctl sometimes reports - 0 != <somenumber>
+* Encoder not tested yet
+
+TODOs
+* Test with a "real" stateful decoder & codec
+  (e.g. Qcom Venus or RPi).
+* Test more v4l2 userspaces in the guest
+
+Future potential features
+* Emulation using libavcodec or similar library
+* Support for VAAPI, OpenMax or v4l2 stateless devices
-- 
2.25.1

Re: [PATCH 1/8] vhost-user-video: Add a README.md with cheat sheet of commands

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  tools/vhost-user-video/README.md | 98 ++++++++++++++++++++++++++++++++
>  1 file changed, 98 insertions(+)
>  create mode 100644 tools/vhost-user-video/README.md
>
> diff --git a/tools/vhost-user-video/README.md b/tools/vhost-user-video/README.md
> new file mode 100644
> index 0000000000..c55e0a7b68
> --- /dev/null
> +++ b/tools/vhost-user-video/README.md

I think as we want this to be visible to the user we should put it in:

  docs/system/devices/vhost-user-video.rst

with the appropriate update to device-emulation.rst to include it.

> @@ -0,0 +1,98 @@
> +# Overview vhost-user-video
> +
> +This vmm translates from virtio-video v3 protocol and writes
> +to a v4l2 mem2mem stateful decoder/encoder device [1]. v3 was
> +chosen as that is what the virtio-video Linux frontend driver
> +currently implements.
> +
> +The primary goal so far is to enable development of virtio-video
> +frontend driver using purely open source software. Using vicodec
> +v4l2 stateful decoder on the host for testing then allows a pure
> +virtual environment for development and testing.
> +
> +Currently the vmm only supports v4l2 stateful devices, and the
> +intention is it will be used with Arm SoCs that implement stateful
> +decode/encode devices such as Qcom Venus, RPi, MediaTek etc.
> +
> +A Qemu + vicodec setup for virtio-video should also allow for
> +CI systems like kernelci, lkft to test the virtio-video interface
> +easily.
> +
> +Currently support for VAAPI or decoding via libavcodec or similar
> +libraries is not implemented, but this could be added in the future.
> +
> +Some example commands are provided below on how to run the daemon
> +and achieve a video decode using vicodec and a link to some test
> +content.
> +
> +[1] https://www.kernel.org/doc/html/latest/userspace-api/media/
> +    v4l/dev-decoder.html
> +
> +[2] https://lwn.net/Articles/760650/
> +
> +# Guest Linux kernel modules
> +CONFIG_MEDIA_SUPPORT=y
> +CONFIG_MEDIA_TEST_SUPPORT=y
> +CONFIG_V4L_TEST_DRIVERS=y
> +CONFIG_VIRTIO_VIDEO=y
> +CONFIG_GDB_SCRIPTS=y

Is GDB_SCRIPTS really needed here?

> +CONFIG_DRM_VIRTIO_GPU=y
> +
> +# Host kernel modules
> +CONFIG_MEDIA_SUPPORT=y
> +CONFIG_MEDIA_TEST_SUPPORT=y
> +CONFIG_V4L_TEST_DRIVERS=y
> +CONFIG_VIDEO_VICODEC=y

this last one isn't set on the default Debian kernel so it might be
worth mentioning that.

> +
> +# Run vhost-user-video daemon with vicodec
> +# (video3 typically is the stateful video)
> +vhost-user-video --socket-path=/tmp/video.sock --v4l2-device=/dev/video3
> +
> +# Qemu command for virtio-video device
> +
> +-device vhost-user-video-pci,chardev=video,id=video
> +-chardev socket,path=/tmp//video.sock,id=video
> +
> +# Example v4l2-ctl decode command
> +wget https://people.linaro.org/~peter.griffin/jelly_640_480-420P.fwht
> +
> +v4l2-ctl -d0 -x width=640,height=480 -v width=640,height=480,pixelformat=YU12
> +--stream-mmap --stream-out-mmap --stream-from jelly_640_480-420P.fwht
> +--stream-to out-jelly-640-480.YU12
> +
> +# Play the raw decoded video with ffplay or mplayer
> +ffplay -loglevel warning -v info -f rawvideo -pixel_format  yuv420p
> +  -video_size "640x480" ./out-jelly-640-480.YU12
> +
> +mplayer -demuxer rawvideo -rawvideo
> +  format=i420:w=640:h=480:fps=25 out-jelly-640-480.YU12
> +
> +# Enable v4l2 debug in virtio-video frontend driver
> +echo 0x1f > /sys/class/video4linux/video0/dev_debug
> +
> +# Enable v4l2 debug in vicodec backend driver
> +echo 0x1f > /sys/class/video4linux/video3/dev_debug
> +

and cut here...

The rest can probably be dropped from user facing documentation as long
as we record it somewhere ourselves (is it in the cards?).

> +# optee-build system qemu virtio-video command
> +make QEMU_VIRTFS_ENABLE=y QEMU_USERNET_ENABLE=y CFG_TA_ASLR=n
> +    QEMU_VHOSTUSER_MEM=y QEMU_VIRTVIDEO_ENABLE=y SSH_PORT_FW=y run-only
> +
> +Current status
> +* Tested with v4l2-ctl from v4l2-utils and vicodec stateful decoder driver
> +* v4l2-compliance - reports
> +Total: 43, Succeeded: 37, Failed: 6, Warnings: 0
> +
> +Known Issues
> +* 6 v4l2-compliance failures remaining
> +* v4l2-ctl 0fps misleading output
> +* v4l2-ctl sometimes reports - 0 != <somenumber>
> +* Encoder not tested yet
> +
> +TODOs
> +* Test with a "real" stateful decoder & codec
> +  (e.g. Qcom Venus or RPi).
> +* Test more v4l2 userspaces in the guest
> +
> +Future potential features
> +* Emulation using libavcodec or similar library
> +* Support for VAAPI, OpenMax or v4l2 stateless devices


-- 
Alex Bennée

Re: [PATCH 1/8] vhost-user-video: Add a README.md with cheat sheet of commands

[Peter Griffin]

Hi Alex,

Thanks for your review :)

On Tue, 11 Jan 2022, Alex Bennée wrote:

> 
> Peter Griffin <peter.griffin@linaro.org> writes:
> 
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  tools/vhost-user-video/README.md | 98 ++++++++++++++++++++++++++++++++
> >  1 file changed, 98 insertions(+)
> >  create mode 100644 tools/vhost-user-video/README.md
> >
> > diff --git a/tools/vhost-user-video/README.md b/tools/vhost-user-video/README.md
> > new file mode 100644
> > index 0000000000..c55e0a7b68
> > --- /dev/null
> > +++ b/tools/vhost-user-video/README.md
> 
> I think as we want this to be visible to the user we should put it in:
> 
>   docs/system/devices/vhost-user-video.rst
> 
> with the appropriate update to device-emulation.rst to include it.

OK, will fix in v2.

> 
> > @@ -0,0 +1,98 @@
> > +# Overview vhost-user-video
> > +
> > +This vmm translates from virtio-video v3 protocol and writes
> > +to a v4l2 mem2mem stateful decoder/encoder device [1]. v3 was
> > +chosen as that is what the virtio-video Linux frontend driver
> > +currently implements.
> > +
> > +The primary goal so far is to enable development of virtio-video
> > +frontend driver using purely open source software. Using vicodec
> > +v4l2 stateful decoder on the host for testing then allows a pure
> > +virtual environment for development and testing.
> > +
> > +Currently the vmm only supports v4l2 stateful devices, and the
> > +intention is it will be used with Arm SoCs that implement stateful
> > +decode/encode devices such as Qcom Venus, RPi, MediaTek etc.
> > +
> > +A Qemu + vicodec setup for virtio-video should also allow for
> > +CI systems like kernelci, lkft to test the virtio-video interface
> > +easily.
> > +
> > +Currently support for VAAPI or decoding via libavcodec or similar
> > +libraries is not implemented, but this could be added in the future.
> > +
> > +Some example commands are provided below on how to run the daemon
> > +and achieve a video decode using vicodec and a link to some test
> > +content.
> > +
> > +[1] https://www.kernel.org/doc/html/latest/userspace-api/media/
> > +    v4l/dev-decoder.html
> > +
> > +[2] https://lwn.net/Articles/760650/
> > +
> > +# Guest Linux kernel modules
> > +CONFIG_MEDIA_SUPPORT=y
> > +CONFIG_MEDIA_TEST_SUPPORT=y
> > +CONFIG_V4L_TEST_DRIVERS=y
> > +CONFIG_VIRTIO_VIDEO=y
> > +CONFIG_GDB_SCRIPTS=y
> 
> Is GDB_SCRIPTS really needed here?

No not really, it just makes for a nicer development env as you can have GDB
attached to both the kernel and the daemon, and gdb scripts gives you seom nice
python helper commands for parsing kernel data structures.

> 
> > +CONFIG_DRM_VIRTIO_GPU=y
> > +
> > +# Host kernel modules
> > +CONFIG_MEDIA_SUPPORT=y
> > +CONFIG_MEDIA_TEST_SUPPORT=y
> > +CONFIG_V4L_TEST_DRIVERS=y
> > +CONFIG_VIDEO_VICODEC=y
> 
> this last one isn't set on the default Debian kernel so it might be
> worth mentioning that.

Will fix in v2.

> 
> > +
> > +# Run vhost-user-video daemon with vicodec
> > +# (video3 typically is the stateful video)
> > +vhost-user-video --socket-path=/tmp/video.sock --v4l2-device=/dev/video3
> > +
> > +# Qemu command for virtio-video device
> > +
> > +-device vhost-user-video-pci,chardev=video,id=video
> > +-chardev socket,path=/tmp//video.sock,id=video
> > +
> > +# Example v4l2-ctl decode command
> > +wget https://people.linaro.org/~peter.griffin/jelly_640_480-420P.fwht
> > +
> > +v4l2-ctl -d0 -x width=640,height=480 -v width=640,height=480,pixelformat=YU12
> > +--stream-mmap --stream-out-mmap --stream-from jelly_640_480-420P.fwht
> > +--stream-to out-jelly-640-480.YU12
> > +
> > +# Play the raw decoded video with ffplay or mplayer
> > +ffplay -loglevel warning -v info -f rawvideo -pixel_format  yuv420p
> > +  -video_size "640x480" ./out-jelly-640-480.YU12
> > +
> > +mplayer -demuxer rawvideo -rawvideo
> > +  format=i420:w=640:h=480:fps=25 out-jelly-640-480.YU12
> > +
> > +# Enable v4l2 debug in virtio-video frontend driver
> > +echo 0x1f > /sys/class/video4linux/video0/dev_debug
> > +
> > +# Enable v4l2 debug in vicodec backend driver
> > +echo 0x1f > /sys/class/video4linux/video3/dev_debug
> > +
> 
> and cut here...
> 
> The rest can probably be dropped from user facing documentation as long
> as we record it somewhere ourselves (is it in the cards?).

Yes it is in the cards, will remove in v2.

kind regards,

Peter.

> 
> > +# optee-build system qemu virtio-video command
> > +make QEMU_VIRTFS_ENABLE=y QEMU_USERNET_ENABLE=y CFG_TA_ASLR=n
> > +    QEMU_VHOSTUSER_MEM=y QEMU_VIRTVIDEO_ENABLE=y SSH_PORT_FW=y run-only
> > +
> > +Current status
> > +* Tested with v4l2-ctl from v4l2-utils and vicodec stateful decoder driver
> > +* v4l2-compliance - reports
> > +Total: 43, Succeeded: 37, Failed: 6, Warnings: 0
> > +
> > +Known Issues
> > +* 6 v4l2-compliance failures remaining
> > +* v4l2-ctl 0fps misleading output
> > +* v4l2-ctl sometimes reports - 0 != <somenumber>
> > +* Encoder not tested yet
> > +
> > +TODOs
> > +* Test with a "real" stateful decoder & codec
> > +  (e.g. Qcom Venus or RPi).
> > +* Test more v4l2 userspaces in the guest
> > +
> > +Future potential features
> > +* Emulation using libavcodec or similar library
> > +* Support for VAAPI, OpenMax or v4l2 stateless devices
> 
> 
> -- 
> Alex Bennée

[PATCH 2/8] MAINTAINERS: Add virtio-video section

[Peter Griffin]

Add myself as maintainer of the virtio-video files added
in this series.

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 MAINTAINERS | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/MAINTAINERS b/MAINTAINERS
index 7543eb4d59..43c53aded8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2012,6 +2012,14 @@ F: hw/virtio/vhost-user-rng-pci.c
 F: include/hw/virtio/vhost-user-rng.h
 F: tools/vhost-user-rng/*
 
+virtio-video
+M: Peter Griffin <peter.griffin@linaro.org>
+S: Supported
+F: hw/display/vhost-user-video.c
+F: hw/display/vhost-user-video-pci.c
+F: include/hw/virtio/vhost-user-video.h
+F: tools/vhost-user-video/*
+
 virtio-crypto
 M: Gonglei <arei.gonglei@huawei.com>
 S: Supported
-- 
2.25.1

Re: [PATCH 2/8] MAINTAINERS: Add virtio-video section

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> Add myself as maintainer of the virtio-video files added
> in this series.
>
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

[PATCH 3/8] vhost-user-video: boiler plate code for vhost-user-video device

[Peter Griffin]

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 hw/display/Kconfig                   |   5 +
 hw/display/meson.build               |   3 +
 hw/display/vhost-user-video.c        | 386 +++++++++++++++++++++++++++
 include/hw/virtio/vhost-user-video.h |  41 +++
 4 files changed, 435 insertions(+)
 create mode 100644 hw/display/vhost-user-video.c
 create mode 100644 include/hw/virtio/vhost-user-video.h

diff --git a/hw/display/Kconfig b/hw/display/Kconfig
index a2306b67d8..186163b015 100644
--- a/hw/display/Kconfig
+++ b/hw/display/Kconfig
@@ -118,6 +118,11 @@ config VHOST_USER_VGA
     default y
     depends on VIRTIO_VGA && VHOST_USER_GPU
 
+config VHOST_USER_VIDEO
+    bool
+    default y
+    depends on VIRTIO && VHOST_USER
+
 config DPCD
     bool
     select AUX
diff --git a/hw/display/meson.build b/hw/display/meson.build
index 861c43ff98..48284528cf 100644
--- a/hw/display/meson.build
+++ b/hw/display/meson.build
@@ -37,6 +37,9 @@ softmmu_ss.add(when: 'CONFIG_MACFB', if_true: files('macfb.c'))
 softmmu_ss.add(when: 'CONFIG_NEXTCUBE', if_true: files('next-fb.c'))
 
 specific_ss.add(when: 'CONFIG_VGA', if_true: files('vga.c'))
+specific_ss.add(when: 'CONFIG_VHOST_USER_VIDEO', if_true: files('vhost-user-video.c'))
+specific_ss.add(when: ['CONFIG_VHOST_USER_VIDEO', 'CONFIG_VIRTIO_PCI' ],
+  if_true: files('vhost-user-video-pci.c'))
 
 if config_all_devices.has_key('CONFIG_QXL')
   qxl_ss = ss.source_set()
diff --git a/hw/display/vhost-user-video.c b/hw/display/vhost-user-video.c
new file mode 100644
index 0000000000..506e350365
--- /dev/null
+++ b/hw/display/vhost-user-video.c
@@ -0,0 +1,386 @@
+/*
+ * Vhost-user VIDEO virtio device
+ *
+ * This is the boilerplate for instantiating a vhost-user device
+ * implementing a virtio-video device.
+ *
+ * The virtio video decoder and encoder devices are virtual devices that
+ * support encoding and decoding respectively.
+ *
+ * The actual back-end for this driver is the vhost-user-video daemon.
+ * The code here just connects up the device in QEMU and allows it to
+ * be instantiated.
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "hw/qdev-properties.h"
+#include "hw/virtio/virtio-bus.h"
+#include "hw/virtio/vhost-user-video.h"
+#include "qemu/error-report.h"
+
+/* currently there is no VIDEO enc/dec defined in Linux virtio_ids.h */
+#define VIRTIO_ID_VIDEO_ENC         30
+#define VIRTIO_ID_VIDEO_DEC         31
+#define MAX_CAPS_LEN 4096
+
+static void vhost_user_video_get_config(VirtIODevice *vdev, uint8_t *config)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    struct virtio_video_config *vconfig = (struct virtio_video_config *)config;
+    int ret;
+    Error *local_err = NULL;
+
+    memset(config, 0, sizeof(struct virtio_video_config));
+
+    ret = vhost_dev_get_config(&video->vhost_dev, config,
+                               sizeof(struct virtio_video_config), &local_err);
+    if (ret) {
+        error_report("vhost-user-video: get device config space failed");
+
+        /*TODO vhost_dev_get_config() fails so for now lets just set it here */
+        vconfig = (struct virtio_video_config *)config;
+        vconfig->version = 0;
+        vconfig->max_caps_length = MAX_CAPS_LEN;
+        vconfig->max_resp_length = MAX_CAPS_LEN;
+        return;
+    }
+}
+
+static void vhost_user_video_start(VirtIODevice *vdev)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+    int ret;
+    int i;
+
+    if (!k->set_guest_notifiers) {
+        error_report("binding does not support guest notifiers");
+        return;
+    }
+
+    ret = vhost_dev_enable_notifiers(&video->vhost_dev, vdev);
+    if (ret < 0) {
+        error_report("Error enabling host notifiers: %d", -ret);
+        return;
+    }
+
+    ret = k->set_guest_notifiers(qbus->parent, video->vhost_dev.nvqs, true);
+    if (ret < 0) {
+        error_report("Error binding guest notifier: %d", -ret);
+        goto err_host_notifiers;
+    }
+
+    video->vhost_dev.acked_features = vdev->guest_features;
+
+    ret = vhost_dev_start(&video->vhost_dev, vdev);
+    if (ret < 0) {
+        error_report("Error starting vhost-user-video: %d", -ret);
+        goto err_guest_notifiers;
+    }
+
+    /*
+     * guest_notifier_mask/pending not used yet, so just unmask
+     * everything here.  virtio-pci will do the right thing by
+     * enabling/disabling irqfd.
+     */
+    for (i = 0; i < video->vhost_dev.nvqs; i++) {
+        vhost_virtqueue_mask(&video->vhost_dev, vdev, i, false);
+    }
+
+    return;
+
+err_guest_notifiers:
+    k->set_guest_notifiers(qbus->parent, video->vhost_dev.nvqs, false);
+err_host_notifiers:
+    vhost_dev_disable_notifiers(&video->vhost_dev, vdev);
+}
+
+static void vhost_user_video_stop(VirtIODevice *vdev)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    BusState *qbus = BUS(qdev_get_parent_bus(DEVICE(vdev)));
+    VirtioBusClass *k = VIRTIO_BUS_GET_CLASS(qbus);
+    int ret;
+
+    if (!k->set_guest_notifiers) {
+        return;
+    }
+
+    vhost_dev_stop(&video->vhost_dev, vdev);
+
+    ret = k->set_guest_notifiers(qbus->parent, video->vhost_dev.nvqs, false);
+    if (ret < 0) {
+        error_report("vhost guest notifier cleanup failed: %d", ret);
+        return;
+    }
+
+    vhost_dev_disable_notifiers(&video->vhost_dev, vdev);
+}
+
+static void vhost_user_video_set_status(VirtIODevice *vdev, uint8_t status)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    bool should_start = status & VIRTIO_CONFIG_S_DRIVER_OK;
+
+    if (!vdev->vm_running) {
+        should_start = false;
+    }
+
+    if (video->vhost_dev.started == should_start) {
+        return;
+    }
+
+    if (should_start) {
+        vhost_user_video_start(vdev);
+    } else {
+        vhost_user_video_stop(vdev);
+    }
+}
+
+static uint64_t vhost_user_video_get_features(VirtIODevice *vdev,
+                                      uint64_t requested_features,
+                                      Error **errp)
+{
+    /* currently only support guest pages */
+    virtio_add_feature(&requested_features,
+                       VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES);
+
+    return requested_features;
+}
+
+static void vhost_user_video_handle_output(VirtIODevice *vdev, VirtQueue *vq)
+{
+    /*
+     * Not normally called; it's the daemon that handles the queue;
+     * however virtio's cleanup path can call this.
+     */
+}
+
+static void vhost_user_video_guest_notifier_mask(VirtIODevice *vdev, int idx,
+                                            bool mask)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    vhost_virtqueue_mask(&video->vhost_dev, vdev, idx, mask);
+}
+
+static bool vhost_user_video_guest_notifier_pending(VirtIODevice *vdev, int idx)
+{
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+    return vhost_virtqueue_pending(&video->vhost_dev, idx);
+}
+
+/*
+ * Chardev connect/disconnect events
+ */
+
+static int vhost_user_video_handle_config_change(struct vhost_dev *dev)
+{
+    int ret;
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(dev->vdev);
+    Error *local_err = NULL;
+
+    ret = vhost_dev_get_config(dev, (uint8_t *)&video->conf.config,
+                               sizeof(struct virtio_video_config), &local_err);
+    if (ret < 0) {
+        error_report("get config space failed");
+        return -1;
+    }
+
+    return 0;
+}
+
+const VhostDevConfigOps video_ops = {
+    .vhost_dev_config_notifier = vhost_user_video_handle_config_change,
+};
+
+static int vhost_user_video_connect(DeviceState *dev)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+
+    if (video->connected) {
+        return 0;
+    }
+    video->connected = true;
+
+    /* restore vhost state */
+    if (virtio_device_started(vdev, vdev->status)) {
+        vhost_user_video_start(vdev);
+    }
+
+    return 0;
+}
+
+static void vhost_user_video_disconnect(DeviceState *dev)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+
+    if (!video->connected) {
+        return;
+    }
+    video->connected = false;
+
+    if (video->vhost_dev.started) {
+        vhost_user_video_stop(vdev);
+    }
+
+    vhost_dev_cleanup(&video->vhost_dev);
+}
+
+static void vhost_user_video_event(void *opaque, QEMUChrEvent event)
+{
+    DeviceState *dev = opaque;
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
+
+    switch (event) {
+    case CHR_EVENT_OPENED:
+        if (vhost_user_video_connect(dev) < 0) {
+            qemu_chr_fe_disconnect(&video->conf.chardev);
+            return;
+        }
+        break;
+    case CHR_EVENT_CLOSED:
+        vhost_user_video_disconnect(dev);
+        break;
+    case CHR_EVENT_BREAK:
+    case CHR_EVENT_MUX_IN:
+    case CHR_EVENT_MUX_OUT:
+        /* Ignore */
+        break;
+    }
+}
+
+static void do_vhost_user_cleanup(VirtIODevice *vdev, VHostUserVIDEO *video)
+{
+    vhost_user_cleanup(&video->vhost_user);
+    virtio_delete_queue(video->command_vq);
+    virtio_delete_queue(video->event_vq);
+    virtio_cleanup(vdev);
+    g_free(video->vhost_dev.vqs);
+    video->vhost_dev.vqs = NULL;
+}
+
+
+static void vhost_user_video_device_realize(DeviceState *dev, Error **errp)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(dev);
+    int ret;
+
+    if (!vhost_user_init(&video->vhost_user, &video->conf.chardev, errp)) {
+        return;
+    }
+
+    /* TODO Implement VIDEO_ENC, currently only support VIDEO_DEC */
+    virtio_init(vdev, "vhost-user-video", VIRTIO_ID_VIDEO_DEC,
+                sizeof(struct virtio_video_config));
+
+    /* one command queue and one event queue */
+    video->vhost_dev.nvqs = 2;
+    video->vhost_dev.vqs = g_new0(struct vhost_virtqueue,
+                                  video->vhost_dev.nvqs);
+
+    ret = vhost_dev_init(&video->vhost_dev, &video->vhost_user,
+                         VHOST_BACKEND_TYPE_USER, 0, errp);
+    if (ret < 0) {
+        error_report("vhost_dev_init failed: %s", strerror(-ret));
+        goto vhost_dev_init_failed;
+    }
+
+    /* One command queue, for sending commands */
+    video->command_vq = virtio_add_queue(vdev, 128,
+                                         vhost_user_video_handle_output);
+
+    if (!video->command_vq) {
+        error_setg_errno(errp, -1, "virtio_add_queue() failed");
+        goto cmd_q_fail;
+    }
+
+    /* One event queue, for sending events */
+    video->event_vq = virtio_add_queue(vdev, 128,
+                                       vhost_user_video_handle_output);
+
+    if (!video->command_vq) {
+        error_setg_errno(errp, -1, "virtio_add_queue() failed");
+        goto event_q_fail;
+    }
+
+    /*
+     * At this point the next event we will get is a connection from
+     * the daemon on the control socket.
+     */
+
+    qemu_chr_fe_set_handlers(&video->conf.chardev,  NULL,
+                             NULL, vhost_user_video_event,
+                             NULL, (void *)dev, NULL, true);
+
+    return;
+
+event_q_fail:
+    virtio_delete_queue(video->event_vq);
+cmd_q_fail:
+    vhost_user_cleanup(&video->vhost_user);
+vhost_dev_init_failed:
+    virtio_cleanup(vdev);
+
+}
+
+static void vhost_user_video_device_unrealize(DeviceState *dev)
+{
+    VirtIODevice *vdev = VIRTIO_DEVICE(dev);
+    VHostUserVIDEO *video = VHOST_USER_VIDEO(dev);
+
+    /* This will stop vhost backend if appropriate. */
+    vhost_user_video_set_status(vdev, 0);
+
+    do_vhost_user_cleanup(vdev, video);
+}
+
+static const VMStateDescription vhost_user_video_vmstate = {
+    .name = "vhost-user-video",
+    .unmigratable = 1,
+};
+
+static Property vhost_user_video_properties[] = {
+    DEFINE_PROP_CHR("chardev", VHostUserVIDEO, conf.chardev),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vhost_user_video_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioDeviceClass *vdc = VIRTIO_DEVICE_CLASS(klass);
+
+    device_class_set_props(dc, vhost_user_video_properties);
+    dc->vmsd = &vhost_user_video_vmstate;
+    set_bit(DEVICE_CATEGORY_MISC, dc->categories);
+    vdc->realize = vhost_user_video_device_realize;
+    vdc->unrealize = vhost_user_video_device_unrealize;
+    vdc->get_features = vhost_user_video_get_features;
+    vdc->get_config = vhost_user_video_get_config;
+    vdc->set_status = vhost_user_video_set_status;
+    vdc->guest_notifier_mask = vhost_user_video_guest_notifier_mask;
+    vdc->guest_notifier_pending = vhost_user_video_guest_notifier_pending;
+}
+
+static const TypeInfo vhost_user_video_info = {
+    .name = TYPE_VHOST_USER_VIDEO,
+    .parent = TYPE_VIRTIO_DEVICE,
+    .instance_size = sizeof(VHostUserVIDEO),
+    .class_init = vhost_user_video_class_init,
+};
+
+static void vhost_user_video_register_types(void)
+{
+    type_register_static(&vhost_user_video_info);
+}
+
+type_init(vhost_user_video_register_types)
diff --git a/include/hw/virtio/vhost-user-video.h b/include/hw/virtio/vhost-user-video.h
new file mode 100644
index 0000000000..09a4374ca6
--- /dev/null
+++ b/include/hw/virtio/vhost-user-video.h
@@ -0,0 +1,41 @@
+/*
+ * vhost-user-video virtio device
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#ifndef _VHOST_USER_VIDEO_H_
+#define _VHOST_USER_VIDEO_H_
+
+#include "standard-headers/linux/virtio_video.h"
+#include "hw/virtio/virtio.h"
+#include "hw/virtio/vhost.h"
+#include "hw/virtio/vhost-user.h"
+#include "chardev/char-fe.h"
+
+#define TYPE_VHOST_USER_VIDEO "vhost-user-video-device"
+#define VHOST_USER_VIDEO(obj) \
+        OBJECT_CHECK(VHostUserVIDEO, (obj), TYPE_VHOST_USER_VIDEO)
+
+typedef struct {
+    CharBackend chardev;
+    struct virtio_video_config config;
+} VHostUserVIDEOConf;
+
+typedef struct {
+    /*< private >*/
+    VirtIODevice parent;
+    VHostUserVIDEOConf conf;
+    struct vhost_virtqueue *vhost_vq;
+    struct vhost_dev vhost_dev;
+    VhostUserState vhost_user;
+    VirtQueue *command_vq;
+    VirtQueue *event_vq;
+    bool connected;
+    /*< public >*/
+} VHostUserVIDEO;
+
+
+#endif /* _VHOST_USER_VIDEO_H_ */
-- 
2.25.1

Re: [PATCH 3/8] vhost-user-video: boiler plate code for vhost-user-video device

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  hw/display/Kconfig                   |   5 +
>  hw/display/meson.build               |   3 +
>  hw/display/vhost-user-video.c        | 386 +++++++++++++++++++++++++++
>  include/hw/virtio/vhost-user-video.h |  41 +++
>  4 files changed, 435 insertions(+)
>  create mode 100644 hw/display/vhost-user-video.c
>  create mode 100644 include/hw/virtio/vhost-user-video.h
>
> diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> index a2306b67d8..186163b015 100644
> --- a/hw/display/Kconfig
> +++ b/hw/display/Kconfig
> @@ -118,6 +118,11 @@ config VHOST_USER_VGA
>      default y
>      depends on VIRTIO_VGA && VHOST_USER_GPU
>  
> +config VHOST_USER_VIDEO
> +    bool
> +    default y
> +    depends on VIRTIO && VHOST_USER
> +
>  config DPCD
>      bool
>      select AUX
> diff --git a/hw/display/meson.build b/hw/display/meson.build
> index 861c43ff98..48284528cf 100644
> --- a/hw/display/meson.build
> +++ b/hw/display/meson.build
> @@ -37,6 +37,9 @@ softmmu_ss.add(when: 'CONFIG_MACFB', if_true: files('macfb.c'))
>  softmmu_ss.add(when: 'CONFIG_NEXTCUBE', if_true: files('next-fb.c'))
>  
>  specific_ss.add(when: 'CONFIG_VGA', if_true: files('vga.c'))
> +specific_ss.add(when: 'CONFIG_VHOST_USER_VIDEO', if_true: files('vhost-user-video.c'))
> +specific_ss.add(when: ['CONFIG_VHOST_USER_VIDEO', 'CONFIG_VIRTIO_PCI' ],
> +  if_true: files('vhost-user-video-pci.c'))
>  
>  if config_all_devices.has_key('CONFIG_QXL')
>    qxl_ss = ss.source_set()
> diff --git a/hw/display/vhost-user-video.c b/hw/display/vhost-user-video.c
> new file mode 100644
> index 0000000000..506e350365
> --- /dev/null
> +++ b/hw/display/vhost-user-video.c
> @@ -0,0 +1,386 @@
> +/*
> + * Vhost-user VIDEO virtio device
> + *
> + * This is the boilerplate for instantiating a vhost-user device
> + * implementing a virtio-video device.
> + *
> + * The virtio video decoder and encoder devices are virtual devices that
> + * support encoding and decoding respectively.
> + *
> + * The actual back-end for this driver is the vhost-user-video daemon.
> + * The code here just connects up the device in QEMU and allows it to
> + * be instantiated.
> + *
> + * Copyright (c) 2021 Linaro Ltd
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "hw/qdev-properties.h"
> +#include "hw/virtio/virtio-bus.h"
> +#include "hw/virtio/vhost-user-video.h"
> +#include "qemu/error-report.h"
> +
> +/* currently there is no VIDEO enc/dec defined in Linux virtio_ids.h */
> +#define VIRTIO_ID_VIDEO_ENC         30
> +#define VIRTIO_ID_VIDEO_DEC         31
> +#define MAX_CAPS_LEN 4096
> +
> +static void vhost_user_video_get_config(VirtIODevice *vdev, uint8_t *config)
> +{
> +    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
> +    struct virtio_video_config *vconfig = (struct virtio_video_config *)config;
> +    int ret;
> +    Error *local_err = NULL;
> +
> +    memset(config, 0, sizeof(struct virtio_video_config));
> +
> +    ret = vhost_dev_get_config(&video->vhost_dev, config,
> +                               sizeof(struct virtio_video_config), &local_err);
> +    if (ret) {
> +        error_report("vhost-user-video: get device config space failed");
> +
> +        /*TODO vhost_dev_get_config() fails so for now lets just set
>  it here */

Is this a lifetime issue? 

Otherwise:

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

Re: [PATCH 3/8] vhost-user-video: boiler plate code for vhost-user-video device

[Peter Griffin]

Hi Alex,

Thanks for reviewing.

On Tue, 11 Jan 2022, Alex Bennée wrote:
> 
> Peter Griffin <peter.griffin@linaro.org> writes:
> 
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  hw/display/Kconfig                   |   5 +
> >  hw/display/meson.build               |   3 +
> >  hw/display/vhost-user-video.c        | 386 +++++++++++++++++++++++++++
> >  include/hw/virtio/vhost-user-video.h |  41 +++
> >  4 files changed, 435 insertions(+)
> >  create mode 100644 hw/display/vhost-user-video.c
> >  create mode 100644 include/hw/virtio/vhost-user-video.h
> >
> > diff --git a/hw/display/Kconfig b/hw/display/Kconfig
> > index a2306b67d8..186163b015 100644
> > --- a/hw/display/Kconfig
> > +++ b/hw/display/Kconfig
> > @@ -118,6 +118,11 @@ config VHOST_USER_VGA
> >      default y
> >      depends on VIRTIO_VGA && VHOST_USER_GPU
> >  
> > +config VHOST_USER_VIDEO
> > +    bool
> > +    default y
> > +    depends on VIRTIO && VHOST_USER
> > +
> >  config DPCD
> >      bool
> >      select AUX
> > diff --git a/hw/display/meson.build b/hw/display/meson.build
> > index 861c43ff98..48284528cf 100644
> > --- a/hw/display/meson.build
> > +++ b/hw/display/meson.build
> > @@ -37,6 +37,9 @@ softmmu_ss.add(when: 'CONFIG_MACFB', if_true: files('macfb.c'))
> >  softmmu_ss.add(when: 'CONFIG_NEXTCUBE', if_true: files('next-fb.c'))
> >  
> >  specific_ss.add(when: 'CONFIG_VGA', if_true: files('vga.c'))
> > +specific_ss.add(when: 'CONFIG_VHOST_USER_VIDEO', if_true: files('vhost-user-video.c'))
> > +specific_ss.add(when: ['CONFIG_VHOST_USER_VIDEO', 'CONFIG_VIRTIO_PCI' ],
> > +  if_true: files('vhost-user-video-pci.c'))
> >  
> >  if config_all_devices.has_key('CONFIG_QXL')
> >    qxl_ss = ss.source_set()
> > diff --git a/hw/display/vhost-user-video.c b/hw/display/vhost-user-video.c
> > new file mode 100644
> > index 0000000000..506e350365
> > --- /dev/null
> > +++ b/hw/display/vhost-user-video.c
> > @@ -0,0 +1,386 @@
> > +/*
> > + * Vhost-user VIDEO virtio device
> > + *
> > + * This is the boilerplate for instantiating a vhost-user device
> > + * implementing a virtio-video device.
> > + *
> > + * The virtio video decoder and encoder devices are virtual devices that
> > + * support encoding and decoding respectively.
> > + *
> > + * The actual back-end for this driver is the vhost-user-video daemon.
> > + * The code here just connects up the device in QEMU and allows it to
> > + * be instantiated.
> > + *
> > + * Copyright (c) 2021 Linaro Ltd
> > + *
> > + * SPDX-License-Identifier: GPL-2.0-or-later
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +#include "qapi/error.h"
> > +#include "hw/qdev-properties.h"
> > +#include "hw/virtio/virtio-bus.h"
> > +#include "hw/virtio/vhost-user-video.h"
> > +#include "qemu/error-report.h"
> > +
> > +/* currently there is no VIDEO enc/dec defined in Linux virtio_ids.h */
> > +#define VIRTIO_ID_VIDEO_ENC         30
> > +#define VIRTIO_ID_VIDEO_DEC         31
> > +#define MAX_CAPS_LEN 4096
> > +
> > +static void vhost_user_video_get_config(VirtIODevice *vdev, uint8_t *config)
> > +{
> > +    VHostUserVIDEO *video = VHOST_USER_VIDEO(vdev);
> > +    struct virtio_video_config *vconfig = (struct virtio_video_config *)config;
> > +    int ret;
> > +    Error *local_err = NULL;
> > +
> > +    memset(config, 0, sizeof(struct virtio_video_config));
> > +
> > +    ret = vhost_dev_get_config(&video->vhost_dev, config,
> > +                               sizeof(struct virtio_video_config), &local_err);
> > +    if (ret) {
> > +        error_report("vhost-user-video: get device config space failed");
> > +
> > +        /*TODO vhost_dev_get_config() fails so for now lets just set
> >  it here */
> 
> Is this a lifetime issue?

I believe so. I spent some time trying to get it working, but then added this
workaround so I could progress on the virtio-video implementation.

I will take another look and see if I can get thie config coming from the daemon
as intended.

Looking at other examples, hw/display/vhost-user-gpu.c get_config() hook seems to call
vhost_dev_config successfully like I'm doing here.

But hw/block/vhost-user-blk.c and hw/virtio/vhost-user-vsock.c seem to have
a slightly different mechanism of using VhostDevConfigOps
.vhost_dev_config_notifier() which I don't recall having looked into that previously.

> 
> Otherwise:
> 
> Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

kind regards,

Peter.

[PATCH 4/8] vhost-user-video: add meson subdir build logic

[Peter Griffin]

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 tools/meson.build | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/tools/meson.build b/tools/meson.build
index 3e5a0abfa2..3314b5efc5 100644
--- a/tools/meson.build
+++ b/tools/meson.build
@@ -24,3 +24,12 @@ endif
 if have_virtiofsd
   subdir('virtiofsd')
 endif
+
+have_virtiovideo = (have_system and
+    have_tools and
+    'CONFIG_LINUX' in config_host)
+
+if have_virtiovideo
+  subdir('vhost-user-video')
+endif
+
-- 
2.25.1

Re: [PATCH 4/8] vhost-user-video: add meson subdir build logic

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  tools/meson.build | 9 +++++++++
>  1 file changed, 9 insertions(+)
>
> diff --git a/tools/meson.build b/tools/meson.build
> index 3e5a0abfa2..3314b5efc5 100644
> --- a/tools/meson.build
> +++ b/tools/meson.build
> @@ -24,3 +24,12 @@ endif
>  if have_virtiofsd
>    subdir('virtiofsd')
>  endif
> +
> +have_virtiovideo = (have_system and
> +    have_tools and
> +    'CONFIG_LINUX' in config_host)

Following the convention of cece116c939d219070b250338439c2d16f94e3da

  have_virtiovideo = (targetos == 'linux' and
      have_tools and
      'CONFIG_VHOST_USER' in config_host)

> +
> +if have_virtiovideo
> +  subdir('vhost-user-video')
> +endif
> +


-- 
Alex Bennée

Re: [PATCH 4/8] vhost-user-video: add meson subdir build logic

[Peter Griffin]

Hi Alex,

Thanks for reviewing.

On Tue, 11 Jan 2022, Alex Bennée wrote:

> 
> Peter Griffin <peter.griffin@linaro.org> writes:
> 
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  tools/meson.build | 9 +++++++++
> >  1 file changed, 9 insertions(+)
> >
> > diff --git a/tools/meson.build b/tools/meson.build
> > index 3e5a0abfa2..3314b5efc5 100644
> > --- a/tools/meson.build
> > +++ b/tools/meson.build
> > @@ -24,3 +24,12 @@ endif
> >  if have_virtiofsd
> >    subdir('virtiofsd')
> >  endif
> > +
> > +have_virtiovideo = (have_system and
> > +    have_tools and
> > +    'CONFIG_LINUX' in config_host)
> 
> Following the convention of cece116c939d219070b250338439c2d16f94e3da
> 
>   have_virtiovideo = (targetos == 'linux' and
>       have_tools and
>       'CONFIG_VHOST_USER' in config_host)
>

Will fix in v2

regards,

Peter.

> > +
> > +if have_virtiovideo
> > +  subdir('vhost-user-video')
> > +endif
> > +
> 
> 
> -- 
> Alex Bennée

[PATCH 5/8] standard-headers: Add virtio_video.h

[Peter Griffin]

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
 1 file changed, 483 insertions(+)
 create mode 100644 include/standard-headers/linux/virtio_video.h

diff --git a/include/standard-headers/linux/virtio_video.h b/include/standard-headers/linux/virtio_video.h
new file mode 100644
index 0000000000..16b5f642a9
--- /dev/null
+++ b/include/standard-headers/linux/virtio_video.h
@@ -0,0 +1,483 @@
+/* SPDX-License-Identifier: BSD-3-Clause */
+/*
+ * Virtio Video Device
+ *
+ * This header is BSD licensed so anyone can use the definitions
+ * to implement compatible drivers/servers:
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ * 3. Neither the name of IBM nor the names of its contributors
+ *    may be used to endorse or promote products derived from this software
+ *    without specific prior written permission.
+ * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+ * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+ * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
+ * FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL IBM OR
+ * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+ * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+ * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
+ * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
+ * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+ * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
+ * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ *
+ * Copyright (C) 2019 OpenSynergy GmbH.
+ */
+
+#ifndef _UAPI_LINUX_VIRTIO_VIDEO_H
+#define _UAPI_LINUX_VIRTIO_VIDEO_H
+
+#include <linux/types.h>
+#include <linux/virtio_config.h>
+
+/*
+ * Feature bits
+ */
+
+/* Guest pages can be used for video buffers. */
+#define VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES 0
+/*
+ * The host can process buffers even if they are non-contiguous memory such as
+ * scatter-gather lists.
+ */
+#define VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG 1
+/* Objects exported by another virtio device can be used for video buffers */
+#define VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT 2
+
+/*
+ * Image formats
+ */
+
+enum virtio_video_format {
+	/* Raw formats */
+	VIRTIO_VIDEO_FORMAT_RAW_MIN = 1,
+	VIRTIO_VIDEO_FORMAT_ARGB8888 = VIRTIO_VIDEO_FORMAT_RAW_MIN,
+	VIRTIO_VIDEO_FORMAT_BGRA8888,
+	VIRTIO_VIDEO_FORMAT_NV12, /* 12  Y/CbCr 4:2:0  */
+	VIRTIO_VIDEO_FORMAT_YUV420, /* 12  YUV 4:2:0     */
+	VIRTIO_VIDEO_FORMAT_YVU420, /* 12  YVU 4:2:0     */
+	VIRTIO_VIDEO_FORMAT_RAW_MAX = VIRTIO_VIDEO_FORMAT_YVU420,
+
+	/* Coded formats */
+	VIRTIO_VIDEO_FORMAT_CODED_MIN = 0x1000,
+	VIRTIO_VIDEO_FORMAT_MPEG2 =
+		VIRTIO_VIDEO_FORMAT_CODED_MIN, /* MPEG-2 Part 2 */
+	VIRTIO_VIDEO_FORMAT_MPEG4, /* MPEG-4 Part 2 */
+	VIRTIO_VIDEO_FORMAT_H264, /* H.264 */
+	VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
+	VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
+	VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
+	VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
+};
+
+enum virtio_video_profile {
+	/* H.264 */
+	VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
+	VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
+	VIRTIO_VIDEO_PROFILE_H264_MAIN,
+	VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
+	VIRTIO_VIDEO_PROFILE_H264_HIGH,
+	VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
+	VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
+	VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
+	VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
+	VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
+	VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
+	VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+	VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+
+	/* HEVC */
+	VIRTIO_VIDEO_PROFILE_HEVC_MIN = 0x200,
+	VIRTIO_VIDEO_PROFILE_HEVC_MAIN = VIRTIO_VIDEO_PROFILE_HEVC_MIN,
+	VIRTIO_VIDEO_PROFILE_HEVC_MAIN10,
+	VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
+	VIRTIO_VIDEO_PROFILE_HEVC_MAX =
+		VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
+
+	/* VP8 */
+	VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x300,
+	VIRTIO_VIDEO_PROFILE_VP8_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP8_MIN,
+	VIRTIO_VIDEO_PROFILE_VP8_PROFILE1,
+	VIRTIO_VIDEO_PROFILE_VP8_PROFILE2,
+	VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
+	VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
+
+	/* VP9 */
+	VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x400,
+	VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
+	VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
+	VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
+	VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+	VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
+};
+
+enum virtio_video_level {
+	/* H.264 */
+	VIRTIO_VIDEO_LEVEL_H264_MIN = 0x100,
+	VIRTIO_VIDEO_LEVEL_H264_1_0 = VIRTIO_VIDEO_LEVEL_H264_MIN,
+	VIRTIO_VIDEO_LEVEL_H264_1_1,
+	VIRTIO_VIDEO_LEVEL_H264_1_2,
+	VIRTIO_VIDEO_LEVEL_H264_1_3,
+	VIRTIO_VIDEO_LEVEL_H264_2_0,
+	VIRTIO_VIDEO_LEVEL_H264_2_1,
+	VIRTIO_VIDEO_LEVEL_H264_2_2,
+	VIRTIO_VIDEO_LEVEL_H264_3_0,
+	VIRTIO_VIDEO_LEVEL_H264_3_1,
+	VIRTIO_VIDEO_LEVEL_H264_3_2,
+	VIRTIO_VIDEO_LEVEL_H264_4_0,
+	VIRTIO_VIDEO_LEVEL_H264_4_1,
+	VIRTIO_VIDEO_LEVEL_H264_4_2,
+	VIRTIO_VIDEO_LEVEL_H264_5_0,
+	VIRTIO_VIDEO_LEVEL_H264_5_1,
+	VIRTIO_VIDEO_LEVEL_H264_MAX = VIRTIO_VIDEO_LEVEL_H264_5_1,
+};
+
+/*
+ * Config
+ */
+
+struct virtio_video_config {
+	__le32 version;
+	__le32 max_caps_length;
+	__le32 max_resp_length;
+};
+
+/*
+ * Commands
+ */
+
+enum virtio_video_cmd_type {
+	/* Command */
+	VIRTIO_VIDEO_CMD_QUERY_CAPABILITY = 0x0100,
+	VIRTIO_VIDEO_CMD_STREAM_CREATE,
+	VIRTIO_VIDEO_CMD_STREAM_DESTROY,
+	VIRTIO_VIDEO_CMD_STREAM_DRAIN,
+	VIRTIO_VIDEO_CMD_RESOURCE_CREATE,
+	VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
+	VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL,
+	VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
+	VIRTIO_VIDEO_CMD_GET_PARAMS,
+	VIRTIO_VIDEO_CMD_SET_PARAMS,
+	VIRTIO_VIDEO_CMD_QUERY_CONTROL,
+	VIRTIO_VIDEO_CMD_GET_CONTROL,
+	VIRTIO_VIDEO_CMD_SET_CONTROL,
+
+	/* Response */
+	VIRTIO_VIDEO_RESP_OK_NODATA = 0x0200,
+	VIRTIO_VIDEO_RESP_OK_QUERY_CAPABILITY,
+	VIRTIO_VIDEO_RESP_OK_RESOURCE_QUEUE,
+	VIRTIO_VIDEO_RESP_OK_GET_PARAMS,
+	VIRTIO_VIDEO_RESP_OK_QUERY_CONTROL,
+	VIRTIO_VIDEO_RESP_OK_GET_CONTROL,
+
+	VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION = 0x0300,
+	VIRTIO_VIDEO_RESP_ERR_OUT_OF_MEMORY,
+	VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID,
+	VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID,
+	VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER,
+	VIRTIO_VIDEO_RESP_ERR_UNSUPPORTED_CONTROL,
+};
+
+struct virtio_video_cmd_hdr {
+	__le32 type; /* One of enum virtio_video_cmd_type */
+	__le32 stream_id;
+};
+
+/* VIRTIO_VIDEO_CMD_QUERY_CAPABILITY */
+enum virtio_video_queue_type {
+	VIRTIO_VIDEO_QUEUE_TYPE_INPUT = 0x100,
+	VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT,
+};
+
+struct virtio_video_query_capability {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__u8 padding[4];
+};
+
+enum virtio_video_planes_layout_flag {
+	VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1 << 0,
+	VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE = 1 << 1,
+};
+
+struct virtio_video_format_range {
+	__le32 min;
+	__le32 max;
+	__le32 step;
+	__u8 padding[4];
+};
+
+struct virtio_video_format_frame {
+	struct virtio_video_format_range width;
+	struct virtio_video_format_range height;
+	__le32 num_rates;
+	__u8 padding[4];
+	/* Followed by struct virtio_video_format_range frame_rates[] */
+};
+
+struct virtio_video_format_desc {
+	__le64 mask;
+	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+	__le32 planes_layout; /* Bitmask with VIRTIO_VIDEO_PLANES_LAYOUT_* */
+	__le32 plane_align;
+	__le32 num_frames;
+	/* Followed by struct virtio_video_format_frame frames[] */
+};
+
+struct virtio_video_query_capability_resp {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 num_descs;
+	__u8 padding[4];
+	/* Followed by struct virtio_video_format_desc descs[] */
+};
+
+/* VIRTIO_VIDEO_CMD_STREAM_CREATE */
+enum virtio_video_mem_type {
+	VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
+	VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT,
+};
+
+struct virtio_video_stream_create {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+	__le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
+	__le32 coded_format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+	__u8 padding[4];
+	__u8 tag[64];
+};
+
+/* VIRTIO_VIDEO_CMD_STREAM_DESTROY */
+struct virtio_video_stream_destroy {
+	struct virtio_video_cmd_hdr hdr;
+};
+
+/* VIRTIO_VIDEO_CMD_STREAM_DRAIN */
+struct virtio_video_stream_drain {
+	struct virtio_video_cmd_hdr hdr;
+};
+
+/* VIRTIO_VIDEO_CMD_RESOURCE_CREATE */
+struct virtio_video_mem_entry {
+	__le64 addr;
+	__le32 length;
+	__u8 padding[4];
+};
+
+struct virtio_video_object_entry {
+	__u8 uuid[16];
+};
+
+#define VIRTIO_VIDEO_MAX_PLANES 8
+
+struct virtio_video_resource_create {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__le32 resource_id;
+	__le32 planes_layout;
+	__le32 num_planes;
+	__le32 plane_offsets[VIRTIO_VIDEO_MAX_PLANES];
+	__le32 num_entries[VIRTIO_VIDEO_MAX_PLANES];
+	/**
+	 * Followed by either
+	 * - struct virtio_video_mem_entry entries[]
+	 *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
+	 * - struct virtio_video_object_entry entries[]
+	 *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
+	 */
+};
+
+/* VIRTIO_VIDEO_CMD_RESOURCE_QUEUE */
+struct virtio_video_resource_queue {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__le32 resource_id;
+	__le64 timestamp;
+	__le32 num_data_sizes;
+	__le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES];
+	__u8 padding[4];
+};
+
+enum virtio_video_buffer_flag {
+	VIRTIO_VIDEO_BUFFER_FLAG_ERR = 0x0001,
+	VIRTIO_VIDEO_BUFFER_FLAG_EOS = 0x0002,
+
+	/* Encoder only */
+	VIRTIO_VIDEO_BUFFER_FLAG_IFRAME = 0x0004,
+	VIRTIO_VIDEO_BUFFER_FLAG_PFRAME = 0x0008,
+	VIRTIO_VIDEO_BUFFER_FLAG_BFRAME = 0x0010,
+};
+
+struct virtio_video_resource_queue_resp {
+	struct virtio_video_cmd_hdr hdr;
+	__le64 timestamp;
+	__le32 flags; /* One of VIRTIO_VIDEO_BUFFER_FLAG_* flags */
+	__le32 size; /* Encoded size */
+};
+
+/* VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL */
+struct virtio_video_resource_destroy_all {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__u8 padding[4];
+};
+
+/* VIRTIO_VIDEO_CMD_QUEUE_CLEAR */
+struct virtio_video_queue_clear {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__u8 padding[4];
+};
+
+/* VIRTIO_VIDEO_CMD_GET_PARAMS */
+struct virtio_video_plane_format {
+	__le32 plane_size;
+	__le32 stride;
+};
+
+struct virtio_video_crop {
+	__le32 left;
+	__le32 top;
+	__le32 width;
+	__le32 height;
+};
+
+struct virtio_video_params {
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
+	__le32 frame_width;
+	__le32 frame_height;
+	__le32 min_buffers;
+	__le32 max_buffers;
+	struct virtio_video_crop crop;
+	__le32 frame_rate;
+	__le32 num_planes;
+	struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
+};
+
+struct virtio_video_get_params {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
+	__u8 padding[4];
+};
+
+struct virtio_video_get_params_resp {
+	struct virtio_video_cmd_hdr hdr;
+	struct virtio_video_params params;
+};
+
+/* VIRTIO_VIDEO_CMD_SET_PARAMS */
+struct virtio_video_set_params {
+	struct virtio_video_cmd_hdr hdr;
+	struct virtio_video_params params;
+};
+
+/* VIRTIO_VIDEO_CMD_QUERY_CONTROL */
+enum virtio_video_control_type {
+	VIRTIO_VIDEO_CONTROL_BITRATE = 1,
+	VIRTIO_VIDEO_CONTROL_PROFILE,
+	VIRTIO_VIDEO_CONTROL_LEVEL,
+	VIRTIO_VIDEO_CONTROL_FORCE_KEYFRAME,
+};
+
+struct virtio_video_query_control_profile {
+	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
+	__u8 padding[4];
+};
+
+struct virtio_video_query_control_level {
+	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
+	__u8 padding[4];
+};
+
+struct virtio_video_query_control {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+	__u8 padding[4];
+	/*
+	 * Followed by a value of struct virtio_video_query_control_*
+	 * in accordance with the value of control.
+	 */
+};
+
+struct virtio_video_query_control_resp_profile {
+	__le32 num;
+	__u8 padding[4];
+	/* Followed by an array le32 profiles[] */
+};
+
+struct virtio_video_query_control_resp_level {
+	__le32 num;
+	__u8 padding[4];
+	/* Followed by an array le32 level[] */
+};
+
+struct virtio_video_query_control_resp {
+	struct virtio_video_cmd_hdr hdr;
+	/* Followed by one of struct virtio_video_query_control_resp_* */
+};
+
+/* VIRTIO_VIDEO_CMD_GET_CONTROL */
+struct virtio_video_get_control {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+	__u8 padding[4];
+};
+
+struct virtio_video_control_val_bitrate {
+	__le32 bitrate;
+	__u8 padding[4];
+};
+
+struct virtio_video_control_val_profile {
+	__le32 profile;
+	__u8 padding[4];
+};
+
+struct virtio_video_control_val_level {
+	__le32 level;
+	__u8 padding[4];
+};
+
+struct virtio_video_get_control_resp {
+	struct virtio_video_cmd_hdr hdr;
+	/* Followed by one of struct virtio_video_control_val_* */
+};
+
+/* VIRTIO_VIDEO_CMD_SET_CONTROL */
+struct virtio_video_set_control {
+	struct virtio_video_cmd_hdr hdr;
+	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
+	__u8 padding[4];
+	/* Followed by one of struct virtio_video_control_val_* */
+};
+
+struct virtio_video_set_control_resp {
+	struct virtio_video_cmd_hdr hdr;
+};
+
+/*
+ * Events
+ */
+
+enum virtio_video_event_type {
+	/* For all devices */
+	VIRTIO_VIDEO_EVENT_ERROR = 0x0100,
+
+	/* For decoder only */
+	VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED = 0x0200,
+};
+
+struct virtio_video_event {
+	__le32 event_type; /* One of VIRTIO_VIDEO_EVENT_* types */
+	__le32 stream_id;
+};
+
+#endif /* _UAPI_LINUX_VIRTIO_VIDEO_H */
-- 
2.25.1

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Michael S. Tsirkin]

On Thu, Dec 09, 2021 at 02:55:58PM +0000, Peter Griffin wrote:
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
>  1 file changed, 483 insertions(+)
>  create mode 100644 include/standard-headers/linux/virtio_video.h

We generally inherit these files from Linux.
Was the driver posted for inclusion in Linux?

-- 
MST

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Peter Griffin]

Hi Michael,

On Fri, 10 Dec 2021, Michael S. Tsirkin wrote:

> On Thu, Dec 09, 2021 at 02:55:58PM +0000, Peter Griffin wrote:
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
> >  1 file changed, 483 insertions(+)
> >  create mode 100644 include/standard-headers/linux/virtio_video.h
> 
> We generally inherit these files from Linux.
> Was the driver posted for inclusion in Linux?

Thanks for reviewing. Yes the Linux virtio-video frontend driver was posted
sometime back on the linux-media ML [1].

One piece of pushback then was not supporting vicodec/FWHT and also no Qemu
support [2] which is what this series is trying to address.

The virtio-video spec however is now at rfc v5. So my rough plan was now I have
something working with Qemu and vicodec I can move both the frontend driver
and the vhost-user-video to latest v5 spec.

I'm a bit unclear what the process is to get the virtio-video spec merged though.
I think I read somewhere they expect a matching frontend driver implementation?

Thanks,

Peter.

[1] https://patchwork.kernel.org/project/linux-media/cover/20200218202753.652093-1-dmitry.sepp@opensynergy.com/
[2] https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg02204.html

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Michael S. Tsirkin]

On Fri, Dec 10, 2021 at 01:09:46PM +0000, Peter Griffin wrote:
> Hi Michael,
> 
> On Fri, 10 Dec 2021, Michael S. Tsirkin wrote:
> 
> > On Thu, Dec 09, 2021 at 02:55:58PM +0000, Peter Griffin wrote:
> > > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > > ---
> > >  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
> > >  1 file changed, 483 insertions(+)
> > >  create mode 100644 include/standard-headers/linux/virtio_video.h
> > 
> > We generally inherit these files from Linux.
> > Was the driver posted for inclusion in Linux?
> 
> Thanks for reviewing. Yes the Linux virtio-video frontend driver was posted
> sometime back on the linux-media ML [1].
> 
> One piece of pushback then was not supporting vicodec/FWHT and also no Qemu
> support [2] which is what this series is trying to address.
> 
> The virtio-video spec however is now at rfc v5. So my rough plan was now I have
> something working with Qemu and vicodec I can move both the frontend driver
> and the vhost-user-video to latest v5 spec.
> 
> I'm a bit unclear what the process is to get the virtio-video spec merged though.
> I think I read somewhere they expect a matching frontend driver implementation?
> 
> Thanks,
> 
> Peter.

No, just that it all looks on track to be merged, and got some acks from
Linux driver maintainers. This is because we don't have experts in
all fields on the TC, so input from linux maintainers is useful.

To get a change into spec the TC needs to vote
on it. The simplest way to do that is described here.

https://github.com/oasis-tcs/virtio-spec/blob/master/README.md#use-of-github-issues


> [1] https://patchwork.kernel.org/project/linux-media/cover/20200218202753.652093-1-dmitry.sepp@opensynergy.com/
> [2] https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg02204.html

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Michael S. Tsirkin]

On Thu, Dec 09, 2021 at 02:55:58PM +0000, Peter Griffin wrote:
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
>  1 file changed, 483 insertions(+)
>  create mode 100644 include/standard-headers/linux/virtio_video.h
> 
> diff --git a/include/standard-headers/linux/virtio_video.h b/include/standard-headers/linux/virtio_video.h
> new file mode 100644
> index 0000000000..16b5f642a9
> --- /dev/null
> +++ b/include/standard-headers/linux/virtio_video.h
> @@ -0,0 +1,483 @@
> +/* SPDX-License-Identifier: BSD-3-Clause */
> +/*
> + * Virtio Video Device
> + *
> + * This header is BSD licensed so anyone can use the definitions
> + * to implement compatible drivers/servers:
> + *
> + * Redistribution and use in source and binary forms, with or without
> + * modification, are permitted provided that the following conditions
> + * are met:
> + * 1. Redistributions of source code must retain the above copyright
> + *    notice, this list of conditions and the following disclaimer.
> + * 2. Redistributions in binary form must reproduce the above copyright
> + *    notice, this list of conditions and the following disclaimer in the
> + *    documentation and/or other materials provided with the distribution.
> + * 3. Neither the name of IBM nor the names of its contributors
> + *    may be used to endorse or promote products derived from this software
> + *    without specific prior written permission.
> + * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
> + * ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
> + * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
> + * FOR A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL IBM OR
> + * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
> + * SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
> + * LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
> + * USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
> + * ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
> + * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
> + * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
> + * SUCH DAMAGE.
> + *
> + * Copyright (C) 2019 OpenSynergy GmbH.
> + */
> +
> +#ifndef _UAPI_LINUX_VIRTIO_VIDEO_H
> +#define _UAPI_LINUX_VIRTIO_VIDEO_H
> +
> +#include <linux/types.h>
> +#include <linux/virtio_config.h>
> +
> +/*
> + * Feature bits
> + */
> +
> +/* Guest pages can be used for video buffers. */
> +#define VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES 0
> +/*
> + * The host can process buffers even if they are non-contiguous memory such as
> + * scatter-gather lists.
> + */
> +#define VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG 1
> +/* Objects exported by another virtio device can be used for video buffers */
> +#define VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT 2
> +
> +/*
> + * Image formats
> + */
> +
> +enum virtio_video_format {
> +	/* Raw formats */
> +	VIRTIO_VIDEO_FORMAT_RAW_MIN = 1,
> +	VIRTIO_VIDEO_FORMAT_ARGB8888 = VIRTIO_VIDEO_FORMAT_RAW_MIN,
> +	VIRTIO_VIDEO_FORMAT_BGRA8888,
> +	VIRTIO_VIDEO_FORMAT_NV12, /* 12  Y/CbCr 4:2:0  */
> +	VIRTIO_VIDEO_FORMAT_YUV420, /* 12  YUV 4:2:0     */
> +	VIRTIO_VIDEO_FORMAT_YVU420, /* 12  YVU 4:2:0     */
> +	VIRTIO_VIDEO_FORMAT_RAW_MAX = VIRTIO_VIDEO_FORMAT_YVU420,
> +
> +	/* Coded formats */
> +	VIRTIO_VIDEO_FORMAT_CODED_MIN = 0x1000,
> +	VIRTIO_VIDEO_FORMAT_MPEG2 =
> +		VIRTIO_VIDEO_FORMAT_CODED_MIN, /* MPEG-2 Part 2 */
> +	VIRTIO_VIDEO_FORMAT_MPEG4, /* MPEG-4 Part 2 */
> +	VIRTIO_VIDEO_FORMAT_H264, /* H.264 */
> +	VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
> +	VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
> +	VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */


Explicitly giving values for all constants is a good idea.
It's not a thing where the numbers don't matter as long as
they are all different. Same for the rest of enums.
I also wonder about the value of min/max here.


> +	VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
> +};
> +
> +enum virtio_video_profile {
> +	/* H.264 */
> +	VIRTIO_VIDEO_PROFILE_H264_MIN = 0x100,
> +	VIRTIO_VIDEO_PROFILE_H264_BASELINE = VIRTIO_VIDEO_PROFILE_H264_MIN,
> +	VIRTIO_VIDEO_PROFILE_H264_MAIN,
> +	VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
> +	VIRTIO_VIDEO_PROFILE_H264_HIGH,
> +	VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
> +	VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
> +	VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
> +	VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
> +	VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
> +	VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
> +	VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +	VIRTIO_VIDEO_PROFILE_H264_MAX = VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
> +
> +	/* HEVC */
> +	VIRTIO_VIDEO_PROFILE_HEVC_MIN = 0x200,
> +	VIRTIO_VIDEO_PROFILE_HEVC_MAIN = VIRTIO_VIDEO_PROFILE_HEVC_MIN,
> +	VIRTIO_VIDEO_PROFILE_HEVC_MAIN10,
> +	VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
> +	VIRTIO_VIDEO_PROFILE_HEVC_MAX =
> +		VIRTIO_VIDEO_PROFILE_HEVC_MAIN_STILL_PICTURE,
> +
> +	/* VP8 */
> +	VIRTIO_VIDEO_PROFILE_VP8_MIN = 0x300,
> +	VIRTIO_VIDEO_PROFILE_VP8_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP8_MIN,
> +	VIRTIO_VIDEO_PROFILE_VP8_PROFILE1,
> +	VIRTIO_VIDEO_PROFILE_VP8_PROFILE2,
> +	VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
> +	VIRTIO_VIDEO_PROFILE_VP8_MAX = VIRTIO_VIDEO_PROFILE_VP8_PROFILE3,
> +
> +	/* VP9 */
> +	VIRTIO_VIDEO_PROFILE_VP9_MIN = 0x400,
> +	VIRTIO_VIDEO_PROFILE_VP9_PROFILE0 = VIRTIO_VIDEO_PROFILE_VP9_MIN,
> +	VIRTIO_VIDEO_PROFILE_VP9_PROFILE1,
> +	VIRTIO_VIDEO_PROFILE_VP9_PROFILE2,
> +	VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +	VIRTIO_VIDEO_PROFILE_VP9_MAX = VIRTIO_VIDEO_PROFILE_VP9_PROFILE3,
> +};
> +
> +enum virtio_video_level {
> +	/* H.264 */
> +	VIRTIO_VIDEO_LEVEL_H264_MIN = 0x100,
> +	VIRTIO_VIDEO_LEVEL_H264_1_0 = VIRTIO_VIDEO_LEVEL_H264_MIN,
> +	VIRTIO_VIDEO_LEVEL_H264_1_1,
> +	VIRTIO_VIDEO_LEVEL_H264_1_2,
> +	VIRTIO_VIDEO_LEVEL_H264_1_3,
> +	VIRTIO_VIDEO_LEVEL_H264_2_0,
> +	VIRTIO_VIDEO_LEVEL_H264_2_1,
> +	VIRTIO_VIDEO_LEVEL_H264_2_2,
> +	VIRTIO_VIDEO_LEVEL_H264_3_0,
> +	VIRTIO_VIDEO_LEVEL_H264_3_1,
> +	VIRTIO_VIDEO_LEVEL_H264_3_2,
> +	VIRTIO_VIDEO_LEVEL_H264_4_0,
> +	VIRTIO_VIDEO_LEVEL_H264_4_1,
> +	VIRTIO_VIDEO_LEVEL_H264_4_2,
> +	VIRTIO_VIDEO_LEVEL_H264_5_0,
> +	VIRTIO_VIDEO_LEVEL_H264_5_1,
> +	VIRTIO_VIDEO_LEVEL_H264_MAX = VIRTIO_VIDEO_LEVEL_H264_5_1,
> +};
> +
> +/*
> + * Config
> + */
> +
> +struct virtio_video_config {
> +	__le32 version;
> +	__le32 max_caps_length;
> +	__le32 max_resp_length;
> +};
> +
> +/*
> + * Commands
> + */
> +
> +enum virtio_video_cmd_type {
> +	/* Command */
> +	VIRTIO_VIDEO_CMD_QUERY_CAPABILITY = 0x0100,
> +	VIRTIO_VIDEO_CMD_STREAM_CREATE,
> +	VIRTIO_VIDEO_CMD_STREAM_DESTROY,
> +	VIRTIO_VIDEO_CMD_STREAM_DRAIN,
> +	VIRTIO_VIDEO_CMD_RESOURCE_CREATE,
> +	VIRTIO_VIDEO_CMD_RESOURCE_QUEUE,
> +	VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL,
> +	VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
> +	VIRTIO_VIDEO_CMD_GET_PARAMS,
> +	VIRTIO_VIDEO_CMD_SET_PARAMS,
> +	VIRTIO_VIDEO_CMD_QUERY_CONTROL,
> +	VIRTIO_VIDEO_CMD_GET_CONTROL,
> +	VIRTIO_VIDEO_CMD_SET_CONTROL,
> +
> +	/* Response */
> +	VIRTIO_VIDEO_RESP_OK_NODATA = 0x0200,
> +	VIRTIO_VIDEO_RESP_OK_QUERY_CAPABILITY,
> +	VIRTIO_VIDEO_RESP_OK_RESOURCE_QUEUE,
> +	VIRTIO_VIDEO_RESP_OK_GET_PARAMS,
> +	VIRTIO_VIDEO_RESP_OK_QUERY_CONTROL,
> +	VIRTIO_VIDEO_RESP_OK_GET_CONTROL,
> +
> +	VIRTIO_VIDEO_RESP_ERR_INVALID_OPERATION = 0x0300,
> +	VIRTIO_VIDEO_RESP_ERR_OUT_OF_MEMORY,
> +	VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID,
> +	VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID,
> +	VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER,
> +	VIRTIO_VIDEO_RESP_ERR_UNSUPPORTED_CONTROL,
> +};
> +
> +struct virtio_video_cmd_hdr {
> +	__le32 type; /* One of enum virtio_video_cmd_type */
> +	__le32 stream_id;
> +};
> +
> +/* VIRTIO_VIDEO_CMD_QUERY_CAPABILITY */
> +enum virtio_video_queue_type {
> +	VIRTIO_VIDEO_QUEUE_TYPE_INPUT = 0x100,
> +	VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT,
> +};
> +
> +struct virtio_video_query_capability {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__u8 padding[4];
> +};
> +
> +enum virtio_video_planes_layout_flag {
> +	VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER = 1 << 0,
> +	VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE = 1 << 1,
> +};
> +
> +struct virtio_video_format_range {
> +	__le32 min;
> +	__le32 max;
> +	__le32 step;
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_format_frame {
> +	struct virtio_video_format_range width;
> +	struct virtio_video_format_range height;
> +	__le32 num_rates;
> +	__u8 padding[4];
> +	/* Followed by struct virtio_video_format_range frame_rates[] */


You can actually do struct virtio_video_format_range frame_rates[]
with the same effect.

> +};
> +
> +struct virtio_video_format_desc {
> +	__le64 mask;
> +	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> +	__le32 planes_layout; /* Bitmask with VIRTIO_VIDEO_PLANES_LAYOUT_* */
> +	__le32 plane_align;
> +	__le32 num_frames;
> +	/* Followed by struct virtio_video_format_frame frames[] */
> +};
> +
> +struct virtio_video_query_capability_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 num_descs;
> +	__u8 padding[4];
> +	/* Followed by struct virtio_video_format_desc descs[] */
> +};
> +
> +/* VIRTIO_VIDEO_CMD_STREAM_CREATE */
> +enum virtio_video_mem_type {
> +	VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES,
> +	VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT,
> +};
> +
> +struct virtio_video_stream_create {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 in_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +	__le32 out_mem_type; /* One of VIRTIO_VIDEO_MEM_TYPE_* types */
> +	__le32 coded_format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> +	__u8 padding[4];
> +	__u8 tag[64];
> +};
> +
> +/* VIRTIO_VIDEO_CMD_STREAM_DESTROY */
> +struct virtio_video_stream_destroy {
> +	struct virtio_video_cmd_hdr hdr;
> +};
> +
> +/* VIRTIO_VIDEO_CMD_STREAM_DRAIN */
> +struct virtio_video_stream_drain {
> +	struct virtio_video_cmd_hdr hdr;
> +};
> +
> +/* VIRTIO_VIDEO_CMD_RESOURCE_CREATE */
> +struct virtio_video_mem_entry {
> +	__le64 addr;
> +	__le32 length;
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_object_entry {
> +	__u8 uuid[16];
> +};
> +
> +#define VIRTIO_VIDEO_MAX_PLANES 8
> +
> +struct virtio_video_resource_create {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__le32 resource_id;
> +	__le32 planes_layout;
> +	__le32 num_planes;
> +	__le32 plane_offsets[VIRTIO_VIDEO_MAX_PLANES];
> +	__le32 num_entries[VIRTIO_VIDEO_MAX_PLANES];
> +	/**
> +	 * Followed by either
> +	 * - struct virtio_video_mem_entry entries[]
> +	 *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
> +	 * - struct virtio_video_object_entry entries[]
> +	 *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
> +	 */
> +};
> +
> +/* VIRTIO_VIDEO_CMD_RESOURCE_QUEUE */
> +struct virtio_video_resource_queue {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__le32 resource_id;
> +	__le64 timestamp;
> +	__le32 num_data_sizes;
> +	__le32 data_sizes[VIRTIO_VIDEO_MAX_PLANES];
> +	__u8 padding[4];
> +};
> +
> +enum virtio_video_buffer_flag {
> +	VIRTIO_VIDEO_BUFFER_FLAG_ERR = 0x0001,
> +	VIRTIO_VIDEO_BUFFER_FLAG_EOS = 0x0002,
> +
> +	/* Encoder only */
> +	VIRTIO_VIDEO_BUFFER_FLAG_IFRAME = 0x0004,
> +	VIRTIO_VIDEO_BUFFER_FLAG_PFRAME = 0x0008,
> +	VIRTIO_VIDEO_BUFFER_FLAG_BFRAME = 0x0010,
> +};
> +
> +struct virtio_video_resource_queue_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le64 timestamp;
> +	__le32 flags; /* One of VIRTIO_VIDEO_BUFFER_FLAG_* flags */
> +	__le32 size; /* Encoded size */
> +};
> +
> +/* VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL */
> +struct virtio_video_resource_destroy_all {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__u8 padding[4];
> +};
> +
> +/* VIRTIO_VIDEO_CMD_QUEUE_CLEAR */
> +struct virtio_video_queue_clear {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__u8 padding[4];
> +};
> +
> +/* VIRTIO_VIDEO_CMD_GET_PARAMS */
> +struct virtio_video_plane_format {
> +	__le32 plane_size;
> +	__le32 stride;
> +};
> +
> +struct virtio_video_crop {
> +	__le32 left;
> +	__le32 top;
> +	__le32 width;
> +	__le32 height;
> +};
> +
> +struct virtio_video_params {
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* types */
> +	__le32 frame_width;
> +	__le32 frame_height;
> +	__le32 min_buffers;
> +	__le32 max_buffers;
> +	struct virtio_video_crop crop;
> +	__le32 frame_rate;
> +	__le32 num_planes;
> +	struct virtio_video_plane_format plane_formats[VIRTIO_VIDEO_MAX_PLANES];
> +};
> +
> +struct virtio_video_get_params {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* types */
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_get_params_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +	struct virtio_video_params params;
> +};
> +
> +/* VIRTIO_VIDEO_CMD_SET_PARAMS */
> +struct virtio_video_set_params {
> +	struct virtio_video_cmd_hdr hdr;
> +	struct virtio_video_params params;
> +};
> +
> +/* VIRTIO_VIDEO_CMD_QUERY_CONTROL */
> +enum virtio_video_control_type {
> +	VIRTIO_VIDEO_CONTROL_BITRATE = 1,
> +	VIRTIO_VIDEO_CONTROL_PROFILE,
> +	VIRTIO_VIDEO_CONTROL_LEVEL,
> +	VIRTIO_VIDEO_CONTROL_FORCE_KEYFRAME,
> +};
> +
> +struct virtio_video_query_control_profile {
> +	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_query_control_level {
> +	__le32 format; /* One of VIRTIO_VIDEO_FORMAT_* */
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_query_control {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> +	__u8 padding[4];
> +	/*
> +	 * Followed by a value of struct virtio_video_query_control_*
> +	 * in accordance with the value of control.
> +	 */
> +};
> +
> +struct virtio_video_query_control_resp_profile {
> +	__le32 num;
> +	__u8 padding[4];
> +	/* Followed by an array le32 profiles[] */
> +};
> +
> +struct virtio_video_query_control_resp_level {
> +	__le32 num;
> +	__u8 padding[4];
> +	/* Followed by an array le32 level[] */
> +};
> +
> +struct virtio_video_query_control_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +	/* Followed by one of struct virtio_video_query_control_resp_* */
> +};
> +
> +/* VIRTIO_VIDEO_CMD_GET_CONTROL */
> +struct virtio_video_get_control {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_control_val_bitrate {
> +	__le32 bitrate;
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_control_val_profile {
> +	__le32 profile;
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_control_val_level {
> +	__le32 level;
> +	__u8 padding[4];
> +};
> +
> +struct virtio_video_get_control_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +	/* Followed by one of struct virtio_video_control_val_* */
> +};
> +
> +/* VIRTIO_VIDEO_CMD_SET_CONTROL */
> +struct virtio_video_set_control {
> +	struct virtio_video_cmd_hdr hdr;
> +	__le32 control; /* One of VIRTIO_VIDEO_CONTROL_* types */
> +	__u8 padding[4];
> +	/* Followed by one of struct virtio_video_control_val_* */
> +};
> +
> +struct virtio_video_set_control_resp {
> +	struct virtio_video_cmd_hdr hdr;
> +};
> +
> +/*
> + * Events
> + */
> +
> +enum virtio_video_event_type {
> +	/* For all devices */
> +	VIRTIO_VIDEO_EVENT_ERROR = 0x0100,
> +
> +	/* For decoder only */
> +	VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED = 0x0200,
> +};
> +
> +struct virtio_video_event {
> +	__le32 event_type; /* One of VIRTIO_VIDEO_EVENT_* types */
> +	__le32 stream_id;
> +};
> +
> +#endif /* _UAPI_LINUX_VIRTIO_VIDEO_H */
> -- 
> 2.25.1

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Peter Maydell]

On Thu, 9 Dec 2021 at 15:03, Peter Griffin <peter.griffin@linaro.org> wrote:
>
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
>  1 file changed, 483 insertions(+)
>  create mode 100644 include/standard-headers/linux/virtio_video.h

As MST notes, content in include/standard-headers/ is from
upstream Linux, and it gets into QEMU by:

 (1) getting it into the Linux kernel tree
 (2) changing QEMU's scripts/update-linux-headers.sh to know that
     it wants this new header file
 (3) running the script to create a patch which updates the headers
     to match a new enough mainline Linux kernel that has the file

This is important because files in standard-headers are used
by QEMU on all host OSes, not just Linux. Things like:

> +#include <linux/virtio_config.h>

or

> +       __le32 version;

won't compile on BSD. The update-linux-headers.sh script takes
care of sanitizing this kind of Linuxism when it copies the
header into the QEMU tree.

The process also ensures that we don't accidentally bring in
support for a kernel feature until its userspace ABI is finalized.

Where a new feature is being worked on on both the kernel and the
QEMU side of things, you can post QEMU RFC patchsets which
temporarily have ad-hoc header file updates (eg created from
a kernel tree that has the not-yet-upstream kernel patches), but
these should be clearly marked as RFCs and we don't take the
QEMU patches until the kernel side is upstream and the QEMU
headers can be cleanly generated from a mainline kernel commit.

thanks
-- PMM

Re: [PATCH 5/8] standard-headers: Add virtio_video.h

[Peter Griffin]

Hi Peter,

On Fri, 10 Dec 2021, Peter Maydell wrote:

> On Thu, 9 Dec 2021 at 15:03, Peter Griffin <peter.griffin@linaro.org> wrote:
> >
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  include/standard-headers/linux/virtio_video.h | 483 ++++++++++++++++++
> >  1 file changed, 483 insertions(+)
> >  create mode 100644 include/standard-headers/linux/virtio_video.h
> 
> As MST notes, content in include/standard-headers/ is from
> upstream Linux, and it gets into QEMU by:
> 
>  (1) getting it into the Linux kernel tree
>  (2) changing QEMU's scripts/update-linux-headers.sh to know that
>      it wants this new header file
>  (3) running the script to create a patch which updates the headers
>      to match a new enough mainline Linux kernel that has the file
> 
> This is important because files in standard-headers are used
> by QEMU on all host OSes, not just Linux. Things like:
> 
> > +#include <linux/virtio_config.h>
> 
> or
> 
> > +       __le32 version;
> 
> won't compile on BSD. The update-linux-headers.sh script takes
> care of sanitizing this kind of Linuxism when it copies the
> header into the QEMU tree.
> 
> The process also ensures that we don't accidentally bring in
> support for a kernel feature until its userspace ABI is finalized.
>
> Where a new feature is being worked on on both the kernel and the
> QEMU side of things, you can post QEMU RFC patchsets which
> temporarily have ad-hoc header file updates (eg created from
> a kernel tree that has the not-yet-upstream kernel patches), but
> these should be clearly marked as RFCs and we don't take the
> QEMU patches until the kernel side is upstream and the QEMU
> headers can be cleanly generated from a mainline kernel commit.

Thanks for the review and the explanation :) I meant to call out this
header file inclusion in the cover letter as I was not sure of the
process on that.

This series should have been marked RFC then as the kernel side is
not merged yet. The purpose of this series is to hopefully unblock
getting the kernel side merged though, as lack of Qemu support and
a test setup that supported vicodec were both mentioned last time
in the review process.

Thanks,

Peter.

[PATCH 6/8] virtio_video: Add Fast Walsh-Hadamard Transform format

[Peter Griffin]

Linux vicodec (Virtual Codec) test driver in Linux implements
FWHT. FWHT was designed to be fast and simple and to have
characteristics of other video codecs and therefore face similar
issues [1].

https://en.wikipedia.org/wiki/Fast_Walsh%E2%80%93Hadamard_transform

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 include/standard-headers/linux/virtio_video.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/standard-headers/linux/virtio_video.h b/include/standard-headers/linux/virtio_video.h
index 16b5f642a9..3b517d50c4 100644
--- a/include/standard-headers/linux/virtio_video.h
+++ b/include/standard-headers/linux/virtio_video.h
@@ -75,6 +75,7 @@ enum virtio_video_format {
 	VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
 	VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
 	VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
+	VIRTIO_VIDEO_FORMAT_FWHT, /* FWHT used by vicodec */
 	VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
 };
 
-- 
2.25.1

Re: [PATCH 6/8] virtio_video: Add Fast Walsh-Hadamard Transform format

[Michael S. Tsirkin]

On Thu, Dec 09, 2021 at 02:55:59PM +0000, Peter Griffin wrote:
> Linux vicodec (Virtual Codec) test driver in Linux implements
> FWHT. FWHT was designed to be fast and simple and to have
> characteristics of other video codecs and therefore face similar
> issues [1].
> 
> https://en.wikipedia.org/wiki/Fast_Walsh%E2%80%93Hadamard_transform
> 
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  include/standard-headers/linux/virtio_video.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/standard-headers/linux/virtio_video.h b/include/standard-headers/linux/virtio_video.h
> index 16b5f642a9..3b517d50c4 100644
> --- a/include/standard-headers/linux/virtio_video.h
> +++ b/include/standard-headers/linux/virtio_video.h
> @@ -75,6 +75,7 @@ enum virtio_video_format {
>  	VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
>  	VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
>  	VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
> +	VIRTIO_VIDEO_FORMAT_FWHT, /* FWHT used by vicodec */
>  	VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,

Is last line still correct? Seems fragile ...
>  };
>  
> -- 
> 2.25.1

Re: [PATCH 6/8] virtio_video: Add Fast Walsh-Hadamard Transform format

[Peter Griffin]

Hi Michael,

On Fri, 10 Dec 2021, Michael S. Tsirkin wrote:

> On Thu, Dec 09, 2021 at 02:55:59PM +0000, Peter Griffin wrote:
> > Linux vicodec (Virtual Codec) test driver in Linux implements
> > FWHT. FWHT was designed to be fast and simple and to have
> > characteristics of other video codecs and therefore face similar
> > issues [1].
> > 
> > https://en.wikipedia.org/wiki/Fast_Walsh%E2%80%93Hadamard_transform
> > 
> > Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> > ---
> >  include/standard-headers/linux/virtio_video.h | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/include/standard-headers/linux/virtio_video.h b/include/standard-headers/linux/virtio_video.h
> > index 16b5f642a9..3b517d50c4 100644
> > --- a/include/standard-headers/linux/virtio_video.h
> > +++ b/include/standard-headers/linux/virtio_video.h
> > @@ -75,6 +75,7 @@ enum virtio_video_format {
> >  	VIRTIO_VIDEO_FORMAT_HEVC, /* HEVC aka H.265*/
> >  	VIRTIO_VIDEO_FORMAT_VP8, /* VP8 */
> >  	VIRTIO_VIDEO_FORMAT_VP9, /* VP9 */
> > +	VIRTIO_VIDEO_FORMAT_FWHT, /* FWHT used by vicodec */
> >  	VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_VP9,
> 
> Is last line still correct? Seems fragile ...

Ah good spot! This is a typo, I should have updated it to
VIRTIO_VIDEO_FORMAT_CODED_MAX = VIRTIO_VIDEO_FORMAT_FWHT,

Will fix in the next iteration.

Peter.

[PATCH 7/8] hw/display: add vhost-user-video-pci

[Peter Griffin]

Add boiler plate code for vhost-user-video-pci.

Example
-device vhost-user-video-pci,chardev=video,id=video
-chardev socket,path=video.sock,id=video

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 hw/display/vhost-user-video-pci.c | 82 +++++++++++++++++++++++++++++++
 1 file changed, 82 insertions(+)
 create mode 100644 hw/display/vhost-user-video-pci.c

diff --git a/hw/display/vhost-user-video-pci.c b/hw/display/vhost-user-video-pci.c
new file mode 100644
index 0000000000..ceeaad2742
--- /dev/null
+++ b/hw/display/vhost-user-video-pci.c
@@ -0,0 +1,82 @@
+/*
+ * Vhost-user VIDEO virtio device PCI glue
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include "qemu/osdep.h"
+#include "hw/qdev-properties.h"
+#include "hw/virtio/vhost-user-video.h"
+#include "hw/virtio/virtio-pci.h"
+
+struct VHostUserVIDEOPCI {
+    VirtIOPCIProxy parent_obj;
+    VHostUserVIDEO vdev;
+};
+
+typedef struct VHostUserVIDEOPCI VHostUserVIDEOPCI;
+
+#define TYPE_VHOST_USER_VIDEO_PCI "vhost-user-video-pci-base"
+
+#define VHOST_USER_VIDEO_PCI(obj) \
+        OBJECT_CHECK(VHostUserVIDEOPCI, (obj), TYPE_VHOST_USER_VIDEO_PCI)
+
+static Property vuvideo_pci_properties[] = {
+    DEFINE_PROP_BIT("ioeventfd", VirtIOPCIProxy, flags,
+                    VIRTIO_PCI_FLAG_USE_IOEVENTFD_BIT, true),
+    DEFINE_PROP_UINT32("vectors", VirtIOPCIProxy, nvectors,
+                       DEV_NVECTORS_UNSPECIFIED),
+    DEFINE_PROP_END_OF_LIST(),
+};
+
+static void vuvideo_pci_realize(VirtIOPCIProxy *vpci_dev, Error **errp)
+{
+    VHostUserVIDEOPCI *dev = VHOST_USER_VIDEO_PCI(vpci_dev);
+    DeviceState *vdev = DEVICE(&dev->vdev);
+
+    if (vpci_dev->nvectors == DEV_NVECTORS_UNSPECIFIED) {
+        vpci_dev->nvectors = 1;
+    }
+
+    qdev_set_parent_bus(vdev, BUS(&vpci_dev->bus), errp);
+    object_property_set_bool(OBJECT(vdev), "realized", true, errp);
+}
+
+static void vuvideo_pci_class_init(ObjectClass *klass, void *data)
+{
+    DeviceClass *dc = DEVICE_CLASS(klass);
+    VirtioPCIClass *k = VIRTIO_PCI_CLASS(klass);
+    PCIDeviceClass *pcidev_k = PCI_DEVICE_CLASS(klass);
+    k->realize = vuvideo_pci_realize;
+    set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+    device_class_set_props(dc, vuvideo_pci_properties);
+    pcidev_k->vendor_id = PCI_VENDOR_ID_REDHAT_QUMRANET;
+    pcidev_k->device_id = 0; /* Set by virtio-pci based on virtio id */
+    pcidev_k->revision = 0x00;
+    pcidev_k->class_id = PCI_CLASS_STORAGE_OTHER;
+}
+
+static void vuvideo_pci_instance_init(Object *obj)
+{
+    VHostUserVIDEOPCI *dev = VHOST_USER_VIDEO_PCI(obj);
+
+    virtio_instance_init_common(obj, &dev->vdev, sizeof(dev->vdev),
+                                TYPE_VHOST_USER_VIDEO);
+}
+
+static const VirtioPCIDeviceTypeInfo vuvideo_pci_info = {
+    .base_name             = TYPE_VHOST_USER_VIDEO_PCI,
+    .non_transitional_name = "vhost-user-video-pci",
+    .instance_size = sizeof(VHostUserVIDEOPCI),
+    .instance_init = vuvideo_pci_instance_init,
+    .class_init    = vuvideo_pci_class_init,
+};
+
+static void vuvideo_pci_register(void)
+{
+    virtio_pci_types_register(&vuvideo_pci_info);
+}
+
+type_init(vuvideo_pci_register);
-- 
2.25.1

Re: [PATCH 7/8] hw/display: add vhost-user-video-pci

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> Add boiler plate code for vhost-user-video-pci.
>
> Example
> -device vhost-user-video-pci,chardev=video,id=video
> -chardev socket,path=video.sock,id=video
>
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée

[PATCH 8/8] tools/vhost-user-video: Add initial vhost-user-video vmm

[Peter Griffin]

This vmm translates from virtio-video v3 protocol and writes
to a v4l2 mem2mem stateful decoder/encoder device [1]. v3 was
chosen as that is what the virtio-video Linux frontend driver
implements.

This allows for testing with the v4l2 vicodec test codec [2]
module in the Linux kernel, and is intended to also be used
with Arm SoCs that implement a v4l2 stateful decoder/encoder
drivers.

The advantage of developing & testing with vicodec is that
is allows quick development on a purely virtual setup with
qemu and a host Linux kernel. Also it allows ci systems like
lkft, kernelci to easily test the virtio interface.

Currently conversion from virtio-video to v4l2 stateless m2m
codec driver or VAAPI drivers is consiered out ot scope as
is emulation of a decoder device using a something like ffmpeg.
Although this could be added in the future.

Note some virtio & v4l2 helpers were based off virtio-video
Linux frontend driver and yavta utility, both GPL v2.

Example host commands
 modprobe vicodec
 vhost-user-video --v4l2-device=/dev/video3 -v --socket-path=video.sock

Run Qemu with
 -device vhost-user-video-pci,chardev=video,id=video

Guest decoder
 v4l2-ctl -d0 -x width=640,height=480 -v width=640,height=480,pixelformat=YU12
   --stream-mmap --stream-out-mmap --stream-from jelly_640_480-420P.fwht
   --stream-to out-jelly-640-480.YU12

[1] https://www.kernel.org/doc/html/latest/userspace-api/media/
    v4l/dev-decoder.html

[2] https://lwn.net/Articles/760650/

Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
---
 tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
 tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
 tools/vhost-user-video/meson.build            |   10 +
 tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
 tools/vhost-user-video/v4l2_backend.h         |   99 +
 tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
 tools/vhost-user-video/virtio_video_helpers.h |  166 ++
 tools/vhost-user-video/vuvideo.h              |   43 +
 8 files changed, 4242 insertions(+)
 create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
 create mode 100644 tools/vhost-user-video/main.c
 create mode 100644 tools/vhost-user-video/meson.build
 create mode 100644 tools/vhost-user-video/v4l2_backend.c
 create mode 100644 tools/vhost-user-video/v4l2_backend.h
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
 create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
 create mode 100644 tools/vhost-user-video/vuvideo.h

diff --git a/tools/vhost-user-video/50-qemu-rpmb.json.in b/tools/vhost-user-video/50-qemu-rpmb.json.in
new file mode 100644
index 0000000000..2b033cda56
--- /dev/null
+++ b/tools/vhost-user-video/50-qemu-rpmb.json.in
@@ -0,0 +1,5 @@
+{
+  "description": "QEMU vhost-user-rpmb",
+  "type": "block",
+  "binary": "@libexecdir@/vhost-user-rpmb"
+}
diff --git a/tools/vhost-user-video/main.c b/tools/vhost-user-video/main.c
new file mode 100644
index 0000000000..a944efadb6
--- /dev/null
+++ b/tools/vhost-user-video/main.c
@@ -0,0 +1,1680 @@
+/*
+ * VIRTIO Video Emulation via vhost-user
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#define G_LOG_DOMAIN "vhost-user-video"
+#define G_LOG_USE_STRUCTURED 1
+
+#include <glib.h>
+#include <gio/gio.h>
+#include <gio/gunixsocketaddress.h>
+#include <glib-unix.h>
+#include <glib/gstdio.h>
+#include <stdio.h>
+#include <string.h>
+#include <inttypes.h>
+#include <fcntl.h>
+#include <sys/ioctl.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <sys/mman.h>
+#include <unistd.h>
+#include <endian.h>
+#include <assert.h>
+
+#include "libvhost-user-glib.h"
+#include "libvhost-user.h"
+#include "standard-headers/linux/virtio_video.h"
+
+#include "qemu/compiler.h"
+#include "qemu/iov.h"
+
+#include "vuvideo.h"
+#include "v4l2_backend.h"
+#include "virtio_video_helpers.h"
+
+#ifndef container_of
+#define container_of(ptr, type, member) ({                      \
+        const typeof(((type *) 0)->member) * __mptr = (ptr);     \
+        (type *) ((char *) __mptr - offsetof(type, member)); })
+#endif
+
+static gchar *socket_path;
+static gchar *v4l2_path;
+static gint socket_fd = -1;
+static gboolean print_cap;
+static gboolean verbose;
+static gboolean debug;
+
+static GOptionEntry options[] = {
+    { "socket-path", 0, 0, G_OPTION_ARG_FILENAME, &socket_path,
+      "Location of vhost-user Unix domain socket, "
+      "incompatible with --fd", "PATH" },
+    { "v4l2-device", 0, 0, G_OPTION_ARG_FILENAME, &v4l2_path,
+      "Location of v4l2 device node", "PATH" },
+    { "fd", 0, 0, G_OPTION_ARG_INT, &socket_fd,
+      "Specify the fd of the backend, "
+      "incompatible with --socket-path", "FD" },
+    { "print-capabilities", 0, 0, G_OPTION_ARG_NONE, &print_cap,
+      "Output to stdout the backend capabilities "
+      "in JSON format and exit", NULL},
+    { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose,
+      "Be more verbose in output", NULL},
+    { "debug", 0, 0, G_OPTION_ARG_NONE, &debug,
+      "Include debug output", NULL},
+    { NULL }
+};
+
+enum {
+    VHOST_USER_VIDEO_MAX_QUEUES = 2,
+};
+
+/* taken from util/iov.c */
+size_t video_iov_size(const struct iovec *iov, const unsigned int iov_cnt)
+{
+    size_t len;
+    unsigned int i;
+
+    len = 0;
+    for (i = 0; i < iov_cnt; i++) {
+        len += iov[i].iov_len;
+    }
+    return len;
+}
+
+static size_t video_iov_to_buf(const struct iovec *iov,
+                               const unsigned int iov_cnt,
+                               size_t offset, void *buf, size_t bytes)
+{
+    size_t done;
+    unsigned int i;
+    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
+        if (offset < iov[i].iov_len) {
+            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
+            memcpy(buf + done, iov[i].iov_base + offset, len);
+            done += len;
+            offset = 0;
+        } else {
+            offset -= iov[i].iov_len;
+        }
+    }
+    assert(offset == 0);
+    return done;
+}
+
+static size_t video_iov_from_buf(const struct iovec *iov, unsigned int iov_cnt,
+                                 size_t offset, const void *buf, size_t bytes)
+{
+    size_t done;
+    unsigned int i;
+    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
+        if (offset < iov[i].iov_len) {
+            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
+            memcpy(iov[i].iov_base + offset, buf + done, len);
+            done += len;
+            offset = 0;
+        } else {
+            offset -= iov[i].iov_len;
+        }
+    }
+    assert(offset == 0);
+    return done;
+}
+
+static void video_panic(VuDev *dev, const char *msg)
+{
+    g_critical("%s\n", msg);
+    exit(EXIT_FAILURE);
+}
+
+static uint64_t video_get_features(VuDev *dev)
+{
+    g_info("%s: replying", __func__);
+    return 0;
+}
+
+static void video_set_features(VuDev *dev, uint64_t features)
+{
+    if (features) {
+        g_autoptr(GString) s = g_string_new("Requested un-handled feature");
+        g_string_append_printf(s, " 0x%" PRIx64 "", features);
+        g_info("%s: %s", __func__, s->str);
+    }
+}
+
+/*
+ * The configuration of the device is static and set when we start the
+ * daemon.
+ */
+static int
+video_get_config(VuDev *dev, uint8_t *config, uint32_t len)
+{
+    VuVideo *v = container_of(dev, VuVideo, dev.parent);
+
+    g_return_val_if_fail(len <= sizeof(struct virtio_video_config), -1);
+    v->virtio_config.version = 0;
+    v->virtio_config.max_caps_length = MAX_CAPS_LEN;
+    v->virtio_config.max_resp_length = MAX_CAPS_LEN;
+
+    memcpy(config, &v->virtio_config, len);
+
+    g_debug("%s: config.max_caps_length = %d", __func__
+           , ((struct virtio_video_config *)config)->max_caps_length);
+    g_debug("%s: config.max_resp_length = %d", __func__
+           , ((struct virtio_video_config *)config)->max_resp_length);
+
+    return 0;
+}
+
+static int
+video_set_config(VuDev *dev, const uint8_t *data,
+                 uint32_t offset, uint32_t size,
+                 uint32_t flags)
+{
+    g_debug("%s: ", __func__);
+    /* ignore */
+    return 0;
+}
+
+/*
+ * Handlers for individual control messages
+ */
+
+static void
+handle_set_params_cmd(struct VuVideo *v, struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret = 0;
+    enum v4l2_buf_type buf_type;
+    struct virtio_video_set_params *cmd =
+        (struct virtio_video_set_params *) vio_cmd->cmd_buf;
+    struct stream *s;
+
+    g_debug("%s: type(x%x) stream_id(%d) %s ", __func__,
+            cmd->hdr.type, cmd->hdr.stream_id,
+            vio_queue_name(le32toh(cmd->params.queue_type)));
+    g_debug("%s: format=0x%x frame_width(%d) frame_height(%d)",
+            __func__, le32toh(cmd->params.format),
+            le32toh(cmd->params.frame_width),
+            le32toh(cmd->params.frame_height));
+    g_debug("%s: min_buffers(%d) max_buffers(%d)", __func__,
+            le32toh(cmd->params.min_buffers), le32toh(cmd->params.max_buffers));
+    g_debug("%s: frame_rate(%d) num_planes(%d)", __func__,
+            le32toh(cmd->params.frame_rate), le32toh(cmd->params.num_planes));
+    g_debug("%s: crop top=%d, left=%d, width=%d, height=%d", __func__,
+            le32toh(cmd->params.crop.left), le32toh(cmd->params.crop.top),
+            le32toh(cmd->params.crop.width), le32toh(cmd->params.crop.height));
+
+    s = find_stream(v, cmd->hdr.stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, cmd->hdr.stream_id);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    buf_type = get_v4l2_buf_type(le32toh(cmd->params.queue_type),
+                                 s->has_mplane);
+
+    ret = v4l2_video_set_format(s->fd, buf_type, &cmd->params);
+    if (ret < 0) {
+        g_error("%s: v4l2_video_set_format() failed", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    if (is_capture_queue(buf_type)) {
+        /* decoder supports composing on CAPTURE */
+        struct v4l2_selection sel;
+        memset(&sel, 0, sizeof(struct v4l2_selection));
+
+        sel.r.left = le32toh(cmd->params.crop.left);
+        sel.r.top = le32toh(cmd->params.crop.top);
+        sel.r.width = le32toh(cmd->params.crop.width);
+        sel.r.height = le32toh(cmd->params.crop.height);
+
+        ret = v4l2_video_set_selection(s->fd, buf_type, &sel);
+        if (ret < 0) {
+            g_printerr("%s: v4l2_video_set_selection failed: %s (%d).\n"
+                       , __func__, g_strerror(errno), errno);
+            cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+            goto out_unlock;
+        }
+    }
+
+    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+
+out_unlock:
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+out:
+    return;
+}
+
+static void
+handle_get_params_cmd(struct VuVideo *v, struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret;
+    struct v4l2_format fmt;
+    struct v4l2_selection sel;
+    enum v4l2_buf_type buf_type;
+    struct virtio_video_get_params *cmd =
+        (struct virtio_video_get_params *) vio_cmd->cmd_buf;
+    struct virtio_video_get_params_resp getparams_reply;
+    struct stream *s;
+
+    g_debug("%s: type(0x%x) stream_id(%d) %s", __func__,
+            cmd->hdr.type, cmd->hdr.stream_id,
+            vio_queue_name(le32toh(cmd->queue_type)));
+
+    s = find_stream(v, cmd->hdr.stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found\n"
+                   , __func__, cmd->hdr.stream_id);
+        getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    getparams_reply.hdr.stream_id = cmd->hdr.stream_id;
+    getparams_reply.params.queue_type = cmd->queue_type;
+
+    buf_type = get_v4l2_buf_type(cmd->queue_type, s->has_mplane);
+
+    ret = v4l2_video_get_format(s->fd, buf_type, &fmt);
+    if (ret < 0) {
+        g_printerr("v4l2_video_get_format failed\n");
+        getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    if (is_capture_queue(buf_type)) {
+        ret = v4l2_video_get_selection(s->fd, buf_type, &sel);
+        if (ret < 0) {
+            g_printerr("v4l2_video_get_selection failed\n");
+            getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+            goto out_unlock;
+        }
+    }
+
+    /* convert from v4l2 to virtio */
+    v4l2_to_virtio_video_params(v->v4l2_dev, &fmt, &sel,
+                                &getparams_reply);
+
+    getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_OK_GET_PARAMS;
+
+out_unlock:
+    vio_cmd->finished = true;
+    send_ctrl_response(vio_cmd, (uint8_t *)&getparams_reply,
+                       sizeof(struct virtio_video_get_params_resp));
+    g_mutex_unlock(&s->mutex);
+out:
+    return;
+}
+
+struct stream *find_stream(struct VuVideo *v, uint32_t stream_id)
+{
+    GList *l;
+    struct stream *s;
+
+    for (l = v->streams; l != NULL; l = l->next) {
+        s = (struct stream *)l->data;
+        if (s->stream_id == stream_id) {
+            return s;
+        }
+    }
+
+    return NULL;
+}
+
+int add_resource(struct stream *s, struct resource *r, uint32_t queue_type)
+{
+
+    if (!s || !r) {
+        return -EINVAL;
+    }
+
+    switch (queue_type) {
+    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
+        s->inputq_resources = g_list_append(s->inputq_resources, r);
+        break;
+
+    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
+        s->outputq_resources = g_list_append(s->outputq_resources, r);
+        break;
+    default:
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
+void free_resource_mem(struct resource *r)
+{
+
+    /*
+     * Frees the memory allocated for resource_queue_cmd
+     * not the memory allocated in resource_create
+     */
+
+    if (r->vio_q_cmd) {
+        g_free(r->vio_q_cmd->cmd_buf);
+        r->vio_q_cmd->cmd_buf = NULL;
+        free(r->vio_q_cmd);
+        r->vio_q_cmd = NULL;
+    }
+}
+
+void remove_all_resources(struct stream *s, uint32_t queue_type)
+{
+    GList **resource_list;
+    struct resource *r;
+
+    /* assumes stream mutex is held by caller */
+
+    switch (queue_type) {
+    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
+        resource_list =  &s->inputq_resources;
+        break;
+    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
+        resource_list = &s->outputq_resources;
+        break;
+    default:
+        g_critical("%s: Invalid virtio queue!", __func__);
+        return;
+    }
+
+    g_debug("%s: resource_list has %d elements", __func__
+            , g_list_length(*resource_list));
+
+    GList *l = *resource_list;
+    while (l != NULL) {
+        GList *next = l->next;
+        r = (struct resource *)l->data;
+        if (r) {
+            g_debug("%s: Removing resource_id(%d) resource=%p"
+                    , __func__, r->vio_resource.resource_id, r);
+
+            /*
+             * Assumes that either QUEUE_CLEAR or normal dequeuing
+             * of buffers will have freed resource_queue cmd memory
+             */
+
+            /* free resource memory allocated in resource_create() */
+            g_free(r->iov);
+            g_free(r);
+            *resource_list = g_list_delete_link(*resource_list, l);
+        }
+        l = next;
+   }
+}
+
+struct resource *find_resource(struct stream *s, uint32_t resource_id,
+                                uint32_t queue_type)
+{
+    GList *l;
+    struct resource *r;
+
+    switch (queue_type) {
+    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
+        l = s->inputq_resources;
+        break;
+
+    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
+        l = s->outputq_resources;
+        break;
+    default:
+        g_error("%s: Invalid queue type!", __func__);
+    }
+
+    for (; l != NULL; l = l->next) {
+        r = (struct resource *)l->data;
+        if (r->vio_resource.resource_id == resource_id) {
+            return r;
+        }
+    }
+
+    return NULL;
+}
+
+struct resource *find_resource_by_v4l2index(struct stream *s,
+                                             enum v4l2_buf_type buf_type,
+                                             uint32_t v4l2_index)
+{
+    GList *l = NULL;
+    struct resource *r;
+
+    switch (buf_type) {
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+        l = s->outputq_resources;
+        break;
+
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+        l = s->inputq_resources;
+        break;
+
+    default:
+        g_error("Unsupported buffer type\n");
+    }
+
+    for (; l != NULL; l = l->next) {
+        r = (struct resource *)l->data;
+        if (r->v4l2_index == v4l2_index) {
+            g_debug("%s: found Resource=%p streamid(%d) resourceid(%d) "
+                    "numplanes(%d) planes_layout(0x%x) vio_q_cmd=%p", __func__,
+                    r, r->stream_id, r->vio_resource.resource_id,
+                    r->vio_resource.num_planes, r->vio_resource.planes_layout,
+                    r->vio_q_cmd);
+            return r;
+        }
+    }
+    return NULL;
+}
+
+#define EVENT_WQ_IDX 1
+
+static void *stream_worker_thread(gpointer data)
+{
+    int ret;
+    struct stream *s = data;
+    VuVideo *v = s->video;
+    VugDev *vugdev = &v->dev;
+    VuDev *vudev = &vugdev->parent;
+    VuVirtq *vq = vu_get_queue(vudev, EVENT_WQ_IDX);
+    VuVirtqElement *elem;
+    size_t len;
+
+    struct v4l2_event ev;
+    struct virtio_video_event vio_event;
+
+    /* select vars */
+    fd_set efds, rfds, wfds;
+    bool have_event, have_read, have_write;
+    enum v4l2_buf_type buf_type;
+
+    fcntl(s->fd, F_SETFL, fcntl(s->fd, F_GETFL) | O_NONBLOCK);
+
+    while (true) {
+            int res;
+
+            g_mutex_lock(&s->mutex);
+
+            /* wait for STREAMING or DESTROYING state */
+            while (s->stream_state != STREAM_DESTROYING &&
+                   s->stream_state != STREAM_STREAMING &&
+                   s->stream_state != STREAM_DRAINING)
+                g_cond_wait(&s->stream_cond, &s->mutex);
+
+            if (s->stream_state == STREAM_DESTROYING) {
+                g_debug("stream worker thread exiting!");
+                s->stream_state = STREAM_DESTROYED;
+                g_cond_signal(&s->stream_cond);
+                g_mutex_unlock(&s->mutex);
+                g_thread_exit(0);
+            }
+
+            g_mutex_unlock(&s->mutex);
+
+            FD_ZERO(&efds);
+            FD_SET(s->fd, &efds);
+            FD_ZERO(&rfds);
+            FD_SET(s->fd, &rfds);
+            FD_ZERO(&wfds);
+            FD_SET(s->fd, &wfds);
+
+            struct timeval tv = { 0 , 500000 };
+            res = select(s->fd + 1, &rfds, &wfds, &efds, &tv);
+            if (res < 0) {
+                g_printerr("%s:%d - select() failed errno(%s)\n", __func__,
+                           __LINE__, g_strerror(errno));
+                break;
+            }
+
+            if (res == 0) {
+                g_debug("%s:%d - select() timeout", __func__, __LINE__);
+                continue;
+            }
+
+            have_event = FD_ISSET(s->fd, &efds);
+            have_read = FD_ISSET(s->fd, &rfds);
+            have_write = FD_ISSET(s->fd, &wfds);
+            /* read is capture queue, write is output queue */
+
+            g_debug("%s:%d have_event=%d, have_write=%d, have_read=%d\n"
+                    , __func__, __LINE__, FD_ISSET(s->fd, &efds)
+                    , FD_ISSET(s->fd, &wfds), FD_ISSET(s->fd, &rfds));
+
+            g_mutex_lock(&s->mutex);
+
+            if (have_event) {
+                g_debug("%s: have_event!", __func__);
+                res = ioctl(s->fd, VIDIOC_DQEVENT, &ev);
+                if (res < 0) {
+                    g_printerr("%s:%d - VIDIOC_DQEVENT failed: errno(%s)\n",
+                               __func__, __LINE__, g_strerror(errno));
+                    break;
+                }
+                v4l2_to_virtio_event(&ev, &vio_event);
+
+                /* get event workqueue */
+                elem = vu_queue_pop(vudev, vq, sizeof(struct VuVirtqElement));
+                if (!elem) {
+                    g_debug("%s:%d\n", __func__, __LINE__);
+                    break;
+                }
+
+                len = video_iov_from_buf(elem->in_sg,
+                                         elem->in_num, 0, (void *) &vio_event,
+                                         sizeof(struct virtio_video_event));
+
+                vu_queue_push(vudev, vq, elem, len);
+                vu_queue_notify(vudev, vq);
+            }
+
+            if (have_read && s->capture_streaming == true) {
+                /* TODO assumes decoder */
+                buf_type = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
+                    : V4L2_BUF_TYPE_VIDEO_CAPTURE;
+
+                ret = v4l2_dequeue_buffer(s->fd, buf_type, s);
+                if (ret < 0) {
+                    g_info("%s: v4l2_dequeue_buffer() failed CAPTURE ret(%d)"
+                           , __func__, ret);
+
+                    if (errno == EPIPE) {
+                        /* dequeued last buf, so stop streaming */
+                        ioctl_streamoff(s, buf_type);
+                    }
+                 }
+            }
+
+            if (have_write  && s->output_streaming == true) {
+                buf_type = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+                    : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+                ret = v4l2_dequeue_buffer(s->fd, buf_type, s);
+                if (ret < 0) {
+                    g_info("%s: v4l2_dequeue_buffer() failed OUTPUT ret(%d)"
+                           , __func__, ret);
+                }
+            }
+
+            g_mutex_unlock(&s->mutex);
+    }
+
+    return NULL;
+}
+
+void handle_queue_clear_cmd(struct VuVideo *v,
+                       struct vu_video_ctrl_command *vio_cmd)
+{
+    struct virtio_video_queue_clear *cmd =
+        (struct virtio_video_queue_clear *)vio_cmd->cmd_buf;
+    int ret = 0;
+    struct stream *s;
+    uint32_t stream_id = le32toh(cmd->hdr.stream_id);
+    enum virtio_video_queue_type queue_type = le32toh(cmd->queue_type);
+
+    g_debug("%s: stream_id(%d) %s\n", __func__, stream_id,
+            vio_queue_name(queue_type));
+
+    if (!v || !cmd) {
+        return;
+    }
+
+    s = find_stream(v, stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    enum v4l2_buf_type buf_type =
+        get_v4l2_buf_type(le32toh(cmd->queue_type), s->has_mplane);
+
+    /*
+     * QUEUE_CLEAR behaviour from virtio-video spec
+     * Return already queued buffers back from the input or the output queue
+     * of the device. The device SHOULD return all of the buffers from the
+     * respective queue as soon as possible without pushing the buffers through
+     * the processing pipeline.
+     *
+     * From v4l2 PoV we issue a VIDIOC_STREAMOFF on the queue which will abort
+     * or finish any DMA in progress, unlocks any user pointer buffers locked
+     * in physical memory, and it removes all buffers from the incoming and
+     * outgoing queues.
+     */
+
+    /* issue streamoff  */
+    ret = ioctl_streamoff(s, buf_type);
+    if (ret < 0) {
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    /* iterate the queues resources list - and send a reply to each one */
+
+    /*
+     * If the processing was stopped due to VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
+     * the device MUST respond with VIRTIO_VIDEO_RESP_OK_NODATA as a response
+     * type and VIRTIO_- VIDEO_BUFFER_FLAG_ERR in flags.
+     */
+
+    g_list_foreach(get_resource_list(s, queue_type),
+                   (GFunc)send_qclear_res_reply, s);
+
+    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+
+out_unlock:
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+out:
+    return;
+}
+
+GList *get_resource_list(struct stream *s, uint32_t queue_type)
+{
+    switch (queue_type) {
+    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
+        return s->inputq_resources;
+        break;
+
+    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
+        return s->outputq_resources;
+        break;
+    default:
+        g_critical("%s: Unknown queue type!", __func__);
+        return NULL;
+    }
+}
+
+void send_ctrl_response(struct vu_video_ctrl_command *vio_cmd,
+                       uint8_t *resp, size_t resp_len)
+{
+    size_t len;
+
+    virtio_video_ctrl_hdr_htole((struct virtio_video_cmd_hdr *)resp);
+
+    /* send virtio_video_resource_queue_resp */
+    len = video_iov_from_buf(vio_cmd->elem.in_sg,
+                             vio_cmd->elem.in_num, 0, resp, resp_len);
+
+    if (len != resp_len) {
+        g_critical("%s: response size incorrect %zu vs %zu",
+                   __func__, len, resp_len);
+    }
+
+    vu_queue_push(vio_cmd->dev, vio_cmd->vq, &vio_cmd->elem, len);
+    vu_queue_notify(vio_cmd->dev, vio_cmd->vq);
+
+    if (vio_cmd->finished) {
+        g_free(vio_cmd->cmd_buf);
+        free(vio_cmd);
+    }
+}
+
+void send_ctrl_response_nodata(struct vu_video_ctrl_command *vio_cmd)
+{
+    send_ctrl_response(vio_cmd, vio_cmd->cmd_buf,
+                       sizeof(struct virtio_video_cmd_hdr));
+}
+
+void send_qclear_res_reply(gpointer data, gpointer user_data)
+{
+    struct resource *r = data;
+    struct vu_video_ctrl_command *vio_cmd = r->vio_q_cmd;
+    struct virtio_video_queue_clear *cmd =
+        (struct virtio_video_queue_clear *) vio_cmd->cmd_buf;
+    struct virtio_video_resource_queue_resp resp;
+
+    /*
+     * only need to send replies for buffers that are
+     * inflight
+     */
+
+    if (r->queued) {
+
+        resp.hdr.stream_id = cmd->hdr.stream_id;
+        resp.hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+        resp.flags = htole32(VIRTIO_VIDEO_BUFFER_FLAG_ERR);
+        resp.timestamp = htole64(r->vio_res_q.timestamp);
+
+        g_debug("%s: stream_id=%d type=0x%x flags=0x%x resource_id=%d t=%llx"
+                , __func__, resp.hdr.stream_id, resp.hdr.type, resp.flags,
+                r->vio_resource.resource_id, resp.timestamp);
+
+        vio_cmd->finished = true;
+        send_ctrl_response(vio_cmd, (uint8_t *) &resp,
+                           sizeof(struct virtio_video_resource_queue_resp));
+    }
+    return;
+}
+
+static int
+handle_resource_create_cmd(struct VuVideo *v,
+                           struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret = 0, i;
+    uint32_t total_entries = 0;
+    uint32_t stream_id ;
+    struct virtio_video_resource_create *cmd =
+        (struct virtio_video_resource_create *)vio_cmd->cmd_buf;
+    struct virtio_video_mem_entry *mem;
+    struct resource *res;
+    struct virtio_video_resource_create *r;
+    struct stream *s;
+    enum virtio_video_mem_type mem_type;
+
+    stream_id = cmd->hdr.stream_id;
+
+    s = find_stream(v, stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    if (le32toh(cmd->resource_id) == 0) {
+        g_critical("%s: resource id 0 is not allowed", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    /* check resource id doesn't already exist */
+    res = find_resource(s, le32toh(cmd->resource_id), le32toh(cmd->queue_type));
+    if (res) {
+        g_critical("%s: resource_id:%d already exists"
+                   , __func__, le32toh(cmd->resource_id));
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
+        goto out_unlock;
+    } else {
+        res = g_new0(struct resource, 1);
+        res->vio_resource.resource_id = le32toh(cmd->resource_id);
+        res->vio_resource.queue_type = le32toh(cmd->queue_type);
+        res->vio_resource.planes_layout = le32toh(cmd->planes_layout);
+
+        res->vio_resource.num_planes = le32toh(cmd->num_planes);
+        r = &res->vio_resource;
+
+        ret = add_resource(s, res, le32toh(cmd->queue_type));
+        if (ret) {
+            g_critical("%s: resource_add id:%d failed"
+                       , __func__, le32toh(cmd->resource_id));
+            cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
+            goto out_unlock;
+        }
+
+        g_debug("%s: resource=%p streamid(%d) resourceid(%d) numplanes(%d)"
+                "planes_layout(0x%x) %s",
+                __func__, res, res->stream_id, r->resource_id, r->num_planes,
+                r->planes_layout, vio_queue_name(r->queue_type));
+    }
+
+    if (r->planes_layout & VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE) {
+        g_debug("%s: streamid(%d) resourceid(%d) planes_layout(0x%x)"
+                , __func__, res->stream_id, r->resource_id, r->planes_layout);
+
+        for (i = 0; i < r->num_planes; i++) {
+            total_entries += le32toh(cmd->num_entries[i]);
+            g_debug("%s: streamid(%d) resourceid(%d) num_entries[%d]=%d"
+                    , __func__, res->stream_id, r->resource_id,
+                    i, le32toh(cmd->num_entries[i]));
+        }
+    } else {
+        total_entries = 1;
+    }
+
+    /*
+     * virtio_video_resource_create is followed by either
+     * - struct virtio_video_mem_entry entries[]
+     *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
+     * - struct virtio_video_object_entry entries[]
+     *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
+     */
+
+    if (r->queue_type == VIRTIO_VIDEO_QUEUE_TYPE_INPUT) {
+        mem_type = s->vio_stream.in_mem_type;
+    } else {
+        mem_type = s->vio_stream.out_mem_type;
+    }
+    /*
+     * Followed by either
+     * - struct virtio_video_mem_entry entries[]
+     *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
+     * - struct virtio_video_object_entry entries[]
+     *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
+     */
+
+    if (mem_type == VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES) {
+        mem = (void *)cmd + sizeof(struct virtio_video_resource_create);
+
+        res->iov = g_malloc0(sizeof(struct iovec) * total_entries);
+        for (i = 0; i < total_entries; i++) {
+            uint64_t len = le32toh(mem[i].length);
+            g_debug("%s: mem[%d] addr=0x%lx", __func__
+                    , i, le64toh(mem[i].addr));
+
+            res->iov[i].iov_len = le32toh(mem[i].length);
+            res->iov[i].iov_base =
+                vu_gpa_to_va(&v->dev.parent, &len, le64toh(mem[i].addr));
+            g_debug("%s: [%d] iov_len = 0x%lx", __func__
+                    , i, res->iov[i].iov_len);
+            g_debug("%s: [%d] iov_base = 0x%p", __func__
+                    , i, res->iov[i].iov_base);
+        }
+        res->iov_count = total_entries;
+
+    } else if (mem_type == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT) {
+        g_critical("%s: VIRTIO_OBJECT not implemented!", __func__);
+        /* TODO implement VIRTIO_OBJECT support */
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    /* check underlying driver supports GUEST_PAGES */
+    enum v4l2_buf_type buf_type =
+        get_v4l2_buf_type(r->queue_type, s->has_mplane);
+
+    ret = v4l2_resource_create(s, buf_type, mem_type, res);
+    if (ret < 0) {
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+
+    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+
+out_unlock:
+    /* send response */
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+out:
+    return ret;
+}
+
+static int
+handle_resource_queue_cmd(struct VuVideo *v,
+                          struct vu_video_ctrl_command *vio_cmd)
+{
+    struct virtio_video_resource_queue *cmd =
+        (struct virtio_video_resource_queue *)vio_cmd->cmd_buf;
+    struct resource *res;
+    struct stream *s;
+    uint32_t stream_id;
+    int ret = 0;
+
+    g_debug("%s: type(0x%x) %s resource_id(%d)", __func__,
+            cmd->hdr.type, vio_queue_name(le32toh(cmd->queue_type)),
+            le32toh(cmd->resource_id));
+    g_debug("%s: num_data_sizes = %d", __func__, le32toh(cmd->num_data_sizes));
+    g_debug("%s: data_sizes[0] = %d", __func__, le32toh(cmd->data_sizes[0]));
+
+    stream_id = cmd->hdr.stream_id;
+
+    s = find_stream(v, stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    if (cmd->resource_id == 0) {
+        g_critical("%s: resource id 0 is not allowed", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
+        goto out_unlock;
+    }
+
+    /* get resource object */
+    res = find_resource(s, le32toh(cmd->resource_id), le32toh(cmd->queue_type));
+    if (!res) {
+        g_critical("%s: resource_id:%d does not exist!"
+                   , __func__, le32toh(cmd->resource_id));
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
+        goto out_unlock;
+    }
+
+    res->vio_res_q.timestamp = le64toh(cmd->timestamp);
+    res->vio_res_q.num_data_sizes = le32toh(cmd->num_data_sizes);
+    res->vio_res_q.queue_type = le32toh(cmd->queue_type);
+    res->vio_q_cmd = vio_cmd;
+
+    g_debug("%s: res=%p res->vio_q_cmd=0x%p", __func__, res, res->vio_q_cmd);
+
+    enum v4l2_buf_type buf_type = get_v4l2_buf_type(
+        cmd->queue_type, s->has_mplane);
+
+
+    ret = v4l2_queue_buffer(s->fd, buf_type, cmd, res, s, v->v4l2_dev);
+    if (ret < 0) {
+        g_critical("%s: v4l2_queue_buffer failed", __func__);
+        /* virtio error set by v4l2_queue_buffer */
+        goto out_unlock;
+    }
+
+    /*
+     * let the stream worker thread do the dequeueing of output and
+     * capture queue buffers and send the resource_queue replies
+     */
+
+    g_mutex_unlock(&s->mutex);
+    return ret;
+
+out_unlock:
+    /* send response */
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+out:
+    return ret;
+}
+
+
+static void
+handle_resource_destroy_all_cmd(struct VuVideo *v,
+                                struct vu_video_ctrl_command *vio_cmd)
+{
+    struct virtio_video_resource_destroy_all *cmd =
+        (struct virtio_video_resource_destroy_all *)vio_cmd->cmd_buf;
+    enum v4l2_buf_type buf_type;
+    struct stream *s;
+    int ret = 0;
+
+    g_debug("%s: type(0x%x) %s stream_id(%d)", __func__,
+            cmd->hdr.type, vio_queue_name(le32toh(cmd->queue_type)),
+            cmd->hdr.stream_id);
+
+    s = find_stream(v, cmd->hdr.stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, cmd->hdr.stream_id);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    buf_type = get_v4l2_buf_type(le32toh(cmd->queue_type), s->has_mplane);
+
+    ret = v4l2_free_buffers(s->fd, buf_type);
+    if (ret) {
+        g_critical("%s: v4l2_free_buffers() failed", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    remove_all_resources(s, le32toh(cmd->queue_type));
+
+    /* free resource objects from queue list */
+    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+
+out:
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+}
+
+static void
+handle_stream_create_cmd(struct VuVideo *v,
+                         struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret = 0;
+    struct stream *s;
+    uint32_t req_stream_id;
+    uint32_t coded_format;
+
+    struct virtio_video_stream_create *cmd =
+        (struct virtio_video_stream_create *)vio_cmd->cmd_buf;
+
+    g_debug("%s: type(0x%x) stream_id(%d) in_mem_type(0x%x) "
+            "out_mem_type(0x%x) coded_format(0x%x)",
+            __func__, cmd->hdr.type, cmd->hdr.stream_id,
+            le32toh(cmd->in_mem_type), le32toh(cmd->out_mem_type),
+            le32toh(cmd->coded_format));
+
+    req_stream_id = cmd->hdr.stream_id;
+    coded_format = le32toh(cmd->coded_format);
+
+    if ((le32toh(cmd->in_mem_type) == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT) ||
+        (le32toh(cmd->out_mem_type) == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT)) {
+        /* TODO implement VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT */
+        g_printerr("%s: MEM_TYPE_VIRTIO_OBJECT not supported yet", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out;
+    }
+
+    if (!find_stream(v, req_stream_id)) {
+        s = g_new0(struct stream, 1);
+        /* copy but bswap */
+        s->vio_stream.in_mem_type = le32toh(cmd->in_mem_type);
+        s->vio_stream.out_mem_type = le32toh(cmd->out_mem_type);
+        s->vio_stream.coded_format = le32toh(cmd->coded_format);
+        strncpy((char *)&s->vio_stream.tag, (char *)cmd->tag,
+                sizeof(cmd->tag) - 1);
+        s->vio_stream.tag[sizeof(cmd->tag) - 1] = 0;
+        s->stream_id = req_stream_id;
+        s->video = v;
+        s->stream_state = STREAM_STOPPED;
+        s->has_mplane = v->v4l2_dev->has_mplane;
+        g_mutex_init(&s->mutex);
+        g_cond_init(&s->stream_cond);
+        v->streams = g_list_append(v->streams, s);
+
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+    } else {
+        g_debug("%s: Stream ID in use - ", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID;
+        goto out;
+    }
+
+    /* set the requested coded format */
+    ret = v4l2_stream_create(v->v4l2_dev, coded_format, s);
+    if (ret < 0) {
+        g_printerr("%s: v4l2_stream_create() failed", __func__);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+
+        v->streams = g_list_remove(v->streams, s);
+        g_free(s);
+    }
+
+    /*
+     * create thread to handle
+     *  - dequeing buffers from output & capture queues
+     *  - sending resource replies for buffers
+     *  - handling EOS and dynamic-resoltion events
+     */
+    s->worker_thread = g_thread_new("vio-video stream worker",
+                                    stream_worker_thread, s);
+
+out:
+    /* send response */
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+}
+
+static void
+handle_stream_drain_cmd(struct VuVideo *v,
+                          struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret;
+    struct stream *s;
+    uint32_t stream_id;
+    struct virtio_video_stream_drain *cmd =
+        (struct virtio_video_stream_drain *)vio_cmd->cmd_buf;
+
+    stream_id = cmd->hdr.stream_id;
+
+    g_debug("%s: stream_id(%d)", __func__, stream_id);
+
+    s = find_stream(v, stream_id);
+    if (!s) {
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID;
+        return;
+    }
+
+    g_debug("%s: Found stream=0x%p", __func__, s);
+
+    g_mutex_lock(&s->mutex);
+
+    ret = v4l2_issue_cmd(s->fd, V4L2_DEC_CMD_STOP, 0);
+    if (ret < 0) {
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        goto out_unlock;
+    }
+    s->stream_state = STREAM_DRAINING;
+    g_cond_signal(&s->stream_cond);
+
+    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+
+out_unlock:
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+}
+
+static void
+handle_stream_destroy_cmd(struct VuVideo *v,
+                          struct vu_video_ctrl_command *vio_cmd)
+{
+    struct stream *s;
+    uint32_t stream_id;
+    struct virtio_video_stream_destroy *cmd =
+        (struct virtio_video_stream_destroy *)vio_cmd->cmd_buf;
+    enum v4l2_buf_type buftype;
+
+    if (!v || !vio_cmd) {
+        return;
+    }
+
+    stream_id = cmd->hdr.stream_id;
+
+    g_debug("%s: stream_id=(%d)", __func__, stream_id);
+
+    s = find_stream(v, stream_id);
+
+    if (!s) {
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID;
+        return;
+    }
+
+    if (s) {
+        g_debug("%s: Found stream=0x%p", __func__, s);
+
+        g_mutex_lock(&s->mutex);
+        /* TODO assumes decoder */
+        buftype = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+            : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+        v4l2_streamoff(buftype, s);
+
+        /* signal worker thread */
+        s->stream_state = STREAM_DESTROYING;
+        g_cond_signal(&s->stream_cond);
+        g_mutex_unlock(&s->mutex);
+
+        /* wait for DESTROYED state */
+        g_mutex_lock(&s->mutex);
+        while (s->stream_state != STREAM_DESTROYED) {
+            g_cond_wait(&s->stream_cond, &s->mutex);
+        }
+
+        /* stream worker thread now exited */
+
+        /* deallocate the buffers */
+        v4l2_free_buffers(s->fd, buftype);
+        remove_all_resources(s, VIRTIO_VIDEO_QUEUE_TYPE_INPUT);
+
+        buftype = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE :
+            V4L2_BUF_TYPE_VIDEO_CAPTURE;
+
+        v4l2_free_buffers(s->fd, buftype);
+        remove_all_resources(s, VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT);
+
+        g_cond_clear(&s->stream_cond);
+
+        v4l2_close(s->fd);
+
+        v->streams = g_list_remove(v->streams, (gconstpointer) s);
+        cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+    }
+
+    /* send response */
+    vio_cmd->finished = true;
+    send_ctrl_response_nodata(vio_cmd);
+    g_mutex_unlock(&s->mutex);
+    g_mutex_clear(&s->mutex);
+    g_free(s);
+
+    return;
+}
+
+struct virtio_video_get_control_resp_level {
+    struct virtio_video_cmd_hdr hdr;
+    struct virtio_video_control_val_level level;
+};
+
+struct virtio_video_get_control_resp_profile {
+    struct virtio_video_cmd_hdr hdr;
+    struct virtio_video_control_val_profile profile;
+};
+
+struct virtio_video_get_control_resp_bitrate {
+    struct virtio_video_cmd_hdr hdr;
+    struct virtio_video_control_val_bitrate bitrate;
+};
+
+static int
+handle_get_control_cmd(struct VuVideo *v, struct vu_video_ctrl_command *vio_cmd)
+{
+    int ret;
+    uint32_t v4l2_control;
+    int32_t value;
+
+    struct virtio_video_get_control_resp ctl_resp_err;
+    struct virtio_video_get_control_resp_level   ctl_resp_level;
+    struct virtio_video_get_control_resp_profile ctl_resp_profile;
+    struct virtio_video_get_control_resp_bitrate ctl_resp_bitrate;
+
+    struct stream *s;
+
+    struct virtio_video_query_control *cmd =
+        (struct virtio_video_query_control *)vio_cmd->cmd_buf;
+
+    g_debug("%s: type(0x%x) stream_id(%d) control(0x%x)", __func__,
+            cmd->hdr.type, cmd->hdr.stream_id, le32toh(cmd->control));
+
+    s = find_stream(v, cmd->hdr.stream_id);
+    if (!s) {
+        g_critical("%s: stream_id(%d) not found", __func__, cmd->hdr.stream_id);
+        goto out;
+    }
+
+    g_mutex_lock(&s->mutex);
+
+    v4l2_control = virtio_video_control_to_v4l2(le32toh(cmd->control));
+    if (!v4l2_control) {
+        goto out_err_unlock;
+    }
+
+
+    switch (le32toh(cmd->control)) {
+    case VIRTIO_VIDEO_CONTROL_BITRATE:
+        g_debug("%s: VIRTIO_VIDEO_CONTROL_BITRATE", __func__);
+
+        ctl_resp_bitrate.hdr.stream_id = cmd->hdr.stream_id;
+        ctl_resp_bitrate.hdr.type = VIRTIO_VIDEO_RESP_OK_GET_PARAMS;
+
+        if (v->v4l2_dev->dev_type == STATEFUL_ENCODER) {
+            ret = v4l2_video_get_control(s->fd, v4l2_control, &value);
+            if (ret < 0) {
+                g_printerr("v4l2_video_get_control() failed\n");
+                goto out_err_unlock;
+            }
+            ctl_resp_bitrate.bitrate.bitrate = htole32(value);
+
+        } else {
+            g_debug("%s: CONTROL_BITRATE unsupported for decoders!", __func__);
+            goto out_err_unlock;
+        }
+
+        vio_cmd->finished = true;
+        send_ctrl_response(vio_cmd, (uint8_t *)&ctl_resp_bitrate,
+                           sizeof(struct virtio_video_get_control_resp_bitrate));
+        break;
+
+    case VIRTIO_VIDEO_CONTROL_PROFILE:
+        g_debug("%s: VIRTIO_VIDEO_CONTROL_PROFILE", __func__);
+
+        ctl_resp_profile.hdr.stream_id = cmd->hdr.stream_id;
+        ctl_resp_profile.hdr.type = VIRTIO_VIDEO_RESP_OK_GET_PARAMS;
+
+        ret = v4l2_video_get_control(s->fd, v4l2_control, &value);
+        if (ret < 0) {
+            g_printerr("v4l2_video_get_control() failed\n");
+            goto out_err_unlock;
+        }
+
+        ctl_resp_profile.profile.profile = htole32(value);
+
+        vio_cmd->finished = true;
+        send_ctrl_response(vio_cmd, (uint8_t *)&ctl_resp_profile,
+                           sizeof(struct virtio_video_get_control_resp_profile));
+
+        /*
+         * TODO need to determine "in use" codec to (h264/vp8/vp9) to map to
+         * v4l2 control for PROFILE?
+         */
+
+        break;
+
+    case VIRTIO_VIDEO_CONTROL_LEVEL:
+        g_debug("%s: VIRTIO_VIDEO_CONTROL_LEVEL", __func__);
+
+        ctl_resp_level.hdr.stream_id = cmd->hdr.stream_id;
+        ctl_resp_level.hdr.type = VIRTIO_VIDEO_RESP_OK_GET_PARAMS;
+
+        ret = v4l2_video_get_control(s->fd, v4l2_control, &value);
+        if (ret < 0) {
+            g_printerr("v4l2_video_get_control() failed\n");
+            goto out_err_unlock;
+        }
+
+        ctl_resp_level.level.level = htole32(value);
+
+        vio_cmd->finished = true;
+        send_ctrl_response(vio_cmd, (uint8_t *)&ctl_resp_level,
+                           sizeof(struct virtio_video_get_control_resp_level));
+        break;
+
+    default:
+        g_critical("Unknown control requested!");
+        goto out_err_unlock;
+        break;
+    }
+
+    return 0;
+
+out_err_unlock:
+    ctl_resp_err.hdr.stream_id = cmd->hdr.stream_id;
+    ctl_resp_err.hdr.type = VIRTIO_VIDEO_RESP_ERR_UNSUPPORTED_CONTROL;
+    vio_cmd->finished = true;
+    send_ctrl_response(vio_cmd, (uint8_t *)&ctl_resp_err,
+                       sizeof(struct virtio_video_get_control_resp));
+    g_mutex_unlock(&s->mutex);
+out:
+    return -EINVAL;
+}
+
+static int
+handle_query_capability_cmd(struct VuVideo *v,
+                            struct vu_video_ctrl_command *cmd)
+{
+    GList *fmt_l;
+    int ret;
+    enum v4l2_buf_type buf_type;
+    struct virtio_video_query_capability *qcmd =
+        (struct virtio_video_query_capability *)cmd->cmd_buf;
+    GByteArray *querycapresp;
+
+    /* hdr bswapped already */
+    g_debug("%s: type(0x%x) stream_id(%d) %s", __func__,
+            qcmd->hdr.type, qcmd->hdr.stream_id,
+            vio_queue_name(le32toh(qcmd->queue_type)));
+
+    buf_type = get_v4l2_buf_type(le32toh(qcmd->queue_type),
+                                 v->v4l2_dev->has_mplane);
+
+    /* enumerate formats */
+    ret = video_enum_formats(v->v4l2_dev, buf_type, &fmt_l, false);
+    if (ret < 0) {
+        g_printerr("video_enum_formats failed");
+        return ret;
+    }
+
+    querycapresp = g_byte_array_new();
+    querycapresp = create_query_cap_resp(qcmd, &fmt_l, querycapresp);
+    cmd->finished = true;
+    send_ctrl_response(cmd, querycapresp->data, querycapresp->len);
+
+    video_free_formats(&fmt_l);
+    g_byte_array_free(querycapresp, true);
+
+    return 0;
+}
+
+/* for v3 virtio-video spec currently */
+static void
+video_handle_ctrl(VuDev *dev, int qidx)
+{
+    VuVirtq *vq = vu_get_queue(dev, qidx);
+    VuVideo *video = container_of(dev, VuVideo, dev.parent);
+    size_t cmd_len, len;
+
+    struct vu_video_ctrl_command *cmd;
+
+    for (;;) {
+
+        cmd = vu_queue_pop(dev, vq, sizeof(struct vu_video_ctrl_command));
+        if (!cmd) {
+            break;
+        }
+
+        cmd->vq = vq;
+        cmd->error = 0;
+        cmd->finished = false;
+        cmd->dev = dev;
+
+        cmd_len = video_iov_size(cmd->elem.out_sg, cmd->elem.out_num);
+        cmd->cmd_buf = g_malloc0(cmd_len);
+        len = video_iov_to_buf(cmd->elem.out_sg, cmd->elem.out_num,
+                         0, cmd->cmd_buf, cmd_len);
+
+        if (len != cmd_len) {
+            g_warning("%s: command size incorrect %zu vs %zu\n",
+                      __func__, len, cmd_len);
+        }
+
+        /* header is first on every cmd struct */
+        cmd->cmd_hdr = (struct virtio_video_cmd_hdr *) cmd->cmd_buf;
+        /*bswap header */
+        virtio_video_ctrl_hdr_letoh(cmd->cmd_hdr);
+
+        switch (cmd->cmd_hdr->type) {
+        case VIRTIO_VIDEO_CMD_QUERY_CAPABILITY:
+            g_debug("Received VIRTIO_VIDEO_CMD_QUERY_CAPABILITY cmd");
+            handle_query_capability_cmd(video, cmd);
+            break;
+
+        case VIRTIO_VIDEO_CMD_STREAM_CREATE:
+            g_debug("Received VIRTIO_VIDEO_CMD_STREAM_CREATE cmd");
+            handle_stream_create_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_STREAM_DESTROY:
+            g_debug("Received VIRTIO_VIDEO_CMD_STREAM_DESTROY cmd");
+            handle_stream_destroy_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_STREAM_DRAIN:
+            g_debug("Received VIRTIO_VIDEO_CMD_STREAM_DRAIN cmd");
+            handle_stream_drain_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_RESOURCE_CREATE:
+            g_debug("Received VIRTIO_VIDEO_CMD_RESOURCE_CREATE cmd");
+            handle_resource_create_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_RESOURCE_QUEUE:
+            g_debug("Received VIRTIO_VIDEO_CMD_RESOURCE_QUEUE cmd");
+            handle_resource_queue_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL:
+            g_debug("Received VIRTIO_VIDEO_CMD_RESOURCE_DESTROY_ALL cmd");
+            handle_resource_destroy_all_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_QUEUE_CLEAR:
+            g_debug("Received VIRTIO_VIDEO_CMD_QUEUE_CLEAR cmd");
+            handle_queue_clear_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_GET_PARAMS:
+            g_debug("Received VIRTIO_VIDEO_CMD_GET_PARAMS cmd");
+            handle_get_params_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_SET_PARAMS:
+            g_debug("Received VIRTIO_VIDEO_CMD_SET_PARAMS cmd");
+            handle_set_params_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_QUERY_CONTROL:
+            g_error("**** VIRTIO_VIDEO_CMD_QUERY_CONTROL unimplemented!");
+            break;
+        case VIRTIO_VIDEO_CMD_GET_CONTROL:
+            g_debug("Received VIRTIO_VIDEO_CMD_GET_CONTROL cmd");
+            handle_get_control_cmd(video, cmd);
+            break;
+        case VIRTIO_VIDEO_CMD_SET_CONTROL:
+            g_error("**** VIRTIO_VIDEO_CMD_SET_CONTROL unimplemented!");
+            break;
+        default:
+            g_error("Unknown VIRTIO_VIDEO command!");
+            break;
+        }
+    }
+}
+
+static void
+video_queue_set_started(VuDev *dev, int qidx, bool started)
+{
+    VuVirtq *vq = vu_get_queue(dev, qidx);
+
+    g_debug("queue started %d:%d\n", qidx, started);
+
+    switch (qidx) {
+    case 0:
+        vu_set_queue_handler(dev, vq, started ? video_handle_ctrl : NULL);
+        break;
+    default:
+        break;
+    }
+}
+
+/*
+ * video_process_msg: process messages of vhost-user interface
+ *
+ * Any that are not handled here are processed by the libvhost library
+ * itself.
+ */
+static int video_process_msg(VuDev *dev, VhostUserMsg *msg, int *do_reply)
+{
+    VuVideo *r = container_of(dev, VuVideo, dev.parent);
+
+    g_debug("%s: msg %d", __func__, msg->request);
+
+    switch (msg->request) {
+    case VHOST_USER_NONE:
+        g_main_loop_quit(r->loop);
+        return 1;
+    default:
+        return 0;
+    }
+
+    return 0;
+}
+
+static const VuDevIface vuiface = {
+    .set_features = video_set_features,
+    .get_features = video_get_features,
+    .queue_set_started = video_queue_set_started,
+    .process_msg = video_process_msg,
+    .get_config = video_get_config,
+    .set_config = video_set_config,
+};
+
+static void video_destroy(VuVideo *v)
+{
+    vug_deinit(&v->dev);
+    if (socket_path) {
+        unlink(socket_path);
+    }
+
+    v4l2_backend_free(v->v4l2_dev);
+}
+
+/* Print vhost-user.json backend program capabilities */
+static void print_capabilities(void)
+{
+    printf("{\n");
+    printf("  \"type\": \"misc\"\n");
+    printf("}\n");
+}
+
+static gboolean hangup(gpointer user_data)
+{
+    GMainLoop *loop = (GMainLoop *) user_data;
+    g_info("%s: caught hangup/quit signal, quitting main loop", __func__);
+    g_main_loop_quit(loop);
+    return true;
+}
+
+int main(int argc, char *argv[])
+{
+    GError *error = NULL;
+    GOptionContext *context;
+    g_autoptr(GSocket) socket = NULL;
+    VuVideo video = {  };
+
+    context = g_option_context_new("vhost-user emulation of video device");
+    g_option_context_add_main_entries(context, options, "vhost-user-video");
+    if (!g_option_context_parse(context, &argc, &argv, &error)) {
+        g_printerr("option parsing failed: %s\n", error->message);
+        exit(1);
+    }
+
+    g_option_context_free(context);
+
+    if (print_cap) {
+        print_capabilities();
+        exit(0);
+    }
+
+    if (!socket_path && socket_fd < 0) {
+        g_printerr("Please specify either --fd or --socket-path\n");
+        exit(EXIT_FAILURE);
+    }
+
+    if (verbose || debug) {
+        g_log_set_handler(NULL, G_LOG_LEVEL_MASK, g_log_default_handler, NULL);
+        if (debug) {
+            g_setenv("G_MESSAGES_DEBUG", "all", true);
+        }
+    } else {
+        g_log_set_handler(NULL,
+                          G_LOG_LEVEL_WARNING | G_LOG_LEVEL_CRITICAL
+                          | G_LOG_LEVEL_ERROR,
+                          g_log_default_handler, NULL);
+    }
+
+    /*
+     * Open the v4l2 device and enumerate supported formats.
+     * Use this to determine whether it is a stateful encoder/decoder.
+     */
+    if (!v4l2_path || !g_file_test(v4l2_path, G_FILE_TEST_EXISTS)) {
+        g_printerr("Please specify a valid --v4l2-device\n");
+        exit(EXIT_FAILURE);
+    } else {
+        video.v4l2_dev = v4l2_backend_init(v4l2_path);
+        if (!video.v4l2_dev) {
+            g_printerr("v4l2 backend init failed!\n");
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /*
+     * Now create a vhost-user socket that we will receive messages
+     * on. Once we have our handler set up we can enter the glib main
+     * loop.
+     */
+    if (socket_path) {
+        g_autoptr(GSocketAddress) addr = g_unix_socket_address_new(socket_path);
+        g_autoptr(GSocket) bind_socket =
+            g_socket_new(G_SOCKET_FAMILY_UNIX, G_SOCKET_TYPE_STREAM,
+                         G_SOCKET_PROTOCOL_DEFAULT, &error);
+
+        if (!g_socket_bind(bind_socket, addr, false, &error)) {
+            g_printerr("Failed to bind to socket at %s (%s).\n",
+                       socket_path, error->message);
+            exit(EXIT_FAILURE);
+        }
+        if (!g_socket_listen(bind_socket, &error)) {
+            g_printerr("Failed to listen on socket %s (%s).\n",
+                       socket_path, error->message);
+        }
+        g_message("awaiting connection to %s", socket_path);
+        socket = g_socket_accept(bind_socket, NULL, &error);
+        if (!socket) {
+            g_printerr("Failed to accept on socket %s (%s).\n",
+                       socket_path, error->message);
+        }
+    } else {
+        socket = g_socket_new_from_fd(socket_fd, &error);
+        if (!socket) {
+            g_printerr("Failed to connect to FD %d (%s).\n",
+                       socket_fd, error->message);
+            exit(EXIT_FAILURE);
+        }
+    }
+
+    /*
+     * Create the main loop first so all the various sources can be
+     * added. As well as catching signals we need to ensure vug_init
+     * can add it's GSource watches.
+     */
+
+    video.loop = g_main_loop_new(NULL, FALSE);
+    /* catch exit signals */
+    g_unix_signal_add(SIGHUP, hangup, video.loop);
+    g_unix_signal_add(SIGINT, hangup, video.loop);
+
+    if (!vug_init(&video.dev, VHOST_USER_VIDEO_MAX_QUEUES,
+                  g_socket_get_fd(socket),
+                  video_panic, &vuiface)) {
+        g_printerr("Failed to initialize libvhost-user-glib.\n");
+        exit(EXIT_FAILURE);
+    }
+
+    g_message("entering main loop, awaiting messages");
+    g_main_loop_run(video.loop);
+    g_message("finished main loop, cleaning up");
+
+    g_main_loop_unref(video.loop);
+    video_destroy(&video);
+}
diff --git a/tools/vhost-user-video/meson.build b/tools/vhost-user-video/meson.build
new file mode 100644
index 0000000000..931e73c15d
--- /dev/null
+++ b/tools/vhost-user-video/meson.build
@@ -0,0 +1,10 @@
+executable('vhost-user-video', files(
+  'main.c', 'v4l2_backend.c', 'virtio_video_helpers.c'),
+  dependencies: [qemuutil, glib, gio, vhost_user],
+  install: true,
+  install_dir: get_option('libexecdir'))
+
+configure_file(input: '50-qemu-rpmb.json.in',
+               output: '50-qemu-rpmb.json',
+               configuration: config_host,
+               install_dir: qemu_datadir / 'vhost-user')
diff --git a/tools/vhost-user-video/v4l2_backend.c b/tools/vhost-user-video/v4l2_backend.c
new file mode 100644
index 0000000000..f90b76b172
--- /dev/null
+++ b/tools/vhost-user-video/v4l2_backend.c
@@ -0,0 +1,1777 @@
+/*
+ * virtio-video video v4l2 backend
+ *
+ * The purpose of this backend is to interface with
+ * v4l2 stateful encoder and decoder devices in the kernel.
+ *
+ * v4l2 stateless devices are NOT supported currently.
+ *
+ * Some v4l2 helper functions taken from yatva
+ *
+ * Copyright (c) 2021 Linaro Ltd
+ * Copyright (C) 2005-2010 Laurent Pinchart
+ *
+ * SPDX-License-Identifier: GPL-2.0-or-later
+ */
+
+#include <linux/videodev2.h>
+
+#include <glib.h>
+#include <glib/gstdio.h>
+
+#include <assert.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdbool.h>
+#include <sys/epoll.h>
+#include <sys/ioctl.h>
+
+#include <unistd.h>
+#include "virtio_video_helpers.h"
+#include "v4l2_backend.h"
+#include "standard-headers/linux/virtio_video.h"
+#include "vuvideo.h"
+
+/* function prototypes */
+
+static const struct v4l2_format_info *v4l2_format_by_fourcc(unsigned int fourcc);
+static const char *v4l2_format_name(unsigned int fourcc);
+static const char *v4l2_buf_type_name(enum v4l2_buf_type type);
+static const char *v4l2_field_name(enum v4l2_field field);
+
+static int video_enum_frame_intervals(struct v4l2_device *dev,
+                                      __u32 pixelformat,
+                                      unsigned int width, unsigned int height,
+                                      GList **p_vid_fmt_frm_rate_l);
+
+static int video_enum_frame_sizes(struct v4l2_device *dev, __u32 pixelformat,
+                                  GList **p_vid_fmt_frm_l);
+static int video_querycap(struct v4l2_device *dev);
+static GByteArray *iterate_frame_rate_list(GByteArray *resp,
+                                            GList *frm_rate_l);
+static GByteArray *iterate_format_frame_list(GByteArray *resp,
+                                             GList *fmt_frm_l);
+static GByteArray *iterate_format_desc_list(GByteArray *resp,
+                                            GList *fmt_desc_l);
+
+/* v4l2 to str tables & helpers taken from yavta to make prettier logs */
+
+static struct v4l2_format_info {
+    const char *name;
+    unsigned int fourcc;
+    unsigned char n_planes;
+} pixel_formats[] = {
+    { "RGB332", V4L2_PIX_FMT_RGB332, 1 },
+    { "RGB444", V4L2_PIX_FMT_RGB444, 1 },
+    { "ARGB444", V4L2_PIX_FMT_ARGB444, 1 },
+    { "XRGB444", V4L2_PIX_FMT_XRGB444, 1 },
+    { "RGB555", V4L2_PIX_FMT_RGB555, 1 },
+    { "ARGB555", V4L2_PIX_FMT_ARGB555, 1 },
+    { "XRGB555", V4L2_PIX_FMT_XRGB555, 1 },
+    { "RGB565", V4L2_PIX_FMT_RGB565, 1 },
+    { "RGB555X", V4L2_PIX_FMT_RGB555X, 1 },
+    { "RGB565X", V4L2_PIX_FMT_RGB565X, 1 },
+    { "BGR666", V4L2_PIX_FMT_BGR666, 1 },
+    { "BGR24", V4L2_PIX_FMT_BGR24, 1 },
+    { "RGB24", V4L2_PIX_FMT_RGB24, 1 },
+    { "BGR32", V4L2_PIX_FMT_BGR32, 1 },
+    { "ABGR32", V4L2_PIX_FMT_ABGR32, 1 },
+    { "XBGR32", V4L2_PIX_FMT_XBGR32, 1 },
+    { "RGB32", V4L2_PIX_FMT_RGB32, 1 },
+    { "ARGB32", V4L2_PIX_FMT_ARGB32, 1 },
+    { "XRGB32", V4L2_PIX_FMT_XRGB32, 1 },
+    { "HSV24", V4L2_PIX_FMT_HSV24, 1 },
+    { "HSV32", V4L2_PIX_FMT_HSV32, 1 },
+    { "Y8", V4L2_PIX_FMT_GREY, 1 },
+    { "Y10", V4L2_PIX_FMT_Y10, 1 },
+    { "Y12", V4L2_PIX_FMT_Y12, 1 },
+    { "Y16", V4L2_PIX_FMT_Y16, 1 },
+    { "UYVY", V4L2_PIX_FMT_UYVY, 1 },
+    { "VYUY", V4L2_PIX_FMT_VYUY, 1 },
+    { "YUYV", V4L2_PIX_FMT_YUYV, 1 },
+    { "YVYU", V4L2_PIX_FMT_YVYU, 1 },
+    { "NV12", V4L2_PIX_FMT_NV12, 1 },
+    { "NV12M", V4L2_PIX_FMT_NV12M, 2 },
+    { "NV21", V4L2_PIX_FMT_NV21, 1 },
+    { "NV21M", V4L2_PIX_FMT_NV21M, 2 },
+    { "NV16", V4L2_PIX_FMT_NV16, 1 },
+    { "NV16M", V4L2_PIX_FMT_NV16M, 2 },
+    { "NV61", V4L2_PIX_FMT_NV61, 1 },
+    { "NV61M", V4L2_PIX_FMT_NV61M, 2 },
+    { "NV24", V4L2_PIX_FMT_NV24, 1 },
+    { "NV42", V4L2_PIX_FMT_NV42, 1 },
+    { "YU12", V4L2_PIX_FMT_YVU420, 1},
+    { "YUV420M", V4L2_PIX_FMT_YUV420M, 3 },
+    { "YUV422M", V4L2_PIX_FMT_YUV422M, 3 },
+    { "YUV444M", V4L2_PIX_FMT_YUV444M, 3 },
+    { "YVU420M", V4L2_PIX_FMT_YVU420M, 3 },
+    { "YVU422M", V4L2_PIX_FMT_YVU422M, 3 },
+    { "YVU444M", V4L2_PIX_FMT_YVU444M, 3 },
+    { "SBGGR8", V4L2_PIX_FMT_SBGGR8, 1 },
+    { "SGBRG8", V4L2_PIX_FMT_SGBRG8, 1 },
+    { "SGRBG8", V4L2_PIX_FMT_SGRBG8, 1 },
+    { "SRGGB8", V4L2_PIX_FMT_SRGGB8, 1 },
+    { "SBGGR10_DPCM8", V4L2_PIX_FMT_SBGGR10DPCM8, 1 },
+    { "SGBRG10_DPCM8", V4L2_PIX_FMT_SGBRG10DPCM8, 1 },
+    { "SGRBG10_DPCM8", V4L2_PIX_FMT_SGRBG10DPCM8, 1 },
+    { "SRGGB10_DPCM8", V4L2_PIX_FMT_SRGGB10DPCM8, 1 },
+    { "SBGGR10", V4L2_PIX_FMT_SBGGR10, 1 },
+    { "SGBRG10", V4L2_PIX_FMT_SGBRG10, 1 },
+    { "SGRBG10", V4L2_PIX_FMT_SGRBG10, 1 },
+    { "SRGGB10", V4L2_PIX_FMT_SRGGB10, 1 },
+    { "SBGGR10P", V4L2_PIX_FMT_SBGGR10P, 1 },
+    { "SGBRG10P", V4L2_PIX_FMT_SGBRG10P, 1 },
+    { "SGRBG10P", V4L2_PIX_FMT_SGRBG10P, 1 },
+    { "SRGGB10P", V4L2_PIX_FMT_SRGGB10P, 1 },
+    { "SBGGR12", V4L2_PIX_FMT_SBGGR12, 1 },
+    { "SGBRG12", V4L2_PIX_FMT_SGBRG12, 1 },
+    { "SGRBG12", V4L2_PIX_FMT_SGRBG12, 1 },
+    { "SRGGB12", V4L2_PIX_FMT_SRGGB12, 1 },
+    { "IPU3_SBGGR10", V4L2_PIX_FMT_IPU3_SBGGR10, 1 },
+    { "IPU3_SGBRG10", V4L2_PIX_FMT_IPU3_SGBRG10, 1 },
+    { "IPU3_SGRBG10", V4L2_PIX_FMT_IPU3_SGRBG10, 1 },
+    { "IPU3_SRGGB10", V4L2_PIX_FMT_IPU3_SRGGB10, 1 },
+    { "DV", V4L2_PIX_FMT_DV, 1 },
+    { "MJPEG", V4L2_PIX_FMT_MJPEG, 1 },
+    { "MPEG", V4L2_PIX_FMT_MPEG, 1 },
+    { "FWHT", V4L2_PIX_FMT_FWHT, 1 },
+};
+
+bool video_is_mplane(enum v4l2_buf_type type)
+{
+    return type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
+        type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE;
+}
+
+bool video_is_splane(enum v4l2_buf_type type)
+{
+    return type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
+        type == V4L2_BUF_TYPE_VIDEO_OUTPUT;
+}
+bool video_is_meta(enum v4l2_buf_type type)
+{
+    return type == V4L2_BUF_TYPE_META_CAPTURE ||
+        type == V4L2_BUF_TYPE_META_OUTPUT;
+}
+
+bool is_capture_queue(enum v4l2_buf_type type)
+{
+    return type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE ||
+        type == V4L2_BUF_TYPE_VIDEO_CAPTURE ||
+        type == V4L2_BUF_TYPE_META_CAPTURE;
+}
+
+bool is_output_queue(enum v4l2_buf_type type)
+{
+    return type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE ||
+        type == V4L2_BUF_TYPE_VIDEO_OUTPUT ||
+        type == V4L2_BUF_TYPE_META_OUTPUT;
+}
+
+static const struct v4l2_format_info *v4l2_format_by_fourcc(unsigned int fourcc)
+{
+    unsigned int i;
+
+    for (i = 0; i < ARRAY_SIZE(pixel_formats); ++i) {
+        if (pixel_formats[i].fourcc == fourcc) {
+            return &pixel_formats[i];
+        }
+    }
+
+    return NULL;
+}
+
+static const char *v4l2_format_name(unsigned int fourcc)
+{
+    const struct v4l2_format_info *info;
+    static char name[5];
+    unsigned int i;
+
+    info = v4l2_format_by_fourcc(fourcc);
+    if (info) {
+        return info->name;
+    }
+
+    for (i = 0; i < 4; ++i) {
+        name[i] = fourcc & 0xff;
+        fourcc >>= 8;
+    }
+
+    name[4] = '\0';
+    return name;
+}
+
+static struct {
+    enum v4l2_buf_type type;
+    bool supported;
+    const char *name;
+} buf_types_array[] = {
+    { V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE, 1, "Video capture mplanes", },
+    { V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE, 1, "Video output mplanes", },
+    { V4L2_BUF_TYPE_VIDEO_CAPTURE, 1, "Video capture", },
+    { V4L2_BUF_TYPE_VIDEO_OUTPUT, 1, "Video output", },
+    { V4L2_BUF_TYPE_VIDEO_OVERLAY, 0, "Video overlay", },
+    { V4L2_BUF_TYPE_META_CAPTURE, 0, "Meta-data capture", },
+    { V4L2_BUF_TYPE_META_OUTPUT, 0, "Meta-data output", },
+};
+
+static const char *v4l2_buf_type_name(enum v4l2_buf_type type)
+{
+    unsigned int i;
+
+    for (i = 0; i < ARRAY_SIZE(buf_types_array); ++i) {
+        if (buf_types_array[i].type == type) {
+            return buf_types_array[i].name;
+        }
+    }
+
+    if (type & V4L2_BUF_TYPE_PRIVATE) {
+        return "Private";
+    } else {
+        return "Unknown";
+    }
+}
+
+static const struct {
+    const char *name;
+    enum v4l2_field field;
+} fields[] = {
+    { "any", V4L2_FIELD_ANY },
+    { "none", V4L2_FIELD_NONE },
+    { "top", V4L2_FIELD_TOP },
+    { "bottom", V4L2_FIELD_BOTTOM },
+    { "interlaced", V4L2_FIELD_INTERLACED },
+    { "seq-tb", V4L2_FIELD_SEQ_TB },
+    { "seq-bt", V4L2_FIELD_SEQ_BT },
+    { "alternate", V4L2_FIELD_ALTERNATE },
+    { "interlaced-tb", V4L2_FIELD_INTERLACED_TB },
+    { "interlaced-bt", V4L2_FIELD_INTERLACED_BT },
+};
+
+static const char *v4l2_field_name(enum v4l2_field field)
+{
+    unsigned int i;
+
+    for (i = 0; i < ARRAY_SIZE(fields); ++i) {
+        if (fields[i].field == field) {
+            return fields[i].name;
+        }
+    }
+
+    return "unknown";
+}
+
+int v4l2_open(const gchar *devname)
+{
+    int fd;
+
+    if (!devname) {
+        return -EINVAL;
+    }
+
+    fd = open(devname, O_RDWR | O_NONBLOCK);
+    if (fd < 0) {
+        g_printerr("Error opening device %s: %s (%d).\n", devname,
+                   g_strerror(errno), errno);
+        return fd;
+    }
+
+    g_print("Device %s opened fd(%d).\n", devname, fd);
+
+    return fd;
+}
+
+int v4l2_close(int fd)
+{
+    int ret;
+    ret = close(fd);
+
+    if (ret < 0) {
+        g_printerr("%s: close failed errno(%s)", __func__, g_strerror(errno));
+    }
+    return ret;
+}
+
+static int video_enum_frame_intervals(struct v4l2_device *dev,
+                                      __u32 pixelformat,
+                                      unsigned int width, unsigned int height,
+                                      GList **p_vid_fmt_frm_rate_l)
+{
+    struct v4l2_frmivalenum ival;
+    GList *vid_fmt_frm_rate_l = NULL;
+    struct video_format_frame_rates *fmt_frm_rate;
+    unsigned int i;
+    int ret = 0;
+
+    for (i = 0; ; ++i) {
+        memset(&ival, 0, sizeof ival);
+        ival.index = i;
+        ival.pixel_format = pixelformat;
+        ival.width = width;
+        ival.height = height;
+        ret = ioctl(dev->fd, VIDIOC_ENUM_FRAMEINTERVALS, &ival);
+        if (ret < 0) {
+            if (errno == EINVAL) /* EINVAL means no more frame intervals */
+                ret = 0;
+            else
+                g_printerr("%s: VIDIOC_ENUM_FRAMEINTERVALS failed %s\n"
+                           , __func__, g_strerror(errno));
+            break;
+        }
+
+        /* driver sanity checks */
+        if (i != ival.index)
+            g_printerr("Warning: driver returned wrong ival index "
+                       "%u.\n", ival.index);
+        if (pixelformat != ival.pixel_format)
+            g_printerr("Warning: driver returned wrong ival pixel "
+                       "format %08x.\n", ival.pixel_format);
+        if (width != ival.width)
+            g_printerr("Warning: driver returned wrong ival width "
+                       "%u.\n", ival.width);
+        if (height != ival.height)
+            g_printerr("Warning: driver returned wrong ival height "
+                       "%u.\n", ival.height);
+
+        if (i != 0) {
+            g_print(", ");
+        }
+
+        /* allocate video_format_frame */
+        fmt_frm_rate = g_new0(struct video_format_frame_rates, 1);
+        /* keep a copy of v4l2 frmsizeenum struct */
+        memcpy(&fmt_frm_rate->v4l_ival, &ival,
+               sizeof(struct v4l2_frmivalenum));
+        vid_fmt_frm_rate_l =
+            g_list_append(vid_fmt_frm_rate_l, fmt_frm_rate);
+
+        switch (ival.type) {
+        case V4L2_FRMIVAL_TYPE_DISCRETE:
+            g_debug("%u/%u",
+                    ival.discrete.numerator,
+                    ival.discrete.denominator);
+
+            fmt_frm_rate->frame_rates.min = ival.discrete.denominator;
+
+            break;
+
+        case V4L2_FRMIVAL_TYPE_CONTINUOUS:
+            g_debug("%u/%u - %u/%u",
+                    ival.stepwise.min.numerator,
+                    ival.stepwise.min.denominator,
+                    ival.stepwise.max.numerator,
+                    ival.stepwise.max.denominator);
+
+            fmt_frm_rate->frame_rates.min = ival.stepwise.min.denominator;
+            fmt_frm_rate->frame_rates.max = ival.stepwise.max.denominator;
+            fmt_frm_rate->frame_rates.step = 1;
+
+            goto out;
+
+        case V4L2_FRMIVAL_TYPE_STEPWISE:
+            g_debug("%u/%u - %u/%u (by %u/%u)",
+                    ival.stepwise.min.numerator,
+                    ival.stepwise.min.denominator,
+                    ival.stepwise.max.numerator,
+                    ival.stepwise.max.denominator,
+                    ival.stepwise.step.numerator,
+                    ival.stepwise.step.denominator);
+
+            fmt_frm_rate->frame_rates.min = ival.stepwise.min.denominator;
+            fmt_frm_rate->frame_rates.max = ival.stepwise.max.denominator;
+            fmt_frm_rate->frame_rates.step = ival.stepwise.step.denominator;
+
+            goto out;
+
+        default:
+            break;
+        }
+    }
+
+out:
+    if (ret == 0) {
+        g_print("\n%s: Enumerated %d frame intervals\n", __func__
+                , g_list_length(vid_fmt_frm_rate_l));
+        g_return_val_if_fail(i == g_list_length(vid_fmt_frm_rate_l), -EINVAL);
+        *p_vid_fmt_frm_rate_l = vid_fmt_frm_rate_l;
+    }
+
+    return ret;
+}
+
+static int video_enum_frame_sizes(struct v4l2_device *dev,
+                                  __u32 pixelformat, GList **p_vid_fmt_frm_l)
+{
+    struct v4l2_frmsizeenum frame;
+    struct video_format_frame *vid_frame = NULL;
+    GList *vid_fmt_frm_l = NULL;
+    unsigned int i;
+    int ret;
+
+    if (!dev) {
+        return -EINVAL;
+    }
+
+    for (i = 0; ; ++i) {
+        memset(&frame, 0, sizeof frame);
+        frame.index = i;
+        frame.pixel_format = pixelformat;
+        ret = ioctl(dev->fd, VIDIOC_ENUM_FRAMESIZES, &frame);
+        if (ret < 0) {
+            if (errno == EINVAL) /* EINVAL means no more frame sizes */
+                ret = 0;
+            else
+                g_printerr("%s: VIDIOC_ENUM_FRAMESIZES failed %s\n",
+                           __func__, g_strerror(errno));
+            break;
+        }
+
+        /* driver sanity checks */
+        if (i != frame.index)
+            g_printerr("Warning: driver returned wrong frame index "
+                       "%u.\n", frame.index);
+        if (pixelformat != frame.pixel_format)
+            g_printerr("Warning: driver returned wrong frame pixel "
+                       "format %08x.\n", frame.pixel_format);
+
+        /* allocate video_format_frame */
+        vid_frame = g_new0(struct video_format_frame, 1);
+        /* keep a copy of v4l2 frmsizeenum struct */
+        memcpy(&vid_frame->v4l_framesize, &frame,
+               sizeof(struct v4l2_frmsizeenum));
+        vid_fmt_frm_l = g_list_append(vid_fmt_frm_l, vid_frame);
+
+        switch (frame.type) {
+        case V4L2_FRMSIZE_TYPE_DISCRETE:
+            g_debug("\tFrame size (D): %ux%u (", frame.discrete.width,
+                    frame.discrete.height);
+
+            vid_frame->frame.width.min = htole32(frame.discrete.width);
+            vid_frame->frame.width.max = htole32(frame.discrete.width);
+            vid_frame->frame.height.min = htole32(frame.discrete.height);
+            vid_frame->frame.height.max = htole32(frame.discrete.height);
+
+            if (video_enum_frame_intervals(dev, frame.pixel_format,
+                                           frame.discrete.width,
+                                           frame.discrete.height,
+                                           &vid_frame->frm_rate_l) < 0)
+                g_printerr("%s: video_enum_frame_intervals failed!", __func__);
+            g_debug(")");
+            break;
+
+        case V4L2_FRMSIZE_TYPE_CONTINUOUS:
+            g_debug("\tFrame size (C): %ux%u - %ux%u (",
+                    frame.stepwise.min_width,
+                    frame.stepwise.min_height,
+                    frame.stepwise.max_width,
+                    frame.stepwise.max_height);
+
+            vid_frame->frame.width.min = htole32(frame.stepwise.min_width);
+            vid_frame->frame.width.max = htole32(frame.stepwise.max_width);
+            vid_frame->frame.width.step = htole32(frame.stepwise.step_width);
+            vid_frame->frame.height.min = htole32(frame.stepwise.min_height);
+            vid_frame->frame.height.max = htole32(frame.stepwise.max_height);
+            vid_frame->frame.height.step = htole32(frame.stepwise.step_height);
+
+            /* driver sanity check */
+            if (frame.stepwise.step_height != 1 ||
+                frame.stepwise.step_width != 1) {
+                g_printerr("Warning: invalid step for continuous framesize");
+            }
+
+            if (video_enum_frame_intervals(dev, frame.pixel_format,
+                                           frame.stepwise.max_width,
+                                           frame.stepwise.max_height,
+                                           &vid_frame->frm_rate_l) < 0)
+                g_printerr("%s: video_enum_frame_intervals failed!\n"
+                           , __func__);
+
+            g_debug(")");
+            break;
+
+        case V4L2_FRMSIZE_TYPE_STEPWISE:
+            g_debug("\tFrame size (S): %ux%u - %ux%u (by %ux%u) (",
+                    frame.stepwise.min_width,
+                    frame.stepwise.min_height,
+                    frame.stepwise.max_width,
+                    frame.stepwise.max_height,
+                    frame.stepwise.step_width,
+                    frame.stepwise.step_height);
+
+            vid_frame->frame.width.min = htole32(frame.stepwise.min_width);
+            vid_frame->frame.width.max = htole32(frame.stepwise.max_width);
+            vid_frame->frame.width.step = htole32(frame.stepwise.step_width);
+            vid_frame->frame.height.min = htole32(frame.stepwise.min_height);
+            vid_frame->frame.height.max = htole32(frame.stepwise.max_height);
+            vid_frame->frame.height.step = htole32(frame.stepwise.step_height);
+
+            if (video_enum_frame_intervals(dev, frame.pixel_format,
+                                       frame.stepwise.max_width,
+                                       frame.stepwise.max_height,
+                                       &vid_frame->frm_rate_l) < 0)
+                g_printerr("%s: video_enum_frame_intervals failed!\n"
+                           , __func__);
+
+            g_debug(")");
+            break;
+
+        default:
+            break;
+        }
+    }
+    if (ret == 0) {
+        g_print("%s: Enumerated %d frame sizes and %d frame intervals\n",
+                __func__, g_list_length(vid_fmt_frm_l),
+                g_list_length(vid_frame->frm_rate_l));
+
+        vid_frame->frame.num_rates =
+            htole32(g_list_length(vid_frame->frm_rate_l));
+
+        g_return_val_if_fail(i == g_list_length(vid_fmt_frm_l), -EINVAL);
+        *p_vid_fmt_frm_l = vid_fmt_frm_l;
+    }
+
+    return ret;
+}
+
+int video_send_decoder_start_cmd(struct v4l2_device *dev)
+{
+    int ret = 0;
+    struct v4l2_decoder_cmd cmd;
+    g_debug("%s: ", __func__);
+
+    cmd.cmd = V4L2_DEC_CMD_START;
+    cmd.flags = 0;
+    cmd.start.speed = 1000;
+
+    memset(&cmd, 0, sizeof cmd);
+
+    ret = ioctl(dev->fd, VIDIOC_DECODER_CMD, &cmd);
+    if (ret < 0) {
+        g_printerr("%s: %s (%d)\n", __func__, g_strerror(errno), errno);
+    }
+
+    return ret;
+}
+
+static int video_querycap(struct v4l2_device *dev)
+{
+        struct v4l2_capability cap;
+        unsigned int caps;
+        bool has_video;
+        bool has_meta;
+        bool has_capture;
+        bool has_output;
+        bool has_mplane;
+        int ret;
+
+        memset(&cap, 0, sizeof cap);
+        ret = ioctl(dev->fd, VIDIOC_QUERYCAP, &cap);
+        if (ret < 0) {
+                return 0;
+        }
+
+        caps = cap.capabilities & V4L2_CAP_DEVICE_CAPS
+             ? cap.device_caps : cap.capabilities;
+
+        has_video = caps & (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+                            V4L2_CAP_VIDEO_CAPTURE |
+                            V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+                            V4L2_CAP_VIDEO_OUTPUT);
+        has_meta = caps & (V4L2_CAP_META_CAPTURE |
+                           V4L2_CAP_META_OUTPUT);
+        has_capture = caps & (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+                              V4L2_CAP_VIDEO_CAPTURE |
+                              V4L2_CAP_META_CAPTURE);
+        has_output = caps & (V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+                             V4L2_CAP_VIDEO_OUTPUT |
+                             V4L2_CAP_META_OUTPUT);
+        has_mplane = caps & (V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+                             V4L2_CAP_VIDEO_OUTPUT_MPLANE |
+                             V4L2_CAP_VIDEO_M2M_MPLANE);
+
+        g_print("Device `%s' on `%s' (driver '%s') "
+                "supports%s%s%s%s %s mplanes.\n",
+                cap.card, cap.bus_info, cap.driver,
+                has_video ? " video," : "",
+                has_meta ? " meta-data," : "",
+                has_capture ? " capture," : "",
+                has_output ? " output," : "",
+                has_mplane ? "with" : "without");
+
+        dev->capabilities = caps;
+        dev->has_mplane = has_mplane;
+
+        return 0;
+}
+
+void v4l2_set_device_type(struct v4l2_device *dev, enum v4l2_buf_type type,
+                          struct v4l2_fmtdesc *fmt_desc)
+{
+    if (fmt_desc->flags & V4L2_FMT_FLAG_COMPRESSED) {
+
+        switch (fmt_desc->pixelformat) {
+        case V4L2_PIX_FMT_H263:
+        case V4L2_PIX_FMT_H264:
+        case V4L2_PIX_FMT_H264_NO_SC:
+        case V4L2_PIX_FMT_H264_MVC:
+        case V4L2_PIX_FMT_MPEG1:
+        case V4L2_PIX_FMT_MPEG2:
+        case V4L2_PIX_FMT_MPEG4:
+        case V4L2_PIX_FMT_XVID:
+        case V4L2_PIX_FMT_VC1_ANNEX_G:
+        case V4L2_PIX_FMT_VC1_ANNEX_L:
+        case V4L2_PIX_FMT_VP8:
+        case V4L2_PIX_FMT_VP9:
+        case V4L2_PIX_FMT_HEVC:
+        case V4L2_PIX_FMT_FWHT:
+            if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT) {
+                dev->dev_type |= STATEFUL_DECODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+                dev->dev_type |= STATEFUL_DECODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE) {
+                dev->dev_type |= STATEFUL_ENCODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+                dev->dev_type |= STATEFUL_ENCODER;
+            }
+            break;
+        case V4L2_PIX_FMT_MPEG2_SLICE:
+        case V4L2_PIX_FMT_FWHT_STATELESS:
+            if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT) {
+                dev->dev_type |= STATELESS_DECODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE) {
+                dev->dev_type |= STATELESS_DECODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE) {
+                dev->dev_type |= STATELESS_ENCODER;
+            }
+            if (type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
+                dev->dev_type |= STATELESS_ENCODER;
+            }
+            break;
+        default:
+            break;
+        }
+    }
+}
+
+enum v4l2_buf_type get_v4l2_buf_type (enum virtio_video_queue_type queue_type,
+                                      bool has_mplane)
+{
+    enum v4l2_buf_type buf_type;
+
+    switch (queue_type) {
+    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
+        buf_type = has_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+            : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+        break;
+    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
+        buf_type = has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
+            : V4L2_BUF_TYPE_VIDEO_CAPTURE;
+        break;
+
+    default:
+        g_warning("%s: Unknown queue_type!", __func__);
+    }
+
+    g_debug("%s: queue_type(0x%x) has_mplane(%d), buf_type(%s)"
+            , __func__, queue_type, has_mplane, v4l2_buf_type_name(buf_type));
+
+    return buf_type;
+}
+
+int v4l2_free_buffers(int fd, enum v4l2_buf_type type)
+{
+    struct v4l2_requestbuffers reqbuf;
+    int ret;
+    g_debug("%s: v4l2_buf_type: %s: Issuing REQBUFS 0"
+            , __func__, v4l2_buf_type_name(type));
+
+    memset(&reqbuf, 0, sizeof(reqbuf));
+    reqbuf.type = type;
+    reqbuf.count = 0;
+
+    /*TODO must save this when creating resource on queue */
+    reqbuf.memory = V4L2_MEMORY_USERPTR;
+
+    /*
+     * Applications can call ioctl VIDIOC_REQBUFS again to change the number
+     * of buffers. Note that if any buffers are still mapped or exported via
+     * DMABUF, then ioctl VIDIOC_REQBUFS can only succeed if the
+     * V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS capability is set. Otherwise ioctl
+     * VIDIOC_REQBUFS will return the EBUSY error code. If
+     * V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS is set, then these buffers are
+     * orphanedand will be freed when they are unmapped or when the exported
+     * DMABUF fds are closed. A count value of zero frees or orphans all
+     * buffers, after aborting or finishing any DMA in progress, an implicit
+     * VIDIOC_STREAMOFF.
+     */
+
+    /* TODO support V4L2_BUF_CAP_SUPPORTS_ORPHANED_BUFS */
+
+    ret = ioctl(fd, VIDIOC_REQBUFS, &reqbuf);
+    if (ret == -1) {
+        if (errno == EBUSY) {
+            g_critical("%s: EBUSY: buffers for %s still mapped or exported!\n"
+                       , __func__, v4l2_buf_type_name(type));
+        } else {
+            g_printerr("VIDIOC_REQBUFS failed: %s (%d)\n"
+                       , g_strerror(errno), errno);
+        }
+        goto out;
+    }
+    g_debug("%s: VIDIOC_REQBUFS capabilities(0x%x) granted(%d)"
+            , __func__, reqbuf.capabilities, reqbuf.count);
+
+out:
+    return ret;
+}
+
+int v4l2_resource_create(struct stream *s, enum v4l2_buf_type type,
+                         enum virtio_video_mem_type mem_type,
+                         struct resource *res)
+{
+    struct v4l2_requestbuffers reqbuf;
+    int ret;
+    g_debug("%s: v4l2_buf_type: %s", __func__, v4l2_buf_type_name(type));
+
+    memset(&reqbuf, 0, sizeof(reqbuf));
+    reqbuf.type = type;
+    reqbuf.count = 1;
+
+    if (is_output_queue(type)) {
+        reqbuf.count = s->output_bufcount + 1;
+    } else if (is_capture_queue(type)) {
+        reqbuf.count = s->capture_bufcount + 1;
+    }
+
+    if (mem_type == VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES) {
+        reqbuf.memory = V4L2_MEMORY_USERPTR;
+    } else if (mem_type == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT) {
+        /* TODO */
+        g_error("%s: VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT not implemented\n"
+                , __func__);
+        ret = -EINVAL;
+        goto out;
+    }
+
+    ret = ioctl(s->fd, VIDIOC_REQBUFS, &reqbuf);
+    if (ret == -1) {
+        g_printerr("VIDIOC_REQBUFS failed: %s (%d)\n"
+                   , g_strerror(errno), errno);
+        goto out;
+    }
+    g_debug("%s: VIDIOC_REQBUFS capabilities(0x%x) granted(%d)!"
+            , __func__, reqbuf.capabilities, reqbuf.count);
+
+    if (is_output_queue(type)) {
+        s->output_bufcount = reqbuf.count;
+        res->v4l2_index = reqbuf.count - 1;
+    } else if (is_capture_queue(type)) {
+        s->capture_bufcount = reqbuf.count;
+        res->v4l2_index = reqbuf.count - 1;
+    }
+
+    res->type = type;
+out:
+    return ret;
+}
+
+/* timestamp in nsecs */
+void convert_to_timeval(uint64_t timestamp, struct timeval *t)
+{
+    uint64_t f_nsecs;
+
+    uint64_t nsecs;
+
+    /* convert to seconds */
+    t->tv_sec = timestamp / 1000000000;
+
+    /* deal with fraction of a second */
+    f_nsecs = t->tv_sec * 1000000000;
+    t->tv_usec = (timestamp - f_nsecs) / 1000;
+
+    /* sanity check above conversion */
+    nsecs = t->tv_sec * 1000000000;
+    nsecs += (t->tv_usec * 1000);
+
+    if (timestamp != nsecs) {
+        g_critical("%s: timestamp != nsecs", __func__);
+    }
+}
+
+int ioctl_streamon(struct stream *s, enum v4l2_buf_type type)
+{
+    int ret = 0;
+    ret = ioctl(s->fd, VIDIOC_STREAMON, &type);
+    if (ret < 0) {
+        g_printerr("VIDIOC_STREAMON failed: fd=(%d) buf type=%s %s (%d).\n"
+                   , s->fd, v4l2_buf_type_name(type), g_strerror(errno), errno);
+    } else {
+        g_debug("%s: VIDIOC_STREAMON OK fd=(%d) buf type: %s"
+                , __func__, s->fd, v4l2_buf_type_name(type));
+        if (is_output_queue(type)) {
+            s->output_streaming = true;
+        }
+        if (is_capture_queue(type)) {
+            s->capture_streaming = true;
+        }
+    }
+    return ret;
+}
+
+int ioctl_streamoff(struct stream *s, enum v4l2_buf_type type)
+{
+    int ret = 0;
+    ret = ioctl(s->fd, VIDIOC_STREAMOFF, &type);
+    if (ret < 0) {
+        g_printerr("VIDIOC_STREAMOFF failed: fd=(%d) buf type=%s: %s (%d).\n"
+                   , s->fd, v4l2_buf_type_name(type), g_strerror(errno), errno);
+    } else {
+        g_debug("%s: VIDIOC_STREAMOFF OK buf type: %s"
+                , __func__, v4l2_buf_type_name(type));
+
+        if (is_output_queue(type)) {
+            s->output_streaming = false;
+        }
+        if (is_capture_queue(type)) {
+            s->capture_streaming = false;
+        }
+
+        /*
+         * if either queue has STREAMOFF applied, then we enter STOPPED
+         * Assumes that s->mutex is held by calling function
+         */
+        s->stream_state = STREAM_STOPPED;
+        g_cond_signal(&s->stream_cond);
+    }
+    return ret;
+}
+
+/* activate streaming on both queues */
+int v4l2_streamon(struct v4l2_device *dev, enum v4l2_buf_type type,
+                  struct stream *s)
+{
+    int ret = 0;
+    bool is_mplane = video_is_mplane(type);
+    enum v4l2_buf_type type2;
+
+    if (!s->subscribed_events) {
+        /* subscribe for SOURCE_CHANGE event */
+        if (dev->sup_dyn_res_switching) {
+            ret = v4l2_subscribe_event(s, V4L2_EVENT_SOURCE_CHANGE, 0);
+            if (ret < 0) {
+                g_printerr("%s: V4L2_EVENT_SOURCE_CHANGE failed\n", __func__);
+            }
+        }
+        /* subscribe for EOS event */
+        ret = v4l2_subscribe_event(s, V4L2_EVENT_EOS, 0);
+        if (ret < 0) {
+            g_printerr("%s: V4L2_EVENT_EOS failed\n", __func__);
+        }
+        s->subscribed_events = true;
+    }
+
+    switch (type) {
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+    case V4L2_BUF_TYPE_META_OUTPUT:
+        if (s->output_streaming == false) {
+            ret |= ioctl_streamon(s, type);
+        }
+
+        if (s->capture_streaming == false) {
+            type2 = is_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE :
+                V4L2_BUF_TYPE_VIDEO_CAPTURE;
+            ret |= ioctl_streamon(s, type2);
+        }
+        break;
+
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+    case V4L2_BUF_TYPE_META_CAPTURE:
+        if (s->capture_streaming == false) {
+            ret |= ioctl_streamon(s, type);
+        }
+        if (s->output_streaming == false) {
+            type2 = is_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE :
+                V4L2_BUF_TYPE_VIDEO_OUTPUT;
+            ret |= ioctl_streamon(s, type2);
+        }
+        break;
+
+    default:
+        g_printerr("%s: unknown v4l2 buffer type!", __func__);
+        ret = EINVAL;
+    }
+
+    if (s->stream_state != STREAM_DRAINING) {
+        s->stream_state = STREAM_STREAMING;
+        g_cond_signal(&s->stream_cond);
+    }
+
+    return ret;
+}
+
+int v4l2_streamoff(enum v4l2_buf_type type, struct stream *s)
+{
+    int ret = 0;
+    bool is_mplane = video_is_mplane(type);
+    enum v4l2_buf_type type2;
+
+    switch (type) {
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
+    case V4L2_BUF_TYPE_VIDEO_OUTPUT:
+    case V4L2_BUF_TYPE_META_OUTPUT:
+        if (s->output_streaming == true) {
+            ret |= ioctl_streamoff(s, type);
+        }
+
+        if (s->capture_streaming == true) {
+            type2 = is_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
+                : V4L2_BUF_TYPE_VIDEO_CAPTURE;
+            ret |= ioctl_streamoff(s, type2);
+        }
+        break;
+
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
+    case V4L2_BUF_TYPE_VIDEO_CAPTURE:
+    case V4L2_BUF_TYPE_META_CAPTURE:
+        if (s->capture_streaming == true) {
+            ret |= ioctl_streamoff(s, type);
+        }
+        if (s->output_streaming == true) {
+            type2 = is_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+                : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+            ret |= ioctl_streamoff(s, type2);
+        }
+        break;
+
+    default:
+        g_printerr("%s: unknown v4l2 buffer type!", __func__);
+        ret = EINVAL;
+    }
+
+    return ret;
+}
+
+int v4l2_subscribe_event(struct stream *s,
+                         uint32_t event_type, uint32_t id)
+{
+    int ret = 0;
+    struct v4l2_event_subscription sub;
+
+    memset(&sub, 0, sizeof(sub));
+    sub.type = event_type;
+    sub.id = 0;
+
+    if (event_type == V4L2_EVENT_SOURCE_CHANGE) {
+        sub.id = id;
+    }
+
+    ret = ioctl(s->fd, VIDIOC_SUBSCRIBE_EVENT, &sub);
+    if (ret < 0) {
+        g_printerr("%s: VIDIOC_SUBSCRIBE_EVENT failed", __func__);
+        return ret;
+    }
+
+    g_debug("%s event(0x%x) OK!", __func__, event_type);
+
+    return ret;
+}
+
+void v4l2_print_event(const struct v4l2_event *ev)
+{
+    g_debug("%s: %ld.%06ld: event %u, pending %u: ", __func__,
+            ev->timestamp.tv_sec, ev->timestamp.tv_nsec / 1000,
+            ev->sequence, ev->pending);
+    switch (ev->type) {
+    case V4L2_EVENT_VSYNC:
+        g_debug("%s: vsync\n", __func__);
+        break;
+    case V4L2_EVENT_EOS:
+        g_debug("%s: eos\n", __func__);
+        break;
+    case V4L2_EVENT_CTRL:
+        g_debug("%s: eos\n", __func__);
+        break;
+    case V4L2_EVENT_FRAME_SYNC:
+        g_debug("%s: frame_sync %d\n", __func__, ev->u.frame_sync.frame_sequence);
+        break;
+    case V4L2_EVENT_SOURCE_CHANGE:
+        g_debug("%s: source_change!: pad/input=%d changes: %x\n"
+                , __func__, ev->id, ev->u.src_change.changes);
+         break;
+    case V4L2_EVENT_MOTION_DET:
+        if (ev->u.motion_det.flags & V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ) {
+            g_debug("%s: motion_det frame %d, regions 0x%x\n", __func__,
+                    ev->u.motion_det.frame_sequence,
+                    ev->u.motion_det.region_mask);
+        } else {
+            g_debug("%s: motion_det regions 0x%x\n", __func__
+                    , ev->u.motion_det.region_mask);
+        }
+        break;
+    default:
+        if (ev->type >= V4L2_EVENT_PRIVATE_START) {
+            g_debug("unknown private event (%08x)\n", ev->type);
+        } else {
+            g_debug("unknown event (%08x)\n", ev->type);
+        }
+        break;
+    }
+}
+
+int v4l2_queue_buffer(int fd, enum v4l2_buf_type type,
+                      struct virtio_video_resource_queue *qcmd,
+                      struct resource *res, struct stream *s,
+                      struct v4l2_device *dev)
+{
+    struct v4l2_buffer vbuf;
+    int ret = 0;
+
+    memset(&vbuf, 0, sizeof(vbuf));
+    vbuf.index = res->v4l2_index;
+
+    vbuf.type = type;
+    vbuf.field = V4L2_FIELD_NONE;
+    vbuf.flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+
+    g_debug("%s: type=%s index=%d", __func__,
+            v4l2_buf_type_name(type), vbuf.index);
+
+    convert_to_timeval(le64toh(qcmd->timestamp), &vbuf.timestamp);
+
+    /* if using GUEST_PAGES queued using USERPTR mechanism */
+    vbuf.memory = V4L2_MEMORY_USERPTR;
+
+    if (video_is_mplane(type)) {
+        /* for mplane length field is number of elements in planes array */
+        vbuf.length = res->vio_resource.num_planes;
+        vbuf.m.planes = g_malloc0(sizeof(struct v4l2_plane)
+                                  * res->vio_resource.num_planes);
+
+        for (int i = 0; i < vbuf.length; i++) {
+            vbuf.m.planes[i].m.userptr = (unsigned long)res->iov[i].iov_base;
+            vbuf.m.planes[i].length = (unsigned long)res->iov[i].iov_len;
+        }
+    } else {
+        /* m is a union of userptr, *planes and fd */
+        vbuf.m.userptr = (unsigned long)res->iov[0].iov_base;
+        vbuf.length = res->iov[0].iov_len;
+        g_debug("%s: iov_base = 0x%p", __func__, res->iov[0].iov_base);
+        g_debug("%s: iov_len = 0x%lx", __func__, res->iov[0].iov_len);
+    }
+
+    ret = v4l2_streamon(dev, type, s);
+    if (ret < 0) {
+        g_printerr("v4l2_streamon failed (%d)", ret);
+        /* only print error, as v4l2_streamon() does both queues */
+    }
+
+    ret = ioctl(fd, VIDIOC_QBUF, &vbuf);
+    if (ret < 0) {
+        qcmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
+        g_printerr("Unable to QBUF: %s (%d).\n", g_strerror(errno), errno);
+        return ret;
+    }
+
+    res->queued = true;
+    if (video_is_mplane(type)) {
+        g_free(vbuf.m.planes);
+    }
+
+    g_debug("%s: Queued resource-id(%d) buf_type=%s v4l2_index(%d) "
+            "virtio_queue(0x%x)", __func__, res->vio_resource.resource_id,
+            v4l2_buf_type_name(type), res->v4l2_index,
+            res->vio_resource.queue_type);
+
+    return ret;
+
+}
+
+int v4l2_dequeue_buffer(int fd, enum v4l2_buf_type type,
+                        struct stream *s)
+{
+    struct v4l2_buffer vbuf;
+    int ret = 0;
+    struct resource *r;
+    struct virtio_video_resource_queue_resp resp;
+    struct vu_video_ctrl_command *vio_cmd;
+
+    memset(&vbuf, 0, sizeof(vbuf));
+
+    vbuf.type = type;
+    vbuf.memory =  V4L2_MEMORY_USERPTR;
+
+    vbuf.field = V4L2_FIELD_NONE;
+    vbuf.flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+
+    if (video_is_mplane(type)) {
+        /* for mplane length field is number of elements in planes array */
+        vbuf.length = VIRTIO_VIDEO_MAX_PLANES;
+        vbuf.m.planes = g_malloc0(sizeof(struct v4l2_plane)
+                                  * VIRTIO_VIDEO_MAX_PLANES);
+
+        g_debug("%s: mplane allocating planes array", __func__);
+    }
+
+    ret = ioctl(fd, VIDIOC_DQBUF, &vbuf);
+    if (ret < 0) {
+        g_printerr("Unable to DQBUF: %s (%d).\n", g_strerror(errno), errno);
+        return ret;
+    }
+
+    g_debug("%s: VIDIOC_DQBUF OK index(%d)!", __func__, vbuf.index);
+
+    if (video_is_mplane(type)) {
+        g_free(vbuf.m.planes);
+    }
+
+    r = find_resource_by_v4l2index(s, type, vbuf.index);
+    if (!r) {
+        g_printerr("%s: Can't find resource for dequeued buffer!", __func__);
+        return -EINVAL;
+    }
+
+    r->queued = false;
+    vio_cmd = r->vio_q_cmd;
+
+    resp.hdr.stream_id = r->stream_id;
+    resp.hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
+    resp.timestamp = htole64(r->vio_res_q.timestamp);
+
+    /* encoder only */
+    resp.size = htole32(vbuf.bytesused);
+
+    if (vbuf.flags & V4L2_BUF_FLAG_LAST &&
+        s->stream_state == STREAM_DRAINING) {
+        resp.flags |= VIRTIO_VIDEO_BUFFER_FLAG_EOS;
+        s->stream_state = STREAM_STOPPED;
+        g_cond_signal(&s->stream_cond);
+    }
+
+    if (vbuf.flags & V4L2_BUF_FLAG_KEYFRAME) {
+        resp.flags |= VIRTIO_VIDEO_BUFFER_FLAG_IFRAME;
+    }
+    if (vbuf.flags & V4L2_BUF_FLAG_PFRAME) {
+        resp.flags |= VIRTIO_VIDEO_BUFFER_FLAG_PFRAME;
+    }
+    if (vbuf.flags & V4L2_BUF_FLAG_BFRAME) {
+        resp.flags |= VIRTIO_VIDEO_BUFFER_FLAG_PFRAME;
+    }
+
+    if (vbuf.flags & V4L2_BUF_FLAG_ERROR) {
+        resp.flags |= VIRTIO_VIDEO_BUFFER_FLAG_ERR;
+        g_critical("%s: V4L2_BUF_FLAG_ERROR\n", __func__);
+    }
+
+    g_debug("%s: Send queue_buffer reply: stream_id=%d type=0x%x "
+            "flags=0x%x resource_id=%d t=%llx", __func__,
+            resp.hdr.stream_id, resp.hdr.type, resp.flags,
+            r->vio_resource.resource_id, resp.timestamp);
+
+    send_ctrl_response(vio_cmd, (uint8_t *) &resp,
+                       sizeof(struct virtio_video_resource_queue_resp));
+
+    vio_cmd->finished = true;
+    free_resource_mem(r);
+
+    return ret;
+}
+
+int v4l2_video_get_selection(int fd, enum v4l2_buf_type type,
+                             struct v4l2_selection *sel)
+{
+    int ret = 0;
+
+    if (!sel) {
+        return -EINVAL;
+    }
+
+    memset(sel, 0, sizeof(struct v4l2_selection));
+
+    sel->type = type;
+
+    if (is_capture_queue(type)) {
+        sel->target = V4L2_SEL_TGT_COMPOSE;
+    } else if (is_output_queue(type)) {
+        sel->target = V4L2_SEL_TGT_CROP;
+    }
+
+    ret = ioctl(fd, VIDIOC_G_SELECTION, sel);
+    if (ret < 0) {
+        g_printerr("Unable to get selection: %s (%d).\n", g_strerror(errno),
+               errno);
+        return ret;
+    }
+
+    g_debug("%s: VIDIOC_G_SELECTION: fd=(%d) %s: left=(%d) "
+            "top=(%d) width=(%d) height=(%d)",
+            __func__, fd, v4l2_buf_type_name(type), sel->r.left,
+            sel->r.top, sel->r.width, sel->r.height);
+
+    return ret;
+}
+
+int v4l2_video_set_selection(int fd, enum v4l2_buf_type type,
+                             struct v4l2_selection *sel)
+{
+    int ret = 0;
+
+    if (!sel) {
+        return -EINVAL;
+    }
+
+    sel->type = type;
+    sel->target = V4L2_SEL_TGT_COMPOSE;
+    /* flags 0 - the driver can adjust the rect size freely */
+    sel->flags = 0;
+
+    ret = ioctl(fd, VIDIOC_S_SELECTION, sel);
+    if (ret < 0) {
+        g_printerr("Unable to set selection: fd=(%d) left=(%d) top=(%d)"
+                   "width=(%d) height=(%d): %s (%d).\n", fd, sel->r.left,
+                   sel->r.top, sel->r.width, sel->r.height, g_strerror(errno),
+                   errno);
+        return ret;
+    }
+
+    g_debug("%s: VIDIOC_S_SELECTION: fd=(%d) left=(%d) "
+            "top=(%d) width=(%d) height=(%d)",
+            __func__, fd, sel->r.left, sel->r.top, sel->r.width, sel->r.height);
+
+    return ret;
+}
+
+int v4l2_issue_cmd(int fd,  uint32_t cmd, uint32_t flags)
+{
+    int ret = 0;
+    struct v4l2_decoder_cmd decoder_cmd;
+    memset(&decoder_cmd, 0, sizeof(struct v4l2_decoder_cmd));
+
+    decoder_cmd.cmd = cmd;
+    decoder_cmd.flags = flags;
+
+    ret = ioctl(fd, VIDIOC_DECODER_CMD, &decoder_cmd);
+    if (ret < 0) {
+        g_printerr("%s: VIDIOC_DECODER_CMD(%d) failed fd(%d): %s: (%d).\n"
+                   , __func__, cmd, fd, g_strerror(errno), errno);
+        return ret;
+    }
+
+    g_debug("%s: VIDIOC_DECODER_CMD(%d) fd(%d)OK\n", __func__, cmd, fd);
+
+    return ret;
+}
+
+int v4l2_video_get_param(int fd, enum v4l2_buf_type type,
+                          struct v4l2_streamparm *sparam)
+{
+    int ret = 0;
+
+    if (!sparam) {
+        return -EINVAL;
+    }
+
+    memset(sparam, 0, sizeof(struct v4l2_streamparm));
+    sparam->type = type;
+
+    ret = ioctl(fd, VIDIOC_G_PARM, sparam);
+    if (ret < 0) {
+        g_printerr("Unable to VIDIOC_G_PARAM: %s (%d).\n", g_strerror(errno),
+               errno);
+        return ret;
+    }
+
+    g_debug("%s: VIDIOC_G_PARM timeperframe (%d/%d)", __func__,
+            sparam->parm.capture.timeperframe.numerator,
+            sparam->parm.capture.timeperframe.denominator);
+
+    return ret;
+}
+
+int v4l2_video_get_format(int fd, enum v4l2_buf_type type,
+                          struct v4l2_format *fmt)
+{
+    unsigned int i;
+    int ret;
+
+    if (!fmt) {
+        return -EINVAL;
+    }
+
+    memset(fmt, 0, sizeof(struct v4l2_format));
+    fmt->type = type;
+
+    ret = ioctl(fd, VIDIOC_G_FMT, fmt);
+    if (ret < 0) {
+        g_printerr("Unable to get format: %s (%d).\n", g_strerror(errno),
+               errno);
+        return ret;
+    }
+
+    if (video_is_mplane(type)) {
+        g_print("Video format: %s (%08x) %ux%u field %s, %u planes:\n",
+                v4l2_format_name(fmt->fmt.pix_mp.pixelformat),
+                fmt->fmt.pix_mp.pixelformat,
+                fmt->fmt.pix_mp.width, fmt->fmt.pix_mp.height,
+                v4l2_field_name(fmt->fmt.pix_mp.field),
+                fmt->fmt.pix_mp.num_planes);
+
+        for (i = 0; i < fmt->fmt.pix_mp.num_planes; i++) {
+            g_print(" * Stride %u, buffer size %u\n",
+                   fmt->fmt.pix_mp.plane_fmt[i].bytesperline,
+                   fmt->fmt.pix_mp.plane_fmt[i].sizeimage);
+        }
+    } else if (video_is_meta(type)) {
+        g_print("Meta-data format: %s (%08x) buffer size %u\n",
+                v4l2_format_name(fmt->fmt.meta.dataformat),
+                fmt->fmt.meta.dataformat,
+                fmt->fmt.meta.buffersize);
+    } else {
+        g_print("Video format: %s (%08x) %ux%u (stride %u) field %s "
+                "buffer size %u\n",
+                v4l2_format_name(fmt->fmt.pix.pixelformat),
+                fmt->fmt.pix.pixelformat,
+                fmt->fmt.pix.width, fmt->fmt.pix.height,
+                fmt->fmt.pix.bytesperline,
+                v4l2_field_name(fmt->fmt.pix_mp.field),
+                fmt->fmt.pix.sizeimage);
+    }
+
+    return 0;
+}
+
+int v4l2_video_get_control(int fd , uint32_t control, int32_t *value)
+{
+    int ret = 0;
+    struct v4l2_control ctrl;
+
+    g_debug("%s:%d", __func__, __LINE__);
+
+    ctrl.id = control;
+
+    ret = ioctl(fd, VIDIOC_G_CTRL, &ctrl);
+    if (ret < 0) {
+        g_printerr("Unable to get control: %s (%d).\n", g_strerror(errno),
+               errno);
+        return ret;
+    }
+
+    *value = ctrl.value;
+    g_debug("%s: ctrl=0x%x value=0x%x", __func__, control, *value);
+
+    return ret;
+}
+
+int v4l2_video_set_format(int fd, enum v4l2_buf_type type,
+                            struct virtio_video_params *p)
+{
+    struct v4l2_format fmt;
+    int ret = 0;
+    unsigned int i;
+    uint32_t pixfmt;
+
+    if (!p) {
+        return -EINVAL;
+    }
+
+    memset(&fmt, 0, sizeof fmt);
+    fmt.type = type;
+    pixfmt = virtio_video_format_to_v4l2(le32toh(p->format));
+
+    if (video_is_mplane(type)) {
+        fmt.fmt.pix_mp.width = le32toh(p->frame_width);
+        fmt.fmt.pix_mp.height = le32toh(p->frame_height);
+        fmt.fmt.pix_mp.pixelformat = pixfmt;
+        /*
+         * V4L2_FIELD_NONE - matches what Linux frontend driver does in
+         * virtio_video_format_from_info()
+         */
+        fmt.fmt.pix_mp.field = V4L2_FIELD_NONE;
+        /*fmt.fmt.pix_mp.num_planes = info->n_planes;*/
+        fmt.fmt.pix_mp.num_planes = le32toh(p->num_planes);
+        fmt.fmt.pix_mp.flags = 0;
+
+        for (i = 0; i < le32toh(p->num_planes); i++) {
+            fmt.fmt.pix_mp.plane_fmt[i].bytesperline =
+                le32toh(p->plane_formats[i].stride);
+            fmt.fmt.pix_mp.plane_fmt[i].sizeimage =
+                le32toh(p->plane_formats[i].plane_size);
+        }
+    } else if (video_is_splane(type)) {
+        fmt.fmt.pix.width = le32toh(p->frame_width);
+        fmt.fmt.pix.height = le32toh(p->frame_height);
+        fmt.fmt.pix.pixelformat = pixfmt;
+        fmt.fmt.pix.field = V4L2_FIELD_NONE;
+        fmt.fmt.pix.bytesperline = le32toh(p->plane_formats[0].stride);
+        fmt.fmt.pix.sizeimage = le32toh(p->plane_formats[0].plane_size);
+        fmt.fmt.pix.priv = V4L2_PIX_FMT_PRIV_MAGIC;
+        fmt.fmt.pix.flags = 0;
+    }
+
+    ret = ioctl(fd, VIDIOC_S_FMT, &fmt);
+    if (ret < 0) {
+        g_printerr("Unable to set format: %s (%d).\n", g_strerror(errno),
+               errno);
+    }
+    return ret;
+}
+
+int v4l2_set_pixel_format(int fd, enum v4l2_buf_type buf_type,
+                          uint32_t pixelformat)
+{
+    int ret = 0;
+    struct v4l2_format cur_fmt;
+
+    g_debug("%s: buf_type=0x%x pixelformat=0x%x", __func__,
+            buf_type, pixelformat);
+
+    /* get the currently set format */
+    ret = v4l2_video_get_format(fd, buf_type, &cur_fmt);
+    if (ret < 0) {
+        g_printerr("%s: v4l2_video_get_format() failed\n", __func__);
+        return ret;
+    }
+
+    /* keep defaults and set correct pixel format */
+    if (video_is_mplane(cur_fmt.type)) {
+        g_print("%s: Format is mplane\n", __func__);
+        cur_fmt.fmt.pix_mp.pixelformat = pixelformat;
+    } else if (video_is_splane(cur_fmt.type)) {
+        g_print("%s: Format is splane\n", __func__);
+        cur_fmt.fmt.pix.pixelformat = pixelformat;
+    }
+
+    ret = ioctl(fd, VIDIOC_S_FMT, &cur_fmt);
+    if (ret < 0) {
+        g_printerr("Unable to set format: %s (%d).\n", g_strerror(errno),
+                   errno);
+    }
+
+    return ret;
+}
+
+int video_enum_formats(struct v4l2_device *dev, enum v4l2_buf_type type,
+                       GList **p_fmt_list, bool only_enum_fmt)
+{
+    struct v4l2_fmtdesc fmt;
+    struct video_format *vid_fmt = NULL;
+    GList *fmt_list = NULL;
+    unsigned int index;
+    int ret = 0;
+
+    if (!dev) {
+        return -EINVAL;
+    }
+
+    for (index = 0; ; ++index) {
+        memset(&fmt, 0, sizeof fmt);
+        fmt.index = index;
+        fmt.type = type;
+        ret = ioctl(dev->fd, VIDIOC_ENUM_FMT, &fmt);
+
+        if (ret < 0) {
+            if (errno == EINVAL) {
+                ret = 0;
+            } else{
+                g_printerr("%s: VIDIOC_ENUM_FMT failed %s\n", __func__,
+                           g_strerror(errno));
+            }
+        break;
+        }
+
+        /* do some driver sanity checks */
+        if (index != fmt.index) {
+            g_warning("v4l2 driver modified index %u.\n", fmt.index);
+        }
+        if (type != fmt.type) {
+            g_warning("v4l2 driver modified type %u.\n", fmt.type);
+        }
+        g_debug("\tFormat %u: %s (%08x)", index,
+               v4l2_format_name(fmt.pixelformat), fmt.pixelformat);
+        g_debug("\tType: %s (%u)", v4l2_buf_type_name(fmt.type),
+               fmt.type);
+        g_debug("\tName: %.32s", fmt.description);
+        g_debug("\tFlags: 0x%x", fmt.flags);
+
+        if (fmt.flags & V4L2_FMT_FLAG_DYN_RESOLUTION &&
+            fmt.flags & V4L2_FMT_FLAG_COMPRESSED) {
+            g_print("dynamic resolution switching supported\n");
+            dev->sup_dyn_res_switching = true;
+        }
+
+        /* test if pixelformat converts to virtio */
+        if (!virtio_video_v4l2_format_to_virtio(fmt.pixelformat)) {
+            g_info("Skipping Format %s (%08x) - no virtio-video equivalent"
+                      , v4l2_format_name(fmt.pixelformat), fmt.pixelformat);
+            continue;
+        }
+
+        /* allocate video_format struct */
+        vid_fmt = g_new0(struct video_format, 1);
+
+        /* keep a copy of v4l2 struct */
+        memcpy(&vid_fmt->fmt, &fmt, sizeof(struct v4l2_fmtdesc));
+
+        /* add it to linked list */
+        fmt_list = g_list_append(fmt_list, vid_fmt);
+
+        if (!only_enum_fmt) {
+            /* pass video_format to enum_frame_sizes */
+            ret = video_enum_frame_sizes(dev, fmt.pixelformat,
+                                         &vid_fmt->vid_fmt_frm_l);
+            if (ret < 0) {
+                g_printerr("video_enum_frame_sizes failed\n");
+            }
+
+            /* convert to virtio format */
+            v4l2_to_virtio_fmtdesc(dev, vid_fmt, type);
+        }
+
+        /* determine type of v4l2 device */
+        v4l2_set_device_type(dev, type, &fmt);
+    }
+
+    if (ret == 0) {
+        g_print("%s: Enumerated %d formats on v4l2 %s queue", __func__
+                , index, v4l2_buf_type_name(type));
+        g_print(" %d formats are representable by virtio-video\n"
+                , g_list_length(fmt_list));
+        if (!only_enum_fmt)
+            g_print("%s: Enumerated %d frame sizes\n", __func__,
+                    g_list_length(vid_fmt->vid_fmt_frm_l));
+
+        *p_fmt_list = fmt_list;
+    }
+
+    return ret;
+}
+
+void video_free_frame_intervals(GList *frm_intervals_l)
+{
+    GList *l;
+    struct video_format_frame_rates *vid_fmt_frm_rate;
+    for (l = frm_intervals_l; l != NULL; l = l->next) {
+        vid_fmt_frm_rate = l->data;
+        g_free(vid_fmt_frm_rate);
+    }
+}
+
+void video_free_frame_sizes(GList *frm_sz_l)
+{
+    GList *l;
+    struct video_format_frame *vid_frame;
+    for (l = frm_sz_l; l != NULL; l = l->next) {
+        vid_frame = l->data;
+        if (vid_frame->frm_rate_l) {
+            video_free_frame_intervals(vid_frame->frm_rate_l);
+        }
+        g_free(vid_frame);
+    }
+}
+
+void video_free_formats(GList **fmt_l)
+{
+    GList *l;
+    struct video_format *vid_fmt;
+
+    for (l = *fmt_l; l != NULL; l = l->next) {
+        vid_fmt = l->data;
+        if (vid_fmt->vid_fmt_frm_l) {
+            video_free_frame_sizes(vid_fmt->vid_fmt_frm_l);
+        }
+
+        g_free(vid_fmt);
+    }
+}
+
+
+static GByteArray *iterate_frame_rate_list(GByteArray *resp, GList *frm_rate_l)
+{
+    struct video_format_frame_rates *vid_fmt_frm_rate;
+
+    /* iterate frame_rate list */
+    for (; frm_rate_l != NULL; frm_rate_l = frm_rate_l->next) {
+        vid_fmt_frm_rate = frm_rate_l->data;
+
+        resp = g_byte_array_append(resp,
+                                   (guint8 *) &vid_fmt_frm_rate->frame_rates,
+                                   sizeof(struct virtio_video_format_range));
+        }
+    return resp;
+}
+
+static GByteArray *iterate_format_frame_list(GByteArray *resp, GList *fmt_frm_l)
+{
+    struct video_format_frame *vid_fmt_frm;
+    GList *frm_rate_l = NULL;
+
+    /* iterate format_frame list */
+    for (; fmt_frm_l != NULL; fmt_frm_l = fmt_frm_l->next) {
+        vid_fmt_frm = fmt_frm_l->data;
+
+        if (!vid_fmt_frm->frm_rate_l) {
+            vid_fmt_frm->frame.num_rates = htole32(0);
+        } else {
+            frm_rate_l = vid_fmt_frm->frm_rate_l;
+            vid_fmt_frm->frame.num_rates = htole32(g_list_length(frm_rate_l));
+
+        }
+
+        g_debug("%s: num_rates(%d)", __func__,
+                le32toh(vid_fmt_frm->frame.num_rates));
+
+        resp = g_byte_array_append(resp,
+                                   (guint8 *) &vid_fmt_frm->frame,
+                                   sizeof(struct virtio_video_format_frame));
+
+        if (frm_rate_l) {
+            resp = iterate_frame_rate_list(resp, frm_rate_l);
+        }
+    }
+
+    return resp;
+}
+
+static GByteArray *iterate_format_desc_list(GByteArray *resp, GList *fmt_desc_l)
+{
+    struct video_format *vid_fmt;
+    GList *fmt_frm_l = NULL;
+
+    for (; fmt_desc_l != NULL; fmt_desc_l = fmt_desc_l->next) {
+        vid_fmt = fmt_desc_l->data;
+
+        /* does video_format have a list of format_frame? */
+        if (!vid_fmt->vid_fmt_frm_l) {
+            vid_fmt->desc.num_frames = htole32(0);
+        } else {
+            fmt_frm_l = vid_fmt->vid_fmt_frm_l;
+            vid_fmt->desc.num_frames = htole32(g_list_length(fmt_frm_l));
+        }
+
+        g_debug("%s: num_frames(%d)", __func__,
+                le32toh(vid_fmt->desc.num_frames));
+
+        resp = g_byte_array_append(resp,
+                                   (guint8 *) &vid_fmt->desc,
+                                   sizeof(struct virtio_video_format_desc));
+
+        if (fmt_frm_l) {
+            resp = iterate_format_frame_list(resp, fmt_frm_l);
+        }
+    }
+
+    return resp;
+}
+
+GByteArray *create_query_cap_resp(struct virtio_video_query_capability *qcmd,
+                           GList **fmt_l, GByteArray *resp)
+{
+    GList *fmt_desc_l;
+    struct virtio_video_query_capability_resp cap_resp;
+
+    fmt_desc_l = *fmt_l;
+
+    cap_resp.hdr.type = VIRTIO_VIDEO_RESP_OK_QUERY_CAPABILITY;
+    cap_resp.hdr.stream_id = qcmd->hdr.stream_id;
+    cap_resp.num_descs = htole32(g_list_length(fmt_desc_l));
+
+    assert(le32toh(cap_resp.num_descs) < MAX_FMT_DESCS);
+
+    resp = g_byte_array_append(resp, (guint8 *) &cap_resp, sizeof(cap_resp));
+    resp = iterate_format_desc_list(resp, fmt_desc_l);
+
+    return resp;
+}
+
+void v4l2_backend_free(struct v4l2_device *dev)
+{
+    if (dev && dev->opened) {
+        close(dev->fd);
+    }
+    g_free(dev);
+}
+
+struct v4l2_device *v4l2_backend_init(const gchar *devname)
+{
+    struct v4l2_device *dev;
+    int ret = 0;
+    GList *vid_output_fmt_l = NULL;
+    GList *vid_capture_fmt_l = NULL;
+    enum v4l2_buf_type buf_type;
+
+    if (!devname) {
+        return NULL;
+    }
+
+    dev = g_malloc0(sizeof(struct v4l2_device));
+
+    /* open the device */
+    dev->fd = v4l2_open(devname);
+    if (dev->fd < 0) {
+        g_printerr("v4l2_open() failed!\n");
+        goto err;
+    }
+
+    dev->opened = 1;
+    dev->devname = devname;
+
+    ret = video_querycap(dev);
+
+    buf_type = dev->has_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+        : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+    /* enumerate coded formats on OUTPUT */
+    ret = video_enum_formats(dev, buf_type,
+                             &vid_output_fmt_l, true);
+    if (ret < 0) {
+        g_printerr("video_enum_formats() failed OUTPUT\n");
+        goto err;
+    }
+
+    buf_type = dev->has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
+        : V4L2_BUF_TYPE_VIDEO_CAPTURE;
+
+    /* enumerate coded formats on CAPTURE */
+    ret = video_enum_formats(dev, buf_type,
+                             &vid_capture_fmt_l, true);
+    if (ret < 0) {
+        g_printerr("video_enum_formats() failed CAPTURE\n");
+        goto err2;
+    }
+
+    if (dev->dev_type & STATEFUL_ENCODER)
+        g_print("%s: %s is a stateful encoder (0x%x)!\n", __func__,
+                devname, dev->dev_type);
+
+    if (dev->dev_type & STATEFUL_DECODER)
+        g_print("%s: %s is a stateful decoder (0x%x)!\n", __func__,
+                devname, dev->dev_type);
+
+    video_free_formats(&vid_output_fmt_l);
+    video_free_formats(&vid_capture_fmt_l);
+
+    if (!(dev->dev_type & STATEFUL_ENCODER ||
+          dev->dev_type & STATEFUL_DECODER)) {
+        g_printerr("v4l2 device not supported! v4l2 backend only supports "
+                   "stateful codec devices currently(%d)!\n", dev->dev_type);
+        goto err3;
+    }
+
+    g_debug("%s: success!\n", __func__);
+    return dev;
+
+err3:
+    video_free_formats(&vid_capture_fmt_l);
+err2:
+    video_free_formats(&vid_output_fmt_l);
+err:
+    v4l2_backend_free(dev);
+    return NULL;
+}
diff --git a/tools/vhost-user-video/v4l2_backend.h b/tools/vhost-user-video/v4l2_backend.h
new file mode 100644
index 0000000000..e3617256b3
--- /dev/null
+++ b/tools/vhost-user-video/v4l2_backend.h
@@ -0,0 +1,99 @@
+/*
+ * Virtio vhost-user VIDEO Device
+ *
+ * Copyright Linaro 2021
+ *
+ * Authors: *     Peter Griffin <peter.griffin@linaro.org>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef V4L2_BACKEND_H
+#define V4L2_BACKEND_H
+
+#include "standard-headers/linux/virtio_video.h"
+#include "virtio_video_helpers.h"
+
+#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0]))
+
+#define MAX_CAPS_LEN 4096
+#define MAX_FMT_DESCS 64
+
+#define STATEFUL_ENCODER (1 << 0)
+#define STATEFUL_DECODER (1 << 1)
+#define STATELESS_ENCODER (1 << 2)
+#define STATELESS_DECODER (1 << 3)
+
+/* Function protoypes */
+
+struct v4l2_device *v4l2_backend_init(const gchar *devname);
+void v4l2_backend_free(struct v4l2_device *dev);
+
+GByteArray *create_query_cap_resp(struct virtio_video_query_capability *qcmd,
+                            GList **fmt_l, GByteArray *querycapresp);
+enum v4l2_buf_type get_v4l2_buf_type (enum virtio_video_queue_type queue_type,
+                                      bool has_mplane);
+
+void v4l2_set_device_type(struct v4l2_device *dev, enum v4l2_buf_type type,
+                          struct v4l2_fmtdesc *fmt_desc);
+int v4l2_video_get_format(int fd, enum v4l2_buf_type type,
+                          struct v4l2_format *fmt);
+
+int v4l2_video_set_format(int fd, enum v4l2_buf_type type,
+                          struct virtio_video_params *p);
+
+int v4l2_video_get_control(int fd, uint32_t control, int32_t *value);
+
+int v4l2_queue_buffer(int fd, enum v4l2_buf_type type,
+                      struct virtio_video_resource_queue *qcmd,
+                      struct resource *res, struct stream *s,
+                      struct v4l2_device *dev);
+
+int v4l2_dequeue_buffer(int fd, enum v4l2_buf_type type,
+                        struct stream *s);
+
+int v4l2_dequeue_event(struct v4l2_device *dev);
+
+int v4l2_set_pixel_format(int fd, enum v4l2_buf_type buf_type,
+                          uint32_t pixelformat);
+int v4l2_release_buffers(int fd, enum v4l2_buf_type type);
+int v4l2_resource_create(struct stream *s, enum v4l2_buf_type type,
+                         enum virtio_video_mem_type mem_type,
+                         struct resource *res);
+int v4l2_subscribe_event(struct stream *s,
+                         uint32_t event_type, uint32_t id);
+
+int v4l2_video_get_param(int fd, enum v4l2_buf_type type,
+                         struct v4l2_streamparm *param);
+
+int v4l2_video_get_selection(int fd, enum v4l2_buf_type type,
+                             struct v4l2_selection *sel);
+
+int v4l2_video_set_selection(int fd, enum v4l2_buf_type type,
+                             struct v4l2_selection *sel);
+
+int video_send_decoder_start_cmd(struct v4l2_device *dev);
+void video_free_frame_intervals(GList *frm_intervals_l);
+void video_free_frame_sizes(GList *frm_sz_l);
+int video_enum_formats(struct v4l2_device *dev, enum v4l2_buf_type type,
+                       GList **p_fmt_list, bool only_enum_fmt);
+void video_free_formats(GList **fmt_l);
+bool video_is_mplane(enum v4l2_buf_type type);
+bool video_is_splane(enum v4l2_buf_type type);
+bool video_is_meta(enum v4l2_buf_type type);
+bool is_capture_queue(enum v4l2_buf_type type);
+bool is_output_queue(enum v4l2_buf_type type);
+int ioctl_streamon(struct stream *s, enum v4l2_buf_type type);
+int ioctl_streamoff(struct stream *s, enum v4l2_buf_type type);
+int v4l2_streamon(struct v4l2_device *dev, enum v4l2_buf_type type,
+                  struct stream *s);
+int v4l2_streamoff(enum v4l2_buf_type type, struct stream *s);
+void v4l2_print_event(const struct v4l2_event *ev);
+int v4l2_open(const gchar *devname);
+int v4l2_close(int fd);
+int v4l2_free_buffers(int fd, enum v4l2_buf_type type);
+void convert_to_timeval(uint64_t timestamp, struct timeval *t);
+int v4l2_issue_cmd(int fd,  uint32_t cmd, uint32_t flags);
+
+#endif
diff --git a/tools/vhost-user-video/virtio_video_helpers.c b/tools/vhost-user-video/virtio_video_helpers.c
new file mode 100644
index 0000000000..71353804ea
--- /dev/null
+++ b/tools/vhost-user-video/virtio_video_helpers.c
@@ -0,0 +1,462 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * virtio-video helpers
+ *
+ * Copyright Linaro 2021
+ * Copyright 2019 OpenSynergy GmbH.
+ *
+ * Authors:
+ *     Peter Griffin <peter.griffin@linaro.org>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include <assert.h>
+#include <errno.h>
+#include <stdbool.h>
+#include <stddef.h>
+#include <inttypes.h>
+
+#include <glib.h>
+#include <glib/gstdio.h>
+
+#include "standard-headers/linux/virtio_video.h"
+#include <linux/videodev2.h>
+#include "v4l2_backend.h"
+#include "virtio_video_helpers.h"
+
+struct virtio_video_convert_table {
+    uint32_t virtio_value;
+    uint32_t v4l2_value;
+};
+
+#define ARRAY_SIZE(a) (sizeof(a) / sizeof((a)[0]))
+
+static struct virtio_video_convert_table level_table[] = {
+    { VIRTIO_VIDEO_LEVEL_H264_1_0, V4L2_MPEG_VIDEO_H264_LEVEL_1_0 },
+    { VIRTIO_VIDEO_LEVEL_H264_1_1, V4L2_MPEG_VIDEO_H264_LEVEL_1_1 },
+    { VIRTIO_VIDEO_LEVEL_H264_1_2, V4L2_MPEG_VIDEO_H264_LEVEL_1_2 },
+    { VIRTIO_VIDEO_LEVEL_H264_1_3, V4L2_MPEG_VIDEO_H264_LEVEL_1_3 },
+    { VIRTIO_VIDEO_LEVEL_H264_2_0, V4L2_MPEG_VIDEO_H264_LEVEL_2_0 },
+    { VIRTIO_VIDEO_LEVEL_H264_2_1, V4L2_MPEG_VIDEO_H264_LEVEL_2_1 },
+    { VIRTIO_VIDEO_LEVEL_H264_2_2, V4L2_MPEG_VIDEO_H264_LEVEL_2_2 },
+    { VIRTIO_VIDEO_LEVEL_H264_3_0, V4L2_MPEG_VIDEO_H264_LEVEL_3_0 },
+    { VIRTIO_VIDEO_LEVEL_H264_3_1, V4L2_MPEG_VIDEO_H264_LEVEL_3_1 },
+    { VIRTIO_VIDEO_LEVEL_H264_3_2, V4L2_MPEG_VIDEO_H264_LEVEL_3_2 },
+    { VIRTIO_VIDEO_LEVEL_H264_4_0, V4L2_MPEG_VIDEO_H264_LEVEL_4_0 },
+    { VIRTIO_VIDEO_LEVEL_H264_4_1, V4L2_MPEG_VIDEO_H264_LEVEL_4_1 },
+    { VIRTIO_VIDEO_LEVEL_H264_4_2, V4L2_MPEG_VIDEO_H264_LEVEL_4_2 },
+    { VIRTIO_VIDEO_LEVEL_H264_5_0, V4L2_MPEG_VIDEO_H264_LEVEL_5_0 },
+    { VIRTIO_VIDEO_LEVEL_H264_5_1, V4L2_MPEG_VIDEO_H264_LEVEL_5_1 },
+    { 0 },
+};
+
+uint32_t virtio_video_level_to_v4l2(uint32_t level)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(level_table); idx++) {
+        if (level_table[idx].virtio_value == level) {
+            return level_table[idx].v4l2_value;
+        }
+    }
+
+    return 0;
+}
+
+uint32_t virtio_video_v4l2_level_to_virtio(uint32_t v4l2_level)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(level_table); idx++) {
+        if (level_table[idx].v4l2_value == v4l2_level) {
+            return level_table[idx].virtio_value;
+        }
+    }
+
+    return 0;
+}
+
+static struct virtio_video_convert_table profile_table[] = {
+    { VIRTIO_VIDEO_PROFILE_H264_BASELINE,
+      V4L2_MPEG_VIDEO_H264_PROFILE_BASELINE },
+    { VIRTIO_VIDEO_PROFILE_H264_MAIN, V4L2_MPEG_VIDEO_H264_PROFILE_MAIN },
+    { VIRTIO_VIDEO_PROFILE_H264_EXTENDED,
+      V4L2_MPEG_VIDEO_H264_PROFILE_EXTENDED },
+    { VIRTIO_VIDEO_PROFILE_H264_HIGH, V4L2_MPEG_VIDEO_H264_PROFILE_HIGH },
+    { VIRTIO_VIDEO_PROFILE_H264_HIGH10PROFILE,
+      V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_10 },
+    { VIRTIO_VIDEO_PROFILE_H264_HIGH422PROFILE,
+      V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_422},
+    { VIRTIO_VIDEO_PROFILE_H264_HIGH444PREDICTIVEPROFILE,
+      V4L2_MPEG_VIDEO_H264_PROFILE_HIGH_444_PREDICTIVE },
+    { VIRTIO_VIDEO_PROFILE_H264_SCALABLEBASELINE,
+      V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_BASELINE },
+    { VIRTIO_VIDEO_PROFILE_H264_SCALABLEHIGH,
+      V4L2_MPEG_VIDEO_H264_PROFILE_SCALABLE_HIGH },
+    { VIRTIO_VIDEO_PROFILE_H264_STEREOHIGH,
+      V4L2_MPEG_VIDEO_H264_PROFILE_STEREO_HIGH },
+    { VIRTIO_VIDEO_PROFILE_H264_MULTIVIEWHIGH,
+      V4L2_MPEG_VIDEO_H264_PROFILE_MULTIVIEW_HIGH },
+    { 0 },
+};
+
+uint32_t virtio_video_profile_to_v4l2(uint32_t profile)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(profile_table); idx++) {
+        if (profile_table[idx].virtio_value == profile) {
+            return profile_table[idx].v4l2_value;
+        }
+    }
+
+    return 0;
+}
+
+uint32_t virtio_video_v4l2_profile_to_virtio(uint32_t v4l2_profile)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(profile_table); idx++) {
+        if (profile_table[idx].v4l2_value == v4l2_profile) {
+            return profile_table[idx].virtio_value;
+        }
+    }
+
+    return 0;
+}
+
+
+static struct virtio_video_convert_table format_table[] = {
+    { VIRTIO_VIDEO_FORMAT_ARGB8888, V4L2_PIX_FMT_ARGB32 },
+    { VIRTIO_VIDEO_FORMAT_BGRA8888, V4L2_PIX_FMT_ABGR32 },
+    { VIRTIO_VIDEO_FORMAT_NV12, V4L2_PIX_FMT_NV12 },
+    { VIRTIO_VIDEO_FORMAT_YUV420, V4L2_PIX_FMT_YUV420 },
+    { VIRTIO_VIDEO_FORMAT_YVU420, V4L2_PIX_FMT_YVU420 },
+    { VIRTIO_VIDEO_FORMAT_MPEG2, V4L2_PIX_FMT_MPEG2 },
+    { VIRTIO_VIDEO_FORMAT_MPEG4, V4L2_PIX_FMT_MPEG4 },
+    { VIRTIO_VIDEO_FORMAT_H264, V4L2_PIX_FMT_H264 },
+    { VIRTIO_VIDEO_FORMAT_HEVC, V4L2_PIX_FMT_HEVC },
+    { VIRTIO_VIDEO_FORMAT_VP8, V4L2_PIX_FMT_VP8 },
+    { VIRTIO_VIDEO_FORMAT_VP9, V4L2_PIX_FMT_VP9 },
+    { VIRTIO_VIDEO_FORMAT_FWHT, V4L2_PIX_FMT_FWHT },
+    { 0 },
+};
+
+uint32_t virtio_video_format_to_v4l2(uint32_t format)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(format_table); idx++) {
+        if (format_table[idx].virtio_value == format) {
+            return format_table[idx].v4l2_value;
+        }
+    }
+
+    return 0;
+}
+
+uint32_t virtio_video_v4l2_format_to_virtio(uint32_t v4l2_format)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(format_table); idx++) {
+        if (format_table[idx].v4l2_value == v4l2_format) {
+            return format_table[idx].virtio_value;
+        }
+    }
+
+    return 0;
+}
+
+/*
+ * TODO FIXME PROFILE and LEVEL seem wrong here as tied to H264 codec.
+ * V4L2_CID_MPEG_VIDEO_VP9_PROFILE
+ * e.g. https://elixir.bootlin.com/linux/v5.12.1/source/
+ * include/uapi/linux/v4l2-controls.h#L669
+ */
+static struct virtio_video_convert_table control_table[] = {
+    { VIRTIO_VIDEO_CONTROL_BITRATE, V4L2_CID_MPEG_VIDEO_BITRATE },
+    { VIRTIO_VIDEO_CONTROL_PROFILE, V4L2_CID_MPEG_VIDEO_H264_PROFILE },
+    { VIRTIO_VIDEO_CONTROL_LEVEL, V4L2_CID_MPEG_VIDEO_H264_LEVEL },
+    { VIRTIO_VIDEO_CONTROL_FORCE_KEYFRAME,
+      V4L2_CID_MPEG_VIDEO_FORCE_KEY_FRAME },
+    { 0 },
+};
+
+uint32_t virtio_video_control_to_v4l2(uint32_t control)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(control_table); idx++) {
+        if (control_table[idx].virtio_value == control) {
+            return control_table[idx].v4l2_value;
+        }
+    }
+
+    return 0;
+}
+
+uint32_t virtio_video_v4l2_control_to_virtio(uint32_t v4l2_control)
+{
+    size_t idx;
+
+    for (idx = 0; idx < ARRAY_SIZE(control_table); idx++) {
+        if (control_table[idx].v4l2_value == v4l2_control) {
+            return control_table[idx].virtio_value;
+        }
+    }
+
+    return 0;
+}
+
+/* new helper functions (not from Linux frontend driver) */
+
+const char *vio_queue_name(enum virtio_video_queue_type queue)
+{
+    if (queue == VIRTIO_VIDEO_QUEUE_TYPE_INPUT) {
+        return "Queue Input";
+    }
+    if (queue == VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT) {
+        return "Queue Output";
+    }
+
+    return "Queue type unknown";
+}
+
+
+__le64 virtio_fmtdesc_generate_mask(GList **p_list)
+{
+    uint64_t mask = 0;
+    unsigned int bit = 0;
+    GList *l;
+
+    for (l = *p_list; l != NULL; l = l->next) {
+        mask |= (1 << bit);
+        bit++;
+    }
+    g_debug("%s: mask(0x%lx)\n", __func__, mask);
+
+    return mask;
+}
+
+/* vio_codedformat endian swapped by upper level */
+
+int v4l2_stream_create(struct v4l2_device *dev, uint32_t vio_codedformat,
+                       struct stream *s)
+{
+    enum v4l2_buf_type buf_type;
+    uint32_t v4l2_pixformat;
+    int ret;
+
+    /* buf type for coded format depends on device type */
+    if (dev->dev_type & STATEFUL_DECODER) {
+        buf_type = dev->has_mplane ?  V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
+            : V4L2_BUF_TYPE_VIDEO_OUTPUT;
+
+    } else if (dev->dev_type & STATEFUL_ENCODER) {
+        buf_type = dev->has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE :
+            V4L2_BUF_TYPE_VIDEO_CAPTURE;
+    } else {
+        g_critical("Unknown device type %d!", dev->dev_type);
+        return -EINVAL;
+    }
+
+    s->fd = v4l2_open(dev->devname);
+    if (s->fd < 0) {
+        g_printerr("Error opening device %s: %s (%d).\n", dev->devname,
+                   g_strerror(errno), errno);
+    }
+
+    v4l2_pixformat = virtio_video_format_to_v4l2(vio_codedformat);
+    if (v4l2_pixformat == 0) {
+        g_error("%s: virtio to v4l2 format translation failed!", __func__);
+        ret = -EINVAL;
+        return ret;
+    }
+
+    /* set the requested coded format */
+    ret = v4l2_set_pixel_format(s->fd, buf_type, v4l2_pixformat);
+    if (ret < 0) {
+        g_printerr("%s: v4l2_video_set_pixel_format() failed", __func__);
+    }
+
+    return ret;
+}
+
+void v4l2_to_virtio_fmtdesc(struct v4l2_device *dev,
+                            struct video_format *vid_fmt,
+                            enum v4l2_buf_type type)
+{
+    struct v4l2_fmtdesc *v4l2_fmtdsc = &vid_fmt->fmt;
+    struct virtio_video_format_desc *virtio_fmtdesc = &vid_fmt->desc;
+    enum v4l2_buf_type buftype;
+    int ret;
+
+    if (!vid_fmt) {
+        return;
+    }
+
+    virtio_fmtdesc->format =
+        htole32(virtio_video_v4l2_format_to_virtio(v4l2_fmtdsc->pixelformat));
+
+    /*
+     * To generate the mask we need to check the FORMAT is already set.
+     * before we enumerate the other queue to generate the mask
+     */
+
+    ret = v4l2_set_pixel_format(dev->fd, type, vid_fmt->fmt.pixelformat);
+    if (ret < 0) {
+        g_printerr("%s: v4l2_video_get_format() failed\n", __func__);
+    }
+
+    /* enumerate formats on the other queue now the format is set */
+    GList *vid_fmts_l = NULL;
+
+    if (is_output_queue(type)) {
+        buftype = video_is_mplane(type) ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
+            : V4L2_BUF_TYPE_VIDEO_CAPTURE;
+    }
+
+    if (is_capture_queue(type)) {
+        buftype = video_is_mplane(type) ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE :
+            V4L2_BUF_TYPE_VIDEO_OUTPUT;
+    }
+
+    ret = video_enum_formats(dev, buftype, &vid_fmts_l, true);
+
+    /*
+     * generate the capability mask. bitset represents the supported
+     * combinations of input and output formats.
+     */
+
+    virtio_fmtdesc->mask = htole64(virtio_fmtdesc_generate_mask(&vid_fmts_l));
+
+    virtio_fmtdesc->planes_layout =
+        htole32(VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER);
+
+    /* TODO need to set plane_align */
+    if ((!v4l2_fmtdsc->flags & V4L2_FMT_FLAG_COMPRESSED) &&
+        (le32toh(virtio_fmtdesc->planes_layout) &
+         VIRTIO_VIDEO_PLANES_LAYOUT_SINGLE_BUFFER)) {
+        g_critical("%s: TODO need to set plane_align field", __func__);
+    }
+
+    virtio_fmtdesc->num_frames = htole32(g_list_length(vid_fmt->vid_fmt_frm_l));
+
+    video_free_formats(&vid_fmts_l);
+}
+
+void v4l2_to_virtio_video_params(struct v4l2_device *dev,
+                                 struct v4l2_format *fmt,
+                                 struct v4l2_selection *sel,
+                                 struct virtio_video_get_params_resp *resp)
+{
+    struct virtio_video_params *vid_params = &resp->params;
+    int i;
+
+    /* min/max_buffers default (taken from crosvm) */
+    vid_params->min_buffers = htole32(1);
+    vid_params->max_buffers = htole32(32);
+
+    if (video_is_mplane(fmt->type)) {
+
+        vid_params->format =
+            virtio_video_v4l2_format_to_virtio(fmt->fmt.pix.pixelformat);
+        vid_params->frame_width = htole32(fmt->fmt.pix_mp.width);
+        vid_params->frame_height = htole32(fmt->fmt.pix_mp.height);
+
+        vid_params->num_planes = htole32(fmt->fmt.pix_mp.num_planes);
+
+        for (i = 0; i < fmt->fmt.pix_mp.num_planes; i++) {
+            vid_params->plane_formats[i].stride = \
+                htole32(fmt->fmt.pix_mp.plane_fmt[i].bytesperline);
+
+            vid_params->plane_formats[i].plane_size = \
+                htole32(fmt->fmt.pix_mp.plane_fmt[i].sizeimage);
+
+            g_debug(" ** Stride %u, buffer size %u\n",
+                    fmt->fmt.pix_mp.plane_fmt[i].bytesperline,
+                    fmt->fmt.pix_mp.plane_fmt[i].sizeimage);
+        }
+    } else if (video_is_splane(fmt->type)) {
+
+        vid_params->format =
+            virtio_video_v4l2_format_to_virtio(fmt->fmt.pix.pixelformat);
+        vid_params->frame_width = htole32(fmt->fmt.pix.width);
+        vid_params->frame_height = htole32(fmt->fmt.pix.height);
+        vid_params->num_planes = htole32(1);
+
+        vid_params->plane_formats[0].stride = \
+            htole32(fmt->fmt.pix.bytesperline);
+
+        vid_params->plane_formats[0].plane_size = \
+            htole32(fmt->fmt.pix.sizeimage);
+
+    }
+
+    /* cropping rectangle */
+    if (is_capture_queue(fmt->type)) {
+        vid_params->crop.left = htole32(sel->r.left);
+        vid_params->crop.top = htole32(sel->r.top);
+        vid_params->crop.width = htole32(sel->r.width);
+        vid_params->crop.height = htole32(sel->r.height);
+
+        g_debug("%s: crop: left=(%d) top=(%d) width=(%d) height=(%d)"
+                , __func__, sel->r.left, sel->r.top, sel->r.width,
+                sel->r.height);
+    }
+
+    /* TODO frame_rate field for encoder */
+}
+
+void v4l2_to_virtio_event(struct v4l2_event *ev,
+                          struct virtio_video_event *vio_ev)
+{
+    g_debug("%s:%d", __func__, __LINE__);
+    g_debug("%ld.%06ld: event %u, pending %u: ",
+            ev->timestamp.tv_sec, ev->timestamp.tv_nsec / 1000,
+            ev->sequence, ev->pending);
+
+    switch (ev->type) {
+    case V4L2_EVENT_VSYNC:
+        g_debug("vsync\n");
+        break;
+    case V4L2_EVENT_EOS:
+        g_debug("eos\n");
+        break;
+    case V4L2_EVENT_CTRL:
+        g_debug("eos\n");
+        break;
+    case V4L2_EVENT_FRAME_SYNC:
+        g_debug("frame_sync %d\n", ev->u.frame_sync.frame_sequence);
+        break;
+    case V4L2_EVENT_SOURCE_CHANGE:
+        g_debug("source_change!: pad/input=%d changes: %x\n"
+                , ev->id, ev->u.src_change.changes);
+
+        vio_ev->event_type =
+            htole32(VIRTIO_VIDEO_EVENT_DECODER_RESOLUTION_CHANGED);
+        /* TODO need proper mapping from v4l2 streamid to virtio streamid */
+        vio_ev->stream_id = htole32(ev->id) + 1;
+        break;
+    case V4L2_EVENT_MOTION_DET:
+        if (ev->u.motion_det.flags & V4L2_EVENT_MD_FL_HAVE_FRAME_SEQ) {
+            g_debug("motion_det frame %d, regions 0x%x\n",
+                    ev->u.motion_det.frame_sequence,
+                    ev->u.motion_det.region_mask);
+        } else {
+            g_debug("motion_det regions 0x%x\n", ev->u.motion_det.region_mask);
+        }
+        break;
+    default:
+        if (ev->type >= V4L2_EVENT_PRIVATE_START) {
+            g_debug("unknown private event (%08x)\n", ev->type);
+        } else {
+            g_debug("unknown event (%08x)\n", ev->type);
+        }
+        break;
+    }
+}
diff --git a/tools/vhost-user-video/virtio_video_helpers.h b/tools/vhost-user-video/virtio_video_helpers.h
new file mode 100644
index 0000000000..9c46f4105a
--- /dev/null
+++ b/tools/vhost-user-video/virtio_video_helpers.h
@@ -0,0 +1,166 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * virtio-video helpers
+ *
+ * Copyright Linaro 2021
+ *
+ * Authors:
+ *     Peter Griffin <peter.griffin@linaro.org>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef VIRTIO_VIDEO_HELPERS_H
+#define VIRTIO_VIDEO_HELPERS_H
+
+#include <stdint.h>
+#include "standard-headers/linux/virtio_video.h"
+#include <linux/videodev2.h>
+#include "libvhost-user-glib.h"
+#include "libvhost-user.h"
+
+/*
+ * Structure to track internal state of VIDEO Device
+ */
+
+typedef struct VuVideo {
+    VugDev dev;
+    struct virtio_video_config virtio_config;
+    GMainLoop *loop;
+    struct v4l2_device *v4l2_dev;
+    GList *streams;
+} VuVideo;
+
+struct v4l2_device {
+    gchar *devname;
+    unsigned int dev_type;
+    unsigned int capabilities;
+    int fd;
+    int epollfd;
+    int opened;
+    bool has_mplane;
+    bool sup_dyn_res_switching;
+};
+
+struct vu_video_ctrl_command {
+    VuVirtqElement elem;
+    VuVirtq *vq;
+    VuDev *dev;
+    struct virtio_video_cmd_hdr *cmd_hdr;
+    uint32_t error;
+    bool finished;
+    uint8_t *cmd_buf;
+};
+
+
+/*
+ * Structure to track internal state of a Stream
+ */
+
+struct stream {
+    struct virtio_video_stream_create vio_stream;
+    uint32_t stream_id;
+    GList *inputq_resources;
+    GList *outputq_resources;
+    VuVideo *video;
+    GThread *worker_thread;
+    uint32_t stream_state;
+    GMutex mutex;
+    GCond stream_cond;
+    bool output_streaming;
+    bool capture_streaming;
+    bool subscribed_events;
+    bool has_mplane;
+    int fd;
+    uint32_t output_bufcount;
+    uint32_t capture_bufcount;
+};
+
+#define STREAM_STOPPED   1
+#define STREAM_STREAMING 2
+#define STREAM_DRAINING  3
+#define STREAM_DESTROYING 4
+#define STREAM_DESTROYED 5
+
+/* Structure to track resources */
+
+struct resource {
+    uint32_t stream_id;
+    struct virtio_video_resource_create vio_resource;
+    struct virtio_video_resource_queue vio_res_q;
+    struct iovec *iov;
+    uint32_t iov_count;
+    uint32_t v4l2_index;
+    enum v4l2_buf_type type;
+    struct vu_video_ctrl_command *vio_q_cmd;
+    bool queued;
+};
+
+struct video_format_frame_rates {
+    struct virtio_video_format_range frame_rates;
+    struct v4l2_frmivalenum v4l_ival;
+};
+
+struct video_format_frame {
+    struct virtio_video_format_frame frame;
+    struct v4l2_frmsizeenum v4l_framesize;
+    GList *frm_rate_l;
+};
+
+struct video_format {
+    struct v4l2_fmtdesc fmt;
+    struct virtio_video_format_desc desc;
+    GList *vid_fmt_frm_l;
+};
+
+/* function prototypes */
+int v4l2_stream_create(struct v4l2_device *dev,
+                       uint32_t vio_codedformat, struct stream *s);
+void v4l2_to_virtio_video_params(struct v4l2_device *dev,
+                                 struct v4l2_format *fmt,
+                                 struct v4l2_selection *sel,
+                                 struct virtio_video_get_params_resp *resp);
+
+void v4l2_to_virtio_fmtdesc(struct v4l2_device *dev,
+                            struct video_format *vid_fmt,
+                            enum v4l2_buf_type type);
+
+void v4l2_to_virtio_event(struct v4l2_event *ev,
+                          struct virtio_video_event *vio_ev);
+
+struct resource *find_resource_by_v4l2index(struct stream *s,
+                                             enum v4l2_buf_type buf_type,
+                                             uint32_t v4l2_index);
+/*
+ * The following conversion helpers and tables taken from Linux
+ * frontend driver from opensynergy
+ */
+
+uint32_t virtio_video_level_to_v4l2(uint32_t level);
+uint32_t virtio_video_v4l2_level_to_virtio(uint32_t v4l2_level);
+uint32_t virtio_video_profile_to_v4l2(uint32_t profile);
+uint32_t virtio_video_v4l2_profile_to_virtio(uint32_t v4l2_profile);
+uint32_t virtio_video_format_to_v4l2(uint32_t format);
+uint32_t virtio_video_v4l2_format_to_virtio(uint32_t v4l2_format);
+uint32_t virtio_video_control_to_v4l2(uint32_t control);
+uint32_t virtio_video_v4l2_control_to_virtio(uint32_t v4l2_control);
+__le64 virtio_fmtdesc_generate_mask(GList **p_list);
+
+/* Helpers for logging */
+const char *vio_queue_name(enum virtio_video_queue_type queue);
+
+static inline void
+virtio_video_ctrl_hdr_letoh(struct virtio_video_cmd_hdr *hdr)
+{
+    hdr->type = le32toh(hdr->type);
+    hdr->stream_id = le32toh(hdr->stream_id);
+}
+
+static inline void
+virtio_video_ctrl_hdr_htole(struct virtio_video_cmd_hdr *hdr)
+{
+    hdr->type = htole32(hdr->type);
+    hdr->stream_id = htole32(hdr->stream_id);
+}
+#endif
diff --git a/tools/vhost-user-video/vuvideo.h b/tools/vhost-user-video/vuvideo.h
new file mode 100644
index 0000000000..de02e05b46
--- /dev/null
+++ b/tools/vhost-user-video/vuvideo.h
@@ -0,0 +1,43 @@
+// SPDX-License-Identifier: GPL-2.0+
+/*
+ * vhost-user-video header
+ *
+ * Copyright Linaro 2021
+ *
+ * Authors:
+ *     Peter Griffin <peter.griffin@linaro.org>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef VUVIDEO_H
+#define VUVIDEO_H
+
+#include "virtio_video_helpers.h"
+#include "v4l2_backend.h"
+#include "vuvideo.h"
+
+size_t video_iov_size(const struct iovec *iov, const unsigned int iov_cnt);
+
+GList *get_resource_list(struct stream *s, uint32_t queue_type);
+void send_qclear_res_reply(gpointer data, gpointer user_data);
+
+struct stream *find_stream(struct VuVideo *v, uint32_t stream_id);
+int add_resource(struct stream *s, struct resource *r, uint32_t queue_type);
+int remove_resource(struct stream *s, struct resource *r, uint32_t queue_type);
+struct resource *find_resource(struct stream *s, uint32_t resource_id,
+                                uint32_t queue_type);
+
+void send_ctrl_response(struct vu_video_ctrl_command *vio_cmd,
+                       uint8_t *resp, size_t resp_len);
+
+void send_ctrl_response_nodata(struct vu_video_ctrl_command *vio_cmd);
+
+void free_resource_mem(struct resource *r);
+void remove_all_resources(struct stream *s, uint32_t queue_type);
+
+void handle_queue_clear_cmd(struct VuVideo *v,
+                           struct vu_video_ctrl_command *vio_cmd);
+
+#endif
-- 
2.25.1

Re: [PATCH 8/8] tools/vhost-user-video: Add initial vhost-user-video vmm

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> This vmm translates from virtio-video v3 protocol and writes
> to a v4l2 mem2mem stateful decoder/encoder device [1]. v3 was
> chosen as that is what the virtio-video Linux frontend driver
> implements.
>
> This allows for testing with the v4l2 vicodec test codec [2]
> module in the Linux kernel, and is intended to also be used
> with Arm SoCs that implement a v4l2 stateful decoder/encoder
> drivers.
>
> The advantage of developing & testing with vicodec is that
> is allows quick development on a purely virtual setup with
> qemu and a host Linux kernel. Also it allows ci systems like
> lkft, kernelci to easily test the virtio interface.
>
> Currently conversion from virtio-video to v4l2 stateless m2m
> codec driver or VAAPI drivers is consiered out ot scope as
> is emulation of a decoder device using a something like ffmpeg.
> Although this could be added in the future.
>
> Note some virtio & v4l2 helpers were based off virtio-video
> Linux frontend driver and yavta utility, both GPL v2.
>
> Example host commands
>  modprobe vicodec
>  vhost-user-video --v4l2-device=/dev/video3 -v --socket-path=video.sock
>
> Run Qemu with
>  -device vhost-user-video-pci,chardev=video,id=video
>
> Guest decoder
>  v4l2-ctl -d0 -x width=640,height=480 -v width=640,height=480,pixelformat=YU12
>    --stream-mmap --stream-out-mmap --stream-from jelly_640_480-420P.fwht
>    --stream-to out-jelly-640-480.YU12
>
> [1] https://www.kernel.org/doc/html/latest/userspace-api/media/
>     v4l/dev-decoder.html
>
> [2] https://lwn.net/Articles/760650/
>
> Signed-off-by: Peter Griffin <peter.griffin@linaro.org>
> ---
>  tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
>  tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
>  tools/vhost-user-video/meson.build            |   10 +
>  tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
>  tools/vhost-user-video/v4l2_backend.h         |   99 +
>  tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
>  tools/vhost-user-video/virtio_video_helpers.h |  166 ++
>  tools/vhost-user-video/vuvideo.h              |   43 +
>  8 files changed, 4242 insertions(+)
>  create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
>  create mode 100644 tools/vhost-user-video/main.c
>  create mode 100644 tools/vhost-user-video/meson.build
>  create mode 100644 tools/vhost-user-video/v4l2_backend.c
>  create mode 100644 tools/vhost-user-video/v4l2_backend.h
>  create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
>  create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
>  create mode 100644 tools/vhost-user-video/vuvideo.h
>
> diff --git a/tools/vhost-user-video/50-qemu-rpmb.json.in b/tools/vhost-user-video/50-qemu-rpmb.json.in
> new file mode 100644
> index 0000000000..2b033cda56
> --- /dev/null
> +++ b/tools/vhost-user-video/50-qemu-rpmb.json.in
> @@ -0,0 +1,5 @@
> +{
> +  "description": "QEMU vhost-user-rpmb",
> +  "type": "block",
> +  "binary": "@libexecdir@/vhost-user-rpmb"
> +}

I'm spotting a copy and paste error here (filename, description and binary).

> diff --git a/tools/vhost-user-video/main.c b/tools/vhost-user-video/main.c
> new file mode 100644
> index 0000000000..a944efadb6
> --- /dev/null
> +++ b/tools/vhost-user-video/main.c
> @@ -0,0 +1,1680 @@
> +/*
> + * VIRTIO Video Emulation via vhost-user
> + *
> + * Copyright (c) 2021 Linaro Ltd
> + *
> + * SPDX-License-Identifier: GPL-2.0-or-later
> + */
> +
> +#define G_LOG_DOMAIN "vhost-user-video"
> +#define G_LOG_USE_STRUCTURED 1
> +
> +#include <glib.h>
> +#include <gio/gio.h>
> +#include <gio/gunixsocketaddress.h>
> +#include <glib-unix.h>
> +#include <glib/gstdio.h>
> +#include <stdio.h>
> +#include <string.h>
> +#include <inttypes.h>
> +#include <fcntl.h>
> +#include <sys/ioctl.h>
> +#include <sys/stat.h>
> +#include <sys/types.h>
> +#include <sys/mman.h>
> +#include <unistd.h>
> +#include <endian.h>
> +#include <assert.h>
> +
> +#include "libvhost-user-glib.h"
> +#include "libvhost-user.h"
> +#include "standard-headers/linux/virtio_video.h"
> +
> +#include "qemu/compiler.h"
> +#include "qemu/iov.h"
> +
> +#include "vuvideo.h"
> +#include "v4l2_backend.h"
> +#include "virtio_video_helpers.h"
> +
> +#ifndef container_of
> +#define container_of(ptr, type, member) ({                      \
> +        const typeof(((type *) 0)->member) * __mptr = (ptr);     \
> +        (type *) ((char *) __mptr - offsetof(type, member)); })
> +#endif
> +
> +static gchar *socket_path;
> +static gchar *v4l2_path;
> +static gint socket_fd = -1;
> +static gboolean print_cap;
> +static gboolean verbose;
> +static gboolean debug;
> +
> +static GOptionEntry options[] = {
> +    { "socket-path", 0, 0, G_OPTION_ARG_FILENAME, &socket_path,
> +      "Location of vhost-user Unix domain socket, "
> +      "incompatible with --fd", "PATH" },
> +    { "v4l2-device", 0, 0, G_OPTION_ARG_FILENAME, &v4l2_path,
> +      "Location of v4l2 device node", "PATH" },
> +    { "fd", 0, 0, G_OPTION_ARG_INT, &socket_fd,
> +      "Specify the fd of the backend, "
> +      "incompatible with --socket-path", "FD" },
> +    { "print-capabilities", 0, 0, G_OPTION_ARG_NONE, &print_cap,
> +      "Output to stdout the backend capabilities "
> +      "in JSON format and exit", NULL},
> +    { "verbose", 'v', 0, G_OPTION_ARG_NONE, &verbose,
> +      "Be more verbose in output", NULL},
> +    { "debug", 0, 0, G_OPTION_ARG_NONE, &debug,
> +      "Include debug output", NULL},
> +    { NULL }
> +};
> +
> +enum {
> +    VHOST_USER_VIDEO_MAX_QUEUES = 2,
> +};
> +
> +/* taken from util/iov.c */
> +size_t video_iov_size(const struct iovec *iov, const unsigned int iov_cnt)
> +{
> +    size_t len;
> +    unsigned int i;
> +
> +    len = 0;
> +    for (i = 0; i < iov_cnt; i++) {
> +        len += iov[i].iov_len;
> +    }
> +    return len;
> +}
> +
> +static size_t video_iov_to_buf(const struct iovec *iov,
> +                               const unsigned int iov_cnt,
> +                               size_t offset, void *buf, size_t bytes)
> +{
> +    size_t done;
> +    unsigned int i;
> +    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
> +        if (offset < iov[i].iov_len) {
> +            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
> +            memcpy(buf + done, iov[i].iov_base + offset, len);
> +            done += len;
> +            offset = 0;
> +        } else {
> +            offset -= iov[i].iov_len;
> +        }
> +    }
> +    assert(offset == 0);
> +    return done;
> +}
> +
> +static size_t video_iov_from_buf(const struct iovec *iov, unsigned int iov_cnt,
> +                                 size_t offset, const void *buf, size_t bytes)
> +{
> +    size_t done;
> +    unsigned int i;
> +    for (i = 0, done = 0; (offset || done < bytes) && i < iov_cnt; i++) {
> +        if (offset < iov[i].iov_len) {
> +            size_t len = MIN(iov[i].iov_len - offset, bytes - done);
> +            memcpy(iov[i].iov_base + offset, buf + done, len);
> +            done += len;
> +            offset = 0;
> +        } else {
> +            offset -= iov[i].iov_len;
> +        }
> +    }
> +    assert(offset == 0);
> +    return done;
> +}
> +
> +static void video_panic(VuDev *dev, const char *msg)
> +{
> +    g_critical("%s\n", msg);
> +    exit(EXIT_FAILURE);
> +}
> +
> +static uint64_t video_get_features(VuDev *dev)
> +{
> +    g_info("%s: replying", __func__);
> +    return 0;
> +}
> +
> +static void video_set_features(VuDev *dev, uint64_t features)
> +{
> +    if (features) {
> +        g_autoptr(GString) s = g_string_new("Requested un-handled feature");
> +        g_string_append_printf(s, " 0x%" PRIx64 "", features);
> +        g_info("%s: %s", __func__, s->str);
> +    }
> +}
> +
> +/*
> + * The configuration of the device is static and set when we start the
> + * daemon.
> + */
> +static int
> +video_get_config(VuDev *dev, uint8_t *config, uint32_t len)
> +{
> +    VuVideo *v = container_of(dev, VuVideo, dev.parent);
> +
> +    g_return_val_if_fail(len <= sizeof(struct virtio_video_config), -1);
> +    v->virtio_config.version = 0;
> +    v->virtio_config.max_caps_length = MAX_CAPS_LEN;
> +    v->virtio_config.max_resp_length = MAX_CAPS_LEN;
> +
> +    memcpy(config, &v->virtio_config, len);
> +
> +    g_debug("%s: config.max_caps_length = %d", __func__
> +           , ((struct virtio_video_config *)config)->max_caps_length);
> +    g_debug("%s: config.max_resp_length = %d", __func__
> +           , ((struct virtio_video_config *)config)->max_resp_length);
> +
> +    return 0;
> +}
> +
> +static int
> +video_set_config(VuDev *dev, const uint8_t *data,
> +                 uint32_t offset, uint32_t size,
> +                 uint32_t flags)
> +{
> +    g_debug("%s: ", __func__);
> +    /* ignore */
> +    return 0;
> +}
> +
> +/*
> + * Handlers for individual control messages
> + */
> +
> +static void
> +handle_set_params_cmd(struct VuVideo *v, struct vu_video_ctrl_command *vio_cmd)
> +{
> +    int ret = 0;
> +    enum v4l2_buf_type buf_type;
> +    struct virtio_video_set_params *cmd =
> +        (struct virtio_video_set_params *) vio_cmd->cmd_buf;
> +    struct stream *s;
> +
> +    g_debug("%s: type(x%x) stream_id(%d) %s ", __func__,
> +            cmd->hdr.type, cmd->hdr.stream_id,
> +            vio_queue_name(le32toh(cmd->params.queue_type)));
> +    g_debug("%s: format=0x%x frame_width(%d) frame_height(%d)",
> +            __func__, le32toh(cmd->params.format),
> +            le32toh(cmd->params.frame_width),
> +            le32toh(cmd->params.frame_height));
> +    g_debug("%s: min_buffers(%d) max_buffers(%d)", __func__,
> +            le32toh(cmd->params.min_buffers), le32toh(cmd->params.max_buffers));
> +    g_debug("%s: frame_rate(%d) num_planes(%d)", __func__,
> +            le32toh(cmd->params.frame_rate), le32toh(cmd->params.num_planes));
> +    g_debug("%s: crop top=%d, left=%d, width=%d, height=%d", __func__,
> +            le32toh(cmd->params.crop.left), le32toh(cmd->params.crop.top),
> +            le32toh(cmd->params.crop.width), le32toh(cmd->params.crop.height));
> +
> +    s = find_stream(v, cmd->hdr.stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found", __func__, cmd->hdr.stream_id);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }

I think you can just return from here as no clean-up is required.

> +
> +    g_mutex_lock(&s->mutex);

It is possible to use the g_autofree stuff to simplify this but as we
are avoiding bringing in QEMU code we can't use WITH_QEMU_LOCK_GUARD :-/

> +
> +    buf_type = get_v4l2_buf_type(le32toh(cmd->params.queue_type),
> +                                 s->has_mplane);
> +
> +    ret = v4l2_video_set_format(s->fd, buf_type, &cmd->params);
> +    if (ret < 0) {
> +        g_error("%s: v4l2_video_set_format() failed", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    if (is_capture_queue(buf_type)) {
> +        /* decoder supports composing on CAPTURE */
> +        struct v4l2_selection sel;
> +        memset(&sel, 0, sizeof(struct v4l2_selection));
> +
> +        sel.r.left = le32toh(cmd->params.crop.left);
> +        sel.r.top = le32toh(cmd->params.crop.top);
> +        sel.r.width = le32toh(cmd->params.crop.width);
> +        sel.r.height = le32toh(cmd->params.crop.height);
> +
> +        ret = v4l2_video_set_selection(s->fd, buf_type, &sel);
> +        if (ret < 0) {
> +            g_printerr("%s: v4l2_video_set_selection failed: %s (%d).\n"
> +                       , __func__, g_strerror(errno), errno);
> +            cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +            goto out_unlock;
> +        }
> +    }
> +
> +    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +
> +out_unlock:
> +    vio_cmd->finished = true;
> +    send_ctrl_response_nodata(vio_cmd);
> +    g_mutex_unlock(&s->mutex);
> +out:
> +    return;
> +}
> +
> +static void
> +handle_get_params_cmd(struct VuVideo *v, struct vu_video_ctrl_command *vio_cmd)
> +{
> +    int ret;
> +    struct v4l2_format fmt;
> +    struct v4l2_selection sel;
> +    enum v4l2_buf_type buf_type;
> +    struct virtio_video_get_params *cmd =
> +        (struct virtio_video_get_params *) vio_cmd->cmd_buf;
> +    struct virtio_video_get_params_resp getparams_reply;
> +    struct stream *s;
> +
> +    g_debug("%s: type(0x%x) stream_id(%d) %s", __func__,
> +            cmd->hdr.type, cmd->hdr.stream_id,
> +            vio_queue_name(le32toh(cmd->queue_type)));
> +
> +    s = find_stream(v, cmd->hdr.stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found\n"
> +                   , __func__, cmd->hdr.stream_id);
> +        getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    g_mutex_lock(&s->mutex);
> +
> +    getparams_reply.hdr.stream_id = cmd->hdr.stream_id;
> +    getparams_reply.params.queue_type = cmd->queue_type;
> +
> +    buf_type = get_v4l2_buf_type(cmd->queue_type, s->has_mplane);
> +
> +    ret = v4l2_video_get_format(s->fd, buf_type, &fmt);
> +    if (ret < 0) {
> +        g_printerr("v4l2_video_get_format failed\n");
> +        getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    if (is_capture_queue(buf_type)) {
> +        ret = v4l2_video_get_selection(s->fd, buf_type, &sel);
> +        if (ret < 0) {
> +            g_printerr("v4l2_video_get_selection failed\n");
> +            getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +            goto out_unlock;
> +        }
> +    }
> +
> +    /* convert from v4l2 to virtio */
> +    v4l2_to_virtio_video_params(v->v4l2_dev, &fmt, &sel,
> +                                &getparams_reply);
> +
> +    getparams_reply.hdr.type = VIRTIO_VIDEO_RESP_OK_GET_PARAMS;
> +
> +out_unlock:
> +    vio_cmd->finished = true;
> +    send_ctrl_response(vio_cmd, (uint8_t *)&getparams_reply,
> +                       sizeof(struct virtio_video_get_params_resp));
> +    g_mutex_unlock(&s->mutex);
> +out:
> +    return;
> +}
> +
> +struct stream *find_stream(struct VuVideo *v, uint32_t stream_id)
> +{
> +    GList *l;
> +    struct stream *s;
> +
> +    for (l = v->streams; l != NULL; l = l->next) {
> +        s = (struct stream *)l->data;
> +        if (s->stream_id == stream_id) {
> +            return s;
> +        }
> +    }
> +
> +    return NULL;
> +}
> +
> +int add_resource(struct stream *s, struct resource *r, uint32_t queue_type)
> +{
> +
> +    if (!s || !r) {
> +        return -EINVAL;
> +    }
> +
> +    switch (queue_type) {
> +    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
> +        s->inputq_resources = g_list_append(s->inputq_resources, r);
> +        break;
> +
> +    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
> +        s->outputq_resources = g_list_append(s->outputq_resources, r);
> +        break;
> +    default:
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
> +void free_resource_mem(struct resource *r)
> +{
> +
> +    /*
> +     * Frees the memory allocated for resource_queue_cmd
> +     * not the memory allocated in resource_create
> +     */
> +
> +    if (r->vio_q_cmd) {
> +        g_free(r->vio_q_cmd->cmd_buf);
> +        r->vio_q_cmd->cmd_buf = NULL;
> +        free(r->vio_q_cmd);
> +        r->vio_q_cmd = NULL;
> +    }
> +}
> +
> +void remove_all_resources(struct stream *s, uint32_t queue_type)
> +{
> +    GList **resource_list;
> +    struct resource *r;
> +
> +    /* assumes stream mutex is held by caller */
> +
> +    switch (queue_type) {
> +    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
> +        resource_list =  &s->inputq_resources;
> +        break;
> +    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
> +        resource_list = &s->outputq_resources;
> +        break;
> +    default:
> +        g_critical("%s: Invalid virtio queue!", __func__);
> +        return;
> +    }
> +
> +    g_debug("%s: resource_list has %d elements", __func__
> +            , g_list_length(*resource_list));
> +
> +    GList *l = *resource_list;
> +    while (l != NULL) {
> +        GList *next = l->next;
> +        r = (struct resource *)l->data;
> +        if (r) {
> +            g_debug("%s: Removing resource_id(%d) resource=%p"
> +                    , __func__, r->vio_resource.resource_id, r);
> +
> +            /*
> +             * Assumes that either QUEUE_CLEAR or normal dequeuing
> +             * of buffers will have freed resource_queue cmd memory
> +             */
> +
> +            /* free resource memory allocated in resource_create() */
> +            g_free(r->iov);
> +            g_free(r);
> +            *resource_list = g_list_delete_link(*resource_list, l);
> +        }
> +        l = next;
> +   }
> +}
> +
> +struct resource *find_resource(struct stream *s, uint32_t resource_id,
> +                                uint32_t queue_type)
> +{
> +    GList *l;
> +    struct resource *r;
> +
> +    switch (queue_type) {
> +    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
> +        l = s->inputq_resources;
> +        break;
> +
> +    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
> +        l = s->outputq_resources;
> +        break;
> +    default:
> +        g_error("%s: Invalid queue type!", __func__);
> +    }
> +
> +    for (; l != NULL; l = l->next) {
> +        r = (struct resource *)l->data;
> +        if (r->vio_resource.resource_id == resource_id) {
> +            return r;
> +        }
> +    }

Given the iteration here would it be worth tracking the struct resource
in a GArray rather than chasing pointers in a linked list?

> +
> +    return NULL;
> +}
> +
> +struct resource *find_resource_by_v4l2index(struct stream *s,
> +                                             enum v4l2_buf_type buf_type,
> +                                             uint32_t v4l2_index)
> +{
> +    GList *l = NULL;
> +    struct resource *r;
> +
> +    switch (buf_type) {
> +    case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> +    case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> +        l = s->outputq_resources;
> +        break;
> +
> +    case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> +    case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> +        l = s->inputq_resources;
> +        break;
> +
> +    default:
> +        g_error("Unsupported buffer type\n");
> +    }
> +
> +    for (; l != NULL; l = l->next) {
> +        r = (struct resource *)l->data;
> +        if (r->v4l2_index == v4l2_index) {
> +            g_debug("%s: found Resource=%p streamid(%d) resourceid(%d) "
> +                    "numplanes(%d) planes_layout(0x%x) vio_q_cmd=%p", __func__,
> +                    r, r->stream_id, r->vio_resource.resource_id,
> +                    r->vio_resource.num_planes, r->vio_resource.planes_layout,
> +                    r->vio_q_cmd);
> +            return r;
> +        }
> +    }
> +    return NULL;
> +}
> +
> +#define EVENT_WQ_IDX 1
> +
> +static void *stream_worker_thread(gpointer data)
> +{
> +    int ret;
> +    struct stream *s = data;
> +    VuVideo *v = s->video;
> +    VugDev *vugdev = &v->dev;
> +    VuDev *vudev = &vugdev->parent;
> +    VuVirtq *vq = vu_get_queue(vudev, EVENT_WQ_IDX);
> +    VuVirtqElement *elem;
> +    size_t len;
> +
> +    struct v4l2_event ev;
> +    struct virtio_video_event vio_event;
> +
> +    /* select vars */
> +    fd_set efds, rfds, wfds;
> +    bool have_event, have_read, have_write;
> +    enum v4l2_buf_type buf_type;
> +
> +    fcntl(s->fd, F_SETFL, fcntl(s->fd, F_GETFL) | O_NONBLOCK);
> +
> +    while (true) {
> +            int res;
> +
> +            g_mutex_lock(&s->mutex);
> +
> +            /* wait for STREAMING or DESTROYING state */
> +            while (s->stream_state != STREAM_DESTROYING &&
> +                   s->stream_state != STREAM_STREAMING &&
> +                   s->stream_state != STREAM_DRAINING)
> +                g_cond_wait(&s->stream_cond, &s->mutex);
> +
> +            if (s->stream_state == STREAM_DESTROYING) {
> +                g_debug("stream worker thread exiting!");
> +                s->stream_state = STREAM_DESTROYED;
> +                g_cond_signal(&s->stream_cond);
> +                g_mutex_unlock(&s->mutex);
> +                g_thread_exit(0);
> +            }
> +
> +            g_mutex_unlock(&s->mutex);
> +
> +            FD_ZERO(&efds);
> +            FD_SET(s->fd, &efds);
> +            FD_ZERO(&rfds);
> +            FD_SET(s->fd, &rfds);
> +            FD_ZERO(&wfds);
> +            FD_SET(s->fd, &wfds);
> +
> +            struct timeval tv = { 0 , 500000 };
> +            res = select(s->fd + 1, &rfds, &wfds, &efds, &tv);
> +            if (res < 0) {
> +                g_printerr("%s:%d - select() failed errno(%s)\n", __func__,
> +                           __LINE__, g_strerror(errno));
> +                break;
> +            }
> +
> +            if (res == 0) {
> +                g_debug("%s:%d - select() timeout", __func__, __LINE__);
> +                continue;
> +            }
> +
> +            have_event = FD_ISSET(s->fd, &efds);
> +            have_read = FD_ISSET(s->fd, &rfds);
> +            have_write = FD_ISSET(s->fd, &wfds);
> +            /* read is capture queue, write is output queue */
> +
> +            g_debug("%s:%d have_event=%d, have_write=%d, have_read=%d\n"
> +                    , __func__, __LINE__, FD_ISSET(s->fd, &efds)
> +                    , FD_ISSET(s->fd, &wfds), FD_ISSET(s->fd, &rfds));
> +
> +            g_mutex_lock(&s->mutex);
> +
> +            if (have_event) {
> +                g_debug("%s: have_event!", __func__);
> +                res = ioctl(s->fd, VIDIOC_DQEVENT, &ev);
> +                if (res < 0) {
> +                    g_printerr("%s:%d - VIDIOC_DQEVENT failed: errno(%s)\n",
> +                               __func__, __LINE__, g_strerror(errno));
> +                    break;
> +                }
> +                v4l2_to_virtio_event(&ev, &vio_event);
> +
> +                /* get event workqueue */
> +                elem = vu_queue_pop(vudev, vq, sizeof(struct VuVirtqElement));
> +                if (!elem) {
> +                    g_debug("%s:%d\n", __func__, __LINE__);
> +                    break;
> +                }
> +
> +                len = video_iov_from_buf(elem->in_sg,
> +                                         elem->in_num, 0, (void *) &vio_event,
> +                                         sizeof(struct virtio_video_event));
> +
> +                vu_queue_push(vudev, vq, elem, len);
> +                vu_queue_notify(vudev, vq);
> +            }
> +
> +            if (have_read && s->capture_streaming == true) {
> +                /* TODO assumes decoder */
> +                buf_type = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE
> +                    : V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +
> +                ret = v4l2_dequeue_buffer(s->fd, buf_type, s);
> +                if (ret < 0) {
> +                    g_info("%s: v4l2_dequeue_buffer() failed CAPTURE ret(%d)"
> +                           , __func__, ret);
> +
> +                    if (errno == EPIPE) {
> +                        /* dequeued last buf, so stop streaming */
> +                        ioctl_streamoff(s, buf_type);
> +                    }
> +                 }
> +            }
> +
> +            if (have_write  && s->output_streaming == true) {
> +                buf_type = s->has_mplane ? V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE
> +                    : V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +
> +                ret = v4l2_dequeue_buffer(s->fd, buf_type, s);
> +                if (ret < 0) {
> +                    g_info("%s: v4l2_dequeue_buffer() failed OUTPUT ret(%d)"
> +                           , __func__, ret);
> +                }
> +            }
> +
> +            g_mutex_unlock(&s->mutex);
> +    }
> +
> +    return NULL;
> +}
> +
> +void handle_queue_clear_cmd(struct VuVideo *v,
> +                       struct vu_video_ctrl_command *vio_cmd)
> +{
> +    struct virtio_video_queue_clear *cmd =
> +        (struct virtio_video_queue_clear *)vio_cmd->cmd_buf;
> +    int ret = 0;
> +    struct stream *s;
> +    uint32_t stream_id = le32toh(cmd->hdr.stream_id);
> +    enum virtio_video_queue_type queue_type = le32toh(cmd->queue_type);
> +
> +    g_debug("%s: stream_id(%d) %s\n", __func__, stream_id,
> +            vio_queue_name(queue_type));
> +
> +    if (!v || !cmd) {
> +        return;
> +    }
> +
> +    s = find_stream(v, stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    g_mutex_lock(&s->mutex);
> +
> +    enum v4l2_buf_type buf_type =
> +        get_v4l2_buf_type(le32toh(cmd->queue_type), s->has_mplane);
> +
> +    /*
> +     * QUEUE_CLEAR behaviour from virtio-video spec
> +     * Return already queued buffers back from the input or the output queue
> +     * of the device. The device SHOULD return all of the buffers from the
> +     * respective queue as soon as possible without pushing the buffers through
> +     * the processing pipeline.
> +     *
> +     * From v4l2 PoV we issue a VIDIOC_STREAMOFF on the queue which will abort
> +     * or finish any DMA in progress, unlocks any user pointer buffers locked
> +     * in physical memory, and it removes all buffers from the incoming and
> +     * outgoing queues.
> +     */
> +
> +    /* issue streamoff  */
> +    ret = ioctl_streamoff(s, buf_type);
> +    if (ret < 0) {
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    /* iterate the queues resources list - and send a reply to each one */
> +
> +    /*
> +     * If the processing was stopped due to VIRTIO_VIDEO_CMD_QUEUE_CLEAR,
> +     * the device MUST respond with VIRTIO_VIDEO_RESP_OK_NODATA as a response
> +     * type and VIRTIO_- VIDEO_BUFFER_FLAG_ERR in flags.
> +     */
> +
> +    g_list_foreach(get_resource_list(s, queue_type),
> +                   (GFunc)send_qclear_res_reply, s);
> +
> +    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +
> +out_unlock:
> +    vio_cmd->finished = true;
> +    send_ctrl_response_nodata(vio_cmd);
> +    g_mutex_unlock(&s->mutex);
> +out:
> +    return;
> +}
> +
> +GList *get_resource_list(struct stream *s, uint32_t queue_type)
> +{
> +    switch (queue_type) {
> +    case VIRTIO_VIDEO_QUEUE_TYPE_INPUT:
> +        return s->inputq_resources;
> +        break;
> +
> +    case VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT:
> +        return s->outputq_resources;
> +        break;
> +    default:
> +        g_critical("%s: Unknown queue type!", __func__);
> +        return NULL;
> +    }
> +}
> +
> +void send_ctrl_response(struct vu_video_ctrl_command *vio_cmd,
> +                       uint8_t *resp, size_t resp_len)
> +{
> +    size_t len;
> +
> +    virtio_video_ctrl_hdr_htole((struct virtio_video_cmd_hdr *)resp);
> +
> +    /* send virtio_video_resource_queue_resp */
> +    len = video_iov_from_buf(vio_cmd->elem.in_sg,
> +                             vio_cmd->elem.in_num, 0, resp, resp_len);
> +
> +    if (len != resp_len) {
> +        g_critical("%s: response size incorrect %zu vs %zu",
> +                   __func__, len, resp_len);
> +    }
> +
> +    vu_queue_push(vio_cmd->dev, vio_cmd->vq, &vio_cmd->elem, len);
> +    vu_queue_notify(vio_cmd->dev, vio_cmd->vq);
> +
> +    if (vio_cmd->finished) {
> +        g_free(vio_cmd->cmd_buf);
> +        free(vio_cmd);
> +    }
> +}
> +
> +void send_ctrl_response_nodata(struct vu_video_ctrl_command *vio_cmd)
> +{
> +    send_ctrl_response(vio_cmd, vio_cmd->cmd_buf,
> +                       sizeof(struct virtio_video_cmd_hdr));
> +}
> +
> +void send_qclear_res_reply(gpointer data, gpointer user_data)
> +{
> +    struct resource *r = data;
> +    struct vu_video_ctrl_command *vio_cmd = r->vio_q_cmd;
> +    struct virtio_video_queue_clear *cmd =
> +        (struct virtio_video_queue_clear *) vio_cmd->cmd_buf;
> +    struct virtio_video_resource_queue_resp resp;
> +
> +    /*
> +     * only need to send replies for buffers that are
> +     * inflight
> +     */
> +
> +    if (r->queued) {
> +
> +        resp.hdr.stream_id = cmd->hdr.stream_id;
> +        resp.hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +        resp.flags = htole32(VIRTIO_VIDEO_BUFFER_FLAG_ERR);
> +        resp.timestamp = htole64(r->vio_res_q.timestamp);
> +
> +        g_debug("%s: stream_id=%d type=0x%x flags=0x%x resource_id=%d t=%llx"
> +                , __func__, resp.hdr.stream_id, resp.hdr.type, resp.flags,
> +                r->vio_resource.resource_id, resp.timestamp);
> +
> +        vio_cmd->finished = true;
> +        send_ctrl_response(vio_cmd, (uint8_t *) &resp,
> +                           sizeof(struct virtio_video_resource_queue_resp));
> +    }
> +    return;
> +}
> +
> +static int
> +handle_resource_create_cmd(struct VuVideo *v,
> +                           struct vu_video_ctrl_command *vio_cmd)
> +{
> +    int ret = 0, i;
> +    uint32_t total_entries = 0;
> +    uint32_t stream_id ;
> +    struct virtio_video_resource_create *cmd =
> +        (struct virtio_video_resource_create *)vio_cmd->cmd_buf;
> +    struct virtio_video_mem_entry *mem;
> +    struct resource *res;
> +    struct virtio_video_resource_create *r;
> +    struct stream *s;
> +    enum virtio_video_mem_type mem_type;
> +
> +    stream_id = cmd->hdr.stream_id;
> +
> +    s = find_stream(v, stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    g_mutex_lock(&s->mutex);
> +
> +    if (le32toh(cmd->resource_id) == 0) {
> +        g_critical("%s: resource id 0 is not allowed", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    /* check resource id doesn't already exist */
> +    res = find_resource(s, le32toh(cmd->resource_id), le32toh(cmd->queue_type));
> +    if (res) {
> +        g_critical("%s: resource_id:%d already exists"
> +                   , __func__, le32toh(cmd->resource_id));
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
> +        goto out_unlock;
> +    } else {
> +        res = g_new0(struct resource, 1);
> +        res->vio_resource.resource_id = le32toh(cmd->resource_id);
> +        res->vio_resource.queue_type = le32toh(cmd->queue_type);
> +        res->vio_resource.planes_layout = le32toh(cmd->planes_layout);
> +
> +        res->vio_resource.num_planes = le32toh(cmd->num_planes);
> +        r = &res->vio_resource;
> +
> +        ret = add_resource(s, res, le32toh(cmd->queue_type));
> +        if (ret) {
> +            g_critical("%s: resource_add id:%d failed"
> +                       , __func__, le32toh(cmd->resource_id));
> +            cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
> +            goto out_unlock;
> +        }
> +
> +        g_debug("%s: resource=%p streamid(%d) resourceid(%d) numplanes(%d)"
> +                "planes_layout(0x%x) %s",
> +                __func__, res, res->stream_id, r->resource_id, r->num_planes,
> +                r->planes_layout, vio_queue_name(r->queue_type));
> +    }
> +
> +    if (r->planes_layout & VIRTIO_VIDEO_PLANES_LAYOUT_PER_PLANE) {
> +        g_debug("%s: streamid(%d) resourceid(%d) planes_layout(0x%x)"
> +                , __func__, res->stream_id, r->resource_id, r->planes_layout);
> +
> +        for (i = 0; i < r->num_planes; i++) {
> +            total_entries += le32toh(cmd->num_entries[i]);
> +            g_debug("%s: streamid(%d) resourceid(%d) num_entries[%d]=%d"
> +                    , __func__, res->stream_id, r->resource_id,
> +                    i, le32toh(cmd->num_entries[i]));
> +        }
> +    } else {
> +        total_entries = 1;
> +    }
> +
> +    /*
> +     * virtio_video_resource_create is followed by either
> +     * - struct virtio_video_mem_entry entries[]
> +     *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
> +     * - struct virtio_video_object_entry entries[]
> +     *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
> +     */
> +
> +    if (r->queue_type == VIRTIO_VIDEO_QUEUE_TYPE_INPUT) {
> +        mem_type = s->vio_stream.in_mem_type;
> +    } else {
> +        mem_type = s->vio_stream.out_mem_type;
> +    }
> +    /*
> +     * Followed by either
> +     * - struct virtio_video_mem_entry entries[]
> +     *   for VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES
> +     * - struct virtio_video_object_entry entries[]
> +     *   for VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT
> +     */
> +
> +    if (mem_type == VIRTIO_VIDEO_MEM_TYPE_GUEST_PAGES) {
> +        mem = (void *)cmd + sizeof(struct virtio_video_resource_create);
> +
> +        res->iov = g_malloc0(sizeof(struct iovec) * total_entries);
> +        for (i = 0; i < total_entries; i++) {
> +            uint64_t len = le32toh(mem[i].length);
> +            g_debug("%s: mem[%d] addr=0x%lx", __func__
> +                    , i, le64toh(mem[i].addr));
> +
> +            res->iov[i].iov_len = le32toh(mem[i].length);
> +            res->iov[i].iov_base =
> +                vu_gpa_to_va(&v->dev.parent, &len, le64toh(mem[i].addr));
> +            g_debug("%s: [%d] iov_len = 0x%lx", __func__
> +                    , i, res->iov[i].iov_len);
> +            g_debug("%s: [%d] iov_base = 0x%p", __func__
> +                    , i, res->iov[i].iov_base);
> +        }
> +        res->iov_count = total_entries;
> +
> +    } else if (mem_type == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT) {
> +        g_critical("%s: VIRTIO_OBJECT not implemented!", __func__);
> +        /* TODO implement VIRTIO_OBJECT support */
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    /* check underlying driver supports GUEST_PAGES */
> +    enum v4l2_buf_type buf_type =
> +        get_v4l2_buf_type(r->queue_type, s->has_mplane);
> +
> +    ret = v4l2_resource_create(s, buf_type, mem_type, res);
> +    if (ret < 0) {
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out_unlock;
> +    }
> +
> +    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +
> +out_unlock:
> +    /* send response */
> +    vio_cmd->finished = true;
> +    send_ctrl_response_nodata(vio_cmd);
> +    g_mutex_unlock(&s->mutex);
> +out:
> +    return ret;
> +}
> +
> +static int
> +handle_resource_queue_cmd(struct VuVideo *v,
> +                          struct vu_video_ctrl_command *vio_cmd)
> +{
> +    struct virtio_video_resource_queue *cmd =
> +        (struct virtio_video_resource_queue *)vio_cmd->cmd_buf;
> +    struct resource *res;
> +    struct stream *s;
> +    uint32_t stream_id;
> +    int ret = 0;
> +
> +    g_debug("%s: type(0x%x) %s resource_id(%d)", __func__,
> +            cmd->hdr.type, vio_queue_name(le32toh(cmd->queue_type)),
> +            le32toh(cmd->resource_id));
> +    g_debug("%s: num_data_sizes = %d", __func__, le32toh(cmd->num_data_sizes));
> +    g_debug("%s: data_sizes[0] = %d", __func__, le32toh(cmd->data_sizes[0]));
> +
> +    stream_id = cmd->hdr.stream_id;
> +
> +    s = find_stream(v, stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found", __func__, stream_id);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    g_mutex_lock(&s->mutex);
> +
> +    if (cmd->resource_id == 0) {
> +        g_critical("%s: resource id 0 is not allowed", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
> +        goto out_unlock;
> +    }
> +
> +    /* get resource object */
> +    res = find_resource(s, le32toh(cmd->resource_id), le32toh(cmd->queue_type));
> +    if (!res) {
> +        g_critical("%s: resource_id:%d does not exist!"
> +                   , __func__, le32toh(cmd->resource_id));
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_RESOURCE_ID;
> +        goto out_unlock;
> +    }
> +
> +    res->vio_res_q.timestamp = le64toh(cmd->timestamp);
> +    res->vio_res_q.num_data_sizes = le32toh(cmd->num_data_sizes);
> +    res->vio_res_q.queue_type = le32toh(cmd->queue_type);
> +    res->vio_q_cmd = vio_cmd;
> +
> +    g_debug("%s: res=%p res->vio_q_cmd=0x%p", __func__, res, res->vio_q_cmd);
> +
> +    enum v4l2_buf_type buf_type = get_v4l2_buf_type(
> +        cmd->queue_type, s->has_mplane);
> +
> +
> +    ret = v4l2_queue_buffer(s->fd, buf_type, cmd, res, s, v->v4l2_dev);
> +    if (ret < 0) {
> +        g_critical("%s: v4l2_queue_buffer failed", __func__);
> +        /* virtio error set by v4l2_queue_buffer */
> +        goto out_unlock;
> +    }
> +
> +    /*
> +     * let the stream worker thread do the dequeueing of output and
> +     * capture queue buffers and send the resource_queue replies
> +     */
> +
> +    g_mutex_unlock(&s->mutex);
> +    return ret;
> +
> +out_unlock:
> +    /* send response */
> +    vio_cmd->finished = true;
> +    send_ctrl_response_nodata(vio_cmd);
> +    g_mutex_unlock(&s->mutex);
> +out:
> +    return ret;
> +}
> +
> +
> +static void
> +handle_resource_destroy_all_cmd(struct VuVideo *v,
> +                                struct vu_video_ctrl_command *vio_cmd)
> +{
> +    struct virtio_video_resource_destroy_all *cmd =
> +        (struct virtio_video_resource_destroy_all *)vio_cmd->cmd_buf;
> +    enum v4l2_buf_type buf_type;
> +    struct stream *s;
> +    int ret = 0;
> +
> +    g_debug("%s: type(0x%x) %s stream_id(%d)", __func__,
> +            cmd->hdr.type, vio_queue_name(le32toh(cmd->queue_type)),
> +            cmd->hdr.stream_id);
> +
> +    s = find_stream(v, cmd->hdr.stream_id);
> +    if (!s) {
> +        g_critical("%s: stream_id(%d) not found", __func__, cmd->hdr.stream_id);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    g_mutex_lock(&s->mutex);
> +
> +    buf_type = get_v4l2_buf_type(le32toh(cmd->queue_type), s->has_mplane);
> +
> +    ret = v4l2_free_buffers(s->fd, buf_type);
> +    if (ret) {
> +        g_critical("%s: v4l2_free_buffers() failed", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    remove_all_resources(s, le32toh(cmd->queue_type));
> +
> +    /* free resource objects from queue list */
> +    cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +
> +out:
> +    vio_cmd->finished = true;
> +    send_ctrl_response_nodata(vio_cmd);
> +    g_mutex_unlock(&s->mutex);
> +}
> +
> +static void
> +handle_stream_create_cmd(struct VuVideo *v,
> +                         struct vu_video_ctrl_command *vio_cmd)
> +{
> +    int ret = 0;
> +    struct stream *s;
> +    uint32_t req_stream_id;
> +    uint32_t coded_format;
> +
> +    struct virtio_video_stream_create *cmd =
> +        (struct virtio_video_stream_create *)vio_cmd->cmd_buf;
> +
> +    g_debug("%s: type(0x%x) stream_id(%d) in_mem_type(0x%x) "
> +            "out_mem_type(0x%x) coded_format(0x%x)",
> +            __func__, cmd->hdr.type, cmd->hdr.stream_id,
> +            le32toh(cmd->in_mem_type), le32toh(cmd->out_mem_type),
> +            le32toh(cmd->coded_format));
> +
> +    req_stream_id = cmd->hdr.stream_id;
> +    coded_format = le32toh(cmd->coded_format);
> +
> +    if ((le32toh(cmd->in_mem_type) == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT) ||
> +        (le32toh(cmd->out_mem_type) == VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT)) {
> +        /* TODO implement VIRTIO_VIDEO_MEM_TYPE_VIRTIO_OBJECT */
> +        g_printerr("%s: MEM_TYPE_VIRTIO_OBJECT not supported yet", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_PARAMETER;
> +        goto out;
> +    }
> +
> +    if (!find_stream(v, req_stream_id)) {
> +        s = g_new0(struct stream, 1);
> +        /* copy but bswap */
> +        s->vio_stream.in_mem_type = le32toh(cmd->in_mem_type);
> +        s->vio_stream.out_mem_type = le32toh(cmd->out_mem_type);
> +        s->vio_stream.coded_format = le32toh(cmd->coded_format);
> +        strncpy((char *)&s->vio_stream.tag, (char *)cmd->tag,
> +                sizeof(cmd->tag) - 1);
> +        s->vio_stream.tag[sizeof(cmd->tag) - 1] = 0;
> +        s->stream_id = req_stream_id;
> +        s->video = v;
> +        s->stream_state = STREAM_STOPPED;
> +        s->has_mplane = v->v4l2_dev->has_mplane;
> +        g_mutex_init(&s->mutex);
> +        g_cond_init(&s->stream_cond);
> +        v->streams = g_list_append(v->streams, s);
> +
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_OK_NODATA;
> +    } else {
> +        g_debug("%s: Stream ID in use - ", __func__);
> +        cmd->hdr.type = VIRTIO_VIDEO_RESP_ERR_INVALID_STREAM_ID;
> +        goto out;
> +    }

Couldn't you avoid the goto by folding this in a level:

  if (le32toh....) {
    ...
  } else if (find_stream()) {
    ...
  } else {
    .. fall through case ..
    v4l_stream_create...
    g_thread_new
  } 

  send_ctrl_response()

I know gotos are the kernel style but we can at least try to avoid them ;-)

I've run out of steam here (3000 lines is a lot for one patch)... I'll
do another pass on the next revision.

-- 
Alex Bennée

Re: [PATCH 0/8] virtio: Add vhost-user based Video decode

[Michael S. Tsirkin]

On Thu, Dec 09, 2021 at 02:55:53PM +0000, Peter Griffin wrote:
> This series adds support for virtio-video decoder devices in Qemu
> and also provides a vhost-user-video vmm implementation.
> 
> The vhost-user-video vmm currently parses virtio-vido v3 protocol
> (as that is what the Linux frontend driver implements).
> It then converts that to a v4l2 mem2mem stateful decoder device.
> Currently this has been tested using v4l2 vicodec test driver in Linux
> [1] but it is intended to be used with Arm SoCs which often implement
> v4l2 stateful decoders/encoders drivers for their video accelerators.
> 
> The primary goal so far has been to allow continuing development
> of virtio-video Linux frontend driver and testing with Qemu. Using
> vicodec on the host allows a purely virtual dev env, and allows for
> ci integration in the future by kernelci etc.
> 
> This series also adds the virtio_video.h header and adds the
> FWHT format which is used by vicodec driver.
> 
> I have tested this VMM using v4l2-ctl from v4l2 utils in the guest
> to do a video decode to a file. This can then be validated using ffplay
> v4l2-compliance tool in the guest has also been run which stresses the
> interface and issues lots of syscall level tests
> 
> See the README.md for example commands on how to configure guest kernel
> and do a video decode using Qemu, vicodec using this VMM.
> 
> Linux virtio-video frontend driver code:
> https://github.com/petegriffin/linux/commits/v5.10-virtio-video-latest

Are you going to post this btw?


> Qemu vmm code:
> https://github.com/petegriffin/qemu/tree/vhost-virtio-video-master-v1
> 
> This is part of a wider initiative by Linaro called
> "project Stratos" for which you can find information here:
> 
>   https://collaborate.linaro.org/display/STR/Stratos+Home
> 
> Applies cleanly to git://git.qemu.org/qemu.git master(a3607def89).
> 
> Thanks,
> 
> Peter.
> 
> [1] https://lwn.net/Articles/760650/

And the spec status?
Does this match this spec:
https://lore.kernel.org/linux-media/CAPBb6MUD_oYeUt8_bGRwAPCPam40Jtktz2F7+A3SO5PFRdUsiw@mail.gmail.com/T/
?
A year ago a new revision was coming RSN ...


> Peter Griffin (8):
>   vhost-user-video: Add a README.md with cheat sheet of commands
>   MAINTAINERS: Add virtio-video section
>   vhost-user-video: boiler plate code for vhost-user-video device
>   vhost-user-video: add meson subdir build logic
>   standard-headers: Add virtio_video.h
>   virtio_video: Add Fast Walsh-Hadamard Transform format
>   hw/display: add vhost-user-video-pci
>   tools/vhost-user-video: Add initial vhost-user-video vmm
> 
>  MAINTAINERS                                   |    8 +
>  hw/display/Kconfig                            |    5 +
>  hw/display/meson.build                        |    3 +
>  hw/display/vhost-user-video-pci.c             |   82 +
>  hw/display/vhost-user-video.c                 |  386 ++++
>  include/hw/virtio/vhost-user-video.h          |   41 +
>  include/standard-headers/linux/virtio_video.h |  484 +++++
>  tools/meson.build                             |    9 +
>  tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
>  tools/vhost-user-video/README.md              |   98 +
>  tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
>  tools/vhost-user-video/meson.build            |   10 +
>  tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
>  tools/vhost-user-video/v4l2_backend.h         |   99 +
>  tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
>  tools/vhost-user-video/virtio_video_helpers.h |  166 ++
>  tools/vhost-user-video/vuvideo.h              |   43 +
>  17 files changed, 5358 insertions(+)
>  create mode 100644 hw/display/vhost-user-video-pci.c
>  create mode 100644 hw/display/vhost-user-video.c
>  create mode 100644 include/hw/virtio/vhost-user-video.h
>  create mode 100644 include/standard-headers/linux/virtio_video.h
>  create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
>  create mode 100644 tools/vhost-user-video/README.md
>  create mode 100644 tools/vhost-user-video/main.c
>  create mode 100644 tools/vhost-user-video/meson.build
>  create mode 100644 tools/vhost-user-video/v4l2_backend.c
>  create mode 100644 tools/vhost-user-video/v4l2_backend.h
>  create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
>  create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
>  create mode 100644 tools/vhost-user-video/vuvideo.h
> 
> -- 
> 2.25.1

Re: [PATCH 0/8] virtio: Add vhost-user based Video decode

[Peter Griffin]

Hi Michael,

Apologies for the delaying in replying to you, I didn't see your email until now
whilst replying to Alex review feedback.

On Tue, 11 Jan 2022, Michael S. Tsirkin wrote:

> On Thu, Dec 09, 2021 at 02:55:53PM +0000, Peter Griffin wrote:
> > This series adds support for virtio-video decoder devices in Qemu
> > and also provides a vhost-user-video vmm implementation.
> > 
> > The vhost-user-video vmm currently parses virtio-vido v3 protocol
> > (as that is what the Linux frontend driver implements).
> > It then converts that to a v4l2 mem2mem stateful decoder device.
> > Currently this has been tested using v4l2 vicodec test driver in Linux
> > [1] but it is intended to be used with Arm SoCs which often implement
> > v4l2 stateful decoders/encoders drivers for their video accelerators.
> > 
> > The primary goal so far has been to allow continuing development
> > of virtio-video Linux frontend driver and testing with Qemu. Using
> > vicodec on the host allows a purely virtual dev env, and allows for
> > ci integration in the future by kernelci etc.
> > 
> > This series also adds the virtio_video.h header and adds the
> > FWHT format which is used by vicodec driver.
> > 
> > I have tested this VMM using v4l2-ctl from v4l2 utils in the guest
> > to do a video decode to a file. This can then be validated using ffplay
> > v4l2-compliance tool in the guest has also been run which stresses the
> > interface and issues lots of syscall level tests
> > 
> > See the README.md for example commands on how to configure guest kernel
> > and do a video decode using Qemu, vicodec using this VMM.
> > 
> > Linux virtio-video frontend driver code:
> > https://github.com/petegriffin/linux/commits/v5.10-virtio-video-latest
> 
> Are you going to post this btw?

Do you think there is much value in posting this version of the driver again?

As it is more or less the same as what Opensynergy folks posted to linux-media ml
plus a few fixups that have landed in the ChromeOS tree over time from using it
in their products and a few extra fixups from me to move it to a newer kernel
and fix a few bugs I found whilst working on the Qemu backend.

> 
> 
> > Qemu vmm code:
> > https://github.com/petegriffin/qemu/tree/vhost-virtio-video-master-v1
> > 
> > This is part of a wider initiative by Linaro called
> > "project Stratos" for which you can find information here:
> > 
> >   https://collaborate.linaro.org/display/STR/Stratos+Home
> > 
> > Applies cleanly to git://git.qemu.org/qemu.git master(a3607def89).
> > 
> > Thanks,
> > 
> > Peter.
> > 
> > [1] https://lwn.net/Articles/760650/
> 
> And the spec status?
> Does this match this spec:
> https://lore.kernel.org/linux-media/CAPBb6MUD_oYeUt8_bGRwAPCPam40Jtktz2F7+A3SO5PFRdUsiw@mail.gmail.com/T/
> ?
> A year ago a new revision was coming RSN ...
>

No unfortunately this VMM currently implements the v3 spec. The next task is to
update both the VMM and the frontend driver above to the v5 spec you linked to
above.

We would like to get the v5 spec voted on and integrated into the standard so
any advice you can give me as to the process for doing that would be appreciated :)

kind regards,

Peter.

> 
> > Peter Griffin (8):
> >   vhost-user-video: Add a README.md with cheat sheet of commands
> >   MAINTAINERS: Add virtio-video section
> >   vhost-user-video: boiler plate code for vhost-user-video device
> >   vhost-user-video: add meson subdir build logic
> >   standard-headers: Add virtio_video.h
> >   virtio_video: Add Fast Walsh-Hadamard Transform format
> >   hw/display: add vhost-user-video-pci
> >   tools/vhost-user-video: Add initial vhost-user-video vmm
> > 
> >  MAINTAINERS                                   |    8 +
> >  hw/display/Kconfig                            |    5 +
> >  hw/display/meson.build                        |    3 +
> >  hw/display/vhost-user-video-pci.c             |   82 +
> >  hw/display/vhost-user-video.c                 |  386 ++++
> >  include/hw/virtio/vhost-user-video.h          |   41 +
> >  include/standard-headers/linux/virtio_video.h |  484 +++++
> >  tools/meson.build                             |    9 +
> >  tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
> >  tools/vhost-user-video/README.md              |   98 +
> >  tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
> >  tools/vhost-user-video/meson.build            |   10 +
> >  tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
> >  tools/vhost-user-video/v4l2_backend.h         |   99 +
> >  tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
> >  tools/vhost-user-video/virtio_video_helpers.h |  166 ++
> >  tools/vhost-user-video/vuvideo.h              |   43 +
> >  17 files changed, 5358 insertions(+)
> >  create mode 100644 hw/display/vhost-user-video-pci.c
> >  create mode 100644 hw/display/vhost-user-video.c
> >  create mode 100644 include/hw/virtio/vhost-user-video.h
> >  create mode 100644 include/standard-headers/linux/virtio_video.h
> >  create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
> >  create mode 100644 tools/vhost-user-video/README.md
> >  create mode 100644 tools/vhost-user-video/main.c
> >  create mode 100644 tools/vhost-user-video/meson.build
> >  create mode 100644 tools/vhost-user-video/v4l2_backend.c
> >  create mode 100644 tools/vhost-user-video/v4l2_backend.h
> >  create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
> >  create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
> >  create mode 100644 tools/vhost-user-video/vuvideo.h
> > 
> > -- 
> > 2.25.1
> 

Re: [PATCH 0/8] virtio: Add vhost-user based Video decode

[Michael S. Tsirkin]

On Thu, Feb 03, 2022 at 12:10:17PM +0000, Peter Griffin wrote:
> Hi Michael,
> 
> Apologies for the delaying in replying to you, I didn't see your email until now
> whilst replying to Alex review feedback.
> 
> On Tue, 11 Jan 2022, Michael S. Tsirkin wrote:
> 
> > On Thu, Dec 09, 2021 at 02:55:53PM +0000, Peter Griffin wrote:
> > > This series adds support for virtio-video decoder devices in Qemu
> > > and also provides a vhost-user-video vmm implementation.
> > > 
> > > The vhost-user-video vmm currently parses virtio-vido v3 protocol
> > > (as that is what the Linux frontend driver implements).
> > > It then converts that to a v4l2 mem2mem stateful decoder device.
> > > Currently this has been tested using v4l2 vicodec test driver in Linux
> > > [1] but it is intended to be used with Arm SoCs which often implement
> > > v4l2 stateful decoders/encoders drivers for their video accelerators.
> > > 
> > > The primary goal so far has been to allow continuing development
> > > of virtio-video Linux frontend driver and testing with Qemu. Using
> > > vicodec on the host allows a purely virtual dev env, and allows for
> > > ci integration in the future by kernelci etc.
> > > 
> > > This series also adds the virtio_video.h header and adds the
> > > FWHT format which is used by vicodec driver.
> > > 
> > > I have tested this VMM using v4l2-ctl from v4l2 utils in the guest
> > > to do a video decode to a file. This can then be validated using ffplay
> > > v4l2-compliance tool in the guest has also been run which stresses the
> > > interface and issues lots of syscall level tests
> > > 
> > > See the README.md for example commands on how to configure guest kernel
> > > and do a video decode using Qemu, vicodec using this VMM.
> > > 
> > > Linux virtio-video frontend driver code:
> > > https://github.com/petegriffin/linux/commits/v5.10-virtio-video-latest
> > 
> > Are you going to post this btw?
> 
> Do you think there is much value in posting this version of the driver again?
> 
> As it is more or less the same as what Opensynergy folks posted to linux-media ml
> plus a few fixups that have landed in the ChromeOS tree over time from using it
> in their products and a few extra fixups from me to move it to a newer kernel
> and fix a few bugs I found whilst working on the Qemu backend.


I guess it needs to be updated to v5 spec then?

> > 
> > 
> > > Qemu vmm code:
> > > https://github.com/petegriffin/qemu/tree/vhost-virtio-video-master-v1
> > > 
> > > This is part of a wider initiative by Linaro called
> > > "project Stratos" for which you can find information here:
> > > 
> > >   https://collaborate.linaro.org/display/STR/Stratos+Home
> > > 
> > > Applies cleanly to git://git.qemu.org/qemu.git master(a3607def89).
> > > 
> > > Thanks,
> > > 
> > > Peter.
> > > 
> > > [1] https://lwn.net/Articles/760650/
> > 
> > And the spec status?
> > Does this match this spec:
> > https://lore.kernel.org/linux-media/CAPBb6MUD_oYeUt8_bGRwAPCPam40Jtktz2F7+A3SO5PFRdUsiw@mail.gmail.com/T/
> > ?
> > A year ago a new revision was coming RSN ...
> >
> 
> No unfortunately this VMM currently implements the v3 spec. The next task is to
> update both the VMM and the frontend driver above to the v5 spec you linked to
> above.
> 
> We would like to get the v5 spec voted on and integrated into the standard so
> any advice you can give me as to the process for doing that would be appreciated :)
> 
> kind regards,
> 
> Peter.

I think as with most cases where the TC lacks experts in the specific
field, we'd like to get acks from experts, one way to do that
is by posting the driver and soliciting acks.
Ideally we'd like to see an open source driver and device
implementations alongside the spec (preferably qemu as it's still
more popular than rust vmm and more people read C than rust).

> > 
> > > Peter Griffin (8):
> > >   vhost-user-video: Add a README.md with cheat sheet of commands
> > >   MAINTAINERS: Add virtio-video section
> > >   vhost-user-video: boiler plate code for vhost-user-video device
> > >   vhost-user-video: add meson subdir build logic
> > >   standard-headers: Add virtio_video.h
> > >   virtio_video: Add Fast Walsh-Hadamard Transform format
> > >   hw/display: add vhost-user-video-pci
> > >   tools/vhost-user-video: Add initial vhost-user-video vmm
> > > 
> > >  MAINTAINERS                                   |    8 +
> > >  hw/display/Kconfig                            |    5 +
> > >  hw/display/meson.build                        |    3 +
> > >  hw/display/vhost-user-video-pci.c             |   82 +
> > >  hw/display/vhost-user-video.c                 |  386 ++++
> > >  include/hw/virtio/vhost-user-video.h          |   41 +
> > >  include/standard-headers/linux/virtio_video.h |  484 +++++
> > >  tools/meson.build                             |    9 +
> > >  tools/vhost-user-video/50-qemu-rpmb.json.in   |    5 +
> > >  tools/vhost-user-video/README.md              |   98 +
> > >  tools/vhost-user-video/main.c                 | 1680 ++++++++++++++++
> > >  tools/vhost-user-video/meson.build            |   10 +
> > >  tools/vhost-user-video/v4l2_backend.c         | 1777 +++++++++++++++++
> > >  tools/vhost-user-video/v4l2_backend.h         |   99 +
> > >  tools/vhost-user-video/virtio_video_helpers.c |  462 +++++
> > >  tools/vhost-user-video/virtio_video_helpers.h |  166 ++
> > >  tools/vhost-user-video/vuvideo.h              |   43 +
> > >  17 files changed, 5358 insertions(+)
> > >  create mode 100644 hw/display/vhost-user-video-pci.c
> > >  create mode 100644 hw/display/vhost-user-video.c
> > >  create mode 100644 include/hw/virtio/vhost-user-video.h
> > >  create mode 100644 include/standard-headers/linux/virtio_video.h
> > >  create mode 100644 tools/vhost-user-video/50-qemu-rpmb.json.in
> > >  create mode 100644 tools/vhost-user-video/README.md
> > >  create mode 100644 tools/vhost-user-video/main.c
> > >  create mode 100644 tools/vhost-user-video/meson.build
> > >  create mode 100644 tools/vhost-user-video/v4l2_backend.c
> > >  create mode 100644 tools/vhost-user-video/v4l2_backend.h
> > >  create mode 100644 tools/vhost-user-video/virtio_video_helpers.c
> > >  create mode 100644 tools/vhost-user-video/virtio_video_helpers.h
> > >  create mode 100644 tools/vhost-user-video/vuvideo.h
> > > 
> > > -- 
> > > 2.25.1
> > 

Re: [PATCH 0/8] virtio: Add vhost-user based Video decode

[Alex Bennée]

Peter Griffin <peter.griffin@linaro.org> writes:

> This series adds support for virtio-video decoder devices in Qemu
> and also provides a vhost-user-video vmm implementation.

This brings up a bunch of failures in CI:

  https://gitlab.com/stsquad/qemu/-/pipelines/445691849/failures

A bunch are probably solved with masking the build when VHOST_USER is
not available but there seem to be some compiler warnings as well which
are probably worth looking into.

Sorry I took so long to get to the review!

-- 
Alex Bennée

Re: [PATCH 0/8] virtio: Add vhost-user based Video decode

[Peter Griffin]

Hi Alex,

On Tue, 11 Jan 2022, Alex Bennée wrote:

> 
> Peter Griffin <peter.griffin@linaro.org> writes:
> 
> > This series adds support for virtio-video decoder devices in Qemu
> > and also provides a vhost-user-video vmm implementation.
> 
> This brings up a bunch of failures in CI:
> 
>   https://gitlab.com/stsquad/qemu/-/pipelines/445691849/failures
> 
> A bunch are probably solved with masking the build when VHOST_USER is
> not available but there seem to be some compiler warnings as well which
> are probably worth looking into.
> 
> Sorry I took so long to get to the review!

No worries, thanks for reviewing the series :) I will take a look at the warnings in the
link above and fix them up for v2.

Thanks,

Peter.