Re: [PATCH v4 00/15] Move HEVC stateless controls out of staging

Benjamin Gaignard
2022-04-06T12:52:02Z

Le 06/04/2022 à 14:46, Adam Ford a écrit :
> On Wed, Apr 6, 2022 at 7:41 AM Benjamin Gaignard
> <benjamin.gaignard@collabora.com> wrote:
>>
>> Le 06/04/2022 à 14:28, Adam Ford a écrit :
>>> On Wed, Apr 6, 2022 at 1:56 AM Benjamin Gaignard
>>> <benjamin.gaignard@collabora.com> wrote:
>>>> Le 05/04/2022 à 23:27, Adam Ford a écrit :
>>>>> On Mon, Apr 4, 2022 at 10:56 AM Benjamin Gaignard
>>>>> <benjamin.gaignard@collabora.com> wrote:
>>>>>> Le 02/04/2022 à 18:59, Adam Ford a écrit :
>>>>>>> On Sat, Apr 2, 2022 at 11:22 AM Adam Ford <aford173@gmail.com> wrote:
>>>>>>>> On Fri, Apr 1, 2022 at 8:18 AM Benjamin Gaignard
>>>>>>>> <benjamin.gaignard@collabora.com> wrote:
>>>>>>>>> Le 31/03/2022 à 08:53, Benjamin Gaignard a écrit :
>>>>>>>>>> Le 30/03/2022 à 20:52, Adam Ford a écrit :
>>>>>>>>>>> On Wed, Mar 30, 2022 at 2:53 AM Benjamin Gaignard
>>>>>>>>>>> <benjamin.gaignard@collabora.com> wrote:
>>>>>>>>>>>> Le 28/02/2022 à 15:08, Benjamin Gaignard a écrit :
>>>>>>>>>>>>> This series aims to make HEVC uapi stable and usable for hardware
>>>>>>>>>>>>> decoder. HEVC uapi is used by 2 mainlined drivers (Cedrus and Hantro)
>>>>>>>>>>>>> and 2 out of the tree drivers (rkvdec and RPI).
>>>>>>>>>>>>>
>>>>>>>>>>>>> After the remarks done on version 2, I have completely reworked to
>>>>>>>>>>>>> patches
>>>>>>>>>>>>> split so changelogs are meaningless. I have also drop "RFC" from the
>>>>>>>>>>>>> titles.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Version 4:
>>>>>>>>>>>>> - Add num_entry_point_offsets field in struct
>>>>>>>>>>>>> v4l2_ctrl_hevc_slice_params
>>>>>>>>>>>>> - Fix V4L2_CID_STATELESS_HEVC_ENTRY_POINT_OFFSETS name
>>>>>>>>>>>>> - Initialize control V4L2_CID_STATELESS_HEVC_ENTRY_POINT_OFFSETS
>>>>>>>>>>>>> - Fix space/tab issue in kernel-doc
>>>>>>>>>>>>> - Add patch to change data_bit_offset definition
>>>>>>>>>>>>> - Fix hantro-media SPDX license
>>>>>>>>>>>>> - put controls under stateless section in v4l2-ctrls-defs.c
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the end fluster tests results on IMX8MQ is 77/147 for HEVC codec.
>>>>>>>>>>>> Dear reviewers,
>>>>>>>>>>>>
>>>>>>>>>>>> This series is waiting for your feedback,
>>>>>>>>>>> I tried several times with the suggested repos for both the kernel and
>>>>>>>>>>> g-streamer without success getting Fluster to pass any tests on the
>>>>>>>>>>> imx8mq. I can try again but I likely won't get to it until this
>>>>>>>>>>> weekend. If I can get it working, I'll test both the 8mq and 8mm.
>>>>>>>>>> Thanks a lot for that.
>>>>>>>>>>
>>>>>>>>>> Benjamin
>>>>>>>>> Adam,
>>>>>>>>>
>>>>>>>>> You may need to check if h265parse and v4l2slh265dec are available on your board.
>>>>>>>> I ran gst-inspect to see what showed up with 265 in the name.
>>>>>>>>
>>>>>>>> # gst-inspect-1.0 |grep 265
>>>>>>>> libav: avdec_h265: libav HEVC (High Efficiency Video Coding) decoder
>>>>>>>> rtp: rtph265depay: RTP H265 depayloader
>>>>>>>> rtp: rtph265pay: RTP H265 payloader
>>>>>>>> typefindfunctions: video/x-h265: h265, x265, 265
>>>>>>>> v4l2codecs: v4l2slh265dec: V4L2 Stateless H.265 Video Decoder
>>>>>>>> videoparsersbad: h265parse: H.265 parser
>>>>>>>>
>>>>>>>> It appears I have both h265parse and v4l2slh265dec.
>>>>>>>>
>>>>>>>>> fluster check if v4l2slh265dec is working fine with this command line:
>>>>>>>>>
>>>>>>>>> gst-launch-1.0 appsrc num-buffers=0 ! h265parse ! v4l2slh265dec ! fakesink
>>>>>>>>>
>>>>>>>>> so if one of them is missing it won't work.
>>>>>>>> gst-launch-1.0 appsrc num-buffers=0 ! h265parse ! v4l2slh265dec ! fakesink
>>>>>>>> Setting pipeline to PAUSED ...
>>>>>>>> 0:00:00.098389938 526 0xaaaaf9d86ac0 ERROR v4l2codecs-decoder
>>>>>>>> gstv4l2decoder.c:725:gst_v4l2_decoder_get_controls:<v4l2decoder2>
>>>>>>>> VIDIOC_G_EXT_CTRLS failed: Invalid argument
>>>>>>>> ERROR: from element
>>>>>>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0: Driver did not
>>>>>>>> report framing and start code method.
>>>>>>>> Additional debug info:
>>>>>>>> ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codech265dec.c(155):
>>>>>>>> gst_v4l2_codec_h265_dec_open ():
>>>>>>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0:
>>>>>>>> gst_v4l2_decoder_get_controls() failed: Invalid argument
>>>>>>>> ERROR: pipeline doesn't want to preroll.
>>>>>>>> ERROR: from element
>>>>>>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0: Could not
>>>>>>>> initialize supporting library.
>>>>>>>> Additional debug info:
>>>>>>>> ../subprojects/gst-plugins-base/gst-libs/gst/video/gstvideodecoder.c(2909):
>>>>>>>> gst_video_decoder_change_state ():
>>>>>>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0:
>>>>>>>> Failed to open decoder
>>>>>>>> ERROR: pipeline doesn't want to preroll.
>>>>>>>> Failed to set pipeline to PAUSED.
>>>>>>>> Setting pipeline to NULL ...
>>>>>>>> Freeing pipeline ...
>>>>>>>>
>>>>>>>> Does this mean I have a wrong version of the kernel and/or incomplete patches?
>>>>>>> I double checked the branches.
>>>>>>>
>>>>>>> Kernel:
>>>>>>> https://gitlab.collabora.com/benjamin.gaignard/for-upstream.git
>>>>>>> branch: origin/HEVC_UAPI_V4
>>>>>>>
>>>>>>> Gstreamer:
>>>>>>> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer.git
>>>>>>> branch: origin/benjamin.gaignard1/gstreamer-HEVC_aligned_with_kernel_5.15
>>>>>>>
>>>>>>>
>>>>>>> I am still not able to run h.265/HEVC tests.
>>>>>> Hello Adam,
>>>>>>
>>>>>> I have updated the following branches with the versions I have used today:
>>>>>>
>>>>>> Kernel:
>>>>>> https://gitlab.collabora.com/benjamin.gaignard/for-upstream.git
>>>>>> branch: origin/HEVC_UAPI_V5 only one change in documentation vs version 4 but rebased in v5.18-rc1
>>>>>>
>>>>>> Gstreamer:
>>>>>> https://gitlab.freedesktop.org/benjamin.gaignard1/gstreamer.git
>>>>>> branch: origin/benjamin.gaignard1/gstreamer-HEVC_aligned_with_kernel_5.15 updated on the latest GST main branch
>>>>>>
>>>>>> I hope this will work fine this time.
>>>>> I wish I had better news for you:
>>>>>
>>>>> dmesg shows the hantro driver is being loaded:
>>>>>
>>>>> [ 38.612243] hantro-vpu 38300000.video-codec: registered
>>>>> nxp,imx8mq-vpu-g1-dec as /dev/video0
>>>>> [ 38.612618] hantro-vpu 38310000.video-codec: registered
>>>>> nxp,imx8mq-vpu-g2-dec as /dev/video1
>>>>>
>>>>> # gst-inspect-1.0 |grep 265
>>>>> libav: avdec_h265: libav HEVC (High Efficiency Video Coding) decoder
>>>>> rtp: rtph265depay: RTP H265 depayloader
>>>>> rtp: rtph265pay: RTP H265 payloader
>>>>> typefindfunctions: video/x-h265: h265, x265, 265
>>>>> v4l2codecs: v4l2slh265dec: V4L2 Stateless H.265 Video Decoder
>>>>> videoparsersbad: h265parse: H.265 parser
>>>>>
>>>>> Fluster reports:
>>>>> GStreamer-H.265-V4L2SL-Gst1.0: GStreamer H.265 V4L2SL decoder for
>>>>> GStreamer 1.0... ❌
>>>> Still the same error about non supported control ?
>>> # gst-lagst-launch-1.0 appsrc num-buffers=0 ! h265parse !
>>> v4l2slh265dec ! fakesink
>>> Setting pipeline to PAUSED ...
>>> 0:00:01.704385508 420 0xaaaaec0a76c0 ERROR v4l2codecs-decoder
>>> gstv4l2decoder.c:725:gst_v4l2_decoder_get_controls:<v4l2decoder2>
>>> VIDIOC_G_EXT_CTRLS failed: Invalid argument
>>> ERROR: from element
>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0: Driver did not
>>> report framing and start code method.
>>> Additional debug info:
>>> ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codech265dec.c(155):
>>> gst_v4l2_codec_h265_dec_open ():
>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0:
>>> gst_v4l2_decoder_get_controls() failed: Invalid argument
>>> ERROR: pipeline doesn't want to preroll.
>>> ERROR: from element
>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0: Could not
>>> initialize supporting library.
>>> Additional debug info:
>>> ../subprojects/gst-plugins-base/gst-libs/gst/video/gstvideodecoder.c(2909):
>>> gst_video_decoder_change_state ():
>>> /GstPipeline:pipeline0/v4l2slh265dec:v4l2slh265dec0:
>>> Failed to open decoder
>>> ERROR: pipeline doesn't want to preroll.
>>> Failed to set pipeline to PAUSED.
>>> Setting pipeline to NULL ...
>>> Freeing pipeline ...
>> What is the result of v4l2-ctl ?
>> Here I have:
>> v4l2-ctl --all -d 1
>> Driver Info:
>> Driver name : hantro-vpu
>> Card type : nxp,imx8mq-vpu-g2-dec
>> Bus info : platform: hantro-vpu
>> Driver version : 5.18.0
>> Capabilities : 0x84204000
>> Video Memory-to-Memory Multiplanar
>> Streaming
>> Extended Pix Format
>> Device Capabilities
>> Device Caps : 0x04204000
>> Video Memory-to-Memory Multiplanar
>> Streaming
>> Extended Pix Format
>> Media Driver Info:
>> Driver name : hantro-vpu
>> Model : hantro-vpu
>> Serial :
>> Bus info : platform: hantro-vpu
>> Media version : 5.18.0
>> Hardware revision: 0x00000000 (0)
>> Driver version : 5.18.0
>> Interface Info:
>> ID : 0x0300000c
>> Type : V4L Video
>> Entity Info:
>> ID : 0x00000001 (1)
>> Name : nxp,imx8mq-vpu-g2-dec-source
>> Function : V4L2 I/O
>> Pad 0x01000002 : 0: Source
>> Link 0x02000008: to remote pad 0x1000004 of entity 'nxp,imx8mq-vpu-g2-dec-proc': Data, Enabled, Immutable
>> Priority: 2
>> Format Video Capture Multiplanar:
>> Width/Height : 48/48
>> Pixel Format : 'VT12' (Y/CbCr 4:2:0 (4x4 Linear))
>> Field : None
>> Number of planes : 1
>> Flags :
>> Colorspace : JPEG
>> Transfer Function : Default
>> YCbCr/HSV Encoding: Default
>> Quantization : Default
>> Plane 0 :
>> Bytes per Line : 48
>> Size Image : 3600
>> Format Video Output Multiplanar:
>> Width/Height : 48/48
>> Pixel Format : 'S265' (HEVC Parsed Slice Data)
>> Field : None
>> Number of planes : 1
>> Flags :
>> Colorspace : JPEG
>> Transfer Function : Default
>> YCbCr/HSV Encoding: Default
>> Quantization : Default
>> Plane 0 :
>> Bytes per Line : 0
>> Size Image : 4608
>>
>> User Controls
>>
>> hantro_hevc_slice_header_skip_b 0x00981a80 (int) : min=0 max=256 step=1 default=0 value=0
>>
>> Codec Controls
>>
>> hevc_profile 0x00990b67 (menu) : min=0 max=2 default=0 value=0
>> 0: Main
>> 1: Main Still Picture
>> 2: Main 10
>> hevc_level 0x00990b68 (menu) : min=0 max=8 default=0 value=0
>> 0: 1
>> 1: 2
>> 2: 2.1
>> 3: 3
>> 4: 3.1
>> 5: 4
>> 6: 4.1
>> 7: 5
>> 8: 5.1
>>
>> Stateless Codec Controls
>>
>> vp9_frame_decode_parameters 0x00a40a2c (unknown): type=261 flags=has-payload
>> vp9_probabilities_updates 0x00a40a2d (unknown): type=260 flags=has-payload
>> hevc_sequence_parameter_set 0x00a40a90 (unknown): type=270 flags=has-payload
>> hevc_picture_parameter_set 0x00a40a91 (unknown): type=271 flags=has-payload
>> hevc_scaling_matrix 0x00a40a93 (unknown): type=273 flags=has-payload
>> hevc_decode_parameters 0x00a40a94 (unknown): type=274 flags=has-payload
>> hevc_decode_mode 0x00a40a95 (menu) : min=1 max=1 default=1 value=1
>> 1: Frame-Based
>> hevc_start_code 0x00a40a96 (menu) : min=1 max=1 default=1 value=1
>> 1: Annex B Start Code
>>
> Mine looks like:
>
> # v4l2-ctl --all -d 1
> Driver Info:
> Driver name : hantro-vpu
> Card type : nxp,imx8mq-vpu-g2-dec
> Bus info : platform: hantro-vpu
> Driver version : 5.18.0
> Capabilities : 0x84204000
> Video Memory-to-Memory Multiplanar
> Streaming
> Extended Pix Format
> Device Capabilities
> Device Caps : 0x04204000
> Video Memory-to-Memory Multiplanar
> Streaming
> Extended Pix Format
> Media Driver Info:
> Driver name : hantro-vpu
> Model : hantro-vpu
> Serial :
> Bus info : platform: hantro-vpu
> Media version : 5.18.0
> Hardware revision: 0x00000000 (0)
> Driver version : 5.18.0
> Interface Info:
> ID : 0x0300000c
> Type : V4L Video
> Entity Info:
> ID : 0x00000001 (1)
> Name : nxp,imx8mq-vpu-g2-dec-source
> Function : V4L2 I/O
> Pad 0x01000002 : 0: Source
> Link 0x02000008: to remote pad 0x1000004 of entity
> 'nxp,imx8mq-vpu-g2-dec-proc': Data, Enabled, Immutable
> Priority: 2
> Format Video Capture Multiplanar:
> Width/Height : 48/48
> Pixel Format : 'VT12' (Y/CbCr 4:2:0 (4x4 Linear))
> Field : None
> Number of planes : 1
> Flags :
> Colorspace : JPEG
> Transfer Function : Default
> YCbCr/HSV Encoding: Default
> Quantization : Default
> Plane 0 :
> Bytes per Line : 48
> Size Image : 3600
> Format Video Output Multiplanar:
> Width/Height : 48/48
> Pixel Format : 'S265' (HEVC Parsed Slice Data)
> Field : None
> Number of planes : 1
> Flags :
> Colorspace : JPEG
> Transfer Function : Default
> YCbCr/HSV Encoding: Default
> Quantization : Default
> Plane 0 :
> Bytes per Line : 0
> Size Image : 4608
>
> User Controls
>
> hantro_hevc_slice_header_skip_b 0x00981a80 (int) : min=0 max=256
> step=1 default=0 value=0
>
> Codec Controls
>
> hevc_profile 0x00990b67 (menu) : min=0 max=2
> default=0 value=0
> 0: Main
> 1: Main Still Picture
> 2: Main 10
> hevc_level 0x00990b68 (menu) : min=0 max=8
> default=0 value=0
> 0: 1
> 1: 2
> 2: 2.1
> 3: 3
> 4: 3.1
> 5: 4
> 6: 4.1
> 7: 5
> 8: 5.1
>
> Stateless Codec Controls
>
> vp9_frame_decode_parameters 0x00a40a2c (unknown): type=261 flags=has-payload
> vp9_probabilities_updates 0x00a40a2d (unknown): type=260 flags=has-payload
> hevc_sequence_parameter_set 0x00a40a90 (unknown): type=270 flags=has-payload
> hevc_picture_parameter_set 0x00a40a91 (unknown): type=271 flags=has-payload
> hevc_scaling_matrix 0x00a40a93 (unknown): type=273 flags=has-payload
> hevc_decode_parameters 0x00a40a94 (unknown): type=274 flags=has-payload
> hevc_decode_mode 0x00a40a95 (menu) : min=1 max=1
> default=1 value=1
> 1: Frame-Based
> hevc_start_code 0x00a40a96 (menu) : min=1 max=1
> default=1 value=1
> 1: Annex B Start Code

It is the same so that suggest the issue is coming from GStreamer plugin.

>
>>>> Benjamin
>>>>
>>>>> adam
>>>>>> Benjamin
>>>>>>
>>>>>>> adam
>>>>>>>> adam
>>>>>>>>> Regards,
>>>>>>>>> Benjamin
>>>>>>>>>
>>>>>>>>>>> adam
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Benjamin
>>>>>>>>>>>>
>>>>>>>>>>>>> Benjamin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Benjamin Gaignard (12):
>>>>>>>>>>>>> media: uapi: HEVC: Add missing fields in HEVC controls
>>>>>>>>>>>>> media: uapi: HEVC: Rename HEVC stateless controls with STATELESS
>>>>>>>>>>>>> prefix
>>>>>>>>>>>>> media: uapi: HEVC: Add document uAPI structure
>>>>>>>>>>>>> media: uapi: HEVC: Define V4L2_CID_STATELESS_HEVC_SLICE_PARAMS
>>>>>>>>>>>>> as a
>>>>>>>>>>>>> dynamic array
>>>>>>>>>>>>> media: uapi: Move parsed HEVC pixel format out of staging
>>>>>>>>>>>>> media: uapi: Add V4L2_CID_STATELESS_HEVC_ENTRY_POINT_OFFSETS
>>>>>>>>>>>>> control
>>>>>>>>>>>>> media: uapi: Move the HEVC stateless control type out of staging
>>>>>>>>>>>>> media: controls: Log HEVC stateless control in .std_log
>>>>>>>>>>>>> media: uapi: Create a dedicated header for Hantro control
>>>>>>>>>>>>> media: uapi: HEVC: fix padding in v4l2 control structures
>>>>>>>>>>>>> media: uapi: Change data_bit_offset definition
>>>>>>>>>>>>> media: uapi: move HEVC stateless controls out of staging
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hans Verkuil (3):
>>>>>>>>>>>>> videodev2.h: add V4L2_CTRL_FLAG_DYNAMIC_ARRAY
>>>>>>>>>>>>> v4l2-ctrls: add support for dynamically allocated arrays.
>>>>>>>>>>>>> vivid: add dynamic array test control
>>>>>>>>>>>>>
>>>>>>>>>>>>> .../userspace-api/media/drivers/hantro.rst | 5 -
>>>>>>>>>>>>> .../media/v4l/ext-ctrls-codec-stateless.rst | 833
>>>>>>>>>>>>> ++++++++++++++++++
>>>>>>>>>>>>> .../media/v4l/ext-ctrls-codec.rst | 780
>>>>>>>>>>>>> ----------------
>>>>>>>>>>>>> .../media/v4l/pixfmt-compressed.rst | 7 +-
>>>>>>>>>>>>> .../media/v4l/vidioc-g-ext-ctrls.rst | 20 +
>>>>>>>>>>>>> .../media/v4l/vidioc-queryctrl.rst | 8 +
>>>>>>>>>>>>> .../media/videodev2.h.rst.exceptions | 5 +
>>>>>>>>>>>>> .../media/test-drivers/vivid/vivid-ctrls.c | 15 +
>>>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ctrls-api.c | 103 ++-
>>>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ctrls-core.c | 198 ++++-
>>>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 37 +-
>>>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ctrls-priv.h | 3 +-
>>>>>>>>>>>>> drivers/media/v4l2-core/v4l2-ctrls-request.c | 13 +-
>>>>>>>>>>>>> drivers/staging/media/hantro/hantro_drv.c | 27 +-
>>>>>>>>>>>>> drivers/staging/media/hantro/hantro_hevc.c | 8 +-
>>>>>>>>>>>>> drivers/staging/media/sunxi/cedrus/cedrus.c | 24 +-
>>>>>>>>>>>>> .../staging/media/sunxi/cedrus/cedrus_dec.c | 10 +-
>>>>>>>>>>>>> .../staging/media/sunxi/cedrus/cedrus_h265.c | 2 +-
>>>>>>>>>>>>> include/media/hevc-ctrls.h | 250 ------
>>>>>>>>>>>>> include/media/v4l2-ctrls.h | 48 +-
>>>>>>>>>>>>> include/uapi/linux/hantro-media.h | 19 +
>>>>>>>>>>>>> include/uapi/linux/v4l2-controls.h | 439 +++++++++
>>>>>>>>>>>>> include/uapi/linux/videodev2.h | 13 +
>>>>>>>>>>>>> 23 files changed, 1697 insertions(+), 1170 deletions(-)
>>>>>>>>>>>>> delete mode 100644 include/media/hevc-ctrls.h
>>>>>>>>>>>>> create mode 100644 include/uapi/linux/hantro-media.h
>>>>>>>>>>>>>