From: Nicolas Dufresne <nicolas.dufresne@collabora.com>
To: Paul Kocialkowski <paulk@sys-base.io>,
devicetree@vger.kernel.org, imx@lists.linux.dev,
linux-arm-kernel@lists.infradead.org,
linux-kernel@vger.kernel.org, linux-media@vger.kernel.org
Cc: Rob Herring <robh@kernel.org>,
Krzysztof Kozlowski <krzk+dt@kernel.org>,
Conor Dooley <conor+dt@kernel.org>,
Shawn Guo <shawnguo@kernel.org>,
Sascha Hauer <s.hauer@pengutronix.de>,
Pengutronix Kernel Team <kernel@pengutronix.de>,
Benjamin Gaignard <benjamin.gaignard@collabora.com>,
Philipp Zabel <p.zabel@pengutronix.de>,
Mauro Carvalho Chehab <mchehab@kernel.org>,
Hans Verkuil <hverkuil@kernel.org>,
Marco Felsch <m.felsch@pengutronix.de>,
Michael Tretter <m.tretter@pengutronix.de>
Subject: Re: [PATCH 02/14] media: uapi: Add H.264 stateless encode support
Date: Tue, 17 Nov 2026 16:30:11 -0500 [thread overview]
Message-ID: <1a1feaf854e6e1c34af468b7501988f2cf68ebca.camel@collabora.com> (raw)
In-Reply-To: <20260522101653.2565125-3-paulk@sys-base.io>
[-- Attachment #1: Type: text/plain, Size: 11186 bytes --]
Hi,
Le vendredi 22 mai 2026 à 12:16 +0200, Paul Kocialkowski a écrit :
> This introduces the stateless H.264 encode params control that allows
> configuring an H.264 encode run.
>
> Note that the current uAPI does not support explicit passing of
> references and works following the sliding window decoded reference
> picture marking process. This may change in the future.
Since its no longer an RFC, you have to support this with a how it can change in
the future without breaking the API. Its fine to create a entry profile for
encoders of course, we need a plan though.
>
> Signed-off-by: Paul Kocialkowski <paulk@sys-base.io>
> ---
> drivers/media/v4l2-core/v4l2-ctrls-core.c | 62 +++++++++++++++++++++++
> drivers/media/v4l2-core/v4l2-ctrls-defs.c | 4 ++
> include/media/v4l2-ctrls.h | 2 +
> include/uapi/linux/v4l2-controls.h | 33 ++++++++++++
> include/uapi/linux/videodev2.h | 1 +
> 5 files changed, 102 insertions(+)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-core.c b/drivers/media/v4l2-core/v4l2-ctrls-core.c
> index 85d07ef44f62..48217811b0f1 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls-core.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls-core.c
> @@ -382,6 +382,9 @@ void v4l2_ctrl_type_op_log(const struct v4l2_ctrl *ctrl)
> case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> pr_cont("H264_PRED_WEIGHTS");
> break;
> + case V4L2_CTRL_TYPE_H264_ENCODE_PARAMS:
> + pr_cont("H264_ENCODE_PARAMS");
> + break;
> case V4L2_CTRL_TYPE_FWHT_PARAMS:
> pr_cont("FWHT_PARAMS");
> break;
> @@ -880,6 +883,7 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
> struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
> struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
> struct v4l2_ctrl_h264_decode_params *p_h264_dec_params;
> + struct v4l2_ctrl_h264_encode_params *p_h264_enc_params;
> struct v4l2_ctrl_hevc_sps *p_hevc_sps;
> struct v4l2_ctrl_hevc_pps *p_hevc_pps;
> struct v4l2_ctrl_hdr10_mastering_display *p_hdr10_mastering;
> @@ -1102,6 +1106,61 @@ static int std_validate_compound(const struct v4l2_ctrl *ctrl, u32 idx,
> zero_reserved(*p_h264_dec_params);
> break;
>
> + case V4L2_CTRL_TYPE_H264_ENCODE_PARAMS:
> + p_h264_enc_params = p;
> +
> + if (p_h264_enc_params->slice_type != V4L2_H264_SLICE_TYPE_B)
> + p_h264_enc_params->flags &=
> + ~V4L2_H264_ENCODE_FLAG_DIRECT_SPATIAL_MV_PRED;
> + if (!(p_h264_enc_params->flags & V4L2_H264_ENCODE_FLAG_IDR_PIC) ||
> + !p_h264_enc_params->nal_ref_idc)
> + p_h264_enc_params->flags &=
> + ~V4L2_H264_ENCODE_FLAG_LONG_TERM_REFERENCE;
> +
> + if (p_h264_enc_params->flags & V4L2_H264_ENCODE_FLAG_IDR_PIC &&
> + p_h264_enc_params->slice_type != V4L2_H264_SLICE_TYPE_I)
> + return -EINVAL;
> + if (p_h264_enc_params->colour_plane_id > 2)
> + return -EINVAL;
> + if (p_h264_enc_params->cabac_init_idc > 2)
> + return -EINVAL;
> + if (p_h264_enc_params->disable_deblocking_filter_idc > 2)
> + return -EINVAL;
> + if (p_h264_enc_params->slice_alpha_c0_offset_div2 < -6 ||
> + p_h264_enc_params->slice_alpha_c0_offset_div2 > 6)
> + return -EINVAL;
> + if (p_h264_enc_params->slice_beta_offset_div2 < -6 ||
> + p_h264_enc_params->slice_beta_offset_div2 > 6)
> + return -EINVAL;
> +
> + if (p_h264_enc_params->slice_type == V4L2_H264_SLICE_TYPE_I ||
> + p_h264_enc_params->slice_type == V4L2_H264_SLICE_TYPE_SI)
> + p_h264_enc_params->num_ref_idx_l0_active_minus1 = 0;
> + if (p_h264_enc_params->slice_type != V4L2_H264_SLICE_TYPE_B)
> + p_h264_enc_params->num_ref_idx_l1_active_minus1 = 0;
> +
> + if (p_h264_enc_params->flags & V4L2_H264_ENCODE_FLAG_IDR_PIC) {
> + p_h264_enc_params->frame_num = 0;
> + p_h264_enc_params->pic_order_cnt_lsb = 0;
> + p_h264_enc_params->delta_pic_order_cnt_bottom = 0;
> + p_h264_enc_params->delta_pic_order_cnt0 = 0;
> + p_h264_enc_params->delta_pic_order_cnt1 = 0;
> + } else {
> + p_h264_enc_params->idr_pic_id = 0;
> + }
> +
> + if (p_h264_enc_params->num_ref_idx_l0_active_minus1 >
> + (V4L2_H264_REF_LIST_LEN - 1))
> + return -EINVAL;
> + if (p_h264_enc_params->num_ref_idx_l1_active_minus1 >
> + (V4L2_H264_REF_LIST_LEN - 1))
> + return -EINVAL;
> + memset(&p_h264_enc_params->reserved0, 0,
> + sizeof(p_h264_enc_params->reserved0));
> + memset(&p_h264_enc_params->reserved1, 0,
> + sizeof(p_h264_enc_params->reserved1));
> + break;
> +
> case V4L2_CTRL_TYPE_VP8_FRAME:
> p_vp8_frame = p;
>
> @@ -1913,6 +1972,9 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
> case V4L2_CTRL_TYPE_H264_PRED_WEIGHTS:
> elem_size = sizeof(struct v4l2_ctrl_h264_pred_weights);
> break;
> + case V4L2_CTRL_TYPE_H264_ENCODE_PARAMS:
> + elem_size = sizeof(struct v4l2_ctrl_h264_encode_params);
> + break;
> case V4L2_CTRL_TYPE_VP8_FRAME:
> elem_size = sizeof(struct v4l2_ctrl_vp8_frame);
> break;
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls-defs.c b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> index ad41f65374e2..4fa6dc7a3b00 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls-defs.c
> @@ -1216,6 +1216,7 @@ const char *v4l2_ctrl_get_name(u32 id)
> case V4L2_CID_STATELESS_H264_PPS: return "H264 Picture Parameter Set";
> case V4L2_CID_STATELESS_H264_SCALING_MATRIX: return "H264 Scaling Matrix";
> case V4L2_CID_STATELESS_H264_PRED_WEIGHTS: return "H264 Prediction Weight Table";
> + case V4L2_CID_STATELESS_H264_ENCODE_PARAMS: return "H264 Encode Parameters";
> case V4L2_CID_STATELESS_H264_SLICE_PARAMS: return "H264 Slice Parameters";
> case V4L2_CID_STATELESS_H264_DECODE_PARAMS: return "H264 Decode Parameters";
> case V4L2_CID_STATELESS_FWHT_PARAMS: return "FWHT Stateless Parameters";
> @@ -1555,6 +1556,9 @@ void v4l2_ctrl_fill(u32 id, const char **name, enum v4l2_ctrl_type *type,
> case V4L2_CID_STATELESS_H264_PRED_WEIGHTS:
> *type = V4L2_CTRL_TYPE_H264_PRED_WEIGHTS;
> break;
> + case V4L2_CID_STATELESS_H264_ENCODE_PARAMS:
> + *type = V4L2_CTRL_TYPE_H264_ENCODE_PARAMS;
> + break;
> case V4L2_CID_STATELESS_VP8_FRAME:
> *type = V4L2_CTRL_TYPE_VP8_FRAME;
> break;
> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> index 31fc1bee3797..54133cda7bc7 100644
> --- a/include/media/v4l2-ctrls.h
> +++ b/include/media/v4l2-ctrls.h
> @@ -42,6 +42,7 @@ struct video_device;
> * @p_h264_scaling_matrix: Pointer to a struct v4l2_ctrl_h264_scaling_matrix.
> * @p_h264_slice_params: Pointer to a struct v4l2_ctrl_h264_slice_params.
> * @p_h264_decode_params: Pointer to a struct v4l2_ctrl_h264_decode_params.
> + * @p_h264_encode_params: Pointer to a struct v4l2_ctrl_h264_encode_params.
> * @p_h264_pred_weights: Pointer to a struct v4l2_ctrl_h264_pred_weights.
> * @p_vp8_frame: Pointer to a VP8 frame params structure.
> * @p_vp9_compressed_hdr_probs: Pointer to a VP9 frame compressed header probs structure.
> @@ -76,6 +77,7 @@ union v4l2_ctrl_ptr {
> struct v4l2_ctrl_h264_scaling_matrix *p_h264_scaling_matrix;
> struct v4l2_ctrl_h264_slice_params *p_h264_slice_params;
> struct v4l2_ctrl_h264_decode_params *p_h264_decode_params;
> + struct v4l2_ctrl_h264_encode_params *p_h264_encode_params;
> struct v4l2_ctrl_h264_pred_weights *p_h264_pred_weights;
> struct v4l2_ctrl_vp8_frame *p_vp8_frame;
> struct v4l2_ctrl_hevc_sps *p_hevc_sps;
> diff --git a/include/uapi/linux/v4l2-controls.h b/include/uapi/linux/v4l2-controls.h
> index 2d30107e047e..9add059ca02a 100644
> --- a/include/uapi/linux/v4l2-controls.h
> +++ b/include/uapi/linux/v4l2-controls.h
> @@ -1568,6 +1568,7 @@ struct v4l2_h264_reference {
>
> #define V4L2_H264_SLICE_FLAG_DIRECT_SPATIAL_MV_PRED 0x01
> #define V4L2_H264_SLICE_FLAG_SP_FOR_SWITCH 0x02
> +#define V4L2_H264_SLICE_FLAG_NUM_REF_IDX_ACTIVE_OVERRIDE 0x04
>
> #define V4L2_CID_STATELESS_H264_SLICE_PARAMS (V4L2_CID_CODEC_STATELESS_BASE + 6)
> /**
> @@ -1707,6 +1708,38 @@ struct v4l2_ctrl_h264_decode_params {
> __u32 flags;
> };
>
> +#define V4L2_H264_ENCODE_FLAG_IDR_PIC 0x01
> +#define V4L2_H264_ENCODE_FLAG_FIELD_PIC 0x02
> +#define V4L2_H264_ENCODE_FLAG_BOTTOM_FIELD 0x04
Just notice you are ahead of Vulkan here, they don't seem to have inerlaced
encoding, maybe they just want to discourage people to use that, and that's why
they only decode it :-D
> +#define V4L2_H264_ENCODE_FLAG_DIRECT_SPATIAL_MV_PRED 0x08
> +#define V4L2_H264_ENCODE_FLAG_NUM_REF_IDX_ACTIVE_OVERRIDE 0x10
> +#define V4L2_H264_ENCODE_FLAG_NO_OUTPUT_OF_PRIOR_PICS 0x20
> +#define V4L2_H264_ENCODE_FLAG_LONG_TERM_REFERENCE 0x40
> +
> +#define V4L2_CID_STATELESS_H264_ENCODE_PARAMS (V4L2_CID_CODEC_STATELESS_BASE + 8)
> +struct v4l2_ctrl_h264_encode_params {
> + __u8 nal_ref_idc;
This is fixed in this implementation, and will be derivable with support for
other reference model. I'd remove.
> + __u8 slice_type;
I think we should do like Vulkan here though. They split the frame invariant
(StdVideoEncodeH264PictureInfo) from the slice parameters
(StdVideoEncodeH264SliceHeader). slice_type does not have to be the same on all
slices, just like many more below.
> + __u8 pic_parameter_set_id;
> + __u8 colour_plane_id;
I'm not aware of any HW capable of != 0 here, so I'd be keen on dropping. We can
make it an encode_params_professional in the future, there will likely be more
things to add by then.
> + __u16 frame_num;
> + __u16 idr_pic_id;
> + __u16 pic_order_cnt_lsb;
> + __u8 reserved0[2];
> + __s32 delta_pic_order_cnt_bottom;
> + __s32 delta_pic_order_cnt0;
> + __s32 delta_pic_order_cnt1;
> + __u8 num_ref_idx_l0_active_minus1;
> + __u8 num_ref_idx_l1_active_minus1;
> + __u8 cabac_init_idc;
> + __s8 slice_qp_delta;
> + __u8 disable_deblocking_filter_idc;
> + __s8 slice_alpha_c0_offset_div2;
> + __s8 slice_beta_offset_div2;
> + __u8 reserved1[6];
This leads the next flag 1 byte off 32bit alignment, what the compiler will do
next can't be predicted. Did you forgot to run pahole?
> + __u32 flags; /* V4L2_H264_ENCODE_FLAG_ */
Missing something like temporal_id, first_mb_in_slice, while splitting the slice
params, these would become a dybamic array and that will define the number of
slices. No long term support I think?
That's all I could find on this pass.
Nicolas
> +};
> +
> /* Stateless FWHT control, used by the vicodec driver */
>
> /* Current FWHT version */
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index becd08fdbddb..1d5e27b65544 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -1964,6 +1964,7 @@ enum v4l2_ctrl_type {
> V4L2_CTRL_TYPE_H264_SLICE_PARAMS = 0x0203,
> V4L2_CTRL_TYPE_H264_DECODE_PARAMS = 0x0204,
> V4L2_CTRL_TYPE_H264_PRED_WEIGHTS = 0x0205,
> + V4L2_CTRL_TYPE_H264_ENCODE_PARAMS = 0x0206,
>
> V4L2_CTRL_TYPE_FWHT_PARAMS = 0x0220,
>
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
next prev parent reply other threads:[~2026-11-17 21:30 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-05-22 10:16 [PATCH 00/14] media: Add V4L2 H.264 stateless encode and VC8000E support Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 01/14] media: h264: Add a more generic reflist builder init Paul Kocialkowski
2026-11-17 21:06 ` Nicolas Dufresne
2026-05-22 10:16 ` [PATCH 02/14] media: uapi: Add H.264 stateless encode support Paul Kocialkowski
2026-11-17 21:30 ` Nicolas Dufresne [this message]
2026-05-22 10:16 ` [PATCH 03/14] media: h264: Add SPS video definitions Paul Kocialkowski
2026-11-17 21:31 ` Nicolas Dufresne
2026-05-22 10:16 ` [PATCH 04/14] media: h264: Add stateless encode core Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 05/14] media: h264: Add stateless encode rbsp Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 06/14] media: h264: Add stateless encode reference management Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 07/14] media: h264: Add stateless encode rate control Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 08/14] media: verisilicon: Report default pixel coding for non-JPEG and fix JPEG case Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 09/14] media: verisilicon: Cancel job with runtime pm put/clk disable on failure Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 10/14] media: hantro: use hantro_decoded_buffer only for dst_vq Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 11/14] media: verisilicon: Add common encoder parm and frameintervals ioctls Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 12/14] media: verisilicon: Add support for the VC8000E H.264 encoder Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 13/14] media: verilisicon: imx8m: Add support for the VC8000E on i.MX8MP Paul Kocialkowski
2026-05-22 10:16 ` [PATCH 14/14] arm64: dts: imx8mp: add VC8000E encoder node Paul Kocialkowski
2026-05-22 11:28 ` Fabio Estevam
2026-05-22 14:47 ` [PATCH 00/14] media: Add V4L2 H.264 stateless encode and VC8000E support Nicolas Dufresne
2026-09-23 12:10 ` Michael Nazzareno Trimarchi
2026-09-23 20:05 ` Paul Kocialkowski
2026-09-24 5:30 ` Michael Nazzareno Trimarchi
2026-09-24 18:09 ` Paul Kocialkowski
2026-09-24 19:38 ` Michael Nazzareno Trimarchi
2026-09-24 19:44 ` Nicolas Dufresne
2026-09-24 20:30 ` Paul Kocialkowski
2026-09-24 20:46 ` Nicolas Dufresne
2026-09-30 4:14 ` Chen-Yu Tsai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1a1feaf854e6e1c34af468b7501988f2cf68ebca.camel@collabora.com \
--to=nicolas.dufresne@collabora.com \
--cc=benjamin.gaignard@collabora.com \
--cc=conor+dt@kernel.org \
--cc=devicetree@vger.kernel.org \
--cc=hverkuil@kernel.org \
--cc=imx@lists.linux.dev \
--cc=kernel@pengutronix.de \
--cc=krzk+dt@kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=m.felsch@pengutronix.de \
--cc=m.tretter@pengutronix.de \
--cc=mchehab@kernel.org \
--cc=p.zabel@pengutronix.de \
--cc=paulk@sys-base.io \
--cc=robh@kernel.org \
--cc=s.hauer@pengutronix.de \
--cc=shawnguo@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox