Re: [PATCH 3/3] media: venus: report the frame size of the codec being enumerated

[Konrad Dybcio <konrad.dybcio@oss.qualcomm.com>]

On 8/12/26 10:01 PM, Dmitry Baryshkov wrote:
> VIDIOC_ENUM_FRAMESIZES takes the pixel format to describe from
> userspace, but both vdec and venc answer it out of frame_width_min() and
> friends, which resolve the capabilities through inst->hfi_codec, the
> codec the instance is currently set to.  Enumerating any format other
> than that one therefore returns the limits of an unrelated codec: on a
> freshly opened decoder, which starts out as H.264, asking about MPEG-2
> on MSM8996 reports the H.264 limits where the firmware describes MPEG-2
> as 16x16 to 1920x1920.

You gave some numbers for MPEG-2, saying H.264 has some different numbers.
I think it would be slightly easier to parse if you either skipped them or
gave numbers for both codecs

> Resolve the capabilities through the format being enumerated instead,
> falling back to the codec in use for the raw formats, which have no
> codec of their own.  Both drivers now share one helper, alongside the
> pixel format to codec mapping that venus_helper_check_codec() already
> carried.
> 
> Assisted-by: Claude:claude-opus-5
> Signed-off-by: Dmitry Baryshkov <dmitry.baryshkov@oss.qualcomm.com>
> ---

[...]

> -bool venus_helper_check_codec(struct venus_inst *inst, u32 v4l2_pixfmt)
> +static u32 venus_pixfmt_to_hfi_codec(u32 v4l2_pixfmt)
>  {
> -	struct venus_core *core = inst->core;
> -	u32 session_type = inst->session_type;
> -	u32 codec;
> -
>  	switch (v4l2_pixfmt) {
>  	case V4L2_PIX_FMT_H264:
> -		codec = HFI_VIDEO_CODEC_H264;
> -		break;
> +		return HFI_VIDEO_CODEC_H264;

We already have this - to_codec_type() in hfi.c

Konrad