Questions about image size listed in VIDIOC_ENUM_FMT

Questions about image size listed in VIDIOC_ENUM_FMT

[Zhaoxuan Zhai]

Hi all,

We are working on a camera driver. We plan to use v4l2 interface to send 
image data to users. We have a question we'd like to ask.

The situation we are facing is as follows.

We have an image processing unit that can process raw data collected by 
the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.

However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the 
width and height of the image must be divisible by 16.

For example, when the sensor provides an image size of 2104x1560, after 
NV12M encoding, it should be pading to  2112x1566. But the additional 8 
rows and 8 columns are padded with 0s and contain no actual content.

So, we would like to ask, in this case, what size should we list in 
VIDIOC_ENUM_FMT? Should it be the actual image size with content, 
2104x1560, or the padded size, 2112x1566?"


Thank you,

Zhaoxuan

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Zhaoxuan Zhai]

Hi all,

Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of 
VIDIOC_ENUM_FMT. I'm sorry for the mistake.

在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> Hi all,
>
> We are working on a camera driver. We plan to use v4l2 interface to 
> send image data to users. We have a question we'd like to ask.
>
> The situation we are facing is as follows.
>
> We have an image processing unit that can process raw data collected 
> by the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
>
> However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the 
> width and height of the image must be divisible by 16.
>
> For example, when the sensor provides an image size of 2104x1560, 
> after NV12M encoding, it should be pading to  2112x1566. But the 
> additional 8 rows and 8 columns are padded with 0s and contain no 
> actual content.
>
> So, we would like to ask, in this case, what size should we list in 
> VIDIOC_ENUM_FMT? Should it be the actual image size with content, 
> 2104x1560, or the padded size, 2112x1566?"
>
>
> Thank you,
>
> Zhaoxuan
>

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Sakari Ailus]

Hi Zhaoxuan,

On Fri, Feb 14, 2025 at 12:19:23PM +0800, Zhaoxuan Zhai wrote:
> Hi all,
> 
> Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of
> VIDIOC_ENUM_FMT. I'm sorry for the mistake.
> 
> 在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> > Hi all,
> > 
> > We are working on a camera driver. We plan to use v4l2 interface to send
> > image data to users. We have a question we'd like to ask.
> > 
> > The situation we are facing is as follows.
> > 
> > We have an image processing unit that can process raw data collected by
> > the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
> > 
> > However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the
> > width and height of the image must be divisible by 16.
> > 
> > For example, when the sensor provides an image size of 2104x1560, after
> > NV12M encoding, it should be pading to  2112x1566. But the additional 8
> > rows and 8 columns are padded with 0s and contain no actual content.
> > 
> > So, we would like to ask, in this case, what size should we list in
> > VIDIOC_ENUM_FMT? Should it be the actual image size with content,
> > 2104x1560, or the padded size, 2112x1566?"

I'd say the actual image size (i.e. where you have pixel data). The
sizeimage field needs to reflect the padding and the user needs to be aware
how the data is laid out in memory.

Also cc Hans.

-- 
Regards,

Sakari Ailus

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Nicolas Dufresne]

Le vendredi 14 février 2025 à 07:08 +0000, Sakari Ailus a écrit :
> Hi Zhaoxuan,
> 
> On Fri, Feb 14, 2025 at 12:19:23PM +0800, Zhaoxuan Zhai wrote:
> > Hi all,
> > 
> > Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of
> > VIDIOC_ENUM_FMT. I'm sorry for the mistake.
> > 
> > 在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> > > Hi all,
> > > 
> > > We are working on a camera driver. We plan to use v4l2 interface to send
> > > image data to users. We have a question we'd like to ask.
> > > 
> > > The situation we are facing is as follows.
> > > 
> > > We have an image processing unit that can process raw data collected by
> > > the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
> > > 
> > > However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the
> > > width and height of the image must be divisible by 16.
> > > 
> > > For example, when the sensor provides an image size of 2104x1560, after
> > > NV12M encoding, it should be pading to  2112x1566. But the additional 8
> > > rows and 8 columns are padded with 0s and contain no actual content.
> > > 
> > > So, we would like to ask, in this case, what size should we list in
> > > VIDIOC_ENUM_FMT? Should it be the actual image size with content,
> > > 2104x1560, or the padded size, 2112x1566?"
> 
> I'd say the actual image size (i.e. where you have pixel data). The
> sizeimage field needs to reflect the padding and the user needs to be aware
> how the data is laid out in memory.

I'd be happy to see spec clarification in this regard. I also think
that image size is best.

For NV12M, you can absorb the padding in per plane bytesperline /
sizeimage of the v4l2_format structure. That means the enumerated size
will match the format. Though, for NV12 (single plane) you'd have to
set at least the padded height and implement the SELECTION API to
return the cropping area.

In practice, for application point of view, it would all have been
easier if the v4l2_format was always padded width/height, with the
display dimension explicitly provided. But I'm pretty sure upstream
drivers implement a mix.

Nicolas

> 
> Also cc Hans.
> 

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Laurent Pinchart]

On Fri, Feb 14, 2025 at 09:57:08AM -0500, Nicolas Dufresne wrote:
> Le vendredi 14 février 2025 à 07:08 +0000, Sakari Ailus a écrit :
> > Hi Zhaoxuan,
> > 
> > On Fri, Feb 14, 2025 at 12:19:23PM +0800, Zhaoxuan Zhai wrote:
> > > Hi all,
> > > 
> > > Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of
> > > VIDIOC_ENUM_FMT. I'm sorry for the mistake.
> > > 
> > > 在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> > > > Hi all,
> > > > 
> > > > We are working on a camera driver. We plan to use v4l2 interface to send
> > > > image data to users. We have a question we'd like to ask.
> > > > 
> > > > The situation we are facing is as follows.
> > > > 
> > > > We have an image processing unit that can process raw data collected by
> > > > the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
> > > > 
> > > > However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the
> > > > width and height of the image must be divisible by 16.
> > > > 
> > > > For example, when the sensor provides an image size of 2104x1560, after
> > > > NV12M encoding, it should be pading to  2112x1566. But the additional 8
> > > > rows and 8 columns are padded with 0s and contain no actual content.
> > > > 
> > > > So, we would like to ask, in this case, what size should we list in
> > > > VIDIOC_ENUM_FMT? Should it be the actual image size with content,
> > > > 2104x1560, or the padded size, 2112x1566?"
> > 
> > I'd say the actual image size (i.e. where you have pixel data). The
> > sizeimage field needs to reflect the padding and the user needs to be aware
> > how the data is laid out in memory.
> 
> I'd be happy to see spec clarification in this regard. I also think
> that image size is best.

Which part of the spec do you think should be clarified ? I've always
thought that ENUM_FMT should report the image dimensions, with padding
being reported through bytesperline and sizeimage. Is there a place
where we say something else ? If so that should indeed be fixed.

> For NV12M, you can absorb the padding in per plane bytesperline /
> sizeimage of the v4l2_format structure. That means the enumerated size
> will match the format. Though, for NV12 (single plane) you'd have to
> set at least the padded height and implement the SELECTION API to
> return the cropping area.

NV12 is indeed ambiguous, as vertical padding reported by sizeimage
could be just at the end of the image, or split between planes.

> In practice, for application point of view, it would all have been
> easier if the v4l2_format was always padded width/height, with the
> display dimension explicitly provided. But I'm pretty sure upstream
> drivers implement a mix.
> 
> > Also cc Hans.

-- 
Regards,

Laurent Pinchart

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Nicolas Dufresne]

Le vendredi 14 février 2025 à 18:40 +0200, Laurent Pinchart a écrit :
> On Fri, Feb 14, 2025 at 09:57:08AM -0500, Nicolas Dufresne wrote:
> > Le vendredi 14 février 2025 à 07:08 +0000, Sakari Ailus a écrit :
> > > Hi Zhaoxuan,
> > > 
> > > On Fri, Feb 14, 2025 at 12:19:23PM +0800, Zhaoxuan Zhai wrote:
> > > > Hi all,
> > > > 
> > > > Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of
> > > > VIDIOC_ENUM_FMT. I'm sorry for the mistake.
> > > > 
> > > > 在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> > > > > Hi all,
> > > > > 
> > > > > We are working on a camera driver. We plan to use v4l2 interface to send
> > > > > image data to users. We have a question we'd like to ask.
> > > > > 
> > > > > The situation we are facing is as follows.
> > > > > 
> > > > > We have an image processing unit that can process raw data collected by
> > > > > the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
> > > > > 
> > > > > However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the
> > > > > width and height of the image must be divisible by 16.
> > > > > 
> > > > > For example, when the sensor provides an image size of 2104x1560, after
> > > > > NV12M encoding, it should be pading to  2112x1566. But the additional 8
> > > > > rows and 8 columns are padded with 0s and contain no actual content.
> > > > > 
> > > > > So, we would like to ask, in this case, what size should we list in
> > > > > VIDIOC_ENUM_FMT? Should it be the actual image size with content,
> > > > > 2104x1560, or the padded size, 2112x1566?"
> > > 
> > > I'd say the actual image size (i.e. where you have pixel data). The
> > > sizeimage field needs to reflect the padding and the user needs to be aware
> > > how the data is laid out in memory.
> > 
> > I'd be happy to see spec clarification in this regard. I also think
> > that image size is best.
> 
> Which part of the spec do you think should be clarified ? I've always
> thought that ENUM_FMT should report the image dimensions, with padding
> being reported through bytesperline and sizeimage. Is there a place
> where we say something else ? If so that should indeed be fixed.

When I read [0] and I don't seen note that explains the expectated
relationship between the results of that enumeration and the SELECTION
API.

That also makes me notice that there is not "queue type" specified,
which if not well documented will be ambiguous in M2M drivers.
Generally codec have more stuff specified, but for scalers, that feels
like a slight concern.

> 
> > For NV12M, you can absorb the padding in per plane bytesperline /
> > sizeimage of the v4l2_format structure. That means the enumerated size
> > will match the format. Though, for NV12 (single plane) you'd have to
> > set at least the padded height and implement the SELECTION API to
> > return the cropping area.
> 
> NV12 is indeed ambiguous, as vertical padding reported by sizeimage
> could be just at the end of the image, or split between planes.

The only way to use NV12 is to have the second plane at exactly
byteperline * height (which is true for all drivers, they have no
choices). But then, when a height padding is used it will endup being
returned by the S_FMT call, but also cropped by the selection API.

I believe the general idea is that the enum_frmsize describes the
display size (or applied to the bitstream/output queue perhaps). I
didn't re-read that part, perhaps this part is all right, but rather
complex, since you then enumerate the results of the combination of
S_FMT + SELECTION. Let me know what you think.

> 
> > In practice, for application point of view, it would all have been
> > easier if the v4l2_format was always padded width/height, with the
> > display dimension explicitly provided. But I'm pretty sure upstream
> > drivers implement a mix.
> > 
> > > Also cc Hans.
> 

[0] https://linuxtv.org/downloads/v4l-dvb-apis/userspace-api/v4l/vidioc-enum-framesizes.html?highlight=enum_framesize

Re: Questions about image size listed in VIDIOC_ENUM_FMT

[Laurent Pinchart]

On Fri, Feb 14, 2025 at 07:08:31AM +0000, Sakari Ailus wrote:
> On Fri, Feb 14, 2025 at 12:19:23PM +0800, Zhaoxuan Zhai wrote:
> > Hi all,
> > 
> > Sorry I made a mistake. It should be VIDIOC_ENUM_FRAMESIZES instead of
> > VIDIOC_ENUM_FMT. I'm sorry for the mistake.
> > 
> > 在 2025/2/14 12:08, Zhaoxuan Zhai 写道:
> > > Hi all,
> > > 
> > > We are working on a camera driver. We plan to use v4l2 interface to send
> > > image data to users. We have a question we'd like to ask.
> > > 
> > > The situation we are facing is as follows.
> > > 
> > > We have an image processing unit that can process raw data collected by
> > > the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.
> > > 
> > > However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the
> > > width and height of the image must be divisible by 16.

I assume this is a requirement of your hardware (can you tell what
hardware you're running on ?) when outputting NV12. The NV12 format
itself doesn't require a 16 pixels alignment.

> > > For example, when the sensor provides an image size of 2104x1560, after
> > > NV12M encoding, it should be pading to  2112x1566. But the additional 8
> > > rows and 8 columns are padded with 0s and contain no actual content.
> > > 
> > > So, we would like to ask, in this case, what size should we list in
> > > VIDIOC_ENUM_FMT? Should it be the actual image size with content,
> > > 2104x1560, or the padded size, 2112x1566?"
> 
> I'd say the actual image size (i.e. where you have pixel data). The
> sizeimage field needs to reflect the padding and the user needs to be aware
> how the data is laid out in memory.

And of course the bytesperline field should report the total line
stride, including the padding.

-- 
Regards,

Laurent Pinchart

Questions about image size listed in VIDIOC_ENUM_FMT

[Zhaoxuan Zhai]

Hi all,

We are working on a camera driver. We plan to use v4l2 interface to send 
image data to users. We have a question we'd like to ask.

The situation we are facing is as follows.

We have an image processing unit that can process raw data collected by 
the sensor into the V4L2_PIX_FMT_NV12M format and send it to the user.

However, due to the requirements of the V4L2_PIX_FMT_NV12M format, the 
width and height of the image must be divisible by 16.

For example, when the sensor provides an image size of 2104x1560, after 
NV12M encoding, it should be pading to  2112x1566. But the additional 8 
rows and 8 columns are padded with 0s and contain no actual content.

So, we would like to ask, in this case, what size should we list in 
VIDIOC_ENUM_FMT? Should it be the actual image size with content, 
2104x1560, or the padded size, 2112x1566?"


Thank you,

Zhaoxuan