Re: [PATCH] media: Documentation: Fix frame interval calculation for raw camera sensors

[Sakari Ailus <sakari.ailus@linux.intel.com>]

Hi Jai,

On Thu, Feb 19, 2026 at 08:26:43PM +0530, Jai Luthra wrote:
> Hi Sakari,
> 
> Thanks for the review.
> 
> Quoting Sakari Ailus (2026-02-19 15:21:10)
> > Hi Jai,
> > 
> > Thanks for the patch.
> > 
> > On Thu, Feb 19, 2026 at 01:20:50PM +0530, Jai Luthra wrote:
> > > The previous frame interval formula used analogue crop dimensions. This
> > > breaks down for some sensors when binning.
> > > 
> > > For example in imx219 the minimum FLL (frame length in lines) can be
> > > lower than the analogue crop height when binning, which would require a
> > > negative VBLANK to represent the actual timing. Similarly, imx283 allows
> > > a lower minimum HMAX (line length) when doing 2x2 or 3x3 binning than
> > > the analogue crop width of the full resolution mode.
> > 
> > V4L2 integer controls are signed so using negative numbers is a non-issue.
> > CCS already does this in some cases actually.
> > 
> 
> Ah, I see. Control being negative was not the only problem, as it would
> also diverge from what sensors and applications already do, but I see you
> agreed on that below :-)
> 
> I'll update the commit message to make this clear(er).

Ack.

> 
> > > 
> > > The CCS specification also describes under section "8.2.6 Line Length
> > > and Frame Length" how the horizontal and vertical readout minimums can
> > > be different when binning.
> > > 
> > > Replace the formula with the underlying hardware concepts of LLP (line
> > > length in pixels) and FLL (frame length in lines). These terms were
> > > chosen to match the CCS specification on raw sensors, as it is a cleaner
> > > reference compared to a typical sensor vendor datasheet.
> > > 
> > > Finally, define the blanking controls relative to the active pixel
> > > readout (post-binning) rather than the analogue crop size. This matches
> > > what most sensor drivers already do, and also what applications like
> > > libcamera expect. In "Figure 42" of CCS specification too, we see a
> > > similar definition:
> > > 
> > >   frame interval = (output width + HBLANK) *
> > >                    (output height + VBLANK) / pixel rate
> > > 
> > > Also add a note in the "Writing camera sensor drivers" guide, to ensure
> > > this formula is followed by new sensor drivers.
> > 
> > We're about to require implementing the common raw sensor model soon by
> > essentially all new drivers so documenting the soon-to-be-obsolete state
> > has limited benefits.
> 
> Makes sense.
> 
> > 
> > But the UAPI documentation remains relevant for quite some time, please see
> > my comments below.
> > 
> > > 
> > > Signed-off-by: Jai Luthra <jai.luthra@ideasonboard.com>
> > > ---
> > >  Documentation/driver-api/media/camera-sensor.rst   | 11 ++++
> > >  .../userspace-api/media/drivers/camera-sensor.rst  | 59 +++++++++++++++-------
> > >  2 files changed, 53 insertions(+), 17 deletions(-)
> > > 
> > > diff --git a/Documentation/driver-api/media/camera-sensor.rst b/Documentation/driver-api/media/camera-sensor.rst
> > > index 94bd1dae82d5c570b2d11c7faee20dd45d2f4be6..8dcac7551f54ac4ffa71173281ae3bbea331c036 100644
> > > --- a/Documentation/driver-api/media/camera-sensor.rst
> > > +++ b/Documentation/driver-api/media/camera-sensor.rst
> > > @@ -120,6 +120,17 @@ The function returns a non-zero value if it succeeded getting the power count or
> > >  runtime PM was disabled, in either of which cases the driver may proceed to
> > >  access the device.
> > >  
> > > +Frame interval
> > > +--------------
> > > +
> > > +If a sensor supports cropping or binning, it is the sensor driver's
> > > +responsibility to ensure that the frame interval formula (see
> > > +:ref:`media_using_camera_sensor_drivers`) remains valid regardless of the
> > > +pipeline configuration. The driver shall adjust the minimum and maximum allowed
> > > +values of ``V4L2_CID_HBLANK`` and ``V4L2_CID_VBLANK`` as needed when the mode
> > > +changes, so that application developers can always rely on the same formula to
> > > +calculate the frame interval.
> > > +
> > >  Rotation, orientation and flipping
> > >  ----------------------------------
> > >  
> > > diff --git a/Documentation/userspace-api/media/drivers/camera-sensor.rst b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > index 75fd9166383fdbb2dabdb6384ed0904c4e78a3c6..30dddea72a12da264fc9c30e37b561c762c09d29 100644
> > > --- a/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > +++ b/Documentation/userspace-api/media/drivers/camera-sensor.rst
> > > @@ -49,35 +49,60 @@ depends on the type of the device.
> > >  Raw camera sensors
> > >  ~~~~~~~~~~~~~~~~~~
> > >  
> > > -Instead of a high level parameter such as frame interval, the frame interval is
> > > -a result of the configuration of a number of camera sensor implementation
> > > -specific parameters. Luckily, these parameters tend to be the same for more or
> > > -less all modern raw camera sensors.
> > > +Instead of a high level parameter such as frame interval, the frame interval on
> > > +a raw camera sensor is determined by a number of sensor-specific parameters.
> > > +These parameters tend to be common across most modern raw camera sensors.
> > >  
> > > -The frame interval is calculated using the following equation::
> > > +The pixel array is the full grid of photosensitive elements on the sensor. A
> > 
> > s/sensor/camera sensor/
> > 
> 
> Will fix.
> 
> > > +subregion of it is selected by the analogue crop. The cropped image may then be
> > > +subject to binning (averaging of a NxN block) or subsampling which
> > > +further reduce the image dimensions. The resulting image is then read out by
> > > +the ADC (analogue-to-digital converter) line by line. After ADC readout,
> > > +optional digital crop or scaling may further reduce the image dimensions, see
> > > +:ref:`VIDIOC_SUBDEV_G_SELECTION <VIDIOC_SUBDEV_G_SELECTION>`.
> > >  
> > > -     frame interval = (analogue crop width + horizontal blanking) *
> > > -                      (analogue crop height + vertical blanking) / pixel rate
> > > +The frame size is determined by two timing parameters: line length in pixels
> > > +(LLP) and frame length in lines (FLL). These are fundamental sensor timing
> > > +registers that control how fast the ADC reads out the image. They may go
> > > +by different names for a particular sensor, like HMAX and VMAX, or HTOTAL and
> > > +VTOTAL, or similar.
> > >  
> > > -The formula is bus independent and is applicable for raw timing parameters on
> > > -large variety of devices beyond camera sensors. Devices that have no analogue
> > > -crop, use the full source image size, i.e. pixel array size.
> > > +LLP is the total number of pixel clock cycles per line, including both the
> > > +active readout width and horizontal blanking. FLL is the total number of lines
> > > +per frame, including both the active readout height and vertical blanking.
> > > +
> > > +The frame interval is::
> > > +
> > > +        frame interval = LLP * FLL / pixel rate
> > 
> > How would this look like if you spell out LLP and FLL? The rest aren't
> > abbreviated either.
> 
>         frame interval = (line length in pixels) *
>                          (frame length in lines) / pixel rate
> 
> This reads much easier, thanks for the suggestion, will update it in v2.
> 
> > 
> > >  
> > >  Horizontal and vertical blanking are specified by ``V4L2_CID_HBLANK`` and
> > >  ``V4L2_CID_VBLANK``, respectively. The unit of the ``V4L2_CID_HBLANK`` control
> > >  is pixels and the unit of the ``V4L2_CID_VBLANK`` is lines. The pixel rate in
> > > -the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the same
> > > -sub-device. The unit of that control is pixels per second.
> > > +the sensor's **pixel array** is specified by ``V4L2_CID_PIXEL_RATE`` in the
> > > +same sub-device. The unit of that control is pixels per second.
> > > +
> > > +The blanking is defined relative to the size of the image being sent out to the
> > > +host over the bus (like CSI-2)::
> > > +
> > > +        LLP = active width + V4L2_CID_HBLANK
> > > +        FLL = active height + V4L2_CID_VBLANK
> > > +
> > > +The driver shall set the minimum and maximum values of ``V4L2_CID_HBLANK`` and
> > > +``V4L2_CID_VBLANK`` such that the resulting LLP and FLL values correspond to the
> > > +range permitted by the sensor hardware for the current mode. Sensors that
> > > +support binning often define a lower minimum for LLP or FLL registers, which
> > > +can help achieve higher framerates when binning.
> > > +
> > > +Application developers can calculate the frame interval using the output
> > > +dimensions and the blanking controls::
> > > +
> > > +        frame interval = (output width + horizontal blanking) *
> > > +                         (output height + vertical blanking) / pixel rate
> > 
> > This is indeed what many presumably non-CCS sensor drivers implement. Are
> > there any that would use the crop rectangle (the imx219 doesn't seem to)?
> > For user space variance in this area is of course bad.
> 
> I am not aware of any, but I haven't checked all drivers.
> 
> The ones I checked (IMX219, IMX283, IMX335, OV5647, OV5640), all set:
> [h/v]blank = [h/v]total - mode->[width/height]
> 
> So IMO updating the documentation to match that is simpler.

I agree. I was just wondering how consistently this was done over the
years.

> 
> > The common raw sensor model introduces two new controls for the purpose so
> > we could re-purpose the old VBLANK/HBLANK controls for this -- apart from
> > the CCS driver.
> > 
> 
> I guess that's my final cue to finally take out time and read that series
> properly :-)

The two new controls will be part of the new version which I hope to post
soon. I'll cc you when I do.

-- 
Kind regards,

Sakari Ailus