Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[johannes.goede@oss.qualcomm.com]

Hi,

On 17-Sep-26 15:47, Bryan O'Donoghue wrote:
> On 15/09/2026 00:38, Laurent Pinchart wrote:
>> We can consider scheduling a third topic. Proposals are welcome.
> 
> From my LPC submission:
> 
> "Should offline ISPs be implemented in drm/accel"
> 
> Where does an offline Camera ISP belong in video4linux or in DRM and why ?
> 
> There are two main types of ISP - inline and offline. An inline ISP takes sensor input directly and produces some kind of processed image on the output. This is a natural fit for v4l2.
> 
> An offline ISP takes a memory buffer given to it and processes that buffer against supplied parameters. This model can be implemented as a video4linux m2m device but it could also be implemented as a DRM device and included in the DAG.
> 
> Taking Qualcomm SoCs as an example the Camera hardware provides two types of ISP. An inline processing engine IFE and an offline processing engine based on firmware ICP/HFI.

Note that one some Qualcomm SoCs the inline-engine can be switched to
offline mode and then run as a M2M ISP. Also the offline ISPs and
inline ISPs often have similar processing-blocks with similar or
identical parameters.

I do not think that splitting this over 2 subsystems is helpful and will
lead to a lot of duplication of efforts related to ISPs.

> A use-case emerges to implement the inline ISP in v4l2 and the offline in drm/accel.
> > Some tension with this design would exist as user-space would have to be involved to update ISP parameters and process statistics. Another tension is including a firmware based system in dma-fencing including watchdogging and resetting if required.
> 
> There are clear benefits to having a loop of NPU/DSP to Offline ISP to Display that drm/accel seems to offer some interesting solutions to.

This sounds to me like we need a way to better integrate drm and v4l2
subsystems. Other areas like hw-video-decoders might benefit from
having better integration there too.

Regards,

Hans