Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[Bryan O'Donoghue <bod@kernel.org>]

On 18/09/2026 10:18, Laurent Pinchart wrote:
>> There are clear benefits to having a loop of NPU/DSP to Offline ISP to
>> Display that drm/accel seems to offer some interesting solutions to.
> What are those interesting solutions ?

The chain I'm interested in is

Inline - V4L2 streaming
-> Produces raw or YUV

CVP (&& ||) NPU - Likely implemented as DRM
-> Produces motion vectors and/or further warps image

Offline - Sharpens, GTM, TNR (could be DRM, or v4lm2m)
-> Produces YUV

Display - DRM

V4L2 owning sensor -> inline engine is already supported and won't 
change. For subsequent steps though each stage hands off in-kernel 
without having to involve userspace.

Given though it depends if you need user-space to "do stuff" at the 
offline stage. If the offline stage has a firmware perhaps user-space 
doesn't need to be involved much past setup whereas if that offline 
stage has no firmware, then user-space really needs to drive it. The 
difference between the OPE on-list right now without a firmware and the 
ICP which does have a firmware.

Taking the example of sharing params and stats between Inline and 
Offline - in this case you are constructing a single ISP around two 
blocks basically in libcamera, whereas what I've described above 
traverses several blocks and isn't really an ISP - the ISP is already @ 
stage 1 - you have an additional fully offline block with its own 
firmware, sessions and runtime context. Stage 1 and stage 3 don't share 
kernel-side stats and params; they have their own isolated contexts, in 
contrast to TFE/OPE on Agatti where its integrated and must live in a 
dedicated libcamera pipeline.

But because you have a functional ISP @ stage 1 - the question is does 
the firmware based offline engine provide better overall system 
functionality as a DRM device ...

Its actually the question I'd like to discuss with colleagues in sunny 
Prague :)

So I think the tension comes from wanting to have the DRM scheduler but 
acknowledging that it is probably only for some use-cases possible.

You implement the device as a DRM device, you are still free to have a 
complex user-space context for that if that is your need - whereas if 
you have a use-case which can tolerate non-intervention from user-space, 
then the DRM scheduler _is_ what you want for complex hand-off from one 
device block to the other.

---
bod