Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

On Sat, Sep 19, 2026 at 10:45:00PM +0100, Bryan O'Donoghue wrote:
> On 18/09/2026 10:18, Laurent Pinchart wrote:
> >> There are clear benefits to having a loop of NPU/DSP to Offline ISP to
> >> Display that drm/accel seems to offer some interesting solutions to.
> > What are those interesting solutions ?
> 
> The chain I'm interested in is
> 
> Inline - V4L2 streaming
> -> Produces raw or YUV
> 
> CVP (&& ||) NPU - Likely implemented as DRM

Possibly, but that's bit very relevant from a userspace point of view.
An NPU will be controlled through a userspace stack, likely involving
Mesa if the stack is open-source, and exposed through higher-level
frameworks/APIs (TF, ONNX, ...). The fact that the kernel driver lives
in DRM won't have any impact.

> -> Produces motion vectors and/or further warps image
> 
> Offline - Sharpens, GTM, TNR (could be DRM, or v4lm2m)
> -> Produces YUV
> 
> Display - DRM

So likely OpenGL or Vulkan, involving wayland or X11. In rare cases
directly interfacing with KMS for embedded devices.

> V4L2 owning sensor -> inline engine is already supported and won't 
> change. For subsequent steps though each stage hands off in-kernel 
> without having to involve userspace.

That's fences, isn't it ?

> Given though it depends if you need user-space to "do stuff" at the 
> offline stage. If the offline stage has a firmware perhaps user-space 
> doesn't need to be involved much past setup whereas if that offline 
> stage has no firmware, then user-space really needs to drive it. The 
> difference between the OPE on-list right now without a firmware and the 
> ICP which does have a firmware.

Cases where userspace wouldn't need to be involved exist, but then
there's even less userspace involvement than you describe above. The
firmware would control all the components directly (sensor, inline and
offline stages), exposing only the end result (possibly with the ability
to additionally capture raw frames).

> Taking the example of sharing params and stats between Inline and 
> Offline - in this case you are constructing a single ISP around two 
> blocks basically in libcamera, whereas what I've described above 
> traverses several blocks and isn't really an ISP - the ISP is already @ 
> stage 1 - you have an additional fully offline block with its own 
> firmware, sessions and runtime context. Stage 1 and stage 3 don't share 
> kernel-side stats and params; they have their own isolated contexts, in 
> contrast to TFE/OPE on Agatti where its integrated and must live in a 
> dedicated libcamera pipeline.
> 
> But because you have a functional ISP @ stage 1 - the question is does 
> the firmware based offline engine provide better overall system 
> functionality as a DRM device ...

I don't see why that would be the case.

> Its actually the question I'd like to discuss with colleagues in sunny 
> Prague :)
> 
> So I think the tension comes from wanting to have the DRM scheduler but 
> acknowledging that it is probably only for some use-cases possible.
> 
> You implement the device as a DRM device, you are still free to have a 
> complex user-space context for that if that is your need - whereas if 
> you have a use-case which can tolerate non-intervention from user-space, 
> then the DRM scheduler _is_ what you want for complex hand-off from one 
> device block to the other.

Not quite. It's a matter of fences.

-- 
Regards,

Laurent Pinchart