Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hello Gjorgji,

On Fri, Sep 18, 2026 at 02:10:07PM +0300, Gjorgji Rosikopulos wrote:
> Hi Laurent, All,
> 
> On 9/15/26 02:38, Laurent Pinchart wrote:
> 
>     We can consider scheduling a third topic. Proposals are welcome.
> 
> 
> Below are my proposed discussion topics. All of them relate to advanced camera
> use cases:
> 
> 1. Electronic Image Stabilization (EIS).
> 2. Multi-frame Bayer or YUV noise reduction filters.
> 3. QUAD CFA reconstruction, including preprocessing algorithms for
> high-resolution Quad Bayer sensors.
> 4. AI-based denoising in Bayer or YUV domains, focusing on image preprocessing
> denoising algorithms.
> 5. DOL and video HDR use cases.
> 6. Background blur for video conferencing applications.
> 7. Face detection algorithms operating directly on image data.
> 8. Multicamera 360 image stitching etc.

Those are all interesting topics, but I won't be able to schedule all of
them in 15 minutes :-)

> All of these use cases and algorithm implementations can be vendor-specific or
> third-party solutions and may run on a GPU, DSP, or other accelerators.
> 
> I also think that, similar to the IPA used for 3A algorithms, we need an
> IPA-like for frame-processing algorithms.
> 
> As discussed, the only practical way to upstream such functionality into
> libcamera is to provide an open-source alternative (even a simplified
> implementation) that defines and covers the use case, including its input/
> output parameters and interfaces.
> 
> The main question is whether these algorithms should live inside libcamera or
> outside of it. Both approaches have their strengths, but I am personally in
> favor of integrating them into libcamera, perhaps as a layer above the pipeline
> handlers, for example through use case plugins or a similar mechanism.

I've long thought that libcamera should make pipeline handlers more
modular. Pipeline handlers would implement components for the pipeline
elements (such as the inline frontend and offline ISP) and declare how
those components are connected. We should be able to move most of the
logic required to run the pipeline into common code.

With a modular approach, inserting elements in the pipeline could become
easier. It's however not clear to me yet if we should try to separate
the processing required for the use cases you list above from the "core"
of pipeline handlers, into a separate layer on top, or if some use cases
would benefit more from being able to inject elements into the pipeline.

Note that we have started experimenting with the concept of layers, see
https://patchwork.libcamera.org/project/libcamera/list/?series=5753/

> We can also learn from Android's approach. Initially, the HAL3 and per-control
> APIs were created to allow applications to implement advanced algorithms such
> as HDR and EIS. Later, CameraX Extensions were introduced so that these
> capabilities could be shared across applications and benefit the broader
> ecosystem:
> 
> https://developer.android.com/media/camera/camerax/extensions-api

It's certainly something we should study more. As usual with Android
there's little documentation on the underlying architecture though :-(

> I believe Linux faces a similar challenge. Once PipeWire and GStreamer plugins
> based on libcamera become widely adopted across Linux distributions, it will
> become increasingly difficult to introduce these advanced use cases at higher
> layers of the software stack. Having a standardized framework within or closely
> integrated with libcamera could make these capabilities more accessible and
> reusable across applications.

No fundamental disagreement there, although we should also be careful
not to reimplement GStreamer or PipeWire inside libcamera.

An important question that will need to be answered is who will be
willing to do all this work.

-- 
Regards,

Laurent Pinchart