Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[Gjorgji Rosikopulos <grosikopulos@mm-sol.com>]

Hi Laurent,

On 10/1/26 02:23, Laurent Pinchart wrote:
> Hello Gjorgji,
> 
> On Fri, Sep 18, 2026 at 02:10:07PM +0300, Gjorgji Rosikopulos wrote:
>> Hi Laurent, All,
>>
>> On 9/15/26 02:38, Laurent Pinchart wrote:
>>
>>     We can consider scheduling a third topic. Proposals are welcome.
>>
>>
>> Below are my proposed discussion topics. All of them relate to advanced camera
>> use cases:
>>
>> 1. Electronic Image Stabilization (EIS).
>> 2. Multi-frame Bayer or YUV noise reduction filters.
>> 3. QUAD CFA reconstruction, including preprocessing algorithms for
>> high-resolution Quad Bayer sensors.
>> 4. AI-based denoising in Bayer or YUV domains, focusing on image preprocessing
>> denoising algorithms.
>> 5. DOL and video HDR use cases.
>> 6. Background blur for video conferencing applications.
>> 7. Face detection algorithms operating directly on image data.
>> 8. Multicamera 360 image stitching etc.
> 
> Those are all interesting topics, but I won't be able to schedule all of
> them in 15 minutes :-)

I agree those topics will require more time to discuss.

> 
>> All of these use cases and algorithm implementations can be vendor-specific or
>> third-party solutions and may run on a GPU, DSP, or other accelerators.
>>
>> I also think that, similar to the IPA used for 3A algorithms, we need an
>> IPA-like for frame-processing algorithms.
>>
>> As discussed, the only practical way to upstream such functionality into
>> libcamera is to provide an open-source alternative (even a simplified
>> implementation) that defines and covers the use case, including its input/
>> output parameters and interfaces.
>>
>> The main question is whether these algorithms should live inside libcamera or
>> outside of it. Both approaches have their strengths, but I am personally in
>> favor of integrating them into libcamera, perhaps as a layer above the pipeline
>> handlers, for example through use case plugins or a similar mechanism.
> 
> I've long thought that libcamera should make pipeline handlers more
> modular. Pipeline handlers would implement components for the pipeline
> elements (such as the inline frontend and offline ISP) and declare how
> those components are connected. We should be able to move most of the
> logic required to run the pipeline into common code.

That will be really good to have.

> 
> With a modular approach, inserting elements in the pipeline could become
> easier. It's however not clear to me yet if we should try to separate
> the processing required for the use cases you list above from the "core"
> of pipeline handlers, into a separate layer on top, or if some use cases
> would benefit more from being able to inject elements into the pipeline.

I think that we will need some scheduler or code for driving different usecases,
also it should be noted that processing order may depend on the parameters,
An example we have preprocessing noise filter which can be enabled disabled,
or HDR. Depending on that the processing blocks participating in the pipleine
may change. But one step at a time.

> 
> Note that we have started experimenting with the concept of layers, see
> https://patchwork.libcamera.org/project/libcamera/list/?series=5753/

Thanks i will check that work.

> 
>> We can also learn from Android's approach. Initially, the HAL3 and per-control
>> APIs were created to allow applications to implement advanced algorithms such
>> as HDR and EIS. Later, CameraX Extensions were introduced so that these
>> capabilities could be shared across applications and benefit the broader
>> ecosystem:
>>
>> https://developer.android.com/media/camera/camerax/extensions-api
> 
> It's certainly something we should study more. As usual with Android
> there's little documentation on the underlying architecture though :-(
> 
>> I believe Linux faces a similar challenge. Once PipeWire and GStreamer plugins
>> based on libcamera become widely adopted across Linux distributions, it will
>> become increasingly difficult to introduce these advanced use cases at higher
>> layers of the software stack. Having a standardized framework within or closely
>> integrated with libcamera could make these capabilities more accessible and
>> reusable across applications.
> 
> No fundamental disagreement there, although we should also be careful
> not to reimplement GStreamer or PipeWire inside libcamera.

I agree on this, but having some concept like bins in gstreamer will also help for the entire usecase,
the question is whether we need to expose using multiple stages of reprocess the control to the app
(gstreamer element, PipeWire or other libcamera application) or have them build in libcamera. Both have benefits,
having ability the application to add intermediate stages gives freedom to experiment and some vendors to implement their
algos outside of the libcamera, On other hand side standard supported application like teams/slack browser will not 
have those extras example with cameraX extensions in Android which have been mentioned earlier.

> 
> An important question that will need to be answered is who will be
> willing to do all this work.

We have two eng in MMS which can start experimenting with those extensions in libcamera, i will actively involved and drive this activity if
there are no objections :-). Once you are back from Prague i can schedule a meeting for further discussions.

>

Regards,
~Gjorgji