Re: [ANNOUNCEMENT] Camera & ISP at Linux Plumbers Conference

[Stefan Klug <stefan.klug@ideasonboard.com>]

Hi Conor,

Thank you for your fast reply.

Quoting Conor Dooley (2026-10-01 00:01:44)
> Hey, just some quick thoughts I had, mostly to fish for more information
> so I can be more informed..
> 
> On Wed, Sep 30, 2026 at 01:18:59PM +0200, Stefan Klug wrote:
> > Hi everyone,
> > 
> > On 15.09.26 01:38, Laurent Pinchart wrote:
> > > Hello,
> > > 
> > > As most of you already know, the Linux Plumbers Conference will host a
> > > Camera & ISP BoF in Prague in three weeks ([1]).
> > > 
> > > The initial proposal for a microconference has unfortunately been
> > > downgraded by the program committee to a 45 minutes BoF. We have
> > > therefore decided to limit the discussion to three topics at most. Two
> > > topics have been approved so far:
> > > 
> > > - RGB-IR support in V4L2 and libcamera
> > > 
> > >   by Rishikesh Donadkar and Devarsh Thakkar, Texas Instruments
> > > 
> > >   The Linux kernel V4L2 API does not support RGB-IR image sensors.
> > >   While building blocks necessary to handle those devices are slowly
> > >   being merged, RGB-IR support itself hasn't been tackled yet.
> > > 
> > >   Rishikesh and Devarsh will also present this topic at the OSS Europe
> > >   conference ([2]).
> > > 
> > > - Camera module identification
> > > 
> > >   by Stefan Klug, Ideas on Board
> > > 
> > >   Camera tuning and calibration depend not only on the image sensor, but
> > >   also on the lens and other characteristics of camera modules. While
> > >   Linux supports identifying image sensors, it completely lacks the
> > >   concept of camera modules. 
> > > 
> > >   Identification of camera modules requires coordination between
> > >   platform firmware (DT or ACPI), the kernel and userspace. Discussions
> > >   will benefit from the presence of DT maintainers.
> > > 
> > LPC is already next week and we won't have enough time on the BoF for a
> > full introduction to the topic. So this email summarizes the Module
> > identification topic and contains the questions I'd like to discuss.
> > 
> > To the device-tree maintainers: I added you to the invitation as this
> > document proposes new device tree properties and I'd love to get some
> > feedback and common understanding on these. So if you'd be able to join,
> > that would be fantastic.
> 
> I'll do my best, there's some schedule conflicts with the RISC-V MC for
> me, so we'll see how it goes.
> 
> > 
> > Most of this mail is introduction and context. If you are short on time,
> > please jump directly to the last headline "### 4. Camera modules without
> > data storage" which contains the most relevant questions.
> 
> > 
> > Q: Does anyone in this group have more details on which information to
> > query and where to get more details from the system?
> > A: ...
> > 
> > ### 2. Camera modules with an embedded EEPROM
> > 
> > To make use of the EEPROM data from user space we need to carry two
> > elements: The EEPROM data itself and some identifier telling which
> > format the data in the EEPROM has. This seems relatively straightforward
> > and we could model it like this in dts:
> > 
> > dts-snippet:
> > imx283_0: sensor@1a {
> >               compatible = "sony,imx283";
> >     ...
> >     eeprom = <&eeprom>;
> >     eeprom-format = "vendor-a-eeprom-fmt";
> 
> It's nvmem-cells you want here I believe, and the associated layouts
> should cover the format?

Oh, that looks good. nvmem-cells seems to be the right thing here. I'm
not completely sure how the format would map, but I believe that would
resolve itself on the first practical implementation.

> 
> > };
> > 
> > Getting the link to the EEPROM from user space is something to
> > investigate. Ideas are either sysfs or media-controller.
> > The eeprom-format must be specified as well, to tell user space how to
> > interpret the EEPROM data, as it is typically not self-contained.
> 
> If you have the format of the eeprom, is it not better to provide the
> parsed version to userspace directly rather? Or are you concerned that
> these things will contain such "random" and non-standardisable info that
> it is a lost cause?

As the data is of no use to the kernel, I don't see the benefit in
standardizing it. On the other hand we are missing practical examples. I
feel it is a bit of a chicken and egg situation. There were some
EEPROMSs, but it is difficult to correlate them with the camera module
so no one really used theme even though everyone would like to have
them.

> 
> > 
> > Q: On a first discussion on that topic we were unsure if it would be
> > preferred to expose the data directly in sysfs,
> > or to symlink to the corresponding sysfs entry for the EEPROM. Any
> > preferences?
> > A: ...
> 
> FWIW, nvmem already has sysfs for this (first option in the nvmem
> Kconfig menu).

Thanks for that info. I need to toy around with that.

> 
> > 
> > Q: Any input/things to consider from ACPI side?
> > A:
> > 
> > Q: What happens if the EEPROM driver is not loaded yet or is just not
> > available? Should it block/defer probing of the sensor?
> > A:
> > 
> > Q: We are missing real-life examples for these cases. So input from
> > vendors would be welcome.
> > A:
> > 
> > ### 3. Camera modules with a sensor that has an embedded OTP memory
> > 
> > This case is similar to the previous one. The big difference is that the
> > OTP memory can be handled by the sensor driver itself, so no additional
> > driver is necessary. In this case a practical solution could be to
> > expose the OTP data via sysfs. Handling in user-space is similar to the
> > EEPROM case. So the device driver could expose an otp_data and
> > otp_format attribute in sysfs for user-space to read.
> > 
> > The property otp_format contains a name of the format of the OTP data.
> > There are cases where the format is specific to the sensor vendor and
> > cases where the format is specific to the module vendor. So it should be
> > possible to specify that in the device tree.
> > 
> > ### 4. Camera modules without data storage
> > 
> > This is the most difficult and most controversial case. In this case we
> > can't identify the module automatically by querying the hardware. It is
> > also not possible to rely on the machine identifier as with example 1.
> > Think of an arbitrary camera module connected to a Raspberry Pi - the
> > machine identifier will not change just by connecting a camera.
> > Therefore we'd like to supply all necessary information with the device
> > tree.
> > 
> > The difficulty is that the combinations are huge and there is no common
> > denominator (for some modules one might be able to get an SKU, for
> > others it will even be difficult to get an official name).
> > 
> > This is quite contrary to typical device tree use-cases where we try to
> > tie down everything as far as possible.
> > A flat list of module identifiers (like compatible strings) won't work
> > as it should be possible to standardize aspects of a module, even if not
> > everything is know. Think of a Raspberry Pi Cam HQ with a screw-on lens.
> > It would be great to be able to specify that module, but the lens is
> > still up to the user.
> > 
> > In [1] I proposed a possible solution to that problem. A tag-based
> > system with a single device tree entry of space separated tags. Every
> > tag carries a prefix to denote a category:
> > 
> > Tag                 Example
> > m:<ModuleName>      m:vc-mipi-imx296
> > v:<Vendor>          v:RaspberryPi
> > l:<Lens>            l:generic-8mm-ircut
> > s:<Sku>             s:B030801
> > u:<user defined>    u:my-selfmade-module
> > 
> > This is a bit like the existing "label" property which carries a human
> > readable label. So an example for a module could be:
> > 
> > imx283_0: sensor@1a {
> >               compatible = "sony,imx283";
> >     ...
> >     module-info = "v:RaspberryPi m:camera-module-hq
> > l:generic-8mm-wide-angle";
> > };
> > 
> > This leads to the immediate question: Should we split the tags into
> > properties?
> 
> Yes, make use of the standard utilities before rolling your own format
> please.
> 
> > imx283_0: sensor@1a {
> >               compatible = "sony,imx283";
> >     ...
> >     module-name = "camera-module-hq";
> >     module-vendor = "RaspberryPi"
> >     module-lens = "generic-8mm-wide-angle";
> > };
> > 
> > Pros: We could try to standardize on some values.
> > Cons: We add a (possibly increasing) number of properties that have no
> > value inside the kernel and will only be populated very sparsely.
> 
> What value do they have anywhere? ELI5 why anyone cares that the rpi
> foundation made the sensor module, for example. What decisions can be
> made on the basis of it? Not trying to be antagonistic, I genuinely
> don't know what kind of things it affects. The lens type or some sort of
> nd filter info I can understand being useful. That said, and I know very
> little about cameras, some of these things could actually vary at runtime
> right?

I think the difficulty here is that we want to identify something which
we can't even name precisely and we don't yet see the full spectrum of
possibilities. Regarding the question who cares if the RPi foundation
manufactured a module: It is just a way of identifying a specific piece
of hardware. So there are many modules out there using an imx477 sensor,
but they could differ in the geometry from other ones.

A compatible string works well for devices we know. But then you can
quickly come up with cases, where only an aspect or part is known.

- Raspberry Pi Camera Module HQ
  - Lens is unknown
  - So we could start to unroll that
    - rpi,cmarea-module-hq-8mm-lens
    - rpi,cmarea-module-hq-9mm-lens
    - ... that doesn't scale

So we could introduce a module-lens property:
- vendor-a-8mm
  vendor-a-8mm-ir-cut
  vendor-b-8mm
  ... 1000 vendors more

That doesn't scale as well.

I tried to further reason about the underlying targets and why my
gutfeeling doesn't like the compatible.

I see two main targets: 

1. Finding a solution for well known devices like laptops or phones.
Upstreaming a compatible string per module could work, but we will often
just not know the vendor of the module. So they would be named
(incorrectly) after the manufacturer of the device.

2. Upstreaming overlays for well known camera modules like the ones from
RaspberryPi.  I don't expect that to gain too much traction quickly
because of missing specs for connectors and therefore no way to model a
camera module in dt in a way that works for multiple processing boards.
Still it would be very helpful to be able to write dt overlays for these
modules that work with an upstream kernel without the need for nasty
hacks (like abusing the label property or so).

3. The same applies for full custom solutions where upstreaming
compatible strings will never happen, but we need the overall mechanics.

So in summary I'm looking for a place to put a blob of information
regarding the camera module that is not of any use to the kernel and
only passed on to userspace. Defining a separate node and maintaining
compatible strings seems like a lot of maintenance work for little gain.

Looking for other places to put that information doesn't lead to really
good candidates. One could store it in the software image itself, but
that is a pain as a user would loose it when flashing a new image.

One could put it in a extra partition with the same issues as above and
a huge technical complexity for use cases like the RaspberryPi.

So the devicetree seems to be a good location for it, just that we're
lacking a place to put it there.

I hope that makes a bit of sense.

Best regards,
Stefan

> 
> On the module name front, it quickly becomes compatible-v2 I think,
> since you're gonna need standardised names for specific modules so that
> decisions can be made on the basis of them. Makes me wonder if you need
> sensors to have an endpoint connection to whatever sits in front of
> them...
> 
> > Q: Would the dtb maintainers accept such a property?
> 
> I mean, at first glance the idea itself seems reasonable. The info isn't
> discoverable and some of it seems to be valuable.
> 
> > A:
> > 
> > Q: Are there other/better ways to convey that information?
> > A:
> > 
> > # Links:
> > 
> > - [1]
> > https://www.linuxtv.org/downloads/presentations/media_summit_2026/Stefan%20-%20Module%20Identification.pdf
> > - [2] https://github.com/systemd/systemd/blob/main/rules.d/70-camera.rules
> > - [3] https://github.com/systemd/systemd/blob/main/hwdb.d/70-cameras.hwdb
> >