Re: RFC: Intel IPU4 driver proof of concept

[Antti Laakso <antti.laakso@linux.intel.com>]

Hi Ruslan,

Thank you for working on this.

On Wed, Mar 04, 2026 at 11:03:51AM +0000, Ruslan Bay wrote:
> Currently there are multiple IPU driver implementations actively
> maintained by Intel engineers:
> 
> - mainline IPU6 [1]
> - downstream IPU6 [2]
> - downstream IPU4/IPU4P [3]
> - staging IPU7 [4]
> - downstream IPU7 [5]
> 
> As mentioned earlier, IPU4 and IPU6 share a large portion of the
> code base, and IPU7 appears architecturally very similar as well.
> 
> The IPU7 TODO mentions working toward a common IPU module [6].
> There was also an attempt to move from ipu6_* back to more
> generic ipu_* naming [7].
> 
> Is a unified IPU core still planned?

We're working on extending ipu6 driver to support ipu7 as well. The
exact schedule is not available yet.

> 
> If so, is it expected to include IPU4/IPU4P support?
> 
> Links:
> [1] https://github.com/torvalds/linux/tree/master/drivers/media/pci/intel/ipu6
> [2] https://github.com/intel/ipu6-drivers
> [3] https://github.com/intel/linux-intel-lts/tree/lts-v5.15.195-android_t-251103T063840Z/drivers/media/pci/intel
> [4] https://github.com/torvalds/linux/tree/master/drivers/staging/media/ipu7
> [5] https://github.com/intel/ipu7-drivers
> [6] https://github.com/torvalds/linux/blob/master/drivers/staging/media/ipu7/TODO#L17
> [7] https://lore.kernel.org/all/20250502154446.88965-6-stanislaw.gruszka@linux.intel.com/
> 
> 
> 
> On Sunday, February 22nd, 2026 at 8:57 PM, Ruslan Bay <ruslanbey@proton.me> wrote:
> 
> > We now have a working IPU4P driver for Ice Lake devices [1][2].
> > 
> > The current IPU4P implementation is based on Intel’s downstream IPU4
> > driver [3]. ISYS capture works with libcamera and has been tested on
> > Surface Pro 7 and Surface Book 3 [4]. The world-facing camera (ov8865)
> > works; the user-facing (ov5693) is still being debugged.
> > 
> > IPU4P and IPU6 both contain PSYS implementations downstream, but in
> > practice only ISYS is usable with libcamera today.
> > 
> > Earlier in this thread Andreas noted that IPU4 and IPU6 share more than
> > 85% of the code base.IPU7 appears architecturally very similar as well.
> > 
> > Before preparing an RFC, I would like clarification on direction:
> > 
> > 1. Is the long-term plan to unify IPU6 and IPU7 under a common driver
> >    structure?
> > 2. If so, should IPU4/IPU4P be aligned on top of that?
> > 3. If not, would it make sense to follow Andreas’ approach [5],
> >    implement IPU4P on top of the IPU6 structure, and move it to
> >    staging while iterating, as has been done for IPU7?
> > 
> > The primary goal is upstream IPU4P support (large Ice Lake user base),
> > but ideally this should align with the Apollo Lake IPU4 work shared
> > earlier [5].
> > 
> > What direction would you recommend?
> > 
> > [1] https://github.com/ruslanbay/ipu4-drivers/tree/main/patches/kernel/v6.19
> > [2] https://github.com/ruslanbay/linux/commits/ipu4-6.19
> > [3] https://github.com/intel/linux-intel-lts/tree/lts-v5.15.195-android_t-251103T063840Z/drivers/media/pci/intel
> > [4] https://github.com/linux-surface/linux-surface/discussions/1353?sort=new
> > [5] https://github.com/Kleist/ipu4-driver
> > 
> > Thanks,
> > Ruslan Bay
> > 
> > On 12/20/23 1:53 PM, Andreas Helbech Kleist wrote:
> > > Hi,
> > > 
> > > As mentioned previously in Bingbu's IPU6 patch series, I'm working on
> > > porting the driver to IPU4. I've now got a hole through so I think it
> > > makes sense sense to share the code.
> > > 
> > > I'm able to capture frames with yavta with the current code, but there
> > > are several issues that needs to be fixed for it to be complete.
> > > 
> > > # How it is tested
> > > ==================
> > > The hardware is a custom x86 PC-like embedded device with the following
> > > video pipeline:
> > > Endoscope -> FPGA -> tc358748 -> IPU4 (E3950/Apollo Lake)
> > > 
> > > See my colleague Claus' description[2] for more info.
> > > 
> > > There is currently no V4L2 subdevice for the FPGA, so we have a custom
> > > ambu-tc358748.c driver which pretends to be an image sensor.
> > > 
> > > $ media-ctl -v \
> > >   -V "\
> > >     \"tc358748 0-000e\"    :0 [fmt:RGB888_1X24/800x800],\
> > >     \"Intel IPU4 CSI2 0\"  :0 [fmt:RGB888_1X24/800x800],\
> > >     \"Intel IPU4 CSI2 0\"  :1 [fmt:RGB888_1X24/800x800]\
> > >     "\
> > >   -l "\
> > >     \"tc358748 0-000e\"    :0 -> \"Intel IPU4 CSI2 0\" :0 [1],\
> > >     \"Intel IPU4 CSI2 0\"  :1 -> \"Intel IPU4 ISYS Capture 12\" :0 [5]\
> > >   "
> > > 
> > > $ yavta --data-prefix -c2 -n2 -I -s 800x800 --file=/tmp/frame-#.bin \
> > >         -f XBGR32 /dev/video12
> > > 
> > > This produces frame-*.bin files containing 800x800x4 bytes of valid
> > > "BGR0" data.
> > > 
> > > # The code
> > > ==========
> > > The code is available at the tag
> > > https://github.com/Kleist/linux/tree/kleist-v6.6-ipu4-hacks-1
> > > (15245fe26e07)
> > > 
> > > 
> > > Note that I haven't renamed the files to ipu4, to make it clear what
> > > the changes are compared to the IPU6 driver.
> > > 
> > > It is based on v6.6 with the IPU6 v2 patches[1] on top, and then my
> > > hacks to make the IPU4 work. This is not meant for upstreaming as it
> > > is. The commits are a cleaned up version of the chronological order I
> > > made the port in. It is not yet in a state where I think an RFC PATCH
> > > series makes sense yet, but I wanted to share it anyway.
> > > 
> > > ## Changes compared to IPU6
> > > diff --stat of the changes in ../ipu6/ compared to the IPU6 v2 patches:
> > > 
> > >  drivers/media/pci/intel/ipu6/Kconfig               |  12 +-
> > >  drivers/media/pci/intel/ipu6/Makefile              |  13 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-bus.c            |   2 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-bus.h            |   6 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-buttress.c       |  71 ++-
> > >  drivers/media/pci/intel/ipu6/ipu6-buttress.h       |   8 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-fw-com.c         |  45 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-fw-com.h         |   2 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-fw-isys.c        | 171 ++++---
> > >  drivers/media/pci/intel/ipu6/ipu6-fw-isys.h        | 237 ++++++----
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-csi2.c      | 219 +++++----
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-csi2.h      |  11 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-queue.c     |  33 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-queue.h     |   8 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-video.c     | 212 +++------
> > >  drivers/media/pci/intel/ipu6/ipu6-isys-video.h     |   4 -
> > >  drivers/media/pci/intel/ipu6/ipu6-isys.c           | 435 +++----------
> > > -----
> > >  drivers/media/pci/intel/ipu6/ipu6-isys.h           |  18 +-
> > >  drivers/media/pci/intel/ipu6/ipu6-mmu.c            | 130 +++++-
> > >  .../pci/intel/ipu6/ipu6-platform-buttress-regs.h   |  98 +---
> > >  .../pci/intel/ipu6/ipu6-platform-isys-csi2-reg.h   | 226 ++-------
> > >  drivers/media/pci/intel/ipu6/ipu6-platform-regs.h  | 172 ++-----
> > >  drivers/media/pci/intel/ipu6/ipu6.c                | 511 ++++++++-----
> > > --------
> > >  drivers/media/pci/intel/ipu6/ipu6.h                |  37 +-
> > >  24 files changed, 1032 insertions(+), 1649 deletions(-)
> > > 
> > > Note that most of the deleted lines are removed because they are not
> > > used in IPU4. E.g. the watermark handling, which I haven't seen an
> > > equivalent for in the old IPU4 driver.
> > > 
> > > ## Ambu-specific tweaks
> > > Note that I'm using a hacked ipu-bridge (AMBU_IPU_BRIDGE) to setup the
> > > fwnode graph for our hardware. You don't want if you're testing this,
> > > so revert at least the "ambu: Add AMBU_IPU_BRIDGE" commit.
> > > 
> > > I'm not sure the right approach for handling this would be going
> > > forward. Of course the ambu-ipu-bridge shouldn't be upstreamed, so I'm
> > > wondering how we can achieve something similar? The ACPI tables from
> > > our BIOS unfortunately don't contain any info about the Toshiba Bridge
> > > (tc358748), so we can't derive the information from there. Maybe some
> > > kind of platform driver could be created which tweaks the ACPI info
> > > before the ipu-bridge driver reads it?
> > > 
> > > What do you typically do when you have some proprietary hardware that
> > > does not provide proper ACPI information? We could carry the ambu-ipu-
> > > bridge patches in our internal kernel tree, but that is not desirable
> > > in the long term.
> > > 
> > > # Inspiration for the IPU4 port
> > > ===============================
> > > We are currently using a Intel LTS 4.19.217 based kernel[3], which
> > > contains the old IPU4 driver. The port was basically made by comparing
> > > mmiotrace's between the old IPU4 driver and the new driver.
> > > 
> > > We're using the IPU4 FW ipu4_cpd_b0.bin extracted from a ClearLinux
> > > package[4].
> > > 
> > > # Known issues
> > > ==============
> > > ## Doesn't yet work with gstreamer for unknown reasons
> > > I get "Unexpected buffer address:" errors from
> > > ipu6_isys_queue_buf_ready, and don't get an image through.
> > > 
> > > ## 64 byte chunks of wrong data
> > > We occasionally get 64 byte aligned 64 byte wrong data (all 0xCC) in
> > > the captured frame*.bin files. This could be a cache invalidation
> > > issue, we haven't looked into this yet. The code currently doesn't use
> > > zlw_invalidate, even though it was ported from the old driver. We
> > > haven't yet tested if enabling this fixes the issue.
> > > 
> > > # Upstreaming
> > > =============
> > > We would like to upstream this driver, probably after the IPU6 driver
> > > has been merged. We're definitely not ready yet (either), but I already
> > > have a couple of questions, that it would be nice to get some input on
> > > from the community.
> > > 
> > > ## How to share code between IPU4 and IPU6
> > > Big parts of the code (approximately 6k out of 7k lines) does not need
> > > to be changed compared to the IPU6 driver, so there is clearly a big
> > > overlap in what the two drivers need to do. I'm not sure how the best
> > > approach would be for sharing this functionality. I see a few options:
> > > 1. Shared driver that supports both IPU's (still split in PCI driver
> > > and -isys driver)
> > > 2. Shared PCI driver that supports both IPU's, but device-specific
> > > intel-ipu4-isys/intel-ipu6-isys drivers
> > > 3. Separate drivers that use a shared "library module" (for lack of a
> > > better term)
> > > 
> > > My gut feeling is that 2. is the right choice, especially if we moved
> > > the shared code in to the PCI driver and the more version-specific code
> > > was moved into the specific drivers.
> > > 
> > > The answer to this could also be input to Bingbu's IPU6 series, maybe
> > > it would make sense to place some files differently if they eventually
> > > will be used in both IPU4 and IPU6 drivers?
> > > 
> > > ## How to implement our platform specific fwnode graph?
> > > As mentioned above, we currently have a hacked ambu-ipu-bridge driver,
> > > which is clearly not upstreamable. What would you typically do if you
> > > need to make a v4l setup where the ACPI table information about
> > > sensors/bridges is missing?
> > > 
> > > /Andreas
> > > 
> > > [1]https://lore.kernel.org/all/20231024112924.3934228-1-bingbu.cao@intel.com/
> > > [2]
> > > https://lore.kernel.org/all/471df7ffdf34b73d186c429a366cfee62963015f.camel@gmail.com/
> > > [3]
> > > https://github.com/intel/linux-intel-lts/tree/lts-v4.19.217-base-211118T072627Z
> > > [4]
> > > https://download.clearlinux.org/releases/32370/clear/source/SRPMS/linux-firmware-ipu-19ww39-104.src.rpm
> > 
> >
>