Re: [RFC PATCH v2 08/18] iommu/riscv: Use MSI table to enable IMSIC access

Thomas Gleixner
2025-09-23T10:12:54Z

On Mon, Sep 22 2025 at 20:56, Jason Gunthorpe wrote:
> On Mon, Sep 22, 2025 at 04:20:43PM -0500, Andrew Jones wrote:
>> > It has to do with each PCI BDF having a unique set of
>> > validation/mapping tables for MSIs that are granular to the interrupt
>> > number.
>>
>> Interrupt numbers (MSI data) aren't used by the RISC-V IOMMU in any way.
>
> Interrupt number is a Linux concept, HW decodes the addr/data pair and
> delivers it to some Linux interrupt. Linux doesn't care how the HW
> treats the addr/data pair, it can ignore data if it wants.

Let me explain this a bit deeper.

As you said, the interrupt number is a pure kernel software construct,
which is mapped to a hardware interrupt source.

The interrupt domain, which is associated to a hardware interrupt
source, creates the mapping and supplies the resulting configuration to
the hardware, so that the hardware is able to raise an interrupt in the
CPU.

In case of MSI, this configuration is the MSI message (address,
data). That's composed by the domain according to the requirements of
the underlying CPU hardware resource. This underlying hardware resource
can be the CPUs interrupt controller itself or some intermediary
hardware entity.

The kernel reflects this in the interrupt domain hierarchy. The simplest
case for MSI is:

[ CPU domain ] --- [ MSI domain ] -- device

The flow is as follows:

device driver allocates an MSI interrupt in the MSI domain

MSI domain allocates an interrupt in the CPU domain

CPU domain allocates an interrupt vector and composes the
address/data pair. If @data is written to @address, the interrupt is
raised in the CPU

MSI domain converts the address/data pair into device format and
writes it into the device.

When the device fires an interrupt it writes @data to @address, which
raises the interrupt in the CPU at the allocated CPU vector. That
vector is then translated to the Linux interrupt number in the
interrupt handling entry code by looking it up in the CPU domain.

With a remapping domain intermediary this looks like this:

[ CPU domain ] --- [ Remap domain] --- [ MSI domain ] -- device

device driver allocates an MSI interrupt in the MSI domain

MSI domain allocates an interrupt in the Remap domain

Remap domain allocates a resource in the remap space, e.g. an entry
in the remap translation table and then allocates an interrupt in the
CPU domain.

CPU domain allocates an interrupt vector and composes the
address/data pair. If @data is written to @address, the interrupt is
raised in the CPU

Remap domain converts the CPU address/data pair to remap table format
and writes it to the alloacted entry in that table. It then composes
a new address/data pair, which points at the remap table entry.

MSI domain converts the remap address/data pair into device format
and writes it into the device.

So when the device fires an interrupt it writes @data to @address,
which triggers the remap unit. The remap unit validates that the
address/data pair is valid for the device and if so it writes the CPU
address/data pair, which raises the interrupt in the CPU at the
allocated vector. That vector is then translated to the Linux
interrupt number in the interrupt handling entry code by looking it
up in the CPU domain.

So from a kernel POV, the address/data pairs are just opaque
configuration values, which are written into the remap table and the
device. Whether the content of @data is relevant or not, is a hardware
implementation detail. That implementation detail is only relevant for
the interrupt domain code, which handle a specific part of the
hierarchy.

The MSI domain does not need to know anything about the content and the
meaning of @address and @data. It just cares about converting that into
the device specific storage format.

The Remap domain does not need to know anything about the content and
the meaning of the CPU domain provided @address and @data. It just cares
about converting that into the remap table specific format.

The hardware entities do not know about the Linux interrupt number at
all. That relationship is purely software managed as a mapping from the
allocated CPU vector to the Linux interrupt number.

Hope that helps.

tglx