IRQ domain logging?

IRQ domain logging?

[Bjorn Helgaas]

Hi Thomas,

IRQ domains and IRQs are critical infrastructure, but we don't really
log anything when we discover controllers or set them up.  Do you
think there would be any value in exposing some of this structure in
dmesg to help people (like me!) understand how these are connected to
devices and drivers?

For example, in a simple qemu system:

  IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
  ACPI: Using IOAPIC for interrupt routing
  ACPI: PCI: Interrupt link LNKA configured for IRQ 10
  ACPI: PCI: Interrupt link GSIA configured for IRQ 16
  hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
  ACPI: \_SB_.GSIA: Enabled at IRQ 16
  pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
  00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
  ata1: SATA max UDMA/133 abar m4096@0xfeadb000 port 0xfeadb100 irq 28 lpm-pol 0

I think these are all wired interrupts, and maybe IRQ==GSI (?), and I
think the ACPI link devices are configurable connections between an
INTx and the IOAPIC, but it's kind of hard to connect them all
together.

This from an arm64 system is even more obscure to me:

  NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
  GICv3: 256 SPIs implemented
  Root IRQ handler: gic_handle_irq
  GICv3: GICv3 features: 16 PPIs
  kvm [1]: vgic interrupt IRQ18
  xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000

I have no clue where irq 67 goes.

Maybe there's no useful way to log anything here, I dunno; it just
occurred to me when looking at Jiri's series to reduce the number of
irqdomain interfaces.  PCI controller drivers do a lot of interrupt
domain setup, and if that were more visible/concrete in dmesg, I think
I might understand it better.

Bjorn

Re: IRQ domain logging?

[Lukas Wunner]

On Tue, Apr 22, 2025 at 04:07:05PM -0500, Bjorn Helgaas wrote:
> This from an arm64 system is even more obscure to me:
> 
>   NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>   GICv3: 256 SPIs implemented
>   Root IRQ handler: gic_handle_irq
>   GICv3: GICv3 features: 16 PPIs
>   kvm [1]: vgic interrupt IRQ18
>   xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000
> 
> I have no clue where irq 67 goes.

There's quite a bit of information available in /proc/interrupts,
/proc/irq/ and /sys/kernel/irq/, I guess that's what most people use.

Thanks,

Lukas

Re: IRQ domain logging?

[Ethan Zhao]

On 4/23/2025 5:07 AM, Bjorn Helgaas wrote:
> Hi Thomas,
> 
> IRQ domains and IRQs are critical infrastructure, but we don't really
> log anything when we discover controllers or set them up.  Do you
> think there would be any value in exposing some of this structure in
> dmesg to help people (like me!) understand how these are connected to
> devices and drivers?
> 
> For example, in a simple qemu system:
> 
>    IOAPIC[0]: apic_id 0, version 32, address 0xfec00000, GSI 0-23
>    ACPI: Using IOAPIC for interrupt routing
>    ACPI: PCI: Interrupt link LNKA configured for IRQ 10
>    ACPI: PCI: Interrupt link GSIA configured for IRQ 16
>    hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
>    ACPI: \_SB_.GSIA: Enabled at IRQ 16
>    pcieport 0000:00:1c.0: PME: Signaling with IRQ 24
>    00:03: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
>    ata1: SATA max UDMA/133 abar m4096@0xfeadb000 port 0xfeadb100 irq 28 lpm-pol 0
> 
> I think these are all wired interrupts, and maybe IRQ==GSI (?), and I
> think the ACPI link devices are configurable connections between an
> INTx and the IOAPIC, but it's kind of hard to connect them all
> together.
> 
> This from an arm64 system is even more obscure to me:
> 
>    NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>    GICv3: 256 SPIs implemented
>    Root IRQ handler: gic_handle_irq
>    GICv3: GICv3 features: 16 PPIs
>    kvm [1]: vgic interrupt IRQ18
>    xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000
> 
> I have no clue where irq 67 goes.
> 
> Maybe there's no useful way to log anything here, I dunno; it just
> occurred to me when looking at Jiri's series to reduce the number of
> irqdomain interfaces.  PCI controller drivers do a lot of interrupt
> domain setup, and if that were more visible/concrete in dmesg, I think
> I might understand it better. 
The current visibility into interrupt routing in systems is fragmented, 
making it challenging to observe the routing behavior of specific 
interrupts or interrupt types. For enthusiasts exploring system 
internals, having a ​traceroute-like tool to map interrupt handling 
paths would significantly enhance transparency and debugging capabilities.

e.g. How an MSI is routed and remapped between different domains on x86

MSI pci-dev-->ioapic-->iommu-->apic-->cpu

But so far it seems there is no enough info from KABI(/proc /sysfs 
syscall, dmesg etc) to compose a complete chain like that ?

Thanks,
Ethan
> 
> Bjorn
> 

Re: IRQ domain logging?

[Thomas Gleixner]

On Wed, Apr 23 2025 at 10:23, Lukas Wunner wrote:
> On Tue, Apr 22, 2025 at 04:07:05PM -0500, Bjorn Helgaas wrote:
>> This from an arm64 system is even more obscure to me:
>> 
>>   NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
>>   GICv3: 256 SPIs implemented
>>   Root IRQ handler: gic_handle_irq
>>   GICv3: GICv3 features: 16 PPIs
>>   kvm [1]: vgic interrupt IRQ18
>>   xhci-hcd xhci-hcd.0.auto: irq 67, io mem 0xfe800000
>> 
>> I have no clue where irq 67 goes.
>
> There's quite a bit of information available in /proc/interrupts,
> /proc/irq/ and /sys/kernel/irq/, I guess that's what most people use.

/sys/kernel/debug/irq/....

Gives the most comprehensive insight into the domain hierarchy.

# cat /sys/kernel/debug/irq/irqs/204
handler:  handle_edge_irq
device:   0000:01:00.0
status:   0x00000000
istate:   0x00004000
ddepth:   0
wdepth:   0
dstate:   0x1d401200
            IRQD_ACTIVATED
            IRQD_IRQ_STARTED
            IRQD_SINGLE_TARGET
            IRQD_AFFINITY_SET
            IRQD_AFFINITY_ON_ACTIVATE
            IRQD_CAN_RESERVE
            IRQD_HANDLE_ENFORCE_IRQCTX
node:     0
affinity: 50
effectiv: 50
pending:  
domain:  PCI-MSIX-0000:01:00.0-12
 hwirq:   0x65
 chip:    PCI-MSIX-0000:01:00.0
  flags:   0x1430
             IRQCHIP_SKIP_SET_WAKE
             IRQCHIP_ONESHOT_SAFE
             IRQCHIP_MOVE_DEFERRED

  address_hi: 0x00000000
  address_lo: 0xfee32000
  msg_data:   0x00000023
 parent:
    domain:  VECTOR
     hwirq:   0xcc
     chip:    APIC
      flags:   0x0
     Vector:    35
     Target:    50
     move_in_progress: 0
     is_managed:       0
     can_reserve:      1
     has_reserved:     0
     cleanup_pending:  0

It shows the complete domain hierarchy for each interrupt an
irq/domains/.... gives extra information about the domains.

You need to enable GENERIC_IRQ_DEBUGFS.

Exposing this in dmesg in a comprehensible and condensed form might be
possible, but needs some thought.

So far I survived pretty good with the debugfs data.

Thanks,

        tglx