[PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[Bjoern Doebel]

After resume, MSI-X interrupts can be silently dropped because the ITS
hardware state does not match the software its_device state. The
in-memory tables pointed at by GITS_BASER survive suspend, but the ITS
has been reset and must be reconfigured via ITS commands, per the GICv3
ITS architecture specification §5.6.1 (Enabling an ITS). Some ITS
implementations also keep internal state that is only populated via ITS
commands rather than by reading guest/firmware memory on demand, so
restoring GITS_BASER alone is not enough.

Before commit 713335b6ee29 ("irqchip/gic-v3-its: Implement
.msi_teardown() callback"), pci_free_irq_vectors() tore down the
its_device (MAPD valid=0, ITT freed) and pci_alloc_irq_vectors() rebuilt
it (MAPD valid=1). Drivers that disabled/re-enabled MSI-X across
suspend/resume (e.g. ENA, NVMe) thus reprogrammed the ITS as a side
effect. After commit 713335b6ee29 ("irqchip/gic-v3-its: Implement
.msi_teardown() callback"), device teardown moved to .msi_teardown(),
which only runs when the MSI domain is removed (driver unbind). Since
the MSI domain persists across suspend/resume, MAPD is never replayed.

Fix this in its_restore_enable() and its_cpu_init_collection() by
walking the preserved software state and re-issuing the ITS commands
needed to bring the hardware back in sync:

  1. For each device, issue MAPD(V=0), zero the ITT, then MAPD(V=1)
     with the same parameters. §5.3.10 makes MAPD(V=1) with a non-zero
     ITT UNPREDICTABLE, so the ITT must be zeroed first.

  2. Restore every CPU's collection (MAPC) and replay MAPTI for events
     targeting that CPU, once all target collections have been mapped.
     The per-event replay is driven by a bool parameter to
     its_cpu_init_collection() so that every ITS on a given CPU gets
     its MAPTIs restored. For the boot CPU, which does not traverse
     its_cpu_init_collections() on resume, replay is driven directly
     from its_restore_enable() for every ITS; the HCC optimisation
     that previously skipped MAPC for memory-resident collections on
     the boot CPU is dropped, matching what secondary CPUs already do
     in their normal cpuhp startup path. For secondary CPUs, replay
     is gated by a cpumask armed by its_restore_enable() once per
     resume cycle so that normal CPU hotplug is unaffected.

GICv4 vLPI state is skipped here: vLPIs are hypervisor-only, replayed
through separate GICv4 VM resume paths, and not relevant to guest
kernels or to this fix.

Tested on EC2 c6gn.16xlarge (ARM64 Graviton). Without the fix,
hibernation resume fails 100% with:

  ena 0000:00:05.0: The ena device sent a completion but the driver
  didn't receive a MSI-X interrupt (cmd 3)
  ena 0000:00:05.0: Failed to create IO CQ. error: -62

With the fix, hibernation resume works reliably.

Fixes: 713335b6ee29 ("irqchip/gic-v3-its: Implement .msi_teardown() callback")
Cc: stable@vger.kernel.org
Cc: Marc Zyngier <maz@kernel.org>
Cc: Thomas Gleixner <tglx@kernel.org>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Cc: David Woodhouse <dwmw@amazon.co.uk>
Cc: Ali Saidi <alisaidi@amazon.com>
Co-developed-by: David Arinzon <darinzon@amazon.com>
Signed-off-by: David Arinzon <darinzon@amazon.com>
Co-developed-by: Zeev Zilberman <zeev@amazon.com>
Signed-off-by: Zeev Zilberman <zeev@amazon.com>
Signed-off-by: Bjoern Doebel <doebel@amazon.de>
Assisted-by: Kiro:claude-opus-4.6
---
Testing: Tested hibernation using Amazon Linux 2023 and kernel 7.1-rc2
on EC2 c6gn, c7gn, and c8gn instances. Without the patch, hibernation
failed to bring up the ENA network device. With the patch, ENA devices
are properly re-initialized on resume.
---
 drivers/irqchip/irq-gic-v3-its.c | 124 ++++++++++++++++++++++++++++---
 1 file changed, 114 insertions(+), 10 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 291d7668cc8da..0d240230037cd 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3283,7 +3283,66 @@ static void its_cpu_init_lpis(void)
 		&paddr);
 }
 
-static void its_cpu_init_collection(struct its_node *its)
+static cpumask_var_t its_restore_pending_cpus;
+
+static void its_restore_device(struct its_device *its_dev)
+{
+	/*
+	 * Bring each device back to a quiescent mapping state, as required
+	 * by GICv3 ITS architecture §5.6.1 (Enabling an ITS) after an ITS
+	 * reset: the device table entries are gone, so software must
+	 * reconfigure them with ITS commands. MAPD(V=1) with a non-zero
+	 * ITT is UNPREDICTABLE (§5.3.10, §5.2.4), so unmap first, zero the
+	 * ITT, and map again with a clean ITT. MAPTI replay is deferred to
+	 * its_cpu_init_collection() so that the collection a given event
+	 * targets is MAPC'd before MAPTI is issued for it.
+	 */
+	its_send_mapd(its_dev, 0);
+	memset(its_dev->itt, 0, its_dev->itt_sz);
+	gic_flush_dcache_to_poc(its_dev->itt, its_dev->itt_sz);
+	its_send_mapd(its_dev, 1);
+}
+
+static void its_cpu_replay_mapti(struct its_node *its)
+{
+	int cpu = smp_processor_id();
+	struct its_device *its_dev;
+	int event;
+
+	/*
+	 * Walk its_device_list without holding its->dev_alloc_lock.
+	 * Device add/remove normally requires that mutex, but this
+	 * function only runs on the resume path, from
+	 * its_cpu_init_collection() on either the boot CPU (called
+	 * directly from its_restore_enable() under its_lock) or a
+	 * secondary CPU (called from its_cpu_init_collections() under
+	 * its_lock). Concurrency with driver MSI alloc/free is excluded
+	 * by the hibernate sequence:
+	 *
+	 *   syscore_resume()            <- its_restore_enable() runs here
+	 *   pm_sleep_enable_secondary_cpus()  <- its_cpu_init() on each CPU
+	 *   dpm_resume_start() / dpm_resume()  <- driver .resume callbacks
+	 *
+	 * See kernel/power/hibernate.c:resume_target_kernel(). Drivers
+	 * cannot add or remove MSI allocations until their .resume
+	 * callbacks run, which is strictly after every CPU has passed
+	 * through its_cpu_init_collection().
+	 */
+	list_for_each_entry(its_dev, &its->its_device_list, entry) {
+		if (its_dev->event_map.vm)
+			continue;
+		for_each_set_bit(event, its_dev->event_map.lpi_map,
+				 its_dev->event_map.nr_lpis) {
+			if (its_dev->event_map.col_map[event] != cpu)
+				continue;
+			its_send_mapti(its_dev,
+				       its_dev->event_map.lpi_base + event,
+				       event);
+		}
+	}
+}
+
+static void its_cpu_init_collection(struct its_node *its, bool replay)
 {
 	int cpu = smp_processor_id();
 	u64 target;
@@ -3320,17 +3379,33 @@ static void its_cpu_init_collection(struct its_node *its)
 
 	its_send_mapc(its, &its->collections[cpu], 1);
 	its_send_invall(its, &its->collections[cpu]);
+
+	/*
+	 * On resume from hibernation, its_restore_enable() has reprogrammed
+	 * the device table but deferred per-event MAPTI replay until each
+	 * target collection is MAPC'd. Now that the local collection is
+	 * mapped, replay MAPTIs for events targeting this CPU on this ITS.
+	 */
+	if (replay)
+		its_cpu_replay_mapti(its);
 }
 
 static void its_cpu_init_collections(void)
 {
 	struct its_node *its;
+	bool replay;
 
-	raw_spin_lock(&its_lock);
+	/*
+	 * On resume from hibernation, its_restore_enable() arms this cpumask
+	 * for every secondary CPU that still needs MAPTI replay. Test-and-
+	 * clear once per CPU and propagate the flag to every ITS on this CPU.
+	 */
+	replay = cpumask_test_and_clear_cpu(smp_processor_id(),
+					    its_restore_pending_cpus);
 
+	raw_spin_lock(&its_lock);
 	list_for_each_entry(its, &its_nodes, entry)
-		its_cpu_init_collection(its);
-
+		its_cpu_init_collection(its, replay);
 	raw_spin_unlock(&its_lock);
 }
 
@@ -5036,8 +5111,22 @@ static void its_restore_enable(void *data)
 	struct its_node *its;
 	int ret;
 
+	/*
+	 * Arm MAPTI replay for every secondary CPU. The boot CPU does not
+	 * go through its_cpu_init_collections() on resume, so it is handled
+	 * directly in the per-ITS loop below; exclude it here to avoid
+	 * leaving a stale bit set.
+	 *
+	 * See §5.6.1 of the GICv3 ITS architecture specification: after an
+	 * ITS reset, software must reconfigure devices, collections and
+	 * translations via ITS commands.
+	 */
+	cpumask_copy(its_restore_pending_cpus, cpu_possible_mask);
+	cpumask_clear_cpu(smp_processor_id(), its_restore_pending_cpus);
+
 	raw_spin_lock(&its_lock);
 	list_for_each_entry(its, &its_nodes, entry) {
+		struct its_device *its_dev;
 		void __iomem *base;
 		int i;
 
@@ -5080,13 +5169,23 @@ static void its_restore_enable(void *data)
 		writel_relaxed(its->ctlr_save, base + GITS_CTLR);
 
 		/*
-		 * Reinit the collection if it's stored in the ITS. This is
-		 * indicated by the col_id being less than the HCC field.
-		 * CID < HCC as specified in the GIC v3 Documentation.
+		 * Reset and remap each device on this ITS. After resume,
+		 * the ITS has no device table entries and ITT contents may
+		 * be stale; per GICv3 ITS §5.3.10, MAPD(V=1) with a non-zero
+		 * ITT is UNPREDICTABLE. Unmap first, zero the ITT, then map
+		 * again.
 		 */
-		if (its->collections[smp_processor_id()].col_id <
-		    GITS_TYPER_HCC(gic_read_typer(base + GITS_TYPER)))
-			its_cpu_init_collection(its);
+		list_for_each_entry(its_dev, &its->its_device_list, entry)
+			its_restore_device(its_dev);
+
+		/*
+		 * Unconditionally MAPC the boot CPU's collection and replay
+		 * MAPTIs for events targeting it, on every ITS. This mirrors
+		 * the unconditional MAPC that secondary CPUs do in their
+		 * cpuhp startup path, and covers both HW-resident and
+		 * memory-resident collections.
+		 */
+		its_cpu_init_collection(its, true);
 	}
 	raw_spin_unlock(&its_lock);
 }
@@ -5826,6 +5925,11 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	if (!itt_pool)
 		return -ENOMEM;
 
+	if (!zalloc_cpumask_var(&its_restore_pending_cpus, GFP_KERNEL)) {
+		gen_pool_destroy(itt_pool);
+		return -ENOMEM;
+	}
+
 	gic_rdists = rdists;
 
 	lpi_prop_prio = irq_prio;
-- 
2.48.2




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[Bjoern Doebel]

Hi all,

gentle ping on this one.

Since the original posting I've re-validated the fix against current
mainline:

  - It still applies cleanly to v7.2-rc1 (and to v7.1.0).

  - I reproduced the original failure on *stock* v7.2-rc1. On EC2
    Graviton instances, hibernation resume fails 100% of the time: the
    ITS comes back reset, MAPD/MAPTI are never replayed, and the ENA
    NIC silently loses its LPIs:

      ena 0000:00:05.0: ... didn't receive a MSI-X interrupt (cmd 3)
      ena 0000:00:05.0: Failed to create IO CQ. error: -62

    The instance then has no networking after resume.

  - With this patch applied, the same kernel survives hibernate/resume
    cleanly: 9/9 cycles with zero failures, across all three Graviton
    generations (Graviton 2/3/4, i.e. Neoverse N1/V1/V2), networking
    fully restored on every resume.

As described in the previous message, this is the fallout from 713335b6ee29
("irqchip/gic-v3-its: Implement .msi_teardown() callback"): device
teardown no longer happens across a suspend/resume that keeps the MSI
domain, so the ITS is never reprogrammed and drops interrupts after the
hardware has been reset.

Could you take a look when you get a chance?

Thanks,
Bjoern




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Re: [PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[Thomas Gleixner]

On Thu, Jul 02 2026 at 11:57, Bjoern Doebel wrote:

> Hi all,
>
> gentle ping on this one.

Marc?

> Since the original posting I've re-validated the fix against current
> mainline:
>
>   - It still applies cleanly to v7.2-rc1 (and to v7.1.0).
>
>   - I reproduced the original failure on *stock* v7.2-rc1. On EC2
>     Graviton instances, hibernation resume fails 100% of the time: the
>     ITS comes back reset, MAPD/MAPTI are never replayed, and the ENA
>     NIC silently loses its LPIs:
>
>       ena 0000:00:05.0: ... didn't receive a MSI-X interrupt (cmd 3)
>       ena 0000:00:05.0: Failed to create IO CQ. error: -62
>
>     The instance then has no networking after resume.
>
>   - With this patch applied, the same kernel survives hibernate/resume
>     cleanly: 9/9 cycles with zero failures, across all three Graviton
>     generations (Graviton 2/3/4, i.e. Neoverse N1/V1/V2), networking
>     fully restored on every resume.
>
> As described in the previous message, this is the fallout from 713335b6ee29
> ("irqchip/gic-v3-its: Implement .msi_teardown() callback"): device
> teardown no longer happens across a suspend/resume that keeps the MSI
> domain, so the ITS is never reprogrammed and drops interrupts after the
> hardware has been reset.
>
> Could you take a look when you get a chance?
>
> Thanks,
> Bjoern
>
>
>
>
> Amazon Web Services Development Center Germany GmbH
> Tamara-Danz-Str. 13
> 10243 Berlin
> Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
> Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
> Sitz: Berlin
> Ust-ID: DE 365 538 597

Re: [PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[Marc Zyngier]

On Thu, 02 Jul 2026 12:57:15 +0100,
Bjoern Doebel <doebel@amazon.de> wrote:
> 
> Hi all,
> 
> gentle ping on this one.
> 
> Since the original posting I've re-validated the fix against current
> mainline:
> 
>   - It still applies cleanly to v7.2-rc1 (and to v7.1.0).
> 
>   - I reproduced the original failure on *stock* v7.2-rc1. On EC2
>     Graviton instances, hibernation resume fails 100% of the time: the
>     ITS comes back reset, MAPD/MAPTI are never replayed, and the ENA
>     NIC silently loses its LPIs:
> 
>       ena 0000:00:05.0: ... didn't receive a MSI-X interrupt (cmd 3)
>       ena 0000:00:05.0: Failed to create IO CQ. error: -62
>

But how did the resumed kernel get there the first place? Surely you
had interrupts to load it, right?

>     The instance then has no networking after resume.
> 
>   - With this patch applied, the same kernel survives hibernate/resume
>     cleanly: 9/9 cycles with zero failures, across all three Graviton
>     generations (Graviton 2/3/4, i.e. Neoverse N1/V1/V2), networking
>     fully restored on every resume.
> 
> As described in the previous message, this is the fallout from 713335b6ee29
> ("irqchip/gic-v3-its: Implement .msi_teardown() callback"): device
> teardown no longer happens across a suspend/resume that keeps the MSI
> domain, so the ITS is never reprogrammed and drops interrupts after the
> hardware has been reset.
> 
> Could you take a look when you get a chance?

What I don't immediately see is how this particular patch influences
anything, given that at this point it isn't doing anything.

Beside that, how does hibernation actually influences LPIs being
released? Can you at least describe the sequence of events?

The other thing that worries me is that in general, (bare metal) GICv3
is unable to deal with hibernation, given that you can't reprogram the
property and pending table addresses. Yes, it *may* work if you are
lucky enough that the boot kernel and the resumed one use the same
memory regions, but this only show that you like playing Russian
roulette.

Thanks,

	M.

-- 
Jazz isn't dead. It just smells funny.

Re: [PATCH] irqchip/gic-v3-its: Reconfigure ITS from software state on resume

[Bjoern Doebel]

Hi Marc,

thanks for your response and apologies for my delay (I'm paying for this
by having to page all this back into my head).

> >   - I reproduced the original failure on *stock* v7.2-rc1. On EC2
> >     Graviton instances, hibernation resume fails 100% of the time: the
> >     ITS comes back reset, MAPD/MAPTI are never replayed, and the ENA
> >     NIC silently loses its LPIs:
> > 
> >       ena 0000:00:05.0: ... didn't receive a MSI-X interrupt (cmd 3)
> >       ena 0000:00:05.0: Failed to create IO CQ. error: -62
> >
> 
> But how did the resumed kernel get there the first place? Surely you
> had interrupts to load it, right?

Pre-hibernate we had a correctly configured guest (VM-1) on a KVM host,
including properly setup ITS etc. Then we hibernated this VM, that is
the guest wrote all its memory content into a snapshot and powered down.
While this contains some guest-side GIC state, it does not contain the
host-side virtual GIC state KVM stores.

Now we resume, that is we launch a new VM (VM-2), potentially on a
different host. This VM configures the underlying vGIC and has working
devices. At some point it then loads the hibernated memory image and
jumps into the resumed kernel. At this point, we are running a VM
configured by VM-2, but we just restored the in-guest ITS state from
VM-1. After this, the network driver no longer receives interrupts from
the device because it relies on mappings VM-1 established while the 2nd
vGIC was never told about them.

The solution to this problem is to replay VM-1's MAPD, MAPC and MAPTI
commands, which re-programs the 2nd vGIC with the mappings VM-1 had
before.

> >     The instance then has no networking after resume.
> > 
> >   - With this patch applied, the same kernel survives hibernate/resume
> >     cleanly: 9/9 cycles with zero failures, across all three Graviton
> >     generations (Graviton 2/3/4, i.e. Neoverse N1/V1/V2), networking
> >     fully restored on every resume.
> > 
> > As described in the previous message, this is the fallout from 713335b6ee29
> > ("irqchip/gic-v3-its: Implement .msi_teardown() callback"): device
> > teardown no longer happens across a suspend/resume that keeps the MSI
> > domain, so the ITS is never reprogrammed and drops interrupts after the
> > hardware has been reset.
> > 
> > Could you take a look when you get a chance?
> 
> What I don't immediately see is how this particular patch influences
> anything, given that at this point it isn't doing anything.
> 
> Beside that, how does hibernation actually influences LPIs being
> released? Can you at least describe the sequence of events?

Before 6.16 this worked "by accident". On suspend, we call ena_suspend()
-> ena_destroy_device() -> ena_disable_msix() -> pci_free_irq_vectors().
This ends up freeing all IRQ vectors and eventually calling
its_irq_domain_free(). This function releases the events from the event
map, finds the event map has run empty, and then tears the its_device
down with its_lpi_free() -> MAPD(V=0) -> its_free_device(). This happens
while devices are frozen, before the snapshot is taken. Hence, no
its_device for the NIC is stored within the hibernation snapshot.

On resume, its_msi_prepare() would then not find anything for that
DeviceID and therefore call its_create_device(). This allocates ITT and
LPI range, and issues MAPD. Eventually, we end up with VM-2's vGIC being
reconfigured from scratch.

6.16 changed the lifetime of the its_device. 713335b6ee29
("irqchip/gic-v3-its: Implement .msi_teardown() callback") moved the
its_device teardown out of its_irq_domain_free() into a new
.msi_teardown() hook, and 03c298760ed9 ("genirq/msi: Engage the
.msi_teardown() callback on domain removal") tied that hook to domain
removal. The empty event map went from being the condition that
triggered teardown to a WARN_ON_ONCE precondition. As 713335b6ee29
explains, coupling the its_device to the event count was wrong, since
drivers legitimately drop to zero MSIs and reallocate.

Our hibernation was relying on exactly that teardown. Suspend still
frees all vectors and empties the event map, but it does not remove the
MSI domain. .msi_teardown() never runs. The its_device survives with
an empty event map, and it therefore ends up in the snapshot. On resume
its_msi_prepare() finds it again via its_find_device(), marks it shared
and returns without issuing MAPD, so its_create_device() is never
reached. VM-2's vGIC is never told about the DeviceID. 

So before 6.16, the snapshot had no its_device and we got full
reconfiguration for free. Since 6.16 the snapshot has an its_device and
nothing reprograms the vGIC. All of this was never meant to support
hibernation. We should always have been restoring ITS state explicitly
on resume.

My patch reconstructs the ITS hardware state from the software state
restored with the image. That state is all present at syscore_resume()
time and does not depend on any device having resumed.
its_restore_enable() already performs the first three steps of the GICv3
ITS §5.6.1 enable sequence and stops short of the fourth, configuring
devices, collections and translations with ITS commands. So we walk
its_device_list and for each device issue MAPD(V=0), zero and flush the
ITT, then issue MAPD(V=1), since §5.3.10 makes MAPD(V=1) with a non-zero
ITT UNPREDICTABLE. Per-event MAPTI replay is deferred to
its_cpu_init_collection(), since an event's target collection has to be
MAPC'd before its MAPTI can be issued. All of it completes before
dpm_resume() runs the driver .resume callbacks, so the ITS is programmed
before any device raises an MSI.

> The other thing that worries me is that in general, (bare metal) GICv3
> is unable to deal with hibernation, given that you can't reprogram the
> property and pending table addresses. Yes, it *may* work if you are
> lucky enough that the boot kernel and the resumed one use the same
> memory regions, but this only show that you like playing Russian
> roulette.

Point taken. This patch is not trying to make hibernation work on bare
metal. It restores ITS command state only and does not touch
GICR_PROPBASER/GICR_PENDBASER, which cannot be re-pointed once
GICR_CTLR.EnableLPIs is set. If a redistributor lost power, replaying
ITS commands will not be enough. I am happy to state that limitation in
the commit message.

Best regards
Bjoern