[PATCH 15/17] gpu: nova-core: retrigger the GSP falcon and clear every latched cause

[John Hubbard <jhubbard@nvidia.com>]

A falcon signals the interrupt tree when its set of enabled causes goes
from empty to non-empty. While any enabled cause stays latched, later
causes produce no signal, and Turing falcons have no INTR_RETRIGGER
register with which to supply one.

The GSP handler cleared its GIN leaf bit and then cleared the falcon's
SWGEN0 latch. A cause that arrived between the two left no record: the
leaf clear discarded it, and the falcon had nothing left to signal.
Swapping the two clears moves the window rather than closing it.

The handler serviced SWGEN0 or reported an unserviceable cause, never
both, so a HALT co-pending with SWGEN0 stayed latched. nova-core's probe
cleared the SWGEN0 latch before draining the tree, so a message posted
in between set a leaf bit that the drain then erased. In every case the
GSP went silent for the life of the device.

Write the falcon's INTR_RETRIGGER register after every clear of the GSP
vector. That supplies the missing signal from whatever causes remain
enabled. Turing falcons have no such register, so skip the write there.

Handle every cause the falcon reports on one invocation, masking the
ones with no recovery path so the re-emit does not raise them again.
Clear the SWGEN0 latch after the tree drain instead of before it.
Neither of these depends on INTR_RETRIGGER, so both apply on Turing.

Assisted-by: Cursor:claude-opus-5
Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/falcon/gsp.rs | 36 ++++++++++++++++++
 drivers/gpu/nova-core/falcon/hal.rs |  8 ++++
 drivers/gpu/nova-core/irq/gsp.rs    | 57 ++++++++++++++++++-----------
 drivers/gpu/nova-core/regs.rs       | 20 ++++++++++
 4 files changed, 100 insertions(+), 21 deletions(-)

diff --git a/drivers/gpu/nova-core/falcon/gsp.rs b/drivers/gpu/nova-core/falcon/gsp.rs
index f9d9e8e0386b..6ee5c1ef1af7 100644
--- a/drivers/gpu/nova-core/falcon/gsp.rs
+++ b/drivers/gpu/nova-core/falcon/gsp.rs
@@ -16,11 +16,13 @@
 use crate::{
     driver::Bar0,
     falcon::{
+        hal,
         Falcon,
         FalconEngine,
         PFalcon2Base,
         PFalconBase, //
     },
+    gpu::Chipset,
     regs,
 };
 
@@ -64,6 +66,40 @@ pub(crate) fn take_swgen0_intr(bar: Bar0<'_>) -> regs::NV_PFALCON_FALCON_IRQSTAT
 
         status
     }
+
+    /// Masks and clears every interrupt cause set in `status`.
+    ///
+    /// A masked cause leaves the falcon's enabled set, so it neither raises the tree again nor
+    /// holds that set non-empty.
+    pub(crate) fn mask_and_clear_intr(bar: Bar0<'_>, status: regs::NV_PFALCON_FALCON_IRQSTAT) {
+        let causes = status.into_raw();
+
+        bar.write(
+            WithBase::of::<Self>(),
+            regs::NV_PFALCON_FALCON_IRQMCLR::zeroed().with_value(causes),
+        );
+        bar.write(
+            WithBase::of::<Self>(),
+            regs::NV_PFALCON_FALCON_IRQSCLR::from(causes),
+        );
+    }
+
+    /// Re-emits the falcon's enabled interrupt causes into the interrupt tree.
+    ///
+    /// The falcon signals the tree on a transition of its enabled causes, so clearing the tree
+    /// leaf while a cause is still latched leaves no transition and no further vector.
+    ///
+    /// Does nothing on Turing, whose falcons do not implement the register.
+    pub(crate) fn retrigger_intr(bar: Bar0<'_>, chipset: Chipset) {
+        if !hal::has_intr_retrigger(chipset) {
+            return;
+        }
+
+        bar.write(
+            WithBase::of::<Self>().at(0),
+            regs::NV_PFALCON_FALCON_INTR_RETRIGGER::zeroed().with_trigger(true),
+        );
+    }
 }
 
 impl<'a> Falcon<'a, Gsp> {
diff --git a/drivers/gpu/nova-core/falcon/hal.rs b/drivers/gpu/nova-core/falcon/hal.rs
index 7e532889a1f4..f0828b32aebb 100644
--- a/drivers/gpu/nova-core/falcon/hal.rs
+++ b/drivers/gpu/nova-core/falcon/hal.rs
@@ -72,6 +72,14 @@ fn signature_reg_fuse_version(
     fn load_method(&self) -> LoadMethod;
 }
 
+/// Returns whether `chipset`'s falcons implement `NV_PFALCON_FALCON_INTR_RETRIGGER`.
+///
+/// Turing falcons do not. Ampere and later do, including GA100, whose falcon otherwise uses the
+/// Turing HAL, so this is keyed on the architecture rather than provided through [`FalconHal`].
+pub(crate) fn has_intr_retrigger(chipset: Chipset) -> bool {
+    !matches!(chipset.arch(), Architecture::Turing)
+}
+
 /// Returns a boxed falcon HAL adequate for `chipset`.
 ///
 /// We use a heap-allocated trait object instead of a statically defined one because the
diff --git a/drivers/gpu/nova-core/irq/gsp.rs b/drivers/gpu/nova-core/irq/gsp.rs
index 1fce315410f3..ecd716b92d4e 100644
--- a/drivers/gpu/nova-core/irq/gsp.rs
+++ b/drivers/gpu/nova-core/irq/gsp.rs
@@ -54,18 +54,17 @@
 
 /// Clears the interrupt state that GSP boot left behind.
 ///
-/// Disables every vector in every implemented leaf, clears the falcon's SWGEN0 latch, clears the
-/// tree's pending bits, and rearms PCI interrupt delivery. On return no vector is enabled, so the
-/// tree delivers nothing.
+/// Disables every vector in every implemented leaf, clears the tree's pending bits, clears the
+/// falcon's SWGEN0 latch, and rearms PCI interrupt delivery. On return no vector is enabled, so
+/// the tree delivers nothing.
 pub(crate) fn quiesce(bar: Bar0<'_>, chipset: Chipset, irq_type: pci::IrqType) {
     let tree = Tree::new(chipset, irq_type, GSP_SUBTREE);
     tree.disable_all_leaves(bar);
-    // GSP boot consumes its notifications by polling the queue, which leaves SWGEN0 latched.
-    // Clear it before the tree drain below, so the drain clears the tree state the clear sets.
-    // Messages already posted raise no interrupt of their own, and the caller's queue drain
-    // covers them.
-    GspFalcon::clear_swgen0_intr(bar);
     tree.drain(bar);
+    // GSP boot consumes its notifications by polling the queue, which leaves SWGEN0 latched, and
+    // the GSP drives no new signal while it is set. Clear it after the tree drain, which erases
+    // every leaf bit and would erase the one a message posted since the clear had set.
+    GspFalcon::clear_swgen0_intr(bar);
     // The `TOP_EN` cycle in `drain` is the rearm for the two enable-cycle methods, but pre-Hopper
     // MSI rearms through a configuration-space write instead. An interrupt delivered before probe
     // leaves delivery un-armed on that path, with no handler to have rearmed it.
@@ -94,6 +93,8 @@ pub(crate) struct GspInterrupt<'a> {
     cmdq: Arc<Cmdq>,
     /// The GIN interrupt tree for this chipset.
     tree: Tree,
+    /// Chipset, for the falcon retrigger, which Turing does not implement.
+    chipset: Chipset,
     /// Device, for logging from interrupt context without taking the command-queue lock.
     dev: ARef<device::Device>,
 }
@@ -112,14 +113,15 @@ pub(crate) fn new(
             bar,
             cmdq,
             tree: Tree::new(chipset, irq_type, GSP_SUBTREE),
+            chipset,
             dev,
         }? Error)
     }
 }
 
 impl irq::ThreadedHandler for GspInterrupt<'_> {
-    /// Top half: clears the GIN leaf, takes the falcon SWGEN0 latch, and rearms PCI interrupt
-    /// delivery.
+    /// Top half: clears the GIN leaf, takes every cause the falcon reports, and rearms PCI
+    /// interrupt delivery.
     fn handle(&self) -> irq::ThreadedIrqReturn {
         let bar = self.bar;
 
@@ -138,27 +140,40 @@ fn handle(&self) -> irq::ThreadedIrqReturn {
         }
         leaf.clear_vectors(bar, GSP_BIT);
 
-        // SWGEN0 is the message-queue notification, so wake the IRQ thread to drain it.
         let status = GspFalcon::take_swgen0_intr(bar);
-        let ret = if status.swgen0() {
-            irq::ThreadedIrqReturn::WakeThread
-        } else {
-            // The tree routes every falcon cause to this vector, so something other than a posted
-            // message fired it, for example a HALT from a GSP crash. There is no recovery path for
-            // those causes, so report the status rather than discarding it.
+
+        // Every cause the falcon reports leaves the falcon's enabled set on this invocation. A
+        // cause left latched holds that set non-empty, and the falcon signals the tree only on a
+        // transition of the set, so no later SWGEN0 would signal at all.
+        let unserviceable = status.with_swgen0(false);
+        if unserviceable.into_raw() != 0 {
+            // The tree routes every falcon cause to this vector, so a cause other than a posted
+            // message also arrives here, for example a HALT from a GSP crash. nova-core has no
+            // recovery path for those, so report the status rather than discarding it, then mask
+            // the cause.
             dev_err!(
                 &self.dev,
-                "GSP interrupt with no SWGEN0, falcon IRQSTAT {:#x}\n",
+                "unserviceable GSP falcon interrupt, IRQSTAT {:#x}\n",
                 status.into_raw()
             );
-            irq::ThreadedIrqReturn::Handled
-        };
+            GspFalcon::mask_and_clear_intr(bar, unserviceable);
+        }
+
+        // The leaf clear above consumed the tree's record of this interrupt, and the falcon signals
+        // the tree only on a transition of its enabled causes, so a cause that arrived while this
+        // handler ran would never reach the CPU. Re-emit to supply that transition.
+        GspFalcon::retrigger_intr(bar, self.chipset);
 
         // Delivery resumes only after this, so it must happen on every path that services the
         // vector, including the fault path above.
         self.tree.rearm_pci_irq(bar, GSP_SUBTREE);
 
-        ret
+        // SWGEN0 is the message-queue notification, so wake the IRQ thread to drain it.
+        if status.swgen0() {
+            irq::ThreadedIrqReturn::WakeThread
+        } else {
+            irq::ThreadedIrqReturn::Handled
+        }
     }
 
     /// IRQ thread: drains and dispatches the GSP-to-CPU message queue.
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 2a0489472a66..01fde2c5e5a6 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -200,6 +200,15 @@ pub(crate) fn usable_fb_size(self) -> u64 {
         6:6     swgen0 => bool;
     }
 
+    /// Masks interrupt causes at the falcon, one bit per cause, in the layout of
+    /// `NV_PFALCON_FALCON_IRQSTAT`.
+    ///
+    /// A masked cause is excluded from the enabled set the falcon signals on, so it cannot be
+    /// raised again by `NV_PFALCON_FALCON_INTR_RETRIGGER`.
+    pub(crate) NV_PFALCON_FALCON_IRQMCLR(u32) @ PFalconBase + 0x00000014 {
+        31:0    value => u32;
+    }
+
     pub(crate) NV_PFALCON_FALCON_MAILBOX0(u32) @ PFalconBase + 0x00000040 {
         31:0    value => u32;
     }
@@ -327,6 +336,17 @@ pub(crate) fn usable_fb_size(self) -> u64 {
         0:0     reset => bool;
     }
 
+    /// Re-emits the falcon's enabled interrupt causes into the interrupt tree.
+    ///
+    /// Write-only. A falcon signals the tree on a transition of its enabled causes, so a handler
+    /// that cleared the tree leaf while a cause was still latched has left no transition behind,
+    /// and this write supplies one. Turing falcons do not implement this register.
+    ///
+    /// OpenRM declares two elements and uses only the first.
+    pub(crate) NV_PFALCON_FALCON_INTR_RETRIGGER(u32)[2] @ PFalconBase + 0x000003e8 {
+        0:0     trigger => bool;
+    }
+
     pub(crate) NV_PFALCON_FBIF_TRANSCFG(u32)[8] @ PFalconBase + 0x00000600 {
         2:2     mem_type => FalconFbifMemType;
         1:0     target ?=> FalconFbifTarget;
-- 
2.55.0