[PATCH v3 06/14] gpu: nova-core: add the GIN interrupt tree and allocate its vectors

[John Hubbard <jhubbard@nvidia.com>]

From: Joel Fernandes <joelagnelf@nvidia.com>

Servicing a GIN leaf has a required order: read its pending bits, then
clear them. Clearing a leaf before reading it discards every vector
latched in it, and nothing reports the loss.

The driver must also allocate a PCI vector for every subtree it enables
at TOP, and register a handler on that vector. MSI-X gives each subtree
its own table entry. Linux masks every entry the driver did not
allocate. An enabled subtree with no entry of its own raises interrupts
that never arrive, and its leaf and TOP bits stay pending and enabled.
MSI instead has one message that the whole tree raises, so a single
entry serves every subtree.

Add an API for one PCIe function's CPU interrupt tree, in which reading
a leaf yields the handle that clears it, and building a tree fails if it
names a subtree the architecture does not implement. Size the vector
allocation to the serviced subtrees, requesting MSI-X entries up to the
highest serviced subtree and falling back to a single MSI rather than a
shared INTx line.

Reviewed-by: Will Pierce <wpierce@nvidia.com>
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
[jhubbard: name the module interrupt_tree with a Tree type that owns the
 BAR mapping, use the canonical NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_*
 register names, express vectors, leaves and subtrees as newtypes, let
 the read of a leaf produce the handle that clears it, add the enable
 guards, take the leaf count and the rearm method from the interrupt
 HAL, and read every implemented leaf in drain() rather than descending
 from the TOP registers, which cannot see a vector that latched while
 disabled]
Signed-off-by: John Hubbard <jhubbard@nvidia.com>
---
 drivers/gpu/nova-core/irq.rs                |  95 +++++++
 drivers/gpu/nova-core/irq/hal.rs            |   8 +-
 drivers/gpu/nova-core/irq/interrupt_tree.rs | 291 +++++++++++++++++++-
 3 files changed, 386 insertions(+), 8 deletions(-)

diff --git a/drivers/gpu/nova-core/irq.rs b/drivers/gpu/nova-core/irq.rs
index d21dee1b89a0..f6ba883d72c5 100644
--- a/drivers/gpu/nova-core/irq.rs
+++ b/drivers/gpu/nova-core/irq.rs
@@ -12,6 +12,23 @@
 mod interrupt_tree;
 mod regs;
 
+use kernel::{
+    device::Bound,
+    irq,
+    pci::{
+        self,
+        IrqType, //
+    },
+    prelude::*, //
+};
+
+use crate::num;
+
+use interrupt_tree::{
+    Subtree,
+    SubtreeSet, //
+};
+
 /// The message-signaled interrupt type a vector allocation obtained.
 ///
 /// nova-core allocates MSI-X or MSI and nothing else, so the level-triggered INTx that
@@ -24,3 +41,81 @@ pub(crate) enum MsiType {
     /// One table entry per subtree.
     MsiX,
 }
+
+/// The PCI interrupt vector that delivers each serviced subtree.
+///
+/// MSI-X raises a separate table entry per subtree, so subtree `N` arrives on entry `N`. MSI has a
+/// single message that every subtree raises, so all of them arrive on the one allocated entry.
+pub(crate) struct SubtreeVectors<'a> {
+    vectors: pci::IrqVectorRegistration<'a>,
+    /// Every subtree nova-core services.
+    serviced: SubtreeSet,
+    /// The type [`alloc_vectors`] obtained, which fixes both the entry each subtree raises and the
+    /// rearm write its handler owes.
+    msi_type: MsiType,
+}
+
+impl SubtreeVectors<'_> {
+    /// Returns the interrupt type these vectors were allocated as.
+    pub(crate) fn msi_type(&self) -> MsiType {
+        self.msi_type
+    }
+
+    /// Returns an [`irq::IrqRequest`] for the vector that delivers `subtree`.
+    ///
+    /// MSI-X gives subtree `N` its own table entry `N`. MSI raises its one message from every
+    /// subtree, and nova-core allocates a single entry for it.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `subtree` is not one nova-core services.
+    pub(crate) fn request_for(&self, subtree: Subtree) -> Result<irq::IrqRequest<'_>> {
+        if !self.serviced.contains(subtree) {
+            return Err(EINVAL);
+        }
+
+        let entry = match self.msi_type {
+            MsiType::MsiX => num::u32_as_usize(subtree.index()),
+            MsiType::Msi => 0,
+        };
+
+        self.vectors.index(entry).map(Into::into)
+    }
+}
+
+/// Allocates the interrupt vectors that the subtrees in `serviced` require.
+///
+/// Every subtree nova-core enables at `TOP` must have an allocated vector with a registered
+/// handler, or the interrupts it raises are lost. Linux masks every MSI-X entry a driver did not
+/// allocate, so the MSI-X request covers every entry up to the highest serviced subtree. A part
+/// whose MSI-X table is smaller than that falls back to a single MSI, which serves the whole tree.
+///
+/// # Errors
+///
+/// `EINVAL` if `serviced` is empty. The error from the MSI request if neither type can be
+/// allocated.
+pub(crate) fn alloc_vectors(
+    pdev: &pci::Device<Bound>,
+    serviced: SubtreeSet,
+) -> Result<SubtreeVectors<'_>> {
+    if serviced.is_empty() {
+        return Err(EINVAL);
+    }
+
+    // One entry per subtree up to and including the highest serviced one.
+    let entries = serviced.span();
+
+    let (vectors, msi_type) = pdev
+        .alloc_irq_vectors(entries, entries, IrqType::MsiX.into())
+        .map(|vectors| (vectors, MsiType::MsiX))
+        .or_else(|_| {
+            pdev.alloc_irq_vectors(1, 1, IrqType::Msi.into())
+                .map(|vectors| (vectors, MsiType::Msi))
+        })?;
+
+    Ok(SubtreeVectors {
+        vectors,
+        serviced,
+        msi_type,
+    })
+}
diff --git a/drivers/gpu/nova-core/irq/hal.rs b/drivers/gpu/nova-core/irq/hal.rs
index 1ea677e37e56..07604458dbbb 100644
--- a/drivers/gpu/nova-core/irq/hal.rs
+++ b/drivers/gpu/nova-core/irq/hal.rs
@@ -25,7 +25,7 @@
         Subtree,
         SubtreeSet, //
     },
-    regs,
+    regs::*,
     MsiType, //
 };
 
@@ -63,7 +63,7 @@ pub(super) fn rearm(self, bar: Bar0<'_>, serviced: SubtreeSet, subtree: Subtree)
         let subtrees = match self {
             // The written value is ignored, so any write rearms delivery.
             Self::ConfigMirrorEoi => {
-                bar.write(regs::NV_XVE_CYA_2, 0u32.into());
+                bar.write(NV_XVE_CYA_2, 0u32.into());
                 return;
             }
             Self::TopEnableCycleServiced => serviced,
@@ -71,10 +71,10 @@ pub(super) fn rearm(self, bar: Bar0<'_>, serviced: SubtreeSet, subtree: Subtree)
         };
 
         bar.write_reg(
-            regs::NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(subtrees),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(subtrees),
         );
         bar.write_reg(
-            regs::NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(subtrees),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(subtrees),
         );
     }
 }
diff --git a/drivers/gpu/nova-core/irq/interrupt_tree.rs b/drivers/gpu/nova-core/irq/interrupt_tree.rs
index 5aa447cf0ec4..0b4dc2fc8ea8 100644
--- a/drivers/gpu/nova-core/irq/interrupt_tree.rs
+++ b/drivers/gpu/nova-core/irq/interrupt_tree.rs
@@ -1,18 +1,42 @@
 // SPDX-License-Identifier: GPL-2.0
 // SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 
-//! Vector addressing in the GIN CPU interrupt tree.
+//! The GIN CPU interrupt tree for one PCIe function.
 //!
 //! A vector's number fixes where it latches: leaf `vector / 32` at bit `vector % 32`, and that
 //! leaf belongs to subtree `vector / 64`. The types here keep those three views apart, so a leaf
 //! index, a set of vectors within one leaf, and a `TOP` bit cannot stand in for one another.
+//!
+//! Servicing a leaf has a required order: read its pending bits, then clear them. Clearing a leaf
+//! before reading it discards every vector latched in it, and nothing reports the loss. Only
+//! [`Tree::read_pending`] produces a [`LeafPending`], and only a [`LeafPending`] can clear, so the
+//! wrong order does not compile.
+//!
+//! Serializing access to the tree is the caller's responsibility.
 
 use kernel::{
+    io::{
+        register::Array,
+        Io, //
+    },
     num::Bounded,
     prelude::*, //
 };
 
-use crate::num;
+use crate::{
+    driver::Bar0,
+    gpu::Chipset,
+    num, //
+};
+
+use super::{
+    hal::{
+        cpu_interrupt_hal,
+        PciIrqRearmMethod, //
+    },
+    regs::*,
+    MsiType, //
+};
 
 /// Number of bits a leaf index occupies, covering the `0..16` leaf register arrays.
 const LEAF_INDEX_BITS: u32 = 4;
@@ -113,7 +137,7 @@ pub(super) const fn contains(self, other: Self) -> bool {
 ///
 /// Exactly one bit is set.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct Subtree(u32);
+pub(crate) struct Subtree(u32);
 
 impl Subtree {
     /// Returns this subtree's index within the tree.
@@ -131,7 +155,7 @@ pub(super) const fn into_raw(self) -> u32 {
 
 /// Set of subtrees, one bit per subtree, in the layout the `TOP` enable registers take.
 #[derive(Clone, Copy, Debug, Eq, PartialEq)]
-pub(super) struct SubtreeSet(u32);
+pub(crate) struct SubtreeSet(u32);
 
 impl SubtreeSet {
     /// Returns whether `subtree` belongs to this set.
@@ -240,3 +264,262 @@ fn from(vector: GinVector) -> Self {
         vector.0.extend()
     }
 }
+
+/// Clears the enables of the vectors set in `vectors` for `leaf` (`LEAF_EN_CLEAR`).
+///
+/// Shared by [`Tree::disable_leaf`] and by [`LeafEnableGuard`]'s [`Drop`], which has no tree to
+/// reach through.
+fn clear_leaf_enables(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+    bar.write(
+        NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::at(*leaf),
+        NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_CLEAR::zeroed().with_vectors(vectors),
+    );
+}
+
+/// Clears the `TOP` enables of every subtree in `serviced` (`TOP_EN_CLEAR`).
+fn clear_top_enables(bar: Bar0<'_>, serviced: SubtreeSet) {
+    bar.write_reg(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_CLEAR::zeroed().with_subtrees(serviced));
+}
+
+/// Clears the pending vectors set in `vectors` for `leaf` (write-1-to-clear).
+fn clear_leaf_pending(bar: Bar0<'_>, leaf: LeafIndex, vectors: LeafMask) {
+    if !vectors.is_empty() {
+        bar.write(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::zeroed().with_vectors(vectors),
+        );
+    }
+}
+
+/// Returns every leaf a tree of `leaves` leaves implements.
+fn implemented_leaves(leaves: LeafCount) -> impl Iterator<Item = LeafIndex> {
+    (0..leaves.into_raw()).filter_map(LeafIndex::try_new)
+}
+
+/// The GIN CPU interrupt tree for a single PCIe function.
+pub(super) struct Tree<'a> {
+    /// Borrowed BAR0, through which every tree register is reached.
+    bar: Bar0<'a>,
+    /// Number of leaves this tree implements.
+    leaves: LeafCount,
+    /// The subtrees this tree enables and services.
+    serviced: SubtreeSet,
+    /// Method that rearms PCI interrupt delivery.
+    rearm: PciIrqRearmMethod,
+}
+
+impl<'a> Tree<'a> {
+    /// Creates a `Tree` for `chipset` covering `serviced`, with the rearm method that `msi_type`
+    /// requires.
+    ///
+    /// Each serviced subtree must have an allocated PCI vector and a registered handler, which
+    /// [`super::alloc_vectors`] sizes the allocation for.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `serviced` names a subtree this architecture does not implement. Such a subtree
+    /// has no `TOP` bit, so nothing would deliver the vectors behind it.
+    pub(super) fn new(
+        bar: Bar0<'a>,
+        chipset: Chipset,
+        msi_type: MsiType,
+        serviced: SubtreeSet,
+    ) -> Result<Self> {
+        let hal = cpu_interrupt_hal(chipset);
+        let leaves = hal.leaf_count();
+
+        if serviced.intersection(leaves.subtree_set()) != serviced {
+            return Err(EINVAL);
+        }
+
+        Ok(Self {
+            bar,
+            leaves,
+            serviced,
+            rearm: hal.pci_irq_rearm_method(msi_type),
+        })
+    }
+
+    /// Rearms PCI interrupt delivery to the CPU after servicing `subtree`, the one subtree the
+    /// calling handler serves.
+    ///
+    /// A handler must call this before it returns, or it receives no further interrupts.
+    pub(super) fn rearm_pci_irq(&self, subtree: Subtree) {
+        self.rearm.rearm(self.bar, self.serviced, subtree);
+    }
+
+    /// Enables this tree's serviced subtrees (`TOP_EN_SET`).
+    pub(super) fn enable_top(&self) {
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_TOP_EN_SET::zeroed().with_subtrees(self.serviced),
+        );
+    }
+
+    /// Disables this tree's serviced subtrees (`TOP_EN_CLEAR`).
+    pub(super) fn disable_top(&self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+
+    /// Enables this tree's serviced subtrees until the returned guard drops.
+    pub(super) fn enable_top_guarded(&self) -> TopEnableGuard<'a> {
+        self.enable_top();
+
+        TopEnableGuard {
+            bar: self.bar,
+            serviced: self.serviced,
+        }
+    }
+
+    /// Enables the vectors set in `vectors` for `leaf` (`LEAF_EN_SET`).
+    ///
+    /// This is the per-vector counterpart of [`Self::enable_top`], which enables whole subtrees.
+    pub(super) fn enable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        self.bar.write(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::at(*leaf),
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_EN_SET::zeroed().with_vectors(vectors),
+        );
+    }
+
+    /// Disables the vectors set in `vectors` for `leaf` (`LEAF_EN_CLEAR`).
+    pub(super) fn disable_leaf(&self, leaf: LeafIndex, vectors: LeafMask) {
+        clear_leaf_enables(self.bar, leaf, vectors);
+    }
+
+    /// Enables `vectors` for `leaf` until the returned guard drops.
+    pub(super) fn enable_leaf_guarded(
+        &self,
+        leaf: LeafIndex,
+        vectors: LeafMask,
+    ) -> LeafEnableGuard<'a> {
+        self.enable_leaf(leaf, vectors);
+
+        LeafEnableGuard {
+            bar: self.bar,
+            leaf,
+            vectors,
+        }
+    }
+
+    /// Reads the vectors pending in `leaf`.
+    pub(super) fn read_pending(&self, leaf: LeafIndex) -> LeafPending<'a> {
+        let pending = self
+            .bar
+            .read(NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF::at(*leaf))
+            .vectors();
+
+        LeafPending {
+            bar: self.bar,
+            leaf,
+            pending,
+        }
+    }
+
+    /// Injects a software interrupt for `vector` via the trigger register.
+    ///
+    /// # Errors
+    ///
+    /// `EINVAL` if `vector` lies outside this tree.
+    // Only the interrupt self-test injects a software interrupt.
+    #[expect(dead_code)]
+    pub(super) fn trigger(&self, vector: GinVector) -> Result {
+        vector.validate(self.leaves)?;
+        self.bar.write_reg(
+            NV_VIRTUAL_FUNCTION_PRIV_CPU_INTR_LEAF_TRIGGER::zeroed().with_vector(vector),
+        );
+
+        Ok(())
+    }
+
+    /// Disables every vector in every implemented leaf (`LEAF_EN_CLEAR`).
+    ///
+    /// Boot, or a driver that ran before this one, can leave leaf enables set for vectors
+    /// nova-core does not service, and such a vector delivers to nova-core's handler once its
+    /// subtree is enabled.
+    ///
+    /// This clears enables outside the subtrees nova-core services, so it is a probe-time
+    /// operation only.
+    pub(super) fn disable_all_leaves(&self) {
+        for leaf in implemented_leaves(self.leaves) {
+            self.disable_leaf(leaf, LeafMask::all());
+        }
+    }
+
+    /// Clears every pending bit in every implemented leaf.
+    ///
+    /// Disables this tree's serviced subtrees at `TOP` for the walk and leaves them disabled, so a
+    /// caller that wants delivery enables them itself once it is ready to receive. The leaves
+    /// cleared reach subtrees the driver does not service, and the `TOP_EN` write does not.
+    ///
+    /// Call `drain()` only during probe. It must not run concurrently with an interrupt handler.
+    pub(super) fn drain(&self) {
+        self.disable_top();
+
+        // `TOP` summarizes enabled leaf bits, so a vector that latched while it was disabled does
+        // not appear there.
+        for leaf in implemented_leaves(self.leaves) {
+            let pending = self.read_pending(leaf);
+            if !pending.vectors().is_empty() {
+                pending.clear();
+            }
+        }
+    }
+}
+
+/// The vectors read pending from one leaf.
+///
+/// Holding one is the proof that the leaf was read, which is what [`Self::clear`] and
+/// [`Self::clear_vectors`] require.
+pub(super) struct LeafPending<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    pending: LeafMask,
+}
+
+impl LeafPending<'_> {
+    /// Returns the vectors that were pending.
+    pub(super) fn vectors(&self) -> LeafMask {
+        self.pending
+    }
+
+    /// Clears every vector that was pending, by writing its bits back (write-1-to-clear).
+    pub(super) fn clear(&self) {
+        self.clear_vectors(self.pending);
+    }
+
+    /// Clears the vectors set in `vectors` (write-1-to-clear), leaving every other pending bit
+    /// set.
+    ///
+    /// A handler that services one vector uses this rather than [`Self::clear`], which clears
+    /// every vector the leaf had pending.
+    pub(super) fn clear_vectors(&self, vectors: LeafMask) {
+        clear_leaf_pending(self.bar, self.leaf, vectors);
+    }
+}
+
+/// Keeps a leaf's vectors enabled for as long as it is held.
+///
+/// Dropping it disables the same vectors, so an error path cannot leave a source enabled with no
+/// handler behind it.
+pub(super) struct LeafEnableGuard<'a> {
+    bar: Bar0<'a>,
+    leaf: LeafIndex,
+    vectors: LeafMask,
+}
+
+impl Drop for LeafEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_leaf_enables(self.bar, self.leaf, self.vectors);
+    }
+}
+
+/// Keeps a tree's serviced subtrees enabled at `TOP` for as long as it is held.
+pub(super) struct TopEnableGuard<'a> {
+    bar: Bar0<'a>,
+    serviced: SubtreeSet,
+}
+
+impl Drop for TopEnableGuard<'_> {
+    fn drop(&mut self) {
+        clear_top_enables(self.bar, self.serviced);
+    }
+}
-- 
2.55.0