[PATCH v3 0/7] Documentation for nova-core

[PATCH v3 0/7] Documentation for nova-core

[Alexandre Courbot]

This series adds some documentation that was relevant to the FWSEC-FRTS
[1] series, but wasn't sent alongside it as it was worked on in
parallel.

It notably introduces a lot of HTMLdocs that adds key explanations to
understand and work on the GSP boot process.

[1] https://lore.kernel.org/all/DB0I8WAH970B.25D3S59AYF85P@nvidia.com/

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Changes in v3:
- Rebase on top of latest nova-next.
- Reorganize patches a bit and use standard prefixes.
- Convert diagrams to only use ASCII characters (thanks Bagas!).
- Move sysmembar documentation to the new `SysmemFlush` type.
- Reword things a bit here and there.
- Reorganize order of `index.rst` for a more natural flow.
- Link to v2: https://lore.kernel.org/rust-for-linux/20250503040802.1411285-1-joelagnelf@nvidia.com/

---
Joel Fernandes (7):
      gpu: nova-core: Add code comments related to devinit
      gpu: nova-core: Clarify sysmembar operations
      gpu: nova-core: Clarify falcon code
      Documentation: gpu: nova-core: Document vbios layout
      Documentation: gpu: nova-core: Document devinit process
      Documentation: gpu: nova-core: Document fwsec operation and layout
      Documentation: gpu: nova-core: Document basics of the Falcon

 Documentation/gpu/nova/core/devinit.rst |  61 +++++++++++
 Documentation/gpu/nova/core/falcon.rst  | 158 +++++++++++++++++++++++++++
 Documentation/gpu/nova/core/fwsec.rst   | 182 ++++++++++++++++++++++++++++++++
 Documentation/gpu/nova/core/vbios.rst   | 180 +++++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst        |   4 +
 drivers/gpu/nova-core/falcon.rs         |  29 +++--
 drivers/gpu/nova-core/fb.rs             |  10 ++
 drivers/gpu/nova-core/gfw.rs            |  39 ++++++-
 drivers/gpu/nova-core/gpu.rs            |   3 +-
 drivers/gpu/nova-core/regs.rs           |  18 +++-
 10 files changed, 669 insertions(+), 15 deletions(-)
---
base-commit: 4092e1b41202ff39aad75a40a03ac1d318443670
change-id: 20250702-nova-docs-b9900d0505b5

Best regards,
-- 
Alexandre Courbot <acourbot@nvidia.com>

[PATCH v3 1/7] gpu: nova-core: Add code comments related to devinit

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

Add several code comments to reduce acronym soup and explain how devinit
magic and bootflow works before driver loads. These are essential for
debug and development of the nova driver.

[acourbot@nvidia.com: reformat and reword a couple of sentences]

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 drivers/gpu/nova-core/gfw.rs  | 39 +++++++++++++++++++++++++++++++++++----
 drivers/gpu/nova-core/regs.rs | 16 ++++++++++++++--
 2 files changed, 49 insertions(+), 6 deletions(-)

diff --git a/drivers/gpu/nova-core/gfw.rs b/drivers/gpu/nova-core/gfw.rs
index d5b68e02d405750b18d634d772f15f413453e80d..8ac1ed18719926493369c2aae9a59b2b55fa2b12 100644
--- a/drivers/gpu/nova-core/gfw.rs
+++ b/drivers/gpu/nova-core/gfw.rs
@@ -1,10 +1,22 @@
 // SPDX-License-Identifier: GPL-2.0
 
-//! GPU Firmware (GFW) support.
+//! GPU Firmware (`GFW`) support, a.k.a `devinit`.
 //!
 //! Upon reset, the GPU runs some firmware code from the BIOS to setup its core parameters. Most of
 //! the GPU is considered unusable until this step is completed, so we must wait on it before
 //! performing driver initialization.
+//!
+//! A clarification about devinit terminology: devinit is a sequence of register read/writes after
+//! reset that performs tasks such as:
+//! 1. Programming VRAM memory controller timings.
+//! 2. Power sequencing.
+//! 3. Clock and PLL configuration.
+//! 4. Thermal management.
+//!
+//! devinit itself is a 'script' which is interpreted by an interpreter program typically running
+//! on the PMU microcontroller.
+//!
+//! Note that the devinit sequence also needs to run during suspend/resume.
 
 use kernel::bindings;
 use kernel::prelude::*;
@@ -14,13 +26,32 @@
 use crate::regs;
 use crate::util;
 
-/// Wait until `GFW` (GPU Firmware) completes, or a 4 seconds timeout elapses.
+/// Wait for the `GFW` (GPU firmware) boot completion signal (`GFW_BOOT`), or a 4 seconds timeout.
+///
+/// Upon GPU reset, several microcontrollers (such as PMU, SEC2, GSP etc) run some firmware code to
+/// setup its core parameters. Most of the GPU is considered unusable until this step is completed,
+/// so it must be waited on very early during driver initialization.
+///
+/// The `GFW` code includes several components that need to execute before the driver loads. These
+/// components are located in the VBIOS ROM and executed in a sequence on these different
+/// microcontrollers. The devinit sequence typically runs on the PMU, and the FWSEC runs on the
+/// GSP.
+///
+/// This function waits for a signal indicating that core initialization is complete. Before this
+/// signal is received, little can be done with the GPU. This signal is set by the FWSEC running on
+/// the GSP in Heavy-secured mode.
 pub(crate) fn wait_gfw_boot_completion(bar: &Bar0) -> Result {
+    // Before accessing the completion status in `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05`, we must
+    // first check `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK`. This is because
+    // `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05` becomes accessible only after the secure firmware
+    // (FWSEC) lowers the privilege level to allow CPU (LS/Light-secured) access. We can only
+    // safely read the status register from CPU (LS/Light-secured) once the mask indicates
+    // that the privilege level has been lowered.
+    //
     // TIMEOUT: arbitrarily large value. GFW starts running immediately after the GPU is put out of
     // reset, and should complete in less time than that.
     util::wait_on(Delta::from_secs(4), || {
-        // Check that FWSEC has lowered its protection level before reading the GFW_BOOT
-        // status.
+        // Check that FWSEC has lowered its protection level before reading the GFW_BOOT status.
         let gfw_booted = regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK::read(bar)
             .read_protection_level0()
             && regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT::read(bar).completed();
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index e8b8aabce3f36abe6a7fba3e11f677e36baa3897..ce08fab8fa96fcacf6947512ebdf4975ebd8772c 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -104,9 +104,21 @@ pub(crate) fn higher_bound(self) -> u64 {
     }
 }
 
-/* PGC6 */
+/*
+ * PGC6 register space.
+ *
+ * `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except
+ * for power rails needed to keep self-refresh working and important registers and hardware
+ * blocks).
+ *
+ * These scratch registers remain powered on even in a low-power state and have a designated group
+ * number.
+ */
 
-register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128 {
+// Privilege level mask register. It dictates whether the host CPU has privilege to access the
+// `PGC6_AON_SECURE_SCRATCH_GROUP_05` register (which it needs to read GFW_BOOT).
+register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128,
+          "Privilege level mask register" {
     0:0     read_protection_level0 as bool, "Set after FWSEC lowers its protection level";
 });
 

-- 
2.50.0

[PATCH v3 2/7] gpu: nova-core: Clarify sysmembar operations

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

sysmembar is a critical operation that the GSP falcon needs to perform
in the reset sequence. Add some code comments to clarify.

[acourbot@nvdidia.com: move relevant documentation to SysmemFlush type]

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 drivers/gpu/nova-core/fb.rs   | 10 ++++++++++
 drivers/gpu/nova-core/gpu.rs  |  3 +--
 drivers/gpu/nova-core/regs.rs |  2 ++
 3 files changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/gpu/nova-core/fb.rs b/drivers/gpu/nova-core/fb.rs
index 172b4a12ba2afc05860cc004fd1f0154402f467a..02c44c79cef500462bab10ea18678749f72404ee 100644
--- a/drivers/gpu/nova-core/fb.rs
+++ b/drivers/gpu/nova-core/fb.rs
@@ -17,6 +17,16 @@
 /// Type holding the sysmem flush memory page, a page of memory to be written into the
 /// `NV_PFB_NISO_FLUSH_SYSMEM_ADDR*` registers and used to maintain memory coherency.
 ///
+/// A system memory page is required for `sysmembar`, which is a GPU-initiated hardware
+/// memory-barrier operation that flushes all pending GPU-side memory writes that were done through
+/// PCIE to system memory. It is required for falcons to be reset as the reset operation involves a
+/// reset handshake. When the falcon acknowledges a reset, it writes into system memory. To ensure
+/// this write is visible to the host and prevent driver timeouts, the falcon must perform a
+/// sysmembar operation to flush its writes.
+///
+/// Because of this, the sysmem flush memory page must be registered as early as possible during
+/// driver initialization, and before any falcon is reset.
+//
 /// Users are responsible for manually calling [`Self::unregister`] before dropping this object,
 /// otherwise the GPU might still use it even after it has been freed.
 pub(crate) struct SysmemFlush {
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 8e32af16b669ca773e63e184d34c3e0427bc9b76..72d40b0124f0c1a2a381484172c289af523511df 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -170,7 +170,7 @@ pub(crate) struct Gpu {
     bar: Devres<Bar0>,
     fw: Firmware,
     /// System memory page required for flushing all pending GPU-side memory writes done through
-    /// PCIE into system memory.
+    /// PCIE into system memory, via sysmembar (A GPU-initiated HW memory-barrier operation).
     sysmem_flush: SysmemFlush,
 }
 
@@ -283,7 +283,6 @@ pub(crate) fn new(
         gfw::wait_gfw_boot_completion(bar)
             .inspect_err(|_| dev_err!(pdev.as_ref(), "GFW boot did not complete"))?;
 
-        // System memory page required for sysmembar to properly flush into system memory.
         let sysmem_flush = SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?;
 
         let gsp_falcon = Falcon::<Gsp>::new(
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index ce08fab8fa96fcacf6947512ebdf4975ebd8772c..e94c1d2190f352e9316937dd3165cc2a390e0ed2 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -51,6 +51,8 @@ pub(crate) fn chipset(self) -> Result<Chipset> {
 
 /* PFB */
 
+// These two registers together hold the physical system memory address that is used by the GPU for
+// perform sysmembar operation (see [`crate::fb::SysmemFlush`]).
 register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR @ 0x00100c10 {
     31:0    adr_39_08 as u32;
 });

-- 
2.50.0

[PATCH v3 3/7] gpu: nova-core: Clarify falcon code

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

Add documentation strings, comments and AES mode for completeness
to the Falcon signatures.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 drivers/gpu/nova-core/falcon.rs | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/drivers/gpu/nova-core/falcon.rs b/drivers/gpu/nova-core/falcon.rs
index c2c6f9eb380ab390befe5af1b3c5df260ccd7595..6ded4a37a08deb71db668f389efad84ff17b317b 100644
--- a/drivers/gpu/nova-core/falcon.rs
+++ b/drivers/gpu/nova-core/falcon.rs
@@ -105,14 +105,22 @@ fn try_from(value: u8) -> Result<Self> {
 /// register.
 #[repr(u8)]
 #[derive(Debug, Default, Copy, Clone)]
+/// Security mode of the Falcon microprocessor.
+/// See falcon.rst for more details.
 pub(crate) enum FalconSecurityModel {
     /// Non-Secure: runs unsigned code without privileges.
     #[default]
     None = 0,
-    /// Low-Secure: runs code with some privileges. Can only be entered from `Heavy` mode, which
-    /// will typically validate the LS code through some signature.
+    /// Light-Secured (LS): Runs signed code with some privileges.
+    /// Entry into this mode is only possible from 'Heavy-secure' mode, which verifies the code's
+    /// signature.
+    ///
+    /// Also known as Low-Secure, Privilege Level 2 or PL2.
     Light = 2,
-    /// High-Secure: runs signed code with full privileges. Signature is validated by boot ROM.
+    /// Heavy-Secured (HS): Runs signed code with full privileges.
+    /// The code's signature is verified by the Falcon Boot ROM (BROM).
+    ///
+    /// Also known as High-Secure, Privilege Level 3 or PL3.
     Heavy = 3,
 }
 impl_from_enum_to_u32!(FalconSecurityModel);
@@ -136,10 +144,13 @@ fn try_from(value: u8) -> Result<Self> {
 }
 
 /// Signing algorithm for a given firmware, used in the [`crate::regs::NV_PFALCON2_FALCON_MOD_SEL`]
-/// register.
+/// register. It is passed to the Falcon Boot ROM (BROM) as a parameter.
 #[repr(u8)]
 #[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
 pub(crate) enum FalconModSelAlgo {
+    /// AES.
+    #[expect(dead_code)]
+    Aes = 0,
     /// RSA3K.
     #[default]
     Rsa3k = 1,
@@ -209,15 +220,18 @@ pub(crate) enum FalconMem {
     Dmem,
 }
 
-/// Target/source of a DMA transfer to/from falcon memory.
+/// Defines the Framebuffer Interface (FBIF) aperture type.
+/// This determines the memory type for external memory access during a DMA transfer, which is
+/// performed by the Falcon's Framebuffer DMA (FBDMA) engine. See falcon.rst for more details.
 #[derive(Debug, Clone, Default)]
 pub(crate) enum FalconFbifTarget {
     /// VRAM.
     #[default]
+    /// Local Framebuffer (GPU's VRAM memory)
     LocalFb = 0,
-    /// Coherent system memory.
+    /// Coherent system memory (System DRAM).
     CoherentSysmem = 1,
-    /// Non-coherent system memory.
+    /// Non-coherent system memory (System DRAM).
     NoncoherentSysmem = 2,
 }
 impl_from_enum_to_u32!(FalconFbifTarget);
@@ -571,3 +585,4 @@ pub(crate) fn signature_reg_fuse_version(
             .signature_reg_fuse_version(self, bar, engine_id_mask, ucode_id)
     }
 }
+

-- 
2.50.0

[PATCH v3 4/7] Documentation: gpu: nova-core: Document vbios layout

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

Add detailed explanation and block diagrams of the layout of the vBIOS
on Nvidia GPUs. This is important to understand how nova-core boots an
Nvidia GPU.

[ Applied Timur Tabi's feedback on providing link to BIT documentation. ]

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 Documentation/gpu/nova/core/vbios.rst | 180 ++++++++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst      |   1 +
 2 files changed, 181 insertions(+)

diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
new file mode 100644
index 0000000000000000000000000000000000000000..55d7dd4a6658c2a20cc5617f96b278bc4ec2ba17
--- /dev/null
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -0,0 +1,180 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+==========
+VBIOS
+==========
+This document describes the layout of the VBIOS image which is a series of concatenated
+images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
+by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
+boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
+the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
+
+The format of the images in the ROM follow the "BIOS Specification" part of the
+PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
+are the ones that contain Falcon ucode and what we are mainly looking for.
+
+As an example, the following are the different image types that can be found in the
+VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
+
+- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
+  likely comes from the "IBM PC/AT" architecture.
+
+- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
+  driver that is used to display UEFI graphics output.
+
+- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
+
+- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
+  contains various different microcodes (also known as an applications) that do
+  a range of different functions. The FWSEC ucode is run in heavy-secure mode and
+  typically runs directly on the GSP (it could be running on a different designated
+  processor in future generations but as of Ampere, it is the GSP). This firmware
+  then loads other firmware ucodes onto the PMU and SEC2 microcontrollers for gfw
+  initialization after GPU reset and before the driver loads (see devinit.rst).
+  The DEVINIT ucode is itself another ucode that is stored in this ROM partition.
+
+Once located, the Falcon ucodes have "Application Interfaces" in their data
+memory (DMEM). For FWSEC, the application interface we use for FWSEC is the
+"DMEM mapper" interface which is configured to run the "FRTS" command. This
+command carves out the WPR2 (Write-Protected Region) in VRAM. It then places
+important power-management data, called 'FRTS', into this region. The WPR2
+region is only accessible to heavy-secure ucode.
+
+.. note::
+   It is not clear why FwSec has 2 different partitions in the ROM, but they both
+   are of type 0xE0 and can be identified as such. This could be subject to change
+   in future generations.
+
+VBIOS ROM Layout
+----------------
+The VBIOS layout is roughly a series of concatenated images as follows:
+(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
+
+.. note::
+   This diagram is created based on an GA-102 Ampere GPU as an example and could
+   vary for future or other GPUs.
+
+Here is a block diagram of the VBIOS layout::
+
+    +----------------------------------------------------------------------------+
+    | VBIOS (Starting at ROM_OFFSET: 0x300000)                                   |
+    +----------------------------------------------------------------------------+
+    | +-----------------------------------------------+                          |
+    | | PciAt Image (Type 0x00)                       |                          |
+    | +-----------------------------------------------+                          |
+    | | +-------------------+                         |                          |
+    | | | ROM Header        |                         |                          |
+    | | | (Signature 0xAA55)|                         |                          |
+    | | +-------------------+                         |                          |
+    | |         | rom header's pci_data_struct_offset |                          |
+    | |         | points to the PCIR structure        |                          |
+    | |         V                                     |                          |
+    | | +-------------------+                         |                          |
+    | | | PCIR Structure    |                         |                          |
+    | | | (Signature "PCIR")|                         |                          |
+    | | | last_image: 0x80  |                         |                          |
+    | | | image_len: size   |                         |                          |
+    | | | in 512-byte units |                         |                          |
+    | | +-------------------+                         |                          |
+    | |         |                                     |                          |
+    | |         | NPDE immediately follows PCIR       |                          |
+    | |         V                                     |                          |
+    | | +-------------------+                         |                          |
+    | | | NPDE Structure    |                         |                          |
+    | | | (Signature "NPDE")|                         |                          |
+    | | | last_image: 0x00  |                         |                          |
+    | | +-------------------+                         |                          |
+    | |                                               |                          |
+    | | +-------------------+                         |                          |
+    | | | BIT Header        | (Signature scanning     |                          |
+    | | | (Signature "BIT") |  provides the location  |                          |
+    | | +-------------------+  of the BIT table)      |                          |
+    | |         | header is                           |                          |
+    | |         | followed by a table of tokens       |                          |
+    | |         V one of which is for falcon data.    |                          |
+    | | +-------------------+                         |                          |
+    | | | BIT Tokens        |                         |                          |
+    | | |  ______________   |                         |                          |
+    | | | | Falcon Data |   |                         |                          |
+    | | | | Token (0x70)|---+------------>------------+--+                       |
+    | | | +-------------+   |  falcon_data_ptr()      |  |                       |
+    | | +-------------------+                         |  V                       |
+    | +-----------------------------------------------+  |                       |
+    |              (no gap between images)               |                       |
+    | +-----------------------------------------------+  |                       |
+    | | EFI Image (Type 0x03)                         |  |                       |
+    | +-----------------------------------------------+  |                       |
+    | | Contains the UEFI GOP driver (Graphics Output)|  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | ROM Header        |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | PCIR Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | NPDE Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | Image data        |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | +-----------------------------------------------+  |                       |
+    |              (no gap between images)               |                       |
+    | +-----------------------------------------------+  |                       |
+    | | First FwSec Image (Type 0xE0)                 |  |                       |
+    | +-----------------------------------------------+  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | ROM Header        |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | PCIR Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | NPDE Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | Image data        |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | +-----------------------------------------------+  |                       |
+    |              (no gap between images)               |                       |
+    | +-----------------------------------------------+  |                       |
+    | | Second FwSec Image (Type 0xE0)                |  |                       |
+    | +-----------------------------------------------+  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | ROM Header        |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | PCIR Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | NPDE Structure    |                         |  |                       |
+    | | +-------------------+                         |  |                       |
+    | |                                               |  |                       |
+    | | +-------------------+                         |  |                       |
+    | | | PMU Lookup Table  | <- falcon_data_offset <----+                       |
+    | | | +-------------+   |    pmu_lookup_table     |                          |
+    | | | | Entry 0x85  |   |                         |                          |
+    | | | | FWSEC_PROD  |   |                         |                          |
+    | | | +-------------+   |                         |                          |
+    | | +-------------------+                         |                          |
+    | |         |                                     |                          |
+    | |         | points to                           |                          |
+    | |         V                                     |                          |
+    | | +-------------------+                         |                          |
+    | | | FalconUCodeDescV3 | <- falcon_ucode_offset  |                          |
+    | | | (FWSEC Firmware)  |    fwsec_header()       |                          |
+    | | +-------------------+                         |                          |
+    | |         |   immediately followed  by...       |                          |
+    | |         V                                     |                          |
+    | | +----------------------------+                |                          |
+    | | | Signatures + FWSEC Ucode   |                |                          |
+    | | | fwsec_sigs(), fwsec_ucode()|                |                          |
+    | | +----------------------------+                |                          |
+    | +-----------------------------------------------+                          |
+    |                                                                            |
+    +----------------------------------------------------------------------------+
+
+Falcon data Lookup
+------------------
+A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
+Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
+used to find the required Falcon ucode based on an application ID.
+
+The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
+tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the
+offset of the same from the start of the VBIOS image. Unfortunately, the offset
+does not account for the EFI image located between the PciAt and FwSec images.
+The `vbios.rs` code compensates for this with appropriate arithmetic.
+
+.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af358b373d4ef47975d5a80213f3e276..f38041fcd595524b204eabf6ca3aad51038682cf 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,3 +28,4 @@ vGPU manager VFIO driver and the nova-drm driver.
 
    core/guidelines
    core/todo
+   core/vbios

-- 
2.50.0

[PATCH v3 5/7] Documentation: gpu: nova-core: Document devinit process

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 Documentation/gpu/nova/core/devinit.rst | 61 +++++++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst        |  1 +
 2 files changed, 62 insertions(+)

diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 0000000000000000000000000000000000000000..70c819a96a00a0a27846e7e96525470d07721a10
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,61 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+Device Initialization (devinit)
+==================================
+The devinit process is complex and subject to change. This document provides a high-level
+overview using the Ampere GPU family as an example. The goal is to provide a conceptual
+overview of the process to aid in understanding the corresponding kernel code.
+
+Device initialization (devinit) is a crucial sequence of register read/write operations
+that occur after a GPU reset. The devinit sequence is essential for properly configuring
+the GPU hardware before it can be used.
+
+The devinit engine is an interpreter program that typically runs on the PMU (Power Management
+Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
+commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the
+FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the
+nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the
+FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while
+devinit runs on the PMU in 'light-secure' mode.
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+
+Low-level Firmware Initialization Flow
+--------------------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
+firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is
+considered unusable until this initialization process completes.
+
+These low-level GPU firmware components are typically:
+
+1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
+2. Executed in sequence on different microcontrollers:
+
+  - The devinit engine typically but not necessarily runs on the PMU.
+  - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
+    heavy-secure mode.
+
+Before the driver can proceed with further initialization, it must wait for a signal
+indicating that core initialization is complete (known as GFW_BOOT). This signal is
+asserted by the FWSEC running on the GSP in heavy-secure mode.
+
+Runtime Considerations
+----------------------
+It's important to note that the devinit sequence also needs to run during suspend/resume
+operations at runtime, not just during initial boot, as it is critical to power management.
+
+Security and Access Control
+---------------------------
+The initialization process involves careful privilege management. For example, before
+accessing certain completion status registers, the driver must check privilege level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
+privilege level to allow CPU (LS/low-secure) access. This is the case, for example,
+when receiving the GFW_BOOT signal.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index f38041fcd595524b204eabf6ca3aad51038682cf..e4e017d926767284b5cee844d8dba32be6bc064c 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -29,3 +29,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/guidelines
    core/todo
    core/vbios
+   core/devinit

-- 
2.50.0

[PATCH v3 6/7] Documentation: gpu: nova-core: Document fwsec operation and layout

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 Documentation/gpu/nova/core/fwsec.rst | 182 ++++++++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst      |   1 +
 2 files changed, 183 insertions(+)

diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 0000000000000000000000000000000000000000..b3e2d8a265e2bd4c9659d5b53b8084bdb0287a26
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,182 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+=========================
+FWSEC (Firmware Security)
+=========================
+This document briefly/conceptually describes the FWSEC (Firmware Security) image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC is to provide a secure boot process. It runs in
+'Heavy-secure' mode, and performs firmware verification after a GPU reset
+before loading various ucode images onto other microcontrollers on the GPU,
+such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only needs
+to perform FRTS, since Secure Boot (SB) has already completed by the time the driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which contains
+data required for power management. Once setup, only HS mode ucode can access it
+(see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
+various ucode images (also known as applications) -- one of them being FWSEC. For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All these
+ucode images are stored in the same ROM partition and the PMU table is used to look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the nova-core driver, the FWSEC contains an 'application interface' called
+DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others.
+For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs and is subject to change
+completely, it is just provided as an example):
+
+Here is a block diagram of the FWSEC memory layout::
+
+   +---------------------------------------------------------------+
+   |                         FWSEC ROM image (type 0xE0)           |
+   |                                                               |
+   |  +---------------------------------+                          |
+   |  |     PMU Falcon Ucode Table      |                          |
+   |  |     (PmuLookupTable)            |                          |
+   |  |  +-------------------------+    |                          |
+   |  |  | Table Header            |    |                          |
+   |  |  | - version: 0x01         |    |                          |
+   |  |  | - header_size: 6        |    |                          |
+   |  |  | - entry_size: 6         |    |                          |
+   |  |  | - entry_count: N        |    |                          |
+   |  |  | - desc_version:3(unused)|    |                          |
+   |  |  +-------------------------+    |                          |
+   |  |         ...                     |                          |
+   |  |  +-------------------------+    |                          |
+   |  |  | Entry for FWSEC (0x85)  |    |                          |
+   |  |  | (PmuLookupTableEntry)   |    |                          |
+   |  |  | - app_id: 0x85 (FWSEC)  |----|----+                     |
+   |  |  | - target_id: 0x01 (PMU) |    |    |                     |
+   |  |  | - data: offset ---------|----|----|---+ look up FWSEC   |
+   |  |  +-------------------------+    |    |   |                 |
+   |  +---------------------------------+    |   |                 |
+   |                                         |   |                 |
+   |                                         |   |                 |
+   |  +---------------------------------+    |   |                 |
+   |  |     FWSEC Ucode Component       |<---+   |                 |
+   |  |     (aka Falcon data)           |        |                 |
+   |  |  +-------------------------+    |        |                 |
+   |  |  | FalconUCodeDescV3       |<---|--------+                 |
+   |  |  | - hdr                   |    |                          |
+   |  |  | - stored_size           |    |                          |
+   |  |  | - pkc_data_offset       |    |                          |
+   |  |  | - interface_offset -----|----|----------------+         |
+   |  |  | - imem_phys_base        |    |                |         |
+   |  |  | - imem_load_size        |    |                |         |
+   |  |  | - imem_virt_base        |    |                |         |
+   |  |  | - dmem_phys_base        |    |                |         |
+   |  |  | - dmem_load_size        |    |                |         |
+   |  |  | - engine_id_mask        |    |                |         |
+   |  |  | - ucode_id              |    |                |         |
+   |  |  | - signature_count       |    |    look up sig |         |
+   |  |  | - signature_versions --------------+          |         |
+   |  |  +-------------------------+    |     |          |         |
+   |  |         (no gap)                |     |          |         |
+   |  |  +-------------------------+    |     |          |         |
+   |  |  | Signatures Section      |<---|-----+          |         |
+   |  |  | (384 bytes per sig)     |    |                |         |
+   |  |  | - RSA-3K Signature 1    |    |                |         |
+   |  |  | - RSA-3K Signature 2    |    |                |         |
+   |  |  |   ...                   |    |                |         |
+   |  |  +-------------------------+    |                |         |
+   |  |                                 |                |         |
+   |  |  +-------------------------+    |                |         |
+   |  |  | IMEM Section (Code)     |    |                |         |
+   |  |  |                         |    |                |         |
+   |  |  | Contains instruction    |    |                |         |
+   |  |  | code etc.               |    |                |         |
+   |  |  +-------------------------+    |                |         |
+   |  |                                 |                |         |
+   |  |  +-------------------------+    |                |         |
+   |  |  | DMEM Section (Data)     |    |                |         |
+   |  |  |                         |    |                |         |
+   |  |  | +---------------------+ |    |                |         |
+   |  |  | | Application         | |<---|----------------+         |
+   |  |  | | Interface Table     | |    |                          |
+   |  |  | | (FalconAppifHdrV1)  | |    |                          |
+   |  |  | | Header:             | |    |                          |
+   |  |  | | - version: 0x01     | |    |                          |
+   |  |  | | - header_size: 4    | |    |                          |
+   |  |  | | - entry_size: 8     | |    |                          |
+   |  |  | | - entry_count: N    | |    |                          |
+   |  |  | |                     | |    |                          |
+   |  |  | | Entries:            | |    |                          |
+   |  |  | | +-----------------+ | |    |                          |
+   |  |  | | | DEVINIT (ID 1)  | | |    |                          |
+   |  |  | | | - id: 0x01      | | |    |                          |
+   |  |  | | | - dmemOffset X -|-|-|----+                          |
+   |  |  | | +-----------------+ | |    |                          |
+   |  |  | | +-----------------+ | |    |                          |
+   |  |  | | | DMEMMAPPER(ID 4)| | |    |                          |
+   |  |  | | | - id: 0x04      | | |    | Used only for DevInit    |
+   |  |  | | |  (NVFW_FALCON_  | | |    | application (not FWSEC)  |
+   |  |  | | |   APPIF_ID_DMEMMAPPER)   |                          |
+   |  |  | | | - dmemOffset Y -|-|-|----|-----+                    |
+   |  |  | | +-----------------+ | |    |     |                    |
+   |  |  | +---------------------+ |    |     |                    |
+   |  |  |                         |    |     |                    |
+   |  |  | +---------------------+ |    |     |                    |
+   |  |  | | DEVINIT Engine      |<|----+     | Used by FWSEC      |
+   |  |  | | Interface           | |    |     |         app.       |
+   |  |  | +---------------------+ |    |     |                    |
+   |  |  |                         |    |     |                    |
+   |  |  | +---------------------+ |    |     |                    |
+   |  |  | | DMEM Mapper (ID 4)  |<|----+-----+                    |
+   |  |  | | (FalconAppifDmemmapperV3)  |                          |
+   |  |  | | - signature: "DMAP" | |    |                          |
+   |  |  | | - version: 0x0003   | |    |                          |
+   |  |  | | - Size: 64 bytes    | |    |                          |
+   |  |  | | - cmd_in_buffer_off | |----|------------+             |
+   |  |  | | - cmd_in_buffer_size| |    |            |             |
+   |  |  | | - cmd_out_buffer_off| |----|------------|-----+       |
+   |  |  | | - cmd_out_buffer_sz | |    |            |     |       |
+   |  |  | | - init_cmd          | |    |            |     |       |
+   |  |  | | - features          | |    |            |     |       |
+   |  |  | | - cmd_mask0/1       | |    |            |     |       |
+   |  |  | +---------------------+ |    |            |     |       |
+   |  |  |                         |    |            |     |       |
+   |  |  | +---------------------+ |    |            |     |       |
+   |  |  | | Command Input Buffer|<|----|------------+     |       |
+   |  |  | | - Command data      | |    |                  |       |
+   |  |  | | - Arguments         | |    |                  |       |
+   |  |  | +---------------------+ |    |                  |       |
+   |  |  |                         |    |                  |       |
+   |  |  | +---------------------+ |    |                  |       |
+   |  |  | | Command Output      |<|----|------------------+       |
+   |  |  | | Buffer              | |    |                          |
+   |  |  | | - Results           | |    |                          |
+   |  |  | | - Status            | |    |                          |
+   |  |  | +---------------------+ |    |                          |
+   |  |  +-------------------------+    |                          |
+   |  +---------------------------------+                          |
+   |                                                               |
+   +---------------------------------------------------------------+
+
+.. note::
+   The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
+   (Video Protected Region) initialization as well. Before the nova-core driver is even
+   loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
+   sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
+   GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
+   async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
+   memory allocators need to wait for this scrubbing to complete).
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index e4e017d926767284b5cee844d8dba32be6bc064c..e3650f53ff53113f31f63f67cf26116b5c070693 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -30,3 +30,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/todo
    core/vbios
    core/devinit
+   core/fwsec

-- 
2.50.0

[PATCH v3 7/7] Documentation: gpu: nova-core: Document basics of the Falcon

[Alexandre Courbot]

From: Joel Fernandes <joelagnelf@nvidia.com>

Instances of the Falcon microcontroller appear in modern Nvidia GPUs and
are crucial to the GPU boot process. Document some concepts which will
make nova-core boot code easier to digest. All the information is
derived from public sources such as public documents, OpenRM and Nouveau
code.

Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
 Documentation/gpu/nova/core/falcon.rst | 158 +++++++++++++++++++++++++++++++++
 Documentation/gpu/nova/index.rst       |   1 +
 2 files changed, 159 insertions(+)

diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst
new file mode 100644
index 0000000000000000000000000000000000000000..33137082eb6c14cecda2fbe6fdb79e63ee9ca2e6
--- /dev/null
+++ b/Documentation/gpu/nova/core/falcon.rst
@@ -0,0 +1,158 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Falcon (FAst Logic Controller)
+==============================
+The following sections describe the Falcon core and the ucode running on it.
+The descriptions are based on the Ampere GPU or earlier designs; however, they
+should mostly apply to future designs as well, but everything is subject to
+change. The overview provided here is mainly tailored towards understanding the
+interactions of nova-core driver with the Falcon.
+
+NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
+handle secure firmware tasks, initialization, and power management. Modern
+NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
+processor) and SEC2 (the security engine)) and also may integrate a RISC-V core.
+This core is capable of running both RISC-V and Falcon code.
+
+The code running on the Falcon cores is also called 'ucode', and will be
+referred to as such in the following sections.
+
+Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
+small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from
+system memory. The nova-core driver must reset and configure the Falcon, load
+its firmware via DMA, and start its CPU.
+
+Falcon security levels
+======================
+Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS)
+modes.
+
+Heavy Secured (HS) also known as Privilege Level 3 (PL3)
+--------------------------------------------------------
+HS ucode is the most trusted code and has access to pretty much everything on
+the chip. The HS binary includes a signature in it which is verified at boot.
+This signature verification is done by the hardware itself, thus establishing a
+root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
+GSP in HS mode. FRTS, which involves setting up and loading content into the WPR
+(Write Protect Region), has to be done by the HS ucode and cannot be done by the
+host CPU or LS ucode.
+
+Light Secured (LS or PL2) and Non Secured (NS or PL0)
+-----------------------------------------------------
+These modes are less secure than HS. Like HS, the LS or NS ucode binary also
+typically includes a signature in it. To load firmware in LS or NS mode onto a
+Falcon, another Falcon needs to be running in HS mode, which also establishes the
+root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter"
+ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
+run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
+example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
+devinit engine onto the PMU in LS mode.
+
+Root of trust establishment
+---------------------------
+To establish a root of trust, the code running on a Falcon must be immutable and
+hardwired into a read-only memory (ROM). This follows industry norms for
+verification of firmware. This code is called the Boot ROM (BROM). The nova-core
+driver on the CPU communicates with Falcon's Boot ROM through various Falcon
+registers prefixed with "BROM" (see regs.rs).
+
+After nova-core driver reads the necessary ucode from VBIOS, it programs the
+BROM and DMA registers to trigger the Falcon to load the HS ucode from the system
+memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified
+by the Falcon's Boot ROM.
+
+Once the verified HS code is running on a Falcon, it can verify and load other
+LS/NS ucode binaries onto other Falcons and start them. The process of signature
+verification is the same as HS; just in this case, the hardware (BROM) doesn't
+compute the signature, but the HS ucode does.
+
+The root of trust is therefore established as follows:
+     Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode.
+
+On an Ampere GPU, for example, the boot verification flow is:
+     Hardware (Boot ROM running on the SEC2) ->
+          HS ucode (Booter running on the SEC2) ->
+               LS ucode (GSP-RM running on the GSP)
+
+.. note::
+     While the CPU can load HS ucode onto a Falcon microcontroller and have it
+     verified by the hardware and run, the CPU itself typically does not load
+     LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
+     HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the
+     SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
+     the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no
+     component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load
+     LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is
+     temporarily made to run HS ucode (which is itself loaded by the CPU via
+     the nova-core driver using a "GSP-provided sequencer") which then loads
+     the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes
+     running its own GSP-RM LS ucode.
+
+Falcon memory subsystem and DMA engine
+======================================
+Falcons have separate instruction and data memories (IMEM/DMEM)
+and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
+DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
+(Framebuffer Interface), to external memory.
+
+DMA transfers are possible from the Falcon's memory to both the system memory
+and the framebuffer memory (VRAM).
+
+To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
+is accessed (also known as aperture type). In the nova-core driver, this is
+determined by the `FalconFbifTarget` enum.
+
+The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
+controls access by the FBDMA to the external memory.
+
+Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows::
+
+               External Memory (Framebuffer / System DRAM)
+                              ^  |
+                              |  |
+                              |  v
+     +-----------------------------------------------------+
+     |                           |                         |
+     |   +---------------+       |                         |
+     |   |     FBIF      |-------+                         |  FALCON
+     |   | (FrameBuffer  |   Memory Interface              |  PROCESSOR
+     |   |  InterFace)   |                                 |
+     |   |  Apertures    |                                 |
+     |   |  Configures   |                                 |
+     |   |  mem access   |                                 |
+     |   +-------^-------+                                 |
+     |           |                                         |
+     |           | FBDMA uses configured FBIF apertures    |
+     |           | to access External Memory
+     |           |
+     |   +-------v--------+      +---------------+
+     |   |    FBDMA       |  cfg |     RISC      |
+     |   | (FrameBuffer   |<---->|     CORE      |----->. Direct Core Access
+     |   |  DMA Engine)   |      |               |      |
+     |   | - Master dev.  |      | (can run both |      |
+     |   +-------^--------+      | Falcon and    |      |
+     |           |        cfg--->| RISC-V code)  |      |
+     |           |        /      |               |      |
+     |           |        |      +---------------+      |    +------------+
+     |           |        |                             |    |   BROM     |
+     |           |        |                             <--->| (Boot ROM) |
+     |           |       /                              |    +------------+
+     |           |      v                               |
+     |   +---------------+                              |
+     |   |    IO-PMP     | Controls access by FBDMA     |
+     |   | (IO Physical  | and other IO Masters         |
+     |   | Memory Protect)                              |
+     |   +-------^-------+                              |
+     |           |                                      |
+     |           | Protected Access Path for FBDMA      |
+     |           v                                      |
+     |   +---------------------------------------+      |
+     |   |       Memory                          |      |
+     |   |   +---------------+  +------------+   |      |
+     |   |   |    IMEM       |  |    DMEM    |   |<-----+
+     |   |   | (Instruction  |  |   (Data    |   |
+     |   |   |  Memory)      |  |   Memory)  |   |
+     |   |   +---------------+  +------------+   |
+     |   +---------------------------------------+
+     +-----------------------------------------------------+
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index e3650f53ff53113f31f63f67cf26116b5c070693..e39cb3163581ea4ff5b441b82e9efa4282c946f8 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -31,3 +31,4 @@ vGPU manager VFIO driver and the nova-drm driver.
    core/vbios
    core/devinit
    core/fwsec
+   core/falcon

-- 
2.50.0

Re: [PATCH v3 0/7] Documentation for nova-core

[Joel Fernandes]

On 7/2/2025 7:00 AM, Alexandre Courbot wrote:
> This series adds some documentation that was relevant to the FWSEC-FRTS
> [1] series, but wasn't sent alongside it as it was worked on in
> parallel.
> 
> It notably introduces a lot of HTMLdocs that adds key explanations to
> understand and work on the GSP boot process.
> 
> [1] https://lore.kernel.org/all/DB0I8WAH970B.25D3S59AYF85P@nvidia.com/
> 
> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>

Thanks a lot Alex!!!

 - Joel


> ---
> Changes in v3:
> - Rebase on top of latest nova-next.
> - Reorganize patches a bit and use standard prefixes.
> - Convert diagrams to only use ASCII characters (thanks Bagas!).
> - Move sysmembar documentation to the new `SysmemFlush` type.
> - Reword things a bit here and there.
> - Reorganize order of `index.rst` for a more natural flow.
> - Link to v2: https://lore.kernel.org/rust-for-linux/20250503040802.1411285-1-joelagnelf@nvidia.com/
> 
> ---
> Joel Fernandes (7):
>       gpu: nova-core: Add code comments related to devinit
>       gpu: nova-core: Clarify sysmembar operations
>       gpu: nova-core: Clarify falcon code
>       Documentation: gpu: nova-core: Document vbios layout
>       Documentation: gpu: nova-core: Document devinit process
>       Documentation: gpu: nova-core: Document fwsec operation and layout
>       Documentation: gpu: nova-core: Document basics of the Falcon
> 
>  Documentation/gpu/nova/core/devinit.rst |  61 +++++++++++
>  Documentation/gpu/nova/core/falcon.rst  | 158 +++++++++++++++++++++++++++
>  Documentation/gpu/nova/core/fwsec.rst   | 182 ++++++++++++++++++++++++++++++++
>  Documentation/gpu/nova/core/vbios.rst   | 180 +++++++++++++++++++++++++++++++
>  Documentation/gpu/nova/index.rst        |   4 +
>  drivers/gpu/nova-core/falcon.rs         |  29 +++--
>  drivers/gpu/nova-core/fb.rs             |  10 ++
>  drivers/gpu/nova-core/gfw.rs            |  39 ++++++-
>  drivers/gpu/nova-core/gpu.rs            |   3 +-
>  drivers/gpu/nova-core/regs.rs           |  18 +++-
>  10 files changed, 669 insertions(+), 15 deletions(-)
> ---
> base-commit: 4092e1b41202ff39aad75a40a03ac1d318443670
> change-id: 20250702-nova-docs-b9900d0505b5
> 
> Best regards,

Re: [PATCH v3 4/7] Documentation: gpu: nova-core: Document vbios layout

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 13971 bytes --]

On Wed, Jul 02, 2025 at 08:00:41PM +0900, Alexandre Courbot wrote:
> diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
> new file mode 100644
> index 0000000000000000000000000000000000000000..55d7dd4a6658c2a20cc5617f96b278bc4ec2ba17
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/vbios.rst
> @@ -0,0 +1,180 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +
> +==========
> +VBIOS
> +==========
> +This document describes the layout of the VBIOS image which is a series of concatenated
> +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
> +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
> +boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
   bootstrap
> +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
> +
> +The format of the images in the ROM follow the "BIOS Specification" part of the
> +PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
> +are the ones that contain Falcon ucode and what we are mainly looking for.
> +
> +As an example, the following are the different image types that can be found in the
> +VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
> +
> +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
> +  likely comes from the "IBM PC/AT" architecture.
> +
> +- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
> +  driver that is used to display UEFI graphics output.
> +
> +- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
> +
> +- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
> +  contains various different microcodes (also known as an applications) that do
              various microcodes?
> +  a range of different functions. The FWSEC ucode is run in heavy-secure mode and
> +  typically runs directly on the GSP (it could be running on a different designated
> +  processor in future generations but as of Ampere, it is the GSP). This firmware
> +  then loads other firmware ucodes onto the PMU and SEC2 microcontrollers for gfw
> +  initialization after GPU reset and before the driver loads (see devinit.rst).
> +  The DEVINIT ucode is itself another ucode that is stored in this ROM partition.
> +
> +Once located, the Falcon ucodes have "Application Interfaces" in their data
> +memory (DMEM). For FWSEC, the application interface we use for FWSEC is the
> +"DMEM mapper" interface which is configured to run the "FRTS" command. This
> +command carves out the WPR2 (Write-Protected Region) in VRAM. It then places
> +important power-management data, called 'FRTS', into this region. The WPR2
> +region is only accessible to heavy-secure ucode.
> +
> +.. note::
> +   It is not clear why FwSec has 2 different partitions in the ROM, but they both
> +   are of type 0xE0 and can be identified as such. This could be subject to change
> +   in future generations.
> +
> +VBIOS ROM Layout
> +----------------
> +The VBIOS layout is roughly a series of concatenated images as follows:
> +(For more explanations of acronyms, see the detailed descriptions in vbios.rs).

Looks OK, but...

> +
> +.. note::
> +   This diagram is created based on an GA-102 Ampere GPU as an example and could
> +   vary for future or other GPUs.
> +
> +Here is a block diagram of the VBIOS layout::

Above sentence (but not the note directive) is redundant, though.

> +
> +    +----------------------------------------------------------------------------+
> +    | VBIOS (Starting at ROM_OFFSET: 0x300000)                                   |
> +    +----------------------------------------------------------------------------+
> +    | +-----------------------------------------------+                          |
> +    | | PciAt Image (Type 0x00)                       |                          |
> +    | +-----------------------------------------------+                          |
> +    | | +-------------------+                         |                          |
> +    | | | ROM Header        |                         |                          |
> +    | | | (Signature 0xAA55)|                         |                          |
> +    | | +-------------------+                         |                          |
> +    | |         | rom header's pci_data_struct_offset |                          |
> +    | |         | points to the PCIR structure        |                          |
> +    | |         V                                     |                          |
> +    | | +-------------------+                         |                          |
> +    | | | PCIR Structure    |                         |                          |
> +    | | | (Signature "PCIR")|                         |                          |
> +    | | | last_image: 0x80  |                         |                          |
> +    | | | image_len: size   |                         |                          |
> +    | | | in 512-byte units |                         |                          |
> +    | | +-------------------+                         |                          |
> +    | |         |                                     |                          |
> +    | |         | NPDE immediately follows PCIR       |                          |
> +    | |         V                                     |                          |
> +    | | +-------------------+                         |                          |
> +    | | | NPDE Structure    |                         |                          |
> +    | | | (Signature "NPDE")|                         |                          |
> +    | | | last_image: 0x00  |                         |                          |
> +    | | +-------------------+                         |                          |
> +    | |                                               |                          |
> +    | | +-------------------+                         |                          |
> +    | | | BIT Header        | (Signature scanning     |                          |
> +    | | | (Signature "BIT") |  provides the location  |                          |
> +    | | +-------------------+  of the BIT table)      |                          |
> +    | |         | header is                           |                          |
> +    | |         | followed by a table of tokens       |                          |
> +    | |         V one of which is for falcon data.    |                          |
> +    | | +-------------------+                         |                          |
> +    | | | BIT Tokens        |                         |                          |
> +    | | |  ______________   |                         |                          |
> +    | | | | Falcon Data |   |                         |                          |
> +    | | | | Token (0x70)|---+------------>------------+--+                       |
> +    | | | +-------------+   |  falcon_data_ptr()      |  |                       |
> +    | | +-------------------+                         |  V                       |
> +    | +-----------------------------------------------+  |                       |
> +    |              (no gap between images)               |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | EFI Image (Type 0x03)                         |  |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | Contains the UEFI GOP driver (Graphics Output)|  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | ROM Header        |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | PCIR Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | NPDE Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | Image data        |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | +-----------------------------------------------+  |                       |
> +    |              (no gap between images)               |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | First FwSec Image (Type 0xE0)                 |  |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | ROM Header        |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | PCIR Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | NPDE Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | Image data        |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | +-----------------------------------------------+  |                       |
> +    |              (no gap between images)               |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | Second FwSec Image (Type 0xE0)                |  |                       |
> +    | +-----------------------------------------------+  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | ROM Header        |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | PCIR Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | NPDE Structure    |                         |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | |                                               |  |                       |
> +    | | +-------------------+                         |  |                       |
> +    | | | PMU Lookup Table  | <- falcon_data_offset <----+                       |
> +    | | | +-------------+   |    pmu_lookup_table     |                          |
> +    | | | | Entry 0x85  |   |                         |                          |
> +    | | | | FWSEC_PROD  |   |                         |                          |
> +    | | | +-------------+   |                         |                          |
> +    | | +-------------------+                         |                          |
> +    | |         |                                     |                          |
> +    | |         | points to                           |                          |
> +    | |         V                                     |                          |
> +    | | +-------------------+                         |                          |
> +    | | | FalconUCodeDescV3 | <- falcon_ucode_offset  |                          |
> +    | | | (FWSEC Firmware)  |    fwsec_header()       |                          |
> +    | | +-------------------+                         |                          |
> +    | |         |   immediately followed  by...       |                          |
> +    | |         V                                     |                          |
> +    | | +----------------------------+                |                          |
> +    | | | Signatures + FWSEC Ucode   |                |                          |
> +    | | | fwsec_sigs(), fwsec_ucode()|                |                          |
> +    | | +----------------------------+                |                          |
> +    | +-----------------------------------------------+                          |
> +    |                                                                            |
> +    +----------------------------------------------------------------------------+
> +
> +Falcon data Lookup
> +------------------
> +A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
> +Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
> +used to find the required Falcon ucode based on an application ID.
> +
> +The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
> +tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the
> +offset of the same from the start of the VBIOS image. Unfortunately, the offset
> +does not account for the EFI image located between the PciAt and FwSec images.
> +The `vbios.rs` code compensates for this with appropriate arithmetic.
> +
> +.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html

The rest looks good.

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 5/7] Documentation: gpu: nova-core: Document devinit process

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 3769 bytes --]

On Wed, Jul 02, 2025 at 08:00:42PM +0900, Alexandre Courbot wrote:
> diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst
> new file mode 100644
> index 0000000000000000000000000000000000000000..70c819a96a00a0a27846e7e96525470d07721a10
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/devinit.rst
> @@ -0,0 +1,61 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==================================
> +Device Initialization (devinit)
> +==================================
> +The devinit process is complex and subject to change. This document provides a high-level
> +overview using the Ampere GPU family as an example. The goal is to provide a conceptual
> +overview of the process to aid in understanding the corresponding kernel code.
> +
> +Device initialization (devinit) is a crucial sequence of register read/write operations
> +that occur after a GPU reset. The devinit sequence is essential for properly configuring
> +the GPU hardware before it can be used.
> +
> +The devinit engine is an interpreter program that typically runs on the PMU (Power Management
> +Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
> +commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the
> +FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the
> +nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the
> +FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while
> +devinit runs on the PMU in 'light-secure' mode.
> +
> +Key Functions of devinit
> +------------------------
> +devinit performs several critical tasks:
> +
> +1. Programming VRAM memory controller timings
> +2. Power sequencing
> +3. Clock and PLL (Phase-Locked Loop) configuration
> +4. Thermal management
> +
> +Low-level Firmware Initialization Flow
> +--------------------------------------
> +Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
> +firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is
> +considered unusable until this initialization process completes.
> +
> +These low-level GPU firmware components are typically:
> +
> +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
> +2. Executed in sequence on different microcontrollers:
> +
> +  - The devinit engine typically but not necessarily runs on the PMU.
> +  - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
> +    heavy-secure mode.
> +
> +Before the driver can proceed with further initialization, it must wait for a signal
> +indicating that core initialization is complete (known as GFW_BOOT). This signal is
> +asserted by the FWSEC running on the GSP in heavy-secure mode.
> +
> +Runtime Considerations
> +----------------------
> +It's important to note that the devinit sequence also needs to run during suspend/resume
> +operations at runtime, not just during initial boot, as it is critical to power management.
> +
> +Security and Access Control
> +---------------------------
> +The initialization process involves careful privilege management. For example, before
> +accessing certain completion status registers, the driver must check privilege level
> +masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
> +privilege level to allow CPU (LS/low-secure) access. This is the case, for example,
> +when receiving the GFW_BOOT signal.

The wording LGTM, thanks!

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 6/7] Documentation: gpu: nova-core: Document fwsec operation and layout

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 542 bytes --]

On Wed, Jul 02, 2025 at 08:00:43PM +0900, Alexandre Courbot wrote:
> +FWSEC Memory Layout
> +-------------------
> +The memory layout of the FWSEC image is as follows (this is using an GA-102
> +Ampere GPU as an example and could vary for future GPUs and is subject to change
> +completely, it is just provided as an example):
> +
> +Here is a block diagram of the FWSEC memory layout::

Above sentence is redundant (as it is implied by preceding paragraph).

Thanks.

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 7/7] Documentation: gpu: nova-core: Document basics of the Falcon

[Bagas Sanjaya]

[-- Attachment #1: Type: text/plain, Size: 9703 bytes --]

On Wed, Jul 02, 2025 at 08:00:44PM +0900, Alexandre Courbot wrote:
> diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst
> new file mode 100644
> index 0000000000000000000000000000000000000000..33137082eb6c14cecda2fbe6fdb79e63ee9ca2e6
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/falcon.rst
> @@ -0,0 +1,158 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +==============================
> +Falcon (FAst Logic Controller)
> +==============================
> +The following sections describe the Falcon core and the ucode running on it.
> +The descriptions are based on the Ampere GPU or earlier designs; however, they
> +should mostly apply to future designs as well, but everything is subject to
> +change. The overview provided here is mainly tailored towards understanding the
> +interactions of nova-core driver with the Falcon.
> +
> +NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
> +handle secure firmware tasks, initialization, and power management. Modern
> +NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
> +processor) and SEC2 (the security engine)) and also may integrate a RISC-V core.
> +This core is capable of running both RISC-V and Falcon code.
> +
> +The code running on the Falcon cores is also called 'ucode', and will be
> +referred to as such in the following sections.
> +
> +Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
> +small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from
> +system memory. The nova-core driver must reset and configure the Falcon, load
> +its firmware via DMA, and start its CPU.
> +
> +Falcon security levels
> +======================
> +Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS)
> +modes.
> +
> +Heavy Secured (HS) also known as Privilege Level 3 (PL3)
> +--------------------------------------------------------
> +HS ucode is the most trusted code and has access to pretty much everything on
> +the chip. The HS binary includes a signature in it which is verified at boot.
> +This signature verification is done by the hardware itself, thus establishing a
> +root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
> +GSP in HS mode. FRTS, which involves setting up and loading content into the WPR
> +(Write Protect Region), has to be done by the HS ucode and cannot be done by the
> +host CPU or LS ucode.
> +
> +Light Secured (LS or PL2) and Non Secured (NS or PL0)
> +-----------------------------------------------------
> +These modes are less secure than HS. Like HS, the LS or NS ucode binary also
> +typically includes a signature in it. To load firmware in LS or NS mode onto a
> +Falcon, another Falcon needs to be running in HS mode, which also establishes the
> +root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter"
> +ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
> +run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
> +example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
> +devinit engine onto the PMU in LS mode.
> +
> +Root of trust establishment
> +---------------------------
> +To establish a root of trust, the code running on a Falcon must be immutable and
> +hardwired into a read-only memory (ROM). This follows industry norms for
> +verification of firmware. This code is called the Boot ROM (BROM). The nova-core
> +driver on the CPU communicates with Falcon's Boot ROM through various Falcon
> +registers prefixed with "BROM" (see regs.rs).
> +
> +After nova-core driver reads the necessary ucode from VBIOS, it programs the
> +BROM and DMA registers to trigger the Falcon to load the HS ucode from the system
> +memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified
> +by the Falcon's Boot ROM.
> +
> +Once the verified HS code is running on a Falcon, it can verify and load other
> +LS/NS ucode binaries onto other Falcons and start them. The process of signature
> +verification is the same as HS; just in this case, the hardware (BROM) doesn't
> +compute the signature, but the HS ucode does.
> +
> +The root of trust is therefore established as follows:
> +     Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode.
> +
> +On an Ampere GPU, for example, the boot verification flow is:
> +     Hardware (Boot ROM running on the SEC2) ->
> +          HS ucode (Booter running on the SEC2) ->
> +               LS ucode (GSP-RM running on the GSP)
> +
> +.. note::
> +     While the CPU can load HS ucode onto a Falcon microcontroller and have it
> +     verified by the hardware and run, the CPU itself typically does not load
> +     LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
> +     HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the
> +     SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
> +     the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no
> +     component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load
> +     LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is
> +     temporarily made to run HS ucode (which is itself loaded by the CPU via
> +     the nova-core driver using a "GSP-provided sequencer") which then loads
> +     the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes
> +     running its own GSP-RM LS ucode.
> +
> +Falcon memory subsystem and DMA engine
> +======================================
> +Falcons have separate instruction and data memories (IMEM/DMEM)
> +and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
> +DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
> +(Framebuffer Interface), to external memory.
> +
> +DMA transfers are possible from the Falcon's memory to both the system memory
> +and the framebuffer memory (VRAM).
> +
> +To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
> +is accessed (also known as aperture type). In the nova-core driver, this is
> +determined by the `FalconFbifTarget` enum.
> +
> +The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
> +controls access by the FBDMA to the external memory.
> +
> +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows::
> +
> +               External Memory (Framebuffer / System DRAM)
> +                              ^  |
> +                              |  |
> +                              |  v
> +     +-----------------------------------------------------+
> +     |                           |                         |
> +     |   +---------------+       |                         |
> +     |   |     FBIF      |-------+                         |  FALCON
> +     |   | (FrameBuffer  |   Memory Interface              |  PROCESSOR
> +     |   |  InterFace)   |                                 |
> +     |   |  Apertures    |                                 |
> +     |   |  Configures   |                                 |
> +     |   |  mem access   |                                 |
> +     |   +-------^-------+                                 |
> +     |           |                                         |
> +     |           | FBDMA uses configured FBIF apertures    |
> +     |           | to access External Memory
> +     |           |
> +     |   +-------v--------+      +---------------+
> +     |   |    FBDMA       |  cfg |     RISC      |
> +     |   | (FrameBuffer   |<---->|     CORE      |----->. Direct Core Access
> +     |   |  DMA Engine)   |      |               |      |
> +     |   | - Master dev.  |      | (can run both |      |
> +     |   +-------^--------+      | Falcon and    |      |
> +     |           |        cfg--->| RISC-V code)  |      |
> +     |           |        /      |               |      |
> +     |           |        |      +---------------+      |    +------------+
> +     |           |        |                             |    |   BROM     |
> +     |           |        |                             <--->| (Boot ROM) |
> +     |           |       /                              |    +------------+
> +     |           |      v                               |
> +     |   +---------------+                              |
> +     |   |    IO-PMP     | Controls access by FBDMA     |
> +     |   | (IO Physical  | and other IO Masters         |
> +     |   | Memory Protect)                              |
> +     |   +-------^-------+                              |
> +     |           |                                      |
> +     |           | Protected Access Path for FBDMA      |
> +     |           v                                      |
> +     |   +---------------------------------------+      |
> +     |   |       Memory                          |      |
> +     |   |   +---------------+  +------------+   |      |
> +     |   |   |    IMEM       |  |    DMEM    |   |<-----+
> +     |   |   | (Instruction  |  |   (Data    |   |
> +     |   |   |  Memory)      |  |   Memory)  |   |
> +     |   |   +---------------+  +------------+   |
> +     |   +---------------------------------------+
> +     +-----------------------------------------------------+

The wording LGTM, thanks!

Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>

-- 
An old man doll... just what I always wanted! - Clara

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v3 4/7] Documentation: gpu: nova-core: Document vbios layout

[Joel Fernandes]

On 7/2/2025 8:20 PM, Bagas Sanjaya wrote:
> On Wed, Jul 02, 2025 at 08:00:41PM +0900, Alexandre Courbot wrote:
>> diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
>> new file mode 100644
>> index 0000000000000000000000000000000000000000..55d7dd4a6658c2a20cc5617f96b278bc4ec2ba17
>> --- /dev/null
>> +++ b/Documentation/gpu/nova/core/vbios.rst
>> @@ -0,0 +1,180 @@
>> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +
>> +==========
>> +VBIOS
>> +==========
>> +This document describes the layout of the VBIOS image which is a series of concatenated
>> +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
>> +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
>> +boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
>
>    bootstrap
>
>> +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
>> +
>> +The format of the images in the ROM follow the "BIOS Specification" part of the
>> +PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
>> +are the ones that contain Falcon ucode and what we are mainly looking for.
>> +
>> +As an example, the following are the different image types that can be found in the
>> +VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
>> +
>> +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
>> +  likely comes from the "IBM PC/AT" architecture.
>> +
>> +- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
>> +  driver that is used to display UEFI graphics output.
>> +
>> +- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
>> +
>> +- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
>> +  contains various different microcodes (also known as an applications) that do
>
>               various microcodes?

The above 2 fixups sounds good to me, I'm guessing Danilo if you're pulling this
then maybe apply the fixups directly to the patch since it is trivial?

> 
>> +
>> +.. note::
>> +   This diagram is created based on an GA-102 Ampere GPU as an example and could
>> +   vary for future or other GPUs.
>> +
>> +Here is a block diagram of the VBIOS layout::
> 
> Above sentence (but not the note directive) is redundant, though.

This suggestion I'd skip.

But thanks for all the review!

 - Joel


> 
>> +
>> +    +----------------------------------------------------------------------------+
>> +    | VBIOS (Starting at ROM_OFFSET: 0x300000)                                   |
>> +    +----------------------------------------------------------------------------+
>> +    | +-----------------------------------------------+                          |
>> +    | | PciAt Image (Type 0x00)                       |                          |
>> +    | +-----------------------------------------------+                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | ROM Header        |                         |                          |
>> +    | | | (Signature 0xAA55)|                         |                          |
>> +    | | +-------------------+                         |                          |
>> +    | |         | rom header's pci_data_struct_offset |                          |
>> +    | |         | points to the PCIR structure        |                          |
>> +    | |         V                                     |                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | PCIR Structure    |                         |                          |
>> +    | | | (Signature "PCIR")|                         |                          |
>> +    | | | last_image: 0x80  |                         |                          |
>> +    | | | image_len: size   |                         |                          |
>> +    | | | in 512-byte units |                         |                          |
>> +    | | +-------------------+                         |                          |
>> +    | |         |                                     |                          |
>> +    | |         | NPDE immediately follows PCIR       |                          |
>> +    | |         V                                     |                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | NPDE Structure    |                         |                          |
>> +    | | | (Signature "NPDE")|                         |                          |
>> +    | | | last_image: 0x00  |                         |                          |
>> +    | | +-------------------+                         |                          |
>> +    | |                                               |                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | BIT Header        | (Signature scanning     |                          |
>> +    | | | (Signature "BIT") |  provides the location  |                          |
>> +    | | +-------------------+  of the BIT table)      |                          |
>> +    | |         | header is                           |                          |
>> +    | |         | followed by a table of tokens       |                          |
>> +    | |         V one of which is for falcon data.    |                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | BIT Tokens        |                         |                          |
>> +    | | |  ______________   |                         |                          |
>> +    | | | | Falcon Data |   |                         |                          |
>> +    | | | | Token (0x70)|---+------------>------------+--+                       |
>> +    | | | +-------------+   |  falcon_data_ptr()      |  |                       |
>> +    | | +-------------------+                         |  V                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    |              (no gap between images)               |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | EFI Image (Type 0x03)                         |  |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | Contains the UEFI GOP driver (Graphics Output)|  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | ROM Header        |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | PCIR Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | NPDE Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | Image data        |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    |              (no gap between images)               |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | First FwSec Image (Type 0xE0)                 |  |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | ROM Header        |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | PCIR Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | NPDE Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | Image data        |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    |              (no gap between images)               |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | Second FwSec Image (Type 0xE0)                |  |                       |
>> +    | +-----------------------------------------------+  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | ROM Header        |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | PCIR Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | NPDE Structure    |                         |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | |                                               |  |                       |
>> +    | | +-------------------+                         |  |                       |
>> +    | | | PMU Lookup Table  | <- falcon_data_offset <----+                       |
>> +    | | | +-------------+   |    pmu_lookup_table     |                          |
>> +    | | | | Entry 0x85  |   |                         |                          |
>> +    | | | | FWSEC_PROD  |   |                         |                          |
>> +    | | | +-------------+   |                         |                          |
>> +    | | +-------------------+                         |                          |
>> +    | |         |                                     |                          |
>> +    | |         | points to                           |                          |
>> +    | |         V                                     |                          |
>> +    | | +-------------------+                         |                          |
>> +    | | | FalconUCodeDescV3 | <- falcon_ucode_offset  |                          |
>> +    | | | (FWSEC Firmware)  |    fwsec_header()       |                          |
>> +    | | +-------------------+                         |                          |
>> +    | |         |   immediately followed  by...       |                          |
>> +    | |         V                                     |                          |
>> +    | | +----------------------------+                |                          |
>> +    | | | Signatures + FWSEC Ucode   |                |                          |
>> +    | | | fwsec_sigs(), fwsec_ucode()|                |                          |
>> +    | | +----------------------------+                |                          |
>> +    | +-----------------------------------------------+                          |
>> +    |                                                                            |
>> +    +----------------------------------------------------------------------------+
>> +
>> +Falcon data Lookup
>> +------------------
>> +A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
>> +Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
>> +used to find the required Falcon ucode based on an application ID.
>> +
>> +The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
>> +tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the
>> +offset of the same from the start of the VBIOS image. Unfortunately, the offset
>> +does not account for the EFI image located between the PciAt and FwSec images.
>> +The `vbios.rs` code compensates for this with appropriate arithmetic.
>> +
>> +.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
> 
> The rest looks good.
> 
> Thanks.
> 

Re: [PATCH v3 6/7] Documentation: gpu: nova-core: Document fwsec operation and layout

[Joel Fernandes]

On 7/2/2025 8:25 PM, Bagas Sanjaya wrote:
> On Wed, Jul 02, 2025 at 08:00:43PM +0900, Alexandre Courbot wrote:
>> +FWSEC Memory Layout
>> +-------------------
>> +The memory layout of the FWSEC image is as follows (this is using an GA-102
>> +Ampere GPU as an example and could vary for future GPUs and is subject to change
>> +completely, it is just provided as an example):
>> +
>> +Here is a block diagram of the FWSEC memory layout::
> 
> Above sentence is redundant (as it is implied by preceding paragraph).
Sure agreed, the "Here is" sentence can be deleted.

thanks,

 - Joel

Re: [PATCH v3 1/7] gpu: nova-core: Add code comments related to devinit

[Miguel Ojeda]

On Wed, Jul 2, 2025 at 1:04 PM Alexandre Courbot <acourbot@nvidia.com> wrote:
>
> +        // Check that FWSEC has lowered its protection level before reading the GFW_BOOT status.

`GFW_BOOT` (another one below), since it is written like that above.

> -/* PGC6 */
> +/*
> + * PGC6 register space.
> + *
> + * `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except
> + * for power rails needed to keep self-refresh working and important registers and hardware
> + * blocks).
> + *
> + * These scratch registers remain powered on even in a low-power state and have a designated group
> + * number.
> + */

I noticed the file uses `/*` in a couple files -- could `//` be used
or there is a reason for it?

(I guess maybe in a different series, since it is already there for
PMC in mainline. It could be a good first issue.)

The idea is to only use `/*` when `//` cannot be used, e.g. within
code, so that they have a different purpose.

Cheers,
Miguel

Re: [PATCH v3 2/7] gpu: nova-core: Clarify sysmembar operations

[Miguel Ojeda]

On Wed, Jul 2, 2025 at 1:05 PM Alexandre Courbot <acourbot@nvidia.com> wrote:
>
> +/// driver initialization, and before any falcon is reset.
> +//
>  /// Users are responsible for manually calling [`Self::unregister`] before dropping this object,

Missing slash.

Added the possibility of catching this automatically in e.g. `rustdoc` to:

    https://github.com/Rust-for-Linux/linux/issues/350

> +// These two registers together hold the physical system memory address that is used by the GPU for
> +// perform sysmembar operation (see [`crate::fb::SysmemFlush`]).

This is a normal comment, so the intra-doc link will (sadly) not do anything.

Cheers,
Miguel

Re: [PATCH v3 3/7] gpu: nova-core: Clarify falcon code

[Miguel Ojeda]

On Wed, Jul 2, 2025 at 1:32 PM Alexandre Courbot <acourbot@nvidia.com> wrote:
>
> +/// Security mode of the Falcon microprocessor.
> +/// See falcon.rst for more details.

Please add a newline between those -- the first paragraph is a title
("short description"). A few more instances below, too.

`falcon.rst`

> +    /// Local Framebuffer (GPU's VRAM memory)

Missing period at the end.

Cheers,
Miguel

Re: [PATCH v3 1/7] gpu: nova-core: Add code comments related to devinit

[Alexandre Courbot]

On Sun Jul 6, 2025 at 10:42 PM JST, Miguel Ojeda wrote:
> On Wed, Jul 2, 2025 at 1:04 PM Alexandre Courbot <acourbot@nvidia.com> wrote:
>>
>> +        // Check that FWSEC has lowered its protection level before reading the GFW_BOOT status.
>
> `GFW_BOOT` (another one below), since it is written like that above.
>
>> -/* PGC6 */
>> +/*
>> + * PGC6 register space.
>> + *
>> + * `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except
>> + * for power rails needed to keep self-refresh working and important registers and hardware
>> + * blocks).
>> + *
>> + * These scratch registers remain powered on even in a low-power state and have a designated group
>> + * number.
>> + */
>
> I noticed the file uses `/*` in a couple files -- could `//` be used
> or there is a reason for it?
>
> (I guess maybe in a different series, since it is already there for
> PMC in mainline. It could be a good first issue.)
>
> The idea is to only use `/*` when `//` cannot be used, e.g. within
> code, so that they have a different purpose.

I cannot think of a good reason to not use //, let me fix it here before
this gets merged as I cannot find any other instance in nova-core
outside of this series.

Re: [PATCH v3 2/7] gpu: nova-core: Clarify sysmembar operations

[Miguel Ojeda]

On Sun, Jul 6, 2025 at 3:46 PM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> Added the possibility of catching this automatically in e.g. `rustdoc` to:
>
>     https://github.com/Rust-for-Linux/linux/issues/350

Filled: https://github.com/rust-lang/rust-clippy/issues/15245 so that
they have it upstream.

Cheers,
Miguel