* [PATCH v4 1/8] gpu: nova-core: Add code comments related to devinit
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 2/8] gpu: nova-core: Clarify sysmembar operations Alexandre Courbot
` (7 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
From: Joel Fernandes <joelagnelf@nvidia.com>
Add several code comments to reduce acronym soup and explain how devinit
magic and bootflow works before driver loads. These are essential for
debug and development of the nova driver.
[acourbot@nvidia.com: reformat and reword a couple of sentences]
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/gfw.rs | 39 +++++++++++++++++++++++++++++++++++----
drivers/gpu/nova-core/regs.rs | 14 ++++++++++++--
2 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/gfw.rs b/drivers/gpu/nova-core/gfw.rs
index d5b68e02d405750b18d634d772f15f413453e80d..8ac1ed18719926493369c2aae9a59b2b55fa2b12 100644
--- a/drivers/gpu/nova-core/gfw.rs
+++ b/drivers/gpu/nova-core/gfw.rs
@@ -1,10 +1,22 @@
// SPDX-License-Identifier: GPL-2.0
-//! GPU Firmware (GFW) support.
+//! GPU Firmware (`GFW`) support, a.k.a `devinit`.
//!
//! Upon reset, the GPU runs some firmware code from the BIOS to setup its core parameters. Most of
//! the GPU is considered unusable until this step is completed, so we must wait on it before
//! performing driver initialization.
+//!
+//! A clarification about devinit terminology: devinit is a sequence of register read/writes after
+//! reset that performs tasks such as:
+//! 1. Programming VRAM memory controller timings.
+//! 2. Power sequencing.
+//! 3. Clock and PLL configuration.
+//! 4. Thermal management.
+//!
+//! devinit itself is a 'script' which is interpreted by an interpreter program typically running
+//! on the PMU microcontroller.
+//!
+//! Note that the devinit sequence also needs to run during suspend/resume.
use kernel::bindings;
use kernel::prelude::*;
@@ -14,13 +26,32 @@
use crate::regs;
use crate::util;
-/// Wait until `GFW` (GPU Firmware) completes, or a 4 seconds timeout elapses.
+/// Wait for the `GFW` (GPU firmware) boot completion signal (`GFW_BOOT`), or a 4 seconds timeout.
+///
+/// Upon GPU reset, several microcontrollers (such as PMU, SEC2, GSP etc) run some firmware code to
+/// setup its core parameters. Most of the GPU is considered unusable until this step is completed,
+/// so it must be waited on very early during driver initialization.
+///
+/// The `GFW` code includes several components that need to execute before the driver loads. These
+/// components are located in the VBIOS ROM and executed in a sequence on these different
+/// microcontrollers. The devinit sequence typically runs on the PMU, and the FWSEC runs on the
+/// GSP.
+///
+/// This function waits for a signal indicating that core initialization is complete. Before this
+/// signal is received, little can be done with the GPU. This signal is set by the FWSEC running on
+/// the GSP in Heavy-secured mode.
pub(crate) fn wait_gfw_boot_completion(bar: &Bar0) -> Result {
+ // Before accessing the completion status in `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05`, we must
+ // first check `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK`. This is because
+ // `NV_PGC6_AON_SECURE_SCRATCH_GROUP_05` becomes accessible only after the secure firmware
+ // (FWSEC) lowers the privilege level to allow CPU (LS/Light-secured) access. We can only
+ // safely read the status register from CPU (LS/Light-secured) once the mask indicates
+ // that the privilege level has been lowered.
+ //
// TIMEOUT: arbitrarily large value. GFW starts running immediately after the GPU is put out of
// reset, and should complete in less time than that.
util::wait_on(Delta::from_secs(4), || {
- // Check that FWSEC has lowered its protection level before reading the GFW_BOOT
- // status.
+ // Check that FWSEC has lowered its protection level before reading the GFW_BOOT status.
let gfw_booted = regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK::read(bar)
.read_protection_level0()
&& regs::NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_0_GFW_BOOT::read(bar).completed();
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index e8b8aabce3f36abe6a7fba3e11f677e36baa3897..3bb38197a890bb32d54b9aa4df4d9ebd740dccca 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -104,9 +104,19 @@ pub(crate) fn higher_bound(self) -> u64 {
}
}
-/* PGC6 */
+// PGC6 register space.
+//
+// `GC6` is a GPU low-power state where VRAM is in self-refresh and the GPU is powered down (except
+// for power rails needed to keep self-refresh working and important registers and hardware
+// blocks).
+//
+// These scratch registers remain powered on even in a low-power state and have a designated group
+// number.
-register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128 {
+// Privilege level mask register. It dictates whether the host CPU has privilege to access the
+// `PGC6_AON_SECURE_SCRATCH_GROUP_05` register (which it needs to read GFW_BOOT).
+register!(NV_PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK @ 0x00118128,
+ "Privilege level mask register" {
0:0 read_protection_level0 as bool, "Set after FWSEC lowers its protection level";
});
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 2/8] gpu: nova-core: Clarify sysmembar operations
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 1/8] gpu: nova-core: Add code comments related to devinit Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 3/8] gpu: nova-core: Clarify falcon code Alexandre Courbot
` (6 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
From: Joel Fernandes <joelagnelf@nvidia.com>
sysmembar is a critical operation that the GSP falcon needs to perform
in the reset sequence. Add some code comments to clarify.
[acourbot@nvdidia.com: move relevant documentation to SysmemFlush type]
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/fb.rs | 10 ++++++++++
drivers/gpu/nova-core/gpu.rs | 3 +--
drivers/gpu/nova-core/regs.rs | 3 +++
3 files changed, 14 insertions(+), 2 deletions(-)
diff --git a/drivers/gpu/nova-core/fb.rs b/drivers/gpu/nova-core/fb.rs
index 172b4a12ba2afc05860cc004fd1f0154402f467a..4a702525fff4f394b75fcf54145ba78e34a1a539 100644
--- a/drivers/gpu/nova-core/fb.rs
+++ b/drivers/gpu/nova-core/fb.rs
@@ -17,6 +17,16 @@
/// Type holding the sysmem flush memory page, a page of memory to be written into the
/// `NV_PFB_NISO_FLUSH_SYSMEM_ADDR*` registers and used to maintain memory coherency.
///
+/// A system memory page is required for `sysmembar`, which is a GPU-initiated hardware
+/// memory-barrier operation that flushes all pending GPU-side memory writes that were done through
+/// PCIE to system memory. It is required for falcons to be reset as the reset operation involves a
+/// reset handshake. When the falcon acknowledges a reset, it writes into system memory. To ensure
+/// this write is visible to the host and prevent driver timeouts, the falcon must perform a
+/// sysmembar operation to flush its writes.
+///
+/// Because of this, the sysmem flush memory page must be registered as early as possible during
+/// driver initialization, and before any falcon is reset.
+///
/// Users are responsible for manually calling [`Self::unregister`] before dropping this object,
/// otherwise the GPU might still use it even after it has been freed.
pub(crate) struct SysmemFlush {
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 8e32af16b669ca773e63e184d34c3e0427bc9b76..72d40b0124f0c1a2a381484172c289af523511df 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -170,7 +170,7 @@ pub(crate) struct Gpu {
bar: Devres<Bar0>,
fw: Firmware,
/// System memory page required for flushing all pending GPU-side memory writes done through
- /// PCIE into system memory.
+ /// PCIE into system memory, via sysmembar (A GPU-initiated HW memory-barrier operation).
sysmem_flush: SysmemFlush,
}
@@ -283,7 +283,6 @@ pub(crate) fn new(
gfw::wait_gfw_boot_completion(bar)
.inspect_err(|_| dev_err!(pdev.as_ref(), "GFW boot did not complete"))?;
- // System memory page required for sysmembar to properly flush into system memory.
let sysmem_flush = SysmemFlush::register(pdev.as_ref(), bar, spec.chipset)?;
let gsp_falcon = Falcon::<Gsp>::new(
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index 3bb38197a890bb32d54b9aa4df4d9ebd740dccca..b934ffe8e81390b36f5a39af39a9b9f337aa66bf 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -51,6 +51,9 @@ pub(crate) fn chipset(self) -> Result<Chipset> {
/* PFB */
+// These two registers together hold the physical system memory address that is used by the GPU for
+// perform sysmembar operation (see `fb::SysmemFlush`).
+
register!(NV_PFB_NISO_FLUSH_SYSMEM_ADDR @ 0x00100c10 {
31:0 adr_39_08 as u32;
});
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 3/8] gpu: nova-core: Clarify falcon code
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 1/8] gpu: nova-core: Add code comments related to devinit Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 2/8] gpu: nova-core: Clarify sysmembar operations Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 4/8] gpu: nova-core: convert `/*` comments to `//` Alexandre Courbot
` (5 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
From: Joel Fernandes <joelagnelf@nvidia.com>
Add documentation strings, comments and AES mode for completeness
to the Falcon signatures.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/falcon.rs | 29 ++++++++++++++++++++++-------
1 file changed, 22 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/nova-core/falcon.rs b/drivers/gpu/nova-core/falcon.rs
index c2c6f9eb380ab390befe5af1b3c5df260ccd7595..50437c67c14a89b6974a121d4408efbcdcb3fdd0 100644
--- a/drivers/gpu/nova-core/falcon.rs
+++ b/drivers/gpu/nova-core/falcon.rs
@@ -105,14 +105,23 @@ fn try_from(value: u8) -> Result<Self> {
/// register.
#[repr(u8)]
#[derive(Debug, Default, Copy, Clone)]
+/// Security mode of the Falcon microprocessor.
+///
+/// See `falcon.rst` for more details.
pub(crate) enum FalconSecurityModel {
/// Non-Secure: runs unsigned code without privileges.
#[default]
None = 0,
- /// Low-Secure: runs code with some privileges. Can only be entered from `Heavy` mode, which
- /// will typically validate the LS code through some signature.
+ /// Light-Secured (LS): Runs signed code with some privileges.
+ /// Entry into this mode is only possible from 'Heavy-secure' mode, which verifies the code's
+ /// signature.
+ ///
+ /// Also known as Low-Secure, Privilege Level 2 or PL2.
Light = 2,
- /// High-Secure: runs signed code with full privileges. Signature is validated by boot ROM.
+ /// Heavy-Secured (HS): Runs signed code with full privileges.
+ /// The code's signature is verified by the Falcon Boot ROM (BROM).
+ ///
+ /// Also known as High-Secure, Privilege Level 3 or PL3.
Heavy = 3,
}
impl_from_enum_to_u32!(FalconSecurityModel);
@@ -136,10 +145,13 @@ fn try_from(value: u8) -> Result<Self> {
}
/// Signing algorithm for a given firmware, used in the [`crate::regs::NV_PFALCON2_FALCON_MOD_SEL`]
-/// register.
+/// register. It is passed to the Falcon Boot ROM (BROM) as a parameter.
#[repr(u8)]
#[derive(Debug, Default, Copy, Clone, PartialEq, Eq)]
pub(crate) enum FalconModSelAlgo {
+ /// AES.
+ #[expect(dead_code)]
+ Aes = 0,
/// RSA3K.
#[default]
Rsa3k = 1,
@@ -209,15 +221,18 @@ pub(crate) enum FalconMem {
Dmem,
}
-/// Target/source of a DMA transfer to/from falcon memory.
+/// Defines the Framebuffer Interface (FBIF) aperture type.
+/// This determines the memory type for external memory access during a DMA transfer, which is
+/// performed by the Falcon's Framebuffer DMA (FBDMA) engine. See falcon.rst for more details.
#[derive(Debug, Clone, Default)]
pub(crate) enum FalconFbifTarget {
/// VRAM.
#[default]
+ /// Local Framebuffer (GPU's VRAM memory).
LocalFb = 0,
- /// Coherent system memory.
+ /// Coherent system memory (System DRAM).
CoherentSysmem = 1,
- /// Non-coherent system memory.
+ /// Non-coherent system memory (System DRAM).
NoncoherentSysmem = 2,
}
impl_from_enum_to_u32!(FalconFbifTarget);
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 4/8] gpu: nova-core: convert `/*` comments to `//`
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (2 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 3/8] gpu: nova-core: Clarify falcon code Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 5/8] Documentation: gpu: nova-core: Document vbios layout Alexandre Courbot
` (4 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
The second form is preferred, and there was no reason to use the first.
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
drivers/gpu/nova-core/regs.rs | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index b934ffe8e81390b36f5a39af39a9b9f337aa66bf..6f774d4f79a5eb253df90e335d05cf6bc1fa5857 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -14,7 +14,7 @@
use crate::gpu::{Architecture, Chipset};
use kernel::prelude::*;
-/* PMC */
+// PMC
register!(NV_PMC_BOOT_0 @ 0x00000000, "Basic revision information about the GPU" {
3:0 minor_revision as u8, "Minor revision of the chip";
@@ -42,14 +42,14 @@ pub(crate) fn chipset(self) -> Result<Chipset> {
}
}
-/* PBUS */
+// PBUS
// TODO[REGA]: this is an array of registers.
register!(NV_PBUS_SW_SCRATCH_0E@0x00001438 {
31:16 frts_err_code as u16;
});
-/* PFB */
+// PFB
// These two registers together hold the physical system memory address that is used by the GPU for
// perform sysmembar operation (see `fb::SysmemFlush`).
@@ -160,7 +160,7 @@ pub(crate) fn usable_fb_size(self) -> u64 {
}
}
-/* PDISP */
+// PDISP
register!(NV_PDISP_VGA_WORKSPACE_BASE @ 0x00625f04 {
3:3 status_valid as bool, "Set if the `addr` field is valid";
@@ -178,7 +178,7 @@ pub(crate) fn vga_workspace_addr(self) -> Option<u64> {
}
}
-/* FUSE */
+// FUSE
register!(NV_FUSE_OPT_FPF_NVDEC_UCODE1_VERSION @ 0x00824100 {
15:0 data as u16;
@@ -192,7 +192,7 @@ pub(crate) fn vga_workspace_addr(self) -> Option<u64> {
15:0 data as u16;
});
-/* PFALCON */
+// PFALCON
register!(NV_PFALCON_FALCON_IRQSCLR @ +0x00000004 {
4:4 halt as bool;
@@ -312,7 +312,7 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
31:0 value as u32;
});
-/* PRISCV */
+// PRISCV
register!(NV_PRISCV_RISCV_BCR_CTRL @ +0x00001668 {
0:0 valid as bool;
@@ -324,7 +324,7 @@ pub(crate) fn mem_scrubbing_done(self) -> bool {
// only be used in HAL modules.
pub(crate) mod gm107 {
- /* FUSE */
+ // FUSE
register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00021c04 {
0:0 display_disabled as bool;
@@ -332,7 +332,7 @@ pub(crate) mod gm107 {
}
pub(crate) mod ga100 {
- /* FUSE */
+ // FUSE
register!(NV_FUSE_STATUS_OPT_DISPLAY @ 0x00820c04 {
0:0 display_disabled as bool;
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 5/8] Documentation: gpu: nova-core: Document vbios layout
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (3 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 4/8] gpu: nova-core: convert `/*` comments to `//` Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 6/8] Documentation: gpu: nova-core: Document devinit process Alexandre Courbot
` (3 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
From: Joel Fernandes <joelagnelf@nvidia.com>
Add detailed explanation and block diagrams of the layout of the vBIOS
on Nvidia GPUs. This is important to understand how nova-core boots an
Nvidia GPU.
[ Applied Timur Tabi's feedback on providing link to BIT documentation. ]
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Documentation/gpu/nova/core/vbios.rst | 181 ++++++++++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 182 insertions(+)
diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
new file mode 100644
index 0000000000000000000000000000000000000000..efd40087480c4fd6cc8ef10ddfaa9f0297417e5b
--- /dev/null
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -0,0 +1,181 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+==========
+VBIOS
+==========
+This document describes the layout of the VBIOS image which is a series of concatenated
+images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
+by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
+bootstrap various microcontrollers (PMU, SEC, GSP) with critical initialization before
+the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
+
+The format of the images in the ROM follow the "BIOS Specification" part of the
+PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
+are the ones that contain Falcon ucode and what we are mainly looking for.
+
+As an example, the following are the different image types that can be found in the
+VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
+
+- PciAt Image (Type 0x00) - This is the standard PCI BIOS image, whose name
+ likely comes from the "IBM PC/AT" architecture.
+
+- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
+ driver that is used to display UEFI graphics output.
+
+- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
+
+- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
+ contains various microcodes (also known as an applications) that do a range
+ of different functions. The FWSEC ucode is run in heavy-secure mode and
+ typically runs directly on the GSP (it could be running on a different
+ designated processor in future generations but as of Ampere, it is the GSP).
+ This firmware then loads other firmware ucodes onto the PMU and SEC2
+ microcontrollers for gfw initialization after GPU reset and before the driver
+ loads (see devinit.rst). The DEVINIT ucode is itself another ucode that is
+ stored in this ROM partition.
+
+Once located, the Falcon ucodes have "Application Interfaces" in their data
+memory (DMEM). For FWSEC, the application interface we use for FWSEC is the
+"DMEM mapper" interface which is configured to run the "FRTS" command. This
+command carves out the WPR2 (Write-Protected Region) in VRAM. It then places
+important power-management data, called 'FRTS', into this region. The WPR2
+region is only accessible to heavy-secure ucode.
+
+.. note::
+ It is not clear why FwSec has 2 different partitions in the ROM, but they both
+ are of type 0xE0 and can be identified as such. This could be subject to change
+ in future generations.
+
+VBIOS ROM Layout
+----------------
+The VBIOS layout is roughly a series of concatenated images laid out as follows::
+
+ +----------------------------------------------------------------------------+
+ | VBIOS (Starting at ROM_OFFSET: 0x300000) |
+ +----------------------------------------------------------------------------+
+ | +-----------------------------------------------+ |
+ | | PciAt Image (Type 0x00) | |
+ | +-----------------------------------------------+ |
+ | | +-------------------+ | |
+ | | | ROM Header | | |
+ | | | (Signature 0xAA55)| | |
+ | | +-------------------+ | |
+ | | | rom header's pci_data_struct_offset | |
+ | | | points to the PCIR structure | |
+ | | V | |
+ | | +-------------------+ | |
+ | | | PCIR Structure | | |
+ | | | (Signature "PCIR")| | |
+ | | | last_image: 0x80 | | |
+ | | | image_len: size | | |
+ | | | in 512-byte units | | |
+ | | +-------------------+ | |
+ | | | | |
+ | | | NPDE immediately follows PCIR | |
+ | | V | |
+ | | +-------------------+ | |
+ | | | NPDE Structure | | |
+ | | | (Signature "NPDE")| | |
+ | | | last_image: 0x00 | | |
+ | | +-------------------+ | |
+ | | | |
+ | | +-------------------+ | |
+ | | | BIT Header | (Signature scanning | |
+ | | | (Signature "BIT") | provides the location | |
+ | | +-------------------+ of the BIT table) | |
+ | | | header is | |
+ | | | followed by a table of tokens | |
+ | | V one of which is for falcon data. | |
+ | | +-------------------+ | |
+ | | | BIT Tokens | | |
+ | | | ______________ | | |
+ | | | | Falcon Data | | | |
+ | | | | Token (0x70)|---+------------>------------+--+ |
+ | | | +-------------+ | falcon_data_ptr() | | |
+ | | +-------------------+ | V |
+ | +-----------------------------------------------+ | |
+ | (no gap between images) | |
+ | +-----------------------------------------------+ | |
+ | | EFI Image (Type 0x03) | | |
+ | +-----------------------------------------------+ | |
+ | | Contains the UEFI GOP driver (Graphics Output)| | |
+ | | +-------------------+ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | +-------------------+ | | |
+ | | | Image data | | | |
+ | | +-------------------+ | | |
+ | +-----------------------------------------------+ | |
+ | (no gap between images) | |
+ | +-----------------------------------------------+ | |
+ | | First FwSec Image (Type 0xE0) | | |
+ | +-----------------------------------------------+ | |
+ | | +-------------------+ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | +-------------------+ | | |
+ | | | Image data | | | |
+ | | +-------------------+ | | |
+ | +-----------------------------------------------+ | |
+ | (no gap between images) | |
+ | +-----------------------------------------------+ | |
+ | | Second FwSec Image (Type 0xE0) | | |
+ | +-----------------------------------------------+ | |
+ | | +-------------------+ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | +-------------------+ | | |
+ | | | | |
+ | | +-------------------+ | | |
+ | | | PMU Lookup Table | <- falcon_data_offset <----+ |
+ | | | +-------------+ | pmu_lookup_table | |
+ | | | | Entry 0x85 | | | |
+ | | | | FWSEC_PROD | | | |
+ | | | +-------------+ | | |
+ | | +-------------------+ | |
+ | | | | |
+ | | | points to | |
+ | | V | |
+ | | +-------------------+ | |
+ | | | FalconUCodeDescV3 | <- falcon_ucode_offset | |
+ | | | (FWSEC Firmware) | fwsec_header() | |
+ | | +-------------------+ | |
+ | | | immediately followed by... | |
+ | | V | |
+ | | +----------------------------+ | |
+ | | | Signatures + FWSEC Ucode | | |
+ | | | fwsec_sigs(), fwsec_ucode()| | |
+ | | +----------------------------+ | |
+ | +-----------------------------------------------+ |
+ | |
+ +----------------------------------------------------------------------------+
+
+.. note::
+ This diagram is created based on an GA-102 Ampere GPU as an example and could
+ vary for future or other GPUs.
+
+.. note::
+ For more explanations of acronyms, see the detailed descriptions in `vbios.rs`.
+
+Falcon data Lookup
+------------------
+A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
+Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
+used to find the required Falcon ucode based on an application ID.
+
+The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
+tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the
+offset of the same from the start of the VBIOS image. Unfortunately, the offset
+does not account for the EFI image located between the PciAt and FwSec images.
+The `vbios.rs` code compensates for this with appropriate arithmetic.
+
+.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af358b373d4ef47975d5a80213f3e276..f38041fcd595524b204eabf6ca3aad51038682cf 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,3 +28,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/todo
+ core/vbios
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 6/8] Documentation: gpu: nova-core: Document devinit process
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (4 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 5/8] Documentation: gpu: nova-core: Document vbios layout Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 7/8] Documentation: gpu: nova-core: Document fwsec operation and layout Alexandre Courbot
` (2 subsequent siblings)
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot, Bagas Sanjaya
From: Joel Fernandes <joelagnelf@nvidia.com>
devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Documentation/gpu/nova/core/devinit.rst | 61 +++++++++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 62 insertions(+)
diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 0000000000000000000000000000000000000000..70c819a96a00a0a27846e7e96525470d07721a10
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,61 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==================================
+Device Initialization (devinit)
+==================================
+The devinit process is complex and subject to change. This document provides a high-level
+overview using the Ampere GPU family as an example. The goal is to provide a conceptual
+overview of the process to aid in understanding the corresponding kernel code.
+
+Device initialization (devinit) is a crucial sequence of register read/write operations
+that occur after a GPU reset. The devinit sequence is essential for properly configuring
+the GPU hardware before it can be used.
+
+The devinit engine is an interpreter program that typically runs on the PMU (Power Management
+Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
+commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the
+FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the
+nova-core driver is even loaded. On an Ampere GPU, the devinit ucode is separate from the
+FWSEC ucode. It is launched by FWSEC, which runs on the GSP in 'heavy-secure' mode, while
+devinit runs on the PMU in 'light-secure' mode.
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+
+Low-level Firmware Initialization Flow
+--------------------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
+firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is
+considered unusable until this initialization process completes.
+
+These low-level GPU firmware components are typically:
+
+1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
+2. Executed in sequence on different microcontrollers:
+
+ - The devinit engine typically but not necessarily runs on the PMU.
+ - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
+ heavy-secure mode.
+
+Before the driver can proceed with further initialization, it must wait for a signal
+indicating that core initialization is complete (known as GFW_BOOT). This signal is
+asserted by the FWSEC running on the GSP in heavy-secure mode.
+
+Runtime Considerations
+----------------------
+It's important to note that the devinit sequence also needs to run during suspend/resume
+operations at runtime, not just during initial boot, as it is critical to power management.
+
+Security and Access Control
+---------------------------
+The initialization process involves careful privilege management. For example, before
+accessing certain completion status registers, the driver must check privilege level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
+privilege level to allow CPU (LS/low-secure) access. This is the case, for example,
+when receiving the GFW_BOOT signal.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index f38041fcd595524b204eabf6ca3aad51038682cf..e4e017d926767284b5cee844d8dba32be6bc064c 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -29,3 +29,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/todo
core/vbios
+ core/devinit
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 7/8] Documentation: gpu: nova-core: Document fwsec operation and layout
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (5 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 6/8] Documentation: gpu: nova-core: Document devinit process Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 6:49 ` [PATCH v4 8/8] Documentation: gpu: nova-core: Document basics of the Falcon Alexandre Courbot
2025-07-08 22:36 ` [PATCH v4 0/8] Documentation for nova-core Danilo Krummrich
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot
From: Joel Fernandes <joelagnelf@nvidia.com>
Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 181 ++++++++++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 182 insertions(+)
diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 0000000000000000000000000000000000000000..c440edbe420c39d7f405316a13472f8a13fe1d8d
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,181 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+
+=========================
+FWSEC (Firmware Security)
+=========================
+This document briefly/conceptually describes the FWSEC (Firmware Security) image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC is to provide a secure boot process. It runs in
+'Heavy-secure' mode, and performs firmware verification after a GPU reset
+before loading various ucode images onto other microcontrollers on the GPU,
+such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only needs
+to perform FRTS, since Secure Boot (SB) has already completed by the time the driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which contains
+data required for power management. Once setup, only HS mode ucode can access it
+(see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
+various ucode images (also known as applications) -- one of them being FWSEC. For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All these
+ucode images are stored in the same ROM partition and the PMU table is used to look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the nova-core driver, the FWSEC contains an 'application interface' called
+DMEMMAPPER. This interface is used to execute the 'FWSEC-FRTS' command, among others.
+For Ampere, FWSEC is running on the GSP in Heavy-secure mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows::
+
+ +---------------------------------------------------------------+
+ | FWSEC ROM image (type 0xE0) |
+ | |
+ | +---------------------------------+ |
+ | | PMU Falcon Ucode Table | |
+ | | (PmuLookupTable) | |
+ | | +-------------------------+ | |
+ | | | Table Header | | |
+ | | | - version: 0x01 | | |
+ | | | - header_size: 6 | | |
+ | | | - entry_size: 6 | | |
+ | | | - entry_count: N | | |
+ | | | - desc_version:3(unused)| | |
+ | | +-------------------------+ | |
+ | | ... | |
+ | | +-------------------------+ | |
+ | | | Entry for FWSEC (0x85) | | |
+ | | | (PmuLookupTableEntry) | | |
+ | | | - app_id: 0x85 (FWSEC) |----|----+ |
+ | | | - target_id: 0x01 (PMU) | | | |
+ | | | - data: offset ---------|----|----|---+ look up FWSEC |
+ | | +-------------------------+ | | | |
+ | +---------------------------------+ | | |
+ | | | |
+ | | | |
+ | +---------------------------------+ | | |
+ | | FWSEC Ucode Component |<---+ | |
+ | | (aka Falcon data) | | |
+ | | +-------------------------+ | | |
+ | | | FalconUCodeDescV3 |<---|--------+ |
+ | | | - hdr | | |
+ | | | - stored_size | | |
+ | | | - pkc_data_offset | | |
+ | | | - interface_offset -----|----|----------------+ |
+ | | | - imem_phys_base | | | |
+ | | | - imem_load_size | | | |
+ | | | - imem_virt_base | | | |
+ | | | - dmem_phys_base | | | |
+ | | | - dmem_load_size | | | |
+ | | | - engine_id_mask | | | |
+ | | | - ucode_id | | | |
+ | | | - signature_count | | look up sig | |
+ | | | - signature_versions --------------+ | |
+ | | +-------------------------+ | | | |
+ | | (no gap) | | | |
+ | | +-------------------------+ | | | |
+ | | | Signatures Section |<---|-----+ | |
+ | | | (384 bytes per sig) | | | |
+ | | | - RSA-3K Signature 1 | | | |
+ | | | - RSA-3K Signature 2 | | | |
+ | | | ... | | | |
+ | | +-------------------------+ | | |
+ | | | | |
+ | | +-------------------------+ | | |
+ | | | IMEM Section (Code) | | | |
+ | | | | | | |
+ | | | Contains instruction | | | |
+ | | | code etc. | | | |
+ | | +-------------------------+ | | |
+ | | | | |
+ | | +-------------------------+ | | |
+ | | | DMEM Section (Data) | | | |
+ | | | | | | |
+ | | | +---------------------+ | | | |
+ | | | | Application | |<---|----------------+ |
+ | | | | Interface Table | | | |
+ | | | | (FalconAppifHdrV1) | | | |
+ | | | | Header: | | | |
+ | | | | - version: 0x01 | | | |
+ | | | | - header_size: 4 | | | |
+ | | | | - entry_size: 8 | | | |
+ | | | | - entry_count: N | | | |
+ | | | | | | | |
+ | | | | Entries: | | | |
+ | | | | +-----------------+ | | | |
+ | | | | | DEVINIT (ID 1) | | | | |
+ | | | | | - id: 0x01 | | | | |
+ | | | | | - dmemOffset X -|-|-|----+ |
+ | | | | +-----------------+ | | | |
+ | | | | +-----------------+ | | | |
+ | | | | | DMEMMAPPER(ID 4)| | | | |
+ | | | | | - id: 0x04 | | | | Used only for DevInit |
+ | | | | | (NVFW_FALCON_ | | | | application (not FWSEC) |
+ | | | | | APPIF_ID_DMEMMAPPER) | |
+ | | | | | - dmemOffset Y -|-|-|----|-----+ |
+ | | | | +-----------------+ | | | | |
+ | | | +---------------------+ | | | |
+ | | | | | | |
+ | | | +---------------------+ | | | |
+ | | | | DEVINIT Engine |<|----+ | Used by FWSEC |
+ | | | | Interface | | | | app. |
+ | | | +---------------------+ | | | |
+ | | | | | | |
+ | | | +---------------------+ | | | |
+ | | | | DMEM Mapper (ID 4) |<|----+-----+ |
+ | | | | (FalconAppifDmemmapperV3) | |
+ | | | | - signature: "DMAP" | | | |
+ | | | | - version: 0x0003 | | | |
+ | | | | - Size: 64 bytes | | | |
+ | | | | - cmd_in_buffer_off | |----|------------+ |
+ | | | | - cmd_in_buffer_size| | | | |
+ | | | | - cmd_out_buffer_off| |----|------------|-----+ |
+ | | | | - cmd_out_buffer_sz | | | | | |
+ | | | | - init_cmd | | | | | |
+ | | | | - features | | | | | |
+ | | | | - cmd_mask0/1 | | | | | |
+ | | | +---------------------+ | | | | |
+ | | | | | | | |
+ | | | +---------------------+ | | | | |
+ | | | | Command Input Buffer|<|----|------------+ | |
+ | | | | - Command data | | | | |
+ | | | | - Arguments | | | | |
+ | | | +---------------------+ | | | |
+ | | | | | | |
+ | | | +---------------------+ | | | |
+ | | | | Command Output |<|----|------------------+ |
+ | | | | Buffer | | | |
+ | | | | - Results | | | |
+ | | | | - Status | | | |
+ | | | +---------------------+ | | |
+ | | +-------------------------+ | |
+ | +---------------------------------+ |
+ | |
+ +---------------------------------------------------------------+
+
+.. note::
+ This is using an GA-102 Ampere GPU as an example and could vary for future GPUs.
+
+.. note::
+ The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
+ (Video Protected Region) initialization as well. Before the nova-core driver is even
+ loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
+ sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
+ GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
+ async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
+ memory allocators need to wait for this scrubbing to complete).
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index e4e017d926767284b5cee844d8dba32be6bc064c..e3650f53ff53113f31f63f67cf26116b5c070693 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -30,3 +30,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/todo
core/vbios
core/devinit
+ core/fwsec
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v4 8/8] Documentation: gpu: nova-core: Document basics of the Falcon
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (6 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 7/8] Documentation: gpu: nova-core: Document fwsec operation and layout Alexandre Courbot
@ 2025-07-08 6:49 ` Alexandre Courbot
2025-07-08 22:36 ` [PATCH v4 0/8] Documentation for nova-core Danilo Krummrich
8 siblings, 0 replies; 11+ messages in thread
From: Alexandre Courbot @ 2025-07-08 6:49 UTC (permalink / raw)
To: Danilo Krummrich, David Airlie, Simona Vetter, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, linux-kernel, linux-doc, rust-for-linux,
Joel Fernandes, Alexandre Courbot, Bagas Sanjaya
From: Joel Fernandes <joelagnelf@nvidia.com>
Instances of the Falcon microcontroller appear in modern Nvidia GPUs and
are crucial to the GPU boot process. Document some concepts which will
make nova-core boot code easier to digest. All the information is
derived from public sources such as public documents, OpenRM and Nouveau
code.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
Reviewed-by: Bagas Sanjaya <bagasdotme@gmail.com>
Signed-off-by: Alexandre Courbot <acourbot@nvidia.com>
---
Documentation/gpu/nova/core/falcon.rst | 158 +++++++++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 159 insertions(+)
diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst
new file mode 100644
index 0000000000000000000000000000000000000000..33137082eb6c14cecda2fbe6fdb79e63ee9ca2e6
--- /dev/null
+++ b/Documentation/gpu/nova/core/falcon.rst
@@ -0,0 +1,158 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+==============================
+Falcon (FAst Logic Controller)
+==============================
+The following sections describe the Falcon core and the ucode running on it.
+The descriptions are based on the Ampere GPU or earlier designs; however, they
+should mostly apply to future designs as well, but everything is subject to
+change. The overview provided here is mainly tailored towards understanding the
+interactions of nova-core driver with the Falcon.
+
+NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
+handle secure firmware tasks, initialization, and power management. Modern
+NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
+processor) and SEC2 (the security engine)) and also may integrate a RISC-V core.
+This core is capable of running both RISC-V and Falcon code.
+
+The code running on the Falcon cores is also called 'ucode', and will be
+referred to as such in the following sections.
+
+Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
+small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from
+system memory. The nova-core driver must reset and configure the Falcon, load
+its firmware via DMA, and start its CPU.
+
+Falcon security levels
+======================
+Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS)
+modes.
+
+Heavy Secured (HS) also known as Privilege Level 3 (PL3)
+--------------------------------------------------------
+HS ucode is the most trusted code and has access to pretty much everything on
+the chip. The HS binary includes a signature in it which is verified at boot.
+This signature verification is done by the hardware itself, thus establishing a
+root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
+GSP in HS mode. FRTS, which involves setting up and loading content into the WPR
+(Write Protect Region), has to be done by the HS ucode and cannot be done by the
+host CPU or LS ucode.
+
+Light Secured (LS or PL2) and Non Secured (NS or PL0)
+-----------------------------------------------------
+These modes are less secure than HS. Like HS, the LS or NS ucode binary also
+typically includes a signature in it. To load firmware in LS or NS mode onto a
+Falcon, another Falcon needs to be running in HS mode, which also establishes the
+root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter"
+ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
+run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
+example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
+devinit engine onto the PMU in LS mode.
+
+Root of trust establishment
+---------------------------
+To establish a root of trust, the code running on a Falcon must be immutable and
+hardwired into a read-only memory (ROM). This follows industry norms for
+verification of firmware. This code is called the Boot ROM (BROM). The nova-core
+driver on the CPU communicates with Falcon's Boot ROM through various Falcon
+registers prefixed with "BROM" (see regs.rs).
+
+After nova-core driver reads the necessary ucode from VBIOS, it programs the
+BROM and DMA registers to trigger the Falcon to load the HS ucode from the system
+memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified
+by the Falcon's Boot ROM.
+
+Once the verified HS code is running on a Falcon, it can verify and load other
+LS/NS ucode binaries onto other Falcons and start them. The process of signature
+verification is the same as HS; just in this case, the hardware (BROM) doesn't
+compute the signature, but the HS ucode does.
+
+The root of trust is therefore established as follows:
+ Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode.
+
+On an Ampere GPU, for example, the boot verification flow is:
+ Hardware (Boot ROM running on the SEC2) ->
+ HS ucode (Booter running on the SEC2) ->
+ LS ucode (GSP-RM running on the GSP)
+
+.. note::
+ While the CPU can load HS ucode onto a Falcon microcontroller and have it
+ verified by the hardware and run, the CPU itself typically does not load
+ LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
+ HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the
+ SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
+ the "SEC2-RTOS" ucode at runtime. This presents a problem: there is no
+ component to load the SEC2-RTOS ucode onto the SEC2. The CPU cannot load
+ LS code, and GSP-RM must run in LS mode. To overcome this, the GSP is
+ temporarily made to run HS ucode (which is itself loaded by the CPU via
+ the nova-core driver using a "GSP-provided sequencer") which then loads
+ the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP then resumes
+ running its own GSP-RM LS ucode.
+
+Falcon memory subsystem and DMA engine
+======================================
+Falcons have separate instruction and data memories (IMEM/DMEM)
+and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
+DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
+(Framebuffer Interface), to external memory.
+
+DMA transfers are possible from the Falcon's memory to both the system memory
+and the framebuffer memory (VRAM).
+
+To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
+is accessed (also known as aperture type). In the nova-core driver, this is
+determined by the `FalconFbifTarget` enum.
+
+The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
+controls access by the FBDMA to the external memory.
+
+Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows::
+
+ External Memory (Framebuffer / System DRAM)
+ ^ |
+ | |
+ | v
+ +-----------------------------------------------------+
+ | | |
+ | +---------------+ | |
+ | | FBIF |-------+ | FALCON
+ | | (FrameBuffer | Memory Interface | PROCESSOR
+ | | InterFace) | |
+ | | Apertures | |
+ | | Configures | |
+ | | mem access | |
+ | +-------^-------+ |
+ | | |
+ | | FBDMA uses configured FBIF apertures |
+ | | to access External Memory
+ | |
+ | +-------v--------+ +---------------+
+ | | FBDMA | cfg | RISC |
+ | | (FrameBuffer |<---->| CORE |----->. Direct Core Access
+ | | DMA Engine) | | | |
+ | | - Master dev. | | (can run both | |
+ | +-------^--------+ | Falcon and | |
+ | | cfg--->| RISC-V code) | |
+ | | / | | |
+ | | | +---------------+ | +------------+
+ | | | | | BROM |
+ | | | <--->| (Boot ROM) |
+ | | / | +------------+
+ | | v |
+ | +---------------+ |
+ | | IO-PMP | Controls access by FBDMA |
+ | | (IO Physical | and other IO Masters |
+ | | Memory Protect) |
+ | +-------^-------+ |
+ | | |
+ | | Protected Access Path for FBDMA |
+ | v |
+ | +---------------------------------------+ |
+ | | Memory | |
+ | | +---------------+ +------------+ | |
+ | | | IMEM | | DMEM | |<-----+
+ | | | (Instruction | | (Data | |
+ | | | Memory) | | Memory) | |
+ | | +---------------+ +------------+ |
+ | +---------------------------------------+
+ +-----------------------------------------------------+
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index e3650f53ff53113f31f63f67cf26116b5c070693..e39cb3163581ea4ff5b441b82e9efa4282c946f8 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -31,3 +31,4 @@ vGPU manager VFIO driver and the nova-drm driver.
core/vbios
core/devinit
core/fwsec
+ core/falcon
--
2.50.0
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v4 0/8] Documentation for nova-core
2025-07-08 6:49 [PATCH v4 0/8] Documentation for nova-core Alexandre Courbot
` (7 preceding siblings ...)
2025-07-08 6:49 ` [PATCH v4 8/8] Documentation: gpu: nova-core: Document basics of the Falcon Alexandre Courbot
@ 2025-07-08 22:36 ` Danilo Krummrich
2025-07-09 1:31 ` Joel Fernandes
8 siblings, 1 reply; 11+ messages in thread
From: Danilo Krummrich @ 2025-07-08 22:36 UTC (permalink / raw)
To: Alexandre Courbot
Cc: David Airlie, Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Jonathan Corbet, nouveau, dri-devel,
linux-kernel, linux-doc, rust-for-linux, Joel Fernandes,
Bagas Sanjaya
On Tue, Jul 08, 2025 at 03:49:40PM +0900, Alexandre Courbot wrote:
> This series adds some documentation that was relevant to the FWSEC-FRTS
> [1] series, but wasn't sent alongside it as it was worked on in
> parallel.
Joel, thanks a lot for this series. It's great to have this documentation --
very much appreciated.
Applied to nova-next, thanks!
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v4 0/8] Documentation for nova-core
2025-07-08 22:36 ` [PATCH v4 0/8] Documentation for nova-core Danilo Krummrich
@ 2025-07-09 1:31 ` Joel Fernandes
0 siblings, 0 replies; 11+ messages in thread
From: Joel Fernandes @ 2025-07-09 1:31 UTC (permalink / raw)
To: Danilo Krummrich, Alexandre Courbot
Cc: David Airlie, Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Jonathan Corbet, nouveau, dri-devel,
linux-kernel, linux-doc, rust-for-linux, Bagas Sanjaya
On 7/8/2025 6:36 PM, Danilo Krummrich wrote:
> On Tue, Jul 08, 2025 at 03:49:40PM +0900, Alexandre Courbot wrote:
>> This series adds some documentation that was relevant to the FWSEC-FRTS
>> [1] series, but wasn't sent alongside it as it was worked on in
>> parallel.
>
> Joel, thanks a lot for this series. It's great to have this documentation --
> very much appreciated.
My pleasure! Thanks! Looking forward to many more. And thanks a lot Alex for
fixing these up and posting.
cheers,
- Joel
>
> Applied to nova-next, thanks!
^ permalink raw reply [flat|nested] 11+ messages in thread