* [PATCH v2 3/7] nova-core: docs: Document vbios layout
[not found] <20250503040802.1411285-1-joelagnelf@nvidia.com>
@ 2025-05-03 4:07 ` Joel Fernandes
2025-05-05 3:00 ` Bagas Sanjaya
2025-05-03 4:07 ` [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout Joel Fernandes
` (2 subsequent siblings)
3 siblings, 1 reply; 13+ messages in thread
From: Joel Fernandes @ 2025-05-03 4:07 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, Joel Fernandes, linux-doc
Add detailed explanation and block diagrams of the layout of the vBIOS
on Nvidia GPUs. This is important to understand how nova-core boots an
Nvidia GPU.
[ Applied Timur Tabi's feedback on providing link to BIT documentation. ]
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/vbios.rst | 177 ++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 178 insertions(+)
create mode 100644 Documentation/gpu/nova/core/vbios.rst
diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
new file mode 100644
index 000000000000..04ced35648cb
--- /dev/null
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -0,0 +1,177 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+==========
+VBIOS
+==========
+This document describes the layout of the VBIOS image which is a series of concatenated
+images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
+by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
+boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
+the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
+
+The format of the images in the ROM follow the "BIOS Specification" part of the
+PCI specification, with Nvidia-specific extensions. The ROM images of type FwSec
+are the ones that contain Falcon ucode and what we are mainly looking for.
+
+As an example, the following are the different image types that can be found in the
+VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
+
+- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's naming likely
+ comes from the "IBM PC/AT" architecture.
+
+- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
+ driver that is used to display UEFI graphics output.
+
+- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
+
+- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware)
+ contains various different microcodes (also known as an applications) that do
+ a range of different functions. The FWSEC ucode is run in heavy-secure mode and
+ typically runs directly on the GSP (it could be running on a different designated
+ processor in future generations but as of Ampere, it is the GSP). This firmware
+ then loads other firmware ucodes onto the PMU and SEC2 microcontrollers for gfw
+ initialization after GPU reset and before the driver loads (see devinit.rst).
+ The DEVINIT ucode is itself another ucode that is stored in this ROM partition.
+
+The Falcon ucodes once located have "Application Interfaces" in the data memory (DMEM)
+of the ucode. For FWSEC, the application interface we use for FWSEC is the "DMEM mapper"
+interface which is configured to run the "FRTS" command. This command performs carving
+out of the WPR2 area (Write protected region) in the VRAM and placing important data
+called 'FRTS' into it which contains power-management data. The WPR2 region is only
+accessible to heavy-secure ucode.
+
+.. note::
+ It is not clear why FwSec has 2 different partitions in the ROM, but they both
+ are of type 0xE0 and can be identified as such. This could be subject to change
+ in future generations.
+
+VBIOS ROM Layout
+----------------
+The VBIOS layout is roughly a series of concatenated images as follows:
+(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
+
+.. note::
+ This diagram is created based on an GA-102 Ampere GPU as an example and could
+ vary for future or other GPUs.
+
+Here is a block diagram of the VBIOS layout::
+
+ ┌────────────────────────────────────────────────────────────────────────┐
+ │ VBIOS (Starting at ROM_OFFSET: 0x300000) │
+ ├────────────────────────────────────────────────────────────────────────┤
+ │ ┌───────────────────────────────────────────────┐ │
+ │ │ PciAt Image (Type 0x00) │ │
+ │ ├───────────────────────────────────────────────┤ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ ROM Header │ │ │
+ │ │ │ (Signature 0xAA55)│ │ │
+ │ │ └───────────────────┘ │ │
+ │ │ │ rom header's pci_data_struct_offset │ │
+ │ │ │ points to the PCIR structure │ │
+ │ │ V │ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ PCIR Structure │ │ │
+ │ │ │ (Signature "PCIR")│ │ │
+ │ │ │ last_image: 0x80 │ │ │
+ │ │ │ image_len: size │ │ │
+ │ │ │ in 512-byte units │ │ │
+ │ │ └───────────────────┘ │ │
+ │ │ │ │ │
+ │ │ │ NPDE immediately follows PCIR │ │
+ │ │ V │ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ NPDE Structure │ │ │
+ │ │ │ (Signature "NPDE")│ │ │
+ │ │ │ last_image: 0x00 │ │ │
+ │ │ └───────────────────┘ │ │
+ │ │ │ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ BIT Header │ (Signature scanning │ │
+ │ │ │ (Signature "BIT") │ provides the location │ │
+ │ │ └───────────────────┘ of the BIT table) │ │
+ │ │ │ header is │ │
+ │ │ | followed by a table of tokens │ │
+ │ │ V one of which is for falcon data. │ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ BIT Tokens │ │ │
+ │ | | ______________ | | |
+ │ │ │ │ Falcon Data │ │ │ │
+ │ │ │ │ Token (0x70)│---+------------>------------┼──+ │
+ │ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
+ │ │ └───────────────────┘ │ V │
+ │ └───────────────────────────────────────────────┘ │ │
+ │ (no gap between images) │ │
+ │ ┌───────────────────────────────────────────────┐ │ │
+ │ │ EFI Image (Type 0x03) │ │ │
+ │ ├───────────────────────────────────────────────┤ │ │
+ │ | Contains the UEFI GOP driver (Graphics Output)| | |
+ │ │ ┌───────────────────┐ │ │ │
+ │ │ │ ROM Header │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ PCIR Structure │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ NPDE Structure │ │ │ │
+ │ │ └───────────────────┘ │ │ │
+ │ │ │ Image data │ │ │ │
+ │ │ └───────────────────┘ │ │ │
+ │ └───────────────────────────────────────────────┘ │ │
+ │ (no gap between images) │ │
+ │ ┌───────────────────────────────────────────────┐ │ │
+ │ │ First FwSec Image (Type 0xE0) │ │ │
+ │ ├───────────────────────────────────────────────┤ │ │
+ │ │ ┌───────────────────┐ │ │ │
+ │ │ │ ROM Header │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ PCIR Structure │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ NPDE Structure │ │ │ │
+ │ │ └───────────────────┘ │ │ │
+ │ │ │ Image data │ │ │ │
+ │ │ └───────────────────┘ │ │ │
+ │ └───────────────────────────────────────────────┘ │ │
+ │ (no gap between images) │ │
+ │ ┌───────────────────────────────────────────────┐ │ │
+ │ │ Second FwSec Image (Type 0xE0) │ │ │
+ │ ├───────────────────────────────────────────────┤ │ │
+ │ │ ┌───────────────────┐ │ │ │
+ │ │ │ ROM Header │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ PCIR Structure │ │ │ │
+ │ │ +───────────────────+ │ │ │
+ │ │ │ NPDE Structure │ │ │ │
+ │ │ └───────────────────┘ │ │ │
+ │ │ │ │ │
+ │ │ ┌───────────────────┐ │ │ │
+ │ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
+ │ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
+ │ │ │ │ Entry 0x85 │ │ │ │
+ │ │ │ │ FWSEC_PROD │ │ │ │
+ │ │ │ └─────────────┘ │ │ │
+ │ │ └───────────────────┘ │ │
+ │ │ │ │ │
+ │ │ │ points to │ │
+ │ │ V │ │
+ │ │ ┌───────────────────┐ │ │
+ │ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
+ │ │ │ (FWSEC Firmware) │ fwsec_header() │ │
+ │ │ └───────────────────┘ │ │
+ │ │ │ immediately followed by... │ │
+ │ │ V │ │
+ │ │ ┌────────────────────────────┐ │ │
+ │ │ │ Signatures + FWSEC Ucode │ │ │
+ │ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
+ │ │ └────────────────────────────┘ │ │
+ │ └───────────────────────────────────────────────┘______________________│
+
+Falcon data Lookup
+------------------
+A key part of the VBIOS extraction code (vbios.rs) is to find the location of the
+Falcon data in the VBIOS which contains the PMU lookup table. This lookup table is
+used to find the required Falcon ucode based on an application ID.
+
+The location of the PMU lookup table is found by scanning the BIT (`BIOS Information Table`_)
+tokens for a token with the id `BIT_TOKEN_ID_FALCON_DATA` (0x70) which indicates the
+offset of the same from the start of the VBIOS image. Unfortunately, the offset
+does not account for the EFI image that sits between the PciAt image and the FwSec
+images, the vbios.rs code compensates for that with appropriate arithmetic.
+
+.. _`BIOS Information Table`: https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af35..91cc802ed94f 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,4 +27,5 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/vbios
core/todo
--
2.43.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 3/7] nova-core: docs: Document vbios layout
2025-05-03 4:07 ` [PATCH v2 3/7] nova-core: docs: Document vbios layout Joel Fernandes
@ 2025-05-05 3:00 ` Bagas Sanjaya
2025-05-05 3:12 ` Bagas Sanjaya
0 siblings, 1 reply; 13+ messages in thread
From: Bagas Sanjaya @ 2025-05-05 3:00 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, David Airlie, Simona Vetter,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
[-- Attachment #1: Type: text/plain, Size: 34082 bytes --]
On Sat, May 03, 2025 at 12:07:55AM -0400, Joel Fernandes wrote:
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +==========
> +VBIOS
> +==========
Separate SPDX line and title heading.
> +- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's naming likely
> + comes from the "IBM PC/AT" architecture.
Combine these two lines on the same bullet list item.
> +Here is a block diagram of the VBIOS layout::
> +
> + ┌────────────────────────────────────────────────────────────────────────┐
> + │ VBIOS (Starting at ROM_OFFSET: 0x300000) │
> + ├────────────────────────────────────────────────────────────────────────┤
> + │ ┌───────────────────────────────────────────────┐ │
> + │ │ PciAt Image (Type 0x00) │ │
> + │ ├───────────────────────────────────────────────┤ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ ROM Header │ │ │
> + │ │ │ (Signature 0xAA55)│ │ │
> + │ │ └───────────────────┘ │ │
> + │ │ │ rom header's pci_data_struct_offset │ │
> + │ │ │ points to the PCIR structure │ │
> + │ │ V │ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ PCIR Structure │ │ │
> + │ │ │ (Signature "PCIR")│ │ │
> + │ │ │ last_image: 0x80 │ │ │
> + │ │ │ image_len: size │ │ │
> + │ │ │ in 512-byte units │ │ │
> + │ │ └───────────────────┘ │ │
> + │ │ │ │ │
> + │ │ │ NPDE immediately follows PCIR │ │
> + │ │ V │ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ NPDE Structure │ │ │
> + │ │ │ (Signature "NPDE")│ │ │
> + │ │ │ last_image: 0x00 │ │ │
> + │ │ └───────────────────┘ │ │
> + │ │ │ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ BIT Header │ (Signature scanning │ │
> + │ │ │ (Signature "BIT") │ provides the location │ │
> + │ │ └───────────────────┘ of the BIT table) │ │
> + │ │ │ header is │ │
> + │ │ | followed by a table of tokens │ │
> + │ │ V one of which is for falcon data. │ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ BIT Tokens │ │ │
> + │ | | ______________ | | |
> + │ │ │ │ Falcon Data │ │ │ │
> + │ │ │ │ Token (0x70)│---+------------>------------┼──+ │
> + │ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
> + │ │ └───────────────────┘ │ V │
> + │ └───────────────────────────────────────────────┘ │ │
> + │ (no gap between images) │ │
> + │ ┌───────────────────────────────────────────────┐ │ │
> + │ │ EFI Image (Type 0x03) │ │ │
> + │ ├───────────────────────────────────────────────┤ │ │
> + │ | Contains the UEFI GOP driver (Graphics Output)| | |
> + │ │ ┌───────────────────┐ │ │ │
> + │ │ │ ROM Header │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ PCIR Structure │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ NPDE Structure │ │ │ │
> + │ │ └───────────────────┘ │ │ │
> + │ │ │ Image data │ │ │ │
> + │ │ └───────────────────┘ │ │ │
> + │ └───────────────────────────────────────────────┘ │ │
> + │ (no gap between images) │ │
> + │ ┌───────────────────────────────────────────────┐ │ │
> + │ │ First FwSec Image (Type 0xE0) │ │ │
> + │ ├───────────────────────────────────────────────┤ │ │
> + │ │ ┌───────────────────┐ │ │ │
> + │ │ │ ROM Header │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ PCIR Structure │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ NPDE Structure │ │ │ │
> + │ │ └───────────────────┘ │ │ │
> + │ │ │ Image data │ │ │ │
> + │ │ └───────────────────┘ │ │ │
> + │ └───────────────────────────────────────────────┘ │ │
> + │ (no gap between images) │ │
> + │ ┌───────────────────────────────────────────────┐ │ │
> + │ │ Second FwSec Image (Type 0xE0) │ │ │
> + │ ├───────────────────────────────────────────────┤ │ │
> + │ │ ┌───────────────────┐ │ │ │
> + │ │ │ ROM Header │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ PCIR Structure │ │ │ │
> + │ │ +───────────────────+ │ │ │
> + │ │ │ NPDE Structure │ │ │ │
> + │ │ └───────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌───────────────────┐ │ │ │
> + │ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
> + │ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
> + │ │ │ │ Entry 0x85 │ │ │ │
> + │ │ │ │ FWSEC_PROD │ │ │ │
> + │ │ │ └─────────────┘ │ │ │
> + │ │ └───────────────────┘ │ │
> + │ │ │ │ │
> + │ │ │ points to │ │
> + │ │ V │ │
> + │ │ ┌───────────────────┐ │ │
> + │ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
> + │ │ │ (FWSEC Firmware) │ fwsec_header() │ │
> + │ │ └───────────────────┘ │ │
> + │ │ │ immediately followed by... │ │
> + │ │ V │ │
> + │ │ ┌────────────────────────────┐ │ │
> + │ │ │ Signatures + FWSEC Ucode │ │ │
> + │ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
> + │ │ └────────────────────────────┘ │ │
> + │ └───────────────────────────────────────────────┘______________________│
Diagram borders look messy in htmldocs output (due to Unicode characters
─ and │), so I use ASCII dash and vertical bar instead:
---- >8 ----
diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
index dd6ac891e5f1d0..c68ef0e7b70124 100644
--- a/Documentation/gpu/nova/core/vbios.rst
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -56,112 +56,113 @@ The VBIOS layout is roughly a series of concatenated images as follows:
Here is a block diagram of the VBIOS layout::
- ┌────────────────────────────────────────────────────────────────────────┐
- │ VBIOS (Starting at ROM_OFFSET: 0x300000) │
- ├────────────────────────────────────────────────────────────────────────┤
- │ ┌───────────────────────────────────────────────┐ │
- │ │ PciAt Image (Type 0x00) │ │
- │ ├───────────────────────────────────────────────┤ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ ROM Header │ │ │
- │ │ │ (Signature 0xAA55)│ │ │
- │ │ └───────────────────┘ │ │
- │ │ │ rom header's pci_data_struct_offset │ │
- │ │ │ points to the PCIR structure │ │
- │ │ V │ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ PCIR Structure │ │ │
- │ │ │ (Signature "PCIR")│ │ │
- │ │ │ last_image: 0x80 │ │ │
- │ │ │ image_len: size │ │ │
- │ │ │ in 512-byte units │ │ │
- │ │ └───────────────────┘ │ │
- │ │ │ │ │
- │ │ │ NPDE immediately follows PCIR │ │
- │ │ V │ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ NPDE Structure │ │ │
- │ │ │ (Signature "NPDE")│ │ │
- │ │ │ last_image: 0x00 │ │ │
- │ │ └───────────────────┘ │ │
- │ │ │ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ BIT Header │ (Signature scanning │ │
- │ │ │ (Signature "BIT") │ provides the location │ │
- │ │ └───────────────────┘ of the BIT table) │ │
- │ │ │ header is │ │
- │ │ | followed by a table of tokens │ │
- │ │ V one of which is for falcon data. │ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ BIT Tokens │ │ │
- │ | | ______________ | | |
- │ │ │ │ Falcon Data │ │ │ │
- │ │ │ │ Token (0x70)│---+------------>------------┼──+ │
- │ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
- │ │ └───────────────────┘ │ V │
- │ └───────────────────────────────────────────────┘ │ │
- │ (no gap between images) │ │
- │ ┌───────────────────────────────────────────────┐ │ │
- │ │ EFI Image (Type 0x03) │ │ │
- │ ├───────────────────────────────────────────────┤ │ │
- │ | Contains the UEFI GOP driver (Graphics Output)| | |
- │ │ ┌───────────────────┐ │ │ │
- │ │ │ ROM Header │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ PCIR Structure │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ NPDE Structure │ │ │ │
- │ │ └───────────────────┘ │ │ │
- │ │ │ Image data │ │ │ │
- │ │ └───────────────────┘ │ │ │
- │ └───────────────────────────────────────────────┘ │ │
- │ (no gap between images) │ │
- │ ┌───────────────────────────────────────────────┐ │ │
- │ │ First FwSec Image (Type 0xE0) │ │ │
- │ ├───────────────────────────────────────────────┤ │ │
- │ │ ┌───────────────────┐ │ │ │
- │ │ │ ROM Header │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ PCIR Structure │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ NPDE Structure │ │ │ │
- │ │ └───────────────────┘ │ │ │
- │ │ │ Image data │ │ │ │
- │ │ └───────────────────┘ │ │ │
- │ └───────────────────────────────────────────────┘ │ │
- │ (no gap between images) │ │
- │ ┌───────────────────────────────────────────────┐ │ │
- │ │ Second FwSec Image (Type 0xE0) │ │ │
- │ ├───────────────────────────────────────────────┤ │ │
- │ │ ┌───────────────────┐ │ │ │
- │ │ │ ROM Header │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ PCIR Structure │ │ │ │
- │ │ +───────────────────+ │ │ │
- │ │ │ NPDE Structure │ │ │ │
- │ │ └───────────────────┘ │ │ │
- │ │ │ │ │
- │ │ ┌───────────────────┐ │ │ │
- │ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
- │ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
- │ │ │ │ Entry 0x85 │ │ │ │
- │ │ │ │ FWSEC_PROD │ │ │ │
- │ │ │ └─────────────┘ │ │ │
- │ │ └───────────────────┘ │ │
- │ │ │ │ │
- │ │ │ points to │ │
- │ │ V │ │
- │ │ ┌───────────────────┐ │ │
- │ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
- │ │ │ (FWSEC Firmware) │ fwsec_header() │ │
- │ │ └───────────────────┘ │ │
- │ │ │ immediately followed by... │ │
- │ │ V │ │
- │ │ ┌────────────────────────────┐ │ │
- │ │ │ Signatures + FWSEC Ucode │ │ │
- │ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
- │ │ └────────────────────────────┘ │ │
- │ └───────────────────────────────────────────────┘______________________│
+ ┌------------------------------------------------------------------------┐
+ | VBIOS (Starting at ROM_OFFSET: 0x300000) |
+ ├------------------------------------------------------------------------┤
+ | ┌-----------------------------------------------┐ |
+ | | PciAt Image (Type 0x00) | |
+ | ├-----------------------------------------------┤ |
+ | | ┌-------------------┐ | |
+ | | | ROM Header | | |
+ | | | (Signature 0xAA55)| | |
+ | | └-------------------┘ | |
+ | | | rom header's pci_data_struct_offset | |
+ | | | points to the PCIR structure | |
+ | | V | |
+ | | ┌-------------------┐ | |
+ | | | PCIR Structure | | |
+ | | | (Signature "PCIR")| | |
+ | | | last_image: 0x80 | | |
+ | | | image_len: size | | |
+ | | | in 512-byte units | | |
+ | | └-------------------┘ | |
+ | | | | |
+ | | | NPDE immediately follows PCIR | |
+ | | V | |
+ | | ┌-------------------┐ | |
+ | | | NPDE Structure | | |
+ | | | (Signature "NPDE")| | |
+ | | | last_image: 0x00 | | |
+ | | └-------------------┘ | |
+ | | | |
+ | | ┌-------------------┐ | |
+ | | | BIT Header | (Signature scanning | |
+ | | | (Signature "BIT") | provides the location | |
+ | | └-------------------┘ of the BIT table) | |
+ | | | header is | |
+ | | | followed by a table of tokens | |
+ | | V one of which is for falcon data. | |
+ | | ┌-------------------┐ | |
+ | | | BIT Tokens | | |
+ | | | ┌-------------┐ | | |
+ | | | | Falcon Data | | | |
+ | | | | Token (0x70)|---+------------>------------┼--+ |
+ | | | └-------------┘ | falcon_data_ptr() | | |
+ | | └-------------------┘ | V |
+ | └-----------------------------------------------┘ | |
+ | (no gap between images) | |
+ | ┌-----------------------------------------------┐ | |
+ | | EFI Image (Type 0x03) | | |
+ | ├-----------------------------------------------┤ | |
+ | | Contains the UEFI GOP driver (Graphics Output)| | |
+ | | ┌-------------------┐ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | └-------------------┘ | | |
+ | | | Image data | | | |
+ | | └-------------------┘ | | |
+ | └-----------------------------------------------┘ | |
+ | (no gap between images) | |
+ | ┌-----------------------------------------------┐ | |
+ | | First FwSec Image (Type 0xE0) | | |
+ | ├-----------------------------------------------┤ | |
+ | | ┌-------------------┐ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | └-------------------┘ | | |
+ | | | Image data | | | |
+ | | └-------------------┘ | | |
+ | └-----------------------------------------------┘ | |
+ | (no gap between images) | |
+ | ┌-----------------------------------------------┐ | |
+ | | Second FwSec Image (Type 0xE0) | | |
+ | ├-----------------------------------------------┤ | |
+ | | ┌-------------------┐ | | |
+ | | | ROM Header | | | |
+ | | +-------------------+ | | |
+ | | | PCIR Structure | | | |
+ | | +-------------------+ | | |
+ | | | NPDE Structure | | | |
+ | | └-------------------┘ | | |
+ | | | | |
+ | | ┌-------------------┐ | | |
+ | | | PMU Lookup Table | <- falcon_data_offset |<-┘ |
+ | | | ┌-------------┐ | pmu_lookup_table | |
+ | | | | Entry 0x85 | | | |
+ | | | | FWSEC_PROD | | | |
+ | | | └-------------┘ | | |
+ | | └-------------------┘ | |
+ | | | | |
+ | | | points to | |
+ | | V | |
+ | | ┌-------------------┐ | |
+ | | | FalconUCodeDescV3 | <- falcon_ucode_offset | |
+ | | | (FWSEC Firmware) | fwsec_header() | |
+ | | └-------------------┘ | |
+ | | | immediately followed by... | |
+ | | V | |
+ | | ┌----------------------------┐ | |
+ | | | Signatures + FWSEC Ucode | | |
+ | | | fwsec_sigs(), fwsec_ucode()| | |
+ | | └----------------------------┘ | |
+ | └-----------------------------------------------┘ |
+ └------------------------------------------------------------------------┘
Falcon data Lookup
------------------
Thanks.
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [PATCH v2 3/7] nova-core: docs: Document vbios layout
2025-05-05 3:00 ` Bagas Sanjaya
@ 2025-05-05 3:12 ` Bagas Sanjaya
0 siblings, 0 replies; 13+ messages in thread
From: Bagas Sanjaya @ 2025-05-05 3:12 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, David Airlie, Simona Vetter,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
On 5/5/25 10:00, Bagas Sanjaya wrote:
> On Sat, May 03, 2025 at 12:07:55AM -0400, Joel Fernandes wrote:
>> +Here is a block diagram of the VBIOS layout::
>> +
>> + ┌────────────────────────────────────────────────────────────────────────┐
>> + │ VBIOS (Starting at ROM_OFFSET: 0x300000) │
>> + ├────────────────────────────────────────────────────────────────────────┤
>> + │ ┌───────────────────────────────────────────────┐ │
>> + │ │ PciAt Image (Type 0x00) │ │
>> + │ ├───────────────────────────────────────────────┤ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ ROM Header │ │ │
>> + │ │ │ (Signature 0xAA55)│ │ │
>> + │ │ └───────────────────┘ │ │
>> + │ │ │ rom header's pci_data_struct_offset │ │
>> + │ │ │ points to the PCIR structure │ │
>> + │ │ V │ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ PCIR Structure │ │ │
>> + │ │ │ (Signature "PCIR")│ │ │
>> + │ │ │ last_image: 0x80 │ │ │
>> + │ │ │ image_len: size │ │ │
>> + │ │ │ in 512-byte units │ │ │
>> + │ │ └───────────────────┘ │ │
>> + │ │ │ │ │
>> + │ │ │ NPDE immediately follows PCIR │ │
>> + │ │ V │ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ NPDE Structure │ │ │
>> + │ │ │ (Signature "NPDE")│ │ │
>> + │ │ │ last_image: 0x00 │ │ │
>> + │ │ └───────────────────┘ │ │
>> + │ │ │ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ BIT Header │ (Signature scanning │ │
>> + │ │ │ (Signature "BIT") │ provides the location │ │
>> + │ │ └───────────────────┘ of the BIT table) │ │
>> + │ │ │ header is │ │
>> + │ │ | followed by a table of tokens │ │
>> + │ │ V one of which is for falcon data. │ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ BIT Tokens │ │ │
>> + │ | | ______________ | | |
>> + │ │ │ │ Falcon Data │ │ │ │
>> + │ │ │ │ Token (0x70)│---+------------>------------┼──+ │
>> + │ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
>> + │ │ └───────────────────┘ │ V │
>> + │ └───────────────────────────────────────────────┘ │ │
>> + │ (no gap between images) │ │
>> + │ ┌───────────────────────────────────────────────┐ │ │
>> + │ │ EFI Image (Type 0x03) │ │ │
>> + │ ├───────────────────────────────────────────────┤ │ │
>> + │ | Contains the UEFI GOP driver (Graphics Output)| | |
>> + │ │ ┌───────────────────┐ │ │ │
>> + │ │ │ ROM Header │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ PCIR Structure │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ NPDE Structure │ │ │ │
>> + │ │ └───────────────────┘ │ │ │
>> + │ │ │ Image data │ │ │ │
>> + │ │ └───────────────────┘ │ │ │
>> + │ └───────────────────────────────────────────────┘ │ │
>> + │ (no gap between images) │ │
>> + │ ┌───────────────────────────────────────────────┐ │ │
>> + │ │ First FwSec Image (Type 0xE0) │ │ │
>> + │ ├───────────────────────────────────────────────┤ │ │
>> + │ │ ┌───────────────────┐ │ │ │
>> + │ │ │ ROM Header │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ PCIR Structure │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ NPDE Structure │ │ │ │
>> + │ │ └───────────────────┘ │ │ │
>> + │ │ │ Image data │ │ │ │
>> + │ │ └───────────────────┘ │ │ │
>> + │ └───────────────────────────────────────────────┘ │ │
>> + │ (no gap between images) │ │
>> + │ ┌───────────────────────────────────────────────┐ │ │
>> + │ │ Second FwSec Image (Type 0xE0) │ │ │
>> + │ ├───────────────────────────────────────────────┤ │ │
>> + │ │ ┌───────────────────┐ │ │ │
>> + │ │ │ ROM Header │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ PCIR Structure │ │ │ │
>> + │ │ +───────────────────+ │ │ │
>> + │ │ │ NPDE Structure │ │ │ │
>> + │ │ └───────────────────┘ │ │ │
>> + │ │ │ │ │
>> + │ │ ┌───────────────────┐ │ │ │
>> + │ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
>> + │ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
>> + │ │ │ │ Entry 0x85 │ │ │ │
>> + │ │ │ │ FWSEC_PROD │ │ │ │
>> + │ │ │ └─────────────┘ │ │ │
>> + │ │ └───────────────────┘ │ │
>> + │ │ │ │ │
>> + │ │ │ points to │ │
>> + │ │ V │ │
>> + │ │ ┌───────────────────┐ │ │
>> + │ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
>> + │ │ │ (FWSEC Firmware) │ fwsec_header() │ │
>> + │ │ └───────────────────┘ │ │
>> + │ │ │ immediately followed by... │ │
>> + │ │ V │ │
>> + │ │ ┌────────────────────────────┐ │ │
>> + │ │ │ Signatures + FWSEC Ucode │ │ │
>> + │ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
>> + │ │ └────────────────────────────┘ │ │
>> + │ └───────────────────────────────────────────────┘______________________│
>
> Diagram borders look messy in htmldocs output (due to Unicode characters
> ─ and │), so I use ASCII dash and vertical bar instead:
>
> ---- >8 ----
> diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
> index dd6ac891e5f1d0..c68ef0e7b70124 100644
> --- a/Documentation/gpu/nova/core/vbios.rst
> +++ b/Documentation/gpu/nova/core/vbios.rst
> @@ -56,112 +56,113 @@ The VBIOS layout is roughly a series of concatenated images as follows:
>
> Here is a block diagram of the VBIOS layout::
>
> - ┌────────────────────────────────────────────────────────────────────────┐
> - │ VBIOS (Starting at ROM_OFFSET: 0x300000) │
> - ├────────────────────────────────────────────────────────────────────────┤
> - │ ┌───────────────────────────────────────────────┐ │
> - │ │ PciAt Image (Type 0x00) │ │
> - │ ├───────────────────────────────────────────────┤ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ ROM Header │ │ │
> - │ │ │ (Signature 0xAA55)│ │ │
> - │ │ └───────────────────┘ │ │
> - │ │ │ rom header's pci_data_struct_offset │ │
> - │ │ │ points to the PCIR structure │ │
> - │ │ V │ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ PCIR Structure │ │ │
> - │ │ │ (Signature "PCIR")│ │ │
> - │ │ │ last_image: 0x80 │ │ │
> - │ │ │ image_len: size │ │ │
> - │ │ │ in 512-byte units │ │ │
> - │ │ └───────────────────┘ │ │
> - │ │ │ │ │
> - │ │ │ NPDE immediately follows PCIR │ │
> - │ │ V │ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ NPDE Structure │ │ │
> - │ │ │ (Signature "NPDE")│ │ │
> - │ │ │ last_image: 0x00 │ │ │
> - │ │ └───────────────────┘ │ │
> - │ │ │ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ BIT Header │ (Signature scanning │ │
> - │ │ │ (Signature "BIT") │ provides the location │ │
> - │ │ └───────────────────┘ of the BIT table) │ │
> - │ │ │ header is │ │
> - │ │ | followed by a table of tokens │ │
> - │ │ V one of which is for falcon data. │ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ BIT Tokens │ │ │
> - │ | | ______________ | | |
> - │ │ │ │ Falcon Data │ │ │ │
> - │ │ │ │ Token (0x70)│---+------------>------------┼──+ │
> - │ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
> - │ │ └───────────────────┘ │ V │
> - │ └───────────────────────────────────────────────┘ │ │
> - │ (no gap between images) │ │
> - │ ┌───────────────────────────────────────────────┐ │ │
> - │ │ EFI Image (Type 0x03) │ │ │
> - │ ├───────────────────────────────────────────────┤ │ │
> - │ | Contains the UEFI GOP driver (Graphics Output)| | |
> - │ │ ┌───────────────────┐ │ │ │
> - │ │ │ ROM Header │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ PCIR Structure │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ NPDE Structure │ │ │ │
> - │ │ └───────────────────┘ │ │ │
> - │ │ │ Image data │ │ │ │
> - │ │ └───────────────────┘ │ │ │
> - │ └───────────────────────────────────────────────┘ │ │
> - │ (no gap between images) │ │
> - │ ┌───────────────────────────────────────────────┐ │ │
> - │ │ First FwSec Image (Type 0xE0) │ │ │
> - │ ├───────────────────────────────────────────────┤ │ │
> - │ │ ┌───────────────────┐ │ │ │
> - │ │ │ ROM Header │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ PCIR Structure │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ NPDE Structure │ │ │ │
> - │ │ └───────────────────┘ │ │ │
> - │ │ │ Image data │ │ │ │
> - │ │ └───────────────────┘ │ │ │
> - │ └───────────────────────────────────────────────┘ │ │
> - │ (no gap between images) │ │
> - │ ┌───────────────────────────────────────────────┐ │ │
> - │ │ Second FwSec Image (Type 0xE0) │ │ │
> - │ ├───────────────────────────────────────────────┤ │ │
> - │ │ ┌───────────────────┐ │ │ │
> - │ │ │ ROM Header │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ PCIR Structure │ │ │ │
> - │ │ +───────────────────+ │ │ │
> - │ │ │ NPDE Structure │ │ │ │
> - │ │ └───────────────────┘ │ │ │
> - │ │ │ │ │
> - │ │ ┌───────────────────┐ │ │ │
> - │ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
> - │ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
> - │ │ │ │ Entry 0x85 │ │ │ │
> - │ │ │ │ FWSEC_PROD │ │ │ │
> - │ │ │ └─────────────┘ │ │ │
> - │ │ └───────────────────┘ │ │
> - │ │ │ │ │
> - │ │ │ points to │ │
> - │ │ V │ │
> - │ │ ┌───────────────────┐ │ │
> - │ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
> - │ │ │ (FWSEC Firmware) │ fwsec_header() │ │
> - │ │ └───────────────────┘ │ │
> - │ │ │ immediately followed by... │ │
> - │ │ V │ │
> - │ │ ┌────────────────────────────┐ │ │
> - │ │ │ Signatures + FWSEC Ucode │ │ │
> - │ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
> - │ │ └────────────────────────────┘ │ │
> - │ └───────────────────────────────────────────────┘______________________│
> + ┌------------------------------------------------------------------------┐
> + | VBIOS (Starting at ROM_OFFSET: 0x300000) |
> + ├------------------------------------------------------------------------┤
> + | ┌-----------------------------------------------┐ |
> + | | PciAt Image (Type 0x00) | |
> + | ├-----------------------------------------------┤ |
> + | | ┌-------------------┐ | |
> + | | | ROM Header | | |
> + | | | (Signature 0xAA55)| | |
> + | | └-------------------┘ | |
> + | | | rom header's pci_data_struct_offset | |
> + | | | points to the PCIR structure | |
> + | | V | |
> + | | ┌-------------------┐ | |
> + | | | PCIR Structure | | |
> + | | | (Signature "PCIR")| | |
> + | | | last_image: 0x80 | | |
> + | | | image_len: size | | |
> + | | | in 512-byte units | | |
> + | | └-------------------┘ | |
> + | | | | |
> + | | | NPDE immediately follows PCIR | |
> + | | V | |
> + | | ┌-------------------┐ | |
> + | | | NPDE Structure | | |
> + | | | (Signature "NPDE")| | |
> + | | | last_image: 0x00 | | |
> + | | └-------------------┘ | |
> + | | | |
> + | | ┌-------------------┐ | |
> + | | | BIT Header | (Signature scanning | |
> + | | | (Signature "BIT") | provides the location | |
> + | | └-------------------┘ of the BIT table) | |
> + | | | header is | |
> + | | | followed by a table of tokens | |
> + | | V one of which is for falcon data. | |
> + | | ┌-------------------┐ | |
> + | | | BIT Tokens | | |
> + | | | ┌-------------┐ | | |
> + | | | | Falcon Data | | | |
> + | | | | Token (0x70)|---+------------>------------┼--+ |
> + | | | └-------------┘ | falcon_data_ptr() | | |
> + | | └-------------------┘ | V |
> + | └-----------------------------------------------┘ | |
> + | (no gap between images) | |
> + | ┌-----------------------------------------------┐ | |
> + | | EFI Image (Type 0x03) | | |
> + | ├-----------------------------------------------┤ | |
> + | | Contains the UEFI GOP driver (Graphics Output)| | |
> + | | ┌-------------------┐ | | |
> + | | | ROM Header | | | |
> + | | +-------------------+ | | |
> + | | | PCIR Structure | | | |
> + | | +-------------------+ | | |
> + | | | NPDE Structure | | | |
> + | | └-------------------┘ | | |
> + | | | Image data | | | |
> + | | └-------------------┘ | | |
> + | └-----------------------------------------------┘ | |
> + | (no gap between images) | |
> + | ┌-----------------------------------------------┐ | |
> + | | First FwSec Image (Type 0xE0) | | |
> + | ├-----------------------------------------------┤ | |
> + | | ┌-------------------┐ | | |
> + | | | ROM Header | | | |
> + | | +-------------------+ | | |
> + | | | PCIR Structure | | | |
> + | | +-------------------+ | | |
> + | | | NPDE Structure | | | |
> + | | └-------------------┘ | | |
> + | | | Image data | | | |
> + | | └-------------------┘ | | |
> + | └-----------------------------------------------┘ | |
> + | (no gap between images) | |
> + | ┌-----------------------------------------------┐ | |
> + | | Second FwSec Image (Type 0xE0) | | |
> + | ├-----------------------------------------------┤ | |
> + | | ┌-------------------┐ | | |
> + | | | ROM Header | | | |
> + | | +-------------------+ | | |
> + | | | PCIR Structure | | | |
> + | | +-------------------+ | | |
> + | | | NPDE Structure | | | |
> + | | └-------------------┘ | | |
> + | | | | |
> + | | ┌-------------------┐ | | |
> + | | | PMU Lookup Table | <- falcon_data_offset |<-┘ |
> + | | | ┌-------------┐ | pmu_lookup_table | |
> + | | | | Entry 0x85 | | | |
> + | | | | FWSEC_PROD | | | |
> + | | | └-------------┘ | | |
> + | | └-------------------┘ | |
> + | | | | |
> + | | | points to | |
> + | | V | |
> + | | ┌-------------------┐ | |
> + | | | FalconUCodeDescV3 | <- falcon_ucode_offset | |
> + | | | (FWSEC Firmware) | fwsec_header() | |
> + | | └-------------------┘ | |
> + | | | immediately followed by... | |
> + | | V | |
> + | | ┌----------------------------┐ | |
> + | | | Signatures + FWSEC Ucode | | |
> + | | | fwsec_sigs(), fwsec_ucode()| | |
> + | | └----------------------------┘ | |
> + | └-----------------------------------------------┘ |
> + └------------------------------------------------------------------------┘
>
Addendum: I'm using Roboto Mono on my htmldocs build (using custom style
sheet). The diagram looks fine with default Alabaster font stack
(Consolas). If you disagree you can ignore above diff.
Thanks.
--
An old man doll... just what I always wanted! - Clara
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
[not found] <20250503040802.1411285-1-joelagnelf@nvidia.com>
2025-05-03 4:07 ` [PATCH v2 3/7] nova-core: docs: Document vbios layout Joel Fernandes
@ 2025-05-03 4:07 ` Joel Fernandes
2025-05-05 3:52 ` Bagas Sanjaya
2025-05-06 16:26 ` Zhi Wang
2025-05-03 4:07 ` [PATCH v2 5/7] docs: nova-core: Document devinit process Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon Joel Fernandes
3 siblings, 2 replies; 13+ messages in thread
From: Joel Fernandes @ 2025-05-03 4:07 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, Joel Fernandes, linux-doc
Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 180 ++++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 181 insertions(+)
create mode 100644 Documentation/gpu/nova/core/fwsec.rst
diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..bed941ac3f2b
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,180 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+=========================
+FWSEC (Firmware Security)
+=========================
+This document briefly/conceptually describes the FWSEC (Firmware Security) image
+and its role in the GPU boot sequence. As such, this information is subject to
+change in the future and is only current as of the Ampere GPU family. However,
+hopefully the concepts described will be useful for understanding the kernel code
+that deals with it. All the information is derived from publicly available
+sources such as public drivers and documentation.
+
+The role of FWSEC to provide secure boot, it is running in Heavy-secure mode. It does
+firmware verification after GPU reset and load various ucode images on to the other
+microcontrollers on the GPU such as the PMU and GSP.
+
+FWSEC itself is an application stored in the VBIOS ROM in the FWSEC partition of
+ROM (see vbios.rst for more details). It contains different commands like FRTS
+(Firmware Runtime Services) and SB (Secure Booting other microcontrollers after
+reset and loading them with other non-FWSEC ucode). The kernel driver only needs to
+to do FRTS, since SB is already already after reset by the time the kernel driver
+is loaded.
+
+The FRTS command carves out the WPR2 region (Write protected region) which contains
+data data required for power management. Once setup, only HS mode ucode can
+access it (see falcon.rst for privilege levels).
+
+The FWSEC image is located in the VBIOS ROM in the partition of the ROM that contains
+various ucode images (also known as applications) -- one of them being FWSEC. For how
+it is extracted, see vbios.rst and the vbios.rs source code.
+
+The Falcon data for each ucode images (including the FWSEC image) is a combination
+of headers, data sections (DMEM) and instruction code sections (IMEM). All these
+ucode images are stored in the same ROM partition and the PMU table is used to look
+up the application to load it based on its application ID (see vbios.rs).
+
+For the purposes of nova-core driver, the FWSEC contains an 'application interface'
+called DMEMMAPPER which is used to the "FWSEC-FRTS" command (among other commands it
+is capable of executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
+mode and runs FRTS.
+
+FWSEC Memory Layout
+-------------------
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs and is subject to change
+completely, it is just provided as an example):
+
+Here is a block diagram of the FWSEC memory layout::
+ ┌───────────────────────────────────────────────────────────────┐
+ │ FWSEC ROM image (type 0xE0) │
+ │ │
+ │ ┌─────────────────────────────────┐ │
+ │ │ PMU Falcon Ucode Table │ │
+ │ │ (PmuLookupTable) │ │
+ │ │ ┌─────────────────────────┐ │ │
+ │ │ │ Table Header │ │ │
+ │ │ │ - version: 0x01 │ │ │
+ │ │ │ - header_size: 6 │ │ │
+ │ │ │ - entry_size: 6 │ │ │
+ │ │ │ - entry_count: N │ │ │
+ │ │ │ - desc_version:3(unused)│ │ │
+ │ │ └─────────────────────────┘ │ │
+ │ │ ... │ │
+ │ │ ┌─────────────────────────┐ │ │
+ │ │ │ Entry for FWSEC (0x85) │ │ │
+ │ │ │ (PmuLookupTableEntry) │ │ │
+ │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
+ │ │ │ - target_id: 0x01 (PMU) │ │ │ │
+ │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
+ │ │ └─────────────────────────┘ │ │ │ application. │
+ │ └─────────────────────────────────┘ │ │ │
+ │ │ │ │
+ │ │ │ │
+ │ ┌─────────────────────────────────┐ │ │ │
+ │ │ FWSEC Ucode Component │<───┘ │ │
+ │ │ (aka Falcon data) │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
+ │ │ │ - hdr │ │ │
+ │ │ │ - stored_size │ │ │
+ │ │ │ - pkc_data_offset │ │ │
+ │ │ │ - interface_offset ─────┼────┼────────────────┐ │
+ │ │ │ - imem_phys_base │ │ │ │
+ │ │ │ - imem_load_size │ │ │ │
+ │ │ │ - imem_virt_base │ │ │ │
+ │ │ │ - dmem_phys_base │ │ │ │
+ │ │ │ - dmem_load_size │ │ │ │
+ │ │ │ - engine_id_mask │ │ │ │
+ │ │ │ - ucode_id │ │ │ │
+ │ │ │ - signature_count │ │ look up sig │ │
+ │ │ │ - signature_versions --------------+ │ │
+ │ │ └─────────────────────────┘ │ │ │ │
+ │ │ (no gap) │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │ │
+ │ │ │ Signatures Section │<───┼─────┘ │ │
+ │ │ │ (384 bytes per sig) │ │ │ │
+ │ │ │ - RSA-3K Signature 1 │ │ │ │
+ │ │ │ - RSA-3K Signature 2 │ │ │ │
+ │ │ │ ... │ │ │ │
+ │ │ └─────────────────────────┘ │ │ │
+ │ │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ IMEM Section (Code) │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ Contains instruction │ │ │ │
+ │ │ │ code etc. │ │ │ │
+ │ │ └─────────────────────────┘ │ │ │
+ │ │ │ │ │
+ │ │ ┌─────────────────────────┐ │ │ │
+ │ │ │ DMEM Section (Data) │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ Application │ │<───┼────────────────┘ │
+ │ │ │ │ Interface Table │ │ │ │
+ │ │ │ │ (FalconAppifHdrV1) │ │ │ │
+ │ │ │ │ Header: │ │ │ │
+ │ │ │ │ - version: 0x01 │ │ │ │
+ │ │ │ │ - header_size: 4 │ │ │ │
+ │ │ │ │ - entry_size: 8 │ │ │ │
+ │ │ │ │ - entry_count: N │ │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ │ Entries: │ │ │ │
+ │ │ │ │ ┌─────────────────┐ │ │ │ │
+ │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
+ │ │ │ │ │ - id: 0x01 │ │ │ │ │
+ │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
+ │ │ │ │ └─────────────────┘ │ │ │ │
+ │ │ │ │ ┌─────────────────┐ │ │ │ │
+ │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
+ │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
+ │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
+ │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
+ │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
+ │ │ │ │ └─────────────────┘ │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
+ │ │ │ │ Interface │ │ │ │ app. │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
+ │ │ │ │ (FalconAppifDmemmapperV3) │ │
+ │ │ │ │ - signature: "DMAP" │ │ │ │
+ │ │ │ │ - version: 0x0003 │ │ │ │
+ │ │ │ │ - Size: 64 bytes │ │ │ │
+ │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
+ │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
+ │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
+ │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
+ │ │ │ │ - init_cmd │ │ │ │ │ │
+ │ │ │ │ - features │ │ │ │ │ │
+ │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │ │
+ │ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │ │
+ │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
+ │ │ │ │ - Command data │ │ │ │ │
+ │ │ │ │ - Arguments │ │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │ │
+ │ │ │ │ │ │ │
+ │ │ │ ┌─────────────────────┐ │ │ │ │
+ │ │ │ │ Command Output │<┼────┼──────────────────┘ │
+ │ │ │ │ Buffer │ │ │ │
+ │ │ │ │ - Results │ │ │ │
+ │ │ │ │ - Status │ │ │ │
+ │ │ │ └─────────────────────┘ │ │ │
+ │ │ └─────────────────────────┘ │ │
+ │ └─────────────────────────────────┘ │
+ │ │
+ └───────────────────────────────────────────────────────────────┘
+
+.. note::
+ The FWSEC image also plays a role in memory scrubbing (ECC initialization) and VPR
+ (Video Protected Region) initialization as well. Before the nova-core driver is even
+ loaded, the FWSEC image is running on the GSP in heavy-secure mode. After the devinit
+ sequence completes, it does VRAM memory scrubbing (ECC initialization). On consumer
+ GPUs, it scrubs only part of memory and then initiates 'async scrubbing'. Before this
+ async scrubbing completes, the unscrubbed VRAM cannot be used for allocation (thus DRM
+ memory allocators need to wait for this scrubbing to complete).
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 91cc802ed94f..22e5712ac6b0 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
+ core/fwsec
core/todo
--
2.43.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
2025-05-03 4:07 ` [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout Joel Fernandes
@ 2025-05-05 3:52 ` Bagas Sanjaya
2025-05-06 16:26 ` Zhi Wang
1 sibling, 0 replies; 13+ messages in thread
From: Bagas Sanjaya @ 2025-05-05 3:52 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, David Airlie,
Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
[-- Attachment #1: Type: text/plain, Size: 13289 bytes --]
On Sat, May 03, 2025 at 12:07:56AM -0400, Joel Fernandes wrote:
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +=========================
> +FWSEC (Firmware Security)
> +=========================
Separate SPDX line from title heading.
> +Here is a block diagram of the FWSEC memory layout::
Add blank line before actual diagram below.
> + ┌───────────────────────────────────────────────────────────────┐
> + │ FWSEC ROM image (type 0xE0) │
> + │ │
> + │ ┌─────────────────────────────────┐ │
> + │ │ PMU Falcon Ucode Table │ │
> + │ │ (PmuLookupTable) │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Table Header │ │ │
> + │ │ │ - version: 0x01 │ │ │
> + │ │ │ - header_size: 6 │ │ │
> + │ │ │ - entry_size: 6 │ │ │
> + │ │ │ - entry_count: N │ │ │
> + │ │ │ - desc_version:3(unused)│ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ │ ... │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Entry for FWSEC (0x85) │ │ │
> + │ │ │ (PmuLookupTableEntry) │ │ │
> + │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
> + │ │ │ - target_id: 0x01 (PMU) │ │ │ │
> + │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
> + │ │ └─────────────────────────┘ │ │ │ application. │
> + │ └─────────────────────────────────┘ │ │ │
> + │ │ │ │
> + │ │ │ │
> + │ ┌─────────────────────────────────┐ │ │ │
> + │ │ FWSEC Ucode Component │<───┘ │ │
> + │ │ (aka Falcon data) │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
> + │ │ │ - hdr │ │ │
> + │ │ │ - stored_size │ │ │
> + │ │ │ - pkc_data_offset │ │ │
> + │ │ │ - interface_offset ─────┼────┼────────────────┐ │
> + │ │ │ - imem_phys_base │ │ │ │
> + │ │ │ - imem_load_size │ │ │ │
> + │ │ │ - imem_virt_base │ │ │ │
> + │ │ │ - dmem_phys_base │ │ │ │
> + │ │ │ - dmem_load_size │ │ │ │
> + │ │ │ - engine_id_mask │ │ │ │
> + │ │ │ - ucode_id │ │ │ │
> + │ │ │ - signature_count │ │ look up sig │ │
> + │ │ │ - signature_versions --------------+ │ │
> + │ │ └─────────────────────────┘ │ │ │ │
> + │ │ (no gap) │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │ │
> + │ │ │ Signatures Section │<───┼─────┘ │ │
> + │ │ │ (384 bytes per sig) │ │ │ │
> + │ │ │ - RSA-3K Signature 1 │ │ │ │
> + │ │ │ - RSA-3K Signature 2 │ │ │ │
> + │ │ │ ... │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ IMEM Section (Code) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ Contains instruction │ │ │ │
> + │ │ │ code etc. │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ DMEM Section (Data) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Application │ │<───┼────────────────┘ │
> + │ │ │ │ Interface Table │ │ │ │
> + │ │ │ │ (FalconAppifHdrV1) │ │ │ │
> + │ │ │ │ Header: │ │ │ │
> + │ │ │ │ - version: 0x01 │ │ │ │
> + │ │ │ │ - header_size: 4 │ │ │ │
> + │ │ │ │ - entry_size: 8 │ │ │ │
> + │ │ │ │ - entry_count: N │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ │ Entries: │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
> + │ │ │ │ │ - id: 0x01 │ │ │ │ │
> + │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
> + │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
> + │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
> + │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
> + │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
> + │ │ │ │ Interface │ │ │ │ app. │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
> + │ │ │ │ (FalconAppifDmemmapperV3) │ │
> + │ │ │ │ - signature: "DMAP" │ │ │ │
> + │ │ │ │ - version: 0x0003 │ │ │ │
> + │ │ │ │ - Size: 64 bytes │ │ │ │
> + │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
> + │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
> + │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
> + │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
> + │ │ │ │ - init_cmd │ │ │ │ │ │
> + │ │ │ │ - features │ │ │ │ │ │
> + │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │ │
> + │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
> + │ │ │ │ - Command data │ │ │ │ │
> + │ │ │ │ - Arguments │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Command Output │<┼────┼──────────────────┘ │
> + │ │ │ │ Buffer │ │ │ │
> + │ │ │ │ - Results │ │ │ │
> + │ │ │ │ - Status │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ └─────────────────────────────────┘ │
> + │ │
> + └───────────────────────────────────────────────────────────────┘
The diagram can look messy on certain fonts in htmldocs output. Please
use ASCII dash (-) and vertical bar (|) instead of Unicode box decoration
variant (─ and │).
Thanks.
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
2025-05-03 4:07 ` [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout Joel Fernandes
2025-05-05 3:52 ` Bagas Sanjaya
@ 2025-05-06 16:26 ` Zhi Wang
2025-05-09 20:56 ` Joel Fernandes
1 sibling, 1 reply; 13+ messages in thread
From: Zhi Wang @ 2025-05-06 16:26 UTC (permalink / raw)
To: Joel Fernandes
Cc: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet, nouveau, dri-devel, Alexandre Courbot,
John Hubbard, Shirish Baskaran, Alistair Popple, Timur Tabi,
Ben Skeggs, rust-for-linux, linux-doc
On Sat, 3 May 2025 00:07:56 -0400
Joel Fernandes <joelagnelf@nvidia.com> wrote:
> Add explanation of fwsec with diagrams. This helps clarify how the
> nova-core falcon boot works.
>
> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> ---
> Documentation/gpu/nova/core/fwsec.rst | 180
> ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst |
> 1 + 2 files changed, 181 insertions(+)
> create mode 100644 Documentation/gpu/nova/core/fwsec.rst
>
> diff --git a/Documentation/gpu/nova/core/fwsec.rst
> b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644
> index 000000000000..bed941ac3f2b
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/fwsec.rst
> @@ -0,0 +1,180 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +=========================
> +FWSEC (Firmware Security)
> +=========================
> +This document briefly/conceptually describes the FWSEC (Firmware
> Security) image +and its role in the GPU boot sequence. As such, this
> information is subject to +change in the future and is only current
> as of the Ampere GPU family. However, +hopefully the concepts
> described will be useful for understanding the kernel code +that
> deals with it. All the information is derived from publicly available
> +sources such as public drivers and documentation. +
> +The role of FWSEC to provide secure boot, it is running in
> Heavy-secure mode. It does +firmware verification after GPU reset and
> load various ucode images on to the other +microcontrollers on the
> GPU such as the PMU and GSP. +
> +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC
> partition of +ROM (see vbios.rst for more details). It contains
> different commands like FRTS +(Firmware Runtime Services) and SB
> (Secure Booting other microcontrollers after +reset and loading them
> with other non-FWSEC ucode). The kernel driver only needs to +to do
> FRTS, since SB is already already after reset by the time the kernel
> driver +is loaded. +
> +The FRTS command carves out the WPR2 region (Write protected region)
> which contains +data data required for power management. Once setup,
repetitive word "data" ^
> only HS mode ucode can +access it (see falcon.rst for privilege
> levels). +
> +The FWSEC image is located in the VBIOS ROM in the partition of the
> ROM that contains +various ucode images (also known as applications)
> -- one of them being FWSEC. For how +it is extracted, see vbios.rst
> and the vbios.rs source code. +
> +The Falcon data for each ucode images (including the FWSEC image) is
> a combination +of headers, data sections (DMEM) and instruction code
> sections (IMEM). All these +ucode images are stored in the same ROM
> partition and the PMU table is used to look +up the application to
> load it based on its application ID (see vbios.rs). +
> +For the purposes of nova-core driver, the FWSEC contains an
> 'application interface' +called DMEMMAPPER which is used to the
> "FWSEC-FRTS" command (among other commands it +is capable of
> executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
> +mode and runs FRTS. +
> +FWSEC Memory Layout
> +-------------------
> +The memory layout of the FWSEC image is as follows (this is using an
> GA-102 +Ampere GPU as an example and could vary for future GPUs and
> is subject to change +completely, it is just provided as an example):
> +
> +Here is a block diagram of the FWSEC memory layout::
> + ┌───────────────────────────────────────────────────────────────┐
> + │ FWSEC ROM image (type 0xE0) │
> + │ │
> + │ ┌─────────────────────────────────┐ │
> + │ │ PMU Falcon Ucode Table │ │
> + │ │ (PmuLookupTable) │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Table Header │ │ │
> + │ │ │ - version: 0x01 │ │ │
> + │ │ │ - header_size: 6 │ │ │
> + │ │ │ - entry_size: 6 │ │ │
> + │ │ │ - entry_count: N │ │ │
> + │ │ │ - desc_version:3(unused)│ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ │ ... │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Entry for FWSEC (0x85) │ │ │
> + │ │ │ (PmuLookupTableEntry) │ │ │
> + │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
> + │ │ │ - target_id: 0x01 (PMU) │ │ │ │
> + │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
> + │ │ └─────────────────────────┘ │ │ │ application. │
> + │ └─────────────────────────────────┘ │ │ │
> + │ │ │ │
> + │ │ │ │
> + │ ┌─────────────────────────────────┐ │ │ │
> + │ │ FWSEC Ucode Component │<───┘ │ │
> + │ │ (aka Falcon data) │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
> + │ │ │ - hdr │ │ │
> + │ │ │ - stored_size │ │ │
> + │ │ │ - pkc_data_offset │ │ │
> + │ │ │ - interface_offset ─────┼────┼────────────────┐ │
> + │ │ │ - imem_phys_base │ │ │ │
> + │ │ │ - imem_load_size │ │ │ │
> + │ │ │ - imem_virt_base │ │ │ │
> + │ │ │ - dmem_phys_base │ │ │ │
> + │ │ │ - dmem_load_size │ │ │ │
> + │ │ │ - engine_id_mask │ │ │ │
> + │ │ │ - ucode_id │ │ │ │
> + │ │ │ - signature_count │ │ look up sig │ │
> + │ │ │ - signature_versions --------------+ │ │
> + │ │ └─────────────────────────┘ │ │ │ │
> + │ │ (no gap) │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │ │
> + │ │ │ Signatures Section │<───┼─────┘ │ │
> + │ │ │ (384 bytes per sig) │ │ │ │
> + │ │ │ - RSA-3K Signature 1 │ │ │ │
> + │ │ │ - RSA-3K Signature 2 │ │ │ │
> + │ │ │ ... │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ IMEM Section (Code) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ Contains instruction │ │ │ │
> + │ │ │ code etc. │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ DMEM Section (Data) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Application │ │<───┼────────────────┘ │
> + │ │ │ │ Interface Table │ │ │ │
> + │ │ │ │ (FalconAppifHdrV1) │ │ │ │
> + │ │ │ │ Header: │ │ │ │
> + │ │ │ │ - version: 0x01 │ │ │ │
> + │ │ │ │ - header_size: 4 │ │ │ │
> + │ │ │ │ - entry_size: 8 │ │ │ │
> + │ │ │ │ - entry_count: N │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ │ Entries: │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
> + │ │ │ │ │ - id: 0x01 │ │ │ │ │
> + │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
> + │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
> + │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
> + │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
> + │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
> + │ │ │ │ Interface │ │ │ │ app. │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
> + │ │ │ │ (FalconAppifDmemmapperV3) │ │
> + │ │ │ │ - signature: "DMAP" │ │ │ │
> + │ │ │ │ - version: 0x0003 │ │ │ │
> + │ │ │ │ - Size: 64 bytes │ │ │ │
> + │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
> + │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
> + │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
> + │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
> + │ │ │ │ - init_cmd │ │ │ │ │ │
> + │ │ │ │ - features │ │ │ │ │ │
> + │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │ │
> + │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
> + │ │ │ │ - Command data │ │ │ │ │
> + │ │ │ │ - Arguments │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Command Output │<┼────┼──────────────────┘ │
> + │ │ │ │ Buffer │ │ │ │
> + │ │ │ │ - Results │ │ │ │
> + │ │ │ │ - Status │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ └─────────────────────────────────┘ │
> + │ │
> + └───────────────────────────────────────────────────────────────┘
> +
> +.. note::
> + The FWSEC image also plays a role in memory scrubbing (ECC
> initialization) and VPR
> + (Video Protected Region) initialization as well. Before the
> nova-core driver is even
> + loaded, the FWSEC image is running on the GSP in heavy-secure
> mode. After the devinit
> + sequence completes, it does VRAM memory scrubbing (ECC
> initialization). On consumer
> + GPUs, it scrubs only part of memory and then initiates 'async
> scrubbing'. Before this
> + async scrubbing completes, the unscrubbed VRAM cannot be used for
> allocation (thus DRM
> + memory allocators need to wait for this scrubbing to complete).
> \ No newline at end of file
> diff --git a/Documentation/gpu/nova/index.rst
> b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0
> 100644 --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
>
> core/guidelines
> core/vbios
> + core/fwsec
> core/todo
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
2025-05-06 16:26 ` Zhi Wang
@ 2025-05-09 20:56 ` Joel Fernandes
0 siblings, 0 replies; 13+ messages in thread
From: Joel Fernandes @ 2025-05-09 20:56 UTC (permalink / raw)
To: Zhi Wang
Cc: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet, nouveau, dri-devel, Alexandre Courbot,
John Hubbard, Shirish Baskaran, Alistair Popple, Timur Tabi,
Ben Skeggs, rust-for-linux, linux-doc
On 5/6/2025 12:26 PM, Zhi Wang wrote:
> On Sat, 3 May 2025 00:07:56 -0400
> Joel Fernandes <joelagnelf@nvidia.com> wrote:
>
>> Add explanation of fwsec with diagrams. This helps clarify how the
>> nova-core falcon boot works.
>>
>> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
>> ---
>> Documentation/gpu/nova/core/fwsec.rst | 180
>> ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst |
>> 1 + 2 files changed, 181 insertions(+)
>> create mode 100644 Documentation/gpu/nova/core/fwsec.rst
>>
>> diff --git a/Documentation/gpu/nova/core/fwsec.rst
>> b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644
>> index 000000000000..bed941ac3f2b
>> --- /dev/null
>> +++ b/Documentation/gpu/nova/core/fwsec.rst
>> @@ -0,0 +1,180 @@
>> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
>> +=========================
>> +FWSEC (Firmware Security)
>> +=========================
>> +This document briefly/conceptually describes the FWSEC (Firmware
>> Security) image +and its role in the GPU boot sequence. As such, this
>> information is subject to +change in the future and is only current
>> as of the Ampere GPU family. However, +hopefully the concepts
>> described will be useful for understanding the kernel code +that
>> deals with it. All the information is derived from publicly available
>> +sources such as public drivers and documentation. +
>> +The role of FWSEC to provide secure boot, it is running in
>> Heavy-secure mode. It does +firmware verification after GPU reset and
>> load various ucode images on to the other +microcontrollers on the
>> GPU such as the PMU and GSP. +
>> +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC
>> partition of +ROM (see vbios.rst for more details). It contains
>> different commands like FRTS +(Firmware Runtime Services) and SB
>> (Secure Booting other microcontrollers after +reset and loading them
>> with other non-FWSEC ucode). The kernel driver only needs to +to do
>> FRTS, since SB is already already after reset by the time the kernel
>> driver +is loaded. +
>> +The FRTS command carves out the WPR2 region (Write protected region)
>> which contains +data data required for power management. Once setup,
>
> repetitive word "data" ^
>
Fixed, thanks!
- Joel
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 5/7] docs: nova-core: Document devinit process
[not found] <20250503040802.1411285-1-joelagnelf@nvidia.com>
2025-05-03 4:07 ` [PATCH v2 3/7] nova-core: docs: Document vbios layout Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout Joel Fernandes
@ 2025-05-03 4:07 ` Joel Fernandes
2025-05-05 4:04 ` Bagas Sanjaya
2025-05-03 4:07 ` [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon Joel Fernandes
3 siblings, 1 reply; 13+ messages in thread
From: Joel Fernandes @ 2025-05-03 4:07 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, Joel Fernandes, linux-doc
devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/devinit.rst | 59 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
drivers/gpu/nova-core/devinit.rs | 6 +--
3 files changed, 63 insertions(+), 3 deletions(-)
create mode 100644 Documentation/gpu/nova/core/devinit.rst
diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 000000000000..85e7033dd3a7
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,59 @@
+.. SPDX-License-Identifier: GPL-2.0
+==================================
+Device Initialization (devinit)
+==================================
+Note that devinit itself is a complicated process, subject to change, and this document
+only roughly provides an overview of the process using the Ampere GPU family as an example.
+The idea is to provide a conceptual overview of the process to help understand the kernel
+code that deals with it.
+
+Device initialization (devinit) is a crucial sequence of register read/write operations
+that occur after a GPU reset. The devinit sequence is essential for properly configuring
+the GPU hardware before it can be used.
+
+The devinit engine is an interpreter program that typically runs on the PMU (Power Management
+Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
+commands. The devinit engine itself is part of the VBIOS ROM in the same ROM image as the
+FWSEC (Firmware Security) image (see fwsec.rst and vbios.rst) and it runs before the
+nova-core driver is even loaded. On an Ampere GPU, the devinit ucode (which is separate
+from the FWSEC ucode), is launched by the FWSEC (FWSEC runs on GSP in heavy-secure mode
+and devinit runs on PMU in light-secure mode).
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+
+Low-level Firmware Initialization Flow
+--------------------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
+firmware (gfw) code to set up the GPU and its core parameters. Most of the GPU is
+considered unusable until this initialization process completes.
+
+These low-level GPU firmware components are typically:
+1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
+2. Executed in sequence on different microcontrollers:
+ - The devinit engine typically but not necessarily runs on the PMU.
+ - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
+ heavy-secure mode.
+
+Before the driver can proceed with further initialization, it must wait for a signal
+indicating that core initialization is complete (known as GFW_BOOT). This signal is
+asserted by the FWSEC running on the GSP in heavy-secure mode.
+
+Runtime Considerations
+---------------------
+It's important to note that the devinit sequence also needs to run during suspend/resume
+operations at runtime, not just during initial boot, as it is critical to power management.
+
+Security and Access Control
+--------------------------
+The initialization process involves careful privilege management. For example, before
+accessing certain completion status registers, the driver must check privilege level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
+privilege level to allow CPU (LS/low-secure) access. For example, to receive the
+GFW_BOOT signal.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 22e5712ac6b0..301435c5cf67 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -29,4 +29,5 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
core/fwsec
+ core/devinit
core/todo
diff --git a/drivers/gpu/nova-core/devinit.rs b/drivers/gpu/nova-core/devinit.rs
index 21fc9b96d498..bd6a5c57c444 100644
--- a/drivers/gpu/nova-core/devinit.rs
+++ b/drivers/gpu/nova-core/devinit.rs
@@ -10,8 +10,8 @@
//! 3. Clock and PLL configuration.
//! 4. Thermal management.
//!
-//! devinit itself is a 'script' which is interpreted by the PMU microcontroller of
-//! the GPU by an interpreter program.
+//! devinit itself is a 'script' which is interpreted by a microcontroller of
+//! the GPU by an interpreter program (typically running on the PMU).
//!
//! Note that the devinit sequence also needs to run during suspend/resume at runtime.
@@ -30,7 +30,7 @@
///
/// The GPU firmware (gfw) code includes several components that execute before the driver loads.
/// These components are located in the VBIOS ROM and are executed in a sequence on these different
-/// microcontrollers. The devinit sequence itself runs on the PMU, and the FWSEC runs on the GSP.
+/// microcontrollers. The devinit sequence typically runs on the PMU, and the FWSEC runs on the GSP.
///
/// This function specifically waits for a signal indicating core initialization is complete before
/// which not much can be done. This signal is setup by the FWSEC running on the GSP in Heavy-secured
--
2.43.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 5/7] docs: nova-core: Document devinit process
2025-05-03 4:07 ` [PATCH v2 5/7] docs: nova-core: Document devinit process Joel Fernandes
@ 2025-05-05 4:04 ` Bagas Sanjaya
2025-05-05 22:15 ` Joel Fernandes
0 siblings, 1 reply; 13+ messages in thread
From: Bagas Sanjaya @ 2025-05-05 4:04 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, David Airlie, Simona Vetter,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
[-- Attachment #1: Type: text/plain, Size: 1037 bytes --]
On Sat, May 03, 2025 at 12:07:57AM -0400, Joel Fernandes wrote:
> +.. SPDX-License-Identifier: GPL-2.0
> +==================================
> +Device Initialization (devinit)
> +==================================
Separate SPDX line from title heading.
> +These low-level GPU firmware components are typically:
> +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
> +2. Executed in sequence on different microcontrollers:
> + - The devinit engine typically but not necessarily runs on the PMU.
> + - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
> + heavy-secure mode.
Please separate numbered list from preceding sentence, and the bullet sublist
from parent numbered list by a line.
> +Runtime Considerations
> +---------------------
> <snipped>...
> +Security and Access Control
> +--------------------------
Match section underline length with the text.
Thanks.
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 5/7] docs: nova-core: Document devinit process
2025-05-05 4:04 ` Bagas Sanjaya
@ 2025-05-05 22:15 ` Joel Fernandes
0 siblings, 0 replies; 13+ messages in thread
From: Joel Fernandes @ 2025-05-05 22:15 UTC (permalink / raw)
To: Bagas Sanjaya, linux-kernel, Danilo Krummrich, Maarten Lankhorst,
Maxime Ripard, Thomas Zimmermann, David Airlie, Simona Vetter,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
On 5/5/2025 12:04 AM, Bagas Sanjaya wrote:
> On Sat, May 03, 2025 at 12:07:57AM -0400, Joel Fernandes wrote:
>> +.. SPDX-License-Identifier: GPL-2.0
>> +==================================
>> +Device Initialization (devinit)
>> +==================================
>
> Separate SPDX line from title heading.
>
>> +These low-level GPU firmware components are typically:
>> +1. Located in the VBIOS ROM in the same ROM partition (see vbios.rst and fwsec.rst).
>> +2. Executed in sequence on different microcontrollers:
>> + - The devinit engine typically but not necessarily runs on the PMU.
>> + - On an Ampere GPU, the FWSEC typically runs on the GSP (GPU System Processor) in
>> + heavy-secure mode.
>
> Please separate numbered list from preceding sentence, and the bullet sublist
> from parent numbered list by a line.
>
>> +Runtime Considerations
>> +---------------------
>> <snipped>...
>> +Security and Access Control
>> +--------------------------
>
> Match section underline length with the text.
Thanks a lot for finding these issues. I have fixed everything and will re-post
patches end of the week. Please let me know if I can add your Tested-by tag.
Also here is my tree with the changes for a preview (top 7 patches):
https://git.kernel.org/pub/scm/linux/kernel/git/jfern/linux.git/log/?h=nova-docs
thanks,
- Joel
^ permalink raw reply [flat|nested] 13+ messages in thread
* [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon
[not found] <20250503040802.1411285-1-joelagnelf@nvidia.com>
` (2 preceding siblings ...)
2025-05-03 4:07 ` [PATCH v2 5/7] docs: nova-core: Document devinit process Joel Fernandes
@ 2025-05-03 4:07 ` Joel Fernandes
2025-05-05 4:14 ` Bagas Sanjaya
3 siblings, 1 reply; 13+ messages in thread
From: Joel Fernandes @ 2025-05-03 4:07 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, Joel Fernandes, linux-doc
Instances of the Falcon microcontroller appear in modern Nvidia GPUs and
are crucial to the GPU boot process. Document some concepts which will
make nova-core boot code easier to digest. All the information is
derived from public sources such as public documents, OpenRM and Nouveau
code.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/falcon.rst | 156 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 1 +
2 files changed, 157 insertions(+)
create mode 100644 Documentation/gpu/nova/core/falcon.rst
diff --git a/Documentation/gpu/nova/core/falcon.rst b/Documentation/gpu/nova/core/falcon.rst
new file mode 100644
index 000000000000..f2b89cc2a159
--- /dev/null
+++ b/Documentation/gpu/nova/core/falcon.rst
@@ -0,0 +1,156 @@
+==============================
+Falcon (FAst Logic Controller)
+==============================
+The following sections describe the Falcon core and the ucode running on it.
+The descriptions are based on the Ampere GPU or earlier designs; however, they
+should mostly apply to future designs as well, but everything is subject to
+change. The overview provided here is mainly tailored towards understanding the
+interactions of nova-core driver with the Falcon.
+
+NVIDIA GPUs embed small RISC-like microcontrollers called Falcon cores, which
+handle secure firmware tasks, initialization, and power management. Modern
+NVIDIA GPUs may have multiple such Falcon instances (e.g., GSP (the GPU system
+processor) and SEC2 (the security engine)) and also may integrate a RISC-V core.
+This core is capable of running both RISC-V and Falcon code.
+
+The code running on the Falcons is also called Ucode and will be referred to as
+such in the following sections.
+
+Falcons have separate instruction and data memories (IMEM/DMEM) and provide a
+small DMA engine (via the FBIF - "Frame Buffer Interface") to load code from
+system memory. The nova-core driver must reset and configure the Falcon, load
+its firmware via DMA, and start its CPU.
+
+Falcon security levels
+======================
+Falcons can run in Non-secure (NS), Light Secure (LS), or Heavy Secure (HS)
+modes.
+
+Heavy Secured (HS) also known as Privilege Level 3 (PL3)
+--------------------------------------------------------
+HS ucode is the most trusted code and has access to pretty much everything on
+the chip. The HS binary includes a signature in it which is verified at boot.
+This signature verification is done by the hardware itself, thus establishing a
+root of trust. For example, the FWSEC-FRTS command (see fwsec.rst) runs on the
+GSP in HS mode. FRTS, which involves setting up and loading content into the WPR
+(Write Protect Region), has to be done by the HS ucode and cannot be done by the
+host CPU or LS ucode.
+
+Light Secured (LS or PL2) and Non Secured (NS or PL0)
+-----------------------------------------------------
+These modes are less secure than HS. Like HS, the LS or NS ucode binary also
+typically includes a signature in it. To load firmware in LS or NS mode onto a
+Falcon, another Falcon needs to be running in HS mode, which also establishes the
+root of trust. For example, in the case of an Ampere GPU, the CPU runs the "Booter"
+ucode in HS mode on the SEC2 Falcon, which then authenticates and runs the
+run-time GSP binary (GSP-RM) in LS mode on the GSP Falcon. Similarly, as an
+example, after reset on an Ampere, FWSEC runs on the GSP which then loads the
+devinit engine onto the PMU in LS mode.
+
+Root of trust establishment
+---------------------------
+To establish a root of trust, the code running of a Falcon has to be something
+that that cannot be erased and is hardwired into a read-only-memory (ROM). This
+follows industry norms for verification of firmware. This code is called the
+Boot ROM (BROM). The nova-core driver on the CPU communicates with Falcon's Boot
+ROM through various Falcon registers prefixed with "BROM" (see regs.rs).
+
+After nova-core driver reads the necessary ucode from VBIOS, it programs the
+BROM and DMA registers to trigger the Falcon to load the HS ucode from the system
+memory into the Falcon's IMEM/DMEM. Once the HS ucode is loaded, it is verified
+by the Falcon's Boot ROM.
+
+Once the verified HS code is running on a Falcon, it can verify and load other
+LS/NS ucode binaries onto other Falcons and start them. The process of signature
+verification is the same as HS; just in this case, the hardware (BROM) doesn't
+compute the signature, but the HS ucode does.
+
+Thus the root of trust is:
+ Hardware (Boot ROM running on the Falcon) -> HS ucode -> LS/NS ucode.
+
+Example on Ampere GPU, the boot verification flow is:
+ Hardware (Boot ROM running on the SEC2) ->
+ HS ucode (Booter running on the SEC2) ->
+ LS ucode (GSP-RM running on the GSP)
+
+.. note::
+ While the CPU can load HS ucode onto a Falcon microcontroller and have it
+ verified by the hardware and run, the CPU itself typically does not load
+ LS or NS ucode and run it. Loading of LS or NS ucode is done mainly by the
+ HS ucode. For example, on an Ampere GPU, after the Booter ucode runs on the
+ SEC2 in HS mode and loads the GSP-RM binary onto the GSP, it needs to run
+ the "SEC2-RTOS" ucode at runtime. This presents a problem where there is
+ no one to load the SEC2-RTOS ucode onto the SEC2 (i.e., the CPU is incapable
+ of loading LS code, and GSP-RM has to run LS mode). To overcome this,
+ the GSP is temporarily made to run HS ucode (which is itself loaded by
+ the CPU via the nova-core driver using a "GSP-provided sequencer")
+ which then loads the SEC2-RTOS ucode onto the SEC2 in LS mode. The GSP
+ then resumes running its own GSP-RM LS ucode.
+
+Falcon memory subsystem and DMA engine
+======================================
+Falcons have separate instruction and data memories (IMEM/DMEM)
+and contains a small DMA engine called FBDMA (Framebuffer DMA) which does
+DMA transfers to/from the IMEM/DMEM memory inside the Falcon via the FBIF
+(Framebuffer Interface), to external memory.
+
+DMA transfers are possible from the Falcon's memory to both the system memory
+and the framebuffer memory (VRAM).
+
+To perform a DMA via the FBDMA, the FBIF is configured to decide how the memory
+is accessed (also known as aperture type). In the nova-core driver, this is
+determined by the `FalconFbifTarget` enum.
+
+The IO-PMP block (Input/Output Physical Memory Protection) unit in the Falcon
+controls access by the FBDMA to the external memory.
+
+Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:
+
+ External Memory (Framebuffer / System DRAM)
+ ▲ │
+ │ │
+ │ ▼
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━┻━━━━━━━━━━━━━━━━━━━━━━┓
+┃ │ ┃
+┃ ┏━━━━━━━━━━━━━━━┓ │ ┃
+┃ ┃ FBIF ┣━━━━━━━┛ ┃ FALCON
+┃ ┃ (FrameBuffer ┃ Memory Interface ┃ PROCESSOR
+┃ ┃ InterFace) ┃ ┃
+┃ ┃ Apertures ┃ ┃
+┃ ┃ Configures ┃ ┃
+┃ ┃ mem access ┃ ┃
+┃ ┗━━━━━━━▲━━━━━━━┛ ┃
+┃ │ ┃
+┃ │ FBDMA uses configured FBIF apertures ┃
+┃ │ to access External Memory
+┃ │
+┃ ┏━━━━━━━▼━━━━━━━┓ ┏━━━━━━━━━━━━━━━━┓
+┃ ┃ FBDMA ┃ cfg ┃ RISC ┃
+┃ ┃ (FrameBuffer ┣<────>┫ CORE ┣─────>. Direct Core Access
+┃ ┃ DMA Engine) ┃ ┃ ┃ ┃
+┃ ┃ - Master dev. ┃ ┃ (can run both ┃ ┃
+┃ ┗━━━━━━━▲━━━━━━━┛ ┃ Falcon and ┃ ┃
+┃ │ cfg-->┃ RISC-V code) ┃ ┃
+┃ │ / ┃ ┃ ┃
+┃ │ | ┗━━━━━━━━━━━━━━━━┛ ┃ ┏━━━━━━━━━━━━┓
+┃ │ │ ┃ ┃ BROM ┃
+┃ │ │ <───>┫ (Boot ROM) ┃
+┃ │ / ┃ ┗━━━━━━━━━━━━┛
+┃ │ ▼ ┃
+┃ ┏━━━━━━━━━━━━━━━┓ ┃
+┃ ┃ IO-PMP ┃ Controls access by FBDMA ┃
+┃ ┃ (IO Physical ┃ and other IO Masters ┃
+┃ ┃ Memory Protect) ┃
+┃ ┗━━━━━━━▲━━━━━━━┛ ┃
+┃ │ ┃
+┃ │ Protected Access Path for FBDMA ┃
+┃ ▼ ┃
+┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃
+┃ ┃ Memory ┃ ┃
+┃ ┃ ┏━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━┓ ┃ ┃
+┃ ┃ ┃ IMEM ┃ ┃ DMEM ┃ ┃<─────┛
+┃ ┃ ┃ (Instruction ┃ ┃ (Data ┃ ┃
+┃ ┃ ┃ Memory) ┃ ┃ Memory) ┃ ┃
+┃ ┃ ┗━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━┛ ┃
+┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
+┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 301435c5cf67..75a98ab63055 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -28,6 +28,7 @@ vGPU manager VFIO driver and the nova-drm driver.
core/guidelines
core/vbios
+ core/falcon
core/fwsec
core/devinit
core/todo
--
2.43.0
^ permalink raw reply related [flat|nested] 13+ messages in thread* Re: [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon
2025-05-03 4:07 ` [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon Joel Fernandes
@ 2025-05-05 4:14 ` Bagas Sanjaya
2025-05-05 21:37 ` Joel Fernandes
0 siblings, 1 reply; 13+ messages in thread
From: Bagas Sanjaya @ 2025-05-05 4:14 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, David Airlie,
Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
[-- Attachment #1: Type: text/plain, Size: 4401 bytes --]
On Sat, May 03, 2025 at 12:07:58AM -0400, Joel Fernandes wrote:
> +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:
> +
> + External Memory (Framebuffer / System DRAM)
> + ▲ │
> + │ │
> + │ ▼
> +┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━┻━━━━━━━━━━━━━━━━━━━━━━┓
> +┃ │ ┃
> +┃ ┏━━━━━━━━━━━━━━━┓ │ ┃
> +┃ ┃ FBIF ┣━━━━━━━┛ ┃ FALCON
> +┃ ┃ (FrameBuffer ┃ Memory Interface ┃ PROCESSOR
> +┃ ┃ InterFace) ┃ ┃
> +┃ ┃ Apertures ┃ ┃
> +┃ ┃ Configures ┃ ┃
> +┃ ┃ mem access ┃ ┃
> +┃ ┗━━━━━━━▲━━━━━━━┛ ┃
> +┃ │ ┃
> +┃ │ FBDMA uses configured FBIF apertures ┃
> +┃ │ to access External Memory
> +┃ │
> +┃ ┏━━━━━━━▼━━━━━━━┓ ┏━━━━━━━━━━━━━━━━┓
> +┃ ┃ FBDMA ┃ cfg ┃ RISC ┃
> +┃ ┃ (FrameBuffer ┣<────>┫ CORE ┣─────>. Direct Core Access
> +┃ ┃ DMA Engine) ┃ ┃ ┃ ┃
> +┃ ┃ - Master dev. ┃ ┃ (can run both ┃ ┃
> +┃ ┗━━━━━━━▲━━━━━━━┛ ┃ Falcon and ┃ ┃
> +┃ │ cfg-->┃ RISC-V code) ┃ ┃
> +┃ │ / ┃ ┃ ┃
> +┃ │ | ┗━━━━━━━━━━━━━━━━┛ ┃ ┏━━━━━━━━━━━━┓
> +┃ │ │ ┃ ┃ BROM ┃
> +┃ │ │ <───>┫ (Boot ROM) ┃
> +┃ │ / ┃ ┗━━━━━━━━━━━━┛
> +┃ │ ▼ ┃
> +┃ ┏━━━━━━━━━━━━━━━┓ ┃
> +┃ ┃ IO-PMP ┃ Controls access by FBDMA ┃
> +┃ ┃ (IO Physical ┃ and other IO Masters ┃
> +┃ ┃ Memory Protect) ┃
> +┃ ┗━━━━━━━▲━━━━━━━┛ ┃
> +┃ │ ┃
> +┃ │ Protected Access Path for FBDMA ┃
> +┃ ▼ ┃
> +┃ ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃
> +┃ ┃ Memory ┃ ┃
> +┃ ┃ ┏━━━━━━━━━━━━━━━┓ ┏━━━━━━━━━━━━━┓ ┃ ┃
> +┃ ┃ ┃ IMEM ┃ ┃ DMEM ┃ ┃<─────┛
> +┃ ┃ ┃ (Instruction ┃ ┃ (Data ┃ ┃
> +┃ ┃ ┃ Memory) ┃ ┃ Memory) ┃ ┃
> +┃ ┃ ┗━━━━━━━━━━━━━━━┛ ┗━━━━━━━━━━━━━┛ ┃
> +┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
> +┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Wrap the diagram in literal code block. Also, note that it can look messy in
htmldocs output with certain fonts due to box drawing characters. Consider using
ASCII constructs (dashes and vertical bars) instead.
Thanks.
--
An old man doll... just what I always wanted! - Clara
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon
2025-05-05 4:14 ` Bagas Sanjaya
@ 2025-05-05 21:37 ` Joel Fernandes
0 siblings, 0 replies; 13+ messages in thread
From: Joel Fernandes @ 2025-05-05 21:37 UTC (permalink / raw)
To: Bagas Sanjaya, linux-kernel, Danilo Krummrich, David Airlie,
Simona Vetter, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
rust-for-linux, linux-doc
On 5/5/2025 12:14 AM, Bagas Sanjaya wrote:
> On Sat, May 03, 2025 at 12:07:58AM -0400, Joel Fernandes wrote:
>> +Conceptual diagram (not exact) of the Falcon and its memory subsystem is as follows:
>> +
>> + External Memory (Framebuffer / System DRAM)
[...]
>
> Wrap the diagram in literal code block. Also, note that it can look messy in
> htmldocs output with certain fonts due to box drawing characters. Consider using
> ASCII constructs (dashes and vertical bars) instead.
>
Done, thanks.
- Joel
^ permalink raw reply [flat|nested] 13+ messages in thread