From: Zhi Wang <zhiw@nvidia.com>
To: Joel Fernandes <joelagnelf@nvidia.com>
Cc: <linux-kernel@vger.kernel.org>,
Danilo Krummrich <dakr@kernel.org>,
"David Airlie" <airlied@gmail.com>,
Simona Vetter <simona@ffwll.ch>,
"Maarten Lankhorst" <maarten.lankhorst@linux.intel.com>,
Maxime Ripard <mripard@kernel.org>,
Thomas Zimmermann <tzimmermann@suse.de>,
"Jonathan Corbet" <corbet@lwn.net>,
<nouveau@lists.freedesktop.org>,
<dri-devel@lists.freedesktop.org>,
Alexandre Courbot <acourbot@nvidia.com>,
John Hubbard <jhubbard@nvidia.com>,
Shirish Baskaran <sbaskaran@nvidia.com>,
Alistair Popple <apopple@nvidia.com>,
Timur Tabi <ttabi@nvidia.com>, "Ben Skeggs" <bskeggs@nvidia.com>,
<rust-for-linux@vger.kernel.org>, <linux-doc@vger.kernel.org>
Subject: Re: [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout
Date: Tue, 6 May 2025 19:26:41 +0300 [thread overview]
Message-ID: <20250506192641.7872cbd0@inno-thin-client> (raw)
In-Reply-To: <20250503040802.1411285-5-joelagnelf@nvidia.com>
On Sat, 3 May 2025 00:07:56 -0400
Joel Fernandes <joelagnelf@nvidia.com> wrote:
> Add explanation of fwsec with diagrams. This helps clarify how the
> nova-core falcon boot works.
>
> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> ---
> Documentation/gpu/nova/core/fwsec.rst | 180
> ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst |
> 1 + 2 files changed, 181 insertions(+)
> create mode 100644 Documentation/gpu/nova/core/fwsec.rst
>
> diff --git a/Documentation/gpu/nova/core/fwsec.rst
> b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644
> index 000000000000..bed941ac3f2b
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/fwsec.rst
> @@ -0,0 +1,180 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +=========================
> +FWSEC (Firmware Security)
> +=========================
> +This document briefly/conceptually describes the FWSEC (Firmware
> Security) image +and its role in the GPU boot sequence. As such, this
> information is subject to +change in the future and is only current
> as of the Ampere GPU family. However, +hopefully the concepts
> described will be useful for understanding the kernel code +that
> deals with it. All the information is derived from publicly available
> +sources such as public drivers and documentation. +
> +The role of FWSEC to provide secure boot, it is running in
> Heavy-secure mode. It does +firmware verification after GPU reset and
> load various ucode images on to the other +microcontrollers on the
> GPU such as the PMU and GSP. +
> +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC
> partition of +ROM (see vbios.rst for more details). It contains
> different commands like FRTS +(Firmware Runtime Services) and SB
> (Secure Booting other microcontrollers after +reset and loading them
> with other non-FWSEC ucode). The kernel driver only needs to +to do
> FRTS, since SB is already already after reset by the time the kernel
> driver +is loaded. +
> +The FRTS command carves out the WPR2 region (Write protected region)
> which contains +data data required for power management. Once setup,
repetitive word "data" ^
> only HS mode ucode can +access it (see falcon.rst for privilege
> levels). +
> +The FWSEC image is located in the VBIOS ROM in the partition of the
> ROM that contains +various ucode images (also known as applications)
> -- one of them being FWSEC. For how +it is extracted, see vbios.rst
> and the vbios.rs source code. +
> +The Falcon data for each ucode images (including the FWSEC image) is
> a combination +of headers, data sections (DMEM) and instruction code
> sections (IMEM). All these +ucode images are stored in the same ROM
> partition and the PMU table is used to look +up the application to
> load it based on its application ID (see vbios.rs). +
> +For the purposes of nova-core driver, the FWSEC contains an
> 'application interface' +called DMEMMAPPER which is used to the
> "FWSEC-FRTS" command (among other commands it +is capable of
> executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
> +mode and runs FRTS. +
> +FWSEC Memory Layout
> +-------------------
> +The memory layout of the FWSEC image is as follows (this is using an
> GA-102 +Ampere GPU as an example and could vary for future GPUs and
> is subject to change +completely, it is just provided as an example):
> +
> +Here is a block diagram of the FWSEC memory layout::
> + ┌───────────────────────────────────────────────────────────────┐
> + │ FWSEC ROM image (type 0xE0) │
> + │ │
> + │ ┌─────────────────────────────────┐ │
> + │ │ PMU Falcon Ucode Table │ │
> + │ │ (PmuLookupTable) │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Table Header │ │ │
> + │ │ │ - version: 0x01 │ │ │
> + │ │ │ - header_size: 6 │ │ │
> + │ │ │ - entry_size: 6 │ │ │
> + │ │ │ - entry_count: N │ │ │
> + │ │ │ - desc_version:3(unused)│ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ │ ... │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Entry for FWSEC (0x85) │ │ │
> + │ │ │ (PmuLookupTableEntry) │ │ │
> + │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
> + │ │ │ - target_id: 0x01 (PMU) │ │ │ │
> + │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
> + │ │ └─────────────────────────┘ │ │ │ application. │
> + │ └─────────────────────────────────┘ │ │ │
> + │ │ │ │
> + │ │ │ │
> + │ ┌─────────────────────────────────┐ │ │ │
> + │ │ FWSEC Ucode Component │<───┘ │ │
> + │ │ (aka Falcon data) │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
> + │ │ │ - hdr │ │ │
> + │ │ │ - stored_size │ │ │
> + │ │ │ - pkc_data_offset │ │ │
> + │ │ │ - interface_offset ─────┼────┼────────────────┐ │
> + │ │ │ - imem_phys_base │ │ │ │
> + │ │ │ - imem_load_size │ │ │ │
> + │ │ │ - imem_virt_base │ │ │ │
> + │ │ │ - dmem_phys_base │ │ │ │
> + │ │ │ - dmem_load_size │ │ │ │
> + │ │ │ - engine_id_mask │ │ │ │
> + │ │ │ - ucode_id │ │ │ │
> + │ │ │ - signature_count │ │ look up sig │ │
> + │ │ │ - signature_versions --------------+ │ │
> + │ │ └─────────────────────────┘ │ │ │ │
> + │ │ (no gap) │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │ │
> + │ │ │ Signatures Section │<───┼─────┘ │ │
> + │ │ │ (384 bytes per sig) │ │ │ │
> + │ │ │ - RSA-3K Signature 1 │ │ │ │
> + │ │ │ - RSA-3K Signature 2 │ │ │ │
> + │ │ │ ... │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ IMEM Section (Code) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ Contains instruction │ │ │ │
> + │ │ │ code etc. │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ DMEM Section (Data) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Application │ │<───┼────────────────┘ │
> + │ │ │ │ Interface Table │ │ │ │
> + │ │ │ │ (FalconAppifHdrV1) │ │ │ │
> + │ │ │ │ Header: │ │ │ │
> + │ │ │ │ - version: 0x01 │ │ │ │
> + │ │ │ │ - header_size: 4 │ │ │ │
> + │ │ │ │ - entry_size: 8 │ │ │ │
> + │ │ │ │ - entry_count: N │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ │ Entries: │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
> + │ │ │ │ │ - id: 0x01 │ │ │ │ │
> + │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
> + │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
> + │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
> + │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
> + │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
> + │ │ │ │ Interface │ │ │ │ app. │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
> + │ │ │ │ (FalconAppifDmemmapperV3) │ │
> + │ │ │ │ - signature: "DMAP" │ │ │ │
> + │ │ │ │ - version: 0x0003 │ │ │ │
> + │ │ │ │ - Size: 64 bytes │ │ │ │
> + │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
> + │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
> + │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
> + │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
> + │ │ │ │ - init_cmd │ │ │ │ │ │
> + │ │ │ │ - features │ │ │ │ │ │
> + │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │ │
> + │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
> + │ │ │ │ - Command data │ │ │ │ │
> + │ │ │ │ - Arguments │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Command Output │<┼────┼──────────────────┘ │
> + │ │ │ │ Buffer │ │ │ │
> + │ │ │ │ - Results │ │ │ │
> + │ │ │ │ - Status │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ └─────────────────────────────────┘ │
> + │ │
> + └───────────────────────────────────────────────────────────────┘
> +
> +.. note::
> + The FWSEC image also plays a role in memory scrubbing (ECC
> initialization) and VPR
> + (Video Protected Region) initialization as well. Before the
> nova-core driver is even
> + loaded, the FWSEC image is running on the GSP in heavy-secure
> mode. After the devinit
> + sequence completes, it does VRAM memory scrubbing (ECC
> initialization). On consumer
> + GPUs, it scrubs only part of memory and then initiates 'async
> scrubbing'. Before this
> + async scrubbing completes, the unscrubbed VRAM cannot be used for
> allocation (thus DRM
> + memory allocators need to wait for this scrubbing to complete).
> \ No newline at end of file
> diff --git a/Documentation/gpu/nova/index.rst
> b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0
> 100644 --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
>
> core/guidelines
> core/vbios
> + core/fwsec
> core/todo
next prev parent reply other threads:[~2025-05-06 16:27 UTC|newest]
Thread overview: 21+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-05-03 4:07 [PATCH v2 0/7] Documentation for nova-core Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 1/7] nova-core: doc: Add code comments related to devinit Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 2/7] nova-core: doc: Clarify sysmembar operations Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 3/7] nova-core: docs: Document vbios layout Joel Fernandes
2025-05-05 3:00 ` Bagas Sanjaya
2025-05-05 3:12 ` Bagas Sanjaya
2025-05-03 4:07 ` [PATCH v2 4/7] nova-core: docs: Document fwsec operation and layout Joel Fernandes
2025-05-05 3:52 ` Bagas Sanjaya
2025-05-06 16:26 ` Zhi Wang [this message]
2025-05-09 20:56 ` Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 5/7] docs: nova-core: Document devinit process Joel Fernandes
2025-05-05 4:04 ` Bagas Sanjaya
2025-05-05 22:15 ` Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 6/7] docs: nova-core: Document basics of the Falcon Joel Fernandes
2025-05-05 4:14 ` Bagas Sanjaya
2025-05-05 21:37 ` Joel Fernandes
2025-05-03 4:07 ` [PATCH v2 7/7] gpu: nova-core: Clarify falcon code Joel Fernandes
2025-05-06 16:21 ` Zhi Wang
2025-05-09 20:59 ` Joel Fernandes
2025-06-30 10:37 ` [PATCH v2 0/7] Documentation for nova-core Danilo Krummrich
2025-06-30 11:33 ` Alexandre Courbot
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250506192641.7872cbd0@inno-thin-client \
--to=zhiw@nvidia.com \
--cc=acourbot@nvidia.com \
--cc=airlied@gmail.com \
--cc=apopple@nvidia.com \
--cc=bskeggs@nvidia.com \
--cc=corbet@lwn.net \
--cc=dakr@kernel.org \
--cc=dri-devel@lists.freedesktop.org \
--cc=jhubbard@nvidia.com \
--cc=joelagnelf@nvidia.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=maarten.lankhorst@linux.intel.com \
--cc=mripard@kernel.org \
--cc=nouveau@lists.freedesktop.org \
--cc=rust-for-linux@vger.kernel.org \
--cc=sbaskaran@nvidia.com \
--cc=simona@ffwll.ch \
--cc=ttabi@nvidia.com \
--cc=tzimmermann@suse.de \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.