* [PATCH 1/6] nova-core: doc: Add code comments related to devinit
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
@ 2025-04-23 22:53 ` Joel Fernandes
2025-04-23 22:53 ` [PATCH 2/6] nova-core: doc: Clarify sysmembar operations Joel Fernandes
` (5 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:53 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes
Add several code comments to reduce acronym soup and explain how devinit magic
and bootflow works before driver loads. These are essential for debug and
development of the nova driver.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
drivers/gpu/nova-core/devinit.rs | 36 ++++++++++++++++++++++++++++----
drivers/gpu/nova-core/regs.rs | 17 +++++++++++++--
2 files changed, 47 insertions(+), 6 deletions(-)
diff --git a/drivers/gpu/nova-core/devinit.rs b/drivers/gpu/nova-core/devinit.rs
index ee5685aff845..890bdc4d9522 100644
--- a/drivers/gpu/nova-core/devinit.rs
+++ b/drivers/gpu/nova-core/devinit.rs
@@ -1,6 +1,21 @@
// SPDX-License-Identifier: GPL-2.0
//! Methods for device initialization.
+//!
+//! A clarification about devinit terminology:
+//! devinit is a sequence of register read/writes after reset that performs tasks
+//! such as:
+//! 1. Programming VRAM memory controller timings.
+//! 2. Power sequencing.
+//! 3. Clock and PLL configuration.
+//! 4. Thermal management.
+//! 5. Performs VRAM memory scrubbing (ECC initialization) - on some GPUs it scrubs
+//! only part of memory and then kicks of 'async scrubbing'.
+//!
+//! devinit itself is a 'script' which is interpreted by the PMU microcontroller of
+//! the GPU by an interpreter program.
+//!
+//! Note that the devinit sequence also needs to run during suspend/resume at runtime.
use kernel::bindings;
use kernel::devres::Devres;
@@ -9,15 +24,28 @@
use crate::driver::Bar0;
use crate::regs;
-/// Wait for devinit FW completion.
+/// Wait for gfw (GPU firmware) boot completion signal (GFW_BOOT).
///
-/// Upon reset, the GPU runs some firmware code to setup its core parameters. Most of the GPU is
-/// considered unusable until this step is completed, so it must be waited on very early during
-/// driver initialization.
+/// Upon reset, several microcontrollers (such as PMU, SEC2, GSP etc) on the GPU run some GPU
+/// firmware (gfw) code to setup its core parameters. Most of the GPU is considered unusable until
+/// this step is completed, so it must be waited on very early during driver initialization.
+///
+/// The GPU firmware (gfw) code includes several components that execute before the driver loads.
+/// These components are located in the VBIOS ROM and are executed in a sequence on these different
+/// microcontrollers. The devinit sequence itself runs on the PMU, and the FWSEC runs on the GSP.
+///
+/// This function specifically waits for a signal indicating core initialization is complete before
+/// which not much can be done. This signal is setup by the FWSEC running on the GSP in high secure
+/// mode.
pub(crate) fn wait_gfw_boot_completion(bar: &Devres<Bar0>) -> Result<()> {
let mut timeout = 2000;
loop {
+ // Before accessing the completion status in PGC6_AON_SECURE_SCRATCH_GROUP_05, we must first
+ // check PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK. This is because the register is
+ // accessible only after secure firmware (FWSEC) lowers the privilege level to allow CPU
+ // (LS/low-secure) access. We can only safely read the status register from CPU (LS/low-secure)
+ // once the mask indicates the privilege level has been appropriately lowered.
let gfw_booted = with_bar!(
bar,
|b| regs::Pgc6AonSecureScratchGroup05PrivLevelMask::read(b)
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index eae5b7c13155..f4a6a382e83f 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -59,13 +59,26 @@
31:4 hi_val => as u32
);
-/* GC6 */
-
+/*
+ * GC6:
+ *
+ * GC6 is a GPU low-power state where VRAM is in self-refresh and the GPU
+ * is powered down (except for power rails needed to keep self-refresh working).
+ *
+ * These scratch registers are "always-on" even in a low-power state and have a
+ * designated group number.
+ */
+
+/// Privilege level mask register (PGC6_AON_SECURE_SCRATCH_GROUP_05_PRIV_LEVEL_MASK)
+/// which dictates whether the host CPU has privilege to access to the
+/// PGC6_AON_SECURE_SCRATCH_GROUP_05 register.
register!(Pgc6AonSecureScratchGroup05PrivLevelMask@0x00118128;
0:0 read_protection_level0_enabled => as_bit bool
);
/* TODO: This is an array of registers. */
+/// PGC6_AON_SECURE_SCRATCH_GROUP_05 scratch register used to check for
+/// GFW boot completion status.
register!(Pgc6AonSecureScratchGroup05@0x00118234;
31:0 value => as u32
);
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 2/6] nova-core: doc: Clarify sysmembar operations
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
2025-04-23 22:53 ` [PATCH 1/6] nova-core: doc: Add code comments related to devinit Joel Fernandes
@ 2025-04-23 22:53 ` Joel Fernandes
2025-04-23 22:53 ` [PATCH 3/6] nova-core: docs: Document vbios layout Joel Fernandes
` (4 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:53 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes
sysmembar is a critical operation that the GSP falcon needs to perform
in the reset sequence. Add some code comments to clarify.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
drivers/gpu/nova-core/gpu.rs | 11 ++++++++++-
drivers/gpu/nova-core/regs.rs | 2 ++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 4d03a0b11b64..61031bccb7d3 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -158,6 +158,9 @@ pub(crate) struct Gpu {
/// MMIO mapping of PCI BAR 0
bar: Devres<Bar0>,
fw: Firmware,
+ /// A system memory page for sysmembar (A GPU-initiated hardware memory-barrier
+ /// operation that flushes all pending GPU-side memory writes that were done
+ /// through PCIE, to system memory).
sysmem_flush: DmaObject,
timer: Timer,
bios: Vbios,
@@ -204,7 +207,13 @@ pub(crate) fn new(
devinit::wait_gfw_boot_completion(&bar)
.inspect_err(|_| pr_err!("GFW boot did not complete"))?;
- // System memory page required for sysmembar to properly flush into system memory.
+ // System memory page required for sysmembar which is a GPU-initiated hardware
+ // memory-barrier operation that flushes all pending GPU-side memory writes that
+ // were done through PCIE, to system memory. It is required for Falcon to be reset
+ // as the reset operation involves a reset handshake. When the falcon acks the
+ // reset, it writes its acknowledgement into system memory, but for this write to
+ // be visible to the host, the falcon needs to do sysmembar to flush
+ // its writes and prevent the driver from timing out.
let sysmem_flush = {
let page = DmaObject::new(
pdev.as_ref(),
diff --git a/drivers/gpu/nova-core/regs.rs b/drivers/gpu/nova-core/regs.rs
index f4a6a382e83f..22906ab1a43a 100644
--- a/drivers/gpu/nova-core/regs.rs
+++ b/drivers/gpu/nova-core/regs.rs
@@ -37,6 +37,8 @@
/* PFB */
+/// These two registers together hold the physical system memory address
+/// that is used by the GPU for perform sysmembar operation (see gpu.rs).
register!(PfbNisoFlushSysmemAddr@0x00100c10;
31:0 adr_39_08 => as u32
);
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 3/6] nova-core: docs: Document vbios layout
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
2025-04-23 22:53 ` [PATCH 1/6] nova-core: doc: Add code comments related to devinit Joel Fernandes
2025-04-23 22:53 ` [PATCH 2/6] nova-core: doc: Clarify sysmembar operations Joel Fernandes
@ 2025-04-23 22:53 ` Joel Fernandes
2025-04-23 23:12 ` Timur Tabi
2025-04-23 22:54 ` [PATCH 4/6] nova-core: docs: Document fwsec operation and layout Joel Fernandes
` (3 subsequent siblings)
6 siblings, 1 reply; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:53 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, Maarten Lankhorst, Maxime Ripard,
Thomas Zimmermann, David Airlie, Simona Vetter, Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes, linux-doc
Add detailed explanation and block diagrams of the layout of the vBIOS
on Nvidia GPUs. This is important to understand how nova-core boots an
Nvidia GPU.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/vbios.rst | 154 ++++++++++++++++++++++++++
1 file changed, 154 insertions(+)
create mode 100644 Documentation/gpu/nova/core/vbios.rst
diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
new file mode 100644
index 000000000000..17411f21b410
--- /dev/null
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -0,0 +1,154 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+==========
+VBIOS
+==========
+
+This document describes the layout of the VBIOS image which is a series of concatenated
+images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
+by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
+boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
+the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
+
+The format of the images in the ROM follow the "BIOS Specification" part of the
+PCI specification, with Nvidia-specific extensions.
+
+As an example, the following are the different image types that can be found in the
+VBIOS of an Ampere GA102 GPU which is supported by the nova-core driver.
+
+- PciAt Image (Type 0x00) - This is the standard PCI BIOS image who's naming likely
+ comes from the "IBM PC/AT" architecture.
+
+- EFI Image (Type 0x03) - This is the EFI BIOS image. It contains the UEFI GOP
+ driver that is used to display UEFI graphics output.
+
+- First FwSec Image (Type 0xE0) - The first FwSec image (Secure Firmware)
+
+- Second FwSec Image (Type 0xE0) - The second FwSec image (Secure Firmware) contains
+ various microcodes that do a range of different functions - all in high secure mode
+ where they are allowed to perform highly privileged register accesses that the CPU
+ cannot do. The 2 most important parts (also known as Application Interfaces) of this
+ are:
+ 1. The devinit engine - this is loaded ontop the PMU before the driver loads
+ and interprets devinit commands which perform critical hardware initialization.
+ This will be described in a separate document.
+ 2. The DMEM mapper - this is loaded onto the GSP and one of the commands it
+ performs is carving out the WPR2 area (Write protected region) and placing
+ important data called 'FRTS' into it which contains things like voltage/frequency
+ curves etc.
+
+It is not clear why FwSec has 2 different images, but they both are of type 0xE0
+and can be identified as such.
+
+VBIOS ROM Layout
+----------------
+
+The VBIOS layout is roughly a series of concatenated images as follows:
+(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
+
+┌────────────────────────────────────────────────────────────────────────┐
+│ VBIOS (Starting at ROM_OFFSET: 0x300000) │
+├────────────────────────────────────────────────────────────────────────┤
+│ ┌───────────────────────────────────────────────┐ │
+│ │ PciAt Image (Type 0x00) │ │
+│ ├───────────────────────────────────────────────┤ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ ROM Header │ │ │
+│ │ │ (Signature 0xAA55)│ │ │
+│ │ └───────────────────┘ │ │
+│ │ │ rom header's pci_data_struct_offset │ │
+│ │ │ points to the PCIR structure │ │
+│ │ V │ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ PCIR Structure │ │ │
+│ │ │ (Signature "PCIR")│ │ │
+│ │ │ last_image: 0x80 │ │ │
+│ │ │ image_len: size │ │ │
+│ │ │ in 512-byte units │ │ │
+│ │ └───────────────────┘ │ │
+│ │ │ │ │
+│ │ │ NPDE immediately follows PCIR │ │
+│ │ V │ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ NPDE Structure │ │ │
+│ │ │ (Signature "NPDE")│ │ │
+│ │ │ last_image: 0x00 │ │ │
+│ │ └───────────────────┘ │ │
+│ │ │ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ BIT Header │ (Signature scanning │ │
+│ │ │ (Signature "BIT") │ provides the location │ │
+│ │ └───────────────────┘ of the BIT table) │ │
+│ │ │ header is │ │
+│ │ | followed by a table of tokens │ │
+│ │ V one of which is for falcon data. │ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ BIT Tokens │ │ │
+| | | ______________ | | |
+│ │ │ │ Falcon Data │ │ │ │
+│ │ │ │ Token (0x70)│---+------------>------------┼──+ │
+│ │ │ └─────────────┘ │ falcon_data_ptr() │ │ │
+│ │ └───────────────────┘ │ V │
+│ └───────────────────────────────────────────────┘ │ │
+│ (no gap between images) │ │
+│ ┌───────────────────────────────────────────────┐ │ │
+│ │ EFI Image (Type 0x03) │ │ │
+│ ├───────────────────────────────────────────────┤ │ │
+| | Contains the UEFI GOP driver (Graphics Output)| | |
+│ │ ┌───────────────────┐ │ │ │
+│ │ │ ROM Header │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ PCIR Structure │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ NPDE Structure │ │ │ │
+│ │ └───────────────────┘ │ │ │
+│ │ │ Image data │ │ │ │
+│ │ └───────────────────┘ │ │ │
+│ └───────────────────────────────────────────────┘ │ │
+│ (no gap between images) │ │
+│ ┌───────────────────────────────────────────────┐ │ │
+│ │ First FwSec Image (Type 0xE0) │ │ │
+│ ├───────────────────────────────────────────────┤ │ │
+│ │ ┌───────────────────┐ │ │ │
+│ │ │ ROM Header │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ PCIR Structure │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ NPDE Structure │ │ │ │
+│ │ └───────────────────┘ │ │ │
+│ │ │ Image data │ │ │ │
+│ │ └───────────────────┘ │ │ │
+│ └───────────────────────────────────────────────┘ │ │
+│ (no gap between images) │ │
+│ ┌───────────────────────────────────────────────┐ │ │
+│ │ Second FwSec Image (Type 0xE0) │ │ │
+│ ├───────────────────────────────────────────────┤ │ │
+│ │ ┌───────────────────┐ │ │ │
+│ │ │ ROM Header │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ PCIR Structure │ │ │ │
+│ │ +───────────────────+ │ │ │
+│ │ │ NPDE Structure │ │ │ │
+│ │ └───────────────────┘ │ │ │
+│ │ │ │ │
+│ │ ┌───────────────────┐ │ │ │
+│ │ │ PMU Lookup Table │ <- falcon_data_offset │<─┘ │
+│ │ │ ┌─────────────┐ │ pmu_lookup_table │ │
+│ │ │ │ Entry 0x85 │ │ │ │
+│ │ │ │ FWSEC_PROD │ │ │ │
+│ │ │ └─────────────┘ │ │ │
+│ │ └───────────────────┘ │ │
+│ │ │ │ │
+│ │ │ points to │ │
+│ │ V │ │
+│ │ ┌───────────────────┐ │ │
+│ │ │ FalconUCodeDescV3 │ <- falcon_ucode_offset │ │
+│ │ │ (FWSEC Firmware) │ fwsec_header() │ │
+│ │ └───────────────────┘ │ │
+│ │ │ immediately followed by... │ │
+│ │ V │ │
+│ │ ┌────────────────────────────┐ │ │
+│ │ │ Signatures + FWSEC Ucode │ │ │
+│ │ │ fwsec_sigs(), fwsec_ucode()│ │ │
+│ │ └────────────────────────────┘ │ │
+│ └───────────────────────────────────────────────┘______________________│
+
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 3/6] nova-core: docs: Document vbios layout
2025-04-23 22:53 ` [PATCH 3/6] nova-core: docs: Document vbios layout Joel Fernandes
@ 2025-04-23 23:12 ` Timur Tabi
2025-04-23 23:30 ` Joel Fernandes
0 siblings, 1 reply; 14+ messages in thread
From: Timur Tabi @ 2025-04-23 23:12 UTC (permalink / raw)
To: corbet@lwn.net, tzimmermann@suse.de, simona@ffwll.ch,
airlied@gmail.com, linux-kernel@vger.kernel.org, dakr@kernel.org,
Joel Fernandes, maarten.lankhorst@linux.intel.com,
mripard@kernel.org
Cc: Shirish Baskaran, Alistair Popple, John Hubbard,
dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org,
linux-doc@vger.kernel.org, Ben Skeggs, Alexandre Courbot
On Wed, 2025-04-23 at 18:53 -0400, Joel Fernandes wrote:
> +This document describes the layout of the VBIOS image which is a series of concatenated
> +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
> +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
> +boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
> +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
You might want to add a link to this page:
https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/6] nova-core: docs: Document vbios layout
2025-04-23 23:12 ` Timur Tabi
@ 2025-04-23 23:30 ` Joel Fernandes
0 siblings, 0 replies; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 23:30 UTC (permalink / raw)
To: Timur Tabi, corbet@lwn.net, tzimmermann@suse.de, simona@ffwll.ch,
airlied@gmail.com, linux-kernel@vger.kernel.org, dakr@kernel.org,
maarten.lankhorst@linux.intel.com, mripard@kernel.org
Cc: Shirish Baskaran, Alistair Popple, John Hubbard,
dri-devel@lists.freedesktop.org, nouveau@lists.freedesktop.org,
linux-doc@vger.kernel.org, Ben Skeggs, Alexandre Courbot
On 4/23/2025 7:12 PM, Timur Tabi wrote:
> On Wed, 2025-04-23 at 18:53 -0400, Joel Fernandes wrote:
>> +This document describes the layout of the VBIOS image which is a series of concatenated
>> +images in the ROM of the GPU. The VBIOS is mirrored onto the BAR 0 space and is read
>> +by both Boot ROM firmware (also known as IFR or init-from-rom firmware) on the GPU to
>> +boot strap various microcontrollers (PMU, SEC, GSP) with critical initialization before
>> +the driver loads, as well as by the nova-core driver in the kernel to boot the GSP.
>
> You might want to add a link to this page:
>
> https://download.nvidia.com/open-gpu-doc/BIOS-Information-Table/1/BIOS-Information-Table.html
Good idea, will do!
- Joel
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 4/6] nova-core: docs: Document fwsec operation and layout
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
` (2 preceding siblings ...)
2025-04-23 22:53 ` [PATCH 3/6] nova-core: docs: Document vbios layout Joel Fernandes
@ 2025-04-23 22:54 ` Joel Fernandes
2025-04-23 22:54 ` [PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1 Joel Fernandes
` (2 subsequent siblings)
6 siblings, 0 replies; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:54 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes, linux-doc
Add explanation of fwsec with diagrams. This helps clarify how the
nova-core falcon boot works.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/fwsec.rst | 173 ++++++++++++++++++++++++++
Documentation/gpu/nova/core/vbios.rst | 3 +-
Documentation/gpu/nova/index.rst | 1 +
3 files changed, 176 insertions(+), 1 deletion(-)
create mode 100644 Documentation/gpu/nova/core/fwsec.rst
diff --git a/Documentation/gpu/nova/core/fwsec.rst b/Documentation/gpu/nova/core/fwsec.rst
new file mode 100644
index 000000000000..03d1f0d67414
--- /dev/null
+++ b/Documentation/gpu/nova/core/fwsec.rst
@@ -0,0 +1,173 @@
+.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
+=========================
+FWSEC (Firmware Security)
+=========================
+
+The role of FWSEC to provide firmware verification and perform secure operations
+such as carving out the WPR2 region (Write protected region). It is critical to
+the boot sequence of the GPU.
+
+The FWSEC image is located in the VBIOS ROM. For how it is extracted, see vbios.rst
+and the vbios.rs source code.
+
+The Falcon data in the FWSEC image is a combination of headers, data sections
+(DMEM) and instruction code sections (IMEM).
+
+FWSEC itself is a central location for all microcodes in the VBIOS that run on
+various microcontrollers (PMU, SEC, GSP etc) some of which (such as devinit) are
+executed after GPU power-on before the nova-core driver is even loaded.
+
+For the purposes of nova-core driver, the FWSEC contains microcode called
+DMEMMAPPER which executes the "FWSEC-FRTS" command (among other commands it is
+capable of executing) that is executed on the GSP to carve out the WPR2 region
+(Write protected region).
+
+FWSEC Memory Layout
+-------------------
+
+The memory layout of the FWSEC image is as follows (this is using an GA-102
+Ampere GPU as an example and could vary for future GPUs):
+
+┌───────────────────────────────────────────────────────────────┐
+│ FWSEC image (type 0xE0) │
+│ │
+│ ┌─────────────────────────────────┐ │
+│ │ PMU Falcon Ucode Table │ │
+│ │ (PmuLookupTable) │ │
+│ │ ┌─────────────────────────┐ │ │
+│ │ │ Table Header │ │ │
+│ │ │ - version: 0x01 │ │ │
+│ │ │ - header_size: 6 │ │ │
+│ │ │ - entry_size: 6 │ │ │
+│ │ │ - entry_count: N │ │ │
+│ │ │ - desc_version:3(unused)│ │ │
+│ │ └─────────────────────────┘ │ │
+│ │ ... │ │
+│ │ ┌─────────────────────────┐ │ │
+│ │ │ Entry for FWSEC (0x85) │ │ │
+│ │ │ (PmuLookupTableEntry) │ │ │
+│ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
+│ │ │ - target_id: 0x01 (PMU) │ │ │ │
+│ │ │ - data: offset ─────────┼────┼────┼───┐ │
+│ │ └─────────────────────────┘ │ │ │ │
+│ └─────────────────────────────────┘ │ │ │
+│ │ │ │
+│ │ │ │
+│ ┌─────────────────────────────────┐ │ │ │
+│ │ FWSEC Ucode Component │<───┘ │ │
+│ │ (aka Falcon data) │ │ │
+│ │ ┌─────────────────────────┐ │ │ │
+│ │ │ FalconUCodeDescV3 │<───┼────────┘ │
+│ │ │ - hdr │ │ │
+│ │ │ - stored_size │ │ │
+│ │ │ - pkc_data_offset │ │ │
+│ │ │ - interface_offset ─────┼────┼────────────────┐ │
+│ │ │ - imem_phys_base │ │ │ │
+│ │ │ - imem_load_size │ │ │ │
+│ │ │ - imem_virt_base │ │ │ │
+│ │ │ - dmem_phys_base │ │ │ │
+│ │ │ - dmem_load_size │ │ │ │
+│ │ │ - engine_id_mask │ │ │ │
+│ │ │ - ucode_id │ │ │ │
+│ │ │ - signature_count │ │ look up sig │ │
+│ │ │ - signature_versions ───┼────┼─────┐using hal │ │
+│ │ └─────────────────────────┘ │ │ │ │
+│ │ (no gap) │ │ │ │
+│ │ ┌─────────────────────────┐ │ │ │ │
+│ │ │ Signatures Section │<───┼─────┘ │ │
+│ │ │ (384 bytes per sig) │ │ │ │
+│ │ │ - RSA-3K Signature 1 │ │ │ │
+│ │ │ - RSA-3K Signature 2 │ │ │ │
+│ │ │ ... │ │ │ │
+│ │ └─────────────────────────┘ │ │ │
+│ │ │ │ │
+│ │ ┌─────────────────────────┐ │ │ │
+│ │ │ IMEM Section (Code) │ │ │ │
+│ │ │ │ │ │ │
+│ │ │ Contains devinit, fwsec │ │ │ │
+│ │ │ instruction code etc. │ │ │ │
+│ │ └─────────────────────────┘ │ │ │
+│ │ │ │ │
+│ │ ┌─────────────────────────┐ │ │ │
+│ │ │ DMEM Section (Data) │ │ │ │
+│ │ │ │ │ │ │
+│ │ │ ┌─────────────────────┐ │ │ │ │
+│ │ │ │ Application │ │<───┼────────────────┘ │
+│ │ │ │ Interface Table │ │ │ │
+│ │ │ │ (FalconAppifHdrV1) │ │ │ │
+│ │ │ │ Header: │ │ │ │
+│ │ │ │ - version: 0x01 │ │ │ │
+│ │ │ │ - header_size: 4 │ │ │ │
+│ │ │ │ - entry_size: 8 │ │ │ │
+│ │ │ │ - entry_count: N │ │ │ │
+│ │ │ │ │ │ │ │
+│ │ │ │ Entries: │ │ │ │
+│ │ │ │ ┌─────────────────┐ │ │ │ │
+│ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
+│ │ │ │ │ - id: 0x01 │ │ │ │ │
+│ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
+│ │ │ │ └─────────────────┘ │ │ │ │
+│ │ │ │ ┌─────────────────┐ │ │ │ │
+│ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
+│ │ │ │ │ - id: 0x04 │ │ │ │ │
+| | | | | (NVFW_FALCON_ | | | | |
+| | | | | APPIF_ID_DMEMMAPPER) | |
+│ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
+│ │ │ │ └─────────────────┘ │ │ │ │ │
+│ │ │ │ ┌─────────────────┐ │ │ │ │ │
+│ │ │ │ │ MULTI_FALCON (5)│ │ │ │ │ │
+│ │ │ │ │ - id: 0x05 │ │ │ │ │ (See note [1]) │
+│ │ │ │ │ - dmemOffset Z ─┼─┼─┼────┼─────┼────────────────┐ │
+│ │ │ │ └─────────────────┘ │ │ │ │ │ │
+│ │ │ └─────────────────────┘ │ │ │ │ │
+│ │ │ │ │ │ │ │
+│ │ │ ┌─────────────────────┐ │ │ │ │ │
+│ │ │ │ DEVINIT Engine │<┼────┘ │ │ │
+│ │ │ │ Interface │ │ │ │ │ │
+│ │ │ └─────────────────────┘ │ │ │ │ │
+│ │ │ │ │ │ │ │
+│ │ │ ┌─────────────────────┐ │ │ │ │ │
+│ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │ │
+│ │ │ │ (FalconAppifDmemmapperV3) | | |
+│ │ │ │ - signature: "DMAP" │ │ │ | |
+│ │ │ │ - version: 0x0003 │ │ │ | |
+│ │ │ │ - Size: 64 bytes │ │ │ | |
+│ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ | |
+│ │ │ │ - cmd_in_buffer_size│ │ │ │ | |
+│ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ | |
+│ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ | |
+│ │ │ │ - init_cmd │ │ │ │ │ | |
+│ │ │ │ - features │ │ │ │ │ | |
+│ │ │ │ - cmd_mask0/1 │ │ │ │ │ V |
+│ │ │ │ - multiTgtTbl-------┼─┼────+────────────+─────+────┐ │
+│ │ │ └─────────────────────┘ │ │ │ │ │ │
+│ │ │ │ │ │ │ │ │
+│ │ │ ┌─────────────────────┐ │ │ │ │ │ │
+│ │ │ │ MULTI_FALCON │<┼────+────────────+─────+────┘ │
+│ │ │ │ - version │ │ │ │ │ |
+│ │ │ │ - targetId (PMU/SEC)│ │ │ │ │ |
+│ │ │ │ - loadType │ │ │ │ │ |
+│ │ │ │ - initStack │ │ │ │ │ |
+│ │ │ └─────────────────────┘ │ │ │ │ |
+│ │ │ │ │ │ │ |
+│ │ │ ┌─────────────────────┐ │ │ │ │ |
+│ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ |
+│ │ │ │ - Command data │ │ │ │ |
+│ │ │ │ - Arguments │ │ │ │ |
+│ │ │ └─────────────────────┘ │ │ │ |
+│ │ │ │ │ │ |
+│ │ │ ┌─────────────────────┐ │ │ │ |
+│ │ │ │ Command Output │<┼────┼──────────────────┘ |
+│ │ │ │ Buffer │ │ │ |
+│ │ │ │ - Results │ │ │ |
+│ │ │ │ - Status │ │ │ |
+│ │ │ └─────────────────────┘ │ │ |
+│ │ └─────────────────────────┘ │ |
+│ └─────────────────────────────────┘ |
+│ │
+└───────────────────────────────────────────────────────────────┘
+
+Note:
+[1] MULTI_FALCON section in DMEM can either be looked up by
+ MULTI_FALCON App Interface, or from multi_tgt_table entry
+ in DMEM_MAPPER's App Interface.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/core/vbios.rst b/Documentation/gpu/nova/core/vbios.rst
index 17411f21b410..da759b56fb2d 100644
--- a/Documentation/gpu/nova/core/vbios.rst
+++ b/Documentation/gpu/nova/core/vbios.rst
@@ -44,7 +44,8 @@ VBIOS ROM Layout
The VBIOS layout is roughly a series of concatenated images as follows:
(For more explanations of acronyms, see the detailed descriptions in vbios.rs).
-
+Note: this diagram is using an GA-102 Ampere GPU as an example and could vary
+ for future GPUs.
┌────────────────────────────────────────────────────────────────────────┐
│ VBIOS (Starting at ROM_OFFSET: 0x300000) │
├────────────────────────────────────────────────────────────────────────┤
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index 2701b3f4af35..c01dcc5657e2 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,4 +27,5 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/fwsec
core/todo
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* [PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
` (3 preceding siblings ...)
2025-04-23 22:54 ` [PATCH 4/6] nova-core: docs: Document fwsec operation and layout Joel Fernandes
@ 2025-04-23 22:54 ` Joel Fernandes
2025-04-24 1:18 ` Alexandre Courbot
2025-04-23 22:54 ` [PATCH 6/6] nova-core: docs: Document devinit process Joel Fernandes
2025-04-24 1:25 ` [PATCH 0/6] Additional documentation for nova-core Alexandre Courbot
6 siblings, 1 reply; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:54 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/drivers/gpu/nova-core/firmware/fwsec.rs b/drivers/gpu/nova-core/firmware/fwsec.rs
index 664319d1d31c..79c21db9d89d 100644
--- a/drivers/gpu/nova-core/firmware/fwsec.rs
+++ b/drivers/gpu/nova-core/firmware/fwsec.rs
@@ -29,11 +29,14 @@
#[repr(C)]
#[derive(Debug)]
+
+/// The header of the Application Interface table, used
+/// to locate the DMEMMAPPER section in the DMEM (see fwsec.rst).
struct FalconAppifHdrV1 {
- ver: u8,
- hdr: u8,
- len: u8,
- cnt: u8,
+ version: u8,
+ header_size: u8,
+ entry_size: u8,
+ entry_count: u8,
}
// SAFETY: any byte sequence is valid for this struct.
unsafe impl FromBytes for FalconAppifHdrV1 {}
@@ -169,14 +172,14 @@ fn patch_command(fw: &mut DmaObject, v3_desc: &FalconUCodeDescV3, cmd: FwsecComm
let hdr_offset = (v3_desc.imem_load_size + v3_desc.interface_offset) as usize;
let hdr: &FalconAppifHdrV1 = unsafe { transmute(fw, hdr_offset) }?;
- if hdr.ver != 1 {
+ if hdr.version != 1 {
return Err(EINVAL);
}
// Find the DMEM mapper section in the firmware.
- for i in 0..hdr.cnt as usize {
+ for i in 0..hdr.entry_count as usize {
let app: &FalconAppifV1 =
- unsafe { transmute(fw, hdr_offset + hdr.hdr as usize + i * hdr.len as usize) }?;
+ unsafe { transmute(fw, hdr_offset + hdr.header_size as usize + i * hdr.entry_size as usize) }?;
if app.id != NVFW_FALCON_APPIF_ID_DMEMMAPPER {
continue;
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1
2025-04-23 22:54 ` [PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1 Joel Fernandes
@ 2025-04-24 1:18 ` Alexandre Courbot
[not found] ` <174546399398.876.3516508778193165894@patchwork.local>
0 siblings, 1 reply; 14+ messages in thread
From: Alexandre Courbot @ 2025-04-24 1:18 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel, Danilo Krummrich, David Airlie,
Simona Vetter
Cc: nouveau, dri-devel, John Hubbard, Shirish Baskaran,
Alistair Popple, Timur Tabi, Ben Skeggs
Since this just renames fields, would you be ok if I squashed this one
into the relevant patch of my series, alongside a
[joelagnelf@nvidia.com: give better names to FalconAppifHdrV1's fields]
?
On Thu Apr 24, 2025 at 7:54 AM JST, Joel Fernandes wrote:
> Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
> ---
> drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++++++++++-------
> 1 file changed, 10 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/gpu/nova-core/firmware/fwsec.rs b/drivers/gpu/nova-core/firmware/fwsec.rs
> index 664319d1d31c..79c21db9d89d 100644
> --- a/drivers/gpu/nova-core/firmware/fwsec.rs
> +++ b/drivers/gpu/nova-core/firmware/fwsec.rs
> @@ -29,11 +29,14 @@
>
> #[repr(C)]
> #[derive(Debug)]
> +
> +/// The header of the Application Interface table, used
> +/// to locate the DMEMMAPPER section in the DMEM (see fwsec.rst).
> struct FalconAppifHdrV1 {
> - ver: u8,
> - hdr: u8,
> - len: u8,
> - cnt: u8,
> + version: u8,
> + header_size: u8,
> + entry_size: u8,
> + entry_count: u8,
> }
> // SAFETY: any byte sequence is valid for this struct.
> unsafe impl FromBytes for FalconAppifHdrV1 {}
> @@ -169,14 +172,14 @@ fn patch_command(fw: &mut DmaObject, v3_desc: &FalconUCodeDescV3, cmd: FwsecComm
> let hdr_offset = (v3_desc.imem_load_size + v3_desc.interface_offset) as usize;
> let hdr: &FalconAppifHdrV1 = unsafe { transmute(fw, hdr_offset) }?;
>
> - if hdr.ver != 1 {
> + if hdr.version != 1 {
> return Err(EINVAL);
> }
>
> // Find the DMEM mapper section in the firmware.
> - for i in 0..hdr.cnt as usize {
> + for i in 0..hdr.entry_count as usize {
> let app: &FalconAppifV1 =
> - unsafe { transmute(fw, hdr_offset + hdr.hdr as usize + i * hdr.len as usize) }?;
> + unsafe { transmute(fw, hdr_offset + hdr.header_size as usize + i * hdr.entry_size as usize) }?;
>
> if app.id != NVFW_FALCON_APPIF_ID_DMEMMAPPER {
> continue;
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 6/6] nova-core: docs: Document devinit process
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
` (4 preceding siblings ...)
2025-04-23 22:54 ` [PATCH 5/6] gpu: nova-core: Clarify fields in FalconAppifHdrV1 Joel Fernandes
@ 2025-04-23 22:54 ` Joel Fernandes
2025-04-24 1:25 ` [PATCH 0/6] Additional documentation for nova-core Alexandre Courbot
6 siblings, 0 replies; 14+ messages in thread
From: Joel Fernandes @ 2025-04-23 22:54 UTC (permalink / raw)
To: linux-kernel, Danilo Krummrich, David Airlie, Simona Vetter,
Maarten Lankhorst, Maxime Ripard, Thomas Zimmermann,
Jonathan Corbet
Cc: nouveau, dri-devel, Alexandre Courbot, John Hubbard,
Shirish Baskaran, Alistair Popple, Timur Tabi, Ben Skeggs,
Joel Fernandes, linux-doc
devinit is mentioned in the code. This patch explains it so it is clear
what it does. devinit is not only essential at boot-time, but also at
runtime due to suspend-resume and things like re-clocking.
Signed-off-by: Joel Fernandes <joelagnelf@nvidia.com>
---
Documentation/gpu/nova/core/devinit.rst | 54 +++++++++++++++++++++++++
Documentation/gpu/nova/index.rst | 2 +
2 files changed, 56 insertions(+)
create mode 100644 Documentation/gpu/nova/core/devinit.rst
diff --git a/Documentation/gpu/nova/core/devinit.rst b/Documentation/gpu/nova/core/devinit.rst
new file mode 100644
index 000000000000..e504ed3efe15
--- /dev/null
+++ b/Documentation/gpu/nova/core/devinit.rst
@@ -0,0 +1,54 @@
+.. SPDX-License-Identifier: GPL-2.0
+==================================
+Device Initialization (devinit)
+==================================
+
+Overview
+--------
+Device initialization (devinit) is a crucial sequence of register read/write operations
+that occur after a GPU reset. The devinit sequence is essential for properly configuring
+the GPU hardware before it can be used.
+
+The devinit is an interpreter program that typically runs on the PMU (Power Management
+Unit) microcontroller of the GPU. This interpreter executes a "script" of initialization
+commands. The devinit interpreter itself is part of the FWSEC (Firmware Security)
+component provided by the GPU firmware in the VBIOS ROM (see fwsec.rst and vbios.rst).
+
+Key Functions of devinit
+------------------------
+devinit performs several critical tasks:
+
+1. Programming VRAM memory controller timings
+2. Power sequencing
+3. Clock and PLL (Phase-Locked Loop) configuration
+4. Thermal management
+5. VRAM memory scrubbing (ECC initialization)
+ - On some GPUs, it scrubs only part of memory and then initiates 'async scrubbing'
+
+Firmware Initialization Flow
+---------------------------
+Upon reset, several microcontrollers on the GPU (such as PMU, SEC2, GSP, etc.) run GPU
+firmware (gfw) code to set up core parameters. Most of the GPU is considered unusable
+until this initialization process completes.
+
+The GPU firmware components are:
+1. Located in the VBIOS ROM
+2. Executed in sequence on different microcontrollers:
+ - The devinit sequence runs on the PMU
+ - The FWSEC runs on the GSP (Graphics System Processor) in high secure mode
+
+Before the driver can proceed with further initialization, it must wait for a signal
+indicating that core initialization is complete (known as GFW_BOOT). This signal is
+set up by the FWSEC running on the GSP in high secure mode.
+
+Runtime Considerations
+---------------------
+It's important to note that the devinit sequence also needs to run during suspend/resume
+operations at runtime, not just during initial boot.
+
+Security and Access Control
+--------------------------
+The initialization process involves careful privilege management. For example, before
+accessing certain completion status registers, the driver must check privilege level
+masks. Some registers are only accessible after secure firmware (FWSEC) lowers the
+privilege level to allow CPU (LS/low-secure) access.
\ No newline at end of file
diff --git a/Documentation/gpu/nova/index.rst b/Documentation/gpu/nova/index.rst
index c01dcc5657e2..301435c5cf67 100644
--- a/Documentation/gpu/nova/index.rst
+++ b/Documentation/gpu/nova/index.rst
@@ -27,5 +27,7 @@ vGPU manager VFIO driver and the nova-drm driver.
:titlesonly:
core/guidelines
+ core/vbios
core/fwsec
+ core/devinit
core/todo
--
2.43.0
^ permalink raw reply related [flat|nested] 14+ messages in thread* Re: [PATCH 0/6] Additional documentation for nova-core
2025-04-23 22:53 [PATCH 0/6] Additional documentation for nova-core Joel Fernandes
` (5 preceding siblings ...)
2025-04-23 22:54 ` [PATCH 6/6] nova-core: docs: Document devinit process Joel Fernandes
@ 2025-04-24 1:25 ` Alexandre Courbot
2025-04-24 8:10 ` Danilo Krummrich
6 siblings, 1 reply; 14+ messages in thread
From: Alexandre Courbot @ 2025-04-24 1:25 UTC (permalink / raw)
To: Joel Fernandes, linux-kernel
Cc: Danilo Krummrich, David Airlie, Simona Vetter, nouveau, dri-devel,
John Hubbard, Shirish Baskaran, Alistair Popple, Timur Tabi,
Ben Skeggs
Thanks a lot for doing this, this was severely missing from the
WPR2 patchset.
Due to the strong focus on documentation, and in order to ease merging,
I think it makes sense to keep this separate from the WPR2 patchset and
merge it on top of it. Danilo, would that work for you?
On Thu Apr 24, 2025 at 7:53 AM JST, Joel Fernandes wrote:
> Hello,
> Please find in this series, several clarifications, diagrams and code comments
> for various things in the nova-core driver. These are essential to develop an
> understanding how nova-core's boot initialization works and aid in development.
>
> These patches are on top of Alex's last posting for GSP WPR2 [1]
>
> [1] https://lore.kernel.org/all/20250420-nova-frts-v1-0-ecd1cca23963@nvidia.com/
>
> Joel Fernandes (6):
> nova-core: doc: Add code comments related to devinit
> nova-core: doc: Clarify sysmembar operations
> nova-core: docs: Document vbios layout
> nova-core: docs: Document fwsec operation and layout
> gpu: nova-core: Clarify fields in FalconAppifHdrV1
> nova-core: docs: Document devinit process
>
> Documentation/gpu/nova/core/devinit.rst | 54 ++++++++
> Documentation/gpu/nova/core/fwsec.rst | 173 ++++++++++++++++++++++++
> Documentation/gpu/nova/core/vbios.rst | 155 +++++++++++++++++++++
> Documentation/gpu/nova/index.rst | 2 +
> drivers/gpu/nova-core/devinit.rs | 36 ++++-
> drivers/gpu/nova-core/firmware/fwsec.rs | 17 ++-
> drivers/gpu/nova-core/gpu.rs | 11 +-
> drivers/gpu/nova-core/regs.rs | 17 ++-
> 8 files changed, 452 insertions(+), 13 deletions(-)
> create mode 100644 Documentation/gpu/nova/core/devinit.rst
> create mode 100644 Documentation/gpu/nova/core/fwsec.rst
> create mode 100644 Documentation/gpu/nova/core/vbios.rst
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [PATCH 0/6] Additional documentation for nova-core
2025-04-24 1:25 ` [PATCH 0/6] Additional documentation for nova-core Alexandre Courbot
@ 2025-04-24 8:10 ` Danilo Krummrich
0 siblings, 0 replies; 14+ messages in thread
From: Danilo Krummrich @ 2025-04-24 8:10 UTC (permalink / raw)
To: Alexandre Courbot, Joel Fernandes
Cc: linux-kernel, David Airlie, Simona Vetter, nouveau, dri-devel,
John Hubbard, Shirish Baskaran, Alistair Popple, Timur Tabi,
Ben Skeggs
On 4/24/25 3:25 AM, Alexandre Courbot wrote:
> Thanks a lot for doing this, this was severely missing from the
> WPR2 patchset.
>
> Due to the strong focus on documentation, and in order to ease merging,
> I think it makes sense to keep this separate from the WPR2 patchset and
> merge it on top of it. Danilo, would that work for you?
Perfectly fine for me!
@Joel: Thanks a lot for this work, appreciate it!
^ permalink raw reply [flat|nested] 14+ messages in thread