[PATCH 0/2] nova-core: add initial support for gpu timer (v2)

[PATCH 0/2] nova-core: add initial support for gpu timer (v2)

[Dave Airlie via B4 Relay]

The GPU has a timer, one of the pieces of information
we need to pass into GSP init later is the timer offset.
    
This series just adds support for setting up the timer
on Turing+, with hals for Turing and Hopper.
    
Right now it calculates the value to send to GSP but
the code to send it will be added later.
    
For some hardware variants sending a proper value
is required for GSP init to succeed.

v2 changes on second patch.

Signed-off-by: Dave Airlie <airlied@redhat.com>
---
Dave Airlie (2):
      rust: time: expose as_nanos for clock sources.
      gpu: nova-core: add support for GPU timer block. (v2)

 drivers/gpu/nova-core/gpu.rs             |  5 +++
 drivers/gpu/nova-core/nova_core.rs       |  1 +
 drivers/gpu/nova-core/timer.rs           | 33 ++++++++++++++
 drivers/gpu/nova-core/timer/hal.rs       | 28 ++++++++++++
 drivers/gpu/nova-core/timer/hal/gh100.rs | 76 ++++++++++++++++++++++++++++++++
 drivers/gpu/nova-core/timer/hal/tu102.rs | 56 +++++++++++++++++++++++
 drivers/gpu/nova-core/timer/regs.rs      | 64 +++++++++++++++++++++++++++
 rust/kernel/time.rs                      |  3 +-
 8 files changed, 265 insertions(+), 1 deletion(-)
---
base-commit: 3497a403264b094ebd61b8b755165d56e7d9dc37
change-id: 20261102-nova-timer-f4b9de3274a1

Best regards,
-- 
Dave Airlie <airlied@redhat.com>

[PATCH 1/2] rust: time: expose as_nanos for clock sources.

[Dave Airlie via B4 Relay]

From: Dave Airlie <airlied@redhat.com>

The nouveau C driver does ktime_to_ns(ktime_get()) to program
the cpu ns into some hw/fw, so allow this method for rust drivers.

Reviewed-by: FUJITA Tomonori <fujita.tomonori@gmail.com>
Signed-off-by: Dave Airlie <airlied@redhat.com>
---
 rust/kernel/time.rs | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/rust/kernel/time.rs b/rust/kernel/time.rs
index 6c0a5e8090d0..a91ddf04785b 100644
--- a/rust/kernel/time.rs
+++ b/rust/kernel/time.rs
@@ -211,8 +211,9 @@ pub fn elapsed(&self) -> Delta {
         Self::now() - *self
     }
 
+    /// Return the raw nanoseconds values since the epoch of clock `C`.
     #[inline]
-    pub(crate) fn as_nanos(&self) -> i64 {
+    pub fn as_nanos(&self) -> i64 {
         self.inner
     }
 

-- 
2.56.0

[PATCH 2/2] gpu: nova-core: add support for GPU timer block. (v2)

[Dave Airlie via B4 Relay]

From: Dave Airlie <airlied@redhat.com>

This is needed to calculate the system timer offset that is passed
to GSP init.

2 HALs exist:

a) tu102 is the gv100 path from open-gpu, the priv level is checked
and the cpu time is written to the gpu timer. no offset is used.

b) gh100 is the gh100 path from open-gpu, where the secure system
timer is read, the offset is calculated and on everything where it
works the offset gets written to secure registers, and also
sent to GSP.

Signed-off-by: Dave Airlie <airlied@redhat.com>
---
v2: kill the read API for now, it would be internal to the timer
    as we just want to read once at startup on gb10b.
    Drop the cpu timer wrapper.
    don't store chipset and remove the pin stuff
    Move registers into timer.
    add support for GH100 WAR properly
    do i64 calcs to avoid overflow and copy RM code closer.
---
 drivers/gpu/nova-core/gpu.rs             |  5 +++
 drivers/gpu/nova-core/nova_core.rs       |  1 +
 drivers/gpu/nova-core/timer.rs           | 33 ++++++++++++++
 drivers/gpu/nova-core/timer/hal.rs       | 28 ++++++++++++
 drivers/gpu/nova-core/timer/hal/gh100.rs | 76 ++++++++++++++++++++++++++++++++
 drivers/gpu/nova-core/timer/hal/tu102.rs | 56 +++++++++++++++++++++++
 drivers/gpu/nova-core/timer/regs.rs      | 64 +++++++++++++++++++++++++++
 7 files changed, 263 insertions(+)

diff --git a/drivers/gpu/nova-core/gpu.rs b/drivers/gpu/nova-core/gpu.rs
index 4e59072cab50..b111a5b802e8 100644
--- a/drivers/gpu/nova-core/gpu.rs
+++ b/drivers/gpu/nova-core/gpu.rs
@@ -53,6 +53,7 @@
         GpuMm,
         VramAddress, //
     },
+    timer::Timer,
     vgpu::VgpuManager, //
 };
 
@@ -331,6 +332,8 @@ struct GspResources<'gpu> {
     // TODO: use different resource types for each boot method, and make the relevant Gsp methods
     // generic against them.
     fsp: Option<Fsp<'gpu>>,
+    /// Timer instance - contains system timer offset
+    timer: Timer,
     /// vGPU state detected before GSP boot.
     vgpu: VgpuManager,
     /// GSP runtime data.
@@ -497,6 +500,8 @@ pub(crate) fn new<'a>(
 
                 fsp: Fsp::try_new(dev, bar, spec.chipset)?,
 
+                timer: Timer::new(bar, spec.chipset)?,
+
                 vgpu: VgpuManager::new(pdev, spec.chipset, fsp.as_mut()),
 
                 gsp <- Gsp::new(pdev, bar),
diff --git a/drivers/gpu/nova-core/nova_core.rs b/drivers/gpu/nova-core/nova_core.rs
index a8f6dc22057b..ee59d81e5105 100644
--- a/drivers/gpu/nova-core/nova_core.rs
+++ b/drivers/gpu/nova-core/nova_core.rs
@@ -27,6 +27,7 @@
 mod sbuffer;
 #[cfg(CONFIG_NOVA_CORE_SELFTESTS)]
 mod selftest;
+mod timer;
 mod vbios;
 mod vgpu;
 
diff --git a/drivers/gpu/nova-core/timer.rs b/drivers/gpu/nova-core/timer.rs
new file mode 100644
index 000000000000..a4206076a2f0
--- /dev/null
+++ b/drivers/gpu/nova-core/timer.rs
@@ -0,0 +1,33 @@
+// SPDX-License-Identifier: GPL-2.0
+
+mod hal;
+mod regs;
+
+use kernel::{
+    prelude::*, //
+};
+
+use crate::{
+    driver::Bar0,
+    gpu::Chipset, //
+};
+
+// struct that just holds the sys timer offset
+// after initialising the GPU timer block.
+pub(crate) struct Timer {
+    sys_timer_offset_ns: i64,
+}
+
+impl Timer {
+    pub(crate) fn new(bar: Bar0<'_>, chipset: Chipset) -> Result<Self> {
+        Ok(Self {
+            sys_timer_offset_ns: hal::timer_hal(chipset).init(bar)?,
+        })
+    }
+
+    // this value will be passed to GspInitRequest SYS_TIMER_OFFSET_NS
+    #[expect(dead_code)]
+    pub(crate) fn sys_timer_offset_ns(&self) -> i64 {
+        self.sys_timer_offset_ns
+    }
+}
diff --git a/drivers/gpu/nova-core/timer/hal.rs b/drivers/gpu/nova-core/timer/hal.rs
new file mode 100644
index 000000000000..7aeff6cdc678
--- /dev/null
+++ b/drivers/gpu/nova-core/timer/hal.rs
@@ -0,0 +1,28 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::prelude::*;
+
+use crate::{
+    driver::Bar0,
+    gpu::{
+        Architecture,
+        Chipset, //
+    },
+};
+
+mod gh100;
+mod tu102;
+
+pub(super) trait TimerHal {
+    /// return system timer offset ns
+    fn init(&self, bar: Bar0<'_>) -> Result<i64>;
+}
+
+pub(super) fn timer_hal(chipset: Chipset) -> &'static dyn TimerHal {
+    match chipset.arch() {
+        Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
+        Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
+            gh100::GH100_HAL
+        }
+    }
+}
diff --git a/drivers/gpu/nova-core/timer/hal/gh100.rs b/drivers/gpu/nova-core/timer/hal/gh100.rs
new file mode 100644
index 000000000000..7f55bf04a3f3
--- /dev/null
+++ b/drivers/gpu/nova-core/timer/hal/gh100.rs
@@ -0,0 +1,76 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use super::TimerHal;
+
+use kernel::{
+    io::Io,
+    prelude::*,
+    time::*, //
+};
+
+use crate::{
+    driver::Bar0,
+    timer::regs, //
+};
+
+struct Gh100;
+
+/// FSP ucode version from which on the fix for bug 3957833 is included.
+const FSP_BUG_3957833_FIX_VERSION: u32 = 0x44c;
+
+/// kfspRequiresBug3957833WAR_HAL: returns `true` if the FSP ucode predates the fix for bug
+/// 3957833 and hence the sys timer offset must not be programmed.
+fn fsp_requires_bug3957833_war(bar: Bar0<'_>) -> bool {
+    bar.read(regs::gh100::NV_GFW_FSP_UCODE_VERSION).full().get() < FSP_BUG_3957833_FIX_VERSION
+}
+
+fn timer_time(bar: Bar0<'_>, time_ns: i64) -> Result<i64> {
+    let mut sec_hi2 = u32::from(bar.read(regs::gh100::NV_PGC6_SCI_SEC_TIMER_TIME_1));
+    let sec_timer_ns: u64;
+    loop {
+        let sec_lo = u32::from(bar.read(regs::gh100::NV_PGC6_SCI_SEC_TIMER_TIME_0));
+        let sec_hi = u32::from(bar.read(regs::gh100::NV_PGC6_SCI_SEC_TIMER_TIME_1));
+        if sec_hi == sec_hi2 {
+            sec_timer_ns = u64::from(sec_hi) << 32 | u64::from(sec_lo);
+            break;
+        }
+        sec_hi2 = u32::from(bar.read(regs::gh100::NV_PGC6_SCI_SEC_TIMER_TIME_1));
+    }
+
+    // The system and the GPU timer are both free-running with unrelated epochs, so the
+    // difference can be negative. Both operands are < 2^62, hence this cannot overflow.
+    let sys_timer_offset_ns = time_ns - sec_timer_ns as i64;
+
+    // The offset is passed to the GSP as an unsigned value and, like in the RM's
+    // tmrSetCurrentTime_GH100, is required to be non-negative. With wall clock time this can
+    // only fail if the system clock is set behind the GPU power-on time.
+    if sys_timer_offset_ns < 0 {
+        return Err(EINVAL);
+    }
+
+    if !fsp_requires_bug3957833_war(bar) {
+        let offset_ns = sys_timer_offset_ns as u64;
+        bar.write_reg(
+            regs::gh100::NV_PGC6_SCI_SYS_TIMER_OFFSET_1::zeroed()
+                .try_with_nsec(((offset_ns >> 32) & 0x1fff_ffff) as u32)?,
+        );
+        bar.write_reg(
+            regs::gh100::NV_PGC6_SCI_SYS_TIMER_OFFSET_0::zeroed()
+                .try_with_nsec(((offset_ns >> 5) & 0x7ff_ffff) as u32)?
+                .with_update(true),
+        );
+    }
+    Ok(sys_timer_offset_ns)
+}
+
+impl TimerHal for Gh100 {
+    fn init(&self, bar: Bar0<'_>) -> Result<i64> {
+        // The open drivers uses wall clock time here not monotonic
+        let sys_time_ns = Instant::<RealTime>::now().as_nanos();
+
+        timer_time(bar, sys_time_ns)
+    }
+}
+
+const GH100: Gh100 = Gh100;
+pub(super) const GH100_HAL: &dyn TimerHal = &GH100;
diff --git a/drivers/gpu/nova-core/timer/hal/tu102.rs b/drivers/gpu/nova-core/timer/hal/tu102.rs
new file mode 100644
index 000000000000..80f26a152065
--- /dev/null
+++ b/drivers/gpu/nova-core/timer/hal/tu102.rs
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use super::TimerHal;
+
+use kernel::{
+    io::Io,
+    prelude::*,
+    time::*, //
+};
+
+use crate::{
+    driver::Bar0,
+    timer::regs, //
+};
+struct Tu102;
+
+// this code will be used in the gb10b hal eventually
+#[expect(dead_code)]
+fn timer_read(bar: Bar0<'_>) -> u64 {
+    let mut hi2: u32 = u32::from(bar.read(regs::NV_PTIMER_TIME_1));
+    loop {
+        let lo = u32::from(bar.read(regs::NV_PTIMER_TIME_0));
+        let hi = u32::from(bar.read(regs::NV_PTIMER_TIME_1));
+        if hi == hi2 {
+            return u64::from(hi) << 32 | u64::from(lo);
+        }
+        hi2 = u32::from(bar.read(regs::NV_PTIMER_TIME_1));
+    }
+}
+
+fn timer_time(bar: Bar0<'_>, time_ns: u64) -> Result {
+    if bar.read(regs::NV_PTIMER_TIME_PRIV_LEVEL_MASK).level0() {
+        bar.write_reg(
+            regs::NV_PTIMER_TIME_1::zeroed().try_with_nsec(((time_ns >> 32) & 0x1fff_ffff) as u32)?);
+        bar.write_reg(
+            regs::NV_PTIMER_TIME_0::zeroed().try_with_nsec(((time_ns >> 5) & 0x7ff_ffff) as u32)?,
+        );
+        Ok(())
+    } else {
+        Err(EINVAL)
+    }
+}
+
+impl TimerHal for Tu102 {
+    fn init(&self, bar: Bar0<'_>) -> Result<i64> {
+        // The RM uses wall clock time here (osGetSystemTime -> ktime_get_real), so the GPU time
+        // matches the host's real time, e.g. for timestamp correlation in userspace.
+        let sys_time_ns = Instant::<RealTime>::now().as_nanos();
+
+        timer_time(bar, sys_time_ns as u64)?;
+        Ok(0_i64)
+    }
+}
+
+const TU102: Tu102 = Tu102;
+pub(super) const TU102_HAL: &dyn TimerHal = &TU102;
diff --git a/drivers/gpu/nova-core/timer/regs.rs b/drivers/gpu/nova-core/timer/regs.rs
new file mode 100644
index 000000000000..04549c761fb0
--- /dev/null
+++ b/drivers/gpu/nova-core/timer/regs.rs
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0
+
+use kernel::io::register;
+
+use crate::{
+    driver::NovaRegisters,
+};
+
+// PTIMER
+
+register! {
+    base: NovaRegisters;
+
+    pub(crate) NV_PTIMER_TIME_PRIV_LEVEL_MASK(u32) @ 0x00009430 {
+        4:4 level0 => bool;
+    }
+
+    pub(crate) NV_PTIMER_TIME_0(u32) @ 0x00009400 {
+        31:5 nsec;
+    }
+
+    pub(crate) NV_PTIMER_TIME_1(u32) @ 0x00009410 {
+        28:0 nsec;
+    }
+}
+
+pub(crate) mod gh100 {
+    use kernel::io::register;
+
+    use crate::driver::NovaRegisters;
+
+    // PGC6/TIMER
+    register! {
+        base: NovaRegisters;
+
+        pub(crate) NV_PGC6_SCI_SEC_TIMER_TIME_0(u32) @ 0x00118f54 {
+            31:5 nsec;
+        }
+
+        pub(crate) NV_PGC6_SCI_SEC_TIMER_TIME_1(u32) @ 0x00118f58 {
+            28:0 nsec;
+        }
+
+        pub(crate) NV_PGC6_SCI_SYS_TIMER_OFFSET_0(u32) @ 0x00118df4 {
+            0:0 update;
+            31:5 nsec;
+        }
+
+        pub(crate) NV_PGC6_SCI_SYS_TIMER_OFFSET_1(u32) @ 0x00118df8 {
+            28:0 nsec;
+        }
+    }
+
+    // FSP PRI space; the timer is its only consumer so far.
+    register! {
+        base: NovaRegisters;
+
+        /// FSP ucode version, encoded as `major.minor` in bits 11:8 and 7:0, written by the FSP
+        /// during boot. (`NV_PFSP_FALCON_COMMON_SCRATCH_GROUP_3(1)` in the hardware headers.)
+        pub(crate) NV_GFW_FSP_UCODE_VERSION(u32) @ 0x008f0334 {
+            11:0 full;
+        }
+    }
+}

-- 
2.56.0

Re: [PATCH 2/2] gpu: nova-core: add support for GPU timer block. (v2)

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] gpu: nova-core: GH100 FSP bug 3957833 WAR applied to Blackwell in timer_time()
--

commit fd289ae7ae729480f9ef4d6e43c04d6aa8f3dbe4
Author: Dave Airlie <airlied@redhat.com>

gpu: nova-core: add support for GPU timer block. (v2)

This commit adds GPU timer block initialization for nova-core to compute and
program the system timer offset passed to GSP initialization, with HAL
implementations for TU102 (Turing/Ampere/Ada) and GH100 (Hopper/Blackwell).

> diff --git a/drivers/gpu/nova-core/timer/hal.rs b/drivers/gpu/nova-core/timer/hal.rs
> new file mode 100644
> index 0000000000000..7aeff6cdc6787
> --- /dev/null
> +++ b/drivers/gpu/nova-core/timer/hal.rs
> @@ -0,0 +1,28 @@

[ ... ]

> +pub(super) fn timer_hal(chipset: Chipset) -> &'static dyn TimerHal {
> +    match chipset.arch() {
> +        Architecture::Turing | Architecture::Ampere | Architecture::Ada => tu102::TU102_HAL,
> +        Architecture::Hopper | Architecture::BlackwellGB10x | Architecture::BlackwellGB20x => {
> +            gh100::GH100_HAL
> +        }
> +    }
> +}
> diff --git a/drivers/gpu/nova-core/timer/hal/gh100.rs b/drivers/gpu/nova-core/timer/hal/gh100.rs
> new file mode 100644
> index 0000000000000..7f55bf04a3f35
> --- /dev/null
> +++ b/drivers/gpu/nova-core/timer/hal/gh100.rs
> @@ -0,0 +1,76 @@

[ ... ]

> +/// FSP ucode version from which on the fix for bug 3957833 is included.
> +const FSP_BUG_3957833_FIX_VERSION: u32 = 0x44c;
> +
> +/// kfspRequiresBug3957833WAR_HAL: returns `true` if the FSP ucode predates the fix for bug
> +/// 3957833 and hence the sys timer offset must not be programmed.
> +fn fsp_requires_bug3957833_war(bar: Bar0<'_>) -> bool {
> +    bar.read(regs::gh100::NV_GFW_FSP_UCODE_VERSION).full().get() < FSP_BUG_3957833_FIX_VERSION
> +}
> +
> +fn timer_time(bar: Bar0<'_>, time_ns: i64) -> Result<i64> {

[ ... ]

> +    if !fsp_requires_bug3957833_war(bar) {
> +        let offset_ns = sys_timer_offset_ns as u64;
> +        bar.write_reg(
> +            regs::gh100::NV_PGC6_SCI_SYS_TIMER_OFFSET_1::zeroed()
> +                .try_with_nsec(((offset_ns >> 32) & 0x1fff_ffff) as u32)?,
> +        );
> +        bar.write_reg(
> +            regs::gh100::NV_PGC6_SCI_SYS_TIMER_OFFSET_0::zeroed()
> +                .try_with_nsec(((offset_ns >> 5) & 0x7ff_ffff) as u32)?
> +                .with_update(true),
> +        );
> +    }
> +    Ok(sys_timer_offset_ns)
> +}

[Severity: High]
Should fsp_requires_bug3957833_war() in drivers/gpu/nova-core/timer/hal/gh100.rs
be restricted to Hopper (GH100) rather than also running on Blackwell?

In drivers/gpu/nova-core/timer/hal.rs, timer_hal() routes Architecture::Hopper,
Architecture::BlackwellGB10x, and Architecture::BlackwellGB20x to
gh100::GH100_HAL:

    Gpu::new()
      -> Timer::new()
        -> hal::timer_hal(chipset).init(bar)
          -> Gh100::init()
            -> timer_time()
              -> fsp_requires_bug3957833_war()

In OpenRM, kfspRequiresBug3957833WAR_HAL is an FSP HAL method implemented only
for GH100 (comparing GH100's FSP ucode version against 0x44c) and stubbed to
return false on Blackwell, which has separate FSP firmware version numbering.

When running on Blackwell (BlackwellGB10x or BlackwellGB20x), if the value in
NV_GFW_FSP_UCODE_VERSION is below 0x44c, won't timer_time() falsely trigger
the GH100 workaround and skip programming NV_PGC6_SCI_SYS_TIMER_OFFSET_1 and
NV_PGC6_SCI_SYS_TIMER_OFFSET_0?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261102-nova-timer-v1-0-b347ead64469@redhat.com?part=2