From: Zhi Wang <zhiw@nvidia.com>
To: <dakr@kernel.org>, <acourbot@nvidia.com>
Cc: <alex@shazbot.org>, <jgg@nvidia.com>, <yishaih@nvidia.com>,
<skolothumtho@nvidia.com>, <kevin.tian@intel.com>,
<airlied@gmail.com>, <simona@ffwll.ch>, <ojeda@kernel.org>,
<alex.gaynor@gmail.com>, <boqun.feng@gmail.com>,
<gary@garyguo.net>, <bjorn3_gh@protonmail.com>,
<lossin@kernel.org>, <a.hindborg@kernel.org>,
<aliceryhl@google.com>, <tmgross@umich.edu>,
<jhubbard@nvidia.com>, <ecourtney@nvidia.com>, <cjia@nvidia.com>,
<smitra@nvidia.com>, <kjaju@nvidia.com>, <alkumar@nvidia.com>,
<ankita@nvidia.com>, <aniketa@nvidia.com>, <kwankhede@nvidia.com>,
<targupta@nvidia.com>, <nova-gpu@lists.linux.dev>,
<linux-kernel@vger.kernel.org>, <zhiwang@kernel.org>,
Zhi Wang <zhiw@nvidia.com>, <kvm@vger.kernel.org>
Subject: [PATCH 00/13] Introduce NVIDIA vGPU manager and VFIO variant driver
Date: Sat, 5 Sep 2026 11:11:03 +0300 [thread overview]
Message-ID: <20260905081116.106613-1-zhiw@nvidia.com> (raw)
Add the nova-core support needed to create and manage NVIDIA vGPU
instances, together with the NVIDIA vGPU VFIO variant driver that consumes
the lifecycle interface.
The control path introduced by this series is:
+-----------------------+ +-----------------------+
| Linux guest | | Windows guest |
| NVIDIA guest driver | | NVIDIA guest driver |
+-----------+-----------+ +-----------+-----------+
| |
+-----------v-----------+ +-----------v-----------+
| QEMU (VFIO user) | | QEMU (VFIO user) |
+-----------+-----------+ +-----------+-----------+
+----------------+----------------+
|
+-------v-------+
| VFIO core |
+-------+-------+
|
+---------------------v---------------------+
| NVIDIA vGPU VFIO variant driver |
+---------------------+---------------------+
|
+---------------+----------------------+
| binds PCI VF | PF lifecycle API
| | open / close / reset
v v
+--------+--------+ +-------------+-------------+
| PCI VF(s) | | nova-core (PF driver) |
| (virtual GPU) | | vGPU Manager |
+--------+--------+ | VRAM / channels ... |
| +-------------+-------------+
| |
+-------------v--------------------------------------v----------------+
| NVIDIA physical GPU |
| VF resources PF / GSP / VRAM |
+---------------------------------------------------------------------+
On the nova-core side, the PF driver owns the GPU firmware-facing part
of vGPU management. It already detects vGPU mode before GSP boot and
supplies the SR-IOV topology to the GPU System Processor firmware
(GSP-RM). This series extends it to:
- Retain the firmware-reported VMMU segment size and FIFO engine
ordering after GSP_INIT, together with nova-core's channel capacity.
- Add VramBlock and VramRegion ownership plus Bar1Map for bounded CPU
mappings of VRAM-backed control structures.
- Query GSP-RM through its command queue for the vGPU type already
assigned to a VF and its properties.
- Build a layout-validated, profile-wide pool of paired framebuffer and
management-heap slots, reserve contiguous channel ranges, and keep
live allocations in a per-PF instance registry.
- Use the typed r000 firmware bindings for control and response layouts,
message IDs, and firmware-defined region sizes.
- Handle GMC transactions over the GSP queues, match GMC replies by
command and sequence, dispatch interleaved RM RPCs from the shared
GSP-to-CPU message queue, and consume stale GMC messages.
- Implement the GMCAPI bootload, shutdown, and cleanup operations with
typed channel and resource maps for each plugin instance.
- Establish the BAR1-backed PluginRpc channel, negotiate its protocol,
and send the VM configuration and plugin BME-state update.
- Use per-instance CeUtils channels to scrub guest framebuffer memory
on allocation, reset, and shutdown. If ownership or completion is
uncertain, keep the affected VRAM and channels out of the allocators.
- Expose the three per-instance plugin log buffers through debugfs with
the header needed by nvlog_decoder.
- Keep resource locking, teardown, and rollback inside nova-core.
- Select the firmware-defined 48-VM WPR2 heap when a device advertises
more than 32 VFs.
On the VFIO side, the NVIDIA vGPU VFIO variant driver:
- Binds only to an NVIDIA VF explicitly selected through
driver_override.
- Derives the guest function ID (GFID) from the SR-IOV VF index and
registers a vfio-pci-core device.
- Presents the vGPU PCI device and subsystem IDs returned by nova-core,
and limits the reported framebuffer BAR size.
- Delegates the remaining PCI and VFIO operations to vfio-pci-core and
uses the standard VFIO physical-device helpers for IOMMUFD.
The ownership boundary between the two drivers is visible at these
lifecycle points:
- Open: after vfio_pci_core_enable(), the variant driver passes the PF
returned by pci_physfn(vf), the GFID, the VF's PCI
domain/bus/device/function (DBDF), and the calling process TGID to
nvidia_vgpu_open(). nova-core validates the PF and GFID, creates and
activates the instance, and returns the guest-visible PCI IDs and
BAR1 size. On failure, the variant driver disables the vfio-pci-core
device; on success, it calls
vfio_pci_core_finish_enable().
- Reset: the variant driver first calls nvidia_vgpu_reset(). nova-core
sends the plugin reset RPC and scrubs the guest framebuffer. Only
after that succeeds is VFIO_DEVICE_RESET passed to vfio-pci-core for
the PCI function reset.
- Close: the variant driver invokes the void nvidia_vgpu_close()
interface to request plugin shutdown, framebuffer scrubbing, and
resource release from nova-core, then closes the vfio-pci-core
device.
The cross-module interface is therefore limited to three GPL-only
symbols in NOVA_CORE_VGPU: nvidia_vgpu_open(), nvidia_vgpu_close(), and
nvidia_vgpu_reset().
This is a ground-up rework of the earlier vGPU RFC [1]. In particular,
the vGPU manager has been moved from the VFIO driver into the nova-core
implementation, and the VFIO side no longer carries copies of RM firmware
headers. The VFIO driver attaches to the VF and uses its PF's nova-core
state for management, as discussed in that thread.
The patches are organized as follows:
1-4 Initialize VgpuManager and add the memory and firmware building
blocks.
5-8 Allocate instances and implement GMC, plugin boot, and RPC.
9-10 Scrub guest framebuffer memory and expose plugin diagnostics.
11-12 Export the lifecycle API and add its VFIO consumer.
13 Select the larger WPR2 heap for 48-VF devices.
The series is based on the nova-core r000 GSP, memory-management,
SR-IOV, Rust bitmap/id-pool, and PCI abstraction work. The exact base
commit is recorded below, and the complete prerequisite stack is
available in [2].
[1] https://lore.kernel.org/kvm/20250903221111.3866249-1-zhiw@nvidia.com/
[2] https://github.com/zhiwang-nvidia/nova-core/tree/zhi/nova-vgpu-wip-nova-gsp-20260902
Alok Kumar (1):
gpu: nova-core: vgpu: add VRAM slot allocator
Zhi Wang (12):
gpu: nova-core: vgpu: add post-GSP-boot vGPU initialization
gpu: nova-core: mm: add VramBlock and Bar1Map
gpu: nova-core: vgpu: add r000 plugin bindings
gpu: nova-core: vgpu: add instance create/destroy
gpu: nova-core: gsp: add GMC transaction helpers
gpu: nova-core: vgpu: add vGPU bootload
gpu: nova-core: vgpu: implement PluginRpc channel and config params
gpu: nova-core: vgpu: scrub guest framebuffer memory with CeUtils
gpu: nova-core: vgpu: export plugin log buffers via debugfs
gpu: nova-core: vgpu: export lifecycle operations to VFIO
vfio/nvidia-vgpu: add the NVIDIA vGPU VFIO variant driver
gpu: nova-core: reserve the 48-VM WPR2 heap
drivers/gpu/nova-core/driver.rs | 42 +-
drivers/gpu/nova-core/fb.rs | 2 +-
drivers/gpu/nova-core/gpu.rs | 184 +++--
drivers/gpu/nova-core/gpu/channel.rs | 3 +
drivers/gpu/nova-core/gsp.rs | 29 +-
drivers/gpu/nova-core/gsp/boot.rs | 13 +-
drivers/gpu/nova-core/gsp/cmdq.rs | 356 +++++++--
drivers/gpu/nova-core/gsp/commands.rs | 78 +-
drivers/gpu/nova-core/gsp/fw.rs | 63 +-
drivers/gpu/nova-core/gsp/fw/commands.rs | 185 ++++-
.../gpu/nova-core/gsp/fw/r000_00/bindings.rs | 175 +++++
drivers/gpu/nova-core/gsp/hal/gh100.rs | 2 +-
drivers/gpu/nova-core/gsp/hal/tu102.rs | 2 +-
drivers/gpu/nova-core/mm.rs | 6 +-
drivers/gpu/nova-core/mm/bar_user.rs | 183 ++++-
drivers/gpu/nova-core/mm/vram.rs | 187 +++++
drivers/gpu/nova-core/nova_core_exports.c | 5 +
drivers/gpu/nova-core/vgpu.rs | 91 ---
drivers/gpu/nova-core/vgpu/bootload.rs | 162 ++++
drivers/gpu/nova-core/vgpu/consts.rs | 33 +
drivers/gpu/nova-core/vgpu/fw.rs | 490 ++++++++++++
drivers/gpu/nova-core/vgpu/fw/commands.rs | 28 +
drivers/gpu/nova-core/vgpu/instance.rs | 705 ++++++++++++++++++
drivers/gpu/nova-core/vgpu/log.rs | 167 +++++
drivers/gpu/nova-core/vgpu/mod.rs | 170 +++++
drivers/gpu/nova-core/vgpu/plugin_rpc.rs | 274 +++++++
drivers/gpu/nova-core/vgpu/scrubber.rs | 474 ++++++++++++
drivers/gpu/nova-core/vgpu/vfio.rs | 282 +++++++
drivers/gpu/nova-core/vgpu/vram.rs | 140 ++++
drivers/vfio/pci/Kconfig | 2 +
drivers/vfio/pci/Makefile | 2 +
drivers/vfio/pci/nvidia-vgpu/Kconfig | 16 +
drivers/vfio/pci/nvidia-vgpu/Makefile | 2 +
drivers/vfio/pci/nvidia-vgpu/main.c | 253 +++++++
include/drm/nvidia_vgpu.h | 28 +
rust/bindings/bindings_helper.h | 1 +
36 files changed, 4536 insertions(+), 299 deletions(-)
create mode 100644 drivers/gpu/nova-core/mm/vram.rs
delete mode 100644 drivers/gpu/nova-core/vgpu.rs
create mode 100644 drivers/gpu/nova-core/vgpu/bootload.rs
create mode 100644 drivers/gpu/nova-core/vgpu/consts.rs
create mode 100644 drivers/gpu/nova-core/vgpu/fw.rs
create mode 100644 drivers/gpu/nova-core/vgpu/fw/commands.rs
create mode 100644 drivers/gpu/nova-core/vgpu/instance.rs
create mode 100644 drivers/gpu/nova-core/vgpu/log.rs
create mode 100644 drivers/gpu/nova-core/vgpu/mod.rs
create mode 100644 drivers/gpu/nova-core/vgpu/plugin_rpc.rs
create mode 100644 drivers/gpu/nova-core/vgpu/scrubber.rs
create mode 100644 drivers/gpu/nova-core/vgpu/vfio.rs
create mode 100644 drivers/gpu/nova-core/vgpu/vram.rs
create mode 100644 drivers/vfio/pci/nvidia-vgpu/Kconfig
create mode 100644 drivers/vfio/pci/nvidia-vgpu/Makefile
create mode 100644 drivers/vfio/pci/nvidia-vgpu/main.c
create mode 100644 include/drm/nvidia_vgpu.h
base-commit: a1dc9fdea9ab15e5df1bd6af1f6de2e40079f02d
--
2.53.0
next reply other threads:[~2026-09-05 8:11 UTC|newest]
Thread overview: 15+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-09-05 8:11 Zhi Wang [this message]
2026-09-05 8:11 ` [PATCH 01/13] gpu: nova-core: vgpu: add post-GSP-boot vGPU initialization Zhi Wang
2026-09-05 8:11 ` [PATCH 02/13] gpu: nova-core: mm: add VramBlock and Bar1Map Zhi Wang
2026-09-05 8:11 ` [PATCH 03/13] gpu: nova-core: vgpu: add VRAM slot allocator Zhi Wang
2026-09-05 8:11 ` [PATCH 04/13] gpu: nova-core: vgpu: add r000 plugin bindings Zhi Wang
2026-09-05 8:11 ` [PATCH 05/13] gpu: nova-core: vgpu: add instance create/destroy Zhi Wang
2026-09-05 8:11 ` [PATCH 06/13] gpu: nova-core: gsp: add GMC transaction helpers Zhi Wang
2026-09-05 8:11 ` [PATCH 07/13] gpu: nova-core: vgpu: add vGPU bootload Zhi Wang
2026-09-05 8:11 ` [PATCH 08/13] gpu: nova-core: vgpu: implement PluginRpc channel and config params Zhi Wang
2026-09-05 8:11 ` [PATCH 09/13] gpu: nova-core: vgpu: scrub guest framebuffer memory with CeUtils Zhi Wang
2026-09-05 8:11 ` [PATCH 10/13] gpu: nova-core: vgpu: export plugin log buffers via debugfs Zhi Wang
2026-09-05 8:11 ` [PATCH 11/13] gpu: nova-core: vgpu: export lifecycle operations to VFIO Zhi Wang
2026-09-05 8:11 ` [PATCH 12/13] vfio/nvidia-vgpu: add the NVIDIA vGPU VFIO variant driver Zhi Wang
2026-09-09 3:00 ` Alex Williamson
2026-09-05 8:11 ` [PATCH 13/13] gpu: nova-core: reserve the 48-VM WPR2 heap Zhi Wang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20260905081116.106613-1-zhiw@nvidia.com \
--to=zhiw@nvidia.com \
--cc=a.hindborg@kernel.org \
--cc=acourbot@nvidia.com \
--cc=airlied@gmail.com \
--cc=alex.gaynor@gmail.com \
--cc=alex@shazbot.org \
--cc=aliceryhl@google.com \
--cc=alkumar@nvidia.com \
--cc=aniketa@nvidia.com \
--cc=ankita@nvidia.com \
--cc=bjorn3_gh@protonmail.com \
--cc=boqun.feng@gmail.com \
--cc=cjia@nvidia.com \
--cc=dakr@kernel.org \
--cc=ecourtney@nvidia.com \
--cc=gary@garyguo.net \
--cc=jgg@nvidia.com \
--cc=jhubbard@nvidia.com \
--cc=kevin.tian@intel.com \
--cc=kjaju@nvidia.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=lossin@kernel.org \
--cc=nova-gpu@lists.linux.dev \
--cc=ojeda@kernel.org \
--cc=simona@ffwll.ch \
--cc=skolothumtho@nvidia.com \
--cc=smitra@nvidia.com \
--cc=targupta@nvidia.com \
--cc=tmgross@umich.edu \
--cc=yishaih@nvidia.com \
--cc=zhiwang@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox