From: Alok Kumar <alkumar@nvidia.com>
To: <dakr@kernel.org>, <acourbot@nvidia.com>, <kvm@vger.kernel.org>,
<rust-for-linux@vger.kernel.org>, <nova-gpu@lists.linux.dev>,
<linux-pci@vger.kernel.org>
Cc: <alex@shazbot.org>, <jgg@nvidia.com>, <yishaih@nvidia.com>,
<skolothumtho@nvidia.com>, <kevin.tian@intel.com>,
<airlied@gmail.com>, <simona@ffwll.ch>, <ojeda@kernel.org>,
<alex.gaynor@gmail.com>, <boqun.feng@gmail.com>,
<gary@garyguo.net>, <bjorn3_gh@protonmail.com>,
<lossin@kernel.org>, <a.hindborg@kernel.org>,
<aliceryhl@google.com>, <tmgross@umich.edu>,
<markus.probst@posteo.de>, <jhubbard@nvidia.com>,
<apopple@nvidia.com>, <joelagnelf@nvidia.com>,
<ecourtney@nvidia.com>, <cjia@nvidia.com>, <smitra@nvidia.com>,
<kjaju@nvidia.com>, <alkumar@nvidia.com>, <ankita@nvidia.com>,
<aniketa@nvidia.com>, <kwankhede@nvidia.com>,
<targupta@nvidia.com>, <linux-kernel@vger.kernel.org>,
<zhiwang@kernel.org>, <zhiw@nvidia.com>, <boqun@kernel.org>,
<daniel.almeida@collabora.com>, <tamird@kernel.org>,
<work@onurozkan.dev>, <jgg@ziepe.ca>, <bhelgaas@google.com>,
<kwilczynski@kernel.org>, <dri-devel@lists.freedesktop.org>,
<peter@colberg.org>, <xinmengs@nvidia.com>
Subject: [PATCH v4 00/32] Introduce NVIDIA vGPU manager
Date: Fri, 6 Nov 2026 11:09:33 +0530 [thread overview]
Message-ID: <cover.1793906667.git.alkumar@nvidia.com> (raw)
Add the nova-core support needed to create and manage NVIDIA vGPU
instances on the current whole-GPU SR-IOV path.
The overall control path is shown below. This posting contains the
nova-core manager and PF-side lifecycle APIs. Typed PF-service
publication and the VFIO variant driver follow separately.
+-----------------------+ +-----------------------+
| Linux guest | | Windows guest |
| NVIDIA guest driver | | NVIDIA guest driver |
+-----------+-----------+ +-----------+-----------+
| |
+-----------v-----------+ +-----------v-----------+
| QEMU (VFIO user) | | QEMU (VFIO user) |
+-----------+-----------+ +-----------+-----------+
+----------------+----------------+
|
+-------v-------+
| VFIO core |
+-------+-------+
|
+---------------------v---------------------+
| NVIDIA vGPU VFIO variant driver |
+---------------------+---------------------+
|
+---------------+----------------------+
| binds PCI VF | PF lifecycle API
| | open / close / reset
v v
+--------+--------+ +-------------+-------------+
| PCI VF(s) | | nova-core (PF driver) |
| (virtual GPU) | | vGPU Manager |
+--------+--------+ | VRAM / channels ... |
| +-------------+-------------+
| |
+-------------v--------------------------------------v----------------+
| NVIDIA physical GPU |
| VF resources PF / GSP / VRAM |
+---------------------------------------------------------------------+
The manager queries the type assigned to a VF by GSP-RM, reserves its
channel and VRAM resources, boots and configures the GSP plugin, and
handles shutdown. It also scrubs guest VRAM before reuse and exposes
per-instance plugin logs through debugfs.
Host resources remain owned until firmware teardown and BAR unmapping
complete. Failed creation rolls back through the instance owner; if
firmware ownership is uncertain or teardown fails, the manager retains
the affected resources until device removal.
This continues the rework of the earlier vGPU RFC [2]: firmware-facing
management belongs to nova-core, while the VFIO driver binds to the VF
and uses the PF's services. The companion VFIO driver does not need its
own copies of the RM firmware headers.
The patches are organized as follows:
1 Retain BAR1 reservations after an unmap failure.
2-8 Boot state, runtime parameters and the 48-VM WPR2 heap.
9-12 VRAM allocation, region views and BAR mappings.
13-14 Synchronous GMC transactions and completion events.
15-19 Plugin bindings, property queries and instance allocation.
20-28 Plugin boot, shutdown and RPC.
29-31 CeUtils scrubbing and plugin logs.
32 SR-IOV PF lifecycle APIs.
Changes since v3:
* Include the BAR1 unmap-failure fix as patch 1.
* Rebase onto r000 firmware stack and drop changes already supplied by the prerequisites.
* Adapt boot-state/VF information handling, firmware encoding and GMC
transactions to the interfaces supplied by that newer stack.
Dependencies:
This series is based on drm-rust-next at
41871f4ddf710339863f957bcfcab5094c234771 plus the r000 GSP firmware
and other integration prerequisites.
The complete integration tree, including the prerequisites and the
companion VFIO driver, is available at:
https://github.com/alokk-nvidia/nova-core/tree/nova-vgpu-wip-20261105
Previous versions:
v3: [1]
v2: https://lore.kernel.org/nova-gpu/20260914074907.1399629-1-zhiw@nvidia.com/
v1: https://lore.kernel.org/nova-gpu/20260905081116.106613-1-zhiw@nvidia.com/
[1] https://lore.kernel.org/nova-gpu/cover.1790580105.git.zhiw@nvidia.com/
[2] https://lore.kernel.org/kvm/20250903221111.3866249-1-zhiw@nvidia.com/
Alok Kumar (2):
gpu: nova-core: gsp: decode the FIFO engine table
gpu: nova-core: vgpu: add VRAM slot allocator
Zhi Wang (30):
gpu: nova-core: mm: retain BAR1 reservations after unmap failure
gpu: nova-core: gsp: pass boot context through setup helpers
gpu: nova-core: gsp: decouple boot context from VgpuManager
gpu: nova-core: vgpu: detect boot state independently
gpu: nova-core: gpu: add a channel ID pool for vGPU
gpu: nova-core: vgpu: initialize runtime parameters after GSP boot
gpu: nova-core: vgpu: reserve the 48-VM WPR2 heap
gpu: nova-core: mm: borrow BarUser for temporary BAR1 access
gpu: nova-core: mm: add VramBlock
gpu: nova-core: mm: add VramRegion
gpu: nova-core: mm: add BarMapping
gpu: nova-core: gsp: add synchronous GMC transactions
gpu: nova-core: gsp: wait for GMC completion events
gpu: nova-core: vgpu: add r000 plugin bindings
gpu: nova-core: gsp: factor out NVKV payload conversion
gpu: nova-core: vgpu: query VF assignments and properties
gpu: nova-core: vgpu: add instance create/destroy
gpu: nova-core: vgpu: encode vGPU boot requests
gpu: nova-core: vgpu: add GSP plugin communication buffers
gpu: nova-core: vgpu: add instance boot
gpu: nova-core: vgpu: add instance shutdown
gpu: nova-core: vgpu: initialize GSP plugin RPC buffers
gpu: nova-core: vgpu: add GSP plugin RPC transactions
gpu: nova-core: vgpu: negotiate the GSP plugin RPC version
gpu: nova-core: vgpu: send GSP plugin configuration parameters
gpu: nova-core: vgpu: update the GSP plugin BME state
gpu: nova-core: vgpu: add CeUtils commands
gpu: nova-core: vgpu: scrub guest VRAM with CeUtils
gpu: nova-core: vgpu: export plugin log buffers via debugfs
gpu: nova-core: vgpu: introduce SR-IOV PF APIs
Documentation/gpu/nova/core/interrupts.rst | 20 +-
drivers/gpu/nova-core/fb.rs | 28 +-
drivers/gpu/nova-core/gpu.rs | 111 ++-
drivers/gpu/nova-core/gpu/channel.rs | 4 +
drivers/gpu/nova-core/gsp.rs | 30 +-
drivers/gpu/nova-core/gsp/boot.rs | 8 +-
drivers/gpu/nova-core/gsp/cmdq.rs | 28 +-
drivers/gpu/nova-core/gsp/cmdq/gmc.rs | 223 +++++-
drivers/gpu/nova-core/gsp/commands.rs | 11 +
drivers/gpu/nova-core/gsp/commands/gmc.rs | 34 +-
drivers/gpu/nova-core/gsp/fw.rs | 44 +-
drivers/gpu/nova-core/gsp/fw/commands.rs | 108 ++-
.../gpu/nova-core/gsp/fw/r000_00/bindings.rs | 175 +++++
drivers/gpu/nova-core/gsp/hal/gh100.rs | 2 +-
drivers/gpu/nova-core/gsp/hal/tu102.rs | 2 +-
drivers/gpu/nova-core/gsp/nvkv.rs | 42 +-
drivers/gpu/nova-core/gsp/nvkv/encode.rs | 12 +
drivers/gpu/nova-core/mm.rs | 1 +
drivers/gpu/nova-core/mm/bar_user.rs | 209 +++++-
drivers/gpu/nova-core/mm/vmm.rs | 18 +-
drivers/gpu/nova-core/mm/vram.rs | 131 ++++
drivers/gpu/nova-core/nova_core.rs | 6 +
drivers/gpu/nova-core/regs.rs | 31 +-
drivers/gpu/nova-core/vgpu.rs | 111 ++-
drivers/gpu/nova-core/vgpu/commands.rs | 321 +++++++++
drivers/gpu/nova-core/vgpu/fw.rs | 70 ++
drivers/gpu/nova-core/vgpu/fw/commands.rs | 470 +++++++++++++
drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs | 496 ++++++++++++++
drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs | 170 +++++
drivers/gpu/nova-core/vgpu/instance.rs | 642 ++++++++++++++++++
drivers/gpu/nova-core/vgpu/log.rs | 171 +++++
drivers/gpu/nova-core/vgpu/scrubber.rs | 173 +++++
drivers/gpu/nova-core/vgpu/vgpu_api.rs | 158 +++++
drivers/gpu/nova-core/vgpu/vram.rs | 179 +++++
34 files changed, 4085 insertions(+), 154 deletions(-)
create mode 100644 drivers/gpu/nova-core/mm/vram.rs
create mode 100644 drivers/gpu/nova-core/vgpu/commands.rs
create mode 100644 drivers/gpu/nova-core/vgpu/fw.rs
create mode 100644 drivers/gpu/nova-core/vgpu/fw/commands.rs
create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_comm.rs
create mode 100644 drivers/gpu/nova-core/vgpu/gsp_plugin_rpc.rs
create mode 100644 drivers/gpu/nova-core/vgpu/instance.rs
create mode 100644 drivers/gpu/nova-core/vgpu/log.rs
create mode 100644 drivers/gpu/nova-core/vgpu/scrubber.rs
create mode 100644 drivers/gpu/nova-core/vgpu/vgpu_api.rs
create mode 100644 drivers/gpu/nova-core/vgpu/vram.rs
base-commit: ccfd86490ddf6fd9c8f59a1a3171c652afb44c7f
--
2.53.0
next reply other threads:[~2026-11-06 5:41 UTC|newest]
Thread overview: 33+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-06 5:39 Alok Kumar [this message]
2026-11-06 5:39 ` [PATCH v4 01/32] gpu: nova-core: mm: retain BAR1 reservations after unmap failure Alok Kumar
2026-11-06 5:39 ` [PATCH v4 02/32] gpu: nova-core: gsp: pass boot context through setup helpers Alok Kumar
2026-11-06 5:39 ` [PATCH v4 03/32] gpu: nova-core: gsp: decouple boot context from VgpuManager Alok Kumar
2026-11-06 5:39 ` [PATCH v4 04/32] gpu: nova-core: vgpu: detect boot state independently Alok Kumar
2026-11-06 5:39 ` [PATCH v4 05/32] gpu: nova-core: gpu: add a channel ID pool for vGPU Alok Kumar
2026-11-06 5:39 ` [PATCH v4 06/32] gpu: nova-core: gsp: decode the FIFO engine table Alok Kumar
2026-11-06 5:39 ` [PATCH v4 07/32] gpu: nova-core: vgpu: initialize runtime parameters after GSP boot Alok Kumar
2026-11-06 5:39 ` [PATCH v4 08/32] gpu: nova-core: vgpu: reserve the 48-VM WPR2 heap Alok Kumar
2026-11-06 5:39 ` [PATCH v4 09/32] gpu: nova-core: mm: borrow BarUser for temporary BAR1 access Alok Kumar
2026-11-06 5:39 ` [PATCH v4 10/32] gpu: nova-core: mm: add VramBlock Alok Kumar
2026-11-06 5:39 ` [PATCH v4 11/32] gpu: nova-core: mm: add VramRegion Alok Kumar
2026-11-06 5:39 ` [PATCH v4 12/32] gpu: nova-core: mm: add BarMapping Alok Kumar
2026-11-06 5:39 ` [PATCH v4 13/32] gpu: nova-core: gsp: add synchronous GMC transactions Alok Kumar
2026-11-06 5:39 ` [PATCH v4 14/32] gpu: nova-core: gsp: wait for GMC completion events Alok Kumar
2026-11-06 5:39 ` [PATCH v4 15/32] gpu: nova-core: vgpu: add r000 plugin bindings Alok Kumar
2026-11-06 5:39 ` [PATCH v4 16/32] gpu: nova-core: vgpu: add VRAM slot allocator Alok Kumar
2026-11-06 5:39 ` [PATCH v4 17/32] gpu: nova-core: gsp: factor out NVKV payload conversion Alok Kumar
2026-11-06 5:39 ` [PATCH v4 18/32] gpu: nova-core: vgpu: query VF assignments and properties Alok Kumar
2026-11-06 5:39 ` [PATCH v4 19/32] gpu: nova-core: vgpu: add instance create/destroy Alok Kumar
2026-11-06 5:39 ` [PATCH v4 20/32] gpu: nova-core: vgpu: encode vGPU boot requests Alok Kumar
2026-11-06 5:39 ` [PATCH v4 21/32] gpu: nova-core: vgpu: add GSP plugin communication buffers Alok Kumar
2026-11-06 5:39 ` [PATCH v4 22/32] gpu: nova-core: vgpu: add instance boot Alok Kumar
2026-11-06 5:39 ` [PATCH v4 23/32] gpu: nova-core: vgpu: add instance shutdown Alok Kumar
2026-11-06 5:39 ` [PATCH v4 24/32] gpu: nova-core: vgpu: initialize GSP plugin RPC buffers Alok Kumar
2026-11-06 5:39 ` [PATCH v4 25/32] gpu: nova-core: vgpu: add GSP plugin RPC transactions Alok Kumar
2026-11-06 5:39 ` [PATCH v4 26/32] gpu: nova-core: vgpu: negotiate the GSP plugin RPC version Alok Kumar
2026-11-06 5:40 ` [PATCH v4 27/32] gpu: nova-core: vgpu: send GSP plugin configuration parameters Alok Kumar
2026-11-06 5:40 ` [PATCH v4 28/32] gpu: nova-core: vgpu: update the GSP plugin BME state Alok Kumar
2026-11-06 5:40 ` [PATCH v4 29/32] gpu: nova-core: vgpu: add CeUtils commands Alok Kumar
2026-11-06 5:40 ` [PATCH v4 30/32] gpu: nova-core: vgpu: scrub guest VRAM with CeUtils Alok Kumar
2026-11-06 5:40 ` [PATCH v4 31/32] gpu: nova-core: vgpu: export plugin log buffers via debugfs Alok Kumar
2026-11-06 5:40 ` [PATCH v4 32/32] gpu: nova-core: vgpu: introduce SR-IOV PF APIs Alok Kumar
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=cover.1793906667.git.alkumar@nvidia.com \
--to=alkumar@nvidia.com \
--cc=a.hindborg@kernel.org \
--cc=acourbot@nvidia.com \
--cc=airlied@gmail.com \
--cc=alex.gaynor@gmail.com \
--cc=alex@shazbot.org \
--cc=aliceryhl@google.com \
--cc=aniketa@nvidia.com \
--cc=ankita@nvidia.com \
--cc=apopple@nvidia.com \
--cc=bhelgaas@google.com \
--cc=bjorn3_gh@protonmail.com \
--cc=boqun.feng@gmail.com \
--cc=boqun@kernel.org \
--cc=cjia@nvidia.com \
--cc=dakr@kernel.org \
--cc=daniel.almeida@collabora.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=ecourtney@nvidia.com \
--cc=gary@garyguo.net \
--cc=jgg@nvidia.com \
--cc=jgg@ziepe.ca \
--cc=jhubbard@nvidia.com \
--cc=joelagnelf@nvidia.com \
--cc=kevin.tian@intel.com \
--cc=kjaju@nvidia.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=kwilczynski@kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-pci@vger.kernel.org \
--cc=lossin@kernel.org \
--cc=markus.probst@posteo.de \
--cc=nova-gpu@lists.linux.dev \
--cc=ojeda@kernel.org \
--cc=peter@colberg.org \
--cc=rust-for-linux@vger.kernel.org \
--cc=simona@ffwll.ch \
--cc=skolothumtho@nvidia.com \
--cc=smitra@nvidia.com \
--cc=tamird@kernel.org \
--cc=targupta@nvidia.com \
--cc=tmgross@umich.edu \
--cc=work@onurozkan.dev \
--cc=xinmengs@nvidia.com \
--cc=yishaih@nvidia.com \
--cc=zhiw@nvidia.com \
--cc=zhiwang@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox