Re: [RFC PATCH] misc: fgds: enable GPU-NVMe direct I/O via POSIX and io_uring

[Li Wang <liwang@kylinos.cn>]

Hi Christian,

>>>> Nor why "fgds" is the name, that's going to be hard to remember, does it
>>>> stand for something?
>>>>
>>> "FGDS" stands for Fast GPUDirect Storage. GPUDirect Storage (GDS) is NVIDIA's 
>>> technology enabling direct I/O between GPU memory and files on NVMe, 
>>> widely used in LLM workloads to bypass CPU overhead.
>>
>> That's nvidia's specific solution, but this works on other devices,
>> right?  Or just for that one platform?
> 
> That was nvidia's specific and very hacky out of tree solution 
Exactly, I completely agree with your viewpoint. In fact, we previously conducted 
a deep analysis of NVIDIA's GDS implementation code, which was also one of the 
motivations for us to develop fgds.

Please allow me to introduce NVIDIA's GDS implementation briefly:
NVIDIA introduced two kernel modules: one module called nvidia-fs, and the other module 
is a customized NVMe driver replacing the Linux kernel's default NVMe driver. 
nvidia-fs creates a device file for every NVIDIA GPU on the machine: /dev/nvidia-fs<gpu-id>.

Applications call cuFileBufRegister to register the GPU memory buffer to perform I/O. 
cuFileBufRegister executes the following operations: calls nvidia-fs via ioctl. The 
implementation of this ioctl allocates a corresponding phony buffer of the same size 
in the host memory, and establishes a mapping between the CPU memory phony buffer 
and the GPU memory buffer.

Then, the application calls cuFileRead/cuFileWrite to perform file I/O operations. 
Its implementation calls ioctl on nvidia-fs with NVFS_IOCTL_READ/NVFS_IOCTL_WRITE 
as parameters. Inside the ioctl implementation, it calls the common kernel interface 
filp->f_op->read_iter/write_iter() on the file on NVME. These interfaces can only take 
the CPU memory phony buffer address as a input, constructing read/write requests sent 
through the block layer to the customized NVMe driver.

The customized NVMe driver intercepts the I/O operations, calls the nvidia-fs interface to 
query, replaces the phony buffer address with the GPU memory dma address, and performs 
DMA transfer between GPU memory and NVMe.

As we can see, this implementation is indeed very hacky and non-elegant. Furthermore, 
as evaluated in this paper published in SC'25 [1], the phony buffer brings a considerable 
performance overhead.

> which as far as I know is pretty much abandoned everywhere.
However, despite the overhead of phony buffers, GDS performance is still significantly higher 
than transferring through CPU host memory (as shown in our performance benchmark tests [2]). 
Therefore, GDS is actually still widely used in the LLM ecosystem. For instance, model loading 
plugins used in inference engines like vLLM and SGLang—such as fastsafetensors and InstantTensor—
both support acceleration via GDS [3,4], with fastsafetensors enabling GDS by default for model loading. 
Furthermore, LMCache, a plugin used for KV cache offloading in vLLM and SGLang, also supports 
GDS acceleration [5]. PyTorch itself also provides file access APIs based on GDS [6].

> 
> AMD came up with something similar, but all those approaches are so fundamentally broken that we didn't even considered upstreaming it.
>> And you are using this as a "bypass" for the normal accel subsystem,
>> shouldn't this be part of that subsystem instead of a custom user/kernel
>> api like you are creating here?
> 
> As far as I know there is a patch set under review and even already partially merged which enables exactly that functionality as general feature for DMA-buf which is vendor independent and should at least in theory work with all drivers.
> 
> I'm really surprised that somebody is still working on the vendor specific stuff.
As you pointed out, every vendor has been inventing their own way and interfaces to support GDS, 
introducing custom kernel modules and proprietary UAPI interfaces, with varying performance that 
leaves developers heavily frustrated. Apologies for not making this clear enough in our commit 
messages, which understandably caused some confusion. We merely borrowed the name "GDS" to describe 
the functional purpose of fgds.

In fact, we believe fgds offers four key advantages: 
(1) GPU platform independence; 
(2) POSIX/io_uring interface compatibility;
(3) Higher performance than GDS;
(4) Minimal kernel footprint and UAPI footprint

Regarding (1), (2), and (3), please allow me to briefly explain the design mechanism of fgds:
fgds turns a GPU memory buffer into a POSIX/io_uring-compatible user-space virtual address via 
three main steps:

Step 1: Utilizing ZONE_DEVICE support, we remap the GPU memory exposed via PCIe BAR into struct pages 
using devm_memremap_pages();

Step 2: Utilizing dma-buf support, the GPU memory buffer is exported as a dma-buf file descriptor (fd). 
Using this fd as a bridge, we look up the corresponding DMA addresses for the GPU memory buffer inside 
the kernel;

Step 3: Through mmap, we insert the struct pages corresponding to the GPU memory buffer into the userspace 
VMA, mapping their physical/DMA addresses directly. The virtual address returned by mmap can then be directly 
passed into standard POSIX or io_uring interfaces.

As you can see, since almost all major GPU vendors support exporting GPU memory buffers via dma-buf, 
all remaining technical dependencies of fgds rely on standard Linux kernel infrastructure. Therefore, 
fgds is completely vendor-agnostic and natively compatible with POSIX/io_uring without introducing any proprietary 
vendor interfaces, which greatly simplifies development, deployment, operations and unifies standard usage. 
Furthermore, because this technique completely eliminates the phony buffer, its performance is significantly 
better than NVIDIA's GDS (as shown in our benchmarks [2]).

In addition, since fgds uses only the most fundamental dma-buf mechanisms, it relies on baseline dma-buf features 
that have been supported in the upstream kernel for a long time, rather than any new dma-buf features currently 
under active development. In fact, before we recently ported fgds to the latest kernel tree, it was developed 
and ran on our internal 6.6 kernel. It has been running stably in our production clusters for over half a year 
across various hardware platforms (including NVIDIA, AMD, and several other vendors) without requiring a single 
line of GPU-platform-specific fgds code modification.

Regarding (4): We believe that implementing GDS-like functionality inherently requires kernel assistance to map 
the GPU memory buffer to a valid userspace virtual address. This inevitably requires userspace-kernel interaction. 
To the best of our knowledge, the mainline kernel currently lacks a dedicated, unified path for this specific interaction, 
which is why various vendors ended up writing their own out-of-tree interfaces. In contrast, fgds introduces 
only one single new ioctl parameter (REG_BUFFER, excluding UNREG_BUFFER), and confines its scope strictly to a 
standalone device driver. We believe this achieves a minimal kernel footprint and minimal UAPI addition.

[1] https://dl.acm.org/doi/10.1145/3712285.3759862
[2] https://github.com/Storage-and-OS-for-AI/fgds
[3] https://github.com/foundation-model-stack/fastsafetensors/blob/main/docs/configuration.md
[4] https://github.com/scitix/InstantTensor/blob/main/csrc/loader_io_cufile.cpp
[5] https://github.com/LMCache/LMCache/blob/dev/lmcache/v1/storage_backend/gds_backend.py
[6] https://docs.pytorch.org/docs/2.14/generated/torch.cuda.gds.GdsFile.html

Thanks,
Li Wang> Regards,
> Christian.
> 
>>
>> thanks,
>>
>> greg k-h