Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe <jgg@nvidia.com>]

On Tue, Jul 09, 2024 at 09:02:25AM -0700, James Bottomley wrote:

> For NVMe and net we do have SPDK and DPDK.  What I find is that people
> tend to use them for niche use cases (like the NVMe KV command set) or
> obscure network routers.  Even though the claim they both make is to
> get the kernel out of the way and do stuff "way faster" the difficulty
> they create by bypassing everything is quite a high burden.

[..]
 
> What all of the prior pass through's taught us is that if the use case
> is big enough it will get pulled into the kernel and the kernel will
> usually manage it better (DB users).  If it remains a niche use case it
> will likely remain out of the kernel, but we won't be hurt by it (NVME
> KV protocol) and sometimes it doesn't really matter and the device
> manufacturers will sort it out on their own (USB tokens).

I don't see it as being linked to big enough use case at all.

The kernel gets involved if there are good technical reasons to do
so. Databases running over real filesystems with O_DIRECT is really
technically better than raw block devices.

While DPDK shows the opposite, userspace is the technically better
option. This is now shown at scale. DPDK is not some niche. A big
chunk of internet traffic is going through DPDKs, especially for
mobile. Many ORAN solutions include DPDK on Linux.

What has been improved kernel-side is the intergation. DPDK
deployments now often use RDMA raw queue pairs instead of VFIO, which
laregly eliminates the "high burden".

There are many other cases, like DPDK, where the right answer is to
reduce the kernel involvement. It is not so simple that things always
get pulled into the kernel.

Jason