Re: [MAINTAINERS SUMMIT] Device Passthrough Considered Harmful?

[Jason Gunthorpe <jgg@nvidia.com>]

On Thu, Jul 11, 2024 at 08:16:55AM -0700, James Bottomley wrote:

> > > What all of the prior pass through's taught us is that if the use
> > > case is big enough it will get pulled into the kernel and the
> > > kernel will usually manage it better (DB users).  If it remains a
> > > niche use case it will likely remain out of the kernel, but we
> > > won't be hurt by it (NVME KV protocol) and sometimes it doesn't
> > > really matter and the device manufacturers will sort it out on
> > > their own (USB tokens).
> > 
> > I don't see it as being linked to big enough use case at all.
> 
> 'it' being fwctl?  

Sorry I ment "it" as "if the use case is big enough it will get pulled
into the kernel"

> I'm in the camp that doesn't squash a novel proposal because the kernel
> should be controlling it.  I'm confident that if the use case becomes
> big enough the kernel will likely do it in the end.

Yes, I agree as well. If there is technical excellence to draw
someting into the kernel then it will definitely happen regardless if
other options are available.

> > While DPDK shows the opposite, userspace is the technically better
> > option. This is now shown at scale. DPDK is not some niche. A big
> > chunk of internet traffic is going through DPDKs, especially for
> > mobile. Many ORAN solutions include DPDK on Linux.
> 
> ORAN being Open Radio Access Network?  

Yes

> But that's a case in point: the kernel doesn't have a LTE stack or
> APN handling for networking.

That isn't really the main point of ORAN, a native LTE stack is
interesting separately but is a different problem space than ORAN.

ORAN is about doing all the complex maths for RF signal processing in
software disaggregated from the antennae hardware, in realtime. It is
some bonkers combination of >= 100G networking and hard realtime
packet processing with incredible math using dedicated accelerator HW.

> I don't disagree: there are many novel protocols and other use cases
> that will never make it into the kernel simply because they won't get
> the adoption;

My point is that prefering userspace/kernel isn't about NOVEL, it is
about technical excellence.

DPDK running a >= 100G NIC with extensive HW packet processing
offloads is technically excellent at doing proprietary traffic
processing work in mobile and cloud networks.

We are already running these DPDK things at near perfect performance
with minimal integration hurdles, there is not a lot of room to
actually be *better* with an in-kernel alternative.

This is what I mean, technically good displaces technically bad, it
has nothing to do with kernel=good userspace=bad.

The design of DPDK, with reduced kernel involvment, is a technically
correct approach to achiving it's goal, and it's goal is legitimate.

To expand a bit, Mellanox has supported both paths for a long time, a
lot of work has been done to improve the netstack and mlx5 to give
matching performance compared to DPDK in some use cases. Yet, I think
the market has spoken and DPDK seems to have much more popularity in
some very big verticals.

Yes, using DPDK carries a heavy development cost and people should
avoid it if they don't see a 5-10x improvment, IMHO.

Jason