Re: Upstream Homa?

[Andrew Lunn <andrew@lunn.ch>]

On Fri, Nov 11, 2022 at 10:59:58AM -0800, John Ousterhout wrote:
> The netlink and 32-bit kernel issues are new for me; I've done some digging to
> learn more, but still have some questions.
> 

> * Is the intent that netlink replaces *all* uses of /proc and ioctl? Homa
> currently uses ioctls on sockets for I/O (its APIs aren't sockets-compatible).
> It looks like switching to netlink would double the number of system calls that
> have to be invoked, which would be unfortunate given Homa's goal of getting the
> lowest possible latency. It also looks like netlink might be awkward for
> dumping large volumes of kernel data to user space (potential for buffer
> overflow?).

I've not looked at the actually code, i'm making general comments.

netlink, like ioctl, is meant for the control plain, not the data
plain. Your statistics should be reported via netlink, for
example. netlink is used to configure routes, setup bonding, bridges
etc. netlink can also dump large volumes of data, it has no problems
dumping the full Internet routing table for example.

How you get real packet data between the userspace and kernel space is
a different question. You say it is not BSD socket compatible. But
maybe there is another existing kernel API which will work? Maybe post
what your ideal API looks like and why sockets don't work. Eric
Dumazet could give you some ideas about what the kernel has which
might do what you need. This is the uAPI point that Stephen raised.

> * By "32 bit kernel problems" are you referring to the lack of atomic 64-bit
> operations and using the facilities of u64_stats_sync.h, or is there a more
> general issue with 64-bit operations?

Those helpers do the real work, and should optimise to pretty much
nothing on an 64 bit kernel, but do the right thing on 32 bit kernels.

But you are right, the general point is that they are not atomic, so
you need to be careful with threads, and any access to a 64 bit values
needs to be protected somehow, hopefully in a way that is optimised
out on 64bit systems.

      Andrew