Re: [RFC net-next v1 1/2] udp: Introduce UDP_STOP_RCV option for UDP

[Willem de Bruijn <willemdebruijn.kernel@gmail.com>]

Kuniyuki Iwashima wrote:
> From: "Jiayuan Chen" <jiayuan.chen@linux.dev>
> Date: Thu, 01 May 2025 06:22:17 +0000
> > 2025/5/1 12:42, "Kuniyuki Iwashima" <kuniyu@amazon.com> wrote:
> > 
> > > 
> > > From: Jiayuan Chen <jiayuan.chen@linux.dev>
> > > 
> > > Date: Thu, 1 May 2025 11:51:08 +0800
> > > 
> > > > 
> > > > For some services we are using "established-over-unconnected" model.
> > > > 
> > > >  
> > > > 
> > > >  '''
> > > > 
> > > >  // create unconnected socket and 'listen()'
> > > > 
> > > >  srv_fd = socket(AF_INET, SOCK_DGRAM)
> > > > 
> > > >  setsockopt(srv_fd, SO_REUSEPORT)
> > > > 
> > > >  bind(srv_fd, SERVER_ADDR, SERVER_PORT)
> > > > 
> > > >  
> > > > 
> > > >  // 'accept()'
> > > > 
> > > >  data, client_addr = recvmsg(srv_fd)
> > > > 
> > > >  
> > > > 
> > > >  // create a connected socket for this request
> > > > 
> > > >  cli_fd = socket(AF_INET, SOCK_DGRAM)
> > > > 
> > > >  setsockopt(cli_fd, SO_REUSEPORT)
> > > > 
> > > >  bind(cli_fd, SERVER_ADDR, SERVER_PORT)
> > > > 
> > > >  connect(cli, client_addr)
> > > > 
> > > >  ...
> > > > 
> > > >  // do handshake with cli_fd
> > > > 
> > > >  '''
> > > > 
> > > >  
> > > > 
> > > >  This programming pattern simulates accept() using UDP, creating a new
> > > > 
> > > >  socket for each client request. The server can then use separate sockets
> > > > 
> > > >  to handle client requests, avoiding the need to use a single UDP socket
> > > > 
> > > >  for I/O transmission.
> > > > 
> > > >  
> > > > 
> > > >  But there is a race condition between the bind() and connect() of the
> > > > 
> > > >  connected socket:
> > > > 
> > > >  We might receive unexpected packets belonging to the unconnected socket
> > > > 
> > > >  before connect() is executed, which is not what we need.
> > > > 
> > > >  (Of course, before connect(), the unconnected socket will also receive
> > > > 
> > > >  packets from the connected socket, which is easily resolved because
> > > > 
> > > >  upper-layer protocols typically require explicit boundaries, and we
> > > > 
> > > >  receive a complete packet before creating a connected socket.)
> > > > 
> > > >  
> > > > 
> > > >  Before this patch, the connected socket had to filter requests at recvmsg
> > > > 
> > > >  time, acting as a dispatcher to some extent. With this patch, we can
> > > > 
> > > >  consider the bind and connect operations to be atomic.
> > > > 
> > > 
> > > SO_ATTACH_REUSEPORT_EBPF is what you want.
> > > 
> > > The socket won't receive any packets until the socket is added to
> > > 
> > > the BPF map.
> > > 
> > > No need to reinvent a subset of BPF functionalities.
> > >
> > 
> > I think this feature is for selecting one socket, not filtering out certain
> > sockets.
> > 
> > Does this mean that I need to first capture all sockets bound to the same
> > port, and then if the kernel selects a socket that I don't want to receive
> > packets on, I'll need to implement an algorithm in the BPF program to
> > choose another socket from the ones I've captured, in order to avoid
> > returning that socket?
> 
> Right.
> 
> If you want a set of sockets to listen on the port, you can implement
> as such with BPF; register the sockets to the BPF map, and if kernel pick
> up other sockets and triggers the BPF prog, just return one of the
> registerd sk.
> 
> Even when you have connect()ed sockets on the same port, kernel will
> fall back to the normal scoring to find the best one, and it's not a
> problem as the last 'result' is one selected by BPF or a connected sk,
> and the packet won't be routed to not-yet-registered unconnected sk.
> 
> 
> > 
> > This looks like it completely bypasses the kernel's built-in scoring
> > logic. Or is expanding BPF_PROG_TYPE_SK_REUSEPORT to have filtering
> > capabilities also an acceptable solution?

Reuseport BPF exists because we want to avoid having to continue to
add custom rules in C for each scenario.

In this case, I did wonder whether it is possible to avoid hitting
the soon-to-be connected socket with the standard reuseport
algorithm in reuseport_select_sock_by_hash.

Setting SO_INCOMING_CPU to a cpu on which no packets arrive will
lower its priority relative to other sockets. It's a bit of a hack,
but should work?