Re: [PATCH resend] netlink.7: note not reliable if NETLINK_NO_ENOBUFS

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi Alexander,

On Thu, Mar 04, 2021 at 03:57:28PM -0500, Alexander Aring wrote:
> This patch adds a note to the netlink manpage that if NETLINK_NO_ENOBUFS
> is set there is no additional handling to make netlink reliable. It just
> disables the error notification.

A bit more background on this toggle.

NETLINK_NO_ENOBUFS also disables netlink broadcast congestion control
which kicks in when the socket buffer gets full. The existing
congestion control algorithm keeps dropping netlink event messages
until the queue is emptied. Note that it might take a while until your
userspace process fully empties the socket queue that is congested
(and during that time _your process is losing every netlink event_).

The usual approach when your process hits ENOBUFS is to resync via
NLM_F_DUMP unicast request. However, getting back to sync with the
kernel subsystem might be expensive if the number of items that are
exposed via netlink is huge.

Note that some people select very large socket buffer queue for
netlink sockets when they notice ENOBUFS. This might however makes
things worse because, as I said, congestion control drops every
netlink message until the queue is emptied. Selecting a large socket
buffer might help to postpone the ENOBUFS error, but once your process
hits ENOBUFS, then the netlink congestion control kicks in and you
will make you lose a lot of event messages (until the queue is empty
again!).

So NETLINK_NO_ENOBUFS from userspace makes sense if:

1) You are subscribed to a netlink broadcast group (so it does _not_
   make sense for unicast netlink sockets).
2) The kernel subsystem delivers the netlink messages you are
   subscribed to from atomic context (e.g. network packet path, if
   the netlink event is triggered by network packets, your process
   might get spammed with a lot of netlink messages in little time,
   depending on your network workload).
3) Your process does not want to resync on lost netlink messages.
   Your process assumes that events might get lost but it does not
   case / it does not want to make any specific action in such case.
4) You want to disable the netlink broadcast congestion control.

To provide an example kernel subsystem, this toggle can be useful with
the connection tracking system, when monitoring for new connection
events in a soft real-time fashion.

> The used word "avoid" receiving ENOBUFS errors can be interpreted
> that netlink tries to do some additional queue handling to avoid
> that such scenario occurs at all, e.g. like zerocopy which tries to
> avoid memory copy. However disable is not the right word here as
> well that in some cases ENOBUFS can be still received. This patch
> makes clear that there will no additional handling to put netlink in
> a more reliable mode.

Right, the NETLINK_NO_ENOBUFS toggle alone itself is not making
netlink more reliable for the broadcast scenario, it just changes the
way it netlink broadcast deals with congestion: userspace process gets
no reports on lost messages and netlink congestion control is
disabled.