Re: nftables netlink cache initialization failure with dnsmasq

[Pablo Neira Ayuso <pablo@netfilter.org>]

Hi,

On Tue, May 06, 2025 at 03:57:23PM +0500, Monib wrote:
> Hello,
> 
> An OpenWRT user here who has been trying to set up split tunneling
> using https://docs.openwrt.melmac.net/pbr/, which uses dnsmasq and
> nftables, but I am having some issues.
> 
> I am encountering an error — "netlink: Error: cache initialization
> failed: Protocol error" — which seems to be produced by nftables. This
> error message was introduced in the following commit:
> https://git.netfilter.org/nftables/commit/?id=a2ddb38f7eb818312c50be78028bc35145c039ae.
> The commit message says: "cache initialization failure (which should
> not ever happen) is not reported to the user."

This commit you refer above is exposing an existing issue.

> The issue starts happening semi-randomly but seems to occur when too
> many DNS requests are made in a short period. Once it appears, the
> relevant nftables sets stop being populated by dnsmasq.
> 
> Here is what I see in the logs:
> 
> Sun Mar 23 17:52:24 2025 daemon.err dnsmasq[4]: nftset inet fw4
> pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
> failed: Protocol error

EPROTO can be reported by libmnl with netlink sequence problems.

Quickly browsing dnsmasq code, it looks like there is a pool of child
processes that are sharing a single nft_ctx handle to handle events,
two or more child processes are racing.

I can expand libnftables(3) manpage to clarify this.

Thanks for reporting.