nftables netlink cache initialization failure with dnsmasq

nftables netlink cache initialization failure with dnsmasq

[Monib]

Hello,

An OpenWRT user here who has been trying to set up split tunneling
using https://docs.openwrt.melmac.net/pbr/, which uses dnsmasq and
nftables, but I am having some issues.

I am encountering an error — "netlink: Error: cache initialization
failed: Protocol error" — which seems to be produced by nftables. This
error message was introduced in the following commit:
https://git.netfilter.org/nftables/commit/?id=a2ddb38f7eb818312c50be78028bc35145c039ae.
The commit message says: "cache initialization failure (which should
not ever happen) is not reported to the user."

The issue starts happening semi-randomly but seems to occur when too
many DNS requests are made in a short period. Once it appears, the
relevant nftables sets stop being populated by dnsmasq.

Here is what I see in the logs:

Sun Mar 23 17:52:24 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:52:33 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg046ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:52:58 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:08 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:22 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg076ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:22 2025 daemon.err dnsmasq[4877]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg076ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:53 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
Sun Mar 23 17:54:53 2025 daemon.err dnsmasq[4]: nftset inet fw4
pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
failed: Protocol error
... (many similar entries)

I ran dnsmasq under strace, hoping to gather more insight:
https://github.com/user-attachments/files/19410818/strace.log.tar.gz

I am still unsure where the actual problem is between dnsmasq and
nftables, but since the error message seems to be coming from
nftables, I am hoping someone here can help me figure out what is
going on or point me in the right direction.

Hardware:
- Router: Linksys EA8100
- SoC: MediaTek MT7621 ver:1 eco:3

Software:
- OpenWRT Version: 24.10.0 (r28427-6df0e3d02a)
- Kernel: 6.6.73
- Dnsmasq: 2.90 (Compile options: IPv6 GNU-getopt no-DBus UBus no-i18n
no-IDN DHCP DHCPv6 no-Lua TFTP conntrack no-ipset nftset auth
cryptohash DNSSEC no-ID loop-detect inotify dumpfile)
- Nftables: v1.1.1 (Commodore Bullmoose #2)

For additional context:
Originally, I thought this was an issue with the PBR packages, so I
posted this on the OpenWRT forum:
https://forum.openwrt.org/t/policy-based-routing-pbr-package-discussion/140639/1961?u=lov432
https://forum.openwrt.org/t/policy-based-routing-pbr-package-discussion/140639/1987?u=lov432

However, they seemed to think it might be related to dnsmasq, then I
created an issue on the OpenWRT GitHub:
https://github.com/openwrt/openwrt/issues/18333

And then finally this post here.

I am sorry for such a scattered post, but I wanted to provide as much
information as possible. Please let me know if you have any questions
or if there is anything else I can do to help make this more clear.

Thanks,
LoV432

Re: nftables netlink cache initialization failure with dnsmasq

[Pablo Neira Ayuso]

Hi,

On Tue, May 06, 2025 at 03:57:23PM +0500, Monib wrote:
> Hello,
> 
> An OpenWRT user here who has been trying to set up split tunneling
> using https://docs.openwrt.melmac.net/pbr/, which uses dnsmasq and
> nftables, but I am having some issues.
> 
> I am encountering an error — "netlink: Error: cache initialization
> failed: Protocol error" — which seems to be produced by nftables. This
> error message was introduced in the following commit:
> https://git.netfilter.org/nftables/commit/?id=a2ddb38f7eb818312c50be78028bc35145c039ae.
> The commit message says: "cache initialization failure (which should
> not ever happen) is not reported to the user."

This commit you refer above is exposing an existing issue.

> The issue starts happening semi-randomly but seems to occur when too
> many DNS requests are made in a short period. Once it appears, the
> relevant nftables sets stop being populated by dnsmasq.
> 
> Here is what I see in the logs:
> 
> Sun Mar 23 17:52:24 2025 daemon.err dnsmasq[4]: nftset inet fw4
> pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
> failed: Protocol error

EPROTO can be reported by libmnl with netlink sequence problems.

Quickly browsing dnsmasq code, it looks like there is a pool of child
processes that are sharing a single nft_ctx handle to handle events,
two or more child processes are racing.

I can expand libnftables(3) manpage to clarify this.

Thanks for reporting.

Re: nftables netlink cache initialization failure with dnsmasq

[Monib]

Hi,

Just wanted to let you know that I forwarded the issue and your
insight to dnsmasq, and they have applied a fix for it.
https://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2025q2/018168.html
Thank you so much for your help and for pointing me in the right direction!

Thanks,
LoV432

On Wed, May 7, 2025 at 1:24 AM Pablo Neira Ayuso <pablo@netfilter.org> wrote:
>
> Hi,
>
> On Tue, May 06, 2025 at 03:57:23PM +0500, Monib wrote:
> > Hello,
> >
> > An OpenWRT user here who has been trying to set up split tunneling
> > using https://docs.openwrt.melmac.net/pbr/, which uses dnsmasq and
> > nftables, but I am having some issues.
> >
> > I am encountering an error — "netlink: Error: cache initialization
> > failed: Protocol error" — which seems to be produced by nftables. This
> > error message was introduced in the following commit:
> > https://git.netfilter.org/nftables/commit/?id=a2ddb38f7eb818312c50be78028bc35145c039ae.
> > The commit message says: "cache initialization failure (which should
> > not ever happen) is not reported to the user."
>
> This commit you refer above is exposing an existing issue.
>
> > The issue starts happening semi-randomly but seems to occur when too
> > many DNS requests are made in a short period. Once it appears, the
> > relevant nftables sets stop being populated by dnsmasq.
> >
> > Here is what I see in the logs:
> >
> > Sun Mar 23 17:52:24 2025 daemon.err dnsmasq[4]: nftset inet fw4
> > pbr_wg_xray_4_dst_ip_cfg066ff5 netlink: Error: cache initialization
> > failed: Protocol error
>
> EPROTO can be reported by libmnl with netlink sequence problems.
>
> Quickly browsing dnsmasq code, it looks like there is a pool of child
> processes that are sharing a single nft_ctx handle to handle events,
> two or more child processes are racing.
>
> I can expand libnftables(3) manpage to clarify this.
>
> Thanks for reporting.