Using netfilter to intercept packets written to an ipvtap device

Using netfilter to intercept packets written to an ipvtap device

[David Arroyo]

I am experimenting with using ipvtap devices for virtual machines &
user-space VPN. I want them to be able to configure their own IP/IPv6
addresses from the network using DHCP, DHCPv6 and SLAAC. IPVlan-based
devices can only receive unicast traffic on the address(es) that are
associated with the interface in the kernel.

I figured I could enable this with a program that snooped DHCP/ICMPv6
packets and configured the addresses it observed interfaces negotiating. I
thought that netfilter queues[1] would be a good way to intercept the
packets I was interested in. However I'm having a bear of a time getting
the client packets (DHCPDISCOVER, nd-router-solicit, etc) that the
VMs write to the tap device.

Here's the the part of my ruleset for SLAAC (full set here[2]):

table inet autoconf {
  chain output {
    type filter hook output priority 0; policy accept;
    counter meta protocol ip6 icmpv6 type { nd-router-solicit } queue;
  }
  chain input {
    type filter hook input priority 0; policy accept;
    counter meta protocol ip6 icmpv6 type { nd-router-advert } queue;
  }
}

And I have a user-space program listening on queue 0 that will configure
link-local addresses on the outdev observed in solicit messages, and
mask those with any prefixes learned in router-advertisements. My program
receives the router adverts (with the physical interface as iif), but not
the solicits.

I've done a lot of tweaking and experimentation, enabling tracing and
such. I can see the packets passing through the "ingress" and "egress"
hooks of the "netdev" family[3]. Here are two example traces of packets I
could use to learn the link-local and global address(es) that the client
is likely to use:

trace id 99f29220 netdev trace egress packet: oif "ipvtap22476" @nh,0,320 0x6000000000203afffe800000000000005e5f67fffefeae20ff020000000000000000000000000001 @th,0,160 0x8800002320000000fe800000000000005e5f67ff
trace id 8a307430 netdev trace ingress packet: iif "ipvtap22476" ether saddr 88:0f:a2:2f:09:60 ether daddr 33:33:00:00:00:01 ip6 saddr fe80::8a0f:a2ff:fe2f:960 ip6 daddr ff02::1 ip6 dscp cs0 ip6 ecn not-ect ip6 hoplimit 255 ip6 flowlabel 0 ip6 nexthdr ipv6-icmp ip6 length 136 icmpv6 type nd-router-advert icmpv6 code no-route icmpv6 parameter-problem 1078460596 @th,64,96 0x30440c0

However, the netdev family does not seem to support queueing to userspace.
All devices are in the same network namespace, so the packets should all
be passing through the same netfilter stack, but I cannot see the outbound
multicast/broadcast DHCP and ICMPv6 packets in any inet family hooks.

I've been plodding through the ipvlan code in the kernel, but it's
slow going. Can anyone help me understand where these packets enter
the kernel and if they pass through any hooks in the ip/ip6 family? Or
alternatively, another way to observe packets written to a tap device
that doesn't involve opening a raw socket for each device.

David

[1]: https://wiki.nftables.org/wiki-nftables/index.php/Queueing_to_userspace
[2]: https://paste.sr.ht/~droyo/afa648300fdd84e41eb083644501d663b56e3477
[3]: https://paste.sr.ht/~droyo/f01f17481efe4f6326a642c8f97dc19662fcf69c
[4]: https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks

Re: Using netfilter to intercept packets written to an ipvtap device

[David Arroyo]

I spent some more time thinking about this and reading the
code, and feel I understand what's going on a bit better.
It looks like frames written to a tap device go directly
to dev_queue_xmit, passing through the egress hook on the
way. And for an ipvlan device in l2 mode like mine, the
driver skips all the ip_* functions that would hit other
hooks and just calls the parent device's xmit methods, so
it makes sense that the inet hooks never see the packets
on their way out.

I've started a different approach using an AF_PACKET socket
and am finding it much less problematic than I feared.

David