Re: [BUG / question] in routing rules, some options (e.g. ipproto, sport) cause rules to be ignored in presence of packet marks

[Ido Schimmel <idosch@idosch.org>]

On Mon, Mar 29, 2021 at 11:52:10PM +0300, Ido Schimmel wrote:
> On Sun, Mar 28, 2021 at 04:05:29PM +0200, Michal Soltys wrote:
> > Hi,
> > 
> > I'm not sure how it behaved in earlier kernels (can check later), but it is
> > / looks bugged in at least recent 5.x+ ones (tests were done with 5.11.8 and
> > 5.10.25).
> > 
> > Consider following setup:
> > 
> > # ip -o ad sh
> > 1: lo    inet 127.0.0.1/8 scope host lo
> > 2: right1    inet 10.0.10.2/24 scope global
> > 3: right2    inet 10.0.20.2/24 scope global
> > 
> > # ip ro sh tab main
> > default via 10.0.10.1 dev right1
> > 10.0.10.0/24 dev right1 proto kernel scope link src 10.0.10.2
> > 10.0.20.0/24 dev right2 proto kernel scope link src 10.0.20.2
> > 
> > # ip ro sh tab 123
> > default via 10.0.20.1 dev right2 src 10.0.20.2
> > 
> > And routing rules:
> > 
> > 0:      from all lookup local
> > 9:      from all fwmark 0x1 ipproto udp sport 1194 lookup 123
> > 10:     from all ipproto udp sport 1194 lookup 123
> > 32766:  from all lookup main
> > 32767:  from all lookup default
> > 
> > This - without any mangling via ipt/nft or by other means - works correctly,
> > for example:
> > 
> > nc -u -p 1194 1.2.3.4 12345
> > 
> > will be routed out correctly via 'right2' using 10.0.20.2
> > 
> > But if we add mark to locally outgoing packets:
> > 
> > iptables -t mangle -A OUTPUT -j MARK --set-mark 1
> > 
> > Then *both* rule 9 and rule 10 will be ignored during reroute check. tcpdump
> > on interface 'right1' will show:
> > 
> > # tcpdump -nvi right1 udp
> > tcpdump: listening on right1, link-type EN10MB (Ethernet), snapshot length
> > 262144 bytes
> > 13:21:59.684928 IP (tos 0x0, ttl 64, id 8801, offset 0, flags [DF], proto
> > UDP (17), length 33)
> >     10.0.20.2.1194 > 1.2.3.4.12345: UDP, length 5
> > 
> > Initial routing decision in rule 10 will set the address correctly, but the
> > packet goes out via interface right1, ignoring both 9 and 10.
> > 
> > If I add another routing roule:
> > 
> > 8:      from all fwmark 0x1 lookup 123
> > 
> > Then the packects will flow correctly - but I *cannot* use (from the ones I
> > tested): sport, dport, ipproto, uidrange - as they will cause the rule to be
> > ignored. For example, this setup of routing rules will fail, if there is any
> > mark set on a packet (nc had uid 1120):
> > 
> > # ip ru sh
> > 0:      from all lookup local
> > 10:     from all ipproto udp lookup 123
> > 10:     from all sport 1194 lookup 123
> > 10:     from all dport 12345 lookup 123
> > 10:     from all uidrange 1120-1120 lookup 123
> > 32766:  from all lookup main
> > 32767:  from all lookup default
> > 
> > Adding correct fwmark to the above rules will have *no* effect either. Only
> > fwmark *alone* will work (or in combination with: iif, from, to - from the
> > ones I tested).
> > 
> > I peeked at fib_rule_match() in net/core/fib_rules.c - but it doesn't look
> > like there is anything wrong there. I initially suspected lack of
> > 'rule->mark &&' in mark related line - but considering that rules such as
> > 'from all fwmark 1 sport 1194 lookup main' also fail, it doesn't look like
> > it's the culprit (and mark_mask covers that test either way).
> > 
> > OTOH, perhaps nf_ip_reroute() / ip_route_me_harder() are somehow the culprit
> > here - but I haven't analyzed them yet. Perhaps it's just an issue of
> > changing output interface incorrectly after ip_route_me_harder() ?
> 
> ip_route_me_harder() does not set source / destination port in the
> flow key, so it explains why fib rules that use them are not hit after
> mangling the packet. These keys were added in 4.17, but I
> don't think this use case every worked. You have a different experience?

It's already tomorrow here, but I think that if you record the
'fib:fib_table_lookup' tracepoint before and after adding the mangling
rules you will see that there is a second lookup for the packet with
zero source / destination port. Something like:

# perf record -a -e fib:fib_table_lookup -- sleep 5
# perf script --stdio

> 
> > 
> > Is this a bug ? Or am I misinterpreting how 'reroute check' works after
> > initial routing decision ? One would expect routing rules during post-mangle
> > check to not be ignored out of the blue, only because packet mark changed on
> > the packet. Not mentioning both marks and routing rules can be used for
> > separate purposes (e.g. marks for shaping).
> >