nft tproxy failed to redirect on one system

nft tproxy failed to redirect on one system

[Carl Lei]

Hi all,

I have two systems with very similar config, but tproxy failed to
redirect on one of them.  I have a program listening on *:1081, and
want to redirect daddr 198.18.0.0/16 & tcp to this program; I set in
the route table like

local 198.18.0.0/16 dev lo proto static scope host

without policy selectors in `ip rule`; and my (incomplete) nftables
config is like:

table inet filter {

    chain input {
        type filter hook input priority filter; policy drop;
        ct state vmap {
            related : accept, established : accept,
            invalid : drop,
            new : jump input-new,
        }
    }

    chain input-new {
        iifname vmap {
            vbr0 : jump input-new-isolated,
            lo : accept,
        }
        icmp type echo-request accept
    }

    chain input-new-isolated {
        meta mark 42 counter accept
        fib daddr . iif type != { local, broadcast, multicast } drop
        icmp type echo-request accept
        icmpv6 type { echo-request, } accept
        reject
    }

    chain mangle {
        type filter hook prerouting priority mangle; policy accept;
        ip daddr 198.18.0.0/16 meta l4proto tcp tproxy ip to :1081 meta
    mark set 42 accept }
}

Sorry for being incomplete, but I added nftrace before these rules and
saw packets went through the same chain of rules, first hitting tproxy
in mangle, then meta mark 42 counter accept in input-new-isolated.  But
on one system it works for local programs AND network-received packets,
on another system it works only for local programs.  On the bad system
the packets instead gets directed to whatever program originally
listening on the original port, or rejected; e.g. I have an nginx
listening on 0.0.0.0:80 but no programs on 443, then curl http in a vm
connected to vbr0 goes to my nginx, and curl https gets rejected.  I
expect them to go to that program listening on 1081.

What should I check next?

Re: nft tproxy failed to redirect on one system

[Carl Lei]

Btw: sent to wrong address, re-sent to list...

On Fri, 11 Aug 2023 12:00:43 +0800
Carl Lei <me@xecycle.info> wrote:

> Sorry for being incomplete, but I added nftrace before these rules and
> saw packets went through the same chain of rules, first hitting tproxy
> in mangle, then meta mark 42 counter accept in input-new-isolated.
> But on one system it works for local programs AND network-received
> packets, on another system it works only for local programs.  On the
> bad system the packets instead gets directed to whatever program
> originally listening on the original port, or rejected; e.g. I have
> an nginx listening on 0.0.0.0:80 but no programs on 443, then curl
> http in a vm connected to vbr0 goes to my nginx, and curl https gets
> rejected.  I expect them to go to that program listening on 1081.  

Looked closer at the trace, I found that on the bad system, when the
packet goes back to input rules, its trace id changed; on the good
system it does not change.  Perhaps this explains it, but I don't know
why it was changed?

Thank you.

Re: nft tproxy failed to redirect on one system

[Pablo Neira Ayuso]

On Mon, Aug 21, 2023 at 03:48:07PM +0800, Carl Lei wrote:
> Btw: sent to wrong address, re-sent to list...
> 
> On Fri, 11 Aug 2023 12:00:43 +0800
> Carl Lei <me@xecycle.info> wrote:
> 
> > Sorry for being incomplete, but I added nftrace before these rules and
> > saw packets went through the same chain of rules, first hitting tproxy
> > in mangle, then meta mark 42 counter accept in input-new-isolated.
> > But on one system it works for local programs AND network-received
> > packets, on another system it works only for local programs.  On the
> > bad system the packets instead gets directed to whatever program
> > originally listening on the original port, or rejected; e.g. I have
> > an nginx listening on 0.0.0.0:80 but no programs on 443, then curl
> > http in a vm connected to vbr0 goes to my nginx, and curl https gets
> > rejected.  I expect them to go to that program listening on 1081.  
> 
> Looked closer at the trace, I found that on the bad system, when the
> packet goes back to input rules, its trace id changed; on the good
> system it does not change.  Perhaps this explains it, but I don't know
> why it was changed?

Could you provide more detailed information on your setup? You refer
to a system where this works fine and another where this does not, but
you do not specify kernel and userspace versions?

Some simple script with 'ip netns' as a reproducer might also help
that might help people jump in a provide feedback.

Thanks.

Re: nft tproxy failed to redirect on one system

[Carl Lei]

On Tue, 22 Aug 2023 12:05:01 +0200
Pablo Neira Ayuso <pablo@netfilter.org> wrote:

> On Mon, Aug 21, 2023 at 03:48:07PM +0800, Carl Lei wrote:
> > Btw: sent to wrong address, re-sent to list...
> > 
> > On Fri, 11 Aug 2023 12:00:43 +0800
> > Carl Lei <me@xecycle.info> wrote:
> > 
> > > Sorry for being incomplete, but I added nftrace before these
> > > rules and saw packets went through the same chain of rules, first
> > > hitting tproxy in mangle, then meta mark 42 counter accept in
> > > input-new-isolated. But on one system it works for local programs
> > > AND network-received packets, on another system it works only for
> > > local programs.  On the bad system the packets instead gets
> > > directed to whatever program originally listening on the original
> > > port, or rejected; e.g. I have an nginx listening on 0.0.0.0:80
> > > but no programs on 443, then curl http in a vm connected to vbr0
> > > goes to my nginx, and curl https gets rejected.  I expect them to
> > > go to that program listening on 1081.  
> > 
> > Looked closer at the trace, I found that on the bad system, when the
> > packet goes back to input rules, its trace id changed; on the good
> > system it does not change.  Perhaps this explains it, but I don't
> > know why it was changed?
> 
> Could you provide more detailed information on your setup? You refer
> to a system where this works fine and another where this does not, but
> you do not specify kernel and userspace versions?

Sorry I forgot that in my first email, indeed I replied to myself
several times and already provided versions, but I did not realize I
sent to a wrong address.  My used versions are (copying from my
previous mails):

"they are both running Arch Linux, the good system running kernel
6.4.8-arch1-1, the bad system I tried both 6.4.4 and 6.4.9; nftables
version both 1:1.0.7-2; the proxy program on 1081 is also identical,
running identical config."

> Some simple script with 'ip netns' as a reproducer might also help
> that might help people jump in a provide feedback.

I will try ASAP to make a reproducer.

Thank you.

Re: nft tproxy failed to redirect on one system

[Carl Lei]

On Tue, 22 Aug 2023 18:16:31 +0800
Carl Lei <me@xecycle.info> wrote:

> On Tue, 22 Aug 2023 12:05:01 +0200
> Pablo Neira Ayuso <pablo@netfilter.org> wrote:
> 
> > On Mon, Aug 21, 2023 at 03:48:07PM +0800, Carl Lei wrote:
> > > Btw: sent to wrong address, re-sent to list...
> > > 
> > > On Fri, 11 Aug 2023 12:00:43 +0800
> > > Carl Lei <me@xecycle.info> wrote:
> > > 
> > > > Sorry for being incomplete, but I added nftrace before these
> > > > rules and saw packets went through the same chain of rules,
> > > > first hitting tproxy in mangle, then meta mark 42 counter
> > > > accept in input-new-isolated. But on one system it works for
> > > > local programs AND network-received packets, on another system
> > > > it works only for local programs.  On the bad system the
> > > > packets instead gets directed to whatever program originally
> > > > listening on the original port, or rejected; e.g. I have an
> > > > nginx listening on 0.0.0.0:80 but no programs on 443, then curl
> > > > http in a vm connected to vbr0 goes to my nginx, and curl https
> > > > gets rejected.  I expect them to go to that program listening
> > > > on 1081.  
> > > 
> > > Looked closer at the trace, I found that on the bad system, when
> > > the packet goes back to input rules, its trace id changed; on the
> > > good system it does not change.  Perhaps this explains it, but I
> > > don't know why it was changed?
> > 
> > Could you provide more detailed information on your setup? You refer
> > to a system where this works fine and another where this does not,
> > but you do not specify kernel and userspace versions?
> 
> Sorry I forgot that in my first email, indeed I replied to myself
> several times and already provided versions, but I did not realize I
> sent to a wrong address.  My used versions are (copying from my
> previous mails):
> 
> "they are both running Arch Linux, the good system running kernel
> 6.4.8-arch1-1, the bad system I tried both 6.4.4 and 6.4.9; nftables
> version both 1:1.0.7-2; the proxy program on 1081 is also identical,
> running identical config."
> 
> > Some simple script with 'ip netns' as a reproducer might also help
> > that might help people jump in a provide feedback.
> 
> I will try ASAP to make a reproducer.

Sorry for the delay, I finally figured out what is going on.

When I do `modprobe br_netfilter`, which automatically sets sysctl
net/bridge/bridge-nf-call-iptables=1, the rule breaks --- the packet
would change trace id when it comes back to INPUT chains.  After I set
that sysctl to 0, the rule works again.

And yes, the packet is coming from a container, having a veth connected
to a bridge on the host.