From: Pablo Neira Ayuso <pablo@netfilter.org>
To: Jeff Cook <jeff@deserettechnology.com>
Cc: netfilter@vger.kernel.org, kaber@trash.net
Subject: Re: Packets marked by iptables only sent to the correct routing table sometimes
Date: Tue, 30 Oct 2012 20:16:10 +0100 [thread overview]
Message-ID: <20121030191610.GA23663@1984> (raw)
In-Reply-To: <20121030191034.GA21096@1984>
On Tue, Oct 30, 2012 at 08:10:34PM +0100, Pablo Neira Ayuso wrote:
> On Tue, Oct 30, 2012 at 11:21:01AM -0600, Jeff Cook wrote:
> > Hello.
> >
> > I am trying to route packets generated by a specific user out over a
> > VPN. I have this configuration:
> >
> > $ sudo iptables -S -t nat
> > -P PREROUTING ACCEPT
> > -P OUTPUT ACCEPT
> > -P POSTROUTING ACCEPT
> > -A POSTROUTING -o tun0 -j MASQUERADE
> >
> > $ sudo iptables -S -t mangle
> > -P PREROUTING ACCEPT
> > -P INPUT ACCEPT
> > -P FORWARD ACCEPT
> > -P OUTPUT ACCEPT
> > -P POSTROUTING ACCEPT
> > -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
> >
> > $ sudo ip rule show
> > 0: from all lookup local
> > 32765: from all fwmark 0xb lookup 11
> > 32766: from all lookup main
> > 32767: from all lookup default
> >
> > $ sudo ip route show table 11
> > 10.8.0.5 dev tun0 proto kernel scope link src 10.8.0.6
> > 10.8.0.6 dev tun0 scope link
> > 10.8.0.1 via 10.8.0.5 dev tun0
> > 0.0.0.0/1 via 10.8.0.5 dev tun0
> ^^^^^^^^^
>
> 23.1.17.194, this doesn't go through tun0
Sorry, I meant: 23.1.17.194, this goes through tun0.
> 209.68.27.16, this doesn't go through tun0
>
> Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0
>
> Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be
> eth0.
>
> > $ sudo iptables -S -t raw
> > -P PREROUTING ACCEPT
> > -P OUTPUT ACCEPT
> > -A OUTPUT -m owner --uid-owner guy -j TRACE
> > -A OUTPUT -p tcp -m tcp --dport 80 -j TRACE
> >
> > It seems that some sites work fine and use the VPN, but others don't and
> > fall back to the normal interface. This is bad. This is a packet trace
> > that used VPN:
> >
> > Oct 27 00:24:28 agent kernel: [612979.976052] TRACE:
> > raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976105] TRACE:
> > raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976164] TRACE:
> > mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976210] TRACE:
> > mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976269] TRACE:
> > nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976320] TRACE:
> > filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976367] TRACE:
> > mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA
> > DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP
> > SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0
> > OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976414] TRACE:
> > nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> >
> > and this is one that didn't:
> >
> > Oct 27 00:22:41 agent kernel: [612873.662559] TRACE:
> > raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662609] TRACE:
> > raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662664] TRACE:
> > mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662709] TRACE:
> > mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662761] TRACE:
> > nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662808] TRACE:
> > filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662855] TRACE:
> > mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA
> > DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP
> > SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> >
> > I have already tried "ip route flush cache", to no avail. I do not know
> > why the first packet goes through the correct routing table, and the
> > second doesn't. Both are marked.
> >
> > Once again, I do not want ALL packets system-wide to go through the VPN,
> > I only want packets from a specific user (UID=999) to go through the
> > VPN. I am testing ipchicken.com and walmart.com via `links`, from the
> > same user, same shell. walmart.com appears to use the VPN; ipchicken.com
> > does not.
> >
> > I have tried iptables -t raw -A OUTPUT -j NOTRACK to circumvent
> > conntrack interference, but this hasn't worked either.
> >
> > Any help appreciated; need this resolved ASAP. If this is something that
> > can't be resolved by volunteers on a mailing list and someone is
> > available as a consultant and can look into this further, would
> > appreciate it; email me privately with rate information and credentials.
> >
> > Thanks
> > Jeff
next prev parent reply other threads:[~2012-10-30 19:16 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-30 17:21 Packets marked by iptables only sent to the correct routing table sometimes Jeff Cook
2012-10-30 19:10 ` Pablo Neira Ayuso
2012-10-30 19:16 ` Pablo Neira Ayuso [this message]
2012-10-30 23:25 ` Jeff Cook
2012-10-30 23:45 ` Ed W
2012-10-31 0:08 ` Pablo Neira Ayuso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121030191610.GA23663@1984 \
--to=pablo@netfilter.org \
--cc=jeff@deserettechnology.com \
--cc=kaber@trash.net \
--cc=netfilter@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).