From: Pablo Neira Ayuso <pablo@netfilter.org>
To: Jeff Cook <jeff@deserettechnology.com>
Cc: netfilter@vger.kernel.org, kaber@trash.net
Subject: Re: Packets marked by iptables only sent to the correct routing table sometimes
Date: Tue, 30 Oct 2012 20:16:10 +0100 [thread overview]
Message-ID: <20121030191610.GA23663@1984> (raw)
In-Reply-To: <20121030191034.GA21096@1984>
On Tue, Oct 30, 2012 at 08:10:34PM +0100, Pablo Neira Ayuso wrote:
> On Tue, Oct 30, 2012 at 11:21:01AM -0600, Jeff Cook wrote:
> > Hello.
> >
> > I am trying to route packets generated by a specific user out over a
> > VPN. I have this configuration:
> >
> > $ sudo iptables -S -t nat
> > -P PREROUTING ACCEPT
> > -P OUTPUT ACCEPT
> > -P POSTROUTING ACCEPT
> > -A POSTROUTING -o tun0 -j MASQUERADE
> >
> > $ sudo iptables -S -t mangle
> > -P PREROUTING ACCEPT
> > -P INPUT ACCEPT
> > -P FORWARD ACCEPT
> > -P OUTPUT ACCEPT
> > -P POSTROUTING ACCEPT
> > -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
> >
> > $ sudo ip rule show
> > 0: from all lookup local
> > 32765: from all fwmark 0xb lookup 11
> > 32766: from all lookup main
> > 32767: from all lookup default
> >
> > $ sudo ip route show table 11
> > 10.8.0.5 dev tun0 proto kernel scope link src 10.8.0.6
> > 10.8.0.6 dev tun0 scope link
> > 10.8.0.1 via 10.8.0.5 dev tun0
> > 0.0.0.0/1 via 10.8.0.5 dev tun0
> ^^^^^^^^^
>
> 23.1.17.194, this doesn't go through tun0
Sorry, I meant: 23.1.17.194, this goes through tun0.
> 209.68.27.16, this doesn't go through tun0
>
> Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0
>
> Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be
> eth0.
>
> > $ sudo iptables -S -t raw
> > -P PREROUTING ACCEPT
> > -P OUTPUT ACCEPT
> > -A OUTPUT -m owner --uid-owner guy -j TRACE
> > -A OUTPUT -p tcp -m tcp --dport 80 -j TRACE
> >
> > It seems that some sites work fine and use the VPN, but others don't and
> > fall back to the normal interface. This is bad. This is a packet trace
> > that used VPN:
> >
> > Oct 27 00:24:28 agent kernel: [612979.976052] TRACE:
> > raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976105] TRACE:
> > raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976164] TRACE:
> > mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999
> > Oct 27 00:24:28 agent kernel: [612979.976210] TRACE:
> > mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976269] TRACE:
> > nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976320] TRACE:
> > filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976367] TRACE:
> > mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA
> > DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP
> > SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0
> > OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:24:28 agent kernel: [612979.976414] TRACE:
> > nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80
> > SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb
> >
> > and this is one that didn't:
> >
> > Oct 27 00:22:41 agent kernel: [612873.662559] TRACE:
> > raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662609] TRACE:
> > raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662664] TRACE:
> > mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999
> > Oct 27 00:22:41 agent kernel: [612873.662709] TRACE:
> > mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662761] TRACE:
> > nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662808] TRACE:
> > filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16
> > LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80
> > SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> > Oct 27 00:22:41 agent kernel: [612873.662855] TRACE:
> > mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA
> > DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP
> > SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT
> > (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb
> >
> > I have already tried "ip route flush cache", to no avail. I do not know
> > why the first packet goes through the correct routing table, and the
> > second doesn't. Both are marked.
> >
> > Once again, I do not want ALL packets system-wide to go through the VPN,
> > I only want packets from a specific user (UID=999) to go through the
> > VPN. I am testing ipchicken.com and walmart.com via `links`, from the
> > same user, same shell. walmart.com appears to use the VPN; ipchicken.com
> > does not.
> >
> > I have tried iptables -t raw -A OUTPUT -j NOTRACK to circumvent
> > conntrack interference, but this hasn't worked either.
> >
> > Any help appreciated; need this resolved ASAP. If this is something that
> > can't be resolved by volunteers on a mailing list and someone is
> > available as a consultant and can look into this further, would
> > appreciate it; email me privately with rate information and credentials.
> >
> > Thanks
> > Jeff
next prev parent reply other threads:[~2012-10-30 19:16 UTC|newest]
Thread overview: 6+ messages / expand[flat|nested] mbox.gz Atom feed top
2012-10-30 17:21 Packets marked by iptables only sent to the correct routing table sometimes Jeff Cook
2012-10-30 19:10 ` Pablo Neira Ayuso
2012-10-30 19:16 ` Pablo Neira Ayuso [this message]
2012-10-30 23:25 ` Jeff Cook
2012-10-30 23:45 ` Ed W
2012-10-31 0:08 ` Pablo Neira Ayuso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20121030191610.GA23663@1984 \
--to=pablo@netfilter.org \
--cc=jeff@deserettechnology.com \
--cc=kaber@trash.net \
--cc=netfilter@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.