Is this possible SYN Proxy bug?

Is this possible SYN Proxy bug?

[İbrahim Ercan]

Hi.
I experienced some errors regarding syn proxy. I observed network
traffic and realized syn proxy doesn't set mss value correctly. Then I
did some tests and here is details of my test.

I set 3 different device. A client, firewall and server. Firewall is
where syn proxy rules located.

Before adding syn proxy rules, I observed mss and wscale values

10.0.0.215.60797 > 10.0.1.213.80: Flags [S], seq 3059817525, win
29200, options [mss 1460,sackOK,TS val 95678003 ecr 0,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60797: Flags [S.], seq 3020500548, ack
3059817526, win 14480, options [mss 1460,sackOK,TS val 12703989 ecr
95678003,nop,wscale 2], length 0

So client sets mss 1460 wscale 7, server sets mss 1460 and wscale 2

Then I added below rules and start tests
iptables -t raw -A PREROUTING -i enp7s0f0 -p tcp -m tcp --syn -j CT --notrack
iptables -A FORWARD -i enp7s0f0 -p tcp -m tcp -m state --state
INVALID,UNTRACKED -j SYNPROXY --sack-perm --timestamp --wscale 2 --mss
1460
iptables -A FORWARD -i enp7s0f0 -p tcp -m tcp -m state --state INVALID -j DROP

With these values here is what I see on external and internal interface

External interface:
10.0.0.215.60800 > 10.0.1.213.80: Flags [S], seq 1609742327, win
29200, options [mss 1460,sackOK,TS val 99453267 ecr 0,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60800: Flags [S.], seq 3294723050, ack
1609742328, win 0, options [mss 1460,sackOK,TS val 5761239 ecr
99453267,nop,wscale 2], length 0

Internal interface:
10.0.0.215.60800 > 10.0.1.213.80: Flags [S], seq 1609742327, win 229,
options [mss 1460,sackOK,TS val 99453267 ecr 5761239,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60800: Flags [S.], seq 1301167703, ack
1609742328, win 14480, options [mss 1460,sackOK,TS val 16479252 ecr
99453267,nop,wscale 2], length 0

Until here there is nothing wrong. Now see what happen when I set
client mss value to 1260 by changing mtu.

External interface
10.0.0.215.60802 > 10.0.1.213.80: Flags [S], seq 36636545, win 25200,
options [mss 1260,sackOK,TS val 99747035 ecr 0,nop,wscale 7], length 0
10.0.1.213.80 > 10.0.0.215.60802: Flags [S.], seq 2342465663, ack
36636546, win 0, options [mss 1260,sackOK,TS val 6054999 ecr
99747035,nop,wscale 2], length 0

Internal interface
10.0.0.215.60802 > 10.0.1.213.80: Flags [S], seq 36636545, win 197,
options [mss 536,sackOK,TS val 99747035 ecr 6054999,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60802: Flags [S.], seq 3600660781, ack
36636546, win 14480, options [mss 1460,sackOK,TS val 16773019 ecr
99747035,nop,wscale 2], length 0

As you can see syn proxy respond to client with same mss value and
open connection to back end with 536. But I suppose, It should send
1460 to client and 1260 to server.

I tried both bridged and router topology with kernel versions 5.0.13
and 4.14.21 and got same results. iptables version is 1.4.21

Regards
--
Ibrahim Ercan

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> Until here there is nothing wrong. Now see what happen when I set
> client mss value to 1260 by changing mtu.
[..]

> Internal interface
> 10.0.0.215.60802 > 10.0.1.213.80: Flags [S], seq 36636545, win 197,
> options [mss 536,sackOK,TS val 99747035 ecr 6054999,nop,wscale 7],
> length 0
> 10.0.1.213.80 > 10.0.0.215.60802: Flags [S.], seq 3600660781, ack
> 36636546, win 14480, options [mss 1460,sackOK,TS val 16773019 ecr
> 99747035,nop,wscale 2], length 0
> 
> As you can see syn proxy respond to client with same mss value and
> open connection to back end with 536. But I suppose, It should send
> 1460 to client and 1260 to server.

Problem is that we do not keep any state.  Syncookes are restricted to 4
mss value:
static __u16 const msstab[] = {
 536,
1300,
1440,   /* 1440, 1452: PPPoE */
1460,
};

So, 1260 forces lowest value supported.

The table was based off a research paper that had mss distribution
tables.  Maybe more recent data is available and if things have changed
we could update the table accordingly.

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Tue, Jun 18, 2019 at 1:40 PM Florian Westphal <fw@strlen.de> wrote:
>
> Problem is that we do not keep any state.  Syncookes are restricted to 4
> mss value:
> static __u16 const msstab[] = {
>  536,
> 1300,
> 1440,   /* 1440, 1452: PPPoE */
> 1460,
> };
>
> So, 1260 forces lowest value supported.
>
> The table was based off a research paper that had mss distribution
> tables.  Maybe more recent data is available and if things have changed
> we could update the table accordingly.

I am confused. So this statement from manual page is just a illusion?
--mss maximum segment size
              Maximum segment size announced to clients. This must
match the backend.

I don't understand why these restriction exist. Why can't we set mss
value same as what client send to us?

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Tue, Jun 18, 2019 at 1:53 PM Andre Paulsberg-Csibi
<Andre.Paulsberg-Csibi@no.ibm.com> wrote:
>
> Maybe a suggestion would be to also have a setting/option where SYNPROXY uses the same MSS as the original packet ?
>
>
As I know, Syn proxy should imitates client and server. To do that, It
should send mss value to client that we set in iptables rule. Same way
it should send mss value to server that what client send already .
These should be default behavior, not an option.

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> On Tue, Jun 18, 2019 at 1:40 PM Florian Westphal <fw@strlen.de> wrote:
> >
> > Problem is that we do not keep any state.  Syncookes are restricted to 4
> > mss value:
> > static __u16 const msstab[] = {
> >  536,
> > 1300,
> > 1440,   /* 1440, 1452: PPPoE */
> > 1460,
> > };
> >
> > So, 1260 forces lowest value supported.
> >
> > The table was based off a research paper that had mss distribution
> > tables.  Maybe more recent data is available and if things have changed
> > we could update the table accordingly.
> 
> I am confused. So this statement from manual page is just a illusion?
> --mss maximum segment size
>               Maximum segment size announced to clients. This must
> match the backend.

?

Your question was about MSS sent to server.

Flow is this:
Client          Synproxy         Server
-> Syn, mss X
    <-Synack,mss M
-> ACK
                     -> Syn, mss Y

M is what you need to configure via --mss switch.

Because Synproxy keeps no state, it can only send
to real server the MSS that was encoded in syncookie (in synack)
packet.  Therefore, X == Y only if the Value from client matches
exactly one for the four values of the mss table, in all other
cases Y is the next lowest available one.  In your case thats 536.

> I don't understand why these restriction exist. Why can't we set mss
> value same as what client send to us?

We only have 2 bits out of the 32Bit Sequence number for MSS. Increasing
mss state table reduces security margin of the cookie.

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Tue, Jun 18, 2019 at 2:59 PM Florian Westphal <fw@strlen.de> wrote:

> >
> > I am confused. So this statement from manual page is just a illusion?
> > --mss maximum segment size
> >               Maximum segment size announced to clients. This must
> > match the backend.
>
> ?
>
> Your question was about MSS sent to server.
>
> Flow is this:
> Client          Synproxy         Server
> -> Syn, mss X
>     <-Synack,mss M
> -> ACK
>                      -> Syn, mss Y
>
> M is what you need to configure via --mss switch.
>
> Because Synproxy keeps no state, it can only send
> to real server the MSS that was encoded in syncookie (in synack)
> packet.  Therefore, X == Y only if the Value from client matches
> exactly one for the four values of the mss table, in all other
> cases Y is the next lowest available one.  In your case thats 536.
>
> > I don't understand why these restriction exist. Why can't we set mss
> > value same as what client send to us?
>
> We only have 2 bits out of the 32Bit Sequence number for MSS. Increasing
> mss state table reduces security margin of the cookie.

My question about both way actually. If you check out my tests, M is
also not correct. Client sends mss 1260 and syn proxy responds 1260
too although I set mss 1460 in iptables.

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> On Tue, Jun 18, 2019 at 2:59 PM Florian Westphal <fw@strlen.de> wrote:
> 
> > >
> > > I am confused. So this statement from manual page is just a illusion?
> > > --mss maximum segment size
> > >               Maximum segment size announced to clients. This must
> > > match the backend.
> >
> > ?
> >
> > Your question was about MSS sent to server.
> >
> > Flow is this:
> > Client          Synproxy         Server
> > -> Syn, mss X
> >     <-Synack,mss M
> > -> ACK
> >                      -> Syn, mss Y
> >
> > M is what you need to configure via --mss switch.
> >
> > Because Synproxy keeps no state, it can only send
> > to real server the MSS that was encoded in syncookie (in synack)
> > packet.  Therefore, X == Y only if the Value from client matches
> > exactly one for the four values of the mss table, in all other
> > cases Y is the next lowest available one.  In your case thats 536.
> >
> > > I don't understand why these restriction exist. Why can't we set mss
> > > value same as what client send to us?
> >
> > We only have 2 bits out of the 32Bit Sequence number for MSS. Increasing
> > mss state table reduces security margin of the cookie.
> 
> My question about both way actually. If you check out my tests, M is
> also not correct. Client sends mss 1260 and syn proxy responds 1260
> too although I set mss 1460 in iptables.

Does this patch fix the problem for you?

diff --git a/net/ipv4/netfilter/ipt_SYNPROXY.c b/net/ipv4/netfilter/ipt_SYNPROXY.c
--- a/net/ipv4/netfilter/ipt_SYNPROXY.c
+++ b/net/ipv4/netfilter/ipt_SYNPROXY.c
@@ -286,6 +286,7 @@ synproxy_tg4(struct sk_buff *skb, const struct xt_action_param *par)
 			opts.options |= XT_SYNPROXY_OPT_ECN;
 
 		opts.options &= info->options;
+		opts.mss = info->mss;
 		if (opts.options & XT_SYNPROXY_OPT_TIMESTAMP)
 			synproxy_init_timestamp_cookie(info, &opts);
 		else
diff --git a/net/ipv6/netfilter/ip6t_SYNPROXY.c b/net/ipv6/netfilter/ip6t_SYNPROXY.c
--- a/net/ipv6/netfilter/ip6t_SYNPROXY.c
+++ b/net/ipv6/netfilter/ip6t_SYNPROXY.c
@@ -300,6 +300,7 @@ synproxy_tg6(struct sk_buff *skb, const struct xt_action_param *par)
 			opts.options |= XT_SYNPROXY_OPT_ECN;
 
 		opts.options &= info->options;
+		opts.mss = info->mss;
 		if (opts.options & XT_SYNPROXY_OPT_TIMESTAMP)
 			synproxy_init_timestamp_cookie(info, &opts);
 		else

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Tue, Jun 18, 2019 at 3:40 PM Florian Westphal <fw@strlen.de> wrote:
>
> Does this patch fix the problem for you?
>
> diff --git a/net/ipv4/netfilter/ipt_SYNPROXY.c b/net/ipv4/netfilter/ipt_SYNPROXY.c
> --- a/net/ipv4/netfilter/ipt_SYNPROXY.c
> +++ b/net/ipv4/netfilter/ipt_SYNPROXY.c
> @@ -286,6 +286,7 @@ synproxy_tg4(struct sk_buff *skb, const struct xt_action_param *par)
>                         opts.options |= XT_SYNPROXY_OPT_ECN;
>
>                 opts.options &= info->options;
> +               opts.mss = info->mss;
>                 if (opts.options & XT_SYNPROXY_OPT_TIMESTAMP)
>                         synproxy_init_timestamp_cookie(info, &opts);
>                 else
> diff --git a/net/ipv6/netfilter/ip6t_SYNPROXY.c b/net/ipv6/netfilter/ip6t_SYNPROXY.c
> --- a/net/ipv6/netfilter/ip6t_SYNPROXY.c
> +++ b/net/ipv6/netfilter/ip6t_SYNPROXY.c
> @@ -300,6 +300,7 @@ synproxy_tg6(struct sk_buff *skb, const struct xt_action_param *par)
>                         opts.options |= XT_SYNPROXY_OPT_ECN;
>
>                 opts.options &= info->options;
> +               opts.mss = info->mss;
>                 if (opts.options & XT_SYNPROXY_OPT_TIMESTAMP)
>                         synproxy_init_timestamp_cookie(info, &opts);
>                 else

I applied this patch and did same test with same setup.
On External interface mss value seems correct. But This time on
internal interface firewall set mss value to 1460 on syn packet rather
than 536.
Here is samples.

External
10.0.0.215.60812 > 10.0.1.213.80: Flags [S], seq 1275328749, win
25200, options [mss 1260,sackOK,TS val 183998290 ecr 0,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60812: Flags [S.], seq 584730658, ack
1275328750, win 0, options [mss 1460,sackOK,TS val 193047 ecr
183998290,nop,wscale 2], length 0

Internal
10.0.0.215.60812 > 10.0.1.213.80: Flags [S], seq 1275328749, win 197,
options [mss 1460,sackOK,TS val 183998290 ecr 193047,nop,wscale 7],
length 0
10.0.1.213.80 > 10.0.0.215.60812: Flags [S.], seq 3022386930, ack
1275328750, win 14480, options [mss 1460,sackOK,TS val 101024266 ecr
183998290,nop,wscale 2], length 0

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

Hi again.
I modified your patch as below and now synproxy send mss values as it
should be. Soom I will test it on real environment.
I also have another question. When I don't provide --wscale option,
both client syn-ack an server syn packets have empty wscale. When I
don't provide --mss option, I realized firewall not set mss value on
client syn-ack, but it sets mss on server syn. Is that what suppose to
happen?

diff -rupN a/net/ipv4/netfilter/ipt_SYNPROXY.c
b/net/ipv4/netfilter/ipt_SYNPROXY.c
--- a/net/ipv4/netfilter/ipt_SYNPROXY.c       2019-06-19
09:51:40.163633231 +0300
+++ b/net/ipv4/netfilter/ipt_SYNPROXY.c      2019-06-20 13:32:18.893025129 +0300
@@ -71,13 +71,13 @@ free_nskb:
 static void
 synproxy_send_client_synack(struct net *net,
                            const struct sk_buff *skb, const struct tcphdr *th,
-                           const struct synproxy_options *opts)
+                           const struct synproxy_options *opts, const
u16 *client_mssinfo)
 {
        struct sk_buff *nskb;
        struct iphdr *iph, *niph;
        struct tcphdr *nth;
        unsigned int tcp_hdr_size;
-       u16 mss = opts->mss;
+       u16 mss = *client_mssinfo;

        iph = ip_hdr(skb);
@@ -266,6 +265,7 @@ synproxy_tg4(struct sk_buff *skb, const
        struct synproxy_net *snet = synproxy_pernet(net);
        struct synproxy_options opts = {};
        struct tcphdr *th, _th;
+        u16 client_mssinfo;

        if (nf_ip_checksum(skb, xt_hooknum(par), par->thoff, IPPROTO_TCP))
                return NF_DROP;
@@ -285,6 +285,8 @@ synproxy_tg4(struct sk_buff *skb, const
                        opts.options |= XT_SYNPROXY_OPT_ECN;

                opts.options &= info->options;
+                client_mssinfo = opts.mss;
+               opts.mss = info->mss;
                if (opts.options & XT_SYNPROXY_OPT_TIMESTAMP)
                        synproxy_init_timestamp_cookie(info, &opts);
                else
@@ -292,7 +294,7 @@ synproxy_tg4(struct sk_buff *skb, const
                                          XT_SYNPROXY_OPT_SACK_PERM |
                                          XT_SYNPROXY_OPT_ECN);

-               synproxy_send_client_synack(net, skb, th, &opts);
+               synproxy_send_client_synack(net, skb, th, &opts,
&client_mssinfo);
                consume_skb(skb);
                return NF_STOLEN;
        } else if (th->ack && !(th->fin || th->rst || th->syn)) {

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> I modified your patch as below and now synproxy send mss values as it
> should be. Soom I will test it on real environment.
> I also have another question. When I don't provide --wscale option,
> both client syn-ack an server syn packets have empty wscale. When I
> don't provide --mss option, I realized firewall not set mss value on
> client syn-ack, but it sets mss on server syn. Is that what suppose to
> happen?

The SYN sent to server should reflect/match the SYN received from
client (mss might be smaller due to msstab encoding).

> diff -rupN a/net/ipv4/netfilter/ipt_SYNPROXY.c
> b/net/ipv4/netfilter/ipt_SYNPROXY.c
> --- a/net/ipv4/netfilter/ipt_SYNPROXY.c       2019-06-19
> 09:51:40.163633231 +0300
> +++ b/net/ipv4/netfilter/ipt_SYNPROXY.c      2019-06-20 13:32:18.893025129 +0300
> @@ -71,13 +71,13 @@ free_nskb:
>  static void
>  synproxy_send_client_synack(struct net *net,
>                             const struct sk_buff *skb, const struct tcphdr *th,
> -                           const struct synproxy_options *opts)
> +                           const struct synproxy_options *opts, const
> u16 *client_mssinfo)
>  {
>         struct sk_buff *nskb;
>         struct iphdr *iph, *niph;
>         struct tcphdr *nth;
>         unsigned int tcp_hdr_size;
> -       u16 mss = opts->mss;
> +       u16 mss = *client_mssinfo;

Yes, something like this is needed, i.e. we need to pass two
mss values -- one from info->mss ("server") that we need to
place in the tcp options sent to client and one containing
the clients mss that we should encode into the cookie.

I think you can pass "u16 client_mssinfo" instead of u16* pointer.

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Fri, Jun 21, 2019 at 2:10 PM Florian Westphal <fw@strlen.de> wrote:
>
> Yes, something like this is needed, i.e. we need to pass two
> mss values -- one from info->mss ("server") that we need to
> place in the tcp options sent to client and one containing
> the clients mss that we should encode into the cookie.
>
> I think you can pass "u16 client_mssinfo" instead of u16* pointer.

Hi Florian.

We tested fixed code on real environment and we are still getting some
errors. We have a customer using syn proxy in front of a point of sale
(POS) application and they reported that about %0.4 of connections are
erroneous.
When I examine traffic from pcap file, I saw connections opens
successfully but somehow something goes wrong after then.
If we deactivates syn proxy, problem goes away. So we are sure it is
caused by syn proxy.
How can I debug syn proxy further? Do you have any suggestion?

Regards.

Re: Is this possible SYN Proxy bug?

[Fernando Fernandez Mancera]

Hi Ibrahim,

On 6/24/19 9:55 AM, İbrahim Ercan wrote:
> On Fri, Jun 21, 2019 at 2:10 PM Florian Westphal <fw@strlen.de> wrote:
>>
>> Yes, something like this is needed, i.e. we need to pass two
>> mss values -- one from info->mss ("server") that we need to
>> place in the tcp options sent to client and one containing
>> the clients mss that we should encode into the cookie.
>>
>> I think you can pass "u16 client_mssinfo" instead of u16* pointer.
> 
> Hi Florian.
> 
> We tested fixed code on real environment and we are still getting some
> errors. We have a customer using syn proxy in front of a point of sale
> (POS) application and they reported that about %0.4 of connections are
> erroneous.
> When I examine traffic from pcap file, I saw connections opens
> successfully but somehow something goes wrong after then.
> If we deactivates syn proxy, problem goes away. So we are sure it is
> caused by syn proxy.
> How can I debug syn proxy further? Do you have any suggestion?
> 
> Regards.
> 

I am working on synproxy right now, I am going to test it in different
environments and debug it. Please let me know if you have any
recommended environment on mind.

Thanks,
Fernando.

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> We tested fixed code on real environment and we are still getting some
> errors.

Can you submit your patch officially?
The MSS fix is needed in any case.

> When I examine traffic from pcap file, I saw connections opens
> successfully but somehow something goes wrong after then.

Do you have an example pcap of a connection stalling?

Re: Is this possible SYN Proxy bug?

[İbrahim Ercan]

On Mon, Jun 24, 2019 at 1:20 PM Florian Westphal <fw@strlen.de> wrote:
>
> İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> > We tested fixed code on real environment and we are still getting some
> > errors.
>
> Can you submit your patch officially?
> The MSS fix is needed in any case.

I submitted. This is my first patch attempt. Sorry for any inconvenience.

>
> > When I examine traffic from pcap file, I saw connections opens
> > successfully but somehow something goes wrong after then.
>
> Do you have an example pcap of a connection stalling?

I will ready and send it to you soon. Can I send a file to this mail
list? Or should I send it directly your personal email?

Re: Is this possible SYN Proxy bug?

[Florian Westphal]

İbrahim Ercan <ibrahim.metu@gmail.com> wrote:
> > > When I examine traffic from pcap file, I saw connections opens
> > > successfully but somehow something goes wrong after then.
> >
> > Do you have an example pcap of a connection stalling?
> 
> I will ready and send it to you soon. Can I send a file to this mail
> list? Or should I send it directly your personal email?

You could pastebin tcpdump text mode, upload a file somewhere, create
a bugzilla ticket on https://bugzilla.netfilter.org/ and attach the pcap
there or send it to my personal email.