Re: ipsec doesn't route TCP with 4.11 kernel

Re: ipsec doesn't route TCP with 4.11 kernel

[Cong Wang]

(Cc'ing netdev and IPSec maintainers)

On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
> I'm not sure how to describe this.
>
> 4.11rc2 worked, after that, no.
>
> My ipsec tunnel comes up ok. ICMP works. UDP works. But TCP, the
> sender [which is the ipsec client] does not reach the destination.
>
> Its not a routing rule issue (since ICMP/UDP work).
> Its not a traffic selector just selecting TCP (I think) since ipsec
> status shows just a subnet, no protocol.
>
> Using tcpdump:
> # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
> NFLOG --nflog-group 5
> # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
> NFLOG --nflog-group 5
> # tcpdump -s 0 -n -i nflog:5
>
> I see that it thinks it is sending the TCP packet, but the server end
> does not receive.
>
> Does anyone have any suggestion to try?
>
> strongswan is 5.5.1 [on ubuntu 17.04]
> kernel is 4.11.0-041100rc8-generic
>
> My rightsubnet is
> rightsubnet = 192.168.128.0/17,10.0.0.0/8
>
> so no specific protocol selected, the result is:
> CHILD_SA sv{1} established with SPIs c05f1b6c_i 0d58815a_o and TS
> 192.168.130.4/32 === 10.0.0.0/8 192.168.128.0/17
>
> I tried changing charondebug net=3, but i'm not sure how to interpret
> the output:
>
> Apr 25 21:06:34 office charon: 04[NET] received packet: from
> 64.7.137.180[4500] to 172.16.0.8[4500] (80 bytes)
> Apr 25 21:06:34 office charon: 04[ENC] parsed INFORMATIONAL request 4 [ ]
> Apr 25 21:06:34 office charon: 04[ENC] generating INFORMATIONAL response 4 [ ]
> Apr 25 21:06:34 office charon: 04[NET] sending packet: from
> 172.16.0.8[4500] to 64.7.137.180[4500] (80 bytes)

Re: ipsec doesn't route TCP with 4.11 kernel

[Steffen Klassert]

On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> (Cc'ing netdev and IPSec maintainers)
> 
> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
> > I'm not sure how to describe this.
> >
> > 4.11rc2 worked, after that, no.

We had some recent IPsec GRO changes, this could influence TCP.
But these changes were introduced before rc2. If I read this correct,
the regression was introduced between rc2 and rc3, right?

> >
> > My ipsec tunnel comes up ok.

When talking about IPsec, I guess you use ESP, right?

> > ICMP works. UDP works. But TCP, the
> > sender [which is the ipsec client] does not reach the destination.
> >
> > Its not a routing rule issue (since ICMP/UDP work).
> > Its not a traffic selector just selecting TCP (I think) since ipsec
> > status shows just a subnet, no protocol.
> >
> > Using tcpdump:
> > # iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j
> > NFLOG --nflog-group 5
> > # iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j
> > NFLOG --nflog-group 5
> > # tcpdump -s 0 -n -i nflog:5
> >
> > I see that it thinks it is sending the TCP packet, but the server end
> > does not receive.
> >
> > Does anyone have any suggestion to try?

If it is a GRO issue, then it is on the receive side, could you do
tcpdump on the receiving interface to see what you get there?

What shows /proc/net/xfrm_stat?

Can you do 'ip -s x s' to see if the SAs are used?

Do you have INET_ESP_OFFLOAD enabled?

Re: ipsec doesn't route TCP with 4.11 kernel

[Don Bowman]

On 27 April 2017 at 04:42, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
>> (Cc'ing netdev and IPSec maintainers)
>>
>> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:

for 'esp' question, i have ' esp = aes256-sha256-modp1536!' is that
what you mean?
its nat-aware tunnel [from my desktop pc to my office]

root@office:~# ip -s x s
src 172.16.0.8 dst 64.7.137.180
        proto esp spi 0x0d588366(223904614) reqid 1(0x00000001) mode tunnel
        replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha256)
0x046cafdf19c5d78d1c29165d96a0b9fce1c500029d77be0fe956dce1bf80a86a
(256 bits) 128
        enc cbc(aes)
0x79ff2fbc2178eb468de6ff16612f0603b514a1d1d5f375c67222294463ec7c62
(256 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x0, oseq 0x28, bitmap 0x00000000
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 42843(sec), hard 43200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          2986(bytes), 40(packets)
          add 2017-04-27 18:08:12 use 2017-04-27 18:08:12
        stats:
          replay-window 0 replay 0 failed 0
src 64.7.137.180 dst 172.16.0.8
        proto esp spi 0xcd366c03(3442895875) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        auth-trunc hmac(sha256)
0x4158741cc971c49417d60165f19ed02249385c7bba808927d4a9e7b45fb30c5b
(256 bits) 128
        enc cbc(aes)
0x77592c79c964787bca5012214b85172b06deb7b3f06aac02e3934dd9ead67c15
(256 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        anti-replay context: seq 0x27, oseq 0x0, bitmap 0xffffffff
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 42873(sec), hard 43200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          4501(bytes), 38(packets)
          add 2017-04-27 18:08:12 use 2017-04-27 18:08:12
        stats:
          replay-window 0 replay 0 failed 0


>> >
>> > My ipsec tunnel comes up ok.
>
> When talking about IPsec, I guess you use ESP, right?
 ...

>
> If it is a GRO issue, then it is on the receive side, could you do
> tcpdump on the receiving interface to see what you get there?

I'm not sure what you mean the receiving interface, you mean the
outer, the native interface?
listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
18:11:32.061501 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:32.788091 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
isakmp: child_sa  inf2
18:11:32.788354 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
isakmp: child_sa  inf2[IR]
18:11:33.066830 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:35.082839 IP 172.16.0.8.3416 > 64.7.137.180.33638: truncated-udplength 0
18:11:37.807945 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
isakmp: child_sa  inf2
18:11:37.808300 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
isakmp: child_sa  inf2[IR]

is what i see there for the 'curl' command that doesn't complete.

>
> What shows /proc/net/xfrm_stat?

root@office:~# cat /proc/net/xfrm_stat
XfrmInError                     0
XfrmInBufferError               0
XfrmInHdrError                  0
XfrmInNoStates                  0
XfrmInStateProtoError           0
XfrmInStateModeError            0
XfrmInStateSeqError             0
XfrmInStateExpired              0
XfrmInStateMismatch             0
XfrmInStateInvalid              0
XfrmInTmplMismatch              0
XfrmInNoPols                    0
XfrmInPolBlock                  0
XfrmInPolError                  0
XfrmOutError                    0
XfrmOutBundleGenError           0
XfrmOutBundleCheckError         0
XfrmOutNoStates                 0
XfrmOutStateProtoError          0
XfrmOutStateModeError           0
XfrmOutStateSeqError            0
XfrmOutStateExpired             0
XfrmOutPolBlock                 0
XfrmOutPolDead                  0
XfrmOutPolError                 0
XfrmFwdHdrError                 0
XfrmOutStateInvalid             0
XfrmAcquireError                0

>
> Can you do 'ip -s x s' to see if the SAs are used?
>
> Do you have INET_ESP_OFFLOAD enabled?
>

CONFIG_INET_ESP=m
CONFIG_INET_ESP_OFFLOAD=m
CONFIG_INET6_ESP=m
CONFIG_INET6_ESP_OFFLOAD=m
CONFIG_NETFILTER_XT_MATCH_ESP=m
CONFIG_IP_VS_PROTO_AH_ESP=y
CONFIG_IP_VS_PROTO_ESP=y


# lsmod |grep esp
esp4                   20480  2
xfrm_algo              16384  5 xfrm_user,esp4,ah4,af_key,xfrm_ipcomp

Re: ipsec doesn't route TCP with 4.11 kernel

[Steffen Klassert]

On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
> On 27 April 2017 at 04:42, Steffen Klassert <steffen.klassert@secunet.com>
> wrote:
> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> >> (Cc'ing netdev and IPSec maintainers)
> >>
> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
> 
> for 'esp' question, i have ' esp = aes256-sha256-modp1536!' is that what
> you mean?
> its nat-aware tunnel [from my desktop pc to my office]
> 
> root@office:~# ip -s x s
> src 172.16.0.8 dst 64.7.137.180
>         proto esp spi 0x0d588366(223904614) reqid 1(0x00000001) mode tunnel
>         replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
>         auth-trunc hmac(sha256)
> 0x046cafdf19c5d78d1c29165d96a0b9fce1c500029d77be0fe956dce1bf80a86a (256
> bits) 128
>         enc cbc(aes)
> 0x79ff2fbc2178eb468de6ff16612f0603b514a1d1d5f375c67222294463ec7c62 (256
> bits)
>         encap type espinudp sport 4500 dport 4500 addr 0.0.0.0

Ok, this is espinudp. This information was important.

> 
> I'm not sure what you mean the receiving interface, you mean the outer, the
> native interface?
> listening on eno1, link-type EN10MB (Ethernet), capture size 262144 bytes
> 18:11:32.061501 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0
> 18:11:32.788091 IP 64.7.137.180.4500 > 172.16.0.8.4500: NONESP-encap:
> isakmp: child_sa  inf2
> 18:11:32.788354 IP 172.16.0.8.4500 > 64.7.137.180.4500: NONESP-encap:
> isakmp: child_sa  inf2[IR]
> 18:11:33.066830 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0
> 18:11:35.082839 IP 172.16.0.8.3416 > 64.7.137.180.33638:
> truncated-udplength 0

This is not a GRO issue as I thought, the TX side is already broken.

Could you please try the patch below?

Subject: [PATCH] esp4: Fix udpencap for local TCP packets.

Locally generated TCP packets are usually cloned, so we
do skb_cow_data() on this packets. After that we need to
reload the pointer to the esp header. On udpencap this
header has an offset to skb_transport_header, so take this
offset into account.

Fixes: commit cac2661c53f ("esp4: Avoid skb_cow_data whenever possible")
Signed-off-by: Steffen Klassert <steffen.klassert@secunet.com>
---
 net/ipv4/esp4.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index b1e2444..ab71fbb 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -223,6 +223,7 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
 	int extralen;
 	int tailen;
 	__be64 seqno;
+	int esp_offset = 0;
 	__u8 proto = *skb_mac_header(skb);
 
 	/* skb is pure payload to encrypt */
@@ -288,6 +289,8 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
 			break;
 		}
 
+		esp_offset = (unsigned char *)esph - (unsigned char *)uh;
+
 		*skb_mac_header(skb) = IPPROTO_UDP;
 	}
 
@@ -397,7 +400,7 @@ static int esp_output(struct xfrm_state *x, struct sk_buff *skb)
 		goto error;
 	nfrags = err;
 	tail = skb_tail_pointer(trailer);
-	esph = ip_esp_hdr(skb);
+	esph = (struct ip_esp_hdr *)(skb_transport_header(skb) + esp_offset);
 
 skip_cow:
 	esp_output_fill_trailer(tail, tfclen, plen, proto);
-- 
2.7.4

Re: ipsec doesn't route TCP with 4.11 kernel

[Eric Dumazet]

On Fri, 2017-04-28 at 09:13 +0200, Steffen Klassert wrote:
>          encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
> 
> Ok, this is espinudp. This information was important.

> This is not a GRO issue as I thought, the TX side is already broken.
> 
> Could you please try the patch below?
> 
> Subject: [PATCH] esp4: Fix udpencap for local TCP packets.
> 
> Locally generated TCP packets are usually cloned, so we
> do skb_cow_data() on this packets. After that we need to
> reload the pointer to the esp header. On udpencap this
> header has an offset to skb_transport_header, so take this
> offset into account.


It looks like locally generated TCP packets could avoid the
skb_cow_data(), if you were using skb_header_cloned() instead of
skb_cloned()  ?

Re: ipsec doesn't route TCP with 4.11 kernel

[Steffen Klassert]

On Fri, Apr 28, 2017 at 09:46:42AM -0700, Eric Dumazet wrote:
> On Fri, 2017-04-28 at 09:13 +0200, Steffen Klassert wrote:
> >          encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
> > 
> > Ok, this is espinudp. This information was important.
> 
> > This is not a GRO issue as I thought, the TX side is already broken.
> > 
> > Could you please try the patch below?
> > 
> > Subject: [PATCH] esp4: Fix udpencap for local TCP packets.
> > 
> > Locally generated TCP packets are usually cloned, so we
> > do skb_cow_data() on this packets. After that we need to
> > reload the pointer to the esp header. On udpencap this
> > header has an offset to skb_transport_header, so take this
> > offset into account.
> 
> 
> It looks like locally generated TCP packets could avoid the
> skb_cow_data(), if you were using skb_header_cloned() instead of
> skb_cloned()  ?

Yes, should be possible in the codepath where we do crypto
with separate src and dst buffers. Would require some
rearrangements to make sure we don't do inplace crypto
in this case.

Thanks for the hint!

Re: ipsec doesn't route TCP with 4.11 kernel

[Don Bowman]

On 28 April 2017 at 03:13, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
>> On 27 April 2017 at 04:42, Steffen Klassert <steffen.klassert@secunet.com>
>> wrote:
>> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
>> >> (Cc'ing netdev and IPSec maintainers)
>> >>
>> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
>>

<snip>

confirmed, with this patch in place that the tcp functions properly.

Re: ipsec doesn't route TCP with 4.11 kernel

[Steffen Klassert]

On Sat, Apr 29, 2017 at 08:39:34PM -0400, Don Bowman wrote:
> On 28 April 2017 at 03:13, Steffen Klassert
> <steffen.klassert@secunet.com> wrote:
> > On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
> >> On 27 April 2017 at 04:42, Steffen Klassert <steffen.klassert@secunet.com>
> >> wrote:
> >> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> >> >> (Cc'ing netdev and IPSec maintainers)
> >> >>
> >> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
> >>
> 
> <snip>
> 
> confirmed, with this patch in place that the tcp functions properly.

Thanks for testing!

I'll make sure to get this fix into the mainline soon.

Re: ipsec doesn't route TCP with 4.11 kernel

[Don Bowman]

On 3 May 2017 at 04:14, Steffen Klassert <steffen.klassert@secunet.com> wrote:
> On Sat, Apr 29, 2017 at 08:39:34PM -0400, Don Bowman wrote:
>> On 28 April 2017 at 03:13, Steffen Klassert
>> <steffen.klassert@secunet.com> wrote:
>> > On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
>> >> On 27 April 2017 at 04:42, Steffen Klassert <steffen.klassert@secunet.com>
>> >> wrote:
>> >> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
>> >> >> (Cc'ing netdev and IPSec maintainers)
>> >> >>
>> >> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
>> >>
>>
>> <snip>
>>
>> confirmed, with this patch in place that the tcp functions properly.
>
> Thanks for testing!
>
> I'll make sure to get this fix into the mainline soon.

Thanks. Let me know if there is any more assistance I can provide.
I've been running the patch for 2 weeks now on 3 machines.

Re: ipsec doesn't route TCP with 4.11 kernel

[Steffen Klassert]

On Tue, May 16, 2017 at 03:05:40PM -0400, Don Bowman wrote:
> On 3 May 2017 at 04:14, Steffen Klassert <steffen.klassert@secunet.com> wrote:
> > On Sat, Apr 29, 2017 at 08:39:34PM -0400, Don Bowman wrote:
> >> On 28 April 2017 at 03:13, Steffen Klassert
> >> <steffen.klassert@secunet.com> wrote:
> >> > On Thu, Apr 27, 2017 at 06:13:38PM -0400, Don Bowman wrote:
> >> >> On 27 April 2017 at 04:42, Steffen Klassert <steffen.klassert@secunet.com>
> >> >> wrote:
> >> >> > On Wed, Apr 26, 2017 at 10:01:34PM -0700, Cong Wang wrote:
> >> >> >> (Cc'ing netdev and IPSec maintainers)
> >> >> >>
> >> >> >> On Tue, Apr 25, 2017 at 6:08 PM, Don Bowman <db@donbowman.ca> wrote:
> >> >>
> >>
> >> <snip>
> >>
> >> confirmed, with this patch in place that the tcp functions properly.
> >
> > Thanks for testing!
> >
> > I'll make sure to get this fix into the mainline soon.
> 
> Thanks. Let me know if there is any more assistance I can provide.
> I've been running the patch for 2 weeks now on 3 machines.

Thanks for testing!
I plan to push it upstream at the beginning of te next week.