IPSEC: tunnel breakage with out-of-order IPv4 fragments

IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

I believe I have found an issue whereby IPv4 fragments arriving
out-of-order will cause an IPSEC tunnel to stop passing any traffic
which arrived fragmented, citing 'SA-icv-failure'.  Packets which were
not fragmented will validate and pass successfully, even once the
condition has been triggered.  I have decrypted the traffic and have
verified that the traffic is arriving correctly.  It appears as if the
condition persists until all connections are closed.

The issue was originally discovered in RHEL 6.5 (2.6.32-431.11.2.el6)
kernel and verified with Fedora 20 running 3.15.0-rc8-nn on x86_64.

The easiest way I have found to reproduce the issue is to use a kernel
without commit c08751c851b78514f6ec5 (Fix data chunk fragmentation for
MTU values which are not multiple of 4) to generate fragmented SCTP
traffic using multiple single-homed connections.  This can be easily
done by running the following command a few times (in parallel) on the
traffic generator:

    dd if=/dev/zero bs=4096 count=100000 | ncat -d 0.02 --sctp -s
<local_ip> <remote_ip> 60000

On the remote end, setup a listener:

    ncat --sctp -l -k -p 60000 > /dev/null

At this point, you should be able to see audit.log errors to the effect of:

type=MAC_IPSEC_EVENT msg=audit(1404931490.964:872): op=SA-icv-failure
src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38605
type=MAC_IPSEC_EVENT msg=audit(1404931490.964:873): op=SA-icv-failure
src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38607
type=MAC_IPSEC_EVENT msg=audit(1404931490.964:874): op=SA-icv-failure
src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38606

I have packet captures and matching audit logs for those interested.
Any help would be greatly appreciated.

Regards,
Karl

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Thu, Jul 10, 2014 at 10:57 AM, Karl Heiss <kheiss@gmail.com> wrote:
> I believe I have found an issue whereby IPv4 fragments arriving
> out-of-order will cause an IPSEC tunnel to stop passing any traffic
> which arrived fragmented, citing 'SA-icv-failure'.  Packets which were
> not fragmented will validate and pass successfully, even once the
> condition has been triggered.  I have decrypted the traffic and have
> verified that the traffic is arriving correctly.  It appears as if the
> condition persists until all connections are closed.
>
> The issue was originally discovered in RHEL 6.5 (2.6.32-431.11.2.el6)
> kernel and verified with Fedora 20 running 3.15.0-rc8-nn on x86_64.
>
> The easiest way I have found to reproduce the issue is to use a kernel
> without commit c08751c851b78514f6ec5 (Fix data chunk fragmentation for
> MTU values which are not multiple of 4) to generate fragmented SCTP
> traffic using multiple single-homed connections.  This can be easily
> done by running the following command a few times (in parallel) on the
> traffic generator:
>
>     dd if=/dev/zero bs=4096 count=100000 | ncat -d 0.02 --sctp -s
> <local_ip> <remote_ip> 60000
>
> On the remote end, setup a listener:
>
>     ncat --sctp -l -k -p 60000 > /dev/null
>
> At this point, you should be able to see audit.log errors to the effect of:
>
> type=MAC_IPSEC_EVENT msg=audit(1404931490.964:872): op=SA-icv-failure
> src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38605
> type=MAC_IPSEC_EVENT msg=audit(1404931490.964:873): op=SA-icv-failure
> src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38607
> type=MAC_IPSEC_EVENT msg=audit(1404931490.964:874): op=SA-icv-failure
> src=10.240.34.75 dst=10.240.34.85 spi=101378814(0x60aeafe) seqno=38606
>
> I have packet captures and matching audit logs for those interested.
> Any help would be greatly appreciated.
>
> Regards,
> Karl

Forgot to mention that the above recreator assumes that an IPSEC
connection in tunnel mode has been configured between the two systems.
Below is my libreswan configuration:

conn test
  left=10.240.6.37
  pfs=yes
  keylife=60m
  ikev2=yes
  authby=secret
  right=10.240.34.75
  auto=start
  type=tunnel
  ike=aes128-sha1;modp2048
  phase2alg=aes128-sha1

Karl

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Steffen Klassert]

On Thu, Jul 10, 2014 at 10:57:21AM -0400, Karl Heiss wrote:
> I believe I have found an issue whereby IPv4 fragments arriving
> out-of-order will cause an IPSEC tunnel to stop passing any traffic
> which arrived fragmented, citing 'SA-icv-failure'.  

Why do you think the arriving order causes our problems? SA-icv-failure
means that the integrity check failed and this is independent of the
packet order, unless you use 64 bit extended sequence numbers (ESN).
I would guess you don't use ESN because you noticed these problems
already with v2.6.32 and this had no ESN support.

If this happens just with fragmented packets, I'd guess there is a
problem when packets get fragmented after IPsec and reassembled
before IPsec processing.

> Packets which were
> not fragmented will validate and pass successfully, even once the
> condition has been triggered.  I have decrypted the traffic and have
> verified that the traffic is arriving correctly.  It appears as if the
> condition persists until all connections are closed.
> 
> The issue was originally discovered in RHEL 6.5 (2.6.32-431.11.2.el6)
> kernel and verified with Fedora 20 running 3.15.0-rc8-nn on x86_64.
> 
> The easiest way I have found to reproduce the issue is to use a kernel
> without commit c08751c851b78514f6ec5 (Fix data chunk fragmentation for
> MTU values which are not multiple of 4) to generate fragmented SCTP
> traffic using multiple single-homed connections.  

Do you have some odd MTU/PMTU value on this route? If so, you might
get post IPsec fragmented packets without that commit.

I have a lot of testcases where I test PMTU discovery and fragmentation
with IPsec. I've never seen such problems, but I don't use sctp. Is there
any chance to reproduce this with another protocol?

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Fri, Jul 11, 2014 at 7:00 AM, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Thu, Jul 10, 2014 at 10:57:21AM -0400, Karl Heiss wrote:
>> I believe I have found an issue whereby IPv4 fragments arriving
>> out-of-order will cause an IPSEC tunnel to stop passing any traffic
>> which arrived fragmented, citing 'SA-icv-failure'.
>
> Why do you think the arriving order causes our problems? SA-icv-failure
> means that the integrity check failed and this is independent of the
> packet order, unless you use 64 bit extended sequence numbers (ESN).
> I would guess you don't use ESN because you noticed these problems
> already with v2.6.32 and this had no ESN support.

Because using a simple scapy script (from a 3rd host with readily
available information), I can kill an existing IPsec connection which
is passing fragmented traffic 100% of the time.  Once the failure
occurs, every subsequent fragmented packet causes an audit entry
citing SA-icv-failure.

>
> If this happens just with fragmented packets, I'd guess there is a
> problem when packets get fragmented after IPsec and reassembled
> before IPsec processing.
>

If I am reading the tcpdump correctly, the issue is that the packets
are getting fragmented after IPsec processing.

    10.240.34.75.35689 > linux.local.60000: sctp (1) [COOKIE ECHO]
08:23:05.821858 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags
[DF], proto ESP (50), length 104)
    linux.local > 10.240.34.75: ESP(spi=0x0d34ea75,seq=0x44be), length 84
08:23:06.823989 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
proto ESP (50), length 1500)
    10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b5), length 1480
08:23:06.824001 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
[none], proto ESP (50), length 32)
    10.240.34.75 > linux.local: esp
08:23:06.824030 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
proto ESP (50), length 1500)
    10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b6), length 1480
08:23:06.824035 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
[none], proto ESP (50), length 32)
    10.240.34.75 > linux.local: esp
08:23:06.824057 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
proto ESP (50), length 1500)
    10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b7), length 1480
08:23:06.824060 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
[none], proto ESP (50), length 32)

>> Packets which were
>> not fragmented will validate and pass successfully, even once the
>> condition has been triggered.  I have decrypted the traffic and have
>> verified that the traffic is arriving correctly.  It appears as if the
>> condition persists until all connections are closed.
>>
>> The issue was originally discovered in RHEL 6.5 (2.6.32-431.11.2.el6)
>> kernel and verified with Fedora 20 running 3.15.0-rc8-nn on x86_64.
>>
>> The easiest way I have found to reproduce the issue is to use a kernel
>> without commit c08751c851b78514f6ec5 (Fix data chunk fragmentation for
>> MTU values which are not multiple of 4) to generate fragmented SCTP
>> traffic using multiple single-homed connections.
>
> Do you have some odd MTU/PMTU value on this route? If so, you might
> get post IPsec fragmented packets without that commit.
>

The MTU for this path is 1500 all the way through.

Yes, this commit most likely fixes the post IPsec fragmentation, as
SCTP was not previously accounting for the IPsec headers when sizing
chunks.  However, if the traffic worked in this case without
reordering, wouldn't we hope it could handle the reordered case, even
with the post IPsec fragmentation?

> I have a lot of testcases where I test PMTU discovery and fragmentation
> with IPsec. I've never seen such problems, but I don't use sctp. Is there
> any chance to reproduce this with another protocol?
>

I can try to reproduce this with TCP, but I think I would have to
specially craft post IPsec fragments since I don't think the stack
would let me fragment them (DF set, TCP takes IPsec headers into
account).  The un-patched SCTP is what specifically allows this.

Karl

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Steffen Klassert]

On Fri, Jul 11, 2014 at 08:51:14AM -0400, Karl Heiss wrote:
> On Fri, Jul 11, 2014 at 7:00 AM, Steffen Klassert
> <steffen.klassert@secunet.com> wrote:
> 
> >
> > If this happens just with fragmented packets, I'd guess there is a
> > problem when packets get fragmented after IPsec and reassembled
> > before IPsec processing.
> >
> 
> If I am reading the tcpdump correctly, the issue is that the packets
> are getting fragmented after IPsec processing.
> 
>     10.240.34.75.35689 > linux.local.60000: sctp (1) [COOKIE ECHO]
> 08:23:05.821858 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags
> [DF], proto ESP (50), length 104)
>     linux.local > 10.240.34.75: ESP(spi=0x0d34ea75,seq=0x44be), length 84
> 08:23:06.823989 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
> proto ESP (50), length 1500)
>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b5), length 1480
> 08:23:06.824001 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
> [none], proto ESP (50), length 32)
>     10.240.34.75 > linux.local: esp
> 08:23:06.824030 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
> proto ESP (50), length 1500)
>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b6), length 1480
> 08:23:06.824035 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
> [none], proto ESP (50), length 32)
>     10.240.34.75 > linux.local: esp
> 08:23:06.824057 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
> proto ESP (50), length 1500)
>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b7), length 1480
> 08:23:06.824060 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
> [none], proto ESP (50), length 32)

Your tcpdump looks interesting. Is it possible that all your
fragmented packets have the id field set to 'id 0'? This should
be only the case if the DF flag is set on that packet, but this
is apparently not the case here. If all the fragmented packets
have id 0, it is not possible to determine the correct fragment
chain. If only one fragment gets lost, all further packets might
be reassembled wrong.

When looking at the code, is seems that sctp sets the DF flag
on packets as the default. The IPsec encapsulation code copies
the DF bit from the inner header and sets 'id 0' in this case.
A first guess would be that someone removes the DF flag after
the IPsec encapsulation.

Is the DF flag set on your inner sctp packets?

> >
> > Do you have some odd MTU/PMTU value on this route? If so, you might
> > get post IPsec fragmented packets without that commit.
> >
> 
> The MTU for this path is 1500 all the way through.
> 
> Yes, this commit most likely fixes the post IPsec fragmentation, as
> SCTP was not previously accounting for the IPsec headers when sizing
> chunks.  However, if the traffic worked in this case without
> reordering, wouldn't we hope it could handle the reordered case, even
> with the post IPsec fragmentation?

Sure, I just wanted to know whether I look into the right direction :)

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Fri, Jul 11, 2014 at 08:51:14AM -0400, Karl Heiss wrote:
>> On Fri, Jul 11, 2014 at 7:00 AM, Steffen Klassert
>> <steffen.klassert@secunet.com> wrote:
>>
>> >
>> > If this happens just with fragmented packets, I'd guess there is a
>> > problem when packets get fragmented after IPsec and reassembled
>> > before IPsec processing.
>> >
>>
>> If I am reading the tcpdump correctly, the issue is that the packets
>> are getting fragmented after IPsec processing.
>>
>>     10.240.34.75.35689 > linux.local.60000: sctp (1) [COOKIE ECHO]
>> 08:23:05.821858 IP (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags
>> [DF], proto ESP (50), length 104)
>>     linux.local > 10.240.34.75: ESP(spi=0x0d34ea75,seq=0x44be), length 84
>> 08:23:06.823989 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
>> proto ESP (50), length 1500)
>>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b5), length 1480
>> 08:23:06.824001 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
>> [none], proto ESP (50), length 32)
>>     10.240.34.75 > linux.local: esp
>> 08:23:06.824030 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
>> proto ESP (50), length 1500)
>>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b6), length 1480
>> 08:23:06.824035 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
>> [none], proto ESP (50), length 32)
>>     10.240.34.75 > linux.local: esp
>> 08:23:06.824057 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 0, flags [+],
>> proto ESP (50), length 1500)
>>     10.240.34.75 > linux.local: ESP(spi=0xedc4a797,seq=0x89b7), length 1480
>> 08:23:06.824060 IP (tos 0x2,ECT(0), ttl 60, id 0, offset 1480, flags
>> [none], proto ESP (50), length 32)
>
> Your tcpdump looks interesting. Is it possible that all your
> fragmented packets have the id field set to 'id 0'? This should
> be only the case if the DF flag is set on that packet, but this
> is apparently not the case here. If all the fragmented packets
> have id 0, it is not possible to determine the correct fragment
> chain. If only one fragment gets lost, all further packets might
> be reassembled wrong.
>

Yes, all fragments have 'id 0'.

> When looking at the code, is seems that sctp sets the DF flag
> on packets as the default. The IPsec encapsulation code copies
> the DF bit from the inner header and sets 'id 0' in this case.
> A first guess would be that someone removes the DF flag after
> the IPsec encapsulation.
>
> Is the DF flag set on your inner sctp packets?
>

Yes, the inner packets have DF set, but the outer do not.

>> >
>> > Do you have some odd MTU/PMTU value on this route? If so, you might
>> > get post IPsec fragmented packets without that commit.
>> >
>>
>> The MTU for this path is 1500 all the way through.
>>
>> Yes, this commit most likely fixes the post IPsec fragmentation, as
>> SCTP was not previously accounting for the IPsec headers when sizing
>> chunks.  However, if the traffic worked in this case without
>> reordering, wouldn't we hope it could handle the reordered case, even
>> with the post IPsec fragmentation?
>
> Sure, I just wanted to know whether I look into the right direction :)
>

Karl

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Steffen Klassert]

On Mon, Jul 14, 2014 at 07:52:23AM -0400, Karl Heiss wrote:
> On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
> <steffen.klassert@secunet.com> wrote:
> >
> > Your tcpdump looks interesting. Is it possible that all your
> > fragmented packets have the id field set to 'id 0'? This should
> > be only the case if the DF flag is set on that packet, but this
> > is apparently not the case here. If all the fragmented packets
> > have id 0, it is not possible to determine the correct fragment
> > chain. If only one fragment gets lost, all further packets might
> > be reassembled wrong.
> >
> 
> Yes, all fragments have 'id 0'.
> 
> > When looking at the code, is seems that sctp sets the DF flag
> > on packets as the default. The IPsec encapsulation code copies
> > the DF bit from the inner header and sets 'id 0' in this case.
> > A first guess would be that someone removes the DF flag after
> > the IPsec encapsulation.
> >
> > Is the DF flag set on your inner sctp packets?
> >
> 
> Yes, the inner packets have DF set, but the outer do not.

So we need to find where the DF flag disappears.

I tried to reproduce your issue with the current net tree,
but I was not able to do so.

With the plain net tree, all packets have the DF flag set
(sctp and esp), no fragments.

With the net tree and commit c08751c851b78514f6ec5 reverted,
I have some packets with the DF flag and some without. The
packets without the DF flag got fragmented after IPsec
processing. But even in this case, the ESP packet has the
DF flag set whenever the inner sctp packet has it set.
And also, the DF flag is set whenever a packet has 'id 0'.
The fragmented packets never have the 'id 0'.

Can you describe your usecase more precisely? Do you use
any additional tunnel like ipip/gre etc. or packet mangling?

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Tue, Jul 15, 2014 at 5:16 AM, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Mon, Jul 14, 2014 at 07:52:23AM -0400, Karl Heiss wrote:
>> On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
>> <steffen.klassert@secunet.com> wrote:
>> >
>> > Your tcpdump looks interesting. Is it possible that all your
>> > fragmented packets have the id field set to 'id 0'? This should
>> > be only the case if the DF flag is set on that packet, but this
>> > is apparently not the case here. If all the fragmented packets
>> > have id 0, it is not possible to determine the correct fragment
>> > chain. If only one fragment gets lost, all further packets might
>> > be reassembled wrong.
>> >
>>
>> Yes, all fragments have 'id 0'.
>>
>> > When looking at the code, is seems that sctp sets the DF flag
>> > on packets as the default. The IPsec encapsulation code copies
>> > the DF bit from the inner header and sets 'id 0' in this case.
>> > A first guess would be that someone removes the DF flag after
>> > the IPsec encapsulation.
>> >
>> > Is the DF flag set on your inner sctp packets?
>> >
>>
>> Yes, the inner packets have DF set, but the outer do not.
>
> So we need to find where the DF flag disappears.

I feel like we may be focusing on two different things.  I am more
interested in figuring out why the receive side does not handle these
packets gracefully.  I would assume that the missing/reordered
fragments may not get reassembled correctly and would be dropped,
which is OK.  However, it is when this event occurs and then every
subsequent, correctly ordered, fragmented packet is dropped that I am
concerned about.  While the sender may be in a broken state, the
receiver should be consistent with receive behavior, agreed?

>
> I tried to reproduce your issue with the current net tree,
> but I was not able to do so.
>
> With the plain net tree, all packets have the DF flag set
> (sctp and esp), no fragments.
>
> With the net tree and commit c08751c851b78514f6ec5 reverted,
> I have some packets with the DF flag and some without. The
> packets without the DF flag got fragmented after IPsec
> processing. But even in this case, the ESP packet has the
> DF flag set whenever the inner sctp packet has it set.
> And also, the DF flag is set whenever a packet has 'id 0'.
> The fragmented packets never have the 'id 0'.

Fragmented packets never having 'id 0' is probably fixed due to
703133de331a7a7df47f31f (ip: generate unique IP identificator if local
fragmentation is allowed).  Sounds like I need to come up with more
detail on the sending side for better recreation (see below).

>
> Can you describe your usecase more precisely? Do you use
> any additional tunnel like ipip/gre etc. or packet mangling?

I apologize, I did leave out one critical bit of information in that
the sender is based on a RHEL 6.5 kernel with a backported 3.4.75 SCTP
stack.  As for other mangling or anything else, the case is as
straightforward as originally described.  I will try and see if I can
find which combination of commits need to be removed to allow this
case on the sending side.  I didn't think to elaborate on the sending
side as I was solely concentrating on the receive aspect :(

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Steffen Klassert]

On Tue, Jul 15, 2014 at 08:13:01AM -0400, Karl Heiss wrote:
> On Tue, Jul 15, 2014 at 5:16 AM, Steffen Klassert
> <steffen.klassert@secunet.com> wrote:
> > On Mon, Jul 14, 2014 at 07:52:23AM -0400, Karl Heiss wrote:
> >> On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
> >> <steffen.klassert@secunet.com> wrote:
> >> >
> >> > Your tcpdump looks interesting. Is it possible that all your
> >> > fragmented packets have the id field set to 'id 0'? This should
> >> > be only the case if the DF flag is set on that packet, but this
> >> > is apparently not the case here. If all the fragmented packets
> >> > have id 0, it is not possible to determine the correct fragment
> >> > chain. If only one fragment gets lost, all further packets might
> >> > be reassembled wrong.
> >> >
> >>
> >> Yes, all fragments have 'id 0'.
> >>
> >> > When looking at the code, is seems that sctp sets the DF flag
> >> > on packets as the default. The IPsec encapsulation code copies
> >> > the DF bit from the inner header and sets 'id 0' in this case.
> >> > A first guess would be that someone removes the DF flag after
> >> > the IPsec encapsulation.
> >> >
> >> > Is the DF flag set on your inner sctp packets?
> >> >
> >>
> >> Yes, the inner packets have DF set, but the outer do not.
> >
> > So we need to find where the DF flag disappears.
> 
> I feel like we may be focusing on two different things.  I am more
> interested in figuring out why the receive side does not handle these
> packets gracefully.  I would assume that the missing/reordered
> fragments may not get reassembled correctly and would be dropped,
> which is OK.  However, it is when this event occurs and then every
> subsequent, correctly ordered, fragmented packet is dropped that I am
> concerned about.  While the sender may be in a broken state, the
> receiver should be consistent with receive behavior, agreed?

Ugh. No, not at all. The sender causes these problems on the receive
side by using 'id 0' on all fragments. The id field is used to
determine which fragments belong to which packet. The id must
be unique for each fragmented packet. I.e. all fragments of a
given packet must have the same id, fragments of other packets
must have different id values. If all fragmented packets have
the same id, they get reassembled in the order they arrive.
Say the second fragment of packet A gets lost, then the first
fragment of Packet A is reassembled with the second fragment
of packet B and so on. This leads to authentication failures
as you observe it.

> 
> >
> > Can you describe your usecase more precisely? Do you use
> > any additional tunnel like ipip/gre etc. or packet mangling?
> 
> I apologize, I did leave out one critical bit of information in that
> the sender is based on a RHEL 6.5 kernel with a backported 3.4.75 SCTP
> stack.  As for other mangling or anything else, the case is as
> straightforward as originally described.  I will try and see if I can
> find which combination of commits need to be removed to allow this
> case on the sending side.  I didn't think to elaborate on the sending
> side as I was solely concentrating on the receive aspect :(

Please try with unpatched kernels from kernel.org on the sender and
the receiver.

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Wed, Jul 16, 2014 at 6:59 AM, Steffen Klassert
<steffen.klassert@secunet.com> wrote:
> On Tue, Jul 15, 2014 at 08:13:01AM -0400, Karl Heiss wrote:
>> On Tue, Jul 15, 2014 at 5:16 AM, Steffen Klassert
>> <steffen.klassert@secunet.com> wrote:
>> > On Mon, Jul 14, 2014 at 07:52:23AM -0400, Karl Heiss wrote:
>> >> On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
>> >> <steffen.klassert@secunet.com> wrote:
>> >> >
>> >> > Your tcpdump looks interesting. Is it possible that all your
>> >> > fragmented packets have the id field set to 'id 0'? This should
>> >> > be only the case if the DF flag is set on that packet, but this
>> >> > is apparently not the case here. If all the fragmented packets
>> >> > have id 0, it is not possible to determine the correct fragment
>> >> > chain. If only one fragment gets lost, all further packets might
>> >> > be reassembled wrong.
>> >> >
>> >>
>> >> Yes, all fragments have 'id 0'.
>> >>
>> >> > When looking at the code, is seems that sctp sets the DF flag
>> >> > on packets as the default. The IPsec encapsulation code copies
>> >> > the DF bit from the inner header and sets 'id 0' in this case.
>> >> > A first guess would be that someone removes the DF flag after
>> >> > the IPsec encapsulation.
>> >> >
>> >> > Is the DF flag set on your inner sctp packets?
>> >> >
>> >>
>> >> Yes, the inner packets have DF set, but the outer do not.
>> >
>> > So we need to find where the DF flag disappears.
>>
>> I feel like we may be focusing on two different things.  I am more
>> interested in figuring out why the receive side does not handle these
>> packets gracefully.  I would assume that the missing/reordered
>> fragments may not get reassembled correctly and would be dropped,
>> which is OK.  However, it is when this event occurs and then every
>> subsequent, correctly ordered, fragmented packet is dropped that I am
>> concerned about.  While the sender may be in a broken state, the
>> receiver should be consistent with receive behavior, agreed?
>
> Ugh. No, not at all. The sender causes these problems on the receive
> side by using 'id 0' on all fragments. The id field is used to
> determine which fragments belong to which packet. The id must
> be unique for each fragmented packet. I.e. all fragments of a
> given packet must have the same id, fragments of other packets
> must have different id values. If all fragmented packets have
> the same id, they get reassembled in the order they arrive.
> Say the second fragment of packet A gets lost, then the first
> fragment of Packet A is reassembled with the second fragment
> of packet B and so on. This leads to authentication failures
> as you observe it.

If this is the case, is that not a security concern?  Anyone can spoof
fragments with just the source and destination IPs, id of 0, the SPI,
and cause any subsequent fragments to be invalidated, regardless of
order.  You say that fragments get reassembled in the order that they
arrive, but the code says otherwise, since it pays attention to the MF
and offset values.  I 100% agree that the receiver cannot possibly
differentiate between fragments when all of them have 'id 0', but it
should be able to recover once the reordering event has passed due to
the MF and offset values.  Should the receive side not be smart enough
to at least drop packets that do not authenticate due to reordering
but subsequent packets which are ordered correctly pass through
cleanly?

>
>>
>> >
>> > Can you describe your usecase more precisely? Do you use
>> > any additional tunnel like ipip/gre etc. or packet mangling?
>>
>> I apologize, I did leave out one critical bit of information in that
>> the sender is based on a RHEL 6.5 kernel with a backported 3.4.75 SCTP
>> stack.  As for other mangling or anything else, the case is as
>> straightforward as originally described.  I will try and see if I can
>> find which combination of commits need to be removed to allow this
>> case on the sending side.  I didn't think to elaborate on the sending
>> side as I was solely concentrating on the receive aspect :(
>
> Please try with unpatched kernels from kernel.org on the sender and
> the receiver.

Re: IPSEC: tunnel breakage with out-of-order IPv4 fragments

[Karl Heiss]

On Wed, Jul 16, 2014 at 7:49 AM, Karl Heiss <kheiss@gmail.com> wrote:
> On Wed, Jul 16, 2014 at 6:59 AM, Steffen Klassert
> <steffen.klassert@secunet.com> wrote:
>> On Tue, Jul 15, 2014 at 08:13:01AM -0400, Karl Heiss wrote:
>>> On Tue, Jul 15, 2014 at 5:16 AM, Steffen Klassert
>>> <steffen.klassert@secunet.com> wrote:
>>> > On Mon, Jul 14, 2014 at 07:52:23AM -0400, Karl Heiss wrote:
>>> >> On Mon, Jul 14, 2014 at 5:33 AM, Steffen Klassert
>>> >> <steffen.klassert@secunet.com> wrote:
>>> >> >
>>> >> > Your tcpdump looks interesting. Is it possible that all your
>>> >> > fragmented packets have the id field set to 'id 0'? This should
>>> >> > be only the case if the DF flag is set on that packet, but this
>>> >> > is apparently not the case here. If all the fragmented packets
>>> >> > have id 0, it is not possible to determine the correct fragment
>>> >> > chain. If only one fragment gets lost, all further packets might
>>> >> > be reassembled wrong.
>>> >> >
>>> >>
>>> >> Yes, all fragments have 'id 0'.
>>> >>
>>> >> > When looking at the code, is seems that sctp sets the DF flag
>>> >> > on packets as the default. The IPsec encapsulation code copies
>>> >> > the DF bit from the inner header and sets 'id 0' in this case.
>>> >> > A first guess would be that someone removes the DF flag after
>>> >> > the IPsec encapsulation.
>>> >> >
>>> >> > Is the DF flag set on your inner sctp packets?
>>> >> >
>>> >>
>>> >> Yes, the inner packets have DF set, but the outer do not.
>>> >
>>> > So we need to find where the DF flag disappears.
>>>
>>> I feel like we may be focusing on two different things.  I am more
>>> interested in figuring out why the receive side does not handle these
>>> packets gracefully.  I would assume that the missing/reordered
>>> fragments may not get reassembled correctly and would be dropped,
>>> which is OK.  However, it is when this event occurs and then every
>>> subsequent, correctly ordered, fragmented packet is dropped that I am
>>> concerned about.  While the sender may be in a broken state, the
>>> receiver should be consistent with receive behavior, agreed?
>>
>> Ugh. No, not at all. The sender causes these problems on the receive
>> side by using 'id 0' on all fragments. The id field is used to
>> determine which fragments belong to which packet. The id must
>> be unique for each fragmented packet. I.e. all fragments of a
>> given packet must have the same id, fragments of other packets
>> must have different id values. If all fragmented packets have
>> the same id, they get reassembled in the order they arrive.
>> Say the second fragment of packet A gets lost, then the first
>> fragment of Packet A is reassembled with the second fragment
>> of packet B and so on. This leads to authentication failures
>> as you observe it.
>
> If this is the case, is that not a security concern?  Anyone can spoof
> fragments with just the source and destination IPs, id of 0, the SPI,
> and cause any subsequent fragments to be invalidated, regardless of
> order.  You say that fragments get reassembled in the order that they
> arrive, but the code says otherwise, since it pays attention to the MF
> and offset values.  I 100% agree that the receiver cannot possibly
> differentiate between fragments when all of them have 'id 0', but it
> should be able to recover once the reordering event has passed due to
> the MF and offset values.  Should the receive side not be smart enough
> to at least drop packets that do not authenticate due to reordering
> but subsequent packets which are ordered correctly pass through
> cleanly?
>

Disregard my earlier statements.  I realized that even with the MF and
offset, there is still no way to be sure that the ordering is correct.

>>
>>>
>>> >
>>> > Can you describe your usecase more precisely? Do you use
>>> > any additional tunnel like ipip/gre etc. or packet mangling?
>>>
>>> I apologize, I did leave out one critical bit of information in that
>>> the sender is based on a RHEL 6.5 kernel with a backported 3.4.75 SCTP
>>> stack.  As for other mangling or anything else, the case is as
>>> straightforward as originally described.  I will try and see if I can
>>> find which combination of commits need to be removed to allow this
>>> case on the sending side.  I didn't think to elaborate on the sending
>>> side as I was solely concentrating on the receive aspect :(
>>
>> Please try with unpatched kernels from kernel.org on the sender and
>> the receiver.