kernel >= 4.0: crashes when using traceroute6 with isatap

kernel >= 4.0: crashes when using traceroute6 with isatap

[Cong Wang]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

(Please always Cc netdev for networking bugs.)


---------- Forwarded message ----------
From: Wolfgang Walter <linux@stwm.de>
Date: Fri, May 1, 2015 at 5:16 PM
Subject: kernel >= 4.0: crashes when using traceroute6 with isatap
To: linux-kernel@vger.kernel.org
Cc: Hannes Frederic Sowa <hannes@stressinduktion.org>


Hello,

kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
isatap-tunnel.

I took an image of the message I got. It is not complete as my vt has not
enough lines.

3.19.3 works fine.

Regards,
--
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

[-- Attachment #2: crash.jpeg --]
[-- Type: image/jpeg, Size: 51041 bytes --]

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Cong Wang]

(Cc'ing netdev.)

On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
>> Hello,
>>
>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
>> isatap-tunnel.
>
> I did some further tests. To trigger the crash you need
>
> * isatap-tunnel (probably any sit-tunnel will do it)
> * raw-socket
> * udp
>
> Using icmpv6 or tcp i.e. does not trigger it.
>

Do you have a script to reproduce it?


Thanks for the bug report!

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Eric Dumazet]

On Wed, 2015-05-06 at 11:15 -0700, Cong Wang wrote:
> (Cc'ing netdev.)
> 
> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> > Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> >> Hello,
> >>
> >> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
> >> isatap-tunnel.
> >
> > I did some further tests. To trigger the crash you need
> >
> > * isatap-tunnel (probably any sit-tunnel will do it)
> > * raw-socket
> > * udp
> >
> > Using icmpv6 or tcp i.e. does not trigger it.
> >
> 
> Do you have a script to reproduce it?
> 
> 
> Thanks for the bug report!
> --

Please Wolfgang try to revert 32dce968dd987adfb0c00946d78dad9154f64759
("ipv6: Allow for partial checksums on non-ufo packets")

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Wolfgang Walter]

Am Mittwoch, 6. Mai 2015, 12:10:00 schrieb Eric Dumazet:
> On Wed, 2015-05-06 at 11:15 -0700, Cong Wang wrote:
> > (Cc'ing netdev.)
> > 
> > On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> > > Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> > >> Hello,
> > >> 
> > >> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with
> > >> an
> > >> isatap-tunnel.
> > > 
> > > I did some further tests. To trigger the crash you need
> > > 
> > > * isatap-tunnel (probably any sit-tunnel will do it)
> > > * raw-socket
> > > * udp
> > > 
> > > Using icmpv6 or tcp i.e. does not trigger it.
> > 
> > Do you have a script to reproduce it?
> > 
> > 
> > Thanks for the bug report!
> > --
> 
> Please Wolfgang try to revert 32dce968dd987adfb0c00946d78dad9154f64759
> ("ipv6: Allow for partial checksums on non-ufo packets")

Indeed, that fixes the problem.

Regards
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Eric Dumazet]

On Thu, 2015-05-07 at 00:04 +0200, Wolfgang Walter wrote:
> Am Mittwoch, 6. Mai 2015, 12:10:00 schrieb Eric Dumazet:
> > On Wed, 2015-05-06 at 11:15 -0700, Cong Wang wrote:
> > > (Cc'ing netdev.)
> > > 
> > > On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> > > > Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> > > >> Hello,
> > > >> 
> > > >> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with
> > > >> an
> > > >> isatap-tunnel.
> > > > 
> > > > I did some further tests. To trigger the crash you need
> > > > 
> > > > * isatap-tunnel (probably any sit-tunnel will do it)
> > > > * raw-socket
> > > > * udp
> > > > 
> > > > Using icmpv6 or tcp i.e. does not trigger it.
> > > 
> > > Do you have a script to reproduce it?
> > > 
> > > 
> > > Thanks for the bug report!
> > > --
> > 
> > Please Wolfgang try to revert 32dce968dd987adfb0c00946d78dad9154f64759
> > ("ipv6: Allow for partial checksums on non-ufo packets")
> 
> Indeed, that fixes the problem.

Yes, setting skb->csum to 0 is clearly wrong for CHECKSUM_PARTIAL

Would you try :

diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 7fde1f265c90e90f16291e6c861b6e242111c25b..694ae630e1ca67e25ab1e5f6dd0b3597db3669b0 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1416,6 +1416,10 @@ alloc_new_skb:
 			data += fragheaderlen;
 			skb->transport_header = (skb->network_header +
 						 fragheaderlen);
+			if (csummode == CHECKSUM_PARTIAL) {
+				skb->csum_start = skb_transport_header(skb) - skb->head;
+				skb->csum_offset = offsetof(struct udphdr, check);
+			}
 			if (fraggap) {
 				skb->csum = skb_copy_and_csum_bits(
 					skb_prev, maxfraglen,

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Wolfgang Walter]

Am Mittwoch, 6. Mai 2015, 15:42:38 schrieben Sie:
> On Thu, 2015-05-07 at 00:04 +0200, Wolfgang Walter wrote:
> > Am Mittwoch, 6. Mai 2015, 12:10:00 schrieb Eric Dumazet:
> > > On Wed, 2015-05-06 at 11:15 -0700, Cong Wang wrote:
> > > > (Cc'ing netdev.)
> > > > 
> > > > On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> > > > > Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> > > > >> Hello,
> > > > >> 
> > > > >> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6
> > > > >> with
> > > > >> an
> > > > >> isatap-tunnel.
> > > > > 
> > > > > I did some further tests. To trigger the crash you need
> > > > > 
> > > > > * isatap-tunnel (probably any sit-tunnel will do it)
> > > > > * raw-socket
> > > > > * udp
> > > > > 
> > > > > Using icmpv6 or tcp i.e. does not trigger it.
> > > > 
> > > > Do you have a script to reproduce it?
> > > > 
> > > > 
> > > > Thanks for the bug report!
> > > > --
> > > 
> > > Please Wolfgang try to revert 32dce968dd987adfb0c00946d78dad9154f64759
> > > ("ipv6: Allow for partial checksums on non-ufo packets")
> > 
> > Indeed, that fixes the problem.
> 
> Yes, setting skb->csum to 0 is clearly wrong for CHECKSUM_PARTIAL
> 
> Would you try :
> 
> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
> index
> 7fde1f265c90e90f16291e6c861b6e242111c25b..694ae630e1ca67e25ab1e5f6dd0b3597d
> b3669b0 100644 --- a/net/ipv6/ip6_output.c
> +++ b/net/ipv6/ip6_output.c
> @@ -1416,6 +1416,10 @@ alloc_new_skb:
>  			data += fragheaderlen;
>  			skb->transport_header = (skb->network_header +
>  						 fragheaderlen);
> +			if (csummode == CHECKSUM_PARTIAL) {
> +				skb->csum_start = skb_transport_header(skb) - skb->head;
> +				skb->csum_offset = offsetof(struct udphdr, check);
> +			}
>  			if (fraggap) {
>  				skb->csum = skb_copy_and_csum_bits(
>  					skb_prev, maxfraglen,

This patch fixes 4.0.1 (and 4.0.2).

Thanks Eric,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Vlad Yasevich]

On 05/06/2015 06:42 PM, Eric Dumazet wrote:
> On Thu, 2015-05-07 at 00:04 +0200, Wolfgang Walter wrote:
>> Am Mittwoch, 6. Mai 2015, 12:10:00 schrieb Eric Dumazet:
>>> On Wed, 2015-05-06 at 11:15 -0700, Cong Wang wrote:
>>>> (Cc'ing netdev.)
>>>>
>>>> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
>>>>> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
>>>>>> Hello,
>>>>>>
>>>>>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with
>>>>>> an
>>>>>> isatap-tunnel.
>>>>>
>>>>> I did some further tests. To trigger the crash you need
>>>>>
>>>>> * isatap-tunnel (probably any sit-tunnel will do it)
>>>>> * raw-socket
>>>>> * udp
>>>>>
>>>>> Using icmpv6 or tcp i.e. does not trigger it.
>>>>
>>>> Do you have a script to reproduce it?
>>>>
>>>>
>>>> Thanks for the bug report!
>>>> --
>>>
>>> Please Wolfgang try to revert 32dce968dd987adfb0c00946d78dad9154f64759
>>> ("ipv6: Allow for partial checksums on non-ufo packets")
>>
>> Indeed, that fixes the problem.
> 
> Yes, setting skb->csum to 0 is clearly wrong for CHECKSUM_PARTIAL
> 
> Would you try :
> 
> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
> index 7fde1f265c90e90f16291e6c861b6e242111c25b..694ae630e1ca67e25ab1e5f6dd0b3597db3669b0 100644
> --- a/net/ipv6/ip6_output.c
> +++ b/net/ipv6/ip6_output.c
> @@ -1416,6 +1416,10 @@ alloc_new_skb:
>  			data += fragheaderlen;
>  			skb->transport_header = (skb->network_header +
>  						 fragheaderlen);
> +			if (csummode == CHECKSUM_PARTIAL) {
> +				skb->csum_start = skb_transport_header(skb) - skb->head;
> +				skb->csum_offset = offsetof(struct udphdr, check);
> +			}
>  			if (fraggap) {
>  				skb->csum = skb_copy_and_csum_bits(
>  					skb_prev, maxfraglen,
> 
> 
> 

So why is this not an issue in __ip_append_data()?

-vlad

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Wolfgang Walter]

Am Mittwoch, 6. Mai 2015, 11:15:18 schrieben Sie:
> (Cc'ing netdev.)
> 
> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> > Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> >> Hello,
> >> 
> >> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
> >> isatap-tunnel.
> > 
> > I did some further tests. To trigger the crash you need
> > 
> > * isatap-tunnel (probably any sit-tunnel will do it)
> > * raw-socket
> > * udp
> > 
> > Using icmpv6 or tcp i.e. does not trigger it.
> 
> Do you have a script to reproduce it?
> 
> 
> Thanks for the bug report!

You need a isatap-server with say ipv4-address $X

Then, on host with 4.0, start isatapd: isatapd --mtu 1280 $X 

then do

	traceroute6 www.google.de

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Vlad Yasevich]

On 05/06/2015 06:11 PM, Wolfgang Walter wrote:
> Am Mittwoch, 6. Mai 2015, 11:15:18 schrieben Sie:
>> (Cc'ing netdev.)
>>
>> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
>>> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
>>>> Hello,
>>>>
>>>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
>>>> isatap-tunnel.
>>>
>>> I did some further tests. To trigger the crash you need
>>>
>>> * isatap-tunnel (probably any sit-tunnel will do it)
>>> * raw-socket
>>> * udp
>>>
>>> Using icmpv6 or tcp i.e. does not trigger it.
>>
>> Do you have a script to reproduce it?
>>
>>
>> Thanks for the bug report!
>
> You need a isatap-server with say ipv4-address $X
>
> Then, on host with 4.0, start isatapd: isatapd --mtu 1280 $X
>
> then do
>
> 	traceroute6 www.google.de
>
> Regards,
>

Hi Walter

Could you try this patch.  Looks like raw passes transhdrlen
of 0 on the first packet and that makes IPv4 behave correctly,
but not IPv6.


diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 7fde1f2..fd9c079 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -1278,7 +1278,7 @@ emsgsize:
 	/* If this is the first and only packet and device
 	 * supports checksum offloading, let's use it.
 	 */
-	if (!skb && sk->sk_protocol == IPPROTO_UDP &&
+	if (transhdrlen && sk->sk_protocol == IPPROTO_UDP &&
 	    length + fragheaderlen < mtu &&
 	    rt->dst.dev->features & NETIF_F_V6_CSUM &&
 	    !exthdrlen)

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Eric Dumazet]

On Tue, 2015-05-12 at 16:18 -0400, Vlad Yasevich wrote:
> On 05/06/2015 06:11 PM, Wolfgang Walter wrote:
> > Am Mittwoch, 6. Mai 2015, 11:15:18 schrieben Sie:
> >> (Cc'ing netdev.)
> >>
> >> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> >>> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> >>>> Hello,
> >>>>
> >>>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
> >>>> isatap-tunnel.
> >>>
> >>> I did some further tests. To trigger the crash you need
> >>>
> >>> * isatap-tunnel (probably any sit-tunnel will do it)
> >>> * raw-socket
> >>> * udp
> >>>
> >>> Using icmpv6 or tcp i.e. does not trigger it.
> >>
> >> Do you have a script to reproduce it?
> >>
> >>
> >> Thanks for the bug report!
> >
> > You need a isatap-server with say ipv4-address $X
> >
> > Then, on host with 4.0, start isatapd: isatapd --mtu 1280 $X
> >
> > then do
> >
> > 	traceroute6 www.google.de
> >
> > Regards,
> >
> 
> Hi Walter
> 
> Could you try this patch.  Looks like raw passes transhdrlen
> of 0 on the first packet and that makes IPv4 behave correctly,
> but not IPv6.
> 
> 
> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
> index 7fde1f2..fd9c079 100644
> --- a/net/ipv6/ip6_output.c
> +++ b/net/ipv6/ip6_output.c
> @@ -1278,7 +1278,7 @@ emsgsize:
>  	/* If this is the first and only packet and device
>  	 * supports checksum offloading, let's use it.
>  	 */
> -	if (!skb && sk->sk_protocol == IPPROTO_UDP &&
> +	if (transhdrlen && sk->sk_protocol == IPPROTO_UDP &&
>  	    length + fragheaderlen < mtu &&
>  	    rt->dst.dev->features & NETIF_F_V6_CSUM &&
>  	    !exthdrlen)

And make sure the checksum is correct ;)

Vlad, can you tell where skb->cum_start and skb->csum_offset are set ?

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Vlad Yasevich]

On 05/12/2015 04:36 PM, Eric Dumazet wrote:
> On Tue, 2015-05-12 at 16:18 -0400, Vlad Yasevich wrote:
>> On 05/06/2015 06:11 PM, Wolfgang Walter wrote:
>>> Am Mittwoch, 6. Mai 2015, 11:15:18 schrieben Sie:
>>>> (Cc'ing netdev.)
>>>>
>>>> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
>>>>> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
>>>>>> Hello,
>>>>>>
>>>>>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with an
>>>>>> isatap-tunnel.
>>>>>
>>>>> I did some further tests. To trigger the crash you need
>>>>>
>>>>> * isatap-tunnel (probably any sit-tunnel will do it)
>>>>> * raw-socket
>>>>> * udp
>>>>>
>>>>> Using icmpv6 or tcp i.e. does not trigger it.
>>>>
>>>> Do you have a script to reproduce it?
>>>>
>>>>
>>>> Thanks for the bug report!
>>>
>>> You need a isatap-server with say ipv4-address $X
>>>
>>> Then, on host with 4.0, start isatapd: isatapd --mtu 1280 $X
>>>
>>> then do
>>>
>>> 	traceroute6 www.google.de
>>>
>>> Regards,
>>>
>>
>> Hi Walter
>>
>> Could you try this patch.  Looks like raw passes transhdrlen
>> of 0 on the first packet and that makes IPv4 behave correctly,
>> but not IPv6.
>>
>>
>> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
>> index 7fde1f2..fd9c079 100644
>> --- a/net/ipv6/ip6_output.c
>> +++ b/net/ipv6/ip6_output.c
>> @@ -1278,7 +1278,7 @@ emsgsize:
>>  	/* If this is the first and only packet and device
>>  	 * supports checksum offloading, let's use it.
>>  	 */
>> -	if (!skb && sk->sk_protocol == IPPROTO_UDP &&
>> +	if (transhdrlen && sk->sk_protocol == IPPROTO_UDP &&
>>  	    length + fragheaderlen < mtu &&
>>  	    rt->dst.dev->features & NETIF_F_V6_CSUM &&
>>  	    !exthdrlen)
> 
> And make sure the checksum is correct ;)
> 
> Vlad, can you tell where skb->cum_start and skb->csum_offset are set ?
> 
> 

For udp, udp6_hwcsum_outgoing(), but with the above patch, the check above will return
false, and we'll fallback to using CHECKSUM_NONE.

Before, the !skb was true since there there was no skb on the queue.  Now, that
we are checking transhdr and raw passing in 0, the check will be false.  This
is what's making IPv4 work correctly in this case.

-vlad

Re: kernel >= 4.0: crashes when using traceroute6 with isatap

[Wolfgang Walter]

Am Dienstag, 12. Mai 2015, 16:18:54 schrieb Vlad Yasevich:
> On 05/06/2015 06:11 PM, Wolfgang Walter wrote:
> > Am Mittwoch, 6. Mai 2015, 11:15:18 schrieben Sie:
> >> (Cc'ing netdev.)
> >> 
> >> On Sat, May 2, 2015 at 5:29 AM, Wolfgang Walter <linux@stwm.de> wrote:
> >>> Am Samstag, 2. Mai 2015, 02:16:36 schrieb Wolfgang Walter:
> >>>> Hello,
> >>>> 
> >>>> kernel 4.0 (and 4.0.1) crashes immediately when I use traceroute6 with
> >>>> an
> >>>> isatap-tunnel.
> >>> 
> >>> I did some further tests. To trigger the crash you need
> >>> 
> >>> * isatap-tunnel (probably any sit-tunnel will do it)
> >>> * raw-socket
> >>> * udp
> >>> 
> >>> Using icmpv6 or tcp i.e. does not trigger it.
> >> 
> >> Do you have a script to reproduce it?
> >> 
> >> 
> >> Thanks for the bug report!
> > 
> > You need a isatap-server with say ipv4-address $X
> > 
> > Then, on host with 4.0, start isatapd: isatapd --mtu 1280 $X
> > 
> > then do
> > 
> > 	traceroute6 www.google.de
> > 
> > Regards,
> 
> Hi Walter
> 
> Could you try this patch.  Looks like raw passes transhdrlen
> of 0 on the first packet and that makes IPv4 behave correctly,
> but not IPv6.
> 
> 
> diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
> index 7fde1f2..fd9c079 100644
> --- a/net/ipv6/ip6_output.c
> +++ b/net/ipv6/ip6_output.c
> @@ -1278,7 +1278,7 @@ emsgsize:
>  	/* If this is the first and only packet and device
>  	 * supports checksum offloading, let's use it.
>  	 */
> -	if (!skb && sk->sk_protocol == IPPROTO_UDP &&
> +	if (transhdrlen && sk->sk_protocol == IPPROTO_UDP &&
>  	    length + fragheaderlen < mtu &&
>  	    rt->dst.dev->features & NETIF_F_V6_CSUM &&
>  	    !exthdrlen)

Ok, that works as fix. No crash and checksums are ok.

Regards,
-- 
Wolfgang Walter
Studentenwerk München
Anstalt des öffentlichen Rechts