Re: Kernel Panic with vti Interfaces

Re: Kernel Panic with vti Interfaces

[Steffen Klassert]

Ccing netdev.

On Tue, Feb 24, 2015 at 11:03:15AM +0100, Andre Valentin wrote:
> Hi!
> 
> I noticed kernel panics on my mips platform while using vti interfaces. After taking a deeper look,
> it seems that if the SA changes or the WAN interface changes I get this error. I could find the
> code where it crashes. But I'm not sure if my solution is right. Perhaps you could take a look.
> 


> From: =?utf8?q?Andr=C3=A9=20Valentin?= <avalentin@marcant.net>
> Date: Tue, 24 Feb 2015 00:24:08 +0100
> Subject: [PATCH] kernel: fix xfrm tunnel checks if vti parent SA/device is in transition
> 
> The outer_mode check results in the error if it is not checked for validity.
> Because I do not know how to handle this situation, I decided to return
> -EINVAL if outer_mode is null.
> 
> ---
>  .../618-net_fixup_xfrm_tunnel_check.patch          |   16 ++++++++++++++++
>  1 files changed, 16 insertions(+), 0 deletions(-)
>  create mode 100644 target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
> 
> diff --git a/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch b/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
> new file mode 100644
> index 0000000..f79b1f7
> --- /dev/null
> +++ b/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
> @@ -0,0 +1,16 @@
> +--- a/include/net/xfrm.h	2015-02-11 08:01:12.000000000 +0100
> ++++ b/include/net/xfrm.h	2015-02-24 00:04:03.102709830 +0100
> +@@ -1805,8 +1805,12 @@ static inline int xfrm_tunnel_check(stru
> + 			tunnel = true;
> + 		break;
> + 	}
> +-	if (tunnel && !(x->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL))
> ++	if (tunnel && !x->outer_mode) {
> ++                printk(KERN_NOTICE "xfrm_tunnel_check: outer_mode is 0, returning -EINVAL\n");
> + 		return -EINVAL;

Returning -EINVAL here would just paper over the real bug.

We should never get a state without outer_mode from a lookup.
Using such a state will lead to a crash anyway, even without
using vti devices. Looks like you get an uninitialized state
with the lookup. When a xfrm_state is initialized, the outer
mode is added and after that inserted to the lookup tables.
It should never loose the outer_mode pointer.

I have never seen this, is the patch above the only locally
applied patch?

Is this reproducible?

Re: Kernel Panic with vti Interfaces

[Andre Valentin]

Hi Steffen!

>> From: =?utf8?q?Andr=C3=A9=20Valentin?= <avalentin@marcant.net>
>> Date: Tue, 24 Feb 2015 00:24:08 +0100
>> Subject: [PATCH] kernel: fix xfrm tunnel checks if vti parent SA/device is in transition
>>
>> The outer_mode check results in the error if it is not checked for validity.
>> Because I do not know how to handle this situation, I decided to return
>> -EINVAL if outer_mode is null.
>>
>> ---
>>  .../618-net_fixup_xfrm_tunnel_check.patch          |   16 ++++++++++++++++
>>  1 files changed, 16 insertions(+), 0 deletions(-)
>>  create mode 100644 target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
>>
>> diff --git a/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch b/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
>> new file mode 100644
>> index 0000000..f79b1f7
>> --- /dev/null
>> +++ b/target/linux/generic/patches-3.18/618-net_fixup_xfrm_tunnel_check.patch
>> @@ -0,0 +1,16 @@
>> +--- a/include/net/xfrm.h    2015-02-11 08:01:12.000000000 +0100
>> ++++ b/include/net/xfrm.h    2015-02-24 00:04:03.102709830 +0100
>> +@@ -1805,8 +1805,12 @@ static inline int xfrm_tunnel_check(stru
>> +                    tunnel = true;
>> +            break;
>> +    }
>> +-   if (tunnel && !(x->outer_mode->flags & XFRM_MODE_FLAG_TUNNEL))
>> ++   if (tunnel && !x->outer_mode) {
>> ++                printk(KERN_NOTICE "xfrm_tunnel_check: outer_mode is 0, returning -EINVAL\n");
>> +            return -EINVAL;
>
> Returning -EINVAL here would just paper over the real bug.
>
> We should never get a state without outer_mode from a lookup.
> Using such a state will lead to a crash anyway, even without
> using vti devices. Looks like you get an uninitialized state
> with the lookup. When a xfrm_state is initialized, the outer
> mode is added and after that inserted to the lookup tables.
> It should never loose the outer_mode pointer.

Okay, I will take a deeper look at the other modifications of
the platform.

Here are some log entries around the event:
Fri Feb 27 00:31:24 2015 daemon.info syslog: 09[ENC] generating CREATE_CHILD_SA request 10 [ N(REKEY_SA) SA No KE TSi TSr ]
Fri Feb 27 00:31:24 2015 daemon.info syslog: 09[NET] sending packet: from a.b.c.d[500] to c.d.e.f[500] (541 bytes)
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.530000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.540000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.550000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 daemon.info syslog: 04[NET] received packet: from c.d.e.f[500] to a.b.c.d[500] (529 bytes)
Fri Feb 27 00:31:25 2015 daemon.info syslog: 04[ENC] parsed CREATE_CHILD_SA response 10 [ SA No KE TSi TSr ]
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.580000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.630000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.640000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.650000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.720000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.730000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.730000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.740000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.750000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.750000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 kern.notice kernel: [255629.790000] xfrm_tunnel_check: outer_mode is 0, returning -EINVAL
Fri Feb 27 00:31:25 2015 daemon.info syslog: 04[IKE] CHILD_SA myhost-vti{111} established with SPIs c65bd3e4_i cd2e85f8_o and TS 0.0.0.0/0 ::/0 === 0.0.0.0/0 ::/0
Fri Feb 27 00:31:25 2015 authpriv.info syslog: 04[IKE] CHILD_SA myhost-vti{111} established with SPIs c65bd3e4_i cd2e85f8_o and TS 0.0.0.0/0 ::/0 === 0.0.0.0/0 ::/0

>
> I have never seen this, is the patch above the only locally
> applied patch?
No, the kernel is coming from the current openwrt 3.18 trunk.
It's 3.18.7 with serveral patches coming from openwrt.
I already checked the IPsec related patches but the error
persisted.

I will remove more patches to be sure and forward my results!

> Is this reproducible?

Yes, it this happens on every pppoe reconnect. That annoyed me
a bit and I dived into it ;-)

Kind regards,

André Valentin



Mit freundlichen Grüßen
 André Valentin
Systemadministrator
--

++++++WIR ZIEHEN UM++++++
Ab dem 09.03.2015 erreichen Sie uns unter folgender Anschrift.

MarcanT GmbH
Herforder Straße 163 a
33609 Bielefeld

Bitte beachten Sie, dass ab dem 09.03.2015 alle Rechnungen und
Korrespondenz nur noch auf die oben genannte Anschrift ausgestellt
werden. Aktualisieren Sie bitte Ihre Stammdaten entsprechend. Wir
wünschen uns, den Umzug für Sie und uns ohne Beeinträchtigungen des
Tagesgeschäftes abwickeln zu können. Sollte es dennoch zu Schwierigkeiten
kommen, hoffen wir auf Ihr Verständnis.
Auf unser Rechenzentrum hat der Umzug keinen Einfluss; die Funktionen
wurden bereits Anfang Februar auf unsere Redundanzrechenzentren verteilt.
++++++++++++++++++++++++++++++++++++

MarcanT GmbH, Ravensberger Str. 10 G, D - 33602 Bielefeld
Fon: +49 (521) 95945-0 | Fax: +49 (521) 95945-18
URL: http://www.marcant.net | http://www.global-m2m.com

Internet * Netzwerk * Mobile Daten
Citrix Silver Solution Advisor

Geschäftsführer: Thorsten Hojas
Handelsregister: AG Bielefeld, HRB 35827 USt-ID Nr.: DE 190203238
___________________________________________________________
Ausserhalb unserer Geschäftszeiten (Montag bis Freitag von 8:30 Uhr bis
17:30 Uhr, ausgenommen gesetzliche Feiertage in NRW) stehen wir Ihnen
gemäß Ihrer jeweiligen Service-Level-Agreements unter der Ihnen
mitgeteilten Telefonnummer für Störungen und Notfälle zur Verfügung.
Sie können natürlich auch gerne jederzeit unter support@marcant.net ein
Ticket eröffnen, welches am nächsten Arbeitstag bearbeitet wird.

Re: Kernel Panic with vti Interfaces

[Steffen Klassert]

On Fri, Feb 27, 2015 at 12:58:10AM +0100, Andre Valentin wrote:
> 
> >
> > I have never seen this, is the patch above the only locally
> > applied patch?
> No, the kernel is coming from the current openwrt 3.18 trunk.
> It's 3.18.7 with serveral patches coming from openwrt.
> I already checked the IPsec related patches but the error
> persisted.
> 
> I will remove more patches to be sure and forward my results!

Please try with an unpatched kernel from kernel.org, thanks!

Re: Kernel Panic with vti Interfaces

[Andre Valentin]

Hi!

Am 27.02.2015 um 11:35 schrieb Steffen Klassert:
> On Fri, Feb 27, 2015 at 12:58:10AM +0100, Andre Valentin wrote:
>>
>>>
>>> I have never seen this, is the patch above the only locally
>>> applied patch?
>> No, the kernel is coming from the current openwrt 3.18 trunk.
>> It's 3.18.7 with serveral patches coming from openwrt.
>> I already checked the IPsec related patches but the error
>> persisted.
>>
>> I will remove more patches to be sure and forward my results!
>
> Please try with an unpatched kernel from kernel.org, thanks!
>

I just retried with vanilla on x86_64 and there no such error.
I'm sorry for the noise. I've been sure that I could exclude
the openwrt patches.  Thank you very much for your help.

Kind regards,

André


Mit freundlichen Grüßen
 André Valentin
Systemadministrator
--

++++++WIR ZIEHEN UM++++++
Ab dem 09.03.2015 erreichen Sie uns unter folgender Anschrift.

MarcanT GmbH
Herforder Straße 163 a
33609 Bielefeld

Bitte beachten Sie, dass ab dem 09.03.2015 alle Rechnungen und
Korrespondenz nur noch auf die oben genannte Anschrift ausgestellt
werden. Aktualisieren Sie bitte Ihre Stammdaten entsprechend. Wir
wünschen uns, den Umzug für Sie und uns ohne Beeinträchtigungen des
Tagesgeschäftes abwickeln zu können. Sollte es dennoch zu Schwierigkeiten
kommen, hoffen wir auf Ihr Verständnis.
Auf unser Rechenzentrum hat der Umzug keinen Einfluss; die Funktionen
wurden bereits Anfang Februar auf unsere Redundanzrechenzentren verteilt.
++++++++++++++++++++++++++++++++++++

MarcanT GmbH, Ravensberger Str. 10 G, D - 33602 Bielefeld
Fon: +49 (521) 95945-0 | Fax: +49 (521) 95945-18
URL: http://www.marcant.net | http://www.global-m2m.com

Internet * Netzwerk * Mobile Daten
Citrix Silver Solution Advisor

Geschäftsführer: Thorsten Hojas
Handelsregister: AG Bielefeld, HRB 35827 USt-ID Nr.: DE 190203238
___________________________________________________________
Ausserhalb unserer Geschäftszeiten (Montag bis Freitag von 8:30 Uhr bis
17:30 Uhr, ausgenommen gesetzliche Feiertage in NRW) stehen wir Ihnen
gemäß Ihrer jeweiligen Service-Level-Agreements unter der Ihnen
mitgeteilten Telefonnummer für Störungen und Notfälle zur Verfügung.
Sie können natürlich auch gerne jederzeit unter support@marcant.net ein
Ticket eröffnen, welches am nächsten Arbeitstag bearbeitet wird.