Oops in authenc: 2.6.26.3

Oops in authenc: 2.6.26.3

[Matt LaPlante]

After upgrading one of my IPSec connected boxes from 2.6.25.x to 2.6.26.x I began to experience the following oopses.  I haven't identified what exactly causes it to trigger, but it has occurred repeatedly, anywhere from 5 to 15 minutes of the box booting the new kernel and running.  I'm not sure what other details are relevant for debugging, so I'll wait for feedback on what specific data is needed.

Ubuntu 8.04
racoon 1:0.6.7-1.1ubuntu1
ipsec-tools 1:0.6.7-1.1ubuntu1

BUG: unable to handle kernel paging request at 5684c077
IP: [<cc96856b>] :authenc:crypto_authenc_givencrypt_done+0x1b/0x20
Oops: 0000 [#1]
Modules linked in: authenc esp4 aead xfrm4_mode_tunnel tun iptable_raw xt_policy
 xt_multiport ipt_ULOG ipt_TTL ipt_REJECT ipt_REDIRECT ipt_recent ipt_NETMAP ipt
_MASQUERADE ipt_LOG ipt_ECN ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_NFQUEUE
xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_conntrack xt_CONNMARK xt_
connmark xt_CLASSIFY iptable_nat nf_nat nf_conntrack_ipv4 iptable_mangle nfnetli
nk iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack xt_
tcpudp ip6table_filter ip6_tables x_tables sit tunnel4 deflate zlib_deflate zlib
_inflate ctr twofish_i586 twofish_common serpent blowfish des_generic cbc aes_i5
86 aes_generic xcbc sha256_generic sha1_generic md5 crypto_null crypto_blkcipher
 af_key psmouse af_packet ipv6 evdev pcspkr sd_mod sg via_rhine ata_piix ata_gen
eric

Pid: 4, comm: events/0 Not tainted (2.6.26.3-Gateway.v11 #2)
EIP: 0060:[<cc96856b>] EFLAGS: 00010296 CPU: 0
EIP is at crypto_authenc_givencrypt_done+0x1b/0x20 [authenc]
EAX: 5684c06f EBX: cbaef540 ECX: 5684c06f EDX: 00000000
ESI: cbaef540 EDI: cbaef530 EBP: cb1a3170 ESP: cb825f74
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process events/0 (pid: 4, ti=cb824000 task=cb81d120 task.ti=cb824000)
Stack: 00000000 cc84f6c0 cb1a318c cb8029c0 cc84f670 cb8029c0 c01267fb cb1d8760
       cb81007b c011007b c0350000 cb8029c0 cb8029c8 c0126ef0 00000000 c0126f49
       00000000 cb81d120 c0129880 cb825fc0 cb825fc0 fffffffc cb8029c0 c0126ef0
Call Trace:
[<cc84f6c0>] async_chainiv_do_postponed+0x50/0x80 [crypto_blkcipher]
[<cc84f670>] async_chainiv_do_postponed_0x0/0x80 [crypto_blkcipher]
[<c01267fb>] run_workqueue+0x8b/0x110
[<c011007b>] handle_vm86_fault+0x5b/0x7c0
[<c0126ef0>] worker_thread+0x0/0x90
[<c0126f49>] worker_thread+0x59/0x90
[<c0129880>] autoremove_wake_function+0x0/0x40
[<c0126ef0>] worker_thread+0x0/0x90
[<c0129502>] kthread+0x42/0x70
[<c01294c0>] kthread+0x0/0x70
[<c0103953>] kernel_thread_helper+0x7/0x14
Code: 09 03 e9 3f ff ff ff 89 f6 8d bc 27 00 00 00 00 85 d2 53 89 c3 75 0f 8b 40 
0c 31 c9 8b 50 38 e8 fc fb ff ff 89 c2 8b 4b 0c 89 c8 <ff> 51 08 5b c3 85 d2 53
89 c3 75 18 8b 40 0c 8b 48 10 8d 50 58
EIP: [<cc96856b>] crypto_authenc_givencrypt_done+0x1b/0x20 [authenc] SS:ESP 0068:cb825f74
Kernel panic - not syncing: Fatal exception in interrupt

-- 
Matt LaPlante

Re: Oops in authenc: 2.6.26.3

[Andrew Morton]

(cc's added)

On Wed, 20 Aug 2008 20:32:45 -0500 Matt LaPlante <kernel1@cyberdogtech.com> wrote:

> After upgrading one of my IPSec connected boxes from 2.6.25.x to 2.6.26.x I began to experience the following oopses.  I haven't identified what exactly causes it to trigger, but it has occurred repeatedly, anywhere from 5 to 15 minutes of the box booting the new kernel and running.  I'm not sure what other details are relevant for debugging, so I'll wait for feedback on what specific data is needed.
> 
> Ubuntu 8.04
> racoon 1:0.6.7-1.1ubuntu1
> ipsec-tools 1:0.6.7-1.1ubuntu1
> 
> BUG: unable to handle kernel paging request at 5684c077
> IP: [<cc96856b>] :authenc:crypto_authenc_givencrypt_done+0x1b/0x20
> Oops: 0000 [#1]
> Modules linked in: authenc esp4 aead xfrm4_mode_tunnel tun iptable_raw xt_policy
>  xt_multiport ipt_ULOG ipt_TTL ipt_REJECT ipt_REDIRECT ipt_recent ipt_NETMAP ipt
> _MASQUERADE ipt_LOG ipt_ECN ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_NFQUEUE
> xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_conntrack xt_CONNMARK xt_
> connmark xt_CLASSIFY iptable_nat nf_nat nf_conntrack_ipv4 iptable_mangle nfnetli
> nk iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack xt_
> tcpudp ip6table_filter ip6_tables x_tables sit tunnel4 deflate zlib_deflate zlib
> _inflate ctr twofish_i586 twofish_common serpent blowfish des_generic cbc aes_i5
> 86 aes_generic xcbc sha256_generic sha1_generic md5 crypto_null crypto_blkcipher
>  af_key psmouse af_packet ipv6 evdev pcspkr sd_mod sg via_rhine ata_piix ata_gen
> eric
> 
> Pid: 4, comm: events/0 Not tainted (2.6.26.3-Gateway.v11 #2)
> EIP: 0060:[<cc96856b>] EFLAGS: 00010296 CPU: 0
> EIP is at crypto_authenc_givencrypt_done+0x1b/0x20 [authenc]
> EAX: 5684c06f EBX: cbaef540 ECX: 5684c06f EDX: 00000000
> ESI: cbaef540 EDI: cbaef530 EBP: cb1a3170 ESP: cb825f74
>  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
> Process events/0 (pid: 4, ti=cb824000 task=cb81d120 task.ti=cb824000)
> Stack: 00000000 cc84f6c0 cb1a318c cb8029c0 cc84f670 cb8029c0 c01267fb cb1d8760
>        cb81007b c011007b c0350000 cb8029c0 cb8029c8 c0126ef0 00000000 c0126f49
>        00000000 cb81d120 c0129880 cb825fc0 cb825fc0 fffffffc cb8029c0 c0126ef0
> Call Trace:
> [<cc84f6c0>] async_chainiv_do_postponed+0x50/0x80 [crypto_blkcipher]
> [<cc84f670>] async_chainiv_do_postponed_0x0/0x80 [crypto_blkcipher]
> [<c01267fb>] run_workqueue+0x8b/0x110
> [<c011007b>] handle_vm86_fault+0x5b/0x7c0
> [<c0126ef0>] worker_thread+0x0/0x90
> [<c0126f49>] worker_thread+0x59/0x90
> [<c0129880>] autoremove_wake_function+0x0/0x40
> [<c0126ef0>] worker_thread+0x0/0x90
> [<c0129502>] kthread+0x42/0x70
> [<c01294c0>] kthread+0x0/0x70
> [<c0103953>] kernel_thread_helper+0x7/0x14
> Code: 09 03 e9 3f ff ff ff 89 f6 8d bc 27 00 00 00 00 85 d2 53 89 c3 75 0f 8b 40 
> 0c 31 c9 8b 50 38 e8 fc fb ff ff 89 c2 8b 4b 0c 89 c8 <ff> 51 08 5b c3 85 d2 53
> 89 c3 75 18 8b 40 0c 8b 48 10 8d 50 58
> EIP: [<cc96856b>] crypto_authenc_givencrypt_done+0x1b/0x20 [authenc] SS:ESP 0068:cb825f74
> Kernel panic - not syncing: Fatal exception in interrupt
> 

Re: Oops in authenc: 2.6.26.3

[Sascha Biberhofer]

I have the same problem on my system, starting with the release of
2.6.26. Shortly afterwards I've had the same problem with the 2.6.25
series starting with 2.6.25.12. I've looked up the changes between
2.6.25.11 and .12 and found commit
c2bd04d8040a91fe2ee2e9fee1a6562ca9792249 (it's commit
872ac8743cb400192a9fce4ba2d3ffd7bb309685 in the 2.6.26 series).
Reverting the commit seems to solve the problem here, I've been running
a 2.6.25.12 kernel without this commit for some weeks now.
In case it's important: I'm using an IPSec ESP transport with AES-256
and sha-256 auth. 

My oops:

Pid: 4, comm: events/0 Not tainted (2.6.25.12 #0)
EIP: 0060:[<c0202d7b>] EFLAGS: 00010296 CPU: 0
EIP is at crypto_authenc_givencrypt_done+0x1b/0x20
EAX: 3e03add6 EBX: c9f29b60 ECX: 3e03add6 EDX: 00000000
ESI: c9f39b60 EDI: c9f39b50 EBP: de8bb7b0 ESP: df833f74
 DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
Process events/0 (pid:4, ti=df832000 task=df829020 task:ti=df832000)

Stack: 00000000 c01f4a60 de8bb7cc df808100 c01f4a10 df808100 c01277eb df829020
       c0383528 0000007b 0000007b df909100 df808108 c0127ee0 00000000 c0127f39
       00000000 df829020 c012a680 df833fc0 df833fc0 fffffffc df808100 c0127ee0

Call trace: [<c01f4a60>] async_chainiv_do_postponed+0x50/0x80
            [<c01f4a10>] async_chainiv_do_postponed+0x0/0x80
            [<c01277eb>] run_workqueue+0x8b/0x100
            [<c0383528>] schedule+0x148/0x280
            [<c0127ee0>] worker_thread+0x0/0x90
            [<c0127f39>] worker_thread+0x59/0x90
            [<c012a680>] autoremove_wake_function+0x0/0x40
            [<c0127ee0>] worker_thread+0x0/0x90
            [<c012a302>] kthread+0x42/0x70
            [<c012a2c0>] kthread+0x0/0x70
            [<c0103867>] kernel_thread_helper+0x7/0x10

Code: ff 09 03 e9 3e ff ff ff 90 8d b4 26 00 00 00 85 d2 53 89 c3 75 0f 8b 40 0c 31 c9 8b 50 38 e8 0c fc ff ff 89 c2 8b 4b 0c
      89 c8 <ff> 51 51 08 5b c3 85 d2 53 89 c3 75 18 8b 40 0c 8b 48 10 8d 50 58


hth, 
  Sascha Biberhofer

Re: Oops in authenc: 2.6.26.3

[Herbert Xu]

On Thu, Aug 21, 2008 at 07:02:25AM +0000, Sascha Biberhofer wrote:
> I have the same problem on my system, starting with the release of
> 2.6.26. Shortly afterwards I've had the same problem with the 2.6.25
> series starting with 2.6.25.12. I've looked up the changes between
> 2.6.25.11 and .12 and found commit
> c2bd04d8040a91fe2ee2e9fee1a6562ca9792249 (it's commit
> 872ac8743cb400192a9fce4ba2d3ffd7bb309685 in the 2.6.26 series).
> Reverting the commit seems to solve the problem here, I've been running
> a 2.6.25.12 kernel without this commit for some weeks now.
> In case it's important: I'm using an IPSec ESP transport with AES-256
> and sha-256 auth. 

Sorry, I was skimping on memory and ended up calling a clobbered
function pointer.

This patch should fix it.

crypto: authenc - Avoid using clobbered request pointer

Authenc works in two stages for encryption, it first encrypts and
then computes an ICV.  The context memory of the request is used
by both operations.  The problem is that when an asynchronous
encryption completes, we will compute the ICV and then reread the
context memory of the encryption to get the original request.

It just happens that we have a buffer of 16 bytes in front of the
request pointer, so ICVs of 16 bytes (such as SHA1) do not trigger
the bug.  However, any attempt to uses a larger ICV instantly kills
the machine when the first asynchronous encryption is completed.

This patch fixes this by saving the request pointer before we start
the ICV computation.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
diff --git a/crypto/authenc.c b/crypto/authenc.c
index 4b22676..fd9f06c 100644
--- a/crypto/authenc.c
+++ b/crypto/authenc.c
@@ -174,8 +174,9 @@ static int crypto_authenc_genicv(struct aead_request *req, u8 *iv,
 static void crypto_authenc_encrypt_done(struct crypto_async_request *req,
 					int err)
 {
+	struct aead_request *areq = req->data;
+
 	if (!err) {
-		struct aead_request *areq = req->data;
 		struct crypto_aead *authenc = crypto_aead_reqtfm(areq);
 		struct crypto_authenc_ctx *ctx = crypto_aead_ctx(authenc);
 		struct ablkcipher_request *abreq = aead_request_ctx(areq);
@@ -185,7 +186,7 @@ static void crypto_authenc_encrypt_done(struct crypto_async_request *req,
 		err = crypto_authenc_genicv(areq, iv, 0);
 	}
 
-	aead_request_complete(req->data, err);
+	aead_request_complete(areq, err);
 }
 
 static int crypto_authenc_encrypt(struct aead_request *req)
@@ -216,14 +217,15 @@ static int crypto_authenc_encrypt(struct aead_request *req)
 static void crypto_authenc_givencrypt_done(struct crypto_async_request *req,
 					   int err)
 {
+	struct aead_request *areq = req->data;
+
 	if (!err) {
-		struct aead_request *areq = req->data;
 		struct skcipher_givcrypt_request *greq = aead_request_ctx(areq);
 
 		err = crypto_authenc_genicv(areq, greq->giv, 0);
 	}
 
-	aead_request_complete(req->data, err);
+	aead_request_complete(areq, err);
 }
 
 static int crypto_authenc_givencrypt(struct aead_givcrypt_request *req)

Re: Oops in authenc: 2.6.26.3

[Matt LaPlante]

On Thu, 21 Aug 2008 18:36:15 +1000
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> On Thu, Aug 21, 2008 at 07:02:25AM +0000, Sascha Biberhofer wrote:
> > I have the same problem on my system, starting with the release of
> > 2.6.26. Shortly afterwards I've had the same problem with the 2.6.25
> > series starting with 2.6.25.12. I've looked up the changes between
> > 2.6.25.11 and .12 and found commit
> > c2bd04d8040a91fe2ee2e9fee1a6562ca9792249 (it's commit
> > 872ac8743cb400192a9fce4ba2d3ffd7bb309685 in the 2.6.26 series).
> > Reverting the commit seems to solve the problem here, I've been running
> > a 2.6.25.12 kernel without this commit for some weeks now.
> > In case it's important: I'm using an IPSec ESP transport with AES-256
> > and sha-256 auth. 
> 
> Sorry, I was skimping on memory and ended up calling a clobbered
> function pointer.
> 
> This patch should fix it.
> 
> crypto: authenc - Avoid using clobbered request pointer
> 
> Authenc works in two stages for encryption, it first encrypts and
> then computes an ICV.  The context memory of the request is used
> by both operations.  The problem is that when an asynchronous
> encryption completes, we will compute the ICV and then reread the
> context memory of the encryption to get the original request.
> 
> It just happens that we have a buffer of 16 bytes in front of the
> request pointer, so ICVs of 16 bytes (such as SHA1) do not trigger
> the bug.  However, any attempt to uses a larger ICV instantly kills
> the machine when the first asynchronous encryption is completed.
> 
> This patch fixes this by saving the request pointer before we start
> the ICV computation.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Acked-by: Matt LaPlante <kernel1@cyberdogtech.com>

Thanks for the quick fix!

-- 
Matt LaPlante

Re: Oops in authenc: 2.6.26.3

[Henrique de Moraes Holschuh]

Cc: stable@kernel.org ?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Oops in authenc: 2.6.26.3

[Herbert Xu]

On Thu, Aug 21, 2008 at 11:56:10AM -0300, Henrique de Moraes Holschuh wrote:
> Cc: stable@kernel.org ?

Once it's merged by Linus I'll push it to stable.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Oops in authenc: 2.6.26.3

[Sascha Biberhofer]

Herbert Xu wrote:
> Sorry, I was skimping on memory and ended up calling a clobbered
> function pointer.
> 
> This patch should fix it.

Works like a charm, thanks for the quick fix. 

  Sascha Biberhofer