Re: Oops in authenc: 2.6.26.3

Re: Oops in authenc: 2.6.26.3

[Andrew Morton]

(cc's added)

On Wed, 20 Aug 2008 20:32:45 -0500 Matt LaPlante <kernel1@cyberdogtech.com> wrote:

> After upgrading one of my IPSec connected boxes from 2.6.25.x to 2.6.26.x I began to experience the following oopses.  I haven't identified what exactly causes it to trigger, but it has occurred repeatedly, anywhere from 5 to 15 minutes of the box booting the new kernel and running.  I'm not sure what other details are relevant for debugging, so I'll wait for feedback on what specific data is needed.
> 
> Ubuntu 8.04
> racoon 1:0.6.7-1.1ubuntu1
> ipsec-tools 1:0.6.7-1.1ubuntu1
> 
> BUG: unable to handle kernel paging request at 5684c077
> IP: [<cc96856b>] :authenc:crypto_authenc_givencrypt_done+0x1b/0x20
> Oops: 0000 [#1]
> Modules linked in: authenc esp4 aead xfrm4_mode_tunnel tun iptable_raw xt_policy
>  xt_multiport ipt_ULOG ipt_TTL ipt_REJECT ipt_REDIRECT ipt_recent ipt_NETMAP ipt
> _MASQUERADE ipt_LOG ipt_ECN ipt_ah ipt_addrtype xt_tcpmss xt_pkttype xt_NFQUEUE
> xt_MARK xt_mark xt_mac xt_limit xt_length xt_helper xt_conntrack xt_CONNMARK xt_
> connmark xt_CLASSIFY iptable_nat nf_nat nf_conntrack_ipv4 iptable_mangle nfnetli
> nk iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack xt_
> tcpudp ip6table_filter ip6_tables x_tables sit tunnel4 deflate zlib_deflate zlib
> _inflate ctr twofish_i586 twofish_common serpent blowfish des_generic cbc aes_i5
> 86 aes_generic xcbc sha256_generic sha1_generic md5 crypto_null crypto_blkcipher
>  af_key psmouse af_packet ipv6 evdev pcspkr sd_mod sg via_rhine ata_piix ata_gen
> eric
> 
> Pid: 4, comm: events/0 Not tainted (2.6.26.3-Gateway.v11 #2)
> EIP: 0060:[<cc96856b>] EFLAGS: 00010296 CPU: 0
> EIP is at crypto_authenc_givencrypt_done+0x1b/0x20 [authenc]
> EAX: 5684c06f EBX: cbaef540 ECX: 5684c06f EDX: 00000000
> ESI: cbaef540 EDI: cbaef530 EBP: cb1a3170 ESP: cb825f74
>  DS: 007b ES: 007b FS: 0000 GS: 0000 SS: 0068
> Process events/0 (pid: 4, ti=cb824000 task=cb81d120 task.ti=cb824000)
> Stack: 00000000 cc84f6c0 cb1a318c cb8029c0 cc84f670 cb8029c0 c01267fb cb1d8760
>        cb81007b c011007b c0350000 cb8029c0 cb8029c8 c0126ef0 00000000 c0126f49
>        00000000 cb81d120 c0129880 cb825fc0 cb825fc0 fffffffc cb8029c0 c0126ef0
> Call Trace:
> [<cc84f6c0>] async_chainiv_do_postponed+0x50/0x80 [crypto_blkcipher]
> [<cc84f670>] async_chainiv_do_postponed_0x0/0x80 [crypto_blkcipher]
> [<c01267fb>] run_workqueue+0x8b/0x110
> [<c011007b>] handle_vm86_fault+0x5b/0x7c0
> [<c0126ef0>] worker_thread+0x0/0x90
> [<c0126f49>] worker_thread+0x59/0x90
> [<c0129880>] autoremove_wake_function+0x0/0x40
> [<c0126ef0>] worker_thread+0x0/0x90
> [<c0129502>] kthread+0x42/0x70
> [<c01294c0>] kthread+0x0/0x70
> [<c0103953>] kernel_thread_helper+0x7/0x14
> Code: 09 03 e9 3f ff ff ff 89 f6 8d bc 27 00 00 00 00 85 d2 53 89 c3 75 0f 8b 40 
> 0c 31 c9 8b 50 38 e8 fc fb ff ff 89 c2 8b 4b 0c 89 c8 <ff> 51 08 5b c3 85 d2 53
> 89 c3 75 18 8b 40 0c 8b 48 10 8d 50 58
> EIP: [<cc96856b>] crypto_authenc_givencrypt_done+0x1b/0x20 [authenc] SS:ESP 0068:cb825f74
> Kernel panic - not syncing: Fatal exception in interrupt
> 

Re: Oops in authenc: 2.6.26.3

[Herbert Xu]

On Thu, Aug 21, 2008 at 07:02:25AM +0000, Sascha Biberhofer wrote:
> I have the same problem on my system, starting with the release of
> 2.6.26. Shortly afterwards I've had the same problem with the 2.6.25
> series starting with 2.6.25.12. I've looked up the changes between
> 2.6.25.11 and .12 and found commit
> c2bd04d8040a91fe2ee2e9fee1a6562ca9792249 (it's commit
> 872ac8743cb400192a9fce4ba2d3ffd7bb309685 in the 2.6.26 series).
> Reverting the commit seems to solve the problem here, I've been running
> a 2.6.25.12 kernel without this commit for some weeks now.
> In case it's important: I'm using an IPSec ESP transport with AES-256
> and sha-256 auth. 

Sorry, I was skimping on memory and ended up calling a clobbered
function pointer.

This patch should fix it.

crypto: authenc - Avoid using clobbered request pointer

Authenc works in two stages for encryption, it first encrypts and
then computes an ICV.  The context memory of the request is used
by both operations.  The problem is that when an asynchronous
encryption completes, we will compute the ICV and then reread the
context memory of the encryption to get the original request.

It just happens that we have a buffer of 16 bytes in front of the
request pointer, so ICVs of 16 bytes (such as SHA1) do not trigger
the bug.  However, any attempt to uses a larger ICV instantly kills
the machine when the first asynchronous encryption is completed.

This patch fixes this by saving the request pointer before we start
the ICV computation.

Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
--
diff --git a/crypto/authenc.c b/crypto/authenc.c
index 4b22676..fd9f06c 100644
--- a/crypto/authenc.c
+++ b/crypto/authenc.c
@@ -174,8 +174,9 @@ static int crypto_authenc_genicv(struct aead_request *req, u8 *iv,
 static void crypto_authenc_encrypt_done(struct crypto_async_request *req,
 					int err)
 {
+	struct aead_request *areq = req->data;
+
 	if (!err) {
-		struct aead_request *areq = req->data;
 		struct crypto_aead *authenc = crypto_aead_reqtfm(areq);
 		struct crypto_authenc_ctx *ctx = crypto_aead_ctx(authenc);
 		struct ablkcipher_request *abreq = aead_request_ctx(areq);
@@ -185,7 +186,7 @@ static void crypto_authenc_encrypt_done(struct crypto_async_request *req,
 		err = crypto_authenc_genicv(areq, iv, 0);
 	}
 
-	aead_request_complete(req->data, err);
+	aead_request_complete(areq, err);
 }
 
 static int crypto_authenc_encrypt(struct aead_request *req)
@@ -216,14 +217,15 @@ static int crypto_authenc_encrypt(struct aead_request *req)
 static void crypto_authenc_givencrypt_done(struct crypto_async_request *req,
 					   int err)
 {
+	struct aead_request *areq = req->data;
+
 	if (!err) {
-		struct aead_request *areq = req->data;
 		struct skcipher_givcrypt_request *greq = aead_request_ctx(areq);
 
 		err = crypto_authenc_genicv(areq, greq->giv, 0);
 	}
 
-	aead_request_complete(req->data, err);
+	aead_request_complete(areq, err);
 }
 
 static int crypto_authenc_givencrypt(struct aead_givcrypt_request *req)

Re: Oops in authenc: 2.6.26.3

[Matt LaPlante]

On Thu, 21 Aug 2008 18:36:15 +1000
Herbert Xu <herbert@gondor.apana.org.au> wrote:

> On Thu, Aug 21, 2008 at 07:02:25AM +0000, Sascha Biberhofer wrote:
> > I have the same problem on my system, starting with the release of
> > 2.6.26. Shortly afterwards I've had the same problem with the 2.6.25
> > series starting with 2.6.25.12. I've looked up the changes between
> > 2.6.25.11 and .12 and found commit
> > c2bd04d8040a91fe2ee2e9fee1a6562ca9792249 (it's commit
> > 872ac8743cb400192a9fce4ba2d3ffd7bb309685 in the 2.6.26 series).
> > Reverting the commit seems to solve the problem here, I've been running
> > a 2.6.25.12 kernel without this commit for some weeks now.
> > In case it's important: I'm using an IPSec ESP transport with AES-256
> > and sha-256 auth. 
> 
> Sorry, I was skimping on memory and ended up calling a clobbered
> function pointer.
> 
> This patch should fix it.
> 
> crypto: authenc - Avoid using clobbered request pointer
> 
> Authenc works in two stages for encryption, it first encrypts and
> then computes an ICV.  The context memory of the request is used
> by both operations.  The problem is that when an asynchronous
> encryption completes, we will compute the ICV and then reread the
> context memory of the encryption to get the original request.
> 
> It just happens that we have a buffer of 16 bytes in front of the
> request pointer, so ICVs of 16 bytes (such as SHA1) do not trigger
> the bug.  However, any attempt to uses a larger ICV instantly kills
> the machine when the first asynchronous encryption is completed.
> 
> This patch fixes this by saving the request pointer before we start
> the ICV computation.
> 
> Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>

Acked-by: Matt LaPlante <kernel1@cyberdogtech.com>

Thanks for the quick fix!

-- 
Matt LaPlante

Re: Oops in authenc: 2.6.26.3

[Henrique de Moraes Holschuh]

Cc: stable@kernel.org ?

-- 
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh

Re: Oops in authenc: 2.6.26.3

[Herbert Xu]

On Thu, Aug 21, 2008 at 11:56:10AM -0300, Henrique de Moraes Holschuh wrote:
> Cc: stable@kernel.org ?

Once it's merged by Linus I'll push it to stable.

Thanks,
-- 
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Oops in authenc: 2.6.26.3

[Sascha Biberhofer]

Herbert Xu wrote:
> Sorry, I was skimping on memory and ended up calling a clobbered
> function pointer.
> 
> This patch should fix it.

Works like a charm, thanks for the quick fix. 

  Sascha Biberhofer