Re: v3.10-rc7 oops soon after boot

Re: v3.10-rc7 oops soon after boot

[Borislav Petkov]

This looks like ulog_timer in net/ipv4/netfilter/ipt_ULOG.c, judging by
your config.

Adding some more people to CC.

On Mon, Jun 24, 2013 at 02:49:34AM -0400, George Spelvin wrote:
> Trying to upgrade from 3.9.4 to 3.10-rc7, I encountered the following
> persistent oops right around the time that init enters runlevel 2.
> The exact timing fluctuates slightly; it doesn't appear to be a particular
> init script triggering it.
> 
> Trying to boot single-user, it occurred immediately after printing
> the password prompt.
> 
> I need this machine to be up during the day, at least, so a full bisect
> would cost me a chunk of a night's sleep, which I'd prefer to avoid
> if anyone has any ideas for abbreviating the process.
> 
> It's a quad-core Phenom with 8 GB of ECC RAM, which has a long history
> of hardware stability.
> 
> The local patches are minor tweaks to device drivers (some linuxpps
> patches I'm working on, and some timeout modifications to the r8169
> driver), and were all in the previous kernel.
> 
> The following is transcribed by hand, but I double-checked the call
> trace and code.
> 
> 
> [ ok ] Cleaning up temporary files....
> [info] Setting console screen modes and fonts
> BUG: unable to handle kernel NULL pointer dereference at        (null)
> IP: [<ffffffff813b5556>] ulog_timer+0x7/0x39
> PGD 216a2b067 PUF 20d853067 PMD 0
> Oops: 0000 [#1] SMP
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.0-rc7-00025-g94f52f4 #24
> Hardware name: MICR_STAR INTERNATIONAL CO.,LTD MS-7376/MS-7376, BIOS V1.7 01/13/2009
> task: ffffffff815c7400 ti: ffffffff815b6000 task.ti: ffffffff815b6000
> RIP: 0010:[<ffffffff813b5556>]  [<ffffffff813b5556>] ulog_timer+0x7/0x39
> RSP: 0018:ffff88021fc03ec0  EFLAGS: 00010206
> RAX: dead000000200200 RBX: 0000000000000100 RCX: ffff8802168ff098
> RDX: ffff88021fc03f00 RSI: 0000000000000000 RDI: 0000000000000000
> RBP: ffffffff815b7fd8 R08: 0000000371f3fc33 R09: ffffffff817465c0
> R10: 0000000000000000 R11: 0000000000000020 R12: 0000000000000000
> R13: ffffffff813b554f R14: 0000000000000000 R15: 0000000000000000
> FS:  0000000000000000(0000) GS:ffff88021fc00000(0000) knlGS:00000000f779e900
> CS: 0020 DS: 0000 ES: 0000 CR0: 000000008005003b
> CR2: 0000000000000000 CR3: 0000000211f93000 CR4: 00000000000007f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Stack:
>  0000000000000100 ffffffff815b7fd8 ffffffff813b554f ffffffff81031f87
>  ffffffff817465c0 ffff8802168ff098 ffff88021fc03f00 ffffffff810321e9
>  ffff88021fc03f00 ffff88021fc03f00 0000000000000101 ffffffff815bb088
> Call Trace:
>  <IRQ>
>  [<ffffffff813b554f>] ? ulog_send+0x76/0x76
>  [<ffffffff81031f87>] ? call_timer_fn.isra.29+0x1c/0x6f
>  [<ffffffff810321e9>] ? run_timer_softirq+0x19a/0x1c2
>  [<ffffffff8102de41>] ? __do_softirq+0xb9/0x171
>  [<ffffffff8102dfb7>] ? irq_exit+0x3a/0x7a
>  [<ffffffff8101969f>] ? smp_apic_timer_interrupt+0x72/0x7e
>  [<ffffffff8142ae8a>] ? apic_timer_interrupt+0x6a/0x70
>  <EOI>
>  [<ffffffff81007d24>] ? default_idle+0x2/0x3
>  [<ffffffff81007e1c>] ? amd_e400_idle+0xa7/0xc1
>  [<ffffffff8104f79e>] ? cpu_startup_entry+0x9c/0xec
>  [<ffffffff8164eb91>] ? start_kernel+0x2bd/0x2c8
>  [<ffffffff8164e6f7>] ? repair_env_string+0x54/0x54
> Code: 80 0d 00 00 e8 47 ae f9 ff c7 03 00 00 00 00 48 c7 43 10 00 00 00 00 48 c7 43 08 00 00 00 00 5b 5d 41 5c c3 41 54 49 89 fc 55 53 <8b> 07 48 89 fb 48 c1 e0 02 49 29 c4 49 8d ac 24 88 0d 00 00 48
> RIP  [<ffffffff813b5556>] ulog_timer+0x7/0x39
>  RSP <ffff88021fc03ec0>
> CR2: 0000000000000000
> ---[ end trace 0673bd6b61636416 ]---
> Kernel panic - not syncing: Fatal exception in interrupt
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: v3.10-rc7 oops soon after boot

[George Spelvin]

> This looks like ulog_timer in net/ipv4/netfilter/ipt_ULOG.c, judging by
> your config.

Ah, yes, it's a firewall/router machine with multiple interfaces.
Exploding based on packet arrivals could explain the variable timing
during init script processing.

Thank you!

Re: v3.10-rc7 oops soon after boot

[Gao feng]

[-- Attachment #1: Type: text/plain, Size: 405 bytes --]

On 06/24/2013 04:47 PM, George Spelvin wrote:
>> This looks like ulog_timer in net/ipv4/netfilter/ipt_ULOG.c, judging by
>> your config.
> 
> Ah, yes, it's a firewall/router machine with multiple interfaces.
> Exploding based on packet arrivals could explain the variable timing
> during init script processing.
> 

Hi George,

Please try the patch below,
I think this bug is introduced by me :(

Thanks!

[-- Attachment #2: 0001-netfilter-ipt_ULOG-fix-incorrect-setting-of-ulog-tim.patch --]
[-- Type: text/x-patch, Size: 1320 bytes --]

From f12c9178b881e0b21efd37b10a33059fd0544a40 Mon Sep 17 00:00:00 2001
From: Gao feng <gaofeng@cn.fujitsu.com>
Date: Mon, 24 Jun 2013 17:04:02 +0800
Subject: [PATCH] netfilter: ipt_ULOG: fix incorrect setting of ulog timer

The parameter of setup_timer should be &ulog->nlgroup[i].
the incorrect parameter will cause kernel panic in
ulog_timer.

Bug introducted in commit 355430671ad93546b34b4e91bdf720f3a704efa4
"netfilter: ipt_ULOG: add net namespace support for ipt_ULOG"

ebt_ULOG doesn't have this problem.

Reported-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com>
---
 net/ipv4/netfilter/ipt_ULOG.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/net/ipv4/netfilter/ipt_ULOG.c b/net/ipv4/netfilter/ipt_ULOG.c
index ff4b781..5200054 100644
--- a/net/ipv4/netfilter/ipt_ULOG.c
+++ b/net/ipv4/netfilter/ipt_ULOG.c
@@ -408,7 +408,8 @@ static int __net_init ulog_tg_net_init(struct net *net)
 	spin_lock_init(&ulog->lock);
 	/* initialize ulog_buffers */
 	for (i = 0; i < ULOG_MAXNLGROUPS; i++)
-		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer, i);
+		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer,
+			    (unsigned long)&ulog->nlgroup[i]);
 
 	ulog->nflognl = netlink_kernel_create(net, NETLINK_NFLOG, &cfg);
 	if (!ulog->nflognl)
-- 
1.8.1.4

Re: v3.10-rc7 oops soon after boot

[George Spelvin]

> Please try the patch below,
> I think this bug is introduced by me :(
> 
> Thanks!

Well, you changed it, but it still crashes.

It's now at del_timer+0x9/0x58, with the code being:

3f 00 55 53 48 89 fb 48 83 ec 10 <48> 83 3f 00 48 c7 47

The backtrace is
 <IRQ>
 ? ulog_send+0x21/0x76
 ? ulog_timer+0x2d/0x39
 ? ulog_send+0x76/0x76
 ? call_timer_fn.isra
 ? run_timer_softirq
 ? __do_softirq
 ? irq_exit
 ? do_IRQ
 ? common_interrupt
 <EOI>

(To give a faster response, I just transcribed a few key
sections.  Is that enough of a clue, or do you need more?)

Re: v3.10-rc7 oops soon after boot

[Gao feng]

[-- Attachment #1: Type: text/plain, Size: 969 bytes --]

On 06/24/2013 05:41 PM, George Spelvin wrote:
>> Please try the patch below,
>> I think this bug is introduced by me :(
>>
>> Thanks!
> 
> Well, you changed it, but it still crashes.
> 
> It's now at del_timer+0x9/0x58, with the code being:

This one should work.

Thanks for your quickly reply.

> 
> 3f 00 55 53 48 89 fb 48 83 ec 10 <48> 83 3f 00 48 c7 47
> 
> The backtrace is
>  <IRQ>
>  ? ulog_send+0x21/0x76
>  ? ulog_timer+0x2d/0x39
>  ? ulog_send+0x76/0x76
>  ? call_timer_fn.isra
>  ? run_timer_softirq
>  ? __do_softirq
>  ? irq_exit
>  ? do_IRQ
>  ? common_interrupt
>  <EOI>
> 
> (To give a faster response, I just transcribed a few key
> sections.  Is that enough of a clue, or do you need more?)
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 


[-- Attachment #2: 0001-netfilter-ipt_ULOG-fix-incorrect-setting-of-ulog-tim.patch --]
[-- Type: text/x-patch, Size: 1626 bytes --]

From f22cb6a9a52497364605c25930ba470ee180ca58 Mon Sep 17 00:00:00 2001
From: Gao feng <gaofeng@cn.fujitsu.com>
Date: Mon, 24 Jun 2013 17:04:02 +0800
Subject: [PATCH] netfilter: ipt_ULOG: fix incorrect setting of ulog timer

The parameter of setup_timer should be &ulog->nlgroup[i].
the incorrect parameter will cause kernel panic in
ulog_timer.

Bug introducted in commit 355430671ad93546b34b4e91bdf720f3a704efa4
"netfilter: ipt_ULOG: add net namespace support for ipt_ULOG"

ebt_ULOG doesn't have this problem.

Reported-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com>
---
 net/ipv4/netfilter/ipt_ULOG.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/net/ipv4/netfilter/ipt_ULOG.c b/net/ipv4/netfilter/ipt_ULOG.c
index ff4b781..1345c20 100644
--- a/net/ipv4/netfilter/ipt_ULOG.c
+++ b/net/ipv4/netfilter/ipt_ULOG.c
@@ -133,7 +133,7 @@ static void ulog_timer(unsigned long data)
 	/* lock to protect against somebody modifying our structure
 	 * from ipt_ulog_target at the same time */
 	spin_lock_bh(&ulog->lock);
-	ulog_send(ulog, data);
+	ulog_send(ulog, *(unsigned int *)data);
 	spin_unlock_bh(&ulog->lock);
 }
 
@@ -408,7 +408,8 @@ static int __net_init ulog_tg_net_init(struct net *net)
 	spin_lock_init(&ulog->lock);
 	/* initialize ulog_buffers */
 	for (i = 0; i < ULOG_MAXNLGROUPS; i++)
-		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer, i);
+		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer,
+			    (unsigned long)&ulog->nlgroup[i]);
 
 	ulog->nflognl = netlink_kernel_create(net, NETLINK_NFLOG, &cfg);
 	if (!ulog->nflognl)
-- 
1.8.1.4

Re: v3.10-rc7 oops soon after boot

[George Spelvin]

Hooray!  This one seems to work.

Thank you very much for the quick response!

Tested-by: George Spelvin <linux@horizon.com>

Re: v3.10-rc7 oops soon after boot

[Pablo Neira Ayuso]

[-- Attachment #1: Type: text/plain, Size: 612 bytes --]

On Mon, Jun 24, 2013 at 05:52:08PM +0800, Gao feng wrote:
> On 06/24/2013 05:41 PM, George Spelvin wrote:
> >> Please try the patch below,
> >> I think this bug is introduced by me :(
> >>
> >> Thanks!
> > 
> > Well, you changed it, but it still crashes.
> > 
> > It's now at del_timer+0x9/0x58, with the code being:
> 
> This one should work.
> 
> Thanks for your quickly reply.

I managed to crash it using a group != 1 (default) and enabling
batching:

iptables -I INPUT -j ULOG --ulog-nlgroup 2 --ulog-qthresh 50

It seems ulog->nlgroup[i] is always left unset.

I have mangled your patch, find it attached.

[-- Attachment #2: 0001-netfilter-ipt_ULOG-fix-incorrect-setting-of-ulog-tim.patch --]
[-- Type: text/x-diff, Size: 2155 bytes --]

From 79da8ea854c06ca9320decbb2f9d2e8d25b2f7e5 Mon Sep 17 00:00:00 2001
From: Gao feng <gaofeng@cn.fujitsu.com>
Date: Mon, 24 Jun 2013 17:04:02 +0800
Subject: [PATCH] netfilter: ipt_ULOG: fix incorrect setting of ulog timer

The parameter of setup_timer should be &ulog->nlgroup[i].
the incorrect parameter will cause kernel panic in
ulog_timer.

Bug introducted in commit 355430671ad93546b34b4e91bdf720f3a704efa4
"netfilter: ipt_ULOG: add net namespace support for ipt_ULOG"

ebt_ULOG doesn't have this problem.

Reported-by: Borislav Petkov <bp@alien8.de>
Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com>
Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>
---
 net/ipv4/netfilter/ipt_ULOG.c |   12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/net/ipv4/netfilter/ipt_ULOG.c b/net/ipv4/netfilter/ipt_ULOG.c
index ff4b781..32b0e97 100644
--- a/net/ipv4/netfilter/ipt_ULOG.c
+++ b/net/ipv4/netfilter/ipt_ULOG.c
@@ -125,15 +125,16 @@ static void ulog_send(struct ulog_net *ulog, unsigned int nlgroupnum)
 /* timer function to flush queue in flushtimeout time */
 static void ulog_timer(unsigned long data)
 {
+	unsigned int groupnum = *((unsigned int *)data);
 	struct ulog_net *ulog = container_of((void *)data,
 					     struct ulog_net,
-					     nlgroup[*(unsigned int *)data]);
+					     nlgroup[groupnum]);
 	pr_debug("timer function called, calling ulog_send\n");
 
 	/* lock to protect against somebody modifying our structure
 	 * from ipt_ulog_target at the same time */
 	spin_lock_bh(&ulog->lock);
-	ulog_send(ulog, data);
+	ulog_send(ulog, groupnum);
 	spin_unlock_bh(&ulog->lock);
 }
 
@@ -407,8 +408,11 @@ static int __net_init ulog_tg_net_init(struct net *net)
 
 	spin_lock_init(&ulog->lock);
 	/* initialize ulog_buffers */
-	for (i = 0; i < ULOG_MAXNLGROUPS; i++)
-		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer, i);
+	for (i = 0; i < ULOG_MAXNLGROUPS; i++) {
+		ulog->nlgroup[i] = i;
+		setup_timer(&ulog->ulog_buffers[i].timer, ulog_timer,
+			    (unsigned long)&ulog->nlgroup[i]);
+	}
 
 	ulog->nflognl = netlink_kernel_create(net, NETLINK_NFLOG, &cfg);
 	if (!ulog->nflognl)
-- 
1.7.10.4

Re: v3.10-rc7 oops soon after boot

[Pablo Neira Ayuso]

On Mon, Jun 24, 2013 at 01:34:41PM +0200, Pablo Neira Ayuso wrote:
> On Mon, Jun 24, 2013 at 05:52:08PM +0800, Gao feng wrote:
> > On 06/24/2013 05:41 PM, George Spelvin wrote:
> > >> Please try the patch below,
> > >> I think this bug is introduced by me :(
> > >>
> > >> Thanks!
> > > 
> > > Well, you changed it, but it still crashes.
> > > 
> > > It's now at del_timer+0x9/0x58, with the code being:
> > 
> > This one should work.
> > 
> > Thanks for your quickly reply.
> 
> I managed to crash it using a group != 1 (default) and enabling
> batching:
> 
> iptables -I INPUT -j ULOG --ulog-nlgroup 2 --ulog-qthresh 50
> 
> It seems ulog->nlgroup[i] is always left unset.
> 
> I have mangled your patch, find it attached.

> From 79da8ea854c06ca9320decbb2f9d2e8d25b2f7e5 Mon Sep 17 00:00:00 2001
> From: Gao feng <gaofeng@cn.fujitsu.com>
> Date: Mon, 24 Jun 2013 17:04:02 +0800
> Subject: [PATCH] netfilter: ipt_ULOG: fix incorrect setting of ulog timer
> 
> The parameter of setup_timer should be &ulog->nlgroup[i].
> the incorrect parameter will cause kernel panic in
> ulog_timer.
> 
> Bug introducted in commit 355430671ad93546b34b4e91bdf720f3a704efa4
> "netfilter: ipt_ULOG: add net namespace support for ipt_ULOG"
> 
> ebt_ULOG doesn't have this problem.
> 
> Reported-by: Borislav Petkov <bp@alien8.de>
> Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com>
> Signed-off-by: Pablo Neira Ayuso <pablo@netfilter.org>

I have applied the patch that I sent you.

Re: v3.10-rc7 oops soon after boot

[Borislav Petkov]

On Mon, Jun 24, 2013 at 05:10:31PM +0200, Pablo Neira Ayuso wrote:
> > I have mangled your patch, find it attached.
> 
> > From 79da8ea854c06ca9320decbb2f9d2e8d25b2f7e5 Mon Sep 17 00:00:00 2001
> > From: Gao feng <gaofeng@cn.fujitsu.com>
> > Date: Mon, 24 Jun 2013 17:04:02 +0800
> > Subject: [PATCH] netfilter: ipt_ULOG: fix incorrect setting of ulog timer
> > 
> > The parameter of setup_timer should be &ulog->nlgroup[i].
> > the incorrect parameter will cause kernel panic in
> > ulog_timer.
> > 
> > Bug introducted in commit 355430671ad93546b34b4e91bdf720f3a704efa4
> > "netfilter: ipt_ULOG: add net namespace support for ipt_ULOG"
> > 
> > ebt_ULOG doesn't have this problem.
> > 
> > Reported-by: Borislav Petkov <bp@alien8.de>

This should be:

Reported-by: George Spelvin <linux@horizon.com>

I only connected the dots...

:-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: v3.10-rc7 oops soon after boot

[George Spelvin]

>> Reported-by: Borislav Petkov <bp@alien8.de>

> This should be:
>
> Reported-by: George Spelvin <linux@horizon.com>
>
> I only connected the dots...

Well, you did a whole lot more than me!  I just lobbed a "d'oh, it
crashes" into the seething ocean of lkml.  (Admittedly, I had reason
to act fast: we're very close to release.)

You figured out what subsystem was at fault and got the right people
involved.  Definitely a valuable contribution.

Me, personally, I don't give a flying f*** about such credit; I had
an itch and was trolling for someone to scratch it.

So feel free to take Reported-by (you are the one who reported it *to
someone who could fix it*), Triaged-by, or whatever.

Re: v3.10-rc7 oops soon after boot

[Borislav Petkov]

On Mon, Jun 24, 2013 at 06:17:08PM -0400, George Spelvin wrote:
> >> Reported-by: Borislav Petkov <bp@alien8.de>
> 
> > This should be:
> >
> > Reported-by: George Spelvin <linux@horizon.com>
> >
> > I only connected the dots...
> 
> Well, you did a whole lot more than me!  I just lobbed a "d'oh, it
> crashes" into the seething ocean of lkml.  (Admittedly, I had reason
> to act fast: we're very close to release.)
> 
> You figured out what subsystem was at fault and got the right people
> involved.  Definitely a valuable contribution.
> 
> Me, personally, I don't give a flying f*** about such credit; I had
> an itch and was trolling for someone to scratch it.
> 
> So feel free to take Reported-by (you are the one who reported it *to
> someone who could fix it*), Triaged-by, or whatever.

I was basically correcting how Reported-by is used - for the bug
reporter.

And yes, it is very important for us (even if you specifically don't
give a rat's rear end :-)) to credit the people - with their permission,
of course - who report bugs. Btw, I assume it is obvious why we need all
those good bug reports...

So I'm perfectly fine with you not giving flying f*ck about crediting as
long as you keep testing -rcs and reporting issues.

:-)

Thanks!

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: v3.10-rc7 oops soon after boot

[Gao feng]

On 06/25/2013 06:17 AM, George Spelvin wrote:
>>> Reported-by: Borislav Petkov <bp@alien8.de>
> 
>> This should be:
>>
>> Reported-by: George Spelvin <linux@horizon.com>
>>
>> I only connected the dots...
> 
> Well, you did a whole lot more than me!  I just lobbed a "d'oh, it
> crashes" into the seething ocean of lkml.  (Admittedly, I had reason
> to act fast: we're very close to release.)
> 
> You figured out what subsystem was at fault and got the right people
> involved.  Definitely a valuable contribution.
> 
> Me, personally, I don't give a flying f*** about such credit; I had
> an itch and was trolling for someone to scratch it.
> 
> So feel free to take Reported-by (you are the one who reported it *to
> someone who could fix it*), Triaged-by, or whatever.

It's my mistake, the reported-by should be you.
of course we should thank Borislav, I didn't notice this bug report
mail until he forwarded it.

Thanks all you guys! Sorry for my mistake..

> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: v3.10-rc7 oops soon after boot

[Gao feng]

On 06/24/2013 07:34 PM, Pablo Neira Ayuso wrote:
> On Mon, Jun 24, 2013 at 05:52:08PM +0800, Gao feng wrote:
>> On 06/24/2013 05:41 PM, George Spelvin wrote:
>>>> Please try the patch below,
>>>> I think this bug is introduced by me :(
>>>>
>>>> Thanks!
>>>
>>> Well, you changed it, but it still crashes.
>>>
>>> It's now at del_timer+0x9/0x58, with the code being:
>>
>> This one should work.
>>
>> Thanks for your quickly reply.
> 
> I managed to crash it using a group != 1 (default) and enabling
> batching:
> 
> iptables -I INPUT -j ULOG --ulog-nlgroup 2 --ulog-qthresh 50
> 
> It seems ulog->nlgroup[i] is always left unset.

Yes, we should set ulog->nlgroup..

> 
> I have mangled your patch, find it attached.
> 

Thanks!

Re: v3.10-rc7 oops soon after boot

[George Spelvin]

> I have mangled your patch, find it attached.

I'm sure it's not a surprise, but just for the record, I can now confirm
that this fixes my crash, too.