Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Mike Snitzer]

On Mon, Oct 31 2011 at  9:40am -0400,
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> On Mon, Oct 31, 2011 at 09:21:58AM -0400, Mike Snitzer wrote:
> > > > It _looks_ like we do not hit the BUG_ON() that. This time we get this instead:
> > > > 
> > > > [ 4024.937870] Unable to handle kernel pointer dereference at virtual kernel address 000003e004d41000
> > > > [ 4024.937886] Oops: 0011 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> > > > [ 4024.937899] Modules linked in: dm_round_robin sunrpc ipv6 qeth_l2 binfmt_misc dm_multipath scsi_dh dm_mod qeth ccwgroup [las
> > > > t unloaded: scsi_wait_scan]
> > > > [ 4024.937925] CPU: 1 Not tainted 3.0.7-50.x.20111021-s390xdefault #1
> > > > [ 4024.937930] Process ksoftirqd/1 (pid: 1942, task: 0000000079c6c750, ksp: 0000000073adfc50)
> > > > [ 4024.937936] Krnl PSW : 0704000180000000 000003e00126263a (dm_softirq_done+0x72/0x140 [dm_mod])
> > > > [ 4024.937959]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:0 PM:0 EA:3
> > > > [ 4024.937966] Krnl GPRS: 000000007b9156b0 000003e004d41100 000000000e14b600 000000000000006d
> > > > [ 4024.937971]            00000000715332b0 000000000c140ce8 000000000090d2ef 0000000000000005
> > > > [ 4024.937977]            0000000000000001 0000000000000101 000000000c140d00 0000000000000000
> > > > [ 4024.937983]            000003e001260000 000003e00126f098 0000000073adfd08 0000000073adfcb8
> > > > [ 4024.938001] Krnl Code: 000003e00126262a: f0a0000407f1        srp     4(11,%r0),2033,0
> > > > [ 4024.938009]            000003e001262630: e31050080004        lg      %r1,8(%r5)
> > > > [ 4024.938017]            000003e001262636: 58b05180            l       %r11,384(%r5)
> > > > [ 4024.938024]           >000003e00126263a: e31010080004        lg      %r1,8(%r1)
> > > > [ 4024.938031]            000003e001262640: e31010500004        lg      %r1,80(%r1)
> > > > [ 4024.938038]            000003e001262646: b9020011            ltgr    %r1,%r1
> > > > [ 4024.938045]            000003e00126264a: a784ffdf            brc     8,3e001262608
> > > > [ 4024.938053]            000003e00126264e: e32050080004        lg      %r2,8(%r5)
> > > > [ 4024.938060] Call Trace:
> > > > [ 4024.938063] ([<070000000040716c>] 0x70000000040716c)
> > > > [ 4024.938069]  [<000000000040d29c>] blk_done_softirq+0xd4/0xf0
> > > > [ 4024.938080]  [<00000000001587c2>] __do_softirq+0xda/0x398
> > > > [ 4024.938088]  [<0000000000158ba0>] run_ksoftirqd+0x120/0x23c
> > > > [ 4024.938095]  [<000000000017c2aa>] kthread+0xa6/0xb0
> > > > [ 4024.938102]  [<000000000061970e>] kernel_thread_starter+0x6/0xc
> > > > [ 4024.938112]  [<0000000000619708>] kernel_thread_starter+0x0/0xc
> > > > [ 4024.938118] INFO: lockdep is turned off.
> > > > [ 4024.938121] Last Breaking-Event-Address:
> > > > [ 4024.938124]  [<000003e001262600>] dm_softirq_done+0x38/0x140 [dm_mod]
> > > > [ 4024.938135]  
> > > > [ 4024.938139] Kernel panic - not syncing: Fatal exception in interrupt
> > > > [ 4024.938144] CPU: 1 Tainted: G      D     3.0.7-50.x.20111021-s390xdefault #1
> > > > [ 4024.938150] Process ksoftirqd/1 (pid: 1942, task: 0000000079c6c750, ksp: 0000000073adfc50)
> > > > [ 4024.938155] 0000000073adf958 0000000073adf8d8 0000000000000002 0000000000000000 
> > > > [ 4024.938164]        0000000073adf978 0000000073adf8f0 0000000073adf8f0 000000000061386a 
> > > > [ 4024.938174]        0000000000000000 0000000000000000 0000000000000005 0000000000100ec6 
> > > > [ 4024.938184]        000000000000000d 000000000000000c 0000000073adf940 0000000000000000 
> > > > [ 4024.938194]        0000000000000000 0000000000100a18 0000000073adf8d8 0000000073adf918 
> > > > [ 4024.938205] Call Trace:
> > > > [ 4024.938208] ([<0000000000100926>] show_trace+0xee/0x144)
> > > > [ 4024.938216]  [<0000000000613694>] panic+0xb0/0x234
> > > > [ 4024.938224]  [<0000000000100ec6>] die+0x15a/0x168
> > > > [ 4024.938230]  [<000000000011fb9e>] do_no_context+0xba/0xf8
> > > > [ 4024.938306]  [<000000000061c074>] do_dat_exception+0x378/0x3e4
> > > > [ 4024.938314]  [<0000000000619e02>] pgm_exit+0x0/0x4
> > > > [ 4024.938319]  [<000003e00126263a>] dm_softirq_done+0x72/0x140 [dm_mod]
> > > > [ 4024.938329] ([<070000000040716c>] 0x70000000040716c)
> > > > [ 4024.938334]  [<000000000040d29c>] blk_done_softirq+0xd4/0xf0
> > > > [ 4024.938341]  [<00000000001587c2>] __do_softirq+0xda/0x398
> > > > [ 4024.938347]  [<0000000000158ba0>] run_ksoftirqd+0x120/0x23c
> > > > [ 4024.938354]  [<000000000017c2aa>] kthread+0xa6/0xb0
> > > > [ 4024.938360]  [<000000000061970e>] kernel_thread_starter+0x6/0xc
> > > > [ 4024.938366]  [<0000000000619708>] kernel_thread_starter+0x0/0xc
> > > > [ 4024.938373] INFO: lockdep is turned off.
> > > > 
> > > > So we thought we might as well upgrade to 3.1 but immediately got a
> > > > 
> > > > kernel BUG at block/blk-flush.c:323!
> > > > 
> > > > which was handled here https://lkml.org/lkml/2011/10/4/105 and
> > > > here https://lkml.org/lkml/2011/10/12/408 .
> > > > 
> > > > But no patches for that one went upstream AFAICS.
> > > 
> > > Well, all I can say is "hm".  You put only a BUG_ON() in the code, which
> > > wasn't triggered, but now we get a completely different oops.  However,
> > > I think it does point to the dm barrier handling code.  Can you turn off
> > > barriers and see if all oopses go away?
> > 
> > There are two 3.1-stable fixes from Jeff Moyer that Jens staged for
> > Linus to pick up (but seems Jens hasn't sent his 3.2 pull to Linus yet):
> > 
> > http://git.kernel.dk/?p=linux-block.git;a=commit;h=8f02b3a09b1b7d2a4d24b8cd7008f2a441f19a14
> > http://git.kernel.dk/?p=linux-block.git;a=commit;h=f26d8f0562da76731cb049943a0e9d9fa81d946a
> 
> Those two fixes would only address the "kernel BUG at block/blk-flush.c:323!" but not the
> crash report above, right?

Right.

> Since looking at the changelog they refer to a patch that went in with 3.1-rc1 while the
> crash report above is with 3.0.7. Oh well...

Good data point.  This is the second request-based DM report I've seen
now with 3.0 (first was with fedora on btrfs and request-based DM).

Will look closer but it should be noted that DM didn't change
significantly in 3.0.  So it is likely a lingering oversight from the
block changes introduced for onstack plugging (from 2.6.39) or some
other change.

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Mike Snitzer]

On Mon, Nov 07 2011 at 10:36am -0500,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Nov 07 2011 at  6:30am -0500,
> Jun'ichi Nomura <j-nomura@ce.jp.nec.com> wrote:
> 
> > On 11/04/11 18:19, Heiko Carstens wrote:
> > > It's the s390 only zfcp device driver.
> > > 
> > > FWIW, yet another use-after-free crash, this time however in multipath_end_io:
> > > 
> > > [96875.870593] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000
> > > [96875.870602] Oops: 0038 [#1]
> > > [96875.870674] PREEMPT SMP DEBUG_PAGEALLOC
> > > [96875.870683] Modules linked in: dm_round_robin sunrpc ipv6 qeth_l2 binfmt_misc dm_multipath scsi_dh dm_mod qeth ccwgroup [la\
> > > st unloaded: scsi_wait_scan]
> > > [96875.870722] CPU: 2 Tainted: G        W   3.0.7-50.x.20111024-s390xdefault #1
> > > [96875.870728] Process udevd (pid: 36697, task: 0000000072c8a3a8, ksp: 0000000057c43868)
> > > [96875.870732] Krnl PSW : 0704200180000000 000003e001347138 (multipath_end_io+0x50/0x140 [dm_multipath])
> > > [96875.870746]            R:0 T:1 IO:1 EX:1 Key:0 M:1 W:0 P:0 AS:0 CC:2 PM:0 EA:3
> > > [96875.870751] Krnl GPRS: 0000000000000000 000003e000000000 6b6b6b6b6b6b6b6b 00000000717ab940
> > > [96875.870755]            0000000000000000 00000000717abab0 0000000000000002 0700000000000008
> > > [96875.870759]            0000000000000002 0000000000000000 0000000058dd37a8 000000006f845478
> > > [96875.870764]            000003e0012e1000 000000005613d1f0 000000007a737bf0 000000007a737ba0
> > > [96875.870768] Krnl Code: 000003e00134712a: b90200dd            ltgr %r13,%r13
> > > [96875.870793]            000003e00134712e: a7840017            brc 8,3e00134715c
> > > [96875.870800]            000003e001347132: e320d0100004        lg %r2,16(%r13)
> > > [96875.870809]           >000003e001347138: e31020180004        lg %r1,24(%r2)
> > > [96875.870818]            000003e00134713e: e31010580004        lg %r1,88(%r1)
> > > [96875.870827]            000003e001347144: b9020011            ltgr %r1,%r1
> > > [96875.870835]            000003e001347148: a784000a            brc 8,3e00134715c
> > > [96875.870841]            000003e00134714c: 41202018            la %r2,24(%r2)
> > > [96875.870889] Call Trace:
> > > [96875.870892] ([<0700000000000008>] 0x700000000000008)
> > > [96875.870897]  [<000003e0012e3662>] dm_softirq_done+0x9a/0x140 [dm_mod]
> > > [96875.870915]  [<000000000040d29c>] blk_done_softirq+0xd4/0xf0
> > > [96875.870925]  [<00000000001587c2>] __do_softirq+0xda/0x398
> > > [96875.870932]  [<000000000010f47e>] do_softirq+0xe2/0xe8
> > > [96875.870940]  [<0000000000158e2c>] irq_exit+0xc8/0xcc
> > > [96875.870945]  [<00000000004ceb48>] do_IRQ+0x910/0x1bfc
> > > [96875.870953]  [<000000000061a164>] io_return+0x0/0x16
> > > [96875.870961]  [<000000000019c84e>] lock_acquire+0xd2/0x204
> > > [96875.870969] ([<000000000019c836>] lock_acquire+0xba/0x204)
> > > [96875.870974]  [<0000000000615f8e>] mutex_lock_killable_nested+0x92/0x520
> > > [96875.870983]  [<0000000000292796>] vfs_readdir+0x8a/0xe4
> > > [96875.870992]  [<00000000002928e0>] SyS_getdents+0x60/0xe8
> > > [96875.870999]  [<0000000000619af2>] sysc_noemu+0x16/0x1c
> > > [96875.871024]  [<000003fffd1ec83e>] 0x3fffd1ec83e
> > > [96875.871028] INFO: lockdep is turned off.
> > > [96875.871031] Last Breaking-Event-Address:
> > > [96875.871037]  [<000003e0012e3660>] dm_softirq_done+0x98/0x140 [dm_mod]
> > > 
> > > static int multipath_end_io(struct dm_target *ti, struct request *clone,
> > > 			    int error, union map_info *map_context)
> > > {
> > > 	struct multipath *m = ti->private;
> > > 	struct dm_mpath_io *mpio = map_context->ptr;
> > > 	struct pgpath *pgpath = mpio->pgpath;
> > > 	struct path_selector *ps;
> > > 	int r;
> > > 
> > > 	r  = do_end_io(m, clone, error, mpio);
> > > 	if (pgpath) {
> > > 		ps = &pgpath->pg->ps; <--- crashes here
> > > 		if (ps->type->end_io)
> > > 			ps->type->end_io(ps, &pgpath->path, mpio->nr_bytes);
> > > 	}
> > > 	mempool_free(mpio, m->mpio_pool);
> > > 
> > > 	return r;
> > > }
> > > 
> > > It crashes when trying to derefence pgpath, which was freed. Since we have
> > > SLUB debugging turned on the freed object tells us that it was allocated
> > > via a call to multipath_ctr() and freed via a call to free_priority_group().
> > 
> > struct pgpath is freed before dm_target when tearing down dm table.
> > So if the problematic completion was being done after freeing pgpath
> > but before freeing dm_target, crash would look like that
> > and what's happening seems the same for these dm crashes:
> > dm table was somehow destroyed while I/O was in-flight.
> 
> Could be the block layer's onstack plugging changes are at the heart of
> this.
> 
> I voiced onstack plugging concerns relative to DM some time ago
> (https://lkml.org/lkml/2011/3/9/450) but somehow convinced myself DM was
> fine to no longer need dm_table_unplug_all() etc.  Unfortunately I
> cannot recall _why_ I felt that was the case.
> 
> So DM needs further review relative to block's onstack plugging changes
> and DM IO completion.

dm_suspend is performed as part of the DM table reload that is being
done my multipathd during path failure.  Seems DM no longer insures
inflight requests have finished during dm_suspend().

Before onstack plugging (< 2.6.39):
dm_suspend() -> dm_wait_for_completion() -> dm_unplug_all() -> dm_table_unplug_all()

After onstack plugging (>= 2.6.39, commit 7eaceaccab5f40bb):
dm_suspend's call to dm_wait_for_completion() no longer unplugs IO
(dm_unplug_all and dm_table_unplug_all were removed without introducing
a clear equivalent).

Mike

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Mike Snitzer]

On Mon, Nov 07 2011 at 12:10pm -0500,
Mike Snitzer <snitzer@redhat.com> wrote:

> On Mon, Nov 07 2011 at 10:36am -0500,
> Mike Snitzer <snitzer@redhat.com> wrote:
> > Could be the block layer's onstack plugging changes are at the heart of
> > this.
> > 
> > I voiced onstack plugging concerns relative to DM some time ago
> > (https://lkml.org/lkml/2011/3/9/450) but somehow convinced myself DM was
> > fine to no longer need dm_table_unplug_all() etc.  Unfortunately I
> > cannot recall _why_ I felt that was the case.
> > 
> > So DM needs further review relative to block's onstack plugging changes
> > and DM IO completion.
> 
> dm_suspend is performed as part of the DM table reload that is being
> done my multipathd during path failure.  Seems DM no longer insures
> inflight requests have finished during dm_suspend().
> 
> Before onstack plugging (< 2.6.39):
> dm_suspend() -> dm_wait_for_completion() -> dm_unplug_all() -> dm_table_unplug_all()
> 
> After onstack plugging (>= 2.6.39, commit 7eaceaccab5f40bb):
> dm_suspend's call to dm_wait_for_completion() no longer unplugs IO
> (dm_unplug_all and dm_table_unplug_all were removed without introducing
> a clear equivalent).

If a missing unplug were a concern then dm_wait_for_completion()'s check
for inflight IO would cause dm_suspend() to hang.

So the onstack plugging changes are unlikely to be the reason for this.

Mike

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Mike Snitzer]

On Tue, Nov 29 2011 at  7:00am -0500,
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> > > > Hmm. Just to be on the safe side, could you try this one:
> > > > 
> > > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> > > > index 5e0090e..e6fad46 100644
> > > > --- a/drivers/md/dm-mpath.c
> > > > +++ b/drivers/md/dm-mpath.c
> > > > @@ -920,8 +920,10 @@ static int multipath_map(struct dm_target *ti,
> > > > struct reque
> > > > st *clone,
> > > >         map_context->ptr = mpio;
> > > >         clone->cmd_flags |= REQ_FAILFAST_TRANSPORT;
> > > >         r = map_io(m, clone, mpio, 0);
> > > > -       if (r < 0 || r == DM_MAPIO_REQUEUE)
> > > > +       if (r < 0 || r == DM_MAPIO_REQUEUE) {
> > > >                 mempool_free(mpio, m->mpio_pool);
> > > > +               map_context->ptr = NULL;
> > > > +       }
> > > > 
> > > >         return r;
> > > >  }
> > > 
> > > With your patch we haven't been able to reproduce the kernel crash until now.
> > > Now we "only" run into I/O stalls, which before your patch we also did. But
> > > repeatedly rebooting and retrying and ignoring the I/O stalls always lead to
> > > a crash.
> > > Gonzalo will run a couple of extra rounds so we can have a feeling if at least
> > > one of the bugs could be fixed with your patch ;)
> > 
> > Hi,
> > 
> > Any update after further testing with Hannes' patch?
> 
> Sorry for the late update, our internal IBM IMAP servers have been down
> for nearly a week :/
> 
> So, we were unable to reproduce the original bug with the patch applied
> during various runs.

OK, so it seems to be a benenficial change (and obviously correct to
me).  Hannes, care to formally post your fix to dm-devel so we can get
it in 3.2-rc?

> However, we ran into this one instead, which is yet another use-after-free bug
> (I need to double check, but I'm quite sure that a freed struct scsi_cmnd
> caused this).

OK, yeah something is causing poisoned (POISON_FREE) memory to be used.

> [ 4906.683654] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000

...

> Gonzalo also tried 2.6.38.8 as suggested and ran into this one:
> 
> [  292.877936] ------------[ cut here ]------------
> [  292.877939] Kernel BUG at 6b6b6b6b6b6b6b6d [verbose debug info unavailable]

Again, more poison.

Seems this test is causing us to fall on our face no matter what.
Likely, best to leave this 2.6.38 blk_unplug crash to one side and
continue focusing on latest upstream.

Mike

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Hannes Reinecke]

On 11/29/2011 09:18 PM, Mike Snitzer wrote:
> On Tue, Nov 29 2011 at  7:00am -0500,
> Heiko Carstens <heiko.carstens@de.ibm.com> wrote:
> 
>>>>> Hmm. Just to be on the safe side, could you try this one:
>>>>>
>>>>> diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
>>>>> index 5e0090e..e6fad46 100644
>>>>> --- a/drivers/md/dm-mpath.c
>>>>> +++ b/drivers/md/dm-mpath.c
>>>>> @@ -920,8 +920,10 @@ static int multipath_map(struct dm_target *ti,
>>>>> struct reque
>>>>> st *clone,
>>>>>         map_context->ptr = mpio;
>>>>>         clone->cmd_flags |= REQ_FAILFAST_TRANSPORT;
>>>>>         r = map_io(m, clone, mpio, 0);
>>>>> -       if (r < 0 || r == DM_MAPIO_REQUEUE)
>>>>> +       if (r < 0 || r == DM_MAPIO_REQUEUE) {
>>>>>                 mempool_free(mpio, m->mpio_pool);
>>>>> +               map_context->ptr = NULL;
>>>>> +       }
>>>>>
>>>>>         return r;
>>>>>  }
>>>>
>>>> With your patch we haven't been able to reproduce the kernel crash until now.
>>>> Now we "only" run into I/O stalls, which before your patch we also did. But
>>>> repeatedly rebooting and retrying and ignoring the I/O stalls always lead to
>>>> a crash.
>>>> Gonzalo will run a couple of extra rounds so we can have a feeling if at least
>>>> one of the bugs could be fixed with your patch ;)
>>>
>>> Hi,
>>>
>>> Any update after further testing with Hannes' patch?
>>
>> Sorry for the late update, our internal IBM IMAP servers have been down
>> for nearly a week :/
>>
>> So, we were unable to reproduce the original bug with the patch applied
>> during various runs.
> 
> OK, so it seems to be a benenficial change (and obviously correct to
> me).  Hannes, care to formally post your fix to dm-devel so we can get
> it in 3.2-rc?
> 
Yep, will do.

Cheers,

Hannes
-- 
Dr. Hannes Reinecke		      zSeries & Storage
hare@suse.de			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: J. Hawn, J. Guild, F. Imendörffer, HRB 16746 (AG Nürnberg)
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Heiko Carstens]

On Tue, Nov 29, 2011 at 03:18:03PM -0500, Mike Snitzer wrote:
> On Tue, Nov 29 2011 at  7:00am -0500,
> Heiko Carstens <heiko.carstens@de.ibm.com> wrote:
> > [ 4906.683654] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000
> 
> ...
> 
> > Gonzalo also tried 2.6.38.8 as suggested and ran into this one:
> > 
> > [  292.877936] ------------[ cut here ]------------
> > [  292.877939] Kernel BUG at 6b6b6b6b6b6b6b6d [verbose debug info unavailable]
> 
> Again, more poison.
> 
> Seems this test is causing us to fall on our face no matter what.
> Likely, best to leave this 2.6.38 blk_unplug crash to one side and
> continue focusing on latest upstream.

Sorry again, for taking so long to come back. This time however with good news:

With 3.2.0-rc4.00255.g77a7300 we were unable to reproduce any I/O stall or
user-after-free bugs even after nearly 3000 test iterations.

The only patches on top we have are:

two patches from Hannes:
http://www.spinics.net/lists/linux-scsi/msg55112.html
http://www.spinics.net/lists/linux-scsi/msg55413.html

and the patch below from Steffen:

Btw. James, any chance to get this one upstream soon? It should be in your
queue for quite some time already, IIRC.

Subject: [PATCH] zfcp: return early from slave_destroy if slave_alloc returned early

From: Steffen Maier <maier@linux.vnet.ibm.com>

zfcp_scsi_slave_destroy erroneously always tried to finish its task
even if the corresponding previous zfcp_scsi_slave_alloc returned
early. This can lead to kernel page faults on accessing uninitialized
fields of struct zfcp_scsi_dev in zfcp_erp_lun_shutdown_wait. Take the
port field of the struct to determine if slave_alloc returned early.

This zfcp bug is exposed by 4e6c82b (in turn fixing f7c9c6b to be
compatible with 21208ae) which can call slave_destroy for a
corresponding previous slave_alloc that did not finish.

This patch is based on James Bottomley's fix suggestion in
http://www.spinics.net/lists/linux-scsi/msg55449.html.

Signed-off-by: Steffen Maier <maier@linux.vnet.ibm.com>
Cc: <stable@kernel.org> #2.6.38+
---

 drivers/s390/scsi/zfcp_scsi.c |    4 ++++
 1 file changed, 4 insertions(+)

diff -urpN linux-2.6/drivers/s390/scsi/zfcp_scsi.c linux-2.6-patched/drivers/s390/scsi/zfcp_scsi.c
--- linux-2.6/drivers/s390/scsi/zfcp_scsi.c	2011-12-01 13:08:32.000000000 +0100
+++ linux-2.6-patched/drivers/s390/scsi/zfcp_scsi.c	2011-12-01 13:08:52.000000000 +0100
@@ -55,6 +55,10 @@ static void zfcp_scsi_slave_destroy(stru
 {
 	struct zfcp_scsi_dev *zfcp_sdev = sdev_to_zfcp(sdev);
 
+	/* if previous slave_alloc returned early, there is nothing to do */
+	if (!zfcp_sdev->port)
+		return;
+
 	zfcp_erp_lun_shutdown_wait(sdev, "scssd_1");
 	put_device(&zfcp_sdev->port->dev);
 }

Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())

[Mike Snitzer]

On Mon, Dec 12 2011 at  7:39am -0500,
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> On Tue, Nov 29, 2011 at 03:18:03PM -0500, Mike Snitzer wrote:
> > On Tue, Nov 29 2011 at  7:00am -0500,
> > Heiko Carstens <heiko.carstens@de.ibm.com> wrote:
> > > [ 4906.683654] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000
> > 
> > ...
> > 
> > > Gonzalo also tried 2.6.38.8 as suggested and ran into this one:
> > > 
> > > [  292.877936] ------------[ cut here ]------------
> > > [  292.877939] Kernel BUG at 6b6b6b6b6b6b6b6d [verbose debug info unavailable]
> > 
> > Again, more poison.
> > 
> > Seems this test is causing us to fall on our face no matter what.
> > Likely, best to leave this 2.6.38 blk_unplug crash to one side and
> > continue focusing on latest upstream.
> 
> Sorry again, for taking so long to come back. This time however with good news:
> 
> With 3.2.0-rc4.00255.g77a7300 we were unable to reproduce any I/O stall or
> user-after-free bugs even after nearly 3000 test iterations.

Great news, so with an eye towards getting these fixes upstream:

> The only patches on top we have are:
> 
> two patches from Hannes:
> http://www.spinics.net/lists/linux-scsi/msg55112.html

Has that scsi_lib.c patch been posted with a formal patch header?

James, I'm not clear on where I should be looking to see what you have
staged but not yet sent to Linus.  Does such a branch exist in your
scsi-misc-2.6 tree?

> http://www.spinics.net/lists/linux-scsi/msg55413.html

Jun'ichi and Hannes said that additional NULL pointer check is needed:
http://www.redhat.com/archives/dm-devel/2011-December/msg00022.html

Hannes said he'd re-post an updated patch (but hasn't yet).

Mike