public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: Mike Snitzer <snitzer@redhat.com>
To: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Hannes Reinecke <hare@suse.de>,
	"Jun'ichi Nomura" <j-nomura@ce.jp.nec.com>,
	James Bottomley <James.Bottomley@HansenPartnership.com>,
	Steffen Maier <maier@linux.vnet.ibm.com>,
	"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>,
	Linux Kernel <linux-kernel@vger.kernel.org>,
	Alan Stern <stern@rowland.harvard.edu>,
	Thadeu Lima de Souza Cascardo <cascardo@linux.vnet.ibm.com>,
	"Taraka R. Bodireddy" <tarak.reddy@in.ibm.com>,
	"Seshagiri N. Ippili" <seshagiri.ippili@in.ibm.com>,
	"Manvanthara B. Puttashankar" <mputtash@in.ibm.com>,
	Jeff Moyer <jmoyer@redhat.com>, Shaohua Li <shaohua.li@intel.com>,
	gmuelas@de.ibm.com, dm-devel@redhat.com
Subject: Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())
Date: Tue, 29 Nov 2011 15:18:03 -0500	[thread overview]
Message-ID: <20111129201803.GB6827@redhat.com> (raw)
In-Reply-To: <20111129120047.GA2456@osiris.boeblingen.de.ibm.com>

On Tue, Nov 29 2011 at  7:00am -0500,
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> > > > Hmm. Just to be on the safe side, could you try this one:
> > > > 
> > > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> > > > index 5e0090e..e6fad46 100644
> > > > --- a/drivers/md/dm-mpath.c
> > > > +++ b/drivers/md/dm-mpath.c
> > > > @@ -920,8 +920,10 @@ static int multipath_map(struct dm_target *ti,
> > > > struct reque
> > > > st *clone,
> > > >         map_context->ptr = mpio;
> > > >         clone->cmd_flags |= REQ_FAILFAST_TRANSPORT;
> > > >         r = map_io(m, clone, mpio, 0);
> > > > -       if (r < 0 || r == DM_MAPIO_REQUEUE)
> > > > +       if (r < 0 || r == DM_MAPIO_REQUEUE) {
> > > >                 mempool_free(mpio, m->mpio_pool);
> > > > +               map_context->ptr = NULL;
> > > > +       }
> > > > 
> > > >         return r;
> > > >  }
> > > 
> > > With your patch we haven't been able to reproduce the kernel crash until now.
> > > Now we "only" run into I/O stalls, which before your patch we also did. But
> > > repeatedly rebooting and retrying and ignoring the I/O stalls always lead to
> > > a crash.
> > > Gonzalo will run a couple of extra rounds so we can have a feeling if at least
> > > one of the bugs could be fixed with your patch ;)
> > 
> > Hi,
> > 
> > Any update after further testing with Hannes' patch?
> 
> Sorry for the late update, our internal IBM IMAP servers have been down
> for nearly a week :/
> 
> So, we were unable to reproduce the original bug with the patch applied
> during various runs.

OK, so it seems to be a benenficial change (and obviously correct to
me).  Hannes, care to formally post your fix to dm-devel so we can get
it in 3.2-rc?

> However, we ran into this one instead, which is yet another use-after-free bug
> (I need to double check, but I'm quite sure that a freed struct scsi_cmnd
> caused this).

OK, yeah something is causing poisoned (POISON_FREE) memory to be used.

> [ 4906.683654] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000

...

> Gonzalo also tried 2.6.38.8 as suggested and ran into this one:
> 
> [  292.877936] ------------[ cut here ]------------
> [  292.877939] Kernel BUG at 6b6b6b6b6b6b6b6d [verbose debug info unavailable]

Again, more poison.

Seems this test is causing us to fall on our face no matter what.
Likely, best to leave this 2.6.38 blk_unplug crash to one side and
continue focusing on latest upstream.

Mike

  reply	other threads:[~2011-11-29 20:18 UTC|newest]

Thread overview: 60+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <201109221318.p8MDIXPP020015@pentland.suse.de>
2011-09-28  0:47 ` [PATCH] block: Free queue resources at blk_release_queue() Jens Axboe
2011-09-28  0:55   ` Linus Torvalds
2011-09-28  1:15     ` Jens Axboe
2011-09-28  1:59       ` Linus Torvalds
2011-09-28  2:02         ` Jens Axboe
2011-09-28  4:10         ` James Bottomley
2011-09-28 14:08           ` Jens Axboe
2011-09-28 14:11             ` James Bottomley
2011-09-28 14:14               ` [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue()) Jens Axboe
2011-09-28 15:22                 ` Linus Torvalds
2011-09-28 15:43                   ` James Bottomley
2011-09-28 17:48                     ` Vivek Goyal
2011-09-28 17:53                       ` Christoph Hellwig
2011-09-28 18:09                         ` Vivek Goyal
2011-09-28 18:16                           ` Christoph Hellwig
2011-09-28 19:05                             ` Eric Seppanen
2011-09-28 19:14                               ` Christoph Hellwig
2011-11-30 10:18                               ` Jens Axboe
2011-11-30 10:26                                 ` Christoph Hellwig
2011-09-28 22:34                             ` Vivek Goyal
2011-09-28 17:59                       ` James Bottomley
2011-10-13 13:09                 ` Steffen Maier
2011-10-14 16:03                   ` James Bottomley
2011-10-17  8:46                     ` Jun'ichi Nomura
2011-10-17 14:06                       ` James Bottomley
2011-10-18 13:31                         ` Jun'ichi Nomura
2011-10-18 15:45                           ` Heiko Carstens
2011-10-18 16:29                             ` James Bottomley
2011-10-31 10:05                               ` Heiko Carstens
2011-10-31 10:42                                 ` James Bottomley
2011-10-31 11:46                                   ` Jun'ichi Nomura
2011-10-31 13:00                                     ` Heiko Carstens
2011-11-02 12:37                                       ` Jun'ichi Nomura
2011-11-02 12:44                                         ` Hannes Reinecke
2011-11-02 13:47                                         ` Heiko Carstens
2011-11-04  4:07                                           ` Jun'ichi Nomura
2011-11-04  9:12                                             ` Heiko Carstens
2011-11-03 18:25                                       ` Mike Snitzer
2011-11-04  9:19                                         ` Heiko Carstens
2011-11-04 13:30                                           ` Mike Snitzer
2011-11-04 13:37                                             ` Hannes Reinecke
2011-11-07 11:31                                             ` Jun'ichi Nomura
2011-11-07 13:42                                               ` Mike Snitzer
2011-11-07 12:23                                             ` Heiko Carstens
2011-11-07 11:30                                           ` Jun'ichi Nomura
2011-11-07 15:36                                             ` Mike Snitzer
2011-11-07 16:43                                               ` Heiko Carstens
2011-11-07 17:10                                               ` Mike Snitzer
2011-11-07 21:44                                                 ` Mike Snitzer
2011-11-09  9:37                                           ` Hannes Reinecke
2011-11-10 16:10                                             ` Heiko Carstens
2011-11-17 16:29                                               ` Mike Snitzer
2011-11-29 12:00                                                 ` Heiko Carstens
2011-11-29 20:18                                                   ` Mike Snitzer [this message]
2011-11-30  7:25                                                     ` Hannes Reinecke
2011-12-12 12:39                                                     ` Heiko Carstens
2011-12-13 16:50                                                       ` Mike Snitzer
2011-10-31 13:21                                   ` Mike Snitzer
2011-10-31 13:40                                     ` Heiko Carstens
2011-10-31 14:01                                       ` Mike Snitzer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20111129201803.GB6827@redhat.com \
    --to=snitzer@redhat.com \
    --cc=James.Bottomley@HansenPartnership.com \
    --cc=axboe@kernel.dk \
    --cc=cascardo@linux.vnet.ibm.com \
    --cc=dm-devel@redhat.com \
    --cc=gmuelas@de.ibm.com \
    --cc=hare@suse.de \
    --cc=heiko.carstens@de.ibm.com \
    --cc=j-nomura@ce.jp.nec.com \
    --cc=jmoyer@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=maier@linux.vnet.ibm.com \
    --cc=mputtash@in.ibm.com \
    --cc=seshagiri.ippili@in.ibm.com \
    --cc=shaohua.li@intel.com \
    --cc=stern@rowland.harvard.edu \
    --cc=tarak.reddy@in.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox