All of lore.kernel.org
 help / color / mirror / Atom feed
From: Mike Snitzer <snitzer@redhat.com>
To: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Hannes Reinecke <hare@suse.de>,
	Jun'ichi Nomura <j-nomura@ce.jp.nec.com>,
	James Bottomley <James.Bottomley@HansenPartnership.com>,
	Steffen Maier <maier@linux.vnet.ibm.com>,
	"linux-scsi@vger.kernel.org" <linux-scsi@vger.kernel.org>,
	Jens Axboe <axboe@kernel.dk>,
	Linux Kernel <linux-kernel@vger.kernel.org>,
	Alan Stern <stern@rowland.harvard.edu>,
	Thadeu Lima de Souza Cascardo <cascardo@linux.vnet.ibm.com>,
	"Taraka R. Bodireddy" <tarak.reddy@in.ibm.com>,
	"Seshagiri N. Ippili" <seshagiri.ippili@in.ibm.com>,
	"Manvanthara B. Puttashankar" <mputtash@in.ibm.com>,
	Jeff Moyer <jmoyer@redhat.com>, Shaohua Li <shaohua.li@intel.com>,
	gmuelas@de.ibm.com, dm-devel@redhat.com
Subject: Re: [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue())
Date: Tue, 29 Nov 2011 15:18:03 -0500	[thread overview]
Message-ID: <20111129201803.GB6827@redhat.com> (raw)
In-Reply-To: <20111129120047.GA2456@osiris.boeblingen.de.ibm.com>

On Tue, Nov 29 2011 at  7:00am -0500,
Heiko Carstens <heiko.carstens@de.ibm.com> wrote:

> > > > Hmm. Just to be on the safe side, could you try this one:
> > > > 
> > > > diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
> > > > index 5e0090e..e6fad46 100644
> > > > --- a/drivers/md/dm-mpath.c
> > > > +++ b/drivers/md/dm-mpath.c
> > > > @@ -920,8 +920,10 @@ static int multipath_map(struct dm_target *ti,
> > > > struct reque
> > > > st *clone,
> > > >         map_context->ptr = mpio;
> > > >         clone->cmd_flags |= REQ_FAILFAST_TRANSPORT;
> > > >         r = map_io(m, clone, mpio, 0);
> > > > -       if (r < 0 || r == DM_MAPIO_REQUEUE)
> > > > +       if (r < 0 || r == DM_MAPIO_REQUEUE) {
> > > >                 mempool_free(mpio, m->mpio_pool);
> > > > +               map_context->ptr = NULL;
> > > > +       }
> > > > 
> > > >         return r;
> > > >  }
> > > 
> > > With your patch we haven't been able to reproduce the kernel crash until now.
> > > Now we "only" run into I/O stalls, which before your patch we also did. But
> > > repeatedly rebooting and retrying and ignoring the I/O stalls always lead to
> > > a crash.
> > > Gonzalo will run a couple of extra rounds so we can have a feeling if at least
> > > one of the bugs could be fixed with your patch ;)
> > 
> > Hi,
> > 
> > Any update after further testing with Hannes' patch?
> 
> Sorry for the late update, our internal IBM IMAP servers have been down
> for nearly a week :/
> 
> So, we were unable to reproduce the original bug with the patch applied
> during various runs.

OK, so it seems to be a benenficial change (and obviously correct to
me).  Hannes, care to formally post your fix to dm-devel so we can get
it in 3.2-rc?

> However, we ran into this one instead, which is yet another use-after-free bug
> (I need to double check, but I'm quite sure that a freed struct scsi_cmnd
> caused this).

OK, yeah something is causing poisoned (POISON_FREE) memory to be used.

> [ 4906.683654] Unable to handle kernel pointer dereference at virtual kernel address 6b6b6b6b6b6b6000

...

> Gonzalo also tried 2.6.38.8 as suggested and ran into this one:
> 
> [  292.877936] ------------[ cut here ]------------
> [  292.877939] Kernel BUG at 6b6b6b6b6b6b6b6d [verbose debug info unavailable]

Again, more poison.

Seems this test is causing us to fall on our face no matter what.
Likely, best to leave this 2.6.38 blk_unplug crash to one side and
continue focusing on latest upstream.

Mike

  reply	other threads:[~2011-11-29 20:18 UTC|newest]

Thread overview: 65+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2011-09-22 13:18 [PATCH] block: Free queue resources at blk_release_queue() Hannes Reinecke
2011-09-28  0:47 ` Jens Axboe
2011-09-28  0:55   ` Linus Torvalds
2011-09-28  1:15     ` Jens Axboe
2011-09-28  1:59       ` Linus Torvalds
2011-09-28  2:02         ` Jens Axboe
2011-09-28  4:10         ` James Bottomley
2011-09-28 14:08           ` Jens Axboe
2011-09-28 14:11             ` James Bottomley
2011-09-28 14:14               ` [GIT PULL] Queue free fix (was Re: [PATCH] block: Free queue resources at blk_release_queue()) Jens Axboe
2011-09-28 15:22                 ` Linus Torvalds
2011-09-28 15:22                   ` Linus Torvalds
2011-09-28 15:43                   ` James Bottomley
2011-09-28 17:48                     ` Vivek Goyal
2011-09-28 17:53                       ` Christoph Hellwig
2011-09-28 18:09                         ` Vivek Goyal
2011-09-28 18:16                           ` Christoph Hellwig
2011-09-28 19:05                             ` Eric Seppanen
2011-09-28 19:05                               ` Eric Seppanen
2011-09-28 19:14                               ` Christoph Hellwig
2011-11-30 10:18                               ` Jens Axboe
2011-11-30 10:26                                 ` Christoph Hellwig
2011-09-28 22:34                             ` Vivek Goyal
2011-09-28 17:59                       ` James Bottomley
2011-10-13 13:09                 ` Steffen Maier
2011-10-14 16:03                   ` James Bottomley
2011-10-17  8:46                     ` Jun'ichi Nomura
2011-10-17 14:06                       ` James Bottomley
2011-10-18 13:31                         ` Jun'ichi Nomura
2011-10-18 15:45                           ` Heiko Carstens
2011-10-18 16:29                             ` James Bottomley
2011-10-31 10:05                               ` Heiko Carstens
2011-10-31 10:42                                 ` James Bottomley
2011-10-31 11:46                                   ` Jun'ichi Nomura
2011-10-31 13:00                                     ` Heiko Carstens
2011-11-02 12:37                                       ` Jun'ichi Nomura
2011-11-02 12:44                                         ` Hannes Reinecke
2011-11-02 12:44                                           ` Hannes Reinecke
2011-11-02 13:47                                         ` Heiko Carstens
2011-11-04  4:07                                           ` Jun'ichi Nomura
2011-11-04  9:12                                             ` Heiko Carstens
2011-11-03 18:25                                       ` Mike Snitzer
2011-11-04  9:19                                         ` Heiko Carstens
2011-11-04 13:30                                           ` Mike Snitzer
2011-11-04 13:37                                             ` Hannes Reinecke
2011-11-07 11:31                                             ` Jun'ichi Nomura
2011-11-07 13:42                                               ` Mike Snitzer
2011-11-07 12:23                                             ` Heiko Carstens
2011-11-07 11:30                                           ` Jun'ichi Nomura
2011-11-07 15:36                                             ` Mike Snitzer
2011-11-07 16:43                                               ` Heiko Carstens
2011-11-07 17:10                                               ` Mike Snitzer
2011-11-07 21:44                                                 ` Mike Snitzer
2011-11-09  9:37                                           ` Hannes Reinecke
2011-11-10 16:10                                             ` Heiko Carstens
2011-11-17 16:29                                               ` Mike Snitzer
2011-11-29 12:00                                                 ` Heiko Carstens
2011-11-29 20:18                                                   ` Mike Snitzer [this message]
2011-11-30  7:25                                                     ` Hannes Reinecke
2011-11-30  7:25                                                       ` Hannes Reinecke
2011-12-12 12:39                                                     ` Heiko Carstens
2011-12-13 16:50                                                       ` Mike Snitzer
2011-10-31 13:21                                   ` Mike Snitzer
2011-10-31 13:40                                     ` Heiko Carstens
2011-10-31 14:01                                       ` Mike Snitzer

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20111129201803.GB6827@redhat.com \
    --to=snitzer@redhat.com \
    --cc=James.Bottomley@HansenPartnership.com \
    --cc=axboe@kernel.dk \
    --cc=cascardo@linux.vnet.ibm.com \
    --cc=dm-devel@redhat.com \
    --cc=gmuelas@de.ibm.com \
    --cc=hare@suse.de \
    --cc=heiko.carstens@de.ibm.com \
    --cc=j-nomura@ce.jp.nec.com \
    --cc=jmoyer@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-scsi@vger.kernel.org \
    --cc=maier@linux.vnet.ibm.com \
    --cc=mputtash@in.ibm.com \
    --cc=seshagiri.ippili@in.ibm.com \
    --cc=shaohua.li@intel.com \
    --cc=stern@rowland.harvard.edu \
    --cc=tarak.reddy@in.ibm.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.