public inbox for linux-kernel@vger.kernel.org
 help / color / mirror / Atom feed
From: "Ted Ts'o" <tytso@mit.edu>
To: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Dave Jones <davej@redhat.com>,
	Wu Fengguang <fengguang.wu@intel.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>
Subject: Re: lockups shortly after booting in current git.
Date: Thu, 29 Mar 2012 16:25:43 -0700	[thread overview]
Message-ID: <20120329232543.GF13970@thunk.org> (raw)
In-Reply-To: <CA+55aFz5Di6K4TzrFrGdyrPQkadt1pkGrWv_9GyaVgUjx_4cwg@mail.gmail.com>

On Thu, Mar 29, 2012 at 02:52:40PM -0700, Linus Torvalds wrote:
> >
> > up 30 minutes with that reverted, no problems so far..
> 
> Goodie. Let it run for a while more, and really pound on it.
> 
> Ted, are there any downsides to just reverting that commit (ie any
> subtle interactions) for now?  That's assuming that Dave's testing
> continues to confirm that it is that one commit.

That commit fixes a race which is seen when you write into fallocated
(and hence uninitialized) disk blocks under *very* heavy memory
pressure.  Furthermore, although theoretically it could trigger under
normal direct I/O writes, it only seems to trigger if you are issuing
a huge number of AIO writes, such that a just-written page can get
evicted from memory, and then read back into memory, before the
workqueue has a chance to update the extent tree.

This race has been around for a little over a year, and no one noticed
until two months ago; it only happens under fairly exotic conditions,
and in fact even after trying very hard to create a simple repro under
lab conditions, we could only reproduce the problem and confirm the
fix on production servers running MySQL on very fast PCIe-attached
flash devices.

Given that Dave was able to hit this problem pretty quickly, if we
confirm that this commit is at fault, the only reasonable thing to do
is to revert it IMO.

    	      	      	   	       	  - Ted

  parent reply	other threads:[~2012-03-29 23:25 UTC|newest]

Thread overview: 19+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2012-03-29 15:55 lockups shortly after booting in current git Dave Jones
2012-03-29 18:23 ` Linus Torvalds
2012-03-29 18:29   ` Dave Jones
     [not found] ` <CA+55aFxOcMt_mcr+ZYwc-SpKbROnh4Gn7jqrFY_SZcBy1Ev7Qw@mail.gmail.com>
     [not found]   ` <20120329182632.GA6891@redhat.com>
2012-03-29 18:36     ` Linus Torvalds
2012-03-29 19:53       ` Dave Jones
2012-03-29 20:10         ` Linus Torvalds
2012-03-29 20:26           ` Dave Jones
2012-03-29 20:38             ` Linus Torvalds
2012-03-29 20:39             ` Ted Ts'o
2012-03-29 20:59               ` Linus Torvalds
2012-03-29 21:12                 ` Dave Jones
2012-03-29 21:45                   ` Ted Ts'o
2012-03-29 21:49                     ` Dave Jones
2012-03-29 21:52                       ` Linus Torvalds
2012-03-29 22:54                         ` Dave Jones
2012-03-29 23:04                           ` Linus Torvalds
2012-03-29 23:35                             ` Ted Ts'o
2012-03-29 23:25                         ` Ted Ts'o [this message]
2012-03-29 21:52                     ` Dave Jones

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20120329232543.GF13970@thunk.org \
    --to=tytso@mit.edu \
    --cc=davej@redhat.com \
    --cc=fengguang.wu@intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox