From mboxrd@z Thu Jan 1 00:00:00 1970 From: Chris Mason Subject: Re: 3.4.4-rt13: btrfs + xfstests 006 = BOOM.. and a bonus rt_mutex deadlock report for absolutely free! Date: Mon, 16 Jul 2012 11:43:04 -0400 Message-ID: <20120716154304.GG25961@shiny.int.fusionio.com> References: <1342072060.7338.102.camel@marge.simpson.net> <20120713125043.GH30128@shiny> <1342260883.7368.30.camel@marge.simpson.net> <1342436144.7659.58.camel@marge.simpson.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Cc: "Chris L. Mason" , "linux-rt-users@vger.kernel.org" , LKML , linux-fsdevel , Thomas Gleixner , Steven Rostedt To: Mike Galbraith Return-path: Content-Disposition: inline In-Reply-To: <1342436144.7659.58.camel@marge.simpson.net> Sender: linux-rt-users-owner@vger.kernel.org List-Id: linux-fsdevel.vger.kernel.org On Mon, Jul 16, 2012 at 04:55:44AM -0600, Mike Galbraith wrote: > On Sat, 2012-07-14 at 12:14 +0200, Mike Galbraith wrote: > > On Fri, 2012-07-13 at 08:50 -0400, Chris Mason wrote: > > > On Wed, Jul 11, 2012 at 11:47:40PM -0600, Mike Galbraith wrote: > > > > Greetings, > > > > > > [ deadlocks with btrfs and the recent RT kernels ] > > > > > > I talked with Thomas about this and I think the problem is the > > > single-reader nature of the RW rwlocks. The lockdep report below > > > mentions that btrfs is calling: > > > > > > > [ 692.963099] [] btrfs_clear_path_blocking+0x32/0x70 > > > > > > In this case, the task has a number of blocking read locks on the btrfs buffers, > > > and we're trying to turn them back into spinning read locks. Even > > > though btrfs is taking the read rwlock, it doesn't think of this as a new > > > lock operation because we were blocking out new writers. > > > > > > If the second task has taken the spinning read lock, it is going to > > > prevent that clear_path_blocking operation from progressing, even though > > > it would have worked on a non-RT kernel. > > > > > > The solution should be to make the blocking read locks in btrfs honor the > > > single-reader semantics. This means not allowing more than one blocking > > > reader and not allowing a spinning reader when there is a blocking > > > reader. Strictly speaking btrfs shouldn't need recursive readers on a > > > single lock, so I wouldn't worry about that part. > > > > > > There is also a chunk of code in btrfs_clear_path_blocking that makes > > > sure to strictly honor top down locking order during the conversion. It > > > only does this when lockdep is enabled because in non-RT kernels we > > > don't need to worry about it. For RT we'll want to enable that as well. > > > > > > I'll give this a shot later today. > > > > I took a poke at it. Did I do something similar to what you had in > > mind, or just hide behind performance stealing paranoid trylock loops? > > Box survived 1000 x xfstests 006 and dbench [-s] massive right off the > > bat, so it gets posted despite skepticism. > > Seems btrfs isn't entirely convinced either. > > [ 2292.336229] use_block_rsv: 1810 callbacks suppressed > [ 2292.336231] ------------[ cut here ]------------ > [ 2292.336255] WARNING: at fs/btrfs/extent-tree.c:6344 use_block_rsv+0x17d/0x190 [btrfs]() > [ 2292.336257] Hardware name: System x3550 M3 -[7944K3G]- > [ 2292.336259] btrfs: block rsv returned -28 This is unrelated. You got far enough into the benchmark to hit an ENOSPC warning. This can be ignored (I just deleted it when we used 3.0 for oracle). re: dbench performance. dbench tends to penalize fairness. I can imagine RT making it slower in general. It also triggers lots of lock contention in btrfs because the dataset is fairly small and the trees don't fan out a lot. -chris