* [BUG] 3.12 vfs weirdness under load
@ 2013-11-13 6:05 Jeff King
2013-11-13 21:36 ` Jeff King
0 siblings, 1 reply; 4+ messages in thread
From: Jeff King @ 2013-11-13 6:05 UTC (permalink / raw)
To: linux-fsdevel; +Cc: Waiman Long, Al Viro
On v3.12, running "make -j16 test" on a git.git checkout produces random
failures in the test suite, whereas it works fine on v3.11. Git's test
suite puts a lot of load on the VFS; re-running failed tests
individually when the system is unloaded lets them pass. I haven't been
able to get a specific syscall strace of what's going on, but some of
the failures I investigated had obvious oddness, like "grep foo bar"
returning non-zero, but then looking in "bar" manually shows that it
contains the expected content.
I bisected this to 232d2d6 (dcache: Translating dentry into pathname
without taking rename_lock, 2013-09-09), and can reliably reproduce the
problem (and running with the parent ac83871 reliably works).
I see the problem on my ext4-over-dmcrypt disk, but also when running
the tests on a tmpfs disk. So I don't think it's filesystem-specific.
Reading the patch for 232d2d6, it seems plausible that it's a locking
problem in the new code, but I don't see anything obviously wrong. I'd
expect others can reproduce by running git's test suite, but if not, I'm
happy to test patches.
-Peff
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] 3.12 vfs weirdness under load
2013-11-13 6:05 [BUG] 3.12 vfs weirdness under load Jeff King
@ 2013-11-13 21:36 ` Jeff King
2013-11-13 21:40 ` Al Viro
0 siblings, 1 reply; 4+ messages in thread
From: Jeff King @ 2013-11-13 21:36 UTC (permalink / raw)
To: linux-fsdevel; +Cc: Waiman Long, Al Viro
On Wed, Nov 13, 2013 at 01:05:58AM -0500, Jeff King wrote:
> On v3.12, running "make -j16 test" on a git.git checkout produces random
> failures in the test suite, whereas it works fine on v3.11.
> [...]
> Reading the patch for 232d2d6, it seems plausible that it's a locking
> problem in the new code, but I don't see anything obviously wrong. I'd
> expect others can reproduce by running git's test suite, but if not, I'm
> happy to test patches.
A word of warning to anyone who wants to reproduce this: I believe it
caused some data loss during my bisect. Git's test suite does a lot of:
mkdir trash.XXXXX
... do some tests in trash.XXXXX
rm -rf trash.XXXXX
>From reading the patch, I'm guessing the bug is causing wrong dir
entries to be returned under load. Random bits of my home directory are
missing this morning after my testing; my theory is that when we are
unlucky enough to hit the race on the "rm", it can end up deleting
random things.
-Peff
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] 3.12 vfs weirdness under load
2013-11-13 21:36 ` Jeff King
@ 2013-11-13 21:40 ` Al Viro
2013-11-14 14:05 ` Jeff King
0 siblings, 1 reply; 4+ messages in thread
From: Al Viro @ 2013-11-13 21:40 UTC (permalink / raw)
To: Jeff King; +Cc: linux-fsdevel, Waiman Long
On Wed, Nov 13, 2013 at 04:36:06PM -0500, Jeff King wrote:
> On Wed, Nov 13, 2013 at 01:05:58AM -0500, Jeff King wrote:
>
> > On v3.12, running "make -j16 test" on a git.git checkout produces random
> > failures in the test suite, whereas it works fine on v3.11.
> > [...]
> > Reading the patch for 232d2d6, it seems plausible that it's a locking
> > problem in the new code, but I don't see anything obviously wrong. I'd
> > expect others can reproduce by running git's test suite, but if not, I'm
> > happy to test patches.
>
> A word of warning to anyone who wants to reproduce this: I believe it
> caused some data loss during my bisect. Git's test suite does a lot of:
>
> mkdir trash.XXXXX
> ... do some tests in trash.XXXXX
> rm -rf trash.XXXXX
>
> >From reading the patch, I'm guessing the bug is causing wrong dir
> entries to be returned under load. Random bits of my home directory are
> missing this morning after my testing; my theory is that when we are
> unlucky enough to hit the race on the "rm", it can end up deleting
> random things.
Move initialization of dentry/vfsmount/mnt to just before the loop in
prepend_path() and see if that fixes what you are seeing...
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [BUG] 3.12 vfs weirdness under load
2013-11-13 21:40 ` Al Viro
@ 2013-11-14 14:05 ` Jeff King
0 siblings, 0 replies; 4+ messages in thread
From: Jeff King @ 2013-11-14 14:05 UTC (permalink / raw)
To: Al Viro; +Cc: linux-fsdevel, Waiman Long
On Wed, Nov 13, 2013 at 09:40:53PM +0000, Al Viro wrote:
> > > Reading the patch for 232d2d6, it seems plausible that it's a locking
> > > problem in the new code, but I don't see anything obviously wrong. I'd
> > > expect others can reproduce by running git's test suite, but if not, I'm
> > > happy to test patches.
> [...]
>
> Move initialization of dentry/vfsmount/mnt to just before the loop in
> prepend_path() and see if that fixes what you are seeing...
Yep, that did the trick. Looks like your fix is already in Linus's tree,
too. Thanks!
-Peff
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-11-14 14:05 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-11-13 6:05 [BUG] 3.12 vfs weirdness under load Jeff King
2013-11-13 21:36 ` Jeff King
2013-11-13 21:40 ` Al Viro
2013-11-14 14:05 ` Jeff King
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).