From: Nick Piggin <npiggin@gmail.com>
To: Dave Chinner <david@fromorbit.com>
Cc: Nick Piggin <npiggin@kernel.dk>,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
Peter Zijlstra <a.p.zijlstra@chello.nl>
Subject: Re: [PATCH 00/46] rcu-walk and dcache scaling
Date: Wed, 8 Dec 2010 02:24:23 +1100 [thread overview]
Message-ID: <AANLkTinAt9PEWMCP55o64E7wsfZ=depU+YXbG8DdGYdO@mail.gmail.com> (raw)
In-Reply-To: <20101207112555.GE16103@dastard>
On Tue, Dec 7, 2010 at 10:25 PM, Dave Chinner <david@fromorbit.com> wrote:
> On Sat, Nov 27, 2010 at 09:15:58PM +1100, Nick Piggin wrote:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/npiggin/linux-npiggin.git vfs-scale-working
>>
>> Here is an new set of vfs patches for review, not that there was much interest
>> last time they were posted. It is structured like:
>>
>> * preparation patches
>> * introduce new locks to take over dcache_lock, then remove it
>> * cleaning up and reworking things for new locks
>> * rcu-walk path walking
>> * start on some fine grained locking steps
>
> Just got this set of traces doing an 8-way parallel remove of 50
> million inodes at about 40M inodes unlinked:
Thanks for testing...
> [ 5954.061633] BUG: sleeping function called from invalid context at arch/x86/mm/fault.c:1081
> [ 5954.062466] in_atomic(): 0, irqs_disabled(): 1, pid: 2927, name: rm
> [ 5954.063122] 3 locks held by rm/2927:
> [ 5954.063476] #0: (&sb->s_type->i_mutex_key#12/1){+.+.+.}, at: [<ffffffff8116f5e1>] do_rmdir+0x81/0x130
> [ 5954.064014] #1: (&sb->s_type->i_mutex_key#12){+.+.+.}, at: [<ffffffff8116d3a8>] vfs_rmdir+0x58/0xe0
> [ 5954.064014] #2: (rcu_read_lock){.+.+..}, at: [<ffffffff811779c0>] shrink_dentry_list+0x0/0x430
> [ 5954.064014] irq event stamp: 1484376719
> [ 5954.064014] hardirqs last enabled at (1484376719): [<ffffffff810ebf07>] __call_rcu+0xd7/0x1a0
> [ 5954.064014] hardirqs last disabled at (1484376718): [<ffffffff810ebe7a>] __call_rcu+0x4a/0x1a0
> [ 5954.064014] softirqs last enabled at (1484376586): [<ffffffff8108b911>] __do_softirq+0x161/0x270
> [ 5954.064014] softirqs last disabled at (1484376581): [<ffffffff8103af1c>] call_softirq+0x1c/0x50
> [ 5954.064014] Pid: 2927, comm: rm Not tainted 2.6.37-rc4-dgc+ #794
> [ 5954.064014] Call Trace:
> [ 5954.064014] [<ffffffff810b95b0>] ? print_irqtrace_events+0xd0/0xe0
> [ 5954.064014] [<ffffffff81076455>] __might_sleep+0xf5/0x130
> [ 5954.064014] [<ffffffff81b1e603>] do_page_fault+0x103/0x4f0
> [ 5954.064014] [<ffffffff810645b8>] ? pvclock_clocksource_read+0x58/0xd0
> [ 5954.064014] [<ffffffff810645b8>] ? pvclock_clocksource_read+0x58/0xd0
> [ 5954.064014] [<ffffffff810645b8>] ? pvclock_clocksource_read+0x58/0xd0
> [ 5954.064014] [<ffffffff81b19b28>] ? trace_hardirqs_off_thunk+0x3a/0x3c
> [ 5954.064014] [<ffffffff810b8e20>] ? usage_match+0x0/0x20
> [ 5954.064014] [<ffffffff81b1af25>] page_fault+0x25/0x30
> [ 5954.064014] [<ffffffff810b8e20>] ? usage_match+0x0/0x20
> [ 5954.064014] [<ffffffff810ba1c8>] ? __bfs+0xc8/0x260
> [ 5954.064014] [<ffffffff810ba123>] ? __bfs+0x23/0x260
> [ 5954.064014] [<ffffffff810ba4d2>] find_usage_backwards+0x42/0x80
> [ 5954.064014] [<ffffffff810bcec4>] check_usage_backwards+0x64/0xf0
> [ 5954.064014] [<ffffffff8104796f>] ? save_stack_trace+0x2f/0x50
> [ 5954.064014] [<ffffffff810bce60>] ? check_usage_backwards+0x0/0xf0
> [ 5954.064014] [<ffffffff810bd9a9>] mark_lock+0x1a9/0x440
> [ 5954.064014] [<ffffffff810be989>] __lock_acquire+0x5a9/0x14b0
> [ 5954.064014] [<ffffffff810be716>] ? __lock_acquire+0x336/0x14b0
> [ 5954.064014] [<ffffffff810645b8>] ? pvclock_clocksource_read+0x58/0xd0
> [ 5954.064014] [<ffffffff810bf944>] lock_acquire+0xb4/0x140
> [ 5954.064014] [<ffffffff81177a1c>] ? shrink_dentry_list+0x5c/0x430
> [ 5954.064014] [<ffffffff81b19d86>] _raw_spin_lock+0x36/0x70
> [ 5954.064014] [<ffffffff81177a1c>] ? shrink_dentry_list+0x5c/0x430
> [ 5954.064014] [<ffffffff81177a1c>] shrink_dentry_list+0x5c/0x430
> [ 5954.064014] [<ffffffff811779c0>] ? shrink_dentry_list+0x0/0x430
> [ 5954.064014] [<ffffffff816b9c7e>] ? do_raw_spin_unlock+0x5e/0xb0
> [ 5954.064014] [<ffffffff81177f2d>] __shrink_dcache_sb+0x13d/0x1c0
> [ 5954.064014] [<ffffffff811784bf>] shrink_dcache_parent+0x32f/0x390
> [ 5954.064014] [<ffffffff8116d31d>] dentry_unhash+0x3d/0x70
> [ 5954.064014] [<ffffffff8116d3b0>] vfs_rmdir+0x60/0xe0
> [ 5954.064014] [<ffffffff8116f673>] do_rmdir+0x113/0x130
> [ 5954.064014] [<ffffffff8103a03a>] ? sysret_check+0x2e/0x69
> [ 5954.064014] [<ffffffff81b19ae9>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [ 5954.064014] [<ffffffff8116f6c5>] sys_unlinkat+0x35/0x40
> [ 5954.064014] [<ffffffff8103a002>] system_call_fastpath+0x16/0x1b
Seems that lockdep exploded.
> [ 5954.092916] BUG: unable to handle kernel NULL pointer dereference at (null)
> [ 5954.093806] IP: [<ffffffff810ba1c8>] __bfs+0xc8/0x260
> [ 5954.094331] PGD 1084e5067 PUD 102368067 PMD 0
> [ 5954.094830] Oops: 0000 [#1] SMP
> [ 5954.095194] last sysfs file: /sys/devices/system/cpu/online
> [ 5954.095760] CPU 6
> [ 5954.095954] Modules linked in:
> [ 5954.096319]
> [ 5954.096483] Pid: 2927, comm: rm Not tainted 2.6.37-rc4-dgc+ #794 /Bochs
> [ 5954.096665] RIP: 0010:[<ffffffff810ba1c8>] [<ffffffff810ba1c8>] __bfs+0xc8/0x260
> [ 5954.096665] RSP: 0018:ffff8801175539a8 EFLAGS: 00010046
> [ 5954.096665] RAX: ffffffff8267d980 RBX: ffffffff8267d980 RCX: ffff880117553a48
> [ 5954.096665] RDX: ffff8801175539d0 RSI: 0000000000000000 RDI: ffff880117553a48
> [ 5954.096665] RBP: ffff880117553a08 R08: 0000000000000000 R09: 0000000000000000
> [ 5954.096665] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000000
> [ 5954.096665] R13: ffffffff810b8e20 R14: ffff880117553a90 R15: 0000000000000000
> [ 5954.096665] FS: 00007f4594cf3700(0000) GS:ffff8800dfa00000(0000) knlGS:0000000000000000
> [ 5954.096665] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 5954.096665] CR2: 00007f2f21e89c60 CR3: 0000000110b0f000 CR4: 00000000000006e0
> [ 5954.096665] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [ 5954.096665] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> [ 5954.096665] Process rm (pid: 2927, threadinfo ffff880117552000, task ffff88010ff10b00)
> [ 5954.096665] Stack:
> [ 5954.096665] ffffffff8267d868 00007fffa02d1428 ffff8800ffffffff ffff880100000000
> [ 5954.096665] 000000000000b720 ffff880117553a48 ffffffff8267d868 ffff880117553a48
> [ 5954.096665] 0000000000000000 ffff88010ff10b00 0000000000000000 ffffffff81dacba0
> [ 5954.096665] Call Trace:
> [ 5954.096665] [<ffffffff810ba4d2>] find_usage_backwards+0x42/0x80
> [ 5954.096665] [<ffffffff810bcec4>] check_usage_backwards+0x64/0xf0
> [ 5954.096665] [<ffffffff8104796f>] ? save_stack_trace+0x2f/0x50
> [ 5954.096665] [<ffffffff810bce60>] ? check_usage_backwards+0x0/0xf0
> [ 5954.096665] [<ffffffff810bd9a9>] mark_lock+0x1a9/0x440
> [ 5954.096665] [<ffffffff810be989>] __lock_acquire+0x5a9/0x14b0
> [ 5954.096665] [<ffffffff810be716>] ? __lock_acquire+0x336/0x14b0
> [ 5954.096665] [<ffffffff810645b8>] ? pvclock_clocksource_read+0x58/0xd0
> [ 5954.096665] [<ffffffff810bf944>] lock_acquire+0xb4/0x140
> [ 5954.096665] [<ffffffff81177a1c>] ? shrink_dentry_list+0x5c/0x430
> [ 5954.096665] [<ffffffff81b19d86>] _raw_spin_lock+0x36/0x70
> [ 5954.096665] [<ffffffff81177a1c>] ? shrink_dentry_list+0x5c/0x430
> [ 5954.096665] [<ffffffff81177a1c>] shrink_dentry_list+0x5c/0x430
> [ 5954.096665] [<ffffffff811779c0>] ? shrink_dentry_list+0x0/0x430
> [ 5954.096665] [<ffffffff816b9c7e>] ? do_raw_spin_unlock+0x5e/0xb0
> [ 5954.096665] [<ffffffff81177f2d>] __shrink_dcache_sb+0x13d/0x1c0
> [ 5954.096665] [<ffffffff811784bf>] shrink_dcache_parent+0x32f/0x390
> [ 5954.096665] [<ffffffff8116d31d>] dentry_unhash+0x3d/0x70
> [ 5954.096665] [<ffffffff8116d3b0>] vfs_rmdir+0x60/0xe0
> [ 5954.096665] [<ffffffff8116f673>] do_rmdir+0x113/0x130
> [ 5954.096665] [<ffffffff8103a03a>] ? sysret_check+0x2e/0x69
> [ 5954.096665] [<ffffffff81b19ae9>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [ 5954.096665] [<ffffffff8116f6c5>] sys_unlinkat+0x35/0x40
> [ 5954.096665] [<ffffffff8103a002>] system_call_fastpath+0x16/0x1b
> [ 5954.096665] Code: 0a 89 05 dc 0f a8 01 48 8b 41 10 48 85 c0 0f 84 1f 01 00 00 48 8d 98 70 01 00 00 48 05 80 01 00 00 45 85 c0 48 0f 44 d8 4c 8b 3b <49> 8b 07 49 39
> [ 5954.096665] RIP [<ffffffff810ba1c8>] __bfs+0xc8/0x260
> [ 5954.096665] RSP <ffff8801175539a8>
> [ 5954.096665] CR2: 0000000000000000
> [ 5954.127991] ---[ end trace 85a6727c2d4e3d90 ]---
So I vfs-scale-working branch may not be entirely in the clear, seeing
as it touches
the code lower in the call chain. However I don't know what can cause
lockdep to go off
the rails like this.
There is a sequence I used to hack around lockdep nesting
restrictions, following this
pattern:
repeat:
spin_lock(&parent->d_lock);
spin_lock_nested(&dentry->d_lock, DENTRY_D_LOCK_NESTED);
/* do stuff */
spin_unlock(&parent->d_lock);
spin_release(&dentry->d_lock.dep_map, 1, _RET_IP_);
parent = dentry;
spin_acquire(&this_parent->d_lock.dep_map, 0, 1, _RET_IP_);
goto repeat;
It's not directly in this call chain, but I wonder if it could have
caused any problem?
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
next prev parent reply other threads:[~2010-12-07 15:24 UTC|newest]
Thread overview: 96+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-11-27 10:15 [PATCH 00/46] rcu-walk and dcache scaling Nick Piggin
2010-11-27 9:44 ` [PATCH 02/46] fs: d_validate fixes Nick Piggin
2010-12-08 1:53 ` Dave Chinner
2010-12-08 6:59 ` Nick Piggin
2010-12-09 0:50 ` Dave Chinner
2010-12-09 4:50 ` Nick Piggin
2010-11-27 9:44 ` [PATCH 03/46] kernel: kmem_ptr_validate considered harmful Nick Piggin
2010-11-27 9:44 ` [PATCH 04/46] fs: dcache documentation cleanup Nick Piggin
2010-11-27 9:44 ` [PATCH 05/46] fs: change d_delete semantics Nick Piggin
2010-11-27 9:44 ` [PATCH 06/46] cifs: dont overwrite dentry name in d_revalidate Nick Piggin
2010-11-27 9:44 ` [PATCH 07/46] jfs: " Nick Piggin
2010-11-27 9:44 ` [PATCH 08/46] fs: change d_compare for rcu-walk Nick Piggin
2010-11-27 9:44 ` [PATCH 09/46] fs: change d_hash " Nick Piggin
2010-11-27 9:44 ` [PATCH 10/46] hostfs: simplify locking Nick Piggin
2010-11-27 9:44 ` [PATCH 11/46] fs: dcache scale hash Nick Piggin
2010-12-09 6:09 ` Dave Chinner
2010-12-09 6:28 ` Nick Piggin
2010-12-09 8:17 ` Dave Chinner
2010-12-09 12:53 ` Nick Piggin
2010-12-09 23:42 ` Dave Chinner
2010-12-10 2:35 ` Nick Piggin
2010-12-10 9:01 ` Dave Chinner
2010-12-13 4:48 ` Nick Piggin
2010-12-13 5:05 ` Nick Piggin
2010-11-27 9:44 ` [PATCH 12/46] fs: dcache scale lru Nick Piggin
2010-12-09 7:22 ` Dave Chinner
2010-12-09 12:34 ` Nick Piggin
2010-11-27 9:44 ` [PATCH 13/46] fs: dcache scale dentry refcount Nick Piggin
2010-11-27 9:44 ` [PATCH 14/46] fs: dcache scale d_unhashed Nick Piggin
2010-11-27 9:44 ` [PATCH 15/46] fs: dcache scale subdirs Nick Piggin
2010-11-27 9:44 ` [PATCH 16/46] fs: scale inode alias list Nick Piggin
2010-11-27 9:44 ` [PATCH 17/46] fs: Use rename lock and RCU for multi-step operations Nick Piggin
2011-01-18 22:32 ` Yehuda Sadeh Weinraub
2011-01-18 22:42 ` Nick Piggin
2011-01-19 22:27 ` Yehuda Sadeh Weinraub
2011-01-19 22:32 ` Nick Piggin
2011-01-25 22:10 ` Yehuda Sadeh Weinraub
2011-01-27 5:18 ` Nick Piggin
2011-02-07 18:52 ` Jim Schutt
2011-02-07 21:04 ` Yehuda Sadeh Weinraub
2011-02-07 21:31 ` Jim Schutt
2011-02-07 22:25 ` Jim Schutt
2011-02-14 17:57 ` Yehuda Sadeh Weinraub
2010-11-27 9:44 ` [PATCH 18/46] fs: increase d_name lock coverage Nick Piggin
2010-11-27 9:44 ` [PATCH 19/46] fs: dcache remove dcache_lock Nick Piggin
2010-11-27 9:44 ` [PATCH 20/46] fs: dcache avoid starvation in dcache multi-step operations Nick Piggin
2010-11-27 9:44 ` [PATCH 21/46] fs: dcache reduce dput locking Nick Piggin
2010-11-27 9:44 ` [PATCH 22/46] fs: dcache reduce locking in d_alloc Nick Piggin
2010-11-27 9:44 ` [PATCH 23/46] fs: dcache reduce dcache_inode_lock Nick Piggin
2010-11-27 9:44 ` [PATCH 24/46] fs: dcache rationalise dget variants Nick Piggin
2010-11-27 9:44 ` [PATCH 25/46] fs: dcache reduce d_parent locking Nick Piggin
2010-11-27 9:44 ` [PATCH 26/46] fs: dcache reduce prune_one_dentry locking Nick Piggin
2010-11-27 9:44 ` [PATCH 27/46] fs: reduce dcache_inode_lock width in lru scanning Nick Piggin
2010-11-27 9:44 ` [PATCH 28/46] fs: use RCU in shrink_dentry_list to reduce lock nesting Nick Piggin
2010-11-27 9:44 ` [PATCH 29/46] fs: consolidate dentry kill sequence Nick Piggin
2010-11-27 9:45 ` [PATCH 30/46] fs: icache RCU free inodes Nick Piggin
2010-11-27 9:45 ` [PATCH 31/46] fs: avoid inode RCU freeing for pseudo fs Nick Piggin
2010-11-27 9:45 ` [PATCH 32/46] kernel: optimise seqlock Nick Piggin
2010-11-27 9:45 ` [PATCH 33/46] fs: rcu-walk for path lookup Nick Piggin
2010-11-27 9:45 ` [PATCH 34/46] fs: fs_struct use seqlock Nick Piggin
2010-11-27 9:45 ` [PATCH 35/46] fs: dcache remove d_mounted Nick Piggin
2010-11-27 9:45 ` [PATCH 36/46] fs: dcache reduce branches in lookup path Nick Piggin
2010-11-27 9:45 ` [PATCH 37/46] fs: cache optimise dentry and inode for rcu-walk Nick Piggin
2010-11-27 9:45 ` [PATCH 38/46] fs: prefetch inode data in dcache lookup Nick Piggin
2010-11-27 9:45 ` [PATCH 39/46] fs: d_revalidate_rcu for rcu-walk Nick Piggin
2010-11-27 9:45 ` [PATCH 40/46] fs: provide rcu-walk aware permission i_ops Nick Piggin
2010-11-27 9:45 ` [PATCH 41/46] fs: provide simple rcu-walk ACL implementation Nick Piggin
2010-11-27 9:45 ` [PATCH 42/46] kernel: add bl_list Nick Piggin
2010-11-27 9:45 ` [PATCH 43/46] bit_spinlock: add required includes Nick Piggin
2010-11-27 9:45 ` [PATCH 44/46] fs: dcache per-bucket dcache hash locking Nick Piggin
2010-11-27 9:45 ` [PATCH 45/46] fs: dcache per-inode inode alias locking Nick Piggin
2010-11-27 9:45 ` [PATCH 46/46] fs: improve scalability of pseudo filesystems Nick Piggin
2010-11-27 9:56 ` [PATCH 01/46] Revert "fs: use RCU read side protection in d_validate" Nick Piggin
2010-12-08 1:16 ` Dave Chinner
2010-12-08 9:38 ` Nick Piggin
2010-12-09 0:44 ` Dave Chinner
2010-12-09 4:38 ` Nick Piggin
2010-12-09 5:16 ` Nick Piggin
2010-11-27 15:04 ` [PATCH 00/46] rcu-walk and dcache scaling Anca Emanuel
2010-11-28 3:28 ` Nick Piggin
2010-11-28 6:24 ` Sedat Dilek
2010-12-01 18:03 ` David Miller
2010-12-03 16:55 ` Nick Piggin
2010-12-07 11:25 ` Dave Chinner
2010-12-07 15:24 ` Nick Piggin [this message]
2010-12-07 15:49 ` Peter Zijlstra
2010-12-07 15:59 ` Nick Piggin
2010-12-07 16:23 ` Peter Zijlstra
2010-12-08 3:28 ` Nick Piggin
2010-12-07 21:56 ` Dave Chinner
2010-12-08 1:47 ` Nick Piggin
2010-12-08 3:32 ` Dave Chinner
2010-12-08 4:28 ` Dave Chinner
2010-12-08 7:09 ` Nick Piggin
2010-12-10 20:32 ` Paul E. McKenney
2010-12-12 14:54 ` Paul E. McKenney
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='AANLkTinAt9PEWMCP55o64E7wsfZ=depU+YXbG8DdGYdO@mail.gmail.com' \
--to=npiggin@gmail.com \
--cc=a.p.zijlstra@chello.nl \
--cc=david@fromorbit.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=npiggin@kernel.dk \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).