From: Mike Snitzer <snitzer@redhat.com>
To: Jens Axboe <jaxboe@fusionio.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"hch@infradead.org" <hch@infradead.org>
Subject: Re: [PATCH 05/10] block: remove per-queue plugging
Date: Thu, 17 Mar 2011 14:46:15 -0400 [thread overview]
Message-ID: <20110317184615.GA11359@redhat.com> (raw)
In-Reply-To: <4D825367.90408@fusionio.com>
[-- Attachment #1: Type: text/plain, Size: 4976 bytes --]
On Thu, Mar 17 2011 at 2:31pm -0400,
Jens Axboe <jaxboe@fusionio.com> wrote:
> On 2011-03-17 16:51, Mike Snitzer wrote:
> > On Tue, Mar 08 2011 at 5:05pm -0500,
> > Mike Snitzer <snitzer@redhat.com> wrote:
> >
> >> On Tue, Mar 08 2011 at 3:27pm -0500,
> >> Jens Axboe <jaxboe@fusionio.com> wrote:
> >>
> >>> On 2011-03-08 21:21, Mike Snitzer wrote:
> >>>> On Tue, Mar 08 2011 at 7:16am -0500,
> >>>> Jens Axboe <jaxboe@fusionio.com> wrote:
> >>>>
> >>>>> On 2011-03-03 23:13, Mike Snitzer wrote:
> >>>>>> I'm now hitting a lockdep issue, while running a 'for-2.6.39/stack-plug'
> >>>>>> kernel, when I try an fsync heavy workload to a request-based mpath
> >>>>>> device (the kernel ultimately goes down in flames, I've yet to look at
> >>>>>> the crashdump I took)
> >>>>>
> >>>>> Mike, can you re-run with the current stack-plug branch? I've fixed the
> >>>>> !CONFIG_BLOCK and rebase issues, and also added a change for this flush
> >>>>> on schedule event. It's run outside of the runqueue lock now, so
> >>>>> hopefully that should solve this one.
> >>>>
> >>>> Works for me, thanks.
> >>>
> >>> Super, thanks! Out of curiousity, did you use dm/md?
> >>
> >> Yes, I've been using a request-based DM multipath device.
> >
> >
> > Against latest 'for-2.6.39/core', I just ran that same fsync heavy
> > workload against XFS (ontop of a DM multipath volume). ffsb induced the
> > following hangs (ripple effect causing NetworkManager to get hung up on
> > this data-only XFS volume, etc):
>
> Ugh. Care to send the recipee for how to reproduce this? Essentially
> just looks like IO got stuck.
Here is the sequence to reproduce with the attached fsync-happy.ffsb
(I've been running the following in a KVM guest):
<create multipath device>
mkfs.xfs /dev/mapper/mpathb
mount /dev/mapper/mpathb /mnt/test
./ffsb fsync-happy.ffsb
And I just verified that the deadlock does _not_ seem to occur without
DM multipath -- by directly using an underlying SCSI device instead.
So multipath is exposing this somehow (could just be changing timing?).
Mike
p.s. though I did get this lockdep warning when unmounting the xfs
filesystem:
=================================
[ INFO: inconsistent lock state ]
2.6.38-rc6-snitm+ #8
---------------------------------
inconsistent {IN-RECLAIM_FS-R} -> {RECLAIM_FS-ON-W} usage.
umount/1524 [HC0[0]:SC0[0]:HE1:SE1] takes:
(iprune_sem){+++++-}, at: [<ffffffff81114a22>] evict_inodes+0x2f/0x107
{IN-RECLAIM_FS-R} state was registered at:
[<ffffffff810727c2>] __lock_acquire+0x3a4/0xd26
[<ffffffff81073227>] lock_acquire+0xe3/0x110
[<ffffffff81372fa2>] down_read+0x51/0x96
[<ffffffff81114d57>] shrink_icache_memory+0x4a/0x215
[<ffffffff810d1e48>] shrink_slab+0xe0/0x164
[<ffffffff810d3e8f>] kswapd+0x5e7/0x9dc
[<ffffffff8105fb7c>] kthread+0xa0/0xa8
[<ffffffff81003a24>] kernel_thread_helper+0x4/0x10
irq event stamp: 73433
hardirqs last enabled at (73433): [<ffffffff81070ffe>] debug_check_no_locks_freed+0x12e/0x145
hardirqs last disabled at (73432): [<ffffffff81070f13>] debug_check_no_locks_freed+0x43/0x145
softirqs last enabled at (72996): [<ffffffff8104a1f1>] __do_softirq+0x1b4/0x1d3
softirqs last disabled at (72991): [<ffffffff81003b1c>] call_softirq+0x1c/0x28
other info that might help us debug this:
2 locks held by umount/1524:
#0: (&type->s_umount_key#24){++++++}, at: [<ffffffff81102a27>] deactivate_super+0x3d/0x4a
#1: (iprune_sem){+++++-}, at: [<ffffffff81114a22>] evict_inodes+0x2f/0x107
stack backtrace:
Pid: 1524, comm: umount Not tainted 2.6.38-rc6-snitm+ #8
Call Trace:
[<ffffffff8107097f>] ? valid_state+0x17e/0x191
[<ffffffff810712e8>] ? check_usage_backwards+0x0/0x81
[<ffffffff81070ae4>] ? mark_lock+0x152/0x22d
[<ffffffff81070c11>] ? mark_held_locks+0x52/0x70
[<ffffffff81070cc8>] ? lockdep_trace_alloc+0x99/0xbb
[<ffffffff810fa98a>] ? kmem_cache_alloc+0x30/0x145
[<ffffffffa014dcdd>] ? kmem_zone_alloc+0x69/0xb1 [xfs]
[<ffffffffa014dd39>] ? kmem_zone_zalloc+0x14/0x35 [xfs]
[<ffffffffa0147ed9>] ? _xfs_trans_alloc+0x27/0x64 [xfs]
[<ffffffffa0148c97>] ? xfs_trans_alloc+0x9f/0xac [xfs]
[<ffffffff810643b7>] ? up_read+0x23/0x3c
[<ffffffffa0133000>] ? xfs_iunlock+0x7e/0xbc [xfs]
[<ffffffffa014c140>] ? xfs_free_eofblocks+0xea/0x1f1 [xfs]
[<ffffffffa014c707>] ? xfs_inactive+0x108/0x3a6 [xfs]
[<ffffffff8106ff27>] ? lockdep_init_map+0xa6/0x11b
[<ffffffffa015787f>] ? xfs_fs_evict_inode+0xf6/0xfe [xfs]
[<ffffffff811147c6>] ? evict+0x24/0x8c
[<ffffffff8111485f>] ? dispose_list+0x31/0xaf
[<ffffffff81114ae3>] ? evict_inodes+0xf0/0x107
[<ffffffff81101660>] ? generic_shutdown_super+0x5c/0xdf
[<ffffffff8110170a>] ? kill_block_super+0x27/0x69
[<ffffffff81101d89>] ? deactivate_locked_super+0x26/0x4b
[<ffffffff81102a2f>] ? deactivate_super+0x45/0x4a
[<ffffffff81118b87>] ? mntput_no_expire+0x105/0x10e
[<ffffffff81119db6>] ? sys_umount+0x2d9/0x304
[<ffffffff81070e9d>] ? trace_hardirqs_on_caller+0x11d/0x141
[<ffffffff81002b82>] ? system_call_fastpath+0x16/0x1b
[-- Attachment #2: fsync-happy.ffsb --]
[-- Type: text/plain, Size: 1642 bytes --]
# Mail server simulation.
time=600
alignio=1
directio=0
#directio=%DIRECTIO%
#directio=0
#callout=/bin/bash
#callout=/usr/local/src/ffsb-6.0-rc2/ltc_tests/enable_lockstat.sh
#callout=/usr/local/src/ffsb-6.0-rc2/ltc_tests/osync.sh
[filesystem0]
location=/mnt/test
num_files=100000
num_dirs=1000
reuse=1
# File sizes range from 1kB to 1MB.
size_weight 1KB 10
size_weight 2KB 15
size_weight 4KB 16
size_weight 8KB 16
size_weight 16KB 15
size_weight 32KB 10
size_weight 64KB 8
size_weight 128KB 4
size_weight 256KB 3
size_weight 512KB 2
size_weight 1MB 1
create_blocksize=1048576
[end0]
[threadgroup0]
num_threads=64
readall_weight=4
create_fsync_weight=2
delete_weight=1
append_weight = 1
append_fsync_weight = 1
stat_weight = 1
# write_weight = 1
# write_fsync_weight = 1
# read_weight = 1
create_weight = 1
writeall_weight = 1
writeall_fsync_weight = 1
open_close_weight = 1
write_size=64KB
write_blocksize=512KB
read_size=64KB
read_blocksize=512KB
[stats]
enable_stats=1
enable_range=1
msec_range 0.00 0.01
msec_range 0.01 0.02
msec_range 0.02 0.05
msec_range 0.05 0.10
msec_range 0.10 0.20
msec_range 0.20 0.50
msec_range 0.50 1.00
msec_range 1.00 2.00
msec_range 2.00 5.00
msec_range 5.00 10.00
msec_range 10.00 20.00
msec_range 20.00 50.00
msec_range 50.00 100.00
msec_range 100.00 200.00
msec_range 200.00 500.00
msec_range 500.00 1000.00
msec_range 1000.00 2000.00
msec_range 2000.00 5000.00
msec_range 5000.00 10000.00
[end]
[end0]
next prev parent reply other threads:[~2011-03-17 18:46 UTC|newest]
Thread overview: 152+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-01-22 1:17 [PATCH 0/10] On-stack explicit block queue plugging Jens Axboe
2011-01-22 1:17 ` [PATCH 01/10] block: add API for delaying work/request_fn a little bit Jens Axboe
2011-01-22 1:17 ` [PATCH 02/10] ide-cd: convert to blk_delay_queue() for a short pause Jens Axboe
2011-01-22 1:19 ` David Miller
2011-01-22 1:17 ` [PATCH 03/10] scsi: convert to blk_delay_queue() Jens Axboe
2011-01-22 1:17 ` [PATCH 04/10] block: initial patch for on-stack per-task plugging Jens Axboe
2011-01-24 19:36 ` Jeff Moyer
2011-01-24 21:23 ` Jens Axboe
2011-03-10 16:54 ` Vivek Goyal
2011-03-10 19:32 ` Jens Axboe
2011-03-10 19:46 ` Vivek Goyal
2011-03-16 8:18 ` Shaohua Li
2011-03-16 17:31 ` Vivek Goyal
2011-03-17 1:00 ` Shaohua Li
2011-03-17 3:19 ` Shaohua Li
2011-03-17 9:44 ` Jens Axboe
2011-03-18 1:55 ` Shaohua Li
2011-03-17 9:43 ` Jens Axboe
2011-03-18 6:36 ` Shaohua Li
2011-03-18 12:54 ` Jens Axboe
2011-03-18 13:52 ` Jens Axboe
2011-03-21 6:52 ` Shaohua Li
2011-03-21 9:20 ` Jens Axboe
2011-03-22 0:32 ` Shaohua Li
2011-03-22 7:36 ` Jens Axboe
2011-03-17 9:39 ` Jens Axboe
2011-01-22 1:17 ` [PATCH 05/10] block: remove per-queue plugging Jens Axboe
2011-01-22 1:31 ` Nick Piggin
2011-03-03 21:23 ` Mike Snitzer
2011-03-03 21:27 ` Mike Snitzer
2011-03-03 22:13 ` Mike Snitzer
2011-03-04 13:02 ` Shaohua Li
2011-03-04 13:20 ` Jens Axboe
2011-03-04 21:43 ` Mike Snitzer
2011-03-04 21:50 ` Jens Axboe
2011-03-04 22:27 ` Mike Snitzer
2011-03-05 20:54 ` Jens Axboe
2011-03-07 10:23 ` Peter Zijlstra
2011-03-07 19:43 ` Jens Axboe
2011-03-07 20:41 ` Peter Zijlstra
2011-03-07 20:46 ` Jens Axboe
2011-03-08 9:38 ` Peter Zijlstra
2011-03-08 9:41 ` Jens Axboe
2011-03-07 0:54 ` Shaohua Li
2011-03-07 8:07 ` Jens Axboe
2011-03-08 12:16 ` Jens Axboe
2011-03-08 20:21 ` Mike Snitzer
2011-03-08 20:27 ` Jens Axboe
2011-03-08 21:36 ` Jeff Moyer
2011-03-09 7:25 ` Jens Axboe
2011-03-08 22:05 ` Mike Snitzer
2011-03-10 0:58 ` Mike Snitzer
2011-04-05 3:05 ` NeilBrown
2011-04-11 4:50 ` NeilBrown
2011-04-11 9:19 ` Jens Axboe
2011-04-11 10:59 ` NeilBrown
2011-04-11 11:04 ` Jens Axboe
2011-04-11 11:26 ` NeilBrown
2011-04-11 11:37 ` Jens Axboe
2011-04-11 12:05 ` NeilBrown
2011-04-11 12:11 ` Jens Axboe
2011-04-11 12:36 ` NeilBrown
2011-04-11 12:48 ` Jens Axboe
2011-04-12 1:12 ` hch
2011-04-12 8:36 ` Jens Axboe
2011-04-12 12:22 ` Dave Chinner
2011-04-12 12:28 ` Jens Axboe
2011-04-12 12:41 ` Dave Chinner
2011-04-12 12:58 ` Jens Axboe
2011-04-12 13:31 ` Dave Chinner
2011-04-12 13:45 ` Jens Axboe
2011-04-12 14:34 ` Dave Chinner
2011-04-12 21:08 ` NeilBrown
2011-04-13 2:23 ` Linus Torvalds
2011-04-13 11:12 ` Peter Zijlstra
2011-04-13 11:23 ` Jens Axboe
2011-04-13 11:41 ` Peter Zijlstra
2011-04-13 15:13 ` Linus Torvalds
2011-04-13 17:35 ` Jens Axboe
2011-04-12 16:58 ` hch
2011-04-12 17:29 ` Jens Axboe
2011-04-12 16:44 ` hch
2011-04-12 16:49 ` Jens Axboe
2011-04-12 16:54 ` hch
2011-04-12 17:24 ` Jens Axboe
2011-04-12 13:40 ` Dave Chinner
2011-04-12 13:48 ` Jens Axboe
2011-04-12 23:35 ` Dave Chinner
2011-04-12 16:50 ` hch
2011-04-15 4:26 ` hch
2011-04-15 6:34 ` Jens Axboe
2011-04-17 22:19 ` NeilBrown
2011-04-18 4:19 ` NeilBrown
2011-04-18 6:38 ` Jens Axboe
2011-04-18 7:25 ` NeilBrown
2011-04-18 8:10 ` Jens Axboe
2011-04-18 8:33 ` NeilBrown
2011-04-18 8:42 ` Jens Axboe
2011-04-18 21:23 ` hch
2011-04-22 15:39 ` hch
2011-04-22 16:01 ` Vivek Goyal
2011-04-22 16:10 ` Vivek Goyal
2011-04-18 21:30 ` hch
2011-04-18 22:38 ` NeilBrown
2011-04-20 10:55 ` hch
2011-04-18 9:19 ` hch
2011-04-18 9:40 ` [dm-devel] " Hannes Reinecke
2011-04-18 9:47 ` Jens Axboe
2011-04-18 9:46 ` Jens Axboe
2011-04-11 11:55 ` NeilBrown
2011-04-11 12:12 ` Jens Axboe
2011-04-11 22:58 ` hch
2011-04-12 6:20 ` Jens Axboe
2011-04-11 16:59 ` hch
2011-04-11 21:14 ` NeilBrown
2011-04-11 22:59 ` hch
2011-04-12 6:18 ` Jens Axboe
2011-03-17 15:51 ` Mike Snitzer
2011-03-17 18:31 ` Jens Axboe
2011-03-17 18:46 ` Mike Snitzer [this message]
2011-03-18 9:15 ` hch
2011-03-08 12:15 ` Jens Axboe
2011-03-04 4:00 ` Vivek Goyal
2011-03-08 12:24 ` Jens Axboe
2011-03-08 22:10 ` blk-throttle: Use blk_plug in throttle code (Was: Re: [PATCH 05/10] block: remove per-queue plugging) Vivek Goyal
2011-03-09 7:26 ` Jens Axboe
2011-01-22 1:17 ` [PATCH 06/10] block: kill request allocation batching Jens Axboe
2011-01-22 9:31 ` Christoph Hellwig
2011-01-24 19:09 ` Jens Axboe
2011-01-22 1:17 ` [PATCH 07/10] fs: make generic file read/write functions plug Jens Axboe
2011-01-24 3:57 ` Dave Chinner
2011-01-24 19:11 ` Jens Axboe
2011-03-04 4:09 ` Vivek Goyal
2011-03-04 13:22 ` Jens Axboe
2011-03-04 13:25 ` hch
2011-03-04 13:40 ` Jens Axboe
2011-03-04 14:08 ` hch
2011-03-04 22:07 ` Jens Axboe
2011-03-04 23:12 ` hch
2011-03-08 12:38 ` Jens Axboe
2011-03-09 10:38 ` hch
2011-03-09 10:52 ` Jens Axboe
2011-01-22 1:17 ` [PATCH 08/10] read-ahead: use plugging Jens Axboe
2011-01-22 1:17 ` [PATCH 09/10] fs: make mpage read/write_pages() plug Jens Axboe
2011-01-22 1:17 ` [PATCH 10/10] fs: make aio plug Jens Axboe
2011-01-24 17:59 ` Jeff Moyer
2011-01-24 19:09 ` Jens Axboe
2011-01-24 19:15 ` Jeff Moyer
2011-01-24 19:22 ` Jens Axboe
2011-01-24 19:29 ` Jeff Moyer
2011-01-24 19:31 ` Jens Axboe
2011-01-24 19:38 ` Jeff Moyer
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110317184615.GA11359@redhat.com \
--to=snitzer@redhat.com \
--cc=hch@infradead.org \
--cc=jaxboe@fusionio.com \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).