From: Mike Snitzer <snitzer@redhat.com>
To: Tejun Heo <tj@kernel.org>
Cc: jaxboe@fusionio.com, k-ueda@ct.jp.nec.com,
j-nomura@ce.jp.nec.com, jamie@shareable.org,
linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org,
linux-raid@vger.kernel.org, hch@lst.de, dm-devel@redhat.com
Subject: Re: [PATCH 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm
Date: Mon, 30 Aug 2010 17:28:20 -0400 [thread overview]
Message-ID: <20100830212819.GA12522@redhat.com> (raw)
In-Reply-To: <20100830190835.GA8458@redhat.com>
On Mon, Aug 30 2010 at 3:08pm -0400,
Mike Snitzer <snitzer@redhat.com> wrote:
> On Mon, Aug 30 2010 at 11:07am -0400,
> Tejun Heo <tj@kernel.org> wrote:
>
> > On 08/30/2010 03:59 PM, Tejun Heo wrote:
> > > Ah... that's probably from "if (!elv_queue_empty(q))" check below,
> > > flushes are on a separate queue but I forgot to update
> > > elv_queue_empty() to check the flush queue. elv_queue_empty() can
> > > return %true spuriously in which case the queue won't be plugged and
> > > restarted later leading to queue hang. I'll fix elv_queue_empty().
> >
> > I think I was too quick to blame elv_queue_empty(). Can you please
> > test whether the following patch fixes the hang?
>
> It does, thanks!
Hmm, but unfortunately I was too quick to say the patch fixed the hang.
It is much more rare, but I can still get a hang. I just got the
following running vgcreate against an DM mpath (rq-based) device:
INFO: task vgcreate:3517 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
vgcreate D ffff88003d677a00 5168 3517 3361 0x00000080
ffff88003d677998 0000000000000046 ffff880000000000 ffff88003d677fd8
ffff880039c84860 ffff88003d677fd8 00000000001d3880 ffff880039c84c30
ffff880039c84c28 00000000001d3880 00000000001d3880 ffff88003d677fd8
Call Trace:
[<ffffffff81389308>] io_schedule+0x73/0xb5
[<ffffffff811c7304>] get_request_wait+0xef/0x17d
[<ffffffff810642be>] ? autoremove_wake_function+0x0/0x39
[<ffffffff811c7890>] __make_request+0x333/0x467
[<ffffffff810251e5>] ? pvclock_clocksource_read+0x50/0xb9
[<ffffffff811c5e91>] generic_make_request+0x342/0x3bf
[<ffffffff81074714>] ? trace_hardirqs_off+0xd/0xf
[<ffffffff81069df2>] ? local_clock+0x41/0x5a
[<ffffffff811c5fe9>] submit_bio+0xdb/0xf8
[<ffffffff810754a4>] ? trace_hardirqs_on+0xd/0xf
[<ffffffff811381a6>] dio_bio_submit+0x7b/0x9c
[<ffffffff81138dbe>] __blockdev_direct_IO+0x7f3/0x97d
[<ffffffff810251e5>] ? pvclock_clocksource_read+0x50/0xb9
[<ffffffff81136d7a>] blkdev_direct_IO+0x57/0x59
[<ffffffff81135f58>] ? blkdev_get_blocks+0x0/0x90
[<ffffffff810ce301>] generic_file_aio_read+0xed/0x5b4
[<ffffffff81077932>] ? lock_release_non_nested+0xd5/0x23b
[<ffffffff810e40f8>] ? might_fault+0x5c/0xac
[<ffffffff810251e5>] ? pvclock_clocksource_read+0x50/0xb9
[<ffffffff8110e131>] do_sync_read+0xcb/0x108
[<ffffffff81074688>] ? trace_hardirqs_off_caller+0x1f/0x9e
[<ffffffff81389a99>] ? __mutex_unlock_slowpath+0x120/0x132
[<ffffffff8119d805>] ? fsnotify_perm+0x4a/0x50
[<ffffffff8119d86c>] ? security_file_permission+0x2e/0x33
[<ffffffff8110e7a3>] vfs_read+0xab/0x107
[<ffffffff81075473>] ? trace_hardirqs_on_caller+0x11d/0x141
[<ffffffff8110e8c2>] sys_read+0x4d/0x74
[<ffffffff81002c32>] system_call_fastpath+0x16/0x1b
no locks held by vgcreate/3517.
I was then able to reproduce it after reboot and another ~5 attempts
(all against 2.6.36-rc2 + your latest FLUSH+FUA patchset and DM
patches).
crash> bt -l 3893
PID: 3893 TASK: ffff88003e65a430 CPU: 0 COMMAND: "vgcreate"
#0 [ffff88003a5298d8] schedule at ffffffff813891d3
/root/git/linux-2.6/kernel/sched.c: 2873
#1 [ffff88003a5299a0] io_schedule at ffffffff81389308
/root/git/linux-2.6/kernel/sched.c: 5128
#2 [ffff88003a5299c0] get_request_wait at ffffffff811c7304
/root/git/linux-2.6/block/blk-core.c: 879
#3 [ffff88003a529a50] __make_request at ffffffff811c7890
/root/git/linux-2.6/block/blk-core.c: 1301
#4 [ffff88003a529ac0] generic_make_request at ffffffff811c5e91
/root/git/linux-2.6/block/blk-core.c: 1536
#5 [ffff88003a529b70] submit_bio at ffffffff811c5fe9
/root/git/linux-2.6/block/blk-core.c: 1632
#6 [ffff88003a529bc0] dio_bio_submit at ffffffff811381a6
/root/git/linux-2.6/fs/direct-io.c: 375
#7 [ffff88003a529bf0] __blockdev_direct_IO at ffffffff81138dbe
/root/git/linux-2.6/fs/direct-io.c: 1087
#8 [ffff88003a529cd0] blkdev_direct_IO at ffffffff81136d7a
/root/git/linux-2.6/fs/block_dev.c: 177
#9 [ffff88003a529d10] generic_file_aio_read at ffffffff810ce301
/root/git/linux-2.6/mm/filemap.c: 1303
#10 [ffff88003a529df0] do_sync_read at ffffffff8110e131
/root/git/linux-2.6/fs/read_write.c: 282
#11 [ffff88003a529f00] vfs_read at ffffffff8110e7a3
/root/git/linux-2.6/fs/read_write.c: 310
#12 [ffff88003a529f40] sys_read at ffffffff8110e8c2
/root/git/linux-2.6/fs/read_write.c: 388
#13 [ffff88003a529f80] system_call_fastpath at ffffffff81002c32
/root/git/linux-2.6/arch/x86/kernel/entry_64.S: 488
RIP: 0000003b602d41a0 RSP: 00007fff55d5b928 RFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff81002c32 RCX: 00007fff55d5b960
RDX: 0000000000001000 RSI: 00007fff55d5a000 RDI: 0000000000000005
RBP: 0000000000000000 R8: 0000000000494ecd R9: 0000000000001000
R10: 000000315c41c160 R11: 0000000000000246 R12: 00007fff55d5a000
R13: 00007fff55d5b0a0 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 0000000000000000 CS: 0033 SS: 002b
next prev parent reply other threads:[~2010-08-30 21:28 UTC|newest]
Thread overview: 88+ messages / expand[flat|nested] mbox.gz Atom feed top
2010-08-30 9:58 [PATCHSET 2.6.36-rc2] block, dm: finish REQ_FLUSH/FUA conversion, take#2 Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` [PATCH 1/5] block: make __blk_rq_prep_clone() copy most command flags Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-09-01 15:30 ` Christoph Hellwig
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` [PATCH 2/5] dm: implement REQ_FLUSH/FUA support for bio-based dm Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-09-01 13:43 ` Mike Snitzer
2010-09-01 13:50 ` Tejun Heo
2010-09-01 13:54 ` Mike Snitzer
2010-09-01 13:56 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` [PATCH 3/5] dm: relax ordering of bio-based flush implementation Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-09-01 13:51 ` Mike Snitzer
2010-09-01 13:51 ` Mike Snitzer
2010-09-01 13:56 ` Tejun Heo
2010-09-01 13:56 ` Tejun Heo
2010-09-03 6:04 ` Kiyoshi Ueda
2010-09-03 9:42 ` Tejun Heo
2010-08-30 9:58 ` [PATCH 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 13:28 ` Mike Snitzer
2010-08-30 13:59 ` Tejun Heo
2010-08-30 15:07 ` Tejun Heo
2010-08-30 19:08 ` Mike Snitzer
2010-08-30 21:28 ` Mike Snitzer [this message]
2010-08-31 10:29 ` Tejun Heo
2010-08-31 13:02 ` Mike Snitzer
2010-08-31 13:14 ` Tejun Heo
2010-08-30 19:08 ` Mike Snitzer
2010-08-30 15:07 ` Tejun Heo
2010-08-30 15:42 ` [PATCH] block: initialize flush request with WRITE_FLUSH instead of REQ_FLUSH Tejun Heo
2010-08-30 15:42 ` Tejun Heo
2010-08-30 15:45 ` [PATCH UPDATED 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm Tejun Heo
2010-08-30 19:18 ` Mike Snitzer
2010-08-30 19:18 ` Mike Snitzer
2010-09-01 7:15 ` Kiyoshi Ueda
2010-09-01 12:25 ` Mike Snitzer
2010-09-02 13:22 ` Tejun Heo
2010-09-02 13:32 ` Tejun Heo
2010-09-03 5:46 ` Kiyoshi Ueda
2010-09-02 17:43 ` [PATCH] block: make sure FSEQ_DATA request has the same rq_disk as the original Tejun Heo
2010-09-03 5:47 ` Kiyoshi Ueda
2010-09-03 9:33 ` Tejun Heo
2010-09-03 10:28 ` Kiyoshi Ueda
2010-09-03 11:42 ` Tejun Heo
2010-09-03 11:51 ` Kiyoshi Ueda
[not found] ` <20100830194731.GA10702@redhat.com>
2010-09-01 10:31 ` [PATCH UPDATED 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm Mikulas Patocka
2010-09-01 11:20 ` Tejun Heo
2010-09-01 12:12 ` Mikulas Patocka
2010-09-01 12:42 ` Tejun Heo
2010-09-01 12:54 ` Mike Snitzer
2010-09-01 15:20 ` Mike Snitzer
2010-09-01 15:35 ` Mikulas Patocka
2010-09-01 17:07 ` Mike Snitzer
2010-09-01 18:59 ` Mike Snitzer
2010-09-02 3:22 ` Mike Snitzer
2010-09-02 10:24 ` Tejun Heo
2010-09-02 15:11 ` Mike Snitzer
2010-09-09 15:26 ` [REGRESSION][BISECTED] virtio-blk serial attribute causes guest to hang [Was: Re: [PATCH UPDATED 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm] Mike Snitzer
2010-09-09 15:44 ` Ryan Harper
2010-09-09 15:57 ` Mike Snitzer
2010-09-09 16:03 ` Ryan Harper
2010-09-09 17:55 ` Mike Snitzer
2010-09-09 18:35 ` Ryan Harper
2010-09-09 19:15 ` Mike Snitzer
2010-09-09 19:43 ` Mike Snitzer
2010-09-09 20:14 ` Mike Snitzer
2010-09-09 20:30 ` Ryan Harper
2010-09-09 21:00 ` [PATCH] virtio-blk: put request that was created to retrieve the device id Mike Snitzer
2010-09-09 21:15 ` Christoph Hellwig
2010-09-17 14:58 ` Ryan Harper
2010-09-21 21:00 ` Christoph Hellwig
2010-10-08 16:06 ` [2.6.36 REGRESSION] " Mike Snitzer
2010-10-09 1:41 ` [PATCH] " Rusty Russell
2010-08-30 15:45 ` [PATCH UPDATED 4/5] dm: implement REQ_FLUSH/FUA support for request-based dm Tejun Heo
2010-08-30 13:59 ` [PATCH " Tejun Heo
2010-08-30 13:28 ` Mike Snitzer
2010-08-30 9:58 ` [PATCH 5/5] block: remove the WRITE_BARRIER flag Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
2010-08-30 9:58 ` Tejun Heo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20100830212819.GA12522@redhat.com \
--to=snitzer@redhat.com \
--cc=dm-devel@redhat.com \
--cc=hch@lst.de \
--cc=j-nomura@ce.jp.nec.com \
--cc=jamie@shareable.org \
--cc=jaxboe@fusionio.com \
--cc=k-ueda@ct.jp.nec.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-raid@vger.kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.