From: Dave Chinner <david@fromorbit.com>
To: Jens Axboe <axboe@kernel.dk>
Cc: Christoph Hellwig <hch@infradead.org>, linux-kernel@vger.kernel.org
Subject: Re: [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices
Date: Wed, 20 Nov 2013 08:30:51 +1100 [thread overview]
Message-ID: <20131119213051.GP11434@dastard> (raw)
In-Reply-To: <20131119160509.GB21343@kernel.dk>
On Tue, Nov 19, 2013 at 09:05:09AM -0700, Jens Axboe wrote:
> On Tue, Nov 19 2013, Christoph Hellwig wrote:
> > On Tue, Nov 19, 2013 at 07:02:18PM +1100, Dave Chinner wrote:
> > > I have no idea if it's related to the above hang, but either way
> > > breaking iostat is a major regression....
> >
> > Both of them are most likely due to the conversion of virtio_blk
> > to the blk-mq code.
> >
> > I've not seen the hang in my heavy xfstests testing, but that was a
> > slightly different codebase than what finally got in, so I'll try
> > to reproduce it once I get some spare QA cycles.
>
> Thanks! Dave, if you still have it in that state, can you dump the
> contents of /sys/block/<devs>/mq/ for the device/devices that are hung?
# find /sys/block/vdb/mq -type f -print -exec cat {} \;
/sys/block/vdb/mq/0/run
1313835
/sys/block/vdb/mq/0/cpu0/completed
546857 207203
/sys/block/vdb/mq/0/cpu0/rq_list
CTX pending:
/sys/block/vdb/mq/0/cpu0/merged
0
/sys/block/vdb/mq/0/cpu0/dispatched
257733 496352
/sys/block/vdb/mq/0/cpu1/completed
547714 200741
/sys/block/vdb/mq/0/cpu1/rq_list
CTX pending:
/sys/block/vdb/mq/0/cpu1/merged
0
/sys/block/vdb/mq/0/cpu1/dispatched
262114 486381
/sys/block/vdb/mq/0/tags
nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
nr_free=0, nr_reserved=1
cpu00: nr_free=0
cpu01: nr_free=0
/sys/block/vdb/mq/0/ipi_redirect
0
/sys/block/vdb/mq/0/queued
1384795
/sys/block/vdb/mq/0/dispatched
0 194596
1 1022691
2 68540
4 10138
8 7686
16 10234
32 0
64 0
128 0
256 0
/sys/block/vdb/mq/0/pending
HCTX pending:
#
It looks like it's hung because of this:
/sys/block/vdb/mq/0/tags
nr_tags=66, reserved_tags=2, batch_move=16, max_cache=32
nr_free=0, nr_reserved=1
cpu00: nr_free=0
cpu01: nr_free=0
No free tags in teh pool, nor any free tags in the per-cpu
magazines. Perhaps there is a leak somewhere?
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2013-11-19 21:30 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-11-19 8:02 [Regression x2, 3.13-git] virtio block mq hang, iostat busted on virtio devices Dave Chinner
2013-11-19 10:36 ` Christoph Hellwig
2013-11-19 16:05 ` Jens Axboe
2013-11-19 16:09 ` Christoph Hellwig
2013-11-19 16:16 ` Jens Axboe
2013-11-19 21:30 ` Dave Chinner [this message]
2013-11-19 21:40 ` Jens Axboe
2013-11-19 20:15 ` Jens Axboe
2013-11-19 21:20 ` Jens Axboe
2013-11-19 21:34 ` Dave Chinner
2013-11-19 21:43 ` Jens Axboe
2013-11-19 22:42 ` Jens Axboe
2013-11-19 22:51 ` Jens Axboe
2013-11-19 23:23 ` Dave Chinner
2013-11-19 23:59 ` Jens Axboe
2013-11-20 0:08 ` Jens Axboe
2013-11-20 1:44 ` Shaohua Li
2013-11-20 1:54 ` Jens Axboe
2013-11-20 2:02 ` Jens Axboe
2013-11-20 2:53 ` Dave Chinner
2013-11-20 3:12 ` Jens Axboe
2013-11-20 8:07 ` Christoph Hellwig
2013-11-20 16:21 ` Jens Axboe
2013-11-20 8:04 ` Christoph Hellwig
2013-11-20 16:20 ` Jens Axboe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20131119213051.GP11434@dastard \
--to=david@fromorbit.com \
--cc=axboe@kernel.dk \
--cc=hch@infradead.org \
--cc=linux-kernel@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).