From: Ming Lei <ming.lei@redhat.com>
To: Jaroslav Pulchart <jaroslav.pulchart@gooddata.com>
Cc: Bart Van Assche <bvanassche@acm.org>,
linux-block@vger.kernel.org, Igor Raits <igor.raits@gooddata.com>,
Daniel Secik <daniel.secik@gooddata.com>,
David Krupicka <david.krupicka@gooddata.com>
Subject: Re: again? - Write I/O queue hangup at random on recent Linus' kernels
Date: Wed, 12 Oct 2022 21:07:33 +0800 [thread overview]
Message-ID: <Y0a8Fax+YbB0M831@T590> (raw)
In-Reply-To: <CAK8fFZ6ruxHsXuGT4qarNxdLLQtAoLsSvV0buFQhdc+TKo3Tag@mail.gmail.com>
On Tue, Oct 11, 2022 at 05:15:05PM +0200, Jaroslav Pulchart wrote:
> Hello,
>
> we disabled the wbt, issue is happening much sooner. The logs are attached
> 1/ "dmesg-20221011.log" form kernel messages
> 2/ "command.logs" from execution of
> (cd /sys/kernel/debug/block/vdc && find . -type f -exec grep -aH . {} \;)
>
From command.logs, looks like it is one qemu or virtio-blk issue.
Maybe you can try to revert 0e9911fa768f ("virtio-blk: support mq_ops->queue_rqs()")
and see if it makes difference.
[1] all busy tags points to in-flight IOs
grep busy temp/command.logs | grep -v -E "dispatch_busy|busy=0"
./hctx27/tags:busy=3
./hctx27/busy:00000000e4e3af26 {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=167, .internal_tag=-1}
./hctx27/busy:00000000a9d6975f {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=168, .internal_tag=-1}
./hctx27/busy:00000000474663fc {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=169, .internal_tag=-1}
./hctx26/tags:busy=2
./hctx26/busy:00000000f32dd525 {.op=WRITE, .cmd_flags=SYNC, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=107, .internal_tag=-1}
./hctx26/busy:0000000055297a93 {.op=READ, .cmd_flags=FAILFAST_DEV|FAILFAST_TRANSPORT|FAILFAST_DRIVER|RAHEAD, .rq_flags=IO_STAT|STATS, .state=in_flight, .tag=108, .internal_tag=-1}
...
Thanks,
Ming
prev parent reply other threads:[~2022-10-12 13:07 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-05 19:43 again? - Write I/O queue hangup at random on recent Linus' kernels Jaroslav Pulchart
2022-10-05 21:03 ` Bart Van Assche
2022-10-06 12:36 ` Jaroslav Pulchart
2022-10-06 14:44 ` Ming Lei
2022-10-06 16:57 ` Bart Van Assche
2022-10-11 15:15 ` Jaroslav Pulchart
2022-10-11 16:53 ` Bart Van Assche
2022-10-12 2:05 ` Yu Kuai
2022-10-12 13:07 ` Ming Lei [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y0a8Fax+YbB0M831@T590 \
--to=ming.lei@redhat.com \
--cc=bvanassche@acm.org \
--cc=daniel.secik@gooddata.com \
--cc=david.krupicka@gooddata.com \
--cc=igor.raits@gooddata.com \
--cc=jaroslav.pulchart@gooddata.com \
--cc=linux-block@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.