From: Kashyap Desai <kashyap.desai@broadcom.com>
To: John Garry <john.garry@huawei.com>,
Hannes Reinecke <hare@suse.com>, Ming Lei <ming.lei@redhat.com>,
Sumit Saxena <sumit.saxena@broadcom.com>,
Bart Van Assche <bvanassche@acm.org>
Cc: chenxiang <chenxiang66@hisilicon.com>,
linux-scsi@vger.kernel.org, linux-block@vger.kernel.org,
Ewan Milne <emilne@redhat.com>, Long Li <longli@microsoft.com>,
"Martin K . Petersen" <martin.petersen@oracle.com>
Subject: RE: [bug report] Hang on sync after dd
Date: Tue, 1 Dec 2020 15:56:50 +0530 [thread overview]
Message-ID: <6cd6f97324474f88a0a748e218c8dddf@mail.gmail.com> (raw)
In-Reply-To: <2847d0e1-ccb1-7be6-2456-274e41ea981b@huawei.com>
[-- Attachment #1: Type: text/plain, Size: 2875 bytes --]
> @Kashyap, have you guys tested megaraid sas much for this?
John - I tested V4 version "scsi: core: Only re-run queue in
scsi_end_request() if device queue is busy" on MR controller.
I used different reduced device queue depth (1 to 16). I can try the exact
same test case with MR controller.
>
> Thanks,
> John
>
>
> Block debugfs info is as follows:
>
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat active cpu101/ cpu96/
> cpu99/ dispatch_busy io_poll sched_tags tags busy cpu102/ cpu97/ ctx_map
> dispatched queued sched_tags_bitmap tags_bitmap cpu100/ cpu103/ cpu98/
> dispatch flags run state type estuary:/sys/kernel/debug/block/sda/hctx8$
> cat
> cpu cpu100/ cpu101/ cpu102/ cpu103/ cpu96/ cpu97/ cpu98/ cpu99/
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu cpu100/ cpu101/
> cpu102/ cpu103/ cpu96/ cpu97/ cpu98/ cpu99/
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu96/ completed
> default_rq_list dispatched merged poll_rq_list read_rq_list
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu96/dispatched
> 0 0
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu97/dispatched
> 0 0
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu98/dispatched
> 0 0
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu99/dispatched
> 0 0
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu100/dispatched
> 3 0
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat cpu100/completed
> 2 0
> estuary:/sys/kernel/debug/block/sda/hctx8$
> estuary:/sys/kernel/debug/block/sda/hctx8$
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat state SCHED_RESTART
When I tested V3 "scsi: core: Only re-run queue in scsi_end_request() if
device queue is busy". I noticed the similar hang and that was fixed in V4
(final patch).
Let me try on MR controller one more time. Hctx state SCHED_RESTART
indicates that someone should kicked-off h/w queue but it was missed. It may
be possible that
When you revert " scsi: core: Only re-run queue in scsi_end_request() if
device queue is busy", actual race condition windows narrows and it may be
actually existing hidden issue.
> estuary:/sys/kernel/debug/block/sda/hctx8$ ls active cpu101 cpu96 cpu99
> dispatch_busy io_poll sched_tags tags busy cpu102 cpu97 ctx_map
> dispatched queued sched_tags_bitmap tags_bitmap
> cpu100 cpu103 cpu98 dispatch flags run state type
> estuary:/sys/kernel/debug/block/sda/hctx8$ cat dispatch 000000007abb596e
> {.op=FLUSH, .cmd_flags=PREFLUSH,
> .rq_flags=FLUSH_SEQ|MQ_INFLIGHT|DONTPREP, .state=idle, .tag=21,
> .internal_tag=-1, .cmd=opcode=0x35 35 00 00 00 00 00 00 00 00 00,
> .retries=0, .result = 0x0, .flags=TAGGED|INITIALIZED|3, .timeout=60.000,
If this issue is reproducible, can you check pending commands. Is there any
pattern in pending command ?
> allocated 2208.876 s ago} estuary:/sys/kernel/debug/block/sda/hctx8$
>
>
> On cpu100, it seems completed is less than number dispatched.
[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4169 bytes --]
next prev parent reply other threads:[~2020-12-01 10:28 UTC|newest]
Thread overview: 9+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-30 11:22 [bug report] Hang on sync after dd John Garry
2020-12-01 10:26 ` Kashyap Desai [this message]
2020-12-01 11:48 ` John Garry
2020-12-01 12:34 ` Ming Lei
2020-12-02 1:44 ` chenxiang (M)
2020-12-02 3:22 ` Ming Lei
2020-12-02 6:22 ` chenxiang (M)
2020-12-02 7:26 ` Ming Lei
2020-12-02 9:06 ` chenxiang (M)
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=6cd6f97324474f88a0a748e218c8dddf@mail.gmail.com \
--to=kashyap.desai@broadcom.com \
--cc=bvanassche@acm.org \
--cc=chenxiang66@hisilicon.com \
--cc=emilne@redhat.com \
--cc=hare@suse.com \
--cc=john.garry@huawei.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-scsi@vger.kernel.org \
--cc=longli@microsoft.com \
--cc=martin.petersen@oracle.com \
--cc=ming.lei@redhat.com \
--cc=sumit.saxena@broadcom.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).