From: Keith Busch <kbusch@kernel.org>
To: Bharat Kumar Gogada <bharatku@xilinx.com>
Cc: "keith.busch@intel.com" <keith.busch@intel.com>,
Keith Busch <keith.busch@linux.intel.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>
Subject: Re: NVMe Poll CQ on timeout
Date: Thu, 19 Sep 2019 08:13:01 -0600 [thread overview]
Message-ID: <20190919141301.GA61660@C02WT3WMHTD6> (raw)
In-Reply-To: <MN2PR02MB633689DBBA6DE9DD7A34043FA5890@MN2PR02MB6336.namprd02.prod.outlook.com>
On Thu, Sep 19, 2019 at 01:47:50PM +0000, Bharat Kumar Gogada wrote:
> Hi All,
>
> We are testing NVMe cards on ARM64 platform, the card uses MSI-X interrupts.
> We are hitting following case in drivers/nvme/host/pci.c
> /*
> * Did we miss an interrupt?
> */
> if (__nvme_poll(nvmeq, req->tag)) {
> dev_warn(dev->ctrl.device,
> "I/O %d QID %d timeout, completion polled\n",
> req->tag, nvmeq->qid);
> return BLK_EH_DONE;
> }
>
> Can anyone tell when does nvme_timeout gets invoked ?
Timeout is invoked when the driver didn't see a completion to a
submitted command.
> In what cases we see this interrupt miss ?
That usually happens for one of two reasons:
1. The device didn't send any MSIx message for a CQE
2. The device sent the MSIx message before posting the CQE
I've also seen h/w errata where the MSIx and CQE are re-ordered, which
can also lead to this.
A hardware trace would provide the most detailed view of what's
happening. You might be able to infer if you carefully account for
commands sent, interrupts received, and spurious interrupts detected.
> We are seeing this issue only for reads with following fio command
> fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randread --bs=128k --direct=0 \
> --size=128M --numjobs=3 --group_reporting --filename=/dev/nvme0n1
>
> We are not seeing issue with --rw=randwrite for same size.
>
> Please let us know what can cause this issue.
_______________________________________________
Linux-nvme mailing list
Linux-nvme@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme
WARNING: multiple messages have this Message-ID (diff)
From: Keith Busch <kbusch@kernel.org>
To: Bharat Kumar Gogada <bharatku@xilinx.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-nvme@lists.infradead.org" <linux-nvme@lists.infradead.org>,
Keith Busch <keith.busch@linux.intel.com>,
"keith.busch@intel.com" <keith.busch@intel.com>
Subject: Re: NVMe Poll CQ on timeout
Date: Thu, 19 Sep 2019 08:13:01 -0600 [thread overview]
Message-ID: <20190919141301.GA61660@C02WT3WMHTD6> (raw)
In-Reply-To: <MN2PR02MB633689DBBA6DE9DD7A34043FA5890@MN2PR02MB6336.namprd02.prod.outlook.com>
On Thu, Sep 19, 2019 at 01:47:50PM +0000, Bharat Kumar Gogada wrote:
> Hi All,
>
> We are testing NVMe cards on ARM64 platform, the card uses MSI-X interrupts.
> We are hitting following case in drivers/nvme/host/pci.c
> /*
> * Did we miss an interrupt?
> */
> if (__nvme_poll(nvmeq, req->tag)) {
> dev_warn(dev->ctrl.device,
> "I/O %d QID %d timeout, completion polled\n",
> req->tag, nvmeq->qid);
> return BLK_EH_DONE;
> }
>
> Can anyone tell when does nvme_timeout gets invoked ?
Timeout is invoked when the driver didn't see a completion to a
submitted command.
> In what cases we see this interrupt miss ?
That usually happens for one of two reasons:
1. The device didn't send any MSIx message for a CQE
2. The device sent the MSIx message before posting the CQE
I've also seen h/w errata where the MSIx and CQE are re-ordered, which
can also lead to this.
A hardware trace would provide the most detailed view of what's
happening. You might be able to infer if you carefully account for
commands sent, interrupts received, and spurious interrupts detected.
> We are seeing this issue only for reads with following fio command
> fio --name=randwrite --ioengine=libaio --iodepth=1 --rw=randread --bs=128k --direct=0 \
> --size=128M --numjobs=3 --group_reporting --filename=/dev/nvme0n1
>
> We are not seeing issue with --rw=randwrite for same size.
>
> Please let us know what can cause this issue.
next prev parent reply other threads:[~2019-09-19 14:13 UTC|newest]
Thread overview: 12+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-09-19 13:47 NVMe Poll CQ on timeout Bharat Kumar Gogada
2019-09-19 13:47 ` Bharat Kumar Gogada
2019-09-19 14:13 ` Keith Busch [this message]
2019-09-19 14:13 ` Keith Busch
-- strict thread matches above, loose matches on Subject: below --
2018-04-26 14:59 Bharat Kumar Gogada
2018-04-26 14:59 ` Bharat Kumar Gogada
2018-05-07 6:57 ` Bharat Kumar Gogada
2018-05-07 6:57 ` Bharat Kumar Gogada
2018-05-07 16:02 ` Keith Busch
2018-05-07 16:02 ` Keith Busch
2018-05-15 13:58 ` Bharat Kumar Gogada
2018-05-15 13:58 ` Bharat Kumar Gogada
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190919141301.GA61660@C02WT3WMHTD6 \
--to=kbusch@kernel.org \
--cc=bharatku@xilinx.com \
--cc=keith.busch@intel.com \
--cc=keith.busch@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvme@lists.infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.