From: John Garry <john.g.garry@oracle.com>
To: Nilay Shroff <nilay@linux.ibm.com>, linux-nvme@lists.infradead.org
Cc: hare@suse.de, kbusch@kernel.org, hch@lst.de, sagi@grimberg.me,
dwagner@suse.de, kanie@linux.alibaba.com, jmeneghi@redhat.com,
randyj@purestorage.com, martin.petersen@oracle.com,
gjoyce@linux.ibm.com
Subject: Re: [PATCH v7 4/9] nvme-multipath: add support for latency I/O policy
Date: Mon, 10 Aug 2026 09:12:26 +0100 [thread overview]
Message-ID: <2d0163f0-ecf3-4f59-9121-587f98eeb54c@oracle.com> (raw)
In-Reply-To: <20260809100825.2014133-5-nilay@linux.ibm.com>
On 09/08/2026 11:07, Nilay Shroff wrote:
> + /*
> + * If latency > ~1s then ignore this sample to prevent EWMA from being
> + * skewed by pathological outliers (multi-second waits, controller
> + * timeouts etc.). This keeps path scores representative of normal
> + * performance and avoids instability from rare spikes. If such high
> + * latency is real, ANA state reporting or keep-alive error counters
> + * will mark the path unhealthy and remove it from the head node list,
> + * so we safely skip such sample here.
> + */
> + if (unlikely(latency > NSEC_PER_SEC)) {
> + stat->nr_ignored++;
> + dev_warn_ratelimited(ns->ctrl->device,
> + "ignoring sample with >1s latency (possible controller stall or timeout)\n");
> + return;
> + }
JFYI, I gave this series a spin and I see this warn a lot:
# ./fio_read.sh nvme1n1
iops: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=libaio, iodepth=100
...
fio-3.38
Starting 10 processes
[ 130.005374] nvme_mpath_add_sample: 29 callbacks suppressed03h:08m:36s]
[ 130.005387] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.009968] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.013176] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.019576] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.022275] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.024762] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.027268] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.130140] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.132316] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
[ 130.139279] nvme nvme2: ignoring sample with >1s latency (possible
controller stall or timeout)
^Cbs: 10 (f=10): [R(10)][0.4%][r=35.8MiB/s][r=9171 IOPS][eta 03h:07m:44s]
fio: terminating on signal 2
iops: (groupid=0, jobs=10): err= 0: pid=821: Mon Aug 10 08:08:28 2026
read: IOPS=9239, BW=36.1MiB/s (37.8MB/s)(1434MiB/39744msec)
slat (usec): min=25, max=110836, avg=129.22, stdev=537.18
Note that I do have many heavy kernel debug options enabled, like kasan
and kmemleak, which may influence this.
next prev parent reply other threads:[~2026-08-10 8:13 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-08-09 10:07 [PATCH v7 0/9] nvme-multipath: introduce latency I/O policy Nilay Shroff
2026-08-09 10:07 ` [PATCH v7 1/9] block: expose blk_stat_{enable,disable}_accounting() to drivers Nilay Shroff
2026-08-10 8:18 ` John Garry
2026-08-09 10:07 ` [PATCH v7 2/9] block: record I/O request start time for passthru request Nilay Shroff
2026-08-10 6:44 ` Hannes Reinecke
2026-08-09 10:07 ` [PATCH v7 3/9] nvme-multipath: pass I/O type to nvme_find_path() Nilay Shroff
2026-08-10 8:57 ` John Garry
2026-08-09 10:07 ` [PATCH v7 4/9] nvme-multipath: add support for latency I/O policy Nilay Shroff
2026-08-10 8:12 ` John Garry [this message]
2026-08-09 10:08 ` [PATCH v7 5/9] nvme: add generic debugfs support Nilay Shroff
2026-08-09 10:08 ` [PATCH v7 6/9] nvme-multipath: add debugfs attribute latency_ewma_shift Nilay Shroff
2026-08-09 10:08 ` [PATCH v7 7/9] nvme-multipath: add debugfs attribute latency_batch_timeout Nilay Shroff
2026-08-09 10:08 ` [PATCH v7 8/9] nvme-multipath: add debugfs attribute latency_stat Nilay Shroff
2026-08-09 10:08 ` [PATCH v7 9/9] nvme-multipath: add documentation for latency I/O policy Nilay Shroff
2026-08-10 6:45 ` Hannes Reinecke
2026-08-10 7:50 ` [PATCH v7 0/9] nvme-multipath: introduce " John Garry
2026-08-10 9:43 ` Guixin Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2d0163f0-ecf3-4f59-9121-587f98eeb54c@oracle.com \
--to=john.g.garry@oracle.com \
--cc=dwagner@suse.de \
--cc=gjoyce@linux.ibm.com \
--cc=hare@suse.de \
--cc=hch@lst.de \
--cc=jmeneghi@redhat.com \
--cc=kanie@linux.alibaba.com \
--cc=kbusch@kernel.org \
--cc=linux-nvme@lists.infradead.org \
--cc=martin.petersen@oracle.com \
--cc=nilay@linux.ibm.com \
--cc=randyj@purestorage.com \
--cc=sagi@grimberg.me \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.