[PATCH 0/9] libnvme: add support for retrieving additional NVMe stat

[PATCH 0/9] libnvme: add support for retrieving additional NVMe stat

[Nilay Shroff]

Hi,

Certain NVMe attributes and statistics can change dynamically at runtime.
However, the current libnvme implementation caches attribute values while
scanning and building the topology. Subsequent libnvme API calls return
these cached values.

While this approach works well for one-shot nvme-cli commands, it is not
suitable for use cases that require upto date information. In particular,
real-time monitoring tools such as nvme-top require access to the latest
(non-cached) values of such attributes.

To address this, this patchset introduces support for retrieving
non-cached values for selected attributes whose values may change
dynamically. The first three patches disable auto-generated accessors
for ana_state, numa_nodes, and iopolicy using "!accessors:none", and
provide custom implementations that always return the latest values.

In addition, libnvme lacks support for retrieving gendisk statistics, which
provide useful insight into disk activity (e.g., number of I/Os processed,
time spent servicing I/O, and in-flight I/O counts). This patchset adds
support for retrieving gendisk I/O statistics for both per-path and namespace
gendisks.

Finally, this series adds support for retrieving diagnostic counters at 
different levels, including per-path, namespace, nshead, and controller.
These counters should improve visibility into NVMe native multipath behavior.

All of the above statistics and counters can be leveraged by tools such as
nvme-top to provide real-time monitoring and analysis.

As usual, feedback and suggestions are welcome!

Nilay Shroff (9):
  libnvme: annotate nvme_path::ana_state with !accessors:none
  libnvme: annotate nvme_path::numa_nodes with !accessors:none
  libnvme: annotate nvme_subsystem::iopolicy with !accessors:none
  libnvme: add support for retrieving per-path gendisk I/O statistics
  libnvme: add support for retrieving namespace gendisk I/O statistics
  libnvme: add support for per-path diagnostic counters
  libnvme: add support for namespace diagnostic counters
  libnvme: add support for nshead diagnostic counters
  libnvme: add support for ctrl diagnostic counters

 libnvme/src/accessors.ld     |   6 -
 libnvme/src/libnvme.ld       |  36 +++
 libnvme/src/nvme/accessors.c |  39 ---
 libnvme/src/nvme/accessors.h |  47 ---
 libnvme/src/nvme/private.h   |  46 ++-
 libnvme/src/nvme/tree.c      | 551 +++++++++++++++++++++++++++++++++++
 libnvme/src/nvme/tree.h      | 314 ++++++++++++++++++++
 7 files changed, 944 insertions(+), 95 deletions(-)

-- 
2.53.0

[PATCH 1/9] libnvme: annotate nvme_path::ana_state with !accessors:none

[Nilay Shroff]

The default accessor generation creates getters that return cached
attribute values. However, for nvme_path::ana_state, a real-time
(non-cached) value is required.

This is particularly useful for tools such as nvme-top, which rely on
up-to-date information for displaying a real-time dashboard.

Annotate nvme_path::ana_state with "!accessors:none" to disable the
auto-generated accessor, and provide a custom implementation of
nvme_path_get_ana_state() that returns the current value.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/accessors.ld     |  2 --
 libnvme/src/libnvme.ld       |  1 +
 libnvme/src/nvme/accessors.c | 13 -------------
 libnvme/src/nvme/accessors.h | 15 ---------------
 libnvme/src/nvme/private.h   |  2 +-
 libnvme/src/nvme/tree.c      | 16 ++++++++++++++++
 libnvme/src/nvme/tree.h      |  8 ++++++++
 7 files changed, 26 insertions(+), 31 deletions(-)

diff --git a/libnvme/src/accessors.ld b/libnvme/src/accessors.ld
index 3cd26f8f7..9e5653b92 100644
--- a/libnvme/src/accessors.ld
+++ b/libnvme/src/accessors.ld
@@ -14,8 +14,6 @@ LIBNVME_ACCESSORS_3 {
 		nvme_path_set_name;
 		nvme_path_get_sysfs_dir;
 		nvme_path_set_sysfs_dir;
-		nvme_path_get_ana_state;
-		nvme_path_set_ana_state;
 		nvme_path_get_numa_nodes;
 		nvme_path_set_numa_nodes;
 		nvme_path_get_grpid;
diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index 6d5d7a0f9..0fd86be71 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -142,6 +142,7 @@ LIBNVME_3 {
 		nvme_path_get_ctrl;
 		nvme_path_get_ns;
 		nvme_path_get_queue_depth;
+		nvme_path_get_ana_state;
 		nvme_filter_paths;
 		nvme_read_config;
 		nvme_read_key;
diff --git a/libnvme/src/nvme/accessors.c b/libnvme/src/nvme/accessors.c
index d3c507d7e..8b095ed4d 100644
--- a/libnvme/src/nvme/accessors.c
+++ b/libnvme/src/nvme/accessors.c
@@ -52,19 +52,6 @@ __public const char *nvme_path_get_sysfs_dir(const struct nvme_path *p)
 	return p->sysfs_dir;
 }
 
-__public void nvme_path_set_ana_state(
-		struct nvme_path *p,
-		const char *ana_state)
-{
-	free(p->ana_state);
-	p->ana_state = ana_state ? strdup(ana_state) : NULL;
-}
-
-__public const char *nvme_path_get_ana_state(const struct nvme_path *p)
-{
-	return p->ana_state;
-}
-
 __public void nvme_path_set_numa_nodes(
 		struct nvme_path *p,
 		const char *numa_nodes)
diff --git a/libnvme/src/nvme/accessors.h b/libnvme/src/nvme/accessors.h
index 5c594dd12..3ede23cbc 100644
--- a/libnvme/src/nvme/accessors.h
+++ b/libnvme/src/nvme/accessors.h
@@ -68,21 +68,6 @@ void nvme_path_set_sysfs_dir(struct nvme_path *p, const char *sysfs_dir);
  */
 const char *nvme_path_get_sysfs_dir(const struct nvme_path *p);
 
-/**
- * nvme_path_set_ana_state() - Set ana_state.
- * @p: The &struct nvme_path instance to update.
- * @ana_state: New string; a copy is stored. Pass NULL to clear.
- */
-void nvme_path_set_ana_state(struct nvme_path *p, const char *ana_state);
-
-/**
- * nvme_path_get_ana_state() - Get ana_state.
- * @p: The &struct nvme_path instance to query.
- *
- * Return: The value of the ana_state field, or NULL if not set.
- */
-const char *nvme_path_get_ana_state(const struct nvme_path *p);
-
 /**
  * nvme_path_set_numa_nodes() - Set numa_nodes.
  * @p: The &struct nvme_path instance to update.
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index 82873fcd6..25fec3216 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -137,8 +137,8 @@ struct nvme_path { /*!generate-accessors*/
 
 	char *name;
 	char *sysfs_dir;
-	char *ana_state;
 	char *numa_nodes;
+	char *ana_state; //!accessors:none
 	int grpid;
 	int queue_depth; //!accessors:none
 };
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 0d836b436..1adc2c57a 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -804,6 +804,22 @@ __public int nvme_path_get_queue_depth(nvme_path_t p)
 	return p->queue_depth;
 }
 
+__public char *nvme_path_get_ana_state(nvme_path_t p)
+{
+	_cleanup_free_ char *ana_state = NULL;
+
+	ana_state = nvme_get_path_attr(p, "ana_state");
+	if (ana_state) {
+		if (!p->ana_state || strcmp(ana_state, p->ana_state)) {
+			free(p->ana_state);
+			p->ana_state = strdup(ana_state);
+
+	}
+	}
+
+	return p->ana_state;
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 07b75677c..ec0f4f2e6 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -679,6 +679,14 @@ int nvme_ns_identify_descs(nvme_ns_t n, struct nvme_ns_id_desc *descs);
  */
 int nvme_path_get_queue_depth(nvme_path_t p);
 
+/**
+ * nvme_path_get_ana_state() - ANA state of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: ANA state of @p
+ */
+char *nvme_path_get_ana_state(nvme_path_t p);
+
 /**
  * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
  * @p:	&nvme_path_t object
-- 
2.53.0

[PATCH 2/9] libnvme: annotate nvme_path::numa_nodes with !accessors:none

[Nilay Shroff]

The default accessor generation creates getters that return cached
attribute values. However, for nvme_path::numa_nodes, a real-time
(non-cached) value is required.

This is particularly useful for tools such as nvme-top, which rely on
up-to-date information for displaying a real-time dashboard.

Annotate nvme_path::numa_nodes with "!accessors:none" to disable the
auto-generated accessor, and provide a custom implementation of
nvme_path_get_numa_nodes() that returns the current value.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/accessors.ld     |  2 --
 libnvme/src/libnvme.ld       |  1 +
 libnvme/src/nvme/accessors.c | 13 -------------
 libnvme/src/nvme/accessors.h | 15 ---------------
 libnvme/src/nvme/private.h   |  2 +-
 libnvme/src/nvme/tree.c      | 15 +++++++++++++++
 libnvme/src/nvme/tree.h      |  8 ++++++++
 7 files changed, 25 insertions(+), 31 deletions(-)

diff --git a/libnvme/src/accessors.ld b/libnvme/src/accessors.ld
index 9e5653b92..433767191 100644
--- a/libnvme/src/accessors.ld
+++ b/libnvme/src/accessors.ld
@@ -14,8 +14,6 @@ LIBNVME_ACCESSORS_3 {
 		nvme_path_set_name;
 		nvme_path_get_sysfs_dir;
 		nvme_path_set_sysfs_dir;
-		nvme_path_get_numa_nodes;
-		nvme_path_set_numa_nodes;
 		nvme_path_get_grpid;
 		nvme_path_set_grpid;
 		nvme_ns_get_nsid;
diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index 0fd86be71..41a170083 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -143,6 +143,7 @@ LIBNVME_3 {
 		nvme_path_get_ns;
 		nvme_path_get_queue_depth;
 		nvme_path_get_ana_state;
+		nvme_path_get_numa_nodes;
 		nvme_filter_paths;
 		nvme_read_config;
 		nvme_read_key;
diff --git a/libnvme/src/nvme/accessors.c b/libnvme/src/nvme/accessors.c
index 8b095ed4d..a98f26234 100644
--- a/libnvme/src/nvme/accessors.c
+++ b/libnvme/src/nvme/accessors.c
@@ -52,19 +52,6 @@ __public const char *nvme_path_get_sysfs_dir(const struct nvme_path *p)
 	return p->sysfs_dir;
 }
 
-__public void nvme_path_set_numa_nodes(
-		struct nvme_path *p,
-		const char *numa_nodes)
-{
-	free(p->numa_nodes);
-	p->numa_nodes = numa_nodes ? strdup(numa_nodes) : NULL;
-}
-
-__public const char *nvme_path_get_numa_nodes(const struct nvme_path *p)
-{
-	return p->numa_nodes;
-}
-
 __public void nvme_path_set_grpid(struct nvme_path *p, int grpid)
 {
 	p->grpid = grpid;
diff --git a/libnvme/src/nvme/accessors.h b/libnvme/src/nvme/accessors.h
index 3ede23cbc..f9323ad11 100644
--- a/libnvme/src/nvme/accessors.h
+++ b/libnvme/src/nvme/accessors.h
@@ -68,21 +68,6 @@ void nvme_path_set_sysfs_dir(struct nvme_path *p, const char *sysfs_dir);
  */
 const char *nvme_path_get_sysfs_dir(const struct nvme_path *p);
 
-/**
- * nvme_path_set_numa_nodes() - Set numa_nodes.
- * @p: The &struct nvme_path instance to update.
- * @numa_nodes: New string; a copy is stored. Pass NULL to clear.
- */
-void nvme_path_set_numa_nodes(struct nvme_path *p, const char *numa_nodes);
-
-/**
- * nvme_path_get_numa_nodes() - Get numa_nodes.
- * @p: The &struct nvme_path instance to query.
- *
- * Return: The value of the numa_nodes field, or NULL if not set.
- */
-const char *nvme_path_get_numa_nodes(const struct nvme_path *p);
-
 /**
  * nvme_path_set_grpid() - Set grpid.
  * @p: The &struct nvme_path instance to update.
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index 25fec3216..dcd9079dd 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -137,8 +137,8 @@ struct nvme_path { /*!generate-accessors*/
 
 	char *name;
 	char *sysfs_dir;
-	char *numa_nodes;
 	char *ana_state; //!accessors:none
+	char *numa_nodes; //!accessors:none
 	int grpid;
 	int queue_depth; //!accessors:none
 };
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 1adc2c57a..2984d0ff7 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -820,6 +820,21 @@ __public char *nvme_path_get_ana_state(nvme_path_t p)
 	return p->ana_state;
 }
 
+__public char *nvme_path_get_numa_nodes(nvme_path_t p)
+{
+	_cleanup_free_ char *numa_nodes = NULL;
+
+	numa_nodes = nvme_get_path_attr(p, "numa_nodes");
+	if (numa_nodes) {
+		if (!p->numa_nodes || strcmp(numa_nodes, p->numa_nodes)) {
+			free(p->numa_nodes);
+			p->numa_nodes = strdup(numa_nodes);
+		}
+	}
+
+	return p->numa_nodes;
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index ec0f4f2e6..1240f3023 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -687,6 +687,14 @@ int nvme_path_get_queue_depth(nvme_path_t p);
  */
 char *nvme_path_get_ana_state(nvme_path_t p);
 
+/**
+ * nvme_path_get_numa_nodes() - Numa nodes of an nvme_path_t object
+ * @p: &nvme_path_t object
+ *
+ * Return: Numa nodes of @p
+ */
+char *nvme_path_get_numa_nodes(nvme_path_t p);
+
 /**
  * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
  * @p:	&nvme_path_t object
-- 
2.53.0

[PATCH 3/9] libnvme: annotate nvme_subsystem::iopolicy with !accessors:none

[Nilay Shroff]

The default accessor generation creates getters that return cached
attribute values. However, for nvme_subsystem::iopolicy, a real-time
(non-cached) value is required.

This is particularly useful for tools such as nvme-top, which rely on
up-to-date information for displaying a real-time dashboard.

Annotate nvme_subsystem::iopolicy with "!accessors:none" to disable the
auto-generated accessor, and provide a custom implementation of
nvme_subsystem_get_iopolicy() that returns the current value.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/accessors.ld     |  2 --
 libnvme/src/libnvme.ld       |  1 +
 libnvme/src/nvme/accessors.c | 13 -------------
 libnvme/src/nvme/accessors.h | 17 -----------------
 libnvme/src/nvme/private.h   |  2 +-
 libnvme/src/nvme/tree.c      | 15 +++++++++++++++
 libnvme/src/nvme/tree.h      |  8 ++++++++
 7 files changed, 25 insertions(+), 33 deletions(-)

diff --git a/libnvme/src/accessors.ld b/libnvme/src/accessors.ld
index 433767191..56348a7e5 100644
--- a/libnvme/src/accessors.ld
+++ b/libnvme/src/accessors.ld
@@ -77,8 +77,6 @@ LIBNVME_ACCESSORS_3 {
 		nvme_subsystem_get_subsystype;
 		nvme_subsystem_get_application;
 		nvme_subsystem_set_application;
-		nvme_subsystem_get_iopolicy;
-		nvme_subsystem_set_iopolicy;
 		nvme_host_get_hostnqn;
 		nvme_host_get_hostid;
 		nvme_host_get_dhchap_host_key;
diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index 41a170083..f842eb770 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -176,6 +176,7 @@ LIBNVME_3 {
 		nvme_subsystem_first_ns;
 		nvme_get_subsystem;
 		nvme_subsystem_get_host;
+		nvme_subsystem_get_iopolicy;
 		nvme_subsystem_lookup_namespace;
 		nvme_subsystem_next_ctrl;
 		nvme_subsystem_next_ns;
diff --git a/libnvme/src/nvme/accessors.c b/libnvme/src/nvme/accessors.c
index a98f26234..55fd11320 100644
--- a/libnvme/src/nvme/accessors.c
+++ b/libnvme/src/nvme/accessors.c
@@ -404,19 +404,6 @@ __public const char *nvme_subsystem_get_application(
 	return p->application;
 }
 
-__public void nvme_subsystem_set_iopolicy(
-		struct nvme_subsystem *p,
-		const char *iopolicy)
-{
-	free(p->iopolicy);
-	p->iopolicy = iopolicy ? strdup(iopolicy) : NULL;
-}
-
-__public const char *nvme_subsystem_get_iopolicy(const struct nvme_subsystem *p)
-{
-	return p->iopolicy;
-}
-
 /****************************************************************************
  * Accessors for: struct nvme_host
  ****************************************************************************/
diff --git a/libnvme/src/nvme/accessors.h b/libnvme/src/nvme/accessors.h
index f9323ad11..9ea96a634 100644
--- a/libnvme/src/nvme/accessors.h
+++ b/libnvme/src/nvme/accessors.h
@@ -575,23 +575,6 @@ void nvme_subsystem_set_application(
  */
 const char *nvme_subsystem_get_application(const struct nvme_subsystem *p);
 
-/**
- * nvme_subsystem_set_iopolicy() - Set iopolicy.
- * @p: The &struct nvme_subsystem instance to update.
- * @iopolicy: New string; a copy is stored. Pass NULL to clear.
- */
-void nvme_subsystem_set_iopolicy(
-		struct nvme_subsystem *p,
-		const char *iopolicy);
-
-/**
- * nvme_subsystem_get_iopolicy() - Get iopolicy.
- * @p: The &struct nvme_subsystem instance to query.
- *
- * Return: The value of the iopolicy field, or NULL if not set.
- */
-const char *nvme_subsystem_get_iopolicy(const struct nvme_subsystem *p);
-
 /****************************************************************************
  * Accessors for: struct nvme_host
  ****************************************************************************/
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index dcd9079dd..84852c8b0 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -230,7 +230,7 @@ struct nvme_subsystem { /*!generate-accessors*/
 	char *firmware; /*!accessors:readonly*/
 	char *subsystype; /*!accessors:readonly*/
 	char *application;
-	char *iopolicy;
+	char *iopolicy; //!accessors:none
 };
 
 struct nvme_host { /*!generate-accessors*/
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 2984d0ff7..18aabe044 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -445,6 +445,21 @@ __public nvme_host_t nvme_subsystem_get_host(nvme_subsystem_t s)
 	return s->h;
 }
 
+__public char *nvme_subsystem_get_iopolicy(nvme_subsystem_t s)
+{
+	_cleanup_free_ char *iopolicy = NULL;
+
+	iopolicy = nvme_get_subsys_attr(s, "iopolicy");
+	if (iopolicy) {
+		if (!s->iopolicy || strcmp(iopolicy, s->iopolicy)) {
+			free(s->iopolicy);
+			s->iopolicy = strdup(iopolicy);
+		}
+	}
+
+	return s->iopolicy;
+}
+
 __public nvme_ns_t nvme_subsystem_first_ns(nvme_subsystem_t s)
 {
 	return list_top(&s->namespaces, struct nvme_ns, entry);
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 1240f3023..39d715508 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -212,6 +212,14 @@ void nvme_free_subsystem(struct nvme_subsystem *s);
  */
 nvme_host_t nvme_subsystem_get_host(nvme_subsystem_t s);
 
+/**
+ * nvme_subsystem_get_iopolicy() - Get subsystem iopolicy name
+ * @s:	subsystem
+ *
+ * Return: The iopolicy configured in subsystem @s
+ */
+char *nvme_subsystem_get_iopolicy(nvme_subsystem_t s);
+
 /**
  * nvme_ctrl_first_ns() - Start namespace iterator
  * @c:	Controller instance
-- 
2.53.0

[PATCH 4/9] libnvme: add support for retrieving per-path gendisk I/O statistics

[Nilay Shroff]

Gendisk I/O statistics provide useful insight into disk activity,
including read/write/discard/flush operations, as well as information
about in-flight I/Os and I/O timing.

Parsing these statistics allows users to determine the number of I/Os
processed, time spent servicing I/O, number of sectors accessed, and
the count of in-flight requests.

Add support for retrieving per-path gendisk I/O statistics. Also add
support for computing deltas of these statistics between samples, such
as I/O ticks, number of sectors, and number of serviced I/Os.

These metrics can be used by tools such as nvme-top to display
real-time disk activity. Per-path metrics are particularly useful when
NVMe native multipath is enabled.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  12 ++
 libnvme/src/nvme/private.h |  26 ++++
 libnvme/src/nvme/tree.c    | 255 +++++++++++++++++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 102 +++++++++++++++
 4 files changed, 395 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index f842eb770..ceb9f9bcb 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -144,6 +144,18 @@ LIBNVME_3 {
 		nvme_path_get_queue_depth;
 		nvme_path_get_ana_state;
 		nvme_path_get_numa_nodes;
+		nvme_path_get_stat;
+		nvme_path_get_read_ios;
+		nvme_path_get_write_ios;
+		nvme_path_get_inflights;
+		nvme_path_get_stat_interval;
+		nvme_path_get_io_ticks;
+		nvme_path_get_read_ticks;
+		nvme_path_get_write_ticks;
+		nvme_path_get_read_sectors;
+		nvme_path_get_write_sectors;
+		nvme_path_reset_stat;
+		nvme_path_update_stat;
 		nvme_filter_paths;
 		nvme_read_config;
 		nvme_read_key;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index 84852c8b0..8e327f1e2 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -20,6 +20,7 @@
 
 #include <nvme/fabrics.h>
 #include <nvme/mi.h>
+#include <nvme/tree.h>
 
 const char *nvme_subsys_sysfs_dir(void);
 const char *nvme_ctrl_sysfs_dir(void);
@@ -128,9 +129,34 @@ struct nvme_transport_handle {
 	struct nvme_log *log;
 };
 
+enum stat_group {
+	READ = 0,
+	WRITE,
+	DISCARD,
+	FLUSH,
+
+	NR_STAT_GROUPS
+};
+
+struct nvme_stat {
+	struct {
+		unsigned long ios;
+		unsigned long merges;
+		unsigned long long sectors;
+		unsigned int ticks;	/* in milliseconds */
+	} group[NR_STAT_GROUPS];
+
+	unsigned int inflights;
+	unsigned int io_ticks;		/* in milliseconds */
+	unsigned int tot_ticks;		/* in milliseconds */
+
+	double ts_ms;			/* timestamp when the stat is updated */
+};
+
 struct nvme_path { /*!generate-accessors*/
 	struct list_node entry;
 	struct list_node nentry;
+	struct nvme_stat stat[2];
 
 	struct nvme_ctrl *c;
 	struct nvme_ns *n;
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 18aabe044..94f900307 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -16,6 +16,7 @@
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
+#include <time.h>
 
 #include <arpa/inet.h>
 #include <netdb.h>
@@ -850,6 +851,260 @@ __public char *nvme_path_get_numa_nodes(nvme_path_t p)
 	return p->numa_nodes;
 }
 
+static nvme_stat_t nvme_path_get_stat(nvme_path_t p, int curr)
+{
+	if (curr < 0 || curr > 1)
+		return NULL;
+
+	return &p->stat[curr];
+}
+
+__public void nvme_path_reset_stat(nvme_path_t p)
+{
+	nvme_stat_t stat = &p->stat[0];
+
+	memset(stat, 0, 2 * sizeof(struct nvme_stat));
+}
+
+static int nvme_update_stat(const char *sysfs_stat_path, nvme_stat_t stat)
+{
+	int n;
+	struct timespec ts;
+	unsigned long rd_ios, rd_merges, wr_ios, wr_merges;
+	unsigned long dc_ios, dc_merges, fl_ios;
+	unsigned long long rd_sectors, wr_sectors, dc_sectors;
+	unsigned int rd_ticks, wr_ticks, dc_ticks, fl_ticks;
+	unsigned int io_ticks, tot_ticks, inflights;
+
+	memset(stat, 0, sizeof(struct nvme_stat));
+
+	n = sscanf(sysfs_stat_path,
+		"%lu %lu %llu %u %lu %lu %llu %u %u %u %u %lu %lu %llu %u %lu %u",
+		&rd_ios, &rd_merges, &rd_sectors, &rd_ticks,
+		&wr_ios, &wr_merges, &wr_sectors, &wr_ticks,
+		&inflights, &io_ticks, &tot_ticks,
+		&dc_ios, &dc_merges, &dc_sectors, &dc_ticks,
+		&fl_ios, &fl_ticks);
+
+	if (n < 17)
+		return -1;
+
+	/* update read stat */
+	stat->group[READ].ios = rd_ios;
+	stat->group[READ].merges = rd_merges;
+	stat->group[READ].sectors = rd_sectors;
+	stat->group[READ].ticks = rd_ticks;
+
+	/* update write stat */
+	stat->group[WRITE].ios = wr_ios;
+	stat->group[WRITE].merges = wr_merges;
+	stat->group[WRITE].sectors = wr_sectors;
+	stat->group[WRITE].ticks = wr_ticks;
+
+	/* update inflight counters and ticks */
+	stat->inflights = inflights;
+	stat->io_ticks = io_ticks;
+	stat->tot_ticks = tot_ticks;
+
+	/* update discard stat */
+	stat->group[DISCARD].ios = dc_ios;
+	stat->group[DISCARD].merges = dc_merges;
+	stat->group[DISCARD].sectors = dc_sectors;
+	stat->group[DISCARD].ticks = dc_ticks;
+
+	/* update flush stat */
+	stat->group[FLUSH].ios = fl_ios;
+	stat->group[FLUSH].ticks = fl_ticks;
+
+	clock_gettime(CLOCK_MONOTONIC, &ts);
+	stat->ts_ms = ts.tv_sec * 1000 + (double)ts.tv_nsec / 1e6;
+
+	return 0;
+}
+
+__public int nvme_path_update_stat(nvme_path_t p, int curr)
+{
+	_cleanup_free_ char *sysfs_stat_path = NULL;
+	nvme_stat_t stat;
+
+	stat = nvme_path_get_stat(p, curr);
+	if (!stat)
+		return -1;
+
+	sysfs_stat_path = nvme_get_path_attr(p, "stat");
+	if (!sysfs_stat_path)
+		return -1;
+
+	return nvme_update_stat(sysfs_stat_path, stat);
+}
+
+static int nvme_stat_get_inflights(nvme_stat_t stat)
+{
+	return stat->inflights;
+}
+
+__public unsigned int nvme_path_get_inflights(nvme_path_t p, int this)
+{
+	nvme_stat_t curr;
+
+	curr = nvme_path_get_stat(p, this);
+	if (!curr)
+		return 0;
+
+	return nvme_stat_get_inflights(curr);
+}
+
+static int nvme_stat_get_io_ticks(nvme_stat_t curr, nvme_stat_t prev)
+{
+	unsigned int delta = 0;
+
+	if (curr->io_ticks > prev->io_ticks)
+		delta = curr->io_ticks - prev->io_ticks;
+
+	return delta;
+}
+
+__public unsigned int nvme_path_get_io_ticks(nvme_path_t p, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_path_get_stat(p, this);
+	prev = nvme_path_get_stat(p, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_io_ticks(curr, prev);
+}
+
+static unsigned int nvme_stat_get_ticks(nvme_stat_t curr,
+		nvme_stat_t prev, enum stat_group grp)
+{
+	unsigned int delta = 0;
+
+	if (curr->group[grp].ticks > prev->group[grp].ticks)
+		delta = curr->group[grp].ticks - prev->group[grp].ticks;
+
+	return delta;
+}
+
+static unsigned int __nvme_path_get_ticks(nvme_path_t p,
+		enum stat_group grp, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_path_get_stat(p, this);
+	prev = nvme_path_get_stat(p, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_ticks(curr, prev, grp);
+}
+
+__public unsigned int nvme_path_get_read_ticks(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_ticks(p, READ, curr);
+}
+
+__public unsigned int nvme_path_get_write_ticks(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_ticks(p, WRITE, curr);
+}
+
+static double nvme_stat_get_interval(nvme_stat_t curr, nvme_stat_t prev)
+{
+	double delta = 0.0;
+
+	if (curr->ts_ms > prev->ts_ms)
+		delta = curr->ts_ms - prev->ts_ms;
+
+	return delta;
+}
+
+__public double nvme_path_get_stat_interval(nvme_path_t p, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_path_get_stat(p, this);
+	prev = nvme_path_get_stat(p, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_interval(curr, prev);
+}
+
+static unsigned long nvme_stat_get_ios(nvme_stat_t curr, nvme_stat_t prev,
+		enum stat_group grp)
+{
+	unsigned long ios = 0;
+
+	if (curr->group[grp].ios > prev->group[grp].ios)
+		ios = curr->group[grp].ios - prev->group[grp].ios;
+
+	return ios;
+}
+
+static unsigned long __nvme_path_get_ios(nvme_path_t p, enum stat_group grp,
+		int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_path_get_stat(p, this);
+	prev = nvme_path_get_stat(p, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_ios(curr, prev, grp);
+}
+
+__public unsigned long nvme_path_get_read_ios(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_ios(p, READ, curr);
+}
+
+__public unsigned long nvme_path_get_write_ios(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_ios(p, WRITE, curr);
+}
+
+static unsigned long long nvme_stat_get_sectors(nvme_stat_t curr,
+		nvme_stat_t prev, enum stat_group grp)
+{
+	unsigned long long sec = 0;
+
+	if (curr->group[grp].sectors > prev->group[grp].sectors)
+		sec = curr->group[grp].sectors - prev->group[grp].sectors;
+
+	return sec;
+}
+
+static unsigned long long __nvme_path_get_sectors(nvme_path_t p,
+		enum stat_group grp, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_path_get_stat(p, this);
+	prev = nvme_path_get_stat(p, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_sectors(curr, prev, grp);
+}
+
+__public unsigned long long nvme_path_get_read_sectors(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_sectors(p, READ, curr);
+}
+
+__public unsigned long long nvme_path_get_write_sectors(nvme_path_t p, int curr)
+{
+	return __nvme_path_get_sectors(p, WRITE, curr);
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 39d715508..3924e061a 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -23,6 +23,7 @@
 typedef struct nvme_ns *nvme_ns_t;
 typedef struct nvme_ns_head *nvme_ns_head_t;
 typedef struct nvme_path *nvme_path_t;
+typedef struct nvme_stat *nvme_stat_t;
 typedef struct nvme_ctrl *nvme_ctrl_t;
 typedef struct nvme_subsystem *nvme_subsystem_t;
 typedef struct nvme_host *nvme_host_t;
@@ -719,6 +720,107 @@ nvme_ctrl_t nvme_path_get_ctrl(nvme_path_t p);
  */
 nvme_ns_t nvme_path_get_ns(nvme_path_t p);
 
+/**
+ * nvme_path_reset_stat() - Resets namespace path nvme stat
+ * @p:	&nvme_path_t object
+ */
+void nvme_path_reset_stat(nvme_path_t p);
+
+/**
+ * nvme_path_update_stat() - Update stat of an nvme_path_t object
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	0 on success, -1 on error
+ */
+int nvme_path_update_stat(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_read_ios() - Calculate and return read IOs
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Num of read IOs processed between two stat samples
+ */
+unsigned long nvme_path_get_read_ios(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_write_ios() - Get write I/Os
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Num of write I/Os processed between two stat samples
+ */
+unsigned long nvme_path_get_write_ios(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_read_ticks() - Get read I/O ticks
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Time, in milliseconds, sepnt processing read I/O requests
+ *		between two stat samples
+ */
+unsigned int nvme_path_get_read_ticks(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_read_sectors() - Get read I/O sectors
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Number of sectors read from the device between two stat samples
+ */
+unsigned long long nvme_path_get_read_sectors(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_write_sectors() - Get write I/O sectors
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Num of sectors written to the device between two stat samples
+ */
+unsigned long long nvme_path_get_write_sectors(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_write_ticks() - Get write I/O ticks
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Time, in milliseconds, sepnt processing write I/O requests
+ *		between two stat samples
+ */
+unsigned int nvme_path_get_write_ticks(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_stat_interval() - Get interval between two stat samples
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Interval, in milliseconds between collection of two consecutive
+ *		stat samples
+ */
+double nvme_path_get_stat_interval(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_io_ticks() - Get I/O ticks
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Time consumed, in milliseconds, processing I/O requests between
+ *		two stat samples
+ */
+unsigned int nvme_path_get_io_ticks(nvme_path_t p, int curr);
+
+/**
+ * nvme_path_get_inflights() - Inflight IOs for nvme_path_t object
+ *
+ * @p:		&nvme_path_t object
+ * @curr:	Index in nvme_path_stat_t object
+ *
+ * Return:	Inflight number of IOs
+ */
+unsigned int nvme_path_get_inflights(nvme_path_t p, int curr);
+
 /**
  * nvme_ctrl_get_transport_handle() - Get associated transport handle
  * @c:	Controller instance
-- 
2.53.0

[PATCH 5/9] libnvme: add support for retrieving namespace gendisk I/O statistics

[Nilay Shroff]

Gendisk I/O statistics provide useful insight into disk activity,
including read/write/discard/flush operations, as well as information
about in-flight I/Os and I/O timing.

Parsing these statistics allows users to determine the number of I/Os
processed, time spent servicing I/O, number of sectors accessed, and
the count of in-flight requests.

Add support for retrieving namespace gendisk I/O statistics. Also add
support for computing deltas of these statistics between samples, such
as I/O ticks, number of sectors, and number of serviced I/Os.

These metrics can be used by tools such as nvme-top to display real-time
disk activity.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  11 +++
 libnvme/src/nvme/private.h |   3 +
 libnvme/src/nvme/tree.c    | 140 +++++++++++++++++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 102 +++++++++++++++++++++++++++
 4 files changed, 256 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index ceb9f9bcb..c9550240d 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -130,6 +130,17 @@ LIBNVME_3 {
 		nvme_ns_get_serial;
 		nvme_ns_get_subsystem;
 		nvme_ns_get_uuid;
+		nvme_ns_reset_stat;
+		nvme_ns_update_stat;
+		nvme_ns_get_stat_interval;
+		nvme_ns_get_read_ios;
+		nvme_ns_get_write_ios;
+		nvme_ns_get_read_ticks;
+		nvme_ns_get_write_ticks;
+		nvme_ns_get_read_sectors;
+		nvme_ns_get_write_sectors;
+		nvme_ns_get_inflights;
+		nvme_ns_get_io_ticks;
 		nvme_ns_identify;
 		nvme_ns_read;
 		nvme_rescan_ns;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index 8e327f1e2..858759a1b 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -184,6 +184,9 @@ struct nvme_ns { /*!generate-accessors*/
 	struct nvme_ns_head *head;
 
 	struct nvme_global_ctx *ctx;
+
+	struct nvme_stat stat[2];
+
 	struct nvme_transport_handle *hdl;
 	__u32 nsid;
 	char *name;
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 94f900307..3ad483557 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -866,6 +866,21 @@ __public void nvme_path_reset_stat(nvme_path_t p)
 	memset(stat, 0, 2 * sizeof(struct nvme_stat));
 }
 
+static nvme_stat_t nvme_ns_get_stat(nvme_ns_t n, int curr)
+{
+	if (curr < 0 || curr > 1)
+		return NULL;
+
+	return &n->stat[curr];
+}
+
+__public void nvme_ns_reset_stat(nvme_ns_t n)
+{
+	nvme_stat_t stat = &n->stat[0];
+
+	memset(stat, 0, 2 * sizeof(struct nvme_stat));
+}
+
 static int nvme_update_stat(const char *sysfs_stat_path, nvme_stat_t stat)
 {
 	int n;
@@ -938,6 +953,22 @@ __public int nvme_path_update_stat(nvme_path_t p, int curr)
 	return nvme_update_stat(sysfs_stat_path, stat);
 }
 
+__public int nvme_ns_update_stat(nvme_ns_t n, int curr)
+{
+	_cleanup_free_ char *sysfs_stat_path = NULL;
+	nvme_stat_t stat;
+
+	stat = nvme_ns_get_stat(n, curr);
+	if (!stat)
+		return -1;
+
+	sysfs_stat_path = nvme_get_ns_attr(n, "stat");
+	if (!sysfs_stat_path)
+		return -1;
+
+	return nvme_update_stat(sysfs_stat_path, stat);
+}
+
 static int nvme_stat_get_inflights(nvme_stat_t stat)
 {
 	return stat->inflights;
@@ -954,6 +985,17 @@ __public unsigned int nvme_path_get_inflights(nvme_path_t p, int this)
 	return nvme_stat_get_inflights(curr);
 }
 
+__public unsigned int nvme_ns_get_inflights(nvme_ns_t n, int this)
+{
+	nvme_stat_t curr;
+
+	curr = nvme_ns_get_stat(n, this);
+	if (!curr)
+		return 0;
+
+	return nvme_stat_get_inflights(curr);
+}
+
 static int nvme_stat_get_io_ticks(nvme_stat_t curr, nvme_stat_t prev)
 {
 	unsigned int delta = 0;
@@ -977,6 +1019,19 @@ __public unsigned int nvme_path_get_io_ticks(nvme_path_t p, int this)
 	return nvme_stat_get_io_ticks(curr, prev);
 }
 
+__public unsigned int nvme_ns_get_io_ticks(nvme_ns_t n, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_ns_get_stat(n, this);
+	prev = nvme_ns_get_stat(n, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_io_ticks(curr, prev);
+}
+
 static unsigned int nvme_stat_get_ticks(nvme_stat_t curr,
 		nvme_stat_t prev, enum stat_group grp)
 {
@@ -1012,6 +1067,30 @@ __public unsigned int nvme_path_get_write_ticks(nvme_path_t p, int curr)
 	return __nvme_path_get_ticks(p, WRITE, curr);
 }
 
+static unsigned int __nvme_ns_get_ticks(nvme_ns_t n,
+		enum stat_group grp, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_ns_get_stat(n, this);
+	prev = nvme_ns_get_stat(n, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_ticks(curr, prev, grp);
+}
+
+__public unsigned int nvme_ns_get_read_ticks(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_ticks(n, READ, curr);
+}
+
+__public unsigned int nvme_ns_get_write_ticks(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_ticks(n, WRITE, curr);
+}
+
 static double nvme_stat_get_interval(nvme_stat_t curr, nvme_stat_t prev)
 {
 	double delta = 0.0;
@@ -1035,6 +1114,19 @@ __public double nvme_path_get_stat_interval(nvme_path_t p, int this)
 	return nvme_stat_get_interval(curr, prev);
 }
 
+__public double nvme_ns_get_stat_interval(nvme_ns_t n, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_ns_get_stat(n, this);
+	prev = nvme_ns_get_stat(n, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_interval(curr, prev);
+}
+
 static unsigned long nvme_stat_get_ios(nvme_stat_t curr, nvme_stat_t prev,
 		enum stat_group grp)
 {
@@ -1070,6 +1162,30 @@ __public unsigned long nvme_path_get_write_ios(nvme_path_t p, int curr)
 	return __nvme_path_get_ios(p, WRITE, curr);
 }
 
+static unsigned long __nvme_ns_get_ios(nvme_ns_t n, enum stat_group grp,
+		int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_ns_get_stat(n, this);
+	prev = nvme_ns_get_stat(n, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_ios(curr, prev, grp);
+}
+
+__public unsigned long nvme_ns_get_read_ios(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_ios(n, READ, curr);
+}
+
+__public unsigned long nvme_ns_get_write_ios(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_ios(n, WRITE, curr);
+}
+
 static unsigned long long nvme_stat_get_sectors(nvme_stat_t curr,
 		nvme_stat_t prev, enum stat_group grp)
 {
@@ -1105,6 +1221,30 @@ __public unsigned long long nvme_path_get_write_sectors(nvme_path_t p, int curr)
 	return __nvme_path_get_sectors(p, WRITE, curr);
 }
 
+static unsigned long long __nvme_ns_get_sectors(nvme_ns_t n,
+		enum stat_group grp, int this)
+{
+	nvme_stat_t curr, prev;
+
+	curr = nvme_ns_get_stat(n, this);
+	prev = nvme_ns_get_stat(n, !this);
+
+	if (!curr || !prev)
+		return 0;
+
+	return nvme_stat_get_sectors(curr, prev, grp);
+}
+
+__public unsigned long long nvme_ns_get_read_sectors(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_sectors(n, READ, curr);
+}
+
+__public unsigned long long nvme_ns_get_write_sectors(nvme_ns_t n, int curr)
+{
+	return __nvme_ns_get_sectors(n, WRITE, curr);
+}
+
 void nvme_free_path(struct nvme_path *p)
 {
 	list_del_init(&p->entry);
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 3924e061a..c69d0c8a0 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -876,6 +876,108 @@ nvme_subsystem_t nvme_ctrl_get_subsystem(nvme_ctrl_t c);
  */
 const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head);
 
+/**
+ * nvme_ns_update_stat() - update the nvme namespace stat
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Returns:	0 on success, -1 on error
+ */
+int nvme_ns_update_stat(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_reset_stat() - Resets nvme namespace stat
+ * @n:	&nvme_ns_t object
+ *
+ */
+void nvme_ns_reset_stat(nvme_ns_t n);
+
+/**
+ * nvme_ns_get_inflights() - Inflight IOs for nvme_ns_t object
+ *
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Inflight number of IOs
+ */
+unsigned int nvme_ns_get_inflights(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_io_ticks() - Get IO ticks
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Time consumed, in milliseconds, processing I/O requests between
+ *		two stat samples
+ */
+unsigned int nvme_ns_get_io_ticks(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_read_ticks() - Get read I/O ticks
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Time, in milliseconds, sepnt processing read I/O requests
+ *		between two stat samples
+ */
+unsigned int nvme_ns_get_read_ticks(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_write_ticks() - Get write I/O ticks
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Time, in milliseconds, sepnt processing write I/O requests
+ *		between two stat samples
+ */
+unsigned int nvme_ns_get_write_ticks(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_stat_interval() - Get interval between two stat samples
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Interval, in milliseconds, between collection of two consecutive
+ *		stat samples
+ */
+double nvme_ns_get_stat_interval(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_read_ios() - Get num of read I/Os
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Num of read IOs processed between two stat samples
+ */
+unsigned long nvme_ns_get_read_ios(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_write_ios() - Get num of write I/Os
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Num of write IOs processed between two consecutive stat samples
+ */
+unsigned long nvme_ns_get_write_ios(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_read_sectors() - Get num of read sectors
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Num of sectors read from the device between two stat samples
+ */
+unsigned long long nvme_ns_get_read_sectors(nvme_ns_t n, int curr);
+
+/**
+ * nvme_ns_get_write_sectors() - Get num of write sectors
+ * @n:		&nvme_ns_t object
+ * @curr:	Index in nvme_ns_stat_t object
+ *
+ * Return:	Num of sectors written to the device between two stat samples
+ */
+unsigned long long nvme_ns_get_write_sectors(nvme_ns_t n, int curr);
+
 /**
  * nvme_ctrl_get_config() - Fabrics configuration of a controller
  * @c:	Controller instance
-- 
2.53.0

[PATCH 6/9] libnvme: add support for per-path diagnostic counters

[Nilay Shroff]

Add support for retrieving per-path diagnostic counters such as
command_retry_count, command_error_count, and multipath_failover_count.
These counters improve visibility into NVMe native multipath behavior
and can be useful for tools such as nvme-top to display real-time
statistics.

Unlike other sysfs attributes, these counters can change dynamically.
Annotate them with "!accessors:none" and provide custom implementations
to always retrieve the latest (non-cached) values.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  3 +++
 libnvme/src/nvme/private.h |  3 +++
 libnvme/src/nvme/tree.c    | 33 +++++++++++++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 27 +++++++++++++++++++++++++++
 4 files changed, 66 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index c9550240d..20dcb24f1 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -167,6 +167,9 @@ LIBNVME_3 {
 		nvme_path_get_write_sectors;
 		nvme_path_reset_stat;
 		nvme_path_update_stat;
+		nvme_path_get_command_retry_count;
+		nvme_path_get_command_error_count;
+		nvme_path_get_multipath_failover_count;
 		nvme_filter_paths;
 		nvme_read_config;
 		nvme_read_key;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index 858759a1b..bbddb9ba0 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -167,6 +167,9 @@ struct nvme_path { /*!generate-accessors*/
 	char *numa_nodes; //!accessors:none
 	int grpid;
 	int queue_depth; //!accessors:none
+	long multipath_failover_count; //!accessors:none
+	long command_retry_count; //!accessors:none
+	long command_error_count; //!accessors:none
 };
 
 struct nvme_ns_head {
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 3ad483557..a197b7a00 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -851,6 +851,39 @@ __public char *nvme_path_get_numa_nodes(nvme_path_t p)
 	return p->numa_nodes;
 }
 
+__public long nvme_path_get_multipath_failover_count(nvme_path_t p)
+{
+	_cleanup_free_ char *failover_count = NULL;
+
+	failover_count = nvme_get_path_attr(p, "multipath_failover_count");
+	if (failover_count)
+		sscanf(failover_count, "%ld", &p->multipath_failover_count);
+
+	return p->multipath_failover_count;
+}
+
+__public long nvme_path_get_command_retry_count(nvme_path_t p)
+{
+	_cleanup_free_ char *retry_count = NULL;
+
+	retry_count = nvme_get_path_attr(p, "command_retry_count");
+	if (retry_count)
+		sscanf(retry_count, "%ld", &p->command_retry_count);
+
+	return p->command_retry_count;
+}
+
+__public long nvme_path_get_command_error_count(nvme_path_t p)
+{
+	_cleanup_free_ char *error_count = NULL;
+
+	error_count = nvme_get_path_attr(p, "command_error_count");
+	if (error_count)
+		sscanf(error_count, "%ld", &p->command_error_count);
+
+	return p->command_error_count;
+}
+
 static nvme_stat_t nvme_path_get_stat(nvme_path_t p, int curr)
 {
 	if (curr < 0 || curr > 1)
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index c69d0c8a0..3c651743d 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -704,6 +704,33 @@ char *nvme_path_get_ana_state(nvme_path_t p);
  */
 char *nvme_path_get_numa_nodes(nvme_path_t p);
 
+/**
+ * nvme_path_get_multipath_failover_count() - Get multipath failover count
+ * @p: &nvme_path_t object
+ *
+ * Return: Number of times I/Os have to be failed over to another active path
+ * from path @p maybe due to any transient error observed on path @p
+ */
+long nvme_path_get_multipath_failover_count(nvme_path_t p);
+
+/**
+ * nvme_path_get_command_retry_count() - Get command retry count
+ * @p: &nvme_path_t object
+ *
+ * Return: Number of times any command issued to the namespace represented by
+ * path @p has to be retried
+ */
+long nvme_path_get_command_retry_count(nvme_path_t p);
+
+/**
+ * nvme_path_get_command_error_count() - Get command error count
+ * @p: &nvme_path_t object
+ *
+ * Return: Number of times command issued to the namespace represented by path
+ * @p returns non-zero status or error
+ */
+long nvme_path_get_command_error_count(nvme_path_t p);
+
 /**
  * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object
  * @p:	&nvme_path_t object
-- 
2.53.0

[PATCH 7/9] libnvme: add support for namespace diagnostic counters

[Nilay Shroff]

Add support for retrieving namespace diagnostic counters such as
command_retry_count and command_error_count. These counters improve
visibility and can be useful for tools such as nvme-top to display
real-time statistics.

Unlike other sysfs attributes, these counters can change dynamically.
Annotate them with "!accessors:none" and provide custom implementations
to always retrieve the latest (non-cached) values.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  2 ++
 libnvme/src/nvme/private.h |  3 +++
 libnvme/src/nvme/tree.c    | 22 ++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 17 +++++++++++++++++
 4 files changed, 44 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index 20dcb24f1..a767eca5e 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -141,6 +141,8 @@ LIBNVME_3 {
 		nvme_ns_get_write_sectors;
 		nvme_ns_get_inflights;
 		nvme_ns_get_io_ticks;
+		nvme_ns_get_command_retry_count;
+		nvme_ns_get_command_error_count;
 		nvme_ns_identify;
 		nvme_ns_read;
 		nvme_rescan_ns;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index bbddb9ba0..f58b13bd7 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -206,6 +206,9 @@ struct nvme_ns { /*!generate-accessors*/
 	uint8_t nguid[16];
 	unsigned char uuid[NVME_UUID_LEN];
 	enum nvme_csi csi;
+
+	long command_retry_count; //!accessors:none
+	long command_error_count; //!accessors:none
 };
 
 struct nvme_ctrl { /*!generate-accessors*/
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index a197b7a00..f82ea4c02 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -2548,6 +2548,28 @@ __public void nvme_ns_get_uuid(nvme_ns_t n, unsigned char out[NVME_UUID_LEN])
 	memcpy(out, n->uuid, NVME_UUID_LEN);
 }
 
+__public long nvme_ns_get_command_retry_count(nvme_ns_t n)
+{
+	_cleanup_free_ char *retry_count = NULL;
+
+	retry_count = nvme_get_ns_attr(n, "command_retry_count");
+	if (retry_count)
+		sscanf(retry_count, "%ld", &n->command_retry_count);
+
+	return n->command_retry_count;
+}
+
+__public long nvme_ns_get_command_error_count(nvme_ns_t n)
+{
+	_cleanup_free_ char *error_count = NULL;
+
+	error_count = nvme_get_ns_attr(n, "command_error_count");
+	if (error_count)
+		sscanf(error_count, "%ld", &n->command_error_count);
+
+	return n->command_error_count;
+}
+
 __public int nvme_ns_identify(nvme_ns_t n, struct nvme_id_ns *ns)
 {
 	struct nvme_transport_handle *hdl;
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 3c651743d..6cff114bb 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -529,6 +529,23 @@ const uint8_t *nvme_ns_get_nguid(nvme_ns_t n);
  */
 void nvme_ns_get_uuid(nvme_ns_t n, unsigned char out[NVME_UUID_LEN]);
 
+/**
+ * nvme_ns_get_command_retry_count() - Get command retry count
+ * @n: &nvme_ns_t object
+ *
+ * Return: Number of times any command issued to namespace @n has to be retried
+ */
+long nvme_ns_get_command_retry_count(nvme_ns_t n);
+
+/**
+ * nvme_ns_get_command_error_count() - Get command error count
+ * @n: &nvme_ns_t object
+ *
+ * Return: Number of times command issued to namespace @n returns non-zero
+ * status or error
+ */
+long nvme_ns_get_command_error_count(nvme_ns_t n);
+
 /**
  * nvme_ns_get_generic_name() - Returns name of generic namespace chardev.
  * @n:	Namespace instance
-- 
2.53.0

[PATCH 8/9] libnvme: add support for nshead diagnostic counters

[Nilay Shroff]

Add support for retrieving nshead diagnostic counters such as requeue_
no_usable_path_count and fail_no_available_path_count.

The requeue_no_usable_path_count counter represents the number of I/Os
requeued to the namespace head when no usable path is available (for
example, due to transient link issues). The fail_no_available_path_count
counter represents the number of I/Os failed when no I/O path is
available (for example, when all paths are removed).

These counters improve visibility into NVMe native multipath behavior
and can be useful for tools such as nvme-top to display real-time
statistics.

Unlike other sysfs attributes, these counters can change dynamically.
Annotate them with "!accessors:none" and provide custom implementations
to always retrieve the latest (non-cached) values.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  2 ++
 libnvme/src/nvme/private.h |  2 ++
 libnvme/src/nvme/tree.c    | 22 ++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 17 +++++++++++++++++
 4 files changed, 43 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index a767eca5e..e3d2ca1bb 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -143,6 +143,8 @@ LIBNVME_3 {
 		nvme_ns_get_io_ticks;
 		nvme_ns_get_command_retry_count;
 		nvme_ns_get_command_error_count;
+		nvme_ns_get_requeue_no_usable_path_count;
+		nvme_ns_get_fail_no_available_path_count;
 		nvme_ns_identify;
 		nvme_ns_read;
 		nvme_rescan_ns;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index f58b13bd7..f7f8b4f76 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -209,6 +209,8 @@ struct nvme_ns { /*!generate-accessors*/
 
 	long command_retry_count; //!accessors:none
 	long command_error_count; //!accessors:none
+	long requeue_no_usable_path_count; //!accessors:none
+	long fail_no_available_path_count; //!accessors:none
 };
 
 struct nvme_ctrl { /*!generate-accessors*/
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index f82ea4c02..0f0761005 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -2570,6 +2570,28 @@ __public long nvme_ns_get_command_error_count(nvme_ns_t n)
 	return n->command_error_count;
 }
 
+__public long nvme_ns_get_requeue_no_usable_path_count(nvme_ns_t n)
+{
+	_cleanup_free_ char *requeue_count = NULL;
+
+	requeue_count = nvme_get_ns_attr(n, "requeue_no_usable_path_count");
+	if (requeue_count)
+		sscanf(requeue_count, "%ld", &n->requeue_no_usable_path_count);
+
+	return n->requeue_no_usable_path_count;
+}
+
+__public long nvme_ns_get_fail_no_available_path_count(nvme_ns_t n)
+{
+	_cleanup_free_ char *fail_count = NULL;
+
+	fail_count = nvme_get_ns_attr(n, "fail_no_available_path_count");
+	if (fail_count)
+		sscanf(fail_count, "%ld", &n->fail_no_available_path_count);
+
+	return n->fail_no_available_path_count;
+}
+
 __public int nvme_ns_identify(nvme_ns_t n, struct nvme_id_ns *ns)
 {
 	struct nvme_transport_handle *hdl;
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index 6cff114bb..a34a31601 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -546,6 +546,23 @@ long nvme_ns_get_command_retry_count(nvme_ns_t n);
  */
 long nvme_ns_get_command_error_count(nvme_ns_t n);
 
+/**
+ * nvme_ns_get_requeue_no_usable_path_count() - Get num of I/O requeue count
+ * @n: &nvme_ns_t object
+ *
+ * Return: Number of I/Os which are re-queued due to the unavalibility of
+ * any usable path (maybe path is currently experiencing transinet link failure)
+ */
+long nvme_ns_get_requeue_no_usable_path_count(nvme_ns_t n);
+
+/**
+ * nvme_ns_get_fail_no_available_path_count() - Get num of I/Os forced to fail
+ * @n: &nvme_ns_t object
+ *
+ * Return: Number of I/Os which are forced to fail due to no path available
+ */
+long nvme_ns_get_fail_no_available_path_count(nvme_ns_t n);
+
 /**
  * nvme_ns_get_generic_name() - Returns name of generic namespace chardev.
  * @n:	Namespace instance
-- 
2.53.0

[PATCH 9/9] libnvme: add support for ctrl diagnostic counters

[Nilay Shroff]

Add support for retrieving ctrl diagnostic counters such as command_
error_count, reset_count and reconnect_count. These counters improve
visibility and can be useful for tools such as nvme-top to display
real-time statistics.

Unlike other sysfs attributes, these counters can change dynamically.
Annotate them with "!accessors:none" and provide custom implementations
to always retrieve the latest (non-cached) values.

Signed-off-by: Nilay Shroff <nilay@linux.ibm.com>
---
 libnvme/src/libnvme.ld     |  3 +++
 libnvme/src/nvme/private.h |  3 +++
 libnvme/src/nvme/tree.c    | 33 +++++++++++++++++++++++++++++++++
 libnvme/src/nvme/tree.h    | 25 +++++++++++++++++++++++++
 4 files changed, 64 insertions(+)

diff --git a/libnvme/src/libnvme.ld b/libnvme/src/libnvme.ld
index e3d2ca1bb..964066e8c 100644
--- a/libnvme/src/libnvme.ld
+++ b/libnvme/src/libnvme.ld
@@ -16,6 +16,9 @@ LIBNVME_3 {
 		nvme_ctrl_get_subsysnqn;
 		nvme_ctrl_get_subsystem;
 		nvme_ctrl_get_transport_handle;
+		nvme_ctrl_get_command_error_count;
+		nvme_ctrl_get_reset_count;
+		nvme_ctrl_get_reconnect_count;
 		nvme_ctrl_identify;
 		nvme_ctrl_next_ns;
 		nvme_ctrl_next_path;
diff --git a/libnvme/src/nvme/private.h b/libnvme/src/nvme/private.h
index f7f8b4f76..12b6351f2 100644
--- a/libnvme/src/nvme/private.h
+++ b/libnvme/src/nvme/private.h
@@ -250,6 +250,9 @@ struct nvme_ctrl { /*!generate-accessors*/
 	bool unique_discovery_ctrl;
 	bool discovered;
 	bool persistent;
+	long command_error_count; //!accessors:none
+	long reset_count; //!accessors:none
+	long reconnect_count; //!accessors:none
 	struct nvme_fabrics_config cfg;
 };
 
diff --git a/libnvme/src/nvme/tree.c b/libnvme/src/nvme/tree.c
index 0f0761005..d15118c21 100644
--- a/libnvme/src/nvme/tree.c
+++ b/libnvme/src/nvme/tree.c
@@ -1400,6 +1400,39 @@ __public const char *nvme_ctrl_get_state(nvme_ctrl_t c)
 	return c->state;
 }
 
+__public long nvme_ctrl_get_command_error_count(nvme_ctrl_t c)
+{
+	_cleanup_free_ char *error_count = NULL;
+
+	error_count = nvme_get_ctrl_attr(c, "command_error_count");
+	if (error_count)
+		sscanf(error_count, "%ld", &c->command_error_count);
+
+	return c->command_error_count;
+}
+
+__public long nvme_ctrl_get_reset_count(nvme_ctrl_t c)
+{
+	_cleanup_free_ char *reset_count = NULL;
+
+	reset_count = nvme_get_ctrl_attr(c, "reset_count");
+	if (reset_count)
+		sscanf(reset_count, "%ld", &c->reset_count);
+
+	return c->reset_count;
+}
+
+__public long nvme_ctrl_get_reconnect_count(nvme_ctrl_t c)
+{
+	_cleanup_free_ char *reconnect_count = NULL;
+
+	reconnect_count = nvme_get_ctrl_attr(c, "reconnect_count");
+	if (reconnect_count)
+		sscanf(reconnect_count, "%ld", &c->reconnect_count);
+
+	return c->reconnect_count;
+}
+
 __public struct nvme_fabrics_config *nvme_ctrl_get_config(nvme_ctrl_t c)
 {
 	return &c->cfg;
diff --git a/libnvme/src/nvme/tree.h b/libnvme/src/nvme/tree.h
index a34a31601..b79254f86 100644
--- a/libnvme/src/nvme/tree.h
+++ b/libnvme/src/nvme/tree.h
@@ -1047,6 +1047,31 @@ unsigned long long nvme_ns_get_write_sectors(nvme_ns_t n, int curr);
  */
 struct nvme_fabrics_config *nvme_ctrl_get_config(nvme_ctrl_t c);
 
+/**
+ * nvme_ctrl_get_command_error_count() - Get admin command error count
+ * @c: Controller instance
+ *
+ * Return: Number of times admin command issued to controller @c failed or
+ * returned error status
+ */
+long nvme_ctrl_get_command_error_count(nvme_ctrl_t c);
+
+/**
+ * nvme_ctrl_get_reset_count() - Get controller reset count
+ * @c: Controller instance
+ *
+ * Return: Number of timer controller @c is reset
+ */
+long nvme_ctrl_get_reset_count(nvme_ctrl_t c);
+
+/**
+ * nvme_ctrl_get_reconnect_count() - Get controller reconnect count
+ * @c: Controller instance
+ *
+ * Return: Number of times controller has to reconnect to the target
+ */
+long nvme_ctrl_get_reconnect_count(nvme_ctrl_t c);
+
 /**
  * nvme_ctrl_identify() - Issues an 'identify controller' command
  * @c:	Controller instance
-- 
2.53.0

Re: [PATCH 1/9] libnvme: annotate nvme_path::ana_state with !accessors:none

[Daniel Wagner]

On Sat, Mar 21, 2026 at 08:58:00PM +0530, Nilay Shroff wrote:
> +__public char *nvme_path_get_ana_state(nvme_path_t p)
> +{
> +	_cleanup_free_ char *ana_state = NULL;
> +
> +	ana_state = nvme_get_path_attr(p, "ana_state");
> +	if (ana_state) {
> +		if (!p->ana_state || strcmp(ana_state, p->ana_state)) {
> +			free(p->ana_state);
> +			p->ana_state = strdup(ana_state);
> +
> +	}

a couple of tabs indention here

> +	}
> +
> +	return p->ana_state;
> +}

Re: [PATCH 4/9] libnvme: add support for retrieving per-path gendisk I/O statistics

[Daniel Wagner]

On Sat, Mar 21, 2026 at 08:58:03PM +0530, Nilay Shroff wrote:
> +static int nvme_update_stat(const char *sysfs_stat_path, nvme_stat_t stat)
> +{
> +	int n;
> +	struct timespec ts;
> +	unsigned long rd_ios, rd_merges, wr_ios, wr_merges;
> +	unsigned long dc_ios, dc_merges, fl_ios;
> +	unsigned long long rd_sectors, wr_sectors, dc_sectors;
> +	unsigned int rd_ticks, wr_ticks, dc_ticks, fl_ticks;
> +	unsigned int io_ticks, tot_ticks, inflights;
> +
> +	memset(stat, 0, sizeof(struct nvme_stat));
> +
> +	n = sscanf(sysfs_stat_path,
> +		"%lu %lu %llu %u %lu %lu %llu %u %u %u %u %lu %lu %llu %u %lu %u",
> +		&rd_ios, &rd_merges, &rd_sectors, &rd_ticks,
> +		&wr_ios, &wr_merges, &wr_sectors, &wr_ticks,
> +		&inflights, &io_ticks, &tot_ticks,
> +		&dc_ios, &dc_merges, &dc_sectors, &dc_ticks,
> +		&fl_ios, &fl_ticks);
> +
> +	if (n < 17)
> +		return -1;

return a proper error code, e.g. -EINVAL or whatever matches better.

> +__public int nvme_path_update_stat(nvme_path_t p, int curr)

Do we need to expose 'curr' to the user? I'd prefer not to expose this
implementation detail to the user if possible.

> +{
> +	_cleanup_free_ char *sysfs_stat_path = NULL;
> +	nvme_stat_t stat;
> +
> +	stat = nvme_path_get_stat(p, curr);
> +	if (!stat)
> +		return -1;

return an error code instead -1. same comment for the rest of the patch.

> +__public unsigned int nvme_path_get_io_ticks(nvme_path_t p, int this)
> +{
> +	nvme_stat_t curr, prev;
> +
> +	curr = nvme_path_get_stat(p, this);
> +	prev = nvme_path_get_stat(p, !this);

Do we really need to expose 'this'?

Re: [PATCH 5/9] libnvme: add support for retrieving namespace gendisk I/O statistics

[Daniel Wagner]

On Sat, Mar 21, 2026 at 08:58:04PM +0530, Nilay Shroff wrote:
> Gendisk I/O statistics provide useful insight into disk activity,
> including read/write/discard/flush operations, as well as information
> about in-flight I/Os and I/O timing.
> 
> Parsing these statistics allows users to determine the number of I/Os
> processed, time spent servicing I/O, number of sectors accessed, and
> the count of in-flight requests.
> 
> Add support for retrieving namespace gendisk I/O statistics. Also add
> support for computing deltas of these statistics between samples, such
> as I/O ticks, number of sectors, and number of serviced I/Os.
> 
> These metrics can be used by tools such as nvme-top to display real-time
> disk activity.

Same comments as in the previous patch.
  - error codes
  - is it necessary to expose 'cur', 'this' via the API?

Re: [PATCH 6/9] libnvme: add support for per-path diagnostic counters

[Daniel Wagner]

On Sat, Mar 21, 2026 at 08:58:05PM +0530, Nilay Shroff wrote:
> Add support for retrieving per-path diagnostic counters such as
> command_retry_count, command_error_count, and multipath_failover_count.
> These counters improve visibility into NVMe native multipath behavior
> and can be useful for tools such as nvme-top to display real-time
> statistics.
> 
> Unlike other sysfs attributes, these counters can change dynamically.
> Annotate them with "!accessors:none" and provide custom implementations
> to always retrieve the latest (non-cached) values.

If I got this right, this and the next patches depend on

   https://lore.kernel.org/all/20260220175024.292898-1-nilay@linux.ibm.com

Should we wait with interfaces here until they are ready? As we
currently still in the development phase of nvme-cli 3 and thus the API
is not stable, we can touch it again. In the past we also added support
for kernel feature which weren't there yet. I don't mind too much, just
asking what would be best here.

The whole series looks pretty good, just my nitpicks and the question
on exposing 'cur'/'this'.

Re: [PATCH 4/9] libnvme: add support for retrieving per-path gendisk I/O statistics

[Nilay Shroff]

On 3/24/26 2:35 PM, Daniel Wagner wrote:
> On Sat, Mar 21, 2026 at 08:58:03PM +0530, Nilay Shroff wrote:
>> +static int nvme_update_stat(const char *sysfs_stat_path, nvme_stat_t stat)
>> +{
>> +	int n;
>> +	struct timespec ts;
>> +	unsigned long rd_ios, rd_merges, wr_ios, wr_merges;
>> +	unsigned long dc_ios, dc_merges, fl_ios;
>> +	unsigned long long rd_sectors, wr_sectors, dc_sectors;
>> +	unsigned int rd_ticks, wr_ticks, dc_ticks, fl_ticks;
>> +	unsigned int io_ticks, tot_ticks, inflights;
>> +
>> +	memset(stat, 0, sizeof(struct nvme_stat));
>> +
>> +	n = sscanf(sysfs_stat_path,
>> +		"%lu %lu %llu %u %lu %lu %llu %u %u %u %u %lu %lu %llu %u %lu %u",
>> +		&rd_ios, &rd_merges, &rd_sectors, &rd_ticks,
>> +		&wr_ios, &wr_merges, &wr_sectors, &wr_ticks,
>> +		&inflights, &io_ticks, &tot_ticks,
>> +		&dc_ios, &dc_merges, &dc_sectors, &dc_ticks,
>> +		&fl_ios, &fl_ticks);
>> +
>> +	if (n < 17)
>> +		return -1;
> 
> return a proper error code, e.g. -EINVAL or whatever matches better.

Yes will return the proper error code.

> 
>> +__public int nvme_path_update_stat(nvme_path_t p, int curr)
> 
> Do we need to expose 'curr' to the user? I'd prefer not to expose this
> implementation detail to the user if possible.
> 

Here, @curr represents the bucket index. As you may have noticed,
nvme_path_t maintains a stats[2] array that stores both the previous
and current statistics.

Typically, at a regular interval (for example, when nvme-top runs with
a 1-second interval), the user/nvme-cli invokes nvme_path_update_stat().
This function reads the gendisk I/O statistics and updates one of the
entries in stats[]. The @curr parameter indicates which bucket should
be updated with the latest values.

After updating the stat, the caller/nvme-cli invokes nvme_path_get_xxx()
APIs to retrieve specific metrics. These APIs also take the current index
and compute results by comparing the current and previous statistics.

In the next interval, the caller/nvme-cli toggles the index. This effectively
swaps the roles of the two buckets—what was previously the current stats becomes
the previous stats, and vice versa. The caller then invokes nvme_path_update_stat()
again with the updated index, followed by the nvme_path_get_xxx() APIs to
retrieve the latest computed statistics.

 From this, you can see that if this logic were handled entirely within
libnvme, the library would need to maintain internal state to track which
bucket to update. Instead, I prefer leaving this responsibility to the
caller, since it already operates within the interval loop. This keeps the
library stateless and gives the caller explicit control over the bucket
selection for each interval. Does this make sense?

>> +{
>> +	_cleanup_free_ char *sysfs_stat_path = NULL;
>> +	nvme_stat_t stat;
>> +
>> +	stat = nvme_path_get_stat(p, curr);
>> +	if (!stat)
>> +		return -1;
> 
> return an error code instead -1. same comment for the rest of the patch.
> 
>> +__public unsigned int nvme_path_get_io_ticks(nvme_path_t p, int this)
>> +{
>> +	nvme_stat_t curr, prev;
>> +
>> +	curr = nvme_path_get_stat(p, this);
>> +	prev = nvme_path_get_stat(p, !this);
> 
> Do we really need to expose 'this'?
> 
As explained above @this is also a bucket index which is used to
calculate/derive the io_ticks comparing the ticks values in the
current and previous stat buckets.

Thanks,
--Nilay

Re: [PATCH 1/9] libnvme: annotate nvme_path::ana_state with !accessors:none

[Nilay Shroff]

On 3/24/26 2:25 PM, Daniel Wagner wrote:
> On Sat, Mar 21, 2026 at 08:58:00PM +0530, Nilay Shroff wrote:
>> +__public char *nvme_path_get_ana_state(nvme_path_t p)
>> +{
>> +	_cleanup_free_ char *ana_state = NULL;
>> +
>> +	ana_state = nvme_get_path_attr(p, "ana_state");
>> +	if (ana_state) {
>> +		if (!p->ana_state || strcmp(ana_state, p->ana_state)) {
>> +			free(p->ana_state);
>> +			p->ana_state = strdup(ana_state);
>> +
>> +	}
> 
> a couple of tabs indention here
> 
Ah, good catch. Will fix.

>> +	}
>> +
>> +	return p->ana_state;
>> +}

Thanks,
--Nilay

Re: [PATCH 5/9] libnvme: add support for retrieving namespace gendisk I/O statistics

[Nilay Shroff]

On 3/24/26 2:36 PM, Daniel Wagner wrote:
> On Sat, Mar 21, 2026 at 08:58:04PM +0530, Nilay Shroff wrote:
>> Gendisk I/O statistics provide useful insight into disk activity,
>> including read/write/discard/flush operations, as well as information
>> about in-flight I/Os and I/O timing.
>>
>> Parsing these statistics allows users to determine the number of I/Os
>> processed, time spent servicing I/O, number of sectors accessed, and
>> the count of in-flight requests.
>>
>> Add support for retrieving namespace gendisk I/O statistics. Also add
>> support for computing deltas of these statistics between samples, such
>> as I/O ticks, number of sectors, and number of serviced I/Os.
>>
>> These metrics can be used by tools such as nvme-top to display real-time
>> disk activity.
> 
> Same comments as in the previous patch.
>    - error codes
>    - is it necessary to expose 'cur', 'this' via the API?

Yeah will use proper error codes.

The same rationale as in the previous patch applies here for exposing cur/this.
The only difference is that the earlier patch handled nvme_path_t statistics,
while this one operates on nvme_ns_t statistics.

Thanks,
--Nilay

Re: [PATCH 6/9] libnvme: add support for per-path diagnostic counters

[Nilay Shroff]

On 3/24/26 2:48 PM, Daniel Wagner wrote:
> On Sat, Mar 21, 2026 at 08:58:05PM +0530, Nilay Shroff wrote:
>> Add support for retrieving per-path diagnostic counters such as
>> command_retry_count, command_error_count, and multipath_failover_count.
>> These counters improve visibility into NVMe native multipath behavior
>> and can be useful for tools such as nvme-top to display real-time
>> statistics.
>>
>> Unlike other sysfs attributes, these counters can change dynamically.
>> Annotate them with "!accessors:none" and provide custom implementations
>> to always retrieve the latest (non-cached) values.
> 
> If I got this right, this and the next patches depend on
> 
>     https://lore.kernel.org/all/20260220175024.292898-1-nilay@linux.ibm.com
> 
Yes you're correct, I am just waiting for Keith's response on that thread here:
https://lore.kernel.org/all/0832f4dd-0ab6-4e11-a404-fdb18ca699b2@linux.ibm.com/

> Should we wait with interfaces here until they are ready? As we
> currently still in the development phase of nvme-cli 3 and thus the API
> is not stable, we can touch it again. In the past we also added support
> for kernel feature which weren't there yet. I don't mind too much, just
> asking what would be best here.
> 

I’m fine either way regarding when to introduce these interfaces.
Even if we include the diagnostic counter patches now, it won’t break
anything. If the corresponding sysfs attributes are not present, the
library will simply return 0 for those counters.

There are essentially two options,

1. Include the diagnostic counter patches now:
    In this case, nvme-cli/nvme-top can start using the libnvme APIs immediately.
    If the kernel does not yet expose these attributes, the counters will appear
    as zero in the dashboard. Once the kernel support lands, the correct values
    will automatically be reflected without requiring further changes to libnvme
    or nvme-cli (unless the sysfs layout changes).

2. Defer the diagnostic counter patches:
    In this case, the counters would be omitted from the nvme-top dashboard for now.
    Once kernel support is available, we would need to update nvme-top (and libnvme)
    to add them.

Given these options, I would lean towards option (1), but I’m happy to go with
your preference.

> The whole series looks pretty good, just my nitpicks and the question
> on exposing 'cur'/'this'.

I will address nitpick comments in next patchset, and I have provided the rationale
about why we need to expose 'cur'/'this' in an earlier reply.

Thanks,
--Nilay

Re: [PATCH 4/9] libnvme: add support for retrieving per-path gendisk I/O statistics

[Daniel Wagner]

Hi Nilay,

Sorry for the late response.

On Tue, Mar 24, 2026 at 06:32:59PM +0530, Nilay Shroff wrote:
> After updating the stat, the caller/nvme-cli invokes nvme_path_get_xxx()
> APIs to retrieve specific metrics. These APIs also take the current index
> and compute results by comparing the current and previous statistics.

Let's try to avoid leaking internal implementation details to the public
API. This makes it unnecessary hard to maintain.

The stats update function could take an argument what it should do:
  - update internal stats, calculate diff -> getters return the diff
  - update internal stats, getters return the absolute value

Thanks,
Daniel

Re: [PATCH 6/9] libnvme: add support for per-path diagnostic counters

[Daniel Wagner]

On Tue, Mar 24, 2026 at 07:24:11PM +0530, Nilay Shroff wrote:
> There are essentially two options,
> 
> 1. Include the diagnostic counter patches now:
>    In this case, nvme-cli/nvme-top can start using the libnvme APIs immediately.
>    If the kernel does not yet expose these attributes, the counters will appear
>    as zero in the dashboard. Once the kernel support lands, the correct values
>    will automatically be reflected without requiring further changes to libnvme
>    or nvme-cli (unless the sysfs layout changes).
> 
> 2. Defer the diagnostic counter patches:
>    In this case, the counters would be omitted from the nvme-top dashboard for now.
>    Once kernel support is available, we would need to update nvme-top (and libnvme)
>    to add them.
> 
> Given these options, I would lean towards option (1), but I’m happy to go with
> your preference.

I’m not sure if I read the room right, but I sensed the patches might be
blocked by the libmultipath project. If they’re just delayed for the
usual reasons, I’m fine with option (1).

FWIW, the 3.0 release is still months away, so there’s no hurry. I still
have a few TODO items and want to wait for feedback from LSFMM. I’ll
likely start the -rc cycle after that. Realistically, 3.0 won't land
until August.

Re: [PATCH 4/9] libnvme: add support for retrieving per-path gendisk I/O statistics

[Nilay Shroff]

On 4/1/26 9:12 PM, Daniel Wagner wrote:
> Hi Nilay,
> 
> Sorry for the late response.
No worries:-)

> 
> On Tue, Mar 24, 2026 at 06:32:59PM +0530, Nilay Shroff wrote:
>> After updating the stat, the caller/nvme-cli invokes nvme_path_get_xxx()
>> APIs to retrieve specific metrics. These APIs also take the current index
>> and compute results by comparing the current and previous statistics.
> 
> Let's try to avoid leaking internal implementation details to the public
> API. This makes it unnecessary hard to maintain.
> 
> The stats update function could take an argument what it should do:
>    - update internal stats, calculate diff -> getters return the diff
>    - update internal stats, getters return the absolute value
> 
Yes got it.. I think we could do this way. I will give it a shot
and respin a new patchset with the above change.

Thanks,
--Nilay

Re: [PATCH 6/9] libnvme: add support for per-path diagnostic counters

[Nilay Shroff]

On 4/1/26 9:24 PM, Daniel Wagner wrote:
> On Tue, Mar 24, 2026 at 07:24:11PM +0530, Nilay Shroff wrote:
>> There are essentially two options,
>>
>> 1. Include the diagnostic counter patches now:
>>     In this case, nvme-cli/nvme-top can start using the libnvme APIs immediately.
>>     If the kernel does not yet expose these attributes, the counters will appear
>>     as zero in the dashboard. Once the kernel support lands, the correct values
>>     will automatically be reflected without requiring further changes to libnvme
>>     or nvme-cli (unless the sysfs layout changes).
>>
>> 2. Defer the diagnostic counter patches:
>>     In this case, the counters would be omitted from the nvme-top dashboard for now.
>>     Once kernel support is available, we would need to update nvme-top (and libnvme)
>>     to add them.
>>
>> Given these options, I would lean towards option (1), but I’m happy to go with
>> your preference.
> 
> I’m not sure if I read the room right, but I sensed the patches might be
> blocked by the libmultipath project. If they’re just delayed for the
> usual reasons, I’m fine with option (1).
> 
I think that the proposed kernel changes shouldn't come in way for
libmultipath work, but let me ping Keith one more time... Oh I see
Keith is already in this mail list..

Keith,
I am not sure if you got a chance to review this:
https://lore.kernel.org/all/0832f4dd-0ab6-4e11-a404-fdb18ca699b2@linux.ibm.com/

But if possible, would you please help let us know your thought/suggestion about
the above thread?

> FWIW, the 3.0 release is still months away, so there’s no hurry. I still
> have a few TODO items and want to wait for feedback from LSFMM. I’ll
> likely start the -rc cycle after that. Realistically, 3.0 won't land
> until August.

Yes understood, sounds reasonable...

Thanks,
--Nilay