* [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns
@ 2025-04-17 13:59 Nilay Shroff
2025-04-17 13:59 ` [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link Nilay Shroff
` (3 more replies)
0 siblings, 4 replies; 14+ messages in thread
From: Nilay Shroff @ 2025-04-17 13:59 UTC (permalink / raw)
To: linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce
Hi,
Recently released Linux kernel v6.15-rc1 added support for easily discovering
multiple paths to a shared NVMe namespace. This multipath information is
exposed to userspace via a new sysfs group attribute named "multipath",
located under /sys/block/<ns-blkdev>/. More details on this functionality
can be found here [1].
This patch series leverages that new functionality to discover multiple paths
to a shared namespace and exposes that information in libnvme so that it can
later be used by nvme-cli.
There are four patches in this series:
The first patch adds support for discovering NVMe paths using the sysfs
"multipath" group attribute.
The second patch adds a new "queue_depth" attribute under the NVMe path
object.
The third patch adds a new "numa_nodes" attribute under the NVMe path object.
The fourth patch extends the sysfs tree dump test to validate multipath link
support
[1]: https://lore.kernel.org/all/20250112124154.60690-1-nilay@linux.ibm.com/
Changes from v1:
- Fixed kernel version typo in the cover letter and commit
- Updated the commit in first patch to explain the change in some
detail (Daniel Wagner)
- Added fourth patch to validate changes (Daniel Wagner)
Nilay Shroff (4):
tree: add support for discovering nvme paths using sysfs multipath
link
tree: add queue-depth attribute for nvme path object
tree: add attribute numa_nodes for NVMe path object
test: extend sysfs tree dump test
src/libnvme.map | 2 +
src/nvme/filters.c | 6 +
src/nvme/filters.h | 9 ++
src/nvme/json.c | 90 +++++++++++++--
src/nvme/private.h | 11 +-
src/nvme/tree.c | 184 ++++++++++++++++++++++---------
src/nvme/tree.h | 25 +++++
test/sysfs/data/tree-pcie.out | 53 +++++++--
test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes
9 files changed, 308 insertions(+), 72 deletions(-)
--
2.49.0
^ permalink raw reply [flat|nested] 14+ messages in thread* [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link 2025-04-17 13:59 [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns Nilay Shroff @ 2025-04-17 13:59 ` Nilay Shroff 2025-04-22 6:24 ` Hannes Reinecke 2025-04-17 13:59 ` [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object Nilay Shroff ` (2 subsequent siblings) 3 siblings, 1 reply; 14+ messages in thread From: Nilay Shroff @ 2025-04-17 13:59 UTC (permalink / raw) To: linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce With the upcoming Linux kernel v6.15, NVMe native multipath now provides a simplified mechanism for discovering all paths to a shared namespace through sysfs. A new "multipath" directory is created under each NVMe head namespace device in "/sys/block/<head>/multipath/". This directory contains symlinks to all namespace path devices that access the same shared namespace. For example, consider a shared namespace accessible via two paths under nvme-subsys1: nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \ +- ns 1 \ +- nvme0 pcie 052e:78:00.0 live optimized +- nvme1 pcie 058e:78:00.0 live optimized The head device `/dev/nvme1n1` will now have the following structure: /sys/block/nvme1n1/multipath/ ├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1 This clearly shows that namespace 1 is accessible through both nvme1c0n1 and nvme1c1n1. This new sysfs structure significantly simplifies multipath discovery and management, making it easier for tools and scripts to enumerate and manage NVMe multipath configurations. So leverage this functionality to update the path links for a shared NVMe namespace, simplifying path discovery and management. This change adds a new struct nvme_ns_head to represent the head of a shared namespace. It contains a list head linking together struct nvme_path objects, where each path corresponds to a shared namespace instance. Additionally, struct nvme_ns has been updated to reference its associated nvme_ns_head, enabling straightforward traversal of all paths to a shared NVMe namespace. Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> --- src/nvme/filters.c | 6 ++ src/nvme/filters.h | 9 +++ src/nvme/private.h | 9 ++- src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- src/nvme/tree.h | 9 +++ 5 files changed, 143 insertions(+), 52 deletions(-) diff --git a/src/nvme/filters.c b/src/nvme/filters.c index ceaba68f..4a8829db 100644 --- a/src/nvme/filters.c +++ b/src/nvme/filters.c @@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns) return scandir(nvme_ctrl_get_sysfs_dir(c), ns, nvme_namespace_filter, alphasort); } + +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths) +{ + return scandir(nvme_ns_head_get_sysfs_dir(head), paths, + nvme_paths_filter, alphasort); +} diff --git a/src/nvme/filters.h b/src/nvme/filters.h index 4ceeffd5..9e9dbb95 100644 --- a/src/nvme/filters.h +++ b/src/nvme/filters.h @@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths); */ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns); +/** + * nvme_scan_ns_head_paths() - Scan for namespace paths + * @head: Namespace head node to scan + * @paths : Pointer to array of dirents + * + * Return: number of entries in @ents + */ +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths); + #endif /* _LIBNVME_FILTERS_H */ diff --git a/src/nvme/private.h b/src/nvme/private.h index 33cdd555..f45c5823 100644 --- a/src/nvme/private.h +++ b/src/nvme/private.h @@ -36,12 +36,19 @@ struct nvme_path { int grpid; }; +struct nvme_ns_head { + struct list_head paths; + struct nvme_ns *n; + + char *sysfs_dir; +}; + struct nvme_ns { struct list_node entry; - struct list_head paths; struct nvme_subsystem *s; struct nvme_ctrl *c; + struct nvme_ns_head *head; int fd; __u32 nsid; diff --git a/src/nvme/tree.c b/src/nvme/tree.c index b0a4696f..bd7fb53e 100644 --- a/src/nvme/tree.c +++ b/src/nvme/tree.c @@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n) nvme_path_t nvme_namespace_first_path(nvme_ns_t ns) { - return list_top(&ns->paths, struct nvme_path, nentry); + return list_top(&ns->head->paths, struct nvme_path, nentry); } nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p) { - return p ? list_next(&ns->paths, p, nentry) : NULL; + return p ? list_next(&ns->head->paths, p, nentry) : NULL; } static void __nvme_free_ns(struct nvme_ns *n) @@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n) free(n->generic_name); free(n->name); free(n->sysfs_dir); + free(n->head->sysfs_dir); + free(n->head); free(n); } @@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p) free(p); } -static void nvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p) -{ - char n_name[32] = { }; - int i, c, nsid, ret; - nvme_ns_t n; - - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid); - if (ret != 3) - return; - - sprintf(n_name, "nvme%dn%d", i, nsid); - nvme_subsystem_for_each_ns(s, n) { - if (!strcmp(n_name, nvme_ns_get_name(n))) { - list_add_tail(&n->paths, &p->nentry); - p->n = n; - } - } -} - static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) { struct nvme_path *p; @@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) } list_node_init(&p->nentry); - nvme_subsystem_set_path_ns(c->s, p); list_node_init(&p->entry); list_add_tail(&c->paths, &p->entry); return 0; @@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name) return NULL; path = NULL; - nvme_ctrl_scan_namespaces(r, c); nvme_ctrl_scan_paths(r, c); + nvme_ctrl_scan_namespaces(r, c); return c; } @@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n) return n->sysfs_dir; } +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head) +{ + return head->sysfs_dir; +} + const char *nvme_ns_get_name(nvme_ns_t n) { return n->name; @@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name) static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) { + int ret; struct nvme_ns *n; + struct nvme_ns_head *head; + struct stat arg; + _cleanup_free_ char *path = NULL; n = calloc(1, sizeof(*n)); if (!n) { @@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) return NULL; } + head = calloc(1, sizeof(*head)); + if (!head) { + errno = ENOMEM; + free(n); + return NULL; + } + + head->n = n; + list_head_init(&head->paths); + ret = asprintf(&path, "%s/%s", sys_path, "multipath"); + if (ret < 0) { + errno = ENOMEM; + goto free_ns_head; + } + /* + * The sysfs-dir "multipath" is available only when nvme multipath + * is configured and we're running kernel version >= 6.14. + */ + ret = stat(path, &arg); + if (ret == 0) { + head->sysfs_dir = path; + path = NULL; + } else + head->sysfs_dir = NULL; + + n->head = head; n->fd = -1; n->name = strdup(name); @@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) if (nvme_ns_init(sys_path, n) != 0) goto free_ns; - list_head_init(&n->paths); list_node_init(&n->entry); nvme_ns_release_fd(n); /* Do not leak fds */ + return n; free_ns: free(n->generic_name); free(n->name); +free_ns_head: + free(head); free(n); return NULL; } @@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name) return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name); } + +static void nvme_ns_head_scan_path(nvme_subsystem_t s, nvme_ns_t n, char *name) +{ + nvme_ctrl_t c; + nvme_path_t p; + + nvme_subsystem_for_each_ctrl(s, c) { + nvme_ctrl_for_each_path(c, p) { + if (!strcmp(nvme_path_get_name(p), name)) { + list_add_tail(&n->head->paths, &p->nentry); + p->n = n; + return; + } + } + } +} + +static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) +{ + struct nvme_ns_head *head = n->head; + + if (nvme_ns_head_get_sysfs_dir(head)) { + struct dirents paths = {}; + int i; + + /* + * When multipath is configured on kernel version >= 6.14, + * we use multipath sysfs link to get each path of a namespace. + */ + paths.num = nvme_scan_ns_head_paths(head, &paths.ents); + + for (i = 0; i < paths.num; i++) + nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name); + } else { + nvme_ctrl_t c; + nvme_path_t p; + int ns_ctrl, ns_nsid, ret; + + /* + * If multipath is not configured or we're running on kernel + * version < 6.14, fallback to the old way. + */ + ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", + &ns_ctrl, &ns_nsid); + if (ret != 2) + return; + + nvme_subsystem_for_each_ctrl(s, c) { + nvme_ctrl_for_each_path(c, p) { + int p_subsys, p_ctrl, p_nsid; + + ret = sscanf(nvme_path_get_name(p), + "nvme%dc%dn%d", + &p_subsys, &p_ctrl, &p_nsid); + if (ret != 3) + continue; + if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { + list_add_tail(&head->paths, &p->nentry); + p->n = n; + } + } + } + } +} + static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, char *name) { @@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, n->s = c->s; n->c = c; list_add_tail(&c->namespaces, &n->entry); - return 0; -} - -static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) -{ - nvme_ctrl_t c; - nvme_path_t p; - int ns_ctrl, ns_nsid, ret; - - ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid); - if (ret != 2) - return; + nvme_subsystem_set_ns_path(c->s, n); - nvme_subsystem_for_each_ctrl(s, c) { - nvme_ctrl_for_each_path(c, p) { - int p_subsys, p_ctrl, p_nsid; - - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", - &p_subsys, &p_ctrl, &p_nsid); - if (ret != 3) - continue; - if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { - list_add_tail(&n->paths, &p->nentry); - p->n = n; - } - } - } + return 0; } static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s, @@ -2917,7 +2977,7 @@ static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s, list_del_init(&p->nentry); p->n = NULL; } - list_head_init(&_n->paths); + list_head_init(&_n->head->paths); __nvme_free_ns(_n); } n->s = s; diff --git a/src/nvme/tree.h b/src/nvme/tree.h index 25d4b31b..9f382e9c 100644 --- a/src/nvme/tree.h +++ b/src/nvme/tree.h @@ -27,6 +27,7 @@ */ typedef struct nvme_ns *nvme_ns_t; +typedef struct nvme_ns_head *nvme_ns_head_t; typedef struct nvme_path *nvme_path_t; typedef struct nvme_ctrl *nvme_ctrl_t; typedef struct nvme_subsystem *nvme_subsystem_t; @@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key); */ const char *nvme_ctrl_get_dhchap_key(nvme_ctrl_t c); +/** + * nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head + * @head: namespace head instance + * + * Returns: sysfs directory name of @head + */ +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head); + /** * nvme_ctrl_set_dhchap_key() - Set controller key * @c: Controller for which the key should be set -- 2.49.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link 2025-04-17 13:59 ` [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link Nilay Shroff @ 2025-04-22 6:24 ` Hannes Reinecke 2025-04-22 14:01 ` Nilay Shroff 0 siblings, 1 reply; 14+ messages in thread From: Hannes Reinecke @ 2025-04-22 6:24 UTC (permalink / raw) To: Nilay Shroff, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/17/25 15:59, Nilay Shroff wrote: > With the upcoming Linux kernel v6.15, NVMe native multipath now provides > a simplified mechanism for discovering all paths to a shared namespace > through sysfs. > > A new "multipath" directory is created under each NVMe head namespace > device in "/sys/block/<head>/multipath/". This directory contains symlinks > to all namespace path devices that access the same shared namespace. > > For example, consider a shared namespace accessible via two paths under > nvme-subsys1: > > nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 > hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 > \ > +- ns 1 > \ > +- nvme0 pcie 052e:78:00.0 live optimized > +- nvme1 pcie 058e:78:00.0 live optimized > > The head device `/dev/nvme1n1` will now have the following structure: > > /sys/block/nvme1n1/multipath/ > ├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 > └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1 > > This clearly shows that namespace 1 is accessible through both nvme1c0n1 > and nvme1c1n1. This new sysfs structure significantly simplifies multipath > discovery and management, making it easier for tools and scripts to enumerate > and manage NVMe multipath configurations. So leverage this functionality to > update the path links for a shared NVMe namespace, simplifying path discovery > and management. > > This change adds a new struct nvme_ns_head to represent the head of a shared > namespace. It contains a list head linking together struct nvme_path objects, > where each path corresponds to a shared namespace instance. Additionally, > struct nvme_ns has been updated to reference its associated nvme_ns_head, > enabling straightforward traversal of all paths to a shared NVMe namespace. > > Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> > --- > src/nvme/filters.c | 6 ++ > src/nvme/filters.h | 9 +++ > src/nvme/private.h | 9 ++- > src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- > src/nvme/tree.h | 9 +++ > 5 files changed, 143 insertions(+), 52 deletions(-) > > diff --git a/src/nvme/filters.c b/src/nvme/filters.c > index ceaba68f..4a8829db 100644 > --- a/src/nvme/filters.c > +++ b/src/nvme/filters.c > @@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns) > return scandir(nvme_ctrl_get_sysfs_dir(c), ns, > nvme_namespace_filter, alphasort); > } > + > +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths) > +{ > + return scandir(nvme_ns_head_get_sysfs_dir(head), paths, > + nvme_paths_filter, alphasort); > +} > diff --git a/src/nvme/filters.h b/src/nvme/filters.h > index 4ceeffd5..9e9dbb95 100644 > --- a/src/nvme/filters.h > +++ b/src/nvme/filters.h > @@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths); > */ > int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns); > > +/** > + * nvme_scan_ns_head_paths() - Scan for namespace paths > + * @head: Namespace head node to scan > + * @paths : Pointer to array of dirents > + * > + * Return: number of entries in @ents > + */ > +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths); > + > #endif /* _LIBNVME_FILTERS_H */ > diff --git a/src/nvme/private.h b/src/nvme/private.h > index 33cdd555..f45c5823 100644 > --- a/src/nvme/private.h > +++ b/src/nvme/private.h > @@ -36,12 +36,19 @@ struct nvme_path { > int grpid; > }; > > +struct nvme_ns_head { > + struct list_head paths; > + struct nvme_ns *n; > + > + char *sysfs_dir; > +}; > + > struct nvme_ns { > struct list_node entry; > - struct list_head paths; > > struct nvme_subsystem *s; > struct nvme_ctrl *c; > + struct nvme_ns_head *head; > > int fd; > __u32 nsid; Why isn't 'nvme_subsystem' moved to 'nvme_ns_head'? That certainly is how the kernel structures are laid out, and it would make sense to follow it here... > diff --git a/src/nvme/tree.c b/src/nvme/tree.c > index b0a4696f..bd7fb53e 100644 > --- a/src/nvme/tree.c > +++ b/src/nvme/tree.c > @@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n) > > nvme_path_t nvme_namespace_first_path(nvme_ns_t ns) > { > - return list_top(&ns->paths, struct nvme_path, nentry); > + return list_top(&ns->head->paths, struct nvme_path, nentry); > } > > nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p) > { > - return p ? list_next(&ns->paths, p, nentry) : NULL; > + return p ? list_next(&ns->head->paths, p, nentry) : NULL; > } > > static void __nvme_free_ns(struct nvme_ns *n) > @@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n) > free(n->generic_name); > free(n->name); > free(n->sysfs_dir); > + free(n->head->sysfs_dir); > + free(n->head); > free(n); > } > > @@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p) > free(p); > } > > -static void nvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p) > -{ > - char n_name[32] = { }; > - int i, c, nsid, ret; > - nvme_ns_t n; > - > - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid); > - if (ret != 3) > - return; > - > - sprintf(n_name, "nvme%dn%d", i, nsid); > - nvme_subsystem_for_each_ns(s, n) { > - if (!strcmp(n_name, nvme_ns_get_name(n))) { > - list_add_tail(&n->paths, &p->nentry); > - p->n = n; > - } > - } > -} > - > static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) > { > struct nvme_path *p; > @@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) > } > > list_node_init(&p->nentry); > - nvme_subsystem_set_path_ns(c->s, p); > list_node_init(&p->entry); > list_add_tail(&c->paths, &p->entry); > return 0; > @@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name) > return NULL; > > path = NULL; > - nvme_ctrl_scan_namespaces(r, c); > nvme_ctrl_scan_paths(r, c); > + nvme_ctrl_scan_namespaces(r, c); > return c; > } > > @@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n) > return n->sysfs_dir; > } > > +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head) > +{ > + return head->sysfs_dir; > +} > + > const char *nvme_ns_get_name(nvme_ns_t n) > { > return n->name; > @@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name) > > static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) > { > + int ret; > struct nvme_ns *n; > + struct nvme_ns_head *head; > + struct stat arg; > + _cleanup_free_ char *path = NULL; > > n = calloc(1, sizeof(*n)); > if (!n) { > @@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) > return NULL; > } > > + head = calloc(1, sizeof(*head)); > + if (!head) { > + errno = ENOMEM; > + free(n); > + return NULL; > + } > + > + head->n = n; > + list_head_init(&head->paths); > + ret = asprintf(&path, "%s/%s", sys_path, "multipath"); > + if (ret < 0) { > + errno = ENOMEM; > + goto free_ns_head; > + } > + /* > + * The sysfs-dir "multipath" is available only when nvme multipath > + * is configured and we're running kernel version >= 6.14. > + */ > + ret = stat(path, &arg); > + if (ret == 0) { > + head->sysfs_dir = path; > + path = NULL; > + } else > + head->sysfs_dir = NULL; > + > + n->head = head; > n->fd = -1; > n->name = strdup(name); > > @@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) > if (nvme_ns_init(sys_path, n) != 0) > goto free_ns; > > - list_head_init(&n->paths); > list_node_init(&n->entry); > > nvme_ns_release_fd(n); /* Do not leak fds */ > + > return n; > > free_ns: > free(n->generic_name); > free(n->name); > +free_ns_head: > + free(head); > free(n); > return NULL; > } > @@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name) > return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name); > } > > + > +static void nvme_ns_head_scan_path(nvme_subsystem_t s, nvme_ns_t n, char *name) > +{ > + nvme_ctrl_t c; > + nvme_path_t p; > + > + nvme_subsystem_for_each_ctrl(s, c) { > + nvme_ctrl_for_each_path(c, p) { > + if (!strcmp(nvme_path_get_name(p), name)) { > + list_add_tail(&n->head->paths, &p->nentry); > + p->n = n; > + return; > + } > + } > + } > +} > + > +static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) > +{ > + struct nvme_ns_head *head = n->head; > + > + if (nvme_ns_head_get_sysfs_dir(head)) { > + struct dirents paths = {}; > + int i; > + > + /* > + * When multipath is configured on kernel version >= 6.14, > + * we use multipath sysfs link to get each path of a namespace. > + */ > + paths.num = nvme_scan_ns_head_paths(head, &paths.ents); > + > + for (i = 0; i < paths.num; i++) > + nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name); > + } else { > + nvme_ctrl_t c; > + nvme_path_t p; > + int ns_ctrl, ns_nsid, ret; > + > + /* > + * If multipath is not configured or we're running on kernel > + * version < 6.14, fallback to the old way. > + */ > + ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", > + &ns_ctrl, &ns_nsid); > + if (ret != 2) > + return; > + > + nvme_subsystem_for_each_ctrl(s, c) { > + nvme_ctrl_for_each_path(c, p) { > + int p_subsys, p_ctrl, p_nsid; > + > + ret = sscanf(nvme_path_get_name(p), > + "nvme%dc%dn%d", > + &p_subsys, &p_ctrl, &p_nsid); > + if (ret != 3) > + continue; > + if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { > + list_add_tail(&head->paths, &p->nentry); > + p->n = n; > + } > + } > + } > + } Can't you just use the existing nvme_subsystem_set_path_ns() function here instead of deleting and open-code it? > +} > + > static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, > char *name) > { > @@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, > n->s = c->s; > n->c = c; > list_add_tail(&c->namespaces, &n->entry); > - return 0; > -} > - > -static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) > -{ > - nvme_ctrl_t c; > - nvme_path_t p; > - int ns_ctrl, ns_nsid, ret; > - > - ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid); > - if (ret != 2) > - return; > + nvme_subsystem_set_ns_path(c->s, n); > > - nvme_subsystem_for_each_ctrl(s, c) { > - nvme_ctrl_for_each_path(c, p) { > - int p_subsys, p_ctrl, p_nsid; > - > - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", > - &p_subsys, &p_ctrl, &p_nsid); > - if (ret != 3) > - continue; > - if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { > - list_add_tail(&n->paths, &p->nentry); > - p->n = n; > - } > - } > - } > + return 0; > } > > static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s, Similar here; better to use the existing functions. > @@ -2917,7 +2977,7 @@ static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s, > list_del_init(&p->nentry); > p->n = NULL; > } > - list_head_init(&_n->paths); > + list_head_init(&_n->head->paths); > __nvme_free_ns(_n); > } > n->s = s; > diff --git a/src/nvme/tree.h b/src/nvme/tree.h > index 25d4b31b..9f382e9c 100644 > --- a/src/nvme/tree.h > +++ b/src/nvme/tree.h > @@ -27,6 +27,7 @@ > */ > > typedef struct nvme_ns *nvme_ns_t; > +typedef struct nvme_ns_head *nvme_ns_head_t; > typedef struct nvme_path *nvme_path_t; > typedef struct nvme_ctrl *nvme_ctrl_t; > typedef struct nvme_subsystem *nvme_subsystem_t; > @@ -1091,6 +1092,14 @@ void nvme_ctrl_set_dhchap_host_key(nvme_ctrl_t c, const char *key); > */ > const char *nvme_ctrl_get_dhchap_key(nvme_ctrl_t c); > > +/** > + * nvme_ns_head_get_sysfs_dir() - sysfs dir of namespave head > + * @head: namespace head instance > + * > + * Returns: sysfs directory name of @head > + */ > +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head); > + > /** > * nvme_ctrl_set_dhchap_key() - Set controller key > * @c: Controller for which the key should be set Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link 2025-04-22 6:24 ` Hannes Reinecke @ 2025-04-22 14:01 ` Nilay Shroff 0 siblings, 0 replies; 14+ messages in thread From: Nilay Shroff @ 2025-04-22 14:01 UTC (permalink / raw) To: Hannes Reinecke, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/22/25 11:54 AM, Hannes Reinecke wrote: > On 4/17/25 15:59, Nilay Shroff wrote: >> With the upcoming Linux kernel v6.15, NVMe native multipath now provides >> a simplified mechanism for discovering all paths to a shared namespace >> through sysfs. >> >> A new "multipath" directory is created under each NVMe head namespace >> device in "/sys/block/<head>/multipath/". This directory contains symlinks >> to all namespace path devices that access the same shared namespace. >> >> For example, consider a shared namespace accessible via two paths under >> nvme-subsys1: >> >> nvme-subsys1 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 >> hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 >> \ >> +- ns 1 >> \ >> +- nvme0 pcie 052e:78:00.0 live optimized >> +- nvme1 pcie 058e:78:00.0 live optimized >> >> The head device `/dev/nvme1n1` will now have the following structure: >> >> /sys/block/nvme1n1/multipath/ >> ├── nvme1c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme1c0n1 >> └── nvme1c1n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme1/nvme1c1n1 >> >> This clearly shows that namespace 1 is accessible through both nvme1c0n1 >> and nvme1c1n1. This new sysfs structure significantly simplifies multipath >> discovery and management, making it easier for tools and scripts to enumerate >> and manage NVMe multipath configurations. So leverage this functionality to >> update the path links for a shared NVMe namespace, simplifying path discovery >> and management. >> >> This change adds a new struct nvme_ns_head to represent the head of a shared >> namespace. It contains a list head linking together struct nvme_path objects, >> where each path corresponds to a shared namespace instance. Additionally, >> struct nvme_ns has been updated to reference its associated nvme_ns_head, >> enabling straightforward traversal of all paths to a shared NVMe namespace. >> >> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> >> --- >> src/nvme/filters.c | 6 ++ >> src/nvme/filters.h | 9 +++ >> src/nvme/private.h | 9 ++- >> src/nvme/tree.c | 162 +++++++++++++++++++++++++++++++-------------- >> src/nvme/tree.h | 9 +++ >> 5 files changed, 143 insertions(+), 52 deletions(-) >> >> diff --git a/src/nvme/filters.c b/src/nvme/filters.c >> index ceaba68f..4a8829db 100644 >> --- a/src/nvme/filters.c >> +++ b/src/nvme/filters.c >> @@ -105,3 +105,9 @@ int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns) >> return scandir(nvme_ctrl_get_sysfs_dir(c), ns, >> nvme_namespace_filter, alphasort); >> } >> + >> +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths) >> +{ >> + return scandir(nvme_ns_head_get_sysfs_dir(head), paths, >> + nvme_paths_filter, alphasort); >> +} >> diff --git a/src/nvme/filters.h b/src/nvme/filters.h >> index 4ceeffd5..9e9dbb95 100644 >> --- a/src/nvme/filters.h >> +++ b/src/nvme/filters.h >> @@ -94,4 +94,13 @@ int nvme_scan_ctrl_namespace_paths(nvme_ctrl_t c, struct dirent ***paths); >> */ >> int nvme_scan_ctrl_namespaces(nvme_ctrl_t c, struct dirent ***ns); >> +/** >> + * nvme_scan_ns_head_paths() - Scan for namespace paths >> + * @head: Namespace head node to scan >> + * @paths : Pointer to array of dirents >> + * >> + * Return: number of entries in @ents >> + */ >> +int nvme_scan_ns_head_paths(nvme_ns_head_t head, struct dirent ***paths); >> + >> #endif /* _LIBNVME_FILTERS_H */ >> diff --git a/src/nvme/private.h b/src/nvme/private.h >> index 33cdd555..f45c5823 100644 >> --- a/src/nvme/private.h >> +++ b/src/nvme/private.h >> @@ -36,12 +36,19 @@ struct nvme_path { >> int grpid; >> }; >> +struct nvme_ns_head { >> + struct list_head paths; >> + struct nvme_ns *n; >> + >> + char *sysfs_dir; >> +}; >> + >> struct nvme_ns { >> struct list_node entry; >> - struct list_head paths; >> struct nvme_subsystem *s; >> struct nvme_ctrl *c; >> + struct nvme_ns_head *head; >> int fd; >> __u32 nsid; > > Why isn't 'nvme_subsystem' moved to 'nvme_ns_head'? > That certainly is how the kernel structures are laid > out, and it would make sense to follow it here... > Ideally yes it could be. However for libnvme, struct nvme_ns actually represents namespace head node when multipath is configured. And struct nvme_subsystem could be directly accessed from struct nvme_ns as we maintain a pointer to the nvme_subsystem from nvme_ns. For instance, take an example of a shared namespace attached to two controllers in a subsystem and so we've got following topology for this subsystem. # nvme show-topology nvme-subsys2 - NQN=nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057 hostnqn=nqn.2014-08.org.nvmexpress:uuid:41528538-e8ad-4eaf-84a7-9c552917d988 \ +- ns 1 \ +- nvme0 pcie 052e:78:00.0 live optimized +- nvme2 pcie 058e:78:00.0 live optimized And multipath sysfs link to reach the shared namespace from head node (nvme2n1): # tree /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/ /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath/ ├── nvme2c0n1 -> ../../../../../pci052e:78/052e:78:00.0/nvme/nvme0/nvme2c0n1 └── nvme2c2n1 -> ../../../../../pci058e:78/058e:78:00.0/nvme/nvme2/nvme2c2n1 If we parse above topology in libnvme then we get, struct nvme_subsystem::name => nvme-subsys2 struct nvme_subsystem::namespaces => <list-of-struct nvme_ns-under-this-subsys> //first path struct nvme_ns_path::name => nvme2c0n1 struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme0/nvme2c0n1/ //second path struct nvme_ns_path::name => nvme2c2n1 struct nvme_ns_path::sysfs_dir => /sys/class/nvme/nvme2/nvme2c2n1/ struct nvme_ns_head::paths => struct nvme_ns_path (first path), struct nvme_ns_path (second path) struct nvme_ns_head::sysfs_dir =>/sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/multipath struct nvme_ns::name => nvme2n1 struct nvme_ns::sysfs_dir => /sys/class/nvme-subsystem/nvme-subsys2/nvme2n1/ struct nvme_ns::head => struct nvme_ns_head struct nvme_ns::s => struct nvme_subsystem So as we could see above, struct nvme_ns::s points to nvme_susbsystem and struct nvme_ns actually represents the head node. >> diff --git a/src/nvme/tree.c b/src/nvme/tree.c >> index b0a4696f..bd7fb53e 100644 >> --- a/src/nvme/tree.c >> +++ b/src/nvme/tree.c >> @@ -564,12 +564,12 @@ nvme_ns_t nvme_subsystem_next_ns(nvme_subsystem_t s, nvme_ns_t n) >> nvme_path_t nvme_namespace_first_path(nvme_ns_t ns) >> { >> - return list_top(&ns->paths, struct nvme_path, nentry); >> + return list_top(&ns->head->paths, struct nvme_path, nentry); >> } >> nvme_path_t nvme_namespace_next_path(nvme_ns_t ns, nvme_path_t p) >> { >> - return p ? list_next(&ns->paths, p, nentry) : NULL; >> + return p ? list_next(&ns->head->paths, p, nentry) : NULL; >> } >> static void __nvme_free_ns(struct nvme_ns *n) >> @@ -579,6 +579,8 @@ static void __nvme_free_ns(struct nvme_ns *n) >> free(n->generic_name); >> free(n->name); >> free(n->sysfs_dir); >> + free(n->head->sysfs_dir); >> + free(n->head); >> free(n); >> } >> @@ -916,25 +918,6 @@ void nvme_free_path(struct nvme_path *p) >> free(p); >> } >> -static void nvme_subsystem_set_path_ns(nvme_subsystem_t s, nvme_path_t p) >> -{ >> - char n_name[32] = { }; >> - int i, c, nsid, ret; >> - nvme_ns_t n; >> - >> - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", &i, &c, &nsid); >> - if (ret != 3) >> - return; >> - >> - sprintf(n_name, "nvme%dn%d", i, nsid); >> - nvme_subsystem_for_each_ns(s, n) { >> - if (!strcmp(n_name, nvme_ns_get_name(n))) { >> - list_add_tail(&n->paths, &p->nentry); >> - p->n = n; >> - } >> - } >> -} >> - >> static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) >> { >> struct nvme_path *p; >> @@ -973,7 +956,6 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) >> } >> list_node_init(&p->nentry); >> - nvme_subsystem_set_path_ns(c->s, p); >> list_node_init(&p->entry); >> list_add_tail(&c->paths, &p->entry); >> return 0; >> @@ -2250,8 +2232,8 @@ nvme_ctrl_t nvme_scan_ctrl(nvme_root_t r, const char *name) >> return NULL; >> path = NULL; >> - nvme_ctrl_scan_namespaces(r, c); >> nvme_ctrl_scan_paths(r, c); >> + nvme_ctrl_scan_namespaces(r, c); >> return c; >> } >> @@ -2323,6 +2305,11 @@ const char *nvme_ns_get_sysfs_dir(nvme_ns_t n) >> return n->sysfs_dir; >> } >> +const char *nvme_ns_head_get_sysfs_dir(nvme_ns_head_t head) >> +{ >> + return head->sysfs_dir; >> +} >> + >> const char *nvme_ns_get_name(nvme_ns_t n) >> { >> return n->name; >> @@ -2749,7 +2736,11 @@ static void nvme_ns_set_generic_name(struct nvme_ns *n, const char *name) >> static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) >> { >> + int ret; >> struct nvme_ns *n; >> + struct nvme_ns_head *head; >> + struct stat arg; >> + _cleanup_free_ char *path = NULL; >> n = calloc(1, sizeof(*n)); >> if (!n) { >> @@ -2757,6 +2748,32 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) >> return NULL; >> } >> + head = calloc(1, sizeof(*head)); >> + if (!head) { >> + errno = ENOMEM; >> + free(n); >> + return NULL; >> + } >> + >> + head->n = n; >> + list_head_init(&head->paths); >> + ret = asprintf(&path, "%s/%s", sys_path, "multipath"); >> + if (ret < 0) { >> + errno = ENOMEM; >> + goto free_ns_head; >> + } >> + /* >> + * The sysfs-dir "multipath" is available only when nvme multipath >> + * is configured and we're running kernel version >= 6.14. >> + */ >> + ret = stat(path, &arg); >> + if (ret == 0) { >> + head->sysfs_dir = path; >> + path = NULL; >> + } else >> + head->sysfs_dir = NULL; >> + >> + n->head = head; >> n->fd = -1; >> n->name = strdup(name); >> @@ -2765,15 +2782,17 @@ static nvme_ns_t nvme_ns_open(const char *sys_path, const char *name) >> if (nvme_ns_init(sys_path, n) != 0) >> goto free_ns; >> - list_head_init(&n->paths); >> list_node_init(&n->entry); >> nvme_ns_release_fd(n); /* Do not leak fds */ >> + >> return n; >> free_ns: >> free(n->generic_name); >> free(n->name); >> +free_ns_head: >> + free(head); >> free(n); >> return NULL; >> } >> @@ -2836,6 +2855,71 @@ nvme_ns_t nvme_scan_namespace(const char *name) >> return __nvme_scan_namespace(nvme_ns_sysfs_dir(), name); >> } >> + >> +static void nvme_ns_head_scan_path(nvme_subsystem_t s, nvme_ns_t n, char *name) >> +{ >> + nvme_ctrl_t c; >> + nvme_path_t p; >> + >> + nvme_subsystem_for_each_ctrl(s, c) { >> + nvme_ctrl_for_each_path(c, p) { >> + if (!strcmp(nvme_path_get_name(p), name)) { >> + list_add_tail(&n->head->paths, &p->nentry); >> + p->n = n; >> + return; >> + } >> + } >> + } >> +} >> + >> +static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) >> +{ >> + struct nvme_ns_head *head = n->head; >> + >> + if (nvme_ns_head_get_sysfs_dir(head)) { >> + struct dirents paths = {}; >> + int i; >> + >> + /* >> + * When multipath is configured on kernel version >= 6.14, >> + * we use multipath sysfs link to get each path of a namespace. >> + */ >> + paths.num = nvme_scan_ns_head_paths(head, &paths.ents); >> + >> + for (i = 0; i < paths.num; i++) >> + nvme_ns_head_scan_path(s, n, paths.ents[i]->d_name); >> + } else { >> + nvme_ctrl_t c; >> + nvme_path_t p; >> + int ns_ctrl, ns_nsid, ret; >> + >> + /* >> + * If multipath is not configured or we're running on kernel >> + * version < 6.14, fallback to the old way. >> + */ >> + ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", >> + &ns_ctrl, &ns_nsid); >> + if (ret != 2) >> + return; >> + >> + nvme_subsystem_for_each_ctrl(s, c) { >> + nvme_ctrl_for_each_path(c, p) { >> + int p_subsys, p_ctrl, p_nsid; >> + >> + ret = sscanf(nvme_path_get_name(p), >> + "nvme%dc%dn%d", >> + &p_subsys, &p_ctrl, &p_nsid); >> + if (ret != 3) >> + continue; >> + if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { >> + list_add_tail(&head->paths, &p->nentry); >> + p->n = n; >> + } >> + } >> + } >> + } > > Can't you just use the existing nvme_subsystem_set_path_ns() function > here instead of deleting and open-code it? > We can but then we don't levearge the newly added (efficient and probably more elegant) way of discovering the multipath information from the head node. So nvme_subsystem_set_ns_path is implemented such that if libnvme is running on the newer kernel supporting multipath sysfs link then we leverage that to discover multipath information. However if libnvme is running on older kernel which doesn't support multipath sysfs link then nvme_subsystem_set_ns_path falls back to the legacy way of discoveing multipath information. >> +} >> + >> static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, >> char *name) >> { >> @@ -2861,33 +2945,9 @@ static int nvme_ctrl_scan_namespace(nvme_root_t r, struct nvme_ctrl *c, >> n->s = c->s; >> n->c = c; >> list_add_tail(&c->namespaces, &n->entry); >> - return 0; >> -} >> - >> -static void nvme_subsystem_set_ns_path(nvme_subsystem_t s, nvme_ns_t n) >> -{ >> - nvme_ctrl_t c; >> - nvme_path_t p; >> - int ns_ctrl, ns_nsid, ret; >> - >> - ret = sscanf(nvme_ns_get_name(n), "nvme%dn%d", &ns_ctrl, &ns_nsid); >> - if (ret != 2) >> - return; >> + nvme_subsystem_set_ns_path(c->s, n); >> - nvme_subsystem_for_each_ctrl(s, c) { >> - nvme_ctrl_for_each_path(c, p) { >> - int p_subsys, p_ctrl, p_nsid; >> - >> - ret = sscanf(nvme_path_get_name(p), "nvme%dc%dn%d", >> - &p_subsys, &p_ctrl, &p_nsid); >> - if (ret != 3) >> - continue; >> - if (ns_ctrl == p_subsys && ns_nsid == p_nsid) { >> - list_add_tail(&n->paths, &p->nentry); >> - p->n = n; >> - } >> - } >> - } >> + return 0; >> } >> static int nvme_subsystem_scan_namespace(nvme_root_t r, nvme_subsystem_t s, > > Similar here; better to use the existing functions. In this patch I re-wrote nvme_subsystem_set_ns_path which is now called either while scanning controler or subsystem. The idea is to use a common function which helps discover the multipath information. And as I stated before, on newer kernel this function uses the multiapth sysfs link to discover multiple paths however on older kernels it falls back to legacy way to discover mulipath infromation. Thanks, --Nilay ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object 2025-04-17 13:59 [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link Nilay Shroff @ 2025-04-17 13:59 ` Nilay Shroff 2025-04-22 6:26 ` Hannes Reinecke 2025-04-17 13:59 ` [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe " Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 4/4] test: extend sysfs tree dump test Nilay Shroff 3 siblings, 1 reply; 14+ messages in thread From: Nilay Shroff @ 2025-04-17 13:59 UTC (permalink / raw) To: linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce Add a new attribute named "queue_depth" under the NVMe path object. This attribute is used by the iopolicy "queue-depth", which was introduced in kernel v6.11. However, the corresponding sysfs attribute for queue depth was only added in kernel v6.14. The queue_depth value can be useful for observing which paths are selected for I/O forwarding, based on the depth of each path. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli. Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> --- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 12 +++++++++++- src/nvme/tree.h | 8 ++++++++ 4 files changed, 21 insertions(+), 1 deletion(-) diff --git a/src/libnvme.map b/src/libnvme.map index 4314705f..e53fad6b 100644 --- a/src/libnvme.map +++ b/src/libnvme.map @@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; nvme_read_config; diff --git a/src/nvme/private.h b/src/nvme/private.h index f45c5823..f94276e2 100644 --- a/src/nvme/private.h +++ b/src/nvme/private.h @@ -34,6 +34,7 @@ struct nvme_path { char *sysfs_dir; char *ana_state; int grpid; + int queue_depth; }; struct nvme_ns_head { diff --git a/src/nvme/tree.c b/src/nvme/tree.c index bd7fb53e..b7a38a07 100644 --- a/src/nvme/tree.c +++ b/src/nvme/tree.c @@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p) return p->name; } +int nvme_path_get_queue_depth(nvme_path_t p) +{ + return p->queue_depth; +} + const char *nvme_path_get_ana_state(nvme_path_t p) { return p->ana_state; @@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p) static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) { struct nvme_path *p; - _cleanup_free_ char *path = NULL, *grpid = NULL; + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL; int ret; nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n", @@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) sscanf(grpid, "%d", &p->grpid); } + queue_depth = nvme_get_path_attr(p, "queue_depth"); + if (queue_depth) { + sscanf(queue_depth, "%d", &p->queue_depth); + } + list_node_init(&p->nentry); list_node_init(&p->entry); list_add_tail(&c->paths, &p->entry); diff --git a/src/nvme/tree.h b/src/nvme/tree.h index 9f382e9c..a9082f8e 100644 --- a/src/nvme/tree.h +++ b/src/nvme/tree.h @@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p); */ const char *nvme_path_get_ana_state(nvme_path_t p); +/** + * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object + * @p: &nvme_path_t object + * + * Return: Queue depth of @p + */ +int nvme_path_get_queue_depth(nvme_path_t p); + /** * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object * @p: &nvme_path_t object -- 2.49.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object 2025-04-17 13:59 ` [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object Nilay Shroff @ 2025-04-22 6:26 ` Hannes Reinecke 2025-04-22 14:32 ` Nilay Shroff 0 siblings, 1 reply; 14+ messages in thread From: Hannes Reinecke @ 2025-04-22 6:26 UTC (permalink / raw) To: Nilay Shroff, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/17/25 15:59, Nilay Shroff wrote: > Add a new attribute named "queue_depth" under the NVMe path object. This > attribute is used by the iopolicy "queue-depth", which was introduced in > kernel v6.11. However, the corresponding sysfs attribute for queue depth > was only added in kernel v6.14. > > The queue_depth value can be useful for observing which paths are selected > for I/O forwarding, based on the depth of each path. To support this, > export the attribute in libnvme.map so it can be accessed via nvme-cli. > > Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> > --- > src/libnvme.map | 1 + > src/nvme/private.h | 1 + > src/nvme/tree.c | 12 +++++++++++- > src/nvme/tree.h | 8 ++++++++ > 4 files changed, 21 insertions(+), 1 deletion(-) > > diff --git a/src/libnvme.map b/src/libnvme.map > index 4314705f..e53fad6b 100644 > --- a/src/libnvme.map > +++ b/src/libnvme.map > @@ -317,6 +317,7 @@ LIBNVME_1_0 { > nvme_path_get_ctrl; > nvme_path_get_name; > nvme_path_get_ns; > + nvme_path_get_queue_depth; > nvme_path_get_sysfs_dir; > nvme_paths_filter; > nvme_read_config; > diff --git a/src/nvme/private.h b/src/nvme/private.h > index f45c5823..f94276e2 100644 > --- a/src/nvme/private.h > +++ b/src/nvme/private.h > @@ -34,6 +34,7 @@ struct nvme_path { > char *sysfs_dir; > char *ana_state; > int grpid; > + int queue_depth; > }; > > struct nvme_ns_head { > diff --git a/src/nvme/tree.c b/src/nvme/tree.c > index bd7fb53e..b7a38a07 100644 > --- a/src/nvme/tree.c > +++ b/src/nvme/tree.c > @@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p) > return p->name; > } > > +int nvme_path_get_queue_depth(nvme_path_t p) > +{ > + return p->queue_depth; > +} > + > const char *nvme_path_get_ana_state(nvme_path_t p) > { > return p->ana_state; > @@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p) > static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) > { > struct nvme_path *p; > - _cleanup_free_ char *path = NULL, *grpid = NULL; > + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL; > int ret; > > nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n", > @@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) > sscanf(grpid, "%d", &p->grpid); > } > > + queue_depth = nvme_get_path_attr(p, "queue_depth"); > + if (queue_depth) { > + sscanf(queue_depth, "%d", &p->queue_depth); > + } > + > list_node_init(&p->nentry); > list_node_init(&p->entry); > list_add_tail(&c->paths, &p->entry); > diff --git a/src/nvme/tree.h b/src/nvme/tree.h > index 9f382e9c..a9082f8e 100644 > --- a/src/nvme/tree.h > +++ b/src/nvme/tree.h > @@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p); > */ > const char *nvme_path_get_ana_state(nvme_path_t p); > > +/** > + * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object > + * @p: &nvme_path_t object > + * > + * Return: Queue depth of @p > + */ > +int nvme_path_get_queue_depth(nvme_path_t p); > + > /** > * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object > * @p: &nvme_path_t object As discussed: saving the 'queue_depth' attribute is a bad idea, as it changes frequently. I'd rather read the attribute value from sysfs whenever 'nvme_path_get_queue_depth' is called, and use the tree structure only to hold the pathname. Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object 2025-04-22 6:26 ` Hannes Reinecke @ 2025-04-22 14:32 ` Nilay Shroff 2025-04-22 17:23 ` Daniel Wagner 0 siblings, 1 reply; 14+ messages in thread From: Nilay Shroff @ 2025-04-22 14:32 UTC (permalink / raw) To: Hannes Reinecke, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/22/25 11:56 AM, Hannes Reinecke wrote: > On 4/17/25 15:59, Nilay Shroff wrote: >> Add a new attribute named "queue_depth" under the NVMe path object. This >> attribute is used by the iopolicy "queue-depth", which was introduced in >> kernel v6.11. However, the corresponding sysfs attribute for queue depth >> was only added in kernel v6.14. >> >> The queue_depth value can be useful for observing which paths are selected >> for I/O forwarding, based on the depth of each path. To support this, >> export the attribute in libnvme.map so it can be accessed via nvme-cli. >> >> Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> >> --- >> src/libnvme.map | 1 + >> src/nvme/private.h | 1 + >> src/nvme/tree.c | 12 +++++++++++- >> src/nvme/tree.h | 8 ++++++++ >> 4 files changed, 21 insertions(+), 1 deletion(-) >> >> diff --git a/src/libnvme.map b/src/libnvme.map >> index 4314705f..e53fad6b 100644 >> --- a/src/libnvme.map >> +++ b/src/libnvme.map >> @@ -317,6 +317,7 @@ LIBNVME_1_0 { >> nvme_path_get_ctrl; >> nvme_path_get_name; >> nvme_path_get_ns; >> + nvme_path_get_queue_depth; >> nvme_path_get_sysfs_dir; >> nvme_paths_filter; >> nvme_read_config; >> diff --git a/src/nvme/private.h b/src/nvme/private.h >> index f45c5823..f94276e2 100644 >> --- a/src/nvme/private.h >> +++ b/src/nvme/private.h >> @@ -34,6 +34,7 @@ struct nvme_path { >> char *sysfs_dir; >> char *ana_state; >> int grpid; >> + int queue_depth; >> }; >> struct nvme_ns_head { >> diff --git a/src/nvme/tree.c b/src/nvme/tree.c >> index bd7fb53e..b7a38a07 100644 >> --- a/src/nvme/tree.c >> +++ b/src/nvme/tree.c >> @@ -903,6 +903,11 @@ const char *nvme_path_get_name(nvme_path_t p) >> return p->name; >> } >> +int nvme_path_get_queue_depth(nvme_path_t p) >> +{ >> + return p->queue_depth; >> +} >> + >> const char *nvme_path_get_ana_state(nvme_path_t p) >> { >> return p->ana_state; >> @@ -921,7 +926,7 @@ void nvme_free_path(struct nvme_path *p) >> static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) >> { >> struct nvme_path *p; >> - _cleanup_free_ char *path = NULL, *grpid = NULL; >> + _cleanup_free_ char *path = NULL, *grpid = NULL, *queue_depth = NULL; >> int ret; >> nvme_msg(r, LOG_DEBUG, "scan controller %s path %s\n", >> @@ -955,6 +960,11 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) >> sscanf(grpid, "%d", &p->grpid); >> } >> + queue_depth = nvme_get_path_attr(p, "queue_depth"); >> + if (queue_depth) { >> + sscanf(queue_depth, "%d", &p->queue_depth); >> + } >> + >> list_node_init(&p->nentry); >> list_node_init(&p->entry); >> list_add_tail(&c->paths, &p->entry); >> diff --git a/src/nvme/tree.h b/src/nvme/tree.h >> index 9f382e9c..a9082f8e 100644 >> --- a/src/nvme/tree.h >> +++ b/src/nvme/tree.h >> @@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p); >> */ >> const char *nvme_path_get_ana_state(nvme_path_t p); >> +/** >> + * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object >> + * @p: &nvme_path_t object >> + * >> + * Return: Queue depth of @p >> + */ >> +int nvme_path_get_queue_depth(nvme_path_t p); >> + >> /** >> * nvme_path_get_ctrl() - Parent controller of an nvme_path_t object >> * @p: &nvme_path_t object > > As discussed: saving the 'queue_depth' attribute is a bad idea, as it > changes frequently. > I'd rather read the attribute value from sysfs whenever > 'nvme_path_get_queue_depth' is called, and use the tree structure only > to hold the pathname. > Yes correct, but I thought we discussed about implementing non-cached version of API in libnvme 2.x. So I implemented cached version of queue_depth API. Daniel can you confirm? Thanks, --Nilay ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object 2025-04-22 14:32 ` Nilay Shroff @ 2025-04-22 17:23 ` Daniel Wagner 2025-04-23 6:13 ` Nilay Shroff 0 siblings, 1 reply; 14+ messages in thread From: Daniel Wagner @ 2025-04-22 17:23 UTC (permalink / raw) To: Nilay Shroff; +Cc: Hannes Reinecke, linux-nvme, hare, kbusch, gjoyce On Tue, Apr 22, 2025 at 08:02:49PM +0530, Nilay Shroff wrote: > > As discussed: saving the 'queue_depth' attribute is a bad idea, as it > > changes frequently. > > I'd rather read the attribute value from sysfs whenever > > 'nvme_path_get_queue_depth' is called, and use the tree structure only > > to hold the pathname. > > > Yes correct, but I thought we discussed about implementing non-cached > version of API in libnvme 2.x. So I implemented cached version of > queue_depth API. Daniel can you confirm? I was under the impression that we agreed on the default being non-cached. As it doesn't really make sense to have a cached anyway. ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object 2025-04-22 17:23 ` Daniel Wagner @ 2025-04-23 6:13 ` Nilay Shroff 0 siblings, 0 replies; 14+ messages in thread From: Nilay Shroff @ 2025-04-23 6:13 UTC (permalink / raw) To: Daniel Wagner; +Cc: Hannes Reinecke, linux-nvme, hare, kbusch, gjoyce On 4/22/25 10:53 PM, Daniel Wagner wrote: > On Tue, Apr 22, 2025 at 08:02:49PM +0530, Nilay Shroff wrote: >>> As discussed: saving the 'queue_depth' attribute is a bad idea, as it >>> changes frequently. >>> I'd rather read the attribute value from sysfs whenever >>> 'nvme_path_get_queue_depth' is called, and use the tree structure only >>> to hold the pathname. >>> >> Yes correct, but I thought we discussed about implementing non-cached >> version of API in libnvme 2.x. So I implemented cached version of >> queue_depth API. Daniel can you confirm? > > I was under the impression that we agreed on the default being > non-cached. As it doesn't really make sense to have a cached anyway. Okay, my bad.. probably some misunderstanding on my part. I will fix this in the next version of this patch. Thanks, --Nilay ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe path object 2025-04-17 13:59 [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object Nilay Shroff @ 2025-04-17 13:59 ` Nilay Shroff 2025-04-22 6:27 ` Hannes Reinecke 2025-04-17 13:59 ` [PATCHv2 4/4] test: extend sysfs tree dump test Nilay Shroff 3 siblings, 1 reply; 14+ messages in thread From: Nilay Shroff @ 2025-04-17 13:59 UTC (permalink / raw) To: linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce Add a new attribute named "numa_nodes" under the NVMe path object. This attribute is used by the iopolicy "numa". The numa_nodes value is stored for each NVMe path and represents the NUMA node(s) associated with it. When the iopolicy is set to "numa", I/O traffic originating from a given NUMA node will be forwarded through the corresponding NVMe path. The numa_nodes attribute is useful for observing which NVMe path the kernel would choose for I/O forwarding based on NUMA affinity. To support this, export the attribute in libnvme.map so it can be accessed via nvme-cli. Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> --- src/libnvme.map | 1 + src/nvme/private.h | 1 + src/nvme/tree.c | 10 ++++++++++ src/nvme/tree.h | 8 ++++++++ 4 files changed, 20 insertions(+) diff --git a/src/libnvme.map b/src/libnvme.map index e53fad6b..ffadc647 100644 --- a/src/libnvme.map +++ b/src/libnvme.map @@ -317,6 +317,7 @@ LIBNVME_1_0 { nvme_path_get_ctrl; nvme_path_get_name; nvme_path_get_ns; + nvme_path_get_numa_nodes; nvme_path_get_queue_depth; nvme_path_get_sysfs_dir; nvme_paths_filter; diff --git a/src/nvme/private.h b/src/nvme/private.h index f94276e2..5bfa027b 100644 --- a/src/nvme/private.h +++ b/src/nvme/private.h @@ -33,6 +33,7 @@ struct nvme_path { char *name; char *sysfs_dir; char *ana_state; + char *numa_nodes; int grpid; int queue_depth; }; diff --git a/src/nvme/tree.c b/src/nvme/tree.c index b7a38a07..ddb6fd70 100644 --- a/src/nvme/tree.c +++ b/src/nvme/tree.c @@ -913,6 +913,11 @@ const char *nvme_path_get_ana_state(nvme_path_t p) return p->ana_state; } +const char *nvme_path_get_numa_nodes(nvme_path_t p) +{ + return p->numa_nodes; +} + void nvme_free_path(struct nvme_path *p) { list_del_init(&p->entry); @@ -920,6 +925,7 @@ void nvme_free_path(struct nvme_path *p) free(p->name); free(p->sysfs_dir); free(p->ana_state); + free(p->numa_nodes); free(p); } @@ -955,6 +961,10 @@ static int nvme_ctrl_scan_path(nvme_root_t r, struct nvme_ctrl *c, char *name) if (!p->ana_state) p->ana_state = strdup("optimized"); + p->numa_nodes = nvme_get_path_attr(p, "numa_nodes"); + if (!p->numa_nodes) + p->numa_nodes = strdup("-1"); + grpid = nvme_get_path_attr(p, "ana_grpid"); if (grpid) { sscanf(grpid, "%d", &p->grpid); diff --git a/src/nvme/tree.h b/src/nvme/tree.h index a9082f8e..f6116c2b 100644 --- a/src/nvme/tree.h +++ b/src/nvme/tree.h @@ -867,6 +867,14 @@ const char *nvme_path_get_sysfs_dir(nvme_path_t p); */ const char *nvme_path_get_ana_state(nvme_path_t p); +/** + * nvme_path_get_numa_nodes() - NUMA nodes of an nvme_path_t object + * @p : &nvme_path_t object + * + * Return: NUMA nodes associated to @p + */ +const char *nvme_path_get_numa_nodes(nvme_path_t p); + /** * nvme_path_get_queue_depth() - Queue depth of an nvme_path_t object * @p: &nvme_path_t object -- 2.49.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe path object 2025-04-17 13:59 ` [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe " Nilay Shroff @ 2025-04-22 6:27 ` Hannes Reinecke 0 siblings, 0 replies; 14+ messages in thread From: Hannes Reinecke @ 2025-04-22 6:27 UTC (permalink / raw) To: Nilay Shroff, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/17/25 15:59, Nilay Shroff wrote: > Add a new attribute named "numa_nodes" under the NVMe path object. This > attribute is used by the iopolicy "numa". The numa_nodes value is stored > for each NVMe path and represents the NUMA node(s) associated with it. > When the iopolicy is set to "numa", I/O traffic originating from a given > NUMA node will be forwarded through the corresponding NVMe path. > > The numa_nodes attribute is useful for observing which NVMe path the > kernel would choose for I/O forwarding based on NUMA affinity. To support > this, export the attribute in libnvme.map so it can be accessed via > nvme-cli. > > Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> > --- > src/libnvme.map | 1 + > src/nvme/private.h | 1 + > src/nvme/tree.c | 10 ++++++++++ > src/nvme/tree.h | 8 ++++++++ > 4 files changed, 20 insertions(+) > Reviewed-by: Hannes Reinecke <hare@suse.de> Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCHv2 4/4] test: extend sysfs tree dump test 2025-04-17 13:59 [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns Nilay Shroff ` (2 preceding siblings ...) 2025-04-17 13:59 ` [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe " Nilay Shroff @ 2025-04-17 13:59 ` Nilay Shroff 2025-04-22 6:28 ` Hannes Reinecke 3 siblings, 1 reply; 14+ messages in thread From: Nilay Shroff @ 2025-04-17 13:59 UTC (permalink / raw) To: linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce In order to validate multipath link support extend the existing sysfs tree dump test. The updated tree-pcie.tar.xz archive includes a sysfs dump with two NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk with a private namespace that does not support multipath. And another nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace attached to multiple controllers. The json tree dump is also updated to include NVMe namespace path information. Since the archive content has changed, the expected output file tree-pcie.out is also updated to reflect the new structure. Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> --- src/nvme/json.c | 90 +++++++++++++++++++++++++++---- test/sysfs/data/tree-pcie.out | 53 ++++++++++++++---- test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes 3 files changed, 124 insertions(+), 19 deletions(-) diff --git a/src/nvme/json.c b/src/nvme/json.c index af28bd48..a2fe8173 100644 --- a/src/nvme/json.c +++ b/src/nvme/json.c @@ -533,25 +533,97 @@ static void json_dump_ctrl(struct json_object *ctrl_array, nvme_ctrl_t c) json_object_array_add(ctrl_array, ctrl_obj); } +static unsigned int json_dump_subsys_multipath(nvme_subsystem_t s, + struct json_object *ns_array) +{ + nvme_ns_t n; + nvme_path_t p; + unsigned int i = 0; + + nvme_subsystem_for_each_ns(s, n) { + struct json_object *ns_obj; + struct json_object *path_array; + + ns_obj = json_object_new_object(); + json_object_object_add(ns_obj, "nsid", + json_object_new_int(nvme_ns_get_nsid(n))); + json_object_object_add(ns_obj, "name", + json_object_new_string(nvme_ns_get_name(n))); + + path_array = json_object_new_array(); + nvme_namespace_for_each_path(n, p) { + struct json_object *path_obj; + struct json_object *ctrl_array; + nvme_ctrl_t c; + + path_obj = json_object_new_object(); + json_object_object_add(path_obj, "path", + json_object_new_string(nvme_path_get_name(p))); + json_object_object_add(path_obj, "ANAState", + json_object_new_string(nvme_path_get_ana_state(p))); + json_object_object_add(path_obj, "NUMANodes", + json_object_new_string(nvme_path_get_numa_nodes(p))); + json_object_object_add(path_obj, "qdepth", + json_object_new_int(nvme_path_get_queue_depth(p))); + + c = nvme_path_get_ctrl(p); + ctrl_array = json_object_new_array(); + json_dump_ctrl(ctrl_array, c); + json_object_object_add(path_obj, "controller", ctrl_array); + json_object_array_add(path_array, path_obj); + } + json_object_object_add(ns_obj, "paths", path_array); + json_object_array_add(ns_array, ns_obj); + i++; + } + return i; +} + +static void json_dump_subsys_non_multipath(nvme_subsystem_t s, + struct json_object *ns_array) +{ + nvme_ctrl_t c; + nvme_ns_t n; + + nvme_subsystem_for_each_ctrl(s, c) { + nvme_ctrl_for_each_ns(c, n) { + struct json_object *ctrl_array; + struct json_object *ns_obj; + + ns_obj = json_object_new_object(); + json_object_object_add(ns_obj, "nsid", + json_object_new_int(nvme_ns_get_nsid(n))); + json_object_object_add(ns_obj, "name", + json_object_new_string(nvme_ns_get_name(n))); + + ctrl_array = json_object_new_array(); + json_dump_ctrl(ctrl_array, c); + json_object_object_add(ns_obj, "controller", ctrl_array); + + json_object_array_add(ns_array, ns_obj); + } + } +} + static void json_dump_subsys(struct json_object *subsys_array, nvme_subsystem_t s) { - nvme_ctrl_t c; struct json_object *subsys_obj = json_object_new_object(); - struct json_object *ctrl_array; + struct json_object *ns_array; json_object_object_add(subsys_obj, "name", json_object_new_string(nvme_subsystem_get_name(s))); json_object_object_add(subsys_obj, "nqn", json_object_new_string(nvme_subsystem_get_nqn(s))); - ctrl_array = json_object_new_array(); - nvme_subsystem_for_each_ctrl(s, c) { - json_dump_ctrl(ctrl_array, c); - } - if (json_object_array_length(ctrl_array)) - json_object_object_add(subsys_obj, "controllers", ctrl_array); + + ns_array = json_object_new_array(); + if (!json_dump_subsys_multipath(s, ns_array)) + json_dump_subsys_non_multipath(s, ns_array); + + if (json_object_array_length(ns_array)) + json_object_object_add(subsys_obj, "namespaces", ns_array); else - json_object_put(ctrl_array); + json_object_put(ns_array); json_object_array_add(subsys_array, subsys_obj); } diff --git a/test/sysfs/data/tree-pcie.out b/test/sysfs/data/tree-pcie.out index 4a755a77..8ea175a6 100644 --- a/test/sysfs/data/tree-pcie.out +++ b/test/sysfs/data/tree-pcie.out @@ -6,23 +6,56 @@ "subsystems":[ { "name":"nvme-subsys0", - "nqn":"nqn.2019-08.org.qemu:subsys1", - "controllers":[ + "nqn":"nqn.1994-11.com.samsung:nvme:PM1743:2.5-inch:S7DFNG0W700063", + "namespaces":[ { - "name":"nvme0", - "transport":"pcie", - "traddr":"0000:0f:00.0" + "nsid":1, + "name":"nvme0n1", + "controller":[ + { + "name":"nvme0", + "transport":"pcie", + "traddr":"0214:90:00.0" + } + ] } ] }, { "name":"nvme-subsys1", - "nqn":"nqn.2019-08.org.qemu:nvme-0", - "controllers":[ + "nqn":"nqn.1994-11.com.samsung:nvme:PM1735a:2.5-inch:S6RTNE0R900057", + "namespaces":[ { - "name":"nvme1", - "transport":"pcie", - "traddr":"0000:00:05.0" + "nsid":1, + "name":"nvme1n1", + "paths":[ + { + "path":"nvme1c1n1", + "ANAState":"optimized", + "NUMANodes":"0-1", + "qdepth":0, + "controller":[ + { + "name":"nvme1", + "transport":"pcie", + "traddr":"052e:78:00.0" + } + ] + }, + { + "path":"nvme1c2n1", + "ANAState":"optimized", + "NUMANodes":"2-3", + "qdepth":0, + "controller":[ + { + "name":"nvme2", + "transport":"pcie", + "traddr":"058e:78:00.0" + } + ] + } + ] } ] } diff --git a/test/sysfs/data/tree-pcie.tar.xz b/test/sysfs/data/tree-pcie.tar.xz index ee11fdeb0108589b954eb03543665e3a10df0e95..d99d2b943248cc2d7586c084ac639b613524cd2f 100644 GIT binary patch literal 12656 zcmV-$F^|suH+ooF000E$*0e?f03iVu0001VFXf~3@Bc9_T>uvgyc~T2mB1+!;Ktfk zvp367fQ?%b7>$8khtFPs4L!X3TgA;(zSjWzmGd%k$kV%T%)YmZ;ZD10i<1%jhws)n z8)2bXVPN%Bqx6+i*h6oKI1k@YDp>3zofS;&cx%6pb@emFJ3f9&!_b2w;pwi|E4$gn zx{h3Ol7wg9Y)_KJ+rj905(XLB^I}qD-s`5Y?(ah!E6j+h074=#LaLJVbu^CZzz^;a z!!f=jw#VH|=EalEVQlVH;rP;v<5f=&=n$xP(5oaH6pj)6fi-d7i2B8@uwb_x7JCEh z4c*7TN$_b`Me3E8x)888-6XFuF_yod>;ft0ZXl|Tl`790QK>mI>}y&)pxOOIQ479% z6=TvQ^wDZ3)ie&7nkKkLPKTQ>O}1I7&{0)}-be-9f#=KXhxiFnns715ZZ5ImV><g5 zQ%OttO290t@3=kOf)?~9*2Jra?8yI@*PVNrEH(oWQCb<wt|{kXhS5^C;#exdVX~@; zz8KGmLP1rjzZSTD`|wn;9!XRE?^_e=(Pogz`1yQ8QEt7+(4STCV<~(m-5<K38DHvC z$9`CMs(J2qI&A=$IG+gfaSJ|lMjt{30;T^(?(laB^jPnmk$o!jAL_(;Ah{7b2rM2a zx%#-Zz<&Hgt`6n&M^6^{Pkzhjn~ZcX87B>GZ#zWe*GJp+ju-}E<BAhl;7mw}IzsvN zw`G|_B1*t(cV2MD1aTgg)wQBB!xz(X({DwF$$|RbWJ=4PMk`|k4!CwhIDs6&+Np*a z5hc!uQr4+H=2(fiBX*BOCKtas@un*myln7>+OdZ8xLzBChBl~tHWG+fW=Z&SV%B42 zoHz3yGa_Q))mzih+cC}GJR-?5Ix%d2@9WCSA7;YS|DUA#Khrbm3FeZY$!u-(9A8`I zM+@6AzqSg#Q8o#9KU#-7HZ{QLZyt~(N;ZTW=3W{EdWgLQyYX-}A@PU66VZ`d1blrX z(~dBoEk!$$YO${=&2FgJ+x5X~>1P5sKUaiO6vtiY)E}nK`QDW@yPEETwac^IlICnu zJ3Ryf@Hwdud3>eB(>`KdLr%~(WWloruD)Ao(a5(Yb<HdgK2enW@8zYm?q^8-tN}#K zS&PiRM=5iSUu(>cP6-B5(@Dh}kaD48QnJk1AFB@z_eQIbold0jyIJur`CQHI(!awY zi7B}a`*?D_hS6eX&L#|_H4YlU{Jb~?Q&U%s!;7gr^Ae@k5>?0y_3}sL3S#+%<BG!7 zL%=xTTl_}Pv=<#c?V&8+Fs4(MoPypP51=OxPPMgc#K>dQ&YG3PqiAX&FQjy6tiFT# zr+Q`1NqiM&1Ur44A2v^jJ&!Y%nz@)8Eil{s)eo><Pv34i0iOe4z*b_hZ`}+pe&S?| z9@OCuP5AWQuHfl97V_EfXqhIho!38PxDn?c^!nE|eNsnb_tLlg)sUBu8U&hC`Uotr zQ0u!W3JDyhx$;kAe;KJYj^|B%$ah`J%5ZjM(~Hb%XOM4U<K|X*$wV@~y!HX;dH@w( zr9&(NPg+l5a8)kO<hpg~)aRBYry3jvF?jAdXILMRdNi%8z+1nTL@brxNVV}<7JH92 zc4pYwPy+<Jp#$%SDh{H3a*_eDtLZ4~WPG%eKuqYJAh)k7gDkg6Y`2=iK1X0<0i)rV zvE?aEx&(>ECLbqg=6jSF(;Q$XV#a;KJ-`2qTv+XlABTh}mUlsFGu1uedYgXvfjjCh zL(m2{O82ec4cOfA0`?Bi?i}sB(keO+%q47<+JuFEE1}yO)QN$IGye>v$ffSzTb|3O zE=($wkOFgp*WjeqMN}JqIf?w~IAq_es8mq|Dq*m2pa_0X`5sqD_qXV1F-L)I`Cq$o zqxa%YWFF!n{Y>U5?BM74=y{(^jJymUc;!;ywV#GI;y`~Zx5IeZlq<gq<wK=CA1hw* z_ARV7XLMn*T_4m~e7mYuID)@rv3&5*cEJs1KkUxpY6J`WS>ZbJJ|aPzc?-?r1zlDq z?XOjS>(2BsK#P3nl%076yb{}KbM3%yG(>+{b2_RbO{ZGiiW%a+)WBj6Y46$eMIrrc zKeRwH{4|STT{b1iJqh70mHducfh%_6^Cn{)vf-ckI9z}7osa;LVT}-4<~&DR9l%-V zdQeH7D>4mr(Twe@=IYTtnBT%-iYjr{_nsU`ss^dzS7+D;KBE<zudHBc8vFkjlSy1G zoaW$lq7Y<yV;rxFJGa<ippC1<N)}E?+mZx|8%OlE1lqsOgu1mlc^`#NHZ}l!0=9MY z%FmZb$z2Nghhpu6i}_RUr`kOh4-U*cP)2jfAahIzdcD3r7BCL&0*bQ3kh9DjTu$}b zz=0k=8Fly7Q6JVf)g&8WBCG~d(YjZY1ZOywAx`5cWe#Zky5!jN5Nq*MSU*t_1JVXf z`qx`4E3pWnBVl_0fd32B&#Me#z_Oj}2ep!KYgJy8<N}4LAl%}ksugOA@g+q#DM1!@ zTA`o2taDpX5q*_930EtA<AHf)XYlFtIekYK6=SM&-#901(mIa$Q&4!DP+B9^Va%nl z#%Sg$Rx_cdFRlEIYV|%Uf1OZ(&svh!A*bDR)Y`<QPqy+SfCjH|0NGh0b!eaFZv?Hb zo7j%6qYWN2R9m+da8aWN79BSpKiO5$;}oAo^`i1A&yV6hm81nRFuGjh!Faq{@J!Ib z4JH~z1eZzPCTu4h_LJuIbvJCj`P2e;R{9czQ)l-PEVd1DHHLv#>XkP!LX<B9u2CHX zB>c(nxKDo-&b{%(Sd?o>ciNr~B=#l$t6wf^7c@&tg8s;{{qLTHQQ`_C|M8K&r8B~_ zmzbbgq|fY8%-<UBOfJ0URVr^m48u>;1l14#lSqVR`>bO)Wu<*wsilw;DfoQGwV{IC z`x;gGcOXPD139>lwibZN3@@%`heV}0;}i|?x+Z`>zzl?J32Y?c%i@+Ylrbg2KIeIo z?c}^yg)kpfdLN&xU3dfE_iz0msx=@)nH&N>ST%$0#@zg3#=R@*Yqx={)o&Lv$c)K= zU0Qn^-Jl^@o@r_UulR!@{}ky6F+jeN7hJ!L)OiTC_i!T##_OaB>&tU(T)hr+*7lHe z9kVgqV*8^J>}NpYWI~tju9M8DHd3c7ikgFIX0>`$vz-hmBhD34F`8tKXj^@oDV6Oy zSPIY?|9BfUNaG+(P;GnZxXfpbykX*I>$j%0=%(o5oo*9qSmU5>O2WXlmKz!Xa}zQI zLkarMfsfWqE^!Je{&TXMx7KS6_rnY@Qj<~hOJLDWn`p<WO`4NW0KLI2C*Bv(nGB`5 zer)zB7Kz_SAXh$ei=u)l6`9)Zk#_Bx@Qlv(j~FAR5bR5>5nB5wuB`S20v;Ie-#G8R zc)`CRO!tmUl#kj&NMv0=3tY7V%-hRB_EBBCY}46FgAFEp*He$-4_--^O%*tJ3}>fJ z$g+qJ5h){*V4wYc-m1#@A@omeF`;L96*t$f9-t=IWt}!DydfS+A%(QF3l)S22g60G zXBKdJjW;A-K&zWbtIKXz0!`L~7jOt=57L2&CPM`rl8pL53Etr~Kt1BLI#^#Mpb;R4 zqVUZ%aWt)p=&k~6b<;>&q##Z}j0uJOWh}6No>F~+JMg7a28aZ9QH*pI$Fpit?!XFa ztz>lbyzKs1mE>(@zp@THc=rn6(?vB!!uDE94pqW|Lddy)fhONCs5+RU0)|2l8iV;b zuWH_m^qTqwe0fy2&4GaIZTlCL=UuO4AIhVj`vSd-2C#<uD5hw|r{_M>68G&2I)Vgn zD^3DK4AlgLE|ORp>_?0qX`8l5m?Pcl72eA>M+er;ZKe1V7#IA!@G@k)=L!DTfsqb= z2e2f5U9B}vITrc8*3J9NeE0;M*xX7q*LxR!C>iu$BCCtk#i!k3e$sML@kBZImuZ?r zIMcO<eMa2b?pKY3+P@crr~OW8UHz24+0r4MjCHs_D$vHtN+jaXs$H{2k)?9x%gp5s zD#QQU^qhK{g3e!lyPa+=bSSIaIZ2_$UQ~_prqVXXW_k8zOg^ach}OVkv^oO?BY~S# zHlS$mQ~AK;3Jte5KuUB=_GC0s<>p3cbV1gtWNW3+BcaEHO2Q&*E&eZ+)B+IwyT#&6 z&DW7;63J+;FLD}(hJ}{0x_k)2KgZud0~GsDQI}o+Mz&eeJ{Z8$u{3xt!4VIoGJbgt zaJn5kn@Bsb>#I~$>6oE6cp_awEf)W#x27AmD{@@cg(PxV0cOfP5Ybwe+T(K~NVVw_ ziD?0Yj`wv1pQrIXkrZIp7>9Hqk*s)x?J0UqUZy<Qisb+N2Ch^2Bw+?DAIxQ09|q#d z*zRy}ewbYuZkb4gG~*d4&7bY7@%zojgdKj>xKSXHYlfHTTvhdK8;zQM(RT`3qXx1= zdDDyyUwRjDz{(aZZNHB1A3E}XTPZW+Z!^Zo?4Xh?Xx1QvteL+Pfas375jD5Bg?LRP zH=F0ARbvHwg_OZN{aR@m2^D2B(pDnV+dfid4+xJ%C`(|<EVy^%t8+D8tzyu6Do-~h zW$J@<kCGkxxN}6cmc#&Vt^j3QRN`6T3f@HzkEW92SrqPrZH~Kue#l~cGD(_-NWn@S ze_g7OMx&oI-e~eZ%4$+BEy_ODXdc13k6&Im|8DpXr?!ZqzQwX#VPk+=(jW&{{Z57> z1^UU`vCZ}f8C(@um+sD1aO;3h>V0&xwo`qj;@<OP?~>OaF)nZ=gB@PWnCRZ(*FNr( z{~lqTan2^mkkE+LW3edSZkq0m=cq>oOw(D8uNig^d17BIFKg+hDJsC57(!FMM0pBN zy0M8^1?Otd)U^z({Hu*{!iTSrBKXMgioppy9(i`5!Z~aNA`y7QX~-jLpe$s#q=97Y z(vxrf`*xjk`X>R+c{2R7$PStnW*!KV$kXa+V#sDGWN0<Oh@^%)x8)K8lp$NUM-_%+ z6~?GFW0bchN^;RZqxl@}p(^uAwvNQSeMqoGPn>HsrD=h9K+j3hvg-wc-HxYkUy3n< z3zv8t;Bws1qxZ=~j4Izq-*_YymBPPo3AFzyn+;z>Ax#t=VMf#)6o|fv$RDtvdP$nO z8)ra|ZB<8(y)BAtCVd0f4uId3DuDK#jqjYkkGjiQ&!p>HEO-{OaE(JBREfW+T~Fbg zp|#L!_uql~GXxh7E6=@MGswWPfta(!n^sxfVIJhkIRY!Z;Po}Q>Dc=@odSV<_F;_C z#vW_{%deu_!xDRwrFNZ9xHS@*xNeW$ewP+Wq?#5FKEnTXAK(q^b&y>5D`c}6+!E{U zL9dMWNx1mYq_t!MWQ*<7srKZN?Vc0qs!P=B9L5g0{M9%(KE2&Gp>zrS`6Q@@4N7kn z?}CUo8aoQmD=dVV<O_Js*$f?j6gL@l$IGD>kfh&Q*-$T1E7Auv;@Q7Ftqhar5&DS( z>mi~Obl@4t<+qk8b0XaA+wZOEV{*R?)N^LIX2-5%$uVUBkom%$Pixg6&pIWY7&FBt zHQ=YWW*U8t@#<KA<}iMRF2`ZCBdjjStE^Qc&1^&#l<S&4wXP>s&8>c@Amj8VDIs2m zuazGumgI$WwQ((`mjjvpgXNU)s-(Qe0Wx<<Qvq9<WSG`tv-DdLd?-%KuR)}aKF(%9 zc|%l3j6s5_gqYg_>=DnKDNJe_&W7SgPsP)+__caJcr5d1(S)}(BT)0~C@Q?f<h(uX zy4f^=6ithZGUFq>YphUQz(T|@&r?A7WaS8JfW(O;{@*hye5k18|C(Ev1(q{?d$h_d zD28`U#9KXt=IiV|=<eEgG>@$3A(Zq`{3}X%K?;VK*E_}3t<0@BgVfUy&Qbi1v0WIo zT0P%CT?CT^`O^|hwAAn2-UF&=u^f~CSF!7!mtc48MX}GDB_$CJ>$4^6z6+_kS%x$e zUb7^L0B-`WU;<IOMJpYjZ6<nxHSJ8PsKG(#wh4pDl(~Rf@o<ULut--%2~j$7gnll+ zhQoX~u1qi__=-I?X4|8=d#c09mS5To*A^ngp$ErWk{a3$+0XLk$tGqVRH}yEiL`b2 zbSUOg7A;|z6O@*)<OPX$s+4`a1VfjdpbKfcUEK8FBS*^2#NYCjGopy^U3i3(LPP4q z+FFs3C==3)JDpK_NH?@dK&Mkkg|`vyhx)6O*P5g{qoPeTJnFENj{o`k3Z<D5C0G}_ zg0!ov^Y|V0eJ6C{_-U!O7*n+w2BmvtV!u;$)I!<KK50SlVYYCBc<iDpMKo&>vQFy2 ze(MY?ro|Szr_)?AalkgH4TzslqQGUr$-pA8W^_NQ2OB9<jN>ZP9q{G632D9)@xh(w zi;L6^wjvgJBaB&$CaHn%qf>37T8%b7#`WNro9GR8&kf<$ZvM)F5v_Ix9~C|pRhIV> zl*V#~GM+R|>yd3NWF69CsWgRTp>Ncs1_%gO*SIaR(9+POh*ivdqoOEv?5~>W{%2IN zxmWqrfNSwhKs;$2F{^vJldhgLY(H-)x5Q122XEv$&@Sw?1=4eCGS}D32>ywS|90mH zgQ%n;z%6{<qhsw0<7dZ~Qh11d&FrOc{2xfPR6-Fa?@Es%<Dd9Mm?foAQw9FQng-5U zB49qEq^kGiW5#rsGmMj7(Wb(ibu`O`qBUh5x;BjpgUtX>SL|6M%!GH#uaq9zd|7FU z4QGXSHU?i!xbzxTI8x>NUd6#hf=(Z6qUg4?ioZIdAQ^q=TOM5krSNcef2D6apq&M# z)w~zY$3MqVk*NfMwklUpaSFHF-N8o-cp^{K8n+A%CVk{}K3Xh__I95ke8olIxAb$( zZ-+d%Ro7%T$;t}?e=wO3$(8ZpF_~TH{m>Km6@ocp@=3xD2zqMhw>v=Tb0W))`@3i; zO+T@ski^{8N9PCHdx+EMUcxd`hPWc53%7*2JSnQw;v|~sAddngpqzAVraOw`YcS8@ zH|}kk?!?t(FPeS$(IDrJwG_A?C-eo2QDt!l)F+?=NBCl9*y7@+4Cj~dP!Ptj5fPiZ z^nG)B=NkUaFWFmr&ByZ>Hf|J--~WqA{>WKas9d3HRmXqb8DJ<|bWU}AEeT#GC?1W@ z^mbPt?bj>ImF~mG!5}0&0hh&kZfbIAWn0te#e18UtImZs#xKEwUvOjA!#|ZQ@^Kv! zR0J#ihB|;a<jg`ci|%k<4!aGEizdSEAFiR-BipUROdiv&fW9dG1v324<8|OCaKJ?2 z9LN|=zyU^a!X*hmlW+3f9t86)<OxZFsjpN7)k#INtyu+e98$EBkv7l_<9W!$JO(^b z<JH$|10Yr76R*05>ILk;qY8Ya&8X6l`>#j0`@u<0C?0NMfp#`{*A3W0w3~T@<w42# z^ULJB^B}I9aq9A|ZrzZpz4X7(-qIJmJ8-+xm0l#M0%~LE=%8WXPN*6#{gl2}E2rzs z2nGE`;qcRV46%%au4yqyU2|WPZzAa0oadAWwCyUhpOB;13IP9)_#+5d?O{$6bH*+@ z<K=BphTz@W<vw;8UE+aC4C$Q1=2M+VY#6yKw4!uy&hRvGFk%iMOBjO2vt->M&}d$r zvesHj{4DrmCH>@2qc_@R@Bh8Rv2Z}6ZsXefCkQ~WC!ws?a&-CI%smm0PvX0BG4wf$ zx-^@Lne688*XqqKnW!TIZK#_`|Gbv!TkaZqj<Q$5Z-ms7Fv>jvG~TIq{6%v&-L#Qv zuRxAmKefs}7u(8JT<8G<u9A@f*hNwA5XiL29PmQh<F5=_CPy0Qa<?DT<s*kaIziR% zXmA!`&QS}BB+K7Tr~Rqe-nRn}8k^BbYA(U890>+bP=~@zG44FX^WQEDldpiygZJF2 zO9v3C<ZIfXv`lb5P8!xLp7tZ{jP3O23j8>5o`~+!8pAqedrrBVLHGAxGkH@VG^!<d z#(~jkhq)f<yXb(y`uQ9b*}s_`H3z;^luZ_uR{rn1vfY}Wp3F*I29T5AH}KsASB={o zLTHH}TaAM-*#0kXsm686_PpfeWaQvLi6^H(##il|dBC|RyjZW5zpcaR;IY6`X0N4F z#<2*=PXF}`r55&hWrVwhi7U?`LxWG=a%Z(Q!jma7WqJf>SKYtElp|OdXgvYy)6<WE zT`4qdBWHRJ;5&g*?btw!M%mBk;1#q~kF)r~sdBIMU8Lv9I?uOare1n%h01E=`H(aF zKPmP31`Rc0WaZW1QK0$^GmK85%ClZb{jM{nJjn&!_^-nX^g*3o<UKepXTxCFg6`3S zj_;|u^>6VW$!6>GB(}nEHWT7?e%6K9Jv{xLi5L&iL}P>P>Ga#UKh4@YQ|r)lujLj2 zsX@6(SGbs6bm!{8@F6!O@-#U}HODYp?*Ss0lS!|oh(9qA*hBtDcix|OX)<nD#!PJ` zCp(pfvKB~tsYtxERvm4|zP8ClH^K;NkR0jW8_<^WL+H?{F?!-xP*Jq-BQ10)XRU+9 z(etL@(9z!TD^L``(K9tyq@pdR+Fi`k0N;1Ws1$O~rJ!Q__AU(=D4r4n4MN#};Y&VM zp?`N4?}NdA%N9|Oz}{!5%yNv!QE6JafTwdj2^Zv1#3GaKTTvX8icpeLgF&R+v-UcN zW``l?;4BgI&a))|@s4)@Zug<RiN3wo7CE{(w!<{I>GNxu70}$yOk^*pZ?3Z=^1wI4 zZVcO7r$JQ=*3abstsbVPQJ;ITMRheH0-9kZzPizx7l;?VH&j^So(Ay%%8@==LcpoB z;^Y;lxVeou`k6kW(-l4;245RGj@)Tp=}1}U3vAaOnv}Oc_%n(dnDlv9GHw@C-&-@d zR6wTs_d&44mo5ckPX2a^YBct>o?!K;Rgthn_2%v#xszYia<-nF6RRLZqpL>Dn2yI2 z_?20Ny2KA~rv7x)kI+cKKfqN9o^=s}q`v+*)GDd)x*M9yXkiW`DhT_C9-|5F0>=a8 zg3rr8m(zM6MnUJ{Dv=}<i~N-e^U-yImHu|!h?RfaX-_aq=L5V>Het5Gx&YB)a5oH| zyz+5(81O=cWcaMm+C^j$DnpJn+WCb6(-IvFP}viGEY#IFVBM&a%7*SlepG@8YbEY5 zS;VJ!N@t9=Dlph_eII!6!F|Q}V-DdtA-uXvI=!mPPiL!WarFjBf<ixJ_CQDQo-pPD z<cy`DSmx9K@^29zg+#1(vY<Gv0)UaFT`IxFJ13ojbMGkwx+HwO*|e7Y^4ww}odu<i zsHk}UQC26N|AP`fjDBLbuluWF6c!4Ww@ZlxDYI{>yc;aw^;02EPBkgIdpAN4tB|bv zjm4mor>>uFv*(EMOdQ<Vp3<fFyoyX7Ws(~LJHUFqHU2QEIw%v#0<fq)?QrWbapO4g zAf0N8Jc}QUqCq$q!&4qv>WzE)Y8K_c9MbWNt(-hkJ5VUh_*HC+UV{#@LiYgCctqHS zctp1P@?!-8;WRX1=wHAv37S0!kj}uYfO)k6Dz<l(4kz`m3<vQkPI|US`n6-fO1r-0 zPp%7_T0e>D6>POCe*T;IbG)PCR%HQCQB98|h*Wa6<cQ-{%rfaZyj4`uwswHI><Tjb z#^l!>rg~*Lx#=npx=k{~Ec!E&NqC1B4_`31#Zf9SeRr2$aIKf&U!BHq)O#{q^yRDx z{n6L$Dsh2Qt@`B;c%QOWCu(TD-p#dgJJ=NTcG3T0p;ZPHVHVZ2^lnE+qqD`G)I(8r z<jz~crK^&PY(KbmuFp(&&adk-!CXF!&ybc8QAi_98un;df12d1+!Dh@(7aT0I3@Wg z$oAMFpCWJK{5)yTwH1BDq<+nc1e0^FzT=72<yjTwAs9;ZRXf6i$qn~m_xjUTmWB_> zcn@l?=4uL;(U?t+`Yuq)>zOS~eyKI20jOyW`hiOD1te0tpZ$_J7Qum4O^y~ERhMh} z4Py3<>op5SLQe-V>8%<cv;ZXDOco5xQtr;Y!Xu^9Mrl0)gHt~5;EPVm82j?YpD<Ss zNzo8Z9?PRD{x`T)XlVCok9TpZE-N|mo_A_dTf3H@HbVCf5L>nufXA`Bvj(3lC4|^& zO(Hku8Y9?)#3iFhHJ_k!IM70x-3!lBe+}1pIsqvto?sz`q%}F?<<f!B{nLSGozSx1 zX7u~hdl{f}uQh;M)?1<GvdZ3z&IWASyTCM3)hH<=PRS!#BjRm>F94jB7w#O{AyBDW z!GSr%CGPX(ZeR0pJ42xh?D_71evZesl--Vx!TREhY<XlPJU2RPJ1ouq-_3b-j02cG zY^bY9mAu)C7g5#5(=nDt<w4mI0g=xYY7RHv$O1wfG_T*0*%hq#f6B9am{WpiO}xSV zlF%Y*h1-ODGrZA(TP@I?+ns)XW9-Uk6uAB>-*?p!r;{9Iwu1U!?T{hKFW#4dcZ(B| z8)?5yMzeB2V_Bx=cn$%y+Vy5kYEJF*7ng$!hA^I;tbX_t`z`<)q5G4{GakH~j>^Tn zf*62Tv~Q9pf(T+mq6mn)MLkt7p|3$HPrR_;NGNElBRp1r1#{`TW;3ZmSmE1I{5{?o zP)<f&KFt<llzLZJ#%j&+8EX7ArJU-ugeUmouR#=7Lem1?5z34dhyo&*JYmsXf7FTd z*aI5@1Pl6{P;ceJSz2Kub%{2WJxif*j7`KbPV2`I$L=M#SB+Eqv4+}B1bxebm+19p zJv#;cLSIqfoh3MmJoO7D=kopP<F~}6i(Xc7WaIc+Z;N1{X>cx!OU%v9zwl_Ed4s`X znrp<EmkJHp1rsvX_1ww<rkS?St(=T?r7Px`w{j6LVVgy~iDf?tdYvH|(lvNNBLQ+B zLD!goxnxpl$^*;jv?m3DdW^Bnz5Ygjn#K|=y;WbFzpAuLi0X)Kwe5q(BpQGw`JO^y zzjNOB&;X5qO!smwVc}*Yi2FvV|H+w%Y7oW5$ASa8?z;E`Jz+nyzG=@h2wu#xg<((X z`u|sh;_~AjQ49fbe$!{cC+gar+eJO^lJc%+`ih|8R!2R-fAWhAYfWZF4*{F>y4PdW zFDpiv+fk9=|J!XxVcukDMsZmoHe#YJ>308VjwbNqYI(z*9VIw=07*fdzXvs#YVEF_ z6~~qqC%(R**`MDWQcat&%F%8!tivoG-7zApfp2ua_6~rZwo3U*)xH)6r*y6yOmlL( zXbHg<+&nwffn@>RC&&e?h=o`f$qg$0EJhzi@~7drHtbp?l>!v&r}_FddU{)r_Y`X; zC9KQMtLoDk%1vj&*3!Sk8#L(8eL38hMJYa0O;kt*rC^B0oqbpFnT%?o=^&E<PV2A@ zV&T6p6s?|#;juLRIS7UHmIN5?4938vuj!1BR}rA0EsiS#tc2<I+$M~3;ruiP@@kx1 z(eZDS@Jl=mO`<kVw^JYHc5xbQcTd~BW4}(0)%9tC2>xSK24Qr`gA!EL&JMZi&Zfi! z0}oxmYKL|G63kn9*wNOXy6A;&!G(ox$K@aau8l(ep~_GsyZV7n0;Zks%pyv{rDf`6 zXwRqP_)N6EbDnyO>fRXT6|O{rdOO13oPw)_t>i?*pv{u6=!d~N)`W`(woKaZJcrf; zF>S{EY<w%Wo${nGb8T1_v5L^=B<<;Mc3UYd1A!=eHFOVuNAm;>FOj)7n^?D%RVY(< z4e3OZ^jV2allo@rKN|G0Mo_oova+t+t8{|d!e%gYw|Ozgp2sWU*fm~^*N+|3u|gtA zXz{*WBpB~IPhlLW*#ZUJGSt+ZJFPepEnvY*H()GLhjouAOKmM*%E43Vs(KoVEmN-% znc#xTw}_o)-bNPQ0;y*j6hcLQJ>vn0v<1efjdjC#K{aQp^f^i`O%V3myxdCbM(gqR zCJ-7Xch!P<QA$!ELCa}r`uTrArUEV-<a5U<#glY@p8#3j#SBz0#4BBJCB_{K;mD?( zk+hi&YjwAL{5Ba8Ny((+o&mNzq#m95BqtY8n<$yVMcSPACRWcJrn>*Gz9@GV3Sj+` zXK)vKzlB(z7Vh`aor+h3sFS|YN!)fzDD%8snwc71`Qn?sQ$ft^oO3Ba0cC5-9dO** zOwHa<O$v+7fsU{>LUyJP(z7wCARl<9__TOu7Xi9vt>Omy6iX#zSzM_30xl$hLnz2F zuEgP4BPq~0c$>7Q?XpAy5D+Z(yQicY)`C{^c+a9@|Aoz|KKzD2?Lsiid;jj3AKb{p zpqW_2-i1h^JmOcf4US8%1+%p+Bd5$#5ms&`AH3c-iP+vGeJ(s_O2iCIrlJxt)|ApP zhIukIH2Rp;Ti?%WUH6y`{Z$O#*(DQzc}2l=$Z)|rOAmiym*cubmS{VhrXp7!6i^i^ zw_m4os;dtImo2s;uo6Swp?vg!3{xwZ?Xw*Cidb$aoVjlz9FCgMStW;Z;l|Do-QcP< zb)+4{=6pg|8`06xVYD;`rVV@!{WkceHupGWnm|(`=L9@+c%N<Ir+>=521~qtiwT;i zIEveiB*6QcCRwQGAa|q-cUNoLj~(7;3>2sLuK-^dw2#_S9Tb{c5OG7A;}Wwv`yuSn zfB1>$nx!KL0b=TpeqMhkasqkgS-aGg!05>;7N(9Pj*QFYr(|ScUGlMymbJ>`MOT@) z|6DYR*-uE@VWECAdX=KRs~$YRc{ZAC+Fg$!F6plGIUg|PtvPc|s52AtAm`bzyhPJ@ z$A!zQ>eKSjseT&c6+GTK{y2_4b|_!gLk0yv)`2|8;N40XX_$M`4qmg)6^gF8XFS8$ zf3$=bzaD(RQ=4dpu_nkbN~V9uj1>figme0tZhRDA?2TJF;ax`d!_agS`MFJ?8dwzL z^*x|-!97eQ4oXG&K=@{LsazKy;Fj~4XACEAt1I)P0@fmr+<dXlNghAe-`K1~j(eOR zQ1%JwDQU~X^XQ6^Pn$tmsTt`s+1Jc9IFXZ!x*moAp(s47kaxW|JRIAbeWIqY<W>!N zzBlAIl>%FVIvR=&(o&E>aC2mwwzq8LKAcLzm`C|A4VBVnkM_SB=-+7z11oIA1>ZNR zG46c)O9Y{R#m<PSb@_R#dgY_rt0K4@KVYnMAjRqry2EPTK$x$5&H2OL27xk`iGP7* zpW}XU)Ga;am(zXWsexl8{P2a}4$;_kFC22pgL)GCX<X(=TKb&jfBCYMtYlGpEp1>{ zQ;u+PWh1vy5<cWR!2`n~YFVa6b`>1U7|z8A{qgFdel!@G;W$}cy5cY?oZ+6oS_cm` z%R&Kq&{il|i^omDK@nryrquStn>Dl&hvgZCAp6NpQW+8d<O6rYTT*^ibgXeu{p^n? zMy7Xj;_+o!q*+urs{xWDDcvCYcSGg((B|3>%FlbLM1NHiIObQ>sghY6Fg3_PD7jQ^ z^?5TYLb4il$H?cjV%bg%z|WDBJ9yG(mSzB4b4;gq_a3IvJin?Wg{>ghMpl7r)GoPZ z@5Jbf9Zi?GaxHg%+|KFq_j=KfdYD-b_#h#3aIdJ%?XZL0Rf-Y~=&-HSEXBCxJC*We zT|N1(EIkecMY&+I#x?5-zK-4Fwj@N=9{S_MVX`>--v_N{RvgMK_qX>^DbCr#!v!2` z<3g|450bv-i@9^Kr|5%Dy^#6la{+m8KrD;84ao;O@zV6E*H#KBM*dh_febBLtv{vu zR;AjQZSBgI68B<?igD_+;+Mys)x>rB+qa5Gl7Zz%vH&}FT}?xW3v4(VJerl!5=t%z zRoEM(BC=EO$bb|FhKwNt=O_8F)V|B+Uw>l3d7*1Lk&4J$Q~K}JV8{ak_t#W3s%m_u zQs>Mt4SzGKxz6javvC@##=$=rhJt60)Lnwa9y5@N@{eV3it*I_{?gn!t*pH>9ynj* zdvuBL340DXIwZ9zK99qu;95%!gccKztyiNYOx=v0kgh}9_8ynCxnLl46n9wXMHB72 z<H-~D*l}xG>^^PFYqY>Yn*R4Z2IEBy$Pl*uL@X%2YydC|JT7I~=Q>1>DXs4({RL-{ z&Ae_-Uz+WTHg}c&9Q-=riJ*)h@+|eG)73C;f~uO(aDN#Ib{$p#{fG5h9Ng+{LK}L{ z8Qcy$hCr*`{0^Su(@!LDkvZkeL=CRDqYifXBTV!mwHt`n&@(zG5Zm+bqShE{=w=(e zaLcT%%b5_oQDbnk%?iJr+2c-3u0nx!7osN{)L9<+dQWa*h(-%>y8d~ZFhB(&`kmv$ z%tD~1ja*!YyF*efZMQlnu7yW=u$#nCu;=NWOAqdS4!e&P@ODyGJ)?)4{+hG_sWKvd ztUl1)5CJj#1-F0xNpqoaf#hG0j;VdJ*1}KvdAjt3>oTad;fx5xTVO&7&k0$EWY>7O z?sw-6B)8miQ8MqOyq5p$uB0VI54=i>u2kw&>fFj^3;hh!z%y`XqGDkLfGf(N)yq}H z*tXw}AF(sx7XK9D{L)^|-pKby^OLbWr(ch&GVH7Mui}2FBpGj3S({kSG1kpz0M(J| zqqdocnYM-#+T|7&qmP)#))F4o{3Jaw3n<qa#09X@#8cQN{m%%=X(<4{$44l=N9_c} z2|r_X_~{NZ08<M3)LaR`oTkwVHMeAxCrInm1CzN4o%Ku`W6HuC6?iwwA4VqI9Wnqh zcZUK4cwLgE&+LQP@n($B_VHD8JHLsq(7v4hR=18wIiZZW{KF_7*k48d3pQ`D(t*-7 zWPX;K?I1P?oR3P=XFY?)y~Po1C!^z=DKhPmF?(gwRo7yMapL^G_!2ELC!eb}ZAN_+ z><-evAEqT)+2_Ts;w2@?m-GiXj-KnhMka!~hjKByeNl~@$62?jg$+Ly=|`0C(<7s; z&w6e1(T@)SX-m3~$vfC+bZXD!g7*#k$|yk?4&eQ`5bd8fx7r*_nMyS7f=w|||3Syy zpm|-1Y~Bf}@60QUeQq;gE}nl*l6}m*<&zL|&4EQVhYB4Ppp#Xnd3jKuBMN?U{)L+^ zRej(>@Xe;FIrU^lixW3VSft8yCAy8Aj=g}kc;gUePLzdcU9GRM1(ZCrUFVspqxbq+ zW&s-~^HD?)ro(}g6*oVBVi6)^j5gC%ZWYUg863Pa2ap(+!*{JqGtn}O5kfRWb)-D3 z$3S0H1loVmiehvU%nroF-GjBAJAG7*enJEnD%vf=rC$w?y#2+C@R2oc%i|JHQZh+* zXivm{lavRqkk*`SP7g4uZWsmk-5PhXpsKl}fyY0k;UCPj)#1#~b}wNQxm<n_e+Nzq zU(TlkMm_<)p0<u=p$Zt)b_hx&$Abddefv?yEYqu*OSs}Wk9@7(uw*dv(l_DtTmNU$ zUQ6lo`91r3v%2~+SVb9LFVGzzulqA((w%6eTG2PRvHSBc=Gd&)IA0A%ub(OaaR}X` zl2!$wej-<WQ7RaPQKJcJ`yp3pZFzWLQXU|{DW*xS5%$tTAv17qpYBSF76bXjNo_Ap zhp_ZQ+j*U-ORF4b;a@2$sDt7ff)w3h6!|Z44ceL7-576^hU;k6mvZSQ!x{CA=O;6) zxcEeN2=L2$>?X*WkK)SB_H{rU=>OOdG#Av*=MA_qTjfcNyPlU2n2CM&B;4L}qJhN^ zfg8jW3XRopa^^INc)ZC~fmw<t4WMXeJ)W(Ob^dx%!__5%aqU$Jt%R()OEwn*fDXc< zc$yLz(y5gNKep5MEy?BYCpVWtICMnRwIhQxX;kZUz*7K<yLz(LgJqo9fcRVW#v=K} zn@8CnM6Lo?Z>71cYJqjJ7(T!^Cik!-wO=vYX%2{M=QI7!hj_;9H{|;Yo8LVS@;^_6 zbe7{70qbc%6K_8RGq@J(dp`s~h|Ze~_7cOE2yp`HQZD~qU*vbCacehJ0TMG0m;SUm zAEDHaJv5E%{fwlf{c%_IPqLhfhEJOBnrd1WEe=ncR*KwS2WjLlSy)|)xV$nyV=j7u zSI_CX61=yBYPll^D{s9WQ0B6`C_$-$or5VHs*J?g?o`%U+f&cq#yi5rMah@2vhmIi z!JN24(%PHk2t16rUx_2XD7ECt45n(!=%Sq*(-<VcoDY)H2e|!R`%XuU_k!~HNjNBD z3V2A;eF$f_J%i>tQFpDMd4riE;{RKz%_He_t__!(Wal}^L77A3CUwmjq=v}BnhSg# z!`)lK^=T`0Q_wruq-pZGl9Ba?sX*_SdMxKf(%9Ll+`N%?V9b<Tp^;zON*0IoCv|%J zc`ydPP5%{UqfvuM{bJ6b_T;NO528PLCYCK;DL&>Cay`z4ZrL&qOZ#m^vs%D%O-iH| zDkRt@l8wM;oYX1*4jcOMp`<=cb!W?!W5cH!PDUpvauKT5!kzW>@Qw~hPez5Jq@Hnl zO@S-sXi|}Wx8(!aqAQ6%qBEYNan{R8QJcNjLqARtmr2M;NLo1>>8Szz$*i`VRJdN; zmMo;BPU|=1!XSEci-(+H0W2|7t!x-&Atx^Am_%eHqyL07(I8ZHAfn%Y-PSC}^qrXd zj^(Oa+n^ZML*(;U=17IvNLWR)4xDF`q69Dm?X$OBGU5inWYL@tc`U%*JkI@8np-;B z-){G7^8iy{*39O4Ed*11NE4Vq$Orgol8+(NWrGgOetQ(uFprrvbr&z1q<NjsArC2U zpO>uud=+TPF8q`1lBYzd{W8;hg`YqYd8BEG&W%mAYPgDQ=^hDR?bVKm0000Q_E^5a e?i*160m@>4;8OrDrTIm%#Ao{g000001X)_XYuZZy literal 19712 zcmV+aKLEh}H+ooF000E$*0e?f03iVu0001VFXf~E|Hw4fT>v?rN~u*F0gLe8Q;sKy zl~ZVt%~nUb&vqJ7+!IT4D6N?P(s$!zut}DBL>mH6y20K>oJlfd(>z26V*cm_=Cfps z4>k7`e7-6Ccq$X}U6E5>T831dBu2v7Jq37BeA0npcaExU$4K%0p^RVIw2r2T<@a;R z7y^uluB0LU{FzV`<eU86U43vDSc?0v&P&{r@l%N=2jo;D+aPjHn88YEcBlwifOI2) z;-*Uwssn1yLk(~BsfdU{$}-vxOrh9~7j)#8L*8BqugUr4AjptI$jFK=@U3^uD*Khb z!fW_S$_6gaZS|6Y{9o7HTQ50*Fn?Ij+(%cS*Y-ikcqXdvD4LJ=&)%%P`boL2sdo{q zydTpvhS9l^(0Xsc>OuEWa}rwnFK^acSho2dYiPKpeZdfTVKhLJ&NdU0Mm}cWsKIJ_ z_$7H**g%B47pZO9#?ZYVMmS?TOpTo8)zu%$Q-J9&QcV}_xCTBnfU5BBw#6YEBI3=) zOu%_)J98QwZ=rA`U3+jAO5ucj(a0N{cr9U8!K-^!dQBJ`ka+Ob4v--GRFU17C*Kg} z!;TQ>C(dC{8RQIEgL=!xyRs(q=vKEPwgdfwbLMv(CiNV7;EX%%d*>-H+;T7!>@Qj% zHnzKjr&UI=q>DC(hmPm3Zh6Z+W4zN~@OVsnUwCu2mj`>tkqon|w)1$|&re-IpB60t z&+mjUNF%Q0&(89hD;>AQu=AIrP7%ILI8%t7hBp-EBx5A5GL#fo4V86H2=NeZCLx0# zrFwW|#a8*f+`@8xd&MhWM1VLQNF7vMm;>mRe7F==G&A*xOna6jDEjR{bmzOY(*FYS zzNEe-S!A9zFA{(z;)vjH?czmZT$T`+Oi&ynKo|7B$!;OR!HbDjMB4pg<Lfu%x<`(e zj&Udko#@k5ozq0QtrW2p6`%GKjwVURh<y^4Qxqb+WVE^er*>OW-^2Vf?2!yr6ezv2 zk-nwZ8Mv*;h-B$|)tc)=S89-h4-+zV<8rLUebyWH$o|^Csrf}H<O$Avc#P$i#ZEyV zMv2D57@{%yEKGM{hK{!50TUxWdn|wQf1%af()=7tD6Z{TeP&dE+UZvt5c?y-_Gp>N z4+?m|BOwdo_7R5~KuyP|S0?bmKlf@ql)uO~R9IS`KT)%Ff^=nSM;Qgk&%9Xe8P#e} zYp&fA+@EfVuJ|JPP6-q37PdZJZ3LkwzhiR`2hjkcNRs){OxHK*{IDgkNp^IK6AQgB zb3vXj8iY2RFv-*uV(rk2q`t(pL8uxd3ydVdB!zn(WM$)8!<F?3V!LIe!Rb6%J4O5> zRu!a$l3^@tDI&y;qxTIdrklZo4=Uf_%eySNbxC?(R4BLk(+cJ`D__f3o=JDuk>=R^ ztXE-dO%1$VwE|Nf%c|sCn+}OlI#(6*VStL{dV4ft*ed&&$gkZm!429dfIoJ9{OhO3 z1}D{Cdk0_r)M9i*KYB9dPwAo$AgG{W<2EBI5B(eSNM8g_8R>|U#15Orv5r8fn0Jha z%2T`EcIK&X4~dQmo3brxORTCWrRh%$BxESDaxXS`2t|mBXee@bvX33DRS|d%jF4Q! zMuY$iY%-*n!Tl<NKNC(3)B}v0tL04jXd2ODZvd^UOjy4+p|Jw<Z+#<=97JwnYNSOe zgMOGTR!wkrVZ#b867Nl6C(BPx+KgjXB(V4QOlnrWjPIGl=;)(_h}yA4FKC1mdtoE7 zXmLCBLuY`IIW-cyLeq47f#gW!94cFZ_xJvlwX>~Q?Y`i2GbOE^XL~Q^Il}A&y@3mg z%AN@$WFF5bHN)sYDt_*!KQFE7mU(mysm_JOB;5YQxJa~A<sJch$+3fHrcqJo3G++< zp{O3G7jCxptl61EbcI^N<ZQbbJ$p8gj<|<H=f#2A4t9=!02m%TF`U$!<(4wpUpZ<A z-2erae!GcpQw~}iR%X3OCb;2GNFWcV3WJO6$sZHdvU{i;&luS2zopsCS$;FI3ppvP z2xz#!+$B3jB-Ij>xpA1ANBaz71$>7?t6@i_Z5+!z?=5kkYGz+Fr6_6h$lx|bsL>fn zgf@y(okvw)H7T-3cCDgoLj^{3KA>Q^G)m2a@^d4^J|#QR8HMhkhRr#!BrELm@cid_ z{aX@Pzf(|GOc=CT(UVef)SM6bG$JI#Y?rkqrRx520#A42=(>212s+<pdpdK;05Hnx zB@XH*?v}?VDUSz$zI!1_e-qLS7sr7eXbrx4udsVwC&joA&ctx-xKv~fM&`mmNwvK+ zCXIf$+~>!iDT;r)ukx$AVb1&Oe|LyW-r7YQF_$~;eely3fj3;9Px!b^(c`zzo`rxd zkD0#_BU|31@Y;BeGPA2n7tyRl$&^4&0HaZ$m`k{x{m-YJ-HwCo3L+lztC_wOUG~(5 z6lJuV41ymt`t~}w!aB*~8u>2>SPKbBDp=<D-anO^-U_?E4Tk(&csi*k>*K-}ySHKx zpBH)U8q}>b)x#TEV3XMZ6Do1)2esHxn7`tuJU5#m0jPf=%wL!Aw}eX*r#QUqG|W%6 z3Sc+O$VpvG{)e>aysIE#2geX-+Z)(>m@k(rCn{&=8W+bHe5^oGw+&b(_E)MJjF*`o zor|(%vk?(T8nWZFFMf!6G0`;51x{L_Jq%FjkGCS<KBBcSE+r60d|B#9<-2@k-Ue;= zU>uz@?NJ6bp(bs%ha0HDmQyz8@!MaA!{WHRM)RL6)$56tAi^vVr*;ORjxbCm^LCd= zxNi<7H+U+|6t3G?ZRl^vb=tFR9Fzu}R-Aq{0fR7)Dz0848A{KE^ZX7Z(&ZzOD*3dh z#XC8WZgS^Wqp#CAXlLl~40|wjNT$jDdB_!oal*H;lR~*b{8$tn!Znxbov9UQ9**Lz zMq0Tda6P8Id!DUxPQBQXlImV?Osp?lGc}7PDmL%AzDyRS)?asqX8|t}w&o%`)3Lh* zV(?8Lj&@kd5uf#Yaz0kGxkqjI&@j9qzDXPSawG(|fHE)?*r`Xk@?9%2XtY8LZJ6a| zUH7AgS10fCpdVGY-}KXhZ4@4o5|h}#MOGsqQ6kT?>myb|TV^I#YEeSi07w4N<+s~j zi3A7<ib@C8ej;7_$`$s-=>!A`(EK040Jx~&&p;%BYTTVWM-L(VW6gIGh~(tmIz&E@ z0~FQye^YqpOHbKAAL3lncu?i8d3%4A28+=M{>nHhHTKdjoWhfAmP`P87E>_eYZ~uh zBpo?5SZ?tfP?A}^E}k<KAMqmgS2BoYdkt%LAisaVTu_`K;u{N@#@_!O66tb6>4k;Q z@d#fiFctvRK~N*2BG1M$vCD{VAU)4*-uOdQGau&%j*Tx$4+&k>LzvV(+$Af&%@j38 z6bp3Ar9w#fTMC|q#W3^gd#>k%3bk~-3LOu@Gp3@?;V-&bM*;CEMMrFBpLl)n3OdR8 zGtYnqc{dA|TywF^xQNP@mU+U^GRF_;n!YijyYu6ISX40%t9yno%YQ%SNO~ZlBfZ8a zy!7qb^8`zF{uutBX)6J=6&mVm_8e_7`v%>d1uq_+?_8DA4B*ZAX5uHe!Zq_w(S~So z6cRJ(Q0O_WYu8HmEoG-#b07+XcA8R>?b=4m>jHl+L^J>U*Al|YdI=M<&<1`f#bW5N zmNUn~K9Ar@p|;ncGTwrLaRZqRBr6;}+w_8j`8hII)gy=)V;KR14+s^fn!|yrT(Rx6 z^@JHPC~)`6w%4;vF60;@{?2OR6%~a3HCbL%uED<VOaa&gAIpa(J5@u&aNWBnl?wXE zWZOtg?PU#W4+;re!OkKtts&O_0hBlT>WIUWHk~q8(c}CG|L%>#@CRTBQ+_{hTnoVt zb^}sUqeHqrFe!f!jl`#2w6$O#h%Z9jwK#p%dBXQKBa-w2gXmG(@djHs`tH3ZtLjdF z1Kuw$(BQLc5l>ZK$jsT;Kz4G-`^I8TGc0#8wddfcQ%YPXW=PkC52$c~Px-^&&p%ya zPxjwDz;%IGCCH`ZCi6w9%X4WvwshI3p4*~~|MTB@fYGb%E@YNK<Qa|r1qkSId*RN@ z!&PDOuIa@MJ$qjnvQeCbJ$+;m#SD@&HL=p9R0)*QcRIEd%5WFy?#G6AJryRdjf<9u z*h9AyW<~y(Xlnz4|0mu+Eib)E-M4y1714g{#l#S8dmN|Xy^SYne^}S|e_@2~VLAAS zq5#w74J*29GuIk(*MyJ2+NCtCm0uI9Z}emWdTAeA?!njE0Z~6L!_<O#O>2#S=0Gr( zWhzYt35aPNs-f_OaJV0jbCU{Pr=vKKoSJZs{y@kd8u&YcS?P^)u`^$Z=1$KVx{Lu% zIQ687o#O!nD#hjFIG_E;mO4&Pda}h;h!A?be%UKqsgnv85iKO_w*NNoVqlcxa>?Ks zQb+T7U9CB?j1)`WX0sxA8gg{MJLc&7jcl*6x)7yZE7a?dM-ap_i^SLmdp^u=sy!)i z{SR8NGgULzM-adO!VSAuRa+=NR1<*f4lK>n^O|{}pjO-Ykx5=m<QqOm>Nv29`BTvC zeTJf#`rTze2eAFfuF1JSAzwKAB~;6gLbB=cRdXU(%J4@>kicwvvC-t1w&g=(Xk+0} zy~W>nlpIT<Md}3zHm?n36Jkz=RhW|3D5L=Zy^Jlk_lmS4wZJP*7LH9IpPJid&6deo z?0$8Iq@8lgzXer$z7V&KrFf%FfUKczZ^uBq->5{)XkHM4Dn6cGqp;Q|ntDj9l-z~) z-asNZOOBhc{^Jv6PKIj&J+T9HYiEM~M=x(kjQy`N{HK6P<(PYhlM;;t1Nkm@B{tIT zV<{KPsl^`2=zp3CuU#D(7)vK)G^U|etx=V>uA&(gNO_-L7Je1r->|Jr3-?>8T7I}7 zRx%WeP?y5UW@}&^d?s|HT>gAOQX+kSuDPnns!|wHR+1tXxDt>!35m+Kt2)*<CimpM zD)8@nI^*d557wLz6L(kClf!5B06#Ng8L5di2_nz4H;iI<8%B1cz?*tNQgKL@yMz)` z3~WqMg1nx|iLL~QJrLZ>j=wzF8{MD^T#|7;t@wWB*aq{xKT-gO4X-d$(?~wSEx4X( zf5#L8AM!n<iT|{yXW<?0)tnL<_xGsLIAIQGD*r{*{cp+o!~=;35cr8;&fOr#K4++t z@o&E4^}-^y6wekd1tR8ztqVE`Md=B8Gt-1>leS&-r($W9I1$DzwqF49l?_NcJ9|lg z+&|K<mObKP?b1O-`Xlmem8C2SS-`G?%{wOAbG1?Lu1-IY&zH{ul=_-Y0HHA{;BSCJ zoOL)}$f&ggj4l-bS!1Eg5vgXFCyn!3$l)mYY5}{ZHL~deI0eFwM#Im*vV%m18=y~F zbR<F*^PTdNnoo8kU`0Ax4-F$RTHU%81JgSfG3Dy*<_Ta<5|nYAdmP_v3Sw==jM)?M zi+Y%BdaZ}&c)%oJ=sw^N<E4a>1{e|zt~LjMG~VGLNl16cFH_O`$@zi0$YtxIeqiIj zE_fJAu`=c0{ok7R#A-_FAqSLM;mFQ|dvSKTqPa{^Ne(vZGERIiX_z%H6M;o}7DT08 zmSsjfe5Y#&7AWE`lBsxdv?>DJ9wj=oqd5QYexKmS-u1}BR!IOZ|6!gwW1PwXb77^m zfWo3GR-D8;04z@PWX^<GOv-DL3{Nh$f+=+B3`;OE&yeVs#m*Dw<|`?&ih>wI%hk1f z(9cp!?FyeH7#5?C_7uIRd%KB^plOj=3O8x?G1SBRaIgY@O{x!Ht(m!4>PhZQV9Kc} z7uaJiZYr?cDT;?y#enPt=O>n%c}Yp6MkBfSl2HlJH!ll>j_+`bUmC=KbDa!L6~z4+ z@C|v*k2P??r#+xWGFUk+tZ7~d-x7}cyD$siwLd{1Q^&X*g>cMx8p2V0g`Na;BRorJ z{!RYUZ~LKZf%l;6&<spa9sy;*eVwwVR-e?O^;F4lmRAWZ+Uymk6Rwx%&;k7b)yg*x z=#J6OU~j{I{_bpP@~m_wgjm?Q)Elfa@>-^%yyLiKG#r&c&@>Z)GCjcCwHwbVWrVV# z+Zp(L2Z!<2>mL$J;Q#|!P7HhtW}&(u!m)H(FvkM21rLS*xs05gc3HA#CZfLpnj07p z(w+;?&EL|JjZ4}#jYObOCwMv#fXn>JsA~d!)B@3>v?;NTFWfRD&Ma-xh@dVz2w{dn zeVYw^d6IaUJ`|fKrPb3u0=z(_f^}YMxq60Xfk_acg4!=BM)DJ#SFCn_+^7eaja;q3 zQQyUlqz`!nhE9m9LhRaZ<8{*`h<QX(iYy+t2smtjPchZUzZOUYmf$mvbk0v+k9P8C zMsRNr)@s<dFHz+O8Yi!ch1LuIUpz7pJAGhhA(99qy+%$;qtvd0Ns?)->{N7IR(+G@ zYX-Ut^$E9!O{Xe!VCi>s_~2+KRs^N>Gx#43%18SoajCV}K4bt)Du&BG{ljtNtI)y5 z0e#cFG;JaO@6Q#yt8u73fvv2&n!lygA5p1uY0pa&G*q!-`PZA@X4L2&p*xbBP^kJ) zaGSEB7UEwkH6tX#FDvy#2E{1SQO@qUR73LlncvH5;`qyPM<8FdTTs=)76-hfF<xn9 z2GP#w(`&H7!>#?(P);0`MpXS@qe7~Q&ELpr)A<p|Moeez-#k=S_SXPfiQ^WU!Z4h_ z4Iu>8+C?Y;js3G~{XMLZ>x-H9J%h&zg19|(u>scb8t;<r7*;}KHk329Ws!x}+7`|j z`wPPa4?1X?sA&dw;4}pwGW|Vzb)DY9(bx^~1)|{HQn%;~yrSZF64HeiXYa2*VMk0Y z+w27Os)Y_Y;Z-ff<XPaC^@^j2)~It)0Ka*igR^3eG-^Ijsh01>IJ22bv!$hD=PZfZ z{AvXPwf`eT_5)RrF_xA~z6{WQAZ_NFS@>5dDB+F(o3>&a2zAttDfO%=xD1^~6@Psk zojXTLV^sZ!%Mid{E{n5t>f_P;GEeJ*X^Zak@G0$R4sf}>Q|%M|`7;o}7e`{HfS#0) zS0%FoPeJ(%3M4KPrTZES30UN<-Hv(<wJhvYhYH{t2m?UX-r0r{KM`tzJcN_Iz&Sfc z%opF%v)NA_=vC<%x-EbKq^HAw@`R3?gs>ujQx0I?z4e=#qbZrZjb^uKE393_0EooA z>Aoqh;NPAD%gvkB%Q{UXG?>j;%1jq7>UBYI;&d46bK!meqbAnmpuvgOnbDBEiM$L@ z9U_ZWmsP*~#<7H)=nMIxmA+!$Mba)kM#eJgm}~_vqV4V*_p<|wC~2(kV>Dg4fZbB@ zIeiPUuZ5~cDL*rFW{t_*UTaDr`vTP7?3n0(#IP#^ff`SM1ahw+@L$(Azb$zFGvnjH zP{vK+6XYA)6qR*}a-klbXGP>aGARRt3uTS#T#v#0*y%Sk<Z2V5>P<!W5>ClyE2pPJ zDxd+8Ckfc#87iYs5a*+YCcr{Y@r$R_aoh)a>f*(&em7kV#0bn@V$R`vuHYix>)J9Y ziE51NL}ZsUC|>6*{1H909RM~q4WFeT7gX>X)CWG5ORmXwAG26(5u?jOQdmURWotvt zJ{xyLijy|?H^G5Fh<*ac_+!my{&0eSai|ZR8pa}6AssV{4s>NcLa%ZCb?1<qYa!@z zBQ*^>w)Prl1IZ>O8yrG7lY;{YZ>dM0MQS@h8)ybj^*x?xT>}DT$H6ixMHB+VcLQ=N zPcPB1IWnJKk{wrEIw*m*M+jPk`8{EQxhQT-C1@RyC{`DUvp@#qhs5KU90523_w?-3 zs`~ZZ*V{uptGk-gP+qWw9s-njTV}KUAPYsGMn`C+^iKH)0EjfjhNen89KoRb!rF*c zn?EZiIPv|-7e1*?>oF_~S6vn~BbIt3Ml+Pc;caV_2SA~8D_qTSUW$g&(VU+b%4jBz z2w2fo`4<ewNJ?F~Kwms|pE-Y|F%@?|A}78wyu?o-Kr1DfoN(f1J8<~j8I`q^;Lvks zt1uaNW2Nj7n9CnVMRp-ZT|ss2ce`{kO2+~ow{biZ#k?nS-btaNR{pCjUOM!Dz;4|u z{8OJadlT^3p=7RSW|<np)I08_+WI#LKp7MC(Kz2#`kJRyA;?H~_K-i&jd47&7t6xt zI)}6tL>OR(j;*L5^oNSHlva%ZcwjT+VbttALAt3G_3q*HYd5lEpfurOc_Ob%N8b(U zOVQ6Y+FnXG%0Qq_Hbr)&0o504D>vlk)<wp;nAKi$w=N5DvzdJwzjrA3dQh#bH-F>- zV1gJYvfD5;w3e2TIkU%PTghT)*gI%-tYRLEK?SK`3MCNW3>M1gV0>4OeqR1o(8;R) z1tfwi;!v{)n5uooe9(=5kgYFtbeTCnde>%YUt-+qD$}Lyts}@2eR3*Q|2#P+<fzfG zi)qyo=cTto^+47Akw9hE7fluS0od-dssEv{XAB9yRX^vUNvLG_Lf|i?5Z&Y5UyZ=^ z)%Y%tWzOJtG<Tq=RPQ+OWm~ky<5<m@0N-nXba&aY07Q9m4dI3JKLSWhPHgcs8T;Q? za)G2Qj0`+fgrqwJ@d?eEc7A0lCyxyt|LBo>DkBp@n|*RfJ5YMil}7(4Nh_dywXnfz z#irPVdMx&@JNW_|=h8u*jDwacjk~&H*8#8k)^z~8>>Ecmo9`^&IZ`J91GAoCg=?+- z3Or3sMV>}#&KKAOt`^O$utl8nkC5w=L$2>+TR}9s<3X-l{>he2w%iq_;08C#p?>B% z$Et(j%xNgtZ5|mB+3*hpwl&K#F2K&`?ffDGkw_VaSQyO=y<c47(MF@U%IROaxxgcS zcmnA+_c7Hv&X|k7Q^Hb6o(GDj-60g48RhxP;ZvS&WdMu&fBuZ-vgYbb#Q8}^>Yvju zCe*a%u2M=vlMTef?qE3M`-)2XMGn@a%hme}jfdwn|GR({{)ZmBEP&m<4>Pn7NHo2j z#MY#D^;eQlZpJA+l)ij;p)S&>yxtJhA(r7nPIrrwBSrsz9cwb)>1Mv`$>%uc$nq2f zUwPRJz^vFP=hLV5VXlRoLPt$+xrc_r?D_u7pyl<lV$|ZjI=1(eVA4<o_@1%qIa7Zc zo>ZD)dfqmsHK=963FAY&3{^Ee+50usDJzCtImrPZHf(p)N~@x`{sEy{Eh9yVls+Wk zUIg#4V9FG==X!{fTiA4Ni^j%2BCpm?RGvC&zoH5r4iPg%zSSve-3tz|yM&n5wt$vc zOd4T)@-fYAix=niWtf34qH#CuIrq&U`)0PnzH#pVa+g@-^S!_ePT#TeMOyQWWin`t zF=-H^0{v4<mZlt(a;7z*X-wM}I&~X>OD-dib{_CDFl+Nr{p7PHqy*LPP%Xw86S4U9 zWKO;j7a5!v7vRSx<8a07+8=0q#@CS&M$rFE23Do(D1Mt)(B@_?Mgg1OCOMw`P%hZ7 zJAjwA+KhB-dLPn_*<sl1<0za|8pF$CmdKI}>M-3;>rn=p0T``tvUL^O6vYikwG2%T zY}yt>gDQ|CC+CQ*Bvp7h3Q44x!hZvrL-CSkkYx7=@DxfHHHzK8RH6Hh8RY+vO1682 zBttArkxBhMa#gd?ZNpZuX8zDPE6&IFc_R^X1(yf8PxlK(tV&k|vLnb#D0njZDjtr? zZ6K>?6z+4S<|`wQJXlhr6L?6Rs~BEIoUeel=a+-;k|(wFGh7I8;3mC<Y2y0G^Xyd8 zxOZQOFJ4u3cU&C{WBJXZAp?h=>&GSIc%3Bz__~v~d6LA>P+2g(`04dMAs7Z!+_%Lc z2*RjIErc2QpG3Lf3rzHE!NhW<S1+R~fP)h9A8n14f2j{kHLpfYIqlmNBgO7}a*tyJ zctgU=8`jBkiT=vwaD+&Q!q{#1BHaEq4wu(9++dCqom3<e>U26~Lm?33sO%Q{M{tEk zL%;x(`i}J=+8)SVZ)U{)WumXfwe{V;a$WV}+IDE;)2Uuq)88VX<@#m?_8BypZRZ_9 zk-1V3#WijgFlz}<Z0sG+X_m&+gLt}~Q=WAf6>GUvWI-@xs{Zo2bQeGr*XC8mqVyH_ zetsOE7Wqa45nRu{mP#FboL1YxHWP#5jh+{HkOX0d@ap1eS$+jwKz8X+jQ8fvd6)px z-0q7P%4OaLjWOf|!t01Js{7m6g>dPCT^S*t$9$`k8CpGTqv<cOBLGC_mCn6Z7O3QM zaJ$xfLQ#IQ72D<2Pt9N$>9>=3zJK@J`0dD!K#Ad&4IQx^b{MY&wuR`9gK8W0n~FyP zHp6vXLh|eTRb#_*pGZ9nvOpioy2WkP*<%$6R`Qi-T-VUGP9YZ2nRj+c!}VE&;NDW6 zzA;nf?J!BqjPDSoeQL&*=s&~aEhWe8$9p?j9G=dAD&Ybda5HEv%?#3LQLwT76*{T@ zYP^$do|@JI)pjRi(jSsXMOq3+(*Z!Brlf3VHNf>6leC6P;o8ts#n$PolgKzA+}zut z(Ww2I9(Orb<4efNsfl(8V%lDQhKL-+o7Dry9bF+72XUgKCaU`~#~Saz$nYtCaYLun zKMG68R-o@B@qb}tJ`8P7HyGWbOS-RpeavT2q*q1v{739)dWHobhF7nn2pMUA)HQ!o zzizWF^B|Eh6h5il0aR$oO~3(QK<leP=1%yqj(t_?>L?1_$6i}fPLH`ThaJU&k|v!H z7Igs7*vMB<Gsx?XAbh_TUtV_4bJK9(fm_$aBF68q4iVlaW+Oe-*jg=CVciVrsV*Zh zBpOAAgi(#LC*A5CY8qt}#ElW1h+xp`1N4~%bU7rLKHyiDU=~_fFgMlG<-_PjFi=X- zmr4T_wP^x8T9GrJwV7%B+ImtZoHM4zG3iE}hzHNa2b$(gND&0y&tW)H>Bt?h4AT3_ zfoeR6p+e;d*Yu|WmnvgmwW!`I+FOLKR^=Cr=(HFF!>h0+3V)q@l)sXn`laODckx53 zjliWqk)A<8cEKoEcg}edf(8^C9DgZja!>ps%0_m&4)R{%^Trmi#q{urb8kMjFD}!h z?2}3px~cWF=0E_N;Fnt|&CGc_9(;W``EFLwHjUQHRH29?;c+JThJF*|ch4MXM_|pY z8GjhKMuqPB47Aw{qar`?BE8-ON!#pQK?A5!_>y+wG18-Rfe+cCK~5~k!@nD$3YWt* z?+^*SbiL4l4;dW_FY$P4^5Y~VuC|3$(0o^nEbs)U5Wb9RQI-wDcG^`Aao7psO}gXr zypAQE{m<PPdFiMttBWai0ncqnun8cK;_S_er^-RV8fin8@>W`5dn_(OhofjY%dIb@ z0QIiMjf5c|Y^m43BP=5}_c{(CDHD4&g~*UbV}A4GZ3&j9!zJ?SZZb)-Q5B=M5>th` zrbug0fS7-lfPfmmXXXZN2zKa;He&R=<W+H(&RZ?<$~sd4jEi#9^F)n{Ft>fyo*Q}8 z22SOqPoALxwwyxzcKB4NtTe@%foZq1A_nms@S&*aO+nKS3u-4!ol%TC9IcMQC+iEE z>Mg}+j27RX4^OHTq*Mw-NYUkdyx!mBQKnp>IAuaSy9mG6s3ki>GL=VdOSiA1Dw@@( zWwLFAs<WPubgVL=UGfP=>^VP}H#UG~9sNl>HA>Z#)T5X7(V#H5gKF}*)zV)Pwv88G zdy#)g6)le-D1q6`PEy)+)mR>-tzTRbLcJAyAKZ4^yfx`h47kOBwq<{9Bp>>$_3lf` zVa>Z#ak4<RuldeVlm~b-MeZIZKDP@AaQ{iTg@TeDn(@MHUg4N+B&d2q-Mqzuzpr%a z^;)n#XKlV*k!pcpHs+0n@2t)+5^q#hf_Zr>mmTxFjdKO!hf`XZa{%J&eY=K)*pKf@ zJe329WTJz>3W+cNOW7NNrpBa@hxNi90J6Y9E;5J>(y8JfhO_TWX5re$X5OtXUDvg| zCe2&VHv=cRBHz$BDYelzjy|hWY!Ep@#(X%$`VcDl0!3Z0EUTB1H7;HHZ?xHFkgS_D z50M$d8=dnOTWqz8`Y!vA<20!Wlu<386NUCfxoI{<8j~+^O*Aa{5-|f7qglT=!V_j^ z9;dXYUam)!%M;(7Tnc_ikC=zI;0%AcjaacHx?ZDsSSG@qrGb^?%T%Nt<h?<&LsSUS zrlcuk)9bhieb&DoPttTT;-?#T-*d~QS5M(j4x><aCurfgugti!dc;)O`gD<&xW;O= z4fO>hXB1tr;(GHzhnNLj(U9+MBk$@oj)fH9UAd}3?E_9ZQK=W)5xs6t;}4U}iNRgZ zPwlU<8gE_Ww5%8gHD>S7c`CTu{_Bl6`bD4|2%x~#15ky#;8t?Pj_Apdm`v*25IZd% zykMuez^6BnGY-zfvZslPaA^WxW2oBa@a;R*cmG03_1ae!2g~7V-$p3H;}vAC!tEG~ zqQXMD$(UiRsC?!G^<eSW9{_L<GEq07kQiX7wt3)5AHIsT5Lb^Jq0kWh?vdUMBCehK zP{=7dT{ATZwC2#Ch$QgkWVW|~|4!=jZkMslY;3-0;&!{4i-41WG7#D7x?>w#r|jic z^IPleB4t_8OJTx>C+m`S!e?}})dLE1;&48r_rmR!nLSXT6uo1ER@WpBVi&r>f6U6k z;iAhAPL>rlua>TMge<QxF(TA8ij!8EhtlKw)T=y^{puyPP+?EFG`_`i5jpXNH;{zb zFF5l%)+HQ-+Ht7I1>@wxyiSRKZuj_Y#Cm)##4oyFW0=5pux{{KdM+d!iW?5d<pf+u zFO~j)eRyFRUC#a;)r31hZ78tk1cc*g!Q(mz1@O!m9Wlo_3es|V8Klu(?cH)LF<fo! zKKyyCypjn_hwTO3zNo279L*9mnt7qhEDlyfGsFSsWO4iwvT)ii8|(erW4v4B$J%x# zWpdGgz$>LiQ<l?6?*ZJ+aiC$Sg`C14Yp(F|1AF`XSjlYwyrH&lk71l66&l(^q(ERa zM2K(QRQ2&a+ckl$5AVJVqiCt-t#XVFSp^NK{)0v<4Q#N4xXn57>DF5?#(0uKeE<I_ zlBG`R92}fP5k}ngX%LPCqbFfIAC?RZMC4CZXp)`+gU@C(Mmrxg*CI3bb<NbBb#Qj; zCJ*Yfx8GB)UnX0b)f4e-GiT_S`>aj=D4Mv|9qTj`nRvkX-P>(+msGX1zw?2$pYL4K zK0kTu9$9v?x%(>lXIzdxX!;2WA+}9bsf24!;c6?WQKr}md?Sw+1E;gPY3BS=;C`d< zSwBcYtQj7Tn`@qH*rv}Xd1VsoOE^m*rJ5_ZWt^QpE_gGtmoFr~z|PwPC9T)K8uU78 zlv$|BwjXPnd=nUOcc|UrB1a#{L@|=PD@O<~n)65vw0<Hv%(fEK7xQ}KxnlNUy{uJ& zgi8o&6TDCxvBuR}!{hi<)o+mofzj>3K&Z<=l<wE9Eu9{BxXpE2_}O#K<yzpeP_qq2 z;Pf(St5e;PN>bNxy{+pa^JKq$Pmk+Em=#(RgdVHKKo%Qn51<JkO2mS_5WsP-8s}Pu z*r2}39f+iF8pgr`7>ijr9%9I=8LQWWti>NCa3WjsX^LoKNX)p>g=z7M@=cOcC5DjH zOtqGO@E_(=s=eVr^)10d3lLRI&?^t%z$|n2lcUq`T?SoO>(hpfhvWQS-iJ23a23HZ z#_JhFVL+GU5ZhG*FLcJa_qXnjOxT>VSXG49HercNG$zvXH|tS>*dNU)6r!nT)*_B{ zf^LZ3fx;P}-FKoJjqX0%QOP{tZEoSp!>i<?E1@aqX-YoPJY8Ad{>qZR+km=L=O+UH z>l?tispon1#%*|o(xfjz6wC*>G@P1gIvk;;n&0=}A}E|czJ^J3X<DKvMf%v51Wz}T zOhkSiCcr-agK|;0Hs6CIr<)G>v`i#&;LX>TiTw^up+on<e6$upUzQM(lL6;%n}7s! z@)h)PiM_C3m2!&WoJ!-7lEEi=$!AG79dhNqkHj)$iJC|1ri>}RgwfswCz>-h@VXc> z{n63WmYP#1BN^(qK5&G;G8zkMdlkA02x}(IA65$^&U;EWP-rx8>azbu4Cuqpr#kyV z@7K5N4FGi1>`vLxICBdwv@fys)jAOMau7Oa%45;)&0eCk4*<Ekv)U*03^x4aTQZv! zQ)NX~T9Y}(CY`?08}RYoH7~7<z_YhW0PbzVb;8aK7DZ%M@&9J<K;ai0dQBpaq}UVH zme^-pC}3D`^YZ>|?l8*ENEgp0Ym6$$3Bd$tT;+P#M_=H6xqK$kO<TP2<jU_{kq=N3 zkj6F_Nv^H1D*`NX9AtSv{cK0ij21vW=h2J_TarY^^OIaqM5fGQ_Z$nVdtsGPjm|pV zBtvOsXHUlIi&*Vzc=(UIEA_F@Q~rMo=dta2816V|+OwZ?wu`CF(ELBrxcNrEDPz&= z7GV14XQ0${h2Way2Aat+nkRZGk$iJnVxu}k7xPFcU5chIUT*Z<3hTCHRNwv6o6U{n zD`IHr@|n$_m}FIJyF-<IykItjxPDseT-Wh<IP(YwD#MtN0#nrGxi{W0_W3db&El-( zV+1bCS*DcDXKOMj=2jB7nw=^f14<M;)=o%FemKU0Ofd7=_A8YTMUXQx9NMj)p~>Rl z2!W1Nn0*ot#;a?~p#56jBQLbxH6Z>wQnF}5P4#nQH2UvT_o>xIQaF36g%I&vD=#G= z;Cw&d+me1(`TNH3nyI2VBtkb^9YD_N+@f7)p*g!7+t)Lwt|#ogRBJE{rffIr=kJ?s z#64jIZh)p=oRy<soZ&rV$uN%;A2|16@5i+=$JA+?E~9CLbNJ;=(0~ZL(|+b4K3nu} z)&%sujF%&@5CsQ!wVpRiE15|-jcirBREH%<+M+~U)wN1hqB5`#Udi%HxSKX9`o7Ft zyv@1VM8Jy9Et_~RK4B@IfQad?)@D4<>&Sz++iy#piX2+W5h{K(;T)pnZSho=cAtSi zyuDp~BW0D;fpWtEkvG;sS>0aWf;`H?Pi4z3O=Qik*9%FQrW~L@lHQX^PmCgQf*_AZ zr%wx6dPAp`lt`4~YbL&jBr7Nd^+;)mJNkf-sh~xcr`)zoqL>Bs7Ui$R*WYP--jmE} z09S<I55|TmN;C+9v!P%cb&&RJ7!BH{?L9Z2*Cb~GaMUTlna0pC4WtquPxK+;w-!y0 zCIuWr=$-JZBEz|dN@hKw_l^w~<&%$(A6<zXgJWPx7c$Mj5sbqt@$;}B%yx5eus<Ni zbpDyX(JY{`ckC41a*99NlgTID@g(AIXg92~Ab_i2(D#g>?#{7W2uqxnI%8Qy<LNAr z$OrpEC#?!dmD9X#Rn_@(75<s$p1SI7HfV3qR;o4A^FuD%Z%&tRn_>M3we@h!;Ei)4 zUdPA#WHs|-m2TGskcMF%<PVSbmiRvy|ESr7*xOUq`@*4@Wnb=4fL5JO^bGJ*cZkdG z2r)$HsJ$!YAD1TTL$&C_`*?xyo$SHC`zU<t_vQFCWbu+qmvW1M)i?1le>-EA0waV* zbohQ~eT%+J5*|(j19C`KYsd`gd_?i?_o)+9QqEjv#BpcM0(G6HZeA{gImq^oZYiG^ zg0sW4?5RVe0)=`H1*(od**T;{<i8V>6tjS#MZKRJVbIrIZFPXkVd)nXS-b@Vk9(bL zuL?a_1ba<J>W+|y@rECBkdt_Fbi#TBjU!HJKj{8jgkPRH{cP*9_adrvkr&CWUQ&j; z$dl|n-g0pW&?L?4XZv&$LS6SpO`L3sJK>=+16F#jnin${Skl@ZJRAAOC#xo|l6H<1 z0bK`;S{Q^~$G_oIZkTM<oV@8_D#IAZ9%}9aXXQ~?;{XBCk`MUk#=*OFFXghfy2YT= zm$xGRa)sV<(*CnK7|}Rp)5kuKO1jr+be!cSE-P~GtAb(i<mU6m^$i{W$2cAizwLO> zcn*S+e~*!j<X@yh%vCYq(EqEbzmv96G?Yg3!0H8*)+jT$+-O^Ee#lLr7l*xHAvR>p z+5=pF1XfoT6#hKEJl49V_Vka~FNJ}e()Xqau(t|lv!7E^m{5)ChHto*7Ctdqq3b*f z#DCy(-4Sho^}SBJ5*4rjoT-k^IVjT*?z@~Eri@Cm-$-pM7=IAA`}<UD*pi`+=!aW? zF_={ptH0uiGF`qhih}Poer@O`BIVZe^6zpC9qY^s0jG)CAno`)TlYb`xXsvUh;{As zb|=`XyXm);Box-zFF%$1-?Z}=htcEpf{9F!M1e9qY%Fa7NyX8fV69=EB@qtgxH}cv zlWR6&ei1hb$uYroE3j-|uQ2B~hyPCB!qitkpJT5y&kmn>Jm3b^qm#Dq^+jycO(()R zb)F2kE`fW`Iwrl{SiDXfwbocjI7`TDAsz$I&;Fsn{pN=(=x!Vc0sA}r7EQ5bq#JY5 z$9|AUQtt^QbJLLDK#{6gxkLM^yY%6?^!j5=t%T(-g2(YrNV0iD&i;MJ!d%`ruaig% zH+TpQi#But@r891L*A4(5G`*J0gjhkJ3PP(x9~!=Te{=g)dFQ`m+G<|=lYJ3i4m8o zdxsYm<jpn)Vsd~4I!4*#A+b?Z(NtaxQa@kX@u$Wy0PP)mi4bGIDPk;wrB=Z>|7XwJ zDB_wIluQl6t;BchSj%<RDCf7JYAJE+&}J$}YaLpKVoHsFLIfK}a7+^IakEa#4-->s zdaXpT3hCb;To#pn4BH&r%M#>-7IDJOV-NW<jfPy8?*Rdv!L2Rn0kHCMLF1R^`KP%S zyhu+TrzZMw95h1h)ZP@;Ec3(esTu`H>}SDAa-VZxmy-NAh`(I~1I6`4m6dQDM_naf z1w5oO;*u_D8=H{ce(BjybH={ySu0Uw@mknK^gn3nv<Y_O(@^cPy*QWK<XD_%=mL{Q z*3dLdH4zwT%zng_V?V2G@2K586U(_Xf9Gb^8EkqL4Wh@cc}hTtwReGK&~@QTR6-|I zOw2OWaO};qf~^qlBw|)1lp$>}+S{L6`~d}N5frF@?K{1-%k5jbKjrbFRoc-wf`@+l zPpYUbKezdzy5na_x&`FpT5Bet)u}tNAVmY@!<~LF<LBF{&F_1CTrNVNyDl8}$$-|> zHcWrh)Q`4{g;y@ZhhYN7i&Yl4Xi(t)>_1qL(@d>^cd=L?Y>6QOht_T>V^fF2s{rbF zef%77>b!CKnQRs^$eE&R{^SJybqlYlm<jBd>Z8wby&Nnivvez`7*b>r)k79>s<+U9 z$h}*ah#F(pfx5ly(<!#zENg!q+M>{VEnO&5SZ8FH`^QpP{m?<OU^N)+vkBtmto2~v zu*&#=b@e32`3m993A&Woh%*Ch-qzzW3Vlu+L;grp3J6_wv8jF*wqv}(cmt=U^V0DY zlncDqb&0_g3SIX%b_wUWq_8x`E1jvK(5&r^LZMbWt&K6wes<Cr8eBWeyy<wzHtDaL z2RqNOc!<4<62M9aL9tV>vvmMv84G7Gg)@CEU0%o$1Q=DtncdUO-8P&k`^tEqi|bPN z#B-GYdM{4|LVK=MH~PC^_NNS6GFH{KMI`iHoz0yJ%u4XnG=GO&pFhD7L!6Va?a<7A z>`frG7)8e;OXul2|3<Tj9~QEp5?%v`0^<Pi9Z!6(hiKgc4KB=Q>Q153%|Hg>sO#Y) zD)rX0X^xm*m6io8H23q{;Lq#8+3YE%g^Dw?+XSkwuap}WpVAz~DSfA#vNl>??RX^q zdV5WmVd0~F!B^^8g8788oa*#Hk5ZhX<SIrVdSHfSBSkX%M-ZD<1TvG_5(!oA%Czmn zTSU!(ncZU;+)V}kh30ZRzDNUB#_KdOxY9QClk^M&`(HdqLoNDi8c%0s?BfxfKW$FO z4aR+P97!<kmL8B|&uhY>esBNYYeX(pstNSY6%=Rp>4PQCFE%!CuEm&!lejh)-T+lk z*{r0tQt#z!Xp*;ey;BKU_m*(q)Vnk49J%IhI&(u2(pFAeGN7d;eC+ml$RCI&9BlVn zr+By5%0}f2?h#VMyO6K0mDtBaKcML`E<}9A1qt?#x8g1WMGbe2Oc-|QgSK=RMVN1l zUrIiYPE)jFRe(2xENRU%lss|YvXY!?|K-**z#G4@K#<R^o>mtcvJ>j@K=YmOF_;`h zz^8rdL5?)@;5Y-l|MI+RSEtYR3>5Mq3@um+yn+@WWSi-`#&izAw$wCYAK>o8N#^JV z9cutGjj9HWLQV_AIKtFZ-H3q?I;_#-vJACR`5_F>ig;bTDO{{05J+9R9T2`8xwq>+ zjx4UAL2>o<r^>F0HE5Q^5i!4B(`^$!ry8z=k-oH=b>~^AA<73yIRAseW;ydgHZMd? z<%|NiD17@1#VoIHF1lDaV0JN`f-l;FMQ)vKzv@!t2Fr9&OIH3$o7jg1aZtj8N?~hI z0-)I6*IJEu6C5>uu821yx((ce%-;YW9&{zS0igm4X1H>PLC+bB7PY3uty0y}uE;eN z>XP7|8@1f&f0_i)unB%EK>h7^Zs|i{Ix4zaP{d9d9-)s9m5B*UdliXWkk8}nE(j8Y zS&7e-zrM~7Uo(y$@nk&Io-}98v&>bl{`>>Sd3Rg{VC%0<5R?iv(qQdCESk9~2}FVe zUXZ{}RQGIZagn-x#|SN9w&<gLGEldu=iXC{!p4Zq*Hm&2w$7PwXzw5N373Lbb}J(} zX|5Cj7rNW*<onPARj-sqOfWJD8O8u7Q>@JY{6E+xNuBKw<<M-OVPgUD0kC1DG_pX> zqVgu44e&`r5Uo!ue_-Vk@!4|>Kz0YP(irXM7u1S2q;N0^;fbDA3F>LUsDc&;KWK$X zZvpo(6^o!}+1+j&Z^wXY*(Dz-)SQxx$m}Eho~bIpbnCMz{X%gCIPEYP@)JubOyHmG zQCnGb7LP)N!G^c45-E#{I<(S?5bz#E2jnXi#IpAK-`iCHXCo!OJf5D*nRTA}Lv3GP zG=wWY7fwxyXy5Mvf*0gPjVtr=hPL2FS|L-`FWNiOlX#bd%m6gMHOT2!X#0*nAYF*u zq#uLg-4N{EbiFZ?3HI_z-+!ucVtL;HyKq6t4*Ivs^ac2H+MdRQtT#lYK;ec1Q2uvW zx7dr}X$~7~1bRhce50MbEC?g5dv_pJ%pW#T=(#YA+hQ$*##!g5g~|ovvYohIL%dRX zW6f*%4}LgKyLdC7pV$z4Qa6R0Y6KwO_yxJjQiXO57=Z!aojOHm$()#~Kc&4TjxsT* zjmT5%(tZHOH*|_+SP(~x;E1w$dGSK>aJn|Ugpr)5m|Pt$7Xro?V2SfZX=v+?vB$rz zfpjWdR0h<g(1`Kr`z-WUnd#-cb)ihrq5*2T_<cgxV_sFZt)ei%W?eF2;=n+O;My2o z9|6eH8=>!ulX?KuW^M7-YCuWDy*?V}nzwIeQ1`p<<f~vmrB(+AvVp{Az47E)vlY%& zsE^c2$<QY0G+B%5Z+<Bj_Q;`!f@}K%_L%TcJKQbO49IrY%~5yg`?~CM{s{Q-d|xQG z`@ZeBEOENYF@miFj+g1yV=$snFo$)8gHrgB*5a9c_lbrJL$A!nOQg;2#bt=4W7Wmm zr88ftuaH4pK#D*m;*j)3PgagDBi;2?{_rtpO=Kf);qE0@cLao}49EkNzZI?fC>zU~ z+Bh~G;y{|)KF2dZoUu-kq8$k^wfOR=Z*p0p_DbYzksecfEkiu2Juds>OOF$fp@}I` zPvlB(@&FpVpoXMRC|~{xHeOIZt#4A)c2W_p(%E+FQxBq##@diE)@iD_H9Iq~$X=Ku z7mjE;MsFP*5$Xop^%iva%Zm}w3i8KrlA8T>Hp8}O!Kx9G70+&V95&j{FE$Z5+b&$r zzK#716uj&S8#Xa?bUuV$hJfVQo=jybkp21Qn;ccjcId}Iahg?qD`JfP4o%S_js0hF zO=wjTL=Ypbl2l|If>$eTp}36(LM-P+f9IDDRNm?|8gn45Iyfr9q8=O|p88LV%Zy4< zh*o@Ros6;;JJjfh=V0j*F)55JVdkc-m2QVfNwhvN#N+UfSYy7ds*OZnoEr>c{40== zvFJPkubX1Jkt5P6rzajU_WEVeoGf2Wg>jf}W)ZbyEh_E_;r_?rOlLAh6_}A()Ld7g zAV)`thWS!bji2c{Lha{dM3{C^-9Aw+JI!n;m~c+1pC5BQ!XbO6Z)355oF|+!`$!f+ z_hHOLblUid5=D)P>Nh%A&|M#Q$wRs-Y$|OT<f|w&RBJxeTI3-w(|5wo$wzobFjYP4 znPiY?1Mh`(3ZWP^b~k2t`9x33&0TM?qkr%kCV<hc_ME*sq&9Up1_LwGa^im(`)i|Y z=Ye724Zl!{-&L{06kQT5`i!3Dr&KvoH)#G=J70+I!yYy%Q{*G|sS{NqjgtS2P3E;} zs9z|u+2%L_%;NdtDdD|-!nh!OiLcq)t8I1bzr}NJEf~AiFOc@`y)E)}c#?=0bZFj_ zE@Esf`>+{#H0TI@|Cec<y|zJyd-TPa6WZur>{>a7dcArITx%H5<MB3gqJ}z657e7- zX!GHlxz^_q>9)LHSa-o1d(DPeZ$_0jZOoWCZ61K$I|{DrJ~D<atZH5DuDwUb7d{QE z05=1tle+}TR2uqd1Cf`P4Ptvs1Rc>FpS_TI>}EW&>;uoz6_7iIVvgJG&8wMF|2*#I zqC4xZriHgW_OdO>Ze+uIW|B<jlT*nytu4?q>DybRI$66~&?=baG~Ls*DKuES@%2+{ zLly6*4zZSzYE(v954s}-$OQONuj=$Ew$!?r|9Wb5o5#wn720%OjojcCwOA9KQKB}Z zKF?#B`%60mV!Y&b<*P^qWK85jk2sP_J~=oh$ZzB%qWx@2duXt@N0r&J`c^qW3^8X@ zy>XN0A$}2<?J#4gSuR+YI>zoq{oV&&od%R|GIUkslZGdHs)@4SW67og^m_n6zWa)8 zffz&PD$kR8r65U@?CQ{f!&<bj2CAytdyiXEOBmntI6T3Z?jZ4PgGMc?MM>l#gZus_ z$S+N@x^ouARps4uf`uEVa&NUj!4)s7OEe9x{(5^eGuIO&#;>93+L<Ps8C3WudB;8+ zt~lbdyA8k;6o}CPNQ@Df$<3nM$laBmGk5v>JrGYxe}3_E=TPIauap_S379Y6>C;_* z#>Jx7LXlg_1KvpBUm(`w_2(Klv~1H)5vM*Uqw6}_x#`@Hx82+s2Fv8)IW1?TdBVhn zRJMdLKn91S1xsBtDA>KfUuz00$Nd9kL0P$?kEA8)TE0^a&#JMmZFy=_3=w*`%3<oK zI;**~rAoo_!G3&3wy)hRbHN)eA`NT6R>yO)$5z{B`xM5;BU5G@T48P#@iNRS&GgKq zx$yFzzQrq{WW*OadT*4MB>y}$!>m$beJ!FS+Ha==)9-=b_&XV_6PcRU>n>A`czXE2 zu$+Y=yP-C3e~A}hDO-3yCcXNORL_pIxd<DtS<3>Fu7(>)qWE}^)qG39D~Q`J2BKfD z?$VOIShzjaqCTd_4~jk{TiTesJS&C`XPCM++QYj>c=>_0LXSg6s;knM9Nb`Q!GJoG z6_w+s$W@t9cYq!1bi5XSh&0sn_43<v86RzO`)_90iR1V6UyNR&YbhxESv)Eh0;Zz$ z1aG~0BgbVO21Nk@4gT`ifpXwktIgrgdFWV1xv_9LLmX($O_82|8G29u_dPmpuETnc z4O}0_r88VZ+is_JBAXyHwOjF1=G{+t?w<JB#3_?D5yIRF5S_oom&<{cCv_{|`Wfuq zE&nx#y_i)ic-buN=b$BqKLS3fPXe;aS+`~~yvnw$!^}i0NrR)Uqevy*Il?eQo&XSk zu>K~y*W(BTC!c-^4y=8@CGZ-dlB!1StuI$yjnsfn6T)jVCJ%`})PJcTISF8+w<nAn z4z^8-n#c}L4dgcGZ~?Ys&~Tp=bn9`N+Lb)&G=(E!d~rD-?f_q{@)L#^qw=~=5REHL z;@Lm%SSSnTuJh7lMI(bsQs&{<q6d%0>?t)cBT9JHzUe>in1-&V=HRa*U=a0_nTEgf z_q{TZr6jp3sPc0=rx+a;G02wuLB*abLXxj&cRQATReZW1SPtU{p40)<0dL0~uIGpO zi%?%96myc*>A?uaD0|Zl+Iqoo;AejuS3Ple%NBA`<oesnmw#f7)eb3la$x$?7PGb; zQ2pYyt;o13Yi!P2)Z`v02FWtFQ^Ehsj+@(~1Ex<Pv>rz#0bO>yx_)s)O%PtuPi<vv zs?m;@k%(!3X!in8Y8Y_P?r|b_OQvt6E6l)Uu05lFtfUe^v>)}WVDj2W@r}IUef=Qo z08%E3TN!zUSVVaJ4o*>`M0HYKq_|$xt2jpHm=DPG=wg~I>ds*#(pX58a!;tgBocTf zm<9>as`GS>I~>V{H+S^tuhGaFTYqc}VRO#r$bX)BL1C-g8%>dV!wFBRMW_*IFwI1b zX}F!}bto*s{(3U^n!Dq~raOgEJ;$u+Lvi}$*eZB3+z)#WgvX}H5z_)m;>Mjk^J;a2 zk?rVa#DK#hF|uVc-!yzU&S#;t=if~J=CNoz2P4|zK7<SvVM@=?MQt1^A7g+&rf6^u z3P{}bv{gJ`nF$EOc77I*P9>{L?2?I(Uv`l5vVuFS0~EIZO*Yi&RoBGQ8o?SNDN;$r zsBE9=84;uvYlRdjQtEiGk9Wy)o&*$r0Y06QY@da8MOD>@rbWb(jbo(OoqyPmc$mZn zw|2`0D>$gt8=EHs@laCY`~|^@ER-R1hpOx1b{4bgEw0BMzHdxlqh{{wFCN@`K}$0& z``W{@XfKzN`Zf(EnBqRh1M4hM+?^IQOp5fW>*`w&?(X>!2KGEaYP)6!IfI=+2qsWS zRtPJhak{?os5z2Wy52Gy4oH}00@M~x3P`O698RK%;w5qs#i*h)6`S42=?9EIm!Vr9 zyhX!JjDI*|Hb;ioRA%D^Z0*61!^QLreE5hpCm|~vM@P=?23;=_D~?xxoUaDSqvB@X z#P*9r(;7);zE#s?4?8k;uif70QmIcv_Hewl7<5r?IHEB>5(uKdIs~%g1*EAE?P^t! z2b#rSM27BNpasn?Uxu1bMQilrX$y>B3R0-uAfn>7#BA#BnODx!3PD@ug)J~(qRsnT zJh%%R16CN3@&WtmMp<SJ+P*YPh#b)T%I@Zgqnr>6|31n6+Cf&<uAJURU~~7cF2tAz zrUsZit4aw2-f{SxNGI%m{IdeG_gB_^R-Fnm{&u9TGaIrZ^wp(z<|c*8T*W7%ePFQc zjxQ&Xz+1TPV3Q~UeqIgWES*$aVuQJ81~YR1M_FY3TpW&!VcE}m2CA>3y7^=+_<U(R zh5yhFSlf`&74;ygd~#7fgkH6y)BYGs|B<n9iCmq4ET_K4hY`1KGs_MdIM#sms20u5 zxKKhZ#~11>l_agEWJKtXsDZA?u@&Vb^2ZQ^A-t1_s3%!GJMu!Z_x2@M9g){`(#B~; z%e}pl$T7AkT$JMJZ=&@Z)Nqm+5_sJSx=P;LStc@$WglmB?ibE(Ixz9AR&9FcCtF#D zNA<qP(2DkYU<m`uZko-NQy^M{Wn$X9d<v`fPM7D-%PcR-^kqy$?r9ZfMI@nqX0p@A zbsmzT-sq{OVmP*)puUbce4}_hQT9?5kuUVHp%x}BjnC--N}%_^<wf<1S$wNZ%gEUH zNSz(4cg3+wL-*(N(43eP_{$D73O?owMTBeT>>!9OzjYOS9OOE~+70OjOpo+qS%-QT z{k+Yz@K}9V(%4`n4W(2MoD0;Yp)UZlG|ne0T|1dSYw?`79FEZ>ttfxBPM6&_T$<#b zPJ|eZs&vjM69u(ug-vCh4E929b)uukJ^0Dz9N+lJ-(jA&Rse|AT^`*>@}*StH(g;3 zr>7es!4hod1wD7Mn3(6|y=H;;>GUH!ko~4HCb*#yJaow3l{36f<>Ec0hP+gJ$x1WI zIh06VK!9@7c}Y%Kq7+MByEMnR^MT32eH6>o^~yQev{^JJ7<Wr7HS$m$H`85%)v6-} zUQV!{>)F$buYb+vvx?5`Cq3V7`)P8teP!H%Rb(zrHtN;!s<u}pDvACV=k8GiJ$s9* zL=aGL^)E$n*$}yB1Sa^3l;HvOe@z$(=(Oeitx<N$L0d#{#^lNLu4`pUgZWtO%*_4G zfABS-LGl`)tne&#)2Tz<S<vV#Ltze`M%U~J5UQ+UHLX8*OA)K3k>k&99tE7IunCzB z$Ou&L(6&N2sS}kmt{6_1)^|1%{1-T7futf_rg_BfY9sgfOUti=97e4gL>gGOE>E@l z03qP9K!l<te#eNKhhKqMHlVo3yTN%M1e&p3PaweqDLgbgME03`EFB~#S{JwmLT9f| z0HKI2v_`&}-Fg8jyiV6o)X>6@_MG@d5ya5xx`c?GWk!n3#o*H`v<d8tLyua}Dyv<M zBO#5CC^s^qP>$ckZHq8#$HmNrjgzziBsxdn&0$ZT%O=oKfUp?oHs9foJfcQ@ZgQ*d zZ)<Y5><w3og;_klv9-PV(IS>{shtNqQfh`+ZcY4ff1gg&1S*YNUZ)zYr!^wTNpm?! z5>W2_IQ-wKh$o>LVJS!0Rv2Re7@++7k8&GHQpkGdw6C5ike=8?GzbPP?S%62`~w&? zq|{x0v5@~MDP%7-DyeYG^OQco=gLX2rL}J)SmdWI4i;xcbCJI}JXtqs8atp-dmDz* zE<-^<t$|y-2InA|khw)o$5aXjydXN9#aWQ}lO4g<brW4Y{UJ|yK&@jDfjGH~Yg$8} zN26Z`6O`MnTboA-K|1MSQIs7ls;5n!F7#sQ>e<JSKY@ZSgk5=Z%G{&O1FZ!Uz)WtS za2fj;@6As0mL(AKJ)@rPnx_e342RTXO!mX)ULkCMk}<hC^iYbyv6FiaJXs)kA7Ud{ z3w$5Ovj?Qtk`sLf;GUz!C0LYwP@y1LAkpFpdDkxWKk&X_$Z;BATb?vc)^x@1Hc`Wl z6rAeX2SJP4{0&eo(dgYawr2ajW9qHN%L2}(N?QI=C)2AYxV?WrC|f$a8I7xs$O#%& zNpjBgJ9fnm>$%QUN*j5F>ymnMy2zVnaulCyLbZV&7%1XIip<(yCpM!#Pz8c(NyJy@ z-Nq6@FOF}zcIhs}&k9h_GG@qJ>R7C%4P4AB=&qUXc8dp9iP*$mAU*}cIUc;LaV#+M zu-ZFa`4)3>ynUx$er6D$^=qj&W{*^&<s+{syMYb&5M^ciLi-W%geP#W&w#L?2lLF} z;}>ZQhLxxXKN^4*$>nIBB5H{MlmL>sP{{3b-2_gHxi{?M=yro)RtFetW6B9yN0l1@ zVx(-grHWef8%nw|+SI5yGgr=kQ&G;t()~>ab5wD^Q3xoL3P0@eU8fIsw^k;s+k?2v z@;zgA9#s+E1!kpJ#<H;4qreEC7`~;jEw100#cSDlHP#RL6<mspPl!a8kjYRNV_WI} zt>~&oM&_3ku?*4p-aF^fUd^$lJu~__Cpy1z*2*4-au!P0)!-60)w`2R#0d)LScsEN z<q*To=^imZ@W|AmXorfdN~OKZY@#d8)6{p^b*NcxsEs;QfiEhQwp8ULZH!VW>go}u zW6#j{V(#@nmNzLSN@ppVrx6P!D$cVKJ9f14pBU}8!Z=Of-N$5}1)=rO;t%#>t=Cpl z>wG+pzl?qtb9)h2UBA6sKy%(Ec5`p9551nayB{3%8z(v$lP|R>-KqmNBY%mCEtt(A zt@l6Erf2i1zf{34>gmTjX)JC#@kw$Rjg${~H<Tm6<ZTp~NG<^puM3`>p7syN9DV(7 zGm6tBihrImVvd|NCK#|fAW*|7idJkv+<!E<>^=y`#VuS7;)ucXXJ3%eO{kv22o#s% zzbVJf68Ir`LkD{_Yn)@j(P0o)!MPGmuE+ZmxU-@1AffOcP`lG-Vg-RNag+rOG0lQ> zwSUJu=)n(N%R=NA(`TNEPrKX!jxX-|8U=NSq5uE@OTqV1mWZUo00GvS0f6w%0RR91 TZqU^+6c;-%0{{R300dcD09`$< -- 2.49.0 ^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [PATCHv2 4/4] test: extend sysfs tree dump test 2025-04-17 13:59 ` [PATCHv2 4/4] test: extend sysfs tree dump test Nilay Shroff @ 2025-04-22 6:28 ` Hannes Reinecke 2025-04-22 10:09 ` Daniel Wagner 0 siblings, 1 reply; 14+ messages in thread From: Hannes Reinecke @ 2025-04-22 6:28 UTC (permalink / raw) To: Nilay Shroff, linux-nvme; +Cc: dwagner, hare, kbusch, gjoyce On 4/17/25 15:59, Nilay Shroff wrote: > In order to validate multipath link support extend the existing sysfs > tree dump test. > > The updated tree-pcie.tar.xz archive includes a sysfs dump with two > NVMe subsystems. The nvme-subsys0 simulates a single-port NVMe disk > with a private namespace that does not support multipath. And another > nvme-subsys1 simulates a dual-port NVMe disk with a shared namespace > attached to multiple controllers. > > The json tree dump is also updated to include NVMe namespace path > information. Since the archive content has changed, the expected > output file tree-pcie.out is also updated to reflect the new structure. > > Signed-off-by: Nilay Shroff <nilay@linux.ibm.com> > --- > src/nvme/json.c | 90 +++++++++++++++++++++++++++---- > test/sysfs/data/tree-pcie.out | 53 ++++++++++++++---- > test/sysfs/data/tree-pcie.tar.xz | Bin 19712 -> 12656 bytes > 3 files changed, 124 insertions(+), 19 deletions(-) > Not a big fan of having tar.gz files in git. Can't we create it on the fly and only store text files? Otherwise: Reviewed-by: Hannes Reinecke <hare@suse.de> Cheers, Hannes -- Dr. Hannes Reinecke Kernel Storage Architect hare@suse.de +49 911 74053 688 SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCHv2 4/4] test: extend sysfs tree dump test 2025-04-22 6:28 ` Hannes Reinecke @ 2025-04-22 10:09 ` Daniel Wagner 0 siblings, 0 replies; 14+ messages in thread From: Daniel Wagner @ 2025-04-22 10:09 UTC (permalink / raw) To: Hannes Reinecke; +Cc: Nilay Shroff, linux-nvme, hare, kbusch, gjoyce On Tue, Apr 22, 2025 at 08:28:48AM +0200, Hannes Reinecke wrote: > Not a big fan of having tar.gz files in git. Can't we create it on the > fly and only store text files? Creating is not really an option, we could just store test data in the tree directly. I don't expect that the size of the repo is increasing a lot as the objects in git are also compressed. My main concern is that we have a lot of new files: $ tar tf config-pcie.tar.xz sys/class/nvme/ sys/class/nvme/nvme1 sys/class/nvme/nvme0 sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/ sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/uevent sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/cntlid sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/address sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/ sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/uevent sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/ext_range sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/metadata_bytes sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/range sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/alignment_offset sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/diskseq sys/devices/pci0000:00/0000:00:06.0/0000:0f:00.0/nvme/nvme0/nvme0c0n1/power/ [...] Currently there are three tar.xz files. If the general mood is that there shouldn't be any tar.xz files around, I am fine to get rid of them. ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2025-04-23 6:53 UTC | newest] Thread overview: 14+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-04-17 13:59 [PATCHv2 0/4] libnvme: add support for discovering multipath of a shared ns Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 1/4] tree: add support for discovering nvme paths using sysfs multipath link Nilay Shroff 2025-04-22 6:24 ` Hannes Reinecke 2025-04-22 14:01 ` Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 2/4] tree: add queue-depth attribute for nvme path object Nilay Shroff 2025-04-22 6:26 ` Hannes Reinecke 2025-04-22 14:32 ` Nilay Shroff 2025-04-22 17:23 ` Daniel Wagner 2025-04-23 6:13 ` Nilay Shroff 2025-04-17 13:59 ` [PATCHv2 3/4] tree: add attribute numa_nodes for NVMe " Nilay Shroff 2025-04-22 6:27 ` Hannes Reinecke 2025-04-17 13:59 ` [PATCHv2 4/4] test: extend sysfs tree dump test Nilay Shroff 2025-04-22 6:28 ` Hannes Reinecke 2025-04-22 10:09 ` Daniel Wagner
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox