From: "Steve Wise" <swise@opengridcomputing.com>
To: "'Leon Romanovsky'" <leon@kernel.org>
Cc: <dsahern@gmail.com>, <stephen@networkplumber.org>,
<netdev@vger.kernel.org>, <linux-rdma@vger.kernel.org>
Subject: RE: [PATCH RFC iproute-next 4/5] rdma: Add MR resource tracking information
Date: Mon, 26 Feb 2018 09:08:58 -0600 [thread overview]
Message-ID: <016501d3af13$ba078210$2e168630$@opengridcomputing.com> (raw)
In-Reply-To: <20180220141223.GH7709@mtr-leonro.local>
> -----Original Message-----
> From: Leon Romanovsky [mailto:leon@kernel.org]
> Sent: Tuesday, February 20, 2018 8:12 AM
> To: Steve Wise <swise@opengridcomputing.com>
> Cc: dsahern@gmail.com; stephen@networkplumber.org;
> netdev@vger.kernel.org; linux-rdma@vger.kernel.org
> Subject: Re: [PATCH RFC iproute-next 4/5] rdma: Add MR resource tracking
> information
>
> On Wed, Feb 14, 2018 at 01:07:01PM -0800, Steve Wise wrote:
> > Sample output:
> >
> > # rdma resource show mr
> > link cxgb4_0/- rkey 0x5e0e lkey 0x5e0e iova 0x7f9b60001c80 mrlen 64
> pgsize 4096 pid 30503 comm rping
> > link cxgb4_0/- rkey 0x510d lkey 0x510d iova 0x17c7ee0 mrlen 16 pgsize
> 4096 pid 30503 comm rping
> > link cxgb4_0/- rkey 0x4a0c lkey 0x4a0c iova 0x17c7e38 mrlen 16 pgsize
4096
> pid 30503 comm rping
> > link cxgb4_0/- rkey 0x480b lkey 0x480b iova 0x224b3f0 mrlen 64 pgsize
> 4096 pid 30498 comm rping
> > link cxgb4_0/- rkey 0x460a lkey 0x460a iova 0x224b350 mrlen 64 pgsize
> 4096 pid 30498 comm rping
> > link cxgb4_0/- rkey 0x4509 lkey 0x4509 iova 0x2245890 mrlen 16 pgsize
> 4096 pid 30498 comm rping
> > link cxgb4_0/- rkey 0x4208 lkey 0x4208 iova 0x22457e8 mrlen 16 pgsize
> 4096 pid 30498 comm rping
> > link mlx4_0/- rkey 0x38010700 lkey 0x38010700 iova 0x7f9b5c002f90 mrlen
> 64 pgsize 4096 pid 30494 comm rping
> > link mlx4_0/- rkey 0x38010600 lkey 0x38010600 iova 0x17c6c80 mrlen 16
> pgsize 4096 pid 30494 comm rping
> > link mlx4_0/- rkey 0x38010500 lkey 0x38010500 iova 0x17c6bd8 mrlen 16
> pgsize 4096 pid 30494 comm rping
> > link mlx4_0/- rkey 0x38010400 lkey 0x38010400 iova 0x1b68430 mrlen 64
> pgsize 4096 pid 30489 comm rping
> > link mlx4_0/- rkey 0x38010300 lkey 0x38010300 iova 0x1b683a0 mrlen 64
> pgsize 4096 pid 30489 comm rping
> > link mlx4_0/- rkey 0x38010200 lkey 0x38010200 iova 0x1b62890 mrlen 16
> pgsize 4096 pid 30489 comm rping
> > link mlx4_0/- rkey 0x38010100 lkey 0x38010100 iova 0x1b627e8 mrlen 16
> pgsize 4096 pid 30489 comm rping
> >
> > Signed-off-by: Steve Wise <swise@opengridcomputing.com>
> > ---
> > include/json_writer.h | 2 +
> > lib/json_writer.c | 11 +++++
> > rdma/res.c | 126
> ++++++++++++++++++++++++++++++++++++++++++++++++++
> > rdma/utils.c | 7 +++
> > 4 files changed, 146 insertions(+)
> >
> > diff --git a/include/json_writer.h b/include/json_writer.h
> > index 1516aaf..34f2ccc 100644
> > --- a/include/json_writer.h
> > +++ b/include/json_writer.h
> > @@ -39,6 +39,7 @@ void jsonw_bool(json_writer_t *self, bool value);
> > void jsonw_float(json_writer_t *self, double number);
> > void jsonw_float_fmt(json_writer_t *self, const char *fmt, double num);
> > void jsonw_uint(json_writer_t *self, uint64_t number);
> > +void jsonw_xint(json_writer_t *self, uint64_t number);
> > void jsonw_hu(json_writer_t *self, unsigned short number);
> > void jsonw_int(json_writer_t *self, int64_t number);
> > void jsonw_null(json_writer_t *self);
> > @@ -49,6 +50,7 @@ void jsonw_string_field(json_writer_t *self, const
char
> *prop, const char *val);
> > void jsonw_bool_field(json_writer_t *self, const char *prop, bool
value);
> > void jsonw_float_field(json_writer_t *self, const char *prop, double
num);
> > void jsonw_uint_field(json_writer_t *self, const char *prop, uint64_t
> num);
> > +void jsonw_xint_field(json_writer_t *self, const char *prop, uint64_t
> num);
> > void jsonw_hu_field(json_writer_t *self, const char *prop, unsigned
short
> num);
> > void jsonw_int_field(json_writer_t *self, const char *prop, int64_t
num);
> > void jsonw_null_field(json_writer_t *self, const char *prop);
> > diff --git a/lib/json_writer.c b/lib/json_writer.c
> > index f3eeaf7..6d73a1b 100644
> > --- a/lib/json_writer.c
> > +++ b/lib/json_writer.c
> > @@ -224,6 +224,11 @@ void jsonw_uint(json_writer_t *self, uint64_t
> num)
> > jsonw_printf(self, "%"PRIu64, num);
> > }
> >
> > +void jsonw_xint(json_writer_t *self, uint64_t num)
> > +{
> > + jsonw_printf(self, "%"PRIx64, num);
> > +}
> > +
> > void jsonw_lluint(json_writer_t *self, unsigned long long int num)
> > {
> > jsonw_printf(self, "%llu", num);
> > @@ -268,6 +273,12 @@ void jsonw_uint_field(json_writer_t *self, const
> char *prop, uint64_t num)
> > jsonw_uint(self, num);
> > }
> >
> > +void jsonw_xint_field(json_writer_t *self, const char *prop, uint64_t
num)
> > +{
> > + jsonw_name(self, prop);
> > + jsonw_xint(self, num);
> > +}
> > +
> > void jsonw_hu_field(json_writer_t *self, const char *prop, unsigned
short
> num)
> > {
> > jsonw_name(self, prop);
> > diff --git a/rdma/res.c b/rdma/res.c
> > index 27c1efd..2b67d25 100644
> > --- a/rdma/res.c
> > +++ b/rdma/res.c
> > @@ -819,6 +819,119 @@ static int res_cq_parse_cb(const struct nlmsghdr
> *nlh, void *data)
> > return MNL_CB_OK;
> > }
> >
> > +static void print_key(struct rd *rd, const char *name, uint32_t val)
> > +{
> > + if (rd->json_output)
> > + jsonw_xint_field(rd->jw, name, val);
> > + else
> > + pr_out("%s 0x%x ", name, val);
> > +}
> > +
> > +static void print_iova(struct rd *rd, uint64_t val)
> > +{
> > + if (rd->json_output)
> > + jsonw_xint_field(rd->jw, "iova", val);
> > + else
> > + pr_out("iova 0x%" PRIx64 " ", val);
> > +}
> > +
> > +static void print_mrlen(struct rd *rd, uint64_t val)
> > +{
> > + if (rd->json_output)
> > + jsonw_uint_field(rd->jw, "mrlen", val);
> > + else
> > + pr_out("mrlen %" PRIu64 " ", val);
> > +}
> > +
> > +static void print_pgsize(struct rd *rd, uint32_t val)
> > +{
> > + if (rd->json_output)
> > + jsonw_uint_field(rd->jw, "pgsize", val);
> > + else
> > + pr_out("pgsize %u ", val);
> > +}
> > +
> > +static int res_mr_parse_cb(const struct nlmsghdr *nlh, void *data)
> > +{
> > + struct nlattr *tb[RDMA_NLDEV_ATTR_MAX] = {};
> > + struct nlattr *nla_table, *nla_entry;
> > + struct rd *rd = data;
> > + const char *name;
> > + uint32_t idx;
> > +
> > + mnl_attr_parse(nlh, 0, rd_attr_cb, tb);
> > + if (!tb[RDMA_NLDEV_ATTR_DEV_INDEX] ||
> > + !tb[RDMA_NLDEV_ATTR_DEV_NAME] ||
> > + !tb[RDMA_NLDEV_ATTR_RES_MR])
> > + return MNL_CB_ERROR;
> > +
> > + name = mnl_attr_get_str(tb[RDMA_NLDEV_ATTR_DEV_NAME]);
> > + idx = mnl_attr_get_u32(tb[RDMA_NLDEV_ATTR_DEV_INDEX]);
> > + nla_table = tb[RDMA_NLDEV_ATTR_RES_MR];
> > +
> > + mnl_attr_for_each_nested(nla_entry, nla_table) {
> > + struct nlattr *nla_line[RDMA_NLDEV_ATTR_MAX] = {};
> > + uint32_t rkey, lkey, pgsize;
> > + uint64_t iova, mrlen;
> > + char *comm = NULL;
> > + uint32_t pid = 0;
> > + int err;
> > +
> > + err = mnl_attr_parse_nested(nla_entry, rd_attr_cb,
nla_line);
> > + if (err != MNL_CB_OK)
> > + return MNL_CB_ERROR;
> > +
> > + if (!nla_line[RDMA_NLDEV_ATTR_RES_RKEY] ||
> > + !nla_line[RDMA_NLDEV_ATTR_RES_LKEY] ||
> > + !nla_line[RDMA_NLDEV_ATTR_RES_IOVA] ||
> > + !nla_line[RDMA_NLDEV_ATTR_RES_MRLEN] ||
> > + !nla_line[RDMA_NLDEV_ATTR_RES_PGSIZE] ||
> > + (!nla_line[RDMA_NLDEV_ATTR_RES_PID] &&
> > + !nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME])) {
> > + return MNL_CB_ERROR;
> > + }
>
> I'm not sure that all this ATTR are needed.
>
Yea you're right. I'll revisit all these and decide which minimal ones are
really required.
> > +
> > + rkey =
> mnl_attr_get_u32(nla_line[RDMA_NLDEV_ATTR_RES_RKEY]);
> > + lkey =
> mnl_attr_get_u32(nla_line[RDMA_NLDEV_ATTR_RES_LKEY]);
> > + iova =
> mnl_attr_get_u64(nla_line[RDMA_NLDEV_ATTR_RES_IOVA]);
> > + mrlen =
> mnl_attr_get_u64(nla_line[RDMA_NLDEV_ATTR_RES_MRLEN]);
> > + pgsize =
> mnl_attr_get_u32(nla_line[RDMA_NLDEV_ATTR_RES_PGSIZE]);
> > +
> > + if (nla_line[RDMA_NLDEV_ATTR_RES_PID]) {
> > + pid =
> mnl_attr_get_u32(nla_line[RDMA_NLDEV_ATTR_RES_PID]);
> > + comm = get_task_name(pid);
> > + }
> > +
> > + if (rd_check_is_filtered(rd, "pid", pid))
>
> free(comm);
>
yup
> > + continue;
> > +
> > + if (nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME])
> > + /* discard const from mnl_attr_get_str */
> > + comm = (char
> *)mnl_attr_get_str(nla_line[RDMA_NLDEV_ATTR_RES_KERN_NAME]);
> > +
> > + if (rd->json_output)
> > + jsonw_start_array(rd->jw);
> > +
> > + print_link(rd, idx, name, 0, nla_line);
> > + print_key(rd, "rkey", rkey);
> > + print_key(rd, "lkey", lkey);
> > + print_iova(rd, iova);
> > + print_mrlen(rd, mrlen);
> > + print_pgsize(rd, pgsize);
> > + print_pid(rd, pid);
> > + print_comm(rd, comm, nla_line);
> > +
> > + if (nla_line[RDMA_NLDEV_ATTR_RES_PID])
> > + free(comm);
> > +
> > + if (rd->json_output)
> > + jsonw_end_array(rd->jw);
> > + else
> > + pr_out("\n");
> > + }
> > + return MNL_CB_OK;
> > +}
> > +
> > RES_FUNC(res_no_args, RDMA_NLDEV_CMD_RES_GET, NULL, true);
> >
> > static const struct
> > @@ -880,6 +993,18 @@ filters
> cq_valid_filters[MAX_NUMBER_OF_FILTERS] = {{ .name = "link",
> >
> > RES_FUNC(res_cq, RDMA_NLDEV_CMD_RES_CQ_GET, cq_valid_filters,
> true);
> >
> > +static const struct
> > +filters mr_valid_filters[MAX_NUMBER_OF_FILTERS] = {{ .name = "link",
> > + .is_number = false },
> > + { .name = "rkey",
> > + .is_number = true },
> > + { .name = "lkey",
> > + .is_number = true },
> > + { .name = "pid",
> > + .is_number = true }};
> > +
> > +RES_FUNC(res_mr, RDMA_NLDEV_CMD_RES_MR_GET, mr_valid_filters,
> true);
> > +
> > static int res_show(struct rd *rd)
> > {
> > const struct rd_cmd cmds[] = {
> > @@ -887,6 +1012,7 @@ static int res_show(struct rd *rd)
> > { "qp", res_qp },
> > { "cm_id", res_cm_id },
> > { "cq", res_cq },
> > + { "mr", res_mr },
> > { 0 }
> > };
> >
> > diff --git a/rdma/utils.c b/rdma/utils.c
> > index 11b34fe..34b195a 100644
> > --- a/rdma/utils.c
> > +++ b/rdma/utils.c
> > @@ -392,6 +392,13 @@ static const enum mnl_attr_data_type
> nldev_policy[RDMA_NLDEV_ATTR_MAX] = {
> > [RDMA_NLDEV_ATTR_RES_CQE] = MNL_TYPE_U32,
> > [RDMA_NLDEV_ATTR_RES_USECNT] = MNL_TYPE_U64,
> > [RDMA_NLDEV_ATTR_RES_POLL_CTX] = MNL_TYPE_U8,
> > + [RDMA_NLDEV_ATTR_RES_MR] = MNL_TYPE_NESTED,
> > + [RDMA_NLDEV_ATTR_RES_MR_ENTRY] = MNL_TYPE_NESTED,
> > + [RDMA_NLDEV_ATTR_RES_RKEY] = MNL_TYPE_U32,
> > + [RDMA_NLDEV_ATTR_RES_LKEY] = MNL_TYPE_U32,
> > + [RDMA_NLDEV_ATTR_RES_IOVA] = MNL_TYPE_U64,
> > + [RDMA_NLDEV_ATTR_RES_MRLEN] = MNL_TYPE_U64,
> > + [RDMA_NLDEV_ATTR_RES_PGSIZE] = MNL_TYPE_U32,
> > };
> >
> > int rd_attr_cb(const struct nlattr *attr, void *data)
> > --
> > 1.8.3.1
> >
next prev parent reply other threads:[~2018-02-26 15:27 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-02-19 20:10 [PATCH RFC iproute-next 0/5] cm_id, cq, mr, and pd resource tracking Steve Wise
2018-02-14 21:05 ` [PATCH RFC iproute-next 1/5] rdma: update rdma_netlink.h Steve Wise
2018-02-14 21:07 ` [PATCH RFC iproute-next 5/5] rdma: Add PD resource tracking information Steve Wise
2018-02-23 14:22 ` Leon Romanovsky
2018-02-26 15:09 ` Steve Wise
2018-02-27 0:47 ` Steve Wise
2018-02-14 21:07 ` [PATCH RFC iproute-next 4/5] rdma: Add MR " Steve Wise
2018-02-20 14:12 ` Leon Romanovsky
2018-02-26 15:08 ` Steve Wise [this message]
2018-02-14 21:07 ` [PATCH RFC iproute-next 2/5] rdma: Add CM_ID " Steve Wise
2018-02-20 12:57 ` Leon Romanovsky
2018-02-20 15:15 ` Parav Pandit
2018-02-26 15:05 ` Steve Wise
2018-02-14 21:07 ` [PATCH RFC iproute-next 3/5] rdma: Add CQ " Steve Wise
2018-02-20 13:09 ` Leon Romanovsky
2018-02-26 15:06 ` Steve Wise
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='016501d3af13$ba078210$2e168630$@opengridcomputing.com' \
--to=swise@opengridcomputing.com \
--cc=dsahern@gmail.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=stephen@networkplumber.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.