Is it ok to use debugfs to dump some ucontext-level driver-defined info?

Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[liweihang]

Hi all,

We want to dump some hns driver-defined information that belongs to a
process to keep track of current memory usage. For example, there is
a ucontext-level(process-level) memory pool to store WQE which is
shared by a lot of QPs, we want to record and query which QPs are using
this pool and how much space each QP is using.

rdmatool don't have a ucontext-level resource tracking currently, is it
ok to achieve that through debugfs?

This may looks like:

$ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
110        6400       256        2048       4096
118        6400       256        2048       0

Or should it be achieved in rdmatool?

Thanks
Weihang

Re: Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[Leon Romanovsky]

On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:
> Hi all,
>
> We want to dump some hns driver-defined information that belongs to a
> process to keep track of current memory usage. For example, there is
> a ucontext-level(process-level) memory pool to store WQE which is
> shared by a lot of QPs, we want to record and query which QPs are using
> this pool and how much space each QP is using.
>
> rdmatool don't have a ucontext-level resource tracking currently, is it
> ok to achieve that through debugfs?
>
> This may looks like:
>
> $ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
> QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
> 110        6400       256        2048       4096
> 118        6400       256        2048       0
>
> Or should it be achieved in rdmatool?

I think so, because PID != ucontext. Why can't it be presented as QP
attribute? Can you please send "rdmatool" example?

Thanks

>
> Thanks
> Weihang

Re: Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[liweihang]

On 2020/12/31 15:59, Leon Romanovsky wrote:
> On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:
>> Hi all,
>>
>> We want to dump some hns driver-defined information that belongs to a
>> process to keep track of current memory usage. For example, there is
>> a ucontext-level(process-level) memory pool to store WQE which is
>> shared by a lot of QPs, we want to record and query which QPs are using
>> this pool and how much space each QP is using.
>>
>> rdmatool don't have a ucontext-level resource tracking currently, is it
>> ok to achieve that through debugfs?
>>
>> This may looks like:
>>
>> $ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
>> QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
>> 110        6400       256        2048       4096
>> 118        6400       256        2048       0
>>
>> Or should it be achieved in rdmatool?
> 
> I think so, because PID != ucontext. Why can't it be presented as QP
> attribute? Can you please send "rdmatool" example?
> 
> Thanks

Hi Leon,

Thanks for your response. If we can achieve it in rdmatool, it may
looks like:

1) We want to get some information of a ucontext (assuming that each
ucontext has an ID), for example, the size of a memory pool that belongs
to a ucontext as I mentioned above:

$ rdma res show ucontext
uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

2) We want to know which ucontext a QP belongs to:

$ rdma res show qp
link hns_0/1 lqpn 1 type GSI ... uctx_id 1
link hns_0/1 lqpn 2 type RC ... uctx_id 1

So the question is, we don't have a ucontext-level restrack currently, and
there in no 'id' for each ucontext.

Thanks
Weihang

Re: Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[Leon Romanovsky]

On Thu, Dec 31, 2020 at 09:36:27AM +0000, liweihang wrote:
> On 2020/12/31 15:59, Leon Romanovsky wrote:
> > On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:
> >> Hi all,
> >>
> >> We want to dump some hns driver-defined information that belongs to a
> >> process to keep track of current memory usage. For example, there is
> >> a ucontext-level(process-level) memory pool to store WQE which is
> >> shared by a lot of QPs, we want to record and query which QPs are using
> >> this pool and how much space each QP is using.
> >>
> >> rdmatool don't have a ucontext-level resource tracking currently, is it
> >> ok to achieve that through debugfs?
> >>
> >> This may looks like:
> >>
> >> $ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
> >> QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
> >> 110        6400       256        2048       4096
> >> 118        6400       256        2048       0
> >>
> >> Or should it be achieved in rdmatool?
> >
> > I think so, because PID != ucontext. Why can't it be presented as QP
> > attribute? Can you please send "rdmatool" example?
> >
> > Thanks
>
> Hi Leon,
>
> Thanks for your response. If we can achieve it in rdmatool, it may
> looks like:
>
> 1) We want to get some information of a ucontext (assuming that each
> ucontext has an ID), for example, the size of a memory pool that belongs
> to a ucontext as I mentioned above:
>
> $ rdma res show ucontext
> uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
> uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048

I have no problems to add "rdma res show ucontext" command, we just need
to find what should be printed.

>
> 2) We want to know which ucontext a QP belongs to:
>
> $ rdma res show qp
> link hns_0/1 lqpn 1 type GSI ... uctx_id 1
> link hns_0/1 lqpn 2 type RC ... uctx_id 1
>
> So the question is, we don't have a ucontext-level restrack currently, and
> there in no 'id' for each ucontext.

We have IDs for every ucontext, it is called "ctxn" and because QP is
not connected directly to ucontext, but through PDs, it is visible
when you check PDs.

[leonro@vm ~]$ ibv_rc_pingpong &
[leonro@vm ~]$ rdma res show
0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
[leonro@vm ~]$ rdma res show qp type RC
link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong
[leonro@vm ~]$ rdma res show pd pdn 3
ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong
                                      ^^^^^^

Thanks

>
> Thanks
> Weihang
>

Re: Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[liweihang]

On 2020/12/31 18:39, Leon Romanovsky wrote:
> On Thu, Dec 31, 2020 at 09:36:27AM +0000, liweihang wrote:
>> On 2020/12/31 15:59, Leon Romanovsky wrote:
>>> On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:
>>>> Hi all,
>>>>
>>>> We want to dump some hns driver-defined information that belongs to a
>>>> process to keep track of current memory usage. For example, there is
>>>> a ucontext-level(process-level) memory pool to store WQE which is
>>>> shared by a lot of QPs, we want to record and query which QPs are using
>>>> this pool and how much space each QP is using.
>>>>
>>>> rdmatool don't have a ucontext-level resource tracking currently, is it
>>>> ok to achieve that through debugfs?
>>>>
>>>> This may looks like:
>>>>
>>>> $ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
>>>> QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
>>>> 110        6400       256        2048       4096
>>>> 118        6400       256        2048       0
>>>>
>>>> Or should it be achieved in rdmatool?
>>>
>>> I think so, because PID != ucontext. Why can't it be presented as QP
>>> attribute? Can you please send "rdmatool" example?
>>>
>>> Thanks
>>
>> Hi Leon,
>>
>> Thanks for your response. If we can achieve it in rdmatool, it may
>> looks like:
>>
>> 1) We want to get some information of a ucontext (assuming that each
>> ucontext has an ID), for example, the size of a memory pool that belongs
>> to a ucontext as I mentioned above:
>>
>> $ rdma res show ucontext
>> uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
>> uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048
> 
> I have no problems to add "rdma res show ucontext" command, we just need
> to find what should be printed.
> 
>>
>> 2) We want to know which ucontext a QP belongs to:
>>
>> $ rdma res show qp
>> link hns_0/1 lqpn 1 type GSI ... uctx_id 1
>> link hns_0/1 lqpn 2 type RC ... uctx_id 1
>>
>> So the question is, we don't have a ucontext-level restrack currently, and
>> there in no 'id' for each ucontext.
> 
> We have IDs for every ucontext, it is called "ctxn" and because QP is
> not connected directly to ucontext, but through PDs, it is visible
> when you check PDs.
> 
> [leonro@vm ~]$ ibv_rc_pingpong &
> [leonro@vm ~]$ rdma res show
> 0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
> [leonro@vm ~]$ rdma res show qp type RC
> link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong
> [leonro@vm ~]$ rdma res show pd pdn 3
> ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong
>                                       ^^^^^^
> 
> Thanks
> 

OK, thank you, we will try to use rdmatool to achieve our goal.

By the way, can we use debugfs to add some trace functions?
Debugfs can quickly get a large amount of information at a time,
it's more convenient than netlink in such situation, especially
for some vendor-defined funtions.

Thanks
Weihang

Re: Is it ok to use debugfs to dump some ucontext-level driver-defined info?

[Leon Romanovsky]

On Tue, Jan 05, 2021 at 03:15:58AM +0000, liweihang wrote:
> On 2020/12/31 18:39, Leon Romanovsky wrote:
> > On Thu, Dec 31, 2020 at 09:36:27AM +0000, liweihang wrote:
> >> On 2020/12/31 15:59, Leon Romanovsky wrote:
> >>> On Tue, Dec 29, 2020 at 01:31:39PM +0000, liweihang wrote:
> >>>> Hi all,
> >>>>
> >>>> We want to dump some hns driver-defined information that belongs to a
> >>>> process to keep track of current memory usage. For example, there is
> >>>> a ucontext-level(process-level) memory pool to store WQE which is
> >>>> shared by a lot of QPs, we want to record and query which QPs are using
> >>>> this pool and how much space each QP is using.
> >>>>
> >>>> rdmatool don't have a ucontext-level resource tracking currently, is it
> >>>> ok to achieve that through debugfs?
> >>>>
> >>>> This may looks like:
> >>>>
> >>>> $ echo 1 > <dbgfs_dir>/hns_roce/hns_0/<pid>/qp
> >>>> QPN        Total(kB)  SQ(kB)     SGE(kB)    RQ(kB)
> >>>> 110        6400       256        2048       4096
> >>>> 118        6400       256        2048       0
> >>>>
> >>>> Or should it be achieved in rdmatool?
> >>>
> >>> I think so, because PID != ucontext. Why can't it be presented as QP
> >>> attribute? Can you please send "rdmatool" example?
> >>>
> >>> Thanks
> >>
> >> Hi Leon,
> >>
> >> Thanks for your response. If we can achieve it in rdmatool, it may
> >> looks like:
> >>
> >> 1) We want to get some information of a ucontext (assuming that each
> >> ucontext has an ID), for example, the size of a memory pool that belongs
> >> to a ucontext as I mentioned above:
> >>
> >> $ rdma res show ucontext
> >> uctx_id	1 pid 20 qp_buf_sz 6400 sq_buf_sz 256 sge_buf_sz 2048 rq_buf_sz 4096
> >> uctx_id	2 pid 20 qp_buf_sz 4800 sq_buf_sz 128 sge_buf_sz 2048 rq_buf_sz 2048
> >
> > I have no problems to add "rdma res show ucontext" command, we just need
> > to find what should be printed.
> >
> >>
> >> 2) We want to know which ucontext a QP belongs to:
> >>
> >> $ rdma res show qp
> >> link hns_0/1 lqpn 1 type GSI ... uctx_id 1
> >> link hns_0/1 lqpn 2 type RC ... uctx_id 1
> >>
> >> So the question is, we don't have a ucontext-level restrack currently, and
> >> there in no 'id' for each ucontext.
> >
> > We have IDs for every ucontext, it is called "ctxn" and because QP is
> > not connected directly to ucontext, but through PDs, it is visible
> > when you check PDs.
> >
> > [leonro@vm ~]$ ibv_rc_pingpong &
> > [leonro@vm ~]$ rdma res show
> > 0: ibp0s9: pd 4 cq 4 qp 4 cm_id 0 mr 1 ctx 1
> > [leonro@vm ~]$ rdma res show qp type RC
> > link ibp0s9/1 lqpn 50 rqpn 0 type RC state INIT rq-psn 16777215 sq-psn 0 path-mig-state MIGRATED pdn 3 pid 479 comm ibv_rc_pingpong
> > [leonro@vm ~]$ rdma res show pd pdn 3
> > ifindex 0 ifname ibp0s9 pdn 3 users 2 ctxn 0 pid 479 comm ibv_rc_pingpong
> >                                       ^^^^^^
> >
> > Thanks
> >
>
> OK, thank you, we will try to use rdmatool to achieve our goal.
>
> By the way, can we use debugfs to add some trace functions?
> Debugfs can quickly get a large amount of information at a time,
> it's more convenient than netlink in such situation, especially
> for some vendor-defined funtions.

If you are talking about connecting your driver to ftrace, so sure go
for it. However for the data, please use rdmatool, we have raw option (-r)
in the rdmatool that dumps in binary format your specific object.

Thanks

>
> Thanks
> Weihang