dumping queue state

dumping queue state

[Steve Wise]

Hey,

I have a need to provide tools for customers to gather runtime state for an
rdma device.   Say, when an application is stuck waiting for some completion
or other rdma event.   This includes hw/fw state of course, and equally as
important, rdma object sw state.  Is debugfs the correct way to export this
sw state?  The data is quite large potentially; each QP, its structures, the
dma queue memory, etc.  Ditto for CQs.  Also MR state, etc etc.  It seems
that would be overloading debugfs to me.  Currently the hw/fw state is being
gathered via ethtool dump commands (--get-dump, --register-dump,
--eeprom-dump).  I am considering using the ethtool --get-dump method for
the low level driver to also include dumping the rdma queue state for the
device.   Is that a reasonable approach?

Any thoughts/suggestions? 

Thanks in advance,

Steve.


---
This email has been checked for viruses by AVG.
http://www.avg.com

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Bart Van Assche]

On Wed, 2017-12-20 at 09:53 -0600, Steve Wise wrote:
> I have a need to provide tools for customers to gather runtime state for an
> rdma device.   Say, when an application is stuck waiting for some completion
> or other rdma event.   This includes hw/fw state of course, and equally as
> important, rdma object sw state.  Is debugfs the correct way to export this
> sw state?  The data is quite large potentially; each QP, its structures, the
> dma queue memory, etc.  Ditto for CQs.  Also MR state, etc etc.  It seems
> that would be overloading debugfs to me.  Currently the hw/fw state is being
> gathered via ethtool dump commands (--get-dump, --register-dump,
> --eeprom-dump).  I am considering using the ethtool --get-dump method for
> the low level driver to also include dumping the rdma queue state for the
> device.   Is that a reasonable approach?
> 
> Any thoughts/suggestions? 

Hello Steve,

I'm not sure what the best solution is for RDMA queues. But if you want you can
have a look at block/blk-mq-debugfs.c. That code has been added recently and has
been very valuable for debugging block layer / SCSI queue lockups.

Bart.

RE: dumping queue state

[Steve Wise]

> Hello Steve,
> 
> I'm not sure what the best solution is for RDMA queues. But if you want you can
> have a look at block/blk-mq-debugfs.c. That code has been added recently and
> has
> been very valuable for debugging block layer / SCSI queue lockups.

Thanks Bart.  

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 4308 bytes --]

On Wed, Dec 20, 2017 at 09:53:33AM -0600, Steve Wise wrote:
> Hey,
>
> I have a need to provide tools for customers to gather runtime state for an
> rdma device.   Say, when an application is stuck waiting for some completion
> or other rdma event.   This includes hw/fw state of course, and equally as
> important, rdma object sw state.  Is debugfs the correct way to export this
> sw state?  The data is quite large potentially; each QP, its structures, the
> dma queue memory, etc.  Ditto for CQs.  Also MR state, etc etc.  It seems
> that would be overloading debugfs to me.  Currently the hw/fw state is being
> gathered via ethtool dump commands (--get-dump, --register-dump,
> --eeprom-dump).  I am considering using the ethtool --get-dump method for
> the low level driver to also include dumping the rdma queue state for the
> device.   Is that a reasonable approach?

In this cycle, I'm going to submit RDMA resource tracking feature, which
will give an infrastructure to do it in RDMA:

The kernel code, it is based on RCU and not final, because my call to
synchronize_rcu is not effective.
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/log/?h=topic/restrack-rcu

There is supplementary part in RDMAtool, which presents global
information, QP information with options to filter.

It is initial stage.

+ /mnt/iproute2/rdma/rdma res
1: mlx5_0: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
2: mlx5_1: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
3: mlx5_2: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
4: mlx5_3: curr/max: pd 2/16777216 cq 3/16777216 qp 2/262144
5: mlx5_4: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
+ /mnt/iproute2/rdma/rdma res show mlx5_4
5: mlx5_4: curr/max: pd 3/16777216 cq 5/16777216 qp 4/262144
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/0
Wrong device name
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/1  7          UD    RTS    0          [mlx5-gsi]
mlx5_4/1  1          GSI   RTS    0          [rdma-mad]
mlx5_4/1  0          SMI   RTS    0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/-
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx5_4/-  8          UD    RESET  0          [ipoib-verbs]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/- -d
DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM            SQ-PSN     RQ-PSN     PATH-MIG
mlx5_4/-  8          ---        UD    RESET  0          [ipoib-verbs]   0          ---        ---
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm -d
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]
+ /mnt/iproute2/rdma/rdma res show qp link mlx5_4/1 display pid,lqpn,comm pid 0-2000
DEV/PORT  LQPN       PID        COMM
mlx5_4/1  7          0          [mlx5-gsi]
mlx5_4/1  1          0          [rdma-mad]
mlx5_4/1  0          0          [rdma-mad]

>
> Any thoughts/suggestions?

Care to try?

>
> Thanks in advance,
>
> Steve.
>
>
> ---
> This email has been checked for viruses by AVG.
> http://www.avg.com
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: dumping queue state

[Steve Wise]

> 
> >
> > Any thoughts/suggestions?
> 
> Care to try?
> 

Sure, I'm building the kernel now.  Where do I get the rdma command that
will work with this?

Is there a way to extend commands with provider-specific data? 

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 608 bytes --]

On Wed, Dec 20, 2017 at 11:00:30AM -0600, Steve Wise wrote:
> >
> > >
> > > Any thoughts/suggestions?
> >
> > Care to try?
> >
>
> Sure, I'm building the kernel now.  Where do I get the rdma command that
> will work with this?

I uploaded iproute2 part.
https://git.kernel.org/pub/scm/linux/kernel/git/leon/iproute2.git/log/?h=topic/restrack

WIP: no manuals, no review yet and not massively tested - but works for me :


>
> Is there a way to extend commands with provider-specific data?
>

In future yes, rdmatool provides "-d" option (detailed output) and IMHO
all vendor specific will got there.

Thanks

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: dumping queue state

[Steve Wise]

> WIP: no manuals, no review yet and not massively tested - but works for me
:
> 
> 
> >
> > Is there a way to extend commands with provider-specific data?
> >
> 
> In future yes, rdmatool provides "-d" option (detailed output) and IMHO
> all vendor specific will got there.

Cool.

Looksie:

[root@stevo1 iproute2]# ./rdma/rdma res
1: mlx4_0: curr/max: pd 4/32764 cq 17/65408 qp 3/163768
2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288
2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288

[root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
cxgb4_0/- 1028       RC    RTS    0          [rdma-cm]
cxgb4_0/- 1026       RC    RTS    0          [rdma-cm]

I'm not sure why the port is not specified on the above cxgb4 QPs which are
connected.  They should have a port number.  But perhaps it isn't filled in
correctly in the cm_id or something?  

[root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0
DEV/PORT  LQPN       TYPE  STATE  PID        COMM
mlx4_0/1  537        RC    RTS    0          [rdma-cm]
mlx4_0/1  536        RC    RTS    0          [rdma-cm]
mlx4_0/1  1          GSI   RTS    0          [rdma-mad]
[root@stevo1 iproute2]#

rdma should probably understand enough transport-specific details to not
display ib-specific stuff on an iwarp qp.  Here PSNs and PATH-MIG are all
IB-specific:

[root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0 -d
DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM
SQ-PSN     RQ-PSN     PATH-MIG
cxgb4_0/- 1028       0          RC    RTS    0          [rdma-cm]       0
0          MIGRATED
cxgb4_0/- 1026       0          RC    RTS    0          [rdma-cm]       0
0          MIGRATED

[root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0 -d
DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM
SQ-PSN     RQ-PSN     PATH-MIG
mlx4_0/1  537        537        RC    RTS    0          [rdma-cm]
13145128   12441844   MIGRATED
mlx4_0/1  536        536        RC    RTS    0          [rdma-cm]
14218681   1982803    MIGRATED
mlx4_0/1  1          ---        GSI   RTS    0          [rdma-mad]      0
---        ---

Looking good though!

Why did you prefer netlink vs debugfs for this stuff?  (I'm sure it was
discussed on the list but I missed it).

Steve.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 3224 bytes --]

On Wed, Dec 20, 2017 at 11:42:22AM -0600, Steve Wise wrote:
> > WIP: no manuals, no review yet and not massively tested - but works for me
> :
> >
> >
> > >
> > > Is there a way to extend commands with provider-specific data?
> > >
> >
> > In future yes, rdmatool provides "-d" option (detailed output) and IMHO
> > all vendor specific will got there.
>
> Cool.
>
> Looksie:
>
> [root@stevo1 iproute2]# ./rdma/rdma res
> 1: mlx4_0: curr/max: pd 4/32764 cq 17/65408 qp 3/163768
> 2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288
> 2: cxgb4_0: curr/max: pd 2/65536 cq 16/24576 qp 2/12288
>
> [root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0
> DEV/PORT  LQPN       TYPE  STATE  PID        COMM
> cxgb4_0/- 1028       RC    RTS    0          [rdma-cm]
> cxgb4_0/- 1026       RC    RTS    0          [rdma-cm]
>
> I'm not sure why the port is not specified on the above cxgb4 QPs which are
> connected.  They should have a port number.  But perhaps it isn't filled in
> correctly in the cm_id or something?

I'm not using CM and take this information from ib_query_qp
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/tree/drivers/infiniband/core/nldev.c?h=topic/restrack-rcu#n236
https://git.kernel.org/pub/scm/linux/kernel/git/leon/linux-rdma.git/tree/drivers/infiniband/core/nldev.c?h=topic/restrack-rcu#n249

Most probably it returns port = 0 and it is not valid for the RDMAtool.

>
> [root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0
> DEV/PORT  LQPN       TYPE  STATE  PID        COMM
> mlx4_0/1  537        RC    RTS    0          [rdma-cm]
> mlx4_0/1  536        RC    RTS    0          [rdma-cm]
> mlx4_0/1  1          GSI   RTS    0          [rdma-mad]
> [root@stevo1 iproute2]#
>
> rdma should probably understand enough transport-specific details to not
> display ib-specific stuff on an iwarp qp.  Here PSNs and PATH-MIG are all
> IB-specific:

Right, it is bug.

>
> [root@stevo1 iproute2]# ./rdma/rdma res show qp link cxgb4_0 -d
> DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM
> SQ-PSN     RQ-PSN     PATH-MIG
> cxgb4_0/- 1028       0          RC    RTS    0          [rdma-cm]       0
> 0          MIGRATED
> cxgb4_0/- 1026       0          RC    RTS    0          [rdma-cm]       0
> 0          MIGRATED
>
> [root@stevo1 iproute2]# ./rdma/rdma res show qp link mlx4_0 -d
> DEV/PORT  LQPN       RQPN       TYPE  STATE  PID        COMM
> SQ-PSN     RQ-PSN     PATH-MIG
> mlx4_0/1  537        537        RC    RTS    0          [rdma-cm]
> 13145128   12441844   MIGRATED
> mlx4_0/1  536        536        RC    RTS    0          [rdma-cm]
> 14218681   1982803    MIGRATED
> mlx4_0/1  1          ---        GSI   RTS    0          [rdma-mad]      0
> ---        ---
>
> Looking good though!

Check filters options too, to filters by numbers and display columns.

>
> Why did you prefer netlink vs debugfs for this stuff?  (I'm sure it was
> discussed on the list but I missed it).

Three reasons:
1. We wanted rdmatool to be part of iproute2.
2. Debugfs requires root access and netlink doesn't
3. Better extensibility, if the device doesn't support the netlink won't
provide attribute and all neltink libraries know how to deal with it.

>
> Steve.
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: dumping queue state

[Steve Wise]

> 
> Check filters options too, to filters by numbers and display columns.

[root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/-
display lqpn,type,state,pid lqpn 1000-1030
DEV/PORT  LQPN       TYPE  STATE  PID
cxgb4_0/- 1027       RC    RTS    0
[root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/-
display lqpn,type,state,pid lqpn 1000-1033
DEV/PORT  LQPN       TYPE  STATE  PID
cxgb4_0/- 1027       RC    RTS    0
cxgb4_0/- 1033       RC    RTS    0
[root@stevo1 ~]#

Working!



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 711 bytes --]

On Wed, Dec 20, 2017 at 01:01:18PM -0600, Steve Wise wrote:
> >
> > Check filters options too, to filters by numbers and display columns.
>
> [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/-
> display lqpn,type,state,pid lqpn 1000-1030
> DEV/PORT  LQPN       TYPE  STATE  PID
> cxgb4_0/- 1027       RC    RTS    0
> [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link cxgb4_0/-
> display lqpn,type,state,pid lqpn 1000-1033
> DEV/PORT  LQPN       TYPE  STATE  PID
> cxgb4_0/- 1027       RC    RTS    0
> cxgb4_0/- 1033       RC    RTS    0
> [root@stevo1 ~]#
>
> Working!

Thanks,
It supports various combinations with "," and "-", like
990,1000-1030,1031-1032,1033

>
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: dumping queue state

[Steve Wise]

> > > Check filters options too, to filters by numbers and display columns.
> >
> > [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link
> cxgb4_0/-
> > display lqpn,type,state,pid lqpn 1000-1030
> > DEV/PORT  LQPN       TYPE  STATE  PID
> > cxgb4_0/- 1027       RC    RTS    0
> > [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link
> cxgb4_0/-
> > display lqpn,type,state,pid lqpn 1000-1033
> > DEV/PORT  LQPN       TYPE  STATE  PID
> > cxgb4_0/- 1027       RC    RTS    0
> > cxgb4_0/- 1033       RC    RTS    0
> > [root@stevo1 ~]#
> >
> > Working!
> 
> Thanks,
> It supports various combinations with "," and "-", like
> 990,1000-1030,1031-1032,1033

While the rdma tool easily allows poking around various resources, it would
be sort of painful to do a full state dump.  Perhaps a fastpath command like
'rdma dumpall' to just dump everything in detail? 



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 1235 bytes --]

On Wed, Dec 20, 2017 at 03:42:56PM -0600, Steve Wise wrote:
> > > > Check filters options too, to filters by numbers and display columns.
> > >
> > > [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link
> > cxgb4_0/-
> > > display lqpn,type,state,pid lqpn 1000-1030
> > > DEV/PORT  LQPN       TYPE  STATE  PID
> > > cxgb4_0/- 1027       RC    RTS    0
> > > [root@stevo1 ~]# /root/stevo/iproute2/rdma/rdma res show qp link
> > cxgb4_0/-
> > > display lqpn,type,state,pid lqpn 1000-1033
> > > DEV/PORT  LQPN       TYPE  STATE  PID
> > > cxgb4_0/- 1027       RC    RTS    0
> > > cxgb4_0/- 1033       RC    RTS    0
> > > [root@stevo1 ~]#
> > >
> > > Working!
> >
> > Thanks,
> > It supports various combinations with "," and "-", like
> > 990,1000-1030,1031-1032,1033
>
> While the rdma tool easily allows poking around various resources, it would
> be sort of painful to do a full state dump.  Perhaps a fastpath command like
> 'rdma dumpall' to just dump everything in detail?

It can be an option after we add more objects, right now, I added only
QP and this one object is "full state" for now. To dump all QPs in detailed
format simply run "rdma res show qp -d" and add "-j" to output it in JSON format.

Thanks

>
>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

RE: dumping queue state

[Steve Wise]

> 
> I'm not using CM and take this information from ib_query_qp

By the way, it will be very useful to dump cm_ids, or at least include the
ip address/port numbers for each connected qp...



--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: dumping queue state

[Leon Romanovsky]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

On Wed, Dec 20, 2017 at 01:31:43PM -0600, Steve Wise wrote:
> >
> > I'm not using CM and take this information from ib_query_qp
>
> By the way, it will be very useful to dump cm_ids, or at least include the
> ip address/port numbers for each connected qp...
>

Will do, we need it for RoCE too.

>
>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]