are IB port counters functioning over --IB-- with the RDMAoE patch set?

are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Or Gerlitz]

Hi Eli, Vlad, 

Using some lab node here, I noticed that the IB port counters were not functioning over --IB-- with the RDMAoE patches present in an OFED drop named OFED-RDMAoE-1.5-rc6, see below.

Was/is this a known issue? if yes, was it fixed? is/was there an ofa bz case open for this? I didn't find such... I was told that under ofed 1.5.1 this doesn't happen, but I wanted to 1st clarify with you.

Or.


> > [root@dodly4 ~]# rdma_bw dodly0
> > 15050: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 |
> > 15050: Local address:  LID 0x03, QPN 0x14004a, PSN 0x5fb3a8 RKey 0x2804260c VAddr 0x002acb54d6a000
> > 15050: Remote address: LID 0x04, QPN 0x250406, PSN 0xf6d97d, RKey 0x28043c04 VAddr 0x002b9026b6a000
> > 15050: Bandwidth peak (#0 to #989): 938.697 MB/sec
> > 15050: Bandwidth average: 938.695 MB/sec --> the link is SDR, so the result makes sense
> > 15050: Service Demand peak (#0 to #989): 3113 cycles/KB
> > 15050: Service Demand Avg  : 3113 cycles/KB
>   
> > [root@dodly4 ~]# perfquery
> > ibwarn: [15057] dump_perfcounters: PortXmitWait not indicated so ignore this counter
> > # Port counters: Lid 3 port 1
> > PortSelect:......................1
> > CounterSelect:...................0x0000
> > SymbolErrors:....................0
> > LinkRecovers:....................0
> > LinkDowned:......................0
> > RcvErrors:.......................0
> > RcvRemotePhysErrors:.............0
> > RcvSwRelayErrors:................0
> > XmtDiscards:.....................0
> > XmtConstraintErrors:.............0
> > RcvConstraintErrors:.............0
> > CounterSelect2:..................0x00
> > LinkIntegrityErrors:.............0
> > ExcBufOverrunErrors:.............0
> > VL15Dropped:.....................0
> > XmtData:.........................72
> > RcvData:.........................0
> > XmtPkts:.........................1
> > RcvPkts:.........................0
> > XmtWait:.........................0
>   
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Or Gerlitz]

> Hi Eli, Vlad,  Using some lab node here, I noticed that the IB port 
> counters were not functioning over --IB-- with the RDMAoE patches 
> present in an OFED drop named OFED-RDMAoE-1.5-rc6, see below.
> Was/is this a known issue? if yes, was it fixed? is/was there an ofa 
> bz case open for this? I didn't find such... I was told that under 
> ofed 1.5.1 this doesn't happen, but I wanted to 1st clarify with you.
Eli, Vlad, anything you can comment here?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Moni Shoua]

Did you try OFED-1.5.1 or even better, OFED-1.5.2?
I know patches for counters with RoCEE were submitted since OFED-1.5 and
I saw it working.


-----Original Message-----
From: Or Gerlitz 
Sent: Thursday, June 03, 2010 14:49
To: Eli Cohen; Vladimir Sokolovsky; Eli Cohen
Cc: Linux RDMA list; Moni Shoua; Shiri Franchi
Subject: Re: are IB port counters functioning over --IB-- with the
RDMAoE patch set?


> Hi Eli, Vlad,  Using some lab node here, I noticed that the IB port 
> counters were not functioning over --IB-- with the RDMAoE patches 
> present in an OFED drop named OFED-RDMAoE-1.5-rc6, see below.
> Was/is this a known issue? if yes, was it fixed? is/was there an ofa 
> bz case open for this? I didn't find such... I was told that under 
> ofed 1.5.1 this doesn't happen, but I wanted to 1st clarify with you.
Eli, Vlad, anything you can comment here?

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Or Gerlitz]

Moni Shoua wrote:
> Did you try OFED-1.5.1 or even better, OFED-1.5.2? I know patches for counters with RoCEE were submitted since OFED-1.5 and I saw it working
Mony, I'm not using ofed, sorry... I am interested in a clarification in 
the context of the upstream submission, e.g does the problem exist in 
the latest patch-set, is there a bz case tracking this, etc.

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Eli Cohen]

Or, counter should work as regular in upstream kernel patches for IB
link layer. For IBoE, they will not work since the SMA does not
support them. I have patches that allow to show counters using sysfs
but those have not yet been sent upstream. They available in
OFED-1.5.2 latest builds.

On Thu, Jun 3, 2010 at 4:33 PM, Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> Moni Shoua wrote:
>>
>> Did you try OFED-1.5.1 or even better, OFED-1.5.2? I know patches for
>> counters with RoCEE were submitted since OFED-1.5 and I saw it working
>
> Mony, I'm not using ofed, sorry... I am interested in a clarification in the
> context of the upstream submission, e.g does the problem exist in the latest
> patch-set, is there a bz case tracking this, etc.
>
> Or.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Or Gerlitz]

Eli Cohen wrote:
> counter should work as regular in upstream kernel patches for IB link layer. 

okay good, can you validate that? basically, I can set some time to clone Roland's tree
and use the iboe branch as a basis for testing that the IB stack is live and kicking as it used to be before the patches. I just need an updated copy of the rest of the patch set (Roland has three patches so far) for that end. Over the review process there were bunch of comments but no new posting, how are you planning to proceed in the review/merge process?

> for IBoE, they will not work since the SMA does not support them
> I have patches that allow to show counters using sysfs 

I am not with you. The counters are read using a MAD sent to the firmware PMA (QP #1), 
this applies for both perfquery and sysfs, isn't it? 

Or.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Eli Cohen]

On Mon, Jun 7, 2010 at 4:29 PM, Or Gerlitz <ogerlitz-smomgflXvOZWk0Htik3J/w@public.gmane.org> wrote:
> Eli Cohen wrote:
>> counter should work as regular in upstream kernel patches for IB link layer.
>
> okay good, can you validate that? basically, I can set some time to clone Roland's tree
> and use the iboe branch as a basis for testing that the IB stack is live and kicking as it used to be before the patches. I just need an updated copy of the rest of the patch set (Roland has three patches so far) for that end. Over the review process there were bunch of comments but no new posting, how are you planning to proceed in the review/merge process?
>
Why are you asking me to validate that? Did you actually encounter a
problem with this?

I did not send any patches since I am not sure how would Roland would
like to handle these changes. There have been a few comments, and
there are some fixes that I pushed to OFED. Roland, would you like me
to send the patches against the iboe branch?

>> for IBoE, they will not work since the SMA does not support them
>> I have patches that allow to show counters using sysfs
>
> I am not with you. The counters are read using a MAD sent to the firmware PMA (QP #1),
> this applies for both perfquery and sysfs, isn't it?

The counters patches will divert the code: for iboe it will not issue
a MAD to the firmware. It will use another command.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: are IB port counters functioning over --IB-- with the RDMAoE patch set?

[Or Gerlitz]

Eli Cohen wrote:
> Why are you asking me to validate that? Did you actually encounter a problem with this?
yes, I did. It didn't work with some ofed drop I was using. Anyway, as I 
said, I can do some validation that IBoE doesn't break upstream IB, just 
need the patches for that end, so once they are available, I will give 
them a try over 2.6.35-rcX

> The counters patches will divert the code: for iboe it will not issue a MAD to the firmware. It will use another command.
Can you be more specific what is the origin for this new design? is it 
HW limitation or firware limitation or something else? In case its not 
hardware limitation, I don't think we need to go for non MAD based 
scheme, at least not for mlx4

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html