Mellanox/RoCE - VLAN support

Mellanox/RoCE - VLAN support

[Klaus Wacker]

Hi all,
we are doing a project on IBM x-blades using Mellanox/RoCE. I have some
difficulties to get RoCE based VLANs to run.
We are currently on Linux 3.8. No OFED material involved.
While non-VLAN devices work ok, i experience the following on VLAN devs
- a GID for the VLAN is created and we use the proper index for gid source
addr.
- a ping on the VLAN-device runs ok
- rdma/roce traffic fails in that the sender overruns the send retry count.

In terms of Mellanox driver level, what gives us Linux 3.8? Are we at most
current driver level?

And do we have to take special setup for VLAN, i.e. special MLX4 module
parms, or special qp parameters to get this going?

Any help on this is very much appreciated.

Kind regards
Klaus Wacker
IBM Systems &Technology Group, Systems Software Development
SW Linux on System z Dev & Service
Phone:  +49-7031-16-3779   IBM Deutschland  E-Mail:  kdwacker-tA70FqPdS9Zhl2p70BpVqQ@public.gmane.orgm
Schoenaicher Str. 220   71032 Boeblingen   Germany
IBM Deutschland Research & Development GmbH / Vorsitzende des
Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart,
HRB 243294

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mellanox/RoCE - VLAN support

[Or Gerlitz]

Klaus Wacker <Klaus.Wacker-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org> wrote:
> Hi all, we are doing a project on IBM x-blades using Mellanox/RoCE. I have some
> difficulties to get RoCE based VLANs to run.
> We are currently on Linux 3.8. No OFED material involved.
> While non-VLAN devices work ok, i experience the following on VLAN devs
> - a GID for the VLAN is created and we use the proper index for gid source addr.
> - a ping on the VLAN-device runs ok
> - rdma/roce traffic fails in that the sender overruns the send retry count.

Do you have any evidence that the packets are actually crossing the
switch? the way we debug RoCE connectivity issues is configuring port
mirroring on the switch and
capture the mirrored packets on the host where packets are mirrored,
and feed them to wireshark (the official/latest versions support
parsing of RoCE/IB packets), on that host you can also observe  L2
mac/vlan header of the packets to see if it matches what you expected.
But even before that, I would check the switch counters to see the
packets aren't dropped.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Mellanox/RoCE - VLAN support

[Jack Morgenstein]

On Friday 12 April 2013 02:13, Or Gerlitz wrote:
> Klaus Wacker <Klaus.Wacker-tA70FqPdS9bQT0dZR+AlfA@public.gmane.org> wrote:
> > Hi all, we are doing a project on IBM x-blades using Mellanox/RoCE. I have some
> > difficulties to get RoCE based VLANs to run.
> > We are currently on Linux 3.8. No OFED material involved.
> > While non-VLAN devices work ok, i experience the following on VLAN devs
> > - a GID for the VLAN is created and we use the proper index for gid source addr.
> > - a ping on the VLAN-device runs ok
> > - rdma/roce traffic fails in that the sender overruns the send retry count.
> 
> Do you have any evidence that the packets are actually crossing the
> switch? the way we debug RoCE connectivity issues is configuring port
> mirroring on the switch and
> capture the mirrored packets on the host where packets are mirrored,
> and feed them to wireshark (the official/latest versions support
> parsing of RoCE/IB packets), on that host you can also observe  L2
> mac/vlan header of the packets to see if it matches what you expected.
> But even before that, I would check the switch counters to see the
> packets aren't dropped.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
Thanks, Or!

Klaus,
In addition, please provide us with the following information:
1. Which ConnectX FW version are you using?
2. Are you using RC or UD qp? (I assume RC if you have a retry count failure, but I would like to be sure).
3. Can you successfully run ucmatose? (tests rdma library for RC connections).
	To run ucmatose:
  		on server:  ucmatose
		on client:  ucmatose -s <IP of desired Mellanox ETH device (ethXX)>

-Jack
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html