"Virtual" ibnetdiscover command fails

"Virtual" ibnetdiscover command fails

[Mathis GAVILLON]

Or, Jack,

I come back to you because I've another problem when I try to use
virtualization of Infiniband.
My host computer is connected to a cluster through Infiniband. I would
like all the rest of the cluster to communicate with the virtual
machine installed on the host.

I've lauched opensm on the host. All is working well except VM is not
listed when I launched ibnetdiscover on the host.
So I try to launch this last command on the virtual machine, and it
fails with this error :

src/query_smp.c:98; send failed; -5
segment core dump

The sminfo command doesn't work much better on the VM :

ibwarn: [3014] _do_madrpc: send failed; Invalid argument
ibwarn: [3014] mad_rpc: _do_madrpc failed; dport (DR path slid 0; dlid 0; 0)
sminfo: iberror: failed: can't resolve sm port (null)

The result of ibstat (on the virtual machine) :
CA 'mlx4_0'
	CA type: MT4100
	Number of ports: 1
	Firmware version: 2.11.500
	Hardware version: 0
	Node GUID: 0x00140500279a774b
	System image GUID: 0x0002c9030031ff63
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 21
		LMC: 0
		SM lid: 21
		Capability mask: 0x02514868
		Port GUID: 0x0014050000000001
		Link layer: InfiniBand



Have you got any idea about the problem ?

For informations, opensm version is 3.3.15 on both VM and host

Thanks

Mathis
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Or Gerlitz]

On 05/02/2013 17:41, Mathis GAVILLON wrote:
> All is working well except VM is not listed when I launched ibnetdiscover on the host.

Yep, this is limitation of the current implementation

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Or Gerlitz]

On 06/02/2013 11:17, Mathis GAVILLON wrote:
> Ok. But what is it possible to do with Infiniband VFs if QP0 is not available ?

EVERYTHING, e.g run IPoIB, iSER, RDS, MPI, etc, etc - except for what 
requires QP0, such as running SM or issuing SMPs for 
discovery/diagnostics purposes


>
>
> 2013/2/5 Jack Morgenstein <jackm-LDSdmyG8hGV8YrgS2mwiifqBs+8SCbDb@public.gmane.org>:
>> Mathis,
>>
>> You cannot use SMP packets on a virtual host (this is a security issue,
>> VFs are not trusted).  Since QP0 (SMP) is not available on VFs,
>> any tool which attempts to use QP0 (SMPs) will fail.
>>
>> Thus, OpenSM will not run over a VF, nor will ibnetdiscover,
>> nor will sminfo (which uses SMP).
>>
>> -Jack

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Mathis GAVILLON]

> EVERYTHING, e.g run IPoIB, iSER, RDS, MPI, etc, etc - except for what
> requires QP0, such as running SM or issuing SMPs for discovery/diagnostics
> purposes

Ok. I just begin with Infiniband technologie so I don't know
everything about this yet.

Thanks

Mathis
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Or Gerlitz]

On 06/02/2013 12:04, Mathis GAVILLON wrote:
> Just a last question : is that possible VFs lid to be different from PF one ?

NO, we've implemented a "shared port" model, so all functions on the 
same IB port use the same lid, each function has its own
virtual GUID though.

Or.


>
> Thanks
>
> 2013/2/6 Mathis GAVILLON <jbiboche-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>> EVERYTHING, e.g run IPoIB, iSER, RDS, MPI, etc, etc - except for what
>>> requires QP0, such as running SM or issuing SMPs for discovery/diagnostics
>>> purposes
>> Ok. I just begin with Infiniband technologie so I don't know
>> everything about this yet.
>>
>> Thanks
>>
>> Mathis

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Sebastian Riemer]

On 06.02.2013 10:22, Or Gerlitz wrote:
> On 06/02/2013 11:17, Mathis GAVILLON wrote:
>> Ok. But what is it possible to do with Infiniband VFs if QP0 is not
>> available ?
> 
> EVERYTHING, e.g run IPoIB, iSER, RDS, MPI, etc, etc - except for what
> requires QP0, such as running SM or issuing SMPs for
> discovery/diagnostics purposes

But SRP isn't provided with SR-IOV I've heared. Is it just a matter of
software or is it a matter of firmware/hardware?

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Sebastian Riemer]

On 06.02.2013 11:20, Or Gerlitz wrote:
> On 06/02/2013 12:04, Mathis GAVILLON wrote:
>> Just a last question : is that possible VFs lid to be different from
>> PF one ?
> 
> NO, we've implemented a "shared port" model, so all functions on the
> same IB port use the same lid, each function has its own
> virtual GUID though.

So if I don't use the unmaintained srptools to get the SRP connection
strings but instead send them directly to the initiator to connect to
the SRP target, then also SRP should be possible with the virtual GUID.
Am I right?

Cheers,
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: "Virtual" ibnetdiscover command fails

[Or Gerlitz]

On 06/02/2013 12:40, Sebastian Riemer wrote:
> So if I don't use the unmaintained srptools to get the SRP connection 
> strings but instead send them directly to the initiator to connect to 
> the SRP target, then also SRP should be possible with the virtual 
> GUID. Am I right?

Basically YES, you can use the initiator VM vGID as the source GID for 
the connection.

Or.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: srptools ("Virtual" ibnetdiscover command fails)

[Bart Van Assche]

On 02/06/13 11:40, Sebastian Riemer wrote:
> On 06.02.2013 11:20, Or Gerlitz wrote:
>> On 06/02/2013 12:04, Mathis GAVILLON wrote:
>>> Just a last question : is that possible VFs lid to be different from
>>> PF one ?
>>
>> NO, we've implemented a "shared port" model, so all functions on the
>> same IB port use the same lid, each function has its own
>> virtual GUID though.
>
> So if I don't use the unmaintained srptools to get the SRP connection
> strings but instead send them directly to the initiator to connect to
> the SRP target, then also SRP should be possible with the virtual GUID.
> Am I right?

It would be appreciated it if someone would step forward and would 
propose to maintain the srptools package. In the meantime, an update for 
srptools 0.4 can be found here: http://github.com/bvanassche/srptools. 
The user-visible changes compared to version 0.4 are:
- srp_daemon keeps working even if the LID changes of the port it is
   using to scan the fabric or if a P_Key change occurs. In other words,
   it is no longer necessary to restart srp_daemon after an IB cable has
   been plugged.
- Added P_Key support to srp_daemon and ibsrpdm.
- Fixed month in srp_daemon.log (OFED bug #2281). srp_daemon now uses
   syslog and logrotate for logging.
- srp_daemon is now only started in InfiniBand ports. It is no longer
   attempted to start srp_daemon on Ethernet ports.
- Fixed a memory leak in srp_daemon that was triggered once during
   every fabric rescan.
- Reduced memory consumption of the srp_daemon process.
- srp_daemon skips now MAD transaction ID 0 after 2**32 rescans.
- Installation: SRPHA_ENABLE=no / SRP_DAEMON_ENABLE=no is only added to
   /etc/infiniband/openibd.conf if these variables did not yet exist in
   that file.
- Changed range of the srp_daemon and ibsrpdm exit codes from 0..127
   into 0..1.
- Changed ibsrpdm such that it uses RMPP.
- Changed ibsrpdm such that it uses the new umad P_Key ABI. Running
   ibsrpdm does no longer cause a warning to be logged ("user_mad:
   process ibsrpdm did not enable P_Key index support / user_mad:
   Documentation/infiniband/user_mad.txt has info on the new ABI").
- Fixed spelling of several help texts and diagnostic messages.

As usual, feedback is welcome.

Bart.

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html