Re: [ewg] rdma/cm: disallow loopback address for iwarp devices

Jeff Squyres
2010-02-09T01:02:55Z

[-- Attachment #1: Type: text/plain, Size: 1480 bytes --]

On Feb 8, 2010, at 7:50 PM, Pradeep Satyanarayana wrote:

> No, there is none. I got this command from one of the mails in the thread. What should I use instead?

You need to compile and run an MPI program. "ring" is a typical test program that sends a message around in a ring. I think that OFED installs those test apps somewhere, but I don't recall where offhand.

ring_c.c is attached. Compile it with:

mpicc ring_c.c -o ring

(you might need the full path to mpicc if it's not in your path?)

A better mpirun command line would be:

/usr/mpi/gcc/openmpi-1.4.1/bin/mpirun -np 2 --host HOSTNAME1,HOSTNAME2 \
--mca btl openib,sm,self --mca btl_openib_cpc_include rdmacm ring

Put in your own HOSTNAME1 and HOSTNAME2 values. You'll also need to ensure that both Open MPI and "ring" are available on both names (preferably in the same filesystem locations on both nodes, for simplicity) and that you can ssh to from one node to the other without being prompted for a password or passphrase.

This will run a 2-process MPI job across the two nodes, passing a message between the two processes a few times before quitting.

The various --mca parameters on this mpirun command line ensure that you are definitely using the OpenFabrics verbs support and forcing the use of RDMA CM.

--
Jeff Squyres
jsquyres-FYB4Gu1CFyUAvxtiuMwx3w@public.gmane.org

For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

[-- Attachment #2: ring_c.c --]
[-- Type: application/octet-stream, Size: 2418 bytes --]

/*
* Copyright (c) 2004-2006 The Trustees of Indiana University and Indiana
* University Research and Technology
* Corporation. All rights reserved.
* Copyright (c) 2006 Cisco Systems, Inc. All rights reserved.
*
* Simple ring test program
*/

#include <stdio.h>
#include "mpi.h"

int main(int argc, char *argv[])
{
int rank, size, next, prev, message, tag = 201;

/* Start up MPI */

MPI_Init(&argc, &argv);
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

/* Calculate the rank of the next process in the ring. Use the
modulus operator so that the last process "wraps around" to
rank zero. */

next = (rank + 1) % size;
prev = (rank + size - 1) % size;

/* If we are the "master" process (i.e., MPI_COMM_WORLD rank 0),
put the number of times to go around the ring in the
message. */

if (0 == rank) {
message = 10;

printf("Process 0 sending %d to %d, tag %d (%d processes in ring)\n",
message, next, tag, size);
MPI_Send(&message, 1, MPI_INT, next, tag, MPI_COMM_WORLD);
printf("Process 0 sent to %d\n", next);
}

/* Pass the message around the ring. The exit mechanism works as
follows: the message (a positive integer) is passed around the
ring. Each time it passes rank 0, it is decremented. When
each processes receives a message containing a 0 value, it
passes the message on to the next process and then quits. By
passing the 0 message first, every process gets the 0 message
and can quit normally. */

while (1) {
MPI_Recv(&message, 1, MPI_INT, prev, tag, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);

if (0 == rank) {
--message;
printf("Process 0 decremented value: %d\n", message);
}

MPI_Send(&message, 1, MPI_INT, next, tag, MPI_COMM_WORLD);
if (0 == message) {
printf("Process %d exiting\n", rank);
break;
}
}

/* The last process does one extra send to process 0, which needs
to be received before the program can exit */

if (0 == rank) {
MPI_Recv(&message, 1, MPI_INT, prev, tag, MPI_COMM_WORLD,
MPI_STATUS_IGNORE);
}

/* All done */

MPI_Finalize();
return 0;
}