basic questions about partitions

basic questions about partitions

[Tom Ammon]

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

How do you configure partitions with opensm? From reading the opensmd 
man page, it looks like you just create a file called 
/etc/osm-partitions.conf, with port GUIDs and such, but is this current? 
I ask because according to the man page the opensm configuration file is 
in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Thanks,

Tom

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: basic questions about partitions

[Mike Heinz]

Tom,

For your first question: yes, an HCA port can belong to multiple partitions. Trivial example: All HCAs must always be able to communicate with the SM, so they must be members of the SM's partition even if they are also members of (for example) the "MPI Traffic" partition.

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Tom Ammon
Sent: Monday, July 12, 2010 2:33 PM
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Subject: basic questions about partitions

  Hi,

I have some basic questions about IB partitions.

Can an HCA port belong to more than 1 partition at a time?

How do you configure partitions with opensm? From reading the opensmd 
man page, it looks like you just create a file called 
/etc/osm-partitions.conf, with port GUIDs and such, but is this current? 
I ask because according to the man page the opensm configuration file is 
in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Thanks,

Tom

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Hal Rosenstock]

Hi Tom,

On 7/12/10, Tom Ammon <tom.ammon-wbocuHtxKic@public.gmane.org> wrote:
>   Hi,
>
> I have some basic questions about IB partitions.
>
> Can an HCA port belong to more than 1 partition at a time?

Yes.

> How do you configure partitions with opensm? From reading the opensmd
> man page, it looks like you just create a file called
> /etc/osm-partitions.conf, with port GUIDs and such, but is this current?

The default location depends on how OpenSM is configured/built.

> I ask because according to the man page the opensm configuration file is
> in /etc/opensm/ . Can you tell opensm where to look for the partitions file?

Yes, with either the -P option on the command line or
partition_config_file line in the options file.

-- Hal

> Thanks,
>
> Tom
>
> --
> Tom Ammon
> Network Engineer
> Office: 801.587.0976
> Mobile: 801.674.9273
>
> Center for High Performance Computing
> University of Utah
> http://www.chpc.utah.edu
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Hal Rosenstock]

On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
<hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Hi Tom,
>
> On 7/12/10, Tom Ammon <tom.ammon-wbocuHtxKic@public.gmane.org> wrote:
>>   Hi,
>>
>> I have some basic questions about IB partitions.
>>
>> Can an HCA port belong to more than 1 partition at a time?
>
> Yes.
>
>> How do you configure partitions with opensm? From reading the opensmd
>> man page, it looks like you just create a file called
>> /etc/osm-partitions.conf, with port GUIDs and such, but is this current?
>
> The default location depends on how OpenSM is configured/built.

I missed this the first time around:

and yes, the syntax indicated is current.

-- Hal

>
>> I ask because according to the man page the opensm configuration file is
>> in /etc/opensm/ . Can you tell opensm where to look for the partitions file?
>
> Yes, with either the -P option on the command line or
> partition_config_file line in the options file.
>
> -- Hal
>
>> Thanks,
>>
>> Tom
>>
>> --
>> Tom Ammon
>> Network Engineer
>> Office: 801.587.0976
>> Mobile: 801.674.9273
>>
>> Center for High Performance Computing
>> University of Utah
>> http://www.chpc.utah.edu
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Tom Ammon]

  Okay, how about this.

With ethernet and a linux server, I can set up up aliases (virtual IP 
addresses) and assign multiple IPs to a single NIC. Can this be done 
with an HCA and IPoIB? For example, could I configure ib0:1 to be 
192.168.0.1/24 and ib0:2 to be 10.0.0.1/24?

I have tried this on a RHEL5 box and it doesn't seem to want to accept 
it, but I don't know if there is another piece somewhere that I need to 
configure.

Tom

On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
>> Hi Tom,
>>
>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>  wrote:
>>>    Hi,
>>>
>>> I have some basic questions about IB partitions.
>>>
>>> Can an HCA port belong to more than 1 partition at a time?
>> Yes.
>>
>>> How do you configure partitions with opensm? From reading the opensmd
>>> man page, it looks like you just create a file called
>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this current?
>> The default location depends on how OpenSM is configured/built.
> I missed this the first time around:
>
> and yes, the syntax indicated is current.
>
> -- Hal
>
>>> I ask because according to the man page the opensm configuration file is
>>> in /etc/opensm/ . Can you tell opensm where to look for the partitions file?
>> Yes, with either the -P option on the command line or
>> partition_config_file line in the options file.
>>
>> -- Hal
>>
>>> Thanks,
>>>
>>> Tom
>>>
>>> --
>>> Tom Ammon
>>> Network Engineer
>>> Office: 801.587.0976
>>> Mobile: 801.674.9273
>>>
>>> Center for High Performance Computing
>>> University of Utah
>>> http://www.chpc.utah.edu
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Hal Rosenstock]

On Mon, Jul 12, 2010 at 3:28 PM, Tom Ammon <tom.ammon-wbocuHtxKic@public.gmane.org> wrote:
>  Okay, how about this.
>
> With ethernet and a linux server, I can set up up aliases (virtual IP
> addresses) and assign multiple IPs to a single NIC. Can this be done with an
> HCA and IPoIB?

Yes.

> For example, could I configure ib0:1 to be 192.168.0.1/24 and
> ib0:2 to be 10.0.0.1/24?

See linux/Documentation/infiniband/ipoib.txt on how to do this.

> I have tried this on a RHEL5 box and it doesn't seem to want to accept it,
> but I don't know if there is another piece somewhere that I need to
> configure.

The procedure is a little different. You'll need to create a child
interface (on a partition) first and then you will be able to
configure it as follows:

echo 0x8001 > /sys/class/net/ib0/create_child
ifconfig ib0.8001 ...

Note that you'll want the 0x8000 bit on for full membership.

-- Hal

> Tom
>
> On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
>>
>> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
>> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
>>>
>>> Hi Tom,
>>>
>>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>  wrote:
>>>>
>>>>   Hi,
>>>>
>>>> I have some basic questions about IB partitions.
>>>>
>>>> Can an HCA port belong to more than 1 partition at a time?
>>>
>>> Yes.
>>>
>>>> How do you configure partitions with opensm? From reading the opensmd
>>>> man page, it looks like you just create a file called
>>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this current?
>>>
>>> The default location depends on how OpenSM is configured/built.
>>
>> I missed this the first time around:
>>
>> and yes, the syntax indicated is current.
>>
>> -- Hal
>>
>>>> I ask because according to the man page the opensm configuration file is
>>>> in /etc/opensm/ . Can you tell opensm where to look for the partitions
>>>> file?
>>>
>>> Yes, with either the -P option on the command line or
>>> partition_config_file line in the options file.
>>>
>>> -- Hal
>>>
>>>> Thanks,
>>>>
>>>> Tom
>>>>
>>>> --
>>>> Tom Ammon
>>>> Network Engineer
>>>> Office: 801.587.0976
>>>> Mobile: 801.674.9273
>>>>
>>>> Center for High Performance Computing
>>>> University of Utah
>>>> http://www.chpc.utah.edu
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>
> --
> Tom Ammon
> Network Engineer
> Office: 801.587.0976
> Mobile: 801.674.9273
>
> Center for High Performance Computing
> University of Utah
> http://www.chpc.utah.edu
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Tom Ammon]

  Hal,

> The procedure is a little different. You'll need to create a child
> interface (on a partition) first and then you will be able to
> configure it as follows:
>
> echo 0x8001>  /sys/class/net/ib0/create_child
> ifconfig ib0.8001 ...
>
> Note that you'll want the 0x8000 bit on for full membership.
What does "full membership" mean? And are you saying that for full 
membership, any value beginning with 0x8 would work? Can an HCA have 
full membership on multiple partitions simultaneously?

Thanks,

Tom


> -- Hal
>
>> Tom
>>
>> On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
>>> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
>>> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>    wrote:
>>>> Hi Tom,
>>>>
>>>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>    wrote:
>>>>>    Hi,
>>>>>
>>>>> I have some basic questions about IB partitions.
>>>>>
>>>>> Can an HCA port belong to more than 1 partition at a time?
>>>> Yes.
>>>>
>>>>> How do you configure partitions with opensm? From reading the opensmd
>>>>> man page, it looks like you just create a file called
>>>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this current?
>>>> The default location depends on how OpenSM is configured/built.
>>> I missed this the first time around:
>>>
>>> and yes, the syntax indicated is current.
>>>
>>> -- Hal
>>>
>>>>> I ask because according to the man page the opensm configuration file is
>>>>> in /etc/opensm/ . Can you tell opensm where to look for the partitions
>>>>> file?
>>>> Yes, with either the -P option on the command line or
>>>> partition_config_file line in the options file.
>>>>
>>>> -- Hal
>>>>
>>>>> Thanks,
>>>>>
>>>>> Tom
>>>>>
>>>>> --
>>>>> Tom Ammon
>>>>> Network Engineer
>>>>> Office: 801.587.0976
>>>>> Mobile: 801.674.9273
>>>>>
>>>>> Center for High Performance Computing
>>>>> University of Utah
>>>>> http://www.chpc.utah.edu
>>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>> --
>> Tom Ammon
>> Network Engineer
>> Office: 801.587.0976
>> Mobile: 801.674.9273
>>
>> Center for High Performance Computing
>> University of Utah
>> http://www.chpc.utah.edu
>>
>>

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Hal Rosenstock]

Tom,

On Tue, Jul 13, 2010 at 12:50 PM, Tom Ammon <tom.ammon-wbocuHtxKic@public.gmane.org> wrote:
>  Hal,
>
>> The procedure is a little different. You'll need to create a child
>> interface (on a partition) first and then you will be able to
>> configure it as follows:
>>
>> echo 0x8001>  /sys/class/net/ib0/create_child
>> ifconfig ib0.8001 ...
>>
>> Note that you'll want the 0x8000 bit on for full membership.
>
> What does "full membership" mean?

For a partition, full members can talk with both full and limited
members whereas limited members can only talk with full members.

> And are you saying that for full
> membership, any value beginning with 0x8 would work?

The 0x8000 bit is the (full) membership bit so 0x8abc is the pkey for
full membership in the 0xabc partition. 15 bits of pkey can be used
for partition although my example above only used 12 bits.

> Can an HCA have full membership on multiple partitions simultaneously?

Yes.

-- Hal

> Thanks,
>
> Tom
>
>
>> -- Hal
>>
>>> Tom
>>>
>>> On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
>>>>
>>>> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
>>>> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>    wrote:
>>>>>
>>>>> Hi Tom,
>>>>>
>>>>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>    wrote:
>>>>>>
>>>>>>   Hi,
>>>>>>
>>>>>> I have some basic questions about IB partitions.
>>>>>>
>>>>>> Can an HCA port belong to more than 1 partition at a time?
>>>>>
>>>>> Yes.
>>>>>
>>>>>> How do you configure partitions with opensm? From reading the opensmd
>>>>>> man page, it looks like you just create a file called
>>>>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this
>>>>>> current?
>>>>>
>>>>> The default location depends on how OpenSM is configured/built.
>>>>
>>>> I missed this the first time around:
>>>>
>>>> and yes, the syntax indicated is current.
>>>>
>>>> -- Hal
>>>>
>>>>>> I ask because according to the man page the opensm configuration file
>>>>>> is
>>>>>> in /etc/opensm/ . Can you tell opensm where to look for the partitions
>>>>>> file?
>>>>>
>>>>> Yes, with either the -P option on the command line or
>>>>> partition_config_file line in the options file.
>>>>>
>>>>> -- Hal
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Tom
>>>>>>
>>>>>> --
>>>>>> Tom Ammon
>>>>>> Network Engineer
>>>>>> Office: 801.587.0976
>>>>>> Mobile: 801.674.9273
>>>>>>
>>>>>> Center for High Performance Computing
>>>>>> University of Utah
>>>>>> http://www.chpc.utah.edu
>>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>>>>>> in
>>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>> --
>>> Tom Ammon
>>> Network Engineer
>>> Office: 801.587.0976
>>> Mobile: 801.674.9273
>>>
>>> Center for High Performance Computing
>>> University of Utah
>>> http://www.chpc.utah.edu
>>>
>>>
>
> --
> Tom Ammon
> Network Engineer
> Office: 801.587.0976
> Mobile: 801.674.9273
>
> Center for High Performance Computing
> University of Utah
> http://www.chpc.utah.edu
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Tom Ammon]

  Hal,

I have some more partition questions. Do I have to configure the nodes 
as members of partition(s) both in opensm and on the individual node (if 
the node is a member of two partitions)? And what about the switches - 
do the switches need be configured as part of any partition?

Tom

On 7/13/2010 11:44 AM, Hal Rosenstock wrote:
> Tom,
>
> On Tue, Jul 13, 2010 at 12:50 PM, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>  wrote:
>>   Hal,
>>
>>> The procedure is a little different. You'll need to create a child
>>> interface (on a partition) first and then you will be able to
>>> configure it as follows:
>>>
>>> echo 0x8001>    /sys/class/net/ib0/create_child
>>> ifconfig ib0.8001 ...
>>>
>>> Note that you'll want the 0x8000 bit on for full membership.
>> What does "full membership" mean?
> For a partition, full members can talk with both full and limited
> members whereas limited members can only talk with full members.
>
>> And are you saying that for full
>> membership, any value beginning with 0x8 would work?
> The 0x8000 bit is the (full) membership bit so 0x8abc is the pkey for
> full membership in the 0xabc partition. 15 bits of pkey can be used
> for partition although my example above only used 12 bits.
>
>> Can an HCA have full membership on multiple partitions simultaneously?
> Yes.
>
> -- Hal
>
>> Thanks,
>>
>> Tom
>>
>>
>>> -- Hal
>>>
>>>> Tom
>>>>
>>>> On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
>>>>> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
>>>>> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>      wrote:
>>>>>> Hi Tom,
>>>>>>
>>>>>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>      wrote:
>>>>>>>    Hi,
>>>>>>>
>>>>>>> I have some basic questions about IB partitions.
>>>>>>>
>>>>>>> Can an HCA port belong to more than 1 partition at a time?
>>>>>> Yes.
>>>>>>
>>>>>>> How do you configure partitions with opensm? From reading the opensmd
>>>>>>> man page, it looks like you just create a file called
>>>>>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this
>>>>>>> current?
>>>>>> The default location depends on how OpenSM is configured/built.
>>>>> I missed this the first time around:
>>>>>
>>>>> and yes, the syntax indicated is current.
>>>>>
>>>>> -- Hal
>>>>>
>>>>>>> I ask because according to the man page the opensm configuration file
>>>>>>> is
>>>>>>> in /etc/opensm/ . Can you tell opensm where to look for the partitions
>>>>>>> file?
>>>>>> Yes, with either the -P option on the command line or
>>>>>> partition_config_file line in the options file.
>>>>>>
>>>>>> -- Hal
>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Tom
>>>>>>>
>>>>>>> --
>>>>>>> Tom Ammon
>>>>>>> Network Engineer
>>>>>>> Office: 801.587.0976
>>>>>>> Mobile: 801.674.9273
>>>>>>>
>>>>>>> Center for High Performance Computing
>>>>>>> University of Utah
>>>>>>> http://www.chpc.utah.edu
>>>>>>>
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe linux-rdma"
>>>>>>> in
>>>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>> --
>>>> Tom Ammon
>>>> Network Engineer
>>>> Office: 801.587.0976
>>>> Mobile: 801.674.9273
>>>>
>>>> Center for High Performance Computing
>>>> University of Utah
>>>> http://www.chpc.utah.edu
>>>>
>>>>
>> --
>> Tom Ammon
>> Network Engineer
>> Office: 801.587.0976
>> Mobile: 801.674.9273
>>
>> Center for High Performance Computing
>> University of Utah
>> http://www.chpc.utah.edu
>>
>>

-- 
Tom Ammon
Network Engineer
Office: 801.587.0976
Mobile: 801.674.9273

Center for High Performance Computing
University of Utah
http://www.chpc.utah.edu

--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: basic questions about partitions

[Hal Rosenstock]

Tom,

On Tue, Jul 13, 2010 at 3:44 PM, Tom Ammon <tom.ammon-wbocuHtxKic@public.gmane.org> wrote:
>  Hal,
>
> I have some more partition questions. Do I have to configure the nodes as
> members of partition(s) both in opensm and on the individual node (if the
> node is a member of two partitions)?

OpenSM needs the configuration we've been discussing. It pushes the
pkeys to the end ports. All that needs to be done at the end port is
whatever the ULP/application needs. I sent the info for IPoIB. The
partitions used there need to match how OpenSM is configured.

> And what about the switches - do the
> switches need be configured as part of any partition?

OpenSM automatically calculates the peer switch port partitions based
on the end port partitions so no explicit configuration is required
for switch ports.

-- Hal

> Tom
>
> On 7/13/2010 11:44 AM, Hal Rosenstock wrote:
>>
>> Tom,
>>
>> On Tue, Jul 13, 2010 at 12:50 PM, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>  wrote:
>>>
>>>  Hal,
>>>
>>>> The procedure is a little different. You'll need to create a child
>>>> interface (on a partition) first and then you will be able to
>>>> configure it as follows:
>>>>
>>>> echo 0x8001>    /sys/class/net/ib0/create_child
>>>> ifconfig ib0.8001 ...
>>>>
>>>> Note that you'll want the 0x8000 bit on for full membership.
>>>
>>> What does "full membership" mean?
>>
>> For a partition, full members can talk with both full and limited
>> members whereas limited members can only talk with full members.
>>
>>> And are you saying that for full
>>> membership, any value beginning with 0x8 would work?
>>
>> The 0x8000 bit is the (full) membership bit so 0x8abc is the pkey for
>> full membership in the 0xabc partition. 15 bits of pkey can be used
>> for partition although my example above only used 12 bits.
>>
>>> Can an HCA have full membership on multiple partitions simultaneously?
>>
>> Yes.
>>
>> -- Hal
>>
>>> Thanks,
>>>
>>> Tom
>>>
>>>
>>>> -- Hal
>>>>
>>>>> Tom
>>>>>
>>>>> On 7/12/2010 1:06 PM, Hal Rosenstock wrote:
>>>>>>
>>>>>> On Mon, Jul 12, 2010 at 3:03 PM, Hal Rosenstock
>>>>>> <hal.rosenstock-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>      wrote:
>>>>>>>
>>>>>>> Hi Tom,
>>>>>>>
>>>>>>> On 7/12/10, Tom Ammon<tom.ammon-wbocuHtxKic@public.gmane.org>      wrote:
>>>>>>>>
>>>>>>>>   Hi,
>>>>>>>>
>>>>>>>> I have some basic questions about IB partitions.
>>>>>>>>
>>>>>>>> Can an HCA port belong to more than 1 partition at a time?
>>>>>>>
>>>>>>> Yes.
>>>>>>>
>>>>>>>> How do you configure partitions with opensm? From reading the
>>>>>>>> opensmd
>>>>>>>> man page, it looks like you just create a file called
>>>>>>>> /etc/osm-partitions.conf, with port GUIDs and such, but is this
>>>>>>>> current?
>>>>>>>
>>>>>>> The default location depends on how OpenSM is configured/built.
>>>>>>
>>>>>> I missed this the first time around:
>>>>>>
>>>>>> and yes, the syntax indicated is current.
>>>>>>
>>>>>> -- Hal
>>>>>>
>>>>>>>> I ask because according to the man page the opensm configuration
>>>>>>>> file
>>>>>>>> is
>>>>>>>> in /etc/opensm/ . Can you tell opensm where to look for the
>>>>>>>> partitions
>>>>>>>> file?
>>>>>>>
>>>>>>> Yes, with either the -P option on the command line or
>>>>>>> partition_config_file line in the options file.
>>>>>>>
>>>>>>> -- Hal
>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Tom
>>>>>>>>
>>>>>>>> --
>>>>>>>> Tom Ammon
>>>>>>>> Network Engineer
>>>>>>>> Office: 801.587.0976
>>>>>>>> Mobile: 801.674.9273
>>>>>>>>
>>>>>>>> Center for High Performance Computing
>>>>>>>> University of Utah
>>>>>>>> http://www.chpc.utah.edu
>>>>>>>>
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> linux-rdma"
>>>>>>>> in
>>>>>>>> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>> --
>>>>> Tom Ammon
>>>>> Network Engineer
>>>>> Office: 801.587.0976
>>>>> Mobile: 801.674.9273
>>>>>
>>>>> Center for High Performance Computing
>>>>> University of Utah
>>>>> http://www.chpc.utah.edu
>>>>>
>>>>>
>>> --
>>> Tom Ammon
>>> Network Engineer
>>> Office: 801.587.0976
>>> Mobile: 801.674.9273
>>>
>>> Center for High Performance Computing
>>> University of Utah
>>> http://www.chpc.utah.edu
>>>
>>>
>
> --
> Tom Ammon
> Network Engineer
> Office: 801.587.0976
> Mobile: 801.674.9273
>
> Center for High Performance Computing
> University of Utah
> http://www.chpc.utah.edu
>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html