[RFC] Proposal to change Node Description naming scheme for HCA's

[RFC] Proposal to change Node Description naming scheme for HCA's

[Ira Weiny]

First, a question: what package installs the openibd script in OFED?  For the
life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]

Right now the "standard" for node description is, AFAIK, "<hostname> HCA-<num>",
where num is simply a counter for the HCA's as they are found in
/sys/class/infiniband.

The problem is resolving this "random" HCA number to an actual HCA on the
host.  I thought about including the node description in ibstat but that seems
a bit short sighted.  I think the better solution would be to append the hca
name (ie mlx4_X, qibX, etc) to the hostname for the Node Description.

Hacking the RHEL start up script is really easy to do this and results in nice
names on the fabric which are easily resolved by the infiniband-diags,
ibverbs, and perftest utilities.

bash-4.1# ibhosts
Ca      : 0x0002c90300325280 ports 1 "ending mlx4_2"
Ca      : 0x001175000079da38 ports 1 "happy qib0"
Ca      : 0x0002c90300108f2e ports 1 "ending mlx4_1"
Ca      : 0x001175000077d90e ports 2 "ending qib0"
Ca      : 0x0002c903004bebda ports 2 "happy mlx4_0"
bash-4.1# hostname
happy
bash-4.1# ibstat mlx4_0
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.8.600
...
bash-4.1# ibv_rc_pingpong -d mlx4_0
  local address:  LID 0x0008, QPN 0x16004a, PSN 0x2e8316
...
bash-4.1# rdma_bw -d mlx4_0
6089: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 |
...

I realize this is really a distro thing but it would be nice if we could agree
to change the current "standard".

I can send a patch for RHEL and OFED (if I someone can point me to the openibd
script or srpm).

Thoughts?
Ira


[*] Last I knew openibd does the same as RHEL's rdma start up script in this regard.

-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC] Proposal to change Node Description naming scheme for HCA's

[Heinz, Michael William]

I like this idea - but it reminds me of a related issue I raised a while back: nodes can often set the HCA description before they received a hostname from DHCP - in which case you end up with saqueries full of "localhost HCA-1".

At the time, QLogic's proposal was to modify the kernel stack so that it extracted the hostname at the time of the query instead of at boot time - but the linux_rdma list did not like that solution.

Any ideas on how we could solve the hostname problem while we're changing the description?

As for what installs the openibd script, I'm pretty sure that's part of ofa_kernel.

-----Original Message-----
From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Ira Weiny
Sent: Friday, March 30, 2012 8:39 PM
To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
Cc: Doug Ledford; Bob Ciotti; James Silva
Subject: [RFC] Proposal to change Node Description naming scheme for HCA's

First, a question: what package installs the openibd script in OFED?  For the life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]

Right now the "standard" for node description is, AFAIK, "<hostname> HCA-<num>", where num is simply a counter for the HCA's as they are found in /sys/class/infiniband.

The problem is resolving this "random" HCA number to an actual HCA on the host.  I thought about including the node description in ibstat but that seems a bit short sighted.  I think the better solution would be to append the hca name (ie mlx4_X, qibX, etc) to the hostname for the Node Description.

Hacking the RHEL start up script is really easy to do this and results in nice names on the fabric which are easily resolved by the infiniband-diags, ibverbs, and perftest utilities.

bash-4.1# ibhosts
Ca      : 0x0002c90300325280 ports 1 "ending mlx4_2"
Ca      : 0x001175000079da38 ports 1 "happy qib0"
Ca      : 0x0002c90300108f2e ports 1 "ending mlx4_1"
Ca      : 0x001175000077d90e ports 2 "ending qib0"
Ca      : 0x0002c903004bebda ports 2 "happy mlx4_0"
bash-4.1# hostname
happy
bash-4.1# ibstat mlx4_0
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.8.600
...
bash-4.1# ibv_rc_pingpong -d mlx4_0
  local address:  LID 0x0008, QPN 0x16004a, PSN 0x2e8316 ...
bash-4.1# rdma_bw -d mlx4_0
6089: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 | ...

I realize this is really a distro thing but it would be nice if we could agree to change the current "standard".

I can send a patch for RHEL and OFED (if I someone can point me to the openibd script or srpm).

Thoughts?
Ira


[*] Last I knew openibd does the same as RHEL's rdma start up script in this regard.

--
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Jason Gunthorpe]

On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:

> Any ideas on how we could solve the hostname problem while we're
> changing the description?

The node description needs to be set from the DCHP notifier script
chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule
triggered on device insertion.

It should probably not be set from the openibd script..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC] Proposal to change Node Description naming scheme for HCA's

[Heinz, Michael William]

I'd agree but the network up/down functionality tends to vary significantly from distro to distro. There's also the question of "what package would we add this functionality to?" I mean, I assume it wouldn't be part of ofa_kernel. Might actually be the installer script that has to patch the DHCP chain.

-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org] 
Sent: Monday, April 02, 2012 2:46 PM
To: Heinz, Michael William
Cc: Ira Weiny; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Doug Ledford; Bob Ciotti; James Silva
Subject: Re: [RFC] Proposal to change Node Description naming scheme for HCA's

On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:

> Any ideas on how we could solve the hostname problem while we're 
> changing the description?

The node description needs to be set from the DCHP notifier script chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule triggered on device insertion.

It should probably not be set from the openibd script..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Ira Weiny]

On Mon, 2 Apr 2012 12:45:45 -0600
Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:
> 
> > Any ideas on how we could solve the hostname problem while we're
> > changing the description?
> 
> The node description needs to be set from the DCHP notifier script
> chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule
> triggered on device insertion.

Jason, I'm confused.  Do you mean DHCP?

It seems are you indicating you would like to see if[up/down] bring ports up/down like they do for IP?

On my ubuntu box (the closest thing I have to Debian) it looks like ifupdown owns /etc/network/if-up.d and that is not specific to DHCP.  I don't think DHCP should be required for IB.

Using udev to set this __seems__ like a better idea although I have not prototyped it.

> 
> It should probably not be set from the openibd script..

I agree there might be better ways but I am not sure I follow your proposal.  Furthermore, I don't know that a start up script of some sort is all that evil.

Finally, I think Michael brings up a good point about which package should own any such scripts.

Ira

> 
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Jason Gunthorpe]

On Tue, Apr 03, 2012 at 07:14:42PM -0700, Ira Weiny wrote:
> On Mon, 2 Apr 2012 12:45:45 -0600
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> > On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:
> > 
> > > Any ideas on how we could solve the hostname problem while we're
> > > changing the description?
> > 
> > The node description needs to be set from the DCHP notifier script
> > chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule
> > triggered on device insertion.
> 
> Jason, I'm confused.  Do you mean DHCP?

Yes..

> It seems are you indicating you would like to see if[up/down] bring
> ports up/down like they do for IP?

No..

> On my ubuntu box (the closest thing I have to Debian) it looks like
> ifupdown owns /etc/network/if-up.d and that is not specific to DHCP.
> I don't think DHCP should be required for IB.

if-up.d/ is a '.d' directory, the idea is individual packages drop
their script files in the directory and the system runs them at
defined times. No package owns the directory.

The purpose of placing a hook here is this call path is used when the
hostname changes via DHCP so you can have a chance to reset the node
description.

> Using udev to set this __seems__ like a better idea although I have
> not prototyped it.

The purpose of the udev hook path is to set the node description on
initial device insertion, which may be before, or after the DHCP
process completes, in such cases.

It may also be before or after the openibd script.. Having init.d
scripts depend on the ordering of hardware discovery and module
loading is considered sketchy these days with all the parallel boot
fancyness and what not.

IMHO we should have a udev rule that also loads the higher level IB
modules when any RDMA capable device is discovered, including the mlx4
IB layer, uverbs, umad, etc. This way systems that have RDMA will load
the right modules and systems that don't, won't. Fully supporting hot
plug, of course.

This would broadly eliminate the openibd script, integrate more
correctly with the modern distro world, be better prepared for systemd
and just be an overall better example for distros to follow :)

> > It should probably not be set from the openibd script..
> 
> I agree there might be better ways but I am not sure I follow your
> proposal.  Furthermore, I don't know that a start up script of some
> sort is all that evil.
> 
> Finally, I think Michael brings up a good point about which package
> should own any such scripts.

udev is like if-up.d/, there is a rules directory packages can drop
hook scripts into that run at the appropriate time.
 
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RE: [RFC] Proposal to change Node Description naming scheme for HCA's

[Heinz, Michael William]

I agree that setting it via the network scripts is preferable - if we can find a fairly consistent way to do it across distros.

Ira - one thing: the qlogic patch didn't hardcode the node description, it simply defaulted to using the current hostname. The name could still be overwritten by the user as in stock OFED.

-----Original Message-----
From: Jason Gunthorpe [mailto:jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org] 
Sent: Wednesday, April 04, 2012 1:12 PM
To: Ira Weiny
Cc: Heinz, Michael William; linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org; Doug Ledford; Bob Ciotti; James Silva
Subject: Re: [RFC] Proposal to change Node Description naming scheme for HCA's

On Tue, Apr 03, 2012 at 07:14:42PM -0700, Ira Weiny wrote:
> On Mon, 2 Apr 2012 12:45:45 -0600
> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> 
> > On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:
> > 
> > > Any ideas on how we could solve the hostname problem while we're 
> > > changing the description?
> > 
> > The node description needs to be set from the DCHP notifier script 
> > chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule 
> > triggered on device insertion.
> 
> Jason, I'm confused.  Do you mean DHCP?

Yes..

> It seems are you indicating you would like to see if[up/down] bring 
> ports up/down like they do for IP?

No..

> On my ubuntu box (the closest thing I have to Debian) it looks like 
> ifupdown owns /etc/network/if-up.d and that is not specific to DHCP.
> I don't think DHCP should be required for IB.

if-up.d/ is a '.d' directory, the idea is individual packages drop their script files in the directory and the system runs them at defined times. No package owns the directory.

The purpose of placing a hook here is this call path is used when the hostname changes via DHCP so you can have a chance to reset the node description.

> Using udev to set this __seems__ like a better idea although I have 
> not prototyped it.

The purpose of the udev hook path is to set the node description on initial device insertion, which may be before, or after the DHCP process completes, in such cases.

It may also be before or after the openibd script.. Having init.d scripts depend on the ordering of hardware discovery and module loading is considered sketchy these days with all the parallel boot fancyness and what not.

IMHO we should have a udev rule that also loads the higher level IB modules when any RDMA capable device is discovered, including the mlx4 IB layer, uverbs, umad, etc. This way systems that have RDMA will load the right modules and systems that don't, won't. Fully supporting hot plug, of course.

This would broadly eliminate the openibd script, integrate more correctly with the modern distro world, be better prepared for systemd and just be an overall better example for distros to follow :)

> > It should probably not be set from the openibd script..
> 
> I agree there might be better ways but I am not sure I follow your 
> proposal.  Furthermore, I don't know that a start up script of some 
> sort is all that evil.
> 
> Finally, I think Michael brings up a good point about which package 
> should own any such scripts.

udev is like if-up.d/, there is a rules directory packages can drop hook scripts into that run at the appropriate time.
 
Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Jason Gunthorpe]

On Wed, Apr 04, 2012 at 05:17:48PM +0000, Heinz, Michael William wrote:
> I agree that setting it via the network scripts is preferable - if
> we can find a fairly consistent way to do it across distros.

OFED only supports two distros, we don't really need to be
distro-agnostic. At this point it is better to integrate well with
each supported distro and 'do things the distro way'. The OFA
direction is now 'upstream first' and 'better distro collaboration'..

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Ira Weiny]

On Wed, 4 Apr 2012 11:12:00 -0600
Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:

> On Tue, Apr 03, 2012 at 07:14:42PM -0700, Ira Weiny wrote:
> > On Mon, 2 Apr 2012 12:45:45 -0600
> > Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
> > 
> > > On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:
> > > 
> > > > Any ideas on how we could solve the hostname problem while we're
> > > > changing the description?
> > > 
> > > The node description needs to be set from the DCHP notifier script
> > > chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule
> > > triggered on device insertion.
> > 
> > Jason, I'm confused.  Do you mean DHCP?
> 
> Yes..
> 
> > It seems are you indicating you would like to see if[up/down] bring
> > ports up/down like they do for IP?
> 
> No..
> 
> > On my ubuntu box (the closest thing I have to Debian) it looks like
> > ifupdown owns /etc/network/if-up.d and that is not specific to DHCP.
> > I don't think DHCP should be required for IB.
> 
> if-up.d/ is a '.d' directory, the idea is individual packages drop
> their script files in the directory and the system runs them at
> defined times. No package owns the directory.
> 
> The purpose of placing a hook here is this call path is used when the
> hostname changes via DHCP so you can have a chance to reset the node
> description.

Ok, I see what you are saying now.

> 
> > Using udev to set this __seems__ like a better idea although I have
> > not prototyped it.
> 
> The purpose of the udev hook path is to set the node description on
> initial device insertion, which may be before, or after the DHCP
> process completes, in such cases.

Yes I think doing this on device insertion is a good thing.

> 
> It may also be before or after the openibd script.. Having init.d
> scripts depend on the ordering of hardware discovery and module
> loading is considered sketchy these days with all the parallel boot
> fancyness and what not.
> 
> IMHO we should have a udev rule that also loads the higher level IB
> modules when any RDMA capable device is discovered, including the mlx4
> IB layer, uverbs, umad, etc. This way systems that have RDMA will load
> the right modules and systems that don't, won't. Fully supporting hot
> plug, of course.

Great idea perhaps you have these rules files and could supply patches to the
various packages?  ie libibumad should probably manage its own rules.

> 
> This would broadly eliminate the openibd script, integrate more
> correctly with the modern distro world, be better prepared for systemd
> and just be an overall better example for distros to follow :)
> 
> > > It should probably not be set from the openibd script..
> > 
> > I agree there might be better ways but I am not sure I follow your
> > proposal.  Furthermore, I don't know that a start up script of some
> > sort is all that evil.
> > 
> > Finally, I think Michael brings up a good point about which package
> > should own any such scripts.
> 
> udev is like if-up.d/, there is a rules directory packages can drop
> hook scripts into that run at the appropriate time.

As I said good idea but each rule file need to be owned by some package.
Preferably the package which needs them.  Node description is "universal" so
who owns its udev rule file, dhcp script file, etc?  I guess this is where the
RHEL rdma package comes in.

One final issue, is there a hook to be able to do this when someone runs
"hostname"?

Ira

>  
> Jason


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Doug Ledford]

[-- Attachment #1: Type: text/plain, Size: 5219 bytes --]

On 04/04/2012 01:12 PM, Jason Gunthorpe wrote:
> On Tue, Apr 03, 2012 at 07:14:42PM -0700, Ira Weiny wrote:
>> On Mon, 2 Apr 2012 12:45:45 -0600
>> Jason Gunthorpe <jgunthorpe-ePGOBjL8dl3ta4EC/59zMFaTQe2KTcn/@public.gmane.org> wrote:
>>
>>> On Mon, Apr 02, 2012 at 03:27:35PM +0000, Heinz, Michael William wrote:
>>>
>>>> Any ideas on how we could solve the hostname problem while we're
>>>> changing the description?
>>>
>>> The node description needs to be set from the DCHP notifier script
>>> chain (eg /etc/network/if-up.d/ on Debian) and also from a udev rule
>>> triggered on device insertion.
>>
>> Jason, I'm confused.  Do you mean DHCP?
> 
> Yes..
> 
>> It seems are you indicating you would like to see if[up/down] bring
>> ports up/down like they do for IP?
> 
> No..
> 
>> On my ubuntu box (the closest thing I have to Debian) it looks like
>> ifupdown owns /etc/network/if-up.d and that is not specific to DHCP.
>> I don't think DHCP should be required for IB.
> 
> if-up.d/ is a '.d' directory, the idea is individual packages drop
> their script files in the directory and the system runs them at
> defined times. No package owns the directory.
> 
> The purpose of placing a hook here is this call path is used when the
> hostname changes via DHCP so you can have a chance to reset the node
> description.
> 
>> Using udev to set this __seems__ like a better idea although I have
>> not prototyped it.
> 
> The purpose of the udev hook path is to set the node description on
> initial device insertion, which may be before, or after the DHCP
> process completes, in such cases.

Well, a udev rule is guaranteed to be before dhcp completes *on that
device*.  It might be that dhcp has completed on another device and that
the other device is actually where the hostname is pulled, but the udev
rule will always be before this given device in the rule has completed dhcp.

> It may also be before or after the openibd script..

Non hot-plug events are going to end up always being before the openibd
script.  This is merely a side effect of the fact that udev start is
called before the normal init script processing is started (this all
changes somewhat in the systemd world though).

> Having init.d
> scripts depend on the ordering of hardware discovery and module
> loading is considered sketchy these days with all the parallel boot
> fancyness and what not.

I would agree with that.  However...

> IMHO we should have a udev rule that also loads the higher level IB
> modules when any RDMA capable device is discovered, including the mlx4
> IB layer, uverbs, umad, etc. This way systems that have RDMA will load
> the right modules and systems that don't, won't. Fully supporting hot
> plug, of course.

One of the niceties of the rdma/openibd init script is the ability to
completely reload the stack.  That goes away if we switch to a udev load
of the stack (well, unless you now create a script in /sbin and have the
udev rule call that script, but the script should no longer be in the
initrddir if it's not an init script).

> This would broadly eliminate the openibd script, integrate more
> correctly with the modern distro world, be better prepared for systemd
> and just be an overall better example for distros to follow :)

This is mostly true, but you do have to sacrifice the one item I listed
above.

>>> It should probably not be set from the openibd script..
>>
>> I agree there might be better ways but I am not sure I follow your
>> proposal.  Furthermore, I don't know that a start up script of some
>> sort is all that evil.
>>
>> Finally, I think Michael brings up a good point about which package
>> should own any such scripts.
> 
> udev is like if-up.d/, there is a rules directory packages can drop
> hook scripts into that run at the appropriate time.

Correct.  On Red Hat Enterprise Linux I could have the rdma package drop
in an /sbin/rdma script that would bring the stack up (and possibly
reload it, but I'm a bit sketchy on that idea given that this would no
longer be an init script but something else), have it drop a small
script in /etc/dhcp/dhclient.d/ to set the node description after dhcp
completes (the script in /sbin would also have to set the node
description from whatever information is available on load just in case
the machine doesn't even use dhcp though), have it drop a rules file in
/etc/udev.d/rules.d for bringing up the stack on device discovery (this
one is a bit tricky though, you basically have to match against all
possible RDMA devices and bring up the stack on the presence of any one
of them, and your script that you call to bring the stack up needs to be
safe against parallel invocations), and then also own the /etc/rdma
directory and the whatever config files it places in there itself.

So, I'd say it's doable to change this over, but I'm not sure I would
recommend it in a minor point release.  I'd probably save this sort of
change for a major release update.


-- 
Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
              GPG KeyID: 0E572FDD
	      http://people.redhat.com/dledford



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 900 bytes --]

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Jason Gunthorpe]

On Thu, Apr 05, 2012 at 01:46:19PM -0400, Doug Ledford wrote:

> > The purpose of the udev hook path is to set the node description on
> > initial device insertion, which may be before, or after the DHCP
> > process completes, in such cases.
> 
> Well, a udev rule is guaranteed to be before dhcp completes *on that
> device*.  It might be that dhcp has completed on another device and that
> the other device is actually where the hostname is pulled, but the udev
> rule will always be before this given device in the rule has completed dhcp.

The case where the hostname comes from management ethernet DHCP is
what I was thinking of..
 
> > IMHO we should have a udev rule that also loads the higher level IB
> > modules when any RDMA capable device is discovered, including the mlx4
> > IB layer, uverbs, umad, etc. This way systems that have RDMA will load
> > the right modules and systems that don't, won't. Fully supporting hot
> > plug, of course.
> 
> One of the niceties of the rdma/openibd init script is the ability to
> completely reload the stack.  That goes away if we switch to a udev load
> of the stack (well, unless you now create a script in /sbin and have the
> udev rule call that script, but the script should no longer be in the
> initrddir if it's not an init script).

I'm not sure how valuable this really is, in practice we shouldn't be
unloading/reloading kernel modules. Every time I've run into problems
an unload of the mlx4 drivers alone was enough to fix them. The verbs,
mad, srp, etc, don't often need to be unloaded in my experience,
certainly never all as a unit. It is also difficult to make this
actually work since you need to be aware of all RDMA using stuff in
the system. You may also need to restart MPI? DB2? RAC? etc?

Do other large scale kernel stacks in RHEL have module restart script?
If so a rdma-stack-restart script in /usr/sbin would seem appropriate
to me.

This would also fix the bug of trying to unload modules on shutdown
via the init script, that is unnecessary/sketchy, IMHO.

> >> I agree there might be better ways but I am not sure I follow your
> >> proposal.  Furthermore, I don't know that a start up script of some
> >> sort is all that evil.
> >>
> >> Finally, I think Michael brings up a good point about which package
> >> should own any such scripts.
> > 
> > udev is like if-up.d/, there is a rules directory packages can drop
> > hook scripts into that run at the appropriate time.
 
> Correct.  On Red Hat Enterprise Linux I could have the rdma package drop
> in an /sbin/rdma script that would bring the stack up (and possibly
> reload it, but I'm a bit sketchy on that idea given that this would no
> longer be an init script but something else), have it drop a small
> script in /etc/dhcp/dhclient.d/ to set the node description after dhcp
> completes (the script in /sbin would also have to set the node
> description from whatever information is available on load just in case
> the machine doesn't even use dhcp though), have it drop a rules file in
> /etc/udev.d/rules.d for bringing up the stack on device discovery
> (this

> one is a bit tricky though, you basically have to match against all
> possible RDMA devices and bring up the stack on the presence of any one
> of them, and your script that you call to bring the stack up needs
> to be

There are a couple independent things here..

1) PCI auto probe will start mlx4_core (and other core drivers, that
   is getting common). This is already done..

2) When mlx4_core/etc is loaded udev should auto load mlx4_en/mlx4_ib/etc
   There is currently no auto-probe system at all for these layered
   drivers, it is annoying it doesn't work automatically :)

3) When a driver is loaded that has the sysfs child attributes
   {node_guid,node_desc,node_type} load the IB user space modules:
     ib_ucm
     ib_umad
     ib_uverbs
   Which pull in the core kernel modules too

4) 'some other sysfs test for iWarp/ROCEE' - load ib_uverbs. If there
    isn't a sysfs already we should add one.

5) Various ULPs can be loaded contingent on the ib_verbs/ib_cm module being
   loaded, or on-demand from other init scripts (e.g. load srp/iser in the
   same way the system loads iscsi)

udev alread has modprobe calling rules, so I think we are good without
any external scripts or complex serialization.

I don't have any rules already that do this, every time I've tangled
with udev I end up screaming at my monitor :(

> So, I'd say it's doable to change this over, but I'm not sure I would
> recommend it in a minor point release.  I'd probably save this sort of
> change for a major release update.

Sure, but it is the sort of thing I think is appropriate for OFED to
prototype so everyone can have something well tested for their next
major releases.

Jason
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Ira Weiny]

On Mon, 2 Apr 2012 15:27:35 +0000
"Heinz, Michael William" <michael.william.heinz-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org> wrote:

> I like this idea - but it reminds me of a related issue I raised a while back: nodes can often set the HCA description before they received a hostname from DHCP - in which case you end up with saqueries full of "localhost HCA-1".
> 
> At the time, QLogic's proposal was to modify the kernel stack so that it extracted the hostname at the time of the query instead of at boot time - but the linux_rdma list did not like that solution.
> 

The reason this is not acceptable is it hard codes hostname at a low level.  We should allow the user to override the Node Descriptor if they so desire.  In the case of the SM being faster than the boot scripts (rdma, openibd, udev, dhcp, or whatever).  It is better that the SM see the Node Description change trap and update the ND there.  As a final "fix" OpenSM has an "update_desc" option in the console to force a resweep if your names from the SA are wrong.

> Any ideas on how we could solve the hostname problem while we're changing the description?
> 
> As for what installs the openibd script, I'm pretty sure that's part of ofa_kernel.

Thanks, I will look for that.

Ira

> 
> -----Original Message-----
> From: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Ira Weiny
> Sent: Friday, March 30, 2012 8:39 PM
> To: linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Cc: Doug Ledford; Bob Ciotti; James Silva
> Subject: [RFC] Proposal to change Node Description naming scheme for HCA's
> 
> First, a question: what package installs the openibd script in OFED?  For the life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]
> 
> Right now the "standard" for node description is, AFAIK, "<hostname> HCA-<num>", where num is simply a counter for the HCA's as they are found in /sys/class/infiniband.
> 
> The problem is resolving this "random" HCA number to an actual HCA on the host.  I thought about including the node description in ibstat but that seems a bit short sighted.  I think the better solution would be to append the hca name (ie mlx4_X, qibX, etc) to the hostname for the Node Description.
> 
> Hacking the RHEL start up script is really easy to do this and results in nice names on the fabric which are easily resolved by the infiniband-diags, ibverbs, and perftest utilities.
> 
> bash-4.1# ibhosts
> Ca      : 0x0002c90300325280 ports 1 "ending mlx4_2"
> Ca      : 0x001175000079da38 ports 1 "happy qib0"
> Ca      : 0x0002c90300108f2e ports 1 "ending mlx4_1"
> Ca      : 0x001175000077d90e ports 2 "ending qib0"
> Ca      : 0x0002c903004bebda ports 2 "happy mlx4_0"
> bash-4.1# hostname
> happy
> bash-4.1# ibstat mlx4_0
> CA 'mlx4_0'
>         CA type: MT26428
>         Number of ports: 2
>         Firmware version: 2.8.600
> ...
> bash-4.1# ibv_rc_pingpong -d mlx4_0
>   local address:  LID 0x0008, QPN 0x16004a, PSN 0x2e8316 ...
> bash-4.1# rdma_bw -d mlx4_0
> 6089: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 | ...
> 
> I realize this is really a distro thing but it would be nice if we could agree to change the current "standard".
> 
> I can send a patch for RHEL and OFED (if I someone can point me to the openibd script or srpm).
> 
> Thoughts?
> Ira
> 
> 
> [*] Last I knew openibd does the same as RHEL's rdma start up script in this regard.
> 
> --
> Ira Weiny
> Member of Technical Staff
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at  http://vger.kernel.org/majordomo-info.html


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Bart Van Assche]

On 03/31/12 00:39, Ira Weiny wrote:

> First, a question: what package installs the openibd script in OFED?  For the
> life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]


That's easy to figure out:

$ rpm -qf /etc/init.d/openibd
kernel-ib-1.5.4-3.0.12+.x86_64

Bart.
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Ira Weiny]

On Thu, 05 Apr 2012 18:31:15 +0000
Bart Van Assche <bvanassche-HInyCGIudOg@public.gmane.org> wrote:

> On 03/31/12 00:39, Ira Weiny wrote:
> 
> > First, a question: what package installs the openibd script in OFED?  For the
> > life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]
> 
> 
> That's easy to figure out:
> 
> $ rpm -qf /etc/init.d/openibd
> kernel-ib-1.5.4-3.0.12+.x86_64

Sure if I had openibd installed.  ;-)

Ira

> 
> Bart.


-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Doug Ledford]

I'm fine with this change.  If people agree, I'll "make it so" in our products.

Sent from my ASUS Eee Pad

Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:

First, a question: what package installs the openibd script in OFED?  For the
life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]

Right now the "standard" for node description is, AFAIK, "<hostname> HCA-<num>",
where num is simply a counter for the HCA's as they are found in
/sys/class/infiniband.

The problem is resolving this "random" HCA number to an actual HCA on the
host.  I thought about including the node description in ibstat but that seems
a bit short sighted.  I think the better solution would be to append the hca
name (ie mlx4_X, qibX, etc) to the hostname for the Node Description.

Hacking the RHEL start up script is really easy to do this and results in nice
names on the fabric which are easily resolved by the infiniband-diags,
ibverbs, and perftest utilities.

bash-4.1# ibhosts
Ca      : 0x0002c90300325280 ports 1 "ending mlx4_2"
Ca      : 0x001175000079da38 ports 1 "happy qib0"
Ca      : 0x0002c90300108f2e ports 1 "ending mlx4_1"
Ca      : 0x001175000077d90e ports 2 "ending qib0"
Ca      : 0x0002c903004bebda ports 2 "happy mlx4_0"
bash-4.1# hostname
happy
bash-4.1# ibstat mlx4_0
CA 'mlx4_0'
        CA type: MT26428
        Number of ports: 2
        Firmware version: 2.8.600
...
bash-4.1# ibv_rc_pingpong -d mlx4_0
  local address:  LID 0x0008, QPN 0x16004a, PSN 0x2e8316
...
bash-4.1# rdma_bw -d mlx4_0
6089: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 |
...

I realize this is really a distro thing but it would be nice if we could agree
to change the current "standard".

I can send a patch for RHEL and OFED (if I someone can point me to the openibd
script or srpm).

Thoughts?
Ira


[*] Last I knew openibd does the same as RHEL's rdma start up script in this regard.

-- 
Ira Weiny
Member of Technical Staff
Lawrence Livermore National Lab
925-423-8008
weiny2-i2BcT+NCU+M@public.gmane.org
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] Proposal to change Node Description naming scheme for HCA's

[Joseph Glanville]

On 2 April 2012 06:49, Doug Ledford <dledford-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org> wrote:
> I'm fine with this change.  If people agree, I'll "make it so" in our products.
>
> Sent from my ASUS Eee Pad
>
> Ira Weiny <weiny2-i2BcT+NCU+M@public.gmane.org> wrote:
>
> First, a question: what package installs the openibd script in OFED?  For the
> life of me I can't find this script in 1.5.4.1 or 3.2 ...  :-/  [*]
>
> Right now the "standard" for node description is, AFAIK, "<hostname> HCA-<num>",
> where num is simply a counter for the HCA's as they are found in
> /sys/class/infiniband.
>
> The problem is resolving this "random" HCA number to an actual HCA on the
> host.  I thought about including the node description in ibstat but that seems
> a bit short sighted.  I think the better solution would be to append the hca
> name (ie mlx4_X, qibX, etc) to the hostname for the Node Description.
>
> Hacking the RHEL start up script is really easy to do this and results in nice
> names on the fabric which are easily resolved by the infiniband-diags,
> ibverbs, and perftest utilities.
>
> bash-4.1# ibhosts
> Ca      : 0x0002c90300325280 ports 1 "ending mlx4_2"
> Ca      : 0x001175000079da38 ports 1 "happy qib0"
> Ca      : 0x0002c90300108f2e ports 1 "ending mlx4_1"
> Ca      : 0x001175000077d90e ports 2 "ending qib0"
> Ca      : 0x0002c903004bebda ports 2 "happy mlx4_0"
> bash-4.1# hostname
> happy
> bash-4.1# ibstat mlx4_0
> CA 'mlx4_0'
>        CA type: MT26428
>        Number of ports: 2
>        Firmware version: 2.8.600
> ...
> bash-4.1# ibv_rc_pingpong -d mlx4_0
>  local address:  LID 0x0008, QPN 0x16004a, PSN 0x2e8316
> ...
> bash-4.1# rdma_bw -d mlx4_0
> 6089: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | sl=0 | iters=1000 | duplex=0 | cma=0 |
> ...
>
> I realize this is really a distro thing but it would be nice if we could agree
> to change the current "standard".
>
> I can send a patch for RHEL and OFED (if I someone can point me to the openibd
> script or srpm).
>
> Thoughts?
> Ira
>
>
> [*] Last I knew openibd does the same as RHEL's rdma start up script in this regard.
>
> --
> Ira Weiny
> Member of Technical Staff
> Lawrence Livermore National Lab
> 925-423-8008
> weiny2-i2BcT+NCU+M@public.gmane.org
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

I have been doing something similar for a while.

Joseph.

-- 
Founder | Director | VP Research
Orion Virtualisation Solutions | www.orionvm.com.au | Phone: 1300 56
99 52 | Mobile: 0428 754 846
--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html