RFC: InfiniBand hotplug agent

RFC: InfiniBand hotplug agent

[Roland Dreier]

I merged new feature for the InfiniBand stack in kernel 2.6.17.  There
will be /sys/class/infiniband/<dev>/node_desc files.  This can be used
by userspace to change the IB node description, which is retrieved by
various network management stuff.  Putting the hostname in the node
description makes things a lot nicer, since it provides a link from
the IB network view back to the real world.

Right now on my systems, I'm using the script below, which takes the
output of hostname and adds it to the node description.  I put this in
/lib/udev/infiniband.agent along with a udev rule like

    SUBSYSTEM="infiniband", RUN+="infiniband.agent"

to run it.

So I have two questions:

 - Is this script reasonable?  I'm by no means a hotplug expert, and
   I'd like to get critiques and suggestions for how to write a proper
   agent.

 - What's the right way to get this into distributions?  File wishlist
   bugs against udev with the rule and the script included?

Thanks,
  Roland

Here's the script:

#!/bin/sh -e
#
# infiniband agent
#

cd /lib/udev/
. ./hotplug.functions

if [ "$ACTION" = "add" ]; then
    DESC_PATH="/sys/$DEVPATH/node_desc"
    if [ -e "$DESC_PATH" ]; then
        DESC="`hostname` (`cat $DESC_PATH`)"
        echo -n $DESC > $DESC_PATH
    fi
fi

exit 0


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: RFC: InfiniBand hotplug agent

[Marco d'Itri]

On Mar 22, Roland Dreier <rdreier@cisco.com> wrote:

>  - What's the right way to get this into distributions?  File wishlist
>    bugs against udev with the rule and the script included?
Yes.

> cd /lib/udev/
> . ./hotplug.functions
I think that this is debian-specific, and you are not using it anyway.

> if [ "$ACTION" = "add" ]; then
Remove this if statement and use such a rule instead:

ACTION="add", SUBSYSTEM="infiniband", RUN+="infiniband.agent"

#!/bin/sh -e

DESC_PATH="/sys/$DEVPATH/node_desc"

# is this actually needed? can node_desc really not exist?
[ -e "$DESC_PATH" ] || exit 0

# by using read instead of cat you save a fork+exec
read OLD_DESC < $DESC_PATH
DESC="`hostname` ($OLD_DESC)"
echo -n $DESC > $DESC_PATH


But probably using hostname(1) in a RUN rule is not such a great idea,
because at boot time the script will be run long before the system
hostname will have been set by an init script.

-- 
ciao,
Marco


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: RFC: InfiniBand hotplug agent

[Roland Dreier]

Thanks for the comments.

    Marco> Remove this if statement and use such a rule instead:

    Marco> ACTION="add", SUBSYSTEM="infiniband", RUN+="infiniband.agent"

Good point.

    Marco> # is this actually needed? can node_desc really not exist?
    Marco> [ -e "$DESC_PATH" ] || exit 0

Certainly on kernels prior to 2.6.17 it won't be there...

    Marco> # by using read instead of cat you save a fork+exec read

Another good point.

    Marco> But probably using hostname(1) in a RUN rule is not such a
    Marco> great idea, because at boot time the script will be run
    Marco> long before the system hostname will have been set by an
    Marco> init script.

Hmm, that's a problem, and indeed if I let the driver be loaded on
boot, I end up with "(none)" as the hostname (I usually blacklist my
IB drivers, because I do a lot of driver development and it's a pain
when the system crashes on boot because of bugs I wrote).

Do you think using /etc/hostname instead (on Debian at least) is a
palatable solution?  It seems this will end up being distro specific
unfortunately...  Here's what I just tested (along with your
ACTION="add" suggestion), and it worked well even when the driver is
loaded at boot time:

#!/bin/sh -e

DESC_PATH="/sys/$DEVPATH/node_desc"

[ -e "$DESC_PATH" ] || exit 0

read OLD_DESC < $DESC_PATH
read HOSTNAME < /etc/hostname
echo -n "$HOSTNAME ($OLD_DESC)" > $DESC_PATH

exit 0

Do you think this is OK for Debian?  Obviously I'll have to come up
with something else for Gentoo, Fedora, etc.

Thanks,
  Roland


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: RFC: InfiniBand hotplug agent

[Greg KH]

On Wed, Mar 22, 2006 at 12:36:44PM -0800, Roland Dreier wrote:
> Hmm, that's a problem, and indeed if I let the driver be loaded on
> boot, I end up with "(none)" as the hostname (I usually blacklist my
> IB drivers, because I do a lot of driver development and it's a pain
> when the system crashes on boot because of bugs I wrote).
> 
> Do you think using /etc/hostname instead (on Debian at least) is a
> palatable solution?  It seems this will end up being distro specific
> unfortunately...  Here's what I just tested (along with your
> ACTION="add" suggestion), and it worked well even when the driver is
> loaded at boot time:

/etc is not even around at early boot time (think module loading from
initramfs/initrd), which is what the "enterprise" distros do.

> #!/bin/sh -e
> 
> DESC_PATH="/sys/$DEVPATH/node_desc"
> 
> [ -e "$DESC_PATH" ] || exit 0
> 
> read OLD_DESC < $DESC_PATH
> read HOSTNAME < /etc/hostname
> echo -n "$HOSTNAME ($OLD_DESC)" > $DESC_PATH

Why does your drivers care about the hostname?  The kernel already knows
this information anyway...

thanks,

greg k-h


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: RFC: InfiniBand hotplug agent

[Scott James Remnant]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

On Wed, 2006-03-22 at 20:47 +0100, Marco d'Itri wrote:

> But probably using hostname(1) in a RUN rule is not such a great idea,
> because at boot time the script will be run long before the system
> hostname will have been set by an init script.
> 
I got sufficiently amused by the three or four things that failed
because of this that I moved the hostname setting to almost the top of
the boot sequence in Ubuntu, with no ill effects.

It's a tiny operation, after all.

Scott
-- 
Scott James Remnant
scott@ubuntu.com

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 191 bytes --]

Re: RFC: InfiniBand hotplug agent

[Roland Dreier]

    Greg> Why does your drivers care about the hostname?  The kernel
    Greg> already knows this information anyway...

As I explained before, this script puts the hostname in the InfiniBand
"node description," which is retrieved by various IB network
management tools.  It makes the output of these tools much easier to
read, since you can see things like "node1 (CA)" and "node2 (CA)" in
the network management tools instead of just two copies of "CA".

Yes, I could modify the driver to look at system_utsname and pick a
way to format that.  But that suffers from the problem of putting
policy in the kernel (who picks the format for node descriptions?),
although that's minor.  And we still have the problem of what to do
before the hostname is set -- put "(none)" in our node description?

Anyway, do you think just having this node description setting in the
kernel is the best way to go?

Thanks,
  Roland


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel

Re: RFC: InfiniBand hotplug agent

[Greg KH]

On Thu, Mar 23, 2006 at 09:11:02AM -0800, Roland Dreier wrote:
>     Greg> Why does your drivers care about the hostname?  The kernel
>     Greg> already knows this information anyway...
> 
> As I explained before, this script puts the hostname in the InfiniBand
> "node description," which is retrieved by various IB network
> management tools.  It makes the output of these tools much easier to
> read, since you can see things like "node1 (CA)" and "node2 (CA)" in
> the network management tools instead of just two copies of "CA".

Ick, the kernel is then printing that data back out somewhere else?  I
really don't want to know... :)

> Yes, I could modify the driver to look at system_utsname and pick a
> way to format that.  But that suffers from the problem of putting
> policy in the kernel (who picks the format for node descriptions?),
> although that's minor.  And we still have the problem of what to do
> before the hostname is set -- put "(none)" in our node description?
> 
> Anyway, do you think just having this node description setting in the
> kernel is the best way to go?

I think that your proposed solution will not work on the current round
of "enterprise" distros, so I don't think it is a real "solution".

As for a proposed one, I have no idea, sorry.

greg k-h


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid\x110944&bid$1720&dat\x121642
_______________________________________________
Linux-hotplug-devel mailing list  http://linux-hotplug.sourceforge.net
Linux-hotplug-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/linux-hotplug-devel