configure MD3000 episode 2

configure MD3000 episode 2

[Thomas Witzel]

I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
the help of this list I managed to configure the RAID etc. However
there is one remaining problem. udev doesn't seem to recognize this
setup. After every reboot I have to manually create the /dev/dm-*
devices with mknod and run kpartx to make the partitions show up. Can
someone please hint to me where I get the appropriate udev scripts ?
Ubuntu does have dmapper and multipath related scripts in the udev
directory, but they don't seem to be working for me.

Thank you so much,
Thomas Witzel

Re: configure MD3000 episode 2

[Konrad Rzeszutek]

On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
> the help of this list I managed to configure the RAID etc. However
> there is one remaining problem. udev doesn't seem to recognize this
> setup. After every reboot I have to manually create the /dev/dm-*
> devices with mknod and run kpartx to make the partitions show up. Can
> someone please hint to me where I get the appropriate udev scripts ?

You can drop this in your udev/rules.d directory:

> cat /etc/udev/rules.d/95-mpdc.rules 
RUN+="socket:/org/kernel/dm/multipath_event"


Obviously you need to have your multipathd to start _before_ your
iSCSI daemon.

> Ubuntu does have dmapper and multipath related scripts in the udev
> directory, but they don't seem to be working for me.
> 
> Thank you so much,
> Thomas Witzel
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel

Re: configure MD3000 episode 2

[Thomas Witzel]

Thanks, I added that. But it didn't change anything. I'm not sure what
you want me to do with iscsi. I don't think I have an iscsi device. I
have heard about the MD3000i before, but thats not what I have. I
think I have a MD3000 (without i) connected via Symbios Logic SAS1068
PCI-X Fusion-MPT SAS  controller.

Thomas

On Mon, Mar 23, 2009 at 10:31 AM, Konrad Rzeszutek
<konrad@virtualiron.com> wrote:
> On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
>> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
>> the help of this list I managed to configure the RAID etc. However
>> there is one remaining problem. udev doesn't seem to recognize this
>> setup. After every reboot I have to manually create the /dev/dm-*
>> devices with mknod and run kpartx to make the partitions show up. Can
>> someone please hint to me where I get the appropriate udev scripts ?
>
> You can drop this in your udev/rules.d directory:
>
>> cat /etc/udev/rules.d/95-mpdc.rules
> RUN+="socket:/org/kernel/dm/multipath_event"
>
>
> Obviously you need to have your multipathd to start _before_ your
> iSCSI daemon.
>
>> Ubuntu does have dmapper and multipath related scripts in the udev
>> directory, but they don't seem to be working for me.
>>
>> Thank you so much,
>> Thomas Witzel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

multipathd dead

[Charlie Sharkey]

 

I have a question about multipathd. It starts up ok,
but later it's status says it is dead. It does show
an error (see below), but would this be enough to kill it ?

Also, is there any other log files besides the messages and
the boot.log files it logs to ?

Thank you,
 charlie

boot.msg file
<6>device-mapper: multipath: version 1.0.4 loaded
<6>device-mapper: multipath round-robin: version 1.0.0 loaded
<3>device-mapper: table: 253:9: multipath: error getting device
<4>device-mapper: ioctl: error adding target to table

Re: configure MD3000 episode 2

[Thomas Witzel]

I'm now looking at the udev monitor and also set the loglevel to debug
and I can see that udev is processing the multipath events, hence
creating the /dev/mapper entries (but not the partition entries). But
there is never anything called that would create the /dev/dm-* devices
and the 95-kpartx rule is also never called it seems.
I have now rigged it to run a script on boot that calls mkdnod and
kpartx, but of course I'd still like udev support. I have not been
able to get any answer from ubuntu guys, so is it a fact that
multipath is basically not supported under ubuntu ? What would be the
next best distribution thats free of annual license fees ?

Thank you very much again,
Thomas


On Mon, Mar 23, 2009 at 11:44 AM, Thomas Witzel <witzel.thomas@gmail.com> wrote:
> Thanks, I added that. But it didn't change anything. I'm not sure what
> you want me to do with iscsi. I don't think I have an iscsi device. I
> have heard about the MD3000i before, but thats not what I have. I
> think I have a MD3000 (without i) connected via Symbios Logic SAS1068
> PCI-X Fusion-MPT SAS  controller.
>
> Thomas
>
> On Mon, Mar 23, 2009 at 10:31 AM, Konrad Rzeszutek
> <konrad@virtualiron.com> wrote:
>> On Fri, Mar 20, 2009 at 08:54:27PM -0400, Thomas Witzel wrote:
>>> I have a Dell NX1950 with a MD3000 running ubuntu server 8.04 . With
>>> the help of this list I managed to configure the RAID etc. However
>>> there is one remaining problem. udev doesn't seem to recognize this
>>> setup. After every reboot I have to manually create the /dev/dm-*
>>> devices with mknod and run kpartx to make the partitions show up. Can
>>> someone please hint to me where I get the appropriate udev scripts ?
>>
>> You can drop this in your udev/rules.d directory:
>>
>>> cat /etc/udev/rules.d/95-mpdc.rules
>> RUN+="socket:/org/kernel/dm/multipath_event"
>>
>>
>> Obviously you need to have your multipathd to start _before_ your
>> iSCSI daemon.
>>
>>> Ubuntu does have dmapper and multipath related scripts in the udev
>>> directory, but they don't seem to be working for me.
>>>
>>> Thank you so much,
>>> Thomas Witzel
>>>
>>> --
>>> dm-devel mailing list
>>> dm-devel@redhat.com
>>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>> --
>> dm-devel mailing list
>> dm-devel@redhat.com
>> https://www.redhat.com/mailman/listinfo/dm-devel
>>
>

Re: configure MD3000 episode 2

[Konrad Rzeszutek]

[-- Attachment #1: Type: text/plain, Size: 4101 bytes --]

On Mon, Mar 23, 2009 at 05:05:37PM -0400, Thomas Witzel wrote:
> I'm now looking at the udev monitor and also set the loglevel to debug
> and I can see that udev is processing the multipath events, hence
> creating the /dev/mapper entries (but not the partition entries). But

The mechanism that does this is a bit more complex. Let me explain
to you and hopefully that will help you out.

When an HBA inits, it ends up calling a bunch of internal sd.c routines
which end up creating uevents (and of course setup the SysFS structures).
There are usually four sets of them:

1) device_add creates this:

[add@/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[ACTION=add]
[DEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[SUBSYSTEM=scsi]
[SEQNUM=8969]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

2) sd_probe creates this guy:

[add@/class/scsi_disk/9:0:0:15]
[ACTION=add]
[DEVPATH=/class/scsi_disk/9:0:0:15]
[SUBSYSTEM=scsi_disk]
[SEQNUM=9088]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

3) add_disk creates this one:

add@/block/sdo]
[ACTION=add]
[DEVPATH=/block/sdo]
[SUBSYSTEM=block]
[SEQNUM=9332]
[MINOR=224]
[MAJOR=8]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]

4) and sg_add this one:

[add@/class/scsi_generic/sg15]
[ACTION=add]
[DEVPATH=/class/scsi_generic/sg15]
[SUBSYSTEM=scsi_generic]
[SEQNUM=9110]
[PHYSDEVPATH=/devices/pci0000:00/0000:00:07.0/0000:04:00.3/0000:0a:01.0/0000:0b:07.0/host9/rport-9:0-17/target9:0:0/9:0:0:15]
[PHYSDEVBUS=scsi]
[PHYSDEVDRIVER=sd]
[MAJOR=21]
[MINOR=15]

the kernel injects them in a netlink (NETLINK_KOBJECT_UEVENT) - I've attached a small C code
that grabs them. At that point the block disk is now usuable and userspace code can
access it by the major:minor numbers.

udev listens to this netlink and based on the udev rules execute them. But there are rules
in there that will short-circuit these uevents. If you see 'RUN=" that means it is done and won't
pass it on to the next one. If you see 'RUN+=' that means process it and pass it on to next rules
that might match the criteria.  Udev is the one that actually creates the /dev/sdo.

There might a udev rule in there that eats up the uevent before it is passed to the multipathd
netlink socket (/org/kernel/dm/multipath_event). You can make it less of chance if you
change the name of the rule to a low number, like 01-mpdc.rules.

Multipathd acts on the #3 uevent - it interrogates the block disk and then calls the
device mapper ioctl (/dev/mapper/control). The device mapper (kernel piece) sends its own
uevent, which looks as so:

[add@/block/dm-19]
[ACTION=add]
[DEVPATH=/block/dm-19]
[SUBSYSTEM=block]
[SEQNUM=9412]
[MINOR=19]
[MAJOR=253]

And udev then creates /dev/dm-19. It then would pass this on to multipathd socket. And
multipathd then does it stuff (which is to check the paths, make sure everthing
is right and kick of a checker path if it hasn't already been done).

> there is never anything called that would create the /dev/dm-* devices
> and the 95-kpartx rule is also never called it seems.

> I have now rigged it to run a script on boot that calls mkdnod and
> kpartx, but of course I'd still like udev support. I have not been

During bootup the udev isn't run. I think what gets called is the
HOTPLUG program, which is defined by default is "/sbin/hotplug".
Look in that program on the initrd image. It might not do any udev stuff at all
and just simple block disk creations.

> able to get any answer from ubuntu guys, so is it a fact that
> multipath is basically not supported under ubuntu ? What would be the
> next best distribution thats free of annual license fees ?

Huh? Ubuntu charges license fees on GPL code? I think you are confusing
what you are paying - it isn't license fee but support fee.

[-- Attachment #2: listen.c --]
[-- Type: text/plain, Size: 1061 bytes --]

#include <sys/socket.h>
#include <linux/netlink.h>
#include <stdio.h>
#include <string.h>
#define MAX_PAYLOAD 1024        /* maximum payload size */
struct sockaddr_nl src_addr;
int sock_fd;
static char buff[MAX_PAYLOAD];
ssize_t buflen;

int
main ()
{
  sock_fd = socket (PF_NETLINK, SOCK_DGRAM, NETLINK_KOBJECT_UEVENT);

  memset (&src_addr, 0, sizeof (src_addr));
  src_addr.nl_family = AF_NETLINK;
  src_addr.nl_pid = getpid ();  /* self pid */
  src_addr.nl_groups = 0xffffffff;

  printf ("Listen..\n");
  bind (sock_fd, (struct sockaddr *) &src_addr, sizeof (src_addr));
  printf ("Receiving..\n");
  while (1)
    {
      buflen = recv (sock_fd, &buff, sizeof (buff), 0);
      printf ("Got data: %d\n", buflen);
      int i, bufpos;
      char *key;
      for (i = 0, bufpos = 0; (bufpos < buflen) && i < MAX_PAYLOAD; i++)
        {
          key = &buff[bufpos];
          printf ("[%s]\n", key);
          bufpos += strlen (key) + 1;
        }
      memset (&buff, 0, MAX_PAYLOAD);
    }
  /* Close Netlink Socket */
  close (sock_fd);
  return 0;
}

[-- Attachment #3: Type: text/plain, Size: 0 bytes --]

Re: configure MD3000 episode 2

[Thomas Witzel]

> Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> what you are paying - it isn't license fee but support fee.

Thats not what I meant. I know that my hardware config does work out
of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
I have not been able to get any of this to work for 3 weeks now and I
have not been able to get help/support on the issue, so I was asking
whether there is a different free distribution that won't put me
through these time consuming google/mailing list/debug orgies to get
some storage mounted.

Thomas

Re: configure MD3000 episode 2

[malahal]

Thomas Witzel [witzel.thomas@gmail.com] wrote:
> > Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> > what you are paying - it isn't license fee but support fee.
> 
> Thats not what I meant. I know that my hardware config does work out
> of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
> I have not been able to get any of this to work for 3 weeks now and I
> have not been able to get help/support on the issue, so I was asking
> whether there is a different free distribution that won't put me
> through these time consuming google/mailing list/debug orgies to get
> some storage mounted.
> 
> Thomas

Maybe CentOS ?

Re: configure MD3000 episode 2

[Thomas Witzel]

I heard people having issues with centos/multipath as well, not clear
whether it applies to me. I just had a mini-breakthrough. Since the
udev documentation says that the default rule should create /dev/dm-*
when the kernel name comes up, unless specified *OTHERWISE* I removed
all those rules that were named dmsetup,dmadm and lvm2. Low and behold
now the devices are created. Now I only have to find
out why the kpartx rule is not called.

Thomas


On Tue, Mar 24, 2009 at 10:48 AM, malahal <malahal@us.ibm.com> wrote:
> Thomas Witzel [witzel.thomas@gmail.com] wrote:
>> > Huh? Ubuntu charges license fees on GPL code? I think you are confusing
>> > what you are paying - it isn't license fee but support fee.
>>
>> Thats not what I meant. I know that my hardware config does work out
>> of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
>> I have not been able to get any of this to work for 3 weeks now and I
>> have not been able to get help/support on the issue, so I was asking
>> whether there is a different free distribution that won't put me
>> through these time consuming google/mailing list/debug orgies to get
>> some storage mounted.
>>
>> Thomas
>
> Maybe CentOS ?
>
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel
>

Re: configure MD3000 episode 2

[Konrad Rzeszutek]

On Tue, Mar 24, 2009 at 10:24:34AM -0400, Thomas Witzel wrote:
> > Huh? Ubuntu charges license fees on GPL code? I think you are confusing
> > what you are paying - it isn't license fee but support fee.
> 
> Thats not what I meant. I know that my hardware config does work out
> of the box with RHEL, but we do not want to pay RHEL, and with ubuntu
> I have not been able to get any of this to work for 3 weeks now and I
> have not been able to get help/support on the issue, so I was asking
> whether there is a different free distribution that won't put me
> through these time consuming google/mailing list/debug orgies to get
> some storage mounted.

Since you say that RHEL works, you could use the clone of RHEL - CentOS.

Re: configure MD3000 episode 2

[Thomas Witzel]

Okay, I'm looking for a man page:
man -k hotplug finds nothing

I'm looking for the file you mentioned:
ls /sbin/hotplug
ls: cannot access /sbin/hotplug: No such file or directory

I'm looking for a package to install related to hotplug:

apt-cache search hotplug
libxi-dev - X11 Input extension library (development headers)
libxi6 - X11 Input extension library
libxi6-dbg - X11 Input extension library (debug package)
fxload - Firmware download to EZ-USB devices
libaws-dev - Ada Web Server development files
atmel-firmware - Firmware for Atmel at76c50x wireless networking chips.

Seems like there is NOTHING on ubuntu even remotely called hotplug,
except /proc/sys/kernel/hotplug
which is not a configuration file.

What distribution are you using ?

Thomas

Re: configure MD3000 episode 2

[Konrad Rzeszutek]

On Tue, Mar 24, 2009 at 10:29:13AM -0400, Thomas Witzel wrote:
> Okay, I'm looking for a man page:
> man -k hotplug finds nothing
> 
> I'm looking for the file you mentioned:
> ls /sbin/hotplug
> ls: cannot access /sbin/hotplug: No such file or directory
> 
> I'm looking for a package to install related to hotplug:
> 
> apt-cache search hotplug
> libxi-dev - X11 Input extension library (development headers)
> libxi6 - X11 Input extension library
> libxi6-dbg - X11 Input extension library (debug package)
> fxload - Firmware download to EZ-USB devices
> libaws-dev - Ada Web Server development files
> atmel-firmware - Firmware for Atmel at76c50x wireless networking chips.
> 
> Seems like there is NOTHING on ubuntu even remotely called hotplug,
> except /proc/sys/kernel/hotplug

I am looking at my Debian box, and it looks as if the initrd
/scripts/init-premount/udev does start the udev daemon. And the
/etc/udev/rules.d has many rules.

Sorry for leading you on this chase.

> which is not a configuration file.

When you create the initrd, does the initrd have your multipathd and
its rules?


> 
> What distribution are you using ?

RHEL, FC, Ubuntu, Debian and SLES. But I haven't dug in the initrd of Debian/Ubuntu
so I can't much help there :-(

> 
> Thomas
> 
> --
> dm-devel mailing list
> dm-devel@redhat.com
> https://www.redhat.com/mailman/listinfo/dm-devel