multipath device issue

multipath device issue

[Dave Young]

[-- Attachment #1: Type: text/plain, Size: 247 bytes --]

Hi,

The multipath device node was not created during testing kdump on
multipath target. dracut drop into emergency shell,  I checked that
'multipathd -B' was running in background. Any idea?

Attached the full log of debug info.

-- 
Thanks
Dave

[-- Attachment #2: multipath.log.gz --]
[-- Type: application/x-gzip, Size: 19704 bytes --]

Re: multipath device issue

[Harald Hoyer]

Am 02.08.2012 07:09, schrieb Dave Young:
> Hi,
> 
> The multipath device node was not created during testing kdump on
> multipath target. dracut drop into emergency shell,  I checked that
> 'multipathd -B' was running in background. Any idea?
> 
> Attached the full log of debug info.
> 

hmm.. I hope, this did not introduce a regression:
http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e

Re: multipath device issue

[Dave Young]

On 08/02/2012 04:07 PM, Harald Hoyer wrote:

> Am 02.08.2012 07:09, schrieb Dave Young:
>> Hi,
>>
>> The multipath device node was not created during testing kdump on
>> multipath target. dracut drop into emergency shell,  I checked that
>> 'multipathd -B' was running in background. Any idea?
>>
>> Attached the full log of debug info.
>>
> 
> hmm.. I hope, this did not introduce a regression:
> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e


Revert this commit does not help in this case.

I tried manually run `multipath` in the emergency shell the
/dev/mapper/mpathb will be created..

-- 
Thanks
Dave

Re: multipath device issue

[Harald Hoyer]

Am 02.08.2012 14:41, schrieb Dave Young:
> On 08/02/2012 04:07 PM, Harald Hoyer wrote:
> 
>> Am 02.08.2012 07:09, schrieb Dave Young:
>>> Hi,
>>>
>>> The multipath device node was not created during testing kdump on
>>> multipath target. dracut drop into emergency shell,  I checked that
>>> 'multipathd -B' was running in background. Any idea?
>>>
>>> Attached the full log of debug info.
>>>
>>
>> hmm.. I hope, this did not introduce a regression:
>> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e
> 
> 
> Revert this commit does not help in this case.
> 
> I tried manually run `multipath` in the emergency shell the
> /dev/mapper/mpathb will be created..
> 

oh.. can you retry with dracut-023? I think your version is missing this patch:

http://git.kernel.org/?p=boot/dracut/dracut.git;a=blobdiff;f=modules.d/90multipath/module-setup.sh;h=31d11088bd65932460f036cabbd7a33cab5c5ec7;hp=f9c1f81354f8932a28561822452107da3aa6c349;hb=24a38bc1cbe461c344ce61c7e10b6d51d5689c93;hpb=82b1739dcee8853c07675d3e79b2f4623c60a364

from

http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=24a38bc1cbe461c344ce61c7e10b6d51d5689c93

Re: multipath device issue

[Dave Young]

On 08/02/2012 09:18 PM, Harald Hoyer wrote:

> Am 02.08.2012 14:41, schrieb Dave Young:
>> On 08/02/2012 04:07 PM, Harald Hoyer wrote:
>>
>>> Am 02.08.2012 07:09, schrieb Dave Young:
>>>> Hi,
>>>>
>>>> The multipath device node was not created during testing kdump on
>>>> multipath target. dracut drop into emergency shell,  I checked that
>>>> 'multipathd -B' was running in background. Any idea?
>>>>
>>>> Attached the full log of debug info.
>>>>
>>>
>>> hmm.. I hope, this did not introduce a regression:
>>> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e
>>
>>
>> Revert this commit does not help in this case.
>>
>> I tried manually run `multipath` in the emergency shell the
>> /dev/mapper/mpathb will be created..
>>
> 
> oh.. can you retry with dracut-023? I think your version is missing this patch:
> 
> http://git.kernel.org/?p=boot/dracut/dracut.git;a=blobdiff;f=modules.d/90multipath/module-setup.sh;h=31d11088bd65932460f036cabbd7a33cab5c5ec7;hp=f9c1f81354f8932a28561822452107da3aa6c349;hb=24a38bc1cbe461c344ce61c7e10b6d51d5689c93;hpb=82b1739dcee8853c07675d3e79b2f4623c60a364
> 
> from
> 
> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=24a38bc1cbe461c344ce61c7e10b6d51d5689c93


I in fact use the dracut-023, but I found I have no 66-kpartx.rules,
maybe I should upgrade udev as well.

Will have a test tomorrow.

-- 
Thanks
Dave

Re: multipath device issue

[Dave Young]

On 08/02/2012 09:28 PM, Dave Young wrote:

> On 08/02/2012 09:18 PM, Harald Hoyer wrote:
> 
>> Am 02.08.2012 14:41, schrieb Dave Young:
>>> On 08/02/2012 04:07 PM, Harald Hoyer wrote:
>>>
>>>> Am 02.08.2012 07:09, schrieb Dave Young:
>>>>> Hi,
>>>>>
>>>>> The multipath device node was not created during testing kdump on
>>>>> multipath target. dracut drop into emergency shell,  I checked that
>>>>> 'multipathd -B' was running in background. Any idea?
>>>>>
>>>>> Attached the full log of debug info.
>>>>>
>>>>
>>>> hmm.. I hope, this did not introduce a regression:
>>>> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=c0cae7f702b5ccf27a80913642aec8bf15fe710e
>>>
>>>
>>> Revert this commit does not help in this case.
>>>
>>> I tried manually run `multipath` in the emergency shell the
>>> /dev/mapper/mpathb will be created..
>>>
>>
>> oh.. can you retry with dracut-023? I think your version is missing this patch:
>>
>> http://git.kernel.org/?p=boot/dracut/dracut.git;a=blobdiff;f=modules.d/90multipath/module-setup.sh;h=31d11088bd65932460f036cabbd7a33cab5c5ec7;hp=f9c1f81354f8932a28561822452107da3aa6c349;hb=24a38bc1cbe461c344ce61c7e10b6d51d5689c93;hpb=82b1739dcee8853c07675d3e79b2f4623c60a364
>>
>> from
>>
>> http://git.kernel.org/?p=boot/dracut/dracut.git;a=commitdiff;h=24a38bc1cbe461c344ce61c7e10b6d51d5689c93
> 
> 
> I in fact use the dracut-023, but I found I have no 66-kpartx.rules,
> maybe I should upgrade udev as well.
> 
> Will have a test tomorrow.


Hi, harald

It has nothing to do with the udev rules.

There's two issues:
1) 90multipath install multipath.sh to pre-trigger hook, but seems
/dev/mapper/mpathX depends on /dev/dm-* population which will be created
after udev trigger.

2) fstab-sys call wait_for_dev /dev/mapper/mpathb in initqueue,
/dev/mapper/mpathb does not geting change to create, so dracut drop to
emergency shell.

I'm not sure what's the right way to resolve this, move multipath.sh to
pre-mount hook looks resolve the issue 1), how to resolve the issue 2)?
comment out it works, but it looks not a right fix.

I'm not familiar with multipath, please correct me for anything wrong.

-- 
Thanks
Dave

Re: multipath device issue

[Harald Hoyer]

Am 03.08.2012 07:36, schrieb Dave Young:
> Hi, harald
> 
> It has nothing to do with the udev rules.
> 
> There's two issues:
> 1) 90multipath install multipath.sh to pre-trigger hook, but seems
> /dev/mapper/mpathX depends on /dev/dm-* population which will be created
> after udev trigger.
> 
> 2) fstab-sys call wait_for_dev /dev/mapper/mpathb in initqueue,
> /dev/mapper/mpathb does not geting change to create, so dracut drop to
> emergency shell.
> 
> I'm not sure what's the right way to resolve this, move multipath.sh to
> pre-mount hook looks resolve the issue 1), how to resolve the issue 2)?
> comment out it works, but it looks not a right fix.
> 
> I'm not familiar with multipath, please correct me for anything wrong.
> 

I think it _has_ to do with the udev rules.

1. the multipathd daemon has to be started before udevd is started.
2. when udev is triggered, 62-multipath.rules should kick in and call
"multipath" for every disk, which adds disks to the daemon, if they fit.
3. of course /etc/multipath.conf has to be configured correctly.

So, if you see /dev/mapper/mpathb be correctly created, when you start multpathd
after all devices exist in /dev, then the 62-multipath.rules
mechanism does not seem to work,

With the next systemd release "rd.udev.log-priority=debug" does work again and
we might be able to debug this issue.

Re: multipath device issue

[Dave Young]

On 08/03/2012 03:37 PM, Harald Hoyer wrote:

> Am 03.08.2012 07:36, schrieb Dave Young:
>> Hi, harald
>>
>> It has nothing to do with the udev rules.
>>
>> There's two issues:
>> 1) 90multipath install multipath.sh to pre-trigger hook, but seems
>> /dev/mapper/mpathX depends on /dev/dm-* population which will be created
>> after udev trigger.
>>
>> 2) fstab-sys call wait_for_dev /dev/mapper/mpathb in initqueue,
>> /dev/mapper/mpathb does not geting change to create, so dracut drop to
>> emergency shell.
>>
>> I'm not sure what's the right way to resolve this, move multipath.sh to
>> pre-mount hook looks resolve the issue 1), how to resolve the issue 2)?
>> comment out it works, but it looks not a right fix.
>>
>> I'm not familiar with multipath, please correct me for anything wrong.
>>

> 
> I think it _has_ to do with the udev rules.


What I should say is "It has nothing to do with the udev rules patches
you mentioned previously" ;)

> 
> 1. the multipathd daemon has to be started before udevd is started.
> 2. when udev is triggered, 62-multipath.rules should kick in and call
> "multipath" for every disk, which adds disks to the daemon, if they fit.
> 3. of course /etc/multipath.conf has to be configured correctly.
> 
> So, if you see /dev/mapper/mpathb be correctly created, when you start multpathd
> after all devices exist in /dev, then the 62-multipath.rules
> mechanism does not seem to work,

>

> With the next systemd release "rd.udev.log-priority=debug" does work again and
> we might be able to debug this issue.


Ok, will see if I can get something from the 62-multipath.rules. Thanks
for the suggestion.


-- 
Thanks
Dave

Re: multipath device issue

[Dave Young]

>> So, if you see /dev/mapper/mpathb be correctly created, when you start multpathd
>> after all devices exist in /dev, then the 62-multipath.rules
>> mechanism does not seem to work,
> 
>>
> 
>> With the next systemd release "rd.udev.log-priority=debug" does work again and
>> we might be able to debug this issue.
> 
> 
> Ok, will see if I can get something from the 62-multipath.rules. Thanks
> for the suggestion.
> 
> 


Hi, here is the debug messages with latest systemd, 62-multipath fails
at run 'socket:/org/kernel/dm/multipath_event':

---
Welcome to Fedora 17 (Beefy Miracle) dracut-023-2.fc18 (Initramfs)!

[    0.799958] systemd[1]: No hostname configured.
[    0.801608] systemd[1]: Set hostname to <localhost>.
[    0.807834] systemd[1]: Cannot add dependency job for unit
systemd-journal-flush.service, ignoring: Unit
systemd-journal-flush.service failed to load: No such file or directory.
See system logs and 'systemctl status systemd-journal-flush.service' for
details.
[  OK  ] Listening on udev Kernel Socket.
[  OK  ] Listening on udev Control Socket.
[  OK  ] Listening on Journal Socket.
         Starting Journal Service...
[    0.825703] systemd-journald[44]: Fixed max_use=11.8M max_size=1.4M
min_size=64.0K keep_free=5.9M
[    0.828702] systemd-journald[44]: Reserving 2695 entries in hash table.
[    0.831147] systemd-journald[44]: Vacuuming...
[  OK  ] Started Journal Service.
[  OK  ] Reached target Sockets.
         Starting Dracut cmdline hook...
         Starting Setup Virtual Console...
[  OK  ] Reached target Swap.
[  OK  ] [    0.850156] usb 1-1: new full-speed USB device number 2
using uhci_hcd
Reached target Local File Systems.
[  OK  ] Started Setup Virtual Console.
[  OK  ] Reached target System Initialization.
dracut-cmdline[46]: Warning: Option 'KEYTABLE' is deprecated, use
'KEYMAP' instead.
dracut-cmdline[46]: Warning: Option 'SYSFONT' is deprecated, use 'FONT'
instead.
[  OK  ] Started Dracut cmdline hook.
         Starting Dracut pre-udev hook...
[  OK  ] Started Dracut pre-udev hook.
         Starting udev Kernel Device Manager...
[  OK  ] Started udev Kernel Device Manager.
         Starting Dracut pre-trigger hook...
[  OK  ] Reached target Basic System.
[    0.950521] systemd-udevd[102]: starting version 188
[    0.970090] device-mapper: multipath: version 1.3.0 loaded
[    1.003546] input: ImExPS/2 Generic Explorer Mouse as
/devices/platform/i8042/serio1/input/input2
[    1.266732] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
[    1.269940] usb 1-1: New USB device strings: Mfr=1, Product=3,
SerialNumber=5
[    1.272323] usb 1-1: Product: QEMU USB Tablet
[    1.273889] usb 1-1: Manufacturer: QEMU 1.0
[    1.275491] usb 1-1: SerialNumber: 42
[    1.311638] input: QEMU 1.0 QEMU USB Tablet as
/devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input3
[    1.315072] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID
v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0
[    1.344055] Refined TSC clocksource calibration: 2491.907 MHz.
[    1.349823] systemd-udevd[163]: failed to execute
'/usr/lib/udev/socket:/org/kernel/dm/multipath_event'
'socket:/org/kernel/dm/multipath_event': No such file or directory
dracut-initqueue[144]: Warning: Could not boot.
dracut-initqueue[144]: Warning: /dev/mapper/mpathb does not exist


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.

dracut:/#

-- 
Thanks
Dave

Re: multipath device issue

[Dave Young]

Cc vivek

On 08/14/2012 02:16 PM, Dave Young wrote:

> 
>>> So, if you see /dev/mapper/mpathb be correctly created, when you start multpathd
>>> after all devices exist in /dev, then the 62-multipath.rules
>>> mechanism does not seem to work,
>>
>>>
>>
>>> With the next systemd release "rd.udev.log-priority=debug" does work again and
>>> we might be able to debug this issue.
>>
>>
>> Ok, will see if I can get something from the 62-multipath.rules. Thanks
>> for the suggestion.
>>
>>
> 
> 
> Hi, here is the debug messages with latest systemd, 62-multipath fails
> at run 'socket:/org/kernel/dm/multipath_event':
> 
> ---
> Welcome to Fedora 17 (Beefy Miracle) dracut-023-2.fc18 (Initramfs)!
> 
> [    0.799958] systemd[1]: No hostname configured.
> [    0.801608] systemd[1]: Set hostname to <localhost>.
> [    0.807834] systemd[1]: Cannot add dependency job for unit
> systemd-journal-flush.service, ignoring: Unit
> systemd-journal-flush.service failed to load: No such file or directory.
> See system logs and 'systemctl status systemd-journal-flush.service' for
> details.
> [  OK  ] Listening on udev Kernel Socket.
> [  OK  ] Listening on udev Control Socket.
> [  OK  ] Listening on Journal Socket.
>          Starting Journal Service...
> [    0.825703] systemd-journald[44]: Fixed max_use=11.8M max_size=1.4M
> min_size=64.0K keep_free=5.9M
> [    0.828702] systemd-journald[44]: Reserving 2695 entries in hash table.
> [    0.831147] systemd-journald[44]: Vacuuming...
> [  OK  ] Started Journal Service.
> [  OK  ] Reached target Sockets.
>          Starting Dracut cmdline hook...
>          Starting Setup Virtual Console...
> [  OK  ] Reached target Swap.
> [  OK  ] [    0.850156] usb 1-1: new full-speed USB device number 2
> using uhci_hcd
> Reached target Local File Systems.
> [  OK  ] Started Setup Virtual Console.
> [  OK  ] Reached target System Initialization.
> dracut-cmdline[46]: Warning: Option 'KEYTABLE' is deprecated, use
> 'KEYMAP' instead.
> dracut-cmdline[46]: Warning: Option 'SYSFONT' is deprecated, use 'FONT'
> instead.
> [  OK  ] Started Dracut cmdline hook.
>          Starting Dracut pre-udev hook...
> [  OK  ] Started Dracut pre-udev hook.
>          Starting udev Kernel Device Manager...
> [  OK  ] Started udev Kernel Device Manager.
>          Starting Dracut pre-trigger hook...
> [  OK  ] Reached target Basic System.
> [    0.950521] systemd-udevd[102]: starting version 188
> [    0.970090] device-mapper: multipath: version 1.3.0 loaded
> [    1.003546] input: ImExPS/2 Generic Explorer Mouse as
> /devices/platform/i8042/serio1/input/input2
> [    1.266732] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
> [    1.269940] usb 1-1: New USB device strings: Mfr=1, Product=3,
> SerialNumber=5
> [    1.272323] usb 1-1: Product: QEMU USB Tablet
> [    1.273889] usb 1-1: Manufacturer: QEMU 1.0
> [    1.275491] usb 1-1: SerialNumber: 42
> [    1.311638] input: QEMU 1.0 QEMU USB Tablet as
> /devices/pci0000:00/0000:00:01.2/usb1/1-1/1-1:1.0/input/input3
> [    1.315072] generic-usb 0003:0627:0001.0001: input,hidraw0: USB HID
> v0.01 Pointer [QEMU 1.0 QEMU USB Tablet] on usb-0000:00:01.2-1/input0
> [    1.344055] Refined TSC clocksource calibration: 2491.907 MHz.
> [    1.349823] systemd-udevd[163]: failed to execute
> '/usr/lib/udev/socket:/org/kernel/dm/multipath_event'
> 'socket:/org/kernel/dm/multipath_event': No such file or directory
> dracut-initqueue[144]: Warning: Could not boot.
> dracut-initqueue[144]: Warning: /dev/mapper/mpathb does not exist
> 
> 
> Entering emergency mode. Exit the shell to continue.
> Type "journalctl" to view system logs.
> 
> dracut:/#
> 



-- 
Thanks
Dave