Re: Update FC22 - doesn't find root fs and doesn't boot

[poma <pomidorabelisima-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

On 28.05.2015 14:19, Gerhard Wiesinger wrote:
> On 27.05.2015 15:27, Harald Hoyer wrote:
>> Am 27.05.2015 um 15:21 schrieb Gerhard Wiesinger:
>>> On 27.05.2015 15:07, Harald Hoyer wrote:
>>>> Am 27.05.2015 um 15:03 schrieb Gerhard Wiesinger:
>>>>> On 27.05.2015 14:43, Harald Hoyer wrote:
>>>>>> Am 27.05.2015 um 14:39 schrieb Gerhard Wiesinger:
>>>>>>> On 27.05.2015 14:25, Harald Hoyer wrote:
>>>>>>>> Am 27.05.2015 um 14:13 schrieb Andrei Borzenkov:
>>>>>>>>> On Wed, May 27, 2015 at 3:09 PM, Gerhard Wiesinger <lists-tcxHB8FAk+J54TAoqtyWWQ@public.gmane.org>
>>>>>>>>> wrote:
>>>>>>>>>> On 27.05.2015 13:56, Harald Hoyer wrote:
>>>>>>>>>>> Am 27.05.2015 um 13:28 schrieb Gerhard Wiesinger:
>>>>>>>>>>>> Hello,
>>>>>>>>>>>>
>>>>>>>>>>>> After upgrading one machine it does't boot anymore. Booting the old
>>>>>>>>>>>> kernel
>>>>>>>>>>>> works well.
>>>>>>>>>>>>
>>>>>>>>>>>> It is a RAID, LUKS, LVM setup. Find relevant details below.
>>>>>>>>>>>>
>>>>>>>>>>>> So it looks RAID assembling doesn't work any more.
>>>>>>>>>>>>
>>>>>>>>>>>> Any ideas?
>>>>>>>>>>>>
>>>>>>>>>>>> Thank you.
>>>>>>>>>>>>
>>>>>>>>>>>> Ciao,
>>>>>>>>>>>> Gerhard
>>>>>>>>>>>>
>>>>>>>>>>>> blkid
>>>>>>>>>>>> /dev/sda1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="5ec6792c-01"
>>>>>>>>>>>> /dev/sdb1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="1c7cffaf-01"
>>>>>>>>>>>> /dev/sdc1: UUID="56e9bde4-5cd1-630d-188c-22a54c1c8c37"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="0079dfc3-01"
>>>>>>>>>>>> /dev/sdd1: UUID="eb295015-e20b-e133-15db-5c9130487300"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="fadeaffe-01"
>>>>>>>>>>>> /dev/sde1: UUID="eb295015-e20b-e133-15db-5c9130487300"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="539b5814-01"
>>>>>>>>>>>> /dev/sdf1: UUID="eb295015-e20b-e133-15db-5c9130487300"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="badeaffe-01"
>>>>>>>>>>>> /dev/sdh1: UUID="51e8e983-8786-b868-dbc3-87357e1a9534"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="15631562-01"
>>>>>>>>>>>> /dev/sdg1: UUID="51e8e983-8786-b868-dbc3-87357e1a9534"
>>>>>>>>>>>> TYPE="linux_raid_member"
>>>>>>>>>>>> PARTUUID="162f0d0b-01"
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> blkid -o udev
>>>>>>>>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=5ec6792c-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=1c7cffaf-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_UUID_ENC=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=0079dfc3-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=fadeaffe-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=539b5814-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_UUID_ENC=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=badeaffe-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=51e8e983-8786-b868-dbc3-87357e1a9534
>>>>>>>>>>>> ID_FS_UUID_ENC=51e8e983-8786-b868-dbc3-87357e1a9534
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=15631562-01
>>>>>>>>>>>>
>>>>>>>>>>>> ID_FS_UUID=51e8e983-8786-b868-dbc3-87357e1a9534
>>>>>>>>>>>> ID_FS_UUID_ENC=51e8e983-8786-b868-dbc3-87357e1a9534
>>>>>>>>>>>> ID_FS_TYPE=linux_raid_member
>>>>>>>>>>>> ID_FS_PARTUUID=162f0d0b-01
>>>>>>>>>>>>
>>>>>>>>>>>> [    2.479836] myhost systemd[1]: Device
>>>>>>>>>>>> dev-disk-by\x2did-usb\x2dUSBDisk_RunDisk_ABCDEFGH\x2d0:0.device appeared
>>>>>>>>>>>> twice
>>>>>>>>>>>> with different sysfs paths /sys
>>>>>>>>>>>>
>>>>>>>>>>>> /devices/pci0000:00/0000:00:04.1/usb2/2-5/2-5:1.0/host9/target9:0:0/9:0:0:0/block/sdh
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and /sys/devices/pci0000:00/0000:00:02.1/usb1/1-5/1-5:1.0/host8/target
>>>>>>>>>>>> 8:0:0/8:0:0:0/block/sdg
>>>>>>>>>>>> [    2.500826] myhost systemd[1]: Device
>>>>>>>>>>>> dev-disk-by\x2did-usb\x2dUSBDisk_RunDisk_ABCDEFGH\x2d0:0\x2dpart1.device
>>>>>>>>>>>> appeared twice with different sysfs p
>>>>>>>>>>>> aths
>>>>>>>>>>>>
>>>>>>>>>>>> /sys/devices/pci0000:00/0000:00:04.1/usb2/2-5/2-5:1.0/host9/target9:0:0/9:0:0:0/block/sdh/sdh1
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> and /sys/devices/pci0000:00/0000:00:02.1/usb1/1-5/1-5:1.
>>>>>>>>>>>> 0/host8/target8:0:0/8:0:0:0/block/sdg/sdg1
>>>>>>>>>>>> [    2.632754] myhost kernel: Switched to clocksource tsc
>>>>>>>>>>>> [   91.162986] myhost systemd[1]: Job dev-md0.device/start timed out.
>>>>>>>>>>>> [   91.163138] myhost systemd[1]: Timed out waiting for device
>>>>>>>>>>>> dev-md0.device.
>>>>>>>>>>>> [   91.163251] myhost systemd[1]: Dependency failed for Cryptography
>>>>>>>>>>>> Setup for
>>>>>>>>>>>> luks-md0.
>>>>>>>>>>>> [   91.163360] myhost systemd[1]: Dependency failed for
>>>>>>>>>>>> dev-mapper-luks\x2dmd0.device.
>>>>>>>>>>>> [   91.163474] myhost systemd[1]: Job
>>>>>>>>>>>> dev-mapper-luks\x2dmd0.device/start
>>>>>>>>>>>> failed with result 'dependency'.
>>>>>>>>>>>> [   91.163586] myhost systemd[1]: Dependency failed for Encrypted
>>>>>>>>>>>> Volumes.
>>>>>>>>>>>> [   91.163694] myhost systemd[1]: Job cryptsetup.target/start failed
>>>>>>>>>>>> with
>>>>>>>>>>>> result 'dependency'.
>>>>>>>>>>>> [   91.163801] myhost systemd[1]: Job
>>>>>>>>>>>> systemd-cryptsetup@luks\x2dmd0.service/start failed with result
>>>>>>>>>>>> 'dependency'.
>>>>>>>>>>>> [   91.163910] myhost systemd[1]: Job dev-md0.device/start failed with
>>>>>>>>>>>> result
>>>>>>>>>>>> 'timeout'.
>>>>>>>>>>>> [   91.164021] myhost systemd[1]: Reached target System Initialization.
>>>>>>>>>>>> [   91.164129] myhost systemd[1]: Starting System Initialization.
>>>>>>>>>>>> [   91.164236] myhost systemd[1]: Reached target Basic System.
>>>>>>>>>>>> [   91.164346] myhost systemd[1]: Starting Basic System.
>>>>>>>>>>>> [  185.740238] myhost dracut-initqueue[340]: Warning: Could not boot.
>>>>>>>>>>>> [  185.741266] myhost systemd[1]: Received SIGRTMIN+20 from PID 341
>>>>>>>>>>>> (plymouthd).
>>>>>>>>>>>> [  185.841257] myhost dracut-initqueue[340]: Warning:
>>>>>>>>>>>> /dev/VolGroup00/root does
>>>>>>>>>>>> not exist
>>>>>>>>>>>> [  185.841403] myhost systemd[1]: Starting Dracut Emergency Shell...
>>>>>>>>>>>> [  185.859366] myhost systemd[1]: Received SIGRTMIN+21 from PID 341
>>>>>>>>>>>> (plymouthd).
>>>>>>>>>>>>
>>>>>>>>>>>> dracut:/# cat "/run/initramfs/rdsosreport.txt"
>>>>>>>>>>>> + cat /lib/dracut/dracut-041-10.fc22.1
>>>>>>>>>>>> dracut-041-10.fc22.1
>>>>>>>>>>>> + cat /proc/cmdline
>>>>>>>>>>>> BOOT_IMAGE=/vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/VolGroup00/root ro
>>>>>>>>>>>> quiet
>>>>>>>>>>>> console=ttyS0,115200n8 rd_NO_PLYMOUTH SYSFONT=latarcyrheb-sun16
>>>>>>>>>>>> LANG=en_US.UTF-
>>>>>>>>>>>> 8 KEYTABLE=de-latin1
>>>>>>>>>>> your kernel cmdline seems to be missing the raid configuration
>>>>>>>>>>>
>>>>>>>>>>> rd.md.uuid=56e9bde4-5cd1-630d-188c-22a54c1c8c37
>>>>>>>>>>> rd.md.uuid=eb295015-e20b-e133-15db-5c9130487300
>>>>>>>>>>> rd.md.uuid=51e8e983-8786-b868-dbc3-87357e1a9534
>>>>>>>>>>>
>>>>>>>>>>> ... whatever is needed to boot your system.
>>>>>>>>>>>
>>>>>>>>>>> # dracut --print-cmdline
>>>>>>>>>>>
>>>>>>>>>>> should tell you what kernel cmdline parameters are needed
>>>>>>>>>> Thank you for the fast reply.
>>>>>>>>>>
>>>>>>>>>> # comment: 3 separate lines!!!!! (maybe a problem)
>>>>>>>>>> dracut --print-cmdline
>>>>>>>>>>      rd.luks.uuid=luks-0c66fc77-a8a0-49e6-b96c-55e886a91f09
>>>>>>>>>> rd.lvm.lv=VolGroup00/root
>>>>>>>>>>      rd.lvm.lv=VolGroup00/swap
>>>>>>>>>>      rd.md.uuid=51e8e983:8786b868:dbc38735:7e1a9534
>>>>>>>>>> rd.md.uuid=56e9bde4:5cd1630d:188c22a5:4c1c8c37
>>>>>>>>>> resume=/dev/mapper/VolGroup00-swap root=/dev/mapper/VolGroup00-root
>>>>>>>>>> rootflags=rw,relatime,data=ordered rootfstype=ext4
>>>>>>>>>>
>>>>>>>>>> dracut --force 4.0.4-301.fc22.x86_64
>>>>>>>>>>
>>>>>>>>>> Still no correct entry in /etc/grub2.cfg
>>>>>>>>> I would be very surprised if dracut made any changes to grub.cfg. As I
>>>>>>>>> understand it, you either build host-specific dracut that will include
>>>>>>>>> this emulated command line or you have to add lines manually to your
>>>>>>>>> boot loader configuration.
>>>>>>>>>
>>>>>>>>>> linux16 /vmlinuz-4.0.4-301.fc22.x86_64 root=/dev/VolGroup00/root ro quiet
>>>>>>>>>> console=ttyS0,115200n8 rd_NO_PLYMOUTH SYSFONT=latarcyrheb-sun16
>>>>>>>>>> LANG=en_US.UTF-8 KEYTABLE=de-latin1
>>>>>>>>>>
>>>>>>>>>> Looks like a bug to me.
>>>>>>>> You have to add the options manually to the grub conf and all in the same
>>>>>>>> line.
>>>>>>>>
>>>>>>>> dracut does not modify your grub configuration.
>>>>>>> OK, I thought it was modifying it, at least in the past.
>>>>>>>
>>>>>>> Added it manually and it worked well.
>>>>>>>
>>>>>>> What's still strange:
>>>>>>>
>>>>>>> Last lines after startup, luks-md0 has already been started as the root fs is
>>>>>>> there ....
>>>>>>> Starting Cryptography Setup for luks-md0...
>>>>>>>             Starting Authorization Manager...
>>>>>>>
>>>>>>> Fedora release 22 (Twenty Two)
>>>>>>> Kernel 4.0.4-301.fc22.x86_64 on an x86_64 (ttyS0)
>>>>>>>
>>>>>>> # Rebooting, why is a passphrase necessary here?
>>>>>>> reboot
>>>>>>> Please enter passphrase for disk luks-md0!
>>>>>>>
>>>>>>> Possible bug here?
>>>>>>>
>>>>>>> Ciao,
>>>>>>> Gerhard
>>>>>>>
>>>>>> Did you add "  rd.luks.uuid=luks-0c66fc77-a8a0-49e6-b96c-55e886a91f09 " also?
>>>>> Yes, of course. :-)
>>>>>
>>>>>> What is your /etc/crypttab ?
>>>>> cat /etc/crypttab
>>>>> luks-md0                /dev/md0        none
>>>>> luks-md1                /dev/md1        /etc/crypto-key-md1
>>>>>
>>>>> BTW: Why did it work in previous versions without the long command line and
>>>>> predefined "hint" parameters?
>>>>>
>>>>> Ciao,
>>>>> Gerhard
>>>>>
>>>> Oh... please, please do not use kernel names like "md0" ... better use UUID=
>>>
>>> changed it, but didn't help. This machine was updated since around FC5. :-)
>>>
>>> Booting:
>>> [FAILED] Failed to start Cryptography Setup for luks-md0.
>>> See "systemctl status 'systemd-cryptsetup@luks\x2dmd0.service'" for details.
>>> [DEPEND] Dependency failed for Encrypted Volumes.
>>> ...
>>> [FAILED] Failed to start Cryptography Setup for luks-md0.
>>> See "systemctl status 'systemd-cryptsetup@luks\x2dmd0.service'" for details.
>>> [DEPEND] Dependency failed for Encrypted Volumes.
>>> ...
>>>           Starting Cryptography Setup for luks-md0...
>>>           Starting Authorization Manager...
>>>
>>> Fedora release 22 (Twenty Two)
>>>
>>> cat /etc/crypttab
>>> luks-md0 UUID="0c66fc77-a8a0-49e6-b96c-55e886a91f09"        none
>>> luks-md1 UUID="aaa6ec1e-e54f-4864-a793-cc95d5fa59d2" /etc/crypto-key-md1
>>>
>>> Ciao,
>>> Gerhard
>>>
>> Does "dracut -f" help to get the update crypttab in the initramfs?
>>
>>
> 
> Removing quotes from the UUID helped.
> 
> Thank you.
> 
> Ciao,
> Gerhard
> 


Without even mention these UUIDs spaghetti.

$ grep ^kernel /etc/dracut.conf.d/custom.conf
kernel_cmdline="rd.auto=1"

# lsinitrd -f

$ lsinitrd -f etc/cmdline.d/01-default.conf
rd.auto=1

This is -not- necessary for MD/RAID setups on Fedora 21:
$ rpm -q dracut
dracut-038-39.git20150518.fc21.x86_64

But it has become -mandatory- on MD/RAID setups on Fedora 22 and Fedora 23:
$ rpm -q dracut
dracut-041-10.fc22.1.x86_64

$ rpm -q dracut
dracut-041-10.git20150219.fc23.x86_64

And already we have fought for this user's side simplification, but it seems developers are very persistent when they do not need to be.