[linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[Vishal Chourasia]

Hi All,

I hope this email finds you well. I am currently facing an issue with
booting a VM using a custom-compiled kernel and would appreciate your
expertise on the matter.

### Problem Description
I have downloaded the `Fedora-Server-KVM-38-1.6.x86_64.qcow2` image and
successfully booted it using `qemu-system-x86_64`. However, when I try
to boot this VM with a custom-compiled kernel using the `-kernel` flag,
it fails to boot. The root filesystem is on LVM, and it seems the kernel
needs to activate volume groups before mounting the root filesystem.

### Steps Taken
1. Copied `/boot/config-*` from inside the VM and used it to build the
kernel on the host machine.
2. Also tried `make localmodconfig` after copying `/boot/config-*`.
3. Built `initrd.img` using `dracut`.

#### Dracut Command
```bash
dracut initrd.img --kmoddir $INSTALL_MOD_PATH/lib/modules/6.4.0+ 6.4.0+
```

#### Qemu Command
```bash
qemu-system-x86_64 -enable-kvm -nographic -vga none -smp 1 -m 2G \
-kernel arch/x86_64/boot/bzImage -initrd initrd.img -append
"root=/dev/vda3 ro console=ttyS0" \
-device virtio-blk-pci,drive=rootdisk -drive
file=Fedora-Server-KVM-38-1.6.x86_64.qcow2,if=none,cache=none,id=rootdisk,format=qcow2
\
-net nic,model=virtio -net user,hostfwd=tcp::5555-:22
```

### Error Observed
The kernel panics with the following message:
```
[    0.843818] List of all partitions:
[    0.844084] fd00         7340032 vda
[    0.844086]  driver: virtio_blk
[    0.844571]   fd01            1024 vda1
f3137bf4-2cf0-4ef6-98a9-dda32e23ddcd
[    0.844573]
[    0.845269]   fd02         1048576 vda2
3030ca3b-6606-4a8e-a5ee-efdc7858fb64
[    0.845270]
[    0.845976]   fd03         6288384 vda3
416b3009-59d3-4ed1-b126-3cb216aa02f4
[    0.845976]
[    0.846593] fd10        20971520 vdb
[    0.846594]  driver: virtio_blk
[    0.847091] 0b00         1048575 sr0
[    0.847092]  driver: sr
[    0.847545] No filesystem could mount root, tried:
[    0.847546]  ext3
[    0.847906]  ext2
[    0.848060]  ext4
[    0.848220]  fuseblk
[    0.848362]  xfs
[    0.848537]  btrfs
[    0.848674]
[    0.848951] Kernel panic - not syncing: VFS: Unable to mount root fs
on unknown-block(253,3)
[    0.849574] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 6.4.0+ #3
[    0.850012] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.o4
[    0.850837] Call Trace:
[    0.851109]  <TASK>
[    0.851261]  dump_stack_lvl+0x47/0x60
[    0.851524]  panic+0x178/0x330
[    0.851745]  mount_block_root+0x278/0x280
[    0.852030]  ? __pfx_ignore_unknown_bootoption+0x10/0x10
[    0.852403]  prepare_namespace+0x156/0x180
[    0.852705]  kernel_init_freeable+0x416/0x460
[    0.853031]  ? __pfx_kernel_init+0x10/0x10
[    0.853328]  kernel_init+0x1a/0x140
[    0.853585]  ret_from_fork+0x2c/0x50
[    0.853858]  </TASK>
[    0.854067] Kernel Offset: 0x16000000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfff)
[    0.854866] ---[ end Kernel panic - not syncing: VFS: Unable to mount
root fs on unknown-block(253,3) ]---
```

### Additional Information
- `/etc/fstab` and `lsblk` output are attached for reference.

#### /etc/fstab
```
/dev/mapper/sysvg-root  /                       xfs     defaults        0 0
UUID=2cd18fbf-7705-4a86-8bb8-c18dafdee822 /boot                   ext4
 defaults        1 2
```

#### lsblk
```
NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0             11:0    1 1024M  0 rom
zram0          251:0    0  3.8G  0 disk [SWAP]
vda            252:0    0    7G  0 disk
├─vda1         252:1    0    1M  0 part
├─vda2         252:2    0    1G  0 part /boot
└─vda3         252:3    0    6G  0 part
  └─sysvg-root 253:0    0    6G  0 lvm  /
```

#### lsinitrd initrd.img
```
Image: initrd.img: 49M
========================================================================
Version: dracut-049-223.git20230119.el8

Arguments: --force --kmoddir '$INSTALL_MOD_PATH/lib/modules/6.4.0+ 6.4.0+'

dracut modules:
bash
systemd
systemd-initrd
nss-softokn
i18n
network-manager
network
ifcfg
drm
plymouth
prefixdevname
crypt
dm
kernel-modules
kernel-modules-extra
kernel-network-modules
lvm
resume
rootfs-block
terminfo
udev-rules
biosdevname
dracut-systemd
usrmount
base
fs-lib
memstrack
microcode_ctl-fw_dir_override
shutdown
```

I would be grateful for any guidance or suggestions you could provide to
resolve this issue.


vishal.c

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[Zdenek Kabelac]

Dne 30. 08. 23 v 8:40 Vishal Chourasia napsal(a):
> Hi All,
> 
> I hope this email finds you well. I am currently facing an issue with
> booting a VM using a custom-compiled kernel and would appreciate your
> expertise on the matter.
> 

Hi

Not really sure how this relates to 'lvm2' yet - your kernel 'stuck' seems to 
be caused by the inability to switch to 'rootfs'  - which should be located on 
/dev/vda3  (according to your qemu exec line) - but kernel panics
reports that device does not have recognizable filesystem.

There seems to be no lvm2 involved at all so far.


> ### Problem Description
> I have downloaded the `Fedora-Server-KVM-38-1.6.x86_64.qcow2` image and
> successfully booted it using `qemu-system-x86_64`. However, when I try
> to boot this VM with a custom-compiled kernel using the `-kernel` flag,
> it fails to boot. The root filesystem is on LVM, and it seems the kernel
> needs to activate volume groups before mounting the root filesystem.

If you believe the filesystem is really on LVM and your /dev/vda3 is just a PV 
- then your boot line is wrong - and you need to be using different naming -
possibly something like:
root=/dev/vgname/lvrootname rd.lvm.lv=vgname/lvrootname

dracut that needs to include lvm2 code and be able to activate such LV prior 
switch to rootfs  (having somewhere inside  lvchange -ay vgname/lvrootname)

But it's not really clear how have you moved from your /dev/vda3 to
something on top of lvm2...

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[Vishal Chourasia]

On 8/30/23 20:01, Zdenek Kabelac wrote:
> Dne 30. 08. 23 v 8:40 Vishal Chourasia napsal(a):
>> Hi All,
>>
>> I hope this email finds you well. I am currently facing an issue with
>> booting a VM using a custom-compiled kernel and would appreciate your
>> expertise on the matter.
>>
> 
> Hi
> 
> Not really sure how this relates to 'lvm2' yet - your kernel 'stuck'
> seems to be caused by the inability to switch to 'rootfs'  - which
> should be located on /dev/vda3  (according to your qemu exec line) - but
> kernel panics
> reports that device does not have recognizable filesystem.
> 
> There seems to be no lvm2 involved at all so far.
> 
> 
>> ### Problem Description
>> I have downloaded the `Fedora-Server-KVM-38-1.6.x86_64.qcow2` image and
>> successfully booted it using `qemu-system-x86_64`. However, when I try
>> to boot this VM with a custom-compiled kernel using the `-kernel` flag,
>> it fails to boot. The root filesystem is on LVM, and it seems the kernel
>> needs to activate volume groups before mounting the root filesystem.
> 
> If you believe the filesystem is really on LVM and your /dev/vda3 is
> just a PV - then your boot line is wrong - and you need to be using
> different naming -
> possibly something like:
> root=/dev/vgname/lvrootname rd.lvm.lv=vgname/lvrootname
# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.4.7-200.fc38.x86_64
root=/dev/mapper/sysvg-root ro console=tty1 console=ttyS0,115200n8
rd.lvm.lv=sysvg/root

I have tried passing "root=/dev/mapper/sysvg-root rd.lvm.lv=sysvg/root"
with the -append option and it hasn't worked either.
> 
> dracut that needs to include lvm2 code and be able to activate such LV
> prior switch to rootfs  (having somewhere inside  lvchange -ay
> vgname/lvrootname)
1. How may I verify that dracut has included lvm2 code or not?
2. Which file in the dracut would contain
   `lvchange -ay vgname/lvrootname` code?
> 
> But it's not really clear how have you moved from your /dev/vda3 to
> something on top of lvm2...
Not sure what you mean by this.
#### lsblk
```
NAME           MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sr0             11:0    1 1024M  0 rom
zram0          251:0    0  3.8G  0 disk [SWAP]
vda            252:0    0    7G  0 disk
├─vda1         252:1    0    1M  0 part
├─vda2         252:2    0    1G  0 part /boot
└─vda3         252:3    0    6G  0 part
  └─sysvg-root 253:0    0    6G  0 lvm  /
```
From lsblk output I see sysvg-root is part of /dev/vda3 and therefore I
tried running qemu with different permutations. /dev/vda3 was one of
them but nothing has worked so far.

>> Regards
> 
> Zdenek
> 

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[Zdenek Kabelac]

Dne 31. 08. 23 v 0:40 Vishal Chourasia napsal(a):
> On 8/30/23 20:01, Zdenek Kabelac wrote:
>> Dne 30. 08. 23 v 8:40 Vishal Chourasia napsal(a):
>>> Hi All,
>>>
> # cat /proc/cmdline
> BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.4.7-200.fc38.x86_64
> root=/dev/mapper/sysvg-root ro console=tty1 console=ttyS0,115200n8
> rd.lvm.lv=sysvg/root
> 
> I have tried passing "root=/dev/mapper/sysvg-root rd.lvm.lv=sysvg/root"
> with the -append option and it hasn't worked either.
>>
>> dracut that needs to include lvm2 code and be able to activate such LV
>> prior switch to rootfs  (having somewhere inside  lvchange -ay
>> vgname/lvrootname)
> 1. How may I verify that dracut has included lvm2 code or not?
> 2. Which file in the dracut would contain
>     `lvchange -ay vgname/lvrootname` code?

Hi

If you are making ramdisk image from the system - dracut should autodetect 
such case and build the image with lvm2 support inside (unless you've 
instructed in dracut.conf to not doing so).

You can check if you dracut ramdisk image contains /usr/sbin/lvm
and also if dirs  in /usr/lib/dracut/hooks do have lvm2 scripts inside as well 
as /etc/lvm/lvm.conf.

Not really sure how are you trying to build this yourself since the Fedora 
anaconda installer should be doing all of this for you automatically - so how 
did you got to the point you don't get this from your installation time ?

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

Re: [linux-lvm] Trouble Booting Custom Kernel with QEMU: dracut-initqueue timeout waiting for /dev/sysvg/root

[Vishal Chourasia]

Sorry for the delay.

On 8/31/23 15:44, Zdenek Kabelac wrote:
> Dne 31. 08. 23 v 0:40 Vishal Chourasia napsal(a):
>> On 8/30/23 20:01, Zdenek Kabelac wrote:
>>> Dne 30. 08. 23 v 8:40 Vishal Chourasia napsal(a):
>>>> Hi All,
>>>>
>> # cat /proc/cmdline
>> BOOT_IMAGE=(hd0,gpt2)/vmlinuz-6.4.7-200.fc38.x86_64
>> root=/dev/mapper/sysvg-root ro console=tty1 console=ttyS0,115200n8
>> rd.lvm.lv=sysvg/root
>>
>> I have tried passing "root=/dev/mapper/sysvg-root rd.lvm.lv=sysvg/root"
>> with the -append option and it hasn't worked either.
>>>
>>> dracut that needs to include lvm2 code and be able to activate such LV
>>> prior switch to rootfs  (having somewhere inside  lvchange -ay
>>> vgname/lvrootname)
>> 1. How may I verify that dracut has included lvm2 code or not?
>> 2. Which file in the dracut would contain
>>     `lvchange -ay vgname/lvrootname` code?
> 
> Hi
> 
> If you are making ramdisk image from the system - dracut should
> autodetect such case and build the image with lvm2 support inside
> (unless you've instructed in dracut.conf to not doing so).
> 
> You can check if you dracut ramdisk image contains /usr/sbin/lvm
Yes. /usr/sbin/lvm is present

After unarchiving the initrd.img

$ find . -name lvm
./usr/sbin/lvm
./etc/lvm

> and also if dirs  in /usr/lib/dracut/hooks do have lvm2 scripts inside

λ ls usr/lib/dracut/hooks
cleanup  cmdline  emergency  initqueue  mount  netroot  pre-mount  pre-pivot  pre-shutdown  pre-trigger  pre-udev  shutdown  shutdown-emergency

λ grep -r -i lvm *
cmdline/30-parse-lvm.sh:if [ -e /etc/lvm/lvm.conf ] && ! getargbool 1 rd.lvm.conf -d -n rd_NO_LVMCONF; then
cmdline/30-parse-lvm.sh:    rm -f -- /etc/lvm/lvm.conf
cmdline/30-parse-lvm.sh:LV_DEVS="$(getargs rd.lvm.vg -d rd_LVM_VG=) $(getargs rd.lvm.lv -d rd_LVM_LV=)"
cmdline/30-parse-lvm.sh:if ! getargbool 1 rd.lvm -d -n rd_NO_LVM \
cmdline/30-parse-lvm.sh:    info "rd.lvm=0: removing LVM activation"
cmdline/30-parse-lvm.sh:    rm -f -- /etc/udev/rules.d/64-lvm*.rules

This seems scripts under cmdline directory parse lvm related parameters
 present in the cmdline

Were you looking for lvm related scripts under pre-mount or mount directories?

> as well as /etc/lvm/lvm.conf.

filtering lines that do not start with '#'

$ grep -v '^[[:space:]]*#\|^$' etc/lvm/lvm.conf
config {
}
devices {
}
allocation {
}
log {
}
backup {
}
shell {
}
global {
}
activation {
}
dmeventd {
}

$ grep -v '^[[:space:]]*#\|^$' etc/lvm/lvmlocal.conf
local {
}
> 
> Not really sure how are you trying to build this yourself since the
> Fedora anaconda installer should be doing all of this for you
> automatically - so how did you got to the point you don't get this from
> your installation time ?

I have a VM booted with qemu-system-x86_64 and I am building kernel and initrd
on the host. I am looking to boot the VM with -kernel and -initrd options
> 
> Regards
> 
> Zdenek
> 

Thanks

vishal.c

_______________________________________________
linux-lvm mailing list
linux-lvm@redhat.com
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/