dracut 008 luks key in external device - still broken

dracut 008 luks key in external device - still broken

[jaivuk]

Hello guys,

I installed F14 into virtualbox and I replicated the HDD layot I have
on my server. Root partition is encrypted together with other
partitions. I created this setup in Anaconda installer, so it should
be considered as "standard" install.
System boots fine (with dracut 006) - it asks for password once during the boot.

Then I installed dracut-008-7.fc15 from
http://koji.fedoraproject.org/koji/packageinfo?packageID=8714, I
updated the grub.conf (created by anaconda) for the new syntax and I
run plymouth-update-initrd so new dracut 008 is active.
Boot was OK without the key on external device.

However when I added rd.luks.key=/mykey:abcd-1234 into the grub
(Instead of abcd-1234 I use real UUID of my key) the boot fails and I
end up in the dracut shell.

Here is how my updated kernel parameters look like:

kernel /vmlinuz-2.6.35.11-83.fc14.i686 ro root=/dev/mapper/vg2-lv_root
rd.luks.uuid=luks-6508ce25-91d1-469a-9423-7b10ef00754e
rd.luks.uuid=luks-73608094-4b4d-48bf-99a6-0493aeb7498d
rd.luks.uuid=luks-9d1124c6-22fe-4572-984b-175c0e307a1f
rd.luks.uuid=luks-eac11ed2-4136-4f73-bda7-1af1c09fe644
rd.md.uuid=eb005502:33822bc2:b956ad0a:be45f8e0
rd.md.uuid=9ce2b0c0:ed400210:451f5dab:694b56f7
rd.md.uuid=5e644250:1dda1a02:9365481e:4e0aee0a
rd.md.uuid=2e0eedaf:41d79b6b:0bed1099:5adc22ef rd.lvm.lv=vg2/lv_root
rd.lvm.lv=vg2/lv_swap rd.dm=0 rd.luks.key=/mykey:abcd-1234
LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us

The change in dracut seem to be quite turbulent and I hope it will
settle a bit. It make take me a while to create patch for dracut 008
and then I can realize you are elsewhere with dracut 009...

dracut.kernel man page confuses me as well:
"           If luksdev is given, the specified key will only be
applied for that LUKS device. Possible values are the same
           as for keydev. Unless you have several LUKS devices, you
don't have to specify this parameter."

I have several luks devices, but one key only. Does it mean I have to
list them all manually for the key again?

So I have to ask you - what logic is used to mount luks partitions
with the key on external device in dracut 008?
(When I modified dracut 005 I had to add delay before USB was checked,
in dracut 008 USB is checked in the loop but if it fails, it does not
ask for password, but ends in shell.)

I have this idea how it can be done (when external key is specified)
I suggest to replace once instance of cryptsetup luksOpen command
which waits for password and blocks boot proccess
with two parallel instances which will try to unlock the same luks device:
- first instance will ask user for password - in the same way it is
done now - cryptsetup luksOpen,
- second intance will:
a) monitor keydev in the loop with sleep. Once keydev is mounted and
the key is found, it will try to unlock luks partition with that key
and when this is successful,
b) Then it will kill the crytpsetup process waiting in the first
instance, first instance will then recheck luks device was
successfully mounted and boot will continue.

In my view this can be implemented without any changes in the
cryptsetup tools. Another option to consider is to try to mount luks
device based on event comming from udev (once key is attached) - and
then perform point b) - what do you think?

In this way once user is asked for password he/she can either enter it
manually or insert the key.

What to you think about my ideas? And can I get any help in
trouble-shooting my problem with dracut 008?

Thank you,

Jaiv

Re: dracut 008 luks key in external device - still broken

[Andrey Borzenkov]

On Sat, Mar 19, 2011 at 5:02 PM, jaivuk <jaivuk-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
> However when I added rd.luks.key=/mykey:abcd-1234 into the grub
> (Instead of abcd-1234 I use real UUID of my key) the boot fails and I
> end up in the dracut shell.
>

If you cannot capture console log, at this point mount some filesystem
rw and dump dmesg ouput there. dracut logs into kmsg so it may provide
some insight into the problem.

> Here is how my updated kernel parameters look like:
>
> kernel /vmlinuz-2.6.35.11-83.fc14.i686 ro root=/dev/mapper/vg2-lv_root
> rd.luks.uuid=luks-6508ce25-91d1-469a-9423-7b10ef00754e
> rd.luks.uuid=luks-73608094-4b4d-48bf-99a6-0493aeb7498d
> rd.luks.uuid=luks-9d1124c6-22fe-4572-984b-175c0e307a1f
> rd.luks.uuid=luks-eac11ed2-4136-4f73-bda7-1af1c09fe644
> rd.md.uuid=eb005502:33822bc2:b956ad0a:be45f8e0
> rd.md.uuid=9ce2b0c0:ed400210:451f5dab:694b56f7
> rd.md.uuid=5e644250:1dda1a02:9365481e:4e0aee0a
> rd.md.uuid=2e0eedaf:41d79b6b:0bed1099:5adc22ef rd.lvm.lv=vg2/lv_root
> rd.lvm.lv=vg2/lv_swap rd.dm=0 rd.luks.key=/mykey:abcd-1234
> LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us
>
> The change in dracut seem to be quite turbulent and I hope it will
> settle a bit. It make take me a while to create patch for dracut 008
> and then I can realize you are elsewhere with dracut 009...
>
> dracut.kernel man page confuses me as well:
> "           If luksdev is given, the specified key will only be
> applied for that LUKS device. Possible values are the same
>           as for keydev. Unless you have several LUKS devices, you
> don't have to specify this parameter."
>
> I have several luks devices, but one key only. Does it mean I have to
> list them all manually for the key again?
>

no

> So I have to ask you - what logic is used to mount luks partitions
> with the key on external device in dracut 008?

Dracut tries to mount all devices that match <keydev> spefication and
checks whether <keypath> exists inside of this filesystem. If yes, it
remembers this device as suitable for specified <luksdev>. Later in
cryptroot-ask it will call

cryptsetup -d "$mntp/$keypath" luksOpen "$device" "$luksname"

where device is real device.

The first thing to try is to use just /keydev; this will try to apply
this key to every LUKS device and will probe all available disks.

> (When I modified dracut 005 I had to add delay before USB was checked,
> in dracut 008 USB is checked in the loop but if it fails, it does not
> ask for password, but ends in shell.)

It re queues cryptroot-ask for after udev queue is settled if key is
not available initially. If device is still not available at this
point, I am not sure what can be done sensibly.

>
> I have this idea how it can be done (when external key is specified)
[...]
>
> In my view this can be implemented without any changes in the
> cryptsetup tools. Another option to consider is to try to mount luks
> device based on event comming from udev (once key is attached) - and
> then perform point b) - what do you think?
>

Tested patches are welcome :)

> And can I get any help in
> trouble-shooting my problem with dracut 008?
>

Please provide dmesg output after failed boot.

Re: dracut 008 luks key in external device - still broken

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 3819 bytes --]

Excerpts from jaivuk's message of Sat Mar 19 15:02:09 +0100 2011:
> However when I added rd.luks.key=/mykey:abcd-1234 into the grub
> (Instead of abcd-1234 I use real UUID of my key) the boot fails and I
> end up in the dracut shell.
> 
> Here is how my updated kernel parameters look like:
> 
> kernel /vmlinuz-2.6.35.11-83.fc14.i686 ro root=/dev/mapper/vg2-lv_root
> rd.luks.uuid=luks-6508ce25-91d1-469a-9423-7b10ef00754e
> rd.luks.uuid=luks-73608094-4b4d-48bf-99a6-0493aeb7498d
> rd.luks.uuid=luks-9d1124c6-22fe-4572-984b-175c0e307a1f
> rd.luks.uuid=luks-eac11ed2-4136-4f73-bda7-1af1c09fe644
> rd.md.uuid=eb005502:33822bc2:b956ad0a:be45f8e0
> rd.md.uuid=9ce2b0c0:ed400210:451f5dab:694b56f7
> rd.md.uuid=5e644250:1dda1a02:9365481e:4e0aee0a
> rd.md.uuid=2e0eedaf:41d79b6b:0bed1099:5adc22ef rd.lvm.lv=vg2/lv_root
> rd.lvm.lv=vg2/lv_swap rd.dm=0 rd.luks.key=/mykey:abcd-1234

UUID should be prefixed with UUID=, so it should be:

  rd.luks.key=/mykey:UUID=abcd-1234

because you can specify labels there, too:

  rd.luks.key=/mykey:LABEL=keys


> The change in dracut seem to be quite turbulent and I hope it will
> settle a bit. It make take me a while to create patch for dracut 008
> and then I can realize you are elsewhere with dracut 009...

Options format is settled with 008. We're not planning any further
changes.


> dracut.kernel man page confuses me as well:
> "If luksdev is given, the specified key will only be applied for that
> LUKS device. Possible values are the same as for keydev. Unless you
> have several LUKS devices, you don't have to specify this parameter."
> 
> I have several luks devices, but one key only. Does it mean I have to
> list them all manually for the key again?

I meant that if you have several luks devices with different keys, you
can specify which key is for which device to avoid unnecessary
decryption of other devices.


> So I have to ask you - what logic is used to mount luks partitions
> with the key on external device in dracut 008?
> (When I modified dracut 005 I had to add delay before USB was checked,
> in dracut 008 USB is checked in the loop but if it fails, it does not
> ask for password, but ends in shell.)

In Dracut 008 devices are probed for key file asynchronously with udev
rules. If device eventually appears, it's probed for key. If key is
found, it's added to the list of possibles keys to try.


> I have this idea how it can be done (when external key is specified)
> I suggest to replace once instance of cryptsetup luksOpen command
> which waits for password and blocks boot proccess
> with two parallel instances which will try to unlock the same luks device:
> - first instance will ask user for password - in the same way it is
> done now - cryptsetup luksOpen,
> - second intance will:
> a) monitor keydev in the loop with sleep. Once keydev is mounted and
> the key is found, it will try to unlock luks partition with that key
> and when this is successful,
> b) Then it will kill the crytpsetup process waiting in the first
> instance, first instance will then recheck luks device was
> successfully mounted and boot will continue.
> 
> In my view this can be implemented without any changes in the
> cryptsetup tools. Another option to consider is to try to mount luks
> device based on event comming from udev (once key is attached) - and
> then perform point b) - what do you think?

As said above, it's based on udev since 008.


> In this way once user is asked for password he/she can either enter it
> manually or insert the key.

You suggest that for every luks device Dracut should stop and let the
user decide to choose between key or password?
-- 
Amadeusz Żołnowski

PGP key fpr: C700 CEDE 0C18 212E 49DA  4653 F013 4531 E1DB FAB5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: dracut 008 luks key in external device - still broken

[Amadeusz Żołnowski]

[-- Attachment #1: Type: text/plain, Size: 3066 bytes --]

Excerpts from jaivuk's message of Sun Mar 20 00:14:56 +0100 2011:
> > It re queues cryptroot-ask for after udev queue is settled if key is
> > not available initially. If device is still not available at this
> > point, I am not sure what can be done sensibly.
> 
> I think it should ask for password. And if it does so it would be good
> if I could still connect the key during the password prompt as I
> explained...

First of all I haven't taken into consideration fact, that one can want
to use dracut for mounting something more than root. That's why there's
only key or only password. But it probably would be possible to
optionally ask for password, too. I will have to think of cases with
multiple devices to be decrypted and improve the module.


> > Please provide dmesg output after failed boot.
> I have dmesg output now. However I do not want to violate any rules of
> this list so can you please advise how can I send you dmesg? I can see
> 3 options:
> - attatch it direclty (it has 50k)
> - send link to some external page where I can store it - do you
> suggest any for this purpose?
> - open a Fedora bugzilla ticket and send you ticket number

Please paste it somewhere.


> Which of these 3 options is the best in this case?
> 
> 2011/3/19 Amadeusz Żołnowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
> > UUID should be prefixed with UUID=, so it should be:
> >
> >  rd.luks.key=/mykey:UUID=abcd-1234
> 
> Thats a good advice - I tried that but no luck :(

What happens? Doesn't decrypt anything? Or just one of devices?


> > In Dracut 008 devices are probed for key file asynchronously with
> > udev rules. If device eventually appears, it's probed for key. If
> > key is found, it's added to the list of possibles keys to try.
>
> I like udev is used, however as I mentioned there is still some bug.

Please specify:
1) How did you build initramfs (which options) and please provide output
it has printed. (Append -v to options to get more verbose output.)
2) How did you specify kernel cmd line options.
3) What is expected to happend and what happens.
4) dmesg
5) Content of /tmp/ from initramfs time would be useful, too. (Content
of every file there.)


> > You suggest that for every luks device Dracut should stop and let
> > the user decide to choose between key or password?
>
> Isn't it the case now? What if I had different key for each luks
> partition?

It's applying keys specified at kernel cmd line. You can specify it
multiple times.


> If I udnerstand current dracut logic correctly then if key is found it
> is store in /tmp so it can be reused?

No. Paths (with devices) to keys are stored there.


> So in case an attempt to open another luks partition failes, then yes
> in my view it would be best if user can either enter key manually or
> insert the USB stick with the key (in case rd.luks.key was specified).

Hm, might be good idea.
-- 
Amadeusz Żołnowski

PGP key fpr: C700 CEDE 0C18 212E 49DA  4653 F013 4531 E1DB FAB5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: dracut 008 luks key in external device - still broken

[jaivuk]

2011/3/20 Amadeusz Żołnowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
> First of all I haven't taken into consideration fact, that one can want
> to use dracut for mounting something more than root. That's why there's
> only key or only password. But it probably would be possible to
> optionally ask for password, too. I will have to think of cases with
> multiple devices to be decrypted and improve the module.

Anaconda by added swap as well, and other things can be added manually
as a kernel parameter...

> What happens? Doesn't decrypt anything? Or just one of devices?
Apparently anything.

> Please specify:
> 1) How did you build initramfs (which options) and please provide output
> it has printed. (Append -v to options to get more verbose output.)
I used:
/usr/libexec/plymouth/plymouth-update-initrd
It takes a while to finish and it does not display anything - with or
without -v...

> 2) How did you specify kernel cmd line options.
It is in my first e-mail and you can see it in dmesg on the line 79 as well.
Link to dmesg is in point 4 below.

> 3) What is expected to happend and what happens.
Expected: I expect my machine boots in the same way as if
rd.luks.keydev is not specified.
What happens: it says it cannot mount md devices and I end up in dracut shell.

> 4) dmesg
http://pastebin.com/nA4qVFcZ

> 5) Content of /tmp/ from initramfs time would be useful, too. (Content
> of every file there.)
It has 3 files:

#console_init.tty0 - it is empty (size 0)

#luks.keys:
*:/dev/sdc1:/mykey

#root.info:
root='block:/dev/mapper/vg2-lv_root'
rflags='ro'
fstype='auto'
netroot=''
NEWROOT='/sysroot'

I added four more points:
6) To prove my key is OK, I tried below from dracut shell: (I'm sure
key for other partitions is fine as well)
#mkdir /mnt
#mkdir /mnt/USB
#mount /dev/sdc1 /mnt/USB
(successful)

#cryptsetup -d "/mnt/USB/mykey" luksOpen /dev/md1
luks-6508ce25-91d1-469a-9423-7b10ef00754e
(successful)

#ls /dev/mapper
control
luks-6508ce25-91d1-469a-9423-7b10ef00754e

7) Is there any option how to enable history and arrow keys in
dracut's shell? (Apart from adding bash into intramfs?)

8) My /etc/dracut.conf contains: (everything else is commented out:)
# list of kernel filesystem modules to be included in the generic initramfs
filesystems+="vfat fat"

# install local /etc/mdadm.conf
mdadmconf="yes"

# install local /etc/lvm/lvm.conf
lvmconf="yes"

9) Also please note that my system boots fine if I remove rd.luks.key
from the kernel options - it asks for pw and all is OK.
Here is dmesg log from successful boot (when rd.luks.key
option is not used):
http://pastebin.com/61yMGff5

Thank you very much for your help.

Re: dracut 008 luks key in external device - still broken

[Andrey Borzenkov]

2011/3/20 jaivuk <jaivuk-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2011/3/20 Amadeusz Żołnowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
>> First of all I haven't taken into consideration fact, that one can want
>> to use dracut for mounting something more than root. That's why there's
>> only key or only password. But it probably would be possible to
>> optionally ask for password, too. I will have to think of cases with
>> multiple devices to be decrypted and improve the module.
>
> Anaconda by added swap as well, and other things can be added manually
> as a kernel parameter...
>
>> What happens? Doesn't decrypt anything? Or just one of devices?
> Apparently anything.
>
[...]
>> 4) dmesg
> http://pastebin.com/nA4qVFcZ
>

It shows that no additional device was seen by dracut. Have you
connected your USB key before booting?

[...]
> I added four more points:
> 6) To prove my key is OK, I tried below from dracut shell: (I'm sure
> key for other partitions is fine as well)
> #mkdir /mnt
> #mkdir /mnt/USB
> #mount /dev/sdc1 /mnt/USB
> (successful)
>
> #cryptsetup -d "/mnt/USB/mykey" luksOpen /dev/md1
> luks-6508ce25-91d1-469a-9423-7b10ef00754e
> (successful)
>
> #ls /dev/mapper
> control
> luks-6508ce25-91d1-469a-9423-7b10ef00754e
>

Could please paste dmesg *after* you have done it?

Re: dracut 008 luks key in external device - still broken

[Andrey Borzenkov]

2011/3/20 Andrey Borzenkov <arvidjaar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2011/3/20 jaivuk <jaivuk-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> 2011/3/20 Amadeusz Żołnowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
>>> First of all I haven't taken into consideration fact, that one can want
>>> to use dracut for mounting something more than root. That's why there's
>>> only key or only password. But it probably would be possible to
>>> optionally ask for password, too. I will have to think of cases with
>>> multiple devices to be decrypted and improve the module.
>>
>> Anaconda by added swap as well, and other things can be added manually
>> as a kernel parameter...
>>
>>> What happens? Doesn't decrypt anything? Or just one of devices?
>> Apparently anything.
>>
> [...]
>>> 4) dmesg
>> http://pastebin.com/nA4qVFcZ
>>
>
> It shows that no additional device was seen by dracut. Have you
> connected your USB key before booting?
>

Oops, sorry :(

[    9.326667] dracut: Probing /dev/sdc1 for /mykey...
[   13.161241] dracut: Found /mykey on /dev/sdc1

> [...]
>> I added four more points:
>> 6) To prove my key is OK, I tried below from dracut shell: (I'm sure
>> key for other partitions is fine as well)
>> #mkdir /mnt
>> #mkdir /mnt/USB
>> #mount /dev/sdc1 /mnt/USB
>> (successful)
>>
>> #cryptsetup -d "/mnt/USB/mykey" luksOpen /dev/md1
>> luks-6508ce25-91d1-469a-9423-7b10ef00754e
>> (successful)
>>
>> #ls /dev/mapper
>> control
>> luks-6508ce25-91d1-469a-9423-7b10ef00754e
>>
>
> Could please paste dmesg *after* you have done it?
>

Re: dracut 008 luks key in external device - still broken

[Andrey Borzenkov]

2011/3/20 Andrey Borzenkov <arvidjaar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
> 2011/3/20 Andrey Borzenkov <arvidjaar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>> 2011/3/20 jaivuk <jaivuk-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>:
>>> 2011/3/20 Amadeusz ¯o³nowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
>>>> First of all I haven't taken into consideration fact, that one can want
>>>> to use dracut for mounting something more than root. That's why there's
>>>> only key or only password. But it probably would be possible to
>>>> optionally ask for password, too. I will have to think of cases with
>>>> multiple devices to be decrypted and improve the module.
>>>
>>> Anaconda by added swap as well, and other things can be added manually
>>> as a kernel parameter...
>>>
>>>> What happens? Doesn't decrypt anything? Or just one of devices?
>>> Apparently anything.
>>>
>> [...]
>>>> 4) dmesg
>>> http://pastebin.com/nA4qVFcZ
>>>
>>
>> It shows that no additional device was seen by dracut. Have you
>> connected your USB key before booting?
>>
>
> Oops, sorry :(
>
> [    9.326667] dracut: Probing /dev/sdc1 for /mykey...
> [   13.161241] dracut: Found /mykey on /dev/sdc1
>

[    3.272805] dracut: dracut-008-7.fc15

Key file parsing was fixed in current master:

commit b20ff981c4b31d650ac0b2e5227f4d6f6eb459e2
Author: Amadeusz ¯o³nowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>
Date:   Mon Mar 7 13:48:02 2011 +0100

    crypt-lib.sh: moved IFS=: from before 'read' to before 'while'

so you need updated modules.d/90crypt/crypt-lib.sh. May be you could
open a bug report in fedora to request update. I confirm that this bug
also happens with dash we have here.

Re: dracut 008 luks key in external device - still broken

[Andrey Borzenkov]

2011/3/20 Amadeusz Żołnowski <aidecoe-2qtfh70TtYba5EbDDlwbIw@public.gmane.org>:
> Excerpts from jaivuk's message of Sun Mar 20 00:14:56 +0100 2011:
>> > It re queues cryptroot-ask for after udev queue is settled if key is
>> > not available initially. If device is still not available at this
>> > point, I am not sure what can be done sensibly.
>>
>> I think it should ask for password. And if it does so it would be good
>> if I could still connect the key during the password prompt as I
>> explained...
>
> First of all I haven't taken into consideration fact, that one can want
> to use dracut for mounting something more than root.

Remembering recent discussion about separate /usr, it seems quite possible :)

> That's why there's
> only key or only password. But it probably would be possible to
> optionally ask for password, too. I will have to think of cases with
> multiple devices to be decrypted and improve the module.
>

 I think this can be combined with below. If keydev was not found
after udevsettle, fall through to password request, combined with
request to insert key if respective kernel option was present.

>> So in case an attempt to open another luks partition failes, then yes
>> in my view it would be best if user can either enter key manually or
>> insert the USB stick with the key (in case rd.luks.key was specified).
>
> Hm, might be good idea.

Yep. Currently if user forgot to plug in stick before booting it is
"Oops, I forgot to attach my key" and reboot. Stop there (after udev
settle) and give user chance to still plug in device,