[dm-crypt] Kernel update: "Failed to access temporary keystore device."

[dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Arno Wagner]

Hi all,

I just tried to upgrade my kernel from 3.10.48 to 3.14.15
(kernel.org). This is Debian wheezy. After the update, I
get "Failed to access temporary keystore device." when 
trying to unlock my LUKS partitions. As far as I can tell
I have not changed anything relevant in the kernel config,
I just did a "make oldconfig" with the old kernel .config.

What is missing here or how do I debug this?

Gr"usse,
Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Milan Broz]

On 08/01/2014 05:57 AM, Arno Wagner wrote:
> I just tried to upgrade my kernel from 3.10.48 to 3.14.15
> (kernel.org). This is Debian wheezy. After the update, I
> get "Failed to access temporary keystore device." when 
> trying to unlock my LUKS partitions. As far as I can tell
> I have not changed anything relevant in the kernel config,
> I just did a "make oldconfig" with the old kernel .config.

Hi,

for some strange reason I am tempting to ask if you read
the FAQ but... ;-)

Well, seriously: this happens when temporary mapped keyslot device
cannot be read (but kernel mapping was created successfully).
Not common problem, I do not even remember someone reported this... 

It seems like some udev/kernel compatibility problem (Debian
has non-standard dm/lvm udev rules btw).
Either bad access rights to device node or device node is missing
(the second is probably the issue).
It is possible you will need to use new udev or something.

Can you paste the command with added --debug?

Can you try to boot Debian provided kernel - does it work?
(Anyway, I am using custom kernel in Debian for years without problem
but I am using unstable repo.)

Milan

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Arno Wagner]

On Fri, Aug 01, 2014 at 08:20:21 CEST, Milan Broz wrote:
> On 08/01/2014 05:57 AM, Arno Wagner wrote:
> > I just tried to upgrade my kernel from 3.10.48 to 3.14.15
> > (kernel.org). This is Debian wheezy. After the update, I
> > get "Failed to access temporary keystore device." when 
> > trying to unlock my LUKS partitions. As far as I can tell
> > I have not changed anything relevant in the kernel config,
> > I just did a "make oldconfig" with the old kernel .config.

Hi Milan,

> 
> Hi,
> 
> for some strange reason I am tempting to ask if you read
> the FAQ but... ;-)

I assure you, I did. The FAQ writer has never heard of this 
problem ;-)
 
> Well, seriously: this happens when temporary mapped keyslot device
> cannot be read (but kernel mapping was created successfully).
> Not common problem, I do not even remember someone reported this... 
> 
> It seems like some udev/kernel compatibility problem (Debian
> has non-standard dm/lvm udev rules btw).

One more reason not to like udev. It used to be that you
just created the right devices manually and things worked...

> Either bad access rights to device node or device node is missing
> (the second is probably the issue).
> It is possible you will need to use new udev or something.
> 
> Can you paste the command with added --debug?

See below, both for 1.6.1 and 1.6.5, which unloaks without 
error (well, without error that gets propagated to the user), 
but never creates the entry in /dev/mapper/. Likely
a bug in 1.6.5, as it probably should tell the user that 
things went wrong.

> Can you try to boot Debian provided kernel - does it work?

Not easily. But it does work with 3.10.51, so the 3.2.x that
Debian stable is stuck at should probably work too. 

Come to think of it, I have /usr/src/linux pointing to a 3.4.67 
source tree, as gcc kernel includes in Debian stable are really 
messed up with 3.5.x and later and I failed to fix it manually.  
(Sometimes I really wonder what the Kernel devs are thinking or 
whether they are thinking at all...) Could that be the problem?

> (Anyway, I am using custom kernel in Debian for years without problem
> but I am using unstable repo.)

I usually run testing, except that I really do not want systemd,
so until I am sure I can do that update without getting that 
atrocity, no update to jessy for me. 

Anyways, if we do not figure this one out, I will just stay
with 3.10.x, it is a longterm-kernel after all. I just
tried 3.14.15 because I have some network issues and wanted to
see whether they may be gone with a newer kernel.

Arno


1.6.5:

# cryptsetup 1.6.5 processing
# "/home/wagner/tools/cryptsetup/cryptsetup-1.6.5/src/.libs/lt-cryptsetup
# --debug luksOpen /dev/md10 c1"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device /dev/md10 context.
# Trying to open and read device /dev/md10.
# Initialising device-mapper backend library.
# Trying to load LUKS1 crypt type from device /dev/md10.
# Crypto backend (gcrypt 1.5.0, flawed whirlpool) initialized.
# Reading LUKS header of size 1024 from device /dev/md10
# Key length 32, device size 419430272 sectors, header size 2050 sectors.
# Timeout set to 0 miliseconds.
# Password retry count set to 3.
# Password verification disabled.
# Iteration time set to 1000 miliseconds.
# Activating volume c1 [keyslot -1] using [none] passphrase.
# Detected kernel Linux 3.14.15 x86_64.
# dm version   OF   [16384] (*1)
# dm versions   OF   [16384] (*1)
# Detected dm-verity version 1.2.0.
# Detected dm-crypt version 1.13.0, dm-ioctl version 4.27.0.
# Device-mapper backend running with UDEV support enabled.
# dm status c1  OF   [16384] (*1)
# Interactive passphrase entry requested.
Enter passphrase for /dev/md10:
# Trying to open key slot 0 [ACTIVE].
# Reading key slot 0 area.
# Using userspace crypto wrapper to access keyslot area.
# Releasing crypt device /dev/md10 context.
# Releasing device-mapper backend.
# Unlocking memory.
Command failed with code 5: Input/output error

1.6.1:
# cryptsetup 1.6.1 processing "cryptsetup --debug luksOpen /dev/md10 c1"
# Running command open.
# Locking memory.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device /dev/md10 context.
# Trying to open and read device /dev/md10.
# Initialising device-mapper backend library.
# Trying to load LUKS1 crypt type from device /dev/md10.
# Crypto backend (gcrypt 1.5.0) initialized.
# Reading LUKS header of size 1024 from device /dev/md10
# Key length 32, device size 419430272 sectors, header size 2050 sectors.
# Timeout set to 0 miliseconds.
# Password retry count set to 3.
# Password verification disabled.
# Iteration time set to 1000 miliseconds.
# Activating volume c1 [keyslot -1] using [none] passphrase.
# dm version   OF   [16384] (*1)
# dm versions   OF   [16384] (*1)
# Detected dm-verity version 1.2.0.
# Detected dm-crypt version 1.13.0, dm-ioctl version 4.27.0.
# Device-mapper backend running with UDEV support enabled.
# dm status c1  OF   [16384] (*1)
# Interactive passphrase entry requested.
Enter passphrase for /dev/md10:
# Trying to open key slot 0 [ACTIVE].
# Reading key slot 0 area.
# Calculated device size is 250 sectors (RW), offset 8.
# DM-UUID is CRYPT-TEMP-temporary-cryptsetup-17411
# Udev cookie 0xd4dc8c7 (semid 9830400) created
# Udev cookie 0xd4dc8c7 (semid 9830400) incremented to 1
# Udev cookie 0xd4dc8c7 (semid 9830400) incremented to 2
# Udev cookie 0xd4dc8c7 (semid 9830400) assigned to CREATE task(0) with
# flags DISABLE_SUBSYSTEM_RULES DISABLE_DISK_RULES DISABLE_OTHER_RULES (0xe)
# dm create temporary-cryptsetup-17411 CRYPT-TEMP-temporary-cryptsetup-17411
# OF   [16384] (*1)
# dm reload temporary-cryptsetup-17411  OFRW    [16384] (*1)
# dm resume temporary-cryptsetup-17411  OFRW    [16384] (*1)
# temporary-cryptsetup-17411: Stacking NODE_ADD (253,0) 0:6 0660
# [verify_udev]
# temporary-cryptsetup-17411: Stacking NODE_READ_AHEAD 256 (flags=1)
# Udev cookie 0xd4dc8c7 (semid 9830400) decremented to 1
# Udev cookie 0xd4dc8c7 (semid 9830400) waiting for zero
# Udev cookie 0xd4dc8c7 (semid 9830400) destroyed
# temporary-cryptsetup-17411: Processing NODE_ADD (253,0) 0:6 0660
# [verify_udev]
# temporary-cryptsetup-17411: Processing NODE_READ_AHEAD 256 (flags=1)
# temporary-cryptsetup-17411 (253:0): read ahead is 256
# temporary-cryptsetup-17411 (253:0): Setting read ahead to 256
Failed to access temporary keystore device.
# Udev cookie 0xd4d53b6 (semid 9863168) created
# Udev cookie 0xd4d53b6 (semid 9863168) incremented to 1
# Udev cookie 0xd4d53b6 (semid 9863168) incremented to 2
# Udev cookie 0xd4d53b6 (semid 9863168) assigned to REMOVE task(2) with
# flags (0x0)
# dm remove temporary-cryptsetup-17411  OFT    [16384] (*1)
# temporary-cryptsetup-17411: Stacking NODE_DEL [verify_udev]
# Udev cookie 0xd4d53b6 (semid 9863168) decremented to 1
# Udev cookie 0xd4d53b6 (semid 9863168) waiting for zero
# Udev cookie 0xd4d53b6 (semid 9863168) destroyed
# temporary-cryptsetup-17411: Processing NODE_DEL [verify_udev]
# Releasing crypt device /dev/md10 context.
# Releasing device-mapper backend.
# Unlocking memory.
Command failed with code 5: Failed to access temporary keystore device.



-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Milan Broz]

On 08/03/2014 02:01 AM, Arno Wagner wrote:
>> Can you paste the command with added --debug?
> 
> See below, both for 1.6.1 and 1.6.5, which unloaks without 
> error (well, without error that gets propagated to the user), 
> but never creates the entry in /dev/mapper/. Likely
> a bug in 1.6.5, as it probably should tell the user that 
> things went wrong.

The 1.6.5 uses different code here (it reads device directly
when decrypting keyslot) and it need more user friendly error
messages here, my bad...

Anyway, seems like in both cases read of device really returns
I/O error while reading keyslot area.
Could you send me strace of the command?
(No need to enter correct password at all.)

BTW if not already there, it is another nice item to FAQ
- warn people that strace and similar debugging output can
easily leak keys or passwords. And yes, people sometimes
post these to lists :)


> 
>> Can you try to boot Debian provided kernel - does it work?
> 
> Not easily. But it does work with 3.10.51, so the 3.2.x that
> Debian stable is stuck at should probably work too. 
> 
> Come to think of it, I have /usr/src/linux pointing to a 3.4.67 
> source tree, as gcc kernel includes in Debian stable are really 
> messed up with 3.5.x and later and I failed to fix it manually.  
> (Sometimes I really wonder what the Kernel devs are thinking or 
> whether they are thinking at all...) Could that be the problem?

Don't think so... kernel should use own includes while compiling
and what's failing here is just plain read (I think). 

> I usually run testing, except that I really do not want systemd,
> so until I am sure I can do that update without getting that 
> atrocity, no update to jessy for me. 

There is a lot of discussion about this on debian devel,
IIRC systemd-shim is possible the way to avoid systemd as init.
(dunno if this will be supported).
 
> Anyways, if we do not figure this one out, I will just stay
> with 3.10.x, it is a longterm-kernel after all. I just
> tried 3.14.15 because I have some network issues and wanted to
> see whether they may be gone with a newer kernel.

Well, it would be interesting to find what's wrong here.

You are using MD device - what kind of raid is that?
(lsblk -t can say more info about storage stack topology as well).

Milan

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Arno Wagner]

On Sun, Aug 03, 2014 at 21:41:46 CEST, Milan Broz wrote:
> On 08/03/2014 02:01 AM, Arno Wagner wrote:
> >> Can you paste the command with added --debug?
> > 
> > See below, both for 1.6.1 and 1.6.5, which unloaks without 
> > error (well, without error that gets propagated to the user), 
> > but never creates the entry in /dev/mapper/. Likely
> > a bug in 1.6.5, as it probably should tell the user that 
> > things went wrong.
> 
> The 1.6.5 uses different code here (it reads device directly
> when decrypting keyslot) and it need more user friendly error
> messages here, my bad...
> 
> Anyway, seems like in both cases read of device really returns
> I/O error while reading keyslot area.
> Could you send me strace of the command?
> (No need to enter correct password at all.)

Looks like it. Strace output from a test container comes
in separate email.
 
> BTW if not already there, it is another nice item to FAQ
> - warn people that strace and similar debugging output can
> easily leak keys or passwords. And yes, people sometimes
> post these to lists :)

Good idea. Added as Item 4.5 and to the warnings at the start.

> > 
> >> Can you try to boot Debian provided kernel - does it work?
> > 
> > Not easily. But it does work with 3.10.51, so the 3.2.x that
> > Debian stable is stuck at should probably work too. 
> > 
> > Come to think of it, I have /usr/src/linux pointing to a 3.4.67 
> > source tree, as gcc kernel includes in Debian stable are really 
> > messed up with 3.5.x and later and I failed to fix it manually.  
> > (Sometimes I really wonder what the Kernel devs are thinking or 
> > whether they are thinking at all...) Could that be the problem?
> 
> Don't think so... kernel should use own includes while compiling
> and what's failing here is just plain read (I think). 
> 
> > I usually run testing, except that I really do not want systemd,
> > so until I am sure I can do that update without getting that 
> > atrocity, no update to jessy for me. 
> 
> There is a lot of discussion about this on debian devel,
> IIRC systemd-shim is possible the way to avoid systemd as init.
> (dunno if this will be supported).

We will see. I have a suspicion that the sudden long-term support
for pre-systemd Debian is not an accident.
  
> > Anyways, if we do not figure this one out, I will just stay
> > with 3.10.x, it is a longterm-kernel after all. I just
> > tried 3.14.15 because I have some network issues and wanted to
> > see whether they may be gone with a newer kernel.
> 
> Well, it would be interesting to find what's wrong here.

Ok, so lets keep poking at it. 

> You are using MD device - what kind of raid is that?
> (lsblk -t can say more info about storage stack topology as well).

It is a 3-way md RAID1 (on 2.5" laptop drives, about one firmware
crash per year...). 

"lsblk -t" does not say a lot:

NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE
md10         0   4096      0    4096     512    1           128

Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 332 bytes --]

On lun., 2014-08-04 at 03:13 +0200, Arno Wagner wrote:
> We will see. I have a suspicion that the sudden long-term support
> for pre-systemd Debian is not an accident.
  
Actually that's completely unrelated. Also, Wheezy LTS will only happen
if Squeeze LTS gains enough traction and volunteers.

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Arno Wagner]

On Mon, Aug 04, 2014 at 07:50:34 CEST, Yves-Alexis Perez wrote:
> On lun., 2014-08-04 at 03:13 +0200, Arno Wagner wrote:
> > We will see. I have a suspicion that the sudden long-term support
> > for pre-systemd Debian is not an accident.
>   
> Actually that's completely unrelated. Also, Wheezy LTS will only happen
> if Squeeze LTS gains enough traction and volunteers.
> 
> Regards,
> -- 
> Yves-Alexis

I will form a definite opinion about that in a few years. At this 
time, I think the timing is highly suggestive.

Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Yves-Alexis Perez]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

On lun., 2014-08-04 at 07:53 +0200, Arno Wagner wrote:
> I will form a definite opinion about that in a few years. At this 
> time, I think the timing is highly suggestive.

I'm a member of the security team, which took the decision to try the
LTS experiment, so I can assure you that's completely unrelated.

Now, maybe people will join the experiment because of that, but that's
not the intention behind it.

Regards,
-- 
Yves-Alexis

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [dm-crypt] Kernel update: "Failed to access temporary keystore device."

[Arno Wagner]

Hi,

just resolved this. I did indeed screw up the kernel config
update. After a second try, cryptsetup works fine with 
stock kernel.org 3.14.15.

Gr"usse,
Arno

On Mon, Aug 04, 2014 at 03:13:30 CEST, Arno Wagner wrote:
> On Sun, Aug 03, 2014 at 21:41:46 CEST, Milan Broz wrote:
> > On 08/03/2014 02:01 AM, Arno Wagner wrote:
> > >> Can you paste the command with added --debug?
> > > 
> > > See below, both for 1.6.1 and 1.6.5, which unloaks without 
> > > error (well, without error that gets propagated to the user), 
> > > but never creates the entry in /dev/mapper/. Likely
> > > a bug in 1.6.5, as it probably should tell the user that 
> > > things went wrong.
> > 
> > The 1.6.5 uses different code here (it reads device directly
> > when decrypting keyslot) and it need more user friendly error
> > messages here, my bad...
> > 
> > Anyway, seems like in both cases read of device really returns
> > I/O error while reading keyslot area.
> > Could you send me strace of the command?
> > (No need to enter correct password at all.)
> 
> Looks like it. Strace output from a test container comes
> in separate email.
>  
> > BTW if not already there, it is another nice item to FAQ
> > - warn people that strace and similar debugging output can
> > easily leak keys or passwords. And yes, people sometimes
> > post these to lists :)
> 
> Good idea. Added as Item 4.5 and to the warnings at the start.
> 
> > > 
> > >> Can you try to boot Debian provided kernel - does it work?
> > > 
> > > Not easily. But it does work with 3.10.51, so the 3.2.x that
> > > Debian stable is stuck at should probably work too. 
> > > 
> > > Come to think of it, I have /usr/src/linux pointing to a 3.4.67 
> > > source tree, as gcc kernel includes in Debian stable are really 
> > > messed up with 3.5.x and later and I failed to fix it manually.  
> > > (Sometimes I really wonder what the Kernel devs are thinking or 
> > > whether they are thinking at all...) Could that be the problem?
> > 
> > Don't think so... kernel should use own includes while compiling
> > and what's failing here is just plain read (I think). 
> > 
> > > I usually run testing, except that I really do not want systemd,
> > > so until I am sure I can do that update without getting that 
> > > atrocity, no update to jessy for me. 
> > 
> > There is a lot of discussion about this on debian devel,
> > IIRC systemd-shim is possible the way to avoid systemd as init.
> > (dunno if this will be supported).
> 
> We will see. I have a suspicion that the sudden long-term support
> for pre-systemd Debian is not an accident.
>   
> > > Anyways, if we do not figure this one out, I will just stay
> > > with 3.10.x, it is a longterm-kernel after all. I just
> > > tried 3.14.15 because I have some network issues and wanted to
> > > see whether they may be gone with a newer kernel.
> > 
> > Well, it would be interesting to find what's wrong here.
> 
> Ok, so lets keep poking at it. 
> 
> > You are using MD device - what kind of raid is that?
> > (lsblk -t can say more info about storage stack topology as well).
> 
> It is a 3-way md RAID1 (on 2.5" laptop drives, about one firmware
> crash per year...). 
> 
> "lsblk -t" does not say a lot:
> 
> NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE
> md10         0   4096      0    4096     512    1           128
> 
> Arno
> 
> -- 
> Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
> GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
> ----
> A good decision is based on knowledge and not on numbers. -  Plato
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -  Plato