[dm-crypt] openLuks failing - semaphore weird bug involved ?

[dm-crypt] openLuks failing - semaphore weird bug involved ?

[Ric Flomag]

[-- Attachment #1: Type: text/plain, Size: 2202 bytes --]

Hi all,

I've been using an external usb hard drive with three (no LVM)  partitions,
one of them crypt-luks (set up using palimpsest), for months. It used to get
auto-mounted under Gnome when plugged. Seahorse provided the key (Ubuntu
11.04).

Since a few days ago, it does not mount anymore. luksOpen yields a "no key
available" error message. No one around me can be suspected to have changed
the password. The other (plain ext4) partition mounts normally.

I can backup or dump the header without a problem. Another usb hardrive,
with the same luks setup (as shown by header dumps), mounts as usual. All
this is reproducible on other systems (another ubuntu 11.04 and a 11.10
beta).

Note that I have the "weird semaphore bug" described here:
http://thread.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/5237/focus=5243,
though, in my case:
  * it happens approx. once every three luksOpen attempts
  * It is not related to luksOpen success or failure (the semaphore error
message is just displayed sometimes with any of both hard drives)
  * Chromium is not involved

Coincidentally, I have started to use my hard drives with ubuntu 11.00 beta
a few days ago. It could have messed up something on one hd and not the
other?

Any hint to try and fix this ?

------------- faulty partition
LUKS header information for /dev/sdb3

Version:           1
Cipher name:       aes
Cipher mode:       cbc-essiv:sha256
Hash spec:         sha1
Payload offset:    2056
MK bits:           256
MK digest:         70 41 d4 78 bd 91 9e 82 c3 9c 61 df e8 6b 4e 28 2f 2b 90
94
MK salt:           a8 c6 7f 6f 35 2b 48 19 cc e7 27 7a 4f a6 ca a3
                   9f 78 6d f8 21 b0 48 56 44 27 2c 53 75 bf 0c 86
MK iterations:     22875
UUID:              4b74a5b1-0c46-4808-98c0-3a51ec5f7c8e

Key Slot 0: ENABLED
    Iterations:             91811
    Salt:                   24 f0 44 14 a7 4d ae e6 b6 2b 42 7c 8d 10 87 39
                              47 05 fd 35 14 46 62 12 f7 5a 7e 99 8a 67 33
f5
    Key material offset:    8
    AF stripes:                4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

[-- Attachment #2: Type: text/html, Size: 2498 bytes --]

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Arno Wagner]

As the header seems to be intact, the possibilities I see are 

- A changed keymap. Some people had problems with special
  symbols moving form ASCII encoding to UTF-8 encoding.
  This yields different passphrase-hashes. See FAQ.

  I am not familiar with seahorse. Maybe its passprase
  input fails and theis leads to cryptsetup getting
  an invalid passphrase as input? Anyways, try to 
  decrypt the key manually and see whether you get a 
  good passphrase for LUKS or an error and in the second
  case, try a manual luksOpen with that passphrase.
 
- Keyslot damage. Look at the first keyslot with a
  hex dumper, e.g. hd or hexdump, and see whether there
  are obvious patterns. Offset and length are in the
  FAQ.

- A broken LUKS version. In that case, can you try
  with, e.g. a life-CD of the old version you used?
  Or download, compile and try manually with the 
  current cryptsetup version? 

The first and last option should yield a complete recovery.
A damaged keyslot is only recoverable with a header backup.

Arno


On Wed, Sep 07, 2011 at 02:30:32PM -0400, Ric Flomag wrote:
> Hi all,
> 
> I've been using an external usb hard drive with three (no LVM)  partitions,
> one of them crypt-luks (set up using palimpsest), for months. It used to get
> auto-mounted under Gnome when plugged. Seahorse provided the key (Ubuntu
> 11.04).
> 
> Since a few days ago, it does not mount anymore. luksOpen yields a "no key
> available" error message. No one around me can be suspected to have changed
> the password. The other (plain ext4) partition mounts normally.
> 
> I can backup or dump the header without a problem. Another usb hardrive,
> with the same luks setup (as shown by header dumps), mounts as usual. All
> this is reproducible on other systems (another ubuntu 11.04 and a 11.10
> beta).
> 
> Note that I have the "weird semaphore bug" described here:
> http://thread.gmane.org/gmane.linux.kernel.device-mapper.dm-crypt/5237/focus=5243,
> though, in my case:
>   * it happens approx. once every three luksOpen attempts
>   * It is not related to luksOpen success or failure (the semaphore error
> message is just displayed sometimes with any of both hard drives)
>   * Chromium is not involved
> 
> Coincidentally, I have started to use my hard drives with ubuntu 11.00 beta
> a few days ago. It could have messed up something on one hd and not the
> other?
> 
> Any hint to try and fix this ?
> 
> ------------- faulty partition
> LUKS header information for /dev/sdb3
> 
> Version:           1
> Cipher name:       aes
> Cipher mode:       cbc-essiv:sha256
> Hash spec:         sha1
> Payload offset:    2056
> MK bits:           256
> MK digest:         70 41 d4 78 bd 91 9e 82 c3 9c 61 df e8 6b 4e 28 2f 2b 90
> 94
> MK salt:           a8 c6 7f 6f 35 2b 48 19 cc e7 27 7a 4f a6 ca a3
>                    9f 78 6d f8 21 b0 48 56 44 27 2c 53 75 bf 0c 86
> MK iterations:     22875
> UUID:              4b74a5b1-0c46-4808-98c0-3a51ec5f7c8e
> 
> Key Slot 0: ENABLED
>     Iterations:             91811
>     Salt:                   24 f0 44 14 a7 4d ae e6 b6 2b 42 7c 8d 10 87 39
>                               47 05 fd 35 14 46 62 12 f7 5a 7e 99 8a 67 33
> f5
>     Key material offset:    8
>     AF stripes:                4000
> Key Slot 1: DISABLED
> Key Slot 2: DISABLED
> Key Slot 3: DISABLED
> Key Slot 4: DISABLED
> Key Slot 5: DISABLED
> Key Slot 6: DISABLED
> Key Slot 7: DISABLED

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Ric Flomag]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]

Claudio and Arno,

Thank you for your kind help. I have tried the following:
 * manually unlock the partition with a Fedora 15 live (it has cryptsetup
1.3), and with an Ubuntu 11.04 live (which has 1.1 without the updates
installed on my system). No luck, cryptsetup returns "no available key".
 * looked at the hex data of the keyslot (between 0x01000 and 0x20400) : no
obvious patterns

As for a possible problem with seahorse: I use the same password on both
hard drives, seahorse shows them correctly, i can manually unlock the other
hard drive with the password and with seahorse... so I don't think that the
problem lies here.

Unfortunately i had not backed up the header nor retrieved the master key.
It's ok though, this hard drive only contains backups and unimportant stuff.
I'll just format the partition if I don't find a solution. And then backup
the header and the master key :D

What worries me is that I have no clue what happened. A damage to the header
is it possible without any signs of it ?

Ric.

[-- Attachment #2: Type: text/html, Size: 1091 bytes --]

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Arno Wagner]

On Thu, Sep 08, 2011 at 11:46:42AM -0400, Ric Flomag wrote:
> Claudio and Arno,
> 
> Thank you for your kind help. I have tried the following:
>  * manually unlock the partition with a Fedora 15 live (it has cryptsetup
> 1.3), and with an Ubuntu 11.04 live (which has 1.1 without the updates
> installed on my system). No luck, cryptsetup returns "no available key".
>  * looked at the hex data of the keyslot (between 0x01000 and 0x20400) : no
> obvious patterns

Which does not mean no patterns. This is the likely explanation.
The key-slots do not have any checksums, as that would decrease
security.

As this problem crops up frequantly, I do have a tool under 
development that does frequency-analysis on the keyslots
in a sector-wise fashion, but I currently have no time to 
finish it. This should be one order of magnitude or so better
than looking at it.
 
> As for a possible problem with seahorse: I use the same password on both
> hard drives, seahorse shows them correctly, i can manually unlock the other
> hard drive with the password and with seahorse... so I don't think that the
> problem lies here.

Agreed.

> Unfortunately i had not backed up the header nor retrieved the master key.
> It's ok though, this hard drive only contains backups and unimportant stuff.
> I'll just format the partition if I don't find a solution. And then backup
> the header and the master key :D

:-) 
You can protect the backup, e.g. by encrypting it with GnuPG.

> What worries me is that I have no clue what happened. A damage to the header
> is it possible without any signs of it ?

Well, if some random-looking data was copied into it, that is hard to
spot. A frequency-analysis would help, but takes some work to
implement. The easy and reliable way would be to compare the
header with its backup.

I do agree that header damage should not happen in an ideal world.
However, from the questions asked here, header damage is a not so
infrequent problem. That is one of the reasons I wrote the FAQ ;-)

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Claudio Moretti]

well, you were lucky you lost only unimportant data :) there were a
lot of people, first of all: me, who did not backup their header on
the main system.. And they (read: me) lost access to their home
partition because the only place where the passphrase had been stored
was.. the damaged root partition..
And that's how i learned i should back up headers (lol)

Though i know losing access to a backup-and-something partition makes
you feel bad too :|

On 08/09/2011, Arno Wagner <arno@wagner.name> wrote:
> On Thu, Sep 08, 2011 at 11:46:42AM -0400, Ric Flomag wrote:
>> Claudio and Arno,
>>
>> Thank you for your kind help. I have tried the following:
>>  * manually unlock the partition with a Fedora 15 live (it has cryptsetup
>> 1.3), and with an Ubuntu 11.04 live (which has 1.1 without the updates
>> installed on my system). No luck, cryptsetup returns "no available key".
>>  * looked at the hex data of the keyslot (between 0x01000 and 0x20400) :
>> no
>> obvious patterns
>
> Which does not mean no patterns. This is the likely explanation.
> The key-slots do not have any checksums, as that would decrease
> security.
>
> As this problem crops up frequantly, I do have a tool under
> development that does frequency-analysis on the keyslots
> in a sector-wise fashion, but I currently have no time to
> finish it. This should be one order of magnitude or so better
> than looking at it.
>
>> As for a possible problem with seahorse: I use the same password on both
>> hard drives, seahorse shows them correctly, i can manually unlock the
>> other
>> hard drive with the password and with seahorse... so I don't think that
>> the
>> problem lies here.
>
> Agreed.
>
>> Unfortunately i had not backed up the header nor retrieved the master key.
>> It's ok though, this hard drive only contains backups and unimportant
>> stuff.
>> I'll just format the partition if I don't find a solution. And then backup
>> the header and the master key :D
>
> :-)
> You can protect the backup, e.g. by encrypting it with GnuPG.
>
>> What worries me is that I have no clue what happened. A damage to the
>> header
>> is it possible without any signs of it ?
>
> Well, if some random-looking data was copied into it, that is hard to
> spot. A frequency-analysis would help, but takes some work to
> implement. The easy and reliable way would be to compare the
> header with its backup.
>
> I do agree that header damage should not happen in an ideal world.
> However, from the questions asked here, header damage is a not so
> infrequent problem. That is one of the reasons I wrote the FAQ ;-)
>
> Arno
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
>
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
>

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Karl O. Pinc]

On 09/08/2011 04:53:45 PM, Claudio Moretti wrote:
> well, you were lucky you lost only unimportant data :) there were a
> lot of people, first of all: me, who did not backup their header on
> the main system.. And they (read: me) lost access to their home
> partition because the only place where the passphrase had been stored
> was.. the damaged root partition..
> And that's how i learned i should back up headers (lol)

As I'm sure has been said by others: it's a lot more important
to back up your _data_ than to backup the luks header.

To paraphrase R. Crumb: "It is better to have data
in times of no luks header than it is to have a luks header
in times of no data."


Karl <kop@meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Arno Wagner]

On Thu, Sep 08, 2011 at 05:21:35PM -0500, Karl O. Pinc wrote:
> On 09/08/2011 04:53:45 PM, Claudio Moretti wrote:
> > well, you were lucky you lost only unimportant data :) there were a
> > lot of people, first of all: me, who did not backup their header on
> > the main system.. And they (read: me) lost access to their home
> > partition because the only place where the passphrase had been stored
> > was.. the damaged root partition..
> > And that's how i learned i should back up headers (lol)
> 
> As I'm sure has been said by others: it's a lot more important
> to back up your _data_ than to backup the luks header.

Indeed. 
 
> To paraphrase R. Crumb: "It is better to have data
> in times of no luks header than it is to have a luks header
> in times of no data."

;-)==)

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Ric Flomag]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

>
> If the drive has smart you could ask the drive's diagnostics
> if there's any indication of trouble.
>
>
I am not familiar with SMART. It reports 4 pending sectors and nothing else
wrong. I have run a "badblocks -nvs
/dev/sdb3<https://one.ubuntu.com/dev/sdb3>",
it did not find any error... And after this, SMART still reports 4 pending
sectors.

Any clue about how investigating further ?

Regards.


2011/9/8 Karl O. Pinc <kop@meme.com>

> On 09/08/2011 10:46:42 AM, Ric Flomag wrote:
>
>
> > What worries me is that I have no clue what happened. A damage to the
> > header
> > is it possible without any signs of it ?
>
> I've not been paying attention, but have a thought.
>
> If the drive has smart you could ask the drive's diagnostics
> if there's any indication of trouble.
>
> Regards,
>
> Karl <kop@meme.com>
> Free Software:  "You don't pay back, you pay forward."
>                 -- Robert A. Heinlein
>
>

[-- Attachment #2: Type: text/html, Size: 1518 bytes --]

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Arno Wagner]

On Thu, Sep 15, 2011 at 11:54:30AM -0400, Ric Flomag wrote:
> >
> > If the drive has smart you could ask the drive's diagnostics
> > if there's any indication of trouble.
> >
> >
> I am not familiar with SMART. It reports 4 pending sectors and nothing else
> wrong. 

That is pretty bad. It means 4 unreadable sectors. Modern HDDs
try pretty had to read damaged sectors ond usually succeed. 

>I have run a "badblocks -nvs
> /dev/sdb3<https://one.ubuntu.com/dev/sdb3>",
> it did not find any error... And after this, SMART still reports 4 pending
> sectors.

Meaning the pending sectors are someplae else.

> Any clue about how investigating further ?

badblocks on /dev/sdb.

Arno

> Regards.
> 
> 
> 2011/9/8 Karl O. Pinc <kop@meme.com>
> 
> > On 09/08/2011 10:46:42 AM, Ric Flomag wrote:
> >
> >
> > > What worries me is that I have no clue what happened. A damage to the
> > > header
> > > is it possible without any signs of it ?
> >
> > I've not been paying attention, but have a thought.
> >
> > If the drive has smart you could ask the drive's diagnostics
> > if there's any indication of trouble.
> >
> > Regards,
> >
> > Karl <kop@meme.com>
> > Free Software:  "You don't pay back, you pay forward."
> >                 -- Robert A. Heinlein
> >
> >

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Ricflomag]

>> I have run a "badblocks -nvs
>> /dev/sdb3<https://one.ubuntu.com/dev/sdb3>",
>> it did not find any error... And after this, SMART still reports 4 pending
>> sectors.
> Meaning the pending sectors are someplae else.
>
>> Any clue about how investigating further ?
> badblocks on /dev/sdb.
>
> Arno
That would tell me where the pending sectors are, outside of the luks 
partition. But it would not help to know why the luks partition fails to 
open, would it ?

Re: [dm-crypt] openLuks failing - semaphore weird bug involved ?

[Arno Wagner]

On Thu, Sep 15, 2011 at 12:15:14PM -0400, Ricflomag wrote:
> 
> >>I have run a "badblocks -nvs
> >>/dev/sdb3<https://one.ubuntu.com/dev/sdb3>",
> >>it did not find any error... And after this, SMART still reports 4 pending
> >>sectors.
> >Meaning the pending sectors are someplae else.
> >
> >>Any clue about how investigating further ?
> >badblocks on /dev/sdb.
> >
> >Arno
> That would tell me where the pending sectors are, outside of the
> luks partition. But it would not help to know why the luks partition
> fails to open, would it ?

It should not. However, the disk is suspect. Solve that first. 
Defective storage (or hardware in general) can cause all sorts 
of strange effects.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier