[dm-crypt] LVM on LUKS: volumes missing

[dm-crypt] LVM on LUKS: volumes missing

[fauno]

[-- Attachment #1.1: Type: text/plain, Size: 832 bytes --]

Hi, maybe I'm too shocked but I couldn't find anything on this issue :)

I have a fully encrypted HD using the LVM on LUKS method from
ArchWiki[^0], with the LUKS header and key file on an external device.

Today I started having some disk failures (root remounted ro, xfs
partition giving errors), and after I decided to reboot to run fsck, I
can't find anything.

When the encrypted partition is opened, I don't see any errors, not even
on dmesg, but LVM can't find any volume.  They're just missing.

Is there anything I can do? Thanks!

FWIW I had the same issue with another HD a few months back, though it
didn't had physical errors.  It didn't had anything important so I
wasn't worried.


[^0]:
https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS


-- 
.oÓ)




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 585 bytes --]

Re: [dm-crypt] LVM on LUKS: volumes missing

[Ondrej Kozina]

Hi fauno,

provided the driver was unlocked successfully it seems unrelated to 
cryptsetup/LUKS to me. Could we move the discussion to lvm mail list?

On 05/30/2016 11:54 PM, fauno wrote:
> Hi, maybe I'm too shocked but I couldn't find anything on this issue :)
>
> I have a fully encrypted HD using the LVM on LUKS method from
> ArchWiki[^0], with the LUKS header and key file on an external device.
>
> Today I started having some disk failures (root remounted ro, xfs
> partition giving errors), and after I decided to reboot to run fsck, I
> can't find anything.

Could you paste here output of pvscan -vvvv --cache 
/dev/mapper/insert_the_unlocked_device_name? Together with your 
/etc/lvm/lvm.conf file? If it's a device with rootfs we're talking about 
you will most probably have to extract /etc/lvm/lvm.conf file from 
initramfs image.

Well, generally, if disk sectors accommodating PV header are damaged, 
lvm2 won't recognise the device...

>
> When the encrypted partition is opened, I don't see any errors, not even
> on dmesg, but LVM can't find any volume.  They're just missing.
>
> Is there anything I can do? Thanks!
>
> FWIW I had the same issue with another HD a few months back, though it
> didn't had physical errors.  It didn't had anything important so I
> wasn't worried.

Ok, try same approach as above for this drive.

Regards
Ondrej

Re: [dm-crypt] LVM on LUKS: volumes missing

[Robert Nichols]

On 05/30/2016 04:54 PM, fauno wrote:
> Hi, maybe I'm too shocked but I couldn't find anything on this issue :)
>
> I have a fully encrypted HD using the LVM on LUKS method from
> ArchWiki[^0], with the LUKS header and key file on an external device.
>
> Today I started having some disk failures (root remounted ro, xfs
> partition giving errors), and after I decided to reboot to run fsck, I
> can't find anything.
>
> When the encrypted partition is opened, I don't see any errors, not even
> on dmesg, but LVM can't find any volume.  They're just missing.

Take a look at the decrypted volume with "hexedit -s". It should start 
out with mostly binary zeros with a little data including the ASCII 
string "LVM2" at the start of a few of the sectors. Starting at about 
the 10th sector there should be a lot of ASCII text. (It's a copy of the 
corresponding file in /etc/lvm/backup, though without the fancy formatting.)

If you're seeing random-appearing binary junk there, then the volume is 
not be decrypted properly. If it's just some of the early sectors that 
are clobbered and the text starting at the 10th sector is intact, then 
this should be recoverable.

-- 
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.

Re: [dm-crypt] LVM on LUKS: volumes missing

[fauno]

[-- Attachment #1.1: Type: text/plain, Size: 869 bytes --]

On 31/05/16 09:52, Robert Nichols wrote:

> Take a look at the decrypted volume with "hexedit -s". It should start
> out with mostly binary zeros with a little data including the ASCII
> string "LVM2" at the start of a few of the sectors. Starting at about
> the 10th sector there should be a lot of ASCII text. (It's a copy of the
> corresponding file in /etc/lvm/backup, though without the fancy
> formatting.)
> 
> If you're seeing random-appearing binary junk there, then the volume is
> not be decrypted properly. If it's just some of the early sectors that
> are clobbered and the text starting at the 10th sector is intact, then
> this should be recoverable.
> 

i've tried reading `hexedit` both on the closed and opened partition and
everything seems to be random junk.  `strings` also can't find anything
resembling LVM data.


-- 
:{


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 585 bytes --]

Re: [dm-crypt] LVM on LUKS: volumes missing

[fauno]

[-- Attachment #1.1: Type: text/plain, Size: 994 bytes --]

On 31/05/16 11:54, fauno wrote:
> On 31/05/16 09:52, Robert Nichols wrote:
> 
>> Take a look at the decrypted volume with "hexedit -s". It should start
>> out with mostly binary zeros with a little data including the ASCII
>> string "LVM2" at the start of a few of the sectors. Starting at about
>> the 10th sector there should be a lot of ASCII text. (It's a copy of the
>> corresponding file in /etc/lvm/backup, though without the fancy
>> formatting.)
>>
>> If you're seeing random-appearing binary junk there, then the volume is
>> not be decrypted properly. If it's just some of the early sectors that
>> are clobbered and the text starting at the 10th sector is intact, then
>> this should be recoverable.
>>
> 
> i've tried reading `hexedit` both on the closed and opened partition and
> everything seems to be random junk.  `strings` also can't find anything
> resembling LVM data.

i'm guessing silence means no one wants to give me the bad news :P


-- 
:D


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 585 bytes --]

Re: [dm-crypt] LVM on LUKS: volumes missing

[Robert Nichols]

On 06/02/2016 07:57 AM, fauno wrote:
> On 31/05/16 11:54, fauno wrote:
>> On 31/05/16 09:52, Robert Nichols wrote:
>>
>>> Take a look at the decrypted volume with "hexedit -s". It should start
>>> out with mostly binary zeros with a little data including the ASCII
>>> string "LVM2" at the start of a few of the sectors. Starting at about
>>> the 10th sector there should be a lot of ASCII text. (It's a copy of the
>>> corresponding file in /etc/lvm/backup, though without the fancy
>>> formatting.)
>>>
>>> If you're seeing random-appearing binary junk there, then the volume is
>>> not be decrypted properly. If it's just some of the early sectors that
>>> are clobbered and the text starting at the 10th sector is intact, then
>>> this should be recoverable.
>>>
>>
>> i've tried reading `hexedit` both on the closed and opened partition and
>> everything seems to be random junk.  `strings` also can't find anything
>> resembling LVM data.
>
> i'm guessing silence means no one wants to give me the bad news :P

By any chance did you reboot into a kernel different from the one that 
was running before? If so, try the old kernel. In the past, there was a 
kernel change that affected a non-default LUKS option (whirlpool hash). 
It's a long shot, but so is anything else at this point.

Let's look at about the only other thing that's correctable, and that 
would be a damaged partition table that has the partition starting in 
the wrong place. Does the output from "fdisk -l" for that drive look 
reasonable? If you look at the sectors with "hexedit -s", does the 
randomness start with the sector where that partition is supposed to begin?

-- 
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.

Re: [dm-crypt] LVM on LUKS: volumes missing

[Arno Wagner]

One thing is that these problems are pretty hard to debug.
Another is that LVM massively complicates things.

Now, if the LUKS container opens cleanly, anything
in it should be decrypted correctly (if it is LVM
atop of LUKS) and decryption with the wrong key is
not actually a possibility. 

That also means you should be able to use LVM
recovery techniques (I assume they exist) on this.

Unfortunately, I cannot help you with LVM as I do not use 
it. I consider it a badly engineered, overly complicated 
thing that decreases reliablity and makes problem 
diagnostics very hard.  Unfortunately, it provides some
increases in convenience and that seems to be all most people
care about, atleast until they are confronted with the hard
realities of reliable engineering, namely that KISS
is more important than any other concern. As LVM
violates that (along with the new RAID superblock
formats, udev, systemd and some other engineering 
abominations that have found their way into mainstream
Linux recently) I do not tolerate LVM on any of my
systems. 

Regards,
Arno


On Thu, Jun 02, 2016 at 14:57:57 CEST, fauno wrote:
> On 31/05/16 11:54, fauno wrote:
> > On 31/05/16 09:52, Robert Nichols wrote:
> > 
> >> Take a look at the decrypted volume with "hexedit -s". It should start
> >> out with mostly binary zeros with a little data including the ASCII
> >> string "LVM2" at the start of a few of the sectors. Starting at about
> >> the 10th sector there should be a lot of ASCII text. (It's a copy of the
> >> corresponding file in /etc/lvm/backup, though without the fancy
> >> formatting.)
> >>
> >> If you're seeing random-appearing binary junk there, then the volume is
> >> not be decrypted properly. If it's just some of the early sectors that
> >> are clobbered and the text starting at the 10th sector is intact, then
> >> this should be recoverable.
> >>
> > 
> > i've tried reading `hexedit` both on the closed and opened partition and
> > everything seems to be random junk.  `strings` also can't find anything
> > resembling LVM data.
> 
> i'm guessing silence means no one wants to give me the bad news :P
> 
> 
> -- 
> :D
> 




> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier

Re: [dm-crypt] LVM on LUKS: volumes missing

[Robert Nichols]

On 06/03/2016 04:42 PM, Arno Wagner wrote:
> One thing is that these problems are pretty hard to debug.
> Another is that LVM massively complicates things.
>
> Now, if the LUKS container opens cleanly, anything
> in it should be decrypted correctly (if it is LVM
> atop of LUKS) and decryption with the wrong key is
> not actually a possibility.
>
> That also means you should be able to use LVM
> recovery techniques (I assume they exist) on this.
>
> Unfortunately, I cannot help you with LVM as I do not use
> it. I consider it a badly engineered, overly complicated
> thing that decreases reliablity and makes problem
> diagnostics very hard.

If the ASCII strings "LABELONE" and "LVM2" cannot be seen in the first 
few sectors of the volume, then that volume is either overwritten or not 
being decrypted correctly.  LVM keeps quite a bit of easily recognized 
ASCII data in the volume header.

In this case the fragile link seems to be the LUKS detached header, as I 
believe there is nothing to associate that header with a device and 
precise starting point for the payload. Yes, there is a check that the 
master key was reconstructed correctly. Now the question is what, if 
anything, does this key decrypt.

-- 
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.

Re: [dm-crypt] LVM on LUKS: volumes missing

[Arno Wagner]

On Sat, Jun 04, 2016 at 00:46:53 CEST, Robert Nichols wrote:
> On 06/03/2016 04:42 PM, Arno Wagner wrote:
> >One thing is that these problems are pretty hard to debug.
> >Another is that LVM massively complicates things.
> >
> >Now, if the LUKS container opens cleanly, anything
> >in it should be decrypted correctly (if it is LVM
> >atop of LUKS) and decryption with the wrong key is
> >not actually a possibility.
> >
> >That also means you should be able to use LVM
> >recovery techniques (I assume they exist) on this.
> >
> >Unfortunately, I cannot help you with LVM as I do not use
> >it. I consider it a badly engineered, overly complicated
> >thing that decreases reliablity and makes problem
> >diagnostics very hard.
> 
> If the ASCII strings "LABELONE" and "LVM2" cannot be seen in the
> first few sectors of the volume, then that volume is either
> overwritten or not being decrypted correctly.  LVM keeps quite a bit
> of easily recognized ASCII data in the volume header.
> 
> In this case the fragile link seems to be the LUKS detached header,
> as I believe there is nothing to associate that header with a device
> and precise starting point for the payload. Yes, there is a check
> that the master key was reconstructed correctly. Now the question is
> what, if anything, does this key decrypt.

That is the one thing with a detached header: As the sector
number goes into the decryption, decryption must start at the
right place. If it does, it will becorrect with LUKS. If not,
"random" data should result with XTS mode, I agree.

Now, in theory it would be possible to try each possible offset 
from the start of the device (depends on how the partition
for the LUKS container was created), until some (later) part
of the decrypted data has some deviation from uniform 
distribution in byte-counts.
 
Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier

Re: [dm-crypt] LVM on LUKS: volumes missing

[fauno]

[-- Attachment #1.1: Type: text/plain, Size: 1328 bytes --]

On 04/06/16 05:06, Arno Wagner wrote:

>> If the ASCII strings "LABELONE" and "LVM2" cannot be seen in the
>> first few sectors of the volume, then that volume is either
>> overwritten or not being decrypted correctly.  LVM keeps quite a bit
>> of easily recognized ASCII data in the volume header.
>>
>> In this case the fragile link seems to be the LUKS detached header,
>> as I believe there is nothing to associate that header with a device
>> and precise starting point for the payload. Yes, there is a check
>> that the master key was reconstructed correctly. Now the question is
>> what, if anything, does this key decrypt.
> 
> That is the one thing with a detached header: As the sector
> number goes into the decryption, decryption must start at the
> right place. If it does, it will becorrect with LUKS. If not,
> "random" data should result with XTS mode, I agree.
> 
> Now, in theory it would be possible to try each possible offset 
> from the start of the device (depends on how the partition
> for the LUKS container was created), until some (later) part
> of the decrypted data has some deviation from uniform 
> distribution in byte-counts.

Hi! Thanks for all the feedback.  I ran out of time for recovering this,
but as soon as I can I'll get back with the results :)

-- 
:>


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 585 bytes --]

[dm-crypt] LVM on LUKS: volumes missing

[fauno]

[-- Attachment #1.1: Type: text/plain, Size: 671 bytes --]

Hi, maybe I'm too shocked but I couldn't find anything on this issue :)

I have a fully encrypted HD using the LVM on LUKS method from
ArchWiki[^0], with the LUKS header and key file on an external device.

Today I started having some disk failures (root remounted ro, xfs
partition giving errors), and after I decided to reboot to run fsck, I
can't find anything.

When the encrypted partition is opened, I don't see any errors, not even
on dmesg, but LVM can't find any volume.  They're just missing.

Is there anything I can do? Thanks!



[^0]:
https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS


-- 
.oÓ)


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 585 bytes --]