[dm-crypt] Device $dev is not a valid LUKS device.

[dm-crypt] Device $dev is not a valid LUKS device.

[Erik Logtenberg]

Hi all,

I've been using dm-crypt for quite some years now, and every now and
then I am very unpleasantly surprised by the message that my entire
volume can no longer be read:

Device /dev/disk/by-path/ip-foo.bar:mainstorage-lun-1 is not a valid
LUKS device.

In this case the iscsi server was rebooted while the volume was still
mounted. The volume is used for making backups and no backups were
currently made, so no actual writes got lost. Nevertheless after reboot
it no longer recognizes my luks volume.

I've had this kind of malice before and at that time was unable to fix
it. So in the end I gave up, created a new luks volume and started from
scratch.

However I hope that is not how every future power failure is going to
end up. Is there anything I can do to fix this volume or anything I can
do to prevent such a simple event from causing such catastrophic results?

Kind regards,

Erik.

Re: [dm-crypt] Device $dev is not a valid LUKS device.

[Arno Wagner]

Hi Erik,

first, you can make a header-backup, see the FAQ. Second, you can 
investigate what destroys the header. The same thing could
well destroy a filesystem, RAID-superblock, etc.
Or put differently: Something is very, very wrong in
your set-up. Loss of the LUKS header can only happen if
einther the user does it (directly or indirectly) or
somethign is broken. 

Arno

On Mon, Mar 04, 2013 at 01:58:31PM +0100, Erik Logtenberg wrote:
> Hi all,
> 
> I've been using dm-crypt for quite some years now, and every now and
> then I am very unpleasantly surprised by the message that my entire
> volume can no longer be read:
> 
> Device /dev/disk/by-path/ip-foo.bar:mainstorage-lun-1 is not a valid
> LUKS device.
> 
> In this case the iscsi server was rebooted while the volume was still
> mounted. The volume is used for making backups and no backups were
> currently made, so no actual writes got lost. Nevertheless after reboot
> it no longer recognizes my luks volume.
> 
> I've had this kind of malice before and at that time was unable to fix
> it. So in the end I gave up, created a new luks volume and started from
> scratch.
> 
> However I hope that is not how every future power failure is going to
> end up. Is there anything I can do to fix this volume or anything I can
> do to prevent such a simple event from causing such catastrophic results?
> 
> Kind regards,
> 
> Erik.
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] Device $dev is not a valid LUKS device.

[Erik Logtenberg]

Hi Arno,

Do you have some pointers on how I can investigate that?

In my previous setup, I had my luks device directly on top of my iscsi
device. That worked great in the beginning, but after a power failure on
the iscsi server, my volume was destroyed. Since the luks device was the
only contents on the iscsi device, there was not much
testing/investigating that I could do.

So I changed the setup: I created an LVM volume on my iscsi device, and
the luks device inside the LVM volume. That way, I thought, I would at
least know what part of the setup caused the malfunction. I mean, if the
iscsi device itself would become corrupted, I wouldn't be able to
activate the LVM volume anymore, but if dm-crypt were to make some
mistake, I would expect to still be able to activate the LVM volume, but
then not be able to read the luks volume.

Right? Well, it turns out I can still activate the LVM volume just fine,
so apparently there are no media errors / overwritten headers, at least
no overwritten LVM headers.
But then, after I activated the LVM volume, dm-crypt can't read the luks
volume. So apparently -only- the luks volume is damaged in some way.
So... how can I investigate this further?

Kind regards,

Erik.


On 03/04/2013 03:04 PM, Arno Wagner wrote:
> Hi Erik,
> 
> first, you can make a header-backup, see the FAQ. Second, you can 
> investigate what destroys the header. The same thing could
> well destroy a filesystem, RAID-superblock, etc.
> Or put differently: Something is very, very wrong in
> your set-up. Loss of the LUKS header can only happen if
> einther the user does it (directly or indirectly) or
> somethign is broken. 
> 
> Arno
> 
> On Mon, Mar 04, 2013 at 01:58:31PM +0100, Erik Logtenberg wrote:
>> Hi all,
>>
>> I've been using dm-crypt for quite some years now, and every now and
>> then I am very unpleasantly surprised by the message that my entire
>> volume can no longer be read:
>>
>> Device /dev/disk/by-path/ip-foo.bar:mainstorage-lun-1 is not a valid
>> LUKS device.
>>
>> In this case the iscsi server was rebooted while the volume was still
>> mounted. The volume is used for making backups and no backups were
>> currently made, so no actual writes got lost. Nevertheless after reboot
>> it no longer recognizes my luks volume.
>>
>> I've had this kind of malice before and at that time was unable to fix
>> it. So in the end I gave up, created a new luks volume and started from
>> scratch.
>>
>> However I hope that is not how every future power failure is going to
>> end up. Is there anything I can do to fix this volume or anything I can
>> do to prevent such a simple event from causing such catastrophic results?
>>
>> Kind regards,
>>
>> Erik.
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@saout.de
>> http://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] Device $dev is not a valid LUKS device.

[Erik Logtenberg]

Lol, whoops. I found out what was the cause of my problem. My mount
script wasn't functioning quite as it should, it tried to luksOpen the
iscsi device directly instead of the lvm volume. Argl.

Fixed my mount script, able to mount my backup volume after all. Nice.

Still, I think I'll make a backup of that header just to be sure, for
future use.

Kind regards,

Erik.


On 03/04/2013 03:18 PM, Erik Logtenberg wrote:
> Hi Arno,
> 
> Do you have some pointers on how I can investigate that?
> 
> In my previous setup, I had my luks device directly on top of my iscsi
> device. That worked great in the beginning, but after a power failure on
> the iscsi server, my volume was destroyed. Since the luks device was the
> only contents on the iscsi device, there was not much
> testing/investigating that I could do.
> 
> So I changed the setup: I created an LVM volume on my iscsi device, and
> the luks device inside the LVM volume. That way, I thought, I would at
> least know what part of the setup caused the malfunction. I mean, if the
> iscsi device itself would become corrupted, I wouldn't be able to
> activate the LVM volume anymore, but if dm-crypt were to make some
> mistake, I would expect to still be able to activate the LVM volume, but
> then not be able to read the luks volume.
> 
> Right? Well, it turns out I can still activate the LVM volume just fine,
> so apparently there are no media errors / overwritten headers, at least
> no overwritten LVM headers.
> But then, after I activated the LVM volume, dm-crypt can't read the luks
> volume. So apparently -only- the luks volume is damaged in some way.
> So... how can I investigate this further?
> 
> Kind regards,
> 
> Erik.
> 
> 
> On 03/04/2013 03:04 PM, Arno Wagner wrote:
>> Hi Erik,
>>
>> first, you can make a header-backup, see the FAQ. Second, you can 
>> investigate what destroys the header. The same thing could
>> well destroy a filesystem, RAID-superblock, etc.
>> Or put differently: Something is very, very wrong in
>> your set-up. Loss of the LUKS header can only happen if
>> einther the user does it (directly or indirectly) or
>> somethign is broken. 
>>
>> Arno
>>
>> On Mon, Mar 04, 2013 at 01:58:31PM +0100, Erik Logtenberg wrote:
>>> Hi all,
>>>
>>> I've been using dm-crypt for quite some years now, and every now and
>>> then I am very unpleasantly surprised by the message that my entire
>>> volume can no longer be read:
>>>
>>> Device /dev/disk/by-path/ip-foo.bar:mainstorage-lun-1 is not a valid
>>> LUKS device.
>>>
>>> In this case the iscsi server was rebooted while the volume was still
>>> mounted. The volume is used for making backups and no backups were
>>> currently made, so no actual writes got lost. Nevertheless after reboot
>>> it no longer recognizes my luks volume.
>>>
>>> I've had this kind of malice before and at that time was unable to fix
>>> it. So in the end I gave up, created a new luks volume and started from
>>> scratch.
>>>
>>> However I hope that is not how every future power failure is going to
>>> end up. Is there anything I can do to fix this volume or anything I can
>>> do to prevent such a simple event from causing such catastrophic results?
>>>
>>> Kind regards,
>>>
>>> Erik.
>>> _______________________________________________
>>> dm-crypt mailing list
>>> dm-crypt@saout.de
>>> http://www.saout.de/mailman/listinfo/dm-crypt
>>
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] Device $dev is not a valid LUKS device.

[Arno Wagner]

Hi Erik,

basically the LUKS header (see FAQ for brief explanation of the 
on-disk format) gets overwritten. The question is by what.
Often, looking at the data that is now there in place of the 
header guves a clue, sometimes not. So look at the start
of the (former) LUKS device.

On Mon, Mar 04, 2013 at 03:18:55PM +0100, Erik Logtenberg wrote:
> Hi Arno,
> 
> Do you have some pointers on how I can investigate that?
> 
> In my previous setup, I had my luks device directly on top of my iscsi
> device. That worked great in the beginning, but after a power failure on
> the iscsi server, my volume was destroyed. Since the luks device was the
> only contents on the iscsi device, there was not much
> testing/investigating that I could do.
> 
> So I changed the setup: I created an LVM volume on my iscsi device, and
> the luks device inside the LVM volume. That way, I thought, I would at
> least know what part of the setup caused the malfunction. I mean, if the
> iscsi device itself would become corrupted, I wouldn't be able to
> activate the LVM volume anymore, but if dm-crypt were to make some
> mistake, I would expect to still be able to activate the LVM volume, but
> then not be able to read the luks volume.
> 
> Right? Well, it turns out I can still activate the LVM volume just fine,
> so apparently there are no media errors / overwritten headers, at least
> no overwritten LVM headers.

I do not use LVM (complicates things which is bad IMO). I also failed
to find any description of its superblock or the superblock
location on the web, pretty bad. However, in an experiment,
pvcreate put the header at 1k offset, while the LUKS header is
at the beginning of the device. So whatever kills the LUKS header
may leave the LVM superblock undamaged. It is interesting, however,
that the LUKS header seems to have been damaged inside the LVM
container.  

> But then, after I activated the LVM volume, dm-crypt can't read the luks
> volume. So apparently -only- the luks volume is damaged in some way.
> So... how can I investigate this further?

Look at what is there instead of the LUKS header. Then find out 
what put it there. I sould also submit that until you find 
out, you shpuld not trust the iSCSI device, something is 
clearly wring with it.

Arno
 
> On 03/04/2013 03:04 PM, Arno Wagner wrote:
> > Hi Erik,
> > 
> > first, you can make a header-backup, see the FAQ. Second, you can 
> > investigate what destroys the header. The same thing could
> > well destroy a filesystem, RAID-superblock, etc.
> > Or put differently: Something is very, very wrong in
> > your set-up. Loss of the LUKS header can only happen if
> > einther the user does it (directly or indirectly) or
> > somethign is broken. 
> > 
> > Arno
> > 
> > On Mon, Mar 04, 2013 at 01:58:31PM +0100, Erik Logtenberg wrote:
> >> Hi all,
> >>
> >> I've been using dm-crypt for quite some years now, and every now and
> >> then I am very unpleasantly surprised by the message that my entire
> >> volume can no longer be read:
> >>
> >> Device /dev/disk/by-path/ip-foo.bar:mainstorage-lun-1 is not a valid
> >> LUKS device.
> >>
> >> In this case the iscsi server was rebooted while the volume was still
> >> mounted. The volume is used for making backups and no backups were
> >> currently made, so no actual writes got lost. Nevertheless after reboot
> >> it no longer recognizes my luks volume.
> >>
> >> I've had this kind of malice before and at that time was unable to fix
> >> it. So in the end I gave up, created a new luks volume and started from
> >> scratch.
> >>
> >> However I hope that is not how every future power failure is going to
> >> end up. Is there anything I can do to fix this volume or anything I can
> >> do to prevent such a simple event from causing such catastrophic results?
> >>
> >> Kind regards,
> >>
> >> Erik.
> >> _______________________________________________
> >> dm-crypt mailing list
> >> dm-crypt@saout.de
> >> http://www.saout.de/mailman/listinfo/dm-crypt
> > 
> 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell