[dm-crypt] LUKS backup headers for recovery

[dm-crypt] LUKS backup headers for recovery

[Two Spirit]

[-- Attachment #1: Type: text/plain, Size: 1447 bytes --]

Hello world,

I'm a heavy believer in the backup mantra "2 is 1 and 1 is none", and start
to feel comfortable when I have 3. Luckily I had backups to handle my
recent data loss with LUKS, but I had to suffer a long restore time as the
capacities get larger.

Are there backup headers/superblocks/metadata (whatever you call it) within
the LUKS container so that if the header
is somehow corrupt, I can utilize the backup copy from within the container
like file systems have. (I understand there is still a question
of pre-decryption / post decryption. Since these are usually long running
file servers, I've found lots of discussions about passphrase recovery
while the systems are still running and not luksClosed). I did google
around for LUKS recovery procedures, but there were lots of bad long
involved processes out there that didn't work or I couldn't get to work.

I now see the  luksHeaderBackup and luksHeaderRestore commands.(My excuse
is that I don't recall them when I first learned about cryptsetup many
years ago.) but it sounds like I have (or some sysadmin has) to make my own
backups of this information else if I don't, I'm screwed if I get
corruption in the LUKS header so it is almost a mandatory procedure --
something I think lot of people would also not have done.

Yes, I have seen a seasoned sysadmin run #rm -rf * from root on a
production server, so I could forsee someone doing something to mess up the
LUKS headers.

[-- Attachment #2: Type: text/html, Size: 1612 bytes --]

Re: [dm-crypt] LUKS backup headers for recovery

[Arno Wagner]

On Thu, Jul 19, 2012 at 11:36:53AM -0700, Two Spirit wrote:
> Hello world,
> 
> I'm a heavy believer in the backup mantra "2 is 1 and 1 is none", and start
> to feel comfortable when I have 3. Luckily I had backups to handle my
> recent data loss with LUKS, but I had to suffer a long restore time as the
> capacities get larger.
> 
> Are there backup headers/superblocks/metadata (whatever you call it) within
> the LUKS container so that if the header
> is somehow corrupt, I can utilize the backup copy from within the container
> like file systems have. 

Not at this time. It is a wish-list item for the future, but its
secuurity-implications are not quite clear at this time. For example,
it compromises the easy way to wipe the header that you have at
the moment. 

> (I understand there is still a question
> of pre-decryption / post decryption. 

Not really. The problem is more that the header is relatively
large and that the second copy decreases the anti-forensic properties.
Just copying th existing header may not be the right approach.

> Since these are usually long running
> file servers, I've found lots of discussions about passphrase recovery
> while the systems are still running and not luksClosed). I did google
> around for LUKS recovery procedures, but there were lots of bad long
> involved processes out there that didn't work or I couldn't get to work.
> 
> I now see the  luksHeaderBackup and luksHeaderRestore commands.(My excuse
> is that I don't recall them when I first learned about cryptsetup many
> years ago.) 

They are also explained in the FAQ, with a special warning 
right at the beginning. Reading documentation from time to
time can decrease your risk and effort.

> but it sounds like I have (or some sysadmin has) to make my own
> backups of this information else if I don't, I'm screwed if I get
> corruption in the LUKS header so it is almost a mandatory procedure --
> something I think lot of people would also not have done.

Not quite. First you have to decide about the security trade-off.
And then, if you want, you can make a header-backup. Or not.
 
> Yes, I have seen a seasoned sysadmin run #rm -rf * from root on a
> production server, so I could forsee someone doing something to 
> mess up the LUKS headers.

It happens, as the mailing-list archives show. But the protection
against that and other things is the data backup, not a header
backup. The header backup is something convenient that decreases
security and as such should not be there by default.

"Those that sacrifice security for convenience shall neither
have security nor convenience", so to say.

The thing is that the header backup does not replace a normal
backup you need anyways. And a header-backup to the same device
does not protect you against a disk failure. There really is no
good way to do this except to explain the issue to the user
and have them decide what they want to do.

Encryption is not something you can do without understanding 
what you do. And until you understand what you do, there will
occasionaly be things that would have been easy with the 
required knowlegde but are hard without it. But you did it
right, it just took a long time to restore, you did not 
permanently lose you data. 

Arno
-- 
Arno Wagner,    Dr. sc. techn., Dipl. Inform.,   Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell 

Re: [dm-crypt] LUKS backup headers for recovery

[Milan Broz]

On 07/19/2012 09:18 PM, Arno Wagner wrote:
> On Thu, Jul 19, 2012 at 11:36:53AM -0700, Two Spirit wrote:
>> I'm a heavy believer in the backup mantra "2 is 1 and 1 is none", and start
>> to feel comfortable when I have 3. Luckily I had backups to handle my
>> recent data loss with LUKS, but I had to suffer a long restore time as the
>> capacities get larger.

While it is simple from that mantra point of view, it complicates many other
things. People are resizing devices, if you place 2nd copy near the end of device,
you create many new problems.
I saw these problems in LVM, GPT an fake-raid metadata. If you have two versions,
which is more recent? Is it still valid if you completely remove one copy?
(Very common confusion - GPT reappears from backup copy.)
And you need atomic counter or timestamp (new LUKS format) and locking.

In enterprise environment you will need to use something better anyway,
(admin need to be able to provide you recovery passphrase), so you end with
a Key Escrow system. (https://fedorahosted.org/volume_key/ is the project based
on libcryptsetup for example).

Without this, you can either store backup of header, or you can print key
on paper and store it somewhere safely (see luksDump --dump-master-key)
with the same effect.

>> Since these are usually long running
>> file servers, I've found lots of discussions about passphrase recovery
>> while the systems are still running and not luksClosed). I did google
>> around for LUKS recovery procedures, but there were lots of bad long
>> involved processes out there that didn't work or I couldn't get to work.

And you are running file server without backup? Do you have backup of /etc?
So adding one 4M file (LUKS header with keyslots) into regular backup is easy.

People are usually highly stressed by situation (data gone and no backup).
And usually they lose ability to recover data not by initial mistake/error,
but by some stupid recovery action without real understanding what caused
the problem.

(And btw there is script in LUKS source recovering LUKS header from running dmcrypt
device, it is mentioned in FAQ... 

Ehm,... actually Arno forgot to update link, after git switch :)
It is here http://code.google.com/p/cryptsetup/source/browse/misc/luks-header-from-active

>> I now see the  luksHeaderBackup and luksHeaderRestore commands.(My excuse
>> is that I don't recall them when I first learned about cryptsetup many
>> years ago.) 

Because I added them later, exactly to simplify all these recovery problems.
You can even open device using backup file (without rewriting header).

>> Yes, I have seen a seasoned sysadmin run #rm -rf * from root on a
>> production server, so I could forsee someone doing something to 
>> mess up the LUKS headers.

There are two groups of people: one group run backups.
The second did not lost data yet :-)

Milan

Re: [dm-crypt] LUKS backup headers for recovery

[Arno Wagner]

On Thu, Jul 19, 2012 at 11:36:59PM +0200, Milan Broz wrote:
> On 07/19/2012 09:18 PM, Arno Wagner wrote:
> > On Thu, Jul 19, 2012 at 11:36:53AM -0700, Two Spirit wrote:
> >> I'm a heavy believer in the backup mantra "2 is 1 and 1 is none", and start
> >> to feel comfortable when I have 3. Luckily I had backups to handle my
> >> recent data loss with LUKS, but I had to suffer a long restore time as the
> >> capacities get larger.
> 
> While it is simple from that mantra point of view, it complicates many other
> things. People are resizing devices, if you place 2nd copy near the end of device,
> you create many new problems.
> I saw these problems in LVM, GPT an fake-raid metadata. If you have two versions,
> which is more recent? Is it still valid if you completely remove one copy?
> (Very common confusion - GPT reappears from backup copy.)
> And you need atomic counter or timestamp (new LUKS format) and locking.

Very true. A relatively simple implementation goes to 
a complex one with surprising behaviour. Not good. 
KISS is still the prime directive for any good engineer.

> In enterprise environment you will need to use something better anyway,
> (admin need to be able to provide you recovery passphrase), so you end with
> a Key Escrow system. (https://fedorahosted.org/volume_key/ is the project based
> on libcryptsetup for example).
>
> Without this, you can either store backup of header, or you can print key
> on paper and store it somewhere safely (see luksDump --dump-master-key)
> with the same effect.

And if you do that make sure to dump all the parameters as well.
cryptsetup is conservative about changing defaults, but it can
happen.
 
> >> Since these are usually long running
> >> file servers, I've found lots of discussions about passphrase recovery
> >> while the systems are still running and not luksClosed). I did google
> >> around for LUKS recovery procedures, but there were lots of bad long
> >> involved processes out there that didn't work or I couldn't get to work.
> 
> And you are running file server without backup? Do you have backup of /etc?
> So adding one 4M file (LUKS header with keyslots) into regular backup is easy.
> 
> People are usually highly stressed by situation (data gone and no backup).
> And usually they lose ability to recover data not by initial mistake/error,
> but by some stupid recovery action without real understanding what caused
> the problem.

Indeed. I have added a note for this at the start of the FAQ.
 
> (And btw there is script in LUKS source recovering LUKS header from running dmcrypt
> device, it is mentioned in FAQ... 
> 
> Ehm,... actually Arno forgot to update link, after git switch :)
> It is here http://code.google.com/p/cryptsetup/source/browse/misc/luks-header-from-active

Wups. Fixed.
 
> >> I now see the  luksHeaderBackup and luksHeaderRestore commands.(My excuse
> >> is that I don't recall them when I first learned about cryptsetup many
> >> years ago.) 
> 
> Because I added them later, exactly to simplify all these recovery problems.
> You can even open device using backup file (without rewriting header).

Which is a very good thing, as any write has a chance of doing more 
damage.

> >> Yes, I have seen a seasoned sysadmin run #rm -rf * from root on a
> >> production server, so I could forsee someone doing something to 
> >> mess up the LUKS headers.
> 
> There are two groups of people: one group run backups.
> The second did not lost data yet :-)

Hehehehe, yes. Makes me remember the first (and only) 
time I lost important data. The universe looks a bit 
different after that experience ;-)

Arno
-- 
Arno Wagner,    Dr. sc. techn., Dipl. Inform.,   Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
One of the painful things about our time is that those who feel certainty 
are stupid, and those with any imagination and understanding are filled 
with doubt and indecision. -- Bertrand Russell