[dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[.. ink ..]

I just added an ability to create an authenticated luks2 device in
zuluCrypt[1] and i am
wondering why these volumes need to be wiped when created. I made it work by
looking at how cryptsetup does it but i don't understand why because i
have so far
failed to find any documentation about it.

[1] https://github.com/mhogomchungu/zuluCrypt/issues/134

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Milan Broz]

On 25/09/2019 21:40, .. ink .. wrote:
> I just added an ability to create an authenticated luks2 device in
> zuluCrypt[1] and i am
> wondering why these volumes need to be wiped when created. I made it work by
> looking at how cryptsetup does it but i don't understand why because i
> have so far
> failed to find any documentation about it.

I think it is explained in the referenced paper, we should add a FAQ about it.

Initial wipe recalculates integrity tags - so you can read the device afterward.

If you skip initialization (wipe), integrity tags for all sectors is incorrect
and read will return integrity failure (EILSEQ errno).

In theory, it is not a problem ("do not read what you did not write").

But it reality it cases many programs to fail because it can access device
through page cache. If the *write* is not aligned to a page, page cache tries
to first read content, then update content, and write it back to the device.

But as said above, all read fails because integrity tags are not initialized,
thus even page-unaligned writes can fail.
(I have seen this problem even in programs like mkfs, where it is apparent bug.)

Milan


> 
> [1] https://github.com/mhogomchungu/zuluCrypt/issues/134
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> https://www.saout.de/mailman/listinfo/dm-crypt
> 

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Christoph Anton Mitterer]

On Thu, 2019-09-26 at 09:41 +0200, Milan Broz wrote:
> Initial wipe recalculates integrity tags - so you can read the device
> afterward.

Maybe it shouldn't be called "wipe" (which people often relate to
secure deletion methods) but "initialise"? :-)

Cheers,
Chris

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Robert Nichols]

On 9/26/19 2:41 AM, Milan Broz wrote:
> On 25/09/2019 21:40, .. ink .. wrote:
>> I just added an ability to create an authenticated luks2 device in
>> zuluCrypt[1] and i am
>> wondering why these volumes need to be wiped when created. I made it work by
>> looking at how cryptsetup does it but i don't understand why because i
>> have so far
>> failed to find any documentation about it.
> 
> I think it is explained in the referenced paper, we should add a FAQ about it.
> 
> Initial wipe recalculates integrity tags - so you can read the device afterward.
> 
> If you skip initialization (wipe), integrity tags for all sectors is incorrect
> and read will return integrity failure (EILSEQ errno).
> 
> In theory, it is not a problem ("do not read what you did not write").
> 
> But it reality it cases many programs to fail because it can access device
> through page cache. If the *write* is not aligned to a page, page cache tries
> to first read content, then update content, and write it back to the device.
> 
> But as said above, all read fails because integrity tags are not initialized,
> thus even page-unaligned writes can fail.
> (I have seen this problem even in programs like mkfs, where it is apparent bug.)

So, LUKS2 is incompatible with LVM thin provisioning or other sparse storage
formats like QCOW2. Good to know.

-- 
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Ondrej Kozina]

On 9/26/19 3:38 PM, Robert Nichols wrote:
> 
> On 9/26/19 2:41 AM, Milan Broz wrote:
>> On 25/09/2019 21:40, .. ink .. wrote:
>>> I just added an ability to create an authenticated luks2 device in
>>> zuluCrypt[1] and i am
>>> wondering why these volumes need to be wiped when created. I made it work by
>>> looking at how cryptsetup does it but i don't understand why because i
>>> have so far
>>> failed to find any documentation about it.
>>
>> I think it is explained in the referenced paper, we should add a FAQ about it.
>>
>> Initial wipe recalculates integrity tags - so you can read the device afterward.
>>
>> If you skip initialization (wipe), integrity tags for all sectors is incorrect
>> and read will return integrity failure (EILSEQ errno).
>>
>> In theory, it is not a problem ("do not read what you did not write").
>>
>> But it reality it cases many programs to fail because it can access device
>> through page cache. If the *write* is not aligned to a page, page cache tries
>> to first read content, then update content, and write it back to the device.
>>
>> But as said above, all read fails because integrity tags are not initialized,
>> thus even page-unaligned writes can fail.
>> (I have seen this problem even in programs like mkfs, where it is apparent bug.)
> 
> So, LUKS2 is incompatible with LVM thin provisioning or other sparse storage
> formats like QCOW2. Good to know.
> 

That is not correct.

First, LUKS2 does not enforce use of authenticated encryption in any 
way. It's optional choice and definitely not default (it's still 
considered experimental feature, unlike LUKS2 format that is new 
upstream default).

Contrary, if you prefer discards (trim) pass-through via dm-crypt you 
may use --persistent --allow-discards option so that it's automatically 
enabled during every following LUKS2 device activation. So it's more 
thin provisioning friendly if that is important for you (for others it 
may not be so).

Yes, authenticated encryption with dm-integrity in interleave mode does 
require device initialization for reason Milan described and moreover 
dm-integrity does not allow discards pass-through so thin provisioning 
does not make sense with it because you won't be able to reclaim free 
space from it.

O.

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Robert Nichols]

On 9/26/19 9:29 AM, Ondrej Kozina wrote:
> On 9/26/19 3:38 PM, Robert Nichols wrote:
>>
>> On 9/26/19 2:41 AM, Milan Broz wrote:
>>> On 25/09/2019 21:40, .. ink .. wrote:
>>>> I just added an ability to create an authenticated luks2 device in
>>>> zuluCrypt[1] and i am
>>>> wondering why these volumes need to be wiped when created. I made it work by
>>>> looking at how cryptsetup does it but i don't understand why because i
>>>> have so far
>>>> failed to find any documentation about it.
>>>
>>> I think it is explained in the referenced paper, we should add a FAQ about it.
>>>
>>> Initial wipe recalculates integrity tags - so you can read the device afterward.
>>>
>>> If you skip initialization (wipe), integrity tags for all sectors is incorrect
>>> and read will return integrity failure (EILSEQ errno).
>>>
>>> In theory, it is not a problem ("do not read what you did not write").
>>>
>>> But it reality it cases many programs to fail because it can access device
>>> through page cache. If the *write* is not aligned to a page, page cache tries
>>> to first read content, then update content, and write it back to the device.
>>>
>>> But as said above, all read fails because integrity tags are not initialized,
>>> thus even page-unaligned writes can fail.
>>> (I have seen this problem even in programs like mkfs, where it is apparent bug.)
>>
>> So, LUKS2 is incompatible with LVM thin provisioning or other sparse storage
>> formats like QCOW2. Good to know.
>>
> 
> That is not correct.
> 
> First, LUKS2 does not enforce use of authenticated encryption in any way. It's optional choice and definitely not default (it's still considered experimental feature, unlike LUKS2 format that is new upstream default).
> 
> Contrary, if you prefer discards (trim) pass-through via dm-crypt you may use --persistent --allow-discards option so that it's automatically enabled during every following LUKS2 device activation. So it's more thin provisioning friendly if that is important for you (for others it may not be so).
> 
> Yes, authenticated encryption with dm-integrity in interleave mode does require device initialization for reason Milan described and moreover dm-integrity does not allow discards pass-through so thin provisioning does not make sense with it because you won't be able to reclaim free space from it.

I wasn't even thinking of trim pass-through. I routinely set up VMs with space allocations that, in total, exceed the space available on the host. Such VMs cannot use authenticated encryption since they would have to immediately use all of their allocated space. I also routinely save system images with partclone or clonezilla, specifically to avoid saving the content of the free space in the filesystems. Such an image would fail authentication when restored. These are fairly common practices. LUKS2 authenticated encryption should come with huge warnings.

-- 
Bob Nichols     "NOSPAM" is really part of my email address.
                 Do NOT delete it.

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Arno Wagner]

On Thu, Sep 26, 2019 at 23:27:27 CEST, Robert Nichols wrote:
> I wasn't even thinking of trim pass-through.  I routinely set up VMs with
> space allocations that, in total, exceed the space available on the host. 
> Such VMs cannot use authenticated encryption since they would have to
> immediately use all of their allocated space.  I also routinely save
> system images with partclone or clonezilla, specifically to avoid saving
> the content of the free space in the filesystems.  Such an image would
> fail authentication when restored.  These are fairly common practices. 
> LUKS2 authenticated encryption should come with huge warnings.

I think a regular explanation what authenticated encryption
does (to those that do not know it) should be entirely enough.
The behavior is not in any way surprising, after all.
Add to that that it is not the default and there is no
need for any "huge warnings". People that do not knwo what 
they are doing will shoot themselves in the foot anyways.

Regards,
Arno
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Milan Broz]

On 26/09/2019 23:27, Robert Nichols wrote:
> I wasn't even thinking of trim pass-through. I routinely set up VMs
> with space allocations that, in total, exceed the space available on
> the host. Such VMs cannot use authenticated encryption since they
> would have to immediately use all of their allocated space. I also
> routinely save system images with partclone or clonezilla,
> specifically to avoid saving the content of the free space in the
> filesystems. Such an image would fail authentication when restored.
> These are fairly common practices. LUKS2 authenticated encryption
> should come with huge warnings.

I do not quite well understand your complaint.

LUKS2 is just on-disk format, it is much more flexible, and allows some
new features like integrity protection, but nobody forces you to use that.
Default options are precisely the same as in LUKS1, and you can still
use LUKS1 if you prefer it.

Actually, even the authenticated encryption does not need to wipe
the space in principle. It is a workaround to avoid bugs (to avoid
reading areas that were never written). You can skip it if your
environment is doing IO operations correctly.
TRIM passthrough is in some situations complicated, so it is not yet
supported for authenticated encryption (dm-integrity backed devices).
It is just missing a feature, that can be implemented later.

A full device wipe is present in many other systems - VeraCrypt by default does it;
Debian installer wipes LUKS devices by writing random data to it.
And these systems use just length preserving encryption.
A common practice is to TRIM the device to free unused space.

We are missing better documentation of common use cases and limitations.
I am well aware of that, but sometimes things take longer than expected,
sorry for that.

Milan

Re: [dm-crypt] Why is it necessary to "wipe" an authenticated luks2 device when creating it?

[Arno Wagner]

On Thu, Sep 26, 2019 at 09:41:39 CEST, Milan Broz wrote:
> On 25/09/2019 21:40, .. ink .. wrote:
> > I just added an ability to create an authenticated luks2 device in
> > zuluCrypt[1] and i am
> > wondering why these volumes need to be wiped when created. I made it work by
> > looking at how cryptsetup does it but i don't understand why because i
> > have so far
> > failed to find any documentation about it.
> 
> I think it is explained in the referenced paper, we should add a FAQ about
> it.
> 
> Initial wipe recalculates integrity tags - so you can read the device afterward.
> 
> If you skip initialization (wipe), integrity tags for all sectors is
> incorrect and read will return integrity failure (EILSEQ errno).
> 
> In theory, it is not a problem ("do not read what you did not write").
> 
> But it reality it cases many programs to fail because it can access device
> through page cache. If the *write* is not aligned to a page, page cache tries
> to first read content, then update content, and write it back to the device.
> 
> But as said above, all read fails because integrity tags are not
> initialized, thus even page-unaligned writes can fail.  (I have seen this
> problem even in programs like mkfs, where it is apparent bug.)

This is a specific problem with anything authenticated: Even non-data
needs to be authenticated, because there is no way to distinguish
it from data on that level. Hence expecting to have to do a full
"initialization" pass at the start on authenticated storage is
perfectly reasonable.

I can add an FAQ secion on authenticated encryption fpr this.
Do you have a link to the reference paper?

Regards,
Arno

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
A good decision is based on knowledge and not on numbers. -- Plato

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier