[dm-crypt] dm-crypt alignment + ssd + raid

[dm-crypt] dm-crypt alignment + ssd + raid

[Philippe Cerfon]

Hi list.

I'd have a few questions about the alignment issues and about how to
generally set up dm-crypt.
I've read http://www.saout.de/pipermail/dm-crypt/2010-February/000584.html,
but it's still not fully clear to me how this alignment thingy works.

I have to scenarios:
A) Sofware RAID6, multiple partitions each to be encrypted. (Using the
kernel md for the software RAID)
B) Single SSD disk, multiple partitions each to be encrypted.


Regarding A:
1) What is generally the best way to do this? I mean how to stack the
different levels of md/RAID, [LVM] (if used at all), dm-crypt.
I'd say one has about this:
(physical disk[s]) --> (MD/RAID6) --> (dm-crypt) --> ([LVM] if used at
all) --> filesystem

Or is another order "better"?
I think having MD/RAID at the bottom makes sense (instead of dm-crypt
at the bottom), at this should make recovery easier, right?
I'm not sure whether I need LVM at all, but I think it makes only
sense to have it on top of dm-crypt in order to use to use it to
enlarge volumes.
I guess one cannot enlarge a LUKS "filesystem", even if the an
unterlying LVM volume would be enlarged?



2) I guess at any of the levels from above, one can partition the
exported block device, right?
So e.g. partition the physical disks that each has one big sdX1, and
create the RAID on it _OR_ create the RAID directly on the disk
withoug partitioning.

Is there any prefered schema, especially so that alignment works
better on any level?


I'd have it done the following way:

(physical disk[s])
---no_partitions--->
(MD/RAID6),... directly on the respective /dev/sdaX... not on any partitions
--- several partitions (e.g. one for boot, root, home, data and so on)
on the RAID device --->
(dm-crypt/LUKS),... one on any partition
---[not using LVM at all]--->
filesystems

Does this sound reasonable?


3) Now regarding the alignment. I use Debian sid, with 2.6.33.
Does this already work out of the box? I guess not.
As far as I understood, I must align each level, right?
How can I do this on each?

When I create the MD/RAID on a non partitioned device. How can I tell
mdadm to do the correct alignment.

And when I create several partitions on that RAID, will cryptsetup
automatically align LUKS?

And last but not least. How do I make the filesystem aligned within
the LUKS device?
Is this dependant from the filesystem type? I'd like to use btrfs and ext4.


4) I guess I have no 4K HDD but would the system automatically get
this done right, just in case I have one?




Regarding B:
5) Are there any special things I have to consider for the alignment
on an SSD or is it the same then above?
(I mean alginment is really important for SSD because of the wear...)

6) Milan wrote, that one needs to align to the 4MB boundary... how&why?



Thanks in advance. Hope that wasn't too off topic :)


Sincerely,
Philippe.

Re: [dm-crypt] dm-crypt alignment + ssd + raid

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 2638 bytes --]

Am 01.06.2010 16:34, schrieb Philippe Cerfon:
> I have to scenarios:
> A) Sofware RAID6, multiple partitions each to be encrypted. (Using the
> kernel md for the software RAID)
> B) Single SSD disk, multiple partitions each to be encrypted.
> 
> 
> Regarding A:
> 1) What is generally the best way to do this? I mean how to stack the
> different levels of md/RAID, [LVM] (if used at all), dm-crypt.
> I'd say one has about this:
> (physical disk[s]) --> (MD/RAID6) --> (dm-crypt) --> ([LVM] if used at
> all) --> filesystem
> 
> Or is another order "better"?
> I think having MD/RAID at the bottom makes sense (instead of dm-crypt
> at the bottom), at this should make recovery easier, right?
> I'm not sure whether I need LVM at all, but I think it makes only
> sense to have it on top of dm-crypt in order to use to use it to
> enlarge volumes.
> I guess one cannot enlarge a LUKS "filesystem", even if the an
> unterlying LVM volume would be enlarged?

I have used both scenarios in the past. The LUKS volume does not know
its payload size, so it will use the maximum space available.

In the scenario LVM->dm-crypt: Once you enlarge the underlying LV, you
can either 'cryptsetup resize' or 'cryptsetup luksClose && cryptsetup
luksOpen' for the volume to get the new size. The former does not even
require unmounting the file system.

The scenario dm-crypt->LVM is easier, as there is no extra layer between
the LV and filesystem.

Combined with a file system that can do online resizing (like ext3 on
newer kernels, or ext4), you can enlarge the file system transparently,
without any downtime. Shrinking is obviously more complicated.

These days, I use LVM on top of dm-crypt. However, a LUKS volume
encrypted with aes-xts-plain should not be bigger than 1TB for security
reasons (I read that here, don't know the exact reason), so this might
be unsuitable for your needs.

> 2) I guess at any of the levels from above, one can partition the
> exported block device, right?
> So e.g. partition the physical disks that each has one big sdX1, and
> create the RAID on it _OR_ create the RAID directly on the disk
> withoug partitioning.

I wouldn't rely on partitions, LVM is way more flexible. (Having /boot
on LVM might not work and require a small partition, depending on the
bootloader)


Sorry, but I am not familiar with the alignment questions you posted,
although I think with 2.6.33 and up-to-date userspace, all alignments
should be correct automatically without any specific interaction from
your end. Someone else will probably give a definitive answer on that.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [dm-crypt] dm-crypt alignment + ssd + raid

[Philippe Cerfon]

Hi Thomas.

Thanks for your answers :)

On Tue, Jun 1, 2010 at 5:00 PM, Thomas Bächler <thomas@archlinux.org> wrote:
> I have used both scenarios in the past. The LUKS volume does not know
> its payload size, so it will use the maximum space available.

Ah nice :-)


> In the scenario LVM->dm-crypt: Once you enlarge the underlying LV, you
> can either 'cryptsetup resize' or 'cryptsetup luksClose && cryptsetup
> luksOpen' for the volume to get the new size. The former does not even
> require unmounting the file system.

Even nicer.


> The scenario dm-crypt->LVM is easier, as there is no extra layer between
> the LV and filesystem.

Well,... I rethought the whole thing.
I still think that RAID should be at the bottom, but then we can have either:
a) disk-->RAID-->dm-crypt-->LVM-->fs
or
b) disk-->RAID-->LVM-->dm-crypt-->fs

(a) seems to be more naturally, as LVM is (as you've said) directly
below the fs,.. but... if I now add new disk because I want to enlarge
the fs,... I'll end up in using at least different master keys, as
dm-crypt is below LVM, right?
This would be avoided with (b) as far as I understand.


> These days, I use LVM on top of dm-crypt. However, a LUKS volume
> encrypted with aes-xts-plain should not be bigger than 1TB for security
> reasons (I read that here, don't know the exact reason), so this might
> be unsuitable for your needs.
Oh :-O strange..

Milan, do you have an idea why this is the case?


>> 2) I guess at any of the levels from above, one can partition the
>> exported block device, right?
>> So e.g. partition the physical disks that each has one big sdX1, and
>> create the RAID on it _OR_ create the RAID directly on the disk
>> withoug partitioning.
> I wouldn't rely on partitions, LVM is way more flexible.
So what is suggested now?

I start e.g. with /dev/sd[a-d],... putting the RAID/MD directly on
sd[a-d] or on sd[a-d]1?
(I ask because this might have an effect on the alignment thingy)

Then the raid gives me the "raid-device" /dev/md0. As I want several
LUKS volumes on my RAID (all with different keys) I could now either
partition md0, or set up LVM, right? Then on top of the
partitions/volumes dm-crypt,... on top of this my filesystems.

Milan,... perhaps you can (if you find some time),... help me with the
alignment thingy. AFAIK cryptsetup 1.1.1 would automatically support
this now with a kernel of at least 2.6.33, right?
But does it also work out of the box for RAID/MD, for LVM and for the
filesystems (as said I'd like to use ext4 and brtfs)? Or do I have to
use special command line switches?

And which userspace do I need to use? Does cfdisk work,... or do I
have to use fdisk -u? And with -u,... is it then automatically algined
or do I have to caculate something?

What about that 4MB boundary you've mentioned with SSDs?

Last but not least,... any other things I have to take care of?

Is there a way to verify, whether the alignment really fits in the end?

Once I understand it, I'll happliy try to contribute to the upcoming FAQ :)


> (Having /boot
> on LVM might not work and require a small partition, depending on the
> bootloader)
I have boot+bootloader anyway separately on usb-stick. ;-)



Thanks in advance,
Philippe.

Re: [dm-crypt] dm-crypt alignment + ssd + raid

[Philippe Cerfon]

Ah I forgot one thing:
I've noticed that discussion about TRIM and SSDs,... so I understand
that for now it is not supported.
But should I now fill an SSD disk with random data before using it or not?


Sincerely,
Philippe.

Re: [dm-crypt] dm-crypt alignment + ssd + raid

[Thomas Bächler]

[-- Attachment #1: Type: text/plain, Size: 2099 bytes --]

Am 04.06.2010 14:31, schrieb Philippe Cerfon:
>> The scenario dm-crypt->LVM is easier, as there is no extra layer between
>> the LV and filesystem.
> 
> Well,... I rethought the whole thing.
> I still think that RAID should be at the bottom, but then we can have either:
> a) disk-->RAID-->dm-crypt-->LVM-->fs
> or
> b) disk-->RAID-->LVM-->dm-crypt-->fs

Yes, both are supported.

> (a) seems to be more naturally, as LVM is (as you've said) directly
> below the fs,.. but... if I now add new disk because I want to enlarge
> the fs,... I'll end up in using at least different master keys, as
> dm-crypt is below LVM, right?
> This would be avoided with (b) as far as I understand.

That is correct. I use the dm-crypt->LVM setup on laptops, where a
second disk will never be added. If you expect the LVM to be split over
several VGs, then a per-logical-volume encryption seems more logical.

>>> 2) I guess at any of the levels from above, one can partition the
>>> exported block device, right?
>>> So e.g. partition the physical disks that each has one big sdX1, and
>>> create the RAID on it _OR_ create the RAID directly on the disk
>>> withoug partitioning.
>> I wouldn't rely on partitions, LVM is way more flexible.
> So what is suggested now?
> 
> I start e.g. with /dev/sd[a-d],... putting the RAID/MD directly on
> sd[a-d] or on sd[a-d]1?

If you boot from an external medium (as you say below), then I see no
need to even partition the drives. Linux handles sd[a-d] (without
partition table) just fine.

> (I ask because this might have an effect on the alignment thingy)
> 
> Then the raid gives me the "raid-device" /dev/md0. As I want several
> LUKS volumes on my RAID (all with different keys) I could now either
> partition md0, or set up LVM, right? Then on top of the
> partitions/volumes dm-crypt,... on top of this my filesystems.

Yes. I don't know how well partitioned md0 is supported. I would prefer
LVM in any case, because it is very flexible - in particular, it does
not rely on a specific physical layout of the volumes.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 262 bytes --]

Re: [dm-crypt] dm-crypt alignment + ssd + raid

[Mikko Rauhala]

pe, 2010-06-04 kello 14:31 +0200, Philippe Cerfon kirjoitti:
> Well,... I rethought the whole thing.
> I still think that RAID should be at the bottom, but then we can have either:
> a) disk-->RAID-->dm-crypt-->LVM-->fs
> or
> b) disk-->RAID-->LVM-->dm-crypt-->fs
> 
> (a) seems to be more naturally, as LVM is (as you've said) directly
> below the fs,.. but... if I now add new disk because I want to enlarge
> the fs,... I'll end up in using at least different master keys, as
> dm-crypt is below LVM, right?

Yes, if you add it as a new physical volume for the LVM, you'd indeed
have two separate dm-crypt devices to initialize.

However, for some needs and configurations, you could simply reshape the
underlying RAID to encompass the new media, and resize the existing
encrypted physical volume. The Linux software raid stack supports
reshaping of at least raid-5 and 6 configurations, IIRC (as well as
raid-5 to raid-6 reshaping).

> I start e.g. with /dev/sd[a-d],... putting the RAID/MD directly on
> sd[a-d] or on sd[a-d]1?

While you can use unpartitioned disks as Thomas said, I'd rather make
them single partition in anticipation of some random tool or another
getting confused about it. (Also, I'm not quite sure if the usual raid
autoconfiguration works without type fd partitions, but it may be...)

> I've noticed that discussion about TRIM and SSDs,... so I understand
> that for now it is not supported.

Correct, though there was that link on this list earlier to a patched
wiper.sh script that you can, if you wish, run sometimes to clear away
the unused areas. "Worked for me."

> But should I now fill an SSD disk with random data before using it or
> not?

If you don't want people to know how much of the drive you have or have
had in use, you should do this. (Note though that conceivably some such
information _could_ be deduced by a resourceful attacker from the
internal metadata of the SSD regardless, depending on what they store -
being black boxes as far as we're concerned and all.)

If you're less worried about that and wish your SSD to perform well
while maximizing its lifetime (not that they're overly prone to fail
quickly these days anyway), filling might be skipped, and trimming used
(for now, only semi-automatically through said script).

-- 
Mikko Rauhala <mjrauhal@cc.helsinki.fi>
University of Helsinki