[dm-crypt] filesystem conversion guidance needed

[dm-crypt] filesystem conversion guidance needed

[lxnf98mm]

I have a setup with 1 drive
The root, swap, and data partitions are encrypted
I want to add a second drive and convert all partitions to raid 1 encrypted
Can I do this conversion safely or do I need to start from scratch
If so I would appreciate a bit of procedure guidance
I have

Debian 6.0.5
kernel 3.4.4
cryptsetup 1.4.1

Thanks
Richard

Re: [dm-crypt] filesystem conversion guidance needed

[Arno Wagner]

While it is possible to do something like this in-place,
any mistake can permanentely destroy all your data. So an
absolute requirement is a complete backup (see FAQ for tips for 
secure backup).

Now, swap is usually not put on RAID, so you can just let 
it be as it is.

For the others, encryption is usually placed on top of
RAID, so here is what you can do:

1. Make degraded RAID1 partitions on the new drive
   (put in "missing" for the missing partitions.
2. Copy the data over.
3. Reboot using the new RAID devices for boot/root
4. Sync the old partitions into the RAID devices.

Notes:
- You need to make the new partitions exactly (!) the
  same size as the old ones aor smaller. Or you
  need to repartition the old drive between steps
  3. and 4. accordingly
- I recomend using partition type 0xfb end kernel-level
  autodetection. For this to work, you have to use
  MD superblock version 0.90.

Gr"usse,
Arno   


On Thu, Feb 28, 2013 at 08:37:21AM -0600, lxnf98mm@gmail.com wrote:
> I have a setup with 1 drive
> The root, swap, and data partitions are encrypted
> I want to add a second drive and convert all partitions to raid 1 encrypted
> Can I do this conversion safely or do I need to start from scratch
> If so I would appreciate a bit of procedure guidance
> I have
> 
> Debian 6.0.5
> kernel 3.4.4
> cryptsetup 1.4.1
> 
> Thanks
> Richard
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] filesystem conversion guidance needed

[lxnf98mm]

I have successfully used the procedure you describe several times to 
convert to raid but never an encrypted partition
When you say "2. Copy the data over." what am I copying
Normally I would

cd old_filesystem; tar cf - . |(cd new_filesystem; tar xf -)

But that would just copy unencrypted data
Would you explain a bit more

Richard

On Thu, 28 Feb 2013, Arno Wagner wrote:

> While it is possible to do something like this in-place,
> any mistake can permanentely destroy all your data. So an
> absolute requirement is a complete backup (see FAQ for tips for
> secure backup).
>
> Now, swap is usually not put on RAID, so you can just let
> it be as it is.
>
> For the others, encryption is usually placed on top of
> RAID, so here is what you can do:
>
> 1. Make degraded RAID1 partitions on the new drive
>   (put in "missing" for the missing partitions.
> 2. Copy the data over.
> 3. Reboot using the new RAID devices for boot/root
> 4. Sync the old partitions into the RAID devices.
>
> Notes:
> - You need to make the new partitions exactly (!) the
>  same size as the old ones aor smaller. Or you
>  need to repartition the old drive between steps
>  3. and 4. accordingly
> - I recomend using partition type 0xfb end kernel-level
>  autodetection. For this to work, you have to use
>  MD superblock version 0.90.
>
> Gr"usse,
> Arno
>
>
> On Thu, Feb 28, 2013 at 08:37:21AM -0600, lxnf98mm@gmail.com wrote:
>> I have a setup with 1 drive
>> The root, swap, and data partitions are encrypted
>> I want to add a second drive and convert all partitions to raid 1 encrypted
>> Can I do this conversion safely or do I need to start from scratch
>> If so I would appreciate a bit of procedure guidance
>> I have
>>
>> Debian 6.0.5
>> kernel 3.4.4
>> cryptsetup 1.4.1
>>
>> Thanks
>> Richard
>> _______________________________________________
>> dm-crypt mailing list
>> dm-crypt@saout.de
>> http://www.saout.de/mailman/listinfo/dm-crypt
>
>

-- 

Re: [dm-crypt] filesystem conversion guidance needed

[Arno Wagner]

On Thu, Feb 28, 2013 at 09:01:29AM -0600, lxnf98mm@gmail.com wrote:
> I have successfully used the procedure you describe several times to
> convert to raid but never an encrypted partition
> When you say "2. Copy the data over." what am I copying

I realize my explanation was a bit low on detailand ambiguous.

You still copy the files, not the encrypted device or the 
filesystem. Copying anything binarily could cause numerous 
problems that can be avoided by copying data on file-level.

The layering is like this

Filesystem
 |
Encryption
 |
RAID          <-- This layer gets inserted in the stack
 |
Raw partitions
 |
Raw disks

Hence the RAID does see the raw encrypted partitions
and mirrors them. 

> Normally I would
> 
> cd old_filesystem; tar cf - . |(cd new_filesystem; tar xf -)
> 
> But that would just copy unencrypted data
> Would you explain a bit more

Say you have / on /dev/mapper/root_part decrypted from
/dev/sda3 and your new disk is /dev/sdb. Yiour target md device is
/dev/md3 (avoid name collisions, you specify the number of the md
device and it is persistent). Then you would 
do somethign like below. You can do this from the running old 
system with a few extra things, see below:

1. Make a partition of smaller or exactly size in /dev/ sdb,
   e.g. /dev/sdb3, give it type fb if you want kernel-level
   auto-detection. I highly recommend doing this.
2. Make a new RAID1 on top of /dev/sdb3, using something
   like this
     mdadm --create -n 2 -l 1 --superblock0.90 /dev/md3 /dev/sdb3 missing
3. Make a new LUKS filesystem in top of md3 and map it to, 
   say /dev/mapper/new_root
4. Create filesystem on new_root and mount it.
5. Copy root over with tar, just as you describe.
   If this is from the live installation, do not forget 
   --one-file-system or you will get a full system copy 
   including a memory image from /proc on the new raid partition.
6. If this is with udev and you copy using the installed system,
   make sure to manually create console and null in /dev/ on
   new_root, as it will otherwise not boot:
      cd /dev/mapper/new_root/dev
      mknod -m 660 console c 5 1
      mknod -m 660 null c 1 3

Now you have a copy of your system. Try to boot it by
your usual mechanism, just with /dev/md3 instead of /dev/sda3
as root and with the new LUKS passphrase, unless you use the same
as before (not a security risk increase in this situation).

So far, you have niot changed anything on the old drive.
If you still do not have a full backuck, at the very least 
make one now.

7. From the new running system, change the type of the old
   partitions to fb for auto-detection.
8. Sync them into the new md partitons like this:
      mdadm --add /dev/md3 /dev/sda3
   Here, all your old data will be overwritten. If the new
   partition is smaller, you may also want to zero the old 
   one before this step, though killing the LUKS header 
   makes that redundant. 
9. boot again and make sure the raid device comes up in a non-degraded
   state.


Done. Repeat for data partition at your laisure.

Note that the scripts handling the encryption on boot
see as only difference that it is now /dev/md<x> instead
of /dev/sda<y>, that represents the encrypted partitions.

Also note that you can check the state of a raid device and
sync progress with a simple "cat /proc/mdstat". Before
step 8, /dev/md3 will just lost one disk as present, 
after step 8, it should list both.

Arno 
-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell

Re: [dm-crypt] filesystem conversion guidance needed

[lxnf98mm]

Thank you
Very clear

Richard

On Thu, 28 Feb 2013, Arno Wagner wrote:

> On Thu, Feb 28, 2013 at 09:01:29AM -0600, lxnf98mm@gmail.com wrote:
>> I have successfully used the procedure you describe several times to
>> convert to raid but never an encrypted partition
>> When you say "2. Copy the data over." what am I copying
>
> I realize my explanation was a bit low on detailand ambiguous.
>
> You still copy the files, not the encrypted device or the
> filesystem. Copying anything binarily could cause numerous
> problems that can be avoided by copying data on file-level.
>
> The layering is like this
>
> Filesystem
> |
> Encryption
> |
> RAID          <-- This layer gets inserted in the stack
> |
> Raw partitions
> |
> Raw disks
>
> Hence the RAID does see the raw encrypted partitions
> and mirrors them.
>
>> Normally I would
>>
>> cd old_filesystem; tar cf - . |(cd new_filesystem; tar xf -)
>>
>> But that would just copy unencrypted data
>> Would you explain a bit more
>
> Say you have / on /dev/mapper/root_part decrypted from
> /dev/sda3 and your new disk is /dev/sdb. Yiour target md device is
> /dev/md3 (avoid name collisions, you specify the number of the md
> device and it is persistent). Then you would
> do somethign like below. You can do this from the running old
> system with a few extra things, see below:
>
> 1. Make a partition of smaller or exactly size in /dev/ sdb,
>   e.g. /dev/sdb3, give it type fb if you want kernel-level
>   auto-detection. I highly recommend doing this.
> 2. Make a new RAID1 on top of /dev/sdb3, using something
>   like this
>     mdadm --create -n 2 -l 1 --superblock0.90 /dev/md3 /dev/sdb3 missing
> 3. Make a new LUKS filesystem in top of md3 and map it to,
>   say /dev/mapper/new_root
> 4. Create filesystem on new_root and mount it.
> 5. Copy root over with tar, just as you describe.
>   If this is from the live installation, do not forget
>   --one-file-system or you will get a full system copy
>   including a memory image from /proc on the new raid partition.
> 6. If this is with udev and you copy using the installed system,
>   make sure to manually create console and null in /dev/ on
>   new_root, as it will otherwise not boot:
>      cd /dev/mapper/new_root/dev
>      mknod -m 660 console c 5 1
>      mknod -m 660 null c 1 3
>
> Now you have a copy of your system. Try to boot it by
> your usual mechanism, just with /dev/md3 instead of /dev/sda3
> as root and with the new LUKS passphrase, unless you use the same
> as before (not a security risk increase in this situation).
>
> So far, you have niot changed anything on the old drive.
> If you still do not have a full backuck, at the very least
> make one now.
>
> 7. From the new running system, change the type of the old
>   partitions to fb for auto-detection.
> 8. Sync them into the new md partitons like this:
>      mdadm --add /dev/md3 /dev/sda3
>   Here, all your old data will be overwritten. If the new
>   partition is smaller, you may also want to zero the old
>   one before this step, though killing the LUKS header
>   makes that redundant.
> 9. boot again and make sure the raid device comes up in a non-degraded
>   state.
>
>
> Done. Repeat for data partition at your laisure.
>
> Note that the scripts handling the encryption on boot
> see as only difference that it is now /dev/md<x> instead
> of /dev/sda<y>, that represents the encrypted partitions.
>
> Also note that you can check the state of a raid device and
> sync progress with a simple "cat /proc/mdstat". Before
> step 8, /dev/md3 will just lost one disk as present,
> after step 8, it should list both.
>
> Arno
>

-- 

Re: [dm-crypt] filesystem conversion guidance needed

[lxnf98mm]

Dr. Wagner,

I received a new drive yesterday and followed your prescription
The patient survived and I am the proud owner of a new encrypted raid
This should go into the FAQ or at least a HowTo for Dummies

Thanks
Richard


On Thu, 28 Feb 2013, Arno Wagner wrote:

> On Thu, Feb 28, 2013 at 09:01:29AM -0600, lxnf98mm@gmail.com wrote:
>> I have successfully used the procedure you describe several times to
>> convert to raid but never an encrypted partition
>> When you say "2. Copy the data over." what am I copying
>
> I realize my explanation was a bit low on detailand ambiguous.
>
> You still copy the files, not the encrypted device or the
> filesystem. Copying anything binarily could cause numerous
> problems that can be avoided by copying data on file-level.
>
> The layering is like this
>
> Filesystem
> |
> Encryption
> |
> RAID          <-- This layer gets inserted in the stack
> |
> Raw partitions
> |
> Raw disks
>
> Hence the RAID does see the raw encrypted partitions
> and mirrors them.
>
>> Normally I would
>>
>> cd old_filesystem; tar cf - . |(cd new_filesystem; tar xf -)
>>
>> But that would just copy unencrypted data
>> Would you explain a bit more
>
> Say you have / on /dev/mapper/root_part decrypted from
> /dev/sda3 and your new disk is /dev/sdb. Yiour target md device is
> /dev/md3 (avoid name collisions, you specify the number of the md
> device and it is persistent). Then you would
> do somethign like below. You can do this from the running old
> system with a few extra things, see below:
>
> 1. Make a partition of smaller or exactly size in /dev/ sdb,
>   e.g. /dev/sdb3, give it type fb if you want kernel-level
>   auto-detection. I highly recommend doing this.
> 2. Make a new RAID1 on top of /dev/sdb3, using something
>   like this
>     mdadm --create -n 2 -l 1 --superblock0.90 /dev/md3 /dev/sdb3 missing
> 3. Make a new LUKS filesystem in top of md3 and map it to,
>   say /dev/mapper/new_root
> 4. Create filesystem on new_root and mount it.
> 5. Copy root over with tar, just as you describe.
>   If this is from the live installation, do not forget
>   --one-file-system or you will get a full system copy
>   including a memory image from /proc on the new raid partition.
> 6. If this is with udev and you copy using the installed system,
>   make sure to manually create console and null in /dev/ on
>   new_root, as it will otherwise not boot:
>      cd /dev/mapper/new_root/dev
>      mknod -m 660 console c 5 1
>      mknod -m 660 null c 1 3
>
> Now you have a copy of your system. Try to boot it by
> your usual mechanism, just with /dev/md3 instead of /dev/sda3
> as root and with the new LUKS passphrase, unless you use the same
> as before (not a security risk increase in this situation).
>
> So far, you have niot changed anything on the old drive.
> If you still do not have a full backuck, at the very least
> make one now.
>
> 7. From the new running system, change the type of the old
>   partitions to fb for auto-detection.
> 8. Sync them into the new md partitons like this:
>      mdadm --add /dev/md3 /dev/sda3
>   Here, all your old data will be overwritten. If the new
>   partition is smaller, you may also want to zero the old
>   one before this step, though killing the LUKS header
>   makes that redundant.
> 9. boot again and make sure the raid device comes up in a non-degraded
>   state.
>
>
> Done. Repeat for data partition at your laisure.
>
> Note that the scripts handling the encryption on boot
> see as only difference that it is now /dev/md<x> instead
> of /dev/sda<y>, that represents the encrypted partitions.
>
> Also note that you can check the state of a raid device and
> sync progress with a simple "cat /proc/mdstat". Before
> step 8, /dev/md3 will just lost one disk as present,
> after step 8, it should list both.
>
> Arno
>

-- 

Re: [dm-crypt] filesystem conversion guidance needed

[Arno Wagner]

Very good, and thank you for the feedback!

Putting this as another "micro HOWTO" into the FAQ
is a good idea, will do it as soon as I find the time.

Arno


On Wed, Mar 06, 2013 at 08:25:02AM -0600, lxnf98mm@gmail.com wrote:
> Dr. Wagner,
> 
> I received a new drive yesterday and followed your prescription
> The patient survived and I am the proud owner of a new encrypted raid
> This should go into the FAQ or at least a HowTo for Dummies
> 
> Thanks
> Richard
> 
> 
> On Thu, 28 Feb 2013, Arno Wagner wrote:
> 
> >On Thu, Feb 28, 2013 at 09:01:29AM -0600, lxnf98mm@gmail.com wrote:
> >>I have successfully used the procedure you describe several times to
> >>convert to raid but never an encrypted partition
> >>When you say "2. Copy the data over." what am I copying
> >
> >I realize my explanation was a bit low on detailand ambiguous.
> >
> >You still copy the files, not the encrypted device or the
> >filesystem. Copying anything binarily could cause numerous
> >problems that can be avoided by copying data on file-level.
> >
> >The layering is like this
> >
> >Filesystem
> >|
> >Encryption
> >|
> >RAID          <-- This layer gets inserted in the stack
> >|
> >Raw partitions
> >|
> >Raw disks
> >
> >Hence the RAID does see the raw encrypted partitions
> >and mirrors them.
> >
> >>Normally I would
> >>
> >>cd old_filesystem; tar cf - . |(cd new_filesystem; tar xf -)
> >>
> >>But that would just copy unencrypted data
> >>Would you explain a bit more
> >
> >Say you have / on /dev/mapper/root_part decrypted from
> >/dev/sda3 and your new disk is /dev/sdb. Yiour target md device is
> >/dev/md3 (avoid name collisions, you specify the number of the md
> >device and it is persistent). Then you would
> >do somethign like below. You can do this from the running old
> >system with a few extra things, see below:
> >
> >1. Make a partition of smaller or exactly size in /dev/ sdb,
> >  e.g. /dev/sdb3, give it type fb if you want kernel-level
> >  auto-detection. I highly recommend doing this.
> >2. Make a new RAID1 on top of /dev/sdb3, using something
> >  like this
> >    mdadm --create -n 2 -l 1 --superblock0.90 /dev/md3 /dev/sdb3 missing
> >3. Make a new LUKS filesystem in top of md3 and map it to,
> >  say /dev/mapper/new_root
> >4. Create filesystem on new_root and mount it.
> >5. Copy root over with tar, just as you describe.
> >  If this is from the live installation, do not forget
> >  --one-file-system or you will get a full system copy
> >  including a memory image from /proc on the new raid partition.
> >6. If this is with udev and you copy using the installed system,
> >  make sure to manually create console and null in /dev/ on
> >  new_root, as it will otherwise not boot:
> >     cd /dev/mapper/new_root/dev
> >     mknod -m 660 console c 5 1
> >     mknod -m 660 null c 1 3
> >
> >Now you have a copy of your system. Try to boot it by
> >your usual mechanism, just with /dev/md3 instead of /dev/sda3
> >as root and with the new LUKS passphrase, unless you use the same
> >as before (not a security risk increase in this situation).
> >
> >So far, you have niot changed anything on the old drive.
> >If you still do not have a full backuck, at the very least
> >make one now.
> >
> >7. From the new running system, change the type of the old
> >  partitions to fb for auto-detection.
> >8. Sync them into the new md partitons like this:
> >     mdadm --add /dev/md3 /dev/sda3
> >  Here, all your old data will be overwritten. If the new
> >  partition is smaller, you may also want to zero the old
> >  one before this step, though killing the LUKS header
> >  makes that redundant.
> >9. boot again and make sure the raid device comes up in a non-degraded
> >  state.
> >
> >
> >Done. Repeat for data partition at your laisure.
> >
> >Note that the scripts handling the encryption on boot
> >see as only difference that it is now /dev/md<x> instead
> >of /dev/sda<y>, that represents the encrypted partitions.
> >
> >Also note that you can check the state of a raid device and
> >sync progress with a simple "cat /proc/mdstat". Before
> >step 8, /dev/md3 will just lost one disk as present,
> >after step 8, it should list both.
> >
> >Arno
> >
> 
> -- 
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner,     Dr. sc. techn., Dipl. Inform.,    Email: arno@wagner.name
GnuPG: ID: CB5D9718  FP: 12D6 C03B 1B30 33BB 13CF  B774 E35C 5FA1 CB5D 9718
----
One of the painful things about our time is that those who feel certainty
are stupid, and those with any imagination and understanding are filled
with doubt and indecision. -- Bertrand Russell