[dm-crypt] dm-crypt on top of DRBD for live migration

[dm-crypt] dm-crypt on top of DRBD for live migration

[Berengar Lehr]

We want to use LVM, dm-crypt and DRBD in a 2-machine setup for KVM.

We think, a proper setup could be something like this (dm-crypt below DRBD):


   Machine 1               Machine 2

      KVM  -> -> -> -> -> ->  KVM
       |   (live migration)    .
       |                       .
      DRBD - - - - - - - - - DRBD
       |                       |
      LVM                     LVM
       |                       |
    dm-crypt                dm-crypt
       |                       |
 Disk/Partition          Disk/Partition

The KVM guest machines should run on machine 1. Live migration to
machine 2 should be supported.

Using this setup, every write to DRBD would be (independently) crypted
on both machines,
leading to additional (unnecessary?) cpu load on machine 2 before live
migrating, and additional
cpu load on machine 1 after live migration.

Could these additional cpu loads be avoided using a setup like this
(dm-crypt in top of DRBD):


   Machine 1               Machine 2

      KVM  -> -> -> -> -> ->  KVM
       |   (live migration)    .
       |                       .(b)
    dm-crypt                dm-crypt
       |                       |(a)
      DRBD - - - - - - - - - DRBD
       |                       |
      LVM                     LVM
       |                       |
 Disk/Partition          Disk/Partition

In this setup, dm-crypt runs on both machines, too, but is not used on
machine 2 until KVM
guests send write-requests after the live migration. So crypting is
done only by one machine
at every time point.

Is such a setup safe and stable?

What about caching at points (a) or (b) on machine 2?
Can KVM read cached, outdated data from dm-crypt after live migration?

Is there a workaround?

Thank You
B. Lehr & M. Müller

--
Mate ist gesunder Schlaf in Halbliterflaschen

Re: [dm-crypt] dm-crypt on top of DRBD for live migration

[Arno Wagner]

I think CPU-load should only be a secondary consideration. 

On the placement of dm-crypt, my intuition would be to place 
it as close to the storage as possible as it works on sector 
layer, and regard DRBD as sort-of part of the buffer-cache. 
I may be wrong. Milan is the person that should know.

An other aspect is that when placing encryption above BRDB, then
the traffic over the network is also encrypted. To me,it does not
make a lot of sense to encrypt data on disk, but put the same 
data unencrypted over a network, even if it is a physically 
separated network. So with dm-crypt below DRBD, you need to encrypt
the data 3 times: 1x local 1x transport and 1x remote, while
with dm-crypt above BRDB, you only need to encrypt once. 

Should the both placents be fine, then definitely place it 
above. Mirroring a dm-crypt or LUKS container should
not cause security problems in itself, two identical containers
with the same key(s) should in fact be a tiny bit more secure
than two containers with different keys and the same data.
(with tiny = does not matter in practice). And with link
encryption in addition, you need to trust 2 mechanisms,
namely the link encryption in addition. That does increase
the attack surface considerably.

On the migration issue, I do not understand the question.
What is your concern here?

Arno 



On Wed, Dec 07, 2011 at 01:30:37PM +0100, Berengar Lehr wrote:
> We want to use LVM, dm-crypt and DRBD in a 2-machine setup for KVM.
> 
> We think, a proper setup could be something like this (dm-crypt below DRBD):
> 
> 
> ?? ??Machine 1 ?? ?? ?? ?? ?? ?? ?? Machine 2
> 
> ?? ?? ?? KVM ??-> -> -> -> -> -> ??KVM
> ?? ?? ?? ??| ?? (live migration) ?? ??.
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? .
> ?? ?? ?? DRBD - - - - - - - - - DRBD
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |
> ?? ?? ?? LVM ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? LVM
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |
> ?? ?? dm-crypt ?? ?? ?? ?? ?? ?? ?? ??dm-crypt
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |
> ??Disk/Partition ?? ?? ?? ?? ??Disk/Partition
> 
> The KVM guest machines should run on machine 1. Live migration to
> machine 2 should be supported.
> 
> Using this setup, every write to DRBD would be (independently) crypted
> on both machines,
> leading to additional (unnecessary?) cpu load on machine 2 before live
> migrating, and additional
> cpu load on machine 1 after live migration.
> 
> Could these additional cpu loads be avoided using a setup like this
> (dm-crypt in top of DRBD):
> 
> 
> ?? ??Machine 1 ?? ?? ?? ?? ?? ?? ?? Machine 2
> 
> ?? ?? ?? KVM ??-> -> -> -> -> -> ??KVM
> ?? ?? ?? ??| ?? (live migration) ?? ??.
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? .(b)
> ?? ?? dm-crypt ?? ?? ?? ?? ?? ?? ?? ??dm-crypt
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |(a)
> ?? ?? ?? DRBD - - - - - - - - - DRBD
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |
> ?? ?? ?? LVM ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? LVM
> ?? ?? ?? ??| ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? |
> ??Disk/Partition ?? ?? ?? ?? ??Disk/Partition
> 
> In this setup, dm-crypt runs on both machines, too, but is not used on
> machine 2 until KVM
> guests send write-requests after the live migration. So crypting is
> done only by one machine
> at every time point.
> 
> Is such a setup safe and stable?
> 
> What about caching at points (a) or (b) on machine 2?
> Can KVM read cached, outdated data from dm-crypt after live migration?
> 
> Is there a workaround?
> 
> Thank You
> B. Lehr & M. M??ller
> 
> --
> Mate ist gesunder Schlaf in Halbliterflaschen
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt on top of DRBD for live migration

[Berengar Lehr]

> On the migration issue, I do not understand the question.
> What is your concern here?
> 
> Arno 

Hy Arno,

thank you for your thoughts. Our concern about live migration from one
machine to another is that using dm-crypt on two different machines
might produce two different disc values even if to master keys are used.
If both machines (they should be running in some kind of parallel mode
during live migration) execute the same write-command writing the same
data would result in the very same value on disc. Using DRBD would hence
not result in write/read problems.
We do not fully understand the way dm-crypt/LUKS works so we considered
the following situation:
Machine M1 is writing to decrypted disk DV1, dm-crypt is writing this
data encrypted to PV1. This is done by generating a random key (RK1)
used for encrypting the data D itself which is again encrypted by the
master key (MK) and written to a special location on the disc (probably
the first bytes of the sector).
Now M2 was parallel writing the same data D to that sector before, using
its own RK2. Now even if both disks have the same MK (which under any
circumstances should be the case if we would be using the second setup).
But due to the difference RK2 and RK1 both machines would write
different data to disk and hence might run into a problem when DRBD
synchronizes the data.
This was the scenario we thought about (not knowing if dm-crypt/LUKS is
using such randomized sector keys) but there might be other problems
using dm-crypt outside the VM but above DRBD. Hope that helps to
understand our considerations.

Thank You
B. Lehr & M. Müller

Re: [dm-crypt] dm-crypt on top of DRBD for live migration

[Arno Wagner]

Hi,

I see. If you place dm-crypt above DRBD, then you will have
the same master key on both machines, as the same LUKS header 
will be on both machines. There is nothing random here. You
will create the LUKS header on one machine, and just use it 
on the other one. 

As to the potential problems, you get the same from the
buffer-cache and filesystem. Hence a situation where M1 and M2
both write to disk already breaks a number of things, because
the filesystm and the buffer-cache both depend on having
exclusive access to the underlying block device. If you 
add plain dm-crypt or LUKS as encryption, it just has the 
same requirement. 

If you look at the picture on the DRBD homepage, then you see that
it is somewhat misleading. The mirroring can always be active,
but only one service layer, filesystem and buffer-cache can. 
And before you can migrate the service layer, you need to 
teardown the filesystem and buffer-cache on the first machine, 
by unmounting the device. Only then can you mount the device 
on the second machine. And only after that can you start the 
service layer on the second machine.

The exception here is if you have a proper cluster-filesystem.
These can have filesystem instances and buffer-cache instances
on several machines accessing the same block device. Here,
you must place encryption below DRBD on each machine, as
dm-crypt is not capable of supporting multiple copies of
itself accessing the same block device concurrently. It
may even work most of the time, but especially in LUKS 
header management could fail, potentially destroying the 
header. 

This (dm-crypt below DRBD) produces indeed two different
raw data sets on both machines, but DRBD only sees the
decrypted versions through dm-crypt and hence no problems
result.

Hope this clears matters up a bit.

Arno


On Mon, Dec 12, 2011 at 09:51:21AM +0100, Berengar Lehr wrote:
> > On the migration issue, I do not understand the question.
> > What is your concern here?
> > 
> > Arno 
> 
> Hy Arno,
> 
> thank you for your thoughts. Our concern about live migration from one
> machine to another is that using dm-crypt on two different machines
> might produce two different disc values even if to master keys are used.
> If both machines (they should be running in some kind of parallel mode
> during live migration) execute the same write-command writing the same
> data would result in the very same value on disc. Using DRBD would hence
> not result in write/read problems.
> We do not fully understand the way dm-crypt/LUKS works so we considered
> the following situation:
> Machine M1 is writing to decrypted disk DV1, dm-crypt is writing this
> data encrypted to PV1. This is done by generating a random key (RK1)
> used for encrypting the data D itself which is again encrypted by the
> master key (MK) and written to a special location on the disc (probably
> the first bytes of the sector).
> Now M2 was parallel writing the same data D to that sector before, using
> its own RK2. Now even if both disks have the same MK (which under any
> circumstances should be the case if we would be using the second setup).
> But due to the difference RK2 and RK1 both machines would write
> different data to disk and hence might run into a problem when DRBD
> synchronizes the data.
> This was the scenario we thought about (not knowing if dm-crypt/LUKS is
> using such randomized sector keys) but there might be other problems
> using dm-crypt outside the VM but above DRBD. Hope that helps to
> understand our considerations.
> 
> Thank You
> B. Lehr & M. M?ller
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] dm-crypt on top of DRBD for live migration

[Milan Broz]

Hi,

I think most of the problems were already discussed,
so just few comments here.

First, dm-crypt is not tested in cluster scenarios at all,
you have been warned :)

Dm-crypt is transparent block device encryption, it always
works on 512bytes sectors (no problem for 4k drives, but just
atomic encryption unit is always 512 bytes) and does not randomize
sectors - so mapping is always 1:1 for plaintext/ciphertext device
(but note LUKS has some header on-disk so there is offset,
plaintext device is smaller than underlying device.)

On 12/07/2011 01:30 PM, Berengar Lehr wrote:
> We want to use LVM, dm-crypt and DRBD in a 2-machine setup for KVM.
> 
> We think, a proper setup could be something like this (dm-crypt below DRBD):
> 
> 
>    Machine 1               Machine 2
> 
>       KVM  -> -> -> -> -> ->  KVM
>        |   (live migration)    .
>        |                       .
>       DRBD - - - - - - - - - DRBD
>        |                       |
>       LVM                     LVM
>        |                       |
>     dm-crypt                dm-crypt
>        |                       |
>  Disk/Partition          Disk/Partition
> 
> The KVM guest machines should run on machine 1. Live migration to
> machine 2 should be supported.

dm-crypt internally uses kernel threads (workqueues) which perform
encryption (and even IO submitting).

So without using a synchronization mechanism data hit disk
in different times for machine 1 and 2.

This is nothing new, IO-scheduler does the same, dm-crypt just add
to the mix more delay (encryption takes some time and
it can depend on load of machine).

It is filesystem responsibility (or application using top-level partition)
to issue FLUSH operation to ensure that data reached disk and DRBD
responsibility to distribute this to both machines.
Both LVM and dmcrypt support FLUSH (or formerly barrier) requests now.

> Using this setup, every write to DRBD would be (independently) crypted
> on both machines,
> leading to additional (unnecessary?) cpu load on machine 2 before live
> migrating, and additional
> cpu load on machine 1 after live migration.

I think CPU load is nothing you should worry about.
It is data consistency what matters here.

> Could these additional cpu loads be avoided using a setup like this
> (dm-crypt in top of DRBD):
> 
> 
>    Machine 1               Machine 2
> 
>       KVM  -> -> -> -> -> ->  KVM
>        |   (live migration)    .
>        |                       .(b)
>     dm-crypt                dm-crypt
>        |                       |(a)
>       DRBD - - - - - - - - - DRBD
>        |                       |
>       LVM                     LVM
>        |                       |
>  Disk/Partition          Disk/Partition
> 
> In this setup, dm-crypt runs on both machines, too, but is not used on
> machine 2 until KVM
> guests send write-requests after the live migration. So crypting is
> done only by one machine
> at every time point.
> 
> Is such a setup safe and stable?
> What about caching at points (a) or (b) on machine 2?
> Can KVM read cached, outdated data from dm-crypt after live migration?

Yes, in this scenario you should create dm-crypt mapping only on one
machine and "migrate" it together with KVM guest.

Or force flush all buffers before migration on the host where is
the guest migrating to (blockdev --flushbufs - but not tested, maybe
it need more force).

As said above, if application (KVM, fs) is properly using some synchronization,
it should work but it is not tested.


Also please note that despite you are exporting ciphertext device here
(IOW data is encrypted on the wire) it is NOT SECURE on network.

In short, attacker cannot decrypt data, but he can replace old content
of sectors (reply attack) etc.
(From the logic how disk encryption works - IV is always constant
for the sector while IV for network data stream block must be unique - nonce
and never repeats). Also there is no authentication/checksum on dm-crypt level
possible because it simply has no space where to store authentication
tag or checksum).

So you need another layer of encryption of the network data stream
(ipsec or something) if you need that.

Milan