osd dm-crypt key management, part... deux?

osd dm-crypt key management, part... deux?

[Sage Weil]

We've had several partial starts to address this problem but haven't 
gotten anything over the line.  A quick summary:

1- Currently we store dm-crypt keys in /etc/ceph/dmcrypt-keys/$osd_uuid, 
on the boot disk.  This lets you throw away OSD disk but not boot disks 
and doesn't help you if someone walks away with a whole server.

2- SUSE had a pull request that made ceph-disk push/pull keys over (s)ftp.  
I can't find it now.. did it get closed?

3- We built a generic "config-key" storage service into the monitor that 
lets you restrict access to certain key/value pairs based on your 
(cephx) authentication key, intending that the osd dm-crypt keys be stored 
there.

4- Red Hat had a DEO project that addressed network-bound encryption in 
the works but that project has been shelved in favor of something 
newer/shinier/better.

5- Lars observes that any solution is incomplete if we don't *also* 
encrypt the swap devices.  (I think we don't need to worry about the boot 
disks, though, unless we're paranoid about /var/log/ceph?)

6- Radoslaw and Adam observe that anything using dm-crypt may have higher 
overhead than something implemented in RADOS itself.  But this is vastly 
more complicated and, if we move in this direction, a long way off.

7- In the meantime, we have nothing better than /etc/ceph/dmcrypt-keys 
upstream...


I suggest we do something simple:

1- Update SUSE's ceph-disk changes to make it easy to plug in 
different key management strategies.

2- Implement a simple mon-based strategy upstream.  We've discussed this a 
fair bit in the past, and were getting stuck on the problem of where to 
store the key-fetching-key.  I.e., we want a key on the disk that you use 
to ask the monitor for the LUKS key, which you then provide to LUKS to 
unlock the actual encryption key.  This means that we need a unencrypted 
spot on the device to store it in.  Milan has indicated that putting it in 
a LUKS key slot would be a bad idea and difficult to maintain.  Instead, I 
propose we create a new GPT partition type called OSD_LOCKBOX (or 
similar), with a tiny filesystem and a few files indicating what to do.  
This will make it easy to store the info we need for the mon scheme, and 
to support new key management approaches later (we can put whatever we 
want there as long as it's not too big).

I put some notes here:

	http://pad.ceph.com/p/osd-key-management

Thoughts?
sage

Re: osd dm-crypt key management, part... deux?

[Radoslaw Zarzynski]

On Wed, Jan 27, 2016 at 3:26 PM, Sage Weil <sweil@redhat.com> wrote:

> 2- Implement a simple mon-based strategy upstream.  We've discussed this a
> fair bit in the past, and were getting stuck on the problem of where to
> store the key-fetching-key.  I.e., we want a key on the disk that you use
> to ask the monitor for the LUKS key, which you then provide to LUKS to
> unlock the actual encryption key.  This means that we need a unencrypted
> spot on the device to store it in.  Milan has indicated that putting it in
> a LUKS key slot would be a bad idea and difficult to maintain.  Instead, I
> propose we create a new GPT partition type called OSD_LOCKBOX (or
> similar), with a tiny filesystem and a few files indicating what to do.
> This will make it easy to store the info we need for the mon scheme, and
> to support new key management approaches later (we can put whatever we
> want there as long as it's not too big).
>
> I put some notes here:
>
>         http://pad.ceph.com/p/osd-key-management
>
> Thoughts?
> sage

Hello Sage,

why we cannot simply fetch the LUKS key from a monitor during
the mount phase? LUKS key wouldn't be stored in any file. Instead,
it would be fetched to memory (RAM) and present there only for
time necessary to initialize dm-crypt.

Of course, code responsible for the setup must have an access to
the OSD's keyring. It may be put into lockbox.

Regards,
Radek

Re: osd dm-crypt key management, part... deux?

[Sage Weil]

On Wed, 27 Jan 2016, Radoslaw Zarzynski wrote:
> On Wed, Jan 27, 2016 at 3:26 PM, Sage Weil <sweil@redhat.com> wrote:
> 
> > 2- Implement a simple mon-based strategy upstream.  We've discussed this a
> > fair bit in the past, and were getting stuck on the problem of where to
> > store the key-fetching-key.  I.e., we want a key on the disk that you use
> > to ask the monitor for the LUKS key, which you then provide to LUKS to
> > unlock the actual encryption key.  This means that we need a unencrypted
> > spot on the device to store it in.  Milan has indicated that putting it in
> > a LUKS key slot would be a bad idea and difficult to maintain.  Instead, I
> > propose we create a new GPT partition type called OSD_LOCKBOX (or
> > similar), with a tiny filesystem and a few files indicating what to do.
> > This will make it easy to store the info we need for the mon scheme, and
> > to support new key management approaches later (we can put whatever we
> > want there as long as it's not too big).
> >
> > I put some notes here:
> >
> >         http://pad.ceph.com/p/osd-key-management
> >
> > Thoughts?
> > sage
> 
> Hello Sage,
> 
> why we cannot simply fetch the LUKS key from a monitor during
> the mount phase? LUKS key wouldn't be stored in any file. Instead,
> it would be fetched to memory (RAM) and present there only for
> time necessary to initialize dm-crypt.

Right, that's basically what I'm suggesting, except with the additional 
requirement that you need the lockbox ceph auth key in order to fetch the 
LUKS key (so that this host can only fetch keys to decrypt its own disks).

> Of course, code responsible for the setup must have an access to
> the OSD's keyring. It may be put into lockbox.

The OSD's key would be in the osd_data, encrypted.  So,

 lockbox key lets us fetch luks key
 luks key lets us set up dmcrypt (by unlocking the real encryption key)
 osd key lets the osd start up and join the cluster.

sage

Re: osd dm-crypt key management, part... deux?

[Milan Broz]

Hi,

just few notes, I know that too late, but anyway...

(Partially based on meeting with Deo author yesterday...)

On 01/27/2016 03:26 PM, Sage Weil wrote:
> We've had several partial starts to address this problem but haven't 
> gotten anything over the line.  A quick summary:
> 
> 1- Currently we store dm-crypt keys in /etc/ceph/dmcrypt-keys/$osd_uuid, 
> on the boot disk.  This lets you throw away OSD disk but not boot disks 
> and doesn't help you if someone walks away with a whole server.
> 
> 2- SUSE had a pull request that made ceph-disk push/pull keys over (s)ftp.  
> I can't find it now.. did it get closed?
> 
> 3- We built a generic "config-key" storage service into the monitor that 
> lets you restrict access to certain key/value pairs based on your 
> (cephx) authentication key, intending that the osd dm-crypt keys be stored 
> there.

My (upstream cryptsetup) plan is to have LUKS2 configurable keyslot and these
should allow plugins that can directly fetch the key for example.

In fact proof-of-concept code was inspired by Ceph OSD use case and
allows fetching key from remote server through libssh (it expects
ssh keys configured such way, that you can run ssh to some server
with public key auth without password - but that's just example).

In LUKS2 header then we store just ssh server/user/dir
https://gitlab.com/cryptsetup/cryptsetup/blob/wip-luks2/misc/luks2_keyslot_example/keyslot_test.c
(please note it is just wip example of standalone binary, interface will change,
and currently it uses real volume key, not unlocking passphrase)

I can imagine that in long-term there can be simple keyslot using
cephx auth as well or whatever Ceph will need.

Integration then would be just to properly format
that keyslot, all luks commands will then work as today automagically.

(There are priority for keyslot, so we can have fallback to other
kesylot type like password-based if network is down.)

The real problem is that LUKS2 can be stable in next months, not earlier.

I have some notes in Devconf presentation but that's relay just overview
of current experiment with LUKS2 (it can still be completely abandoned :)
https://mbroz.fedorapeople.org/talks/DevConf2016/devconf2016-luks2.pdf

> 
> 4- Red Hat had a DEO project that addressed network-bound encryption in 
> the works but that project has been shelved in favor of something 
> newer/shinier/better.

Yes, Deo (formerly Petera) is dead, replaced with Clevis/Tang project.

For more info see Devconf presentation
http://slides.com/npmccallum/devconf16#/ (there is also youtube recording)

This approach is usable for you as well but it requires another
configuration and dependencies. It will probably use on-disk LUKS2 data
in long-term, just with intermediate implementation using LUKS1 and something
for metadata store (IIRC they mentioned file on boot disk).

The major risk is that the new cryptographic protocol is not yet properly
reviewed and if broken, half of it need to be redesigned.
(Also it is not in any distro yet.)

> 5- Lars observes that any solution is incomplete if we don't *also* 
> encrypt the swap devices.  (I think we don't need to worry about the boot 
> disks, though, unless we're paranoid about /var/log/ceph?)

It depends on your threat model...

But you can easily create encrypted swap using key from /dev/urandom
(iow regenerated every boot).

This is common solution if you do not store hibernation data in swap.
(Just one line in crypttab/fstab.)

> 6- Radoslaw and Adam observe that anything using dm-crypt may have higher 
> overhead than something implemented in RADOS itself.  But this is vastly 
> more complicated and, if we move in this direction, a long way off.

Gluster implemented encryption this way (no idea if it is used in reality),
it is months of development. It can provide almost native performance but
risk of doing something wrong and later redesign is quite high.

Also you can be hit by FIPS (and similar) certification requirements later...
(for government users)

But yes, there will be always overhead for dmcrypt, but because OSDs usually
run on quite powerful hw (with AESNI etc) device-mapper team can help to
solve performance issues.

> 7- In the meantime, we have nothing better than /etc/ceph/dmcrypt-keys 
> upstream...
> 
> 
> I suggest we do something simple:
> 
> 1- Update SUSE's ceph-disk changes to make it easy to plug in 
> different key management strategies.
> 
> 2- Implement a simple mon-based strategy upstream.  We've discussed this a 
> fair bit in the past, and were getting stuck on the problem of where to 
> store the key-fetching-key.  I.e., we want a key on the disk that you use 
> to ask the monitor for the LUKS key, which you then provide to LUKS to 
> unlock the actual encryption key.  This means that we need a unencrypted 
> spot on the device to store it in.  Milan has indicated that putting it in 
> a LUKS key slot would be a bad idea and difficult to maintain.  Instead, I 
> propose we create a new GPT partition type called OSD_LOCKBOX (or 
> similar), with a tiny filesystem and a few files indicating what to do.  
> This will make it easy to store the info we need for the mon scheme, and 
> to support new key management approaches later (we can put whatever we 
> want there as long as it's not too big).

I would like to have LUKS1 to be stable format, so hacking with
keyslots would be fragile solution... (Despite it is possible.)

As I mentioned, LUKS2 should solve this but it is probably not usable
in this timeframe you need.

I just want to mention LUKS2 just idea but it can simplify the whole configuration
in future for you.

Milan

Re: osd dm-crypt key management, part... deux?

[Sage Weil]

On Wed, 10 Feb 2016, Milan Broz wrote:
> On 01/27/2016 03:26 PM, Sage Weil wrote:
> > We've had several partial starts to address this problem but haven't 
> > gotten anything over the line.  A quick summary:
> > 
> > 1- Currently we store dm-crypt keys in /etc/ceph/dmcrypt-keys/$osd_uuid, 
> > on the boot disk.  This lets you throw away OSD disk but not boot disks 
> > and doesn't help you if someone walks away with a whole server.
> > 
> > 2- SUSE had a pull request that made ceph-disk push/pull keys over (s)ftp.  
> > I can't find it now.. did it get closed?
> > 
> > 3- We built a generic "config-key" storage service into the monitor that 
> > lets you restrict access to certain key/value pairs based on your 
> > (cephx) authentication key, intending that the osd dm-crypt keys be stored 
> > there.
> 
> My (upstream cryptsetup) plan is to have LUKS2 configurable keyslot and these
> should allow plugins that can directly fetch the key for example.
> 
> In fact proof-of-concept code was inspired by Ceph OSD use case and
> allows fetching key from remote server through libssh (it expects
> ssh keys configured such way, that you can run ssh to some server
> with public key auth without password - but that's just example).
> 
> In LUKS2 header then we store just ssh server/user/dir
> https://gitlab.com/cryptsetup/cryptsetup/blob/wip-luks2/misc/luks2_keyslot_example/keyslot_test.c
> (please note it is just wip example of standalone binary, interface will change,
> and currently it uses real volume key, not unlocking passphrase)
> 
> I can imagine that in long-term there can be simple keyslot using
> cephx auth as well or whatever Ceph will need.

This sounds really promising! I wish it were arriving a bit sooner... :/

> > 5- Lars observes that any solution is incomplete if we don't *also* 
> > encrypt the swap devices.  (I think we don't need to worry about the boot 
> > disks, though, unless we're paranoid about /var/log/ceph?)
> 
> It depends on your threat model...
> 
> But you can easily create encrypted swap using key from /dev/urandom
> (iow regenerated every boot).
> 
> This is common solution if you do not store hibernation data in swap.
> (Just one line in crypttab/fstab.)

Good to know!  Lars, does this fully address your concerns?  I forget if 
the boot drive (and /var/log) was part of your concern.

sage