Re: Improving Data-At-Rest encryption in Ceph

[Lars Marowsky-Bree <lmb@suse.com>]

On 2015-12-14T14:17:08, Radoslaw Zarzynski <rzarzynski@mirantis.com> wrote:

Hi all,

great to see this revived.

However, I have come to see some concerns with handling the encryption
within Ceph itself.

The key part to any such approach is formulating the threat scenario.
For the use cases we have seen, the data-at-rest encryption matters so
they can confidently throw away disks without leaking data. It's not
meant as a defense against an online attacker. There usually is no
problem with "a few" disks being privileged, or one or two nodes that
need an admin intervention for booting (to enter some master encryption
key somehow, somewhere).

However, that requires *all* data on the OSDs to be encrypted.

Crucially, that includes not just the file system meta data (so not just
the data), but also the root and especially the swap partition. Those
potentially include swapped out data, coredumps, logs, etc.

(As an optional feature, it'd be cool if an OSD could be moved to a
different chassis and continue operating there, to speed up recovery.
Another optional feature would be to eventually be able, for those
customers that trust them ;-), supply the key to the on-disk encryption
(OPAL et al).)

The proposal that Joshua posted a while ago essentially remained based
on dm-crypt, but put in simple hooks to retrieve the keys from some
"secured" server via sftp/ftps instead of loading them from the root fs.
Similar to deo, that ties the key to being on the network and knowing
the OSD UUID.

This would then also be somewhat easily extensible to utilize the same
key management server via initrd/dracut.

Yes, this means that each OSD disk is separately encrypted, but given
modern CPUs, this is less of a problem. It does have the benefit of
being completely transparent to Ceph, and actually covering the whole
node.

Of course, one of the key issues is always the key server.
Putting/retrieving/deleting keys is reasonably simple, but the question
of how to ensure HA for it is a bit tricky. But doable; people have been
building HA ftp/http servers for a while ;-) Also, a single key server
setup could theoretically serve multiple Ceph clusters.

It's not yet perfect, but I think the approach is superior to being
implemented in Ceph natively. If there's any encryption that should be
implemented in Ceph, I believe it'd be the on-the-wire encryption to
protect against evasedroppers.

Other scenarios would require client-side encryption.

> Current data at rest encryption is achieved through dm-crypt placed
> under OSD’s filestore. This solution is a generic one and cannot
> leverage Ceph-specific characteristics. The best example is that
> encryption is done multiple times - one time for each replica. Another
> issue is lack of granularity - either OSD encrypts nothing, or OSD
> encrypts everything (with dm-crypt on).

True. But for the threat scenario, a holistic approach to encryption
seems actually required.

> Cryptographic keys are stored on filesystem of storage node that hosts
> OSDs. Changing them require redeploying the OSDs.

This is solvable by storing the key on an external key server.

Changing the key is only necessary if the key has been exposed. And with
dm-crypt, that's still possible - it's not the actual encryption key
that's stored, but the secret that is needed to unlock it, and that can
be re-encrypted quite fast. (In theory; it's not implemented yet for
the Ceph OSDs.)


> Data incoming from Ceph clients would be encrypted by primary OSD. It
> would replicate ciphertext to non-primary members of an acting set.

This still exposes data in coredumps or on swap on the primary OSD, and
metadata on the secondaries.


Regards,
    Lars

-- 
Architect Storage/HA
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg)
"Experience is the name everyone gives to their mistakes." -- Oscar Wilde

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html