Proposal: data-at-rest encryption

Proposal: data-at-rest encryption

[Joshua Schmid]

Hi all,


just as a quick preface:

I'm fairly new to ceph, so forgive me any blunders :) But back to the topic.

after following several discussions on how to implement data-at-rest
encryption in ceph i want to add my two cents.



What is used:

a dedicated FTPS server for key-handling


How it will work:

Specify your key-server in ceph.conf.
Preparing and activating an OSD as usual via --dmcrypt. The newly
created key will be uploaded and deleted locally. On the initial unlock
it will already be retrieved via network.


Why is this a good way:

Taking in consideration that MONs are not the best place to put the keys
imho, a dedicated machine is a good place to put them since you are able
to take care of additional security arrangements.


What needs attention:

- The dedicated server is a single point of failure.
- If you add more servers, is rsync enough?
- Prevent swapping on key retrieval. (mlockall)



This is what i did so far:

https://github.com/jschmid1/ceph/commit/fee26890c24bd3a7b8865295546297e3f144e6d0?diff=unified


ANY feedback is welcome :)


-- 
Freundliche Grüße - Kind regards,
Joshua Schmid
Trainee - Storage
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nürnberg
--------------------------------------------------------------------------------------------------------------------
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
--------------------------------------------------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal: data-at-rest encryption

[Sage Weil]

Hi Joshua!

On Mon, 24 Aug 2015, Joshua Schmid wrote:
> Hi all,
> 
> 
> just as a quick preface:
> 
> I'm fairly new to ceph, so forgive me any blunders :) But back to the topic.
> 
> after following several discussions on how to implement data-at-rest
> encryption in ceph i want to add my two cents.
> 
> 
> What is used:
> 
> a dedicated FTPS server for key-handling
> 
> 
> How it will work:
> 
> Specify your key-server in ceph.conf.
> Preparing and activating an OSD as usual via --dmcrypt. The newly
> created key will be uploaded and deleted locally. On the initial unlock
> it will already be retrieved via network.
> 
> 
> Why is this a good way:
> 
> Taking in consideration that MONs are not the best place to put the keys
> imho, a dedicated machine is a good place to put them since you are able
> to take care of additional security arrangements.
> 
> 
> What needs attention:
> 
> - The dedicated server is a single point of failure.
> - If you add more servers, is rsync enough?
> - Prevent swapping on key retrieval. (mlockall)
> 
> 
> 
> This is what i did so far:
> 
> https://github.com/jschmid1/ceph/commit/fee26890c24bd3a7b8865295546297e3f144e6d0?diff=unified
> 
> 
> ANY feedback is welcome :)

Overall the ceph-disk changes look pretty good, and it looks like Andrew 
and David have both reviewed.  My only real concern/request is that the 
key server be as pluggable as possible.  You're using ftps here, but we'd 
also like to allow deo[1], or even the mon config-key service.

With the original mon proposal, we also wanted an additional layer of 
security (beyond simply access to the storage network) by 
storing some key-fetching-key on the disk.  It looks like the ftps 
access is unauthenticated... is that right?  I would assume (I'm not hte 
expert!) that most key management systems require some credentials to 
store/fetch keys?

sage

Re: Proposal: data-at-rest encryption

[Joshua Schmid]

On 08/27/2015 02:49 AM, Sage Weil wrote:
> Hi Joshua!
> 

Hi Sage,


> Overall the ceph-disk changes look pretty good, and it looks like Andrew 
> and David have both reviewed.  My only real concern/request is that the 
> key server be as pluggable as possible.  You're using ftps here, but we'd 
> also like to allow deo[1], or even the mon config-key service.

Thank for having a look!
I think this should do:

https://github.com/jschmid1/ceph/commit/7dd64c70bcb8d986568d6f379a6fbf9a0e40a441

Service of choice can now be set in the ceph.conf and will be handled
separately. This is currently only for unlocking/mapping but will be
extended for locking/new if this solution is acceptable.


> 
> With the original mon proposal, we also wanted an additional layer of 
> security (beyond simply access to the storage network) by 
> storing some key-fetching-key on the disk.

Like deo does it? (From the deo README)

"""
Second, we will add a new random key to the pre-existing LUKS encrypted
disk and then encrypt it using Deo in a known location.
"""


  It looks like the ftps
> access is unauthenticated... is that right?  I would assume (I'm not hte 
> expert!) that most key management systems require some credentials to 
> store/fetch keys?

Its totally unauthenticated, thats right. It'd be possible to require
USER/PASS for ftp.
> 
> sage
> 

-- 
Freundliche Grüße - Kind regards,
Joshua Schmid
Trainee - Storage
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nürnberg
--------------------------------------------------------------------------------------------------------------------
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
--------------------------------------------------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal: data-at-rest encryption

[Sage Weil]

On Thu, 27 Aug 2015, Joshua Schmid wrote:
> 
> On 08/27/2015 02:49 AM, Sage Weil wrote:
> > Hi Joshua!
> > 
> 
> Hi Sage,
> 
> 
> > Overall the ceph-disk changes look pretty good, and it looks like Andrew 
> > and David have both reviewed.  My only real concern/request is that the 
> > key server be as pluggable as possible.  You're using ftps here, but we'd 
> > also like to allow deo[1], or even the mon config-key service.
> 
> Thank for having a look!
> I think this should do:
> 
> https://github.com/jschmid1/ceph/commit/7dd64c70bcb8d986568d6f379a6fbf9a0e40a441
> 
> Service of choice can now be set in the ceph.conf and will be handled
> separately. This is currently only for unlocking/mapping but will be
> extended for locking/new if this solution is acceptable.

Yep!  It'd probably be much cleaner to wrap this up in a class with 
fetch_key() and create_key(), etc. methods so that there is only one place 
that has to instantiate the implemetnation based on type.

> > With the original mon proposal, we also wanted an additional layer of 
> > security (beyond simply access to the storage network) by 
> > storing some key-fetching-key on the disk.
> 
> Like deo does it? (From the deo README)
> 
> """
> Second, we will add a new random key to the pre-existing LUKS encrypted
> disk and then encrypt it using Deo in a known location.
> """
> 
> 
>   It looks like the ftps
> > access is unauthenticated... is that right?  I would assume (I'm not hte 
> > expert!) that most key management systems require some credentials to 
> > store/fetch keys?
> 
> Its totally unauthenticated, thats right. It'd be possible to require
> USER/PASS for ftp.

Yeah.  If we have a general method to store a key-fetching-key on the disk 
(in the LUKS table?  I forget if this was practical) on the LUKS disk the 
might work?  Hopefully such that various backends can all use it (e.g., as 
the ftps password, or as a mon key)..

sage

Re: Proposal: data-at-rest encryption

[Joshua Schmid]

On 08/27/2015 03:38 PM, Sage Weil wrote:
> On Thu, 27 Aug 2015, Joshua Schmid wrote:
>>
>> On 08/27/2015 02:49 AM, Sage Weil wrote:
>>> Hi Joshua!
>>>
>>
>> Hi Sage,
>>
>>
>>> Overall the ceph-disk changes look pretty good, and it looks like Andrew 
>>> and David have both reviewed.  My only real concern/request is that the 
>>> key server be as pluggable as possible.  You're using ftps here, but we'd 
>>> also like to allow deo[1], or even the mon config-key service.
>>
>> Thank for having a look!
>> I think this should do:
>>
>> https://github.com/jschmid1/ceph/commit/7dd64c70bcb8d986568d6f379a6fbf9a0e40a441
>>
>> Service of choice can now be set in the ceph.conf and will be handled
>> separately. This is currently only for unlocking/mapping but will be
>> extended for locking/new if this solution is acceptable.
> 
> Yep!  It'd probably be much cleaner to wrap this up in a class with 
> fetch_key() and create_key(), etc. methods so that there is only one place 
> that has to instantiate the implemetnation based on type.
> 

I dont know if i understood you correctly here. But, since ceph-disk is
classless wouldn't it be a bit strange to introduce one?
I now put anything associated with fetching/creating key in a seperate
method retrieve_key() and create_key() which will behave accordingly.



>>> With the original mon proposal, we also wanted an additional layer of 
>>> security (beyond simply access to the storage network) by 
>>> storing some key-fetching-key on the disk.
>>
>> Like deo does it? (From the deo README)
>>
>> """
>> Second, we will add a new random key to the pre-existing LUKS encrypted
>> disk and then encrypt it using Deo in a known location.
>> """
>>
>>
>>   It looks like the ftps
>>> access is unauthenticated... is that right?  I would assume (I'm not hte 
>>> expert!) that most key management systems require some credentials to 
>>> store/fetch keys?
>>
>> Its totally unauthenticated, thats right. It'd be possible to require
>> USER/PASS for ftp.
> 
> Yeah.  If we have a general method to store a key-fetching-key on the disk 
> (in the LUKS table?  I forget if this was practical) on the LUKS disk the 
> might work?  Hopefully such that various backends can all use it (e.g., as 
> the ftps password, or as a mon key)..


I guess putting it in the LuksKeySlot[1] might work. But plain-dmcrypt
has to be handled differently then. For compatibility reasons i suggest
to avoid that. But storing it on the root partition is not a good idea
either. This area definitely needs more heavy thinking..


> 
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
Freundliche Grüße - Kind regards,
Joshua Schmid
Trainee - Storage
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nürnberg
--------------------------------------------------------------------------------------------------------------------
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
--------------------------------------------------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal: data-at-rest encryption

[Joshua Schmid]

On 08/28/2015 01:32 AM, Shinobu wrote:
> Hello,
> 
> Just question.
> If the key is broken, how could ceph (maybe named kss standing for key
> store server :0) recover, and where information to restore it could be
> retrieved?
> Any blueprint?

Hi,

i don't know if i got your question right.
Are you asking what ceph will do if the keyserver is down and where to
get the information from on how to restore the OSD?

Well, there will be a timeout if the KSS is unreachable. But in a
productive environment it might not be a bad idea to add HA to your KSS.

> 
> Shinobu
> 
> On Thu, Aug 27, 2015 at 10:38 PM, Sage Weil <sage@newdream.net> wrote:
> 
>> On Thu, 27 Aug 2015, Joshua Schmid wrote:
>>>
>>> On 08/27/2015 02:49 AM, Sage Weil wrote:
>>>> Hi Joshua!
>>>>
>>>
>>> Hi Sage,
>>>
>>>
>>>> Overall the ceph-disk changes look pretty good, and it looks like
>> Andrew
>>>> and David have both reviewed.  My only real concern/request is that the
>>>> key server be as pluggable as possible.  You're using ftps here, but
>> we'd
>>>> also like to allow deo[1], or even the mon config-key service.
>>>
>>> Thank for having a look!
>>> I think this should do:
>>>
>>>
>> https://github.com/jschmid1/ceph/commit/7dd64c70bcb8d986568d6f379a6fbf9a0e40a441
>>>
>>> Service of choice can now be set in the ceph.conf and will be handled
>>> separately. This is currently only for unlocking/mapping but will be
>>> extended for locking/new if this solution is acceptable.
>>
>> Yep!  It'd probably be much cleaner to wrap this up in a class with
>> fetch_key() and create_key(), etc. methods so that there is only one place
>> that has to instantiate the implemetnation based on type.
>>
>>>> With the original mon proposal, we also wanted an additional layer of
>>>> security (beyond simply access to the storage network) by
>>>> storing some key-fetching-key on the disk.
>>>
>>> Like deo does it? (From the deo README)
>>>
>>> """
>>> Second, we will add a new random key to the pre-existing LUKS encrypted
>>> disk and then encrypt it using Deo in a known location.
>>> """
>>>
>>>
>>>   It looks like the ftps
>>>> access is unauthenticated... is that right?  I would assume (I'm not
>> hte
>>>> expert!) that most key management systems require some credentials to
>>>> store/fetch keys?
>>>
>>> Its totally unauthenticated, thats right. It'd be possible to require
>>> USER/PASS for ftp.
>>
>> Yeah.  If we have a general method to store a key-fetching-key on the disk
>> (in the LUKS table?  I forget if this was practical) on the LUKS disk the
>> might work?  Hopefully such that various backends can all use it (e.g., as
>> the ftps password, or as a mon key)..
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 
> 

-- 
Freundliche Grüße - Kind regards,
Joshua Schmid
Trainee - Storage
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nürnberg
--------------------------------------------------------------------------------------------------------------------
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
--------------------------------------------------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Proposal: data-at-rest encryption

[Shinobu Kinjo]

Hi,

Thanks for your good comeback!

 > to your KSS -;

Shinobu

----- Original Message -----
From: "Joshua Schmid" <jschmid@suse.de>
To: skinjo@redhat.com, "Sage Weil" <sage@newdream.net>
Cc: "Ceph Development" <ceph-devel@vger.kernel.org>
Sent: Friday, August 28, 2015 5:11:34 PM
Subject: Re: Proposal: data-at-rest encryption



On 08/28/2015 01:32 AM, Shinobu wrote:
> Hello,
> 
> Just question.
> If the key is broken, how could ceph (maybe named kss standing for key
> store server :0) recover, and where information to restore it could be
> retrieved?
> Any blueprint?

Hi,

i don't know if i got your question right.
Are you asking what ceph will do if the keyserver is down and where to
get the information from on how to restore the OSD?

Well, there will be a timeout if the KSS is unreachable. But in a
productive environment it might not be a bad idea to add HA to your KSS.

> 
> Shinobu
> 
> On Thu, Aug 27, 2015 at 10:38 PM, Sage Weil <sage@newdream.net> wrote:
> 
>> On Thu, 27 Aug 2015, Joshua Schmid wrote:
>>>
>>> On 08/27/2015 02:49 AM, Sage Weil wrote:
>>>> Hi Joshua!
>>>>
>>>
>>> Hi Sage,
>>>
>>>
>>>> Overall the ceph-disk changes look pretty good, and it looks like
>> Andrew
>>>> and David have both reviewed.  My only real concern/request is that the
>>>> key server be as pluggable as possible.  You're using ftps here, but
>> we'd
>>>> also like to allow deo[1], or even the mon config-key service.
>>>
>>> Thank for having a look!
>>> I think this should do:
>>>
>>>
>> https://github.com/jschmid1/ceph/commit/7dd64c70bcb8d986568d6f379a6fbf9a0e40a441
>>>
>>> Service of choice can now be set in the ceph.conf and will be handled
>>> separately. This is currently only for unlocking/mapping but will be
>>> extended for locking/new if this solution is acceptable.
>>
>> Yep!  It'd probably be much cleaner to wrap this up in a class with
>> fetch_key() and create_key(), etc. methods so that there is only one place
>> that has to instantiate the implemetnation based on type.
>>
>>>> With the original mon proposal, we also wanted an additional layer of
>>>> security (beyond simply access to the storage network) by
>>>> storing some key-fetching-key on the disk.
>>>
>>> Like deo does it? (From the deo README)
>>>
>>> """
>>> Second, we will add a new random key to the pre-existing LUKS encrypted
>>> disk and then encrypt it using Deo in a known location.
>>> """
>>>
>>>
>>>   It looks like the ftps
>>>> access is unauthenticated... is that right?  I would assume (I'm not
>> hte
>>>> expert!) that most key management systems require some credentials to
>>>> store/fetch keys?
>>>
>>> Its totally unauthenticated, thats right. It'd be possible to require
>>> USER/PASS for ftp.
>>
>> Yeah.  If we have a general method to store a key-fetching-key on the disk
>> (in the LUKS table?  I forget if this was practical) on the LUKS disk the
>> might work?  Hopefully such that various backends can all use it (e.g., as
>> the ftps password, or as a mon key)..
>>
>> sage
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> 
> 
> 

-- 
Freundliche Grüße - Kind regards,
Joshua Schmid
Trainee - Storage
SUSE Linux GmbH - Maxfeldstr. 5 - 90409 Nürnberg
--------------------------------------------------------------------------------------------------------------------
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard,
Jennifer Guild, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)
--------------------------------------------------------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html