[dm-crypt] LUKS in failover cluster

[dm-crypt] LUKS in failover cluster

[Sohl, Jacob (LNG-SEA)]

[-- Attachment #1: Type: text/plain, Size: 1808 bytes --]

Hi all,

I've been working on a design for an encrypted fileserver using RHEL6.x.
On a single server the stack is pretty simple:

SAN LUNs > LUKS > LVM > XFS > Samba Server

 

But I would like to have a second node for High-Availability failover
(SAN storage is available to both nodes). I'm looking at Red Hat Cluster
Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM,
XFS and Samba resources. In the event of node failure, it will migrate
all resources to the healthy node. But the resources are only available
if the SAN volumes are decrypted:

cryptsetup luksOpen /dev/sdc1 crypt_vol

 

Is it possible to have the raw volumes decrypted on both systems, maybe
during boot. So the LUKS device (/dev/mapper/crypt_vol) will be
available on the backup node in the event of primary node failure. The
other resources - LVM, XFS, Samba - would only be on one node at a time,
so no filesystem access from the passive node. If this is not possible
then can you suggest another solution?

 

Also, scalability is a requirement in my design, hence XFS. I was
thinking I needed to use multiple LUKS PVs in LVM to grow the
filesystem. But I would end up with multiple LUKS devices to keep track
of. I recently found out that LUKS can resize. Would it be better to
create one LUKS device on top of LVM? Then create a filesystem on that?
(Though that would affect resource dependencies.)

But basically:

SAN LUNs > LVM > LUKS > XFS > Samba Server

 

Other people will be accessing/managing this system, so I want
manageability through simplicity. Don't want to have the wrong volumes
(re)encrypted, headers damaged, etc.

Anyways, thanks for your help.

 

Jacob Sohl  |  Systems Engineer

Applied Discovery(r)

Mobile: 360-620-2695

 


[-- Attachment #2: Type: text/html, Size: 16082 bytes --]

Re: [dm-crypt] LUKS in failover cluster

[Arno Wagner]

Hi,

On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote:
> Hi all,
> 
> I've been working on a design for an encrypted fileserver using RHEL6.x.
> On a single server the stack is pretty simple:
> 
> SAN LUNs > LUKS > LVM > XFS > Samba Server
> 
>  
> 
> But I would like to have a second node for High-Availability failover
> (SAN storage is available to both nodes). I'm looking at Red Hat Cluster
> Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM,
> XFS and Samba resources. In the event of node failure, it will migrate
> all resources to the healthy node. But the resources are only available
> if the SAN volumes are decrypted:
> 
> cryptsetup luksOpen /dev/sdc1 crypt_vol
>
>  
> 
> Is it possible to have the raw volumes decrypted on both systems, maybe
> during boot. So the LUKS device (/dev/mapper/crypt_vol) will be
> available on the backup node in the event of primary node failure. The
> other resources - LVM, XFS, Samba - would only be on one node at a time,
> so no filesystem access from the passive node. If this is not possible
> then can you suggest another solution?

You can map ("decrypt") the devices and never use them. The
LVM/mount/whatever is completely optional.

In fact the mapper tool (cryptsetup) does nothing except
to decrypt the raw encrypted device to a raw decrypted device
under /dev/mapper/.
 
  
> Also, scalability is a requirement in my design, hence XFS. I was
> thinking I needed to use multiple LUKS PVs in LVM to grow the
> filesystem. But I would end up with multiple LUKS devices to keep track
> of. I recently found out that LUKS can resize. 

LUKS _cannot_ resize. The thing is that LUKS does not care about
device size. So if you enlarge a device/partition with an intact
LUKS header, "cryptsetup luksOpen" will just map the larger 
device.

If you have multiple devices, you can "slave" the additional ones
to a "master" device using something like "decrypt_derived". 
This just takes the master key from the opened "master" container,
runns it though a hash and uses this as key for the "slave"
device. 

> Would it be better to
> create one LUKS device on top of LVM? Then create a filesystem on that?
> (Though that would affect resource dependencies.)

Sre you sure you need LVM at all?

> But basically:
> 
> SAN LUNs > LVM > LUKS > XFS > Samba Server
> 
>  
> 
> Other people will be accessing/managing this system, so I want
> manageability through simplicity. 

Hence the question whether you actually need LVM. 
It strikes me that typically LVM is not needed and
just complicates matters. 

> Don't want to have the wrong volumes
> (re)encrypted, headers damaged, etc.

I think this setup is already too complicated for most people
to manage. A full backup should be part of your plans. 
Resize with backup is actually easier, as you can just
backup, kill everything and do a clean new config.

> Anyways, thanks for your help.
> 

Just to make sure: You _have_ read the FAQ?
 
Arno

> 
> Jacob Sohl  |  Systems Engineer
> 
> Applied Discovery(r)
> 
> Mobile: 360-620-2695
> 
>  
> 

> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt


-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] LUKS in failover cluster

[Milan Broz]

On 10/08/2011 06:52 AM, Arno Wagner wrote:
>> Also, scalability is a requirement in my design, hence XFS. I was
>> thinking I needed to use multiple LUKS PVs in LVM to grow the
>> filesystem. But I would end up with multiple LUKS devices to keep track
>> of. I recently found out that LUKS can resize. 
> 
> LUKS _cannot_ resize. The thing is that LUKS does not care about
> device size. So if you enlarge a device/partition with an intact
> LUKS header, "cryptsetup luksOpen" will just map the larger 
> device.

yes, but crytsetup resize is designed that it it supports open
(active) LUKS containers. You do not need to activate/deactivate,
resize operation is enough.

> If you have multiple devices, you can "slave" the additional ones
> to a "master" device using something like "decrypt_derived". 
> This just takes the master key from the opened "master" container,
> runns it though a hash and uses this as key for the "slave"
> device. 
> 
>> Would it be better to
>> create one LUKS device on top of LVM? Then create a filesystem on that?
>> (Though that would affect resource dependencies.)
> 
> Sre you sure you need LVM at all?

LVM is very useful in such configurations, you can dynamically manage
LVs over encrypted storage.
Scripts for managing such stack is not complicated but of course
it is yet another layer to care of.

Because you mentioned cluster, I will just repeat

- if you manage it in HA LVM style (the device
stack is always active exclusively only on one node) it will work.
(You just need to create own resource for cluster suite - add LUKS
to the stack. So cluster suite will handle exclusivity and failover itself.)

For using LUKS in cluster for shared storage (visible to all nodes)
- if encryption (mapping) runs in server and plaintext device is exported
and shared to nodes (using iscsi + clvm or so) this will work

- if you want to run dm-crypt on every node (for the same device,
IOW export ciphertext device) this is not supported.

(I think that I said on IRC that with proper used cluster fs and
flush requests this may work - but it is not tested. The problem is of course
with internal crypto queues so without flushing these queues different
nodes will see different storage content. And moreover, there is no cluster
infrastructure like clvmd to manage such devices consistently.)

Milan

Re: [dm-crypt] LUKS in failover cluster

[Sohl, Jacob (LNG-SEA)]

> -----Original Message-----
> From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On
> Behalf Of Arno Wagner
> Sent: Friday, October 07, 2011 9:53 PM
> To: dm-crypt@saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> Hi,
> 
> On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote:
> > Hi all,
> >
> > I've been working on a design for an encrypted fileserver using
> RHEL6.x.
> > On a single server the stack is pretty simple:
> >
> > SAN LUNs > LUKS > LVM > XFS > Samba Server
> >
> >
> >
> > But I would like to have a second node for High-Availability
failover
> > (SAN storage is available to both nodes). I'm looking at Red Hat
> Cluster
> > Suite with corosyn, rgmanager. rgmanager has the ability to manage
> LVM,
> > XFS and Samba resources. In the event of node failure, it will
> migrate
> > all resources to the healthy node. But the resources are only
> available
> > if the SAN volumes are decrypted:
> >
> > cryptsetup luksOpen /dev/sdc1 crypt_vol
> >
> >
> >
> > Is it possible to have the raw volumes decrypted on both systems,
> maybe
> > during boot. So the LUKS device (/dev/mapper/crypt_vol) will be
> > available on the backup node in the event of primary node failure.
> The
> > other resources - LVM, XFS, Samba - would only be on one node at a
> time,
> > so no filesystem access from the passive node. If this is not
> possible
> > then can you suggest another solution?
> 
> You can map ("decrypt") the devices and never use them. The
> LVM/mount/whatever is completely optional.
> 
> In fact the mapper tool (cryptsetup) does nothing except
> to decrypt the raw encrypted device to a raw decrypted device
> under /dev/mapper/.
> 

Ok, thank you. That seemed logical, but I just recently started working
with encryption so I'm still learning.

> 
> > Also, scalability is a requirement in my design, hence XFS. I was
> > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > filesystem. But I would end up with multiple LUKS devices to keep
> track
> > of. I recently found out that LUKS can resize.
> 
> LUKS _cannot_ resize. The thing is that LUKS does not care about
> device size. So if you enlarge a device/partition with an intact
> LUKS header, "cryptsetup luksOpen" will just map the larger
> device.
> 
> If you have multiple devices, you can "slave" the additional ones
> to a "master" device using something like "decrypt_derived".
> This just takes the master key from the opened "master" container,
> runns it though a hash and uses this as key for the "slave"
> device.
> 
> > Would it be better to
> > create one LUKS device on top of LVM? Then create a filesystem on
> that?
> > (Though that would affect resource dependencies.)
> 
> Sre you sure you need LVM at all?

Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins
can/will only attach storage in 4TB increments. So I have to combine all
of the LUNS into one Logical Volume in order to create a 50TB+ XFS
filesystem. That's why I ask about encrypting multiple physical LUNS vs
a single Logical Volume.

> 
> > But basically:
> >
> > SAN LUNs > LVM > LUKS > XFS > Samba Server
> >
> >
> >
> > Other people will be accessing/managing this system, so I want
> > manageability through simplicity.
> 
> Hence the question whether you actually need LVM.
> It strikes me that typically LVM is not needed and
> just complicates matters.
> 
> > Don't want to have the wrong volumes
> > (re)encrypted, headers damaged, etc.
> 
> I think this setup is already too complicated for most people
> to manage. A full backup should be part of your plans.
> Resize with backup is actually easier, as you can just
> backup, kill everything and do a clean new config.
> 

I will be making full backups with xfsdump. But it doesn't seem
practical to be moving 50TB around. Both LVM and XFS allow for designed
for online resize. I was planning to "pvcreate" an encrypted LUN then
expand the Logical Volume and Filesystem.

> > Anyways, thanks for your help.
> >
> 
> Just to make sure: You _have_ read the FAQ?

Yes I have read the FAQ. I guess I am just a little nervous about this
project. This is my first time working with encryption and 50TB of
critical client data is at stake. Just being thorough and making sure I
fully understand everything.
Thanks again,
Jacob Sohl

> 
> Arno
> 
> >
> > Jacob Sohl  |  Systems Engineer
> >
> > Applied Discovery(r)
> >
> > Mobile: 360-620-2695
> >
> >
> >
> 
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt
> 
> 
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50
1E25
> 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> 
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS in failover cluster

[Sohl, Jacob (LNG-SEA)]

> -----Original Message-----
> From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On
> Behalf Of Milan Broz
> Sent: Saturday, October 08, 2011 12:51 AM
> To: dm-crypt@saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> 
> On 10/08/2011 06:52 AM, Arno Wagner wrote:
> >> Also, scalability is a requirement in my design, hence XFS. I was
> >> thinking I needed to use multiple LUKS PVs in LVM to grow the
> >> filesystem. But I would end up with multiple LUKS devices to keep
> track
> >> of. I recently found out that LUKS can resize.
> >
> > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > device size. So if you enlarge a device/partition with an intact
> > LUKS header, "cryptsetup luksOpen" will just map the larger
> > device.
> 
> yes, but crytsetup resize is designed that it it supports open
> (active) LUKS containers. You do not need to activate/deactivate,
> resize operation is enough.
> 
> > If you have multiple devices, you can "slave" the additional ones
> > to a "master" device using something like "decrypt_derived".
> > This just takes the master key from the opened "master" container,
> > runns it though a hash and uses this as key for the "slave"
> > device.
> >
> >> Would it be better to
> >> create one LUKS device on top of LVM? Then create a filesystem on
> that?
> >> (Though that would affect resource dependencies.)
> >
> > Sre you sure you need LVM at all?
> 
> LVM is very useful in such configurations, you can dynamically manage
> LVs over encrypted storage.
> Scripts for managing such stack is not complicated but of course
> it is yet another layer to care of.
> 
> Because you mentioned cluster, I will just repeat
> 
> - if you manage it in HA LVM style (the device
> stack is always active exclusively only on one node) it will work.
> (You just need to create own resource for cluster suite - add LUKS
> to the stack. So cluster suite will handle exclusivity and failover
> itself.)

^ HA LVM is what I will be doing.

> 
> For using LUKS in cluster for shared storage (visible to all nodes)
> - if encryption (mapping) runs in server and plaintext device is
> exported
> and shared to nodes (using iscsi + clvm or so) this will work
> 
> - if you want to run dm-crypt on every node (for the same device,
> IOW export ciphertext device) this is not supported.
> 
> (I think that I said on IRC that with proper used cluster fs and
> flush requests this may work - but it is not tested. The problem is of
> course
> with internal crypto queues so without flushing these queues different
> nodes will see different storage content. And moreover, there is no
> cluster
> infrastructure like clvmd to manage such devices consistently.)

Thank you for clarifying this for me, I didn't really understand in IRC
what you meant about "flush requests". I'm not a programmer and don't
really know low-level hardware stuff that much. It's been a lot of stuff
for me to learn.

> 
> Milan



> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS in failover cluster

[Arno Wagner]

On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote:
[...]
> > You can map ("decrypt") the devices and never use them. The
> > LVM/mount/whatever is completely optional.
> > 
> > In fact the mapper tool (cryptsetup) does nothing except
> > to decrypt the raw encrypted device to a raw decrypted device
> > under /dev/mapper/.
> > 
> 
> Ok, thank you. That seemed logical, but I just recently started working
> with encryption so I'm still learning.

That is fine. Making sure now will safe you grief later.
 
> > 
> > > Also, scalability is a requirement in my design, hence XFS. I was
> > > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > > filesystem. But I would end up with multiple LUKS devices to keep
> > track
> > > of. I recently found out that LUKS can resize.
> > 
> > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > device size. So if you enlarge a device/partition with an intact
> > LUKS header, "cryptsetup luksOpen" will just map the larger
> > device.
> > 
> > If you have multiple devices, you can "slave" the additional ones
> > to a "master" device using something like "decrypt_derived".
> > This just takes the master key from the opened "master" container,
> > runns it though a hash and uses this as key for the "slave"
> > device.
> > 
> > > Would it be better to
> > > create one LUKS device on top of LVM? Then create a filesystem on
> > that?
> > > (Though that would affect resource dependencies.)
> > 
> > Sre you sure you need LVM at all?
> 
> Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins
> can/will only attach storage in 4TB increments. So I have to combine all
> of the LUNS into one Logical Volume in order to create a 50TB+ XFS
> filesystem. That's why I ask about encrypting multiple physical LUNS vs
> a single Logical Volume.

50TB? Ok, that is a bit larger. I did have such a dataset
for research a few years back on tape. Took several days 
days to get 15TB of it to disk for my largest measurement.

Well, yes, I think LVM is the best way here.

> > 
> > > But basically:
> > >
> > > SAN LUNs > LVM > LUKS > XFS > Samba Server
> > >
> > >
> > >
> > > Other people will be accessing/managing this system, so I want
> > > manageability through simplicity.
> > 
> > Hence the question whether you actually need LVM.
> > It strikes me that typically LVM is not needed and
> > just complicates matters.
> > 
> > > Don't want to have the wrong volumes
> > > (re)encrypted, headers damaged, etc.
> > 
> > I think this setup is already too complicated for most people
> > to manage. A full backup should be part of your plans.
> > Resize with backup is actually easier, as you can just
> > backup, kill everything and do a clean new config.
> > 
> 
> I will be making full backups with xfsdump. But it doesn't seem
> practical to be moving 50TB around. Both LVM and XFS allow for designed
> for online resize. I was planning to "pvcreate" an encrypted LUN then
> expand the Logical Volume and Filesystem.

For that volume, run a full compare as well. I found that
with good hardware, and an IBM Tape-Library I still had 
something like 1 unexplained bit-error every 10TB or so
when pulling the data from the tape-library again. That was 
a few years back, but still.

> > > Anyways, thanks for your help.
> > >
> > 
> > Just to make sure: You _have_ read the FAQ?
> 
> Yes I have read the FAQ. 

Good!

> I guess I am just a little nervous about this project. 

Understandable.

> This is my first time working with encryption and 50TB of
> critical client data is at stake. Just being thorough and making sure I
> fully understand everything.

And that is all you can do. I think you are approaching this
perfectly right. I take it the 50TB are alredy on XFS+LVM
at this time and you are just adding the encryption layer?

Make sure to use XTS mode or plain64 with ESSIV for these
volume sizes.

And let us know what your expeiences are!

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] LUKS in failover cluster

[Sohl, Jacob (LNG-SEA)]

> -----Original Message-----
> From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On
> Behalf Of Arno Wagner
> Sent: Tuesday, October 11, 2011 8:13 PM
> To: dm-crypt@saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote:
> [...]
> > > You can map ("decrypt") the devices and never use them. The
> > > LVM/mount/whatever is completely optional.
> > >
> > > In fact the mapper tool (cryptsetup) does nothing except
> > > to decrypt the raw encrypted device to a raw decrypted device
> > > under /dev/mapper/.
> > >
> >
> > Ok, thank you. That seemed logical, but I just recently started
> working
> > with encryption so I'm still learning.
> 
> That is fine. Making sure now will safe you grief later.
> 
> > >
> > > > Also, scalability is a requirement in my design, hence XFS. I
was
> > > > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > > > filesystem. But I would end up with multiple LUKS devices to
keep
> > > track
> > > > of. I recently found out that LUKS can resize.
> > >
> > > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > > device size. So if you enlarge a device/partition with an intact
> > > LUKS header, "cryptsetup luksOpen" will just map the larger
> > > device.
> > >
> > > If you have multiple devices, you can "slave" the additional ones
> > > to a "master" device using something like "decrypt_derived".
> > > This just takes the master key from the opened "master" container,
> > > runns it though a hash and uses this as key for the "slave"
> > > device.
> > >
> > > > Would it be better to
> > > > create one LUKS device on top of LVM? Then create a filesystem
on
> > > that?
> > > > (Though that would affect resource dependencies.)
> > >
> > > Sre you sure you need LVM at all?
> >
> > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN
admins
> > can/will only attach storage in 4TB increments. So I have to combine
> all
> > of the LUNS into one Logical Volume in order to create a 50TB+ XFS
> > filesystem. That's why I ask about encrypting multiple physical LUNS
> vs
> > a single Logical Volume.
> 
> 50TB? Ok, that is a bit larger. I did have such a dataset
> for research a few years back on tape. Took several days
> days to get 15TB of it to disk for my largest measurement.
> 
> Well, yes, I think LVM is the best way here.
> 
> > >
> > > > But basically:
> > > >
> > > > SAN LUNs > LVM > LUKS > XFS > Samba Server
> > > >
> > > >
> > > >
> > > > Other people will be accessing/managing this system, so I want
> > > > manageability through simplicity.
> > >
> > > Hence the question whether you actually need LVM.
> > > It strikes me that typically LVM is not needed and
> > > just complicates matters.
> > >
> > > > Don't want to have the wrong volumes
> > > > (re)encrypted, headers damaged, etc.
> > >
> > > I think this setup is already too complicated for most people
> > > to manage. A full backup should be part of your plans.
> > > Resize with backup is actually easier, as you can just
> > > backup, kill everything and do a clean new config.
> > >
> >
> > I will be making full backups with xfsdump. But it doesn't seem
> > practical to be moving 50TB around. Both LVM and XFS allow for
> designed
> > for online resize. I was planning to "pvcreate" an encrypted LUN
then
> > expand the Logical Volume and Filesystem.
> 
> For that volume, run a full compare as well. I found that
> with good hardware, and an IBM Tape-Library I still had
> something like 1 unexplained bit-error every 10TB or so
> when pulling the data from the tape-library again. That was
> a few years back, but still.
> 

Thanks, I've never been very good at doing backups so any advice is
appreciated.

> > > > Anyways, thanks for your help.
> > > >
> > >
> > > Just to make sure: You _have_ read the FAQ?
> >
> > Yes I have read the FAQ.
> 
> Good!
> 
> > I guess I am just a little nervous about this project.
> 
> Understandable.
> 
> > This is my first time working with encryption and 50TB of
> > critical client data is at stake. Just being thorough and making
sure
> I
> > fully understand everything.
> 
> And that is all you can do. I think you are approaching this
> perfectly right. I take it the 50TB are alredy on XFS+LVM
> at this time and you are just adding the encryption layer?

Currently the data is spread across multiple ext3 volumes of 8TB or less
and is a pain to manage. That is the reason we are moving to XFS. The
encryption is currently being handled by a Brocade switch blade. The
encryption blade caused us major issues last year, resulting in a
multi-day outage. That is the why we are moving to LUKS.

> 
> Make sure to use XTS mode or plain64 with ESSIV for these
> volume sizes.

I don't know what this means. Haha. But I will look it up.

> 
> And let us know what your expeiences are!
> 

Sure thing. This project has been my first exposure to XFS, LUKS and
clustering. And I'm the main Linux guy on my team. I've been working
with Red Hat, but the solution architect laughed when he saw what we
were trying to do. Haha. But he's been doing research and talking to his
Red Hat contacts. And you guys have helped out tremendously! Thank you!

Jacob

> Arno
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50
1E25
> 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> 
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS in failover cluster

[Scott McCarty]

That was a good laugh, maybe a chuckle even :-)

Scott McCarty, RHCE, RHCSA, LPIC-1
Solutions Architect
Email: smccarty@redhat.com
Phone: 312-660-3535
Web: http://people.redhat.com/smccarty


----- Original Message -----
From: "Jacob Sohl (LNG-SEA)" <jacob.sohl@applieddiscovery.com>
To: "Arno Wagner" <arno@wagner.name>, dm-crypt@saout.de
Cc: "LNG-SEA Systems Engineers ADI" <LNG-SEASystemsEngineersADI@lexisnexis.com>, "Scott McCarty" <smccarty@redhat.com>
Sent: Wednesday, October 12, 2011 3:20:51 PM
Subject: RE: [dm-crypt] LUKS in failover cluster



> -----Original Message-----
> From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On
> Behalf Of Arno Wagner
> Sent: Tuesday, October 11, 2011 8:13 PM
> To: dm-crypt@saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote:
> [...]
> > > You can map ("decrypt") the devices and never use them. The
> > > LVM/mount/whatever is completely optional.
> > >
> > > In fact the mapper tool (cryptsetup) does nothing except
> > > to decrypt the raw encrypted device to a raw decrypted device
> > > under /dev/mapper/.
> > >
> >
> > Ok, thank you. That seemed logical, but I just recently started
> working
> > with encryption so I'm still learning.
> 
> That is fine. Making sure now will safe you grief later.
> 
> > >
> > > > Also, scalability is a requirement in my design, hence XFS. I
was
> > > > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > > > filesystem. But I would end up with multiple LUKS devices to
keep
> > > track
> > > > of. I recently found out that LUKS can resize.
> > >
> > > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > > device size. So if you enlarge a device/partition with an intact
> > > LUKS header, "cryptsetup luksOpen" will just map the larger
> > > device.
> > >
> > > If you have multiple devices, you can "slave" the additional ones
> > > to a "master" device using something like "decrypt_derived".
> > > This just takes the master key from the opened "master" container,
> > > runns it though a hash and uses this as key for the "slave"
> > > device.
> > >
> > > > Would it be better to
> > > > create one LUKS device on top of LVM? Then create a filesystem
on
> > > that?
> > > > (Though that would affect resource dependencies.)
> > >
> > > Sre you sure you need LVM at all?
> >
> > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN
admins
> > can/will only attach storage in 4TB increments. So I have to combine
> all
> > of the LUNS into one Logical Volume in order to create a 50TB+ XFS
> > filesystem. That's why I ask about encrypting multiple physical LUNS
> vs
> > a single Logical Volume.
> 
> 50TB? Ok, that is a bit larger. I did have such a dataset
> for research a few years back on tape. Took several days
> days to get 15TB of it to disk for my largest measurement.
> 
> Well, yes, I think LVM is the best way here.
> 
> > >
> > > > But basically:
> > > >
> > > > SAN LUNs > LVM > LUKS > XFS > Samba Server
> > > >
> > > >
> > > >
> > > > Other people will be accessing/managing this system, so I want
> > > > manageability through simplicity.
> > >
> > > Hence the question whether you actually need LVM.
> > > It strikes me that typically LVM is not needed and
> > > just complicates matters.
> > >
> > > > Don't want to have the wrong volumes
> > > > (re)encrypted, headers damaged, etc.
> > >
> > > I think this setup is already too complicated for most people
> > > to manage. A full backup should be part of your plans.
> > > Resize with backup is actually easier, as you can just
> > > backup, kill everything and do a clean new config.
> > >
> >
> > I will be making full backups with xfsdump. But it doesn't seem
> > practical to be moving 50TB around. Both LVM and XFS allow for
> designed
> > for online resize. I was planning to "pvcreate" an encrypted LUN
then
> > expand the Logical Volume and Filesystem.
> 
> For that volume, run a full compare as well. I found that
> with good hardware, and an IBM Tape-Library I still had
> something like 1 unexplained bit-error every 10TB or so
> when pulling the data from the tape-library again. That was
> a few years back, but still.
> 

Thanks, I've never been very good at doing backups so any advice is
appreciated.

> > > > Anyways, thanks for your help.
> > > >
> > >
> > > Just to make sure: You _have_ read the FAQ?
> >
> > Yes I have read the FAQ.
> 
> Good!
> 
> > I guess I am just a little nervous about this project.
> 
> Understandable.
> 
> > This is my first time working with encryption and 50TB of
> > critical client data is at stake. Just being thorough and making
sure
> I
> > fully understand everything.
> 
> And that is all you can do. I think you are approaching this
> perfectly right. I take it the 50TB are alredy on XFS+LVM
> at this time and you are just adding the encryption layer?

Currently the data is spread across multiple ext3 volumes of 8TB or less
and is a pain to manage. That is the reason we are moving to XFS. The
encryption is currently being handled by a Brocade switch blade. The
encryption blade caused us major issues last year, resulting in a
multi-day outage. That is the why we are moving to LUKS.

> 
> Make sure to use XTS mode or plain64 with ESSIV for these
> volume sizes.

I don't know what this means. Haha. But I will look it up.

> 
> And let us know what your expeiences are!
> 

Sure thing. This project has been my first exposure to XFS, LUKS and
clustering. And I'm the main Linux guy on my team. I've been working
with Red Hat, but the solution architect laughed when he saw what we
were trying to do. Haha. But he's been doing research and talking to his
Red Hat contacts. And you guys have helped out tremendously! Thank you!

Jacob

> Arno
> --
> Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> arno@wagner.name
> GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50
1E25
> 338F
> ----
> Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> 
> If it's in the news, don't worry about it.  The very definition of
> "news" is "something that hardly ever happens." -- Bruce Schneier
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt

Re: [dm-crypt] LUKS in failover cluster

[Sohl, Jacob (LNG-SEA)]

Yeah, it was a "crazy-fun-project" laugh. An opportunity to learn a lot and create something cool.

> -----Original Message-----
> From: Scott McCarty [mailto:smccarty@redhat.com]
> Sent: Wednesday, October 12, 2011 12:55 PM
> To: Sohl, Jacob (LNG-SEA)
> Cc: LNG-SEA Systems Engineers ADI; Arno Wagner; dm-crypt@saout.de
> Subject: Re: [dm-crypt] LUKS in failover cluster
> 
> That was a good laugh, maybe a chuckle even :-)
> 
> Scott McCarty, RHCE, RHCSA, LPIC-1
> Solutions Architect
> Email: smccarty@redhat.com
> Phone: 312-660-3535
> Web: http://people.redhat.com/smccarty
> 
> 
> ----- Original Message -----
> From: "Jacob Sohl (LNG-SEA)" <jacob.sohl@applieddiscovery.com>
> To: "Arno Wagner" <arno@wagner.name>, dm-crypt@saout.de
> Cc: "LNG-SEA Systems Engineers ADI" <LNG-
> SEASystemsEngineersADI@lexisnexis.com>, "Scott McCarty"
> <smccarty@redhat.com>
> Sent: Wednesday, October 12, 2011 3:20:51 PM
> Subject: RE: [dm-crypt] LUKS in failover cluster
> 
> 
> 
> > -----Original Message-----
> > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On
> > Behalf Of Arno Wagner
> > Sent: Tuesday, October 11, 2011 8:13 PM
> > To: dm-crypt@saout.de
> > Subject: Re: [dm-crypt] LUKS in failover cluster
> >
> > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA)
> wrote:
> > [...]
> > > > You can map ("decrypt") the devices and never use them. The
> > > > LVM/mount/whatever is completely optional.
> > > >
> > > > In fact the mapper tool (cryptsetup) does nothing except
> > > > to decrypt the raw encrypted device to a raw decrypted device
> > > > under /dev/mapper/.
> > > >
> > >
> > > Ok, thank you. That seemed logical, but I just recently started
> > working
> > > with encryption so I'm still learning.
> >
> > That is fine. Making sure now will safe you grief later.
> >
> > > >
> > > > > Also, scalability is a requirement in my design, hence XFS. I
> was
> > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the
> > > > > filesystem. But I would end up with multiple LUKS devices to
> keep
> > > > track
> > > > > of. I recently found out that LUKS can resize.
> > > >
> > > > LUKS _cannot_ resize. The thing is that LUKS does not care about
> > > > device size. So if you enlarge a device/partition with an intact
> > > > LUKS header, "cryptsetup luksOpen" will just map the larger
> > > > device.
> > > >
> > > > If you have multiple devices, you can "slave" the additional ones
> > > > to a "master" device using something like "decrypt_derived".
> > > > This just takes the master key from the opened "master"
> container,
> > > > runns it though a hash and uses this as key for the "slave"
> > > > device.
> > > >
> > > > > Would it be better to
> > > > > create one LUKS device on top of LVM? Then create a filesystem
> on
> > > > that?
> > > > > (Though that would affect resource dependencies.)
> > > >
> > > > Sre you sure you need LVM at all?
> > >
> > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN
> admins
> > > can/will only attach storage in 4TB increments. So I have to
> combine
> > all
> > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS
> > > filesystem. That's why I ask about encrypting multiple physical
> LUNS
> > vs
> > > a single Logical Volume.
> >
> > 50TB? Ok, that is a bit larger. I did have such a dataset
> > for research a few years back on tape. Took several days
> > days to get 15TB of it to disk for my largest measurement.
> >
> > Well, yes, I think LVM is the best way here.
> >
> > > >
> > > > > But basically:
> > > > >
> > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server
> > > > >
> > > > >
> > > > >
> > > > > Other people will be accessing/managing this system, so I want
> > > > > manageability through simplicity.
> > > >
> > > > Hence the question whether you actually need LVM.
> > > > It strikes me that typically LVM is not needed and
> > > > just complicates matters.
> > > >
> > > > > Don't want to have the wrong volumes
> > > > > (re)encrypted, headers damaged, etc.
> > > >
> > > > I think this setup is already too complicated for most people
> > > > to manage. A full backup should be part of your plans.
> > > > Resize with backup is actually easier, as you can just
> > > > backup, kill everything and do a clean new config.
> > > >
> > >
> > > I will be making full backups with xfsdump. But it doesn't seem
> > > practical to be moving 50TB around. Both LVM and XFS allow for
> > designed
> > > for online resize. I was planning to "pvcreate" an encrypted LUN
> then
> > > expand the Logical Volume and Filesystem.
> >
> > For that volume, run a full compare as well. I found that
> > with good hardware, and an IBM Tape-Library I still had
> > something like 1 unexplained bit-error every 10TB or so
> > when pulling the data from the tape-library again. That was
> > a few years back, but still.
> >
> 
> Thanks, I've never been very good at doing backups so any advice is
> appreciated.
> 
> > > > > Anyways, thanks for your help.
> > > > >
> > > >
> > > > Just to make sure: You _have_ read the FAQ?
> > >
> > > Yes I have read the FAQ.
> >
> > Good!
> >
> > > I guess I am just a little nervous about this project.
> >
> > Understandable.
> >
> > > This is my first time working with encryption and 50TB of
> > > critical client data is at stake. Just being thorough and making
> sure
> > I
> > > fully understand everything.
> >
> > And that is all you can do. I think you are approaching this
> > perfectly right. I take it the 50TB are alredy on XFS+LVM
> > at this time and you are just adding the encryption layer?
> 
> Currently the data is spread across multiple ext3 volumes of 8TB or
> less
> and is a pain to manage. That is the reason we are moving to XFS. The
> encryption is currently being handled by a Brocade switch blade. The
> encryption blade caused us major issues last year, resulting in a
> multi-day outage. That is the why we are moving to LUKS.
> 
> >
> > Make sure to use XTS mode or plain64 with ESSIV for these
> > volume sizes.
> 
> I don't know what this means. Haha. But I will look it up.
> 
> >
> > And let us know what your expeiences are!
> >
> 
> Sure thing. This project has been my first exposure to XFS, LUKS and
> clustering. And I'm the main Linux guy on my team. I've been working
> with Red Hat, but the solution architect laughed when he saw what we
> were trying to do. Haha. But he's been doing research and talking to
> his
> Red Hat contacts. And you guys have helped out tremendously! Thank you!
> 
> Jacob
> 
> > Arno
> > --
> > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
> > arno@wagner.name
> > GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50
> 1E25
> > 338F
> > ----
> > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans
> >
> > If it's in the news, don't worry about it.  The very definition of
> > "news" is "something that hardly ever happens." -- Bruce Schneier
> > _______________________________________________
> > dm-crypt mailing list
> > dm-crypt@saout.de
> > http://www.saout.de/mailman/listinfo/dm-crypt