* [dm-crypt] LUKS in failover cluster @ 2011-10-08 0:54 Sohl, Jacob (LNG-SEA) 2011-10-08 4:52 ` Arno Wagner 0 siblings, 1 reply; 9+ messages in thread From: Sohl, Jacob (LNG-SEA) @ 2011-10-08 0:54 UTC (permalink / raw) To: dm-crypt [-- Attachment #1: Type: text/plain, Size: 1808 bytes --] Hi all, I've been working on a design for an encrypted fileserver using RHEL6.x. On a single server the stack is pretty simple: SAN LUNs > LUKS > LVM > XFS > Samba Server But I would like to have a second node for High-Availability failover (SAN storage is available to both nodes). I'm looking at Red Hat Cluster Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM, XFS and Samba resources. In the event of node failure, it will migrate all resources to the healthy node. But the resources are only available if the SAN volumes are decrypted: cryptsetup luksOpen /dev/sdc1 crypt_vol Is it possible to have the raw volumes decrypted on both systems, maybe during boot. So the LUKS device (/dev/mapper/crypt_vol) will be available on the backup node in the event of primary node failure. The other resources - LVM, XFS, Samba - would only be on one node at a time, so no filesystem access from the passive node. If this is not possible then can you suggest another solution? Also, scalability is a requirement in my design, hence XFS. I was thinking I needed to use multiple LUKS PVs in LVM to grow the filesystem. But I would end up with multiple LUKS devices to keep track of. I recently found out that LUKS can resize. Would it be better to create one LUKS device on top of LVM? Then create a filesystem on that? (Though that would affect resource dependencies.) But basically: SAN LUNs > LVM > LUKS > XFS > Samba Server Other people will be accessing/managing this system, so I want manageability through simplicity. Don't want to have the wrong volumes (re)encrypted, headers damaged, etc. Anyways, thanks for your help. Jacob Sohl | Systems Engineer Applied Discovery(r) Mobile: 360-620-2695 [-- Attachment #2: Type: text/html, Size: 16082 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-08 0:54 [dm-crypt] LUKS in failover cluster Sohl, Jacob (LNG-SEA) @ 2011-10-08 4:52 ` Arno Wagner 2011-10-08 7:50 ` Milan Broz 2011-10-11 23:34 ` Sohl, Jacob (LNG-SEA) 0 siblings, 2 replies; 9+ messages in thread From: Arno Wagner @ 2011-10-08 4:52 UTC (permalink / raw) To: dm-crypt Hi, On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote: > Hi all, > > I've been working on a design for an encrypted fileserver using RHEL6.x. > On a single server the stack is pretty simple: > > SAN LUNs > LUKS > LVM > XFS > Samba Server > > > > But I would like to have a second node for High-Availability failover > (SAN storage is available to both nodes). I'm looking at Red Hat Cluster > Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM, > XFS and Samba resources. In the event of node failure, it will migrate > all resources to the healthy node. But the resources are only available > if the SAN volumes are decrypted: > > cryptsetup luksOpen /dev/sdc1 crypt_vol > > > > Is it possible to have the raw volumes decrypted on both systems, maybe > during boot. So the LUKS device (/dev/mapper/crypt_vol) will be > available on the backup node in the event of primary node failure. The > other resources - LVM, XFS, Samba - would only be on one node at a time, > so no filesystem access from the passive node. If this is not possible > then can you suggest another solution? You can map ("decrypt") the devices and never use them. The LVM/mount/whatever is completely optional. In fact the mapper tool (cryptsetup) does nothing except to decrypt the raw encrypted device to a raw decrypted device under /dev/mapper/. > Also, scalability is a requirement in my design, hence XFS. I was > thinking I needed to use multiple LUKS PVs in LVM to grow the > filesystem. But I would end up with multiple LUKS devices to keep track > of. I recently found out that LUKS can resize. LUKS _cannot_ resize. The thing is that LUKS does not care about device size. So if you enlarge a device/partition with an intact LUKS header, "cryptsetup luksOpen" will just map the larger device. If you have multiple devices, you can "slave" the additional ones to a "master" device using something like "decrypt_derived". This just takes the master key from the opened "master" container, runns it though a hash and uses this as key for the "slave" device. > Would it be better to > create one LUKS device on top of LVM? Then create a filesystem on that? > (Though that would affect resource dependencies.) Sre you sure you need LVM at all? > But basically: > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > Other people will be accessing/managing this system, so I want > manageability through simplicity. Hence the question whether you actually need LVM. It strikes me that typically LVM is not needed and just complicates matters. > Don't want to have the wrong volumes > (re)encrypted, headers damaged, etc. I think this setup is already too complicated for most people to manage. A full backup should be part of your plans. Resize with backup is actually easier, as you can just backup, kill everything and do a clean new config. > Anyways, thanks for your help. > Just to make sure: You _have_ read the FAQ? Arno > > Jacob Sohl | Systems Engineer > > Applied Discovery(r) > > Mobile: 360-620-2695 > > > > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-08 4:52 ` Arno Wagner @ 2011-10-08 7:50 ` Milan Broz 2011-10-12 0:06 ` Sohl, Jacob (LNG-SEA) 2011-10-11 23:34 ` Sohl, Jacob (LNG-SEA) 1 sibling, 1 reply; 9+ messages in thread From: Milan Broz @ 2011-10-08 7:50 UTC (permalink / raw) To: dm-crypt On 10/08/2011 06:52 AM, Arno Wagner wrote: >> Also, scalability is a requirement in my design, hence XFS. I was >> thinking I needed to use multiple LUKS PVs in LVM to grow the >> filesystem. But I would end up with multiple LUKS devices to keep track >> of. I recently found out that LUKS can resize. > > LUKS _cannot_ resize. The thing is that LUKS does not care about > device size. So if you enlarge a device/partition with an intact > LUKS header, "cryptsetup luksOpen" will just map the larger > device. yes, but crytsetup resize is designed that it it supports open (active) LUKS containers. You do not need to activate/deactivate, resize operation is enough. > If you have multiple devices, you can "slave" the additional ones > to a "master" device using something like "decrypt_derived". > This just takes the master key from the opened "master" container, > runns it though a hash and uses this as key for the "slave" > device. > >> Would it be better to >> create one LUKS device on top of LVM? Then create a filesystem on that? >> (Though that would affect resource dependencies.) > > Sre you sure you need LVM at all? LVM is very useful in such configurations, you can dynamically manage LVs over encrypted storage. Scripts for managing such stack is not complicated but of course it is yet another layer to care of. Because you mentioned cluster, I will just repeat - if you manage it in HA LVM style (the device stack is always active exclusively only on one node) it will work. (You just need to create own resource for cluster suite - add LUKS to the stack. So cluster suite will handle exclusivity and failover itself.) For using LUKS in cluster for shared storage (visible to all nodes) - if encryption (mapping) runs in server and plaintext device is exported and shared to nodes (using iscsi + clvm or so) this will work - if you want to run dm-crypt on every node (for the same device, IOW export ciphertext device) this is not supported. (I think that I said on IRC that with proper used cluster fs and flush requests this may work - but it is not tested. The problem is of course with internal crypto queues so without flushing these queues different nodes will see different storage content. And moreover, there is no cluster infrastructure like clvmd to manage such devices consistently.) Milan ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-08 7:50 ` Milan Broz @ 2011-10-12 0:06 ` Sohl, Jacob (LNG-SEA) 0 siblings, 0 replies; 9+ messages in thread From: Sohl, Jacob (LNG-SEA) @ 2011-10-12 0:06 UTC (permalink / raw) To: Milan Broz, dm-crypt > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Milan Broz > Sent: Saturday, October 08, 2011 12:51 AM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > > On 10/08/2011 06:52 AM, Arno Wagner wrote: > >> Also, scalability is a requirement in my design, hence XFS. I was > >> thinking I needed to use multiple LUKS PVs in LVM to grow the > >> filesystem. But I would end up with multiple LUKS devices to keep > track > >> of. I recently found out that LUKS can resize. > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > device size. So if you enlarge a device/partition with an intact > > LUKS header, "cryptsetup luksOpen" will just map the larger > > device. > > yes, but crytsetup resize is designed that it it supports open > (active) LUKS containers. You do not need to activate/deactivate, > resize operation is enough. > > > If you have multiple devices, you can "slave" the additional ones > > to a "master" device using something like "decrypt_derived". > > This just takes the master key from the opened "master" container, > > runns it though a hash and uses this as key for the "slave" > > device. > > > >> Would it be better to > >> create one LUKS device on top of LVM? Then create a filesystem on > that? > >> (Though that would affect resource dependencies.) > > > > Sre you sure you need LVM at all? > > LVM is very useful in such configurations, you can dynamically manage > LVs over encrypted storage. > Scripts for managing such stack is not complicated but of course > it is yet another layer to care of. > > Because you mentioned cluster, I will just repeat > > - if you manage it in HA LVM style (the device > stack is always active exclusively only on one node) it will work. > (You just need to create own resource for cluster suite - add LUKS > to the stack. So cluster suite will handle exclusivity and failover > itself.) ^ HA LVM is what I will be doing. > > For using LUKS in cluster for shared storage (visible to all nodes) > - if encryption (mapping) runs in server and plaintext device is > exported > and shared to nodes (using iscsi + clvm or so) this will work > > - if you want to run dm-crypt on every node (for the same device, > IOW export ciphertext device) this is not supported. > > (I think that I said on IRC that with proper used cluster fs and > flush requests this may work - but it is not tested. The problem is of > course > with internal crypto queues so without flushing these queues different > nodes will see different storage content. And moreover, there is no > cluster > infrastructure like clvmd to manage such devices consistently.) Thank you for clarifying this for me, I didn't really understand in IRC what you meant about "flush requests". I'm not a programmer and don't really know low-level hardware stuff that much. It's been a lot of stuff for me to learn. > > Milan > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-08 4:52 ` Arno Wagner 2011-10-08 7:50 ` Milan Broz @ 2011-10-11 23:34 ` Sohl, Jacob (LNG-SEA) 2011-10-12 3:13 ` Arno Wagner 1 sibling, 1 reply; 9+ messages in thread From: Sohl, Jacob (LNG-SEA) @ 2011-10-11 23:34 UTC (permalink / raw) To: Arno Wagner, dm-crypt > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Friday, October 07, 2011 9:53 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > Hi, > > On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote: > > Hi all, > > > > I've been working on a design for an encrypted fileserver using > RHEL6.x. > > On a single server the stack is pretty simple: > > > > SAN LUNs > LUKS > LVM > XFS > Samba Server > > > > > > > > But I would like to have a second node for High-Availability failover > > (SAN storage is available to both nodes). I'm looking at Red Hat > Cluster > > Suite with corosyn, rgmanager. rgmanager has the ability to manage > LVM, > > XFS and Samba resources. In the event of node failure, it will > migrate > > all resources to the healthy node. But the resources are only > available > > if the SAN volumes are decrypted: > > > > cryptsetup luksOpen /dev/sdc1 crypt_vol > > > > > > > > Is it possible to have the raw volumes decrypted on both systems, > maybe > > during boot. So the LUKS device (/dev/mapper/crypt_vol) will be > > available on the backup node in the event of primary node failure. > The > > other resources - LVM, XFS, Samba - would only be on one node at a > time, > > so no filesystem access from the passive node. If this is not > possible > > then can you suggest another solution? > > You can map ("decrypt") the devices and never use them. The > LVM/mount/whatever is completely optional. > > In fact the mapper tool (cryptsetup) does nothing except > to decrypt the raw encrypted device to a raw decrypted device > under /dev/mapper/. > Ok, thank you. That seemed logical, but I just recently started working with encryption so I'm still learning. > > > Also, scalability is a requirement in my design, hence XFS. I was > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > filesystem. But I would end up with multiple LUKS devices to keep > track > > of. I recently found out that LUKS can resize. > > LUKS _cannot_ resize. The thing is that LUKS does not care about > device size. So if you enlarge a device/partition with an intact > LUKS header, "cryptsetup luksOpen" will just map the larger > device. > > If you have multiple devices, you can "slave" the additional ones > to a "master" device using something like "decrypt_derived". > This just takes the master key from the opened "master" container, > runns it though a hash and uses this as key for the "slave" > device. > > > Would it be better to > > create one LUKS device on top of LVM? Then create a filesystem on > that? > > (Though that would affect resource dependencies.) > > Sre you sure you need LVM at all? Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins can/will only attach storage in 4TB increments. So I have to combine all of the LUNS into one Logical Volume in order to create a 50TB+ XFS filesystem. That's why I ask about encrypting multiple physical LUNS vs a single Logical Volume. > > > But basically: > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > Other people will be accessing/managing this system, so I want > > manageability through simplicity. > > Hence the question whether you actually need LVM. > It strikes me that typically LVM is not needed and > just complicates matters. > > > Don't want to have the wrong volumes > > (re)encrypted, headers damaged, etc. > > I think this setup is already too complicated for most people > to manage. A full backup should be part of your plans. > Resize with backup is actually easier, as you can just > backup, kill everything and do a clean new config. > I will be making full backups with xfsdump. But it doesn't seem practical to be moving 50TB around. Both LVM and XFS allow for designed for online resize. I was planning to "pvcreate" an encrypted LUN then expand the Logical Volume and Filesystem. > > Anyways, thanks for your help. > > > > Just to make sure: You _have_ read the FAQ? Yes I have read the FAQ. I guess I am just a little nervous about this project. This is my first time working with encryption and 50TB of critical client data is at stake. Just being thorough and making sure I fully understand everything. Thanks again, Jacob Sohl > > Arno > > > > > Jacob Sohl | Systems Engineer > > > > Applied Discovery(r) > > > > Mobile: 360-620-2695 > > > > > > > > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@saout.de > > http://www.saout.de/mailman/listinfo/dm-crypt > > > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans > > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-11 23:34 ` Sohl, Jacob (LNG-SEA) @ 2011-10-12 3:13 ` Arno Wagner 2011-10-12 19:20 ` Sohl, Jacob (LNG-SEA) 0 siblings, 1 reply; 9+ messages in thread From: Arno Wagner @ 2011-10-12 3:13 UTC (permalink / raw) To: dm-crypt On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: [...] > > You can map ("decrypt") the devices and never use them. The > > LVM/mount/whatever is completely optional. > > > > In fact the mapper tool (cryptsetup) does nothing except > > to decrypt the raw encrypted device to a raw decrypted device > > under /dev/mapper/. > > > > Ok, thank you. That seemed logical, but I just recently started working > with encryption so I'm still learning. That is fine. Making sure now will safe you grief later. > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > filesystem. But I would end up with multiple LUKS devices to keep > > track > > > of. I recently found out that LUKS can resize. > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > device size. So if you enlarge a device/partition with an intact > > LUKS header, "cryptsetup luksOpen" will just map the larger > > device. > > > > If you have multiple devices, you can "slave" the additional ones > > to a "master" device using something like "decrypt_derived". > > This just takes the master key from the opened "master" container, > > runns it though a hash and uses this as key for the "slave" > > device. > > > > > Would it be better to > > > create one LUKS device on top of LVM? Then create a filesystem on > > that? > > > (Though that would affect resource dependencies.) > > > > Sre you sure you need LVM at all? > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > can/will only attach storage in 4TB increments. So I have to combine all > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > filesystem. That's why I ask about encrypting multiple physical LUNS vs > a single Logical Volume. 50TB? Ok, that is a bit larger. I did have such a dataset for research a few years back on tape. Took several days days to get 15TB of it to disk for my largest measurement. Well, yes, I think LVM is the best way here. > > > > > But basically: > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > manageability through simplicity. > > > > Hence the question whether you actually need LVM. > > It strikes me that typically LVM is not needed and > > just complicates matters. > > > > > Don't want to have the wrong volumes > > > (re)encrypted, headers damaged, etc. > > > > I think this setup is already too complicated for most people > > to manage. A full backup should be part of your plans. > > Resize with backup is actually easier, as you can just > > backup, kill everything and do a clean new config. > > > > I will be making full backups with xfsdump. But it doesn't seem > practical to be moving 50TB around. Both LVM and XFS allow for designed > for online resize. I was planning to "pvcreate" an encrypted LUN then > expand the Logical Volume and Filesystem. For that volume, run a full compare as well. I found that with good hardware, and an IBM Tape-Library I still had something like 1 unexplained bit-error every 10TB or so when pulling the data from the tape-library again. That was a few years back, but still. > > > Anyways, thanks for your help. > > > > > > > Just to make sure: You _have_ read the FAQ? > > Yes I have read the FAQ. Good! > I guess I am just a little nervous about this project. Understandable. > This is my first time working with encryption and 50TB of > critical client data is at stake. Just being thorough and making sure I > fully understand everything. And that is all you can do. I think you are approaching this perfectly right. I take it the 50TB are alredy on XFS+LVM at this time and you are just adding the encryption layer? Make sure to use XTS mode or plain64 with ESSIV for these volume sizes. And let us know what your expeiences are! Arno -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-12 3:13 ` Arno Wagner @ 2011-10-12 19:20 ` Sohl, Jacob (LNG-SEA) 2011-10-12 19:54 ` Scott McCarty 0 siblings, 1 reply; 9+ messages in thread From: Sohl, Jacob (LNG-SEA) @ 2011-10-12 19:20 UTC (permalink / raw) To: Arno Wagner, dm-crypt; +Cc: Scott McCarty, LNG-SEA Systems Engineers ADI > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Tuesday, October 11, 2011 8:13 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: > [...] > > > You can map ("decrypt") the devices and never use them. The > > > LVM/mount/whatever is completely optional. > > > > > > In fact the mapper tool (cryptsetup) does nothing except > > > to decrypt the raw encrypted device to a raw decrypted device > > > under /dev/mapper/. > > > > > > > Ok, thank you. That seemed logical, but I just recently started > working > > with encryption so I'm still learning. > > That is fine. Making sure now will safe you grief later. > > > > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > > filesystem. But I would end up with multiple LUKS devices to keep > > > track > > > > of. I recently found out that LUKS can resize. > > > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > > device size. So if you enlarge a device/partition with an intact > > > LUKS header, "cryptsetup luksOpen" will just map the larger > > > device. > > > > > > If you have multiple devices, you can "slave" the additional ones > > > to a "master" device using something like "decrypt_derived". > > > This just takes the master key from the opened "master" container, > > > runns it though a hash and uses this as key for the "slave" > > > device. > > > > > > > Would it be better to > > > > create one LUKS device on top of LVM? Then create a filesystem on > > > that? > > > > (Though that would affect resource dependencies.) > > > > > > Sre you sure you need LVM at all? > > > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > > can/will only attach storage in 4TB increments. So I have to combine > all > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > > filesystem. That's why I ask about encrypting multiple physical LUNS > vs > > a single Logical Volume. > > 50TB? Ok, that is a bit larger. I did have such a dataset > for research a few years back on tape. Took several days > days to get 15TB of it to disk for my largest measurement. > > Well, yes, I think LVM is the best way here. > > > > > > > > But basically: > > > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > > manageability through simplicity. > > > > > > Hence the question whether you actually need LVM. > > > It strikes me that typically LVM is not needed and > > > just complicates matters. > > > > > > > Don't want to have the wrong volumes > > > > (re)encrypted, headers damaged, etc. > > > > > > I think this setup is already too complicated for most people > > > to manage. A full backup should be part of your plans. > > > Resize with backup is actually easier, as you can just > > > backup, kill everything and do a clean new config. > > > > > > > I will be making full backups with xfsdump. But it doesn't seem > > practical to be moving 50TB around. Both LVM and XFS allow for > designed > > for online resize. I was planning to "pvcreate" an encrypted LUN then > > expand the Logical Volume and Filesystem. > > For that volume, run a full compare as well. I found that > with good hardware, and an IBM Tape-Library I still had > something like 1 unexplained bit-error every 10TB or so > when pulling the data from the tape-library again. That was > a few years back, but still. > Thanks, I've never been very good at doing backups so any advice is appreciated. > > > > Anyways, thanks for your help. > > > > > > > > > > Just to make sure: You _have_ read the FAQ? > > > > Yes I have read the FAQ. > > Good! > > > I guess I am just a little nervous about this project. > > Understandable. > > > This is my first time working with encryption and 50TB of > > critical client data is at stake. Just being thorough and making sure > I > > fully understand everything. > > And that is all you can do. I think you are approaching this > perfectly right. I take it the 50TB are alredy on XFS+LVM > at this time and you are just adding the encryption layer? Currently the data is spread across multiple ext3 volumes of 8TB or less and is a pain to manage. That is the reason we are moving to XFS. The encryption is currently being handled by a Brocade switch blade. The encryption blade caused us major issues last year, resulting in a multi-day outage. That is the why we are moving to LUKS. > > Make sure to use XTS mode or plain64 with ESSIV for these > volume sizes. I don't know what this means. Haha. But I will look it up. > > And let us know what your expeiences are! > Sure thing. This project has been my first exposure to XFS, LUKS and clustering. And I'm the main Linux guy on my team. I've been working with Red Hat, but the solution architect laughed when he saw what we were trying to do. Haha. But he's been doing research and talking to his Red Hat contacts. And you guys have helped out tremendously! Thank you! Jacob > Arno > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans > > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-12 19:20 ` Sohl, Jacob (LNG-SEA) @ 2011-10-12 19:54 ` Scott McCarty 2011-10-12 20:26 ` Sohl, Jacob (LNG-SEA) 0 siblings, 1 reply; 9+ messages in thread From: Scott McCarty @ 2011-10-12 19:54 UTC (permalink / raw) To: Jacob Sohl (LNG-SEA); +Cc: dm-crypt, Arno Wagner, LNG-SEA Systems Engineers ADI That was a good laugh, maybe a chuckle even :-) Scott McCarty, RHCE, RHCSA, LPIC-1 Solutions Architect Email: smccarty@redhat.com Phone: 312-660-3535 Web: http://people.redhat.com/smccarty ----- Original Message ----- From: "Jacob Sohl (LNG-SEA)" <jacob.sohl@applieddiscovery.com> To: "Arno Wagner" <arno@wagner.name>, dm-crypt@saout.de Cc: "LNG-SEA Systems Engineers ADI" <LNG-SEASystemsEngineersADI@lexisnexis.com>, "Scott McCarty" <smccarty@redhat.com> Sent: Wednesday, October 12, 2011 3:20:51 PM Subject: RE: [dm-crypt] LUKS in failover cluster > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Tuesday, October 11, 2011 8:13 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: > [...] > > > You can map ("decrypt") the devices and never use them. The > > > LVM/mount/whatever is completely optional. > > > > > > In fact the mapper tool (cryptsetup) does nothing except > > > to decrypt the raw encrypted device to a raw decrypted device > > > under /dev/mapper/. > > > > > > > Ok, thank you. That seemed logical, but I just recently started > working > > with encryption so I'm still learning. > > That is fine. Making sure now will safe you grief later. > > > > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > > filesystem. But I would end up with multiple LUKS devices to keep > > > track > > > > of. I recently found out that LUKS can resize. > > > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > > device size. So if you enlarge a device/partition with an intact > > > LUKS header, "cryptsetup luksOpen" will just map the larger > > > device. > > > > > > If you have multiple devices, you can "slave" the additional ones > > > to a "master" device using something like "decrypt_derived". > > > This just takes the master key from the opened "master" container, > > > runns it though a hash and uses this as key for the "slave" > > > device. > > > > > > > Would it be better to > > > > create one LUKS device on top of LVM? Then create a filesystem on > > > that? > > > > (Though that would affect resource dependencies.) > > > > > > Sre you sure you need LVM at all? > > > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > > can/will only attach storage in 4TB increments. So I have to combine > all > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > > filesystem. That's why I ask about encrypting multiple physical LUNS > vs > > a single Logical Volume. > > 50TB? Ok, that is a bit larger. I did have such a dataset > for research a few years back on tape. Took several days > days to get 15TB of it to disk for my largest measurement. > > Well, yes, I think LVM is the best way here. > > > > > > > > But basically: > > > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > > manageability through simplicity. > > > > > > Hence the question whether you actually need LVM. > > > It strikes me that typically LVM is not needed and > > > just complicates matters. > > > > > > > Don't want to have the wrong volumes > > > > (re)encrypted, headers damaged, etc. > > > > > > I think this setup is already too complicated for most people > > > to manage. A full backup should be part of your plans. > > > Resize with backup is actually easier, as you can just > > > backup, kill everything and do a clean new config. > > > > > > > I will be making full backups with xfsdump. But it doesn't seem > > practical to be moving 50TB around. Both LVM and XFS allow for > designed > > for online resize. I was planning to "pvcreate" an encrypted LUN then > > expand the Logical Volume and Filesystem. > > For that volume, run a full compare as well. I found that > with good hardware, and an IBM Tape-Library I still had > something like 1 unexplained bit-error every 10TB or so > when pulling the data from the tape-library again. That was > a few years back, but still. > Thanks, I've never been very good at doing backups so any advice is appreciated. > > > > Anyways, thanks for your help. > > > > > > > > > > Just to make sure: You _have_ read the FAQ? > > > > Yes I have read the FAQ. > > Good! > > > I guess I am just a little nervous about this project. > > Understandable. > > > This is my first time working with encryption and 50TB of > > critical client data is at stake. Just being thorough and making sure > I > > fully understand everything. > > And that is all you can do. I think you are approaching this > perfectly right. I take it the 50TB are alredy on XFS+LVM > at this time and you are just adding the encryption layer? Currently the data is spread across multiple ext3 volumes of 8TB or less and is a pain to manage. That is the reason we are moving to XFS. The encryption is currently being handled by a Brocade switch blade. The encryption blade caused us major issues last year, resulting in a multi-day outage. That is the why we are moving to LUKS. > > Make sure to use XTS mode or plain64 with ESSIV for these > volume sizes. I don't know what this means. Haha. But I will look it up. > > And let us know what your expeiences are! > Sure thing. This project has been my first exposure to XFS, LUKS and clustering. And I'm the main Linux guy on my team. I've been working with Red Hat, but the solution architect laughed when he saw what we were trying to do. Haha. But he's been doing research and talking to his Red Hat contacts. And you guys have helped out tremendously! Thank you! Jacob > Arno > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans > > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [dm-crypt] LUKS in failover cluster 2011-10-12 19:54 ` Scott McCarty @ 2011-10-12 20:26 ` Sohl, Jacob (LNG-SEA) 0 siblings, 0 replies; 9+ messages in thread From: Sohl, Jacob (LNG-SEA) @ 2011-10-12 20:26 UTC (permalink / raw) To: Scott McCarty; +Cc: dm-crypt, Arno Wagner, LNG-SEA Systems Engineers ADI Yeah, it was a "crazy-fun-project" laugh. An opportunity to learn a lot and create something cool. > -----Original Message----- > From: Scott McCarty [mailto:smccarty@redhat.com] > Sent: Wednesday, October 12, 2011 12:55 PM > To: Sohl, Jacob (LNG-SEA) > Cc: LNG-SEA Systems Engineers ADI; Arno Wagner; dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > That was a good laugh, maybe a chuckle even :-) > > Scott McCarty, RHCE, RHCSA, LPIC-1 > Solutions Architect > Email: smccarty@redhat.com > Phone: 312-660-3535 > Web: http://people.redhat.com/smccarty > > > ----- Original Message ----- > From: "Jacob Sohl (LNG-SEA)" <jacob.sohl@applieddiscovery.com> > To: "Arno Wagner" <arno@wagner.name>, dm-crypt@saout.de > Cc: "LNG-SEA Systems Engineers ADI" <LNG- > SEASystemsEngineersADI@lexisnexis.com>, "Scott McCarty" > <smccarty@redhat.com> > Sent: Wednesday, October 12, 2011 3:20:51 PM > Subject: RE: [dm-crypt] LUKS in failover cluster > > > > > -----Original Message----- > > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > > Behalf Of Arno Wagner > > Sent: Tuesday, October 11, 2011 8:13 PM > > To: dm-crypt@saout.de > > Subject: Re: [dm-crypt] LUKS in failover cluster > > > > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) > wrote: > > [...] > > > > You can map ("decrypt") the devices and never use them. The > > > > LVM/mount/whatever is completely optional. > > > > > > > > In fact the mapper tool (cryptsetup) does nothing except > > > > to decrypt the raw encrypted device to a raw decrypted device > > > > under /dev/mapper/. > > > > > > > > > > Ok, thank you. That seemed logical, but I just recently started > > working > > > with encryption so I'm still learning. > > > > That is fine. Making sure now will safe you grief later. > > > > > > > > > > > Also, scalability is a requirement in my design, hence XFS. I > was > > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > > > filesystem. But I would end up with multiple LUKS devices to > keep > > > > track > > > > > of. I recently found out that LUKS can resize. > > > > > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > > > device size. So if you enlarge a device/partition with an intact > > > > LUKS header, "cryptsetup luksOpen" will just map the larger > > > > device. > > > > > > > > If you have multiple devices, you can "slave" the additional ones > > > > to a "master" device using something like "decrypt_derived". > > > > This just takes the master key from the opened "master" > container, > > > > runns it though a hash and uses this as key for the "slave" > > > > device. > > > > > > > > > Would it be better to > > > > > create one LUKS device on top of LVM? Then create a filesystem > on > > > > that? > > > > > (Though that would affect resource dependencies.) > > > > > > > > Sre you sure you need LVM at all? > > > > > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN > admins > > > can/will only attach storage in 4TB increments. So I have to > combine > > all > > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > > > filesystem. That's why I ask about encrypting multiple physical > LUNS > > vs > > > a single Logical Volume. > > > > 50TB? Ok, that is a bit larger. I did have such a dataset > > for research a few years back on tape. Took several days > > days to get 15TB of it to disk for my largest measurement. > > > > Well, yes, I think LVM is the best way here. > > > > > > > > > > > But basically: > > > > > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > > > manageability through simplicity. > > > > > > > > Hence the question whether you actually need LVM. > > > > It strikes me that typically LVM is not needed and > > > > just complicates matters. > > > > > > > > > Don't want to have the wrong volumes > > > > > (re)encrypted, headers damaged, etc. > > > > > > > > I think this setup is already too complicated for most people > > > > to manage. A full backup should be part of your plans. > > > > Resize with backup is actually easier, as you can just > > > > backup, kill everything and do a clean new config. > > > > > > > > > > I will be making full backups with xfsdump. But it doesn't seem > > > practical to be moving 50TB around. Both LVM and XFS allow for > > designed > > > for online resize. I was planning to "pvcreate" an encrypted LUN > then > > > expand the Logical Volume and Filesystem. > > > > For that volume, run a full compare as well. I found that > > with good hardware, and an IBM Tape-Library I still had > > something like 1 unexplained bit-error every 10TB or so > > when pulling the data from the tape-library again. That was > > a few years back, but still. > > > > Thanks, I've never been very good at doing backups so any advice is > appreciated. > > > > > > Anyways, thanks for your help. > > > > > > > > > > > > > Just to make sure: You _have_ read the FAQ? > > > > > > Yes I have read the FAQ. > > > > Good! > > > > > I guess I am just a little nervous about this project. > > > > Understandable. > > > > > This is my first time working with encryption and 50TB of > > > critical client data is at stake. Just being thorough and making > sure > > I > > > fully understand everything. > > > > And that is all you can do. I think you are approaching this > > perfectly right. I take it the 50TB are alredy on XFS+LVM > > at this time and you are just adding the encryption layer? > > Currently the data is spread across multiple ext3 volumes of 8TB or > less > and is a pain to manage. That is the reason we are moving to XFS. The > encryption is currently being handled by a Brocade switch blade. The > encryption blade caused us major issues last year, resulting in a > multi-day outage. That is the why we are moving to LUKS. > > > > > Make sure to use XTS mode or plain64 with ESSIV for these > > volume sizes. > > I don't know what this means. Haha. But I will look it up. > > > > > And let us know what your expeiences are! > > > > Sure thing. This project has been my first exposure to XFS, LUKS and > clustering. And I'm the main Linux guy on my team. I've been working > with Red Hat, but the solution architect laughed when he saw what we > were trying to do. Haha. But he's been doing research and talking to > his > Red Hat contacts. And you guys have helped out tremendously! Thank you! > > Jacob > > > Arno > > -- > > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > > arno@wagner.name > > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 > 1E25 > > 338F > > ---- > > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans > > > > If it's in the news, don't worry about it. The very definition of > > "news" is "something that hardly ever happens." -- Bruce Schneier > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@saout.de > > http://www.saout.de/mailman/listinfo/dm-crypt ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-10-12 20:45 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-10-08 0:54 [dm-crypt] LUKS in failover cluster Sohl, Jacob (LNG-SEA) 2011-10-08 4:52 ` Arno Wagner 2011-10-08 7:50 ` Milan Broz 2011-10-12 0:06 ` Sohl, Jacob (LNG-SEA) 2011-10-11 23:34 ` Sohl, Jacob (LNG-SEA) 2011-10-12 3:13 ` Arno Wagner 2011-10-12 19:20 ` Sohl, Jacob (LNG-SEA) 2011-10-12 19:54 ` Scott McCarty 2011-10-12 20:26 ` Sohl, Jacob (LNG-SEA)
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.