From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id sIs9mLnkVxzq for ; Sat, 8 Oct 2011 03:08:16 +0200 (CEST) Received: from mx03.lexisnexis.com (mx03.lexisnexis.com [198.185.18.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Sat, 8 Oct 2011 03:08:15 +0200 (CEST) Received: from lngseaexcp002.legal.regn.net ([10.215.36.22]) by mailgate2.lexis-nexis.com (8.12.11.20060308/8.12.11) with ESMTP id p980wOv3003550 for ; Fri, 7 Oct 2011 20:58:24 -0400 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01CC8554.E12E5982" Date: Fri, 7 Oct 2011 17:54:48 -0700 Message-ID: From: "Sohl, Jacob (LNG-SEA)" Subject: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dm-crypt@saout.de This is a multi-part message in MIME format. ------_=_NextPart_001_01CC8554.E12E5982 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi all, I've been working on a design for an encrypted fileserver using RHEL6.x. On a single server the stack is pretty simple: SAN LUNs > LUKS > LVM > XFS > Samba Server =20 But I would like to have a second node for High-Availability failover (SAN storage is available to both nodes). I'm looking at Red Hat Cluster Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM, XFS and Samba resources. In the event of node failure, it will migrate all resources to the healthy node. But the resources are only available if the SAN volumes are decrypted: cryptsetup luksOpen /dev/sdc1 crypt_vol =20 Is it possible to have the raw volumes decrypted on both systems, maybe during boot. So the LUKS device (/dev/mapper/crypt_vol) will be available on the backup node in the event of primary node failure. The other resources - LVM, XFS, Samba - would only be on one node at a time, so no filesystem access from the passive node. If this is not possible then can you suggest another solution? =20 Also, scalability is a requirement in my design, hence XFS. I was thinking I needed to use multiple LUKS PVs in LVM to grow the filesystem. But I would end up with multiple LUKS devices to keep track of. I recently found out that LUKS can resize. Would it be better to create one LUKS device on top of LVM? Then create a filesystem on that? (Though that would affect resource dependencies.) But basically: SAN LUNs > LVM > LUKS > XFS > Samba Server =20 Other people will be accessing/managing this system, so I want manageability through simplicity. Don't want to have the wrong volumes (re)encrypted, headers damaged, etc. Anyways, thanks for your help. =20 Jacob Sohl | Systems Engineer Applied Discovery(r) Mobile: 360-620-2695 =20 ------_=_NextPart_001_01CC8554.E12E5982 Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi all,

I’ve been = working on a design for an encrypted fileserver using RHEL6.x. On a = single server the stack is pretty simple:

SAN LUNs > LUKS = > LVM > XFS > Samba Server

 

But I would like to = have a second node for High-Availability failover (SAN storage is = available to both nodes). I’m looking at Red Hat Cluster Suite = with corosyn, rgmanager. rgmanager has the ability to manage LVM, XFS = and Samba resources. In the event of node failure, it will migrate all = resources to the healthy node. But the resources are only available if = the SAN volumes are decrypted:

cryptsetup luksOpen = /dev/sdc1 crypt_vol

 

Is it possible to = have the raw volumes decrypted on both systems, maybe during boot. So = the LUKS device (/dev/mapper/crypt_vol) will be available on the backup = node in the event of primary node failure. The other resources - LVM, = XFS, Samba – would only be on one node at a time, so no filesystem = access from the passive node. If this is not possible then can you = suggest another solution?

 

Also, scalability is = a requirement in my design, hence XFS. I was thinking I needed to use = multiple LUKS PVs in LVM to grow the filesystem. But I would end up with = multiple LUKS devices to keep track of. I recently found out that LUKS = can resize. Would it be better to create one LUKS device on top of LVM? = Then create a filesystem on that? (Though that would affect resource = dependencies.)

But basically:

SAN LUNs > LVM = > LUKS > XFS > Samba Server

 

Other people will be = accessing/managing this system, so I want manageability through = simplicity. Don’t want to have the wrong volumes (re)encrypted, = headers damaged, etc.

Anyways, thanks for your = help.

 

Jacob Sohl  = |  Systems Engineer

Applied = Discovery®

Mobile: 360-620-2695

 

------_=_NextPart_001_01CC8554.E12E5982-- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 5YQtLAuUSKP3 for ; Sat, 8 Oct 2011 06:52:33 +0200 (CEST) Received: from v4.tansi.org (ns.km33513-03.keymachine.de [87.118.94.3]) by mail.saout.de (Postfix) with ESMTP for ; Sat, 8 Oct 2011 06:52:33 +0200 (CEST) Received: from gatewagner.dyndns.org (84-74-163-71.dclient.hispeed.ch [84.74.163.71]) by v4.tansi.org (Postfix) with ESMTPA id 1CDC41404001 for ; Sat, 8 Oct 2011 06:52:33 +0200 (CEST) Date: Sat, 8 Oct 2011 06:52:32 +0200 From: Arno Wagner Message-ID: <20111008045232.GB23870@tansi.org> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dm-crypt@saout.de Hi, On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote: > Hi all, > > I've been working on a design for an encrypted fileserver using RHEL6.x. > On a single server the stack is pretty simple: > > SAN LUNs > LUKS > LVM > XFS > Samba Server > > > > But I would like to have a second node for High-Availability failover > (SAN storage is available to both nodes). I'm looking at Red Hat Cluster > Suite with corosyn, rgmanager. rgmanager has the ability to manage LVM, > XFS and Samba resources. In the event of node failure, it will migrate > all resources to the healthy node. But the resources are only available > if the SAN volumes are decrypted: > > cryptsetup luksOpen /dev/sdc1 crypt_vol > > > > Is it possible to have the raw volumes decrypted on both systems, maybe > during boot. So the LUKS device (/dev/mapper/crypt_vol) will be > available on the backup node in the event of primary node failure. The > other resources - LVM, XFS, Samba - would only be on one node at a time, > so no filesystem access from the passive node. If this is not possible > then can you suggest another solution? You can map ("decrypt") the devices and never use them. The LVM/mount/whatever is completely optional. In fact the mapper tool (cryptsetup) does nothing except to decrypt the raw encrypted device to a raw decrypted device under /dev/mapper/. > Also, scalability is a requirement in my design, hence XFS. I was > thinking I needed to use multiple LUKS PVs in LVM to grow the > filesystem. But I would end up with multiple LUKS devices to keep track > of. I recently found out that LUKS can resize. LUKS _cannot_ resize. The thing is that LUKS does not care about device size. So if you enlarge a device/partition with an intact LUKS header, "cryptsetup luksOpen" will just map the larger device. If you have multiple devices, you can "slave" the additional ones to a "master" device using something like "decrypt_derived". This just takes the master key from the opened "master" container, runns it though a hash and uses this as key for the "slave" device. > Would it be better to > create one LUKS device on top of LVM? Then create a filesystem on that? > (Though that would affect resource dependencies.) Sre you sure you need LVM at all? > But basically: > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > Other people will be accessing/managing this system, so I want > manageability through simplicity. Hence the question whether you actually need LVM. It strikes me that typically LVM is not needed and just complicates matters. > Don't want to have the wrong volumes > (re)encrypted, headers damaged, etc. I think this setup is already too complicated for most people to manage. A full backup should be part of your plans. Resize with backup is actually easier, as you can just backup, kill everything and do a clean new config. > Anyways, thanks for your help. > Just to make sure: You _have_ read the FAQ? Arno > > Jacob Sohl | Systems Engineer > > Applied Discovery(r) > > Mobile: 360-620-2695 > > > > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id bD_aqKtLRyAc for ; Sat, 8 Oct 2011 09:50:55 +0200 (CEST) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mail.saout.de (Postfix) with ESMTP for ; Sat, 8 Oct 2011 09:50:54 +0200 (CEST) Received: from int-mx02.intmail.prod.int.phx2.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p987orBR013154 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Sat, 8 Oct 2011 03:50:53 -0400 Received: from [10.36.7.238] (vpn1-7-238.ams2.redhat.com [10.36.7.238]) by int-mx02.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id p987opAN002763 for ; Sat, 8 Oct 2011 03:50:52 -0400 Message-ID: <4E9000DB.4000308@redhat.com> Date: Sat, 08 Oct 2011 09:50:51 +0200 From: Milan Broz MIME-Version: 1.0 References: <20111008045232.GB23870@tansi.org> In-Reply-To: <20111008045232.GB23870@tansi.org> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dm-crypt@saout.de On 10/08/2011 06:52 AM, Arno Wagner wrote: >> Also, scalability is a requirement in my design, hence XFS. I was >> thinking I needed to use multiple LUKS PVs in LVM to grow the >> filesystem. But I would end up with multiple LUKS devices to keep track >> of. I recently found out that LUKS can resize. > > LUKS _cannot_ resize. The thing is that LUKS does not care about > device size. So if you enlarge a device/partition with an intact > LUKS header, "cryptsetup luksOpen" will just map the larger > device. yes, but crytsetup resize is designed that it it supports open (active) LUKS containers. You do not need to activate/deactivate, resize operation is enough. > If you have multiple devices, you can "slave" the additional ones > to a "master" device using something like "decrypt_derived". > This just takes the master key from the opened "master" container, > runns it though a hash and uses this as key for the "slave" > device. > >> Would it be better to >> create one LUKS device on top of LVM? Then create a filesystem on that? >> (Though that would affect resource dependencies.) > > Sre you sure you need LVM at all? LVM is very useful in such configurations, you can dynamically manage LVs over encrypted storage. Scripts for managing such stack is not complicated but of course it is yet another layer to care of. Because you mentioned cluster, I will just repeat - if you manage it in HA LVM style (the device stack is always active exclusively only on one node) it will work. (You just need to create own resource for cluster suite - add LUKS to the stack. So cluster suite will handle exclusivity and failover itself.) For using LUKS in cluster for shared storage (visible to all nodes) - if encryption (mapping) runs in server and plaintext device is exported and shared to nodes (using iscsi + clvm or so) this will work - if you want to run dm-crypt on every node (for the same device, IOW export ciphertext device) this is not supported. (I think that I said on IRC that with proper used cluster fs and flush requests this may work - but it is not tested. The problem is of course with internal crypto queues so without flushing these queues different nodes will see different storage content. And moreover, there is no cluster infrastructure like clvmd to manage such devices consistently.) Milan From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3x2MubvwtVQt for ; Wed, 12 Oct 2011 01:34:27 +0200 (CEST) Received: from mx03.lexisnexis.com (mx03.lexisnexis.com [198.185.18.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Wed, 12 Oct 2011 01:34:26 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 11 Oct 2011 16:34:21 -0700 Message-ID: In-Reply-To: <20111008045232.GB23870@tansi.org> References: <20111008045232.GB23870@tansi.org> From: "Sohl, Jacob (LNG-SEA)" Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Arno Wagner , dm-crypt@saout.de > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Friday, October 07, 2011 9:53 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster >=20 > Hi, >=20 > On Fri, Oct 07, 2011 at 05:54:48PM -0700, Sohl, Jacob (LNG-SEA) wrote: > > Hi all, > > > > I've been working on a design for an encrypted fileserver using > RHEL6.x. > > On a single server the stack is pretty simple: > > > > SAN LUNs > LUKS > LVM > XFS > Samba Server > > > > > > > > But I would like to have a second node for High-Availability failover > > (SAN storage is available to both nodes). I'm looking at Red Hat > Cluster > > Suite with corosyn, rgmanager. rgmanager has the ability to manage > LVM, > > XFS and Samba resources. In the event of node failure, it will > migrate > > all resources to the healthy node. But the resources are only > available > > if the SAN volumes are decrypted: > > > > cryptsetup luksOpen /dev/sdc1 crypt_vol > > > > > > > > Is it possible to have the raw volumes decrypted on both systems, > maybe > > during boot. So the LUKS device (/dev/mapper/crypt_vol) will be > > available on the backup node in the event of primary node failure. > The > > other resources - LVM, XFS, Samba - would only be on one node at a > time, > > so no filesystem access from the passive node. If this is not > possible > > then can you suggest another solution? >=20 > You can map ("decrypt") the devices and never use them. The > LVM/mount/whatever is completely optional. >=20 > In fact the mapper tool (cryptsetup) does nothing except > to decrypt the raw encrypted device to a raw decrypted device > under /dev/mapper/. >=20 Ok, thank you. That seemed logical, but I just recently started working with encryption so I'm still learning. >=20 > > Also, scalability is a requirement in my design, hence XFS. I was > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > filesystem. But I would end up with multiple LUKS devices to keep > track > > of. I recently found out that LUKS can resize. >=20 > LUKS _cannot_ resize. The thing is that LUKS does not care about > device size. So if you enlarge a device/partition with an intact > LUKS header, "cryptsetup luksOpen" will just map the larger > device. >=20 > If you have multiple devices, you can "slave" the additional ones > to a "master" device using something like "decrypt_derived". > This just takes the master key from the opened "master" container, > runns it though a hash and uses this as key for the "slave" > device. >=20 > > Would it be better to > > create one LUKS device on top of LVM? Then create a filesystem on > that? > > (Though that would affect resource dependencies.) >=20 > Sre you sure you need LVM at all? Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins can/will only attach storage in 4TB increments. So I have to combine all of the LUNS into one Logical Volume in order to create a 50TB+ XFS filesystem. That's why I ask about encrypting multiple physical LUNS vs a single Logical Volume. >=20 > > But basically: > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > Other people will be accessing/managing this system, so I want > > manageability through simplicity. >=20 > Hence the question whether you actually need LVM. > It strikes me that typically LVM is not needed and > just complicates matters. >=20 > > Don't want to have the wrong volumes > > (re)encrypted, headers damaged, etc. >=20 > I think this setup is already too complicated for most people > to manage. A full backup should be part of your plans. > Resize with backup is actually easier, as you can just > backup, kill everything and do a clean new config. >=20 I will be making full backups with xfsdump. But it doesn't seem practical to be moving 50TB around. Both LVM and XFS allow for designed for online resize. I was planning to "pvcreate" an encrypted LUN then expand the Logical Volume and Filesystem. > > Anyways, thanks for your help. > > >=20 > Just to make sure: You _have_ read the FAQ? Yes I have read the FAQ. I guess I am just a little nervous about this project. This is my first time working with encryption and 50TB of critical client data is at stake. Just being thorough and making sure I fully understand everything. Thanks again, Jacob Sohl >=20 > Arno >=20 > > > > Jacob Sohl | Systems Engineer > > > > Applied Discovery(r) > > > > Mobile: 360-620-2695 > > > > > > >=20 > > _______________________________________________ > > dm-crypt mailing list > > dm-crypt@saout.de > > http://www.saout.de/mailman/listinfo/dm-crypt >=20 >=20 > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans >=20 > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vpECXo9yWdgq for ; Wed, 12 Oct 2011 02:06:20 +0200 (CEST) Received: from mx03.lexisnexis.com (mx03.lexisnexis.com [198.185.18.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Wed, 12 Oct 2011 02:06:20 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 11 Oct 2011 17:06:16 -0700 Message-ID: In-Reply-To: <4E9000DB.4000308@redhat.com> References: <20111008045232.GB23870@tansi.org> <4E9000DB.4000308@redhat.com> From: "Sohl, Jacob (LNG-SEA)" Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Milan Broz , dm-crypt@saout.de > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Milan Broz > Sent: Saturday, October 08, 2011 12:51 AM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster >=20 >=20 > On 10/08/2011 06:52 AM, Arno Wagner wrote: > >> Also, scalability is a requirement in my design, hence XFS. I was > >> thinking I needed to use multiple LUKS PVs in LVM to grow the > >> filesystem. But I would end up with multiple LUKS devices to keep > track > >> of. I recently found out that LUKS can resize. > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > device size. So if you enlarge a device/partition with an intact > > LUKS header, "cryptsetup luksOpen" will just map the larger > > device. >=20 > yes, but crytsetup resize is designed that it it supports open > (active) LUKS containers. You do not need to activate/deactivate, > resize operation is enough. >=20 > > If you have multiple devices, you can "slave" the additional ones > > to a "master" device using something like "decrypt_derived". > > This just takes the master key from the opened "master" container, > > runns it though a hash and uses this as key for the "slave" > > device. > > > >> Would it be better to > >> create one LUKS device on top of LVM? Then create a filesystem on > that? > >> (Though that would affect resource dependencies.) > > > > Sre you sure you need LVM at all? >=20 > LVM is very useful in such configurations, you can dynamically manage > LVs over encrypted storage. > Scripts for managing such stack is not complicated but of course > it is yet another layer to care of. >=20 > Because you mentioned cluster, I will just repeat >=20 > - if you manage it in HA LVM style (the device > stack is always active exclusively only on one node) it will work. > (You just need to create own resource for cluster suite - add LUKS > to the stack. So cluster suite will handle exclusivity and failover > itself.) ^ HA LVM is what I will be doing. >=20 > For using LUKS in cluster for shared storage (visible to all nodes) > - if encryption (mapping) runs in server and plaintext device is > exported > and shared to nodes (using iscsi + clvm or so) this will work >=20 > - if you want to run dm-crypt on every node (for the same device, > IOW export ciphertext device) this is not supported. >=20 > (I think that I said on IRC that with proper used cluster fs and > flush requests this may work - but it is not tested. The problem is of > course > with internal crypto queues so without flushing these queues different > nodes will see different storage content. And moreover, there is no > cluster > infrastructure like clvmd to manage such devices consistently.) Thank you for clarifying this for me, I didn't really understand in IRC what you meant about "flush requests". I'm not a programmer and don't really know low-level hardware stuff that much. It's been a lot of stuff for me to learn. >=20 > Milan > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id XOtbeYg2hbu6 for ; Wed, 12 Oct 2011 05:13:31 +0200 (CEST) Received: from v4.tansi.org (ns.km33513-03.keymachine.de [87.118.94.3]) by mail.saout.de (Postfix) with ESMTP for ; Wed, 12 Oct 2011 05:13:30 +0200 (CEST) Received: from gatewagner.dyndns.org (84-74-163-71.dclient.hispeed.ch [84.74.163.71]) by v4.tansi.org (Postfix) with ESMTPA id 1CA2D1404001 for ; Wed, 12 Oct 2011 05:13:30 +0200 (CEST) Date: Wed, 12 Oct 2011 05:13:29 +0200 From: Arno Wagner Message-ID: <20111012031329.GA29873@tansi.org> References: <20111008045232.GB23870@tansi.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: dm-crypt@saout.de On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: [...] > > You can map ("decrypt") the devices and never use them. The > > LVM/mount/whatever is completely optional. > > > > In fact the mapper tool (cryptsetup) does nothing except > > to decrypt the raw encrypted device to a raw decrypted device > > under /dev/mapper/. > > > > Ok, thank you. That seemed logical, but I just recently started working > with encryption so I'm still learning. That is fine. Making sure now will safe you grief later. > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > filesystem. But I would end up with multiple LUKS devices to keep > > track > > > of. I recently found out that LUKS can resize. > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > device size. So if you enlarge a device/partition with an intact > > LUKS header, "cryptsetup luksOpen" will just map the larger > > device. > > > > If you have multiple devices, you can "slave" the additional ones > > to a "master" device using something like "decrypt_derived". > > This just takes the master key from the opened "master" container, > > runns it though a hash and uses this as key for the "slave" > > device. > > > > > Would it be better to > > > create one LUKS device on top of LVM? Then create a filesystem on > > that? > > > (Though that would affect resource dependencies.) > > > > Sre you sure you need LVM at all? > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > can/will only attach storage in 4TB increments. So I have to combine all > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > filesystem. That's why I ask about encrypting multiple physical LUNS vs > a single Logical Volume. 50TB? Ok, that is a bit larger. I did have such a dataset for research a few years back on tape. Took several days days to get 15TB of it to disk for my largest measurement. Well, yes, I think LVM is the best way here. > > > > > But basically: > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > manageability through simplicity. > > > > Hence the question whether you actually need LVM. > > It strikes me that typically LVM is not needed and > > just complicates matters. > > > > > Don't want to have the wrong volumes > > > (re)encrypted, headers damaged, etc. > > > > I think this setup is already too complicated for most people > > to manage. A full backup should be part of your plans. > > Resize with backup is actually easier, as you can just > > backup, kill everything and do a clean new config. > > > > I will be making full backups with xfsdump. But it doesn't seem > practical to be moving 50TB around. Both LVM and XFS allow for designed > for online resize. I was planning to "pvcreate" an encrypted LUN then > expand the Logical Volume and Filesystem. For that volume, run a full compare as well. I found that with good hardware, and an IBM Tape-Library I still had something like 1 unexplained bit-error every 10TB or so when pulling the data from the tape-library again. That was a few years back, but still. > > > Anyways, thanks for your help. > > > > > > > Just to make sure: You _have_ read the FAQ? > > Yes I have read the FAQ. Good! > I guess I am just a little nervous about this project. Understandable. > This is my first time working with encryption and 50TB of > critical client data is at stake. Just being thorough and making sure I > fully understand everything. And that is all you can do. I think you are approaching this perfectly right. I take it the 50TB are alredy on XFS+LVM at this time and you are just adding the encryption layer? Make sure to use XTS mode or plain64 with ESSIV for these volume sizes. And let us know what your expeiences are! Arno -- Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F ---- Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans If it's in the news, don't worry about it. The very definition of "news" is "something that hardly ever happens." -- Bruce Schneier From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 9kwWqX6gFTLz for ; Wed, 12 Oct 2011 21:26:15 +0200 (CEST) Received: from mx03.lexisnexis.com (mx03.lexisnexis.com [198.185.18.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Wed, 12 Oct 2011 21:26:15 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Wed, 12 Oct 2011 12:20:51 -0700 Message-ID: In-Reply-To: <20111012031329.GA29873@tansi.org> References: <20111008045232.GB23870@tansi.org> <20111012031329.GA29873@tansi.org> From: "Sohl, Jacob (LNG-SEA)" Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Arno Wagner , dm-crypt@saout.de Cc: Scott McCarty , LNG-SEA Systems Engineers ADI > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Tuesday, October 11, 2011 8:13 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster >=20 > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: > [...] > > > You can map ("decrypt") the devices and never use them. The > > > LVM/mount/whatever is completely optional. > > > > > > In fact the mapper tool (cryptsetup) does nothing except > > > to decrypt the raw encrypted device to a raw decrypted device > > > under /dev/mapper/. > > > > > > > Ok, thank you. That seemed logical, but I just recently started > working > > with encryption so I'm still learning. >=20 > That is fine. Making sure now will safe you grief later. >=20 > > > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > > filesystem. But I would end up with multiple LUKS devices to keep > > > track > > > > of. I recently found out that LUKS can resize. > > > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > > device size. So if you enlarge a device/partition with an intact > > > LUKS header, "cryptsetup luksOpen" will just map the larger > > > device. > > > > > > If you have multiple devices, you can "slave" the additional ones > > > to a "master" device using something like "decrypt_derived". > > > This just takes the master key from the opened "master" container, > > > runns it though a hash and uses this as key for the "slave" > > > device. > > > > > > > Would it be better to > > > > create one LUKS device on top of LVM? Then create a filesystem on > > > that? > > > > (Though that would affect resource dependencies.) > > > > > > Sre you sure you need LVM at all? > > > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > > can/will only attach storage in 4TB increments. So I have to combine > all > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > > filesystem. That's why I ask about encrypting multiple physical LUNS > vs > > a single Logical Volume. >=20 > 50TB? Ok, that is a bit larger. I did have such a dataset > for research a few years back on tape. Took several days > days to get 15TB of it to disk for my largest measurement. >=20 > Well, yes, I think LVM is the best way here. >=20 > > > > > > > But basically: > > > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > > manageability through simplicity. > > > > > > Hence the question whether you actually need LVM. > > > It strikes me that typically LVM is not needed and > > > just complicates matters. > > > > > > > Don't want to have the wrong volumes > > > > (re)encrypted, headers damaged, etc. > > > > > > I think this setup is already too complicated for most people > > > to manage. A full backup should be part of your plans. > > > Resize with backup is actually easier, as you can just > > > backup, kill everything and do a clean new config. > > > > > > > I will be making full backups with xfsdump. But it doesn't seem > > practical to be moving 50TB around. Both LVM and XFS allow for > designed > > for online resize. I was planning to "pvcreate" an encrypted LUN then > > expand the Logical Volume and Filesystem. >=20 > For that volume, run a full compare as well. I found that > with good hardware, and an IBM Tape-Library I still had > something like 1 unexplained bit-error every 10TB or so > when pulling the data from the tape-library again. That was > a few years back, but still. >=20 Thanks, I've never been very good at doing backups so any advice is appreciated. > > > > Anyways, thanks for your help. > > > > > > > > > > Just to make sure: You _have_ read the FAQ? > > > > Yes I have read the FAQ. >=20 > Good! >=20 > > I guess I am just a little nervous about this project. >=20 > Understandable. >=20 > > This is my first time working with encryption and 50TB of > > critical client data is at stake. Just being thorough and making sure > I > > fully understand everything. >=20 > And that is all you can do. I think you are approaching this > perfectly right. I take it the 50TB are alredy on XFS+LVM > at this time and you are just adding the encryption layer? Currently the data is spread across multiple ext3 volumes of 8TB or less and is a pain to manage. That is the reason we are moving to XFS. The encryption is currently being handled by a Brocade switch blade. The encryption blade caused us major issues last year, resulting in a multi-day outage. That is the why we are moving to LUKS. >=20 > Make sure to use XTS mode or plain64 with ESSIV for these > volume sizes. I don't know what this means. Haha. But I will look it up. >=20 > And let us know what your expeiences are! >=20 Sure thing. This project has been my first exposure to XFS, LUKS and clustering. And I'm the main Linux guy on my team. I've been working with Red Hat, but the solution architect laughed when he saw what we were trying to do. Haha. But he's been doing research and talking to his Red Hat contacts. And you guys have helped out tremendously! Thank you! Jacob > Arno > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans >=20 > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id z61pamCpTVCO for ; Wed, 12 Oct 2011 22:26:25 +0200 (CEST) Received: from mx03.lexisnexis.com (mx03.lexisnexis.com [198.185.18.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.saout.de (Postfix) with ESMTPS for ; Wed, 12 Oct 2011 22:26:24 +0200 (CEST) Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Date: Wed, 12 Oct 2011 13:26:08 -0700 Message-ID: In-Reply-To: <19f1feb1-bc78-4977-b233-eb8837d281d1@keith.crunchtools.com> References: <19f1feb1-bc78-4977-b233-eb8837d281d1@keith.crunchtools.com> From: "Sohl, Jacob (LNG-SEA)" Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Scott McCarty Cc: dm-crypt@saout.de, Arno Wagner , LNG-SEA Systems Engineers ADI WWVhaCwgaXQgd2FzIGEgImNyYXp5LWZ1bi1wcm9qZWN0IiBsYXVnaC4gQW4gb3Bwb3J0dW5pdHkg dG8gbGVhcm4gYSBsb3QgYW5kIGNyZWF0ZSBzb21ldGhpbmcgY29vbC4NCg0KPiAtLS0tLU9yaWdp bmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiBTY290dCBNY0NhcnR5IFttYWlsdG86c21jY2FydHlA cmVkaGF0LmNvbV0NCj4gU2VudDogV2VkbmVzZGF5LCBPY3RvYmVyIDEyLCAyMDExIDEyOjU1IFBN DQo+IFRvOiBTb2hsLCBKYWNvYiAoTE5HLVNFQSkNCj4gQ2M6IExORy1TRUEgU3lzdGVtcyBFbmdp bmVlcnMgQURJOyBBcm5vIFdhZ25lcjsgZG0tY3J5cHRAc2FvdXQuZGUNCj4gU3ViamVjdDogUmU6 IFtkbS1jcnlwdF0gTFVLUyBpbiBmYWlsb3ZlciBjbHVzdGVyDQo+IA0KPiBUaGF0IHdhcyBhIGdv b2QgbGF1Z2gsIG1heWJlIGEgY2h1Y2tsZSBldmVuIDotKQ0KPiANCj4gU2NvdHQgTWNDYXJ0eSwg UkhDRSwgUkhDU0EsIExQSUMtMQ0KPiBTb2x1dGlvbnMgQXJjaGl0ZWN0DQo+IEVtYWlsOiBzbWNj YXJ0eUByZWRoYXQuY29tDQo+IFBob25lOiAzMTItNjYwLTM1MzUNCj4gV2ViOiBodHRwOi8vcGVv cGxlLnJlZGhhdC5jb20vc21jY2FydHkNCj4gDQo+IA0KPiAtLS0tLSBPcmlnaW5hbCBNZXNzYWdl IC0tLS0tDQo+IEZyb206ICJKYWNvYiBTb2hsIChMTkctU0VBKSIgPGphY29iLnNvaGxAYXBwbGll ZGRpc2NvdmVyeS5jb20+DQo+IFRvOiAiQXJubyBXYWduZXIiIDxhcm5vQHdhZ25lci5uYW1lPiwg ZG0tY3J5cHRAc2FvdXQuZGUNCj4gQ2M6ICJMTkctU0VBIFN5c3RlbXMgRW5naW5lZXJzIEFESSIg PExORy0NCj4gU0VBU3lzdGVtc0VuZ2luZWVyc0FESUBsZXhpc25leGlzLmNvbT4sICJTY290dCBN Y0NhcnR5Ig0KPiA8c21jY2FydHlAcmVkaGF0LmNvbT4NCj4gU2VudDogV2VkbmVzZGF5LCBPY3Rv YmVyIDEyLCAyMDExIDM6MjA6NTEgUE0NCj4gU3ViamVjdDogUkU6IFtkbS1jcnlwdF0gTFVLUyBp biBmYWlsb3ZlciBjbHVzdGVyDQo+IA0KPiANCj4gDQo+ID4gLS0tLS1PcmlnaW5hbCBNZXNzYWdl LS0tLS0NCj4gPiBGcm9tOiBkbS1jcnlwdC1ib3VuY2VzQHNhb3V0LmRlIFttYWlsdG86ZG0tY3J5 cHQtYm91bmNlc0BzYW91dC5kZV0gT24NCj4gPiBCZWhhbGYgT2YgQXJubyBXYWduZXINCj4gPiBT ZW50OiBUdWVzZGF5LCBPY3RvYmVyIDExLCAyMDExIDg6MTMgUE0NCj4gPiBUbzogZG0tY3J5cHRA c2FvdXQuZGUNCj4gPiBTdWJqZWN0OiBSZTogW2RtLWNyeXB0XSBMVUtTIGluIGZhaWxvdmVyIGNs dXN0ZXINCj4gPg0KPiA+IE9uIFR1ZSwgT2N0IDExLCAyMDExIGF0IDA0OjM0OjIxUE0gLTA3MDAs IFNvaGwsIEphY29iIChMTkctU0VBKQ0KPiB3cm90ZToNCj4gPiBbLi4uXQ0KPiA+ID4gPiBZb3Ug Y2FuIG1hcCAoImRlY3J5cHQiKSB0aGUgZGV2aWNlcyBhbmQgbmV2ZXIgdXNlIHRoZW0uIFRoZQ0K PiA+ID4gPiBMVk0vbW91bnQvd2hhdGV2ZXIgaXMgY29tcGxldGVseSBvcHRpb25hbC4NCj4gPiA+ ID4NCj4gPiA+ID4gSW4gZmFjdCB0aGUgbWFwcGVyIHRvb2wgKGNyeXB0c2V0dXApIGRvZXMgbm90 aGluZyBleGNlcHQNCj4gPiA+ID4gdG8gZGVjcnlwdCB0aGUgcmF3IGVuY3J5cHRlZCBkZXZpY2Ug dG8gYSByYXcgZGVjcnlwdGVkIGRldmljZQ0KPiA+ID4gPiB1bmRlciAvZGV2L21hcHBlci8uDQo+ ID4gPiA+DQo+ID4gPg0KPiA+ID4gT2ssIHRoYW5rIHlvdS4gVGhhdCBzZWVtZWQgbG9naWNhbCwg YnV0IEkganVzdCByZWNlbnRseSBzdGFydGVkDQo+ID4gd29ya2luZw0KPiA+ID4gd2l0aCBlbmNy eXB0aW9uIHNvIEknbSBzdGlsbCBsZWFybmluZy4NCj4gPg0KPiA+IFRoYXQgaXMgZmluZS4gTWFr aW5nIHN1cmUgbm93IHdpbGwgc2FmZSB5b3UgZ3JpZWYgbGF0ZXIuDQo+ID4NCj4gPiA+ID4NCj4g PiA+ID4gPiBBbHNvLCBzY2FsYWJpbGl0eSBpcyBhIHJlcXVpcmVtZW50IGluIG15IGRlc2lnbiwg aGVuY2UgWEZTLiBJDQo+IHdhcw0KPiA+ID4gPiA+IHRoaW5raW5nIEkgbmVlZGVkIHRvIHVzZSBt dWx0aXBsZSBMVUtTIFBWcyBpbiBMVk0gdG8gZ3JvdyB0aGUNCj4gPiA+ID4gPiBmaWxlc3lzdGVt LiBCdXQgSSB3b3VsZCBlbmQgdXAgd2l0aCBtdWx0aXBsZSBMVUtTIGRldmljZXMgdG8NCj4ga2Vl cA0KPiA+ID4gPiB0cmFjaw0KPiA+ID4gPiA+IG9mLiBJIHJlY2VudGx5IGZvdW5kIG91dCB0aGF0 IExVS1MgY2FuIHJlc2l6ZS4NCj4gPiA+ID4NCj4gPiA+ID4gTFVLUyBfY2Fubm90XyByZXNpemUu IFRoZSB0aGluZyBpcyB0aGF0IExVS1MgZG9lcyBub3QgY2FyZSBhYm91dA0KPiA+ID4gPiBkZXZp Y2Ugc2l6ZS4gU28gaWYgeW91IGVubGFyZ2UgYSBkZXZpY2UvcGFydGl0aW9uIHdpdGggYW4gaW50 YWN0DQo+ID4gPiA+IExVS1MgaGVhZGVyLCAiY3J5cHRzZXR1cCBsdWtzT3BlbiIgd2lsbCBqdXN0 IG1hcCB0aGUgbGFyZ2VyDQo+ID4gPiA+IGRldmljZS4NCj4gPiA+ID4NCj4gPiA+ID4gSWYgeW91 IGhhdmUgbXVsdGlwbGUgZGV2aWNlcywgeW91IGNhbiAic2xhdmUiIHRoZSBhZGRpdGlvbmFsIG9u ZXMNCj4gPiA+ID4gdG8gYSAibWFzdGVyIiBkZXZpY2UgdXNpbmcgc29tZXRoaW5nIGxpa2UgImRl Y3J5cHRfZGVyaXZlZCIuDQo+ID4gPiA+IFRoaXMganVzdCB0YWtlcyB0aGUgbWFzdGVyIGtleSBm cm9tIHRoZSBvcGVuZWQgIm1hc3RlciINCj4gY29udGFpbmVyLA0KPiA+ID4gPiBydW5ucyBpdCB0 aG91Z2ggYSBoYXNoIGFuZCB1c2VzIHRoaXMgYXMga2V5IGZvciB0aGUgInNsYXZlIg0KPiA+ID4g PiBkZXZpY2UuDQo+ID4gPiA+DQo+ID4gPiA+ID4gV291bGQgaXQgYmUgYmV0dGVyIHRvDQo+ID4g PiA+ID4gY3JlYXRlIG9uZSBMVUtTIGRldmljZSBvbiB0b3Agb2YgTFZNPyBUaGVuIGNyZWF0ZSBh IGZpbGVzeXN0ZW0NCj4gb24NCj4gPiA+ID4gdGhhdD8NCj4gPiA+ID4gPiAoVGhvdWdoIHRoYXQg d291bGQgYWZmZWN0IHJlc291cmNlIGRlcGVuZGVuY2llcy4pDQo+ID4gPiA+DQo+ID4gPiA+IFNy ZSB5b3Ugc3VyZSB5b3UgbmVlZCBMVk0gYXQgYWxsPw0KPiA+ID4NCj4gPiA+IFllcywgSSBkbyBu ZWVkIExWTS4gSSdtIGRlYWxpbmcgd2l0aCB+NTBUQiBvZiBkYXRhIGFuZCB0aGUgU0FODQo+IGFk bWlucw0KPiA+ID4gY2FuL3dpbGwgb25seSBhdHRhY2ggc3RvcmFnZSBpbiA0VEIgaW5jcmVtZW50 cy4gU28gSSBoYXZlIHRvDQo+IGNvbWJpbmUNCj4gPiBhbGwNCj4gPiA+IG9mIHRoZSBMVU5TIGlu dG8gb25lIExvZ2ljYWwgVm9sdW1lIGluIG9yZGVyIHRvIGNyZWF0ZSBhIDUwVEIrIFhGUw0KPiA+ ID4gZmlsZXN5c3RlbS4gVGhhdCdzIHdoeSBJIGFzayBhYm91dCBlbmNyeXB0aW5nIG11bHRpcGxl IHBoeXNpY2FsDQo+IExVTlMNCj4gPiB2cw0KPiA+ID4gYSBzaW5nbGUgTG9naWNhbCBWb2x1bWUu DQo+ID4NCj4gPiA1MFRCPyBPaywgdGhhdCBpcyBhIGJpdCBsYXJnZXIuIEkgZGlkIGhhdmUgc3Vj aCBhIGRhdGFzZXQNCj4gPiBmb3IgcmVzZWFyY2ggYSBmZXcgeWVhcnMgYmFjayBvbiB0YXBlLiBU b29rIHNldmVyYWwgZGF5cw0KPiA+IGRheXMgdG8gZ2V0IDE1VEIgb2YgaXQgdG8gZGlzayBmb3Ig bXkgbGFyZ2VzdCBtZWFzdXJlbWVudC4NCj4gPg0KPiA+IFdlbGwsIHllcywgSSB0aGluayBMVk0g aXMgdGhlIGJlc3Qgd2F5IGhlcmUuDQo+ID4NCj4gPiA+ID4NCj4gPiA+ID4gPiBCdXQgYmFzaWNh bGx5Og0KPiA+ID4gPiA+DQo+ID4gPiA+ID4gU0FOIExVTnMgPiBMVk0gPiBMVUtTID4gWEZTID4g U2FtYmEgU2VydmVyDQo+ID4gPiA+ID4NCj4gPiA+ID4gPg0KPiA+ID4gPiA+DQo+ID4gPiA+ID4g T3RoZXIgcGVvcGxlIHdpbGwgYmUgYWNjZXNzaW5nL21hbmFnaW5nIHRoaXMgc3lzdGVtLCBzbyBJ IHdhbnQNCj4gPiA+ID4gPiBtYW5hZ2VhYmlsaXR5IHRocm91Z2ggc2ltcGxpY2l0eS4NCj4gPiA+ ID4NCj4gPiA+ID4gSGVuY2UgdGhlIHF1ZXN0aW9uIHdoZXRoZXIgeW91IGFjdHVhbGx5IG5lZWQg TFZNLg0KPiA+ID4gPiBJdCBzdHJpa2VzIG1lIHRoYXQgdHlwaWNhbGx5IExWTSBpcyBub3QgbmVl ZGVkIGFuZA0KPiA+ID4gPiBqdXN0IGNvbXBsaWNhdGVzIG1hdHRlcnMuDQo+ID4gPiA+DQo+ID4g PiA+ID4gRG9uJ3Qgd2FudCB0byBoYXZlIHRoZSB3cm9uZyB2b2x1bWVzDQo+ID4gPiA+ID4gKHJl KWVuY3J5cHRlZCwgaGVhZGVycyBkYW1hZ2VkLCBldGMuDQo+ID4gPiA+DQo+ID4gPiA+IEkgdGhp bmsgdGhpcyBzZXR1cCBpcyBhbHJlYWR5IHRvbyBjb21wbGljYXRlZCBmb3IgbW9zdCBwZW9wbGUN Cj4gPiA+ID4gdG8gbWFuYWdlLiBBIGZ1bGwgYmFja3VwIHNob3VsZCBiZSBwYXJ0IG9mIHlvdXIg cGxhbnMuDQo+ID4gPiA+IFJlc2l6ZSB3aXRoIGJhY2t1cCBpcyBhY3R1YWxseSBlYXNpZXIsIGFz IHlvdSBjYW4ganVzdA0KPiA+ID4gPiBiYWNrdXAsIGtpbGwgZXZlcnl0aGluZyBhbmQgZG8gYSBj bGVhbiBuZXcgY29uZmlnLg0KPiA+ID4gPg0KPiA+ID4NCj4gPiA+IEkgd2lsbCBiZSBtYWtpbmcg ZnVsbCBiYWNrdXBzIHdpdGggeGZzZHVtcC4gQnV0IGl0IGRvZXNuJ3Qgc2VlbQ0KPiA+ID4gcHJh Y3RpY2FsIHRvIGJlIG1vdmluZyA1MFRCIGFyb3VuZC4gQm90aCBMVk0gYW5kIFhGUyBhbGxvdyBm b3INCj4gPiBkZXNpZ25lZA0KPiA+ID4gZm9yIG9ubGluZSByZXNpemUuIEkgd2FzIHBsYW5uaW5n IHRvICJwdmNyZWF0ZSIgYW4gZW5jcnlwdGVkIExVTg0KPiB0aGVuDQo+ID4gPiBleHBhbmQgdGhl IExvZ2ljYWwgVm9sdW1lIGFuZCBGaWxlc3lzdGVtLg0KPiA+DQo+ID4gRm9yIHRoYXQgdm9sdW1l LCBydW4gYSBmdWxsIGNvbXBhcmUgYXMgd2VsbC4gSSBmb3VuZCB0aGF0DQo+ID4gd2l0aCBnb29k IGhhcmR3YXJlLCBhbmQgYW4gSUJNIFRhcGUtTGlicmFyeSBJIHN0aWxsIGhhZA0KPiA+IHNvbWV0 aGluZyBsaWtlIDEgdW5leHBsYWluZWQgYml0LWVycm9yIGV2ZXJ5IDEwVEIgb3Igc28NCj4gPiB3 aGVuIHB1bGxpbmcgdGhlIGRhdGEgZnJvbSB0aGUgdGFwZS1saWJyYXJ5IGFnYWluLiBUaGF0IHdh cw0KPiA+IGEgZmV3IHllYXJzIGJhY2ssIGJ1dCBzdGlsbC4NCj4gPg0KPiANCj4gVGhhbmtzLCBJ J3ZlIG5ldmVyIGJlZW4gdmVyeSBnb29kIGF0IGRvaW5nIGJhY2t1cHMgc28gYW55IGFkdmljZSBp cw0KPiBhcHByZWNpYXRlZC4NCj4gDQo+ID4gPiA+ID4gQW55d2F5cywgdGhhbmtzIGZvciB5b3Vy IGhlbHAuDQo+ID4gPiA+ID4NCj4gPiA+ID4NCj4gPiA+ID4gSnVzdCB0byBtYWtlIHN1cmU6IFlv dSBfaGF2ZV8gcmVhZCB0aGUgRkFRPw0KPiA+ID4NCj4gPiA+IFllcyBJIGhhdmUgcmVhZCB0aGUg RkFRLg0KPiA+DQo+ID4gR29vZCENCj4gPg0KPiA+ID4gSSBndWVzcyBJIGFtIGp1c3QgYSBsaXR0 bGUgbmVydm91cyBhYm91dCB0aGlzIHByb2plY3QuDQo+ID4NCj4gPiBVbmRlcnN0YW5kYWJsZS4N Cj4gPg0KPiA+ID4gVGhpcyBpcyBteSBmaXJzdCB0aW1lIHdvcmtpbmcgd2l0aCBlbmNyeXB0aW9u IGFuZCA1MFRCIG9mDQo+ID4gPiBjcml0aWNhbCBjbGllbnQgZGF0YSBpcyBhdCBzdGFrZS4gSnVz dCBiZWluZyB0aG9yb3VnaCBhbmQgbWFraW5nDQo+IHN1cmUNCj4gPiBJDQo+ID4gPiBmdWxseSB1 bmRlcnN0YW5kIGV2ZXJ5dGhpbmcuDQo+ID4NCj4gPiBBbmQgdGhhdCBpcyBhbGwgeW91IGNhbiBk by4gSSB0aGluayB5b3UgYXJlIGFwcHJvYWNoaW5nIHRoaXMNCj4gPiBwZXJmZWN0bHkgcmlnaHQu IEkgdGFrZSBpdCB0aGUgNTBUQiBhcmUgYWxyZWR5IG9uIFhGUytMVk0NCj4gPiBhdCB0aGlzIHRp bWUgYW5kIHlvdSBhcmUganVzdCBhZGRpbmcgdGhlIGVuY3J5cHRpb24gbGF5ZXI/DQo+IA0KPiBD dXJyZW50bHkgdGhlIGRhdGEgaXMgc3ByZWFkIGFjcm9zcyBtdWx0aXBsZSBleHQzIHZvbHVtZXMg b2YgOFRCIG9yDQo+IGxlc3MNCj4gYW5kIGlzIGEgcGFpbiB0byBtYW5hZ2UuIFRoYXQgaXMgdGhl IHJlYXNvbiB3ZSBhcmUgbW92aW5nIHRvIFhGUy4gVGhlDQo+IGVuY3J5cHRpb24gaXMgY3VycmVu dGx5IGJlaW5nIGhhbmRsZWQgYnkgYSBCcm9jYWRlIHN3aXRjaCBibGFkZS4gVGhlDQo+IGVuY3J5 cHRpb24gYmxhZGUgY2F1c2VkIHVzIG1ham9yIGlzc3VlcyBsYXN0IHllYXIsIHJlc3VsdGluZyBp biBhDQo+IG11bHRpLWRheSBvdXRhZ2UuIFRoYXQgaXMgdGhlIHdoeSB3ZSBhcmUgbW92aW5nIHRv IExVS1MuDQo+IA0KPiA+DQo+ID4gTWFrZSBzdXJlIHRvIHVzZSBYVFMgbW9kZSBvciBwbGFpbjY0 IHdpdGggRVNTSVYgZm9yIHRoZXNlDQo+ID4gdm9sdW1lIHNpemVzLg0KPiANCj4gSSBkb24ndCBr bm93IHdoYXQgdGhpcyBtZWFucy4gSGFoYS4gQnV0IEkgd2lsbCBsb29rIGl0IHVwLg0KPiANCj4g Pg0KPiA+IEFuZCBsZXQgdXMga25vdyB3aGF0IHlvdXIgZXhwZWllbmNlcyBhcmUhDQo+ID4NCj4g DQo+IFN1cmUgdGhpbmcuIFRoaXMgcHJvamVjdCBoYXMgYmVlbiBteSBmaXJzdCBleHBvc3VyZSB0 byBYRlMsIExVS1MgYW5kDQo+IGNsdXN0ZXJpbmcuIEFuZCBJJ20gdGhlIG1haW4gTGludXggZ3V5 IG9uIG15IHRlYW0uIEkndmUgYmVlbiB3b3JraW5nDQo+IHdpdGggUmVkIEhhdCwgYnV0IHRoZSBz b2x1dGlvbiBhcmNoaXRlY3QgbGF1Z2hlZCB3aGVuIGhlIHNhdyB3aGF0IHdlDQo+IHdlcmUgdHJ5 aW5nIHRvIGRvLiBIYWhhLiBCdXQgaGUncyBiZWVuIGRvaW5nIHJlc2VhcmNoIGFuZCB0YWxraW5n IHRvDQo+IGhpcw0KPiBSZWQgSGF0IGNvbnRhY3RzLiBBbmQgeW91IGd1eXMgaGF2ZSBoZWxwZWQg b3V0IHRyZW1lbmRvdXNseSEgVGhhbmsgeW91IQ0KPiANCj4gSmFjb2INCj4gDQo+ID4gQXJubw0K PiA+IC0tDQo+ID4gQXJubyBXYWduZXIsIERyLiBzYy4gdGVjaG4uLCBEaXBsLiBJbmZvcm0uLCBD SVNTUCAtLSBFbWFpbDoNCj4gPiBhcm5vQHdhZ25lci5uYW1lDQo+ID4gR251UEc6ICBJRDogMUUy NTMzOEYgIEZQOiAwQzMwIDU3ODIgOUQ5MyBGNzg1IEU3OUMgIDAyOTYgNzk3RiA2QjUwDQo+IDFF MjUNCj4gPiAzMzhGDQo+ID4gLS0tLQ0KPiA+IEN1ZGRseSBVSSdzIGFyZSB0aGUgbWFuaWZlc3Rh dGlvbiBvZiB3aXNoZnVsIHRoaW5raW5nLiAtLSBEeWxhbiBFdmFucw0KPiA+DQo+ID4gSWYgaXQn cyBpbiB0aGUgbmV3cywgZG9uJ3Qgd29ycnkgYWJvdXQgaXQuICBUaGUgdmVyeSBkZWZpbml0aW9u IG9mDQo+ID4gIm5ld3MiIGlzICJzb21ldGhpbmcgdGhhdCBoYXJkbHkgZXZlciBoYXBwZW5zLiIg LS0gQnJ1Y2UgU2NobmVpZXINCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fXw0KPiA+IGRtLWNyeXB0IG1haWxpbmcgbGlzdA0KPiA+IGRtLWNyeXB0QHNh b3V0LmRlDQo+ID4gaHR0cDovL3d3dy5zYW91dC5kZS9tYWlsbWFuL2xpc3RpbmZvL2RtLWNyeXB0 DQo= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail.saout.de ([127.0.0.1]) by localhost (mail.saout.de [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id rETyJ8dvc8CO for ; Wed, 12 Oct 2011 22:45:02 +0200 (CEST) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by mail.saout.de (Postfix) with ESMTP for ; Wed, 12 Oct 2011 22:45:01 +0200 (CEST) Received: from mx3-phx2.redhat.com (mx01.colomx.prod.int.phx2.redhat.com [10.5.7.1]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id p9CJt35P026298 for ; Wed, 12 Oct 2011 15:55:03 -0400 Date: Wed, 12 Oct 2011 15:54:57 -0400 (EDT) From: Scott McCarty Message-ID: <19f1feb1-bc78-4977-b233-eb8837d281d1@keith.crunchtools.com> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit MIME-Version: 1.0 Subject: Re: [dm-crypt] LUKS in failover cluster List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Jacob Sohl (LNG-SEA)" Cc: dm-crypt@saout.de, Arno Wagner , LNG-SEA Systems Engineers ADI That was a good laugh, maybe a chuckle even :-) Scott McCarty, RHCE, RHCSA, LPIC-1 Solutions Architect Email: smccarty@redhat.com Phone: 312-660-3535 Web: http://people.redhat.com/smccarty ----- Original Message ----- From: "Jacob Sohl (LNG-SEA)" To: "Arno Wagner" , dm-crypt@saout.de Cc: "LNG-SEA Systems Engineers ADI" , "Scott McCarty" Sent: Wednesday, October 12, 2011 3:20:51 PM Subject: RE: [dm-crypt] LUKS in failover cluster > -----Original Message----- > From: dm-crypt-bounces@saout.de [mailto:dm-crypt-bounces@saout.de] On > Behalf Of Arno Wagner > Sent: Tuesday, October 11, 2011 8:13 PM > To: dm-crypt@saout.de > Subject: Re: [dm-crypt] LUKS in failover cluster > > On Tue, Oct 11, 2011 at 04:34:21PM -0700, Sohl, Jacob (LNG-SEA) wrote: > [...] > > > You can map ("decrypt") the devices and never use them. The > > > LVM/mount/whatever is completely optional. > > > > > > In fact the mapper tool (cryptsetup) does nothing except > > > to decrypt the raw encrypted device to a raw decrypted device > > > under /dev/mapper/. > > > > > > > Ok, thank you. That seemed logical, but I just recently started > working > > with encryption so I'm still learning. > > That is fine. Making sure now will safe you grief later. > > > > > > > > Also, scalability is a requirement in my design, hence XFS. I was > > > > thinking I needed to use multiple LUKS PVs in LVM to grow the > > > > filesystem. But I would end up with multiple LUKS devices to keep > > > track > > > > of. I recently found out that LUKS can resize. > > > > > > LUKS _cannot_ resize. The thing is that LUKS does not care about > > > device size. So if you enlarge a device/partition with an intact > > > LUKS header, "cryptsetup luksOpen" will just map the larger > > > device. > > > > > > If you have multiple devices, you can "slave" the additional ones > > > to a "master" device using something like "decrypt_derived". > > > This just takes the master key from the opened "master" container, > > > runns it though a hash and uses this as key for the "slave" > > > device. > > > > > > > Would it be better to > > > > create one LUKS device on top of LVM? Then create a filesystem on > > > that? > > > > (Though that would affect resource dependencies.) > > > > > > Sre you sure you need LVM at all? > > > > Yes, I do need LVM. I'm dealing with ~50TB of data and the SAN admins > > can/will only attach storage in 4TB increments. So I have to combine > all > > of the LUNS into one Logical Volume in order to create a 50TB+ XFS > > filesystem. That's why I ask about encrypting multiple physical LUNS > vs > > a single Logical Volume. > > 50TB? Ok, that is a bit larger. I did have such a dataset > for research a few years back on tape. Took several days > days to get 15TB of it to disk for my largest measurement. > > Well, yes, I think LVM is the best way here. > > > > > > > > But basically: > > > > > > > > SAN LUNs > LVM > LUKS > XFS > Samba Server > > > > > > > > > > > > > > > > Other people will be accessing/managing this system, so I want > > > > manageability through simplicity. > > > > > > Hence the question whether you actually need LVM. > > > It strikes me that typically LVM is not needed and > > > just complicates matters. > > > > > > > Don't want to have the wrong volumes > > > > (re)encrypted, headers damaged, etc. > > > > > > I think this setup is already too complicated for most people > > > to manage. A full backup should be part of your plans. > > > Resize with backup is actually easier, as you can just > > > backup, kill everything and do a clean new config. > > > > > > > I will be making full backups with xfsdump. But it doesn't seem > > practical to be moving 50TB around. Both LVM and XFS allow for > designed > > for online resize. I was planning to "pvcreate" an encrypted LUN then > > expand the Logical Volume and Filesystem. > > For that volume, run a full compare as well. I found that > with good hardware, and an IBM Tape-Library I still had > something like 1 unexplained bit-error every 10TB or so > when pulling the data from the tape-library again. That was > a few years back, but still. > Thanks, I've never been very good at doing backups so any advice is appreciated. > > > > Anyways, thanks for your help. > > > > > > > > > > Just to make sure: You _have_ read the FAQ? > > > > Yes I have read the FAQ. > > Good! > > > I guess I am just a little nervous about this project. > > Understandable. > > > This is my first time working with encryption and 50TB of > > critical client data is at stake. Just being thorough and making sure > I > > fully understand everything. > > And that is all you can do. I think you are approaching this > perfectly right. I take it the 50TB are alredy on XFS+LVM > at this time and you are just adding the encryption layer? Currently the data is spread across multiple ext3 volumes of 8TB or less and is a pain to manage. That is the reason we are moving to XFS. The encryption is currently being handled by a Brocade switch blade. The encryption blade caused us major issues last year, resulting in a multi-day outage. That is the why we are moving to LUKS. > > Make sure to use XTS mode or plain64 with ESSIV for these > volume sizes. I don't know what this means. Haha. But I will look it up. > > And let us know what your expeiences are! > Sure thing. This project has been my first exposure to XFS, LUKS and clustering. And I'm the main Linux guy on my team. I've been working with Red Hat, but the solution architect laughed when he saw what we were trying to do. Haha. But he's been doing research and talking to his Red Hat contacts. And you guys have helped out tremendously! Thank you! Jacob > Arno > -- > Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: > arno@wagner.name > GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 > 338F > ---- > Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans > > If it's in the news, don't worry about it. The very definition of > "news" is "something that hardly ever happens." -- Bruce Schneier > _______________________________________________ > dm-crypt mailing list > dm-crypt@saout.de > http://www.saout.de/mailman/listinfo/dm-crypt