[dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Ryan Lynch]

Would it be possible to run a shared-disk, clustered filesystem over a
dm-crypt block device, which in turn runs on a shared iSCSI device?
I'd be interested in knowing if anyone has tried, or has theoretical
knowledge of why it would (not) work.

Multiple Linux machines can simultaneously R/W mount a clustered
filesystem (like GFS/GFS2 or OCFS). Performance can suffer when
multiple hosts write in parallel, but otherwise it works pretty well
(in my limited experience with GFS2). All of the participating hosts
need some kind of shared access directly to the same block device:
I've used iSCSI and DRBD, but I think FC is common, too. Each host
also runs a DLM (distributed lock manager) daemon, which sends and
receives info about which hosts are writing to which inodes, so they
can all keep their caches consistent with the disk. (better
explanation on WP, here:
http://en.wikipedia.org/wiki/Global_File_System#Differences_from_a_local_filesystem)

From my limited understanding of how iSCSI, GFS2 work, and dm-crypt
work, I have no idea how they'll cooperate with each other, though.
I'd like to test it, when I get a chance, but it would be nice to know
a little more, in advance.

Ryan B. Lynch
ryan.b.lynch@gmail.com

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Arno Wagner]

On Fri, Apr 23, 2010 at 05:20:05PM -0400, Ryan Lynch wrote:
> Would it be possible to run a shared-disk, clustered filesystem over a
> dm-crypt block device, which in turn runs on a shared iSCSI device?
> I'd be interested in knowing if anyone has tried, or has theoretical
> knowledge of why it would (not) work.

Thoretically, it should work. A block device is a block device.
The question becomes performance as performance characteristics
cannot be hidden beind an uniform block device interface.

> Multiple Linux machines can simultaneously R/W mount a clustered
> filesystem (like GFS/GFS2 or OCFS). Performance can suffer when
> multiple hosts write in parallel, but otherwise it works pretty well
> (in my limited experience with GFS2). All of the participating hosts
> need some kind of shared access directly to the same block device:
> I've used iSCSI and DRBD, but I think FC is common, too. Each host
> also runs a DLM (distributed lock manager) daemon, which sends and
> receives info about which hosts are writing to which inodes, so they
> can all keep their caches consistent with the disk. (better
> explanation on WP, here:
> http://en.wikipedia.org/wiki/Global_File_System#Differences_from_a_local_filesystem)
>
> From my limited understanding of how iSCSI, GFS2 work, and dm-crypt
> work, I have no idea how they'll cooperate with each other, though.
> I'd like to test it, when I get a chance, but it would be nice to know
> a little more, in advance.

As I said, GFS2 does not care whether the block device it
runs on is directly hardware or in any way transformed
by LVM/dm-crypt, whatever. The only practical experience
I have is RAID1/5/6 -> dm-crypt -> ext3 -> NFS export. I have not 
noticed any additional problems there in several years of 
doing it. There are of course the individual problems of
the layers. E.g. slow access because files are heavily
fragmented (don't ask) or NFS (old version) being unreliable
when not used with TCP.

You can get pretty bad "synergies" even locally though.
Once I had  XFS on a Linux software RADI5 and they interacted
so badly when both did a resync/FS check, that it would have
taken months to finish. Basically both seemd to work, then 
backed off, then tried again, backed off.... The RAID resync 
speed was arounf 10kB/sec.

I think unless somebody has the exact same configuration and
workload as you do, you really need to try ot out.

One thing though: If this works currently, and you just want
to put in the dm-crypt layer, chances are it is still going to 
work with dm-cryopt.

Arno

 
> Ryan B. Lynch
> ryan.b.lynch@gmail.com
> _______________________________________________
> dm-crypt mailing list
> dm-crypt@saout.de
> http://www.saout.de/mailman/listinfo/dm-crypt
> 

-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Ryan Lynch]

On Sat, Apr 24, 2010 at 11:55, Arno Wagner <arno@wagner.name> wrote:
> On Fri, Apr 23, 2010 at 05:20:05PM -0400, Ryan Lynch wrote:
>> Would it be possible to run a shared-disk, clustered filesystem over a
>> dm-crypt block device, which in turn runs on a shared iSCSI device?
>> I'd be interested in knowing if anyone has tried, or has theoretical
>> knowledge of why it would (not) work.

> As I said, GFS2 does not care whether the block device it
> runs on is directly hardware or in any way transformed
> by LVM/dm-crypt, whatever. The only practical experience
> I have is RAID1/5/6 -> dm-crypt -> ext3 -> NFS export. I have not
> noticed any additional problems there in several years of
> doing it. There are of course the individual problems of
> the layers. E.g. slow access because files are heavily
> fragmented (don't ask) or NFS (old version) being unreliable
> when not used with TCP.

This seems logical, today. After a good night's sleep, and considering
your response, the source of my trepidation is a little clearer to me.
I think I should be asking two specific questions:

    1) Do dm-crypt's individual cipher chains correspond to disk
sectors? To put that another way, can a single chain encompass a
larger area of disk? If larger chains are possible, can a cipher chain
ever cross outside the boundaries of GFS2's locking granularity?

    2) Is the LUKS/dm-crypt "mounting" (?) attachment process
read-only and immune to concurrency problems? (Sorry, I'm not sure of
the right terminology, here--"mount" is wrong, there's no FS involved,
but what is the process of setting up the virtual block device
called?)

Based on what I've read, so far, I believe there's no problem WRT
question #1. But I really don't know about question #2.


> You can get pretty bad "synergies" even locally though.
> Once I had  XFS on a Linux software RADI5 and they interacted
> so badly when both did a resync/FS check, that it would have
> taken months to finish. Basically both seemd to work, then
> backed off, then tried again, backed off.... The RAID resync
> speed was arounf 10kB/sec.

I'm prepared for a pretty big performance hit. For my application, 2x
hosts at >= 1 MB/sec, each, for relatively local file reads/writes w/o
any lock contention is fine. Against a SATA RAID array over a single
GigE switch, I can do 100x that. I haven't even really started to
worry about this issue, yet.


> I think unless somebody has the exact same configuration and
> workload as you do, you really need to try ot out.

I'll be happy if it works and passes some basic tests, but I don't
have the knowledge or resources to exhaustively test it. Someone more
knowledgeable would have to tell me whether there are any hidden
cryptographic risks, or potential crash/data loss bugs on uncommon
corner cases that my tests don't cover.

Thanks for the feedback.

-Ryan

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Arno Wagner]

On Sat, Apr 24, 2010 at 06:41:30PM -0400, Ryan Lynch wrote:
> On Sat, Apr 24, 2010 at 11:55, Arno Wagner <arno@wagner.name> wrote:
> > On Fri, Apr 23, 2010 at 05:20:05PM -0400, Ryan Lynch wrote:
> >> Would it be possible to run a shared-disk, clustered filesystem over a
> >> dm-crypt block device, which in turn runs on a shared iSCSI device?
> >> I'd be interested in knowing if anyone has tried, or has theoretical
> >> knowledge of why it would (not) work.
> 
> > As I said, GFS2 does not care whether the block device it
> > runs on is directly hardware or in any way transformed
> > by LVM/dm-crypt, whatever. The only practical experience
> > I have is RAID1/5/6 -> dm-crypt -> ext3 -> NFS export. I have not
> > noticed any additional problems there in several years of
> > doing it. There are of course the individual problems of
> > the layers. E.g. slow access because files are heavily
> > fragmented (don't ask) or NFS (old version) being unreliable
> > when not used with TCP.
> 
> This seems logical, today. After a good night's sleep, and considering
> your response, the source of my trepidation is a little clearer to me.

Good. 

> I think I should be asking two specific questions:
> 
>     1) Do dm-crypt's individual cipher chains correspond to disk
> sectors? To put that another way, can a single chain encompass a
> larger area of disk? If larger chains are possible, can a cipher chain
> ever cross outside the boundaries of GFS2's locking granularity?

Should all be 512 byte blocks, as these are the smallest
addressable units when talking to disks. All larger units 
come from the filesystems, i.e. above dm-crypt. The dm-crypt
documentation also stipulates 512 byte sectors.
 
>     2) Is the LUKS/dm-crypt "mounting" (?) attachment process
> read-only and immune to concurrency problems? (Sorry, I'm not sure of
> the right terminology, here--"mount" is wrong, there's no FS involved,
> but what is the process of setting up the virtual block device
> called?)

I would propose to call the process "mapping".

AFAIK dm-crypt mapping is completely no-disk-access, i.e. not even 
read as there is no metadata on disk. Well, to be exact, the 
partition table gets read if you map a partition. I am not sure 
about LUKS.

> Based on what I've read, so far, I believe there's no problem WRT
> question #1. But I really don't know about question #2.

LUKS could be a problem with regard to #2. I agree on #1.

> > You can get pretty bad "synergies" even locally though.
> > Once I had ??XFS on a Linux software RADI5 and they interacted
> > so badly when both did a resync/FS check, that it would have
> > taken months to finish. Basically both seemd to work, then
> > backed off, then tried again, backed off.... The RAID resync
> > speed was arounf 10kB/sec.
> 
> I'm prepared for a pretty big performance hit. For my application, 2x
> hosts at >= 1 MB/sec, each, for relatively local file reads/writes w/o
> any lock contention is fine. Against a SATA RAID array over a single
> GigE switch, I can do 100x that. I haven't even really started to
> worry about this issue, yet.

Sounds sensible to me with requirements this low. My intuition is
that this should work with more than a bit of margin, even if there
are some performance problems.

> > I think unless somebody has the exact same configuration and
> > workload as you do, you really need to try ot out.
> 
> I'll be happy if it works and passes some basic tests, but I don't
> have the knowledge or resources to exhaustively test it. Someone more
> knowledgeable would have to tell me whether there are any hidden
> cryptographic risks, or potential crash/data loss bugs on uncommon
> corner cases that my tests don't cover.

There are no crypto-risks that I can see. There should also not be 
crash or data-loss issues (other than the combined ones of the 
layers). The only risk I see is potentially low performance, but
even that may not be a problem for your requirements.

One data-loss risk I see is that dm-crypt does not "sync" until
unmapped (found that out when zeroing a new dm-crypt partition,
the start of the partition only got overwritten after unmapping).
So for a hard sync where you need to be sure the data is on disk,
you would need to remove the mapping. But this is already 
problematic with modern disks in a power-loss scenario. You do
not really get a write-through these days. But if you force
disk flushes by umounting (the only reliable way these days, it
seems), remember to unmap dm-crypt as well.

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Ryan Lynch]

On Sat, Apr 24, 2010 at 20:19, Arno Wagner <arno@wagner.name> wrote:
> On Sat, Apr 24, 2010 at 06:41:30PM -0400, Ryan Lynch wrote:
>>     2) Is the LUKS/dm-crypt "mounting" (?) attachment process
>> read-only and immune to concurrency problems? (Sorry, I'm not sure of
>> the right terminology, here--"mount" is wrong, there's no FS involved,
>> but what is the process of setting up the virtual block device
>> called?)
>
> I would propose to call the process "mapping".
>
> AFAIK dm-crypt mapping is completely no-disk-access, i.e. not even
> read as there is no metadata on disk. Well, to be exact, the
> partition table gets read if you map a partition. I am not sure
> about LUKS.
>
>> Based on what I've read, so far, I believe there's no problem WRT
>> question #1. But I really don't know about question #2.
>
> LUKS could be a problem with regard to #2. I agree on #1.

I found the LUKS specification
(http://code.google.com/p/cryptsetup/wiki/Specification), and it
doesn't seem like there are any write operations to the LUKS header
during a normal mapping. Initialization and key management, yes, so I
might have to be careful if I want to add/change/revoke a passphrase,
but I haven't finished digesting that part, yet.

Also, the man page for 'cryptsetup' mentions an interesting option:

       --non-exclusive

              This  option  is  ignored. Non-exclusive access to the
same block device can cause data corruption thus this mode is no
              longer supported by cryptsetup.

There are references to it in the release notes for 1.07 and 1.10, the
latter also mentioning that the option is currently disabled. There's
a ticket mentioning it that seems to explain a little more
(http://code.google.com/p/cryptsetup/issues/detail?id=34&can=1). I'm
still trying to understand it, right now.

I also had an idea to test the basic concept re: LUKS, with a much
simpler local configuration:
    - Allocate a file big enough to hold a LUKS/dm-crypt volume (a few
10s of MBs should be enough).
    - Set up a first loopback device on the new file.
    - Set up a second loopback device, backed by either the file or
the first loopback device.
    - Initialize an encrypted volume via the first loop device.
    - Checksum the "raw" LUKS header via the first loop device (e.g.,
`dd ... | md5sum`, can't remember the exact header size).
    - Map the encrypted volume via the first loop device..
    - Checksum the "raw" LUKS header, again, via the first loop
device, and compare it to the original. If it's changed, we have a
problem.
    - Map the encrypted volume on the second loop device.But
    - Checksum the "raw" LUKS header, via both loop devices, and
compare w/ previous. If it's changed, we have a problem.
    - Repeat w/ un-mapping operations, too.

I'm a little worried that this might be the exact situation that the
'--non-exclusive' option was supposed to handle. If it doesn't work,
I'm probably just going to set up an iSCSI target and attach from two
different machines. Less convenient, but if this works, I'll have to
do it, anyway.

(BTW: If gmazyland happens to read this, and he doesn't mind
explaining this for me, I would really appreciate the help.)


> One data-loss risk I see is that dm-crypt does not "sync" until
> unmapped (found that out when zeroing a new dm-crypt partition,
> the start of the partition only got overwritten after unmapping).
> So for a hard sync where you need to be sure the data is on disk,
> you would need to remove the mapping. But this is already
> problematic with modern disks in a power-loss scenario. You do
> not really get a write-through these days. But if you force
> disk flushes by umounting (the only reliable way these days, it
> seems), remember to unmap dm-crypt as well.

I don't think I understand this part very well: Is dm-crypt operating
synchronously WRT the underlying device? If it's really async, then I
don't think any of this is going to work, unless it's possible to
disable the block layer cache. Each host's lock manager daemon
notifies the other hosts when it performs any kind of write operation,
so that if the other hosts can drop any affected bits from their FS
caches. But the lock managers aren't aware of the block layer cache
(if any), so wouldn't that create the potential for concurrency
problems?

I just glanced over the DM I/O docs, in the kernel source. Looks like
sync and async are both supported. Just when I thought I had this
thing figured out...

-Ryan

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Milan Broz]

Hi,

short summary because there is mix of problems and dangerous ideas :-)

1) if you have dm-crypt device, and exported plaintext block device
above dm-crypt using iSCSI od DRBD (IOW the encryption run only on one place)
to several nodes - this should work (in theory, not sure if anyone using that).
(using cluster aware FS is required here but that's another level)

(note you are exporting "plaintext" device - so maybe this vetoes
your use case)


2) you should not export underlying (ciphertext) block device
to several nodes and activate dm-crypt layer on several nodes in parallel
(iow every node run its own block lever encryption).

dm-crypt was neither designed nor tested to be cluster aware. With the crypto
queue inside (which works asynchronously) you will probably hit data corruption.
(And I never saw request for cluster support here, need to think what
is needed to change here...)


3) LUKS is only key management system, you can remove it from
picture completely here for simplification.
(FYI there is no write request during LUKS activation on LUKS metadata)


On 04/25/2010 07:12 AM, Ryan Lynch wrote:
> Also, the man page for 'cryptsetup' mentions an interesting option:
> 
>        --non-exclusive
> 
>               This  option  is  ignored. Non-exclusive access to the
> same block device can cause data corruption thus this mode is no
>               longer supported by cryptsetup.
> 
> There are references to it in the release notes for 1.07 and 1.10

Yes, I mentioned it here: http://code.google.com/p/cryptsetup/wiki/Cryptsetup110

    * Removes support for dangerous non-exclusive option 
    (it is ignored now, LUKS device must be always opened exclusive) 

Multiple problems here, but probably the most important is that
system see two devices - managing cache separately, so you can
get data corruption - one device see old data, (f)sync will not help here.

> I also had an idea to test the basic concept re: LUKS, with a much
> simpler local configuration:
>     - Allocate a file big enough to hold a LUKS/dm-crypt volume (a few
> 10s of MBs should be enough).
>     - Set up a first loopback device on the new file.
>     - Set up a second loopback device, backed by either the file or
> the first loopback device.

waste of time, it will not work for *data* access properly
(here probably loop stop it anyway).
(and LUKS metadata are written using direct-io, and read-only
for activation, so you see it works in that case - but it is not important here.)

Adding loop to the mix will cause another level of headache, you have been warned ;-)

>> One data-loss risk I see is that dm-crypt does not "sync" until
>> unmapped (found that out when zeroing a new dm-crypt partition,

What do you mean? If you need sync, issue it explicitly, or use direct-io.
(try dd oflag=direct ... )

There is barrier concept and device-mapper fully supports it now.
(fsync() issues barrier request in the end for the device and waits for it,
it is in core device-mapper, no need additional code in dm-crypt)

>> the start of the partition only got overwritten after unmapping).
>> So for a hard sync where you need to be sure the data is on disk,
>> you would need to remove the mapping.

No. see above.


> I don't think I understand this part very well: Is dm-crypt operating
> synchronously WRT the underlying device?

Basically, no. But it will sync if it requested.
Not sure if I understand which level you mean, but
there is queue for crypto requests and these are processed later.

Milan

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Arno Wagner]

On Sun, Apr 25, 2010 at 11:38:01AM +0200, Milan Broz wrote:
[...]
> >> the start of the partition only got overwritten after unmapping).
> >> So for a hard sync where you need to be sure the data is on disk,
> >> you would need to remove the mapping.
> 
> No. see above.

Ah. So when I saw old data in the raw device, that really
was cache content, not device content? 

Arno
-- 
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email: arno@wagner.name 
GnuPG:  ID: 1E25338F  FP: 0C30 5782 9D93 F785 E79C  0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

If it's in the news, don't worry about it.  The very definition of 
"news" is "something that hardly ever happens." -- Bruce Schneier 

Re: [dm-crypt] Sharing encrypted block devices, for GFS2 over iSCSI?

[Ryan Lynch]

On Sun, Apr 25, 2010 at 05:38, Milan Broz <mbroz@redhat.com> wrote:
> short summary because there is mix of problems and dangerous ideas :-)

Milan, first of all, thank you for taking the time to help me
understand this. It's intimidating for me to approach a complex topic
like this, from a cold start. Your input is invaluable.


> Multiple problems here, but probably the most important is that
> system see two devices - managing cache separately, so you can
> get data corruption - one device see old data, (f)sync will not help here.

So, if I under stand you correctly, the problem is with *READS FROM*
invalidated cache, not writes to cache that don't hit the disk? Let me
see if I get this right:

    - I have an unmapped encrypted volume, /dev/cipher.
    - I create two identical dm-crypt mappings, one as /dev/plainA,
the other as /dev/plainB.
    - I read the first 512 bytes of /dev/plainA, which caches
/dev/cipher's bytes 0-512 in the /dev/plainA-specific block buffer.
    - I write the first 512 bytes of /dev/plainB, using the direct-io
flag, which alters /dev/cipher's bytes 0-512, but /dev/plainA's block
buffer doesn't get notified that its cache is now invalid.
    - I read the first 512 bytes of /dev/plainA, again, and see the
unchanged invalid cache contents instead of the updated contents on
the underlying /dev/cipher.

I understand how O_DIRECT affects write operations. But I can't find a
clear reference about how it affects reads. Do block device reads
still get buffered, or do read requests always go directly to the
underlying device? In the above example, if I passed the the 'direct'
or 'cio' flags to 'dd' for the read operations, would I be reading
from the cache or directly from underlying device?

As for a cluster FS like GFS2, I believe the lock managers take
responsibility for dropping their invalid read caches when one host
notifies the others of a write operation. Otherwise, how would GFS2
work, at all?

-Ryan