Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?

Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?

[Lance Reed]

I am attempting to setup encrypted user home directories via eCryptfs using
gluster as a backend.

Very simple setup currently has a small two node gluster cluster mounted by
a separate client.  Normal gluster client and NFS mount / file options are
working fine.


e.g. https://wiki.archlinux.org/index.php/ECryptfs#Encrypting_a_home_directory

In my attempts lay ecryptfs on top of the mounted native gluster setup, I am
unable to edit a file, write etc.  I either get zero length or fixed sizes.

Only log messages I get are:
"Either the lower file is not in a valid eCryptfs format, or the key could
not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO"

I am posting in this forum to see if anyone knows of any reason why this may
be failing from the ecryptfs side and I should stop banging my head against
the wall...

I am trying Centos / RHEL.
See these bugs:
Bug 762976 - (GLUSTER-1244) ecryptfs does not work when the directory to be
encrypted is on gluster mount
https://bugzilla.redhat.com/show_bug.cgi?id=762976

A non-empty file created on glusterfs with ecryptfs reports as a file of
size zero
https://bugzilla.redhat.com/show_bug.cgi?id=989702#c1

These look to be issues with O_DIRECT usage in fuse but are suppose to be
fixed now.

I was hoping someone might have an idea or remember some of this to help me
figure out if using glusterfs for a backend with  eCryptfs is even an option.

Is it possible that this bug is still the core problem?
"ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs"
https://bugs.launchpad.net/ecryptfs/+bug/277578
It is old but still seems to be open..

versions of the code I am using:
glusterfs-cli-3.4.2-1.el6.x86_64
glusterfs-libs-3.4.2-1.el6.x86_64
glusterfs-fuse-3.4.2-1.el6.x86_64
glusterfs-server-3.4.2-1.el6.x86_64
ecryptfs-utils-82-6.el6_1.3.x86_64
glusterfs-3.4.2-1.el6.x86_64


Thanks very much in advance for any ideas on the problem.

Re: Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?

[Tyler Hicks]

[-- Attachment #1: Type: text/plain, Size: 2791 bytes --]

Hi Lance!

On 2014-02-26 16:06:20, Lance Reed wrote:
> I am attempting to setup encrypted user home directories via eCryptfs using
> gluster as a backend.
> 
> Very simple setup currently has a small two node gluster cluster mounted by
> a separate client.  Normal gluster client and NFS mount / file options are
> working fine.
> 
> 
> e.g. https://wiki.archlinux.org/index.php/ECryptfs#Encrypting_a_home_directory
> 
> In my attempts lay ecryptfs on top of the mounted native gluster setup, I am
> unable to edit a file, write etc.  I either get zero length or fixed sizes.
> 
> Only log messages I get are:
> "Either the lower file is not in a valid eCryptfs format, or the key could
> not be retrieved. Plaintext passthrough mode is not enabled; returning -EIO"
> 
> I am posting in this forum to see if anyone knows of any reason why this may
> be failing from the ecryptfs side and I should stop banging my head against
> the wall...
> 
> I am trying Centos / RHEL.
> See these bugs:
> Bug 762976 - (GLUSTER-1244) ecryptfs does not work when the directory to be
> encrypted is on gluster mount
> https://bugzilla.redhat.com/show_bug.cgi?id=762976
> 
> A non-empty file created on glusterfs with ecryptfs reports as a file of
> size zero
> https://bugzilla.redhat.com/show_bug.cgi?id=989702#c1
> 
> These look to be issues with O_DIRECT usage in fuse but are suppose to be
> fixed now.
> 
> I was hoping someone might have an idea or remember some of this to help me
> figure out if using glusterfs for a backend with  eCryptfs is even an option.

eCryptfs mounted on top of glusterfs is something that I've never tried
and I don't recall anyone talking with upstream eCryptfs about it,
either. It wouldn't surprise me if it doesn't work. :/

I haven't paid much attention to glusterfs, but I thought the answer to
encryption with glusterfs was hekafs?

While briefly refreshing my memory on hekafs, it sounds like it is
geared towards cloud storage providers. Maybe it is too complex for your
needs?

> 
> Is it possible that this bug is still the core problem?
> "ecryptfs does not work properly over nfs, cifs, samba, WebDAV, or aufs"
> https://bugs.launchpad.net/ecryptfs/+bug/277578
> It is old but still seems to be open..

That bug is a mess. It needs to be reevaluated and split up into
separate bug reports for individual lower filesystems. There is no
single fix for that bug and it will never be closed in its current
state.

> 
> versions of the code I am using:
> glusterfs-cli-3.4.2-1.el6.x86_64
> glusterfs-libs-3.4.2-1.el6.x86_64
> glusterfs-fuse-3.4.2-1.el6.x86_64
> glusterfs-server-3.4.2-1.el6.x86_64
> ecryptfs-utils-82-6.el6_1.3.x86_64
> glusterfs-3.4.2-1.el6.x86_64

Kernel version?

Tyler

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?

[Lance Reed]

Tyler,

Thanks for the response!
Red Hat Enterprise Linux Server release 6.4 (Santiago) (and Centos 6.4)
2.6.32-358.14.1.el6.x86_64

I am actually planning to use this in a cloud setup.  Mostly for an internal
shared NFS like but HA space.
I had thought HekaFS had gone a bit dormant...(not a lot of talk since May
2013...)
I will look at it again.  Thanks for the tip and taking a look.
Any thought would be greatly appreciated.

Cheers,

Lance


If this is a double post, I apologize.. oddness on email vs web interface:

Re: Can anyone confirm or deny if ecryptfs will work with a glusterfs backend?

[Tyler Hicks]

[-- Attachment #1: Type: text/plain, Size: 1695 bytes --]

On 2014-02-26 18:35:09, Lance Reed wrote:
> Tyler,
> 
> Thanks for the response!
> Red Hat Enterprise Linux Server release 6.4 (Santiago) (and Centos 6.4)
> 2.6.32-358.14.1.el6.x86_64

While I've not made any recent changes that I would expect to improve
glusterfs support, that is a somewhat old kernel and a newer version
may, obviously, have different results.

However, I should say that even if eCryptfs suddenly works on top of
glusterfs in a newer kernel, there's still a fundamental problem with
eCryptfs stacked on top of any type of remote filesystem. eCryptfs does
not expect the lower/backend filesystem to be modified by another
machine. So you could have cache coherency problems if two or more
clients are reading a file and then one of them updates the file.

Fixing the cache coherency issues would be step 2, after eCryptfs
actually starts working on top of popular remote filesystems (nfs, cifs,
glusterfs, etc.). Unfortunately, I'm not aware of anyone working on step
1 at the moment.

Tyler

> 
> I am actually planning to use this in a cloud setup.  Mostly for an internal
> shared NFS like but HA space.
> I had thought HekaFS had gone a bit dormant...(not a lot of talk since May
> 2013...)
> I will look at it again.  Thanks for the tip and taking a look.
> Any thought would be greatly appreciated.
> 
> Cheers,
> 
> Lance
> 
> 
> If this is a double post, I apologize.. oddness on email vs web interface:
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe ecryptfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]