Selective Compression/Encryption

Selective Compression/Encryption

[Lee Trager]

Currently compression and I assume if encryption is implemented it is
turned on or off during mount. There are however many times when a user may
want to select which files/directories they want to compress or encrypt.
This will also be helpful when implementing btrfs support in grub for
example. We can say the disk can be compressed/encrypted except for /boot so 
compression/encryption doesn't have to be implemented in grub.

I was thinking of adding this functionality to the userspace application
btrfstune. The way I was thinking of doing this is when btrfstune +c is
applied to a directory or file the directory(and all its contents) or
file will always be compressed reguardless of how the filesystem is
mounted. The opposite would happen when btrfstune -c is used.

Would this be a reasonable thing to implement? Any suggestions before I
start doing this?

Lee

Re: Selective Compression/Encryption

[Miguel Figueiredo Mascarenhas Sousa Filipe]

Hi there,

On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
> Currently compression and I assume if encryption is implemented it is
> turned on or off during mount. There are however many times when a user may
> want to select which files/directories they want to compress or encrypt.
> This will also be helpful when implementing btrfs support in grub for
> example. We can say the disk can be compressed/encrypted except for /boot so
> compression/encryption doesn't have to be implemented in grub.
>
> I was thinking of adding this functionality to the userspace application
> btrfstune. The way I was thinking of doing this is when btrfstune +c is
> applied to a directory or file the directory(and all its contents) or
> file will always be compressed reguardless of how the filesystem is
> mounted. The opposite would happen when btrfstune -c is used.
>
> Would this be a reasonable thing to implement? Any suggestions before I
> start doing this?
>

Things like compression or encription should be used at the "volume" level.
So.. if a user wants a specific set of files or dirs ..they should
create a mount-point/volume like:

private_vol
bigarchives_vol

and set those volumes as compressed or encripted volumes

Regarding usability, the best would be for the sub-volume creation
tool to optionally allow passing encription/compression arguments.


and then:
 should mount those volumes somewhere like: ~/Confidential or ~/Archives.

Basically, do it at the directory level (which in btrfs is at the
sub-volume level).
File-level granularity is totally unmanageable in the long term.

Kind regards,


-- 
Miguel Sousa Filipe

Re: Selective Compression/Encryption

[Chris Mason]

On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> Currently compression and I assume if encryption is implemented it is
> turned on or off during mount. There are however many times when a user may
> want to select which files/directories they want to compress or encrypt.
> This will also be helpful when implementing btrfs support in grub for
> example. We can say the disk can be compressed/encrypted except for /boot so 
> compression/encryption doesn't have to be implemented in grub.
> 
> I was thinking of adding this functionality to the userspace application
> btrfstune. The way I was thinking of doing this is when btrfstune +c is
> applied to a directory or file the directory(and all its contents) or
> file will always be compressed reguardless of how the filesystem is
> mounted. The opposite would happen when btrfstune -c is used.
> 

This was my plan, but btrfstune probably isn't the best program to do it
(the ext2 tune program is mostly aimed at the super block level things).

I think it would be better to make a setattr style program to call the
ioctls.  There is already a per file compression flag, and the code
should already be checking it.

-chris

Re: Selective Compression/Encryption

[Lee Trager]

On Tue, Dec 09, 2008 at 05:22:18PM +0100, Christian Hesse wrote:
> On Tuesday 09 December 2008, Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
> > Hi there,
> >
> > On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
> > > Currently compression and I assume if encryption is implemented it is
> > > turned on or off during mount. There are however many times when a user
> > > may want to select which files/directories they want to compress or
> > > encrypt. This will also be helpful when implementing btrfs support in
> > > grub for example. We can say the disk can be compressed/encrypted except
> > > for /boot so compression/encryption doesn't have to be implemented in
> > > grub.
> 
> You could just use an additional partition for /boot that has compression an 
> encryption disabled...
>
Yes you could but many distributions are moving away from that such as
ubuntu. Also while this works well on desktop and server environments
where space is not an issue on embedded systems where free space is
measured in kb or mb it starts becomming a huge issue.
 
> > > I was thinking of adding this functionality to the userspace application
> > > btrfstune. The way I was thinking of doing this is when btrfstune +c is
> > > applied to a directory or file the directory(and all its contents) or
> > > file will always be compressed reguardless of how the filesystem is
> > > mounted. The opposite would happen when btrfstune -c is used.
> > >
> > > Would this be a reasonable thing to implement? Any suggestions before I
> > > start doing this?
> >
> > Things like compression or encription should be used at the "volume" level.
> 
> That was what I said some time ago when I asked why encryption support for 
> btrfs is planned. On a volume level you can use dm-crypt and the fs can 
> ignore that part completely.
> 
> The answer was that different users on a home partition could use their own 
> encryption key. That sounds like volume level is out of bet. ;)
> 
It does seem that doing it with volumes would limit user control and add
lots of complexity to such a simple task.

Miguel, what is your reason for suggesting this be done at the volume level?

> > So.. if a user wants a specific set of files or dirs ..they should
> > create a mount-point/volume like:
> >
> > private_vol
> > bigarchives_vol
> >
> > and set those volumes as compressed or encripted volumes
> >
> > Regarding usability, the best would be for the sub-volume creation
> > tool to optionally allow passing encription/compression arguments.
> >
> >
> > and then:
> >  should mount those volumes somewhere like: ~/Confidential or ~/Archives.
> >
> > Basically, do it at the directory level (which in btrfs is at the
> > sub-volume level).
> > File-level granularity is totally unmanageable in the long term.
> >
> > Kind regards,
> 
> -- 
> Regards,
> Chris

Lee

Re: Selective Compression/Encryption

[Lee Trager]

Would you suggest using the existing chattr/lsattr commands from
e2fsprogs for userspace control and just add support at the kernel
level?

Miguel suggested this be done at the volume level. Do you have any
thoughts on that?

Thanks,

Lee

On Tue, Dec 09, 2008 at 11:35:16AM -0500, Chris Mason wrote:
> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> > Currently compression and I assume if encryption is implemented it is
> > turned on or off during mount. There are however many times when a user may
> > want to select which files/directories they want to compress or encrypt.
> > This will also be helpful when implementing btrfs support in grub for
> > example. We can say the disk can be compressed/encrypted except for /boot so 
> > compression/encryption doesn't have to be implemented in grub.
> > 
> > I was thinking of adding this functionality to the userspace application
> > btrfstune. The way I was thinking of doing this is when btrfstune +c is
> > applied to a directory or file the directory(and all its contents) or
> > file will always be compressed reguardless of how the filesystem is
> > mounted. The opposite would happen when btrfstune -c is used.
> > 
> 
> This was my plan, but btrfstune probably isn't the best program to do it
> (the ext2 tune program is mostly aimed at the super block level things).
> 
> I think it would be better to make a setattr style program to call the
> ioctls.  There is already a per file compression flag, and the code
> should already be checking it.
> 
> -chris
> 

Re: Selective Compression/Encryption

[Joshua J. Berry]

On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> > Currently compression and I assume if encryption is implemented it is
> > turned on or off during mount. There are however many times when a user
> > may want to select which files/directories they want to compress or
> > encrypt. This will also be helpful when implementing btrfs support in
> > grub for example. We can say the disk can be compressed/encrypted except
> > for /boot so compression/encryption doesn't have to be implemented in
> > grub.
> >
> > I was thinking of adding this functionality to the userspace application
> > btrfstune. The way I was thinking of doing this is when btrfstune +c is
> > applied to a directory or file the directory(and all its contents) or
> > file will always be compressed reguardless of how the filesystem is
> > mounted. The opposite would happen when btrfstune -c is used.
>
> This was my plan, but btrfstune probably isn't the best program to do it
> (the ext2 tune program is mostly aimed at the super block level things).
>
> I think it would be better to make a setattr style program to call the
> ioctls.  There is already a per file compression flag, and the code
> should already be checking it.

Is there some reason this can't be done with the existing extended attribute 
facilities?

It seems like xattrs would be preferable to some btrfs-specific tunable, as 
programs like rsync or backup tools would be able to preserve (and restore) 
these bits with no extra work required.

-- Josh


-- 
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

Re: Selective Compression/Encryption

[jim owens]

Joshua J. Berry wrote:
> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
>>> Currently compression and I assume if encryption is implemented it is
>>> turned on or off during mount. There are however many times when a user
>>> may want to select which files/directories they want to compress or
>>> encrypt. This will also be helpful when implementing btrfs support in
>>> grub for example. We can say the disk can be compressed/encrypted except
>>> for /boot so compression/encryption doesn't have to be implemented in
>>> grub.
>>>
>>> I was thinking of adding this functionality to the userspace application
>>> btrfstune. The way I was thinking of doing this is when btrfstune +c is
>>> applied to a directory or file the directory(and all its contents) or
>>> file will always be compressed reguardless of how the filesystem is
>>> mounted. The opposite would happen when btrfstune -c is used.
>> This was my plan, but btrfstune probably isn't the best program to do it
>> (the ext2 tune program is mostly aimed at the super block level things).
>>
>> I think it would be better to make a setattr style program to call the
>> ioctls.  There is already a per file compression flag, and the code
>> should already be checking it.
> 
> Is there some reason this can't be done with the existing extended attribute 
> facilities?
> 
> It seems like xattrs would be preferable to some btrfs-specific tunable, as 
> programs like rsync or backup tools would be able to preserve (and restore) 
> these bits with no extra work required.

I had the same thought... that many btrfs per-file tunables would
be better implemented in xattrs (I've read christoph's earlier
objections and hope to get around those issues).

I have been working on changing the xattr code with the first
step getting it functioning properly when selinux is enabled
so we can see just how costly btrfs xattrs are in actual use.

jim

Re: Selective Compression/Encryption

[Miguel Figueiredo Mascarenhas Sousa Filipe]

On Tue, Dec 9, 2008 at 6:09 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
> On Tue, Dec 09, 2008 at 05:22:18PM +0100, Christian Hesse wrote:
>> On Tuesday 09 December 2008, Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
>> > Hi there,
>> >
>> > On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
>> > > Currently compression and I assume if encryption is implemented it is
>> > > turned on or off during mount. There are however many times when a user
>> > > may want to select which files/directories they want to compress or
>> > > encrypt. This will also be helpful when implementing btrfs support in
>> > > grub for example. We can say the disk can be compressed/encrypted except
>> > > for /boot so compression/encryption doesn't have to be implemented in
>> > > grub.
>>
>> You could just use an additional partition for /boot that has compression an
>> encryption disabled...
>>
> Yes you could but many distributions are moving away from that such as
> ubuntu. Also while this works well on desktop and server environments
> where space is not an issue on embedded systems where free space is
> measured in kb or mb it starts becomming a huge issue.
>
>> > > I was thinking of adding this functionality to the userspace application
>> > > btrfstune. The way I was thinking of doing this is when btrfstune +c is
>> > > applied to a directory or file the directory(and all its contents) or
>> > > file will always be compressed reguardless of how the filesystem is
>> > > mounted. The opposite would happen when btrfstune -c is used.
>> > >
>> > > Would this be a reasonable thing to implement? Any suggestions before I
>> > > start doing this?
>> >
>> > Things like compression or encription should be used at the "volume" level.
>>
>> That was what I said some time ago when I asked why encryption support for
>> btrfs is planned. On a volume level you can use dm-crypt and the fs can
>> ignore that part completely.
>>
>> The answer was that different users on a home partition could use their own
>> encryption key. That sounds like volume level is out of bet. ;)
>>
> It does seem that doing it with volumes would limit user control and add
> lots of complexity to such a simple task.
>
> Miguel, what is your reason for suggesting this be done at the volume level?

Well, basically, the main issue I have with file level configuration,
is maintenance nightmare in the long term.

Redundancy level settings (raid0,1,5,6..), and compression/encription,
to me, seem like a sub-volume setting.
Its the right granularity to manage and mantain data and interact with it.
If I want this folders compressed.. I can just move them to my
compressed sub-vol.
If I want files: a, b/weird_path/D, and NSA/* confidential, I put them
in my encripted sub-vol.

Imagine working on a confidencial project, where you add new
files/folders practilly everyday, and every single day, you must
remember to (and bother to do extra word so that..) set that file as a
encripted one.

Also, by using sub-vols like this, and if redudancy is applied at the
same level, things just play nice, I can have distinct mount-points
that represent data that I want with a given set of
redundancy/compression and encription rules.

It becomes easier to mantain, and use.

Just imagine having a "confidencial" or a "compressed" mountpoint/disk
 ICON in you desktop, and just knowing that files in there are
encripted and/or compressed.
Versus, having to browser your file and look for a specific
"encripted" or "compressed" flag, that the file explorer shows if they
play nice...

Now... if its at the file level, It can be troublesome to look for
allfiles encripted/compressed.
Also, knowing the encription key to each file (if it's set at the file
level) ... argh!!


Doing this file based, seems to me that will lead to complication the
long term maintenance of things.

In the long term, when someone picks up, or mounts a old, used system,
I can't imagine the butload of work involved in finding out things
like:
- what in this 4TB fs is compressed, what is encripted?
- (in case of redundancy at the file level): how many redundancy
levels do I have, and which files respectively?
- why is rsync/scp 'ing this folder/BIG_NESTED_TREE_OF_FILES allways
"stop" at folder/**/fileB and folder/z*/* ?
 (might it be that some unsuspecting file in there has a trully beasty
compression ratio, and that 4GB fileB (in the middle of other 4GB
files), is instead a 80Gb file compressed ?


I also don't like extended attributes, we're just creating a whole new
orthogonal dimension/namespace were we store/manipulate information..
it also complicates backups (does the backup system understand and
respects ext. attrs?)

It's messier doing these things with extended attributes or very
custom tools, instead of levelaging/creating a volume management tool,
for managing this, such tool will allready be there to manage the
fisical storage <-> filesystems/volumes mappings and settings.. might
has well leverage it.



Kind regards,



>
>> > So.. if a user wants a specific set of files or dirs ..they should
>> > create a mount-point/volume like:
>> >
>> > private_vol
>> > bigarchives_vol
>> >
>> > and set those volumes as compressed or encripted volumes
>> >
>> > Regarding usability, the best would be for the sub-volume creation
>> > tool to optionally allow passing encription/compression arguments.
>> >
>> >
>> > and then:
>> >  should mount those volumes somewhere like: ~/Confidential or ~/Archives.
>> >
>> > Basically, do it at the directory level (which in btrfs is at the
>> > sub-volume level).
>> > File-level granularity is totally unmanageable in the long term.
>> >
>> > Kind regards,
>>
>> --
>> Regards,
>> Chris
>
> Lee
>



-- 
Miguel Sousa Filipe

Re: Selective Compression/Encryption

[Diego Calleja]

El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <lt73@cs.drexel.edu> es=
cribi=C3=B3:

> It does seem that doing it with volumes would limit user control and =
add
> lots of complexity to such a simple task.

IMHO, WRT compression it's the contrary. Compression on a per-file basi=
s has
never been very succesful (just look at how many windows users use it)
because it implies taking a decision for every file on the system. OTOH=
,
volume-level is just a single option to be enabled.

I'm of course not arguing that file-level compression shouldn't be poss=
ible,
im just saying that is way more difficult to administer and that most p=
eople
(including sysadmins) is most likely to use compression in a per-volume=
 basis.
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Selective Compression/Encryption

[jim owens]

Diego Calleja wrote:
> El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <lt73@cs.drexel.edu> =
escribi=C3=B3:
>=20
>> It does seem that doing it with volumes would limit user control and=
 add
>> lots of complexity to such a simple task.
>=20
> IMHO, WRT compression it's the contrary. Compression on a per-file ba=
sis has
> never been very succesful (just look at how many windows users use it=
)
> because it implies taking a decision for every file on the system. OT=
OH,
> volume-level is just a single option to be enabled.
>=20
> I'm of course not arguing that file-level compression shouldn't be po=
ssible,
> im just saying that is way more difficult to administer and that most=
 people
> (including sysadmins) is most likely to use compression in a per-volu=
me basis.

While I have not gotten far enough to prove it is feasible...

My idea on controlling features like compression is that
the default mode is inherited from the parent in the
directory tree.  Thus you can turn it on/off at whatever
granularity you want.

jim
--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Selective Compression/Encryption

[calin]

> While I have not gotten far enough to prove it is feasible...
> 
> My idea on controlling features like compression is that
> the default mode is inherited from the parent in the
> directory tree.  Thus you can turn it on/off at whatever
> granularity you want.
> 

That seems like the ideal solution to me.  Similar to how default ACLs work.

-John

RE: Selective Compression/Encryption

[Mattos, Oliver]

SGF2aW5nIGEgMyBwb3NzaWJsZSBzdGF0ZXMgZm9yIGVhY2ggZmlsZSB3b3VsZCBzZWVtIHNlbnNp
YmxlOg0KDQoxLiAgQ29tcHJlc3Npb24gRW5hYmxlZCAtIHRoaXMgZmlsZSBvciBmb2xkZXIgd2ls
bCBiZSBjb21wcmVzc2VkLg0KMi4gIENvbXByZXNzaW9uIERpc2FibGVkIC0gdGhpcyBmaWxlIG9y
IGZvbGRlciB3aWxsIG5ldmVyIGJlIGNvbXByZXNzZWQuDQozLiAgTm90IFNwZWNpZmllZCAtIFRo
aXMgd2lsbCBpbmhlcml0IHRoZSBjb21wcmVzc2lvbiBzdGF0ZSBmcm9tIGl0J3MgcGFyZW50Lg0K
DQpUbyBrZWVwIHRoaXMgc3lzdGVtIGluIGEgY29uc2lzdGVudCBzdGF0ZSwgaWYgYSBmb2xkZXIg
aXMgY2hhbmdlZCBmcm9tIHNheSBzdGF0ZSAyIHRvIHN0YXRlIDEsIHRoZW4gYWxsIHRoZSBjaGls
ZHJlbiBvZiB0aGF0IGZvbGRlciB3aG8gYXJlIGluIHN0YXRlIDMgbmVlZCB0byBiZSBjb21wcmVz
c2VkIC0gcG90ZW50aWFsbHkgYSB2ZXJ5IHRpbWUgY29uc3VtaW5nIG9wZXJhdGlvbi4NCg0KSWYg
dGhpcyBvcGVyYXRpb24gaXMgY2FuY2VsbGVkLCBvciBmYWlscywgdGhlbiB0aGUgY29tcHJlc3Np
b24gc3RhdGVzIGluIHRoZSBmaWxlc3lzdGVtIHdpbGwgYmUgaW5jb25zaXN0ZW50LiAgTm8gaWRl
YSBob3cgaXQncyBiZXN0IHRvIHNvbHZlIHRoYXQgLSBlaXRoZXIganVzdCBjb3BlIHdpdGggdGhl
IGluY29uc2lzdGVuY3kgYW5kIGxldCBhIGZpbGVzeXN0ZW0gY2hlY2sgdGlkeSBpdCB1cCAocmVt
ZW1iZXIgdGhpcyBpc3N1ZSB3b3VsZCBuZXZlciBlbmRhbmdlciBkYXRhLCBhbmQgb25seSBvY2N1
cnMgaW4gdGhlIGNhc2Ugb2YgYSBzeXN0ZW0gZmFpbHVyZSB3aGlsZSBjaGFuZ2luZyB0aGUgY29t
cHJlc3Npb24gZmxhZ3Mgb24gYSBkaXJlY3RvcnkpLiAgQW4gYWx0ZXJuYXRpdmUgc29sdXRpb24g
Y291bGQgYmUgdG8gbWFrZSB0aGUgYXR0cmlidXRlIGNoYW5nZSBhc2NocnluY2hyb25vdXMgYW5k
IGhhdmUgYW4gIm9wZXJhdGlvbiBxdWV1ZSIsIHNvIGluIHRoZSBjYXNlIG9mIGEgc3lzdGVtIGZh
aWx1cmUgdGhlIG9wZXJhdGlvbiBjYW4gY29udGludWUgd2hlcmUgaXQgbGVmdCBvZmYuDQo=

Re: Selective Compression/Encryption

[Jeremy Sanders]

Miguel Figueiredo Mascarenhas Sousa Filipe wrote:

> Things like compression or encription should be used at the "volume"
> level. So.. if a user wants a specific set of files or dirs ..they should
> create a mount-point/volume like:
> 
> private_vol
> bigarchives_vol

Does a normal user have permissions to do this?

I could easily see the case where a normal user has a set of very
compressible files in a directory. They shouldn't have to get a sysadmin go
to the trouble of setting this up.

Jeremy

Re: Selective Compression/Encryption

[Miguel Figueiredo Mascarenhas Sousa Filipe]

Hi there,

On Tue, Dec 9, 2008 at 11:03 PM, Oliver Mattos
<oliver.mattos08@imperial.ac.uk> wrote:
> Hi,
>
> It looks to me like if all the attributes were per file, and files
> automatically inherited settings from the parent, and users weren't
> allowed to change inherited permissions (only root could), then you
> could get the same effect as per volume controls.
>
> The above could be enforced at the option of the sysAdmin\distro.
>
> On the other hand if the attributes are per volume then you can't
> individually set attributes for individual files unless you create one
> volume per file, which doesn't sound like a clean solution.
>
> As far as I can see, all the use cases that call for per-volume
> configuration are also satisfied by per file configuration with
> inheritance, but not vice versa.
>

You have convinced me.
If this works with default inheritance enabled, so that not only
child/leaf files of a dir tree get the dir setting, but also new
files, then, it fits my usage model and "wanted" features :)


>
> An example where I would want compressed and uncompressed files in the
> same folder is where I have a Virtual Machine with two disks, each
> stored in a disk image on a btrfs partition - disk 1 is the root
> filesystem and compressed to save space, since it contains a lot of
> "zero" bytes, but disk 2 is the virtual swap disk, which isn't
> compressed due to needing fast low overhead access.  Since both files
> are associated with the same virtual machine I want to keep them in the
> same folder on the same volume.
>

That's a good realistic scenario were per-volume settings indeed fail
to behave "properly".


> To resolve the problem with "random" attributes being set on random
> files, I propose a utility is created to reset all attributes on all
> files in a directory tree.  Another utility could list all non-inherited
> (ie. not the same as the parent) attributes, then you can very quickly
> see which directories are compressed\encrypted.
>
> Thanks
> Oliver
>
> On Tue, 2008-12-09 at 22:14 +0000, Miguel Figueiredo Mascarenhas Sousa
> Filipe wrote:
>> On Tue, Dec 9, 2008 at 6:09 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
>> > On Tue, Dec 09, 2008 at 05:22:18PM +0100, Christian Hesse wrote:
>> >> On Tuesday 09 December 2008, Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
>> >> > Hi there,
>> >> >
>> >> > On Tue, Dec 9, 2008 at 2:59 PM, Lee Trager <lt73@cs.drexel.edu> wrote:
>> >> > > Currently compression and I assume if encryption is implemented it is
>> >> > > turned on or off during mount. There are however many times when a user
>> >> > > may want to select which files/directories they want to compress or
>> >> > > encrypt. This will also be helpful when implementing btrfs support in
>> >> > > grub for example. We can say the disk can be compressed/encrypted except
>> >> > > for /boot so compression/encryption doesn't have to be implemented in
>> >> > > grub.
>> >>
>> >> You could just use an additional partition for /boot that has compression an
>> >> encryption disabled...
>> >>
>> > Yes you could but many distributions are moving away from that such as
>> > ubuntu. Also while this works well on desktop and server environments
>> > where space is not an issue on embedded systems where free space is
>> > measured in kb or mb it starts becomming a huge issue.
>> >
>> >> > > I was thinking of adding this functionality to the userspace application
>> >> > > btrfstune. The way I was thinking of doing this is when btrfstune +c is
>> >> > > applied to a directory or file the directory(and all its contents) or
>> >> > > file will always be compressed reguardless of how the filesystem is
>> >> > > mounted. The opposite would happen when btrfstune -c is used.
>> >> > >
>> >> > > Would this be a reasonable thing to implement? Any suggestions before I
>> >> > > start doing this?
>> >> >
>> >> > Things like compression or encription should be used at the "volume" level.
>> >>
>> >> That was what I said some time ago when I asked why encryption support for
>> >> btrfs is planned. On a volume level you can use dm-crypt and the fs can
>> >> ignore that part completely.
>> >>
>> >> The answer was that different users on a home partition could use their own
>> >> encryption key. That sounds like volume level is out of bet. ;)
>> >>
>> > It does seem that doing it with volumes would limit user control and add
>> > lots of complexity to such a simple task.
>> >
>> > Miguel, what is your reason for suggesting this be done at the volume level?
>>
>> Well, basically, the main issue I have with file level configuration,
>> is maintenance nightmare in the long term.
>>
>> Redundancy level settings (raid0,1,5,6..), and compression/encription,
>> to me, seem like a sub-volume setting.
>> Its the right granularity to manage and mantain data and interact with it.
>> If I want this folders compressed.. I can just move them to my
>> compressed sub-vol.
>> If I want files: a, b/weird_path/D, and NSA/* confidential, I put them
>> in my encripted sub-vol.
>>
>> Imagine working on a confidencial project, where you add new
>> files/folders practilly everyday, and every single day, you must
>> remember to (and bother to do extra word so that..) set that file as a
>> encripted one.
>>
>> Also, by using sub-vols like this, and if redudancy is applied at the
>> same level, things just play nice, I can have distinct mount-points
>> that represent data that I want with a given set of
>> redundancy/compression and encription rules.
>>
>> It becomes easier to mantain, and use.
>>
>> Just imagine having a "confidencial" or a "compressed" mountpoint/disk
>>  ICON in you desktop, and just knowing that files in there are
>> encripted and/or compressed.
>> Versus, having to browser your file and look for a specific
>> "encripted" or "compressed" flag, that the file explorer shows if they
>> play nice...
>>
>> Now... if its at the file level, It can be troublesome to look for
>> allfiles encripted/compressed.
>> Also, knowing the encription key to each file (if it's set at the file
>> level) ... argh!!
>>
>>
>> Doing this file based, seems to me that will lead to complication the
>> long term maintenance of things.
>>
>> In the long term, when someone picks up, or mounts a old, used system,
>> I can't imagine the butload of work involved in finding out things
>> like:
>> - what in this 4TB fs is compressed, what is encripted?
>> - (in case of redundancy at the file level): how many redundancy
>> levels do I have, and which files respectively?
>> - why is rsync/scp 'ing this folder/BIG_NESTED_TREE_OF_FILES allways
>> "stop" at folder/**/fileB and folder/z*/* ?
>>  (might it be that some unsuspecting file in there has a trully beasty
>> compression ratio, and that 4GB fileB (in the middle of other 4GB
>> files), is instead a 80Gb file compressed ?
>>
>>
>> I also don't like extended attributes, we're just creating a whole new
>> orthogonal dimension/namespace were we store/manipulate information..
>> it also complicates backups (does the backup system understand and
>> respects ext. attrs?)
>>
>> It's messier doing these things with extended attributes or very
>> custom tools, instead of levelaging/creating a volume management tool,
>> for managing this, such tool will allready be there to manage the
>> fisical storage <-> filesystems/volumes mappings and settings.. might
>> has well leverage it.
>>
>>
>>
>> Kind regards,
>>
>>
>>
>> >
>> >> > So.. if a user wants a specific set of files or dirs ..they should
>> >> > create a mount-point/volume like:
>> >> >
>> >> > private_vol
>> >> > bigarchives_vol
>> >> >
>> >> > and set those volumes as compressed or encripted volumes
>> >> >
>> >> > Regarding usability, the best would be for the sub-volume creation
>> >> > tool to optionally allow passing encription/compression arguments.
>> >> >
>> >> >
>> >> > and then:
>> >> >  should mount those volumes somewhere like: ~/Confidential or ~/Archives.
>> >> >
>> >> > Basically, do it at the directory level (which in btrfs is at the
>> >> > sub-volume level).
>> >> > File-level granularity is totally unmanageable in the long term.
>> >> >
>> >> > Kind regards,
>> >>
>> >> --
>> >> Regards,
>> >> Chris
>> >
>> > Lee
>> >
>>
>>
>>
>
>



-- 
Miguel Sousa Filipe

Re: Selective Compression/Encryption

[Miguel Figueiredo Mascarenhas Sousa Filipe]

On Wed, Dec 10, 2008 at 9:32 AM, Jeremy Sanders
<jeremy@jeremysanders.net> wrote:
> Miguel Figueiredo Mascarenhas Sousa Filipe wrote:
>
>> Things like compression or encription should be used at the "volume"
>> level. So.. if a user wants a specific set of files or dirs ..they should
>> create a mount-point/volume like:
>>
>> private_vol
>> bigarchives_vol
>
> Does a normal user have permissions to do this?
>
> I could easily see the case where a normal user has a set of very
> compressible files in a directory. They shouldn't have to get a sysadmin go
> to the trouble of setting this up.
>

that's a policy decision.
Distro's will probably allow a admin/wheel user to do such things.

Big deployments, shell servers, or university setups, will all have a
policy decision and usualy hack up costumised tools for managing users
quota, settings, passwds and things like that.

A costumised useradd would probably create such volumes for each user,
for instance, and a costumised passwd, would reencript stored data
with new passwd. (or something similar...)...
I know I did similar things when was a university sysadm, pam_useradd
and pam_mountd were heavily used and costumised.

Kind regards,


-- 
Miguel Sousa Filipe

Re: Selective Compression/Encryption

[Chris Mason]

On Tue, 2008-12-09 at 18:50 -0500, jim owens wrote:
> Diego Calleja wrote:
> > El Tue, 9 Dec 2008 13:09:51 -0500, "Lee Trager" <lt73@cs.drexel.edu=
> escribi=C3=B3:
> >=20
> >> It does seem that doing it with volumes would limit user control a=
nd add
> >> lots of complexity to such a simple task.
> >=20
> > IMHO, WRT compression it's the contrary. Compression on a per-file =
basis has
> > never been very succesful (just look at how many windows users use =
it)
> > because it implies taking a decision for every file on the system. =
OTOH,
> > volume-level is just a single option to be enabled.
> >=20
> > I'm of course not arguing that file-level compression shouldn't be =
possible,
> > im just saying that is way more difficult to administer and that mo=
st people
> > (including sysadmins) is most likely to use compression in a per-vo=
lume basis.
>=20
> While I have not gotten far enough to prove it is feasible...
>=20
> My idea on controlling features like compression is that
> the default mode is inherited from the parent in the
> directory tree.  Thus you can turn it on/off at whatever
> granularity you want.
>=20

I had planned to make the bits inheritable from the directory inode
flags.  There are two different discussions around xattrs for this.  On=
e
is using xattrs to store the flag, which I'd would rather avoid because
it is checked in some performance critical places.

The second is using xattr programs to set the flag, which I don't reall=
y
have an opinion on.  The idea of having the flags backed up by backup
programs or rsync is really nice, but do any of the backup programs
actually copy out all the xattrs?

-chris


--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" =
in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Selective Compression/Encryption

[Chris Mason]

On Tue, 2008-12-09 at 15:22 -0500, jim owens wrote:
> Joshua J. Berry wrote:

[ storing compression flags ]

> > 
> > It seems like xattrs would be preferable to some btrfs-specific tunable, as 
> > programs like rsync or backup tools would be able to preserve (and restore) 
> > these bits with no extra work required.
> 
> I had the same thought... that many btrfs per-file tunables would
> be better implemented in xattrs (I've read christoph's earlier
> objections and hope to get around those issues).
> 
> I have been working on changing the xattr code with the first
> step getting it functioning properly when selinux is enabled
> so we can see just how costly btrfs xattrs are in actual use.

Not really on topic, but how are things broken today with selinux?

-chris

btrfs with selinux

[jim owens]

Chris Mason wrote:
> On Tue, 2008-12-09 at 15:22 -0500, jim owens wrote:
>> I have been working on changing the xattr code with the first
>> step getting it functioning properly when selinux is enabled
>> so we can see just how costly btrfs xattrs are in actual use.
> 
> Not really on topic, but how are things broken today with selinux?

With selinux enabled you can not create any files on
a btrfs filesystem (as of dec9 git tree with fedora 9),
even as root!

There are 2 things needed to make it work:

1) the /etc/selinux load-into-kernel database must be
    patched to recognize btrfs has xattrs. One of our
    security people, Paul Moore, has submitted it to
    the upstream refpolicy.  But it won't be merged
    until I finish my testing.

After the database is patched, the dec9 git tree
will allow file create on btrfs... but the selinux
xattrs are not set.  Thus "cp -a" will copy the
files but all "selinux context" values are wrong.

2) I have btrfs patches to interface correctly with
    the LSM so we save the selinux context. I'll be
    sending them up as soon as I have finished testing.

jim

P.S. sane people just disable selinux on install :)

Re: Selective Compression/Encryption

[Christoph Hellwig]

On Wed, Dec 10, 2008 at 08:44:47AM -0500, Chris Mason wrote:
> I had planned to make the bits inheritable from the directory inode
> flags.  There are two different discussions around xattrs for this.  One
> is using xattrs to store the flag, which I'd would rather avoid because
> it is checked in some performance critical places.
> 
> The second is using xattr programs to set the flag, which I don't really
> have an opinion on.  The idea of having the flags backed up by backup
> programs or rsync is really nice, but do any of the backup programs
> actually copy out all the xattrs?

xfsdump does :)  But I think especially the compressed bit is much
better off in the bit for the set/get flags ioctls used by chattr /
lsattr.  These are implemented by all Linux filesystems, and even have
a compressed bit allocated already.  (and xfsdump of course backs them
up, too ;-))  *run*

Re: Selective Compression/Encryption

[Linda Knippers]

Christoph Hellwig wrote:
> On Wed, Dec 10, 2008 at 08:44:47AM -0500, Chris Mason wrote:
>> I had planned to make the bits inheritable from the directory inode
>> flags.  There are two different discussions around xattrs for this.  One
>> is using xattrs to store the flag, which I'd would rather avoid because
>> it is checked in some performance critical places.
>>
>> The second is using xattr programs to set the flag, which I don't really
>> have an opinion on.  The idea of having the flags backed up by backup
>> programs or rsync is really nice, but do any of the backup programs
>> actually copy out all the xattrs?
> 
> xfsdump does :)  But I think especially the compressed bit is much
> better off in the bit for the set/get flags ioctls used by chattr /
> lsattr.  These are implemented by all Linux filesystems, and even have
> a compressed bit allocated already.  (and xfsdump of course backs them
> up, too ;-))  *run*

The 'star' command has an -xattr option.

-- ljk

Re: Selective Compression/Encryption

[Joshua J. Berry]

On Wednesday 10 December 2008 05:44:47 Chris Mason wrote:
[snip]
> The second is using xattr programs to set the flag, which I don't really
> have an opinion on.  The idea of having the flags backed up by backup
> programs or rsync is really nice, but do any of the backup programs
> actually copy out all the xattrs?

rsync has the -X option.  rdiff-backup autodetects whether xattrs are 
supported.

Those are the programs I use; dunno about others.

-- Josh


-- 
Joshua J. Berry

"I haven't lost my mind -- it's backed up on tape somewhere."
    -- /usr/games/fortune

Re: Selective Compression/Encryption

[Lee Trager]

On Tue, Dec 09, 2008 at 03:22:29PM -0500, jim owens wrote:
> Joshua J. Berry wrote:
>> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
>>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
>>>> Currently compression and I assume if encryption is implemented it is
>>>> turned on or off during mount. There are however many times when a user
>>>> may want to select which files/directories they want to compress or
>>>> encrypt. This will also be helpful when implementing btrfs support in
>>>> grub for example. We can say the disk can be compressed/encrypted except
>>>> for /boot so compression/encryption doesn't have to be implemented in
>>>> grub.
>>>>
>>>> I was thinking of adding this functionality to the userspace application
>>>> btrfstune. The way I was thinking of doing this is when btrfstune +c is
>>>> applied to a directory or file the directory(and all its contents) or
>>>> file will always be compressed reguardless of how the filesystem is
>>>> mounted. The opposite would happen when btrfstune -c is used.
>>> This was my plan, but btrfstune probably isn't the best program to do it
>>> (the ext2 tune program is mostly aimed at the super block level things).
>>>
>>> I think it would be better to make a setattr style program to call the
>>> ioctls.  There is already a per file compression flag, and the code
>>> should already be checking it.
>>
>> Is there some reason this can't be done with the existing extended 
>> attribute facilities?
>>
>> It seems like xattrs would be preferable to some btrfs-specific 
>> tunable, as programs like rsync or backup tools would be able to 
>> preserve (and restore) these bits with no extra work required.
>
> I had the same thought... that many btrfs per-file tunables would
> be better implemented in xattrs (I've read christoph's earlier
> objections and hope to get around those issues).
I agree that implementing it in the xattr would be best but what
userspace tools should be used to control it? lsattr/chattr or our own?
lsattr/chattr does seem to do what we want for compression but it really
has nothing for encryption. Should we also support attribute(a), no
modification(i), etc support in xattrs?
>
> I have been working on changing the xattr code with the first
> step getting it functioning properly when selinux is enabled
> so we can see just how costly btrfs xattrs are in actual use.
Could you send me to xattr code you are working on or will you be
submitting patches soon?

Thanks,

Lee

>
> jim

Re: Selective Compression/Encryption

[Chris Mason]

On Thu, 2008-12-11 at 17:03 -0500, Lee Trager wrote:
> On Tue, Dec 09, 2008 at 03:22:29PM -0500, jim owens wrote:
> > Joshua J. Berry wrote:
> >> On Tuesday 09 December 2008 08:35:16 Chris Mason wrote:
> >>> On Tue, 2008-12-09 at 09:59 -0500, Lee Trager wrote:
> >>>> Currently compression and I assume if encryption is implemented it is
> >>>> turned on or off during mount. There are however many times when a user
> >>>> may want to select which files/directories they want to compress or
> >>>> encrypt. This will also be helpful when implementing btrfs support in
> >>>> grub for example. We can say the disk can be compressed/encrypted except
> >>>> for /boot so compression/encryption doesn't have to be implemented in
> >>>> grub.
> >>>>
> >>>> I was thinking of adding this functionality to the userspace application
> >>>> btrfstune. The way I was thinking of doing this is when btrfstune +c is
> >>>> applied to a directory or file the directory(and all its contents) or
> >>>> file will always be compressed reguardless of how the filesystem is
> >>>> mounted. The opposite would happen when btrfstune -c is used.
> >>> This was my plan, but btrfstune probably isn't the best program to do it
> >>> (the ext2 tune program is mostly aimed at the super block level things).
> >>>
> >>> I think it would be better to make a setattr style program to call the
> >>> ioctls.  There is already a per file compression flag, and the code
> >>> should already be checking it.
> >>
> >> Is there some reason this can't be done with the existing extended 
> >> attribute facilities?
> >>
> >> It seems like xattrs would be preferable to some btrfs-specific 
> >> tunable, as programs like rsync or backup tools would be able to 
> >> preserve (and restore) these bits with no extra work required.
> >
> > I had the same thought... that many btrfs per-file tunables would
> > be better implemented in xattrs (I've read christoph's earlier
> > objections and hope to get around those issues).
> I agree that implementing it in the xattr would be best but what
> userspace tools should be used to control it? lsattr/chattr or our own?
> lsattr/chattr does seem to do what we want for compression but it really
> has nothing for encryption. Should we also support attribute(a), no
> modification(i), etc support in xattrs?

Just to clarify, I'm fine with xattrs as a user interface to set the
per-inode flags.  The flag for this should be stored in the inode
though, and not as a btrfs xattr.  There is already a flag field in the
inode for this use (see NODATASUM and NODATACOW)

-chris