FileSystem level compression support.

FileSystem level compression support.

[Miguel Sousa Filipe]

Hi there,

Providing compression and/or encryption at the file system layer has
proven to be contentious and polemic, at least in the linux kernel
world.

However, I would like to probe you guys (specially Chis Mason) about
your thoughts on the issue.
I have seen quite a few benchs were compression provides better
performance, since it reduces the amount of IO.
It seems to be a cpu/io trade-off. And given that cpu power is more
easily accessible than fast IO.. it is starting to look a good idea
that should be more deeply explored.

Is compression for data on BTRFS a feature that is:
- completely out of the table - it will not happen at the BTRFS layer
- considered, but no efforts on developing this feature are planed.
- considered, and design decisions are contemplating it.. but no code
yet.. not this soon. (not for 1.0)


Kind regards!

-- 
Miguel Sousa Filipe

Re: FileSystem level compression support.

[Ric Wheeler]

Miguel Sousa Filipe wrote:
> Hi there,
>
> Providing compression and/or encryption at the file system layer has
> proven to be contentious and polemic, at least in the linux kernel
> world.
>
> However, I would like to probe you guys (specially Chis Mason) about
> your thoughts on the issue.
> I have seen quite a few benchs were compression provides better
> performance, since it reduces the amount of IO.
> It seems to be a cpu/io trade-off. And given that cpu power is more
> easily accessible than fast IO.. it is starting to look a good idea
> that should be more deeply explored.
>
> Is compression for data on BTRFS a feature that is:
> - completely out of the table - it will not happen at the BTRFS layer
> - considered, but no efforts on developing this feature are planed.
> - considered, and design decisions are contemplating it.. but no code
> yet.. not this soon. (not for 1.0)
>
>
> Kind regards!
>
>   
One big issue with compression is that it is very workload dependent. 
For example, if you are storing lots of video, sound or image files, the 
format itself it normally already compressed so you end up doing nothing 
but burning CPU cycles ;-) Also, for very small files, compression works 
best when you bunch them up (say compress the contents of a subdirectory).

Also note that some storage devices can do compression for you beneath 
the file system or do other fancy features like block level data 
de-duplication.

ric

Re: FileSystem level compression support.

[Chris Mason]

On Wed, 2008-06-11 at 12:13 +0100, Miguel Sousa Filipe wrote:
> Hi there,
> 
> Providing compression and/or encryption at the file system layer has
> proven to be contentious and polemic, at least in the linux kernel
> world.
> 
> However, I would like to probe you guys (specially Chis Mason) about
> your thoughts on the issue.
> I have seen quite a few benchs were compression provides better
> performance, since it reduces the amount of IO.
> It seems to be a cpu/io trade-off. And given that cpu power is more
> easily accessible than fast IO.. it is starting to look a good idea
> that should be more deeply explored.
> 
> Is compression for data on BTRFS a feature that is:
> - completely out of the table - it will not happen at the BTRFS layer
> - considered, but no efforts on developing this feature are planed.
> - considered, and design decisions are contemplating it.. but no code
> yet.. not this soon. (not for 1.0)

Compression is definitely on the table, although it probably won't be in
in until after 1.0 unless someone volunteers.

There are lots of different ways to code compression, and some are much
harder than others.  If you restrict the compression support to only
files small enough to have their data packed into the btree, things get
much much easier.

-chris