New tool to recursive compress / decompress of files

New tool to recursive compress / decompress of files

[Petr Bena]

Hi, I like the compress feature, but unlike NTFS, compressing files in
btrfs is not so simple. Just changing the flags using chattr will not
compress anything, only newly written data would be.

For this reason I decided to create a new tool that makes this simple,
you can find it here: https://github.com/benapetr/compress

It's very simple to use, for example to compress whole folder, you would do

compress -rv <folder_name>

the tool would recursively scan the folder, change all folders to +c
and copy all files, preserving attributes to a temporary file within
folder, then removes the original and move the copied, now fully
compressed file back.

It's written in python despite I loathe it (I am a C++ programmer),
because most of people love it for some reason, any because this would
make it more simple for people to deploy it and modify it.

The tool right now is in early alpha stage, probably full of bugs, I
made it few hours ago, but I would like to hear any feedback, whether
there would be any use for such a tool, feature requests etc :)

Thanks

Re: New tool to recursive compress / decompress of files

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1524 bytes --]

On Sat, Mar 21, 2015 at 06:56:12PM +0100, Petr Bena wrote:
> Hi, I like the compress feature, but unlike NTFS, compressing files in
> btrfs is not so simple. Just changing the flags using chattr will not
> compress anything, only newly written data would be.
> 
> For this reason I decided to create a new tool that makes this simple,
> you can find it here: https://github.com/benapetr/compress
> 
> It's very simple to use, for example to compress whole folder, you would do
> 
> compress -rv <folder_name>
> 
> the tool would recursively scan the folder, change all folders to +c
> and copy all files, preserving attributes to a temporary file within
> folder, then removes the original and move the copied, now fully
> compressed file back.

   What does it do that this doesn't?

$ chattr -R +c $dir
$ btrfs fi defrag -r -c $dir

   (Sorry to do this to you, but better early in the life of the
project than later).

   Hugo.

> It's written in python despite I loathe it (I am a C++ programmer),
> because most of people love it for some reason, any because this would
> make it more simple for people to deploy it and modify it.
> 
> The tool right now is in early alpha stage, probably full of bugs, I
> made it few hours ago, but I would like to hear any feedback, whether
> there would be any use for such a tool, feature requests etc :)

-- 
Hugo Mills             | You shouldn't anthropomorphise computers. They
hugo@... carfax.org.uk | really don't like that.
http://carfax.org.uk/  |
PGP: 65E74AC0          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: New tool to recursive compress / decompress of files

[Petr Bena]

I would be certainly useful if this kind of tricks were mentioned on
https://btrfs.wiki.kernel.org/index.php/Compression or anywhere on
wiki

On Sat, Mar 21, 2015 at 7:08 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
> On Sat, Mar 21, 2015 at 06:56:12PM +0100, Petr Bena wrote:
>> Hi, I like the compress feature, but unlike NTFS, compressing files in
>> btrfs is not so simple. Just changing the flags using chattr will not
>> compress anything, only newly written data would be.
>>
>> For this reason I decided to create a new tool that makes this simple,
>> you can find it here: https://github.com/benapetr/compress
>>
>> It's very simple to use, for example to compress whole folder, you would do
>>
>> compress -rv <folder_name>
>>
>> the tool would recursively scan the folder, change all folders to +c
>> and copy all files, preserving attributes to a temporary file within
>> folder, then removes the original and move the copied, now fully
>> compressed file back.
>
>    What does it do that this doesn't?
>
> $ chattr -R +c $dir
> $ btrfs fi defrag -r -c $dir
>
>    (Sorry to do this to you, but better early in the life of the
> project than later).
>
>    Hugo.
>
>> It's written in python despite I loathe it (I am a C++ programmer),
>> because most of people love it for some reason, any because this would
>> make it more simple for people to deploy it and modify it.
>>
>> The tool right now is in early alpha stage, probably full of bugs, I
>> made it few hours ago, but I would like to hear any feedback, whether
>> there would be any use for such a tool, feature requests etc :)
>
> --
> Hugo Mills             | You shouldn't anthropomorphise computers. They
> hugo@... carfax.org.uk | really don't like that.
> http://carfax.org.uk/  |
> PGP: 65E74AC0          |

Re: New tool to recursive compress / decompress of files

[Petr Bena]

I am just wondering, does that defragment trick work for non-root users?

Also, even if it did work, I could as well make my tools use this
defragment trick instead, because I will probably never remember that
hacky syntax of defrag command :P doing something like compress -r
<folder> or decompress -r <folder> is just much more easy to remember
than multiple commands.

That thing I made is just a wrapper for other commands anyway

On Sat, Mar 21, 2015 at 7:12 PM, Petr Bena <benapetr@gmail.com> wrote:
> I would be certainly useful if this kind of tricks were mentioned on
> https://btrfs.wiki.kernel.org/index.php/Compression or anywhere on
> wiki
>
> On Sat, Mar 21, 2015 at 7:08 PM, Hugo Mills <hugo@carfax.org.uk> wrote:
>> On Sat, Mar 21, 2015 at 06:56:12PM +0100, Petr Bena wrote:
>>> Hi, I like the compress feature, but unlike NTFS, compressing files in
>>> btrfs is not so simple. Just changing the flags using chattr will not
>>> compress anything, only newly written data would be.
>>>
>>> For this reason I decided to create a new tool that makes this simple,
>>> you can find it here: https://github.com/benapetr/compress
>>>
>>> It's very simple to use, for example to compress whole folder, you would do
>>>
>>> compress -rv <folder_name>
>>>
>>> the tool would recursively scan the folder, change all folders to +c
>>> and copy all files, preserving attributes to a temporary file within
>>> folder, then removes the original and move the copied, now fully
>>> compressed file back.
>>
>>    What does it do that this doesn't?
>>
>> $ chattr -R +c $dir
>> $ btrfs fi defrag -r -c $dir
>>
>>    (Sorry to do this to you, but better early in the life of the
>> project than later).
>>
>>    Hugo.
>>
>>> It's written in python despite I loathe it (I am a C++ programmer),
>>> because most of people love it for some reason, any because this would
>>> make it more simple for people to deploy it and modify it.
>>>
>>> The tool right now is in early alpha stage, probably full of bugs, I
>>> made it few hours ago, but I would like to hear any feedback, whether
>>> there would be any use for such a tool, feature requests etc :)
>>
>> --
>> Hugo Mills             | You shouldn't anthropomorphise computers. They
>> hugo@... carfax.org.uk | really don't like that.
>> http://carfax.org.uk/  |
>> PGP: 65E74AC0          |

Re: New tool to recursive compress / decompress of files

[Roman Mamedov]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

On Sat, 21 Mar 2015 18:56:12 +0100
Petr Bena <benapetr@gmail.com> wrote:

> unlike NTFS, compressing files in btrfs is not so simple

There shouldn't be any need to micro-manage compression on Btrfs on a
per-folder or per-file basis. Just mount the whole volume as compress=[method]
(but not compress-force), there shouldn't be any downside, on the contrary,
with the current ratio of CPU core count and their performance to disk I/O
speed, you are likely to even see a speed-up. Also files which are detected to
be incompressible are automatically skipped from compression (at least that's
what it tries to do by design).

If you want higher performance and less fragmentation on certain files/folders,
you are supposed to set them NOCOW, at which point the compression is also
automatically disabled.

-- 
With respect,
Roman

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: New tool to recursive compress / decompress of files

[Petr Bena]

Ok, but what if I am just a non-root user who want to transparently
compress some of their data? What if I am a root user who does just
want to compress some large folder transparently and doesn't want to
mess up with subvolumes?

I know that from sysadmin point of view, there is no need for this,
but from regular user point of view, who have btrfs on their laptop,
this could make life easier.

On Sat, Mar 21, 2015 at 7:22 PM, Roman Mamedov <rm@romanrm.net> wrote:
> On Sat, 21 Mar 2015 18:56:12 +0100
> Petr Bena <benapetr@gmail.com> wrote:
>
>> unlike NTFS, compressing files in btrfs is not so simple
>
> There shouldn't be any need to micro-manage compression on Btrfs on a
> per-folder or per-file basis. Just mount the whole volume as compress=[method]
> (but not compress-force), there shouldn't be any downside, on the contrary,
> with the current ratio of CPU core count and their performance to disk I/O
> speed, you are likely to even see a speed-up. Also files which are detected to
> be incompressible are automatically skipped from compression (at least that's
> what it tries to do by design).
>
> If you want higher performance and less fragmentation on certain files/folders,
> you are supposed to set them NOCOW, at which point the compression is also
> automatically disabled.
>
> --
> With respect,
> Roman

Re: New tool to recursive compress / decompress of files

[Petr Bena]

There is a dozen of cases where you want to just compress some folders
in your system, but you don't want to compress whole device, would you
create a subvolume for each of them?

On Sat, Mar 21, 2015 at 7:25 PM, Petr Bena <benapetr@gmail.com> wrote:
> Ok, but what if I am just a non-root user who want to transparently
> compress some of their data? What if I am a root user who does just
> want to compress some large folder transparently and doesn't want to
> mess up with subvolumes?
>
> I know that from sysadmin point of view, there is no need for this,
> but from regular user point of view, who have btrfs on their laptop,
> this could make life easier.
>
> On Sat, Mar 21, 2015 at 7:22 PM, Roman Mamedov <rm@romanrm.net> wrote:
>> On Sat, 21 Mar 2015 18:56:12 +0100
>> Petr Bena <benapetr@gmail.com> wrote:
>>
>>> unlike NTFS, compressing files in btrfs is not so simple
>>
>> There shouldn't be any need to micro-manage compression on Btrfs on a
>> per-folder or per-file basis. Just mount the whole volume as compress=[method]
>> (but not compress-force), there shouldn't be any downside, on the contrary,
>> with the current ratio of CPU core count and their performance to disk I/O
>> speed, you are likely to even see a speed-up. Also files which are detected to
>> be incompressible are automatically skipped from compression (at least that's
>> what it tries to do by design).
>>
>> If you want higher performance and less fragmentation on certain files/folders,
>> you are supposed to set them NOCOW, at which point the compression is also
>> automatically disabled.
>>
>> --
>> With respect,
>> Roman

Re: New tool to recursive compress / decompress of files

[Marc MERLIN]

On Sat, Mar 21, 2015 at 07:26:41PM +0100, Petr Bena wrote:
> There is a dozen of cases where you want to just compress some folders
> in your system, but you don't want to compress whole device, would you
> create a subvolume for each of them?
 
Actually with the previous reply you got, I'm not sure there is any case at
all.

Compression makes your filesystem faster and is transparent to you.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/  

Re: New tool to recursive compress / decompress of files

[David Sterba]

On Sat, Mar 21, 2015 at 07:12:52PM +0100, Petr Bena wrote:
> I would be certainly useful if this kind of tricks were mentioned on
> https://btrfs.wiki.kernel.org/index.php/Compression or anywhere on
> wiki

There's a gap between developers (who don't want to write documentation
because they know how to do things) and users (who expect there's enough
documentaion how to do things).

We're getting better and the user queries and complaints help to close
this gap but docs are still not perfect and somebody has to find the
time to write it.

The wiki is open, I'm merging documentation updates to btrfs-progs (ie.
man pages) right away, but this still does not seem to encourage enough
people. No coding skills rquired. Would github-style of pull requests
help to improve that? Or plain mails with the portions of text that'd
go to manpages (ie. no bothering with formatting)?

Re: New tool to recursive compress / decompress of files

[David Sterba]

On Sat, Mar 21, 2015 at 07:16:47PM +0100, Petr Bena wrote:
> I am just wondering, does that defragment trick work for non-root users?

Defragmentation works where read and write would work for the user. Root
can also defrag files that do not have the wite rwx bit set, but it's a
special case anyway.

> Also, even if it did work, I could as well make my tools use this
> defragment trick instead, because I will probably never remember that
> hacky syntax of defrag command :P doing something like compress -r
> <folder> or decompress -r <folder> is just much more easy to remember
> than multiple commands.

The tools should provide a rich set of options to do basic operations
and could provide options to do complex things, but writing wrappers is
sometimes inevitable.

In your case, we can enhance the defrag command to set the +c bit on
directories/files on the way, plus set the compression method via
properties according to the user selected method. All of this fits into
the idea of core tool support.