testing compression in reiserfs4.1

testing compression in reiserfs4.1

[mangoo]

I was long looking for a Linux filesystem that would support r/w transparent compression.

jffs2 is stable, but is dedicated to flash devices mainly, and hence the maximum size of this filesystem is 4 GB.

ext2 + compression patches does not work with 2.6 kernels, has no journaling (it's ext2, after all), and is not considered stable (which doesn't bother me that much for my purposes).

IBM's jfs supports compression, but not under Linux.


And there is (will be) reiserfs4.1, which will at last have compression support?

Is it to be released anytime soon?

If not, where can I get 4.1 sources and how can I patch my kernel with the current, most up-to-date reiserfs4.1-alpha-don't-use version?

Features I need is compression, hardlinks and read/write (of course) - is the current state OK to test it on a /mnt/test partition with non-critical data?


Tomek


------------------------------------------------------------------
Jan Pawel II 1920 - 2005
http://link.interia.pl/f1871

Re: testing compression in reiserfs4.1

[Edward Shishkin]

mangoo@interia.pl wrote:

>I was long looking for a Linux filesystem that would support r/w transparent compression.
>
>jffs2 is stable, but is dedicated to flash devices mainly, and hence the maximum size of this filesystem is 4 GB.
>
>ext2 + compression patches does not work with 2.6 kernels, has no journaling (it's ext2, after all), and is not considered stable (which doesn't bother me that much for my purposes).
>
>IBM's jfs supports compression, but not under Linux.
>
>
>And there is (will be) reiserfs4.1, which will at last have compression support?
>
>Is it to be released anytime soon?
>
>If not, where can I get 4.1 sources and how can I patch my kernel with the current, most up-to-date reiserfs4.1-alpha-don't-use version?
>
>Features I need is compression, hardlinks and read/write (of course) - is the current state OK to test it on a /mnt/test partition with non-critical data?
>
>
>  
>
Hello, it looks mostly stable, but currently there is no human interface 
to create such
files. Also it would be fine to make it a bit smart: there is a lot of 
incompressible files
so flush algorithm should know when compression should be switched off.
We are working on it now, wait for beta-release.

Thanks,
Edward.

>Tomek
>
>
>------------------------------------------------------------------
>Jan Pawel II 1920 - 2005
>http://link.interia.pl/f1871 
>
>
>
>  
>

one idea about reiser4 compress

[Jun OKAJIMA]

Hello.

Assume you install a distribution on a compressed partition with reiser4.
you usually do like this --

| CD | Installer with decompression | -> | Re-compress with Reiser4 | HDD |

You dont think decompression -> recompression cycle is wasteful?
If CD contains compressed files with reiser4 compression format,
I guess it is possible to do like this --

| CD with reiser4 compressed files | Installer | -> | HDD | 

I mean, this eliminates an overhead of re-compression and make whole 
isntall process simpler and faster. Maybe you could do install like this --
 # cp -a /mnt/cdrom /mnt/target_hdd
And this way would be much faster than usual installer today.

How about this idea?


                --- Okajima, Jun. Tokyo, Japan.




>mangoo@interia.pl wrote:
>
>>I was long looking for a Linux filesystem that would support r/w transparent 
compression.
>>
>>jffs2 is stable, but is dedicated to flash devices mainly, and hence the maximum size 
of this filesystem is 4 GB.
>>
>>ext2 + compression patches does not work with 2.6 kernels, has no journaling (it's 
ext2, after all), and is not considered stable (which doesn't bother me that much for 
my purposes).
>>
>>IBM's jfs supports compression, but not under Linux.
>>
>>
>>And there is (will be) reiserfs4.1, which will at last have compression support?
>>
>>Is it to be released anytime soon?
>>
>>If not, where can I get 4.1 sources and how can I patch my kernel with the current, 
most up-to-date reiserfs4.1-alpha-don't-use version?
>>
>>Features I need is compression, hardlinks and read/write (of course) - is the current 
state OK to test it on a /mnt/test partition with non-critical data?
>>
>>
>>  
>>
>Hello, it looks mostly stable, but currently there is no human interface 
>to create such
>files. Also it would be fine to make it a bit smart: there is a lot of 
>incompressible files
>so flush algorithm should know when compression should be switched off.
>We are working on it now, wait for beta-release.
>
>Thanks,
>Edward.
>
>>Tomek
>>
>>
>>------------------------------------------------------------------
>>Jan Pawel II 1920 - 2005
>>http://link.interia.pl/f1871 
>>
>>
>>
>>  
>>
>

Re: one idea about reiser4 compress

[Linuxhippy]

Hi Jun!

>Assume you install a distribution on a compressed partition with reiser4.
>you usually do like this --
>
>| CD | Installer with decompression | -> | Re-compress with Reiser4 | HDD |
>
>You dont think decompression -> recompression cycle is wasteful?
>If CD contains compressed files with reiser4 compression format,
>I guess it is possible to do like this --
>
>| CD with reiser4 compressed files | Installer | -> | HDD | 
>
>I mean, this eliminates an overhead of re-compression and make whole 
>isntall process simpler and faster. Maybe you could do install like this --
> # cp -a /mnt/cdrom /mnt/target_hdd
>And this way would be much faster than usual installer today.
>
>How about this idea?
>  
>

To be honest I do not think your idea can be realized in a useful way 
because of:
a.) Every Installer uses its own compression format, most time those 
packages have additional fines in it like install scripts and so on.
b.) Reiser4.1 compression will eb definitivly optimized for speed, 
installer-packages are usually heavily optimized for size.
c.) You would need new API's to archive this. Either through syscalls or 
via reiser's special files.

Conclusion: It would be a major hassle to implement such stuff for next 
to null advantages. There are so many (important) features waiting to be 
implemented ;-)

lg Clemens

Re: one idea about reiser4 compress

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linuxhippy wrote:
> Hi Jun!
> 
>> Assume you install a distribution on a compressed partition with reiser4.
>> you usually do like this --
>>
>> | CD | Installer with decompression | -> | Re-compress with Reiser4 |
>> HDD |
>>
>> You dont think decompression -> recompression cycle is wasteful?
>> If CD contains compressed files with reiser4 compression format,
>> I guess it is possible to do like this --
>>
>> | CD with reiser4 compressed files | Installer | -> | HDD |
>> I mean, this eliminates an overhead of re-compression and make whole
>> isntall process simpler and faster. Maybe you could do install like
>> this --
>> # cp -a /mnt/cdrom /mnt/target_hdd
>> And this way would be much faster than usual installer today.
>>
>> How about this idea?
>>  
>>
> 
> To be honest I do not think your idea can be realized in a useful way
> because of:
> a.) Every Installer uses its own compression format, most time those
> packages have additional fines in it like install scripts and so on.

So support zisofs, if we're talking about a CD.

> b.) Reiser4.1 compression will eb definitivly optimized for speed,
> installer-packages are usually heavily optimized for size.

Right.  Hopefully, we'd have different algorithms supported, and zisofs
compression is gzip, which might be useful.  If it's not, then this
becomes a transcoding problem (convert gzip-to-zlib with no
decompression/recompression) which is beyond the scope of Reiser.

> c.) You would need new API's to archive this. Either through syscalls or
> via reiser's special files.

Special files are supposed to be easier than that.  Assuming that we
have a decent standard to allow custom extensions, the API should be no
big deal.  Look at XML's namespaces for a good exaple of such a standard.

> Conclusion: It would be a major hassle to implement such stuff for next
> to null advantages. There are so many (important) features waiting to be
> implemented ;-)

I agree there, but I won't be the one implementing it, and chances are
you won't either.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQllhh3gHNmZLgCUhAQJjRQ/+JWxU8ozwZqxZu1kC2r5L58Ij8ivm2ZDu
2+SRIh3qF8wtk049gx+a5YUWP66wD3FGhzeGwxzhoJ3exGTsP/HRASYWM7TB0/t0
+e96RaC4kn36mjWPEU2M0KFblZG3UUNjHL9Tpvsta3crOyhz4vc9ODpFPKzfqVqE
kKz1SGtA//noXbpdldmD4U30AoXApMH8396T4f62g3xTrdNvTfX9a/HuqTzGRXnm
whObzSmNPIvhRvaU/fYLQHFvKh6zat5rRXbt1AEy3264SzU9X0JnfeIjjF5KTMN0
mzjUt26E4sTKKLgAsMaJun35wsmq2dXsj16LHx+CujI6dcXC2pHK0HP5GpcSxJ5g
CVmsiMSd3GzDi5YbRjHoeLNwHA8nZThljTL3PbRrUsROHGsQe2k68G2RN8555g0S
apc3X45NMxmJtkZuIWv4GCzq4lpbj+DCLiSioIfhLoY+uxRELu9yLCRcgebuOcmo
dUcBnZS4uXSU0lYp6K0C+XvqFgi9Q/Q0YrseqbHNzABFzA/NHf7I/yDqg9yXQvxK
hcQ7y0lpeXO6rL2YEWCTCU5625M9HoZVnN1glzBSVpoqvscnxsrA/+1oxfDvLnAb
DSFfD9cvSmOtg83K/Li4Sirt8Bn7mjTaMi3vC/V2tuKrQ24QtRTBvhtbTtpITgkr
WPqgsq1/fHo=
=KIBS
-----END PGP SIGNATURE-----

Re: one idea about reiser4 compress

[Linuxhippy]

Hi again ;-)

>>Conclusion: It would be a major hassle to implement such stuff for next
>>to null advantages. There are so many (important) features waiting to be
>>implemented ;-)
>>    
>>
>
>I agree there, but I won't be the one implementing it, and chances are
>you won't either.
>  
>
Wow - a friend - nice ;-)
Well, if we both wont implement anything - we could let Jun implement 
the install-to-compressed-fs feature, right?

lg Clemens

Re: one idea about reiser4 compress

[Edward Shishkin]

Linuxhippy wrote:

> Hi Jun!
>
>> Assume you install a distribution on a compressed partition with 
>> reiser4.
>> you usually do like this --
>>
>> | CD | Installer with decompression | -> | Re-compress with Reiser4 | 
>> HDD |
>>
>> You dont think decompression -> recompression cycle is wasteful?
>> If CD contains compressed files with reiser4 compression format,
>> I guess it is possible to do like this --
>>
>> | CD with reiser4 compressed files | Installer | -> | HDD |
>> I mean, this eliminates an overhead of re-compression and make whole 
>> isntall process simpler and faster. Maybe you could do install like 
>> this --
>> # cp -a /mnt/cdrom /mnt/target_hdd
>> And this way would be much faster than usual installer today.
>>
>> How about this idea?
>>  
>

Fine, but it makes sense only if source and target files are powered by 
the same object plugin.
Also there is no any chances for 'cp' to implement this (who will inform 
the file system if we
don't need to decompress?)

> To be honest I do not think your idea can be realized in a useful way 
> because of:
> a.) Every Installer uses its own compression format, most time those 
> packages have additional fines in it like install scripts and so on.
> b.) Reiser4.1 compression will eb definitivly optimized for speed, 
> installer-packages are usually heavily optimized for size.
> c.) You would need new API's to archive this. Either through syscalls 
> or via reiser's special files.
>
> Conclusion: It would be a major hassle to implement such stuff for 
> next to null advantages. There are so many (important) features 
> waiting to be implemented ;-) 


Fast backup of cryptcompress files is a queued feature.
It should be a business of file reader/writer (not flush algorithm). 
Also each cluster after compression is supposed
to be ciphered (not in 4.1), this means we need to copy iv-seed of 
source file and store it in disk stat-data of target file
(I think to use object id as default iv-seed). This is a victim of fast 
backup...

Edward.

Re: one idea about reiser4 compress

[Jun OKAJIMA]

>
>Fine, but it makes sense only if source and target files are powered by 
>the same object plugin.
>Also there is no any chances for 'cp' to implement this (who will inform 
>the file system if we
>don't need to decompress?)
>

I am afraid if I got something wrong.
When you are going to do # cp -a /mnt/cdrom /mnt/install_hdd stuff,
the compression plug-in must not be enabled.
the pseudo transcript is like this.
Note: In this psuedo sample, enabling compression needs remounting,
but I hope this could be done with IOCTL to a dir or such and not need remounting.

                    --- Okajima.


-----------------
# mount -t reiser4 -o no-compression /dev/cdrom /mnt
# ls -la /mnt/cdrom/bin
-rwxr-xr-x    1 root     root         976 Dec 24  2002 arch
-rwxr-xr-x    1 root     root        3498 Mar  7  2002 cat
-rwxr-xr-x    1 root     root        8901 Mar 19  2002 chgrp
# /mnt/cdrom/cat
 EFL loader: illegal format.
# cp -a /mnt/cdrom /mnt/install_hdd
( wait a while... )
# mount -o remount,with-compression none /mnt/install_hdd
# ls -la /mnt/install_hdd
-rwxr-xr-x    1 root     root         2684 Dec 24  2002 arch
-rwxr-xr-x    1 root     root         9308 Mar  7  2002 cat
-rwxr-xr-x    1 root     root        15976 Mar 19  2002 chgrp