ext5

ext5

[Anonymous Remailer (austria)]

will there be ext5? ext4 works just fine so far. but it could be even more faster. otherwise i have to jump to btrfs (when it's done).

Re: ext5

[tytso]

On Wed, Feb 10, 2010 at 12:40:05AM +0100, Anonymous Remailer (austria) wrote:
> 
> will there be ext5? ext4 works just fine so far. but it could be
> even more faster. otherwise i have to jump to btrfs (when it's
> done).

We currently don't have any plans for an "ext5".  There might be some
new features that might gradually trickle into ext4; for example
there's someone who I may be mentoring who is interested in working on
an idea I've had to add read-only compression to ext4.  (Actually, the
design I've sketched out makes 90% of the work be file system
independent, so it's something that could be retrofitted into other
filesystems: xfs, btrfs, etc.)

The benchmarks I've seen don't show that btrfs is that much faster;
for some workloads its faster, for others its slower.  Of course,
there may be some file system tuning that still remains to be done,
both for btrfs and ext4, that may change the performance numbers
slightly.

The main reason why I suspect people will be interested in btrfs is to
because of some of its features (i.e., file system level snapshots,
data level checksums, etc.) that are unlikely to show up in ext4.  I
also suspect that btrfs will take a while to mature, as all file
systems do.  ZFS for example took good five years to development, and
perhaps another 3-4 before people started really trusting it for
critical production uses.

Regards,

						- Ted

Re: ext5

[Manish Katiyar]

On Thu, Feb 11, 2010 at 3:20 AM,  <tytso@mit.edu> wrote:
> On Wed, Feb 10, 2010 at 12:40:05AM +0100, Anonymous Remailer (austria) wrote:
>>
>> will there be ext5? ext4 works just fine so far. but it could be
>> even more faster. otherwise i have to jump to btrfs (when it's
>> done).
>
> We currently don't have any plans for an "ext5".  There might be some
> new features that might gradually trickle into ext4; for example
> there's someone who I may be mentoring who is interested in working on
> an idea I've had to add read-only compression to ext4.  (Actually, the
> design I've sketched out makes 90% of the work be file system
> independent,
Hi Ted,

Is this design somewhere on net so that we can read/lookup it up ?

Thanks -
Manish
> so it's something that could be retrofitted into other
> filesystems: xfs, btrfs, etc.)
>
> The benchmarks I've seen don't show that btrfs is that much faster;
> for some workloads its faster, for others its slower.  Of course,
> there may be some file system tuning that still remains to be done,
> both for btrfs and ext4, that may change the performance numbers
> slightly.
>
> The main reason why I suspect people will be interested in btrfs is to
> because of some of its features (i.e., file system level snapshots,
> data level checksums, etc.) that are unlikely to show up in ext4.  I
> also suspect that btrfs will take a while to mature, as all file
> systems do.  ZFS for example took good five years to development, and
> perhaps another 3-4 before people started really trusting it for
> critical production uses.
>
> Regards,
>
>                                                - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
Thanks -
Manish
==================================
[$\*.^ -- I miss being one of them
==================================
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: ext5

[tytso]

On Thu, Feb 11, 2010 at 09:08:36AM +0530, Manish Katiyar wrote:
> 
> Is this design somewhere on net so that we can read/lookup it up ?

Not yet, but the basic idea is to do the compression in userspace,
using libz with regular resync points every 64k or 128k of
uncompressed data.  An array, indexed by each fixed-block of
uncompressed data, is located at the beginning of the file indicating
where each compressed block begins.  The file is stored on-disk
written by the installer in a compressed format, and then the
installer flips an attribute bit which marks the file as containing
compressed data, and which also makes the file immutable.  (Any
attempt to open the file read/write will result in an error.)

The advantage of this scheme is that it minimizes in-kernel
complexity, since the kernel doesn't have to deal with compressing
data, and it means we can use a simpler format which is more efficient
from a compression standpoint since we don't have to deal with random
access writes.

Another advantage of this scheme is that it's relatively easy to do
the bulk of the work in a file system-independent layer.  Some minor
work would be needed to interface the attribute bit and the mmap
reading function into each client filesystem, but it should be
possible to isolate 95% of the work into a filesystem independent
layer, and then make this something which multiple file systems could
take advantage of --- not just ext3 and ext4, but also xfs, btrfs,
etc.

						- Ted

Re: ext5

[Jan Kara]

> On Thu, Feb 11, 2010 at 09:08:36AM +0530, Manish Katiyar wrote:
> > 
> > Is this design somewhere on net so that we can read/lookup it up ?
> 
> Not yet, but the basic idea is to do the compression in userspace,
> using libz with regular resync points every 64k or 128k of
> uncompressed data.  An array, indexed by each fixed-block of
> uncompressed data, is located at the beginning of the file indicating
> where each compressed block begins.  The file is stored on-disk
> written by the installer in a compressed format, and then the
> installer flips an attribute bit which marks the file as containing
> compressed data, and which also makes the file immutable.  (Any
> attempt to open the file read/write will result in an error.)
  From the first reading, this sounds like what zisofs is doing.
The reading part is already implemented in kernel in fs/isofs/compress.c
so that might be lifted to a generic level and used for ext4 as well..

								Honza
-- 
Jan Kara <jack@suse.cz>
SuSE CR Labs

Re: ext5

[tytso]

On Thu, Feb 11, 2010 at 06:55:58PM +0100, Jan Kara wrote:
>   From the first reading, this sounds like what zisofs is doing.
> The reading part is already implemented in kernel in fs/isofs/compress.c
> so that might be lifted to a generic level and used for ext4 as well..

I didn't know about zisofs; thanks for pointing that out!  I'll
mention it to the student I'm helping out....

						- Ted

Re: ext5

[Ron Johnson]

On 2010-02-10 15:50, tytso@mit.edu wrote:
> On Wed, Feb 10, 2010 at 12:40:05AM +0100, Anonymous Remailer (austria) wrote:
>> will there be ext5? ext4 works just fine so far. but it could be
>> even more faster. otherwise i have to jump to btrfs (when it's
>> done).
> 
> We currently don't have any plans for an "ext5".  There might be some
> new features that might gradually trickle into ext4; for example
> there's someone who I may be mentoring who is interested in working on
> an idea I've had to add read-only compression to ext4.  (Actually, the
> design I've sketched out makes 90% of the work be file system
> independent, so it's something that could be retrofitted into other
> filesystems: xfs, btrfs, etc.)

I guess that means every file on the fs?

Windows-like per-file compression would be darned useful in certain 
circumstances.  Big mbox files, for example.

-- 
"Hell hath no fury like the vast robot armies of a woman scorned."
Walt

Re: ext5

[tytso]

On Wed, Feb 10, 2010 at 11:18:21PM -0600, Ron Johnson wrote:
> >We currently don't have any plans for an "ext5".  There might be some
> >new features that might gradually trickle into ext4; for example
> >there's someone who I may be mentoring who is interested in working on
> >an idea I've had to add read-only compression to ext4.  (Actually, the
> >design I've sketched out makes 90% of the work be file system
> >independent, so it's something that could be retrofitted into other
> >filesystems: xfs, btrfs, etc.)
> 
> I guess that means every file on the fs?

No, I mean per-file compression, but a compressed file is immutable.
This is basically what Mac OS X has recently added, and while I
haven't looked at their implementation, Apple being one of those
closed source companies and all, I wouldn't be surprised if they did
things the same way.

> Windows-like per-file compression would be darned useful in certain
> circumstances.  Big mbox files, for example.

The problem with mbox files is that some mail readers try to smart
about how they modify them to avoid needing to rewrite the whole mbox
file; mutt will seak to the middle of the file, write to the end of
the file, and then trim off any excess space by using the truncate
system call.  This is *hard* to support if the mbox file is
compressed; you can do it using a stacker-style compression technique,
but it's not as efficient, and it has a lot of complexity in the
kernel.

The idea with read-only compressed files is that they are useful for
large executables or large static files, where compressing them means
that it takes less time to read them off of an HDD.

     	      	   	   	     	    - Ted

Re: ext5

[Ron Johnson]

On 2010-02-11 00:44, tytso@mit.edu wrote:
> On Wed, Feb 10, 2010 at 11:18:21PM -0600, Ron Johnson wrote:
>>> We currently don't have any plans for an "ext5".  There might be some
>>> new features that might gradually trickle into ext4; for example
>>> there's someone who I may be mentoring who is interested in working on
>>> an idea I've had to add read-only compression to ext4.  (Actually, the
>>> design I've sketched out makes 90% of the work be file system
>>> independent, so it's something that could be retrofitted into other
>>> filesystems: xfs, btrfs, etc.)
>> I guess that means every file on the fs?
> 
> No, I mean per-file compression, but a compressed file is immutable.
> This is basically what Mac OS X has recently added, and while I
> haven't looked at their implementation, Apple being one of those
> closed source companies and all, I wouldn't be surprised if they did
> things the same way.
> 
>> Windows-like per-file compression would be darned useful in certain
>> circumstances.  Big mbox files, for example.
> 
> The problem with mbox files is that some mail readers try to smart
> about how they modify them to avoid needing to rewrite the whole mbox
> file; mutt will seak to the middle of the file, write to the end of
> the file, and then trim off any excess space by using the truncate
> system call.  This is *hard* to support if the mbox file is
> compressed; you can do it using a stacker-style compression technique,
> but it's not as efficient, and it has a lot of complexity in the
> kernel.

I guess that's how Windows does it?

> The idea with read-only compressed files is that they are useful for
> large executables or large static files, where compressing them means
> that it takes less time to read them off of an HDD.

Sure.  Anything is better than nothing!

-- 
"Hell hath no fury like the vast robot armies of a woman scorned."
Walt

Re: ext5

[Goswin von Brederlow]

tytso@mit.edu writes:

> On Wed, Feb 10, 2010 at 11:18:21PM -0600, Ron Johnson wrote:
>> >We currently don't have any plans for an "ext5".  There might be some
>> >new features that might gradually trickle into ext4; for example
>> >there's someone who I may be mentoring who is interested in working on
>> >an idea I've had to add read-only compression to ext4.  (Actually, the
>> >design I've sketched out makes 90% of the work be file system
>> >independent, so it's something that could be retrofitted into other
>> >filesystems: xfs, btrfs, etc.)
>> 
>> I guess that means every file on the fs?
>
> No, I mean per-file compression, but a compressed file is immutable.
> This is basically what Mac OS X has recently added, and while I
> haven't looked at their implementation, Apple being one of those
> closed source companies and all, I wouldn't be surprised if they did
> things the same way.

Do that with fuse or like ecryptfs. That is nothing that needs to be
inside any FS and is far better to have as layer on top of the fs of
your choice.

>> Windows-like per-file compression would be darned useful in certain
>> circumstances.  Big mbox files, for example.
>
> The problem with mbox files is that some mail readers try to smart
> about how they modify them to avoid needing to rewrite the whole mbox
> file; mutt will seak to the middle of the file, write to the end of
> the file, and then trim off any excess space by using the truncate
> system call.  This is *hard* to support if the mbox file is
> compressed; you can do it using a stacker-style compression technique,
> but it's not as efficient, and it has a lot of complexity in the
> kernel.

You need a block based compression for that so that you can recompress
and replace blocks in any order. It also helps if you can have the
compressed blocks in random order so that you can put the block at the
end and zero fill the old location when it grows during recompression.

Otherwise you will be moving the compressed data around a lot to
accomodate changes in the middle of a file.

> The idea with read-only compressed files is that they are useful for
> large executables or large static files, where compressing them means
> that it takes less time to read them off of an HDD.

Or when you only have so much flash.

MfG
        Goswin

Re: ext5

[Ron Johnson]

On 2010-02-11 15:41, Goswin von Brederlow wrote:
> tytso@mit.edu writes:
>> The idea with read-only compressed files is that they are useful for
>> large executables or large static files, where compressing them means
>> that it takes less time to read them off of an HDD.
> 
> Or when you only have so much flash.

Isn't that what squashfs is for?

-- 
"Hell hath no fury like the vast robot armies of a woman scorned."
Walt

Re: ext5

[tytso]

On Thu, Feb 11, 2010 at 10:47:15PM -0600, Ron Johnson wrote:
> On 2010-02-11 15:41, Goswin von Brederlow wrote:
> >tytso@mit.edu writes:
> >>The idea with read-only compressed files is that they are useful for
> >>large executables or large static files, where compressing them means
> >>that it takes less time to read them off of an HDD.
> >
> >Or when you only have so much flash.
> 
> Isn't that what squashfs is for?

The problem with the squashfs, fuse, ecryptfs approaches for the use
case that I am envisioning is that it's an all-or-nothing sort of
thing.  You may not want to encrypt all of the files in a file system.
Sure, you can play games with bind mounts, and/or accept the
performance hit of passing everything through fuse even for files that
aren't encrypted, but I think that's going to significantly inhibit
adoption of the technology.

Something which allows compressed and uncompressed files to co-exist
without any performance hits to the uncompressed files, and which
allows for a gradual transition after you upgrade rpm/dpkg and as
packages get upgraded is going to much easier time with adoption
rates.

							- Ted