ext4 roadmap

ext4 roadmap

[Dave Kleikamp]

It has been discussed in the conference call that we need to better
define the requirements for ext4 and set some kind of target for when
new function should be added, so we can then work on stabilizing the
code and getting it production-ready.  The list below is based on the
candidate features list in Documentation/filesystems/ext4.txt.  It is by
no means complete, as some other ideas are being floated around, and I
very well may have missed some features.

I plan to put this in a wiki, but I'm not sure what the best place is.
Neither http://fedoraproject.org/wiki/ext3-devel nor
https://mail.clusterfs.com/wikis/lustre/Ext3Development has been kept up
to date, but I'd be happy to use either.

Here's something to start with:

Multiblock Allocation
---------------------
Primary developer: Alex Tomas
Status:	Patches available
Changes on-disk format: No
Target Date: ?
e2fsprogs changes: No

Delayed Allocation
------------------
Primary developer: Alex Tomas
Status:	Patches available
Changes on-disk format: No
Target Date: ?
e2fsprogs changes: No

Break 32000 subdirectory limit
------------------------------
Primary developer: Andreas? Ted?
Status:	Patches available
Changes on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

nsec timestamps
---------------
Primary developer: Andreas? Alex?
Status: Patches available
Change on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

inode version on disk
---------------------
Primary developer: ?
Status: Prototype available
Change on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

uninitialized groups
--------------------
Primary developer: ?
Status: Prototype available
Change on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

journal checksumming
--------------------
Primary developer: Andreas?
Status: Prototype available
Change on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

persistent preallocation
------------------------
Primary developer: Suparna?
Status: In development
Change on-disk format: Yes
Target Date: ?
e2fsprogs changes: Needed

-- 
David Kleikamp
IBM Linux Technology Center

Re: ext4 roadmap

[Grzegorz Kulewski]

Hi,

On Wed, 1 Nov 2006, Dave Kleikamp wrote:
> It has been discussed in the conference call that we need to better
> define the requirements for ext4 and set some kind of target for when
> new function should be added, so we can then work on stabilizing the
> code and getting it production-ready.  The list below is based on the
> candidate features list in Documentation/filesystems/ext4.txt.  It is by
> no means complete, as some other ideas are being floated around, and I
> very well may have missed some features.

Is there any chance to get compression and/or encryption in ext4?

Compression could be very usefull for directories (or whole trees) with 
big amount text or nearly text documents (like man pages, ebuilds in 
Gentoo, many other things).

But compression (on small files) would probably only work good if they 
will be aggregated somehow - like tail in reiserfs3 or something. Maybe 
there could be some attribute per directory meaning "there are 
only/mainly small files in this directory, try to store them together" and 
another "try to compress contents of this directory"?

As to the encryption - everybody knows why it can be usefull (of course 
not to store your warez and porn in a safe place... :-) - to protect 
important company secrets!). Since encryption algorithms are already 
available in kernel and encryption does not change lenght of data it 
should be easily doable. No?


Also, while working on openvz vservers I found that having per-dir quota 
would be usefull. Basically I mean I want to be able to say that if ext4 
is mounted under /foo then whole content of /foo/001 could only take 20GB 
(and maybe even that additionally /foo/001/tmp can take only 2GB - then 
both limits work for /foo/001/tmp of course). And of course I want to be 
able to know how much of this 20GB or 2GB is used. The same goes for 
inodes.

People in openvz done some artificial filesystem (simfs) to do that but it 
does not look too good, is not journaled (== it looses it's calculations 
during crash or any other strange event, including probably accessing that 
part of the native filesystem directly under it and recalculation it's 
quota can take really long time) and so on. Making it part of the native 
filesystem could work better. And probably this can be usefull not only 
for vserver solutions.

So, is there any chance to see something like that in ext4?



Thanks,

Grzegorz Kulewski

Re: ext4 roadmap

[Andreas Dilger]

On Nov 02, 2006  00:52 +0100, Grzegorz Kulewski wrote:
> Is there any chance to get compression and/or encryption in ext4?

Note that all of the features mentioned in the list at least have some
kind of prototype already or people working on it.  It isn't really a
"feature request list".  I'm not sure how many people are interested
in compression/encryption, but if they are they should come forward
with a patch.

Note that there was previously a compression patch for ext2, maybe that
could be moved forward for ext4?

Cheers, Andreas
--
Andreas Dilger
Principal Software Engineer
Cluster File Systems, Inc.

Re: ext4 roadmap

[Grzegorz Kulewski]

On Thu, 2 Nov 2006, Andreas Dilger wrote:
> On Nov 02, 2006  00:52 +0100, Grzegorz Kulewski wrote:
>> Is there any chance to get compression and/or encryption in ext4?
>
> Note that all of the features mentioned in the list at least have some
> kind of prototype already or people working on it.  It isn't really a
> "feature request list".  I'm not sure how many people are interested
> in compression/encryption, but if they are they should come forward
> with a patch.

I understand that. Unfortunatelly I don't have enough time to write 
something like that and my knowledge about ext3/4 in the kernel is pretty 
limited too.

***This is only an idea if somebody with more code knowledge has time and 
thinks that this is worth implementing. This is simply something that I, 
a user (maybe a little more advanced one, but still user) of ext3, thinks 
would be usefull.***

Also (as to the compression) I think Linux's VFS and page cache and so on 
operates rather on data amounts of constant length and isn't too well 
suited for making them smaller (or larger) somewhere in the middle 
between userspace and the disk. But maybe I am completly wrong about it. 
But if I am not wrong extending core Linux components to allow easy 
generic compression (or other manipulation that changes size) should be 
probably done in cooperation with something that works pretty close with 
core kernel components (like ext3/4 in my opinion does). This could 
possibly allow to avoid having 10 different implementation for each 
filesystem in the future. Compression and especially encryption should not 
be (in my opinion) too much fs dependent (fs should not care too much 
about data only about metadata) and should have only minimal per-fs 
support (additional metadata to allow discovering that file was 
compressed and to allow seeks for example). But again, maybe I am 
completly wrong.

As to the encryption and per-dir quota they should be probably easier (I 
hope). Maybe even I can try to hack something like that if somebody could 
tell me how to do it in a way to not break too much in the process). But 
they would probably require some ext3 on-disk filesystem layout changes 
(and e2progs changes too) so maybe ext4-development time is good to 
introduce them?


> Note that there was previously a compression patch for ext2, maybe that
> could be moved forward for ext4?

I know about ext2 patch (I even think there was more than one) and I even 
used it in those old days when linux distro could be made to fit on 250MB 
disk without too much problems. I think I ever know (one of?) the author 
of it personally. Unfortunatelly IIRC this was rather some not very pretty 
and efficient hack. It should be probably reimplemented from scratch 
rather than ported. I think too much (journaling, ...) changed since then 
in ext3/4. But again, maybe I am completly wrong and those old bits of 
code could be made to work on newer kernels somehow.


Thanks,

Grzegorz Kulewski