[PATCH 0/2] LogFS take two

[PATCH 0/2] LogFS take two

[Jörn Engel]

Motivation:

Linux currently has 1-2 flash filesystems to choose from, JFFS2 and
YAFFS.  The latter has never made a serious attempt of kernel
integration, which may disqualify it to some.

The two main problems of JFFS2 are memory consumption and mount time.
Unlike most filesystems, there is no tree structure of any sorts on
the medium, so the complete medium needs to be scanned at mount time
and a tree structure kept in-memory while the filesystem is mounted.
With bigger devices, both mount time and memory consumption increase
linearly.

JFFS2 has recently gained summary support, which helps reduce mount
time by a constant factor.  Linear scalability remains.  YAFFS also
appears to be better by a constant factor, yet still scales linearly.

LogFS has an on-medium tree, fairly similar to Ext2 in structure, so
mount times are O(1).  In absolute terms, the OLPC system has mount
times of ~3.3s for JFFS2 and ~60ms for LogFS.


Motivation 2:

Flash is becoming increasingly common in standard PC hardware.  Nearly
a dozen different manufacturers have announced Solid State Disks
(SSDs), the OLPC and the Intel Classmate no longer contain hard disks
and ASUS announced a flash-only Laptop series for regular consumers.
And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards,
etc.

Flash behaves significantly different to hard disks.  In order to use
flash, the current standard practice is to add an emulation layer and
an old-fashioned hard disk filesystem.  As can be expected, this is
eating up some of the benefits flash can offer over hard disks.

In principle it is possible to achieve better performance with a flash
filesystem than with the current emulated approach.  In practice our
current flash filesystems are not even near that theoretical goal.
LogFS in its current state is already closer.


Current state:

LogFS works and survives my testcases.  It has fairly good chances of
not eating your data during regular operation.  There are still two
known bugs that will eat data if the filesystem is uncleanly
unmounted.  Also still missing is wear leveling.

Handling of read/write/erase errors currently is BUG().  It is on my
list, no need to remind me. :)

Overall I consider this to be -mm material.  It would be good to get
some review and have the usual allyesconfig crowd build it and find
coverity bugs and the like.

http://logfs.org/logfs/ may have some further information.


Shameless plug:

I have quit my job last November to concentrate on LogFS.  While I
have found one sponsor kind enough to fund me, my monetary reserves
are fairly stressed.  Fairly soon I will be forced to take an
old-fashioned job again and work on other less exciting stuff.  So if
anyone needs a fast flash filesystem and has spare money to spend,
please contact me.

Jörn

-- 
Everything should be made as simple as possible, but not simpler.
-- Albert Einstein
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] LogFS take two

[Albert Cahalan]

joern@lazybastard.org, akpm@osdl.org,
linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org,
shaggy@linux.vnet.ibm.com, dgc@sgi.com

Re: [PATCH 0/2] LogFS take two

You seem to be missing the immutable bit. This is really useful
for dealing with buggy or badly-designed things running as root.
I've used to to protect /dev/null from becoming a normal file
filled with junk, and to protect /etc/resolv.conf from "helpful"
network management daemons that don't know my DNS servers.

Anything else missing?

BTW, BSD offers an unprivileged immutable bit as well. I'm sure
it's useful for the apps that trash their own config files.
Actually, this bit alone would do fine, and we could really use
a way to protect writable device files from deletion or permission
bit changes.

Re: [PATCH 0/2] LogFS take two

[Thomas Gleixner]

On Mon, 2007-05-07 at 23:59 +0200, Jörn Engel wrote:

> LogFS has an on-medium tree, fairly similar to Ext2 in structure, so
> mount times are O(1).  In absolute terms, the OLPC system has mount
> times of ~3.3s for JFFS2 and ~60ms for LogFS.

Impressive number

> Motivation 2:
> 
> Flash is becoming increasingly common in standard PC hardware.  Nearly
> a dozen different manufacturers have announced Solid State Disks
> (SSDs), the OLPC and the Intel Classmate no longer contain hard disks
> and ASUS announced a flash-only Laptop series for regular consumers.

With a hardware controller which allows no direct access to the flash.

> And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards,
> etc.

which again do not allow direct access to the flash

> Flash behaves significantly different to hard disks.  In order to use
> flash, the current standard practice is to add an emulation layer and
> an old-fashioned hard disk filesystem.  As can be expected, this is
> eating up some of the benefits flash can offer over hard disks.
> 
> In principle it is possible to achieve better performance with a flash
> filesystem than with the current emulated approach. 

Err, where does JFFS2 use a block emulation layer ?

> Current state:
> 
> LogFS works and survives my testcases.  It has fairly good chances of
> not eating your data during regular operation.  There are still two
> known bugs that will eat data if the filesystem is uncleanly
> unmounted.  Also still missing is wear leveling.

Are you going to make logfs play with UBI ?

> Handling of read/write/erase errors currently is BUG().  It is on my
> list, no need to remind me. :)
> 
> Overall I consider this to be -mm material.

I don't. It seems fs developers tend to have their own view of how to
get stuff mainline.

The code is far from being useful on real world hardware. The error
handling via BUG() is just making it useless. 

Also please fix the coding style and other issues from the seperate
review.

Some useful comments would make a functional review way easier.

>   It would be good to get
> some review and have the usual allyesconfig crowd build it 

make allyesconfig does not work for you ?

	tglx


-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] LogFS take two

[Jörn Engel]

On Tue, 8 May 2007 09:39:37 +0200, Thomas Gleixner wrote:
> 
> > Motivation 2:
> > 
> > Flash is becoming increasingly common in standard PC hardware.  Nearly
> > a dozen different manufacturers have announced Solid State Disks
> > (SSDs), the OLPC and the Intel Classmate no longer contain hard disks
> > and ASUS announced a flash-only Laptop series for regular consumers.
> 
> With a hardware controller which allows no direct access to the flash.
> 
> > And that doesn't even mention the ubiquitous USB-Sticks, SD-Cards,
> > etc.
> 
> which again do not allow direct access to the flash

I know that and I have talked to manufacturers.  Not allowing direct
access is common practice today, but I didn't encounter much opposition
against allowing it in the future.  What appears to be holding them back
is that there would be absolutely no value in it right now.  With direct
flash access, which filesystem should users choose for their 32GB SSD?

> > Flash behaves significantly different to hard disks.  In order to use
> > flash, the current standard practice is to add an emulation layer and
> > an old-fashioned hard disk filesystem.  As can be expected, this is
> > eating up some of the benefits flash can offer over hard disks.
> > 
> > In principle it is possible to achieve better performance with a flash
> > filesystem than with the current emulated approach. 
> 
> Err, where does JFFS2 use a block emulation layer ?

It doesn't.  Motivation 2 is about SSDs, USB sticks, SD-Cards, etc.
JFFS2 is motivation 1.

> Are you going to make logfs play with UBI ?

It is not very high on my priority list.

> > Handling of read/write/erase errors currently is BUG().  It is on my
> > list, no need to remind me. :)
> > 
> > Overall I consider this to be -mm material.
> 
> I don't. It seems fs developers tend to have their own view of how to
> get stuff mainline.

Maybe.  My view is that I have to solve any problems found until people
consider the code good enough by whatever metric.  The final criterium
appears to be quite fuzzy.

> The code is far from being useful on real world hardware. The error
> handling via BUG() is just making it useless. 

On NOR hardware?  How many write/erase failures does one commonly
encounter there?  Those things will need to get sorted, sure.  But
I doubt whether LogFS is useless on _all_ hardware because of this.

> Also please fix the coding style and other issues from the seperate
> review.

Sure.

> Some useful comments would make a functional review way easier.

Common problem.  Implementor doesn't know what comments would be useful
and reviewer doesn't know where to start without useful comments.  I
will try to add some and would love to see suggestions.

> >   It would be good to get
> > some review and have the usual allyesconfig crowd build it 
> 
> make allyesconfig does not work for you ?

It does.  But I don't have a coverity license, just to give one example.

Jörn

-- 
The wise man seeks everything in himself; the ignorant man tries to get
everything from somebody else.
-- unknown
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/2] LogFS take two

[Jörn Engel]

On Tue, 8 May 2007 01:53:38 -0400, Albert Cahalan wrote:
> 
> You seem to be missing the immutable bit. This is really useful
> for dealing with buggy or badly-designed things running as root.
> I've used to to protect /dev/null from becoming a normal file
> filled with junk, and to protect /etc/resolv.conf from "helpful"
> network management daemons that don't know my DNS servers.

Sounds useful.  Onto my todo list.  And since the list is slowly getting
too long to be memorized, I've added it to my wiki:
http://logfs.org/logfs/todo

> Anything else missing?
> 
> BTW, BSD offers an unprivileged immutable bit as well. I'm sure
> it's useful for the apps that trash their own config files.
> Actually, this bit alone would do fine, and we could really use
> a way to protect writable device files from deletion or permission
> bit changes.

It would be relatively easy to add this as well.  The biggest obstacle I
see is getting support in chattr(1).  Adding Ted to Cc:, as he is the
maintainer.

What remains to be decided is whether such a flag is a useful addition.
My gut feeling is yes, but I would like to have more than two votes in
favor.

Jörn

-- 
Fools ignore complexity.  Pragmatists suffer it.
Some can avoid it.  Geniuses remove it.
-- Perlis's Programming Proverb #58, SIGPLAN Notices, Sept.  1982
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html