JFFS2 Support for Large Flash Designs

JFFS2 Support for Large Flash Designs

[Johnson, Charles F]

My understanding is that JFFS2 does not support flash file systems
larger than 4GB.  What would be the issues with supporting more than 4GB
of NAND flash.

Charles Johnson
Ultra-Mobility Group
Platform Software Engineering
Intel Corporation
charles.f.johnson@intel.com

Re: JFFS2 Support for Large Flash Designs

[Josh Boyer]

On 3/6/07, Johnson, Charles F <charles.f.johnson@intel.com> wrote:
> My understanding is that JFFS2 does not support flash file systems
> larger than 4GB.  What would be the issues with supporting more than 4GB
> of NAND flash.

Creating JFFS3.  Or LogFS.  Seriously.  And support in MTD for devices > 4GiB.

To support more than 4GiB, you need to change the internal variables
used to track sizes and offsets.  But as I said before, the log
structured nature of JFFS2 will present practical limits long before
4GiB because JFFS2 scales linearly.  Scanning a 1GiB JFFS2 filesystem
can already take minutes depending on various factors such as CPU
speed and filesystem utilization.

LogFS is under work to avoid these scaling issues as it doesn't
require a full scan of flash.  JFFS3, when it comes about, will do
something similar from what I understand.

josh

RE: JFFS2 Support for Large Flash Designs

[Johnson, Charles F]

What is this list's opinion of YAFFS ? 


Charles Johnson
Ultra-Mobility Group
Platform Software Engineering
Intel Corporation
charles.f.johnson@intel.com
503-712-5181

-----Original Message-----
From: Josh Boyer [mailto:jwboyer@gmail.com] 
Sent: Tuesday, March 06, 2007 5:37 PM
To: Johnson, Charles F
Cc: linux-mtd@lists.infradead.org
Subject: Re: JFFS2 Support for Large Flash Designs

On 3/6/07, Johnson, Charles F <charles.f.johnson@intel.com> wrote:
> My understanding is that JFFS2 does not support flash file systems
> larger than 4GB.  What would be the issues with supporting more than
4GB
> of NAND flash.

Creating JFFS3.  Or LogFS.  Seriously.  And support in MTD for devices >
4GiB.

To support more than 4GiB, you need to change the internal variables
used to track sizes and offsets.  But as I said before, the log
structured nature of JFFS2 will present practical limits long before
4GiB because JFFS2 scales linearly.  Scanning a 1GiB JFFS2 filesystem
can already take minutes depending on various factors such as CPU
speed and filesystem utilization.

LogFS is under work to avoid these scaling issues as it doesn't
require a full scan of flash.  JFFS3, when it comes about, will do
something similar from what I understand.

josh

Re: JFFS2 Support for Large Flash Designs

[Josh Boyer]

On 3/6/07, Johnson, Charles F <charles.f.johnson@intel.com> wrote:
> What is this list's opinion of YAFFS ?

Personally, I've never used it so I have no opinion.  My understanding
is that it only works on NAND flash.  Charles Manning and the yaffs
devel list would be able to answer your questions, along with a
handful of people here.

josh

Re: JFFS2 Support for Large Flash Designs

[Jörn Engel]

Disclaimer: I am the LogFS developer.  Everything below is my personal
opinion based on possibly outdated information.  Apply a grain of salt.

On Tue, 6 March 2007 17:38:04 -0800, Johnson, Charles F wrote:
> 
> What is this list's opinion of YAFFS ? 

Advantages:
o It exists and is working.
o From what I heard it is faster than JFFS2.
o Has a relatively simple design.

Disadvantages:
o Code is not in mainline Linux kernel.  I generally distrust such code;
  apart from the support question, it is an indication of other problems.
o Only works on raw NAND flash.
o Has O(n) properties, just like JFFS2.  It will hit the same
  scalability problem at some time in the future.
o No compression support.

Likely the best solution for your problem at the moment.

>> LogFS.

Advantages:
o Code exists.
o Fast mount time (~60ms).
o Has O(1) mount time and memory overhead, O(log(n)) for most other
  things, O(sqrt(n)) for GC.

Disadvantages:
o Still under development.  Medium format is constantly changing.
o Still has known bugs, see http://lazybastard.org/logfs/BUGS
o Currently needs to write more data to flash than JFFS2/YAFFS.

>> JFFS3

Advantages:
???

Disadvantages:
o To my knowledge, code does not exist (in public) yet.
o Garbage collection was an open issue, last time I looked.

Jörn

-- 
Invincibility is in oneself, vulnerability is in the opponent.
-- Sun Tzu

Re: JFFS2 Support for Large Flash Designs

[Vitaly Wool]

Johnson, Charles F wrote:

>My understanding is that JFFS2 does not support flash file systems
>larger than 4GB.  What would be the issues with supporting more than 4GB
>of NAND flash.
>  
>
Excuse me for the stupid (probably) question, but still: are you sure
you want to use a single 4GB partition for read/write operations??

Vitaly

Re: JFFS2 Support for Large Flash Designs

[Charles Manning]

On Wednesday 07 March 2007 19:47, Vitaly Wool wrote:
> Johnson, Charles F wrote:
> >My understanding is that JFFS2 does not support flash file systems
> >larger than 4GB.  What would be the issues with supporting more than 4GB
> >of NAND flash.
>
> Excuse me for the stupid (probably) question, but still: are you sure
> you want to use a single 4GB partition for read/write operations??

I think that for many applications a large partition is preferable to many 
small ones.

If you think of something simple like an MP3 player or such, the user does not 
want to split up their data according to partitions. Partitions just provide 
an artificial constraint that is confusing to many people.

-- Charles

RE: JFFS2 Support for Large Flash Designs

[ian]

On Wednesday 07 March 2007 08:01, Joern Engel wrote:
> Code is not in mainline Linux kernel.  I generally distrust
> such code; apart from the support question, it is an
> indication of other problems.

There are many software components that have intimate knowledge 
of the kernel that are not in "mainline Linux kernel".  There is 
nothing wrong with this. There are many many drivers that are 
outside the "mainline"; much of the support for embedded 
processors and their peripherals is not in the "mainline".
There are some positives of having components maintained outside 
the context of "linux".

Yaffs actually has a life outside linux.  Using "outside the 
mainline" as an argument is weak at best. The 2.6 kernel 
makefile makes building a module outside the tree trivial -- so 
lots of the older build hassles and arguments no longer exist.

Some random programs I build that know about the kernel 
internals: busybox, e2fsprogs, genext2fs, Glibc, net-tools, 
pthreads, uClibc, util-linux, wireless_tools the list goes on, 
with plenty of drivers/modules too.

-imcd

RE: JFFS2 Support for Large Flash Designs

[Thomas Gleixner]

On Wed, 2007-03-07 at 12:23 -0500, ian@brightstareng.com wrote:
> Some random programs I build that know about the kernel 
> internals: busybox, e2fsprogs, genext2fs, Glibc, net-tools, 
> pthreads, uClibc, util-linux, wireless_tools the list goes on, 
> with plenty of drivers/modules too.

There is an important difference between userland tools which have
kernel knowledge and drivers:

The user space interface is _STABLE_ - the kernel interfaces are _NOT_

That's the reason why out of tree drivers, filesystems ... are often out
of sync and contain more #ifdefs than code to cope with the dozens of
kernel versions.

SoC devices out of tree are a seperate steaming pile of ... I have seen
too many chip vendor kernels explode when you just try to do anything
else than run "hello world" on them.

I share Joerns distrust fully.

	tglx

Re: JFFS2 Support for Large Flash Designs

[Lennert Buytenhek]

On Wed, Mar 07, 2007 at 06:48:07PM +0100, Thomas Gleixner wrote:

> SoC devices out of tree are a seperate steaming pile of ... I have
> seen too many chip vendor kernels explode when you just try to do
> anything else than run "hello world" on them.

Having spent quite some time looking at ARM SoC vendor ports, I'd
fully agree with Thomas here.  It's usually better to throw all the
crap they made away and start over from scratch, than to try to make
something useful out of what is there.

RE: JFFS2 Support for Large Flash Designs

[ian]

On Wednesday 07 March 2007 12:48, Thomas Gleixner wrote:
> SoC devices out of tree are a seperate steaming pile of ... I
> have seen too many chip vendor kernels explode when you just
> try to do anything else than run "hello world" on them.

I agree that this is far from ideal -- I have to deal with this 
all the time.  But I also appreciate that some vendors do 
actually bother to contribute some base code to support their 
designs, even if I have to go in and fix it.

The vendors need to do better job at establishing a framework to 
solicit fixes, improvements and new code from their customers, 
integrate it, and feed it up the tree into the mainline. It is 
clearly to their benefit.

-imcd

RE: JFFS2 Support for Large Flash Designs

[Thomas Gleixner]

On Wed, 2007-03-07 at 13:17 -0500, ian@brightstareng.com wrote:
> On Wednesday 07 March 2007 12:48, Thomas Gleixner wrote:
> > SoC devices out of tree are a seperate steaming pile of ... I
> > have seen too many chip vendor kernels explode when you just
> > try to do anything else than run "hello world" on them.
> 
> I agree that this is far from ideal -- I have to deal with this 
> all the time.  But I also appreciate that some vendors do 
> actually bother to contribute some base code to support their 
> designs, even if I have to go in and fix it.
> 
> The vendors need to do better job at establishing a framework to 
> solicit fixes, improvements and new code from their customers, 
> integrate it, and feed it up the tree into the mainline. It is 
> clearly to their benefit.

As well as it would be for out of tree drivers and such. The time people
spend to catch up with mainline to maintain their #ifdef mess would be
much more useful spent when the stuff would be in tree.

	tglx

Re: JFFS2 Support for Large Flash Designs

[Jörn Engel]

Please keep me on the Cc: list.

On Wed, 7 March 2007 12:23:07 -0500, ian@brightstareng.com wrote:
> 
> There are some positives of having components maintained outside 
> the context of "linux".

Can you name those?

> Yaffs actually has a life outside linux.  Using "outside the 
> mainline" as an argument is weak at best. The 2.6 kernel 
> makefile makes building a module outside the tree trivial -- so 
> lots of the older build hassles and arguments no longer exist.

If building it were our only problem...

Problems that cause much more trouble are:
o Following the changing in-kernel APIs.  I maintain several
  out-of-tree projects myself and can tell you it is a pain.  At times I
  don't update my kernel for month because I lack the time to update the
  cowlink patches.
o Problem support.  Unless the code is in mainline, the only person that
  will be willing to assist a user is the one responsible for the
  out-of-tree project.  Such support can be good, on average it tends to
  be somewhat worse.
o Code maturity.  Face it, there is often a reason why the code is not
  in mainline.

How much of the above applies to YAFFS is a seperate question I don't
want to answer.  But in general, being wary of out-of-tree code is a
good thing.

Jörn

-- 
"Error protection by error detection and correction."
-- from a university class