JFFS2 & NAND

JFFS2 & NAND

[Thomas Gleixner]

Since the discussion is broken and I'm answering my question to myself for a 
few days, I ask me, if the interest on this problem is gone.

I have spent a lot of time to make this running. It does, except one 
unresolved issue, when we fail in flush_wbuf and wbuf contains more than the 
actual node.
I'm willing to solve this too, but as I mentioned before, my knowlegde of 
JFFS2 is not deep enough yet. Is there anybody, who want's to implement this 
basicly or give me at least some pointers how to do it.

Thomas

Re: JFFS2 & NAND

[Peter De Schrijver]

Hi,

On Thu, Feb 14, 2002 at 04:57:00PM +0100, Thomas Gleixner wrote:
> Since the discussion is broken and I'm answering my question to myself for a 
> few days, I ask me, if the interest on this problem is gone.
> 

No, certainly not. Unfortunately I don't have any hardware available now
to test on. As soon as I have, I will certainly test JFFS2 on NAND and
help fixing bugs.

> I have spent a lot of time to make this running. It does, except one 
> unresolved issue, when we fail in flush_wbuf and wbuf contains more than the 
> actual node.
> I'm willing to solve this too, but as I mentioned before, my knowlegde of 
> JFFS2 is not deep enough yet. Is there anybody, who want's to implement this 
> basicly or give me at least some pointers how to do it.
> 

Peter.

Re: JFFS2 & NAND

[David Woodhouse]

gleixner@autronix.de said:
> Since the discussion is broken and I'm answering my question to myself
> for a  few days, I ask me, if the interest on this problem is gone.

> I have spent a lot of time to make this running. It does, except one
> unresolved issue, when we fail in flush_wbuf and wbuf contains more
> than the  actual node. I'm willing to solve this too, but as I
> mentioned before, my knowlegde of  JFFS2 is not deep enough yet. Is
> there anybody, who want's to implement this  basicly or give me at
> least some pointers how to do it.

I'm sorry I haven't been particularly responsive for the last few days. It's
not that I'm not interested, more that the last few days included 24 hours
in a tin can coming back from linux.conf.au and I don't think I'm awake
enough yet to attempt it :)

--
dwmw2

Re: JFFS2 & NAND

[Thomas Gleixner]

On Thursday, 14. February 2002 18:27, David Woodhouse wrote:
> gleixner@autronix.de said:
> I'm sorry I haven't been particularly responsive for the last few days.
> It's not that I'm not interested, more that the last few days included 24
> hours in a tin can coming back from linux.conf.au and I don't think I'm
> awake enough yet to attempt it :)
Ok no problem, i was a little bit irritated, that a lot of other issues were 
answered and did not know, why the discussion was broken. So this was only a 
well meant wakeup call.

I Include my latest statement on this, to have this new thread up to date.

Actual state of JFFS2 on NAND

I implemented a new structure into nand_chip structure, which describes the 
usage of the oob area. You can read and modify this structure from the 
filesystem driver via two functions, which i have implemented into the mtd 
structure. (get_oobcfg, set_oobcfg).
The mtd_oob_config structure is:
/* configuration for out of band data (NAND,DOC...) */
struct mtd_oob_config {
         int     oob_size;       /* size of out of band area */
         int     ecc_pos[6];     /* position of ECC bytes inside oob */
         int     badblock_pos;   /* position of bad block flag inside oob -1 
= inactive */
         int     eccvalid_pos;   /* position of ECC valid flag inside oob -1 
= inactive */
         int     fsdata_pos;     /* position to store filesystem specific 
information */
         int     fsdata_len;     /* length available for filesystem specific 
data */
};
With this structure we can decide at mount time, which scheme we use. Either 
the fs driver modifies it to match it's requirements or the fs driver uses 
the information given by the device driver. At the moment i use the 
information given by the device driver.

I have now fixed the most NAND related FIXME's inside JFFS2, except the 
problem with wbuf_flush fail. The cleanmarker in the oob area and the bad 
block marking is working pretty good. I modified the scan routine to collect 
bad block information. I have a old SmartMediaCard, which has bad blocks on 
it. The information is collected correct. Then I erased one of the bad blocks 
and destroyed the bad block flag. When gc erased this block again it was 
detected as bad (not totaly erased) and marked bad again. 

I ran some stress tests on the filesystem and encountered no serious problem, 
except when i used a wornout old SMCard from a digicam i ran into the 
flush_wbuf bug.

Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[David Woodhouse]

gleixner@autronix.de said:
> I implemented a new structure into nand_chip structure, which
> describes the  usage of the oob area. You can read and modify this
> structure from the  filesystem driver via two functions, which i have
> implemented into the mtd  structure. (get_oobcfg, set_oobcfg).

I'm slightly concerned that this may be overkill. The JFFS2 code has to
recognise certain types of hardware and arrange its own data in the spare
area accordingly. Does it really need to make that information available, or
just do its own thing? Is the existing mtd->{oobblock,oobsize,ecctype}
really not sufficient?


> I have now fixed the most NAND related FIXME's inside JFFS2, except
> the  problem with wbuf_flush fail. The cleanmarker in the oob area and
> the bad  block marking is working pretty good. I modified the scan
> routine to collect  bad block information. I have a old
> SmartMediaCard, which has bad blocks on  it. The information is
> collected correct. Then I erased one of the bad blocks  and destroyed
> the bad block flag. When gc erased this block again it was  detected
> as bad (not totaly erased) and marked bad again.  

Excellent. 

>  I ran some stress tests on the filesystem and encountered no serious
> problem,  except when i used a wornout old SMCard from a digicam i ran
> into the  flush_wbuf bug. 

OK, let's work on getting your current changes merged, then we can deal 
with the wbuf_flush failure handling. Can you send me a patch against the 
current CVS tree -- I don't think I've changed anything on the 
jffs2-nand-branch recently (since the 5th of February in fact).

--
dwmw2

Re: JFFS2 & NAND

[Thomas Gleixner]

On Sunday, 17. February 2002 10:51, David Woodhouse wrote:
> gleixner@autronix.de said:
> > I implemented a new structure into nand_chip structure, which
> > describes the  usage of the oob area. You can read and modify this
> > structure from the  filesystem driver via two functions, which i have
> > implemented into the mtd  structure. (get_oobcfg, set_oobcfg).
>
> I'm slightly concerned that this may be overkill. The JFFS2 code has to
> recognise certain types of hardware and arrange its own data in the spare
> area accordingly. Does it really need to make that information available,
> or just do its own thing? Is the existing mtd->{oobblock,oobsize,ecctype}
> really not sufficient?
No. Both the chip layer e.g. NAND, DOC, or something else and every 
filesystem layer has to know about the arrangement of the oob data. Maybe we 
could reduce the information to a filesystem type and define the oob-layout 
as constants in a .h file. That would be less overhead but gives the 
flexibility for both chip and filesystem driver to use free oob layouts.
It would be not a big hack to rearrange it that way.

> OK, let's work on getting your current changes merged, then we can deal
> with the wbuf_flush failure handling. Can you send me a patch against the
> current CVS tree -- I don't think I've changed anything on the
> jffs2-nand-branch recently (since the 5th of February in fact).
Comes along, if we have fixed above question.
 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[David Woodhouse]

gleixner@autronix.de said:
> Maybe we  could reduce the information to a filesystem type and define
> the oob-layout  as constants in a .h file. That would be less overhead
> but gives the  flexibility for both chip and filesystem driver to use
> free oob layouts. It would be not a big hack to rearrange it that way.

I think that might be best. After all, there are only a limited number of 
such arrangements. 

I'd actually like to overhaul all the oob and ecc stuff in the mtd_info 
structure - I don't like any of it very much. If we can make do with the 
existing API for a while longer, then we can give some serious thought to 
its replacement. 

--
dwmw2

Re: JFFS2 & NAND

[Thomas Gleixner]

On Sunday, 17. February 2002 18:25, David Woodhouse wrote:
> gleixner@autronix.de said:
> > Maybe we  could reduce the information to a filesystem type and define
> > the oob-layout  as constants in a .h file. That would be less overhead
> > but gives the  flexibility for both chip and filesystem driver to use
> > free oob layouts. It would be not a big hack to rearrange it that way.
>
> I think that might be best. After all, there are only a limited number of
> such arrangements.
>
> I'd actually like to overhaul all the oob and ecc stuff in the mtd_info
> structure - I don't like any of it very much. If we can make do with the
> existing API for a while longer, then we can give some serious thought to
> its replacement.
I looked for the usage of eectype in MTD. There's only nand.c, which writes a 
value to this field. We could use a OR-value for the oob-configuration and 
use this field for our oob-layout selection.
 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[Thomas Gleixner]

> On Sunday, 17. February 2002 18:25, David Woodhouse wrote:
> > I'd actually like to overhaul all the oob and ecc stuff in the mtd_info
> > structure - I don't like any of it very much. If we can make do with the
> > existing API for a while longer, then we can give some serious thought to
> > its replacement.
Would it be ok to use a CONFIG option to tell the chip device driver which 
OOB configuration to use ?
-- 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[David Woodhouse]

gleixner@autronix.de said:
> Would it be ok to use a CONFIG option to tell the chip device driver
> which  OOB configuration to use ?

switch(mtd->ecctype) ?


--
dwmw2

Re: JFFS2 & NAND

[Thomas Gleixner]

On Monday, 18. February 2002 09:46, David Woodhouse wrote:
> gleixner@autronix.de said:
> > Would it be ok to use a CONFIG option to tell the chip device driver
> > which  OOB configuration to use ?
>
> switch(mtd->ecctype) ?
Yes inside JFFS2, but who tells the NAND driver, which layout to use.
I think we should support this, to keep the possibility to implement other fs 
than JFFS2. (e.g. this ugly DOS-FAT of SMCard). So we could extend the 
CONFIG_NAND_ECC by a CONFIG_NAND_ECC_JFFS or something like that.
-- 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[David Woodhouse]

gleixner@autronix.de said:
> Yes inside JFFS2, but who tells the NAND driver, which layout to use.

Do we really need the NAND driver to be able to use more than one layout? 

Either we have hardware ECC which determines the layout, or the NAND driver 
implements software ECC and gets to choose. Can't the filesystem just deal 
with the ECC layout used by the NAND driver, as determined by the 
underlying hardware?


--
dwmw2

Re: JFFS2 & NAND

[Thomas Gleixner]

On Monday, 18. February 2002 10:29, David Woodhouse wrote:
> gleixner@autronix.de said:
> > Yes inside JFFS2, but who tells the NAND driver, which layout to use.
>
> Do we really need the NAND driver to be able to use more than one layout?
>
> Either we have hardware ECC which determines the layout, or the NAND driver
> implements software ECC and gets to choose. Can't the filesystem just deal
> with the ECC layout used by the NAND driver, as determined by the
> underlying hardware?
Also hardware ECC does not determine the layout. It just builds the ECC, you 
have to read it out of the CLPD and write it at a appropriate place in the 
OOB area. Same on read.  
We can use a fixed layout for every chip-type. There are 2 good reasons not 
to do that.
1. The SMC DOS-FAT oob layout is more than ugly. To press our idea of oob 
usage into this, is like you slice a bread in many small pieces to fit into 
the subdivisions of your tupperbox, instead of rearanging the contents of the 
box.

2. I don't want to put too much restrictions into the oob layout, so we would 
make it hard for somebody, who want's to implement a different filesystem 
than JFFS2.

-- 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de  

Re: JFFS2 & NAND

[Thomas Gleixner]

I hacked it, so that JFFS2 uses the c->mtd->ecctype to decide which oob 
layout is given. I tested the system again and destroyed the ECC data on one 
page. When i tried to mount JFFS2 again it failed, because the nand driver 
returned -EIO, as the bad page ECC was seen. 
That's not the way it can go. You have one bad ECC page and all your data are 
lost. 
There are two possibilities to react:
1. Return the raw page data without ECC correction and return 0 (success) and 
let the fs driver detect the invalid stuff on it's own, what JFFS2 actually 
does.

2. Return the raw page data without ECC correction and return a value > 0 to 
inform the fs driver that ECC failed and data maybe corrupted.

I think the 2nd one is the better one. But it gives us a lot of hacking 
inside JFFS2.

-- 
Thomas
__________________________________________________
Thomas Gleixner, autronix automation GmbH
auf dem berg 3, d-88690 uhldingen-muehlhofen
fon: +49 7556 919891 , fax: +49 7556 919886
mail: gleixner@autronix.de, http://www.autronix.de