nand oob layout assumptions

nand oob layout assumptions

[David Updegraff]

Hi.

So I have a little ducttape-string-and-bailing wire driver here now that 
deals with these newer bigger Samsung K9xxx chips; am hitting snags 
around the declarations of oob areas.

The ecc calcluations within the driver are easy to deal with, straightup 
algorithm of ecc-space-needed = 3 * #-of-256byte chunks in the page.

But how to cope with that exposed oobinfo structure defn. in mtd.h that 
declares the all the universe uses 6 bytes for ecc in oob?

It seems we need the lo-level driver to declare (and allocate?) that 
oobinfo struct, and while we're at breaking everything by doing so, add 
a badblock_pos element to it, since these samsung chips mark badblocks 
not at offset:5 but offset:0 in oob.

I have no solution dreamed up yet on how to cope with the MEMSETOOBSEL
ioctl, though.  Do you?

-dbu.

Re: nand oob layout assumptions

[Thomas Gleixner]

On Thursday 25 March 2004 19:44, David Updegraff wrote:
> Hi.
>
> So I have a little ducttape-string-and-bailing wire driver here now that
> deals with these newer bigger Samsung K9xxx chips; am hitting snags
> around the declarations of oob areas.
>
> The ecc calcluations within the driver are easy to deal with, straightup
> algorithm of ecc-space-needed = 3 * #-of-256byte chunks in the page.
>
> But how to cope with that exposed oobinfo structure defn. in mtd.h that
> declares the all the universe uses 6 bytes for ecc in oob?

This was valid until now, as we only had 256/512 byte pages and we therefor 
only had 3 or 6 byte ecc. But this is neither a declaration for the universe 
nor for the eternity. What about s/6/24/ ?

> It seems we need the lo-level driver to declare (and allocate?) that
> oobinfo struct, and while we're at breaking everything by doing so, add
> a badblock_pos element to it, since these samsung chips mark badblocks
> not at offset:5 but offset:0 in oob.

Nope, this is done by the filesystem driver, as the filesystem decides how to 
use the oob area. This interface was introduced to make the usage of 
different fs on diffferent partitions possible. The userspace acess is also 
covered by this. There's nothing to change and nothing to break.

The badblockpos is a given value per chip not per parition / filesystem. 5 for 
256/512 byte pages and 0 for the big pages. This must be handled by the nand 
driver itself depending on the chip type and the fs driver must have access 
to this information through the mtd->chip interface. 

> I have no solution dreamed up yet on how to cope with the MEMSETOOBSEL
> ioctl, though.  Do you?

Where's the problem ? 
copy_from_user(&mtd->oobinfo ,(void *)arg, sizeof(struct nand_oobinfo))
I can't see anything to dream of :)

There are more complicated unaddressed issues:

The changes you are doing go for a full page write AFAICS, which will waste a 
lot of space on the device, as we have to flush the writebuffer in jffs2 over 
a timeout to "guarantee" that data is written to flash, or to write a full 
page even for small nodes in YAFFS, which does a page aligned write. 
The "big" devices allow up to 4 consecutive writes to a page, which should be 
made available to the fs driver. This can either mean 4 * 256/512 bytes or a 
4 * random size. The ecc stuff must be made aware of this possibility. The 
256/512byte pagesize types could use this feature too. 

We should check the possibility to use the 16 bit buswith with the same 
driver. The command interface is still 8 bits, only the data interface is 16 
bit. It should be possible to combine those in one go.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[Charles Manning]

On Friday 26 March 2004 07:58, Thomas Gleixner wrote:
> On Thursday 25 March 2004 19:44, David Updegraff wrote:
> > Hi.
> >
> > So I have a little ducttape-string-and-bailing wire driver here now that
> > deals with these newer bigger Samsung K9xxx chips; am hitting snags
> > around the declarations of oob areas.
> >
> > The ecc calcluations within the driver are easy to deal with, straightup
> > algorithm of ecc-space-needed = 3 * #-of-256byte chunks in the page.
> >
> > But how to cope with that exposed oobinfo structure defn. in mtd.h that
> > declares the all the universe uses 6 bytes for ecc in oob?
>
> This was valid until now, as we only had 256/512 byte pages and we therefor
> only had 3 or 6 byte ecc. But this is neither a declaration for the
> universe nor for the eternity. What about s/6/24/ ?
>
> > It seems we need the lo-level driver to declare (and allocate?) that
> > oobinfo struct, and while we're at breaking everything by doing so, add
> > a badblock_pos element to it, since these samsung chips mark badblocks
> > not at offset:5 but offset:0 in oob.
>
> Nope, this is done by the filesystem driver, as the filesystem decides how
> to use the oob area. This interface was introduced to make the usage of
> different fs on diffferent partitions possible. The userspace acess is also
> covered by this. There's nothing to change and nothing to break.
>
> The badblockpos is a given value per chip not per parition / filesystem. 5
> for 256/512 byte pages and 0 for the big pages. This must be handled by the
> nand driver itself depending on the chip type and the fs driver must have
> access to this information through the mtd->chip interface.
>
> > I have no solution dreamed up yet on how to cope with the MEMSETOOBSEL
> > ioctl, though.  Do you?
>
> Where's the problem ?
> copy_from_user(&mtd->oobinfo ,(void *)arg, sizeof(struct nand_oobinfo))
> I can't see anything to dream of :)
>
> There are more complicated unaddressed issues:
>
> The changes you are doing go for a full page write AFAICS, which will waste
> a lot of space on the device, as we have to flush the writebuffer in jffs2
> over a timeout to "guarantee" that data is written to flash, or to write a
> full page even for small nodes in YAFFS, which does a page aligned write.
> The "big" devices allow up to 4 consecutive writes to a page, which should
> be made available to the fs driver. This can either mean 4 * 256/512 bytes
> or a 4 * random size. The ecc stuff must be made aware of this possibility.
> The 256/512byte pagesize types could use this feature too.
>
> We should check the possibility to use the 16 bit buswith with the same
> driver. The command interface is still 8 bits, only the data interface is
> 16 bit. It should be possible to combine those in one go.

I think it is time we re-think the whole mtd interface for NAND. There are a 
bunch of new NAND parts out there that bring a whole new bunch of issues.

For instance, the bad block handling is different in different NAND parts. It 
is wrong, IMHO, to be handling bad-block logic in the file system. Instead, 
the bad block handling should be done in mtd so that each different chip can 
have its own methods and the file system can be kept "clean". 

This is, BTW, the approach I have taken for YAFFS2. YAFFS2 has no ECC calcs 
or bad block logic in the file system "guts". Instead the mtd glue layer is 
currently tasked with this job, though in time I hope the actiual mtd ends up 
with the job.

-- Charles

Re: nand oob layout assumptions

[Thomas Gleixner]

On Saturday 27 March 2004 08:40, Charles Manning wrote:
> >
> > We should check the possibility to use the 16 bit buswith with the same
> > driver. The command interface is still 8 bits, only the data interface is
> > 16 bit. It should be possible to combine those in one go.
>
> I think it is time we re-think the whole mtd interface for NAND. There are
> a bunch of new NAND parts out there that bring a whole new bunch of issues.
>
> For instance, the bad block handling is different in different NAND parts.
> It is wrong, IMHO, to be handling bad-block logic in the file system.
> Instead, the bad block handling should be done in mtd so that each
> different chip can have its own methods and the file system can be kept
> "clean".

I have already considered this. 
But the fs must be aware of the bad block marker position in the OOB area, as 
it can not use this byte for storing fs dependend data. The OOB usage is 
given by the fs layer.

> This is, BTW, the approach I have taken for YAFFS2. YAFFS2 has no ECC calcs
> or bad block logic in the file system "guts". Instead the mtd glue layer is
> currently tasked with this job, though in time I hope the actiual mtd ends
> up with the job.

Not sure. The generic nand driver provides mechanisms to tell you where the 
bad block marker is located and a basic protection against the erasing of bad 
blocks. All other bad block handling tasks should be done in the fs layer.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[David Woodhouse]

On Sat, 2004-03-27 at 09:07 +0100, Thomas Gleixner wrote:
> > For instance, the bad block handling is different in different NAND parts.

And in some cases in the _same_ NAND parts in different environments.
Consider the DiskOnChip, where on first format we look at the 'bad
block' markers in the OOB areas and build a separate bad block bitmap in
a dedicated sector. Then we erase the good blocks and use _every_ byte
in the OOB area. That's why I added the ->block_is_bad() callback from
the NAND code to the board-specific code, but I'm not sure that's
sufficient.

> > It is wrong, IMHO, to be handling bad-block logic in the file system.
> > Instead, the bad block handling should be done in mtd so that each
> > different chip can have its own methods and the file system can be kept
> > "clean".
> 
> I have already considered this. 
> But the fs must be aware of the bad block marker position in the OOB area, as 
> it can not use this byte for storing fs dependend data. The OOB usage is 
> given by the fs layer.

It might not be in the OOB area. The fs does need to be told which bytes
of the OOB area are reserved for hardware ECC and/or bad block markers,
but the hardware driver should still be responsible for bad block
handling.

> Not sure. The generic nand driver provides mechanisms to tell you where the 
> bad block marker is located and a basic protection against the erasing of bad 
> blocks. All other bad block handling tasks should be done in the fs layer.

Other than noticing them and avoiding them, there really isn't much
handling to be done.

-- 
dwmw2

Re: nand oob layout assumptions

[Thomas Gleixner]

On Saturday 27 March 2004 11:24, David Woodhouse wrote:
> On Sat, 2004-03-27 at 09:07 +0100, Thomas Gleixner wrote:
> > > For instance, the bad block handling is different in different NAND
> > > parts.
>
> And in some cases in the _same_ NAND parts in different environments.
> Consider the DiskOnChip, where on first format we look at the 'bad
> block' markers in the OOB areas and build a separate bad block bitmap in
> a dedicated sector. Then we erase the good blocks and use _every_ byte
> in the OOB area. That's why I added the ->block_is_bad() callback from
> the NAND code to the board-specific code, but I'm not sure that's
> sufficient.

What else would you need ? 

I'm not happy to build in the bad block bitmap stuff as a general solution. 
Next will be translation tables, bad block remapping and all kind of crap, 
which is not neccecary to run a NAND FLASH aware filesystem. 

> > > It is wrong, IMHO, to be handling bad-block logic in the file system.
> > > Instead, the bad block handling should be done in mtd so that each
> > > different chip can have its own methods and the file system can be kept
> > > "clean".
> >
> > I have already considered this.
> > But the fs must be aware of the bad block marker position in the OOB
> > area, as it can not use this byte for storing fs dependend data. The OOB
> > usage is given by the fs layer.
>
> It might not be in the OOB area. The fs does need to be told which bytes
> of the OOB area are reserved for hardware ECC and/or bad block markers,
> but the hardware driver should still be responsible for bad block
> handling.

The NAND specs have defined to have the bad block markers in the OOB area, 
which is quite clever, as it does not restrict the usage of any byte in the 
main area.

Nand.c needs bad block handling in the sense of preventing erase of bad 
blocks. Maybe the prevention of reading / writing bad blocks too. And the 
possibility to have a bad block detection function, which covers specific 
cases like DOC.

But this does not change the interface. 

ECC positions are a totaly different thing. If we want to have a fixed scheme 
for storing ECC bytes, which is not neccecary even with hardware ECC, as the 
hardware ECC only does the ECC calculation and no placing of ECC data into 
OOB, then we have to use _ONE_ final ECC placement scheme for all filesystems 
and therefor we can only use the SmartmediaCard layout, as doing not so would 
prevent the development of a SmartMedia FAT fs driver. This would break YAFFS 
support complete. JFFS2 uses the SmartMedia layout anyway.

> > Not sure. The generic nand driver provides mechanisms to tell you where
> > the bad block marker is located and a basic protection against the
> > erasing of bad blocks. All other bad block handling tasks should be done
> > in the fs layer.
>
> Other than noticing them and avoiding them, there really isn't much
> handling to be done.

True, but it still is a job of the filesystem and therefor it must have access 
to this information.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[David Woodhouse]

On Sat, 2004-03-27 at 12:10 +0100, Thomas Gleixner wrote:
> What else would you need ? 

Nothing more, but it does live where it is in the hardware driver.

> I'm not happy to build in the bad block bitmap stuff as a general solution. 
> Next will be translation tables, bad block remapping and all kind of crap, 
> which is not neccecary to run a NAND FLASH aware filesystem. 

We don't need to build it in as a general solution -- the DiskOnChip
code will do it itself. All we need to do is keep the API sufficiently
versatile that the FS doesn't need to know about the peculiarities of
the DiskOnChip -- it just gets told which bytes of the OOB area it can
use, and does so.

The DiskOnChip uses the first 6 bytes of the OOB area for hardware ECC,
and bad block information is elsewhere on the medium.

> The NAND specs have defined to have the bad block markers in the OOB area, 
> which is quite clever, as it does not restrict the usage of any byte in the 
> main area.

No, the NAND specs have defined the bad block markers to be in the OOB
area when the device is shipped. What you do with it after that is up to
you. And in the case of the DiskOnChip that means the bad block
information is moved. We _really_ want to honour that and keep it within
the hardware driver. I don't want any possibility of erasing bad blocks,
and I'm not overly happy with thinking non-bad blocks are bad just
because the device has already been used with NFTL and has ECC data in
byte 5.

> Nand.c needs bad block handling in the sense of preventing erase of bad 
> blocks. Maybe the prevention of reading / writing bad blocks too. And the 
> possibility to have a bad block detection function, which covers specific 
> cases like DOC.
> 
> But this does not change the interface. 

Except that in the file system we need to call the device's
->is_bad_block() function rather than poking at the OOB area.
 
> ECC positions are a totaly different thing. If we want to have a fixed scheme 
> for storing ECC bytes, which is not neccecary even with hardware ECC, as the 
> hardware ECC only does the ECC calculation and no placing of ECC data into 
> OOB, 

On the DiskOnChip I think it has to be the first six bytes of the OOB.
You read <data + ECC> and check a _bit_ to see if there was an error.
You don't read the ECC syndrome and compare with what you separately
read from the medium.

> then we have to use _ONE_ final ECC placement scheme for all filesystems 
> and therefor we can only use the SmartmediaCard layout, as doing not so would 
> prevent the development of a SmartMedia FAT fs driver. This would break YAFFS 
> support complete. JFFS2 uses the SmartMedia layout anyway.

We don't need a fixed ECC scheme. But we do need to tell the file system
what we support. Those should include the SmartMedia software ECC, and
may include the DiskOnChip hardware ECC too. 

JFFS2 uses the DiskOnChip ECC on DiskOnChip already, doesn't it?

> > Other than noticing them and avoiding them, there really isn't much
> > handling to be done.
> 
> True, but it still is a job of the filesystem and therefor it must have access 
> to this information.

Yes. It _absolutely_ needs access to the information 'is this a bad
block?'. What it doesn't need is stuff like "what byte in the OOB area
would tell us if this was a bad block".

-- 
dwmw2

Re: nand oob layout assumptions

[Thomas Gleixner]

On Saturday 27 March 2004 12:25, David Woodhouse wrote:
> On Sat, 2004-03-27 at 12:10 +0100, Thomas Gleixner wrote:
> > What else would you need ?
>
> Nothing more, but it does live where it is in the hardware driver.
>
> > I'm not happy to build in the bad block bitmap stuff as a general
> > solution. Next will be translation tables, bad block remapping and all
> > kind of crap, which is not neccecary to run a NAND FLASH aware
> > filesystem.
>
> We don't need to build it in as a general solution -- the DiskOnChip
> code will do it itself. All we need to do is keep the API sufficiently
> versatile that the FS doesn't need to know about the peculiarities of
> the DiskOnChip -- it just gets told which bytes of the OOB area it can
> use, and does so.
>
> The DiskOnChip uses the first 6 bytes of the OOB area for hardware ECC,
> and bad block information is elsewhere on the medium.

OK, that's DOC specific. And the functions is there. Nobody is going to remove 
it.

> > The NAND specs have defined to have the bad block markers in the OOB
> > area, which is quite clever, as it does not restrict the usage of any
> > byte in the main area.
>
> No, the NAND specs have defined the bad block markers to be in the OOB
> area when the device is shipped. What you do with it after that is up to
> you. And in the case of the DiskOnChip that means the bad block
> information is moved. We _really_ want to honour that and keep it within
> the hardware driver. I don't want any possibility of erasing bad blocks,
> and I'm not overly happy with thinking non-bad blocks are bad just
> because the device has already been used with NFTL and has ECC data in
> byte 5.

So the evil thing is moving the bad block information to a different place.
If some braindead hardware engineers decide to move this information then we 
unfortunately have to deal with the consequences in a hardware specific 
driver which is aware of that crap, but moving it should _NOT_ be encouraged 
by providing a generic possibility to do so. 

We have built a HW ECC which does error correction into a FPGA. The ECC 
positions are programmable, so I can use YAFFS and JFFS2 with it. I did not 
need an extra driver and an extra bad block mechanism. In general should 
people learn to do hardware / software codesign rather than blowing up the 
code with support for braindead hardware.

The nand driver does prevent the erasure of bad blocks already and I'm not 
going to change this.  

> > Nand.c needs bad block handling in the sense of preventing erase of bad
> > blocks. Maybe the prevention of reading / writing bad blocks too. And the
> > possibility to have a bad block detection function, which covers specific
> > cases like DOC.
> >
> > But this does not change the interface.
>
> Except that in the file system we need to call the device's
> ->is_bad_block() function rather than poking at the OOB area.

Correct. No arguments against.

>
> We don't need a fixed ECC scheme. But we do need to tell the file system
> what we support. Those should include the SmartMedia software ECC, and
> may include the DiskOnChip hardware ECC too.
> JFFS2 uses the DiskOnChip ECC on DiskOnChip already, doesn't it?

Yep, but on DOC this is enforced by the hardware. All other things can use an 
ECC placement scheme which fits the filesystem specific OOB layout.

I'm not happy to say that the ECC placements are given by the nand driver, as 
it will break the existing YAFFS code at least, which is not a real good idea, 
as this code is used already in production environment.

> > > Other than noticing them and avoiding them, there really isn't much
> > > handling to be done.
> >
> > True, but it still is a job of the filesystem and therefor it must have
> > access to this information.
>
> Yes. It _absolutely_ needs access to the information 'is this a bad
> block?'. What it doesn't need is stuff like "what byte in the OOB area
> would tell us if this was a bad block".

It needs the information where it is so the fs driver does not put data into 
this place. 

We discussed this already two years ago and decided against a function which 
shuffles around the fs oob data into places which are not used by ecc and bad 
block information and another function which rebuilds the data on read into a 
byte stream which is expected by the fs driver.

The argument there was: removing transparency and flexiblity is evil. 

And I say it's still evil, especially if it breaks existing code.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[David Updegraff]

In all this discussion, the need for anyone outside the driver to know
hardware positions of ECC and badblock markers still elludes me.  Yes, 
various FS need to scribble oob info of their own in a particular 
format, and in a way that won't scribble over ecc/badblock, but is there 
anything more than that?

So, again, what if the 'oobsize' and 'oob_buf' (or a renamed relative 
thereof) were interpreted to mean "region available in OOB for misc. 
file system housekeeping, that will not stomp on private ECC or badblock 
areas and will not be interpreted by nand driver".  And let the driver 
put it wherever it wants/needs to in OOB.  Driver gets to ignorant of FS 
details, FS gets to be ignorant of chip-HW-dependent ECC and badblock 
layout.

>>What else would you need ? 
> 
> 
> Nothing more, but it does live where it is in the hardware driver.
> 
> 
>>I'm not happy to build in the bad block bitmap stuff as a general solution. 
>>Next will be translation tables, bad block remapping and all kind of crap, 
>>which is not neccecary to run a NAND FLASH aware filesystem. 
> 
> 
> We don't need to build it in as a general solution -- the DiskOnChip
> code will do it itself. All we need to do is keep the API sufficiently
> versatile that the FS doesn't need to know about the peculiarities of
> the DiskOnChip -- it just gets told which bytes of the OOB area it can
> use, and does so.
> 
> The DiskOnChip uses the first 6 bytes of the OOB area for hardware ECC,
> and bad block information is elsewhere on the medium.
> 
> 
>>The NAND specs have defined to have the bad block markers in the OOB area, 
>>which is quite clever, as it does not restrict the usage of any byte in the 
>>main area.
> 
> 
> No, the NAND specs have defined the bad block markers to be in the OOB
> area when the device is shipped. What you do with it after that is up to
> you. And in the case of the DiskOnChip that means the bad block
> information is moved. We _really_ want to honour that and keep it within
> the hardware driver. I don't want any possibility of erasing bad blocks,
> and I'm not overly happy with thinking non-bad blocks are bad just
> because the device has already been used with NFTL and has ECC data in
> byte 5.
> 
> 
>>Nand.c needs bad block handling in the sense of preventing erase of bad 
>>blocks. Maybe the prevention of reading / writing bad blocks too. And the 
>>possibility to have a bad block detection function, which covers specific 
>>cases like DOC.
>>
>>But this does not change the interface. 
> 
> 
> Except that in the file system we need to call the device's
> ->is_bad_block() function rather than poking at the OOB area.
>  
> 
>>ECC positions are a totaly different thing. If we want to have a fixed scheme 
>>for storing ECC bytes, which is not neccecary even with hardware ECC, as the 
>>hardware ECC only does the ECC calculation and no placing of ECC data into 
>>OOB, 
> 
> 
> On the DiskOnChip I think it has to be the first six bytes of the OOB.
> You read <data + ECC> and check a _bit_ to see if there was an error.
> You don't read the ECC syndrome and compare with what you separately
> read from the medium.
> 
> 
>>then we have to use _ONE_ final ECC placement scheme for all filesystems 
>>and therefor we can only use the SmartmediaCard layout, as doing not so would 
>>prevent the development of a SmartMedia FAT fs driver. This would break YAFFS 
>>support complete. JFFS2 uses the SmartMedia layout anyway.
> 
> 
> We don't need a fixed ECC scheme. But we do need to tell the file system
> what we support. Those should include the SmartMedia software ECC, and
> may include the DiskOnChip hardware ECC too. 
> 
> JFFS2 uses the DiskOnChip ECC on DiskOnChip already, doesn't it?
> 
> 
>>>Other than noticing them and avoiding them, there really isn't much
>>>handling to be done.
>>
>>True, but it still is a job of the filesystem and therefor it must have access 
>>to this information.
> 
> 
> Yes. It _absolutely_ needs access to the information 'is this a bad
> block?'. What it doesn't need is stuff like "what byte in the OOB area
> would tell us if this was a bad block".
> 

Re: nand oob layout assumptions

[David Woodhouse]

On Sat, 2004-03-27 at 10:13 -0600, David Updegraff wrote:
> In all this discussion, the need for anyone outside the driver to know
> hardware positions of ECC and badblock markers still elludes me.  Yes, 
> various FS need to scribble oob info of their own in a particular 
> format, and in a way that won't scribble over ecc/badblock, but is there 
> anything more than that?

Nope. The file system just need to be told which are bad blocks, where
the ECC data are, and which bytes are still available. In fact, it
possibly doesn't even need to know where the ECC data are in the general
case.

> So, again, what if the 'oobsize' and 'oob_buf' (or a renamed relative 
> thereof) were interpreted to mean "region available in OOB for misc. 
> file system housekeeping, that will not stomp on private ECC or badblock 
> areas and will not be interpreted by nand driver".  And let the driver 
> put it wherever it wants/needs to in OOB.  Driver gets to ignorant of FS 
> details, FS gets to be ignorant of chip-HW-dependent ECC and badblock 
> layout.

Seems reasonable. We can make it a bitmask of available bytes, perhaps?

-- 
dwmw2

Re: nand oob layout assumptions

[Thomas Gleixner]

On Saturday 27 March 2004 17:18, David Woodhouse wrote:
> On Sat, 2004-03-27 at 10:13 -0600, David Updegraff wrote:
> > In all this discussion, the need for anyone outside the driver to know
> > hardware positions of ECC and badblock markers still elludes me.  Yes,
> > various FS need to scribble oob info of their own in a particular
> > format, and in a way that won't scribble over ecc/badblock, but is there
> > anything more than that?
>
> Nope. The file system just need to be told which are bad blocks, where
> the ECC data are, and which bytes are still available. In fact, it
> possibly doesn't even need to know where the ECC data are in the general
> case.
>
> > So, again, what if the 'oobsize' and 'oob_buf' (or a renamed relative
> > thereof) were interpreted to mean "region available in OOB for misc.
> > file system housekeeping, that will not stomp on private ECC or badblock
> > areas and will not be interpreted by nand driver".  And let the driver
> > put it wherever it wants/needs to in OOB.  Driver gets to ignorant of FS
> > details, FS gets to be ignorant of chip-HW-dependent ECC and badblock
> > layout.
>
> Seems reasonable. We can make it a bitmask of available bytes, perhaps?

I'm tired to repeat myself. 

1. This will break existing code and I'm not willing to do so
We accepted in a long discussion 2 years ago, that we support fs supplied ECC 
layouts. YAFFS relies on this. 

2. Putting stuff into any random place is not a thing I want to have.
Changing one line in the placement code will break all existing fs at once. 
Changing an oob layout in a fs driver just breaks this and solely this 
filesystem. 

It may be better and nicer and more clever to have this independent, but
will you explain the people which are using YAFFS on a SmartMedia Card, that 
they can either freeze the current code or accept that the already existing 
cards are not longer readable ? This is valid for soldered flash too. Update 
of existing hardware to a more recent kernel is then not longer possible 
without loosing data. 

I'm not against an _extended_ API, which provides such things for new 
developments, but I'm cowardly refusing to break compability of existing 
stuff.

Can you please accept the matter of fact that breaking existing 
implementations, which are used on removable medias, is not possible.
Thats like changing the FAT or the ISO driver is not possible even if there 
could be better designs available.

Breaking such things only for the sake of a slightly cleaner interface is the 
playfield of M$ and friends. Linux has a good standing for long term 
stability of interfaces and this is a reason for industrial users to choose 
it. They are tired of changing stuff in short intervals, as the lifecycle of 
such products are higher than the M$ incompabilty rate. And now we want to go 
for the same ?

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[Charles Manning]

On Sunday 28 March 2004 04:18, David Woodhouse wrote:
> On Sat, 2004-03-27 at 10:13 -0600, David Updegraff wrote:
> > In all this discussion, the need for anyone outside the driver to know
> > hardware positions of ECC and badblock markers still elludes me.  Yes,
> > various FS need to scribble oob info of their own in a particular
> > format, and in a way that won't scribble over ecc/badblock, but is there
> > anything more than that?
>
> Nope. The file system just need to be told which are bad blocks, where
> the ECC data are, and which bytes are still available. In fact, it
> possibly doesn't even need to know where the ECC data are in the general
> case.

I agree with the idea of abstacting this as much as possible. Where the EEC 
is stores (if indeed it even is stored in OOB) should not be the FS's concern 
and should not be known to the FS. The FS should just be told that the ECC 
passed, failed or fixed a single bit error.

Unless we go for one abstract interface it will [continue to] be hell to 
support more than one FS (it is already a /dev/ass/pain to try get YAFFS and 
JFFS2 going on one machine due to different OOB layouts). The [already bad] 
situation will only get worse when trying to deal with all the fun that the 
newer NAND devices bring us.
>
> > So, again, what if the 'oobsize' and 'oob_buf' (or a renamed relative
> > thereof) were interpreted to mean "region available in OOB for misc.
> > file system housekeeping, that will not stomp on private ECC or badblock
> > areas and will not be interpreted by nand driver".  And let the driver
> > put it wherever it wants/needs to in OOB.  Driver gets to ignorant of FS
> > details, FS gets to be ignorant of chip-HW-dependent ECC and badblock
> > layout.
>
> Seems reasonable. We can make it a bitmask of available bytes, perhaps?

No bitmask. That just passes mucky knowledge through the interface.

Rather:
* Have the mtd tell the FS the number of unused bytes in the OOB.
* Pass a byte array through the interface and the mtd packs/unpacks this 
around the ECC, bad block markers and other stuff.

-- Charles

Re: nand oob layout assumptions

[Thomas Gleixner]

On Sunday 28 March 2004 10:06, Charles Manning wrote:
> I agree with the idea of abstacting this as much as possible. Where the EEC
> is stores (if indeed it even is stored in OOB) should not be the FS's
> concern and should not be known to the FS. The FS should just be told that
> the ECC passed, failed or fixed a single bit error.

No problem with that

> Unless we go for one abstract interface it will [continue to] be hell to
> support more than one FS (it is already a /dev/ass/pain to try get YAFFS
> and JFFS2 going on one machine due to different OOB layouts). The [already
> bad] situation will only get worse when trying to deal with all the fun
> that the newer NAND devices bring us.

He ? I'm running YAFFS and JFFS2 since a long time on the same device and it 
works without any PITA. 

I just want to bring back into memories, that we had a long discussion about 
an abstract interface 2 years ago. There was no way to find a common solution 
as nobody was willing to break either JFFS2 or YAFFS1 or both. 

Again. I agree with the idea of an abstract interface as long as we keep the 
current stuff running. 

> No bitmask. That just passes mucky knowledge through the interface.
>
> Rather:
> * Have the mtd tell the FS the number of unused bytes in the OOB.
> * Pass a byte array through the interface and the mtd packs/unpacks this
> around the ECC, bad block markers and other stuff.

No objections.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[Charles Manning]

On Saturday 27 March 2004 20:07, Thomas Gleixner wrote:
> On Saturday 27 March 2004 08:40, Charles Manning wrote:
> > > We should check the possibility to use the 16 bit buswith with the same
> > > driver. The command interface is still 8 bits, only the data interface
> > > is 16 bit. It should be possible to combine those in one go.
> >
> > I think it is time we re-think the whole mtd interface for NAND. There
> > are a bunch of new NAND parts out there that bring a whole new bunch of
> > issues.
> >
> > For instance, the bad block handling is different in different NAND
> > parts. It is wrong, IMHO, to be handling bad-block logic in the file
> > system. Instead, the bad block handling should be done in mtd so that
> > each different chip can have its own methods and the file system can be
> > kept "clean".
>
> I have already considered this.
> But the fs must be aware of the bad block marker position in the OOB area,
> as it can not use this byte for storing fs dependend data. The OOB usage is
> given by the fs layer.

No. Part of the deal is that the OOB area should not be shown in "raw" form 
to the fs. There should be no l physical placement knowledge in the fs, only 
abstract.


For example if the OOB area is used as follows (standard SmartMedia layout as 
used by YAFFS1):
    __u8  spare0[4];
    __u8  pageStatus; 
    __u8  blockStatus; // bad block marker
   __u8  spare1[2];
    __u8  ecc1[3];
    __u8  spare2[2];;
    __u8  ecc2[3];

This should be shown to the fs as:
  __u8 spare[8];
  ECCResult (OK, fixed, failed)
  BlockOK (OK, bad)

or some such.


>
> > This is, BTW, the approach I have taken for YAFFS2. YAFFS2 has no ECC
> > calcs or bad block logic in the file system "guts". Instead the mtd glue
> > layer is currently tasked with this job, though in time I hope the
> > actiual mtd ends up with the job.
>
> Not sure. The generic nand driver provides mechanisms to tell you where the
> bad block marker is located and a basic protection against the erasing of
> bad blocks. All other bad block handling tasks should be done in the fs
> layer.

But...One of the issues is that the bad block marking strategy of the 
different NAND devices does not necessarily just depend on a fixed-location 
bad block marker.  Some (particularly the 2k page devices) say that any pages 
that are not completely 0xFF when shipped are bad. Therefore one needs to 
track this badness as well as future badness.

The fs should not know about bad block marking positions. The fs should be 
saying to the mtd "mark this block bad" or asking "is this block bad". How 
the mtd chooses to record the fact that the block is bad is not the fs's 
concern.

For example, in a recent implementation of a 2K page NAND system I use block 
0 to store a list of "manufacturing bad blocks". If the array is not found at 
boot, then the driver assumes that this is the first boot for the device and 
builds such a list. Subsequent failures are marked by stomping 0s on the OOB. 
The next time the driver boots, the NAND "badness" is scanned into an array 
that is held in RAM to remember which blocks are bad.

IMHO...

-- Charles

Re: nand oob layout assumptions

[Thomas Gleixner]

On Sunday 28 March 2004 09:34, Charles Manning wrote:
> > But the fs must be aware of the bad block marker position in the OOB
> > area, as it can not use this byte for storing fs dependend data. The OOB
> > usage is given by the fs layer.
>
> No. Part of the deal is that the OOB area should not be shown in "raw" form
> to the fs. There should be no l physical placement knowledge in the fs,
> only abstract.
>
> This should be shown to the fs as:
>   __u8 spare[8];
>   ECCResult (OK, fixed, failed)
>   BlockOK (OK, bad)

OK, I already have agreed, that we can use such an abstract form for new 
developments, but we _MUST_ maintain compability for the existing 
implementatitons. I like abstract models very much, but I will always keep in 
view what consequences we will have, if we change things.

That's all I'm insisting on.

YAFFS1 uses a different ECC placement than JFFS2. 
If we change this in general now, we will either break YAFFS or JFFS2 or even 
both. 

This prevents users to upgrade their kernels. They will have format 
incompabilities so they loose data on upgrading or loose the interoperability 
of systems. Will you tell your customer to either upgrade 1000 already sold 
devices or stick with the current code or accepting that interoperability is 
not given ? That's not what Open Source Software stands for.

I do not have any objections against an API extension, but breaking things I'm 
_NEVER_ going to accept.

-- 
Thomas
________________________________________________________________________
linutronix - competence in embedded & realtime linux
http://www.linutronix.de
mail: tglx@linutronix.de

Re: nand oob layout assumptions

[Charles Manning]

On Sunday 28 March 2004 19:51, Thomas Gleixner wrote:
> On Sunday 28 March 2004 09:34, Charles Manning wrote:
> > > But the fs must be aware of the bad block marker position in the OOB
> > > area, as it can not use this byte for storing fs dependend data. The
> > > OOB usage is given by the fs layer.
> >
> > No. Part of the deal is that the OOB area should not be shown in "raw"
> > form to the fs. There should be no l physical placement knowledge in the
> > fs, only abstract.
> >
> > This should be shown to the fs as:
> >   __u8 spare[8];
> >   ECCResult (OK, fixed, failed)
> >   BlockOK (OK, bad)
>
> OK, I already have agreed, that we can use such an abstract form for new
> developments, but we _MUST_ maintain compability for the existing
> implementatitons. I like abstract models very much, but I will always keep
> in view what consequences we will have, if we change things.
>
> That's all I'm insisting on.
>
> YAFFS1 uses a different ECC placement than JFFS2.
> If we change this in general now, we will either break YAFFS or JFFS2 or
> even both.
>
> This prevents users to upgrade their kernels. They will have format
> incompabilities so they loose data on upgrading or loose the
> interoperability of systems. Will you tell your customer to either upgrade
> 1000 already sold devices or stick with the current code or accepting that
> interoperability is not given ? That's not what Open Source Software stands
> for.
>
> I do not have any objections against an API extension, but breaking things
> I'm _NEVER_ going to accept.

I agree with that fully. We should continue to support the currect 
interfaces, but should perhaps depreciate them. As well as support what is 
already out there we need to also ensure that we provide a more healthy 
framework to move forward too.

Here's what I suggest:

* Keep the existing functions/interface (== the legacy interface) for the 
current concepts so we don't break current implementations.

* Extend the interface with new functions for the new interface.

* Future devices will not necessarily have to support both interfaces. For 
example, if a device has a bad block marking strategy that is not consistent 
with the legacy interface (ie the assumptions in the legacy interface are 
broken for that device)  then it might support the new interface, but not the 
legacy interface.

Current JFFS2, YAFFS1 and current 512byte page NAND devices can use the 
legacy interface and will not get broken.

Future JFFS2, YAFFS1 and 512byte NAND can be written to cope with either. 

YAFFS2 will require the new interface to run in YAFFS2 mode (eg 2k page 
support) and will support either interface to run in YAFFS1 mode.

-- CHarles