DOC filesystem questions

DOC filesystem questions

[Chuck Meade]

I have a DiskOnChip device soldered onto a PowerPC target and
would appreciate if someone could verify if I am using the 
correct commands to put a filesystem on it.

Also I am concerned from what I have read in the archives about
the bad block table.  From what I see in the archives and in
Karim Yaghmour's book, I needed to save this off.  Unfortunately
however I cannot run DOS or its utilities on this target board,
so I don't think that there is any way to do this.

I use the following commands:

  nftl_format /dev/mtd0
  fdisk /dev/nftla

(I know it's unnecessary for now to use partitioning when I just
have 1 partition, but I may be using multiple partitions in the
near future)

  mkfs.ext3 /dev/nftla1
  mount /dev/nftla1 /mnt

This filesystem on the DOC looks like it is working well.

 Questions:

1. Have I destroyed any bad block table in the device by
   doing this command sequence?

2. If so, is this a bad block table needed by firmware
   internal to the DOC, or just needed by some DOS utilities
   that I won't be using anyway?

3. What I would really prefer to use is JFFS2 on this DOC.
   I have seen messages from David that this may now be 
   possible.
   When I use other targets with NOR flash I typically write
   an MTD partition map file for the target, then I use dd to
   put my jffs2 image file into the right mtdblock partition.
   How would I do this with a DOC device?
   (Meaning how do you specify the MTD partition map for your
   DOC so you are able to specify /dev/mtdblock3, for example,
   as a destination for the fs image).

Thanks very much,
Chuck Meade

Re: DOC filesystem questions

[David Woodhouse]

On Wed, 2003-08-06 at 05:27, Chuck Meade wrote:
> I have a DiskOnChip device soldered onto a PowerPC target and
> would appreciate if someone could verify if I am using the 
> correct commands to put a filesystem on it.
> 
> Also I am concerned from what I have read in the archives about
> the bad block table.  From what I see in the archives and in
> Karim Yaghmour's book, I needed to save this off.  Unfortunately
> however I cannot run DOS or its utilities on this target board,
> so I don't think that there is any way to do this.

You can read from /dev/mtd0 after first loading the DiskOnChip driver.
It's in there somewhere (see the NFTL and nftl_format code for more
details of precisely where). In fact, you could probably hack nftldump
to dump it, quite easily. 

> I use the following commands:
> 
>   nftl_format /dev/mtd0
>   fdisk /dev/nftla

Not without 'insmod nftl' or 'reboot -f' between them, I hope.

> 1. Have I destroyed any bad block table in the device by
>    doing this command sequence?

Yes.

> 2. If so, is this a bad block table needed by firmware
>    internal to the DOC, or just needed by some DOS utilities
>    that I won't be using anyway?

Neither. It's a bad block table needed by _all_ DiskOnChip drivers,
including the Linux driver. There is no firmware internal to the DOC now
-- there used to be a BIOS driver in there but you've removed it.

The bad block table is there to prevent the drivers from using blocks
which were marked bad during the factory testing. 

> 3. What I would really prefer to use is JFFS2 on this DOC.
>    I have seen messages from David that this may now be 
>    possible.

It's becoming possible. Give me a few weeks.

>    When I use other targets with NOR flash I typically write
>    an MTD partition map file for the target, then I use dd to
>    put my jffs2 image file into the right mtdblock partition.
>    How would I do this with a DOC device?

See a response I sent to the list a week or so ago -- basically we don't
currently do partitioning on the DiskOnChip's _raw_ flash, although
there's a patch around which hacks up hardcoded partitions, and the
INFTL format on the newest DiskOnChip units does seem to have a
partitioning scheme we might be able to abuse.


-- 
dwmw2

RE: DOC filesystem questions

[Chuck Meade]

David Woodhouse wrote:

> > Also I am concerned from what I have read in the archives about
> > the bad block table.  From what I see in the archives and in
> > Karim Yaghmour's book, I needed to save this off.  Unfortunately
> > however I cannot run DOS or its utilities on this target board,
> > so I don't think that there is any way to do this.
> 
> You can read from /dev/mtd0 after first loading the DiskOnChip driver.
> It's in there somewhere (see the NFTL and nftl_format code for more
> details of precisely where). In fact, you could probably hack nftldump
> to dump it, quite easily. 

It looks like nftldump sees the bad unit table as being the 7680 
bytes starting at byte offset 512 into the device.  So this would
certainly be possible.  After looking more closely at nftl_format
I don't know if it is absolutely necessary to save it though:

nftl_format actually creates a bad unit table at offset 512
beyond the offset that the user provides on the command line.  The
size of the bad unit table it creates is based on the actual number
of erase units in the device, so I suppose it will not always be
7680.  It looks for blocks marked bad in their oob data to create
this table.  It also (by default) will do read/write/erase checking
on each block not already marked bad in its oob data, and any
failures are added to the bad unit table as well.

So it seems the news isn't that bad.  If we always leave the oob
data alone when the oob is marked bad for a block at the factory,
then nftl_format will include it in the BadUnitTable it creates.
Then preserving the factory-set bad block table would only be
critical if it marks blocks bad that are not marked bad in their
oob data.

Is this correct?  It appears that nftl_format is already doing
a good job of preserving bad block data, at least that which comes
from the oob data and r/w/e testing.

> > I use the following commands:
> > 
> >   nftl_format /dev/mtd0
> >   fdisk /dev/nftla
> 
> Not without 'insmod nftl' or 'reboot -f' between them, I hope.

You are correct.  I missed entering that line as I transcribed
my notes.  There is an 'insmod nftl' between those 2 lines in
the command sequence that I used.

> > 3. What I would really prefer to use is JFFS2 on this DOC.
> >    I have seen messages from David that this may now be 
> >    possible.
> 
> It's becoming possible. Give me a few weeks.

That is great news!

Thanks,
Chuck

RE: DOC filesystem questions

[David Woodhouse]

On Thu, 2003-08-07 at 17:09, Chuck Meade wrote:
> It looks like nftldump sees the bad unit table as being the 7680 
> bytes starting at byte offset 512 into the device. 

Er, there's a copy of the Bad Unit Table in each of the two eraseblocks
containing an NFTL Media Header, at offset 512 within each block. It's
not unconditionally offset 512 in the device.

The Bad Unit Table is one byte per logical erase unit, where the
relationship from physical to logical erase units (UnitSizeFactor) is
given in the Media Header itself (i.e. in the struct in the first 512
bytes). It's _normally_ 1:1 but that isn't mandatory.

In devices with 16KiB erase size, the Bad Unit Table can also be
16384-512 bytes, not 7680 (8192-512)

> nftl_format actually creates a bad unit table at offset 512
> beyond the offset that the user provides on the command line.  The
> size of the bad unit table it creates is based on the actual number
> of erase units in the device, so I suppose it will not always be
> 7680.

That's correct. I believe that if it would be more than would fit then
we have to change the UnitSizeFactor though.

>   It looks for blocks marked bad in their oob data to create
> this table.  It also (by default) will do read/write/erase checking
> on each block not already marked bad in its oob data, and any
> failures are added to the bad unit table as well.

The OOB check isn't necessarily correct. I have in front of me a Toshiba
TC58V64AFT datasheet which says...

	"At the time of shipment, all data bytes in a Valid Block
	 at FFh.  For Bad Blocks, all bytes are not in the FFh
	 state. Please don't perform erase operation to Bad Blocks."

Basically a block is bad if it contains anything but 0xFF when it
_arrives_ but obviously if the device has been subsequently used, many
other blocks could be in that state.

Likewise, the custom of setting the 5th byte of the OOB area to
something other than 0xFF gives you false positives on used DiskOnChip
devices, since we use the first six OOB bytes for DiskOnChip hardware
ECC. 

We _must_ retain the bad block table which was on the device when it
arrived from the factory. 

> So it seems the news isn't that bad.  If we always leave the oob
> data alone when the oob is marked bad for a block at the factory,
> then nftl_format will include it in the BadUnitTable it creates.
> Then preserving the factory-set bad block table would only be
> critical if it marks blocks bad that are not marked bad in their
> oob data.

It is critical, because it _does_ mark blocks bad which are not marked
bad in their oob-data, and because there are blocks which would
_apparently_ be marked bad in their oob-data which _aren't_ actually
bad.

I've fixed up the generic NAND code to allow the board code to register
a 'nand_block_bad()' function, and the DiskOnChip driver needs to hook
into that correctly, using the BBT as appropriate.

And I need to come up with a solution for the JFFS2-on-DiskOnChip
situation, where we don't have the NFTL Media Header block which would
contain a BBT. I'll probably end up with an INFTL header with its
'partition table' instead. 

-- 
dwmw2

RE: DOC filesystem questions

[Chuck Meade]

David Woodhouse wrote:

> We _must_ retain the bad block table which was on the device when it
> arrived from the factory. 

OK I see.  Would it be valuable then to have a Linux command-line
utility which captures the BBT before calling nftl_format, then 
is used to restore the BBT after calling nftl_format but before
nftl tries to mount any partitions (via 'insmod nftl' or whatever)?

Since nftl_format can be called with an offset, I believe the utility
would not necessarily put the BBT back at the exact device offset
from which it was read.  For instance, when reading the initial BBT
we might find the media headers in the first 2 blocks on the device,
and read/save the BBT from there.  However, if nftl_format is called
with an offset, we would need to write the saved BBT back to the
device at the blocks with media headers written beyond that offset
by nftl_format.

Does such a utility for the Linux command line sound like it would
preserve the BBT correctly?

Thanks,
Chuck

RE: DOC filesystem questions

[David Woodhouse]

On Thu, 2003-08-07 at 17:56, Chuck Meade wrote:
> David Woodhouse wrote:
> 
> > We _must_ retain the bad block table which was on the device when it
> > arrived from the factory. 
> 
> OK I see.  Would it be valuable then to have a Linux command-line
> utility which captures the BBT before calling nftl_format, then 
> is used to restore the BBT after calling nftl_format but before
> nftl tries to mount any partitions (via 'insmod nftl' or whatever)?

... or which writes it back as _part_ of the nftl_format process,
perhaps?

Yes, it would be extremely useful to do this.



> Since nftl_format can be called with an offset, I believe the utility
> would not necessarily put the BBT back at the exact device offset
> from which it was read.  For instance, when reading the initial BBT
> we might find the media headers in the first 2 blocks on the device,
> and read/save the BBT from there.  However, if nftl_format is called
> with an offset, we would need to write the saved BBT back to the
> device at the blocks with media headers written beyond that offset
> by nftl_format.
> 
> Does such a utility for the Linux command line sound like it would
> preserve the BBT correctly?
> 
> Thanks,
> Chuck

-- 
dwmw2

RE: DOC filesystem questions

[Chuck Meade]

David Woodhouse wrote:
> > > We _must_ retain the bad block table which was on the device when it
> > > arrived from the factory. 
> > 
> > OK I see.  Would it be valuable then to have a Linux command-line
> > utility which captures the BBT before calling nftl_format, then 
> > is used to restore the BBT after calling nftl_format but before
> > nftl tries to mount any partitions (via 'insmod nftl' or whatever)?
> 
> ... or which writes it back as _part_ of the nftl_format process,
> perhaps?
> 
> Yes, it would be extremely useful to do this.

I will work on this then and feed it back to the list for
evaluation when I have something that works.

Chuck

RE: DOC filesystem questions

[David Woodhouse]

On Fri, 2003-08-08 at 14:42, Chuck Meade wrote:
> I will work on this then and feed it back to the list for
> evaluation when I have something that works.

Cool, thanks. If you can make it eat the same file format as the DOS
DFORMAT tool does, that would be great.

-- 
dwmw2

RE: DOC filesystem questions

[Chuck Meade]

> On Thu, 2003-08-07 at 17:56, Chuck Meade wrote:
> > David Woodhouse wrote:
> > 
> > > We _must_ retain the bad block table which was on the device when it
> > > arrived from the factory. 
> > 
> > OK I see.  Would it be valuable then to have a Linux command-line
> > utility which captures the BBT before calling nftl_format, then 
> > is used to restore the BBT after calling nftl_format but before
> > nftl tries to mount any partitions (via 'insmod nftl' or whatever)?
> 
> ... or which writes it back as _part_ of the nftl_format process,
> perhaps?
> 
> Yes, it would be extremely useful to do this.

I have been working on the utilities to do this.  It is working on my
DOC Millennium device that is soldered to the target board.  I now
have a requirement to do the same with a DIMM DOC device.
The device has the following markings:
  MD-2220-D16-V3
  DiskOnChip B-1.21
  DIMM  (serial number follows)

This device is identified as a DOC 2000 at bootup, while my other one is
identified as a DOC Millennium.

I am starting out saving and restoring the bad block table from the command
line using dd.  Since the bad block table is to be saved before the 
nftl_format, and restored afterwards, I thought it would be handy to do
it with a shell script that uses dd.

This approach is working on the DOC Millennium device.  It is not working,
however, on my DOC 2000 device.  I get:

Read of bad sector table failed (err -5)

when I boot after restoring the bad block table via dd.
The -5 is an EIO being returned because it is unhappy with the ECC
that is read out.  I modified doc2000.c to print out the ECC codes that
are being written in doc_write_ecc, and read in doc_read_ecc.  They
do not match at all.

Is there a known problem with ECC reading/writing with DOC 2000
devices, or a subset of DOC 2000 devices?

Thanks very much,
Chuck Meade

RE: DOC filesystem questions

[Chuck Meade]

> > On Thu, 2003-08-07 at 17:56, Chuck Meade wrote:
> > > David Woodhouse wrote:
> > > 
> > > > We _must_ retain the bad block table which was on the device when it
> > > > arrived from the factory. 
> > > 
> > > OK I see.  Would it be valuable then to have a Linux command-line
> > > utility which captures the BBT before calling nftl_format, then 
> > > is used to restore the BBT after calling nftl_format but before
> > > nftl tries to mount any partitions (via 'insmod nftl' or whatever)?
> > 
> > ... or which writes it back as _part_ of the nftl_format process,
> > perhaps?
> > 
> > Yes, it would be extremely useful to do this.
> 
> I have been working on the utilities to do this.  It is working on my
> DOC Millennium device that is soldered to the target board.  I now
> have a requirement to do the same with a DIMM DOC device.
> The device has the following markings:
>   MD-2220-D16-V3
>   DiskOnChip B-1.21
>   DIMM  (serial number follows)
> 
> This device is identified as a DOC 2000 at bootup, while my other one is
> identified as a DOC Millennium.
> 
> I am starting out saving and restoring the bad block table from the command
> line using dd.  Since the bad block table is to be saved before the 
> nftl_format, and restored afterwards, I thought it would be handy to do
> it with a shell script that uses dd.
> 
> This approach is working on the DOC Millennium device.  It is not working,
> however, on my DOC 2000 device.  I get:
> 
> Read of bad sector table failed (err -5)
> 
> when I boot after restoring the bad block table via dd.
> The -5 is an EIO being returned because it is unhappy with the ECC
> that is read out.  I modified doc2000.c to print out the ECC codes that
> are being written in doc_write_ecc, and read in doc_read_ecc.  They
> do not match at all.
> 
> Is there a known problem with ECC reading/writing with DOC 2000
> devices, or a subset of DOC 2000 devices?

Looks like this is a case of "I just didn't look at it long enough".
The dd operation (of course) just plants a new ECC in the OOB area, 
which already has the ECC from the nftl_format operation.  So you 
get a bogus ECC combination that is a bitwise and of the 2 ECCs.
Any zero bit in either ECC is zero in the ECC in the OOB area 
after doing the dd.  Of course, since it is writing flash.

So what I will do is read out the media headers, replacing the bad
block table in my local copy, then erase the sector it is in and
write back the whole sector, with the media header and the (now
valid) bad block table.

Chuck