Latest jffs2 broken?

Latest jffs2 broken?

[Peter Grayson]

I am trying to use the latest mtd from cvs. There appears to be a
problem with jffs2 -- I'm surprised this has not tripped up anyone else
yet. I am seeing a kernel oops when I do the following procedure: erase
an mtd partition, mount the partition as jffs2, touch a file, unmount,
and remount.

This is fully reproducible for me. I am running the latest mtd cvs (as
of 8/23/2005) on a 2.6.10 kernel. I am using a 32MB partition for this
test, but it is not obvious that the partition size matters. My nand
part has a 2048-byte page size and 128 kB eraseblock size.

I have included the output showing the problem. Debug level is set to 1.

Has anyone else run into this?

Pete

---- Debug output below ----

# flash_eraseall -jq /dev/mtd3
# mount /mnt/mtdb3
# touch /mnt/mtdb3/foo
jffs2: No clean, dirty _or_ erasable blocks to GC from! Where are they
all?
jffs2: Couldn't find erase block to garbage collect!
# umount /mnt/mtdb3
# mount /mnt/mtdb3
kernel BUG in read_unknown at fs/jffs2/readinode.c:353!
Oops: Exception in kernel mode, sig: 5 [#1]
TASK = c0601190[74] 'jffs2_gcd_mtd3' THREAD: c044c000
Last syscall: -1
NIP [c00ad818] jffs2_do_read_inode_internal+0x804/0x1220
LR [c00ad1f4] jffs2_do_read_inode_internal+0x1e0/0x1220
Call trace:
 [c00ae2a4] jffs2_do_crccheck_inode+0x70/0xcc
 [c00b2de4] jffs2_garbage_collect_pass+0x284/0xa80
 [c00b4fc8] jffs2_garbage_collect_thread+0x1a0/0x1f4
 [c0004f6c] kernel_thread+0x44/0x60

Re: Latest jffs2 broken?

[Ferenc Havasi]

Hi Pete,

Peter Grayson wrote:

>This is fully reproducible for me. I am running the latest mtd cvs (as
>of 8/23/2005) on a 2.6.10 kernel. I am using a 32MB partition for this
>test, but it is not obvious that the partition size matters. My nand
>part has a 2048-byte page size and 128 kB eraseblock size.
>
>---- Debug output below ----
>
># flash_eraseall -jq /dev/mtd3
># mount /mnt/mtdb3
># touch /mnt/mtdb3/foo
>jffs2: No clean, dirty _or_ erasable blocks to GC from! Where are they
>all?
>jffs2: Couldn't find erase block to garbage collect!
># umount /mnt/mtdb3
># mount /mnt/mtdb3
>kernel BUG in read_unknown at fs/jffs2/readinode.c:353!
>Oops: Exception in kernel mode, sig: 5 [#1]
>TASK = c0601190[74] 'jffs2_gcd_mtd3' THREAD: c044c000
>Last syscall: -1
>NIP [c00ad818] jffs2_do_read_inode_internal+0x804/0x1220
>LR [c00ad1f4] jffs2_do_read_inode_internal+0x1e0/0x1220
>Call trace:
> [c00ae2a4] jffs2_do_crccheck_inode+0x70/0xcc
> [c00b2de4] jffs2_garbage_collect_pass+0x284/0xa80
> [c00b4fc8] jffs2_garbage_collect_thread+0x1a0/0x1f4
> [c0004f6c] kernel_thread+0x44/0x60
>  
>
Is this problem still present at you?

Something very strange happened here. All of the relevant lists are
empty, there is an unknow (?) node that is REF_UNCHECKED.

Did you try which earlier snapshot works correctly? It can be a good
starting point where the problem is. 2005-07-30? 2005-08-01? 2005-08-05?

You also may attach some more from the log...

Bye,
Ferenc

Re: Latest jffs2 broken?

[Peter Grayson]

On Fri, 2005-08-26 at 14:24 +0200, Ferenc Havasi wrote:
> Is this problem still present at you?

Yes.

> Something very strange happened here. All of the relevant lists are
> empty, there is an unknow (?) node that is REF_UNCHECKED.
> 
> Did you try which earlier snapshot works correctly? It can be a good
> starting point where the problem is. 2005-07-30? 2005-08-01? 2005-08-05?

I have not yet had a chance to figure out when this broke. If I had to
guess, I would say that the problem has something to do with the new
frag tree stuff. I was hoping that Artem might just know what is
happening.

I will try to find some time next week to do this search to figure out
when the breakage occurred.

> You also may attach some more from the log...

There was no more to the log. At debug level 2, there are lots of
messages like this at mount time:

  jffs_check_nand_cleanmarker returned 0
  jffs2_check_oob_empty returned 0

But those are not very interesting. I did omit some of the CPU-specific
stuff from the kernel oops printout. Here is the complete trace again:

# flash_eraseall -jq /dev/mtd3
# mount /mnt/mtdb3/
# touch /mnt/mtdb3/foo
# jffs2: No clean, dirty _or_ erasable blocks to GC from! Where are they
all?
jffs2: Couldn't find erase block to garbage collect!

# umount /mnt/mtdb3
# mount /mnt/mtdb3
# kernel BUG in read_unknown at fs/jffs2/readinode.c:353!
Oops: Exception in kernel mode, sig: 5 [#1]
PREEMPT
NIP: C00ADDA8 LR: C00AD804 SP: C0633E10 REGS: c0633d60 TRAP: 0700    Not
tainted
MSR: 00029030 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c03303f0[83] 'jffs2_gcd_mtd3' THREAD: c0632000
Last syscall: -1
GPR00: 00000001 C0633E10 C03303F0 00000000 0003FF80 0000000B 00000000
00000800
GPR08: 00000800 00000000 00000000 00000000 24008048 00000000 00000000
00000000
GPR16: C0633E20 C0633E24 00000000 00000000 C01D1E0C C0452000 00000000
00000000
GPR24: C0633E90 C0615400 C0656434 C01D7000 C01D1E00 C0632000 00000000
C0452000
NIP [c00adda8] jffs2_do_read_inode_internal+0x71c/0x1084
LR [c00ad804] jffs2_do_read_inode_internal+0x178/0x1084
Call trace:
 [c00ae780] jffs2_do_crccheck_inode+0x70/0xcc
 [c00b32f0] jffs2_garbage_collect_pass+0x27c/0xa78
 [c00b5558] jffs2_garbage_collect_thread+0x1a0/0x1f4
 [c0004f6c] kernel_thread+0x44/0x60

Pete

Re: Latest jffs2 broken?

[Peter Grayson]

[-- Attachment #1: Type: text/plain, Size: 1050 bytes --]

> Did you try which earlier snapshot works correctly? It can be a good
> starting point where the problem is. 2005-07-30? 2005-08-01? 2005-08-05?

This problem does not occur with 2005-08-01, but does with 2005-08-02.
There were some significant changes to the frag tree code that went in
between these two snapshots.

It appears that the offending code is in readinode.c. There is code in
jffs2_get_inode_nodes(), read_dnode(), and read_more() that attempts to
optimize the NAND (writebuffered) cases to avoid redundant reads of
flash. When I removed these optimizations, the problem goes away and
everything is quite happy. It was not obvious to me what is wrong with
this code, but honestly, I'm too tired to figure it out now.

Regardless, I have attached the patch that makes latest mtd cvs work for
me. I do not have high hopes that the powers that be will want to apply
this patch, but I wanted to put it out there in case someone else is
having the same trouble. Maybe someone more familiar with this code
could take another look too.

Pete


[-- Attachment #2: jffs2-simpler-readinode.patch --]
[-- Type: text/x-patch, Size: 6398 bytes --]

[JFFS2] Remove tricky writebuffered optimizations

This patch removes some read optimizations from readinode.c. These
optimizations cause serious problems on at least one NAND flash device.

[From: Peter Grayson <pgrayson@realmsys.com>]
diff -Narup mtd/fs/jffs2/readinode.c mtd-readinode-fix/fs/jffs2/readinode.c
--- mtd/fs/jffs2/readinode.c	2005-08-30 15:47:52.000000000 -0600
+++ mtd-readinode-fix/fs/jffs2/readinode.c	2005-08-31 22:51:51.456533976 -0600
@@ -230,61 +230,7 @@ static inline int read_dnode(struct jffs
 			goto free_out;
 		}
 
-		if (jffs2_is_writebuffered(c) && csize != 0) {
-			/* At this point we are supposed to check the data CRC
-			 * of our unchecked node. But thus far, we do not
-			 * know whether the node is valid or obsolete. To
-			 * figure this out, we need to walk all the nodes of
-			 * the inode and build the inode fragtree. We don't
-			 * want to spend time checking data of nodes which may
-			 * later be found to be obsolete. So we put off the full
-			 * data CRC checking until we have read all the inode
-			 * nodes and have started building the fragtree.
-			 *
-			 * The fragtree is being built starting with nodes
-			 * having the highest version number, so we'll be able
-			 * to detect whether a node is valid (i.e., it is not
-			 * overlapped by a node with higher version) or not.
-			 * And we'll be able to check only those nodes, which
-			 * are not obsolete.
-			 *
-			 * Of course, this optimization only makes sense in case
-			 * of NAND flashes (or other flashes whith
-			 * !jffs2_can_mark_obsolete()), since on NOR flashes
-			 * nodes are marked obsolete physically.
-			 *
-			 * Since NAND flashes (or other flashes with
-			 * jffs2_is_writebuffered(c)) are anyway read by
-			 * fractions of c->wbuf_pagesize, and we have just read
-			 * the node header, it is likely that the starting part
-			 * of the node data is also read when we read the
-			 * header. So we don't mind to check the CRC of the
-			 * starting part of the data of the node now, and check
-			 * the second part later (in jffs2_check_node_data()).
-			 * Of course, we will not need to re-read and re-check
-			 * the NAND page which we have just read. This is why we
-			 * read the whole NAND page at jffs2_get_inode_nodes(),
-			 * while we needed only the node header.
-			 */
-			unsigned char *buf;
-
-			/* 'buf' will point to the start of data */
-			buf = (unsigned char *)rd + sizeof(*rd);
-			/* len will be the read data length */
-			len = min_t(uint32_t, rdlen - sizeof(*rd), csize);
-			tn->partial_crc = crc32(0, buf, len);
-
-			JFFS2_DBG_READINODE("Calculates CRC (%#08x) for %d bytes, csize %d\n", tn->partial_crc, len, csize);
-
-			/* If we actually calculated the whole data CRC
-			 * and it is wrong, drop the node. */
-			if (len >= csize && unlikely(tn->partial_crc != je32_to_cpu(rd->data_crc))) {
-				JFFS2_NOTICE("wrong data CRC in data node at 0x%08x: read %#08x, calculated %#08x.\n",
-					ref_offset(ref), tn->partial_crc, je32_to_cpu(rd->data_crc));
-				goto free_out;
-			}
-
-		} else if (csize == 0) {
+		if (csize == 0) {
 			/*
 			 * We checked the header CRC. If the node has no data, adjust
 			 * the space accounting now. For other nodes this will be done
@@ -400,30 +346,18 @@ static inline int read_unknown(struct jf
 static int read_more(struct jffs2_sb_info *c, struct jffs2_raw_node_ref *ref,
 		     int right_size, int *rdlen, unsigned char *buf, unsigned char *bufstart)
 {
-	int right_len, err, len;
+	int err, len;
 	size_t retlen;
 	uint32_t offs;
 
-	if (jffs2_is_writebuffered(c)) {
-		right_len = c->wbuf_pagesize - (bufstart - buf);
-		if (right_size + (int)(bufstart - buf) > c->wbuf_pagesize)
-			right_len += c->wbuf_pagesize;
-	} else
-		right_len = right_size;
-
-	if (*rdlen == right_len)
+	if (*rdlen == right_size)
 		return 0;
 
 	/* We need to read more data */
 	offs = ref_offset(ref) + *rdlen;
-	if (jffs2_is_writebuffered(c)) {
-		bufstart = buf + c->wbuf_pagesize;
-		len = c->wbuf_pagesize;
-	} else {
-		bufstart = buf + *rdlen;
-		len = right_size - *rdlen;
-	}
-	
+	bufstart = buf + *rdlen;
+	len = right_size - *rdlen;
+
 	JFFS2_DBG_READINODE("read more %d bytes\n", len);
 
 	err = jffs2_flash_read(c, offs, len, &retlen, bufstart);
@@ -439,7 +373,7 @@ static int read_more(struct jffs2_sb_inf
 		return -EIO;
 	}
 
-	*rdlen = right_len;
+	*rdlen = right_size;
 
 	return 0;
 }
@@ -463,24 +397,11 @@ static int jffs2_get_inode_nodes(struct 
 	
 	JFFS2_DBG_READINODE("ino #%u\n", f->inocache->ino);
 
-	if (jffs2_is_writebuffered(c)) {
-		/*
-		 * If we have the write buffer, we assume the minimal I/O unit
-		 * is c->wbuf_pagesize. We implement some optimizations which in
-		 * this case and we need a temporary buffer of size =
-		 * 2*c->wbuf_pagesize bytes (see comments in read_dnode()).
-		 * Basically, we want to read not only the node header, but the
-		 * whole wbuf (NAND page in case of NAND) or 2, if the node
-		 * header overlaps the border between the 2 wbufs.
-		 */
-		len = 2*c->wbuf_pagesize;
-	} else {
-		/*
-		 * When there is no write buffer, the size of the temporary
-		 * buffer is the size of the larges node header.
-		 */
-		len = sizeof(union jffs2_node_union);
-	}
+	/*
+	 * When there is no write buffer, the size of the temporary
+	 * buffer is the size of the larges node header.
+	 */
+	len = sizeof(union jffs2_node_union);
 
 	/* FIXME: in case of NOR and available ->point() this
 	 * needs to be fixed. */
@@ -513,23 +434,8 @@ static int jffs2_get_inode_nodes(struct 
 		 * to minimize the amount of flash IO we assume the node has
 		 * size = JFFS2_MIN_NODE_HEADER.
 		 */
-		if (jffs2_is_writebuffered(c)) {
-			/* 
-			 * We treat 'buf' as 2 adjacent wbufs. We want to
-			 * adjust bufstart such as it points to the
-			 * beginning of the node within this wbuf.
-			 */
-			bufstart = buf + (ref_offset(ref) % c->wbuf_pagesize);
-			/* We will read either one wbuf or 2 wbufs. */
-			len = c->wbuf_pagesize - (bufstart - buf);
-			if (JFFS2_MIN_NODE_HEADER + (int)(bufstart - buf) > c->wbuf_pagesize) {
-				/* The header spans the border of the first wbuf */
-				len += c->wbuf_pagesize;
-			}
-		} else {
-			bufstart = buf;
-			len = JFFS2_MIN_NODE_HEADER;
-		}
+		bufstart = buf;
+		len = JFFS2_MIN_NODE_HEADER;
 
 		JFFS2_DBG_READINODE("read %d bytes at %#08x(%d).\n", len, ref_offset(ref), ref_flags(ref));
 

Re: Latest jffs2 broken?

[Artem B. Bityuckiy]

On Wed, 2005-08-31 at 23:02 -0600, Peter Grayson wrote:
> > Did you try which earlier snapshot works correctly? It can be a good
> > starting point where the problem is. 2005-07-30? 2005-08-01? 2005-08-05?
> 
> This problem does not occur with 2005-08-01, but does with 2005-08-02.
> There were some significant changes to the frag tree code that went in
> between these two snapshots.
> 
> It appears that the offending code is in readinode.c. There is code in
> jffs2_get_inode_nodes(), read_dnode(), and read_more() that attempts to
> optimize the NAND (writebuffered) cases to avoid redundant reads of
> flash. When I removed these optimizations, the problem goes away and
> everything is quite happy. It was not obvious to me what is wrong with
> this code, but honestly, I'm too tired to figure it out now.
> 
> Regardless, I have attached the patch that makes latest mtd cvs work for
> me. I do not have high hopes that the powers that be will want to apply
> this patch, but I wanted to put it out there in case someone else is
> having the same trouble. Maybe someone more familiar with this code
> could take another look too.
> 
I have had my vacation, I will glance what is going on.

Re: Latest jffs2 broken?

[Peter Grayson]

On Tue, 2005-09-06 at 13:00 +0400, Artem B. Bityuckiy wrote:
> I have had my vacation, I will glance what is going on.

Actually, I have just found that an error in the nand driver I am using
is the cause of this problem. The problem only happens with the latest
jffs2 code, but I now know it is not the jffs2 code. In fact, I am now
seeing the performance benefits of the new frag tree stuff and I am
really impressed. Good job!

Pete