jffs2 Oops on mount

jffs2 Oops on mount

[Ladislav Michl]

Hi,

I'm experiencind following Oops while mounting jffs2 NAND partiton:

Checked all inodes but still 0x13364c bytes of unchecked space?
Unable to handle kernel NULL pointer dereference at virtual address 00000000
pgd = c0004000
[00000000] *pgd=00000000
Internal error: Oops: 817 [#1]
Modules linked in: nsswitch-04 nsquade1-03 nsblank-02 nsblank-01 nscpu-00 nsdetect nssysutil mmc_block mmc_core
CPU: 0
PC is at jffs2_garbage_collect_pass+0x1e8/0x53c
LR is at 0x1
pc : [<c00d0e88>]    lr : [<00000001>]    Not tainted
sp : c3e25ef0  ip : 60000093  fp : c3e25f58
r10: c3e380e8  r9 : 00000000  r8 : c3e38000
r7 : 00000000  r6 : c3e38000  r5 : c3e3802c  r4 : c3ec98a8
r3 : c01d7940  r2 : 00000000  r1 : 00000000  r0 : 00000046
Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  Segment user
Control: 317F  Table: 1342C000  DAC: 00000015
Process jffs2_gcd_mtd5 (pid: 138, stack limit = 0xc3e24194)
Stack: (0xc3e25ef0 to 0xc3e26000)
5ee0:                                     c3e25f0c c3e25f00 c00428d0 c004281c 
5f00: c3e25f38 c3e25f10 c0042f7c c00428c4 20000013 c3db8040 c3db8234 c3e25f5c 
5f20: 00000000 00000000 c3e25f44 c3e25f48 20000013 c3e24000 c3e38000 00000000 
5f40: 00000000 00000000 00000000 c3e25ff4 c3e25f5c c00d3d80 c00d0cb0 00000001 
5f60: 00000000 00000080 00000000 00000000 00000000 c34e1f4c c34e1f40 c003aefc 
5f80: c003aa70 40172c60 c3e25fa4 c3e25f98 c003af2c c003ae90 00000000 00000000 
5fa0: 00000000 c3e25fb0 c001dd84 c0034758 00000000 c3e38000 c00d3c9c c003aa60 
5fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 
5fe0: 00000000 00000000 00000000 c3e25ff8 c003aa60 c00d3cac 00000000 00000000 
Backtrace: 
[<c00d0ca0>] (jffs2_garbage_collect_pass+0x0/0x53c) from [<c00d3d80>] (jffs2_garbage_collect_thread+0xe4/0x120)
[<c00d3c9c>] (jffs2_garbage_collect_thread+0x0/0x120) from [<c003aa60>] (do_exit+0x0/0x3d0)
 r6 = 00000000  r5 = 00000000  r4 = 00000000 
Code: e59f0348 ebfd9ea4 eaffffec ebfd9ea2 (e5877000) 
 <0>Kernel panic - not syncing: Fatal exception

Kernel version is 2.6.15 with CVS jffs2 from 10. Mar 2006 and omap2 patch
applied (http://www.muru.com/linux/omap/patches/patch-2.6.15-omap2.bz2).

You can find jffs2 image and kernel objdump here:
ftp://ftp.linux-mips.org/pub/linux/mips/people/ladis/jffs2_oops/

I'm ready to do any testing you ask for.

Thanks a lot,
	ladis

Re: jffs2 Oops on mount

[Ladislav Michl]

On Wed, Mar 22, 2006 at 12:54:09PM +0100, Ladislav Michl wrote:
> Checked all inodes but still 0x13364c bytes of unchecked space?
> Unable to handle kernel NULL pointer dereference at virtual address 00000000
Well, this is harmless (caused by BUG() defined as (*(int *)0 = 0))

With more debugs enabled it ends like this:
Skipping ino #228 already checked
Skipping ino #229 already checked
Skipping ino #270 already checked
Skipping check of ino #273 with nlink zero
[lots of skipping check of ino #xxx with nlink zero]
Skipping check of ino #800 with nlink zero
Skipping check of ino #801 with nlink zero
Skipping ino #802 already checked
Skipping check of ino #803 with nlink zero
Checked all inodes but still 0x13364c bytes of unchecked space?

Where checked_ino is 804 and highest_ino 803. ino 802 is already checked
because userspace requested read meanwhile.

Ideas?

Thanks,
	ladis

jffs2 BUG() on mount

[Ladislav Michl]

On Wed, Mar 22, 2006 at 08:49:39PM +0100, Ladislav Michl wrote:
> With more debugs enabled it ends like this:
> Skipping ino #228 already checked
> Skipping ino #229 already checked
> Skipping ino #270 already checked
> Skipping check of ino #273 with nlink zero
> [lots of skipping check of ino #xxx with nlink zero]
> Skipping check of ino #800 with nlink zero
> Skipping check of ino #801 with nlink zero
> Skipping ino #802 already checked
> Skipping check of ino #803 with nlink zero
> Checked all inodes but still 0x13364c bytes of unchecked space?
> 
> Where checked_ino is 804 and highest_ino 803. ino 802 is already checked
> because userspace requested read meanwhile.

jffs2_scan_inode_node adds also inodes with zero nlink to
unchecked_space, but jffs2_garbage_collect_pass skips them. That
probably leads to above problem. After applying following patch
BUG() is no longer triggered.

Index: fs/jffs2/gc.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/gc.c,v
retrieving revision 1.159
diff -u -r1.159 gc.c
--- fs/jffs2/gc.c	18 Nov 2005 07:27:45 -0000	1.159
+++ fs/jffs2/gc.c	23 Mar 2006 17:39:00 -0000
@@ -211,12 +211,6 @@
 			continue;
 		}
 
-		if (!ic->nlink) {
-			D1(printk(KERN_DEBUG "Skipping check of ino #%d with nlink zero\n",
-				  ic->ino));
-			spin_unlock(&c->inocache_lock);
-			continue;
-		}
 		switch(ic->state) {
 		case INO_STATE_CHECKEDABSENT:
 		case INO_STATE_PRESENT:

I'm not sure what is correct solution. With this change there are lots of
JFFS2 warning: (138) jffs2_get_inode_nodes: Eep. No valid nodes for ino #465.
JFFS2 warning: (138) jffs2_do_read_inode_internal: no data nodes found for ino #465
Returned error for crccheck of ino #465. Expect badness...
warnings. Jffs2 summary feature is disabled. I'd be very gratefull for ideas.

	ladis

jffs2 BUG() on mount

[Ladislav Michl]

(Resend in hope it will not be eaten by spam filter)

On Wed, Mar 22, 2006 at 08:49:39PM +0100, Ladislav Michl wrote:
> With more debugs enabled it ends like this:
> Skipping ino #228 already checked
> Skipping ino #229 already checked
> Skipping ino #270 already checked
> Skipping check of ino #273 with nlink zero
> [lots of skipping check of ino #xxx with nlink zero]
> Skipping check of ino #800 with nlink zero
> Skipping check of ino #801 with nlink zero
> Skipping ino #802 already checked
> Skipping check of ino #803 with nlink zero
> Checked all inodes but still 0x13364c bytes of unchecked space?
> 
> Where checked_ino is 804 and highest_ino 803. ino 802 is already checked
> because userspace requested read meanwhile.

jffs2_scan_inode_node adds also inodes with zero nlink to
unchecked_space, but jffs2_garbage_collect_pass skips them. That
probably leads to above problem. After applying following patch
BUG() is no longer triggered.

Index: fs/jffs2/gc.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/gc.c,v
retrieving revision 1.159
diff -u -r1.159 gc.c
--- fs/jffs2/gc.c	18 Nov 2005 07:27:45 -0000	1.159
+++ fs/jffs2/gc.c	23 Mar 2006 17:39:00 -0000
@@ -211,12 +211,6 @@
 			continue;
 		}
 
-		if (!ic->nlink) {
-			D1(printk(KERN_DEBUG "Skipping check of ino #%d with nlink zero\n",
-				  ic->ino));
-			spin_unlock(&c->inocache_lock);
-			continue;
-		}
 		switch(ic->state) {
 		case INO_STATE_CHECKEDABSENT:
 		case INO_STATE_PRESENT:

I'm not sure what is correct solution. With this change there are lots of
JFFS2 warning: (138) jffs2_get_inode_nodes: Eep. No valid nodes for ino #465.
JFFS2 warning: (138) jffs2_do_read_inode_internal: no data nodes found for ino #465
Returned error for crccheck of ino #465. Expect badness...
warnings. Jffs2 summary feature is disabled. I'd be very gratefull for ideas.

	ladis

Re: jffs2 BUG() on mount

[Artem B. Bityutskiy]

Ladislav Michl wrote:
> jffs2_scan_inode_node adds also inodes with zero nlink to
> unchecked_space, but jffs2_garbage_collect_pass skips them. That
> probably leads to above problem. After applying following patch
> BUG() is no longer triggered.
> 

<snip>

> I'm not sure what is correct solution. With this change there are lots of
> JFFS2 warning: (138) jffs2_get_inode_nodes: Eep. No valid nodes for ino #465.
> JFFS2 warning: (138) jffs2_do_read_inode_internal: no data nodes found for ino #465
> Returned error for crccheck of ino #465. Expect badness...
> warnings. Jffs2 summary feature is disabled. I'd be very gratefull for ideas.

Actually, there is a jffs2_build_filesystem() function which walks all 
inodes with nlink == 0 and pretends to dispense with them. I believe 
that function is the right place to fix. I ganced at it, and it appeared 
to be thet it calls jffs2_mark_node_obsolete() for all nodes of this 
inode. jffs2_mark_node_obsolete() is a huge and fearsom monster-function 
which may do something wrong.

-- 
Best Regards,
Artem B. Bityutskiy,
St.-Petersburg, Russia.

Re: jffs2 BUG() on mount

[Ladislav Michl]

On Tue, Apr 04, 2006 at 11:34:30AM +0400, Artem B. Bityutskiy wrote:
> Actually, there is a jffs2_build_filesystem() function which walks all 
> inodes with nlink == 0 and pretends to dispense with them. I believe 
> that function is the right place to fix. I ganced at it, and it appeared 
> to be thet it calls jffs2_mark_node_obsolete() for all nodes of this 
> inode. jffs2_mark_node_obsolete() is a huge and fearsom monster-function 
> which may do something wrong.

Problem appeared to be GC thread waiting for inode being read. On next run
this inode was incorrectly skipped before jffs2_do_read_inode_internal
marked it obsolete. Bug was fixed by decrementing c->checked_ino before
jffs2_garbage_collect_pass returns (for INO_STATE_READING) so on next GC
run it is checked again.

Many thanks to David Woodhouse and Artem B. Bityuckiy for help with
debugging.

Best regards,
	ladis