Re: jffs2 kernel dump with 2.6.22-rc7

Re: jffs2 kernel dump with 2.6.22-rc7

[shanevolpe]

I changed the debug level for JFFS2 in the kernel from 0 to 1
(CONFIG_JFFS2_FS_DEBUG=1) and now the kernel seems stable.  I also
verified that it is stable with a debug level of 2.

The actions leading up to the core dump do not cause any debug
information to be displayed so I do not believe it is the actual debug
code but maybe some race condition.

I verified the above statement by setting the debug level to 0 then
copying a large file (which causes the kernel to core dump and then
setting the debug level to 2 and repeating the same copy.  The second
copy with the debug level set to 2 did not cause the core dump and
there were NO JFFS2 debug messages displayed.  I repeated the above
events several times and in different orders, the results were the
same.
Regards,
Shane

On 7/9/07, shanevolpe@gmail.com <shanevolpe@gmail.com> wrote:
> I'm working on an embedded system which uses the PXA270 (xscale) processor with NOR flash0.  I just updated the kernel to 2.6.22-rc7 from 2.6.20.14 (however the jffs2 was first created with  2.6.19) and now two things are happening.
> 1.)  I 'm getting the following message during system boot:
>             jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at 0x00c4531c: 0x0044 instead
>             Further such events for this erase block will not be printed
>             Old JFFS2 bitmask found at 0x00c45720
>             you cannot use older JFFS2 file systems with newer kernels
> 2.) I'm getting a core dump when writing to the file system, I have appended the core dump to the end of this message.
>      Also it seems as if the reported blocks in issue 1, increases every time the core dump of issue 2 happens.
>
> This problem is very repeatable and I have recreated it on two systems now.
> Regards,
> Shane
>
> **********************CORE DUMP****************************
>
>
>
>
> <4>argh. node added in wrong place
>
> <4>argh. node added in wrong place
>
> <1>Unable to handle kernel NULL pointer dereference at virtual address  00000000
>
> <1>pgd = c32e8000
>
> <1> [00000000] *pgd=a3214031, *pte=00000000, *ppte=00000000
>
> < 4>Internal error: Oops: 817 [#1] PREEMPT
>
> <4 >Modules linked in: ucb1400_ts
>
> <4>CPU: 0    Not tainted  (2.6 .22-rc7 #16)
>
> <4>PC is at jffs2_link_node_ref+0x20/0x1d0
>
>  <4>LR is at jffs2_add_physical_node_ref+0xb4/0x178
>
> <4>pc : [<c01073d4> ]    lr : [<c01094e4>]    psr: 60000013
>
> <4>sp : c32c7b98  ip :  00000000  fp : c32c7bcc
>
> <4>r10: c06123c8  r9 : c2bf8478  r8 : c06123c8
>
> < 4>r7 : 00b7d082  r6 : c0630c00  r5 : c0596924  r4 : c0596924
>
> <4>r3 : 000009e0  r2 : 00b7d082  r1 : c0596924  r0 : c0630c00
>
> <4>Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  Segment user
>
> <4>Control: 0000397f  Table: a32e8000  DAC:  00000015
>
> <4>Process ipkg (pid: 13880, stack limit = 0xc32c6268 )
>
> <4>Stack: (0xc32c7b98 to 0xc32c8000)
>
> < 4>7b80:                                                       c32c7c34 c246d995
>
> <4>7ba0: c03f0bf8 00000002 c32c7c20 c32c6000 c0596924 c0630c00 00b7d082 000009e0
>
> <4>7bc0: c32c7bfc c32c7bd0 c01094e4 c01073c0 c06123c8 c246d000 00000000 c249e528
>
>  <4>7be0: 00b7d080 00000000 00b7d080 c0630c00 c32c7c64 c32c7c00 c010c334 c010943c
>
> <4>7c00: 00b7d080  00000000 c32c7c34 00000b18 000009dd c322d898 00000000 00000002
>
> <4>7c20: c249e528  00000044 c246d000 00000999 c24f1000 000009dd c0107b68 00000000
>
> <4>7c40:  00000010 00000000 00000000 00000000 c249e528 c2bf8478 c32c7cd4 c32c7c68
>
> < 4>7c60: c010cab8 c010bf98 00000999 00000003 00000000 00000099 00000000  00000000
>
> <4>7c80: 00000070 00000006 c24f1000 c0630c00 00000000  00000000 00000999 00001000
>
> <4>7ca0: 00002fc0 c246d000 00007000  00000000 c249e550 c249e528 c2bf84a8 c044be20
>
> <4>7cc0: 00000000 00000000 c32c7d24 c32c7cd8 c0106b70 c010c7c8  00007000 00001000
>
> <4>7ce0: c32c7cf4 00000007 00001000  c0630c00 00000000 00000000 c32c7d24 c02d1c20
>
> <4>7d00: 00001000 c044be20  00001000 c03ef678 00000000 c32c6000 c32c7dcc c32c7d28
>
> <4>7d20: c00635d4 c01069e8  00000001 c32c7ea0 00000001 00001000 c37bf660 c2bf8540
>
> <4>7d40: c02d1c20 c2bf84a8  00000000 be8626f0 c32c7f30 00000000 c004844c 00000001
>
> <4>7d60: 00000000  c044be20 c04d6820 c32c7da8 00000000 00007000 c32c7da4 c32c7d88
>
> <4>7d80: c003f220 c003ea00  00000799 1f9e8785 c2be801c c2bf84a8 c32c7dcc e52c7da8
>
> <4>7da0: c009b0a0 ffffffff c2bf84a8 00007000  00000000 00000000 00000000 00001000
>
> <4>7dc0: c32c7e54 c32c7dd0 c0063cec c0063154  00007000 00000000 c32c7ef0 00001000
>
> <4>7de0: 00000000  c32c7df0 c003f220 c32c7ef0 c32c7f30 c32c7ea0 c30a7848 c3483478
>
> <4>7e00: c37bf660 c2bf8540 00000000 00000001  00000799 1f05f105 00000799 1f05f105
>
> <4>7e20: 00000400  00001000 c32c7e94 c2bf8514 c32c7ea0 c2bf84a8 c32c7f30 00007000
>
> <4>7e40: 00000000  00000001 c32c7e94 c32c7e58 c0063dd0 c0063820 00000001 c32c6000
>
> <4>7e60: c37bf660 c2bf8540 00000000  c32c7ea0 c37bf660 c32c7f78 00000000 c00230a8
>
> <4>7e80: c32c6000 00000000 c32c7f54 c32c7e98 c0084970 c0063d68  00007000 00000000
>
> <4>7ea0: c32c7ebc c32c7eb0 00000000  00000001 ffffffff c37bf660 00000000 00000000
>
> <4>7ec0: 00000000  00000000 c04d6820 00000817 00000000 00000000 c037dffc c04d6820
>
> <4>7ee0: c0050070 c32c7ee4 c32c7ee4 4014e000  00007000 00000000 c0022264 c0029d3c
>
> <4>7f00: c0075124 c0072c88 00100073  00001000 00000000 0004001f 00000000 ffffffff
>
> <4>7f20: 000000c5 c00230a8  00000001 00000001 be8626f0 00001000 c37bf660 c37bf660
>
> <4>7f40: be8626f0 c32c7f78 c32c7f74 c32c7f58 c00851b4 c00848cc c00539c8 c37bf660
>
> <4>7f60: fffffff7 00007000 c32c7fa4 c32c7f78 c00856c8 c008510c 00007000 00000000
>
> <4>7f80: 00000000 00000000 00042040 00001000  be8626f0 00000004 00000000 c32c7fa8
>
> <4>7fa0: c0022f00 c0085690 00042040  00001000 00000006 be8626f0 00001000 00000000
>
> <4>7fc0:  00042040 00001000 be8626f0 00000004 00001000 00000000 4014e000 00042040
>
> <4>7fe0: 00000004 be861668 400a49f4 400e801c 60000010 00000006  00000000 00000000
>
> <4>Backtrace:
>
> <4> [<c01073b4>] (jffs2_link_node_ref+0x0/0x1d0) from [<c01094e4>]  (jffs2_add_physical_node_ref+0xb4/0x178)
>
> <4> r8:000009e0 r7:00b7d082 r6:c0630c00 r5:c0596924 r4:c32c6000
>
> <4>[<c0109430>] (jffs2_add_physical_node_ref+0x0/0x178)  from [<c010c334>] (jffs2_write_dnode+0x3a8/0x450)
>
> < 4>[<c010bf8c>] (jffs2_write_dnode+0x0/0x450) from [<c010cab8> ] (jffs2_write_inode_range+0x2fc/0x484)
>
> <4>[<c010c7bc> ] (jffs2_write_inode_range+0x0/0x484) from [<c0106b70>] (jffs2_commit_write+0x194/0x2dc )
>
> <4>[<c01069dc>] (jffs2_commit_write+0x0/0x2dc ) from [<c00635d4>] (generic_file_buffered_write+0x48c/0x6cc)
>
>  <4>[<c0063148>] (generic_file_buffered_write+0x0/0x6cc) from [ <c0063cec>] (__generic_file_aio_write_nolock+0x4d8/0x548)
>
> <4> [<c0063814>] (__generic_file_aio_write_nolock+0x0/0x548) from [<c0063dd0>]  (generic_file_aio_write+0x74/0xe8)
>
> <4>[<c0063d5c>]  (generic_file_aio_write+0x0/0xe8) from [<c0084970>] (do_sync_write+0xb0/0x100)
>
> <4>[<c00848c0>] (do_sync_write+0x0/0x100) from  [<c00851b4>] (vfs_write+0xb4/0xdc)
>
> <4> r6:c32c7f78 r5:be8626f0 r4:c37bf660
>
> <4>[<c0085100>] (vfs_write+0x0/0xdc) from  [<c00856c8>] (sys_write+0x44/0x70)
>
> <4> r6: 00007000 r5:fffffff7 r4:c37bf660
>
> <4>[<c0085684>]  (sys_write+0x0/0x70) from [<c0022f00>] (ret_fast_syscall+0x0/0x2c)
>
> <4> r7:00000004 r6:be8626f0 r5:00001000 r4:00042040
>
> < 4>Code: e591c024 e1a04001 e59b8004 e35c0000 (058cc000)
>
> <6>note: ipkg [13880] exited with preempt_count 1
>
> <3>BUG: scheduling while atomic: ipkg/0x40000001/ 13880
>
> <3>BUG: scheduling while atomic: ipkg/0x40000001/13880
>



-- 
Registered Linux User: #293401

Re: jffs2 kernel dump with 2.6.22-rc7

[giulio fedel]

In jffs2_do_unlink() in fs/jffs2/write.c
the jffs2_complete_reservation(c) is called even if 
jffs2_reserve_space() is not called. This cause an unmatched 
up(&c->alloc_sem) so the locking mechanism does not work.
People using MTD_CAP_NORFLASH flash _and_ CONFIG_JFFS2_SUMMARY do not 
see the problem (see  os-linux.h for the definition of 
jffs2_can_mark_obsolete(c)).

I think this is your problem (it was the mine).

Regards.

     Giulio Fedel


--- fs/jffs2/write.c.orig       2007-08-17 19:01:04.000000000 +0200
+++ fs/jffs2/write.c    2007-08-17 19:00:41.000000000 +0200
@@ -549,6 +549,7 @@ int jffs2_do_unlink(struct jffs2_sb_info
                 /* File it. This will mark the old one obsolete. */
                 jffs2_add_fd_to_list(c, fd, &dir_f->dents);
                 up(&dir_f->sem);
+               jffs2_complete_reservation(c);
         } else {
                 struct jffs2_full_dirent **prev = &dir_f->dents;
                 uint32_t nhash = full_name_hash(name, namelen);
@@ -605,7 +606,6 @@ int jffs2_do_unlink(struct jffs2_sb_info
                 up(&dead_f->sem);
         }

-       jffs2_complete_reservation(c);

         return 0;
  }

Re: jffs2 kernel dump with 2.6.22-rc7

[Zac Medico]

giulio fedel wrote:
> In jffs2_do_unlink() in fs/jffs2/write.c
> the jffs2_complete_reservation(c) is called even if 
> jffs2_reserve_space() is not called. This cause an unmatched 
> up(&c->alloc_sem) so the locking mechanism does not work.
> People using MTD_CAP_NORFLASH flash _and_ CONFIG_JFFS2_SUMMARY do not 
> see the problem (see  os-linux.h for the definition of 
> jffs2_can_mark_obsolete(c)).
> 
> I think this is your problem (it was the mine).

I can confirm that your patch solves the problem for me. I've been
unable to use jffs2 since 2.6.22 because of this problem. Thanks for
the fix!

Zac

Re: jffs2 kernel dump with 2.6.22-rc7

[Josh Boyer]

On 8/17/07, giulio fedel <giulio.fedel@andorsystems.com> wrote:
> In jffs2_do_unlink() in fs/jffs2/write.c
> the jffs2_complete_reservation(c) is called even if
> jffs2_reserve_space() is not called. This cause an unmatched
> up(&c->alloc_sem) so the locking mechanism does not work.
> People using MTD_CAP_NORFLASH flash _and_ CONFIG_JFFS2_SUMMARY do not
> see the problem (see  os-linux.h for the definition of
> jffs2_can_mark_obsolete(c)).
>
> I think this is your problem (it was the mine).

Hm.  That would be a regression introduced by commit
a491486a2087ac3dfc00efb4f838c8d684afaf54 I think.

Your patch seems like the correct fix.  Could you make it a strip
level of 1 and add the appropriate Signed-off-by line?  If so, we
should probably push this to -stable for 2.6.22 and be sure to get it
into 2.6.23.

josh

RE: jffs2 kernel dump with 2.6.22-rc7

[Joakim Tjernlund]

> -----Original Message-----
> From: Josh Boyer [mailto:jwboyer@gmail.com] 
> Sent: den 18 augusti 2007 23:43
> To: giulio fedel; David Woodhouse; Joakim.Tjernlund@transmode.se
> Cc: linux-mtd@lists.infradead.org
> Subject: Re: jffs2 kernel dump with 2.6.22-rc7
> 
> On 8/17/07, giulio fedel <giulio.fedel@andorsystems.com> wrote:
> > In jffs2_do_unlink() in fs/jffs2/write.c
> > the jffs2_complete_reservation(c) is called even if
> > jffs2_reserve_space() is not called. This cause an unmatched
> > up(&c->alloc_sem) so the locking mechanism does not work.
> > People using MTD_CAP_NORFLASH flash _and_ 
> CONFIG_JFFS2_SUMMARY do not
> > see the problem (see  os-linux.h for the definition of
> > jffs2_can_mark_obsolete(c)).
> >
> > I think this is your problem (it was the mine).
> 
> Hm.  That would be a regression introduced by commit
> a491486a2087ac3dfc00efb4f838c8d684afaf54 I think.

Looks like it.

> 
> Your patch seems like the correct fix.  Could you make it a strip
> level of 1 and add the appropriate Signed-off-by line?  If so, we
> should probably push this to -stable for 2.6.22 and be sure to get it
> into 2.6.23.

Yes, can't be long until .23 gets released now, hopefully this 
correction will make it in time.

> 
> josh
> 
> 

Re: jffs2 kernel dump with 2.6.22-rc7

[David Woodhouse]

On Sat, 2007-08-18 at 16:42 -0500, Josh Boyer wrote:
> On 8/17/07, giulio fedel <giulio.fedel@andorsystems.com> wrote:
> > In jffs2_do_unlink() in fs/jffs2/write.c
> > the jffs2_complete_reservation(c) is called even if
> > jffs2_reserve_space() is not called. This cause an unmatched
> > up(&c->alloc_sem) so the locking mechanism does not work.
> > People using MTD_CAP_NORFLASH flash _and_ CONFIG_JFFS2_SUMMARY do not
> > see the problem (see  os-linux.h for the definition of
> > jffs2_can_mark_obsolete(c)).
> >
> > I think this is your problem (it was the mine).
> 
> Hm.  That would be a regression introduced by commit
> a491486a2087ac3dfc00efb4f838c8d684afaf54 I think.
> 
> Your patch seems like the correct fix. 

Hm, maybe. On the other hand, it does make it clear that we're violating
the "No writes to flash without alloc_sem" held assumption. That
assumption, when you're holding alloc_sem, means you're allowed to hold
on to a _valid_ node while not holding the erase_completion_lock
spinlock, although you're not allowed to hold onto an invalid one
(because it might disappear if the eraseblock in which it resides is
deleted). If valid nodes can become invalid, that's going to be...
fun :)

I think I'd prefer to make the can_mark_obsolete path also hold
alloc_sem while it's doing its thing.

Giulio, please could you verify that this patch also fixes the problem?

diff --git a/fs/jffs2/write.c b/fs/jffs2/write.c
index bc61859..664c164 100644
--- a/fs/jffs2/write.c
+++ b/fs/jffs2/write.c
@@ -566,6 +566,9 @@ int jffs2_do_unlink(struct jffs2_sb_info *c, struct jffs2_inode_info *dir_f,
 		struct jffs2_full_dirent **prev = &dir_f->dents;
 		uint32_t nhash = full_name_hash(name, namelen);
 
+		/* We don't actually want to reserve any space, but we do
+		   want to be holding the alloc_sem when we write to flash */
+		down(&c->alloc_sem);
 		down(&dir_f->sem);
 
 		while ((*prev) && (*prev)->nhash <= nhash) {


-- 
dwmw2

Re: jffs2 kernel dump with 2.6.22-rc7

[Zac Medico]

[-- Attachment #1: Type: text/plain, Size: 946 bytes --]

David Woodhouse wrote:
> I think I'd prefer to make the can_mark_obsolete path also hold
> alloc_sem while it's doing its thing.
> 
> Giulio, please could you verify that this patch also fixes the problem?
> 
> diff --git a/fs/jffs2/write.c b/fs/jffs2/write.c
> index bc61859..664c164 100644
> --- a/fs/jffs2/write.c
> +++ b/fs/jffs2/write.c
> @@ -566,6 +566,9 @@ int jffs2_do_unlink(struct jffs2_sb_info *c, struct jffs2_inode_info *dir_f,
>  		struct jffs2_full_dirent **prev = &dir_f->dents;
>  		uint32_t nhash = full_name_hash(name, namelen);
>  
> +		/* We don't actually want to reserve any space, but we do
> +		   want to be holding the alloc_sem when we write to flash */
> +		down(&c->alloc_sem);
>  		down(&dir_f->sem);
>  
>  		while ((*prev) && (*prev)->nhash <= nhash) {
> 
> 

I've tested this patch with a 2.6.23-rc3-git2 snapshot and it solves
the problem for me (see attachment for an example of what I was
experiencing).

Zac

[-- Attachment #2: bug.txt --]
[-- Type: text/plain, Size: 2376 bytes --]

Kernel BUG at ffffffff8823eedc [verbose debug info unavailable]
invalid opcode: 0000 [1] PREEMPT SMP
CPU 0
Modules linked in: squashfs loop jffs2 block2mtd mtdblock mtd_blkdevs mtdchar mtd 8139too e100 mii evdev psmouse cpufreq_ondemand freq_table ipv6 snd_pcm_oss snd_mixer_oss snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_timer snd soundcore snd_page_alloc nfsd exportfs lockd sunrpc ipt_REJECT xt_tcpudp ipt_LOG xt_limit ipt_recent nf_conntrack_ipv4 xt_state nf_conntrack nfnetlink iptable_filter ip_tables x_tables rtc usbcore ntfs dm_mod
Pid: 7379, comm: rsync Not tainted 2.6.23-rc2-0807-x86-64 #1
RIP: 0010:[<ffffffff8823eedc>]  [<ffffffff8823eedc>] :jffs2:jffs2_link_node_ref+0x1d/0x153
RSP: 0000:ffff810038547c38  EFLAGS: 00210246
RAX: 0000000000000000 RBX: ffffc200008cef40 RCX: ffff8100380bb5b0
RDX: 0000000001ea8dff RSI: ffffc200008cef40 RDI: ffff810037dba000
RBP: ffff810038547c68 R08: ffff8100380bb5b0 R09: 0000000000000044
R10: 0000000000000044 R11: 0000000000200296 R12: ffff810037dba000
R13: 0000000000000044 R14: ffff810037dba188 R15: ffff8100380bb5b0
FS:  0000000000000000(0000) GS:ffffffff80581000(0063) knlGS:00000000f7c748d0
CS:  0010 DS: 002b ES: 002b CR0: 000000008005003b
CR2: 00000000f69fa000 CR3: 000000003ed35000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process rsync (pid: 7379, threadinfo ffff810038546000, task ffff8100202608a0)
Stack:  0000000000000001 ffffffffffffff10 ffffffff80280bce ffffc200008cef40
 ffff810037dba000 0000000001ea8dff ffff810038547ca8 ffffffff88240540
 0000004438b4cca8 0000000001ea8dfc 0000000000000000 ffff810037dba000
Call Trace:
 [<ffffffff80280bce>] kmem_cache_alloc+0x49/0x6f
 [<ffffffff88240540>] :jffs2:jffs2_add_physical_node_ref+0x9e/0x129
 [<ffffffff88242e92>] :jffs2:jffs2_write_dnode+0x2d0/0x337
 [<ffffffff88247d06>] :jffs2:jffs2_do_setattr+0x385/0x52d
 [<ffffffff80239eee>] current_fs_time+0x22/0x29
 [<ffffffff88247f31>] :jffs2:jffs2_setattr+0xd/0xf
 [<ffffffff80298b22>] notify_change+0x160/0x30e
 [<ffffffff802831bf>] sys_fchmodat+0xac/0xd5
 [<ffffffff8029a1ba>] mntput_no_expire+0x20/0xa2
 [<ffffffff80224cba>] sys32_lstat64+0x29/0x34
 [<ffffffff802831fb>] sys_chmod+0x13/0x15
 [<ffffffff802245c2>] ia32_sysret+0x0/0xa

Re: jffs2 kernel dump with 2.6.22-rc7

[giulio fedel]

It's ok for me.

     Giulio


David Woodhouse wrote:
> On Sat, 2007-08-18 at 16:42 -0500, Josh Boyer wrote:
...
> 
> Hm, maybe. On the other hand, it does make it clear that we're violating
> the "No writes to flash without alloc_sem" held assumption. That
> assumption, when you're holding alloc_sem, means you're allowed to hold
> on to a _valid_ node while not holding the erase_completion_lock
> spinlock, although you're not allowed to hold onto an invalid one
> (because it might disappear if the eraseblock in which it resides is
> deleted). If valid nodes can become invalid, that's going to be...
> fun :)
> 
> I think I'd prefer to make the can_mark_obsolete path also hold
> alloc_sem while it's doing its thing.
> 
> Giulio, please could you verify that this patch also fixes the problem?
> 
> diff --git a/fs/jffs2/write.c b/fs/jffs2/write.c
> index bc61859..664c164 100644
> --- a/fs/jffs2/write.c
> +++ b/fs/jffs2/write.c
> @@ -566,6 +566,9 @@ int jffs2_do_unlink(struct jffs2_sb_info *c, struct jffs2_inode_info *dir_f,
>  		struct jffs2_full_dirent **prev = &dir_f->dents;
>  		uint32_t nhash = full_name_hash(name, namelen);
>  
> +		/* We don't actually want to reserve any space, but we do
> +		   want to be holding the alloc_sem when we write to flash */
> +		down(&c->alloc_sem);
>  		down(&dir_f->sem);
>  
>  		while ((*prev) && (*prev)->nhash <= nhash) {
> 
> 

jffs2 kernel dump with 2.6.22-rc7

[shanevolpe]

I'm working on an embedded system which uses the PXA270 (xscale)
processor with NOR flash0.  I just updated the kernel to 2.6.22-rc7
from 2.6.20.14 (however the jffs2 was first created with 2.6.19) and
now two things are happening.
1.)  I 'm getting the following message during system boot:
            jffs2_scan_eraseblock(): Magic bitmask 0x1985 not found at
0x00c4531c: 0x0044 instead
            Further such events for this erase block will not be printed
            Old JFFS2 bitmask found at 0x00c45720
            you cannot use older JFFS2 file systems with newer kernels
2.) I'm getting a core dump when writing to the file system, I have
appended the core dump to the end of this message.
     Also it seems as if the reported blocks in issue 1, increases
every time the core dump of issue 2 happens.

This problem is very repeatable and I have recreated it on two systems now.
Regards,
Shane

**********************CORE DUMP****************************


<4>argh. node added in wrong place
<4>argh. node added in wrong place
<1>Unable to handle kernel NULL pointer dereference at virtual address 00000000
<1>pgd = c32e8000
<1> [00000000] *pgd=a3214031, *pte=00000000, *ppte=00000000
<4>Internal error: Oops: 817 [#1] PREEMPT
<4 >Modules linked in: ucb1400_ts
<4>CPU: 0    Not tainted  (2.6 .22-rc7 #16)
<4>PC is at jffs2_link_node_ref+0x20/0x1d0
<4>LR is at jffs2_add_physical_node_ref+0xb4/0x178
<4>pc : [<c01073d4> ]    lr : [<c01094e4>]    psr: 60000013
<4>sp : c32c7b98  ip : 00000000  fp : c32c7bcc
<4>r10: c06123c8  r9 : c2bf8478  r8 : c06123c8
<4>r7 : 00b7d082  r6 : c0630c00  r5 : c0596924  r4 : c0596924
<4>r3 : 000009e0  r2 : 00b7d082  r1 : c0596924  r0 : c0630c00
<4>Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  Segment user
<4>Control: 0000397f  Table: a32e8000  DAC: 00000015
<4>Process ipkg (pid: 13880, stack limit = 0xc32c6268 )
<4>Stack: (0xc32c7b98 to 0xc32c8000)
<4>7b80:                                                       c32c7c34 c246d995
<4>7ba0: c03f0bf8 00000002 c32c7c20 c32c6000 c0596924 c0630c00 00b7d082 000009e0
<4>7bc0: c32c7bfc c32c7bd0 c01094e4 c01073c0 c06123c8 c246d000 00000000 c249e528
<4>7be0: 00b7d080 00000000 00b7d080 c0630c00 c32c7c64 c32c7c00 c010c334 c010943c
<4>7c00: 00b7d080 00000000 c32c7c34 00000b18 000009dd c322d898 00000000 00000002
<4>7c20: c249e528 00000044 c246d000 00000999 c24f1000 000009dd c0107b68 00000000
<4>7c40: 00000010 00000000 00000000 00000000 c249e528 c2bf8478 c32c7cd4 c32c7c68
<4>7c60: c010cab8 c010bf98 00000999 00000003 00000000 00000099 00000000 00000000
<4>7c80: 00000070 00000006 c24f1000 c0630c00 00000000 00000000 00000999 00001000
<4>7ca0: 00002fc0 c246d000 00007000 00000000 c249e550 c249e528 c2bf84a8 c044be20
<4>7cc0: 00000000 00000000 c32c7d24 c32c7cd8 c0106b70 c010c7c8 00007000 00001000
<4>7ce0: c32c7cf4 00000007 00001000 c0630c00 00000000 00000000 c32c7d24 c02d1c20
<4>7d00: 00001000 c044be20 00001000 c03ef678 00000000 c32c6000 c32c7dcc c32c7d28
<4>7d20: c00635d4 c01069e8 00000001 c32c7ea0 00000001 00001000 c37bf660 c2bf8540
<4>7d40: c02d1c20 c2bf84a8 00000000 be8626f0 c32c7f30 00000000 c004844c 00000001
<4>7d60: 00000000 c044be20 c04d6820 c32c7da8 00000000 00007000 c32c7da4 c32c7d88
<4>7d80: c003f220 c003ea00 00000799 1f9e8785 c2be801c c2bf84a8 c32c7dcc e52c7da8
<4>7da0: c009b0a0 ffffffff c2bf84a8 00007000 00000000 00000000 00000000 00001000
<4>7dc0: c32c7e54 c32c7dd0 c0063cec c0063154 00007000 00000000 c32c7ef0 00001000
<4>7de0: 00000000 c32c7df0 c003f220 c32c7ef0 c32c7f30 c32c7ea0 c30a7848 c3483478
<4>7e00: c37bf660 c2bf8540 00000000 00000001 00000799 1f05f105 00000799 1f05f105
<4>7e20: 00000400 00001000 c32c7e94 c2bf8514 c32c7ea0 c2bf84a8 c32c7f30 00007000
<4>7e40: 00000000 00000001 c32c7e94 c32c7e58 c0063dd0 c0063820 00000001 c32c6000
<4>7e60: c37bf660 c2bf8540 00000000 c32c7ea0 c37bf660 c32c7f78 00000000 c00230a8
<4>7e80: c32c6000 00000000 c32c7f54 c32c7e98 c0084970 c0063d68 00007000 00000000
<4>7ea0: c32c7ebc c32c7eb0 00000000 00000001 ffffffff c37bf660 00000000 00000000
<4>7ec0: 00000000 00000000 c04d6820 00000817 00000000 00000000 c037dffc c04d6820
<4>7ee0: c0050070 c32c7ee4 c32c7ee4 4014e000 00007000 00000000 c0022264 c0029d3c
<4>7f00: c0075124 c0072c88 00100073 00001000 00000000 0004001f 00000000 ffffffff
<4>7f20: 000000c5 c00230a8 00000001 00000001 be8626f0 00001000 c37bf660 c37bf660
<4>7f40: be8626f0 c32c7f78 c32c7f74 c32c7f58 c00851b4 c00848cc c00539c8 c37bf660
<4>7f60: fffffff7 00007000 c32c7fa4 c32c7f78 c00856c8 c008510c 00007000 00000000
<4>7f80: 00000000 00000000 00042040 00001000 be8626f0 00000004 00000000 c32c7fa8
<4>7fa0: c0022f00 c0085690 00042040 00001000 00000006 be8626f0 00001000 00000000
<4>7fc0: 00042040 00001000 be8626f0 00000004 00001000 00000000 4014e000 00042040
<4>7fe0: 00000004 be861668 400a49f4 400e801c 60000010 00000006 00000000 00000000

<4>Backtrace:
<4> [<c01073b4>] (jffs2_link_node_ref+0x0/0x1d0) from [<c01094e4>]
(jffs2_add_physical_node_ref+0xb4/0x178)
<4> r8:000009e0 r7:00b7d082 r6:c0630c00 r5:c0596924 r4:c32c6000
<4>[<c0109430>] (jffs2_add_physical_node_ref+0x0/0x178) from
[<c010c334>] (jffs2_write_dnode+0x3a8/0x450)
<4>[<c010bf8c>] (jffs2_write_dnode+0x0/0x450) from [<c010cab8> ]
(jffs2_write_inode_range+0x2fc/0x484)
<4>[<c010c7bc> ] (jffs2_write_inode_range+0x0/0x484) from [<c0106b70>]
(jffs2_commit_write+0x194/0x2dc )
<4>[<c01069dc>] (jffs2_commit_write+0x0/0x2dc ) from [<c00635d4>]
(generic_file_buffered_write+0x48c/0x6cc)
<4>[<c0063148>] (generic_file_buffered_write+0x0/0x6cc) from [
<c0063cec>] (__generic_file_aio_write_nolock+0x4d8/0x548)
<4> [<c0063814>] (__generic_file_aio_write_nolock+0x0/0x548) from
[<c0063dd0>] (generic_file_aio_write+0x74/0xe8)
<4>[<c0063d5c>] (generic_file_aio_write+0x0/0xe8) from [<c0084970>]
(do_sync_write+0xb0/0x100)
<4>[<c00848c0>] (do_sync_write+0x0/0x100) from [<c00851b4>]
(vfs_write+0xb4/0xdc)
<4> r6:c32c7f78 r5:be8626f0 r4:c37bf660
<4>[<c0085100>] (vfs_write+0x0/0xdc) from [<c00856c8>] (sys_write+0x44/0x70)
<4> r6: 00007000 r5:fffffff7 r4:c37bf660
<4>[<c0085684>] (sys_write+0x0/0x70) from [<c0022f00>]
(ret_fast_syscall+0x0/0x2c)
<4> r7:00000004 r6:be8626f0 r5:00001000 r4:00042040
<4>Code: e591c024 e1a04001 e59b8004 e35c0000 (058cc000)
<6>note: ipkg [13880] exited with preempt_count 1
<3>BUG: scheduling while atomic: ipkg/0x40000001/ 13880
<3>BUG: scheduling while atomic: ipkg/0x40000001/13880