JFFS2 SUMMARY nodes aren't working

JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On a fresh JFFS2 root FS image with summary nodes there is no
difference in mount time when enabling summary support in kernel(2.6.20,
powerpc).

 Jocke

Re: JFFS2 SUMMARY nodes aren't working

[Artem Bityutskiy]

On Thu, 2007-03-22 at 14:15 +0100, Joakim Tjernlund wrote:
> On a fresh JFFS2 root FS image with summary nodes there is no
> difference in mount time when enabling summary support in kernel(2.6.20,
> powerpc).

I would suggest to CC Ferenc Havasi <havasi@inf.u-szeged.hu>

-- 
Best regards,
Artem Bityutskiy (Битюцкий Артём)

Re: JFFS2 SUMMARY nodes aren't working

[Vitaly Wool]

Joakim Tjernlund wrote:

>On a fresh JFFS2 root FS image with summary nodes there is no
>difference in mount time when enabling summary support in kernel(2.6.20,
>powerpc).
>  
>
Can you please elaborate on how you got the image with summary nodes?

Vitaly

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Thu, 2007-03-22 at 16:36 +0300, Vitaly Wool wrote:
> Joakim Tjernlund wrote:
> 
> >On a fresh JFFS2 root FS image with summary nodes there is no
> >difference in mount time when enabling summary support in kernel(2.6.20,
> >powerpc).
> >  
> >
> Can you please elaborate on how you got the image with summary nodes?

>From my JFFS2 build script:
mkfs.jffs2 -e 0x20000 -b -d jffs2root -o "${JFFS2FILE}.tmp" || exit
sumtool -e 0x20000 -b -i "${JFFS2FILE}.tmp" -o "${JFFS2FILE}" || exit


sumtool --version
sumtool: revision 1.9
> 
> Vitaly
> 
> 

Re: JFFS2 SUMMARY nodes aren't working

[Vitaly Wool]

Joakim Tjernlund wrote:

>On Thu, 2007-03-22 at 16:36 +0300, Vitaly Wool wrote:
>  
>
>>Joakim Tjernlund wrote:
>>
>>    
>>
>>>On a fresh JFFS2 root FS image with summary nodes there is no
>>>difference in mount time when enabling summary support in kernel(2.6.20,
>>>powerpc).
>>> 
>>>
>>>      
>>>
>>Can you please elaborate on how you got the image with summary nodes?
>>    
>>
>
>>From my JFFS2 build script:
>mkfs.jffs2 -e 0x20000 -b -d jffs2root -o "${JFFS2FILE}.tmp" || exit
>sumtool -e 0x20000 -b -i "${JFFS2FILE}.tmp" -o "${JFFS2FILE}" || exit
>  
>
Ok, can you try starting from a bare filesystem? I.e. erase the
partition, copy the files, unmount, mount and see how much time it takes?

Vitaly

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Thu, 2007-03-22 at 14:52 +0100, Joakim Tjernlund wrote:
> On Thu, 2007-03-22 at 16:36 +0300, Vitaly Wool wrote:
> > Joakim Tjernlund wrote:
> > 
> > >On a fresh JFFS2 root FS image with summary nodes there is no
> > >difference in mount time when enabling summary support in kernel(2.6.20,
> > >powerpc).
> > >  
> > >
> > Can you please elaborate on how you got the image with summary nodes?
> 
> >From my JFFS2 build script:
> mkfs.jffs2 -e 0x20000 -b -d jffs2root -o "${JFFS2FILE}.tmp" || exit
> sumtool -e 0x20000 -b -i "${JFFS2FILE}.tmp" -o "${JFFS2FILE}" || exit
> 
> 
> sumtool --version
> sumtool: revision 1.9
> > 
> > Vitaly
> > 
> > 

I found that an all empty FS took about the same time to mount
which I find very strange.

Looking into the code in scan.c I found that even though I have
cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
you really need to scan the whole area after the cleanmarker?
Should be enough to only check a few words after the clean marker
and if that is 0xffffffff then assume that the EB is clean
since it had a valid clean marker at the beginning.

Below is the code in scan thats doing the scanning:

 	more_empty:
			inbuf_ofs = ofs - buf_ofs;
			(printk(KERN_CRIT "Found empty flash at 0x%08x  0x%08x 0x%08x \n", ofs, inbuf_ofs, buf_len));
			while (inbuf_ofs < buf_len) {
				if (*(uint32_t *)(&buf[inbuf_ofs]) != 0xffffffff) {
					printk(KERN_WARNING "Empty flash at 0x%08x ends at 0x%08x\n",
					       empty_start, ofs);
					if ((err = jffs2_scan_dirty_space(c, jeb, ofs-empty_start)))
						return err;
					goto scan_more;
				}

				inbuf_ofs+=4;
				ofs += 4;
			}
			/* Ran off end. */

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote:
> On Thu, 2007-03-22 at 14:52 +0100, Joakim Tjernlund wrote:
> > On Thu, 2007-03-22 at 16:36 +0300, Vitaly Wool wrote:
> > > Joakim Tjernlund wrote:
> > > 
> > > >On a fresh JFFS2 root FS image with summary nodes there is no
> > > >difference in mount time when enabling summary support in kernel(2.6.20,
> > > >powerpc).
> > > >  
> > > >
> > > Can you please elaborate on how you got the image with summary nodes?
> > 
> > >From my JFFS2 build script:
> > mkfs.jffs2 -e 0x20000 -b -d jffs2root -o "${JFFS2FILE}.tmp" || exit
> > sumtool -e 0x20000 -b -i "${JFFS2FILE}.tmp" -o "${JFFS2FILE}" || exit
> > 
> > 
> > sumtool --version
> > sumtool: revision 1.9
> > > 
> > > Vitaly
> > > 
> > > 
> 
> I found that an all empty FS took about the same time to mount
> which I find very strange.
> 
> Looking into the code in scan.c I found that even though I have
> cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> you really need to scan the whole area after the cleanmarker?
> Should be enough to only check a few words after the clean marker
> and if that is 0xffffffff then assume that the EB is clean
> since it had a valid clean marker at the beginning.
> 
> Below is the code in scan thats doing the scanning:
> 
>  	more_empty:
> 			inbuf_ofs = ofs - buf_ofs;
> 			(printk(KERN_CRIT "Found empty flash at 0x%08x  0x%08x 0x%08x \n", ofs, inbuf_ofs, buf_len));
> 			while (inbuf_ofs < buf_len) {
> 				if (*(uint32_t *)(&buf[inbuf_ofs]) != 0xffffffff) {
> 					printk(KERN_WARNING "Empty flash at 0x%08x ends at 0x%08x\n",
> 					       empty_start, ofs);
> 					if ((err = jffs2_scan_dirty_space(c, jeb, ofs-empty_start)))
> 						return err;
> 					goto scan_more;
> 				}
> 
> 				inbuf_ofs+=4;
> 				ofs += 4;
> 			}
> 			/* Ran off end. */
> 

This ugly hack makes mount extreamly fast(and I am not using SUMMARY
either):

diff --git a/fs/jffs2/scan.c b/fs/jffs2/scan.c
index b98661a..383fa2c 100644
--- a/fs/jffs2/scan.c
+++ b/fs/jffs2/scan.c
@@ -850,6 +850,9 @@ scan_more:
                                jffs2_link_node_ref(c, jeb, ofs | REF_NORMAL, c
 
                                ofs += PAD(c->cleanmarker_size);
+                               if (*(uint32_t *)(&buf[ofs-buf_ofs]) == 0xfffff
+                                       return BLK_STATE_CLEANMARKER;
+                               }
                        }
                        break;

Re: JFFS2 SUMMARY nodes aren't working

[Josh Boyer]

On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote: 
> 
> I found that an all empty FS took about the same time to mount
> which I find very strange.
> 
> Looking into the code in scan.c I found that even though I have
> cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> you really need to scan the whole area after the cleanmarker?
> Should be enough to only check a few words after the clean marker
> and if that is 0xffffffff then assume that the EB is clean
> since it had a valid clean marker at the beginning.

No,  you can't just read a few words.  On some flash, the cleanmarker
cannot be marked invalid in place so you still need to scan the rest of
the block.  Sucks.

josh

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Mon, 2007-03-26 at 07:37 -0500, Josh Boyer wrote:
> On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote: 
> > 
> > I found that an all empty FS took about the same time to mount
> > which I find very strange.
> > 
> > Looking into the code in scan.c I found that even though I have
> > cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> > you really need to scan the whole area after the cleanmarker?
> > Should be enough to only check a few words after the clean marker
> > and if that is 0xffffffff then assume that the EB is clean
> > since it had a valid clean marker at the beginning.
> 
> No,  you can't just read a few words.  On some flash, the cleanmarker
> cannot be marked invalid in place so you still need to scan the rest of
> the block.  Sucks.

Not quite true, for the non XIP case you don't read more than
EMPTY_SCAN_SIZE(c->sector_size) before declaring it free.

JFFS2 has relied on this behavior in 2.4 and I see no reason to
change that.

Re: JFFS2 SUMMARY nodes aren't working

[Josh Boyer]

On Mon, 2007-03-26 at 15:16 +0200, Joakim Tjernlund wrote:
> On Mon, 2007-03-26 at 07:37 -0500, Josh Boyer wrote:
> > On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote: 
> > > 
> > > I found that an all empty FS took about the same time to mount
> > > which I find very strange.
> > > 
> > > Looking into the code in scan.c I found that even though I have
> > > cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> > > you really need to scan the whole area after the cleanmarker?
> > > Should be enough to only check a few words after the clean marker
> > > and if that is 0xffffffff then assume that the EB is clean
> > > since it had a valid clean marker at the beginning.
> > 
> > No,  you can't just read a few words.  On some flash, the cleanmarker
> > cannot be marked invalid in place so you still need to scan the rest of
> > the block.  Sucks.
> 
> Not quite true, for the non XIP case you don't read more than
> EMPTY_SCAN_SIZE(c->sector_size) before declaring it free.
> 
> JFFS2 has relied on this behavior in 2.4 and I see no reason to
> change that.

Well, yes you're right.  Too early in the morning for me to be emailing.

But now I'm confused.  Are you running with XIP or no?  If not, then you
shouldn't be hitting this issue from what I can tell.  The check below
the code you quoted earlier explicitly looks to see if the first node
found is a cleanmarker and then returns.

I'm confused how you're hitting this issue unless you have XIP turned
on.

josh

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Mon, 2007-03-26 at 09:02 -0500, Josh Boyer wrote:
> On Mon, 2007-03-26 at 15:16 +0200, Joakim Tjernlund wrote:
> > On Mon, 2007-03-26 at 07:37 -0500, Josh Boyer wrote:
> > > On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote: 
> > > > 
> > > > I found that an all empty FS took about the same time to mount
> > > > which I find very strange.
> > > > 
> > > > Looking into the code in scan.c I found that even though I have
> > > > cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> > > > you really need to scan the whole area after the cleanmarker?
> > > > Should be enough to only check a few words after the clean marker
> > > > and if that is 0xffffffff then assume that the EB is clean
> > > > since it had a valid clean marker at the beginning.
> > > 
> > > No,  you can't just read a few words.  On some flash, the cleanmarker
> > > cannot be marked invalid in place so you still need to scan the rest of
> > > the block.  Sucks.
> > 
> > Not quite true, for the non XIP case you don't read more than
> > EMPTY_SCAN_SIZE(c->sector_size) before declaring it free.
> > 
> > JFFS2 has relied on this behavior in 2.4 and I see no reason to
> > change that.
> 
> Well, yes you're right.  Too early in the morning for me to be emailing.
> 
> But now I'm confused.  Are you running with XIP or no?  If not, then you
> shouldn't be hitting this issue from what I can tell.  The check below
> the code you quoted earlier explicitly looks to see if the first node
> found is a cleanmarker and then returns.
> 
> I'm confused how you're hitting this issue unless you have XIP turned on

I don't run XIP, but I use cfi_cmdset_0001 flash driver and that one
has support for the point/unpoint methods. 

XIP needs point but point is also an optimization that has
its own use and cfi_cmdset_0001 enables them by default.
This means that all users of above flashdriver(Intel flashes)
will use point if the flash mapping is linear. 

   Jocke

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Mon, 2007-03-26 at 17:09 +0200, Joakim Tjernlund wrote:
> On Mon, 2007-03-26 at 09:02 -0500, Josh Boyer wrote:
> > On Mon, 2007-03-26 at 15:16 +0200, Joakim Tjernlund wrote:
> > > On Mon, 2007-03-26 at 07:37 -0500, Josh Boyer wrote:
> > > > On Fri, 2007-03-23 at 18:24 +0100, Joakim Tjernlund wrote: 
> > > > > 
> > > > > I found that an all empty FS took about the same time to mount
> > > > > which I find very strange.
> > > > > 
> > > > > Looking into the code in scan.c I found that even though I have
> > > > > cleanmarkers in all EB:s, JFFS2 still scans the whole EB. Do
> > > > > you really need to scan the whole area after the cleanmarker?
> > > > > Should be enough to only check a few words after the clean marker
> > > > > and if that is 0xffffffff then assume that the EB is clean
> > > > > since it had a valid clean marker at the beginning.
> > > > 
> > > > No,  you can't just read a few words.  On some flash, the cleanmarker
> > > > cannot be marked invalid in place so you still need to scan the rest of
> > > > the block.  Sucks.
> > > 
> > > Not quite true, for the non XIP case you don't read more than
> > > EMPTY_SCAN_SIZE(c->sector_size) before declaring it free.
> > > 
> > > JFFS2 has relied on this behavior in 2.4 and I see no reason to
> > > change that.
> > 
> > Well, yes you're right.  Too early in the morning for me to be emailing.
> > 
> > But now I'm confused.  Are you running with XIP or no?  If not, then you
> > shouldn't be hitting this issue from what I can tell.  The check below
> > the code you quoted earlier explicitly looks to see if the first node
> > found is a cleanmarker and then returns.
> > 
> > I'm confused how you're hitting this issue unless you have XIP turned on
> 
> I don't run XIP, but I use cfi_cmdset_0001 flash driver and that one
> has support for the point/unpoint methods. 
> 
> XIP needs point but point is also an optimization that has
> its own use and cfi_cmdset_0001 enables them by default.
> This means that all users of above flashdriver(Intel flashes)
> will use point if the flash mapping is linear. 
> 

eeh, just to compare I disabled point in scan.c and tested diffrent
buf_size'es and was quite surprised that lowering the size made
mount go much faster, dropped from ~7 to ~2 seconds:

diff --git a/fs/jffs2/scan.c b/fs/jffs2/scan.c
index b98661a..cb66a1e 100644
--- a/fs/jffs2/scan.c
+++ b/fs/jffs2/scan.c
@@ -97,8 +97,10 @@ int jffs2_scan_medium(struct jffs2_sb_info *c)
 #ifndef __ECOS
        size_t pointlen;
 
+       c->mtd->point = c->mtd->unpoint = NULL;
        if (c->mtd->point) {
                ret = c->mtd->point (c->mtd, 0, c->mtd->size, &pointlen, &flashbuf);
+
                if (!ret && pointlen < c->mtd->size) {
                        /* Don't muck about if it won't let us point to the whole flash */
                        D1(printk(KERN_DEBUG "MTD point returned len too short: 0x%zx\n", pointlen));
@@ -115,7 +117,7 @@ int jffs2_scan_medium(struct jffs2_sb_info *c)
                if (jffs2_cleanmarker_oob(c))
                        buf_size = c->sector_size;
                else
-                       buf_size = PAGE_SIZE;
+                       buf_size = PAGE_SIZE/32;
 
                /* Respect kmalloc limitations */
                if (buf_size > 128*1024)

Re: JFFS2 SUMMARY nodes aren't working

[Ferenc Havasi]

Joakim Tjernlund írta:
> On a fresh JFFS2 root FS image with summary nodes there is no
> difference in mount time when enabling summary support in kernel(2.6.20,
> powerpc).
>   
If you use NOR flash with relatively small erase block, summary will not 
really chance to speed up the mount process.

Anyway you should check the summary is really working or not. Use the 
following way:

Enable the JFFS33 debugging options (min. debug level 1) in your kernel 
configuration and recompile the kernel. You  see the following messages 
in your kernel log. When you load the jffs2 kernel module you can see 
the following message:
"JFFS2 version 2.2. NAND SUMMARY" (in case of summary is enabled, maybe 
the NAND is skipped if you doesn't have writebuffer compiled in).

If summary is enabled and something went wrong, maybe the image on flash 
has no summary info:
"no summary found in jeb 0x%08x. Apply original scan." where %08x is 
replaced by the offset of the jeb.

If you see debug information which begin with "jffs2_sum_" and there is 
no error in your log it means the summary works.

Ferenc

Re: JFFS2 SUMMARY nodes aren't working

[Joakim Tjernlund]

On Tue, 2007-03-27 at 14:53 +0200, Ferenc Havasi wrote:
> Joakim Tjernlund írta:
> > On a fresh JFFS2 root FS image with summary nodes there is no
> > difference in mount time when enabling summary support in kernel(2.6.20,
> > powerpc).
> >   
> If you use NOR flash with relatively small erase block, summary will not 
> really chance to speed up the mount process.
> 
> Anyway you should check the summary is really working or not. Use the 
> following way:
> 
> Enable the JFFS33 debugging options (min. debug level 1) in your kernel 
> configuration and recompile the kernel. You  see the following messages 
> in your kernel log. When you load the jffs2 kernel module you can see 
> the following message:
> "JFFS2 version 2.2. NAND SUMMARY" (in case of summary is enabled, maybe 
> the NAND is skipped if you doesn't have writebuffer compiled in).
> 
> If summary is enabled and something went wrong, maybe the image on flash 
> has no summary info:
> "no summary found in jeb 0x%08x. Apply original scan." where %08x is 
> replaced by the offset of the jeb.
> 
> If you see debug information which begin with "jffs2_sum_" and there is 
> no error in your log it means the summary works.
> 
> Ferenc

Thanks, but it turned out that summary wasn't my problem. It was
excessive scanning of empty erase blocks that took time.

See my later posts and [PATCH] Speed up mount for cfi_cmdset_0001 users

  Jocke