jffs2 & new linux zlib

jffs2 & new linux zlib

[Clive Davies]

In testing jffs2 in linux 2.5.4-rmk1 I've found that kmalloc cannot allocate 
objects of the size returned by zlib_deflate_workspacesize() and 
zlib_inflate_workspacesize(). The patch below uses vmalloc to sucessfully 
allocate the memory, however I'm not sure that this is the best approach., 
maybe it would be better to try and reduce the size of the deflate_workspace 
& inflate workspace in zlib? (This might be a ridiculous idea, I haven't a 
clue how zlib works) Any suggestions?

--- linux_2.5.4-rmk1/fs/jffs2/compr_zlib.c	Mon Feb 11 01:50:06 2002
+++ linux/fs/jffs2/compr_zlib.c	Tue Feb 19 13:33:24 2002
@@ -80,8 +80,7 @@ int zlib_compress(unsigned char *data_in
 		return -1;
 
 #ifdef __KERNEL__
-	strm.workspace = kmalloc(zlib_deflate_workspacesize(),
-				 GFP_KERNEL);
+	strm.workspace = vmalloc(zlib_deflate_workspacesize());
 	if (strm.workspace == NULL) {
 		printk(KERN_WARNING "deflateInit alloc of workspace failed\n");
 		return -1;
@@ -123,7 +122,7 @@ int zlib_compress(unsigned char *data_in
 
 	}
 	zlib_deflateEnd(&strm);
-	kfree(strm.workspace);
+	vfree(strm.workspace);
 
 	D1(printk(KERN_DEBUG "zlib compressed %ld bytes into %ld\n", strm.total_in, 
strm.total_out));
 
@@ -137,7 +136,7 @@ int zlib_compress(unsigned char *data_in
 
  out_err:
 	zlib_deflateEnd(&strm);
-	kfree(strm.workspace);
+	vfree(strm.workspace);
 	return -1;
 }
 
@@ -148,8 +147,7 @@ void zlib_decompress(unsigned char *data
 	int ret;
 
 #ifdef __KERNEL__
-	strm.workspace = kmalloc(zlib_inflate_workspacesize(),
-				 GFP_KERNEL);
+	strm.workspace = vmalloc(zlib_inflate_workspacesize());
 	if (strm.workspace == NULL) {
 		printk(KERN_WARNING "inflateInit alloc of workspace failed\n");
 		return;
@@ -177,5 +175,5 @@ void zlib_decompress(unsigned char *data
 		printk(KERN_NOTICE "inflate returned %d\n", ret);
 	}
 	zlib_inflateEnd(&strm);
-	kfree(strm.workspace);
+	vfree(strm.workspace);
 }

Slow jffs2 startup on DOM

[Michael Michael]

Hello I'm using jffs2 on a DOM with the block device
driver. I find the startup times very slow. It seams
to me from reading the source that jffs2 may be
scanning every block with this driver. It takes about
3 min for the first mount. Is there anyway to fix this
?
Would it be  possible to add say a checkpoint such
that if the disk  is "clean" we proceed with normal
startup ?
I do write to the DOM but I can certianly ensure that
its clean except on sudden power downs.  I suspect
that a lot of people who write to the devices can
ensure that the writes are checkpointed so we don't
have to do a full scan.

Mike
 


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[Gad Hayisraeli]

----- Original Message -----
From: "Michael Michael" <memmel2@yahoo.com>
To: <linux-mtd@lists.infradead.org>
Sent: Wednesday, February 20, 2002 8:33 AM
Subject: Slow jffs2 startup on DOM


> Hello I'm using jffs2 on a DOM with the block device
> driver. I find the startup times very slow. It seams
> to me from reading the source that jffs2 may be
> scanning every block with this driver. It takes about
> 3 min for the first mount. Is there anyway to fix this
> ?
> Would it be  possible to add say a checkpoint such
> that if the disk  is "clean" we proceed with normal
> startup ?
> I do write to the DOM but I can certianly ensure that
> its clean except on sudden power downs.  I suspect
> that a lot of people who write to the devices can
> ensure that the writes are checkpointed so we don't
> have to do a full scan.
>
> Mike
>

in my case (mediaEngine + assabet), i encountered some slowness like you
did, just when i previously shutdown the device without unmounting + syncing
first
(this commands are regularly handled by  S0Reboot on rc6.d dir)

Gad

Re: Slow jffs2 startup on DOM

[Michael Michael]

Ahh thanks I know what it is. I'm using a nfs root and
have modified nfs to read the DOM as a persistant
local cache at boot (cool IMHO). So I'm not unmounting
the DOM clean when I unmount the NFS root. Thanks just
needed to look in the right place and "know" for sure
that it can be fast.
I had already tested and seen that if you mount/umount
its fast the second time but I guess I'm always
unclean it the case of the NFS root since I can't
manually unmount the Cache in this case oops. Of
course a nice message from JFF2 saying its doing a
clean/dirty mount would be nice of maybe I missed that
to : )

Thanks agian I'll go back into hiding

Mike

--- Gad Hayisraeli <gad@syete.co.il> wrote:
> 
> ----- Original Message -----
> From: "Michael Michael" <memmel2@yahoo.com>
> To: <linux-mtd@lists.infradead.org>
> Sent: Wednesday, February 20, 2002 8:33 AM
> Subject: Slow jffs2 startup on DOM
> 
> 
> > Hello I'm using jffs2 on a DOM with the block
> device
> > driver. I find the startup times very slow. It
> seams
> > to me from reading the source that jffs2 may be
> > scanning every block with this driver. It takes
> about
> > 3 min for the first mount. Is there anyway to fix
> this
> > ?
> > Would it be  possible to add say a checkpoint such
> > that if the disk  is "clean" we proceed with
> normal
> > startup ?
> > I do write to the DOM but I can certianly ensure
> that
> > its clean except on sudden power downs.  I suspect
> > that a lot of people who write to the devices can
> > ensure that the writes are checkpointed so we
> don't
> > have to do a full scan.
> >
> > Mike
> >
> 
> in my case (mediaEngine + assabet), i encountered
> some slowness like you
> did, just when i previously shutdown the device
> without unmounting + syncing
> first
> (this commands are regularly handled by  S0Reboot on
> rc6.d dir)
> 
> Gad
> 
> 
> 
> 
>
______________________________________________________
> Linux MTD discussion mailing list
>
http://lists.infradead.org/mailman/listinfo/linux-mtd/


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[David Woodhouse]

gad@syete.co.il said:
>  in my case (mediaEngine + assabet), i encountered some slowness like
> you did, just when i previously shutdown the device without unmounting
> + syncing first

Er, if that makes a difference to JFFS2 I'll be _very_ surprised. JFFS2 
does nothing special on unmount. Checkpointing, either on unmount or 
periodic opportunistic checkpoints, has been discussed a lot but never 
actually implemented. I'm happy to give pointers if you seriously intend to 
work on it.

--
dwmw2

Re: Slow jffs2 startup on DOM

[Michael Michael]

I think in this case its caused by the underlying
block device cleaning things up. Not JFFS2. I think
he's right.

Remeber this is the ide/block driver (DOM) not regular
mtd.

--- David Woodhouse <dwmw2@infradead.org> wrote:
> 
> gad@syete.co.il said:
> >  in my case (mediaEngine + assabet), i encountered
> some slowness like
> > you did, just when i previously shutdown the
> device without unmounting
> > + syncing first
> 
> Er, if that makes a difference to JFFS2 I'll be
> _very_ surprised. JFFS2 
> does nothing special on unmount. Checkpointing,
> either on unmount or 
> periodic opportunistic checkpoints, has been
> discussed a lot but never 
> actually implemented. I'm happy to give pointers if
> you seriously intend to 
> work on it.
> 
> --
> dwmw2
> 
> 
> 
>
______________________________________________________
> Linux MTD discussion mailing list
>
http://lists.infradead.org/mailman/listinfo/linux-mtd/


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[David Woodhouse]

memmel2@yahoo.com said:
> I think in this case its caused by the underlying block device
> cleaning things up. Not JFFS2. I think he's right. 

I didn't think the DOM knew anything about when it was opened/closed by the 
operating system, so it didn't get a chance to behave differently on an 
unmount? It's just an IDE drive, isn't it?

--
dwmw2

Re: Slow jffs2 startup on DOM

[Michael Michael]

Umm I'm thinking about this a bit more

the problem is I think in 
linux/drivers/mtd/devices/blkmtd.c

I think that it is using the buffered device through
the io buf. Thus even when you unmount you havent
flushed the io buf of the underlying block device.
Its my understanding the the linux block devices are
not raw but read and right through the io buf.

in blkmtd.c blkmtd_readpage and  write_queue_task it
does a
 brw_kiovec which is in buffer.c at this point I'm
getting a bit lost but it looks like the io is
definatly buffered  with the writes occurring through
a io buffer in a thread. The point is there not
"direct"
and not garunteed to occur until write_queue_task
runs.


I think blkmtd needs to use the new raw device api not
the linux block devices i.e...

block_dev_directIO in fs/block_dev.c umm Nope dangit
that calls evenutually brw_kiovec.  So its got to be
the write thread which should be synchronous on writes
???
I'm pretty sure thats the root of the problem a quick
look showed the other mtd device writing directly so
this is the "big difference" why is this in a thread ?


I'm sure the Gad Hayisraeli is right about umount/sync
working the question is why. 
Answer  blkmtd writes in a thread and umount/sync
wakes it up correctly. I don't think it should do
that.

Did I win a star : )

--- David Woodhouse <dwmw2@infradead.org> wrote:
> 
> memmel2@yahoo.com said:
> > I think in this case its caused by the underlying
> block device
> > cleaning things up. Not JFFS2. I think he's right.
> 
> 
> I didn't think the DOM knew anything about when it
> was opened/closed by the 
> operating system, so it didn't get a chance to
> behave differently on an 
> unmount? It's just an IDE drive, isn't it?
> 
> --
> dwmw2
> 
> 


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[Michael Michael]

Hack attack ...
Okay to test I added a call to  blockmtd_sync right at
the end of block_mtd  write
here's the monster diff
*** blkmtd.c	Thu Feb 21 05:28:39 2002
--- blkmtd.c.mike	Thu Feb 21 05:28:19 2002
***************
*** 843,852 ****
--- 843,853 ----
        SetPageUptodate(pages[pagecnt]);
        flush_dcache_page(pages[pagecnt]);
        page_cache_release(pages[pagecnt]);
      }
      kfree(pages);
+ 		blkmtd_sync(mtd);
      return 0;
    }

This fixed everything for me so I guess I was right.
But I really  this driver needs to be cleaned out to
just write direct try it with and without this change.

Mike



--- Michael Michael <memmel2@yahoo.com> wrote:
> Umm I'm thinking about this a bit more
> 
> the problem is I think in 
> linux/drivers/mtd/devices/blkmtd.c
> 
> I think that it is using the buffered device through
> the io buf. Thus even when you unmount you havent
> flushed the io buf of the underlying block device.
> Its my understanding the the linux block devices are
> not raw but read and right through the io buf.
> 
> in blkmtd.c blkmtd_readpage and  write_queue_task it
> does a
>  brw_kiovec which is in buffer.c at this point I'm
> getting a bit lost but it looks like the io is
> definatly buffered  with the writes occurring
> through
> a io buffer in a thread. The point is there not
> "direct"
> and not garunteed to occur until write_queue_task
> runs.
> 
> 
> I think blkmtd needs to use the new raw device api
> not
> the linux block devices i.e...
> 
> block_dev_directIO in fs/block_dev.c umm Nope dangit
> that calls evenutually brw_kiovec.  So its got to be
> the write thread which should be synchronous on
> writes
> ???
> I'm pretty sure thats the root of the problem a
> quick
> look showed the other mtd device writing directly so
> this is the "big difference" why is this in a thread
> ?
> 
> 
> I'm sure the Gad Hayisraeli is right about
> umount/sync
> working the question is why. 
> Answer  blkmtd writes in a thread and umount/sync
> wakes it up correctly. I don't think it should do
> that.
> 
> Did I win a star : )
> 
> --- David Woodhouse <dwmw2@infradead.org> wrote:
> > 
> > memmel2@yahoo.com said:
> > > I think in this case its caused by the
> underlying
> > block device
> > > cleaning things up. Not JFFS2. I think he's
> right.
> > 
> > 
> > I didn't think the DOM knew anything about when it
> > was opened/closed by the 
> > operating system, so it didn't get a chance to
> > behave differently on an 
> > unmount? It's just an IDE drive, isn't it?
> > 
> > --
> > dwmw2
> > 
> > 
> 
> 
> __________________________________________________
> Do You Yahoo!?
> Yahoo! Sports - Coverage of the 2002 Olympic Games
> http://sports.yahoo.com
> 
>
______________________________________________________
> Linux MTD discussion mailing list
>
http://lists.infradead.org/mailman/listinfo/linux-mtd/


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[Simon Evans]

Hi,

Sorry for not replying earlier :)

When I originally wrote blkmtd it did do synchronous writes and
there was no write thread. However this made the device *really*
slow. Obviously, If you are doing a small amount of writing compared
to reading then you could remove all the thread stuff and the outgoing
write queue and just do a brw_kiovec in blkmtd_write.

The reason the writeout stuff is so complex is because the writes
go out in the order they are done and we also have to maintain multiple
copies of pages we haven't written out. So if we write to Page 1,
then Page 2 then Page 1 again the outgoing queue has 

copy of first page 1
copy of page 2
copy of new page 1

and then the readpage has to look at this queue to find the newest
version. The reason you cant discard the first page 1 is because if you
dropped the copy of the first page 1 & made you queue look like:

copy of page 2
copy of new page 1

and the power went off just after it had written out the page 2, the
device would be inconsistent.

NOTE - we are ignoring the cache on the IDE device and assuming
it does the correct thing even if it practice it does not.

Basically, blkmtd is trying to maintain a consistent device however it 
does use a lot of memory in doing so.

If you wanted to make the code really simple you could just do synchronous 
writes. I should really do a version that does this and benchmark it as
it was a long time ago when I first had it like this.

If you wanted a faster version but werent worried about the order writes
occured, you could just mark the pages dirty and then use vm pressure
in writepage() to do write outs. You'll still have to walk the dirty
list on sync() to make sure everything is written out however.

Another thing to bear in mind is that blkmtd reads and writes in whole
pages. If you just change one byte, it has to read the page that it is
in and write it out again - not good.

Anyway, hope some of this info helps

cheers
si

>Umm I'm thinking about this a bit more
>
>the problem is I think in 
>linux/drivers/mtd/devices/blkmtd.c
>
>I think that it is using the buffered device through
>the io buf. Thus even when you unmount you havent
>flushed the io buf of the underlying block device.
>Its my understanding the the linux block devices are
>not raw but read and right through the io buf.
>
>in blkmtd.c blkmtd_readpage and  write_queue_task it
>does a
> brw_kiovec which is in buffer.c at this point I'm
>getting a bit lost but it looks like the io is
>definatly buffered  with the writes occurring through
>a io buffer in a thread. The point is there not
>"direct"
>and not garunteed to occur until write_queue_task
>runs.
>
>
>I think blkmtd needs to use the new raw device api not
>the linux block devices i.e...
>
>block_dev_directIO in fs/block_dev.c umm Nope dangit
>that calls evenutually brw_kiovec.  So its got to be
>the write thread which should be synchronous on writes
>???
>I'm pretty sure thats the root of the problem a quick
>look showed the other mtd device writing directly so
>this is the "big difference" why is this in a thread ?
>
>
>I'm sure the Gad Hayisraeli is right about umount/sync
>working the question is why. 
>Answer  blkmtd writes in a thread and umount/sync
>wakes it up correctly. I don't think it should do
>that.
>
>Did I win a star : )
>

Re: Slow jffs2 startup on DOM

[Michael Michael]

--- Simon Evans <spse@secret.org.uk> wrote:

I'm glad you wrote. I was hesitant to change the code
because of the complexity. Do you think you can add
synchronus write vs the current system as a compile
time option for this driver.
Would it be possible for you to ifdef and move the
code around a bit so both options are avialable. 

Its easy for me just to wack out all the thread stuff
and do a direct write but I think its better if your
approach and the direct write approach are both
avialable as compile options. 

I gave some responses below since I honestly don't
think its needed and if it is should be somewhere else
not the driver.  But its easy to make it optional ...


>
> Hi,
> 
> Sorry for not replying earlier :)
> 
> When I originally wrote blkmtd it did do synchronous
> writes and
> there was no write thread. However this made the
> device *really*
> slow. 
I could agian be wrong here but doesn't the page cache
handle all this buffering of writes at a higher level
? I.E there are other buffer mechanisms for write.  So
unless there is a miss in the page cache everything is
cozy. It just looks to me like your repeating the
actions of the page cache.  If you have to cache at
the device level this means to me something is wrong
with the page cache which was designed to cache on top
of slow ide block devices. 

If you are really slow on writes then maybe MTD could
do the buffering. So this write buffer would be a MTD
option. It may even make a lot of sense there for
people since it would be and option for all devices. 
Most of the MTD devices are fairly slow for writes (
IMHO ) so it makes more sense there if its really
needed. So it should be and option to MTD not in the
device driver.


>Obviously, If you are doing a small amount of
> writing compared
> to reading then you could remove all the thread
> stuff and the outgoing
> write queue and just do a brw_kiovec in
> blkmtd_write.
> 
> The reason the writeout stuff is so complex is
> because the writes
> go out in the order they are done and we also have
> to maintain multiple
> copies of pages we haven't written out. So if we
> write to Page 1,
> then Page 2 then Page 1 again the outgoing queue has
> 
> 
> copy of first page 1
> copy of page 2
> copy of new page 1
> 
> and then the readpage has to look at this queue to
> find the newest
> version. The reason you cant discard the first page
> 1 is because if you
> dropped the copy of the first page 1 & made you
> queue look like:
> 
> copy of page 2
> copy of new page 1
> 
> and the power went off just after it had written out
> the page 2, the
> device would be inconsistent.
> 
> NOTE - we are ignoring the cache on the IDE device
> and assuming
> it does the correct thing even if it practice it
> does not.
> 
> Basically, blkmtd is trying to maintain a consistent
> device however it 
> does use a lot of memory in doing so.
> 
> If you wanted to make the code really simple you
> could just do synchronous 
> writes. I should really do a version that does this
> and benchmark it as
> it was a long time ago when I first had it like
> this.
> 
> If you wanted a faster version but werent worried
> about the order writes
> occured, you could just mark the pages dirty and
> then use vm pressure
> in writepage() to do write outs. You'll still have
> to walk the dirty
> list on sync() to make sure everything is written
> out however.
> 
> Another thing to bear in mind is that blkmtd reads
> and writes in whole
> pages. If you just change one byte, it has to read
> the page that it is
> in and write it out again - not good.

See above this would only happen if the page cache is
missing badly.
> 
> Anyway, hope some of this info helps
> 
> cheers
> si
> 
> >Umm I'm thinking about this a bit more
> >
> >the problem is I think in 
> >linux/drivers/mtd/devices/blkmtd.c
> >
> >I think that it is using the buffered device
> through
> >the io buf. Thus even when you unmount you havent
> >flushed the io buf of the underlying block device.
> >Its my understanding the the linux block devices
> are
> >not raw but read and right through the io buf.
> >
> >in blkmtd.c blkmtd_readpage and  write_queue_task
> it
> >does a
> > brw_kiovec which is in buffer.c at this point I'm
> >getting a bit lost but it looks like the io is
> >definatly buffered  with the writes occurring
> through
> >a io buffer in a thread. The point is there not
> >"direct"
> >and not garunteed to occur until write_queue_task
> >runs.
> >
> >
> >I think blkmtd needs to use the new raw device api
> not
> >the linux block devices i.e...
> >
> >block_dev_directIO in fs/block_dev.c umm Nope
> dangit
> >that calls evenutually brw_kiovec.  So its got to
> be
> >the write thread which should be synchronous on
> writes
> >???
> >I'm pretty sure thats the root of the problem a
> quick
> >look showed the other mtd device writing directly
> so
> >this is the "big difference" why is this in a
> thread ?
> >
> >
> >I'm sure the Gad Hayisraeli is right about
> umount/sync
> >working the question is why. 
> >Answer  blkmtd writes in a thread and umount/sync
> >wakes it up correctly. I don't think it should do
> >that.
> >
> >Did I win a star : )
> >
> 
> 
>
______________________________________________________
> Linux MTD discussion mailing list
>
http://lists.infradead.org/mailman/listinfo/linux-mtd/


__________________________________________________
Do You Yahoo!?
Yahoo! Sports - Coverage of the 2002 Olympic Games
http://sports.yahoo.com

Re: Slow jffs2 startup on DOM

[Gad Hayisraeli]

you'll be very surprised, becasue many times that i had rebooted the machine
(mediaEngine, strataflash intel) and the loading of all the files was VERY
slow, untill i had done "shutdown -r now" and after that the machine
returned to normal operation, almost...
still i encountering a little slowness sometimes, and i suspect that the
jffs layer is fixing things online, but i'm not sure. (i got rid of the jffs
debug info so i cannot see its operation)

maybe you can think of some e2fsck for jffs2, instead of fixing the blocks
everytime without the user's knowledge ?

Gad

----- Original Message -----
From: "David Woodhouse" <dwmw2@infradead.org>
To: "Gad Hayisraeli" <gad@syete.co.il>
Cc: "Michael Michael" <memmel2@yahoo.com>; <linux-mtd@lists.infradead.org>
Sent: Thursday, February 21, 2002 10:33 AM
Subject: Re: Slow jffs2 startup on DOM


>
> gad@syete.co.il said:
> >  in my case (mediaEngine + assabet), i encountered some slowness like
> > you did, just when i previously shutdown the device without unmounting
> > + syncing first
>
> Er, if that makes a difference to JFFS2 I'll be _very_ surprised. JFFS2
> does nothing special on unmount. Checkpointing, either on unmount or
> periodic opportunistic checkpoints, has been discussed a lot but never
> actually implemented. I'm happy to give pointers if you seriously intend
to
> work on it.
>
> --
> dwmw2
>
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Slow jffs2 startup on DOM

[Joakim Tjernlund]

I see this too. Every time I boot with a fresh JFFS2 image or if
I remove som big files and reboot(does not matter if I wait before I
reboot), kupdate
runs like hell, consuming almost all cpu for a while before it calms down.
If I monitor the output from 'df', I see that the Use field slowly rises to
100% and
then its drops back to my real usage of the JFFS2 partition.

      Jocke

----- Original Message -----
From: "Gad Hayisraeli" <gad@syete.co.il>
To: "David Woodhouse" <dwmw2@infradead.org>
Cc: "Michael Michael" <memmel2@yahoo.com>; <linux-mtd@lists.infradead.org>
Sent: den 24 februari 2002 10:18
Subject: Re: Slow jffs2 startup on DOM


> you'll be very surprised, becasue many times that i had rebooted the
machine
> (mediaEngine, strataflash intel) and the loading of all the files was VERY
> slow, untill i had done "shutdown -r now" and after that the machine
> returned to normal operation, almost...
> still i encountering a little slowness sometimes, and i suspect that the
> jffs layer is fixing things online, but i'm not sure. (i got rid of the
jffs
> debug info so i cannot see its operation)
>
> maybe you can think of some e2fsck for jffs2, instead of fixing the blocks
> everytime without the user's knowledge ?
>
> Gad
>
> ----- Original Message -----
> From: "David Woodhouse" <dwmw2@infradead.org>
> To: "Gad Hayisraeli" <gad@syete.co.il>
> Cc: "Michael Michael" <memmel2@yahoo.com>; <linux-mtd@lists.infradead.org>
> Sent: Thursday, February 21, 2002 10:33 AM
> Subject: Re: Slow jffs2 startup on DOM
>
>
> >
> > gad@syete.co.il said:
> > >  in my case (mediaEngine + assabet), i encountered some slowness like
> > > you did, just when i previously shutdown the device without unmounting
> > > + syncing first
> >
> > Er, if that makes a difference to JFFS2 I'll be _very_ surprised. JFFS2
> > does nothing special on unmount. Checkpointing, either on unmount or
> > periodic opportunistic checkpoints, has been discussed a lot but never
> > actually implemented. I'm happy to give pointers if you seriously intend
> to
> > work on it.
> >
> > --
> > dwmw2
> >
> >
> >
> > ______________________________________________________
> > Linux MTD discussion mailing list
> > http://lists.infradead.org/mailman/listinfo/linux-mtd/
>
>
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: Slow jffs2 startup on DOM

[David Woodhouse]

(moved to correct list)

Joakim.Tjernlund@lumentis.se said:
> I see this too. Every time I boot with a fresh JFFS2 image or if I
> remove som big files and reboot(does not matter if I wait before I
> reboot), kupdate runs like hell, consuming almost all cpu for a while
> before it calms down. If I monitor the output from 'df', I see that
> the Use field slowly rises to 100% and then its drops back to my real
> usage of the JFFS2 partition. 

That one I can explain, although it shouldn't make any difference whether 
you reboot cleanly or not.

When we obsoleted the final node in an eraseblock, we used to stick it 
straight on the erase_pending_list for recycling. That screwed the wear 
levelling - if you continually create and remove {lots of,large} files, 
then you'll keep using the same blocks over and over again.

So I took that out, and we just file those blocks on the dirty_list for 
later erasure by the garbage-collection code. However, the mount code will 
observe that they're entirely empty and file them for immediate erasure, 
and this is what makes kupdated eat your CPU on remount.

I see two options for fixing this.

First, we could partially revert the earlier change - mostly just erase 
the block straight away as we used to, but _occasionally_ put it on the 
dirty_list to keep the wear levelling moving.

Secondly, we could prevent the code from erasing the all-dirty blocks
immediately on mount, by filing them at the front of the dirty_list so 
they're first up for garbage-collection. 

I've just committed the first option to CVS, which should render the second 
pointless - there will rarely be many such blocks anyway. 

Thomas - this may break the NAND support, as it takes some NAND-specific 
code out of #if 0. Although the NAND support was already broken, because 
that code really should have been used :)

In the 1 in 64 times that we just stick this block on the dirty_list instead
of erasing it immediately (or putting it on the erase_pending_wbuf list),
what stops the GC code from subsequently erasing it before the wbuf is
flushed? (Until today, that was 64 in 64 times.)

Index: nodemgmt.c
===================================================================
RCS file: /home/cvs/mtd/fs/jffs2/nodemgmt.c,v
retrieving revision 1.55
retrieving revision 1.56
diff -u -p -r1.55 -r1.56
--- nodemgmt.c	2002/02/21 23:50:46	1.55
+++ nodemgmt.c	2002/02/24 11:55:23	1.56
@@ -31,7 +31,7 @@
  * provisions above, a recipient may use your version of this file
  * under either the RHEPL or the GPL.
  *
- * $Id: nodemgmt.c,v 1.55 2002/02/21 23:50:46 gleixner Exp $
+ * $Id: nodemgmt.c,v 1.56 2002/02/24 11:55:23 dwmw2 Exp $
  *
  */
 
@@ -348,13 +348,17 @@ void jffs2_mark_node_obsolete(struct jff
 		D2(printk(KERN_DEBUG "Not moving nextblock 0x%08x to dirty/erase_pending list\n", jeb->offset));
 	} else if (jeb == c->gcblock) {
 		D2(printk(KERN_DEBUG "Not moving gcblock 0x%08x to dirty/erase_pending list\n", jeb->offset));
-#if 0 /* We no longer do this here. It can screw the wear levelling. If you have a lot of static
-	 data and a few blocks free, and you just create new files and keep deleting/overwriting
-	 them, then you'd keep erasing and reusing those blocks without ever moving stuff around.
-	 So we leave completely obsoleted blocks on the dirty_list and let the GC delete them 
-	 when it finds them there. That way, we still get the 'once in a while, take a clean block'
-	 to spread out the flash usage */
-	} else if (!jeb->used_size) {
+	} else if (!jeb->used_size && (jiffies % 64)) {
+		/* We no longer do this unconditionally. It can screw the wear
+		   levelling. If you have a lot of static data and a few
+		   blocks free, and you just create new files and keep
+		   deleting/overwriting them, then you'd keep erasing and
+		   reusing those blocks without ever moving stuff around.  So
+		   occasionally we leave completely obsoleted blocks on the
+		   dirty_list and let the GC delete them when it finds them
+		   there. That way, we still get the 'once in a while, take a
+		   clean block' to spread out the flash usage.
+		*/
 		D1(printk(KERN_DEBUG "Eraseblock at 0x%08x completely dirtied. Removing from (dirty?) list...\n", jeb->offset));
 		list_del(&jeb->list);
 		if (c->wbuf_len) {
@@ -387,7 +391,6 @@ void jffs2_mark_node_obsolete(struct jff
 			jffs2_erase_pending_trigger(c);
 		}
 		D1(printk(KERN_DEBUG "Done OK\n"));
-#endif
 	} else if (jeb->dirty_size == ref->totlen) {
 		D1(printk(KERN_DEBUG "Eraseblock at 0x%08x is freshly dirtied. Removing from clean list...\n", jeb->offset));
 		list_del(&jeb->list);

--
dwmw2

RE: Slow jffs2 startup on DOM

[Joakim Tjernlund]

> >    Any chance to see this patch in the jffs2-2_4-branch too? 
> 
> Have you tested it? I haven't :)

Not yet. I will start with the other stuff that went in during the weekend :-)
 
> I've actually changed round a lot of that code in the trunk now, fixing the
> NAND problem I mentioned - that cleanup won't go into the stable branch, but
> the original one probably can.
> 
> To be honest, I'm more interested in working out why kupdated blocks and 
> fixing that, rather than trying to work around it by reducing the symptoms.

hmm, thinking of it, I remember something about kupdate on LK. A quick
search gives: http://www.cs.helsinki.fi/linux/linux-kernel/2002-07/0213.html
Looks like Andrew Morton is on it.

 Jocke