public inbox for linux-mtd@lists.infradead.org
 help / color / mirror / Atom feed
* mounting jffs2
@ 2005-04-13 13:25 Sergei Sharonov
  2005-04-13 13:50 ` Artem B. Bityuckiy
  2005-04-14  9:01 ` Ferenc Havasi
  0 siblings, 2 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-13 13:25 UTC (permalink / raw)
  To: linux-mtd

Hi,

It looks like there is a long delay before filesystem becomes available 
after mount returns. Test setup: jffs2 on 256 MByte NAND chip, 2.6.10 linux
with recent mtd snapshot. Filesystem contains one 200 MByte file written 
and fsynced in 1 kByte chunks. Checking ps it appears that gcd was the only 
thing running (RWN).
Mount and touch were run immediately one after another. I think it may be
related to the long umount time problem that I reported before. At least
the symptoms are similar: something is still going on after mount completes.
Note negligible user and sys times during touch.

# time mount /mnt/flash/

real    1m33.538s
user    0m0.020s
sys     1m33.421s
# time touch /mnt/flash/junk

real    11m10.058s
user    0m0.010s
sys     0m0.010s
#

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 13:25 mounting jffs2 Sergei Sharonov
@ 2005-04-13 13:50 ` Artem B. Bityuckiy
  2005-04-13 14:22   ` Sergei Sharonov
  2005-04-14 20:39   ` Sergei Sharonov
  2005-04-14  9:01 ` Ferenc Havasi
  1 sibling, 2 replies; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-13 13:50 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

On Wed, 2005-04-13 at 13:25 +0000, Sergei Sharonov wrote: 
> Hi,
> 
> It looks like there is a long delay before filesystem becomes available 
> after mount returns. Test setup: jffs2 on 256 MByte NAND chip, 2.6.10 linux
> with recent mtd snapshot. Filesystem contains one 200 MByte file written 
> and fsynced in 1 kByte chunks.
Believe me, if you'll write in 4K chunks, you'll effect much better
JFFS2 performance. 

> Checking ps it appears that gcd was the only 
> thing running (RWN).
It isn't bug, it is a feature. To facilitate faster mount JFFS2 defers a
lot of work to the GC thread and performs this work just after mount.
You may read filesystem but can't write. Any writer will be blocked - in
your case on 11m10.058s time interval.

> Mount and touch were run immediately one after another. I think it may be
> related to the long umount time problem that I reported before.
Measure the time of unmount just after mount. If if is also around
11m10.058s - this is a bug. Theoretically GC thread must forget its
deferred work and quit immediately.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 13:50 ` Artem B. Bityuckiy
@ 2005-04-13 14:22   ` Sergei Sharonov
  2005-04-13 15:02     ` Artem B. Bityuckiy
  2005-04-13 15:18     ` Artem B. Bityuckiy
  2005-04-14 20:39   ` Sergei Sharonov
  1 sibling, 2 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-13 14:22 UTC (permalink / raw)
  To: linux-mtd

Hi,

> Believe me, if you'll write in 4K chunks, you'll effect much better
> JFFS2 performance. 

I will try that. My log data is comming in 1 kB chunks, so in order to do
4 kB at a time I will have to implement some sort of <power fail safe>
buffering.

> It isn't bug, it is a feature. To facilitate faster mount JFFS2 defers a
> lot of work to the GC thread and performs this work just after mount.
> You may read filesystem but can't write. Any writer will be blocked - in
> your case on 11m10.058s time interval.

Hmm.. ls is not a writer but it is blocked as well.
 
> > Mount and touch were run immediately one after another. I think it may be
> > related to the long umount time problem that I reported before.
> Measure the time of unmount just after mount. If if is also around
> 11m10.058s - this is a bug. Theoretically GC thread must forget its
> deferred work and quit immediately.

Doesn't seem to be the case. Continuing the previous test:

# time mount /mnt/flash/

real    1m33.527s
user    0m0.010s
sys     1m33.471s
# time touch /mnt/flash/junk

real    11m26.228s
user    0m0.010s
sys     0m0.000s
#
#   <------------- previous post ends here ----------
#
# time umount /mnt/flash/

real    0m0.255s
user    0m0.010s
sys     0m0.240s
# time mount /mnt/flash/

real    1m33.517s
user    0m0.010s
sys     1m33.431s
# time umount /mnt/flash/

real    11m25.534s
user    0m0.000s
sys     0m0.250s
#

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 14:22   ` Sergei Sharonov
@ 2005-04-13 15:02     ` Artem B. Bityuckiy
  2005-04-13 17:51       ` Sergei Sharonov
  2005-04-13 15:18     ` Artem B. Bityuckiy
  1 sibling, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-13 15:02 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

On Wed, 2005-04-13 at 14:22 +0000, Sergei Sharonov wrote:
> I will try that. My log data is comming in 1 kB chunks, so in order to do
> 4 kB at a time I will have to implement some sort of <power fail safe>
> buffering.
It isn't obligatory to write by 4K. But if you write *intensively*, it
will be very helpful. Otherwise, just stay with 1K. And of course, the
best way to try both variants. You may try 4k without the "power fail"
things and decide whether it helps or not.

> Hmm.. ls is not a writer but it is blocked as well.
It may be blocked but for shorter time (checking one ore few inodes, not
all). Do you have very big files or directories with a lot of L1
children ?

> # time umount /mnt/flash/
> 
> real    11m25.534s
> user    0m0.000s
> sys     0m0.250s
Well, this is bad. I may suggest where to dig:

on unmount jffs2_stop_garbage_collect_thread() is called, sends SIGKILL
to the GC thread and waits for it. Meanwhile, GC thread revolves around
at jffs2_garbage_collect_pass() calling jffs2_do_crccheck_inode() for
all inodes. Theoretically it returns after it checks an inode and
processes signals (background.c:115).

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 14:22   ` Sergei Sharonov
  2005-04-13 15:02     ` Artem B. Bityuckiy
@ 2005-04-13 15:18     ` Artem B. Bityuckiy
  1 sibling, 0 replies; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-13 15:18 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

On Wed, 2005-04-13 at 14:22 +0000, Sergei Sharonov wrote:
> # time umount /mnt/flash/
> 
> real    11m25.534s
> user    0m0.000s
> sys     0m0.250s
Try to change 

D1(printk(KERN_DEBUG "jffs2_garbage_collect_thread(): SIGKILL
received.\n"));

to

printk(KERN_DEBUG "jffs2_garbage_collect_thread(): SIGKILL
received.\n");

at background.c:116 and see whether GC thread receives SIGKILL on
unmount and when it receives it (just after or on 11 min).

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 15:02     ` Artem B. Bityuckiy
@ 2005-04-13 17:51       ` Sergei Sharonov
  2005-04-14  8:41         ` Artem B. Bityuckiy
  0 siblings, 1 reply; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-13 17:51 UTC (permalink / raw)
  To: linux-mtd

Hi,
> > Hmm.. ls is not a writer but it is blocked as well.
> It may be blocked but for shorter time (checking one ore few inodes, not
> all). Do you have very big files or directories with a lot of L1
> children ?

Nope. Just three files: one is 200 MBytes, two others are small. 

# time mount /mnt/flash/

real    1m33.528s
user    0m0.010s
sys     1m33.421s
# time ls /mnt/flash/
blockno1  junk      test1

real    20m54.573s
user    0m0.010s
sys     9m32.519s
#

This is not usable for embedded application. Have I run into a bug or that
is just how things are with jffs2 on big NANDs? Is that a problem with
a 200 MByte file? 

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 17:51       ` Sergei Sharonov
@ 2005-04-14  8:41         ` Artem B. Bityuckiy
  2005-04-14 12:39           ` Sergei Sharonov
  0 siblings, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-14  8:41 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

> Nope. Just three files: one is 200 MBytes, two others are small.

That's all I need to know. JFFS2 was originally developed to work with
small NOR flashes. After NAND support was added it became possible to
utilize it on top of NAND, but with large flashes it is not very usable.
200 MB file is very large for JFFS2. It eats a lot of RAM when
you open the file and iget() is very slow with it (iget() happens when
you open the file, stat it, ls -l, etc).

This is why I started designing JFFS3, though I do this in background and 
don't really spend a lot of time (sponsors are needed :-)). Take a glimpse 
at the evolving document at 
http://www.linux-mtd.infradead.org/tech/JFFS3design.pdf, especially at the 
"JFFS2 analysis chapter", which is far not complete though. This will help 
you to realize why it is difficult to use JFFS2 on large flashes, 
especially if you have a 200 MB file :-)

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 13:25 mounting jffs2 Sergei Sharonov
  2005-04-13 13:50 ` Artem B. Bityuckiy
@ 2005-04-14  9:01 ` Ferenc Havasi
  2005-04-14  9:15   ` Artem B. Bityuckiy
  1 sibling, 1 reply; 24+ messages in thread
From: Ferenc Havasi @ 2005-04-14  9:01 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

Hi Sergei,

>It looks like there is a long delay before filesystem becomes available 
>after mount returns. Test setup: jffs2 on 256 MByte NAND chip, 2.6.10 linux
>  
>
Just an idea: you may try out our summary patch. ( 
http://www.inf.u-szeged.hu/jffs2/mount.php )

It makes the mount time faster and there is an experimental option, 
where you can disable the check the CRC of nodes which have good 
summary. (most of the nodes will have it, if you use it correctly)

Regards,
Ferenc

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14  9:01 ` Ferenc Havasi
@ 2005-04-14  9:15   ` Artem B. Bityuckiy
  2005-04-20 14:59     ` Sergei Sharonov
  0 siblings, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-14  9:15 UTC (permalink / raw)
  To: Ferenc Havasi; +Cc: linux-mtd

> Just an idea: you may try out our summary patch. (
> http://www.inf.u-szeged.hu/jffs2/mount.php )
Yes, summary will help greatly with mount time, but unfortunately it 
doesn't help with memory consumption and iget() (in Sergey's case the time 
of the background checking is the same).

> It makes the mount time faster and there is an experimental option,
> where you can disable the check the CRC of nodes which have good
> summary. (most of the nodes will have it, if you use it correctly)
Albeit I used to claim it is a good attribute not to check CRCs for
blocks with summaries, now I believe it is wrong and dangerous and must 
not be used in JFFS2.

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14  8:41         ` Artem B. Bityuckiy
@ 2005-04-14 12:39           ` Sergei Sharonov
  2005-04-14 13:11             ` Artem B. Bityuckiy
  0 siblings, 1 reply; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-14 12:39 UTC (permalink / raw)
  To: linux-mtd

Hi,

thanks, Artem.

> > Nope. Just three files: one is 200 MBytes, two others are small.
> 
> That's all I need to know. JFFS2 was originally developed to work with
> small NOR flashes. After NAND support was added it became possible to
> utilize it on top of NAND, but with large flashes it is not very usable.
> 200 MB file is very large for JFFS2. It eats a lot of RAM when
> you open the file and iget() is very slow with it (iget() happens when
> you open the file, stat it, ls -l, etc).

That's what I was afraid of. Looks like JFFS2 is not very usable on large
NAND chips. Will the situation improve if the 200 MByte file
is split into, say, multiple 20 MByte files? I guess not..

Other possibilities that I would like to investigate:

1. Partition NAND into two blocks. One will be small and will buffer incomming
data while the second big partition is being mounted. Problem: if power
cycles just before the readout the user will have to wait 20+ minutes before 
he can get to data. Argh..

2. Try YAFFS. Can anybody offer an opinion ;) ? 
JFFS2 is well supported, every resonable question or problem is followed 
up within hours. What about YAFFS? I also have a feeling that jffs2 has had 
a bit more testing and larger user base. How mature is YAFFS?

3. Try summary patch. How stable is it with jffs2? Is anybody using it
in production or is it still pretty much experimental? I recall that
initially summary was not created/updated dynamically, e.g. it was not 
usefull for log data. Is it still the case?

4. Combine (1) with (3). (1) will give fast write access, (3) hopefuly will
give fast read access????

5. Look for a commercial solution. I recall somebody was talking at 
Embedded Conference about porting their filesystem to linux.

> This is why I started designing JFFS3, though I do this in background and 
> don't really spend a lot of time (sponsors are needed ). Take a glimpse 
> at the evolving document at 
> http://www.linux-mtd.infradead.org/tech/JFFS3design.pdf, especially at the 
> "JFFS2 analysis chapter", which is far not complete though. This will help 
> you to realize why it is difficult to use JFFS2 on large flashes, 
> especially if you have a 200 MB file 

What's the point in having a big NAND if you cannot fill it ;) ?

Best regards,

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14 12:39           ` Sergei Sharonov
@ 2005-04-14 13:11             ` Artem B. Bityuckiy
  2005-04-14 14:15               ` Sergei Sharonov
  0 siblings, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-14 13:11 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

Sergei Sharonov wrote:
> Hi,
Hello :-)

> That's what I was afraid of. Looks like JFFS2 is not very usable on 
large
> NAND chips. Will the situation improve if the 200 MByte file
> is split into, say, multiple 20 MByte files? I guess not..
Yes, it is an option. Some thing will be much better. If you formulate 
specific questions, I may try do get concrete answers.

> Other possibilities that I would like to investigate:
> 1. Partition NAND into two blocks.
We usually call flash sector "block", so I am confused what do you mean. 
Just split your flash on 2 partitions?

> 2. Try YAFFS. Can anybody offer an opinion ;) ? 
> JFFS2 is well supported, every resonable question or problem is followed 
> up within hours. What about YAFFS? I also have a feeling that jffs2 has 
had 
> a bit more testing and larger user base. How mature is YAFFS?
I have no Idea. And I'd like to know this too :-)

> 3. Try summary patch. How stable is it with jffs2? Is anybody using it
> in production or is it still pretty much experimental?
AFAIK, it is quite good.

> I recall that
> initially summary was not created/updated dynamically, e.g. it was not 
> usefull for log data. Is it still the case?
No, it isn't.

> 4. Combine (1) with (3). (1) will give fast write access, (3) hopefuly 
will
> give fast read access????
You'll end up with quick mount, but the "touch your_200mb_file" command 
(just after mount) should be still be slow. touch your_20mb_file will be 
~10 times faster.

> 5. Look for a commercial solution. I recall somebody was talking at 
> Embedded Conference about porting their filesystem to linux.
May be. The other case is just to start developing JFFS3. We have some 
ideas.

> What's the point in having a big NAND if you cannot fill it ;) ?
JFFS2 wasn't designed for such large flashes/files. It has scalability 
problems. And you can fill it if you're ready to spend a lot of RAM/time.


--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14 13:11             ` Artem B. Bityuckiy
@ 2005-04-14 14:15               ` Sergei Sharonov
  2005-04-14 15:18                 ` Artem B. Bityuckiy
  0 siblings, 1 reply; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-14 14:15 UTC (permalink / raw)
  To: linux-mtd

Hi,

> > Other possibilities that I would like to investigate:
> > 1. Partition NAND into two blocks.
> We usually call flash sector "block", so I am confused what do you mean. 
> Just split your flash on 2 partitions?

Sorry, I meant two partitions. (blocks is sense of /dev/mtdblock ;)

> > 2. Try YAFFS. Can anybody offer an opinion ;) ? 
...
> I have no Idea. And I'd like to know this too 

It looks like I would need YAFFS2 in order to handle 2 kB pages. And it is
pretty new. Also the traffic is definitely lower on yaffs list.

> > 3. Try summary patch. How stable is it with jffs2? Is anybody using it
> > in production or is it still pretty much experimental?
> AFAIK, it is quite good.

That is good to know. Thanks.

> > 4. Combine (1) with (3). (1) will give fast write access, (3) hopefuly 
> will
> > give fast read access????
> You'll end up with quick mount, but the "touch your_200mb_file" command 
> (just after mount) should be still be slow. touch your_20mb_file will be 
> ~10 times faster.

I'll run a test with 10x20 MByte files and post results.

> > 5. Look for a commercial solution. I recall somebody was talking at 
> > Embedded Conference about porting their filesystem to linux.
> May be. The other case is just to start developing JFFS3. We have some 
> ideas.

What would be a time frame for a stable JFFS3 solution assuming there is a 
support?
 
> > What's the point in having a big NAND if you cannot fill it ;) ?
> JFFS2 wasn't designed for such large flashes/files. It has scalability 
> problems. And you can fill it if you're ready to spend a lot of RAM/time.

Got 32 MB, can probably go to 64 MB. Will that be sufficient to support 
256 MB NAND assuming 1 kB writes? BTW, does it make sense to increase write 
size above 1 page (2 kB). I understood from previous discussions that jffs2 
will split it on a page boundary anyway?

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14 14:15               ` Sergei Sharonov
@ 2005-04-14 15:18                 ` Artem B. Bityuckiy
  0 siblings, 0 replies; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-14 15:18 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

Sergei Sharonov wrote:
> Sorry, I meant two partitions. (blocks is sense of /dev/mtdblock ;)
It's hard to answer, I don't know your task and requirements.

> I'll run a test with 10x20 MByte files and post results.
It would be nice. If you provide mount time speeds with depending on how 
many data is placed on flash, it would be nice. I then add these numbers 
to the JFFS3 document for reference (the brief description of your 
system would be useful as well).

> What would be a time frame for a stable JFFS3 solution assuming there is a 
> support?
Design is even not ready yet. But I'd say it could take 4-6 months
depending on requirements, fetures to implement, people involved, etc.

> Got 32 MB, can probably go to 64 MB. Will that be sufficient to support 
> 256 MB NAND assuming 1 kB writes? BTW, does it make sense to increase write 
> size above 1 page (2 kB). I understood from previous discussions that jffs2 
> will split it on a page boundary anyway?
No. JFFS (even not JFFS2, but the Linux Page Cache) split on *RAM* page 
boundary which is PAGE_SIZE bytes big and is, as a rule, 4KiB. And 4KiB 
is a JFFS2's lovely size :-)

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-13 13:50 ` Artem B. Bityuckiy
  2005-04-13 14:22   ` Sergei Sharonov
@ 2005-04-14 20:39   ` Sergei Sharonov
  1 sibling, 0 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-14 20:39 UTC (permalink / raw)
  To: linux-mtd

Hi

> Believe me, if you'll write in 4K chunks, you'll effect much better
> JFFS2 performance. 

200 MByte file written on a 256 MByte NAND in 4096 byte chunks instead of 1024.
It is somewhat better (?) but still unusable. Mount time is still at 1m31s, 
but any operation after that including ls or touch are 10-14 minutes and
have inconsistent real time, sys is ~6m25s.
BTW, does it make difference if I run test on pristine, freshly
erased flash or after I created and deleted a few large files?

# time mount /mnt/flash/

real    1m31.304s
user    0m0.020s
sys     1m31.199s
# time touch /mnt/flash/test1

real    14m23.449s
user    0m0.010s
sys     6m24.856s
# umount /mnt/flash/
# time mount /mnt/flash/

real    1m31.306s
user    0m0.010s
sys     1m31.219s
# time touch /mnt/flash/test1

real    10m48.944s
user    0m0.000s
sys     6m25.156s
#
# umount /mnt/flash/
# time mount /mnt/flash/

real    1m31.300s
user    0m0.010s
sys     1m31.239s
# time touch /mnt/flash/test1

real    14m30.499s
user    0m0.000s
sys     6m24.746s
#

I will try next 10 x 20 MByte files but am afraid there is no
quick fix here.

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-14  9:15   ` Artem B. Bityuckiy
@ 2005-04-20 14:59     ` Sergei Sharonov
  2005-04-20 15:20       ` Artem B. Bityuckiy
                         ` (2 more replies)
  0 siblings, 3 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-20 14:59 UTC (permalink / raw)
  To: linux-mtd

Hi,
> > Just an idea: you may try out our summary patch. (
> > http://www.inf.u-szeged.hu/jffs2/mount.php )
> Yes, summary will help greatly with mount time, but unfortunately it 
> doesn't help with memory consumption and iget() (in Sergey's case the time 
> of the background checking is the same).

Hmm.. then what improvement does summary patch provide? In my case mount
returns in 1.5 minutes but then I cannot do anything to flash for
another 20+ minutes.

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-20 14:59     ` Sergei Sharonov
@ 2005-04-20 15:20       ` Artem B. Bityuckiy
  2005-04-20 16:34         ` Sergei Sharonov
  2005-04-21  7:48       ` Artem B. Bityuckiy
  2005-04-21 14:30       ` Ferenc Havasi
  2 siblings, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-20 15:20 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

On Wed, 2005-04-20 at 14:59 +0000, Sergei Sharonov wrote:
> Hmm.. then what improvement does summary patch provide? In my case mount
> returns in 1.5 minutes but then I cannot do anything to flash for
> another 20+ minutes.
Poor poor you :-)

Summary should accelerate mounting but leave the other characteristics 
inact in theory.

If you see that mount time only grows, that's a bug.

Read the chapter I referred you to - it will help you to understand the
nature of JFFS2 limitations.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-20 15:20       ` Artem B. Bityuckiy
@ 2005-04-20 16:34         ` Sergei Sharonov
  0 siblings, 0 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-20 16:34 UTC (permalink / raw)
  To: linux-mtd

Hi,
> On Wed, 2005-04-20 at 14:59 +0000, Sergei Sharonov wrote:
> > Hmm.. then what improvement does summary patch provide? In my case mount
> > returns in 1.5 minutes but then I cannot do anything to flash for
> > another 20+ minutes.
> Poor poor you 

More like stuck ;-)
I believe I got the hardware and the OS right - the only trouble is
a filesystem. I cannot believe that I am the only one who runs linux
with large FLASH and requires power-fail safe operation. So far I
checked into:
1. JFFS2 - seems stable, good support, mainstream, but long mount time
makes it unusable with large NAND FLASH.
2. YAFFS2 - still playing with it. Has a lot of promise but is not
IMHO a stable/usable product at this point of time.
3. Reliance/FlashFX Pro - a commercial product that promises to do
what I need but is not yet available for linux. I was told end of Q2,
early Q3.
4. Samsung RFS - there were a few announcements last summer. They never
answered my email though.
5. Stuff like TargetFFS, etc. that has not been ported (and no
intention?) to linux.
6. JFFS3 - still in design stage.
7. M-Systems software - AFAIK I cannot use it with raw NAND FLASH.

If anybody knows of a solution, please post.


> Summary should accelerate mounting but leave the other characteristics 
> inact in theory.

What exactly can I do after mount returns: ls is blocked, write is blocked..

Sergei Sharonov 
 

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-20 14:59     ` Sergei Sharonov
  2005-04-20 15:20       ` Artem B. Bityuckiy
@ 2005-04-21  7:48       ` Artem B. Bityuckiy
  2005-04-21 14:30       ` Ferenc Havasi
  2 siblings, 0 replies; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-21  7:48 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

Sergei Sharonov wrote:
> What exactly can I do after mount returns: ls is blocked, write is
blocked..

ls mustn't not blocked in theory.

Again shortly:

* ls needs iget() of the ls-ed directory inode and of all its parent
inodes.
* ls -l is ls + iget() of all L1 children inodes of the ls-ed directory.
* iget() is slow for large files like your 200MiB 
* iget() is slow for large directories (i.e., tere are a lot of L1
direntries)
* just after mount JFFS2 performs checking
* checking = fake iget() of all the filesystem inodes; fake means iget()
without actually going to the Linux Inode Cache;
* GC cant proceed while checking is in progress;
* reads may proceed;

If you have one 200MiB file then just after mount:
* jffs2 starts checking fs, you must see it typing 'top';
* ls must be quick;
* ls -l needs iget() and will do iget() of your large file.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-20 14:59     ` Sergei Sharonov
  2005-04-20 15:20       ` Artem B. Bityuckiy
  2005-04-21  7:48       ` Artem B. Bityuckiy
@ 2005-04-21 14:30       ` Ferenc Havasi
  2005-04-21 15:31         ` Artem B. Bityuckiy
  2005-04-21 15:34         ` Sergei Sharonov
  2 siblings, 2 replies; 24+ messages in thread
From: Ferenc Havasi @ 2005-04-21 14:30 UTC (permalink / raw)
  To: Sergei Sharonov; +Cc: linux-mtd

Hi Sergei,

>Hmm.. then what improvement does summary patch provide? In my case mount
>returns in 1.5 minutes but then I cannot do anything to flash for
>another 20+ minutes.
>  
>
Yes, that is the CRC checking time.

Anyway, we are now working on a new boot time speed up technique. We 
call it centralized summary. The "old" summary is erase block summary.

Centralized summary will store information not at the end of every erase 
block, but in a centralized space. It'll stores it at the umount time, 
so if you do a clean umount the next mount will be very fast. Now it a 
prototype state.

If you would like to test it I can send a patch. I would be also useful 
for us, because we don't have real NAND device, so we cannot make real 
measurements. (Forutnatelly we can use Artem's NAND simulator.)

Bye,
Ferenc

P.s.: Artem: you wrote that you don't think that swithing off the CRC 
checking of summarized node is safe. Can you explain why? (I don't know 
the behavior of NAND deeply) Becuse it may help for Sergei...

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-21 14:30       ` Ferenc Havasi
@ 2005-04-21 15:31         ` Artem B. Bityuckiy
  2005-04-26 13:35           ` Zoltan Sogor
  2005-04-21 15:34         ` Sergei Sharonov
  1 sibling, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-21 15:31 UTC (permalink / raw)
  To: Ferenc Havasi; +Cc: linux-mtd

Hello Ferenc,

> Yes, that is the CRC checking time.

I doubt that actual CRC checking takes so much time. It is mainly *read*
operation.

> P.s.: Artem: you wrote that you don't think that swithing off the CRC 
> checking of summarized node is safe. Can you explain why? (I don't
know 
> the behavior of NAND deeply) Becuse it may help for Sergei...

Because what your option does is

+#ifdef CONFIG_JFFS2_SUMMARY_NOCRCCHECK	
+	ic->state = INO_STATE_CHECKEDABSENT;
+#endif

which is incorrect. You should not skip the checking process for inodes.
When I proposed not to check CRCs I meant another thing. And now I'm
uncertain that what I proposed is absolutely correct.

Please, bear in mind that the primary goal of the checking process is
not to check CRCs but rather to detect which nodes are valid, which are
obsolete. This is done by building fake fragtrees.

Correct me if I'm wrong.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-21 14:30       ` Ferenc Havasi
  2005-04-21 15:31         ` Artem B. Bityuckiy
@ 2005-04-21 15:34         ` Sergei Sharonov
  1 sibling, 0 replies; 24+ messages in thread
From: Sergei Sharonov @ 2005-04-21 15:34 UTC (permalink / raw)
  To: linux-mtd

Hi,
> Anyway, we are now working on a new boot time speed up technique. We 
> call it centralized summary. The "old" summary is erase block summary.
..................
> If you would like to test it I can send a patch. I would be also useful 
> for us, because we don't have real NAND device, so we cannot make real 
> measurements. (Forutnatelly we can use Artem's NAND simulator.)

Ferenc, I appreciate your help. Unfortunately I don't think the patch will
improve my situation since I need to handle un-clean umounts. e.g. power 
failures and still preserve reasonable mount time.

I did a few tests with yaffs1 and yaffs2. With yaffs1, 128 MByte NAND and 100
MBytes worth of files I am getting 18 s mount time. And after 18 s filesystem
seems to be fully operational (touch, ls -l, etc.) Treat data as 
preliminary since I just built yaffs1 yesterday ;-)
I also did write/read/verify/erase of total of 6.8 GBytes of data without any 
errors. Have not done any power fail testing yet.

Now, the bad news yaffs2 did not work for me - files would disapear or get
corrupted. I would need yaffs2 to handle devices with 2 kB page size.

Sergei Sharonov

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-21 15:31         ` Artem B. Bityuckiy
@ 2005-04-26 13:35           ` Zoltan Sogor
  2005-04-26 13:51             ` Artem B. Bityuckiy
  0 siblings, 1 reply; 24+ messages in thread
From: Zoltan Sogor @ 2005-04-26 13:35 UTC (permalink / raw)
  To: linux-mtd

Hi Artem!

Artem B. Bityuckiy wrote:

>Because what your option does is
>
>+#ifdef CONFIG_JFFS2_SUMMARY_NOCRCCHECK	
>+	ic->state = INO_STATE_CHECKEDABSENT;
>+#endif
>
>which is incorrect. You should not skip the checking process for inodes.
>When I proposed not to check CRCs I meant another thing. And now I'm
>uncertain that what I proposed is absolutely correct.
>
>Please, bear in mind that the primary goal of the checking process is
>not to check CRCs but rather to detect which nodes are valid, which are
>obsolete. This is done by building fake fragtrees.
>
>Correct me if I'm wrong.
>  
>
I think our nocrc support works fine, and it is correct.

Everything is checkd and built but not at mount time. It will be checked
at the moment when you want to use the inode:

Mount:
Function jffs2_do_crccheck_inode() calls jffs2_do_read_inode_internal(),
which calls jffs2_get_inode_nodes() to collect some necessary info and
do sanity checks (crc, page boundary, ...).
After then jffs2_get_inode_nodes() can appoint which node is valid and
which is obsolated.

Read:
If you want to read an inode then the kernel calls jffs2_do_read_inode()
to get the inode.
jffs2_do_read_inode() checks that the inode is in which state
(UNCHECKED, CHECKEDABSENT, PRESENT, CHECKING, ...) In our point of view
state UNCHECKED and CHECKEDABSENT are the same, it means that in both
cases it calls jffs2_do_read_inode_internal().

In both situation nodes are checked.

I think it is necessery because if you mount the fs as readonly these
checks should happen.

Zoltan Sogor

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-26 13:35           ` Zoltan Sogor
@ 2005-04-26 13:51             ` Artem B. Bityuckiy
  2005-04-27  9:39               ` Zoltan Sogor
  0 siblings, 1 reply; 24+ messages in thread
From: Artem B. Bityuckiy @ 2005-04-26 13:51 UTC (permalink / raw)
  To: Zoltan Sogor; +Cc: linux-mtd

Hello Zoltan,

> Read:
> If you want to read an inode then the kernel calls jffs2_do_read_inode()
> to get the inode.
But if I don't want? There are thousands of inodes and I don't have to 
read them all. What then? Will GC move dirt along with valid data? Will 
I see fewer free space then there is actually present? I suppose yes and 
this is incorrect.

-- 
Best regards, Artem B. Bityuckiy
Oktet Labs (St. Petersburg), Software Engineer.
+78124286709 (office) +79112449030 (mobile)
E-mail: dedekind@oktetlabs.ru, web: http://www.oktetlabs.ru

^ permalink raw reply	[flat|nested] 24+ messages in thread

* Re: mounting jffs2
  2005-04-26 13:51             ` Artem B. Bityuckiy
@ 2005-04-27  9:39               ` Zoltan Sogor
  0 siblings, 0 replies; 24+ messages in thread
From: Zoltan Sogor @ 2005-04-27  9:39 UTC (permalink / raw)
  To: linux-mtd; +Cc: dedekind

Hello Artem!

Artem B. Bityuckiy wrote:

> Hello Zoltan,
>
>> Read:
>> If you want to read an inode then the kernel calls jffs2_do_read_inode()
>> to get the inode.
>
> But if I don't want? There are thousands of inodes and I don't have to 
> read them all. What then? Will GC move dirt along with valid data? 
> Will I see fewer free space then there is actually present? I suppose 
> yes and this is incorrect.
>
In this point of view I think you are right! The nocrc option is
incorrect and it will be removed from our summary patch.

But ... :)

We are developing a new technologic solution for faster mount. The name
of the solution is Centralized Summary. It stores summary information
from the whole flash in contrast to summary (Erase Block Summary). It
can cooperate with summary for fast recover after powerloss.

Zoltan

^ permalink raw reply	[flat|nested] 24+ messages in thread

end of thread, other threads:[~2005-04-27  9:39 UTC | newest]

Thread overview: 24+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-04-13 13:25 mounting jffs2 Sergei Sharonov
2005-04-13 13:50 ` Artem B. Bityuckiy
2005-04-13 14:22   ` Sergei Sharonov
2005-04-13 15:02     ` Artem B. Bityuckiy
2005-04-13 17:51       ` Sergei Sharonov
2005-04-14  8:41         ` Artem B. Bityuckiy
2005-04-14 12:39           ` Sergei Sharonov
2005-04-14 13:11             ` Artem B. Bityuckiy
2005-04-14 14:15               ` Sergei Sharonov
2005-04-14 15:18                 ` Artem B. Bityuckiy
2005-04-13 15:18     ` Artem B. Bityuckiy
2005-04-14 20:39   ` Sergei Sharonov
2005-04-14  9:01 ` Ferenc Havasi
2005-04-14  9:15   ` Artem B. Bityuckiy
2005-04-20 14:59     ` Sergei Sharonov
2005-04-20 15:20       ` Artem B. Bityuckiy
2005-04-20 16:34         ` Sergei Sharonov
2005-04-21  7:48       ` Artem B. Bityuckiy
2005-04-21 14:30       ` Ferenc Havasi
2005-04-21 15:31         ` Artem B. Bityuckiy
2005-04-26 13:35           ` Zoltan Sogor
2005-04-26 13:51             ` Artem B. Bityuckiy
2005-04-27  9:39               ` Zoltan Sogor
2005-04-21 15:34         ` Sergei Sharonov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox