* Re: jffs2 copy too long after mounting [not found] <50881109.3010107@linux.vnet.ibm.com> @ 2012-10-25 1:41 ` hejianet 2012-10-26 1:45 ` hejianet 0 siblings, 1 reply; 12+ messages in thread From: hejianet @ 2012-10-25 1:41 UTC (permalink / raw) To: linux-mtd Hi all I think I find the reason why there are so many similar logs("Check potential deletion dirent"). It seems in jffs2_garbage_collect_deletion_dirent(), jffs2 tries to find "whether this deletion dirent is still needed to actively delete a 'real' dirent with the same name that's still somewhere else on the flash." I wonder why there is such a necessary handling procedure? There must be a newest valid direntry. Why we need to remove the older one and "write out a new deletion dirent to replace it"? What if jffs2 just did not search the older deletion dirent allover the flash? Thanks. B.R. Jia On 2012-10-23 15:13, hejianet wrote: > Hi Artem, thanks for your reply > Now I can reproduce it after 2-3 days stress tests: > 1.just do lots of small writing(4k bytes each time) radomly in one of 1000 > thousand file > 2.remove the 1000 files,now the jffs2 fs is *very* dirty. > 3.I dynamically increase the threshold nr_free_blocks to trigger the gc > in jffs2_thread_should_wake(): > if (c->nr_free_blocks + c->nr_erasing_blocks < c->resv_blocks_gctrigger && > (dirty > c->nospc_dirty_size)) > ret = 1; > 4. I watched after a while, there were tens of thousands of lines (I enable the > jffs2 log level to be 1) > Check potential deletion dirent at 008ddd24 > Check potential deletion dirent at 008ddcf4 > Check potential deletion dirent at 008ddcc4 > Check potential deletion dirent at 008ddc68 > Check potential deletion dirent at 008dca34 > Check potential deletion dirent at 008dca04 > Check potential deletion dirent at 0088d77c > Check potential deletion dirent at 0088d74c > Check potential deletion dirent at 0088d71c > Check potential deletion dirent at 0088d6ec > Check potential deletion dirent at 0088d6bc > Check potential deletion dirent at 0088c648 > Check potential deletion dirent at 0088c618 > Check potential deletion dirent at 0088c2a8 > Check potential deletion dirent at 0088c278 > Check potential deletion dirent at 0088c248 > Check potential deletion dirent at 0088c218 > Check potential deletion dirent at 0088c1e8 > Check potential deletion dirent at 00886b50 > Check potential deletion dirent at 00885388 > Check potential deletion dirent at 00885358 > Check potential deletion dirent at 008852a8 > Check potential deletion dirent at 00885278 > 5. I did find that the free blocks number of jffs2 is very slowly increased > > So could you give me any ideas, it is normal or abnormal? > Thanks a lot > On 2012-08-24 15:53, Artem Bityutskiy wrote: >> On Thu, 2012-08-02 at 11:04 +0800, hejianet wrote: >>> hi All >>> my kernel is 2.6.16 >>> after mounting the nand flash with jffs2 partition, the first copy operation >>> for a 15M file(from jffs2 partition to jffs2 partition) take 9 >>> minutes,but the >>> second copy operation takes 1 minutes. >>> the disk usage(df result) is over 85%. >>> I wonder is there any patch ported to such old kernel? >> His is because how JFFS2 is designed. When the mount operation is >> finished, JFFS2 does not really finish mounting. There is the background >> process which keeps working - it basically reads all files in the system >> and checks the CRC, which is needed to find out which nodes are new and >> which are obsolete. >> >> So what happens is that you mount JFFS2, the background thread keeps >> working (see 'top' for example). >> >> While the background thread works, you can read files. But you cannot >> write - writes will be blocked. Well, it is a bit more complex - you can >> write a little, but from some point your writes will be blocked until >> the background thread finishes. Then the write will continue and finish. >> >> In you case, you mount, start writing your 15MiB file, write gets >> blocked, waits for the background thread, then continues write. The >> background thread takes ages to finish on big partitions. >> >> The second write is fast because you do not wait for the background >> thread any longer. >> > > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-10-25 1:41 ` jffs2 copy too long after mounting hejianet @ 2012-10-26 1:45 ` hejianet 2012-11-12 15:42 ` Artem Bityutskiy 0 siblings, 1 reply; 12+ messages in thread From: hejianet @ 2012-10-26 1:45 UTC (permalink / raw) To: linux-mtd Sorry,maybe my question is long and bothering:( I try to make it more easily and clearly in jffs2_garbage_collect_deletion_dirent(), what does it mean, why in nand, we can't actually mark nodes obsolete pernamently /* On a medium where we can't actually mark nodes obsolete pernamently, such as NAND flash, ..... */ On 2012-10-25 09:41, hejianet wrote: > Hi all > I think I find the reason why there are so many similar logs("Check > potential deletion dirent"). > It seems in jffs2_garbage_collect_deletion_dirent(), jffs2 tries to > find "whether this deletion dirent is still needed to actively delete > a 'real' dirent with the same name that's still somewhere else on the > flash." > I wonder why there is such a necessary handling procedure? There must > be a newest valid direntry. Why we need to remove the older one and > "write out a new deletion dirent to replace it"? What if jffs2 just > did not search the older deletion dirent allover the flash? > Thanks. > B.R. > Jia > > > On 2012-10-23 15:13, hejianet wrote: >> Hi Artem, thanks for your reply >> Now I can reproduce it after 2-3 days stress tests: >> 1.just do lots of small writing(4k bytes each time) radomly in one of 1000 >> thousand file >> 2.remove the 1000 files,now the jffs2 fs is *very* dirty. >> 3.I dynamically increase the threshold nr_free_blocks to trigger the gc >> in jffs2_thread_should_wake(): >> if (c->nr_free_blocks + c->nr_erasing_blocks < c->resv_blocks_gctrigger && >> (dirty > c->nospc_dirty_size)) >> ret = 1; >> 4. I watched after a while, there were tens of thousands of lines (I enable the >> jffs2 log level to be 1) >> Check potential deletion dirent at 008ddd24 >> Check potential deletion dirent at 008ddcf4 >> Check potential deletion dirent at 008ddcc4 >> Check potential deletion dirent at 008ddc68 >> Check potential deletion dirent at 008dca34 >> Check potential deletion dirent at 008dca04 >> Check potential deletion dirent at 0088d77c >> Check potential deletion dirent at 0088d74c >> Check potential deletion dirent at 0088d71c >> Check potential deletion dirent at 0088d6ec >> Check potential deletion dirent at 0088d6bc >> Check potential deletion dirent at 0088c648 >> Check potential deletion dirent at 0088c618 >> Check potential deletion dirent at 0088c2a8 >> Check potential deletion dirent at 0088c278 >> Check potential deletion dirent at 0088c248 >> Check potential deletion dirent at 0088c218 >> Check potential deletion dirent at 0088c1e8 >> Check potential deletion dirent at 00886b50 >> Check potential deletion dirent at 00885388 >> Check potential deletion dirent at 00885358 >> Check potential deletion dirent at 008852a8 >> Check potential deletion dirent at 00885278 >> 5. I did find that the free blocks number of jffs2 is very slowly increased >> >> So could you give me any ideas, it is normal or abnormal? >> Thanks a lot >> On 2012-08-24 15:53, Artem Bityutskiy wrote: >>> On Thu, 2012-08-02 at 11:04 +0800, hejianet wrote: >>>> hi All >>>> my kernel is 2.6.16 >>>> after mounting the nand flash with jffs2 partition, the first copy operation >>>> for a 15M file(from jffs2 partition to jffs2 partition) take 9 >>>> minutes,but the >>>> second copy operation takes 1 minutes. >>>> the disk usage(df result) is over 85%. >>>> I wonder is there any patch ported to such old kernel? >>> His is because how JFFS2 is designed. When the mount operation is >>> finished, JFFS2 does not really finish mounting. There is the background >>> process which keeps working - it basically reads all files in the system >>> and checks the CRC, which is needed to find out which nodes are new and >>> which are obsolete. >>> >>> So what happens is that you mount JFFS2, the background thread keeps >>> working (see 'top' for example). >>> >>> While the background thread works, you can read files. But you cannot >>> write - writes will be blocked. Well, it is a bit more complex - you can >>> write a little, but from some point your writes will be blocked until >>> the background thread finishes. Then the write will continue and finish. >>> >>> In you case, you mount, start writing your 15MiB file, write gets >>> blocked, waits for the background thread, then continues write. The >>> background thread takes ages to finish on big partitions. >>> >>> The second write is fast because you do not wait for the background >>> thread any longer. >>> >> >> ______________________________________________________ >> Linux MTD discussion mailing list >> http://lists.infradead.org/mailman/listinfo/linux-mtd/ >> >> >> > > > > > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-10-26 1:45 ` hejianet @ 2012-11-12 15:42 ` Artem Bityutskiy 0 siblings, 0 replies; 12+ messages in thread From: Artem Bityutskiy @ 2012-11-12 15:42 UTC (permalink / raw) To: hejianet; +Cc: linux-mtd [-- Attachment #1: Type: text/plain, Size: 665 bytes --] On Fri, 2012-10-26 at 09:45 +0800, hejianet wrote: > Sorry,maybe my question is long and bothering:( > I try to make it more easily and clearly > in jffs2_garbage_collect_deletion_dirent(), > what does it mean, why in nand, we can't actually mark nodes obsolete > pernamently > /* On a medium where we can't actually mark nodes obsolete > pernamently, such as NAND flash, ..... */ Because in NAND you cannot change the contents of a NAND page which has already been written, because you have ECC for the data in the OOB. So you cannot just mark a node as obsolete "in-place". In NOR you can do this. -- Best Regards, Artem Bityutskiy [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* jffs2 copy too long after mounting
@ 2012-08-02 3:04 hejianet
2012-08-02 3:16 ` hejianet
` (2 more replies)
0 siblings, 3 replies; 12+ messages in thread
From: hejianet @ 2012-08-02 3:04 UTC (permalink / raw)
To: linux-mtd
hi All
my kernel is 2.6.16
after mounting the nand flash with jffs2 partition, the first copy operation
for a 15M file(from jffs2 partition to jffs2 partition) take 9
minutes,but the
second copy operation takes 1 minutes.
the disk usage(df result) is over 85%.
I wonder is there any patch ported to such old kernel?
Thanks
B.R.
Jia
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: jffs2 copy too long after mounting 2012-08-02 3:04 hejianet @ 2012-08-02 3:16 ` hejianet 2012-08-06 22:54 ` Brian Norris 2012-08-02 5:58 ` hejianet 2012-08-24 7:53 ` Artem Bityutskiy 2 siblings, 1 reply; 12+ messages in thread From: hejianet @ 2012-08-02 3:16 UTC (permalink / raw) To: linux-mtd On 2012-08-02 11:04, hejianet wrote: > hi All > my kernel is 2.6.16 > after mounting the nand flash with jffs2 partition, the first copy operation > for a 15M file(from jffs2 partition to jffs2 partition) take 9 > minutes,but the > second copy operation takes 1 minutes. > the disk usage(df result) is over 85%. > I wonder is there any patch ported to such old kernel? > Thanks > > B.R. > Jia > > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > besides, how to search all the archives of linux-mtd? http://lists.infradead.org/pipermail/linux-mtd/ is a link, but it couldn't be searched totally by date. I have review all pages month by month :( ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-02 3:16 ` hejianet @ 2012-08-06 22:54 ` Brian Norris 0 siblings, 0 replies; 12+ messages in thread From: Brian Norris @ 2012-08-06 22:54 UTC (permalink / raw) To: hejianet; +Cc: linux-mtd On Wed, Aug 1, 2012 at 8:16 PM, hejianet <hejianet@linux.vnet.ibm.com> wrote: > besides, how to search all the archives of linux-mtd? > http://lists.infradead.org/pipermail/linux-mtd/ is a link, but it > couldn't be searched totally by date. I have review all pages > month by month :( You can do a search of this particular archive using a third-party search engine, although you may not get results sorted by date. For example, with Google, just include the following in your search term: site:lists.infradead.org/pipermail/linux-mtd Also, there are other archives that track the same mailing list. For example, gmane has a pretty good interface for search or threaded browsing: http://dir.gmane.org/gmane.linux.drivers.mtd (Try "search the list" or browse with the various options, like "On the web, using frames and threads") Hope that helps. Brian ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-02 3:04 hejianet 2012-08-02 3:16 ` hejianet @ 2012-08-02 5:58 ` hejianet 2012-08-06 23:02 ` Brian Norris 2012-08-24 7:53 ` Artem Bityutskiy 2 siblings, 1 reply; 12+ messages in thread From: hejianet @ 2012-08-02 5:58 UTC (permalink / raw) To: linux-mtd On 2012-08-02 11:04, hejianet wrote: > hi All > my kernel is 2.6.16 > after mounting the nand flash with jffs2 partition, the first copy operation > for a 15M file(from jffs2 partition to jffs2 partition) take 9 > minutes,but the > second copy operation takes 1 minutes. > the disk usage(df result) is over 85%. > I wonder is there any patch ported to such old kernel? > Thanks > > B.R. > Jia > > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > > Is there anyway to show the summary result of jffs2's fragmentation? Is there anyway to manually make defragmentation? Thanks Jia ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-02 5:58 ` hejianet @ 2012-08-06 23:02 ` Brian Norris 2012-08-09 5:31 ` hejianet 0 siblings, 1 reply; 12+ messages in thread From: Brian Norris @ 2012-08-06 23:02 UTC (permalink / raw) To: hejianet; +Cc: linux-mtd On Wed, Aug 1, 2012 at 10:58 PM, hejianet <hejianet@linux.vnet.ibm.com> wrote: > Is there anyway to show the summary result of jffs2's fragmentation? > Is there anyway to manually make defragmentation? I'm not a JFFS2 expert, but I don't believe fragmentation really is an issue in quite the same way as in spinning-disk, block storage filesystems like FAT. Access time to all blocks on an MTD is uniform. Also, I quite doubt there is a tool that displays the low-level details of file/block mapping. Regards, Brian ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-06 23:02 ` Brian Norris @ 2012-08-09 5:31 ` hejianet 0 siblings, 0 replies; 12+ messages in thread From: hejianet @ 2012-08-09 5:31 UTC (permalink / raw) To: linux-mtd [-- Attachment #1: Type: text/plain, Size: 1774 bytes --] Thanks, Brian Here is my latest investigation on the copy_too_long_time issue: I defined the log level of jffs2 as "noisy",and inserted timestamp for each jffs2 printk log. I watched that, there are more lines of such printk compared to good case.(bad case is first copy, takes 9 minutes, good case is second copy, takes 1 minute) 1344387878.230899000:<7>Check potential deletion dirent at 0081aad8 1344387878.233899000:<7>Check potential deletion dirent at 00819960 1344387878.236899000:<7>Check potential deletion dirent at 00819388 1344387878.239899000:<7>Check potential deletion dirent at 00818d5c 1344387878.242899000:<7>Check potential deletion dirent at 00818a7c 1344387878.245899000:<7>Check potential deletion dirent at 008141d0 1344387878.248899000:<7>Check potential deletion dirent at 0080e9a0 1344387878.251899000:<7>Check potential deletion dirent at 0080defc 1344387878.254899000:<7>Check potential deletion dirent at 0080dc8c Thanks for any suggestions or instructions. kernel version:2.6.16 attached the full logs. On 2012-08-07 07:02, Brian Norris wrote: > On Wed, Aug 1, 2012 at 10:58 PM, hejianet <hejianet@linux.vnet.ibm.com> wrote: >> Is there anyway to show the summary result of jffs2's fragmentation? >> Is there anyway to manually make defragmentation? > I'm not a JFFS2 expert, but I don't believe fragmentation really is an > issue in quite the same way as in spinning-disk, block storage > filesystems like FAT. Access time to all blocks on an MTD is uniform. > > Also, I quite doubt there is a tool that displays the low-level > details of file/block mapping. > > Regards, > Brian > > ______________________________________________________ > Linux MTD discussion mailing list > http://lists.infradead.org/mailman/listinfo/linux-mtd/ > > [-- Attachment #2: copy_too_long.zip --] [-- Type: application/octet-stream, Size: 572140 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-02 3:04 hejianet 2012-08-02 3:16 ` hejianet 2012-08-02 5:58 ` hejianet @ 2012-08-24 7:53 ` Artem Bityutskiy 2012-10-23 7:13 ` hejianet 2 siblings, 1 reply; 12+ messages in thread From: Artem Bityutskiy @ 2012-08-24 7:53 UTC (permalink / raw) To: hejianet; +Cc: linux-mtd [-- Attachment #1: Type: text/plain, Size: 1429 bytes --] On Thu, 2012-08-02 at 11:04 +0800, hejianet wrote: > hi All > my kernel is 2.6.16 > after mounting the nand flash with jffs2 partition, the first copy operation > for a 15M file(from jffs2 partition to jffs2 partition) take 9 > minutes,but the > second copy operation takes 1 minutes. > the disk usage(df result) is over 85%. > I wonder is there any patch ported to such old kernel? His is because how JFFS2 is designed. When the mount operation is finished, JFFS2 does not really finish mounting. There is the background process which keeps working - it basically reads all files in the system and checks the CRC, which is needed to find out which nodes are new and which are obsolete. So what happens is that you mount JFFS2, the background thread keeps working (see 'top' for example). While the background thread works, you can read files. But you cannot write - writes will be blocked. Well, it is a bit more complex - you can write a little, but from some point your writes will be blocked until the background thread finishes. Then the write will continue and finish. In you case, you mount, start writing your 15MiB file, write gets blocked, waits for the background thread, then continues write. The background thread takes ages to finish on big partitions. The second write is fast because you do not wait for the background thread any longer. -- Best Regards, Artem Bityutskiy [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-08-24 7:53 ` Artem Bityutskiy @ 2012-10-23 7:13 ` hejianet 2012-11-12 15:35 ` Artem Bityutskiy 0 siblings, 1 reply; 12+ messages in thread From: hejianet @ 2012-10-23 7:13 UTC (permalink / raw) To: dedekind1; +Cc: linux-mtd Hi Artem, thanks for your reply Now I can reproduce it after 2-3 days stress tests: 1.just do lots of small writing(4k bytes each time) radomly in one of 1000 thousand file 2.remove the 1000 files,now the jffs2 fs is *very* dirty. 3.I dynamically increase the threshold nr_free_blocks to trigger the gc in jffs2_thread_should_wake(): if (c->nr_free_blocks + c->nr_erasing_blocks < c->resv_blocks_gctrigger && (dirty > c->nospc_dirty_size)) ret = 1; 4. I watched after a while, there were tens of thousands of lines (I enable the jffs2 log level to be 1) Check potential deletion dirent at 008ddd24 Check potential deletion dirent at 008ddcf4 Check potential deletion dirent at 008ddcc4 Check potential deletion dirent at 008ddc68 Check potential deletion dirent at 008dca34 Check potential deletion dirent at 008dca04 Check potential deletion dirent at 0088d77c Check potential deletion dirent at 0088d74c Check potential deletion dirent at 0088d71c Check potential deletion dirent at 0088d6ec Check potential deletion dirent at 0088d6bc Check potential deletion dirent at 0088c648 Check potential deletion dirent at 0088c618 Check potential deletion dirent at 0088c2a8 Check potential deletion dirent at 0088c278 Check potential deletion dirent at 0088c248 Check potential deletion dirent at 0088c218 Check potential deletion dirent at 0088c1e8 Check potential deletion dirent at 00886b50 Check potential deletion dirent at 00885388 Check potential deletion dirent at 00885358 Check potential deletion dirent at 008852a8 Check potential deletion dirent at 00885278 5. I did find that the free blocks number of jffs2 is very slowly increased So could you give me any ideas, it is normal or abnormal? Thanks a lot On 2012-08-24 15:53, Artem Bityutskiy wrote: > On Thu, 2012-08-02 at 11:04 +0800, hejianet wrote: >> hi All >> my kernel is 2.6.16 >> after mounting the nand flash with jffs2 partition, the first copy operation >> for a 15M file(from jffs2 partition to jffs2 partition) take 9 >> minutes,but the >> second copy operation takes 1 minutes. >> the disk usage(df result) is over 85%. >> I wonder is there any patch ported to such old kernel? > His is because how JFFS2 is designed. When the mount operation is > finished, JFFS2 does not really finish mounting. There is the background > process which keeps working - it basically reads all files in the system > and checks the CRC, which is needed to find out which nodes are new and > which are obsolete. > > So what happens is that you mount JFFS2, the background thread keeps > working (see 'top' for example). > > While the background thread works, you can read files. But you cannot > write - writes will be blocked. Well, it is a bit more complex - you can > write a little, but from some point your writes will be blocked until > the background thread finishes. Then the write will continue and finish. > > In you case, you mount, start writing your 15MiB file, write gets > blocked, waits for the background thread, then continues write. The > background thread takes ages to finish on big partitions. > > The second write is fast because you do not wait for the background > thread any longer. > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: jffs2 copy too long after mounting 2012-10-23 7:13 ` hejianet @ 2012-11-12 15:35 ` Artem Bityutskiy 0 siblings, 0 replies; 12+ messages in thread From: Artem Bityutskiy @ 2012-11-12 15:35 UTC (permalink / raw) To: hejianet; +Cc: linux-mtd [-- Attachment #1: Type: text/plain, Size: 327 bytes --] On Tue, 2012-10-23 at 15:13 +0800, hejianet wrote: > Hi Artem, thanks for your reply > Now I can reproduce it after 2-3 days stress tests: You should use perf or oprofile and find out where the kernel actually spends time. Logs is not the best source of information in this case. -- Best Regards, Artem Bityutskiy [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2012-11-12 15:42 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <50881109.3010107@linux.vnet.ibm.com>
2012-10-25 1:41 ` jffs2 copy too long after mounting hejianet
2012-10-26 1:45 ` hejianet
2012-11-12 15:42 ` Artem Bityutskiy
2012-08-02 3:04 hejianet
2012-08-02 3:16 ` hejianet
2012-08-06 22:54 ` Brian Norris
2012-08-02 5:58 ` hejianet
2012-08-06 23:02 ` Brian Norris
2012-08-09 5:31 ` hejianet
2012-08-24 7:53 ` Artem Bityutskiy
2012-10-23 7:13 ` hejianet
2012-11-12 15:35 ` Artem Bityutskiy
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).