data loss on jffs2 filesystem on dataflash

data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Hi,
I loose files, which I write on a jffs2 filesystem.

I start with a freshly erased mtd partition. Then I start the following 
sequence:

mount /conf
echo aaa > /conf/aaa
echo aaa > /conf/aaa
umount /conf
mount /conf

After remounting, the file gets lost (I think because the GC removes the 
inode). I use kernel 2.6.13 with mtd out of cvs of today.
The machine is a AT91RM9200 with a serial dataflash. I used already
jffs2 (most time readonly) as rootfs here for months without problems. 
The only specialty is, that the dataflash has a unfamiliar erase size of 
8448 bytes.

Maybe someone can give me a pointer, what might be wrong.

Best regards
Peter

Here a log of the problem:
(The lines PM: are write and erase logs at the mtd device)

mount /conf
JFFS2 write-buffering enabled (8448)
[JFFS2 DBG] (109) __jffs2_dbg_dump_block_lists_nolock: dump JFFS2 blocks 
lists:
[JFFS2 DBG] (109) __jffs2_dbg_dump_block_lists_nolock: dump JFFS2 blocks 
lists:
[JFFS2 DBG] (109) jffs2_do_read_inode: read inode #1
[JFFS2 DBG] (109) jffs2_do_read_inode: creating inocache for root inode
[JFFS2 DBG] (109) jffs2_add_ino_cache: add c0e213c4 (ino #1)
[JFFS2 DBG] (109) jffs2_do_read_inode_internal: ino #1 nlink is 1
[JFFS2 DBG] (109) jffs2_get_inode_nodes: ino #1
[JFFS2 DBG] (109) jffs2_get_inode_nodes: nodes of inode #1 were read, 
the highest version is 0, latest_mctime 3221420188, mctime_ver 0.
[root@armbox /root]$echo  aaa > /conf/aaa
[JFFS2 DBG] (94) jffs2_add_ino_cache: add c0e213dc (ino #2)
[JFFS2 DBG] (94) jffs2_add_fd_to_list: add dirent "aaa", ino #2
[JFFS2 DBG] (111) jffs2_add_full_dnode_to_inode: adding node 0x00-0x04 
@0x006a1770 on flash, newfrag *c0f343b8
[root@armbox /root]$
jffs2: No clean, dirty _or_ erasable blocks to GC from! Where are they all?
jffs2: Couldn't find erase block to garbage collect!
PM: dataflash_write: 8574720 .. 8583168

[root@armbox /root]$echo aaa > /conf/aaa
[JFFS2 DBG] (94) jffs2_truncate_fragtree: truncating fragtree to 
0x00000000 bytes
[JFFS2 DBG] (112) jffs2_add_full_dnode_to_inode: adding node 0x00-0x04 
@0x006a17fc on flash, newfrag *c0f343b8
[root@armbox /root]$jffs2: No clean, dirty _or_ erasable blocks to GC 
from! Where are they all?
jffs2: Couldn't find erase block to garbage collect!
PM: dataflash_write: 8574720 .. 8583168

[root@armbox /root]$umount /conf
[JFFS2 DBG] (113) jffs2_kill_fragtree: killing
[root@armbox /root]$mount /conf
JFFS2 write-buffering enabled (8448)
[JFFS2 DBG] (114) jffs2_scan_eraseblock: no summary found in jeb 
0x006a1700. Apply original scan.
[JFFS2 DBG] (114) jffs2_add_ino_cache: add c0f933c4 (ino #2)
[JFFS2 DBG] (114) __jffs2_dbg_dump_block_lists_nolock: dump JFFS2 blocks 
lists:
[JFFS2 DBG] (114) __jffs2_dbg_dump_block_lists_nolock: dump JFFS2 blocks 
lists:
[JFFS2 DBG] (114) jffs2_do_read_inode: read inode #1
[JFFS2 DBG] (114) jffs2_do_read_inode: creating inocache for root inode
[JFFS2 DBG] (114) jffs2_add_ino_cache: add c0f933ac (ino #1)
[JFFS2 DBG] (114) jffs2_do_read_inode_internal: ino #1 nlink is 1
[JFFS2 DBG] (114) jffs2_get_inode_nodes: ino #1
[JFFS2 DBG] (114) jffs2_get_inode_nodes: nodes of inode #1 were read, 
the highest version is 0, latest_mctime 3221420188, mctime_ver 0.
[JFFS2 DBG] (8) jffs2_del_ino_cache: del c0f933c4 (ino #2)

PM: dataflash_erase: addr=8574720 len=8448

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> Hi,
> I loose files, which I write on a jffs2 filesystem.
Do you have sumary enabled ? If yes, try to disable it.

> Maybe someone can give me a pointer, what might be wrong.
Frankly, I do not know DataFlash specificity but I have several basic 
questions bellow :)

> mount /conf
> JFFS2 write-buffering enabled (8448)
> [JFFS2 DBG] (109) __jffs2_dbg_dump_block_lists_nolock: dump JFFS2 blocks 
[snip]
> [JFFS2 DBG] (111) jffs2_add_full_dnode_to_inode: adding node 0x00-0x04 
Why I don't see messages from scan.c? Strange.

> [root@armbox /root]$
[snip]
> PM: dataflash_write: 8574720 .. 8583168
[snip]
> PM: dataflash_write: 8574720 .. 8583168
two nodes are written to the same place ? That's odd...

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityuckiy wrote:
> Peter Menzebach wrote:
...
>> I loose files, which I write on a jffs2 filesystem.
> 
> Do you have sumary enabled ? If yes, try to disable it.
It's not enabled.
> 
...
>> [JFFS2 DBG] (111) jffs2_add_full_dnode_to_inode: adding node 0x00-0x04 
> 
> Why I don't see messages from scan.c? Strange.
My fault, new log attached

...
>> PM: dataflash_write: 8574720 .. 8583168
> 
> [snip]
> 
>> PM: dataflash_write: 8574720 .. 8583168
> 
> two nodes are written to the same place ? That's odd...

Yes, that's what I thought, too.

new log...

[42949436.200000] jffs2_get_sb(): dev_name "/dev/mtdblock/2"
[42949436.200000] jffs2_get_sb(): path_lookup() returned 0, inode c0fcde94
[42949436.200000] jffs2_get_sb_mtd(): New superblock for device 2 ("etcfs")
[42949436.200000] JFFS2 write-buffering enabled (8448)
[42949436.200000] Allocating readbuf of 4096 bytes
[42949436.200000] jffs2_scan_eraseblock(): Scanning block at 0x0
[42949436.200000] Block at 0x00000000 is empty (erased)
[42949436.200000] jffs2_scan_eraseblock(): Scanning block at 0x2100
[42949436.210000] Block at 0x00002100 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0x4200
[42949436.210000] Block at 0x00004200 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0x6300
[42949436.210000] Block at 0x00006300 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0x8400
[42949436.210000] Block at 0x00008400 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0xa500
[42949436.210000] Block at 0x0000a500 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0xc600
[42949436.210000] Block at 0x0000c600 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0xe700
[42949436.210000] Block at 0x0000e700 is empty (erased)
[42949436.210000] jffs2_scan_eraseblock(): Scanning block at 0x10800
[42949436.220000] Block at 0x00010800 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x12900
[42949436.220000] Block at 0x00012900 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x14a00
[42949436.220000] Block at 0x00014a00 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x16b00
[42949436.220000] Block at 0x00016b00 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x18c00
[42949436.220000] Block at 0x00018c00 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x1ad00
[42949436.220000] Block at 0x0001ad00 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x1ce00
[42949436.220000] Block at 0x0001ce00 is empty (erased)
[42949436.220000] jffs2_scan_eraseblock(): Scanning block at 0x1ef00
[42949436.220000] Block at 0x0001ef00 is empty (erased)
[42949436.220000] Scanned flash completely
[42949436.220000] [JFFS2 DBG] (102) __jffs2_dbg_dump_block_lists_nolock: 
dump JFFS2 blocks lists:
[42949436.230000] flash_size: 0x021000
[42949436.230000] used_size: 0x000000
[42949436.230000] dirty_size: 0x000000
[42949436.230000] wasted_size: 0x000000
[42949436.230000] unchecked_size: 0x000000
[42949436.230000] free_size: 0x021000
[42949436.230000] erasing_size: 0x000000
[42949436.230000] bad_size: 0x000000
[42949436.230000] sector_size: 0x002100
[42949436.230000] jffs2_reserved_blocks size: 0x000000
[42949436.230000] nextblock: NULL
[42949436.230000] gcblock: NULL
[42949436.230000] clean_list: empty
[42949436.230000] very_dirty_list: empty
[42949436.230000] dirty_list: empty
[42949436.230000] erasable_list: empty
[42949436.230000] erasing_list: empty
[42949436.230000] erase_pending_list: empty
[42949436.230000] erasable_pending_wbuf_list: empty
[42949436.230000] bad_list: empty
[42949436.230000] bad_used_list: empty
[42949436.230000] Pass 1 complete
[42949436.230000] Pass 2 starting
[42949436.230000] Pass 2a starting
[42949436.230000] Pass 2 complete
[42949436.230000] Pass 3 complete
[42949436.230000] [JFFS2 DBG] (102) __jffs2_dbg_dump_block_lists_nolock: 
dump JFFS2 blocks lists:
[42949436.230000] flash_size: 0x021000
[42949436.230000] used_size: 0x000000
[42949436.230000] dirty_size: 0x000000
[42949436.230000] wasted_size: 0x000000
[42949436.230000] unchecked_size: 0x000000
[42949436.230000] free_size: 0x021000
[42949436.230000] erasing_size: 0x000000
[42949436.230000] bad_size: 0x000000
[42949436.230000] sector_size: 0x002100
[42949436.230000] jffs2_reserved_blocks size: 0x000000
[42949436.230000] nextblock: NULL
[42949436.230000] gcblock: NULL
[42949436.230000] clean_list: empty
[42949436.230000] very_dirty_list: empty
[42949436.230000] dirty_list: empty
[42949436.230000] erasable_list: empty
[42949436.230000] erasing_list: empty
[42949436.230000] erase_pending_list: empty
[42949436.230000] erasable_pending_wbuf_list: empty
[42949436.230000] bad_list: empty
[42949436.230000] bad_used_list: empty
[42949436.230000] Not rotating empty clean_list
[42949436.230000] Not rotating empty very_dirty_list
[42949436.230000] Not rotating empty dirty_list
[42949436.230000] Not rotating empty erasable_list
[42949436.230000] Not rotating empty erase_pending_list
[42949436.230000] Rotating free_list by 7
[42949436.230000] Erase block at front of free_list is at 0000e700
[42949436.230000] JFFS2 trigger levels (size 132 KiB, block size 8 KiB, 
16 blocks)
[42949436.230000] Blocks required to allow deletion:    2 (16 KiB)
[42949436.230000] Blocks required to allow writes:      3 (24 KiB)
[42949436.230000] Blocks required to quiesce GC thread: 4 (33 KiB)
[42949436.230000] Blocks required to allow GC merges:   3 (24 KiB)
[42949436.230000] Blocks required to GC bad blocks:     0 (0 KiB)
[42949436.230000] Amount of dirty space required to GC: 9799 bytes
[42949436.230000] jffs2_do_fill_super(): Getting root inode
[42949436.230000] jffs2_read_inode(): inode->i_ino == 1
[42949436.230000] [JFFS2 DBG] (102) jffs2_do_read_inode: read inode #1
[42949436.230000] [JFFS2 DBG] (102) jffs2_do_read_inode: creating 
inocache for root inode
[42949436.230000] [JFFS2 DBG] (102) jffs2_add_ino_cache: add c0ebc3c4 
(ino #1)
[42949436.230000] [JFFS2 DBG] (102) jffs2_do_read_inode_internal: ino #1 
nlink is 1
[42949436.230000] [JFFS2 DBG] (102) jffs2_get_inode_nodes: ino #1
[42949436.230000] [JFFS2 DBG] (102) jffs2_get_inode_nodes: nodes of 
inode #1 were read, the highest version is 0, latest_mctime 3221420412, 
mctime_ver 0.
[42949436.230000] jffs2_read_inode() returning
[42949436.230000] jffs2_do_fill_super(): d_alloc_root()
[42949436.230000] JFFS2: Garbage collect thread is pid 103
[42949436.230000] jffs2_thread_should_wake(): nr_free_blocks 16, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949436.230000] jffs2_garbage_collect_thread sleeping...

[root@armbox /root]$echo aaa > /conf/aaa
[root@armbox /root]$d[42949498.310000] jffs2: No clean, dirty _or_ 
erasable blocks to GC from! Where are they all?
[42949498.310000] jffs2: Couldn't find erase block to garbage collect!
[42949498.320000] PM: dataflash_write: 1022208 .. 1030656
mesg -c
[42949495.620000] jffs2_lookup()
[42949495.620000] jffs2_create()
[42949495.620000] jffs2_new_inode(): dir_i 1, mode 0x81a4
[42949495.620000] [JFFS2 DBG] (94) jffs2_add_ino_cache: add c0ebc3dc 
(ino #2)
[42949495.620000] jffs2_do_new_inode(): Assigned ino# 2
[42949495.620000] jffs2_reserve_space(): Requested 0x44 bytes
[42949495.620000] jffs2_reserve_space(): alloc sem got
[42949495.620000] jffs2_find_nextblock(): new nextblock = 0x0000e700
[42949495.620000] jffs2_do_reserve_space(): Giving 0x2100 bytes at 0xe700
[42949495.620000] jffs2_do_create(): reserved 0x2100 bytes
[42949495.620000] jffs2_add_physical_node_ref(): Node at 0xe700(2), size 
0x44
[42949495.620000] jffs2_write_dnode wrote node at 0x0000e700(2) with 
dsize 0x0, csize 0x0, node_crc 0xbb60398c, data_crc 0x00000000, totlen 
0x00000044
[42949495.620000] jffs2_do_create created file with mode 0x81a4
[42949495.620000] jffs2_complete_reservation()
[42949495.620000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949495.620000] jffs2_reserve_space(): Requested 0x2c bytes
[42949495.620000] jffs2_reserve_space(): alloc sem got
[42949495.620000] jffs2_do_reserve_space(): Giving 0x20bc bytes at 0xe744
[42949495.620000] jffs2_write_dirent(ino #1, name at *0xc1dcb970 
"aaa"->ino #2, name_crc 0x0f46aa3f)
[42949495.620000] jffs2_add_physical_node_ref(): Node at 0xe744(2), size 
0x2c
[42949495.620000] [JFFS2 DBG] (94) jffs2_add_fd_to_list: add dirent 
"aaa", ino #2
[42949495.620000] jffs2_complete_reservation()
[42949495.620000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949495.620000] jffs2_create: Created ino #2 with mode 100644, nlink 
1(1). nrpages 0
[42949495.640000] jffs2_prepare_write()
[42949495.640000] jffs2_read_inode_range: ino #2, range 
0x00000000-0x00001000
[42949495.640000] Filling non-frag hole from 0-4096
[42949495.640000] end prepare_write(). pg->flags 209
[42949495.640000] jffs2_commit_write(): ino #2, page at 0x0, range 0-4, 
flags 209
[42949495.640000] jffs2_write_inode_range(): Ino #2, ofs 0x0, len 0x4
[42949495.640000] jffs2_reserve_space(): Requested 0xc4 bytes
[42949495.640000] jffs2_reserve_space(): alloc sem got
[42949495.640000] jffs2_do_reserve_space(): Giving 0x2090 bytes at 0xe770
[42949495.640000] jffs2_add_physical_node_ref(): Node at 0xe770(2), size 
0x48
[42949495.640000] jffs2_write_dnode wrote node at 0x0000e770(2) with 
dsize 0x4, csize 0x4, node_crc 0x33de5e85, data_crc 0x56bc8289, totlen 
0x00000048
[42949495.640000] [JFFS2 DBG] (105) jffs2_add_full_dnode_to_inode: 
adding node 0x00-0x04 @0x0000e770 on flash, newfrag *c0eac3b8
[42949495.640000] Obsoleting node at 0x0000e700 of len 0x44: <7>Wasting
[42949495.640000] jffs2_complete_reservation()
[42949495.640000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949495.640000] increasing writtenlen by 4
[42949495.640000] jffs2_commit_write() returning 4
[42949498.310000] jffs2_write_super()
[42949498.310000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949498.310000] jffs2_erase_pending_blocks completed
[42949498.310000] jffs2_flush_wbuf_gc() called for ino #0...
[42949498.310000] jffs2_flush_wbuf_gc() calls gc pass
[42949498.310000] jffs2: No clean, dirty _or_ erasable blocks to GC 
from! Where are they all?
[42949498.310000] jffs2: Couldn't find erase block to garbage collect!
[42949498.320000] PM: dataflash_write: 1022208 .. 1030656
[42949498.460000] jffs2_flush_wbuf_gc() ends...

[root@armbox /root]$echo aaa > /conf/aaa
[root@armbox /root]$[42949518.460000] jffs2: No clean, dirty _or_ 
erasable blocks to GC from! Where are they all?
[42949518.460000] jffs2: Couldn't find erase block to garbage collect!
[42949518.470000] PM: dataflash_write: 1022208 .. 1030656

[root@armbox /root]$dmesg -c
[42949515.170000] jffs2_setattr(): ino #2
[42949515.170000] jffs2_reserve_space(): Requested 0x44 bytes
[42949515.170000] jffs2_reserve_space(): alloc sem got
[42949515.170000] jffs2_do_reserve_space(): Giving 0x2048 bytes at 0xe7b8
[42949515.170000] jffs2_add_physical_node_ref(): Node at 0xe7b8(2), size 
0x44
[42949515.170000] jffs2_write_dnode wrote node at 0x0000e7b8(2) with 
dsize 0x0, csize 0x0, node_crc 0x247c3272, data_crc 0x00000000, totlen 
0x00000044
[42949515.170000] [JFFS2 DBG] (94) jffs2_truncate_fragtree: truncating 
fragtree to 0x00000000 bytes
[42949515.170000] Obsoleting node at 0x0000e770 of len 0x48: <7>Wasting
[42949515.170000] jffs2_complete_reservation()
[42949515.170000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949515.180000] jffs2_prepare_write()
[42949515.180000] jffs2_read_inode_range: ino #2, range 
0x00000000-0x00001000
[42949515.190000] Filling non-frag hole from 0-4096
[42949515.190000] end prepare_write(). pg->flags 209
[42949515.190000] jffs2_commit_write(): ino #2, page at 0x0, range 0-4, 
flags 209
[42949515.190000] jffs2_write_inode_range(): Ino #2, ofs 0x0, len 0x4
[42949515.190000] jffs2_reserve_space(): Requested 0xc4 bytes
[42949515.190000] jffs2_reserve_space(): alloc sem got
[42949515.190000] jffs2_do_reserve_space(): Giving 0x2004 bytes at 0xe7fc
[42949515.190000] jffs2_add_physical_node_ref(): Node at 0xe7fc(2), size 
0x48
[42949515.190000] jffs2_write_dnode wrote node at 0x0000e7fc(2) with 
dsize 0x4, csize 0x4, node_crc 0x8f17e80f, data_crc 0x56bc8289, totlen 
0x00000048
[42949515.190000] [JFFS2 DBG] (107) jffs2_add_full_dnode_to_inode: 
adding node 0x00-0x04 @0x0000e7fc on flash, newfrag *c0eac3b8
[42949515.190000] Obsoleting node at 0x0000e7b8 of len 0x44: <7>Wasting
[42949515.190000] jffs2_complete_reservation()
[42949515.190000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949515.190000] increasing writtenlen by 4
[42949515.190000] jffs2_commit_write() returning 4
[42949518.460000] jffs2_write_super()
[42949518.460000] jffs2_thread_should_wake(): nr_free_blocks 15, 
nr_erasing_blocks 0, dirty_size 0x0: no
[42949518.460000] jffs2_erase_pending_blocks completed
[42949518.460000] jffs2_flush_wbuf_gc() called for ino #0...
[42949518.460000] jffs2_flush_wbuf_gc() calls gc pass
[42949518.460000] jffs2: No clean, dirty _or_ erasable blocks to GC 
from! Where are they all?
[42949518.460000] jffs2: Couldn't find erase block to garbage collect!
[42949518.470000] PM: dataflash_write: 1022208 .. 1030656
[42949518.610000] jffs2_flush_wbuf_gc() ends...


-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:

[snip]
 > [42949498.320000] PM: dataflash_write: 1022208 .. 1030656
[snip]
> [42949495.640000] jffs2_write_dnode wrote node at 0x0000e770(2) with 
> dsize 0x4, csize 0x4, node_crc 0x33de5e85, data_crc 0x56bc8289, totlen 
> 0x00000048
JFFS2 writes a node at offset 0x0000e770.

[snip]
> [42949518.470000] PM: dataflash_write: 1022208 .. 1030656
[snip]
> [42949515.170000] Obsoleting node at 0x0000e770 of len 0x48: <7>Wasting
After the second echo, it obsolets the old data node.

[snip]
> [42949515.190000] jffs2_write_dnode wrote node at 0x0000e7fc(2) with 
> dsize 0x4, csize 0x4, node_crc 0x8f17e80f, data_crc 0x56bc8289, totlen 
> 0x00000048
And writes the new node at 0x0000e7fc

But your messages show that the write goes to the same place.

May be (guessing):
1. Your messages are invalid?
2. Something goes on with JFFS2 wbuf and it is committed to the same 
page (unlikely)?
3. Your flash driver is incorrect?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Thanks for your help!

I found (at least partly) out, whats happening: Some parts of code are 
relying still on the assumption, that the erase size/sector size is 
fixed to a power of 2.

Example nodelist.c:
len = ofs & (c->wbuf_pagesize - 1)

So the jffs2 code is broken, if it *should* support erase sizes which 
are not 2^X.

Best regards
Peter



-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> Thanks for your help!
> 
> I found (at least partly) out, whats happening: Some parts of code are 
> relying still on the assumption, that the erase size/sector size is 
> fixed to a power of 2.
> 
> Example nodelist.c:
> len = ofs & (c->wbuf_pagesize - 1)
> 
Ah!
What's your erasesize?

I'll fix this.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> Thanks for your help!
> 
> I found (at least partly) out, whats happening: Some parts of code are 
> relying still on the assumption, that the erase size/sector size is 
> fixed to a power of 2.
What's your c->wbuf_pagesize ?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> short note: The sector size is 1056 bytes, but I report 8*1056 or 
> 16*1056 byte to the mtd layer. As far as I see, this seems to be better 
> for jffs2, and otherwise I had to patch the mkfs.jffs2 every time I use 
> it ;)

Hello Peter,

I've returned to explore your problem. Do you have any success?
I have a vague idea what happens, need to think a little. Unfortunately, 
I have no DataFlash to just debug this.

Please, write me your flash name - I want to glance at its data sheet.

Thanks,

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> Artem B. Bityutskiy wrote:
> 
>> I've returned to explore your problem. Do you have any success?
> 
> 
> No, not at then moment. If I have some time, I can try to rewrite the 
> chipset driver, that it reports a sector size of 1024.
> 
>> I have a vague idea what happens, need to think a little. 
>> Unfortunately, I have no DataFlash to just debug this.
>>
>> Please, write me your flash name - I want to glance at its data sheet.
> 
> Here the datasheet:
> http://www.atmel.com/dyn/resources/prod_documents/DOC1638.PDF
> 

Peter, please, let we be public. This may be helpful for others.
Ok, I'll write you what I think later.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> No, not at then moment. If I have some time, I can try to rewrite the 
> chipset driver, that it reports a sector size of 1024.
Well, I don't think this is the right approach, this is rather
a dirty'n'ugly approach...

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Artem B. Bityutskiy wrote:
> Peter Menzebach wrote:
> 
>> No, not at then moment. If I have some time, I can try to rewrite the 
>> chipset driver, that it reports a sector size of 1024.
> 
> 
> I glanced at the manual. Uhh, DataFlash is very specific beast. It 
> suppoers page program with built-in erase command... So DataFlash 
> effectively may be considered as a block device. Then you may use any FS 
> on it providing you have wrote proper driver? Why do you need JFFS2 then 
> :-) ?
> 
> JFFS2 orients to "classical" flashes. They have no "write page with 
> built-in erase" operation.
> 
> Didn't read the manual carefully, what do they refer by "Main memory 
> array"?
> 
> BTW, having 8*1056 write buffer is not perfect ides, better make it as 
> small as possible, i.e., 1056 bytes.
> 
Ups, I sent the mail to LKML instead of MTD ML by accident :-)

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
>> Artem B. Bityutskiy wrote:
>> I glanced at the manual. Uhh, DataFlash is very specific beast. It 
>> suppoers page program with built-in erase command... So DataFlash 
>> effectively may be considered as a block device. Then you may use any 
>> FS on it providing you have wrote proper driver? Why do you need JFFS2 
>> then :-) ?
I do need some wear leveling (and a filesystem which supports block 
sizes of 1056 bytes ;) )

>> JFFS2 orients to "classical" flashes. They have no "write page with 
>> built-in erase" operation.
>>
>> Didn't read the manual carefully, what do they refer by "Main memory 
>> array"?
The main memory array is the flash itself with pages of 1056 bytes

>> BTW, having 8*1056 write buffer is not perfect ides, better make it as 
>> small as possible, i.e., 1056 bytes.
At the moment, the write buffer = reported erase block size. Is this 
neccessary that this is equal?

my reasons for setting erase block size to 8(16)*1056:
1. my lazyness -> I have to patch the mkfs.jff2 every time, since it 
makes a guess, that -e 1056 is not a very good idea (at lest the 
versions some times ago)

2. It seemed to me, that jffs2 boots faster, if the number of blocks is 
smaller.

3. A guess without proof: jffs2 likes blocks which are not too small

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

On Tue, 2005-09-20 at 15:38 +0200, Peter Menzebach wrote:
> I do need some wear leveling (and a filesystem which supports block 
> sizes of 1056 bytes ;) )
Ok.

> The main memory array is the flash itself with pages of 1056 bytes
Ok, just strange name. On the picture they call it "Flash memory array".

> my reasons for setting erase block size to 8(16)*1056:
> 1. my lazyness -> I have to patch the mkfs.jff2 every time, since it 
> makes a guess, that -e 1056 is not a very good idea (at lest the 
> versions some times ago)
-e specifies the *erasesize*. In your case, make it = DataFlash block
size = 8*1056. So, all is OK. Although you may make it 1056 (DataFlash
allows this?), it is bad idea.

But In your logs, I saw that you have *write buffer* size = 8*1056!
Write buffer size is another thing. It is the minimal flash IO unit.
JFFS2 assumes that it cannot write 1 byte or 100 bytes, it assumes that
it can only write 'write buffer size' bytes. And the goal of the write
buffer is to accumulate many small JFFS2 writes in RAM, and when the
write buffer becomes full, it is flushed to flash. 

So, in your case, make write buffer = Data Flash page size = 1056.


> 2. It seemed to me, that jffs2 boots faster, if the number of blocks is 
> smaller.
I guess we have got confused with terminology. Please, let use this:

eraseblock - the minimal erase unit
IO block - the minimal IO unit = write buffer size

eraseblocks are usually larger then IO blocks.

The larger is the IO block, the more data you loose due to unclean
reboots in case of JFFS2. So, don't make it larger then 1056.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> On Tue, 2005-09-20 at 15:38 +0200, Peter Menzebach wrote:

...
> 
> But In your logs, I saw that you have *write buffer* size = 8*1056!
> Write buffer size is another thing. It is the minimal flash IO unit.
> JFFS2 assumes that it cannot write 1 byte or 100 bytes, it assumes that
> it can only write 'write buffer size' bytes. And the goal of the write
> buffer is to accumulate many small JFFS2 writes in RAM, and when the
> write buffer becomes full, it is flushed to flash. 
> 
> So, in your case, make write buffer = Data Flash page size = 1056.
> 
How can this reported from the flash device to the mtd layer and then to 
jffs2?
In the code, I see at the moment only, that in the device driver 
mtd_info.erasesize is set, later on in jffs2 I see, that this has become 
the sector_size, which becomes then the wbuf_pagesize.

Is here mtd_info.ooblock to be set?


Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Andrew Victor]

hi,

> How can this reported from the flash device to the mtd layer and then to 
> jffs2?

Via mtd_info.erasesize


> In the code, I see at the moment only, that in the device driver 
> mtd_info.erasesize is set, later on in jffs2 I see, that this has become 
> the sector_size, which becomes then the wbuf_pagesize.

To reduce the number of erase blocks, JFFS2 may also concatenate a
number of erase-blocks in a 'virtual block'.  That is what sector_size
and wbuf_pagesize might get set to.


> Is here mtd_info.ooblock to be set?

No.  All bytes on a DataFlash device are directly addressable, so there
is not really an OOB area.


Regards,
  Andrew Victor

Re: data loss on jffs2 filesystem on dataflash

[Jörn Engel]

On Tue, 20 September 2005 17:11:00 +0200, Andrew Victor wrote:
> 
> > How can this reported from the flash device to the mtd layer and then to 
> > jffs2?
> 
> Via mtd_info.erasesize
> 
> > In the code, I see at the moment only, that in the device driver 
> > mtd_info.erasesize is set, later on in jffs2 I see, that this has become 
> > the sector_size, which becomes then the wbuf_pagesize.
> 
> To reduce the number of erase blocks, JFFS2 may also concatenate a
> number of erase-blocks in a 'virtual block'.  That is what sector_size
> and wbuf_pagesize might get set to.

Not any more.  That strategy can cause data loss and other fun things
which we don't want.

> > Is here mtd_info.ooblock to be set?
> 
> No.  All bytes on a DataFlash device are directly addressable, so there
> is not really an OOB area.

Jörn

-- 
Good warriors cause others to come to them and do not go to others.
-- Sun Tzu

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Hi Andrew,

On Tue, 2005-09-20 at 17:11 +0200, Andrew Victor wrote:
> > In the code, I see at the moment only, that in the device driver 
> > mtd_info.erasesize is set, later on in jffs2 I see, that this has become 
> > the sector_size, which becomes then the wbuf_pagesize.
> 
> To reduce the number of erase blocks, JFFS2 may also concatenate a
> number of erase-blocks in a 'virtual block'.  That is what sector_size
> and wbuf_pagesize might get set to.
Yes, right. Albeit this is fixed, this doesn't matter.

The question is *why you set wbuf size = eraseblock size?* In case of
DataFlash, I assume, wbuf_size = DataFlash page size (the minimal
writing unit). Eraseblock size must be = DataFlash block size.

Yes, it is theoretically correct to set write buffer size = eraseblock
size, but this is *really* bad. This is wasteful (imagine, any sync will
will add lots of padding and waste the whole eraseblock).

And, the size wbuf has nothing to do with "concatenate number of
eraseblocks", whare is the relation?

> > Is here mtd_info.ooblock to be set?
> 
> No.  All bytes on a DataFlash device are directly addressable, so there
> is not really an OOB area.
Stop, mtd_info.ooblock is the size of NAND page in case of NAND. I would
read it as 'the size of block which contains OOB'. Agree, not very good
name.

So, I offered to fill mtd_info.ooblock by the size of DataFlash page,
and set.

Comments? Thanks.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Andrew Victor]

hi Artem,

> > To reduce the number of erase blocks, JFFS2 may also concatenate a
> > number of erase-blocks in a 'virtual block'.  That is what sector_size
> > and wbuf_pagesize might get set to.
> Yes, right. Albeit this is fixed, this doesn't matter.
> 
> The question is *why you set wbuf size = eraseblock size?

I don't.  I set the wbuf_pagesize to the sector_size.
(Which at that time was 1 or more eraseblocks.  And I think sector_size
is/was the smallest unit of flash that JFFS2 would ever erase/write).


> In case of
> DataFlash, I assume, wbuf_size = DataFlash page size (the minimal
> writing unit). Eraseblock size must be = DataFlash block size.

Maybe there is some confusion here.  Dataflash has both 'pages' and
'blocks'.  With a 'page' being the smallest unit that can be
erased/written.

The MTD-device's erasesize is therefore set to the smallest unit - the
Dataflash page size (528 or 1056, depending on device)


> Yes, it is theoretically correct to set write buffer size = eraseblock
> size, but this is *really* bad. This is wasteful (imagine, any sync will
> will add lots of padding and waste the whole eraseblock).

Dataflash does not use Padding nodes.


> And, the size wbuf has nothing to do with "concatenate number of
> eraseblocks", whare is the relation?

Maybe it would be more optimal to set the wbuf_size to eraseblocksize? 
I don't know.

I do know that JFFS2-on-Dataflash was working fine when I committed the
changes in February.


Regards,
  Andrew Victor

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Andrew,

Andrew Victor wrote:
> I don't.  I set the wbuf_pagesize to the sector_size.
> (Which at that time was 1 or more eraseblocks.  And I think sector_size
> is/was the smallest unit of flash that JFFS2 would ever erase/write).
You used to think or still think :-) ?
eraseblock, which is of size = c->sector_size is the smallest JFFS2 
erase unit.
The smallest write unit is usually *smaller* then eraseblock and = 
c->wbuf_pagesize. Your DataFlash port has c->wbuf_pagesize = 
c->sector_size which is wasteful. This leads to much faster Flash wear. 
This makes unneeded loads to JFFS2. This should work, but not the best 
approach.

> Maybe there is some confusion here.  Dataflash has both 'pages' and
> 'blocks'.  With a 'page' being the smallest unit that can be
> erased/written.
Yes, but it is impossible to make JFFS2 eraseblock = DataFlash page 
size. Why? Because the page is too small. JFFS2's nodes cannot overlap 
eraseblock's borders. So, we cannot fit large 4K JFFS2 nodes to such 
small (1056 bytes) eraseblocks. Thus, the most logical approach is to 
make eraseblock size = DataFlash page size.

> 
> The MTD-device's erasesize is therefore set to the smallest unit - the
> Dataflash page size (528 or 1056, depending on device)
Well, may be this is right that *MTD* reports erasesize = DataFlash page 
size. But JFFS2 then must realize that it works on top of DataFlash and 
use blocks as erase units.

> Dataflash does not use Padding nodes.
I'm about JFFS2 - it pads wbuf. But well, it does not matter anyway. If 
your wbuf is 8K, you write there 100 byters then do sync, you wate (8K - 
100) bytes. It doen't matter whether you write padding or not. You waste 
the space, period. :-)

> Maybe it would be more optimal to set the wbuf_size to eraseblocksize? 
> I don't know.
Grrr. No! We don't understand each other at all :-)

> I do know that JFFS2-on-Dataflash was working fine when I committed the
> changes in February.
Now it doesn't :-) Sure, you don't have to fix it :-) We were just 
curious. I guess Peter should deal with this.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> Andrew,
> 
> Andrew Victor wrote:
> 
>> I do know that JFFS2-on-Dataflash was working fine when I committed the
>> changes in February.
> 
> Now it doesn't :-) Sure, you don't have to fix it :-) We were just 
> curious. I guess Peter should deal with this.
> 
Ok guys,
I will try, to get it to work, but at the moment I'm a little bit lost.

So here, what I saw in the data sheet, code and your answers:

Dataflash:

The dataflash has a smallest unit = dataflash page, which can be written 
and erased (528/1056 bytes)

The dataflash has blocks, which are 8 * pagesize. This units may be used 
for programming/erasing, but is not used in the driver

The dataflash has sectors, which are 256 pages (sector0+1 are special). 
These sectors are only relevant in terms of wear leveling, since each 
page of a sector must be reprogrammed at least once each 10.000 writes.

jffs2:
jffs2 nodes are 4 K
the wbuf_size should be the smallest size, which you can re-program

My assumptions now:

We have to set the jffs2->sector_size to 8 * dataflash pagesize
We have to set the jffs2->wbuf_size to dataflash pagesize (528/1056)
We have to set the mtd->erasesize to 8 * dataflash pagesize


And still I didn't get the point, why it isn't working today....

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

On Wed, 2005-09-21 at 12:27 +0200, Peter Menzebach wrote:
> My assumptions now:
> 
> We have to set the jffs2->sector_size to 8 * dataflash pagesize
> We have to set the jffs2->wbuf_size to dataflash pagesize (528/1056)
Yes, IMO.

> We have to set the mtd->erasesize to 8 * dataflash pagesize
For JFFS2 it is OK. But for other purposes it may be not OK. So, that's
up to you. MTD is NOR/NAND-centric at the moment. You may fix this.

> And still I didn't get the point, why it isn't working today....
Me too so far :-) If you send me a DataFlash-ed board - I'll dig
this :-)

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Artem B. Bityutskiy wrote:
>>We have to set the mtd->erasesize to 8 * dataflash pagesize
> 
> For JFFS2 it is OK. But for other purposes it may be not OK. So, that's
> up to you. MTD is NOR/NAND-centric at the moment. You may fix this.
> 
I mean, MTD probably must returm both - page and block sized. JFFS2 
should use block.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> On Wed, 2005-09-21 at 12:27 +0200, Peter Menzebach wrote:
> 
>>My assumptions now:
>>
>>We have to set the jffs2->sector_size to 8 * dataflash pagesize
>>We have to set the jffs2->wbuf_size to dataflash pagesize (528/1056)
> 
> Yes, IMO.
> 
Ok, sector_size now 8*1056, wbuf_size = 1056 erase_size=8*1056
-> kernel oops
-> ok, found SECTOR_ADDR was defined wrong
-> after fixing stable and not working correctly as before ;)
...
>>And still I didn't get the point, why it isn't working today....
> 
> Me too so far :-) If you send me a DataFlash-ed board - I'll dig
> this :-)

Ok, Ok, now I am digging a bit....
with some more printfs, I now looked a bit clearer, and have another 
question:
when jffs2_flush_wbuf is called, how is this area now markes as written.
Sometimes later jffs2_do_reserve_space is called an returns the same 
address, which was written to flash....

Any hints?

Best regards
Peter


-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> Ok, Ok, now I am digging a bit....
> with some more printfs, I now looked a bit clearer, and have another 
> question:
> when jffs2_flush_wbuf is called, how is this area now markes as written.
> Sometimes later jffs2_do_reserve_space is called an returns the same 
> address, which was written to flash....
> 
The space is accounted by corresponding fields in jffs2_eraseblock 
structure: free_size, wasted_size, used_size.

Jffs2 always writes to c->sector_size - jeb->free_size.

The corresponding place in jffs2_flush_wbuf, where the parameters a 
adjusted is wbuf.c:506.

But.. Argh! Look at line 488: if (pad && !jffs2_dataflash(c))

Why !jffs2_dataflash(c)??? I bet this is the bug.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
>
> But.. Argh! Look at line 488: if (pad && !jffs2_dataflash(c))
> 
> Why !jffs2_dataflash(c)??? I bet this is the bug.
> 
Nope, it's OK there, only when padding.

In my logs, wbuf_len is always 0.

Then when I write the flash,
free_size, wasted_size, used_size is adjusted with 0.

How is the intention of the write buffer?
- I write 10 bytes at the beginning
- Then after 2 seconds, the buffer is flushed
- I it correct, to adjust with wbuf_len, or must it be wbuf_size?

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

On Wed, 2005-09-21 at 18:10 +0200, Peter Menzebach wrote:
> Artem B. Bityutskiy wrote:
> >
> > But.. Argh! Look at line 488: if (pad && !jffs2_dataflash(c))
> > 
> > Why !jffs2_dataflash(c)??? I bet this is the bug.
> > 
> Nope, it's OK there, only when padding.
OK, its late here and my head is working only on 2%. But anyway, today I
insist this is a bug. :-)

I don't understand what is the pad parameter at all. AFAICS, it is just
an old rudiment. What is padding? If the write buffer is not full, we
fill the rest of it by padding and flush it. Why we may not want this? I
have no idea. This is something odd. Why DataFlash is something
special??? May be I'm too stupid today to understand this? Later.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> On Wed, 2005-09-21 at 18:10 +0200, Peter Menzebach wrote:
> 
>>Artem B. Bityutskiy wrote:
>>
>>>But.. Argh! Look at line 488: if (pad && !jffs2_dataflash(c))
>>>
>>>Why !jffs2_dataflash(c)??? I bet this is the bug.
>>>
>>
>>Nope, it's OK there, only when padding.
> 
> OK, its late here and my head is working only on 2%. But anyway, today I
> insist this is a bug. :-)
> 
> I don't understand what is the pad parameter at all. AFAICS, it is just
> an old rudiment. What is padding? If the write buffer is not full, we
> fill the rest of it by padding and flush it. Why we may not want this? I
> have no idea. This is something odd. Why DataFlash is something
> special??? May be I'm too stupid today to understand this? Later.
> 
You are right (both):
It's late, and it was the bug....
Tomorrow is another day....

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Peter Menzebach wrote:
> Artem B. Bityutskiy wrote:
> 
>> On Wed, 2005-09-21 at 18:10 +0200, Peter Menzebach wrote:
>>
>>> Artem B. Bityutskiy wrote:
>>>
>>>> But.. Argh! Look at line 488: if (pad && !jffs2_dataflash(c))
>>>>
>>>> Why !jffs2_dataflash(c)??? I bet this is the bug.
>>>>
>>>
>>> Nope, it's OK there, only when padding.
>>
>>
>> OK, its late here and my head is working only on 2%. But anyway, today I
>> insist this is a bug. :-)
>>
>> I don't understand what is the pad parameter at all. AFAICS, it is just
>> an old rudiment. What is padding? If the write buffer is not full, we
>> fill the rest of it by padding and flush it. Why we may not want this? I
>> have no idea. This is something odd. Why DataFlash is something
>> special??? May be I'm too stupid today to understand this? Later.
>>
> You are right (both):
> It's late, and it was the bug....
> Tomorrow is another day....
> 
Ok, so far one step again...
but I don't find the place, where jeb->free_size and c->free_size is 
adjusted for the *data* (wbuf_len). I find this only for the padding.
Any ideas?

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> Ok, so far one step again...
> but I don't find the place, where jeb->free_size and c->free_size is 
> adjusted for the *data* (wbuf_len). I find this only for the padding.
> Any ideas?

Hmm, at the very end of __jffs2_flush_wbuf() I suppose.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> Artem B. Bityutskiy wrote:
>> But In your logs, I saw that you have *write buffer* size = 8*1056!
>> Write buffer size is another thing. It is the minimal flash IO unit.
>> JFFS2 assumes that it cannot write 1 byte or 100 bytes, it assumes that
>> it can only write 'write buffer size' bytes. And the goal of the write
>> buffer is to accumulate many small JFFS2 writes in RAM, and when the
>> write buffer becomes full, it is flushed to flash.
>> So, in your case, make write buffer = Data Flash page size = 1056.
>>
> How can this reported from the flash device to the mtd layer and then to 
> jffs2?
> In the code, I see at the moment only, that in the device driver 
> mtd_info.erasesize is set, later on in jffs2 I see, that this has become 
> the sector_size, which becomes then the wbuf_pagesize.
I see, in jffs2_dataflash_setup(): c->wbuf_pagesize = c->sector_size;
I don't know why Andrew Victor set wbuf size to Sector Size. May be 
you'll ask him?

Ok, theoretically this should work as well. But having smaller wbuf size 
is better.

> Is here mtd_info.ooblock to be set?
Yes, I in case of NAND we use this, I guess you may use it as well ... 
May be it is time to introduce a 'writesize' field to mtd_info, instead 
of 'oobblock' ... ?


-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> Peter Menzebach wrote:
> 
>> Artem B. Bityutskiy wrote:
>>
>>> But In your logs, I saw that you have *write buffer* size = 8*1056!
>>> Write buffer size is another thing. It is the minimal flash IO unit.
>>> JFFS2 assumes that it cannot write 1 byte or 100 bytes, it assumes that
>>> it can only write 'write buffer size' bytes. And the goal of the write
>>> buffer is to accumulate many small JFFS2 writes in RAM, and when the
>>> write buffer becomes full, it is flushed to flash.
>>> So, in your case, make write buffer = Data Flash page size = 1056.
>>>
>> How can this reported from the flash device to the mtd layer and then 
>> to jffs2?
>> In the code, I see at the moment only, that in the device driver 
>> mtd_info.erasesize is set, later on in jffs2 I see, that this has 
>> become the sector_size, which becomes then the wbuf_pagesize.
> 
> I see, in jffs2_dataflash_setup(): c->wbuf_pagesize = c->sector_size;
> I don't know why Andrew Victor set wbuf size to Sector Size. May be 
> you'll ask him?
> 
> Ok, theoretically this should work as well. But having smaller wbuf size 
> is better.

Ok, wirh the answer from Andrew, the conclusion is for me at the moment:

At the moment the mtd layer only reports one usable "tile" value to 
jffs2, which is the erasesize.

So the erase_size is the only sensful value, which can be used as 
sector_size and wbuf_pagesize.

I can fix the mkfs.jffs2 (if needed), that it can generate correct 
images for a erase_size of 1056.

Open questions:

I think, the original problem with the lost files is not solved, if I 
reduce the erase size?

Make sector_size=erase_size=wbuf_size=1056 sense?

Best regards
Peter






-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Jörn Engel]

On Tue, 20 September 2005 18:11:03 +0400, Artem B. Bityutskiy wrote:
> Jörn Engel wrote:
> >Still can't.  Block devices have the attribute that writing AAA... to
> >a block containing BBB... gives you one of three possible results in
> >case of power failure:
> >
> >1. BBB...BBB all written
> >2. AAA...AAA nothing written
> >3. AAA...BBB partially written.
> >
> >Flash doesn't have 3, but two more cases:
> >4. FFF...FFF erased, nothing written
> >5. AAA...FFF erased, partially written
> >
> >Plus the really obnoxious
> >6. FFF...FFF partially erased.  Looks fine but some bits may flip
> >   randomly, writes may not stick, etc.
> >
> >Now try finding a filesystem that is robust if 4-6 happens. ;)
> Don't underastand this. If you mean the atomicity, CRC may help here. 
> And no problems. Or may be you missed the the fact that we have 
> eraseblock size = writeblock size?

Image this block contains the used block bitmap of ext2.  All blocks
handled with this bitmap are free, so the complete block contains
zeroes.  Now we allocate a single block, writing a single one to the
bitmap.  When writing, a power failure happens right after the erase:

Before:
00000000000

After:
FFFFFFFFFF

Gee, every single block is used now.  Good thing this happened to the
block bitmap, so we only lost some space on the fs.  Somewhere else,
we'd have data corruption.

Any hard disk filesystem will suffer some kind of corruption in such a
case.  They heavily depend on the non-existance of 4-6.

> >>JFFS2 orients to "classical" flashes. They have no "write page with 
> >>built-in erase" operation.
> >What does this thing do?
> It erases individual page, then writes there. To put it differently, in 
> your terminology, eraseblock size = writeblock size.

Neat.

> P.S. I actually missed the mailing list, this should have gone to the 
> MTD ML. So let's move there please.

As always.  You fuck it up, I get to fix it. ;)

Jörn

-- 
Homo Sapiens is a goal, not a description.
-- unknown

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

On Tue, 2005-09-20 at 16:36 +0200, Jörn Engel wrote:
> Image this block contains the used block bitmap of ext2.  All blocks
> handled with this bitmap are free, so the complete block contains
> zeroes.  Now we allocate a single block, writing a single one to the
> bitmap.  When writing, a power failure happens right after the erase:
> 
> Before:
> 00000000000
> 
> After:
> FFFFFFFFFF
> 
> Gee, every single block is used now.  Good thing this happened to the
> block bitmap, so we only lost some space on the fs.  Somewhere else,
> we'd have data corruption.
> 
> Any hard disk filesystem will suffer some kind of corruption in such a
> case.  They heavily depend on the non-existance of 4-6.

Oh OK, Got it, thanks.

> > P.S. I actually missed the mailing list, this should have gone to the 
> > MTD ML. So let's move there please.
> 
> As always.  You fuck it up, I get to fix it. ;)
hehe :-)

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> Example nodelist.c:
> len = ofs & (c->wbuf_pagesize - 1)
I suppose if you change this on
len = ofs % c->wbuf_pagesize;

everything should be fine.

There is a high chance that this is the only place like this.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> The chance is not too high  ;)  .
> 
> I have done some further search...
> Here one other patch for fix and the other is only for better look...
> 
> Now I will do a little test again...
> 
Oh, but you said JFFS2 worked with your dataflash for ages? How could it?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityuckiy wrote:
> Peter Menzebach wrote:
> 
>> The chance is not too high  ;)  .
>>
>> I have done some further search...
>> Here one other patch for fix and the other is only for better look...
>>
>> Now I will do a little test again...
>>
> Oh, but you said JFFS2 worked with your dataflash for ages? How could it?
> 
Andrew Victor have had some patches for fix, and I worked with this old 
version since last week: http://maxim.org.za/AT91RM9200/

He committed his changes to the mtd cvs at the beginning of this year.
And I now have upgraded to 2.6.13, and found some problems in the recent 
version.

Best regards
Peter


-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

Peter Menzebach wrote:
> -	if (!jffs2_can_mark_obsolete(c) && c->nextblock && (c->nextblock->free_size & (c->wbuf_pagesize-1))) {
> +	if (!jffs2_can_mark_obsolete(c) && c->nextblock && (c->nextblock->free_size % c->wbuf_pagesize)) {
[snip]
> -		uint32_t skip = c->nextblock->free_size & (c->wbuf_pagesize-1);
> +		uint32_t skip = c->nextblock->free_size % c->wbuf_pagesize;
[snip]
> -#ifdef CONFIG_JFFS2_FS_WRITEBUFFER
>  #define PAGE_DIV(x) ( ((unsigned long)(x) / (unsigned long)(c->wbuf_pagesize)) * (unsigned long)(c->wbuf_pagesize) )
>  #define PAGE_MOD(x) ( (unsigned long)(x) % (unsigned long)(c->wbuf_pagesize) )
> -#else
> -#define PAGE_DIV(x) ( (x) & (~(c->wbuf_pagesize - 1)) )
> -#define PAGE_MOD(x) ( (x) & (c->wbuf_pagesize - 1) )
> -#endif
Looks sane.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityuckiy wrote:
> Peter Menzebach wrote:
> 
>> -    if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
>> (c->nextblock->free_size & (c->wbuf_pagesize-1))) {
>> +    if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
>> (c->nextblock->free_size % c->wbuf_pagesize)) {
> 
> [snip]
> 
>> -        uint32_t skip = c->nextblock->free_size & (c->wbuf_pagesize-1);
>> +        uint32_t skip = c->nextblock->free_size % c->wbuf_pagesize;
> 
> [snip]
> 
>> -#ifdef CONFIG_JFFS2_FS_WRITEBUFFER
>>  #define PAGE_DIV(x) ( ((unsigned long)(x) / (unsigned 
>> long)(c->wbuf_pagesize)) * (unsigned long)(c->wbuf_pagesize) )
>>  #define PAGE_MOD(x) ( (unsigned long)(x) % (unsigned 
>> long)(c->wbuf_pagesize) )
>> -#else
>> -#define PAGE_DIV(x) ( (x) & (~(c->wbuf_pagesize - 1)) )
>> -#define PAGE_MOD(x) ( (x) & (c->wbuf_pagesize - 1) )
>> -#endif
> 
> Looks sane.
> 
Ok, one step forward. With these patches I can now copy a generated 
rootfs to flash, mount it, and read it without problems.

But with writing to an empty flash, there are still these problems. I 
can confirm, that the mtd interface works properly when reading, 
writing, erasing flash. For testing, I have replaced only the jffs2 with 
an old version, leaving the mtd part the same. There everything works fine.

My guess at the moment is, that the write buffering has some problems 
somewhere. I can see, that everytime a whole pagesize is written to 
flash. Is that intended?

Any ideas?

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

On Thu, 2005-09-15 at 11:14 +0200, Peter Menzebach wrote:
> My guess at the moment is, that the write buffering has some problems 
> somewhere. I can see, that everytime a whole pagesize is written to 
> flash. Is that intended?
This is OK. The whole goal of the write-buffer is to accumulate small
writes and to flush the whole page at once. It is assumed that the
minimal flash IO unit is c->wbuf_pagesize.

Could you please reproduce the problem, gather the log and send it
again, but with 2 notes:

1. please, output messages synchronously to a serial console (try dmesg
-n 8). When you use dmesg you see only the last output and loose
everything which does not fit the internal Linux output ring buffer.

2. AFAIU, after remount files disappear. Please, also send the log of
the second mount/

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityuckiy wrote:
> Peter Menzebach wrote:
> 
>> -    if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
>> (c->nextblock->free_size & (c->wbuf_pagesize-1))) {
>> +    if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
>> (c->nextblock->free_size % c->wbuf_pagesize)) {
> 
> [snip]
> 
>> -        uint32_t skip = c->nextblock->free_size & (c->wbuf_pagesize-1);
>> +        uint32_t skip = c->nextblock->free_size % c->wbuf_pagesize;
> 
> [snip]
> 
>> -#ifdef CONFIG_JFFS2_FS_WRITEBUFFER
>>  #define PAGE_DIV(x) ( ((unsigned long)(x) / (unsigned 
>> long)(c->wbuf_pagesize)) * (unsigned long)(c->wbuf_pagesize) )
>>  #define PAGE_MOD(x) ( (unsigned long)(x) % (unsigned 
>> long)(c->wbuf_pagesize) )
>> -#else
>> -#define PAGE_DIV(x) ( (x) & (~(c->wbuf_pagesize - 1)) )
>> -#define PAGE_MOD(x) ( (x) & (c->wbuf_pagesize - 1) )
>> -#endif
> 
> Looks sane.
> 
Do you put this into cvs or shall I submit it later again?

Best regards
Peter

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> Do you put this into cvs or shall I submit it later again?
No I didn't. I would like to commit this lated, together with other 
DataFlash fixes, unless this is really important to you to commit this now.


-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityuckiy]

BTW, try to disable EBS. It is a new feature and may have a bug.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Peter Menzebach wrote:
> Hi,
> I loose files, which I write on a jffs2 filesystem.
> 
> I start with a freshly erased mtd partition. Then I start the following 
> sequence:
> 
> mount /conf
> echo aaa > /conf/aaa
> echo aaa > /conf/aaa
> umount /conf
> mount /conf
> 
> After remounting, the file gets lost (I think because the GC removes the 
> inode). I use kernel 2.6.13 with mtd out of cvs of today.
> The machine is a AT91RM9200 with a serial dataflash. I used already
> jffs2 (most time readonly) as rootfs here for months without problems. 
> The only specialty is, that the dataflash has a unfamiliar erase size of 
> 8448 bytes.
> 
OK guys,
here we are with a patch for it. I made several tests now with
wbuf->pagesize of 1056 and 8*1056 and erase size of 8*1056 and it looks
good so far.

Open is now the question, if we should implement a possibility to adjust
the wbuf_pagesize independently from the jffs2 sectorsize. For my
purposes I can live with a wbuf_size = jffs2 sector_size = 8*1056, but
if we find a nice solution, I would implement it...

IMHO the cleanest solution would be, that the dataflash/mtd layer
reports it's true erase_size = smallest write size = 528/1056 bytes, and
should not make a further guess about later use.

JFFS2 now recognizes, that it can't work with this parameters, and
should adjust it to multiples of the size internally.

Please comment to patch and how to proceed...

Best regards
Peter

P.S. Sry I can't manage to send the patch as attachment. (it get's 
rejected by the mailing list)

diff -urN mtda/fs/jffs2/nodelist.c mtdb/fs/jffs2/nodelist.c
--- mtda/fs/jffs2/nodelist.c	2005-09-22 13:47:32.000000000 +0200
+++ mtdb/fs/jffs2/nodelist.c	2005-09-22 13:44:50.000000000 +0200
@@ -412,7 +412,7 @@

  	/* Calculate how many bytes were already checked */
  	ofs = ref_offset(ref) + sizeof(struct jffs2_raw_inode);
-	len = ofs & (c->wbuf_pagesize - 1);
+	len = ofs % c->wbuf_pagesize;
  	if (likely(len))
  		len = c->wbuf_pagesize - len;

diff -urN mtda/fs/jffs2/os-linux.h mtdb/fs/jffs2/os-linux.h
--- mtda/fs/jffs2/os-linux.h	2005-09-22 13:47:36.000000000 +0200
+++ mtdb/fs/jffs2/os-linux.h	2005-09-22 13:44:53.000000000 +0200
@@ -65,8 +65,12 @@

  #define jffs2_is_readonly(c) (OFNI_BS_2SFFJ(c)->s_flags & MS_RDONLY)

-#ifndef CONFIG_JFFS2_FS_WRITEBUFFER
+/*
  #define SECTOR_ADDR(x) ( ((unsigned long)(x) & ~(c->sector_size-1)) )
+*/
+#ifndef CONFIG_JFFS2_FS_WRITEBUFFER
+#define SECTOR_ADDR(x) ( (((unsigned long)(x) / c->sector_size) * 
c->sector_size) )
+

  #ifdef CONFIG_JFFS2_SUMMARY
  #define jffs2_can_mark_obsolete(c) (0)
diff -urN mtda/fs/jffs2/scan.c mtdb/fs/jffs2/scan.c
--- mtda/fs/jffs2/scan.c	2005-09-22 13:47:32.000000000 +0200
+++ mtdb/fs/jffs2/scan.c	2005-09-22 13:44:50.000000000 +0200
@@ -233,12 +233,12 @@
  		c->nextblock->dirty_size = 0;
  	}
  #ifdef CONFIG_JFFS2_FS_WRITEBUFFER
-	if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
(c->nextblock->free_size & (c->wbuf_pagesize-1))) {
+	if (!jffs2_can_mark_obsolete(c) && c->nextblock && 
(c->nextblock->free_size % c->wbuf_pagesize)) {
  		/* If we're going to start writing into a block which already
  		   contains data, and the end of the data isn't page-aligned,
  		   skip a little and align it. */

-		uint32_t skip = c->nextblock->free_size & (c->wbuf_pagesize-1);
+		uint32_t skip = c->nextblock->free_size % c->wbuf_pagesize;

  		D1(printk(KERN_DEBUG "jffs2_scan_medium(): Skipping %d bytes in 
nextblock to ensure page alignment\n",
  			  skip));
diff -urN mtda/fs/jffs2/wbuf.c mtdb/fs/jffs2/wbuf.c
--- mtda/fs/jffs2/wbuf.c	2005-09-22 13:47:32.000000000 +0200
+++ mtdb/fs/jffs2/wbuf.c	2005-09-22 13:44:50.000000000 +0200
@@ -28,12 +28,12 @@
  static unsigned char *brokenbuf;
  #endif

+#define PAGE_DIV(x) ( ((unsigned long)(x) / (unsigned 
long)(c->wbuf_pagesize)) * (unsigned long)(c->wbuf_pagesize) )
+#define PAGE_MOD(x) ( (unsigned long)(x) % (unsigned 
long)(c->wbuf_pagesize) )
+
  /* max. erase failures before we mark a block bad */
  #define MAX_ERASE_FAILURES 	2

-/* two seconds timeout for timed wbuf-flushing */
-#define WBUF_FLUSH_TIMEOUT	2 * HZ
-
  struct jffs2_inodirty {
  	uint32_t ino;
  	struct jffs2_inodirty *next;
@@ -434,7 +434,7 @@
  	   if we have a switch to next page, we will not have
  	   enough remaining space for this.
  	*/
-	if (pad && !jffs2_dataflash(c)) {
+	if (pad ) {
  		c->wbuf_len = PAD(c->wbuf_len);

  		/* Pad with JFFS2_DIRTY_BITMASK initially.  this helps out ECC'd NOR
@@ -483,9 +483,9 @@
  	}

  	spin_lock(&c->erase_completion_lock);
-
+		
  	/* Adjust free size of the block if we padded. */
-	if (pad && !jffs2_dataflash(c)) {
+	if (pad) {
  		struct jffs2_eraseblock *jeb;

  		jeb = &c->blocks[c->wbuf_ofs / c->sector_size];
@@ -602,15 +602,6 @@

  	return ret;
  }
-
-#ifdef CONFIG_JFFS2_FS_WRITEBUFFER
-#define PAGE_DIV(x) ( ((unsigned long)(x) / (unsigned 
long)(c->wbuf_pagesize)) * (unsigned long)(c->wbuf_pagesize) )
-#define PAGE_MOD(x) ( (unsigned long)(x) % (unsigned 
long)(c->wbuf_pagesize) )
-#else
-#define PAGE_DIV(x) ( (x) & (~(c->wbuf_pagesize - 1)) )
-#define PAGE_MOD(x) ( (x) & (c->wbuf_pagesize - 1) )
-#endif
-
  int jffs2_flash_writev(struct jffs2_sb_info *c, const struct kvec 
*invecs, unsigned long count, loff_t to, size_t *retlen, uint32_t ino)
  {
  	struct kvec outvecs[3];
@@ -654,7 +645,7 @@
  	   erase block. Anything else, and you die.
  	   New block starts at xxx000c (0-b = block header)
  	*/
-	if (SECTOR_ADDR(to) != SECTOR_ADDR(c->wbuf_ofs)) {
+	if (PAGE_DIV(to) != PAGE_DIV(c->wbuf_ofs)) {
  		/* It's a write to a new block */
  		if (c->wbuf_len) {
  			D1(printk(KERN_DEBUG "jffs2_flash_writev() to 0x%lx causes flush of 
wbuf at 0x%08x\n", (unsigned long)to, c->wbuf_ofs));
@@ -905,7 +896,7 @@
  		goto exit;

  	/* if we read in a different block, return */
-	if (SECTOR_ADDR(ofs) != SECTOR_ADDR(c->wbuf_ofs))
+	if (PAGE_DIV(ofs) != PAGE_DIV(c->wbuf_ofs))
  		goto exit;

  	if (ofs >= c->wbuf_ofs) {
@@ -1104,7 +1095,7 @@
  		return 0;

  	if (!c->mtd->block_markbad)
-		return 1; // What else can we do?
+		return 1;

  	D1(printk(KERN_WARNING "jffs2_write_nand_badblock(): Marking bad 
block at %08x\n", bad_offset));
  	ret = c->mtd->block_markbad(c->mtd, bad_offset);
diff -urN mtda/drivers/mtd/mtdpart.c mtdb/drivers/mtd/mtdpart.c
--- mtda/drivers/mtd/mtdpart.c	2005-02-08 18:11:13.000000000 +0100
+++ mtdb/drivers/mtd/mtdpart.c	2005-09-14 13:20:18.000000000 +0200
@@ -465,8 +465,9 @@
  		if (slave->offset == MTDPART_OFS_APPEND)
  			slave->offset = cur_offset;
  		if (slave->offset == MTDPART_OFS_NXTBLK) {
-			u_int32_t emask = master->erasesize-1;
-			slave->offset = (cur_offset + emask) & ~emask;
+			slave->offset = cur_offset;
+			if ((cur_offset % master->erasesize) != 0)
+				slave->offset = ((cur_offset / master->erasesize) + 1) * 
master->erasesize;
  			if (slave->offset != cur_offset) {
  				printk(KERN_NOTICE "Moving partition %d: "
  				       "0x%08x -> 0x%08x\n", i,

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

On Thu, 2005-09-22 at 14:30 +0200, Peter Menzebach wrote:
> OK guys,
> here we are with a patch for it. I made several tests now with
> wbuf->pagesize of 1056 and 8*1056 and erase size of 8*1056 and it looks
> good so far.
> 
> Open is now the question, if we should implement a possibility to adjust
> the wbuf_pagesize independently from the jffs2 sectorsize. For my
> purposes I can live with a wbuf_size = jffs2 sector_size = 8*1056, but
> if we find a nice solution, I would implement it...
You can leave. But do you want to have more CPU load, more Flash IO,
faster flash wear, more lost data after Unclean reboot?

If no, make wbuf_size = DataFlash page size. And IMO, it is *MUST* for
CVS commit.

> IMHO the cleanest solution would be, that the dataflash/mtd layer
> reports it's true erase_size = smallest write size = 528/1056 bytes, and
> should not make a further guess about later use.
I thinks the cleanest if it reports *both* sizes. Let the upper (above
MTD) layers decide what erasesize to use.

> P.S. Sry I can't manage to send the patch as attachment. (it get's 
> rejected by the mailing list)
That's usual thing, unfortunately.


> -#ifndef CONFIG_JFFS2_FS_WRITEBUFFER
> +/*
>   #define SECTOR_ADDR(x) ( ((unsigned long)(x) & ~(c->sector_size-1)) )
> +*/
What for this comment ?

>   	struct kvec outvecs[3];
> @@ -654,7 +645,7 @@
>   	   erase block. Anything else, and you die.
>   	   New block starts at xxx000c (0-b = block header)
>   	*/
> -	if (SECTOR_ADDR(to) != SECTOR_ADDR(c->wbuf_ofs)) {
> +	if (PAGE_DIV(to) != PAGE_DIV(c->wbuf_ofs)) {
Why this change ?

>   	if (!c->mtd->block_markbad)
> -		return 1; // What else can we do?
> +		return 1;
Why this change ?

> 
>   	D1(printk(KERN_WARNING "jffs2_write_nand_badblock(): Marking bad 
> block at %08x\n", bad_offset));
>   	ret = c->mtd->block_markbad(c->mtd, bad_offset);
> diff -urN mtda/drivers/mtd/mtdpart.c mtdb/drivers/mtd/mtdpart.c
> --- mtda/drivers/mtd/mtdpart.c	2005-02-08 18:11:13.000000000 +0100
> +++ mtdb/drivers/mtd/mtdpart.c	2005-09-14 13:20:18.000000000 +0200
> @@ -465,8 +465,9 @@
I think this is better to send as a distinct patch.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Artem B. Bityutskiy wrote:
> On Thu, 2005-09-22 at 14:30 +0200, Peter Menzebach wrote:
> 
>>OK guys,
>>here we are with a patch for it. I made several tests now with
>>wbuf->pagesize of 1056 and 8*1056 and erase size of 8*1056 and it looks
>>good so far.
>>
>>Open is now the question, if we should implement a possibility to adjust
>>the wbuf_pagesize independently from the jffs2 sectorsize. For my
>>purposes I can live with a wbuf_size = jffs2 sector_size = 8*1056, but
>>if we find a nice solution, I would implement it...
> 
> You can leave. But do you want to have more CPU load, more Flash IO,
> faster flash wear, more lost data after Unclean reboot?
> 
> If no, make wbuf_size = DataFlash page size. And IMO, it is *MUST* for
> CVS commit.
> 
But in this case mtd_info structure has to be changed/extended. 
Otherwise the information does not get to the jffs2 layer. How 
should/could this be managed?

> 
>>IMHO the cleanest solution would be, that the dataflash/mtd layer
>>reports it's true erase_size = smallest write size = 528/1056 bytes, and
>>should not make a further guess about later use.
> 
Problem is here, how does the user get the information about the erase 
size used by jffs2?

> I thinks the cleanest if it reports *both* sizes. Let the upper (above
> MTD) layers decide what erasesize to use.
> 
Are these two values typically existing? In the dataflash case, it's 
only one value.

If they make sense, we have a write_chunk_size, and an erasesize size...

>>-#ifndef CONFIG_JFFS2_FS_WRITEBUFFER
>>+/*
>>  #define SECTOR_ADDR(x) ( ((unsigned long)(x) & ~(c->sector_size-1)) )
>>+*/
> 
> What for this comment ?
gets out (I commented out instead of removing)

>>-	if (SECTOR_ADDR(to) != SECTOR_ADDR(c->wbuf_ofs)) {
>>+	if (PAGE_DIV(to) != PAGE_DIV(c->wbuf_ofs)) {
> Why this change ?
c->sector_size != c->wbuf_ofs

>>  	if (!c->mtd->block_markbad)
>>-		return 1; // What else can we do?
>>+		return 1;
> 
> Why this change ?
C++ comment

> 
>>diff -urN mtda/drivers/mtd/mtdpart.c mtdb/drivers/mtd/mtdpart.c
>>--- mtda/drivers/mtd/mtdpart.c	2005-02-08 18:11:13.000000000 +0100
>>+++ mtdb/drivers/mtd/mtdpart.c	2005-09-14 13:20:18.000000000 +0200
>>@@ -465,8 +465,9 @@
> 
> I think this is better to send as a distinct patch.
> 
OK


Best regards
Peter
-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Peter Menzebach wrote:
> But in this case mtd_info structure has to be changed/extended. 
> Otherwise the information does not get to the jffs2 layer. How 
> should/could this be managed?
Yes. As a hacky solution you may use mtd->oobblock to inform about 
DataFlash page size.

>>> IMHO the cleanest solution would be, that the dataflash/mtd layer
>>> reports it's true erase_size = smallest write size = 528/1056 bytes, and
>>> should not make a further guess about later use.
>>
>>
> Problem is here, how does the user get the information about the erase 
> size used by jffs2?
He detects that the flash is DataFlash (mtd->type == MTD_DATAFLASH). 
Afterwards he knows, that
1. mtd->erasesize is the size of DataFlash block
2. mtd->oobblock is the size of DataFlash page

May be this is not the best solution. Consult tglx1.

> Are these two values typically existing? In the dataflash case, it's 
> only one value.
Sorry, didn't get it. I mean, MTD provide information about both 
DataFlash block size and DataFlash page size. That's obvious.

>>> -    if (SECTOR_ADDR(to) != SECTOR_ADDR(c->wbuf_ofs)) {
>>> +    if (PAGE_DIV(to) != PAGE_DIV(c->wbuf_ofs)) {
>>
>> Why this change ?
> 
> c->sector_size != c->wbuf_ofs
Of course.
c->sector_size = eraseblock size
c->wbuf_ofs - the absolute offset from the beginning of the partition 
where wbuf will be written.

The if() just check if JFFS2 switched to another eraseblock (because we 
are at the end of eraseblock and there is anyway too few space to write 
any node). So, SECTOR_ADDR() is absolutely correct. What's wrong?

> 
>>>      if (!c->mtd->block_markbad)
>>> -        return 1; // What else can we do?
>>> +        return 1;
>>
>>
>> Why this change ?
> 
> C++ comment
Well, it does not relate to the DataFlash stuff. It is generally good 
idea to send this as a distinct patch. But don't want to be an ass, so, 
let it be :-) Although dwmw2 is usually hit the celling when I edit his 
comments and I would prefer if you just change it to /* */ ... :-)

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Andrew Victor]

hi,

> > Problem is here, how does the user get the information about the erase 
> > size used by jffs2?
> He detects that the flash is DataFlash (mtd->type == MTD_DATAFLASH). 
> Afterwards he knows, that
> 1. mtd->erasesize is the size of DataFlash block
> 2. mtd->oobblock is the size of DataFlash page

No.  You can't change mtd->erasesize.
There are other existing applications that access the raw MTD layer and
want the real erasesize.


> Sorry, didn't get it. I mean, MTD provide information about both 
> DataFlash block size and DataFlash page size. That's obvious.

This magic minimum block size could always be calculated in JFFS2 (in
jffs2_dataflash_setup).  It is a JFFS2-specific value.
Just calculate an 'N' where:
   (N * mtd->erasesize) >= jffs2_minimum_block_size
and
   (mtd->size % (N * mtd->erasesize)) == 0


Regards,
  Andrew Victor

Re: data loss on jffs2 filesystem on dataflash

[Artem B. Bityutskiy]

Andrew Victor wrote:
> No.  You can't change mtd->erasesize.
> There are other existing applications that access the raw MTD layer and
> want the real erasesize.
Ok guys, I don't actually know the best solution. Just think up 
something. I don't care about MTD as much, sorry :-) You may joint the 
IRC and talk to tglx1 and others.

My objections are:
1. jffs2_dataflash_setup() should set c->wbuf_pagesize = DataFlash page
2. c->sector_size should be set to DataFlash block size
3. The way how you find out DataFlash page and DataFlash block size must 
work on any other DataFlash.
4. When JFFS2 erases an eraseblock, it must erase c->sector_size bytes, 
not c->wbuf_pagesize bytes/

I think that's fair, isn't it?

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Andrew Victor wrote:
> hi,
Hi,
> 
>>>Problem is here, how does the user get the information about the erase 
>>>size used by jffs2?
>>
>>He detects that the flash is DataFlash (mtd->type == MTD_DATAFLASH). 
>>Afterwards he knows, that
>>1. mtd->erasesize is the size of DataFlash block
>>2. mtd->oobblock is the size of DataFlash page
> 
> 
> No.  You can't change mtd->erasesize.
> There are other existing applications that access the raw MTD layer and
> want the real erasesize.
I agree. The erase size reported to the mtd layer should be always the 
physical erase size (528/1056).

>>Sorry, didn't get it. I mean, MTD provide information about both 
>>DataFlash block size and DataFlash page size. That's obvious.
> 
> 
> This magic minimum block size could always be calculated in JFFS2 (in
> jffs2_dataflash_setup).  It is a JFFS2-specific value.
> Just calculate an 'N' where:
>    (N * mtd->erasesize) >= jffs2_minimum_block_size
> and
>    (mtd->size % (N * mtd->erasesize)) == 0

That would possible.
Set jffs2->wbuf_size to erasesize and calculate the jffs2->sector size 
as you outlined...

Best regards
Peter


-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Andrew Victor wrote:

> This magic minimum block size could always be calculated in JFFS2 (in
> jffs2_dataflash_setup).  It is a JFFS2-specific value.
> Just calculate an 'N' where:
>    (N * mtd->erasesize) >= jffs2_minimum_block_size
> and
>    (mtd->size % (N * mtd->erasesize)) == 0

How about this in jffs2_dataflash_setup:

--- /arm/src/mtd/fs/jffs2/wbuf.c	2005-09-22 13:33:42.000000000 +0200
+++ wbuf.c	2005-09-22 17:52:17.000000000 +0200
@@ -1204,14 +1197,28 @@
  	
  	/* Initialize write buffer */
  	init_rwsem(&c->wbuf_sem);
-	c->wbuf_pagesize = c->sector_size;
+	c->wbuf_pagesize = c->mtd->erasesize;
+
+	c->sector_size = ((JFFS2_MIN_SECTOR_SIZE + c->mtd->erasesize - 1) /
+			  c->mtd->erasesize) * c->mtd->erasesize;
+
+	while ((5*c->sector_size < c->mtd->size) &&
+		((c->mtd->size % c->sector_size) != 0))
+	{
+		c->sector_size+=c->mtd->erasesize;
+	}
+	if ((c->mtd->size % c->sector_size) != 0) {
+		printk(KERN_WARNING "JFFS2 no suitable sector size found\n");
+		return (1);
+	};
+	
  	c->wbuf_ofs = 0xFFFFFFFF;

  	c->wbuf = kmalloc(c->wbuf_pagesize, GFP_KERNEL);
  	if (!c->wbuf)
  		return -ENOMEM;

-	printk(KERN_INFO "JFFS2 write-buffering enabled (%i)\n", 
c->wbuf_pagesize);
+	printk(KERN_INFO "JFFS2 write-buffering enabled (%i) erasesize 
(%i)\n", c->wbuf_pagesize, c->sector_size);

  	return 0;
  }


-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1

Re: data loss on jffs2 filesystem on dataflash

[Peter Menzebach]

Peter Menzebach wrote:
forget the last patch, I made up my mind. If I do not find a good sector 
size, I should resize the size of flash for jffs2. See below.

The question is, if the call for jffs2_flash_setup should be moved 
before the size checks in jffs2_do_fill_super?

Best regards
Peter


--- /arm/src/mtd/fs/jffs2/wbuf.c	2005-09-22 13:33:42.000000000 +0200
+++ wbuf.c	2005-09-22 18:52:17.000000000 +0200
@@ -1200,18 +1193,37 @@
  }

  int jffs2_dataflash_setup(struct jffs2_sb_info *c) {
+	int min_sector_size;
  	c->cleanmarker_size = 0;		/* No cleanmarkers needed */
  	
  	/* Initialize write buffer */
  	init_rwsem(&c->wbuf_sem);
-	c->wbuf_pagesize = c->sector_size;
+	c->wbuf_pagesize = c->mtd->erasesize;
+
+	min_sector_size = ((JFFS2_MIN_SECTOR_SIZE + c->mtd->erasesize - 1) / 
c->mtd->erasesize) * c->mtd->erasesize;
+
+	c->sector_size = min_sector_size;
+
+	c->flash_size = c->mtd->size;
+	
+	while ((c->sector_size < (2 * min_sector_size)) && (c->flash_size % 
c->sector_size != 0)) {
+		c->sector_size+=c->mtd->erasesize;
+	}
+
+	if ((c->flash_size % c->sector_size) != 0) {
+		c->sector_size = min_sector_size;
+		c->flash_size = (c->flash_size / min_sector_size) * min_sector_size;
+		printk(KERN_INFO "JFFS2 flash size and erase size adjusted size %dKiB 
erasesize %dB\n", c->flash_size/1024, c->sector_size);
+	};
+	
+	
  	c->wbuf_ofs = 0xFFFFFFFF;

  	c->wbuf = kmalloc(c->wbuf_pagesize, GFP_KERNEL);
  	if (!c->wbuf)
  		return -ENOMEM;

-	printk(KERN_INFO "JFFS2 write-buffering enabled (%i)\n", 
c->wbuf_pagesize);
+	printk(KERN_INFO "JFFS2 write-buffering enabled (%i) erasesize 
(%i)\n", c->wbuf_pagesize, c->sector_size);

  	return 0;
  }

-- 
Peter Menzebach
Menzebach und Wolff IT-Consulting GbR
Phone +49 751 355 387 1