jffs2 with sync burst mode

jffs2 with sync burst mode

[Konstantin Kletschke]

Hi people!

These days I have time again for my embedded system to investigate what
weird shit is happening here with the probplems introduced trying to
configure the Flash device to run in sync burst mode.
It is an intel 28F128K3 device with a motorola i.MX cpu.

I now set up a debug environment: I am running a classical kernel
locatedt in FLASH which is succesfully copied and decompressed into
SDRAM. It mounts its root via nfs from server and from there I mount
/dev/mtdblockX for testing purposes with jffs2 and mtd debug turned on.

First weird thing is, reading writing from the device works perfect.
I diffed original and copy and diff putted out no difference. I copied
from flash back into root, copied the copy and diffed on the PC also, no
differences. I tried small files, big files (i.e. libc, 1.5MB) and tried
also /dev/urandom.

Then I populated the rootfs in /dev/mtdblock with the rootfs, booted
with nfs-root again and mounted:

sh-2.05b# mount -t jffs2 /dev/mtdblock3 /mnt
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0004206c: Read
0x9acd7a6e, calculated 0x7fb7dc11
Name for which CRC failed is (now) 'ld-uClibc-0.ø', ino #248
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0009a2b0: Read
0xb08c7dea, calculated 0xd3eba660
Name for which CRC failed is (now) 'modules.', ino #285
jffs2_scan_dirent_node(): Name CRC failed on node at 0x001126e8: Read
0x839452cb, calculated 0xa6182a48
Name for which CRC failed is (now) 'openssh-host-key­ï.sh', ino #376
jffs2_scan_dirent_node(): Name CRC failed on node at 0x002c97ec: Read
0xb08c7dea, calculated 0x0bb60059
Name for which CRC failed is (now) 'modules.ieee', ino #541
jffs2_scan_dirent_node(): Name CRC failed on node at 0x002c9828: Read
0xa33d20d0, calculated 0x2ed7968c
Name for which CRC failed is (now) 'modules.ieee1394µ', ino #0
sh-2.05b# umount /mnt

I did "cat /dev/mtdblock3 > file1" into nfs-root (as a side note, I did
the same again with dd bs=1024k:
konsti@synertronixx3:/usr/src/debian_arm/ > md5sum ddmtdblock3 
e871d71c52d93c1cafb4a8f4ff47a741  ddmtdblock3
konsti@synertronixx3:/usr/src/debian_arm/ > md5sum mtdblock3 
e871d71c52d93c1cafb4a8f4ff47a741  mtdblock3
).

This is emacs in hexl-mode on these files:

jffs2_scan_dirent_node(): Name CRC failed on node at 0x0004206c: Read
0x9acd7a6e, calculated 0x7fb7dc11
Name for which CRC failed is (now) 'ld-uClibc-0.ø', ino #248

00042070: 3b00 0000 7939 3f2d 0700 0000 f600 0000  ;...y9?-........                  
00042080: f800 0000 16a9 ef41 1308 0000 06d6 16d5  .......A........                  
00042090: 6e7a cd9a 6c64 2d75 436c 6962 632d 302e  nz..ld-uClibc-0.                  
000420a0: 392e 3236 2e73 6fff 8519 02c0 3d08 0000  9.26.so.....=...                  
000420b0: b345 d380 f800 0000 0100 0000 ed81 0000  .E..............

ASCII code of the "crossed o" is f8, second line, first byte.

jffs2_scan_dirent_node(): Name CRC failed on node at 0x0009a2b0: Read
0xb08c7dea, calculated 0xd3eba660
Name for which CRC failed is (now) 'modules.', ino #285

0009a2b0: 8519 01e0 3b00 0000 7939 3f2d 1701 0000  ....;...y9?-....                  
0009a2c0: 1b01 0000 1d01 0000 fbb0 ef41 1308 0000  ...........A....                  
0009a2d0: 2c46 14cc ea7d 8cb0 6d6f 6475 6c65 732e  ,F...}..modules.                  
0009a2e0: 6965 6565 3133 3934 6d61 70ff 8519 02c0  ieee1394map.....                  
0009a2f0: 8d00 0000 915a b293 1d01 0000 0100 0000  .....Z..........

What should become "ieee1394map" has become nothing, probably second
line, the 0000.

jffs2_scan_dirent_node(): Name CRC failed on node at 0x001126e8: Read
0x839452cb, calculated 0xa6182a48
Name for which CRC failed is (now) 'openssh-host-key­ï.sh', ino #376

001126e0: 0418 0017 1272 f3ff 8519 01e0 3e00 0000  .....r......>...                  
001126f0: 4bc9 e11a 0a00 0000 7601 0000 7801 0000  K.......v...x...                  
00112700: 9aad ef41 1608 0000 2312 0de0 cb52 9483  ...A....#....R..                  
00112710: 6f70 656e 7373 682d 686f 7374 2d6b 6579  openssh-host-key                  
00112720: 6765 6e2e 7368 ffff 8519 02c0 2801 0000  gen.sh......(...                  
00112730: 91d9 c5e8 7801 0000 0100 0000 ed81 0000  ....x...........

The "i" with two dots is ef third byte, second line.

Name for which CRC failed is (now) 'modules.ieee', ino #541
jffs2_scan_dirent_node(): Name CRC failed on node at 0x002c9828: Read
0xa33d20d0, calculated 0x2ed7968c

002c9820: 3133 3934 6d61 70ff 8519 01e0 4000 0000  1394map.....@...                  
002c9830: e41e 0191 e401 0000 4700 0000 0000 0000  ........G.......                  
002c9840: b500 0000 1800 367c 6ede 69d5 d020 3da3  ......6|n.i.. =.                  
002c9850: 6d6f 6475 6c65 732e 6965 6565 3133 3934  modules.ieee1394                  
002c9860: 6d61 702e 7465 6d70 8519 02c0 4400 0000  map.temp....D...                  
002c9870: 1dfb f798 1e02 0000 0100 0000 a481 0000  ................

Again kernel read "00" (line above third or fourth byte) instead of
1394map.temp.

jffs2_scan_dirent_node(): Name CRC failed on node at 0x002c9828: Read
0xa33d20d0, calculated 0x2ed7968c
Name for which CRC failed is (now) 'modules.ieee1394µ', ino #0

002c9820: 3133 3934 6d61 70ff 8519 01e0 4000 0000  1394map.....@...                  
002c9830: e41e 0191 e401 0000 4700 0000 0000 0000  ........G.......                  
002c9840: b500 0000 1800 367c 6ede 69d5 d020 3da3  ......6|n.i.. =.                  
002c9850: 6d6f 6475 6c65 732e 6965 6565 3133 3934  modules.ieee1394                  
002c9860: 6d61 702e 7465 6d70 8519 02c0 4400 0000  map.temp....D...                  
002c9870: 1dfb f798 1e02 0000 0100 0000 a481 0000  ................

The µ is B5, second line, first byte.

The Kernel is completely XIP unaware or such, it is 
Linux 192.168.1.133 2.6.11-rc2-mm1-imx1 #169 Wed Mar 9 12:49:57 CET 2005
armv4tl GNU/Linux
The error messages are the same if I reboot or try to mount the fs as
root-fs initially also.

Where is jffs2 stumbling upon?
Doesn't jffs cope correctly with cancelled or wrapped around word reads?
But why is Kernel correctly copied then and 6 files broken and not a
dozen? Why could this be unreproducable on less complex filesystem
structures?

Kind Regards, Konsti


-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

> Hi people!
>
> These days I have time again for my embedded system to investigate what
> weird shit is happening here with the probplems introduced trying to
> configure the Flash device to run in sync burst mode.
> It is an intel 28F128K3 device with a motorola i.MX cpu.
>
> I now set up a debug environment: I am running a classical kernel
> locatedt in FLASH which is succesfully copied and decompressed into
> SDRAM. It mounts its root via nfs from server and from there I mount
> /dev/mtdblockX for testing purposes with jffs2 and mtd debug turned on.
>
> First weird thing is, reading writing from the device works perfect.
> I diffed original and copy and diff putted out no difference. I copied
> from flash back into root, copied the copy and diffed on the PC also, 

First of all we ask to try the latest MTD snapshot. Did you do that?

Just general suggestions:

Try to mount empty filesystem then copy your tree there by "cp -r -a", 
umount, mount. This will exclude errors during JFFS2 image 
creation/flashing.

Try to use mtdram device.

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 13:15 +0000 schrieb Artem B. Bityuckiy:

> First of all we ask to try the latest MTD snapshot. Did you do that?

I did not but testet now: same picture.
 
> Try to mount empty filesystem then copy your tree there by "cp -r -a", 
> umount, mount. This will exclude errors during JFFS2 image 
> creation/flashing.

The image is dumped from another embedded system running fine
with this image, it was halted and then dumped with the bootloader onto
PC. Its no difference if this image is then flashed or dd'ed

> Try to use mtdram device.

Well if I figure out how to do this and what to do to test I will try.

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 13:15 +0000 schrieb Artem B. Bityuckiy:

> Try to mount empty filesystem then copy your tree there by "cp -r -a", 
> umount, mount. This will exclude errors during JFFS2 image 
> creation/flashing.

Ok. I did following:
I copied the root-fs directory into the nfs export so I can reach it
from the embedded system. I put a tar with /dev into it also.

I erased /dev/mtdblock3 with bootloader and
bootet with nfsroot again. I mounted empty /dev/mtdblock3, cd'ed into it
and did "cp -a -r /usr/src/root/* ."

sh-2.05b# mount -t jffs2 /dev/mtdblock3 /mnt
jffs2_scan_dirent_node(): Name CRC failed on node at 0x003a52c8: Read 0x839452cb, calculated 0x262752af
Name for which CRC failed is (now) 'openssh-host-key', ino #350
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0043566c: Read 0x9fcfc7a9, calculated 0x445e56d8
', ino #269ich CRC failed is (now) 'libpthread-0
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0047aa4c: Read 0xb9570647, calculated 0x6ac2e146
Name for which CRC failed is (now) 'linux-wlan-nÒ', ino #210

i.e.:

003a52c0: 0000 0000 ff45 79b6 8519 01e0 3e00 0000  .....Ey.....>... 
003a52d0: 4bc9 e11a 4901 0000 1600 0000 5e01 0000  K...I.......^... 
003a52e0: 9900 0000 1608 0000 333a 7af0 cb52 9483  ........3:z..R.. 
003a52f0: 6f70 656e 7373 682d 686f 7374 2d6b 6579  openssh-host-key 
003a5300: 6765 6e2e 7368 ffff 8519 02c0 2801 0000  gen.sh......(... 
003a5310: 91d9 c5e8 5e01 0000 0200 0000 ed81 0000  ....^........... 
003a5320: 0000 0000 9902 0000 9900 0000 9900 0000  ................ 

00435660: 0000 0000 0000 0000 b882 2ec7 8519 01e0  ................ 
00435670: 3c00 0000 c001 e8b0 fb00 0000 1300 0000  <............... 
00435680: 0d01 0000 9400 0000 1408 0000 55ff 2d7c  ............U.-| 
00435690: a9c7 cf9f 6c69 6270 7468 7265 6164 2d30  ....libpthread-0 
004356a0: 2e39 2e32 362e 736f 8519 02c0 a906 0000  .9.26.so........ 
004356b0: 4a1e 6fb8 0d01 0000 0200 0000 a481 0000  J.o............. 
004356c0: 0000 0000 0010 0000 9400 0000 9400 0000  ................ 

Filenames truncated by 0000?

0047aa40: 0000 0000 0000 0000 3d99 4484 8519 01e0  ........=.D..... 
0047aa50: 3c00 0000 c001 e8b0 d100 0000 0200 0000  <............... 
0047aa60: d200 0000 9100 0000 1408 0000 56db cdde  ............V... 
0047aa70: 4706 57b9 6c69 6e75 782d 776c 616e 2d6e  G.W.linux-wlan-n 
0047aa80: 672d 7072 652d 7570 8519 02c0 8006 0000  g-pre-up........ 
0047aa90: fefe 5565 d200 0000 0200 0000 ed81 0000  ..Ue............ 
0047aaa0: 0000 0000 0010 0000 9100 0000 9100 0000  ................ 
0047aab0: 9100 0000 0000 0000 3c06 0000 0010 0000  ........<....... 

The `O is d2, third line first byte.

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

On Thu, 10 Mar 2005, Konstantin Kletschke wrote:
> The image is dumped from another embedded system running fine
> with this image, it was halted and then dumped with the bootloader onto
> PC. Its no difference if this image is then flashed or dd'ed

OK, why does this mean JFFS2 is guilty of this? Why don't you suspect your 
drivers? Why don't you suspect that you flashed your (say, correct) image 
incorrectly?

> Well if I figure out how to do this and what to do to test I will try.
Just configure your kernel properly: switch on the "Test driver using RAM" 
(MTD_MTDRAM) option in your kernel configuration. Type "modprobe mtdram" 
if you have configured it as the kernel module.

If you do this your JFFS2 will work over emulated 4MiB in-RAM flash and 
this will help you to exclude driver problems.

HTH, Artem.

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

> 00435660: 0000 0000 0000 0000 b882 2ec7 8519 01e0  ................
> 00435670: 3c00 0000 c001 e8b0 fb00 0000 1300 0000  <...............
            ^
            node length
> 00435680: 0d01 0000 9400 0000 1408 0000 55ff 2d7c  ............U.-|
                                ^
                                name length
> 00435690: a9c7 cf9f 6c69 6270 7468 7265 6164 2d30  ....libpthread-0
            ^^^^ ^^^^
            Name CRC
> 004356a0: 2e39 2e32 362e 736f 8519 02c0 a906 0000  .9.26.so........
> 004356b0: 4a1e 6fb8 0d01 0000 0200 0000 a481 0000  J.o.............
> 004356c0: 0000 0000 0010 0000 9400 0000 9400 0000  ................
Where has this dump come from?

This dump looks good:
* the length of node is 0x3c
* the length of the name is 0x14
* the name CRC - don't know, didn't calculate.

Looks like JFFS2 really reads messed data from flash and I think this 
isn't JFFS2'd failure.

I'd suggest to hack the write function and after each write read the data 
that has just been written again and check it has is the same as the 
original data. This check might be done either on MTD layer or on JFFS2 
layer.

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 15:54 +0000 schrieb Artem B. Bityuckiy:

> OK, why does this mean JFFS2 is guilty of this? Why don't you suspect your 
> drivers? Why don't you suspect that you flashed your (say, correct) image 
> incorrectly?

The image is created in different ways, copied, dd'ed, mkfs.jffs2. Same
error picture.
All other constellations in software on this Hardware are working
perfect.
Kernel XIP out of this Flash device with this settings runs perfect.

No matter if partitions are cat'ed or dd'ed or files copied and such:
identical md5sums read back. 

I tested cramfs as rootfilesystem which works ok also.

> Just configure your kernel properly: switch on the "Test driver using RAM" 
> (MTD_MTDRAM) option in your kernel configuration. Type "modprobe mtdram" 
> if you have configured it as the kernel module.
> 
> If you do this your JFFS2 will work over emulated 4MiB in-RAM flash and 
> this will help you to exclude driver problems.

No clue until now. On Host-PC or on the embedded system?

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 16:21 +0000 schrieb Artem B. Bityuckiy:
> > 00435660: 0000 0000 0000 0000 b882 2ec7 8519 01e0  ................
> > 00435670: 3c00 0000 c001 e8b0 fb00 0000 1300 0000  <...............
>             ^
>             node length
> > 00435680: 0d01 0000 9400 0000 1408 0000 55ff 2d7c  ............U.-|
>                                 ^
>                                 name length
> > 00435690: a9c7 cf9f 6c69 6270 7468 7265 6164 2d30  ....libpthread-0
>             ^^^^ ^^^^
>             Name CRC
> > 004356a0: 2e39 2e32 362e 736f 8519 02c0 a906 0000  .9.26.so........
> > 004356b0: 4a1e 6fb8 0d01 0000 0200 0000 a481 0000  J.o.............
> > 004356c0: 0000 0000 0010 0000 9400 0000 9400 0000  ................
> Where has this dump come from?

My initial Mail:

I did "cat /dev/mtdblock3 > file1" into nfs-root (as a side note, I did
the same again with dd bs=1024k:
konsti at synertronixx3:/usr/src/debian_arm/ > md5sum ddmtdblock3 
e871d71c52d93c1cafb4a8f4ff47a741  ddmtdblock3
konsti at synertronixx3:/usr/src/debian_arm/ > md5sum mtdblock3 
e871d71c52d93c1cafb4a8f4ff47a741  mtdblock3
).

This is emacs in hexl-mode on these files:

> This dump looks good:

And this dump is read from the same system right after umounting.

> * the length of node is 0x3c
> * the length of the name is 0x14
> * the name CRC - don't know, didn't calculate.
> 
> Looks like JFFS2 really reads messed data from flash and I think this 
> isn't JFFS2'd failure.

> I'd suggest to hack the write function and after each write read the data 
> that has just been written again and check it has is the same as the 
> original data. This check might be done either on MTD layer or on JFFS2 
> layer.

But the error seems quite systematic so I suspect software. But of
course, still my setup my be broken and jffs2 triggers something. I have
no clue what though.
I will look into the code and see if I find the "write" to read
afterwards... :/

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 16:21 +0000 schrieb Artem B. Bityuckiy:

> Looks like JFFS2 really reads messed data from flash and I think this 
> isn't JFFS2'd failure.

So it reads wrong length of name (always too less)
and wrong data which always is one byte 
located somewhere before the name itself in flash.

When Hardware is broken, why does it not read name lengths too long
sometimes or reads data which is not found in the image a couple of
bytes before the filename? Is this possible? 

Well, may be... hence my question. But I have no clue anymore where to
search. I hacked onto this burst mode for weeks and only jffs2 is not
working in always a similair pattern.
Thats why I show up here, may be burst mode is broken for this system
and that only for jffs2. But I don't know how this filesystems works
exactly when reading and I am trying to find out which fucking bug in
hardware it could trigger, all other tests run fine.

Konsti


-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

> No clue until now. On Host-PC or on the embedded system?
Does it make sense to do it on host?
Surely on the board. This way you will have the same JFFS2 but the 
underlying flash will be another.

> This is emacs in hexl-mode on these files:
Hmm, so this means that when you dump the image it looks good, but when 
JFFS2 reads direntry nodes they aren't good. I'd print similar hexdump 
from JFFS2 and see what it actually reads.

Insert the corresponding printk() to jffs2_scan_dirent_node() function in 
scan.c (print the 'rd' contents).

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

> Well, may be... hence my question. But I have no clue anymore where to
> search. I hacked onto this burst mode for weeks and only jffs2 is not
> working in always a similair pattern.
> Thats why I show up here, may be burst mode is broken for this system
> and that only for jffs2. But I don't know how this filesystems works
> exactly when reading and I am trying to find out which fucking bug in
> hardware it could trigger, all other tests run fine.
Well, then I offer to start debugging. Insert printk() calls and start 
catching error step by step.

All JFFS2 read/write goes through mtd->write(), mtd->writev(), 
mtd->read(). You might just find the corresponded handlers and insert 
printk's there. This will give us additional information.

--
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 17:17 +0000 schrieb Artem B. Bityuckiy:

> Well, then I offer to start debugging. Insert printk() calls and start 
> catching error step by step.

Mine or jffs2's or mtd's error...


	printk("totlen: %x\nnsize: %x\n", rd->totlen,rd->nsize);

before

	rc = crc32(0, rd, sizeof(*rd)-8);

and

        printk("rd->nsize before memcpy: %x\n", rd->nsize);
        printk("address of rd->name: %x\n", rd->name[0]);
        memcpy(&fd->name, rd->name, rd->nsize);
        printk("rd->nsize after memcpy: %x\n", rd->nsize);
        fd->name[rd->nsize] = 0;
        printk("fd->name: %s\n", fd->name);



totlen: 3e        
nsize: 16 
rd->nsize before memcpy: 16
address of rd->name: 6f    
rd->nsize after memcpy: 16
fd->name: openssh-host-key
jffs2_scan_dirent_node(): Name CRC failed on node at 0x003a52c8: Read 0x839452cb, calculated 0x262752af
Name for which CRC failed is (now) 'openssh-host-key', ino #350

totlen: 3c           
nsize: 14 
rd->nsize before memcpy: 14
address of rd->name: 6c    
rd->nsize after memcpy: 14
fd->name: linux-wlan-nÒ   
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0047aa4c: Read 0xb9570647, calculated 0x6ac2e146
Name for which CRC failed is (now) 'linux-wlan-nÒ', ino #210

This is cat "/dev/mtdblock3 > file" afterwards:

003a52c0: 0000 0000 ff45 79b6 8519 01e0 3e00 0000  .....Ey.....>... 
003a52d0: 4bc9 e11a 4901 0000 1600 0000 5e01 0000  K...I.......^... 
003a52e0: 9900 0000 1608 0000 333a 7af0 cb52 9483  ........3:z..R.. 
003a52f0: 6f70 656e 7373 682d 686f 7374 2d6b 6579  openssh-host-key 
003a5300: 6765 6e2e 7368 ffff 8519 02c0 2801 0000  gen.sh......(... 
003a5310: 91d9 c5e8 5e01 0000 0200 0000 ed81 0000  ....^........... 
003a5320: 0000 0000 9902 0000 9900 0000 9900 0000  ................

0047aa40: 0000 0000 0000 0000 3d99 4484 8519 01e0  ........=.D..... 
0047aa50: 3c00 0000 c001 e8b0 d100 0000 0200 0000  <............... 
0047aa60: d200 0000 9100 0000 1408 0000 56db cdde  ............V... 
0047aa70: 4706 57b9 6c69 6e75 782d 776c 616e 2d6e  G.W.linux-wlan-n 
0047aa80: 672d 7072 652d 7570 8519 02c0 8006 0000  g-pre-up........ 
0047aa90: fefe 5565 d200 0000 0200 0000 ed81 0000  ..Ue............ 
0047aaa0: 0000 0000 0010 0000 9100 0000 9100 0000  ................

So it could be rd->name contains garbage, "d2" in second case and
memcopy is terminated by "00". May be...

> All JFFS2 read/write goes through mtd->write(), mtd->writev(), 
> mtd->read(). You might just find the corresponded handlers and insert 
> printk's there. This will give us additional information.

Ok, I will try this tomorrow, I have to leave know, sadly as debugging
gets interesting :)

Kind Regards and thanks for your help, Konsti


-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Look at this quick, dirty hack:

        printk("rd->nsize before memcpy: %x\n", rd->nsize);
        printk("rd->name: %c %x\n", rd->name[0], rd->name[0]);
        printk("rd->name: %c %x\n", rd->name[11], rd->name[11]);
        printk("rd->name: %c %x\n", rd->name[12], rd->name[12]);
        printk("rd->name: %c %x\n", rd->name[13], rd->name[13]);
        memcpy(&fd->name, rd->name, rd->nsize);
        printk("rd->nsize after memcpy: %x\n", rd->nsize);
        printk("rd->name: %c %x\n", rd->name[0], rd->name[0]);
        printk("rd->name: %c %x\n", rd->name[11], rd->name[11]);
        printk("rd->name: %c %x\n", rd->name[12], rd->name[12]);
        printk("rd->name: %c %x\n", rd->name[13], rd->name[13]);


rd->nsize before memcpy: 14
rd->name: l 6c             
rd->name: n 6e
rd->name: g 67
rd->name: - 2d
rd->nsize after memcpy: 14
rd->name: l 6c            
rd->name: n 6e
rd->name: g 67
rd->name: - 2d
fd->name: linux-wlan-nÒ
jffs2_scan_dirent_node(): Name CRC failed on node at 0x0047aa4c: Read 0xb9570647, calculated 0x6ac2e146
Name for which CRC failed is (now) 'linux-wlan-nÒ', ino #210



        printk("rd->nsize before memcpy: %x\n", rd->nsize);
        printk("rd->name: %c %x\n", rd->name[0], rd->name[0]);
        printk("rd->name: %c %x\n", rd->name[14], rd->name[14]);
        printk("rd->name: %c %x\n", rd->name[15], rd->name[15]);
        printk("rd->name: %c %x\n", rd->name[16], rd->name[16]);
        printk("rd->name: %c %x\n", rd->name[17], rd->name[17]);
        memcpy(&fd->name, rd->name, rd->nsize);
        printk("rd->nsize after memcpy: %x\n", rd->nsize);
        printk("rd->name: %c %x\n", rd->name[0], rd->name[0]);
        printk("rd->name: %c %x\n", rd->name[14], rd->name[14]);
        printk("rd->name: %c %x\n", rd->name[15], rd->name[15]);
        printk("rd->name: %c %x\n", rd->name[16], rd->name[16]);
        printk("rd->name: %c %x\n", rd->name[17], rd->name[17]);



rd->nsize before memcpy: 16
rd->name: o 6f             
rd->name: e 65
rd->name: y 79
rd->name: g 67
rd->name: e 65
rd->nsize after memcpy: 16
rd->name: o 6f            
rd->name: e 65
rd->name: y 79
rd->name: g 67
rd->name: e 65
fd->name: openssh-host-key
jffs2_scan_dirent_node(): Name CRC failed on node at 0x003a52c8: Read 0x839452cb, calculated 0x262752af
Name for which CRC failed is (now) 'openssh-host-key', ino #350

So rd->name and rd->size seems to be both ok just before and after memcpy.

Weird...

Konsti 


-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Steve Wahl]

Konsti,

Just a suggestion: Copying, DD, mkfs.jffs2 all will write large
buffers to your flash, probably aligned on nice address boundaries,
too.  The filesystem *may* be writing small buffers of wierd lengths
and perhaps not well aligned; and this may be what causes your
problems to surface.

[ I don't actually know what kind of writes and reads the filesystem
does. I subscribe to this list for the MTD half, not the JFFS
part. :-) ]

Look real closely at bursting with odd lengths and/or alignment; see
if JFFS2 is doing any of these, and then see if that causes your
hardware to react badly.

Since your raw dumps of the flash after the fact appear good(?), I'd
concentrate on the read operations.  (If the dumps were bad, I'd look
at the writes.)

If your hardware screws up with unusual alignment, well, it's not all
that rare of a bug to find.

--> Steve

On Thu, Mar 10, 2005 at 05:40:24PM +0100, Konstantin Kletschke wrote:
> Am 2005-03-10 15:54 +0000 schrieb Artem B. Bityuckiy:
> 
> > OK, why does this mean JFFS2 is guilty of this? Why don't you suspect your 
> > drivers? Why don't you suspect that you flashed your (say, correct) image 
> > incorrectly?
> 
> The image is created in different ways, copied, dd'ed, mkfs.jffs2. Same
> error picture.
> All other constellations in software on this Hardware are working
> perfect.
> Kernel XIP out of this Flash device with this settings runs perfect.
> 
> No matter if partitions are cat'ed or dd'ed or files copied and such:
> identical md5sums read back. 
> 
> I tested cramfs as rootfilesystem which works ok also.
> 
> > Just configure your kernel properly: switch on the "Test driver using RAM" 
> > (MTD_MTDRAM) option in your kernel configuration. Type "modprobe mtdram" 
> > if you have configured it as the kernel module.
> > 
> > If you do this your JFFS2 will work over emulated 4MiB in-RAM flash and 
> > this will help you to exclude driver problems.
> 
> No clue until now. On Host-PC or on the embedded system?
> 
> Konsti
> 
> -- 
> GPG KeyID EF62FCEF
> Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF
> 
> ______________________________________________________
> Linux MTD discussion mailing list
> http://lists.infradead.org/mailman/listinfo/linux-mtd/

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 18:01 +0100 schrieb Joakim Tjernlund:

> I think there is old data/garbage in the CPU:s cache thats causing your problem.
> Do you invalidate the data cache before you burst copy data from flash?

I drunk a couple of beer after several hours of hacking onto this and I
think in the same direction as you at the moument.
If you followed the thread in the mailing list you see my debugging
printk effords, which show that memcpy() reads the wrong line after page
boundary to the next line.
I thought first that page wrap mode of sync burst mode is freaking out,
but source and destiantion of memcpy() is in RAM.

So DataCache is the problem to check up. I don't know yet if _I_
invalidate cache before bursting. I should... or jffs2 or mtd-tools...

May I copy this mail into the ml, since you addressed me without it?

> Whats your CPU?

Motorla ARM9 core implemented on their i.MX architecture.

Regards, Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-10 13:25 -0600 schrieb Steve Wahl:

> [ I don't actually know what kind of writes and reads the filesystem
> does. I subscribe to this list for the MTD half, not the JFFS

Me neither, I hope the people here will guide me to find out :)

> Since your raw dumps of the flash after the fact appear good(?), I'd
> concentrate on the read operations.  (If the dumps were bad, I'd look
> at the writes.)

Yeah.

> If your hardware screws up with unusual alignment, well, it's not all
> that rare of a bug to find.

Well lets see. If you look at my debug outputs, I am actually tracing
the problem down to the memcpy function itself.
I suspect there is something wrong with the data cache.
memcpy misreads the correct the original and puts garbage into the copy,
which is most often a line above the original. I don't suspect burst
wrap mode or such, because too much other things work well in this mode
and both, source and copy of the memcpy function are actually in RAM
already...

Regards, Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-11 00:38 +0100 schrieb Joakim Tjernlund:

> Invalidating dcache should probably go into the copy_from routine, if needed.
> I don't known if your ARM CPU is cache coherent or not. If it is, you shouldn't need
> to invalidate the dcache.

IIRC it is coherent, but I have to taka a look at this :)
> 
> It could still be an alignment problem, I think the ->name field is at an
> unligned address if my memory serves me right.

I don't exclude an alignment problem...

> > May I copy this mail into the ml, since you addressed me without it?
> 
> Sure, I just hit the wrong reply button.

:)

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

jffs2 with sync burst mode

[Konstantin Kletschke]

Actually I tried to debug, what happens in, before and after memcpy() in
fs/jffs2/scan.c line 773 (actual cvs) located in
jffs2_scan_dirent_node(), as I suspect some issues with dcache.
Therefore I copied memcpy() out of arch/arm/boot/compressed/misc.c (why
is it _there?) above jffs2_scan_dirent_node() to memcpykgk().

The "Name CRC failed on node at ..." is away. I populated the root-fs
on the system with "cp -a -r" over nfs and rebooted with this as
root-fs. No problems. But mounting takes ages :)

Whats next? :D

Konsti



-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

Konstantin Kletschke wrote:
 > The "Name CRC failed on node at ..." is away. I populated the root-fs
Good.

 > root-fs. No problems. But mounting takes ages :)
How much time does "ages" mean ?
Do you stand that it was faster before?
How large is your flash?
Do you have directories with huge files number in it (say, ~ 10000)?

 > Whats next? :D
Profiling might help.
Try to put fewver files to your flash and test how quck is it mounted.

P.S. :

 > I erased /dev/mtdblock3 with bootloader and
If you erase your flash with flash_eraseall -j it might work a bit faster.

Q: What's -j ?
A: This means that flash_eraseall ought to write special node called 
cleanmarker at the beginning of each block.

Q: Why is this better then to erase using my bootloader?
A: your bootloader probably doesn't write cleanmarkers.

Q: What happens if there are no cleanmarkers?
A: Nothing critical. JFFS2 will just re-erase such blocks and put 
cleanmarkers itself. This is just time wasting.

Q: Why does JFFS2 re-erase blocks which have no cleanmarkers but contain 
all FF (i.e., are erased)?
A: Because it was observed that if the power loss occurs when block is 
almost completely erased, there might be "unstable" bits in it. I.e., 
they are read sometimes as "1" and sometimes as "0" (This effect is 
described in David Woodhouse's page). Cleanmarker does guarantee that 
the block was successfully erased. All FFs don't.

Q: Does this affect mount time?
A: It might, but I guess not very much. The difference is that if there 
is clean marker at the beginning of the block, JFFS2 quickly detects 
that it is free. Otherwise, JFFS2 scans it trying to find some data, 
thus, wasting time.

P.P.S:
JFFS2 has problems with mount time but it should be noticable when you 
work with relatively large partitions, which isn't very typical for NOR.

If you are sure that the problem is not your slow flash driver, you 
might try Ferenc's "summary" patch.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-11 14:15 +0300 schrieb Artem B. Bityuckiy:
> Konstantin Kletschke wrote:
> > The "Name CRC failed on node at ..." is away. I populated the root-fs
> Good.

But is moving the memcpy function a valid fix? There must be something
really wrong which should be fixed :/ DCache corruption i.e. missing
invalidating at the right time?

> 
> > root-fs. No problems. But mounting takes ages :)
> How much time does "ages" mean ?
> Do you stand that it was faster before?

Sorry, my apologies! CONFIG_MTD_DEBUG_VERBOSE was on 2, on 0 I get
reasonable mount time again, I am very sorry for misleading you!

> How large is your flash?

16MB, the root-fs is 5MB...

> Do you have directories with huge files number in it (say, ~ 10000)?

No.

> A: This means that flash_eraseall ought to write special node called 
> cleanmarker at the beginning of each block.

> If you are sure that the problem is not your slow flash driver, you 
> might try Ferenc's "summary" patch.

But if I understood correct the not created-cleanmarkers-by-bootloader
problem is "solved" on second boot? I see this here also, the 1st mount
takes ages, the following are fast.

Regards, sorry, Konstantin

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Konstantin Kletschke]

Am 2005-03-11 11:44 +0100 schrieb Konstantin Kletschke:
> Actually I tried to debug, what happens in, before and after memcpy() in
> fs/jffs2/scan.c line 773 (actual cvs) located in
> jffs2_scan_dirent_node(), as I suspect some issues with dcache.
> Therefore I copied memcpy() out of arch/arm/boot/compressed/misc.c (why
> is it _there?) above jffs2_scan_dirent_node() to memcpykgk().

No I am puzzled.

I think I replaced arch/arm/lib/memcpy.S with 

static inline __ptr_t memcpykgk(__ptr_t __dest, __const __ptr_t __src,
                size_t __n)

out of arch/arm/boot/compressed/misc.c by accident, though it works now.

The inline helps probably. So, is it correct memcpy() in lib/string.c
and out of arch/arm/boot/compressed/misc.c are NOT used in general arm
code?

Konsti

-- 
GPG KeyID EF62FCEF
Fingerprint: 13C9 B16B 9844 EC15 CC2E  A080 1E69 3FDA EF62 FCEF

Re: jffs2 with sync burst mode

[Artem B. Bityuckiy]

Konstantin Kletschke wrote:
> But is moving the memcpy function a valid fix? There must be something
> really wrong which should be fixed :/ DCache corruption i.e. missing
> invalidating at the right time?
Err, how can ti be valid? Do you see any problem in JFFS2 itself?

AFAIU, you have problems in layers which are bellow JFFS2. JFFS2 itself 
is correct. Your fix in JFFS2 is not actually fix, you have just hidden 
your problem, right?

Unfortunately, I can't help you with your Dcache problems. In fact, I 
have very limited NOR experience for now, and I might help if you find 
bugs in JFFS2, not in underlying drivers. May be anybody else...


> But if I understood correct the not created-cleanmarkers-by-bootloader
> problem is "solved" on second boot? I see this here also, the 1st mount
> takes ages, the following are fast.
This depends on how long have you used JFFS2 after the first mount. 
Blocks aren't erased immediately on mount. Instead, this work is done 
later. So, *some* blocks *might* stay not re-erased.

-- 
Best Regards,
Artem B. Bityuckiy,
St.-Petersburg, Russia.