jffs2: mount problems when XATTR is enabled

jffs2: mount problems when XATTR is enabled

[Paul Wakeman]

I'm seeing lots of jffs2 errors at boot when mounting filesystems. My
kernel has XATTR enabled. If I build my kernel without
CONFIG_JFFS2_FS_XATTR, the errors don't occur. What are the
requirements for a working jffs2 with XATTR? Are there dependencies of
userspace utilities - mount, mkfs.jffs2 etc?

Errors are of the form:-

Node totlen on flash (0xffffffff) != totlen from node ref (0x0000013c)
at 0x00322d24
Node totlen on flash (0xffffffff) != totlen from node ref (0x000007e0)
at 0x00322544

The filesystem mounts but "ls" gives errors:

# ls /
ls: Input/output error

My board has NOR and NAND flash. The errors occur in NOR partitions.

System info:-
Freescale MX31 CPU (ARM11 core)
2.6.19 kernel
mkfs.jffs2 version 1.50
gcc 4.1.1 (glibc)
busybox 1.1.3

I tried patching my kernel with the MTD code from the latest 2.6.22
kernel but it results in the same problems. I also tried the latest
mkfs.jffs2 from git (which reports the same version number though I
don't think it is...).

Curiously, if I boot an NFS root and then mount the partition
manually, the errors do not occur.

Any ideas?

Thanks

Paul

Re: jffs2: mount problems when XATTR is enabled

[KaiGai Kohei]

Paul,

I like to confirm your situation at first.

Did you pack any xattr in the filesystem image?
If you didn't specify a xattr related option in mkfs.jffs2, any xattr
on the host filesystem is not packed.

Paul Wakeman wrote:
> I'm seeing lots of jffs2 errors at boot when mounting filesystems. My
> kernel has XATTR enabled. If I build my kernel without
> CONFIG_JFFS2_FS_XATTR, the errors don't occur. What are the
> requirements for a working jffs2 with XATTR? Are there dependencies of
> userspace utilities - mount, mkfs.jffs2 etc?
> 
> Errors are of the form:-
> 
> Node totlen on flash (0xffffffff) != totlen from node ref (0x0000013c)
> at 0x00322d24
> Node totlen on flash (0xffffffff) != totlen from node ref (0x000007e0)
> at 0x00322544

jffs2_mark_node_obsolete() generates above messages, when the length of
node on the medium is differ from expected one.
It seems to me jffs2 tries to access a cleared region because left hand
is 0xffffff. In addition, 0x013c (316byte) and 0x07e0 (2016byte) are
too long, if these are xattr nodes.

Is it possible to retry it using the kernel with debug messages.
(See fs/jffs2/debug.h)

> The filesystem mounts but "ls" gives errors:
> 
> # ls /
> ls: Input/output error
> 
> My board has NOR and NAND flash. The errors occur in NOR partitions.
> 
> System info:-
> Freescale MX31 CPU (ARM11 core)
> 2.6.19 kernel
> mkfs.jffs2 version 1.50
> gcc 4.1.1 (glibc)
> busybox 1.1.3
> 
> I tried patching my kernel with the MTD code from the latest 2.6.22
> kernel but it results in the same problems. I also tried the latest
> mkfs.jffs2 from git (which reports the same version number though I
> don't think it is...).
> 
> Curiously, if I boot an NFS root and then mount the partition
> manually, the errors do not occur.

Is it really same kernel, and same partition?

> Any ideas?
> 
> Thanks
> 
> Paul
-- 
Open Source Software Promotion Center, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Re: jffs2: mount problems when XATTR is enabled

[Paul Wakeman]

KaiGai Kohei <kaigai@ak.jp.nec.com> wrote:

> Did you pack any xattr in the filesystem image?

No. The initial rootfs has no files with xattrs. Applications add
attributes when the system operates. These errors occur before the
first xattr is written.

> is 0xffffff. In addition, 0x013c (316byte) and 0x07e0 (2016byte) are
> too long, if these are xattr nodes.

I don't think they're xattr nodes. These errors happen on first boot
when there are no xattrs.

> Is it possible to retry it using the kernel with debug messages.
> (See fs/jffs2/debug.h)

Ok. I'll send the log later.

> > Curiously, if I boot an NFS root and then mount the partition
> > manually, the errors do not occur.
> 
> Is it really same kernel, and same partition?

Yes. 

I mount the partition using

mount -t jffs2 /dev/mtdblock2 /mnt/tmp

I can then modify files under /mnt/tmp and then umount without errors.
There is a delay while I think jffs2 writes its clean markers to erased
sectors within the partition. If I boot with root=/dev/mtdblock2, the
errors occur and although the filesystem mounts, ls gives errors.

I'm beginning to think this is a timing issue at boot, perhaps due to a
bug in another driver. I'll investigate that.

Thanks for your reply.

Paul



      ____________________________________________________________________________________
Luggage? GPS? Comic books? 
Check out fitting gifts for grads at Yahoo! Search
http://search.yahoo.com/search?fr=oni_on_mail&p=graduation+gifts&cs=bz

Re: jffs2: mount problems when XATTR is enabled

[KaiGai Kohei]

Paul,

 >> Did you pack any xattr in the filesystem image?
 >
 > No. The initial rootfs has no files with xattrs. Applications add
 > attributes when the system operates. These errors occur before the
 > first xattr is written.

I wonder why jffs2/xattr code gave an effect your flash device
without any xattr node, because any xattr related implementation
is invoked only when jffs2 found xattr nodes on the medium, or
xattr related operations are done (like setfacl).

>> Is it possible to retry it using the kernel with debug messages.
>> (See fs/jffs2/debug.h)
> 
> Ok. I'll send the log later.

I think Kconfig related to JFFS2 also helps us to understand the situation.

 > mount -t jffs2 /dev/mtdblock2 /mnt/tmp
 >
 > I can then modify files under /mnt/tmp and then umount without errors.
 > There is a delay while I think jffs2 writes its clean markers to erased
 > sectors within the partition. If I boot with root=/dev/mtdblock2, the
 > errors occur and although the filesystem mounts, ls gives errors.
 >
 > I'm beginning to think this is a timing issue at boot, perhaps due to a
 > bug in another driver. I'll investigate that.

What's happen when you boot the board with NFS root, and mount the errored
partition manually again?
If you can reproduce the error, I recommend you to try the same test
on mtdram pseudo mtd device next.

-- 
Open Source Software Promotion Center, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Re: jffs2: mount problems when XATTR is enabled

[Paul Wakeman]

[-- Attachment #1: Type: text/plain, Size: 2424 bytes --]

KaiGai Kohei <kaigai@ak.jp.nec.com> wrote:

>  >> Did you pack any xattr in the filesystem image?
>  >
>  > No. The initial rootfs has no files with xattrs. Applications add
>  > attributes when the system operates. These errors occur before the
>  > first xattr is written.
> 
> I wonder why jffs2/xattr code gave an effect your flash device
> without any xattr node, because any xattr related implementation
> is invoked only when jffs2 found xattr nodes on the medium, or
> xattr related operations are done (like setfacl).

Maybe it causes timing differences just by it being compiled in?

My boot logs show calls to jffs_build_xattr_subsystem().

> >> Is it possible to retry it using the kernel with debug messages.
> >> (See fs/jffs2/debug.h)
> > 
> > Ok. I'll send the log later.
> 
> I think Kconfig related to JFFS2 also helps us to understand the
> situation.

Attached are two compressed logs captured on the serial port.

jffs2-dbg-flashboot.log.gz
jffs2-dbg-nfsboot.log.gz

The logs show kernel boot messages and shell command output which will
hopefully show the symptoms. The same kernel was used in each case.

The root partition is in NOR flash (mtd2). Applications use xattr
features only in /mnt/userdata (also in NOR at mtd3). There are 3
partitions in NAND (mtd6-8).

At the end of the nfsboot log, there are shell commands that show that
manually mounting /dev/mtdblock2 (which gave all the errors in the
flash boot case) mounts cleanly. 

The jffs2 kernel config is:-

# CONFIG_JFFS_FS is not set
CONFIG_JFFS2_FS=y
CONFIG_JFFS2_FS_DEBUG=2
CONFIG_JFFS2_FS_WRITEBUFFER=y
# CONFIG_JFFS2_SUMMARY is not set
CONFIG_JFFS2_FS_XATTR=y
# CONFIG_JFFS2_FS_POSIX_ACL is not set
# CONFIG_JFFS2_FS_SECURITY is not set
# CONFIG_JFFS2_COMPRESSION_OPTIONS is not set
CONFIG_JFFS2_ZLIB=y
CONFIG_JFFS2_RTIME=y
# CONFIG_JFFS2_RUBIN is not set

> What's happen when you boot the board with NFS root, and mount the
> errored
> partition manually again?

See the attached nfsboot log. No errors!

> If you can reproduce the error, I recommend you to try the same test
> on mtdram pseudo mtd device next.

Unfortunately this only seems to happen if the system is booted using
the jffs2 rootfs.

Thanks again for your reply.

Paul



       
____________________________________________________________________________________Ready for the edge of your seat? 
Check out tonight's top picks on Yahoo! TV. 
http://tv.yahoo.com/

[-- Attachment #2: 1698775580-jffs2-dbg-flashboot.log.gz --]
[-- Type: application/x-gzip, Size: 52358 bytes --]

[-- Attachment #3: 467851458-jffs2-dbg-nfsboot.log.gz --]
[-- Type: application/x-gzip, Size: 51999 bytes --]

Re: jffs2: mount problems when XATTR is enabled

[KaiGai Kohei]

Paul Wakeman wrote:
>> I wonder why jffs2/xattr code gave an effect your flash device
>> without any xattr node, because any xattr related implementation
>> is invoked only when jffs2 found xattr nodes on the medium, or
>> xattr related operations are done (like setfacl).
> 
> Maybe it causes timing differences just by it being compiled in?

Is it means there are differences of kernels between flash case and
NFS one? We can confirm it using binary hash such as md5sum.

> My boot logs show calls to jffs_build_xattr_subsystem().

But it reports the partition contains no JFFS2_NODETYPE_XATTR and JFFS2_NODETYPE_XREF
nodes. jffs_build_xattr_subsystem() did just an empty loop.

| JFFS2 notice: (1) jffs2_build_xattr_subsystem: complete building xattr subsystem
| , 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.

>>>> Is it possible to retry it using the kernel with debug messages.
>>>> (See fs/jffs2/debug.h)
>>> Ok. I'll send the log later.
>> I think Kconfig related to JFFS2 also helps us to understand the
>> situation.
> 
> Attached are two compressed logs captured on the serial port.
> 
> jffs2-dbg-flashboot.log.gz
> jffs2-dbg-nfsboot.log.gz
> 
> The logs show kernel boot messages and shell command output which will
> hopefully show the symptoms. The same kernel was used in each case.

In the tail of jffs2-dbg-flashboot.log.gz, it shows you can execute 'df' and
'ls' command, although 'ls' reported EIO.
However, success of 'df' means it can open and read /etc/mtab, at least.
Is it installed in /dev/mtdblock2 ?

Is it possible to send the following logs?
- "ls /" after "dmesg -n 8"
 It seems to me that some warnning or notice messages from kernel are filtered
 due to log message level configuration.
 Xattr implementation has a possibility to return -EIO, but I could not found
 those messages in jffs2-dbg-flashboot.log.gz.

- "strace ls /"
 It helps us to know what system call returns -EIO, and whether applications
 actually use xattr features, or not.

> The root partition is in NOR flash (mtd2). Applications use xattr
> features only in /mnt/userdata (also in NOR at mtd3). There are 3
> partitions in NAND (mtd6-8).

There is a possibility that applications use xattr features implicitly
via library functions, I think.
For example, the recent ls in busybox supports to display security context
of SELinux. It calls fgetxattr(2) via libselinux.

> At the end of the nfsboot log, there are shell commands that show that
> manually mounting /dev/mtdblock2 (which gave all the errors in the
> flash boot case) mounts cleanly.

Hmm...
Currently, I have no idea to explain your situation...

--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Re: jffs2: mount problems when XATTR is enabled

[KaiGai Kohei]

KaiGai Kohei wrote:
> Is it possible to send the following logs?
> - "ls /" after "dmesg -n 8"
>  It seems to me that some warnning or notice messages from kernel are filtered
>  due to log message level configuration.
>  Xattr implementation has a possibility to return -EIO, but I could not found
>  those messages in jffs2-dbg-flashboot.log.gz.

In addition, is it possible to execute "ls <any other directories/files>" ?
e.g) $ ls /mnt

-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>

Re: jffs2: mount problems when XATTR is enabled

[Paul Wakeman]

KaiGai Kohei <kaigai@ak.jp.nec.com> wrote:

> > Maybe it causes timing differences just by it being compiled in?
> 
> Is it means there are differences of kernels between flash case and
> NFS one? We can confirm it using binary hash such as md5sum

No, it's the same kernel. See the cat /proc/version output in the
previous logs.

> In the tail of jffs2-dbg-flashboot.log.gz, it shows you can execute
> 'df' and
> 'ls' command, although 'ls' reported EIO.

Actually, it was the OS trying to fetch 'ls' from the filesystem that
returned EIO. 'ls' doesn't run at all. Using the following:-

cd /bin
./<TAB><TAB>

to have the shell's auto-completion list the directory contents, 'ls'
is missing. When trying to manually create the link to busybox, it
claims 'ls' already exists. So I create it in /usr/bin instead and then
'ls' works as expected.

> However, success of 'df' means it can open and read /etc/mtab, at
> least.
> Is it installed in /dev/mtdblock2 ?

It is symlinked to /proc/mounts

> Is it possible to send the following logs?
> - "ls /" after "dmesg -n 8"
>  It seems to me that some warnning or notice messages from kernel are
> filtered
>  due to log message level configuration.

I'll need to set the debug level early at boot since the EIO errors
come from the system failing to load 'ls' not the 'ls' executable
itself.

Paul



       
____________________________________________________________________________________
Be a better Heartthrob. Get better relationship answers from someone who knows. Yahoo! Answers - Check it out. 
http://answers.yahoo.com/dir/?link=list&sid=396545433

Re: jffs2: mount problems when XATTR is enabled

[Paul Wakeman]

KaiGai Kohei <kaigai@ak.jp.nec.com> wrote:

> KaiGai Kohei wrote:
> > Is it possible to send the following logs?
> > - "ls /" after "dmesg -n 8"

I managed to boot a debug kernel when /mnt/userdata was showing jffs2
errors. "ls /mnt/userdata" returns IO error.

aseries# ls /mnt/userdata
JFFS2 warning: (1783) jffs2_get_inode_nodes: Eep. No valid nodes for
ino #2.
JFFS2 warning: (1783) jffs2_do_read_inode_internal: no data nodes found
for ino #2
ls: /mnt/userdata/4242-1111-1111-0001: Input/output error
aseries# echo 8 > /proc/sys/kernel/printk
aseries# ls /mnt/userdata
jffs2_follow_link(): target path is 'busybox'
jffs2_follow_link(): target path is 'ld-2.4.so'
jffs2_follow_link(): target path is 'libcrypt-2.4.so'
jffs2_follow_link(): target path is 'libc-2.4.so'
jffs2_readdir() for dir_i #1
Dirent 0: ".", ino #1
Dirent 1: "..", ino #1
Dirent 2: "4242-1111-1111-0001", ino #2, type 8
Skipping deletion dirent "4242-1111-1111-0001.new"
jffs2_readdir() for dir_i #1
Skipping dirent: "4242-1111-1111-0001", ino #2, type 8, because curofs
2 < offset 4
Skipping dirent: "4242-1111-1111-0001.new", ino #0, type 0, because
curofs 3 < offset 4
ls: /mnt/userdata/4242-1111-1111-0001: Input/output error

> >  It seems to me that some warnning or notice messages from kernel
> are filtered
> >  due to log message level configuration.
> >  Xattr implementation has a possibility to return -EIO, but I could
> not found
> >  those messages in jffs2-dbg-flashboot.log.gz.
> 
> In addition, is it possible to execute "ls <any other
> directories/files>" ?
> e.g) $ ls /mnt

Yes it is.

aseries# ls /mnt
jffs2_follow_link(): target path is 'busybox'
jffs2_follow_link(): target path is 'ld-2.4.so'
jffs2_follow_link(): target path is 'libcrypt-2.4.so'
jffs2_follow_link(): target path is 'libc-2.4.so'
jffs2_readdir() for dir_i #9
Dirent 0: ".", ino #9
Dirent 1: "..", ino #1
Dirent 2: "nfs", ino #497, type 4
Dirent 3: "src", ino #499, type 4
Dirent 4: "userdata", ino #494, type 4
Dirent 5: "maps", ino #496, type 4
Dirent 6: "rwfs", ino #498, type 4
Dirent 7: "cdrom", ino #492, type 4
Dirent 8: "crash", ino #493, type 4
Dirent 9: "floppy", ino #495, type 4
jffs2_readdir() for dir_i #9
Skipping dirent: "nfs", ino #497, type 4, because curofs 2 < offset 10
Skipping dirent: "src", ino #499, type 4, because curofs 3 < offset 10
Skipping dirent: "userdata", ino #494, type 4, because curofs 4 <
offset 10
Skipping dirent: "maps", ino #496, type 4, because curofs 5 < offset 10
Skipping dirent: "rwfs", ino #498, type 4, because curofs 6 < offset 10
Skipping dirent: "cdrom", ino #492, type 4, because curofs 7 < offset
10
Skipping dirent: "crash", ino #493, type 4, because curofs 8 < offset
10
Skipping dirent: "floppy", ino #495, type 4, because curofs 9 < offset
10
cdrom   crash   userdata    floppy  maps    nfs     rwfs    src

The /mnt/userdata partition is the one where files have XATTRs.

Are others using XATTR with jffs2?

Paul



       
____________________________________________________________________________________
Boardwalk for $500? In 2007? Ha! Play Monopoly Here and Now (it's updated for today's economy) at Yahoo! Games.
http://get.games.yahoo.com/proddesc?gamekey=monopolyherenow  

Re: jffs2: mount problems when XATTR is enabled

[KaiGai Kohei]

Paul Wakeman wrote:
> KaiGai Kohei <kaigai@ak.jp.nec.com> wrote:
> 
>> KaiGai Kohei wrote:
>>> Is it possible to send the following logs?
>>> - "ls /" after "dmesg -n 8"
> 
> I managed to boot a debug kernel when /mnt/userdata was showing jffs2
> errors. "ls /mnt/userdata" returns IO error.
> 
> aseries# ls /mnt/userdata
> JFFS2 warning: (1783) jffs2_get_inode_nodes: Eep. No valid nodes for
> ino #2.
> JFFS2 warning: (1783) jffs2_do_read_inode_internal: no data nodes found
> for ino #2
> ls: /mnt/userdata/4242-1111-1111-0001: Input/output error

Could you get the above messages in the second ls with the most
detailed kernel messages?

> aseries# echo 8 > /proc/sys/kernel/printk
> aseries# ls /mnt/userdata
> jffs2_follow_link(): target path is 'busybox'
> jffs2_follow_link(): target path is 'ld-2.4.so'
> jffs2_follow_link(): target path is 'libcrypt-2.4.so'
> jffs2_follow_link(): target path is 'libc-2.4.so'
These are required by loading busybox and libraries depends on.

> jffs2_readdir() for dir_i #1
> Dirent 0: ".", ino #1
> Dirent 1: "..", ino #1
> Dirent 2: "4242-1111-1111-0001", ino #2, type 8
> Skipping deletion dirent "4242-1111-1111-0001.new"
busybox tried to getdents(2), and it could obtain three entries and
one dead entry.

> jffs2_readdir() for dir_i #1
> Skipping dirent: "4242-1111-1111-0001", ino #2, type 8, because curofs
> 2 < offset 4
> Skipping dirent: "4242-1111-1111-0001.new", ino #0, type 0, because
> curofs 3 < offset 4
busybox tries to getdents(2) for rest of directory entries,
but no more entries are exist.

> ls: /mnt/userdata/4242-1111-1111-0001: Input/output error
ls in busybox tries to call lstat(2) for all the entries except for "."
and ".." next to getdents(2).
It seems to me something happen on sys_lstat() or its call chain.
Maybe, it fails to resolve the path name, because jffs2 does not have
its i_ops->getattr() handler and generic_fillattr() never fails.

In the above trial, I intended that we can get some warnings from
jffs2/xattr subsystem if incorrect xattr cause the -EIO.

BTW, is the /bin/ls stored on NFS partition when you booted it with
NFS root? What is happen on using /mnt/tmp/bin/ls ?
Is it possible to reproduce with NFS root?

| aseries# df
| Filesystem           1k-blocks      Used Available Use% Mounted on
| /dev/root            150231148 121530312  20946292  85% /
| tmpfs                    55284       520     54764   1% /dev
| rwfs                      1024       836       188  82% /mnt/rwfs
| rwfs                      1024       836       188  82% /tmp
| rwfs                      1024       836       188  82% /var
| rwfs                      1024       836       188  82% /etc
| shm                      55284         4     55280   0% /dev/shm
| /dev/mtdblock6           15360       532     14828   3% /usr/local
| /dev/mtdblock3            2048       388      1660  19% /mnt/userdata
| /dev/mtdblock7            1024        68       956   7% /mnt/crash
| /dev/mtdblock8          114688      3044    111644   3% /mnt/maps
| aseries# ls /
| bin            home           opt            sbin           var
| boot           lib            proc           share
| dev            linuxrc        root           sys
| dir            mnt            rootfs.jffs2   tmp
| etc            null.jffs2     rootfs2.jffs2  usr
| aseries# mount -t jffs2 /dev/mtdblock2 /mnt/tmp
| JFFS2 notice: (1770) jffs2_build_xattr_subsystem: complete building xattr subsys
| tem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
| aseries# ls /mnt/tmp
| bin      etc      linuxrc  proc     share    usr
| dev      home     mnt      root     sys      var
| dir      lib      opt      sbin     tmp
aseries# /mnt/tmp/bin/ls /mnt/tmp
What happen?

>>>  It seems to me that some warnning or notice messages from kernel
>> are filtered
>>>  due to log message level configuration.
>>>  Xattr implementation has a possibility to return -EIO, but I could
>> not found
>>>  those messages in jffs2-dbg-flashboot.log.gz.
>> In addition, is it possible to execute "ls <any other
>> directories/files>" ?
>> e.g) $ ls /mnt
> 
> Yes it is.
> 
> aseries# ls /mnt
> jffs2_follow_link(): target path is 'busybox'
> jffs2_follow_link(): target path is 'ld-2.4.so'
> jffs2_follow_link(): target path is 'libcrypt-2.4.so'
> jffs2_follow_link(): target path is 'libc-2.4.so'
> jffs2_readdir() for dir_i #9
> Dirent 0: ".", ino #9
> Dirent 1: "..", ino #1
> Dirent 2: "nfs", ino #497, type 4
> Dirent 3: "src", ino #499, type 4
> Dirent 4: "userdata", ino #494, type 4
> Dirent 5: "maps", ino #496, type 4
> Dirent 6: "rwfs", ino #498, type 4
> Dirent 7: "cdrom", ino #492, type 4
> Dirent 8: "crash", ino #493, type 4
> Dirent 9: "floppy", ino #495, type 4
> jffs2_readdir() for dir_i #9
> Skipping dirent: "nfs", ino #497, type 4, because curofs 2 < offset 10
> Skipping dirent: "src", ino #499, type 4, because curofs 3 < offset 10
> Skipping dirent: "userdata", ino #494, type 4, because curofs 4 <
> offset 10
> Skipping dirent: "maps", ino #496, type 4, because curofs 5 < offset 10
> Skipping dirent: "rwfs", ino #498, type 4, because curofs 6 < offset 10
> Skipping dirent: "cdrom", ino #492, type 4, because curofs 7 < offset
> 10
> Skipping dirent: "crash", ino #493, type 4, because curofs 8 < offset
> 10
> Skipping dirent: "floppy", ino #495, type 4, because curofs 9 < offset
> 10
> cdrom   crash   userdata    floppy  maps    nfs     rwfs    src
> 
> The /mnt/userdata partition is the one where files have XATTRs.
> 
> Are others using XATTR with jffs2?
> 
> Paul

-- 
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai@ak.jp.nec.com>