* UBIFS: assert failed in ubifs_tnc_next_ent
@ 2013-10-23 22:09 Brian Norris
2013-10-24 1:31 ` Brian Norris
0 siblings, 1 reply; 4+ messages in thread
From: Brian Norris @ 2013-10-23 22:09 UTC (permalink / raw)
To: Artem Bityutskiy; +Cc: David Woodhouse, linux-mtd
Hi all,
I'm running a 3.8 kernel with UBIFS. I can trigger some UBIFS asserts
after a few hours of running a filesystem benchmark (like bonnie++) in a
loop. I've tried backporting a few UBIFS bugfixes, e.g.:
commit 605c912bb843c024b1ed173dc427cd5c08e5d54d
Author: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Date: Fri Jun 28 14:15:15 2013 +0300
UBIFS: fix a horrid bug
commit 33f1a63ae84dfd9ad298cf275b8f1887043ced36
Author: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
Date: Fri Jun 28 14:15:14 2013 +0300
UBIFS: prepare to fix a horrid bug
They did not help.
I have run various tests (including the MTD read/page/stress test
modules as well as a few hand-rolled user-space tests), and my driver
seems to be working fine.
I am not able to reproduce the failure on nandsim with either my 3.8
kernel/embedded hardware or with Linus' head of tree on my PC. Perhaps
there's something related to NAND geometry or to driver timing that
uncovers the filesystem bug.
Do you have any idea what would cause this type of problem?
Thanks,
Brian
NAND chip:
NAND device: Manufacturer ID: 0x2c, Chip ID: 0x44 (Micron MT29F32G08CBADAWP), 4096MiB, page size: 8192, OOB size: 744
Log:
(I didn't enable UBI/UBIFS debugging messages yet, because the test
takes a few hours to trigger this, and I would be overwhelmed with logs
before I could here...)
UBIFS assert failed in ubifs_tnc_next_ent at 2776 (pid 2143)
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270)
[<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
UBIFS error (pid 2143): ubifs_validate_entry: bad extended attribute entry node
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0)
[<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0) from [<c01a5670>] (matches_name.clone.22+0xbc/0xd0)
[<c01a5670>] (matches_name.clone.22+0xbc/0xd0) from [<c01a56cc>] (resolve_collision+0x48/0x304)
[<c01a56cc>] (resolve_collision+0x48/0x304) from [<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270)
[<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
magic 0x6101831
crc 0x2715b167
node_type 0 (inode node)
group_type 2 (last of node group)
sqnum 80750934
len 160
key (5899336, inode)
creat_sqnum 80641253
size 236552
nlink 2
atime 13941.0
mtime 13963.0
ctime 13963.0
uid 0
gid 0
mode 16832
flags 0x1
xattr_cnt 0
xattr_size 0
xattr_names 0
compr_type 0x0
data len 0
UBIFS error (pid 2143): ubifs_readdir: cannot find next direntry, error -22
UBIFS assert failed in ubifs_tnc_next_ent at 2776 (pid 2143)
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270)
[<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
UBIFS error (pid 2143): ubifs_validate_entry: bad extended attribute entry node
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0)
[<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0) from [<c01a5670>] (matches_name.clone.22+0xbc/0xd0)
[<c01a5670>] (matches_name.clone.22+0xbc/0xd0) from [<c01a56cc>] (resolve_collision+0x48/0x304)
[<c01a56cc>] (resolve_collision+0x48/0x304) from [<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270)
[<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
magic 0x6101831
crc 0xef854d68
node_type 0 (inode node)
group_type 2 (last of node group)
sqnum 80750937
len 160
key (5899336, inode)
creat_sqnum 80641253
size 236488
nlink 2
atime 13941.0
mtime 13963.0
ctime 13963.0
uid 0
gid 0
mode 16832
flags 0x1
xattr_cnt 0
xattr_size 0
xattr_names 0
compr_type 0x0
data len 0
UBIFS error (pid 2143): ubifs_readdir: cannot find next direntry, error -22
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: UBIFS: assert failed in ubifs_tnc_next_ent
2013-10-23 22:09 UBIFS: assert failed in ubifs_tnc_next_ent Brian Norris
@ 2013-10-24 1:31 ` Brian Norris
2013-10-24 16:41 ` Brian Norris
0 siblings, 1 reply; 4+ messages in thread
From: Brian Norris @ 2013-10-24 1:31 UTC (permalink / raw)
To: Artem Bityutskiy; +Cc: David Woodhouse, linux-mtd
On Wed, Oct 23, 2013 at 03:09:50PM -0700, Brian Norris wrote:
> I am not able to reproduce the failure on nandsim with either my 3.8
> kernel/embedded hardware or with Linus' head of tree on my PC. Perhaps
> there's something related to NAND geometry or to driver timing that
> uncovers the filesystem bug.
Scratch that. It just took a lot longer (over 700 runs of bonnie++ vs.
100 to 200 on real hardware), but I was able to reproduce the bug with
nandsim on 3.8. I'll retry on 3.12-rc and with a few extra backported
fixes on 3.8.
The asserts using nandsim:
UBIFS assert failed in ubifs_tnc_next_ent at 2776 (pid 4262)
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270)
[<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
UBIFS error (pid 4262): ubifs_validate_entry: bad extended attribute entry node
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0)
[<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0) from [<c01a5670>] (matches_name.clone.22+0xbc/0xd0)
[<c01a5670>] (matches_name.clone.22+0xbc/0xd0) from [<c01a56cc>] (resolve_collision+0x48/0x304)
[<c01a56cc>] (resolve_collision+0x48/0x304) from [<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270)
[<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
magic 0x6101831
crc 0xf87567a6
node_type 0 (inode node)
group_type 2 (last of node group)
sqnum 324111858
len 160
key (23300960, inode)
creat_sqnum 324002935
size 263248
nlink 2
atime 20882.0
mtime 20884.0
ctime 20884.0
uid 0
gid 0
mode 16832
flags 0x1
xattr_cnt 0
xattr_size 0
xattr_names 0
compr_type 0x0
data len 0
UBIFS error (pid 4262): ubifs_readdir: cannot find next direntry, error -22
UBIFS assert failed in ubifs_tnc_next_ent at 2776 (pid 4262)
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270)
[<c01a81f0>] (ubifs_tnc_next_ent+0x1c4/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
UBIFS error (pid 4262): ubifs_validate_entry: bad extended attribute entry node
[<c0015efc>] (unwind_backtrace+0x0/0x138) from [<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0)
[<c01a5570>] (lnc_add_directly.clone.21+0x7c/0xc0) from [<c01a5670>] (matches_name.clone.22+0xbc/0xd0)
[<c01a5670>] (matches_name.clone.22+0xbc/0xd0) from [<c01a56cc>] (resolve_collision+0x48/0x304)
[<c01a56cc>] (resolve_collision+0x48/0x304) from [<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270)
[<c01a8174>] (ubifs_tnc_next_ent+0x148/0x270) from [<c0198238>] (ubifs_readdir+0x2e4/0x5a0)
[<c0198238>] (ubifs_readdir+0x2e4/0x5a0) from [<c00bc8e8>] (vfs_readdir+0x90/0xac)
[<c00bc8e8>] (vfs_readdir+0x90/0xac) from [<c00bc9e8>] (sys_getdents+0x74/0xcc)
[<c00bc9e8>] (sys_getdents+0x74/0xcc) from [<c000ebc0>] (ret_fast_syscall+0x0/0x30)
magic 0x6101831
crc 0x76707438
node_type 0 (inode node)
group_type 2 (last of node group)
sqnum 324111861
len 160
key (23300960, inode)
creat_sqnum 324002935
size 263184
nlink 2
atime 20882.0
mtime 20884.0
ctime 20884.0
uid 0
gid 0
mode 16832
flags 0x1
xattr_cnt 0
xattr_size 0
xattr_names 0
compr_type 0x0
data len 0
UBIFS error (pid 4262): ubifs_readdir: cannot find next direntry, error -22
Brian
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: UBIFS: assert failed in ubifs_tnc_next_ent
2013-10-24 1:31 ` Brian Norris
@ 2013-10-24 16:41 ` Brian Norris
2013-10-24 17:31 ` Ezequiel Garcia
0 siblings, 1 reply; 4+ messages in thread
From: Brian Norris @ 2013-10-24 16:41 UTC (permalink / raw)
To: Artem Bityutskiy; +Cc: David Woodhouse, linux-mtd
On Wed, Oct 23, 2013 at 06:31:50PM -0700, Brian Norris wrote:
> I'll retry on 3.12-rc and with a few extra backported
> fixes on 3.8.
It looks like I was just missing a few upstream bugfixes. That's what I
get for not properly following -stable properly :(
With the additional backporting of these two commits, my filesystem
tests survived overnight:
commit 2928f0d0c5ebd6c9605c0d98207a44376387c298
Author: Adam Thomas <adamthomas1111@gmail.com>
Date: Sat Feb 2 22:32:31 2013 +0000
UBIFS: fix use of freed ubifs_orphan objects
commit 8afd500cb52a5d00bab4525dd5a560d199f979b9
Author: Adam Thomas <adamthomas1111@gmail.com>
Date: Sat Feb 2 22:35:08 2013 +0000
UBIFS: fix double free of ubifs_orphan objects
Sorry for the noise. I guess this is a lesson for others, though. Greg
KH isn't joking when he says "all users must upgrade"!
Brian
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: UBIFS: assert failed in ubifs_tnc_next_ent
2013-10-24 16:41 ` Brian Norris
@ 2013-10-24 17:31 ` Ezequiel Garcia
0 siblings, 0 replies; 4+ messages in thread
From: Ezequiel Garcia @ 2013-10-24 17:31 UTC (permalink / raw)
To: Brian Norris; +Cc: linux-mtd, David Woodhouse, Artem Bityutskiy
On Thu, Oct 24, 2013 at 09:41:44AM -0700, Brian Norris wrote:
> On Wed, Oct 23, 2013 at 06:31:50PM -0700, Brian Norris wrote:
> > I'll retry on 3.12-rc and with a few extra backported
> > fixes on 3.8.
>
> It looks like I was just missing a few upstream bugfixes. That's what I
> get for not properly following -stable properly :(
>
> With the additional backporting of these two commits, my filesystem
> tests survived overnight:
>
> commit 2928f0d0c5ebd6c9605c0d98207a44376387c298
> Author: Adam Thomas <adamthomas1111@gmail.com>
> Date: Sat Feb 2 22:32:31 2013 +0000
>
> UBIFS: fix use of freed ubifs_orphan objects
>
> commit 8afd500cb52a5d00bab4525dd5a560d199f979b9
> Author: Adam Thomas <adamthomas1111@gmail.com>
> Date: Sat Feb 2 22:35:08 2013 +0000
>
> UBIFS: fix double free of ubifs_orphan objects
>
> Sorry for the noise. I guess this is a lesson for others, though. Greg
> KH isn't joking when he says "all users must upgrade"!
>
Interesting findings and interesting lesson :-)
I'll keep this in mind and thanks for sharing your findings!
--
Ezequiel García, Free Electrons
Embedded Linux, Kernel and Android Engineering
http://free-electrons.com
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2013-10-24 17:31 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2013-10-23 22:09 UBIFS: assert failed in ubifs_tnc_next_ent Brian Norris
2013-10-24 1:31 ` Brian Norris
2013-10-24 16:41 ` Brian Norris
2013-10-24 17:31 ` Ezequiel Garcia
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).