Disappearing directories

Disappearing directories

[Christoph Mammitzsch]

[-- Attachment #1: Type: text/plain, Size: 2955 bytes --]

Hello,

I think I might have stumbled across a bug in jffs2, that caused our
JFFS2 NAND flash partitions to lose subdirectories now and then.

The embedded system in question was running on linux 2.6.38.2, but using
the nandsim module, I was able to recreate the bug on workstations
running linux 3.0.0 as well as linux 3.2.0.

The attached script should demonstrate the bug, however since the ways
of the garbage collector are not exactly deterministic, it might not
work every time. When the bug hits, the following lines can be found in
dmesg:
> [ 7917.087199] JFFS2 notice: (9354) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
> [ 7917.433015] JFFS2 error: (9409) jffs2_build_inode_pass1: child dir "subdir" (ino #6) of dir ino #2 appears to be a hard link
> [ 7917.433058] JFFS2 notice: (9409) jffs2_build_xattr_subsystem: complete building xattr subsystem, 0 of xdatum (0 unchecked, 0 orphan) and 0 of xref (0 dead, 0 orphan) found.
> [ 7917.436272] JFFS2 warning: (9411) jffs2_get_inode_nodes: Eep. No valid nodes for ino #6.
> [ 7917.436277] JFFS2 warning: (9411) jffs2_do_read_inode_internal: no data nodes found for ino #6

Here is, what I think happens:

"mkdir tmpdir" creates inode 2 and links to it from inode 1 using the
name tmpdir. This entry shares an eraseblock with the DELETE[12] files.
"mkdir tmpdir/subdir" creates inode 6 and links to it from inode 2 using
the name tmpdir. This entry shares an eraseblock with the KEEP[12] files.
"mv tmpdir/subdir ." and "rmdir tmpdir" create three consecutive log
entries:
 - create new link to inode 6 from inode 1 using the name subdir
 - delete entry named subdir from inode 2
 - delete entry named tmpdir from inode 1
These these three steps share an eraseblock with DELETE[34]

Now after "rm DELETE*", the creation of inode 6 and the link from inode
2 to it share an eraseblock with lots of valid data, while all the other
entries sit in very dirty eraseblocks. These eraseblocks are very likely
to be garbage collected during the following "cp ../rnd DELETE" sequence.

If that happens, inode 2 as well as the link to it vanish. The deletion
of the link from inode 2 to inode 6 also disappears. However the
creation of a link from inode 2 to inode 6 still remains in flash, since
it is "protected" by the heap of valid data.

This does not cause any trouble as long as the filesystem stays mounted.
However if you unmount the partition at this point and then remount it,
jffs2 can't get around the fact, that there are now two links to inode
6, even if one is from a directory, that itself doesn't exist anymore.


Best Regards,

i. A. Christoph Mammitzsch
-- 
Symeo GmbH
Professor-Messerschmitt-Str. 3
85579 Neubiberg / München
phone: +49 (89) 6607796-301

Symeo GmbH; Managing Directors: Dirk Brunnengräber, Christoph Rommel;
Registered Office: Munich; Commercial Register: Munich, HRB 157340

[-- Attachment #2: trigger_bug.sh --]
[-- Type: text/plain, Size: 1563 bytes --]

#!/bin/sh

# load needed modules
modprobe nandsim first_id_byte=0x2c second_id_byte=0xa1 third_id_byte=0x80 fourth_id_byte=0x15 parts=128
modprobe mtdblock

# generate some arbitrary data
dd if=/dev/urandom of=rnd bs=2048 count=64

# clean and mount simulated NAND partition
flash_erase /dev/mtd0 0 0
mkdir -p mnt_jffs2
mount -t jffs2 /dev/mtdblock0 mnt_jffs2

# set up a critical node structure
cd mnt_jffs2
mkdir tmpdir
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd KEEP1
mkdir tmpdir/subdir
cp ../rnd KEEP2
cp ../rnd DELETE3
mv tmpdir/subdir .
rmdir tmpdir
cp ../rnd DELETE4
rm DELETE*
# and excercise the garbage collector
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd DELETE3
cp ../rnd DELETE4
cp ../rnd DELETE5
cp ../rnd DELETE6
cp ../rnd DELETE7
cp ../rnd DELETE8
cp ../rnd DELETE9
cp ../rnd DELETE1
cp ../rnd DELETE2
cp ../rnd DELETE3
cp ../rnd DELETE4
cp ../rnd DELETE5
cp ../rnd DELETE6
cp ../rnd DELETE7
cp ../rnd DELETE8
cp ../rnd DELETE9
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
cp ../rnd DELETE
rm DELETE*
cd ..

# unmount and remount partition. Notice how ls barfs after remount, but not before.
ls -l mnt_jffs2
umount mnt_jffs2
mount -t jffs2 /dev/mtdblock0 mnt_jffs2
ls -l mnt_jffs2

# unmount again for next run
umount mnt_jffs2

Re: Disappearing directories (jffs2 on nand flash)

[Christoph Mammitzsch]

Am 04.07.2014 19:00, schrieb Christoph Mammitzsch:

> The embedded system in question was running on linux 2.6.38.2, but using
> the nandsim module, I was able to recreate the bug on workstations
> running linux 3.0.0 as well as linux 3.2.0.

Update: I compiled the newest linux kernel (commit
812c5fa82bae9f377f49000d7636c19a8b61735c) and ran the test script on
that kernel. The bug still showed 9 out of 10 times.


Regards,

Christoph Mammitzsch

Re: Disappearing directories (jffs2 on nand flash)

[Christoph Mammitzsch]

Hello,

I am a little astonished at the amount of feedback I got for my
bugreport. Apparently jffs2 losing entire directories is not as big a
deal as I thought. Does that mean that JFFS2 is dead and I should use
another flash filesystem instead, or did I simply post on the wrong
mailing list?


Regards,

Christoph Mammitzsch

Re: Disappearing directories (jffs2 on nand flash)

[Richard Weinberger]

On Sat, Aug 2, 2014 at 9:21 PM, Christoph Mammitzsch
<Christoph.Mammitzsch@symeo.com> wrote:
> Hello,
>
> I am a little astonished at the amount of feedback I got for my
> bugreport. Apparently jffs2 losing entire directories is not as big a
> deal as I thought. Does that mean that JFFS2 is dead and I should use
> another flash filesystem instead, or did I simply post on the wrong
> mailing list?

Is there any reason why you are not using UBIFS?

-- 
Thanks,
//richard

Re: Disappearing directories (jffs2 on nand flash)

[Christoph Mammitzsch]

Am 03.08.2014 01:13, schrieb Richard Weinberger:
> On Sat, Aug 2, 2014 at 9:21 PM, Christoph Mammitzsch
> <Christoph.Mammitzsch@symeo.com> wrote:
>> Hello,
>>
>> I am a little astonished at the amount of feedback I got for my
>> bugreport. Apparently jffs2 losing entire directories is not as big a
>> deal as I thought. Does that mean that JFFS2 is dead and I should use
>> another flash filesystem instead, or did I simply post on the wrong
>> mailing list?
> 
> Is there any reason why you are not using UBIFS?

We introduced our first linux based product 2006, so UBIFS was not an
option. Our second linux based product was designed in 2011. Back then
UBIFS was still relatively new, and it seemed beneficial to keep
differences between our two products minimal, so we kept using JFFS2.

Switching to UBIFS now means we would have to update and test quite a
lot of scripts. And providing an update for existing stations would be
even more problematic. Therefore I was hoping for a bugfix, so we could
just deliver a new kernel image and be done with it.


Regards,

Christoph Mammitzsch

Re: Disappearing directories (jffs2 on nand flash)

[Richard Weinberger]

Am 05.08.2014 11:22, schrieb Christoph Mammitzsch:
> Am 03.08.2014 01:13, schrieb Richard Weinberger:
>> On Sat, Aug 2, 2014 at 9:21 PM, Christoph Mammitzsch
>> <Christoph.Mammitzsch@symeo.com> wrote:
>>> Hello,
>>>
>>> I am a little astonished at the amount of feedback I got for my
>>> bugreport. Apparently jffs2 losing entire directories is not as big a
>>> deal as I thought. Does that mean that JFFS2 is dead and I should use
>>> another flash filesystem instead, or did I simply post on the wrong
>>> mailing list?
>>
>> Is there any reason why you are not using UBIFS?
> 
> We introduced our first linux based product 2006, so UBIFS was not an
> option. Our second linux based product was designed in 2011. Back then
> UBIFS was still relatively new, and it seemed beneficial to keep
> differences between our two products minimal, so we kept using JFFS2.
> 
> Switching to UBIFS now means we would have to update and test quite a
> lot of scripts. And providing an update for existing stations would be
> even more problematic. Therefore I was hoping for a bugfix, so we could
> just deliver a new kernel image and be done with it.

I can understand your decisions.
TBH jffs2 is more of less abandoned, don't expect anyone to fix issues fast.

Is the issue you see a regression?
If you can find the first bad commit it might give someone a hint...
git bisect will help you.

Thanks,
//richard