UBIFS corruptions - update

UBIFS corruptions - update

[Manfred Spraul]

Hi Richard,

as promised I tried to add backtrace support to my mtdram modification.

I don't like the result, the workflow is way to complex (good old 
ksymoops, ...)

Just dump_stack(), and then retain the output from dmesg would be simple 
and more efficient.

Thus I have abandoned the patch.


What I have are the results:

- chk_fs corruptions: With 4.15-rc2, the issue still exists.

The corruptions start when the background worker thread erases a page:

1st instance:

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-3754.bin/download

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/rec-syms-3753.txt/download

2nd instance:

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-5311.bin/download


- the garbage_collect_leb issue:

The issue happens during ubifs_tnc_end_commit(), called by sync().

I'm not sure about the root cause: Should ubifs_check_node() skip the 
index LEB? Or should the nodes never cross a page boundary?

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-167836.bin/download

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/rec-syms-167837.txt/download

https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/r-167836.txt/download


Without xattr, the garbage_collect_leb issue did not happen, I tested 
around 40.000 images.


@Richard: Is there something where I could support? I fear more data 
would just cause confusion.

For me, there appears to be two reproducible issues:

- garbage_collect_leb: Fairly rare

- Failure with chk_fs enabled. Easy to reproduce, more common.


--

     Manfred

Re: UBIFS corruptions - update

[Richard Weinberger]

Hi Manfred,

Am Mittwoch, 13. Dezember 2017, 20:29:54 CET schrieb Manfred Spraul:
> Hi Richard,
> 
> as promised I tried to add backtrace support to my mtdram modification.
> 
> I don't like the result, the workflow is way to complex (good old
> ksymoops, ...)
> 
> Just dump_stack(), and then retain the output from dmesg would be simple
> and more efficient.
> 
> Thus I have abandoned the patch.
> 
> 
> What I have are the results:
> 
> - chk_fs corruptions: With 4.15-rc2, the issue still exists.
> 
> The corruptions start when the background worker thread erases a page:
> 
> 1st instance:
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-3
> 754.bin/download
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/rec-sym
> s-3753.txt/download
> 
> 2nd instance:
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-5
> 311.bin/download
> 
> 
> - the garbage_collect_leb issue:
> 
> The issue happens during ubifs_tnc_end_commit(), called by sync().
> 
> I'm not sure about the root cause: Should ubifs_check_node() skip the
> index LEB? Or should the nodes never cross a page boundary?
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/image-1
> 67836.bin/download
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/rec-sym
> s-167837.txt/download
> 
> https://sourceforge.net/projects/calculix-rpm/files/ubifs-2/11_xattr/r-16783
> 6.txt/download
> 
> 
> Without xattr, the garbage_collect_leb issue did not happen, I tested
> around 40.000 images.
> 
> 
> @Richard: Is there something where I could support? I fear more data
> would just cause confusion.

A day with more than 24h would be a good start. ;-)

I was able to play with your replay program and found a problem in UBIFS.
So I confirm we do have a problem with xattrs. The problem is the way how 
UBIFS kills all inodes that represent xattrs when a file is unlinked.
With a small enough LEB size and many "perfectly" timed power-cuts we can end 
up with a situation where UBIFS forgets about the relationship between an 
inode and it's xattrs inodes. This will later confuse the free space 
accounting (the LEB property tree) and then cause UBIFS to unmap a wrong LEB.

While playing with your replay tool I've reworked also your flight recorder to 
operate directly in MTD core and the tracing ring buffer. 
I'll send patches soon. This way of power-cut testing is really nice, and I 
want this mainline and part of mtd-utils.

Thanks,
//richard