read-only FS with error -2

read-only FS with error -2

[Ben Greear]

Hello,

I did something to my OpenWRT system (running 4.14.78 + 4.19 backports kernel),
and it went read-only.  Upon reboot, it is still read-only.  Is there something
I can to do debug and/or fix this?

root@LF-R7800-6ab8:/home/lanforge# dmesg|grep -i ubif
[   11.595662] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 143
[   11.674298] UBIFS (ubi0:1): recovery needed
[   12.028518] UBIFS (ubi0:1): recovery completed
[   12.028598] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
[   12.031847] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
[   12.039815] UBIFS (ubi0:1): FS size: 83042304 bytes (79 MiB, 654 LEBs), journal size 4190208 bytes (3 MiB, 33 LEBs)
[   12.049711] UBIFS (ubi0:1): reserved for root: 3922293 bytes (3830 KiB)
[   12.059957] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 036CCB1F-5608-4069-B5A0-D22544398634, small LPT model
[   12.085050] mount_root: switching to ubifs overlay
[   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2
[   31.942588] UBIFS error (ubi0:1 pid 570): ubifs_lookup: dead directory entry 'dhcpd', error -2
[   31.949813] UBIFS warning (ubi0:1 pid 570): ubifs_ro_mode.part.0: switched to read-only mode, error -2
[   31.985721] [<c0782b58>] (dump_stack) from [<c0472824>] (ubifs_lookup+0x2d8/0x394)
[   31.992757] [<c0472824>] (ubifs_lookup) from [<c03f70b0>] (lookup_slow+0xe0/0x15c)
[   31.152936] kmodloader: done loading kernel modules from /etc/modules.d/*


And the backtrace in question:

....
[   31.387880] Unsafe core_pattern used with fs.suid_dumpable=2.
[   31.387880] Pipe handler or fully qualified core dump path required.
[   31.387880] Set kernel.core_pattern before fs.suid_dumpable.
[   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2
[   31.942588] UBIFS error (ubi0:1 pid 570): ubifs_lookup: dead directory entry 'dhcpd', error -2
[   31.949813] UBIFS warning (ubi0:1 pid 570): ubifs_ro_mode.part.0: switched to read-only mode, error -2
[   31.958326] CPU: 1 PID: 570 Comm: sh Not tainted 4.14.78 #0
[   31.967477] Hardware name: Generic DT based system
[   31.972987] [<c030f58c>] (unwind_backtrace) from [<c030b7a4>] (show_stack+0x14/0x20)
[   31.977827] [<c030b7a4>] (show_stack) from [<c0782b58>] (dump_stack+0x88/0x9c)
[   31.985721] [<c0782b58>] (dump_stack) from [<c0472824>] (ubifs_lookup+0x2d8/0x394)
[   31.992757] [<c0472824>] (ubifs_lookup) from [<c03f70b0>] (lookup_slow+0xe0/0x15c)
[   32.000309] [<c03f70b0>] (lookup_slow) from [<c03f8a84>] (lookup_one_len_unlocked+0xe8/0x104)
[   32.007866] [<c03f8a84>] (lookup_one_len_unlocked) from [<c049f6f8>] (ovl_lookup_single+0x28/0x2dc)
[   32.016457] [<c049f6f8>] (ovl_lookup_single) from [<c049f9fc>] (ovl_lookup_layer+0x50/0x178)
[   32.025308] [<c049f9fc>] (ovl_lookup_layer) from [<c049ffb8>] (ovl_lookup+0xac/0x670)
[   32.033988] [<c049ffb8>] (ovl_lookup) from [<c03f70b0>] (lookup_slow+0xe0/0x15c)
[   32.041711] [<c03f70b0>] (lookup_slow) from [<c03f9644>] (walk_component+0x108/0x2fc)
[   32.049175] [<c03f9644>] (walk_component) from [<c03f9ee0>] (path_lookupat+0x164/0x1e4)
[   32.056904] [<c03f9ee0>] (path_lookupat) from [<c03fbca8>] (filename_lookup+0x90/0xf0)
[   32.064725] [<c03fbca8>] (filename_lookup) from [<c03f1314>] (vfs_statx+0x68/0xcc)
[   32.072704] [<c03f1314>] (vfs_statx) from [<c03f1ad8>] (SyS_stat64+0x2c/0x50)
[   32.080257] [<c03f1ad8>] (SyS_stat64) from [<c0307e60>] (ret_fast_syscall+0x0/0x54)
....

Any help will be appreciated!

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: read-only FS with error -2

[Richard Weinberger]

Ben,

On Fri, Oct 26, 2018 at 8:39 PM Ben Greear <greearb@candelatech.com> wrote:
>
> Hello,
>
> I did something to my OpenWRT system (running 4.14.78 + 4.19 backports kernel),

What did you do? :)

> and it went read-only.  Upon reboot, it is still read-only.  Is there something
> I can to do debug and/or fix this?
>
> root@LF-R7800-6ab8:/home/lanforge# dmesg|grep -i ubif
> [   11.595662] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 143
> [   11.674298] UBIFS (ubi0:1): recovery needed
> [   12.028518] UBIFS (ubi0:1): recovery completed
> [   12.028598] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
> [   12.031847] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
> [   12.039815] UBIFS (ubi0:1): FS size: 83042304 bytes (79 MiB, 654 LEBs), journal size 4190208 bytes (3 MiB, 33 LEBs)
> [   12.049711] UBIFS (ubi0:1): reserved for root: 3922293 bytes (3830 KiB)
> [   12.059957] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 036CCB1F-5608-4069-B5A0-D22544398634, small LPT model
> [   12.085050] mount_root: switching to ubifs overlay
> [   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2

Hmm, UBIFS is unable to find a inode on your MTD.
Can you please mount UBIFS again with "/sys/kernel/debug/ubifs/chk_fs" set to 1?
This will run a self-check and give us maybe more details.

Did you see in the past odd logs? ECC errors or such?

-- 
Thanks,
//richard

Re: read-only FS with error -2

[Ben Greear]

On 10/27/2018 01:36 PM, Richard Weinberger wrote:
> Ben,
>
> On Fri, Oct 26, 2018 at 8:39 PM Ben Greear <greearb@candelatech.com> wrote:
>>
>> Hello,
>>
>> I did something to my OpenWRT system (running 4.14.78 + 4.19 backports kernel),
>
> What did you do? :)

Well, I created 64 station vdevs on an ath10k NIC, which if nothing else, was hitting a zillion
WARN_ON bugs in the mac80211 stack.  That flooded dmesg so I don't know if there
were any more serious bugs earlier on.

>> and it went read-only.  Upon reboot, it is still read-only.  Is there something
>> I can to do debug and/or fix this?
>>
>> root@LF-R7800-6ab8:/home/lanforge# dmesg|grep -i ubif
>> [   11.595662] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 143
>> [   11.674298] UBIFS (ubi0:1): recovery needed
>> [   12.028518] UBIFS (ubi0:1): recovery completed
>> [   12.028598] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
>> [   12.031847] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
>> [   12.039815] UBIFS (ubi0:1): FS size: 83042304 bytes (79 MiB, 654 LEBs), journal size 4190208 bytes (3 MiB, 33 LEBs)
>> [   12.049711] UBIFS (ubi0:1): reserved for root: 3922293 bytes (3830 KiB)
>> [   12.059957] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 036CCB1F-5608-4069-B5A0-D22544398634, small LPT model
>> [   12.085050] mount_root: switching to ubifs overlay
>> [   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2
>
> Hmm, UBIFS is unable to find a inode on your MTD.
> Can you please mount UBIFS again with "/sys/kernel/debug/ubifs/chk_fs" set to 1?
> This will run a self-check and give us maybe more details.
>
> Did you see in the past odd logs? ECC errors or such?

I reinstalled already, and the problem went away.  How exactly would I
do that test above?  The root file system is RO on bootup in the problem
case...would I remount root after twiddling the debugfs flag?

Thanks,
Ben

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: read-only FS with error -2

[Richard Weinberger]

Ben,

Am Sonntag, 28. Oktober 2018, 00:37:33 CET schrieb Ben Greear:
> > What did you do? :)
> 
> Well, I created 64 station vdevs on an ath10k NIC, which if nothing else, was hitting a zillion
> WARN_ON bugs in the mac80211 stack.  That flooded dmesg so I don't know if there
> were any more serious bugs earlier on.

Hmm, a WARN_ON itself is no problem but maybe the driver corrupted kernel memory and confused UBIFS.
I saw such issues more than once.
Linux is not a microkernel. :-)
 
> >> and it went read-only.  Upon reboot, it is still read-only.  Is there something
> >> I can to do debug and/or fix this?
> >>
> >> root@LF-R7800-6ab8:/home/lanforge# dmesg|grep -i ubif
> >> [   11.595662] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 143
> >> [   11.674298] UBIFS (ubi0:1): recovery needed
> >> [   12.028518] UBIFS (ubi0:1): recovery completed
> >> [   12.028598] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
> >> [   12.031847] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
> >> [   12.039815] UBIFS (ubi0:1): FS size: 83042304 bytes (79 MiB, 654 LEBs), journal size 4190208 bytes (3 MiB, 33 LEBs)
> >> [   12.049711] UBIFS (ubi0:1): reserved for root: 3922293 bytes (3830 KiB)
> >> [   12.059957] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 036CCB1F-5608-4069-B5A0-D22544398634, small LPT model
> >> [   12.085050] mount_root: switching to ubifs overlay
> >> [   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2
> >
> > Hmm, UBIFS is unable to find a inode on your MTD.
> > Can you please mount UBIFS again with "/sys/kernel/debug/ubifs/chk_fs" set to 1?
> > This will run a self-check and give us maybe more details.
> >
> > Did you see in the past odd logs? ECC errors or such?
> 
> I reinstalled already, and the problem went away.  How exactly would I
> do that test above?  The root file system is RO on bootup in the problem
> case...would I remount root after twiddling the debugfs flag?

You need to set the flag before mounting UBIFS.
If UBIFS is the rootfs, boot from somewhere else. e.g. NFS.
Or do it in an initramfs.

Thanks,
//richard

Re: read-only FS with error -2

[Ben Greear]

On 10/28/2018 04:17 AM, Richard Weinberger wrote:
> Ben,
>
> Am Sonntag, 28. Oktober 2018, 00:37:33 CET schrieb Ben Greear:
>>> What did you do? :)
>>
>> Well, I created 64 station vdevs on an ath10k NIC, which if nothing else, was hitting a zillion
>> WARN_ON bugs in the mac80211 stack.  That flooded dmesg so I don't know if there
>> were any more serious bugs earlier on.
>
> Hmm, a WARN_ON itself is no problem but maybe the driver corrupted kernel memory and confused UBIFS.
> I saw such issues more than once.
> Linux is not a microkernel. :-)
>
>>>> and it went read-only.  Upon reboot, it is still read-only.  Is there something
>>>> I can to do debug and/or fix this?
>>>>
>>>> root@LF-R7800-6ab8:/home/lanforge# dmesg|grep -i ubif
>>>> [   11.595662] UBIFS (ubi0:1): background thread "ubifs_bgt0_1" started, PID 143
>>>> [   11.674298] UBIFS (ubi0:1): recovery needed
>>>> [   12.028518] UBIFS (ubi0:1): recovery completed
>>>> [   12.028598] UBIFS (ubi0:1): UBIFS: mounted UBI device 0, volume 1, name "rootfs_data"
>>>> [   12.031847] UBIFS (ubi0:1): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes
>>>> [   12.039815] UBIFS (ubi0:1): FS size: 83042304 bytes (79 MiB, 654 LEBs), journal size 4190208 bytes (3 MiB, 33 LEBs)
>>>> [   12.049711] UBIFS (ubi0:1): reserved for root: 3922293 bytes (3830 KiB)
>>>> [   12.059957] UBIFS (ubi0:1): media format: w4/r0 (latest is w5/r0), UUID 036CCB1F-5608-4069-B5A0-D22544398634, small LPT model
>>>> [   12.085050] mount_root: switching to ubifs overlay
>>>> [   31.942522] UBIFS error (ubi0:1 pid 570): ubifs_iget: failed to read inode 1623, error -2
>>>
>>> Hmm, UBIFS is unable to find a inode on your MTD.
>>> Can you please mount UBIFS again with "/sys/kernel/debug/ubifs/chk_fs" set to 1?
>>> This will run a self-check and give us maybe more details.
>>>
>>> Did you see in the past odd logs? ECC errors or such?
>>
>> I reinstalled already, and the problem went away.  How exactly would I
>> do that test above?  The root file system is RO on bootup in the problem
>> case...would I remount root after twiddling the debugfs flag?
>
> You need to set the flag before mounting UBIFS.
> If UBIFS is the rootfs, boot from somewhere else. e.g. NFS.
> Or do it in an initramfs.

That is not an easy thing to do on random embedded systems.  Possibly I could add a kernel
command line option...is that supported?

Thanks,
Ben

>
> Thanks,
> //richard
>
>
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: read-only FS with error -2

[Richard Weinberger]

Ben,

Am Sonntag, 28. Oktober 2018, 16:08:13 CET schrieb Ben Greear: 
> That is not an easy thing to do on random embedded systems.  Possibly I could add a kernel

Debugging random non-development systems is never easy.

> command line option...is that supported?

The UBIFS module takes no parameters. But patches are welcome.
For now you can just use your initramfs (AFAIK OpenWRT has one), boot with rdinit=/bin/sh ...

Or apply a hacky patch like this one to enable chk_fs always...

diff --git a/fs/ubifs/debug.h b/fs/ubifs/debug.h
index 64c6977c189b..ea0ffc0a9660 100644
--- a/fs/ubifs/debug.h
+++ b/fs/ubifs/debug.h
@@ -231,7 +231,7 @@ static inline int dbg_is_chk_lprops(const struct ubifs_info *c)
 }
 static inline int dbg_is_chk_fs(const struct ubifs_info *c)
 {
-	return !!(ubifs_dbg.chk_fs || c->dbg->chk_fs);
+	return 1;
 }
 static inline int dbg_is_tst_rcvry(const struct ubifs_info *c)
 {

Thanks,
//richard

Re: read-only FS with error -2

[Ben Greear]

On 10/28/2018 08:21 AM, Richard Weinberger wrote:
> Ben,
>
> Am Sonntag, 28. Oktober 2018, 16:08:13 CET schrieb Ben Greear:
>> That is not an easy thing to do on random embedded systems.  Possibly I could add a kernel
>
> Debugging random non-development systems is never easy.
>
>> command line option...is that supported?
>
> The UBIFS module takes no parameters. But patches are welcome.

I'm fully occupied with the wifi stack issues...hoping other people
can work on different issues.

> For now you can just use your initramfs (AFAIK OpenWRT has one), boot with rdinit=/bin/sh ...
>
> Or apply a hacky patch like this one to enable chk_fs always...

Problem is, when I re-install then the problem goes away.  Would it
be bad to enable this code all the time on production systems?

Could UBIFS just enable debugging and print out any pertinent
info when it detects the failure instead of (just) going into RO
mode?

Thanks,
Ben

>
> diff --git a/fs/ubifs/debug.h b/fs/ubifs/debug.h
> index 64c6977c189b..ea0ffc0a9660 100644
> --- a/fs/ubifs/debug.h
> +++ b/fs/ubifs/debug.h
> @@ -231,7 +231,7 @@ static inline int dbg_is_chk_lprops(const struct ubifs_info *c)
>  }
>  static inline int dbg_is_chk_fs(const struct ubifs_info *c)
>  {
> -	return !!(ubifs_dbg.chk_fs || c->dbg->chk_fs);
> +	return 1;
>  }
>  static inline int dbg_is_tst_rcvry(const struct ubifs_info *c)
>  {
>
> Thanks,
> //richard
>
>

-- 
Ben Greear <greearb@candelatech.com>
Candela Technologies Inc  http://www.candelatech.com

Re: read-only FS with error -2

[Richard Weinberger]

Ben,

Am Sonntag, 28. Oktober 2018, 19:44:46 CET schrieb Ben Greear:
> > For now you can just use your initramfs (AFAIK OpenWRT has one), boot with rdinit=/bin/sh ...
> >
> > Or apply a hacky patch like this one to enable chk_fs always...
> 
> Problem is, when I re-install then the problem goes away.  Would it
> be bad to enable this code all the time on production systems?

Did you face the problem more than *once*?

chk_fs makes mounting very slow, it is basically a fsck upon mount.
 
> Could UBIFS just enable debugging and print out any pertinent
> info when it detects the failure instead of (just) going into RO
> mode?

UBIFS prints a ton on debug information upon unexpected situations.
But if your logs are gone and you have only very the last one (after a reboot),
UBIFS cannot do much...

Thanks,
//richard