[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

Hi u-booters,

I have a short script in my u-boot environment which chooses which of
two ubifs partitions to boot
by attempting to read a release file in each one.

Unfortunately, after an unclean shutdown sometimes the ubifsmount
fails. ?(By "unclean shutdown"
I mean that the board was power cycled while doing some low bandwidth logging.)

The strange thing is that Linux has no problem mounting the partition
as its root filesystem. ?This is
very confusing because it looks like the ubifs implementation in
u-boot is just a copy of the one in Linux.

Has anyone else seen this problem?

Regards,

Alex

PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:


> U-Boot 2011.06 (Feb 10 2012 - 12:29:06)
> OpenRD-Base
> SoC: ? Kirkwood 88F6281_A1
> DRAM: ?128 MiB
> NAND: ?512 MiB
> *** Warning - bad CRC, using default environment
> In: ? ?serial
> Out: ? serial
> Err: ? serial
> Net: ? egiga0
> 88E6351 Initialized on egiga0
> Hit any key to stop autoboot: ?0
> Creating 1 MTD partitions on "nand0":
> 0x000001000000-0x000010000000 : "mtd=2"
> UBI: attaching mtd1 to ubi0
> UBI: physical eraseblock size: ? 131072 bytes (128 KiB)
> UBI: logical eraseblock size: ? ?126976 bytes
> UBI: smallest flash I/O unit: ? ?2048
> UBI: sub-page size: ? ? ? ? ? ? ?512
> UBI: VID header offset: ? ? ? ? ?2048 (aligned 2048)
> UBI: data offset: ? ? ? ? ? ? ? ?4096
> UBI: attached mtd1 to ubi0
> UBI: MTD device name: ? ? ? ? ? ?"mtd=2"
> UBI: MTD device size: ? ? ? ? ? ?240 MiB
> UBI: number of good PEBs: ? ? ? ?1913
> UBI: number of bad PEBs: ? ? ? ? 7
> UBI: max. allowed volumes: ? ? ? 128
> UBI: wear-leveling threshold: ? ?4096
> UBI: number of internal volumes: 1
> UBI: number of user volumes: ? ? 1
> UBI: available PEBs: ? ? ? ? ? ? 0
> UBI: total number of reserved PEBs: 1913
> UBI: number of PEBs reserved for bad PEB handling: 19
> UBI: max/mean erase counter: 7/1
> UBIFS: recovery needed
> Error reading superblock on volume 'ubi:rootfs'!
> UBI: mtd1 is detached from ubi0

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

On 7 March 2012 14:42, Alex Zeffertt <azeffertt@cambridgesys.com> wrote:
> Hi u-booters,
>
> I have a short script in my u-boot environment which chooses which of
> two ubifs partitions to boot
> by attempting to read a release file in each one.
>
> Unfortunately, after an unclean shutdown sometimes the ubifsmount
> fails. ?(By "unclean shutdown"
> I mean that the board was power cycled while doing some low bandwidth logging.)
>
> The strange thing is that Linux has no problem mounting the partition
> as its root filesystem. ?This is
> very confusing because it looks like the ubifs implementation in
> u-boot is just a copy of the one in Linux.
>
> Has anyone else seen this problem?
>
> Regards,
>
> Alex
>
> PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:
>
>
>> U-Boot 2011.06 (Feb 10 2012 - 12:29:06)
>> OpenRD-Base
>> SoC: ? Kirkwood 88F6281_A1
>> DRAM: ?128 MiB
>> NAND: ?512 MiB
>> *** Warning - bad CRC, using default environment
>> In: ? ?serial
>> Out: ? serial
>> Err: ? serial
>> Net: ? egiga0
>> 88E6351 Initialized on egiga0
>> Hit any key to stop autoboot: ?0
>> Creating 1 MTD partitions on "nand0":
>> 0x000001000000-0x000010000000 : "mtd=2"
>> UBI: attaching mtd1 to ubi0
>> UBI: physical eraseblock size: ? 131072 bytes (128 KiB)
>> UBI: logical eraseblock size: ? ?126976 bytes
>> UBI: smallest flash I/O unit: ? ?2048
>> UBI: sub-page size: ? ? ? ? ? ? ?512
>> UBI: VID header offset: ? ? ? ? ?2048 (aligned 2048)
>> UBI: data offset: ? ? ? ? ? ? ? ?4096
>> UBI: attached mtd1 to ubi0
>> UBI: MTD device name: ? ? ? ? ? ?"mtd=2"
>> UBI: MTD device size: ? ? ? ? ? ?240 MiB
>> UBI: number of good PEBs: ? ? ? ?1913
>> UBI: number of bad PEBs: ? ? ? ? 7
>> UBI: max. allowed volumes: ? ? ? 128
>> UBI: wear-leveling threshold: ? ?4096
>> UBI: number of internal volumes: 1
>> UBI: number of user volumes: ? ? 1
>> UBI: available PEBs: ? ? ? ? ? ? 0
>> UBI: total number of reserved PEBs: 1913
>> UBI: number of PEBs reserved for bad PEB handling: 19
>> UBI: max/mean erase counter: 7/1
>> UBIFS: recovery needed
>> Error reading superblock on volume 'ubi:rootfs'!
>> UBI: mtd1 is detached from ubi0

I've been comparing the linux and u-boot implementations, and it looks
like the following fix is in the kernel
but not in u-boot.  I don't really understand it, but it looks like a
candidate.  Might porting this change to
u-boot fix the issue?

 - Alex

commit 2ef13294d29bcfb306e0d360f1b97f37b647b0c0
Author: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>
Date:   Sun Sep 19 18:34:26 2010 +0300

    UBIFS: introduce new flags for RO mounts

    Commit 2fde99cb55fb9d9b88180512a5e8a5d939d27fec "UBIFS: mark VFS SB RO too"
    introduced regression. This commit made UBIFS set the 'MS_RDONLY'
flag in the
    VFS superblock when it switches to R/O mode due to an error. This was done
    to make VFS show the R/O UBIFS flag in /proc/mounts.

    However, several places in UBIFS relied on the 'MS_RDONLY' flag
and assume this
    flag can only change when we re-mount. For example, 'ubifs_put_super()'.

    This patch introduces new UBIFS flag - 'c->ro_mount' which changes only when
    we re-mount, and preserves the way UBIFS was originally mounted
(R/W or R/O).
    This allows us to de-initialize UBIFS cleanly in 'ubifs_put_super()'.

    This patch also changes all 'ubifs_assert(!c->ro_media)' assertions to
    'ubifs_assert(!c->ro_media && !c->ro_mount)', because we never should write
    anything if the FS was mounter R/O.

    All the places where we test for 'MS_RDONLY' flag in the VFS SB were changed
    and now we test the 'c->ro_mount' flag instead, because it preserves the
    original UBIFS mount type, unlike the 'MS_RDONLY' flag.

    Signed-off-by: Artem Bityutskiy <Artem.Bityutskiy@nokia.com>

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Stefan Roese]

Hi Alex,

On Wednesday 07 March 2012 16:14:28 Alex Zeffertt wrote:
> On 7 March 2012 14:42, Alex Zeffertt <azeffertt@cambridgesys.com> wrote:
> > Hi u-booters,
> > 
> > I have a short script in my u-boot environment which chooses which of
> > two ubifs partitions to boot
> > by attempting to read a release file in each one.
> > 
> > Unfortunately, after an unclean shutdown sometimes the ubifsmount
> > fails.  (By "unclean shutdown"
> > I mean that the board was power cycled while doing some low bandwidth
> > logging.)
> > 
> > The strange thing is that Linux has no problem mounting the partition
> > as its root filesystem.  This is
> > very confusing because it looks like the ubifs implementation in
> > u-boot is just a copy of the one in Linux.

Correct. The U-Boot version is mainly a copy of the Linux version. With minor 
changes. But its based on an older Linux version. Many changes have gone into 
the UBI Linux code.
 
> > Has anyone else seen this problem?
> > 
> > Regards,
> > 
> > Alex
> > 
> > PS My kernel is linux-3.0.0/armv5tel and the full u-boot trace is below:
> >> U-Boot 2011.06 (Feb 10 2012 - 12:29:06)
> >> OpenRD-Base
> >> SoC:   Kirkwood 88F6281_A1
> >> DRAM:  128 MiB
> >> NAND:  512 MiB
> >> *** Warning - bad CRC, using default environment
> >> In:    serial
> >> Out:   serial
> >> Err:   serial
> >> Net:   egiga0
> >> 88E6351 Initialized on egiga0
> >> Hit any key to stop autoboot:  0
> >> Creating 1 MTD partitions on "nand0":
> >> 0x000001000000-0x000010000000 : "mtd=2"
> >> UBI: attaching mtd1 to ubi0
> >> UBI: physical eraseblock size:   131072 bytes (128 KiB)
> >> UBI: logical eraseblock size:    126976 bytes
> >> UBI: smallest flash I/O unit:    2048
> >> UBI: sub-page size:              512
> >> UBI: VID header offset:          2048 (aligned 2048)
> >> UBI: data offset:                4096
> >> UBI: attached mtd1 to ubi0
> >> UBI: MTD device name:            "mtd=2"
> >> UBI: MTD device size:            240 MiB
> >> UBI: number of good PEBs:        1913
> >> UBI: number of bad PEBs:         7
> >> UBI: max. allowed volumes:       128
> >> UBI: wear-leveling threshold:    4096
> >> UBI: number of internal volumes: 1
> >> UBI: number of user volumes:     1
> >> UBI: available PEBs:             0
> >> UBI: total number of reserved PEBs: 1913
> >> UBI: number of PEBs reserved for bad PEB handling: 19
> >> UBI: max/mean erase counter: 7/1
> >> UBIFS: recovery needed
> >> Error reading superblock on volume 'ubi:rootfs'!
> >> UBI: mtd1 is detached from ubi0
> 
> I've been comparing the linux and u-boot implementations, and it looks
> like the following fix is in the kernel
> but not in u-boot.  I don't really understand it, but it looks like a
> candidate.  Might porting this change to
> u-boot fix the issue?

Hard to tell. Might be worth a try, if its not too complicated. It would be 
great if you could report the outcome of this. And best to send this patch to 
the list as well.

Thanks,
Stefan

--
DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-0 Fax: (+49)-8142-66989-80 Email: office at denx.de

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

>> I've been comparing the linux and u-boot implementations, and it looks
>> like the following fix is in the kernel
>> but not in u-boot. ?I don't really understand it, but it looks like a
>> candidate. ?Might porting this change to
>> u-boot fix the issue?
>
> Hard to tell. Might be worth a try, if its not too complicated. It would be
> great if you could report the outcome of this. And best to send this patch to
> the list as well.
>

Thanks, I've ported that one changeset to u-boot.  (I first tried
updating u-boot/fs/ubifs to the latest kernel code but this was too
difficult without any real understanding.)

Unfortunately I've got to wait for a repro before I can tell if it
fixes the issue... so I may go quiet for a few weeks.

Regards,

Alex

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Stefan Roese]

Hi Alex,

On Thursday 08 March 2012 10:49:33 Alex Zeffertt wrote:
> >> I've been comparing the linux and u-boot implementations, and it looks
> >> like the following fix is in the kernel
> >> but not in u-boot.  I don't really understand it, but it looks like a
> >> candidate.  Might porting this change to
> >> u-boot fix the issue?
> > 
> > Hard to tell. Might be worth a try, if its not too complicated. It would
> > be great if you could report the outcome of this. And best to send this
> > patch to the list as well.
> 
> Thanks, I've ported that one changeset to u-boot.  (I first tried
> updating u-boot/fs/ubifs to the latest kernel code but this was too
> difficult without any real understanding.)
> 
> Unfortunately I've got to wait for a repro before I can tell if it
> fixes the issue... so I may go quiet for a few weeks.

Understood. Thanks for the status update so far.

Thanks, 
Stefan

--
DENX Software Engineering GmbH,      MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich,  Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-0 Fax: (+49)-8142-66989-80 Email: office at denx.de

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

On 7 March 2012 14:42, Alex Zeffertt <azeffertt@cambridgesys.com> wrote:
> Hi u-booters,
>
> I have a short script in my u-boot environment which chooses which of
> two ubifs partitions to boot
> by attempting to read a release file in each one.
>
> Unfortunately, after an unclean shutdown sometimes the ubifsmount
> fails. ?(By "unclean shutdown"
> I mean that the board was power cycled while doing some low bandwidth logging.)
>
> The strange thing is that Linux has no problem mounting the partition
> as its root filesystem. ?This is
> very confusing because it looks like the ubifs implementation in
> u-boot is just a copy of the one in Linux.
>
> Has anyone else seen this problem?
>
>

I've now managed to repro this problem and add some debug.  It looks
like u-boot is simply running out of memory whilst trying to mount a
filesystem that "needs recovery".  (Error -12 is -ENOMEM.)  The
partition it is mounting is 240MB, but only about 40MB full.

The debug output is below after the "ubifsmount" command:



U-Boot 2011.06-00009-g3b6754e-dirty (Mar 08 2012 - 16:30:13)
OpenRD-Base

SoC:   Kirkwood 88F6281_A1
DRAM:  128 MiB
NAND:  512 MiB
In:    serial
Out:   serial
Err:   serial
Net:   egiga0
88E6351 Initialized on egiga0
Marvell>> ubi part rootfs 2048
Creating 1 MTD partitions on "nand0":
0x000001000000-0x000010000000 : "mtd=2"
UBI: attaching mtd1 to ubi0
UBI: physical eraseblock size:   131072 bytes (128 KiB)
UBI: logical eraseblock size:    126976 bytes
UBI: smallest flash I/O unit:    2048
UBI: sub-page size:              512
UBI: VID header offset:          2048 (aligned 2048)
UBI: data offset:                4096
UBI: attached mtd1 to ubi0
UBI: MTD device name:            "mtd=2"
UBI: MTD device size:            240 MiB
UBI: number of good PEBs:        1913
UBI: number of bad PEBs:         7
UBI: max. allowed volumes:       128
UBI: wear-leveling threshold:    4096
UBI: number of internal volumes: 1
UBI: number of user volumes:     1
UBI: available PEBs:             0
UBI: total number of reserved PEBs: 1913
UBI: number of PEBs reserved for bad PEB handling: 19
UBI: max/mean erase counter: 7/1
Marvell>> ubifsmount rootfs
UBIFS: recovery needed
UBIFS error (pid 0): replay_bud: insert_node failed: err=-12
UBIFS error (pid 0): replay_buds: replay_bud failed: err=-12
UBIFS error (pid 0): ubifs_replay_journal: replay_buds failed: err=-12
UBIFS error (pid 0): mount_ubifs: ubifs_replay_journal failed: err=-12
UBIFS error (pid 0): ubifs_fill_super: mount_ubifs failed: err=-12
UBIFS error (pid 0): ubifs_get_sb: ubifs_fill_super failed: err=-12
Error reading superblock on volume 'ubi:rootfs'!
Marvell>>



Does this help explain the issue?

Regards,

Alex

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Wolfgang Denk]

Dear Alex,

In message <CANNiwJ+91o+A5Ef6w9V0g=rvj1yA939+ZzTSnKA4DmR67oFe0w@mail.gmail.com> you wrote:
>
> I've now managed to repro this problem and add some debug.  It looks
> like u-boot is simply running out of memory whilst trying to mount a
> filesystem that "needs recovery".  (Error -12 is -ENOMEM.)  The
> partition it is mounting is 240MB, but only about 40MB full.

Can you please try out and test what happens when you increase the
size of the malloc arena?  "include/configs/openrd.h" includes
"include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to
1 MiB - try changing it to 4 MiB.


BTW: nice to "seeing" you again :-) Hope all is well for you.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Our OS who art in CPU, UNIX be thy name.
Thy programs run, thy syscalls done,
In kernel as it is in user!

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

On 8 March 2012 18:18, Wolfgang Denk <wd@denx.de> wrote:
> In message <CANNiwJ+91o+A5Ef6w9V0g=rvj1yA939+ZzTSnKA4DmR67oFe0w@mail.gmail.com> you wrote:
>>
>> I've now managed to repro this problem and add some debug. ?It looks
>> like u-boot is simply running out of memory whilst trying to mount a
>> filesystem that "needs recovery". ?(Error -12 is -ENOMEM.) ?The
>> partition it is mounting is 240MB, but only about 40MB full.
>
> Can you please try out and test what happens when you increase the
> size of the malloc arena? ?"include/configs/openrd.h" includes
> "include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to
> 1 MiB - try changing it to 4 MiB.

Thanks Wolfgang, that worked!  Is there any reason why you would not
set this variable to the maximum possible.  For example, is this
memory available to the kernel after it is booted?  How do you know
how big this might need to be in order to mount a filesystem that
"needs recovery"?

Regards,

Alex

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Wolfgang Denk]

Dear Alex,

In message <CANNiwJKREXiOBfiA_1v_-QGMoV_A59Ce0ADcVfK1xLMFkReFgw@mail.gmail.com> you wrote:
>
> > Can you please try out and test what happens when you increase the
> > size of the malloc arena?  "include/configs/openrd.h" includes
> > "include/configs/mv-common.h" which sets CONFIG_SYS_MALLOC_LEN to
> > 1 MiB - try changing it to 4 MiB.
> 
> Thanks Wolfgang, that worked!  Is there any reason why you would not
> set this variable to the maximum possible. ...

Define "maximum possible".  The more memory we reserve here, the less
will be available for loading Linux and root filesystem images and
such.  So it's always a compromize: make it as big as needed, but not
(much) bigger.

>                                        ...  For example, is this
> memory available to the kernel after it is booted?  ...

When the kernel boots, all of U-Boot is gone. The kernel has no idea
of how U-Boot used the memory; all such information is gone, too.

>                                                ...  How do you know
> how big this might need to be in order to mount a filesystem that
> "needs recovery"?

I don't know of ways to determine this in advance.  Eventually all we
can do is thry what you did: increase the malloc arena when it turns
out to be too small.

If you are willing to invest a little more time, it would be
interesting to test how much memory is actually needed on your system.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
In the realm of scientific observation, luck is granted only to those
who are prepared.                                     - Louis Pasteur

[U-Boot] ubifsmount reports "Error reading superblock", but linux can mount FS

[Alex Zeffertt]

On 9 March 2012 14:02, Wolfgang Denk <wd@denx.de> wrote:
> In message <CANNiwJKREXiOBfiA_1v_-QGMoV_A59Ce0ADcVfK1xLMFkReFgw@mail.gmail.com> you wrote:
>> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?... ?How do you know
>> how big this might need to be in order to mount a filesystem that
>> "needs recovery"?
>
> I don't know of ways to determine this in advance. ?Eventually all we
> can do is thry what you did: increase the malloc arena when it turns
> out to be too small.
>
> If you are willing to invest a little more time, it would be
> interesting to test how much memory is actually needed on your system.
>

I know that 1MiB was too little and 4MiB was enough.  However, I don't
know whether a filesystem corruption could occur that caused
ubifsmount to require more than 4MiB.  For that reason I made the
malloc area 64MiB.  This is half my total RAM, which leaves the other
half for downloading images.

Regards,

Alex