bad node type after using ubiupdatevol

bad node type after using ubiupdatevol

[Per Smitt]

Hi

I am having some issues with ubifs. I have a system with two rootfs and
when I upgrade I planned to simply update the rootfs volume currently not 
in use with the new filesystem.

When loading the system for the first time using U-Boot it always works to
write both rootfs volumes but once I try to upgrade I run into problems.

My upgrade process:
ubiupdatevol /dev/ubi0_$INDEX "$FS_IMG"
sync
mount -t ubifs ubi0:rootfs$INDEX "$MOUNT_POINT"
# Migrate config files like passwords, hostname, application specific config
sync
umount "$MOUNT_POINT"
sync
fw_setenv boot_index "$INDEX"    (Telling U-Boot environment which rootfs to use)
reboot

Yes I was littering with syncs in a desperate attempt to get it to work. I am also
checking for error codes after every command and I get return code 0 all the time.

When rebooting I get the following in the log:
UBIFS error (pid 418): ubifs_read_node: bad node type (255 but expected 1)                                
UBIFS error (pid 418): ubifs_read_node: bad node at LEB 68:27440                                                                      
UBIFS error (pid 418): do_readpage: cannot read page 1000 of inode 617, error -22                                                     
UBIFS error (pid 418): ubifs_read_node: bad node type (255 but expected 1)                                                            
UBIFS error (pid 418): ubifs_read_node: bad node at LEB 68:27440                                                                      
UBIFS error (pid 418): do_readpage: cannot read page 1000 of inode 617, error -22      

It seems random which files are affected as I can sometimes upgrade without
any error message. It might even be possible that the upgrade completed without
any error at all, I cannot say.

Do anyone see anything outright wrong here that I simply am not aware of?

Best regards,
Per Smitt​

Re: bad node type after using ubiupdatevol

[Artem Bityutskiy]

Hi,

On Mon, 2014-10-20 at 11:51 +0000, Per Smitt wrote:
> Hi
> 
> I am having some issues with ubifs. I have a system with two rootfs and
> when I upgrade I planned to simply update the rootfs volume currently not 
> in use with the new filesystem.
> 
> When loading the system for the first time using U-Boot it always works to
> write both rootfs volumes but once I try to upgrade I run into problems.
> 
> My upgrade process:
> ubiupdatevol /dev/ubi0_$INDEX "$FS_IMG"
> sync
> mount -t ubifs ubi0:rootfs$INDEX "$MOUNT_POINT"
> # Migrate config files like passwords, hostname, application specific config
> sync
> umount "$MOUNT_POINT"
> sync
> fw_setenv boot_index "$INDEX"    (Telling U-Boot environment which rootfs to use)

Can you experiment and put a very long sleep here, say, 20 minutes. The
idea is to wait until UBI finishes all the background operations. This
should not matter normally, but may be for your HW/SW it does for some
reasons.

> reboot

I'd also tried to validate the flash with MTD tests.

> Yes I was littering with syncs in a desperate attempt to get it to work. I am also
> checking for error codes after every command and I get return code 0 all the time.

sync is only the FS sync, it does not flush the UBI queue, because it
should not be necessary.

> When rebooting I get the following in the log:
> UBIFS error (pid 418): ubifs_read_node: bad node type (255 but expected 1)                                
> UBIFS error (pid 418): ubifs_read_node: bad node at LEB 68:27440                                                                      
> UBIFS error (pid 418): do_readpage: cannot read page 1000 of inode 617, error -22                                                     
> UBIFS error (pid 418): ubifs_read_node: bad node type (255 but expected 1)                                                            
> UBIFS error (pid 418): ubifs_read_node: bad node at LEB 68:27440                                                                      
> UBIFS error (pid 418): do_readpage: cannot read page 1000 of inode 617, error -22      

> It seems random which files are affected as I can sometimes upgrade without
> any error message. It might even be possible that the upgrade completed without
> any error at all, I cannot say.
> 
> Do anyone see anything outright wrong here that I simply am not aware of?

What may, in theory cause issues may be related to how you flash your
images. There are whole bunch of tricks related to this, see [1] and
[2]. It is possible that you see the issues only after the update just
because you hit them this way first. If you do enough UBIFS operations
before update, you may also hit issues due to "improper" flashing.

To exclude those, please either use ubiformat, or just start with fully
erase flash, and attach it to UBI, so UBI formats it. Then you may be
sure it is not your u-boot/firmware/whatever flasher doing nasty things.

Artem

Re: SV: bad node type after using ubiupdatevol

[Artem Bityutskiy]

First of all, do you have fastmap enabled or disabled (check
your .config)

On Mon, 2014-10-20 at 13:57 +0000, Per Smitt wrote:
> > Can you experiment and put a very long sleep here, say, 20 minutes. The
> > idea is to wait until UBI finishes all the background operations. This
> > should not matter normally, but may be for your HW/SW it does for some
> > reasons.
> 
> I just experimented here with a manual delay somewhere between
> 10 seconds and one minute. I simply disabled the automatic reboot 
> in the firmware upgrade and manually rebooted once the writing was
> done. 

My hunch is that this just hides/reveals symptoms, and the root-cause is
not related to volume update at all. I thing this is related to flasher.
Namely, if your flasher does not skip empty pages, but writes 0xFFs to
them. See the links below.

> > What may, in theory cause issues may be related to how you flash your
> > images. There are whole bunch of tricks related to this, see [1] and
> > [2]. It is possible that you see the issues only after the update just
> > because you hit them this way first. If you do enough UBIFS operations
> > before update, you may also hit issues due to "improper" flashing.
> 
> What are the references [1] and [2] you refer to? I didn't see any references 
> added in the email.

Oh, here:

1. http://www.linux-mtd.infradead.org/doc/ubi.html#L_flasher_algo
2. http://www.linux-mtd.infradead.org/faq/ubifs.html#L_free_space_fixup

Could you please try to re-create your image with '--space-fixup' and
try what you do (without adding any delays).

But validating the flash is a good idea irrespective.

> > To exclude those, please either use ubiformat, or just start with fully
> > erase flash, and attach it to UBI, so UBI formats it. Then you may be
> > sure it is not your u-boot/firmware/whatever flasher doing nasty things.
> 
> I have a hard time doing this since I have only one mtd-partition except
> for my kernel partition. I could perhaps try to boot Linux from the SD-card
> and then write the ubifs from there.

Yea, booting form an SD card would be the way to go. But you can start
with just quick-trying '--space-fixup' first.

Artem.

Re: bad node type after using ubiupdatevol

[Artem Bityutskiy]

On Tue, 2014-10-21 at 09:05 +0000, Per Smitt wrote:
> > First of all, do you have fastmap enabled or disabled (check
> > your .config)
> 
> It is not at all in my config. I am running a Freescale i.MX28 yocto
> solution and it uses kernel 2.6.35.3.

Oh, it is so old. There might have been fixes since then. And I am not
sure how out-of-date the MTD tests are in this kernel.

Artem

Re: bad node type after using ubiupdatevol

[Per Smitt]

> > > First of all, do you have fastmap enabled or disabled (check
> > > your .config)
> >
> > It is not at all in my config. I am running a Freescale i.MX28 yocto
> > solution and it uses kernel 2.6.35.3.
> 
> Oh, it is so old. There might have been fixes since then. And I am not
> sure how out-of-date the MTD tests are in this kernel.

The kernel is old but the mtd-utils are slightly fresher. 

* commit dcea43eba91642939c82739387147da26d572758
| Author: Aaron Sierra <asierra@xes-inc.com>
| Date:   Fri Sep 27 11:34:04 2013 -0500

But if I understand you correctly most part of the UBIFS code is in
kernel drivers so it doesn't matter which version of mtd-utils I have.

What would you recommend in this situation. Should I try to compile a 
later kernel with my toolchain and see if I can get Linux running with 
that one? If so which version of the Linux kernel should I use?

/Per