Mounting issue with old rootfs and new kernel

Mounting issue with old rootfs and new kernel

[Jaap de Jong]

Hi

I'm hoping for some pointers.

I have this a created with openembedded classic.

It works just fine when running with an old kernel (2.6.35)

Now with the same rootfs and a newer kernel (4.9.28) it damages the old 
rootfs in such a way that it becomes unusable.

This is the error it shows:

                         [    1.523437] ubi0 error: 
ubi_read_volume_table: the layout volume was not found
                         [    1.531250] ubi0 error: ubi_attach_mtd_dev: 
failed to attach mtd3, error -22
                         [    1.539062] UBI error: cannot attach mtd3
                         [    1.546875] Kernel panic - not syncing: VFS: 
Unable to mount root fs on unknown-block(0,0)
                         [    1.546875] Rebooting in 1 seconds..RomBOOT

As far as I can see the kernel configuration seems to be ok.

Any ideas?

Thanks!

Jaap

Re: Mounting issue with old rootfs and new kernel

[Richard Weinberger]

Jaap,

On Thu, Nov 30, 2017 at 4:22 PM, Jaap de Jong <jaap.dejong@nedap.com> wrote:
> Hi
>
> I'm hoping for some pointers.
>
> I have this a created with openembedded classic.
>
> It works just fine when running with an old kernel (2.6.35)
>
> Now with the same rootfs and a newer kernel (4.9.28) it damages the old
> rootfs in such a way that it becomes unusable.
>
> This is the error it shows:
>
>                         [    1.523437] ubi0 error: ubi_read_volume_table:
> the layout volume was not found
>                         [    1.531250] ubi0 error: ubi_attach_mtd_dev:
> failed to attach mtd3, error -22

Are these really the only erros/warnings from UBI?

>                         [    1.539062] UBI error: cannot attach mtd3
>                         [    1.546875] Kernel panic - not syncing: VFS:
> Unable to mount root fs on unknown-block(0,0)
>                         [    1.546875] Rebooting in 1 seconds..RomBOOT
>
> As far as I can see the kernel configuration seems to be ok.
>
> Any ideas?

If the MTD layout had changed I'd expect more errors from UBI.
Is this NAND?

Did you compare the MTD partition layout and number of bad blocks?

-- 
Thanks,
//richard

Re: Mounting issue with old rootfs and new kernel

[Richard Weinberger]

Jaap,

Am Donnerstag, 30. November 2017, 16:42:33 CET schrieb Jaap de Jong:
> Hi Richard,
> 
> On 30-11-17 16:28, Richard Weinberger wrote:
> > Jaap,
> > 
> > On Thu, Nov 30, 2017 at 4:22 PM, Jaap de Jong <jaap.dejong@nedap.com> 
wrote:
> >> Hi
> >> 
> >> I'm hoping for some pointers.
> >> 
> >> I have this a created with openembedded classic.
> >> 
> >> It works just fine when running with an old kernel (2.6.35)
> >> 
> >> Now with the same rootfs and a newer kernel (4.9.28) it damages the old
> >> rootfs in such a way that it becomes unusable.
> >> 
> >> This is the error it shows:
> >>                          [    1.523437] ubi0 error:
> >>                          ubi_read_volume_table: the layout volume was
> >>                          not found [    1.531250] ubi0 error:
> >>                          ubi_attach_mtd_dev: failed to attach mtd3,
> >>                          error -22> 
> > Are these really the only erros/warnings from UBI?
> 
> Yes. Also ran it without 'quiet' as kernel parameter and that also does
> not show extra errors.

Hm, but U-Boot comes first? Maybe it damaged the UBI image already.

> If I boot u-boot and try to mount it there, some other errors are show
> although basically the same
> 
>     U-Boot> ubi part rootfs
>     UBI: mtd1 is detached from ubi0
>     Creating 1 MTD partitions on "nand0":
>     0x000000100000-0x000020000000 : "mtd=3"
>     UBI: attaching mtd1 to ubi0
>     UBI: physical eraseblock size:   131072 bytes (128 KiB)
>     UBI: logical eraseblock size:    129024 bytes
>     UBI: smallest flash I/O unit:    2048
>     UBI: sub-page size:              512
>     UBI: VID header offset:          512 (aligned 512)
>     UBI: data offset:                2048
>     UBI: attached mtd1 to ubi0
>     UBI: MTD device name:            "mtd=3"
>     UBI: MTD device size:            511 MiB
>     UBI: number of good PEBs:        4088
>     UBI: number of bad PEBs:         0
>     UBI: max. allowed volumes:       128
>     UBI: wear-leveling threshold:    4096
>     UBI: number of internal volumes: 1
>     UBI: number of user volumes:     1
>     UBI: available PEBs:             40
>     UBI: total number of reserved PEBs: 4048
>     UBI: number of PEBs reserved for bad PEB handling: 40
>     UBI: max/mean erase counter: 2/0
> 
>     U-Boot> ubifsmount rootfs
>     UBIFS error (pid 0): ubifs_read_node: bad node type (255 but expected 6)
> UBIFS error (pid 0): ubifs_read_node: bad node at LEB 0:0
>     Error reading superblock on volume 'ubi:rootfs'!
> 
> >>                          [    1.539062] UBI error: cannot attach mtd3
> >>                          [    1.546875] Kernel panic - not syncing: VFS:
> >>                          Unable to mount root fs on unknown-block(0,0) [
> >>                             1.546875] Rebooting in 1 seconds..RomBOOT
> >> 
> >> As far as I can see the kernel configuration seems to be ok.
> >> 
> >> Any ideas?
> > 
> > If the MTD layout had changed I'd expect more errors from UBI.
> > Is this NAND?
> 
> Yes nandflash
> 
> > Did you compare the MTD partition layout and number of bad blocks?
> 
> Do you mean before and after?

Yes. Something must be different.
Page size? Sub pages? Number or erase blocks, etc...

Thanks,
//richard

Re: Mounting issue with old rootfs and new kernel

[Jaap de Jong]

On 30-11-17 17:20, Richard Weinberger wrote:
> Jaap,
> 
> Am Donnerstag, 30. November 2017, 16:42:33 CET schrieb Jaap de Jong:
>> Hi Richard,
>>
>> On 30-11-17 16:28, Richard Weinberger wrote:
>>> Jaap,
>>>
>>> On Thu, Nov 30, 2017 at 4:22 PM, Jaap de Jong <jaap.dejong@nedap.com>
> wrote:
>>>> Hi
>>>>
>>>> I'm hoping for some pointers.
>>>>
>>>> I have this a created with openembedded classic.
>>>>
>>>> It works just fine when running with an old kernel (2.6.35)
>>>>
>>>> Now with the same rootfs and a newer kernel (4.9.28) it damages the old
>>>> rootfs in such a way that it becomes unusable.
>>>>
>>>> This is the error it shows:
>>>>                           [    1.523437] ubi0 error:
>>>>                           ubi_read_volume_table: the layout volume was
>>>>                           not found [    1.531250] ubi0 error:
>>>>                           ubi_attach_mtd_dev: failed to attach mtd3,
>>>>                           error -22>
>>> Are these really the only erros/warnings from UBI?
>>
>> Yes. Also ran it without 'quiet' as kernel parameter and that also does
>> not show extra errors.
> 
> Hm, but U-Boot comes first? Maybe it damaged the UBI image already.
That is the same U-Boot as before which didn't damage the rootfs.
Also U-boot is still able to mount and read files from it even after it 
has been damaged by the new kernel.

So what do I have so far?
1) flash a unit with uboot, old kernel and old rootfs, boot it --> fine
2) flash a unit with uboot, new kernel and new rootfs, boot it --> fine
3) flash a unit with uboot, new kernel and old rootfs, boot it --> fine
4) as with 1) but afterwards boot it with new kernel --> rootfs damaged

> 
>> If I boot u-boot and try to mount it there, some other errors are show
>> although basically the same
>>
>>      U-Boot> ubi part rootfs
>>      UBI: mtd1 is detached from ubi0
>>      Creating 1 MTD partitions on "nand0":
>>      0x000000100000-0x000020000000 : "mtd=3"
>>      UBI: attaching mtd1 to ubi0
>>      UBI: physical eraseblock size:   131072 bytes (128 KiB)
>>      UBI: logical eraseblock size:    129024 bytes
>>      UBI: smallest flash I/O unit:    2048
>>      UBI: sub-page size:              512
>>      UBI: VID header offset:          512 (aligned 512)
>>      UBI: data offset:                2048
>>      UBI: attached mtd1 to ubi0
>>      UBI: MTD device name:            "mtd=3"
>>      UBI: MTD device size:            511 MiB
>>      UBI: number of good PEBs:        4088
>>      UBI: number of bad PEBs:         0
>>      UBI: max. allowed volumes:       128
>>      UBI: wear-leveling threshold:    4096
>>      UBI: number of internal volumes: 1
>>      UBI: number of user volumes:     1
>>      UBI: available PEBs:             40
>>      UBI: total number of reserved PEBs: 4048
>>      UBI: number of PEBs reserved for bad PEB handling: 40
>>      UBI: max/mean erase counter: 2/0
>>
>>      U-Boot> ubifsmount rootfs
>>      UBIFS error (pid 0): ubifs_read_node: bad node type (255 but expected 6)
>> UBIFS error (pid 0): ubifs_read_node: bad node at LEB 0:0
>>      Error reading superblock on volume 'ubi:rootfs'!
>>
>>>>                           [    1.539062] UBI error: cannot attach mtd3
>>>>                           [    1.546875] Kernel panic - not syncing: VFS:
>>>>                           Unable to mount root fs on unknown-block(0,0) [
>>>>                              1.546875] Rebooting in 1 seconds..RomBOOT
>>>>
>>>> As far as I can see the kernel configuration seems to be ok.
>>>>
>>>> Any ideas?
>>>
>>> If the MTD layout had changed I'd expect more errors from UBI.
>>> Is this NAND?
>>
>> Yes nandflash
>>
>>> Did you compare the MTD partition layout and number of bad blocks?
>>
>> Do you mean before and after?
> 
> Yes. Something must be different.
> Page size? Sub pages? Number or erase blocks, etc...
Will come back on this, need some more time...

> 
> Thanks,
> //richard
> 
> 

Re: Mounting issue with old rootfs and new kernel

[Jaap de Jong]

On 01-12-17 15:26, Jaap de Jong wrote:
> 
> 
> On 30-11-17 17:20, Richard Weinberger wrote:
>> Jaap,
>>
>> Am Donnerstag, 30. November 2017, 16:42:33 CET schrieb Jaap de Jong:
>>> Hi Richard,
>>>
>>> On 30-11-17 16:28, Richard Weinberger wrote:
>>>> Jaap,
>>>>
>>>> On Thu, Nov 30, 2017 at 4:22 PM, Jaap de Jong <jaap.dejong@nedap.com>
>> wrote:
>>>>> Hi
>>>>>
>>>>> I'm hoping for some pointers.
>>>>>
>>>>> I have this a created with openembedded classic.
>>>>>
>>>>> It works just fine when running with an old kernel (2.6.35)
>>>>>
>>>>> Now with the same rootfs and a newer kernel (4.9.28) it damages the 
>>>>> old
>>>>> rootfs in such a way that it becomes unusable.
>>>>>
>>>>> This is the error it shows:
>>>>>                           [    1.523437] ubi0 error:
>>>>>                           ubi_read_volume_table: the layout volume was
>>>>>                           not found [    1.531250] ubi0 error:
>>>>>                           ubi_attach_mtd_dev: failed to attach mtd3,
>>>>>                           error -22>
>>>> Are these really the only erros/warnings from UBI?
>>>
>>> Yes. Also ran it without 'quiet' as kernel parameter and that also does
>>> not show extra errors.
>>
>> Hm, but U-Boot comes first? Maybe it damaged the UBI image already.
> That is the same U-Boot as before which didn't damage the rootfs.
> Also U-boot is still able to mount and read files from it even after it 
> has been damaged by the new kernel.
> 
> So what do I have so far?
> 1) flash a unit with uboot, old kernel and old rootfs, boot it --> fine
> 2) flash a unit with uboot, new kernel and new rootfs, boot it --> fine
> 3) flash a unit with uboot, new kernel and old rootfs, boot it --> fine
> 4) as with 1) but afterwards boot it with new kernel --> rootfs damaged
> 
>>
>>> If I boot u-boot and try to mount it there, some other errors are show
>>> although basically the same
>>>
>>>      U-Boot> ubi part rootfs
>>>      UBI: mtd1 is detached from ubi0
>>>      Creating 1 MTD partitions on "nand0":
>>>      0x000000100000-0x000020000000 : "mtd=3"
>>>      UBI: attaching mtd1 to ubi0
>>>      UBI: physical eraseblock size:   131072 bytes (128 KiB)
>>>      UBI: logical eraseblock size:    129024 bytes
>>>      UBI: smallest flash I/O unit:    2048
>>>      UBI: sub-page size:              512
>>>      UBI: VID header offset:          512 (aligned 512)
>>>      UBI: data offset:                2048
>>>      UBI: attached mtd1 to ubi0
>>>      UBI: MTD device name:            "mtd=3"
>>>      UBI: MTD device size:            511 MiB
>>>      UBI: number of good PEBs:        4088
>>>      UBI: number of bad PEBs:         0
>>>      UBI: max. allowed volumes:       128
>>>      UBI: wear-leveling threshold:    4096
>>>      UBI: number of internal volumes: 1
>>>      UBI: number of user volumes:     1
>>>      UBI: available PEBs:             40
>>>      UBI: total number of reserved PEBs: 4048
>>>      UBI: number of PEBs reserved for bad PEB handling: 40
>>>      UBI: max/mean erase counter: 2/0
>>>
>>>      U-Boot> ubifsmount rootfs
>>>      UBIFS error (pid 0): ubifs_read_node: bad node type (255 but 
>>> expected 6)
>>> UBIFS error (pid 0): ubifs_read_node: bad node at LEB 0:0
>>>      Error reading superblock on volume 'ubi:rootfs'!
>>>
>>>>>                           [    1.539062] UBI error: cannot attach mtd3
>>>>>                           [    1.546875] Kernel panic - not 
>>>>> syncing: VFS:
>>>>>                           Unable to mount root fs on 
>>>>> unknown-block(0,0) [
>>>>>                              1.546875] Rebooting in 1 seconds..RomBOOT
>>>>>
>>>>> As far as I can see the kernel configuration seems to be ok.
>>>>>
>>>>> Any ideas?
>>>>
>>>> If the MTD layout had changed I'd expect more errors from UBI.
>>>> Is this NAND?
>>>
>>> Yes nandflash
>>>
>>>> Did you compare the MTD partition layout and number of bad blocks?
>>>
>>> Do you mean before and after?
>>
>> Yes. Something must be different.
>> Page size? Sub pages? Number or erase blocks, etc...
> Will come back on this, need some more time...
Ok mtd partition is the same for both
	kernel cmdline:
root=ubi0 rw ubi.mtd=3 rootfstype=ubifs 
mtdparts=atmel_nand:128K(bootstrap),256K(u-boot-env),640K(u-boot),-(rootfs)

As mentioned before (all with the saem parameters)
1) old kernel and old rootfs, boot it --> fine
2) new kernel and new rootfs, boot it --> fine
3) new kernel and old rootfs, boot it --> fine
If I run 1) and then boot the old rootfs with a new kernel --> rootfs 
damaged



	

Re: Mounting issue with old uboot and new rootfs

[Richard Weinberger]

Jaap,

Am Montag, 11. Dezember 2017, 16:03:25 CET schrieb Jaap de Jong:
> Hi All,
> 
> Some time ago I posted a question with a slightly different subject.
> Now that I found out a bit more the previous subject is no longer relevant.
> 
> But I still have issues with mounting in a mixed environment.
> I have this board with an older u-boot (2010.09) in combination with a more
> recent kernel (4.9.28).
> 
> The parameters in uboot:
>         root=ubi0 rw ubi.mtd=3 rootfstype=ubifs
>        
> mtdparts=atmel_nand:128K(bootstrap),256K(u-boot-env),640K(u-boot),-(rootfs)
> 
> U-boot runs `ubi part rootfs` as one of the steps in the startup process to
> load the kernel. When doing that u-boot reports that the ubi volume is
> resized. This is due to the fact that the rootfs is written with the u-boot
> `nand write` command, writing a ubi formatted image.
> 
> When booting the unit the kernel panics:
>         [    1.523437] ubi0 error: ubi_read_volume_table: the layout volume
> was not found [    1.539062] ubi0 error: ubi_attach_mtd_dev: failed to
> attach mtd3, error -22 [    1.546875] UBI error: cannot attach mtd3
>         [    1.546875] Kernel panic - not syncing: VFS: Unable to mount root
> fs on unknown-block(0,0) [    1.546875] Rebooting in 1 seconds..RomBOOT
> Then u-boot restarts and tells:
>         UBI warning: process_lvol: volume table copy #1 is corrupted
>         UBI: create volume table (copy #1)
>         UBI: volume table was restored
> But is able to load the kernel and transfer control to it.
> This second run of this kernel does not panic anymore and just starts as
> expected! Also, new reboots don't show u-boot issues!
> 
> Some other combinations:
>         u-boot        kernel        result
>         2010.09     2.6.35        fine
>         2010.09     4.9.28        panic
>         2016.03     2.6.35        fine
>         2016.03     4.9.28        fine
> 
> One thing that I noticed is that the newer u-boot resizes to around 40 less
> LEBs than the older u-boot does. Related?

Resize? Or missing?
This is definitely odd and should not happen.

Thanks,
//richard

Re: Mounting issue with old uboot and new rootfs

[Jaap de Jong]

On 12-12-17 17:29, Richard Weinberger wrote:
> Jaap,
> 
> Am Montag, 11. Dezember 2017, 16:03:25 CET schrieb Jaap de Jong:
>> Hi All,
>>
>> Some time ago I posted a question with a slightly different subject.
>> Now that I found out a bit more the previous subject is no longer relevant.
>>
>> But I still have issues with mounting in a mixed environment.
>> I have this board with an older u-boot (2010.09) in combination with a more
>> recent kernel (4.9.28).
>>
>> The parameters in uboot:
>>          root=ubi0 rw ubi.mtd=3 rootfstype=ubifs
>>         
>> mtdparts=atmel_nand:128K(bootstrap),256K(u-boot-env),640K(u-boot),-(rootfs)
>>
>> U-boot runs `ubi part rootfs` as one of the steps in the startup process to
>> load the kernel. When doing that u-boot reports that the ubi volume is
>> resized. This is due to the fact that the rootfs is written with the u-boot
>> `nand write` command, writing a ubi formatted image.
>>
>> When booting the unit the kernel panics:
>>          [    1.523437] ubi0 error: ubi_read_volume_table: the layout volume
>> was not found [    1.539062] ubi0 error: ubi_attach_mtd_dev: failed to
>> attach mtd3, error -22 [    1.546875] UBI error: cannot attach mtd3
>>          [    1.546875] Kernel panic - not syncing: VFS: Unable to mount root
>> fs on unknown-block(0,0) [    1.546875] Rebooting in 1 seconds..RomBOOT
>> Then u-boot restarts and tells:
>>          UBI warning: process_lvol: volume table copy #1 is corrupted
>>          UBI: create volume table (copy #1)
>>          UBI: volume table was restored
>> But is able to load the kernel and transfer control to it.
>> This second run of this kernel does not panic anymore and just starts as
>> expected! Also, new reboots don't show u-boot issues!
>>
>> Some other combinations:
>>          u-boot        kernel        result
>>          2010.09     2.6.35        fine
>>          2010.09     4.9.28        panic
>>          2016.03     2.6.35        fine
>>          2016.03     4.9.28        fine
>>
>> One thing that I noticed is that the newer u-boot resizes to around 40 less
>> LEBs than the older u-boot does. Related?
> 
> Resize? Or missing?
> This is definitely odd and should not happen.
Below the output of the old uboot when doing its `ubi part rootfs` on a 
new freshly flashed ubi root file system:
- resizing the volume

     Creating 1 MTD partitions on "nand0":
     0x000000100000-0x000020000000 : "mtd=3"
     UBI: attaching mtd1 to ubi0
     UBI: physical eraseblock size:   131072 bytes (128 KiB)
     UBI: logical eraseblock size:    129024 bytes
     UBI: smallest flash I/O unit:    2048
     UBI: sub-page size:              512
     UBI: VID header offset:          512 (aligned 512)
     UBI: data offset:                2048
     UBI: volume 0 ("my-rootfs") re-sized from 933 to 4042 LEBs
     UBI: attached mtd1 to ubi0
     UBI: MTD device name:            "mtd=3"
     UBI: MTD device size:            511 MiB
     UBI: number of good PEBs:        4086
     UBI: number of bad PEBs:         2
     UBI: max. allowed volumes:       128
     UBI: wear-leveling threshold:    4096
     UBI: number of internal volumes: 1
     UBI: number of user volumes:     1
     UBI: available PEBs:             0
     UBI: total number of reserved PEBs: 4086
     UBI: number of PEBs reserved for bad PEB handling: 40
     UBI: max/mean erase counter: 1/0

Then the kernel panics with this, causing a reboot.
During that new startup, the old uboot again shows an interesting error:
- volume table #1 corruption

     Creating 1 MTD partitions on "nand0":
     0x000000100000-0x000020000000 : "mtd=3"
     UBI: attaching mtd1 to ubi0
     UBI: physical eraseblock size:   131072 bytes (128 KiB)
     UBI: logical eraseblock size:    129024 bytes
     UBI: smallest flash I/O unit:    2048
     UBI: sub-page size:              512
     UBI: VID header offset:          512 (aligned 512)
     UBI: data offset:                2048
     UBI warning: process_lvol: volume table copy #1 is corrupted
     UBI: create volume table (copy #1)
     UBI: volume table was restored
     UBI: attached mtd1 to ubi0
     UBI: MTD device name:            "mtd=3"
     UBI: MTD device size:            511 MiB
     UBI: number of good PEBs:        4086
     UBI: number of bad PEBs:         2
     UBI: max. allowed volumes:       128
     UBI: wear-leveling threshold:    4096
     UBI: number of internal volumes: 1
     UBI: number of user volumes:     1
     UBI: available PEBs:             0
     UBI: total number of reserved PEBs: 4086
     UBI: number of PEBs reserved for bad PEB handling: 40
     UBI: max/mean erase counter: 1/0

The kernel does not panic and boots as to be expected, but dmesg shows 
two interesting message (only the ubi messages):
- volume table #2 corruption
- peb reservation

     ubi0: default fastmap pool size: 200
     ubi0: default fastmap WL pool size: 100
     ubi0: attaching mtd3
     ubi0: scanning is finished
     ubi0 warning: ubi_read_volume_table: volume table copy #2 is corrupted
     ubi0: volume table was restored
     ubi0 warning: ubi_eba_init: cannot reserve enough PEBs for bad PEB 
handling, reserved 35, need 75
     ubi0: attached mtd3 (name "rootfs", size 511 MiB)
     ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 129024 bytes
     ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 512
     ubi0: VID header offset: 512 (aligned 512), data offset: 2048
     ubi0: good PEBs: 4083, bad PEBs: 5, corrupted PEBs: 0
     ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128
     ubi0: max/mean erase counter: 1/0, WL threshold: 4096, image 
sequence number: 1296107501
     ubi0: available PEBs: 0, total reserved PEBs: 4083, PEBs reserved 
for bad PEB handling: 35
     ubi0: background thread "ubi_bgt0d" started, PID 94

Things are settled after this


> 
> Thanks,
> //richard
> 

Re: Mounting issue with old uboot and new rootfs

[Richard Weinberger]

Jaap,

Am Mittwoch, 13. Dezember 2017, 08:45:44 CET schrieb Jaap de Jong:
> On 12-12-17 17:29, Richard Weinberger wrote:
> > Jaap,
> > 
> > Am Montag, 11. Dezember 2017, 16:03:25 CET schrieb Jaap de Jong:
> >> Hi All,
> >> 
> >> Some time ago I posted a question with a slightly different subject.
> >> Now that I found out a bit more the previous subject is no longer
> >> relevant.
> >> 
> >> But I still have issues with mounting in a mixed environment.
> >> I have this board with an older u-boot (2010.09) in combination with a
> >> more
> >> recent kernel (4.9.28).
> >> 
> >> The parameters in uboot:
> >>          root=ubi0 rw ubi.mtd=3 rootfstype=ubifs
> >> 
> >> mtdparts=atmel_nand:128K(bootstrap),256K(u-boot-env),640K(u-boot),-(rootf
> >> s)
> >> 
> >> U-boot runs `ubi part rootfs` as one of the steps in the startup process
> >> to
> >> load the kernel. When doing that u-boot reports that the ubi volume is
> >> resized. This is due to the fact that the rootfs is written with the
> >> u-boot
> >> `nand write` command, writing a ubi formatted image.
> >> 
> >> When booting the unit the kernel panics:
> >>          [    1.523437] ubi0 error: ubi_read_volume_table: the layout
> >>          volume
> >> 
> >> was not found [    1.539062] ubi0 error: ubi_attach_mtd_dev: failed to
> >> attach mtd3, error -22 [    1.546875] UBI error: cannot attach mtd3
> >> 
> >>          [    1.546875] Kernel panic - not syncing: VFS: Unable to mount
> >>          root
> >> 
> >> fs on unknown-block(0,0) [    1.546875] Rebooting in 1 seconds..RomBOOT
> >> 
> >> Then u-boot restarts and tells:
> >>          UBI warning: process_lvol: volume table copy #1 is corrupted
> >>          UBI: create volume table (copy #1)
> >>          UBI: volume table was restored
> >> 
> >> But is able to load the kernel and transfer control to it.
> >> This second run of this kernel does not panic anymore and just starts as
> >> expected! Also, new reboots don't show u-boot issues!
> >> 
> >> Some other combinations:
> >>          u-boot        kernel        result
> >>          2010.09     2.6.35        fine
> >>          2010.09     4.9.28        panic
> >>          2016.03     2.6.35        fine
> >>          2016.03     4.9.28        fine
> >> 
> >> One thing that I noticed is that the newer u-boot resizes to around 40
> >> less
> >> LEBs than the older u-boot does. Related?
> > 
> > Resize? Or missing?
> > This is definitely odd and should not happen.
> 
> Below the output of the old uboot when doing its `ubi part rootfs` on a
> new freshly flashed ubi root file system:
> - resizing the volume
> 
>      Creating 1 MTD partitions on "nand0":
>      0x000000100000-0x000020000000 : "mtd=3"
>      UBI: attaching mtd1 to ubi0
>      UBI: physical eraseblock size:   131072 bytes (128 KiB)
>      UBI: logical eraseblock size:    129024 bytes
>      UBI: smallest flash I/O unit:    2048
>      UBI: sub-page size:              512
>      UBI: VID header offset:          512 (aligned 512)
>      UBI: data offset:                2048
>      UBI: volume 0 ("my-rootfs") re-sized from 933 to 4042 LEBs

Does everything work as expected if you don't set the resize flag in ubinize?
Maybe this is the culprit.

Thanks,
//richard

Re: Mounting issue with old uboot and new rootfs

[Jaap de Jong]

On 13-12-17 10:43, Richard Weinberger wrote:
> Jaap,
> 
> Am Mittwoch, 13. Dezember 2017, 08:45:44 CET schrieb Jaap de Jong:
>> On 12-12-17 17:29, Richard Weinberger wrote:
>>> Jaap,
>>>
>>> Am Montag, 11. Dezember 2017, 16:03:25 CET schrieb Jaap de Jong:
>>>> Hi All,
>>>>
>>>> Some time ago I posted a question with a slightly different subject.
>>>> Now that I found out a bit more the previous subject is no longer
>>>> relevant.
>>>>
>>>> But I still have issues with mounting in a mixed environment.
>>>> I have this board with an older u-boot (2010.09) in combination with a
>>>> more
>>>> recent kernel (4.9.28).
>>>>
>>>> The parameters in uboot:
>>>>           root=ubi0 rw ubi.mtd=3 rootfstype=ubifs
>>>>
>>>> mtdparts=atmel_nand:128K(bootstrap),256K(u-boot-env),640K(u-boot),-(rootf
>>>> s)
>>>>
>>>> U-boot runs `ubi part rootfs` as one of the steps in the startup process
>>>> to
>>>> load the kernel. When doing that u-boot reports that the ubi volume is
>>>> resized. This is due to the fact that the rootfs is written with the
>>>> u-boot
>>>> `nand write` command, writing a ubi formatted image.
>>>>
>>>> When booting the unit the kernel panics:
>>>>           [    1.523437] ubi0 error: ubi_read_volume_table: the layout
>>>>           volume
>>>>
>>>> was not found [    1.539062] ubi0 error: ubi_attach_mtd_dev: failed to
>>>> attach mtd3, error -22 [    1.546875] UBI error: cannot attach mtd3
>>>>
>>>>           [    1.546875] Kernel panic - not syncing: VFS: Unable to mount
>>>>           root
>>>>
>>>> fs on unknown-block(0,0) [    1.546875] Rebooting in 1 seconds..RomBOOT
>>>>
>>>> Then u-boot restarts and tells:
>>>>           UBI warning: process_lvol: volume table copy #1 is corrupted
>>>>           UBI: create volume table (copy #1)
>>>>           UBI: volume table was restored
>>>>
>>>> But is able to load the kernel and transfer control to it.
>>>> This second run of this kernel does not panic anymore and just starts as
>>>> expected! Also, new reboots don't show u-boot issues!
>>>>
>>>> Some other combinations:
>>>>           u-boot        kernel        result
>>>>           2010.09     2.6.35        fine
>>>>           2010.09     4.9.28        panic
>>>>           2016.03     2.6.35        fine
>>>>           2016.03     4.9.28        fine
>>>>
>>>> One thing that I noticed is that the newer u-boot resizes to around 40
>>>> less
>>>> LEBs than the older u-boot does. Related?
>>>
>>> Resize? Or missing?
>>> This is definitely odd and should not happen.
>>
>> Below the output of the old uboot when doing its `ubi part rootfs` on a
>> new freshly flashed ubi root file system:
>> - resizing the volume
>>
>>       Creating 1 MTD partitions on "nand0":
>>       0x000000100000-0x000020000000 : "mtd=3"
>>       UBI: attaching mtd1 to ubi0
>>       UBI: physical eraseblock size:   131072 bytes (128 KiB)
>>       UBI: logical eraseblock size:    129024 bytes
>>       UBI: smallest flash I/O unit:    2048
>>       UBI: sub-page size:              512
>>       UBI: VID header offset:          512 (aligned 512)
>>       UBI: data offset:                2048
>>       UBI: volume 0 ("my-rootfs") re-sized from 933 to 4042 LEBs
> 
> Does everything work as expected if you don't set the resize flag in ubinize?
> Maybe this is the culprit.
Yes, that was my last experiment and that works. It turns off the code 
in the old uboot that modifies the volume in such a way that the new 
kernel is not able to deal with it.
The strange thing is, that an old kernel doesn't mind.

Re: Mounting issue with old uboot and new rootfs

[Richard Weinberger]

Am Mittwoch, 13. Dezember 2017, 11:18:02 CET schrieb Jaap de Jong:
> > Does everything work as expected if you don't set the resize flag in
> > ubinize? Maybe this is the culprit.
> 
> Yes, that was my last experiment and that works. It turns off the code
> in the old uboot that modifies the volume in such a way that the new
> kernel is not able to deal with it.
> The strange thing is, that an old kernel doesn't mind.

Can you please rule out U-Boot first?
IOW don't attach UBI from U-Boot and load the kernel via TFTP/NFS, etc...
I'm still not sure whether this is a regression in Linux or U-Boot.

Thanks,
//richard

Re: Mounting issue with old uboot and new rootfs

[Jaap de Jong]

On 13-12-17 17:42, Richard Weinberger wrote:
> Am Mittwoch, 13. Dezember 2017, 11:18:02 CET schrieb Jaap de Jong:
>>> Does everything work as expected if you don't set the resize flag in
>>> ubinize? Maybe this is the culprit.
>>
>> Yes, that was my last experiment and that works. It turns off the code
>> in the old uboot that modifies the volume in such a way that the new
>> kernel is not able to deal with it.
>> The strange thing is, that an old kernel doesn't mind.
> 
> Can you please rule out U-Boot first?
> IOW don't attach UBI from U-Boot and load the kernel via TFTP/NFS, etc...
> I'm still not sure whether this is a regression in Linux or U-Boot.
Sure. Did that and then there is no corruption. The old version uboot 
modifies the volume in a way that the new kernel can't handle. The old 
kernel on the other hand is able to mount that 'mangled' volume.

Jaap.

Re: Mounting issue with old uboot and new rootfs

[Jaap de Jong]

On 14-12-17 08:28, Jaap de Jong wrote:
> 
> 
> On 13-12-17 17:42, Richard Weinberger wrote:
>> Am Mittwoch, 13. Dezember 2017, 11:18:02 CET schrieb Jaap de Jong:
>>>> Does everything work as expected if you don't set the resize flag in
>>>> ubinize? Maybe this is the culprit.
>>>
>>> Yes, that was my last experiment and that works. It turns off the code
>>> in the old uboot that modifies the volume in such a way that the new
>>> kernel is not able to deal with it.
>>> The strange thing is, that an old kernel doesn't mind.
>>
>> Can you please rule out U-Boot first?
>> IOW don't attach UBI from U-Boot and load the kernel via TFTP/NFS, etc...
>> I'm still not sure whether this is a regression in Linux or U-Boot.
> Sure. Did that and then there is no corruption. The old version uboot 
> modifies the volume in a way that the new kernel can't handle. The old 
> kernel on the other hand is able to mount that 'mangled' volume.
In addition to this: In the situation where you replace the old kernel 
with a new kernel, the filesystem is damaged by the new kernel in a way 
that it is unusable. The way I reproduce this is by flashing a unit with 
an old uboot/kernel/filesystem. Start it: the old uboot resizes the fs 
and the old kernel also changes something with it. Restart with a new 
kernel. Bricked.
The first run of the new kernel shows:
  ubi0 error: ubi_read_volume_table: the layout volume was not found
  ubi0 error: ubi_attach_mtd_dev: failed to attach mtd3, error -22
  UBI error: cannot attach mtd3
  Kernel panic - not syncing: VFS: Unable to mount root fs on 
unknown-block(0,0)
  Rebooting in 1 seconds..

The last run of uboot:
  Creating 1 MTD partitions on "nand0":
  0x000000100000-0x000020000000 : "mtd=3"
  UBI: attaching mtd1 to ubi0
  UBI: physical eraseblock size:   131072 bytes (128 KiB)
  UBI: logical eraseblock size:    129024 bytes
  UBI: smallest flash I/O unit:    2048
  UBI: sub-page size:              512
  UBI: VID header offset:          512 (aligned 512)
  UBI: data offset:                2048
  UBI error: ubi_read_volume_table: the layout volume was not found
  UBI error: ubi_init: cannot attach mtd1
  UBI error: ubi_init: UBI error: cannot initialize UBI, error -22
  UBI init error -22
  UBIFS not mounted, use ubifs mount to mount volume first!
  UBIFS not mounted, use ubifs mount to mount volume first!
  UBIFS not mounted, use ubifs mount to mount volume first!
  UBIFS not mounted, use ubifs mount to mount volume first!

Same setup but bypassing the first uboot run: no issues.

To put my experiments in a table:

uboot kernel  kernel  result
       1st run 2nd run
-     old     old     fine
old   old     old     fine
new   old     old     fine

-     old     new     fine
old   old     new     bricked
new   old     new     bricked

-     new     new     fine
old   new     new     bricked
new   new     new     fine

So I see 2 issues...