Firmware update error: Cannot UBI create volume

Firmware update error: Cannot UBI create volume

[rgrs]

[-- Attachment #1: Type: text/plain, Size: 5011 bytes --]

Hi All,

I have a strange behavior where only one image is visible in webUI.
I get UBI "no space left" errors when I try to update/flash new image.
I dont have steps to reproduce and dont know how my BMC ended up in this state.

How do I recover from this error? /proc/mtd shows all partitions/volumes properly.
Is there any way to recover alternate partition without erasing SPI from uboot?

Logs:

Dec 17 09:26:21 obmc phosphor-version-software-manager[1531]: Untaring
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:21 obmc phosphor-mapper[1300]: Found invalid association on path /xyz/openbmc_project/software/c35db0b8
Dec 17 09:26:22 obmc phosphor-image-updater[1464]: BMC image activating - BMC reboots are disabled.
Dec 17 09:26:22 obmc systemd[1]: Starting Enable a guard that blocks BMC reboot...
Dec 17 09:26:23 obmc systemd[1]: reboot-guard-enable.service: Succeeded.
Dec 17 09:26:23 obmc systemd[1]: Started Enable a guard that blocks BMC reboot.
Dec 17 09:26:24 obmc systemd[1]: Created slice system-obmc\x2dflash\x2dbmc\x2dubiro.slice.
Dec 17 09:26:24 obmc systemd[1]: Starting Store read-only images c35db0b8 to BMC storage...
Dec 17 09:26:24 obmc systemd[1]: Starting Create BMC read-write ubi volume...
Dec 17 09:26:25 obmc systemd[1]: obmc-flash-bmc-ubirw.service: Succeeded.
Dec 17 09:26:25 obmc systemd[1]: Started Create BMC read-write ubi volume.
Dec 17 09:26:27 obmc kernel: ubi4 error: ubi_create_volume: not enough PEBs, only 315 available
Dec 17 09:26:27 obmc kernel: ubi4 error: ubi_create_volume: cannot create volume 0, error -28
Dec 17 09:26:27 obmc obmc-flash-bmc[2205]: ubimkvol: error!: cannot UBI create volume
Dec 17 09:26:27 obmc obmc-flash-bmc[2205]:           error 28 (No space left on device)
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro@c35db0b8.service: Main process exited, code=exited, status=255/EXCN
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro@c35db0b8.service: Failed with result 'exit-code'.
Dec 17 09:26:27 obmc systemd[1]: Failed to start Store read-only images c35db0b8 to BMC storage.
Dec 17 09:26:27 obmc phosphor-image-updater[1464]: BMC activation has ended - BMC reboots are re-enabled.
Dec 17 09:26:27 obmc systemd[1]: obmc-flash-bmc-ubiro@c35db0b8.service: Triggering OnFailure= dependencies.
Dec 17 09:26:27 obmc systemd[1]: Starting Removes the guard that blocks BMC reboot...
Dec 17 09:26:27 obmc systemd[1]: Created slice system-obmc\x2dflash\x2dbmc\x2dubiro\x2dremove.slice.
Dec 17 09:26:27 obmc systemd[1]: Starting Deletes read-only and kernel ubi volume c35db0b8...
Dec 17 09:26:28 obmc systemd[1]: reboot-guard-disable.service: Succeeded.
Dec 17 09:26:28 obmc systemd[1]: Started Removes the guard that blocks BMC reboot.
Dec 17 09:26:28 obmc systemd[1]: obmc-flash-bmc-ubiro-remove@c35db0b8.service: Succeeded.
Dec 17 09:26:28 obmc systemd[1]: Started Deletes read-only and kernel ubi volume c35db0b8.
root@obmc:~#
root@obmc:~#

root@obmc:~# ubinfo
UBI version:                    1
Count of UBI devices:           2
UBI control device major/minor: 10:59
Present UBI devices:            ubi0, ubi4

root@obmc:~# ubinfo -d 0 -n 0
Volume ID:   0 (on ubi0)
Type:        static
Alignment:   1
Size:        293 LEBs (19164544 bytes, 18.2 MiB)
Data bytes:  19128320 bytes (18.2 MiB)
State:       OK
Name:        rofs-956f13ca
Character device major/minor: 246:1
root@obmc:~# ubinfo -d 0 -n 1
Volume ID:   1 (on ubi0)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-956f13ca
Character device major/minor: 246:2
root@obmc:~# ubinfo -d 0 -n 2
Volume ID:   2 (on ubi0)
Type:        dynamic
Alignment:   1
Size:        97 LEBs (6344576 bytes, 6.0 MiB)
State:       OK
Name:        rwfs
Character device major/minor: 246:3
root@obmc:~# ubinfo -d 0 -n 3
ubinfo: error!: cannot get information about UBI volume 3 on ubi0
        error 2 (No such file or directory)

root@obmc:~# ubinfo -d 4 -n 0
ubinfo: error!: cannot get information about UBI volume 0 on ubi4
        error 2 (No such file or directory)
root@obmc:~# ubinfo -d 4 -n 1
Volume ID:   1 (on ubi4)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-7e7e1530
Character device major/minor: 245:2
root@obmc:~# ubinfo -d 4 -n 2
Volume ID:   2 (on ubi4)
Type:        dynamic
Alignment:   1
Size:        97 LEBs (6344576 bytes, 6.0 MiB)
State:       OK
Name:        rwfs
Character device major/minor: 245:3
root@obmc:~# ubinfo -d 4 -n 3
Volume ID:   3 (on ubi4)
Type:        static
Alignment:   1
Size:        43 LEBs (2812544 bytes, 2.6 MiB)
Data bytes:  2767008 bytes (2.6 MiB)
State:       OK
Name:        kernel-956f13ca
Character device major/minor: 245:4
root@obmc:~#

[-- Attachment #2: Type: text/html, Size: 60722 bytes --]

Re: Firmware update error: Cannot UBI create volume

[Adriana Kobylak]

On 2019-12-17 03:52, rgrs wrote:
> Hi All,
> 
> I have a strange behavior where only one image is visible in webUI.
> I get UBI "no space left" errors when I try to update/flash new image.
> I dont have steps to reproduce and dont know how my BMC ended up in
> this state.
> 
> How do I recover from this error? /proc/mtd shows all
> partitions/volumes properly.
> Is there any way to recover alternate partition without erasing SPI
> from uboot?

Seems there's an 'orphan' volume which you can manually delete to make 
space. A while back there was an issue at some point where not all the 
volumes were deleted during a firmware update.

> 
> root@obmc:~# ubinfo -d 4 -n 1
> Volume ID: 1 (on ubi4)
> Type: static
> Alignment: 1
> Size: 43 LEBs (2812544 bytes, 2.6 MiB)
> Data bytes: 2767008 bytes (2.6 MiB)
> State: OK
> Name: kernel-7e7e1530

This seems to be the 'orphan' since there's not a corresponding 
'rofs-7e7e1530' volume, you can manually delete it with "ubirmvol 
/dev/ubi4 -n 1". You can also list all the volumes of a device with 
"ubinfo -a -d 0" for example.

Re: Firmware update error: Cannot UBI create volume

[rgrs]

Thanks :)
I am able to flash new images now.

~Raj


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Tuesday, December 17, 2019 10:30 PM, Adriana Kobylak <anoo@linux.ibm.com> wrote:

> On 2019-12-17 03:52, rgrs wrote:
>
> > Hi All,
> > I have a strange behavior where only one image is visible in webUI.
> > I get UBI "no space left" errors when I try to update/flash new image.
> > I dont have steps to reproduce and dont know how my BMC ended up in
> > this state.
> > How do I recover from this error? /proc/mtd shows all
> > partitions/volumes properly.
> > Is there any way to recover alternate partition without erasing SPI
> > from uboot?
>
> Seems there's an 'orphan' volume which you can manually delete to make
> space. A while back there was an issue at some point where not all the
> volumes were deleted during a firmware update.
>
> > root@obmc:~# ubinfo -d 4 -n 1
> > Volume ID: 1 (on ubi4)
> > Type: static
> > Alignment: 1
> > Size: 43 LEBs (2812544 bytes, 2.6 MiB)
> > Data bytes: 2767008 bytes (2.6 MiB)
> > State: OK
> > Name: kernel-7e7e1530
>
> This seems to be the 'orphan' since there's not a corresponding
> 'rofs-7e7e1530' volume, you can manually delete it with "ubirmvol
> /dev/ubi4 -n 1". You can also list all the volumes of a device with
> "ubinfo -a -d 0" for example.

Re: Firmware update error: Cannot UBI create volume

[rgrs]

Hi Adriana,

Was this bug fixed recently after 2.7.0 release?
My codebase is aligned with 2.7.0 and I'm hitting this issue often.

Thanks,
Raj


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Wednesday, December 18, 2019 1:56 PM, rgrs <rgrs@protonmail.com> wrote:

> Thanks :)
> I am able to flash new images now.
>
> ~Raj
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Tuesday, December 17, 2019 10:30 PM, Adriana Kobylak anoo@linux.ibm.com wrote:
>
> > On 2019-12-17 03:52, rgrs wrote:
> >
> > > Hi All,
> > > I have a strange behavior where only one image is visible in webUI.
> > > I get UBI "no space left" errors when I try to update/flash new image.
> > > I dont have steps to reproduce and dont know how my BMC ended up in
> > > this state.
> > > How do I recover from this error? /proc/mtd shows all
> > > partitions/volumes properly.
> > > Is there any way to recover alternate partition without erasing SPI
> > > from uboot?
> >
> > Seems there's an 'orphan' volume which you can manually delete to make
> > space. A while back there was an issue at some point where not all the
> > volumes were deleted during a firmware update.
> >
> > > root@obmc:~# ubinfo -d 4 -n 1
> > > Volume ID: 1 (on ubi4)
> > > Type: static
> > > Alignment: 1
> > > Size: 43 LEBs (2812544 bytes, 2.6 MiB)
> > > Data bytes: 2767008 bytes (2.6 MiB)
> > > State: OK
> > > Name: kernel-7e7e1530
> >
> > This seems to be the 'orphan' since there's not a corresponding
> > 'rofs-7e7e1530' volume, you can manually delete it with "ubirmvol
> > /dev/ubi4 -n 1". You can also list all the volumes of a device with
> > "ubinfo -a -d 0" for example.