Romulus to use Virtual PNOR

Romulus to use Virtual PNOR

[Lei YU]

This email is to notify that Romulus is going to use VirtualPNOR feature, if
no objections are received.

It **impacts** to existing Romulus systems, that they must do PNOR code update
when the feature is enabled.

A little background on this topic could be found at:
https://lists.ozlabs.org/pipermail/openbmc/2018-May/011822.html
https://lists.ozlabs.org/pipermail/openbmc/2018-November/014112.html

I did not receive any feedback, so I choose to use VritualPNOR feature,
which has below benefits:
1. It requires minor code changes for a system to switch to VirtualPNOR;
2. It gets full features, including PNOR version, code verification, and code
   update via new interface;
3. When OpenBMC switches to Redfish, it will get Redfish code update for free,
   because IBM will implement Redfish code update based on VirtualPNOR.

The related changes are:
* OpenBMC: https://gerrit.openbmc-project.xyz/#/q/topic:ubifs-pnor-for-romulus
* op-build: https://github.com/open-power/op-build/pull/2578

Any objections?

Thanks!

--
BRs,
Lei YU

Re: Romulus to use Virtual PNOR

[Alexander Amelkin]

[-- Attachment #1.1: Type: text/plain, Size: 1466 bytes --]

25.01.2019 10:40, Lei YU wrote:
> This email is to notify that Romulus is going to use VirtualPNOR feature, if
> no objections are received.
>
> It **impacts** to existing Romulus systems, that they must do PNOR code update
> when the feature is enabled.
>
> A little background on this topic could be found at:
> https://lists.ozlabs.org/pipermail/openbmc/2018-May/011822.html
> https://lists.ozlabs.org/pipermail/openbmc/2018-November/014112.html
>
> I did not receive any feedback, so I choose to use VritualPNOR feature,
> which has below benefits:
> 1. It requires minor code changes for a system to switch to VirtualPNOR;
> 2. It gets full features, including PNOR version, code verification, and code
>    update via new interface;
> 3. When OpenBMC switches to Redfish, it will get Redfish code update for free,
>    because IBM will implement Redfish code update based on VirtualPNOR.
>
> The related changes are:
> * OpenBMC: https://gerrit.openbmc-project.xyz/#/q/topic:ubifs-pnor-for-romulus
> * op-build: https://github.com/open-power/op-build/pull/2578
>
> Any objections?

Well, the main concern is how it will affect vesnin, which is P8-based (like palmetto).

All this virtual PNOR functionality is based on mbox/hiomap, which is not supported for P8 in openpower firmware.

As a result, we suspect that palmetto and (what concerns us most) vesnin will lose firmware update support completely.

Alexander.
YADRO



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Romulus to use Virtual PNOR

[Stewart Smith]

Alexander Amelkin <a.amelkin@yadro.com> writes:
> 25.01.2019 10:40, Lei YU wrote:
>> This email is to notify that Romulus is going to use VirtualPNOR feature, if
>> no objections are received.
>>
>> It **impacts** to existing Romulus systems, that they must do PNOR code update
>> when the feature is enabled.
>>
>> A little background on this topic could be found at:
>> https://lists.ozlabs.org/pipermail/openbmc/2018-May/011822.html
>> https://lists.ozlabs.org/pipermail/openbmc/2018-November/014112.html
>>
>> I did not receive any feedback, so I choose to use VritualPNOR feature,
>> which has below benefits:
>> 1. It requires minor code changes for a system to switch to VirtualPNOR;
>> 2. It gets full features, including PNOR version, code verification, and code
>>    update via new interface;
>> 3. When OpenBMC switches to Redfish, it will get Redfish code update for free,
>>    because IBM will implement Redfish code update based on VirtualPNOR.
>>
>> The related changes are:
>> * OpenBMC: https://gerrit.openbmc-project.xyz/#/q/topic:ubifs-pnor-for-romulus
>> * op-build: https://github.com/open-power/op-build/pull/2578
>>
>> Any objections?
>
> Well, the main concern is how it will affect vesnin, which is P8-based (like palmetto).
>
> All this virtual PNOR functionality is based on mbox/hiomap, which is
> not supported for P8 in openpower firmware.

For various reasons, this is now changing, so you'll be able to do hiomap
on P8.

> As a result, we suspect that palmetto and (what concerns us most)
> vesnin will lose firmware update support completely.

I think this is a valid concern, and I'd be concerned if there's tight
coupling between VPNOR and the firmware update APIs.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Romulus to use Virtual PNOR

[Alexander Amelkin]

[-- Attachment #1.1: Type: text/plain, Size: 1581 bytes --]


06.02.2019 3:39, Stewart Smith wrote:
> Alexander Amelkin <a.amelkin@yadro.com> writes:
>> 25.01.2019 10:40, Lei YU wrote:
>>> This email is to notify that Romulus is going to use VirtualPNOR feature, if
>>> no objections are received.
>>>
>>> ...
>>>
>>> Any objections?
>> Well, the main concern is how it will affect vesnin, which is P8-based (like palmetto).
>>
>> All this virtual PNOR functionality is based on mbox/hiomap, which is
>> not supported for P8 in openpower firmware.
> For various reasons, this is now changing, so you'll be able to do hiomap
> on P8.

Could you please elaborate on this? Some GitHub issues or discussion threads anywhere?

The problem as we see it is that P8 must be supported everywhere throughout op-build components.
Even if there was no support in hostboot for hiomap being added, we could be able to add it ourselves. But that still wouldn't work with SBE, as while the SBE sources for P8 are public, they nonetheless require a proprietary toolchain to build that, I quote the README.md, "cannot be open-sourced". The op-build framework takes pre-built SBE binaries for P8.

Anyway, if hiomap support is indeed being added for P8, we'd love to know more about it.

>
>> As a result, we suspect that palmetto and (what concerns us most)
>> vesnin will lose firmware update support completely.
> I think this is a valid concern, and I'd be concerned if there's tight
> coupling between VPNOR and the firmware update APIs.

So far it looks to me like there is such tight coupling.

Thanks.

Alexander.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Romulus to use Virtual PNOR

[Andrew Jeffery]

On Thu, 7 Feb 2019, at 01:31, Alexander Amelkin wrote:
> 
> 06.02.2019 3:39, Stewart Smith wrote:
> > Alexander Amelkin <a.amelkin@yadro.com> writes:
> >> 25.01.2019 10:40, Lei YU wrote:
> >>> This email is to notify that Romulus is going to use VirtualPNOR feature, if
> >>> no objections are received.
> >>>
> >>> ...
> >>>
> >>> Any objections?
> >> Well, the main concern is how it will affect vesnin, which is P8-based (like palmetto).
> >>
> >> All this virtual PNOR functionality is based on mbox/hiomap, which is
> >> not supported for P8 in openpower firmware.
> > For various reasons, this is now changing, so you'll be able to do hiomap
> > on P8.
> 
> Could you please elaborate on this? Some GitHub issues or discussion 
> threads anywhere?
> 
> The problem as we see it is that P8 must be supported everywhere 
> throughout op-build components.
> Even if there was no support in hostboot for hiomap being added, we 
> could be able to add it ourselves. But that still wouldn't work with 
> SBE, as while the SBE sources for P8 are public, they nonetheless 
> require a proprietary toolchain to build that, I quote the README.md, 
> "cannot be open-sourced". The op-build framework takes pre-built SBE 
> binaries for P8.
> 
> Anyway, if hiomap support is indeed being added for P8, we'd love to 
> know more about it.

Hostboot support for hiomap on P8 has landed as of[1] (on the master-p8
branch). This should with the latest skiboot stable releases once I've fixed up
the P8 fallback support in skiboot master[2]. From there we need to get the
BMC-side fixed up (add the hiomap daemon) and we should be good to go.

[1] https://github.com/open-power/hostboot/commit/adb33e73b32c133dc9c6d23b695884d0d8c0ec9e
[2] https://github.com/open-power/skiboot/issues/217

> 
> >
> >> As a result, we suspect that palmetto and (what concerns us most)
> >> vesnin will lose firmware update support completely.
> > I think this is a valid concern, and I'd be concerned if there's tight
> > coupling between VPNOR and the firmware update APIs.
> 
> So far it looks to me like there is such tight coupling.

Yes, this is something I've been concerned about since its introduction.
The coupling is unfortunate and should be unravelled IMO, but that's
work that someone needs to take on.

Andrew

Re: Romulus to use Virtual PNOR

[Stewart Smith]

Lei YU <mine260309@gmail.com> writes:
> This email is to notify that Romulus is going to use VirtualPNOR feature, if
> no objections are received.

Why?

It's proved to be massively complex on Witherspoon and a tremendous
source of bugs (both on BMC and host).

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Romulus to use Virtual PNOR

[Joel Stanley]

On Fri, 25 Jan 2019 at 18:11, Lei YU <mine260309@gmail.com> wrote:
>
> This email is to notify that Romulus is going to use VirtualPNOR feature, if
> no objections are received.
>
> It **impacts** to existing Romulus systems, that they must do PNOR code update
> when the feature is enabled.
>
> A little background on this topic could be found at:
> https://lists.ozlabs.org/pipermail/openbmc/2018-May/011822.html
> https://lists.ozlabs.org/pipermail/openbmc/2018-November/014112.html
>
> I did not receive any feedback, so I choose to use VritualPNOR feature,
> which has below benefits:
> 1. It requires minor code changes for a system to switch to VirtualPNOR;
> 2. It gets full features, including PNOR version, code verification, and code
>    update via new interface;
> 3. When OpenBMC switches to Redfish, it will get Redfish code update for free,
>    because IBM will implement Redfish code update based on VirtualPNOR.
>
> The related changes are:
> * OpenBMC: https://gerrit.openbmc-project.xyz/#/q/topic:ubifs-pnor-for-romulus
> * op-build: https://github.com/open-power/op-build/pull/2578
>
> Any objections?

I would prefer this not to happen.

The "virtual pnor" feature is tightly coupled to UBI. The preferred
filesystem to use on small NOR chips is JFFS2, which suits the
technology and the size that BMCs ship on.

Moving to vpnor breaks features such as full pnor flashing from the host.

Cheers,

Joel

Re: Romulus to use Virtual PNOR

[Lei YU]

On Tue, Feb 12, 2019 at 11:57 AM Joel Stanley <joel@jms.id.au> wrote:
>
> On Fri, 25 Jan 2019 at 18:11, Lei YU <mine260309@gmail.com> wrote:
> >
> > This email is to notify that Romulus is going to use VirtualPNOR feature, if
> > no objections are received.
> >
> > It **impacts** to existing Romulus systems, that they must do PNOR code update
> > when the feature is enabled.
> >
> > A little background on this topic could be found at:
> > https://lists.ozlabs.org/pipermail/openbmc/2018-May/011822.html
> > https://lists.ozlabs.org/pipermail/openbmc/2018-November/014112.html
> >
> > I did not receive any feedback, so I choose to use VritualPNOR feature,
> > which has below benefits:
> > 1. It requires minor code changes for a system to switch to VirtualPNOR;
> > 2. It gets full features, including PNOR version, code verification, and code
> >    update via new interface;
> > 3. When OpenBMC switches to Redfish, it will get Redfish code update for free,
> >    because IBM will implement Redfish code update based on VirtualPNOR.
> >
> > The related changes are:
> > * OpenBMC: https://gerrit.openbmc-project.xyz/#/q/topic:ubifs-pnor-for-romulus
> > * op-build: https://github.com/open-power/op-build/pull/2578
> >
> > Any objections?
>
> I would prefer this not to happen.
>
> The "virtual pnor" feature is tightly coupled to UBI. The preferred
> filesystem to use on small NOR chips is JFFS2, which suits the
> technology and the size that BMCs ship on.
>
> Moving to vpnor breaks features such as full pnor flashing from the host.

Based on the discussion, let's keep using static flash layout for PNOR on
Romulus.

This brings the questions:
1. Shall we move the legacy code update service
   (org.openbmc.control.Flash.service) to the new one
   (xyz.openbmc_project.Software.xxx.service)?
2. How do we plan to support PNOR version, image verification for static flash
   layout?

Re: Romulus to use Virtual PNOR

[Alexander Amelkin]

[-- Attachment #1.1: Type: text/plain, Size: 1009 bytes --]

14.02.2019 6:03, Lei YU wrote:
> Based on the discussion, let's keep using static flash layout for PNOR on
> Romulus.
>
> This brings the questions:
> 1. Shall we move the legacy code update service
>    (org.openbmc.control.Flash.service) to the new one
>    (xyz.openbmc_project.Software.xxx.service)?
> 2. How do we plan to support PNOR version, image verification for static flash
>    layout?

What's the problem with version and image verification?

The installed PNOR reports its version via IPMI FRU now, so we know the installed version.

The version supplied in the update can be checked using the image container header.
Image integrity/validity checks can also be performed using container header information.
The header meta-information can be then saved on BMC side to perform PNOR contents checks before starting the host.

You could also reserve a section in PNOR itself for that meta-info, but IMO that would be less secure than storing the info in BMC.

Alexander.



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Romulus to use Virtual PNOR

[Lei YU]

> What's the problem with version and image verification?

By that, I mean there is no service on Romulus to implement the interface of
/xyz/openbmc_project/Software/Version.interface.yaml and no pnor code's
verification during code update as how Witherspoon does.

> The installed PNOR reports its version via IPMI FRU now, so we know the installed version.
Great! Could you kindly let me know how do you implement this?

Re: Romulus to use Virtual PNOR

[Alexander Amelkin]

[-- Attachment #1.1.1: Type: text/plain, Size: 367 bytes --]

15.02.2019 9:48, Lei YU wrote:
>
>> The installed PNOR reports its version via IPMI FRU now, so we know the installed version.
> Great! Could you kindly let me know how do you implement this?
Please see systemFwIpmiFruInv::buildProductInfoArea() function in
hostboot/src/usr/ipmiext/ipmifruinv.C

That is where hostboot builds the opfw FRU.

Alexander.



[-- Attachment #1.1.2: Type: text/html, Size: 1155 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: Romulus to use Virtual PNOR

[Andrew Geissler]

On Wed, Feb 13, 2019 at 9:03 PM Lei YU <mine260309@gmail.com> wrote:
>
> Based on the discussion, let's keep using static flash layout for PNOR on
> Romulus.
>
> This brings the questions:
> 1. Shall we move the legacy code update service
>    (org.openbmc.control.Flash.service) to the new one
>    (xyz.openbmc_project.Software.xxx.service)?

Yes please. I asked Adriana to open a story to track doing this. I'm
working on getting
firmware update support in Redfish and I'd like to just use the
xyz.openbmc_project.Software.xxx interfaces for both BMC and Bios updates.
Some day these xyz.openbmc_project.Software.xxx would also be used for
updates to power supplies and other devices in the system as well via
the Redfish UpdateService.

> 2. How do we plan to support PNOR version, image verification for static flash
>    layout?

Re: Romulus to use Virtual PNOR

[Adriana Kobylak]

On 2019-02-14 07:13, Andrew Geissler wrote:
> On Wed, Feb 13, 2019 at 9:03 PM Lei YU <mine260309@gmail.com> wrote:
>> 
>> Based on the discussion, let's keep using static flash layout for PNOR 
>> on
>> Romulus.
>> 
>> This brings the questions:
>> 1. Shall we move the legacy code update service
>>    (org.openbmc.control.Flash.service) to the new one
>>    (xyz.openbmc_project.Software.xxx.service)?
> 
> Yes please. I asked Adriana to open a story to track doing this.

https://github.com/ibm-openbmc/dev/issues/417

The first step would be to support just updating the pnor using the
libflash library that the legacy code uses. We can look into the
other features like signature validation afterwards.