Parse PELs sent by OpenPOWER host firmware

Parse PELs sent by OpenPOWER host firmware

[Deepak Kodihalli]

Hi,

The SELs/eSELs (System Event Log) that OpenBMC receives from 
hostboot/skiboot today are not interpreted, and are dumped as-is in a 
property of the error log D-Bus object. This means that if there's a PEL 
(Platform Event Log) hidden in the eSEL, admins/users of the error log 
need to depend on external tools to parse the PEL to human readable text.

Here's a proposal to have the BMC itself parse such eSELs (the 
motivation is to make the OpenBMC error logs more useful in this 
context, without needing the extra steps to parse such errors) :

- Separate this from phosphor-logging. This code can't go to a phosphor 
app, because it's OpenPOWER specific. Have, for example, an 
openpower-logging app implement a D-Bus interface to parse out a PEL 
from an eSEL. This app will host it's own D-Bus service.

- Such an app can watch for error D-Bus objects being created in the 
phosphor namespace. If it finds an eSEL in an error log, it can attempt 
to parse a PEL (if there's one), and store the PELs parsed text 
representation in a D-Bus property.

- The app can create a D-Bus object with the same path as the original 
object and have a D-Bus object-manager at the error log root path, so 
that when someone uses the OpenBMC REST API to read an error log, 
properties across all D-Bus services will be returned 
(phosphor-rest-server does this today already). This means with the 
existing REST API to enumerate error logs, a parsed PEL would be 
returned, if present.

- To parse the PEL, use opal-elog-parse.

I have some quickly hacked-up code based on the proposal above, and I 
can submit that for review. In general, this is appearing to me to be 
one of the ways to add host-specific information to phosphor D-Bus 
objects, without breaking the phosphor D-Bus API or changing the REST URIs.

Thanks,
Deepak

Re: Parse PELs sent by OpenPOWER host firmware

[Stewart Smith]

Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> writes:
> The SELs/eSELs (System Event Log) that OpenBMC receives from 
> hostboot/skiboot today are not interpreted, and are dumped as-is in a 
> property of the error log D-Bus object. This means that if there's a PEL 
> (Platform Event Log) hidden in the eSEL, admins/users of the error log 
> need to depend on external tools to parse the PEL to human readable
> text.

My lack of love for PELs is probably fairly well known, but even if we
replace them down the line, I think there's opportunity here for making
things better for users (including myself).

> Here's a proposal to have the BMC itself parse such eSELs (the 
> motivation is to make the OpenBMC error logs more useful in this 
> context, without needing the extra steps to parse such errors) :
>
> - Separate this from phosphor-logging. This code can't go to a phosphor 
> app, because it's OpenPOWER specific. Have, for example, an 
> openpower-logging app implement a D-Bus interface to parse out a PEL 
> from an eSEL. This app will host it's own D-Bus service.

It would be a bonus if we could take different actions based on log
format. If in the future we had a hypothetical other format, it'd be
great if it was *really* easy to plug into OpenBMC. I suspect other
platforms may have similar things (e.g. Google has done protobuf based
logs in the past at least, which are structurally somewhat similar to
PELs but different serialisation format).

> - Such an app can watch for error D-Bus objects being created in the 
> phosphor namespace. If it finds an eSEL in an error log, it can attempt 
> to parse a PEL (if there's one), and store the PELs parsed text 
> representation in a D-Bus property.
>
> - The app can create a D-Bus object with the same path as the original 
> object and have a D-Bus object-manager at the error log root path, so 
> that when someone uses the OpenBMC REST API to read an error log, 
> properties across all D-Bus services will be returned 
> (phosphor-rest-server does this today already). This means with the 
> existing REST API to enumerate error logs, a parsed PEL would be 
> returned, if present.

Is thre any reason for this approach rather than something like
phosphor-logging having a list of properties/formats and external
parsing tools? Kind of like mime type/action lists?

> - To parse the PEL, use opal-elog-parse.

+1

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Parse PELs sent by OpenPOWER host firmware

[Alexander Amelkin]

14.03.2018 03:36, Stewart Smith wrote:
> Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> writes:
>> The SELs/eSELs (System Event Log) that OpenBMC receives from 
>> hostboot/skiboot today are not interpreted, and are dumped as-is in a 
>> property of the error log D-Bus object. This means that if there's a PEL 
>> (Platform Event Log) hidden in the eSEL, admins/users of the error log 
>> need to depend on external tools to parse the PEL to human readable
>> text.
> My lack of love for PELs is probably fairly well known, but even if we
> replace them down the line, I think there's opportunity here for making
> things better for users (including myself).

Just my 2 cents. The IPMI-standardized SELs are good first of all
because of IPMI-standardized PEFs and PETs.

Being able to decode the OEM SEL records is not just convenient for the
users, but will also let them easily attach PEFs to such events if they
want to. That's of course when PEFs are supported in OpenBMC which is
currently not true, AFAIK.

As a maintainer of ipmitool I can promise to promptly include the
OpenPOWER event record decoder into ipmitool upstream code. All I need
is the information on how to decode those events. So far I'm planning to
try and deduce this information from hostboot and opal sources.

I guess these logs don't need to be fully decoded at BMC or ipmitool
levels as they contain too much of a too deeply technical information,
but some high level hints beyond "Unknown #0x0c" and "System Event #0x01
| Undetermined system hardware failure" that we see now in ipmitool need
to be given to the end user.

Alexander.


>
>> Here's a proposal to have the BMC itself parse such eSELs (the 
>> motivation is to make the OpenBMC error logs more useful in this 
>> context, without needing the extra steps to parse such errors) :
>>
>> - Separate this from phosphor-logging. This code can't go to a phosphor 
>> app, because it's OpenPOWER specific. Have, for example, an 
>> openpower-logging app implement a D-Bus interface to parse out a PEL 
>> from an eSEL. This app will host it's own D-Bus service.
> It would be a bonus if we could take different actions based on log
> format. If in the future we had a hypothetical other format, it'd be
> great if it was *really* easy to plug into OpenBMC. I suspect other
> platforms may have similar things (e.g. Google has done protobuf based
> logs in the past at least, which are structurally somewhat similar to
> PELs but different serialisation format).
>
>> - Such an app can watch for error D-Bus objects being created in the 
>> phosphor namespace. If it finds an eSEL in an error log, it can attempt 
>> to parse a PEL (if there's one), and store the PELs parsed text 
>> representation in a D-Bus property.
>>
>> - The app can create a D-Bus object with the same path as the original 
>> object and have a D-Bus object-manager at the error log root path, so 
>> that when someone uses the OpenBMC REST API to read an error log, 
>> properties across all D-Bus services will be returned 
>> (phosphor-rest-server does this today already). This means with the 
>> existing REST API to enumerate error logs, a parsed PEL would be 
>> returned, if present.
> Is thre any reason for this approach rather than something like
> phosphor-logging having a list of properties/formats and external
> parsing tools? Kind of like mime type/action lists?
>
>> - To parse the PEL, use opal-elog-parse.
> +1
>

Re: Parse PELs sent by OpenPOWER host firmware

[Stewart Smith]

Alexander Amelkin <a.amelkin@yadro.com> writes:
> 14.03.2018 03:36, Stewart Smith wrote:
>> Deepak Kodihalli <dkodihal@linux.vnet.ibm.com> writes:
>>> The SELs/eSELs (System Event Log) that OpenBMC receives from 
>>> hostboot/skiboot today are not interpreted, and are dumped as-is in a 
>>> property of the error log D-Bus object. This means that if there's a PEL 
>>> (Platform Event Log) hidden in the eSEL, admins/users of the error log 
>>> need to depend on external tools to parse the PEL to human readable
>>> text.
>> My lack of love for PELs is probably fairly well known, but even if we
>> replace them down the line, I think there's opportunity here for making
>> things better for users (including myself).
>
> Just my 2 cents. The IPMI-standardized SELs are good first of all
> because of IPMI-standardized PEFs and PETs.
>
> Being able to decode the OEM SEL records is not just convenient for the
> users, but will also let them easily attach PEFs to such events if they
> want to. That's of course when PEFs are supported in OpenBMC which is
> currently not true, AFAIK.
>
> As a maintainer of ipmitool I can promise to promptly include the
> OpenPOWER event record decoder into ipmitool upstream code. All I need
> is the information on how to decode those events. So far I'm planning to
> try and deduce this information from hostboot and opal sources.

(Adding in Tim from Raptor as we were talking about this yesterday)

There's opal-elog-parse, which is part of ppc64-diag, and there's
https://github.com/open-power/libopalevents which is a quite
bitrotted attempt to turn parsing it into a shared library rather than
just a binary.

there's also a (internal but in the process of being open sourced)
implementation called 'errl' which is what has shipped on IBM FSP
systems forever (but that for whatever reason we ended up doing
opal-elog-parse rather than open sourcing it back in 2014).

From hostboot, ./src/build/debug/eSEL.pl is the script that extracts
things from the BMC IIRC.

Now, there *is* a spec for the Platfrom Error Log format, which I have
previously tried to get out into the open... so I encourage you to find
all your IBM contacts and bug them about it.

That being said, I'm not convinced PEL is the right thing to continue to
exist. It's Yet Another Custom Binary Format, and there's probably
enough of those in the world as it is.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: Parse PELs sent by OpenPOWER host firmware

[Vasant Hegde]

On 03/13/2018 07:09 PM, Deepak Kodihalli wrote:
> Hi,
> 
> The SELs/eSELs (System Event Log) that OpenBMC receives from hostboot/skiboot 
> today are not interpreted, and are dumped as-is in a property of the error log 
> D-Bus object. This means that if there's a PEL (Platform Event Log) hidden in 
> the eSEL, admins/users of the error log need to depend on external tools to 
> parse the PEL to human readable text.

Yes.  We are having hard time with this. Any improvement is welcome.


> 
> Here's a proposal to have the BMC itself parse such eSELs (the motivation is to 
> make the OpenBMC error logs more useful in this context, without needing the 
> extra steps to parse such errors) :
> 
> - Separate this from phosphor-logging. This code can't go to a phosphor app, 
> because it's OpenPOWER specific. Have, for example, an openpower-logging app 
> implement a D-Bus interface to parse out a PEL from an eSEL. This app will host 
> it's own D-Bus service.
> 
> - Such an app can watch for error D-Bus objects being created in the phosphor 
> namespace. If it finds an eSEL in an error log, it can attempt to parse a PEL 
> (if there's one), and store the PELs parsed text representation in a D-Bus 
> property.
> 
> - The app can create a D-Bus object with the same path as the original object 
> and have a D-Bus object-manager at the error log root path, so that when someone 
> uses the OpenBMC REST API to read an error log, properties across all D-Bus 
> services will be returned (phosphor-rest-server does this today already). This 
> means with the existing REST API to enumerate error logs, a parsed PEL would be 
> returned, if present.
> 
> - To parse the PEL, use opal-elog-parse.

+1. I think it has capability to detect eSEL wrapper as well. It should just 
work fine.

-Vasant