checkstop processing

checkstop processing

[Sergey Kachkin]

[-- Attachment #1: Type: text/plain, Size: 678 bytes --]

Hi all,

i'm investigating the checkstop processing and looking for a way to isolate
a faulty component with OpenBmc.
So far SEL logs available via REST are not really helpful.

Is there any data source in the openbmc to troubleshoot checkstops?

I guess eSEL binary data parsed with eSEL.pl can be more informative but do
we have any procedure to grab the binary sel data and parse it with the
latest obmc?

Currently it seems that IPL checkstop analysis is not really working. i
mean that faulty component is not deconfigured on the next boot and gard
list is empty.
It can be easily duplicated by injecting an error manually via putscom.

thanks in advance,

regards,
Sergey

[-- Attachment #2: Type: text/html, Size: 909 bytes --]

Re: checkstop processing

[Joel Stanley]

On Tue, Nov 14, 2017 at 8:04 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
> Hi all,
>
> i'm investigating the checkstop processing and looking for a way to isolate
> a faulty component with OpenBmc.
> So far SEL logs available via REST are not really helpful.
>
> Is there any data source in the openbmc to troubleshoot checkstops?
>
> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
> we have any procedure to grab the binary sel data and parse it with the
> latest obmc?
>
> Currently it seems that IPL checkstop analysis is not really working. i mean
> that faulty component is not deconfigured on the next boot and gard list is
> empty.
> It can be easily duplicated by injecting an error manually via putscom.

I think you've identified an area that would be great for improvement.

I'd like to expand the scope beyond just checkstop to other boot
failures: I've tried to boot machines recently that have failed to
even start hostboot, and I haven't known what has failed.

A tool that inspects recent error logs, and the state of the SBE would
be useful. We can leverage libpdbg to talk to the host.

Cheers,

Joel

Re: checkstop processing

[Oliver]

On Tue, Nov 14, 2017 at 2:42 PM, Joel Stanley <joel@jms.id.au> wrote:
> On Tue, Nov 14, 2017 at 8:04 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
>> Hi all,
>>
>> i'm investigating the checkstop processing and looking for a way to isolate
>> a faulty component with OpenBmc.
>> So far SEL logs available via REST are not really helpful.
>>
>> Is there any data source in the openbmc to troubleshoot checkstops?
>>
>> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
>> we have any procedure to grab the binary sel data and parse it with the
>> latest obmc?
>>
>> Currently it seems that IPL checkstop analysis is not really working. i mean
>> that faulty component is not deconfigured on the next boot and gard list is
>> empty.
>> It can be easily duplicated by injecting an error manually via putscom.
>
> I think you've identified an area that would be great for improvement.
>
> I'd like to expand the scope beyond just checkstop to other boot
> failures: I've tried to boot machines recently that have failed to
> even start hostboot, and I haven't known what has failed.
>
> A tool that inspects recent error logs, and the state of the SBE would
> be useful. We can leverage libpdbg to talk to the host.

The SBE stores some state information in cfam 2809 that we can use to
find out the currents istep. I think we can also dump the SBE trace
buffer out of PIB memory on non-secure systems too. Parsing the trace
buffer requires the tracehash file from the SBE build, but we can
probably able to add that to the squashfs file for the host firmware.

>
> Cheers,
>
> Joel

Re: checkstop processing

[Stewart Smith]

Oliver <oohall@gmail.com> writes:
> On Tue, Nov 14, 2017 at 2:42 PM, Joel Stanley <joel@jms.id.au> wrote:
>> On Tue, Nov 14, 2017 at 8:04 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
>>> Hi all,
>>>
>>> i'm investigating the checkstop processing and looking for a way to isolate
>>> a faulty component with OpenBmc.
>>> So far SEL logs available via REST are not really helpful.
>>>
>>> Is there any data source in the openbmc to troubleshoot checkstops?
>>>
>>> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
>>> we have any procedure to grab the binary sel data and parse it with the
>>> latest obmc?
>>>
>>> Currently it seems that IPL checkstop analysis is not really working. i mean
>>> that faulty component is not deconfigured on the next boot and gard list is
>>> empty.
>>> It can be easily duplicated by injecting an error manually via putscom.
>>
>> I think you've identified an area that would be great for improvement.
>>
>> I'd like to expand the scope beyond just checkstop to other boot
>> failures: I've tried to boot machines recently that have failed to
>> even start hostboot, and I haven't known what has failed.
>>
>> A tool that inspects recent error logs, and the state of the SBE would
>> be useful. We can leverage libpdbg to talk to the host.
>
> The SBE stores some state information in cfam 2809 that we can use to
> find out the currents istep. I think we can also dump the SBE trace
> buffer out of PIB memory on non-secure systems too. Parsing the trace
> buffer requires the tracehash file from the SBE build, but we can
> probably able to add that to the squashfs file for the host firmware.

This would be ideal to put in a sensor for boot progress.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: checkstop processing

[Stewart Smith]

Joel Stanley <joel@jms.id.au> writes:
> On Tue, Nov 14, 2017 at 8:04 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
>> Hi all,
>>
>> i'm investigating the checkstop processing and looking for a way to isolate
>> a faulty component with OpenBmc.
>> So far SEL logs available via REST are not really helpful.
>>
>> Is there any data source in the openbmc to troubleshoot checkstops?
>>
>> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
>> we have any procedure to grab the binary sel data and parse it with the
>> latest obmc?
>>
>> Currently it seems that IPL checkstop analysis is not really working. i mean
>> that faulty component is not deconfigured on the next boot and gard list is
>> empty.
>> It can be easily duplicated by injecting an error manually via putscom.
>
> I think you've identified an area that would be great for improvement.

Understatement of the year right there :)

This (of course) isn't an OpenBMC specific problem, but rather an
opportunity for OpenBMC to clearly excel against other BMC
implementations.

I'd love to see even the parsed ESELs show up through the REST API,
rather than the current mess which is literally just "printf("ESEL=%02x
%02x %02x...)".

If we have a PEL hidden in there, there's existing userspace to parse it
too (opal-elog-parse), and there's no reason why the BMC couldn't just
output the text representation of it all in addition to the binary.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: checkstop processing

[Oliver]

On Tue, Nov 14, 2017 at 8:34 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
> Hi all,
>
> i'm investigating the checkstop processing and looking for a way to isolate
> a faulty component with OpenBmc.

What did you have in mind? The IPL time checkstop analysis that
hostboot does *should* handle all this stuff for you. I'm not sure how
straightforward porting that functionality to the BMC would be since
it might require access to data from the system's MRW.

> So far SEL logs available via REST are not really helpful.
>
> Is there any data source in the openbmc to troubleshoot checkstops?
>
> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
> we have any procedure to grab the binary sel data and parse it with the
> latest obmc?
>
> Currently it seems that IPL checkstop analysis is not really working. i mean
> that faulty component is not deconfigured on the next boot and gard list is
> empty.
> It can be easily duplicated by injecting an error manually via putscom.

What errors are you injecting and what are you using to check for GARD
records? There's an open bug (SW404983) concerning hostboot generating
bad gard records which the openpower gard tool doesn't understand and
a side effect of that bug is that hostboot might overwrite records
rather than creating a new one. You might be getting bitten by that.

>
> thanks in advance,
>
> regards,
> Sergey
>

Re: checkstop processing

[Balbir Singh]

On Tue, Nov 14, 2017 at 8:34 AM, Sergey Kachkin <s.kachkin@gmail.com> wrote:
> Hi all,
>
> i'm investigating the checkstop processing and looking for a way to isolate
> a faulty component with OpenBmc.
> So far SEL logs available via REST are not really helpful.
>
> Is there any data source in the openbmc to troubleshoot checkstops?
>

Not yet! I guess you'd want to use some of the built-in pdbg infrastructure
to look at the checkstop issues.



> I guess eSEL binary data parsed with eSEL.pl can be more informative but do
> we have any procedure to grab the binary sel data and parse it with the
> latest obmc?
>

The workflow as I understand is

1. run IPMI commands from another host, extract eSEL logs
2. Decode those logs with eSEL.pl

Hostboot has gotten better at decoding checkstops at boot, so thats a
good first step



> Currently it seems that IPL checkstop analysis is not really working. i mean
> that faulty component is not deconfigured on the next boot and gard list is
> empty.
> It can be easily duplicated by injecting an error manually via putscom.

I've seen the opposite to be honest. What error are you injecting?

Balbir Singh.