new port seeing ipmid exiting with seg fault

new port seeing ipmid exiting with seg fault

[Patton, Schuyler]

[-- Attachment #1: Type: text/plain, Size: 3879 bytes --]

Hi all,

In our port the ipmid is exiting with a seg fault.  Does anyone have any suggestions on what to look at or what the problem might be? I have included some info I collected from systemctl and journalctl. Thanks in advance for any pointers, suggestions.

root@evb-am62xx:~# systemctl status phosphor-ipmi-host
x phosphor-ipmi-host.service - Phosphor Inband IPMI
     Loaded: loaded (/lib/systemd/system/phosphor-ipmi-host.service; enabled; vendor preset: enabled)
    Drop-In: /lib/systemd/system/phosphor-ipmi-host.service.d
             `-10-override.conf
     Active: failed (Result: core-dump) since Mon 2022-08-29 15:01:40 UTC; 3min 8s ago
   Duration: 1.163s
    Process: 368 ExecStart=/usr/bin/env ipmid (code=dumped, signal=SEGV)
   Main PID: 368 (code=dumped, signal=SEGV)

Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Scheduled restart job, restart counter is at 2.
Aug 29 15:01:40 evb-am62xx systemd[1]: Stopped Phosphor Inband IPMI.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Start request repeated too quickly.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Failed with result 'core-dump'.
Aug 29 15:01:40 evb-am62xx systemd[1]: Failed to start Phosphor Inband IPMI.


root@evb-am62xx:~# journalctl | grep ipmi
Jan 01 00:00:04 evb-am62xx systemd[1]: /lib/systemd/system/phosphor-ipmi-net@.socket:3: Invalid interface name, ignoring: sys-subsystem-net-devices-%i.device
Jan 01 00:00:04 evb-am62xx systemd[1]: Created slice Slice /system/phosphor-ipmi-net.
Aug 29 15:01:19 evb-am62xx systemd[1]: Listening on phosphor-ipmi-net@eth0.socket.
Aug 29 15:01:21 evb-am62xx ipmid[329]: JSON file not found
Aug 29 15:01:22 evb-am62xx systemd-coredump[339]: Process 334 (netipmid) of user 0 dumped core.
Aug 29 15:01:22 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Main process exited, code=dumped, status=11/SEGV
Aug 29 15:01:22 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Failed with result 'core-dump'.
Aug 29 15:01:23 evb-am62xx systemd-coredump[338]: Process 329 (ipmid) of user 0 dumped core.
Aug 29 15:01:23 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Main process exited, code=dumped, status=11/SEGV
Aug 29 15:01:23 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Failed with result 'core-dump'.
Aug 29 15:01:38 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Scheduled restart job, restart counter is at 1.
Aug 29 15:01:38 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Scheduled restart job, restart counter is at 1.
Aug 29 15:01:39 evb-am62xx systemd-coredump[373]: Process 370 (netipmid) of user 0 dumped core.
Aug 29 15:01:39 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Main process exited, code=dumped, status=11/SEGV
Aug 29 15:01:39 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Failed with result 'core-dump'.
Aug 29 15:01:39 evb-am62xx systemd-coredump[371]: Process 368 (ipmid) of user 0 dumped core.
Aug 29 15:01:39 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Main process exited, code=dumped, status=11/SEGV
Aug 29 15:01:39 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Failed with result 'core-dump'.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Scheduled restart job, restart counter is at 2.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Scheduled restart job, restart counter is at 2.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Start request repeated too quickly.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-host.service: Failed with result 'core-dump'.
Aug 29 15:01:40 evb-am62xx systemd[1]: phosphor-ipmi-net@eth0.service: Job phosphor-ipmi-net@eth0.service/start failed with result 'dependency'.

Regards,
Schuyler Patton
Sitara MPU System Applications
Texas Instruments


[-- Attachment #2: Type: text/html, Size: 7652 bytes --]

Re: new port seeing ipmid exiting with seg fault

[Andrew Jeffery]

Hi Schuyler,

On Tue, 30 Aug 2022, at 00:45, Patton, Schuyler wrote:
> Hi all,
>
> In our port the ipmid is exiting with a seg fault.  Does anyone have 
> any suggestions on what to look at or what the problem might be? I have 
> included some info I collected from systemctl and journalctl. Thanks in 
> advance for any pointers, suggestions.

You might find some useful tricks for debugging the segfault in my blog 
post here:

https://amboar.github.io/notes/2022/01/13/openbmc-development-workflow.html

Broadly, the idea is to get gdb hooked up to debug either the process 
or the core dump. If you can extract the core dump from the system then 
you can use the bbdbg script in concert with your local OpenBMC build 
tree to generate a rootfs context to debug the core using 
gdb-multiarch. The other way to go is to get gdbserver onto the BMC and 
then use remote gdb (again using e.g. bbdbg to generate the context in 
which gdb runs).

Hope that helps,

Andrew

RE: [EXTERNAL] Re: new port seeing ipmid exiting with seg fault

[Patton, Schuyler]

Hi Andrew and all,

I have used remote gdb to debug the cause of the seg faults for phosphor-host-ipmid to this structure (MetaPassStruct in user_channel/passwd_mgr.cpp). This structure appears to initialized correctly, extremely large values. I notice that a commit (05703ad - Anton Blanchard, I am using a 64 bit machine) was made to change structure elements of the MetaPassStruct and the commit message mentioned this would fix seg faults. I added a bbappends that pulls in the ipmid current top of tree of a23af1 and this included 05703ad commit but the ipmid now fails before getting to the passwdmgr that was seg faulting earlier. 

Apologies as I am new to OpenBMC but I was looking for guidance on which ipmid commit I should use beyond the bfd3a17 commit that seg faults.

Regards,
Schuyler

-----Original Message-----
From: Andrew Jeffery <andrew@aj.id.au> 
Sent: Monday, August 29, 2022 9:00 PM
To: Patton, Schuyler <spatton@ti.com>; openbmc@lists.ozlabs.org
Subject: [EXTERNAL] Re: new port seeing ipmid exiting with seg fault

Hi Schuyler,

On Tue, 30 Aug 2022, at 00:45, Patton, Schuyler wrote:
> Hi all,
>
> In our port the ipmid is exiting with a seg fault.  Does anyone have 
> any suggestions on what to look at or what the problem might be? I 
> have included some info I collected from systemctl and journalctl. 
> Thanks in advance for any pointers, suggestions.

You might find some useful tricks for debugging the segfault in my blog post here:

https://amboar.github.io/notes/2022/01/13/openbmc-development-workflow.html

Broadly, the idea is to get gdb hooked up to debug either the process or the core dump. If you can extract the core dump from the system then you can use the bbdbg script in concert with your local OpenBMC build tree to generate a rootfs context to debug the core using gdb-multiarch. The other way to go is to get gdbserver onto the BMC and then use remote gdb (again using e.g. bbdbg to generate the context in which gdb runs).

Hope that helps,

Andrew

Re: [EXTERNAL] Re: new port seeing ipmid exiting with seg fault

[Andrew Jeffery]

On Thu, 22 Sep 2022, at 15:14, Patton, Schuyler wrote:
> Hi Andrew and all,
>
> I have used remote gdb to debug the cause of the seg faults for 
> phosphor-host-ipmid to this structure (MetaPassStruct in 
> user_channel/passwd_mgr.cpp). This structure appears to initialized 
> correctly, extremely large values. I notice that a commit (05703ad - 
> Anton Blanchard, I am using a 64 bit machine) was made to change 
> structure elements of the MetaPassStruct and the commit message 
> mentioned this would fix seg faults. I added a bbappends that pulls in 
> the ipmid current top of tree of a23af1 and this included 05703ad 
> commit but the ipmid now fails before getting to the passwdmgr that was 
> seg faulting earlier. 
>
> Apologies as I am new to OpenBMC but I was looking for guidance on 
> which ipmid commit I should use beyond the bfd3a17 commit that seg 
> faults.
>

You'll need to reproduce the issue against the upstream code before we 
can help much further here. It sounds like you're doing a fair bit of 
munging of the history and it's unclear what upstream base you're 
working from and what changes you've introduced. It's not really 
something we can help with as it is.

Andrew

Re: [EXTERNAL] Re: new port seeing ipmid exiting with seg fault

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

Hello Schuyler,

On Fri, Sep 23, 2022 at 09:36:30AM +0930, Andrew Jeffery wrote:
> On Thu, 22 Sep 2022, at 15:14, Patton, Schuyler wrote:
> > Apologies as I am new to OpenBMC but I was looking for guidance on 
> > which ipmid commit I should use beyond the bfd3a17 commit that seg 
> > faults.
> >
> 
> You'll need to reproduce the issue against the upstream code before we 
> can help much further here. 

To reiterate, this is really the answer for all repositories.  We don't
have people that support release branches of code.  Most people work off
HEAD in this project.

If you find a bug either the latest commit of a repository or the version
pointed to by the recipe in the latest commit of openbmc/openbmc, someone
is much more likely to look into it.  If you are trying to report bugs on
something older it is quite likely going to get very little response.

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]