systemd + run-postinsts.service

systemd + run-postinsts.service

[Patrick Ohly]

Hello!

I'm using systemd from OE-core master to boot core-image-minimal. I'm
seeing a boot failure where "Run pending postinsts" (aka
run-postinsts.service) gets stuck during booting under very specific
circumstances:

     1. The package has a pkg_postinst_${PN} which (intentionally)
        failed during image building, thus delaying its execution to the
        first boot.
     2. The package uses "inherit systemd" to enable its own systemd
        service.

It seems to be "systemctl start <service name>" which is hanging,
because I can reproduce the same behavior without systemd.bbclass in the
following pkg_postinst:

pkg_postinst_${PN} () {
#!/bin/sh -e
   # Cannot run during image building...
   if [ x"$D" != "x" ]; then
      exit 1
   fi

   # Do something on target during first boot.

   systemctl enable foobar
   systemctl start foobar # <====
}

It works without the last line. The actual recipe is a bit more complex,
but I can try to create a dummy one if that would help.

Any idea what could be causing this and, more importantly, how to fix
it?

Re: systemd + run-postinsts.service

[Jonathan Liu]

On 24/03/2015 8:57 AM, Patrick Ohly wrote:
> Hello!
>
> I'm using systemd from OE-core master to boot core-image-minimal. I'm
> seeing a boot failure where "Run pending postinsts" (aka
> run-postinsts.service) gets stuck during booting under very specific
> circumstances:
>
>       1. The package has a pkg_postinst_${PN} which (intentionally)
>          failed during image building, thus delaying its execution to the
>          first boot.
>       2. The package uses "inherit systemd" to enable its own systemd
>          service.
>
> It seems to be "systemctl start <service name>" which is hanging,
> because I can reproduce the same behavior without systemd.bbclass in the
> following pkg_postinst:
>
> pkg_postinst_${PN} () {
> #!/bin/sh -e
>     # Cannot run during image building...
>     if [ x"$D" != "x" ]; then
>        exit 1
>     fi
>
>     # Do something on target during first boot.
>
>     systemctl enable foobar
>     systemctl start foobar # <====
> }
>
> It works without the last line. The actual recipe is a bit more complex,
> but I can try to create a dummy one if that would help.
>
> Any idea what could be causing this and, more importantly, how to fix
> it?
>
>
>
I am not sure how well systemd works with executing systemctl to 
manipulate services from within a service that is starting...

What is the definition of foobar.service?
foobar.service may depend on other services/mounts/targets to start 
before it but since "systemctl start" is blocking, those dependencies 
may not be satisfied yet until sometime after run-postinsts.service 
which can result in a circular dependency deadlock. See 
DefaultDependencies, After, Before in 
http://www.freedesktop.org/software/systemd/man/systemd.unit.html.

Things to try:
1. Refactor foobar.service and your image recipe to not require calling 
systemctl from within a systemd service
2. Change "systemctl start foobar" to "systemctl start --no-block 
foobar" to let the startup continue without waiting until foobar is started

Regards,
Jonathan

Re: systemd + run-postinsts.service

[Patrick Ohly]

On Tue, 2015-03-24 at 09:33 +1100, Jonathan Liu wrote:
> I am not sure how well systemd works with executing systemctl to 
> manipulate services from within a service that is starting...
>
> What is the definition of foobar.service?

Manipulating services within a service works. You were on the right
track when suspecting a dependency deadlock.

> foobar.service may depend on other services/mounts/targets to start 
> before it but since "systemctl start" is blocking, those dependencies 
> may not be satisfied yet until sometime after run-postinsts.service 
> which can result in a circular dependency deadlock. See 
> DefaultDependencies, After, Before in 
> http://www.freedesktop.org/software/systemd/man/systemd.unit.html.
> 
> Things to try:
> 1. Refactor foobar.service and your image recipe to not require calling 
> systemctl from within a systemd service
> 2. Change "systemctl start foobar" to "systemctl start --no-block 
> foobar" to let the startup continue without waiting until foobar is started

"--no-block" did indeed help. The recipe uses only the default
dependencies, but that already seems enough to prevent starting it at
the time when run-postinsts.service runs.

I guess the situation hasn't come up before. But when it does, it
probably breaks systemd.bbclass for most services. I wonder whether
--no-block should be added unconditionally to systemd.bblcass'
"systemctl start" invocation?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.