Shutting down a container?

Shutting down a container?

[Michael Tokarev]

What's the supposed way to stop a container?

Let's assume I've a container that's running a complete
system inside, together with its own init and the like.
It should clean some stuff when "shutting down", it's
not wise to kill it right away.  So I run something like
a shutdown in the container.  And it goes up to `halt'
and just sits here, with only one process left - it's
the init process with virtual PID=1, which is reading
from it's /dev/initctl.

I tried to change /etc/init.d/halt to kill a process with
pid=1, but that does not work - apparently the signal is
ignored (even with kill -9) -- ie, init continues as if
there was no signal.

_But_, when I attempt to run strace on the ("virtual")
init process, it gets killed with `kill -9' immediately.

It also can be killed (also with -9) from the host system
regardless of strace.

Should there be some different init perhaps, that will
exit given some command or signal, instead of sysvinit?
Or is there other way to shutdown a container?

P.S.  Using debian lenny with 2.6.31.6 kernel and lxc
utils from yesterdays git.

Thanks!

/mjt

Re: Shutting down a container?

[Daniel Lezcano]

Michael Tokarev wrote:
> What's the supposed way to stop a container?
>
> Let's assume I've a container that's running a complete
> system inside, together with its own init and the like.
> It should clean some stuff when "shutting down", it's
> not wise to kill it right away.  So I run something like
> a shutdown in the container.  And it goes up to `halt'
> and just sits here, with only one process left - it's
> the init process with virtual PID=1, which is reading
> from it's /dev/initctl.
>
> I tried to change /etc/init.d/halt to kill a process with
> pid=1, but that does not work - apparently the signal is
> ignored (even with kill -9) -- ie, init continues as if
> there was no signal.
>
> _But_, when I attempt to run strace on the ("virtual")
> init process, it gets killed with `kill -9' immediately.
>
> It also can be killed (also with -9) from the host system
> regardless of strace.
>
> Should there be some different init perhaps, that will
> exit given some command or signal, instead of sysvinit?
> Or is there other way to shutdown a container?
>
> P.S.  Using debian lenny with 2.6.31.6 kernel and lxc
> utils from yesterdays git.
>   
Unfortunately, it's not supported by the kernel yet.

Random thought : may be it could be interesting to have the kernel to 
send a signal to the init's parent when:

 * LINUX_REBOOT_CMD_POWER_OFF : the parent process kills the pid 1

 * LINUX_REBOOT_CMD_RESTART : the parent process kills the pid 1 and 
relaunch the container

 * LINUX_REBOOT_CMD_HALT: the parent process freeze the container
 
...

may be later:
 * LINUX_REBOOT_CMD_SW_SUSPEND : the parent process checkpoint the container

Well it's just an idea.

Re: [lxc-devel] Shutting down a container?

[Andrian Nord]

On Fri, Nov 13, 2009 at 06:58:01PM +0100, Daniel Lezcano wrote:
> Michael Tokarev wrote:
> >
> > Should there be some different init perhaps, that will
> > exit given some command or signal, instead of sysvinit?
> > Or is there other way to shutdown a container?
> >

There is two techniques, that seems to be 'common':
1) Write your own init script, which would run every process you need
and traping some signal via 'trap' that will exec an shutdown event.
It has big advantage of lower overhead for rc-subsystem, but much more
complicated to maintain.

2) Second technique is to rewrite container's /etc/inittab of sysvinit
to have line like that:
ca:12345:ctrlaltdel:/sbin/shutdown -hP now

This is, obviously, what init should execute on ctrl-alt-del shortcut,
actually triggered by SIGINT signal sent to init process.

So, full-automated solution is
1) find container's init's pid via lxc-ps (speaking of lxc utilities).
2) sed/echo-in container's inittab via /proc/${initpid}/root/etc/inittab
3) kill -INT ${initpid}
4) in loop, until all non-inits are gone:
	take random not-init container process from lxc-ps and 'wait'
	for it's termination.
5) 'kill -9' init, or just 'lxc-stop' container.

Waiting for pid may be problematic, as you may wait only for your child,
but if you'll do this via rc-system, it should work. At least it works
for openrc =)

This is hack, yes, but there is no common way of inserting commands into
container by now (there was such thread into lxc-devel maillist), but
solution is found and needs implementing.

Much worse situation is with syncing start procedure / waiting for
network readyness =).

Re: [lxc-devel] Shutting down a container?

[Michael Tokarev]

[I'm not really sure anymore what's the proper address:
  the question at hand seems to be generic enough for
  containers@]

Andrian Nord wrote:
> On Fri, Nov 13, 2009 at 06:58:01PM +0100, Daniel Lezcano wrote:
>> Michael Tokarev wrote:
>>> Should there be some different init perhaps, that will
>>> exit given some command or signal, instead of sysvinit?
>>> Or is there other way to shutdown a container?
>>>
> 
> There is two techniques, that seems to be 'common':
> 1) Write your own init script, which would run every process you need
> and traping some signal via 'trap' that will exec an shutdown event.
> It has big advantage of lower overhead for rc-subsystem, but much more
> complicated to maintain.

Well, I don't quite see the complexity.  sysvinit here is nothing more
than an equivalent of run-parts.  It's the /etc/init.d/rc (or whatever
the path is on your distro) shell script who does all the fun stuff,
init itself basically does nothing but catching SIGCHLD and (re)starting
whatever is in inittab (which, in this case, should be very limited).

So I guess that a custom simple init-alike is all that's needed.  The
main difference is that it should exit when there's no more processes
running or after some signal (like /dev/initctl, since process with pid=1
ignores signals).

> 2) Second technique is to rewrite container's /etc/inittab of sysvinit
> to have line like that:
> ca:12345:ctrlaltdel:/sbin/shutdown -hP now
> 
> This is, obviously, what init should execute on ctrl-alt-del shortcut,
> actually triggered by SIGINT signal sent to init process.

For now I'm not worried about how to signal the "guest" to shut down.
it can be easily done by, say, attaching something to /dev/tty1 and
sening a command using lxc-console.  Do unix sockets work between a
container and "host system" ?

What does not work for now is the final step.  init process should exit
but it does not do that and can't be killed from within the container.

[]
> This is hack, yes, but there is no common way of inserting commands into
> container by now (there was such thread into lxc-devel maillist), but
> solution is found and needs implementing.

I'll dig there.  But yet again, I'm fine with modifying the container
initscripts or something, the question really is what to do.

By the way, I for one see no reason why pid=1 in a container is special
when delivering signals or the like.

> Much worse situation is with syncing start procedure / waiting for
> network readyness =).

That again can be done by placing something into container that will
ping a given address when everything else is done.

Thanks!

/mjt

Re: [lxc-devel] Shutting down a container?

[Andrian Nord]

containers@ ml, imo, is more kernel-developing. Your questions are about
userspace and they probably should be asked in ml of userspace utilities
you are using (because implementing many of workarounds of current
kernelspace incapabilities is userspace utilities work, and your problem
may be already solved, but in userspace).

On Sat, Nov 14, 2009 at 02:17:45PM +0300, Michael Tokarev wrote:
> Well, I don't quite see the complexity.  sysvinit here is nothing more
> than an equivalent of run-parts.  It's the /etc/init.d/rc (or whatever
> the path is on your distro) shell script who does all the fun stuff,
> init itself basically does nothing but catching SIGCHLD and (re)starting
> whatever is in inittab (which, in this case, should be very limited).
> 
> So I guess that a custom simple init-alike is all that's needed.  The
> main difference is that it should exit when there's no more processes
> running or after some signal (like /dev/initctl, since process with pid=1
> ignores signals).

Yeap, but sysvinit is universal aproach, so if you use it, you may jasu
use common rc system for container's distro and never mind any problems
with initialising etc (writing a good init is not a trivial task, imo).

> For now I'm not worried about how to signal the "guest" to shut down.
> it can be easily done by, say, attaching something to /dev/tty1 and
> sening a command using lxc-console.  Do unix sockets work between a
> container and "host system" ?

Problem is that
a) lxc-console may not work, if you haven't enabled ttys
b) you may have hudrends of containers

Anyway, attaching to tty is also a bad hack, imo, but just least common =)

No, if you are isolating network, unix sockets will also break. There
is a thread about 'af_unix isolation' at lxc-devel ml, check it for
possible (but, sure, not trivial) workaround.

> 
> What does not work for now is the final step.  init process should exit
> but it does not do that and can't be killed from within the container.
> 
> []

As I said before - you should wait for pids of container and lxc-stop it
when all pids except sysvinit are gone. If you are using custom init,
you may change this behaviour (as you are not worried about kernel panic
or init die anymore) and silently quit when all processes are gone.

> By the way, I for one see no reason why pid=1 in a container is special
> when delivering signals or the like.

It's not special. This is normal behaviour of sysvinit, read it's
manpages.

> That again can be done by placing something into container that will
> ping a given address when everything else is done.

Yes, yes, that is. Questing is in some universal approach, that will
work without hacks or assumptions about container's structure =)