nsenter and SIGSTOP

nsenter and SIGSTOP

[Zbigniew Jędrzejewski-Szmek]

Hi,
I've hit a bit of a problem with nsenter and systemd-nspawn.
When nsenter is used to enter the PID namespace created with
systemd-nspawn, and the container's init attempts a shutdown,
it hangs because nsenter is suspended.

The sequence of events leading to the hang is:

1. nsenter launches a shell inside the container with
   PPID=0 as seen inside the container,
2. systemd with PID=1 goes through the shutdown sequence,
   issuing the equivalent(*) of

   kill(-1, SIGSTOP)
   kill(-1, SIGTERM)
   kill(_1, SIGCONT)
   reboot(RB_HALT_SYSTEM)

Now, nsenter has a stanza in continue_as_child where it stops itself
when the child gets stopped. Unfortunately, this means that nsenter
gets stopped in response to kill(-1, SIGSTOP) which hits the child.
Then the child dies on kill(-1, SIGTERM), is resumed with kill(-1,
SIGCONT) and exits (it prints "exit", so it's easy to see that it
terminated properly. Then the shell becomes a zombie, since nsenter it
it's parent and it's sleeping. Meanwhile, init executes reboot, and
hangs in there, since the container waits for the PID namespace to
become empty (I'm guessing here, but that seems logical). When then
I type 'fg' to continue nsenter, the child gets collected and the
container successfully exits.

This is with kernel 3.9-rc6 from Fedora.

Thanks,
Zbyszek

Re: [systemd-devel] nsenter and SIGSTOP

[Eric W. Biederman]

WmJpZ25pZXcgSsSZZHJ6ZWpld3NraS1Tem1layA8emJ5c3pla0Bpbi53YXcucGw+IHdyaXRlczoK
Cj4gSGksCj4gSSd2ZSBoaXQgYSBiaXQgb2YgYSBwcm9ibGVtIHdpdGggbnNlbnRlciBhbmQgc3lz
dGVtZC1uc3Bhd24uCj4gV2hlbiBuc2VudGVyIGlzIHVzZWQgdG8gZW50ZXIgdGhlIFBJRCBuYW1l
c3BhY2UgY3JlYXRlZCB3aXRoCj4gc3lzdGVtZC1uc3Bhd24sIGFuZCB0aGUgY29udGFpbmVyJ3Mg
aW5pdCBhdHRlbXB0cyBhIHNodXRkb3duLAo+IGl0IGhhbmdzIGJlY2F1c2UgbnNlbnRlciBpcyBz
dXNwZW5kZWQuCj4KPiBUaGUgc2VxdWVuY2Ugb2YgZXZlbnRzIGxlYWRpbmcgdG8gdGhlIGhhbmcg
aXM6Cj4KPiAxLiBuc2VudGVyIGxhdW5jaGVzIGEgc2hlbGwgaW5zaWRlIHRoZSBjb250YWluZXIg
d2l0aAo+ICAgIFBQSUQ9MCBhcyBzZWVuIGluc2lkZSB0aGUgY29udGFpbmVyLAo+IDIuIHN5c3Rl
bWQgd2l0aCBQSUQ9MSBnb2VzIHRocm91Z2ggdGhlIHNodXRkb3duIHNlcXVlbmNlLAo+ICAgIGlz
c3VpbmcgdGhlIGVxdWl2YWxlbnQoKikgb2YKPgo+ICAgIGtpbGwoLTEsIFNJR1NUT1ApCgpUaGlz
IGJhZmZsZXMgbWUuICBJIGFtIG5vdCBjZXJ0YWluIHdoeSBzb21lb25lIHdob3VsZCBzZW5kIFNJ
R1NUT1AKd2hlbiB0aGUgd2FudCBwcm9jZXNzZXMgdG8gZXhpdC4gIEknbSBub3QgZXZlbiBzYXlp
bmcgaXQncyB3cm9uZyBqdXN0CnNheWluZyB0aGF0IGlzIG9kZC4KCj4gICAga2lsbCgtMSwgU0lH
VEVSTSkKPiAgICBraWxsKF8xLCBTSUdDT05UKQo+ICAgIHJlYm9vdChSQl9IQUxUX1NZU1RFTSkK
Pgo+IE5vdywgbnNlbnRlciBoYXMgYSBzdGFuemEgaW4gY29udGludWVfYXNfY2hpbGQgd2hlcmUg
aXQgc3RvcHMgaXRzZWxmCj4gd2hlbiB0aGUgY2hpbGQgZ2V0cyBzdG9wcGVkLiBVbmZvcnR1bmF0
ZWx5LCB0aGlzIG1lYW5zIHRoYXQgbnNlbnRlcgo+IGdldHMgc3RvcHBlZCBpbiByZXNwb25zZSB0
byBraWxsKC0xLCBTSUdTVE9QKSB3aGljaCBoaXRzIHRoZSBjaGlsZC4KPiBUaGVuIHRoZSBjaGls
ZCBkaWVzIG9uIGtpbGwoLTEsIFNJR1RFUk0pLCBpcyByZXN1bWVkIHdpdGgga2lsbCgtMSwKPiBT
SUdDT05UKSBhbmQgZXhpdHMgKGl0IHByaW50cyAiZXhpdCIsIHNvIGl0J3MgZWFzeSB0byBzZWUg
dGhhdCBpdAo+IHRlcm1pbmF0ZWQgcHJvcGVybHkuIFRoZW4gdGhlIHNoZWxsIGJlY29tZXMgYSB6
b21iaWUsIHNpbmNlIG5zZW50ZXIgaXQKPiBpdCdzIHBhcmVudCBhbmQgaXQncyBzbGVlcGluZy4g
TWVhbndoaWxlLCBpbml0IGV4ZWN1dGVzIHJlYm9vdCwgYW5kCj4gaGFuZ3MgaW4gdGhlcmUsIHNp
bmNlIHRoZSBjb250YWluZXIgd2FpdHMgZm9yIHRoZSBQSUQgbmFtZXNwYWNlIHRvCj4gYmVjb21l
IGVtcHR5IChJJ20gZ3Vlc3NpbmcgaGVyZSwgYnV0IHRoYXQgc2VlbXMgbG9naWNhbCkuCgpJIGV4
cGVjdCB0aGUgaGFuZyBpcyBpbiB0aGUgcGlkIG5hbWVzcGFjZSBpbml0IGV4aXRpbmcuCmluIGtl
cm5lbC9waWRfbmFtZXNwYWNlLng6emFwX3BpZF9uc19wcm9jZXNzZXMoKSBoYXMgdGhlIGJhdmlv
dXIgb2YKYmxvY2tpbmcgdW50aWwgYWxsIGNoaWxkcmVuIG9mIGluaXQgaGF2ZSBiZWVuIHJlYXBl
ZCB0aGF0IHlvdSBkZXNjcmliZS4KCj4gV2hlbiB0aGVuCj4gSSB0eXBlICdmZycgdG8gY29udGlu
dWUgbnNlbnRlciwgdGhlIGNoaWxkIGdldHMgY29sbGVjdGVkIGFuZCB0aGUKPiBjb250YWluZXIg
c3VjY2Vzc2Z1bGx5IGV4aXRzLgo+Cj4gVGhpcyBpcyB3aXRoIGtlcm5lbCAzLjktcmM2IGZyb20g
RmVkb3JhLgoKRm9yIG5zZW50ZXIgYW5kIHRoZSBwaWQgbmFtZXNwYWNlIHRoZXkgYXJlIHdvcmtp
bmcgYXMgZGVzaWduZWQuICBCdXQKZ2l2ZW4gdGhpcyBvdXRjb2RlIGl0IHdvdWxkIGJlIG5pY2Ug
aWYgd2UgY291bGQgZ2V0IGEgU0lHQ09OVCB3aGVuIHRoZQpjaGlsZCB3YWtlcyB1cCBhZ2Fpbi4K
ClRoZSBjdXJyZW50IGJlaGF2aW9yIHN1cHBvcnRzIGJlaW5nIGFibGUgdG8gdHlwZSBzdXNwZW5k
IGluIHlvdXIgc2hlbGwKaW4gdGhlIG5hbWVzcGFjZSBhbmQgYWJsZSB0byB3b3JrIG91dHNpZGUg
dGhlIG5hbWVzcGFjZS4KCkkgY2FuJ3QgdGhpbmsgb2YgYSB3YXkgb2ZmIHRoZSB0b3Agb2YgbXkg
aGVhZCB0byB3YWtlIG5zZW50ZXIgdXAgd2hlbgppdCdzIGNoaWxkIGlzIHdva2VuIHVwIHVuZGVy
bmVhdGggaXQsIGJ1dCBpdCBzb3VuZHMgbGlrZSB0aGF0IHdvdWxkIGJlCm5pY2UgdG8gZG8uCgpG
b3IgdGhlIHNob3J0IHRlcm0gSSB3b3VsZCByZWNvbW1lbmQgbm90IHR5cGluZyAicmVib290ICYg
ZXhpdCIgaW5zdGVhZApvZiAicmVib290IiBmcm9tIGEgc2hlbGwgc3RhcnRlZCB3aXRoIG5zZW50
ZXIsIGFuZCBvdGhlcndpc2Ugbm90IGxlYXZpbmcKcHJvY2Vzc2VzIHdpdGggcGFyZW50cyBvdXRz
aWRlIHRoZSBwaWQgbmFtZXNwYWNlIGFyb3VuZC4KCk9mIGNvdXJzZSB0aGF0IHNlZGluZyBTSUdT
VE9QIGJlZm9yZSBzZW5kaW5nIFNJR1RFUk0gc2VlbXMgbWlnaHR5IGZpc2h5CmFzIHdlbGwuCgpF
cmljCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fCnN5c3Rl
bWQtZGV2ZWwgbWFpbGluZyBsaXN0CnN5c3RlbWQtZGV2ZWxAbGlzdHMuZnJlZWRlc2t0b3Aub3Jn
Cmh0dHA6Ly9saXN0cy5mcmVlZGVza3RvcC5vcmcvbWFpbG1hbi9saXN0aW5mby9zeXN0ZW1kLWRl
dmVsCg==

Re: [systemd-devel] nsenter and SIGSTOP

[Lennart Poettering]

On Sat, 20.04.13 15:27, Eric W. Biederman (ebiederm@xmission.com) wrote:

> Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:
> 
> > Hi,
> > I've hit a bit of a problem with nsenter and systemd-nspawn.
> > When nsenter is used to enter the PID namespace created with
> > systemd-nspawn, and the container's init attempts a shutdown,
> > it hangs because nsenter is suspended.
> >
> > The sequence of events leading to the hang is:
> >
> > 1. nsenter launches a shell inside the container with
> >    PPID=0 as seen inside the container,
> > 2. systemd with PID=1 goes through the shutdown sequence,
> >    issuing the equivalent(*) of
> >
> >    kill(-1, SIGSTOP)
> 
> This baffles me.  I am not certain why someone whould send SIGSTOP
> when the want processes to exit.  I'm not even saying it's wrong just
> saying that is odd.

This is how the killing spree generally worked on sysvinit systems too:
if we kill all remaining processes, then this might kill processes that
are children of others. Now, in general it is preferable getting the
SIGTERM/SIGKILL delivered to all these processes before possible SIGCHLD
signals for their dying children. Why? Simply to avoid log spew with
messages such as "Worker child process xyz died abnormally" generated by
the various system daemons.

By pausing all processes before delivering the SIGTERM/SIGKILL we can
ensure that if they are unpaused again the SIGTERM/SIGKILL is guaranteed
to be queued for each process and since on Linux lower-numbered signals
are guaranteed to be dispatched before higher-numbered signals (and
SIGKILL/SIGTERM is lower than SIGCHLD) we get the desired behaviour that
daemons exit quickly on SIGTERM/SIGKILL before handling the SIGCHLD.

Here's the code in sysvinit:

http://svn.savannah.nongnu.org/viewvc/sysvinit/trunk/src/killall5.c?revision=117&root=sysvinit&view=markup

Look for "kill(-1, SIGSTOP)".

Here's the code in systemd:

http://cgit.freedesktop.org/systemd/systemd/tree/src/core/killall.c#n185

Lennart

-- 
Lennart Poettering - Red Hat, Inc.

Re: nsenter and SIGSTOP

[Zbigniew Jędrzejewski-Szmek]

On Sat, Apr 20, 2013 at 03:27:46PM -0700, Eric W. Biederman wrote:
> Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:
> 
> > Hi,
> > I've hit a bit of a problem with nsenter and systemd-nspawn.
> > When nsenter is used to enter the PID namespace created with
> > systemd-nspawn, and the container's init attempts a shutdown,
> > it hangs because nsenter is suspended.
> >
> > The sequence of events leading to the hang is:
> >
> > 1. nsenter launches a shell inside the container with
> >    PPID=0 as seen inside the container,
> > 2. systemd with PID=1 goes through the shutdown sequence,
> >    issuing the equivalent(*) of
> >
> >    kill(-1, SIGSTOP)
> 
> This baffles me.  I am not certain why someone whould send SIGSTOP
> when the want processes to exit.  I'm not even saying it's wrong just
> saying that is odd.
Like Lennart wrote, it's for atomicity of the subsequent killing.

> >    kill(-1, SIGTERM)
> >    kill(_1, SIGCONT)
> >    reboot(RB_HALT_SYSTEM)
> >
> > Now, nsenter has a stanza in continue_as_child where it stops itself
> > when the child gets stopped. Unfortunately, this means that nsenter
> > gets stopped in response to kill(-1, SIGSTOP) which hits the child.
> > Then the child dies on kill(-1, SIGTERM), is resumed with kill(-1,
> > SIGCONT) and exits (it prints "exit", so it's easy to see that it
> > terminated properly. Then the shell becomes a zombie, since nsenter it
> > it's parent and it's sleeping. Meanwhile, init executes reboot, and
> > hangs in there, since the container waits for the PID namespace to
> > become empty (I'm guessing here, but that seems logical).
> 
> I expect the hang is in the pid namespace init exiting.
> in kernel/pid_namespace.x:zap_pid_ns_processes() has the baviour of
> blocking until all children of init have been reaped that you describe.
> 
> > When then
> > I type 'fg' to continue nsenter, the child gets collected and the
> > container successfully exits.
> >
> > This is with kernel 3.9-rc6 from Fedora.
> 
> For nsenter and the pid namespace they are working as designed.  But
> given this outcode it would be nice if we could get a SIGCONT when the
> child wakes up again.
I don't know how the kernel could know what is wanted. nsenter
signalled itself, and the kernel had nothing to with that.

> The current behavior supports being able to type suspend in your shell
> in the namespace and able to work outside the namespace.
> 
> I can't think of a way off the top of my head to wake nsenter up when
> it's child is woken up underneath it, but it sounds like that would be
> nice to do.
> 
> For the short term I would recommend not typing "reboot & exit" instead
> of "reboot" from a shell started with nsenter, and otherwise not leaving
> processes with parents outside the pid namespace around.
'reboot & exit' would suffer from the same problem, just with a race.
Even 'exec reboot' would, since the container shuts down quite quickly,
and the 'reboot' process could get SIGSTOPped before exiting.

> Of course that seding SIGSTOP before sending SIGTERM seems mighty fishy
> as well.
It's not entirely fishy, but I think that the implementation in
systemd might require some revisiting. systemd currently stops (and
resumes) all processes, even the ones which are exempt from killing.
But it's independent of this problem, since systemd does not exempt
the injected shell from killing.

Whether nsenter should be "fixed" depends on the main purpose of
nsenter.  If it's supposed to be used to launch arbitrary services,
then it might be changed, if comfortable use of a shell is more
important. I'll post a patch to remove the self-suspend, but I'm not
really sure if it should be applied. Probably not.

For systemd-nspawn, we'll grow our own facility to enter the
container, since we want to set the environment and find the container
by name and in general integrate with systemd-nspawn. So there's
little reason to modify nsenter for this purpose. 

Zbyszek

[not-for-applying PATCH] nsenter: do not self-suspend if child is suspended

[Zbigniew Jędrzejewski-Szmek]

nsenter would kill(self, SIGSTOP) when the child was stopped,
making it possible to use 'suspend -f' in the child shell to
suspend both the shell and the parent nsenter process. Unfortunately
this interferes with other uses of SIGSTOP, since nsenter process
is not visible in the child PID namespace and can be only resumed
from outside.
---
 sys-utils/nsenter.c | 13 ++-----------
 1 file changed, 2 insertions(+), 11 deletions(-)

diff --git a/sys-utils/nsenter.c b/sys-utils/nsenter.c
index 106349c..5cef4bf 100644
--- a/sys-utils/nsenter.c
+++ b/sys-utils/nsenter.c
@@ -129,7 +129,6 @@ static void continue_as_child(void)
 {
 	pid_t child = fork();
 	int status;
-	pid_t ret;
 
 	if (child < 0)
 		err(EXIT_FAILURE, _("fork failed"));
@@ -138,16 +137,8 @@ static void continue_as_child(void)
 	if (child == 0)
 		return;
 
-	for (;;) {
-		ret = waitpid(child, &status, WUNTRACED);
-		if ((ret == child) && (WIFSTOPPED(status))) {
-			/* The child suspended so suspend us as well */
-			kill(getpid(), SIGSTOP);
-			kill(child, SIGCONT);
-		} else {
-			break;
-		}
-	}
+	waitpid(child, &status, 0);
+
 	/* Return the child's exit code if possible */
 	if (WIFEXITED(status)) {
 		exit(WEXITSTATUS(status));
-- 
1.8.2.562.g931e949

Re: nsenter and SIGSTOP

[Eric W. Biederman]

Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:

> On Sat, Apr 20, 2013 at 03:27:46PM -0700, Eric W. Biederman wrote:
>> Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:
>> 
>> > Hi,
>> > I've hit a bit of a problem with nsenter and systemd-nspawn.
>> > When nsenter is used to enter the PID namespace created with
>> > systemd-nspawn, and the container's init attempts a shutdown,
>> > it hangs because nsenter is suspended.
>> >
>> > The sequence of events leading to the hang is:
>> >
>> > 1. nsenter launches a shell inside the container with
>> >    PPID=0 as seen inside the container,
>> > 2. systemd with PID=1 goes through the shutdown sequence,
>> >    issuing the equivalent(*) of
>> >
>> >    kill(-1, SIGSTOP)
>> 
>> This baffles me.  I am not certain why someone whould send SIGSTOP
>> when the want processes to exit.  I'm not even saying it's wrong just
>> saying that is odd.
> Like Lennart wrote, it's for atomicity of the subsequent killing.

When you don't do kill(-1, SIGTERM) that makes sense.
>
>> >    kill(-1, SIGTERM)
>> >    kill(_1, SIGCONT)
>> >    reboot(RB_HALT_SYSTEM)
>> >
>> > Now, nsenter has a stanza in continue_as_child where it stops itself
>> > when the child gets stopped. Unfortunately, this means that nsenter
>> > gets stopped in response to kill(-1, SIGSTOP) which hits the child.
>> > Then the child dies on kill(-1, SIGTERM), is resumed with kill(-1,
>> > SIGCONT) and exits (it prints "exit", so it's easy to see that it
>> > terminated properly. Then the shell becomes a zombie, since nsenter it
>> > it's parent and it's sleeping. Meanwhile, init executes reboot, and
>> > hangs in there, since the container waits for the PID namespace to
>> > become empty (I'm guessing here, but that seems logical).
>> 
>> I expect the hang is in the pid namespace init exiting.
>> in kernel/pid_namespace.x:zap_pid_ns_processes() has the baviour of
>> blocking until all children of init have been reaped that you describe.
>> 
>> > When then
>> > I type 'fg' to continue nsenter, the child gets collected and the
>> > container successfully exits.
>> >
>> > This is with kernel 3.9-rc6 from Fedora.
>> 
>> For nsenter and the pid namespace they are working as designed.  But
>> given this outcode it would be nice if we could get a SIGCONT when the
>> child wakes up again.
> I don't know how the kernel could know what is wanted. nsenter
> signalled itself, and the kernel had nothing to with that.

No.  However it is possible to get a notification when the child wakes
up, and even more when the child is killed (SIGCHLD).

The question is can those facilities be used without making the code
incomprehensible both to readers and to users of nsenter.

>> The current behavior supports being able to type suspend in your shell
>> in the namespace and able to work outside the namespace.
>> 
>> I can't think of a way off the top of my head to wake nsenter up when
>> it's child is woken up underneath it, but it sounds like that would be
>> nice to do.
>> 
>> For the short term I would recommend not typing "reboot & exit" instead
>> of "reboot" from a shell started with nsenter, and otherwise not leaving
>> processes with parents outside the pid namespace around.
> 'reboot & exit' would suffer from the same problem, just with a race.
> Even 'exec reboot' would, since the container shuts down quite quickly,
> and the 'reboot' process could get SIGSTOPped before exiting.

Well when this happens with ssh "reboot & exit" has a pretty good track
record of working.  'shutdown -r "now + 1 minute" &' might even be
better.

When you are interactive I don't imaginge going "doh!" and typing fg
is not going to be particularly hard either.

>> Of course that seding SIGSTOP before sending SIGTERM seems mighty fishy
>> as well.
> It's not entirely fishy, but I think that the implementation in
> systemd might require some revisiting. systemd currently stops (and
> resumes) all processes, even the ones which are exempt from killing.
> But it's independent of this problem, since systemd does not exempt
> the injected shell from killing.
>
> Whether nsenter should be "fixed" depends on the main purpose of
> nsenter.  If it's supposed to be used to launch arbitrary services,
> then it might be changed, if comfortable use of a shell is more
> important. I'll post a patch to remove the self-suspend, but I'm not
> really sure if it should be applied. Probably not.

Launching completely arbitrary services really isn't the main purposes.
Any time use setns to launch processes in a pid namespace the process
tree becomes multi-rooted which is not a good place to be.  So at the
very least everything you "launch" needs to be daemonized.

nsenter should remove the need to run sshd do in a container.

As I see it the main purpose of nsenter is to be a simple, easily
understood tool that let's you get inside of a container and do things.
It should be comfortable and useful to use as much as possible.

Which means it should be possible to use the "suspend" command in bash
possible.  There are folks for whom their workflow breaks when that
command does not work.

So I guess I am saying I would bias nsenter towards the interactive users
rather than scripted automation.

> For systemd-nspawn, we'll grow our own facility to enter the
> container, since we want to set the environment and find the container
> by name and in general integrate with systemd-nspawn. So there's
> little reason to modify nsenter for this purpose. 

Sounds reaasonable to me.  Just make certain multiple roots in the pid
namespace doing mess you up.

Eric

Re: [systemd-devel] nsenter and SIGSTOP

[Zbigniew Jędrzejewski-Szmek]

T24gU3VuLCBBcHIgMjEsIDIwMTMgYXQgMDk6MTg6MzRBTSAtMDcwMCwgRXJpYyBXLiBCaWVkZXJt
YW4gd3JvdGU6Cj4gWmJpZ25pZXcgSsSZZHJ6ZWpld3NraS1Tem1layA8emJ5c3pla0Bpbi53YXcu
cGw+IHdyaXRlczoKPiAKPiA+IE9uIFNhdCwgQXByIDIwLCAyMDEzIGF0IDAzOjI3OjQ2UE0gLTA3
MDAsIEVyaWMgVy4gQmllZGVybWFuIHdyb3RlOgo+ID4+IFpiaWduaWV3IErEmWRyemVqZXdza2kt
U3ptZWsgPHpieXN6ZWtAaW4ud2F3LnBsPiB3cml0ZXM6Cj4gPj4gCj4gPj4gPiBIaSwKPiA+PiA+
IEkndmUgaGl0IGEgYml0IG9mIGEgcHJvYmxlbSB3aXRoIG5zZW50ZXIgYW5kIHN5c3RlbWQtbnNw
YXduLgo+ID4+ID4gV2hlbiBuc2VudGVyIGlzIHVzZWQgdG8gZW50ZXIgdGhlIFBJRCBuYW1lc3Bh
Y2UgY3JlYXRlZCB3aXRoCj4gPj4gPiBzeXN0ZW1kLW5zcGF3biwgYW5kIHRoZSBjb250YWluZXIn
cyBpbml0IGF0dGVtcHRzIGEgc2h1dGRvd24sCj4gPj4gPiBpdCBoYW5ncyBiZWNhdXNlIG5zZW50
ZXIgaXMgc3VzcGVuZGVkLgo+ID4+ID4KPiA+PiA+IFRoZSBzZXF1ZW5jZSBvZiBldmVudHMgbGVh
ZGluZyB0byB0aGUgaGFuZyBpczoKPiA+PiA+Cj4gPj4gPiAxLiBuc2VudGVyIGxhdW5jaGVzIGEg
c2hlbGwgaW5zaWRlIHRoZSBjb250YWluZXIgd2l0aAo+ID4+ID4gICAgUFBJRD0wIGFzIHNlZW4g
aW5zaWRlIHRoZSBjb250YWluZXIsCj4gPj4gPiAyLiBzeXN0ZW1kIHdpdGggUElEPTEgZ29lcyB0
aHJvdWdoIHRoZSBzaHV0ZG93biBzZXF1ZW5jZSwKPiA+PiA+ICAgIGlzc3VpbmcgdGhlIGVxdWl2
YWxlbnQoKikgb2YKPiA+PiA+Cj4gPj4gPiAgICBraWxsKC0xLCBTSUdTVE9QKQo+ID4+IAo+ID4+
IFRoaXMgYmFmZmxlcyBtZS4gIEkgYW0gbm90IGNlcnRhaW4gd2h5IHNvbWVvbmUgd2hvdWxkIHNl
bmQgU0lHU1RPUAo+ID4+IHdoZW4gdGhlIHdhbnQgcHJvY2Vzc2VzIHRvIGV4aXQuICBJJ20gbm90
IGV2ZW4gc2F5aW5nIGl0J3Mgd3JvbmcganVzdAo+ID4+IHNheWluZyB0aGF0IGlzIG9kZC4KPiA+
IExpa2UgTGVubmFydCB3cm90ZSwgaXQncyBmb3IgYXRvbWljaXR5IG9mIHRoZSBzdWJzZXF1ZW50
IGtpbGxpbmcuCj4gCj4gV2hlbiB5b3UgZG9uJ3QgZG8ga2lsbCgtMSwgU0lHVEVSTSkgdGhhdCBt
YWtlcyBzZW5zZS4KQmVjYXVzZSBub3QgYWxsIHByb2Nlc3NlcyBhcmUga2lsbGVkOiBkdXJpbmcg
bm9ybWFsIHNodXRkb3duIHByb2Nlc3Nlcwp3aXRoIGFyZ3ZbMF0gYmVnaW5uaW5nIHdpdGggQCBh
cmUgc3BhcmVkCihodHRwOi8vd3d3LmZyZWVkZXNrdG9wLm9yZy93aWtpL1NvZnR3YXJlL3N5c3Rl
bWQvUm9vdFN0b3JhZ2VEYWVtb25zKS4KCj4gPj4gPiAgICBraWxsKC0xLCBTSUdURVJNKQo+ID4+
ID4gICAga2lsbChfMSwgU0lHQ09OVCkKPiA+PiA+ICAgIHJlYm9vdChSQl9IQUxUX1NZU1RFTSkK
PiA+PiA+Cj4gPj4gPiBOb3csIG5zZW50ZXIgaGFzIGEgc3RhbnphIGluIGNvbnRpbnVlX2FzX2No
aWxkIHdoZXJlIGl0IHN0b3BzIGl0c2VsZgo+ID4+ID4gd2hlbiB0aGUgY2hpbGQgZ2V0cyBzdG9w
cGVkLiBVbmZvcnR1bmF0ZWx5LCB0aGlzIG1lYW5zIHRoYXQgbnNlbnRlcgo+ID4+ID4gZ2V0cyBz
dG9wcGVkIGluIHJlc3BvbnNlIHRvIGtpbGwoLTEsIFNJR1NUT1ApIHdoaWNoIGhpdHMgdGhlIGNo
aWxkLgo+ID4+ID4gVGhlbiB0aGUgY2hpbGQgZGllcyBvbiBraWxsKC0xLCBTSUdURVJNKSwgaXMg
cmVzdW1lZCB3aXRoIGtpbGwoLTEsCj4gPj4gPiBTSUdDT05UKSBhbmQgZXhpdHMgKGl0IHByaW50
cyAiZXhpdCIsIHNvIGl0J3MgZWFzeSB0byBzZWUgdGhhdCBpdAo+ID4+ID4gdGVybWluYXRlZCBw
cm9wZXJseS4gVGhlbiB0aGUgc2hlbGwgYmVjb21lcyBhIHpvbWJpZSwgc2luY2UgbnNlbnRlciBp
dAo+ID4+ID4gaXQncyBwYXJlbnQgYW5kIGl0J3Mgc2xlZXBpbmcuIE1lYW53aGlsZSwgaW5pdCBl
eGVjdXRlcyByZWJvb3QsIGFuZAo+ID4+ID4gaGFuZ3MgaW4gdGhlcmUsIHNpbmNlIHRoZSBjb250
YWluZXIgd2FpdHMgZm9yIHRoZSBQSUQgbmFtZXNwYWNlIHRvCj4gPj4gPiBiZWNvbWUgZW1wdHkg
KEknbSBndWVzc2luZyBoZXJlLCBidXQgdGhhdCBzZWVtcyBsb2dpY2FsKS4KPiA+PiAKPiA+PiBJ
IGV4cGVjdCB0aGUgaGFuZyBpcyBpbiB0aGUgcGlkIG5hbWVzcGFjZSBpbml0IGV4aXRpbmcuCj4g
Pj4gaW4ga2VybmVsL3BpZF9uYW1lc3BhY2UueDp6YXBfcGlkX25zX3Byb2Nlc3NlcygpIGhhcyB0
aGUgYmF2aW91ciBvZgo+ID4+IGJsb2NraW5nIHVudGlsIGFsbCBjaGlsZHJlbiBvZiBpbml0IGhh
dmUgYmVlbiByZWFwZWQgdGhhdCB5b3UgZGVzY3JpYmUuCj4gPj4gCj4gPj4gPiBXaGVuIHRoZW4K
PiA+PiA+IEkgdHlwZSAnZmcnIHRvIGNvbnRpbnVlIG5zZW50ZXIsIHRoZSBjaGlsZCBnZXRzIGNv
bGxlY3RlZCBhbmQgdGhlCj4gPj4gPiBjb250YWluZXIgc3VjY2Vzc2Z1bGx5IGV4aXRzLgo+ID4+
ID4KPiA+PiA+IFRoaXMgaXMgd2l0aCBrZXJuZWwgMy45LXJjNiBmcm9tIEZlZG9yYS4KPiA+PiAK
PiA+PiBGb3IgbnNlbnRlciBhbmQgdGhlIHBpZCBuYW1lc3BhY2UgdGhleSBhcmUgd29ya2luZyBh
cyBkZXNpZ25lZC4gIEJ1dAo+ID4+IGdpdmVuIHRoaXMgb3V0Y29kZSBpdCB3b3VsZCBiZSBuaWNl
IGlmIHdlIGNvdWxkIGdldCBhIFNJR0NPTlQgd2hlbiB0aGUKPiA+PiBjaGlsZCB3YWtlcyB1cCBh
Z2Fpbi4KPiA+IEkgZG9uJ3Qga25vdyBob3cgdGhlIGtlcm5lbCBjb3VsZCBrbm93IHdoYXQgaXMg
d2FudGVkLiBuc2VudGVyCj4gPiBzaWduYWxsZWQgaXRzZWxmLCBhbmQgdGhlIGtlcm5lbCBoYWQg
bm90aGluZyB0byB3aXRoIHRoYXQuCj4gCj4gTm8uICBIb3dldmVyIGl0IGlzIHBvc3NpYmxlIHRv
IGdldCBhIG5vdGlmaWNhdGlvbiB3aGVuIHRoZSBjaGlsZCB3YWtlcwo+IHVwLCBhbmQgZXZlbiBt
b3JlIHdoZW4gdGhlIGNoaWxkIGlzIGtpbGxlZCAoU0lHQ0hMRCkuClJpZ2h0LCBidXQgdGhhdCBk
b2Vzbid0IGhlbHAgYXQgYWxsLCBzaW5jZSBuc2VudGVyIGlzIHNsZWVwaW5nLiBJdCdsbApnZXQg
dGhlIG5vdGlmaWNhdGlvbiwgd2hlbiBpdCB3YWtlcyB1cCwgYnV0IHRoZXJlJ3Mgbm90aGluZyB0
byB3YWtlIGl0CnVwLgoKPiBUaGUgcXVlc3Rpb24gaXMgY2FuIHRob3NlIGZhY2lsaXRpZXMgYmUg
dXNlZCB3aXRob3V0IG1ha2luZyB0aGUgY29kZQo+IGluY29tcHJlaGVuc2libGUgYm90aCB0byBy
ZWFkZXJzIGFuZCB0byB1c2VycyBvZiBuc2VudGVyLgo+IAo+ID4+IFRoZSBjdXJyZW50IGJlaGF2
aW9yIHN1cHBvcnRzIGJlaW5nIGFibGUgdG8gdHlwZSBzdXNwZW5kIGluIHlvdXIgc2hlbGwKPiA+
PiBpbiB0aGUgbmFtZXNwYWNlIGFuZCBhYmxlIHRvIHdvcmsgb3V0c2lkZSB0aGUgbmFtZXNwYWNl
Lgo+ID4+IAo+ID4+IEkgY2FuJ3QgdGhpbmsgb2YgYSB3YXkgb2ZmIHRoZSB0b3Agb2YgbXkgaGVh
ZCB0byB3YWtlIG5zZW50ZXIgdXAgd2hlbgo+ID4+IGl0J3MgY2hpbGQgaXMgd29rZW4gdXAgdW5k
ZXJuZWF0aCBpdCwgYnV0IGl0IHNvdW5kcyBsaWtlIHRoYXQgd291bGQgYmUKPiA+PiBuaWNlIHRv
IGRvLgo+ID4+IAo+ID4+IEZvciB0aGUgc2hvcnQgdGVybSBJIHdvdWxkIHJlY29tbWVuZCBub3Qg
dHlwaW5nICJyZWJvb3QgJiBleGl0IiBpbnN0ZWFkCj4gPj4gb2YgInJlYm9vdCIgZnJvbSBhIHNo
ZWxsIHN0YXJ0ZWQgd2l0aCBuc2VudGVyLCBhbmQgb3RoZXJ3aXNlIG5vdCBsZWF2aW5nCj4gPj4g
cHJvY2Vzc2VzIHdpdGggcGFyZW50cyBvdXRzaWRlIHRoZSBwaWQgbmFtZXNwYWNlIGFyb3VuZC4K
PiA+ICdyZWJvb3QgJiBleGl0JyB3b3VsZCBzdWZmZXIgZnJvbSB0aGUgc2FtZSBwcm9ibGVtLCBq
dXN0IHdpdGggYSByYWNlLgo+ID4gRXZlbiAnZXhlYyByZWJvb3QnIHdvdWxkLCBzaW5jZSB0aGUg
Y29udGFpbmVyIHNodXRzIGRvd24gcXVpdGUgcXVpY2tseSwKPiA+IGFuZCB0aGUgJ3JlYm9vdCcg
cHJvY2VzcyBjb3VsZCBnZXQgU0lHU1RPUHBlZCBiZWZvcmUgZXhpdGluZy4KPiAKPiBXZWxsIHdo
ZW4gdGhpcyBoYXBwZW5zIHdpdGggc3NoICJyZWJvb3QgJiBleGl0IiBoYXMgYSBwcmV0dHkgZ29v
ZCB0cmFjawo+IHJlY29yZCBvZiB3b3JraW5nLiAgJ3NodXRkb3duIC1yICJub3cgKyAxIG1pbnV0
ZSIgJicgbWlnaHQgZXZlbiBiZQo+IGJldHRlci4KPiAKPiBXaGVuIHlvdSBhcmUgaW50ZXJhY3Rp
dmUgSSBkb24ndCBpbWFnaW5nZSBnb2luZyAiZG9oISIgYW5kIHR5cGluZyBmZwo+IGlzIG5vdCBn
b2luZyB0byBiZSBwYXJ0aWN1bGFybHkgaGFyZCBlaXRoZXIuCldlJ3JlIHRyeWluZyB0byBnZXQg
dGhpbmdzIHRvIHdvcmsgd2l0aG91dCBrbHVkZ2VzIGxpa2Ugc2xlZXBpbmcgb3IgCm1hbnVhbCBw
cm9kZGluZy4gRm9yIGRlYnVnZ2luZyB0aGF0J3MgZmluZSwgYnV0IHBlb3BsZSB1c2Ugc3lzdGVt
ZC1uc3Bhd24KY29udGFpbmVycyBmb3Igc2VydmljZXMsIGFuZCBleHBlY3QgdGhlbSB0byAianVz
dCB3b3JrIi4KCj4gPj4gT2YgY291cnNlIHRoYXQgc2VkaW5nIFNJR1NUT1AgYmVmb3JlIHNlbmRp
bmcgU0lHVEVSTSBzZWVtcyBtaWdodHkgZmlzaHkKPiA+PiBhcyB3ZWxsLgo+ID4gSXQncyBub3Qg
ZW50aXJlbHkgZmlzaHksIGJ1dCBJIHRoaW5rIHRoYXQgdGhlIGltcGxlbWVudGF0aW9uIGluCj4g
PiBzeXN0ZW1kIG1pZ2h0IHJlcXVpcmUgc29tZSByZXZpc2l0aW5nLiBzeXN0ZW1kIGN1cnJlbnRs
eSBzdG9wcyAoYW5kCj4gPiByZXN1bWVzKSBhbGwgcHJvY2Vzc2VzLCBldmVuIHRoZSBvbmVzIHdo
aWNoIGFyZSBleGVtcHQgZnJvbSBraWxsaW5nLgo+ID4gQnV0IGl0J3MgaW5kZXBlbmRlbnQgb2Yg
dGhpcyBwcm9ibGVtLCBzaW5jZSBzeXN0ZW1kIGRvZXMgbm90IGV4ZW1wdAo+ID4gdGhlIGluamVj
dGVkIHNoZWxsIGZyb20ga2lsbGluZy4KPiA+Cj4gPiBXaGV0aGVyIG5zZW50ZXIgc2hvdWxkIGJl
ICJmaXhlZCIgZGVwZW5kcyBvbiB0aGUgbWFpbiBwdXJwb3NlIG9mCj4gPiBuc2VudGVyLiAgSWYg
aXQncyBzdXBwb3NlZCB0byBiZSB1c2VkIHRvIGxhdW5jaCBhcmJpdHJhcnkgc2VydmljZXMsCj4g
PiB0aGVuIGl0IG1pZ2h0IGJlIGNoYW5nZWQsIGlmIGNvbWZvcnRhYmxlIHVzZSBvZiBhIHNoZWxs
IGlzIG1vcmUKPiA+IGltcG9ydGFudC4gSSdsbCBwb3N0IGEgcGF0Y2ggdG8gcmVtb3ZlIHRoZSBz
ZWxmLXN1c3BlbmQsIGJ1dCBJJ20gbm90Cj4gPiByZWFsbHkgc3VyZSBpZiBpdCBzaG91bGQgYmUg
YXBwbGllZC4gUHJvYmFibHkgbm90Lgo+IAo+IExhdW5jaGluZyBjb21wbGV0ZWx5IGFyYml0cmFy
eSBzZXJ2aWNlcyByZWFsbHkgaXNuJ3QgdGhlIG1haW4gcHVycG9zZXMuCj4gQW55IHRpbWUgdXNl
IHNldG5zIHRvIGxhdW5jaCBwcm9jZXNzZXMgaW4gYSBwaWQgbmFtZXNwYWNlIHRoZSBwcm9jZXNz
Cj4gdHJlZSBiZWNvbWVzIG11bHRpLXJvb3RlZCB3aGljaCBpcyBub3QgYSBnb29kIHBsYWNlIHRv
IGJlLiAgU28gYXQgdGhlCj4gdmVyeSBsZWFzdCBldmVyeXRoaW5nIHlvdSAibGF1bmNoIiBuZWVk
cyB0byBiZSBkYWVtb25pemVkLgo+IAo+IG5zZW50ZXIgc2hvdWxkIHJlbW92ZSB0aGUgbmVlZCB0
byBydW4gc3NoZCBkbyBpbiBhIGNvbnRhaW5lci4KPiAKPiBBcyBJIHNlZSBpdCB0aGUgbWFpbiBw
dXJwb3NlIG9mIG5zZW50ZXIgaXMgdG8gYmUgYSBzaW1wbGUsIGVhc2lseQo+IHVuZGVyc3Rvb2Qg
dG9vbCB0aGF0IGxldCdzIHlvdSBnZXQgaW5zaWRlIG9mIGEgY29udGFpbmVyIGFuZCBkbyB0aGlu
Z3MuCj4gSXQgc2hvdWxkIGJlIGNvbWZvcnRhYmxlIGFuZCB1c2VmdWwgdG8gdXNlIGFzIG11Y2gg
YXMgcG9zc2libGUuCj4gCj4gV2hpY2ggbWVhbnMgaXQgc2hvdWxkIGJlIHBvc3NpYmxlIHRvIHVz
ZSB0aGUgInN1c3BlbmQiIGNvbW1hbmQgaW4gYmFzaAo+IHBvc3NpYmxlLiAgVGhlcmUgYXJlIGZv
bGtzIGZvciB3aG9tIHRoZWlyIHdvcmtmbG93IGJyZWFrcyB3aGVuIHRoYXQKPiBjb21tYW5kIGRv
ZXMgbm90IHdvcmsuCj4gCj4gU28gSSBndWVzcyBJIGFtIHNheWluZyBJIHdvdWxkIGJpYXMgbnNl
bnRlciB0b3dhcmRzIHRoZSBpbnRlcmFjdGl2ZSB1c2Vycwo+IHJhdGhlciB0aGFuIHNjcmlwdGVk
IGF1dG9tYXRpb24uCkFncmVlZC4KCj4gPiBGb3Igc3lzdGVtZC1uc3Bhd24sIHdlJ2xsIGdyb3cg
b3VyIG93biBmYWNpbGl0eSB0byBlbnRlciB0aGUKPiA+IGNvbnRhaW5lciwgc2luY2Ugd2Ugd2Fu
dCB0byBzZXQgdGhlIGVudmlyb25tZW50IGFuZCBmaW5kIHRoZSBjb250YWluZXIKPiA+IGJ5IG5h
bWUgYW5kIGluIGdlbmVyYWwgaW50ZWdyYXRlIHdpdGggc3lzdGVtZC1uc3Bhd24uIFNvIHRoZXJl
J3MKPiA+IGxpdHRsZSByZWFzb24gdG8gbW9kaWZ5IG5zZW50ZXIgZm9yIHRoaXMgcHVycG9zZS4g
Cj4gCj4gU291bmRzIHJlYWFzb25hYmxlIHRvIG1lLiAgSnVzdCBtYWtlIGNlcnRhaW4gbXVsdGlw
bGUgcm9vdHMgaW4gdGhlIHBpZAo+IG5hbWVzcGFjZSBkb2luZyBtZXNzIHlvdSB1cC4KWWVhaCwg
bXVsdGlwbGUgcm9vdHMgd2l0aCB1bmtpbGxhYmxlIHpvbWJpZSBwcm9jZXNzZXMgc3VyZWx5IGFy
ZSBlbm91Z2gKdG8gbWFrZSBwZW9wbGUgY29uZnVzZWQuIEknbSBzdGlsbCB0cnlpbmcgdG8gd3Jh
cCBteSBoZWFkIGFyb3VuZCBQSUQKYW5kIG1vdW50IG5hbWVzcGFjZXMsIGFuZCBJIGtub3cgdGhh
dCB1c2VyIG5hbWVzcGFjZXMgYWRkIGFub3RoZXIgbGV2ZWwKb2YgZnVuIDopLgoKWmJ5c3plawpf
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpzeXN0ZW1kLWRl
dmVsIG1haWxpbmcgbGlzdApzeXN0ZW1kLWRldmVsQGxpc3RzLmZyZWVkZXNrdG9wLm9yZwpodHRw
Oi8vbGlzdHMuZnJlZWRlc2t0b3Aub3JnL21haWxtYW4vbGlzdGluZm8vc3lzdGVtZC1kZXZlbAo=

Re: nsenter and SIGSTOP

[Eric W. Biederman]

Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:

> On Sun, Apr 21, 2013 at 09:18:34AM -0700, Eric W. Biederman wrote:
>> Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:
>> 
>> > On Sat, Apr 20, 2013 at 03:27:46PM -0700, Eric W. Biederman wrote:
>> >> Zbigniew Jędrzejewski-Szmek <zbyszek@in.waw.pl> writes:
>> >> 
>> >> > Hi,
>> >> > I've hit a bit of a problem with nsenter and systemd-nspawn.
>> >> > When nsenter is used to enter the PID namespace created with
>> >> > systemd-nspawn, and the container's init attempts a shutdown,
>> >> > it hangs because nsenter is suspended.
>> >> >
>> >> > The sequence of events leading to the hang is:
>> >> >
>> >> > 1. nsenter launches a shell inside the container with
>> >> >    PPID=0 as seen inside the container,
>> >> > 2. systemd with PID=1 goes through the shutdown sequence,
>> >> >    issuing the equivalent(*) of
>> >> >
>> >> >    kill(-1, SIGSTOP)
>> >> 
>> >> This baffles me.  I am not certain why someone whould send SIGSTOP
>> >> when the want processes to exit.  I'm not even saying it's wrong just
>> >> saying that is odd.
>> > Like Lennart wrote, it's for atomicity of the subsequent killing.
>> 
>> When you don't do kill(-1, SIGTERM) that makes sense.
> Because not all processes are killed: during normal shutdown processes
> with argv[0] beginning with @ are spared
> (http://www.freedesktop.org/wiki/Software/systemd/RootStorageDaemons).

I wasn't arguing.  But since you bring up root storage daemons and
initial ram disks, I don't believe any of those actually apply to a
container.

My point was only that using SIGSTOP makes sense when it becomes clear
that kill(-1, SIGTERM) is not happening.

>> No.  However it is possible to get a notification when the child wakes
>> up, and even more when the child is killed (SIGCHLD).
> Right, but that doesn't help at all, since nsenter is sleeping. It'll
> get the notification, when it wakes up, but there's nothing to wake it
> up.

I believe you could make it all work with a poll based on a periodic
wake up.  If you don't send yourself SIGSTOP I don't believe the outer
shell will recognize what has happened, but you should be able to wake
yourself up with a timer and see verify you child is still stopped.

>> Well when this happens with ssh "reboot & exit" has a pretty good track
>> record of working.  'shutdown -r "now + 1 minute" &' might even be
>> better.
>> 
>> When you are interactive I don't imaginge going "doh!" and typing fg
>> is not going to be particularly hard either.
> We're trying to get things to work without kludges like sleeping or 
> manual prodding. For debugging that's fine, but people use systemd-nspawn
> containers for services, and expect them to "just work".

Like I said below nsenter is about interactive/debugging case.  More
sysadmin debugging than application debugging but I guess that makes
it debugging.

Essentially that was the point of implementing setns for the pid
namespace.  So you exceptional cases could be handled without having to
got to the expense of having to depend on a functioning login daemon in
the container.

>> So I guess I am saying I would bias nsenter towards the interactive users
>> rather than scripted automation.
> Agreed.
>
>> > For systemd-nspawn, we'll grow our own facility to enter the
>> > container, since we want to set the environment and find the container
>> > by name and in general integrate with systemd-nspawn. So there's
>> > little reason to modify nsenter for this purpose. 
>> 
>> Sounds reaasonable to me.  Just make certain multiple roots in the pid
>> namespace doing mess you up.
> Yeah, multiple roots with unkillable zombie processes surely are enough
> to make people confused. I'm still trying to wrap my head around PID
> and mount namespaces, and I know that user namespaces add another level
> of fun :).

Well this isn't a case of unkillable zombies.  This is a case of
processes not being reaped.  And a weird process that doesn't fully die
until all of it's children are dead (similar to the leader of a thread
group).  Now honestly I would not be adverse in theory to handling this
weird corrner case better in zap_pid_ns_processes.  Right now
zap_pid_ns_processes is the best so far.

For launching new services in a container simply sending a message to
the init process is probably what you want.  I think those messages
already traverse unix domain sockets so it insn't too shabby.

Eric