restart (mktree) program usage

restart (mktree) program usage

[Sukadev Bhattiprolu]

I have a usage question on the 'restart' (formerly mktree) program.

In the following container c/r case: 

	- create a container
	- log in to the container,
	- restore filesystem(s) from snapshot
	- restart application from checkpoint

On restart, suppose the user wants to restore the original pids.
But he does not want to create a new pid-ns, (since he just created
the container, and is sure the original pids are available).

To accomplish this the user has to specify the arguments in the following
order right  (since -pids implies --pidns).

	restart --pids --no-pidns

IOW, the order of the arguments matters.

Would it be easier to understand if --pids did not imply --pidns ?

(Or in fact the reverse seems to make more sense -i.e --pidns implies
--pids, with a new option, --no-pids if user absolutely hates the pids
he was dealt before checkpoint :-)

	$ restart
		don't create pid ns, don't try to restore pids

	$ restart --pids
		try to restore original pids, don't create pid-ns

	$ restart --pidns
		create new pid-namespace and restore original pids

	$ restart --pidns --no-pids
	$ restart --no-pids --pidns
		create new pid-namespace, DO NOT restore pids

Or maybe drop implying and let user explicitly specify one, none or
both of --pids, --pidns ?

Sukadev

Re: restart (mktree) program usage

[Oren Laadan]

Sukadev Bhattiprolu wrote:
> I have a usage question on the 'restart' (formerly mktree) program.
> 
> In the following container c/r case: 
> 
> 	- create a container
> 	- log in to the container,
> 	- restore filesystem(s) from snapshot
> 	- restart application from checkpoint

FWIW, I'd expect that future versions of 'restart' will be capable
of doing this entire setup, (filesystem(s) included), as it matures.

Note that this use case that you suggest will only work to restart
subtrees; it is unsuitable for full containers (with pids) because
the pid of init (1) will already be in use.

Perhaps we should think of some "plugin" architecture for 'restart'
that will allow the user to ask it to execute some work at between
creating a new container and actually restarting into it ?

> 
> On restart, suppose the user wants to restore the original pids.
> But he does not want to create a new pid-ns, (since he just created
> the container, and is sure the original pids are available).
> 
> To accomplish this the user has to specify the arguments in the following
> order right  (since -pids implies --pidns).
> 
> 	restart --pids --no-pidns
> 
> IOW, the order of the arguments matters.
> 
> Would it be easier to understand if --pids did not imply --pidns ?

Either that, or simply change the semantics such that the order
does not matter:

> 
> (Or in fact the reverse seems to make more sense -i.e --pidns implies
> --pids, with a new option, --no-pids if user absolutely hates the pids
> he was dealt before checkpoint :-)
> 
> 	$ restart
> 		don't create pid ns, don't try to restore pids
> 
> 	$ restart --pids
> 		try to restore original pids, don't create pid-ns
> 
> 	$ restart --pidns
> 		create new pid-namespace and restore original pids
> 
> 	$ restart --pidns --no-pids
> 	$ restart --no-pids --pidns
> 		create new pid-namespace, DO NOT restore pids

Yes, something like that.

Oren.

> 
> Or maybe drop implying and let user explicitly specify one, none or
> both of --pids, --pidns ?
> 
> Sukadev
> _______________________________________________
> Containers mailing list
> Containers-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA@public.gmane.org
> https://lists.linux-foundation.org/mailman/listinfo/containers

Re: restart (mktree) program usage

[Sukadev Bhattiprolu]

Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
| 
| 
| Sukadev Bhattiprolu wrote:
| > I have a usage question on the 'restart' (formerly mktree) program.
| > 
| > In the following container c/r case: 
| > 
| > 	- create a container
| > 	- log in to the container,
| > 	- restore filesystem(s) from snapshot
| > 	- restart application from checkpoint
| 
| FWIW, I'd expect that future versions of 'restart' will be capable
| of doing this entire setup, (filesystem(s) included), as it matures.
| 
| Note that this use case that you suggest will only work to restart
| subtrees; it is unsuitable for full containers (with pids) because
| the pid of init (1) will already be in use.

True. But if originally the application was started as:

	Create container
	Login to contaienr
	Set up filesystem
	Start application

The application would not be using the pid 1 right - even if the
application was started from an rc script in the container ?

| 
| Perhaps we should think of some "plugin" architecture for 'restart'
| that will allow the user to ask it to execute some work at between
| creating a new container and actually restarting into it ?

Yes, that would be really useful I think for things like restoring file
system to its snapshot. Without that there is somewhat of an assymetry
in starting an application in a container and restarting it from a
checkpoint.

Re: restart (mktree) program usage

[Oren Laadan]

Sukadev Bhattiprolu wrote:
> Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
> | 
> | 
> | Sukadev Bhattiprolu wrote:
> | > I have a usage question on the 'restart' (formerly mktree) program.
> | > 
> | > In the following container c/r case: 
> | > 
> | > 	- create a container
> | > 	- log in to the container,
> | > 	- restore filesystem(s) from snapshot
> | > 	- restart application from checkpoint
> | 
> | FWIW, I'd expect that future versions of 'restart' will be capable
> | of doing this entire setup, (filesystem(s) included), as it matures.
> | 
> | Note that this use case that you suggest will only work to restart
> | subtrees; it is unsuitable for full containers (with pids) because
> | the pid of init (1) will already be in use.
> 
> True. But if originally the application was started as:
> 
> 	Create container
> 	Login to contaienr

Actually, I'm not sure what you mean by "login to container" ?

> 	Set up filesystem
> 	Start application
> 

[One way to solve this is - after the file systems are setup, to
start a container inside that container :p ... or not].

Ok, let's assume that the application was started inside an existing
container to begin with, but was checkpointed as a whole-container --
with the ancestor(s) processes there -  then we basically have a
checkpoint image that holds more processes than we really care about.

So what ?  The simple fact is that the checkpoint image contains a
task with pid 1. So it won't restart unless in a new container.

I'd suggest checkpointing only a subtree, but that may not be an
option, e.g.  if the application already has some orphan processes -
unreachable under subtree !.

Or, I'd suggest to use a userspace tool to chop data away from the
checkpoint image (e.g. remove processes ...). But if you remove the
init process, you again face a problem with the orphans.

<warning> Here's a crazy idea: </warning>

Userspace maniuplation of the checkpoint image is a powerful tool.
I can imagine, for instance, a flag RESTART_I_AM_FINE_THANK_YOU with
which a process tells the restart to to let it be  (like the current
ghost tasks, but without exiting).

How is that useful ?  combine that with some checkpoint image tweaks,
and you can drop the init(1) task from the checkpoint image, and have
the real init task in the new container participate in the restart
without really restoring its state.... voila.

In fact, just like the proposed cradvise() would be able to tell the
kernel to use a given resource instead of recreating it from the
image, such a flag could tell the kernel to do so for processes.

Ok... a bit carried away. But maybe someone will find this idea not
only cool, but also useful :)

[Or, you start a new container, setup file systems, and then restart
into a new - nested - container :p ...]

Oren.

> The application would not be using the pid 1 right - even if the
> application was started from an rc script in the container ?
> 
> | 
> | Perhaps we should think of some "plugin" architecture for 'restart'
> | that will allow the user to ask it to execute some work at between
> | creating a new container and actually restarting into it ?
> 
> Yes, that would be really useful I think for things like restoring file
> system to its snapshot. Without that there is somewhat of an assymetry
> in starting an application in a container and restarting it from a
> checkpoint.

Re: restart (mktree) program usage

[Serge E. Hallyn]

Quoting Oren Laadan (orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org):
> 
> 
> Sukadev Bhattiprolu wrote:
> > Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
> > | 
> > | 
> > | Sukadev Bhattiprolu wrote:
> > | > I have a usage question on the 'restart' (formerly mktree) program.
> > | > 
> > | > In the following container c/r case: 
> > | > 
> > | > 	- create a container
> > | > 	- log in to the container,
> > | > 	- restore filesystem(s) from snapshot
> > | > 	- restart application from checkpoint
> > | 
> > | FWIW, I'd expect that future versions of 'restart' will be capable
> > | of doing this entire setup, (filesystem(s) included), as it matures.
> > | 
> > | Note that this use case that you suggest will only work to restart
> > | subtrees; it is unsuitable for full containers (with pids) because
> > | the pid of init (1) will already be in use.
> > 
> > True. But if originally the application was started as:
> > 
> > 	Create container
> > 	Login to contaienr
> 
> Actually, I'm not sure what you mean by "login to container" ?

I assume he was thinking of a system container created with liblxc
or libvirt, and literally logging in on its console or over ssh.

...

> > Yes, that would be really useful I think for things like restoring file
> > system to its snapshot. Without that there is somewhat of an assymetry
> > in starting an application in a container and restarting it from a
> > checkpoint.

Fine, they are different operations :)

-serge

Re: restart (mktree) program usage

[Sukadev Bhattiprolu]

Serge E. Hallyn [serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org] wrote:
| > > True. But if originally the application was started as:
| > > 
| > > 	Create container
| > > 	Login to contaienr
| > 
| > Actually, I'm not sure what you mean by "login to container" ?
| 
| I assume he was thinking of a system container created with liblxc
| or libvirt, and literally logging in on its console or over ssh.

Yes, but...
| 
| ...
| 
| > > Yes, that would be really useful I think for things like restoring file
| > > system to its snapshot. Without that there is somewhat of an assymetry
| > > in starting an application in a container and restarting it from a
| > > checkpoint.
| 
| Fine, they are different operations :)

...more than the assymetry, I guess I am not sure what is the "recommended"
way to restart an application, while ensuring filesystem (and network)
are correctly restored by user.

	ns_exec --cpuim -- /usr/bin/restart --wait < checkpoint-image.1

assumes fs/network are setup. Hence the "login" to the container.


| 
| -serge

Re: restart (mktree) program usage

[Serge E. Hallyn]

Quoting Sukadev Bhattiprolu (sukadev-23VcF4HTsmIX0ybBhKVfKdBPR1lH4CV8@public.gmane.org):
> Serge E. Hallyn [serue-r/Jw6+rmf7HQT0dZR+AlfA@public.gmane.org] wrote:
> | > > True. But if originally the application was started as:
> | > > 
> | > > 	Create container
> | > > 	Login to contaienr
> | > 
> | > Actually, I'm not sure what you mean by "login to container" ?
> | 
> | I assume he was thinking of a system container created with liblxc
> | or libvirt, and literally logging in on its console or over ssh.
> 
> Yes, but...
> | 
> | ...
> | 
> | > > Yes, that would be really useful I think for things like restoring file
> | > > system to its snapshot. Without that there is somewhat of an assymetry
> | > > in starting an application in a container and restarting it from a
> | > > checkpoint.
> | 
> | Fine, they are different operations :)
> 
> ...more than the assymetry, I guess I am not sure what is the "recommended"
> way to restart an application, while ensuring filesystem (and network)
> are correctly restored by user.
> 
> 	ns_exec --cpuim -- /usr/bin/restart --wait < checkpoint-image.1
> 
> assumes fs/network are setup. Hence the "login" to the container.

Right, this is stuff to discuss at the plumbers talk and BOF.  How
much of this do we want /usr/bin/restart to handle?  It already does
pidns, and IMO having it handle everything needed makes sense.  The
only problem is we can't have it handle filesystem yet because we
haven't yet decided where/how to checkpont the fs :)  I don't even
mean mounts, I mean a btrfs snapshot or rsync.  Do we want to create
a 'container_checkpoint' program which wraps 'checkpoint' and does
filesystem and network interface snapshotting?

I'm not asking for an answer in this thread, but for it to be discussed
at the bof :)

-serge

Re: restart (mktree) program usage

[Oren Laadan]

Serge E. Hallyn wrote:
> Quoting Oren Laadan (orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org):
>>
>> Sukadev Bhattiprolu wrote:
>>> Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
>>> | 
>>> | 
>>> | Sukadev Bhattiprolu wrote:
>>> | > I have a usage question on the 'restart' (formerly mktree) program.
>>> | > 
>>> | > In the following container c/r case: 
>>> | > 
>>> | > 	- create a container
>>> | > 	- log in to the container,
>>> | > 	- restore filesystem(s) from snapshot
>>> | > 	- restart application from checkpoint
>>> | 
>>> | FWIW, I'd expect that future versions of 'restart' will be capable
>>> | of doing this entire setup, (filesystem(s) included), as it matures.
>>> | 
>>> | Note that this use case that you suggest will only work to restart
>>> | subtrees; it is unsuitable for full containers (with pids) because
>>> | the pid of init (1) will already be in use.
>>>
>>> True. But if originally the application was started as:
>>>
>>> 	Create container
>>> 	Login to contaienr
>> Actually, I'm not sure what you mean by "login to container" ?
> 
> I assume he was thinking of a system container created with liblxc
> or libvirt, and literally logging in on its console or over ssh.

This is exactly my point:

* If you checkpoint full container, you get the sshd (and init) as
well, so you can't restore into an "existing" container, but create
a new one.

* If you checkpoint subtree, you will miss orphans, and you will
give up leak-detection.

I'd assume most users of this scenarios will prefer full container.

If you want to restart by first creating a new container, than set
it up, then log-in to that container, and finally run restart -
you will _have to_ modify the checkpoint image to remove unwanted
parts, and adjust the remaining accordingly.

For instance, you will have to remove (or tell the kernel to skip)
the data for the init task, and the ssh-server, and then change the
ppid of the root of your subtree to be 1 instead of what it was
(probably the old ssh-server), and so on.

It's doable, but I'm unsure if it's worth the extra work.

Oren.

> 
>>> Yes, that would be really useful I think for things like restoring file
>>> system to its snapshot. Without that there is somewhat of an assymetry
>>> in starting an application in a container and restarting it from a
>>> checkpoint.
> 
> Fine, they are different operations :)
> 
> -serge

Re: restart (mktree) program usage

[Serge E. Hallyn]

Quoting Oren Laadan (orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org):
> 
> 
> Serge E. Hallyn wrote:
> > Quoting Oren Laadan (orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org):
> >>
> >> Sukadev Bhattiprolu wrote:
> >>> Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
> >>> | 
> >>> | 
> >>> | Sukadev Bhattiprolu wrote:
> >>> | > I have a usage question on the 'restart' (formerly mktree) program.
> >>> | > 
> >>> | > In the following container c/r case: 
> >>> | > 
> >>> | > 	- create a container
> >>> | > 	- log in to the container,
> >>> | > 	- restore filesystem(s) from snapshot
> >>> | > 	- restart application from checkpoint
> >>> | 
> >>> | FWIW, I'd expect that future versions of 'restart' will be capable
> >>> | of doing this entire setup, (filesystem(s) included), as it matures.
> >>> | 
> >>> | Note that this use case that you suggest will only work to restart
> >>> | subtrees; it is unsuitable for full containers (with pids) because
> >>> | the pid of init (1) will already be in use.
> >>>
> >>> True. But if originally the application was started as:
> >>>
> >>> 	Create container
> >>> 	Login to contaienr
> >> Actually, I'm not sure what you mean by "login to container" ?
> > 
> > I assume he was thinking of a system container created with liblxc
> > or libvirt, and literally logging in on its console or over ssh.
> 
> This is exactly my point:
> 
> * If you checkpoint full container, you get the sshd (and init) as
> well, so you can't restore into an "existing" container, but create
> a new one.
> 
> * If you checkpoint subtree, you will miss orphans, and you will
> give up leak-detection.
> 
> I'd assume most users of this scenarios will prefer full container.

Yup, I think we are agreed.

-serge

Re: restart (mktree) program usage

[Cedric Le Goater]

On 09/17/2009 03:35 AM, Sukadev Bhattiprolu wrote:
> Oren Laadan [orenl-RdfvBDnrOixBDgjK7y7TUQ@public.gmane.org] wrote:
> | 
> | 
> | Sukadev Bhattiprolu wrote:
> | > I have a usage question on the 'restart' (formerly mktree) program.
> | > 
> | > In the following container c/r case: 
> | > 
> | > 	- create a container
> | > 	- log in to the container,
> | > 	- restore filesystem(s) from snapshot
> | > 	- restart application from checkpoint
> | 
> | FWIW, I'd expect that future versions of 'restart' will be capable
> | of doing this entire setup, (filesystem(s) included), as it matures.
> | 
> | Note that this use case that you suggest will only work to restart
> | subtrees; it is unsuitable for full containers (with pids) because
> | the pid of init (1) will already be in use.
> 
> True. But if originally the application was started as:
> 
> 	Create container
> 	Login to contaienr
> 	Set up filesystem
> 	Start application

You should take a look at lxc, which is doing what you want.

Metacluster (IBM's proprietary C/R solution) is fully integrated with lxc 
containers and the current lxc development plans focus on externalizing 
some home made C/R plugins we've been doing for it. We should then spend 
some time on blcr.

And we expect you guys to work on oren's patchset ? but I guess that will 
all depend on how 'proprietary' is mktree, by 'proprietary' I mean 
dependent on the kernel patch. 

Cheers,

C.