Re: [RFC v3][PATCH 1/9] Create syscalls: sys_checkpoint, sys_restart

["Serge E. Hallyn" <serue@us.ibm.com>]

Quoting Oren Laadan (orenl@cs.columbia.edu):
> 
> 
> Serge E. Hallyn wrote:
> > Quoting Oren Laadan (orenl@cs.columbia.edu):
> >>
> >> Serge E. Hallyn wrote:
> >>> Quoting Oren Laadan (orenl@cs.columbia.edu):
> >>>> Create trivial sys_checkpoint and sys_restore system calls. They will
> >>>> enable to checkpoint and restart an entire container, to and from a
> >>>> checkpoint image file descriptor.
> >>>>
> >>>> The syscalls take a file descriptor (for the image file) and flags as
> >>>> arguments. For sys_checkpoint the first argument identifies the target
> >>>> container; for sys_restart it will identify the checkpoint image.
> >>>>
> >>>> Signed-off-by: Oren Laadan <orenl@cs.columbia.edu>
> >>>> ---
> >> [...]
> >>
> >>>> +/**
> >>>> + * sys_checkpoint - checkpoint a container
> >>>> + * @pid: pid of the container init(1) process
> >>>> + * @fd: file to which dump the checkpoint image
> >>>> + * @flags: checkpoint operation flags
> >>>> + */
> >>>> +asmlinkage long sys_checkpoint(pid_t pid, int fd, unsigned long flags)
> >>>> +{
> >>>> +	pr_debug("sys_checkpoint not implemented yet\n");
> >>>> +	return -ENOSYS;
> >>>> +}
> >>>> +/**
> >>>> + * sys_restart - restart a container
> >>>> + * @crid: checkpoint image identifier
> >>> So can we compare your api to Andrey's?
> >>>
> >>> You've explained before that crid is used to tie together multiple
> >>> calls to checkpoint, but why do you have to specify it for restart?
> >>> Can't it just come from the fd?  Or, the fd will be passed in
> >>> seek()d to the right position for the data for this task, so the crid
> >>> won't be available there?
> >> I added the 'crid' inside to support a mode of operation in which we
> >> would like the checkpoint data to remain in memory across multiple
> >> system calls. Here are example scenarios:
> >>
> >> 1) We will want to reduce down time by first buffering the checkpoint
> >> image in memory, then resuming the container, and only then writing
> >> the data back to a (the) file descriptor.
> >> So instead of:
> >>   freeze -> checkpoint and write back -> unfreeze
> >> We want:
> >>   freeze -> checkpoint to buffer -> unfreeze -> write back
> >> I envision each of these steps to be a separate invocation of a syscall.
> >> to the 'crid' returned by the sys_checkpoint() at the 2nd step, will be
> >> used to identify that data in the 4th step. (Note, that between the
> >> unfreeze and the write-back, another checkpoint may be already taken).
> >>
> >> 2) A task may want to take a checkpoint (e.g. of itself, or a whole
> >> container) and keep that checkpoint in memory; at a later time it may
> >> want to revert to that checkpoint. Moreover, it may keep multiple such
> >> checkpoints (to where it may want to return). 'crid' tells sys_restart
> >> which one to use.
> >>
> >> Note that this 'crid' will in fact be tied to resources that are kept
> >> by the kernel - e.g. references to COW pages (when we add that).
> >> Louis suggested to use a specialized FD instead of a numeric 'crid'
> >> (that is: create a anonymous inode and a struct file that represent
> >> that checkpoint in the kernel, and return an FD to it). This approach
> >> has pros and cons of 'crid' (see the archives of the containers
> >> mailing list). For now I kept 'crid', but I'm definitely open to change
> >> it to a FD.
> >>
> >> Oren.
> > 
> > Oh, so the crid identifies one checkpoint inside the file - the single
> > file can store multiple checkpoints?
> 
> Not quite. Let me rephrase the motivation first:
> 
> There are occasions when we would like to keep the checkpoint data in the
> kernel for some (relatively long) time, between syscalls. By "checkpoint
> data" I mean references to memory contents (pages) and all the other data.
> 
> The two scenarios above are two examples: between the syscall to checkpoint
> and the syscall to unfreeze and then write-back the data to a file (first
> example), and for some time until a task may want to "go back in time"
> (second example, useful for ultra fast "undo" for a task).
> 
> Note that in both cases when I say "keep in kernel" I mean before it is
> written to a file, or to the network. Simply in memory, in some efficient
> manner.
> 
> Subsequent syscalls will need to refer to a specific checkpoint data that
> is kept in memory - e.g. to write-back to a file-descriptor, or to clean
> up, or to restart from it. (At any single time a specific container may
> have multiple checkpoints associated with it - eg. because they have not
> yet been written back to storage but already taken).
> 
> Once the data is written back to a file descriptor, the in-kernel data can
> be discarded and cleaned-up.
> 
> The main reason why I want to keep the data in the kernel and not instead
> copy to user space, is efficiency: most of the checkpoint data is the memory
> footprint; by keeping the data in the kernel, one can merely keep a COW
> reference instead of a whole copy of everything (save space and copy time).
> 
> So, if we have keep data in kernel between syscalls, then we must have a
> way to refer to it. The current implementation uses a very simple 'crid'
> value to do that - although, clearly, at the moment it isn't used.
> 
> I hope this explains better.

Ah, ok.  So we're either using an fd or a crid.

Personally I'd then prefer two syscalls, which are wrappers around
a more flexible in-kernel api.  That way we can start with just
	sys_restart(int fd, long flags)
and add
	sys_restart_crid(int crid, long flags)
later.

-serge