Re: aufs as root vor openvz CT

Re: aufs as root vor openvz CT

[Hans-Peter Jansen]

Dear Junjiro, dear Sergey,

first of all, I just try to transform the communication between you two in 
order to help both parties. I'm not affiliated in any of the two, its just a 
poor mans AUFS contribution.

On Montag, 21. Juli 2014 15:33:15 Sergey Korshunoff wrote:
> > I won't be helpful for you until you give me enough information.
> > Do you have any special reason not to give the information?
> 
> I'm trying to use a aufs as a root fs for container. I use openvz
> kernels 2.6.18 and 2.6.32. While container starts, stops and works
> well it can not be suspended. There is nothing special to a kernel
> version and even union fs type (aufs, uniionfs). Problem is common to
> any stacked fs which opens fd itself. This is why I do not try to
> give info asked in README
> 
> Suspend count currently a fd opened by a stacked fs (internal fd) as a
> fd opened by application in container. I'm trying to ask openvz
> developers (https://bugzilla.openvz.org/show_bug.cgi?id=3024) about a
> method allowing to exclude internal fd of stacked fs from a suspend
> set.
> 
> Suspend perform a closing of the opened fd's. This fd's are reopened
> while resume is performed. Internal fd's of the stacked fs must not be
> counted by this processes.

Junjiro, openvz is a container technology (CT) similar to docker. I can 
imagine, that these technologies have a problem, that describes as follows:

In order to perform a proper suspend/resume cycle, the CT has to arrange 
everything to the state before the suspend. Therefore, they track the opened 
files, and restore their state during resume. Layering FS (LFS) do cause 
problems in this scenario, since files opened by them don't belong to any 
application running in the container.

As it stands, a LFS can't do much to solve the problem in itself, other then 
hinting the CT during suspend, which fd is used by the LFS itself. The CT 
should simply ignore those on resume, given, that opening the fd and restoring 
its state in the application layer should restore the LFS arrangement 
implicitly.

Since this issue is generic to LFS using their own fds, I'm sending this to 
the new linux-unionfs ML, too, and suggest to discuss it any further over 
there.

So, either define a special open flag for LFS internal fds, which, I guess, is 
rather unpopular, or define a respective fcntl, that those LFS should 
implement. Otherwise, every CT needs to detect the internal fds of every LFS 
implementation heuristically, which is the worst solution, of course.

Pete

Re: aufs as root vor openvz CT

[sfjro]

Hello Pete,
Thank you very much for your contribution.
I've repeatedly asked Sergey how to reproduce and the details, but got
no good answers. I hope your contribution will make the details clearer.


Hans-Peter Jansen:
> In order to perform a proper suspend/resume cycle, the CT has to arrange 
> everything to the state before the suspend. Therefore, they track the opened 
> files, and restore their state during resume. Layering FS (LFS) do cause 
> problems in this scenario, since files opened by them don't belong to any 
> application running in the container.

I'd ask generic questions (to anyone).
- Is such handling done in the container, or in the generic linux
  checkpoint/restart?
- If it is done in the generic feature, then is the linux
  checkpoint/restart feature specific to the containers?
- If not, we will be able to reproduce without any containers, no
  openvz, no old kernel.

As a first step, I need to know the detail steps to reproduce the
problem since I don't know much about suspend/resume.


> So, either define a special open flag for LFS internal fds, which, I guess, is 
> rather unpopular, or define a respective fcntl, that those LFS should 
> implement. Otherwise, every CT needs to detect the internal fds of every LFS 
> implementation heuristically, which is the worst solution, of course.

It depends upon how CT finds the opend files.
I have another idea.
- add a notifier-chain into the suspend feature (if there is not
  currently).
- any module who opens a file internaly regists itself to the
  notifier-chain.
- when the suspend event is notified via the notifier-chain, the module
  handle it.


J. R. Okajima

------------------------------------------------------------------------------
Want fast and easy access to all the code in your enterprise? Index and
search up to 200,000 lines of code with a free copy of Black Duck
Code Sight - the same software that powers the world's largest code
search on Ohloh, the Black Duck Open Hub! Try it now.
http://p.sf.net/sfu/bds

Re: aufs as root vor openvz CT

[Sergey Korshunoff]

Hello!

2014-07-23 6:59 GMT+04:00, sfjro@users.sourceforge.net
<sfjro@users.sourceforge.net>:
> I'd ask generic questions (to anyone).
> - Is such handling done in the container, or in the generic linux
>   checkpoint/restart?
> - If it is done in the generic feature, then is the linux
>   checkpoint/restart feature specific to the containers?
> - If not, we will be able to reproduce without any containers, no
>   openvz, no old kernel.
>
> As a first step, I need to know the detail steps to reproduce the
> problem since I don't know much about suspend/resume.

A generic hibernate with aufs  as a root fs is OK, ie there is no problem
executing a  hibernate command and then restarting OS from a saved state.

A problem is with a containers suspend/resume. Currently a openvz.org team
is working on RHEL7 based kernel (version 3.10). I don;t know if
suspend/resume for contaimers can be done with mainstream kernel. I
will to test a RHEL7 based kernel
when a openvz.team released it.

Then I will try to make a livecd like http://download.openvz.org/livecd/ but
based on gentoo and with kernels 2.6.32 and 3.10.x which can be used
to demontrate a problem.

Re: aufs as root vor openvz CT

[Sergey Korshunoff]

Succes story.
I modified a checkpoint part of the openvz 2.6.32 kernel (cpt_mm.c)
which handles a vma areas
by replacing vma->vm_file with vma->vm_prfile if vma->prfile != 0.
After that a checkpoint-restore is performed without any error.

I think that CRIU utils must know about that field in the vma struct
(if there is that filed) before CRIU can suspend a container located
inside an aufs file system.

Another solution may be to use vma_ops in aufs and to keep
vma->vm_file unchanged,