Re: How much of a mess does OpenVZ make? ;) Was: What can OpenVZ do?

[Mike Waychison <mikew@google.com>]

Oren Laadan wrote:
> 
> Mike Waychison wrote:
>> Linus Torvalds wrote:
>>> On Thu, 12 Mar 2009, Sukadev Bhattiprolu wrote:
>>>
>>>> Ying Han [yinghan@google.com] wrote:
>>>> | Hi Serge:
>>>> | I made a patch based on Oren's tree recently which implement a new
>>>> | syscall clone_with_pid. I tested with checkpoint/restart process tree
>>>> | and it works as expected.
>>>>
>>>> Yes, I think we had a version of clone() with pid a while ago.
>>> Are people _at_all_ thinking about security?
>>>
>>> Obviously not.
>>>
>>> There's no way we can do anything like this. Sure, it's trivial to do 
>>> inside the kernel. But it also sounds like a _wonderful_ attack vector 
>>> against badly written user-land software that sends signals and has small 
>>> races.
>> I'm not really sure how this is different than a malicious app going off 
>> and spawning thousands of threads in an attempt to hit a target pid from 
>> a security pov.  Sure, it makes it easier, but it's not like there is 
>> anything in place to close the attack vector.
>>
>>> Quite frankly, from having followed the discussion(s) over the last few 
>>> weeks about checkpoint/restart in various forms, my reaction to just about 
>>> _all_ of this is that people pushing this are pretty damn borderline. 
>>>
>>> I think you guys are working on all the wrong problems. 
>>>
>>> Let's face it, we're not going to _ever_ checkpoint any kind of general 
>>> case process. Just TCP makes that fundamentally impossible in the general 
>>> case, and there are lots and lots of other cases too (just something as 
>>> totally _trivial_ as all the files in the filesystem that don't get rolled 
>>> back).
>> In some instances such as ours, TCP is probably the easiest thing to 
>> migrate.  In an rpc-based cluster application, TCP is nothing more than 
>> an RPC channel and applications already have to handle RPC channel 
>> failure and re-establishment.
>>
>> I agree that this is not the 'general case' as you mention above 
>> however.  This is the bit that sorta bothers me with the way the 
>> implementation has been going so far on this list.  The implementation 
>> that folks are building on top of Oren's patchset tries to be everything 
>> to everybody.  For our purposes, we need to have the flexibility of 
>> choosing *how* we checkpoint.  The line seems to be arbitrarily drawn at 
>> the kernel being responsible for checkpointing and restoring all 
>> resources associated with a task, and leaving userland with nothing more 
>> than transporting filesystem bits.  This approach isn't flexible enough: 
>>   Consider the case where we want to stub out most of the TCP file 
>> descriptors with ECONNRESETed sockets because we know that they are RPC 
>> sockets and can re-establish themselves, but we want to use some other 
>> mechanism for TCP sockets we don't know much about.  The current 
>> monolithic approach has zero flexibility for doing anything like this, 
>> and I figure out how we could even fit anything like this in.
> 
> The flexibility exists, but wasn't spelled out, so here it is:
> 
> 1) Similar to madvice(), I envision a cradvice() that could tell the c/r
> something about specific resources, e.g.:
>  * cradvice(CR_ADV_MEM, ptr, len)  -> don't save that memory, it's scratch
>  * cradvice(CR_ADV_SOCK, fd, CR_ADV_SOCK_RESET)  -> reset connection on restart
> etc .. (nevermind the exact interface right now)
> 
> 2) Tasks can ask to be notified (e.g. register a signal) when a checkpoint
> or a restart complete successfully. At that time they can do their private
> house-keeping if they know better.
> 
> 3) If restoring some resource is significantly easier in user space (e.g. a
> file-descriptor of some special device which user space knows how to
> re-initialize), then the restarting task can prepare it ahead of time,
> and, call:
>   * cradvice(CR_ADV_USERFD, fd, 0)  -> use the fd in place instead of trying
> 				       to restore it yourself.

This would be called by the embryo process (mktree.c?) before calling 
sys_restart?

> 
> Method #3 is what I used in Zap to implement distributed checkpoints, where
> it is so much easier to recreate all network connections in user space then
> putting that logic into the kernel.
> 
> Now, on the other hand, doing the c/r from userland is much less flexible
> than in the kernel (e.g. epollfd, futex state and much more) and requires
> exposing tremendous amount of in-kernel data to user space. And we all know
> than exposing internals is always a one-way ticket :(
> 
> [...]
> 
> Oren.
> 
> 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>