Re: RFC: Self-snapshotting in Linux

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Ben Nizette wrote:
> On Wed, 2008-04-16 at 16:44 +0800, Peter Teoh wrote:
> > Essentially, to reiterate the key idea:   able to snapshot the current
> > kernel+userspace permanent.....restore from another snapshot....and
> > then switch back again if needed etc.....will the implementation be
> > difficult...if not impossible????
> >
>
> As I see it the main thing is that VMWare doesn't have to worry about
> trying to put hardware in to (and pull out of) low power modes.  VMWare
> hardware is never left in an undefined state by poorly written drivers
> etc.
>
> I think hibernation is about what you want; snapshotting as you describe
> it should fall down for about the same reasons

I guess there is the hardware / drivers issue.  I would like to claim
I've found hibernation to be reliable but unfortunately that's not
100% true.  As you say, that's inherent to snapshotting on
unvirtualised hardware - calling it snapshotting instead of
hibernation wouldn't change anything.

I think VMware's real advantages are more about management and
convenience.
1) Disk virtualisation.  It's much more convenient to create a VMware
image than a new linux partition.   Linux will let you use disk images
on a filesystem using the loopback driver, but that tends not to be
well supported by installers.  wubi[1] is the obvious counterexample
here.
2) Linux hibernation does the in-memory snapshot OK, but the
*filesystem* obviouslly isn't a part of that snapshot.  In some
unfortunate circumstances, you can end up resuming a memory image
which is not consistent with the filesystem and corrupting it.
IMHO filesystems need to include a "last modified" timestamp and check
it on resume.

[1] http://wubi-installer.org/

Re: RFC: Self-snapshotting in Linux

[Pavel Machek]

> Ben Nizette wrote:
> > On Wed, 2008-04-16 at 16:44 +0800, Peter Teoh wrote:
> > > Essentially, to reiterate the key idea:   able to snapshot the current
> > > kernel+userspace permanent.....restore from another snapshot....and
> > > then switch back again if needed etc.....will the implementation be
> > > difficult...if not impossible????
> > >
> >
> > As I see it the main thing is that VMWare doesn't have to worry about
> > trying to put hardware in to (and pull out of) low power modes.  VMWare
> > hardware is never left in an undefined state by poorly written drivers
> > etc.
> >
> > I think hibernation is about what you want; snapshotting as you describe
> > it should fall down for about the same reasons
> 
> I guess there is the hardware / drivers issue.  I would like to claim
> I've found hibernation to be reliable but unfortunately that's not
> 100% true.  

So debug it, it is open source after all. Or at least file a bugs.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Pavel Machek wrote:
>> I guess there is the hardware / drivers issue.  I would like to claim
>> I've found hibernation to be reliable but unfortunately that's not
>> 100% true.
>>     
> So debug it, it is open source after all. Or at least file a bugs.
>   
I definitely want to debug it; I love hibernating.  It's a regression 
and it'll affect my distro kernel once I upgrade to Ubuntu Hardy.

<http://bugzilla.kernel.org/show_bug.cgi?id=10196> "Backtraces during 
hibernate / resume".

I posted the backtraces but nobody could tell me what they were - they 
don't actually say whether they're OOPs or BUGs etc.  They might be 
softlockups - some times I got softlockup warnings, though thats not in 
the log I posted.

The backtraces have been harder to reproduce across different versions / 
configs, but there's definitely an issue where it hangs every time after 
it successfully suspends. It then tends to lock up about a minute after 
resuming.

I'm confused about some details (exactly which versions / configs 
produced which symptoms).  I don't have the machine right now, but I'll 
get back to it some time soon.  I think I managed to get a pair of 
broken and non-broken kernels to work on, and it seems to be a 
regression going from 2.6.22 to 2.6.23.

Any help in clearing up my confusion would be appreciated.  Is it worth 
trying to narrow down exactly where the hang happens between writing the 
hibernation image and shutting down?  If so how?  I suspect that's going 
to be easier than bisecting it.

Alan

Re: RFC: Self-snapshotting in Linux

[Pavel Machek]

On Wed 2008-04-16 14:50:46, Alan Jenkins wrote:
> Pavel Machek wrote:
>>> I guess there is the hardware / drivers issue.  I would like to claim
>>> I've found hibernation to be reliable but unfortunately that's not
>>> 100% true.
>>>     
>> So debug it, it is open source after all. Or at least file a bugs.
>>   
> I definitely want to debug it; I love hibernating.  It's a regression and 
> it'll affect my distro kernel once I upgrade to Ubuntu Hardy.
>
> <http://bugzilla.kernel.org/show_bug.cgi?id=10196> "Backtraces during 
> hibernate / resume".
>
> I posted the backtraces but nobody could tell me what they were - they 
> don't actually say whether they're OOPs or BUGs etc.  They might be 
> softlockups - some times I got softlockup warnings, though thats not in the 
> log I posted.

Not sure what that is, I never seen that before.

Can you try vanilla 2.6.25-rc9 or something?

> I'm confused about some details (exactly which versions / configs produced 
> which symptoms).  I don't have the machine right now, but I'll get back to 
> it some time soon.  I think I managed to get a pair of broken and 
> non-broken kernels to work on, and it seems to be a regression going from 
> 2.6.22 to 2.6.23.

If you can do a git bisect between those two, we could probably fix it
very quickly.

> Any help in clearing up my confusion would be appreciated.  Is it worth 
> trying to narrow down exactly where the hang happens between writing the 
> hibernation image and shutting down?  If so how?  I suspect that's going to 
> be easier than bisecting it.

Adding printks?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Pavel Machek wrote:
> On Wed 2008-04-16 14:50:46, Alan Jenkins wrote:
>   
>> Pavel Machek wrote:
>>     
>>>> I guess there is the hardware / drivers issue.  I would like to claim
>>>> I've found hibernation to be reliable but unfortunately that's not
>>>> 100% true.
>>>>     
>>>>         
>>> So debug it, it is open source after all. Or at least file a bugs.
>>>   
>>>       
>> I definitely want to debug it; I love hibernating.  It's a regression and 
>> it'll affect my distro kernel once I upgrade to Ubuntu Hardy.
>>
>> <http://bugzilla.kernel.org/show_bug.cgi?id=10196> "Backtraces during 
>> hibernate / resume".
>>
>> I posted the backtraces but nobody could tell me what they were - they 
>> don't actually say whether they're OOPs or BUGs etc.  They might be 
>> softlockups - some times I got softlockup warnings, though thats not in the 
>> log I posted.
>>     
>
> Not sure what that is, I never seen that before.
>
> Can you try vanilla 2.6.25-rc9 or something?
>   
I've just retested with 2.6.25.  I reproduced the hang that happens
after (otherwise successful) hibernation.  After a second hibernation &
resume I reproduced the mysterious call-traces.

I posted this information, a full system log (including 2 hibernation
cycles and the calltraces), and the kernel config to the bugzilla entry.

This was in response to Rafael's request on the bugzilla.  So I expect
someone will look at it anyway, but I thought I should copy you on this.

Thanks,

Alan

Re: RFC: Self-snapshotting in Linux

[Nigel Cunningham]

Hi.

On Wed, 2008-04-16 at 03:27 -0700, Alan Jenkins wrote:
> I guess there is the hardware / drivers issue.  I would like to claim
> I've found hibernation to be reliable but unfortunately that's not
> 100% true.  As you say, that's inherent to snapshotting on
> unvirtualised hardware - calling it snapshotting instead of
> hibernation wouldn't change anything.

On a particular set of hardware, it should be possible to make it 100%
reliable. If you've found bugs, please report them. We can only fix
issues if we know they exist.

Regards,

Nigel

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Nigel Cunningham wrote:
> Hi.
>
> On Wed, 2008-04-16 at 03:27 -0700, Alan Jenkins wrote:
>   
>> I guess there is the hardware / drivers issue.  I would like to claim
>> I've found hibernation to be reliable but unfortunately that's not
>> 100% true.  As you say, that's inherent to snapshotting on
>> unvirtualised hardware - calling it snapshotting instead of
>> hibernation wouldn't change anything.
>>     
>
> On a particular set of hardware, it should be possible to make it 100%
> reliable. If you've found bugs, please report them. We can only fix
> issues if we know they exist.
>   
Thanks!

I already replied to Pavels similar comment - I had posted on Bugzilla.  
I guess it missed the right people because it started off in the wrong 
category.  (The first symptom I reported was my 2.13Ghz cpu running at 
1Ghz after a sucessful hibernation cycle).  At Rafaels request, I shall 
be retesting with the latest 2.6.25-rc kernel on Saturday.  I look 
forward to getting it fixed - I know it should be possible because it's 
a regression.

Alan

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Scott Lovenberg wrote:
> Peter Teoh wrote:
> Maybe you load up another kernel to handle the snapshot, and then hand
> the system back to it afterwards?  What do you think?

Isn't that just what Ying Huans kexec-based hibernation does?

Re: RFC: Self-snapshotting in Linux

[Peter Teoh]

On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
> Scott Lovenberg wrote:
>
> > Peter Teoh wrote:
>  > Maybe you load up another kernel to handle the snapshot, and then hand
>  > the system back to it afterwards?  What do you think?
>
>
> Isn't that just what Ying Huans kexec-based hibernation does?
>

This list is awesome.   After I read up on this kexec-based hibernation thing:

http://kerneltrap.org/node/11756

I realized it is about the same idea.   Some differences though:

My original starting point was VMWare's snapshot idea.   Drawing an
analogy from there, the idea is to freeze and restore back entire
kernel + userspace application.   For integrity reason, filesystem
should be included in the frozen image as well.

Currently, what we are doing now is to have a bank of Norton
Ghost-based images of the entire OS and just selectively restoring
back the OS we want to work on.   Very fast - less than 30secs the
entire OS can be restored back.   But problem is that it need to be
boot up - which is very slow.   And there userspace state cannot be
frozen and restored back.

VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
access requirements.   There goes Xen's virtualization approach as
well.

Another approach is this (from an email by Scott Lovenberg) - using
RELOCATABLE kernel (or may be not?????I really don't know, but idea is
below):

a.   Assuming we have 32G (64bit hardware can do that) of memory, but
we want to have 7 32-bit OS running (not concurrently) - so then
memory is partition into 8 x 4GB each - the lowest 4GB reserved for
the current running OS.   Each OS will be housed into each 4G of
memory.   When each OS is running, it will access its own partition on
the harddisk/memory, security concerns put aside.   Switching from one
OS to another OS is VOLUNTARILY done by the user - equivalent to that
of "desktop" feature in Solaris CDE. Restoring back essentially is
just copying from each of the 4GB into the lowest 4GB memory range.
Because only the lowest 4gb is used, only 32 bit instruction is
needed, 64bit is needed only when copying from one 4GB memory
partition into the lowest 4GB region, and vice versa.   And together
with using  partitioning of harddisk for each OS, switching among the
different OS kernel should be in seconds, much less than 1 minute,
correct?

Technically, does the above sound feasible?


-- 
Regards,
Peter Teoh

Re: RFC: Self-snapshotting in Linux

[Disconnect]

On Wed, Apr 16, 2008 at 11:06 AM, Peter Teoh <htmldeveloper@gmail.com> wrote:
>  This list is awesome.   After I read up on this kexec-based hibernation thing:
>
>  http://kerneltrap.org/node/11756
>
>  I realized it is about the same idea.   Some differences though:
>
>  My original starting point was VMWare's snapshot idea.   Drawing an
>  analogy from there, the idea is to freeze and restore back entire
>  kernel + userspace application.   For integrity reason, filesystem
>  should be included in the frozen image as well.
[snip]

It sounds like what you want is the existing ghost setup plus
swsusp/tuxonice.  Hibernate the system, which saves out your userspace
and kernel state, do the ghost save, then when you restore it'll boot
up right where it left off.

http://www.tuxonice.net

Re: RFC: Self-snapshotting in Linux

[Vivek Goyal]

On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
> On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
> > Scott Lovenberg wrote:
> >
> > > Peter Teoh wrote:
> >  > Maybe you load up another kernel to handle the snapshot, and then hand
> >  > the system back to it afterwards?  What do you think?
> >
> >
> > Isn't that just what Ying Huans kexec-based hibernation does?
> >
> 
> This list is awesome.   After I read up on this kexec-based hibernation thing:
> 
> http://kerneltrap.org/node/11756
> 
> I realized it is about the same idea.   Some differences though:
> 
> My original starting point was VMWare's snapshot idea.   Drawing an
> analogy from there, the idea is to freeze and restore back entire
> kernel + userspace application.   For integrity reason, filesystem
> should be included in the frozen image as well.
> 
> Currently, what we are doing now is to have a bank of Norton
> Ghost-based images of the entire OS and just selectively restoring
> back the OS we want to work on.   Very fast - less than 30secs the
> entire OS can be restored back.   But problem is that it need to be
> boot up - which is very slow.   And there userspace state cannot be
> frozen and restored back.
> 
> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
> access requirements.   There goes Xen's virtualization approach as
> well.
> 
> Another approach is this (from an email by Scott Lovenberg) - using
> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
> below):
> 
> a.   Assuming we have 32G (64bit hardware can do that) of memory, but
> we want to have 7 32-bit OS running (not concurrently) - so then
> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
> the current running OS.   Each OS will be housed into each 4G of
> memory.   When each OS is running, it will access its own partition on
> the harddisk/memory, security concerns put aside.   Switching from one
> OS to another OS is VOLUNTARILY done by the user - equivalent to that
> of "desktop" feature in Solaris CDE. Restoring back essentially is
> just copying from each of the 4GB into the lowest 4GB memory range.
> Because only the lowest 4gb is used, only 32 bit instruction is
> needed, 64bit is needed only when copying from one 4GB memory
> partition into the lowest 4GB region, and vice versa.   And together
> with using  partitioning of harddisk for each OS, switching among the
> different OS kernel should be in seconds, much less than 1 minute,
> correct?
> 

[CCing Huang and Eric]

I think Huang is doing something very similar in kexec based hibernation
and probably that idea can be extended to achive above.

Currently if system has got 4G of memory then one can reserve some
amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
and let it run from there. Huang's implementation is also targetting
the same thing where more than one kernel be in RAM at the same time
(in mutually exclusive RAM locations) and one can switch between those
kernels using kexec techniques.

To begin with, he is targetting co-existence of just two kernels and
second kernel can be used to save/resume the hibernated image.

In fact, because of RELOCATABLE nature of kernel, you don't have to
copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
running). At max one might require first 640 KB of memory and that
can be worked out, if need be.

This will indeed need to put devices into some kind of sleep state so
that next kernel can resume it.

So I think a variant of above is possible where on a large memory system
multiple kernels can coexist (while accessing separate disk partitions)
and one ought to be able to switch between kernels.

Technically, there are few important pieces. kexec, relocatable kernel,
hibernation, kexec based hibernation. First three pieces are already
in place and fourth one is under development and after that I think
it is just a matter of putting everything together.

Thanks
Vivek

Re: RFC: Self-snapshotting in Linux

[Scott Lovenberg]

Scott Lovenberg wrote:
> Vivek Goyal wrote:
>> On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>>   
>>> On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>>>     
>>>> Scott Lovenberg wrote:
>>>>
>>>>       
>>>>> Peter Teoh wrote:
>>>>>         
>>>>  > Maybe you load up another kernel to handle the snapshot, and then hand
>>>>  > the system back to it afterwards?  What do you think?
>>>>
>>>>
>>>> Isn't that just what Ying Huans kexec-based hibernation does?
>>>>
>>>>       
>>> This list is awesome.   After I read up on this kexec-based hibernation thing:
>>>
>>> http://kerneltrap.org/node/11756
>>>
>>> I realized it is about the same idea.   Some differences though:
>>>
>>> My original starting point was VMWare's snapshot idea.   Drawing an
>>> analogy from there, the idea is to freeze and restore back entire
>>> kernel + userspace application.   For integrity reason, filesystem
>>> should be included in the frozen image as well.
>>>
>>> Currently, what we are doing now is to have a bank of Norton
>>> Ghost-based images of the entire OS and just selectively restoring
>>> back the OS we want to work on.   Very fast - less than 30secs the
>>> entire OS can be restored back.   But problem is that it need to be
>>> boot up - which is very slow.   And there userspace state cannot be
>>> frozen and restored back.
>>>
>>> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
>>> access requirements.   There goes Xen's virtualization approach as
>>> well.
>>>
>>> Another approach is this (from an email by Scott Lovenberg) - using
>>> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
>>> below):
>>>
>>> a.   Assuming we have 32G (64bit hardware can do that) of memory, but
>>> we want to have 7 32-bit OS running (not concurrently) - so then
>>> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
>>> the current running OS.   Each OS will be housed into each 4G of
>>> memory.   When each OS is running, it will access its own partition on
>>> the harddisk/memory, security concerns put aside.   Switching from one
>>> OS to another OS is VOLUNTARILY done by the user - equivalent to that
>>> of "desktop" feature in Solaris CDE. Restoring back essentially is
>>> just copying from each of the 4GB into the lowest 4GB memory range.
>>> Because only the lowest 4gb is used, only 32 bit instruction is
>>> needed, 64bit is needed only when copying from one 4GB memory
>>> partition into the lowest 4GB region, and vice versa.   And together
>>> with using  partitioning of harddisk for each OS, switching among the
>>> different OS kernel should be in seconds, much less than 1 minute,
>>> correct?
>>>
>>>     
>>
>> [CCing Huang and Eric]
>>
>> I think Huang is doing something very similar in kexec based hibernation
>> and probably that idea can be extended to achive above.
>>
>> Currently if system has got 4G of memory then one can reserve some
>> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
>> and let it run from there. Huang's implementation is also targetting
>> the same thing where more than one kernel be in RAM at the same time
>> (in mutually exclusive RAM locations) and one can switch between those
>> kernels using kexec techniques.
>>
>> To begin with, he is targetting co-existence of just two kernels and
>> second kernel can be used to save/resume the hibernated image.
>>
>> In fact, because of RELOCATABLE nature of kernel, you don't have to
>> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
>> running). At max one might require first 640 KB of memory and that
>> can be worked out, if need be.
>>
>> This will indeed need to put devices into some kind of sleep state so
>> that next kernel can resume it.
>>
>> So I think a variant of above is possible where on a large memory system
>> multiple kernels can coexist (while accessing separate disk partitions)
>> and one ought to be able to switch between kernels.
>>
>> Technically, there are few important pieces. kexec, relocatable kernel,
>> hibernation, kexec based hibernation. First three pieces are already
>> in place and fourth one is under development and after that I think
>> it is just a matter of putting everything together.
>>
>> Thanks
>> Vivek
>>   
Let's try this again, without the HTML ;)
> What about the way that the kernel does interrupt masks on CPUs during 
> a critical section of code on SMP machines?  It basically flushes the 
> TLB, and the cache, moves the process in critical section to a (now) 
> isolated CPU, and reroutes interrupts to another CPU.  If you took 
> that basic model and applied it to kernels instead of CPUs, you could 
> probably get the desired hand off of freezing one after flushing its 
> caches back (or sideways and then back in SMP) and moving the mm to 
> your unfrozen kernel and routing the processes there. After 
> snapshotting, flush the cache back again, and reroute each process to 
> the once again unfrozen kernel, handing them back again?  Would this 
> basic model work for isolation and snapshotting and then transitioning 
> back?  Oh, yeah, and block each process so it doesn't try to run 
> anything during snapshot :-).  Or, save PCs and then load them back 
> again, I guess... although that's a waste, and a disaster waiting to 
> happen... not that I've let that deter me before :-).  Unfortunately, 
> this is so far out of my skill range and knowledge base, that I can't 
> speak intelligently on it at any lower level.  Can someone fill in the 
> gaps for me?

Re: RFC: Self-snapshotting in Linux

[Vivek Goyal]

On Wed, Apr 16, 2008 at 04:07:00PM -0400, Scott Lovenberg wrote:
> Vivek Goyal wrote:
>> On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>>   
>>> On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>>>     
>>>> Scott Lovenberg wrote:
>>>>
>>>>       
>>>>> Peter Teoh wrote:
>>>>>         
>>>>  > Maybe you load up another kernel to handle the snapshot, and then hand
>>>>  > the system back to it afterwards?  What do you think?
>>>>
>>>>
>>>> Isn't that just what Ying Huans kexec-based hibernation does?
>>>>
>>>>       
>>> This list is awesome.   After I read up on this kexec-based hibernation thing:
>>>
>>> http://kerneltrap.org/node/11756
>>>
>>> I realized it is about the same idea.   Some differences though:
>>>
>>> My original starting point was VMWare's snapshot idea.   Drawing an
>>> analogy from there, the idea is to freeze and restore back entire
>>> kernel + userspace application.   For integrity reason, filesystem
>>> should be included in the frozen image as well.
>>>
>>> Currently, what we are doing now is to have a bank of Norton
>>> Ghost-based images of the entire OS and just selectively restoring
>>> back the OS we want to work on.   Very fast - less than 30secs the
>>> entire OS can be restored back.   But problem is that it need to be
>>> boot up - which is very slow.   And there userspace state cannot be
>>> frozen and restored back.
>>>
>>> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
>>> access requirements.   There goes Xen's virtualization approach as
>>> well.
>>>
>>> Another approach is this (from an email by Scott Lovenberg) - using
>>> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
>>> below):
>>>
>>> a.   Assuming we have 32G (64bit hardware can do that) of memory, but
>>> we want to have 7 32-bit OS running (not concurrently) - so then
>>> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
>>> the current running OS.   Each OS will be housed into each 4G of
>>> memory.   When each OS is running, it will access its own partition on
>>> the harddisk/memory, security concerns put aside.   Switching from one
>>> OS to another OS is VOLUNTARILY done by the user - equivalent to that
>>> of "desktop" feature in Solaris CDE. Restoring back essentially is
>>> just copying from each of the 4GB into the lowest 4GB memory range.
>>> Because only the lowest 4gb is used, only 32 bit instruction is
>>> needed, 64bit is needed only when copying from one 4GB memory
>>> partition into the lowest 4GB region, and vice versa.   And together
>>> with using  partitioning of harddisk for each OS, switching among the
>>> different OS kernel should be in seconds, much less than 1 minute,
>>> correct?
>>>
>>>     
>>
>> [CCing Huang and Eric]
>>
>> I think Huang is doing something very similar in kexec based hibernation
>> and probably that idea can be extended to achive above.
>>
>> Currently if system has got 4G of memory then one can reserve some
>> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
>> and let it run from there. Huang's implementation is also targetting
>> the same thing where more than one kernel be in RAM at the same time
>> (in mutually exclusive RAM locations) and one can switch between those
>> kernels using kexec techniques.
>>
>> To begin with, he is targetting co-existence of just two kernels and
>> second kernel can be used to save/resume the hibernated image.
>>
>> In fact, because of RELOCATABLE nature of kernel, you don't have to
>> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
>> running). At max one might require first 640 KB of memory and that
>> can be worked out, if need be.
>>
>> This will indeed need to put devices into some kind of sleep state so
>> that next kernel can resume it.
>>
>> So I think a variant of above is possible where on a large memory system
>> multiple kernels can coexist (while accessing separate disk partitions)
>> and one ought to be able to switch between kernels.
>>
>> Technically, there are few important pieces. kexec, relocatable kernel,
>> hibernation, kexec based hibernation. First three pieces are already
>> in place and fourth one is under development and after that I think
>> it is just a matter of putting everything together.
>>
>> Thanks
>> Vivek
>>   
> What about the way that the kernel does interrupt masks on CPUs during a 
> critical section of code on SMP machines?  It basically flushes the TLB, 
> and the cache, moves the process in critical section to a (now) isolated 
> CPU, and reroutes interrupts to another CPU.  If you took that basic model 
> and applied it to kernels instead of CPUs, you could probably get the 
> desired hand off of freezing one after flushing its caches back (or 
> sideways and then back in SMP) and moving the mm to your unfrozen kernel 
> and routing the processes there. After snapshotting, flush the cache back 
> again, and reroute each process to the once again unfrozen kernel, handing 
> them back again?  Would this basic model work for isolation and 
> snapshotting and then transitioning back?  Oh, yeah, and block each process 
> so it doesn't try to run anything during snapshot :-).  Or, save PCs and 
> then load them back again, I guess... although that's a waste, and a 
> disaster waiting to happen... not that I've let that deter me before :-).  
> Unfortunately, this is so far out of my skill range and knowledge base, 
> that I can't speak intelligently on it at any lower level.  Can someone 
> fill in the gaps for me?

Not very sure what you are saying here but one important piece missing
from your prposal seems to be state of various devices between kernels.

Thanks
Vivek

Re: RFC: Self-snapshotting in Linux

[Scott Lovenberg]

Vivek Goyal wrote:
> On Wed, Apr 16, 2008 at 04:07:00PM -0400, Scott Lovenberg wrote:
>   
>> Vivek Goyal wrote:
>>     
>>> On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>>>   
>>>       
>>>> On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>>>>     
>>>>         
>>>>> Scott Lovenberg wrote:
>>>>>
>>>>>       
>>>>>           
>>>>>> Peter Teoh wrote:
>>>>>>         
>>>>>>             
>>>>>  > Maybe you load up another kernel to handle the snapshot, and then hand
>>>>>  > the system back to it afterwards?  What do you think?
>>>>>
>>>>>
>>>>> Isn't that just what Ying Huans kexec-based hibernation does?
>>>>>
>>>>>       
>>>>>           
>>>> This list is awesome.   After I read up on this kexec-based hibernation thing:
>>>>
>>>> http://kerneltrap.org/node/11756
>>>>
>>>> I realized it is about the same idea.   Some differences though:
>>>>
>>>> My original starting point was VMWare's snapshot idea.   Drawing an
>>>> analogy from there, the idea is to freeze and restore back entire
>>>> kernel + userspace application.   For integrity reason, filesystem
>>>> should be included in the frozen image as well.
>>>>
>>>> Currently, what we are doing now is to have a bank of Norton
>>>> Ghost-based images of the entire OS and just selectively restoring
>>>> back the OS we want to work on.   Very fast - less than 30secs the
>>>> entire OS can be restored back.   But problem is that it need to be
>>>> boot up - which is very slow.   And there userspace state cannot be
>>>> frozen and restored back.
>>>>
>>>> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
>>>> access requirements.   There goes Xen's virtualization approach as
>>>> well.
>>>>
>>>> Another approach is this (from an email by Scott Lovenberg) - using
>>>> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
>>>> below):
>>>>
>>>> a.   Assuming we have 32G (64bit hardware can do that) of memory, but
>>>> we want to have 7 32-bit OS running (not concurrently) - so then
>>>> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
>>>> the current running OS.   Each OS will be housed into each 4G of
>>>> memory.   When each OS is running, it will access its own partition on
>>>> the harddisk/memory, security concerns put aside.   Switching from one
>>>> OS to another OS is VOLUNTARILY done by the user - equivalent to that
>>>> of "desktop" feature in Solaris CDE. Restoring back essentially is
>>>> just copying from each of the 4GB into the lowest 4GB memory range.
>>>> Because only the lowest 4gb is used, only 32 bit instruction is
>>>> needed, 64bit is needed only when copying from one 4GB memory
>>>> partition into the lowest 4GB region, and vice versa.   And together
>>>> with using  partitioning of harddisk for each OS, switching among the
>>>> different OS kernel should be in seconds, much less than 1 minute,
>>>> correct?
>>>>
>>>>     
>>>>         
>>> [CCing Huang and Eric]
>>>
>>> I think Huang is doing something very similar in kexec based hibernation
>>> and probably that idea can be extended to achive above.
>>>
>>> Currently if system has got 4G of memory then one can reserve some
>>> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
>>> and let it run from there. Huang's implementation is also targetting
>>> the same thing where more than one kernel be in RAM at the same time
>>> (in mutually exclusive RAM locations) and one can switch between those
>>> kernels using kexec techniques.
>>>
>>> To begin with, he is targetting co-existence of just two kernels and
>>> second kernel can be used to save/resume the hibernated image.
>>>
>>> In fact, because of RELOCATABLE nature of kernel, you don't have to
>>> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
>>> running). At max one might require first 640 KB of memory and that
>>> can be worked out, if need be.
>>>
>>> This will indeed need to put devices into some kind of sleep state so
>>> that next kernel can resume it.
>>>
>>> So I think a variant of above is possible where on a large memory system
>>> multiple kernels can coexist (while accessing separate disk partitions)
>>> and one ought to be able to switch between kernels.
>>>
>>> Technically, there are few important pieces. kexec, relocatable kernel,
>>> hibernation, kexec based hibernation. First three pieces are already
>>> in place and fourth one is under development and after that I think
>>> it is just a matter of putting everything together.
>>>
>>> Thanks
>>> Vivek
>>>   
>>>       
>> What about the way that the kernel does interrupt masks on CPUs during a 
>> critical section of code on SMP machines?  It basically flushes the TLB, 
>> and the cache, moves the process in critical section to a (now) isolated 
>> CPU, and reroutes interrupts to another CPU.  If you took that basic model 
>> and applied it to kernels instead of CPUs, you could probably get the 
>> desired hand off of freezing one after flushing its caches back (or 
>> sideways and then back in SMP) and moving the mm to your unfrozen kernel 
>> and routing the processes there. After snapshotting, flush the cache back 
>> again, and reroute each process to the once again unfrozen kernel, handing 
>> them back again?  Would this basic model work for isolation and 
>> snapshotting and then transitioning back?  Oh, yeah, and block each process 
>> so it doesn't try to run anything during snapshot :-).  Or, save PCs and 
>> then load them back again, I guess... although that's a waste, and a 
>> disaster waiting to happen... not that I've let that deter me before :-).  
>> Unfortunately, this is so far out of my skill range and knowledge base, 
>> that I can't speak intelligently on it at any lower level.  Can someone 
>> fill in the gaps for me?
>>     
>
> Not very sure what you are saying here but one important piece missing
> from your prposal seems to be state of various devices between kernels.
>
> Thanks
> Vivek
>
>   
Aha!  Now I see what you're getting at.  While I don't have any good 
solutions, I do now fully understand and appreciate the problem at hand! :)
I was proposing something to the effect of rerouting the processes to 
the new kernel and then firing it up at the same time as shutting down 
the one that was running.  Then just putting all the userspace processes 
on ice (IO block or something) until the snapshot happened and thawing 
them back out on the original.  Well, it looked pretty in my minds' eye 
at that moment.  Do you have any suggestions on this, or areas of 
research that might be promising?

Re: RFC: Self-snapshotting in Linux

[Peter Teoh]

On Thu, Apr 17, 2008 at 4:07 AM, Scott Lovenberg
<scott.lovenberg@gmail.com> wrote:
>
>  Vivek Goyal wrote:
>
>  On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>
>
>  On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>
>
>  Scott Lovenberg wrote:
>
>
>
>  Peter Teoh wrote:
>
>  > Maybe you load up another kernel to handle the snapshot, and then hand
>  > the system back to it afterwards? What do you think?
>
>
> Isn't that just what Ying Huans kexec-based hibernation does?
>
>
>  This list is awesome. After I read up on this kexec-based hibernation
> thing:
>
> http://kerneltrap.org/node/11756
>
> I realized it is about the same idea. Some differences though:
>
> My original starting point was VMWare's snapshot idea. Drawing an
> analogy from there, the idea is to freeze and restore back entire
> kernel + userspace application. For integrity reason, filesystem
> should be included in the frozen image as well.
>
> Currently, what we are doing now is to have a bank of Norton
> Ghost-based images of the entire OS and just selectively restoring
> back the OS we want to work on. Very fast - less than 30secs the
> entire OS can be restored back. But problem is that it need to be
> boot up - which is very slow. And there userspace state cannot be
> frozen and restored back.
>
> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
> access requirements. There goes Xen's virtualization approach as
> well.
>
> Another approach is this (from an email by Scott Lovenberg) - using
> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
> below):
>
> a. Assuming we have 32G (64bit hardware can do that) of memory, but
> we want to have 7 32-bit OS running (not concurrently) - so then
> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
> the current running OS. Each OS will be housed into each 4G of
> memory. When each OS is running, it will access its own partition on
> the harddisk/memory, security concerns put aside. Switching from one
> OS to another OS is VOLUNTARILY done by the user - equivalent to that
> of "desktop" feature in Solaris CDE. Restoring back essentially is
> just copying from each of the 4GB into the lowest 4GB memory range.
> Because only the lowest 4gb is used, only 32 bit instruction is
> needed, 64bit is needed only when copying from one 4GB memory
> partition into the lowest 4GB region, and vice versa. And together
> with using partitioning of harddisk for each OS, switching among the
> different OS kernel should be in seconds, much less than 1 minute,
> correct?
>
>
>  [CCing Huang and Eric]
>
> I think Huang is doing something very similar in kexec based hibernation
> and probably that idea can be extended to achive above.
>
> Currently if system has got 4G of memory then one can reserve some
> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
> and let it run from there. Huang's implementation is also targetting
> the same thing where more than one kernel be in RAM at the same time
> (in mutually exclusive RAM locations) and one can switch between those
> kernels using kexec techniques.
>
> To begin with, he is targetting co-existence of just two kernels and
> second kernel can be used to save/resume the hibernated image.
>
> In fact, because of RELOCATABLE nature of kernel, you don't have to
> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
> running). At max one might require first 640 KB of memory and that
> can be worked out, if need be.
>
> This will indeed need to put devices into some kind of sleep state so
> that next kernel can resume it.
>
> So I think a variant of above is possible where on a large memory system
> multiple kernels can coexist (while accessing separate disk partitions)
> and one ought to be able to switch between kernels.
>
> Technically, there are few important pieces. kexec, relocatable kernel,
> hibernation, kexec based hibernation. First three pieces are already
> in place and fourth one is under development and after that I think
> it is just a matter of putting everything together.
>
> Thanks
> Vivek

Wow...this is amazing discussion...I love it.

Can I asked a few questions?

>
>  What about the way that the kernel does interrupt masks on CPUs during a
> critical section of code on SMP machines?  It basically flushes the TLB, and
> the cache, moves the process in critical section to a (now) isolated CPU,

1.   Where is this isolation from multiple running CPU to single
running CPU currently done in the kernel?   If the CPU are executing
some inter-CPU order dependent stuff, like memory barriers, then can u
just freeze them?   And when resuming - is it necessary to restore
back in the same order?

2.   At the userspace level, what is the mechanism of freezing (APIs,
syscall, ioctl) everything - swap, fs, physical memory etc?

I tool a look at s2ram, and saw that it uses ioctl() with
SNAPSHOT_SET/GET_SWAP_PAGE and all the amazing work done:

http://lkml.org/lkml/2008/1/31/566

which are snapshotting for the swap, how about the fs and physical
mem...where is it done?   Or is it not necessary to be be done?

> and reroutes interrupts to another CPU.  If you took that basic model and
> applied it to kernels instead of CPUs, you could probably get the desired
> hand off of freezing one after flushing its caches back (or sideways and
> then back in SMP) and moving the mm to your unfrozen kernel and routing the
> processes there. After snapshotting, flush the cache back again, and reroute
> each process to the once again unfrozen kernel, handing them back again?
> Would this basic model work for isolation and snapshotting and then
> transitioning back?  Oh, yeah, and block each process so it doesn't try to
> run anything during snapshot :-).  Or, save PCs and then load them back

I have not fully understand the detail patch as mentioned above, but
this blocking of processes - can we just set all the processes to not
runnable-to-be-resume state, and during resuming just set these back
to runnable?   Or is more complicated than this?

There are some scenario whereby if the state is broken, it may be
difficult to be restored back again - eg, TCP/IP state machine.
while download something, if the network driver just freeze, and later
restored back again, will it be able to continue where it left off,
downloading the HTTP continuing from where it left off?   Or SSH
traffic....?    Can it worked?   Or unless the traffic is
time-sensitive?  (eg, password within depends on time)

> again, I guess... although that's a waste, and a disaster waiting to
> happen... not that I've let that deter me before :-).  Unfortunately, this
> is so far out of my skill range and knowledge base, that I can't speak
> intelligently on it at any lower level.  Can someone fill in the gaps for
> me?
>


-- 
Regards,
Peter Teoh

Re: RFC: Self-snapshotting in Linux

[Alan Jenkins]

Peter Teoh wrote:
> On Thu, Apr 17, 2008 at 4:07 AM, Scott Lovenberg
> <scott.lovenberg@gmail.com> wrote:
>   
>>  Vivek Goyal wrote:
>>
>>  On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>>
>>
>>  On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>>
>>
>>  Scott Lovenberg wrote:
>>
>>
>>
>>  Peter Teoh wrote:
>>
>>  > Maybe you load up another kernel to handle the snapshot, and then hand
>>  > the system back to it afterwards? What do you think?
>>
>>
>> Isn't that just what Ying Huans kexec-based hibernation does?
>>
>>
>>  This list is awesome. After I read up on this kexec-based hibernation
>> thing:
>>
>> http://kerneltrap.org/node/11756
>>
>> I realized it is about the same idea. Some differences though:
>>
>> My original starting point was VMWare's snapshot idea. Drawing an
>> analogy from there, the idea is to freeze and restore back entire
>> kernel + userspace application. For integrity reason, filesystem
>> should be included in the frozen image as well.
>>
>> Currently, what we are doing now is to have a bank of Norton
>> Ghost-based images of the entire OS and just selectively restoring
>> back the OS we want to work on. Very fast - less than 30secs the
>> entire OS can be restored back. But problem is that it need to be
>> boot up - which is very slow. And there userspace state cannot be
>> frozen and restored back.
>>
>> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
>> access requirements. There goes Xen's virtualization approach as
>> well.
>>
>> Another approach is this (from an email by Scott Lovenberg) - using
>> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
>> below):
>>
>> a. Assuming we have 32G (64bit hardware can do that) of memory, but
>> we want to have 7 32-bit OS running (not concurrently) - so then
>> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
>> the current running OS. Each OS will be housed into each 4G of
>> memory. When each OS is running, it will access its own partition on
>> the harddisk/memory, security concerns put aside. Switching from one
>> OS to another OS is VOLUNTARILY done by the user - equivalent to that
>> of "desktop" feature in Solaris CDE. Restoring back essentially is
>> just copying from each of the 4GB into the lowest 4GB memory range.
>> Because only the lowest 4gb is used, only 32 bit instruction is
>> needed, 64bit is needed only when copying from one 4GB memory
>> partition into the lowest 4GB region, and vice versa. And together
>> with using partitioning of harddisk for each OS, switching among the
>> different OS kernel should be in seconds, much less than 1 minute,
>> correct?
>>
>>
>>  [CCing Huang and Eric]
>>
>> I think Huang is doing something very similar in kexec based hibernation
>> and probably that idea can be extended to achive above.
>>
>> Currently if system has got 4G of memory then one can reserve some
>> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
>> and let it run from there. Huang's implementation is also targetting
>> the same thing where more than one kernel be in RAM at the same time
>> (in mutually exclusive RAM locations) and one can switch between those
>> kernels using kexec techniques.
>>
>> To begin with, he is targetting co-existence of just two kernels and
>> second kernel can be used to save/resume the hibernated image.
>>
>> In fact, because of RELOCATABLE nature of kernel, you don't have to
>> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
>> running). At max one might require first 640 KB of memory and that
>> can be worked out, if need be.
>>
>> This will indeed need to put devices into some kind of sleep state so
>> that next kernel can resume it.
>>
>> So I think a variant of above is possible where on a large memory system
>> multiple kernels can coexist (while accessing separate disk partitions)
>> and one ought to be able to switch between kernels.
>>
>> Technically, there are few important pieces. kexec, relocatable kernel,
>> hibernation, kexec based hibernation. First three pieces are already
>> in place and fourth one is under development and after that I think
>> it is just a matter of putting everything together.
>>
>> Thanks
>> Vivek
>>     
>
> Wow...this is amazing discussion...I love it.
>
> Can I asked a few questions?
>
>   
>>  What about the way that the kernel does interrupt masks on CPUs during a
>> critical section of code on SMP machines?  It basically flushes the TLB, and
>> the cache, moves the process in critical section to a (now) isolated CPU,
>>     
>
> 1.   Where is this isolation from multiple running CPU to single
> running CPU currently done in the kernel?   If the CPU are executing
> some inter-CPU order dependent stuff, like memory barriers, then can u
> just freeze them?   And when resuming - is it necessary to restore
> back in the same order?
>
> 2.   At the userspace level, what is the mechanism of freezing (APIs,
> syscall, ioctl) everything - swap, fs, physical memory etc?
>
> I tool a look at s2ram, and saw that it uses ioctl() with
> SNAPSHOT_SET/GET_SWAP_PAGE and all the amazing work done:
>
> http://lkml.org/lkml/2008/1/31/566
>
> which are snapshotting for the swap, how about the fs and physical
> mem...where is it done?   Or is it not necessary to be be done?
>   
Um.  You want to look at s2disk, not s2ram.  s2ram shouldn't really do 
anything with snapshots; it just puts the hardware into a special 
low-power state (or rather, asks the kernel to do so).  If it does any 
snapshotting, that's more a hang-over from s2disk - you should look at 
the original.

At a high level, the answer to your question is that hibernation 
(remember that's the primary use-case here) "snapshots" the system in 
memory, after quiesing processes, some (but not all) kernel threads, and 
DMA, so that nothing can be writing to the memory at the same time.  You 
could call this a snapshot of physical memory, although it will exclude 
certain areas by request of the BIOS.  This snapshot is then written to 
a hibernation file/partition.

You're probably a bit confused about swap because the current swap file 
is usually used to store the hibernation image.  When one talks about 
hibernation the "snapshot" is just the saved state of memory.  The other 
things you ask about - swap and filesystem - are _already_ on disk in a 
consistent, persistent format; you don't need to "snapshot" them.

In theory this is all well documented.  You're probably better off 
getting a better overview starting from published articles or even 
Howtos, rather than asking people to tell you what their code is 
supposed to do - because they already did that when they wrote the 
docs.  I don't know if there's a really great index for hibernation as a 
whole, but you should obviouslly be reading the relevant stuff under 
Documentation/ in the kernel source tree.
>> and reroutes interrupts to another CPU.  If you took that basic model and
>> applied it to kernels instead of CPUs, you could probably get the desired
>> hand off of freezing one after flushing its caches back (or sideways and
>> then back in SMP) and moving the mm to your unfrozen kernel and routing the
>> processes there. After snapshotting, flush the cache back again, and reroute
>> each process to the once again unfrozen kernel, handing them back again?
>> Would this basic model work for isolation and snapshotting and then
>> transitioning back?  Oh, yeah, and block each process so it doesn't try to
>> run anything during snapshot :-).  Or, save PCs and then load them back
>>     
>
> I have not fully understand the detail patch as mentioned above, but
> this blocking of processes - can we just set all the processes to not
> runnable-to-be-resume state, and during resuming just set these back
> to runnable?   Or is more complicated than this?
>
> There are some scenario whereby if the state is broken, it may be
> difficult to be restored back again - eg, TCP/IP state machine.
> while download something, if the network driver just freeze, and later
> restored back again, will it be able to continue where it left off,
> downloading the HTTP continuing from where it left off?   Or SSH
> traffic....?    Can it worked?   Or unless the traffic is
> time-sensitive?  (eg, password within depends on time)
>   
Yah.  I think it can work for short intervals, but in general userspace 
needs to be able to restart network connections.  TCP connections will 
time out for a number of reasons, and your IP address might even 
change.  So you should use a reliable downloader like wget; use screen 
for persistent SSH connections, etc.  General techniques that were 
originally used to cope with unreliable network connections, dialup, etc.

Alan

Re: RFC: Self-snapshotting in Linux

[Scott Lovenberg]

Alan Jenkins wrote:
> Peter Teoh wrote:
>> On Thu, Apr 17, 2008 at 4:07 AM, Scott Lovenberg
>> <scott.lovenberg@gmail.com> wrote:
>>  
>>>  Vivek Goyal wrote:
>>>
>>>  On Wed, Apr 16, 2008 at 11:06:05PM +0800, Peter Teoh wrote:
>>>
>>>
>>>  On 4/16/08, Alan Jenkins <alan-jenkins@tuffmail.co.uk> wrote:
>>>
>>>
>>>  Scott Lovenberg wrote:
>>>
>>>
>>>
>>>  Peter Teoh wrote:
>>>
>>>  > Maybe you load up another kernel to handle the snapshot, and then 
>>> hand
>>>  > the system back to it afterwards? What do you think?
>>>
>>>
>>> Isn't that just what Ying Huans kexec-based hibernation does?
>>>
>>>
>>>  This list is awesome. After I read up on this kexec-based hibernation
>>> thing:
>>>
>>> http://kerneltrap.org/node/11756
>>>
>>> I realized it is about the same idea. Some differences though:
>>>
>>> My original starting point was VMWare's snapshot idea. Drawing an
>>> analogy from there, the idea is to freeze and restore back entire
>>> kernel + userspace application. For integrity reason, filesystem
>>> should be included in the frozen image as well.
>>>
>>> Currently, what we are doing now is to have a bank of Norton
>>> Ghost-based images of the entire OS and just selectively restoring
>>> back the OS we want to work on. Very fast - less than 30secs the
>>> entire OS can be restored back. But problem is that it need to be
>>> boot up - which is very slow. And there userspace state cannot be
>>> frozen and restored back.
>>>
>>> VMWare images is slow, and cannot meet bare-metal CPU/direct hardware
>>> access requirements. There goes Xen's virtualization approach as
>>> well.
>>>
>>> Another approach is this (from an email by Scott Lovenberg) - using
>>> RELOCATABLE kernel (or may be not?????I really don't know, but idea is
>>> below):
>>>
>>> a. Assuming we have 32G (64bit hardware can do that) of memory, but
>>> we want to have 7 32-bit OS running (not concurrently) - so then
>>> memory is partition into 8 x 4GB each - the lowest 4GB reserved for
>>> the current running OS. Each OS will be housed into each 4G of
>>> memory. When each OS is running, it will access its own partition on
>>> the harddisk/memory, security concerns put aside. Switching from one
>>> OS to another OS is VOLUNTARILY done by the user - equivalent to that
>>> of "desktop" feature in Solaris CDE. Restoring back essentially is
>>> just copying from each of the 4GB into the lowest 4GB memory range.
>>> Because only the lowest 4gb is used, only 32 bit instruction is
>>> needed, 64bit is needed only when copying from one 4GB memory
>>> partition into the lowest 4GB region, and vice versa. And together
>>> with using partitioning of harddisk for each OS, switching among the
>>> different OS kernel should be in seconds, much less than 1 minute,
>>> correct?
>>>
>>>
>>>  [CCing Huang and Eric]
>>>
>>> I think Huang is doing something very similar in kexec based 
>>> hibernation
>>> and probably that idea can be extended to achive above.
>>>
>>> Currently if system has got 4G of memory then one can reserve some
>>> amount of RAM, lets say 128 MB (with in 4G) and load the kernel there
>>> and let it run from there. Huang's implementation is also targetting
>>> the same thing where more than one kernel be in RAM at the same time
>>> (in mutually exclusive RAM locations) and one can switch between those
>>> kernels using kexec techniques.
>>>
>>> To begin with, he is targetting co-existence of just two kernels and
>>> second kernel can be used to save/resume the hibernated image.
>>>
>>> In fact, because of RELOCATABLE nature of kernel, you don't have to
>>> copy the kernel to lower 4GB of memory (Assuming all 64bit kernels
>>> running). At max one might require first 640 KB of memory and that
>>> can be worked out, if need be.
>>>
>>> This will indeed need to put devices into some kind of sleep state so
>>> that next kernel can resume it.
>>>
>>> So I think a variant of above is possible where on a large memory 
>>> system
>>> multiple kernels can coexist (while accessing separate disk partitions)
>>> and one ought to be able to switch between kernels.
>>>
>>> Technically, there are few important pieces. kexec, relocatable kernel,
>>> hibernation, kexec based hibernation. First three pieces are already
>>> in place and fourth one is under development and after that I think
>>> it is just a matter of putting everything together.
>>>
>>> Thanks
>>> Vivek
>>>     
>>
>> Wow...this is amazing discussion...I love it.
>>
>> Can I asked a few questions?
>>
>>  
>>>  What about the way that the kernel does interrupt masks on CPUs 
>>> during a
>>> critical section of code on SMP machines?  It basically flushes the 
>>> TLB, and
>>> the cache, moves the process in critical section to a (now) isolated 
>>> CPU,
>>>     
>>
>> 1.   Where is this isolation from multiple running CPU to single
>> running CPU currently done in the kernel?   If the CPU are executing
>> some inter-CPU order dependent stuff, like memory barriers, then can u
>> just freeze them?   And when resuming - is it necessary to restore
>> back in the same order?
>>
IIRC, linux/kernel/arch/x86/smp_32.c.  I think that the barrier calls 
are macroed in.  I'm not sure about whether or not you can just freeze 
them, but I would think so long as the thread hasn't completed its 
critical section, nothing can really go too wrong (except for the 
universe imploding - but that's a risk we must take!) so long as all the 
memory and registers are put back as they were.  Unfortunately this is 
well above my knowledge to speak on with any authority whatsoever.  My 
gut says that if the barriers are implemented in a rational way and good 
programming principles have been used, it should mostly take care of 
itself (I know the code checks if the process loses the CPU it's on and 
acts accordingly).  I wish I could comment further, but I have to spend 
some heads down time in the code and get a VMWare box up that I can 
break a few dozen times.
>> 2.   At the userspace level, what is the mechanism of freezing (APIs,
>> syscall, ioctl) everything - swap, fs, physical memory etc?
>>
I'm not sure off the top of my head.  Peter, do you have any ideas on this?
>> I tool a look at s2ram, and saw that it uses ioctl() with
>> SNAPSHOT_SET/GET_SWAP_PAGE and all the amazing work done:
>>
>> http://lkml.org/lkml/2008/1/31/566
>>
>> which are snapshotting for the swap, how about the fs and physical
>> mem...where is it done?   Or is it not necessary to be be done?
>>   
> Um.  You want to look at s2disk, not s2ram.  s2ram shouldn't really do 
> anything with snapshots; it just puts the hardware into a special 
> low-power state (or rather, asks the kernel to do so).  If it does any 
> snapshotting, that's more a hang-over from s2disk - you should look at 
> the original.
>
> At a high level, the answer to your question is that hibernation 
> (remember that's the primary use-case here) "snapshots" the system in 
> memory, after quiesing processes, some (but not all) kernel threads, 
> and DMA, so that nothing can be writing to the memory at the same 
> time.  You could call this a snapshot of physical memory, although it 
> will exclude certain areas by request of the BIOS.  This snapshot is 
> then written to a hibernation file/partition.
>
> You're probably a bit confused about swap because the current swap 
> file is usually used to store the hibernation image.  When one talks 
> about hibernation the "snapshot" is just the saved state of memory.  
> The other things you ask about - swap and filesystem - are _already_ 
> on disk in a consistent, persistent format; you don't need to 
> "snapshot" them.
>
> In theory this is all well documented.  You're probably better off 
> getting a better overview starting from published articles or even 
> Howtos, rather than asking people to tell you what their code is 
> supposed to do - because they already did that when they wrote the 
> docs.  I don't know if there's a really great index for hibernation as 
> a whole, but you should obviouslly be reading the relevant stuff under 
> Documentation/ in the kernel source tree.
>>> and reroutes interrupts to another CPU.  If you took that basic 
>>> model and
>>> applied it to kernels instead of CPUs, you could probably get the 
>>> desired
>>> hand off of freezing one after flushing its caches back (or sideways 
>>> and
>>> then back in SMP) and moving the mm to your unfrozen kernel and 
>>> routing the
>>> processes there. After snapshotting, flush the cache back again, and 
>>> reroute
>>> each process to the once again unfrozen kernel, handing them back 
>>> again?
>>> Would this basic model work for isolation and snapshotting and then
>>> transitioning back?  Oh, yeah, and block each process so it doesn't 
>>> try to
>>> run anything during snapshot :-).  Or, save PCs and then load them back
>>>     
>>
>> I have not fully understand the detail patch as mentioned above, but
>> this blocking of processes - can we just set all the processes to not
>> runnable-to-be-resume state, and during resuming just set these back
>> to runnable?   Or is more complicated than this?
>>
>> There are some scenario whereby if the state is broken, it may be
>> difficult to be restored back again - eg, TCP/IP state machine.
>> while download something, if the network driver just freeze, and later
>> restored back again, will it be able to continue where it left off,
>> downloading the HTTP continuing from where it left off?   Or SSH
>> traffic....?    Can it worked?   Or unless the traffic is
>> time-sensitive?  (eg, password within depends on time)
>>   
> Yah.  I think it can work for short intervals, but in general 
> userspace needs to be able to restart network connections.  TCP 
> connections will time out for a number of reasons, and your IP address 
> might even change.  So you should use a reliable downloader like wget; 
> use screen for persistent SSH connections, etc.  General techniques 
> that were originally used to cope with unreliable network connections, 
> dialup, etc.
>
> Alan

RFC: Self-snapshotting in Linux

[Peter Teoh]

Sorry if this is a a crazy idea.....just be forewarned....

First, I would like to make a reference to VMWare's snapshot (name not
important, emphasize the idea) feature - hopefully u are familiar with
it.   This is a feature whereby u can freeze the entire OS (kernel +
userspace application running) and then later reload itself next time,
continuing where it left off, without reboot from ground zero.

Next, can I ask, is such a feature useful in Linux?   Ie, able to
restart the kernel + userspace application from where u left off, the
last time round.    Not JUST the normal suspend/resume feature, but
more important able to CHOOSE among the different available images for
u to resume on.   Eg, u want to freeze the current 2.6.25-rc6 kernel,
save it, and then restore back the 2.6.23-rc5 image, work on it, save
it, and then restore the previous image again.   All done without
virtualization as in the VMWare sense - which really is CPU intensive
and slow things  down a lot.   Now we can directly execute each OS
kernel image on the CPU, and since saving and restoring is quite fast
(eg, zipping up the entire physical memory before saving into
permanent storage) - I supposed this will be much faster than the
normal initialization/fsck work done normally....or did I missed out
anything?

Essentially, to reiterate the key idea:   able to snapshot the current
kernel+userspace permanent.....restore from another snapshot....and
then switch back again if needed etc.....will the implementation be
difficult...if not impossible????

-- 
Regards,
Peter Teoh

Re: RFC: Self-snapshotting in Linux

[eric miao]

Hibernation maybe ... try "echo "disk" > /sys/power/state"

On Wed, Apr 16, 2008 at 4:44 PM, Peter Teoh <htmldeveloper@gmail.com> wrote:
> Sorry if this is a a crazy idea.....just be forewarned....
>
>  First, I would like to make a reference to VMWare's snapshot (name not
>  important, emphasize the idea) feature - hopefully u are familiar with
>  it.   This is a feature whereby u can freeze the entire OS (kernel +
>  userspace application running) and then later reload itself next time,
>  continuing where it left off, without reboot from ground zero.
>
>  Next, can I ask, is such a feature useful in Linux?   Ie, able to
>  restart the kernel + userspace application from where u left off, the
>  last time round.    Not JUST the normal suspend/resume feature, but
>  more important able to CHOOSE among the different available images for
>  u to resume on.   Eg, u want to freeze the current 2.6.25-rc6 kernel,
>  save it, and then restore back the 2.6.23-rc5 image, work on it, save
>  it, and then restore the previous image again.   All done without
>  virtualization as in the VMWare sense - which really is CPU intensive
>  and slow things  down a lot.   Now we can directly execute each OS
>  kernel image on the CPU, and since saving and restoring is quite fast
>  (eg, zipping up the entire physical memory before saving into
>  permanent storage) - I supposed this will be much faster than the
>  normal initialization/fsck work done normally....or did I missed out
>  anything?
>
>  Essentially, to reiterate the key idea:   able to snapshot the current
>  kernel+userspace permanent.....restore from another snapshot....and
>  then switch back again if needed etc.....will the implementation be
>  difficult...if not impossible????
>
>  --
>  Regards,
>  Peter Teoh
>  --
>  To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>  the body of a message to majordomo@vger.kernel.org
>  More majordomo info at  http://vger.kernel.org/majordomo-info.html
>  Please read the FAQ at  http://www.tux.org/lkml/
>



-- 
Cheers
- eric

Re: RFC: Self-snapshotting in Linux

[Ben Nizette]

On Wed, 2008-04-16 at 16:44 +0800, Peter Teoh wrote:
> Essentially, to reiterate the key idea:   able to snapshot the current
> kernel+userspace permanent.....restore from another snapshot....and
> then switch back again if needed etc.....will the implementation be
> difficult...if not impossible????
> 

As I see it the main thing is that VMWare doesn't have to worry about
trying to put hardware in to (and pull out of) low power modes.  VMWare
hardware is never left in an undefined state by poorly written drivers
etc.

I think hibernation is about what you want; snapshotting as you describe
it should fall down for about the same reasons :-)

	--Ben.

Re: RFC: Self-snapshotting in Linux

[Scott Lovenberg]

[-- Attachment #1: Type: text/plain, Size: 3501 bytes --]

Peter Teoh wrote:
> Sorry if this is a a crazy idea.....just be forewarned....
>
> First, I would like to make a reference to VMWare's snapshot (name not
> important, emphasize the idea) feature - hopefully u are familiar with
> it.   This is a feature whereby u can freeze the entire OS (kernel +
> userspace application running) and then later reload itself next time,
> continuing where it left off, without reboot from ground zero.
>
> Next, can I ask, is such a feature useful in Linux?   Ie, able to
> restart the kernel + userspace application from where u left off, the
> last time round.    Not JUST the normal suspend/resume feature, but
> more important able to CHOOSE among the different available images for
> u to resume on.   Eg, u want to freeze the current 2.6.25-rc6 kernel,
> save it, and then restore back the 2.6.23-rc5 image, work on it, save
> it, and then restore the previous image again.   All done without
> virtualization as in the VMWare sense - which really is CPU intensive
> and slow things  down a lot.   Now we can directly execute each OS
> kernel image on the CPU, and since saving and restoring is quite fast
> (eg, zipping up the entire physical memory before saving into
> permanent storage) - I supposed this will be much faster than the
> normal initialization/fsck work done normally....or did I missed out
> anything?
>
> Essentially, to reiterate the key idea:   able to snapshot the current
> kernel+userspace permanent.....restore from another snapshot....and
> then switch back again if needed etc.....will the implementation be
> difficult...if not impossible????
>
>   
Hrm... well, the kernel has the ability since 2.6.20 (although its in 
testing AFAIK) to relocate itself on a crash.  Reference:


      Relocatable kernel support for x86

This feature (enabled with CONFIG_RELOCATABLE) isn't very noticeable for 
end-users but it's quite interesting from a kernel POV. Until now, it 
was a requirement that a i386 kernel was loaded at a fixed memory 
address in order to work, loading it in a different place wouldn't work. 
This feature allows to compile a kernel that can be loaded at different 
4K-aligned addresses, but always below 1 GB, with no runtime overhead. 
Kdump users (a feature introduced in 2.6.13 
<http://kernelnewbies.org/Linux_2_6_13> that triggers kexec 
<http://lwn.net/Articles/108595/> in a kernel crash in order to boot a 
kernel that has been previously loaded at an 'empty' address, then runs 
that kernel, saves the memory where the crashed kernel was placed, dumps 
it in a file and continues booting the system) will benefit from this 
because until now the "rescue kernel" needed to be compiled with 
different configuration options in order to make it bootable at a 
different address. With a relocatable kernel, the same kernel can be 
booted at different addresses. (commit 1 
<http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=968de4f02621db35b8ae5239c8cfc6664fb872d8>, 
2 
<http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=e69f202d0a1419219198566e1c22218a5c71a9a6>, 
3 
<http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=be274eeaf20b4c7155242645d5e2c48b023e609b>, 
4) 
<http://git.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=fd593d12770d4a0d1ff095d44b96436c18479ee8>



Maybe you load up another kernel to handle the snapshot, and then hand 
the system back to it afterwards?  What do you think?

[-- Attachment #2: scott_lovenberg.vcf --]
[-- Type: text/x-vcard, Size: 525 bytes --]

begin:vcard
fn:Scott Lovenberg
n:Lovenberg;Scott
org:The CleftStone Works
adr:;;;Kutztown;PA;19530;USA
email;internet:scott.lovenberg@gmail.com
tel;cell:570-856-2999
note;quoted-printable:Interests:=0D=0A=
	Programming, *NIX, Slackware, Samba, C/C++, Perl, Java, .NET, Kernel Hack=
	ing, Virtual Machines, FOSS, Security, Operating Systems, Various Softwar=
	e Stacks.  =
	=0D=0A=
	=0D=0A=
	Always have time for "shop talk"; feel free to drop me a line.
x-mozilla-html:TRUE
url:www.cleftstoneworks.com
version:2.1
end:vcard

Re: RFC: Self-snapshotting in Linux

[Pavel Machek]

> Sorry if this is a a crazy idea.....just be forewarned....
> 
> First, I would like to make a reference to VMWare's snapshot (name not
> important, emphasize the idea) feature - hopefully u are familiar with
> it.   This is a feature whereby u can freeze the entire OS (kernel +
> userspace application running) and then later reload itself next time,
> continuing where it left off, without reboot from ground zero.
> 
> Next, can I ask, is such a feature useful in Linux?   Ie, able to
> restart the kernel + userspace application from where u left off, the
> last time round.    Not JUST the normal suspend/resume feature, but
> more important able to CHOOSE among the different available images
> for
> u to resume on.   Eg, u want to freeze the current 2.6.25-rc6
> kernel,

Could we use english on this list?

Anyway, hibernate, cp /dev/root /somewhere/root.1; cp /dev/swap
/somewhere/swap.1

If you want to resume selected version, copy back that set of devices,
and proceed normally.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: RFC: Self-snapshotting in Linux

[Pavel Machek]

On Wed 2008-04-16 23:21:05, Peter Teoh wrote:
> Thank you for your reply.
> 
> Just some questions...
> 
> On 4/16/08, Pavel Machek <pavel@ucw.cz> wrote:
> > > Sorry if this is a a crazy idea.....just be forewarned....
> >  >
> >  > First, I would like to make a reference to VMWare's snapshot (name not
> >  > important, emphasize the idea) feature - hopefully u are familiar with
> >  > it.   This is a feature whereby u can freeze the entire OS (kernel +
> >  > userspace application running) and then later reload itself next time,
> >  > continuing where it left off, without reboot from ground zero.
> >  >
> >  > Next, can I ask, is such a feature useful in Linux?   Ie, able to
> >  > restart the kernel + userspace application from where u left off, the
> >  > last time round.    Not JUST the normal suspend/resume feature, but
> >  > more important able to CHOOSE among the different available images
> >  > for
> >  > u to resume on.   Eg, u want to freeze the current 2.6.25-rc6
> >  > kernel,
> >
> >
> > Could we use english on this list?
> >
> >  Anyway, hibernate, cp /dev/root /somewhere/root.1; cp /dev/swap
> >  /somewhere/swap.1
> >
> 
> The current userspace suspend actually saved the entire state of the
> memory (which applications are oepne/closed) into the swap area,
> right?

> So the 2nd line above of backingup the swap should allow us to restore
>  back state of the userspace applications later - correct?

Yes, but you need to match memory state with filesystem state...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: RFC: Self-snapshotting in Linux

[Greg Freemyer]

On Wed, Apr 16, 2008 at 4:44 AM, Peter Teoh <htmldeveloper@gmail.com> wrote:
> Sorry if this is a a crazy idea.....just be forewarned....
>
>  First, I would like to make a reference to VMWare's snapshot (name not
>  important, emphasize the idea) feature - hopefully u are familiar with
>  it.   This is a feature whereby u can freeze the entire OS (kernel +
>  userspace application running) and then later reload itself next time,
>  continuing where it left off, without reboot from ground zero.
>
>  Next, can I ask, is such a feature useful in Linux?   Ie, able to
>  restart the kernel + userspace application from where u left off, the
>  last time round.    Not JUST the normal suspend/resume feature, but
>  more important able to CHOOSE among the different available images for
>  u to resume on.   Eg, u want to freeze the current 2.6.25-rc6 kernel,
>  save it, and then restore back the 2.6.23-rc5 image, work on it, save
>  it, and then restore the previous image again.   All done without
>  virtualization as in the VMWare sense - which really is CPU intensive
>  and slow things  down a lot.   Now we can directly execute each OS
>  kernel image on the CPU, and since saving and restoring is quite fast
>  (eg, zipping up the entire physical memory before saving into
>  permanent storage) - I supposed this will be much faster than the
>  normal initialization/fsck work done normally....or did I missed out
>  anything?
>
>  Essentially, to reiterate the key idea:   able to snapshot the current
>  kernel+userspace permanent.....restore from another snapshot....and
>  then switch back again if needed etc.....will the implementation be
>  difficult...if not impossible????
>
>  --
>  Regards,
>  Peter Teoh

By using LVM-2 snapshots you can already do this with data drives.

So if you setup a XEN setup where the DomU boot drives are Dom0 LVs
you should be golden.

Ie. from Dom0 you can create/restore your snapshots of your DomU instances.

It may also be possible to make LVM snapshots work with the root
partition of the physical boot drive, but that would take initrd magic
I have never tried.

Greg
-- 
Greg Freemyer
Litigation Triage Solutions Specialist
http://www.linkedin.com/in/gregfreemyer
First 99 Days Litigation White Paper -
http://www.norcrossgroup.com/forms/whitepapers/99%20Days%20whitepaper.pdf

The Norcross Group
The Intersection of Evidence & Technology
http://www.norcrossgroup.com