RE: Switching Kernels without Rebooting?

RE: Switching Kernels without Rebooting?

[Torrey Hoffman]

Jesse Pollard wrote:

[why switching kernels is very hard, and...]
 
> Before you even try switching kernels, first implement a process
> checkpoint/restart. The process must be resumed after a boot 
> using the same
> kernel, with all I/O resumed. Now get it accepted into the kernel.

Hear, hear!  That would be a useful feature, maybe not network servers, 
but for pure number crunching apps it would save people having to write 
all the state saving and recovery that is needed now for long term 
computations.

For bonus points, make it work for clusters to synchronously save and
restore state for the apps running on all the nodes at once...

Torrey

Re: Switching Kernels without Rebooting?

[C. Slater]

Would anyone else like to point out some other task somewhat related 
and have me do it? :-)

> > Before you even try switching kernels, first implement a process
> > checkpoint/restart. The process must be resumed after a boot
> > using the same
> > kernel, with all I/O resumed. Now get it accepted into the kernel.
> 
> Hear, hear!  That would be a useful feature, maybe not network servers, 
> but for pure number crunching apps it would save people having to write 
> all the state saving and recovery that is needed now for long term 
> computations.

Get a computer with hibernation support. That's just about what it is.

> 
> For bonus points, make it work for clusters to synchronously save and
> restore state for the apps running on all the nodes at once...

Bash script.

> 
> Torrey

Re: Switching Kernels without Rebooting?

[Jesse Pollard]

On Wed, 11 Jul 2001, C. Slater wrote:
>Would anyone else like to point out some other task somewhat related 
>and have me do it? :-)
>
>> > Before you even try switching kernels, first implement a process
>> > checkpoint/restart. The process must be resumed after a boot
>> > using the same
>> > kernel, with all I/O resumed. Now get it accepted into the kernel.
>> 
>> Hear, hear!  That would be a useful feature, maybe not network servers, 
>> but for pure number crunching apps it would save people having to write 
>> all the state saving and recovery that is needed now for long term 
>> computations.
>
>Get a computer with hibernation support. That's just about what it is.

Bzzzt wrong anser. Hibernation stops the entire kernel. checkpoint restart
stops processes, saves the entire state of the process. hibernation
is just halt the processor.

>> 
>> For bonus points, make it work for clusters to synchronously save and
>> restore state for the apps running on all the nodes at once...
>
>Bash script.

doesn't work - remember once the kernel is suspended it can't tell
another system that is has done so.

A full checkpoint/restart can potentially allow a process to migrate
from one node to another. It also allows other processing to be done
while the process is checkpointed:

	a. how do you reconstruct a software raid 5 while the system
	   is "suspended"
	b. how do you migrate to a different platform if the system is
	   suspended

Answer - you can't.

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.

Re: Switching Kernels without Rebooting?

[Ian Stirling]

> 
> On Wed, 11 Jul 2001, C. Slater wrote:
> >Would anyone else like to point out some other task somewhat related 
> >and have me do it? :-)
> >
> >> > Before you even try switching kernels, first implement a process
> >> > checkpoint/restart. The process must be resumed after a boot
> >> > using the same
> >> > kernel, with all I/O resumed. Now get it accepted into the kernel.
> >> 
> >> Hear, hear!  That would be a useful feature, maybe not network servers, 
> >> but for pure number crunching apps it would save people having to write 
> >> all the state saving and recovery that is needed now for long term 
> >> computations.
> >
> >Get a computer with hibernation support. That's just about what it is.
> 
> Bzzzt wrong anser. Hibernation stops the entire kernel. checkpoint restart
> stops processes, saves the entire state of the process. hibernation
> is just halt the processor.

Hibernation may not be.
I've just suspended to disk after the list line, pulled the power supplies,
taken the RAM chip out, shorted the pins to make really sure, then powered
back up.
Everything just resumed fine.

All I'd need to do kernel migration is a quick vi of the
disk file.

(well, almost)

Re: Switching Kernels without Rebooting?

[Jesse Pollard]

---------  Received message begins Here  ---------

> 
> > 
> > On Wed, 11 Jul 2001, C. Slater wrote:
> > >Would anyone else like to point out some other task somewhat related 
> > >and have me do it? :-)
> > >
> > >> > Before you even try switching kernels, first implement a process
> > >> > checkpoint/restart. The process must be resumed after a boot
> > >> > using the same
> > >> > kernel, with all I/O resumed. Now get it accepted into the kernel.
> > >> 
> > >> Hear, hear!  That would be a useful feature, maybe not network servers, 
> > >> but for pure number crunching apps it would save people having to write 
> > >> all the state saving and recovery that is needed now for long term 
> > >> computations.
> > >
> > >Get a computer with hibernation support. That's just about what it is.
> > 
> > Bzzzt wrong anser. Hibernation stops the entire kernel. checkpoint restart
> > stops processes, saves the entire state of the process. hibernation
> > is just halt the processor.
> 
> Hibernation may not be.
> I've just suspended to disk after the list line, pulled the power supplies,
> taken the RAM chip out, shorted the pins to make really sure, then powered
> back up.
> Everything just resumed fine.
> 
> All I'd need to do kernel migration is a quick vi of the
> disk file.
> 
> (well, almost)

That sounds more like a memory dump to disk, and reload after power restored.
Either that or possibly a separate power supply for RAM (something like a
trickle discharge capacitor; I've read that some capacitors can hold a charge
for about 3 days. Whether that would work for a large RAM or not, I have no
idea).

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: pollard@navo.hpc.mil

Any opinions expressed are solely my own.

Re: Switching Kernels without Rebooting?

[Michael H. Warfield]

On Thu, Jul 12, 2001 at 07:54:10AM -0500, Jesse Pollard wrote:

> > > On Wed, 11 Jul 2001, C. Slater wrote:
> > > >Would anyone else like to point out some other task somewhat related 
> > > >and have me do it? :-)
> > > >
> > > >> > Before you even try switching kernels, first implement a process
> > > >> > checkpoint/restart. The process must be resumed after a boot
> > > >> > using the same
> > > >> > kernel, with all I/O resumed. Now get it accepted into the kernel.
> > > >> 
> > > >> Hear, hear!  That would be a useful feature, maybe not network servers, 
> > > >> but for pure number crunching apps it would save people having to write 
> > > >> all the state saving and recovery that is needed now for long term 
> > > >> computations.
> > > >
> > > >Get a computer with hibernation support. That's just about what it is.
> > > 
> > > Bzzzt wrong anser. Hibernation stops the entire kernel. checkpoint restart
> > > stops processes, saves the entire state of the process. hibernation
> > > is just halt the processor.
> > 
> > Hibernation may not be.
> > I've just suspended to disk after the list line, pulled the power supplies,
> > taken the RAM chip out, shorted the pins to make really sure, then powered
> > back up.
> > Everything just resumed fine.
> > 
> > All I'd need to do kernel migration is a quick vi of the
> > disk file.
> > 
> > (well, almost)

> That sounds more like a memory dump to disk, and reload after power restored.
> Either that or possibly a separate power supply for RAM (something like a
> trickle discharge capacitor; I've read that some capacitors can hold a charge
> for about 3 days. Whether that would work for a large RAM or not, I have no
> idea).

	It's a suspend to disk.  Lots of Laptops can do it and my Toshiba
Tecra 8100 can do it from the BIOS if I have a magic Windows partition with
an appropriate suspend file in it (which would be unencrypted, which would
be unacceptable - so I had to look for a Linux solution for the suspend
to disk problem).

	Check out the swsusp project up at Source Forge
<http://sourceforge.net/projects/swsusp/>.  It allows me to suspend
into the swap space by hitting Alt-SysRQ-D.  Great for changing
batteries on laptops (and, no, normal suspend does not survive a battery
change) but also REALLY GREAT for forensic security analysis of compromised
systems.  I hit the console of a compromised system and hit Alt-SysRq-D
and it flushs the dirty buffers, dumps memory to swap (preserving all
my "volatiles") and the shuts down.  I can snapshot the hard drive and
then restart the system where it left off for live running analysis.  If
that gets screwed up, I can restore the image again and restart again from
the same spot again.  I've also got all the memory and CPU state in that
disk image for "in-vitro" analysis by tools like Weitse's "The Coroner's
Toolkit".

	But that doesn't solve ANY of the problems with changing the kernel
itself.  Suspending and restoring the system is the easy part (and swsusp
still has some problems restoring X Windows).  Restoring a system to
a different kernel is orders of magnitude worse, if not down right
impossible for all the reasons given over internal structures and
interfaces.

	I would LOVE to have something like swsusp in the main line kernel,
however, just so I didn't have to convince IT departments to apply this
custom kernel patch to their production systems BEFORE they get their butts
kicked by some snott nosed script kiddie.  :-/

> -------------------------------------------------------------------------
> Jesse I Pollard, II
> Email: pollard@navo.hpc.mil
> 
> Any opinions expressed are solely my own.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

-- 
 Michael H. Warfield    |  (770) 985-6132   |  mhw@WittsEnd.com
  (The Mad Wizard)      |  (678) 463-0932   |  http://www.wittsend.com/mhw/
  NIC whois:  MHW9      |  An optimist believes we live in the best of all
 PGP Key: 0xDF1DD471    |  possible worlds.  A pessimist is sure of it!

swsusp [was Re: Switching Kernels without Rebooting?]

[Pavel Machek]

Hi!

> > That sounds more like a memory dump to disk, and reload after power restored.
> > Either that or possibly a separate power supply for RAM (something like a
> > trickle discharge capacitor; I've read that some capacitors can hold a charge
> > for about 3 days. Whether that would work for a large RAM or not, I have no
> > idea).
> 
> 	It's a suspend to disk.  Lots of Laptops can do it and my Toshiba
> Tecra 8100 can do it from the BIOS if I have a magic Windows partition with
> an appropriate suspend file in it (which would be unencrypted, which would
> be unacceptable - so I had to look for a Linux solution for the suspend
> to disk problem).
> 
> 	Check out the swsusp project up at Source Forge
> <http://sourceforge.net/projects/swsusp/>.  It allows me to suspend
> into the swap space by hitting Alt-SysRQ-D.  Great for changing
> batteries on laptops (and, no, normal suspend does not survive a battery
> change) but also REALLY GREAT for forensic security analysis of compromised
> systems.  I hit the console of a compromised system and hit Alt-SysRq-D
> and it flushs the dirty buffers, dumps memory to swap (preserving all
> my "volatiles") and the shuts down.  I can snapshot the hard drive and
> then restart the system where it left off for live running analysis.  If
> that gets screwed up, I can restore the image again and restart again from
> the same spot again.  I've also got all the memory and CPU state in that
> disk image for "in-vitro" analysis by tools like Weitse's "The Coroner's
> Toolkit".
> 
> 	But that doesn't solve ANY of the problems with changing the kernel
> itself.  Suspending and restoring the system is the easy part (and swsusp
> still has some problems restoring X Windows).  Restoring a system to
> a different kernel is orders of magnitude worse, if not down right
> impossible for all the reasons given over internal structures and
> interfaces.
> 
> 	I would LOVE to have something like swsusp in the main line kernel,
> however, just so I didn't have to convince IT departments to apply this
> custom kernel patch to their production systems BEFORE they get their butts
> kicked by some snott nosed script kiddie.  :-/

Patience. swsusp is needed for ACPI S4 support. And I guess ACPI S4 is
good enough reason to push it to Linus.
								Pavel
-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: Switching Kernels without Rebooting?

[Pavel Machek]

Hi!

> Would anyone else like to point out some other task somewhat related 
> and have me do it? :-)

Ummm, I need someone to cook me lunch tommorow ;-).

> > > Before you even try switching kernels, first implement a process
> > > checkpoint/restart. The process must be resumed after a boot
> > > using the same
> > > kernel, with all I/O resumed. Now get it accepted into the kernel.
> > 
> > Hear, hear!  That would be a useful feature, maybe not network servers, 
> > but for pure number crunching apps it would save people having to write 
> > all the state saving and recovery that is needed now for long term 
> > computations.
> 
> Get a computer with hibernation support. That's just about what it
> is.

No. Hibernation can be done (see sw_susp patches). This is per-process
-> different. And you could implement that "live upgrade" similar
way. Checkpoint all. Reboot with new kernel. Restart all. That's close
enough to live upgrade.

(Ouch, what are you going to do with programs that behave differently
on different kernel releases? What if you have X using some kernel
driver that goes away in new release?)

-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: Switching Kernels without Rebooting?

[Wilfried Weissmann]

Torrey Hoffman wrote:
> 
> Jesse Pollard wrote:
> 
> [why switching kernels is very hard, and...]
> 
> > Before you even try switching kernels, first implement a process
> > checkpoint/restart. The process must be resumed after a boot
> > using the same
> > kernel, with all I/O resumed. Now get it accepted into the kernel.
> 
> Hear, hear!  That would be a useful feature, maybe not network servers,
> but for pure number crunching apps it would save people having to write
> all the state saving and recovery that is needed now for long term
> computations.

There is a checkpointing and resumeing lib at
ftp://gutemine.geo.uni-koeln.de/pub/chkpt/
I am not sure if it has been ported to linux yet, but it might be worth
a look.

> 
> For bonus points, make it work for clusters to synchronously save and
> restore state for the apps running on all the nodes at once...
> 
> Torrey

bye,
Wilfried