Re: Switching Kernels without Rebooting?

Re: Switching Kernels without Rebooting?

[C. Slater]

>
> >     - Replace all saved structures
>
> > what if the layout of these changes as it often does?
>
> You would want to convert all structures into a neutral encoding scheme
> that would support transferring structures across versions. BER comes to
> mind, as it provides for an easy way to ignore stuff you don't understand
> and support multiple versions of the same object in a single encoding.
>
> However, this would be a truly massive task. And the big challenge would
be
> what to do when an older kernel doesn't understand something essential. It
> could be simplified significantly by supporting live replacement only of
> kernels of the same version, but this seems to defeat much of the purpose.
>
> DS

I don't think that it would be possible to switch kernels when one was not
properly set up to do it, if thats what you mean. You could only switch
between kernels that have been compiled to support live switching.

I do see you'r point with the datastructures changeing. We would need to use
some format that all properly setup kernels could understand, then we would
only need to write enough to convert the structs to the middle format and
back when they change. I am not familer with BER, but if it is suitable, it
may help.

Are you saying that swaping the kernels out altogether would be a massive
task, or that saveing/restoring the datastructures would be a massive task.

  Colin

Re: Switching Kernels without Rebooting?

[FORT David]

C. Slater wrote:

>
>>>    - Replace all saved structures
>>>
>>>what if the layout of these changes as it often does?
>>>
>>You would want to convert all structures into a neutral encoding scheme
>>that would support transferring structures across versions. BER comes to
>>mind, as it provides for an easy way to ignore stuff you don't understand
>>and support multiple versions of the same object in a single encoding.
>>
>>However, this would be a truly massive task. And the big challenge would
>>
>be
>
>>what to do when an older kernel doesn't understand something essential. It
>>could be simplified significantly by supporting live replacement only of
>>kernels of the same version, but this seems to defeat much of the purpose.
>>
>>DS
>>
>
>I don't think that it would be possible to switch kernels when one was not
>properly set up to do it, if thats what you mean. You could only switch
>between kernels that have been compiled to support live switching.
>
>I do see you'r point with the datastructures changeing. We would need to use
>some format that all properly setup kernels could understand, then we would
>only need to write enough to convert the structs to the middle format and
>back when they change. I am not familer with BER, but if it is suitable, it
>may help.
>
>Are you saying that swaping the kernels out altogether would be a massive
>task, or that saveing/restoring the datastructures would be a massive task.
>
>  Colin
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
I remembered that this thread was longly discussed 1 or 2 years ago on 
linux-future
and came to no conclusive end.

-- 
 HomePage: http://www.enlightened-popo.net  
-- This was sent by Djinn running Linux 2.4.5 -- 

Re: Switching Kernels without Rebooting?

[Helge Hafting]

"C. Slater" wrote:

> I don't think that it would be possible to switch kernels when one was not
> properly set up to do it, if thats what you mean. You could only switch
> between kernels that have been compiled to support live switching.
> 
Sure.  
> I do see you'r point with the datastructures changeing. We would need to use
> some format that all properly setup kernels could understand, 

That seems completely out of question.  The structures a 2.4.7 
kernel understands might be insufficient to express the setup 
a future 2.6.9 kernel is using to do its stuff better.  (And vice
versa, if future kernels drop a 2.4.7 feature deemed obsolete.
But what if that feature is in use when you decide to upgrade?) 
You can easily deal with simple stuff like struct
rearrangement and type conversions, but what to do when whole data
structures
change completely?  

Example: something changes from two linked lists representation to a
single tree or 4 hashtables.  You'll have a very hard time inventing
a generic data format to deal with that kind of changes.  It might
happen.  Look at differences in 2.2 and 2.4 VM with the big pagecache
change in early 2.3.  And the dentry cache that suddenly appeared.

And of course the rules change too, from time to time.  
Many releases have a list of "active pages".  what kind exactly is that?
The rules may change, what to do if the new kernel don't allow
one particular kind of page on that list, but the old running kernel
have a bunch?

This was jsut some made-up examples, I guess you'll run into a ton
of such issues.  New releases aren't simply fixes and tweaks, there
are frequent design changes.

> Are you saying that swaping the kernels out altogether would be a massive
> task, or that saveing/restoring the datastructures would be a massive task.

All you need to swap kernel images is memory.  Swapping structures
can't be done in a generic way, you'll need code that convert the
structures of one particular kernel release to those of a
particular other kernel.  And I don't think you'll have the usual
kernel developers do that.

A "long-term uptime" distro might do this kind of work for a few
selected kernels, but I cannot imagine it happen for the regular
ones.

Helge Hafting

Re: Switching Kernels without Rebooting?

[C. Slater]

Unless we find some other way to do it, i think we will have to limit this
to only switching between kernels with the same minor version. We probably
would not beable to swap between 2.4 and 2.6 anyways, though it depends on
what changes are made.

  Colin

----- Original Message -----
From: "Helge Hafting" <helgehaf@idb.hist.no>
To: "C. Slater" <cslater@wcnet.org>
Cc: <linux-kernel@vger.kernel.org>
Sent: Wednesday, July 11, 2001 5:10 AM
Subject: Re: Switching Kernels without Rebooting?


> "C. Slater" wrote:
>
> > I don't think that it would be possible to switch kernels when one was
not
> > properly set up to do it, if thats what you mean. You could only switch
> > between kernels that have been compiled to support live switching.
> >
> Sure.
> > I do see you'r point with the datastructures changeing. We would need to
use
> > some format that all properly setup kernels could understand,
>
> That seems completely out of question.  The structures a 2.4.7
> kernel understands might be insufficient to express the setup
> a future 2.6.9 kernel is using to do its stuff better.  (And vice
> versa, if future kernels drop a 2.4.7 feature deemed obsolete.
> But what if that feature is in use when you decide to upgrade?)
> You can easily deal with simple stuff like struct
> rearrangement and type conversions, but what to do when whole data
> structures
> change completely?
>
> Example: something changes from two linked lists representation to a
> single tree or 4 hashtables.  You'll have a very hard time inventing
> a generic data format to deal with that kind of changes.  It might
> happen.  Look at differences in 2.2 and 2.4 VM with the big pagecache
> change in early 2.3.  And the dentry cache that suddenly appeared.
>
> And of course the rules change too, from time to time.
> Many releases have a list of "active pages".  what kind exactly is that?
> The rules may change, what to do if the new kernel don't allow
> one particular kind of page on that list, but the old running kernel
> have a bunch?
>
> This was jsut some made-up examples, I guess you'll run into a ton
> of such issues.  New releases aren't simply fixes and tweaks, there
> are frequent design changes.
>
> > Are you saying that swaping the kernels out altogether would be a
massive
> > task, or that saveing/restoring the datastructures would be a massive
task.
>
> All you need to swap kernel images is memory.  Swapping structures
> can't be done in a generic way, you'll need code that convert the
> structures of one particular kernel release to those of a
> particular other kernel.  And I don't think you'll have the usual
> kernel developers do that.
>
> A "long-term uptime" distro might do this kind of work for a few
> selected kernels, but I cannot imagine it happen for the regular
> ones.
>
> Helge Hafting

Re: Switching Kernels without Rebooting? [MOSIX]

[Carlos O'Donell Jr.]

On Wed, Jul 11, 2001 at 11:41:54AM -0400, C. Slater wrote:
> Unless we find some other way to do it, i think we will have to limit this
> to only switching between kernels with the same minor version. We probably
> would not beable to swap between 2.4 and 2.6 anyways, though it depends on
> what changes are made.
> 
>   Colin

Just thinking...

If you had enough money, and were inclined enough, one could setup the
following system:

- 2 Boxes, Running MOSIX (similar processors).

a. Start processes on Box 1.
b. Migrate processes to Box 2.

If the need to upgrade the kernel arises, you can migrate the processes
back to Box 1. Upgrade the kenrel on Box 2, recompile MOSIX.
If the first two digits of the MOSIX version are the same, you can migrate
the processes back to Box 2 (now running the latest kernel).

The stubs inplace for your process will run local kernel functions that
are not specifically host dependant, thus taking advantage of the newer
kernel features, and possibly newer hardware on Box 2, at an application
level.

Obviously, Box 1 could be smaller and less expensive.
Take note that if Box 1 were to fail, you process would die, since the
kernel stubs need to be in place on the original machine.

There are many cons to this system, but I will not ruin the decidely
happy mood of this linux-future-istic conversation ;)

Cheers,
Carlos O'Donell Jr.
-------------------------
Baldric Project
http://www.baldric.uwo.ca
-------------------------

Re: Switching Kernels without Rebooting?

[Helge Hafting]

"C. Slater" wrote:
> 
> Unless we find some other way to do it, i think we will have to limit this
> to only switching between kernels with the same minor version. We probably
> would not beable to swap between 2.4 and 2.6 anyways, though it depends on
> what changes are made.

Minor versions won't help you.  Different minor versions try to stay
interface-compatible with each other.  But data structures not
exposed to interfaces can still be rewritten completely.

Lots of nice ideas and implementations have piled up for 2.5.  Those
who proves immensely successfull in 2.5 may get backported to 2.4
once they get enough testing.  Try reading a few months worth of
kernel patches and you'll see that things change in stable kernels
too.

Helge Hafting

Re: Switching Kernels without Rebooting?

[Paul Jakma]

On Wed, 11 Jul 2001, Helge Hafting wrote:

> That seems completely out of question.  The structures a 2.4.7
> kernel understands might be insufficient to express the setup
> a future 2.6.9 kernel is using to do its stuff better.

however, it might be handy if say you needed to upgrade a stable
kernel due to a bug fix or security update.

no?

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org
PGP5 key: http://www.clubi.ie/jakma/publickey.txt
-------------------------------------------
Fortune:
I found Rome a city of bricks and left it a city of marble.
		-- Augustus Caesar

Re: Switching Kernels without Rebooting?

[Rik van Riel]

On Wed, 11 Jul 2001, Paul Jakma wrote:
> On Wed, 11 Jul 2001, Helge Hafting wrote:
>
> > That seems completely out of question.  The structures a 2.4.7
> > kernel understands might be insufficient to express the setup
> > a future 2.6.9 kernel is using to do its stuff better.
>
> however, it might be handy if say you needed to upgrade a stable
> kernel due to a bug fix or security update.

One thing which always surprises me in this discussion
(it comes up about once a year, it seems) is that
nobody participating in this discussion ever starts
writing any code for it.

Is this a feature which is only wanted by people who
don't want to code, or is this just a signal that the
amount of trouble involved just isn't worth it?

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Switching Kernels without Rebooting?

[C. Slater]

Does it come up often? Well, I have a sourceforge project setup and am
currently only waiting on finalizing how it's going to be done. So we have
about proved the first possibility wrong, and if you ever hear anything else
about this in a while, we will have proved the second wrong too. Soo, while
we are at it, ill say, that if anyone wants to help with it, email me. We
especialy need people that either have ideas on how to do this or have a
good knowledge of the kernel, mainly memory, processes, and initilization.

  Colin

----- Original Message -----
From: Rik van Riel <riel@conectiva.com.br>
To: Paul Jakma <paul@clubi.ie>
Cc: Helge Hafting <helgehaf@idb.hist.no>; C. Slater <cslater@wcnet.org>;
<linux-kernel@vger.kernel.org>
Sent: Wednesday, July 11, 2001 06:14 PM
Subject: Re: Switching Kernels without Rebooting?


> On Wed, 11 Jul 2001, Paul Jakma wrote:
> > On Wed, 11 Jul 2001, Helge Hafting wrote:
> >
> > > That seems completely out of question.  The structures a 2.4.7
> > > kernel understands might be insufficient to express the setup
> > > a future 2.6.9 kernel is using to do its stuff better.
> >
> > however, it might be handy if say you needed to upgrade a stable
> > kernel due to a bug fix or security update.
>
> One thing which always surprises me in this discussion
> (it comes up about once a year, it seems) is that
> nobody participating in this discussion ever starts
> writing any code for it.
>
> Is this a feature which is only wanted by people who
> don't want to code, or is this just a signal that the
> amount of trouble involved just isn't worth it?
>
> Rik
> --
> Virtual memory is like a game you can't win;
> However, without VM there's truly nothing to lose...
>
> http://www.surriel.com/ http://distro.conectiva.com/
>
> Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Switching Kernels without Rebooting?

[Andreas Dilger]

Colin Slater writes:
> Does it come up often? Well, I have a sourceforge project setup and am
> currently only waiting on finalizing how it's going to be done. So we have
> about proved the first possibility wrong, and if you ever hear anything else
> about this in a while, we will have proved the second wrong too. Soo, while
> we are at it, ill say, that if anyone wants to help with it, email me. We
> especialy need people that either have ideas on how to do this or have a
> good knowledge of the kernel, mainly memory, processes, and initilization.

Not to be overly negative, I don't intend this email as an insult, but rather
as a "shed a little light" on the discussion email.  I would be _happy_ if
you actually succeed in your project, but your comments come out as follows:

a) we want this "sounds real good" feature
b) we don't know how we will do it, beyond some hand waving ideas
c) we want kernel experts who know what they are doing to help us
d) kernel experts who have replied so far (negatively) don't know what
   they are talking about, so please butt out
e) you have "started coding" by setting up a sourceforge project

Note that you are talking about a VERY difficult problem, which is
not available on 99.9% of systems out there.  Maybe on a few highly
specialized *nixes which were designed for this (Sequent or such),
and probably have extra hardware support to help along.  I'm _pretty_
sure that Solaris and AIX and HP/UX do NOT do this, and don't you think
they would want to if it were easy?  It would be easier than under
Linux from the perspective that their kernels change far less often,
and have relatively static interfaces.

The best proposal I've heard so far was to use MOSIX to do live job
migration between machines, and then upgrade the kernel like normal.
In the end, it is the jobs that are running on the kernel, and not
the kernel or the individual machine that are the most important.  One
person pointed out that there is a single point of failure in the
MOSIX "stub" machine, which doesn't help you in the end (how do you
update the kernel there?).  If you can figure a way to enhance MOSIX
to allow migrating the MOSIX "stub" processes to another machine, you
will have solved your problem in a much easier way, IMHO.

Note also that you need to look at the _specific_ reason why you want to
do live kernel upgrades, besides it "sounds real good".  If you have such
tight uptime deadlines that you can't take 5 minutes of downtime to boot
a new kernel, then you are probably using a load balancing cluster anyways
in case of hardware failure, so live kernel updates are not needed here.

Note that all real-world high-availability systems I ever worked on
still allowed for SCHEDULED maintenance downtime, but highly frowned
upon UNSCHEDULED downtime.  Even IBM's S/390 99.999% uptime numbers
exclude downtime for SCHEDULED outages, which are simply a fact of life.



Please prove everyone wrong by developing a way to do this, or even
showing a proof-of-concept (i.e. a user-space framework for translating
every kernel data structures from one kernel version to another, that
works across, say, a large fraction of the 2.2 kernel, or maybe from
2.4.0-test until 2.4.current).  It doesn't have to be in-kernel (yet).

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: Switching Kernels without Rebooting?

[C. Slater]

I will say that you are incredibly correct. Accualy rather funny.

> Not to be overly negative, I don't intend this email as an insult, but
rather
> as a "shed a little light" on the discussion email.  I would be _happy_ if
> you actually succeed in your project, but your comments come out as
follows:

> a) we want this "sounds real good" feature
 But at least it sounds good.

> b) we don't know how we will do it, beyond some hand waving ideas
We don't. We would like to change that.

> c) we want kernel experts who know what they are doing to help us
  Quite correct

> d) kernel experts who have replied so far (negatively) don't know what
>    they are talking about, so please butt out
We would like any information that they have. I hope they do not.

> e) you have "started coding" by setting up a sourceforge project
That line is hillarious to me. And you are right! I merely intended to show
that we are trying to go somewhere beyond a mailing list thread. To avoid
anything more i will say *trying* agin.

> Note that you are talking about a VERY difficult problem, which is
> not available on 99.9% of systems out there.  Maybe on a few highly
> specialized *nixes which were designed for this (Sequent or such),
> and probably have extra hardware support to help along.  I'm _pretty_
> sure that Solaris and AIX and HP/UX do NOT do this, and don't you think
> they would want to if it were easy?  It would be easier than under
> Linux from the perspective that their kernels change far less often,
> and have relatively static interfaces.
>
> The best proposal I've heard so far was to use MOSIX to do live job
> migration between machines, and then upgrade the kernel like normal.
> In the end, it is the jobs that are running on the kernel, and not
> the kernel or the individual machine that are the most important.  One
> person pointed out that there is a single point of failure in the
> MOSIX "stub" machine, which doesn't help you in the end (how do you
> update the kernel there?).  If you can figure a way to enhance MOSIX
> to allow migrating the MOSIX "stub" processes to another machine, you
> will have solved your problem in a much easier way, IMHO.

Unfortunatly I have not heard this yet. I have not been able to look at the
list
archives to see all of what has been posted there.

> Note also that you need to look at the _specific_ reason why you want to
> do live kernel upgrades, besides it "sounds real good".  If you have such
> tight uptime deadlines that you can't take 5 minutes of downtime to boot
> a new kernel, then you are probably using a load balancing cluster anyways
> in case of hardware failure, so live kernel updates are not needed here.
>
> Note that all real-world high-availability systems I ever worked on
> still allowed for SCHEDULED maintenance downtime, but highly frowned
> upon UNSCHEDULED downtime.  Even IBM's S/390 99.999% uptime numbers
> exclude downtime for SCHEDULED outages, which are simply a fact of life

> Please prove everyone wrong by developing a way to do this, or even
> showing a proof-of-concept (i.e. a user-space framework for translating
> every kernel data structures from one kernel version to another, that
> works across, say, a large fraction of the 2.2 kernel, or maybe from
> 2.4.0-test until 2.4.current).  It doesn't have to be in-kernel (yet).
>
> Cheers, Andreas
> --
> Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
>                  \  would they cancel out, leaving him still hungry?"
> http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Thanks for you'r insight. Will try.

Re: Switching Kernels without Rebooting?

[Rik van Riel]

On Wed, 11 Jul 2001, C. Slater wrote:

> > a) we want this "sounds real good" feature
>  But at least it sounds good.

And nothing wrong with that. It seems an excelent
opportunity to learn lots about every part of the
kernel.

> > b) we don't know how we will do it, beyond some hand waving ideas
> We don't. We would like to change that.
>
> > c) we want kernel experts who know what they are doing to help us
>   Quite correct

I guess there are two things to do here:

(1) analyse the general idea of what you want to achieve,
    breaking it down in sub-goals which may be achievable

(2) learn about how the kernel works, you may want to go to

	http://kernelnewbies.org/

I won't have time to put in a project as huge and difficult
as upgrading the kernel "live", but I'll be around to try
and teach people about how the kernel works.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Switching Kernels without Rebooting?

[Albert D. Cahalan]

Rik van Riel writes:

> I won't have time to put in a project as huge and difficult
> as upgrading the kernel "live", but I'll be around to try
> and teach people about how the kernel works.

I think I see a business opportunity here.

Live upgrades require data structure conversion and other horrors.
You can't just write the code and expect it to maintain itself.
You'd need to rewrite half of it every time, for every patch level.

The 24x7 places might be willing to pay somebody to do this.
It's consulting work really. The customer says "I want to go
from 2.4.8 to 2.4.12", you say "OK, $320405 please.", and you
make a custom upgrade procedure for them.

Re: Switching Kernels without Rebooting?

[Mike Borrelli]

How often would a company that demands 24x7 uptime /want/ to upgrade their 
kernel?  It seems to me that when the choice been decided to take that 
kind of a step in a production environment, that someone has done lots of 
tests with the new target kernel, so that even if they don't have the 
extra hardware to bring up another server in parallel, the most downtime 
that would be suffered would be the time it takes to do two boots (boot 
the new kernel, find out it doesn't work, reboot the old one.)

Not to discourage anyone, but is this really necessary, or is it something 
to be worked on just to say that it can be done?

Just a random comment from someone who knows very little.

Regards,
Mike

On Thu Jul 12 12:23:31 2001 Albert D. Cahalan said...
> Rik van Riel writes:
> 
> > I won't have time to put in a project as huge and difficult
> > as upgrading the kernel "live", but I'll be around to try
> > and teach people about how the kernel works.
> 
> I think I see a business opportunity here.
> 
> Live upgrades require data structure conversion and other horrors.
> You can't just write the code and expect it to maintain itself.
> You'd need to rewrite half of it every time, for every patch level.
> 
> The 24x7 places might be willing to pay somebody to do this.
> It's consulting work really. The customer says "I want to go
> from 2.4.8 to 2.4.12", you say "OK, $320405 please.", and you
> make a custom upgrade procedure for them.
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Switching Kernels without Rebooting?

[Rik van Riel]

On Thu, 12 Jul 2001, Albert D. Cahalan wrote:

> I think I see a business opportunity here.

	[snip technically risky idea]

> The 24x7 places might be willing to pay somebody to do this.

Unlikely. They need hardware redundancy anyway, so they'll
just upgrade their cluster node-by-node, without doing
risky and potentially data-corrupting things like live
kernel upgrades.

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Switching Kernels without Rebooting?

[Pau Aliagas]

On Thu, 12 Jul 2001, Rik van Riel wrote:

> On Thu, 12 Jul 2001, Albert D. Cahalan wrote:
>
> > I think I see a business opportunity here.
>
> 	[snip technically risky idea]
>
> > The 24x7 places might be willing to pay somebody to do this.
>
> Unlikely. They need hardware redundancy anyway, so they'll
> just upgrade their cluster node-by-node, without doing
> risky and potentially data-corrupting things like live
> kernel upgrades.

I see business in a different way: instead of ISP or ASP you provide a
backup cluster node where you can migrate your processes before rebooting.
Everything keeps on working, no magic involved.

So we can invent the CNP (Cluster Node Provider)

Pau

Re: Switching Kernels without Rebooting?

[Chris Friesen]

"Albert D. Cahalan" wrote:

> The 24x7 places might be willing to pay somebody to do this.
> It's consulting work really. The customer says "I want to go
> from 2.4.8 to 2.4.12", you say "OK, $320405 please.", and you
> make a custom upgrade procedure for them.

Speaking as someone who is working on what will eventually be a five 9's project
based on linux, there is almost zero chance that we would make use of something
like this.  Applications and kernels are tested together and verified together,
and the likelihood of changing either one and not the other one is very low (and
in fact they are shipped together as a single image).

We have hardware redundancy, and upgrades are controlled by the application,
since it knows exactly what state must be transferred and what the differences
are between versions.  After all the state has been transferred we then do an IP
takeover so that the rest of the system knows to talk to the new side.  At this
point we can test the new side for a while.  If we're satisfied with how its
performing, we can then take down the inactive side and upgrade it and then
bring it back into sync with the active side.  If we don't like it, we can
always abort and switch back to the old version.

-- 
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: cfriesen@nortelnetworks.com

Re: Switching Kernels without Rebooting?

[Ralf Baechle]

On Wed, Jul 11, 2001 at 05:44:45PM -0600, Andreas Dilger wrote:

> The best proposal I've heard so far was to use MOSIX to do live job
> migration between machines, and then upgrade the kernel like normal.
> In the end, it is the jobs that are running on the kernel, and not
> the kernel or the individual machine that are the most important.  One
> person pointed out that there is a single point of failure in the
> MOSIX "stub" machine, which doesn't help you in the end (how do you
> update the kernel there?).  If you can figure a way to enhance MOSIX
> to allow migrating the MOSIX "stub" processes to another machine, you
> will have solved your problem in a much easier way, IMHO.

Virtual machines a la VM are also nice for this.  Build a HA cluster from
two VMs, then upgrade one after another.  All that's required is HA stuff
as it already is available.

  Ralf

Re: Switching Kernels without Rebooting?

[Rik van Riel]

On Wed, 11 Jul 2001, C. Slater wrote:

> Does it come up often? Well, I have a sourceforge project setup and am
> currently only waiting on finalizing how it's going to be done.

I hope you have fun waiting.

If you're really serious about this feature, however,
you may want to start looking into the technical
details behind your wish to get an idea of exactly
how much work it would be to implement this feature.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

RE: Switching Kernels without Rebooting?

[David Schwartz]

> One thing which always surprises me in this discussion
> (it comes up about once a year, it seems) is that
> nobody participating in this discussion ever starts
> writing any code for it.

> Is this a feature which is only wanted by people who
> don't want to code, or is this just a signal that the
> amount of trouble involved just isn't worth it?

> Rik
> --

	Doesn't it make sense to decide on a feature set and method of
implementation _before_ you begin coding? Or does it make sense to just
start coding something that might never work or do what anybody wants?

	When you decide to implement something, do you usually code before you
decide exactly what it is you're trying to implement and whether anybody
wants it? I certainly don't.

	This isn't a very good example because this a rather bad idea overall. But
if you think it's stupid and will never work, just say that. Kill with legal
blows, especially when you're right.

	DS

Re: Switching Kernels without Rebooting?

[Kai Henningsen]

riel@conectiva.com.br (Rik van Riel)  wrote on 11.07.01 in <Pine.LNX.4.33L.0107111913010.9899-100000@imladris.rielhome.conectiva>:

> One thing which always surprises me in this discussion
> (it comes up about once a year, it seems) is that
> nobody participating in this discussion ever starts
> writing any code for it.
>
> Is this a feature which is only wanted by people who
> don't want to code, or is this just a signal that the
> amount of trouble involved just isn't worth it?

Maybe it's a sign that the people who *would* be able to contribute have  
all looked at the problem already (surely most people are annoyed how a  
reboot interrupts everything), and have already concluded for themselves  
that it's not possible with reasonable effort ... but there is a steady  
influx of new people who don't understand enough of the problem and have  
to ask.

What I'd *really* like (but don't see how to get there) would be a "save  
system state, shutdown, change kernel and/or hardware, reboot, restore  
state" system (where state is like "I'm logged in on this console, in this  
current directory, and under X I have Netscape running and this page  
displayed" but I don't care about the exact state of Squid or even if my  
ISDN line is dialled in, because those "fix themselves").

I suspect to do this right would need a means of storing per-process state  
controlled by the process (because only that process knows what needs to  
be saved, and what can easily be reconstructed - for example, open file  
descriptors to a place where we store cookies don't need to be saved, just  
routinely reopened), and then every user-visible non-transient program  
needs to implement it - and I don't see *that* happen in the next ten  
years.

But it *does* have the advantage of not needing to save kernel-internal  
state.

MfG Kai

Re: Switching Kernels without Rebooting?

[Helge Hafting]

Kai Henningsen wrote:

> What I'd *really* like (but don't see how to get there) would be a "save
> system state, shutdown, change kernel and/or hardware, reboot, restore
> state" system (where state is like "I'm logged in on this console, in this
> current directory, and under X I have Netscape running and this page
> displayed" but I don't care about the exact state of Squid or even if my
> ISDN line is dialled in, because those "fix themselves").

Consider os/2 then.  All workplace-shell aware programs is supposed to
save
state in this way.  And yes - they do start up in the same state after
reboot if you want to.  Editors come up on the page you left, filesystem
folders comes up, and so on.  

> and then every user-visible non-transient program
> needs to implement it - and I don't see *that* happen in the next ten
> years.

Consider a patch for konqueror or a few other webpage/fs-view programs
and you'll go a long way - all in userspace.

Helge Hafting

Re: Switching Kernels without Rebooting?

[Kai Henningsen]

helgehaf@idb.hist.no (Helge Hafting)  wrote on 12.07.01 in <3B4D7685.9AC1DED@idb.hist.no>:

> Kai Henningsen wrote:
>
> > What I'd *really* like (but don't see how to get there) would be a "save
> > system state, shutdown, change kernel and/or hardware, reboot, restore
> > state" system (where state is like "I'm logged in on this console, in this
> > current directory, and under X I have Netscape running and this page
> > displayed" but I don't care about the exact state of Squid or even if my
> > ISDN line is dialled in, because those "fix themselves").
>
> Consider os/2 then.  All workplace-shell aware programs is supposed to
> save
> state in this way.

The keyword is "supposed". Because I remember from my OS/2 days that most  
didn't.

OTOH, Borland's DOS IDE does. It's a mixed bag.

>  And yes - they do start up in the same state after
> reboot if you want to.  Editors come up on the page you left, filesystem
> folders comes up, and so on.

Most programs from IBM got it right, most others didn't, as far as I can  
recall.

> > and then every user-visible non-transient program
> > needs to implement it - and I don't see *that* happen in the next ten
> > years.
>
> Consider a patch for konqueror or a few other webpage/fs-view programs
> and you'll go a long way - all in userspace.

Well, Netscape *can* sort of do it (for one window).

But how do I make it happen for bash? login? xdm? Amd so on ... anyway, I  
simply don't have the time for such a project. I'm spread too thin as it  
is.

MfG Kai

Re: Switching Kernels without Rebooting?

[Hua Zhong]

-> kaih@khms.westfalen.de (Kai Henningsen)  wrote:
> riel@conectiva.com.br (Rik van Riel)  wrote on 11.07.01 in <Pine.LNX.4.33L.0107111913010.9899-100000@imladris.rielhome.conectiva>: 
> I suspect to do this right would need a means of storing per-process state  
> controlled by the process (because only that process knows what needs to  
> be saved, and what can easily be reconstructed - for example, open file  
> descriptors to a place where we store cookies don't need to be saved, just  
> routinely reopened), and then every user-visible non-transient program  
> needs to implement it - and I don't see *that* happen in the next ten  
> years.

This would be the easiest way to do in the sense that application authors take care of their own stuff, and kernel developpers only need to define rules/interfaces.

One scheme is that we can define a new signal number (e.g., SIGCKPT).  When we send the signal to the process, it checkpoints itself (saves everything it needs for a restart).  Then we define another signal (e.e., SIGRSUM).  When we send the signal to it, it then knows that it should resume from the last checkpointed point.  This is user-level checkpoint/restart, and there are already certain packages available (Condor, libckpt, etc).

If we want total transparency (i.e., applications don't need to be aware and everything is taken care of by the kernel), then the kernel needs substantial changes (I've written a kernel module to do this).

swsusp again [was Re: Switching Kernels without Rebooting?]

[Pavel Machek]

Hi!

> What I'd *really* like (but don't see how to get there) would be a "save  
> system state, shutdown, change kernel and/or hardware, reboot, restore  
> state" system (where state is like "I'm logged in on this console, in this  
> current directory, and under X I have Netscape running and this page  
> displayed" but I don't care about the exact state of Squid or even if my  
> ISDN line is dialled in, because those "fix themselves").

Suspend-to-disk, change hardware, restore-from-disk, load neccessary
modules seems quite easy to do with swsusp. It is very different from
suspend-to-disk, change kernel, restore-from-disk (which is guaranteed
to kill you if kernel changes size).

								Pavel
-- 
I'm pavel@ucw.cz. "In my country we have almost anarchy and I don't care."
Panos Katsaloulis describing me w.r.t. patents at discuss@linmodems.org

Re: swsusp again [was Re: Switching Kernels without Rebooting?]

[Alan Cox]

> Suspend-to-disk, change hardware, restore-from-disk, load neccessary
> modules seems quite easy to do with swsusp. It is very different from
> suspend-to-disk, change kernel, restore-from-disk (which is guaranteed
> to kill you if kernel changes size).

It works for most hw changes. I've used swsusp to replace a burned out 3c509
without rebooting 8)

Re: Switching Kernels without Rebooting?

[Kip Macy]

In the future when Linux is more heavily used at the enterprise level
there will likely be upgrade/revert modules to allow such a transition to
take place.

			-Kip

On Wed, 11 Jul 2001, Paul Jakma wrote:

> On Wed, 11 Jul 2001, Helge Hafting wrote:
> 
> > That seems completely out of question.  The structures a 2.4.7
> > kernel understands might be insufficient to express the setup
> > a future 2.6.9 kernel is using to do its stuff better.
> 
> however, it might be handy if say you needed to upgrade a stable
> kernel due to a bug fix or security update.
> 
> no?
> 
> regards,
> -- 
> Paul Jakma	paul@clubi.ie	paul@jakma.org
> PGP5 key: http://www.clubi.ie/jakma/publickey.txt
> -------------------------------------------
> Fortune:
> I found Rome a city of bricks and left it a city of marble.
> 		-- Augustus Caesar
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

Re: Switching Kernels without Rebooting?

[Rik van Riel]

On Wed, 11 Jul 2001, Kip Macy wrote:

> In the future when Linux is more heavily used at the enterprise level
> there will likely be upgrade/revert modules to allow such a transition
> to take place.

Only if somebody takes the trouble to write them, which
isn't something I see happening in the near future.

Not only would this feature be a LOT of work, it would
(probably) also be very invasive all over the kernel.
OTOH, if the kernel was compiled with -g maybe it'd have
enough info to locate its data structures ?

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: Switching Kernels without Rebooting?

[Jesse Pollard]

On Wed, 11 Jul 2001, Kip Macy wrote:
>In the future when Linux is more heavily used at the enterprise level
>there will likely be upgrade/revert modules to allow such a transition to
>take place.

I use some of the largest UNIX supercomputers ever built (IBM SP, Cray T3E,
SV1, YMP, XMP, J90, SGI Origin). None of them can start of a new kernel from an
earlier version. There are too many things that will fail:

	Any network activity
	Active disk I/O
	Locked memory
	File modification
	File structures
	Disk structures (yes they change...)
	Clock Synchronization (SMP and cluster)
	Shared memory (SMP and cluster)
	semaphores (SMP and cluster)
	login sessions
	device status
	shared disks and distributed file systems (cluster)
	pipes

Before you even try switching kernels, first implement a process
checkpoint/restart. The process must be resumed after a boot using the same
kernel, with all I/O resumed. Now get it accepted into the kernel.

Anything else is just another name for "reboot using new kernel".
	

-- 
-------------------------------------------------------------------------
Jesse I Pollard, II
Email: jesse@cats-chateau.net

Any opinions expressed are solely my own.

Re: Switching Kernels without Rebooting?

[Hua Zhong]

-> Jesse Pollard <jesse@cats-chateau.net>  wrote:
> Before you even try switching kernels, first implement a process
> checkpoint/restart. The process must be resumed after a boot using the same
> kernel, with all I/O resumed. Now get it accepted into the kernel.
> 
> Anything else is just another name for "reboot using new kernel".

Exactly.  You may want to take a look at http://www.checkpointing.org

Re: Switching Kernels without Rebooting?

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.33.0107112310590.962-100000@fogarty.jakma.org>
By author:    Paul Jakma <paul@clubi.ie>
In newsgroup: linux.dev.kernel
>
> On Wed, 11 Jul 2001, Helge Hafting wrote:
> 
> > That seems completely out of question.  The structures a 2.4.7
> > kernel understands might be insufficient to express the setup
> > a future 2.6.9 kernel is using to do its stuff better.
> 
> however, it might be handy if say you needed to upgrade a stable
> kernel due to a bug fix or security update.
> 
> no?
> 

No.  You have no guarantee that the state or state mangler won't
propagate the bug into the new kernel, even if it has been fixed.
Since many, if not most, bug fixes or security upgrades are related to
state getting mucked up, this is a very serious thing.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt

Re: Switching Kernels without Rebooting?

[Ville Herva]

On Wed, Jul 11, 2001 at 11:12:12PM +0100, you [Paul Jakma] claimed:
> On Wed, 11 Jul 2001, Helge Hafting wrote:
> 
> > That seems completely out of question.  The structures a 2.4.7
> > kernel understands might be insufficient to express the setup
> > a future 2.6.9 kernel is using to do its stuff better.
> 
> however, it might be handy if say you needed to upgrade a stable
> kernel due to a bug fix or security update.
> 
> no?

<clueless>
In that case you might get a way with a simpler approach. Perhaps you could
just replace the changed function(s) with new ones and scan the kernel for
calls to them. Each call should then be changed to point to the new
function. This might work provided the function interfaces don't change
(which might just be true for simple maintenance bug fixes and security
fixes.) It might even be useful for kernel development.

Of course this takes complex locking and the details are propably very
thorny.

I'm not sure if this is possible, IANAKH. But AFAIK this is roughly what
MSVC6.0 edit and continue does for userspace programs. 
</clueless>


-- v --

v@iki.fi