[PATCH] CPU hot swap for 2.4.3 + s390 support

[PATCH] CPU hot swap for 2.4.3 + s390 support

[Anton Blanchard]

Hi,

You can find a new version of the hot swap cpu patch at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch

The version for s390 (you need to first apply the 2.4.3 kernel
patch available on the IBM s390 Linux website) is at:

http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390

Many thanks to Heiko Carstens <Heiko.Carstens@de.ibm.com> for adding
s390 support and fixing a few bugs in the initial implementation.
You should be able to attach and detach CPUs depending on workload
in your s390 Linux guest images :)

One of the advantages of this patch is that it removes cpu_logical_map()
and cpu_number_map() which people had a tendency to get wrong.

It should also be easy to support more than BITS_PER_LONG cpus
as there is no concept of online_cpu_map any more.

Anton

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Peter Rival]

Has anyone looked into memory hot swap/hot add support?  Especially with
systems with Chipkill coming out, this would be great to support...

 - Pete

Anton Blanchard wrote:

> Hi,
>
> You can find a new version of the hot swap cpu patch at:
>
> http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch
>
> The version for s390 (you need to first apply the 2.4.3 kernel
> patch available on the IBM s390 Linux website) is at:
>
> http://samba.org/~anton/patches/cpu_hotswap-2.4.3-patch-s390
>
> Many thanks to Heiko Carstens <Heiko.Carstens@de.ibm.com> for adding
> s390 support and fixing a few bugs in the initial implementation.
> You should be able to attach and detach CPUs depending on workload
> in your s390 Linux guest images :)
>
> One of the advantages of this patch is that it removes cpu_logical_map()
> and cpu_number_map() which people had a tendency to get wrong.
>
> It should also be easy to support more than BITS_PER_LONG cpus
> as there is no concept of online_cpu_map any more.
>
> Anton
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Chris Wedgwood]

On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:

    Has anyone looked into memory hot swap/hot add support? 

Adding memory probably isn't going to be too hard... but taking
existing memory off line is tricky. You have to find some way of
finding all the pages that are in use and then dealing with them
appropriately, and when some are locked or contain kernel data this
would be extremely difficult I should think.

    Especially with systems with Chipkill coming out, this would be
    great to support...

Chipkill?


  --cw

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Mitch Adair]

> Adding memory probably isn't going to be too hard... but taking
> existing memory off line is tricky. You have to find some way of
> finding all the pages that are in use and then dealing with them
> appropriately, and when some are locked or contain kernel data this
> would be extremely difficult I should think.

Wouldn't that be lot of the same issues as a "swapoff" with some portion of
that in use?  (except for the kernel data case of course...)

	M

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Chris Wedgwood]

On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:

    Wouldn't that be lot of the same issues as a "swapoff" with some
    portion of that in use?  (except for the kernel data case of
    course...)

No. Swapoff makes pages allocated to userland applications in swap
move back into main memory -- this is much easier because:

   - the pages are on disk, we _know_ the aren't locked my mlock or
     pinned for IO (kiobufs, whatever)

   - there are no kernel pages/buffers in this area, even harder than
     the above to deal with



  --cw

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Rik van Riel]

On Sun, 6 May 2001, Chris Wedgwood wrote:
> On Sat, May 05, 2001 at 11:34:16AM -0500, Mitch Adair wrote:
> 
>     Wouldn't that be lot of the same issues as a "swapoff" with some
>     portion of that in use?  (except for the kernel data case of
>     course...)
> 
> No. Swapoff makes pages allocated to userland applications in swap
> move back into main memory -- this is much easier because:
> 
>    - the pages are on disk, we _know_ the aren't locked my mlock or
>      pinned for IO (kiobufs, whatever)
> 
>    - there are no kernel pages/buffers in this area, even harder than
>      the above to deal with

Details, details ... ;)

For the "pinned for IO" case, we can simply wait until they're
unpinned, this shouldn't take too long. Mlock isn't an issue
either, just remove the page from the page tables, relocate the
page and put it back. This may in theory violate some mlock
behaviour, but in practice it'll be no worse than normal
scheduling latency.

Kernel pages/buffers are no big deal since they're accessed
through the kernel's page tables (for which the same rule
applies as what we do with the mlock()ed pages).

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Rik van Riel]

On Sat, 5 May 2001, Mitch Adair wrote:

> Wouldn't that be lot of the same issues as a "swapoff" with some
> portion of that in use?  (except for the kernel data case of
> course...)

Basically, yes.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Peter Rival]

Chris Wedgwood wrote:

> On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
>
>     Has anyone looked into memory hot swap/hot add support?
>
> Adding memory probably isn't going to be too hard... but taking
> existing memory off line is tricky. You have to find some way of
> finding all the pages that are in use and then dealing with them
> appropriately, and when some are locked or contain kernel data this
> would be extremely difficult I should think.
>

Hrmm...  I agree this is a hard problem.  I know people smarter than I have
been thinking about this type of problem at Compaq.  While I haven't talked to
them directly, my only guess would be that we'd have to hand-rewrite some page
tables after copying the page contents to a new area.  It's late Saturday and
I really haven't thought this through fully, so I'm not even sure that would
work, but it's something like how we support replicated text segments on our
GS series...don't know why it wouldn't work here.  *shrug*

>     Especially with systems with Chipkill coming out, this would be
>     great to support...
>
> Chipkill?
>

It's the IBM technology that works around bad memory by detecting single-bit
errors and removing the chip that caused it from use.  I'd think of this as a
big hammer version of that in software.  Besides, eventually you'll want to
replace the DIMM that has the bad chip, and what better way then while the
system is still running (as long as it's stable, of course ;)  I'm just
thinking out loud, so someone can correct me if I'm being loopy...

 - Pete

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Chris Wedgwood]

On Sat, May 05, 2001 at 08:08:59PM -0400, Peter Rival wrote:

    Hrmm...  I agree this is a hard problem.  I know people smarter
    than I have been thinking about this type of problem at Compaq.

It's hard with current memory allocation and management paradigms, if
we wanted to abstract things more and make (break) certain rules, I'm
sure it can me made to work -- the only thing is, we would loose
_MUCH_ speed and efficiency (and waste much more space), so much so I
doubt anyone would serious want to know about it.

We would also have to violate certain assumptions of RT applications.

    While I haven't talked to them directly, my only guess would be
    that we'd have to hand-rewrite some page tables after copying the
    page contents to a new area.

That in itself isn't too bad, except if the pages are mlocked this is
nasty, you have to block all access to the page during copy, very bad
for RT stuff.

Not only that, what if the pages themselves have kernel allocations in
them? We cannot find these (at present) let alone have _any_ idea how
to move them. I guess it could be fidged to work using the MMU if we
were allowed to _COMPLETELY_ lock the system duing the removal phase
from all interrupts and such like... seems pretty horrible to me.

    It's late Saturday and I really haven't thought this through
    fully, so I'm not even sure that would work, but it's something
    like how we support replicated text segments on our GS
    series...don't know why it wouldn't work here.  *shrug*

There have been demonstrations in the past of this sort of thing. I
think HP may have done one. Not with a commodity OS though.



Actually, I just thought of a kill, what about platforms that have
physically mapped page-tables? This makes like even harder as you have
to move them :(

    It's the IBM technology that works around bad memory by detecting
    single-bit errors and removing the chip that caused it from use.

I think Solaris claims to do this right now... no idea if it works, I
know of at least one Solaris 7 machine with a dicky memory bit and it
keeps moaning about parity corrections so I guess it doesn't lock it
out. Maybe later versions (8) do?

    I'd think of this as a big hammer version of that in software.

It's much easier to do in hardware with current OS design I should
think.

    Besides, eventually you'll want to replace the DIMM that has the
    bad chip, and what better way then while the system is still
    running (as long as it's stable, of course ;) I'm just thinking
    out loud, so someone can correct me if I'm being loopy...

Again, you could do this is hardware... have the hardware route writes
to the memory elsewhere and only take reads from the old memory until
the 'refresh' cycles have copied all the data over. Hmm, when I think
about this, doing this in the memory controller chipset seems much
easier I wonder if someone hasn't actually done it...



  --cw

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Rik van Riel]

On Sun, 6 May 2001, Chris Wedgwood wrote:
> On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
> 
>     Has anyone looked into memory hot swap/hot add support? 
> 
> Adding memory probably isn't going to be too hard... but taking
> existing memory off line is tricky. You have to find some way of
> finding all the pages that are in use and then dealing with them
> appropriately, and when some are locked or contain kernel data this
> would be extremely difficult I should think.

Actually:

1. the kernel uses virtual memory itself and accesses its
   data structures through page tables
2. reverse mapping stuff is easy (though it costs 8 bytes
   of overhead per mapped pte, probably doubling page table
   overhead)

This only leaves two issues, the first is device drivers and
the second is the question whether we'd want the overhead
needed to implement the (fairly easy) memory relocation.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Chris Wedgwood]

On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:

    This only leaves two issues, the first is device drivers and the
    second is the question whether we'd want the overhead needed to
    implement the (fairly easy) memory relocation.

How do you relocate

  -- pages which are mlocked without violating RT contraints?

  -- pages which contain kernel pointers and might be accessed from
     interrupt context?



  --cw

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Rik van Riel]

On Sun, 6 May 2001, Chris Wedgwood wrote:
> On Sat, May 05, 2001 at 11:19:09PM -0300, Rik van Riel wrote:
> 
>     This only leaves two issues, the first is device drivers and the
>     second is the question whether we'd want the overhead needed to
>     implement the (fairly easy) memory relocation.
> 
> How do you relocate
> 
>   -- pages which are mlocked without violating RT contraints?

Fuck RT constraints. Linux doesn't have infinitely small
scheduling latencies, it's easy to copy a page without
increasing the scheduling latencies much.

>   -- pages which contain kernel pointers and might be accessed from
>      interrupt context?

Block interrupts while copying a kernel page?


Remember that this is an emergency situation, so it's ok
to use "big hammer" solutions.

regards,

Rik
--
Virtual memory is like a game you can't win;
However, without VM there's truly nothing to lose...

http://www.surriel.com/		http://distro.conectiva.com/

Send all your spam to aardvark@nl.linux.org (spam digging piggy)

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[David Woodhouse]

cw@f00f.org said:
>  How do you relocate
>   -- pages which are mlocked without violating RT contraints?
>   -- pages which contain kernel pointers and might be accessed from
>      interrupt context? 

Those two are the same problem, essentially. You have to copy the page, 
then map it into the same virtual address (be that userspace or 
kernelspace) as the old one. Mark the page readonly when you start to copy 
it, and have a fault handler which immediately marks it writable and 
returns. If the source is writable by the time you've finished the copy, 
repeat.

If you have to repeat yourself more than $n times, you're probably 
experiencing livelock. At that point, do what Rik said - to hell with the 
RT constraints, disable interrupts and do the copy. At least your cache is 
warm :)

--
dwmw2

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Aaron Lehmann]

On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
> Has anyone looked into memory hot swap/hot add support?

How do you hotswap RAM? What happens to the data that was on the
removed memory module?

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Stephen Beynon]

On Sun, 6 May 2001, Aaron Lehmann wrote:

> On Sat, May 05, 2001 at 10:43:27AM -0400, Peter Rival wrote:
> > Has anyone looked into memory hot swap/hot add support?
> 
> How do you hotswap RAM? What happens to the data that was on the
> removed memory module?

Dont know about the s390 - but on some hardware you get notification when
someone presses the hotswap request button.  You would then be requiried
to move all the data on the memory elsewhere.  Then you tell the hardware
it is ok to hotswap.  On many systems this will make the little LED light
up to say the hotswap is safe :)

Stephen

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Dwayne C. Litzenberger]

[-- Attachment #1: Type: text/plain, Size: 196 bytes --]

Hey, this is cool.

How far away is the capability to "teleport" processes from one machine to
another over the network?  Think of the uptime!

-- 
Dwayne C. Litzenberger - dlitz@dlitz.net

[-- Attachment #2: Type: application/pgp-signature, Size: 240 bytes --]

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Aaron Lehmann]

On Sun, May 06, 2001 at 01:15:53AM -0600, Dwayne C. Litzenberger wrote:
> Hey, this is cool.
> 
> How far away is the capability to "teleport" processes from one machine to
> another over the network?  Think of the uptime!

http://www.mosix.org

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Ben Ford]

Dwayne C. Litzenberger wrote:

>Hey, this is cool.
>
>How far away is the capability to "teleport" processes from one machine to
>another over the network?  Think of the uptime!
>

It is here.  Look at Mosix.

-- 
I'd rather listen to Newton than to Mundie [MS flunkie who made a speech on
the evil-ness of open source]. He may have been dead for almost three
hundred years, but despite that he stinks up the room less.

Linus

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Jakob Østergaard]

On Sun, May 06, 2001 at 10:06:42AM -0700, Ben Ford wrote:
> Dwayne C. Litzenberger wrote:
> 
> >Hey, this is cool.
> >
> >How far away is the capability to "teleport" processes from one machine to
> >another over the network?  Think of the uptime!
> >
> 
> It is here.  Look at Mosix.

No.  Not for uptime.

The "responsibility" for process completion does not get delegated. A process
will always be bound to it's home-node (in mosix terms), no matter how far
it's "teleported".   If the home-node fails, the process won't know what hit
it.

There are good reasons why mosix let's processes depend on their home nodes.

This is not meant as backstabbing mosix, it's a great environment for a lot
of things.

But it's not the universal silver bullet.

-- 
................................................................
:   jakob@unthought.net   : And I see the elder races,         :
:.........................: putrid forms of man                :
:   Jakob Østergaard      : See him rise and claim the earth,  :
:        OZ9ABN           : his downfall is at hand.           :
:.........................:............{Konkhra}...............:

Re: [PATCH] CPU hot swap for 2.4.3 + s390 support

[Bruce Harada]

> > >How far away is the capability to "teleport" processes from one machine to
> > >another over the network?  Think of the uptime!
> > >
> > 
> > It is here.  Look at Mosix.
> 
> No.  Not for uptime.
> 
> The "responsibility" for process completion does not get delegated. A process
> will always be bound to it's home-node (in mosix terms), no matter how far
> it's "teleported".   If the home-node fails, the process won't know what hit
> it.
> 
> There are good reasons why mosix let's processes depend on their home nodes.
> 
> This is not meant as backstabbing mosix, it's a great environment for a lot
> of things.
> 
> But it's not the universal silver bullet.

Take a look at

http://citeseer.nj.nec.com/299905.html

for something along the lines of what you want, I think (transparent process
migration between nodes). As a bonus, it's also architecture-independent.


Bruce