Re: Scaling noise

[William Lee Irwin III <wli@holomorphy.com>]

On Mer, 2003-09-03 at 19:15, William Lee Irwin III wrote:
>> Independent operating system instances running under a hypervisor don't
>> qualify as a cache-coherent cluster that I can tell; it's merely dynamic
>> partitioning, which is great, but nothing to do with clustering or SMP.

On Wed, Sep 03, 2003 at 07:32:12PM +0100, Alan Cox wrote:
> Now add a clusterfs and tell me the difference, other than there being a
> lot less sharing going on...

The sharing matters; e.g. libc and other massively shared bits are
replicated in memory once per instance, which increases memory and
cache footprint(s). A number of other consequences of the sharing loss:

The number of systems to manage proliferates.

Pagecache access suddenly involves cross-instance communication instead
of swift memory access and function calls, with potentially enormous
invalidation latencies.

Userspace IPC goes from shared memory and pipes and sockets inside
a single instance (which are just memory copies) to cross-instance
data traffic, which involves slinging memory around through the
hypervisor's interface, which is slower.

The limited size of a single instance bounds the size of individual
applications, which at various times would like to have larger memory
footprints or consume more cpu time than fits in a single instance.
i.e. something resembling external fragmentation of system resources.

Process migration is confined to within a single instance without
some very ugly bits; things such as forking servers and dynamic task
creation algorithms like thread pools fall apart here.

There's suddenly competition for and a need for dynamic shifting around
of resources not shared across instances, like private disk space and
devices, shares of cpu, IP numbers and other system identifiers, and
even such things as RAM and virtual cpus.

AFAICT this raises more issues than it addresses. Not that the issues
aren't worth addressing, but there's a lot more to do than Larry
saying "I think this is a good idea" before expecting anyone to even
think it's worth thinking about.


-- wli