Re: Distro kernel and 'virtualization server' vs. 'server that sometimes runs virtual instances' rant (was: Re: Re: [GIT PULL] Xen APIC hooks (with io_apic_ops))

[Tim Post <echo@echoreply.us>]

On Thu, 2009-05-28 at 08:03 -0400, Luke S Crawford wrote:
> Dan Magenheimer <dan.magenheimer@oracle.com> writes:
> > such as is needed for huge corporate data centers and "clouds".
> > However, the majority of users (individuals and small businesses)
> > will probably be most happy with their distro (and distro kernel)
> > as dom0 since it is convenient and familiar.

> I've been selling VPSs using Xen since 2005.  After the 
> marketing people convince the middle managers that virtualization is the 
> way to go, someone like me has to actually bang on the thing with a spanner
> or rub it with a greasy rag until it works.  

So have I, since (pre) 2.0.7. I was one of the first (and only) to offer
OpenSSI (paravirtualized) as an offering.

> I also do contracting for some of those 'large corporate data centers' of 
> which you speak.  (corporate data centers seem to be the worst in terms of 
> operational efficiency.  do you know how many Linux installations I've seen 
> where the customer pays a few hundred extra per box for integrated KVM over 
> IP functionality rather than the much cheaper and more useful serial 
> consoles?  Oy.  You expect me to tell you why your server crashed when 
> you have no console logs of the backtrace?)

They are in business to make money, which is why real system integrators
flourish and stand out from the crowd who read "linux for dummies
version (x), now including KVM!!"

You either know how Xen and Linux works or you don't. Most DC "hands and
eyes" just follow a pre-set procedure and can't be bothered to deviate
from it or handle special cases. Again, that's why we have jobs.

> But I'm getting sidetracked.   My point is that small companies need good 
> tools more than large corporations do.   The big guys can just keep 
> throwing money at the problem until their stuff mostly works.

Here we go again. Writing your own tools is not too difficult, it makes
you money using LGPL libraries that are (reasonably) self explanatory.
Xen is a tool in your toolbox. All too often many fail to realize the
difference between Xen the hypervisor and the tools provided.

> the last thing I want is all the cowboy hackery that goes into my favorite
> desktop OS to be included in my Xen Dom0.   I moved to Ubuntu on my laptop
> last year and I was amazed how easy it was.  everything just worked.  
> making new hardware work was easier than windows.  

Desktop OS? We have to draw a line here. There is desktop and server
virtualization. If you want to try xyz-distro on your desktop, use
Virtualbox. If you want to put virtual machines to work, use Xen.

What, exactly is cowboy hackery? A dom-0 that might be a little slower
if you boot it without Xen? 

> But do I want that on my Xen Dom0?  certainly not until you get that thing
> working where I can reboot the Dom0 without killing everything.

Mmmm, then work on getting xenstored into a stub domain.

> This is what I think is wrong about the default install of Xen;  it is setup 
> so that you can run your desktop in the dom0, and spin up DomUs as needed.
> It tries to be a virtualization server and a desktop at the same time,
> and it gives up stability for this. 

The only reason that you should be using Xen on a desktop is to test
stuff that you want to propagate to servers. You've already said that
you make your living as an integrator selling the use of computers that
use Xen.

Xen is meant for production, it can be used on a desktop.

> If you've ever run a Xen host and have forgotten to change the default 
> dom0-min-mem of 192MiB, you'd know most (especially x86_64) linux 
> installations are not stable under load with that much memory.

I have , and I don't forget to change it.

> In my work, people mostly use the  'I take this Linux box, I set it up,
> and I use it for three years' model.  They don't need any of the fancy 
> 'computing on demand'  -  they just want to move 16 of those crusty P3
> servers that are killing their power bill and crashing due to bad hardware
> twice a month on to a nice shiny new 8 core box with 32GiB ram and a
> warranty.   I've seen lots of people who buy ec2 instances and do the
> same thing; they leave it on all the time. (the basic ec2 instances are
> particularly unsuited to this usage, but people do it anyhow.) 

Have you even looked at / tried Eucalyptus ?

> I'm not going to say memory overcommit is never useful for anyone;
> but I can say it is never useful for me.  32GiB registered ecc ddr2
> is around $600.  That's not very many billable hours.  That's around
> half the approximate cost of an unplanned reboot of one of my servers.
> (I'm only counting money lost due to SLA and time to clean up;  if you
> count loss to reputation, it gets even worse)

I don't have this problem. I export PV guest vitals over xenbus and set
up watches on them. As for overcommitment, the first step is knowing how
much memory each domain's kernel has actually promised to running
processes. That much is already in the tree. 

> Right now, I'm using CentOS5 with the xen.org kernels, but it sure
> would be nice if there was some pared down pre-built dom0 configuration 
> available. (I personally give my Dom0 1024MiB out of 32GiB)  It could be
> based on centos, or on ttylinux, or whatever.  just something standard, small,
> and simple.  Make it good enough that people use it.  When I see a problem,
> I want fifty other guys to have seen the problem first.  

I don't want to seem combative or antagonistic .. however, if I give you
a screw driver and a wrench, I'd expect that you'd use them in your own
way. Xen is no different.

> I'm thinking about starting such a project myself once I get a few other 
> things done.  If nothing else, I can distribute kickstart files of a minimal
> dom0.

Just as many others have done with debootstrap. I know your frustrated
with dom-0 not being in mainline, we all are. However, it seems the
tools frustrate you the most. Xen gives us a solid hypervisor, solid low
level libraries and some examples on how to use them. I can't see (at
this point) why you are so seemingly disgruntled?

> RedHat is talking about doing it with KVM  -  see the Red Hat Enterprise 
> Virtualization hypervisor  - they claim you will have a KVM 'dom0'  that 
> uses only 64M ram- which seems funny to me, as my perception of KVM  has 
> always been that it was optimized to run virtual instances as needed on 
> a box that usually ran applications on the bare metal, like a desktop.

Eh, that funny thing we call "market research" influences that. People
want easy desktop virtualization. Desktop virtualization is
_most_decidedly_not_ IAAS. There will _always_ be a market for people
who can make tools (or modify the existing ones) to suit some need.

I agree with some of what you have to say, I always appreciate a rant
and I do not mean to seem unfriendly .. however, I also fail to see the
basis?

Maybe I missed something, entirely possible.

Cheers,
--Tim