XCP - FYI - An easy way to wedge (and fix) a Cloud

["dwight at supercomputer.org" <dwight@supercomputer.org>]

This is mostly FYI. I know someone else is going to run into this.

It turns out that it's real easy to wedge an entire Cloud with
the default configurations in XCP 0.1.1. We saw this recently
with our Development Cloud.

It turns out that /var/log had filled up the root filesystem on
the master.  500M+ worth of messages in there. After I tracked 
down the problem, and freed this space up, everything started  
working again.

When this happens, various things either fail mysteriously 
(including a failure of the slaves and master to reboot),
xsconsole wedging (on the master and slaves), and OpenXenCenter 
not being able to connect, and at best messages that aren't
helpful.

I would recommend, at the very least, that compression of the
logs in logrotate.conf be turned on. I'd also strongly  recommend
that this be the default in release 0.5.

Myself, I've taken this further, by putting logrotate into the
hourly cronjob. And we're going to change our automatic 
installation scripts to put /var on a separate, large disk 
volume, not on the root filesystem.

Having /var separate from the root filesystem is generally
a wise move for servers, so that /var doesn't impact the root.

I'd also add that having grub available would've been helpful.

   -dwight-