Re: ESXi and ideal hardware spec

[Josh Durgin <josh.durgin@inktank.com>]

On 08/27/2012 11:17 AM, Stephen Perkins wrote:
> Are there any thoughts on placing all Ceph nodes as single virtual machines
> running on top of an ESXi hypervisor?   What I mean by this is that each
> brick runs ESXi and then only runs one virtual machine.

This sounds like it's going to have a bunch of overhead, but if
the performance under ESXi is acceptable to you, it might make sense.
I'm not aware of anyone testing OSDs on ESXi, so you could run into
performance issues that don't appear on real hardware.

> Are there any advantages of running the MON and MDS servers as independent
> virtual machines on the same physical brick as an OSD virtual machine
> (rather than just running the processes)?  Multi-port Ethernet systems can
> segregate traffic between the instances...

If your kernel/glibc doesn't support syncfs(2), running the monitor in
a virtual machine can be beneficial since it won't affect the OSD when
calling sync(2). I don't see many other advantages, since the monitors
are pretty light-weight.

It might be a disadvantage to run the MDS in vm, since it will make a
large chunk of memory unusable by the OSD, even after the MDS stops
using some of it, and vice versa. You may prefer to be able to control
the amount of memory available to each independently, but I don't see
much advantage there. The OSD will require more memory during recovery
than during normal usage, so running the MDS and OSD as processes lets
them use the memory the other doesn't need at the moment. The MDS
effectively serves as a cache, so having lots of memory for it is
important for metadata-heavy workloads.

Josh