ESXi and ideal hardware spec

ESXi and ideal hardware spec

[Stephen Perkins]

Hi all,

As part of the ideal hardware spec thread that has been going around, I
wanted to ask comments on virtualizing the OS.  Since my deployment would
eventually be co-located at a remote "lights out" environment,  I require
that I have support for:

	out of band remote access on dedicated port
		- Dell DRAC
		- HP iLO
		- SuperMicro IPMI with KVM
		- ASUS with Remote Management Chip
		- VMWare on the actual virtual machine instances.

	Remote monitoring of physical hardware
		- Dell OpenManage
		- HP Insight Manager
		- SuperMicro SuperDoctor III
		- VMWare

	Integration with a Service Monitor solution
		- Nagios
		- Zabbix
		- VMWare

One thing that integrates well with all of these is the free VMWare ESXi
Hypervisor.

Are there any thoughts on placing all Ceph nodes as single virtual machines
running on top of an ESXi hypervisor?   What I mean by this is that each
brick runs ESXi and then only runs one virtual machine.  

Are there any advantages of running the MON and MDS servers as independent
virtual machines on the same physical brick as an OSD virtual machine
(rather than just running the processes)?  Multi-port Ethernet systems can
segregate traffic between the instances...

- Steve

Re: ESXi and ideal hardware spec

[Josh Durgin]

On 08/27/2012 11:17 AM, Stephen Perkins wrote:
> Are there any thoughts on placing all Ceph nodes as single virtual machines
> running on top of an ESXi hypervisor?   What I mean by this is that each
> brick runs ESXi and then only runs one virtual machine.

This sounds like it's going to have a bunch of overhead, but if
the performance under ESXi is acceptable to you, it might make sense.
I'm not aware of anyone testing OSDs on ESXi, so you could run into
performance issues that don't appear on real hardware.

> Are there any advantages of running the MON and MDS servers as independent
> virtual machines on the same physical brick as an OSD virtual machine
> (rather than just running the processes)?  Multi-port Ethernet systems can
> segregate traffic between the instances...

If your kernel/glibc doesn't support syncfs(2), running the monitor in
a virtual machine can be beneficial since it won't affect the OSD when
calling sync(2). I don't see many other advantages, since the monitors
are pretty light-weight.

It might be a disadvantage to run the MDS in vm, since it will make a
large chunk of memory unusable by the OSD, even after the MDS stops
using some of it, and vice versa. You may prefer to be able to control
the amount of memory available to each independently, but I don't see
much advantage there. The OSD will require more memory during recovery
than during normal usage, so running the MDS and OSD as processes lets
them use the memory the other doesn't need at the moment. The MDS
effectively serves as a cache, so having lots of memory for it is
important for metadata-heavy workloads.

Josh