The idea is borrowed from Linux kernel: if the vCPU is just scheduled 
out and put to run-queue, it's likely cache-hot on its current pCPU, and 
it may be scheduled in in a short period of time; however, if vCPU is 
migrated to another pCPU, it need to re-warm the cache - that's the 
meaning of migration cost.

The patch introduces an option migration_cost to depress too aggressive 
vCPU migration (actually we really see migration frequency is very high 
most of the time.), while in the meantime keeping load balance works in 
certain degree.

Linux kernel uses 0.5ms by default. Considering the cost may be higher 
(e.g. VMCS impact) than in native, migration_cost=1ms is chosen for our 
tests, which are performed on a 4x 6-core Dunnington platform. In 24-VM 
case, there is ~2% stable performance gain for enterprise workloads like 
SPECjbb and sysbench. If HVM is with stubdom, the gain is more: 4% for 
the same workloads.

The best value may vary on different platforms based on different cache 
hierarchy and with different workloads. Due to resource limit, we 
haven't test many combinations. And we plans to try more in future. 
Welcome to evaluate and give feedback on what's suitable / not suitable 
for you.

Signed-off-by: Xiaowei Yang <xiaowei.yang@intel.com>


Thanks,
Xiaowei