From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Cooper Subject: Re: [PATCH v2] x86/HVM: Avoid cache flush operations during hvm_load Date: Tue, 2 Jun 2015 13:56:03 +0100 Message-ID: <556DA7E3.4070500@citrix.com> References: <1433249220-23845-1-git-send-email-ross.lagerwall@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1433249220-23845-1-git-send-email-ross.lagerwall@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Ross Lagerwall , xen-devel@lists.xen.org Cc: Keir Fraser , Jan Beulich List-Id: xen-devel@lists.xenproject.org On 02/06/15 13:47, Ross Lagerwall wrote: > An MTRR record is processed for each vCPU during hvm_load. Each MTRR > record sets several mtrrs, each of which flushes the cache on all pCPUs. > This can take some time and trip the watchdog for HVM guests with many > CPUs. > > To fix this, introduce a flag which prevents flushing the cache on x86 > while loading the restore records and instead does a single cache flush > at the end of hvm_load. > > This reduces the time to restore an HVM guest with 32 vCPUs by about 5 > seconds on an Intel Xeon CPU E7-2870. > > Signed-off-by: Ross Lagerwall > --- > > In v2: Code moved into arch hooks since it's x86 specific. > > xen/arch/x86/hvm/mtrr.c | 5 +++++ > xen/arch/x86/hvm/save.c | 10 ++++++++++ > xen/common/hvm/save.c | 15 ++++++++++----- > xen/include/asm-x86/mtrr.h | 9 +++++++++ > xen/include/xen/hvm/save.h | 1 + > 5 files changed, 35 insertions(+), 5 deletions(-) > > diff --git a/xen/arch/x86/hvm/mtrr.c b/xen/arch/x86/hvm/mtrr.c > index a69ee62..f21b367 100644 > --- a/xen/arch/x86/hvm/mtrr.c > +++ b/xen/arch/x86/hvm/mtrr.c > @@ -65,6 +65,8 @@ static const uint8_t mm_type_tbl[MTRR_NUM_TYPES][PAT_TYPE_NUMS] = { > #undef RS > }; > > +DEFINE_PER_CPU(bool_t, memory_type_changed_ignore); Thinking about this, this should be "memory_type_changed_defer" and there should be a second bool "memory_type_changed_wanted". This way, we don't suffer a memory_type_changed() in arch_hvm_load_post() if there were no MTRR records loaded; we support partial loads of state via this path as well. ~Andrew