From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: High vm-exit latencies during kvm boot-up/shutdown Date: Wed, 24 Oct 2007 18:52:53 +0200 Message-ID: <471F7865.7070506@qumranet.com> References: <471D2D8C.1080202@web.de> <10EA09EFD8728347A513008B6B0DA77A02441127@pdsmsx411.ccr.corp.intel.com> <471D96DC.7070809@web.de> <10EA09EFD8728347A513008B6B0DA77A024414D9@pdsmsx411.ccr.corp.intel.com> <471DBA1A.2080108@web.de> <471DC311.2050003@qumranet.com> <471DF76B.7040001@siemens.com> <471E02F7.6080408@qumranet.com> <471E0818.6060405@siemens.com> <471E1290.2000208@qumranet.com> <471E1A77.90808@siemens.com> <471E1DCD.5040301@qumranet.com> <471E21D7.3000309@siemens.com> <471E238E.6040005@qumranet.com> <471E27B0.1090001@siemens.com> <471E29C5.1060807@qumranet.com> <471F0464.4000704@siemens.com> <471F07C0.8040104@qumranet.com> <471F0D57.3020209@siemens.com> <471F116D.9080903@qumranet.com> <471F7143.8040406@siemens.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org To: Jan Kiszka Return-path: In-Reply-To: <471F7143.8040406-kv7WeFo6aLtBDgjK7y7TUQ@public.gmane.org> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org Errors-To: kvm-devel-bounces-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org List-Id: kvm.vger.kernel.org Jan Kiszka wrote: > Avi Kivity wrote: > >> Jan Kiszka wrote: >> >>> Avi Kivity wrote: >>> >>> >>>> Jan Kiszka wrote: >>>> >>>> >>>>> I ran >>>>> >>>>> user/kvmctl user/test/bootstrap user/test/smp.flat >>>>> >>>>> with the busy loop hacked into bootstrap, but I got no latency spots >>>>> this time. And what should I look for in the output of kvm_stat? >>>>> >>>>> >>>>> >>>>> >>>> The first numeric column is the total number of exits; the second is the >>>> rate of change (per second). For a spinning workload, both irq_exits >>>> and exits rate should equal the timer frequency. >>>> >>>> >>> It matches roughly, but this isn't surprising given the lack of latency >>> peaks in this scenario. >>> >>> >>> >> Okay, that's expected. But I really can't think what causes the latency >> spike you see running the bios code. 300 microseconds is enough for 3000 >> cache misses. No instruction can cause so many except rep movs and >> friends, and these are supposed to be preemptible. >> >> You might try isolating it by adding code to output markers to some I/O >> port and scattering it throughout the bios. A binary search should find >> the problem spot quickly. >> >> > > Got it! It's wbinvd from smm_init in rombios32.c! Anyone any comments on > this? > Ha! A real life 300usec instruction! Unfortunately, it cannot be trapped on Intel (it can be trapped on AMD). Looks like a minor hole in VT, as a guest can generate very high latencies this way. For our bios, we can remove it, but there's no way to ensure an untrusted guest doesn't use it. -- error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Still grepping through log files to find problems? Stop. Now Search log events and configuration files using AJAX and a browser. Download your FREE copy of Splunk now >> http://get.splunk.com/