From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: performance with guests running 2.4 kernels (specifically RHEL3) Date: Mon, 21 Apr 2008 12:19:53 +0300 Message-ID: <480C5C39.4040300@qumranet.com> References: <48054518.3000104@cisco.com> <4805BCF1.6040605@qumranet.com> <4807BD53.6020304@cisco.com> <48085485.3090205@qumranet.com> <480C188F.3020101@cisco.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: kvm-devel To: "David S. Ahern" Return-path: In-Reply-To: <480C188F.3020101@cisco.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: kvm-devel-bounces@lists.sourceforge.net Errors-To: kvm-devel-bounces@lists.sourceforge.net List-Id: kvm.vger.kernel.org David S. Ahern wrote: > I added the traces and captured data over another apparent lockup of the guest. > This seems to be representative of the sequence (pid/vcpu removed). > > (+4776) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016127c ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ] > (+3632) VMENTRY > (+4552) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016104a ] > (+ 0) PAGE_FAULT [ errorcode = 0x0000000b, virt = 0x00000000 fffb61c8 ] > (+ 54928) VMENTRY > Can you oprofile the host to see where the 54K cycles are spent? > (+4568) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ] > (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db4 gpte = 0x00000000 41c5d363 ] > (+8432) VMENTRY > (+3936) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ] > (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ] > (+ 13832) VMENTRY > > > (+5768) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016127c ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ] > (+3712) VMENTRY > (+4576) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016104a ] > (+ 0) PAGE_FAULT [ errorcode = 0x0000000b, virt = 0x00000000 fffb61d0 ] > (+ 0) PTE_WRITE [ gpa = 0x00000000 3d5981d0 gpte = 0x00000000 3d55d047 ] > This indeed has the accessed bit clear. > (+ 65216) VMENTRY > (+4232) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ] > (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db4 gpte = 0x00000000 3d598363 ] > This has the accessed bit set and the user bit clear, and the pte pointing at the previous pte_write gpa. Looks like a kmap_atomic(). > (+8640) VMENTRY > (+3936) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ] > (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ] > (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ] > (+ 14160) VMENTRY > > I can forward a more complete time snippet if you'd like. vcpu0 + corresponding > vcpu1 files have 85000 total lines and compressed the files total ~500k. > > I did not see the FLOODED trace come out during this sample though I did bump > the count from 3 to 4 as you suggested. > > > Bumping the count was supposed to remove the flooding... > Correlating rip addresses to the 2.4 kernel: > > c0160d00-c0161290 = page_referenced > > It looks like the event is kscand running through the pages. I suspected this > some time ago, and tried tweaking the kscand_work_percent sysctl variable. It > appeared to lower the peak of the spikes, but maybe I imagined it. I believe > lowering that value makes kscand wake up more often but do less work (page > scanning) each time it is awakened. > > What does 'top' in the guest show (perhaps sorted by total cpu time rather than instantaneous usage)? What host kernel are you running? How many host cpus? -- error compiling committee.c: too many arguments to function ------------------------------------------------------------------------- This SF.net email is sponsored by the 2008 JavaOne(SM) Conference Don't miss this year's exciting event. There's still time to save $100. Use priority code J8TL2D2. http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone