Re: performance with guests running 2.4 kernels (specifically RHEL3)

Avi Kivity
2008-04-21T09:19:53Z

David S. Ahern wrote:
> I added the traces and captured data over another apparent lockup of the guest.
> This seems to be representative of the sequence (pid/vcpu removed).
>
> (+4776) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016127c ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+3632) VMENTRY
> (+4552) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016104a ]
> (+ 0) PAGE_FAULT [ errorcode = 0x0000000b, virt = 0x00000000 fffb61c8 ]
> (+ 54928) VMENTRY
>

Can you oprofile the host to see where the 54K cycles are spent?

> (+4568) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db4 gpte = 0x00000000 41c5d363 ]
> (+8432) VMENTRY
> (+3936) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ]
> (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ]
> (+ 13832) VMENTRY
>
>
> (+5768) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016127c ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+3712) VMENTRY
> (+4576) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c016104a ]
> (+ 0) PAGE_FAULT [ errorcode = 0x0000000b, virt = 0x00000000 fffb61d0 ]
> (+ 0) PTE_WRITE [ gpa = 0x00000000 3d5981d0 gpte = 0x00000000 3d55d047 ]
>

This indeed has the accessed bit clear.

> (+ 65216) VMENTRY
> (+4232) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610e7 ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db4 ]
> (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db4 gpte = 0x00000000 3d598363 ]
>

This has the accessed bit set and the user bit clear, and the pte
pointing at the previous pte_write gpa. Looks like a kmap_atomic().

> (+8640) VMENTRY
> (+3936) VMEXIT [ exitcode = 0x00000000, rip = 0x00000000 c01610ee ]
> (+ 0) PAGE_FAULT [ errorcode = 0x00000003, virt = 0x00000000 c0009db0 ]
> (+ 0) PTE_WRITE [ gpa = 0x00000000 00009db0 gpte = 0x00000000 00000000 ]
> (+ 14160) VMENTRY
>
> I can forward a more complete time snippet if you'd like. vcpu0 + corresponding
> vcpu1 files have 85000 total lines and compressed the files total ~500k.
>
> I did not see the FLOODED trace come out during this sample though I did bump
> the count from 3 to 4 as you suggested.
>
>
>

Bumping the count was supposed to remove the flooding...

> Correlating rip addresses to the 2.4 kernel:
>
> c0160d00-c0161290 = page_referenced
>
> It looks like the event is kscand running through the pages. I suspected this
> some time ago, and tried tweaking the kscand_work_percent sysctl variable. It
> appeared to lower the peak of the spikes, but maybe I imagined it. I believe
> lowering that value makes kscand wake up more often but do less work (page
> scanning) each time it is awakened.
>
>

What does 'top' in the guest show (perhaps sorted by total cpu time
rather than instantaneous usage)?

What host kernel are you running? How many host cpus?

--
error compiling committee.c: too many arguments to function

-------------------------------------------------------------------------
This SF.net email is sponsored by the 2008 JavaOne(SM) Conference
Don't miss this year's exciting event. There's still time to save $100.
Use priority code J8TL2D2.