* dirty page tracking in kvm/qemu -- page faults inevitable?
@ 2014-07-29 22:12 Chris Friesen
2014-07-30 6:09 ` Xiao Guangrong
0 siblings, 1 reply; 6+ messages in thread
From: Chris Friesen @ 2014-07-29 22:12 UTC (permalink / raw)
To: avi, mtosatti, kvm
Hi,
I've got an issue where we're hitting major performance penalties while
doing live migration, and it seems like it might be due to page faults
triggering hypervisor exits, and then we get stuck waiting for the
iothread lock which is held by the qemu dirty page scanning code.
Accordingly, I'm trying to figure out the actual mechanism whereby dirty
pages are tracked in qemu/kvm. I've got an Ivy Bridge CPU, a 3.4 kernel
on the host, and qemu 1.4.
Looking at the qemu code, it seems to be calling down into kvm to get
the dirty page information.
Looking at kvm, most of what I read seems to be doing the usual "mark it
read-only and then when we take a page fault mark it as dirty" trick.
However, I read something about Intel EPT having hardware support for
tracking dirty pages. It seems like this might avoid the need for a
page fault, but might only be available on Haswell or later CPUs--is
that correct? Is it supported in kvm? If so, when was support added?
Thanks,
Chris
P.S. Please CC me on reply, I'm not subscribed to the list.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dirty page tracking in kvm/qemu -- page faults inevitable?
2014-07-29 22:12 dirty page tracking in kvm/qemu -- page faults inevitable? Chris Friesen
@ 2014-07-30 6:09 ` Xiao Guangrong
2014-07-30 7:41 ` Chris Friesen
0 siblings, 1 reply; 6+ messages in thread
From: Xiao Guangrong @ 2014-07-30 6:09 UTC (permalink / raw)
To: Chris Friesen, avi, mtosatti, kvm
On 07/30/2014 06:12 AM, Chris Friesen wrote:
> Hi,
>
> I've got an issue where we're hitting major performance penalties while doing live migration, and it seems like it might
> be due to page faults triggering hypervisor exits, and then we get stuck waiting for the iothread lock which is held by
> the qemu dirty page scanning code.
I am afraid that using dirty-bit instead of write-protection may cause the case
even more worse for iothread-lock because we need to walk whole sptes to get
dirty-set pages, however currently we only need to walk the page set in the
bitmap.
>
> Accordingly, I'm trying to figure out the actual mechanism whereby dirty pages are tracked in qemu/kvm. I've got an Ivy Bridge CPU, a 3.4 kernel on the host, and qemu 1.4.
>
> Looking at the qemu code, it seems to be calling down into kvm to get the dirty page information.
>
> Looking at kvm, most of what I read seems to be doing the usual "mark it read-only and then when we take a page fault mark it as dirty" trick.
>
> However, I read something about Intel EPT having hardware support for tracking dirty pages. It seems like this might avoid the need for a page fault, but might only be available on Haswell or later CPUs--is that correct? Is it supported in kvm? If so, when was support added?
Actually, i have implemented the prototype long time ago, maybe it's the time to benchmark
it and post it out.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dirty page tracking in kvm/qemu -- page faults inevitable?
2014-07-30 6:09 ` Xiao Guangrong
@ 2014-07-30 7:41 ` Chris Friesen
2014-07-30 15:42 ` Paolo Bonzini
0 siblings, 1 reply; 6+ messages in thread
From: Chris Friesen @ 2014-07-30 7:41 UTC (permalink / raw)
To: Xiao Guangrong, avi, mtosatti, kvm
On 07/30/2014 12:09 AM, Xiao Guangrong wrote:
> On 07/30/2014 06:12 AM, Chris Friesen wrote:
>> Hi,
>>
>> I've got an issue where we're hitting major performance penalties while doing live migration, and it seems like it might
>> be due to page faults triggering hypervisor exits, and then we get stuck waiting for the iothread lock which is held by
>> the qemu dirty page scanning code.
>
> I am afraid that using dirty-bit instead of write-protection may cause the case
> even more worse for iothread-lock because we need to walk whole sptes to get
> dirty-set pages, however currently we only need to walk the page set in the
> bitmap.
I found a document at
"http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf"
which talks about the benefits of Haswell. One of the items reads:
"New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates
major cause of vmexits"
Is that accurate? If so, then it seems like it should allow for the VM
to run without trying to exit the hypervisor, and as long as it just
does in-memory operations it won't contend on the iothread lock.
Chris
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dirty page tracking in kvm/qemu -- page faults inevitable?
2014-07-30 7:41 ` Chris Friesen
@ 2014-07-30 15:42 ` Paolo Bonzini
2014-07-30 16:02 ` Chris Friesen
0 siblings, 1 reply; 6+ messages in thread
From: Paolo Bonzini @ 2014-07-30 15:42 UTC (permalink / raw)
To: Chris Friesen, Xiao Guangrong, avi, mtosatti, kvm
Il 30/07/2014 09:41, Chris Friesen ha scritto:
>> I am afraid that using dirty-bit instead of write-protection may cause the case
>> even more worse for iothread-lock because we need to walk whole sptes to get
>> dirty-set pages, however currently we only need to walk the page set in the
>> bitmap.
>
> I found a document at
> "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf"
> which talks about the benefits of Haswell. One of the items reads:
>
> "New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates
> major cause of vmexits"
>
> Is that accurate? If so, then it seems like it should allow for the VM
> to run without trying to exit the hypervisor, and as long as it just
> does in-memory operations it won't contend on the iothread lock.
True, but:
1) the problem is fishing the information out of the page tables and
passing it up to userspace. You have to process the whole EPT tree one
page at a time, instead of doing it 64 bits at a time. Also, one source
of bad performance is having to split all entries of the EPT page tables
down to 4K, and you get that anyway.
2) You should not get to userspace simply for marking a page as locked.
As you describe it, your problem seems to be contention between QEMU
threads, KVM is not involved.
3) What version of QEMU are you using? Things have been improving
steadily, and we probably will get to using atomic operations instead of
the iothread lock to protect the migration dirty bitmap.
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dirty page tracking in kvm/qemu -- page faults inevitable?
2014-07-30 15:42 ` Paolo Bonzini
@ 2014-07-30 16:02 ` Chris Friesen
2014-07-30 16:18 ` Paolo Bonzini
0 siblings, 1 reply; 6+ messages in thread
From: Chris Friesen @ 2014-07-30 16:02 UTC (permalink / raw)
To: Paolo Bonzini, Xiao Guangrong, avi, mtosatti, kvm
On 07/30/2014 09:42 AM, Paolo Bonzini wrote:
> Il 30/07/2014 09:41, Chris Friesen ha scritto:
>> I found a document at
>> "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf"
>> which talks about the benefits of Haswell. One of the items reads:
>>
>> "New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates
>> major cause of vmexits"
>>
>> Is that accurate? If so, then it seems like it should allow for the VM
>> to run without trying to exit the hypervisor, and as long as it just
>> does in-memory operations it won't contend on the iothread lock.
> 2) You should not get to userspace simply for marking a page as locked.
> As you describe it, your problem seems to be contention between QEMU
> threads, KVM is not involved.
What about writing to a page where we're tracking dirty pages? Would
that get back up to qemu or would that be handled entirely in the kvm
kernel module?
I was assuming that it was due to the page faults since as far as I know
the app in the VM is just doing packet processing from/to memory-mapped
circular buffers--the qemu threads in question aren't doing "normal" I/O
but something is causing them to try to acquire the iothread lock.
> 3) What version of QEMU are you using? Things have been improving
> steadily, and we probably will get to using atomic operations instead of
> the iothread lock to protect the migration dirty bitmap.
We're currently on 1.4.2. We're looking at trying out 1.7 to see if
it's better, but we've got some local patches that would need to get ported.
Chris
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: dirty page tracking in kvm/qemu -- page faults inevitable?
2014-07-30 16:02 ` Chris Friesen
@ 2014-07-30 16:18 ` Paolo Bonzini
0 siblings, 0 replies; 6+ messages in thread
From: Paolo Bonzini @ 2014-07-30 16:18 UTC (permalink / raw)
To: Chris Friesen, Xiao Guangrong, mtosatti, kvm
Il 30/07/2014 18:02, Chris Friesen ha scritto:
> On 07/30/2014 09:42 AM, Paolo Bonzini wrote:
>> Il 30/07/2014 09:41, Chris Friesen ha scritto:
>>> I found a document at
>>> "http://ftp.software-sources.co.il/Processor_Architecture_Update-Bob_Valentine.pdf"
>>>
>>> which talks about the benefits of Haswell. One of the items reads:
>>>
>>> "New Accessed and Dirty bits for Extended Page Tables (EPT) eliminates
>>> major cause of vmexits"
>>>
>>> Is that accurate? If so, then it seems like it should allow for the VM
>>> to run without trying to exit the hypervisor, and as long as it just
>>> does in-memory operations it won't contend on the iothread lock.
>
>> 2) You should not get to userspace simply for marking a page as locked.
>> As you describe it, your problem seems to be contention between QEMU
>> threads, KVM is not involved.
>
> What about writing to a page where we're tracking dirty pages? Would
> that get back up to qemu or would that be handled entirely in the kvm
> kernel module?
It's handle inside the kernel module.
Every now and then QEMU asks the kernel for the dirty pages and ORs the
bitmap returned by KVM with its own. All this is done under the
iothread lock.
> I was assuming that it was due to the page faults since as far as I know
> the app in the VM is just doing packet processing from/to memory-mapped
> circular buffers--the qemu threads in question aren't doing "normal" I/O
> but something is causing them to try to acquire the iothread lock.
>
>> 3) What version of QEMU are you using? Things have been improving
>> steadily, and we probably will get to using atomic operations instead of
>> the iothread lock to protect the migration dirty bitmap.
>
> We're currently on 1.4.2. We're looking at trying out 1.7 to see if
> it's better, but we've got some local patches that would need to get
> ported.
>From a quick "git describe" 2.0 is needed. The patches end at commit
ae2810c (memory: syncronize kvm bitmap using bitmaps operations,
2013-11-05).
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-07-30 16:18 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2014-07-29 22:12 dirty page tracking in kvm/qemu -- page faults inevitable? Chris Friesen
2014-07-30 6:09 ` Xiao Guangrong
2014-07-30 7:41 ` Chris Friesen
2014-07-30 15:42 ` Paolo Bonzini
2014-07-30 16:02 ` Chris Friesen
2014-07-30 16:18 ` Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox