From mboxrd@z Thu Jan 1 00:00:00 1970 From: Takuya Yoshikawa Subject: Re: [PATCH 0/4] KVM: Dirty logging optimization using rmap Date: Wed, 16 Nov 2011 17:17:32 +0900 Message-ID: <4EC3719C.5000701@oss.ntt.co.jp> References: <20111114182041.43570cdf.yoshikawa.takuya@oss.ntt.co.jp> <4EC0EC90.1090202@redhat.com> <4EC0F3D3.9090907@oss.ntt.co.jp> <4EC10BFE.7050704@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: mtosatti@redhat.com, takuya.yoshikawa@gmail.com, kvm@vger.kernel.org, qemu-devel@nongnu.org To: Avi Kivity Return-path: In-Reply-To: <4EC10BFE.7050704@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org Sender: qemu-devel-bounces+gceq-qemu-devel=gmane.org@nongnu.org List-Id: kvm.vger.kernel.org Adding qemu-devel to Cc. (2011/11/14 21:39), Avi Kivity wrote: > On 11/14/2011 12:56 PM, Takuya Yoshikawa wrote: >> (2011/11/14 19:25), Avi Kivity wrote: >>> On 11/14/2011 11:20 AM, Takuya Yoshikawa wrote: >>>> This is a revised version of my previous work. I hope that >>>> the patches are more self explanatory than before. >>>> >>> >>> It looks good. I'll let Marcelo (or anyone else?) review it as well >>> before applying. >>> >>> Do you have performance measurements? >>> >> >> For VGA, 30-40us became 3-5us when the display was quiet, with a >> enough warmed up guest. >> > > That's a nice improvement. > >> Near the criterion, the number was not different much from the >> original version. >> >> For live migration, I forgot the number but the result was good. >> But my test case was not enough to cover every pattern, so I changed >> the criterion to be a bit conservative. >> >> More tests may be able to find a better criterion. >> I am not in a hurry about this, so it is OK to add some tests >> before merging this. > > I think we can merge is as is, it's clear we get an improvement. I did a simple test to show numbers! Here, a 4GB guest was being migrated locally during copying a file in it. Case 1. corresponds to the original method and case 2 does to the optimized one. Small numbers are, probably, from VGA: Case 1. about 30us Case 2. about 3us Other numbers are from the system RAM (triggered by live migration): Case 1. about 500us, 2000us Case 2. about 80us, 2000us (not exactly averaged, see below for details) * 2000us was when rmap was not used, so equal to that of case 1. So I can say that my patch worked well for both VGA and live migration. Takuya === measurement snippet === Case 1. kvm_mmu_slot_remove_write_access() only (same as the original method): qemu-system-x86-25413 [000] 6546.215009: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.215010: funcgraph_entry: ! 2039.512 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.217051: funcgraph_exit: ! 2040.487 us | } qemu-system-x86-25413 [002] 6546.217347: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [002] 6546.217349: funcgraph_entry: ! 571.121 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [002] 6546.217921: funcgraph_exit: ! 572.525 us | } qemu-system-x86-25413 [000] 6546.314583: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.314585: funcgraph_entry: + 29.598 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.314616: funcgraph_exit: + 31.053 us | } qemu-system-x86-25413 [000] 6546.314784: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.314785: funcgraph_entry: ! 2002.591 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.316788: funcgraph_exit: ! 2003.537 us | } qemu-system-x86-25413 [000] 6546.317082: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.317083: funcgraph_entry: ! 624.445 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.317709: funcgraph_exit: ! 625.861 us | } qemu-system-x86-25413 [000] 6546.414261: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.414263: funcgraph_entry: + 29.593 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.414293: funcgraph_exit: + 30.944 us | } qemu-system-x86-25413 [000] 6546.414528: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.414529: funcgraph_entry: ! 1990.363 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.416520: funcgraph_exit: ! 1991.370 us | } qemu-system-x86-25413 [000] 6546.416775: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.416776: funcgraph_entry: ! 594.333 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.417371: funcgraph_exit: ! 595.415 us | } qemu-system-x86-25413 [000] 6546.514133: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.514135: funcgraph_entry: + 24.032 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.514160: funcgraph_exit: + 25.074 us | } qemu-system-x86-25413 [000] 6546.514312: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.514313: funcgraph_entry: ! 2035.365 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.516349: funcgraph_exit: ! 2036.298 us | } qemu-system-x86-25413 [000] 6546.516642: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.516643: funcgraph_entry: ! 598.308 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.517242: funcgraph_exit: ! 599.344 us | } qemu-system-x86-25413 [000] 6546.613895: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.613897: funcgraph_entry: + 27.765 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.613926: funcgraph_exit: + 29.051 us | } qemu-system-x86-25413 [000] 6546.614052: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.614053: funcgraph_entry: ! 1401.083 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.615454: funcgraph_exit: ! 1401.778 us | } qemu-system-x86-25413 [000] 6546.615656: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.615657: funcgraph_entry: ! 405.810 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.616063: funcgraph_exit: ! 406.415 us | } qemu-system-x86-25413 [001] 6546.713523: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.713525: funcgraph_entry: + 33.166 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.713559: funcgraph_exit: + 34.644 us | } qemu-system-x86-25413 [003] 6546.713688: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [003] 6546.713691: funcgraph_entry: ! 1872.491 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [003] 6546.715564: funcgraph_exit: ! 1874.775 us | } qemu-system-x86-25413 [001] 6546.715829: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.715830: funcgraph_entry: + 25.002 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.715856: funcgraph_exit: + 26.177 us | } qemu-system-x86-25413 [001] 6546.715969: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.715970: funcgraph_entry: ! 604.399 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.716575: funcgraph_exit: ! 605.502 us | } qemu-system-x86-25413 [000] 6546.813387: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.813389: funcgraph_entry: + 32.248 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.813422: funcgraph_exit: + 33.592 us | } qemu-system-x86-25413 [000] 6546.813565: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.813566: funcgraph_entry: ! 1970.585 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.815537: funcgraph_exit: ! 1971.752 us | } 2. + rmap (my method): qemu-system-x86-7772 [000] 6096.185229: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.185230: funcgraph_entry: ! 2090.108 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.187322: funcgraph_exit: ! 2091.865 us | } qemu-system-x86-7772 [001] 6096.187634: funcgraph_entry: 6.623 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.187765: funcgraph_entry: ! 110.571 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.284811: funcgraph_entry: 2.343 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.284971: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.284972: funcgraph_entry: ! 1999.656 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.286973: funcgraph_exit: ! 2000.955 us | } qemu-system-x86-7772 [000] 6096.287255: funcgraph_entry: + 79.547 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.384401: funcgraph_entry: 4.977 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.384512: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.384513: funcgraph_entry: ! 1887.579 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.386401: funcgraph_exit: ! 1889.068 us | } qemu-system-x86-7772 [001] 6096.386631: funcgraph_entry: + 80.816 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.484212: funcgraph_entry: 4.249 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.484280: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.484280: funcgraph_entry: ! 1872.626 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.486154: funcgraph_exit: ! 1873.982 us | } qemu-system-x86-7772 [001] 6096.486398: funcgraph_entry: + 99.259 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.584165: funcgraph_entry: 2.354 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.584450: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.584451: funcgraph_entry: ! 2012.011 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.586464: funcgraph_exit: ! 2013.625 us | } qemu-system-x86-7772 [000] 6096.586791: funcgraph_entry: + 74.855 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.683636: funcgraph_entry: 2.386 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.683749: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.683749: funcgraph_entry: ! 1922.750 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.685674: funcgraph_exit: ! 1924.311 us | } qemu-system-x86-7772 [000] 6096.685926: funcgraph_entry: + 76.410 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.783620: funcgraph_entry: 2.195 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.783715: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.783716: funcgraph_entry: ! 2110.459 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.785827: funcgraph_exit: ! 2111.781 us | } qemu-system-x86-7772 [000] 6096.786243: funcgraph_entry: 3.493 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.786334: funcgraph_entry: ! 186.657 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.883166: funcgraph_entry: 2.289 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.883376: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.883376: funcgraph_entry: ! 2031.813 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.885410: funcgraph_exit: ! 2033.311 us | } qemu-system-x86-7772 [000] 6096.885696: funcgraph_entry: + 53.929 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.983096: funcgraph_entry: 2.177 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.983288: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.983288: funcgraph_entry: ! 2096.257 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.985386: funcgraph_exit: ! 2097.685 us | } qemu-system-x86-7772 [000] 6096.985697: funcgraph_entry: + 78.442 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.082658: funcgraph_entry: 2.286 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.082800: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6097.082801: funcgraph_entry: ! 2049.809 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6097.084851: funcgraph_exit: ! 2051.107 us | } qemu-system-x86-7772 [000] 6097.085290: funcgraph_entry: + 69.201 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.182384: funcgraph_entry: 2.239 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.182545: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6097.182546: funcgraph_entry: ! 2158.647 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6097.184706: funcgraph_exit: ! 2160.028 us | } From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([140.186.70.92]:34746) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RQag1-0002Jl-4p for qemu-devel@nongnu.org; Wed, 16 Nov 2011 03:17:02 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1RQafy-0004WE-Qj for qemu-devel@nongnu.org; Wed, 16 Nov 2011 03:17:01 -0500 Received: from serv2.oss.ntt.co.jp ([222.151.198.100]:37373) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1RQafx-0004Vo-Sk for qemu-devel@nongnu.org; Wed, 16 Nov 2011 03:16:58 -0500 Message-ID: <4EC3719C.5000701@oss.ntt.co.jp> Date: Wed, 16 Nov 2011 17:17:32 +0900 From: Takuya Yoshikawa MIME-Version: 1.0 References: <20111114182041.43570cdf.yoshikawa.takuya@oss.ntt.co.jp> <4EC0EC90.1090202@redhat.com> <4EC0F3D3.9090907@oss.ntt.co.jp> <4EC10BFE.7050704@redhat.com> In-Reply-To: <4EC10BFE.7050704@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 0/4] KVM: Dirty logging optimization using rmap List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Avi Kivity Cc: mtosatti@redhat.com, takuya.yoshikawa@gmail.com, kvm@vger.kernel.org, qemu-devel@nongnu.org Adding qemu-devel to Cc. (2011/11/14 21:39), Avi Kivity wrote: > On 11/14/2011 12:56 PM, Takuya Yoshikawa wrote: >> (2011/11/14 19:25), Avi Kivity wrote: >>> On 11/14/2011 11:20 AM, Takuya Yoshikawa wrote: >>>> This is a revised version of my previous work. I hope that >>>> the patches are more self explanatory than before. >>>> >>> >>> It looks good. I'll let Marcelo (or anyone else?) review it as well >>> before applying. >>> >>> Do you have performance measurements? >>> >> >> For VGA, 30-40us became 3-5us when the display was quiet, with a >> enough warmed up guest. >> > > That's a nice improvement. > >> Near the criterion, the number was not different much from the >> original version. >> >> For live migration, I forgot the number but the result was good. >> But my test case was not enough to cover every pattern, so I changed >> the criterion to be a bit conservative. >> >> More tests may be able to find a better criterion. >> I am not in a hurry about this, so it is OK to add some tests >> before merging this. > > I think we can merge is as is, it's clear we get an improvement. I did a simple test to show numbers! Here, a 4GB guest was being migrated locally during copying a file in it. Case 1. corresponds to the original method and case 2 does to the optimized one. Small numbers are, probably, from VGA: Case 1. about 30us Case 2. about 3us Other numbers are from the system RAM (triggered by live migration): Case 1. about 500us, 2000us Case 2. about 80us, 2000us (not exactly averaged, see below for details) * 2000us was when rmap was not used, so equal to that of case 1. So I can say that my patch worked well for both VGA and live migration. Takuya === measurement snippet === Case 1. kvm_mmu_slot_remove_write_access() only (same as the original method): qemu-system-x86-25413 [000] 6546.215009: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.215010: funcgraph_entry: ! 2039.512 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.217051: funcgraph_exit: ! 2040.487 us | } qemu-system-x86-25413 [002] 6546.217347: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [002] 6546.217349: funcgraph_entry: ! 571.121 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [002] 6546.217921: funcgraph_exit: ! 572.525 us | } qemu-system-x86-25413 [000] 6546.314583: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.314585: funcgraph_entry: + 29.598 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.314616: funcgraph_exit: + 31.053 us | } qemu-system-x86-25413 [000] 6546.314784: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.314785: funcgraph_entry: ! 2002.591 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.316788: funcgraph_exit: ! 2003.537 us | } qemu-system-x86-25413 [000] 6546.317082: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.317083: funcgraph_entry: ! 624.445 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.317709: funcgraph_exit: ! 625.861 us | } qemu-system-x86-25413 [000] 6546.414261: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.414263: funcgraph_entry: + 29.593 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.414293: funcgraph_exit: + 30.944 us | } qemu-system-x86-25413 [000] 6546.414528: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.414529: funcgraph_entry: ! 1990.363 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.416520: funcgraph_exit: ! 1991.370 us | } qemu-system-x86-25413 [000] 6546.416775: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.416776: funcgraph_entry: ! 594.333 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.417371: funcgraph_exit: ! 595.415 us | } qemu-system-x86-25413 [000] 6546.514133: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.514135: funcgraph_entry: + 24.032 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.514160: funcgraph_exit: + 25.074 us | } qemu-system-x86-25413 [000] 6546.514312: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.514313: funcgraph_entry: ! 2035.365 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.516349: funcgraph_exit: ! 2036.298 us | } qemu-system-x86-25413 [000] 6546.516642: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.516643: funcgraph_entry: ! 598.308 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.517242: funcgraph_exit: ! 599.344 us | } qemu-system-x86-25413 [000] 6546.613895: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.613897: funcgraph_entry: + 27.765 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.613926: funcgraph_exit: + 29.051 us | } qemu-system-x86-25413 [000] 6546.614052: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.614053: funcgraph_entry: ! 1401.083 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.615454: funcgraph_exit: ! 1401.778 us | } qemu-system-x86-25413 [000] 6546.615656: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.615657: funcgraph_entry: ! 405.810 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.616063: funcgraph_exit: ! 406.415 us | } qemu-system-x86-25413 [001] 6546.713523: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.713525: funcgraph_entry: + 33.166 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.713559: funcgraph_exit: + 34.644 us | } qemu-system-x86-25413 [003] 6546.713688: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [003] 6546.713691: funcgraph_entry: ! 1872.491 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [003] 6546.715564: funcgraph_exit: ! 1874.775 us | } qemu-system-x86-25413 [001] 6546.715829: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.715830: funcgraph_entry: + 25.002 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.715856: funcgraph_exit: + 26.177 us | } qemu-system-x86-25413 [001] 6546.715969: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [001] 6546.715970: funcgraph_entry: ! 604.399 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [001] 6546.716575: funcgraph_exit: ! 605.502 us | } qemu-system-x86-25413 [000] 6546.813387: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.813389: funcgraph_entry: + 32.248 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.813422: funcgraph_exit: + 33.592 us | } qemu-system-x86-25413 [000] 6546.813565: funcgraph_entry: | write_protect_slot() { qemu-system-x86-25413 [000] 6546.813566: funcgraph_entry: ! 1970.585 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-25413 [000] 6546.815537: funcgraph_exit: ! 1971.752 us | } 2. + rmap (my method): qemu-system-x86-7772 [000] 6096.185229: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.185230: funcgraph_entry: ! 2090.108 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.187322: funcgraph_exit: ! 2091.865 us | } qemu-system-x86-7772 [001] 6096.187634: funcgraph_entry: 6.623 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.187765: funcgraph_entry: ! 110.571 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.284811: funcgraph_entry: 2.343 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.284971: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.284972: funcgraph_entry: ! 1999.656 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.286973: funcgraph_exit: ! 2000.955 us | } qemu-system-x86-7772 [000] 6096.287255: funcgraph_entry: + 79.547 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.384401: funcgraph_entry: 4.977 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.384512: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.384513: funcgraph_entry: ! 1887.579 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.386401: funcgraph_exit: ! 1889.068 us | } qemu-system-x86-7772 [001] 6096.386631: funcgraph_entry: + 80.816 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.484212: funcgraph_entry: 4.249 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.484280: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.484280: funcgraph_entry: ! 1872.626 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.486154: funcgraph_exit: ! 1873.982 us | } qemu-system-x86-7772 [001] 6096.486398: funcgraph_entry: + 99.259 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.584165: funcgraph_entry: 2.354 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.584450: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.584451: funcgraph_entry: ! 2012.011 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.586464: funcgraph_exit: ! 2013.625 us | } qemu-system-x86-7772 [000] 6096.586791: funcgraph_entry: + 74.855 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.683636: funcgraph_entry: 2.386 us | write_protect_slot(); qemu-system-x86-7772 [001] 6096.683749: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [001] 6096.683749: funcgraph_entry: ! 1922.750 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [001] 6096.685674: funcgraph_exit: ! 1924.311 us | } qemu-system-x86-7772 [000] 6096.685926: funcgraph_entry: + 76.410 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.783620: funcgraph_entry: 2.195 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.783715: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.783716: funcgraph_entry: ! 2110.459 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.785827: funcgraph_exit: ! 2111.781 us | } qemu-system-x86-7772 [000] 6096.786243: funcgraph_entry: 3.493 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.786334: funcgraph_entry: ! 186.657 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.883166: funcgraph_entry: 2.289 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.883376: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.883376: funcgraph_entry: ! 2031.813 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.885410: funcgraph_exit: ! 2033.311 us | } qemu-system-x86-7772 [000] 6096.885696: funcgraph_entry: + 53.929 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.983096: funcgraph_entry: 2.177 us | write_protect_slot(); qemu-system-x86-7772 [000] 6096.983288: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6096.983288: funcgraph_entry: ! 2096.257 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6096.985386: funcgraph_exit: ! 2097.685 us | } qemu-system-x86-7772 [000] 6096.985697: funcgraph_entry: + 78.442 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.082658: funcgraph_entry: 2.286 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.082800: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6097.082801: funcgraph_entry: ! 2049.809 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6097.084851: funcgraph_exit: ! 2051.107 us | } qemu-system-x86-7772 [000] 6097.085290: funcgraph_entry: + 69.201 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.182384: funcgraph_entry: 2.239 us | write_protect_slot(); qemu-system-x86-7772 [000] 6097.182545: funcgraph_entry: | write_protect_slot() { qemu-system-x86-7772 [000] 6097.182546: funcgraph_entry: ! 2158.647 us | kvm_mmu_slot_remove_write_access(); qemu-system-x86-7772 [000] 6097.184706: funcgraph_exit: ! 2160.028 us | }