Re: PML (Page Modification Logging) design for Xen

[Kai Huang <kai.huang@linux.intel.com>]

On 02/12/2015 02:25 PM, Tian, Kevin wrote:
>> From: Kai Huang [mailto:kai.huang@linux.intel.com]
>> Sent: Thursday, February 12, 2015 10:35 AM
>>
>> On 02/11/2015 09:13 PM, Jan Beulich wrote:
>>>>>> On 11.02.15 at 12:52, <andrew.cooper3@citrix.com> wrote:
>>>> On 11/02/15 08:28, Kai Huang wrote:
>>>>> With PML, we don't have to use write protection but just clear D-bit
>>>>> of EPT entry of guest memory to do dirty logging, with an additional
>>>>> PML buffer full VMEXIT for 512 dirty GPAs. Theoretically, this can
>>>>> reduce hypervisor overhead when guest is in dirty logging mode, and
>>>>> therefore more CPU cycles can be allocated to guest, so it's expected
>>>>> benchmarks in guest will have better performance comparing to
>> non-PML.
>>>> One issue with basic EPT A/D tracking was the scan of the EPT tables.
>>>> Here, hardware will give us a list of affected gfns, but how is Xen
>>>> supposed to efficiently clear the dirty bits again?  Using EPT
>>>> misconfiguration is no better than the existing fault path.
>>> Why not? The misconfiguration exit ought to clear the D bit for all
>>> 511 entries in the L1 table (and set it for the one entry that is
>>> currently serving the access). All further D bit handling will then
>>> be PML based.
>> Indeed, we clear D-bit in EPT misconfiguration. In my understanding, the
>> sequences are as follows:
>>
>> 1) PML enabled for the domain.
>> 2) ept_invalidate_emt (or ept_invalidate_emt_range) is called.
>> 3) Guest accesses specific GPA (which has been invalidated by step 2),
>> and EPT misconfig is triggered.
>> 4) Then resolve_misconfig is called, which fixes up GFN (above GPA >>
>> 12) to p2m_ram_logdirty, and calls ept_p2m_type_to_flags, in which we
>> clear D-bit of EPT entry (instead of clear W-bit) if p2m type is
>> p2m_ram_logdirty. Then dirty logging of this GFN will be handled by PML.
>>
>> The above 2) ~ 4) will be repeated when log-dirty radix tree is cleared.
> is ept_invalidate_emt required by existing logdirty mode or by PML enable?
It's in existing logdirty code.
> can we clear D bits directly when log-dirty radix tree is cleared to reduce
> EPT misconfig exits for repeatedly dirtied pages?
Theoretically we can, and looks logdirty for video ram is done in this 
way (logdirty for the page is re-enabled while it is reported to 
dirty_bitmap). One thing is looks video ram logdirty only exists for HAP 
mode.
But in current log dirty implementation for global logdirty, at common 
paging layer, the log-dirty radix tree is cleaned in single step after 
reporting all dirty pages to userspace. And it just calls 
ept_invalidate_emt essentially. Therefore we need to modify logdirty 
common code at paging layer to achieve this, which is more like logdirty 
enhancement but not related to PML enabling directly. And any change of 
interface in paging layer requires modification in shadow mode 
accordingly, so currently I just choose not to do it.

Thanks,
-Kai
>
> Thanks
> Kevin
>