Re: Why is 'emulate' as good as writable PT's?

[Andrew Theurer <habanero@us.ibm.com>]

Ian Pratt wrote:
>> I was wondering, perhaps we are not just triggering writable pagetables when we shouldn't, but maybe we are flushing them back too early.  I
>> added some xen perf counters to get an idea of why we are flushing back
>> wtpt's (run on SDET again):
>
> Are these numbers taken on a uniprocessor guest (or dom0?)
Yes.
>
>>                         modified:  0 <=10  <=20  <=30  <=40  <=50
>> 1 writable pt updates    T=1086   0  612   194   111    49    85
>> 2 ptwr_fl: called from ptwr_emulated_update because wtpt exists T=0
>> 3 ptwr_fl: called from ptwr_do_page_fault because wtpt is used T=338
>> 4 ptwr_fl: called from spurious_page_fault  T=0
>> 5 ptwr_fl: called from fixup_page_fault  T=0
>> 6 ptwr_fl: called from cleanup_wpt, do_mmuext_op (active) T=467
>> 7 ptwr_fl: called from cleanup_wpt, do_mmuext_op (inactive) T=0
>> 8 ptwr_fl: called from cleanup_wpt, update_va_mapping (active) T=280
>> 9 ptwr_fl: called from cleanup_wpt, update_va_mapping (inactive) T=0
>> 10 ptwr_flush: called from cleanup_wpt, do_mmu_update (active) T=1
>> 11 ptwr_flush: called from cleanup_wpt, do_mmu_update (inactive) T=0
>> line 3:  I think we can just goto emulate instead of flushing back the
>> wtpt here, right?  I've tried this, but no real difference in
>> performance.  Could we increase the number of wtpt's we keep track of,
>> so we don't have to flush back or emulate?
>
> This will happen as part of a fork when we move on to the next page in
> the PT. It should be harmless unless we're flopping back and forth.
OK
>
>> line 6: We seem to call cleanup_writable_pagetables unconditionally
>> here, and if either of the active or inactive pages are used, they get
>> flushed back.  Do we always need to do this?
>
> What's the op? is it a TLB flush, invplg, or cr3 load?
I don't know, but I'll find out.
>  
>> line 8: Also call cleanup_writable_pagetables unconditionally here.
>> Do the wtpt's always need this to happen?  Is is possible the
>> update_va_mapping call is for an address space which does not affect
>> the wtpt?
>
> It's interesting to understand what the interaction is here. I'd like to
> know 
>
>> line 10: Not seeing many flushes here, so I guess it's not an issue.
>>
>> Sorry if these questions seem odd.  There's a good chance I am not
>> "getting it" :)
>
> This is useful work. It's been on our todo list to re-profile this on
> newer kernels. Once upon a time we had it quite nicely tuned...
>
> Could you find out all the kernel EIPs that are triggering writeable
> pagetables with any frequency and list them for us. It might be good to
> turn everything into using update mmuop and then just turn on direct
> writes just for the fork case which is where we know need it.
I think I can do that.  I'll just use something similar to the EIP 
logging Xen has for finding out what triggered the wtpt flushes.

Thanks,

Andrew