From mboxrd@z Thu Jan 1 00:00:00 1970 From: Andrew Theurer Subject: Re: Why is 'emulate' as good as writable PT's? Date: Tue, 13 Jun 2006 09:47:49 -0500 Message-ID: <448ED015.8040808@us.ibm.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ian Pratt Cc: Xen development list , Rolf Neugebauer List-Id: xen-devel@lists.xenproject.org Ian Pratt wrote: >> I was wondering, perhaps we are not just triggering writable pagetables when we shouldn't, but maybe we are flushing them back too early. I >> added some xen perf counters to get an idea of why we are flushing back >> wtpt's (run on SDET again): > > Are these numbers taken on a uniprocessor guest (or dom0?) Yes. > >> modified: 0 <=10 <=20 <=30 <=40 <=50 >> 1 writable pt updates T=1086 0 612 194 111 49 85 >> 2 ptwr_fl: called from ptwr_emulated_update because wtpt exists T=0 >> 3 ptwr_fl: called from ptwr_do_page_fault because wtpt is used T=338 >> 4 ptwr_fl: called from spurious_page_fault T=0 >> 5 ptwr_fl: called from fixup_page_fault T=0 >> 6 ptwr_fl: called from cleanup_wpt, do_mmuext_op (active) T=467 >> 7 ptwr_fl: called from cleanup_wpt, do_mmuext_op (inactive) T=0 >> 8 ptwr_fl: called from cleanup_wpt, update_va_mapping (active) T=280 >> 9 ptwr_fl: called from cleanup_wpt, update_va_mapping (inactive) T=0 >> 10 ptwr_flush: called from cleanup_wpt, do_mmu_update (active) T=1 >> 11 ptwr_flush: called from cleanup_wpt, do_mmu_update (inactive) T=0 >> line 3: I think we can just goto emulate instead of flushing back the >> wtpt here, right? I've tried this, but no real difference in >> performance. Could we increase the number of wtpt's we keep track of, >> so we don't have to flush back or emulate? > > This will happen as part of a fork when we move on to the next page in > the PT. It should be harmless unless we're flopping back and forth. OK > >> line 6: We seem to call cleanup_writable_pagetables unconditionally >> here, and if either of the active or inactive pages are used, they get >> flushed back. Do we always need to do this? > > What's the op? is it a TLB flush, invplg, or cr3 load? I don't know, but I'll find out. > >> line 8: Also call cleanup_writable_pagetables unconditionally here. >> Do the wtpt's always need this to happen? Is is possible the >> update_va_mapping call is for an address space which does not affect >> the wtpt? > > It's interesting to understand what the interaction is here. I'd like to > know > >> line 10: Not seeing many flushes here, so I guess it's not an issue. >> >> Sorry if these questions seem odd. There's a good chance I am not >> "getting it" :) > > This is useful work. It's been on our todo list to re-profile this on > newer kernels. Once upon a time we had it quite nicely tuned... > > Could you find out all the kernel EIPs that are triggering writeable > pagetables with any frequency and list them for us. It might be good to > turn everything into using update mmuop and then just turn on direct > writes just for the fork case which is where we know need it. I think I can do that. I'll just use something similar to the EIP logging Xen has for finding out what triggered the wtpt flushes. Thanks, Andrew