From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jean Guyader Subject: Re: [PATCH] permute with 2MB chunk Date: Tue, 25 Mar 2008 12:34:27 +0000 Message-ID: <47E8F153.4020307@eu.citrix.com> References: Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------070104090909070909080807" Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Ian Pratt Cc: xen-devel@lists.xensource.com, Keir Fraser , "Cui, Dexuan" List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --------------070104090909070909080807 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Ian Pratt wrote: >> We also tested building an HVM guest with the permuted ordering of >> pages, versus reverse ordering, versus normal ordering. Only the > permuted >> ordering showed the problem. We assume that the permute() function has > an >> unfortunate interaction with the memory allocator in certain HVM guest > OSes, >> causing poor cache utilisation. > > It's still very odd that the permutation fn only seems to effect Linux > running as a HVM guest and not as a PV guest. I still think there's > something we're not quite understanding. > > Jean: have you definitely verified that building a domain with the > permute function does not affect Linux PV guests? > Here a new version of the permute patch, it has to be applied instead of the previous one. Now it works with PV guests, sorry for the delay. Signed-off-by: Jean Guyader -- Jean Guyader --------------070104090909070909080807 Content-Type: text/x-diff; name="fix-permute2.patch" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="fix-permute2.patch" diff -r 76c9cf11ce23 tools/libxc/xc_domain_save.c --- a/tools/libxc/xc_domain_save.c Fri Mar 21 09:45:34 2008 +0000 +++ b/tools/libxc/xc_domain_save.c Tue Mar 25 12:31:42 2008 +0000 @@ -123,6 +123,32 @@ static inline int count_bits ( int nr, v for ( i = 0; i < (nr / (sizeof(unsigned long)*8)); i++, p++ ) count += hweight32(*p); return count; +} + +static inline int permute(unsigned long i, unsigned long nr, unsigned long order_nr) +{ + /* Need a simple permutation function so that we scan pages in a + pseudo random order, enabling us to get a better estimate of + the domain's page dirtying rate as we go (there are often + contiguous ranges of pfns that have similar behaviour, and we + want to mix them up. */ + + unsigned char keep = 9; /* chunk of 2 MB */ + unsigned char shift_high = (order_nr - keep) / 2; + unsigned char shift_low = order_nr - keep - (order_nr - keep) / 2; + + /* Check if the permutation gives an out of range number. */ + do + { + unsigned long high = (i >> (keep + shift_low)); + unsigned long low = (i >> keep) & ((1 << shift_low) - 1); + i = (i & ((1 << keep) - 1)) | + (low << (shift_high + keep)) | (high << keep); + } + while (i >= nr); + + + return (i); } static uint64_t tv_to_us(struct timeval *new) @@ -735,6 +761,7 @@ static xen_pfn_t *map_and_save_p2m_table p2m_frame_list[i/FPP] = mfn_to_pfn(p2m_frame_list[i/FPP]); } + memset(&ctxt, 0, sizeof (ctxt)); if ( xc_vcpu_getcontext(xc_handle, dom, 0, &ctxt.c) ) { ERROR("Could not get vcpu context"); @@ -828,6 +855,8 @@ int xc_domain_save(int xc_handle, int io /* base of the region in which domain memory is mapped */ unsigned char *region_base = NULL; + + int order_nr = 0; /* bitmap of pages: - that should be sent this iteration (unless later marked as skip); @@ -937,6 +966,11 @@ int xc_domain_save(int xc_handle, int io /* pretend we sent all the pages last iteration */ sent_last_iter = p2m_size; + + /* calculate the power of 2 order of p2m_size, e.g. + 15->4 16->4 17->5 */ + for ( i = p2m_size-1, order_nr = 0; i ; i >>= 1, order_nr++ ) + continue; /* Setup to_send / to_fix and to_skip bitmaps */ to_send = malloc(BITMAP_SIZE); @@ -1088,7 +1122,7 @@ int xc_domain_save(int xc_handle, int io (batch < MAX_BATCH_SIZE) && (N < p2m_size); N++ ) { - int n = N; + int n = permute(N, p2m_size, order_nr); if ( debug ) { --------------070104090909070909080807 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xensource.com http://lists.xensource.com/xen-devel --------------070104090909070909080807--