From mboxrd@z Thu Jan 1 00:00:00 1970 From: tupeng212 Subject: Re: alloc_heap_pages is low efficient with more CPUs Date: Sat, 13 Oct 2012 14:46:54 +0800 Message-ID: <2012101314464865629623@gmail.com> References: Reply-To: tupeng212 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============1305624968587899190==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Keir Fraser Cc: xen-devel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============1305624968587899190== Content-Type: multipart/alternative; boundary="----=_001_NextPart606012520113_=----" This is a multi-part message in MIME format. ------=_001_NextPart606012520113_=---- Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 V2hhdCBpZiB5b3UgcmVwbGFjZSB0bGJmbHVzaF9maWx0ZXIoKSBjYWxsIHdpdGggY3B1c19jbGVh cigmZXh0cmFfY3B1c19tYXNrKT8gDQo6ICB5b3UgbWVhbiBqdXN0IGNsZWFyIGl0LCAgbWF5YmUg YSBsaXR0bGUgdmlvbGVudC4uLCAgeW91ICdkIGxpa2UgdG8gZG8gaXQgYXQgYW55IG90aGVyIHBs YWNlLg0KDQpJIGFzc3VtZSB5b3Ugc2VlIGxvdHMgb2YgbG9vcGluZyBpbiBvbmUgb2YgdGhvc2Ug dHdvIGZ1bmN0aW9ucywgYnV0IG9ubHkgc2luZ2xlLXBhZ2UtYXQtYS10aW1lIGNhbGxzIGludG8g YWxsb2NfZG9taGVhcF9wYWdlcygpLT5hbGxvY19oZWFwX3BhZ2VzKCk/DQo6ICBJbiBwb3B1bGF0 ZV9waHlzbWFwLCAgYWxsIHBhZ2VzIGFyZSAyTSBzaXplLCANCnN0YXRpYyB2b2lkIHBvcHVsYXRl X3BoeXNtYXAoc3RydWN0IG1lbW9wX2FyZ3MgKmEpIA0Kew0KICAgIGZvciAoIGkgPSBhLT5ucl9k b25lOyBpIDwgYS0+bnJfZXh0ZW50czsgaSsrICkNCiAgICB7DQpwYWdlID0gYWxsb2NfZG9taGVh cF9wYWdlcyhkLCBhLT5leHRlbnRfb3JkZXIsIGEtPm1lbWZsYWdzKSAtPmFsbG9jX2hlYXBfcGFn ZXMgOyAgLy9hLT5leHRlbnRfb3JkZXIgPSA5LCBhbHdheXMgMk0gc2l6ZQ0KfQ0KLy95b3UgbWVh biBtb3ZlIHRoYXQgYmxvY2sgYW5kIFRMQi1mbHVzaCBoZXJlIHRvIGF2b2lkIGZvciBsb29wID8N Cn0NCg0KDQoNCnR1cGVuZzIxMg0KDQpGcm9tOiBLZWlyIEZyYXNlcg0KRGF0ZTogMjAxMi0xMC0x MyAxNDozMA0KVG86IHR1cGVuZzIxMg0KU3ViamVjdDogUmU6IFtYZW4tZGV2ZWxdIGFsbG9jX2hl YXBfcGFnZXMgaXMgbG93IGVmZmljaWVudCB3aXRoIG1vcmUgQ1BVcw0KV2hhdCBpZiB5b3UgcmVw bGFjZSB0bGJmbHVzaF9maWx0ZXIoKSBjYWxsIHdpdGggY3B1c19jbGVhcigmZXh0cmFfY3B1c19t YXNrKT8gU2VlbXMgYSBiaXQgc2lsbHkgdG8gZG8sIGJ1dCBJ4oCZZCBsaWtlIHRvIGtub3cgaG93 IG11Y2ggYSBmZXcgY3B1bWFzayBvcGVyYXRpb25zIHBlciBwYWdlIGlzIGNvc3RpbmcgKG1vc3Qg YXJlIG9mIGNvdXJzZSBtdWNoIHF1aWNrZXIgdGhhbiB0bGJmbHVzaF9maWx0ZXIgYXMgdGhleSBv cGVyYXRlIG9uIDY0IGJpdHMgcGVyIGl0ZXJhdGlvbiwgcmF0aGVyIHRoYW4gb25lIGJpdCBwZXIg aXRlcmF0aW9uKS4NCg0KSWYgdGhhdCBpcyBzdWl0YWJseSBmYXN0LCBJIHRoaW5rIHdlIGNhbiBo YXZlIGEgZ28gYXQgZml4aW5nIHRoaXMgYnkgcHVsbGluZyB0aGUgVExCLWZsdXNoIGxvZ2ljIG91 dCBvZiBhbGxvY19oZWFwX3BhZ2VzKCkgYW5kIGludG8gdGhlIGxvb3BzIGluIGluY3Jld2FzZV9y ZXNlcnZhdGlvbigpIGFuZCBwb3B1bGF0ZV9waHlzbWFwKCkgaW4gY29tbW9uL21lbW9yeS5jLiBJ IGFzc3VtZSB5b3Ugc2VlIGxvdHMgb2YgbG9vcGluZyBpbiBvbmUgb2YgdGhvc2UgdHdvIGZ1bmN0 aW9ucywgYnV0IG9ubHkgc2luZ2xlLXBhZ2UtYXQtYS10aW1lIGNhbGxzIGludG8gYWxsb2NfZG9t aGVhcF9wYWdlcygpLT5hbGxvY19oZWFwX3BhZ2VzKCk/DQoNCiAgLS0gS2Vpcg0KDQpPbiAxMy8x MC8yMDEyIDA3OjIxLCAidHVwZW5nMjEyIiA8dHVwZW5nMjEyQGdtYWlsLmNvbT4gd3JvdGU6DQoN Cg0KSWYgdGhlIHRsYmZsdXNoX2ZpbHRlcigpIGFuZCBjcHVtYXNrX29yKCkgbGluZXMgYXJlIGNv bW1lbnRlZCBvdXQgZnJvbSB0aGUNCmlmKG5lZWRfdGxiZmx1c2gpIGJsb2NrIGluIGFsbG9jX2hl YXBfcGFnZXMoKSwgd2hhdCBhcmUgdGhlIGRvbWFpbiBjcmVhdGlvbg0KdGltZXMgbGlrZSB0aGVu PyANCjogWW91IG1lYW4gcmVtb3ZpbmcgdGhlc2UgY29kZSBmcm9tIGFsbG9jX2hlYXBfcGFnZXMs IHRoZW4gdHJ5IGl0Lg0KSSBkaWRuJ3QgZG8gaXQgYXMgeW91IHNhaWQsIGJ1dCBJIGNhbGN1bGF0 ZWQgdGhlIHdob2xlIHRpbWUgb2YgaWYobmVlZF90bGJmbHVzaCkgYmxvY2sNCmJ5IHVzaW5nIHRp bWUxPU5PVygpIC4uLmJsb2NrIC4uLiB0aW1lMj1OT1coKSwgdGltZT10aW1lMi10aW1lMSwgaXRz IHVuaXQgaXMgbnMsIGFuZCBzID0gbnMgKiAxMF45DQppdCBvY2N1cHkgaGlnaCByYXRlIG9mIHRo ZSB3aG9sZSB0aW1lLiB3aG9sZSBzdGFydGluZyB0aW1lIGlzIDMwcywgdGhlbiB0aGlzIGJsb2Nr IG1heSBiZSAyOXMuDQogDQpCeSB0aGUgd2F5IGl0IGxvb2tzIGxpa2UgeW91IGFyZSBub3QgdXNp bmcgeGVuLXVuc3RhYmxlIG9yDQp4ZW4tNC4yLCBjYW4geW91IHRyeSB3aXRoIG9uZSBvZiB0aGVz ZSBsYXRlciB2ZXJzaW9ucyBvZiBYZW4/DQo6IGZvcnR1bmF0ZWx5LCBvdGhlciBncm91cHMgaGF2 ZSB0byB1c2UgeGVuLTQuMiwgd2UgaGF2ZSByZXBlYXRlZCB0aGlzIGV4cGVyaW1lbnQgb24gDQp0 aGF0IHNvdXJjZSBjb2RlIHRvbywgaXQgY2hhbmdlZCBub3RoaW5nLCB0aW1lIGlzIHN0aWxsIHZl cnkgbG9uZyBpbiBzZWNvbmQgc3RhcnRpbmcuDQogDQoNCg0KdHVwZW5nDQogDQog ------=_001_NextPart606012520113_=---- Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable =EF=BB=BF Re: [Xen-devel] alloc_heap_pages is low efficient with = more CPUs
What if you repl= ace=20 tlbflush_filter() call with cpus_clear(&extra_cpus_mask)?=20
:  you mean= just=20 clear it,  maybe a little violent..,  you 'd like to do it at an= y=20 other place.
&n= bsp;
I assume you see= lots of=20 looping in one of those two functions, but only single-page-at-a-time call= s into=20 alloc_domheap_pages()->alloc_heap_pages()?
:  In populate_physmap= , =20 all pages are 2M size,
static void populate_physmap(struct memop_args *a= )=20
{
    for ( i =3D a->nr_done= ; i < a->nr_extents; i++ )
    {
page =3D alloc_domheap_pages(d, = a->extent_order, a->memflags)=20 ->alloc_heap_pages ;  //a->extent_order =3D 9, always 2M size
}
//you mean move that block and TLB-flush h= ere to=20 avoid for loop ?
}
tupeng212
 
Date: 2012-10-13 14:30
To: tupeng212<= /DIV>
Subject: Re: [Xen-devel] alloc_heap_pages is low efficien= t with=20 more CPUs
tupeng212@gmail.com>=20 wrote:
If the=20 tlbflush_filter() and cpumask_or() lines are commented out from=20 the
if(need_tlbflush) block in alloc_heap_pages(), what are the domai= n=20 creation
times like then?
: You mean removing these code from=20 alloc_heap_pages, then try it.
I didn't do it as you said, but I calc= ulated=20 the whole time of if(need_tlbflush) block
by using time1=3DNOW() ...b= lock ...=20 time2=3DNOW(), time=3Dtime2-time1, its unit is ns, and s =3D ns * 10^9it occupy=20 high rate of the whole time. whole starting time is 30s, then this block= may=20 be 29s.
 
By the way it looks like you are not using xen-unst= able=20 or
xen-4.2, can you try with one of these later versions of Xen?
:= =20 fortunately, other groups have to use xen-4.2, we have repeated this=20 experiment on
that source code too, it changed nothing, time is stil= l very=20 long in second starting.
 
tupeng
 
 

------=_001_NextPart606012520113_=------ --===============1305624968587899190== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1305624968587899190==--