From mboxrd@z Thu Jan 1 00:00:00 1970 From: Keir Fraser Subject: Re: alloc_heap_pages is low efficient with more CPUs Date: Sat, 13 Oct 2012 09:59:19 +0100 Message-ID: References: <2012101314464865629623@gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="B_3432967176_20080749" Return-path: In-Reply-To: <2012101314464865629623@gmail.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: tupeng212 Cc: xen-devel List-Id: xen-devel@lists.xenproject.org > This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. --B_3432967176_20080749 Content-type: multipart/alternative; boundary="B_3432967176_20047710" --B_3432967176_20047710 Content-type: text/plain; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable If the allocations are 2M size, we can do better quite easily I think. Please try the attached patch. -- Keir On 13/10/2012 07:46, "tupeng212" wrote: > What if you replace tlbflush_filter() call with cpus_clear(&extra_cpus_ma= sk)? > : you mean just clear it, maybe a little violent.., you 'd like to do = it at > any other place. > =20 > I assume you see lots of looping in one of those two functions, but only > single-page-at-a-time calls into alloc_domheap_pages()->alloc_heap_pages(= )? > : In populate_physmap, all pages are 2M size, > static void populate_physmap(struct memop_args *a) > { > for ( i =3D a->nr_done; i < a->nr_extents; i++ ) > { > page =3D alloc_domheap_pages(d, a->extent_order, a->memflags) ->alloc_heap_= pages > ; //a->extent_order =3D 9, always 2M size > } > //you mean move that block and TLB-flush here to avoid for loop ? > } >=20 > tupeng212 > =20 > From: Keir Fraser > Date: 2012-10-13 14:30 > To: tupeng212 > Subject: Re: [Xen-devel] alloc_heap_pages is low efficient with more CPUs > What if you replace tlbflush_filter() call with cpus_clear(&extra_cpus_ma= sk)? > Seems a bit silly to do, but I=B9d like to know how much a few cpumask > operations per page is costing (most are of course much quicker than > tlbflush_filter as they operate on 64 bits per iteration, rather than one= bit > per iteration). >=20 > If that is suitably fast, I think we can have a go at fixing this by pull= ing > the TLB-flush logic out of alloc_heap_pages() and into the loops in > increwase_reservation() and populate_physmap() in common/memory.c. I assu= me > you see lots of looping in one of those two functions, but only > single-page-at-a-time calls into alloc_domheap_pages()->alloc_heap_pages(= )? >=20 > -- Keir >=20 > On 13/10/2012 07:21, "tupeng212" wrote: >=20 >> If the tlbflush_filter() and cpumask_or() lines are commented out from = the >> if(need_tlbflush) block in alloc_heap_pages(), what are the domain crea= tion >> times like then? >> : You mean removing these code from alloc_heap_pages, then try it. >> I didn't do it as you said, but I calculated the whole time of >> if(need_tlbflush) block >> by using time1=3DNOW() ...block ... time2=3DNOW(), time=3Dtime2-time1, its un= it is >> ns, and s =3D ns * 10^9 >> it occupy high rate of the whole time. whole starting time is 30s, then= this >> block may be 29s. >> =20 >> By the way it looks like you are not using xen-unstable or >> xen-4.2, can you try with one of these later versions of Xen? >> : fortunately, other groups have to use xen-4.2, we have repeated this >> experiment on=20 >> that source code too, it changed nothing, time is still very long in se= cond >> starting. >> =20 >> =20 >>=20 >> tupeng >> =20 >> =20 >>=20 >=20 --B_3432967176_20047710 Content-type: text/html; charset="ISO-8859-1" Content-transfer-encoding: quoted-printable Re: [Xen-devel] alloc_heap_pages is low efficient with more CPUs</TI= TLE> </HEAD> <BODY> <FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><SPAN STYLE=3D'font-size:11pt= '>If the allocations are 2M size, we can do better quite easily I think. Ple= ase try the attached patch.<BR> <BR>  -- Keir<BR> <BR> On 13/10/2012 07:46, "tupeng212" <<a href=3D"tupeng212@gmail.com= ">tupeng212@gmail.com</a>> wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><= SPAN STYLE=3D'font-size:11pt'><FONT COLOR=3D"#000080">What if you replace tlbflu= sh_filter() call with cpus_clear(&extra_cpus_mask)? <BR> :  you mean just clear it,  maybe a little violent..,  you '= d like to do it at any other place.<BR> </FONT></SPAN><FONT COLOR=3D"#000080"><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:1= 0pt'> <BR> </SPAN></FONT><SPAN STYLE=3D'font-size:11pt'>I assume you see lots of looping= in one of those two functions, but only single-page-at-a-time calls into al= loc_domheap_pages()->alloc_heap_pages()?<BR> :  In populate_physmap,  all pages are 2M size, <BR> </SPAN><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'>static void populate_phy= smap(struct memop_args *a) <BR> {<BR>     for ( i =3D a->nr_done; i < a->nr_extents; i= ++ )<BR>     {<BR> page =3D alloc_domheap_pages(d, a->extent_order, a->memflags) ->allo= c_heap_pages ;  //a->extent_order =3D 9, always 2M size<BR> }<BR> //you mean move that block and TLB-flush here to avoid for loop ?<BR> }<BR> <HR ALIGN=3DLEFT SIZE=3D"1" WIDTH=3D"100%">tupeng212<BR>  <BR> </SPAN></FONT></FONT><FONT SIZE=3D"1"><SPAN STYLE=3D'font-size:9pt'><B>From:</B= > Keir Fraser <<a href=3D"mailto:keir.xen@gmail.com">mailto:keir.xen@gmail.= com</a>> <BR> <B>Date:</B> 2012-10-13 14:30<BR> <B>To:</B> tupeng212 <<a href=3D"mailto:tupeng212@gmail.com">mailto:tupeng= 212@gmail.com</a>> <BR> <B>Subject:</B> Re: [Xen-devel] alloc_heap_pages is low efficient with more= CPUs<BR> </SPAN></FONT><SPAN STYLE=3D'font-size:11pt'>What if you replace tlbflush_fil= ter() call with cpus_clear(&extra_cpus_mask)? Seems a bit silly to do, b= ut I’d like to know how much a few cpumask operations per page is cost= ing (most are of course much quicker than tlbflush_filter as they operate on= 64 bits per iteration, rather than one bit per iteration).<BR> <BR> If that is suitably fast, I think we can have a go at fixing this by pullin= g the TLB-flush logic out of alloc_heap_pages() and into the loops in increw= ase_reservation() and populate_physmap() in common/memory.c. I assume you se= e lots of looping in one of those two functions, but only single-page-at-a-t= ime calls into alloc_domheap_pages()->alloc_heap_pages()?<BR> <BR>   -- Keir<BR> <BR> On 13/10/2012 07:21, "tupeng212" <<a href=3D"tupeng212@gmail.com= ">tupeng212@gmail.com</a>> wrote:<BR> <BR> </SPAN></FONT><BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helvetica, Arial"><= FONT COLOR=3D"#000080"><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10pt'>If the &nbs= p;tlbflush_filter() and cpumask_or() lines are commented out from  the<= BR> if(need_tlbflush) block in alloc_heap_pages(), what are the domain  cr= eation<BR> times like then? <BR> : You mean removing these code from  alloc_heap_pages, then try it.<BR= > I didn't do it as you said, but I calculated  the whole time of if(nee= d_tlbflush) block<BR> by using time1=3DNOW() ...block ...  time2=3DNOW(), time=3Dtime2-time1, its = unit is ns, and s =3D ns * 10^9<BR> it occupy  high rate of the whole time. whole starting time is 30s, th= en this block may  be 29s.<BR>  <BR> By the way it looks like you are not using xen-unstable  or<BR> xen-4.2, can you try with one of these later versions of Xen?<BR> :  fortunately, other groups have to use xen-4.2, we have repeated thi= s  experiment on <BR> that source code too, it changed nothing, time is still very  long in = second starting.<BR>  <BR>  <BR> <HR ALIGN=3DLEFT SIZE=3D"1" WIDTH=3D"100%"> tupeng<BR>  <BR>  <BR> <BR> </SPAN></FONT></FONT></FONT></BLOCKQUOTE><FONT FACE=3D"Calibri, Verdana, Helv= etica, Arial"><FONT COLOR=3D"#000080"><FONT SIZE=3D"2"><SPAN STYLE=3D'font-size:10= pt'><BR> </SPAN></FONT></FONT></FONT></BLOCKQUOTE> </BODY> </HTML> --B_3432967176_20047710-- --B_3432967176_20080749 Content-type: application/octet-stream; name="00-reduce-tlbflush_filter" Content-disposition: attachment; filename="00-reduce-tlbflush_filter" Content-transfer-encoding: base64 ZGlmZiAtciBhMTU1OTZhNjE5ZWQgeGVuL2NvbW1vbi9wYWdlX2FsbG9jLmMKLS0tIGEveGVu L2NvbW1vbi9wYWdlX2FsbG9jLmMJVGh1IE9jdCAwNCAxMDo0NDo0MyAyMDEyICswMjAwCisr KyBiL3hlbi9jb21tb24vcGFnZV9hbGxvYy5jCVNhdCBPY3QgMTMgMDk6NTc6MjYgMjAxMiAr MDEwMApAQCAtMzAzLDkgKzMwMywxMCBAQCBzdGF0aWMgc3RydWN0IHBhZ2VfaW5mbyAqYWxs b2NfaGVhcF9wYWdlCiAgICAgdW5zaWduZWQgaW50IGZpcnN0X25vZGUsIGksIGosIHpvbmUg PSAwLCBub2RlbWFza19yZXRyeSA9IDA7CiAgICAgdW5zaWduZWQgaW50IG5vZGUgPSAodWlu dDhfdCkoKG1lbWZsYWdzID4+IF9NRU1GX25vZGUpIC0gMSk7CiAgICAgdW5zaWduZWQgbG9u ZyByZXF1ZXN0ID0gMVVMIDw8IG9yZGVyOwotICAgIGNwdW1hc2tfdCBleHRyYV9jcHVzX21h c2ssIG1hc2s7CiAgICAgc3RydWN0IHBhZ2VfaW5mbyAqcGc7CiAgICAgbm9kZW1hc2tfdCBu b2RlbWFzayA9IChkICE9IE5VTEwgKSA/IGQtPm5vZGVfYWZmaW5pdHkgOiBub2RlX29ubGlu ZV9tYXA7CisgICAgYm9vbF90IG5lZWRfdGxiZmx1c2ggPSAwOworICAgIHVpbnQzMl90IHRs YmZsdXNoX3RpbWVzdGFtcCA9IDA7CiAKICAgICBpZiAoIG5vZGUgPT0gTlVNQV9OT19OT0RF ICkKICAgICB7CkBAIC00MTcsMjAgKzQxOCwxOSBAQCBzdGF0aWMgc3RydWN0IHBhZ2VfaW5m byAqYWxsb2NfaGVhcF9wYWdlCiAgICAgaWYgKCBkICE9IE5VTEwgKQogICAgICAgICBkLT5s YXN0X2FsbG9jX25vZGUgPSBub2RlOwogCi0gICAgY3B1c19jbGVhcihtYXNrKTsKLQogICAg IGZvciAoIGkgPSAwOyBpIDwgKDEgPDwgb3JkZXIpOyBpKysgKQogICAgIHsKICAgICAgICAg LyogUmVmZXJlbmNlIGNvdW50IG11c3QgY29udGludW91c2x5IGJlIHplcm8gZm9yIGZyZWUg cGFnZXMuICovCiAgICAgICAgIEJVR19PTihwZ1tpXS5jb3VudF9pbmZvICE9IFBHQ19zdGF0 ZV9mcmVlKTsKICAgICAgICAgcGdbaV0uY291bnRfaW5mbyA9IFBHQ19zdGF0ZV9pbnVzZTsK IAotICAgICAgICBpZiAoIHBnW2ldLnUuZnJlZS5uZWVkX3RsYmZsdXNoICkKKyAgICAgICAg aWYgKCBwZ1tpXS51LmZyZWUubmVlZF90bGJmbHVzaCAmJgorICAgICAgICAgICAgIChwZ1tp XS50bGJmbHVzaF90aW1lc3RhbXAgPD0gdGxiZmx1c2hfY3VycmVudF90aW1lKCkpICYmCisg ICAgICAgICAgICAgKCFuZWVkX3RsYmZsdXNoIHx8CisgICAgICAgICAgICAgIChwZ1tpXS50 bGJmbHVzaF90aW1lc3RhbXAgPiB0bGJmbHVzaF90aW1lc3RhbXApKSApCiAgICAgICAgIHsK LSAgICAgICAgICAgIC8qIEFkZCBpbiBleHRyYSBDUFVzIHRoYXQgbmVlZCBmbHVzaGluZyBi ZWNhdXNlIG9mIHRoaXMgcGFnZS4gKi8KLSAgICAgICAgICAgIGNwdXNfYW5kbm90KGV4dHJh X2NwdXNfbWFzaywgY3B1X29ubGluZV9tYXAsIG1hc2spOwotICAgICAgICAgICAgdGxiZmx1 c2hfZmlsdGVyKGV4dHJhX2NwdXNfbWFzaywgcGdbaV0udGxiZmx1c2hfdGltZXN0YW1wKTsK LSAgICAgICAgICAgIGNwdXNfb3IobWFzaywgbWFzaywgZXh0cmFfY3B1c19tYXNrKTsKKyAg ICAgICAgICAgIG5lZWRfdGxiZmx1c2ggPSAxOworICAgICAgICAgICAgdGxiZmx1c2hfdGlt ZXN0YW1wID0gcGdbaV0udGxiZmx1c2hfdGltZXN0YW1wOwogICAgICAgICB9CiAKICAgICAg ICAgLyogSW5pdGlhbGlzZSBmaWVsZHMgd2hpY2ggaGF2ZSBvdGhlciB1c2VzIGZvciBmcmVl IHBhZ2VzLiAqLwpAQCAtNDQwLDEwICs0NDAsMTUgQEAgc3RhdGljIHN0cnVjdCBwYWdlX2lu Zm8gKmFsbG9jX2hlYXBfcGFnZQogCiAgICAgc3Bpbl91bmxvY2soJmhlYXBfbG9jayk7CiAK LSAgICBpZiAoIHVubGlrZWx5KCFjcHVzX2VtcHR5KG1hc2spKSApCisgICAgaWYgKCBuZWVk X3RsYmZsdXNoICkKICAgICB7Ci0gICAgICAgIHBlcmZjX2luY3IobmVlZF9mbHVzaF90bGJf Zmx1c2gpOwotICAgICAgICBmbHVzaF90bGJfbWFzaygmbWFzayk7CisgICAgICAgIGNwdW1h c2tfdCBtYXNrID0gY3B1X29ubGluZV9tYXA7CisgICAgICAgIHRsYmZsdXNoX2ZpbHRlciht YXNrLCB0bGJmbHVzaF90aW1lc3RhbXApOworICAgICAgICBpZiAoICFjcHVzX2VtcHR5KG1h c2spICkKKyAgICAgICAgeworICAgICAgICAgICAgcGVyZmNfaW5jcihuZWVkX2ZsdXNoX3Rs Yl9mbHVzaCk7CisgICAgICAgICAgICBmbHVzaF90bGJfbWFzaygmbWFzayk7CisgICAgICAg IH0KICAgICB9CiAKICAgICByZXR1cm4gcGc7Cg== --B_3432967176_20080749 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --B_3432967176_20080749--