Re: [Xen-devel] alloc_heap_pages is low efficient with = more CPUs

From mboxrd@z Thu Jan 1 00:00:00 1970 From: tupeng212 Subject: Re: alloc_heap_pages is low efficient with more CPUs Date: Fri, 12 Oct 2012 20:24:16 +0800 Message-ID: <201210122024123436447@gmail.com> References: Reply-To: tupeng212 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============9116212788730668343==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Keir Fraser , xen-devel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============9116212788730668343== Content-Type: multipart/alternative; boundary="----=_001_NextPart540553674733_=----" This is a multi-part message in MIME format. ------=_001_NextPart540553674733_=---- Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 V2l0aCAxNiBDUFVzIHlvdSBmaW5kIGRvbWFpbiBzdGFydHVwIHRha2VzIDNzIGFsd2F5cy4gDQo6 Tm8NCldpdGggMTZDUFVzLCBmaXJzdCAwLjNzLCBzZWNvbmQgM3MNCldpdGggNjRDUFVzLCBmaXJz dCAzcywgc2Vjb25kIDMwcy4NCg0KV2l0aCA2NCBDUFVzIHlvdSBmaW5kIGl0IHRha2VzIDNzIGZp cnN0IHRpbWUsIHRoZW4gMzBzIGluIGZ1dHVyZT8gDQo6IFllcw0KDQpBbmQgdGhpcyBpcyBkdWUg dG8gY29zdCBvZiB0bGJmbHVzaF9maWx0ZXIoKSAobm90IGFjdHVhbCBUTEIgZmx1c2hlcywgYmVj YXVzZSB5b3UgYWx3YXlzIGVuZCB1cCB3aXRoIG1hc2s9MCk/IA0KOiBZZXMsIGl0IGNvc3RzIG11 Y2ggaW4gdGxiZmx1c2hfZmlsdGVyKCkgaW4gdGhlIGp1ZGdlbWVudC4NClRMQiBmbHVzaGluZyBp cyByZWFsbHkgdmVyeSBmYXN0LCBpdCBqdXN0IHNlbmRzIGEgSVBJIHRvIHJlbGF0ZWQgQ1BVLg0K SW4gdGhlIHN0YXJ0aW5nIHByb2Nlc3MncyBhbGxvY2F0aW9uLCBpdCBhbHdheXMgZW5kcyB1cCB3 aXRoIG1hc2s9MCB3aGljaCBzZWVtcyBuZWVkbGVzcy4NCg0KSWYgdGxiZmx1c2hfZmlsdGVyKCkg d2VyZSB0aGF0IGV4cGVuc2l2ZSBJ4oCZZCBleHBlY3QgdGhlIDE2LUNQVSBjYXNlIHRvIGhhdmUg c2xvd2Rvd24gYWZ0ZXIgdGhlIGZpcnN0IGRvbWFpbiBzdGFydHVwLCB0b28uDQo6IFllcywgeW91 IGFyZSByaWdodCwgMTZDUFUgc2xvd3MgZG93biB0b28gYWZ0ZXIgaXRzIGZpcnN0IHN0YXJ0dXAu DQoNClRoZSByZWFzb24gaXMgdmVyeSBjbGVhciwgSSBoYXZlIGRpc2N1c3NlZCBpdCB3aXRoIG90 aGVycywgdGxiZmx1c2hfZmlsdGVyKCkgaXMgbG93IGVmZmljaWVudCBpcyBubyBkb3VidCwgDQpC dXQgSSBkb24ndCBrbm93IGhvdyB0byBpbXByb3ZlIGl0IC4NCg0KYW5kIEkgYWxzbyB1c2VkICB4 ZW4gb3Byb2ZpbGUgdG8gZmluZCB0aGUgZm9sbG93aW5nIHR3byBmdW5jdGlvbnMgYXJlIGNhbGxl ZCBoaWdoIGZyZXF1ZW50bHk6DQphbGxvY19oZWFwX3BhZ2VzOiA0MCUNCl9fbmV4dF9jcHUgOiAy MCUNCm90aGVyczogMC54JQ0KLi4uLi4NCmFsbG9jX2hlYXBfcGFnZXMgLT4gdGxiZmx1c2hfZmls dGVyIC0+IGZvcl9lYWNoX2NwdV9tYXNrIG5leHRfY3B1ICAtPiBfX25leHRfY3B1IA0KaXQgc2Vl bXMgdHJhdmVsaW5nIGFtb25nIENQVXMgaXMgZXhwZW5zaXZlLg== ------=_001_NextPart540553674733_=---- Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable =EF=BB=BF Re: [Xen-devel] alloc_heap_pages is low efficient with = more CPUs

With 16 CPUs you find domain startup ta= kes 3s=20 always.

:No

With 16CPUs, first 0.3s, second 3s

With 64CPUs, first 3s, second 30s.

With 64 CPUs you find it takes 3s first= time,=20 then 30s in future?

: Yes

And this is due to cost of tlbflush_fil= ter() (not=20 actual TLB flushes, because you always end up with mask=3D0)?

: Yes, it costs much in tlbflush_filter= () in the=20 judgement.

TLB flushing is really very fast, it ju= st sends a=20 IPI to related CPU.

In the starting process's allocation, i= t always=20 ends up with mask=3D0 which seems needless.

If tlbflush_filter() were that expensiv= e I=E2=80=99d=20 expect the 16-CPU case to have slowdown after the first domain startup,=20 too.

: Yes, you are right, 16CPU slows down = too after=20 its first startup.

The reason is very clear, I have discus= sed it=20 with others, tlbflush_filter() is low efficient is no doubt,

But I don't know how to improve it .

and I also used xen opr= ofile to=20 find the following two functions are called high frequently:

alloc_heap_pages: 40%

__next_cpu : 20%

others: 0.x%

.....

alloc_heap_pages -> tlbflush_filter -> for_each_cpu_mask=20 next_cpu -> __next_cpu

it seems traveling among CPUs is expensive.

------=_001_NextPart540553674733_=------ --===============9116212788730668343== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============9116212788730668343==--