From mboxrd@z Thu Jan 1 00:00:00 1970 From: tupeng212 Subject: alloc_heap_pages is low efficient with more CPUs Date: Thu, 11 Oct 2012 23:18:52 +0800 Message-ID: <201210112318468920764@gmail.com> Reply-To: tupeng212 Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="===============5297671841591368988==" Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel List-Id: xen-devel@lists.xenproject.org This is a multi-part message in MIME format. --===============5297671841591368988== Content-Type: multipart/alternative; boundary="----=_001_NextPart007376380103_=----" This is a multi-part message in MIME format. ------=_001_NextPart007376380103_=---- Content-Type: text/plain; charset="gb2312" Content-Transfer-Encoding: base64 SSBhbSBjb25mdXNlZCB3aXRoIGEgcHJvYmxlbToNCkkgaGF2ZSBhIGJsYWRlIHdpdGggNjQgcGh5 c2ljYWwgQ1BVcyBhbmQgNjRHIHBoeXNpY2FsIFJBTSwgYW5kIGRlZmluZWQgb25seSBvbmUgVk0g d2l0aCAxIENQVSBhbmQgNDBHIFJBTS4NCkZvciB0aGUgZmlyc3QgdGltZSBJIHN0YXJ0ZWQgdGhl IFZNLCBpdCBqdXN0IHRvb2sgM3MsIEJ1dCBmb3IgdGhlIHNlY29uZCBzdGFydGluZyBpdCB0b29r IDMwcy4gDQpBZnRlciBzdHVkaWVkIGl0IGJ5IHByaW50aW5nIGxvZywgSSBoYXZlIGxvY2F0ZWQg YSBwbGFjZSBpbiB0aGUgaHlwZXJ2aXNvciB3aGVyZSBjb3N0IHRvbyBtdWNoIHRpbWUsIA0Kb2Nj dXBpZWQgOTglIG9mIHRoZSB3aG9sZSBzdGFydGluZyB0aW1lLg0KDQp4ZW4vY29tbW9uL3BhZ2Vf YWxsb2MuYyANCi8qIEFsbG9jYXRlIDJeQG9yZGVyIGNvbnRpZ3VvdXMgcGFnZXMuICovDQpzdGF0 aWMgc3RydWN0IHBhZ2VfaW5mbyAqYWxsb2NfaGVhcF9wYWdlcygNCiAgICB1bnNpZ25lZCBpbnQg em9uZV9sbywgdW5zaWduZWQgaW50IHpvbmVfaGksDQogICAgdW5zaWduZWQgaW50IG5vZGUsIHVu c2lnbmVkIGludCBvcmRlciwgdW5zaWduZWQgaW50IG1lbWZsYWdzKQ0Kew0KICAgICAgICBpZiAo IHBnW2ldLnUuZnJlZS5uZWVkX3RsYmZsdXNoICkNCiAgICAgICAgew0KICAgICAgICAgICAgLyog QWRkIGluIGV4dHJhIENQVXMgdGhhdCBuZWVkIGZsdXNoaW5nIGJlY2F1c2Ugb2YgdGhpcyBwYWdl LiAqLw0KICAgICAgICAgICAgY3B1c19hbmRub3QoZXh0cmFfY3B1c19tYXNrLCBjcHVfb25saW5l X21hcCwgbWFzayk7DQogICAgICAgICAgICB0bGJmbHVzaF9maWx0ZXIoZXh0cmFfY3B1c19tYXNr LCBwZ1tpXS50bGJmbHVzaF90aW1lc3RhbXApOw0KICAgICAgICAgICAgY3B1c19vcihtYXNrLCBt YXNrLCBleHRyYV9jcHVzX21hc2spOw0KICAgICAgICB9DQoNCn0NCg0KMSBpbiB0aGUgZmlyc3Qg c3RhcnRpbmcsIG1vc3Qgb2YgbmVlZF90bGJmbHVzaD1OVUxMLCBzbyBjb3N0IGxpdHRsZTsgaW4g dGhlIHNlY29uZCBvbmUsIG1vc3Qgb2YgUkFNIGhhdmUgYmVlbiB1c2VkIA0KICB0aHVzIG1ha2Vz IG5lZWRfdGxiZmx1c2g9dHJ1ZSwgc28gY29zdCBtdWNoLg0KDQoyIGJ1dCBJIHJlcGVhdGVkIHRo ZSBzYW1lIGV4cGVyaW1lbnQgaW4gYW5vdGhlciBibGFkZSB3aGljaCBjb250YWlucyAxNiBwaHlz aWNhbCBDUFVzIGFuZCA2NEcgcGh5c2ljYWwgUkFNLCB0aGUgc2Vjb25kDQogIHN0YXJ0aW5nIGNv c3QgM3MuIEFmdGVyIEkgdHJhY2VkIHRoZSBwcm9jZXNzIGJldHdlZW4gdGhlIHR3byBzZWNvbmQg c3RhcnRpbmdzLCBmb3VuZCB0aGF0IGNvdW50IGVudGVyaW5nIGludG8gdGhlIGp1ZGdlbWVudCBv Zg0KICBwZ1tpXS51LmZyZWUubmVlZF90bGJmbHVzaCBpcyB0aGUgc2FtZSwgYnV0IG51bWJlciBv ZiBDUFVzIGxlYWRzIHRvIHRoZSBkaWZmZXJlbmNlLg0KDQozIFRoZSBjb2RlIEkgcGFzdGVkIGFp bXMgdG8gY29tcHV0ZSBhIG1hc2sgdG8gZGV0ZXJtaW5lIHdoZXRoZXIgaXQgc2hvdWxkIGZsdXNo IENQVSdzIFRMQi4gSSB0cmFjZWQgdGhlIHZhbHVlcyBpbiBzdGFydGluZyBwZXJpb2QgYmVsb3c6 DQoNCmNwdXNfYW5kbm90KGV4dHJhX2NwdXNfbWFzaywgY3B1X29ubGluZV9tYXAsIG1hc2spOw0K Ly9hZnRlciwgbWFzaz0wLCBjcHVfb25saW5lX21hcD0weEZGRkZGRkZGRkZGRkZGRkYsIGV4dHJh X2NwdXNfbWFzaz0weEZGRkZGRkZGRkZGRkZGRkYNCnRsYmZsdXNoX2ZpbHRlcihleHRyYV9jcHVz X21hc2ssIHBnW2ldLnRsYmZsdXNoX3RpbWVzdGFtcCk7DQovL2FmdGVyLCBtYXNrPTAsIGV4dHJh X2NwdXNfbWFzaz0wDQpjcHVzX29yKG1hc2ssIG1hc2ssIGV4dHJhX2NwdXNfbWFzayk7DQovL2Fm dGVyLCBtYXNrPTAsIGV4dHJhX2NwdXNfbWFzaz0wDQoNCiAgZXZlcnkgdGltZSBpdCBzdGFydHMg d2l0aCBtYXNrPTAsIGFuZCBlbmRzIHdpdGggdGhlIHNhbWUgcmVzdWx0IG1hc2s9MCwgc28gbGVh ZHMgdG8gZmx1c2ggQ1BVJ3MgVExCIGRlZmluaXRlbHksICANCiAgd2hpY2ggc2VlbXMgbWVhbmlu Z2xlc3MgaW4gdGhlIHN0YXJpbmcgcHJvY2Vzcy4NCg0KNCBUaGUgcHJvYmxlbSBpcywgdGhpcyBz ZWVtZWQgbWVhbmluZ2xlc3MgY29kZSBpcyB2ZXJ5IHRpbWUtY29uc3VtaW5nLCBDUFVzIGdldCBt b3JlLCB0aW1lIGNvc3RzIG1vcmUsIGl0IHRha2VzIDMwcyBoZXJlIGluIG15IGJsYWRlDQogIHdp dGggNjQgQ1BVcyB3aGljaCBtYXkgbmVlZCBzb21lIHNvbHV0aW9uIHRvIGltcHJvdmUgdGhlIGVm ZmljaWVuY3kuDQogIA0KDQoNCg0KdHVwZW5n ------=_001_NextPart007376380103_=---- Content-Type: text/html; charset="gb2312" Content-Transfer-Encoding: quoted-printable
I am confused with a problem:
I have a blade with 64 physical CP= Us and 64G physical RAM, and defined on= ly one VM with 1 CPU and 40G RAM.<= /DIV>
For the first time I started the V= M, it just took 3s, But for the se= cond starting it took 30s. 
After studied it by printing log, I&nbs= p;have located a place in the hypervisor&nbs= p;where cost too much time, 
occupied 98% of the whole starting time= .
 
xen/common/page_alloc.c
/* Allocate 2^@order contiguous pages. */
static struct page_info *alloc_heap_pages(
    unsigned int zone_lo, unsigned=  int zone_hi,
    unsigned int node, unsigned&nb= sp;int order, unsigned int memflags)
{
        if ( pg[i].= u.free.need_tlbflush )
        {
           &nb= sp;/* Add in extra CPUs that need flush= ing because of this page. */
           &nb= sp;cpus_andnot(extra_cpus_mask, cpu_online_map, mask);
           &nb= sp;tlbflush_filter(extra_cpus_mask, pg[i].tlbflush_timestamp);
           &nb= sp;cpus_or(mask, mask, extra_cpus_mask);
        }
 
}
 
1 in the first starting, most of n= eed_tlbflush=3DNULL, so cost little; in the = second one, most of RAM have been used&= nbsp;
  thus makes need_tlbflush=3Dtrue, so c= ost much.
 
2 but I repeated the same experiment&nb= sp;in another blade which contains 16 physic= al CPUs and 64G physical RAM, the secon= d
  starting cost 3s. After I traced=  the=20 process between the two second=20 startings, found that count entering into th= e judgement=20 of
  pg[i].u.free.need_tlbflush is the same,&nb= sp;but number of CPUs leads to the diff= erence.
 
3 The code I pasted aims to comput= e a mask to determine whether it should=  flush CPU's TLB. I traced the values&n= bsp;in starting period below:
 
cpus_andnot(extra_cpus_mask, cpu_online_map, mask);
//after, mask=3D0, cpu_online_map=3D0xFFFFFFFFFFFFFFFF,&n= bsp;extra_cpus_mask=3D0xFFFFFFFFFFFFFFFF
tlbflush_filter(extra_cpus_mask, pg[i].tlbflush_timestamp);
//after, mask=3D0, extra_cpus_mask=3D0
cpus_or(mask, mask, extra_cpus_mask);
//after, mask=3D0, extra_cpus_mask=3D0
 
  every time it starts with mask= =3D0, and ends with the same result mas= k=3D0, so leads to flush CPU's TLB defi= nitely,  
 =20 which seems meaningless in the staring proce= ss.
 
4 The problem is, this seemed meaningle= ss code is very time-consuming, CPUs get&nbs= p;more, time costs more, it takes 30s h= ere in my blade
  with 64 CPUs which may need = ;some solution to improve the efficiency.
  
tupeng
------=_001_NextPart007376380103_=------ --===============5297671841591368988== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============5297671841591368988==--