2. In dom0, only two netback q= ueues process are scheduled, other two process aren't scheduled.

<= /div>

&= nbsp; Are there any issues in my test? In theory, can we achieve 9~10Gb/s be= tween DomUs on different hosts using netfront/netback?

&nb= sp; The testing environment details are as follows:

= ;1. Hardware

a. CPU: Intel(R) Xeon(R) CPU E5645 @ 2= .40GHz, 2 CPU 6 Cores with Hyper Thread enabled

b.= NIC: Intel Corporation 82599EB 10-Gigabit SFI/SFP+ Network Connection (rev 0= 1)

a. HostOS: SLES 12 (Ker= nel 3.16-7,git commit d0335e4feea0d3f7a8af3116c5dc166239da7521 )<= /span>

&= nbsp; b. NIC Driver: IXGBE 3.21.2

&nb= sp; c. OVS: 2.1.3

d. MTU: 1600

<= div> &= nbsp; e. Dom0=EF=BC=9A6U6G

f. queue number: 4=

= g. xen 4.4

h. DomU: 4U4= G

3. Networking Environment:

a. All ne= twork flows are transmit/receive through OVS

b. Se= nder server and receiver server are connected directly between 10GE NIC

= ; 4. Testing Tools:

a. Sender: netperf=

&= nbsp; b. Receiver: netserver

----------

= zhangleiqiang (Trump)

Best Regards

= --Apple-Mail-043A107B-8FBB-48C3-8335-0EE6E2C01279-- --===============1327411302560573387== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel --===============1327411302560573387==-- From mboxrd@z Thu Jan 1 00:00:00 1970 From: David Vrabel Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 10:57:04 +0000 Message-ID: <547D9B00.2090506@citrix.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: zhangleiqiang , xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org On 02/12/14 08:30, zhangleiqiang wrote: > Hi, all > I am testing the performance of xen netfront-netback driver that > with multi-queues support. The throughput from domU to remote dom0 is > 9.2Gb/s, but the throughput from domU to remote domU is only 3.6Gb/s, I > think the bottleneck is the throughput from dom0 to local domU. However, > we have done some testing and found the throughput from dom0 to local > domU is 5.8Gb/s. > And if we send packets from one DomU to other 3 DomUs on different > host simultaneously, the sum of throughout can reach 9Gbps. It seems > like the bottleneck is the receiver? > After some analysis, I found that even the max_queue of > netfront/back is set to 4, there are some strange results as follows: > 1. In domU, only one rx queue deal with softirq > 2. In dom0, only two netback queues process are scheduled, other two > process aren't scheduled. Multiqueue only has benefits if you have multiple flows since the source/destination addresses are hashed to a queue number. This probably explains why only some of the queues are being used in your test. David From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 11:01:33 +0000 Message-ID: <20141202110133.GA5768@zion.uk.xensource.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: zhangleiqiang Cc: wei.liu2@citrix.com, xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org T24gVHVlLCBEZWMgMDIsIDIwMTQgYXQgMDQ6MzA6NDlQTSArMDgwMCwgemhhbmdsZWlxaWFuZyB3 cm90ZToKPiBIaSwgYWxsCj4gICAgIEkgYW0gdGVzdGluZyB0aGUgcGVyZm9ybWFuY2Ugb2YgeGVu IG5ldGZyb250LW5ldGJhY2sgZHJpdmVyIHRoYXQgd2l0aCBtdWx0aS1xdWV1ZXMgc3VwcG9ydC4g VGhlIHRocm91Z2hwdXQgZnJvbSBkb21VIHRvIHJlbW90ZSBkb20wIGlzIDkuMkdiL3MsIGJ1dCB0 aGUgdGhyb3VnaHB1dCBmcm9tIGRvbVUgdG8gcmVtb3RlIGRvbVUgaXMgb25seSAzLjZHYi9zLCBJ IHRoaW5rIHRoZSBib3R0bGVuZWNrIGlzIHRoZSB0aHJvdWdocHV0IGZyb20gZG9tMCB0byBsb2Nh bCBkb21VLiBIb3dldmVyLCB3ZSBoYXZlIGRvbmUgc29tZSB0ZXN0aW5nIGFuZCBmb3VuZCB0aGUg dGhyb3VnaHB1dCBmcm9tIGRvbTAgdG8gbG9jYWwgZG9tVSBpcyA1LjhHYi9zLgo+ICAgICBBbmQg aWYgd2Ugc2VuZCBwYWNrZXRzIGZyb20gb25lIERvbVUgdG8gb3RoZXIgMyBEb21VcyBvbiBkaWZm ZXJlbnQgaG9zdCBzaW11bHRhbmVvdXNseSwgdGhlIHN1bSBvZiB0aHJvdWdob3V0IGNhbiByZWFj aCA5R2Jwcy4gSXQgc2VlbXMgbGlrZSB0aGUgYm90dGxlbmVjayBpcyB0aGUgcmVjZWl2ZXI/Cj4g ICAgIEFmdGVyIHNvbWUgYW5hbHlzaXMsIEkgZm91bmQgdGhhdCBldmVuIHRoZSBtYXhfcXVldWUg b2YgbmV0ZnJvbnQvYmFjayBpcyBzZXQgdG8gNCwgdGhlcmUgYXJlIHNvbWUgc3RyYW5nZSByZXN1 bHRzIGFzIGZvbGxvd3M6Cj4gICAgIDEuIEluIGRvbVUsIG9ubHkgb25lIHJ4IHF1ZXVlIGRlYWwg d2l0aCBzb2Z0aXJxCgpUcnkgdG8gYmluZCBpcnEgdG8gZGlmZmVyZW50IHZjcHVzPwoKPiAgICAg Mi4gSW4gZG9tMCwgb25seSB0d28gbmV0YmFjayBxdWV1ZXMgcHJvY2VzcyBhcmUgc2NoZWR1bGVk LCBvdGhlciB0d28gcHJvY2VzcyBhcmVuJ3Qgc2NoZWR1bGVkLgoKSG93IG1hbnkgRG9tMCB2Y3B1 IGRvIHlvdSBoYXZlPyBJZiBpdCBvbmx5IGhhcyB0d28gdGhlbiB0aGVyZSB3aWxsIG9ubHkKYmUg dHdvIHByb2Nlc3NlcyBydW5uaW5nIGF0IGEgdGltZS4KCj4gCj4gICAgIEFyZSB0aGVyZSBhbnkg aXNzdWVzIGluIG15IHRlc3Q/IEluIHRoZW9yeSwgY2FuIHdlIGFjaGlldmUgOX4xMEdiL3MgYmV0 d2VlbiBEb21VcyBvbiBkaWZmZXJlbnQgaG9zdHMgdXNpbmcgbmV0ZnJvbnQvbmV0YmFjaz8KPiAg ICAgCj4gICAgICBUaGUgdGVzdGluZyBlbnZpcm9ubWVudCBkZXRhaWxzIGFyZSBhcyBmb2xsb3dz Ogo+ICAgIDEuIEhhcmR3YXJlCj4gICAgICAgIGEuIENQVTogSW50ZWwoUikgWGVvbihSKSBDUFUg RTU2NDUgQCAyLjQwR0h6LCAyIENQVSA2IENvcmVzIHdpdGggSHlwZXIgVGhyZWFkIGVuYWJsZWQK PiAgICAgICAgYi4gTklDOiBJbnRlbCBDb3Jwb3JhdGlvbiA4MjU5OUVCIDEwLUdpZ2FiaXQgU0ZJ L1NGUCsgTmV0d29yayBDb25uZWN0aW9uIChyZXYgMDEpCj4gICAgMi4gU29md2FyZToKPiAgICAg ICAgYS4gSG9zdE9TOiBTTEVTIDEyIChLZXJuZWwgMy4xNi03LGdpdCBjb21taXQgZDAzMzVlNGZl ZWEwZDNmN2E4YWYzMTE2YzVkYzE2NjIzOWRhNzUyMSApCgpBbmQgdGhpcyBpcyBhIFN1U0Uga2Vy bmVsPwoKPiAgICAgICAgYi4gTklDIERyaXZlcjogSVhHQkUgMy4yMS4yIAo+ICAgICAgICBjLiBP VlM6IDIuMS4zCj4gICAgICAgIGQuIE1UVTogMTYwMAo+ICAgICAgICBlLiBEb20w77yaNlU2Rwo+ ICAgICAgICBmLiBxdWV1ZSBudW1iZXI6IDQKPiAgICAgICAgZy4geGVuIDQuNAo+ICAgICAgICBo LiBEb21VOiA0VTRHCj4gICAgMy4gTmV0d29ya2luZyBFbnZpcm9ubWVudDoKPiAgICAgICAgYS4g QWxsIG5ldHdvcmsgZmxvd3MgYXJlIHRyYW5zbWl0L3JlY2VpdmUgdGhyb3VnaCBPVlMKPiAgICAg ICAgYi4gU2VuZGVyIHNlcnZlciBhbmQgcmVjZWl2ZXIgc2VydmVyIGFyZSBjb25uZWN0ZWQgZGly ZWN0bHkgYmV0d2VlbiAxMEdFIE5JQwo+ICAgIDQuIFRlc3RpbmcgVG9vbHM6Cj4gICAgICAgIGEu IFNlbmRlcjogbmV0cGVyZgo+ICAgICAgICBiLiBSZWNlaXZlcjogbmV0c2VydmVyCj4gCj4gCj4g LS0tLS0tLS0tLQo+IHpoYW5nbGVpcWlhbmcgKFRydW1wKQo+IEJlc3QgUmVnYXJkcwoKPiBfX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+IFhlbi1kZXZlbCBt YWlsaW5nIGxpc3QKPiBYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwo+IGh0dHA6Ly9saXN0cy54ZW4u b3JnL3hlbi1kZXZlbAoKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fClhlbi1kZXZlbCBtYWlsaW5nIGxpc3QKWGVuLWRldmVsQGxpc3RzLnhlbi5vcmcKaHR0 cDovL2xpc3RzLnhlbi5vcmcveGVuLWRldmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 11:53:35 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE23931FBB@SZXEMA512-MBX.china.huawei.com> References: <547D9B00.2090506@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <547D9B00.2090506@citrix.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: David Vrabel , zhangleiqiang , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: xen-devel-bounces@lists.xen.org > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of David Vrabel > Sent: Tuesday, December 02, 2014 6:57 PM > To: zhangleiqiang; xen-devel@lists.xen.org > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On 02/12/14 08:30, zhangleiqiang wrote: > > Hi, all > > I am testing the performance of xen netfront-netback driver that > > with multi-queues support. The throughput from domU to remote dom0 is > > 9.2Gb/s, but the throughput from domU to remote domU is only 3.6Gb/s, > > I think the bottleneck is the throughput from dom0 to local domU. > > However, we have done some testing and found the throughput from dom0 > > to local domU is 5.8Gb/s. > > And if we send packets from one DomU to other 3 DomUs on different > > host simultaneously, the sum of throughout can reach 9Gbps. It seems > > like the bottleneck is the receiver? > > After some analysis, I found that even the max_queue of > > netfront/back is set to 4, there are some strange results as follows: > > 1. In domU, only one rx queue deal with softirq > > 2. In dom0, only two netback queues process are scheduled, other > > two process aren't scheduled. > > Multiqueue only has benefits if you have multiple flows since the > source/destination addresses are hashed to a queue number. This probably > explains why only some of the queues are being used in your test. The hash method you mentioned is used for selection of netback process or netfront rx queue? Indeed, there are 4 netback processes running in Dom0, because there are only one DomU running in Dom0 and so four netback processes are running in Dom0 (the max_queue param of netback kernel module is set to 4). The phenomenon is that only 2 of these four netback process were running with about 70% cpu usage, and another two use little CPU. > David > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 11:50:59 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: In-Reply-To: <20141202110133.GA5768@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , zhangleiqiang Cc: "Xiaoding (B)" , Zhuangyuxin , "Luohao (brian)" , "Yuzhou (C)" , "xen-devel@lists.xen.org" List-Id: xen-devel@lists.xenproject.org PiAtLS0tLU9yaWdpbmFsIE1lc3NhZ2UtLS0tLQ0KPiBGcm9tOiB4ZW4tZGV2ZWwtYm91bmNlc0Bs aXN0cy54ZW4ub3JnDQo+IFttYWlsdG86eGVuLWRldmVsLWJvdW5jZXNAbGlzdHMueGVuLm9yZ10g T24gQmVoYWxmIE9mIFdlaSBMaXUNCj4gU2VudDogVHVlc2RheSwgRGVjZW1iZXIgMDIsIDIwMTQg NzowMiBQTQ0KPiBUbzogemhhbmdsZWlxaWFuZw0KPiBDYzogd2VpLmxpdTJAY2l0cml4LmNvbTsg eGVuLWRldmVsQGxpc3RzLnhlbi5vcmcNCj4gU3ViamVjdDogUmU6IFtYZW4tZGV2ZWxdIFBvb3Ig bmV0d29yayBwZXJmb3JtYW5jZSBiZXR3ZWVuIERvbVUgd2l0aA0KPiBtdWx0aXF1ZXVlIHN1cHBv cnQNCj4gDQo+IE9uIFR1ZSwgRGVjIDAyLCAyMDE0IGF0IDA0OjMwOjQ5UE0gKzA4MDAsIHpoYW5n bGVpcWlhbmcgd3JvdGU6DQo+ID4gSGksIGFsbA0KPiA+ICAgICBJIGFtIHRlc3RpbmcgdGhlIHBl cmZvcm1hbmNlIG9mIHhlbiBuZXRmcm9udC1uZXRiYWNrIGRyaXZlciB0aGF0IHdpdGgNCj4gbXVs dGktcXVldWVzIHN1cHBvcnQuIFRoZSB0aHJvdWdocHV0IGZyb20gZG9tVSB0byByZW1vdGUgZG9t MCBpcyA5LjJHYi9zLA0KPiBidXQgdGhlIHRocm91Z2hwdXQgZnJvbSBkb21VIHRvIHJlbW90ZSBk b21VIGlzIG9ubHkgMy42R2IvcywgSSB0aGluayB0aGUNCj4gYm90dGxlbmVjayBpcyB0aGUgdGhy b3VnaHB1dCBmcm9tIGRvbTAgdG8gbG9jYWwgZG9tVS4gSG93ZXZlciwgd2UgaGF2ZQ0KPiBkb25l IHNvbWUgdGVzdGluZyBhbmQgZm91bmQgdGhlIHRocm91Z2hwdXQgZnJvbSBkb20wIHRvIGxvY2Fs IGRvbVUgaXMNCj4gNS44R2Ivcy4NCj4gPiAgICAgQW5kIGlmIHdlIHNlbmQgcGFja2V0cyBmcm9t IG9uZSBEb21VIHRvIG90aGVyIDMgRG9tVXMgb24gZGlmZmVyZW50DQo+IGhvc3Qgc2ltdWx0YW5l b3VzbHksIHRoZSBzdW0gb2YgdGhyb3VnaG91dCBjYW4gcmVhY2ggOUdicHMuIEl0IHNlZW1zIGxp a2UgdGhlDQo+IGJvdHRsZW5lY2sgaXMgdGhlIHJlY2VpdmVyPw0KPiA+ICAgICBBZnRlciBzb21l IGFuYWx5c2lzLCBJIGZvdW5kIHRoYXQgZXZlbiB0aGUgbWF4X3F1ZXVlIG9mIG5ldGZyb250L2Jh Y2sNCj4gaXMgc2V0IHRvIDQsIHRoZXJlIGFyZSBzb21lIHN0cmFuZ2UgcmVzdWx0cyBhcyBmb2xs b3dzOg0KPiA+ICAgICAxLiBJbiBkb21VLCBvbmx5IG9uZSByeCBxdWV1ZSBkZWFsIHdpdGggc29m dGlycQ0KPiANCj4gVHJ5IHRvIGJpbmQgaXJxIHRvIGRpZmZlcmVudCB2Y3B1cz8NCg0KRG8geW91 IG1lYW4gd2UgdHJ5IHRvIGJpbmQgaXJxIHRvIGRpZmZlcmVudCB2Y3B1cyBpbiBEb21VPyBJIHdp bGwgdHJ5IGl0IG5vdy4NCg0KPiANCj4gPiAgICAgMi4gSW4gZG9tMCwgb25seSB0d28gbmV0YmFj ayBxdWV1ZXMgcHJvY2VzcyBhcmUgc2NoZWR1bGVkLCBvdGhlciB0d28NCj4gcHJvY2VzcyBhcmVu J3Qgc2NoZWR1bGVkLg0KPiANCj4gSG93IG1hbnkgRG9tMCB2Y3B1IGRvIHlvdSBoYXZlPyBJZiBp dCBvbmx5IGhhcyB0d28gdGhlbiB0aGVyZSB3aWxsIG9ubHkgYmUNCj4gdHdvIHByb2Nlc3NlcyBy dW5uaW5nIGF0IGEgdGltZS4NCg0KRG9tMCBoYXMgNiB2Y3B1cywgYW5kIDZHIG1lbW9yeS4gVGhl cmUgYXJlIG9ubHkgb25lIERvbVUgcnVubmluZyBpbiBEb20wIGFuZCBzbyBmb3VyIG5ldGJhY2sg cHJvY2Vzc2VzIGFyZSBydW5uaW5nIGluIERvbTAgKGJlY2F1c2UgdGhlIG1heF9xdWV1ZSBwYXJh bSBvZiBuZXRiYWNrIGtlcm5lbCBtb2R1bGUgaXMgc2V0IHRvIDQpLiANClRoZSBwaGVub21lbm9u IGlzIHRoYXQgb25seSAyIG9mIHRoZXNlIGZvdXIgbmV0YmFjayBwcm9jZXNzIHdlcmUgcnVubmlu ZyB3aXRoIGFib3V0IDcwJSBjcHUgdXNhZ2UsIGFuZCBhbm90aGVyIHR3byB1c2UgbGl0dGxlIENQ VS4NCklzIHRoZXJlIGEgaGFzaCBhbGdvcml0aG0gdG8gZGV0ZXJtaW5lIHdoaWNoIG5ldGJhY2sg cHJvY2VzcyB0byBoYW5kbGUgdGhlIGlucHV0IHBhY2tldD8NCg0KPiA+DQo+ID4gICAgIEFyZSB0 aGVyZSBhbnkgaXNzdWVzIGluIG15IHRlc3Q/IEluIHRoZW9yeSwgY2FuIHdlIGFjaGlldmUgOX4x MEdiL3MNCj4gYmV0d2VlbiBEb21VcyBvbiBkaWZmZXJlbnQgaG9zdHMgdXNpbmcgbmV0ZnJvbnQv bmV0YmFjaz8NCj4gPg0KPiA+ICAgICAgVGhlIHRlc3RpbmcgZW52aXJvbm1lbnQgZGV0YWlscyBh cmUgYXMgZm9sbG93czoNCj4gPiAgICAxLiBIYXJkd2FyZQ0KPiA+ICAgICAgICBhLiBDUFU6IElu dGVsKFIpIFhlb24oUikgQ1BVIEU1NjQ1IEAgMi40MEdIeiwgMiBDUFUgNiBDb3JlcyB3aXRoDQo+ IEh5cGVyIFRocmVhZCBlbmFibGVkDQo+ID4gICAgICAgIGIuIE5JQzogSW50ZWwgQ29ycG9yYXRp b24gODI1OTlFQiAxMC1HaWdhYml0IFNGSS9TRlArIE5ldHdvcmsNCj4gQ29ubmVjdGlvbiAocmV2 IDAxKQ0KPiA+ICAgIDIuIFNvZndhcmU6DQo+ID4gICAgICAgIGEuIEhvc3RPUzogU0xFUyAxMiAo S2VybmVsIDMuMTYtNyxnaXQgY29tbWl0DQo+ID4gZDAzMzVlNGZlZWEwZDNmN2E4YWYzMTE2YzVk YzE2NjIzOWRhNzUyMSApDQo+IA0KPiBBbmQgdGhpcyBpcyBhIFN1U0Uga2VybmVsPw0KDQpObywg SSBqdXN0IGNvbXBpbGUgRG9tMCBhbmQgRG9tVSBrZXJuZWwgdXNpbmcgMy4xNi03IHRhZyBmcm9t IGtlcm5lbC5vcmcuDQoNCj4gPiAgICAgICAgYi4gTklDIERyaXZlcjogSVhHQkUgMy4yMS4yDQo+ ID4gICAgICAgIGMuIE9WUzogMi4xLjMNCj4gPiAgICAgICAgZC4gTVRVOiAxNjAwDQo+ID4gICAg ICAgIGUuIERvbTDvvJo2VTZHDQo+ID4gICAgICAgIGYuIHF1ZXVlIG51bWJlcjogNA0KPiA+ICAg ICAgICBnLiB4ZW4gNC40DQo+ID4gICAgICAgIGguIERvbVU6IDRVNEcNCj4gPiAgICAzLiBOZXR3 b3JraW5nIEVudmlyb25tZW50Og0KPiA+ICAgICAgICBhLiBBbGwgbmV0d29yayBmbG93cyBhcmUg dHJhbnNtaXQvcmVjZWl2ZSB0aHJvdWdoIE9WUw0KPiA+ICAgICAgICBiLiBTZW5kZXIgc2VydmVy IGFuZCByZWNlaXZlciBzZXJ2ZXIgYXJlIGNvbm5lY3RlZCBkaXJlY3RseSBiZXR3ZWVuDQo+IDEw R0UgTklDDQo+ID4gICAgNC4gVGVzdGluZyBUb29sczoNCj4gPiAgICAgICAgYS4gU2VuZGVyOiBu ZXRwZXJmDQo+ID4gICAgICAgIGIuIFJlY2VpdmVyOiBuZXRzZXJ2ZXINCj4gPg0KPiA+DQo+ID4g LS0tLS0tLS0tLQ0KPiA+IHpoYW5nbGVpcWlhbmcgKFRydW1wKQ0KPiA+IEJlc3QgUmVnYXJkcw0K PiANCj4gPiBfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXw0K PiA+IFhlbi1kZXZlbCBtYWlsaW5nIGxpc3QNCj4gPiBYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZw0K PiA+IGh0dHA6Ly9saXN0cy54ZW4ub3JnL3hlbi1kZXZlbA0KPiANCj4gDQo+IF9fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fDQo+IFhlbi1kZXZlbCBtYWlsaW5n IGxpc3QNCj4gWGVuLWRldmVsQGxpc3RzLnhlbi5vcmcNCj4gaHR0cDovL2xpc3RzLnhlbi5vcmcv eGVuLWRldmVsDQpfX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f XwpYZW4tZGV2ZWwgbWFpbGluZyBsaXN0Clhlbi1kZXZlbEBsaXN0cy54ZW4ub3JnCmh0dHA6Ly9s aXN0cy54ZW4ub3JnL3hlbi1kZXZlbAo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 12:11:51 +0000 Message-ID: <20141202121151.GD5768@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > -----Original Message----- > > From: xen-devel-bounces@lists.xen.org > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei Liu > > Sent: Tuesday, December 02, 2014 7:02 PM > > To: zhangleiqiang > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > multiqueue support > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > Hi, all > > > I am testing the performance of xen netfront-netback driver that with > > multi-queues support. The throughput from domU to remote dom0 is 9.2Gb/s, > > but the throughput from domU to remote domU is only 3.6Gb/s, I think the > > bottleneck is the throughput from dom0 to local domU. However, we have > > done some testing and found the throughput from dom0 to local domU is > > 5.8Gb/s. > > > And if we send packets from one DomU to other 3 DomUs on different > > host simultaneously, the sum of throughout can reach 9Gbps. It seems like the > > bottleneck is the receiver? > > > After some analysis, I found that even the max_queue of netfront/back > > is set to 4, there are some strange results as follows: > > > 1. In domU, only one rx queue deal with softirq > > > > Try to bind irq to different vcpus? > > Do you mean we try to bind irq to different vcpus in DomU? I will try it now. > Yes. Given the fact that you have two backend threads running while only one DomU vcpu is busy, it smells like misconfiguration in DomU. If this phenomenon persists after correctly binding irqs, you might want to check traffic is steering correctly to different queues. > > > > > 2. In dom0, only two netback queues process are scheduled, other two > > process aren't scheduled. > > > > How many Dom0 vcpu do you have? If it only has two then there will only be > > two processes running at a time. > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running in Dom0 and so four netback processes are running in Dom0 (because the max_queue param of netback kernel module is set to 4). > The phenomenon is that only 2 of these four netback process were running with about 70% cpu usage, and another two use little CPU. > Is there a hash algorithm to determine which netback process to handle the input packet? > I think that's whatever default algorithm Linux kernel is using. We don't currently support other algorithms. Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 14:46:36 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141202121151.GD5768@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu Cc: "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org Thanks for your reply, Wei. I do the following testing just now and found the results as follows: There are three DomUs (4U4G) are running on Host A (6U6G) and one DomU (4U4G) is running on Host B (6U6G), I send packets from three DomUs to the DomU on Host B simultaneously. 1. The "top" output of Host B as follows: top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.8 si, 1.9 st %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, 9.5 si, 0.4 st %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 1.4 si, 1.4 st %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 6.9 si, 0.9 st KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 buffers KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7440 root 20 0 0 0 0 R 71.10 0.000 8:15.38 vif4.0-q3-guest 7434 root 20 0 0 0 0 R 59.14 0.000 9:00.58 vif4.0-q0-guest 18 root 20 0 0 0 0 R 33.89 0.000 2:35.06 ksoftirqd/2 28 root 20 0 0 0 0 S 20.93 0.000 3:01.81 ksoftirqd/4 As shown above, only two netback related processes (vif4.0-*) are running with high cpu usage, and the other 2 netback processes are idle. The "ps" result of vif4.0-* processes as follows: root 7434 50.5 0.0 0 0 ? R 09:23 11:29 [vif4.0-q0-guest] root 7435 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q0-deall] root 7436 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-guest] root 7437 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-deall] root 7438 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-guest] root 7439 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-deall] root 7440 48.1 0.0 0 0 ? R 09:23 10:55 [vif4.0-q3-guest] root 7441 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q3-deall] root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 0:00 grep --color=auto 2. The "rx" related content in /proc/interupts in receiver DomU (on Host B): 73: 2 0 2925405 0 xen-dyn-event eth0-q0-rx 75: 43 93 0 118 xen-dyn-event eth0-q1-rx 77: 2 3376 14 1983 xen-dyn-event eth0-q2-rx 79: 2414666 0 9 0 xen-dyn-event eth0-q3-rx As shown above, it seems like that only q0 and q3 handles the interrupt triggered by packet receving. Any advise? Thanks. ---------- zhangleiqiang (Trump) Best Regards > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@citrix.com] > Sent: Tuesday, December 02, 2014 8:12 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > > -----Original Message----- > > > From: xen-devel-bounces@lists.xen.org > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei Liu > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > To: zhangleiqiang > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > multiqueue support > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > Hi, all > > > > I am testing the performance of xen netfront-netback driver > > > > that with > > > multi-queues support. The throughput from domU to remote dom0 is > > > 9.2Gb/s, but the throughput from domU to remote domU is only > > > 3.6Gb/s, I think the bottleneck is the throughput from dom0 to local > > > domU. However, we have done some testing and found the throughput > > > from dom0 to local domU is 5.8Gb/s. > > > > And if we send packets from one DomU to other 3 DomUs on > > > > different > > > host simultaneously, the sum of throughout can reach 9Gbps. It seems > > > like the bottleneck is the receiver? > > > > After some analysis, I found that even the max_queue of > > > > netfront/back > > > is set to 4, there are some strange results as follows: > > > > 1. In domU, only one rx queue deal with softirq > > > > > > Try to bind irq to different vcpus? > > > > Do you mean we try to bind irq to different vcpus in DomU? I will try it now. > > > > Yes. Given the fact that you have two backend threads running while only one > DomU vcpu is busy, it smells like misconfiguration in DomU. > > If this phenomenon persists after correctly binding irqs, you might want to > check traffic is steering correctly to different queues. > > > > > > > > 2. In dom0, only two netback queues process are scheduled, > > > > other two > > > process aren't scheduled. > > > > > > How many Dom0 vcpu do you have? If it only has two then there will > > > only be two processes running at a time. > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running in > Dom0 and so four netback processes are running in Dom0 (because the > max_queue param of netback kernel module is set to 4). > > The phenomenon is that only 2 of these four netback process were running > with about 70% cpu usage, and another two use little CPU. > > Is there a hash algorithm to determine which netback process to handle the > input packet? > > > > I think that's whatever default algorithm Linux kernel is using. > > We don't currently support other algorithms. > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 2 Dec 2014 15:58:32 +0000 Message-ID: <20141202155832.GH5768@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Tue, Dec 02, 2014 at 02:46:36PM +0000, Zhangleiqiang (Trump) wrote: > Thanks for your reply, Wei. > > I do the following testing just now and found the results as follows: > > There are three DomUs (4U4G) are running on Host A (6U6G) and one DomU (4U4G) is running on Host B (6U6G), I send packets from three DomUs to the DomU on Host B simultaneously. > > 1. The "top" output of Host B as follows: > > top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 > Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie > %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.8 si, 1.9 st > %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, 9.5 si, 0.4 st > %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st > %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 1.4 si, 1.4 st > %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st > %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 6.9 si, 0.9 st > KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 buffers > KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 cached Mem > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 7440 root 20 0 0 0 0 R 71.10 0.000 8:15.38 vif4.0-q3-guest > 7434 root 20 0 0 0 0 R 59.14 0.000 9:00.58 vif4.0-q0-guest > 18 root 20 0 0 0 0 R 33.89 0.000 2:35.06 ksoftirqd/2 > 28 root 20 0 0 0 0 S 20.93 0.000 3:01.81 ksoftirqd/4 > > > As shown above, only two netback related processes (vif4.0-*) are running with high cpu usage, and the other 2 netback processes are idle. The "ps" result of vif4.0-* processes as follows: > > root 7434 50.5 0.0 0 0 ? R 09:23 11:29 [vif4.0-q0-guest] > root 7435 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q0-deall] > root 7436 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-guest] > root 7437 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-deall] > root 7438 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-guest] > root 7439 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-deall] > root 7440 48.1 0.0 0 0 ? R 09:23 10:55 [vif4.0-q3-guest] > root 7441 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q3-deall] > root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 0:00 grep --color=auto > > > 2. The "rx" related content in /proc/interupts in receiver DomU (on Host B): > > 73: 2 0 2925405 0 xen-dyn-event eth0-q0-rx > 75: 43 93 0 118 xen-dyn-event eth0-q1-rx > 77: 2 3376 14 1983 xen-dyn-event eth0-q2-rx > 79: 2414666 0 9 0 xen-dyn-event eth0-q3-rx > > As shown above, it seems like that only q0 and q3 handles the interrupt triggered by packet receving. > > Any advise? Thanks. Netback selects queue based on the return value of skb_get_queue_mapping. The queue mapping is set by core driver or ndo_select_queue (if specified by individual driver). In this case netback doesn't have its implementation of ndo_select_queue, so it's up to core driver to decide which queue to dispatch the packet to. I think you need to inspect why Dom0 only steers traffic to these two queues but not all of them. Don't know which utility is handy for this job. Probably tc(8) is useful? Wei. > ---------- > zhangleiqiang (Trump) > > Best Regards > > > > -----Original Message----- > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > Sent: Tuesday, December 02, 2014 8:12 PM > > To: Zhangleiqiang (Trump) > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); Xiaoding > > (B); Yuzhou (C); Zhuangyuxin > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > multiqueue support > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > > > -----Original Message----- > > > > From: xen-devel-bounces@lists.xen.org > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei Liu > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > To: zhangleiqiang > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > > multiqueue support > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > Hi, all > > > > > I am testing the performance of xen netfront-netback driver > > > > > that with > > > > multi-queues support. The throughput from domU to remote dom0 is > > > > 9.2Gb/s, but the throughput from domU to remote domU is only > > > > 3.6Gb/s, I think the bottleneck is the throughput from dom0 to local > > > > domU. However, we have done some testing and found the throughput > > > > from dom0 to local domU is 5.8Gb/s. > > > > > And if we send packets from one DomU to other 3 DomUs on > > > > > different > > > > host simultaneously, the sum of throughout can reach 9Gbps. It seems > > > > like the bottleneck is the receiver? > > > > > After some analysis, I found that even the max_queue of > > > > > netfront/back > > > > is set to 4, there are some strange results as follows: > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > Try to bind irq to different vcpus? > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I will try it now. > > > > > > > Yes. Given the fact that you have two backend threads running while only one > > DomU vcpu is busy, it smells like misconfiguration in DomU. > > > > If this phenomenon persists after correctly binding irqs, you might want to > > check traffic is steering correctly to different queues. > > > > > > > > > > > 2. In dom0, only two netback queues process are scheduled, > > > > > other two > > > > process aren't scheduled. > > > > > > > > How many Dom0 vcpu do you have? If it only has two then there will > > > > only be two processes running at a time. > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running in > > Dom0 and so four netback processes are running in Dom0 (because the > > max_queue param of netback kernel module is set to 4). > > > The phenomenon is that only 2 of these four netback process were running > > with about 70% cpu usage, and another two use little CPU. > > > Is there a hash algorithm to determine which netback process to handle the > > input packet? > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > We don't currently support other algorithms. > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zoltan Kiss Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 02 Dec 2014 17:25:22 +0000 Message-ID: <547DF602.2040704@linaro.org> References: <547D9B00.2090506@citrix.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FBB@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE23931FBB@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" , David Vrabel , zhangleiqiang , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org On 02/12/14 11:53, Zhangleiqiang (Trump) wrote: >> -----Original Message----- >> From: xen-devel-bounces@lists.xen.org >> [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of David Vrabel >> Sent: Tuesday, December 02, 2014 6:57 PM >> To: zhangleiqiang; xen-devel@lists.xen.org >> Subject: Re: [Xen-devel] Poor network performance between DomU with >> multiqueue support >> >> On 02/12/14 08:30, zhangleiqiang wrote: >>> Hi, all >>> I am testing the performance of xen netfront-netback driver that >>> with multi-queues support. The throughput from domU to remote dom0 is >>> 9.2Gb/s, but the throughput from domU to remote domU is only 3.6Gb/s, >>> I think the bottleneck is the throughput from dom0 to local domU. >>> However, we have done some testing and found the throughput from dom0 >>> to local domU is 5.8Gb/s. >>> And if we send packets from one DomU to other 3 DomUs on different >>> host simultaneously, the sum of throughout can reach 9Gbps. It seems >>> like the bottleneck is the receiver? >>> After some analysis, I found that even the max_queue of >>> netfront/back is set to 4, there are some strange results as follows: >>> 1. In domU, only one rx queue deal with softirq >>> 2. In dom0, only two netback queues process are scheduled, other >>> two process aren't scheduled. >> >> Multiqueue only has benefits if you have multiple flows since the >> source/destination addresses are hashed to a queue number. This probably >> explains why only some of the queues are being used in your test. > > The hash method you mentioned is used for selection of netback process or netfront rx queue? It's used in both direction to select the queue. > Indeed, there are 4 netback processes running in Dom0, because there are only one DomU running in Dom0 and so four netback processes are running in Dom0 (the max_queue param of netback kernel module is set to 4). > The phenomenon is that only 2 of these four netback process were running with about 70% cpu usage, and another two use little CPU. > >> David >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xen.org >> http://lists.xen.org/xen-devel > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Wed, 3 Dec 2014 14:43:37 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141202155832.GH5768@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@citrix.com] > Sent: Tuesday, December 02, 2014 11:59 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Tue, Dec 02, 2014 at 02:46:36PM +0000, Zhangleiqiang (Trump) wrote: > > Thanks for your reply, Wei. > > > > I do the following testing just now and found the results as follows: > > > > There are three DomUs (4U4G) are running on Host A (6U6G) and one DomU > (4U4G) is running on Host B (6U6G), I send packets from three DomUs to the > DomU on Host B simultaneously. > > > > 1. The "top" output of Host B as follows: > > > > top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 > > Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie > > %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.8 > > si, 1.9 st > > %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, 9.5 > > si, 0.4 st > > %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 1.7 > > si, 0.0 st > > %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 1.4 > > si, 1.4 st > > %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, 0.3 > > si, 0.0 st > > %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 6.9 si, 0.9 > st > > KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 > buffers > > KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 > cached Mem > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM > TIME+ COMMAND > > 7440 root 20 0 0 0 0 R 71.10 0.000 > 8:15.38 vif4.0-q3-guest > > 7434 root 20 0 0 0 0 R 59.14 0.000 > 9:00.58 vif4.0-q0-guest > > 18 root 20 0 0 0 0 R 33.89 0.000 > 2:35.06 ksoftirqd/2 > > 28 root 20 0 0 0 0 S 20.93 0.000 > 3:01.81 ksoftirqd/4 > > > > > > As shown above, only two netback related processes (vif4.0-*) are running > with high cpu usage, and the other 2 netback processes are idle. The "ps" > result of vif4.0-* processes as follows: > > > > root 7434 50.5 0.0 0 0 ? R 09:23 11:29 > [vif4.0-q0-guest] > > root 7435 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q0-deall] > > root 7436 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q1-guest] > > root 7437 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q1-deall] > > root 7438 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q2-guest] > > root 7439 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q2-deall] > > root 7440 48.1 0.0 0 0 ? R 09:23 10:55 > [vif4.0-q3-guest] > > root 7441 0.0 0.0 0 0 ? S 09:23 0:00 > [vif4.0-q3-deall] > > root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 0:00 > grep --color=auto > > > > > > 2. The "rx" related content in /proc/interupts in receiver DomU (on Host B): > > > > 73: 2 0 2925405 0 xen-dyn-event > eth0-q0-rx > > 75: 43 93 0 118 xen-dyn-event > eth0-q1-rx > > 77: 2 3376 14 1983 xen-dyn-event > eth0-q2-rx > > 79: 2414666 0 9 0 xen-dyn-event > eth0-q3-rx > > > > As shown above, it seems like that only q0 and q3 handles the interrupt > triggered by packet receving. > > > > Any advise? Thanks. > > Netback selects queue based on the return value of skb_get_queue_mapping. > The queue mapping is set by core driver or ndo_select_queue (if specified by > individual driver). In this case netback doesn't have its implementation of > ndo_select_queue, so it's up to core driver to decide which queue to dispatch > the packet to. I think you need to inspect why Dom0 only steers traffic to > these two queues but not all of them. > > Don't know which utility is handy for this job. Probably tc(8) is useful? Thanks Wei. I think the reason for the above results that only two netback/netfront processes works hard is the queue select method. I have tried to send packets from multiple host/vm to a vm, and all of the netback/netfront processes are running with high cpu usage a few times. However, I find another issue. Even using 6 queues and making sure that all of these 6 netback processes running with high cpu usage (indeed, any of it running with 87% cpu usage), the whole VM receive throughout is not very higher than results when using 4 queues. The results are from 4.5Gbps to 5.04 Gbps using TCP with 512 bytes length and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes length. According to the testing result from WIKI: http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_performance_testing, The VM receive throughput is also more lower than VM transmit. I am wondering why the VM receive throughout cannot be up to 8-10Gbps as VM transmit under multi-queue? I also tried to send packets directly from Local Dom0 to DomU, the DomU receive throughput can reach about 8-12Gbps, so I am also wondering why transmitting packets from Dom0 to Remote DomU can only reach about 4-5Gbps throughout? > Wei. > > > ---------- > > zhangleiqiang (Trump) > > > > Best Regards > > > > > > > -----Original Message----- > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > To: Zhangleiqiang (Trump) > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); > > > Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > multiqueue support > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > > > > -----Original Message----- > > > > > From: xen-devel-bounces@lists.xen.org > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei Liu > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > To: zhangleiqiang > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > Subject: Re: [Xen-devel] Poor network performance between DomU > > > > > with multiqueue support > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > > Hi, all > > > > > > I am testing the performance of xen netfront-netback > > > > > > driver that with > > > > > multi-queues support. The throughput from domU to remote dom0 is > > > > > 9.2Gb/s, but the throughput from domU to remote domU is only > > > > > 3.6Gb/s, I think the bottleneck is the throughput from dom0 to > > > > > local domU. However, we have done some testing and found the > > > > > throughput from dom0 to local domU is 5.8Gb/s. > > > > > > And if we send packets from one DomU to other 3 DomUs on > > > > > > different > > > > > host simultaneously, the sum of throughout can reach 9Gbps. It > > > > > seems like the bottleneck is the receiver? > > > > > > After some analysis, I found that even the max_queue of > > > > > > netfront/back > > > > > is set to 4, there are some strange results as follows: > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I will try it > now. > > > > > > > > > > Yes. Given the fact that you have two backend threads running while > > > only one DomU vcpu is busy, it smells like misconfiguration in DomU. > > > > > > If this phenomenon persists after correctly binding irqs, you might > > > want to check traffic is steering correctly to different queues. > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are scheduled, > > > > > > other two > > > > > process aren't scheduled. > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then there > > > > > will only be two processes running at a time. > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running > > > > in > > > Dom0 and so four netback processes are running in Dom0 (because the > > > max_queue param of netback kernel module is set to 4). > > > > The phenomenon is that only 2 of these four netback process were > > > > running > > > with about 70% cpu usage, and another two use little CPU. > > > > Is there a hash algorithm to determine which netback process to > > > > handle the > > > input packet? > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > We don't currently support other algorithms. > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 4 Dec 2014 10:50:21 +0000 Message-ID: <20141204105021.GA16532@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Wed, Dec 03, 2014 at 02:43:37PM +0000, Zhangleiqiang (Trump) wrote: > > -----Original Message----- > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > Sent: Tuesday, December 02, 2014 11:59 PM > > To: Zhangleiqiang (Trump) > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); Xiaoding > > (B); Yuzhou (C); Zhuangyuxin > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > multiqueue support > > > > On Tue, Dec 02, 2014 at 02:46:36PM +0000, Zhangleiqiang (Trump) wrote: > > > Thanks for your reply, Wei. > > > > > > I do the following testing just now and found the results as follows: > > > > > > There are three DomUs (4U4G) are running on Host A (6U6G) and one DomU > > (4U4G) is running on Host B (6U6G), I send packets from three DomUs to the > > DomU on Host B simultaneously. > > > > > > 1. The "top" output of Host B as follows: > > > > > > top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 > > > Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie > > > %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.8 > > > si, 1.9 st > > > %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, 9.5 > > > si, 0.4 st > > > %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 1.7 > > > si, 0.0 st > > > %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 1.4 > > > si, 1.4 st > > > %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, 0.3 > > > si, 0.0 st > > > %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 6.9 si, 0.9 > > st > > > KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 > > buffers > > > KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 > > cached Mem > > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM > > TIME+ COMMAND > > > 7440 root 20 0 0 0 0 R 71.10 0.000 > > 8:15.38 vif4.0-q3-guest > > > 7434 root 20 0 0 0 0 R 59.14 0.000 > > 9:00.58 vif4.0-q0-guest > > > 18 root 20 0 0 0 0 R 33.89 0.000 > > 2:35.06 ksoftirqd/2 > > > 28 root 20 0 0 0 0 S 20.93 0.000 > > 3:01.81 ksoftirqd/4 > > > > > > > > > As shown above, only two netback related processes (vif4.0-*) are running > > with high cpu usage, and the other 2 netback processes are idle. The "ps" > > result of vif4.0-* processes as follows: > > > > > > root 7434 50.5 0.0 0 0 ? R 09:23 11:29 > > [vif4.0-q0-guest] > > > root 7435 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q0-deall] > > > root 7436 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q1-guest] > > > root 7437 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q1-deall] > > > root 7438 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q2-guest] > > > root 7439 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q2-deall] > > > root 7440 48.1 0.0 0 0 ? R 09:23 10:55 > > [vif4.0-q3-guest] > > > root 7441 0.0 0.0 0 0 ? S 09:23 0:00 > > [vif4.0-q3-deall] > > > root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 0:00 > > grep --color=auto > > > > > > > > > 2. The "rx" related content in /proc/interupts in receiver DomU (on Host B): > > > > > > 73: 2 0 2925405 0 xen-dyn-event > > eth0-q0-rx > > > 75: 43 93 0 118 xen-dyn-event > > eth0-q1-rx > > > 77: 2 3376 14 1983 xen-dyn-event > > eth0-q2-rx > > > 79: 2414666 0 9 0 xen-dyn-event > > eth0-q3-rx > > > > > > As shown above, it seems like that only q0 and q3 handles the interrupt > > triggered by packet receving. > > > > > > Any advise? Thanks. > > > > Netback selects queue based on the return value of skb_get_queue_mapping. > > The queue mapping is set by core driver or ndo_select_queue (if specified by > > individual driver). In this case netback doesn't have its implementation of > > ndo_select_queue, so it's up to core driver to decide which queue to dispatch > > the packet to. I think you need to inspect why Dom0 only steers traffic to > > these two queues but not all of them. > > > > Don't know which utility is handy for this job. Probably tc(8) is useful? > > Thanks Wei. > > I think the reason for the above results that only two > netback/netfront processes works hard is the queue select method. I > have tried to send packets from multiple host/vm to a vm, and all of > the netback/netfront processes are running with high cpu usage a few > times. > A few times? You might want to check some patches to rework RX stall detection by David Vrabel that went in after 3.16. > However, I find another issue. Even using 6 queues and making sure > that all of these 6 netback processes running with high cpu usage > (indeed, any of it running with 87% cpu usage), the whole VM receive > throughout is not very higher than results when using 4 queues. The > results are from 4.5Gbps to 5.04 Gbps using TCP with 512 bytes length > and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes length. > I would like to ask if you're still using 4U4G (4 CPU 4 G?) configuration? If so, please make sure there are at least the same number of vcpus as queues. > According to the testing result from WIKI: > http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_performance_testing, > The VM receive throughput is also more lower than VM transmit. > I think that's expected, because guest RX data path still uses grant_copy while guest TX uses grant_map to do zero-copy transmit. > I am wondering why the VM receive throughout cannot be up to 8-10Gbps > as VM transmit under multi-queue? I also tried to send packets > directly from Local Dom0 to DomU, the DomU receive throughput can > reach about 8-12Gbps, so I am also wondering why transmitting packets > from Dom0 to Remote DomU can only reach about 4-5Gbps throughout? If data is from Dom0 to DomU then SKB is probably not fragmented by network stack. You can use tcpdump to check that. Wei. > > > Wei. > > > > > ---------- > > > zhangleiqiang (Trump) > > > > > > Best Regards > > > > > > > > > > -----Original Message----- > > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > > To: Zhangleiqiang (Trump) > > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); > > > > Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > > multiqueue support > > > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > > > > > -----Original Message----- > > > > > > From: xen-devel-bounces@lists.xen.org > > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei Liu > > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > > To: zhangleiqiang > > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > > Subject: Re: [Xen-devel] Poor network performance between DomU > > > > > > with multiqueue support > > > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > > > Hi, all > > > > > > > I am testing the performance of xen netfront-netback > > > > > > > driver that with > > > > > > multi-queues support. The throughput from domU to remote dom0 is > > > > > > 9.2Gb/s, but the throughput from domU to remote domU is only > > > > > > 3.6Gb/s, I think the bottleneck is the throughput from dom0 to > > > > > > local domU. However, we have done some testing and found the > > > > > > throughput from dom0 to local domU is 5.8Gb/s. > > > > > > > And if we send packets from one DomU to other 3 DomUs on > > > > > > > different > > > > > > host simultaneously, the sum of throughout can reach 9Gbps. It > > > > > > seems like the bottleneck is the receiver? > > > > > > > After some analysis, I found that even the max_queue of > > > > > > > netfront/back > > > > > > is set to 4, there are some strange results as follows: > > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I will try it > > now. > > > > > > > > > > > > > Yes. Given the fact that you have two backend threads running while > > > > only one DomU vcpu is busy, it smells like misconfiguration in DomU. > > > > > > > > If this phenomenon persists after correctly binding irqs, you might > > > > want to check traffic is steering correctly to different queues. > > > > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are scheduled, > > > > > > > other two > > > > > > process aren't scheduled. > > > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then there > > > > > > will only be two processes running at a time. > > > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running > > > > > in > > > > Dom0 and so four netback processes are running in Dom0 (because the > > > > max_queue param of netback kernel module is set to 4). > > > > > The phenomenon is that only 2 of these four netback process were > > > > > running > > > > with about 70% cpu usage, and another two use little CPU. > > > > > Is there a hash algorithm to determine which netback process to > > > > > handle the > > > > input packet? > > > > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > > > We don't currently support other algorithms. > > > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 4 Dec 2014 12:09:33 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141204105021.GA16532@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@citrix.com] > Sent: Thursday, December 04, 2014 6:50 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; xen-devel@lists.xen.org; zhangleiqiang; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Wed, Dec 03, 2014 at 02:43:37PM +0000, Zhangleiqiang (Trump) wrote: > > > -----Original Message----- > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > Sent: Tuesday, December 02, 2014 11:59 PM > > > To: Zhangleiqiang (Trump) > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); > > > Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > multiqueue support > > > > > > On Tue, Dec 02, 2014 at 02:46:36PM +0000, Zhangleiqiang (Trump) wrote: > > > > Thanks for your reply, Wei. > > > > > > > > I do the following testing just now and found the results as follows: > > > > > > > > There are three DomUs (4U4G) are running on Host A (6U6G) and one > > > > DomU > > > (4U4G) is running on Host B (6U6G), I send packets from three DomUs > > > to the DomU on Host B simultaneously. > > > > > > > > 1. The "top" output of Host B as follows: > > > > > > > > top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 > > > > Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie > > > > %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, > > > > 0.8 si, 1.9 st > > > > %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, > > > > 9.5 si, 0.4 st > > > > %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, > > > > 1.7 si, 0.0 st > > > > %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, > > > > 1.4 si, 1.4 st > > > > %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, > > > > 0.3 si, 0.0 st > > > > %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, > > > > 6.9 si, 0.9 > > > st > > > > KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 > > > buffers > > > > KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 > > > cached Mem > > > > > > > > PID USER PR NI VIRT RES SHR > S %CPU %MEM > > > TIME+ COMMAND > > > > 7440 root 20 0 0 0 0 R 71.10 0.000 > > > 8:15.38 vif4.0-q3-guest > > > > 7434 root 20 0 0 0 0 R 59.14 0.000 > > > 9:00.58 vif4.0-q0-guest > > > > 18 root 20 0 0 0 0 R 33.89 0.000 > > > 2:35.06 ksoftirqd/2 > > > > 28 root 20 0 0 0 0 S 20.93 0.000 > > > 3:01.81 ksoftirqd/4 > > > > > > > > > > > > As shown above, only two netback related processes (vif4.0-*) are > > > > running > > > with high cpu usage, and the other 2 netback processes are idle. The "ps" > > > result of vif4.0-* processes as follows: > > > > > > > > root 7434 50.5 0.0 0 0 ? R 09:23 > 11:29 > > > [vif4.0-q0-guest] > > > > root 7435 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q0-deall] > > > > root 7436 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q1-guest] > > > > root 7437 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q1-deall] > > > > root 7438 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q2-guest] > > > > root 7439 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q2-deall] > > > > root 7440 48.1 0.0 0 0 ? R 09:23 > 10:55 > > > [vif4.0-q3-guest] > > > > root 7441 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q3-deall] > > > > root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 > 0:00 > > > grep --color=auto > > > > > > > > > > > > 2. The "rx" related content in /proc/interupts in receiver DomU (on Host > B): > > > > > > > > 73: 2 0 2925405 0 xen-dyn-event > > > eth0-q0-rx > > > > 75: 43 93 0 118 xen-dyn-event > > > eth0-q1-rx > > > > 77: 2 3376 14 1983 xen-dyn-event > > > eth0-q2-rx > > > > 79: 2414666 0 9 0 xen-dyn-event > > > eth0-q3-rx > > > > > > > > As shown above, it seems like that only q0 and q3 handles the > > > > interrupt > > > triggered by packet receving. > > > > > > > > Any advise? Thanks. > > > > > > Netback selects queue based on the return value of > skb_get_queue_mapping. > > > The queue mapping is set by core driver or ndo_select_queue (if > > > specified by individual driver). In this case netback doesn't have > > > its implementation of ndo_select_queue, so it's up to core driver to > > > decide which queue to dispatch the packet to. I think you need to > > > inspect why Dom0 only steers traffic to these two queues but not all of > them. > > > > > > Don't know which utility is handy for this job. Probably tc(8) is useful? > > > > Thanks Wei. > > > > > I think the reason for the above results that only two > > netback/netfront processes works hard is the queue select method. I > > have tried to send packets from multiple host/vm to a vm, and all of > > the netback/netfront processes are running with high cpu usage a few > > times. > > > > A few times? You might want to check some patches to rework RX stall > detection by David Vrabel that went in after 3.16. Thanks for your suggest. I have switched to latest stable branch 3.17.4 and I find the patches you mentioned are not merged in this branch too, I will merge this patch and try again. > > However, I find another issue. Even using 6 queues and making sure > > that all of these 6 netback processes running with high cpu usage > > (indeed, any of it running with 87% cpu usage), the whole VM receive > > throughout is not very higher than results when using 4 queues. The > > results are from 4.5Gbps to 5.04 Gbps using TCP with 512 bytes length > > and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes length. > > > > I would like to ask if you're still using 4U4G (4 CPU 4 G?) configuration? If so, > please make sure there are at least the same number of vcpus as queues. Sorry for misleading you, 4U4G means 4 CPU and 4 G memory, :). I also found that the max_queue of netback is determinated by min(online_cpu, module_param) yesterday, so when using 6 queues in the previous testing, I used VM with 6 CPU and 6 G Memory. > > According to the testing result from WIKI: > > http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_perf > > ormance_testing, The VM receive throughput is also more lower than VM > > transmit. > > > > I think that's expected, because guest RX data path still uses grant_copy while > guest TX uses grant_map to do zero-copy transmit. As I understand, the RX process is as follows: 1. Phy NIC receive packet 2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do the "RX" operation, and the packet is stored into SKB which is also owned/shared with netback 4. NetBack notify netfront through event channel that a packet is receiving 5. Netfront grant a buffer for receiving and notify netback the GR (if using grant-resue mechanism, netfront just notify the GR to netback) through IO Ring 6. NetBack do the grant_copy to copy packet from its SKB to the buffer referenced by GR, and notify netfront through event channel 7. Netfront copy the data from buffer to user-level app's SKB Am I right? Why not using zero-copy transmit in guest RX data pash too ? > > I am wondering why the VM receive throughout cannot be up to 8-10Gbps > > as VM transmit under multi-queue? I also tried to send packets > > directly from Local Dom0 to DomU, the DomU receive throughput can > > reach about 8-12Gbps, so I am also wondering why transmitting packets > > from Dom0 to Remote DomU can only reach about 4-5Gbps throughout? > > If data is from Dom0 to DomU then SKB is probably not fragmented by network > stack. You can use tcpdump to check that. In our testing , the MTU is set to 1600. However, even testing with packets whose length are 1024 (small than 1600), the throughout between Dom0 to Local DomU is more higher than that between Dom0 to Remote DomU. So maybe the fragment is not the reason for it. > Wei. > > > > > > Wei. > > > > > > > ---------- > > > > zhangleiqiang (Trump) > > > > > > > > Best Regards > > > > > > > > > > > > > -----Original Message----- > > > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > > > To: Zhangleiqiang (Trump) > > > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao > > > > > (brian); Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > > > Subject: Re: [Xen-devel] Poor network performance between DomU > > > > > with multiqueue support > > > > > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) > wrote: > > > > > > > -----Original Message----- > > > > > > > From: xen-devel-bounces@lists.xen.org > > > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei > > > > > > > Liu > > > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > > > To: zhangleiqiang > > > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > > > Subject: Re: [Xen-devel] Poor network performance between > > > > > > > DomU with multiqueue support > > > > > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > > > > Hi, all > > > > > > > > I am testing the performance of xen netfront-netback > > > > > > > > driver that with > > > > > > > multi-queues support. The throughput from domU to remote > > > > > > > dom0 is 9.2Gb/s, but the throughput from domU to remote domU > > > > > > > is only 3.6Gb/s, I think the bottleneck is the throughput > > > > > > > from dom0 to local domU. However, we have done some testing > > > > > > > and found the throughput from dom0 to local domU is 5.8Gb/s. > > > > > > > > And if we send packets from one DomU to other 3 DomUs > > > > > > > > on different > > > > > > > host simultaneously, the sum of throughout can reach 9Gbps. > > > > > > > It seems like the bottleneck is the receiver? > > > > > > > > After some analysis, I found that even the max_queue > > > > > > > > of netfront/back > > > > > > > is set to 4, there are some strange results as follows: > > > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I > > > > > > will try it > > > now. > > > > > > > > > > > > > > > > Yes. Given the fact that you have two backend threads running > > > > > while only one DomU vcpu is busy, it smells like misconfiguration in > DomU. > > > > > > > > > > If this phenomenon persists after correctly binding irqs, you > > > > > might want to check traffic is steering correctly to different queues. > > > > > > > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are > > > > > > > > scheduled, other two > > > > > > > process aren't scheduled. > > > > > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then > > > > > > > there will only be two processes running at a time. > > > > > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU > > > > > > running in > > > > > Dom0 and so four netback processes are running in Dom0 (because > > > > > the max_queue param of netback kernel module is set to 4). > > > > > > The phenomenon is that only 2 of these four netback process > > > > > > were running > > > > > with about 70% cpu usage, and another two use little CPU. > > > > > > Is there a hash algorithm to determine which netback process > > > > > > to handle the > > > > > input packet? > > > > > > > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > > > > > We don't currently support other algorithms. > > > > > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 4 Dec 2014 13:05:31 +0000 Message-ID: <20141204130531.GD16532@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Thu, Dec 04, 2014 at 12:09:33PM +0000, Zhangleiqiang (Trump) wrote: [...] > > > However, I find another issue. Even using 6 queues and making sure > > > that all of these 6 netback processes running with high cpu usage > > > (indeed, any of it running with 87% cpu usage), the whole VM receive > > > throughout is not very higher than results when using 4 queues. The > > > results are from 4.5Gbps to 5.04 Gbps using TCP with 512 bytes length > > > and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes length. > > > > > > > I would like to ask if you're still using 4U4G (4 CPU 4 G?) configuration? If so, > > please make sure there are at least the same number of vcpus as queues. > > Sorry for misleading you, 4U4G means 4 CPU and 4 G memory, :). I also > found that the max_queue of netback is determinated by min(online_cpu, > module_param) yesterday, so when using 6 queues in the previous > testing, I used VM with 6 CPU and 6 G Memory. > > > > According to the testing result from WIKI: > > > http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_perf > > > ormance_testing, The VM receive throughput is also more lower than VM > > > transmit. > > > > > > > I think that's expected, because guest RX data path still uses grant_copy while > > guest TX uses grant_map to do zero-copy transmit. > > As I understand, the RX process is as follows: > 1. Phy NIC receive packet > 2. XEN Hypervisor trigger interrupt to Dom0 > 3. Dom0' s NIC driver do the "RX" operation, and the packet is stored into SKB which is also owned/shared with netback > 4. NetBack notify netfront through event channel that a packet is receiving > 5. Netfront grant a buffer for receiving and notify netback the GR (if using grant-resue mechanism, netfront just notify the GR to netback) through IO Ring > 6. NetBack do the grant_copy to copy packet from its SKB to the buffer referenced by GR, and notify netfront through event channel > 7. Netfront copy the data from buffer to user-level app's SKB > > Am I right? Step 4 is not correct, netback won't notify netfront at that point. Step 5 is not correct, all grant refs are pre-allocated and granted before that. Other steps look correct. > Why not using zero-copy transmit in guest RX data pash too ? > A rogue / buggy guest might hold the mapping for arbitrary long period of time. > > > > I am wondering why the VM receive throughout cannot be up to 8-10Gbps > > > as VM transmit under multi-queue? I also tried to send packets > > > directly from Local Dom0 to DomU, the DomU receive throughput can > > > reach about 8-12Gbps, so I am also wondering why transmitting packets > > > from Dom0 to Remote DomU can only reach about 4-5Gbps throughout? > > > > If data is from Dom0 to DomU then SKB is probably not fragmented by network > > stack. You can use tcpdump to check that. > > In our testing , the MTU is set to 1600. However, even testing with > packets whose length are 1024 (small than 1600), the throughout > between Dom0 to Local DomU is more higher than that between Dom0 to > Remote DomU. So maybe the fragment is not the reason for it. > Don't have much idea about this, sorry. Wei. > > > Wei. > > > > > > > > > Wei. > > > > > > > > > ---------- > > > > > zhangleiqiang (Trump) > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > > > > To: Zhangleiqiang (Trump) > > > > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao > > > > > > (brian); Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > > > > Subject: Re: [Xen-devel] Poor network performance between DomU > > > > > > with multiqueue support > > > > > > > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) > > wrote: > > > > > > > > -----Original Message----- > > > > > > > > From: xen-devel-bounces@lists.xen.org > > > > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei > > > > > > > > Liu > > > > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > > > > To: zhangleiqiang > > > > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > > > > Subject: Re: [Xen-devel] Poor network performance between > > > > > > > > DomU with multiqueue support > > > > > > > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > > > > > Hi, all > > > > > > > > > I am testing the performance of xen netfront-netback > > > > > > > > > driver that with > > > > > > > > multi-queues support. The throughput from domU to remote > > > > > > > > dom0 is 9.2Gb/s, but the throughput from domU to remote domU > > > > > > > > is only 3.6Gb/s, I think the bottleneck is the throughput > > > > > > > > from dom0 to local domU. However, we have done some testing > > > > > > > > and found the throughput from dom0 to local domU is 5.8Gb/s. > > > > > > > > > And if we send packets from one DomU to other 3 DomUs > > > > > > > > > on different > > > > > > > > host simultaneously, the sum of throughout can reach 9Gbps. > > > > > > > > It seems like the bottleneck is the receiver? > > > > > > > > > After some analysis, I found that even the max_queue > > > > > > > > > of netfront/back > > > > > > > > is set to 4, there are some strange results as follows: > > > > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I > > > > > > > will try it > > > > now. > > > > > > > > > > > > > > > > > > > Yes. Given the fact that you have two backend threads running > > > > > > while only one DomU vcpu is busy, it smells like misconfiguration in > > DomU. > > > > > > > > > > > > If this phenomenon persists after correctly binding irqs, you > > > > > > might want to check traffic is steering correctly to different queues. > > > > > > > > > > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are > > > > > > > > > scheduled, other two > > > > > > > > process aren't scheduled. > > > > > > > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then > > > > > > > > there will only be two processes running at a time. > > > > > > > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU > > > > > > > running in > > > > > > Dom0 and so four netback processes are running in Dom0 (because > > > > > > the max_queue param of netback kernel module is set to 4). > > > > > > > The phenomenon is that only 2 of these four netback process > > > > > > > were running > > > > > > with about 70% cpu usage, and another two use little CPU. > > > > > > > Is there a hash algorithm to determine which netback process > > > > > > > to handle the > > > > > > input packet? > > > > > > > > > > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > > > > > > > We don't currently support other algorithms. > > > > > > > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zoltan Kiss Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 04 Dec 2014 13:35:17 +0000 Message-ID: <54806315.6010007@linaro.org> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" , Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org On 04/12/14 12:09, Zhangleiqiang (Trump) wrote: >> I think that's expected, because guest RX data path still uses grant_copy while >> >guest TX uses grant_map to do zero-copy transmit. > As I understand, the RX process is as follows: > 1. Phy NIC receive packet > 2. XEN Hypervisor trigger interrupt to Dom0 > 3. Dom0' s NIC driver do the "RX" operation, and the packet is stored into SKB which is also owned/shared with netback Not that easy. There is something between the NIC driver and netback which directs the packets, e.g. the old bridge driver, ovs, or the IP stack of the kernel. > 4. NetBack notify netfront through event channel that a packet is receiving > 5. Netfront grant a buffer for receiving and notify netback the GR (if using grant-resue mechanism, netfront just notify the GR to netback) through IO Ring It looks a bit confusing in the code, but netfront put "requests" on the ring buffer, which contains the grant ref of the guest page where the backend can copy. When the packet comes, netback consumes these requests and send back a response telling the guest the grant copy of the packet finished, it can start handling the data. (sending a response means it's placing a response in the ring and trigger the event channel) And ideally netback should always have requests in the ring, so it doesn't have to wait for the guest to fill it up. > 6. NetBack do the grant_copy to copy packet from its SKB to the buffer referenced by GR, and notify netfront through event channel > 7. Netfront copy the data from buffer to user-level app's SKB Or wherever that SKB should go, yes. Like with any received packet on a real network interface. > > Am I right? Why not using zero-copy transmit in guest RX data pash too ? Because that means you are mapping that memory to the guest, and you won't have any guarantee when the guest will release them. And netback can't just unmap them forcibly after a timeout, because finding a correct timeout value would be quite impossible. A malicious/buggy/overloaded guest can hold on to Dom0 memory indefinitely, but it even becomes worse if the memory came from another guest: you can't shutdown that guest for example, until all its memory is returned to him. Regards, Zoli From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 4 Dec 2014 14:31:12 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE23933926@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> <54806315.6010007@linaro.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <54806315.6010007@linaro.org> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Zoltan Kiss , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > Sent: Thursday, December 04, 2014 9:35 PM > To: Zhangleiqiang (Trump); Wei Liu; xen-devel@lists.xen.org > Cc: Xiaoding (B); Zhuangyuxin; zhangleiqiang; Luohao (brian); Yuzhou (C) > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > > > On 04/12/14 12:09, Zhangleiqiang (Trump) wrote: > >> I think that's expected, because guest RX data path still uses > >> grant_copy while > >> >guest TX uses grant_map to do zero-copy transmit. > > As I understand, the RX process is as follows: > > 1. Phy NIC receive packet > > 2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do > > the "RX" operation, and the packet is stored into SKB which is also > > owned/shared with netback > Not that easy. There is something between the NIC driver and netback which > directs the packets, e.g. the old bridge driver, ovs, or the IP stack of the kernel. > > 4. NetBack notify netfront through event channel that a packet is > > receiving 5. Netfront grant a buffer for receiving and notify netback > > the GR (if using grant-resue mechanism, netfront just notify the GR to > > netback) through IO Ring > It looks a bit confusing in the code, but netfront put "requests" on the ring > buffer, which contains the grant ref of the guest page where the backend can > copy. When the packet comes, netback consumes these requests and send > back a response telling the guest the grant copy of the packet finished, it can > start handling the data. (sending a response means it's placing a response in > the ring and trigger the event channel) And ideally netback should always have > requests in the ring, so it doesn't have to wait for the guest to fill it up. > > 6. NetBack do the grant_copy to copy packet from its SKB to the buffer > > referenced by GR, and notify netfront through event channel 7. > > Netfront copy the data from buffer to user-level app's SKB > Or wherever that SKB should go, yes. Like with any received packet on a real > network interface. > > > > Am I right? Why not using zero-copy transmit in guest RX data pash too ? > Because that means you are mapping that memory to the guest, and you won't > have any guarantee when the guest will release them. And netback can't just > unmap them forcibly after a timeout, because finding a correct timeout value > would be quite impossible. > A malicious/buggy/overloaded guest can hold on to Dom0 memory indefinitely, > but it even becomes worse if the memory came from another > guest: you can't shutdown that guest for example, until all its memory is > returned to him. Thanks for your detailed explanation about RX data path, I have get it, :) About the issue that poor performance between DomU to DomU, but high throughout between Dom0 to remote Dom0/DomU mentioned in my previous mail, do you have any idea about it? I am wondering if netfront/netback can be optimized to reach the 10Gbps throughout between DomUs running on different hosts connected with 10GE network. Currently, it seems like the TX is not the bottleneck, because we can reach the aggregate throughout of 9Gbps when sending packets from one DomU to other 3 DomUs running on different host. So I think the bottleneck maybe the RX, are you agreed with me? I am wondering what is the main reason that prevent RX to reach the higher throughout? Compared to KVM+virtio+vhost, which can reach high throughout, the RX has extra grantcopy operation, and the grantcopy operation may be one reason for it. Do you have any idea about it too? > > Regards, > > Zoli From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Thu, 4 Dec 2014 14:37:54 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2393394C@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> <20141204130531.GD16532@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141204130531.GD16532@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org Thanks for your detailed explanation, Wei. I am wondering if netfront/netback can be optimized to reach the 10Gbps throughout between DomUs running on different hosts connected with 10GE network. Currently, it seems like the RX is the bottleneck, which also consist with the testing result in xenwiki: http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_performance_testing I am wondering what factors prevent RX to reach the higher throughout? You have mentioned that one reason is that guest RX data path still uses grant_copy while guest TX uses grant_map to do zero-copy transmit. Do you know any other factors or ongoing work to optimize the RX data path? ---------- zhangleiqiang (Trump) Best Regards > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@citrix.com] > Sent: Thursday, December 04, 2014 9:06 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; xen-devel@lists.xen.org; zhangleiqiang; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Thu, Dec 04, 2014 at 12:09:33PM +0000, Zhangleiqiang (Trump) wrote: > [...] > > > > However, I find another issue. Even using 6 queues and making sure > > > > that all of these 6 netback processes running with high cpu usage > > > > (indeed, any of it running with 87% cpu usage), the whole VM > > > > receive throughout is not very higher than results when using 4 > > > > queues. The results are from 4.5Gbps to 5.04 Gbps using TCP with > > > > 512 bytes length and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes > length. > > > > > > > > > > I would like to ask if you're still using 4U4G (4 CPU 4 G?) > > > configuration? If so, please make sure there are at least the same number > of vcpus as queues. > > > > > Sorry for misleading you, 4U4G means 4 CPU and 4 G memory, :). I also > > found that the max_queue of netback is determinated by min(online_cpu, > > module_param) yesterday, so when using 6 queues in the previous > > testing, I used VM with 6 CPU and 6 G Memory. > > > > > > > According to the testing result from WIKI: > > > > http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_ > > > > perf ormance_testing, The VM receive throughput is also more lower > > > > than VM transmit. > > > > > > > > > > I think that's expected, because guest RX data path still uses > > > grant_copy while guest TX uses grant_map to do zero-copy transmit. > > > > As I understand, the RX process is as follows: > > 1. Phy NIC receive packet > > 2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do > > the "RX" operation, and the packet is stored into SKB which is also > > owned/shared with netback 4. NetBack notify netfront through event > > channel that a packet is receiving 5. Netfront grant a buffer for > > receiving and notify netback the GR (if using grant-resue mechanism, > > netfront just notify the GR to netback) through IO Ring 6. NetBack do > > the grant_copy to copy packet from its SKB to the buffer referenced by > > GR, and notify netfront through event channel 7. Netfront copy the > > data from buffer to user-level app's SKB > > > > Am I right? > > Step 4 is not correct, netback won't notify netfront at that point. > > Step 5 is not correct, all grant refs are pre-allocated and granted before that. > > Other steps look correct. > > > Why not using zero-copy transmit in guest RX data pash too ? > > > > A rogue / buggy guest might hold the mapping for arbitrary long period of time. > > > > > > > I am wondering why the VM receive throughout cannot be up to > > > > 8-10Gbps as VM transmit under multi-queue? I also tried to send > > > > packets directly from Local Dom0 to DomU, the DomU receive > > > > throughput can reach about 8-12Gbps, so I am also wondering why > > > > transmitting packets from Dom0 to Remote DomU can only reach about > 4-5Gbps throughout? > > > > > > If data is from Dom0 to DomU then SKB is probably not fragmented by > > > network stack. You can use tcpdump to check that. > > > > In our testing , the MTU is set to 1600. However, even testing with > > packets whose length are 1024 (small than 1600), the throughout > > between Dom0 to Local DomU is more higher than that between Dom0 to > > Remote DomU. So maybe the fragment is not the reason for it. > > > > Don't have much idea about this, sorry. > > Wei. > > > > > > Wei. > > > > > > > > > > > > Wei. > > > > > > > > > > > ---------- > > > > > > zhangleiqiang (Trump) > > > > > > > > > > > > Best Regards > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > > > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > > > > > To: Zhangleiqiang (Trump) > > > > > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao > > > > > > > (brian); Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > > > > > Subject: Re: [Xen-devel] Poor network performance between > > > > > > > DomU with multiqueue support > > > > > > > > > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang > > > > > > > (Trump) > > > wrote: > > > > > > > > > -----Original Message----- > > > > > > > > > From: xen-devel-bounces@lists.xen.org > > > > > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of > > > > > > > > > Wei Liu > > > > > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > > > > > To: zhangleiqiang > > > > > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > > > > > Subject: Re: [Xen-devel] Poor network performance > > > > > > > > > between DomU with multiqueue support > > > > > > > > > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang > wrote: > > > > > > > > > > Hi, all > > > > > > > > > > I am testing the performance of xen > > > > > > > > > > netfront-netback driver that with > > > > > > > > > multi-queues support. The throughput from domU to remote > > > > > > > > > dom0 is 9.2Gb/s, but the throughput from domU to remote > > > > > > > > > domU is only 3.6Gb/s, I think the bottleneck is the > > > > > > > > > throughput from dom0 to local domU. However, we have > > > > > > > > > done some testing and found the throughput from dom0 to local > domU is 5.8Gb/s. > > > > > > > > > > And if we send packets from one DomU to other 3 > > > > > > > > > > DomUs on different > > > > > > > > > host simultaneously, the sum of throughout can reach 9Gbps. > > > > > > > > > It seems like the bottleneck is the receiver? > > > > > > > > > > After some analysis, I found that even the > > > > > > > > > > max_queue of netfront/back > > > > > > > > > is set to 4, there are some strange results as follows: > > > > > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? > > > > > > > > I will try it > > > > > now. > > > > > > > > > > > > > > > > > > > > > > Yes. Given the fact that you have two backend threads > > > > > > > running while only one DomU vcpu is busy, it smells like > > > > > > > misconfiguration in > > > DomU. > > > > > > > > > > > > > > If this phenomenon persists after correctly binding irqs, > > > > > > > you might want to check traffic is steering correctly to different > queues. > > > > > > > > > > > > > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are > > > > > > > > > > scheduled, other two > > > > > > > > > process aren't scheduled. > > > > > > > > > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then > > > > > > > > > there will only be two processes running at a time. > > > > > > > > > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU > > > > > > > > running in > > > > > > > Dom0 and so four netback processes are running in Dom0 > > > > > > > (because the max_queue param of netback kernel module is set to > 4). > > > > > > > > The phenomenon is that only 2 of these four netback > > > > > > > > process were running > > > > > > > with about 70% cpu usage, and another two use little CPU. > > > > > > > > Is there a hash algorithm to determine which netback > > > > > > > > process to handle the > > > > > > > input packet? > > > > > > > > > > > > > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > > > > > > > > > We don't currently support other algorithms. > > > > > > > > > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 5 Dec 2014 01:17:16 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141204105021.GA16532@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@citrix.com] > Sent: Thursday, December 04, 2014 6:50 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; xen-devel@lists.xen.org; zhangleiqiang; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Wed, Dec 03, 2014 at 02:43:37PM +0000, Zhangleiqiang (Trump) wrote: > > > -----Original Message----- > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > Sent: Tuesday, December 02, 2014 11:59 PM > > > To: Zhangleiqiang (Trump) > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao (brian); > > > Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > multiqueue support > > > > > > On Tue, Dec 02, 2014 at 02:46:36PM +0000, Zhangleiqiang (Trump) wrote: > > > > Thanks for your reply, Wei. > > > > > > > > I do the following testing just now and found the results as follows: > > > > > > > > There are three DomUs (4U4G) are running on Host A (6U6G) and one > > > > DomU > > > (4U4G) is running on Host B (6U6G), I send packets from three DomUs > > > to the DomU on Host B simultaneously. > > > > > > > > 1. The "top" output of Host B as follows: > > > > > > > > top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 > > > > Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie > > > > %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, > > > > 0.8 si, 1.9 st > > > > %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, > > > > 9.5 si, 0.4 st > > > > %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, > > > > 1.7 si, 0.0 st > > > > %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, > > > > 1.4 si, 1.4 st > > > > %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, > > > > 0.3 si, 0.0 st > > > > %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, > > > > 6.9 si, 0.9 > > > st > > > > KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 > > > buffers > > > > KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 > > > cached Mem > > > > > > > > PID USER PR NI VIRT RES SHR > S %CPU %MEM > > > TIME+ COMMAND > > > > 7440 root 20 0 0 0 0 R 71.10 0.000 > > > 8:15.38 vif4.0-q3-guest > > > > 7434 root 20 0 0 0 0 R 59.14 0.000 > > > 9:00.58 vif4.0-q0-guest > > > > 18 root 20 0 0 0 0 R 33.89 0.000 > > > 2:35.06 ksoftirqd/2 > > > > 28 root 20 0 0 0 0 S 20.93 0.000 > > > 3:01.81 ksoftirqd/4 > > > > > > > > > > > > As shown above, only two netback related processes (vif4.0-*) are > > > > running > > > with high cpu usage, and the other 2 netback processes are idle. The "ps" > > > result of vif4.0-* processes as follows: > > > > > > > > root 7434 50.5 0.0 0 0 ? R 09:23 > 11:29 > > > [vif4.0-q0-guest] > > > > root 7435 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q0-deall] > > > > root 7436 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q1-guest] > > > > root 7437 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q1-deall] > > > > root 7438 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q2-guest] > > > > root 7439 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q2-deall] > > > > root 7440 48.1 0.0 0 0 ? R 09:23 > 10:55 > > > [vif4.0-q3-guest] > > > > root 7441 0.0 0.0 0 0 ? S 09:23 > 0:00 > > > [vif4.0-q3-deall] > > > > root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 > 0:00 > > > grep --color=auto > > > > > > > > > > > > 2. The "rx" related content in /proc/interupts in receiver DomU (on Host > B): > > > > > > > > 73: 2 0 2925405 0 xen-dyn-event > > > eth0-q0-rx > > > > 75: 43 93 0 118 xen-dyn-event > > > eth0-q1-rx > > > > 77: 2 3376 14 1983 xen-dyn-event > > > eth0-q2-rx > > > > 79: 2414666 0 9 0 xen-dyn-event > > > eth0-q3-rx > > > > > > > > As shown above, it seems like that only q0 and q3 handles the > > > > interrupt > > > triggered by packet receving. > > > > > > > > Any advise? Thanks. > > > > > > Netback selects queue based on the return value of > skb_get_queue_mapping. > > > The queue mapping is set by core driver or ndo_select_queue (if > > > specified by individual driver). In this case netback doesn't have > > > its implementation of ndo_select_queue, so it's up to core driver to > > > decide which queue to dispatch the packet to. I think you need to > > > inspect why Dom0 only steers traffic to these two queues but not all of > them. > > > > > > Don't know which utility is handy for this job. Probably tc(8) is useful? > > > > Thanks Wei. > > > > > I think the reason for the above results that only two > > netback/netfront processes works hard is the queue select method. I > > have tried to send packets from multiple host/vm to a vm, and all of > > the netback/netfront processes are running with high cpu usage a few > > times. > > > > A few times? You might want to check some patches to rework RX stall > detection by David Vrabel that went in after 3.16. > > > However, I find another issue. Even using 6 queues and making sure > > that all of these 6 netback processes running with high cpu usage > > (indeed, any of it running with 87% cpu usage), the whole VM receive > > throughout is not very higher than results when using 4 queues. The > > results are from 4.5Gbps to 5.04 Gbps using TCP with 512 bytes length > > and 4.3Gbps to 5.78Gbps using TCP with 1460 bytes length. > > > > I would like to ask if you're still using 4U4G (4 CPU 4 G?) configuration? If so, > please make sure there are at least the same number of vcpus as queues. > > > According to the testing result from WIKI: > > http://wiki.xen.org/wiki/Xen-netback_and_xen-netfront_multi-queue_perf > > ormance_testing, The VM receive throughput is also more lower than VM > > transmit. > > > > I think that's expected, because guest RX data path still uses grant_copy while > guest TX uses grant_map to do zero-copy transmit. As far as I know, there are three main grant-related operations used in split device model: grant mapping, grant transfer and grant copy. Grant transfer has not used now, and grant mapping and grant transfer both involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer has this overhead? Does grant copy surely has more overhead than grant mapping? >>From the code, I see that in TX, netback will do gnttab_batch_copy as well as gnttab_map_refs:

 //netback.c:xenvif_tx_action
	xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops);

	if (nr_cops == 0)
		return 0;

	gnttab_batch_copy(queue->tx_copy_ops, nr_cops);
	if (nr_mops != 0) {
		ret = gnttab_map_refs(queue->tx_map_ops,
				      NULL,
				      queue->pages_to_map,
				      nr_mops);
		BUG_ON(ret);
	}

> > I am wondering why the VM receive throughout cannot be up to 8-10Gbps > > as VM transmit under multi-queue? I also tried to send packets > > directly from Local Dom0 to DomU, the DomU receive throughput can > > reach about 8-12Gbps, so I am also wondering why transmitting packets > > from Dom0 to Remote DomU can only reach about 4-5Gbps throughout? > > If data is from Dom0 to DomU then SKB is probably not fragmented by network > stack. You can use tcpdump to check that. > > Wei. > > > > > > Wei. > > > > > > > ---------- > > > > zhangleiqiang (Trump) > > > > > > > > Best Regards > > > > > > > > > > > > > -----Original Message----- > > > > > From: Wei Liu [mailto:wei.liu2@citrix.com] > > > > > Sent: Tuesday, December 02, 2014 8:12 PM > > > > > To: Zhangleiqiang (Trump) > > > > > Cc: Wei Liu; zhangleiqiang; xen-devel@lists.xen.org; Luohao > > > > > (brian); Xiaoding (B); Yuzhou (C); Zhuangyuxin > > > > > Subject: Re: [Xen-devel] Poor network performance between DomU > > > > > with multiqueue support > > > > > > > > > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) > wrote: > > > > > > > -----Original Message----- > > > > > > > From: xen-devel-bounces@lists.xen.org > > > > > > > [mailto:xen-devel-bounces@lists.xen.org] On Behalf Of Wei > > > > > > > Liu > > > > > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > > > > > To: zhangleiqiang > > > > > > > Cc: wei.liu2@citrix.com; xen-devel@lists.xen.org > > > > > > > Subject: Re: [Xen-devel] Poor network performance between > > > > > > > DomU with multiqueue support > > > > > > > > > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > > > > > Hi, all > > > > > > > > I am testing the performance of xen netfront-netback > > > > > > > > driver that with > > > > > > > multi-queues support. The throughput from domU to remote > > > > > > > dom0 is 9.2Gb/s, but the throughput from domU to remote domU > > > > > > > is only 3.6Gb/s, I think the bottleneck is the throughput > > > > > > > from dom0 to local domU. However, we have done some testing > > > > > > > and found the throughput from dom0 to local domU is 5.8Gb/s. > > > > > > > > And if we send packets from one DomU to other 3 DomUs > > > > > > > > on different > > > > > > > host simultaneously, the sum of throughout can reach 9Gbps. > > > > > > > It seems like the bottleneck is the receiver? > > > > > > > > After some analysis, I found that even the max_queue > > > > > > > > of netfront/back > > > > > > > is set to 4, there are some strange results as follows: > > > > > > > > 1. In domU, only one rx queue deal with softirq > > > > > > > > > > > > > > Try to bind irq to different vcpus? > > > > > > > > > > > > Do you mean we try to bind irq to different vcpus in DomU? I > > > > > > will try it > > > now. > > > > > > > > > > > > > > > > Yes. Given the fact that you have two backend threads running > > > > > while only one DomU vcpu is busy, it smells like misconfiguration in > DomU. > > > > > > > > > > If this phenomenon persists after correctly binding irqs, you > > > > > might want to check traffic is steering correctly to different queues. > > > > > > > > > > > > > > > > > > > > 2. In dom0, only two netback queues process are > > > > > > > > scheduled, other two > > > > > > > process aren't scheduled. > > > > > > > > > > > > > > How many Dom0 vcpu do you have? If it only has two then > > > > > > > there will only be two processes running at a time. > > > > > > > > > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU > > > > > > running in > > > > > Dom0 and so four netback processes are running in Dom0 (because > > > > > the max_queue param of netback kernel module is set to 4). > > > > > > The phenomenon is that only 2 of these four netback process > > > > > > were running > > > > > with about 70% cpu usage, and another two use little CPU. > > > > > > Is there a hash algorithm to determine which netback process > > > > > > to handle the > > > > > input packet? > > > > > > > > > > > > > > > > I think that's whatever default algorithm Linux kernel is using. > > > > > > > > > > We don't currently support other algorithms. > > > > > > > > > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 5 Dec 2014 12:42:33 +0000 Message-ID: <20141205124233.GD31446@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: [...] > > I think that's expected, because guest RX data path still uses grant_copy while > > guest TX uses grant_map to do zero-copy transmit. > > As far as I know, there are three main grant-related operations used in split device model: grant mapping, grant transfer and grant copy. > Grant transfer has not used now, and grant mapping and grant transfer both involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer has this overhead? Transfer is not used so I can't tell. Grant unmap causes TLB flush. I saw in an email the other day XenServer folks has some planned improvement to avoid TLB flush in Xen to upstream in 4.6 window. I can't speak for sure it will get upstreamed as I don't work on that. > Does grant copy surely has more overhead than grant mapping? > At the very least the zero-copy TX path is faster than previous copying path. But speaking of the micro operation I'm not sure. There was once persistent map prototype netback / netfront that establishes a memory pool between FE and BE then use memcpy to copy data. Unfortunately that prototype was not done right so the result was not good. > >From the code, I see that in TX, netback will do gnttab_batch_copy as well as gnttab_map_refs: > >

 //netback.c:xenvif_tx_action
> 	xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops);
> 
> 	if (nr_cops == 0)
> 		return 0;
> 
> 	gnttab_batch_copy(queue->tx_copy_ops, nr_cops);
> 	if (nr_mops != 0) {
> 		ret = gnttab_map_refs(queue->tx_map_ops,
> 				      NULL,
> 				      queue->pages_to_map,
> 				      nr_mops);
> 		BUG_ON(ret);
> 	}
>

> The copy is for the packet header. Mapping is for packet data. We need to copy header from guest so that it doesn't change under netback's feet. Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zoltan Kiss Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 05 Dec 2014 15:18:07 +0000 Message-ID: <5481CCAF.1040102@linaro.org> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141205124233.GD31446@zion.uk.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On 05/12/14 12:42, Wei Liu wrote: > On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: > [...] >>> I think that's expected, because guest RX data path still uses grant_copy while >>> guest TX uses grant_map to do zero-copy transmit. >> >> As far as I know, there are three main grant-related operations used in split device model: grant mapping, grant transfer and grant copy. >> Grant transfer has not used now, and grant mapping and grant transfer both involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer has this overhead? > > Transfer is not used so I can't tell. Grant unmap causes TLB flush. > > I saw in an email the other day XenServer folks has some planned > improvement to avoid TLB flush in Xen to upstream in 4.6 window. I can't > speak for sure it will get upstreamed as I don't work on that. > >> Does grant copy surely has more overhead than grant mapping? >> > > At the very least the zero-copy TX path is faster than previous copying > path. > > But speaking of the micro operation I'm not sure. > > There was once persistent map prototype netback / netfront that > establishes a memory pool between FE and BE then use memcpy to copy > data. Unfortunately that prototype was not done right so the result was > not good. > >> >From the code, I see that in TX, netback will do gnttab_batch_copy as well as gnttab_map_refs: >> >>

 //netback.c:xenvif_tx_action
>> 	xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops);
>>
>> 	if (nr_cops == 0)
>> 		return 0;
>>
>> 	gnttab_batch_copy(queue->tx_copy_ops, nr_cops);
>> 	if (nr_mops != 0) {
>> 		ret = gnttab_map_refs(queue->tx_map_ops,
>> 				      NULL,
>> 				      queue->pages_to_map,
>> 				      nr_mops);
>> 		BUG_ON(ret);
>> 	}
>>

>> > > The copy is for the packet header. Mapping is for packet data. > > We need to copy header from guest so that it doesn't change under > netback's feet. It is also important because if the above mentioned "TLB flush avoidance" patch goes in to Xen, it will be important to grant copy the header rather than grant map plus memcpy. The latter is the old way, it touches the page so you can't avoid TLB flush. > > Wei. > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel > From mboxrd@z Thu Jan 1 00:00:00 1970 From: Zoltan Kiss Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 05 Dec 2014 15:20:55 +0000 Message-ID: <5481CD57.607@linaro.org> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> <54806315.6010007@linaro.org> <3A6795EA1206904E94BEC8EF9DF109AE23933926@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE23933926@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" , "xen-devel@lists.xen.org" Cc: jonathan.davies@citrix.com, "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On 04/12/14 14:31, Zhangleiqiang (Trump) wrote: >> -----Original Message----- >> From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] >> Sent: Thursday, December 04, 2014 9:35 PM >> To: Zhangleiqiang (Trump); Wei Liu; xen-devel@lists.xen.org >> Cc: Xiaoding (B); Zhuangyuxin; zhangleiqiang; Luohao (brian); Yuzhou (C) >> Subject: Re: [Xen-devel] Poor network performance between DomU with >> multiqueue support >> >> >> >> On 04/12/14 12:09, Zhangleiqiang (Trump) wrote: >>>> I think that's expected, because guest RX data path still uses >>>> grant_copy while >>>>> guest TX uses grant_map to do zero-copy transmit. >>> As I understand, the RX process is as follows: >>> 1. Phy NIC receive packet >>> 2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do >>> the "RX" operation, and the packet is stored into SKB which is also >>> owned/shared with netback >> Not that easy. There is something between the NIC driver and netback which >> directs the packets, e.g. the old bridge driver, ovs, or the IP stack of the kernel. >>> 4. NetBack notify netfront through event channel that a packet is >>> receiving 5. Netfront grant a buffer for receiving and notify netback >>> the GR (if using grant-resue mechanism, netfront just notify the GR to >>> netback) through IO Ring >> It looks a bit confusing in the code, but netfront put "requests" on the ring >> buffer, which contains the grant ref of the guest page where the backend can >> copy. When the packet comes, netback consumes these requests and send >> back a response telling the guest the grant copy of the packet finished, it can >> start handling the data. (sending a response means it's placing a response in >> the ring and trigger the event channel) And ideally netback should always have >> requests in the ring, so it doesn't have to wait for the guest to fill it up. > >>> 6. NetBack do the grant_copy to copy packet from its SKB to the buffer >>> referenced by GR, and notify netfront through event channel 7. >>> Netfront copy the data from buffer to user-level app's SKB >> Or wherever that SKB should go, yes. Like with any received packet on a real >> network interface. >>> >>> Am I right? Why not using zero-copy transmit in guest RX data pash too ? >> Because that means you are mapping that memory to the guest, and you won't >> have any guarantee when the guest will release them. And netback can't just >> unmap them forcibly after a timeout, because finding a correct timeout value >> would be quite impossible. >> A malicious/buggy/overloaded guest can hold on to Dom0 memory indefinitely, >> but it even becomes worse if the memory came from another >> guest: you can't shutdown that guest for example, until all its memory is >> returned to him. > > Thanks for your detailed explanation about RX data path, I have get it, :) > > About the issue that poor performance between DomU to DomU, but high throughout between Dom0 to remote Dom0/DomU mentioned in my previous mail, do you have any idea about it? > > I am wondering if netfront/netback can be optimized to reach the 10Gbps throughout between DomUs running on different hosts connected with 10GE network. Currently, it seems like the TX is not the bottleneck, because we can reach the aggregate throughout of 9Gbps when sending packets from one DomU to other 3 DomUs running on different host. So I think the bottleneck maybe the RX, are you agreed with me? > > I am wondering what is the main reason that prevent RX to reach the higher throughout? Compared to KVM+virtio+vhost, which can reach high throughout, the RX has extra grantcopy operation, and the grantcopy operation may be one reason for it. Do you have any idea about it too? It's quite sure that the grant copy is the bottleneck for a single queue RX traffic. I don't know what's the plan to help that, currently only a faster CPU can help you with that. > >> >> Regards, >> >> Zoli From mboxrd@z Thu Jan 1 00:00:00 1970 From: Konrad Rzeszutek Wilk Subject: Re: Poor network performance between DomU with multiqueue support Date: Fri, 5 Dec 2014 13:27:02 -0500 Message-ID: <20141205182702.GA4754@laptop.dumpdata.com> References: <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> <54806315.6010007@linaro.org> <3A6795EA1206904E94BEC8EF9DF109AE23933926@SZXEMA512-MBX.china.huawei.com> <5481CD57.607@linaro.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <5481CD57.607@linaro.org> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Zoltan Kiss Cc: "Zhangleiqiang (Trump)" , jonathan.davies@citrix.com, "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Fri, Dec 05, 2014 at 03:20:55PM +0000, Zoltan Kiss wrote: > > > On 04/12/14 14:31, Zhangleiqiang (Trump) wrote: > >>-----Original Message----- > >>From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > >>Sent: Thursday, December 04, 2014 9:35 PM > >>To: Zhangleiqiang (Trump); Wei Liu; xen-devel@lists.xen.org > >>Cc: Xiaoding (B); Zhuangyuxin; zhangleiqiang; Luohao (brian); Yuzhou (C) > >>Subject: Re: [Xen-devel] Poor network performance between DomU with > >>multiqueue support > >> > >> > >> > >>On 04/12/14 12:09, Zhangleiqiang (Trump) wrote: > >>>>I think that's expected, because guest RX data path still uses > >>>>grant_copy while > >>>>>guest TX uses grant_map to do zero-copy transmit. > >>>As I understand, the RX process is as follows: > >>>1. Phy NIC receive packet > >>>2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver do > >>>the "RX" operation, and the packet is stored into SKB which is also > >>>owned/shared with netback > >>Not that easy. There is something between the NIC driver and netback which > >>directs the packets, e.g. the old bridge driver, ovs, or the IP stack of the kernel. > >>>4. NetBack notify netfront through event channel that a packet is > >>>receiving 5. Netfront grant a buffer for receiving and notify netback > >>>the GR (if using grant-resue mechanism, netfront just notify the GR to > >>>netback) through IO Ring > >>It looks a bit confusing in the code, but netfront put "requests" on the ring > >>buffer, which contains the grant ref of the guest page where the backend can > >>copy. When the packet comes, netback consumes these requests and send > >>back a response telling the guest the grant copy of the packet finished, it can > >>start handling the data. (sending a response means it's placing a response in > >>the ring and trigger the event channel) And ideally netback should always have > >>requests in the ring, so it doesn't have to wait for the guest to fill it up. > > > >>>6. NetBack do the grant_copy to copy packet from its SKB to the buffer > >>>referenced by GR, and notify netfront through event channel 7. > >>>Netfront copy the data from buffer to user-level app's SKB > >>Or wherever that SKB should go, yes. Like with any received packet on a real > >>network interface. > >>> > >>>Am I right? Why not using zero-copy transmit in guest RX data pash too ? > >>Because that means you are mapping that memory to the guest, and you won't > >>have any guarantee when the guest will release them. And netback can't just > >>unmap them forcibly after a timeout, because finding a correct timeout value > >>would be quite impossible. > >>A malicious/buggy/overloaded guest can hold on to Dom0 memory indefinitely, > >>but it even becomes worse if the memory came from another > >>guest: you can't shutdown that guest for example, until all its memory is > >>returned to him. > > > >Thanks for your detailed explanation about RX data path, I have get it, :) > > > >About the issue that poor performance between DomU to DomU, but high throughout between Dom0 to remote Dom0/DomU mentioned in my previous mail, do you have any idea about it? > > > >I am wondering if netfront/netback can be optimized to reach the 10Gbps throughout between DomUs running on different hosts connected with 10GE network. Currently, it seems like the TX is not the bottleneck, because we can reach the aggregate throughout of 9Gbps when sending packets from one DomU to other 3 DomUs running on different host. So I think the bottleneck maybe the RX, are you agreed with me? > > > >I am wondering what is the main reason that prevent RX to reach the higher throughout? Compared to KVM+virtio+vhost, which can reach high throughout, the RX has extra grantcopy operation, and the grantcopy operation may be one reason for it. Do you have any idea about it too? > It's quite sure that the grant copy is the bottleneck for a single queue RX > traffic. I don't know what's the plan to help that, currently only a faster > CPU can help you with that. Could the Intel QuickData help with that? > > > > >> > >>Regards, > >> > >>Zoli > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xen.org > http://lists.xen.org/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Mon, 8 Dec 2014 06:50:15 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2394B199@SZXEMA512-MBX.china.huawei.com> References: <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393371E@SZXEMA512-MBX.china.huawei.com> <54806315.6010007@linaro.org> <3A6795EA1206904E94BEC8EF9DF109AE23933926@SZXEMA512-MBX.china.huawei.com> <5481CD57.607@linaro.org> <20141205182702.GA4754@laptop.dumpdata.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141205182702.GA4754@laptop.dumpdata.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Konrad Rzeszutek Wilk , Zoltan Kiss , "xen-devel@lists.xen.org" Cc: "jonathan.davies@citrix.com" , "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org > On Fri, Dec 05, 2014 at 03:20:55PM +0000, Zoltan Kiss wrote: > > > > > > On 04/12/14 14:31, Zhangleiqiang (Trump) wrote: > > >>-----Original Message----- > > >>From: Zoltan Kiss [mailto:zoltan.kiss@linaro.org] > > >>Sent: Thursday, December 04, 2014 9:35 PM > > >>To: Zhangleiqiang (Trump); Wei Liu; xen-devel@lists.xen.org > > >>Cc: Xiaoding (B); Zhuangyuxin; zhangleiqiang; Luohao (brian); Yuzhou > > >>(C) > > >>Subject: Re: [Xen-devel] Poor network performance between DomU with > > >>multiqueue support > > >> > > >> > > >> > > >>On 04/12/14 12:09, Zhangleiqiang (Trump) wrote: > > >>>>I think that's expected, because guest RX data path still uses > > >>>>grant_copy while > > >>>>>guest TX uses grant_map to do zero-copy transmit. > > >>>As I understand, the RX process is as follows: > > >>>1. Phy NIC receive packet > > >>>2. XEN Hypervisor trigger interrupt to Dom0 3. Dom0' s NIC driver > > >>>do the "RX" operation, and the packet is stored into SKB which is > > >>>also owned/shared with netback > > >>Not that easy. There is something between the NIC driver and netback > > >>which directs the packets, e.g. the old bridge driver, ovs, or the IP stack of > the kernel. > > >>>4. NetBack notify netfront through event channel that a packet is > > >>>receiving 5. Netfront grant a buffer for receiving and notify > > >>>netback the GR (if using grant-resue mechanism, netfront just > > >>>notify the GR to > > >>>netback) through IO Ring > > >>It looks a bit confusing in the code, but netfront put "requests" on > > >>the ring buffer, which contains the grant ref of the guest page > > >>where the backend can copy. When the packet comes, netback consumes > > >>these requests and send back a response telling the guest the grant > > >>copy of the packet finished, it can start handling the data. > > >>(sending a response means it's placing a response in the ring and > > >>trigger the event channel) And ideally netback should always have requests > in the ring, so it doesn't have to wait for the guest to fill it up. > > > > > >>>6. NetBack do the grant_copy to copy packet from its SKB to the > > >>>buffer referenced by GR, and notify netfront through event channel 7. > > >>>Netfront copy the data from buffer to user-level app's SKB > > >>Or wherever that SKB should go, yes. Like with any received packet > > >>on a real network interface. > > >>> > > >>>Am I right? Why not using zero-copy transmit in guest RX data pash too ? > > >>Because that means you are mapping that memory to the guest, and you > > >>won't have any guarantee when the guest will release them. And > > >>netback can't just unmap them forcibly after a timeout, because > > >>finding a correct timeout value would be quite impossible. > > >>A malicious/buggy/overloaded guest can hold on to Dom0 memory > > >>indefinitely, but it even becomes worse if the memory came from > > >>another > > >>guest: you can't shutdown that guest for example, until all its > > >>memory is returned to him. > > > > > >Thanks for your detailed explanation about RX data path, I have get > > >it, :) > > > > > >About the issue that poor performance between DomU to DomU, but high > throughout between Dom0 to remote Dom0/DomU mentioned in my previous > mail, do you have any idea about it? > > > > > >I am wondering if netfront/netback can be optimized to reach the 10Gbps > throughout between DomUs running on different hosts connected with 10GE > network. Currently, it seems like the TX is not the bottleneck, because we can > reach the aggregate throughout of 9Gbps when sending packets from one > DomU to other 3 DomUs running on different host. So I think the bottleneck > maybe the RX, are you agreed with me? > > > > > >I am wondering what is the main reason that prevent RX to reach the higher > throughout? Compared to KVM+virtio+vhost, which can reach high throughout, > the RX has extra grantcopy operation, and the grantcopy operation may be one > reason for it. Do you have any idea about it too? > > It's quite sure that the grant copy is the bottleneck for a single > > queue RX traffic. I don't know what's the plan to help that, currently > > only a faster CPU can help you with that. > > Could the Intel QuickData help with that? Thanks for your hit. I am looking for method which is independent on hardware. Because I have seen that virtio can reach the 10Gbps throughout, and I think PV network protocol which is the mainline of XEN should also reach the throughout. However, the testing results show that it is not ideal, so I am wondering what the possible reason is and if PV network protocol can be optimized. > > > > > > > >> > > >>Regards, > > >> > > >>Zoli > > > > _______________________________________________ > > Xen-devel mailing list > > Xen-devel@lists.xen.org > > http://lists.xen.org/xen-devel From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Mon, 8 Dec 2014 06:44:26 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141205124233.GD31446@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Zhangleiqiang (Trump)" , "Luohao (brian)" , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org > On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: > [...] > > > I think that's expected, because guest RX data path still uses > > > grant_copy while guest TX uses grant_map to do zero-copy transmit. > > > > As far as I know, there are three main grant-related operations used in split > device model: grant mapping, grant transfer and grant copy. > > Grant transfer has not used now, and grant mapping and grant transfer both > involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer > has this overhead? > > Transfer is not used so I can't tell. Grant unmap causes TLB flush. > > I saw in an email the other day XenServer folks has some planned improvement > to avoid TLB flush in Xen to upstream in 4.6 window. I can't speak for sure it will > get upstreamed as I don't work on that. > > > Does grant copy surely has more overhead than grant mapping? > > > > At the very least the zero-copy TX path is faster than previous copying path. > > But speaking of the micro operation I'm not sure. > > There was once persistent map prototype netback / netfront that establishes a > memory pool between FE and BE then use memcpy to copy data. Unfortunately > that prototype was not done right so the result was not good. The newest mail about persistent grant I can find is sent from 16 Nov 2012 (http://lists.xen.org/archives/html/xen-devel/2012-11/msg00832.html). Why is it not done right and not merged into upstream? And I also search for virtio support in XEN, and I find that the one who are familiar with it is you, too, (http://wiki.xen.org/wiki/Virtio_On_Xen), :-). I am wondering what is the current state for virtio on XEN? > > >From the code, I see that in TX, netback will do gnttab_batch_copy as well > as gnttab_map_refs: > > > >

 //netback.c:xenvif_tx_action
> > 	xenvif_tx_build_gops(queue, budget, &nr_cops, &nr_mops);
> >
> > 	if (nr_cops == 0)
> > 		return 0;
> >
> > 	gnttab_batch_copy(queue->tx_copy_ops, nr_cops);
> > 	if (nr_mops != 0) {
> > 		ret = gnttab_map_refs(queue->tx_map_ops,
> > 				      NULL,
> > 				      queue->pages_to_map,
> > 				      nr_mops);
> > 		BUG_ON(ret);
> > 	}
> >

> > > > The copy is for the packet header. Mapping is for packet data. > > We need to copy header from guest so that it doesn't change under netback's > feet. > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Mon, 8 Dec 2014 10:13:04 +0000 Message-ID: <20141208101304.GB17128@zion.uk.xensource.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Mon, Dec 08, 2014 at 06:44:26AM +0000, Zhangleiqiang (Trump) wrote: > > On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: > > [...] > > > > I think that's expected, because guest RX data path still uses > > > > grant_copy while guest TX uses grant_map to do zero-copy transmit. > > > > > > As far as I know, there are three main grant-related operations used in split > > device model: grant mapping, grant transfer and grant copy. > > > Grant transfer has not used now, and grant mapping and grant transfer both > > involve "TLB" refresh work for hypervisor, am I right? Or only grant transfer > > has this overhead? > > > > Transfer is not used so I can't tell. Grant unmap causes TLB flush. > > > > I saw in an email the other day XenServer folks has some planned improvement > > to avoid TLB flush in Xen to upstream in 4.6 window. I can't speak for sure it will > > get upstreamed as I don't work on that. > > > > > Does grant copy surely has more overhead than grant mapping? > > > > > > > At the very least the zero-copy TX path is faster than previous copying path. > > > > But speaking of the micro operation I'm not sure. > > > > There was once persistent map prototype netback / netfront that establishes a > > memory pool between FE and BE then use memcpy to copy data. Unfortunately > > that prototype was not done right so the result was not good. > > The newest mail about persistent grant I can find is sent from 16 Nov > 2012 > (http://lists.xen.org/archives/html/xen-devel/2012-11/msg00832.html). > Why is it not done right and not merged into upstream? AFAICT there's one more memcpy than necessary, i.e. frontend memcpy data into the pool then backend memcpy data out of the pool, when backend should be able to use the page in pool directly. > > And I also search for virtio support in XEN, and I find that the one > who are familiar with it is you, too, > (http://wiki.xen.org/wiki/Virtio_On_Xen), :-). I am wondering what is > the current state for virtio on XEN? Yes, it was me. I never have the time to revisit that. I don't think we support virtio network at the moment. Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Mon, 8 Dec 2014 13:08:18 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2394C523@SZXEMA512-MBX.china.huawei.com> References: <20141202110133.GA5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23931FA1@SZXEMA512-MBX.china.huawei.com> <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> <20141208101304.GB17128@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <20141208101304.GB17128@zion.uk.xensource.com> Content-Language: zh-CN List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Wei Liu , "xen-devel@lists.xen.org" Cc: "Xiaoding (B)" , Zhuangyuxin , zhangleiqiang , "Luohao (brian)" , "Yuzhou (C)" List-Id: xen-devel@lists.xenproject.org > On Mon, Dec 08, 2014 at 06:44:26AM +0000, Zhangleiqiang (Trump) wrote: > > > On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: > > > [...] > > > > > I think that's expected, because guest RX data path still uses > > > > > grant_copy while guest TX uses grant_map to do zero-copy transmit. > > > > > > > > As far as I know, there are three main grant-related operations > > > > used in split > > > device model: grant mapping, grant transfer and grant copy. > > > > Grant transfer has not used now, and grant mapping and grant > > > > transfer both > > > involve "TLB" refresh work for hypervisor, am I right? Or only > > > grant transfer has this overhead? > > > > > > Transfer is not used so I can't tell. Grant unmap causes TLB flush. > > > > > > I saw in an email the other day XenServer folks has some planned > > > improvement to avoid TLB flush in Xen to upstream in 4.6 window. I > > > can't speak for sure it will get upstreamed as I don't work on that. > > > > > > > Does grant copy surely has more overhead than grant mapping? > > > > > > > > > > At the very least the zero-copy TX path is faster than previous copying path. > > > > > > But speaking of the micro operation I'm not sure. > > > > > > There was once persistent map prototype netback / netfront that > > > establishes a memory pool between FE and BE then use memcpy to copy > > > data. Unfortunately that prototype was not done right so the result was not > good. > > > > The newest mail about persistent grant I can find is sent from 16 Nov > > 2012 > > (http://lists.xen.org/archives/html/xen-devel/2012-11/msg00832.html). > > Why is it not done right and not merged into upstream? > > AFAICT there's one more memcpy than necessary, i.e. frontend memcpy data > into the pool then backend memcpy data out of the pool, when backend should > be able to use the page in pool directly. Memcpy should cheaper than grant_copy because the former needs not the "hypercall" which will cause "VM Exit" to "XEN Hypervisor", am I right? For RX path, using memcpy based on persistent grant table may have higher performance than using grant copy now. I have seen "move grant copy to guest" and "Fix grant copy alignment problem" as optimization methods used in "NetChannel2" (http://www-archive.xenproject.org/files/xensummit_fall07/16_JoseRenatoSantos.pdf). Unfortunately, NetChannel2 seems not be supported from 2.6.32. Do you know them and are them be helpful for RX path optimization under current upstream implementation? By the way, after rethinking the testing results for multi-queue pv (kernel 3.17.4+XEN 4.4) implementation, I find that when using four queues for netback/netfront, there will be about 3 netback process running with high CPU usage on receive Dom0 (about 85% usage per process running on one CPU core), and the aggregate throughout is only about 5Gbps. I doubt that there may be some bug or pitfall in current multi-queue implementation, because for 5Gbps throughout, occurring about all of 3 CPU core for packet receiving is somehow abnormal. > > > > And I also search for virtio support in XEN, and I find that the one > > who are familiar with it is you, too, > > (http://wiki.xen.org/wiki/Virtio_On_Xen), :-). I am wondering what is > > the current state for virtio on XEN? > > Yes, it was me. I never have the time to revisit that. I don't think we support > virtio network at the moment. > > Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: Re: Poor network performance between DomU with multiqueue support Date: Mon, 8 Dec 2014 13:55:34 +0000 Message-ID: <20141208135534.GA21374@zion.uk.xensource.com> References: <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> <20141208101304.GB17128@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394C523@SZXEMA512-MBX.china.huawei.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: Content-Disposition: inline In-Reply-To: <3A6795EA1206904E94BEC8EF9DF109AE2394C523@SZXEMA512-MBX.china.huawei.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: "Zhangleiqiang (Trump)" Cc: "Luohao (brian)" , Wei Liu , Zhuangyuxin , zhangleiqiang , "Yuzhou (C)" , "xen-devel@lists.xen.org" , "Xiaoding (B)" List-Id: xen-devel@lists.xenproject.org On Mon, Dec 08, 2014 at 01:08:18PM +0000, Zhangleiqiang (Trump) wrote: > > On Mon, Dec 08, 2014 at 06:44:26AM +0000, Zhangleiqiang (Trump) wrote: > > > > On Fri, Dec 05, 2014 at 01:17:16AM +0000, Zhangleiqiang (Trump) wrote: > > > > [...] > > > > > > I think that's expected, because guest RX data path still uses > > > > > > grant_copy while guest TX uses grant_map to do zero-copy transmit. > > > > > > > > > > As far as I know, there are three main grant-related operations > > > > > used in split > > > > device model: grant mapping, grant transfer and grant copy. > > > > > Grant transfer has not used now, and grant mapping and grant > > > > > transfer both > > > > involve "TLB" refresh work for hypervisor, am I right? Or only > > > > grant transfer has this overhead? > > > > > > > > Transfer is not used so I can't tell. Grant unmap causes TLB flush. > > > > > > > > I saw in an email the other day XenServer folks has some planned > > > > improvement to avoid TLB flush in Xen to upstream in 4.6 window. I > > > > can't speak for sure it will get upstreamed as I don't work on that. > > > > > > > > > Does grant copy surely has more overhead than grant mapping? > > > > > > > > > > > > > At the very least the zero-copy TX path is faster than previous copying path. > > > > > > > > But speaking of the micro operation I'm not sure. > > > > > > > > There was once persistent map prototype netback / netfront that > > > > establishes a memory pool between FE and BE then use memcpy to copy > > > > data. Unfortunately that prototype was not done right so the result was not > > good. > > > > > > The newest mail about persistent grant I can find is sent from 16 Nov > > > 2012 > > > (http://lists.xen.org/archives/html/xen-devel/2012-11/msg00832.html). > > > Why is it not done right and not merged into upstream? > > > > AFAICT there's one more memcpy than necessary, i.e. frontend memcpy data > > into the pool then backend memcpy data out of the pool, when backend should > > be able to use the page in pool directly. > > Memcpy should cheaper than grant_copy because the former needs not the > "hypercall" which will cause "VM Exit" to "XEN Hypervisor", am I > right? For RX path, using memcpy based on persistent grant table may > have higher performance than using grant copy now. In theory yes. Unfortunately nobody has benchmarked that properly. If you're interested in doing work on optimising RX performance, you might want to sync up with XenServer folks? > > I have seen "move grant copy to guest" and "Fix grant copy alignment > problem" as optimization methods used in "NetChannel2" > (http://www-archive.xenproject.org/files/xensummit_fall07/16_JoseRenatoSantos.pdf). > Unfortunately, NetChannel2 seems not be supported from 2.6.32. Do you > know them and are them be helpful for RX path optimization under > current upstream implementation? Not sure, that's long before I ever started working on Xen. > > By the way, after rethinking the testing results for multi-queue pv > (kernel 3.17.4+XEN 4.4) implementation, I find that when using four > queues for netback/netfront, there will be about 3 netback process > running with high CPU usage on receive Dom0 (about 85% usage per > process running on one CPU core), and the aggregate throughout is only > about 5Gbps. I doubt that there may be some bug or pitfall in current > multi-queue implementation, because for 5Gbps throughout, occurring > about all of 3 CPU core for packet receiving is somehow abnormal. > 3.17.4 doesn't contain David Vrabel's fixes. Look for bc96f648df1bbc2729abbb84513cf4f64273a1f1 f48da8b14d04ca87ffcffe68829afd45f926ec6a ecf08d2dbb96d5a4b4bcc53a39e8d29cc8fef02e in David Miller's net tree. BTW there are some improvement planned for 4.6: "[Xen-devel] [PATCH v3 0/2] gnttab: Improve scaleability". This is orthogonal to the problem you're trying to solve but it should help improve performance in general. Wei. From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Zhangleiqiang (Trump)" Subject: Re: Poor network performance between DomU with multiqueue support Date: Tue, 9 Dec 2014 02:51:55 +0000 Message-ID: <3A6795EA1206904E94BEC8EF9DF109AE2394DA29@SZXEMA512-MBX.china.huawei.com> References: <20141202121151.GD5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393216B@SZXEMA512-MBX.china.huawei.com> <20141202155832.GH5768@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2393301C@SZXEMA512-MBX.china.huawei.com> <20141204105021.GA16532@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE23933CDA@SZXEMA512-MBX.china.huawei.com> <20141205124233.GD31446@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394A187@SZXEMA512-MBX.china.huawei.com> <20141208101304.GB17128@zion.uk.xensource.com> <3A6795EA1206904E94BEC8EF9DF109AE2394C523@SZXEMA512-MBX.china.huawei.com> <20141208135534.GA21374@zion.uk.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: