From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net Date: Thu, 28 Sep 2017 01:28:54 +0300 Message-ID: <20170928012009-mutt-send-email-mst@kernel.org> References: <1506067355-5771-1-git-send-email-jasowang@redhat.com> <20170926164055-mutt-send-email-mst@kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Jason Wang Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, virtualization@lists.linux-foundation.org List-Id: virtualization@lists.linuxfoundation.org T24gV2VkLCBTZXAgMjcsIDIwMTcgYXQgMDg6Mjc6MzdBTSArMDgwMCwgSmFzb24gV2FuZyB3cm90 ZToKPiAKPiAKPiBPbiAyMDE35bm0MDnmnIgyNuaXpSAyMTo0NSwgTWljaGFlbCBTLiBUc2lya2lu IHdyb3RlOgo+ID4gT24gRnJpLCBTZXAgMjIsIDIwMTcgYXQgMDQ6MDI6MzBQTSArMDgwMCwgSmFz b24gV2FuZyB3cm90ZToKPiA+ID4gSGk6Cj4gPiA+IAo+ID4gPiBUaGlzIHNlcmllcyB0cmllcyB0 byBpbXBsZW1lbnQgYmFzaWMgdHggYmF0Y2hlZCBwcm9jZXNzaW5nLiBUaGlzIGlzCj4gPiA+IGRv bmUgYnkgcHJlZmV0Y2hpbmcgZGVzY3JpcHRvciBpbmRpY2VzIGFuZCB1cGRhdGUgdXNlZCByaW5n IGluIGEKPiA+ID4gYmF0Y2guIFRoaXMgaW50ZW5kcyB0byBzcGVlZCB1cCB1c2VkIHJpbmcgdXBk YXRpbmcgYW5kIGltcHJvdmUgdGhlCj4gPiA+IGNhY2hlIHV0aWxpemF0aW9uLgo+ID4gSW50ZXJl c3RpbmcsIHRoYW5rcyBmb3IgdGhlIHBhdGNoZXMuIFNvIElJVUMgbW9zdCBvZiB0aGUgZ2FpbiBp cyByZWFsbHkKPiA+IG92ZXJjb21pbmcgc29tZSBvZiB0aGUgc2hvcnRjb21pbmdzIG9mIHZpcnRp byAxLjAgd3J0IGNhY2hlIHV0aWxpemF0aW9uPwo+IAo+IFllcy4KPiAKPiBBY3R1YWxseSwgbG9v a3MgbGlrZSBiYXRjaGluZyBpbiAxLjEgaXMgbm90IGFzIGVhc3kgYXMgaW4gMS4wLgo+IAo+IElu IDEuMCwgd2UgY291bGQgZG8gc29tZXRoaW5nIGxpa2U6Cj4gCj4gYmF0Y2ggdXBkYXRlIHVzZWQg cmluZyBieSB1c2VyIGNvcHlfdG9fdXNlcigpCj4gc21wX3dtYigpCj4gdXBkYXRlIHVzZWRfaWR4 Cj4gSW4gMS4xLCB3ZSBuZWVkIG1vcmUgbWVtb3J5IGJhcnJpZXJzLCBjYW4ndCBiZW5lZml0IGZy b20gZmFzdCBjb3B5IGhlbHBlcnM/Cj4gCj4gZm9yICgpIHsKPiDCoMKgwqAgdXBkYXRlIGRlc2Mu YWRkcgo+IMKgwqDCoCBzbXBfd21iKCkKPiDCoMKgwqAgdXBkYXRlIGRlc2MuZmxhZwo+IH0KClll cyBidXQgc21wX3dtYiBpcyBhIE5PUCBvbiBlLmcuIHg4Ni4gV2UgY2FuIHN3aXRjaCB0byBvdGhl ciB0eXBlcyBvZgpiYXJyaWVycyBhcyB3ZWxsLiAgV2UgZG8gbmVlZCB0byBkbyB0aGUgdXBkYXRl cyBpbiBvcmRlciwgc28gd2UgbWlnaHQKbmVlZCBuZXcgQVBJcyBmb3IgdGhhdCB0byBhdm9pZCBy ZS1kb2luZyB0aGUgdHJhbnNsYXRpb24gYWxsIHRoZSB0aW1lLgoKSW4gMS4wIHRoZSBsYXN0IHVw ZGF0ZSBpcyBhIGNhY2hlIG1pc3MgYWx3YXlzLiBZb3UgbmVlZCBiYXRjaGluZyB0byBnZXQKbGVz cyBtaXNzZXMuIEluIDEuMSB5b3UgZG9uJ3QgaGF2ZSBpdCBzbyBmdW5kYW1lbnRhbGx5IHRoZXJl IGlzIGxlc3MKbmVlZCBmb3IgYmF0Y2hpbmcuIEJ1dCBiYXRjaGluZyBkb2VzIG5vdCBhbHdheXMg d29yay4gIERQREsgZ3V5cyAod2hpY2gKYmF0Y2ggdGhpbmdzIGFnZ3Jlc3NpdmVseSkgYWxyZWFk eSB0cmllZCAxLjEgYW5kIHNhdyBwZXJmb3JtYW5jZSBnYWlucwpzbyB3ZSBkbyBub3QgbmVlZCB0 byBhcmd1ZSB0aGVvcmV0aWNhbGx5LgoKCgo+ID4gCj4gPiBXaGljaCBpcyBmYWlyIGVub3VnaCAo MS4wIGlzIGFscmVhZHkgZGVwbG95ZWQpIGJ1dCBJIHdvdWxkIGxpa2UgdG8gYXZvaWQKPiA+IG1h a2luZyAxLjEgc3VwcG9ydCBoYXJkZXIsIGFuZCB0aGlzIHBhdGNoc2V0IGRvZXMgdGhpcyB1bmZv cnR1bmF0ZWx5LAo+IAo+IEkgdGhpbmsgdGhlIG5ldyBBUElzIGRvIG5vdCBleHBvc2UgbW9yZSBp bnRlcm5hbCBkYXRhIHN0cnVjdHVyZSBvZiB2aXJ0aW8KPiB0aGFuIGJlZm9yZT8gKHZxLT5oZWFk cyBoYXMgYWxyZWFkeSBiZWVuIHVzZWQgYnkgdmhvc3RfbmV0IGZvciB5ZWFycykuCgpGb3Igc3Vy ZSB3ZSBtaWdodCBuZWVkIHRvIGNoYW5nZSB2cmluZ191c2VkX2VsZW0uCgo+IENvbnNpZGVyIHRo ZSBsYXlvdXQgaXMgcmUtZGVzaWduZWQgY29tcGxldGVseSwgSSBkb24ndCBzZWUgYW4gZWFzeSBt ZXRob2QgdG8KPiByZXVzZSBjdXJyZW50IDEuMCBBUEkgZm9yIDEuMS4KCkN1cnJlbnQgQVBJIGp1 c3Qgc2F5cyB5b3UgZ2V0IGJ1ZmZlcnMgdGhlbiB5b3UgdXNlIHRoZW0uIEl0IGlzIG5vdCB0aWVk CnRvIGFjdHVhbCBzZXBhcmF0ZSB1c2VkIHJpbmcuCgoKPiA+IHNlZSBjb21tZW50cyBvbiBpbmRp dmlkdWFsIHBhdGNoZXMuIEknbSBzdXJlIGl0IGNhbiBiZSBhZGRyZXNzZWQgdGhvdWdoLgo+ID4g Cj4gPiA+IFRlc3Qgc2hvd3MgYWJvdXQgfjIyJSBpbXByb3ZlbWVudCBpbiB0eCBwc3MuCj4gPiBJ cyB0aGlzIHdpdGggb3Igd2l0aG91dCB0eCBuYXBpIGluIGd1ZXN0Pwo+IAo+IE1vb25HZW4gaXMg dXNlZCBpbiBndWVzdCBmb3IgYmV0dGVyIG51bWJlcnMuCj4gCj4gVGhhbmtzCgpOb3Qgc3VyZSBJ IHVuZGVyc3RhbmQuIERpZCB5b3Ugc2V0IG5hcGlfdHggdG8gdHJ1ZSBvciBmYWxzZT8KCj4gPiAK PiA+ID4gUGxlYXNlIHJldmlldy4KPiA+ID4gCj4gPiA+IEphc29uIFdhbmcgKDUpOgo+ID4gPiAg ICB2aG9zdDogc3BsaXQgb3V0IHJpbmcgaGVhZCBmZXRjaGluZyBsb2dpYwo+ID4gPiAgICB2aG9z dDogaW50cm9kdWNlIGhlbHBlciB0byBwcmVmZXRjaCBkZXNjIGluZGV4Cj4gPiA+ICAgIHZob3N0 OiBpbnRyb2R1Y2Ugdmhvc3RfYWRkX3VzZWRfaWR4KCkKPiA+ID4gICAgdmhvc3RfbmV0OiByZW5h bWUgVkhPU1RfUlhfQkFUQ0ggdG8gVkhPU1RfTkVUX0JBVENICj4gPiA+ICAgIHZob3N0X25ldDog YmFzaWMgdHggdmlydHF1ZXVlIGJhdGNoZWQgcHJvY2Vzc2luZwo+ID4gPiAKPiA+ID4gICBkcml2 ZXJzL3Zob3N0L25ldC5jICAgfCAyMjEgKysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0t LS0tLS0tLS0tLS0tLS0tLS0KPiA+ID4gICBkcml2ZXJzL3Zob3N0L3Zob3N0LmMgfCAxNjUgKysr KysrKysrKysrKysrKysrKysrKysrKysrKysrKy0tLS0tLQo+ID4gPiAgIGRyaXZlcnMvdmhvc3Qv dmhvc3QuaCB8ICAgOSArKwo+ID4gPiAgIDMgZmlsZXMgY2hhbmdlZCwgMjcwIGluc2VydGlvbnMo KyksIDEyNSBkZWxldGlvbnMoLSkKPiA+ID4gCj4gPiA+IC0tIAo+ID4gPiAyLjcuNApfX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwpWaXJ0dWFsaXphdGlvbiBt YWlsaW5nIGxpc3QKVmlydHVhbGl6YXRpb25AbGlzdHMubGludXgtZm91bmRhdGlvbi5vcmcKaHR0 cHM6Ly9saXN0cy5saW51eGZvdW5kYXRpb24ub3JnL21haWxtYW4vbGlzdGluZm8vdmlydHVhbGl6 YXRpb24= From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752206AbdI0W25 (ORCPT ); Wed, 27 Sep 2017 18:28:57 -0400 Received: from mx1.redhat.com ([209.132.183.28]:41096 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751355AbdI0W24 (ORCPT ); Wed, 27 Sep 2017 18:28:56 -0400 DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com CB099257E95 Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx10.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=mst@redhat.com Date: Thu, 28 Sep 2017 01:28:54 +0300 From: "Michael S. Tsirkin" To: Jason Wang Cc: virtualization@lists.linux-foundation.org, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, kvm@vger.kernel.org Subject: Re: [PATCH net-next RFC 0/5] batched tx processing in vhost_net Message-ID: <20170928012009-mutt-send-email-mst@kernel.org> References: <1506067355-5771-1-git-send-email-jasowang@redhat.com> <20170926164055-mutt-send-email-mst@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.39]); Wed, 27 Sep 2017 22:28:55 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Sep 27, 2017 at 08:27:37AM +0800, Jason Wang wrote: > > > On 2017年09月26日 21:45, Michael S. Tsirkin wrote: > > On Fri, Sep 22, 2017 at 04:02:30PM +0800, Jason Wang wrote: > > > Hi: > > > > > > This series tries to implement basic tx batched processing. This is > > > done by prefetching descriptor indices and update used ring in a > > > batch. This intends to speed up used ring updating and improve the > > > cache utilization. > > Interesting, thanks for the patches. So IIUC most of the gain is really > > overcoming some of the shortcomings of virtio 1.0 wrt cache utilization? > > Yes. > > Actually, looks like batching in 1.1 is not as easy as in 1.0. > > In 1.0, we could do something like: > > batch update used ring by user copy_to_user() > smp_wmb() > update used_idx > In 1.1, we need more memory barriers, can't benefit from fast copy helpers? > > for () { >     update desc.addr >     smp_wmb() >     update desc.flag > } Yes but smp_wmb is a NOP on e.g. x86. We can switch to other types of barriers as well. We do need to do the updates in order, so we might need new APIs for that to avoid re-doing the translation all the time. In 1.0 the last update is a cache miss always. You need batching to get less misses. In 1.1 you don't have it so fundamentally there is less need for batching. But batching does not always work. DPDK guys (which batch things aggressively) already tried 1.1 and saw performance gains so we do not need to argue theoretically. > > > > Which is fair enough (1.0 is already deployed) but I would like to avoid > > making 1.1 support harder, and this patchset does this unfortunately, > > I think the new APIs do not expose more internal data structure of virtio > than before? (vq->heads has already been used by vhost_net for years). For sure we might need to change vring_used_elem. > Consider the layout is re-designed completely, I don't see an easy method to > reuse current 1.0 API for 1.1. Current API just says you get buffers then you use them. It is not tied to actual separate used ring. > > see comments on individual patches. I'm sure it can be addressed though. > > > > > Test shows about ~22% improvement in tx pss. > > Is this with or without tx napi in guest? > > MoonGen is used in guest for better numbers. > > Thanks Not sure I understand. Did you set napi_tx to true or false? > > > > > Please review. > > > > > > Jason Wang (5): > > > vhost: split out ring head fetching logic > > > vhost: introduce helper to prefetch desc index > > > vhost: introduce vhost_add_used_idx() > > > vhost_net: rename VHOST_RX_BATCH to VHOST_NET_BATCH > > > vhost_net: basic tx virtqueue batched processing > > > > > > drivers/vhost/net.c | 221 ++++++++++++++++++++++++++++---------------------- > > > drivers/vhost/vhost.c | 165 +++++++++++++++++++++++++++++++------ > > > drivers/vhost/vhost.h | 9 ++ > > > 3 files changed, 270 insertions(+), 125 deletions(-) > > > > > > -- > > > 2.7.4