From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Michael S. Tsirkin" Subject: Re: [PATCH RFC 1/2] virtio-net: bql support Date: Sun, 6 Jan 2019 22:17:37 -0500 Message-ID: <20190106221506-mutt-send-email-mst@kernel.org> References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101528-mutt-send-email-mst@kernel.org> <0fa99d9b-e510-d7eb-db1b-831bd7610ce9@redhat.com> <20181230134106-mutt-send-email-mst@kernel.org> <20190102085457-mutt-send-email-mst@kernel.org> <17d2ab21-1c9a-2bb9-166f-2863d019cb0b@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Return-path: Content-Disposition: inline In-Reply-To: <17d2ab21-1c9a-2bb9-166f-2863d019cb0b@redhat.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: virtualization-bounces@lists.linux-foundation.org Errors-To: virtualization-bounces@lists.linux-foundation.org To: Jason Wang Cc: netdev@vger.kernel.org, linux-kernel@vger.kernel.org, virtualization@lists.linux-foundation.org, maxime.coquelin@redhat.com, wexu@redhat.com, "David S. Miller" List-Id: virtualization@lists.linuxfoundation.org T24gTW9uLCBKYW4gMDcsIDIwMTkgYXQgMTA6MTQ6MzdBTSArMDgwMCwgSmFzb24gV2FuZyB3cm90 ZToKPiAKPiBPbiAyMDE5LzEvMiDkuIvljYg5OjU5LCBNaWNoYWVsIFMuIFRzaXJraW4gd3JvdGU6 Cj4gPiBPbiBXZWQsIEphbiAwMiwgMjAxOSBhdCAxMToyODo0M0FNICswODAwLCBKYXNvbiBXYW5n IHdyb3RlOgo+ID4gPiBPbiAyMDE4LzEyLzMxIOS4iuWNiDI6NDUsIE1pY2hhZWwgUy4gVHNpcmtp biB3cm90ZToKPiA+ID4gPiBPbiBUaHUsIERlYyAyNywgMjAxOCBhdCAwNjowMDozNlBNICswODAw LCBKYXNvbiBXYW5nIHdyb3RlOgo+ID4gPiA+ID4gT24gMjAxOC8xMi8yNiDkuIvljYgxMToxOSwg TWljaGFlbCBTLiBUc2lya2luIHdyb3RlOgo+ID4gPiA+ID4gPiBPbiBUaHUsIERlYyAwNiwgMjAx OCBhdCAwNDoxNzozNlBNICswODAwLCBKYXNvbiBXYW5nIHdyb3RlOgo+ID4gPiA+ID4gPiA+IE9u IDIwMTgvMTIvNiDkuIrljYg2OjU0LCBNaWNoYWVsIFMuIFRzaXJraW4gd3JvdGU6Cj4gPiA+ID4g PiA+ID4gPiBXaGVuIHVzZV9uYXBpIGlzIHNldCwgbGV0J3MgZW5hYmxlIEJRTHMuICBOb3RlOiBz b21lIG9mIHRoZSBpc3N1ZXMgYXJlCj4gPiA+ID4gPiA+ID4gPiBzaW1pbGFyIHRvIHdpZmkuICBJ dCdzIHdvcnRoIGNvbnNpZGVyaW5nIHdoZXRoZXIgc29tZXRoaW5nIHNpbWlsYXIgdG8KPiA+ID4g PiA+ID4gPiA+IGNvbW1pdCAzNjE0OGMyYmJmYmUgKCJtYWM4MDIxMTogQWRqdXN0IFRTUSBwYWNp bmcgc2hpZnQiKSBtaWdodCBiZQo+ID4gPiA+ID4gPiA+ID4gYmVuZWZpdGlhbC4KPiA+ID4gPiA+ ID4gPiBJJ3ZlIHBsYXllZCBhIHNpbWlsYXIgcGF0Y2ggc2V2ZXJhbCBkYXlzIGJlZm9yZS4gVGhl IHRyaWNreSBwYXJ0IGlzIHRoZSBtb2RlCj4gPiA+ID4gPiA+ID4gc3dpdGNoaW5nIGJldHdlZW4g bmFwaSBhbmQgbm8gbmFwaS4gV2Ugc2hvdWxkIG1ha2Ugc3VyZSB3aGVuIHRoZSBwYWNrZXQgaXMK PiA+ID4gPiA+ID4gPiBzZW50IGFuZCB0cmFrY2VkIGJ5IEJRTCzCoCBpdCBzaG91bGQgYmUgY29u c3VtZWQgYnkgQlFMIGFzIHdlbGwuIEkgZGlkIGl0IGJ5Cj4gPiA+ID4gPiA+ID4gdHJhY2tpbmcg aXQgdGhyb3VnaCBza2ItPmNiLsKgIEFuZCBkZWFsIHdpdGggdGhlIGZyZWV6ZSBieSByZXNldCB0 aGUgQlFMCj4gPiA+ID4gPiA+ID4gc3RhdHVzLiBQYXRjaCBhdHRhY2hlZC4KPiA+ID4gPiA+ID4g PiAKPiA+ID4gPiA+ID4gPiBCdXQgd2hlbiB0ZXN0aW5nIHdpdGggdmhvc3QtbmV0LCBJIGRvbid0 IHZlcnkgYSBzdGFibGUgcGVyZm9ybWFuY2UsCj4gPiA+ID4gPiA+IFNvIGhvdyBhYm91dCBpbmNy ZWFzaW5nIFRTUSBwYWNpbmcgc2hpZnQgdGhlbj8KPiA+ID4gPiA+IEkgY2FuIHRlc3QgdGhpcy4g QnV0IGNoYW5naW5nIGRlZmF1bHQgVENQIHZhbHVlIGlzIG11Y2ggbW9yZSB0aGFuIGEKPiA+ID4g PiA+IHZpcnRpby1uZXQgc3BlY2lmaWMgdGhpbmcuCj4gPiA+ID4gV2VsbCBzYW1lIGxvZ2ljIGFz IHdpZmkgYXBwbGllcy4gVW5wcmVkaWN0YWJsZSBsYXRlbmNpZXMgcmVsYXRlZAo+ID4gPiA+IHRv IHJhZGlvIGluIG9uZSBjYXNlLCB0byBob3N0IHNjaGVkdWxlciBpbiB0aGUgb3RoZXIuCj4gPiA+ ID4gCj4gPiA+ID4gPiA+ID4gaXQgd2FzCj4gPiA+ID4gPiA+ID4gcHJvYmFibHkgYmVjYXVzZSB3 ZSBiYXRjaCB0aGUgdXNlZCByaW5nIHVwZGF0aW5nIHNvIHR4IGludGVycnVwdCBtYXkgY29tZQo+ ID4gPiA+ID4gPiA+IHJhbmRvbWx5LiBXZSBwcm9iYWJseSBuZWVkIHRvIGltcGxlbWVudCB0aW1l IGJvdW5kZWQgY29hbGVzY2luZyBtZWNoYW5pc20KPiA+ID4gPiA+ID4gPiB3aGljaCBjb3VsZCBi ZSBjb25maWd1cmVkIGZyb20gdXNlcnNwYWNlLgo+ID4gPiA+ID4gPiBJIGRvbid0IHRoaW5rIGl0 J3MgcmVhc29uYWJsZSB0byBleHBlY3QgdXNlcnNwYWNlIHRvIGJlIHRoYXQgc21hcnQgLi4uCj4g PiA+ID4gPiA+IFdoeSBkbyB3ZSBuZWVkIHRpbWUgYm91bmRlZD8gdXNlZCByaW5nIGlzIGFsd2F5 cyB1cGRhdGVkIHdoZW4gcmluZwo+ID4gPiA+ID4gPiBiZWNvbWVzIGVtcHR5Lgo+ID4gPiA+ID4g V2UgZG9uJ3QgYWRkIHVzZWQgd2hlbiBtZWFucyBCUUwgbWF5IG5vdCBzZWUgdGhlIGNvbnN1bWVk IHBhY2tldCBpbiB0aW1lLgo+ID4gPiA+ID4gQW5kIHRoZSBkZWxheSB2YXJpZXMgYmFzZWQgb24g dGhlIHdvcmtsb2FkIHNpbmNlIHdlIGNvdW50IHBhY2tldHMgbm90IGJ5dGVzCj4gPiA+ID4gPiBv ciB0aW1lIGJlZm9yZSBkb2luZyB0aGUgYmF0Y2hlZCB1cGRhdGluZy4KPiA+ID4gPiA+IAo+ID4g PiA+ID4gVGhhbmtzCj4gPiA+ID4gU29ycnkgSSBzdGlsbCBkb24ndCBnZXQgaXQuCj4gPiA+ID4g V2hlbiBub3RoaW5nIGlzIG91dHN0YW5kaW5nIHRoZW4gd2UgZG8gdXBkYXRlIHRoZSB1c2VkLgo+ ID4gPiA+IFNvIGlmIEJRTCBzdG9wcyB1c2Vyc3BhY2UgZnJvbSBzZW5kaW5nIHBhY2tldHMgdGhl bgo+ID4gPiA+IHdlIGdldCBhbiBpbnRlcnJ1cHQgYW5kIHBhY2tldHMgc3RhcnQgZmxvd2luZyBh Z2Fpbi4KPiA+ID4gWWVzLCBidXQgaG93IGFib3V0IHRoZSBjYXNlcyBvZiBtdWx0aXBsZSBmbG93 cy4gVGhhdCdzIHdoZXJlIEkgc2VlIHVuc3RhYmxlCj4gPiA+IHJlc3VsdHMuCj4gPiA+IAo+ID4g PiAKPiA+ID4gPiBJdCBtaWdodCBiZSBzdWJvcHRpbWFsLCB3ZSBtaWdodCBuZWVkIHRvIHR1bmUg aXQgYnV0IEkgZG91YnQgcnVubmluZwo+ID4gPiA+IHRpbWVycyBpcyBhIHNvbHV0aW9uLCB0aW1l ciBpbnRlcnJ1cHRzIGNhdXNlIFZNIGV4aXRzLgo+ID4gPiBQcm9iYWJseSBub3QgYSB0aW1lciBi dXQgYSB0aW1lIGNvdW50ZXIgKG9yIGV2ZW50IGJ5dGUgY291bnRlcikgaW4gdmhvc3QgdG8KPiA+ ID4gYWRkIHVzZWQgYW5kIHNpZ25hbCBndWVzdCBpZiBpdCBleGNlZWRzIGEgdmFsdWUgaW5zdGVh ZCBvZiB3YWl0aW5nIHRoZQo+ID4gPiBudW1iZXIgb2YgcGFja2V0cy4KPiA+ID4gCj4gPiA+IAo+ ID4gPiBUaGFua3MKPiA+IFdlbGwgd2UgYWxyZWFkeSBoYXZlIFZIT1NUX05FVF9XRUlHSFQgLSBp cyBpdCB0b28gYmlnIHRoZW4/Cj4gCj4gCj4gSSdtIG5vdCBzdXJlLCBpdCBtaWdodCBiZSB0b28g YmlnLgo+IAo+IAo+ID4gCj4gPiBBbmQgbWF5YmUgd2Ugc2hvdWxkIGV4cG9zZSB0aGUgIk1PUkUi IGZsYWcgaW4gdGhlIGRlc2NyaXB0b3IgLQo+ID4gZG8geW91IHRoaW5rIHRoYXQgd2lsbCBoZWxw Pwo+ID4gCj4gCj4gSSBkb24ndCBrbm93LiBCdXQgaG93IGEgIm1vcmUiIGZsYWcgY2FuIGhlbHAg aGVyZT8KPiAKPiBUaGFua3MKCkl0IHNvdW5kcyBsaWtlIHdlIHNob3VsZCBiZSBhIGJpdCBtb3Jl IGFnZ3Jlc3NpdmUgaW4gdXBkYXRpbmcgdXNlZCByaW5nLgpCdXQgaWYgd2UganVzdCBkbyBpdCBu YWl2ZWx5IHdlIHdpbGwgaGFybSBwZXJmb3JtYW5jZSBmb3Igc3VyZSBhcyB0aGF0CmlzIGhvdyB3 ZSBhcmUgZG9pbmcgYmF0Y2hpbmcgcmlnaHQgbm93LiBJbnN0ZWFkIHdlIGNvdWxkIG1ha2UgZ3Vl c3QKY29udHJvbCBiYXRjaGluZyB1c2luZyB0aGUgbW9yZSBmbGFnIC0gaWYgdGhhdCdzIG5vdCBz ZXQgd2Ugd3JpdGUgb3V0CnRoZSB1c2VkIHJpbmcuCgo+IAo+ID4gCl9fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fClZpcnR1YWxpemF0aW9uIG1haWxpbmcgbGlz dApWaXJ0dWFsaXphdGlvbkBsaXN0cy5saW51eC1mb3VuZGF0aW9uLm9yZwpodHRwczovL2xpc3Rz LmxpbnV4Zm91bmRhdGlvbi5vcmcvbWFpbG1hbi9saXN0aW5mby92aXJ0dWFsaXphdGlvbg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60BAAC43387 for ; Mon, 7 Jan 2019 03:17:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2F73820859 for ; Mon, 7 Jan 2019 03:17:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726417AbfAGDRk (ORCPT ); Sun, 6 Jan 2019 22:17:40 -0500 Received: from mx1.redhat.com ([209.132.183.28]:37470 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726278AbfAGDRj (ORCPT ); Sun, 6 Jan 2019 22:17:39 -0500 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 51F2A89AD5; Mon, 7 Jan 2019 03:17:39 +0000 (UTC) Received: from redhat.com (ovpn-120-33.rdu2.redhat.com [10.10.120.33]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7E67A60C45; Mon, 7 Jan 2019 03:17:37 +0000 (UTC) Date: Sun, 6 Jan 2019 22:17:37 -0500 From: "Michael S. Tsirkin" To: Jason Wang Cc: linux-kernel@vger.kernel.org, maxime.coquelin@redhat.com, tiwei.bie@intel.com, wexu@redhat.com, jfreimann@redhat.com, "David S. Miller" , virtualization@lists.linux-foundation.org, netdev@vger.kernel.org Subject: Re: [PATCH RFC 1/2] virtio-net: bql support Message-ID: <20190106221506-mutt-send-email-mst@kernel.org> References: <20181205225323.12555-1-mst@redhat.com> <20181205225323.12555-2-mst@redhat.com> <21384cb5-99a6-7431-1039-b356521e1bc3@redhat.com> <20181226101528-mutt-send-email-mst@kernel.org> <0fa99d9b-e510-d7eb-db1b-831bd7610ce9@redhat.com> <20181230134106-mutt-send-email-mst@kernel.org> <20190102085457-mutt-send-email-mst@kernel.org> <17d2ab21-1c9a-2bb9-166f-2863d019cb0b@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <17d2ab21-1c9a-2bb9-166f-2863d019cb0b@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.26]); Mon, 07 Jan 2019 03:17:39 +0000 (UTC) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Jan 07, 2019 at 10:14:37AM +0800, Jason Wang wrote: > > On 2019/1/2 下午9:59, Michael S. Tsirkin wrote: > > On Wed, Jan 02, 2019 at 11:28:43AM +0800, Jason Wang wrote: > > > On 2018/12/31 上午2:45, Michael S. Tsirkin wrote: > > > > On Thu, Dec 27, 2018 at 06:00:36PM +0800, Jason Wang wrote: > > > > > On 2018/12/26 下午11:19, Michael S. Tsirkin wrote: > > > > > > On Thu, Dec 06, 2018 at 04:17:36PM +0800, Jason Wang wrote: > > > > > > > On 2018/12/6 上午6:54, Michael S. Tsirkin wrote: > > > > > > > > When use_napi is set, let's enable BQLs. Note: some of the issues are > > > > > > > > similar to wifi. It's worth considering whether something similar to > > > > > > > > commit 36148c2bbfbe ("mac80211: Adjust TSQ pacing shift") might be > > > > > > > > benefitial. > > > > > > > I've played a similar patch several days before. The tricky part is the mode > > > > > > > switching between napi and no napi. We should make sure when the packet is > > > > > > > sent and trakced by BQL,  it should be consumed by BQL as well. I did it by > > > > > > > tracking it through skb->cb.  And deal with the freeze by reset the BQL > > > > > > > status. Patch attached. > > > > > > > > > > > > > > But when testing with vhost-net, I don't very a stable performance, > > > > > > So how about increasing TSQ pacing shift then? > > > > > I can test this. But changing default TCP value is much more than a > > > > > virtio-net specific thing. > > > > Well same logic as wifi applies. Unpredictable latencies related > > > > to radio in one case, to host scheduler in the other. > > > > > > > > > > > it was > > > > > > > probably because we batch the used ring updating so tx interrupt may come > > > > > > > randomly. We probably need to implement time bounded coalescing mechanism > > > > > > > which could be configured from userspace. > > > > > > I don't think it's reasonable to expect userspace to be that smart ... > > > > > > Why do we need time bounded? used ring is always updated when ring > > > > > > becomes empty. > > > > > We don't add used when means BQL may not see the consumed packet in time. > > > > > And the delay varies based on the workload since we count packets not bytes > > > > > or time before doing the batched updating. > > > > > > > > > > Thanks > > > > Sorry I still don't get it. > > > > When nothing is outstanding then we do update the used. > > > > So if BQL stops userspace from sending packets then > > > > we get an interrupt and packets start flowing again. > > > Yes, but how about the cases of multiple flows. That's where I see unstable > > > results. > > > > > > > > > > It might be suboptimal, we might need to tune it but I doubt running > > > > timers is a solution, timer interrupts cause VM exits. > > > Probably not a timer but a time counter (or event byte counter) in vhost to > > > add used and signal guest if it exceeds a value instead of waiting the > > > number of packets. > > > > > > > > > Thanks > > Well we already have VHOST_NET_WEIGHT - is it too big then? > > > I'm not sure, it might be too big. > > > > > > And maybe we should expose the "MORE" flag in the descriptor - > > do you think that will help? > > > > I don't know. But how a "more" flag can help here? > > Thanks It sounds like we should be a bit more aggressive in updating used ring. But if we just do it naively we will harm performance for sure as that is how we are doing batching right now. Instead we could make guest control batching using the more flag - if that's not set we write out the used ring. > > >