From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mail.toke.dk ([2001:470:dc45:1000::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1gNLNd-0006yF-VC for ath10k@lists.infradead.org; Thu, 15 Nov 2018 17:24:40 +0000 From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= Subject: Re: [PATCH v3 3/6] mac80211: Add airtime accounting and scheduling to TXQs In-Reply-To: <8e7847ff-4c88-10ae-2223-2fc7321641d9@nbd.name> References: <1542063113-22438-1-git-send-email-rmanohar@codeaurora.org> <1542063113-22438-4-git-send-email-rmanohar@codeaurora.org> <871s7nv9pl.fsf@toke.dk> <8e7847ff-4c88-10ae-2223-2fc7321641d9@nbd.name> Date: Thu, 15 Nov 2018 09:24:22 -0800 Message-ID: <87sh02tfsp.fsf@toke.dk> MIME-Version: 1.0 List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 Sender: "ath10k" Errors-To: ath10k-bounces+kvalo=adurom.com@lists.infradead.org To: Felix Fietkau , Rajkumar Manoharan , linux-wireless@vger.kernel.org, ath10k@lists.infradead.org Cc: make-wifi-fast@lists.bufferbloat.net RmVsaXggRmlldGthdSA8bmJkQG5iZC5uYW1lPiB3cml0ZXM6Cgo+IE9uIDIwMTgtMTEtMTQgMTg6 NDAsIFRva2UgSMO4aWxhbmQtSsO4cmdlbnNlbiB3cm90ZToKPj4+IFRoaXMgcGFydCBkb2Vzbid0 IHJlYWxseSBtYWtlIG11Y2ggc2Vuc2UgdG8gbWUsIGJ1dCBtYXliZSBJJ20KPj4+IG1pc3VuZGVy c3RhbmRpbmcgaG93IHRoZSBjb2RlIHdvcmtzLgo+Pj4gTGV0J3MgYXNzdW1lIHdlIGhhdmUgYSBk cml2ZXIgbGlrZSBhdGg5ayBvciBtdDc2LCB3aGljaCB0cmllcyB0byBrZWVwIGEKPj4+IG51bWJl ciBvZiBhZ2dyZWdhdGVzIGluIHRoZSBoYXJkd2FyZSBxdWV1ZSwgYW5kIHRoZSBoYXJkd2FyZSBx dWV1ZSBpcwo+Pj4gY3VycmVudGx5IGVtcHR5Lgo+Pj4gSWYgdGhlIGN1cnJlbnQgdHhxIGVudHJ5 IGlzIGtlcHQgYXQgdGhlIGhlYWQgb2YgdGhlIHNjaGVkdWxlIGxpc3QsCj4+PiB3b3VsZG4ndCB0 aGUgY29kZSBqdXN0IHB1bGwgZnJvbSB0aGF0IG9uZSBvdmVyIGFuZCBvdmVyIGFnYWluLCB1bnRp bAo+Pj4gZW5vdWdoIHBhY2tldHMgYXJlIHRyYW5zbWl0dGVkIGJ5IHRoZSBoYXJkd2FyZSBhbmQg dGhlaXIgdHggc3RhdHVzCj4+PiBwcm9jZXNzZWQ/Cj4+PiBJdCBzZWVtcyB0byBtZSB0aGF0IHdo aWxlIGZhaXJuZXNzIGlzIHN0aWxsIHByZXNlcnZlZCBpbiB0aGUgbG9uZyBydW4sCj4+PiB0aGlz IGNvdWxkIGxlYWQgdG8gcmF0aGVyIGJ1cnN0eSBzY2hlZHVsaW5nLCB3aGljaCBtYXkgbm90IGJl Cj4+PiBwYXJ0aWN1bGFybHkgbGF0ZW5jeSBmcmllbmRseS4KPj4gCj4+IFllcywgaXQnbGwgYmUg YSBiaXQgbW9yZSBidXJzdHkgd2hlbiB0aGUgaGFyZHdhcmUgcXVldWUgaXMgY29tcGxldGVseQo+ PiBlbXB0eS4gSG93ZXZlciwgd2hlbiBhIFRYIGNvbXBsZXRpb24gY29tZXMgYmFjaywgdGhhdCB3 aWxsIGFkanVzdCB0aGUKPj4gZGVmaWNpdCBvZiB0aGF0IHN0YSBhbmQgY2F1c2UgaXQgdG8gYmUg cm90YXRlZCBvbiB0aGUgbmV4dCBkZXF1ZXVlLiBUaGlzCj4+IG9idmlvdXNseSByZWxpZXMgb24g dGhlIGZhY3QgdGhhdCB0aGUgbG93ZXItbGV2ZWwgaGFyZHdhcmUgcXVldWUgaXMKPj4gc3VmZmlj aWVudGx5IHNoYWxsb3cgdG8gbm90IGFkZCBhIGxvdCBvZiBsYXRlbmN5LiBCdXQgd2Ugd2FudCB0 aGF0IHRvIGJlCj4+IHRoZSBjYXNlIGFueXdheS4gSW4gcHJhY3RpY2UsIGl0IHdvcmtzIHF1aXRl IHdlbGwgZm9yIGF0aDlrLCBidXQgbm90IHNvCj4+IHdlbGwgZm9yIGF0aDEwayBiZWNhdXNlIGl0 IGhhcyBhIGxhcmdlIGJ1ZmZlciBpbiBmaXJtd2FyZS4KPj4gCj4+IElmIHdlIHJlcXVldWUgdGhl IFRYUSBhdCB0aGUgZW5kIG9mIHRoZSBsaXN0LCBhIHN0YXRpb24gdGhhdCBpcyB0YWtpbmcKPj4g dXAgdG9vIG11Y2ggYWlydGltZSB3aWxsIGZhaWwgdG8gYmUgdGhyb3R0bGVkIHByb3Blcmx5LCBz byB0aGUKPj4gcXVldWUtYXQtaGVhZCBpcyBraW5kYSBuZWVkZWQgdG8gZW5zdXJlIGZhaXJuZXNz Li4uCj4gVGhhbmtzIGZvciB0aGUgZXhwbGFuYXRpb24sIHRoYXQgbWFrZXMgc2Vuc2UgdG8gbWUu IEkgaGF2ZSBhbiBpZGVhIG9uCj4gaG93IHRvIG1pdGlnYXRlIHRoZSBidXJzdGluZXNzIHdpdGhp biB0aGUgZHJpdmVyLiBJJ2xsIHdyaXRlIGl0IGRvd24gaW4KPiBwc2V1ZG9jb2RlLCBwbGVhc2Ug bGV0IG1lIGtub3cgaWYgeW91IHRoaW5rIHRoYXQnbGwgd29yay4KCkkgZG9uJ3QgdGhpbmsgaXQg d2lsbCwgdW5mb3J0dW5hdGVseS4gRm9yIGV4YW1wbGUsIGNvbnNpZGVyIHRoZSBjYXNlCndoZXJl IHRoZXJlIGFyZSB0d28gc3RhdGlvbnMgcXVldWVkOyBvbmUgd2l0aCBhIGxhcmdlIG5lZ2F0aXZl IGRlZmljaXQKKHNheSwgLTEwbXMpLCBhbmQgb25lIHdpdGggYSBwb3NpdGl2ZSBkZWZpY2l0LgoK SW4gdGhpcyBjYXNlLCB3ZSByZWFsbHkgbmVlZCB0byB0aHJvdHRsZSB0aGUgc3RhdGlvbiB3aXRo IGEgbmVnYXRpdmUKZGVmaWNpdC4gQnV0IGlmIHRoZSBkcml2ZXIgbG9vcHMgYW5kIGNhY2hlcyB0 eHFzLCB3ZSdsbCBnZXQgc29tZXRoaW5nCmxpa2UgdGhlIGZvbGxvd2luZzoKCi0gRmlyc3QgZHJp dmVyIGxvb3AgaXRlcmF0aW9uOiByZXR1cm5zIFRYUSB3aXRoIHBvc2l0aXZlIGRlZmljaXQuCi0g U2Vjb25kIGRyaXZlciBsb29wIGl0ZXJhdGlvbjogT25seSB0aGUgbmVnYXRpdmUtZGVmaWNpdCBU WFEgaXMgaW4gdGhlCiAgbWFjODAyMTEgbGlzdCwgc28gaXQgd2lsbCBsb29wIHVudGlsIHRoYXQg VFhRJ3MgZGVmaWNpdCB0dXJucyBwb3NpdGl2ZQogIGFuZCByZXR1cm4gaXQuCgpCZWNhdXNlIG9m IHRoaXMsIHRoZSBuZWdhdGl2ZS1kZWZpY2l0IHN0YXRpb24gd29uJ3QgYmUgdGhyb3R0bGVkLCBh bmQgd2UKd29uJ3QgZ2V0IGZhaXJuZXNzLgoKSG93IG1hbnkgZnJhbWVzIHdpbGwgbXQ3NiBxdWV1 ZSB1cCBiZWxvdyB0aGUgZHJpdmVyIHBvaW50PyBJLmUuLCBob3cKbXVjaCBidXJzdGluZXNzIGFy ZSB5b3UgZXhwZWN0aW5nIHRoaXMgd2lsbCBpbnRyb2R1Y2Ugb24gdGhhdCBkcml2ZXI/CgpUYWtp bmcgYSBzdGVwIGJhY2ssIGl0J3MgY2xlYXIgdGhhdCBpdCB3b3VsZCBiZSBnb29kIHRvIGJlIGFi bGUgdG8KZGVxdWV1ZSBwYWNrZXRzIHRvIG11bHRpcGxlIFNUQXMgYXQgb25jZSAod2UgbmVlZCB0 aGF0IGZvciBNVS1NSU1PIG9uCmF0aDEwayBhcyB3ZWxsKS4gSG93ZXZlciwgSSBkb24ndCB0aGlu ayB3ZSBjYW4gZG8gdGhhdCB3aXRoIHRoZQpyb3VuZC1yb2JpbiBmYWlybmVzcyBzY2hlZHVsZXI7 IHNvIHdlIGFyZSBnb2luZyB0byBuZWVkIGEgZGlmZmVyZW50CmFsZ29yaXRobS4gSSAqdGhpbmsq IGl0IG1heSBiZSBwb3NzaWJsZSB0byBkbyB0aGlzIHdpdGggYSB2aXJ0dWFsLXRpbWUKc2NoZWR1 bGVyLCBidXQgSSBoYXZlbid0IHNhdCBkb3duIGFuZCB3b3JrZWQgb3V0IHRoZSBkZXRhaWxzIHll dC4uLgoKLVRva2UKCl9fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fCmF0aDEwayBtYWlsaW5nIGxpc3QKYXRoMTBrQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDov L2xpc3RzLmluZnJhZGVhZC5vcmcvbWFpbG1hbi9saXN0aW5mby9hdGgxMGsK From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 943CEC43441 for ; Thu, 15 Nov 2018 17:24:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5696A21582 for ; Thu, 15 Nov 2018 17:24:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=toke.dk header.i=@toke.dk header.b="WLMS9fgB" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5696A21582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=toke.dk Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-wireless-owner@vger.kernel.org Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388351AbeKPDdK (ORCPT ); Thu, 15 Nov 2018 22:33:10 -0500 Received: from mail.toke.dk ([52.28.52.200]:44483 "EHLO mail.toke.dk" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388265AbeKPDdJ (ORCPT ); Thu, 15 Nov 2018 22:33:09 -0500 From: Toke =?utf-8?Q?H=C3=B8iland-J=C3=B8rgensen?= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=toke.dk; s=20161023; t=1542302665; bh=+61GhQNjnU8wdyMTs/FDeoj8smhkEy+LIIT7JyOcQ8I=; h=From:To:Cc:Subject:In-Reply-To:References:Date:From; b=WLMS9fgBj7/v8CdJxutgr4scinP9Us1dkY6uB3ClTzI6CLvv5qyQZnJBC+2LF5HgK ux+LLl6G6Y6qqNjuUOGnZBFGpZaCH7300v/U6mIP1/m1bwFszCL+MWiDFWcDQ7qz/t 6t0dYkbjYJ+zu1thTtxucSt4bn+Qmoq6TBrewo58TAVGZ+pI0tOs0YO55SonK9Zogs KbxiUq8Adg3uuVf+VlNNfp4/YBF5oX/wPLg8ytZkIJIIxhdLu/FlK2955OxZKY8xMm 1LWZoBCGOivT2i+KhcMT7Gx6ORIWDcuk2afSdvrNT7l5qYZyXMZtwyRzQMLDl+z0EC yfNDQQlr1KXtg== To: Felix Fietkau , Rajkumar Manoharan , linux-wireless@vger.kernel.org, ath10k@lists.infradead.org Cc: make-wifi-fast@lists.bufferbloat.net Subject: Re: [PATCH v3 3/6] mac80211: Add airtime accounting and scheduling to TXQs In-Reply-To: <8e7847ff-4c88-10ae-2223-2fc7321641d9@nbd.name> References: <1542063113-22438-1-git-send-email-rmanohar@codeaurora.org> <1542063113-22438-4-git-send-email-rmanohar@codeaurora.org> <871s7nv9pl.fsf@toke.dk> <8e7847ff-4c88-10ae-2223-2fc7321641d9@nbd.name> Date: Thu, 15 Nov 2018 09:24:22 -0800 X-Clacks-Overhead: GNU Terry Pratchett Message-ID: <87sh02tfsp.fsf@toke.dk> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Sender: linux-wireless-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-wireless@vger.kernel.org Felix Fietkau writes: > On 2018-11-14 18:40, Toke H=C3=B8iland-J=C3=B8rgensen wrote: >>> This part doesn't really make much sense to me, but maybe I'm >>> misunderstanding how the code works. >>> Let's assume we have a driver like ath9k or mt76, which tries to keep a >>> number of aggregates in the hardware queue, and the hardware queue is >>> currently empty. >>> If the current txq entry is kept at the head of the schedule list, >>> wouldn't the code just pull from that one over and over again, until >>> enough packets are transmitted by the hardware and their tx status >>> processed? >>> It seems to me that while fairness is still preserved in the long run, >>> this could lead to rather bursty scheduling, which may not be >>> particularly latency friendly. >>=20 >> Yes, it'll be a bit more bursty when the hardware queue is completely >> empty. However, when a TX completion comes back, that will adjust the >> deficit of that sta and cause it to be rotated on the next dequeue. This >> obviously relies on the fact that the lower-level hardware queue is >> sufficiently shallow to not add a lot of latency. But we want that to be >> the case anyway. In practice, it works quite well for ath9k, but not so >> well for ath10k because it has a large buffer in firmware. >>=20 >> If we requeue the TXQ at the end of the list, a station that is taking >> up too much airtime will fail to be throttled properly, so the >> queue-at-head is kinda needed to ensure fairness... > Thanks for the explanation, that makes sense to me. I have an idea on > how to mitigate the burstiness within the driver. I'll write it down in > pseudocode, please let me know if you think that'll work. I don't think it will, unfortunately. For example, consider the case where there are two stations queued; one with a large negative deficit (say, -10ms), and one with a positive deficit. In this case, we really need to throttle the station with a negative deficit. But if the driver loops and caches txqs, we'll get something like the following: - First driver loop iteration: returns TXQ with positive deficit. - Second driver loop iteration: Only the negative-deficit TXQ is in the mac80211 list, so it will loop until that TXQ's deficit turns positive and return it. Because of this, the negative-deficit station won't be throttled, and we won't get fairness. How many frames will mt76 queue up below the driver point? I.e., how much burstiness are you expecting this will introduce on that driver? Taking a step back, it's clear that it would be good to be able to dequeue packets to multiple STAs at once (we need that for MU-MIMO on ath10k as well). However, I don't think we can do that with the round-robin fairness scheduler; so we are going to need a different algorithm. I *think* it may be possible to do this with a virtual-time scheduler, but I haven't sat down and worked out the details yet... -Toke