From mboxrd@z Thu Jan 1 00:00:00 1970 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Subject: Aw: Re: dvb usb issues since kernel 4.9 From: Jesper Dangaard Brouer Message-Id: <20180110104514.4cd0e94a@redhat.com> Date: Wed, 10 Jan 2018 10:45:14 +0100 To: Linus Torvalds Cc: Eric Dumazet , Josef Griebichler , Peter Zijlstra , Mauro Carvalho Chehab , Alan Stern , Greg Kroah-Hartman , USB list , Rik van Riel , Paolo Abeni , Hannes Frederic Sowa , linux-kernel , netdev , Jonathan Corbet , LMML , David Miller , Marek Majkowski List-ID: T24gVHVlLCA5IEphbiAyMDE4IDEwOjU4OjMwIC0wODAwIExpbnVzIFRvcnZhbGRzIDx0b3J2YWxk c0BsaW51eC1mb3VuZGF0aW9uLm9yZz4gd3JvdGU6Cgo+IFNvIEkgcmVhbGx5IHRoaW5rICJ5b3Ug Y2FuIHVzZSB1cCA5MCUgb2YgQ1BVIHRpbWUgd2l0aCBhIFVEUCBwYWNrZXQKPiBmbG9vZCBmcm9t IHRoZSBzYW1lIG5ldHdvcmsiIGlzIHZlcnkgdmVyeSB2ZXJ5IGRpZmZlcmVudCAtIGFuZAo+IGhv bmVzdGx5IG5vdCBhdCBhbGwgYXMgaW1wb3J0YW50IC0gYXMgInlvdSB3YW50IHRvIGJlIGFibGUg dG8gdXNlIGEKPiBVU0IgRFZCIHJlY2VpdmVyIGFuZCB3YXRjaC9yZWNvcmQgVFYiLgo+IAo+IEJl Y2F1c2UgdGhhdCB3aG9sZSAiVURQIHBhY2tldCBmbG9vZCBmcm9tIHRoZSBzYW1lIG5ldHdvcmsi IHJlYWxseSBpcwo+IHNvbWV0aGluZyB5b3UgX2Z1bmRhbWVudGFsbHlfIGhhdmUgb3RoZXIgbWl0 aWdhdGlvbnMgZm9yLgo+IAo+IEkgYmV0IHRoYXQgd2hvbGUgY29tbWl0IHdhcyBpbnRyb2R1Y2Vk IGJlY2F1c2Ugb2YgYSBiZW5jaG1hcmsgdGVzdCwKPiByYXRoZXIgdGhhbiByZWFsIGxpZmUuIE5v PwoKSSBiZWxpZXZlIHRoaXMgaGF2ZSBoYXBwZW5lZCBpbiByZWFsLWxpZmUuICBJbiB0aGUgZm9y bSBvZiBETlMgc2VydmVycwpub3QgYmVpbmcgYWJsZSB0byByZWNvdmVyIGFmdGVyIGxvbmcgb3V0 YWdlLCB3aGVyZSBETlMtVFRMIGhhZCB0aW1lb3V0CmNhdXNpbmcgbGVnaXRpbWF0ZSB0cmFmZmlj IHRvIG92ZXJsb2FkIHRoZWlyIEROUyBzZXJ2ZXJzLiAgVGhlIGdvb2RwdXQKYW5zd2Vycy9zZWMg ZnJvbSB0aGVpciBETlMgc2VydmVycyB3ZXJlIHRvbyBsb3csIHdoZW4gYnJpbmdpbmcgdGhlbQpv bmxpbmUgYWdhaW4uIChCYXNlZCBvbiB0YWxrIG92ZXIgYmVlciBhdCBOZXREZXZDb25mIGZyb20g YSBndXkKY2xhaW1pbmcgdGhleSByYW4gRE5TIGZvciBBV1MpLgoKClRoZSBjb21taXQgNGNkMTNj MjFiMjA3ICgic29mdGlycTogTGV0IGtzb2Z0aXJxZCBkbyBpdHMgam9iIikgdHJpZXMgdG8KYWRk cmVzcyBhIGZ1bmRhbWVudGFsIHByb2JsZW0gdGhhdCB0aGUgbmV0d29yayBzdGFjayBoYXZlIHdo ZW4KaW50ZXJhY3Rpbmcgd2l0aCBzb2Z0aXJxIGluIG92ZXJsb2FkIHNpdHVhdGlvbnMuCihNYXli ZSB3ZSBjYW4gY29tZSB1cCB3aXRoIGEgYmV0dGVyIHNvbHV0aW9uPykKCkJlZm9yZSB0aGlzIGNv bW1pdCwgd2hlbiBhcHBsaWNhdGlvbiBydW4gb24gc2FtZSBDUFUgYXMgc29mdGlycSwgdGhlCmtl cm5lbCBoYXZlIGEgYmFkICJkcm9wIG9mZiBjbGlmZiIgYmVoYXZpb3IsIHdoZW4gcmVhY2hpbmcg YWJvdmUgdGhlCnNhdHVyYXRpb24gcG9pbnQuCgpUaGlzIGlzIGNvbmZpcm1lZCBpbiBDbG91ZEZs YXJlIGJsb2dwb3N0WzFdLCB3aGljaCB1c2VkIGEga2VybmVsIHRoYXQKcHJlZGF0ZXMgdGhpcyBj b21taXQuIEZyb21bMV0gc2VjdGlvbjogIkEgbm90ZSBvbiBOVU1BIHBlcmZvcm1hbmNlIgpRdW90 ZToiCiAgMS4gUnVuIHJlY2VpdmVyIG9uIGFub3RoZXIgQ1BVLCBidXQgb24gdGhlIHNhbWUgTlVN QSBub2RlIGFzIHRoZSBSWAogICAgIHF1ZXVlLiBUaGUgcGVyZm9ybWFuY2UgYXMgd2Ugc2F3IGFi b3ZlIGlzIGFyb3VuZCAzNjBrcHBzLgoKICAyLiBXaXRoIHJlY2VpdmVyIG9uIGV4YWN0bHkgc2Ft ZSBDUFUgYXMgdGhlIFJYIHF1ZXVlIHdlIGNhbiBnZXQgdXAgdG8KICAgICB+NDMwa3Bwcy4gQnV0 IGl0IGNyZWF0ZXMgaGlnaCB2YXJpYWJpbGl0eS4gVGhlIHBlcmZvcm1hbmNlIGRyb3BzIGRvd24K ICAgICB0byB6ZXJvIGlmIHRoZSBOSUMgaXMgb3ZlcndoZWxtZWQgd2l0aCBwYWNrZXRzLiIKClRo ZSBiZWhhdmlvciBwcm9ibGVtIGhlcmUgaXMgInBlcmZvcm1hbmNlIGRyb3BzIGRvd24gdG8gemVy byBpZiB0aGUgTklDCmlzIG92ZXJ3aGVsbWVkIHdpdGggcGFja2V0cyIuICBUaGF0IGlzIGEgYmFk IHdheSB0byBoYW5kbGUgb3ZlcmxvYWQuCk5vdCBvbmx5IHdoZW4gYXR0YWNrZWQsIGJ1dCBhbHNv IHdoZW4gYnJpbmdpbmcgYSBzZXJ2aWNlIG9ubGluZSBhZnRlcgphbiBvdXRhZ2UuCgpXaGF0IGVz c2VudGlhbGx5IGhhcHBlbnMgaXMgdGhhdDoKIDEuIHNvZnRpcnEgTkFQSSBlbnF1ZXVlIDY0IHBh Y2tldHMgaW50byBzb2NrZXQuIAogMi4gYXBwbGljYXRpb24gZGVxdWV1ZSAxIHBhY2tldCBhbmQg aW52b2tlIGxvY2FsX2JoX2VuYWJsZSgpCiAzLiBjYXVzaW5nIHNvZnRpcnEgdG8gcnVuIGluIGFw cC10aW1lc2xpY2UsIGFnYWluIGVucSA2NCBwYWNrZXRzCiA0LiBhcHAgb25seSBzZWUgZ29vZHB1 dCBvZiAxLzEyOCBvZiBwYWNrZXRzCgpUaGF0IGlzIGVzc2VudGlhbGx5IHdoYXQgRXJpYyBzb2x2 ZWQgd2l0aCBoaXMgY29tbWl0LCBhdm9pZGluZyAoMykKbG9jYWxfYmhfZW5hYmxlKCkgdG8gaW52 b2tlIHNvZnRpcnEgaWYga3NvZnRpcnFkIGlzIGFscmVhZHkgcnVubmluZy4KCk1heWJlIHdlIGNh biBjb21lIHVwIHdpdGggYSBiZXR0ZXIgc29sdXRpb24/CihhcyBJIGRvIGFncmVlIHRoaXMgd2Fz IGEgdG9vIGJpZy1oYW1tZXIgYWZmZWN0aW5nIG90aGVyIHVzZS1jYXNlcykKCgpbMV0gaHR0cHM6 Ly9ibG9nLmNsb3VkZmxhcmUuY29tL2hvdy10by1yZWNlaXZlLWEtbWlsbGlvbi1wYWNrZXRzLwoK cC5zLiBSZWdhcmRpbmcgcXVvdGVbMV0gcG9pbnQgIjEuIiwgYWZ0ZXIgUGFvbG8gQWJlbmkgb3B0 aW1pemVkIHRoZSBVRFAKY29kZSwgdGhhdCBzdGF0ZW1lbnQgaXMgbm8gbG9uZ2VyIHRydWUuICBJ dCBub3cgKHNpZ25pZmljYW50bHkpIGZhc3RlciB0bwpydW4vcGluIHlvdXIgVURQIGFwcGxpY2F0 aW9uIHRvIGFub3RoZXIgQ1BVIHRoYW4gdGhlIFJYLUNQVS4K From mboxrd@z Thu Jan 1 00:00:00 1970 Return-path: Received: from mx1.redhat.com ([209.132.183.28]:48724 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752649AbeAJJpb (ORCPT ); Wed, 10 Jan 2018 04:45:31 -0500 Date: Wed, 10 Jan 2018 10:45:14 +0100 From: Jesper Dangaard Brouer To: Linus Torvalds Cc: Eric Dumazet , Josef Griebichler , Peter Zijlstra , Mauro Carvalho Chehab , Alan Stern , Greg Kroah-Hartman , USB list , Rik van Riel , Paolo Abeni , Hannes Frederic Sowa , linux-kernel , netdev , Jonathan Corbet , LMML , David Miller , Marek Majkowski Subject: Re: dvb usb issues since kernel 4.9 Message-ID: <20180110104514.4cd0e94a@redhat.com> In-Reply-To: References: <20180107090336.03826df2@vento.lan> <20180108074324.3c153189@vento.lan> <20180108223109.66c91554@redhat.com> <20180108214427.GT29822@worktop.programming.kicks-ass.net> <20180108231656.3bbd1968@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: linux-media-owner@vger.kernel.org List-ID: On Tue, 9 Jan 2018 10:58:30 -0800 Linus Torvalds wrote: > So I really think "you can use up 90% of CPU time with a UDP packet > flood from the same network" is very very very different - and > honestly not at all as important - as "you want to be able to use a > USB DVB receiver and watch/record TV". > > Because that whole "UDP packet flood from the same network" really is > something you _fundamentally_ have other mitigations for. > > I bet that whole commit was introduced because of a benchmark test, > rather than real life. No? I believe this have happened in real-life. In the form of DNS servers not being able to recover after long outage, where DNS-TTL had timeout causing legitimate traffic to overload their DNS servers. The goodput answers/sec from their DNS servers were too low, when bringing them online again. (Based on talk over beer at NetDevConf from a guy claiming they ran DNS for AWS). The commit 4cd13c21b207 ("softirq: Let ksoftirqd do its job") tries to address a fundamental problem that the network stack have when interacting with softirq in overload situations. (Maybe we can come up with a better solution?) Before this commit, when application run on same CPU as softirq, the kernel have a bad "drop off cliff" behavior, when reaching above the saturation point. This is confirmed in CloudFlare blogpost[1], which used a kernel that predates this commit. From[1] section: "A note on NUMA performance" Quote:" 1. Run receiver on another CPU, but on the same NUMA node as the RX queue. The performance as we saw above is around 360kpps. 2. With receiver on exactly same CPU as the RX queue we can get up to ~430kpps. But it creates high variability. The performance drops down to zero if the NIC is overwhelmed with packets." The behavior problem here is "performance drops down to zero if the NIC is overwhelmed with packets". That is a bad way to handle overload. Not only when attacked, but also when bringing a service online after an outage. What essentially happens is that: 1. softirq NAPI enqueue 64 packets into socket. 2. application dequeue 1 packet and invoke local_bh_enable() 3. causing softirq to run in app-timeslice, again enq 64 packets 4. app only see goodput of 1/128 of packets That is essentially what Eric solved with his commit, avoiding (3) local_bh_enable() to invoke softirq if ksoftirqd is already running. Maybe we can come up with a better solution? (as I do agree this was a too big-hammer affecting other use-cases) [1] https://blog.cloudflare.com/how-to-receive-a-million-packets/ p.s. Regarding quote[1] point "1.", after Paolo Abeni optimized the UDP code, that statement is no longer true. It now (significantly) faster to run/pin your UDP application to another CPU than the RX-CPU. -- Best regards, Jesper Dangaard Brouer MSc.CS, Principal Kernel Engineer at Red Hat LinkedIn: http://www.linkedin.com/in/brouer