From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755138AbcB2PDT (ORCPT ); Mon, 29 Feb 2016 10:03:19 -0500 Received: from mail-pa0-f44.google.com ([209.85.220.44]:33019 "EHLO mail-pa0-f44.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752070AbcB2PDQ (ORCPT ); Mon, 29 Feb 2016 10:03:16 -0500 Subject: Re: Softirq priority inversion from "softirq: reduce latencies" To: Mike Galbraith References: <56D1E8B6.6090003@hurleysoftware.com> <1456638957.3676.12.camel@gmail.com> <20160228170109.GA16322@electric-eye.fr.zoreil.com> <1456721889.3488.67.camel@gmail.com> Cc: Francois Romieu , Eric Dumazet , David Miller , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Greg KH , dmaengine@vger.kernel.org, John Ogness , Sebastian Andrzej Siewior , Andrew Morton , Thomas Gleixner From: Peter Hurley Message-ID: <56D45DAF.5070709@hurleysoftware.com> Date: Mon, 29 Feb 2016 07:03:11 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <1456721889.3488.67.camel@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 02/28/2016 08:58 PM, Mike Galbraith wrote: > On Sun, 2016-02-28 at 18:01 +0100, Francois Romieu wrote: >> Mike Galbraith : >> [...] >>> Hrm, relatively new + tasklet woes rings a bell. Ah, that.. >>> >>> >>> What's worse is that at the point where this code was written it was >>> already well known that tasklets are a steaming pile of crap and >>> should die. >>> >>> >>> Source thereof https://lwn.net/Articles/588457/ Thanks but not applicable. tglx's POV has everything to do with the tasklet interface and not the general concept of bottom-half interrupt processing in a timely manner. In any event, the problem created by Eric's change is not restricted to tasklets, but rather applies to all softirq. >> tasklets are ingrained in the dmaengine API (see Documentation/dmaengine/client.txt >> and drivers/dma/virt-dma.h::vchan_cookie_complete). >> >> Moving everything to irq context or handling his own sub-{jiffy/ms} timer >> while losing async dma doesn't exactly smell like roses either. :o( > > https://lwn.net/Articles/239633/ > > If I'm listening properly, the root cause is that there is a timing > constraint involved, which is being exposed because one softirq raises > another (ew). Not the case. The softirq is raised from interrupt. Before Eric's change, when an interrupt raises a new softirq while processing another softirq, the new softirq is immediately processed *after the existing softirq completes*. After Eric's change, when an interrupt raises a new softirq while processing another softirq and _that softirq wakes a process_, the new softirq is *deferred to normal process priority*. This happens even if the new softirq is higher priority than the one currently running, which is flat-out wrong. The reason this happens repeatedly and regularly is because 1. The time window while NET_RX softirq is running is big. 2. NET_RX softirq will almost always wake a process for a received packet. The reason why Eric's change is so effective for Eric's workload is that it fixes the problem where NET_RX keeps getting new network packets so it keeps looping, servicing more NET_RX softirq. However, I'm pointing out that Eric's sledgehammer approach to fixing the NET_RX softirq bug is having significant side-effects in other subsystems. > Processing timeout happens, freshly raised tasklet > wanders off to SCHED_NORMAL kthread context where its constraint dies. > > Given the dma stuff apparently works fine in -rt (or did, see below), > timing constraints can't be super tight, so perhaps we could grow > realtime workqueue support for the truly deserving. The tricky bit > would be being keeping everybody and his brother from abusing it. > > WRT -rt: if dma tasklets really do have hard (ish) constraints, -rt > recently "broke" in the same way.. of all softirqs which are deferred > to kthread context, due to a recent change, only timer/hrtimer are > executed at realtime priority by default. > > -Mike >