From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755049AbbBKEmw (ORCPT ); Tue, 10 Feb 2015 23:42:52 -0500 Received: from mail-wi0-f178.google.com ([209.85.212.178]:37812 "EHLO mail-wi0-f178.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752196AbbBKEbW (ORCPT ); Tue, 10 Feb 2015 23:31:22 -0500 Message-ID: <1423629076.9360.28.camel@gmail.com> Subject: Re: process_backlog interruptions with 3.10.47-rt50 From: Mike Galbraith To: trevor_davenport@selinc.com Cc: linux-kernel@vger.kernel.org Date: Wed, 11 Feb 2015 05:31:16 +0100 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.12.9 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 2015-02-10 at 13:43 -0800, trevor_davenport@selinc.com wrote: > I've recently encountered a problem after upgrading from 3.0.57-rt82 to > 3.10.47-rt50 where process_backlog gets interrupted and does not resume > for a while, which results in packets not being processed in time. I see > net_rx_action, which then calls process_backlog (as the poll method to > process the backlog of packets queued up the netif_rx) but then after the > interruption, it does not finish for about 5ms. In the older kernel it > would finish based on the priority of ksoftirqd. This is no longer the > case. > > I have priorities configured so that hard interrupts are highest, > ksoftirqd next (both are SCHED_FIFO) and then my program is currently > SCHED_OTHER but I still do not see the rx softirq finish before my program > runs. > > This is all on a single core powerpc device. I do not see these problems > with a net device which uses NAPI directly (as such i'm updating my driver > to use NAPI) but it seems like there is a real bug here somewhere. I have > not been able to find any mention of similar problems (perhaps few people > are using netif_rx these days). > > I've attached a recording from perf which shows the problem. Specifically, > you see net_rx_action run at time 213.079014 and then it doesn't finish > until about 5ms later at time 213.084953 which i not the case on the older > kernels. It seems something has changed with softirq handling or > process_backlog needed adapted for it. My suspicion is this has something > to do with the work mentioned in 210dc110063cf040d3209fddf766f6fcafccdc34 > but I'm not an expert with this area of the kernel. Your suspicion is correct. Your net traffic is being handled by your SCHED_OTHER database task, which lost the CPU for a while due to it being a SCHED_OTHER task. It's a behavior change from previous rt kernels, but not a bad one. At the rt mini-summit of whatever year that was, this change was shown to be a massive win. Low priority network traffic now won't hinder a high priority task getting to the CPU, and should a high priority task block due to your low priority task having been preempted while holding the sirq lock it wants, PI will kick in. -Mike