From: Dmitry Safonov <dima@arista.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: linux-kernel@vger.kernel.org, 0x7f454c46@gmail.com,
Andrew Morton <akpm@linux-foundation.org>,
David Miller <davem@davemloft.net>,
Eric Dumazet <edumazet@google.com>,
Hannes Frederic Sowa <hannes@stressinduktion.org>,
Ingo Molnar <mingo@kernel.org>,
"Levin, Alexander (Sasha Levin)" <alexander.levin@verizon.com>,
Linus Torvalds <torvalds@linux-foundation.org>,
Paolo Abeni <pabeni@redhat.com>,
"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
Peter Zijlstra <peterz@infradead.org>,
Radu Rendec <rrendec@arista.com>, Rik van Riel <riel@redhat.com>,
Stanislaw Gruszka <sgruszka@redhat.com>,
Thomas Gleixner <tglx@linutronix.de>,
Wanpeng Li <wanpeng.li@hotmail.com>
Subject: Re: [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd context
Date: Wed, 10 Jan 2018 21:20:31 +0000 [thread overview]
Message-ID: <1515619231.3350.36.camel@arista.com> (raw)
In-Reply-To: <CAFTL4hyPrwo+RXUkBp5FS_MC6qiMsK5bCVg2_tn0uPu6d_fgqg@mail.gmail.com>
On Tue, 2018-01-09 at 17:03 +0100, Frederic Weisbecker wrote:
> 2018-01-09 14:36 UTC+01:00, Dmitry Safonov <dima@arista.com>:
> > Warning: Not merge-ready
> >
> > I. Current workflow of ksoftirqd.
> > Softirqs are processed in the context of ksoftirqd iff they are
> > being raised very frequently. How it works:
> > do_softirq() and invoke_softirq() deffer pending softirq iff
> > ksoftirqd is in runqueue. Ksoftirqd is scheduled mostly in the
> > end of processed softirqs if 2ms were not enough to process all
> > pending softirqs.
> >
> > Here is pseudo-picture of the workflow (for simplicity on UMP):
> > ------------- ------------------ ------------------
> > | ksoftirqd | | User's process | | Softirqs |
> > ------------- ------------------ ------------------
> > Not scheduled Running
> > |
> > o------------------------o
> > |
> > __do_softirq()
> > |
> > 2ms & softirq
> > pending?
> > Schedule ksoftirqd
> > |
> > Scheduled o------------------------o
> > |
> > o--------------------o
> > |
> > Running Scheduled
> > |
> > o--------------------o
> > |
> > Not scheduled Running
> >
> > Timegraph for the workflow,
> > dash (-) means ksoftirqd not scheduled;
> > equal(=) ksoftirqd is scheduled, a softirq may still be
> > pending
> >
> > Pending softirqs
> > | | | | | | | | |
> > v v v v | | | | v
> > Processing o-----o | | | | o--o
> > softirqs | | | | | | | |
> > | | | | | | | |
> > | | | | | | | |
> > Userspace o-o o=========o | | | | o----o o---------o
> > <-2ms-> | | | | | |
> > | v v v v |
> > Ksoftirqd o----------o
> >
> > II. Corner-conditions.
> > During testing of commit [1] on some non-mainstream driver,
> > I've found that due to platform specifics, the IRQ is being
> > raised too late (after softirq has been processed).
>
> I'm a bit confused about that part. I would expect the softirq to be
> raised by the IRQ.
The rx-softirq is raised by napi_schedule(), which is called on
receiving an interrupt from device, yes.
> So I guess in this scenario the softirq is raised by something else
> and you expect the upcoming IRQ to handle the softirq, right? (sorry
> I'm not used to networking code).
So, the softirq is served by after upcomming IRQ. But in the end of the
softirq there is no yet-pending softirq. And just after irq_exit(),
there emerges another pending softirq.
ITOW, here is what I see on the trace:
python-8597 [001] d... 16635.495480: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495498: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495503: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495533: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495542: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495567: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495576: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495601: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495615: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495627: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495637: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495668: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495684: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495703: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495710: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495723: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495736: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495755: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495769: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495788: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495804: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495822: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495834: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495857: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495868: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495891: rcu_irq_exit <-irq_exit
python-8597 [001] d.s1 16635.495907: rcu_irq_exit <-irq_exit
python-8597 [001] d..1 16635.495908: __do_softirq <-
do_softirq_own_stack
python-8597 [001] d... 16635.495939: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495959: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.495973: __do_softirq <-irq_exit
python-8597 [001] d... 16635.495988: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496002: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496014: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496027: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496044: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496059: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496088: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496102: __do_softirq <-irq_exit
python-8597 [001] d.s. 16635.496108: rcu_irq_exit <-irq_exit
python-8597 [001] dN.. 16635.496131: rcu_irq_exit <-irq_exit
ksoftirqd/1-14 [001] d... 16635.496132: __do_softirq <-
run_ksoftirqd
ksoftirqd/1-14 [001] d.s1 16635.496145: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496171: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496191: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496204: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496218: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496231: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496250: rcu_irq_exit <-irq_exit
python-8597 [001] d... 16635.496263: __do_softirq <-irq_exit
python-8597 [001] d... 16635.496283: rcu_irq_exit <-irq_exit
Ksoftirqd here is very rarely scheduled, the python process here is
receiver, and looks CPU-starving:
%Cpu1 : 0.6 us, 9.5 sy, 0.0 ni, 0.0 id, 0.0 wa, 35.3 hi, 54.6
si, 0.0 st
[..]
PID USER PR NI VIRT RES SHR S %CPU
%MEM TIME+ COMMAND
8597 root 20 0 7664 5736 3924 R 10 0.0 4:00.69 python
That's what I tried to picture there with diagrams.
(the source of __do_softirq() also show how ksoftirq got scheduled)
--
Thanks,
Dmitry
next prev parent reply other threads:[~2018-01-10 21:20 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-01-09 13:36 [RFC 0/2] Net softirq deferring to ksoftirqd Dmitry Safonov
2018-01-09 13:36 ` [RFC 1/2] softirq: Defer net rx/tx processing to ksoftirqd context Dmitry Safonov
2018-01-09 16:03 ` Frederic Weisbecker
2018-01-10 21:20 ` Dmitry Safonov [this message]
2018-01-09 18:02 ` Eric Dumazet
2018-01-10 21:48 ` Dmitry Safonov
2018-01-11 2:13 ` Linus Torvalds
2018-01-11 3:22 ` Frederic Weisbecker
2018-01-11 4:19 ` Linus Torvalds
2018-01-11 4:44 ` Frederic Weisbecker
2018-01-11 14:31 ` Dmitry Safonov
2018-01-11 16:20 ` Eric Dumazet
2018-01-11 16:32 ` Peter Zijlstra
2018-01-11 16:38 ` David Miller
2018-01-11 16:43 ` Peter Zijlstra
2018-01-11 18:48 ` Linus Torvalds
2018-01-11 19:15 ` Eric Dumazet
2018-01-11 19:43 ` Linus Torvalds
2018-01-11 19:48 ` Eric Dumazet
2018-01-11 20:03 ` Linus Torvalds
2018-01-11 20:16 ` Eric Dumazet
2018-01-11 20:22 ` Linus Torvalds
2018-01-11 20:34 ` Dmitry Safonov
2018-01-11 20:37 ` Eric Dumazet
2018-01-11 20:40 ` Linus Torvalds
2018-01-11 20:46 ` Dmitry Safonov
2018-01-11 20:53 ` Eric Dumazet
2018-01-11 21:13 ` Dmitry Safonov
2018-01-12 5:41 ` Frederic Weisbecker
2018-01-12 5:23 ` Mike Galbraith
2018-01-12 10:13 ` Peter Zijlstra
2018-01-12 17:26 ` Linus Torvalds
2018-01-12 17:44 ` Mike Galbraith
2018-01-12 17:51 ` Linus Torvalds
2018-01-12 18:15 ` Mike Galbraith
2018-01-12 18:44 ` Mike Galbraith
2018-01-17 20:30 ` David Miller
2018-01-17 21:06 ` Linus Torvalds
2018-01-17 21:49 ` David Miller
2018-01-17 21:54 ` Thomas Gleixner
2018-01-17 21:58 ` Linus Torvalds
2018-01-17 22:00 ` Thomas Gleixner
2018-01-17 22:02 ` Eric Dumazet
2018-01-17 22:24 ` David Miller
2018-01-17 22:54 ` Thomas Gleixner
2018-02-06 16:56 ` Sebastian Andrzej Siewior
2018-02-13 16:36 ` Thomas Gleixner
2018-01-17 21:54 ` Linus Torvalds
2018-01-17 21:58 ` Thomas Gleixner
2018-01-12 19:28 ` Thomas Gleixner
2018-01-12 19:53 ` Frederic Weisbecker
2018-01-12 14:58 ` Frederic Weisbecker
2018-01-12 15:15 ` Mike Galbraith
2018-01-12 17:00 ` Frederic Weisbecker
2018-01-12 18:08 ` Mike Galbraith
2018-01-12 17:24 ` Linus Torvalds
2018-01-09 13:36 ` [RFC 2/2] softirq: Introduce mask for __do_softirq() Dmitry Safonov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1515619231.3350.36.camel@arista.com \
--to=dima@arista.com \
--cc=0x7f454c46@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=alexander.levin@verizon.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=frederic@kernel.org \
--cc=hannes@stressinduktion.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@kernel.org \
--cc=pabeni@redhat.com \
--cc=paulmck@linux.vnet.ibm.com \
--cc=peterz@infradead.org \
--cc=riel@redhat.com \
--cc=rrendec@arista.com \
--cc=sgruszka@redhat.com \
--cc=tglx@linutronix.de \
--cc=torvalds@linux-foundation.org \
--cc=wanpeng.li@hotmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox