From: Tio Zhang <tiozhang@didiglobal.com>
To: <mingo@redhat.com>, <peterz@infradead.org>,
<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
<rostedt@goodmis.org>, <bsingharora@gmail.com>, <corbet@lwn.net>,
<akpm@linux-foundation.org>
Cc: <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
<dietmar.eggemann@arm.com>, <bsegall@google.com>,
<mgorman@suse.de>, <bristot@redhat.com>, <vschneid@redhat.com>,
<tiozhang@didiglobal.com>, <zyhtheonly@gmail.com>,
<zyhtheonly@yeah.net>, <fuyuanli@didiglobal.com>
Subject: [PATCHSET] sched/delayacct: get task SOFTIRQ delay
Date: Tue, 2 Apr 2024 19:21:12 +0800 [thread overview]
Message-ID: <20240402112112.GA17370@didi-ThinkCentre-M930t-N000> (raw)
We can only get IRQ/SOFTIRQ delay in total now in Delay accounting, but
getting SOFTIRQ delay and IRQ delay separetely would help users reduce
such delays in a more convenient way.
For IRQ delay, we can tuning irq CPU affinity or using threaded-irq.
For SOFTIRQ delay, we can tuning rps/xps or using kernel threads for NAPI.
And this is an example stack a task is delayed mainly by SOFTIRQ(delay
by receiving packets when sending packets):
...
ip_rcv
__netif_receive_skb_core
__netif_receive_skb
process_backlog
net_rx_action
do_softirq
__local_bh_enable_ip
ip_finish_output2
ip_finish_output
ip_output
ip_local_out
ip_send_skb
udp_send_skb
udp_sendmsg
inet_sendmsg
sock_sendmsg
__sys_sendto
do_syscall_64
__libc_sendto
...
So this patchset tries to make SOFTIRQ delay observeable in Delay
accounting and available in taskstats.
(also update tools/accounting/getdelays.c)
Also for backward compatibility, we dont want to change the meaning of
origin IRQ/SOFTIRQ delay, instead we can get real IRQ(interrupt) delay by
the origin IRQ/SOFTIRQ delay minus SOFTIRQ delay added by this patch.
With this patch, the example above results by getdelays.c:
# ./getdelays -t 4600 -d
print delayacct stats ON
TGID 4600
CPU count real total virtual total delay total delay average
3973 10700014780 10698803222 312345815813 78.617ms
IO count delay total delay average
0 0 0.000ms
SWAP count delay total delay average
0 0 0.000ms
RECLAIM count delay total delay average
0 0 0.000ms
THRASHING count delay total delay average
0 0 0.000ms
COMPACT count delay total delay average
0 0 0.000ms
WPCOPY count delay total delay average
40 266859 0.007ms
IRQ count delay total delay average
13450 17756373906 1.320ms
SOFTIRQ count delay total delay average
13450 17639154300 1.311ms
We find out SOFTIRQ impact the delay most, then tune RPS to reduce this.
next reply other threads:[~2024-04-02 11:21 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-04-02 11:21 Tio Zhang [this message]
2024-04-02 11:24 ` [PATCH 1/3] sched: make softirq cputime accounting separately in irqtime Tio Zhang
2024-04-02 11:27 ` [PATCH 2/3] delayacct: get delay of SOFTIRQ Tio Zhang
2024-04-02 11:29 ` [PATCH 3/3] delayacct/taskstats: make soft_delay available in taskstats Tio Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240402112112.GA17370@didi-ThinkCentre-M930t-N000 \
--to=tiozhang@didiglobal.com \
--cc=akpm@linux-foundation.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=bsingharora@gmail.com \
--cc=corbet@lwn.net \
--cc=dietmar.eggemann@arm.com \
--cc=fuyuanli@didiglobal.com \
--cc=juri.lelli@redhat.com \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mgorman@suse.de \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=zyhtheonly@gmail.com \
--cc=zyhtheonly@yeah.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox