Linux Documentation
 help / color / mirror / Atom feed
From: Tio Zhang <tiozhang@didiglobal.com>
To: <mingo@redhat.com>, <peterz@infradead.org>,
	<juri.lelli@redhat.com>, <vincent.guittot@linaro.org>,
	<rostedt@goodmis.org>, <bsingharora@gmail.com>, <corbet@lwn.net>,
	<akpm@linux-foundation.org>
Cc: <linux-kernel@vger.kernel.org>, <linux-doc@vger.kernel.org>,
	<dietmar.eggemann@arm.com>, <bsegall@google.com>,
	<mgorman@suse.de>, <bristot@redhat.com>, <vschneid@redhat.com>,
	<tiozhang@didiglobal.com>, <zyhtheonly@gmail.com>,
	<zyhtheonly@yeah.net>, <fuyuanli@didiglobal.com>
Subject: [PATCHSET] sched/delayacct: get task SOFTIRQ delay
Date: Tue, 2 Apr 2024 19:21:12 +0800	[thread overview]
Message-ID: <20240402112112.GA17370@didi-ThinkCentre-M930t-N000> (raw)

We can only get IRQ/SOFTIRQ delay in total now in Delay accounting, but
getting SOFTIRQ delay and IRQ delay separetely would help users reduce
such delays in a more convenient way.
For IRQ delay, we can tuning irq CPU affinity or using threaded-irq.
For SOFTIRQ delay, we can tuning rps/xps or using kernel threads for NAPI.
And this is an example stack a task is delayed mainly by SOFTIRQ(delay
by receiving packets when sending packets):
	...
	ip_rcv
	__netif_receive_skb_core
	__netif_receive_skb
	process_backlog
	net_rx_action
	do_softirq
	__local_bh_enable_ip
	ip_finish_output2
	ip_finish_output
	ip_output
	ip_local_out
	ip_send_skb
	udp_send_skb
	udp_sendmsg
	inet_sendmsg
	sock_sendmsg
	__sys_sendto
	do_syscall_64
	__libc_sendto
	...

So this patchset tries to make SOFTIRQ delay observeable in Delay
accounting and available in taskstats.
(also update tools/accounting/getdelays.c)

Also for backward compatibility, we dont want to change the meaning of
origin IRQ/SOFTIRQ delay, instead we can get real IRQ(interrupt) delay by
the origin IRQ/SOFTIRQ delay minus SOFTIRQ delay added by this patch.

With this patch, the example above results by getdelays.c:
	# ./getdelays -t 4600 -d
	print delayacct stats ON
	TGID	4600
	
	CPU             count     real total  virtual total    delay total  delay average
                 	3973    10700014780    10698803222   312345815813         78.617ms
	IO              count    delay total  delay average
                    	0              0              0.000ms
	SWAP            count    delay total  delay average
                    	0              0              0.000ms
	RECLAIM         count    delay total  delay average
                    	0              0              0.000ms
	THRASHING       count    delay total  delay average
                    	0              0              0.000ms
	COMPACT         count    delay total  delay average
                    	0              0              0.000ms
	WPCOPY          count    delay total  delay average
                   	40         266859             0.007ms
	IRQ             count    delay total  delay average
                	13450    17756373906          1.320ms
	SOFTIRQ         count    delay total  delay average
        	        13450    17639154300          1.311ms

We find out SOFTIRQ impact the delay most, then tune RPS to reduce this.

             reply	other threads:[~2024-04-02 11:21 UTC|newest]

Thread overview: 4+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2024-04-02 11:21 Tio Zhang [this message]
2024-04-02 11:24 ` [PATCH 1/3] sched: make softirq cputime accounting separately in irqtime Tio Zhang
2024-04-02 11:27   ` [PATCH 2/3] delayacct: get delay of SOFTIRQ Tio Zhang
2024-04-02 11:29     ` [PATCH 3/3] delayacct/taskstats: make soft_delay available in taskstats Tio Zhang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20240402112112.GA17370@didi-ThinkCentre-M930t-N000 \
    --to=tiozhang@didiglobal.com \
    --cc=akpm@linux-foundation.org \
    --cc=bristot@redhat.com \
    --cc=bsegall@google.com \
    --cc=bsingharora@gmail.com \
    --cc=corbet@lwn.net \
    --cc=dietmar.eggemann@arm.com \
    --cc=fuyuanli@didiglobal.com \
    --cc=juri.lelli@redhat.com \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mgorman@suse.de \
    --cc=mingo@redhat.com \
    --cc=peterz@infradead.org \
    --cc=rostedt@goodmis.org \
    --cc=vincent.guittot@linaro.org \
    --cc=vschneid@redhat.com \
    --cc=zyhtheonly@gmail.com \
    --cc=zyhtheonly@yeah.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox