From: Christophe Leroy <christophe.leroy@csgroup.eu>
To: Eric Dumazet <edumazet@google.com>,
"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
David Dworken <ddworken@google.com>,
Willem de Bruijn <willemb@google.com>
Cc: "David S. Miller" <davem@davemloft.net>
Subject: 126 ms irqsoff Latency - Possibly due to commit 190cc82489f4 ("tcp: change source port randomizarion at connect() time")
Date: Sat, 1 Oct 2022 17:16:08 +0000 [thread overview]
Message-ID: <03a06114-bc63-bc01-be38-535bcc394612@csgroup.eu> (raw)
Hi,
With recent kernels I have a huge irqsoff latency in my boards, shortly
after startup, from the call to net_get_random_once() in
__inet_hash_connect().
On a non instrumented kernel, IRQs are disabled during approximately 80
milliseconds. With the traces in goes to 126 milliseconds.
Was apparently introduced by commit 190cc82489f4 ("tcp: change source
port randomizarion at connect() time")
Trace below.
Would there be a way to perform the call to get_random_bytes() without
disabling IRQ ?
Thanks
Christophe
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 6.0.0-rc5-s3k-dev-02351-gebc95f69a7d4
# --------------------------------------------------------------------
# latency: 126337 us, #8207/8207, CPU#0 | (M:preempt VP:0, KP:0, SP:0 HP:0)
# -----------------
# | task: CORSurv-352 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: _raw_spin_lock_irqsave
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
CORSurv-352 0d.... 4us : _raw_spin_lock_irqsave
CORSurv-352 0d.... 13us+: preempt_count_add
<-_raw_spin_lock_irqsave
CORSurv-352 0d..1. 25us+: do_raw_spin_lock
<-_raw_spin_lock_irqsave
CORSurv-352 0d..1. 36us : get_random_bytes <-__inet_hash_connect
CORSurv-352 0d..1. 45us : _get_random_bytes.part.0
<-__inet_hash_connect
CORSurv-352 0d..1. 55us : crng_make_state
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 65us+: ktime_get_seconds <-crng_make_state
CORSurv-352 0d..1. 77us+: crng_fast_key_erasure <-crng_make_state
CORSurv-352 0d..1. 89us+: chacha_block_generic
<-crng_fast_key_erasure
CORSurv-352 0d..1. 101us+: chacha_permute <-chacha_block_generic
CORSurv-352 0d..1. 129us : chacha_block_generic
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 139us+: chacha_permute <-chacha_block_generic
CORSurv-352 0d..1. 160us : chacha_block_generic
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 170us+: chacha_permute <-chacha_block_generic
CORSurv-352 0d..1. 191us : chacha_block_generic
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 200us+: chacha_permute <-chacha_block_generic
CORSurv-352 0d..1. 221us : chacha_block_generic
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 231us+: chacha_permute <-chacha_block_generic
8182 x the above two line
CORSurv-352 0d..1. 126275us : chacha_block_generic
<-_get_random_bytes.part.0
CORSurv-352 0d..1. 126285us+: chacha_permute <-chacha_block_generic
CORSurv-352 0d..1. 126309us : _raw_spin_unlock_irqrestore
<-__do_once_done
CORSurv-352 0d..1. 126318us+: do_raw_spin_unlock
<-_raw_spin_unlock_irqrestore
CORSurv-352 0d..1. 126330us+: _raw_spin_unlock_irqrestore
CORSurv-352 0d..1. 126346us+: trace_hardirqs_on
<-_raw_spin_unlock_irqrestore
CORSurv-352 0d..1. 126387us : <stack trace>
=> tcp_v4_connect
=> __inet_stream_connect
=> inet_stream_connect
=> __sys_connect
=> system_call_exception
=> ret_from_syscall
next reply other threads:[~2022-10-01 17:16 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-10-01 17:16 Christophe Leroy [this message]
2022-10-01 17:43 ` 126 ms irqsoff Latency - Possibly due to commit 190cc82489f4 ("tcp: change source port randomizarion at connect() time") Eric Dumazet
2022-10-01 17:58 ` Eric Dumazet
2022-10-01 22:16 ` Jason A. Donenfeld
2022-10-01 22:31 ` Eric Dumazet
2022-10-01 22:37 ` Jason A. Donenfeld
2022-10-01 22:34 ` Jason A. Donenfeld
2022-10-01 22:37 ` Eric Dumazet
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=03a06114-bc63-bc01-be38-535bcc394612@csgroup.eu \
--to=christophe.leroy@csgroup.eu \
--cc=davem@davemloft.net \
--cc=ddworken@google.com \
--cc=edumazet@google.com \
--cc=netdev@vger.kernel.org \
--cc=willemb@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox