From: Jakub Kicinski <kuba@kernel.org>
To: Tariq Toukan <ttoukan.linux@gmail.com>
Cc: netdev@vger.kernel.org, edumazet@google.com, pabeni@redhat.com,
borisp@nvidia.com, john.fastabend@gmail.com, maximmi@nvidia.com,
tariqt@nvidia.com, vfedorenko@novek.ru,
Ran Rozenstein <ranro@nvidia.com>,
"gal@nvidia.com" <gal@nvidia.com>,
"David S. Miller" <davem@davemloft.net>
Subject: Re: [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser
Date: Tue, 2 Aug 2022 08:40:21 -0700 [thread overview]
Message-ID: <20220802084021.2ee26764@kernel.org> (raw)
In-Reply-To: <84406eec-289b-edde-759a-cf0b2c39c150@gmail.com>
On Tue, 2 Aug 2022 17:54:01 +0300 Tariq Toukan wrote:
> Hi Jakub,
>
> The device offload flow got broken, we started getting the call trace
> below in our regressions tests.
>
> Bisecting points to this one as the offending commit.
>
> I taking a look, but I'm less familiar with this change.
> Probably you have a direction?
Sorry about that, I'll take a look. Would you be able to run it thru
decode_stacktrace before I get to the office and start digging?
> [ 407.560799] rcu: INFO: rcu_sched self-detected stall on CPU
> [ 407.561734] rcu: 1-....: (5248 ticks this GP)
> idle=51b/1/0x4000000000000000 softirq=41347/41347 fqs=2625
> [ 407.563101] (t=5250 jiffies g=65669 q=4492 ncpus=10)
> [ 407.563859] NMI backtrace for cpu 1
> [ 407.564430] CPU: 1 PID: 45266 Comm: iperf Not tainted
> 5.19.0-rc7_for-upstream_min-debug_5f35d2896553 #1
> [ 407.565766] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009),
> BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
> [ 407.567319] Call Trace:
> [ 407.567772] <IRQ>
> [ 407.568168] dump_stack_lvl+0x34/0x44
> [ 407.568763] nmi_cpu_backtrace.cold+0x30/0x70
> [ 407.569462] ? lapic_can_unplug_cpu+0x70/0x70
> [ 407.570146] nmi_trigger_cpumask_backtrace+0xef/0x100
> [ 407.570897] trigger_single_cpu_backtrace+0x24/0x27
> [ 407.571635] rcu_dump_cpu_stacks+0xa0/0xd9
> [ 407.572278] rcu_sched_clock_irq.cold+0x111/0x2d3
> [ 407.573004] update_process_times+0x5b/0x90
> [ 407.584227] tick_sched_timer+0x88/0xa0
> [ 407.584855] ? tick_sched_do_timer+0xf0/0xf0
> [ 407.585515] __hrtimer_run_queues+0x139/0x290
> [ 407.586199] hrtimer_interrupt+0x10e/0x240
> [ 407.586844] __sysvec_apic_timer_interrupt+0x56/0xd0
> [ 407.587575] sysvec_apic_timer_interrupt+0x6d/0x90
> [ 407.588293] </IRQ>
> [ 407.588699] <TASK>
> [ 407.589118] asm_sysvec_apic_timer_interrupt+0x16/0x20
> [ 407.589886] RIP: 0010:tls_device_decrypted+0x7a/0x2e0
> [ 407.590641] Code: 83 e7 01 45 0f b6 e7 41 83 f7 01 48 85 d2 45 0f
> b6 ff 74 1b 0f b6 82 83 00 00 00 48 8b 12 c0 e8 06 41 21 c4 83 f0 01 41
> 21 c7 <48> 85 d2 75 e5 8b 05 d3 71 ca 00 85 c0 0f 8f 4b 01 00 00 48 8b 85
> [ 407.593152] RSP: 0018:ffff8881a113bb60 EFLAGS: 00000202
> [ 407.593927] RAX: 0000000000000003 RBX: ffff88810e2d6000 RCX:
> 0000000000000000
> [ 407.594913] RDX: ffff8881792b80d8 RSI: ffff88814ef46800 RDI:
> ffff8881792b8000
> [ 407.595904] RBP: ffff88814ef46800 R08: 7fffffffffffffff R09:
> 0000000000000001
> [ 407.596907] R10: ffff888105cd2200 R11: 0000000000022dd1 R12:
> 0000000000000000
> [ 407.597901] R13: ffff888108f45000 R14: ffff8881792b8000 R15:
> 0000000000000001
> [ 407.598897] ? tls_rx_rec_wait+0x225/0x250
> [ 407.599539] tls_rx_one_record+0xe4/0x2d0
> [ 407.600169] tls_sw_recvmsg+0x380/0x910
> [ 407.600773] ? 0xffffffff81000000
> [ 407.601336] inet6_recvmsg+0x47/0xc0
> [ 407.601932] ____sys_recvmsg+0x109/0x120
> [ 407.602554] ? _copy_from_user+0x26/0x60
> [ 407.603171] ? iovec_from_user+0x4a/0x150
> [ 407.603803] ___sys_recvmsg+0xa4/0xe0
> [ 407.604394] ? mlx5e_ktls_add_rx+0x349/0x430 [mlx5_core]
> [ 407.605262] ? mlx5e_ktls_add_rx+0x3af/0x430 [mlx5_core]
> [ 407.606082] ? tls_device_attach+0x60/0xe0
> [ 407.606721] ? tls_set_device_offload_rx+0x11b/0x220
> [ 407.607464] __sys_recvmsg+0x4e/0x90
> [ 407.608050] do_syscall_64+0x3d/0x90
> [ 407.608631] entry_SYSCALL_64_after_hwframe+0x46/0xb0
> [ 407.609400] RIP: 0033:0x7f7a41f856dd
next prev parent reply other threads:[~2022-08-02 15:41 UTC|newest]
Thread overview: 30+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-22 23:50 [PATCH net-next v3 0/7] tls: rx: decrypt from the TCP queue Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 1/7] tls: rx: wrap recv_pkt accesses in helpers Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 2/7] tls: rx: factor SW handling out of tls_rx_one_record() Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 3/7] tls: rx: don't free the output in case of zero-copy Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 4/7] tls: rx: device: keep the zero copy status with offload Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 5/7] tcp: allow tls to decrypt directly from the tcp rcv queue Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 6/7] tls: rx: device: add input CoW helper Jakub Kicinski
2022-07-22 23:50 ` [PATCH net-next v3 7/7] tls: rx: do not use the standard strparser Jakub Kicinski
2022-07-26 9:27 ` Paolo Abeni
2022-07-26 17:01 ` Jakub Kicinski
2022-07-26 17:26 ` Paolo Abeni
2022-08-02 14:54 ` Tariq Toukan
2022-08-02 15:40 ` Jakub Kicinski [this message]
2022-08-04 1:24 ` Jakub Kicinski
2022-08-04 6:13 ` Tariq Toukan
2022-08-04 8:05 ` Tariq Toukan
2022-08-04 15:35 ` Jakub Kicinski
2022-08-07 6:01 ` Tariq Toukan
2022-08-04 15:59 ` Jakub Kicinski
2022-08-07 6:01 ` Tariq Toukan
2022-08-08 5:24 ` Tariq Toukan
2022-08-08 18:24 ` Jakub Kicinski
2022-08-09 8:47 ` Tariq Toukan
2023-03-09 15:15 ` Tariq Toukan
2023-03-09 17:54 ` Jakub Kicinski
2023-03-12 17:59 ` Tariq Toukan
2023-03-13 18:22 ` Jakub Kicinski
2023-03-15 20:26 ` Tariq Toukan
2023-03-16 1:41 ` Jakub Kicinski
2022-07-26 22:00 ` [PATCH net-next v3 0/7] tls: rx: decrypt from the TCP queue patchwork-bot+netdevbpf
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220802084021.2ee26764@kernel.org \
--to=kuba@kernel.org \
--cc=borisp@nvidia.com \
--cc=davem@davemloft.net \
--cc=edumazet@google.com \
--cc=gal@nvidia.com \
--cc=john.fastabend@gmail.com \
--cc=maximmi@nvidia.com \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=ranro@nvidia.com \
--cc=tariqt@nvidia.com \
--cc=ttoukan.linux@gmail.com \
--cc=vfedorenko@novek.ru \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).