From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11AC6C433F5 for ; Wed, 12 Jan 2022 12:09:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=J0NZ1hlxGlkbRW45oJ1vzkcSSfrieJFcC8sXVrAy8Js=; b=jhAH0ec1RZn4B+ 5TbJcuVkh+WJOUuZgt1VeJLeEkuGkY24jYa5x/D4+zE3GXb+gpi0z0uFmpu4h83ob159evNbJycOr VbRVUM5nOysDIgGBjY7+reHCAvLfDzAzv7xA3iffZZq5G9cP4SumK8QdZCzh/yWLIXk2PzmlLE4lk Ra4yBZJUEUcRQBhS1ayoDMpkCg4DoO5p6bISDO1s8uBzHzznRMO5c/v/2bJzq3tZF66L5OxKBDAMc u09YYQDk1rTUIL2exYeJaOySSZuMBs1TSSeoB1T0Kz98tBM+rUyGVp3S4ctVHGRouoUkIKrlRFVfu IYJ+zug1RY4pu/IWx0vw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1n7cQ9-002SGG-BD; Wed, 12 Jan 2022 12:08:05 +0000 Received: from mail-wm1-x330.google.com ([2a00:1450:4864:20::330]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1n7cQ5-002SFF-AK for linux-arm-kernel@lists.infradead.org; Wed, 12 Jan 2022 12:08:04 +0000 Received: by mail-wm1-x330.google.com with SMTP id bg19-20020a05600c3c9300b0034565e837b6so1630004wmb.1 for ; Wed, 12 Jan 2022 04:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=4LPkI+w4jHIHw9E16rqxjr7FBdGSROFOqJnifdLeE1A=; b=mQGnIpWSYtGtQ9f5ZaETvIiO1+uxsbdhu3K3+hAFI6zCW3xhQH3k6Soyp1WKeRUjTk ShkMyDWl38n00nB139NvAYRhMBSjJCcNnQ0f/6XwE/52LSuSN9Y1Em/uHI8PDrcxQNc6 lYhabjtOGqPZ+AgRQvO0ieJfbAittjUpPZ5a1JeKbIbEvBXmBb8M7WjI0ZnjnGHxLYEL 0753Du4X6yAwoS++LtITQ4yQ7/lIAbXXNlhD9Lg2zqrFtfGsUM7LA0qQlOdhoSomDjrb QvUrI1SCsT046uxP2CRas++3oH/cmK5kSAkHiRj8vaCTyksUtV5N1wSGUcX/EwNwOw1i US3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=4LPkI+w4jHIHw9E16rqxjr7FBdGSROFOqJnifdLeE1A=; b=HA19dTlxVtq6kyuteXHilOJq/ocu7Y01BhXOQyB9NuRUGCmnzpHpn66QPW5DQnj45H IY77BxPcjsSrLvNzjATa+/VZlYCYkzzwagbBArkEn9t5wQu1RD/L41njIXQ3QMiJedv8 KFHPFiA0amFD2avHNXKL1nsb/pP2JKgzcdTd73jrC5b78PJbXRjpKjcUIADSaTwxXgHA aaIfJ3vaC+ScaZJFmYjZxAjKMD4fgQ8+VyMcaHa2qy5X3wH+TlrRwm8IuzQ4Vu7nF1f1 uf43jirJUukS1a1P04i0gGCD1qn1oa233BCy10sWIMTkwU5kVspAcUVYyu39dD8ZxpF4 Mz7w== X-Gm-Message-State: AOAM530nx3vP89ckAoGd3t1+4NSM0rBF+RP6aU6E54mHBM2YfTkdMncU QR3YShQyURHG/BY2NtppmnRfS4+aa1A= X-Google-Smtp-Source: ABdhPJzxqJJBL4FmyHYN/gZzO9xvwSRllcEpCFNKDursLjJAVjfFdOjJLzkDDn/eeuzEAiibhEqHkQ== X-Received: by 2002:a05:600c:1e8b:: with SMTP id be11mr6629689wmb.51.1641989278552; Wed, 12 Jan 2022 04:07:58 -0800 (PST) Received: from Red ([2a01:cb1d:3d5:a100:264b:feff:fe03:2806]) by smtp.googlemail.com with ESMTPSA id 11sm16079844wrz.63.2022.01.12.04.07.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jan 2022 04:07:58 -0800 (PST) Date: Wed, 12 Jan 2022 13:07:52 +0100 From: Corentin Labbe To: herbert@gondor.apana.org.au, davem@davemloft.net, linux-arm-kernel@lists.infradead.org, netdev@vger.kernel.org, linux-crypto@vger.kernel.org, steffen.klassert@secunet.com Cc: linux-kernel@vger.kernel.org Subject: Re: crypto: BUG: spinlock recursion when doing iperf over ipsec with crypto hardware device Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220112_040801_418882_A16EDC5F X-CRM114-Status: GOOD ( 21.75 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Le Tue, Jan 11, 2022 at 10:47:12AM +0100, Corentin Labbe a =E9crit : I forgot to add some mailing list (linux-crypto) and I add ipsec maintainer= as spinlock is from ipsec code. And I have now identified the spin_lock which is stuck: It is the one from net/xfrm/xfrm_input.c:649 This is the patch used to identify it: diff --git a/include/linux/spinlock.h b/include/linux/spinlock.h index a732d62dbe35..63d388fe0363 100644 --- a/include/linux/spinlock.h +++ b/include/linux/spinlock.h @@ -354,6 +354,12 @@ static __always_inline void spin_lock(spinlock_t *lock) raw_spin_lock(&lock->rlock); } = +static __always_inline void spin_lock2(spinlock_t *lock, int vid) +{ + raw_spin_lock(&lock->rlock); + atomic64_set(&lock->rlock.id, vid); \ +} + static __always_inline void spin_lock_bh(spinlock_t *lock) { raw_spin_lock_bh(&lock->rlock); diff --git a/include/linux/spinlock_types_raw.h b/include/linux/spinlock_ty= pes_raw.h index 91cb36b65a17..247ed688e9e7 100644 --- a/include/linux/spinlock_types_raw.h +++ b/include/linux/spinlock_types_raw.h @@ -2,6 +2,7 @@ #define __LINUX_SPINLOCK_TYPES_RAW_H = #include +#include = #if defined(CONFIG_SMP) # include @@ -16,6 +17,7 @@ typedef struct raw_spinlock { #ifdef CONFIG_DEBUG_SPINLOCK unsigned int magic, owner_cpu; void *owner; + atomic64_t id; #endif #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; diff --git a/kernel/locking/spinlock_debug.c b/kernel/locking/spinlock_debu= g.c index 14235671a1a7..685981594c54 100644 --- a/kernel/locking/spinlock_debug.c +++ b/kernel/locking/spinlock_debug.c @@ -27,6 +27,7 @@ void __raw_spin_lock_init(raw_spinlock_t *lock, const cha= r *name, lock->magic =3D SPINLOCK_MAGIC; lock->owner =3D SPINLOCK_OWNER_INIT; lock->owner_cpu =3D -1; + atomic64_set(&lock->id, 666); \ } = EXPORT_SYMBOL(__raw_spin_lock_init); @@ -61,11 +62,12 @@ static void spin_dump(raw_spinlock_t *lock, const char = *msg) msg, raw_smp_processor_id(), current->comm, task_pid_nr(current)); printk(KERN_EMERG " lock: %pS, .magic: %08x, .owner: %s/%d, " - ".owner_cpu: %d\n", + ".owner_cpu: %d ID=3D%lu\n", lock, READ_ONCE(lock->magic), owner ? owner->comm : "", owner ? task_pid_nr(owner) : -1, - READ_ONCE(lock->owner_cpu)); + READ_ONCE(lock->owner_cpu), + atomic64_read(&lock->id)); dump_stack(); } = diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 144238a50f3d..f874c4af9220 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -596,7 +596,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32= spi, int encap_type) } = lock: - spin_lock(&x->lock); + spin_lock2(&x->lock, 15); = if (unlikely(x->km.state !=3D XFRM_STATE_VALID)) { if (x->km.state =3D=3D XFRM_STATE_ACQ) @@ -646,7 +646,7 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32= spi, int encap_type) resume: dev_put(skb->dev); = - spin_lock(&x->lock); + spin_lock2(&x->lock, 14); if (nexthdr < 0) { if (nexthdr =3D=3D -EBADMSG) { xfrm_audit_state_icvfail(x, skb, And I got: [ 49.170676] BUG: spinlock recursion on CPU#0, 1c15000.crypto-/58 [ 49.176747] lock: 0xc43642a8, .magic: dead4ead, .owner: 1c15000.crypto-= /58, .owner_cpu: 0 ID=3D14 Anyway, I believed that spinlock was from crypto/crypto_engine.c but the st= ack trace said the problem is from a basic spin_lock() and crypto_engine us= e only spin_lock_irq() Regards > Hello > = > I work on adding crypto/engine support to my crypto driver sun4i-ss. > Selftests are passed but when doing iperf over ipsec I hit a spinlock rec= ursion BUG. > = > This bug happen also against my sun8i-ce driver. > But this bug happen only on ARM (SoCs H3) and not on arm64 (SoCs H6, a64 = tested) for it. > = > This is the full trace: > iperf 3.10.1 > Linux buildroot 5.15.13-00003-g14f78eca130d #199 SMP Fri Jan 7 08:35:20 C= ET 2022 armv7l > Control connection MSS 1386 > Time: Fri, 07 Jan 2022 08:52:27 GMT > Connecting to host ipsec.lava.local, port 5201 > Cookie: ylt5ilt5n4jnw72qu2lik5njq63rfalkshz6 > TCP MSS: 1386 (default) > [ 5] local 192.168.1.204 port 33828 connected to 192.168.1.40 port 5201 > Starting Test: protocol: TCP, 1 streams, 131072 byte blocks, omitting 0 s= econds, 10 second test, tos 0 > [ 44.412526] BUG: spinlock recursion on CPU#0, 1c15000.crypto-/111 > [ 44.418674] lock: 0xc3f40268, .magic: dead4ead, .owner: 1c15000.crypt= o-/111, .owner_cpu: 0 > [ 44.427040] CPU: 0 PID: 111 Comm: 1c15000.crypto- Not tainted 5.15.13-= 00003-g14f78eca130d #199 > [ 44.435647] Hardware name: Allwinner sun8i Family > [ 44.440354] [] (unwind_backtrace) from [] (show_st= ack+0x10/0x14) > [ 44.448110] [] (show_stack) from [] (dump_stack_lv= l+0x40/0x4c) > [ 44.455685] [] (dump_stack_lvl) from [] (do_raw_sp= in_lock+0x11c/0x120) > [ 44.463953] [] (do_raw_spin_lock) from [] (xfrm_in= put+0x17c/0x1304) > [ 44.471961] [] (xfrm_input) from [] (xfrm4_esp_rcv= +0x44/0x84) > [ 44.479451] [] (xfrm4_esp_rcv) from [] (ip_protoco= l_deliver_rcu+0x2c/0x2b8) > [ 44.488156] [] (ip_protocol_deliver_rcu) from [] (= ip_local_deliver_finish+0x64/0x80) > [ 44.497636] [] (ip_local_deliver_finish) from [] (= ip_sublist_rcv_finish+0x3c/0x50) > [ 44.506944] [] (ip_sublist_rcv_finish) from [] (ip= _sublist_rcv+0x180/0x1a0) > [ 44.515645] [] (ip_sublist_rcv) from [] (ip_list_r= cv+0xf0/0x110) > [ 44.523391] [] (ip_list_rcv) from [] (__netif_rece= ive_skb_list_core+0x194/0x1fc) > [ 44.532527] [] (__netif_receive_skb_list_core) from [] (netif_receive_skb_list_internal+0x1ec/0x304) > [ 44.543394] [] (netif_receive_skb_list_internal) from [] (napi_complete_done+0x130/0x1c8) > [ 44.553306] [] (napi_complete_done) from [] (stmma= c_napi_poll_rx+0xa7c/0xbd8) > [ 44.562182] [] (stmmac_napi_poll_rx) from [] (__na= pi_poll+0x28/0x170) > [ 44.570359] [] (__napi_poll) from [] (net_rx_actio= n+0xf0/0x278) > [ 44.578016] [] (net_rx_action) from [] (__do_softi= rq+0x104/0x29c) > [ 44.585848] [] (__do_softirq) from [] (irq_exit+0x= bc/0x100) > [ 44.593162] [] (irq_exit) from [] (handle_domain_i= rq+0x60/0x78) > [ 44.600824] [] (handle_domain_irq) from [] (gic_ha= ndle_irq+0x7c/0x90) > [ 44.609007] [] (gic_handle_irq) from [] (__irq_svc= +0x5c/0x78) > [ 44.616491] Exception stack(0xc1b79e38 to 0xc1b79e80) > [ 44.621541] 9e20: = c1c13c00 c0de5100 > [ 44.629712] 9e40: c1b78000 00000002 c3f40240 ffffe000 ec010000 c1b7800= 0 00000002 00000000 > [ 44.637883] 9e60: c3f40268 c0de5100 c3c369c0 c1b79e88 c0809388 c080b9d= 4 a0000113 ffffffff > [ 44.646050] [] (__irq_svc) from [] (xfrm_replay_ad= vance+0x11c/0x3dc) > [ 44.654143] [] (xfrm_replay_advance) from [] (xfrm= _input+0x4d0/0x1304) > [ 44.662408] [] (xfrm_input) from [] (crypto_finali= ze_request+0x5c/0xc4) > [ 44.670766] [] (crypto_finalize_request) from [] (= sun8i_ce_cipher_run+0x34/0x3c) > [ 44.679900] [] (sun8i_ce_cipher_run) from [] (cryp= to_pump_work+0x1a8/0x330) > [ 44.688600] [] (crypto_pump_work) from [] (kthread= _worker_fn+0xd8/0x220) > [ 44.697045] [] (kthread_worker_fn) from [] (kthrea= d+0x15c/0x180) > [ 44.704793] [] (kthread) from [] (ret_from_fork+0x= 14/0x24) > [ 44.712017] Exception stack(0xc1b79fb0 to 0xc1b79ff8) > [ 44.717066] 9fa0: 00000000 0000000= 0 00000000 00000000 > [ 44.725237] 9fc0: 00000000 00000000 00000000 00000000 00000000 0000000= 0 00000000 00000000 > [ 44.733406] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > [ ID] Interval Transfer Bitrate Retr Cwnd > [ 5] 0.00-1.11 sec 1.11 MBytes 8.39 Mbits/sec 1 1.35 KBytes = = > [ 5] 1.11-2.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 46.438402] sched: RT throttling activated > [ 46.438402] sched: RT throttling activated > [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 1.35 KBytes = = > [ 65.398401] rcu: INFO: rcu_sched self-detected stall on CPU > [ 65.403987] rcu: 0-....: (2068 ticks this GP) idle=3Dafd/1/0x40000004= softirq=3D3056/3057 fqs=3D1050 = > [ 65.412774] (t=3D2102 jiffies g=3D1437 q=3D434) > [ 65.416873] NMI backtrace for cpu 0 > [ 65.420361] CPU: 0 PID: 111 Comm: 1c15000.crypto- Not tainted 5.15.13-= 00003-g14f78eca130d #199 > [ 65.428965] Hardware name: Allwinner sun8i Family > [ 65.433669] [] (unwind_backtrace) from [] (show_st= ack+0x10/0x14) > [ 65.441424] [] (show_stack) from [] (dump_stack_lv= l+0x40/0x4c) > [ 65.448998] [] (dump_stack_lvl) from [] (nmi_cpu_b= acktrace+0xdc/0x110) > [ 65.457267] [] (nmi_cpu_backtrace) from [] (nmi_tr= igger_cpumask_backtrace+0x108/0x168) > [ 65.466918] [] (nmi_trigger_cpumask_backtrace) from [] (rcu_dump_cpu_stacks+0x128/0x15c) > [ 65.476745] [] (rcu_dump_cpu_stacks) from [] (rcu_= sched_clock_irq+0x798/0x950) > [ 65.485707] [] (rcu_sched_clock_irq) from [] (upda= te_process_times+0x9c/0xd0) > [ 65.494581] [] (update_process_times) from [] (tic= k_sched_timer+0x7c/0xf0) > [ 65.503197] [] (tick_sched_timer) from [] (__hrtim= er_run_queues+0x15c/0x218) > [ 65.511984] [] (__hrtimer_run_queues) from [] (hrt= imer_interrupt+0x124/0x2b0) > [ 65.520857] [] (hrtimer_interrupt) from [] (arch_t= imer_handler_phys+0x28/0x30) > [ 65.529820] [] (arch_timer_handler_phys) from [] (= handle_percpu_devid_irq+0x78/0x13c) > [ 65.539388] [] (handle_percpu_devid_irq) from [] (= handle_domain_irq+0x5c/0x78) > [ 65.548352] [] (handle_domain_irq) from [] (gic_ha= ndle_irq+0x7c/0x90) > [ 65.556533] [] (gic_handle_irq) from [] (__irq_svc= +0x5c/0x78) > [ 65.564017] Exception stack(0xc1b79a30 to 0xc1b79a78) > [ 65.569067] 9a20: 00000000 0000000= 0 000003dc 000003da > [ 65.577238] 9a40: c3f40268 ffffe000 00000000 c1b78000 00000002 0000000= 0 c3f40268 c0de5100 > [ 65.585408] 9a60: c0b496d0 c1b79a80 c016c5b0 c016c538 80000113 ffffffff > [ 65.592015] [] (__irq_svc) from [] (do_raw_spin_lo= ck+0x78/0x120) > [ 65.599759] [] (do_raw_spin_lock) from [] (xfrm_in= put+0x17c/0x1304) > [ 65.607765] [] (xfrm_input) from [] (xfrm4_esp_rcv= +0x44/0x84) > [ 65.615255] [] (xfrm4_esp_rcv) from [] (ip_protoco= l_deliver_rcu+0x2c/0x2b8) > [ 65.623960] [] (ip_protocol_deliver_rcu) from [] (= ip_local_deliver_finish+0x64/0x80) > [ 65.633440] [] (ip_local_deliver_finish) from [] (= ip_sublist_rcv_finish+0x3c/0x50) > [ 65.642747] [] (ip_sublist_rcv_finish) from [] (ip= _sublist_rcv+0x180/0x1a0) > [ 65.651447] [] (ip_sublist_rcv) from [] (ip_list_r= cv+0xf0/0x110) > [ 65.659193] [] (ip_list_rcv) from [] (__netif_rece= ive_skb_list_core+0x194/0x1fc) > [ 65.668329] [] (__netif_receive_skb_list_core) from [] (netif_receive_skb_list_internal+0x1ec/0x304) > [ 65.679196] [] (netif_receive_skb_list_internal) from [] (napi_complete_done+0x130/0x1c8) > [ 65.689108] [] (napi_complete_done) from [] (stmma= c_napi_poll_rx+0xa7c/0xbd8) > [ 65.697982] [] (stmmac_napi_poll_rx) from [] (__na= pi_poll+0x28/0x170) > [ 65.706159] [] (__napi_poll) from [] (net_rx_actio= n+0xf0/0x278) > [ 65.713816] [] (net_rx_action) from [] (__do_softi= rq+0x104/0x29c) > [ 65.721647] [] (__do_softirq) from [] (irq_exit+0x= bc/0x100) > [ 65.728960] [] (irq_exit) from [] (handle_domain_i= rq+0x60/0x78) > [ 65.736620] [] (handle_domain_irq) from [] (gic_ha= ndle_irq+0x7c/0x90) > [ 65.744799] [] (gic_handle_irq) from [] (__irq_svc= +0x5c/0x78) > [ 65.752281] Exception stack(0xc1b79e38 to 0xc1b79e80) > [ 65.757329] 9e20: = c1c13c00 c0de5100 > [ 65.765500] 9e40: c1b78000 00000002 c3f40240 ffffe000 ec010000 c1b7800= 0 00000002 00000000 > [ 65.773671] 9e60: c3f40268 c0de5100 c3c369c0 c1b79e88 c0809388 c080b9d= 4 a0000113 ffffffff > [ 65.781837] [] (__irq_svc) from [] (xfrm_replay_ad= vance+0x11c/0x3dc) > [ 65.789929] [] (xfrm_replay_advance) from [] (xfrm= _input+0x4d0/0x1304) > [ 65.798196] [] (xfrm_input) from [] (crypto_finali= ze_request+0x5c/0xc4) > [ 65.806552] [] (crypto_finalize_request) from [] (= sun8i_ce_cipher_run+0x34/0x3c) > [ 65.815686] [] (sun8i_ce_cipher_run) from [] (cryp= to_pump_work+0x1a8/0x330) > [ 65.824384] [] (crypto_pump_work) from [] (kthread= _worker_fn+0xd8/0x220) > [ 65.832827] [] (kthread_worker_fn) from [] (kthrea= d+0x15c/0x180) > [ 65.840574] [] (kthread) from [] (ret_from_fork+0x= 14/0x24) > [ 65.847799] Exception stack(0xc1b79fb0 to 0xc1b79ff8) > [ 65.852847] 9fa0: 00000000 0000000= 0 00000000 00000000 > [ 65.861017] 9fc0: 00000000 00000000 00000000 00000000 00000000 0000000= 0 00000000 00000000 > [ 65.869186] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > [ 128.458402] rcu: INFO: rcu_sched self-detected stall on CPU > [ 128.463984] rcu: 0-....: (8328 ticks this GP) idle=3Dafd/1/0x40000004= softirq=3D3056/3057 fqs=3D3872 = > [ 128.472769] (t=3D8408 jiffies g=3D1437 q=3D435) > [ 128.476867] NMI backtrace for cpu 0 > [ 128.480354] CPU: 0 PID: 111 Comm: 1c15000.crypto- Not tainted 5.15.13-= 00003-g14f78eca130d #199 > [ 128.488958] Hardware name: Allwinner sun8i Family > [ 128.493660] [] (unwind_backtrace) from [] (show_st= ack+0x10/0x14) > [ 128.501413] [] (show_stack) from [] (dump_stack_lv= l+0x40/0x4c) > [ 128.508985] [] (dump_stack_lvl) from [] (nmi_cpu_b= acktrace+0xdc/0x110) > [ 128.517252] [] (nmi_cpu_backtrace) from [] (nmi_tr= igger_cpumask_backtrace+0x108/0x168) > [ 128.526904] [] (nmi_trigger_cpumask_backtrace) from [] (rcu_dump_cpu_stacks+0x128/0x15c) > [ 128.536730] [] (rcu_dump_cpu_stacks) from [] (rcu_= sched_clock_irq+0x798/0x950) > [ 128.545691] [] (rcu_sched_clock_irq) from [] (upda= te_process_times+0x9c/0xd0) > [ 128.554564] [] (update_process_times) from [] (tic= k_sched_timer+0x7c/0xf0) > [ 128.563179] [] (tick_sched_timer) from [] (__hrtim= er_run_queues+0x15c/0x218) > [ 128.571966] [] (__hrtimer_run_queues) from [] (hrt= imer_interrupt+0x124/0x2b0) > [ 128.580838] [] (hrtimer_interrupt) from [] (arch_t= imer_handler_phys+0x28/0x30) > [ 128.589800] [] (arch_timer_handler_phys) from [] (= handle_percpu_devid_irq+0x78/0x13c) > [ 128.599369] [] (handle_percpu_devid_irq) from [] (= handle_domain_irq+0x5c/0x78) > [ 128.608332] [] (handle_domain_irq) from [] (gic_ha= ndle_irq+0x7c/0x90) > [ 128.616514] [] (gic_handle_irq) from [] (__irq_svc= +0x5c/0x78) > [ 128.623997] Exception stack(0xc1b79a30 to 0xc1b79a78) > [ 128.629047] 9a20: 00000000 0000000= 0 000003dc 000003da > [ 128.637218] 9a40: c3f40268 ffffe000 00000000 c1b78000 00000002 0000000= 0 c3f40268 c0de5100 > [ 128.645388] 9a60: c0b496d0 c1b79a80 c016c5b0 c016c538 80000113 ffffffff > [ 128.651995] [] (__irq_svc) from [] (do_raw_spin_lo= ck+0x78/0x120) > [ 128.659738] [] (do_raw_spin_lock) from [] (xfrm_in= put+0x17c/0x1304) > [ 128.667744] [] (xfrm_input) from [] (xfrm4_esp_rcv= +0x44/0x84) > [ 128.675233] [] (xfrm4_esp_rcv) from [] (ip_protoco= l_deliver_rcu+0x2c/0x2b8) > [ 128.683936] [] (ip_protocol_deliver_rcu) from [] (= ip_local_deliver_finish+0x64/0x80) > [ 128.693408] [] (ip_local_deliver_finish) from [] (= ip_sublist_rcv_finish+0x3c/0x50) > [ 128.702716] [] (ip_sublist_rcv_finish) from [] (ip= _sublist_rcv+0x180/0x1a0) > [ 128.711417] [] (ip_sublist_rcv) from [] (ip_list_r= cv+0xf0/0x110) > [ 128.719163] [] (ip_list_rcv) from [] (__netif_rece= ive_skb_list_core+0x194/0x1fc) > [ 128.728299] [] (__netif_receive_skb_list_core) from [] (netif_receive_skb_list_internal+0x1ec/0x304) > [ 128.739165] [] (netif_receive_skb_list_internal) from [] (napi_complete_done+0x130/0x1c8) > [ 128.749077] [] (napi_complete_done) from [] (stmma= c_napi_poll_rx+0xa7c/0xbd8) > [ 128.757951] [] (stmmac_napi_poll_rx) from [] (__na= pi_poll+0x28/0x170) > [ 128.766129] [] (__napi_poll) from [] (net_rx_actio= n+0xf0/0x278) > [ 128.773786] [] (net_rx_action) from [] (__do_softi= rq+0x104/0x29c) > [ 128.781617] [] (__do_softirq) from [] (irq_exit+0x= bc/0x100) > [ 128.788930] [] (irq_exit) from [] (handle_domain_i= rq+0x60/0x78) > [ 128.796589] [] (handle_domain_irq) from [] (gic_ha= ndle_irq+0x7c/0x90) > [ 128.804768] [] (gic_handle_irq) from [] (__irq_svc= +0x5c/0x78) > [ 128.812250] Exception stack(0xc1b79e38 to 0xc1b79e80) > [ 128.817298] 9e20: = c1c13c00 c0de5100 > [ 128.825469] 9e40: c1b78000 00000002 c3f40240 ffffe000 ec010000 c1b7800= 0 00000002 00000000 > [ 128.833640] 9e60: c3f40268 c0de5100 c3c369c0 c1b79e88 c0809388 c080b9d= 4 a0000113 ffffffff > [ 128.841806] [] (__irq_svc) from [] (xfrm_replay_ad= vance+0x11c/0x3dc) > [ 128.849898] [] (xfrm_replay_advance) from [] (xfrm= _input+0x4d0/0x1304) > [ 128.858163] [] (xfrm_input) from [] (crypto_finali= ze_request+0x5c/0xc4) > [ 128.866518] [] (crypto_finalize_request) from [] (= sun8i_ce_cipher_run+0x34/0x3c) > [ 128.875651] [] (sun8i_ce_cipher_run) from [] (cryp= to_pump_work+0x1a8/0x330) > [ 128.884349] [] (crypto_pump_work) from [] (kthread= _worker_fn+0xd8/0x220) > [ 128.892790] [] (kthread_worker_fn) from [] (kthrea= d+0x15c/0x180) > [ 128.900537] [] (kthread) from [] (ret_from_fork+0x= 14/0x24) > [ 128.907760] Exception stack(0xc1b79fb0 to 0xc1b79ff8) > [ 128.912809] 9fa0: 00000000 0000000= 0 00000000 00000000 > [ 128.920978] 9fc0: 00000000 00000000 00000000 00000000 00000000 0000000= 0 00000000 00000000 > [ 128.929147] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > = > None on my drivers are using spinlock so the only one which remains is cr= ypto/engine one (queue_lock) > = > I have started to bisect this, but this lead to unrelated commits. (I tri= ed twice with different starting point) > = > Known broken version are > 5.13 5.14 5.15.13 next-20220105 5.10 > I randomly hit also "NOHZ tick-stop error: Non-RCU local softirq work is = pending, handler #08!!!". > On 5.6 and 5.7 I got: NOHZ: local_softirq_pending 08 > = > Working version are 5.4.170 and 5.5 > = > The normal iperf performance is at least 30Mbit/s, when bug is present, i= perf never end. > But When the bug is present with "NOHZ: local_softirq_pending 08", iperf = give 1Mbit/s. > = > I started also to test other crypto driver which use the crypto/crypto_en= gine > On my sun8i-ss, no kernel trace are given, but iperf performance decrease= is present. > I also tested omap-aes and it seems un-impacted. > = > Any help to debug that is welcome. > Thanks in advance > = > Regards > = _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel