From: Leon Romanovsky <leon@kernel.org>
To: Yacan Liu <liuyacan@corp.netease.com>
Cc: jgg@ziepe.ca, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: An issue of kernel stall caused by infiniband mad packet
Date: Wed, 22 Feb 2023 20:11:48 +0200 [thread overview]
Message-ID: <Y/Za5NUcROu4Srrj@unreal> (raw)
In-Reply-To: <4f5e51dc-6116-aad6-de8b-7a1bf7c17656@corp.netease.com>
On Wed, Feb 22, 2023 at 08:40:52PM +0800, Yacan Liu wrote:
> Hi folks:
>
> We recently encountered a kernel stall issue related to infiniband mad
> packet. We believe this is caused by some abnormal operations of user-level
> programs.But I am not sure if the kernel driver needs to do some protection.
>
> In our cluster, multiple nodes will establish multiple RDMA connections with
> each other. Then, when we start the program, the kernel prompts the
> following error after one minute:
>
> [ 5489.530125] rcu: INFO: rcu_sched self-detected stall on CPU
> [ 5489.535714] rcu: 55-....: (19027 ticks this GP)
> idle=236/1/0x4000000000000000 softirq=13510/13510 fqs=4359
> [ 5489.545554] (t=21015 jiffies g=279845 q=701559)
> [ 5489.545557] NMI backtrace for cpu 55
> [ 5489.545559] CPU: 55 PID: 44979 Comm: kworker/u196:1 Tainted: G OE
> 5.10.91-netease3-1 #2
> [ 5489.545560] Hardware name: Inspur SA5212M5/YZMB-00882-10T, BIOS 4.1.21
> 08/25/2021
> [ 5489.545582] Workqueue: ib-comp-unb-wq ib_cq_poll_work [ib_core]
> [ 5489.545584] Call Trace:
> [ 5489.545586]
> [ 5489.545590] dump_stack+0x6b/0x83
> [ 5489.545593] nmi_cpu_backtrace.cold+0x32/0x69
> [ 5489.545596] ? lapic_can_unplug_cpu+0x80/0x80
> [ 5489.545599] nmi_trigger_cpumask_backtrace+0xd7/0xe0
> [ 5489.545602] rcu_dump_cpu_stacks+0xa2/0xd0
> [ 5489.545604] rcu_sched_clock_irq.cold+0x1ff/0x3d6
> [ 5489.545607] ? trigger_load_balance+0x5a/0x240
> [ 5489.545610] update_process_times+0x8c/0xc0
> [ 5489.545613] tick_sched_handle+0x22/0x60
> [ 5489.545615] tick_sched_timer+0x7c/0xb0
> [ 5489.545617] ? tick_do_update_jiffies64.part.0+0xc0/0xc0
> [ 5489.545619] __hrtimer_run_queues+0x12a/0x270
> [ 5489.545621] hrtimer_interrupt+0x110/0x2c0
> [ 5489.545623] __sysvec_apic_timer_interrupt+0x5f/0xd0
> [ 5489.545627] asm_call_irq_on_stack+0xf/0x20
> [ 5489.545627]
> [ 5489.545629] sysvec_apic_timer_interrupt+0x72/0x80
> [ 5489.545631] asm_sysvec_apic_timer_interrupt+0x12/0x20
>
> OS: Debian 11
> Kernel: 5.10.91
> ofed: MLNX_OFED_LINUX-5.5-1.0.3.2
Please approach your NVidia support representative. There are some
differences between upstream and MOFED code in MAD area.
Thanks
prev parent reply other threads:[~2023-02-22 18:11 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-22 12:40 An issue of kernel stall caused by infiniband mad packet Yacan Liu
2023-02-22 18:11 ` Leon Romanovsky [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y/Za5NUcROu4Srrj@unreal \
--to=leon@kernel.org \
--cc=jgg@ziepe.ca \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=liuyacan@corp.netease.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.