From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out-179.mta0.migadu.com (out-179.mta0.migadu.com [91.218.175.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9F7081C01 for ; Thu, 28 Aug 2025 02:40:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=91.218.175.179 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756348859; cv=none; b=m9Mm1u5Qdwvmu51wcMtp3rDflTi5kWvAl05igl4CnCr7TZZ/E7uwE9NdOmhZhDNbPv5jYK4NGA/7x8CrX4ZS8Cr/0hqOg6gspSb1cmWb+yRWB9qTor1cy5NcyVkQ829MLakBXl4PnvaKJiiRafMu8iBeGfbUntKhGOYwk3dLTVE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1756348859; c=relaxed/simple; bh=YUnBOV0DRlWVcBC/QtoOy0ZWvZB0grZmo4YXjOzoDH4=; h=Message-ID:Date:MIME-Version:Subject:To:Cc:References:From: In-Reply-To:Content-Type; b=HSCHtlhMccpgfnicceWQGh67AMDJvuGkTdS+i6+v620yMKW9bj6wbcoucBonEM4iiFFD7pArZXj77dJcs8wKKyM/8LErrWWpXx9nPx6xoUHSLF9BMWgbdXUd3NphKLOT5B/Fj2/Dk30mKHcPCke1ED0VRQov+wpTb8s89de+wL8= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev; spf=pass smtp.mailfrom=linux.dev; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b=sSWH2+Y8; arc=none smtp.client-ip=91.218.175.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=linux.dev Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=linux.dev Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="sSWH2+Y8" Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1756348854; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HSzIAxQjIPhumyUd628BNPGzi5Ifl6qfzdJVclskNWo=; b=sSWH2+Y89Sm08DUMkjrMZaBSEVTDFUgl6aZHg+OOf5M4VfsyxQ2qbAFMJBorFxxptJAJUA N+5m2Czl2ETEyuR0aNaVuo+IJbDT4UDVZGdd9zEhvMrejc+HtGKmHRRhl/twR46DF58Hnw VL/7AaPBqGinEbipvV8+0STYe0Mvcb8= Date: Thu, 28 Aug 2025 10:40:47 +0800 Precedence: bulk X-Mailing-List: bpf@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Subject: Re: [BUG] Deadlock triggered by bpfsnoop funcgraph feature Content-Language: en-US To: Alexei Starovoitov , "Paul E. McKenney" Cc: bpf , Alexei Starovoitov , Andrii Nakryiko , Daniel Borkmann , Jiri Olsa References: <8dcc144e-3142-4e0d-a852-155781e41eb4@linux.dev> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Leon Hwang In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT On 28/8/25 08:42, Alexei Starovoitov wrote: > On Tue, Aug 26, 2025 at 7:58 PM Leon Hwang wrote: >> >> >> >> On 27/8/25 10:23, Alexei Starovoitov wrote: >>> On Tue, Aug 26, 2025 at 7:13 PM Leon Hwang wrote: >>>> >>>> Hi, >>>> >>>> I’ve encountered a reproducible deadlock while developing the funcgraph >>>> feature for bpfsnoop [0]. >>> >>> debug it pls. >> >> It’s quite difficult for me. I’ve tried debugging it but didn’t succeed. >> >>> Sounds like you're implying that the root cause is in bpf, >>> but why do you think so? >>> >>> You're attaching to things that shouldn't be attached to. >>> Like rcu_lockdep_current_cpu_online() >>> so effectively you're recursing in that lockdep code. >>> See big lock there. It will dead lock for sure. >> >> If a function that acquires a lock can be traced by a tracing program, >> bpfsnoop’s funcgraph will attempt to trace it as well. In such cases, a >> deadlock is highly likely to occur. >> >> With bpfsnoop I try my best to avoid such deadlock issues. But what >> about other bpf tracing tools? If they don’t handle this properly, the >> kernel is very likely to crash. > > bpf infra is trying hard not to crash it, but debug kernel is a different > category. rcu_read_lock_held() doesn't exist in production kernels. > You can propose adding "notrace" for it, but in general that doesn't scale. > Same with rcu_lockdep_current_cpu_online(). > It probably deserves "notrace" too. Indeed, it doesn't scale. When I run ./bpfsnoop -k "htab_*_elem" --output-fgraph --fgraph-debug --fgraph-exclude 'rcu_read_lock_*held,rcu_lockdep_current_cpu_online,*raw_spin_*lock*,kvfree,show_stack,put_task_stack', the kernel doesn’t panic, but the OS eventually stalls and becomes unresponsive to key presses. It seems preferable to avoid running BPF programs continuously in such cases. Thanks, Leon