From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 858633F8ECC for ; Tue, 9 Jun 2026 10:47:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.49 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781002059; cv=none; b=NsBxJazbOKft4MmcV46vkI1OLlOiAE8MgcLQdjnCcUuCSlrwy9+0CziBRItsVODpBzmGcBF+nTlNRmvrJIOPZ0pGshXah8+l+Ts9Vm0yRFIhgVhywR7IVwKVj1XoIywwgGzFQdTrgiAuRXDQE10MeVn/uy2X9c3wAfeC0G/catA= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1781002059; c=relaxed/simple; bh=5BU7BTvCm2MLAN3ZJE6HJEn8mf1z0MvGNLlbq+J5GtQ=; h=From:To:Cc:Subject:Date:Message-ID:MIME-Version; b=Qy682qS1dSH+aRIkF6XMj9gEhinr7f46c4AA6Ip9Sevl6SERGBvkd/ifmNUMDp339CvOXOMmGahi7cVhOzmDEm4xmZrzVnufuE8ntWqu4Bb3u6vr7O+GA49H1/hwjf9XYxTtdvDdZSmxacmKhbuHBVw8v7gl/TUSjdCwL6nON4g= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=readmodwrite.com; spf=none smtp.mailfrom=readmodwrite.com; dkim=pass (2048-bit key) header.d=readmodwrite-com.20251104.gappssmtp.com header.i=@readmodwrite-com.20251104.gappssmtp.com header.b=TxVYvbcT; arc=none smtp.client-ip=209.85.128.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=readmodwrite.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=readmodwrite.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=readmodwrite-com.20251104.gappssmtp.com header.i=@readmodwrite-com.20251104.gappssmtp.com header.b="TxVYvbcT" Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-490af320e2aso58923165e9.2 for ; Tue, 09 Jun 2026 03:47:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=readmodwrite-com.20251104.gappssmtp.com; s=20251104; t=1781002056; x=1781606856; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=K5aAcU4MtL73dOMrhKLE4+sZ4BDx51H3GnN5hUoWvrE=; b=TxVYvbcTNrHM95AVW5mFxqYF2LkPX5jzr8cZd7CTBQxnSKgJBPNwhiao9Nh8Y85DKb MV9775Lnt7/o/zR24JNuE8Jj2LzVAATlEK2T264gbl19/FWXa5muBXfsR3kaeHWzbb/J J+YrlD68LP+TOXkGcLV7bp0gPsMsEzqTsXZ4/tn3ibpHxu60+9/SZhcp/ELvvRvE9qdk z+c+9zsWkiFLALPC8OIeSVHfFHvXRY/08mALKb6tAoEA9PCsaN/TD5W76N9nSrHgZQzw NhvzXMduprELAHRcS6ETXEtO7ivk6xKYUMjm6xVzO36HfIITFayDCn00XFPXs8ayayyQ 34GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1781002056; x=1781606856; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=K5aAcU4MtL73dOMrhKLE4+sZ4BDx51H3GnN5hUoWvrE=; b=fwfzJhl4eOH4h9bGFdSOqtaEefmd4JsjhwCCF8WnhaMvi9+8tFZFvga83r3LhIJClp Mk2zKQrZiU4nVHZYzA+gwxuheQtcrbtr6Lv0IjrwMjh2oivC1RigDnTmADcNXYpbSCpV c35OLL5MFVrfUnu9UO7sbDV1X5Hr/zxbymLZcXZx4lN6DlNYLjgEPenqj3UbJkNaeMzU bSHSPP6xtIYzabYLElC8nlXqgOIRXIhw74vm9DWcxG7bIu3B8jqnFqM9QI7AQaRfVQ8j duFh0t2POhz2SgL2zipo0pbOBcVkn5slxtrcr2kTnd8w7qwI3iXE5rmspIAsX5M6Y52t ZuoQ== X-Forwarded-Encrypted: i=1; AFNElJ/old4LWxJ9CjJ8qe5L3uaOI8WTuFXziLppuIbdSuSnUy4FRa66yewkwbuRiopR/ebHvruCmGmhD/JTB2M=@vger.kernel.org X-Gm-Message-State: AOJu0YzOzIpttn8VofSt/P7FW9eJr0EYuSt+wOYelMEsN8upCjWZFAzZ KPbHAlSczTB72WWhOIHJCbkBrdFi0PXLHlNVJsR2RpwmKh/x4L755pgCZpCFPSMBM6M= X-Gm-Gg: Acq92OHWU3Y8+jgZ7zZGUkY0EOSpr13Xk7fQGfeQeHw/KfQu9D1LTQRfkK/+qgMCbKU y3g0Yk87FeyoCaLY8JYCpPO+zEcaHcXok8KYEpJCZAqXcYKtspqbBTW9dQ5XK/WzbwFhixmVlnc jfni2Nt8//vKl5zAtnLkVRb2QqW4lmdCPYV0888jsaoAoiOxm1+LhbdXVkv+kiSrLVDybD6ZwJx M4BBYKbire3zDis1wEwec0Aw8svZNJHDOY2kS5odSdzALiFjAaWfvq3MIOwbkqSs+zHik4ULAsv x9PJntjFteL+JH7LvjVHddc8FpFqnKMIep02271mooXypfcn/ymsKOPB7J5ECWOC8e0LxIItdTb sLGV/jV/2iVLaODynW5klhkl2dSjqjim6j8S7hNIAi5DSvhd2j93v0mKotHZOsJBD9njgTHFY/O /t0+xs9GM+ar3UKEgr91Wkbl7b57ORGTfCuy7zy1Y= X-Received: by 2002:a05:600c:3e15:b0:490:adb6:793d with SMTP id 5b1f17b1804b1-490c25f67d9mr319951875e9.26.1781002055673; Tue, 09 Jun 2026 03:47:35 -0700 (PDT) Received: from matt-Precision-5490.. ([2a09:bac6:37a8:ebe::178:1cb]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-490bc3a87dasm457413665e9.7.2026.06.09.03.47.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Jun 2026 03:47:34 -0700 (PDT) From: Matt Fleming X-Google-Original-From: Matt Fleming To: Tejun Heo , Andrea Righi Cc: "Paul E . McKenney" , sched-ext@lists.linux.dev, linux-kernel@vger.kernel.org, kernel-team@cloudflare.com Subject: sched_ext/lavd hard lockup in old call_rcu_tasks_generic needadjust path Date: Tue, 9 Jun 2026 11:47:33 +0100 Message-ID: <20260609104733.1184001-1-mfleming@cloudflare.com> X-Mailer: git-send-email 2.43.0 Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Hi there, We're investigating a hard lockup on a 6.18.33-based kernel with scx_lavd running. The vmcore shows CPU#67 stuck in: native_queued_spin_lock_slowpath _raw_spin_lock task_rq_lock sched_ext_free __put_task_struct rcu_core handle_softirqs irq_exit_rcu sysvec_apic_timer_interrupt The rq lock being waited on is for CPU#66. Another CPU in the same dump is in: sched_ext_free scx_exit_task bpf_task_storage_delete call_rcu_tasks_generic+547 _printk console_unlock wake_up_q try_to_wake_up __task_rq_lock The `call_rcu_tasks_generic+547` site is the old needadjust path that prints: Switching RCU Tasks Trace to per-CPU callback queuing. So the current theory is that a task teardown under rq lock enters bpf_task_storage_delete(), hits the callback-queue adjustment printk, then console wakeup tries to acquire the same rq lock again. We found related upstream changes: 3063b33a347c ("rcu-tasks: Avoid raw-spinlocked wakeups from call_rcu_tasks_generic()") d245698d727a ("cgroup: Defer task cgroup unlink until after the task is done switching out") 7900aa699c34 ("sched_ext: Fix cgroup exit ordering by moving sched_ext_free() to finish_task_switch()") 7c405fb3279b ("rcu: Use an intermediate irq_work to start process_srcu()") But none appears to directly fix the old 6.18 needadjust printk path. Would backporting d245698d727a and 7900aa699c34 be useful, or should the needadjust printk path itself be deferred away from rq-locked callers? Thanks, Matt