From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF7B8EE49AF for ; Sat, 19 Aug 2023 06:18:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245456AbjHSGKN (ORCPT ); Sat, 19 Aug 2023 02:10:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245241AbjHSGJi (ORCPT ); Sat, 19 Aug 2023 02:09:38 -0400 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48DE6420E for ; Fri, 18 Aug 2023 23:09:37 -0700 (PDT) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-d672f55d48dso2316771276.2 for ; Fri, 18 Aug 2023 23:09:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1692425376; x=1693030176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xLEkCfvmRAN107gaM4JSoAwXnog1kAUr56Dto8cWxno=; b=hbjoTKbrlGc4RQScBJIOnFpMszxl41iAM6V6igHiwztP2qf+hRtTVXVrBybmLjfIa9 GHD1qKkl/MYblXDTD8V5c7OOxPrLXxeZt878LO0ZynQjtHhzwUixowMe18BEbU/Rj18l usZ8+20HkPgJn5i4k20lVrBeeddOPqm4H5Wc3RwGWjKpvahIwcNMOgxdkg96WPec6XlT W0YqNnyhnOiwh7g6wm1duFHHK4Bd6v2rIHUR71DNpuCaJb+PpDsswiYK+GCauVFYcNQ3 Ou9cETFTiohjFtvJQSfpHIdmwKBLpOC2opsyy0RY9iHA0g8ZvcpN8Ncjp7pM7OtEzYrz +jkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692425376; x=1693030176; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xLEkCfvmRAN107gaM4JSoAwXnog1kAUr56Dto8cWxno=; b=JAytvDG9T9SAWQHT0fzySL2x7YeW/MUAR99CA0AveqLwWstWp50eBcvgoDSj+yWplK 6A8iO+Vz+27gf8U9ChptKKbUmWewSimWwsAb5qLlNZ9fjUQHzuQN106NrJhGCwcRmRFL tKmA6b/lMKn40h7y65wTTOYGTzMzhGrACOYRuVTNSyPWDrNayZjhXllHugybIUFDIXHc EvvLCiVbrA18vhwHnLtdEWxtGRhNA9o8i956OuJrlP/FO3uu38XNttpeJkm++VsCmfOC r9c9l7K5gnfDrMLV2smiJpLwYm413JcQn1teWtOqZ1Lfx9JHnVdHhXSp1tjH2h6L4gHB 4Wjg== X-Gm-Message-State: AOJu0YyNXt9clLPwjk31vI8bLib2aeTYq2BWZZS01aCFq2rOh21aV3bz My89Uc0l5P2PrsuT8bkDKcNlXf5OGsGH+9w8hYYTl8i5iCk+2tDsWMqm7k+BZ3MUcV9UMpuC/EZ DmCntzbZ04Hy51IYu1+89eB1Co5ynmM5VfSNZ1xnhMMkfCvBk9VIeAG3EKnFL3pFUJpggeBk= X-Google-Smtp-Source: AGHT+IHyrq/h1ke1Rd+OfQXPO7gX0ISqxfIoU/Z4bjSP41hXov+553fyGTAJVmC2jSFIkS7vcN80e61t384z X-Received: from jstultz-noogler2.c.googlers.com ([fda3:e722:ac3:cc00:24:72f4:c0a8:600]) (user=jstultz job=sendgmr) by 2002:a25:d851:0:b0:d0b:d8cd:e661 with SMTP id p78-20020a25d851000000b00d0bd8cde661mr8991ybg.12.1692425376156; Fri, 18 Aug 2023 23:09:36 -0700 (PDT) Date: Sat, 19 Aug 2023 06:08:39 +0000 In-Reply-To: <20230819060915.3001568-1-jstultz@google.com> Mime-Version: 1.0 References: <20230819060915.3001568-1-jstultz@google.com> X-Mailer: git-send-email 2.42.0.rc1.204.g551eb34607-goog Message-ID: <20230819060915.3001568-6-jstultz@google.com> Subject: [PATCH v5 05/19] locking/mutex: Rework task_struct::blocked_on From: John Stultz To: LKML Cc: Peter Zijlstra , Joel Fernandes , Qais Yousef , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Valentin Schneider , Steven Rostedt , Ben Segall , Zimuzo Ezeozue , Youssef Esmat , Mel Gorman , Daniel Bristot de Oliveira , Will Deacon , Waiman Long , Boqun Feng , "Paul E . McKenney" , kernel-team@android.com, "Connor O'Brien" , John Stultz Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org From: Peter Zijlstra Track the blocked-on relation for mutexes, this allows following this relation at schedule time. task | blocked-on v mutex | owner v task Cc: Joel Fernandes Cc: Qais Yousef Cc: Ingo Molnar Cc: Peter Zijlstra Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Valentin Schneider Cc: Steven Rostedt Cc: Ben Segall Cc: Zimuzo Ezeozue Cc: Youssef Esmat Cc: Mel Gorman Cc: Daniel Bristot de Oliveira Cc: Will Deacon Cc: Waiman Long Cc: Boqun Feng Cc: "Paul E . McKenney" Cc: kernel-team@android.com Signed-off-by: Peter Zijlstra (Intel) [minor changes while rebasing] Signed-off-by: Juri Lelli Signed-off-by: Peter Zijlstra (Intel) Signed-off-by: Connor O'Brien [jstultz: Fix blocked_on tracking in __mutex_lock_common in error paths] Signed-off-by: John Stultz --- v2: * Fixed blocked_on tracking in error paths that was causing crashes v4: * Ensure we clear blocked_on when waking ww_mutexes to die or wound. This is critical so we don't get ciruclar blocked_on relationships that can't be resolved. v5: * Fix potential bug where the skip_wait path might clear blocked_on when that path never set it * Slight tweaks to where we set blocked_on to make it consistent, along with extra WARN_ON correctness checking * Minor comment changes --- include/linux/sched.h | 5 +---- kernel/fork.c | 3 +-- kernel/locking/mutex-debug.c | 9 +++++---- kernel/locking/mutex.c | 10 ++++++++++ kernel/locking/ww_mutex.h | 17 +++++++++++++++-- 5 files changed, 32 insertions(+), 12 deletions(-) diff --git a/include/linux/sched.h b/include/linux/sched.h index 4bc8a4912539..8b98e3933bd9 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1140,10 +1140,7 @@ struct task_struct { struct rt_mutex_waiter *pi_blocked_on; #endif -#ifdef CONFIG_DEBUG_MUTEXES - /* Mutex deadlock detection: */ - struct mutex_waiter *blocked_on; -#endif + struct mutex *blocked_on; /* lock we're blocked on */ #ifdef CONFIG_DEBUG_ATOMIC_SLEEP int non_block_count; diff --git a/kernel/fork.c b/kernel/fork.c index d2e12b6d2b18..42b022fc3c4c 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2458,9 +2458,8 @@ __latent_entropy struct task_struct *copy_process( lockdep_init_task(p); #endif -#ifdef CONFIG_DEBUG_MUTEXES p->blocked_on = NULL; /* not blocked yet */ -#endif + #ifdef CONFIG_BCACHE p->sequential_io = 0; p->sequential_io_avg = 0; diff --git a/kernel/locking/mutex-debug.c b/kernel/locking/mutex-debug.c index bc8abb8549d2..7228909c3e62 100644 --- a/kernel/locking/mutex-debug.c +++ b/kernel/locking/mutex-debug.c @@ -52,17 +52,18 @@ void debug_mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter, { lockdep_assert_held(&lock->wait_lock); - /* Mark the current thread as blocked on the lock: */ - task->blocked_on = waiter; + /* Current thread can't be already blocked (since it's executing!) */ + DEBUG_LOCKS_WARN_ON(task->blocked_on); } void debug_mutex_remove_waiter(struct mutex *lock, struct mutex_waiter *waiter, struct task_struct *task) { + struct mutex *blocked_on = READ_ONCE(task->blocked_on); + DEBUG_LOCKS_WARN_ON(list_empty(&waiter->list)); DEBUG_LOCKS_WARN_ON(waiter->task != task); - DEBUG_LOCKS_WARN_ON(task->blocked_on != waiter); - task->blocked_on = NULL; + DEBUG_LOCKS_WARN_ON(blocked_on && blocked_on != lock); INIT_LIST_HEAD(&waiter->list); waiter->task = NULL; diff --git a/kernel/locking/mutex.c b/kernel/locking/mutex.c index 67ff08beebd6..8af9269ce8d9 100644 --- a/kernel/locking/mutex.c +++ b/kernel/locking/mutex.c @@ -622,6 +622,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas goto err_early_kill; } + current->blocked_on = lock; set_current_state(state); trace_contention_begin(lock, LCB_F_MUTEX); for (;;) { @@ -661,6 +662,10 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas first = __mutex_waiter_is_first(lock, &waiter); + /* + * Gets reset by unlock path(). + */ + current->blocked_on = lock; set_current_state(state); /* * Here we order against unlock; we must either see it change @@ -681,6 +686,7 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas } raw_spin_lock_irqsave(&lock->wait_lock, flags); acquired: + current->blocked_on = NULL; __set_current_state(TASK_RUNNING); if (ww_ctx) { @@ -711,9 +717,11 @@ __mutex_lock_common(struct mutex *lock, unsigned int state, unsigned int subclas return 0; err: + current->blocked_on = NULL; __set_current_state(TASK_RUNNING); __mutex_remove_waiter(lock, &waiter); err_early_kill: + WARN_ON(current->blocked_on); trace_contention_end(lock, ret); raw_spin_unlock_irqrestore(&lock->wait_lock, flags); debug_mutex_free_waiter(&waiter); @@ -924,6 +932,8 @@ static noinline void __sched __mutex_unlock_slowpath(struct mutex *lock, unsigne next = waiter->task; debug_mutex_wake_waiter(lock, waiter); + WARN_ON(next->blocked_on != lock); + next->blocked_on = NULL; wake_q_add(&wake_q, next); } diff --git a/kernel/locking/ww_mutex.h b/kernel/locking/ww_mutex.h index 8b94f4b89e74..8bb334491732 100644 --- a/kernel/locking/ww_mutex.h +++ b/kernel/locking/ww_mutex.h @@ -284,6 +284,13 @@ __ww_mutex_die(struct MUTEX *lock, struct MUTEX_WAITER *waiter, #ifndef WW_RT debug_mutex_wake_waiter(lock, waiter); #endif + /* + * When waking up the task to die, be sure to clear the + * blocked_on pointer. Otherwise we can see circular + * blocked_on relationships that can't resolve. + */ + WARN_ON(waiter->task->blocked_on != lock); + waiter->task->blocked_on = NULL; wake_q_add(wake_q, waiter->task); } @@ -331,9 +338,15 @@ static bool __ww_mutex_wound(struct MUTEX *lock, * it's wounded in __ww_mutex_check_kill() or has a * wakeup pending to re-read the wounded state. */ - if (owner != current) + if (owner != current) { + /* + * When waking up the task to wound, be sure to clear the + * blocked_on pointer. Otherwise we can see circular + * blocked_on relationships that can't resolve. + */ + owner->blocked_on = NULL; wake_q_add(wake_q, owner); - + } return true; } -- 2.42.0.rc1.204.g551eb34607-goog