From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932297AbbHGOCh (ORCPT ); Fri, 7 Aug 2015 10:02:37 -0400 Received: from mail.kernel.org ([198.145.29.136]:58498 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S932229AbbHGOBZ (ORCPT ); Fri, 7 Aug 2015 10:01:25 -0400 Message-Id: <20150807140121.618156144@goodmis.org> User-Agent: quilt/0.61-1 Date: Fri, 07 Aug 2015 10:01:13 -0400 From: Steven Rostedt To: linux-kernel@vger.kernel.org, linux-rt-users Cc: Thomas Gleixner , Carsten Emde , Sebastian Andrzej Siewior , John Kacur , Paul Gortmaker , , Dave Chinner Subject: [PATCH RT 4/6] xfs: Disable percpu SB on PREEMPT_RT_FULL References: <20150807140109.299360883@goodmis.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15 Content-Disposition: inline; filename=0004-xfs-Disable-percpu-SB-on-PREEMPT_RT_FULL.patch Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org 3.12.44-rt62-rc1 stable review patch. If anyone has any objections, please let me know. ------------------ From: Steven Rostedt Running a test on a large CPU count box with xfs, I hit a live lock with the following backtraces on several CPUs: Call Trace: [] __const_udelay+0x28/0x30 [] xfs_icsb_lock_cntr+0x2a/0x40 [xfs] [] xfs_icsb_modify_counters+0x71/0x280 [xfs] [] xfs_trans_reserve+0x171/0x210 [xfs] [] xfs_create+0x24d/0x6f0 [xfs] [] ? avc_has_perm_flags+0xfb/0x1e0 [] xfs_vn_mknod+0xbb/0x1e0 [xfs] [] xfs_vn_create+0x13/0x20 [xfs] [] vfs_create+0xcd/0x130 [] do_last+0xb8f/0x1240 [] path_openat+0xc2/0x490 Looking at the code I see it was stuck at: STATIC void xfs_icsb_lock_cntr( xfs_icsb_cnts_t *icsbp) { while (test_and_set_bit(XFS_ICSB_FLAG_LOCK, &icsbp->icsb_flags)) { ndelay(1000); } } In xfs_icsb_modify_counters() the code is fine. There's a preempt_disable() called when taking this bit spinlock and a preempt_enable() after it is released. The issue is that not all locations are protected by preempt_disable() when PREEMPT_RT is set. Namely the places that grab all CPU cntr locks. STATIC void xfs_icsb_lock_all_counters( xfs_mount_t *mp) { xfs_icsb_cnts_t *cntp; int i; for_each_online_cpu(i) { cntp = (xfs_icsb_cnts_t *)per_cpu_ptr(mp->m_sb_cnts, i); xfs_icsb_lock_cntr(cntp); } } STATIC void xfs_icsb_disable_counter() { [...] xfs_icsb_lock_all_counters(mp); [...] xfs_icsb_unlock_all_counters(mp); } STATIC void xfs_icsb_balance_counter_locked() { [...] xfs_icsb_disable_counter(); [...] } STATIC void xfs_icsb_balance_counter( xfs_mount_t *mp, xfs_sb_field_t fields, int min_per_cpu) { spin_lock(&mp->m_sb_lock); xfs_icsb_balance_counter_locked(mp, fields, min_per_cpu); spin_unlock(&mp->m_sb_lock); } Now, when PREEMPT_RT is not enabled, that spin_lock() disables preemption. But for PREEMPT_RT, it does not. Although with my test box I was not able to produce a task state of all tasks, but I'm assuming that some task called the xfs_icsb_lock_all_counters() and was preempted by an RT task and could not finish, causing all callers of that lock to block indefinitely. Dave Chinner has stated that the scalability of that code will probably be negated by PREEMPT_RT, and that it is probably best to just disable the code in question. Also, this code has been rewritten in newer kernels. Link: http://lkml.kernel.org/r/20150504004844.GA21261@dastard Cc: stable-rt@vger.kernel.org Suggested-by: Dave Chinner Signed-off-by: Steven Rostedt --- fs/xfs/xfs_linux.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/xfs/xfs_linux.h b/fs/xfs/xfs_linux.h index f9bb590acc0e..8483f8cf432a 100644 --- a/fs/xfs/xfs_linux.h +++ b/fs/xfs/xfs_linux.h @@ -131,7 +131,7 @@ typedef __uint64_t __psunsigned_t; /* * Feature macros (disable/enable) */ -#ifdef CONFIG_SMP +#if defined(CONFIG_SMP) && !defined(CONFIG_PREEMPT_RT_FULL) #define HAVE_PERCPU_SB /* per cpu superblock counters are a 2.6 feature */ #else #undef HAVE_PERCPU_SB /* per cpu superblock counters are a 2.6 feature */ -- 2.1.4