From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2993914AbdDYWhV (ORCPT ); Tue, 25 Apr 2017 18:37:21 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:55128 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1952814AbdDYWgq (ORCPT ); Tue, 25 Apr 2017 18:36:46 -0400 Date: Tue, 25 Apr 2017 15:36:42 -0700 From: "Paul E. McKenney" To: Mike Galbraith Cc: LKML Subject: Re: TREE_SRCU slows hotplug by factor ~16 Reply-To: paulmck@linux.vnet.ibm.com References: <1493002089.4145.7.camel@gmx.de> <20170424033250.GX3956@linux.vnet.ibm.com> <1493011484.4166.13.camel@gmx.de> <20170424062220.GZ3956@linux.vnet.ibm.com> <1493019303.4617.6.camel@gmx.de> <20170424162442.GI3956@linux.vnet.ibm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170424162442.GI3956@linux.vnet.ibm.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17042522-0052-0000-0000-000001E65D55 X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00006973; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000208; SDB=6.00852441; UDB=6.00421361; IPR=6.00631279; BA=6.00005312; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015176; XFM=3.00000013; UTC=2017-04-25 22:36:43 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17042522-0053-0000-0000-0000503D1D84 Message-Id: <20170425223642.GA10931@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-04-25_16:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1704250383 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, Apr 24, 2017 at 09:24:42AM -0700, Paul E. McKenney wrote: > On Mon, Apr 24, 2017 at 09:35:03AM +0200, Mike Galbraith wrote: > > On Sun, 2017-04-23 at 23:22 -0700, Paul E. McKenney wrote: > > > > > Could you please collect an ftrace (or whatever) showing the timestamp > > > sequence of calls to synchronize_srcu(), synchronize_srcu_expedited(), > > > and call_srcu() during the execution of the stress script? If it is easy > > > to do, also the timestamp sequence of returns from synchronize_srcu() > > > and synchronize_srcu_expedited()? > > > > The first two minutes below config bits, trace_printk([enter/exit]) in > > each function. If you (unlikely, but..) want the whole trace, holler. > > > > # RCU Subsystem > > CONFIG_TREE_RCU=y > > CONFIG_RCU_EXPERT=y > > CONFIG_SRCU=y > > # CONFIG_CLASSIC_SRCU is not set > > CONFIG_TREE_SRCU=y > > # CONFIG_TASKS_RCU is not set > > CONFIG_RCU_STALL_COMMON=y > > CONFIG_RCU_FANOUT=64 > > CONFIG_RCU_FANOUT_LEAF=16 > > # CONFIG_RCU_FAST_NO_HZ is not set > > # CONFIG_TREE_RCU_TRACE is not set > > CONFIG_RCU_KTHREAD_PRIO=0 > > CONFIG_RCU_NOCB_CPU=y > > CONFIG_RCU_NOCB_CPU_NONE=y > > # CONFIG_RCU_NOCB_CPU_ZERO is not set > > # CONFIG_RCU_NOCB_CPU_ALL is not set > > # RCU Debugging > > # CONFIG_PROVE_RCU is not set > > # CONFIG_SPARSE_RCU_POINTER is not set > > # CONFIG_RCU_PERF_TEST is not set > > # CONFIG_RCU_TORTURE_TEST is not set > > CONFIG_RCU_CPU_STALL_TIMEOUT=60 > > # CONFIG_RCU_TRACE is not set > > # CONFIG_RCU_EQS_DEBUG is not set > > Thank you for the data, extremely helpful! I now know that I must > immediately fix this the hard way rather than trying several simpler > possibilities that, according to your trace data, would be complete > wastes of time. > > My initial calculation was actually optimistic. There was one > exit-to-enter interval at about 80 milliseconds, but the second > longest is 342 microseconds, and the vast majority are under 200 > microseconds. > > Back to the drawing board! > > This will take some time, but I should have a patch for you in a few days, > hopefully sooner. Making progress, might have something late tomorrow (Wednesday) Pacific Time, will definitely have something Thursday Pacific Time. Thanx, Paul