From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754622AbdEHNY3 (ORCPT ); Mon, 8 May 2017 09:24:29 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58704 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753302AbdEHNY1 (ORCPT ); Mon, 8 May 2017 09:24:27 -0400 Date: Mon, 8 May 2017 06:24:23 -0700 From: "Paul E. McKenney" To: mitko@stz-bg.com Cc: Steven Rostedt , Josh Triplett , linux-kernel@vger.kernel.org, Mathieu Desnoyers , Lai Jiangshan Subject: Re: BUG: rcu_sched detected stalls on CPUs Reply-To: paulmck@linux.vnet.ibm.com References: <20170120101912.00498a6e@gandalf.local.home> <9976847ad11196cfb6a586217c1bc523@stz-bg.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9976847ad11196cfb6a586217c1bc523@stz-bg.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-TM-AS-GCONF: 00 x-cbid: 17050813-0008-0000-0000-000002152BED X-IBM-SpamModules-Scores: X-IBM-SpamModules-Versions: BY=3.00007034; HX=3.00000240; KW=3.00000007; PH=3.00000004; SC=3.00000209; SDB=6.00857757; UDB=6.00424913; IPR=6.00637186; BA=6.00005332; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00015357; XFM=3.00000014; UTC=2017-05-08 13:24:23 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17050813-0009-0000-0000-000035308891 Message-Id: <20170508132423.GX3956@linux.vnet.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-05-08_08:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1703280000 definitions=main-1705080071 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, May 08, 2017 at 09:43:15AM +0300, mitko@stz-bg.com wrote: > On 2017-01-20 17:19, Steven Rostedt wrote: > >On Fri, 20 Jan 2017 10:43:50 +0200 > >mitko@stz-bg.com wrote: > > > >>[1.] One line summary of the problem: > >> > >>rcu_sched detected stalls on CPUs and few minutes server not respond. > > > >Is this reproducible? Or was this a one time ordeal? > > > >> > >>[2.] Full description of the problem/report: > >> > >>Load of my server (postgres database) isnt big less then 0.50 and when > >>error occured rcu_sched detected stalls on CPUs > >>server freeze and nothing is work for 3-5 minute. > >>No network, no video signal, no keyboard, no mouse. Nothing is worked. > >>After these few minutes everything continue normal. > >>This usual is happend once per day. When I check in google find a lots > >>of ppl complain of this error, but no solution. > >>Do any one know can help me to resolve it ? I spoke with few > >>friends and > >>they trying to convince me the problem is in CPU. > >>I did not believe after a 3 years working CPU suddenly stop working > >>correctly, but I might be wrong. > >> > >>[3.] Keywords (i.e., modules, networking, kernel): > >> > >>kernel > >> > >>[4.] Kernel information > >>[4.1.] Kernel version (from /proc/version): > >> > >>Linux version 4.4.38 (root@hive64) (gcc version 5.4.0 (GCC) ) #2 > >>SMP Sun > >>Dec 11 16:11:02 CST 2016 > >> > > > >Have you tried a newer version of the kernel? > > > >-- Steve > > > Hello, > yesterday I change to new kernel: 4.9.26 and still no effect. > > I trying to figure out what I need to buy because I read on google a > lots of posts about that problem, > some ppl suggest is BIOS firmware bug, some ppl tell that when they > swap CPU problem is resolved. > May be problem was started when I first time boot 4.x kernel and > there have cpu microcode updates. > Im 3.x kernels this feature was not in kernel, but I don't know, > only guess. > > Can some one point me clearly: You need to change this one and > problem will be solved ? Looking back at your earlier email, I see the following: [Wed Jan 4 10:19:12 2017] rcu_sched kthread starved for 60004 jiffies! g61967937 c61967936 f0x0 s3 ->state=0x1 That indicates that the task "rcu_sched" isn't being allowed to run. The "->state=0x1" indicates that this task is currently asleep and the "s3" indicates that it is looking to start a grace period, but has not progressed through this task. My guess is that your system get very busy, and that rcu_sched's normal scheduling priority is not sufficient for it to be allowed to run. I therefore suggest that you try raising the rcu_sched task's priority. You can find this task like this: ps -ef | grep rcu_sched On my laptop, I get this: $ ps -ef | grep rcu_sched root 7 2 0 Apr11 ? 00:19:19 [rcu_sched] paulmck 18307 22926 0 06:11 pts/35 00:00:00 grep --color=auto rcu_sche You can use the chrt command to set the priority. For example, given that on my laptop, rcu_sched is PID 7: sudo chrt -f -p 1 7 You can double-check as follows: $ chrt -p 7 pid 7's current scheduling policy: SCHED_FIFO pid 7's current scheduling priority: 1 Does that help? Thanx, Paul > [239940.067938] clocksource: timekeeping watchdog on CPU26: Marking > clocksource 'tsc' as unstable because the skew is too large: > [239940.067943] clocksource: 'hpet' wd_now: > ecb521ce wd_last: ca45912d mask: ffffffff > [239940.067946] clocksource: 'tsc' cs_now: > 2d12df5f88d08 cs_last: 29fffbe2d36d6 mask: ffffffffffffffff > [239940.068357] clocksource: Switched to clocksource hpet > [240000.066457] INFO: rcu_sched detected stalls on CPUs/tasks: > [240000.066488] 2-...: (38 GPs behind) idle=f1a/0/0 > softirq=2417028/2417028 fqs=0 > [240000.066491] 3-...: (27 GPs behind) idle=0bc/0/0 > softirq=3076046/3076047 fqs=0 > [240000.066494] 4-...: (1006 GPs behind) idle=308/0/0 > softirq=1474922/1474922 fqs=0 > [240000.066497] 5-...: (8034 GPs behind) idle=7b4/0/0 > softirq=69165/69166 fqs=0 > [240000.066499] 6-...: (1665 GPs behind) idle=716/0/0 > softirq=16465/16465 fqs=0 > [240000.066502] 7-...: (8034 GPs behind) idle=bd0/0/0 > softirq=15923/15924 fqs=0 > [240000.066505] 8-...: (1665 GPs behind) idle=a62/0/0 > softirq=93861/93861 fqs=0 > [240000.066507] 9-...: (1258 GPs behind) idle=a8c/0/0 > softirq=21608/21608 fqs=0 > [240000.066510] 11-...: (1665 GPs behind) idle=8e6/0/0 > softirq=18844/18844 fqs=0 > [240000.066512] 13-...: (1665 GPs behind) idle=ab8/0/0 > softirq=21278/21278 fqs=0 > [240000.066515] 14-...: (4854 GPs behind) idle=578/0/0 > softirq=25624/25626 fqs=0 > [240000.066517] 15-...: (8034 GPs behind) idle=f5e/0/0 > softirq=18114/18115 fqs=0 > [240000.066520] 16-...: (1665 GPs behind) idle=b66/0/0 > softirq=19511/19511 fqs=0 > [240000.066528] 17-...: (0 ticks this GP) idle=e18/0/0 > softirq=16163/16163 fqs=0 > [240000.066531] 18-...: (1665 GPs behind) idle=a18/0/0 > softirq=16424/16424 fqs=0 > [240000.066533] 19-...: (1665 GPs behind) idle=944/0/0 > softirq=17920/17920 fqs=0 > [240000.066549] 20-...: (20991 GPs behind) idle=750/0/0 > softirq=17549/17552 fqs=0 > [240000.066552] 21-...: (0 ticks this GP) idle=9e0/0/0 > softirq=18937/18937 fqs=0 > [240000.066555] 22-...: (14519 GPs behind) idle=95e/0/0 > softirq=16122/16124 fqs=0 > [240000.066557] 23-...: (1665 GPs behind) idle=bbc/0/0 > softirq=16316/16316 fqs=0 > [240000.066560] 24-...: (0 ticks this GP) idle=c56/0/0 > softirq=20070/20070 fqs=0 > [240000.066563] 27-...: (8034 GPs behind) idle=bdc/0/0 > softirq=18291/18292 fqs=0 > [240000.066566] 28-...: (8034 GPs behind) idle=4f8/0/0 > softirq=22313/22314 fqs=0 > [240000.066568] 29-...: (3443 GPs behind) idle=22a/0/0 > softirq=24975/24976 fqs=0 > [240000.066571] 30-...: (1665 GPs behind) idle=940/0/0 > softirq=89382/89382 fqs=0 > [240000.066572] (detected by 10, t=60002 jiffies, g=4843721, > c=4843720, q=659) > [240000.066578] Task dump for CPU 2: > [240000.066581] swapper/2 R running task 0 0 > 1 0x00200000 > [240000.066588] 0000000000000083 ffffffffa5cecd79 ffff8aca3eea1e00 > ffffffffa64cab40 > [240000.066593] 0000000000000005 ffffffffa64cad38 ffffa96c80073eb0 > ffffffffa5ba9d93 > [240000.066597] ffff8aca3eea1e00 000000000e94fec6 ffff8aca38535940 > 0000000000000002 > [240000.066601] Call Trace: > [240000.066617] [] ? intel_idle+0x109/0x110 > [240000.066625] [] ? cpuidle_enter_state+0x103/0x2b0 > [240000.066629] [] ? cpuidle_enter+0x17/0x20 > [240000.066638] [] ? call_cpuidle+0x23/0x40 > [240000.066641] [] ? cpu_startup_entry+0x147/0x220 > [240000.066647] [] ? start_secondary+0x12d/0x140 > [240000.066650] Task dump for CPU 3: > [240000.066651] swapper/3 R running task 0 0 > 1 0x00200000 > > Regards, > Mitko >