Re: Bug report for RCU stalled warning [3.10.69]

Re: Bug report for RCU stalled warning [3.10.69]

[Paul E. McKenney]

[ Adding LKML on CC so that others can find this. ]

On Wed, Oct 11, 2017 at 12:21:39PM +0800, Wang YanQing wrote:
> Hi, Paul McKenney.
> 
> I have received many machine-stopped-respone reports, after reboot and
> inspect message, all of them show RCU stalled, but I can't figure out
> how to fix it. I can't update the kernel, it is the painful point, so I
> need to fix it in 3.10. I have attached four messages come from different
> cpu and broads(so I guess it is a BUG instead of hardware fault), any
> suggestion is welcome.

The first step is of course to report this to your distro, as they are
the ones who do the care and feeding of such old kernels.  Please include
the information below in that report, as it might help your distro find
and fix the problem.

It looks like the stalled CPU is idle, and that the activity resulting
from the stall-warning message gets things going again.  Callbacks are
being processed, so no OOM.  But you are getting the splat every 60
seconds.  The system has only two CPUs, and is x86.

If you cannot upgrade the kernel, my ability to help is limited.  And the
diagnostics printed with the v3.10 CPU stall warnings are also quite
limited.  However, there are some things you could try as workarounds:

1.	Check to make sure that the rcu_sched kthread is getting
	the CPU time that it needs.  Preventing this kthread from
	running would create exactly this output, assuming that
	the stall warning got it going again temporarily.

2.	It looks like the disturbance of the RCU CPU stall warning
	is getting things going again.  Try artificially providing
	this disturbance, for example, by running a usermode program
	or script that runs on each CPU in turn, then sleeps for
	(say) five seconds.

3.	If you can reconfigure your kernel, try building with
	CONFIG_RCU_FAST_NO_HZ=n.

4.	Was the system running reliably on some earlier version?
	If so, consider reverting back to that version, and include
	the version information in your report to your distro.  If
	your distro provides individual patches, you should consider
	bisecting so as to locate the offending patch.

Good luck with it!

							Thanx, Paul

Re: Bug report for RCU stalled warning [3.10.69]

[Paul E. McKenney]

On Thu, Oct 12, 2017 at 01:38:24PM -0700, Paul E. McKenney wrote:
> [ Adding LKML on CC so that others can find this. ]
> 
> On Wed, Oct 11, 2017 at 12:21:39PM +0800, Wang YanQing wrote:
> > Hi, Paul McKenney.
> > 
> > I have received many machine-stopped-respone reports, after reboot and
> > inspect message, all of them show RCU stalled, but I can't figure out
> > how to fix it. I can't update the kernel, it is the painful point, so I
> > need to fix it in 3.10. I have attached four messages come from different
> > cpu and broads(so I guess it is a BUG instead of hardware fault), any
> > suggestion is welcome.
> 
> The first step is of course to report this to your distro, as they are
> the ones who do the care and feeding of such old kernels.  Please include
> the information below in that report, as it might help your distro find
> and fix the problem.
> 
> It looks like the stalled CPU is idle, and that the activity resulting
> from the stall-warning message gets things going again.  Callbacks are
> being processed, so no OOM.  But you are getting the splat every 60
> seconds.  The system has only two CPUs, and is x86.
> 
> If you cannot upgrade the kernel, my ability to help is limited.  And the
> diagnostics printed with the v3.10 CPU stall warnings are also quite
> limited.  However, there are some things you could try as workarounds:
> 
> 1.	Check to make sure that the rcu_sched kthread is getting
> 	the CPU time that it needs.  Preventing this kthread from
> 	running would create exactly this output, assuming that
> 	the stall warning got it going again temporarily.
> 
> 2.	It looks like the disturbance of the RCU CPU stall warning
> 	is getting things going again.  Try artificially providing
> 	this disturbance, for example, by running a usermode program
> 	or script that runs on each CPU in turn, then sleeps for
> 	(say) five seconds.
> 
> 3.	If you can reconfigure your kernel, try building with
> 	CONFIG_RCU_FAST_NO_HZ=n.

And if you can reconfigure kernel, in v3.10, building with
CONFIG_RCU_CPU_STALL_INFO and CONFIG_RCU_CPU_STALL_VERBOSE will provide
more information on the CPUs and tasks stalling the grace period.

							Thanx, Paul

> 4.	Was the system running reliably on some earlier version?
> 	If so, consider reverting back to that version, and include
> 	the version information in your report to your distro.  If
> 	your distro provides individual patches, you should consider
> 	bisecting so as to locate the offending patch.
> 
> Good luck with it!
> 
> 							Thanx, Paul