From mboxrd@z Thu Jan 1 00:00:00 1970 From: Ville =?iso-8859-1?Q?Syrj=E4l=E4?= Subject: Re: S3 resume regression [1cf4f629d9d2 ("cpu/hotplug: Move online calls to hotplugged cpu")] Date: Tue, 1 Nov 2016 22:47:37 +0200 Message-ID: <20161101204737.GB4617@intel.com> References: <20160809172057.GZ4329@intel.com> <20161027172852.GE4617@intel.com> <20161027192006.GF4617@intel.com> <20161027203745.GH4617@intel.com> <20161028155603.GI4617@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Return-path: Content-Disposition: inline In-Reply-To: Sender: linux-pm-owner@vger.kernel.org To: Thomas Gleixner Cc: Feng Tang , feng.tang@intel.com, "Rafael J. Wysocki" , "Rafael J. Wysocki" , Steven Rostedt , Sebastian Andrzej Siewior , linux-arch@vger.kernel.org, Rik van Riel , "Srivatsa S. Bhat" , Peter Zijlstra , Arjan van de Ven , Rusty Russell , Oleg Nesterov , Tejun Heo , Andrew Morton , Paul McKenney , Linus Torvalds , Paul Turner , Linux Kernel Mailing List , "Zhang, Rui" List-Id: linux-arch.vger.kernel.org On Fri, Oct 28, 2016 at 08:58:41PM +0200, Thomas Gleixner wrote: > On Fri, 28 Oct 2016, Ville Syrjälä wrote: > > On Thu, Oct 27, 2016 at 10:41:18PM +0200, Thomas Gleixner wrote: > > > On Thu, 27 Oct 2016, Ville Syrjälä wrote: > > > > On Thu, Oct 27, 2016 at 09:25:05PM +0200, Thomas Gleixner wrote: > > > > > So it would be interesting whether that hunk in resume_broadcast() is > > > > > sufficient. > > > > > > > > So far it looks like the answer is yes. > > > > > > > > Looks to be about 5 seconds slower than acpi-idle in resuming, but > > > > I suppose that's not all that surprising ;) > > > > > > Well, set it to 1msec then. If that works reliably then we really can do > > > that unconditionally. There is no harm in firing a useless timer during > > > resume once. > > > > I narrowed down the required timeout, and looks like 25ms is the > > minimum that works. With 24ms I already started to have failures. So > > maybe just bump it up by an order of magnitude to 250ms for some > > safety margin? I left the thing running for the weekend and it failed 26 out of 16057 times with the 25ms timeout. Looks like it takes ~5 minutes to resume when it fails, but eventually it does come back. > > Sure, but what puzzles me is that we need a timeout that big. What happens > between broadcast_resume() and broadcast_resume() + 25ms? > > IOW, what is the event/resume function which we need to bridge. We should > really try to track than down. My hunch would be that SMM trap in the DSDT/SSDT since that's where things ended up last time I was tracing these resume problems. Though I can't recall if that was just with acpi-idle or if intel_idle landed in the same spot as well. I guess I can try to repeat that test tomorrow, or I'll try your function tracer method if the other thing fails. > > You might try to enable function tracing and do a tracing_off() when that > 25ms timeout fires. > > Something like > > stop_trace = true; > > in broadcast_resume() and then in the broadcast timer function: > > if (stop_trace) { > stop_trace = false; > tracing_off(); > } > > Then when the machine is up read the trace, compress and upload it > somewhere or send it in private mail if it's not that big. > > Thanks, > > tglx -- Ville Syrjälä Intel OTC From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com ([192.55.52.93]:59842 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751487AbcKAUru (ORCPT ); Tue, 1 Nov 2016 16:47:50 -0400 Date: Tue, 1 Nov 2016 22:47:37 +0200 From: Ville =?iso-8859-1?Q?Syrj=E4l=E4?= Subject: Re: S3 resume regression [1cf4f629d9d2 ("cpu/hotplug: Move online calls to hotplugged cpu")] Message-ID: <20161101204737.GB4617@intel.com> References: <20160809172057.GZ4329@intel.com> <20161027172852.GE4617@intel.com> <20161027192006.GF4617@intel.com> <20161027203745.GH4617@intel.com> <20161028155603.GI4617@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Sender: linux-arch-owner@vger.kernel.org List-ID: To: Thomas Gleixner Cc: Feng Tang , feng.tang@intel.com, "Rafael J. Wysocki" , "Rafael J. Wysocki" , Steven Rostedt , Sebastian Andrzej Siewior , linux-arch@vger.kernel.org, Rik van Riel , "Srivatsa S. Bhat" , Peter Zijlstra , Arjan van de Ven , Rusty Russell , Oleg Nesterov , Tejun Heo , Andrew Morton , Paul McKenney , Linus Torvalds , Paul Turner , Linux Kernel Mailing List , "Zhang, Rui" , Len Brown , Linux PM , Linux ACPI Message-ID: <20161101204737.9mnLGvWPX7WwMv2c6C46XINyOQXS7g3SgQVFxWpqUNo@z> On Fri, Oct 28, 2016 at 08:58:41PM +0200, Thomas Gleixner wrote: > On Fri, 28 Oct 2016, Ville Syrjälä wrote: > > On Thu, Oct 27, 2016 at 10:41:18PM +0200, Thomas Gleixner wrote: > > > On Thu, 27 Oct 2016, Ville Syrjälä wrote: > > > > On Thu, Oct 27, 2016 at 09:25:05PM +0200, Thomas Gleixner wrote: > > > > > So it would be interesting whether that hunk in resume_broadcast() is > > > > > sufficient. > > > > > > > > So far it looks like the answer is yes. > > > > > > > > Looks to be about 5 seconds slower than acpi-idle in resuming, but > > > > I suppose that's not all that surprising ;) > > > > > > Well, set it to 1msec then. If that works reliably then we really can do > > > that unconditionally. There is no harm in firing a useless timer during > > > resume once. > > > > I narrowed down the required timeout, and looks like 25ms is the > > minimum that works. With 24ms I already started to have failures. So > > maybe just bump it up by an order of magnitude to 250ms for some > > safety margin? I left the thing running for the weekend and it failed 26 out of 16057 times with the 25ms timeout. Looks like it takes ~5 minutes to resume when it fails, but eventually it does come back. > > Sure, but what puzzles me is that we need a timeout that big. What happens > between broadcast_resume() and broadcast_resume() + 25ms? > > IOW, what is the event/resume function which we need to bridge. We should > really try to track than down. My hunch would be that SMM trap in the DSDT/SSDT since that's where things ended up last time I was tracing these resume problems. Though I can't recall if that was just with acpi-idle or if intel_idle landed in the same spot as well. I guess I can try to repeat that test tomorrow, or I'll try your function tracer method if the other thing fails. > > You might try to enable function tracing and do a tracing_off() when that > 25ms timeout fires. > > Something like > > stop_trace = true; > > in broadcast_resume() and then in the broadcast timer function: > > if (stop_trace) { > stop_trace = false; > tracing_off(); > } > > Then when the machine is up read the trace, compress and upload it > somewhere or send it in private mail if it's not that big. > > Thanks, > > tglx -- Ville Syrjälä Intel OTC