RE: 2.6.16-rc5: known regressions

RE: 2.6.16-rc5: known regressions

[Yu, Luming]

>Subject    : S3 sleep hangs the second time - 600X
>References : http://bugzilla.kernel.org/show_bug.cgi?id=5989
>Submitter  : Sanjoy Mahajan <sanjoy@mrao.cam.ac.uk>
>Handled-By : Luming Yu <luming.yu@intel.com>
>Status     : is being debugged,
>             we might want to change the default back for 2.6.16:
>             http://lkml.org/lkml/2006/2/25/101
>

Accordint to bug report, the BIOS DSDT is modified.
I don't know how these changes affect the results
of suspend/resume. But, it is clear this is NOT right approach 
to fix problem. Hence, I need the testing report with 
un-modified DSDT on TP 600X, bios 1.11.

--Luming

Re: 2.6.16-rc5: known regressions

[Sanjoy Mahajan]

>> Subject    : S3 sleep hangs the second time - 600X
>> References : http://bugzilla.kernel.org/show_bug.cgi?id=5989

From: "Yu, Luming" <luming.yu@intel.com>
> According to bug report, the BIOS DSDT is modified.  I don't know
> how these changes affect the results of suspend/resume. But, it is
> clear this is NOT right approach to fix problem. Hence, I need the
> testing report with un-modified DSDT on TP 600X, bios 1.11.

I'll try it, although I don't think I'll get any data on the problem.
The unmodified DSDT (bios 1.11) lacks an S3 sleep object, so I had to
modify the DSDT even to get S3 to sleep at all.  See
<http://bugzilla.kernel.org/show_bug.cgi?id=3534> for that discussion.
In additional comment #4 there (2004-10-14), you said:

  The root cause of [the missing S3 object] failure is that linux is
  using element in

  const char      *acpi_gbl_sleep_state_names[ACPI_S_STATE_COUNT] =
  {
	  "\_S0_",
	  "\_S1_",
	  "\_S2_",
	  "\_S3_",
	  "\_S4_",
	  "\_S5_"
  };

  to call acpi_get_sleep_type_data, but your box define _S3 under the
  device PNP0A03. So, the evaluating \_S3 will fail.

  The workaround in DSDT is to change _S3 to \_S3_ .
  We can fix it in acpi driver soon.

It looks unchanged in a recent acpi driver
(drivers/acpi/utilities/utglobal.c, line 170, 2.6.16-rc2), so I
suspect S3 won't happen with the vanilla DSDT.

(Sorry, I was away for 10 days and also just saw your info requests in
the bugme #5989.)

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

Re: 2.6.16-rc5: known regressions

[Matthew Garrett]

On Fri, Mar 03, 2006 at 02:59:22AM +0000, Sanjoy Mahajan wrote:

> I'll try it, although I don't think I'll get any data on the problem.
> The unmodified DSDT (bios 1.11) lacks an S3 sleep object, so I had to
> modify the DSDT even to get S3 to sleep at all.  See
> <http://bugzilla.kernel.org/show_bug.cgi?id=3534> for that discussion.

I think it's arguably a bit extreme to describe "My setup is so 
unsupported that I had to modify my firmware to enable sleep and then 
override the kernel's sanity checks and it's stopped working with 
2.6.16" as a regression.

-- 
Matthew Garrett | mjg59@srcf.ucam.org

Re: 2.6.16-rc5: known regressions

[Sanjoy Mahajan]

>> I'll try it, although I don't think I'll get any data on the problem.
>> The unmodified DSDT (bios 1.11) lacks an S3 sleep object, so I had to
>> modify the DSDT even to get S3 to sleep at all.  See
>> <http://bugzilla.kernel.org/show_bug.cgi?id=3534> for that discussion.

> I think it's arguably a bit extreme to describe "My setup is so 
> unsupported that I had to modify my firmware to enable sleep and then 
> override the kernel's sanity checks and it's stopped working with 
> 2.6.16" as a regression.

I agree, and that was the point of 'picture of me hanging head in
shame', so there's no need to rub it in.

Anyway, the TP600X w/ vanilla DSDT *was* unsupported (circa 2.6.11),
but now the ACPI interpreter can interpret the vanilla DSDT and go
into S3 sleep (before, it would complain about a missing S3 sleep
object because the DSDT used a funny syntax).  There were other
problems in the vanilla DSDT (e.g. probably using fn-F7 to switch to
an external display doesn't work) but I'll investigate them one at a
time.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

[Re: bugme #5989, head no longer hanging in shame]

From: "Yu, Luming" <luming.yu@intel.com>
> I suggest you to retest, and post dmesg with UN-modified BIOS.

I'm now running/testing an unmodified DSDT with 2.6.16-rc5.  For a while
I had no S3 hangs, but I just noticed them again.  The error is the same
as with the modified DSDT (with slightly different offsets):

exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 width) Address=0000000023FDFFC0
exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 width) Address=00000000000000B2

repeated endlessly.

I think the problem resurfaced once I decided to let my sleep.sh script
leave the thermal driver loaded before going into S3 (suspecting that
the bug might come back if I did that).

So I susect that my modified DSDT didn't cause the S3 problems, it
merely exposed one even in the minimal configuration discussed in the
#5989 report.

Which makes me wonder about another bug that disappeared when I switched
to the vanilla DSDT: While printing (via gs+hpijs to an HP photosmart
2710 via the wireless card), the system makes double-beeps as if it were
having the AC adapter plugged and unplugged.  These noises happen when
printing via the wireless card or via USB (to a different HP inkjet),
but not when printing via the parallel port to a Lexmark laserprinter
(using just gs).  Since I didn't do anything to the battery code in the
DSDT, I now wonder whether changing the DSDT merely exposed the issue
but didn't create it.

[From an earlier msg:]
> I think the truth is, for 5989, we need to fix thermal and processor
> driver issue.

I agree, although I think the processor driver is not the culprit.  My
earlier testing with the (with the modified DSDT) worked fine with the
processor module loaded, but hung with processor + thermal loaded.

-Sanjoy

`A society of sheep must in time beget a government of wolves.'
   - Bertrand de Jouvenal

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Thomas Renninger]

On Fri, 2006-03-10 at 00:26 -0500, Sanjoy Mahajan wrote:
> [Re: bugme #5989, head no longer hanging in shame]
> 
> From: "Yu, Luming" <luming.yu@intel.com>
> > I suggest you to retest, and post dmesg with UN-modified BIOS.
> 
> I'm now running/testing an unmodified DSDT with 2.6.16-rc5.  For a while
> I had no S3 hangs, but I just noticed them again.  The error is the same
> as with the modified DSDT (with slightly different offsets):
> 
> exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
> exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 width) Address=0000000023FDFFC0
> exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 width) Address=00000000000000B2
> 
> repeated endlessly.

This sounds like the problem Daniel had on his Samsung P35 recently.
He could fix it by getting rid of some asus_unhide_smbus stuff or the
otherway around, adding asus_unhide_smbus quirks in the S3 resume code.

This thread was recently posted on lkml:
Re: [patch] smbus unhiding kills thermal management

Here are some more details, for me that sounds related...:
https://bugzilla.novell.com/show_bug.cgi?id=173420

     Thomas

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> This sounds like the problem Daniel had on his Samsung P35 recently.
> He could fix it by getting rid of some asus_unhide_smbus stuff or the
> otherway around, adding asus_unhide_smbus quirks in the S3 resume code.
> 
> This thread was recently posted on lkml:
> Re: [patch] smbus unhiding kills thermal management
> 

That seems likely, thanks for the pointer: Besides the ACPI sleep
hangs, this machine (TP 600X) has fan troubles upon S3 resume.  The
problems don't do harm (the damn fan keeps turning on when it
shouldn't), but that's probably chance.  Various patches that I tested
for S3 resume hangs reversed this fan behavior, making the fan refuse
to turn on when it should have.  The same problem happened after
resume from swsusp (bugzilla #5000).

> https://bugzilla.novell.com/show_bug.cgi?id=173420

>From Comment #30 at the above url: "The Linux ACPI code seems to
actively prevent the fan from running and that worries me."

I saw that as well, and found the following recipe would work around
the problem:

1. Set the trip point to, say, 70 C -- well above the actual
   temperature.

2. Then set the trip to anything reasonable that's under the current
   temperature (27 C always works).  Now the fan turns on, and behaves
   fine from then.

My explanation is that, before step 1, the fan is off but the OS
thinks it's on.  So the dialogue goes something like:

Hardware (from EC or BIOS?): Ack, I'm overheating, turn on the fan now!
OS: There, there, take it easy.  I've checked bit fields in my
     memory, and the fan is on.  So I don't have to do anything.
Hardware: Ack, ...
OS: There, there, ...
[Hence the 100% kacpid CPU usage]

Based on this explanation, I added a resume method to the fan driver.
It would turn on the fan and mark it as on.  So then the internal OS
state matched the actual state.  The fix didn't work for at least one
reason: ACPI drivers didn't have suspend/resume methods (though now
there are test patches to add those methods).

Another fix, probably worth doing anyway, is to turn on the fan if the
BIOS asks for it, whether or not the OS thinks it's on.  The chance of
the two pieces of information getting out of synch, and the hardware
damage it can cause, is enough to make it worthwhile.  The reverse
case can try to optimize (if BIOS asks to shut off the fan, shut it
off only if OS thinks it's on).  That creates no danger: just extra
fan noise if the fan is on but the OS thinks it's off.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Carl-Daniel Hailfinger]

Sanjoy Mahajan wrote:
> That seems likely, thanks for the pointer: Besides the ACPI sleep
> hangs, this machine (TP 600X) has fan troubles upon S3 resume.  The
> problems don't do harm (the damn fan keeps turning on when it
> shouldn't), but that's probably chance.  Various patches that I tested
> for S3 resume hangs reversed this fan behavior, making the fan refuse
> to turn on when it should have.  The same problem happened after
> resume from swsusp (bugzilla #5000).

Please try kernel 2.6.16.17 (just released). It has the SMBus fix which
may fix resume and fan behaviour.


Regards,
Carl-Daniel
-- 
http://www.hailfinger.org/

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Pavel Machek]

Hi!

> > https://bugzilla.novell.com/show_bug.cgi?id=173420
> 
> >From Comment #30 at the above url: "The Linux ACPI code seems to
> actively prevent the fan from running and that worries me."
> 
> I saw that as well, and found the following recipe would work around
> the problem:
> 
> 1. Set the trip point to, say, 70 C -- well above the actual
>    temperature.
> 
> 2. Then set the trip to anything reasonable that's under the current
>    temperature (27 C always works).  Now the fan turns on, and behaves
>    fine from then.
> 
> My explanation is that, before step 1, the fan is off but the OS
> thinks it's on.  So the dialogue goes something like:
> 
> Hardware (from EC or BIOS?): Ack, I'm overheating, turn on the fan now!
> OS: There, there, take it easy.  I've checked bit fields in my
>      memory, and the fan is on.  So I don't have to do anything.
> Hardware: Ack, ...
> OS: There, there, ...
> [Hence the 100% kacpid CPU usage]
> 
> Based on this explanation, I added a resume method to the fan driver.
> It would turn on the fan and mark it as on.  So then the internal OS
> state matched the actual state.  The fix didn't work for at least one
> reason: ACPI drivers didn't have suspend/resume methods (though now
> there are test patches to add those methods).

Can you redo your patches with those methods?

> Another fix, probably worth doing anyway, is to turn on the fan if the
> BIOS asks for it, whether or not the OS thinks it's on.  The chance of
> the two pieces of information getting out of synch, and the hardware
> damage it can cause, is enough to make it worthwhile.  The reverse

There should be 0% hardware damage chance. Fan failure means overheats
mean emergency power cutoff. I even tested it with paper into fan
blades several times. It mostly works.
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

>From: "Yu, Luming" <luming.yu@intel.com>
>> I suggest you to retest, and post dmesg with UN-modified BIOS.
>
>I'm now running/testing an unmodified DSDT with 2.6.16-rc5.  
>For a while
>I had no S3 hangs, but I just noticed them again.  The error 
>is the same
>as with the modified DSDT (with slightly different offsets):

I assume you have tested ec_intr=0 and ec_intr=1.

>
>exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 
>width) Address=0000000023FDFFC0
>exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 
>width) Address=0000000023FDFFC0
>exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>width) Address=00000000000000B2
>
>repeated endlessly.

I need calltrace for this 

>
>I think the problem resurfaced once I decided to let my sleep.sh script
>leave the thermal driver loaded before going into S3 (suspecting that
>the bug might come back if I did that).

Clealy, it's thermal related. We need to narrow down here.

>
>So I susect that my modified DSDT didn't cause the S3 problems, it
>merely exposed one even in the minimal configuration discussed in the
>#5989 report.

The ground rule is Don't use modified DSDT.
If you do that,  the results won't be trusted.

>
>Which makes me wonder about another bug that disappeared when 
>I switched
>to the vanilla DSDT: While printing (via gs+hpijs to an HP photosmart
>2710 via the wireless card), the system makes double-beeps as 
>if it were
>having the AC adapter plugged and unplugged.  These noises happen when
>printing via the wireless card or via USB (to a different HP inkjet),

Interesting, open bug for this.

>but not when printing via the parallel port to a Lexmark laserprinter
>(using just gs).  Since I didn't do anything to the battery code in the
>DSDT, I now wonder whether changing the DSDT merely exposed the issue
>but didn't create it.
>
>[From an earlier msg:]
>> I think the truth is, for 5989, we need to fix thermal and processor
>> driver issue.
>
>I agree, although I think the processor driver is not the culprit.  My
>earlier testing with the (with the modified DSDT) worked fine with the
>processor module loaded, but hung with processor + thermal loaded.
>

ok, we need to start from thermal.  

BTW, do you still think this is a regression?

Thanks,
Luming

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> I assume you have tested ec_intr=0 and ec_intr=1.

Right: I forgot to mention it, but I did test it both ways, and
ec_intr=0 is fine.

>> These noises happen when printing via the wireless card or via USB
>> (to a different HP inkjet),

> Interesting, open bug for this.

I cannot reproduce it with the vanilla DSDT, only with the modified
one.  But:

> The ground rule is Don't use modified DSDT.
> If you do that,  the results won't be trusted.

>> exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>> width) Address=00000000000000B2
>>
>> repeated endlessly.

> I need calltrace for this 

Looking at /proc/acpi/debug_level, I see several debugging choices
that might give the calltrace you want.  Let me know which ones are
essential (I'd turn all of them on; however, I found when trying to
track this down earlier that the bug would slither away if I had too
much debugging turned on):

ACPI_LV_DISPATCH	       0x00000100 [ ]
ACPI_LV_EXEC		       0x00000200 [ ]
ACPI_LV_NAMES		       0x00000400 [ ]
ACPI_LV_FUNCTIONS	       0x00200000 [ ]

By the way, a long standing buglet for me is that 'cat
/proc/acpi/debug_level' truncates the output to 1024 bytes.  So I have
to do 'cat /proc/acpi/debug_level | cat' so that the first cat doesn't
find that its stdout is a tty and try to reduce its buffer size from
4096 (big enough) to 1024.  A patch is available at
<http://bugzilla.kernel.org/show_bug.cgi?id=5076>

> BTW, do you still think this is a regression?

I'm 95% sure, because booting with ec_intr=0 avoids the problem, so
the commit that made ec_intr=1 the default almost certainly also makes
this bug appear.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

> exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>>> width) Address=00000000000000B2
>>>
>>> repeated endlessly.
>
>> I need calltrace for this 
>
>Looking at /proc/acpi/debug_level, I see several debugging choices
>that might give the calltrace you want.  Let me know which ones are
>essential (I'd turn all of them on; however, I found when trying to
>track this down earlier that the bug would slither away if I had too
>much debugging turned on):

What do you mean of "slither away" ? 
bug go away?

>
>ACPI_LV_DISPATCH	       0x00000100 [ ]
>ACPI_LV_EXEC		       0x00000200 [ ]
>ACPI_LV_NAMES		       0x00000400 [ ]
>ACPI_LV_FUNCTIONS	       0x00200000 [ ]
>
>By the way, a long standing buglet for me is that 'cat
>/proc/acpi/debug_level' truncates the output to 1024 bytes.  So I have
>to do 'cat /proc/acpi/debug_level | cat' so that the first cat doesn't
>find that its stdout is a tty and try to reduce its buffer size from
>4096 (big enough) to 1024.  A patch is available at
><http://bugzilla.kernel.org/show_bug.cgi?id=5076>

let's start from:

echo -n 0x10 > /proc/acpi/debug_layer
echo -n 0x10 > /proc/acpi/debug_level

>
>> BTW, do you still think this is a regression?
>
>I'm 95% sure, because booting with ec_intr=0 avoids the problem, so
>the commit that made ec_intr=1 the default almost certainly also makes
>this bug appear.

why NOT 100% sure? :-)

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> What do you mean of "slither away" ? 
> bug go away?

I can no longer trigger it, at least not with the usual procedure.  I
doubt that it goes away (i.e. that it is solved), only that it
slithers into hiding, like bugs that disappear when compiling a C
program with -g but show up when compiling without -g.

> echo -n 0x10 > /proc/acpi/debug_layer
> echo -n 0x10 > /proc/acpi/debug_level

Oh, I always have more info turned on in my sleep.sh script
(debug_layer = 0xFFFF3FFF to begin with, and the script sets
debug_level to 0x1F).  I'll attach the slightly trimmed log file to
the bugme report.  If it's too much information, let me know and I'll
retest with just the above settings.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

Actually, there's no point only attaching it at bugzilla, since the
relevant output is shorter than I thought:

/proc/cmdline: auto BOOT_IMAGE=16-rc5 ro root=305 idebus=66 apm=off acpi=force pci=noacpi console=ttyS0,115200 console=tty0 acpi_sleep=s3_bios cpufreq.debug=7 acpi_debug=0xffffffff

where 16-rc5 is my LILO label for 2.6.16-rc5

The extract from the dmesgs:

Stopping tasks: ========================================================|
Execute Method: [\_SB_.LID0._PSW] (Node c1564808)
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFF40
 acpi_ec-0458 [37] ec_intr_read          : Read [02] from address [32]
 acpi_ec-0508 [37] ec_intr_write         : Wrote [06] to address [32]
Execute Method: [\_SB_.SLPB._PSW] (Node c1564708)
Execute Method: [\_S3_] (Node c157a988)
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFF40
Execute Method: [\_PTS] (Node c157ab48)
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFF40
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFF40
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFF40
exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 width) Address=0000000023FDFFC0
exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC4
exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 width) Address=0000000023FDFFC4
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 width) Address=00000000000000B2
exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 width) Address=0000000023FDFFC0
exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 width) Address=0000000023FDFFC0
exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 width) Address=00000000000000B2

And then these above four lines (exregion-0185, -0185, -0185, -0290)
repeat until I reboot.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

>width) Address=0000000023FDFFC0
>exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>width) Address=00000000000000B2
>exregion-0185 [35] ex_system_memory_space: system_memory 0 (32 
>width) Address=0000000023FDFFC0
>exregion-0185 [36] ex_system_memory_space: system_memory 0 (32 
>width) Address=0000000023FDFFC0
>exregion-0185 [36] ex_system_memory_space: system_memory 1 (32 
>width) Address=0000000023FDFFC0
>exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>width) Address=00000000000000B2
>
>And then these above four lines (exregion-0185, -0185, -0185, -0290)
>repeat until I reboot.
>

If I understand correctly, it was due to  LEqual(S_AH, 0xA6) awlays
true.
SMM bios code didn't  respond , or respond correctly 
to the request by "store 0x81, APMD"  due to thermal module caused
issue?
I need the acpi trace log before _PTS to see what kind of thermal
related methods got called.

    Method (SMPI, 1, NotSerialized)
    {
        Store (S_AX, Local0)
        Store (0x81, APMD)
        While (LEqual (S_AH, 0xA6))
        {
            Sleep (0x64)
            Store (Local0, S_AX)
            Store (0x81, APMD)
        }
    }

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> I need the acpi trace log before _PTS to see what kind of thermal
> related methods got called.

Alas, I've included all the dmesg's.  

Below is the script that I use to enter S3 sleep.  It unloads rid of
troublesome modules and stop services that don't sleep well.  Then
(for debugging) it sends the kernel version and boot parameters across
the serial console (the @@@@ SLEEP line), raises the debug level to
0x1F, does a sync (in case the sleep hangs, since this is my
production machine), and then enters mem sleep.

So nothing in it should trigger any thermal methods; except that I
usually have the THM2 trip point raised to 45C with a polling time of
100 seconds.  So once in a while a thermal poll will happen sleep is
being set up.  I am not sure whether it would be reported in the
dmesgs if it happened; but the S3 failure happens much more often than
such a thermal polling would happen, so I doubt the S3 failure
requires a thermal poll.

#!/bin/bash -x
# S3 (suspend to memory), with cleanups before and after
sync
ifdown eth0
remove='prism54 xircom_cb xircom_tulip_cb' 
remove2='snd_pcm_oss snd_cs46xx'
modprobe -rv $remove
modprobe -rv $remove2
/etc/init.d/chrony stop  > /dev/null

sleep 1

(echo "@@@@ SLEEP" ; date ; uname -a ; cat /proc/cmdline ) > /dev/tts/0
echo 0x0000001F > /proc/acpi/debug_level
sync
sleep 2
echo -n mem > /sys/power/state
[stuff for wakeup snipped]

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

>> I need the acpi trace log before _PTS to see what kind of thermal
>> related methods got called.
>
>Alas, I've included all the dmesg's.  

I need the full log  for S3 suspend failure not just snippets.
Please attach it on bugzilla.kernel.org

The log for S3 suspend success cannot help me to track down.


>
>Below is the script that I use to enter S3 sleep.  It unloads rid of
>troublesome modules and stop services that don't sleep well.  Then
>(for debugging) it sends the kernel version and boot parameters across
>the serial console (the @@@@ SLEEP line), raises the debug level to
>0x1F, does a sync (in case the sleep hangs, since this is my
>production machine), and then enters mem sleep.
>
>So nothing in it should trigger any thermal methods; except that I
>usually have the THM2 trip point raised to 45C with a polling time of
>100 seconds.  So once in a while a thermal poll will happen sleep is
>being set up.  I am not sure whether it would be reported in the
>dmesgs if it happened; but the S3 failure happens much more often than
>such a thermal polling would happen, so I doubt the S3 failure
>requires a thermal poll.

Could you try to mute thermal poll?

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> Could you try to mute thermal poll?

Done.  The sleep.sh script now has

echo 0 > /proc/acpi/thermal_zone/THM2/polling_frequency
echo 0 > /proc/acpi/thermal_zone/THM0/polling_frequency
sleep 1

> I need the full log  for S3 suspend failure not just snippets.
> Please attach it on bugzilla.kernel.org

Done.

> The log for S3 suspend success cannot help me to track down.

For completeness, I didn't excise that portion of the log.  It's not
many lines, plus it doesn't make it harder to find the failing
portion: The suspend failure happens after the second "@@@@ SLEEP" in
the log.

Should I turn on more acpi_debug_level debugging?

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

Thanks for your debug information.

>
>> Could you try to mute thermal poll?
>
>Done.  The sleep.sh script now has
>
>echo 0 > /proc/acpi/thermal_zone/THM2/polling_frequency
>echo 0 > /proc/acpi/thermal_zone/THM0/polling_frequency
>sleep 1

Hmm,  could you file dmesges with tmermal module loaded and
unloaded?

>
>> I need the full log  for S3 suspend failure not just snippets.
>> Please attach it on bugzilla.kernel.org
>
>Done.

I saw this acpi_debug=0xffffffff.

I used to used to use acpi_debug_layer=0x10 acpi_debug_level=0x10
Could you try that?

Re: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Sanjoy Mahajan]

> Hmm, could you file dmesgs with thermal module loaded and unloaded?

Filed at bugzilla.

> I saw this acpi_debug=0xffffffff.

Sorry, it's a legacy from trying to debug #5112, and I've removed it
for getting the above dmesgs.  I'm not even sure what that option does
since it's not documented in the kernel-parameters.txt, but it does
increase the amount of debugging info.

> I used to used to use acpi_debug_layer=0x10 acpi_debug_level=0x10
> Could you try that?

For the above dmesgs I booted with acpi_dbg_level=0x10
acpi_dbg_layer=0x10 and then did two sleep-wake cycles with no thermal
module (both went fine), then one cycle with the thermal module loaded
(went fine), and then the usual failing second sleep with the thermal
module still loaded.  The sleep-wake cycles themselves (i.e. once the
system booted) were done with acpi_debug_level=0x1F rather than the
0x10 boot value.

Let me know if there's a different permutation of debug options that I
should try.  I wasn't sure whether you meant that I should leave all
the debug values at 0x10.  Or whether I should still include
acpi_debug=0xffffffff on top of the other options.

-Sanjoy

`Never underestimate the evil of which men of power are capable.'
         --Bertrand Russell, _War Crimes in Vietnam_, chapter 1.

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

>> Hmm, could you file dmesgs with thermal module loaded and unloaded?
>
>Filed at bugzilla.

Excellent! .

>Let me know if there's a different permutation of debug options that I
>should try.  I wasn't sure whether you meant that I should leave all
>the debug values at 0x10.  Or whether I should still include
>acpi_debug=0xffffffff on top of the other options.

So far, it's ok,  I saw these,  Could you do bisection to find out
which methods or which thermal zone cause trouble?
To do that, you have to hack thermal.c by commenting out 
some calls of evaluating methods below.
I hope it is easy for you!	 :-)

Thanks,
Luming

Execute Method: [\_TZ_.THM0._TMP] (Node c157bf88)
Execute Method: [\_TZ_.THM0._PSV] (Node c157be48)
Execute Method: [\_TZ_.THM0._TC1] (Node c157bdc8)
Execute Method: [\_TZ_.THM0._TC2] (Node c157bd88)
Execute Method: [\_TZ_.THM0._TSP] (Node c157bd48)
Execute Method: [\_TZ_.THM0._AC0] (Node c157bf48)
Execute Method: [\_TZ_.THM0._SCP] (Node c157bec8)
Execute Method: [\_TZ_.THM0._TMP] (Node c157bf88)
ACPI: Thermal Zone [THM0] (47 C)
Execute Method: [\_TZ_.THM2._TMP] (Node c157bb88)
Execute Method: [\_TZ_.THM2._AC0] (Node c157bb48)
Execute Method: [\_TZ_.THM2._SCP] (Node c157bac8)
Execute Method: [\_TZ_.THM2._TMP] (Node c157bb88)
Execute Method: [\_TZ_.PFN0._ON_] (Node c157a2c8)
Execute Method: [\_TZ_.PFN0._STA] (Node c157a308)
ACPI: Thermal Zone [THM2] (40 C)
Execute Method: [\_TZ_.THM6._TMP] (Node c157b948)
Execute Method: [\_TZ_.THM6._AC0] (Node c157b908)
Execute Method: [\_TZ_.THM6._SCP] (Node c157b888)
Execute Method: [\_TZ_.THM6._TMP] (Node c157b948)
ACPI: Thermal Zone [THM6] (30 C)
Execute Method: [\_TZ_.THM7._TMP] (Node c157b708)
Execute Method: [\_TZ_.THM7._AC0] (Node c157b6c8)
Execute Method: [\_TZ_.THM7._SCP] (Node c157b648)
Execute Method: [\_TZ_.THM7._TMP] (Node c157b708)
ACPI: Thermal Zone [THM7] (33 C)

RE: 2.6.16-rc5: known regressions [TP 600X S3, vanilla DSDT]

[Yu, Luming]

>> exregion-0185 [36] ex_system_memory_space: system_memory 0 
>(32 width) Address=0000000023FDFFC0
>> exregion-0185 [36] ex_system_memory_space: system_memory 1 
>(32 width) Address=0000000023FDFFC0
>> exregion-0290 [36] ex_system_io_space_han: system_iO 1 (8 
>width) Address=00000000000000B2
>> 
>> repeated endlessly.

Hmm.. interesting.  This looks like same error with TP600X.

>
>This sounds like the problem Daniel had on his Samsung P35 recently.
>He could fix it by getting rid of some asus_unhide_smbus stuff or the
>otherway around, adding asus_unhide_smbus quirks in the S3 resume code.
>
>This thread was recently posted on lkml:
>Re: [patch] smbus unhiding kills thermal management
>
>Here are some more details, for me that sounds related...:
>https://bugzilla.novell.com/show_bug.cgi?id=173420
>

But this Samsung P35 don't have _GLK. So, I think TP 600x has
a different problem with Samsung P35.

Actually, Sanjoy has a workaround to solve TP 600X S3 issue.
What we need to do is to come up with a clean patch. 
It is on to-do list. 

Thanks,
Luming