I need some serious help to debug suspend to ram problem

I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Hi,

I hit a dead end when trying to understand 
why my notebook can't resume from suspend to ram
if this is done two times a row.

Single suspend/resume cycle works almost perfectly 
(beep that goes through the sound card is muted... 
no morse code for me... :-(

)

I compiled a minimal kernel (absolutely nothing but disk drivers, all experimental option like nohz
turned off)

But I had to turn SMP, since without it system won't resume first time I suspend it.
(How could this affect suspend?)


With SMP and minimal kernel (of course  no closed drivers), I get same behavior, first resume works
second hangs.

I then added some debug code to real mode wakeup code, I put there in first place instructions, that will save some magic value to rtc (to alarm registers that I know are preserved during boot cycle), and I discovered 
sad thing that first time bios does pass control to linux, but second time (when it hangs), it doesn't.


I tried to update bios, and I got same results.

Of course it does work with that @#$%^& OS


I then proceeded to test recently posted low memory corruption patch, and it did show that that @#$%^& BIOS does corrupt low memory
I then reserved all low memory, but system began to hand after first suspend, in exactly same way, but as expected I soon discovered,
that that forces real mode page to be above 1M, ok, then I reserved almost all low memory except 100K window in the middle, so low allocations will work, but be placed in region bios less likely to corrupt, and still that didn't help, still same hang.


You did face lot of such situations, so maybe you know about anything I can do.

System is ACER aspire 5720G.

I now use 2.6.27-rc5, but same did happen on 2.6.26, and if I remember correctly 2.6.25.
(I bought this system in April and installed linux during first day)
I installed hardy there)



Note: I do use nvidia drivers, and without them display stays dark after resume, BUT, this bug is unrelated,
while screen is dark, system does work, I can see hdd led blinking when I issue commands, I can turn system off, suspend again (and get hard hang of resume...), I even stored dmesg after first suspend without nvidia driver, and there are no errors
(except that EC storm , which is harmless here (I tried  patches that fix it, but they didn't affect this suspend bug)

All tests were done without nvidia driver.

I would be very glad if you tell me any suggestions how to fix/debug that thing.

And lastly, suggestion to everybody:

NEVER BUY ACER LAPTOPS, THEY ARE JUST BROKEN

Best regards,
	Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Rafael J. Wysocki]

On Saturday, 20 of September 2008, Maxim Levitsky wrote:
> Hi,
> 
> I hit a dead end when trying to understand 
> why my notebook can't resume from suspend to ram
> if this is done two times a row.
> 
> Single suspend/resume cycle works almost perfectly 
> (beep that goes through the sound card is muted... 
> no morse code for me... :-(
> 
> )
> 
> I compiled a minimal kernel (absolutely nothing but disk drivers, all experimental option like nohz
> turned off)
> 
> But I had to turn SMP, since without it system won't resume first time I suspend it.
> (How could this affect suspend?)

It could if the system is 64-bit.  In which case please have a look at
http://bugzilla.kernel.org/show_bug.cgi?id=11237

> With SMP and minimal kernel (of course  no closed drivers), I get same behavior,
> first resume works second hangs.
> 
> I then added some debug code to real mode wakeup code, I put there in first
> place instructions, that will save some magic value to rtc (to alarm
> registers that I know are preserved during boot cycle), and I discovered   
> sad thing that first time bios does pass control to linux, but second time
> (when it hangs), it doesn't. 
> 
> 
> I tried to update bios, and I got same results.
> 
> Of course it does work with that @#$%^& OS

So we're doing something wrong.  Please try the appended patch.

> I then proceeded to test recently posted low memory corruption patch, and
> it did show that that @#$%^& BIOS does corrupt low memory 
> I then reserved all low memory, but system began to hand after first suspend,
> in exactly same way, but as expected I soon discovered, that that forces real
> mode page to be above 1M, ok, then I reserved almost all low memory except
> 100K window in the middle, so low allocations will work, but be placed in
> region bios less likely to corrupt, and still that didn't help, still same hang.   
>  
> You did face lot of such situations, so maybe you know about anything I can do.
> 

Actually, I didn't, but some people did.  Again,
http://bugzilla.kernel.org/show_bug.cgi?id=11237 is the place to look at.

Thanks,
Rafael

---
From: Rafael J. Wysocki <rjw@sisk.pl>

ACPI suspend: Always use the 32-bit waking vector

According to the ACPI specification 2.0c and later, the 64-bit waking vector
should be cleared and the 32-bit waking vector should be used, unless we want
the wake-up code to be called by the BIOS in Protected Mode.  Moreover, some
systems (for example HP dv5-1004nr) are known to fail to resume if the 64-bit
waking vector is used.  Therefore, modify the code to clear the 64-bit waking
vector, for FACS version 1 or greater, and set the 32-bit one before suspend.

Signed-off-by: Rafael J. Wysocki <rjw@sisk.pl>
---
 drivers/acpi/hardware/hwsleep.c |   37 +++++++++++--------------------------
 1 file changed, 11 insertions(+), 26 deletions(-)

Index: linux-2.6/drivers/acpi/hardware/hwsleep.c
===================================================================
--- linux-2.6.orig/drivers/acpi/hardware/hwsleep.c
+++ linux-2.6/drivers/acpi/hardware/hwsleep.c
@@ -78,19 +78,17 @@ acpi_set_firmware_waking_vector(acpi_phy
 		return_ACPI_STATUS(status);
 	}
 
-	/* Set the vector */
+	/*
+	 * According to the ACPI specification 2.0c and later, the 64-bit
+	 * waking vector should be cleared and the 32-bit waking vector should
+	 * be used, unless we want the wake-up code to be called by the BIOS in
+	 * Protected Mode.  Some systems (for example HP dv5-1004nr) are known
+	 * to fail to resume if the 64-bit vector is used.
+	 */
+	if (facs->version >= 1)
+		facs->xfirmware_waking_vector = 0;
 
-	if ((facs->length < 32) || (!(facs->xfirmware_waking_vector))) {
-		/*
-		 * ACPI 1.0 FACS or short table or optional X_ field is zero
-		 */
-		facs->firmware_waking_vector = (u32) physical_address;
-	} else {
-		/*
-		 * ACPI 2.0 FACS with valid X_ field
-		 */
-		facs->xfirmware_waking_vector = physical_address;
-	}
+	facs->firmware_waking_vector = (u32)physical_address;
 
 	return_ACPI_STATUS(AE_OK);
 }
@@ -134,20 +132,7 @@ acpi_get_firmware_waking_vector(acpi_phy
 	}
 
 	/* Get the vector */
-
-	if ((facs->length < 32) || (!(facs->xfirmware_waking_vector))) {
-		/*
-		 * ACPI 1.0 FACS or short table or optional X_ field is zero
-		 */
-		*physical_address =
-		    (acpi_physical_address) facs->firmware_waking_vector;
-	} else {
-		/*
-		 * ACPI 2.0 FACS with valid X_ field
-		 */
-		*physical_address =
-		    (acpi_physical_address) facs->xfirmware_waking_vector;
-	}
+	*physical_address = (acpi_physical_address)facs->firmware_waking_vector;
 
 	return_ACPI_STATUS(AE_OK);
 }

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Rafael J. Wysocki wrote:
> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>> Hi,
>>
>> I hit a dead end when trying to understand 
>> why my notebook can't resume from suspend to ram
>> if this is done two times a row.
>>
>> Single suspend/resume cycle works almost perfectly 
>> (beep that goes through the sound card is muted... 
>> no morse code for me... :-(
>>
>> )
>>
>> I compiled a minimal kernel (absolutely nothing but disk drivers, all experimental option like nohz
>> turned off)
>>
>> But I had to turn SMP, since without it system won't resume first time I suspend it.
>> (How could this affect suspend?)
> 
> It could if the system is 64-bit.  In which case please have a look at
> http://bugzilla.kernel.org/show_bug.cgi?id=11237
> 
>> With SMP and minimal kernel (of course  no closed drivers), I get same behavior,
>> first resume works second hangs.
>>
>> I then added some debug code to real mode wakeup code, I put there in first
>> place instructions, that will save some magic value to rtc (to alarm
>> registers that I know are preserved during boot cycle), and I discovered   
>> sad thing that first time bios does pass control to linux, but second time
>> (when it hangs), it doesn't. 
>>
>>
>> I tried to update bios, and I got same results.
>>
>> Of course it does work with that @#$%^& OS
> 
> So we're doing something wrong.  Please try the appended patch.
Thanks a lot, but this didn't help.

It still has same pattern, first suspend/resume works perfectly, 
second suspend/resume hangs hard.
It always happens like this, first resume always work 
(unless I turn off smp in kernel (I test this again), or reserve all low memory)

Also note that if I suspend the system to ram, resume, and then suspend to disk, 
then I can suspend to ram and resume, it seems that

on suspend to ram cycle somehow arms BIOS or something else, 
so second resume in a row doesn't work.

I run 32 bit kernel here, this is a long story 
(this bios doesn't turn fan on when running 64-bit version, 
I could update it, and I know that fan issue is fixed there, 
but new bios introduces bigger bug, namely it makes fan to run almost always 
regardless of 32/64 type of os.
And it doesn't fix this suspend/resume issue, I tested this. 
I could start/stop fan manually with a script, but this could fail, 
and maybe I will do so someday.)

The bugzilla seems to be unrelated here, since bios does 
pass control there, but corrupts memory.
Here I also have seen that bios corrupts memory, 
but everything resumes fine first time, and on second time,
bios doesn't pass control (I put set of instructions in beginning 
of wakeup real mode assembly file, no page tables, GDT/LDT are used there)


Best regards,
	Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Rafael J. Wysocki wrote:
> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>> Hi,
>>
>> I hit a dead end when trying to understand 
>> why my notebook can't resume from suspend to ram
>> if this is done two times a row.
>>
>> Single suspend/resume cycle works almost perfectly 
>> (beep that goes through the sound card is muted... 
>> no morse code for me... :-(
>>
>> )
>>
>> I compiled a minimal kernel (absolutely nothing but disk drivers, all experimental option like nohz
>> turned off)
>>
>> But I had to turn SMP, since without it system won't resume first time I suspend it.
>> (How could this affect suspend?)
> 
> It could if the system is 64-bit.  In which case please have a look at
> http://bugzilla.kernel.org/show_bug.cgi?id=11237
> 
>> With SMP and minimal kernel (of course  no closed drivers), I get same behavior,
>> first resume works second hangs.
>>
>> I then added some debug code to real mode wakeup code, I put there in first
>> place instructions, that will save some magic value to rtc (to alarm
>> registers that I know are preserved during boot cycle), and I discovered   
>> sad thing that first time bios does pass control to linux, but second time
>> (when it hangs), it doesn't. 
>>
>>
>> I tried to update bios, and I got same results.
>>
>> Of course it does work with that @#$%^& OS
> 
> So we're doing something wrong.  Please try the appended patch.
> 
>> I then proceeded to test recently posted low memory corruption patch, and
>> it did show that that @#$%^& BIOS does corrupt low memory 
>> I then reserved all low memory, but system began to hand after first suspend,
>> in exactly same way, but as expected I soon discovered, that that forces real
>> mode page to be above 1M, ok, then I reserved almost all low memory except
>> 100K window in the middle, so low allocations will work, but be placed in
>> region bios less likely to corrupt, and still that didn't help, still same hang.   
>>  

More information, I compiled kernels back to 2.6.19, and they all have exactly the same issue.


Any ideas?

Best regards,
	Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Maxim Levitsky wrote:
> Rafael J. Wysocki wrote:
>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>>> Hi,
>>>
>>> I hit a dead end when trying to understand why my notebook can't 
>>> resume from suspend to ram
>>> if this is done two times a row.
>>>
>>> Single suspend/resume cycle works almost perfectly (beep that goes 
>>> through the sound card is muted... no morse code for me... :-(
>>>
>>> )
>>>
>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
>>> experimental option like nohz
>>> turned off)
>>>
>>> But I had to turn SMP, since without it system won't resume first 
>>> time I suspend it.
>>> (How could this affect suspend?)
>>
>> It could if the system is 64-bit.  In which case please have a look at
>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
>>
>>> With SMP and minimal kernel (of course  no closed drivers), I get 
>>> same behavior,
>>> first resume works second hangs.
>>>
>>> I then added some debug code to real mode wakeup code, I put there in 
>>> first
>>> place instructions, that will save some magic value to rtc (to alarm
>>> registers that I know are preserved during boot cycle), and I 
>>> discovered   sad thing that first time bios does pass control to 
>>> linux, but second time
>>> (when it hangs), it doesn't.
>>>
>>> I tried to update bios, and I got same results.
>>>
>>> Of course it does work with that @#$%^& OS
>>
>> So we're doing something wrong.  Please try the appended patch.
> Thanks a lot, but this didn't help.
> 
> It still has same pattern, first suspend/resume works perfectly, second 
> suspend/resume hangs hard.
> It always happens like this, first resume always work (unless I turn off 
> smp in kernel (I test this again), or reserve all low memory)
> 
> Also note that if I suspend the system to ram, resume, and then suspend 
> to disk, then I can suspend to ram and resume, it seems that
> 
> on suspend to ram cycle somehow arms BIOS or something else, so second 
> resume in a row doesn't work.
> 
> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
> fan on when running 64-bit version, I could update it, and I know that 
> fan issue is fixed there, but new bios introduces bigger bug, namely it 
> makes fan to run almost always regardless of 32/64 type of os.
> And it doesn't fix this suspend/resume issue, I tested this. I could 
> start/stop fan manually with a script, but this could fail, and maybe I 
> will do so someday.)
> 
> The bugzilla seems to be unrelated here, since bios does pass control 
> there, but corrupts memory.
> Here I also have seen that bios corrupts memory, but everything resumes 
> fine first time, and on second time,
> bios doesn't pass control (I put set of instructions in beginning of 
> wakeup real mode assembly file, no page tables, GDT/LDT are used there)

I did same test for kernel without SMP, yes it hangs on first resume, but bios
does pass control to linux, so while this is a minor bug, it is unrelated.

I also tested noapic, pci=nommconf. No luck.

Pattern is always the same, first resume works always, second doesn't.
It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
which has a sigmatel codec)
> 
> 
> Best regards,
>     Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Rafael J. Wysocki]

On Sunday, 21 of September 2008, Maxim Levitsky wrote:
> Maxim Levitsky wrote:
> > Rafael J. Wysocki wrote:
> >> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
> >>> Hi,
> >>>
> >>> I hit a dead end when trying to understand why my notebook can't 
> >>> resume from suspend to ram
> >>> if this is done two times a row.
> >>>
> >>> Single suspend/resume cycle works almost perfectly (beep that goes 
> >>> through the sound card is muted... no morse code for me... :-(
> >>>
> >>> )
> >>>
> >>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
> >>> experimental option like nohz
> >>> turned off)
> >>>
> >>> But I had to turn SMP, since without it system won't resume first 
> >>> time I suspend it.
> >>> (How could this affect suspend?)
> >>
> >> It could if the system is 64-bit.  In which case please have a look at
> >> http://bugzilla.kernel.org/show_bug.cgi?id=11237
> >>
> >>> With SMP and minimal kernel (of course  no closed drivers), I get 
> >>> same behavior,
> >>> first resume works second hangs.
> >>>
> >>> I then added some debug code to real mode wakeup code, I put there in 
> >>> first
> >>> place instructions, that will save some magic value to rtc (to alarm
> >>> registers that I know are preserved during boot cycle), and I 
> >>> discovered   sad thing that first time bios does pass control to 
> >>> linux, but second time
> >>> (when it hangs), it doesn't.
> >>>
> >>> I tried to update bios, and I got same results.
> >>>
> >>> Of course it does work with that @#$%^& OS
> >>
> >> So we're doing something wrong.  Please try the appended patch.
> > Thanks a lot, but this didn't help.
> > 
> > It still has same pattern, first suspend/resume works perfectly, second 
> > suspend/resume hangs hard.
> > It always happens like this, first resume always work (unless I turn off 
> > smp in kernel (I test this again), or reserve all low memory)
> > 
> > Also note that if I suspend the system to ram, resume, and then suspend 
> > to disk, then I can suspend to ram and resume, it seems that
> > 
> > on suspend to ram cycle somehow arms BIOS or something else, so second 
> > resume in a row doesn't work.
> > 
> > I run 32 bit kernel here, this is a long story (this bios doesn't turn 
> > fan on when running 64-bit version, I could update it, and I know that 
> > fan issue is fixed there, but new bios introduces bigger bug, namely it 
> > makes fan to run almost always regardless of 32/64 type of os.
> > And it doesn't fix this suspend/resume issue, I tested this. I could 
> > start/stop fan manually with a script, but this could fail, and maybe I 
> > will do so someday.)
> > 
> > The bugzilla seems to be unrelated here, since bios does pass control 
> > there, but corrupts memory.
> > Here I also have seen that bios corrupts memory, but everything resumes 
> > fine first time, and on second time,
> > bios doesn't pass control (I put set of instructions in beginning of 
> > wakeup real mode assembly file, no page tables, GDT/LDT are used there)
> 
> I did same test for kernel without SMP, yes it hangs on first resume, but bios
> does pass control to linux, so while this is a minor bug, it is unrelated.

Still, I'd be interested in debugging this one too, if possible.  That may be
easier too. ;-)

> I also tested noapic, pci=nommconf. No luck.
> 
> Pattern is always the same, first resume works always, second doesn't.
> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
> which has a sigmatel codec)

If you have more than 2 GB of RAM, you can try iommu=soft .

I guess that all of the /sys/power/pm_test tests are passed?

Thanks,
Rafael

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Rafael J. Wysocki wrote:
> On Sunday, 21 of September 2008, Maxim Levitsky wrote:
>> Maxim Levitsky wrote:
>>> Rafael J. Wysocki wrote:
>>>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>>>>> Hi,
>>>>>
>>>>> I hit a dead end when trying to understand why my notebook can't 
>>>>> resume from suspend to ram
>>>>> if this is done two times a row.
>>>>>
>>>>> Single suspend/resume cycle works almost perfectly (beep that goes 
>>>>> through the sound card is muted... no morse code for me... :-(
>>>>>
>>>>> )
>>>>>
>>>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
>>>>> experimental option like nohz
>>>>> turned off)
>>>>>
>>>>> But I had to turn SMP, since without it system won't resume first 
>>>>> time I suspend it.
>>>>> (How could this affect suspend?)
>>>> It could if the system is 64-bit.  In which case please have a look at
>>>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
>>>>
>>>>> With SMP and minimal kernel (of course  no closed drivers), I get 
>>>>> same behavior,
>>>>> first resume works second hangs.
>>>>>
>>>>> I then added some debug code to real mode wakeup code, I put there in 
>>>>> first
>>>>> place instructions, that will save some magic value to rtc (to alarm
>>>>> registers that I know are preserved during boot cycle), and I 
>>>>> discovered   sad thing that first time bios does pass control to 
>>>>> linux, but second time
>>>>> (when it hangs), it doesn't.
>>>>>
>>>>> I tried to update bios, and I got same results.
>>>>>
>>>>> Of course it does work with that @#$%^& OS
>>>> So we're doing something wrong.  Please try the appended patch.
>>> Thanks a lot, but this didn't help.
>>>
>>> It still has same pattern, first suspend/resume works perfectly, second 
>>> suspend/resume hangs hard.
>>> It always happens like this, first resume always work (unless I turn off 
>>> smp in kernel (I test this again), or reserve all low memory)
>>>
>>> Also note that if I suspend the system to ram, resume, and then suspend 
>>> to disk, then I can suspend to ram and resume, it seems that
>>>
>>> on suspend to ram cycle somehow arms BIOS or something else, so second 
>>> resume in a row doesn't work.
>>>
>>> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
>>> fan on when running 64-bit version, I could update it, and I know that 
>>> fan issue is fixed there, but new bios introduces bigger bug, namely it 
>>> makes fan to run almost always regardless of 32/64 type of os.
>>> And it doesn't fix this suspend/resume issue, I tested this. I could 
>>> start/stop fan manually with a script, but this could fail, and maybe I 
>>> will do so someday.)
>>>
>>> The bugzilla seems to be unrelated here, since bios does pass control 
>>> there, but corrupts memory.
>>> Here I also have seen that bios corrupts memory, but everything resumes 
>>> fine first time, and on second time,
>>> bios doesn't pass control (I put set of instructions in beginning of 
>>> wakeup real mode assembly file, no page tables, GDT/LDT are used there)
>> I did same test for kernel without SMP, yes it hangs on first resume, but bios
>> does pass control to linux, so while this is a minor bug, it is unrelated.
> 
> Still, I'd be interested in debugging this one too, if possible.  That may be
> easier too. ;-)

I take a look at that.

> 
>> I also tested noapic, pci=nommconf. No luck.
>>
>> Pattern is always the same, first resume works always, second doesn't.
>> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
>> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
>> which has a sigmatel codec)
> 
> If you have more than 2 GB of RAM, you can try iommu=soft .
> 
> I guess that all of the /sys/power/pm_test tests are passed?

Well, I didn't run /sys/power/pm_test. 
But this system has rock solid suspend to disk, I use it always.

iommu? I don't think this mobo has it, it has PM965/GM965/GL960
(according to lspci)


Best regards,
	Maxim  Levitsky
> 
> Thanks,
> Rafael

Re: I need some serious help to debug suspend to ram problem

[Rafael J. Wysocki]

On Monday, 22 of September 2008, Maxim Levitsky wrote:
> Rafael J. Wysocki wrote:
> > On Sunday, 21 of September 2008, Maxim Levitsky wrote:
> >> Maxim Levitsky wrote:
> >>> Rafael J. Wysocki wrote:
> >>>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
> >>>>> Hi,
> >>>>>
> >>>>> I hit a dead end when trying to understand why my notebook can't 
> >>>>> resume from suspend to ram
> >>>>> if this is done two times a row.
> >>>>>
> >>>>> Single suspend/resume cycle works almost perfectly (beep that goes 
> >>>>> through the sound card is muted... no morse code for me... :-(
> >>>>>
> >>>>> )
> >>>>>
> >>>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
> >>>>> experimental option like nohz
> >>>>> turned off)
> >>>>>
> >>>>> But I had to turn SMP, since without it system won't resume first 
> >>>>> time I suspend it.
> >>>>> (How could this affect suspend?)
> >>>> It could if the system is 64-bit.  In which case please have a look at
> >>>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
> >>>>
> >>>>> With SMP and minimal kernel (of course  no closed drivers), I get 
> >>>>> same behavior,
> >>>>> first resume works second hangs.
> >>>>>
> >>>>> I then added some debug code to real mode wakeup code, I put there in 
> >>>>> first
> >>>>> place instructions, that will save some magic value to rtc (to alarm
> >>>>> registers that I know are preserved during boot cycle), and I 
> >>>>> discovered   sad thing that first time bios does pass control to 
> >>>>> linux, but second time
> >>>>> (when it hangs), it doesn't.
> >>>>>
> >>>>> I tried to update bios, and I got same results.
> >>>>>
> >>>>> Of course it does work with that @#$%^& OS
> >>>> So we're doing something wrong.  Please try the appended patch.
> >>> Thanks a lot, but this didn't help.
> >>>
> >>> It still has same pattern, first suspend/resume works perfectly, second 
> >>> suspend/resume hangs hard.
> >>> It always happens like this, first resume always work (unless I turn off 
> >>> smp in kernel (I test this again), or reserve all low memory)
> >>>
> >>> Also note that if I suspend the system to ram, resume, and then suspend 
> >>> to disk, then I can suspend to ram and resume, it seems that
> >>>
> >>> on suspend to ram cycle somehow arms BIOS or something else, so second 
> >>> resume in a row doesn't work.
> >>>
> >>> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
> >>> fan on when running 64-bit version, I could update it, and I know that 
> >>> fan issue is fixed there, but new bios introduces bigger bug, namely it 
> >>> makes fan to run almost always regardless of 32/64 type of os.
> >>> And it doesn't fix this suspend/resume issue, I tested this. I could 
> >>> start/stop fan manually with a script, but this could fail, and maybe I 
> >>> will do so someday.)
> >>>
> >>> The bugzilla seems to be unrelated here, since bios does pass control 
> >>> there, but corrupts memory.
> >>> Here I also have seen that bios corrupts memory, but everything resumes 
> >>> fine first time, and on second time,
> >>> bios doesn't pass control (I put set of instructions in beginning of 
> >>> wakeup real mode assembly file, no page tables, GDT/LDT are used there)
> >> I did same test for kernel without SMP, yes it hangs on first resume, but bios
> >> does pass control to linux, so while this is a minor bug, it is unrelated.
> > 
> > Still, I'd be interested in debugging this one too, if possible.  That may be
> > easier too. ;-)
> 
> I take a look at that.
> 
> > 
> >> I also tested noapic, pci=nommconf. No luck.
> >>
> >> Pattern is always the same, first resume works always, second doesn't.
> >> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
> >> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
> >> which has a sigmatel codec)
> > 
> > If you have more than 2 GB of RAM, you can try iommu=soft .
> > 
> > I guess that all of the /sys/power/pm_test tests are passed?
> 
> Well, I didn't run /sys/power/pm_test. 
> But this system has rock solid suspend to disk, I use it always.

Please look at http://bugzilla.kernel.org/show_bug.cgi?id=11415 .

Perhaps you can add some information into it.

Thanks,
Rafael

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Rafael J. Wysocki wrote:
> On Monday, 22 of September 2008, Maxim Levitsky wrote:
>> Rafael J. Wysocki wrote:
>>> On Sunday, 21 of September 2008, Maxim Levitsky wrote:
>>>> Maxim Levitsky wrote:
>>>>> Rafael J. Wysocki wrote:
>>>>>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> I hit a dead end when trying to understand why my notebook can't 
>>>>>>> resume from suspend to ram
>>>>>>> if this is done two times a row.
>>>>>>>
>>>>>>> Single suspend/resume cycle works almost perfectly (beep that goes 
>>>>>>> through the sound card is muted... no morse code for me... :-(
>>>>>>>
>>>>>>> )
>>>>>>>
>>>>>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
>>>>>>> experimental option like nohz
>>>>>>> turned off)
>>>>>>>
>>>>>>> But I had to turn SMP, since without it system won't resume first 
>>>>>>> time I suspend it.
>>>>>>> (How could this affect suspend?)
>>>>>> It could if the system is 64-bit.  In which case please have a look at
>>>>>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
>>>>>>
>>>>>>> With SMP and minimal kernel (of course  no closed drivers), I get 
>>>>>>> same behavior,
>>>>>>> first resume works second hangs.
>>>>>>>
>>>>>>> I then added some debug code to real mode wakeup code, I put there in 
>>>>>>> first
>>>>>>> place instructions, that will save some magic value to rtc (to alarm
>>>>>>> registers that I know are preserved during boot cycle), and I 
>>>>>>> discovered   sad thing that first time bios does pass control to 
>>>>>>> linux, but second time
>>>>>>> (when it hangs), it doesn't.
>>>>>>>
>>>>>>> I tried to update bios, and I got same results.
>>>>>>>
>>>>>>> Of course it does work with that @#$%^& OS
>>>>>> So we're doing something wrong.  Please try the appended patch.
>>>>> Thanks a lot, but this didn't help.
>>>>>
>>>>> It still has same pattern, first suspend/resume works perfectly, second 
>>>>> suspend/resume hangs hard.
>>>>> It always happens like this, first resume always work (unless I turn off 
>>>>> smp in kernel (I test this again), or reserve all low memory)
>>>>>
>>>>> Also note that if I suspend the system to ram, resume, and then suspend 
>>>>> to disk, then I can suspend to ram and resume, it seems that
>>>>>
>>>>> on suspend to ram cycle somehow arms BIOS or something else, so second 
>>>>> resume in a row doesn't work.
>>>>>
>>>>> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
>>>>> fan on when running 64-bit version, I could update it, and I know that 
>>>>> fan issue is fixed there, but new bios introduces bigger bug, namely it 
>>>>> makes fan to run almost always regardless of 32/64 type of os.
>>>>> And it doesn't fix this suspend/resume issue, I tested this. I could 
>>>>> start/stop fan manually with a script, but this could fail, and maybe I 
>>>>> will do so someday.)
>>>>>
>>>>> The bugzilla seems to be unrelated here, since bios does pass control 
>>>>> there, but corrupts memory.
>>>>> Here I also have seen that bios corrupts memory, but everything resumes 
>>>>> fine first time, and on second time,
>>>>> bios doesn't pass control (I put set of instructions in beginning of 
>>>>> wakeup real mode assembly file, no page tables, GDT/LDT are used there)
>>>> I did same test for kernel without SMP, yes it hangs on first resume, but bios
>>>> does pass control to linux, so while this is a minor bug, it is unrelated.
>>> Still, I'd be interested in debugging this one too, if possible.  That may be
>>> easier too. ;-)
>> I take a look at that.
>>
>>>> I also tested noapic, pci=nommconf. No luck.
>>>>
>>>> Pattern is always the same, first resume works always, second doesn't.
>>>> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
>>>> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
>>>> which has a sigmatel codec)
>>> If you have more than 2 GB of RAM, you can try iommu=soft .
>>>
>>> I guess that all of the /sys/power/pm_test tests are passed?
>> Well, I didn't run /sys/power/pm_test. 
>> But this system has rock solid suspend to disk, I use it always.
> 
> Please look at http://bugzilla.kernel.org/show_bug.cgi?id=11415 .

Hi,


I took a look there, but it doesn't seem to be similar to my issue,
my issue is much bigger :-(

They tell that 2.6.24 works, but here nothing works, I was never able to do
two suspends in row.

What I did find interesting was that they mention hardware locks of several kind there, so I am thinking
could that be related to EC code, could it be that EC code confuses it somehow, so next boot doesn't work?
Some hardware lock that kernel forgets to unlock, and that prevents bios from resume

Here ec switches to polled mode almost instantly, due to that bogus 'interrupt storm', I tried to increase interrupt threshold,
and no more polled mode, but nether working second resume :-(

apart from usual "ACPI: EC: GPE storm detected, disabling EC GPE"
there is nothing just nothing interesting in dmesg after first resume, and everything works :-(

What could it be.....


Best regards,
	Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Rafael J. Wysocki]

On Saturday, 27 of September 2008, Maxim Levitsky wrote:
> Rafael J. Wysocki wrote:
> > On Monday, 22 of September 2008, Maxim Levitsky wrote:
> >> Rafael J. Wysocki wrote:
> >>> On Sunday, 21 of September 2008, Maxim Levitsky wrote:
> >>>> Maxim Levitsky wrote:
> >>>>> Rafael J. Wysocki wrote:
> >>>>>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
> >>>>>>> Hi,
> >>>>>>>
> >>>>>>> I hit a dead end when trying to understand why my notebook can't 
> >>>>>>> resume from suspend to ram
> >>>>>>> if this is done two times a row.
> >>>>>>>
> >>>>>>> Single suspend/resume cycle works almost perfectly (beep that goes 
> >>>>>>> through the sound card is muted... no morse code for me... :-(
> >>>>>>>
> >>>>>>> )
> >>>>>>>
> >>>>>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
> >>>>>>> experimental option like nohz
> >>>>>>> turned off)
> >>>>>>>
> >>>>>>> But I had to turn SMP, since without it system won't resume first 
> >>>>>>> time I suspend it.
> >>>>>>> (How could this affect suspend?)
> >>>>>> It could if the system is 64-bit.  In which case please have a look at
> >>>>>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
> >>>>>>
> >>>>>>> With SMP and minimal kernel (of course  no closed drivers), I get 
> >>>>>>> same behavior,
> >>>>>>> first resume works second hangs.
> >>>>>>>
> >>>>>>> I then added some debug code to real mode wakeup code, I put there in 
> >>>>>>> first
> >>>>>>> place instructions, that will save some magic value to rtc (to alarm
> >>>>>>> registers that I know are preserved during boot cycle), and I 
> >>>>>>> discovered   sad thing that first time bios does pass control to 
> >>>>>>> linux, but second time
> >>>>>>> (when it hangs), it doesn't.
> >>>>>>>
> >>>>>>> I tried to update bios, and I got same results.
> >>>>>>>
> >>>>>>> Of course it does work with that @#$%^& OS
> >>>>>> So we're doing something wrong.  Please try the appended patch.
> >>>>> Thanks a lot, but this didn't help.
> >>>>>
> >>>>> It still has same pattern, first suspend/resume works perfectly, second 
> >>>>> suspend/resume hangs hard.
> >>>>> It always happens like this, first resume always work (unless I turn off 
> >>>>> smp in kernel (I test this again), or reserve all low memory)
> >>>>>
> >>>>> Also note that if I suspend the system to ram, resume, and then suspend 
> >>>>> to disk, then I can suspend to ram and resume, it seems that
> >>>>>
> >>>>> on suspend to ram cycle somehow arms BIOS or something else, so second 
> >>>>> resume in a row doesn't work.
> >>>>>
> >>>>> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
> >>>>> fan on when running 64-bit version, I could update it, and I know that 
> >>>>> fan issue is fixed there, but new bios introduces bigger bug, namely it 
> >>>>> makes fan to run almost always regardless of 32/64 type of os.
> >>>>> And it doesn't fix this suspend/resume issue, I tested this. I could 
> >>>>> start/stop fan manually with a script, but this could fail, and maybe I 
> >>>>> will do so someday.)
> >>>>>
> >>>>> The bugzilla seems to be unrelated here, since bios does pass control 
> >>>>> there, but corrupts memory.
> >>>>> Here I also have seen that bios corrupts memory, but everything resumes 
> >>>>> fine first time, and on second time,
> >>>>> bios doesn't pass control (I put set of instructions in beginning of 
> >>>>> wakeup real mode assembly file, no page tables, GDT/LDT are used there)
> >>>> I did same test for kernel without SMP, yes it hangs on first resume, but bios
> >>>> does pass control to linux, so while this is a minor bug, it is unrelated.
> >>> Still, I'd be interested in debugging this one too, if possible.  That may be
> >>> easier too. ;-)
> >> I take a look at that.
> >>
> >>>> I also tested noapic, pci=nommconf. No luck.
> >>>>
> >>>> Pattern is always the same, first resume works always, second doesn't.
> >>>> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
> >>>> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
> >>>> which has a sigmatel codec)
> >>> If you have more than 2 GB of RAM, you can try iommu=soft .
> >>>
> >>> I guess that all of the /sys/power/pm_test tests are passed?
> >> Well, I didn't run /sys/power/pm_test. 
> >> But this system has rock solid suspend to disk, I use it always.
> > 
> > Please look at http://bugzilla.kernel.org/show_bug.cgi?id=11415 .
> 
> Hi,
> 
> 
> I took a look there, but it doesn't seem to be similar to my issue,
> my issue is much bigger :-(
> 
> They tell that 2.6.24 works, but here nothing works, I was never able to do
> two suspends in row.
> 
> What I did find interesting was that they mention hardware locks of several kind there, so I am thinking
> could that be related to EC code, could it be that EC code confuses it somehow, so next boot doesn't work?
> Some hardware lock that kernel forgets to unlock, and that prevents bios from resume
> 
> Here ec switches to polled mode almost instantly, due to that bogus 'interrupt storm', I tried to increase interrupt threshold,
> and no more polled mode, but nether working second resume :-(

Have you tried the patch from http://bugzilla.kernel.org/show_bug.cgi?id=10724#c142 ?

Rafael

Re: I need some serious help to debug suspend to ram problem

[Maxim Levitsky]

Rafael J. Wysocki wrote:
> On Saturday, 27 of September 2008, Maxim Levitsky wrote:
>> Rafael J. Wysocki wrote:
>>> On Monday, 22 of September 2008, Maxim Levitsky wrote:
>>>> Rafael J. Wysocki wrote:
>>>>> On Sunday, 21 of September 2008, Maxim Levitsky wrote:
>>>>>> Maxim Levitsky wrote:
>>>>>>> Rafael J. Wysocki wrote:
>>>>>>>> On Saturday, 20 of September 2008, Maxim Levitsky wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I hit a dead end when trying to understand why my notebook can't 
>>>>>>>>> resume from suspend to ram
>>>>>>>>> if this is done two times a row.
>>>>>>>>>
>>>>>>>>> Single suspend/resume cycle works almost perfectly (beep that goes 
>>>>>>>>> through the sound card is muted... no morse code for me... :-(
>>>>>>>>>
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all 
>>>>>>>>> experimental option like nohz
>>>>>>>>> turned off)
>>>>>>>>>
>>>>>>>>> But I had to turn SMP, since without it system won't resume first 
>>>>>>>>> time I suspend it.
>>>>>>>>> (How could this affect suspend?)
>>>>>>>> It could if the system is 64-bit.  In which case please have a look at
>>>>>>>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
>>>>>>>>
>>>>>>>>> With SMP and minimal kernel (of course  no closed drivers), I get 
>>>>>>>>> same behavior,
>>>>>>>>> first resume works second hangs.
>>>>>>>>>
>>>>>>>>> I then added some debug code to real mode wakeup code, I put there in 
>>>>>>>>> first
>>>>>>>>> place instructions, that will save some magic value to rtc (to alarm
>>>>>>>>> registers that I know are preserved during boot cycle), and I 
>>>>>>>>> discovered   sad thing that first time bios does pass control to 
>>>>>>>>> linux, but second time
>>>>>>>>> (when it hangs), it doesn't.
>>>>>>>>>
>>>>>>>>> I tried to update bios, and I got same results.
>>>>>>>>>
>>>>>>>>> Of course it does work with that @#$%^& OS
>>>>>>>> So we're doing something wrong.  Please try the appended patch.
>>>>>>> Thanks a lot, but this didn't help.
>>>>>>>
>>>>>>> It still has same pattern, first suspend/resume works perfectly, second 
>>>>>>> suspend/resume hangs hard.
>>>>>>> It always happens like this, first resume always work (unless I turn off 
>>>>>>> smp in kernel (I test this again), or reserve all low memory)
>>>>>>>
>>>>>>> Also note that if I suspend the system to ram, resume, and then suspend 
>>>>>>> to disk, then I can suspend to ram and resume, it seems that
>>>>>>>
>>>>>>> on suspend to ram cycle somehow arms BIOS or something else, so second 
>>>>>>> resume in a row doesn't work.
>>>>>>>
>>>>>>> I run 32 bit kernel here, this is a long story (this bios doesn't turn 
>>>>>>> fan on when running 64-bit version, I could update it, and I know that 
>>>>>>> fan issue is fixed there, but new bios introduces bigger bug, namely it 
>>>>>>> makes fan to run almost always regardless of 32/64 type of os.
>>>>>>> And it doesn't fix this suspend/resume issue, I tested this. I could 
>>>>>>> start/stop fan manually with a script, but this could fail, and maybe I 
>>>>>>> will do so someday.)
>>>>>>>
>>>>>>> The bugzilla seems to be unrelated here, since bios does pass control 
>>>>>>> there, but corrupts memory.
>>>>>>> Here I also have seen that bios corrupts memory, but everything resumes 
>>>>>>> fine first time, and on second time,
>>>>>>> bios doesn't pass control (I put set of instructions in beginning of 
>>>>>>> wakeup real mode assembly file, no page tables, GDT/LDT are used there)
>>>>>> I did same test for kernel without SMP, yes it hangs on first resume, but bios
>>>>>> does pass control to linux, so while this is a minor bug, it is unrelated.
>>>>> Still, I'd be interested in debugging this one too, if possible.  That may be
>>>>> easier too. ;-)
>>>> I take a look at that.
>>>>
>>>>>> I also tested noapic, pci=nommconf. No luck.
>>>>>>
>>>>>> Pattern is always the same, first resume works always, second doesn't.
>>>>>> It is sad since first resume is almost perfect (when I have free time I need to look at sound codec datasheet
>>>>>> and fix few issues there, anyways here alsa has few issues, all this is trivial, I already fixed all issues with desktop
>>>>>> which has a sigmatel codec)
>>>>> If you have more than 2 GB of RAM, you can try iommu=soft .
>>>>>
>>>>> I guess that all of the /sys/power/pm_test tests are passed?
>>>> Well, I didn't run /sys/power/pm_test. 
>>>> But this system has rock solid suspend to disk, I use it always.
>>> Please look at http://bugzilla.kernel.org/show_bug.cgi?id=11415 .
>> Hi,
>>
>>
>> I took a look there, but it doesn't seem to be similar to my issue,
>> my issue is much bigger :-(
>>
>> They tell that 2.6.24 works, but here nothing works, I was never able to do
>> two suspends in row.
>>
>> What I did find interesting was that they mention hardware locks of several kind there, so I am thinking
>> could that be related to EC code, could it be that EC code confuses it somehow, so next boot doesn't work?
>> Some hardware lock that kernel forgets to unlock, and that prevents bios from resume
>>
>> Here ec switches to polled mode almost instantly, due to that bogus 'interrupt storm', I tried to increase interrupt threshold,
>> and no more polled mode, but nether working second resume :-(
> 
> Have you tried the patch from http://bugzilla.kernel.org/show_bug.cgi?id=10724#c142 ?
> 

No, but just did,
EC storm issue is gone, but thats all that is gone, still same hang after second resume

No ec storm message after first resume ether,  now no error messages at all in dmesg....

Best regards,
	Maxim Levitsky

Re: I need some serious help to debug suspend to ram problem

[Pavel Machek]

Hi!

>>> I hit a dead end when trying to understand why my notebook can't 
>>> resume from suspend to ram
>>> if this is done two times a row.
>>>
>>> Single suspend/resume cycle works almost perfectly (beep that goes 
>>> through the sound card is muted... no morse code for me... :-(
>>>
>>> )
>>>
>>> I compiled a minimal kernel (absolutely nothing but disk drivers, all experimental option like nohz
>>> turned off)
>>>
>>> But I had to turn SMP, since without it system won't resume first time I suspend it.
>>> (How could this affect suspend?)
>>
>> It could if the system is 64-bit.  In which case please have a look at
>> http://bugzilla.kernel.org/show_bug.cgi?id=11237
>>
>>> With SMP and minimal kernel (of course  no closed drivers), I get same behavior,
>>> first resume works second hangs.
>>>
>>> I then added some debug code to real mode wakeup code, I put there in first
>>> place instructions, that will save some magic value to rtc (to alarm
>>> registers that I know are preserved during boot cycle), and I 
>>> discovered   sad thing that first time bios does pass control to 
>>> linux, but second time
>>> (when it hangs), it doesn't. 
>>>
>>>
>>> I tried to update bios, and I got same results.
>>>
>>> Of course it does work with that @#$%^& OS
>>
>> So we're doing something wrong.  Please try the appended patch.
>>
>>> I then proceeded to test recently posted low memory corruption patch, and
>>> it did show that that @#$%^& BIOS does corrupt low memory I then 
>>> reserved all low memory, but system began to hand after first 
>>> suspend,
>>> in exactly same way, but as expected I soon discovered, that that forces real
>>> mode page to be above 1M, ok, then I reserved almost all low memory except
>>> 100K window in the middle, so low allocations will work, but be placed in
>>> region bios less likely to corrupt, and still that didn't help, still 
>>> same hang.    
>
> More information, I compiled kernels back to 2.6.19, and they all have exactly the same issue.
>

I must say you have done quite a lot of homework...

Ok, it looks like BIOS hangs before it can pass control to kernel. I
have seen it once before, and it was before we were using i8259 and
BIOS expected us to use ioapic (or vice-versa; its *long* time ago).

Good luck...
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html