RE: ACPICA regression checker

RE: ACPICA regression checker

[Brown, Len]

[adding linux-acpi@vger.kernel.org]

Thomas,
I like the way you think.

my reply is below your message...

-----Original Message-----
From: Thomas Renninger [mailto:trenn@suse.de] 
Sent: Monday, July 10, 2006 10:23 AM
To: Brown, Len
Cc: Moore, Robert; intel-linux-acpi@linux.intel.com; Yu, Ling L
Subject: ACPICA regression checker

Hi,

I like to build up a pool of ACPI BIOSes to automatically check them
with new versions of iasl compiler/disassembler.

Therefore I am asking for some ideas whether my approach is OK, or maybe
someone can give some enhancements or just has a good idea.

First I thought about a new mailing list people can send their acpidump
to and I extract needed info, put it in a directory on my ftp where
everybody can see some statistics or can run the validations on his own.

Then the already existing html/php front-end on acpi.sf.net came into my
mind. If this is OK with you, I'll try to write some kind of wget
wrapper to upload/download DSDTs more or less automatically (hmm, just
thought about: do people load their whole acpidump up there or just
DSDT? In second case we have a problem). I like to add some dozens
(hundreds?) of BIOSes which are not meant to get fixed (as the current
main purpose there ATM?), but as a playground for ACPI developers to see
some weird/huge IA64 DSDTs, check whether modifications still
compile/disassemble correctly on all BIOSes and so on.

The only test is to disassemble and compile with iasl, parse
warning/error messages and display some statistics (compared to previous
versions, ..) of the results.

So a final run could show something like:

            iasl ver 01012006   |  iasl ver 0x0y2007 | diff ver1/ver2?
          -------------------------------------------------------------
Warnings                        |                    |
------------------------------------------------------------------------
Errors                          |                    |

Additional errors/warnings in xy DSDTs
Fixed xy   errors/warnings in xy different DSDTs
... ??? ...

I got at least two bugs that are regressions (or better missing
workarounds). I expect that with above checking those could have find at
implementation time.


To make this work properly I would first try to implement what Robert
suggested to avoid compile/disassemble errors with additional SSDTs:
http://rudin.suse.de:8888/show_bug.cgi?id=179702 (comment #21):
-------------
The only real solution (that I can think of) to the disassembly problem
(external methods) is probably something like allowing the specification
of
additional tables in order to resolve the methods. 

Something like:

iasl -e dsdt.aml -d ssdt.aml
--------------
Not sure whether someone already started with this one, if not I'll give
it a try...

What do you think about all this?
Better use a separate mailing list or web server?

Thanks,

        Thomas
------------

This is a good line of thought.
But first, There are a couple of things going on
that maybe not everybody is aware of.

1.
the functional tests that Yu-Ling has published under Validation
on acpi.sourcforge.net.  These are feature-level tests intended
to test the end product.  eg. a release of SuSE Linux.
We run these internally, and I've heard that some distros
are interested in running them too.  These tests need additional
work to automate and to cover more cases.

2.
Over the last year, Intel has developed an ASL Test Suite consisting
of over 1700 tests.   These are AML tests that run on top the ACPICA
interpreter used by Linux, but compiled as a standalone application.
This found over 230 issues -- about 60 are still
unresolved.  All remaining open issues are filed in bugzilla.
You may have seen about 30 of these in the AML-Interpreter
subcategory under ACPI in bugzilla.kernel.org.  You may have
noticed in my latest check-ins for ACPICA that Bob's notes refer
to BZ 300 or so -- these are additional bug reports we filed
in an internal bugzilla for bugs that didn't look easy to
describe as Linux kernel bugs -- which is all that bugzilla.kernel.org
allows.

We'll be publishing this test suite along with ACPICA very soon --
indeed, the paperwork is already in the pipeline.

We now run this test suite before every ACPICA release.
I'd like to move to running it between every ACPICA check-in,
which I'd like to check-into Linux individually.

3.
Platform ACPI compliance Test
This is sorely needed by the community to replace WHQL as the test
that vendors run to make sure they got their ACPI implementation right.
It would be a turn-key test that runs on a platform and gives
a yes/no on compliance, and diagnostics on failures.

This would be a voluntary thing for vendors to run, but I expect
that many of them would run it if somebody else wrote the tests
for them.

Some folks at Intel have given this some thought, but it isn't
currently a committed/funded project, so I can't promise anything.

My expectation that a reasonable implementation would be a loadable
Linux device driver that could use any/all kernel hooks and talk to
the real hardware via ACPI.

---
That brings me to your idea.
I think it works for testing the ASL compiler.
However, the problem with using DSDT images to test the interpreter
is that unless they are running on the real hardware,
you can't really see if they work.  You really
need to do #3 to get where you want to be -- run the test
and the firmware on the HW it was written for.

The other problem with DSDTs is that it is taboo for Intel
to traffic in firmware images that are products of its customers --
-- this could potentially piss off some of our customers.
I've never posted a DSDT to sourceforge.net, and I'm really
not permitted to.

thanks,
-Len

ps. please point me to the two bugs -- maybe they show a hole
in our ASL test suite.

RE: ACPICA regression checker

[Thomas Renninger]

On Tue, 2006-07-11 at 22:01 -0400, Brown, Len wrote:
> [adding linux-acpi@vger.kernel.org]
> 
> Thomas,
> I like the way you think.
> 
> my reply is below your message...
> 
> -----Original Message-----
> From: Thomas Renninger [mailto:trenn@suse.de] 
> Sent: Monday, July 10, 2006 10:23 AM
> To: Brown, Len
> Cc: Moore, Robert; intel-linux-acpi@linux.intel.com; Yu, Ling L
> Subject: ACPICA regression checker
> 
> Hi,
> 
> I like to build up a pool of ACPI BIOSes to automatically check them
> with new versions of iasl compiler/disassembler.
> 
> Therefore I am asking for some ideas whether my approach is OK, or maybe
> someone can give some enhancements or just has a good idea.
> 
> First I thought about a new mailing list people can send their acpidump
> to and I extract needed info, put it in a directory on my ftp where
> everybody can see some statistics or can run the validations on his own.
> 
> Then the already existing html/php front-end on acpi.sf.net came into my
> mind. If this is OK with you, I'll try to write some kind of wget
> wrapper to upload/download DSDTs more or less automatically (hmm, just
> thought about: do people load their whole acpidump up there or just
> DSDT? In second case we have a problem). I like to add some dozens
> (hundreds?) of BIOSes which are not meant to get fixed (as the current
> main purpose there ATM?), but as a playground for ACPI developers to see
> some weird/huge IA64 DSDTs, check whether modifications still
> compile/disassemble correctly on all BIOSes and so on.
> 
> The only test is to disassemble and compile with iasl, parse
> warning/error messages and display some statistics (compared to previous
> versions, ..) of the results.
> 
> So a final run could show something like:
> 
>             iasl ver 01012006   |  iasl ver 0x0y2007 | diff ver1/ver2?
>           -------------------------------------------------------------
> Warnings                        |                    |
> ------------------------------------------------------------------------
> Errors                          |                    |
> 
> Additional errors/warnings in xy DSDTs
> Fixed xy   errors/warnings in xy different DSDTs
> ... ??? ...
> 
> I got at least two bugs that are regressions (or better missing
> workarounds). I expect that with above checking those could have find at
> implementation time.
> 
> 
> To make this work properly I would first try to implement what Robert
> suggested to avoid compile/disassemble errors with additional SSDTs:
> http://rudin.suse.de:8888/show_bug.cgi?id=179702 (comment #21):
> -------------
> The only real solution (that I can think of) to the disassembly problem
> (external methods) is probably something like allowing the specification
> of
> additional tables in order to resolve the methods. 
> 
> Something like:
> 
> iasl -e dsdt.aml -d ssdt.aml
> --------------
> Not sure whether someone already started with this one, if not I'll give
> it a try...
> 
> What do you think about all this?
> Better use a separate mailing list or web server?
> 
> Thanks,
> 
>         Thomas
> ------------
> 
> This is a good line of thought.
> But first, There are a couple of things going on
> that maybe not everybody is aware of.
> 
> 1.
> the functional tests that Yu-Ling has published under Validation
> on acpi.sourcforge.net.  These are feature-level tests intended
> to test the end product.  eg. a release of SuSE Linux.
> We run these internally, and I've heard that some distros
> are interested in running them too.  These tests need additional
> work to automate and to cover more cases.
I already have an internal package for this.
Still I also like to add a "disassemble and recompile and report
errors/warnings tool" for ACPI validation. My experience is that most
(at least a lot) potential (unfortunately only syntactical) bugs are
found by that.
> 
> 2.
> Over the last year, Intel has developed an ASL Test Suite consisting
> of over 1700 tests.   These are AML tests that run on top the ACPICA
> interpreter used by Linux, but compiled as a standalone application.
> This found over 230 issues -- about 60 are still
> unresolved.  All remaining open issues are filed in bugzilla.
> You may have seen about 30 of these in the AML-Interpreter
> subcategory under ACPI in bugzilla.kernel.org.  You may have
> noticed in my latest check-ins for ACPICA that Bob's notes refer
> to BZ 300 or so -- these are additional bug reports we filed
> in an internal bugzilla for bugs that didn't look easy to
> describe as Linux kernel bugs -- which is all that bugzilla.kernel.org
> allows.
> 
> We'll be publishing this test suite along with ACPICA very soon --
> indeed, the paperwork is already in the pipeline.
Great.
> 
> We now run this test suite before every ACPICA release.
> I'd like to move to running it between every ACPICA check-in,
> which I'd like to check-into Linux individually.
But these are some generated corner cases, not real BIOS parts?
This sounds similar to what I like to do, but with (hopefully all :) )
real BIOSes out there. Especially some weird Windows/vendor workarounds
could be outlined by that.
> 
> 3.
> Platform ACPI compliance Test
> This is sorely needed by the community to replace WHQL as the test
> that vendors run to make sure they got their ACPI implementation right.
(WHQL? Windows Head Quarters L..?)
> It would be a turn-key test that runs on a platform and gives
> a yes/no on compliance, and diagnostics on failures.
> 
> This would be a voluntary thing for vendors to run, but I expect
> that many of them would run it if somebody else wrote the tests
> for them.
I doubt that many vendors will run it. Say some vendors on some of their
servers they like to Linux certificate.
> 
> Some folks at Intel have given this some thought, but it isn't
> currently a committed/funded project, so I can't promise anything.
> 
> My expectation that a reasonable implementation would be a loadable
> Linux device driver that could use any/all kernel hooks and talk to
> the real hardware via ACPI.
A Windows/Linux/Other OS compliance test? Probably hard to implement?
> 
> ---
> That brings me to your idea.
> I think it works for testing the ASL compiler.
> However, the problem with using DSDT images to test the interpreter
> is that unless they are running on the real hardware,
> you can't really see if they work.
In the past you often got a clue that something is bogus with your
tables. You can at least say: "This one is critical, something probably
won't work" or "This error is harmless" or "This might cause problems".
Of course there might be hidden things...

> You really
> need to do #3 to get where you want to be -- run the test
> and the firmware on the HW it was written for.
But this is not possible for all hardware (including already sold one).
Even new hardware will only be tested by vendors for some percent of
machines (the servers they like to linux certificate). Most
workstations/laptops we will still be on our own.
In the end it would be the user himself who needs to run this if he
experience problems in ACPI parts.
> 
> The other problem with DSDTs is that it is taboo for Intel
> to traffic in firmware images that are products of its customers --
> -- this could potentially piss off some of our customers.
> I've never posted a DSDT to sourceforge.net, and I'm really
> not permitted to.
As mentioned, this is not intended to fix DSDTs, but:
 a) to find compatibility problems and regressions early. The whole
    ACPI thing is a huge, complicated struct of code with lots of
    workarounds and so on. Fixing one thing might break old BIOSes.
 b) a playground for ACPI developers. We could see some vendor/windows
    specific functions on some hardware (especially laptops, I am
    thinking about hotkey, bluetooth/wlan/display control, ...).
    This won't be enough to implement their functionality, but maybe
    at least find a concept how to get this running in future or 
    whatever...

What hangs in my mind all the time is some kind of function/exec tracker
that logs all ACPI kernel activity at runtime. The output should be some
debug file which acpiexec can be filled with and regenerate the ACPI
paths that were executed.
Like that you can tell a person to switch on ACPI debug. Then send
acpidump and debug file. Then you can "offline debug" what was executed
and even know what hardware accesses returned.
But this is probably not easy to implement. I really like to invest some
time here, not sure how far I could come...
The last step then would be another debug module that allows calling of
arbitrary ACPI functions (e.g. that special hotkey/fan/.. stuff that
normally is not used). I wrote a small module to export EC variables to
sysfs. That was easy and could be done for all ACPI funcs. Of course
only for debug purposes.
These are only ideas yet...
> 
> thanks,
> -Len
> 
> ps. please point me to the two bugs -- maybe they show a hole
> in our ASL test suite.

http://rudin.suse.de:8888/show_bug.cgi?id=189488
-> Bogus _PSS package, customer says that already worked. Maybe with PSB
fallback to use legacy AMD tables, don't know... If that already worked
(with ACPI) we should have seen the regression by simple
disassembling/recompiling.

http://rudin.suse.de:8888/show_bug.cgi?id=160671
-> Ok, this one would not have been found by simple
disassembling/recompiling.

Nothing critical, not really worth looking at.

Thanks,

      Thomas

RE: ACPICA regression checker

[Thomas Renninger]

I didn't see this is going to linux-acpi... these links are using an
internal proxy the outside address is bugzilla.novell.com:

> http://rudin.suse.de:8888/show_bug.cgi?id=189488
https://bugzilla.novell.com/show_bug.cgi?id=189488
> -> Bogus _PSS package, customer says that already worked. Maybe with PSB
> fallback to use legacy AMD tables, don't know... If that already worked
> (with ACPI) we should have seen the regression by simple
> disassembling/recompiling.
> 
> http://rudin.suse.de:8888/show_bug.cgi?id=160671
https://bugzilla.novell.com/show_bug.cgi?id=160671
> -> Ok, this one would not have been found by simple
> disassembling/recompiling.
> 
> Nothing critical, not really worth looking at.

    Thomas

RE: ACPICA regression checker

[Brown, Len]

>> 1.
>> the functional tests that Yu-Ling has published under Validation
>> on acpi.sourcforge.net.  These are feature-level tests intended
>> to test the end product.  eg. a release of SuSE Linux.
>> We run these internally, and I've heard that some distros
>> are interested in running them too.  These tests need additional
>> work to automate and to cover more cases.

>I already have an internal package for this.

>Still I also like to add a "disassemble and recompile and report
>errors/warnings tool" for ACPI validation. My experience is that most
>(at least a lot) potential (unfortunately only syntactical) bugs are
>found by that.

DSDT disassemble/re-compile is a great test of the BIOS.
Indeed, we recommend to BIOS vendors that even if they us the MS
compiler for production, that they use ours for static error checking.

However, the compiler isn't directly related to how the interpreter
will handle a DSDT.  For example, the compiler will ERROR out on
implicit return, but the interpreter has a work-around to handle that.

But that doesn't mean the end of your idea.  It may be that we
want to cook up a special test version of the compiler that can
be used to behave like the interpreter to look for issues that
will crop-up when the interpreter is faced with the DSDT.


>> 2.
>> Over the last year, Intel has developed an ASL Test Suite consisting
>> of over 1700 tests.   These are AML tests that run on top the ACPICA
>> interpreter used by Linux, but compiled as a standalone application.
>> This found over 230 issues -- about 60 are still
>> unresolved.  All remaining open issues are filed in bugzilla.
>> You may have seen about 30 of these in the AML-Interpreter
>> subcategory under ACPI in bugzilla.kernel.org.  You may have
>> noticed in my latest check-ins for ACPICA that Bob's notes refer
>> to BZ 300 or so -- these are additional bug reports we filed
>> in an internal bugzilla for bugs that didn't look easy to
>> describe as Linux kernel bugs -- which is all that 
>> bugzilla.kernel.org allows.
>> 
>> We'll be publishing this test suite along with ACPICA very soon --
>> indeed, the paperwork is already in the pipeline.

>Great.
 
>> We now run this test suite before every ACPICA release.
>> I'd like to move to running it between every ACPICA check-in,
>> which I'd like to check-into Linux individually.

>But these are some generated corner cases, not real BIOS parts?
>This sounds similar to what I like to do, but with (hopefully all :) )
>real BIOSes out there. Especially some weird Windows/vendor workarounds
>could be outlined by that.

Right, this is testing against the language specification itself
using a large list of small synthetic tests.  I think it has helped
enormously over the past year in reducing regressions at the AML
level.  However, it doesn't look for strange things that programmers
might do in a production DSDT.  BTW. the strange things are generally
BIOS programmer mistakes that are simply not caught by the MS compiler
and by chance happen to not break the MS interpreter.  It doesn't break,
and the BIOS writer declares victory and moveds on to the next
project...

>> 3.
>> Platform ACPI compliance Test
>> This is sorely needed by the community to replace WHQL as the test
>> that vendors run to make sure they got their ACPI 
>> implementation right.

>(WHQL? Windows Head Quarters L..?)

WHQL is the test that HW vendors must pass to get a Windows Logo
on their box.  Without a Windows Logo, they basically have no product.
I think it stands for Windows HW Quality Lab.

>> It would be a turn-key test that runs on a platform and gives
>> a yes/no on compliance, and diagnostics on failures.
>> 
>> This would be a voluntary thing for vendors to run, but I expect
>> that many of them would run it if somebody else wrote the tests
>> for them.

>I doubt that many vendors will run it. Say some vendors on 
>some of their servers they like to Linux certificate.

"If you build it, they will come":-)
Yes, I agree,vendors who care only about Windows will not bother.
However, it is critical that those vendors that _do_ care
about Linux have the tests to run -- and there are
some significant vendors that do care very much about Linux...

>> Some folks at Intel have given this some thought, but it isn't
>> currently a committed/funded project, so I can't promise anything.
>> 
>> My expectation that a reasonable implementation would be a loadable
>> Linux device driver that could use any/all kernel hooks and talk to
>> the real hardware via ACPI.

>A Windows/Linux/Other OS compliance test? Probably hard to implement?

No, it is an ACPI specification platform compliance test --
it would test how the hardware+firmware platform implements
the ACPI specification.

It doesn't say anything about Windows or Linux, but it will
happen to be implemented using Linux b/c that is both open,
and the tool we have on hand.

>> That brings me to your idea.
>> I think it works for testing the ASL compiler.
>> However, the problem with using DSDT images to test the interpreter
>> is that unless they are running on the real hardware,
>> you can't really see if they work.

>In the past you often got a clue that something is bogus with your
>tables. You can at least say: "This one is critical, something probably
>won't work" or "This error is harmless" or "This might cause problems".
>Of course there might be hidden things...

Yes, per above, I think it would be reasonable to customize the
compiler so that it could have a mode where it would be focused
on its job as a static AML checker looking for incompatibilities
with our interpreter.  I think it happens to do some of this now,
but it will complain about stuff that the interpreter has workarounds
for.  (and these workarounds must stay forever for the installed base).



>> You really
>> need to do #3 to get where you want to be -- run the test
>> and the firmware on the HW it was written for.

>But this is not possible for all hardware (including already sold one).
>Even new hardware will only be tested by vendors for some percent of
>machines (the servers they like to linux certificate). Most
>workstations/laptops we will still be on our own.
>In the end it would be the user himself who needs to run this if he
>experience problems in ACPI parts.

I think both would have value.  Running on the HW can check more things
(including everything that examining the DSDT off the hardware can do).
I agree that checking the DSDT off the hardware is more practical.

>> The other problem with DSDTs is that it is taboo for Intel
>> to traffic in firmware images that are products of its customers --
>> -- this could potentially piss off some of our customers.
>> I've never posted a DSDT to sourceforge.net, and I'm really
>> not permitted to.

>As mentioned, this is not intended to fix DSDTs, but:
> a) to find compatibility problems and regressions early. The whole
>    ACPI thing is a huge, complicated struct of code with lots of
>    workarounds and so on. Fixing one thing might break old BIOSes.
> b) a playground for ACPI developers. We could see some vendor/windows
>    specific functions on some hardware (especially laptops, I am
>    thinking about hotkey, bluetooth/wlan/display control, ...).
>    This won't be enough to implement their functionality, but maybe
>    at least find a concept how to get this running in future or 
>    whatever...

Again, this is very close to a line that I'm not permitted to cross
as an Intel employee.

>What hangs in my mind all the time is some kind of 
>function/exec tracker that logs all ACPI kernel activity at runtime.
>The output should be some
>debug file which acpiexec can be filled with and regenerate the ACPI
>paths that were executed.
>
>Like that you can tell a person to switch on ACPI debug. Then send
>acpidump and debug file. Then you can "offline debug" what was executed
>and even know what hardware accesses returned.
>But this is probably not easy to implement. I really like to 
>invest some time here, not sure how far I could come...

Interesting idea.  Bob should comment on this.
Today we basically we do this with a very large text log
and a very smart human reading.

We actually don't have to use that debugging method too often --
it is sort of the last resort, buy yes, automating it would
be pretty cool if that is possible.

>The last step then would be another debug module that allows calling of
>arbitrary ACPI functions (e.g. that special hotkey/fan/.. stuff that
>normally is not used). I wrote a small module to export EC variables to
>sysfs. That was easy and could be done for all ACPI funcs. Of course
>only for debug purposes.

These linux-hosted tests, that may use a debug device driver, that
run on the real hardware is exactly what I was referring to in #3.
Luming is thinking about this.


>http://rudin.suse.de:8888/show_bug.cgi?id=189488
>-> Bogus _PSS package, customer says that already worked. 
>Maybe with PSB
>fallback to use legacy AMD tables, don't know... If that already worked
>(with ACPI) we should have seen the regression by simple
>disassembling/recompiling.

https://bugzilla.novell.com/show_bug.cgi?id=189488

I'll look at this.

thanks,
-Len

>http://rudin.suse.de:8888/show_bug.cgi?id=160671
>-> Ok, this one would not have been found by simple
>disassembling/recompiling.
>
>Nothing critical, not really worth looking at.
>
>Thanks,
>
>      Thomas
>

Re: ACPICA regression checker

[Matthew Garrett]

On Tue, Jul 11, 2006 at 10:01:30PM -0400, Brown, Len wrote:

> Platform ACPI compliance Test
> This is sorely needed by the community to replace WHQL as the test
> that vendors run to make sure they got their ACPI implementation right.
> It would be a turn-key test that runs on a platform and gives
> a yes/no on compliance, and diagnostics on failures.

Based on our previous conversations, would this require that platforms 
reinitialise video themselves? And, what's probably a harder question, 
is there any real chance of there being sufficient political will in 
Intel to make passing this a requirement for claiming ACPI compliance?

-- 
Matthew Garrett | mjg59@srcf.ucam.org

RE: ACPICA regression checker

[Brown, Len]

>On Tue, Jul 11, 2006 at 10:01:30PM -0400, Brown, Len wrote:
>
>> Platform ACPI compliance Test
>> This is sorely needed by the community to replace WHQL as the test
>> that vendors run to make sure they got their ACPI 
>> implementation right.
>> It would be a turn-key test that runs on a platform and gives
>> a yes/no on compliance, and diagnostics on failures.
>
>Based on our previous conversations, would this require that platforms 
>reinitialise video themselves?

That is a good example of what should be verified, though I don't
know of an automatic way to test it.

Note that Intel already tells the BIOS vendors this when we tell them
what we think they should do, and we do this in our example designs.
(though 18 months ago that was not the case).

On systems where the vendor cares about Linux enough
to try to get a Centrino Mobile Technology brand, there is a simple
compliance test (not fancy like WHQL) and they will fail if they
don't re-initialize video on resume from S3.

Of course the population of vendors who ship Linux on their laptops
is small, and the subset of them that go for a CMT logo is even smaller.
But those are the ones for us to focus on, it is futile to try to
get a Windows-only vendor to lift a finger to support Linux.

>And, what's probably a harder question, 
>is there any real chance of there being sufficient political will in 
>Intel to make passing this a requirement for claiming ACPI compliance?

My view is that it is impossible for anybody to claim ACPI compliance
because there is no ACPI compliance test to certify it.

I doubt that a test based on Linux would fly in the committee that
defines
the ACPI spec -- particularly since the Linux distributors have not even
bothered to join the commmitte and get a vote at the table.

However, what we really do care about here are the vendors that _do_
care about Linux.  They would be interested in running a Linux based
test if they were available.

-Len