LSB inclusion of ALSA

LSB inclusion of ALSA

[Robert Schweikert]

Hi,

Currently ALSA is a trial use module for LSB
(http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it
is the intention to make ALSA mandatory with the LSB 4.1 release
targeted for Q1 2010. One of the requirements of a mandatory module is
the existence of a test suite for the module that covers a good chunk of
the interfaces provided by the LSB spec. An initial exploratory look at
the existing test cases showed that tests appear to be tied to hardware
drivers and are interactive.

Is this initial impression correct?

>From an LSB testing perspective, interface testing is  very important;
i.e. I call the interface with the arguments required and I get the
expected result back. Further the tests need to run in an automated
fashion and produce a certain output (automated processing). The output
can be created by retrofitting existing tests or by specifically writing
tests within the LSB test framework.

IMHO it would be best to have tests that are useful to both, the project
and the LSB and in a best case scenario we wouldn't have to start over
as the existing tests already cover a good part of the interface.

With this as background there are a few questions that have been raised
while discussing this during LSB workgroup sessions.

1.) Is there interest from the community side to participate in this
effort and accept patches?
2.) Is it reasonable to expect that the existing tests can be used in
some way by using a dummy sound device or a sound loop device to verify
output?
One requirement for LSB testing is that a test has only two states, pass
or fail, and a recovery or try again mode is not supported. Further a
test must pass or fail.
3.) Would the community be more comfortable if this is an effort that
creates new tests separate from the existing HW focused tests?
4.) Is anyone interested in helping with this effort, writing tests,
answering questions about ALSA for those not familiar with the interface
and or sound in general?

Comments, thoughts, etc. are much appreciated.

Thanks,
Robert

-- 
Robert Schweikert                           MAY THE SOURCE BE WITH YOU
Software Engineer Consultant                          LINUX
rschweikert@novell.com 
781-464-8147

Novell
Making IT Work As One

Re: LSB inclusion of ALSA

[Clemens Ladisch]

Robert Schweikert wrote:
> Currently ALSA is a trial use module for LSB
> (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it
> is the intention to make ALSA mandatory with the LSB 4.1 release
> targeted for Q1 2010. One of the requirements of a mandatory module is
> the existence of a test suite for the module that covers a good chunk of
> the interfaces provided by the LSB spec. An initial exploratory look at
> the existing test cases showed that tests appear to be tied to hardware
> drivers and are interactive.
> 
> Is this initial impression correct?

Yes.

> From an LSB testing perspective, interface testing is  very important;
> i.e. I call the interface with the arguments required and I get the
> expected result back.

In the case of a hardware interface library like libasound, there is not
always a result that comes back where it could by easily tested.

While many parts of the library can be tested without special hardware
(config, sequencer+MIDI, timers), the parts that are most likely to be
used (PCM, mixer) require at least some kernel driver and contain many
functions that are used to handle hardware differences.

> 1.) Is there interest from the community side to participate in this
> effort and accept patches?

I'm just an individual and cannot speak for "the community", whatever
that is, but there is certainly interest to accept patches.

> 2.) Is it reasonable to expect that the existing tests can be used in
> some way by using a dummy sound device or a sound loop device to verify
> output?

There is a loopback driver, but it is not part of the LSB specification.
This means that a test using this driver would not work with any
alternate ALSA implementation.

I can easily imagine LSB-compliant computers that do not have any sound
card (e.g., most servers).  Even if the ALSA interface (i.e., libasound)
is installed, some parts cannot be used in any meaningful way.  What is
a test supposed to do in this case?  It could just exit with "pass",
but this wouldn't actually _test_ much.

(Hmmm, a dummy ALSA implementation that just returns -ENODEV when the
application tries to open a device would be compliant, because it cannot
be distinguished from the 'real' ALSA with no installed sound card.  :-)

> 3.) Would the community be more comfortable if this is an effort that
> creates new tests separate from the existing HW focused tests?

The current interactive tests are mostly intended to test drivers, not
the interface.  I don't think it would be desirable or even possible to
integrate them into a test suite with strict pass/fail requirements.

> 4.) Is anyone interested in helping with this effort, writing tests,
> answering questions about ALSA for those not familiar with the interface
> and or sound in general?

Yes.

Since most of libasound is somewhat undocumented (the doxygen reference
is just a skeleton), I'm wondering who is going to write the tests ...


Best regards,
Clemens

Re: LSB inclusion of ALSA

[Takashi Iwai]

At Tue, 16 Jun 2009 13:59:45 +0200,
Clemens Ladisch wrote:
> 
> Robert Schweikert wrote:
> > Currently ALSA is a trial use module for LSB
> > (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it
> > is the intention to make ALSA mandatory with the LSB 4.1 release
> > targeted for Q1 2010. One of the requirements of a mandatory module is
> > the existence of a test suite for the module that covers a good chunk of
> > the interfaces provided by the LSB spec. An initial exploratory look at
> > the existing test cases showed that tests appear to be tied to hardware
> > drivers and are interactive.
> > 
> > Is this initial impression correct?
> 
> Yes.
> 
> > From an LSB testing perspective, interface testing is  very important;
> > i.e. I call the interface with the arguments required and I get the
> > expected result back.
> 
> In the case of a hardware interface library like libasound, there is not
> always a result that comes back where it could by easily tested.
> 
> While many parts of the library can be tested without special hardware
> (config, sequencer+MIDI, timers), the parts that are most likely to be
> used (PCM, mixer) require at least some kernel driver and contain many
> functions that are used to handle hardware differences.

True.  That's the very high hurdle that an automatic testing would
face.

> > 1.) Is there interest from the community side to participate in this
> > effort and accept patches?
> 
> I'm just an individual and cannot speak for "the community", whatever
> that is, but there is certainly interest to accept patches.

Good to hear :)

> > 2.) Is it reasonable to expect that the existing tests can be used in
> > some way by using a dummy sound device or a sound loop device to verify
> > output?
> 
> There is a loopback driver, but it is not part of the LSB specification.
> This means that a test using this driver would not work with any
> alternate ALSA implementation.

Yes, I mentioned about this in the last conf call with LSB guys.

Another candidate is alsa-lib file plugin (and null plugin).  They can
be used to simulate in a certain level of I/O.

> I can easily imagine LSB-compliant computers that do not have any sound
> card (e.g., most servers).  Even if the ALSA interface (i.e., libasound)
> is installed, some parts cannot be used in any meaningful way.  What is
> a test supposed to do in this case?  It could just exit with "pass",
> but this wouldn't actually _test_ much.

The test can never cover 100% all real cases.
So, the primary question is: what kind of test do we need.

>From the LSB perspective, I suppose, the API compatibility is the
biggest issue.  The functionality, either working or not on the real
hardware, is a subsystem bug.  And, the subtle tests would be likely
needed done manually with the real hardware.


> (Hmmm, a dummy ALSA implementation that just returns -ENODEV when the
> application tries to open a device would be compliant, because it cannot
> be distinguished from the 'real' ALSA with no installed sound card.  :-)

You mean the dummy sound driver?

> > 3.) Would the community be more comfortable if this is an effort that
> > creates new tests separate from the existing HW focused tests?
> 
> The current interactive tests are mostly intended to test drivers, not
> the interface.  I don't think it would be desirable or even possible to
> integrate them into a test suite with strict pass/fail requirements.

An automatic test would be hard, yes.  But, including "manual tests"
by human makes a lot sense.  For example, providing a set of WAV files
with different format, or giving a script to handle with sox, etc.

In my case, I've been testing like the following:

- Mixer adjustment via alsamixer
- Play with different format bits (8, 16, 24, 32, float), sample
  rates, and channels via aplay
- Play with different access methods (mmap, r/w) via aplay
- Speaker configuration tests via speaker-test
- Buffer size, period size tests via aplay
- Recording via arecord; this is really hard to automate right now...
- SPDIF AC3 output via ac3dec to a surround receiver
- SPDIF input recording by cabling to SPDIF input 

So, most of them are real hardware tests...


> > 4.) Is anyone interested in helping with this effort, writing tests,
> > answering questions about ALSA for those not familiar with the interface
> > and or sound in general?
> 
> Yes.
> 
> Since most of libasound is somewhat undocumented (the doxygen reference
> is just a skeleton), I'm wondering who is going to write the tests ...

Heh, first response, first victim? :)

As mentioned, we need to focus on the definition of what kind of tests
at first.


thanks,

Takashi

Re: LSB inclusion of ALSA

[Clemens Ladisch]

Takashi Iwai wrote:
> Clemens Ladisch wrote:
> > There is a loopback driver, but it is not part of the LSB specification.
> > This means that a test using this driver would not work with any
> > alternate ALSA implementation.
> 
> Yes, I mentioned about this in the last conf call with LSB guys.
> 
> Another candidate is alsa-lib file plugin (and null plugin).  They can
> be used to simulate in a certain level of I/O.

I don't think that LSB guarantees the availability of these plugins
either.  (And they don't get the timing correct, but maybe we don't care
about that.)

> > I can easily imagine LSB-compliant computers that do not have any sound
> > card (e.g., most servers).  Even if the ALSA interface (i.e., libasound)
> > is installed, some parts cannot be used in any meaningful way.  What is
> > a test supposed to do in this case?  It could just exit with "pass",
> > but this wouldn't actually _test_ much.
> 
> The test can never cover 100% all real cases.
> So, the primary question is: what kind of test do we need.
> 
> From the LSB perspective, I suppose, the API compatibility is the
> biggest issue.  The functionality, either working or not on the real
> hardware, is a subsystem bug.  And, the subtle tests would be likely
> needed done manually with the real hardware.

So ALSA's LSB tests would be like the Qt4 tests, whose description is:
| check run-time presence of interfaces and absence of critical errors
| in simple use cases

> > (Hmmm, a dummy ALSA implementation that just returns -ENODEV when the
> > application tries to open a device would be compliant, because it cannot
> > be distinguished from the 'real' ALSA with no installed sound card.  :-)
> 
> You mean the dummy sound driver?

No.  I meant that the ALSA interfaces, in case they are only required
for strict LSB compliance but not for actual sound output, could be
implemented like this to save memory:

	int snd_pcm_open(...)
	{
		return -ENODEV;
	}

This is purely theoretical, but my point is that this would be a
conforming implementation of the ALSA interface.


Best regards,
Clemens

Re: LSB inclusion of ALSA

[Robert Schweikert]

Clemens Ladisch wrote:
> Takashi Iwai wrote:
>   
>> Clemens Ladisch wrote:
>>     
>>> There is a loopback driver, but it is not part of the LSB specification.
>>> This means that a test using this driver would not work with any
>>> alternate ALSA implementation.
>>>       
>> Yes, I mentioned about this in the last conf call with LSB guys.
>>
>> Another candidate is alsa-lib file plugin (and null plugin).  They can
>> be used to simulate in a certain level of I/O.
>>     
>
> I don't think that LSB guarantees the availability of these plugins
> either.  (And they don't get the timing correct, but maybe we don't care
> about that.)
>   
If we need special libraries for testing that's fine. These do not have
to be part of the LSB.
>   
>>> I can easily imagine LSB-compliant computers that do not have any sound
>>> card (e.g., most servers).  Even if the ALSA interface (i.e., libasound)
>>> is installed, some parts cannot be used in any meaningful way.  What is
>>> a test supposed to do in this case?  It could just exit with "pass",
>>> but this wouldn't actually _test_ much.
>>>       
>> The test can never cover 100% all real cases.
>> So, the primary question is: what kind of test do we need.
>>
>> From the LSB perspective, I suppose, the API compatibility is the
>> biggest issue.  The functionality, either working or not on the real
>> hardware, is a subsystem bug.  And, the subtle tests would be likely
>> needed done manually with the real hardware.
>>     
>
> So ALSA's LSB tests would be like the Qt4 tests, whose description is:
> | check run-time presence of interfaces and absence of critical errors
> | in simple use cases
>   
Well, hopefully not quite that rudimentary but definitely in line with
this train of thought. Just like the ALSA project cannot test all
possible configurations, neither can the LSB. From a driver - hardware
perspective the LSB probably can test less than those more deeply
involved in the project.

One of the key parts of any effort is that the tests are accepted by the
project and maintained in some way. If we just create LSB tests that are
separate from the project things will diverge and become stale.
>   
>>> (Hmmm, a dummy ALSA implementation that just returns -ENODEV when the
>>> application tries to open a device would be compliant, because it cannot
>>> be distinguished from the 'real' ALSA with no installed sound card.  :-)
>>>       
>> You mean the dummy sound driver?
>>     
>
> No.  I meant that the ALSA interfaces, in case they are only required
> for strict LSB compliance but not for actual sound output, could be
> implemented like this to save memory:
>
> 	int snd_pcm_open(...)
> 	{
> 		return -ENODEV;
> 	}
>
> This is purely theoretical, but my point is that this would be a
> conforming implementation of the ALSA interface.
>   
But this would fail the test where we load the dummy driver and test
another return value (see previous post for more details).
>
> Best regards,
> Clemens
>   

-- 
Robert Schweikert                           MAY THE SOURCE BE WITH YOU
Software Engineer Consultant                          LINUX
rschweikert@novell.com 
781-464-8147

Novell
Making IT Work As One

Re: LSB inclusion of ALSA

[Theodore Tso]

On Wed, Jun 17, 2009 at 10:52:38AM +0200, Clemens Ladisch wrote:
> Takashi Iwai wrote:
> > Yes, I mentioned about this in the last conf call with LSB guys.
> > 
> > Another candidate is alsa-lib file plugin (and null plugin).  They can
> > be used to simulate in a certain level of I/O.
> 
> I don't think that LSB guarantees the availability of these plugins
> either.  (And they don't get the timing correct, but maybe we don't care
> about that.)

That's not necessarily fatal, *if* the LSB sound tests can include the
plugin in question.  Is the plugin interface a stable, published
interface?  That is, is it guaranteed to remain stable from release to
release of ALSA?  If so, then the test suite can provide the plugin,
just for the purposes of running the test.

> > From the LSB perspective, I suppose, the API compatibility is the
> > biggest issue.  The functionality, either working or not on the real
> > hardware, is a subsystem bug.  And, the subtle tests would be likely
> > needed done manually with the real hardware.

This is true, but some kind of minimal functionality testing is often
useful to make sure that the *meaning* of the parameters haven't
changed.  For example, to take an extreme example, if a library were
to accidentally change a long to "long long" in a function signature,
but because of how other parameters and what happened to be in the
registers on an x86 system, the program might not crash, and if were
testing with a no-op output function for the program, we might not
notice a problem.

So it's nice to be able to do a semantic test "does this function
actually do roughly what it is spec'ed to do", and not just a simple
API test.  On the other hand, we also recognize that perfection can
require far more resources than are available, so often we have to
settle for a basic test.

						- Ted

Re: LSB inclusion of ALSA

[Clemens Ladisch]

Theodore Tso wrote:
> On Wed, Jun 17, 2009 at 10:52:38AM +0200, Clemens Ladisch wrote:
> > Takashi Iwai wrote:
> > > Another candidate is alsa-lib file plugin (and null plugin).  They can
> > > be used to simulate in a certain level of I/O.
> > 
> > I don't think that LSB guarantees the availability of these plugins
> > either.  (And they don't get the timing correct, but maybe we don't care
> > about that.)
> 
> That's not necessarily fatal, *if* the LSB sound tests can include the
> plugin in question.  Is the plugin interface a stable, published
> interface?  That is, is it guaranteed to remain stable from release to
> release of ALSA?

Yes.

However, the plugin interface itself is not part of LSB.  If there were
some alternative ALSA implementation that wanted to be LSB conformant,
the test would not work.  (There are currently two alternatives, both
named "SALSA", but neither one implements the full interface.)


Best regards,
Clemens

Re: LSB inclusion of ALSA

[Robert Schweikert]

Clemens Ladisch wrote:
> Robert Schweikert wrote:
>   
>> Currently ALSA is a trial use module for LSB
>> (http://www.linuxfoundation.org/collaborate/workgroups/lsb) 4.0 and it
>> is the intention to make ALSA mandatory with the LSB 4.1 release
>> targeted for Q1 2010. One of the requirements of a mandatory module is
>> the existence of a test suite for the module that covers a good chunk of
>> the interfaces provided by the LSB spec. An initial exploratory look at
>> the existing test cases showed that tests appear to be tied to hardware
>> drivers and are interactive.
>>
>> Is this initial impression correct?
>>     
>
> Yes.
>
>   
>> From an LSB testing perspective, interface testing is  very important;
>> i.e. I call the interface with the arguments required and I get the
>> expected result back.
>>     
>
> In the case of a hardware interface library like libasound, there is not
> always a result that comes back where it could by easily tested.
>
> While many parts of the library can be tested without special hardware
> (config, sequencer+MIDI, timers), the parts that are most likely to be
> used (PCM, mixer) require at least some kernel driver and contain many
> functions that are used to handle hardware differences.
>   
Understood, but this is functional testing (current tests seem to cover
this) and would required the appropriate hardware (a gazilion sound
cards). LSB makes no guarantees that the drivers exist and have no bugs.
LSB does strive to guarantee that a particular interface exists and
behaves as specified.

So for interfaces that require a driver one could test that an
appropriate error condition is returned if no driver exists, and we
could test that no error condition is triggered when a dummy driver is
connected. Correct?
>   
>> 1.) Is there interest from the community side to participate in this
>> effort and accept patches?
>>     
>
> I'm just an individual and cannot speak for "the community", whatever
> that is, but there is certainly interest to accept patches.
>
>   
>> 2.) Is it reasonable to expect that the existing tests can be used in
>> some way by using a dummy sound device or a sound loop device to verify
>> output?
>>     
>
> There is a loopback driver, but it is not part of the LSB specification.
> This means that a test using this driver would not work with any
> alternate ALSA implementation.
>
> I can easily imagine LSB-compliant computers that do not have any sound
> card (e.g., most servers).  Even if the ALSA interface (i.e., libasound)
> is installed, some parts cannot be used in any meaningful way.  What is
> a test supposed to do in this case?  It could just exit with "pass",
> but this wouldn't actually _test_ much.
>
> (Hmmm, a dummy ALSA implementation that just returns -ENODEV when the
> application tries to open a device would be compliant, because it cannot
> be distinguished from the 'real' ALSA with no installed sound card.  :-)
>   
Sorry, didn't quite get this, why a dummy ALSA implementation. Shouldn't
the ALSA interfaces that depend on a driver already return -ENODEV today?

But you raise a good point w.r.t. hardware tie in. Within the LSB work
group we are working on a concept to allow an application to test for
module/component/subcomponent (name not yet decided) presence. For
example a distribution can be LSB certified, but on a server install
libasound may not be present. An LSB certified application at install
time can query whether the LSB module/component/subcomponent that
contains libasound is installed on the system and then take appropriate
action. This would solve the problem you point out.
>   
>> 3.) Would the community be more comfortable if this is an effort that
>> creates new tests separate from the existing HW focused tests?
>>     
>
> The current interactive tests are mostly intended to test drivers, not
> the interface.  I don't think it would be desirable or even possible to
> integrate them into a test suite with strict pass/fail requirements.
>   
Good to know, sounds like we are better of starting from scratch.
>   
>> 4.) Is anyone interested in helping with this effort, writing tests,
>> answering questions about ALSA for those not familiar with the interface
>> and or sound in general?
>>     
>
> Yes.
>
> Since most of libasound is somewhat undocumented (the doxygen reference
> is just a skeleton), I'm wondering who is going to write the tests ...
>   
I have some time to work on this, but not having much of a clue how
sound works, and maybe that is not needed, I will need some amount of
hand holding.

Robert
>
> Best regards,
> Clemens
>   

-- 
Robert Schweikert                           MAY THE SOURCE BE WITH YOU
Software Engineer Consultant                          LINUX
rschweikert@novell.com 
781-464-8147

Novell
Making IT Work As One

Re: LSB inclusion of ALSA

[Clemens Ladisch]

Robert Schweikert wrote:
> Clemens Ladisch wrote:
> > While many parts of the library can be tested without special hardware
> > (config, sequencer+MIDI, timers), the parts that are most likely to be
> > used (PCM, mixer) require at least some kernel driver and contain many
> > functions that are used to handle hardware differences.
> 
> Understood, but this is functional testing (current tests seem to cover
> this) and would required the appropriate hardware (a gazilion sound
> cards). LSB makes no guarantees that the drivers exist and have no bugs.
> LSB does strive to guarantee that a particular interface exists and
> behaves as specified.
> 
> So for interfaces that require a driver one could test that an
> appropriate error condition is returned if no driver exists, and we
> could test that no error condition is triggered when a dummy driver is
> connected. Correct?

Yes.

This would require that the test suite has the capability to load and
unload kernel modules.  Could this ever be a problem?


Best regards,
Clemens