alsa-utils 1.1.8: axfer tests take ages tu run.

alsa-utils 1.1.8: axfer tests take ages tu run.

[Elimar Riesebieter]

[-- Attachment #1.1: Type: text/plain, Size: 343 bytes --]

Hi,

what are the tests in alsa-utils-1.1.8/axfer/test/ good for? mapper-test and
container-test are running very long time. I Can't find any docu in
the source? If the need as long as on my 8-core the source is not
ready for distribution, though.

Elimar
-- 
  You cannot propel yourself forward by
  patting yourself on the back.

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Elimar Riesebieter]

[-- Attachment #1.1: Type: text/plain, Size: 477 bytes --]

* Elimar Riesebieter <riesebie@lxtec.de> [2019-01-26 15:49 +0100]:

> Hi,
> 
> what are the tests in alsa-utils-1.1.8/axfer/test/ good for? mapper-test and
> container-test are running very long time. I Can't find any docu in
> the source? If the need as long as on my 8-core the source is not
> ready for distribution, though.

 container-test took 44min but passed!

Elimar
-- 
  On the keyboard of life you have always
  to keep a finger at the escape key;-)

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Takashi Sakamoto]

Hi,

On Sat, Jan 26, 2019 at 05:04:32PM +0100, Elimar Riesebieter wrote:
> * Elimar Riesebieter <riesebie@lxtec.de> [2019-01-26 15:49 +0100]:
> 
> > Hi,
> > 
> > what are the tests in alsa-utils-1.1.8/axfer/test/ good for? mapper-test and
> > container-test are running very long time. I Can't find any docu in
> > the source? If the need as long as on my 8-core the source is not
> > ready for distribution, though.
> 
>  container-test took 44min but passed!

It's expected.

The execution time of unit tests for axfer is file I/O bound, thus
it's independent of processors. However, I guess it takes 20-50 minutes
to finish all iterations.


I design the 'container-test' to iterate a set of building/parsing
content of media container with memory comparison for 635,904 times.
Each trial consists of the different combination of parameters
described below:

* buffer type for audio data frame (=2)
 * linear buffer and interleaved for SND_PCM_ACCESS_MMAP_INTERLEAVED
 * linear buffer and non-interleaved for SND_PCM_ACCESS_MMAP_NONINTERLEAVED
* type of media container and format of audio data frame (19 + 7 + 4 + 39 = 69)
 * 19 formats for RIFF/Wave container
 * 7 formats for AU container
 * 4 formats for VOC container
 * 39 formats for raw container
* the number of samples in an audio data frame (=128)
 * 1-128
* the number of audio data frames in buffer (=6)
 * 23
 * 1047
 * 2071
 * 3095
 * 4119
 * 4500
 * (23-4500 with 1024 step)
* the number of audio data frames per second (=6)
 * 44.1kHz
 * 48.0
 * 88.2
 * 96.0
 * 176.4
 * 192.0

When building/parsing media container, each implementation of media
container executes I/Os to actual files.


Like container-test, the 'mapper-test' iterates a set of muxing/demuxing
between the buffers of audio data frame and the containers with memory
comparison for 10,752 times. Each trial consists of the different
combination of parameters described below

* buffer type for audio data frame (=4)
 * linear buffer and interleaved for SND_PCM_ACCESS_MMAP_INTERLEAVED
 * linear buffer and non-interleaved for SND_PCM_ACCESS_MMAP_NONINTERLEAVED
 * linear buffer and interleaved for SND_PCM_ACCESS_RW_INTERLEAVED
 * vector buffer and noninterleaved for SND_PCM_ACCESS_RW_NONINTERLEAVED
* type of media container and format of audio data frame (=7)
 * 7 formats for RIFF/Wave
* the number of samples in an audio data frame (=32)
 * 1-32
* the number of audio data frames in buffer (=6)
 * 23
 * 1047
 * 2071
 * 3095
 * 4119
 * 4500
 * (23-4500 with 1024 step)
* the number of audio data frames per second (=1)
 * 48.0 kHz
* type of mapper (=2)
 * single
 * multiple

When muxing/demuxing for buffer of audio data frame, it uses internal
implementation of RIFF/Wave container, thus executes I/Os to actual
files.


I think it preferable to shorten the execution time of each unit test;
e.g. several minutes at maximum, however the purpose of unit test is to
detect bugs in advance and this program handles audio data frame between
the several types of buffer for I/O to sound device and the several
types of media container with combinations of many parameters. At
present it's reasonable to takes such long time to finish the tests.

Of course, we can shorten the duration time by eliminating the range
of parameters. Actually I've investigated to reduce iteration and reduce
for the number of audio data frames in buffer by 6 options. But any
reasonable explanation except for the duration time is required to be
worth for it, IMO.


Thanks

Takashi Sakamoto

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Elimar Riesebieter]

[-- Attachment #1.1: Type: text/plain, Size: 4677 bytes --]

* Takashi Sakamoto <o-takashi@sakamocchi.jp> [2019-01-27 12:44 +0900]:

> Hi,
> 
> On Sat, Jan 26, 2019 at 05:04:32PM +0100, Elimar Riesebieter wrote:
> > * Elimar Riesebieter <riesebie@lxtec.de> [2019-01-26 15:49 +0100]:
> > 
> > > Hi,
> > > 
> > > what are the tests in alsa-utils-1.1.8/axfer/test/ good for? mapper-test and
> > > container-test are running very long time. I Can't find any docu in
> > > the source? If the need as long as on my 8-core the source is not
> > > ready for distribution, though.
> > 
> >  container-test took 44min but passed!
> 
> It's expected.
> 
> The execution time of unit tests for axfer is file I/O bound, thus
> it's independent of processors. However, I guess it takes 20-50 minutes
> to finish all iterations.

FYI this will block build daemons on distribution machines.
alsa-utils 1.1.7 builds within a finger snip, though.

> 
> I design the 'container-test' to iterate a set of building/parsing
> content of media container with memory comparison for 635,904 times.
> Each trial consists of the different combination of parameters
> described below:
> 
> * buffer type for audio data frame (=2)
>  * linear buffer and interleaved for SND_PCM_ACCESS_MMAP_INTERLEAVED
>  * linear buffer and non-interleaved for SND_PCM_ACCESS_MMAP_NONINTERLEAVED
> * type of media container and format of audio data frame (19 + 7 + 4 + 39 = 69)
>  * 19 formats for RIFF/Wave container
>  * 7 formats for AU container
>  * 4 formats for VOC container
>  * 39 formats for raw container
> * the number of samples in an audio data frame (=128)
>  * 1-128
> * the number of audio data frames in buffer (=6)
>  * 23
>  * 1047
>  * 2071
>  * 3095
>  * 4119
>  * 4500
>  * (23-4500 with 1024 step)
> * the number of audio data frames per second (=6)
>  * 44.1kHz
>  * 48.0
>  * 88.2
>  * 96.0
>  * 176.4
>  * 192.0
> 
> When building/parsing media container, each implementation of media
> container executes I/Os to actual files.
> 
> 
> Like container-test, the 'mapper-test' iterates a set of muxing/demuxing
> between the buffers of audio data frame and the containers with memory
> comparison for 10,752 times. Each trial consists of the different
> combination of parameters described below
> 
> * buffer type for audio data frame (=4)
>  * linear buffer and interleaved for SND_PCM_ACCESS_MMAP_INTERLEAVED
>  * linear buffer and non-interleaved for SND_PCM_ACCESS_MMAP_NONINTERLEAVED
>  * linear buffer and interleaved for SND_PCM_ACCESS_RW_INTERLEAVED
>  * vector buffer and noninterleaved for SND_PCM_ACCESS_RW_NONINTERLEAVED
> * type of media container and format of audio data frame (=7)
>  * 7 formats for RIFF/Wave
> * the number of samples in an audio data frame (=32)
>  * 1-32
> * the number of audio data frames in buffer (=6)
>  * 23
>  * 1047
>  * 2071
>  * 3095
>  * 4119
>  * 4500
>  * (23-4500 with 1024 step)
> * the number of audio data frames per second (=1)
>  * 48.0 kHz
> * type of mapper (=2)
>  * single
>  * multiple
> 
> When muxing/demuxing for buffer of audio data frame, it uses internal
> implementation of RIFF/Wave container, thus executes I/Os to actual
> files.

Thanks for explanation.

> 
> I think it preferable to shorten the execution time of each unit test;
> e.g. several minutes at maximum, however the purpose of unit test is to
> detect bugs in advance and this program handles audio data frame between
> the several types of buffer for I/O to sound device and the several
> types of media container with combinations of many parameters. At
> present it's reasonable to takes such long time to finish the tests.

This means you need a sound device on the building machine? Thats
contra productive! I assume that at least our Debian buildd's don't
have a sound device. How are those tests handled in that case? We
need a properly solution for Linux distributions here!

> Of course, we can shorten the duration time by eliminating the range
> of parameters. Actually I've investigated to reduce iteration and reduce
> for the number of audio data frames in buffer by 6 options. But any
> reasonable explanation except for the duration time is required to be
> worth for it, IMO.

Well, what do you think to outbound your tests to i.e.
$PREFIX/doc/examples to let the user decide whether to run them or
not. A configure option to cancel the tests in the build process
would be an option to be decided by the distribution maintainer.

Elimar
Member of "Debian ALSA Maintainers"
-- 
  "Talking much about oneself can also
   be a means to conceal oneself."
         -Friedrich Nietzsche

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Takashi Sakamoto]

Hi,

On Sun, Jan 27, 2019 at 11:03:04AM +0100, Elimar Riesebieter wrote:
> > I think it preferable to shorten the execution time of each unit test;
> > e.g. several minutes at maximum, however the purpose of unit test is to
> > detect bugs in advance and this program handles audio data frame between
> > the several types of buffer for I/O to sound device and the several
> > types of media container with combinations of many parameters. At
> > present it's reasonable to takes such long time to finish the tests.
> 
> This means you need a sound device on the building machine? Thats
> contra productive! I assume that at least our Debian buildd's don't
> have a sound device. How are those tests handled in that case? We
> need a properly solution for Linux distributions here!
 
Execution of unit test independent from actual sound device is one of
my intention for the tests, and actually the they performs. Executables
of the test include duplicated code of axfer for this intention. Please
refer to 'axfer/test/Makefile.am'.

> > Of course, we can shorten the duration time by eliminating the range
> > of parameters. Actually I've investigated to reduce iteration and reduce
> > for the number of audio data frames in buffer by 6 options. But any
> > reasonable explanation except for the duration time is required to be
> > worth for it, IMO.
> 
> Well, what do you think to outbound your tests to i.e.
> $PREFIX/doc/examples to let the user decide whether to run them or
> not. A configure option to cancel the tests in the build process
> would be an option to be decided by the distribution maintainer.

The alsa-utils uses GNU Automake to implement test. As long as I know,
any Makefile generated by GNU Automake doesn't run test unless
'check' target is given explicitly.
https://www.gnu.org/software/automake/manual/html_node/Tests.html

Thereore, it depends on yout configuration of debhelper to execute the
tests in building time of deb package. Please recheck your 'debian/rules'
and adds appropriate directives to suppress execution of test for axfer
(perhaps, 'override_dh_auto_test'). For example:

```
override_dh_auto_test:
        true
```

For your information, alsa-utils includes no other test programs.


Thanks

Takashi Sakamoto

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Takashi Sakamoto]

On Sun, Jan 27, 2019 at 09:14:02PM +0900, Takashi Sakamoto wrote:
> Execution of unit test independent from actual sound device is one of
> my intention for the tests, and actually the they performs. Executables
> of the test include duplicated code of axfer for this intention. Please
> refer to 'axfer/test/Makefile.am'.
> 
> > > Of course, we can shorten the duration time by eliminating the range
> > > of parameters. Actually I've investigated to reduce iteration and reduce
> > > for the number of audio data frames in buffer by 6 options. But any
> > > reasonable explanation except for the duration time is required to be
> > > worth for it, IMO.
> > 
> > Well, what do you think to outbound your tests to i.e.
> > $PREFIX/doc/examples to let the user decide whether to run them or
> > not. A configure option to cancel the tests in the build process
> > would be an option to be decided by the distribution maintainer.
> 
> The alsa-utils uses GNU Automake to implement test. As long as I know,
> any Makefile generated by GNU Automake doesn't run test unless
> 'check' target is given explicitly.
> https://www.gnu.org/software/automake/manual/html_node/Tests.html
> 
> Thereore, it depends on yout configuration of debhelper to execute the
> tests in building time of deb package. Please recheck your 'debian/rules'
> and adds appropriate directives to suppress execution of test for axfer
> (perhaps, 'override_dh_auto_test'). For example:
> 
> ```
> override_dh_auto_test:
>         true
> ```
> 
> For your information, alsa-utils includes no other test programs.

Or indicate test target explicitly in a directive of
override_dh_auto_test:

```
override_dh_auto_test:
        # 'container-test' and 'mapper-test' for axfer take a long time.
        # They are designed to check robustness of internal
	# implementation of axfer and do not necessarily need to run in
	# a process of deb packaging. Let's suppress them.
        dh_auto_test -- TESTS=''
```


Thanks

Takashi Sakamoto

Re: alsa-utils 1.1.8: axfer tests take ages tu run.

[Elimar Riesebieter]

[-- Attachment #1.1: Type: text/plain, Size: 2400 bytes --]

* Takashi Sakamoto <o-takashi@sakamocchi.jp> [2019-01-27 21:49 +0900]:

> On Sun, Jan 27, 2019 at 09:14:02PM +0900, Takashi Sakamoto wrote:
> > Execution of unit test independent from actual sound device is one of
> > my intention for the tests, and actually the they performs. Executables
> > of the test include duplicated code of axfer for this intention. Please
> > refer to 'axfer/test/Makefile.am'.
> > 
> > > > Of course, we can shorten the duration time by eliminating the range
> > > > of parameters. Actually I've investigated to reduce iteration and reduce
> > > > for the number of audio data frames in buffer by 6 options. But any
> > > > reasonable explanation except for the duration time is required to be
> > > > worth for it, IMO.
> > > 
> > > Well, what do you think to outbound your tests to i.e.
> > > $PREFIX/doc/examples to let the user decide whether to run them or
> > > not. A configure option to cancel the tests in the build process
> > > would be an option to be decided by the distribution maintainer.
> > 
> > The alsa-utils uses GNU Automake to implement test. As long as I know,
> > any Makefile generated by GNU Automake doesn't run test unless
> > 'check' target is given explicitly.
> > https://www.gnu.org/software/automake/manual/html_node/Tests.html
> > 
> > Thereore, it depends on yout configuration of debhelper to execute the
> > tests in building time of deb package. Please recheck your 'debian/rules'
> > and adds appropriate directives to suppress execution of test for axfer
> > (perhaps, 'override_dh_auto_test'). For example:
> > 
> > ```
> > override_dh_auto_test:
> >         true
> > ```
> > 
> > For your information, alsa-utils includes no other test programs.
> 
> Or indicate test target explicitly in a directive of
> override_dh_auto_test:
> 
> ```
> override_dh_auto_test:
>         # 'container-test' and 'mapper-test' for axfer take a long time.
>         # They are designed to check robustness of internal
> 	# implementation of axfer and do not necessarily need to run in
> 	# a process of deb packaging. Let's suppress them.
>         dh_auto_test -- TESTS=''

Wow, I never noticed the availability of dh_auto_test. Thanks alot!

Thanks for cooperation
Elimar
-- 
  355/113: Not the famous irrational number pi,
           but an incredible simulation!
	                -unknown

[-- Attachment #1.2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

[-- Attachment #2: Type: text/plain, Size: 0 bytes --]