Re: [PATCH v5 00/16] mSBC tests

[Siarhei Siamashka <siarhei.siamashka@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 2455 bytes --]

On Fri, 14 Dec 2012 16:03:21 +0100
Frédéric Dalleau  <frederic.dalleau@linux.intel.com> wrote:

> Hi,
> 
> v5 shoudl fix the few remaining issues.  I could test neon and armv6
> and raised verbosity of commit messages.

Thanks for the update. The verbose commit messages make a really big
difference.

> However, before releasing a new version of the library, I believe we should
> write some sort of automated testing and test against another implementation.
> Siarhei shared a script some time ago, it could be a good start.
> But where could we store some decent audio materials? Should we use generators
> like sine wave, or just record ambient sound??

It is a good idea to make use of autotools test framework (a simple
one, not dejagnu based):
    http://www.gnu.org/software/automake/manual/html_node/Tests.html

The tests could generate some deterministic pseudorandom data and
encode it with the sbc encoder.

Testing for potential overflows and the other bugs in the code is quite
effective with the plain uniformly distributed random data. The
checksums (CRC32 or some cryptographic hashes) can be calculated for
the encoded data and compared with the reference values. This test
ensures that the code works exactly as it used to work before (no
regressions, no improvements).

Additionally some sine waves (or more complex signals consisting
of multiple pseudorandom sine waves) can be used for one more test,
which could check the errors accumulated for the encoding/decoding
roundtrip.

And finally, an integrated benchmark would be also useful in this
sbc test framework. Compressing/decompressing white noise may not
provide relevant results, but maybe some sine waves can be selected
which represent more "realistic" audio data. Or one can be compared
to each other, and if there are no significant differences in
performance for different data, then it's easier for us.

An example of an autotools based test framework employing pseudorandom
data generation and checksums verification can be found in the pixman
library:
    http://cgit.freedesktop.org/pixman
Coincidentally, I looked into pseudorandom number generators recently
and found this one to be quite good:
    http://burtleburtle.net/bob/rand/smallprng.html

As for comparing with the reference sbc encoder/decoder (win32
binary), I used to have one more script. It is attached.

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: sbc_encode_test.rb --]
[-- Type: application/x-ruby, Size: 3067 bytes --]