Re: Testing SBC filtering functions

[Siarhei Siamashka <siarhei.siamashka@nokia.com>]

[-- Attachment #1: Type: text/plain, Size: 4053 bytes --]

On Monday 29 December 2008 14:04:29 ext Marcel Holtmann wrote:
> Hi Siarhei,
> 
> > > I wrote this script to make some tests on the SBC decoder and encoder using
> > > the recommended testing procedure with the reference bitstreams, the
> > > reference codec and PEAQ. 
> > > http://net.cs.uni-tuebingen.de/html/nexgenvoip/html/
> > > I got little bit confused with all the latest patches and whether they are
> > > included or not. Just send me an email on which patch you like to have
> > > tested. Running the tests just takes half an hour; me to answer my emails
> > > maybe a bit longer.
> > 
> > Please try the following patch (apply it to the latest git):
> > http://marc.info/?l=linux-bluetooth&m=123054787830678&w=2
> > 
> > You can try the patch "as is", and also with SBC_HIGH_PRECISION define
> > uncommented. High precision mode is naturally more likely to pass the
> > conformance tests.
> > 
> > I used my own script for testing with 'tiny_psnr' tool for comparing original file
> > before encoding and the final decoded result (it measures standard deviation
> > and PSNR). It would be interesting to see how our results correlate.
> 
> can you open source tiny_psnr an we merge it into the BlueZ source?

It is already open source. This very simple tool is used in ffmpeg project
regression tests (and is part of ffmpeg distribution). It does not do
anything extraordinary, but just analyzes the difference between several
audio files and gets standard deviation and PSNR statistics. I just did not
feel like reinventing the wheel and used it for estimating quality :)

I only use the following patch on top of it (it can do automatic shift detection):
http://article.gmane.org/gmane.comp.video.ffmpeg.devel/74597
I find this patch quite useful (especially as it turns out for SBC), but ffmpeg
developers do not think so...

A crude ruby script (sorry, I don't speak shell scripting language) is attached.
I have been using it for testing audio quality.

For example, here is the test of a high precision sbcenc build (bitpool=255):

./sbc_encode_test.rb BigBuckBunny-stereo.flac
[2, 48000]
["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
--- comparing original / sbcenc + sbcdec ---
stddev:  108.67 PSNR: 55.60 bytes:114519261/114520000

--- comparing original / sbcenc + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
stddev:    0.01 PSNR:130.99 bytes:114519552/114519552

Test of a standard sbcenc build (bitpool=255):

./sbc_encode_test.rb BigBuckBunny-stereo.flac
[2, 48000]
["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
--- comparing original / sbcenc + sbcdec ---
stddev:  108.71 PSNR: 55.59 bytes:114519261/114520000

--- comparing original / sbcenc + sbc_decoder.exe ---
stddev:    2.07 PSNR: 89.98 bytes:114519260/114520000

--- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
stddev:    1.77 PSNR: 91.34 bytes:114519552/114519552

Having high bitrate is useful for detecting bugs in analysis filter because
its contribution to quality loss is more visible in this configuration and
other factors play lesser role.

I tested a lot of various files and settings configurations. According to this
script, the quality of high precision sbcenc build matches the quality of
reference encoder pretty well. Standard 16-bit fixed point sbcenc build
introduces a very minor quality loss, noticeable only at extremely high
bitrates.

By the way, 'sbcdec' seems to introduce quite a noticeable distortion and
is orders of magnitude less precise than the encoder.

Of course it would be intertesting to see if the quality estimation done by
using tiny_psnr is adequate and if it can replace a 6000 EUR tool for this
particular purpose.


Best regards,
Siarhei Siamashka

[-- Attachment #2: sbc_encode_test.rb --]
[-- Type: application/x-ruby, Size: 2748 bytes --]