Re: Testing SBC filtering functions

[Siarhei Siamashka <siarhei.siamashka@nokia.com>]

[-- Attachment #1: Type: text/plain, Size: 5248 bytes --]

On Monday 29 December 2008 16:36:15 ext Siarhei Siamashka wrote:
> On Monday 29 December 2008 14:04:29 ext Marcel Holtmann wrote:
> > Hi Siarhei,
> > 
> > > > I wrote this script to make some tests on the SBC decoder and encoder using
> > > > the recommended testing procedure with the reference bitstreams, the
> > > > reference codec and PEAQ. 
> > > > http://net.cs.uni-tuebingen.de/html/nexgenvoip/html/
> > > > I got little bit confused with all the latest patches and whether they are
> > > > included or not. Just send me an email on which patch you like to have
> > > > tested. Running the tests just takes half an hour; me to answer my emails
> > > > maybe a bit longer.
> > > 
> > > Please try the following patch (apply it to the latest git):
> > > http://marc.info/?l=linux-bluetooth&m=123054787830678&w=2
> > > 
> > > You can try the patch "as is", and also with SBC_HIGH_PRECISION define
> > > uncommented. High precision mode is naturally more likely to pass the
> > > conformance tests.
> > > 
> > > I used my own script for testing with 'tiny_psnr' tool for comparing original file
> > > before encoding and the final decoded result (it measures standard deviation
> > > and PSNR). It would be interesting to see how our results correlate.
> > 
> > can you open source tiny_psnr an we merge it into the BlueZ source?
> 
> It is already open source. This very simple tool is used in ffmpeg project
> regression tests (and is part of ffmpeg distribution). It does not do
> anything extraordinary, but just analyzes the difference between several
> audio files and gets standard deviation and PSNR statistics. I just did not
> feel like reinventing the wheel and used it for estimating quality :)
> 
> I only use the following patch on top of it (it can do automatic shift detection):
> http://article.gmane.org/gmane.comp.video.ffmpeg.devel/74597
> I find this patch quite useful (especially as it turns out for SBC), but ffmpeg
> developers do not think so...
> 
> A crude ruby script (sorry, I don't speak shell scripting language) is attached.
> I have been using it for testing audio quality.
> 
> For example, here is the test of a high precision sbcenc build (bitpool=255):
> 
> ./sbc_encode_test.rb BigBuckBunny-stereo.flac
> [2, 48000]
> ["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
> --- comparing original / sbcenc + sbcdec ---
> stddev:  108.67 PSNR: 55.60 bytes:114519261/114520000
> 
> --- comparing original / sbcenc + sbc_decoder.exe ---
> stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000
> 
> --- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
> stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000
> 
> --- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
> stddev:    0.01 PSNR:130.99 bytes:114519552/114519552
> 
> Test of a standard sbcenc build (bitpool=255):
> 
> ./sbc_encode_test.rb BigBuckBunny-stereo.flac
> [2, 48000]
> ["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
> --- comparing original / sbcenc + sbcdec ---
> stddev:  108.71 PSNR: 55.59 bytes:114519261/114520000
> 
> --- comparing original / sbcenc + sbc_decoder.exe ---
> stddev:    2.07 PSNR: 89.98 bytes:114519260/114520000
> 
> --- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
> stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000
> 
> --- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
> stddev:    1.77 PSNR: 91.34 bytes:114519552/114519552

[...] 

> By the way, 'sbcdec' seems to introduce quite a noticeable distortion and
> is orders of magnitude less precise than the encoder.

Please disregard this statement. I only tried to add support for sbcdec as the
last minute change (I did not pay any attention to the decoder audio quality
before today) and forgot to change its output to a normal WAV format
before comparing files. Sorry about that. A fixed script is attached and
updated results are below.

Test of a high precision sbcenc build (bitpool=255):

./sbc_encode_test.rb BigBuckBunny-stereo.flac
[2, 48000]
["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
--- comparing original / sbcenc + sbcdec ---
stddev:    4.49 PSNR: 83.26 bytes:114519260/114520000

--- comparing original / sbcenc + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
stddev:    0.01 PSNR:130.99 bytes:114519552/114519552

Test of a standard sbcenc build (bitpool=255):

./sbc_encode_test.rb BigBuckBunny-stereo.flac
[2, 48000]
["-j -S -s8 -B16 -b255", "-j -l16 -n8 -r1569000"]
--- comparing original / sbcenc + sbcdec ---
stddev:    4.85 PSNR: 82.60 bytes:114519260/114520000

--- comparing original / sbcenc + sbc_decoder.exe ---
stddev:    2.07 PSNR: 89.98 bytes:114519260/114520000

--- comparing original / sbc_encoder.exe + sbc_decoder.exe ---
stddev:    1.09 PSNR: 95.56 bytes:114519260/114520000

--- comparing sbcenc + sbc_decoder.exe / sbc_encoder.exe + sbc_decoder.exe
stddev:    1.77 PSNR: 91.34 bytes:114519552/114519552

Still the sbcdec seems to be the major contributor to quality loss in all cases.


Best regards,
Siarhei Siamashka

[-- Attachment #2: sbc_encode_test.rb --]
[-- Type: application/x-ruby, Size: 2811 bytes --]