gstreamer and sbcdec problem

gstreamer and sbcdec problem

[Marian Harbach]

Hi,

I have the same problem as described in this post:

http://www.spinics.net/lists/linux-bluetooth/msg03627.html

I found that depending on filesize, the original input is delayed by some
bytes (180-260 for 16kHz PCM input) and additionally the first few samples
are overwritten when doing sbcenc followed by sbcdec.

Is there a fix for this issue? 

Cheers
Marian

Re: gstreamer and sbcdec problem

[Siarhei Siamashka]

[-- Attachment #1: Type: text/plain, Size: 1730 bytes --]

On Tuesday 15 December 2009, Marian Harbach wrote:
> Hi,
>
> I have the same problem as described in this post:
>
> http://www.spinics.net/lists/linux-bluetooth/msg03627.html

This looks like either some completely random garbage gets there (valgrind
can probably help), or the decoder is not reinitialized properly and has
some stale data from the previous decode in some of its internal buffers.

> I found that depending on filesize, the original input is delayed by some
> bytes (180-260 for 16kHz PCM input) 

SBC introduces some delay to the audio data due to the way how it works.
Some of the trailing samples of the buffer you feed to the 'sbc_encode' 
function are actually not represented fully in the encoded data for this
frame, but are kept in the ring buffer and get processed later on
next 'sbc_encode' call.

The reference binary encoder introduces exactly the same delay.

> and additionally the first few samples 
> are overwritten when doing sbcenc followed by sbcdec.

Could you please clarify this part? In what way and where do they get
overwritten (in files, memory buffers, ...)?

> Is there a fix for this issue?

My guess is that the client application can be aware of this delay and take it
into account somehow.

Also I don't have much trust in the current bluez SBC decoder implementation.
Its quality may be not the very best. I was considering to eventually review
its code, do some refactoring and merge some of its parts with the encoder
(SBC encode and decode are quite symmetric), but did not find some spare
time for this yet. Considering that bluez got SBC sink support now, improving
the decoder may make sense.

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

AW: gstreamer and sbcdec problem

[Marian Harbach]

Hey thanks for the reply. To clarify:

I use gstreamer to run a 16kHz PCM wav file through one sbc encoder/decoder
step and save the results back to a 16kHz PCM wav file. When I examine the
original file and the result file in a tool like Audacity, there is an
offset between the two files of about 0.0125 secs. If I delay one file
accordingly, the waveforms in the remaining parts fit. Yet, the first couple
of samples of the original waveform have no resamblances with the resulting
waveform. 

If that delay mentioned above was actually a fixed value, I'd have no
problem in my application. But the delay depends on input filesize and also
increases when running the result file through an sbcenc/sbcdec pair again.
At least that is the pattern that I see for now.

I hope that helps.
Cheers
Marian

-----Ursprüngliche Nachricht-----
Von: linux-bluetooth-owner@vger.kernel.org
[mailto:linux-bluetooth-owner@vger.kernel.org] Im Auftrag von Siarhei
Siamashka
Gesendet: Mittwoch, 16. Dezember 2009 01:04
An: Marian Harbach
Cc: linux-bluetooth@vger.kernel.org
Betreff: Re: gstreamer and sbcdec problem

On Tuesday 15 December 2009, Marian Harbach wrote:
> Hi,
>
> I have the same problem as described in this post:
>
> http://www.spinics.net/lists/linux-bluetooth/msg03627.html

This looks like either some completely random garbage gets there (valgrind
can probably help), or the decoder is not reinitialized properly and has
some stale data from the previous decode in some of its internal buffers.

> I found that depending on filesize, the original input is delayed by 
> some bytes (180-260 for 16kHz PCM input)

SBC introduces some delay to the audio data due to the way how it works.
Some of the trailing samples of the buffer you feed to the 'sbc_encode' 
function are actually not represented fully in the encoded data for this
frame, but are kept in the ring buffer and get processed later on next
'sbc_encode' call.

The reference binary encoder introduces exactly the same delay.

> and additionally the first few samples are overwritten when doing 
> sbcenc followed by sbcdec.

Could you please clarify this part? In what way and where do they get
overwritten (in files, memory buffers, ...)?

> Is there a fix for this issue?

My guess is that the client application can be aware of this delay and take
it into account somehow.

Also I don't have much trust in the current bluez SBC decoder
implementation.
Its quality may be not the very best. I was considering to eventually review
its code, do some refactoring and merge some of its parts with the encoder
(SBC encode and decode are quite symmetric), but did not find some spare
time for this yet. Considering that bluez got SBC sink support now,
improving the decoder may make sense.

--
Best regards,
Siarhei Siamashka

Re: AW: gstreamer and sbcdec problem

[Siarhei Siamashka]

[-- Attachment #1: Type: text/plain, Size: 2166 bytes --]

On Wednesday 16 December 2009, Marian Harbach wrote:
> Hey thanks for the reply. To clarify:
>
> I use gstreamer to run a 16kHz PCM wav file through one sbc encoder/decoder
> step and save the results back to a 16kHz PCM wav file. When I examine the
> original file and the result file in a tool like Audacity, there is an
> offset between the two files of about 0.0125 secs. If I delay one file
> accordingly, the waveforms in the remaining parts fit.

Yes, this is something to be expected unless I'm mistaken. You can google
for "SBC algorithmic delay" or something like this.

> Yet, the first couple of samples of the original waveform have no
> resamblances with the resulting waveform.

This may actually need a bit of tweaking. When starting encoding, we don't
have a preceding history, but older nonexisting samples are still needed as
input for the polyphase filter. The encoder from bluez currently treats those
missing samples as zero. Having a quick look at A2DP spec again, I don't see
any special explanations about how the encoding process should start.
Experimenting a bit with the reference binary encoder and trying to guess
how it deals with the initial samples may help.

> If that delay mentioned above was actually a fixed value, I'd have no
> problem in my application.

It should be a fixed value, which only depends on encoding parameters.

> But the delay depends on input filesize 

This is strange. The encoder/decoder works with the stream of data and is not
even supposed to know how big is the input file.

Either your tool is not very precise at measuring this delay, or there could
be some bugs in sbc or in the gstreamer layer.

I suggest you to experiment with standalone sbcenc/sbcdec comand line tools
and check if the same behavior is also reproduced there.

> and also increases when running the result file through an sbcenc/sbcdec
> pair again. At least that is the pattern that I see for now.

Sure, there is a constant delay introduced on each encode/decode operation.
Repeating it multiple times will result in this delay accumulating.

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: gstreamer and sbcdec problem

[João Paulo Rechi Vita]

On Tue, Dec 15, 2009 at 10:03 PM, Siarhei Siamashka
<siarhei.siamashka@gmail.com> wrote:

> Also I don't have much trust in the current bluez SBC decoder implementation.
> Its quality may be not the very best. I was considering to eventually review
> its code, do some refactoring and merge some of its parts with the encoder
> (SBC encode and decode are quite symmetric), but did not find some spare
> time for this yet. Considering that bluez got SBC sink support now, improving
> the decoder may make sense.
>

IMO it would make sense to export SBC codec in a library, since
encoding and decoding is done outside bluetoothd (ALSA, gstreamer or
PulseAudio). Also, the codec could be used for applications different
from A2DP streaming, and the more people using it, more tested (and
eventually improved) the code gets.

-- 
João Paulo Rechi Vita
MSc Computer Science Student
Computer Engineer
IC / Unicamp
http://jprvita.wordpress.com/

Re: gstreamer and sbcdec problem

[Siarhei Siamashka]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Wednesday 16 December 2009, João Paulo Rechi Vita wrote:
> On Tue, Dec 15, 2009 at 10:03 PM, Siarhei Siamashka
>
> <siarhei.siamashka@gmail.com> wrote:
> > Also I don't have much trust in the current bluez SBC decoder
> > implementation. Its quality may be not the very best. I was considering
> > to eventually review its code, do some refactoring and merge some of its
> > parts with the encoder (SBC encode and decode are quite symmetric), but
> > did not find some spare time for this yet. Considering that bluez got SBC
> > sink support now, improving the decoder may make sense.
>
> IMO it would make sense to export SBC codec in a library, since
> encoding and decoding is done outside bluetoothd (ALSA, gstreamer or
> PulseAudio). Also, the codec could be used for applications different
> from A2DP streaming, and the more people using it, more tested (and
> eventually improved) the code gets.

I don't have first hand information regarding this matter myself, but
according to SBC wikipedia page [1]:
"The patent owners wrote that they allow the free usage of SBC in Bluetooth 
application, with the view to boost the use of this technology. All 
applications outside Bluetooth are however not free."

So having SBC codec tightly coupled with bluez at least makes some sense. From
purely technical point of view, a standalone library could be better.


1. http://en.wikipedia.org/wiki/SBC_(codec)

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: gstreamer and sbcdec problem

[João Paulo Rechi Vita]

2009/12/17 Siarhei Siamashka <siarhei.siamashka@gmail.com>:
> On Wednesday 16 December 2009, João Paulo Rechi Vita wrote:
>> On Tue, Dec 15, 2009 at 10:03 PM, Siarhei Siamashka
>>
>> <siarhei.siamashka@gmail.com> wrote:
>> > Also I don't have much trust in the current bluez SBC decoder
>> > implementation. Its quality may be not the very best. I was considering
>> > to eventually review its code, do some refactoring and merge some of its
>> > parts with the encoder (SBC encode and decode are quite symmetric), but
>> > did not find some spare time for this yet. Considering that bluez got SBC
>> > sink support now, improving the decoder may make sense.
>>
>> IMO it would make sense to export SBC codec in a library, since
>> encoding and decoding is done outside bluetoothd (ALSA, gstreamer or
>> PulseAudio). Also, the codec could be used for applications different
>> from A2DP streaming, and the more people using it, more tested (and
>> eventually improved) the code gets.
>
> I don't have first hand information regarding this matter myself, but
> according to SBC wikipedia page [1]:
> "The patent owners wrote that they allow the free usage of SBC in Bluetooth
> application, with the view to boost the use of this technology. All
> applications outside Bluetooth are however not free."
>

And if you continue reading the same paragraph: "The patent will
expire June 2, 2010.".

But anyway, gstreamer and pulseaudio uses SBC for bluetooth
application, so it's under the free usage terms (and also, there is a
copy of sbc.c in pulseaudio tree).

-- 
João Paulo Rechi Vita
MSc Computer Science Student
Computer Engineer
IC / Unicamp
http://jprvita.wordpress.com/

Re: gstreamer and sbcdec problem

[Siarhei Siamashka]

[-- Attachment #1: Type: text/plain, Size: 3602 bytes --]

On Thursday 17 December 2009, João Paulo Rechi Vita wrote:
> 2009/12/17 Siarhei Siamashka <siarhei.siamashka@gmail.com>:
> > On Wednesday 16 December 2009, João Paulo Rechi Vita wrote:
> >> On Tue, Dec 15, 2009 at 10:03 PM, Siarhei Siamashka
> >>
> >> <siarhei.siamashka@gmail.com> wrote:
> >> > Also I don't have much trust in the current bluez SBC decoder
> >> > implementation. Its quality may be not the very best. I was
> >> > considering to eventually review its code, do some refactoring and
> >> > merge some of its parts with the encoder (SBC encode and decode are
> >> > quite symmetric), but did not find some spare time for this yet.
> >> > Considering that bluez got SBC sink support now, improving the decoder
> >> > may make sense.
> >>
> >> IMO it would make sense to export SBC codec in a library, since
> >> encoding and decoding is done outside bluetoothd (ALSA, gstreamer or
> >> PulseAudio). Also, the codec could be used for applications different
> >> from A2DP streaming, and the more people using it, more tested (and
> >> eventually improved) the code gets.
> >
> > I don't have first hand information regarding this matter myself, but
> > according to SBC wikipedia page [1]:
> > "The patent owners wrote that they allow the free usage of SBC in
> > Bluetooth application, with the view to boost the use of this technology.
> > All applications outside Bluetooth are however not free."
>
> And if you continue reading the same paragraph: "The patent will
> expire June 2, 2010.".

So should we wait for half a year before bringing this discussion up again,
or something? ;-)

> But anyway, gstreamer and pulseaudio uses SBC for bluetooth
> application, so it's under the free usage terms (and also, there is a
> copy of sbc.c in pulseaudio tree).

In any case, everything can be categorized into
1. administrative part (splitting off SBC into a separate library, make a
repository for it, make a formal release, promote it and start using it in
different bluetooth related projects).
2. technical part (fixing bugs and improving the codec)

I'm not very much interested in the "administrative part" myself :-)

For the "technical part", the first thing to do is to change SBC decoder
to use Q31 fixed point arithmetics in the synthesis filter. Currently it
uses lower precision calculations and is somewhat optimized (to the point
of becoming a bit obfuscated). After this change, the output of SBC decoder
should differ from from the output of the reference SBC decoder binary
probably only in the least significant bit. More significant differences (if
any) are caused by the bugs in the code, most likely overflows on
calculations (I already fixed a bug with caused by missing clamping of output
in the decoder, but who knows how many of them could be still there?). After
all the bugs get ironed out, support for Q15 calculations can be added for
better performance and SIMD optimizations. Precision loss of Q15 fixed point
implementation should be carefully checked and minimized.

Additionally, more SIMD optimizations can still improve performance for both
encoder and decoder.

As for the other changes, maybe handling of the initial samples in the encoder
could be tweaked (for example not by using zero samples in the initial
"history" buffer, but putting something like a mirror reflection of the input
data there). Still I'm not sure if it's a really big problem, maybe Marian can
provide a bit more information about what is the real impact in practice.

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: gstreamer and sbcdec problem

[João Paulo Rechi Vita]

2009/12/17 Siarhei Siamashka <siarhei.siamashka@gmail.com>:
> On Thursday 17 December 2009, João Paulo Rechi Vita wrote:
>> 2009/12/17 Siarhei Siamashka <siarhei.siamashka@gmail.com>:
>> > On Wednesday 16 December 2009, João Paulo Rechi Vita wrote:
>> >> On Tue, Dec 15, 2009 at 10:03 PM, Siarhei Siamashka
>> >>
>> >> <siarhei.siamashka@gmail.com> wrote:
>> >> > Also I don't have much trust in the current bluez SBC decoder
>> >> > implementation. Its quality may be not the very best. I was
>> >> > considering to eventually review its code, do some refactoring and
>> >> > merge some of its parts with the encoder (SBC encode and decode are
>> >> > quite symmetric), but did not find some spare time for this yet.
>> >> > Considering that bluez got SBC sink support now, improving the decoder
>> >> > may make sense.
>> >>
>> >> IMO it would make sense to export SBC codec in a library, since
>> >> encoding and decoding is done outside bluetoothd (ALSA, gstreamer or
>> >> PulseAudio). Also, the codec could be used for applications different
>> >> from A2DP streaming, and the more people using it, more tested (and
>> >> eventually improved) the code gets.
>> >
>> > I don't have first hand information regarding this matter myself, but
>> > according to SBC wikipedia page [1]:
>> > "The patent owners wrote that they allow the free usage of SBC in
>> > Bluetooth application, with the view to boost the use of this technology.
>> > All applications outside Bluetooth are however not free."
>>
>> And if you continue reading the same paragraph: "The patent will
>> expire June 2, 2010.".
>
> So should we wait for half a year before bringing this discussion up again,
> or something? ;-)
>

No, of course not, hehe. I've just brought that up because in a few
months non-bluetooth application can benefit of this re-factor too.

>> But anyway, gstreamer and pulseaudio uses SBC for bluetooth
>> application, so it's under the free usage terms (and also, there is a
>> copy of sbc.c in pulseaudio tree).
>
> In any case, everything can be categorized into
> 1. administrative part (splitting off SBC into a separate library, make a
> repository for it, make a formal release, promote it and start using it in
> different bluetooth related projects).
> 2. technical part (fixing bugs and improving the codec)
>

At least for now, I don't think SBC should be moved to a different
repository, but only be split in a separate library and have a
separated release tarball. But I would like to know what other devs
think about this separation.

And part 2 should be done independently of making SBC a lib or not.

> I'm not very much interested in the "administrative part" myself :-)
>
> For the "technical part", the first thing to do is to change SBC decoder
> to use Q31 fixed point arithmetics in the synthesis filter. Currently it
> uses lower precision calculations and is somewhat optimized (to the point
> of becoming a bit obfuscated). After this change, the output of SBC decoder
> should differ from from the output of the reference SBC decoder binary
> probably only in the least significant bit. More significant differences (if
> any) are caused by the bugs in the code, most likely overflows on
> calculations (I already fixed a bug with caused by missing clamping of output
> in the decoder, but who knows how many of them could be still there?). After
> all the bugs get ironed out, support for Q15 calculations can be added for
> better performance and SIMD optimizations. Precision loss of Q15 fixed point
> implementation should be carefully checked and minimized.
>
> Additionally, more SIMD optimizations can still improve performance for both
> encoder and decoder.
>
> As for the other changes, maybe handling of the initial samples in the encoder
> could be tweaked (for example not by using zero samples in the initial
> "history" buffer, but putting something like a mirror reflection of the input
> data there). Still I'm not sure if it's a really big problem, maybe Marian can
> provide a bit more information about what is the real impact in practice.
>

So from you comments it seems you are interested and have the
technical knowledge to work these improvements. Please do this when
you find the time and send patches to the ML.

-- 
João Paulo Rechi Vita
MSc Computer Science Student
Computer Engineer
IC / Unicamp
http://jprvita.wordpress.com/