SBC encoder/decoder API & errors handling

[Siarhei Siamashka <siarhei.siamashka@gmail.com>]

[-- Attachment #1: Type: Text/Plain, Size: 1038 bytes --]

Hello,

When I tried to run some SBC encoder tests a few days ago, I noticed that 
there is a regression introduced by commit:
http://git.kernel.org/?p=bluetooth/bluez.git;a=commit;h=c43f8bdcc1d527e2d77481a66217771038be3acd

It is caused by the change from 'int' to 'size_t' for the type of variable 
'encoded' in 'sbcenc.c'. After this modification, the check for 'encoded <= 0' 
does not catch the case when 'sbc_encode' tries to return a negative number in 
'encoded' variable. Later we end up calling 'write' function with a negative 
size for the data block.

Right now, in the case of error 'sbc_encode' function may either return a 
negative number as a return value, or return a negative value in 'encoded' 
variable. But this second type of error is apparently not handled by anything 
other than 'sbcenc' tool at the moment.

Any opinions about how to fix it in the best way? Because it is a flaw in the 
library API, comments from the interested parties are welcome.

-- 
Best regards,
Siarhei Siamashka

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 198 bytes --]