webrtc question

webrtc question

[Ed Sutter]

I apologize if this is the wrong list for this question, but I'm about 
to embark
on a iMX6 -based multimedia project that will be greatly helped by webRTC
functionality.

Still getting up to speed on this, but it will not be Android, and it 
will need
to run outside the scope of a browser.  I believe this is referred to as 
"native
webRTC".

Has anything been done with iMX6-linux regarding webRTC?
Does hardware acceleration support for VP8 encode/decode work on iMX6?
Any pointers would be appreciated.
Regards,
Ed

Re: webrtc question

[Otavio Salvador]

Hello Ed,

On Tue, Jan 6, 2015 at 3:36 PM, Ed Sutter <ed.sutter@alcatel-lucent.com> wrote:
> I apologize if this is the wrong list for this question, but I'm about to
> embark on a iMX6 -based multimedia project that will be greatly helped by webRTC
> functionality.
>
> Still getting up to speed on this, but it will not be Android, and it will
> need to run outside the scope of a browser.  I believe this is referred to as
> "native webRTC".
>
> Has anything been done with iMX6-linux regarding webRTC?
> Does hardware acceleration support for VP8 encode/decode work on iMX6?
> Any pointers would be appreciated.

It all depends on what you intend to do and the tooling you want to
use. From your description it seems that the webRTC will be the
'communication channel' and the processing of the packages will be a
second step (audio / video / whatever). For VP8 it does work, as it is
supported in Chromium.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: webrtc question

[Ed Sutter]

Otavio thanks for responding,
> Hello Ed,
>
> On Tue, Jan 6, 2015 at 3:36 PM, Ed Sutter <ed.sutter@alcatel-lucent.com> wrote:
>> I apologize if this is the wrong list for this question, but I'm about to
>> embark on a iMX6 -based multimedia project that will be greatly helped by webRTC
>> functionality.
>>
>> Still getting up to speed on this, but it will not be Android, and it will
>> need to run outside the scope of a browser.  I believe this is referred to as
>> "native webRTC".
>>
>> Has anything been done with iMX6-linux regarding webRTC?
>> Does hardware acceleration support for VP8 encode/decode work on iMX6?
>> Any pointers would be appreciated.
> It all depends on what you intend to do and the tooling you want to
> use. From your description it seems that the webRTC will be the
> 'communication channel' and the processing of the packages will be a
> second step (audio / video / whatever). For VP8 it does work, as it is
> supported in Chromium.
>
Yea, the first step is to make my iMX6 based device be compatible with the
communication channels (and data formats, i.e. VP8/opus/etc..) 
used/supported
by webrtc.  The difference will be that (in certain modes) there is no 
user on
the device; it will be headless and autonomous, so I need to be able to feed
media to and extract media from the underlying webrtc base platform on 
the iMX6.
Does Chromium provide any non-browser type of hooks into webrtc down below?
Thanks,

Re: webrtc question

[Dmitriy B.]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

Hi,

 From your description it seems that the webRTC will be the
>> 'communication channel' and the processing of the packages will be a
>> second step (audio / video / whatever). For VP8 it does work, as it is
>> supported in Chromium.
>>
>
Unfortunately, it is not that simple. webrtc & libjingle is a long going
project from Google that is done as a big C++ framework that has everything
you need: encoding, decoding libs, communication libs and debug stuff.
Chrome is just hooked up to use that framework when webrtc actions appear
in users javascript. webrtc does not depend on HTML5 video playback for
example, where we have with patches imx-vpu in action. At least that is how
it did work around a year ago.


> Yea, the first step is to make my iMX6 based device be compatible with the
> communication channels (and data formats, i.e. VP8/opus/etc..)
> used/supported
> by webrtc.  The difference will be that (in certain modes) there is no
> user on
> the device; it will be headless and autonomous, so I need to be able to
> feed
> media to and extract media from the underlying webrtc base platform on the
> iMX6.
> Does Chromium provide any non-browser type of hooks into webrtc down below?
>

No, chromium links with webrtc which links with libvpx for
encoding/decoding. Libvpx then decides what to use.

I investigated this some time ago while working on a similar device (webrtc
app linked with Qt5 front end on ARM linux-native device communicating
other similar devices, it was a robotics project). You need to integrate
imx vpu encoding/decoding support to webrtc/libvpx internals or use Android
where it should be passed through from standard video acceleration
frameworks.

I've added Carlos to cc, he might know more on this topic.

Best Regards,
Dmitry Beykun

[-- Attachment #2: Type: text/html, Size: 2414 bytes --]

Re: webrtc question

[Ed Sutter]

Hi Dmitry,
Questions embedded,
Thanks
Ed
>
>          From your description it seems that the webRTC will be the
>         'communication channel' and the processing of the packages
>         will be a
>         second step (audio / video / whatever). For VP8 it does work,
>         as it is
>         supported in Chromium.
>
>
> Unfortunately, it is not that simple. webrtc & libjingle is a long 
> going project from Google that is done as a big C++ framework that has 
> everything you need: encoding, decoding libs, communication libs and 
> debug stuff. Chrome is just hooked up to use that framework when 
> webrtc actions appear in users javascript. webrtc does not depend on 
> HTML5 video playback for example, where we have with patches imx-vpu 
> in action. At least that is how it did work around a year ago.
Similarly, HTML5 doesn't use VP8 unless webrtc is involved right?
I'm hoping I don't need to get near any HTML at all, and can use 
libjingle along with the audio and video engines to do this.
>
>     Yea, the first step is to make my iMX6 based device be compatible
>     with the
>     communication channels (and data formats, i.e. VP8/opus/etc..)
>     used/supported
>     by webrtc.  The difference will be that (in certain modes) there
>     is no user on
>     the device; it will be headless and autonomous, so I need to be
>     able to feed
>     media to and extract media from the underlying webrtc base
>     platform on the iMX6.
>     Does Chromium provide any non-browser type of hooks into webrtc
>     down below?
>
>
> No, chromium links with webrtc which links with libvpx for 
> encoding/decoding. Libvpx then decides what to use.
>
> I investigated this some time ago while working on a similar device 
> (webrtc app linked with Qt5 front end on ARM linux-native device 
> communicating other similar devices, it was a robotics project). You 
> need to integrate imx vpu encoding/decoding support to webrtc/libvpx 
> internals or use Android where it should be passed through from 
> standard video acceleration frameworks.
So does that imply that there is no hardware (imx-vpu) acceleration 
support for VP8?
If this is true, is there any data on how the impacts CPU usage when VP8 
is used?

BTW... did you ever complete that project?  If not, was it due to 
iMX6/webrtc issues?

Re: webrtc question

[Dmitriy B.]

[-- Attachment #1: Type: text/plain, Size: 4265 bytes --]

2015-01-07 1:10 GMT+03:00 Ed Sutter <ed.sutter@alcatel-lucent.com>:

> Hi Dmitry,
> Questions embedded,
> Thanks
> Ed
>
>>
>>          From your description it seems that the webRTC will be the
>>         'communication channel' and the processing of the packages
>>         will be a
>>         second step (audio / video / whatever). For VP8 it does work,
>>         as it is
>>         supported in Chromium.
>>
>>
>> Unfortunately, it is not that simple. webrtc & libjingle is a long going
>> project from Google that is done as a big C++ framework that has everything
>> you need: encoding, decoding libs, communication libs and debug stuff.
>> Chrome is just hooked up to use that framework when webrtc actions appear
>> in users javascript. webrtc does not depend on HTML5 video playback for
>> example, where we have with patches imx-vpu in action. At least that is how
>> it did work around a year ago.
>>
> Similarly, HTML5 doesn't use VP8 unless webrtc is involved right?
> I'm hoping I don't need to get near any HTML at all, and can use libjingle
> along with the audio and video engines to do this.


IIRC, no, <video> tag is handled by Chrome, but when you start "webrtc
context", Chrome just proxies everything to webrtc library and waits for
answer from it. It is due to webrtc nature of being a separate library
ready for integration to any application. This is how Google wants it.

What do you mean by not getting near HTML? If you just want to use webrtc
in your application it is one situation, when you want a "webrtc context"
inside Chrome being accelerated - that is another situation. Sadly, both of
these situations will need you to dig into webrtc sources and add imx-vpu
support. There is no "easy way" here.


>
>>     Yea, the first step is to make my iMX6 based device be compatible
>>     with the
>>     communication channels (and data formats, i.e. VP8/opus/etc..)
>>     used/supported
>>     by webrtc.  The difference will be that (in certain modes) there
>>     is no user on
>>     the device; it will be headless and autonomous, so I need to be
>>     able to feed
>>     media to and extract media from the underlying webrtc base
>>     platform on the iMX6.
>>     Does Chromium provide any non-browser type of hooks into webrtc
>>     down below?
>>
>>
>> No, chromium links with webrtc which links with libvpx for
>> encoding/decoding. Libvpx then decides what to use.
>>
>> I investigated this some time ago while working on a similar device
>> (webrtc app linked with Qt5 front end on ARM linux-native device
>> communicating other similar devices, it was a robotics project). You need
>> to integrate imx vpu encoding/decoding support to webrtc/libvpx internals
>> or use Android where it should be passed through from standard video
>> acceleration frameworks.
>>
> So does that imply that there is no hardware (imx-vpu) acceleration
> support for VP8?
> If this is true, is there any data on how the impacts CPU usage when VP8
> is used?
>

Chrome HTML5 VP8 video playback acceleration != WebRTC VP8 decoding
acceleration! (see explanation above about webrtc being separate library
with own encoders/decoders).

HTML5 VP8 playback works, webrtc video decoding does not.


> BTW... did you ever complete that project?  If not, was it due to
> iMX6/webrtc issues?


This was a long going project on Samsung Exynos SoCs and we ended up just
running it there unaccelerated, just bare webrtc+libvpx, since 480p was
enough for robotics. We were trying to move to i.MX, but at that point only
LTIB existed and it was soft-float (which was a showstopper).

But now, you have much more chances to get webrtc running great on i.MX
when HTML5 acceleration works in Chrome. Just adapt that code to webrtc's
internals and link your Chrome with it.

I remember asking Carlos (author of imx patches for Chrome) about this on
IRC, but, unfortunately, I don't remember his answers. Hope he will explain
better than me :)

Other than here, you might want to ask at webrtc google group (
https://groups.google.com/forum/#!forum/discuss-webrtc) about how to
integrate video acceleration to the library.

Best Regards,
Dmitry Beykun

[-- Attachment #2: Type: text/html, Size: 5745 bytes --]

Re: webrtc question

[Ed Sutter]

>                From your description it seems that the webRTC will be the
>                 'communication channel' and the processing of the packages
>                 will be a
>                 second step (audio / video / whatever). For VP8 it
>         does work,
>                 as it is
>                 supported in Chromium.
>
>
>         Unfortunately, it is not that simple. webrtc & libjingle is a
>         long going project from Google that is done as a big C++
>         framework that has everything you need: encoding, decoding
>         libs, communication libs and debug stuff. Chrome is just
>         hooked up to use that framework when webrtc actions appear in
>         users javascript. webrtc does not depend on HTML5 video
>         playback for example, where we have with patches imx-vpu in
>         action. At least that is how it did work around a year ago.
>
>     Similarly, HTML5 doesn't use VP8 unless webrtc is involved right?
>     I'm hoping I don't need to get near any HTML at all, and can use
>     libjingle along with the audio and video engines to do this.
>
>
> IIRC, no, <video> tag is handled by Chrome, but when you start "webrtc 
> context", Chrome just proxies everything to webrtc library and waits 
> for answer from it. It is due to webrtc nature of being a separate 
> library ready for integration to any application. This is how Google 
> wants it.
>
> What do you mean by not getting near HTML? If you just want to use 
> webrtc in your application it is one situation, when you want a 
> "webrtc context" inside Chrome being accelerated - that is another 
> situation. Sadly, both of these situations will need you to dig into 
> webrtc sources and add imx-vpu support. There is no "easy way" here.
My use of this will (hopefully) not require a browser; hence, no HTML.  
I just want to use webrtc libraries in my application.
I've dug into this a few years ago (pre-webrtc-availability), using VP8 
alone in a different project.  Wasn't too messy but that
was prior to the point when all this "stuff" was integrated into 
webrtc.   Also, then I was working on a PC (no acceleration needed).
The difference here is that iMX6-quad != Intel-I7-quad; so I assume I'll 
need as much acceleration as I can get.
Really not sure what burden this will put on the iMX6Q, so that is yet 
to be determined.
>
>
>
>             Yea, the first step is to make my iMX6 based device be
>         compatible
>             with the
>             communication channels (and data formats, i.e. VP8/opus/etc..)
>             used/supported
>             by webrtc.  The difference will be that (in certain modes)
>         there
>             is no user on
>             the device; it will be headless and autonomous, so I need
>         to be
>             able to feed
>             media to and extract media from the underlying webrtc base
>             platform on the iMX6.
>             Does Chromium provide any non-browser type of hooks into
>         webrtc
>             down below?
>
>
>         No, chromium links with webrtc which links with libvpx for
>         encoding/decoding. Libvpx then decides what to use.
>
>         I investigated this some time ago while working on a similar
>         device (webrtc app linked with Qt5 front end on ARM
>         linux-native device communicating other similar devices, it
>         was a robotics project). You need to integrate imx vpu
>         encoding/decoding support to webrtc/libvpx internals or use
>         Android where it should be passed through from standard video
>         acceleration frameworks.
>
>     So does that imply that there is no hardware (imx-vpu)
>     acceleration support for VP8?
>     If this is true, is there any data on how the impacts CPU usage
>     when VP8 is used?
>
>
> Chrome HTML5 VP8 video playback acceleration != WebRTC VP8 decoding 
> acceleration! (see explanation above about webrtc being separate 
> library with own encoders/decoders).
> HTML5 VP8 playback works, webrtc video decoding does not.
Hmmm...(get ready for a naive comment)...
I would have thought once you get into VP8 then regardless of who calls 
it, the acceleration would be the same.
I realize it can be tuned and run in different modes; however I would 
have thought that
the bowels of the algorithm that acceleration would be applied to would 
be below this.
>
>     BTW... did you ever complete that project?  If not, was it due to
>     iMX6/webrtc issues?
>
>
> This was a long going project on Samsung Exynos SoCs and we ended up 
> just running it there unaccelerated, just bare webrtc+libvpx, since 
> 480p was enough for robotics. We were trying to move to i.MX, but at 
> that point only LTIB existed and it was soft-float (which was a 
> showstopper).
>
> But now, you have much more chances to get webrtc running great on 
> i.MX when HTML5 acceleration works in Chrome. Just adapt that code to 
> webrtc's internals and link your Chrome with it.
>
> I remember asking Carlos (author of imx patches for Chrome) about this 
> on IRC, but, unfortunately, I don't remember his answers. Hope he will 
> explain better than me :)
>
> Other than here, you might want to ask at webrtc google group 
> (https://groups.google.com/forum/#!forum/discuss-webrtc 
> <https://groups.google.com/forum/#%21forum/discuss-webrtc>) about how 
> to integrate video acceleration to the library.
>
Thanks very much for your comments.
Regards,
Ed

Re: webrtc question

[Dmitriy B.]

[-- Attachment #1: Type: text/plain, Size: 3210 bytes --]

2015-01-07 16:40 GMT+03:00 Ed Sutter <ed.sutter@alcatel-lucent.com>:
>
> IIRC, no, <video> tag is handled by Chrome, but when you start "webrtc
>> context", Chrome just proxies everything to webrtc library and waits for
>> answer from it. It is due to webrtc nature of being a separate library
>> ready for integration to any application. This is how Google wants it.
>>
>> What do you mean by not getting near HTML? If you just want to use webrtc
>> in your application it is one situation, when you want a "webrtc context"
>> inside Chrome being accelerated - that is another situation. Sadly, both of
>> these situations will need you to dig into webrtc sources and add imx-vpu
>> support. There is no "easy way" here.
>>
> My use of this will (hopefully) not require a browser; hence, no HTML.  I
> just want to use webrtc libraries in my application.
> I've dug into this a few years ago (pre-webrtc-availability), using VP8
> alone in a different project.  Wasn't too messy but that
> was prior to the point when all this "stuff" was integrated into webrtc.
>  Also, then I was working on a PC (no acceleration needed).
> The difference here is that iMX6-quad != Intel-I7-quad; so I assume I'll
> need as much acceleration as I can get.
> Really not sure what burden this will put on the iMX6Q, so that is yet to
> be determined.


People successfully ran various video software on i.MX6 (xbmc, gstreamer,
etc.), so with right amount of time you'll be able to add enc/dec support
to webrtc.

App that I've been working on was also just webrtc compiled for ARM
application, without Chrome or WebKit. On Exynos4412 we've been able to
render lowres video on libvpx CPU decoding, when webrtc is compiled in
hardfloat mode.


>  Hmmm...(get ready for a naive comment)...
> I would have thought once you get into VP8 then regardless of who calls
> it, the acceleration would be the same.
> I realize it can be tuned and run in different modes; however I would have
> thought that
> the bowels of the algorithm that acceleration would be applied to would be
> below this.
>

Embedded world has no "standardization" around video acceleration. There is
no VDPAU/VAAPI here. :)

For i.MX there's now gstreamer-imx available, both in 1.0 and 0.10 versions
and a library for lowlevel VPU access - imx-vpu. You can try to hook
gstreamer with webrtc, but I guess that will be really hard, unless webrtc
added support for gstreamer (which is unlikely, but check for that). Better
look at what Carlos did for Chrome browser and adapt that to libvpx. IIRC
he did not use gstreamer directly there.

For debugging purposes you can compile webrtc examples that come with the
library, you can find "p2p web cam chat" application there, as well as
other examples and unittests. Web cam app uses both encoder and decoder, so
it is great for performance testing right out of the box.

Also, just in case, if you'll be doing all this on i.MX, please share
bitbake recipes, if you will write ones. webrtc has a big load of
dependencies that might need fine tune in Yocto.


> Thanks very much for your comments.

Regards,
> Ed
>

Best Regards,
Dmitry Beykun

[-- Attachment #2: Type: text/html, Size: 4364 bytes --]