[dizzy] Choppy gstreamer video (MPEG TS over UDP)

[dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Nikolay Dimitrov]

Hi guys,

I have an issue with gstreamer, running on imx6q sabresd (dizzy). The
gstreamer pipeline has to play a live video stream over the network.
The video stream is MPEG TS over UDP unicast, carrying h264 video + mp3
audio. Here's the pipeline:

gst-launch udpsrc port=5000 ! decodebin2 ! mfw_v4lsink
The biggest issue right now with this setup is that the video is very
choppy - shortly after the playback starts (<10s) the video becomes
choppy, e.g. after decoding 1 frame it pauses for 2-3s before
displaying the next frame (all frames in between are skipped, so it
looks like a slideshow). Also, gstreamer is giving warnings in the
console:

WARNING: from element 
/GstPipeline:pipeline0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0: A 
lot of buffer
s are being dropped.
Additional debug info:
/home/picmaster/work/yocto-dizzy-sabresd/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.1
0.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesink.c(2875): 
gst_base_sink_is_too_late (): /GstPipeline:pipe
line0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0:
There may be a timestamping problem, or this computer is too slow.

As I'm by no means a gstreamer expert, I tried different combinations
of queues in all possible places in the pipeline, but with no success.
What's interesting to note is that if I make a local file copy of the
live stream, it plays without the choppines:

gst-launch playbin2 uri=file:///tmp/stream.ts

Has someone seen such behavior of gstreamer, or even better - has an
idea how to solve it? Thanks in advance for taking your time to read
and respond.

Kind regards,
Nikolay

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Daiane Angolini]

On Fri, Nov 14, 2014 at 4:46 PM, Nikolay Dimitrov <picmaster@mail.bg> wrote:
> Hi guys,
>
> I have an issue with gstreamer, running on imx6q sabresd (dizzy). The
> gstreamer pipeline has to play a live video stream over the network.
> The video stream is MPEG TS over UDP unicast, carrying h264 video + mp3
> audio. Here's the pipeline:
>
> gst-launch udpsrc port=5000 ! decodebin2 ! mfw_v4lsink
> The biggest issue right now with this setup is that the video is very
> choppy - shortly after the playback starts (<10s) the video becomes
> choppy, e.g. after decoding 1 frame it pauses for 2-3s before
> displaying the next frame (all frames in between are skipped, so it
> looks like a slideshow). Also, gstreamer is giving warnings in the
> console:
>
> WARNING: from element
> /GstPipeline:pipeline0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0: A lot
> of buffer
> s are being dropped.
> Additional debug info:
> /home/picmaster/work/yocto-dizzy-sabresd/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.1
> 0.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesink.c(2875):
> gst_base_sink_is_too_late (): /GstPipeline:pipe
> line0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0:
> There may be a timestamping problem, or this computer is too slow.
>
> As I'm by no means a gstreamer expert, I tried different combinations
> of queues in all possible places in the pipeline, but with no success.
> What's interesting to note is that if I make a local file copy of the
> live stream, it plays without the choppines:
>
> gst-launch playbin2 uri=file:///tmp/stream.ts
>
> Has someone seen such behavior of gstreamer, or even better - has an
> idea how to solve it? Thanks in advance for taking your time to read
> and respond.

I have never-ever got a good streaming audio+video.

There are several imx-community pages with how-tos to streaming (i.e.
[1][2]) you can use to get a better line instead of using decodebin
(which I´m not sure is going to really know your streaming caps
instead of guessing)

Another thing, your network does matter, a lot! So start with a local
network or even ppp.

I haven´t tested gstreamer 1.x, maybe it can be better for streaming
audio+video.

Daiane
[1]https://community.freescale.com/docs/DOC-94646
[2]https://community.freescale.com/docs/DOC-97781

>
> Kind regards,
> Nikolay
> --
> _______________________________________________
> meta-freescale mailing list
> meta-freescale@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/meta-freescale

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Nikolay Dimitrov]

Hi Daiane,

On 11/14/2014 10:25 PM, Daiane Angolini wrote:
> On Fri, Nov 14, 2014 at 4:46 PM, Nikolay Dimitrov <picmaster@mail.bg> wrote:
>> Hi guys,
>>
>> I have an issue with gstreamer, running on imx6q sabresd (dizzy). The
>> gstreamer pipeline has to play a live video stream over the network.
>> The video stream is MPEG TS over UDP unicast, carrying h264 video + mp3
>> audio. Here's the pipeline:
>>
>> gst-launch udpsrc port=5000 ! decodebin2 ! mfw_v4lsink
>> The biggest issue right now with this setup is that the video is very
>> choppy - shortly after the playback starts (<10s) the video becomes
>> choppy, e.g. after decoding 1 frame it pauses for 2-3s before
>> displaying the next frame (all frames in between are skipped, so it
>> looks like a slideshow). Also, gstreamer is giving warnings in the
>> console:
>>
>> WARNING: from element
>> /GstPipeline:pipeline0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0: A lot
>> of buffer
>> s are being dropped.
>> Additional debug info:
>> /home/picmaster/work/yocto-dizzy-sabresd/build/tmp/work/cortexa9hf-vfp-neon-poky-linux-gnueabi/gstreamer/0.1
>> 0.36-r2/gstreamer-0.10.36/libs/gst/base/gstbasesink.c(2875):
>> gst_base_sink_is_too_late (): /GstPipeline:pipe
>> line0/MFW_GST_V4LSINK_INFO_T:mfw_gst_v4lsink_info_t0:
>> There may be a timestamping problem, or this computer is too slow.
>>
>> As I'm by no means a gstreamer expert, I tried different combinations
>> of queues in all possible places in the pipeline, but with no success.
>> What's interesting to note is that if I make a local file copy of the
>> live stream, it plays without the choppines:
>>
>> gst-launch playbin2 uri=file:///tmp/stream.ts
>>
>> Has someone seen such behavior of gstreamer, or even better - has an
>> idea how to solve it? Thanks in advance for taking your time to read
>> and respond.
>
> I have never-ever got a good streaming audio+video.
>
> There are several imx-community pages with how-tos to streaming (i.e.
> [1][2]) you can use to get a better line instead of using decodebin
> (which I´m not sure is going to really know your streaming caps
> instead of guessing)

Playbin was very unstable for most of my tests (either refuses to play 
stream at all, or freezes on 1st or consequent frames). Then I started 
to use decodebin to have more control of the pipeline, and more stable 
results (to some minimal extent).

Most, if not all of these examples are hard to be applied in my case: my 
customer has already deployed infrastructure for the previous product 
generation, where they use MPEG-TS transport over multicast UDP, and I 
can't deviate from that. Also currently gstreamer-0.10 just doesn't 
support multicast, that's why I'm testing on unicast for now, and will 
have to fix the multicast bug soon.

> Another thing, your network does matter, a lot! So start with a local
> network or even ppp.

Totally agree. I'm using Gigabit LAN, which is dedicated to this 
development. Also, when deployed, the final product will also run on 
dedicated network segments with dedicated media servers, so I don't 
expect issues with that.

> I haven´t tested gstreamer 1.x, maybe it can be better for streaming
> audio+video.

I don't know whether the FSL plugins are compatible with Gstreamer-1.0. 
If they're, I can give it a try.

> [1]https://community.freescale.com/docs/DOC-94646
> [2]https://community.freescale.com/docs/DOC-97781

Thanks for sharing the links, the Freescale community was the first 
place where I looked for information.

Regards,
Nikolay

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Daiane Angolini]

On Sat, Nov 15, 2014 at 9:02 PM, Nikolay Dimitrov <picmaster@mail.bg> wrote:
> Hi Daiane,
>
>

>
>
> I don't know whether the FSL plugins are compatible with Gstreamer-1.0. If
> they're, I can give it a try.

http://git.yoctoproject.org/cgit/cgit.cgi/meta-fsl-arm/tree/recipes-multimedia/gstreamer/gstreamer1.0-plugins-imx_0.9.9.bb
http://git.yoctoproject.org/cgit/cgit.cgi/meta-fsl-arm/tree/recipes-multimedia/gstreamer/gst1.0-fsl-plugin_4.0.1.bb

Again, I have never tested video+audio streaming with gstreamer 1.0

Daiane

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Marco Trillo]

Hi Nikolay,

On 11/16/2014 12:02 AM, Nikolay Dimitrov wrote:
> Playbin was very unstable for most of my tests (either refuses to play 
> stream at all, or freezes on 1st or consequent frames). Then I started 
> to use decodebin to have more control of the pipeline, and more stable 
> results (to some minimal extent).

We have been doing extensive tests with MPEG-TS streaming over UDP and 
RTP with GStreamer 0.10, and we agree with your results.
Playbin2 is unrealible for UDP or RTP streaming. However, I don't think 
using "decodebin2" will be an improvement -- it is better to build the 
full pipeline manually.

That way you can configure your elements individually. In particular, we 
found that setting the `low-latency' property to true in `vpudec' and 
the `qos' property to false in `aiurdemux' provided better results.

Also, RTP streaming worked much better for us than raw UDP.

>
> Most, if not all of these examples are hard to be applied in my case: 
> my customer has already deployed infrastructure for the previous 
> product generation, where they use MPEG-TS transport over multicast 
> UDP, and I can't deviate from that. Also currently gstreamer-0.10 just 
> doesn't support multicast, that's why I'm testing on unicast for now, 
> and will have to fix the multicast bug soon.

gstreamer-0.10 does support multicast perfectly well: consult the 
`multicast-group' and `port' properties of `udpsrc' for more information 
(with gst-inspect).

Make sure your kernel has the multicast support enabled:
# zcat /proc/config.gz | grep -i multicast

For some reason, the default config for some kernels (linux-wandboard 
and linux-boundary IIRC) doesn't enable multicast.

>
>> Another thing, your network does matter, a lot! So start with a local
>> network or even ppp.
>
> Totally agree. I'm using Gigabit LAN, which is dedicated to this 
> development. Also, when deployed, the final product will also run on 
> dedicated network segments with dedicated media servers, so I don't 
> expect issues with that.

Yes; as said, we found the same choppy playback results and the network 
was perfectly fine. An alternative client (FFMPEG or VLC) worked fined 
at the same point.

Kind regards,
Marco.

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Nikolay Dimitrov]

Hi Marco,

On 11/18/2014 01:01 PM, Marco Trillo wrote:
> Hi Nikolay,
>
> On 11/16/2014 12:02 AM, Nikolay Dimitrov wrote:
>> Playbin was very unstable for most of my tests (either refuses to play
>> stream at all, or freezes on 1st or consequent frames). Then I started
>> to use decodebin to have more control of the pipeline, and more stable
>> results (to some minimal extent).
>
> We have been doing extensive tests with MPEG-TS streaming over UDP and
> RTP with GStreamer 0.10, and we agree with your results.
> Playbin2 is unrealible for UDP or RTP streaming. However, I don't think
> using "decodebin2" will be an improvement -- it is better to build the
> full pipeline manually.

Yes. I went to use decodebin just to be able to force the usage of
udpsrc in multicast mode, because I was not sure what was actually
happening inside playbin2. But tend to agree that custom pipelines (and
even C-application) are the way to go.

> That way you can configure your elements individually. In particular, we
> found that setting the `low-latency' property to true in `vpudec' and
> the `qos' property to false in `aiurdemux' provided better results.

"low-latency" definitely lowers the time to display 1st frame (< 1s),
but need to test whether it affects long-term stability. Hope to be
able to test soon "qos" property.

> Also, RTP streaming worked much better for us than raw UDP.

It's quite possible. Unfortunately, RTP this is not an option for me at
the moment.

>> Most, if not all of these examples are hard to be applied in my case:
>> my customer has already deployed infrastructure for the previous
>> product generation, where they use MPEG-TS transport over multicast
>> UDP, and I can't deviate from that. Also currently gstreamer-0.10 just
>> doesn't support multicast, that's why I'm testing on unicast for now,
>> and will have to fix the multicast bug soon.
>
> gstreamer-0.10 does support multicast perfectly well: consult the
> `multicast-group' and `port' properties of `udpsrc' for more information
> (with gst-inspect).
>
> Make sure your kernel has the multicast support enabled:
> # zcat /proc/config.gz | grep -i multicast
>
> For some reason, the default config for some kernels (linux-wandboard
> and linux-boundary IIRC) doesn't enable multicast.

Yep, I also found this out the hard way. After enabling the
CONFIG_IP_MULTICAST it's much better now. I can't remember exactly what
was the error message I saw several days ago, but it was something like:

"added **something**: -1, Not found"

So this is why I said that the multicast didn't worked out of the box.
Obviously, I wasn't right and now it works. But I can't reproduce this
error message anymore.

>>> Another thing, your network does matter, a lot! So start with a local
>>> network or even ppp.
>>
>> Totally agree. I'm using Gigabit LAN, which is dedicated to this
>> development. Also, when deployed, the final product will also run on
>> dedicated network segments with dedicated media servers, so I don't
>> expect issues with that.
>
> Yes; as said, we found the same choppy playback results and the network
> was perfectly fine. An alternative client (FFMPEG or VLC) worked fined
> at the same point.

Same here, tests performed with ffmpeg & vlc on x86 machines showed
that at least the TS stream and LAN network were OK.

Regards,
Nikolay

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Marco Trillo]

Hi,

On 11/18/2014 04:06 PM, Nikolay Dimitrov wrote:
>> That way you can configure your elements individually. In particular, we
>> found that setting the `low-latency' property to true in `vpudec' and
>> the `qos' property to false in `aiurdemux' provided better results.
>
> "low-latency" definitely lowers the time to display 1st frame (< 1s),
> but need to test whether it affects long-term stability. Hope to be
> able to test soon "qos" property.

One small correction: the `qos' property belongs to the video sink (we 
are using `mfw_v4lsink'), not to `aiurdemux' as I said in my first e-mail...

Regarding the time-to-first-frame you mention, we are also running in a 
big show-stopper issue with `aiurdemux': depending on the number of PIDs 
present in the program table -- for example if the program contains 
subtitles or multiple audio streams --, the demuxer takes a long, long 
time to provide the pads. Only in relatively simple programs (one audio 
and one video PID) the demuxer seems to behave right.

It is quite possible this is not an issue for you if you control the 
source stream. In any case, more information is available at: 
<https://community.freescale.com/thread/324048>

Kind regards,
Marco

Re: [dizzy] Choppy gstreamer video (MPEG TS over UDP)

[Nikolay Dimitrov]

Hi Marco,

On 11/19/2014 09:36 AM, Marco Trillo wrote:
> Hi,
>
> On 11/18/2014 04:06 PM, Nikolay Dimitrov wrote:
>>> That way you can configure your elements individually. In particular, we
>>> found that setting the `low-latency' property to true in `vpudec' and
>>> the `qos' property to false in `aiurdemux' provided better results.
>>
>> "low-latency" definitely lowers the time to display 1st frame (< 1s),
>> but need to test whether it affects long-term stability. Hope to be
>> able to test soon "qos" property.
>
> One small correction: the `qos' property belongs to the video sink (we
> are using `mfw_v4lsink'), not to `aiurdemux' as I said in my first
> e-mail...

Thanks for the clarification.

> Regarding the time-to-first-frame you mention, we are also running in a
> big show-stopper issue with `aiurdemux': depending on the number of PIDs
> present in the program table -- for example if the program contains
> subtitles or multiple audio streams --, the demuxer takes a long, long
> time to provide the pads. Only in relatively simple programs (one audio
> and one video PID) the demuxer seems to behave right.

Well, I actually ditched aiurdemux and started using the latest
Fluendo's MPEG TS demuxer. I can't say with confidence that it's the 
best solution, but at least it allows me to go forward.

I personally think that the amount of available programs and elementary
streams in the MPEG TS is not an issue at all. The PAT is transmitted
regularly (at most on 140ms), in order to reduce the "time to 1st
acquisition". Also PMTs are sent regularly. The problem is actually how
the control packets trigger the additional demuxer logic.

For example, I noticed is that the Fluendo MPEG TS demuxer rely for
some reason on the regular update of PAT/PMT/CAT tables with the
current_next_indicator bit. The streams I have access to doesn't update
this bit regularly (if at all), that's why the demuxer waits
practically for inifinity before demuxing packets. What I did as an
experiment was to force the parsing of PAT/PMT/CAT table disregarding
this update bit, and suddenly the demuxer started to work quickly.

> It is quite possible this is not an issue for you if you control the
> source stream. In any case, more information is available at:
> <https://community.freescale.com/thread/324048>

Thanks, this was definitely useful info, and a very good remark -
mapping tables just describe relations between children and parent
logical streams, but packets of the actual streams (including PMT) can
start to arrive much later, so the demuxer should always work with
whatever is available *now*, not after 15min when the user is dead from
boredom :D.

In the end, I still have issues with the Gstreamer pipeline stability
with network streams - it sometimes freezes either for a while (1-3s),
or fully, begging for Ctrl-C. I plan to look at whether some of the
pipeline elements are mutually blocking each other, causing a deadlock.

Kind regards,
Nikolay

PS: Please forgive me if someone feels this is going too off-topic.