AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Florian Neuhaus]

Hi Laurent,

Thank you for your help, see my notes below:

Laurent Pinchart wrote on 2013-01-30:

>> Will result in the following and the following (screen flickers and
>> goes black again): [ 5293.617095] omapdss DISPC error: FIFO UNDERFLOW
>> on gfx, disabling the overlay [ 5293.678283] omapdss DISPC error: FIFO
>> UNDERFLOW on vid2, disabling the overlay
>> 
>> Output of mediactl -p while streaming:
>> http://pastebin.com/d9zDfKXu
>> 
>> OMAPDSS-config:
>> http://pastebin.com/JjF0CcCS
>> 
>> Now my questions:
>> Is this behaviour expected?
> 
> I don't think so. I'm not an expert on the OMAP DSS, but I wouldn't consider
> the above messages as normal.

Just as a note: This does not happen, if I disable the fb0 upon start of the
streamer.
My DSS config before the streamer start:

fb0 --- gfx --- lcd --- LCD
fb1 --- vid1 -/
        vid2 /

Description: I am using fb0 for the framebuffer-console. The fb1 is connected
with the overlay vid1 and used as the colorkey in your streamer application.
vid2 is directly used from within the streamer app for the ISP-output (at least
I think so ;)). Everything is connected to the lcd-manager and outputted
to a physical attached LCD.

My DSS config when streamer is running:

fb0     gfx --- lcd --- LCD
fb1 --- vid1 -/
        vid2 /

With this workaround the streamer will continue streaming after a blank/unblank.

> As buffers
> will stop flowing until the screen is unblanked, the live application
> will exit after a short select() timeout. This is an application issue.

If the AF/AEWB unit is enabled, the timeout doesn't happen as the
H3A-unit delivers still OMAP3_ISP_EVENT_EXCEPTION events.

> It doesn't explain the omapdss error, Tomi might be able to provide more
> information about that (but he is currently away until beginning of
> February if I'm not mistaken).

That would be very nice!

Regards,
Florian

Re: AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Laurent Pinchart]

Hi Florian,

On Thursday 31 January 2013 13:06:53 Florian Neuhaus wrote:
> Hi Laurent,
> 
> Thank you for your help, see my notes below:
> 
> Laurent Pinchart wrote on 2013-01-30:
> >> Will result in the following and the following (screen flickers and goes
> >> black again):
> >> [ 5293.617095] omapdss DISPC error: FIFO UNDERFLOW on gfx, disabling the
> >> overlay
> >> [ 5293.678283] omapdss DISPC error: FIFO UNDERFLOW on vid2, disabling the
> >> overlay
> >> 
> >> Output of mediactl -p while streaming:
> >> http://pastebin.com/d9zDfKXu
> >> 
> >> OMAPDSS-config:
> >> http://pastebin.com/JjF0CcCS
> >> 
> >> Now my questions:
> >> Is this behaviour expected?
> > 
> > I don't think so. I'm not an expert on the OMAP DSS, but I wouldn't
> > consider the above messages as normal.
> 
> Just as a note: This does not happen, if I disable the fb0 upon start of the
> streamer.
> My DSS config before the streamer start:
> 
> fb0 --- gfx --- lcd --- LCD
> fb1 --- vid1 -/
>         vid2 /
> 
> Description: I am using fb0 for the framebuffer-console. The fb1 is
> connected with the overlay vid1 and used as the colorkey in your streamer
> application. vid2 is directly used from within the streamer app for the
> ISP-output (at least I think so ;)). Everything is connected to the
> lcd-manager and outputted to a physical attached LCD.
> 
> My DSS config when streamer is running:
> 
> fb0     gfx --- lcd --- LCD
> fb1 --- vid1 -/
>         vid2 /
> 
> With this workaround the streamer will continue streaming after a
> blank/unblank.
>
> > As buffers will stop flowing until the screen is unblanked, the live
> > application will exit after a short select() timeout. This is an
> > application issue.
> 
> If the AF/AEWB unit is enabled, the timeout doesn't happen as the
> H3A-unit delivers still OMAP3_ISP_EVENT_EXCEPTION events.

Ah right. My live application should still be fixed not to timeout when AEWB 
is disabled, but that's out of scope here.

> > It doesn't explain the omapdss error, Tomi might be able to provide more
> > information about that (but he is currently away until beginning of
> > February if I'm not mistaken).
> 
> That would be very nice!

As this seems to be mostly a DSS issue I'll let Tomi handle it :-)

-- 
Regards,

Laurent Pinchart

AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Florian Neuhaus]

Hi Tomi,

Tomi Valkeinen wrote on 2013-02-07:

> FIFO underflow means that the DSS hardware wasn't able to fetch enough 
> pixel data in time to output them to the panel. Sometimes this happens 
> because of plain misconfiguration, but usually it happens because of 
> the hardware just can't do things fast enough with the configuration 
> the user has set.
> 
> In this case I see that you are using VRFB rotation on fb0, and the 
> rotation is
> 270 degrees. Rotating the fb is heavy, especially 90 and 270 degrees. 
> It may be that when the DSS is resumed, there's a peak in the mem 
> usage as DSS suddenly needs to fetch lots of data.
> 
> Another issue that could be involved is power management. After the 
> DSS is suspended, parts of OMAP may be put to sleep. When the DSS is 
> resumed, these parts need to be woken up, and it may be that there's a 
> higher mem latency for a short period of time right after resume. 
> Which could again cause DSS not getting enough pixel data.
> 
> You say the issue doesn't happen if you disable fb0. What happens if 
> you disable fb0, blank the screen, then unblank the screen, and after 
> that enable fb0 again?

By "disable fb0" do you mean disconnect fb0 from ovl0 or disable ovl0?
I have done both:
http://pastebin.com/Bxm1Z2RY

This works as expected.

Further tests I have done:

Enable fb1/ovl1 and hit some keys on the keyboard to let fb0/ovl0 update in the
background causes a fifo underflow too:
http://pastebin.com/f3JnMLsV

This happens only, if I enable the vrfb (rotate=3). So the whole thing
seems to be a rotation issue. Do you have some hints to trace down
the problem?

> How about if you disable VRFB rotation, either totally, or set the 
> rotation to 0 or 180 degrees?

Disable rotation is not an option for me, as we have a "wrong" oriented
portrait display with 480x800 which we must use in landscape mode...

> And you can also tune the PM so that deeper sleep states are prevented.
> I don't remember right away how this is done, though.
> 
>  Tomi

Regards,
Florian

P.S.
@Laurent: Do you use your streamer on a headless device? What is your DSS-config?
Do you have a framebuffer-console on fb0?

Re: AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Tomi Valkeinen]

[-- Attachment #1: Type: text/plain, Size: 3507 bytes --]

On 2013-02-14 11:30, Florian Neuhaus wrote:
> Hi Tomi,
> 
> Tomi Valkeinen wrote on 2013-02-07:
> 
>> FIFO underflow means that the DSS hardware wasn't able to fetch enough 
>> pixel data in time to output them to the panel. Sometimes this happens 
>> because of plain misconfiguration, but usually it happens because of 
>> the hardware just can't do things fast enough with the configuration 
>> the user has set.
>>
>> In this case I see that you are using VRFB rotation on fb0, and the 
>> rotation is
>> 270 degrees. Rotating the fb is heavy, especially 90 and 270 degrees. 
>> It may be that when the DSS is resumed, there's a peak in the mem 
>> usage as DSS suddenly needs to fetch lots of data.
>>
>> Another issue that could be involved is power management. After the 
>> DSS is suspended, parts of OMAP may be put to sleep. When the DSS is 
>> resumed, these parts need to be woken up, and it may be that there's a 
>> higher mem latency for a short period of time right after resume. 
>> Which could again cause DSS not getting enough pixel data.
>>
>> You say the issue doesn't happen if you disable fb0. What happens if 
>> you disable fb0, blank the screen, then unblank the screen, and after 
>> that enable fb0 again?
> 
> By "disable fb0" do you mean disconnect fb0 from ovl0 or disable ovl0?
> I have done both:
> http://pastebin.com/Bxm1Z2RY
> 
> This works as expected.

I think both disconnecting fb0 and ovl0, and disabling ovl0 end up doing
the same, which is disabling ovl0. Which is what I meant.

So, if I understand correctly, this only happens at unblank, and can be
circumvented by temporarily keeping ovl0 disabled during the unblank,
and enabling ovl0 afterwards works fine.

So for some reason the time of unblank is "extra heavy" for the memory bus.

Archit, I have a feeling that enabling the LCD is heavier on the memory
bus than what happens at VBLANK, even if both start fetching the pixels
for a fresh frame. You've been twiddling with the FIFO stuff, am I right
there?

> Further tests I have done:
> 
> Enable fb1/ovl1 and hit some keys on the keyboard to let fb0/ovl0 update in the
> background causes a fifo underflow too:
> http://pastebin.com/f3JnMLsV
> 
> This happens only, if I enable the vrfb (rotate=3). So the whole thing
> seems to be a rotation issue. Do you have some hints to trace down
> the problem?

Not rotation issue as such, but memory bandwidth issue.

>> How about if you disable VRFB rotation, either totally, or set the 
>> rotation to 0 or 180 degrees?
> 
> Disable rotation is not an option for me, as we have a "wrong" oriented
> portrait display with 480x800 which we must use in landscape mode...

I understand, I only meant that for testing purposes. VRFB rotation with
0 and 180 cause a slight impact on the mem bus, whereas 90 and 270
rotation cause a large impact. Also, as I mentioned earlier, the PM may
also affect this, as things may have been shut down in the OMAP. So
disabling PM related features could also "fix" the problem.

In many cases underflows are rather hard to debug and solve. There are
things in the DSS hardware like FIFO thresholds and prefetch, and VRFB
tile sizes, which can be changed (although unfortunately only by
modifying the drivers). How they should be changed if a difficult
question, though, and whether it'll help is also a question mark.

If you want to tweak those, I suggest you study them from the TRM.

 Tomi



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

Re: AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Laurent Pinchart]

[-- Attachment #1: Type: text/plain, Size: 3838 bytes --]

Hi Tomi,

On Thursday 14 February 2013 12:08:18 Tomi Valkeinen wrote:
> On 2013-02-14 11:30, Florian Neuhaus wrote:
> > Tomi Valkeinen wrote on 2013-02-07:
> >> FIFO underflow means that the DSS hardware wasn't able to fetch enough
> >> pixel data in time to output them to the panel. Sometimes this happens
> >> because of plain misconfiguration, but usually it happens because of
> >> the hardware just can't do things fast enough with the configuration
> >> the user has set.
> >> 
> >> In this case I see that you are using VRFB rotation on fb0, and the
> >> rotation is
> >> 270 degrees. Rotating the fb is heavy, especially 90 and 270 degrees.
> >> It may be that when the DSS is resumed, there's a peak in the mem
> >> usage as DSS suddenly needs to fetch lots of data.
> >> 
> >> Another issue that could be involved is power management. After the
> >> DSS is suspended, parts of OMAP may be put to sleep. When the DSS is
> >> resumed, these parts need to be woken up, and it may be that there's a
> >> higher mem latency for a short period of time right after resume.
> >> Which could again cause DSS not getting enough pixel data.
> >> 
> >> You say the issue doesn't happen if you disable fb0. What happens if
> >> you disable fb0, blank the screen, then unblank the screen, and after
> >> that enable fb0 again?
> > 
> > By "disable fb0" do you mean disconnect fb0 from ovl0 or disable ovl0?
> > I have done both:
> > http://pastebin.com/Bxm1Z2RY
> > 
> > This works as expected.
> 
> I think both disconnecting fb0 and ovl0, and disabling ovl0 end up doing
> the same, which is disabling ovl0. Which is what I meant.
> 
> So, if I understand correctly, this only happens at unblank, and can be
> circumvented by temporarily keeping ovl0 disabled during the unblank,
> and enabling ovl0 afterwards works fine.
> 
> So for some reason the time of unblank is "extra heavy" for the memory bus.
> 
> Archit, I have a feeling that enabling the LCD is heavier on the memory
> bus than what happens at VBLANK, even if both start fetching the pixels
> for a fresh frame. You've been twiddling with the FIFO stuff, am I right
> there?
> 
> > Further tests I have done:
> > 
> > Enable fb1/ovl1 and hit some keys on the keyboard to let fb0/ovl0 update
> > in the background causes a fifo underflow too:
> > http://pastebin.com/f3JnMLsV
> > 
> > This happens only, if I enable the vrfb (rotate=3). So the whole thing
> > seems to be a rotation issue. Do you have some hints to trace down
> > the problem?
> 
> Not rotation issue as such, but memory bandwidth issue.
> 
> >> How about if you disable VRFB rotation, either totally, or set the
> >> rotation to 0 or 180 degrees?
> > 
> > Disable rotation is not an option for me, as we have a "wrong" oriented
> > portrait display with 480x800 which we must use in landscape mode...
> 
> I understand, I only meant that for testing purposes. VRFB rotation with
> 0 and 180 cause a slight impact on the mem bus, whereas 90 and 270
> rotation cause a large impact. Also, as I mentioned earlier, the PM may
> also affect this, as things may have been shut down in the OMAP. So
> disabling PM related features could also "fix" the problem.
> 
> In many cases underflows are rather hard to debug and solve. There are
> things in the DSS hardware like FIFO thresholds and prefetch, and VRFB
> tile sizes, which can be changed (although unfortunately only by
> modifying the drivers). How they should be changed if a difficult
> question, though, and whether it'll help is also a question mark.

Naive question here, instead of killing the overlay completely when an 
underflow happens, couldn't the DSS driver somehow recover from that condition 
by restarting whatever needs to be restarted ?

> If you want to tweak those, I suggest you study them from the TRM.

-- 
Regards,

Laurent Pinchart

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 490 bytes --]

Re: AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Tomi Valkeinen]

[-- Attachment #1: Type: text/plain, Size: 1111 bytes --]

On 2013-02-14 13:07, Laurent Pinchart wrote:

>> In many cases underflows are rather hard to debug and solve. There are
>> things in the DSS hardware like FIFO thresholds and prefetch, and VRFB
>> tile sizes, which can be changed (although unfortunately only by
>> modifying the drivers). How they should be changed if a difficult
>> question, though, and whether it'll help is also a question mark.
> 
> Naive question here, instead of killing the overlay completely when an 
> underflow happens, couldn't the DSS driver somehow recover from that condition 
> by restarting whatever needs to be restarted ?

Yes. Killing the overlay is just the safest choice. Presumably if an
underflow happens, the problem is still there, and it'll just happen
again if you re-enable the overlay. Obviously this is not always the
case, as this problem at hand shows.

There's much to improve with the DSS driver's error handling, though. I
think first step would be to remove it totally from DSS, and move it to
omapfb/omapdrm. It's a bit difficult to handle the errors at the lowest
level.

 Tomi



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 899 bytes --]

Re: AW: omapdss/omap3isp/omapfb: Picture from omap3isp can't recover after a blank/unblank (or overlay disables after resuming)

[Archit Taneja]

On Thursday 14 February 2013 03:38 PM, Tomi Valkeinen wrote:
> On 2013-02-14 11:30, Florian Neuhaus wrote:
>> Hi Tomi,
>>
>> Tomi Valkeinen wrote on 2013-02-07:
>>
>>> FIFO underflow means that the DSS hardware wasn't able to fetch enough
>>> pixel data in time to output them to the panel. Sometimes this happens
>>> because of plain misconfiguration, but usually it happens because of
>>> the hardware just can't do things fast enough with the configuration
>>> the user has set.
>>>
>>> In this case I see that you are using VRFB rotation on fb0, and the
>>> rotation is
>>> 270 degrees. Rotating the fb is heavy, especially 90 and 270 degrees.
>>> It may be that when the DSS is resumed, there's a peak in the mem
>>> usage as DSS suddenly needs to fetch lots of data.
>>>
>>> Another issue that could be involved is power management. After the
>>> DSS is suspended, parts of OMAP may be put to sleep. When the DSS is
>>> resumed, these parts need to be woken up, and it may be that there's a
>>> higher mem latency for a short period of time right after resume.
>>> Which could again cause DSS not getting enough pixel data.
>>>
>>> You say the issue doesn't happen if you disable fb0. What happens if
>>> you disable fb0, blank the screen, then unblank the screen, and after
>>> that enable fb0 again?
>>
>> By "disable fb0" do you mean disconnect fb0 from ovl0 or disable ovl0?
>> I have done both:
>> http://pastebin.com/Bxm1Z2RY
>>
>> This works as expected.
>
> I think both disconnecting fb0 and ovl0, and disabling ovl0 end up doing
> the same, which is disabling ovl0. Which is what I meant.
>
> So, if I understand correctly, this only happens at unblank, and can be
> circumvented by temporarily keeping ovl0 disabled during the unblank,
> and enabling ovl0 afterwards works fine.
>
> So for some reason the time of unblank is "extra heavy" for the memory bus.
>
> Archit, I have a feeling that enabling the LCD is heavier on the memory
> bus than what happens at VBLANK, even if both start fetching the pixels
> for a fresh frame. You've been twiddling with the FIFO stuff, am I right
> there?

I don't think it's heavier in terms of memory bandwidth when you enable 
the LCD compared to the fetch of new frame content on a VBLANK. In both 
cases, the DSS tries to preload a few set of lines.

In terms of responsiveness from the interconnect, it's possible that DSS 
has a lower priority(in the round robin queue of initiators) compared to 
when it was active.

I don't know if LCD enable vs VBLANK has some sort of difference with VRFB.

>
>> Further tests I have done:
>>
>> Enable fb1/ovl1 and hit some keys on the keyboard to let fb0/ovl0 update in the
>> background causes a fifo underflow too:
>> http://pastebin.com/f3JnMLsV
>>
>> This happens only, if I enable the vrfb (rotate=3). So the whole thing
>> seems to be a rotation issue. Do you have some hints to trace down
>> the problem?
>
> Not rotation issue as such, but memory bandwidth issue.
>
>>> How about if you disable VRFB rotation, either totally, or set the
>>> rotation to 0 or 180 degrees?
>>
>> Disable rotation is not an option for me, as we have a "wrong" oriented
>> portrait display with 480x800 which we must use in landscape mode...
>
> I understand, I only meant that for testing purposes. VRFB rotation with
> 0 and 180 cause a slight impact on the mem bus, whereas 90 and 270
> rotation cause a large impact. Also, as I mentioned earlier, the PM may
> also affect this, as things may have been shut down in the OMAP. So
> disabling PM related features could also "fix" the problem.
>
> In many cases underflows are rather hard to debug and solve. There are
> things in the DSS hardware like FIFO thresholds and prefetch, and VRFB
> tile sizes, which can be changed (although unfortunately only by
> modifying the drivers). How they should be changed if a difficult
> question, though, and whether it'll help is also a question mark.

One thing which might help to debug this is to comment out the underflow 
error handling, and see if it continues to happen, this will tell if it 
was just a temporary bandwidth issue, or it's some consistent PM related 
thing

Archit