PROBLEM: AST2500 BMC video output disabled by reboot (regression)

PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Nick Bowler]

Hi,

I recently noticed that on current kernels I lose video output from
my Blackbird's AST2500 BMC after a reboot, which makes it difficult to
boot the system again (the video output will come on only after Linux
is booted again).

With Linux 6.6 and earlier, there is no problem rebooting, everything
is fine.

I bisected to this commit:

  ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
  commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
  Author: Douglas Anderson <dianders@chromium.org>
  Date:   Fri Sep 1 16:39:53 2023 -0700

      drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers

This commit does not revert cleanly on 6.14, but I can easily undo
the changes this commit made to the ast driver, and this is enough
to restore working behaviour.

Let me know if you need any more info!

Thanks,
  Nick

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Doug Anderson]

Hi,

On Sun, Apr 20, 2025 at 9:26 PM Nick Bowler <nbowler@draconx.ca> wrote:
>
> Hi,
>
> I recently noticed that on current kernels I lose video output from
> my Blackbird's AST2500 BMC after a reboot, which makes it difficult to
> boot the system again (the video output will come on only after Linux
> is booted again).
>
> With Linux 6.6 and earlier, there is no problem rebooting, everything
> is fine.
>
> I bisected to this commit:
>
>   ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
>   commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
>   Author: Douglas Anderson <dianders@chromium.org>
>   Date:   Fri Sep 1 16:39:53 2023 -0700
>
>       drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers
>
> This commit does not revert cleanly on 6.14, but I can easily undo
> the changes this commit made to the ast driver, and this is enough
> to restore working behaviour.
>
> Let me know if you need any more info!

Bleh. That's not good. If I had to guess there's some subtle bug /
missing timing constraint that's being triggered here. A few things to
try:

1. Add a several second delay after the call to
"drm_atomic_helper_shutdown()", like msleep(5000) or something like
that. That's kind of a shot in the dark, but it's fairly common for
panels to get upset if you turn them off and then turn them on again
too quickly. This would be my blind guess of what is happening.

2. Could you give more details about what panel you're using? Ideally
it'd be great if you could say which device tree you're using too.

3. Any chance you can gather the `dmesg` from a failing boot and
provide it somehow? Are there any errors in the logs from the failing
boot?

-Doug

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Nick Bowler]

[-- Attachment #1: Type: text/plain, Size: 2472 bytes --]

Hi Doug,

On Mon, Apr 28, 2025 at 01:40:25PM -0700, Doug Anderson wrote:
> On Sun, Apr 20, 2025 at 9:26■PM Nick Bowler <nbowler@draconx.ca> wrote:
> > I recently noticed that on current kernels I lose video output from
> > my Blackbird's AST2500 BMC after a reboot
[...]
> >   ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
> >   commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
> >   Author: Douglas Anderson <dianders@chromium.org>
> >   Date:   Fri Sep 1 16:39:53 2023 -0700
> >
> >       drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers
[...]
> Bleh. That's not good. If I had to guess there's some subtle bug /
> missing timing constraint that's being triggered here. A few things to
> try:
> 
> 1. Add a several second delay after the call to
> "drm_atomic_helper_shutdown()", like msleep(5000) or something like
> that. That's kind of a shot in the dark, but it's fairly common for
> panels to get upset if you turn them off and then turn them on again
> too quickly. This would be my blind guess of what is happening.

Adding msleep(5000) does nothing except that once the video turns off
it now takes 5 seconds longer to reboot.

> 2. Could you give more details about what panel you're using?

According to the documentation I have for the machine, the video output
of the AST2500 BMC is connected to an IT66121 HDMI transmitter.

Then in turn I have that connected to some generic HDMI->VGA adapter
(PrimeCables branded).  I also tried with another much more expensive
device (Extron DVI-RGB 200) and observe no difference in behaviour.

i think these devices are working and there's just no output signal
on the hdmi port.

> Ideally it'd be great if you could say which device tree you're using too.

Not sure how to answer this.  Do you want me to look at something
specific in /proc/device-tree?  Or dump it somehow?

> 3. Any chance you can gather the `dmesg` from a failing boot and
> provide it somehow? Are there any errors in the logs from the failing
> boot?

To clarify, there is no boot failure.  There is just no video output
after rebooting.  I can then boot Linux again by any method that works
without being able to see the screen, and then everything is fine once
I do that.

I've attached the dmesg output (gzipped) from after such a reboot.
Except for the order and the timestamps, the messages are identical to
when I boot after rebooting a kernel which does not disable the video.

Thanks,
  Nick

[-- Attachment #2: boot-novid.log.gz --]
[-- Type: application/gzip, Size: 11710 bytes --]

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Doug Anderson]

Hi,

On Wed, Apr 30, 2025 at 6:28 AM Nick Bowler <nbowler@draconx.ca> wrote:
>
> Hi Doug,
>
> On Mon, Apr 28, 2025 at 01:40:25PM -0700, Doug Anderson wrote:
> > On Sun, Apr 20, 2025 at 9:26■PM Nick Bowler <nbowler@draconx.ca> wrote:
> > > I recently noticed that on current kernels I lose video output from
> > > my Blackbird's AST2500 BMC after a reboot
> [...]
> > >   ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
> > >   commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
> > >   Author: Douglas Anderson <dianders@chromium.org>
> > >   Date:   Fri Sep 1 16:39:53 2023 -0700
> > >
> > >       drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers
> [...]
> > Bleh. That's not good. If I had to guess there's some subtle bug /
> > missing timing constraint that's being triggered here. A few things to
> > try:
> >
> > 1. Add a several second delay after the call to
> > "drm_atomic_helper_shutdown()", like msleep(5000) or something like
> > that. That's kind of a shot in the dark, but it's fairly common for
> > panels to get upset if you turn them off and then turn them on again
> > too quickly. This would be my blind guess of what is happening.
>
> Adding msleep(5000) does nothing except that once the video turns off
> it now takes 5 seconds longer to reboot.

Dang. Thanks for checking.


> > 2. Could you give more details about what panel you're using?
>
> According to the documentation I have for the machine, the video output
> of the AST2500 BMC is connected to an IT66121 HDMI transmitter.
>
> Then in turn I have that connected to some generic HDMI->VGA adapter
> (PrimeCables branded).  I also tried with another much more expensive
> device (Extron DVI-RGB 200) and observe no difference in behaviour.
>
> i think these devices are working and there's just no output signal
> on the hdmi port.

I've got a pile of crappy/generic HDMI to VGA adapters and I've had
mixed success with them working. They're not really passive adapters,
so they somehow need power to convert the HDMI to VGA. They seem to
run off the power given to them by the HDMI port and I've seen cases
where (I suspect) they get into a bad state. I've seen cases where
they need to be plugged in at just the right time in order to work and
I suspect that there's some sort of chicken-and-egg problem here.
Maybe in most states the HDMI port doesn't get power if nothing is
plugged in but (because of their design) they can't report plugged in
without getting power? ...but then there are maybe some cases where
power is given anyway? I'm spitballing here.

I don't know much about the Extron DVI-RGB 200.

Do you happen to have anything that's just a normal HDMI sink, like a
TV or a standard monitor that takes HDMI?


> > Ideally it'd be great if you could say which device tree you're using too.
>
> Not sure how to answer this.  Do you want me to look at something
> specific in /proc/device-tree?  Or dump it somehow?

Ah, I get it. On many device tree boards people use a dts file that
lives in the Linux source base and then bundle it with the kernel.
Looks like yours is provided by your firmware?


> > 3. Any chance you can gather the `dmesg` from a failing boot and
> > provide it somehow? Are there any errors in the logs from the failing
> > boot?
>
> To clarify, there is no boot failure.  There is just no video output
> after rebooting.  I can then boot Linux again by any method that works
> without being able to see the screen, and then everything is fine once
> I do that.

Super weird. So every other boot works?

I guess I'd be interested in other types of tests to see what's going
on. Aside from trying some other, more standard HDMI sinks, I'd love
to see the results of:

1. HDMI is supposed to be hotpluggable. If you've got a boot where the
display isn't working, what if you unplug the HDMI and plug it back
in. Does it fix it?

2. Does the hotplug experience change if you boot with the revert?
AKA: boot up with the revert (so everything is working normally),
unplug HDMI, wait a few seconds, plug HDMI back in? Is this different
than #1?

3. What about if you fully power off and then power on? Does the
display work reliably in this case, or are things different between
ToT and with the revert?

4. What about if you fully power off, unplug the HDMI, wait a few
seconds, plug the HDMI, and power on? Does that work? Are things
different between ToT and with the revert?


> I've attached the dmesg output (gzipped) from after such a reboot.
> Except for the order and the timestamps, the messages are identical to
> when I boot after rebooting a kernel which does not disable the video.

Thanks! dmesg could still be useful but I was hoping for some error
messages. I guess not.

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Nick Bowler]

Hi,

On Wed, Apr 30, 2025 at 10:05:44AM -0700, Doug Anderson wrote:
> On Wed, Apr 30, 2025 at 6:28 AM Nick Bowler <nbowler@draconx.ca> wrote:
> > On Mon, Apr 28, 2025 at 01:40:25PM -0700, Doug Anderson wrote:
> > > On Sun, Apr 20, 2025 at 9:26 PM Nick Bowler <nbowler@draconx.ca> wrote:
> > > > I recently noticed that on current kernels I lose video output from
> > > > my Blackbird's AST2500 BMC after a reboot
> > [...]
> > > >   ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
> > > >   commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
> > > >   Author: Douglas Anderson <dianders@chromium.org>
> > > >   Date:   Fri Sep 1 16:39:53 2023 -0700
> > > >
> > > >       drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers
[...]
> Do you happen to have anything that's just a normal HDMI sink, like a
> TV or a standard monitor that takes HDMI?

I can probably find something else to try later.

> > To clarify, there is no boot failure.  There is just no video output
> > after rebooting.  I can then boot Linux again by any method that works
> > without being able to see the screen, and then everything is fine once
> > I do that.
> 
> Super weird. So every other boot works?

On a new/broken kernel, every time I run "reboot" the video turns off
when Linux does whatever it does to make the system restart.

The video comes on again if I manage to boot it up again.

The problem is that I have to do that without using the screen.  So I
can boot Linux via the serial port, or via the BMC web interface, or
by just typing on the keyboard without seeing what is happening.

> I guess I'd be interested in other types of tests to see what's going
> on. Aside from trying some other, more standard HDMI sinks, I'd love
> to see the results of:
> 
> 1. HDMI is supposed to be hotpluggable. If you've got a boot where the
> display isn't working, what if you unplug the HDMI and plug it back
> in. Does it fix it?

Unplugging/replugging the cable when the video is off after rebooting
does not cause it to start working again.

> 2. Does the hotplug experience change if you boot with the revert?
> AKA: boot up with the revert (so everything is working normally),
> unplug HDMI, wait a few seconds, plug HDMI back in? Is this different
> than #1?

I have certainly never noticed any problem related to hot plugging while
the video output is working but this is not something I ever normally
do.  I can try it later.

> 3. What about if you fully power off and then power on? Does the
> display work reliably in this case, or are things different between
> ToT and with the revert?

There is never any problem with the video output if I fully power off/on
the system.  The problem only occurs on a reboot.

> 4. What about if you fully power off, unplug the HDMI, wait a few
> seconds, plug the HDMI, and power on? Does that work? Are things
> different between ToT and with the revert?

Again there is never any problem if I fully power off/on the system.

Thanks,
  Nick

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Doug Anderson]

Hi,

On Wed, Apr 30, 2025 at 10:52 AM Nick Bowler <nbowler@draconx.ca> wrote:
>
> > > To clarify, there is no boot failure.  There is just no video output
> > > after rebooting.  I can then boot Linux again by any method that works
> > > without being able to see the screen, and then everything is fine once
> > > I do that.
> >
> > Super weird. So every other boot works?
>
> On a new/broken kernel, every time I run "reboot" the video turns off
> when Linux does whatever it does to make the system restart.
>
> The video comes on again if I manage to boot it up again.
>
> The problem is that I have to do that without using the screen.  So I
> can boot Linux via the serial port, or via the BMC web interface, or
> by just typing on the keyboard without seeing what is happening.

Ohhhhhh! I was assuming that your problem looked like this:

1. Boot up Linux, video works.

2. Type "reboot" and the system boots back to Linux w/ no video.

3. Log into Linux via UART or ssh and type "reboot". System boots back
to Linux and video works.

...and then if you repeat steps #2 and #3 it keeps alternating with
Linux having video and not having video.

It sounds like instead what you're saying is that you're typing
"reboot" and then booting into some other operating system / BIOS and
the video doesn't work in that other (non-Linux) environment. Is that
right?

If I (finally) got it right, then I'm not really sure what to suggest.
It would sure seem like a bug in the other OS/BIOS. We could possibly
land the revert you have. I'd imagine it's papering over the bug, but
it's no real skin off my teeth. As a test, I'd imagine that if you
turned the screen off via commandline (I never really do this, but a
search turns up [1]) and then reboot with the screen off then I'd
imagine you run into the same trouble with or without the patch. Is
that true?

FWIW: The original incentive for adding all the shutdown calls (aside
from the fact that it's documented that they should be there) was to
make sure embedded panels shutdown properly. It looks as if no
embedded panels will really be used with this driver anyway.

[1] https://www.baeldung.com/linux/cli-monitor-turn-on-off

Re: PROBLEM: AST2500 BMC video output disabled by reboot (regression)

[Thomas Zimmermann]

Hi,

it's been a while since you sent this report. I assume, the problem is 
this there?

Am 30.04.25 um 15:28 schrieb Nick Bowler:
> Hi Doug,
>
> On Mon, Apr 28, 2025 at 01:40:25PM -0700, Doug Anderson wrote:
>> On Sun, Apr 20, 2025 at 9:26■PM Nick Bowler <nbowler@draconx.ca> wrote:
>>> I recently noticed that on current kernels I lose video output from
>>> my Blackbird's AST2500 BMC after a reboot
> [...]
>>>    ce3d99c8349584bc0fbe1e21918a3ea1155343aa is the first bad commit
>>>    commit ce3d99c8349584bc0fbe1e21918a3ea1155343aa
>>>    Author: Douglas Anderson <dianders@chromium.org>
>>>    Date:   Fri Sep 1 16:39:53 2023 -0700
>>>
>>>        drm: Call drm_atomic_helper_shutdown() at shutdown time for misc drivers
> [...]
>> Bleh. That's not good. If I had to guess there's some subtle bug /
>> missing timing constraint that's being triggered here. A few things to
>> try:
>>
>> 1. Add a several second delay after the call to
>> "drm_atomic_helper_shutdown()", like msleep(5000) or something like
>> that. That's kind of a shot in the dark, but it's fairly common for
>> panels to get upset if you turn them off and then turn them on again
>> too quickly. This would be my blind guess of what is happening.
> Adding msleep(5000) does nothing except that once the video turns off
> it now takes 5 seconds longer to reboot.
>
>> 2. Could you give more details about what panel you're using?
> According to the documentation I have for the machine, the video output
> of the AST2500 BMC is connected to an IT66121 HDMI transmitter.

That dmesg refers to a SIL164. I always thought these where only for 
DVI. With the IT66121, I'd expect the warning from [1] in the dmesg.

[1] 
https://elixir.bootlin.com/linux/v6.16.3/source/drivers/gpu/drm/ast/ast_main.c#L196


The ast driver doesn't do much during shutdown. Could you please 
out-comment the lines at either [2] xor [3] and report on either effect? 
These calls turn of the video chip's memory access and sync pulses. Not 
doing that might resolve the problem.

[2] 
https://elixir.bootlin.com/linux/v6.16.3/source/drivers/gpu/drm/ast/ast_mode.c#L835
[3] 
https://elixir.bootlin.com/linux/v6.16.3/source/drivers/gpu/drm/ast/ast_mode.c#L839

Best regards
Thomas

>
> Then in turn I have that connected to some generic HDMI->VGA adapter
> (PrimeCables branded).  I also tried with another much more expensive
> device (Extron DVI-RGB 200) and observe no difference in behaviour.
>
> i think these devices are working and there's just no output signal
> on the hdmi port.
>
>> Ideally it'd be great if you could say which device tree you're using too.
> Not sure how to answer this.  Do you want me to look at something
> specific in /proc/device-tree?  Or dump it somehow?
>
>> 3. Any chance you can gather the `dmesg` from a failing boot and
>> provide it somehow? Are there any errors in the logs from the failing
>> boot?
> To clarify, there is no boot failure.  There is just no video output
> after rebooting.  I can then boot Linux again by any method that works
> without being able to see the screen, and then everything is fine once
> I do that.
>
> I've attached the dmesg output (gzipped) from after such a reboot.
> Except for the order and the timestamps, the messages are identical to
> when I boot after rebooting a kernel which does not disable the video.
>
> Thanks,
>    Nick

-- 
--
Thomas Zimmermann
Graphics Driver Developer
SUSE Software Solutions Germany GmbH
Frankenstrasse 146, 90461 Nuernberg, Germany
GF: Ivo Totev, Andrew Myers, Andrew McDonald, Boudien Moerman
HRB 36809 (AG Nuernberg)