i.MX8MP Hantro G2 HEVC decoding problems

i.MX8MP Hantro G2 HEVC decoding problems

[Krzysztof Hałasa]

Hi,

I'm trying to decode H.265 full HD RTP stream on an i.MX8MP CPU. Should
I expect it to work, or is there something missing?

Small streams (640x360 etc.) are decoded. With 1080p, the board panics
or does something similar.

Generally Hantro G2 doesn't finish decoding some frame. I have a test
video - it can probably choke on random frames, including P-frames
(this is I/P-frame only stream, produced by the H2 counterpart).

The details are foggy at this point, but I'm receiving interrupts:

# dmesg | grep irq
...
[   75.002276] hantro_g2_irq: 0x1100 masked 0x1000 <<< status registers,
                                                shows decoding complete

[   75.143611] hantro_g2_irq: 0x2101 masked 0x0 <<< this is the problem

What could that mean? DEC_E is probably "ERROR", lack of bit 12 is "not
ready", but why do I get this and what does bit 13 possibly mean?

#define G2_REG_INTERRUPT                G2_SWREG(1)
#define G2_REG_INTERRUPT_DEC_RDY_INT    BIT(12)
#define G2_REG_INTERRUPT_DEC_ABORT_E    BIT(5)
#define G2_REG_INTERRUPT_DEC_IRQ_DIS    BIT(4)
#define G2_REG_INTERRUPT_DEC_E          BIT(0)

... then the condition doesn't seem to improve:
[   95.198362] hantro_watchdog:126: frame processing timed out!
[   95.323410] imx-pgc imx-pgc-domain.8: failed to power down ADB400

and it can randomly panic, show an SError, or do nothing of the sort.

How do I even start debugging such stuff?

H.264 doesn't have this problem.
-- 
Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa

Re: i.MX8MP Hantro G2 HEVC decoding problems

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 3621 bytes --]

Hi Krzysztof,

Le lundi 03 novembre 2025 à 14:52 +0100, Krzysztof Hałasa a écrit :
> Hi,
> 
> I'm trying to decode H.265 full HD RTP stream on an i.MX8MP CPU. Should
> I expect it to work, or is there something missing?
> 
> Small streams (640x360 etc.) are decoded. With 1080p, the board panics
> or does something similar.

If its size related, always make sure you have enough CMA for this type of
system. Though lower description seems to indicated its not the issue.

> 
> Generally Hantro G2 doesn't finish decoding some frame. I have a test
> video - it can probably choke on random frames, including P-frames
> (this is I/P-frame only stream, produced by the H2 counterpart).
> 
> The details are foggy at this point, but I'm receiving interrupts:
> 
> # dmesg | grep irq
> ...
> [   75.002276] hantro_g2_irq: 0x1100 masked 0x1000 <<< status registers,
>                                                 shows decoding complete
> 
> [   75.143611] hantro_g2_irq: 0x2101 masked 0x0 <<< this is the problem
> 
> What could that mean? DEC_E is probably "ERROR", lack of bit 12 is "not
> ready", but why do I get this and what does bit 13 possibly mean?

BUS Error, typically an AXI errors. See pending patch below:

> 
> #define G2_REG_INTERRUPT                G2_SWREG(1)
> #define G2_REG_INTERRUPT_DEC_RDY_INT    BIT(12)
> #define G2_REG_INTERRUPT_DEC_ABORT_E    BIT(5)
> #define G2_REG_INTERRUPT_DEC_IRQ_DIS    BIT(4)
> #define G2_REG_INTERRUPT_DEC_E          BIT(0)
> 
> ... then the condition doesn't seem to improve:
> [   95.198362] hantro_watchdog:126: frame processing timed out!
> [   95.323410] imx-pgc imx-pgc-domain.8: failed to power down ADB400
> 
> and it can randomly panic, show an SError, or do nothing of the sort.

In queue for the next kernel release, we have a rewrite of the IRQ handler:

https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/19c286b755072a22a063052f530a6b1fac8a1f63

The important take away for this chip is that it does not always stop on errors.
It can also raise a dry IRQ (I don't have explanation, only happen on IMX8
boards). The new handler properly check if the hardware is still running, which
avoid programming over a live IP. The G2 has fancy internal clock gating, so
doing so can lead to CPU hang if the register you set is hooked to a gated
portion of the IP. At least this is my mental model of it.

And there is an important bug fix for missing references, without that, you get
thousands of error IRQs while the IP is fetch from address 0x0 and its
surrounding. This large burst of IRQ, combine with multiple memory related
erratas of IMX8 boards cause random other things to fall appart.

https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/47825b1646a6a9eca0f90baa3d4f98947c2add96

> 
> How do I even start debugging such stuff?
> 
> H.264 doesn't have this problem.

The Hantro G1 driver is a lot more robust indeed, it has been deployed on
ChromeOS for many years (though that chips is a bit obsolete now). HEVC was
contributed later on, I don't think it have had that many users. It has been
mostly tested with reliable media, though it can also fails there if you seek to
a GOPs with CRA keyframes and RASL (skippable) leading frames, and your
userspace does not pre-emptively skip these (GStreamer, which I'll fix soon).

I'm sure we will find more things to harden around this driver, so your feedback
is all very appreciated.

regards,
Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: i.MX8MP Hantro G2 HEVC decoding problems

[Krzysztof Hałasa]

Hi Nicolas,

> In queue for the next kernel release, we have a rewrite of the IRQ handler:
>
> https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/19c286b755072a22a063052f530a6b1fac8a1f63
>
> The important take away for this chip is that it does not always stop on errors.
> It can also raise a dry IRQ (I don't have explanation, only happen on IMX8
> boards). The new handler properly check if the hardware is still running, which
> avoid programming over a live IP. The G2 has fancy internal clock gating, so
> doing so can lead to CPU hang if the register you set is hooked to a gated
> portion of the IP. At least this is my mental model of it.

Right. This patch seems to fix the crashes. The (possibly badly
transformed by GStreamer?) HEVC stream isn't displayed correctly yet,
though a direct HEVC 1080p RTP from camera is now ok (certain rate of
packet loss is possible here). Then:

> And there is an important bug fix for missing references, without that, you get
> thousands of error IRQs while the IP is fetch from address 0x0 and its
> surrounding. This large burst of IRQ, combine with multiple memory related
> erratas of IMX8 boards cause random other things to fall appart.
>
> https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/47825b1646a6a9eca0f90baa3d4f98947c2add96

After applying the above, the stream from GStreamer is fixed, too. It
seems it all works fine now.

From time to time I'm getting:
hantro_g2_irq: 0x80100 masked 0x0
hantro-vpu 38310000.video-codec: not all macroblocks were decoded.

But I guess it's due to my network setup.


Thank you very much for your help. You have saved me a lot of work -
a work with an uncertain outcome.
-- 
Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa

Re: i.MX8MP Hantro G2 HEVC decoding problems

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 2826 bytes --]

Hi Krzysztof,

Le mardi 04 novembre 2025 à 09:12 +0100, Krzysztof Hałasa a écrit :
> Hi Nicolas,
> 
> > In queue for the next kernel release, we have a rewrite of the IRQ handler:
> > 
> > https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/19c286b755072a22a063052f530a6b1fac8a1f63
> > 
> > The important take away for this chip is that it does not always stop on
> > errors.
> > It can also raise a dry IRQ (I don't have explanation, only happen on IMX8
> > boards). The new handler properly check if the hardware is still running,
> > which
> > avoid programming over a live IP. The G2 has fancy internal clock gating, so
> > doing so can lead to CPU hang if the register you set is hooked to a gated
> > portion of the IP. At least this is my mental model of it.
> 
> Right. This patch seems to fix the crashes. The (possibly badly
> transformed by GStreamer?) HEVC stream isn't displayed correctly yet,
> though a direct HEVC 1080p RTP from camera is now ok (certain rate of
> packet loss is possible here). Then:
> 
> > And there is an important bug fix for missing references, without that, you
> > get
> > thousands of error IRQs while the IP is fetch from address 0x0 and its
> > surrounding. This large burst of IRQ, combine with multiple memory related
> > erratas of IMX8 boards cause random other things to fall appart.
> > 
> > https://gitlab.freedesktop.org/linux-media/media-committers/-/commit/47825b1646a6a9eca0f90baa3d4f98947c2add96
> 
> After applying the above, the stream from GStreamer is fixed, too. It
> seems it all works fine now.
> 
> From time to time I'm getting:
> hantro_g2_irq: 0x80100 masked 0x0
> hantro-vpu 38310000.video-codec: not all macroblocks were decoded.
> 
> But I guess it's due to my network setup.
> 
> 
> Thank you very much for your help. You have saved me a lot of work -
> a work with an uncertain outcome.

Glad my recent fixes helped. These fixes took me several weeks to find and fix,
now be ware that the IRQ coding pattern needs to be extended to all other hantro
cores type, its in my todo, but not has urgent as it does not break as badly.

As for the remaining error, it matches my observation. Though, I strongly
believe its yet another hidden problem in the driver. More research needed, that
hard part is being able to isolate the bitstream that triggers it. I have very
little information about this specific error, most of the time, there is no
evident visual artifacts (so its not really broken for the users).

On another IMX8, the MQ, we often see the display controller fails at the same
time we get this error. Though, the MP have a bit less problems, I believe the
memory QoS is working and the that interconnect does a smoother job at load
balancing the bandwidth.

Nicolas

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: i.MX8MP Hantro G2 HEVC decoding problems

[Krzysztof Hałasa]

Hi Nicolas,

Nicolas Dufresne <nicolas@ndufresne.ca> writes:

>> From time to time I'm getting:
>> hantro_g2_irq: 0x80100 masked 0x0
>> hantro-vpu 38310000.video-codec: not all macroblocks were decoded.

> As for the remaining error, it matches my observation. Though, I strongly
> believe its yet another hidden problem in the driver. More research needed, that
> hard part is being able to isolate the bitstream that triggers it. I have very
> little information about this specific error, most of the time, there is no
> evident visual artifacts (so its not really broken for the users).

I see. I don't know if the above IRQ status comes with artifacts or not.
What I have is a printk() in the logs from time to time. I'll let you
know if I have a stream reliably triggering it.

Thanks again,
-- 
Krzysztof "Chris" Hałasa

Sieć Badawcza Łukasiewicz
Przemysłowy Instytut Automatyki i Pomiarów PIAP
Al. Jerozolimskie 202, 02-486 Warszawa