Re: USB transaction errors causing ERROR unknown event type 37

Re: USB transaction errors causing ERROR unknown event type 37

[Mathias Nyman]

On 27.2.2023 6.13, Fengyi WU wrote:
> Hi,
> 
> We have a board NXP i.MX8MPlus EVK. We are running Linux 5.4.24 from NXP on the SoC.
> 
> There is an analog to USB converter connected to the SoC via USB.
> The converter is this.
> https://www.linuxtv.org/wiki/index.php/The_Imaging_Source_DFG-USB2pro#Making_it_Work
> 
> We want to play video on the EVK and the result is that with 640*480 resolution, the video quality is very bad
> And the FPS is only 10.
> Before we applied the patch from Peter Chen, the 640*480 resolution is not playable.
> So the patch from Peter is usable, but not enough for us to make a good video quality.
> 
> The patch is this:
> https://patchwork.kernel.org/project/linux-usb/patch/1573836603-10871-2-git-send-email-mathias.nyman@linux.intel.com/
> 
> And we saw a lot of ERROR unknown event type 37 log on the console while the video is playing when it is not well working
> [  272.942827] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  272.999240] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.055696] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.112372] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.168736] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.225463] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.281947] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.338204] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.394560] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.451048] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.506856] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> [  273.563624] xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
> 
> We do tried the same USB converter device on Ubuntu kernel and everything is working fine, and the USB converter kernel driver between NXP kernel and ubuntu kernel is the same.
> So the problem we think is from the USB 3.0 xhci core driver.

Event type 37 is a "host controller event", usually seen when event ring is full.

The patch from Peter Chen also touches this area, making sure we update the event ring
dequeue pointer in hardware more often.

This error can happen if interrupts are disabled for a long time, or we have a lot of
TRBs on the event ring that don't generate interrupts.

There are some changes in this area since 5.4 kernel.
Is it possible to try a more recent kernel?

For example in kernel 5.5 we are giving back URBs in tasklet context, see:
36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context

Other things to try:
- set xhci->quirks |= XHCI_AVOID_BEI to make isoc transfers interrupt more often
- adjust the xhci interrupt moderation interval (IMODI)
- Add xhci dynamic debug and tracing to get more details of what is really going on


Thanks
Mathias

RE: USB transaction errors causing ERROR unknown event type 37

[Fengyi WU]

Mathias

We followed your suggestion and tried to add this patch to the 5.4 kernel,nothing has changed.
36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context

We also adjusted the xhci interrupt moderation interval (IMODI) from 40000 to 10000.
Nothing changed either.

And we are thinking to try the 5.5 kernel next, but not sure if this will make any difference or not.

Thanks.


Internal

-----Original Message-----
From: Fengyi WU
Sent: Tuesday, February 28, 2023 11:08 AM
To: Mathias Nyman <mathias.nyman@intel.com>; peter.chen@nxp.com
Cc: linux-usb@vger.kernel.org; Peter Chen <peter.chen@kernel.org>; mathias.nyman@linux.intel.com
Subject: RE: USB transaction errors causing ERROR unknown event type 37

Mathias

Thanks for the kind reply.
We do tried the ubuntu kernel 5.4, it was working without any problem as I mentioned. 
Let me give you more info, so we can debug and analysis together.
I did a trace using the command below, can you check the atttached trace file?
echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable

I do found Event Ring Full Error inside the trace file. 

Thanks.
Jeff


Internal

-----Original Message-----
From: Mathias Nyman <mathias.nyman@intel.com>
Sent: Monday, February 27, 2023 6:38 PM
To: Fengyi WU <fengyi.wu@se.com>; peter.chen@nxp.com
Cc: linux-usb@vger.kernel.org; Peter Chen <peter.chen@kernel.org>; mathias.nyman@linux.intel.com
Subject: Re: USB transaction errors causing ERROR unknown event type 37

[External email: Use caution with links and attachments]

________________________________



On 27.2.2023 6.13, Fengyi WU wrote:
> Hi,
>
> We have a board NXP i.MX8MPlus EVK. We are running Linux 5.4.24 from NXP on the SoC.
>
> There is an analog to USB converter connected to the SoC via USB.
> The converter is this.
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.
> linuxtv.org%2Fwiki%2Findex.php%2FThe_Imaging_Source_DFG-USB2pro%23Maki
> ng_it_Work&data=05%7C01%7Cfengyi.wu%40se.com%7Cb7f04f2f54d749f2c19a08d
> b18a62dc4%7C6e51e1adc54b4b39b5980ffe9ae68fef%7C0%7C0%7C638130874232170
> 496%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTi
> I6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=wrZSuQ9hBAMNkCblWrTQYV2
> BemA12DNr5Q6rbzBUviw%3D&reserved=0
>
> We want to play video on the EVK and the result is that with 640*480 
> resolution, the video quality is very bad And the FPS is only 10.
> Before we applied the patch from Peter Chen, the 640*480 resolution is not playable.
> So the patch from Peter is usable, but not enough for us to make a good video quality.
>
> The patch is this:
> https://eur02.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatc
> hwork.kernel.org%2Fproject%2Flinux-usb%2Fpatch%2F1573836603-10871-2-gi
> t-send-email-mathias.nyman%40linux.intel.com%2F&data=05%7C01%7Cfengyi.
> wu%40se.com%7Cb7f04f2f54d749f2c19a08db18a62dc4%7C6e51e1adc54b4b39b5980
> ffe9ae68fef%7C0%7C0%7C638130874232170496%7CUnknown%7CTWFpbGZsb3d8eyJWI
> joiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7
> C%7C%7C&sdata=qNm3Yzs3bGkbvwKggC4FxcbSXpzi9NOupKs5Qe81uuw%3D&reserved=
> 0
>
> And we saw a lot of ERROR unknown event type 37 log on the console 
> while the video is playing when it is not well working [  272.942827] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  272.999240] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.055696] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.112372] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.168736] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.225463] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.281947] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.338204] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.394560] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.451048] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.506856] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37 [  273.563624] 
> xhci-hcd xhci-hcd.1.auto: ERROR unknown event type 37
>
> We do tried the same USB converter device on Ubuntu kernel and everything is working fine, and the USB converter kernel driver between NXP kernel and ubuntu kernel is the same.
> So the problem we think is from the USB 3.0 xhci core driver.

Event type 37 is a "host controller event", usually seen when event ring is full.

The patch from Peter Chen also touches this area, making sure we update the event ring dequeue pointer in hardware more often.

This error can happen if interrupts are disabled for a long time, or we have a lot of TRBs on the event ring that don't generate interrupts.

There are some changes in this area since 5.4 kernel.
Is it possible to try a more recent kernel?

For example in kernel 5.5 we are giving back URBs in tasklet context, see:
36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context

Other things to try:
- set xhci->quirks |= XHCI_AVOID_BEI to make isoc transfers interrupt more often
- adjust the xhci interrupt moderation interval (IMODI)
- Add xhci dynamic debug and tracing to get more details of what is really going on


Thanks
Mathias

Re: USB transaction errors causing ERROR unknown event type 37

[Mathias Nyman]

Hi

(squashing and modifying the two top posted emails a bit to make this more readable)

On 28.2.2023 6.37, Fengyi WU wrote:

> We followed your suggestion and tried to add this patch to the 5.4 kernel,nothing has changed.
> 36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context
> 
> We also adjusted the xhci interrupt moderation interval (IMODI) from 40000 to 10000.
> Nothing changed either.
> 
> And we are thinking to try the 5.5 kernel next, but not sure if this will make any difference or not.
> 

I'd recommend trying a more recent upstream 6.1 or 6.2 kernel

> We do tried the ubuntu kernel 5.4, it was working without any problem as I mentioned.

If the problem is only visible with a custom NXP kernel then there isn't much I can do.

> Let me give you more info, so we can debug and analysis together.
> I did a trace using the command below, can you check the atttached trace file?
> echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
> echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
> 
> I do found Event Ring Full Error inside the trace file.

Trace shows there are a lot of isoc TRBs with BEI (Block Event Interrupt) flag queued.
These will fill the event ring with transfer events without issuing interrupts.

Did setting the XHCI_AVOID_BEI flag for this host help?

An more reasonable interrupt rate can be achieved by setting both XHCI_AVOID_BEI
flag, and adding:
"edc649a82341 xhci: Tune interrupt blocking for isochronous transfers"
from 5.10 kernel.

This in addition to "36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context"
and the patch by Peter Chen mentioned earlier

Thanks
Mathias

RE: USB transaction errors causing ERROR unknown event type 37

[Fengyi WU]

Hi,Mathias

We do tried the kernel 5.15, and added some latest patches on USB driver as you suggested.it is becoming worse.
Not sure what the problem is and we are trying to contact with NXP

Thanks.


Internal

-----Original Message-----
From: Mathias Nyman <mathias.nyman@linux.intel.com> 
Sent: Tuesday, February 28, 2023 8:57 PM
To: Fengyi WU <fengyi.wu@se.com>; Mathias Nyman <mathias.nyman@intel.com>; peter.chen@nxp.com
Cc: linux-usb@vger.kernel.org; Peter Chen <peter.chen@kernel.org>
Subject: Re: USB transaction errors causing ERROR unknown event type 37

[External email: Use caution with links and attachments]

________________________________



Hi

(squashing and modifying the two top posted emails a bit to make this more readable)

On 28.2.2023 6.37, Fengyi WU wrote:

> We followed your suggestion and tried to add this patch to the 5.4 kernel,nothing has changed.
> 36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet 
> context
>
> We also adjusted the xhci interrupt moderation interval (IMODI) from 40000 to 10000.
> Nothing changed either.
>
> And we are thinking to try the 5.5 kernel next, but not sure if this will make any difference or not.
>

I'd recommend trying a more recent upstream 6.1 or 6.2 kernel

> We do tried the ubuntu kernel 5.4, it was working without any problem as I mentioned.

If the problem is only visible with a custom NXP kernel then there isn't much I can do.

> Let me give you more info, so we can debug and analysis together.
> I did a trace using the command below, can you check the atttached trace file?
> echo 81920 > /sys/kernel/debug/tracing/buffer_size_kb
> echo 1 > /sys/kernel/debug/tracing/events/xhci-hcd/enable
>
> I do found Event Ring Full Error inside the trace file.

Trace shows there are a lot of isoc TRBs with BEI (Block Event Interrupt) flag queued.
These will fill the event ring with transfer events without issuing interrupts.

Did setting the XHCI_AVOID_BEI flag for this host help?

An more reasonable interrupt rate can be achieved by setting both XHCI_AVOID_BEI flag, and adding:
"edc649a82341 xhci: Tune interrupt blocking for isochronous transfers"
from 5.10 kernel.

This in addition to "36dc01657b49 usb: host: xhci: Support running urb giveback in tasklet context"
and the patch by Peter Chen mentioned earlier

Thanks
Mathias