Linux Crash with Gadget serial driver

Linux Crash with Gadget serial driver

[GARG Shivam]

Hi All,

My Device is a Linux machine that is having a Gadget Serial driver. Physically device side we have USB-C port while at host side we have USB port. This device is used as COM port to access the serial port.

My Device side Linux is getting panic as soon as I attached the USB to FreeBSD. Any view on this?

If I stopped the enumeration at FreeBSD using command (sysctl hw.usb.disable_enumeration=1), so device side Linux is stopped crashing.

Can anyone please help?

Thanks
Shivam

Re: Linux Crash with Gadget serial driver

[Peter Chen]

On Wed, Aug 26, 2020 at 1:36 AM GARG Shivam <shivam.garg@thalesgroup.com> wrote:
>
> Hi All,
>
> My Device is a Linux machine that is having a Gadget Serial driver. Physically device side we have USB-C port while at host side we have USB port. This device is used as COM port to access the serial port.
>
> My Device side Linux is getting panic as soon as I attached the USB to FreeBSD. Any view on this?
>

You may post your panic log and kernel version info as well.

Peter

RE: Linux Crash with Gadget serial driver

[GARG Shivam]

I am with Kernel version 4.1.

Unable to get the panic log (Backtrace).

Thanks
Shivam


-----Original Message-----
From: Peter Chen <hzpeterchen@gmail.com> 
Sent: Wednesday, August 26, 2020 7:26 AM
To: GARG Shivam <shivam.garg@thalesgroup.com>
Cc: Linux-usb@vger.kernel.org
Subject: Re: Linux Crash with Gadget serial driver

On Wed, Aug 26, 2020 at 1:36 AM GARG Shivam <shivam.garg@thalesgroup.com> wrote:
>
> Hi All,
>
> My Device is a Linux machine that is having a Gadget Serial driver. Physically device side we have USB-C port while at host side we have USB port. This device is used as COM port to access the serial port.
>
> My Device side Linux is getting panic as soon as I attached the USB to FreeBSD. Any view on this?
>

You may post your panic log and kernel version info as well.

Peter

Re: Linux Crash with Gadget serial driver

[Greg KH]

On Wed, Aug 26, 2020 at 09:09:32AM +0000, GARG Shivam wrote:
> I am with Kernel version 4.1.

4.1 is over 6 years old and over 500 thousand changes and fixes behind
what we are all working with now.

Please work with your vendor who is forcing you to remain on an obsolete
kernel version like this, as you are paying them for that type of
support, there is nothing that we can do here with this, sorry.

good luck!

greg k-h

RE: Linux Crash with Gadget serial driver

[GARG Shivam]

Hi All,

I agree it is an old kernel but still can you look into the backtrace and guide me how I can proceed to stop the host crash.

Below is the backtrace info that is coming as soon as I connect my gadget serial driver supported device(USB) to FreeBSD.

Backtrace #1

root@(none):~# INFO: rcu_sched detected stalls on CPUs/tasks:
6: (39 GPs behind) idle=83d/1/0 softirq=0/0 fqs=5228
(detected by 4, t=5254 jiffies, g=8752, c=8751, q=75869)
Task dump for CPU 6:
swapper/6 R running task 0 0 1 0x00000804
Call Trace:
[c0000001f01379f0] [c000000000005e18] .do_IRQ+0x84/0x118 (unreliable)
[c0000001f0137a80] [c00000000001793c] exc_0x500_common+0xfc/0x100
- interrupt: 501 at .book3e_idle+0x24/0x4c
LR = .book3e_idle+0x24/0x4c
[c0000001f0137d70] [c0000000000094a8] .arch_cpu_idle+0x34/0xa0 (unreliable)
[c0000001f0137de0] [c000000000086bd0] .cpu_startup_entry+0x204/0x2ac
[c0000001f0137ee0] [c00000000001edf8] .start_secondary+0x2b0/0x2d4
[c0000001f0137f90] [c00000000000046c] start_secondary_prolog+0x10/0x14

This is also seen sometimes:

INFO: task udhcpc:3285 blocked for more than 120 seconds.
Tainted: G O 4.1.8-rt8+gbd51baffc04e #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
udhcpc D 00003fff87215f78 0 3285 3245 0x00000000
Call Trace:
[c0000001c42ef710] [c000000000008878] .__switch_to+0x8c/0xc4
[c0000001c42ef790] [c0000000009604cc] .__schedule+0x230/0x53c
[c0000001c42ef840] [c00000000096081c] .schedule+0x44/0xe4
[c0000001c42ef8c0] [c000000000963de4] .schedule_timeout+0x194/0x210
[c0000001c42ef9a0] [c0000000009616c4] .wait_for_common+0x170/0x1d0
[c0000001c42efa60] [c00000000009d4ec] .wait_rcu_gp+0x5c/0x74
[c0000001c42efb10] [c00000000009ee7c] .synchronize_sched+0x4c/0x70
[c0000001c42efb80] [c0000000007df974] .synchronize_net+0x20/0x4c
[c0000001c42efbf0] [c000000000906874] .__unregister_prot_hook+0x124/0x1c8
[c0000001c42efc80] [c000000000907014] .packet_do_bind+0x184/0x1e4
[c0000001c42efd10] [c0000000007c2d8c] .SyS_bind+0xf4/0x110
[c0000001c42efe30] [c000000000000698] system_call+0x38/0xc4


Thanks
Shivam



-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org> 
Sent: Wednesday, August 26, 2020 3:06 PM
To: GARG Shivam <shivam.garg@thalesgroup.com>
Cc: Peter Chen <hzpeterchen@gmail.com>; Linux-usb@vger.kernel.org
Subject: Re: Linux Crash with Gadget serial driver

On Wed, Aug 26, 2020 at 09:09:32AM +0000, GARG Shivam wrote:
> I am with Kernel version 4.1.

4.1 is over 6 years old and over 500 thousand changes and fixes behind what we are all working with now.

Please work with your vendor who is forcing you to remain on an obsolete kernel version like this, as you are paying them for that type of support, there is nothing that we can do here with this, sorry.

good luck!

greg k-h

RE: Linux Crash with Gadget serial driver

[GARG Shivam]

Hi All,

I think backtrace info get messed up due to email configuration. Let me post again.

root@(none):~# INFO: rcu_sched detected stalls on CPUs/tasks:
6: (39 GPs behind) idle=83d/1/0 softirq=0/0 fqs=5228
(detected by 4, t=5254 jiffies, g=8752, c=8751, q=75869)
Task dump for CPU 6:
swapper/6 R running task 0 0 1 0x00000804
Call Trace:
[c0000001f01379f0] [c000000000005e18] .do_IRQ+0x84/0x118 (unreliable)
[c0000001f0137a80] [c00000000001793c] exc_0x500_common+0xfc/0x100
- interrupt: 501 at .book3e_idle+0x24/0x4c
LR = .book3e_idle+0x24/0x4c
[c0000001f0137d70] [c0000000000094a8] .arch_cpu_idle+0x34/0xa0 (unreliable)
[c0000001f0137de0] [c000000000086bd0] .cpu_startup_entry+0x204/0x2ac
[c0000001f0137ee0] [c00000000001edf8] .start_secondary+0x2b0/0x2d4
[c0000001f0137f90] [c00000000000046c] start_secondary_prolog+0x10/0x14


This is also seen sometimes:

INFO: task udhcpc:3285 blocked for more than 120 seconds.
Tainted: G O 4.1.8-rt8+gbd51baffc04e #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
udhcpc D 00003fff87215f78 0 3285 3245 0x00000000
Call Trace:
[c0000001c42ef710] [c000000000008878] .__switch_to+0x8c/0xc4
[c0000001c42ef790] [c0000000009604cc] .__schedule+0x230/0x53c
[c0000001c42ef840] [c00000000096081c] .schedule+0x44/0xe4
[c0000001c42ef8c0] [c000000000963de4] .schedule_timeout+0x194/0x210
[c0000001c42ef9a0] [c0000000009616c4] .wait_for_common+0x170/0x1d0
[c0000001c42efa60] [c00000000009d4ec] .wait_rcu_gp+0x5c/0x74
[c0000001c42efb10] [c00000000009ee7c] .synchronize_sched+0x4c/0x70
[c0000001c42efb80] [c0000000007df974] .synchronize_net+0x20/0x4c
[c0000001c42efbf0] [c000000000906874] .__unregister_prot_hook+0x124/0x1c8
[c0000001c42efc80] [c000000000907014] .packet_do_bind+0x184/0x1e4
[c0000001c42efd10] [c0000000007c2d8c] .SyS_bind+0xf4/0x110
[c0000001c42efe30] [c000000000000698] system_call+0x38/0xc4

Thanks
Shivam
-----Original Message-----
From: GARG Shivam 
Sent: Wednesday, September 23, 2020 12:07 PM
To: Linux-usb@vger.kernel.org
Cc: Peter Chen <hzpeterchen@gmail.com>; Greg KH <gregkh@linuxfoundation.org>
Subject: RE: Linux Crash with Gadget serial driver

Hi All,

I agree it is an old kernel but still can you look into the backtrace and guide me how I can proceed to stop the host crash.

Below is the backtrace info that is coming as soon as I connect my gadget serial driver supported device(USB) to FreeBSD.

Backtrace #1

root@(none):~# INFO: rcu_sched detected stalls on CPUs/tasks:
6: (39 GPs behind) idle=83d/1/0 softirq=0/0 fqs=5228 (detected by 4, t=5254 jiffies, g=8752, c=8751, q=75869) Task dump for CPU 6:
swapper/6 R running task 0 0 1 0x00000804 Call Trace:
[c0000001f01379f0] [c000000000005e18] .do_IRQ+0x84/0x118 (unreliable) [c0000001f0137a80] [c00000000001793c] exc_0x500_common+0xfc/0x100 - interrupt: 501 at .book3e_idle+0x24/0x4c LR = .book3e_idle+0x24/0x4c [c0000001f0137d70] [c0000000000094a8] .arch_cpu_idle+0x34/0xa0 (unreliable) [c0000001f0137de0] [c000000000086bd0] .cpu_startup_entry+0x204/0x2ac [c0000001f0137ee0] [c00000000001edf8] .start_secondary+0x2b0/0x2d4 [c0000001f0137f90] [c00000000000046c] start_secondary_prolog+0x10/0x14

This is also seen sometimes:

INFO: task udhcpc:3285 blocked for more than 120 seconds.
Tainted: G O 4.1.8-rt8+gbd51baffc04e #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
udhcpc D 00003fff87215f78 0 3285 3245 0x00000000 Call Trace:
[c0000001c42ef710] [c000000000008878] .__switch_to+0x8c/0xc4 [c0000001c42ef790] [c0000000009604cc] .__schedule+0x230/0x53c [c0000001c42ef840] [c00000000096081c] .schedule+0x44/0xe4 [c0000001c42ef8c0] [c000000000963de4] .schedule_timeout+0x194/0x210 [c0000001c42ef9a0] [c0000000009616c4] .wait_for_common+0x170/0x1d0 [c0000001c42efa60] [c00000000009d4ec] .wait_rcu_gp+0x5c/0x74 [c0000001c42efb10] [c00000000009ee7c] .synchronize_sched+0x4c/0x70 [c0000001c42efb80] [c0000000007df974] .synchronize_net+0x20/0x4c [c0000001c42efbf0] [c000000000906874] .__unregister_prot_hook+0x124/0x1c8
[c0000001c42efc80] [c000000000907014] .packet_do_bind+0x184/0x1e4 [c0000001c42efd10] [c0000000007c2d8c] .SyS_bind+0xf4/0x110 [c0000001c42efe30] [c000000000000698] system_call+0x38/0xc4


Thanks
Shivam



-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org>
Sent: Wednesday, August 26, 2020 3:06 PM
To: GARG Shivam <shivam.garg@thalesgroup.com>
Cc: Peter Chen <hzpeterchen@gmail.com>; Linux-usb@vger.kernel.org
Subject: Re: Linux Crash with Gadget serial driver

On Wed, Aug 26, 2020 at 09:09:32AM +0000, GARG Shivam wrote:
> I am with Kernel version 4.1.

4.1 is over 6 years old and over 500 thousand changes and fixes behind what we are all working with now.

Please work with your vendor who is forcing you to remain on an obsolete kernel version like this, as you are paying them for that type of support, there is nothing that we can do here with this, sorry.

good luck!

greg k-h

Re: Linux Crash with Gadget serial driver

[Greg KH]

On Wed, Sep 23, 2020 at 06:36:40AM +0000, GARG Shivam wrote:
> Hi All,
> 
> I agree it is an old kernel but still can you look into the backtrace and guide me how I can proceed to stop the host crash.

You are on your own here, sorry.  Please work with the company that you
are paying to give you support for such an old and obsolete and insecure
kernel tree.  You don't want that money to go to waste, right?  :)

good luck!

greg k-h

RE: Linux Crash with Gadget serial driver

[GARG Shivam]

Actually, we are using USB-C port at my host but vendor does not support USB-C. It is a customization that we did at our end in the hardware.

That is the reason, we are stuck a between Yacto and freebsd. If just, want to stop the kernel crash.

Thanks
Shivam

-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org> 
Sent: Wednesday, September 23, 2020 12:14 PM
To: GARG Shivam <shivam.garg@thalesgroup.com>
Cc: Linux-usb@vger.kernel.org; Peter Chen <hzpeterchen@gmail.com>
Subject: Re: Linux Crash with Gadget serial driver

On Wed, Sep 23, 2020 at 06:36:40AM +0000, GARG Shivam wrote:
> Hi All,
> 
> I agree it is an old kernel but still can you look into the backtrace and guide me how I can proceed to stop the host crash.

You are on your own here, sorry.  Please work with the company that you are paying to give you support for such an old and obsolete and insecure kernel tree.  You don't want that money to go to waste, right?  :)

good luck!

greg k-h

Re: Linux Crash with Gadget serial driver

[Greg KH]

On Wed, Sep 23, 2020 at 06:51:52AM +0000, GARG Shivam wrote:
> Actually, we are using USB-C port at my host but vendor does not support USB-C. It is a customization that we did at our end in the hardware.
> 
> That is the reason, we are stuck a between Yacto and freebsd. If just, want to stop the kernel crash.

I understand the problem, but you are working with a heavily modified
and patched version of a kernel that was released over 5 years and
430277 changes ago.  An almost impossible task to debug for us, don't
you think?

good luck!

greg k-h

RE: Linux Crash with Gadget serial driver

[GARG Shivam]

Hi Greg,

I agree and respect your decision this is the reason I start debugging it month back and found few areas that can be useful for any expert.

It will be a great help if community giver some hint how to proceed from here.

My finding is : 

1. FreeBSD machine is doing the continuous enumeration to USB device and due to this host side Linux is crashing. In case we stop the enumeration (command is sysctl hw.usb.disable_enumeration=1) from FreeBSD side so host side linux stop crashing.
2. In case we modify the usb_gadget_map_request function by adding it some of printk(around 25) so host side Linux Stop crashing.
3. In case we add printk at entry and exit of fsl_udc_core.c functions .host side hsm again stop crashing.

Any hint will make my day.

Thanks
Shivam



-----Original Message-----
From: Greg KH <gregkh@linuxfoundation.org> 
Sent: Wednesday, September 23, 2020 12:34 PM
To: GARG Shivam <shivam.garg@thalesgroup.com>
Cc: Linux-usb@vger.kernel.org; Peter Chen <hzpeterchen@gmail.com>
Subject: Re: Linux Crash with Gadget serial driver

On Wed, Sep 23, 2020 at 06:51:52AM +0000, GARG Shivam wrote:
> Actually, we are using USB-C port at my host but vendor does not support USB-C. It is a customization that we did at our end in the hardware.
> 
> That is the reason, we are stuck a between Yacto and freebsd. If just, want to stop the kernel crash.

I understand the problem, but you are working with a heavily modified and patched version of a kernel that was released over 5 years and
430277 changes ago.  An almost impossible task to debug for us, don't you think?

good luck!

greg k-h