dwc3 gadget and USB3

dwc3 gadget and USB3

[Joakim Tjernlund]

we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
use USB0 in gadget mode running either NCM or RNDIS ethernet on top.

When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
networking(NCM or RNDIS) works well.

However, when we connect the gadget directly to the PC/laptop which uses USB3
we see something odd:
  Ping from PC to gadget works.
  Ping from gadget to laptop does not. However if we also ping from PC at the same time we
  see gadget to PC start working.
Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.

Any idea what might be wrong or how to debug this?
Kernel 5.15.87

 Jocke 

Re: dwc3 gadget and USB3

[Greg KH]

On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> 
> When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> networking(NCM or RNDIS) works well.
> 
> However, when we connect the gadget directly to the PC/laptop which uses USB3
> we see something odd:
>   Ping from PC to gadget works.
>   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
>   see gadget to PC start working.
> Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> 
> Any idea what might be wrong or how to debug this?
> Kernel 5.15.87

5.15.y is very old, does this also happen on 6.2?

thanks,

greg k-h

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > 
> > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > networking(NCM or RNDIS) works well.
> > 
> > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > we see something odd:
> >   Ping from PC to gadget works.
> >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> >   see gadget to PC start working.
> > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > 
> > Any idea what might be wrong or how to debug this?
> > Kernel 5.15.87
> 
> 5.15.y is very old, does this also happen on 6.2?
> 

I just tried 6.1.15 and the problem remains, I hope that is close enough ?

Re: dwc3 gadget and USB3

[gregkh]

On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > 
> > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > networking(NCM or RNDIS) works well.
> > > 
> > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > we see something odd:
> > >   Ping from PC to gadget works.
> > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > >   see gadget to PC start working.
> > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > 
> > > Any idea what might be wrong or how to debug this?
> > > Kernel 5.15.87
> > 
> > 5.15.y is very old, does this also happen on 6.2?
> > 
> 
> I just tried 6.1.15 and the problem remains, I hope that is close enough ?

It's good enough :)

Have any logs at all that show any problems?  Also, you might want to
 cc:  the dwc3 maintainer...

 thanks,

 greg k-hj

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > 
> > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > networking(NCM or RNDIS) works well.
> > > > 
> > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > we see something odd:
> > > >   Ping from PC to gadget works.
> > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > >   see gadget to PC start working.
> > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > 
> > > > Any idea what might be wrong or how to debug this?
> > > > Kernel 5.15.87
> > > 
> > > 5.15.y is very old, does this also happen on 6.2?
> > > 
> > 
> > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> 
> It's good enough :)
> 
> Have any logs at all that show any problems?
> 
No, don't know where to start. There are no errors logged.

>   Also, you might want to
>  cc:  the dwc3 maintainer...

I thought I did but that look like old info, added Thinh Nguyen now, thanks

 Jocke

> 
>  thanks,
> 
>  greg k-hj

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > 
> > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > networking(NCM or RNDIS) works well.
> > > > > 
> > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > we see something odd:
> > > > >   Ping from PC to gadget works.
> > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > >   see gadget to PC start working.
> > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > 
> > > > > Any idea what might be wrong or how to debug this?
> > > > > Kernel 5.15.87
> > > > 
> > > > 5.15.y is very old, does this also happen on 6.2?
> > > > 
> > > 
> > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > 
> > It's good enough :)
> > 
> > Have any logs at all that show any problems?
> > 
> No, don't know where to start. There are no errors logged.
> 
> >   Also, you might want to
> >  cc:  the dwc3 maintainer...
> 
> I thought I did but that look like old info, added Thinh Nguyen now, thanks
> 
>  Jocke
> 
> > 
> >  thanks,
> > 
> >  greg k-hj
> 

Found and USBC Dock and connected that between gadget an PC and this also works well.
Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.

Found out something interesting, on PC:
cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
echo 0 > usb2_hardware_lpm

Now ping works normally.

So LPM does not seem to work properly on gadget. Can I disable LPM somehow
on gadget side? 

 Jocke

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > 
> > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > networking(NCM or RNDIS) works well.
> > > > > > 
> > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > we see something odd:
> > > > > >   Ping from PC to gadget works.
> > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > >   see gadget to PC start working.
> > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > 
> > > > > > Any idea what might be wrong or how to debug this?
> > > > > > Kernel 5.15.87
> > > > > 
> > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > 
> > > > 
> > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > 
> > > It's good enough :)
> > > 
> > > Have any logs at all that show any problems?
> > > 
> > No, don't know where to start. There are no errors logged.
> > 
> > >   Also, you might want to
> > >  cc:  the dwc3 maintainer...
> > 
> > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > 
> >  Jocke
> > 
> > > 
> > >  thanks,
> > > 
> > >  greg k-hj
> > 
> 
> Found and USBC Dock and connected that between gadget an PC and this also works well.
> Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> 
> Found out something interesting, on PC:
> cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> echo 0 > usb2_hardware_lpm
> 
> Now ping works normally.
> 
> So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> on gadget side? 
> 
>  Jocke

Found some DTS quirks to disable LPM, work fine :)
One observation:

ping over NCM to Linux PC:
PING 169.254.100.102 (169.254.100.102): 56 data bytes
64 bytes from 169.254.100.102: seq=0 ttl=64 time=2.166 ms
64 bytes from 169.254.100.102: seq=1 ttl=64 time=2.168 ms
64 bytes from 169.254.100.102: seq=2 ttl=64 time=2.333 ms

ping over NCM to Windows 10 PC:
PING 169.254.100.102 (169.254.100.102): 56 data bytes
64 bytes from 169.254.100.102: seq=0 ttl=128 time=0.921 ms
64 bytes from 169.254.100.102: seq=1 ttl=128 time=0.963 ms
64 bytes from 169.254.100.102: seq=2 ttl=128 time=1.143 ms
64 bytes from 169.254.100.102: seq=3 ttl=128 time=1.161 ms

NCM to Windows appears to have much lower latency.

 Jocke

Re: dwc3 gadget and USB3

[Thinh Nguyen]

On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > 
> > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > 
> > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > we see something odd:
> > > > > > >   Ping from PC to gadget works.
> > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > >   see gadget to PC start working.
> > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > 
> > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > Kernel 5.15.87
> > > > > > 
> > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > 
> > > > > 
> > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > 
> > > > It's good enough :)
> > > > 
> > > > Have any logs at all that show any problems?
> > > > 
> > > No, don't know where to start. There are no errors logged.
> > > 
> > > >   Also, you might want to
> > > >  cc:  the dwc3 maintainer...
> > > 
> > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > 
> > >  Jocke
> > > 
> > > > 
> > > >  thanks,
> > > > 
> > > >  greg k-hj
> > > 
> > 
> > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > 
> > Found out something interesting, on PC:
> > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > echo 0 > usb2_hardware_lpm
> > 
> > Now ping works normally.
> > 
> > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > on gadget side? 
> > 

There's no option in gadget configfs to allow you to do that at the
moment. You can disable LPM in dwc3 controller in the devicetree with
"snps,dis_enblslpm_quirk" instead.

If the host puts the gadget in suspend, the gadget won't be able to
communicate with the host until the host wakes the gadget up and starts
talking to the gadget again. The gadget may be able to signal the host
to wakeup via remote wakeup. Did you check if the device is in suspend?
If it's in suspend, is the gadget enabled with remote wakeup? Did the
NCM driver sent a remote wakeup signal to the host? I didn't verify, but
I suspect the NCM gadget driver isn't configured/implemented with remote
wakeup.

You can work around this by disabling LPM, which removes any power
saving as you've tested.

BR,
Thinh

> >  Jocke
> 
> Found some DTS quirks to disable LPM, work fine :)
> One observation:
> 
> ping over NCM to Linux PC:
> PING 169.254.100.102 (169.254.100.102): 56 data bytes
> 64 bytes from 169.254.100.102: seq=0 ttl=64 time=2.166 ms
> 64 bytes from 169.254.100.102: seq=1 ttl=64 time=2.168 ms
> 64 bytes from 169.254.100.102: seq=2 ttl=64 time=2.333 ms
> 
> ping over NCM to Windows 10 PC:
> PING 169.254.100.102 (169.254.100.102): 56 data bytes
> 64 bytes from 169.254.100.102: seq=0 ttl=128 time=0.921 ms
> 64 bytes from 169.254.100.102: seq=1 ttl=128 time=0.963 ms
> 64 bytes from 169.254.100.102: seq=2 ttl=128 time=1.143 ms
> 64 bytes from 169.254.100.102: seq=3 ttl=128 time=1.161 ms
> 
> NCM to Windows appears to have much lower latency.
> 

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > > 
> > > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > > 
> > > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > > we see something odd:
> > > > > > > >   Ping from PC to gadget works.
> > > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > > >   see gadget to PC start working.
> > > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > > 
> > > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > > Kernel 5.15.87
> > > > > > > 
> > > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > > 
> > > > > > 
> > > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > > 
> > > > > It's good enough :)
> > > > > 
> > > > > Have any logs at all that show any problems?
> > > > > 
> > > > No, don't know where to start. There are no errors logged.
> > > > 
> > > > >   Also, you might want to
> > > > >  cc:  the dwc3 maintainer...
> > > > 
> > > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > > 
> > > >  Jocke
> > > > 
> > > > > 
> > > > >  thanks,
> > > > > 
> > > > >  greg k-hj
> > > > 
> > > 
> > > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > > 
> > > Found out something interesting, on PC:
> > > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > > echo 0 > usb2_hardware_lpm
> > > 
> > > Now ping works normally.
> > > 
> > > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > > on gadget side? 
> > > 
> 
> There's no option in gadget configfs to allow you to do that at the
> moment. You can disable LPM in dwc3 controller in the devicetree with
> "snps,dis_enblslpm_quirk" instead.

Yes, I found that. Thanks.

> 
> If the host puts the gadget in suspend, the gadget won't be able to
> communicate with the host until the host wakes the gadget up and starts
> talking to the gadget again. The gadget may be able to signal the host
> to wakeup via remote wakeup. Did you check if the device is in suspend?
> If it's in suspend, is the gadget enabled with remote wakeup? Did the
> NCM driver sent a remote wakeup signal to the host? I didn't verify, but
> I suspect the NCM gadget driver isn't configured/implemented with remote
> wakeup.

Then maybe NCM/RNDIS should inform/disable LPM in the device driver?
One cannot have half an impl. of this feature.

> 
> You can work around this by disabling LPM, which removes any power
> saving as you've tested.

Yes, we don't require LPM so this will work for us.

> 
> BR,
> Thinh
> 
> > >  Jocke
> > 
> > Found some DTS quirks to disable LPM, work fine :)
> > One observation:
> > 
> > ping over NCM to Linux PC:
> > PING 169.254.100.102 (169.254.100.102): 56 data bytes
> > 64 bytes from 169.254.100.102: seq=0 ttl=64 time=2.166 ms
> > 64 bytes from 169.254.100.102: seq=1 ttl=64 time=2.168 ms
> > 64 bytes from 169.254.100.102: seq=2 ttl=64 time=2.333 ms
> > 
> > ping over NCM to Windows 10 PC:
> > PING 169.254.100.102 (169.254.100.102): 56 data bytes
> > 64 bytes from 169.254.100.102: seq=0 ttl=128 time=0.921 ms
> > 64 bytes from 169.254.100.102: seq=1 ttl=128 time=0.963 ms
> > 64 bytes from 169.254.100.102: seq=2 ttl=128 time=1.143 ms
> > 64 bytes from 169.254.100.102: seq=3 ttl=128 time=1.161 ms
> > 
> > NCM to Windows appears to have much lower latency.
> > 

Re: dwc3 gadget and USB3

[Thinh Nguyen]

On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > > > 
> > > > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > > > 
> > > > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > > > we see something odd:
> > > > > > > > >   Ping from PC to gadget works.
> > > > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > > > >   see gadget to PC start working.
> > > > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > > > 
> > > > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > > > Kernel 5.15.87
> > > > > > > > 
> > > > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > > > 
> > > > > > > 
> > > > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > > > 
> > > > > > It's good enough :)
> > > > > > 
> > > > > > Have any logs at all that show any problems?
> > > > > > 
> > > > > No, don't know where to start. There are no errors logged.
> > > > > 
> > > > > >   Also, you might want to
> > > > > >  cc:  the dwc3 maintainer...
> > > > > 
> > > > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > > > 
> > > > >  Jocke
> > > > > 
> > > > > > 
> > > > > >  thanks,
> > > > > > 
> > > > > >  greg k-hj
> > > > > 
> > > > 
> > > > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > > > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > > > 
> > > > Found out something interesting, on PC:
> > > > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > > > echo 0 > usb2_hardware_lpm
> > > > 
> > > > Now ping works normally.
> > > > 
> > > > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > > > on gadget side? 
> > > > 
> > 
> > There's no option in gadget configfs to allow you to do that at the
> > moment. You can disable LPM in dwc3 controller in the devicetree with
> > "snps,dis_enblslpm_quirk" instead.
> 
> Yes, I found that. Thanks.

Also note that LPM is for device operating in usb2 speed. If it's usb3
speeds, then you need to disable U3 suspend.

> 
> > 
> > If the host puts the gadget in suspend, the gadget won't be able to
> > communicate with the host until the host wakes the gadget up and starts
> > talking to the gadget again. The gadget may be able to signal the host
> > to wakeup via remote wakeup. Did you check if the device is in suspend?
> > If it's in suspend, is the gadget enabled with remote wakeup? Did the
> > NCM driver sent a remote wakeup signal to the host? I didn't verify, but
> > I suspect the NCM gadget driver isn't configured/implemented with remote
> > wakeup.
> 
> Then maybe NCM/RNDIS should inform/disable LPM in the device driver?
> One cannot have half an impl. of this feature.

Most drivers don't fully support every feature. They are implemented as
needed. I'm not familiar with NCM/RNDIS driver. Perhaps you can ping its
maintainer to see its current status to see if it needs to be updated.
Unless there's a quirk from the host/device, we shouldn't have to
disable LPM.

BR,
Thinh

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Thu, 2023-03-09 at 21:09 +0000, Thinh Nguyen wrote:
> On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > > > > 
> > > > > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > > > > 
> > > > > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > > > > we see something odd:
> > > > > > > > > >   Ping from PC to gadget works.
> > > > > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > > > > >   see gadget to PC start working.
> > > > > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > > > > 
> > > > > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > > > > Kernel 5.15.87
> > > > > > > > > 
> > > > > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > > > > 
> > > > > > > It's good enough :)
> > > > > > > 
> > > > > > > Have any logs at all that show any problems?
> > > > > > > 
> > > > > > No, don't know where to start. There are no errors logged.
> > > > > > 
> > > > > > >   Also, you might want to
> > > > > > >  cc:  the dwc3 maintainer...
> > > > > > 
> > > > > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > > > > 
> > > > > >  Jocke
> > > > > > 
> > > > > > > 
> > > > > > >  thanks,
> > > > > > > 
> > > > > > >  greg k-hj
> > > > > > 
> > > > > 
> > > > > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > > > > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > > > > 
> > > > > Found out something interesting, on PC:
> > > > > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > > > > echo 0 > usb2_hardware_lpm
> > > > > 
> > > > > Now ping works normally.
> > > > > 
> > > > > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > > > > on gadget side? 
> > > > > 
> > > 
> > > There's no option in gadget configfs to allow you to do that at the
> > > moment. You can disable LPM in dwc3 controller in the devicetree with
> > > "snps,dis_enblslpm_quirk" instead.
> > 
> > Yes, I found that. Thanks.
> 
> Also note that LPM is for device operating in usb2 speed. If it's usb3
> speeds, then you need to disable U3 suspend.

ATM I have:
	snps,dis_enblslpm_quirk;
	snps,dis_u2_susphy_quirk;
	snps,dis_u3_susphy_quirk;
	snps,usb2-gadget-lpm-disable;

That fixes it, I have no need for any LPM/suspend

> 
> > 
> > > 
> > > If the host puts the gadget in suspend, the gadget won't be able to
> > > communicate with the host until the host wakes the gadget up and starts
> > > talking to the gadget again. The gadget may be able to signal the host
> > > to wakeup via remote wakeup. Did you check if the device is in suspend?
> > > If it's in suspend, is the gadget enabled with remote wakeup? Did the
> > > NCM driver sent a remote wakeup signal to the host? I didn't verify, but
> > > I suspect the NCM gadget driver isn't configured/implemented with remote
> > > wakeup.
> > 
> > Then maybe NCM/RNDIS should inform/disable LPM in the device driver?
> > One cannot have half an impl. of this feature.
> 
> Most drivers don't fully support every feature. They are implemented as
> needed. I'm not familiar with NCM/RNDIS driver. Perhaps you can ping its
> maintainer to see its current status to see if it needs to be updated.
> Unless there's a quirk from the host/device, we shouldn't have to
> disable LPM.
> 
Right but isn't LPM something else? One can read in Documentation/ABI/testing/sysfs-bus-usb:
What:		/sys/bus/usb/devices/.../power/usb2_hardware_lpm
Date:		September 2011
Contact:	Andiry Xu <andiry.xu@amd.com>
Description:
		If CONFIG_PM is set and a USB 2.0 lpm-capable device is plugged
		in to a xHCI host which support link PM, it will perform a LPM
		test; if the test is passed and host supports USB2 hardware LPM
		(xHCI 1.0 feature), USB2 hardware LPM will be enabled for the
		device and the USB device directory will contain a file named
		power/usb2_hardware_lpm.  The file holds a string value (enable
		or disable) indicating whether or not USB2 hardware LPM is
		enabled for the device. Developer can write y/Y/1 or n/N/0 to
		the file to enable/disable the feature.

What:		/sys/bus/usb/devices/.../power/usb3_hardware_lpm_u1
		/sys/bus/usb/devices/.../power/usb3_hardware_lpm_u2
Date:		November 2015
Contact:	Kevin Strasser <kevin.strasser@linux.intel.com>
		Lu Baolu <baolu.lu@linux.intel.com>
Description:
		If CONFIG_PM is set and a USB 3.0 lpm-capable device is plugged
		in to a xHCI host which supports link PM, it will check if U1
		and U2 exit latencies have been set in the BOS descriptor; if
		the check is passed and the host supports USB3 hardware LPM,
		USB3 hardware LPM will be enabled for the device and the USB
		device directory will contain two files named
		power/usb3_hardware_lpm_u1 and power/usb3_hardware_lpm_u2. These
		files hold a string value (enable or disable) indicating whether
		or not USB3 hardware LPM U1 or U2 is enabled for the device.

This seems to indicate that LPM is on device driver/controller level rather than
on higher levels like RNDIS/NCM ?


While I remember, I tried to create both a NCM and a ttyACM on top of my dwc3 gadget
but then I got som bind error when activating the UDC. Is that expected ?
I figured both could live on the same device and would just mux the two protocols.

 Jocke

Re: dwc3 gadget and USB3

[Thinh Nguyen]

On Fri, Mar 10, 2023, Joakim Tjernlund wrote:
> On Thu, 2023-03-09 at 21:09 +0000, Thinh Nguyen wrote:
> > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > > > > > 
> > > > > > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > > > > > 
> > > > > > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > > > > > we see something odd:
> > > > > > > > > > >   Ping from PC to gadget works.
> > > > > > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > > > > > >   see gadget to PC start working.
> > > > > > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > > > > > 
> > > > > > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > > > > > Kernel 5.15.87
> > > > > > > > > > 
> > > > > > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > > > > > 
> > > > > > > > > 
> > > > > > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > > > > > 
> > > > > > > > It's good enough :)
> > > > > > > > 
> > > > > > > > Have any logs at all that show any problems?
> > > > > > > > 
> > > > > > > No, don't know where to start. There are no errors logged.
> > > > > > > 
> > > > > > > >   Also, you might want to
> > > > > > > >  cc:  the dwc3 maintainer...
> > > > > > > 
> > > > > > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > > > > > 
> > > > > > >  Jocke
> > > > > > > 
> > > > > > > > 
> > > > > > > >  thanks,
> > > > > > > > 
> > > > > > > >  greg k-hj
> > > > > > > 
> > > > > > 
> > > > > > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > > > > > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > > > > > 
> > > > > > Found out something interesting, on PC:
> > > > > > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > > > > > echo 0 > usb2_hardware_lpm
> > > > > > 
> > > > > > Now ping works normally.
> > > > > > 
> > > > > > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > > > > > on gadget side? 
> > > > > > 
> > > > 
> > > > There's no option in gadget configfs to allow you to do that at the
> > > > moment. You can disable LPM in dwc3 controller in the devicetree with
> > > > "snps,dis_enblslpm_quirk" instead.
> > > 
> > > Yes, I found that. Thanks.
> > 
> > Also note that LPM is for device operating in usb2 speed. If it's usb3
> > speeds, then you need to disable U3 suspend.

Actually, you can't disable U3. The device has to obey the host if it
requests for U3 entry.

> 
> ATM I have:
> 	snps,dis_enblslpm_quirk;
> 	snps,dis_u2_susphy_quirk;
> 	snps,dis_u3_susphy_quirk;
> 	snps,usb2-gadget-lpm-disable;
> 
> That fixes it, I have no need for any LPM/suspend

Note that adding these 2 won't help if the host send U3 request.
	snps,dis_u2_susphy_quirk;
	snps,dis_u3_susphy_quirk;

That's just disabling the phy from going into lower power state. You
need to prevent the host from autosuspending and put the device in U3.

For usb2 speed, you only need "snps,usb2-gadget-lpm-disable".


> 
> > 
> > > 
> > > > 
> > > > If the host puts the gadget in suspend, the gadget won't be able to
> > > > communicate with the host until the host wakes the gadget up and starts
> > > > talking to the gadget again. The gadget may be able to signal the host
> > > > to wakeup via remote wakeup. Did you check if the device is in suspend?
> > > > If it's in suspend, is the gadget enabled with remote wakeup? Did the
> > > > NCM driver sent a remote wakeup signal to the host? I didn't verify, but
> > > > I suspect the NCM gadget driver isn't configured/implemented with remote
> > > > wakeup.
> > > 
> > > Then maybe NCM/RNDIS should inform/disable LPM in the device driver?
> > > One cannot have half an impl. of this feature.
> > 
> > Most drivers don't fully support every feature. They are implemented as
> > needed. I'm not familiar with NCM/RNDIS driver. Perhaps you can ping its
> > maintainer to see its current status to see if it needs to be updated.
> > Unless there's a quirk from the host/device, we shouldn't have to
> > disable LPM.
> > 
> Right but isn't LPM something else? One can read in Documentation/ABI/testing/sysfs-bus-usb:
> What:		/sys/bus/usb/devices/.../power/usb2_hardware_lpm
> Date:		September 2011
> Contact:	Andiry Xu <andiry.xu@amd.com>
> Description:
> 		If CONFIG_PM is set and a USB 2.0 lpm-capable device is plugged
> 		in to a xHCI host which support link PM, it will perform a LPM
> 		test; if the test is passed and host supports USB2 hardware LPM
> 		(xHCI 1.0 feature), USB2 hardware LPM will be enabled for the
> 		device and the USB device directory will contain a file named
> 		power/usb2_hardware_lpm.  The file holds a string value (enable
> 		or disable) indicating whether or not USB2 hardware LPM is
> 		enabled for the device. Developer can write y/Y/1 or n/N/0 to
> 		the file to enable/disable the feature.
> 
> What:		/sys/bus/usb/devices/.../power/usb3_hardware_lpm_u1
> 		/sys/bus/usb/devices/.../power/usb3_hardware_lpm_u2
> Date:		November 2015
> Contact:	Kevin Strasser <kevin.strasser@linux.intel.com>
> 		Lu Baolu <baolu.lu@linux.intel.com>
> Description:
> 		If CONFIG_PM is set and a USB 3.0 lpm-capable device is plugged
> 		in to a xHCI host which supports link PM, it will check if U1
> 		and U2 exit latencies have been set in the BOS descriptor; if
> 		the check is passed and the host supports USB3 hardware LPM,
> 		USB3 hardware LPM will be enabled for the device and the USB
> 		device directory will contain two files named
> 		power/usb3_hardware_lpm_u1 and power/usb3_hardware_lpm_u2. These
> 		files hold a string value (enable or disable) indicating whether
> 		or not USB3 hardware LPM U1 or U2 is enabled for the device.
> 
> This seems to indicate that LPM is on device driver/controller level rather than
> on higher levels like RNDIS/NCM ?

The controller will handle the LPM. However, the UDC driver will a get
notification from the controller if it goes in to suspend (L2). It will
in turn invoke the suspend/resume gadget driver's callback from the
RNDIS/NCM driver's ops. So, the RNDIS/NCM can keep track of whether the
gadget is in suspend to initiate remote wakeup.

> 
> 
> While I remember, I tried to create both a NCM and a ttyACM on top of my dwc3 gadget
> but then I got som bind error when activating the UDC. Is that expected ?
> I figured both could live on the same device and would just mux the two protocols.
> 

I'm not clear what you meant here. The gadget driver and the UDC driver
should be separated and abstracted from each other.

Also, please note that all this talking points are based on the
assumption that the issue is due to the host putting the device in
suspend base on your experiment. We didn't review the logs.

IMO, the best option should be to prevent suspend from the host side as
it looks like your device may operate in usb3 speed also?

BR,
Thinh

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Fri, 2023-03-10 at 22:58 +0000, Thinh Nguyen wrote:
> On Fri, Mar 10, 2023, Joakim Tjernlund wrote:
> > On Thu, 2023-03-09 at 21:09 +0000, Thinh Nguyen wrote:
> > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > > > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > > 
> 
SNIP ...

> > 
> > 
> > While I remember, I tried to create both a NCM and a ttyACM on top of my dwc3 gadget
> > but then I got som bind error when activating the UDC. Is that expected ?
> > I figured both could live on the same device and would just mux the two protocols.
> > 
> 
> I'm not clear what you meant here. The gadget driver and the UDC driver
> should be separated and abstracted from each other.
> 
> Also, please note that all this talking points are based on the
> assumption that the issue is due to the host putting the device in
> suspend base on your experiment. We didn't review the logs.
> 
> IMO, the best option should be to prevent suspend from the host side as
> it looks like your device may operate in usb3 speed also?
> 
> BR,
> Thinh

This script best explain I think:
#! /bin/sh

stop() {
    g="/sys/kernel/config/usb_gadget/g1"
    if [ -e "${g}"/UDC ]; then
	echo "" >  "${g}"/UDC
	rm -f "${g}"/os_desc/c.1
	rm -f "${g}"/configs/c.1/ncm.usb0
	rm -f "${g}"/configs/c.1/acm.GS0
	[ -d "${g}"/configs/c.1/strings/0x409 ] && rmdir "${g}"/configs/c.1/strings/0x409
	[ -d "${g}"/functions/acm.GS0 ] && rmdir "${g}"/functions/acm.GS0
	[ -d "${g}"/functions/ncm.usb0 ] && rmdir "${g}"/functions/ncm.usb0
	[ -d "${g}"/c.1/strings ] && rmdir "${g}"/c.1/strings
	[ -d "${g}"/c.1/strings/0x409 ] && rmdir "${g}"/c.1/strings/0x409
	[ -d "${g}"/configs/c.1 ] && rmdir "${g}"/configs/c.1
	[ -d "${g}" ] && rmdir  "${g}"
    fi
}

start() {
    mkdir -p /sys/kernel/config/usb_gadget/g1
    cd /sys/kernel/config/usb_gadget/g1

    ###############################
    # Populate Device-Level Stuff #
    ###############################

    #Setting device class/subclass/protocol to these values
    # alerts the OS that this is a composite device with
    # IADs in it's firmware.
    # ref: https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/usb-interface-association-descriptor
    echo "0xEF" > bDeviceClass
    echo "0x02" > bDeviceSubClass
    echo "0x01" > bDeviceProtocol

    echo "0x15A2" > idVendor
    echo "0x0001" > idProduct

    #mkdir strings/0x409
    #echo "1234567" > strings/0x409/serialnumber
    #echo "Infinera" > strings/0x409/manufacturer
    #echo "NDU" > strings/0x409/product

    #enable use of os_desc's (important for RNDIS & NCM enablement on Windows):
    echo 1       > os_desc/use
    echo 0xbc    > os_desc/b_vendor_code #or 0xcd ?
    echo MSFT100 > os_desc/qw_sign

    #################################
    # Populate Individual Functions #
    #################################

    #The order functions are populated here will be reflected in the
    # order of descriptors written.

    #########
    # RNDIS #
    #########
    #Note! If RNDIS is enabled, it *has* to be the first function! Otherwise, Windows 10 will report error 10 (failed to start device).
    # (It's unclear why this is the case..)
    # https://docs.microsoft.com/en-us/answers/questions/474108/does-rndis-need-to-be-listed-as-the-first-function.html
    # https://stackoverflow.com/questions/68365739/windows-rndis-compatible-device-does-rndis-need-to-be-listed-as-the-first-funct
    if [ 0 = 1 ]
    then
	mkdir functions/rndis.usb0
	mkdir -p functions/rndis.usb0/os_desc/interface.rndis

	# Set compatible / sub-compatible id's so that Windows can match this
	# function to RNDIS6 driver more easily.
	echo RNDIS   > functions/rndis.usb0/os_desc/interface.rndis/compatible_id
	echo 5162001 > functions/rndis.usb0/os_desc/interface.rndis/sub_compatible_id

	mkdir -p configs/c.1
	mkdir -p configs/c.1/strings/0x409
	echo "0xC0" > configs/c.1/bmAttributes # Self powered
	echo "conf1" > configs/c.1/strings/0x409/configuration
	ln -s functions/rndis.usb0 configs/c.1
	if [ ! -L os_desc/c.1 ]
	then
	    ln -s configs/c.1 os_desc
	fi
    fi

    #########
    # NCM   #
    #########
    #Usually I test with *either* RNDIS or NCM enabled, but not both, hence the if(0) here..
    if [ 1 = 1 ]
    then
	mkdir functions/ncm.usb0
	mkdir -p functions/ncm.usb0/os_desc/interface.ncm
	# Set compatible id so that Windows 10 can match this function to
	# NCM driver more easily.
	echo WINNCM   > functions/ncm.usb0/os_desc/interface.ncm/compatible_id

	mkdir -p configs/c.1
	mkdir -p configs/c.1/strings/0x409
	echo "0xC0" > configs/c.1/bmAttributes # Self powered
	echo "conf1" > configs/c.1/strings/0x409/configuration
	ln -s functions/ncm.usb0 configs/c.1
	if [ ! -L os_desc/c.1 ]
	then
	    ln -s configs/c.1 os_desc
	fi
    fi

    #########
    # ACM   #
    #########
    if [ 1 = 1 ]
    then
	mkdir -p functions/acm.GS0

	mkdir -p configs/c.1
	mkdir -p configs/c.1/strings/0x409
	echo "0xC0" > configs/c.1/bmAttributes # Self powered
	ln -fs functions/acm.GS0 configs/c.1
	if [ ! -L os_desc/c.1 ]
	then
	    ln -s configs/c.1 os_desc
	fi
    fi

    #Activate/bind USB gadget
    echo 2f00000.usb > UDC
}
command=$@

case ${command} in
    start)
	echo "Starting usb-lc"
	start
	#ifconfig lc 169.254.100.101 netmask 255.255.255.252 # for debug
    ;;
    stop)
	echo "Stopping usb-lc"
	stop
    ;;
    restart)
	stop
	start
	;;

    *)
	echo "Create USB Gadget device dual NCM and rndis V0.1.1 for MAC, Linux and Windows"
	exit 1
	;;
esac


This gives me an error:

./config_gadget.sh start
Starting usb-lc
sh: write error: Device or resource busy

the line causing that is:
echo 2f00000.usb > UDC

Just create either NCM or ACM works, but not both

Re: dwc3 gadget and USB3

[Joakim Tjernlund]

On Fri, 2023-03-10 at 22:58 +0000, Thinh Nguyen wrote:
> On Fri, Mar 10, 2023, Joakim Tjernlund wrote:
> > On Thu, 2023-03-09 at 21:09 +0000, Thinh Nguyen wrote:
> > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > > > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > > > we are using fsl-ls1043a-rdb based design but with a ls1023a SOC and
> > > > > > > > > > > > use USB0 in gadget mode running either NCM or RNDIS ethernet on top.
> > > > > > > > > > > > 
> > > > > > > > > > > > When we connect the gadget to a PC(Linux of Windows) over an USB2 hub,
> > > > > > > > > > > > networking(NCM or RNDIS) works well.
> > > > > > > > > > > > 
> > > > > > > > > > > > However, when we connect the gadget directly to the PC/laptop which uses USB3
> > > > > > > > > > > > we see something odd:
> > > > > > > > > > > >   Ping from PC to gadget works.
> > > > > > > > > > > >   Ping from gadget to laptop does not. However if we also ping from PC at the same time we
> > > > > > > > > > > >   see gadget to PC start working.
> > > > > > > > > > > > Seems like ping from the PC tiggers the gadget to see incoming pkgs somehow.
> > > > > > > > > > > > 
> > > > > > > > > > > > Any idea what might be wrong or how to debug this?
> > > > > > > > > > > > Kernel 5.15.87
> > > > > > > > > > > 
> > > > > > > > > > > 5.15.y is very old, does this also happen on 6.2?
> > > > > > > > > > > 
> > > > > > > > > > 
> > > > > > > > > > I just tried 6.1.15 and the problem remains, I hope that is close enough ?
> > > > > > > > > 
> > > > > > > > > It's good enough :)
> > > > > > > > > 
> > > > > > > > > Have any logs at all that show any problems?
> > > > > > > > > 
> > > > > > > > No, don't know where to start. There are no errors logged.
> > > > > > > > 
> > > > > > > > >   Also, you might want to
> > > > > > > > >  cc:  the dwc3 maintainer...
> > > > > > > > 
> > > > > > > > I thought I did but that look like old info, added Thinh Nguyen now, thanks
> > > > > > > > 
> > > > > > > >  Jocke
> > > > > > > > 
> > > > > > > > > 
> > > > > > > > >  thanks,
> > > > > > > > > 
> > > > > > > > >  greg k-hj
> > > > > > > > 
> > > > > > > 
> > > > > > > Found and USBC Dock and connected that between gadget an PC and this also works well.
> > > > > > > Seems like a hub, regardless of USB2/USB3, make the usb network function in both directions.
> > > > > > > 
> > > > > > > Found out something interesting, on PC:
> > > > > > > cd /sys/devices/pci0000:00/0000:00:14.0/usb1/1-1/power  # Where my gadget is connected
> > > > > > > echo 0 > usb2_hardware_lpm
> > > > > > > 
> > > > > > > Now ping works normally.
> > > > > > > 
> > > > > > > So LPM does not seem to work properly on gadget. Can I disable LPM somehow
> > > > > > > on gadget side? 
> > > > > > > 
> > > > > 
> > > > > There's no option in gadget configfs to allow you to do that at the
> > > > > moment. You can disable LPM in dwc3 controller in the devicetree with
> > > > > "snps,dis_enblslpm_quirk" instead.
> > > > 
> > > > Yes, I found that. Thanks.
> > > 
> > > Also note that LPM is for device operating in usb2 speed. If it's usb3
> > > speeds, then you need to disable U3 suspend.
> 
> Actually, you can't disable U3. The device has to obey the host if it
> requests for U3 entry.
> 
> > 
> > ATM I have:
> > 	snps,dis_enblslpm_quirk;
> > 	snps,dis_u2_susphy_quirk;
> > 	snps,dis_u3_susphy_quirk;
> > 	snps,usb2-gadget-lpm-disable;
> > 
> > That fixes it, I have no need for any LPM/suspend
> 
> Note that adding these 2 won't help if the host send U3 request.
> 	snps,dis_u2_susphy_quirk;
> 	snps,dis_u3_susphy_quirk;
> 
> That's just disabling the phy from going into lower power state. You
> need to prevent the host from autosuspending and put the device in U3.
> 
> For usb2 speed, you only need "snps,usb2-gadget-lpm-disable".

Just checked with HW and out USB device is only wired for USB2.

 Jocke

Re: dwc3 gadget and USB3

[Thinh Nguyen]

On Sat, Mar 11, 2023, Joakim Tjernlund wrote:
> On Fri, 2023-03-10 at 22:58 +0000, Thinh Nguyen wrote:
> > On Fri, Mar 10, 2023, Joakim Tjernlund wrote:
> > > On Thu, 2023-03-09 at 21:09 +0000, Thinh Nguyen wrote:
> > > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > > On Thu, 2023-03-09 at 19:56 +0000, Thinh Nguyen wrote:
> > > > > > On Thu, Mar 09, 2023, Joakim Tjernlund wrote:
> > > > > > > On Thu, 2023-03-09 at 17:32 +0100, Joakim Tjernlund wrote:
> > > > > > > > On Wed, 2023-03-08 at 22:26 +0100, Joakim Tjernlund wrote:
> > > > > > > > > On Wed, 2023-03-08 at 19:58 +0100, gregkh@linuxfoundation.org wrote:
> > > > > > > > > > On Wed, Mar 08, 2023 at 06:12:51PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > > On Wed, 2023-03-08 at 18:25 +0100, Greg KH wrote:
> > > > > > > > > > > > On Wed, Mar 08, 2023 at 05:10:17PM +0000, Joakim Tjernlund wrote:
> > > > > > > > > > > > 
> > 
> SNIP ...
> 
> > > 
> > > 
> > > While I remember, I tried to create both a NCM and a ttyACM on top of my dwc3 gadget
> > > but then I got som bind error when activating the UDC. Is that expected ?
> > > I figured both could live on the same device and would just mux the two protocols.
> > > 
> > 
> > I'm not clear what you meant here. The gadget driver and the UDC driver
> > should be separated and abstracted from each other.
> > 
> > Also, please note that all this talking points are based on the
> > assumption that the issue is due to the host putting the device in
> > suspend base on your experiment. We didn't review the logs.
> > 
> > IMO, the best option should be to prevent suspend from the host side as
> > it looks like your device may operate in usb3 speed also?
> > 
> > BR,
> > Thinh
> 
> This script best explain I think:
> #! /bin/sh
> 
> stop() {
>     g="/sys/kernel/config/usb_gadget/g1"
>     if [ -e "${g}"/UDC ]; then
> 	echo "" >  "${g}"/UDC
> 	rm -f "${g}"/os_desc/c.1
> 	rm -f "${g}"/configs/c.1/ncm.usb0
> 	rm -f "${g}"/configs/c.1/acm.GS0
> 	[ -d "${g}"/configs/c.1/strings/0x409 ] && rmdir "${g}"/configs/c.1/strings/0x409
> 	[ -d "${g}"/functions/acm.GS0 ] && rmdir "${g}"/functions/acm.GS0
> 	[ -d "${g}"/functions/ncm.usb0 ] && rmdir "${g}"/functions/ncm.usb0
> 	[ -d "${g}"/c.1/strings ] && rmdir "${g}"/c.1/strings
> 	[ -d "${g}"/c.1/strings/0x409 ] && rmdir "${g}"/c.1/strings/0x409
> 	[ -d "${g}"/configs/c.1 ] && rmdir "${g}"/configs/c.1
> 	[ -d "${g}" ] && rmdir  "${g}"
>     fi
> }
> 
> start() {
>     mkdir -p /sys/kernel/config/usb_gadget/g1
>     cd /sys/kernel/config/usb_gadget/g1
> 
>     ###############################
>     # Populate Device-Level Stuff #
>     ###############################
> 
>     #Setting device class/subclass/protocol to these values
>     # alerts the OS that this is a composite device with
>     # IADs in it's firmware.
>     # ref: https://urldefense.com/v3/__https://docs.microsoft.com/en-us/windows-hardware/drivers/usbcon/usb-interface-association-descriptor__;!!A4F2R9G_pg!f2DmukBCz-n2dXRTDTd22YjHdXgKXWZwCLtuwoB4SdjW-U7s2s9Cyez4-JjmHJNzhetxdX06a6iohV_1yj7bpq2Ha28LvwALSw$ 
>     echo "0xEF" > bDeviceClass
>     echo "0x02" > bDeviceSubClass
>     echo "0x01" > bDeviceProtocol
> 
>     echo "0x15A2" > idVendor
>     echo "0x0001" > idProduct
> 
>     #mkdir strings/0x409
>     #echo "1234567" > strings/0x409/serialnumber
>     #echo "Infinera" > strings/0x409/manufacturer
>     #echo "NDU" > strings/0x409/product
> 
>     #enable use of os_desc's (important for RNDIS & NCM enablement on Windows):
>     echo 1       > os_desc/use
>     echo 0xbc    > os_desc/b_vendor_code #or 0xcd ?
>     echo MSFT100 > os_desc/qw_sign
> 
>     #################################
>     # Populate Individual Functions #
>     #################################
> 
>     #The order functions are populated here will be reflected in the
>     # order of descriptors written.
> 
>     #########
>     # RNDIS #
>     #########
>     #Note! If RNDIS is enabled, it *has* to be the first function! Otherwise, Windows 10 will report error 10 (failed to start device).
>     # (It's unclear why this is the case..)
>     # https://urldefense.com/v3/__https://docs.microsoft.com/en-us/answers/questions/474108/does-rndis-need-to-be-listed-as-the-first-function.html__;!!A4F2R9G_pg!f2DmukBCz-n2dXRTDTd22YjHdXgKXWZwCLtuwoB4SdjW-U7s2s9Cyez4-JjmHJNzhetxdX06a6iohV_1yj7bpq2Ha28URWCTNQ$ 
>     # https://urldefense.com/v3/__https://stackoverflow.com/questions/68365739/windows-rndis-compatible-device-does-rndis-need-to-be-listed-as-the-first-funct__;!!A4F2R9G_pg!f2DmukBCz-n2dXRTDTd22YjHdXgKXWZwCLtuwoB4SdjW-U7s2s9Cyez4-JjmHJNzhetxdX06a6iohV_1yj7bpq2Ha2-HtzzXog$ 
>     if [ 0 = 1 ]
>     then
> 	mkdir functions/rndis.usb0
> 	mkdir -p functions/rndis.usb0/os_desc/interface.rndis
> 
> 	# Set compatible / sub-compatible id's so that Windows can match this
> 	# function to RNDIS6 driver more easily.
> 	echo RNDIS   > functions/rndis.usb0/os_desc/interface.rndis/compatible_id
> 	echo 5162001 > functions/rndis.usb0/os_desc/interface.rndis/sub_compatible_id
> 
> 	mkdir -p configs/c.1
> 	mkdir -p configs/c.1/strings/0x409
> 	echo "0xC0" > configs/c.1/bmAttributes # Self powered
> 	echo "conf1" > configs/c.1/strings/0x409/configuration
> 	ln -s functions/rndis.usb0 configs/c.1
> 	if [ ! -L os_desc/c.1 ]
> 	then
> 	    ln -s configs/c.1 os_desc
> 	fi
>     fi
> 
>     #########
>     # NCM   #
>     #########
>     #Usually I test with *either* RNDIS or NCM enabled, but not both, hence the if(0) here..
>     if [ 1 = 1 ]
>     then
> 	mkdir functions/ncm.usb0
> 	mkdir -p functions/ncm.usb0/os_desc/interface.ncm
> 	# Set compatible id so that Windows 10 can match this function to
> 	# NCM driver more easily.
> 	echo WINNCM   > functions/ncm.usb0/os_desc/interface.ncm/compatible_id
> 
> 	mkdir -p configs/c.1
> 	mkdir -p configs/c.1/strings/0x409
> 	echo "0xC0" > configs/c.1/bmAttributes # Self powered
> 	echo "conf1" > configs/c.1/strings/0x409/configuration
> 	ln -s functions/ncm.usb0 configs/c.1
> 	if [ ! -L os_desc/c.1 ]
> 	then
> 	    ln -s configs/c.1 os_desc
> 	fi
>     fi
> 
>     #########
>     # ACM   #
>     #########
>     if [ 1 = 1 ]
>     then
> 	mkdir -p functions/acm.GS0
> 
> 	mkdir -p configs/c.1
> 	mkdir -p configs/c.1/strings/0x409
> 	echo "0xC0" > configs/c.1/bmAttributes # Self powered
> 	ln -fs functions/acm.GS0 configs/c.1
> 	if [ ! -L os_desc/c.1 ]
> 	then
> 	    ln -s configs/c.1 os_desc
> 	fi
>     fi
> 
>     #Activate/bind USB gadget
>     echo 2f00000.usb > UDC
> }
> command=$@
> 
> case ${command} in
>     start)
> 	echo "Starting usb-lc"
> 	start
> 	#ifconfig lc 169.254.100.101 netmask 255.255.255.252 # for debug
>     ;;
>     stop)
> 	echo "Stopping usb-lc"
> 	stop
>     ;;
>     restart)
> 	stop
> 	start
> 	;;
> 
>     *)
> 	echo "Create USB Gadget device dual NCM and rndis V0.1.1 for MAC, Linux and Windows"
> 	exit 1
> 	;;
> esac
> 
> 
> This gives me an error:
> 
> ./config_gadget.sh start
> Starting usb-lc
> sh: write error: Device or resource busy
> 
> the line causing that is:
> echo 2f00000.usb > UDC
> 
> Just create either NCM or ACM works, but not both

Not sure. You may need to trace through the code path and see where it
fails.

BR,
Thinh