SMMU Fault after setting RUN_STOP during gadget_pullup

SMMU Fault after setting RUN_STOP during gadget_pullup

[Krishna Kurapati PSSNV]

[-- Attachment #1: Type: text/plain, Size: 4951 bytes --]

Hi all,

I am facing one SMMU fault with FAR=0x00 on current targets running on 
5.10 and 5.15 with following signature:

20694.894996:   arm-smmu 15000000.apps-smmu: Unhandled arm-smmu context
fault from a600000.dwc3!
20694.905122:   arm-smmu 15000000.apps-smmu: FAR    = 0x0000000000000000
20694.930631:   arm-smmu 15000000.apps-smmu: PAR    = 0x0000000000000000
20694.938191:   arm-smmu 15000000.apps-smmu: FSR    = 0x40000402 [TF W SS ]
20694.946061:   arm-smmu 15000000.apps-smmu: FSYNR0    = 0x1a0011
20694.953022:   arm-smmu 15000000.apps-smmu: FSYNR1    = 0xaa00
20694.967045:   arm-smmu 15000000.apps-smmu: TTBR0  = 0x0000000013537042
20694.974600:   arm-smmu 15000000.apps-smmu: TTBR1  = 0x00000000f58cd1ac
20694.982171:   arm-smmu 15000000.apps-smmu: SCTLR  = 0x0a5f00e7 ACTLR = 
0x00000003
20694.990847:   arm-smmu 15000000.apps-smmu: CBAR  = 0x0001f300
20694.997696:   arm-smmu 15000000.apps-smmu: MAIR0 = 0xf404ff44 MAIR1 = 
0x0000efe4
20695.021604:   arm-smmu 15000000.apps-smmu: soft 
iova-to-phys=0x0000000000000000

This is happening when peripheral mode is kicking in and is reproducible 
randomly after long intervals. The SMMU fault happens soon after Run 
stop is set in gadget_pullup.

I tried to dump the dwc registers from offset 0xc100 (GSBUSCFG0) to 
0xcc10 (OSTS) (dump attached) by implementing an SMMU fault handler for 
dwc3 and see that only the following registers are non-zero. All other 
registers show value 0x00:

GSTS: 0x7E800020
GSNPSID: 0x5533330A
GUID: 0x00050A88
GUSB2PHYCFG0: 0x00002400
GUSB3PIPECTL0: 0x030E0002
DCFG: 0x008C0804
DCTL: 0x008C0804
DEVTEN: 0x008C0804
DSTS: 0x0003C5B4

As per the spec, when core soft reset is done:

"This bit clears the interrupts and all the CSRs except GSTS, GSNPSID, 
GGPIO, GUID, GUSB2PHYCFGn registers, GUSB3PIPECTLn registers, DCFG, 
DCTL, DEVTEN, and DSTS registers."

We thought soft reset timed-out but in the dumped registers, we don't 
see CORESOFT_RST bit (Bit 30) set in DCTL (Core soft reset is 
successful). We tried to check the dwc3_readl and dwc3_writel traces and 
see the following happening during gadget pullup:
		
//Write and readback Core soft reset in gadget_pullup
20694.424864:  dwc3_readl   base=0xffffffc0147ac000 offset=50948 
value=15728640
20694.424867:  dwc3_writel   base=0xffffffc0147ac000 offset=50948 
value=1089470464
20694.425853:  dwc3_readl   base=0xffffffc0147ac000 offset=50948 
value=15728640

//dwc3_event_buffers_setup as part of gadget_pullup
20694.425856:  dwc3_writel   base=0xffffffc0147ac000 offset=50176 
value=4026527744
20694.427826:  dwc3_writel   base=0xffffffc0147ac000 offset=50180  value=0
20694.428815:  dwc3_writel   base=0xffffffc0147ac000 offset=50184 
value=4096
20694.430780:  dwc3_writel   base=0xffffffc0147ac000 offset=50188  value=0

//Write imod if any in __dwc3_gadget_start
20694.443587:  dwc3_writel   base=0xffffffc0147ac000 offset=51712  value=0

//Modify GRXTHRCFG in __dwc3_gadget_start
20694.445557:  dwc3_readl   base=0xffffffc0147ac000 offset=49420  value=0
20694.445560:  dwc3_writel   base=0xffffffc0147ac000 offset=49420  value=0

//DCFG read/writes to setup nump in in __dwc3_gadget_start
20694.446547:  dwc3_readl   base=0xffffffc0147ac000 offset=50944 
value=526340
20694.446550:  dwc3_writel   base=0xffffffc0147ac000 offset=50944 
value=788484

//Set Ignore Packet Pending bit __dwc3_gadget_start
20694.446553:  dwc3_readl   base=0xffffffc0147ac000 offset=50944 
value=788484
20694.446555:  dwc3_writel   base=0xffffffc0147ac000 offset=50944 
value=9177092

//Ep transfer resource config and ep0 out/in enable happened here

//dwc3_gadget_enable_irq modifies DEVTEN for enabling IRQ with bits
20694.629290:  dwc3_writel   base=0xffffffc0147ac000 offset=50952 
value=3671

//DCTL read in run stop
20694.629313:  dwc3_readl   base=0xffffffc0147ac000 offset=50948 
value=15728640

//__dwc3_gadget_set_speed modifies speed appropriately
20694.629316:  dwc3_readl   base=0xffffffc0147ac000 offset=50944 
value=9177092
20694.629318:  dwc3_writel   base=0xffffffc0147ac000 offset=50944 
value=9177092

//Write run stop bit in DCTL via dwc3_gadget_dctl_write_safe()
20694.629321:  dwc3_writel   base=0xffffffc0147ac000 offset=50948 
value=2163212288

//Read DSTS to check if DWC3_DSTS_DEVCTRLHLT is cleared or not
20694.629325:  dwc3_readl   base=0xffffffc0147ac000 offset=50956 
value=9568260

Everything seems to be fine from the logs up until this point but SMMU 
fault occurs soon after the RUN_STOP bit is written and DSTS is checked 
for DWC3_DSTS_DEVCTRLHLT bit and SMMU handler begins reading registers 
for dumping them. I am not sure why GEVTADDRH/GEVTADDRL/ DEPCMD 
registers too are 0x00 in the dumps collected in smmu fault handler. Is 
there any reason why this could happen ? Any debug pointers in this 
regard would be helpful.

Attached are the reg dump from SMMU fault handler, dwc3 readl/writel 
traces and usb gadget traces of the crash instance.

Thanks in advance,
Krishna,

[-- Attachment #2: traces_n_dumps.7z --]
[-- Type: application/octet-stream, Size: 106635 bytes --]

Re: SMMU Fault after setting RUN_STOP during gadget_pullup

[Greg KH]

On Fri, Dec 23, 2022 at 07:26:49PM +0530, Krishna Kurapati PSSNV wrote:
> Hi all,
> 
> I am facing one SMMU fault with FAR=0x00 on current targets running on 5.10
> and 5.15 with following signature:

What about 6.1?

thanks,

greg k-h

Re: SMMU Fault after setting RUN_STOP during gadget_pullup

[Krishna Kurapati PSSNV]

On 12/23/2022 7:56 PM, Greg KH wrote:
> On Fri, Dec 23, 2022 at 07:26:49PM +0530, Krishna Kurapati PSSNV wrote:
>> Hi all,
>>
>> I am facing one SMMU fault with FAR=0x00 on current targets running on 5.10
>> and 5.15 with following signature:
> 
> What about 6.1?
> 
> thanks,
> 
> greg k-h

Hi Greg,

  Current mobile targets are running on 5.10 and 5.15 only till now. No 
target on 6.1 hasn't come up completely yet. So not sure if this issue 
is there on 6.1.

  I can try putting 6.1 on a minimal busybox ramdisk image on the mobile 
target and see if the issue comes up but seeing that the issue is random 
and seen only after long duration, I might take some time to get back 
with the results.

In the meantime, I would be happy to try out any ideas coming up from 
community in the meantime on current 5.15 build.

Regards,
Krishna,

Re: SMMU Fault after setting RUN_STOP during gadget_pullup

[Greg KH]

On Fri, Dec 23, 2022 at 08:34:08PM +0530, Krishna Kurapati PSSNV wrote:
> 
> 
> On 12/23/2022 7:56 PM, Greg KH wrote:
> > On Fri, Dec 23, 2022 at 07:26:49PM +0530, Krishna Kurapati PSSNV wrote:
> > > Hi all,
> > > 
> > > I am facing one SMMU fault with FAR=0x00 on current targets running on 5.10
> > > and 5.15 with following signature:
> > 
> > What about 6.1?
> > 
> > thanks,
> > 
> > greg k-h
> 
> Hi Greg,
> 
>  Current mobile targets are running on 5.10 and 5.15 only till now. No
> target on 6.1 hasn't come up completely yet. So not sure if this issue is
> there on 6.1.
> 
>  I can try putting 6.1 on a minimal busybox ramdisk image on the mobile
> target and see if the issue comes up but seeing that the issue is random and
> seen only after long duration, I might take some time to get back with the
> results.
> 
> In the meantime, I would be happy to try out any ideas coming up from
> community in the meantime on current 5.15 build.

We can't do much about older kernels, as you know, sorry.  If you can
reproduce the issue on a modern kernel (i.e. 6.1), then we can look into
it.  If you can't, then 'git bisect' is your friend :)

good luck!

greg k-h