Re: Sierra Wireless EM9191 integration issues in mhi+wwan

[Thomas Perrot <thomas.perrot@bootlin.com>]

[-- Attachment #1: Type: text/plain, Size: 5344 bytes --]

Hello Aleksander,

On Tue, 2021-10-12 at 21:44 +0200, Aleksander Morgado wrote:
> > > I'm working on a setup using a RPi CM4 based board and a Sierra
> > > Wireless EM9191 module, running OpenWRT 21.02 and backported
> > > mhi+wwan
> > > drivers (from latest 5.15 rc). The kernel also has the
> > > mhi_pci_dev_info entry written by Thomas, as per
> > > https://forum.sierrawireless.com/t/sierra-wireless-airprime-em919x-pcie-support/24927
> > > 
> > > The EM9191 is now running 02.08.01.00 (latest firmware from
> > > Sierra),
> > > upgraded in several steps back from the original 01.04.01.02 the
> > > module came with. The firmware upgrades were done with
> > > qmi-firmware-update and the module in USB mode  in a desktop PC.
> > > 
> > > Most of the system boots end up with the mhi driver reporting
> > > that the
> > > module firmware crashed in different ways:
> > > 
> > > [    7.060730] mhi-pci-generic 0000:01:00.0: MHI PCI device
> > > found:
> > > sierra-em919x
> > > [    7.067906] mhi-pci-generic 0000:01:00.0: BAR 0: assigned [mem
> > > 0x600000000-0x600000fff 64bit]
> > > [    7.076455] mhi-pci-generic 0000:01:00.0: enabling device
> > > (0000 ->
> > > 0002)
> > > [    7.083277] mhi-pci-generic 0000:01:00.0: using shared MSI
> > > [    7.089508] mhi mhi0: Requested to power ON
> > > [    7.094080] mhi mhi0: Attempting power on with EE: PASS
> > > THROUGH,
> > > state: SYS ERROR
> > > [    7.180371] mhi mhi0: local ee: INVALID_EE state: RESET device
> > > ee:
> > > PASS THROUGH state: SYS ERROR
> > > [    7.187146] mhi mhi0: Power on setup success
> > > [    7.187219] mhi mhi0: Handling state transition: PBL
> > > [    7.189165] mhi mhi0: System error detected
> > > [    7.189178] mhi mhi0: Device MHI is not in valid state
> > > [    7.189189] mhi-pci-generic 0000:01:00.0: firmware crashed (7)
> > > [    7.213682] mhi mhi0: Handling state transition: SYS ERROR
> > > [    7.219183] mhi mhi0: Transitioning from PM state: Linkdown or
> > > Error Fatal Detect to: SYS ERROR Process
> > > [    7.228590] mhi-pci-generic 0000:01:00.0: firmware crashed (6)
> > > [    7.234429] mhi mhi0: Failed to transition from PM state:
> > > Linkdown
> > > or Error Fatal Detect to: SYS ERROR Process
> > > [    7.244433] mhi mhi0: Exiting with PM state: Linkdown or Error
> > > Fatal Detect, MHI state: RESET
> > > [    7.252963] mhi mhi0: Handling state transition: DISABLE
> > > [    7.258278] mhi mhi0: Processing disable transition with PM
> > > state:
> > > Linkdown or Error Fatal Detect
> > > [    7.267155] mhi mhi0: Waiting for all pending event ring
> > > processing
> > > to complete
> > > [    7.274480] mhi mhi0: Waiting for all pending threads to
> > > complete
> > > [    7.280576] mhi mhi0: Reset all active channels and remove MHI
> > > devices
> > > [    7.287110] mhi mhi0: Resetting EV CTXT and CMD CTXT
> > > [    7.292077] mhi mhi0: Exiting with PM state: DISABLE, MHI
> > > state:
> > > RESET
> > > [    7.298683] mhi-pci-generic 0000:01:00.0: failed to power up
> > > MHI
> > > controller
> > > [    7.306184] mhi-pci-generic: probe of 0000:01:00.0 failed with
> > > error
> > > -110
> > > 
> > > Some of the boots successfully finish and I can talk to both the
> > > QMI
> > > and MBIM ports exposed by the WWAN subsystem, but the success
> > > rate is
> > > extremely low.
> > > 
> > > Thomas, are you seeing similar issues in your setup?
> > 
> > On our setup, using i.MX6DL based board and a PCIe Sierra Wireless
> > EM9190 module, running Yocto and Linux 5.13, we don't have much
> > success
> > for the moment, qmi and mbim commands very often end in timeout.
> > 
> > Otherwise, when responses are received, we also can observe strange
> > things: unexpected messages, response to previous commands or queue
> > buffer issue.
> > 
> 
> Are you testing this with qmicli + mbimcli + ModemManager? And if so,
> are you running the qmicli/mbimcli commands with the "-p" option in
> order to always use the intermediate qmi-proxy/mbim-proxy? I'd
> suggest
> to always do that to avoid having multiple processes talking to the
> ports at the same time.
> 

In first, I'm testing with qmicli/mbimcli commands with the "-p" option
in order to always use the intermediate qmi-proxy/mbim-proxy that that
I run in debug mode beforehand.

> 
> > I updated the topic opened on the Sierra Wireless forum, with our
> > latest progress and as well as additional information.
> > 
> > In addition, we observed some strange behavior of the EM919x after
> > warm
> > reboots.
> > 
> 
> Is the log after the warm reboots similar to the one I showed in my
> first email, i.e. with the 2 reported "firmware crashes"? Or does it
> look totally different?

After warm reboots, we observe no explain message indicating an error,
but we use an old firmware version.

> And, do you always see the module booting properly on cold boots? Or
> do you see failed boots like the one i showed in my first email?

The module doesn't always booting properly, you see failed boots like
the one you showed.

Best regards,
Thomas

> 

-- 
Thomas Perrot, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 659 bytes --]