error -110 sending stop command

error -110 sending stop command

[Timo Teras]

Hi,

I'm having the following errors on my SD-card when writing to them:

mmc0: Timeout waiting for hardware interrupt.
mmcblk0: error -110 sending stop command, original cmd response 0x900,
card status 0x400e00

Basically this happens when writing to the SD, and after this the
SD-card becomes unusable.

After this, the kernel [mmcqd/0] task is hogging CPU, and mmc0
interrupts are coming in at very fast pace (seems 10k+ interrupts
per second). Waiting long enough the following messages appear too:

[  896.871617] mmc0: Card stuck in programming state! mmcblk0 card_busy_detect
[  898.235070] mmc0: card never left busy state
[  898.235078] mmc0: tried to reset card
[  898.235087] blk_update_request: I/O error, dev mmcblk0, sector 16384
[  898.235098] Buffer I/O error on dev mmcblk0p1, logical block 8192, lost async page write
[  898.235107] blk_update_request: I/O error, dev mmcblk0, sector 16385
[  898.235114] Buffer I/O error on dev mmcblk0p1, logical block 8193, lost async page write
[  898.237103] mmcblk0: error -110 sending status command, retrying
[  898.238069] mmcblk0: error -110 sending status command, retrying
[  898.239057] mmcblk0: error -110 sending status command, aborting

After which the SD does not work at all until reboot.

This is pretty much what Richard Ash described in the thread 'error -110
sending stop command' on Zynq platform. Also the thread 'Unstable mmc
operation on armada38x' from Tor Krill looks the same issue.

The hardware I have is http://www.pcengines.ch/apu2c4.htm and on bootup
dmesg says:
[    3.082221] sdhci: Secure Digital Host Controller Interface driver
[    3.082228] sdhci: Copyright(c) Pierre Ossman
[    3.083135] sdhci-pci 0000:00:14.7: SDHCI controller found [1022:7813] (rev 1)
[    3.083495] sdhci-pci 0000:00:14.7: No vmmc regulator found
[    3.083503] sdhci-pci 0000:00:14.7: No vqmmc regulator found
[    3.084994] mmc0: SDHCI controller on PCI [0000:00:14.7] using ADMA 64-bit
[    3.156461] mmc0: new high speed SDHC card at address 0007
[    3.347692] mmcblk0: mmc0:0007 SD04G 3.71 GiB 
[    3.349924]  mmcblk0: p1

I have verified that this issues exist in 4.1.20 and 4.4.10.

Any suggestions how to debug this further? Perhaps some quirks to test?
Or already committed patches/fixes that might resolve this issue?

Thanks,
Timo

Re: error -110 sending stop command

[Timo Teras]

On Tue, 24 May 2016 09:37:46 +0300
Timo Teras <timo.teras@iki.fi> wrote:

> I'm having the following errors on my SD-card when writing to them:
> 
> mmc0: Timeout waiting for hardware interrupt.
> mmcblk0: error -110 sending stop command, original cmd response 0x900,
> card status 0x400e00
>[...] 
>
> Any suggestions how to debug this further? Perhaps some quirks to
> test? Or already committed patches/fixes that might resolve this
> issue?

If I force PIO mode (sdhci.debug_quirks=0x60), I get:
[  111.201763] mmc0: Timeout waiting for hardware interrupt.
[  111.201887] mmc0: Got data interrupt 0x00000020 even though no data operation was in progress.
[  111.202032] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x400e00    

And enabling SDHCI_QUIRK_NO_MULTIBLOCK (sdhci.debug_quirks=0x200000)
seems to fix all immediate issues, but it also degrades the performance
seriously.
 
/Timo

Re: error -110 sending stop command

[Tor Krill]

Hi Timo and others,

Sorry for the late reply.

On tis, 2016-05-24 at 17:37 +0300, Timo Teras wrote:
> On Tue, 24 May 2016 09:37:46 +0300
> Timo Teras <timo.teras@iki.fi> wrote:
> 
> > I'm having the following errors on my SD-card when writing to them:
> > 
> > mmc0: Timeout waiting for hardware interrupt.
> > mmcblk0: error -110 sending stop command, original cmd response
> > 0x900,
> > card status 0x400e00
> > [...] 
> > 
> > Any suggestions how to debug this further? Perhaps some quirks to
> > test? Or already committed patches/fixes that might resolve this
> > issue?

As a comment on your first email. We still have problems with our board
and the eMMC on it. (A solid run clearfog and the  MicroSoM A388)

With any recent kernel we experience write operations that fail and
then the no response on the status poll. After this any communication
with the chip seems fruitless.

Since we could not rule out that this could be an issue with the
specific eMMC we are currently investigating if we can reproduce the
problems with an SD-card instead and have to wait for tests on new SOMs
without the eMMC. (These arrived yesterday and we should hopefully be
able to test these shortly)

> If I force PIO mode (sdhci.debug_quirks=0x60), I get:
> [  111.201763] mmc0: Timeout waiting for hardware interrupt.
> [  111.201887] mmc0: Got data interrupt 0x00000020 even though no
> data operation was in progress.
> [  111.202032] mmcblk0: error -110 sending stop command, original cmd
> response 0x900, card status 0x400e00    
> 
> And enabling SDHCI_QUIRK_NO_MULTIBLOCK (sdhci.debug_quirks=0x200000)
> seems to fix all immediate issues, but it also degrades the
> performance
> seriously.

This is somewhat analogous with what i have seen. We have stable
operation when we use the Marvell/solid-run provided SDK kernel 3.x
something.

I have also tried bisecting the kernel without any definite culprit
identified. I can however observe what seems to be a more stable system
on older versions of the kernel.

For some time i had this commit as a candidate for the problem:

https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/
?id=d3df0465db00cf4ed9f90d0bfc3b827d32b9c796

But it turned out to be a false positive. After a few hours of testing
i got the same problem.

Thus it seems like timing/performance could be the problem here, at
least for us. As earlier mentioned, when enabling debug logging on the
subsystem the problem goes away or at least makes the bug less
frequently triggered.

But as said above, we are currently trying to isolate the problem to
sw/hw, controller, driver etc.

Best!

/Tor

Re: error -110 sending stop command

[Richard Ash]

On 25 May 2016 at 08:22, Tor Krill <tor@openproducts.com> wrote:
> Hi Timo and others,
>
> Sorry for the late reply.
>
> On tis, 2016-05-24 at 17:37 +0300, Timo Teras wrote:
>> On Tue, 24 May 2016 09:37:46 +0300
>> Timo Teras <timo.teras@iki.fi> wrote:
>>
>> > I'm having the following errors on my SD-card when writing to them:
>> >
>> > mmc0: Timeout waiting for hardware interrupt.
>> > mmcblk0: error -110 sending stop command, original cmd response
>> > 0x900,
>> > card status 0x400e00
>> > [...]
>> >
>> > Any suggestions how to debug this further? Perhaps some quirks to
>> > test? Or already committed patches/fixes that might resolve this
>> > issue?
>
> As a comment on your first email. We still have problems with our board
> and the eMMC on it. (A solid run clearfog and the  MicroSoM A388)
>
> With any recent kernel we experience write operations that fail and
> then the no response on the status poll. After this any communication
> with the chip seems fruitless.
>
> Since we could not rule out that this could be an issue with the
> specific eMMC we are currently investigating if we can reproduce the
> problems with an SD-card instead and have to wait for tests on new SOMs
> without the eMMC. (These arrived yesterday and we should hopefully be
> able to test these shortly)
>
>> If I force PIO mode (sdhci.debug_quirks=0x60), I get:
>> [  111.201763] mmc0: Timeout waiting for hardware interrupt.
>> [  111.201887] mmc0: Got data interrupt 0x00000020 even though no
>> data operation was in progress.
>> [  111.202032] mmcblk0: error -110 sending stop command, original cmd
>> response 0x900, card status 0x400e00

Quick update to say that after banging my head against this for a
while I managed to RMA the microSD cards I was using and a ZedBoard
(Zynq eval board) back to my supplier (UK) and then on to the flash
vendor (Taiwan), who reproduced the problem and declared it to be a
bug in the firmware on the microSD card, which is fixed by more recent
firmware releases. I have since tested a number of cards with the more
recent firmware without problems. I also found that the AMD SD host
controller in my laptop (uses SDHCI-PCI driver) had the same problem,
same error message.

My microSD cards were from Phison (Toshiba flash chips, but Phison do
the firmware), the ones with 'FW 82.99' on the back caused the
problem, ones marked 'FW 82.101' are trouble free.

Richard

Re: error -110 sending stop command

[Tor Krill]

On fre, 2016-06-03 at 15:20 +0100, Richard Ash wrote:
> On 25 May 2016 at 08:22, Tor Krill <tor@openproducts.com> wrote:

> > On tis, 2016-05-24 at 17:37 +0300, Timo Teras wrote:
> > > On Tue, 24 May 2016 09:37:46 +0300
> > > Timo Teras <timo.teras@iki.fi> wrote:
> > > 
> > > > I'm having the following errors on my SD-card when writing to
> > > > them:
> > > > 
> > > > mmc0: Timeout waiting for hardware interrupt.
> > > > mmcblk0: error -110 sending stop command, original cmd response
> > > > 0x900,
> > > > card status 0x400e00
> > > > [...]
> > > > 
> > > > Any suggestions how to debug this further? Perhaps some quirks
> > > > to
> > > > test? Or already committed patches/fixes that might resolve
> > > > this
> > > > issue?
> > 
> > As a comment on your first email. We still have problems with our
> > board
> > and the eMMC on it. (A solid run clearfog and the  MicroSoM A388)
> > 
> > With any recent kernel we experience write operations that fail and
> > then the no response on the status poll. After this any
> > communication
> > with the chip seems fruitless.
> > 
> > Since we could not rule out that this could be an issue with the
> > specific eMMC we are currently investigating if we can reproduce
> > the
> > problems with an SD-card instead and have to wait for tests on new
> > SOMs
> > without the eMMC. (These arrived yesterday and we should hopefully
> > be
> > able to test these shortly)
> > 
> > > If I force PIO mode (sdhci.debug_quirks=0x60), I get:
> > > [  111.201763] mmc0: Timeout waiting for hardware interrupt.
> > > [  111.201887] mmc0: Got data interrupt 0x00000020 even though no
> > > data operation was in progress.
> > > [  111.202032] mmcblk0: error -110 sending stop command, original
> > > cmd
> > > response 0x900, card status 0x400e00
> 
> Quick update to say that after banging my head against this for a
> while I managed to RMA the microSD cards I was using and a ZedBoard
> (Zynq eval board) back to my supplier (UK) and then on to the flash
> vendor (Taiwan), who reproduced the problem and declared it to be a
> bug in the firmware on the microSD card, which is fixed by more
> recent
> firmware releases. I have since tested a number of cards with the
> more
> recent firmware without problems. I also found that the AMD SD host
> controller in my laptop (uses SDHCI-PCI driver) had the same problem,
> same error message.
> 
> My microSD cards were from Phison (Toshiba flash chips, but Phison do
> the firmware), the ones with 'FW 82.99' on the back caused the
> problem, ones marked 'FW 82.101' are trouble free.
> 

To add our comment on this. We have had a similar experience. When
running the system on other chips, i.e. sd-cards, the system seems
stable. Thus it seems like the culprit where the eMMC on the SolidRun
SOC in our case as well. 

/Tor