Re: dw_mmc: HLE errors

[Jaehoon Chung <jh80.chung@samsung.com>]

On 11/24/2015 10:55 AM, Jorge Ramirez-Ortiz wrote:
> On 11/23/2015 07:11 PM, Jaehoon Chung wrote:
>> Dear, Jorge.
>>
>> On 11/24/2015 02:29 AM, Jorge Ramirez-Ortiz wrote:
>>> On 11/23/2015 11:57 AM, Doug Anderson wrote:
>>>> Jorge,
>>>>
>>>> On Mon, Nov 23, 2015 at 6:10 AM, Jorge Ramirez-Ortiz
>>>> <jorge.ramirez-ortiz@linaro.org> wrote:
>>>>> Doug/Jaehoon,
>>>>>
>>>>> Were there any follow ups to this thread [1] from March 30, 2015?
>>>>> We are seeing HLE errors on 3.18 and we are trying to determine if a solution
>>>>> was ever delivered.
>>>>> On inspection, I can't find anything specific in recent kernels that address
>>>>> this particular issue (was the actual root cause identified?)
>>>>>
>>>>> I put together a possible work-around that avoids the HLE storm from occurring
>>>>> for this specific SoC [2].
>>>>> However we'd rather not merge this -or any other similar fix- if there is a
>>>>> generic solution already that we can pick up from mainline.
>>>> Nothing landed that I'm aware of.  Are you on SDIO, SD or eMMC?
>>>> Trying to do UHS?
>>> SD even without UHS (yet, that is coming now)
>> If you want to use the upper mode than UHS-DDR50 for SD-card, you need to apply the below patch.
> 
> ACK
> 
>>
>> https://patchwork.kernel.org/patch/7456121/
>>
>> Actually, this is not relevant to HLE error.
>>
>> When sd-card is inserted/removed quickly, then sometime dwmmc controller is occurred the HLE error.
>> (Now, i can't see HLE error.)
>> So i had applied the some reset processing at my official repository.(It's not generic solution.)
> 
> Thanks, I'll have a look now.
> 
> I believe this to be your official repo:
> https://github.com/jh80chung/dw-mmc
> 
> Please let me know if it is not.

Sorry. it's not official repo (Samsung). So i can't share URL. :(
It's just my personal git repository.  I will work on that repository.. :)

Best Regards,
Jaehoon Chung

> 
> 
>>
>>>> I know that this patch mattered for me for UHS:
>>>>
>>>>   7c5209c315ea mmc: core: Increase delay for voltage to stabilize from
>>>> 3.3V to 1.8V
>>>>
>>>>
>>>> Also important for UHS (for at least some folks) were patches like:
>>>>
>>>>   9c85f37a2984 mmc: core: Add mmc_regulator_set_vqmmc()
>>>>
>>>> ...that attempted to get voltages more proper...
>>> ack
>>>
>>>>
>>>> In the ChromeOS tree we did just land treating HLE errors as data and
>>>> cmd errors <https://patchwork.kernel.org/patch/5978711/>.  It's not
>>>> wonderful but it's better than letting an interrupt go off forever...
>>> Yes I did try this patch on 3.18 but it didn't seem to be enough for us.
>>> Even though it would prevent the interrupt storm from flooding the kernel, once
>>> the event triggered and the interrupt was handled no more card
>>> insertions/ejections would be detected.
>> If HLE error will be reproduce with the generic sequence, I think we can find the generic solution.
>> So could you explain to me in more detail? If i can reproduce with v3.18, i will try to test it.
>> Your case will be helpful to me for solving the HLE error.
> 
> 
> Yes, the issue is relatively easy to reproduce.
> 
> On this platform:
> https://www.96boards.org/products/ce/hikey/
> 
> Using either debian [1] or android [2] releases and the latest UEFI [3]
> [1] https://builds.96boards.org/snapshots/hikey/linaro/debian/379/
> [2] https://builds.96boards.org/snapshots/hikey/linaro/aosp/197/
> [3] https://builds.96boards.org/snapshots/hikey/linaro/uefi/89/
> 
> The kernel tree between android and debian is shared [4].
> We are using the "hikey" branch (v3.18)
> [4] https://github.com/96boards/linux
> 
> For my tests and to be able to handed the interrupt storm and monitor the
> registers while it happens, I patched the kernel with a Xenomai [5] co-kernel.
> This is my kernel tree [6]
> [5] http://xenomai.org/
> [6] http://git.xenomai.org/ipipe-jro.git/log/?h=hikey
> 
> To reproduce the problem all it was required was to insert/remove the SD card
> rapidly until it triggers this condition:
> [  229.974525] dwmmc_k3 f723e000.dwmmc1: Busy; trying anyway
>  
> When it triggered, and after patching the interrupt handler with some debug info
> to show the distance between interrupts and the content of the MINTSTS register,
> I could see the following:
> mci_isr: 0x1000,  3333 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  3333 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  3333 ns
> mci_isr: 0x1000,  2500 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  2500 ns
> mci_isr: 0x1000,  3333 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  3333 ns
> mci_isr: 0x1000,  3334 ns
> mci_isr: 0x1000,  2500 ns
> mci_isr: 0x1000,  3334 ns
> [...]
> 
> Notice that since the Xenomai co-kernel runs with a higher priority than the
> Linux kernel, I was able to output this information to the console.
> 
> I put together a fix based on this commit from Doug;
> mmc: dw_mmc: Don't start commands while busy
> https://lkml.org/lkml/2015/2/20/508
> 
> In Doug's commit, we would delay sending a command until the SDMCC_STATUS_BUSY
> cleared.
> However if it never cleared, we'd go ahead and submit the command anyway.
> 
> I believe this is what was causing the HLE to be raised.
> In order to prevent that from happening, I think we should abort the operation
> completely.
> My "extension" for the Hikey platform looks like this:
> https://github.com/96boards/linux/commit/fe8d7f714d420121cec460e69f6529044a2cb6d
> 
> It could be made generic or the fix could have some other form of course.
> I was only targeting the Hikey platform when I wrote this hoping that it would
> have been fixed upstream.
> 
> Having said all of this, I am not sure what would cause the host status to
> remain busy for so long (which is Ulf's biggest concern)
> I also tried increasing some of the timers that wait for the voltages to ramp up
> after power on but it didnt make any difference.
> 
> I captured most of the information above under this bug for reference.
> https://bugs.96boards.org/show_bug.cgi?id=175
> 
> 
> 
> 
>