obscure microsd detection issue between U-Boot and kernel

obscure microsd detection issue between U-Boot and kernel

[Tim Harvey]

Greetings,

I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
for a specific set of microsd cards if I have accessed the microsd in
U-Boot with UHS/1.8V the kernel will not recognize that microsd when
scanning.

The issue does not occur with all microsd cards but seems to appear
with a large sample size of a specific card/model (Kingston SDC32 32GB
SDR104 card). I do not see a signal integrity issue on the scope.

Instrumenting the kernel the issue is that the host reports a CRC
error as soon as the first mmc_send_if_cond call which occurs in
mmc_rescan_try_freq.

I can avoid the issue by either not accessing the microsd in U-Boot or
by disabling UHS/1.8V mode in U-Boot therefore what I think is
happening is that U-Boot leaves the card in UHS/1.8V signalling mode
and when the kernel scans it sets the voltage back to 3.3V
standard/default and default timings then issues its clock cycles to
'reset' the card and the card does not recognize the reset. I'm
wondering if this is because the reset is done via clock cycles after
the kernel has set the I/O voltage back to 3.3V when perhaps the card
is still in 1.8V mode (although I don't see how that would cause an
issue)?

Is there some sort of MMC 'reset' I can/should do in U-Boot before
booting the kernel? Has anyone encountered anything like this before?

Best Regards,

Tim

Re: obscure microsd detection issue between U-Boot and kernel

[Christian Loehle]

On 5/31/24 21:47, Tim Harvey wrote:
> Greetings,
> 
> I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
> for a specific set of microsd cards if I have accessed the microsd in
> U-Boot with UHS/1.8V the kernel will not recognize that microsd when
> scanning.
> 
> The issue does not occur with all microsd cards but seems to appear
> with a large sample size of a specific card/model (Kingston SDC32 32GB
> SDR104 card). I do not see a signal integrity issue on the scope.
> 
> Instrumenting the kernel the issue is that the host reports a CRC
> error as soon as the first mmc_send_if_cond call which occurs in
> mmc_rescan_try_freq.
> 
> I can avoid the issue by either not accessing the microsd in U-Boot or
> by disabling UHS/1.8V mode in U-Boot therefore what I think is
> happening is that U-Boot leaves the card in UHS/1.8V signalling mode
> and when the kernel scans it sets the voltage back to 3.3V
> standard/default and default timings then issues its clock cycles to
> 'reset' the card and the card does not recognize the reset. I'm
> wondering if this is because the reset is done via clock cycles after
> the kernel has set the I/O voltage back to 3.3V when perhaps the card
> is still in 1.8V mode (although I don't see how that would cause an
> issue)?

It will cause an issue for many cards and might break some cards.

> 
> Is there some sort of MMC 'reset' I can/should do in U-Boot before
> booting the kernel? Has anyone encountered anything like this before?

There is no 'switching back' to 3.3V signalling from UHS 1.8V.
The only way this can be done is therefore a full power-off.
Is that done correctly for your system?
AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
lines can also sustain the card somewhat, as leakage is only limited
within operating voltage.

Re: obscure microsd detection issue between U-Boot and kernel

[Tim Harvey]

On Mon, Jun 3, 2024 at 1:18 AM Christian Loehle
<christian.loehle@arm.com> wrote:
>
> On 5/31/24 21:47, Tim Harvey wrote:
> > Greetings,
> >
> > I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
> > for a specific set of microsd cards if I have accessed the microsd in
> > U-Boot with UHS/1.8V the kernel will not recognize that microsd when
> > scanning.
> >
> > The issue does not occur with all microsd cards but seems to appear
> > with a large sample size of a specific card/model (Kingston SDC32 32GB
> > SDR104 card). I do not see a signal integrity issue on the scope.
> >
> > Instrumenting the kernel the issue is that the host reports a CRC
> > error as soon as the first mmc_send_if_cond call which occurs in
> > mmc_rescan_try_freq.
> >
> > I can avoid the issue by either not accessing the microsd in U-Boot or
> > by disabling UHS/1.8V mode in U-Boot therefore what I think is
> > happening is that U-Boot leaves the card in UHS/1.8V signalling mode
> > and when the kernel scans it sets the voltage back to 3.3V
> > standard/default and default timings then issues its clock cycles to
> > 'reset' the card and the card does not recognize the reset. I'm
> > wondering if this is because the reset is done via clock cycles after
> > the kernel has set the I/O voltage back to 3.3V when perhaps the card
> > is still in 1.8V mode (although I don't see how that would cause an
> > issue)?
>
> It will cause an issue for many cards and might break some cards.
>
> >
> > Is there some sort of MMC 'reset' I can/should do in U-Boot before
> > booting the kernel? Has anyone encountered anything like this before?
>
> There is no 'switching back' to 3.3V signalling from UHS 1.8V.
> The only way this can be done is therefore a full power-off.
> Is that done correctly for your system?
> AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
> lines can also sustain the card somewhat, as leakage is only limited
> within operating voltage.

Hi Christian,

Are you saying the only way to properly reset from 1.8V is to have a
VDD supply on the microSD card that can be turned off before booting
to Linux? We have never had that before and never encountered
something like this.

Best Regards,

Tim

Re: obscure microsd detection issue between U-Boot and kernel

[Christian Loehle]

On 6/3/24 22:28, Tim Harvey wrote:
> On Mon, Jun 3, 2024 at 1:18 AM Christian Loehle
> <christian.loehle@arm.com> wrote:
>>
>> On 5/31/24 21:47, Tim Harvey wrote:
>>> Greetings,
>>>
>>> I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
>>> for a specific set of microsd cards if I have accessed the microsd in
>>> U-Boot with UHS/1.8V the kernel will not recognize that microsd when
>>> scanning.
>>>
>>> The issue does not occur with all microsd cards but seems to appear
>>> with a large sample size of a specific card/model (Kingston SDC32 32GB
>>> SDR104 card). I do not see a signal integrity issue on the scope.
>>>
>>> Instrumenting the kernel the issue is that the host reports a CRC
>>> error as soon as the first mmc_send_if_cond call which occurs in
>>> mmc_rescan_try_freq.
>>>
>>> I can avoid the issue by either not accessing the microsd in U-Boot or
>>> by disabling UHS/1.8V mode in U-Boot therefore what I think is
>>> happening is that U-Boot leaves the card in UHS/1.8V signalling mode
>>> and when the kernel scans it sets the voltage back to 3.3V
>>> standard/default and default timings then issues its clock cycles to
>>> 'reset' the card and the card does not recognize the reset. I'm
>>> wondering if this is because the reset is done via clock cycles after
>>> the kernel has set the I/O voltage back to 3.3V when perhaps the card
>>> is still in 1.8V mode (although I don't see how that would cause an
>>> issue)?
>>
>> It will cause an issue for many cards and might break some cards.
>>
>>>
>>> Is there some sort of MMC 'reset' I can/should do in U-Boot before
>>> booting the kernel? Has anyone encountered anything like this before?
>>
>> There is no 'switching back' to 3.3V signalling from UHS 1.8V.
>> The only way this can be done is therefore a full power-off.
>> Is that done correctly for your system?
>> AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
>> lines can also sustain the card somewhat, as leakage is only limited
>> within operating voltage.
> 
> Hi Christian,
> 
> Are you saying the only way to properly reset from 1.8V is to have a
> VDD supply on the microSD card that can be turned off before booting
> to Linux? We have never had that before and never encountered
> something like this.

Yes, the only safe way to use UHS-I really anyway.
You could disable UHS for u-boot but that still leaves (potentially)
problematic warm-reboots of the board.
Having a (software-controlled) switchable regulator on SD VCC is pretty
common for this reason and you should be able to find it in most dts
for host controllers with sd-uhs-* property.
I'm afraid that the relevant spec section isn't available in the
simplified version.

Kind Regards,
Christian

Re: obscure microsd detection issue between U-Boot and kernel

[Michael Walle]

[-- Attachment #1: Type: text/plain, Size: 3476 bytes --]

On Tue Jun 4, 2024 at 9:47 AM CEST, Christian Loehle wrote:
> On 6/3/24 22:28, Tim Harvey wrote:
> > On Mon, Jun 3, 2024 at 1:18 AM Christian Loehle
> > <christian.loehle@arm.com> wrote:
> >>
> >> On 5/31/24 21:47, Tim Harvey wrote:
> >>> Greetings,
> >>>
> >>> I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
> >>> for a specific set of microsd cards if I have accessed the microsd in
> >>> U-Boot with UHS/1.8V the kernel will not recognize that microsd when
> >>> scanning.
> >>>
> >>> The issue does not occur with all microsd cards but seems to appear
> >>> with a large sample size of a specific card/model (Kingston SDC32 32GB
> >>> SDR104 card). I do not see a signal integrity issue on the scope.
> >>>
> >>> Instrumenting the kernel the issue is that the host reports a CRC
> >>> error as soon as the first mmc_send_if_cond call which occurs in
> >>> mmc_rescan_try_freq.
> >>>
> >>> I can avoid the issue by either not accessing the microsd in U-Boot or
> >>> by disabling UHS/1.8V mode in U-Boot therefore what I think is
> >>> happening is that U-Boot leaves the card in UHS/1.8V signalling mode
> >>> and when the kernel scans it sets the voltage back to 3.3V
> >>> standard/default and default timings then issues its clock cycles to
> >>> 'reset' the card and the card does not recognize the reset. I'm
> >>> wondering if this is because the reset is done via clock cycles after
> >>> the kernel has set the I/O voltage back to 3.3V when perhaps the card
> >>> is still in 1.8V mode (although I don't see how that would cause an
> >>> issue)?
> >>
> >> It will cause an issue for many cards and might break some cards.
> >>
> >>>
> >>> Is there some sort of MMC 'reset' I can/should do in U-Boot before
> >>> booting the kernel? Has anyone encountered anything like this before?
> >>
> >> There is no 'switching back' to 3.3V signalling from UHS 1.8V.
> >> The only way this can be done is therefore a full power-off.
> >> Is that done correctly for your system?
> >> AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
> >> lines can also sustain the card somewhat, as leakage is only limited
> >> within operating voltage.
> > 
> > Hi Christian,
> > 
> > Are you saying the only way to properly reset from 1.8V is to have a
> > VDD supply on the microSD card that can be turned off before booting
> > to Linux? We have never had that before and never encountered
> > something like this.
>
> Yes, the only safe way to use UHS-I really anyway.

I can second that. And also keep in mind that the actual supply
voltage must be below that threshold. Thus, the power-off time also
depends on the capacitance on that supply line after the power load
switch. There are switches with dedicated output discharge circuits
built-in.

That being said, from my experience there are cards which will work
when switching back from 1V8 to 3V3 signalling and some don't. I
haven't digged deeper into that topic, though.

-michael

> You could disable UHS for u-boot but that still leaves (potentially)
> problematic warm-reboots of the board.
> Having a (software-controlled) switchable regulator on SD VCC is pretty
> common for this reason and you should be able to find it in most dts
> for host controllers with sd-uhs-* property.
> I'm afraid that the relevant spec section isn't available in the
> simplified version.
>
> Kind Regards,
> Christian


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 297 bytes --]

Re: obscure microsd detection issue between U-Boot and kernel

[Tim Harvey]

On Tue, Jun 4, 2024 at 1:14 AM Michael Walle <michael@walle.cc> wrote:
>
> On Tue Jun 4, 2024 at 9:47 AM CEST, Christian Loehle wrote:
> > On 6/3/24 22:28, Tim Harvey wrote:
> > > On Mon, Jun 3, 2024 at 1:18 AM Christian Loehle
> > > <christian.loehle@arm.com> wrote:
> > >>
> > >> On 5/31/24 21:47, Tim Harvey wrote:
> > >>> Greetings,
> > >>>
> > >>> I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
> > >>> for a specific set of microsd cards if I have accessed the microsd in
> > >>> U-Boot with UHS/1.8V the kernel will not recognize that microsd when
> > >>> scanning.
> > >>>
> > >>> The issue does not occur with all microsd cards but seems to appear
> > >>> with a large sample size of a specific card/model (Kingston SDC32 32GB
> > >>> SDR104 card). I do not see a signal integrity issue on the scope.
> > >>>
> > >>> Instrumenting the kernel the issue is that the host reports a CRC
> > >>> error as soon as the first mmc_send_if_cond call which occurs in
> > >>> mmc_rescan_try_freq.
> > >>>
> > >>> I can avoid the issue by either not accessing the microsd in U-Boot or
> > >>> by disabling UHS/1.8V mode in U-Boot therefore what I think is
> > >>> happening is that U-Boot leaves the card in UHS/1.8V signalling mode
> > >>> and when the kernel scans it sets the voltage back to 3.3V
> > >>> standard/default and default timings then issues its clock cycles to
> > >>> 'reset' the card and the card does not recognize the reset. I'm
> > >>> wondering if this is because the reset is done via clock cycles after
> > >>> the kernel has set the I/O voltage back to 3.3V when perhaps the card
> > >>> is still in 1.8V mode (although I don't see how that would cause an
> > >>> issue)?
> > >>
> > >> It will cause an issue for many cards and might break some cards.
> > >>
> > >>>
> > >>> Is there some sort of MMC 'reset' I can/should do in U-Boot before
> > >>> booting the kernel? Has anyone encountered anything like this before?
> > >>
> > >> There is no 'switching back' to 3.3V signalling from UHS 1.8V.
> > >> The only way this can be done is therefore a full power-off.
> > >> Is that done correctly for your system?
> > >> AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
> > >> lines can also sustain the card somewhat, as leakage is only limited
> > >> within operating voltage.
> > >
> > > Hi Christian,
> > >
> > > Are you saying the only way to properly reset from 1.8V is to have a
> > > VDD supply on the microSD card that can be turned off before booting
> > > to Linux? We have never had that before and never encountered
> > > something like this.
> >
> > Yes, the only safe way to use UHS-I really anyway.
>
> I can second that. And also keep in mind that the actual supply
> voltage must be below that threshold. Thus, the power-off time also
> depends on the capacitance on that supply line after the power load
> switch. There are switches with dedicated output discharge circuits
> built-in.
>
> That being said, from my experience there are cards which will work
> when switching back from 1V8 to 3V3 signalling and some don't. I
> haven't digged deeper into that topic, though.
>
> -michael
>
> > You could disable UHS for u-boot but that still leaves (potentially)
> > problematic warm-reboots of the board.
> > Having a (software-controlled) switchable regulator on SD VCC is pretty
> > common for this reason and you should be able to find it in most dts
> > for host controllers with sd-uhs-* property.
> > I'm afraid that the relevant spec section isn't available in the
> > simplified version.
> >
> > Kind Regards,
> > Christian
>

Thanks for both of your responses here!

I have a situation where I can re-create a problem switching from 1.8V
back to 3.3V with a specific card on a board that has ESD protection
between the IMX8MM host and the microSD connector (Semtech
ECLAMP2410P.TCT [1]) but works just fine on a previous revision of
that same PCB that does not have the ESD protection added. Boards with
proper ESD protection are the first time I've seen this issue occur.
Does this make sense at all?

I believe that the MMC device is 'reset' via a series of CLK cycles
with CMD/DAT non-asserted so I'm having a difficult time understanding
how this wouldn't work if the host has been reset to 3.3V.

Best Regards,

Tim
[1] https://www.semtech.com/products/circuit-protection/esd-emi-filter-devices/eclamp2410p

Re: obscure microsd detection issue between U-Boot and kernel

[Christian Loehle]

On 6/20/24 17:48, Tim Harvey wrote:
> On Tue, Jun 4, 2024 at 1:14 AM Michael Walle <michael@walle.cc> wrote:
>>
>> On Tue Jun 4, 2024 at 9:47 AM CEST, Christian Loehle wrote:
>>> On 6/3/24 22:28, Tim Harvey wrote:
>>>> On Mon, Jun 3, 2024 at 1:18 AM Christian Loehle
>>>> <christian.loehle@arm.com> wrote:
>>>>>
>>>>> On 5/31/24 21:47, Tim Harvey wrote:
>>>>>> Greetings,
>>>>>>
>>>>>> I'm seeing an issue on an imx8mm board (imx8mm-venice-gw73xx) where
>>>>>> for a specific set of microsd cards if I have accessed the microsd in
>>>>>> U-Boot with UHS/1.8V the kernel will not recognize that microsd when
>>>>>> scanning.
>>>>>>
>>>>>> The issue does not occur with all microsd cards but seems to appear
>>>>>> with a large sample size of a specific card/model (Kingston SDC32 32GB
>>>>>> SDR104 card). I do not see a signal integrity issue on the scope.
>>>>>>
>>>>>> Instrumenting the kernel the issue is that the host reports a CRC
>>>>>> error as soon as the first mmc_send_if_cond call which occurs in
>>>>>> mmc_rescan_try_freq.
>>>>>>
>>>>>> I can avoid the issue by either not accessing the microsd in U-Boot or
>>>>>> by disabling UHS/1.8V mode in U-Boot therefore what I think is
>>>>>> happening is that U-Boot leaves the card in UHS/1.8V signalling mode
>>>>>> and when the kernel scans it sets the voltage back to 3.3V
>>>>>> standard/default and default timings then issues its clock cycles to
>>>>>> 'reset' the card and the card does not recognize the reset. I'm
>>>>>> wondering if this is because the reset is done via clock cycles after
>>>>>> the kernel has set the I/O voltage back to 3.3V when perhaps the card
>>>>>> is still in 1.8V mode (although I don't see how that would cause an
>>>>>> issue)?
>>>>>
>>>>> It will cause an issue for many cards and might break some cards.
>>>>>
>>>>>>
>>>>>> Is there some sort of MMC 'reset' I can/should do in U-Boot before
>>>>>> booting the kernel? Has anyone encountered anything like this before?
>>>>>
>>>>> There is no 'switching back' to 3.3V signalling from UHS 1.8V.
>>>>> The only way this can be done is therefore a full power-off.
>>>>> Is that done correctly for your system?
>>>>> AFAIR spec dictates 500ms of <0.5V on VCC. Note that  driving CLK/signal
>>>>> lines can also sustain the card somewhat, as leakage is only limited
>>>>> within operating voltage.
>>>>
>>>> Hi Christian,
>>>>
>>>> Are you saying the only way to properly reset from 1.8V is to have a
>>>> VDD supply on the microSD card that can be turned off before booting
>>>> to Linux? We have never had that before and never encountered
>>>> something like this.
>>>
>>> Yes, the only safe way to use UHS-I really anyway.
>>
>> I can second that. And also keep in mind that the actual supply
>> voltage must be below that threshold. Thus, the power-off time also
>> depends on the capacitance on that supply line after the power load
>> switch. There are switches with dedicated output discharge circuits
>> built-in.
>>
>> That being said, from my experience there are cards which will work
>> when switching back from 1V8 to 3V3 signalling and some don't. I
>> haven't digged deeper into that topic, though.
>>
>> -michael
>>
>>> You could disable UHS for u-boot but that still leaves (potentially)
>>> problematic warm-reboots of the board.
>>> Having a (software-controlled) switchable regulator on SD VCC is pretty
>>> common for this reason and you should be able to find it in most dts
>>> for host controllers with sd-uhs-* property.
>>> I'm afraid that the relevant spec section isn't available in the
>>> simplified version.
>>>
>>> Kind Regards,
>>> Christian
>>
> 
> Thanks for both of your responses here!
> 
> I have a situation where I can re-create a problem switching from 1.8V
> back to 3.3V with a specific card on a board that has ESD protection
> between the IMX8MM host and the microSD connector (Semtech
> ECLAMP2410P.TCT [1]) but works just fine on a previous revision of
> that same PCB that does not have the ESD protection added. Boards with
> proper ESD protection are the first time I've seen this issue occur.
> Does this make sense at all?

I have some hypothesis but that isn't even worth sharing, see below.

> 
> I believe that the MMC device is 'reset' via a series of CLK cycles
> with CMD/DAT non-asserted so I'm having a difficult time understanding
> how this wouldn't work if the host has been reset to 3.3V.

My answer is simple but not very satisfying:
It is out of spec.
After the CMD11 UHS voltage switch anything on CMD, DAT and CLK you
drive from the host at 3.3V without a proper powercycle is out of spec,
the card's behavior isn't covered by the SD spec and the card isn't to
blame.

If you want a really detailed answer try asking your SD card vendor,
my guess would be their answer is "the host is out of spec".

Kind Regards,
Christian