MTK mt8173-evb: mainline unstable, and big-endian boot failures

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Hello,

I've been meaning to report this for awhile, but finally getting around to it.

In kernelci.org, the mt8173-evb board has been rather unstable with
mainline and linux-next lately.  A look a the history of boots in
kernelci.org with various trees[1] shows that it's failing to boot
quite a lot.

It seems to be pretty stable up through v4.4 but
unreliable/unpredictable in v4.5-rc1 and lots of places in between.
It's hard to bisect because it doesn't reliably fail, but when it does
fail to finish booting, it's typically after it reaches a shell, and
trying to run various simple shell commands fails to complete, like
this example from next-20160125[2].

To me this suggests possibly an issue with the idle driver where
characters might be getting lost on the serial console, which causes
the automated boot scripts to fail because the shell is still waiting
for the command to be finished.

Also, a separate issue, I've also noticed that builds with
CONFIG_CPU_BIG_ENDIAN=y have been failing to boot since early in the
merge window[3].  For the big-endian failures, the last lines in the
boot log are always:

[...]
[    0.518078] usbcore: registered new interface driver usbhid
[    0.518780] usbhid: USB HID core driver
[    0.520295] NET: Registered protocol family 17
[    0.520934] 9pnet: Installing 9P2000 support
[    0.521549] Key type dns_resolver registered
[    0.524558] registered taskstats version 1

and in a successfull boot (e.g. little-endian) this is the followed by:

[    0.525458] hctosys: unable to open rtc device (rtc0)
[    0.533128] usb_vbus: disabling
[    0.534117] ALSA device list:
[    0.534497]   No soundcards found.
[    0.535423] Freeing unused kernel memory: 696K (ffffffc0008fa000 -
ffffffc0009a8000)

I have yet to debug this any further, but still wanted to report it in
case anyone else has the time to debug it further.

Thanks,

Kevin

[1] http://kernelci.org/boot/mt8173-evb/
[2] http://storage.kernelci.org/next/next-20160125/arm64-defconfig/lab-khilman/boot-mt8173-evb.html
[3] http://kernelci.org/boot/mt8173-evb/job/mainline/kernel/v4.5-rc1/defconfig/defconfig+CONFIG_CPU_BIG_ENDIAN=y/lab/lab-khilman/?_id=56a5812b59b514bed8fa5dc0

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Kevin Hilman <khilman@baylibre.com> writes:

[...]

> It seems to be pretty stable up through v4.4 but
> unreliable/unpredictable in v4.5-rc1 and lots of places in between.
> It's hard to bisect because it doesn't reliably fail, but when it does
> fail to finish booting, it's typically after it reaches a shell, and
> trying to run various simple shell commands fails to complete, like
> this example from next-20160125[2].
>
> To me this suggests possibly an issue with the idle driver where
> characters might be getting lost on the serial console, which causes
> the automated boot scripts to fail because the shell is still waiting
> for the command to be finished.

FWIW, I think I confirmed my suspicion that it's an idle driver issue
because adding 'nohlt' on the cmdline, which forces cpuidle_poll mode
(essentially disabling cpuidle) makes mainline boot quite reliably on my
mt8173-evb board now.

Kevin

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Eddie Huang]

On Thu, 2016-01-28 at 13:41 +0800, Koan-Sin Tan wrote:
> Hi Kevin, 
> 
> On Thu, Jan 28, 2016 at 9:16 AM, Kevin Hilman <khilman@baylibre.com>
> wrote:
>         Kevin Hilman <khilman@baylibre.com> writes:
>         
>         [...]
>         
>         > It seems to be pretty stable up through v4.4 but
>         > unreliable/unpredictable in v4.5-rc1 and lots of places in
>         between.
>         > It's hard to bisect because it doesn't reliably fail, but
>         when it does
>         > fail to finish booting, it's typically after it reaches a
>         shell, and
>         > trying to run various simple shell commands fails to
>         complete, like
>         > this example from next-20160125[2].
>         >
>         > To me this suggests possibly an issue with the idle driver
>         where
>         > characters might be getting lost on the serial console,
>         which causes
>         > the automated boot scripts to fail because the shell is
>         still waiting
>         > for the command to be finished.
>         
>         FWIW, I think I confirmed my suspicion that it's an idle
>         driver issue
>         because adding 'nohlt' on the cmdline, which forces
>         cpuidle_poll mode
>         (essentially disabling cpuidle) makes mainline boot quite
>         reliably on my
>         mt8173-evb board now.
> 
> 
>    Thanks for the information. I'll try to reproduce it. The problem
> is that I have been using 4.5-rc1 based kernel for several days, but
> didn't run into such problem. Can tell me which version of firmware
> you are using? 

Hi Kevin,

I use firmware [0] to run kernel v4.5-rc1 on my MT8173-evb, both branch
v20150716 (psci-0.2 atf) and v20150902 (psci-1.0 atf) can't reproduce
this problem.

[0]: https://github.com/ibanezchen/evb-utils

Eddie

>   
>    And, YES, I can reproduce the CONFIG_CPU_BIG_ENDIAN=y problem. The
> 'nohlt' command line option didn't stop the problem. We'll investigate
> it.
> 
> 
> -- 
> // freedom

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Hi Eddie, Freedom,

Eddie Huang <eddie.huang@mediatek.com> writes:

> On Thu, 2016-01-28 at 13:41 +0800, Koan-Sin Tan wrote:

[...]

>>    Thanks for the information. I'll try to reproduce it. The problem
>> is that I have been using 4.5-rc1 based kernel for several days, but
>> didn't run into such problem. Can tell me which version of firmware
>> you are using? 
>
> I use firmware [0] to run kernel v4.5-rc1 on my MT8173-evb, both branch
> v20150716 (psci-0.2 atf) and v20150902 (psci-1.0 atf) can't reproduce
> this problem.

I'm running firmware from the master branch of the same repo.  The top
commit is the PSCI 1.0 support.

It's not exactly easy to reproduce the problem, and you it seems you
will not likely see it unless you have lots of input on the serial
console.  Just booting to a shell may not reproduce the problem, you may
need to try to copy/paste a bunch of commands in order to see characters
being dropped.

For example, I'm able to reproduce pretty easily by just booting to a
busybox ramdisk[1], and then copy/pasting a bunch of commands[2] to the
shell prompt.

What I noticed is that sometimes it works, and sometimes it doesn't.
you can tell because you don't get back to a shell problem, but rather
to the shell's ">" prompt telling you it's still waiting for input to
finish.

Ususally it fails for me within 4-5 times of copy/pasting these lines.

If I add nohlt to the command-line I can repeatedly copy/paste and there
is never any loss of characters.

Could you please try this experiment on v4.5-rc1 and let me know if it's
working for you?

Thanks

Kevin

[1] http://storage.kernelci.org/images/rootfs/buildroot/arm64/rootfs.cpio.gz
[2]
dmesg -n 1
DMESG=$(readlink -f /bin/dmesg)
[[ $DMESG = "/bin/dmesg" ]] && dmesg -l warn | awk '{ print "[WARN] " $0}'
[[ $DMESG = "/bin/dmesg" ]] && dmesg -l err | awk '{ print "[ERR] " $0}'
[[ $DMESG = "/bin/dmesg" ]] && dmesg -l alert | awk '{ print "[ERR] " $0}'
[[ -e /var/log/messages ]] && grep kern.warn /var/log/messages | awk '{print "[WARN] " $0 }'
[[ -e /var/log/messages ]] && grep kern.err /var/log/messages | awk '{print "[ERR] " $0 }'
[[ -e /var/log/messages ]] && grep kern.alert /var/log/messages | awk '{print "[ERR] " $0 }'
echo "hello, world"

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Eddie Huang]

Hi Kevin,

On Fri, 2016-01-29 at 16:09 -0800, Kevin Hilman wrote:
> Hi Eddie, Freedom,
> 
> Eddie Huang <eddie.huang@mediatek.com> writes:
> 
> > On Thu, 2016-01-28 at 13:41 +0800, Koan-Sin Tan wrote:
> 
> [...]
> 
> >>    Thanks for the information. I'll try to reproduce it. The problem
> >> is that I have been using 4.5-rc1 based kernel for several days, but
> >> didn't run into such problem. Can tell me which version of firmware
> >> you are using? 
> >
> > I use firmware [0] to run kernel v4.5-rc1 on my MT8173-evb, both branch
> > v20150716 (psci-0.2 atf) and v20150902 (psci-1.0 atf) can't reproduce
> > this problem.
> 
> I'm running firmware from the master branch of the same repo.  The top
> commit is the PSCI 1.0 support.
> 
> It's not exactly easy to reproduce the problem, and you it seems you
> will not likely see it unless you have lots of input on the serial
> console.  Just booting to a shell may not reproduce the problem, you may
> need to try to copy/paste a bunch of commands in order to see characters
> being dropped.
> 
> For example, I'm able to reproduce pretty easily by just booting to a
> busybox ramdisk[1], and then copy/pasting a bunch of commands[2] to the
> shell prompt.
> 
> What I noticed is that sometimes it works, and sometimes it doesn't.
> you can tell because you don't get back to a shell problem, but rather
> to the shell's ">" prompt telling you it's still waiting for input to
> finish.
> 
> Ususally it fails for me within 4-5 times of copy/pasting these lines.
> 
> If I add nohlt to the command-line I can repeatedly copy/paste and there
> is never any loss of characters.
> 
> Could you please try this experiment on v4.5-rc1 and let me know if it's
> working for you?
> 
> Thanks
> 
> Kevin
> 
> [1] http://storage.kernelci.org/images/rootfs/buildroot/arm64/rootfs.cpio.gz
> [2]
> dmesg -n 1
> DMESG=$(readlink -f /bin/dmesg)
> [[ $DMESG = "/bin/dmesg" ]] && dmesg -l warn | awk '{ print "[WARN] " $0}'
> [[ $DMESG = "/bin/dmesg" ]] && dmesg -l err | awk '{ print "[ERR] " $0}'
> [[ $DMESG = "/bin/dmesg" ]] && dmesg -l alert | awk '{ print "[ERR] " $0}'
> [[ -e /var/log/messages ]] && grep kern.warn /var/log/messages | awk '{print "[WARN] " $0 }'
> [[ -e /var/log/messages ]] && grep kern.err /var/log/messages | awk '{print "[ERR] " $0 }'
> [[ -e /var/log/messages ]] && grep kern.alert /var/log/messages | awk '{print "[ERR] " $0 }'
> echo "hello, world"
> 

Thanks your patience to provide detail steps, we can reproduce this
problem now. It seems something wrong in ATF, we still investigate how
to fix it. We will provide you new ATF firmware once we have solution

Regards,
Eddie

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Hi Eddie,

Eddie Huang <eddie.huang@mediatek.com> writes:

[...]

> Thanks your patience to provide detail steps, we can reproduce this
> problem now. It seems something wrong in ATF, we still investigate how
> to fix it. We will provide you new ATF firmware once we have solution

Great, I'm glad you can reproduce it.  I was starting to worry it was
only my hardware with the problem.

Please let me know if there are firmware updates you'd like me to test
and I'll be happy to.

Thanks,

Kevin

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Fan Chen]

Hi Kevin,

I just checked that the problem cannot be reproduced (tested 100 times
with your script) after flashing new firmware which we uploaded to
github[0] yesterday.
This latest firmware got many fixes and also supports cluster idle, so
we prefer testing with this new one
Would you please test in your side? Thanks for your great help!! 

[0] https://github.com/ibanezchen/evb-utils/tree/master


Best regards,
Fan

On Mon, 2016-02-01 at 17:17 -0800, Kevin Hilman wrote:
> Hi Eddie,
> 
> Eddie Huang <eddie.huang@mediatek.com> writes:
> 
> [...]
> 
> > Thanks your patience to provide detail steps, we can reproduce this
> > problem now. It seems something wrong in ATF, we still investigate how
> > to fix it. We will provide you new ATF firmware once we have solution
> 
> Great, I'm glad you can reproduce it.  I was starting to worry it was
> only my hardware with the problem.
> 
> Please let me know if there are firmware updates you'd like me to test
> and I'll be happy to.
> 
> Thanks,
> 
> Kevin

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Hi Fan,

Fan Chen <fan.chen@mediatek.com> writes:

> I just checked that the problem cannot be reproduced (tested 100 times
> with your script) after flashing new firmware which we uploaded to
> github[0] yesterday.

Great news!

> This latest firmware got many fixes and also supports cluster idle, so
> we prefer testing with this new one Would you please test in your
> side? Thanks for your great help!!

I tested the latest firmware and it's booted 20+ times in a row without
failure, which is better than it ever did in the past, so it seems to
have fixed the idle problem.

I will keep an eye on the failures in kernelci.org and let you know if I
notice any other failures.

Thanks again for the updated firmware!

Speaking of updated firmware, is there any chance to make the default
version of the firmware significantly less noisy.  There is a TON of
output from the firmware before fastboot starts which makes the logs for
this board very noisy (same complaint on the mt8135.)

Kevin

>
>
> Best regards,
> Fan
>
> On Mon, 2016-02-01 at 17:17 -0800, Kevin Hilman wrote:
>> Hi Eddie,
>> 
>> Eddie Huang <eddie.huang@mediatek.com> writes:
>> 
>> [...]
>> 
>> > Thanks your patience to provide detail steps, we can reproduce this
>> > problem now. It seems something wrong in ATF, we still investigate how
>> > to fix it. We will provide you new ATF firmware once we have solution
>> 
>> Great, I'm glad you can reproduce it.  I was starting to worry it was
>> only my hardware with the problem.
>> 
>> Please let me know if there are firmware updates you'd like me to test
>> and I'll be happy to.
>> 
>> Thanks,
>> 
>> Kevin

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Fan Chen]

Hi Kevin,

On Tue, 2016-02-02 at 10:46 -0800, Kevin Hilman wrote:
> Hi Fan,
> 
> Fan Chen <fan.chen@mediatek.com> writes:
> 
> > I just checked that the problem cannot be reproduced (tested 100 times
> > with your script) after flashing new firmware which we uploaded to
> > github[0] yesterday.
> 
> Great news!
> 
> > This latest firmware got many fixes and also supports cluster idle, so
> > we prefer testing with this new one Would you please test in your
> > side? Thanks for your great help!!
> 
> I tested the latest firmware and it's booted 20+ times in a row without
> failure, which is better than it ever did in the past, so it seems to
> have fixed the idle problem.
> 
> I will keep an eye on the failures in kernelci.org and let you know if I
> notice any other failures.
> 
> Thanks again for the updated firmware!
> 
Thanks to keep notice us failures as well.

> Speaking of updated firmware, is there any chance to make the default
> version of the firmware significantly less noisy.  There is a TON of
> output from the firmware before fastboot starts which makes the logs for
> this board very noisy (same complaint on the mt8135.)
> 
I will discuss with Eddie if it's possible to release a version with
light logs, but I'm afraid the answer could be given after our Chinese
New Year vacation. :P
 

> Kevin
> 

Fan

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Kevin Hilman]

Hi Fan,

On Tue, Feb 2, 2016 at 5:38 PM, Fan Chen <fan.chen@mediatek.com> wrote:

>> Speaking of updated firmware, is there any chance to make the default
>> version of the firmware significantly less noisy.  There is a TON of
>> output from the firmware before fastboot starts which makes the logs for
>> this board very noisy (same complaint on the mt8135.)
>>
> I will discuss with Eddie if it's possible to release a version with
> light logs, but I'm afraid the answer could be given after our Chinese
> New Year vacation. :P

Thank you, it's not a big rush on that feature request, but it will be
helpful whenever you can get to it.  Thanks for looking into it.  I
hope you enjoy your  vacation.

Gu?ni?n h?o  :)

Kevin

MTK mt8173-evb: mainline unstable, and big-endian boot failures

[Johannes Berg]

On Thu, 2016-01-28 at 17:31 +0800, Koan-Sin Tan wrote:
>?
> > I have yet to debug this any further, but still wanted to report it
> > in case anyone else has the time to debug it further.
> bisect showed that the culprit of the big endian problem is a
> "regmap-mmio: Use native endianness for read/write"
> patch [1]. I saw Johannes already proposed to revert it [2].
> 

Mark has real fixes for it instead:
https://git.kernel.org/cgit/linux/kernel/git/broonie/regmap.git/log/?h=fix/mmio

johannes