exynos4412: misc issues on Hardkernel Odroid boards

exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello,

I've collected some of the more serious remaining problems on the
Hardkernel Odroid boards (here on an X2).

1) When using cpufreq-exynos and having selected the 'ondemand'
governor, the system shutdown/reboot process doesn't complete and hangs
before the end. I'm currently hotfixing this by switching to
'performance' governor in a custom initscript.

2) When reading the 'clk_summary' debugfs entry the system locks up
immediately. Currently hotfixing this by having the ISP powerdomain
always-on. The problem seems to be known, at least it was mentioned here:
http://www.spinics.net/lists/linux-samsung-soc/msg39488.html

3) Maybe related to (1). I'm experiencing high SoC temperatures after
shutting the system down and leaving the AC connector plugged into the
board. The shutdown procedure is like mentioned in (1): Switch to
'performance' and then do a normal 'shutdown -h now'. I was wondering if
the Odroid boards need a special shutdown procedure (like designed in
the odroid-restart handler patch) after all.

4) Spinlock BUGs triggered by the sdhci subsystem (so for the people
using the system with a SD card). This is also a known problem, I think
first mentioned here: http://www.spinics.net/lists/linux-mmc/msg27277.html
Currently fixing this with this patch:
https://github.com/tobiasjakobi/linux-odroid/commit/abc749843dd7022d01322dca3db0181211a30cd8


With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Krzysztof Kozlowski]

On Feb 7, 2015 9:27 PM, "Tobias Jakobi" <tjakobi@math.uni-bielefeld.de> wrote:
>
> Hello,
>
> I've collected some of the more serious remaining problems on the
> Hardkernel Odroid boards (here on an X2).
>
> 1) When using cpufreq-exynos and having selected the 'ondemand'
> governor, the system shutdown/reboot process doesn't complete and hangs
> before the end. I'm currently hotfixing this by switching to
> 'performance' governor in a custom initscript.
>
> 2) When reading the 'clk_summary' debugfs entry the system locks up
> immediately. Currently hotfixing this by having the ISP powerdomain
> always-on. The problem seems to be known, at least it was mentioned here:
> http://www.spinics.net/lists/linux-samsung-soc/msg39488.html
>
> 3) Maybe related to (1). I'm experiencing high SoC temperatures after
> shutting the system down and leaving the AC connector plugged into the
> board. The shutdown procedure is like mentioned in (1): Switch to
> 'performance' and then do a normal 'shutdown -h now'. I was wondering if
> the Odroid boards need a special shutdown procedure (like designed in
> the odroid-restart handler patch) after all.
>
> 4) Spinlock BUGs triggered by the sdhci subsystem (so for the people
> using the system with a SD card). This is also a known problem, I think
> first mentioned here: http://www.spinics.net/lists/linux-mmc/msg27277.html
> Currently fixing this with this patch:
> https://github.com/tobiasjakobi/linux-odroid/commit/abc749843dd7022d01322dca3db0181211a30cd8

The fourth problem looks similar to:
https://lkml.org/lkml/2015/2/4/63
Could you give that patch a try?

Best regards,
Krzysztof

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Marek Szyprowski]

Hello,

On 2015-02-08 16:00, Tobias Jakobi wrote:
> Hello,
>
> I've collected some of the more serious remaining problems on the
> Hardkernel Odroid boards (here on an X2).
>
> 1) When using cpufreq-exynos and having selected the 'ondemand'
> governor, the system shutdown/reboot process doesn't complete and hangs
> before the end. I'm currently hotfixing this by switching to
> 'performance' governor in a custom initscript.

I didn't observe any issues with 'ondemand' governor. I've just tested it
on next-20150204 with "[PATCH v2 0/3] Add support for hardware reset of eMMC
card on reboot" patches added
(http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot works 
fine.

> 2) When reading the 'clk_summary' debugfs entry the system locks up
> immediately. Currently hotfixing this by having the ISP powerdomain
> always-on. The problem seems to be known, at least it was mentioned here:
> http://www.spinics.net/lists/linux-samsung-soc/msg39488.html

This is known issue. To properly solve it, we need to rewrite exynos4 clock
driver and integrate power domain support for handling of ISP clocks.
Sylwester can provide more details.

> 3) Maybe related to (1). I'm experiencing high SoC temperatures after
> shutting the system down and leaving the AC connector plugged into the
> board. The shutdown procedure is like mentioned in (1): Switch to
> 'performance' and then do a normal 'shutdown -h now'. I was wondering if
> the Odroid boards need a special shutdown procedure (like designed in
> the odroid-restart handler patch) after all.

Right. That's because CPU is busy looping instead of proper shutdown. 
I've just
posted a patch for solving this issue on all Exynos SoCs, see "[PATCH] 
exynos:
pmu: use PS_HOLD based poweroff for all supported SoCs" thread.

> 4) Spinlock BUGs triggered by the sdhci subsystem (so for the people
> using the system with a SD card). This is also a known problem, I think
> first mentioned here: http://www.spinics.net/lists/linux-mmc/msg27277.html
> Currently fixing this with this patch:
> https://github.com/tobiasjakobi/linux-odroid/commit/abc749843dd7022d01322dca3db0181211a30cd8

The fix for this issue has been queued to mmc-next:
https://git.linaro.org/people/ulf.hansson/mmc.git/commit/017210d1c0dc2e2d3b142985cb31d90b98dc0f0f

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello!

Marek Szyprowski wrote:
> I didn't observe any issues with 'ondemand' governor. I've just tested it
> on next-20150204 with "[PATCH v2 0/3] Add support for hardware reset of
> eMMC
> card on reboot" patches added
> (http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot works
> fine.
Since I'm not using an eMMC card, I don't think the pwrseq stuff is
going to make a difference here.
Have you checked this with the PS_HOLD patch applied, or without?
Anyway, guess I'm going to recheck this, maybe with some other governors
too.


> This is known issue. To properly solve it, we need to rewrite exynos4 clock
> driver and integrate power domain support for handling of ISP clocks.
> Sylwester can provide more details.
Oh, that sounds like quite some work. Looking forward to hear from
Sylwester about this.


> Right. That's because CPU is busy looping instead of proper shutdown.
> I've just
> posted a patch for solving this issue on all Exynos SoCs, see "[PATCH]
> exynos:
> pmu: use PS_HOLD based poweroff for all supported SoCs" thread.
Like stated in my other mail, this got fixed by your patch. Thanks!


> The fix for this issue has been queued to mmc-next:
> https://git.linaro.org/people/ulf.hansson/mmc.git/commit/017210d1c0dc2e2d3b142985cb31d90b98dc0f0f
Have removed my hotfix and applied this one. Haven't encountered and
spinlock BUGs yet, but I'll keep on testing.


With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello Marek,

sorry for the late response!


On 2015-02-10 12:38, Marek Szyprowski wrote:
> Hello,
> 
> On 2015-02-09 21:28, Tobias Jakobi wrote:
>> Hello!
>> 
>> Marek Szyprowski wrote:
>>> I didn't observe any issues with 'ondemand' governor. I've just 
>>> tested it
>>> on next-20150204 with "[PATCH v2 0/3] Add support for hardware reset 
>>> of
>>> eMMC
>>> card on reboot" patches added
>>> (http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot 
>>> works
>>> fine.
>> Since I'm not using an eMMC card, I don't think the pwrseq stuff is
>> going to make a difference here.
>> Have you checked this with the PS_HOLD patch applied, or without?
>> Anyway, guess I'm going to recheck this, maybe with some other 
>> governors
>> too.
> 
> PS_HOLD patch is related to shutdown behavior. Reboot doesn't use it. 
> Could you
> point which exactly version did you use and provide the .config which 
> causes
> problems with normal reboot when system runs from SD card? For eMMC you 
> would
> need patches from last linux-next.

I'm currently using this tree:
https://github.com/tobiasjakobi/linux-odroid

In this is the config:
https://raw.githubusercontent.com/tobiasjakobi/odroid-environment/master/sourcecode/system/vanilla-3.19-debug.conf

Upon issuing reboot I get this here on the UART:
chidori ~ # reboot

Broadcast message from root@chidori (ttySAC1) (Mon Feb 23 17:04:44 
2015):

The system is going down for reboot NOW!
INIT: Switching to runlevel: 6
INIT: Sending processes the TERM signal
chidori ~ #  * Stopping local                                            
  [ ok ]
  * Stopping vixie-cron ...                                               
  [ ok ]
  * Saving random seed ...                                                
  [ ok ]
  * Deactivating additional swap space ...                                
  [ ok ]
  * Stopping sshd ...                                                     
  [ ok ]
  * Unmounting network filesystems ...                                    
  [ ok ]
  * Stopping chronyd ...                                                  
  [ ok ]
  * Stopping BlueTooth NAP ...                                            
  [ ok ]
  * Stopping dnsmasq ...                                                  
  [ ok ]
  * Stopping syslog-ng ...                                                
  [ ok ]
  * Shutting down bluetooth ...                                           
  [ ok ]
  * Stopping D-BUS system messagebus ...                                  
  [ ok ]
  * Unmounting loop devices
  * Unmounting filesystems
  *   Unmounting /usr/local/emu ...                                       
  [ ok ]
  *   Unmounting /home ...                                                
  [ ok ]
  *   Unmounting /var ...                                                 
  [ ok ]
  * Deactivating swap devices ...                                         
  [ ok ]
  * Stopping udev ...                                                     
  [ ok ]
  * Saving the shutdown time ...                                          
  [ ok ]
  * Terminating remaining processes ...                                   
  [ ok ]
  * Killing remaining processes ...                                       
  [ ok ]
  * Saving dependency cache ...                                           
  [ ok ]
  * Remounting remaining filesystems read-only ...
  *   Remounting / read only ...
[  113.526626] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)             
  [ ok ]
[  115.609592] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[


Notice the open bracket in the last line. Seems like there is still 
output in the kernel buffer, but the UART is maybe switched off before 
one can see that. At least there is nothing after that, and I have to 
manually restart the board then.

With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Marek Szyprowski]

Hello,

On 2015-02-24 14:39, Tobias Jakobi wrote:
> Hello Marek,
>
> sorry for the late response!
>
>
> On 2015-02-10 12:38, Marek Szyprowski wrote:
>> Hello,
>>
>> On 2015-02-09 21:28, Tobias Jakobi wrote:
>>> Hello!
>>>
>>> Marek Szyprowski wrote:
>>>> I didn't observe any issues with 'ondemand' governor. I've just 
>>>> tested it
>>>> on next-20150204 with "[PATCH v2 0/3] Add support for hardware 
>>>> reset of
>>>> eMMC
>>>> card on reboot" patches added
>>>> (http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot 
>>>> works
>>>> fine.
>>> Since I'm not using an eMMC card, I don't think the pwrseq stuff is
>>> going to make a difference here.
>>> Have you checked this with the PS_HOLD patch applied, or without?
>>> Anyway, guess I'm going to recheck this, maybe with some other 
>>> governors
>>> too.
>>
>> PS_HOLD patch is related to shutdown behavior. Reboot doesn't use it. 
>> Could you
>> point which exactly version did you use and provide the .config which 
>> causes
>> problems with normal reboot when system runs from SD card? For eMMC 
>> you would
>> need patches from last linux-next.
>
> I'm currently using this tree:
> https://github.com/tobiasjakobi/linux-odroid
>
> In this is the config:
> https://raw.githubusercontent.com/tobiasjakobi/odroid-environment/master/sourcecode/system/vanilla-3.19-debug.conf 
>
>
> Upon issuing reboot I get this here on the UART:
> chidori ~ # reboot
>
> Broadcast message from root@chidori (ttySAC1) (Mon Feb 23 17:04:44 2015):
>
> The system is going down for reboot NOW!
> INIT: Switching to runlevel: 6
> INIT: Sending processes the TERM signal
> chidori ~ #  * Stopping 
> local                                             [ ok ]
>  * Stopping vixie-cron 
> ...                                                [ ok ]
>  * Saving random seed 
> ...                                                 [ ok ]
>  * Deactivating additional swap space 
> ...                                 [ ok ]
>  * Stopping sshd 
> ...                                                      [ ok ]
>  * Unmounting network filesystems 
> ...                                     [ ok ]
>  * Stopping chronyd 
> ...                                                   [ ok ]
>  * Stopping BlueTooth NAP 
> ...                                             [ ok ]
>  * Stopping dnsmasq 
> ...                                                   [ ok ]
>  * Stopping syslog-ng 
> ...                                                 [ ok ]
>  * Shutting down bluetooth 
> ...                                            [ ok ]
>  * Stopping D-BUS system messagebus 
> ...                                   [ ok ]
>  * Unmounting loop devices
>  * Unmounting filesystems
>  *   Unmounting /usr/local/emu 
> ...                                        [ ok ]
>  *   Unmounting /home 
> ...                                                 [ ok ]
>  *   Unmounting /var 
> ...                                                  [ ok ]
>  * Deactivating swap devices 
> ...                                          [ ok ]
>  * Stopping udev 
> ...                                                      [ ok ]
>  * Saving the shutdown time 
> ...                                           [ ok ]
>  * Terminating remaining processes 
> ...                                    [ ok ]
>  * Killing remaining processes 
> ...                                        [ ok ]
>  * Saving dependency cache 
> ...                                            [ ok ]
>  * Remounting remaining filesystems read-only ...
>  *   Remounting / read only ...
> [  113.526626] EXT4-fs (mmcblk0p2): re-mounted. Opts: 
> (null)              [ ok ]
> [  115.609592] sd 0:0:0:0: [sda] Synchronizing SCSI cache
> [
>
>
> Notice the open bracket in the last line. Seems like there is still 
> output in the kernel buffer, but the UART is maybe switched off before 
> one can see that. At least there is nothing after that, and I have to 
> manually restart the board then.

I've just checked and it works fine with my Odroid U3+ board booted from 
SD card. I have really no idea what wrong here... Could you check if the 
same problem exist with vanilla v4.0-rc1 kernel?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Marek Szyprowski]

Hello Tobias,

On 2015-03-03 15:10, Marek Szyprowski wrote:
> Hello,
>
> On 2015-02-24 14:39, Tobias Jakobi wrote:
>> Hello Marek,
>>
>> sorry for the late response!
>>
>>
>> On 2015-02-10 12:38, Marek Szyprowski wrote:
>>> Hello,
>>>
>>> On 2015-02-09 21:28, Tobias Jakobi wrote:
>>>> Hello!
>>>>
>>>> Marek Szyprowski wrote:
>>>>> I didn't observe any issues with 'ondemand' governor. I've just 
>>>>> tested it
>>>>> on next-20150204 with "[PATCH v2 0/3] Add support for hardware 
>>>>> reset of
>>>>> eMMC
>>>>> card on reboot" patches added
>>>>> (http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot 
>>>>> works
>>>>> fine.
>>>> Since I'm not using an eMMC card, I don't think the pwrseq stuff is
>>>> going to make a difference here.
>>>> Have you checked this with the PS_HOLD patch applied, or without?
>>>> Anyway, guess I'm going to recheck this, maybe with some other 
>>>> governors
>>>> too.
>>>
>>> PS_HOLD patch is related to shutdown behavior. Reboot doesn't use 
>>> it. Could you
>>> point which exactly version did you use and provide the .config 
>>> which causes
>>> problems with normal reboot when system runs from SD card? For eMMC 
>>> you would
>>> need patches from last linux-next.
>>
>> I'm currently using this tree:
>> https://github.com/tobiasjakobi/linux-odroid
>>
>> In this is the config:
>> https://raw.githubusercontent.com/tobiasjakobi/odroid-environment/master/sourcecode/system/vanilla-3.19-debug.conf 
>>
>>
>> Upon issuing reboot I get this here on the UART:
>> chidori ~ # reboot
>>
>> Broadcast message from root@chidori (ttySAC1) (Mon Feb 23 17:04:44 
>> 2015):
>>
>> The system is going down for reboot NOW!
>> INIT: Switching to runlevel: 6
>> INIT: Sending processes the TERM signal
>> chidori ~ #  * Stopping 
>> local                                             [ ok ]
>>  * Stopping vixie-cron 
>> ...                                                [ ok ]
>>  * Saving random seed 
>> ...                                                 [ ok ]
>>  * Deactivating additional swap space 
>> ...                                 [ ok ]
>>  * Stopping sshd 
>> ...                                                      [ ok ]
>>  * Unmounting network filesystems 
>> ...                                     [ ok ]
>>  * Stopping chronyd 
>> ...                                                   [ ok ]
>>  * Stopping BlueTooth NAP 
>> ...                                             [ ok ]
>>  * Stopping dnsmasq 
>> ...                                                   [ ok ]
>>  * Stopping syslog-ng 
>> ...                                                 [ ok ]
>>  * Shutting down bluetooth 
>> ...                                            [ ok ]
>>  * Stopping D-BUS system messagebus 
>> ...                                   [ ok ]
>>  * Unmounting loop devices
>>  * Unmounting filesystems
>>  *   Unmounting /usr/local/emu 
>> ...                                        [ ok ]
>>  *   Unmounting /home 
>> ...                                                 [ ok ]
>>  *   Unmounting /var 
>> ...                                                  [ ok ]
>>  * Deactivating swap devices 
>> ...                                          [ ok ]
>>  * Stopping udev 
>> ...                                                      [ ok ]
>>  * Saving the shutdown time 
>> ...                                           [ ok ]
>>  * Terminating remaining processes 
>> ...                                    [ ok ]
>>  * Killing remaining processes 
>> ...                                        [ ok ]
>>  * Saving dependency cache 
>> ...                                            [ ok ]
>>  * Remounting remaining filesystems read-only ...
>>  *   Remounting / read only ...
>> [  113.526626] EXT4-fs (mmcblk0p2): re-mounted. Opts: 
>> (null)              [ ok ]
>> [  115.609592] sd 0:0:0:0: [sda] Synchronizing SCSI cache
>> [
>>
>>
>> Notice the open bracket in the last line. Seems like there is still 
>> output in the kernel buffer, but the UART is maybe switched off 
>> before one can see that. At least there is nothing after that, and I 
>> have to manually restart the board then.
>
> I've just checked and it works fine with my Odroid U3+ board booted 
> from SD card. I have really no idea what wrong here... Could you check 
> if the same problem exist with vanilla v4.0-rc1 kernel?

I believe that patch e91d863d1aaeda0349aed42dfbd75c98e042a2c9 ("serial: 
samsung:
Clear operation mode on UART shutdown") solves this issue. It has been 
merged to
v4.0-rc7. Could you confirm it?

Best regards
-- 
Marek Szyprowski, PhD
Samsung R&D Institute Poland

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello Marek,

Marek Szyprowski wrote:
> Hello Tobias,
> 
> On 2015-03-03 15:10, Marek Szyprowski wrote:
>> Hello,
>>
>> On 2015-02-24 14:39, Tobias Jakobi wrote:
>>> Hello Marek,
>>>
>>> sorry for the late response!
>>>
>>>
>>> On 2015-02-10 12:38, Marek Szyprowski wrote:
>>>> Hello,
>>>>
>>>> On 2015-02-09 21:28, Tobias Jakobi wrote:
>>>>> Hello!
>>>>>
>>>>> Marek Szyprowski wrote:
>>>>>> I didn't observe any issues with 'ondemand' governor. I've just
>>>>>> tested it
>>>>>> on next-20150204 with "[PATCH v2 0/3] Add support for hardware
>>>>>> reset of
>>>>>> eMMC
>>>>>> card on reboot" patches added
>>>>>> (http://www.spinics.net/lists/linux-mmc/msg30621.html) and reboot
>>>>>> works
>>>>>> fine.
>>>>> Since I'm not using an eMMC card, I don't think the pwrseq stuff is
>>>>> going to make a difference here.
>>>>> Have you checked this with the PS_HOLD patch applied, or without?
>>>>> Anyway, guess I'm going to recheck this, maybe with some other
>>>>> governors
>>>>> too.
>>>>
>>>> PS_HOLD patch is related to shutdown behavior. Reboot doesn't use
>>>> it. Could you
>>>> point which exactly version did you use and provide the .config
>>>> which causes
>>>> problems with normal reboot when system runs from SD card? For eMMC
>>>> you would
>>>> need patches from last linux-next.
>>>
>>> I'm currently using this tree:
>>> https://github.com/tobiasjakobi/linux-odroid
>>>
>>> In this is the config:
>>> https://raw.githubusercontent.com/tobiasjakobi/odroid-environment/master/sourcecode/system/vanilla-3.19-debug.conf
>>>
>>>
>>> Upon issuing reboot I get this here on the UART:
>>> chidori ~ # reboot
>>>
>>> Broadcast message from root@chidori (ttySAC1) (Mon Feb 23 17:04:44
>>> 2015):
>>>
>>> The system is going down for reboot NOW!
>>> INIT: Switching to runlevel: 6
>>> INIT: Sending processes the TERM signal
>>> chidori ~ #  * Stopping
>>> local                                             [ ok ]
>>>  * Stopping vixie-cron
>>> ...                                                [ ok ]
>>>  * Saving random seed
>>> ...                                                 [ ok ]
>>>  * Deactivating additional swap space
>>> ...                                 [ ok ]
>>>  * Stopping sshd
>>> ...                                                      [ ok ]
>>>  * Unmounting network filesystems
>>> ...                                     [ ok ]
>>>  * Stopping chronyd
>>> ...                                                   [ ok ]
>>>  * Stopping BlueTooth NAP
>>> ...                                             [ ok ]
>>>  * Stopping dnsmasq
>>> ...                                                   [ ok ]
>>>  * Stopping syslog-ng
>>> ...                                                 [ ok ]
>>>  * Shutting down bluetooth
>>> ...                                            [ ok ]
>>>  * Stopping D-BUS system messagebus
>>> ...                                   [ ok ]
>>>  * Unmounting loop devices
>>>  * Unmounting filesystems
>>>  *   Unmounting /usr/local/emu
>>> ...                                        [ ok ]
>>>  *   Unmounting /home
>>> ...                                                 [ ok ]
>>>  *   Unmounting /var
>>> ...                                                  [ ok ]
>>>  * Deactivating swap devices
>>> ...                                          [ ok ]
>>>  * Stopping udev
>>> ...                                                      [ ok ]
>>>  * Saving the shutdown time
>>> ...                                           [ ok ]
>>>  * Terminating remaining processes
>>> ...                                    [ ok ]
>>>  * Killing remaining processes
>>> ...                                        [ ok ]
>>>  * Saving dependency cache
>>> ...                                            [ ok ]
>>>  * Remounting remaining filesystems read-only ...
>>>  *   Remounting / read only ...
>>> [  113.526626] EXT4-fs (mmcblk0p2): re-mounted. Opts:
>>> (null)              [ ok ]
>>> [  115.609592] sd 0:0:0:0: [sda] Synchronizing SCSI cache
>>> [
>>>
>>>
>>> Notice the open bracket in the last line. Seems like there is still
>>> output in the kernel buffer, but the UART is maybe switched off
>>> before one can see that. At least there is nothing after that, and I
>>> have to manually restart the board then.
>>
>> I've just checked and it works fine with my Odroid U3+ board booted
>> from SD card. I have really no idea what wrong here... Could you check
>> if the same problem exist with vanilla v4.0-rc1 kernel?
> 
> I believe that patch e91d863d1aaeda0349aed42dfbd75c98e042a2c9 ("serial:
> samsung:
> Clear operation mode on UART shutdown") solves this issue. It has been
> merged to
> v4.0-rc7. Could you confirm it?
I've tried some reboot cycles without modifying the cpu governor and so
far all reboot worked. So this issue looks like its fixed.


With best wishes,
Tobias


> 
> Best regards

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello!

Marek Szyprowski wrote:
>> 4) Spinlock BUGs triggered by the sdhci subsystem (so for the people
>> using the system with a SD card). This is also a known problem, I think
>> first mentioned here:
>> http://www.spinics.net/lists/linux-mmc/msg27277.html
>> Currently fixing this with this patch:
>> https://github.com/tobiasjakobi/linux-odroid/commit/abc749843dd7022d01322dca3db0181211a30cd8
>>
> 
> The fix for this issue has been queued to mmc-next:
> https://git.linaro.org/people/ulf.hansson/mmc.git/commit/017210d1c0dc2e2d3b142985cb31d90b98dc0f0f
This fix doesn't work for me. I have it applied, but encountered this
just now:

[   33.159544] BUG: scheduling while atomic: mmcqd/0/789/0x00000002
[   33.159558] Modules linked in: bridge stp llc bnep btrfs xor xor_neon
zlib_inflate zlib_deflate raid6_pq ecb btusb bluetooth usb_storage
[   33.159625] Preemption disabled at:[<c037de64>]
sdhci_do_set_ios+0x24/0x6a0

[   33.159673] CPU: 0 PID: 789 Comm: mmcqd/0 Not tainted 3.19.0-debug+ #1
[   33.159683] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[   33.159692] Backtrace:
[   33.159724] [<c00122e0>] (dump_backtrace) from [<c00124f8>]
(show_stack+0x18/0x1c)
[   33.159734]  r6:c06b6db4 r5:ffffffff r4:00000000 r3:dc8cb100
[   33.159777] [<c00124e0>] (show_stack) from [<c04bf2cc>]
(dump_stack+0x88/0xc8)
[   33.159796] [<c04bf244>] (dump_stack) from [<c04bbc9c>]
(__schedule_bug+0xac/0xbc)
[   33.159805]  r6:00000000 r5:ee791780 r4:ee3a4b00 r3:eda18000
[   33.159841] [<c04bbbf0>] (__schedule_bug) from [<c04c0d48>]
(__schedule+0x4b8/0x600)
[   33.159859] [<c04c0890>] (__schedule) from [<c04c0ec8>]
(schedule+0x38/0x84)
[   33.159868]  r10:c06f4374 r9:c0701c4c r8:eda19c78 r7:ee3a4b00
r6:600000d3 r5:c06c5994
[   33.159898]  r4:c06c5990
[   33.159919] [<c04c0e90>] (schedule) from [<c04c1274>]
(schedule_preempt_disabled+0x18/0x24)
[   33.159939] [<c04c125c>] (schedule_preempt_disabled) from
[<c04c3c8c>] (__mutex_lock_slowpath+0x190/0x428)
[   33.159955] [<c04c3afc>] (__mutex_lock_slowpath) from [<c04c3f38>]
(mutex_lock+0x14/0x2c)
[   33.159964]  r10:ed8a295c r9:ffffffff r8:00000000 r7:ee017668
r6:00000002 r5:ee021580
[   33.159994]  r4:c06c5990
[   33.160012] [<c04c3f24>] (mutex_lock) from [<c03a16f4>]
(clk_prepare_lock+0x58/0x100)
[   33.160021]  r4:c0704bac r3:ee370a00
[   33.160044] [<c03a169c>] (clk_prepare_lock) from [<c03a2284>]
(clk_round_rate+0x18/0x34)
[   33.160053]  r5:ee021580 r4:02faf080
[   33.160078] [<c03a226c>] (clk_round_rate) from [<c037fb70>]
(sdhci_s3c_set_clock+0x1d8/0x1fc)
[   33.160088]  r5:ee017400 r4:02faf080
[   33.160113] [<c037f998>] (sdhci_s3c_set_clock) from [<c037fbf0>]
(sdhci_cmu_set_clock+0x5c/0x1cc)
[   33.160122]  r9:eda19e8c r8:60000053 r7:ee270e00 r6:ee017000
r5:02faf080 r4:ee017400
[   33.160161] [<c037fb94>] (sdhci_cmu_set_clock) from [<c037df24>]
(sdhci_do_set_ios+0xe4/0x6a0)
[   33.160171]  r9:eda19e8c r8:60000053 r7:ee017518 r6:ee017000
r5:ee0172c0 r4:ee017400
[   33.160209] [<c037de40>] (sdhci_do_set_ios) from [<c037e510>]
(sdhci_set_ios+0x30/0x40)
[   33.160218]  r10:ed8a295c r9:eda19e8c r8:ee0173bc r7:ed8a2a80
r6:02faf080 r5:ee0172c0
[   33.160248]  r4:ee017000 r3:c06d09a8
[   33.160275] [<c037e4e0>] (sdhci_set_ios) from [<c0369614>]
(__mmc_set_clock+0x74/0x78)
[   33.160285]  r5:ee0172c0 r4:ee017000
[   33.160308] [<c03695a0>] (__mmc_set_clock) from [<c0369670>]
(mmc_ungate_clock+0x28/0x30)
[   33.160317]  r6:ee01725c r5:ee01724c r4:ee017000 r3:00000000
[   33.160351] [<c0369648>] (mmc_ungate_clock) from [<c036bcf0>]
(mmc_host_clk_hold+0x50/0x7c)
[   33.160369] [<c036bca0>] (mmc_host_clk_hold) from [<c036797c>]
(mmc_start_request+0xcc/0x110)
[   33.160378]  r6:00000001 r5:ee017000 r4:ed8a2970 r3:ed8a2a0c
[   33.160409] [<c03678b0>] (mmc_start_request) from [<c03688fc>]
(mmc_start_req+0x288/0x38c)
[   33.160419]  r5:00000000 r4:ee017000
[   33.160443] [<c0368674>] (mmc_start_req) from [<c0377b48>]
(mmc_blk_issue_rw_rq+0xcc/0xaf4)
[   33.160452]  r10:eda10000 r9:ed8a2814 r8:ed965000 r7:00000000
r6:ed965000 r5:00000000
[   33.160482]  r4:ed8a2970
[   33.160500] [<c0377a7c>] (mmc_blk_issue_rw_rq) from [<c0378674>]
(mmc_blk_issue_rq+0x104/0x4d8)
[   33.160510]  r10:24590060 r9:00000001 r8:ed8a2800 r7:ed8a2800
r6:ed965000 r5:eda10000
[   33.160539]  r4:ed8a2814
[   33.160558] [<c0378570>] (mmc_blk_issue_rq) from [<c0379970>]
(mmc_queue_thread+0xb8/0x15c)
[   33.160567]  r10:ed965000 r9:00000001 r8:24590060 r7:00000000
r6:eda18000 r5:eda28000
[   33.160597]  r4:ed8a2814
[   33.160619] [<c03798b8>] (mmc_queue_thread) from [<c00435a8>]
(kthread+0xe8/0x100)
[   33.160628]  r10:00000000 r9:00000000 r8:00000000 r7:c03798b8
r6:ed8a2814 r5:00000000
[   33.160657]  r4:ee3bdbc0
[   33.160677] [<c00434c0>] (kthread) from [<c000ec60>]
(ret_from_fork+0x14/0x34)
[   33.160686]  r7:00000000 r6:00000000 r5:c00434c0 r4:ee3bdbc0




With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Krzysztof Kozlowski]

2015-02-14 19:14 GMT+01:00 Tobias Jakobi <tjakobi@math.uni-bielefeld.de>:
> Hello!
>
> Marek Szyprowski wrote:
>>> 4) Spinlock BUGs triggered by the sdhci subsystem (so for the people
>>> using the system with a SD card). This is also a known problem, I think
>>> first mentioned here:
>>> http://www.spinics.net/lists/linux-mmc/msg27277.html
>>> Currently fixing this with this patch:
>>> https://github.com/tobiasjakobi/linux-odroid/commit/abc749843dd7022d01322dca3db0181211a30cd8
>>>
>>
>> The fix for this issue has been queued to mmc-next:
>> https://git.linaro.org/people/ulf.hansson/mmc.git/commit/017210d1c0dc2e2d3b142985cb31d90b98dc0f0f
> This fix doesn't work for me. I have it applied, but encountered this
> just now:
>
> [   33.159544] BUG: scheduling while atomic: mmcqd/0/789/0x00000002
> [   33.159558] Modules linked in: bridge stp llc bnep btrfs xor xor_neon
> zlib_inflate zlib_deflate raid6_pq ecb btusb bluetooth usb_storage
> [   33.159625] Preemption disabled at:[<c037de64>]
> sdhci_do_set_ios+0x24/0x6a0

Looking at the backtrace this seems very likely however I have
troubles reproducing this. Any special tree or config is required
(except MMC_CLKGATE and DEBUG_ATOMIC_SLEEP)?

Best regards,
Krzysztof

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello Krzysztof,

On 2015-02-17 14:24, Krzysztof Kozlowski wrote:
> Looking at the backtrace this seems very likely however I have
> troubles reproducing this. Any special tree or config is required
> (except MMC_CLKGATE and DEBUG_ATOMIC_SLEEP)?
> 
> Best regards,
> Krzysztof
The kernel was build with this config:
https://github.com/tobiasjakobi/odroid-environment/blob/master/sourcecode/system/vanilla-3.19-debug.conf

I didn't have DEBUG_ATOMIC_SLEEP enabled back then, but I could try to 
reproduce it again.

With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Tobias Jakobi]

Hello!

Tobias Jakobi wrote:
> Hello Krzysztof,
> 
> On 2015-02-17 14:24, Krzysztof Kozlowski wrote:
>> Looking at the backtrace this seems very likely however I have
>> troubles reproducing this. Any special tree or config is required
>> (except MMC_CLKGATE and DEBUG_ATOMIC_SLEEP)?
>>
>> Best regards,
>> Krzysztof
> The kernel was build with this config:
> https://github.com/tobiasjakobi/odroid-environment/blob/master/sourcecode/system/vanilla-3.19-debug.conf
> 
> 
> I didn't have DEBUG_ATOMIC_SLEEP enabled back then, but I could try to
> reproduce it again.
This issue is still happening with 4.0.0-rc4.

[    7.336824] BUG: sleeping function called from invalid context at
kernel/locking/mutex.c:97
[    7.336832] in_atomic(): 1, irqs_disabled(): 128, pid: 796, name: mmcqd/0
[    7.336838] Preemption disabled at:[<c038ce44>]
sdhci_do_set_ios+0x24/0x6a0
[    7.336883] CPU: 0 PID: 796 Comm: mmcqd/0 Not tainted 4.0.0-rc4-debug+ #3
[    7.336889] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree)
[    7.336894] Backtrace:
[    7.336921] [<c0012444>] (dump_backtrace) from [<c001265c>]
(show_stack+0x18/0x1c)
[    7.336925]  r6:c06cd554 r5:ffffffff r4:00000000 r3:dc8ba200
[    7.336951] [<c0012644>] (show_stack) from [<c04d1d0c>]
(dump_stack+0x88/0xc8)
[    7.336966] [<c04d1c84>] (dump_stack) from [<c00491bc>]
(___might_sleep+0x14c/0x1cc)
[    7.336970]  r6:00000000 r5:ffff1b58 r4:00000000 r3:ed9fe000
[    7.336983] [<c0049070>] (___might_sleep) from [<c00492a4>]
(__might_sleep+0x68/0xa0)
[    7.336987]  r5:00000061 r4:c05e0440
[    7.337002] [<c004923c>] (__might_sleep) from [<c04d6bb4>]
(mutex_lock+0x24/0x44)
[    7.337006]  r7:ed9ba668 r6:00000002 r5:02faf080 r4:c06dc3c0
[    7.337031] [<c04d6b90>] (mutex_lock) from [<c03b0870>]
(clk_prepare_lock+0x58/0x100)
[    7.337035]  r4:c071b574 r3:ee0e5f00
[    7.337047] [<c03b0818>] (clk_prepare_lock) from [<c03b1620>]
(clk_round_rate+0x1c/0x40)
[    7.337051]  r5:02faf080 r4:ed9a4080
[    7.337064] [<c03b1604>] (clk_round_rate) from [<c038eb68>]
(sdhci_s3c_set_clock+0x1d8/0x1fc)
[    7.337068]  r5:ed9ba3c0 r4:02faf080
[    7.337078] [<c038e990>] (sdhci_s3c_set_clock) from [<c038ebe8>]
(sdhci_cmu_set_clock+0x5c/0x1cc)
[    7.337082]  r9:eda48c14 r8:60010053 r7:ee294e00 r6:ed9ba000
r5:02faf080 r4:ed9ba3c0
[    7.337098] [<c038eb8c>] (sdhci_cmu_set_clock) from [<c038cf04>]
(sdhci_do_set_ios+0xe4/0x6a0)
[    7.337102]  r9:eda48c14 r8:60010053 r7:ed9ba4d8 r6:ed9ba000
r5:ed9ba2b8 r4:ed9ba3c0
[    7.337118] [<c038ce20>] (sdhci_do_set_ios) from [<c038d4f0>]
(sdhci_set_ios+0x30/0x3c)
[    7.337122]  r10:ed9ffe8c r9:eda48c14 r8:ee118820 r7:eda48d5c
r6:02faf080 r5:ed9ba2b8
[    7.337135]  r4:ed9ba000 r3:c06e73e8
[    7.337158] [<c038d4c0>] (sdhci_set_ios) from [<c0377e9c>]
(__mmc_set_clock+0x74/0x78)
[    7.337162]  r5:ed9ba2b8 r4:ed9ba000
[    7.337173] [<c0377e28>] (__mmc_set_clock) from [<c0377ef8>]
(mmc_ungate_clock+0x28/0x30)
[    7.337177]  r6:ed9ba258 r5:ed9ba248 r4:ed9ba000 r3:00000000
[    7.337194] [<c0377ed0>] (mmc_ungate_clock) from [<c037a5b4>]
(mmc_host_clk_hold+0x50/0x7c)
[    7.337202] [<c037a564>] (mmc_host_clk_hold) from [<c03760a8>]
(mmc_pre_req+0x2c/0x50)
[    7.337206]  r6:eda48c4c r5:00000001 r4:ed9ba000 r3:c0389aac
[    7.337220] [<c037607c>] (mmc_pre_req) from [<c0376f90>]
(mmc_start_req+0x38/0x368)
[    7.337224]  r6:ee118820 r5:00000000 r4:ed9ba000 r3:eda48c48
[    7.337238] [<c0376f58>] (mmc_start_req) from [<c0386a14>]
(mmc_blk_issue_rw_rq+0xcc/0xaf4)
[    7.337242]  r10:eda39000 r9:eda48c14 r8:ee118820 r7:00000000
r6:ee118820 r5:00000000
[    7.337255]  r4:eda48c4c
[    7.337263] [<c0386948>] (mmc_blk_issue_rw_rq) from [<c0387540>]
(mmc_blk_issue_rq+0x104/0x4d8)
[    7.337267]  r10:24590000 r9:eda48c14 r8:eda48c00 r7:eda48c00
r6:ee118820 r5:eda39000
[    7.337279]  r4:eda48c14
[    7.337287] [<c038743c>] (mmc_blk_issue_rq) from [<c0388854>]
(mmc_queue_thread+0xd0/0x18c)
[    7.337291]  r10:ed9fe000 r9:eda48c14 r8:00000001 r7:ee118820
r6:24590000 r5:00000000
[    7.337304]  r4:eda60000
[    7.337319] [<c0388784>] (mmc_queue_thread) from [<c0044084>]
(kthread+0xf4/0x110)
[    7.337322]  r10:00000000 r9:00000000 r8:00000000 r7:c0388784
r6:eda48c14 r5:00000000
[    7.337335]  r4:ee1020c0
[    7.337345] [<c0043f90>] (kthread) from [<c000eca0>]
(ret_from_fork+0x14/0x34)
[    7.337348]  r7:00000000 r6:00000000 r5:c0043f90 r4:ee1020c0


With best wishes,
Tobias

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Krzysztof Kozlowski]

2015-03-20 2:20 GMT+01:00 Tobias Jakobi <liquid.acid@gmx.net>:
> Hello!
>
> Tobias Jakobi wrote:
>> Hello Krzysztof,
>>
>> On 2015-02-17 14:24, Krzysztof Kozlowski wrote:
>>> Looking at the backtrace this seems very likely however I have
>>> troubles reproducing this. Any special tree or config is required
>>> (except MMC_CLKGATE and DEBUG_ATOMIC_SLEEP)?
>>>
>>> Best regards,
>>> Krzysztof
>> The kernel was build with this config:
>> https://github.com/tobiasjakobi/odroid-environment/blob/master/sourcecode/system/vanilla-3.19-debug.conf
>>
>>
>> I didn't have DEBUG_ATOMIC_SLEEP enabled back then, but I could try to
>> reproduce it again.
> This issue is still happening with 4.0.0-rc4.
>
> [    7.336824] BUG: sleeping function called from invalid context at
> kernel/locking/mutex.c:97

Hi,

Yes, Pawel Osmialowski's patch fixed one such issue but the rest still
exists. I looked at this and it was not quite easy fix. I don't have
spare time now for this issue so if anyone is interested in fixing
this then it would be great.

BTW Tobias, the OOPS/dmesg of yours is wrapped around 72 char. It is
harder to read it this way.

Best regards,
Krzysztof

Re: exynos4412: misc issues on Hardkernel Odroid boards

[Paul Osmialowski]

On Fri, 20 Mar 2015, Krzysztof Kozlowski wrote:

> 2015-03-20 2:20 GMT+01:00 Tobias Jakobi <liquid.acid@gmx.net>:
>> Hello!
>>
>> Tobias Jakobi wrote:
>>> Hello Krzysztof,
>>>
>>> On 2015-02-17 14:24, Krzysztof Kozlowski wrote:
>>>> Looking at the backtrace this seems very likely however I have
>>>> troubles reproducing this. Any special tree or config is required
>>>> (except MMC_CLKGATE and DEBUG_ATOMIC_SLEEP)?
>>>>
>>>> Best regards,
>>>> Krzysztof
>>> The kernel was build with this config:
>>> https://github.com/tobiasjakobi/odroid-environment/blob/master/sourcecode/system/vanilla-3.19-debug.conf
>>>
>>>
>>> I didn't have DEBUG_ATOMIC_SLEEP enabled back then, but I could try to
>>> reproduce it again.
>> This issue is still happening with 4.0.0-rc4.
>>
>> [    7.336824] BUG: sleeping function called from invalid context at
>> kernel/locking/mutex.c:97
>
> Hi,
>
> Yes, Pawel Osmialowski's patch fixed one such issue but the rest still
> exists. I looked at this and it was not quite easy fix. I don't have
> spare time now for this issue so if anyone is interested in fixing
> this then it would be great.
>
> BTW Tobias, the OOPS/dmesg of yours is wrapped around 72 char. It is
> harder to read it this way.
>
> Best regards,
> Krzysztof
>

Hi Guys,

Unfortunately, I'm not on Trats2 devices anymore, however, I found some 
time to have a quick look at the problem. And it looks like solving this 
would require more fundamental/general approach.

My patch which solves problem with call to clk_set_rate() uses the same 
approach as found in sdhci_set_power() function: unlock host->lock 
spinlock before doing operation which may sleep (e.g. while trying to 
acquire mutex).

For this new bug following fix probably would be sufficient, as 
sdhci_s3c_consider_clock() is also called with spinlock acquired in 
sdhci_do_set_ios(), which should be released before calling 
clk_round_rate():

--- sdhci-s3c.c.old	2015-03-23 16:55:09.933609335 +0100
+++ sdhci-s3c.c	2015-03-23 17:02:15.497610082 +0100
@@ -105,10 +105,11 @@
   * @src: The source clock index.
   * @wanted: The clock frequency wanted.
   */
-static unsigned int sdhci_s3c_consider_clock(struct sdhci_s3c *ourhost,
+static unsigned int sdhci_s3c_consider_clock(struct sdhci_host *host,
  					     unsigned int src,
  					     unsigned int wanted)
  {
+	struct sdhci_s3c *ourhost = to_s3c(host);
  	unsigned long rate;
  	struct clk *clksrc = ourhost->clk_bus[src];
  	int shift;
@@ -121,7 +122,9 @@
  	 * speed possible with selected clock source and skip the 
division.
  	 */
  	if (ourhost->no_divider) {
+		spin_unlock_irq(&host->lock);
  		rate = clk_round_rate(clksrc, wanted);
+		spin_lock_irq(&host->lock);
  		return wanted - rate;
  	}

@@ -171,7 +174,7 @@
  	}

  	for (src = 0; src < MAX_BUS_CLK; src++) {
-		delta = sdhci_s3c_consider_clock(ourhost, src, clock);
+		delta = sdhci_s3c_consider_clock(host, src, clock);
  		if (delta < best) {
  			best = delta;
  			best_src = src;

(Again, currently I have no device to test it).

Thing is, from quick look at the code I can't say what would happend in 
two other places where clk_round_rate() is called (and fortunately, 
clk_set_rate() call addressed by my previous patch is called from one 
place only). And what about other clk_ functions called from sdhci-s3c.c 
functions? Few lines below sdhci_s3c_consider_clock() call which we know 
is called with host->lock spinlock held we have calls 
to clk_prepare_enable() / clk_disable_unprepare() - will they fail too? 
Internally, they also call clk_prepare_lock() which causes all of the 
toruble.

Wrapping all of such places with spin_unlock_irq(&host->lock) / 
spin_lock_irq(&host->lock) seems for me like asking for troubles. IMHO 
more general approach should be proposed.

Best regards,
Paul