Re: [PATCH 0/4] PREEMPT_RT support for sparc64

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

> Have tested it on UltraSparc T4 (Niagara4). 
> 
> Allen Pais (4):
>   sparc64: use generic rwsem spinlocks rt
>   sparc64: allow forced irq threading
>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>   sparc64: convert ctx_alloc_lock raw_spinlock_t
> 
>  arch/sparc/Kconfig                      |    7 +++----
>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>  arch/sparc/kernel/smp_64.c              |    4 ++--
>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>  6 files changed, 28 insertions(+), 29 deletions(-)
> 

Sebastian, Any idea on the status of these patches?

Thanks,
Allen

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Sebastian Andrzej Siewior]

On 02/05/2014 04:31 AM, Allen Pais wrote:
> 
>> Have tested it on UltraSparc T4 (Niagara4). 
>>
>> Allen Pais (4):
>>   sparc64: use generic rwsem spinlocks rt
>>   sparc64: allow forced irq threading
>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>
>>  arch/sparc/Kconfig                      |    7 +++----
>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>
> 
> Sebastian, Any idea on the status of these patches?

I took that all in one I think so they are part of -RT. These individual
should go upstream via DaveM / the sparc tree.

> 
> Thanks,
> Allen

Sebastian

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

>>>
>>> Allen Pais (4):
>>>   sparc64: use generic rwsem spinlocks rt
>>>   sparc64: allow forced irq threading
>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>
>>>  arch/sparc/Kconfig                      |    7 +++----
>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>
>>
>> Sebastian, Any idea on the status of these patches?
> 
> I took that all in one I think so they are part of -RT. These individual
> should go upstream via DaveM / the sparc tree.
> 

 I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
list and check.

Thanks,
Allen

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Sebastian Andrzej Siewior]

On 02/05/2014 11:38 AM, Allen Pais wrote:
>>>>
>>>> Allen Pais (4):
>>>>   sparc64: use generic rwsem spinlocks rt
>>>>   sparc64: allow forced irq threading
>>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>>
>>>>  arch/sparc/Kconfig                      |    7 +++----
>>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>>
>>>
>>> Sebastian, Any idea on the status of these patches?
>>
>> I took that all in one I think so they are part of -RT. These individual
>> should go upstream via DaveM / the sparc tree.
>>
> 
>  I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
> list and check.

The rwsem thingy is the only -RT specific and it depends on -RT only
Kconfig symbols. The spin_lock_t to raw_spinlock_t conversations can
also go upstream.

> Thanks,
> Allen
> 
Sebastian

Re: [PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

On Wednesday 05 February 2014 04:13 PM, Sebastian Andrzej Siewior wrote:
> On 02/05/2014 11:38 AM, Allen Pais wrote:
>>>>>
>>>>> Allen Pais (4):
>>>>>   sparc64: use generic rwsem spinlocks rt
>>>>>   sparc64: allow forced irq threading
>>>>>   sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t
>>>>>   sparc64: convert ctx_alloc_lock raw_spinlock_t
>>>>>
>>>>>  arch/sparc/Kconfig                      |    7 +++----
>>>>>  arch/sparc/include/asm/mmu_64.h         |    2 +-
>>>>>  arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
>>>>>  arch/sparc/kernel/smp_64.c              |    4 ++--
>>>>>  arch/sparc/mm/init_64.c                 |   14 +++++++-------
>>>>>  arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
>>>>>  6 files changed, 28 insertions(+), 29 deletions(-)
>>>>>
>>>>
>>>> Sebastian, Any idea on the status of these patches?
>>>
>>> I took that all in one I think so they are part of -RT. These individual
>>> should go upstream via DaveM / the sparc tree.
>>>
>>
>>  I think DaveM has to pick irq threading patch. The rest are -RT. Will ping the sparc
>> list and check.
> 
> The rwsem thingy is the only -RT specific and it depends on -RT only
> Kconfig symbols. The spin_lock_t to raw_spinlock_t conversations can
> also go upstream.

They've all been sent upstream. Will check with DaveM on the status.

Thanks again.

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
> 
> 
> 06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>> In the attempt of get PREEMPT_RT working on sparc64 using
>> linux-stable-rt version 3.10.22-rt19+, the kernel crash
>> with the following trace:
>>
>> [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>> [ 1487.027885] Call Trace:
>> [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>> [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>> [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>> [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>> [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>> [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>> [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>> [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>> [ 1487.027922]  [0000000000000000]           (null)
>>

Now, consistently I've been getting sun4v_data_access_exception.
Here's the trace:

[ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
[ 4673.360124]               \|/ ____ \|/
[ 4673.360124]               "@'/ .. \`@"
[ 4673.360124]               /_| \__/ |_\
[ 4673.360124]                  \__U_/
[ 4673.360128] hackbench(4183): Dax [#1]
[ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G        W    3.10.24-rt22+ #12
[ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
[ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000    Tainted: G        W   
[ 4673.360157] TPC: <tsb_flush+0x4/0x40>
[ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
[ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
[ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
[ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
[ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
[ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
[ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
[ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
[ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
[ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
[ 4673.360193] Call Trace:
[ 4673.360198]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[ 4673.360203]  [000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360210]  [000000000052e520] unmap_single_vma+0x400/0x6c0
[ 4673.360213]  [000000000052e808] unmap_vmas+0x28/0x60
[ 4673.360220]  [0000000000530cc8] exit_mmap+0x88/0x160
[ 4673.360226]  [000000000045e0d4] mmput+0x34/0xe0
[ 4673.360236]  [00000000004669fc] do_exit+0x1fc/0xa40
[ 4673.360241]  [0000000000467270] do_group_exit+0x30/0xe0
[ 4673.360245]  [000000000046733c] SyS_exit_group+0x1c/0x40
[ 4673.360256]  [0000000000406234] linux_sparc_syscall+0x34/0x44
[ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
[ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
[ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
[ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
[ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
[ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
[ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
[ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
[ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
[ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44



- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:

>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>   with the following trace:
>>>
>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>   [ 1487.027885] Call Trace:
>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>   [ 1487.027922]  [0000000000000000]           (null)
>  Now, consistently I've been getting sun4v_data_access_exception.
>  Here's the trace:
>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.

I've never dived at sparc's tlb before, but it seems now I'm understanding.

arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
you collect flush requests before you really flush all of them.

In RT you collect them too, but you are able to be preempted in any moment.
So, you may switch to other process with unflushed tlb, which is very bad.

Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
We will look if this robust fix helps.


>  [ 4673.360124]               \|/ ____ \|/
>  [ 4673.360124]               "@'/ .. \`@"
>  [ 4673.360124]               /_| \__/ |_\
>  [ 4673.360124]                  \__U_/
>  [ 4673.360128] hackbench(4183): Dax [#1]
>  [ 4673.360137] CPU: 5 PID: 4183 Comm: hackbench Tainted: G        W    3.10.24-rt22+ #12
>  [ 4673.360141] task: fffff80f9c793840 ti: fffff80f9b270000 task.ti: fffff80f9b270000
>  [ 4673.360146] TSTATE: 0000004411e01606 TPC: 0000000000407b64 TNPC: 0000000000407b68 Y: 00000000    Tainted: G        W
>  [ 4673.360157] TPC: <tsb_flush+0x4/0x40>
>  [ 4673.360160] g0: fffff80f9c7c54b8 g1: 0000000000000001 g2: 0000000000008000 g3: 0000000000000000
>  [ 4673.360163] g4: fffff80f9c793840 g5: fffff80fcfc9c000 g6: fffff80f9b270000 g7: 0000000000000000
>  [ 4673.360167] o0: 0000080000000130 o1: 000003ffffe00400 o2: 0000000000878e48 o3: 0000000000000000
>  [ 4673.360170] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f9b272ec1 ret_pc: 00000000004520d0
>  [ 4673.360177] RPC: <flush_tsb_user+0x70/0x120>
>  [ 4673.360180] l0: 0000000000000001 l1: fffff80fd0800870 l2: 0000080000000000 l3: 00000000000001ff
>  [ 4673.360183] l4: fffff80f9852ea00 l5: fffff80f9852ee10 l6: 0000000000a87000 l7: 0000000000000000
>  [ 4673.360185] i0: fffff80fd0800868 i1: 0000000000000000 i2: 0000000000000000 i3: 0000000000000000
>  [ 4673.360187] i4: 0000000000000002 i5: 0000000000000030 i6: fffff80f9b272f71 i7: 00000000004515a8
>  [ 4673.360192] I7: <flush_tlb_pending+0x68/0xe0>
>  [ 4673.360193] Call Trace:
>  [ 4673.360198]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
>  [ 4673.360203]  [000000000045185c] arch_leave_lazy_mmu_mode+0x3c/0x60
>  [ 4673.360210]  [000000000052e520] unmap_single_vma+0x400/0x6c0
>  [ 4673.360213]  [000000000052e808] unmap_vmas+0x28/0x60
>  [ 4673.360220]  [0000000000530cc8] exit_mmap+0x88/0x160
>  [ 4673.360226]  [000000000045e0d4] mmput+0x34/0xe0
>  [ 4673.360236]  [00000000004669fc] do_exit+0x1fc/0xa40
>  [ 4673.360241]  [0000000000467270] do_group_exit+0x30/0xe0
>  [ 4673.360245]  [000000000046733c] SyS_exit_group+0x1c/0x40
>  [ 4673.360256]  [0000000000406234] linux_sparc_syscall+0x34/0x44
>  [ 4673.360260] Caller[00000000004515a8]: flush_tlb_pending+0x68/0xe0
>  [ 4673.360264] Caller[000000000045185c]: arch_leave_lazy_mmu_mode+0x3c/0x60
>  [ 4673.360267] Caller[000000000052e520]: unmap_single_vma+0x400/0x6c0
>  [ 4673.360270] Caller[000000000052e808]: unmap_vmas+0x28/0x60
>  [ 4673.360274] Caller[0000000000530cc8]: exit_mmap+0x88/0x160
>  [ 4673.360277] Caller[000000000045e0d4]: mmput+0x34/0xe0
>  [ 4673.360280] Caller[00000000004669fc]: do_exit+0x1fc/0xa40
>  [ 4673.360284] Caller[0000000000467270]: do_group_exit+0x30/0xe0
>  [ 4673.360287] Caller[000000000046733c]: SyS_exit_group+0x1c/0x40
>  [ 4673.360291] Caller[0000000000406234]: linux_sparc_syscall+0x34/0x44
>
>  - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.
> 

Kirill, Well the change works. So far the machine is up and no stall or crashes
with Hackbench. I'll run it for longer period and check.

Thanks,

Allen

 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>    [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>    [ 1487.027885] Call Trace:
>>>>>    [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>    [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>    [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>    [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>    [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>    [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>    [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>    [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>    [ 1487.027922]  [0000000000000000]           (null)
>>>   Now, consistently I've been getting sun4v_data_access_exception.
>>>   Here's the trace:
>>>   [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>>  I've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>>  arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>>  you collect flush requests before you really flush all of them.
>>
>>  In RT you collect them too, but you are able to be preempted in any moment.
>>  So, you may switch to other process with unflushed tlb, which is very bad.
>>
>>  Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>>  We will look if this robust fix helps.
>
> Kirill, Well the change works. So far the machine is up and no stall or crashes
> with Hackbench. I'll run it for longer period and check.

Ok, good.

But I don't know is this the best fix. May we have to implement another optimization
for RT.

For example, collect only batches which does not require smp call function. Or the
main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The other serious thing is to know does __set_pte_at() execute in preemption disable
context on !RT kernel. Because the place is interesting.

If yes, we have to do the same for RT. If not, then no.

Kirill

>
> Thanks,
>
> Allen
>
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
> 12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>    [ 1487.027885] Call Trace:
>>>>>>    [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>    [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>    [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>    [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>    [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>    [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>    [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>    [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>    [ 1487.027922]  [0000000000000000]           (null)

>>
>> Kirill, Well the change works. So far the machine is up and no stall or crashes
>> with Hackbench. I'll run it for longer period and check.
> 
> Ok, good.
> 
> But I don't know is this the best fix. May we have to implement another optimization
> for RT.

No, unfortunately, the system hit a stall on about 8 cpu's. 
CPU: 31 PID: 28675 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
[ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000    Tainted: G      D W   
TPC: <do_raw_spin_lock+0xb4/0x120>
[ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
[ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
[ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
[ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
[ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
[ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
[ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
[ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
[ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
[ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
[ 5725.097692] Call Trace:
[ 5725.097697]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5725.097702]  [0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
[ 5725.097707]  [00000000008790ac] rt_spin_lock+0xc/0x40
[ 5725.097712]  [00000000008610bc] unix_stream_sendmsg+0x15c/0x380
[ 5725.097717]  [00000000007ac114] sock_aio_write+0xf4/0x120
[ 5725.097722]  [000000000055891c] do_sync_write+0x5c/0xa0
[ 5725.097727]  [0000000000559e1c] vfs_write+0x15c/0x180
[ 5725.097732]  [0000000000559ef8] SyS_write+0x38/0x80
[ 5725.097738]  [0000000000406234] linux_sparc_syscall+0x34/0x44

This(above) on a few cpu's and this(below) on the other

BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
[ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
[ 5728.378347] irq event stamp: 0
[ 5728.378350] hardirqs last  enabled at (0): [<          (null)>]           (null)
[ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378361] softirqs last  enabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
[ 5728.378364] softirqs last disabled at (0): [<          (null)>]           (null)
[ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
[ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
[ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000    Tainted: G      D W   
[ 5728.378378] TPC: <do_exit+0xb4/0xa40>
[ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
[ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
[ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
[ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
[ 5728.378390] RPC: <do_exit+0x9c/0xa40>
[ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
[ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
[ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
[ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
[ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
[ 5728.378406] Call Trace:
[ 5728.378410]  [000000000087a650] perfctr_irq+0x3d0/0x420
[ 5728.378415]  [00000000004209f4] tl0_irq15+0x14/0x20
[ 5728.378419]  [000000000042f608] stick_get_tick+0x8/0x20
[ 5728.378422]  [000000000042fa24] __delay+0x24/0x60
[ 5728.378426]  [0000000000671e58] do_raw_spin_lock+0xb8/0x120
[ 5728.378430]  [0000000000879b14] _raw_spin_lock+0x54/0x80
[ 5728.378435]  [00000000004a1978] load_balance+0x538/0x860
[ 5728.378438]  [00000000004a2154] idle_balance+0x134/0x1c0
[ 5728.378442]  [0000000000877d54] switch_to_pc+0x1f4/0x2c0
[ 5728.378445]  [0000000000877ec4] schedule+0x24/0xc0
[ 5728.378449]  [0000000000876860] schedule_timeout+0x1c0/0x2a0
[ 5728.378452]  [0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
[ 5728.378456]  [00000000007ac23c] sock_aio_read+0xfc/0x120
[ 5728.378460]  [0000000000558adc] do_sync_read+0x5c/0xa0
[ 5728.378464]  [000000000055a04c] vfs_read+0x10c/0x120
[ 5728.378467]  [000000000055a118] SyS_read+0x38/0x80

> 
> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..
> 
> The other serious thing is to know does __set_pte_at() execute in preemption disable
> context on !RT kernel. Because the place is interesting.
> 
> If yes, we have to do the same for RT. If not, then no.

I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

Thanks,

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>
>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)
>>>  Kirill, Well the change works. So far the machine is up and no stall or crashes
>>>  with Hackbench. I'll run it for longer period and check.
>>  Ok, good.
>>
>>  But I don't know is this the best fix. May we have to implement another optimization
>>  for RT.
>
> No, unfortunately, the system hit a stall on about 8 cpu's.
> CPU: 31 PID: 28675 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
> [ 5725.097645] task: fffff80f929da8c0 ti: fffff80f8a4fc000 task.ti: fffff80f8a4fc000
> [ 5725.097649] TSTATE: 0000000011001604 TPC: 0000000000671e54 TNPC: 0000000000671e58 Y: 00000000    Tainted: G      D W
> TPC: <do_raw_spin_lock+0xb4/0x120>
> [ 5725.097657] g0: 0000000000671e4c g1: 00000000000000ff g2: 0000000002625010 g3: 0000000000000000
> [ 5725.097661] g4: fffff80f929da8c0 g5: fffff80fd649c000 g6: fffff80f8a4fc000 g7: 0000000000000000
> [ 5725.097664] o0: 0000000000000001 o1: 00000000009dfc00 o2: 0000000000000000 o3: 0000000000000000
> [ 5725.097667] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f8a4fee21 ret_pc: 0000000000671e58
> [ 5725.097671] RPC: <do_raw_spin_lock+0xb8/0x120>
> [ 5725.097675] l0: 000000000933b401 l1: 000000003b99d190 l2: 0000000000e25c00 l3: 0000000000000000
> [ 5725.097678] l4: 0000000000000000 l5: 0000000000000000 l6: 0000000000000000 l7: fffff801001254c8
> [ 5725.097682] i0: fffff80f89a367c8 i1: 0000000000878be4 i2: 0000000000000000 i3: 0000000000000000
> [ 5725.097685] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80f8a4feed1 i7: 0000000000879b14
> [ 5725.097690] I7: <_raw_spin_lock+0x54/0x80>
> [ 5725.097692] Call Trace:
> [ 5725.097697]  [0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5725.097702]  [0000000000878be4] rt_spin_lock_slowlock+0x24/0x340
> [ 5725.097707]  [00000000008790ac] rt_spin_lock+0xc/0x40
> [ 5725.097712]  [00000000008610bc] unix_stream_sendmsg+0x15c/0x380
> [ 5725.097717]  [00000000007ac114] sock_aio_write+0xf4/0x120
> [ 5725.097722]  [000000000055891c] do_sync_write+0x5c/0xa0
> [ 5725.097727]  [0000000000559e1c] vfs_write+0x15c/0x180
> [ 5725.097732]  [0000000000559ef8] SyS_write+0x38/0x80
> [ 5725.097738]  [0000000000406234] linux_sparc_syscall+0x34/0x44

No ideas right now.

> This(above) on a few cpu's and this(below) on the other
>
> BUG: soft lockup - CPU#13 stuck for 22s! [hackbench:28701]
> [ 5728.378345] Modules linked in: binfmt_misc usb_storage ehci_pci ehci_hcd sg n2_rng rng_core ext4 jbd2 crc16 sr_mod mpt2sas scsi_transport_sas raid_class sunvnet sunvdc dm_mirror dm_region_hash dm_log dm_mod be2iscsi iscsi_boot_sysfs bnx2i cnic uio ipv6 cxgb4i cxgb4 cxgb3i libcxgbi cxgb3 mdio libiscsi_tcp libiscsi scsi_transport_iscsi
> [ 5728.378347] irq event stamp: 0
> [ 5728.378350] hardirqs last  enabled at (0): [<          (null)>]           (null)
> [ 5728.378356] hardirqs last disabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378361] softirqs last  enabled at (0): [<000000000045eb38>] copy_process+0x418/0x1080
> [ 5728.378364] softirqs last disabled at (0): [<          (null)>]           (null)
> [ 5728.378368] CPU: 13 PID: 28701 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #13
> [ 5728.378371] task: fffff80f90efbb80 ti: fffff80f925ac000 task.ti: fffff80f925ac000
> [ 5728.378374] TSTATE: 0000000011001604 TPC: 00000000004668b4 TNPC: 00000000004668b8 Y: 00000000    Tainted: G      D W
> [ 5728.378378] TPC: <do_exit+0xb4/0xa40>
> [ 5728.378380] g0: 0000000000003f40 g1: 00000000000000ff g2: fffff80f90efbeb0 g3: 0000000000000002
> [ 5728.378383] g4: fffff80f90efbb80 g5: fffff80fd1c9c000 g6: fffff80f925ac000 g7: 0000000000000000
> [ 5728.378385] o0: fffff80f90efbb80 o1: fffff80f925ac400 o2: 000000000087a654 o3: 0000000000000000
> [ 5728.378387] o4: 0000000000000000 o5: fffff80f925aff40 sp: fffff80fff98f671 ret_pc: 000000000046689c
> [ 5728.378390] RPC: <do_exit+0x9c/0xa40>
> [ 5728.378393] l0: fffff80f90efbb80 l1: 0000004480001603 l2: 000000000087a650 l3: 0000000000000400
> [ 5728.378395] l4: 0000000000000000 l5: 0000000000000003 l6: 0000000000000000 l7: 0000000000000008
> [ 5728.378397] i0: 000000000000000a i1: 000000000000000d i2: 000000000042f608 i3: 0000000000000000
> [ 5728.378400] i4: 000000000000004f i5: 0000000000000002 i6: fffff80fff98f741 i7: 000000000087a650
> [ 5728.378405] I7: <perfctr_irq+0x3d0/0x420>
> [ 5728.378406] Call Trace:
> [ 5728.378410]  [000000000087a650] perfctr_irq+0x3d0/0x420
> [ 5728.378415]  [00000000004209f4] tl0_irq15+0x14/0x20
> [ 5728.378419]  [000000000042f608] stick_get_tick+0x8/0x20
> [ 5728.378422]  [000000000042fa24] __delay+0x24/0x60
> [ 5728.378426]  [0000000000671e58] do_raw_spin_lock+0xb8/0x120
> [ 5728.378430]  [0000000000879b14] _raw_spin_lock+0x54/0x80
> [ 5728.378435]  [00000000004a1978] load_balance+0x538/0x860
> [ 5728.378438]  [00000000004a2154] idle_balance+0x134/0x1c0
> [ 5728.378442]  [0000000000877d54] switch_to_pc+0x1f4/0x2c0
> [ 5728.378445]  [0000000000877ec4] schedule+0x24/0xc0
> [ 5728.378449]  [0000000000876860] schedule_timeout+0x1c0/0x2a0
> [ 5728.378452]  [0000000000860ac0] unix_stream_recvmsg+0x240/0x6e0
> [ 5728.378456]  [00000000007ac23c] sock_aio_read+0xfc/0x120
> [ 5728.378460]  [0000000000558adc] do_sync_read+0x5c/0xa0
> [ 5728.378464]  [000000000055a04c] vfs_read+0x10c/0x120
> [ 5728.378467]  [000000000055a118] SyS_read+0x38/0x80
>
>>  For example, collect only batches which does not require smp call function. Or the
>>  main goal of lazy tlb was to prevent smp calls?! It's good to discover this..
>>
>>  The other serious thing is to know does __set_pte_at() execute in preemption disable
>>  context on !RT kernel. Because the place is interesting.
>>
>>  If yes, we have to do the same for RT. If not, then no.
>
> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.

++all above. May we have to add one more crutch... Put preempt_disable() at begining of
__set_pte_at() and enable at end...


> Thanks,
>
> Allen
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 12 February 2014 06:15 PM, Kirill Tkhai wrote:
> 
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)

>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...
> 
I'll look into it. Thanks again.

Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,
> 
> 12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>> On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>
>>>  12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>     [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>     [ 1487.027885] Call Trace:
>>>>>>>>     [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>     [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>     [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>     [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>     [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>     [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>     [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>     [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>     [ 1487.027922]  [0000000000000000]           (null)
>>
>> I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
> 
> ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
> __set_pte_at() and enable at end...


 I realized locking in tsb is very tricky. My attempts to try and get hackbench run
without causing a stall failed. So here's what I tried to fix it, am not sure if it's
an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
with hackbench and dd, the system did not stall :)

diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
index 9eb10b4..24dcd29 100644
--- a/arch/sparc/mm/tsb.c
+++ b/arch/sparc/mm/tsb.c
@@ -6,6 +6,7 @@
 #include <linux/kernel.h>
 #include <linux/preempt.h>
 #include <linux/slab.h>
+#include <linux/locallock.h>
 #include <asm/page.h>
 #include <asm/pgtable.h>
 #include <asm/mmu_context.h>
@@ -14,6 +15,7 @@
 #include <asm/oplib.h>
 
 extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
+static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
 
 static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
 {
@@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
 void flush_tsb_user(struct tlb_batch *tb)
 {
        struct mm_struct *mm = tb->mm;
-       unsigned long nentries, base, flags;
+       unsigned long nentries, base;
 
-       raw_spin_lock_irqsave(&mm->context.lock, flags);
+       local_lock(tsb_lock);
 
        base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
        nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
@@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
                __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
        }
 #endif
-       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
+       local_unlock(tsb_lock);
 }


Thanks,

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 07:54, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>>  12.02.2014, 16:15, "Allen Pais" <allen.pais@oracle.com>:
>>>  On Wednesday 12 February 2014 05:13 PM, Kirill Tkhai wrote:
>>>>   12.02.2014, 15:29, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>>>>      [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>>>>>      [ 1487.027885] Call Trace:
>>>>>>>>>      [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>>>>>      [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>>>>>      [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>>>>>      [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>>>>>      [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>>>>>      [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>>>>>      [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>>>>>      [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>>>>>      [ 1487.027922]  [0000000000000000]           (null)
>>>  I am not convinced that I've covered all tlb/smp code. Guess I'll need to dig more.
>>  ++all above. May we have to add one more crutch... Put preempt_disable() at begining of
>>  __set_pte_at() and enable at end...
>
>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)
>
> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
> index 9eb10b4..24dcd29 100644
> --- a/arch/sparc/mm/tsb.c
> +++ b/arch/sparc/mm/tsb.c
> @@ -6,6 +6,7 @@
>  #include <linux/kernel.h>
>  #include <linux/preempt.h>
>  #include <linux/slab.h>
> +#include <linux/locallock.h>
>  #include <asm/page.h>
>  #include <asm/pgtable.h>
>  #include <asm/mmu_context.h>
> @@ -14,6 +15,7 @@
>  #include <asm/oplib.h>
>
>  extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>
>  static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>  {
> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>  void flush_tsb_user(struct tlb_batch *tb)
>  {
>         struct mm_struct *mm = tb->mm;
> -       unsigned long nentries, base, flags;
> +       unsigned long nentries, base;
>
> -       raw_spin_lock_irqsave(&mm->context.lock, flags);
> +       local_lock(tsb_lock);
>
>         base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>         nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>                 __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>         }
>  #endif
> -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
> +       local_unlock(tsb_lock);

It seems to be not good for me. Tsb setup is in tsb_grow() and it must
be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..

Which last stack stack has you received with tb->active, permanently set to zero?

>  }
>
> Thanks,
>
> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

>> diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>> index 9eb10b4..24dcd29 100644
>> --- a/arch/sparc/mm/tsb.c
>> +++ b/arch/sparc/mm/tsb.c
>> @@ -6,6 +6,7 @@
>>  #include <linux/kernel.h>
>>  #include <linux/preempt.h>
>>  #include <linux/slab.h>
>> +#include <linux/locallock.h>
>>  #include <asm/page.h>
>>  #include <asm/pgtable.h>
>>  #include <asm/mmu_context.h>
>> @@ -14,6 +15,7 @@
>>  #include <asm/oplib.h>
>>
>>  extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>> +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>
>>  static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>  {
>> @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>  void flush_tsb_user(struct tlb_batch *tb)
>>  {
>>         struct mm_struct *mm = tb->mm;
>> -       unsigned long nentries, base, flags;
>> +       unsigned long nentries, base;
>>
>> -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>> +       local_lock(tsb_lock);
>>
>>         base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>         nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>> @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>                 __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>         }
>>  #endif
>> -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>> +       local_unlock(tsb_lock);
> 
> It seems to be not good for me. Tsb setup is in tsb_grow() and it must
> be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
> 
> Which last stack stack has you received with tb->active, permanently set to zero?
> 

I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
a bit tricky to actually debug.

Yes, tb->active was set to zero.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>  diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>  index 9eb10b4..24dcd29 100644
>>>  --- a/arch/sparc/mm/tsb.c
>>>  +++ b/arch/sparc/mm/tsb.c
>>>  @@ -6,6 +6,7 @@
>>>   #include <linux/kernel.h>
>>>   #include <linux/preempt.h>
>>>   #include <linux/slab.h>
>>>  +#include <linux/locallock.h>
>>>   #include <asm/page.h>
>>>   #include <asm/pgtable.h>
>>>   #include <asm/mmu_context.h>
>>>  @@ -14,6 +15,7 @@
>>>   #include <asm/oplib.h>
>>>
>>>   extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>  +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>
>>>   static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>   {
>>>  @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>   void flush_tsb_user(struct tlb_batch *tb)
>>>   {
>>>          struct mm_struct *mm = tb->mm;
>>>  -       unsigned long nentries, base, flags;
>>>  +       unsigned long nentries, base;
>>>
>>>  -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>  +       local_lock(tsb_lock);
>>>
>>>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>  @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>          }
>>>   #endif
>>>  -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>  +       local_unlock(tsb_lock);
>>  It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>  be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>
>>  Which last stack stack has you received with tb->active, permanently set to zero?
>
> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
> a bit tricky to actually debug.
>
> Yes, tb->active was set to zero.

If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.

> - Allen
> --
> To unsubscribe from this list: send the line "unsubscribe sparclinux" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
> 
> 
> 19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>  diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>  index 9eb10b4..24dcd29 100644
>>>>  --- a/arch/sparc/mm/tsb.c
>>>>  +++ b/arch/sparc/mm/tsb.c
>>>>  @@ -6,6 +6,7 @@
>>>>   #include <linux/kernel.h>
>>>>   #include <linux/preempt.h>
>>>>   #include <linux/slab.h>
>>>>  +#include <linux/locallock.h>
>>>>   #include <asm/page.h>
>>>>   #include <asm/pgtable.h>
>>>>   #include <asm/mmu_context.h>
>>>>  @@ -14,6 +15,7 @@
>>>>   #include <asm/oplib.h>
>>>>
>>>>   extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>  +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>
>>>>   static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>   {
>>>>  @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>   void flush_tsb_user(struct tlb_batch *tb)
>>>>   {
>>>>          struct mm_struct *mm = tb->mm;
>>>>  -       unsigned long nentries, base, flags;
>>>>  +       unsigned long nentries, base;
>>>>
>>>>  -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>  +       local_lock(tsb_lock);
>>>>
>>>>          base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>          nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>  @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>                  __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>          }
>>>>   #endif
>>>>  -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>  +       local_unlock(tsb_lock);
>>>  It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>  be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>
>>>  Which last stack stack has you received with tb->active, permanently set to zero?
>>
>> I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>> a bit tricky to actually debug.
>>
>> Yes, tb->active was set to zero.
> 
> If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>
Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:

>  On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>>   19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>>    diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>>    index 9eb10b4..24dcd29 100644
>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>    @@ -6,6 +6,7 @@
>>>>>     #include <linux/kernel.h>
>>>>>     #include <linux/preempt.h>
>>>>>     #include <linux/slab.h>
>>>>>    +#include <linux/locallock.h>
>>>>>     #include <asm/page.h>
>>>>>     #include <asm/pgtable.h>
>>>>>     #include <asm/mmu_context.h>
>>>>>    @@ -14,6 +15,7 @@
>>>>>     #include <asm/oplib.h>
>>>>>
>>>>>     extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>>    +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>
>>>>>     static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>>     {
>>>>>    @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>>     void flush_tsb_user(struct tlb_batch *tb)
>>>>>     {
>>>>>            struct mm_struct *mm = tb->mm;
>>>>>    -       unsigned long nentries, base, flags;
>>>>>    +       unsigned long nentries, base;
>>>>>
>>>>>    -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>>    +       local_lock(tsb_lock);
>>>>>
>>>>>            base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>>            nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>>    @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>>                    __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>>            }
>>>>>     #endif
>>>>>    -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>>    +       local_unlock(tsb_lock);
>>>>    It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>>    be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>
>>>>    Which last stack stack has you received with tb->active, permanently set to zero?
>>>   I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>>   a bit tricky to actually debug.
>>>
>>>   Yes, tb->active was set to zero.
>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
You may get good stack without sun4v_data_access_exception error, which was in the first or second
message.

>  - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 19 February 2014 02:55 PM, Kirill Tkhai wrote:
> 19.02.2014, 13:13, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 19 February 2014 02:27 PM, Kirill Tkhai wrote:
>>>   19.02.2014, 12:12, "Allen Pais" <allen.pais@oracle.com>:
>>>>>>    diff --git a/arch/sparc/mm/tsb.c b/arch/sparc/mm/tsb.c
>>>>>>    index 9eb10b4..24dcd29 100644
>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>
>>>>>>
>>>>>>     extern struct tsb swapper_tsb[KERNEL_TSB_NENTRIES];
>>>>>>    +static DEFINE_LOCAL_IRQ_LOCK(tsb_lock);
>>>>>>
>>>>>>     static inline unsigned long tsb_hash(unsigned long vaddr, unsigned long hash_sh
>>>>>>     {
>>>>>>    @@ -71,9 +73,9 @@ static void __flush_tsb_one(struct tlb_batch *tb, unsigned lon
>>>>>>     void flush_tsb_user(struct tlb_batch *tb)
>>>>>>     {
>>>>>>            struct mm_struct *mm = tb->mm;
>>>>>>    -       unsigned long nentries, base, flags;
>>>>>>    +       unsigned long nentries, base;
>>>>>>
>>>>>>    -       raw_spin_lock_irqsave(&mm->context.lock, flags);
>>>>>>    +       local_lock(tsb_lock);
>>>>>>
>>>>>>            base = (unsigned long) mm->context.tsb_block[MM_TSB_BASE].tsb;
>>>>>>            nentries = mm->context.tsb_block[MM_TSB_BASE].tsb_nentries;
>>>>>>    @@ -90,7 +92,7 @@ void flush_tsb_user(struct tlb_batch *tb)
>>>>>>                    __flush_tsb_one(tb, HPAGE_SHIFT, base, nentries);
>>>>>>            }
>>>>>>     #endif
>>>>>>    -       raw_spin_unlock_irqrestore(&mm->context.lock, flags);
>>>>>>    +       local_unlock(tsb_lock);
>>>>>    It seems to be not good for me. Tsb setup is in tsb_grow() and it must
>>>>>    be synchronized with flushing. Flushing is also being made in flush_tsb_user_page()..
>>>>>
>>>>>    Which last stack stack has you received with tb->active, permanently set to zero?
>>>>   I agree with you point about flushing in flush_tbs_user_page too. Like i said, this is
>>>>   a bit tricky to actually debug.
>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.
> 
> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

I agree, I hope to get more a few opinions on the patches/issues. Meanwhile, Lemme see if I could
manage a way to stick to the old protectors and fix the stalls.

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

Kirill,

>>>>>>    --- a/arch/sparc/mm/tsb.c
>>>>>>    +++ b/arch/sparc/mm/tsb.c
>>>>>>    @@ -6,6 +6,7 @@
>>>>>>     #include <linux/kernel.h>
>>>>>>     #include <linux/preempt.h>
>>>>>>     #include <linux/slab.h>
>>>>>>    +#include <linux/locallock.h>
>>>>>>     #include <asm/page.h>
>>>>>>     #include <asm/pgtable.h>
>>>>>>     #include <asm/mmu_context.h>
>>>>>>    @@ -14,6 +15,7 @@
>>>>>>     #include <asm/oplib.h>

>>>>
>>>>   Yes, tb->active was set to zero.
>>>   If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>  Sorry, my bad. tb->active was set to one when I ran the test with the above patch.

  The CPU now does not stall, the change I did was remove debug lockdep from the config.
Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.

1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
[53990.477387] kernel BUG at kernel/rtmutex.c:738!
[53990.477393]               \|/ ____ \|/
[53990.477393]               "@'/ .. \`@"
[53990.477393]               /_| \__/ |_\
[53990.477393]                  \__U_/
[53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
[53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
[53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000    Tainted: G      D W   
[53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
[53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
[53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
[53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
[53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
[53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
[53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
[53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
[53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
[53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
[53990.477462] I7: <rt_spin_lock+0x20/0x60>
[53990.477464] Call Trace:
[53990.477470]  [0000000000876ee0] rt_spin_lock+0x20/0x60
[53990.477476]  [000000000052ee60] unmap_single_vma+0x200/0x6c0
[53990.477482]  [000000000052f348] unmap_vmas+0x28/0x60
[53990.477488]  [0000000000531868] exit_mmap+0x88/0x160
[53990.477492]  [000000000045e0e8] mmput+0x48/0x100
[53990.477496]  [0000000000466a3c] do_exit+0x1fc/0xa40
[53990.477500]  [0000000000427f00] die_if_kernel+0x1a0/0x340
[53990.477506]  [00000000004294a8] sun4v_data_access_exception+0x108/0x120
[53990.477512]  [0000000000406c08] sun4v_dacc+0x28/0x34
[53990.477517]  [0000000000407b64] tsb_flush+0x4/0x40
[53990.477523]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53990.477528]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53990.477534]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53990.477539]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53990.477544]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53990.477548]  [000000000052cab8] handle_mm_fault+0xd8/0x160

and

2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
[53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
[53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
[53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000    Tainted: G      D W   
[53998.070236] TPC: <stick_get_tick+0x8/0x20>
[53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
[53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
[53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
[53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
[53998.070264] RPC: <tl0_irq15+0x14/0x20>
[53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
[53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
[53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
[53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
[53998.070280] I7: <__delay+0x10/0x60>
[53998.070282] Call Trace:
[53998.070286]  [000000000042fa10] __delay+0x10/0x60
[53998.070291]  [000000000066f8b8] do_raw_spin_lock+0xb8/0x120
[53998.070300]  [0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
[53998.070306]  [0000000000452074] flush_tsb_user+0x14/0x120
[53998.070309]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[53998.070312]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[53998.070325]  [000000000053cad8] ptep_clear_flush+0x38/0x60
[53998.070328]  [000000000052b47c] do_wp_page+0x1dc/0x880
[53998.070331]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
[53998.070334]  [000000000052cab8] handle_mm_fault+0xd8/0x160
[53998.070339]  [0000000000879724] do_sparc64_fault+0x404/0x700
[53998.070342]  [0000000000407ae0] sparc64_realfault_common+0x10/0x20



But strangely, during boot-up I have more crash messages. 
Here's what I see

[  520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
tex.c:659
[  520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
[  520.570803] INFO: lockdep is turned off.
[  520.570805] irq event stamp: 4502
[  520.570806] hardirqs last  enabled at (4501): [<00000000004d68c4>] rcu_note_c
ontext_switch+0xa4/0x300
[  520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
lock_irq+0x10/0x80
[  520.570822] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  520.570828] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G        W    3.10.24-r
t22+ #25
[  520.570835] Call Trace:
[  520.570842]  [0000000000495f0c] __might_sleep+0xec/0x160
[  520.570846]  [0000000000876ed8] rt_spin_lock+0x18/0x60
[  520.570852]  [00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
[  520.570855]  [00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
[  520.570858]  [0000000000462a14] console_unlock+0x394/0x400
[  520.570861]  [0000000000463108] vprintk_emit+0x3a8/0x5a0
[  520.570863]  [0000000000874378] printk+0x38/0x4c
[  520.570874]  [000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
sas]
[  520.570882]  [0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
[  520.570893]  [000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
[  520.570900]  [0000000000686120] local_pci_probe+0x20/0x40
[  520.570903]  [000000000068680c] pci_device_probe+0xec/0x100
[  520.570907]  [00000000006ee574] driver_probe_device+0x74/0x220
[  520.570909]  [00000000006ee7a8] __driver_attach+0x88/0xa0
[  520.570913]  [00000000006eca0c] bus_for_each_dev+0x6c/0xa0
[  520.570916]  [00000000006ee39c] driver_attach+0x1c/0x40


and this one 

[  519.160755] =================================
[  519.160756] [ INFO: inconsistent lock state ]
[  519.160760] 3.10.24-rt22+ #25 Not tainted
[  519.160761] ---------------------------------
[  519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
[  519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
[  519.160778]  (&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
simple_irq+0xc/0xe0
[  519.160779] {IN-HARDIRQ-W} state was registered at:
[  519.160785]   [<00000000004ba7c0>] lock_acquire+0x60/0x100
[  519.160791]   [<0000000000877930>] _raw_spin_lock+0x30/0x80
[  519.160795]   [<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
[  519.160800]   [<00000000004cf118>] generic_handle_irq+0x38/0x60
[  519.160804]   [<000000000087bf44>] handler_irq+0xc4/0x100
[  519.160808]   [<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
[  519.160812]   [<000000000042ced4>] arch_cpu_idle+0x54/0xe0
[  519.160817]   [<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
[  519.160822]   [<0000000000870f18>] smp_callin+0x100/0x110
[  519.160825]   [<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
[  519.160827]   [<          (null)>]           (null)
[  519.160829] irq event stamp: 19
[  519.160832] hardirqs last  enabled at (19): [<0000000000877cc4>] _raw_spin_un
lock_irq+0x24/0x60
[  519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
ck_irq+0x10/0x80
[  519.160841] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
0x418/0x1080
[  519.160843] softirqs last disabled at (0): [<          (null)>]           (nu
ll)
[  519.160844] 
[  519.160844] other info that might help us debug this:
[  519.160844]  Possible unsafe locking scenario:
[  519.160844] 
[  519.160845]        CPU0
[  519.160845]        ----
[  519.160847]   lock(&irq_desc_lock_class);
[  519.160848]   <Interrupt>
[  519.160850]     lock(&irq_desc_lock_class);
[  519.160850] 
[  519.160850]  *** DEADLOCK ***
[  519.160850] 
[  519.160852] no locks held by irq/36-MSIQ/640.
[  519.160853] 
[  519.160853] stack backtrace:
[  519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
[  519.160856] Call Trace:
[  519.160860]  [00000000004b50b4] print_usage_bug+0x234/0x2e0
[  519.160862]  [00000000004b5728] mark_lock+0x5c8/0x800
[  519.160864]  [00000000004ba238] __lock_acquire+0x7b8/0xce0
[  519.160866]  [00000000004ba7c0] lock_acquire+0x60/0x100
[  519.160868]  [0000000000877930] _raw_spin_lock+0x30/0x80
[  519.160870]  [00000000004d314c] handle_simple_irq+0xc/0xe0
[  519.160872]  [00000000004cf118] generic_handle_irq+0x38/0x60
[  519.160877]  [0000000000447870] sparc64_msiq_interrupt+0x50/0x120
[  519.160880]  [00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
[  519.160883]  [00000000004d019c] irq_thread+0xdc/0x140
[  519.160888]  [0000000000489560] kthread+0x80/0xa0
[  519.160893]  [0000000000406104] ret_from_syscall+0x1c/0x2c
[  519.160894]  [0000000000000000]           (null)
[  519.160897] ------------[ cut here ]------------


what do you think?

- Allen

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

26.02.2014, 11:52, "Allen Pais" <allen.pais@oracle.com>:
> Kirill,
>
>>>>>>>     --- a/arch/sparc/mm/tsb.c
>>>>>>>     +++ b/arch/sparc/mm/tsb.c
>>>>>>>     @@ -6,6 +6,7 @@
>>>>>>>      #include <linux/kernel.h>
>>>>>>>      #include <linux/preempt.h>
>>>>>>>      #include <linux/slab.h>
>>>>>>>     +#include <linux/locallock.h>
>>>>>>>      #include <asm/page.h>
>>>>>>>      #include <asm/pgtable.h>
>>>>>>>      #include <asm/mmu_context.h>
>>>>>>>     @@ -14,6 +15,7 @@
>>>>>>>      #include <asm/oplib.h>
>>>>>    Yes, tb->active was set to zero.
>>>>    If tb->active is zero, flush_tsb_user() is never called, because of tlb_nr is permanently zero.
>>>   Sorry, my bad. tb->active was set to one when I ran the test with the above patch.
>
>   The CPU now does not stall, the change I did was remove debug lockdep from the config.
> Now the system runs(cyclicttest/hackbench) producing two of the below mentioned crashes.
>
> 1. This is as the messages says, sleeping in atomic context. Am not sure who's holding the lock.
> [53990.477387] kernel BUG at kernel/rtmutex.c:738!
> [53990.477393]               \|/ ____ \|/
> [53990.477393]               "@'/ .. \`@"
> [53990.477393]               /_| \__/ |_\
> [53990.477393]                  \__U_/
> [53990.477396] hackbench(11777): Kernel bad sw trap 5 [#2]
> [53990.477403] CPU: 35 PID: 11777 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
> [53990.477408] task: fffff80f931f9600 ti: fffff80f905ec000 task.ti: fffff80f905ec000
> [53990.477413] TSTATE: 0000004411e01600 TPC: 0000000000876ca4 TNPC: 0000000000876ca8 Y: 00000000    Tainted: G      D W
> [53990.477419] TPC: <rt_spin_lock_slowlock+0x304/0x340>
> [53990.477423] g0: 000000000000000e g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de1800
> [53990.477427] g4: fffff80f931f9600 g5: fffff80fd74a0000 g6: fffff80f905ec000 g7: 726e656c2f72746d
> [53990.477430] o0: 00000000009bcee8 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
> [53990.477434] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80f905ee6f1 ret_pc: 0000000000876c9c
> [53990.477439] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
> [53990.477444] l0: fffff80f905eefb0 l1: fffff80f931f9600 l2: fffff80f931f9c50 l3: 0000000000a8d800
> [53990.477448] l4: 0000000000000000 l5: 0000000000de1400 l6: 0000000000de1440 l7: 0000000000000001
> [53990.477452] i0: fffff80f9026ae70 i1: 0000000000000293 i2: 0000000000000000 i3: 0000000000000000
> [53990.477456] i4: 0000000000000002 i5: 0000000000000001 i6: fffff80f905ee831 i7: 0000000000876ee0
> [53990.477462] I7: <rt_spin_lock+0x20/0x60>
> [53990.477464] Call Trace:
> [53990.477470]  [0000000000876ee0] rt_spin_lock+0x20/0x60
> [53990.477476]  [000000000052ee60] unmap_single_vma+0x200/0x6c0
> [53990.477482]  [000000000052f348] unmap_vmas+0x28/0x60
> [53990.477488]  [0000000000531868] exit_mmap+0x88/0x160
> [53990.477492]  [000000000045e0e8] mmput+0x48/0x100
> [53990.477496]  [0000000000466a3c] do_exit+0x1fc/0xa40
> [53990.477500]  [0000000000427f00] die_if_kernel+0x1a0/0x340
> [53990.477506]  [00000000004294a8] sun4v_data_access_exception+0x108/0x120
> [53990.477512]  [0000000000406c08] sun4v_dacc+0x28/0x34
> [53990.477517]  [0000000000407b64] tsb_flush+0x4/0x40
> [53990.477523]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53990.477528]  [0000000000451800] tlb_batch_add+0x1e0/0x200
> [53990.477534]  [000000000053cad8] ptep_clear_flush+0x38/0x60
> [53990.477539]  [000000000052b47c] do_wp_page+0x1dc/0x880
> [53990.477544]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53990.477548]  [000000000052cab8] handle_mm_fault+0xd8/0x160
>
> and
>
> 2. [53998.070198] BUG: NMI Watchdog detected LOCKUP on CPU35, ip 0042f608, registers:
> [53998.070206] CPU: 35 PID: 11694 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #25
> [53998.070211] task: fffff80f91c20000 ti: fffff80f8f40c000 task.ti: fffff80f8f40c000
> [53998.070216] TSTATE: 0000000011e01606 TPC: 000000000042f608 TNPC: 000000000042f60c Y: 00000000    Tainted: G      D W
> [53998.070236] TPC: <stick_get_tick+0x8/0x20>
> [53998.070241] g0: 0000000000000000 g1: 000000000042f600 g2: 00000000076c64ec g3: 0000000007a9b280
> [53998.070246] g4: fffff80f91c20000 g5: fffff80fd74a0000 g6: fffff80f8f40c000 g7: 0000000000000000
> [53998.070251] o0: 0000000000000001 o1: fffff80f8f40c400 o2: 000000000042fa28 o3: 0000000000000000
> [53998.070255] o4: 000000000000004f o5: 0000000000000002 sp: fffff80f8f40ee01 ret_pc: 00000000004209f4
> [53998.070264] RPC: <tl0_irq15+0x14/0x20>
> [53998.070267] l0: 0000000000001000 l1: 0000000011001605 l2: 000000000042fa24 l3: 0000000000000400
> [53998.070270] l4: 000000000000000e l5: 0000000000000001 l6: 0000000000000000 l7: 0000000000000008
> [53998.070272] i0: 0000311023c1caaa i1: fffff80f8f40c400 i2: 000000000066f8b0 i3: 0000000000000000
> [53998.070275] i4: fffff80f8ab8e098 i5: fffff80f893f2a70 i6: fffff80f8f40eeb1 i7: 000000000042fa10
> [53998.070280] I7: <__delay+0x10/0x60>
> [53998.070282] Call Trace:
> [53998.070286]  [000000000042fa10] __delay+0x10/0x60
> [53998.070291]  [000000000066f8b8] do_raw_spin_lock+0xb8/0x120
> [53998.070300]  [0000000000877b08] _raw_spin_lock_irqsave+0x68/0xa0
> [53998.070306]  [0000000000452074] flush_tsb_user+0x14/0x120
> [53998.070309]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
> [53998.070312]  [0000000000451800] tlb_batch_add+0x1e0/0x200
> [53998.070325]  [000000000053cad8] ptep_clear_flush+0x38/0x60
> [53998.070328]  [000000000052b47c] do_wp_page+0x1dc/0x880
> [53998.070331]  [000000000052beac] handle_pte_fault+0x38c/0x7c0
> [53998.070334]  [000000000052cab8] handle_mm_fault+0xd8/0x160
> [53998.070339]  [0000000000879724] do_sparc64_fault+0x404/0x700
> [53998.070342]  [0000000000407ae0] sparc64_realfault_common+0x10/0x20
>
> But strangely, during boot-up I have more crash messages.
> Here's what I see
>
> [  520.570799] BUG: sleeping function called from invalid context at kernel/rtmu
> tex.c:659
> [  520.570802] in_atomic(): 0, irqs_disabled(): 1, pid: 2140, name: modprobe
> [  520.570803] INFO: lockdep is turned off.
> [  520.570805] irq event stamp: 4502
> [  520.570806] hardirqs last  enabled at (4501): [<00000000004d68c4>] rcu_note_c
> ontext_switch+0xa4/0x300
> [  520.570815] hardirqs last disabled at (4502): [<0000000000877a30>] _raw_spin_
> lock_irq+0x10/0x80
> [  520.570822] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [  520.570828] softirqs last disabled at (0): [<          (null)>]           (nu
> ll)
> [  520.570834] CPU: 18 PID: 2140 Comm: modprobe Tainted: G        W    3.10.24-r
> t22+ #25
> [  520.570835] Call Trace:
> [  520.570842]  [0000000000495f0c] __might_sleep+0xec/0x160
> [  520.570846]  [0000000000876ed8] rt_spin_lock+0x18/0x60
> [  520.570852]  [00000000006e0f78] sunhv_console_write_paged+0x1d8/0x200
> [  520.570855]  [00000000004625e0] call_console_drivers.clone.2+0x120/0x1c0
> [  520.570858]  [0000000000462a14] console_unlock+0x394/0x400
> [  520.570861]  [0000000000463108] vprintk_emit+0x3a8/0x5a0
> [  520.570863]  [0000000000874378] printk+0x38/0x4c
> [  520.570874]  [000000001024e78c] _base_make_ioc_operational+0xeac/0x1440 [mpt2
> sas]
> [  520.570882]  [0000000010253100] mpt2sas_base_attach+0x1720/0x1ae0 [mpt2sas]
> [  520.570893]  [000000001025b4fc] _scsih_probe+0x4fc/0x700 [mpt2sas]
> [  520.570900]  [0000000000686120] local_pci_probe+0x20/0x40
> [  520.570903]  [000000000068680c] pci_device_probe+0xec/0x100
> [  520.570907]  [00000000006ee574] driver_probe_device+0x74/0x220
> [  520.570909]  [00000000006ee7a8] __driver_attach+0x88/0xa0
> [  520.570913]  [00000000006eca0c] bus_for_each_dev+0x6c/0xa0
> [  520.570916]  [00000000006ee39c] driver_attach+0x1c/0x40

This should be fixed like that's done for 8250
[patch drivers-serial-cleanup-locking-for-rt.patch]


> and this one
>
> [  519.160755] =================================
> [  519.160756] [ INFO: inconsistent lock state ]
> [  519.160760] 3.10.24-rt22+ #25 Not tainted
> [  519.160761] ---------------------------------
> [  519.160763] inconsistent {IN-HARDIRQ-W} -> {HARDIRQ-ON-W} usage.
> [  519.160766] irq/36-MSIQ/640 [HC0[0]:SC0[0]:HE1:SE1] takes:
> [  519.160778]  (&irq_desc_lock_class){?.....}, at: [<00000000004d314c>] handle_
> simple_irq+0xc/0xe0
> [  519.160779] {IN-HARDIRQ-W} state was registered at:
> [  519.160785]   [<00000000004ba7c0>] lock_acquire+0x60/0x100
> [  519.160791]   [<0000000000877930>] _raw_spin_lock+0x30/0x80
> [  519.160795]   [<00000000004d2e6c>] handle_fasteoi_irq+0xc/0x180
> [  519.160800]   [<00000000004cf118>] generic_handle_irq+0x38/0x60
> [  519.160804]   [<000000000087bf44>] handler_irq+0xc4/0x100
> [  519.160808]   [<0000000000426b2c>] valid_addr_bitmap_patch+0x74/0x288
> [  519.160812]   [<000000000042ced4>] arch_cpu_idle+0x54/0xe0
> [  519.160817]   [<00000000004a85bc>] cpu_startup_entry+0x19c/0x340
> [  519.160822]   [<0000000000870f18>] smp_callin+0x100/0x110
> [  519.160825]   [<0000000000870a78>] after_lock_tlb+0x1ac/0x1c4
> [  519.160827]   [<          (null)>]           (null)
> [  519.160829] irq event stamp: 19
> [  519.160832] hardirqs last  enabled at (19): [<0000000000877cc4>] _raw_spin_un
> lock_irq+0x24/0x60
> [  519.160835] hardirqs last disabled at (18): [<0000000000877a30>] _raw_spin_lo
> ck_irq+0x10/0x80
> [  519.160841] softirqs last  enabled at (0): [<000000000045eb58>] copy_process+
> 0x418/0x1080
> [  519.160843] softirqs last disabled at (0): [<          (null)>]           (nu
> ll)
> [  519.160844]
> [  519.160844] other info that might help us debug this:
> [  519.160844]  Possible unsafe locking scenario:
> [  519.160844]
> [  519.160845]        CPU0
> [  519.160845]        ----
> [  519.160847]   lock(&irq_desc_lock_class);
> [  519.160848]   <Interrupt>
> [  519.160850]     lock(&irq_desc_lock_class);
> [  519.160850]
> [  519.160850]  *** DEADLOCK ***
> [  519.160850]
> [  519.160852] no locks held by irq/36-MSIQ/640.
> [  519.160853]
> [  519.160853] stack backtrace:
> [  519.160855] CPU: 9 PID: 640 Comm: irq/36-MSIQ Not tainted 3.10.24-rt22+ #25
> [  519.160856] Call Trace:
> [  519.160860]  [00000000004b50b4] print_usage_bug+0x234/0x2e0
> [  519.160862]  [00000000004b5728] mark_lock+0x5c8/0x800
> [  519.160864]  [00000000004ba238] __lock_acquire+0x7b8/0xce0
> [  519.160866]  [00000000004ba7c0] lock_acquire+0x60/0x100
> [  519.160868]  [0000000000877930] _raw_spin_lock+0x30/0x80
> [  519.160870]  [00000000004d314c] handle_simple_irq+0xc/0xe0
> [  519.160872]  [00000000004cf118] generic_handle_irq+0x38/0x60
> [  519.160877]  [0000000000447870] sparc64_msiq_interrupt+0x50/0x120
> [  519.160880]  [00000000004d05fc] irq_forced_thread_fn+0x1c/0x80
> [  519.160883]  [00000000004d019c] irq_thread+0xdc/0x140
> [  519.160888]  [0000000000489560] kthread+0x80/0xa0
> [  519.160893]  [0000000000406104] ret_from_syscall+0x1c/0x2c
> [  519.160894]  [0000000000000000]           (null)
> [  519.160897] ------------[ cut here ]------------
>
> what do you think?
>
> - Allen
--
To unsubscribe from this list: send the line "unsubscribe sparclinux" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Fri, 28 Feb 2014 18:51:23 +0400

> This should be fixed like that's done for 8250
> [patch drivers-serial-cleanup-locking-for-rt.patch]

Like the patch below?

Is this transformation legitimate outside of the -rt tree?  I wish the commit
messages for the 8250 change was more verbose, explaining exactly why this
is needed.

And if it's needed, all the sparc serial/console drivers need this too.

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: David Miller <davem@davemloft.net>
Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)

> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Here's an updated patch that takes care of all of the Sun serial
drivers, and is compile tested:

--------------------
sparc: serial: Clean up the locking for -rt

Signed-off-by: David S. Miller <davem@davemloft.net>

diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
index cf86e72..dc697ce 100644
--- a/drivers/tty/serial/sunhv.c
+++ b/drivers/tty/serial/sunhv.c
@@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (port->sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&port->lock);
-	} else
-		spin_lock(&port->lock);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 
 	while (n > 0) {
 		unsigned long ra = __pa(con_write_page);
@@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static inline void sunhv_console_putchar(struct uart_port *port, char c)
@@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	unsigned long flags;
 	int i, locked = 1;
 
-	local_irq_save(flags);
+	if (port->sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&port->lock, flags);
+	else
+		spin_lock_irqsave(&port->lock, flags);
 	if (port->sysrq) {
 		locked = 0;
 	} else if (oops_in_progress) {
@@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
 	}
 
 	if (locked)
-		spin_unlock(&port->lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&port->lock, flags);
 }
 
 static struct console sunhv_console = {
diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
index 380fb53..5faa8e9 100644
--- a/drivers/tty/serial/sunsab.c
+++ b/drivers/tty/serial/sunsab.c
@@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, n, sunsab_console_putchar);
 	sunsab_tec_wait(up);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int sunsab_console_setup(struct console *con, char *options)
diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
index db79b76..9a0f24f 100644
--- a/drivers/tty/serial/sunsu.c
+++ b/drivers/tty/serial/sunsu.c
@@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
 	unsigned int ier;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	/*
 	 *	First save the UER then disable the interrupts
@@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
 	serial_out(up, UART_IER, ier);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 /*
diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
index 45a8c6a..a2c40ed 100644
--- a/drivers/tty/serial/sunzilog.c
+++ b/drivers/tty/serial/sunzilog.c
@@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
 	unsigned long flags;
 	int locked = 1;
 
-	local_irq_save(flags);
-	if (up->port.sysrq) {
-		locked = 0;
-	} else if (oops_in_progress) {
-		locked = spin_trylock(&up->port.lock);
-	} else
-		spin_lock(&up->port.lock);
+	if (up->port.sysrq || oops_in_progress)
+		locked = spin_trylock_irqsave(&up->port.lock, flags);
+	else
+		spin_lock_irqsave(&up->port.lock, flags);
 
 	uart_console_write(&up->port, s, count, sunzilog_putchar);
 	udelay(2);
 
 	if (locked)
-		spin_unlock(&up->port.lock);
-	local_irq_restore(flags);
+		spin_unlock_irqrestore(&up->port.lock, flags);
 }
 
 static int __init sunzilog_console_setup(struct console *con, char *options)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
> From: David Miller <davem@davemloft.net>
> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
> 
>> From: Kirill Tkhai <tkhai@yandex.ru>
>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>
>>> This should be fixed like that's done for 8250
>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>
>> Like the patch below?

tested. Sounds good to me.
Tested-by: Allen Pais <allen.pais@oracle.com>

> 
> Here's an updated patch that takes care of all of the Sun serial
> drivers, and is compile tested:
> 
> --------------------
> sparc: serial: Clean up the locking for -rt
> 
> Signed-off-by: David S. Miller <davem@davemloft.net>
> 
> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> diff --git a/drivers/tty/serial/sunsab.c b/drivers/tty/serial/sunsab.c
> index 380fb53..5faa8e9 100644
> --- a/drivers/tty/serial/sunsab.c
> +++ b/drivers/tty/serial/sunsab.c
> @@ -844,20 +844,16 @@ static void sunsab_console_write(struct console *con, const char *s, unsigned n)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, n, sunsab_console_putchar);
>  	sunsab_tec_wait(up);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int sunsab_console_setup(struct console *con, char *options)
> diff --git a/drivers/tty/serial/sunsu.c b/drivers/tty/serial/sunsu.c
> index db79b76..9a0f24f 100644
> --- a/drivers/tty/serial/sunsu.c
> +++ b/drivers/tty/serial/sunsu.c
> @@ -1295,13 +1295,10 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	unsigned int ier;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	/*
>  	 *	First save the UER then disable the interrupts
> @@ -1319,8 +1316,7 @@ static void sunsu_console_write(struct console *co, const char *s,
>  	serial_out(up, UART_IER, ier);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  /*
> diff --git a/drivers/tty/serial/sunzilog.c b/drivers/tty/serial/sunzilog.c
> index 45a8c6a..a2c40ed 100644
> --- a/drivers/tty/serial/sunzilog.c
> +++ b/drivers/tty/serial/sunzilog.c
> @@ -1195,20 +1195,16 @@ sunzilog_console_write(struct console *con, const char *s, unsigned int count)
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (up->port.sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&up->port.lock);
> -	} else
> -		spin_lock(&up->port.lock);
> +	if (up->port.sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&up->port.lock, flags);
> +	else
> +		spin_lock_irqsave(&up->port.lock, flags);
>  
>  	uart_console_write(&up->port, s, count, sunzilog_putchar);
>  	udelay(2);
>  
>  	if (locked)
> -		spin_unlock(&up->port.lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&up->port.lock, flags);
>  }
>  
>  static int __init sunzilog_console_setup(struct console *con, char *options)
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:00:20 +0530

> On Wednesday 05 March 2014 01:58 AM, David Miller wrote:
>> From: David Miller <davem@davemloft.net>
>> Date: Tue, 04 Mar 2014 14:10:03 -0500 (EST)
>> 
>>> From: Kirill Tkhai <tkhai@yandex.ru>
>>> Date: Fri, 28 Feb 2014 18:51:23 +0400
>>>
>>>> This should be fixed like that's done for 8250
>>>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>>>
>>> Like the patch below?
> 
> tested. Sounds good to me.
> Tested-by: Allen Pais <allen.pais@oracle.com>

Great, I've queued this up into my tree.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-06 16:36:50 [-0500]:

>Great, I've queued this up into my tree.

And I just took that in my -RT.

Sebastian

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

Hi, David!

On 04.03.2014 23:10, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Fri, 28 Feb 2014 18:51:23 +0400
> 
>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
> 
> Like the patch below?

Exactly this.

> Is this transformation legitimate outside of the -rt tree?  I wish the commit
> messages for the 8250 change was more verbose, explaining exactly why this
> is needed.
> 
> And if it's needed, all the sparc serial/console drivers need this too.

port->lock is mutex, so we should not disable interrupts before we are
doing spin_lock(). In RT spin_trylock_irqsave() does not disable irqs.

The patch is made to be good for both RT and !RT cases, so everything
is ok!

Really I don't know, why the original patch is not in upstream
***shrugging***. It's from 2009, everything is possible.


> diff --git a/drivers/tty/serial/sunhv.c b/drivers/tty/serial/sunhv.c
> index cf86e72..dc697ce 100644
> --- a/drivers/tty/serial/sunhv.c
> +++ b/drivers/tty/serial/sunhv.c
> @@ -433,13 +433,10 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	unsigned long flags;
>  	int locked = 1;
>  
> -	local_irq_save(flags);
> -	if (port->sysrq) {
> -		locked = 0;
> -	} else if (oops_in_progress) {
> -		locked = spin_trylock(&port->lock);
> -	} else
> -		spin_lock(&port->lock);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  
>  	while (n > 0) {
>  		unsigned long ra = __pa(con_write_page);
> @@ -470,8 +467,7 @@ static void sunhv_console_write_paged(struct console *con, const char *s, unsign
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static inline void sunhv_console_putchar(struct uart_port *port, char c)
> @@ -492,7 +488,10 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	unsigned long flags;
>  	int i, locked = 1;
>  
> -	local_irq_save(flags);
> +	if (port->sysrq || oops_in_progress)
> +		locked = spin_trylock_irqsave(&port->lock, flags);
> +	else
> +		spin_lock_irqsave(&port->lock, flags);
>  	if (port->sysrq) {
>  		locked = 0;
>  	} else if (oops_in_progress) {
> @@ -507,8 +506,7 @@ static void sunhv_console_write_bychar(struct console *con, const char *s, unsig
>  	}
>  
>  	if (locked)
> -		spin_unlock(&port->lock);
> -	local_irq_restore(flags);
> +		spin_unlock_irqrestore(&port->lock, flags);
>  }
>  
>  static struct console sunhv_console = {
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:10:03 [-0500]:

>> This should be fixed like that's done for 8250
>> [patch drivers-serial-cleanup-locking-for-rt.patch]
>
>Like the patch below?

The patch looks good.

>Is this transformation legitimate outside of the -rt tree?  I wish the commit
>messages for the 8250 change was more verbose, explaining exactly why this
>is needed.

Looking at it, it looks like the transformation is legitimate in
vanila, too.

In -RT the spin_lock_irqsave() does not spin but sleep if the lock is
taken. Before that, local_irq_save() is invoked which disables
interrupts even on -RT. That is why you might_sleep() triggers a warning
here.
In the ->sysrq and oops_in_progress case it is save to trylock the lock
i.e. this is what we do now anyway except for ->sysrq where we assume
that the lock is already taken.

The spin_lock_irqsave() grabs the lock and disables the interrupts on
vanila (the same behavior) and on -RT it won't disable interrupts.

>And if it's needed, all the sparc serial/console drivers need this too.

Let me forward the 8250 to GregKH and see how he reacts. Would be nice
if it is another patch that gets removed from that queue…

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 19 Feb 2014 13:25:38 +0400

> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
> You may get good stack without sun4v_data_access_exception error, which was in the first or second
> message.

My suspicion is that what happens when we get the data access error is
that we sample the tlb batch count as non-zero, preempt, then come
back from preemption seeing the tlb batch in a completely different state.

And that's what leads to the crash, in the one trace I saw the TSB address
passed to tsb_flush() (register %o0) was some garbage like 0x103.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 05.03.2014 00:03, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 19 Feb 2014 13:25:38 +0400
> 
>> It seems for me it's better to decide the problem not changing protector of tsb like in patch above.
>> You may get good stack without sun4v_data_access_exception error, which was in the first or second
>> message.
> 
> My suspicion is that what happens when we get the data access error is
> that we sample the tlb batch count as non-zero, preempt, then come
> back from preemption seeing the tlb batch in a completely different state.
> 
> And that's what leads to the crash, in the one trace I saw the TSB address
> passed to tsb_flush() (register %o0) was some garbage like 0x103.
> 

I suggested to set tb_active to zero just for experiment. This way

diff --git a/arch/sparc/mm/tlb.c b/arch/sparc/mm/tlb.c
index b12cb5e..e1d1fd6 100644
--- a/arch/sparc/mm/tlb.c
+++ b/arch/sparc/mm/tlb.c
@@ -54,7 +54,7 @@ void arch_enter_lazy_mmu_mode(void)
 {
 	struct tlb_batch *tb = &__get_cpu_var(tlb_batch);

-	tb->active = 1;
+	tb->active = 0;
 }

 void arch_leave_lazy_mmu_mode(void)

Last Allen's stack (from 26 feb. 11:52) still contains
flush_tlb_pending(). Strange, why this is so, maybe
bad initialized per-cpu tlb_batch, and something bad is with BSS...

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 19 Feb 2014 09:23:46 +0530

>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
> with hackbench and dd, the system did not stall :)

If using a local lock fixes your problem, it just means we're being
preempted during flush_tsb_user() which is absolutely not allowed.

The batch is a per-cpu object, and must be forcibly flushed if we
preempt.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Allen Pais]

On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
> From: Allen Pais <allen.pais@oracle.com>
> Date: Wed, 19 Feb 2014 09:23:46 +0530
> 
>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>> with hackbench and dd, the system did not stall :)
> 
> If using a local lock fixes your problem, it just means we're being
> preempted during flush_tsb_user() which is absolutely not allowed.

David, Its my bad, I realized my mistake of using local locks.
Am still looking into the tsb flush code. It's left me sleepless. :)
 

> The batch is a per-cpu object, and must be forcibly flushed if we
> preempt.
> 

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Allen Pais <allen.pais@oracle.com>
Date: Wed, 05 Mar 2014 10:04:56 +0530

> On Wednesday 05 March 2014 01:31 AM, David Miller wrote:
>> From: Allen Pais <allen.pais@oracle.com>
>> Date: Wed, 19 Feb 2014 09:23:46 +0530
>> 
>>>  I realized locking in tsb is very tricky. My attempts to try and get hackbench run
>>> without causing a stall failed. So here's what I tried to fix it, am not sure if it's
>>> an appropriate fix, I would love to get comments. I have tested this fix for over 24 hours
>>> with hackbench and dd, the system did not stall :)
>> 
>> If using a local lock fixes your problem, it just means we're being
>> preempted during flush_tsb_user() which is absolutely not allowed.
> 
> David, Its my bad, I realized my mistake of using local locks.
> Am still looking into the tsb flush code. It's left me sleepless. :)

So let's do a simple test, in flush_tlb_user() assert to make sure we
cannot be preempted by the -rt kernel.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 15:43:06 +0400

> For example, collect only batches which does not require smp call function. Or the
> main goal of lazy tlb was to prevent smp calls?! It's good to discover this..

The goal of the lazy tlb stuff is to only have one (potential)
synchronization point with other cpus.

Then we sweep away the TSB entries, after which any TLB miss must
enter the full fault path and synchronize with the current thread
doing the flush.

Then we kill the TLB entries, both local and remote.

The tsb_grow() code path is quite the animal, as any of you who have
read the comment above it's implementation can attest :-) It took 6
months to get that code right in a non-RT context back when it was
originally written.

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[David Miller]

From: Kirill Tkhai <tkhai@yandex.ru>
Date: Wed, 12 Feb 2014 12:33:58 +0400

> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
> 
>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>   with the following trace:
>>>>
>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>   [ 1487.027885] Call Trace:
>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>  Now, consistently I've been getting sun4v_data_access_exception.
>>  Here's the trace:
>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
> 
> I've never dived at sparc's tlb before, but it seems now I'm understanding.
> 
> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
> you collect flush requests before you really flush all of them.
> 
> In RT you collect them too, but you are able to be preempted in any moment.
> So, you may switch to other process with unflushed tlb, which is very bad.
> 
> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
> We will look if this robust fix helps.

Sorry for coming into this discussion so late.

Indeed, the pending flushes are per-cpu and we must flush them out in the
event of a preemption.

PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
fact that's where I copied the logic from.  Does PowerPC not work with
-rt? :-)
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Kirill Tkhai]

On 04.03.2014 23:55, David Miller wrote:
> From: Kirill Tkhai <tkhai@yandex.ru>
> Date: Wed, 12 Feb 2014 12:33:58 +0400
> 
>> 12.02.2014, 11:48, "Allen Pais" <allen.pais@oracle.com>:
>>
>>>  On Wednesday 12 February 2014 02:43 AM, Kirill Tkhai wrote:
>>>>   06.01.2014, 07:56, "Allen Pais" <allen.pais@oracle.com>:
>>>>>   In the attempt of get PREEMPT_RT working on sparc64 using
>>>>>   linux-stable-rt version 3.10.22-rt19+, the kernel crash
>>>>>   with the following trace:
>>>>>
>>>>>   [ 1487.027884] I7: <rt_mutex_setprio+0x3c/0x2c0>
>>>>>   [ 1487.027885] Call Trace:
>>>>>   [ 1487.027887]  [00000000004967dc] rt_mutex_setprio+0x3c/0x2c0
>>>>>   [ 1487.027892]  [00000000004afe20] task_blocks_on_rt_mutex+0x180/0x200
>>>>>   [ 1487.027895]  [0000000000819114] rt_spin_lock_slowlock+0x94/0x300
>>>>>   [ 1487.027897]  [0000000000817ebc] __schedule+0x39c/0x53c
>>>>>   [ 1487.027899]  [00000000008185fc] schedule+0x1c/0xc0
>>>>>   [ 1487.027908]  [000000000048fff4] smpboot_thread_fn+0x154/0x2e0
>>>>>   [ 1487.027913]  [000000000048753c] kthread+0x7c/0xa0
>>>>>   [ 1487.027920]  [00000000004060c4] ret_from_syscall+0x1c/0x2c
>>>>>   [ 1487.027922]  [0000000000000000]           (null)
>>>  Now, consistently I've been getting sun4v_data_access_exception.
>>>  Here's the trace:
>>>  [ 4673.360121] sun4v_data_access_exception: ADDR[0000080000000000] CTX[0000] TYPE[0004], going.
>>
>> I've never dived at sparc's tlb before, but it seems now I'm understanding.
>>
>> arch_enter_lazy_mmu_mode() makes possible delayed tlb flushing. In !RT kernel
>> you collect flush requests before you really flush all of them.
>>
>> In RT you collect them too, but you are able to be preempted in any moment.
>> So, you may switch to other process with unflushed tlb, which is very bad.
>>
>> Try to not to set tb->active = 1; in arch_enter_lazy_mmu_mode(). Set it to zero.
>> We will look if this robust fix helps.
> 
> Sorry for coming into this discussion so late.
> 
> Indeed, the pending flushes are per-cpu and we must flush them out in the
> event of a preemption.
> 
> PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
> fact that's where I copied the logic from.  Does PowerPC not work with
> -rt? :-)
> 

It does not work, but we will :)

Re: [PATCH 3/4] sparc64: convert spinlock_t to raw_spinlock_t in mmu_context_t

[Sebastian Andrzej Siewior]

* David Miller | 2014-03-04 14:55:23 [-0500]:

>PowerPC does the same exact thing with arch_enter_lazy_mmu_mode(), in
>fact that's where I copied the logic from.  Does PowerPC not work with
>-rt? :-)

It is only PPC64 as far as I can see and I have mostly 32bit book-e
around. That preempt-lazy stuff also touches PPC64 bits so I assume
PPC64 works.

Sebastian

[PATCH 0/4] PREEMPT_RT support for sparc64

[Allen Pais]

PREEMPT_RT support for sparc64
All the patches are based on v3.10.24-rt22
commit e88b1c51941d964e77ef7e9099075ae2e8dafd75

Have tested it on UltraSparc T4 (Niagara4). 

Allen Pais (4):
  [V1] sparc64: use generic rwsem spinlocks rt
  [V1] sparc64: support forced irq threading
  [V1] sparc64: convert spinloc_t to raw_spinlock_t in mmu_context_t
  [V1] sparc64: convert ctx_alloc_lock raw_spinlock_t

 arch/sparc/Kconfig                      |    7 +++----
 arch/sparc/include/asm/mmu_64.h         |    2 +-
 arch/sparc/include/asm/mmu_context_64.h |   10 +++++-----
 arch/sparc/kernel/smp_64.c              |    4 ++--
 arch/sparc/mm/init_64.c                 |   14 +++++++-------
 arch/sparc/mm/tsb.c                     |   20 ++++++++++----------
 6 files changed, 28 insertions(+), 29 deletions(-)

-- 
1.7.10.4