Re: Cyclictest results on Sparc64 with PREEMPT_RT

[Allen Pais <allen.pais@oracle.com>]

[-- Attachment #1: Type: text/plain, Size: 3797 bytes --]

Sebastian,
> 
> This is the stack where the NMI got injected. Not sure why this is more
> than once here but then this the sparc64 trace I see :)
> 
>> [ 1150.135604]  [00000000004acc00] in_lock_functions+0x0/0x40
>> [ 1150.135608]  [000000000080a038] add_preempt_count+0xd8/0x140
>> [ 1150.135610]  [000000000080617c] __schedule+0x1c/0x500
>> [ 1150.135613]  [0000000000806b7c] schedule+0x1c/0xc0
>> [ 1150.135615]  [0000000000806f8c] schedule_preempt_disabled+0xc/0x40
>> [ 1150.135617]  [000000000049dd10] cpu_startup_entry+0x150/0x300
> 
> And this where the CPU was before the NMI. Doesn't look blocking.
> in_lock_functions() compares a few values no locking involved so the CPU
> probably was here while the NMI hit and an usec later it might be an
> instruction later. What I thnig is odd, is that it is exactly at the
> begin of the function, not an instruction later.
> 

I haven't made much progress yet. These appear when the machine is under
stress(hackbench/dd). There's also another issue that popped up while
I ran hack bench, here's the brief trace

[ 6694.884398] kernel BUG at kernel/rtmutex.c:738!
[ 6694.884402]               \|/ ____ \|/
[ 6694.884402]               "@'/ .. \`@"
[ 6694.884402]               /_| \__/ |_\
[ 6694.884402]                  \__U_/
[ 6694.884403] hackbench(18821): Kernel bad sw trap 5 [#2]
[ 6694.884408] CPU: 8 PID: 18821 Comm: hackbench Tainted: G      D W    3.10.24-rt22+ #11
[ 6694.884410] task: fffff80f8f4a2580 ti: fffff80f8ebd4000 task.ti: fffff80f8ebd4000
[ 6694.884413] TSTATE: 0000004411001603 TPC: 0000000000878ec4 TNPC: 0000000000878ec8 Y: 00000000    Tainted: G      D W   
[ 6694.884425] TPC: <rt_spin_lock_slowlock+0x304/0x340>
[ 6694.884427] g0: 0000000000000000 g1: 0000000000000000 g2: 0000000000000000 g3: 0000000000de5800
[ 6694.884429] g4: fffff80f8f4a2580 g5: fffff80fd089c000 g6: fffff80f8ebd4000 g7: 726e656c2f72746d
[ 6694.884430] o0: 00000000009bfaf0 o1: 00000000000002e2 o2: 0000000000000000 o3: 0000000000000001
[ 6694.884432] o4: 0000000000000002 o5: 0000000000000000 sp: fffff80fff9b70d1 ret_pc: 0000000000878ebc
[ 6694.884434] RPC: <rt_spin_lock_slowlock+0x2fc/0x340>
[ 6694.884437] l0: fffff80fff9b7990 l1: fffff80f8f4a2580 l2: fffff80f8f4a2bd0 l3: 000001001fb75040
[ 6694.884438] l4: 0000000000000000 l5: 0000000000e25c00 l6: 0000000000000008 l7: 0000000000000008
[ 6694.884440] i0: fffff80f97836070 i1: 0000000000512400 i2: 0000000000000001 i3: 0000000000000000
[ 6694.884441] i4: 0000000000000002 i5: 0000000000000000 i6: fffff80fff9b7211 i7: 00000000008790ac
[ 6694.884444] I7: <rt_spin_lock+0xc/0x40>
[ 6694.884445] Call Trace:
[ 6694.884448]  [00000000008790ac] rt_spin_lock+0xc/0x40
[ 6694.884454]  [000000000052e30c] unmap_single_vma+0x1ec/0x6c0
[ 6694.884456]  [000000000052e808] unmap_vmas+0x28/0x60
[ 6694.884459]  [0000000000530cc8] exit_mmap+0x88/0x160
[ 6694.884465]  [000000000045e0d4] mmput+0x34/0xe0
[ 6694.884469]  [00000000004669fc] do_exit+0x1fc/0xa40
[ 6694.884473]  [000000000087a650] perfctr_irq+0x3d0/0x420
[ 6694.884477]  [00000000004209f4] tl0_irq15+0x14/0x20
[ 6694.884482]  [0000000000671e4c] do_raw_spin_lock+0xac/0x120
[ 6694.884485]  [0000000000879cc8] _raw_spin_lock_irqsave+0x68/0xa0
[ 6694.884488]  [0000000000452074] flush_tsb_user+0x14/0x120
[ 6694.884490]  [00000000004515a8] flush_tlb_pending+0x68/0xe0
[ 6694.884492]  [0000000000451800] tlb_batch_add+0x1e0/0x200
[ 6694.884496]  [000000000053bef8] ptep_clear_flush+0x38/0x60
[ 6694.884498]  [000000000052a9fc] do_wp_page+0x1dc/0x860
[ 6694.884500]  [000000000052b3f8] handle_pte_fault+0x378/0x7c0

These are the two issues I have ran into with stress. Otherwise the machine is quite stable
with light load(compress/decompress and building the kernel).

Attached are the graphs of the system on light load.

Thanks,

Allen


[-- Attachment #2: plot_OL.png --]
[-- Type: image/png, Size: 3800 bytes --]

[-- Attachment #3: plot_RT.png --]
[-- Type: image/png, Size: 3548 bytes --]