sparc: deadlock using perf with sched tracepoints and user callstacks

sparc: deadlock using perf with sched tracepoints and user callstacks

[David Ahern]

I am hitting a deadlock with 'perf sched record -g':

[ 8138.140030] ------------[ cut here ]------------
[ 8138.159054] WARNING: CPU: 758 PID: 12488 at 
/opt/dahern/linux.git/arch/sparc/kernel/nmi.c:80 perfctr_irq+0x1f8/0x2b4()
[ 8138.203152] Watchdog detected hard LOCKUP on cpu 758
[ 8138.222874] Modules linked in: ipt_REJECT nf_reject_ipv4 
nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT 
nf_reject_ipv6 xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_state 
nf_conntrack ip6table_filter ip6_tables x_tables ipv6 cdc_ether usbnet 
mii ixgbe mdio igb i2c_algo_bit i2c_core ptp crc32c_sparc64 
camellia_sparc64 des_sparc64 des_generic md5_sparc64 sha512_sparc64 
sha1_sparc64 uio_pdrv_genirq uio usb_storage mpt3sas scsi_transport_sas 
raid_class sha256_sparc64 aes_sparc64 sunvnet sunvdc
[ 8138.410969] CPU: 758 PID: 12488 Comm: perf Not tainted 4.0.0-rc6+ #6
[ 8138.437146] Call Trace:
[ 8138.447193]  [000000000045cdd4] warn_slowpath_common+0x7c/0xa0
[ 8138.471238]  [000000000045ce90] warn_slowpath_fmt+0x30/0x40
[ 8138.494189]  [0000000000983e38] perfctr_irq+0x1f8/0x2b4
[ 8138.515716]  [00000000004209f4] tl0_irq15+0x14/0x20
[ 8138.535791]  [00000000009839ec] _raw_spin_trylock_bh+0x68/0x108
[ 8138.560180]  [0000000000980018] __schedule+0xcc/0x710
[ 8138.580981]  [00000000009806dc] preempt_schedule_common+0x10/0x3c
[ 8138.606082]  [000000000098077c] _cond_resched+0x34/0x44
[ 8138.627603]  [0000000000565990] kmem_cache_alloc_node+0x24/0x1a0
[ 8138.652345]  [0000000000450b60] tsb_grow+0xac/0x488
[ 8138.672429]  [0000000000985040] do_sparc64_fault+0x4dc/0x6e4
[ 8138.695736]  [0000000000407c2c] sparc64_realfault_common+0x10/0x20
[ 8138.721202]  [00000000006f2e24] NG4copy_from_user+0xa4/0x3c0
[ 8138.744510]  [000000000044f900] perf_callchain_user+0x5c/0x6c
[ 8138.768182]  [0000000000517b5c] perf_callchain+0x16c/0x19c
[ 8138.790774]  [0000000000515f84] perf_prepare_sample+0x68/0x218
[ 8138.814801] ---[ end trace 42ca6294b1ff7573 ]---

The scheduler tracepoints can be hit with the run queue lock taken. Per 
the above while walking the userspace callchains page faults can cause 
paths where __schedule gets called again. The deadlock happens because 
it wants the runqueue lock again.

For PowerPC (b59a1bfcc2406ea75346977ad016cfe909a762ac) the solution was 
to disable pagefaults while walking userspace stacks. That does not work 
for sparc; rather that change results in process terminating due to a 
SIGBUS:

diff --git a/arch/sparc/kernel/perf_event.c b/arch/sparc/kernel/perf_event.c
index 86eebfa3b158..ebe0287e7542 100644
--- a/arch/sparc/kernel/perf_event.c
+++ b/arch/sparc/kernel/perf_event.c
@@ -21,7 +21,7 @@

  #include <asm/stacktrace.h>
  #include <asm/cpudata.h>
-#include <asm/uaccess.h>
+#include <linux/uaccess.h>
  #include <linux/atomic.h>
  #include <asm/nmi.h>
  #include <asm/pcr.h>
@@ -1832,8 +1832,11 @@ perf_callchain_user(struct perf_callchain_entry 
*entry, struct pt_regs *regs)
                 return;

         flushw_user();
+
+       pagefault_disable();
         if (test_thread_flag(TIF_32BIT))
                 perf_callchain_user_32(entry, regs);
         else
                 perf_callchain_user_64(entry, regs);
+       pagefault_enable();
  }


Perhaps its the pending PTO but I have not been able to find a solution. 
Any ideas? How do I avoid page faults on sparc? (i.e., if page is not 
present just terminate walking the stack)

David

Re: sparc: deadlock using perf with sched tracepoints and user callstacks

[David Miller]

From: David Ahern <david.ahern@oracle.com>
Date: Tue, 31 Mar 2015 13:49:46 -0600

> The scheduler tracepoints can be hit with the run queue lock
> taken. Per the above while walking the userspace callchains page
> faults can cause paths where __schedule gets called again. The
> deadlock happens because it wants the runqueue lock again.
> 
> For PowerPC (b59a1bfcc2406ea75346977ad016cfe909a762ac) the solution
> was to disable pagefaults while walking userspace stacks. That does
> not work for sparc; rather that change results in process terminating
> due to a SIGBUS:

What is different in the PowerPC fault path vs. the Sparc one that
causes the SIGBUS for sparc where the PowerPC does not see it?

I don't understand where the SIGBUS comes from.  In a pagefault
disable section on sparc, in_atomic() should be true, thus we'll call
do_kernel_fault(), fall through to the TSTATE_PRIV code path and
simply run the exception handler.

Re: sparc: deadlock using perf with sched tracepoints and user callstacks

[David Ahern]

On 3/31/15 5:38 PM, David Miller wrote:
> What is different in the PowerPC fault path vs. the Sparc one that
> causes the SIGBUS for sparc where the PowerPC does not see it?
>
> I don't understand where the SIGBUS comes from.  In a pagefault
> disable section on sparc, in_atomic() should be true, thus we'll call
> do_kernel_fault(), fall through to the TSTATE_PRIV code path and
> simply run the exception handler.
>

That's what I was hoping someone could explain to me. I'll dig deeper on 
Monday.

Re: sparc: deadlock using perf with sched tracepoints and user callstacks

[David Miller]

From: David Ahern <david.ahern@oracle.com>
Date: Tue, 31 Mar 2015 17:52:53 -0600

> On 3/31/15 5:38 PM, David Miller wrote:
>> What is different in the PowerPC fault path vs. the Sparc one that
>> causes the SIGBUS for sparc where the PowerPC does not see it?
>>
>> I don't understand where the SIGBUS comes from.  In a pagefault
>> disable section on sparc, in_atomic() should be true, thus we'll call
>> do_kernel_fault(), fall through to the TSTATE_PRIV code path and
>> simply run the exception handler.
>>
> 
> That's what I was hoping someone could explain to me. I'll dig deeper
> on Monday.

I've looked a few times at this and can't figure out what might be
going wrong just from visual inspection.  I'll try reproducing the
problem and adding some diagnostics.

Re: sparc: deadlock using perf with sched tracepoints and user callstacks

[David Ahern]

On 4/22/15 5:22 PM, David Miller wrote:
> From: David Ahern <david.ahern@oracle.com>
> Date: Tue, 31 Mar 2015 17:52:53 -0600
>
>> On 3/31/15 5:38 PM, David Miller wrote:
>>> What is different in the PowerPC fault path vs. the Sparc one that
>>> causes the SIGBUS for sparc where the PowerPC does not see it?
>>>
>>> I don't understand where the SIGBUS comes from.  In a pagefault
>>> disable section on sparc, in_atomic() should be true, thus we'll call
>>> do_kernel_fault(), fall through to the TSTATE_PRIV code path and
>>> simply run the exception handler.
>>>
>>
>> That's what I was hoping someone could explain to me. I'll dig deeper
>> on Monday.
>
> I've looked a few times at this and can't figure out what might be
> going wrong just from visual inspection.  I'll try reproducing the
> problem and adding some diagnostics.
>

All you need is 'perf sched record -g -- make -j 256'

gcc related tasks will start erroring out. Odd that it is mainly gcc and 
friends getting hit.

I threw a printk into all of the sigbus places under arch/sparc. One 
that kept triggering is sun4v_do_mna. I was not able to get any info 
about why.

Re: sparc: deadlock using perf with sched tracepoints and user callstacks

[David Miller]

From: David Ahern <david.ahern@oracle.com>
Date: Wed, 22 Apr 2015 18:39:53 -0600

> gcc related tasks will start erroring out. Odd that it is mainly gcc
> and friends getting hit.
> 
> I threw a printk into all of the sigbus places under arch/sparc. One
> that kept triggering is sun4v_do_mna. I was not able to get any info
> about why.

Unaligned memory accesses are not a good sign, it usually means
userspace is getting corrupted.