[GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Frederic Weisbecker]

Ingo,

I've tested a little this set and it seems to work well. I can't test
the C expression parser (need some libdwarf cross-version compatiblity),
but I can create kprobes via low-level parameters, then launch
perf record and perf trace successfully. I'll need to update
perf trace to handle variables that start with '$' characters though.

Anyway, it's still unstable but I guess pulling it would
help Masami to continue without the need to handle the
whole batch.

Please pull the tracing/kprobes branch that can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
	tracing/kprobes

Thanks,
	Frederic.

Masami Hiramatsu (7):
      tracing/kprobes: Add $ prefix to special variables
      tracing/kprobes: Remove '$ra' special variable
      tracing/kprobes: Make special variable names more self-explainable
      tracing/kprobes: Avoid field name confliction
      tracing/kprobes: Robustify fixed field names against variable field names conflicts
      perf: Add perf probe subcommand, a kprobe-event setup helper
      perf probe: Add perf probe command support without libdwarf

 Documentation/trace/kprobetrace.txt |   27 +-
 kernel/trace/trace_kprobe.c         |  160 +++++---
 tools/perf/Makefile                 |   10 +
 tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
 tools/perf/builtin.h                |    1 +
 tools/perf/perf.c                   |    1 +
 tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
 tools/perf/util/probe-finder.h      |   70 ++++
 8 files changed, 1271 insertions(+), 74 deletions(-)
 create mode 100644 tools/perf/builtin-probe.c
 create mode 100644 tools/perf/util/probe-finder.c
 create mode 100644 tools/perf/util/probe-finder.h

[PATCH] kill-the-bkl/reiserfs: fix reiserfs lock to cpu_add_remove_lock dependency

[Frederic Weisbecker]

While creating the reiserfs workqueue during the journal
initialization, we are holding the reiserfs lock, but
create_workqueue() also holds the cpu_add_remove_lock, creating
then the following dependency:

- reiserfs lock -> cpu_add_remove_lock

But we also have the following existing dependencies:

- mm->mmap_sem -> reiserfs lock
- cpu_add_remove_lock -> cpu_hotplug.lock -> slub_lock -> sysfs_mutex

The merged dependency chain then becomes:

- mm->mmap_sem -> reiserfs lock -> cpu_add_remove_lock ->
	cpu_hotplug.lock -> slub_lock -> sysfs_mutex

But when we fill a dir entry in sysfs_readir(), we are holding the
sysfs_mutex and we also might fault while copying the directory entry
to the user, leading to the following dependency:

- sysfs_mutex -> mm->mmap_sem

The end result is then a lock inversion between sysfs_mutex and
mm->mmap_sem, as reported in the following lockdep warning:

[ INFO: possible circular locking dependency detected ]
2.6.31-07095-g25a3912 #4
-------------------------------------------------------
udevadm/790 is trying to acquire lock:
 (&mm->mmap_sem){++++++}, at: [<c1098942>] might_fault+0x72/0xc0

but task is already holding lock:
 (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #5 (sysfs_mutex){+.+.+.}:
      [...]

-> #4 (slub_lock){+++++.}:
      [...]

-> #3 (cpu_hotplug.lock){+.+.+.}:
      [...]

-> #2 (cpu_add_remove_lock){+.+.+.}:
      [...]

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
      [...]

-> #0 (&mm->mmap_sem){++++++}:
      [...]

This can be fixed by relaxing the reiserfs lock while creating the
workqueue.
This is fine to relax the lock here, we just keep it around to pass
through reiserfs lock checks and for paranoid reasons.

Reported-by: Alexander Beregalov <a.beregalov@gmail.com>
Tested-by: Alexander Beregalov <a.beregalov@gmail.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Alexander Beregalov <a.beregalov@gmail.com>
Cc: Laurent Riffard <laurent.riffard@free.fr>
---
 fs/reiserfs/journal.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index 04e3c42..2f8a7e7 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2933,8 +2933,11 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 	}
 
 	reiserfs_mounted_fs_count++;
-	if (reiserfs_mounted_fs_count <= 1)
+	if (reiserfs_mounted_fs_count <= 1) {
+		reiserfs_write_unlock(sb);
 		commit_wq = create_workqueue("reiserfs");
+		reiserfs_write_lock(sb);
+	}
 
 	INIT_DELAYED_WORK(&journal->j_work, flush_async_commits);
 	journal->j_work_sb = sb;
-- 
1.6.2.3

Re: [PATCH] kill-the-bkl/reiserfs: fix reiserfs lock to cpu_add_remove_lock dependency

[Frederic Weisbecker]

On Tue, Oct 13, 2009 at 12:25:02AM +0200, Frederic Weisbecker wrote:
> This can be fixed by relaxing the reiserfs lock while creating the
> workqueue.
> This is fine to relax the lock here, we just keep it around to pass
> through reiserfs lock checks and for paranoid reasons.


Nothing related to kprobes that said...
Sorry for the noise. I've already posted this patch, but it
looks like these locking inversions don't want to live
their own life and follow me everywhere...

[PATCH 1/7] tracing/kprobes: Add $ prefix to special variables

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Add $ prefix to the special variables(e.g. sa, rv) of kprobe-tracer.
This resolves consistency issues between kprobe_events and perf-kprobe.

The main goal is to avoid conflicts between local variable names of
probed functions, used by perf probe, and special variables used
in the kprobe event creation interface (stack values, etc...) and
also available from perf probe.

ie: we don't want rv (return value) to conflict with a local variable
named rv in a probed function.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222740.1684.91170.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 Documentation/trace/kprobetrace.txt |   20 ++++++------
 kernel/trace/trace_kprobe.c         |   60 +++++++++++++++++++++-------------
 2 files changed, 47 insertions(+), 33 deletions(-)

diff --git a/Documentation/trace/kprobetrace.txt b/Documentation/trace/kprobetrace.txt
index 9b8f7c6..33f5318 100644
--- a/Documentation/trace/kprobetrace.txt
+++ b/Documentation/trace/kprobetrace.txt
@@ -36,13 +36,13 @@ Synopsis of kprobe_events
 
  FETCHARGS	: Arguments. Each probe can have up to 128 args.
   %REG	: Fetch register REG
-  sN	: Fetch Nth entry of stack (N >= 0)
-  sa	: Fetch stack address.
   @ADDR	: Fetch memory at ADDR (ADDR should be in kernel)
   @SYM[+|-offs]	: Fetch memory at SYM +|- offs (SYM should be a data symbol)
-  aN	: Fetch function argument. (N >= 0)(*)
-  rv	: Fetch return value.(**)
-  ra	: Fetch return address.(**)
+  $sN	: Fetch Nth entry of stack (N >= 0)
+  $sa	: Fetch stack address.
+  $aN	: Fetch function argument. (N >= 0)(*)
+  $rv	: Fetch return value.(**)
+  $ra	: Fetch return address.(**)
   +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(***)
   NAME=FETCHARG: Set NAME as the argument name of FETCHARG.
 
@@ -85,13 +85,13 @@ Usage examples
 To add a probe as a new event, write a new definition to kprobe_events
 as below.
 
-  echo p:myprobe do_sys_open dfd=a0 filename=a1 flags=a2 mode=a3 > /sys/kernel/debug/tracing/kprobe_events
+  echo p:myprobe do_sys_open dfd=$a0 filename=$a1 flags=$a2 mode=$a3 > /sys/kernel/debug/tracing/kprobe_events
 
  This sets a kprobe on the top of do_sys_open() function with recording
 1st to 4th arguments as "myprobe" event. As this example shows, users can
 choose more familiar names for each arguments.
 
-  echo r:myretprobe do_sys_open rv ra >> /sys/kernel/debug/tracing/kprobe_events
+  echo r:myretprobe do_sys_open $rv $ra >> /sys/kernel/debug/tracing/kprobe_events
 
  This sets a kretprobe on the return point of do_sys_open() function with
 recording return value and return address as "myretprobe" event.
@@ -138,11 +138,11 @@ events, you need to enable it.
 #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
 #              | |       |          |         |
            <...>-1447  [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
-           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) rv=fffffffffffffffe ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $rv=fffffffffffffffe $ra=ffffffff81367a3a
            <...>-1447  [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
-           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) rv=3 ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3 $ra=ffffffff81367a3a
            <...>-1447  [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
-           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) rv=3 ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3 $ra=ffffffff81367a3a
 
 
  Each line shows when the kernel hits an event, and <- SYMBOL means kernel
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 97309d4..f63ead0 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -220,24 +220,24 @@ static int probe_arg_string(char *buf, size_t n, struct fetch_func *ff)
 	int ret = -EINVAL;
 
 	if (ff->func == fetch_argument)
-		ret = snprintf(buf, n, "a%lu", (unsigned long)ff->data);
+		ret = snprintf(buf, n, "$a%lu", (unsigned long)ff->data);
 	else if (ff->func == fetch_register) {
 		const char *name;
 		name = regs_query_register_name((unsigned int)((long)ff->data));
 		ret = snprintf(buf, n, "%%%s", name);
 	} else if (ff->func == fetch_stack)
-		ret = snprintf(buf, n, "s%lu", (unsigned long)ff->data);
+		ret = snprintf(buf, n, "$s%lu", (unsigned long)ff->data);
 	else if (ff->func == fetch_memory)
 		ret = snprintf(buf, n, "@0x%p", ff->data);
 	else if (ff->func == fetch_symbol) {
 		struct symbol_cache *sc = ff->data;
 		ret = snprintf(buf, n, "@%s%+ld", sc->symbol, sc->offset);
 	} else if (ff->func == fetch_retvalue)
-		ret = snprintf(buf, n, "rv");
+		ret = snprintf(buf, n, "$rv");
 	else if (ff->func == fetch_ip)
-		ret = snprintf(buf, n, "ra");
+		ret = snprintf(buf, n, "$ra");
 	else if (ff->func == fetch_stack_address)
-		ret = snprintf(buf, n, "sa");
+		ret = snprintf(buf, n, "$sa");
 	else if (ff->func == fetch_indirect) {
 		struct indirect_fetch_data *id = ff->data;
 		size_t l = 0;
@@ -429,12 +429,10 @@ static int split_symbol_offset(char *symbol, unsigned long *offset)
 #define PARAM_MAX_ARGS 16
 #define PARAM_MAX_STACK (THREAD_SIZE / sizeof(unsigned long))
 
-static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
+static int parse_probe_vars(char *arg, struct fetch_func *ff, int is_return)
 {
 	int ret = 0;
 	unsigned long param;
-	long offset;
-	char *tmp;
 
 	switch (arg[0]) {
 	case 'a':	/* argument */
@@ -456,14 +454,6 @@ static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
 		} else
 			ret = -EINVAL;
 		break;
-	case '%':	/* named register */
-		ret = regs_query_register_offset(arg + 1);
-		if (ret >= 0) {
-			ff->func = fetch_register;
-			ff->data = (void *)(unsigned long)ret;
-			ret = 0;
-		}
-		break;
 	case 's':	/* stack */
 		if (arg[1] == 'a') {
 			ff->func = fetch_stack_address;
@@ -478,6 +468,31 @@ static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
 			}
 		}
 		break;
+	default:
+		ret = -EINVAL;
+	}
+	return ret;
+}
+
+static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
+{
+	int ret = 0;
+	unsigned long param;
+	long offset;
+	char *tmp;
+
+	switch (arg[0]) {
+	case '$':
+		ret = parse_probe_vars(arg + 1, ff, is_return);
+		break;
+	case '%':	/* named register */
+		ret = regs_query_register_offset(arg + 1);
+		if (ret >= 0) {
+			ff->func = fetch_register;
+			ff->data = (void *)(unsigned long)ret;
+			ret = 0;
+		}
+		break;
 	case '@':	/* memory or symbol */
 		if (isdigit(arg[1])) {
 			ret = strict_strtoul(arg + 1, 0, &param);
@@ -489,8 +504,7 @@ static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
 			ret = split_symbol_offset(arg + 1, &offset);
 			if (ret)
 				break;
-			ff->data = alloc_symbol_cache(arg + 1,
-							      offset);
+			ff->data = alloc_symbol_cache(arg + 1, offset);
 			if (ff->data)
 				ff->func = fetch_symbol;
 			else
@@ -544,11 +558,11 @@ static int create_trace_probe(int argc, char **argv)
 	 *  - Add kprobe: p[:[GRP/]EVENT] KSYM[+OFFS]|KADDR [FETCHARGS]
 	 *  - Add kretprobe: r[:[GRP/]EVENT] KSYM[+0] [FETCHARGS]
 	 * Fetch args:
-	 *  aN	: fetch Nth of function argument. (N:0-)
-	 *  rv	: fetch return value
-	 *  ra	: fetch return address
-	 *  sa	: fetch stack address
-	 *  sN	: fetch Nth of stack (N:0-)
+	 *  $aN	: fetch Nth of function argument. (N:0-)
+	 *  $rv	: fetch return value
+	 *  $ra	: fetch return address
+	 *  $sa	: fetch stack address
+	 *  $sN	: fetch Nth of stack (N:0-)
 	 *  @ADDR	: fetch memory at ADDR (ADDR should be in kernel)
 	 *  @SYM[+|-offs] : fetch memory at SYM +|- offs (SYM is a data symbol)
 	 *  %REG	: fetch register REG
-- 
1.6.2.3

[PATCH 2/7] tracing/kprobes: Remove '$ra' special variable

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Remove '$ra' (return address) because it is already shown at the head of
each entry.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222748.1684.12711.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 Documentation/trace/kprobetrace.txt |   13 ++++++-------
 kernel/trace/trace_kprobe.c         |   11 -----------
 2 files changed, 6 insertions(+), 18 deletions(-)

diff --git a/Documentation/trace/kprobetrace.txt b/Documentation/trace/kprobetrace.txt
index 33f5318..4208253 100644
--- a/Documentation/trace/kprobetrace.txt
+++ b/Documentation/trace/kprobetrace.txt
@@ -1,4 +1,4 @@
-                         Kprobe-based Event Tracer
+                        Kprobe-based Event Tracer
                          =========================
 
                  Documentation is written by Masami Hiramatsu
@@ -42,7 +42,6 @@ Synopsis of kprobe_events
   $sa	: Fetch stack address.
   $aN	: Fetch function argument. (N >= 0)(*)
   $rv	: Fetch return value.(**)
-  $ra	: Fetch return address.(**)
   +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(***)
   NAME=FETCHARG: Set NAME as the argument name of FETCHARG.
 
@@ -91,10 +90,10 @@ as below.
 1st to 4th arguments as "myprobe" event. As this example shows, users can
 choose more familiar names for each arguments.
 
-  echo r:myretprobe do_sys_open $rv $ra >> /sys/kernel/debug/tracing/kprobe_events
+  echo r:myretprobe do_sys_open $rv >> /sys/kernel/debug/tracing/kprobe_events
 
  This sets a kretprobe on the return point of do_sys_open() function with
-recording return value and return address as "myretprobe" event.
+recording return value as "myretprobe" event.
  You can see the format of these events via
 /sys/kernel/debug/tracing/events/kprobes/<EVENT>/format.
 
@@ -138,11 +137,11 @@ events, you need to enable it.
 #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
 #              | |       |          |         |
            <...>-1447  [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
-           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $rv=fffffffffffffffe $ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $rv=fffffffffffffffe
            <...>-1447  [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
-           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3 $ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3
            <...>-1447  [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
-           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3 $ra=ffffffff81367a3a
+           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3
 
 
  Each line shows when the kernel hits an event, and <- SYMBOL means kernel
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index f63ead0..ba6d3bd 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -85,11 +85,6 @@ static __kprobes unsigned long fetch_retvalue(struct pt_regs *regs,
 	return regs_return_value(regs);
 }
 
-static __kprobes unsigned long fetch_ip(struct pt_regs *regs, void *dummy)
-{
-	return instruction_pointer(regs);
-}
-
 static __kprobes unsigned long fetch_stack_address(struct pt_regs *regs,
 						   void *dummy)
 {
@@ -234,8 +229,6 @@ static int probe_arg_string(char *buf, size_t n, struct fetch_func *ff)
 		ret = snprintf(buf, n, "@%s%+ld", sc->symbol, sc->offset);
 	} else if (ff->func == fetch_retvalue)
 		ret = snprintf(buf, n, "$rv");
-	else if (ff->func == fetch_ip)
-		ret = snprintf(buf, n, "$ra");
 	else if (ff->func == fetch_stack_address)
 		ret = snprintf(buf, n, "$sa");
 	else if (ff->func == fetch_indirect) {
@@ -448,9 +441,6 @@ static int parse_probe_vars(char *arg, struct fetch_func *ff, int is_return)
 		if (is_return && arg[1] == 'v') {
 			ff->func = fetch_retvalue;
 			ff->data = NULL;
-		} else if (is_return && arg[1] == 'a') {
-			ff->func = fetch_ip;
-			ff->data = NULL;
 		} else
 			ret = -EINVAL;
 		break;
@@ -560,7 +550,6 @@ static int create_trace_probe(int argc, char **argv)
 	 * Fetch args:
 	 *  $aN	: fetch Nth of function argument. (N:0-)
 	 *  $rv	: fetch return value
-	 *  $ra	: fetch return address
 	 *  $sa	: fetch stack address
 	 *  $sN	: fetch Nth of stack (N:0-)
 	 *  @ADDR	: fetch memory at ADDR (ADDR should be in kernel)
-- 
1.6.2.3

[PATCH 3/7] tracing/kprobes: Make special variable names more self-explainable

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Rename special variables to more self-explainable names as below:
- $rv to $retval
- $sa to $stack
- $aN to $argN
- $sN to $stackN

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222759.1684.3319.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 Documentation/trace/kprobetrace.txt |   22 +++++++-------
 kernel/trace/trace_kprobe.c         |   52 ++++++++++++++++-------------------
 2 files changed, 35 insertions(+), 39 deletions(-)

diff --git a/Documentation/trace/kprobetrace.txt b/Documentation/trace/kprobetrace.txt
index 4208253..1541524 100644
--- a/Documentation/trace/kprobetrace.txt
+++ b/Documentation/trace/kprobetrace.txt
@@ -35,13 +35,13 @@ Synopsis of kprobe_events
  MEMADDR	: Address where the probe is inserted.
 
  FETCHARGS	: Arguments. Each probe can have up to 128 args.
-  %REG	: Fetch register REG
-  @ADDR	: Fetch memory at ADDR (ADDR should be in kernel)
+  %REG		: Fetch register REG
+  @ADDR		: Fetch memory at ADDR (ADDR should be in kernel)
   @SYM[+|-offs]	: Fetch memory at SYM +|- offs (SYM should be a data symbol)
-  $sN	: Fetch Nth entry of stack (N >= 0)
-  $sa	: Fetch stack address.
-  $aN	: Fetch function argument. (N >= 0)(*)
-  $rv	: Fetch return value.(**)
+  $stackN	: Fetch Nth entry of stack (N >= 0)
+  $stack	: Fetch stack address.
+  $argN		: Fetch function argument. (N >= 0)(*)
+  $retval	: Fetch return value.(**)
   +|-offs(FETCHARG) : Fetch memory at FETCHARG +|- offs address.(***)
   NAME=FETCHARG: Set NAME as the argument name of FETCHARG.
 
@@ -84,13 +84,13 @@ Usage examples
 To add a probe as a new event, write a new definition to kprobe_events
 as below.
 
-  echo p:myprobe do_sys_open dfd=$a0 filename=$a1 flags=$a2 mode=$a3 > /sys/kernel/debug/tracing/kprobe_events
+  echo p:myprobe do_sys_open dfd=$arg0 filename=$arg1 flags=$arg2 mode=$arg3 > /sys/kernel/debug/tracing/kprobe_events
 
  This sets a kprobe on the top of do_sys_open() function with recording
 1st to 4th arguments as "myprobe" event. As this example shows, users can
 choose more familiar names for each arguments.
 
-  echo r:myretprobe do_sys_open $rv >> /sys/kernel/debug/tracing/kprobe_events
+  echo r:myretprobe do_sys_open $retval >> /sys/kernel/debug/tracing/kprobe_events
 
  This sets a kretprobe on the return point of do_sys_open() function with
 recording return value as "myretprobe" event.
@@ -137,11 +137,11 @@ events, you need to enable it.
 #           TASK-PID    CPU#    TIMESTAMP  FUNCTION
 #              | |       |          |         |
            <...>-1447  [001] 1038282.286875: myprobe: (do_sys_open+0x0/0xd6) dfd=3 filename=7fffd1ec4440 flags=8000 mode=0
-           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $rv=fffffffffffffffe
+           <...>-1447  [001] 1038282.286878: myretprobe: (sys_openat+0xc/0xe <- do_sys_open) $retval=fffffffffffffffe
            <...>-1447  [001] 1038282.286885: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=40413c flags=8000 mode=1b6
-           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3
+           <...>-1447  [001] 1038282.286915: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
            <...>-1447  [001] 1038282.286969: myprobe: (do_sys_open+0x0/0xd6) dfd=ffffff9c filename=4041c6 flags=98800 mode=10
-           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $rv=3
+           <...>-1447  [001] 1038282.286976: myretprobe: (sys_open+0x1b/0x1d <- do_sys_open) $retval=3
 
 
  Each line shows when the kernel hits an event, and <- SYMBOL means kernel
diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index ba6d3bd..3313fa7 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -215,22 +215,22 @@ static int probe_arg_string(char *buf, size_t n, struct fetch_func *ff)
 	int ret = -EINVAL;
 
 	if (ff->func == fetch_argument)
-		ret = snprintf(buf, n, "$a%lu", (unsigned long)ff->data);
+		ret = snprintf(buf, n, "$arg%lu", (unsigned long)ff->data);
 	else if (ff->func == fetch_register) {
 		const char *name;
 		name = regs_query_register_name((unsigned int)((long)ff->data));
 		ret = snprintf(buf, n, "%%%s", name);
 	} else if (ff->func == fetch_stack)
-		ret = snprintf(buf, n, "$s%lu", (unsigned long)ff->data);
+		ret = snprintf(buf, n, "$stack%lu", (unsigned long)ff->data);
 	else if (ff->func == fetch_memory)
 		ret = snprintf(buf, n, "@0x%p", ff->data);
 	else if (ff->func == fetch_symbol) {
 		struct symbol_cache *sc = ff->data;
 		ret = snprintf(buf, n, "@%s%+ld", sc->symbol, sc->offset);
 	} else if (ff->func == fetch_retvalue)
-		ret = snprintf(buf, n, "$rv");
+		ret = snprintf(buf, n, "$retval");
 	else if (ff->func == fetch_stack_address)
-		ret = snprintf(buf, n, "$sa");
+		ret = snprintf(buf, n, "$stack");
 	else if (ff->func == fetch_indirect) {
 		struct indirect_fetch_data *id = ff->data;
 		size_t l = 0;
@@ -427,40 +427,36 @@ static int parse_probe_vars(char *arg, struct fetch_func *ff, int is_return)
 	int ret = 0;
 	unsigned long param;
 
-	switch (arg[0]) {
-	case 'a':	/* argument */
-		ret = strict_strtoul(arg + 1, 10, &param);
-		if (ret || param > PARAM_MAX_ARGS)
-			ret = -EINVAL;
-		else {
-			ff->func = fetch_argument;
-			ff->data = (void *)param;
-		}
-		break;
-	case 'r':	/* retval or retaddr */
-		if (is_return && arg[1] == 'v') {
+	if (strcmp(arg, "retval") == 0) {
+		if (is_return) {
 			ff->func = fetch_retvalue;
 			ff->data = NULL;
 		} else
 			ret = -EINVAL;
-		break;
-	case 's':	/* stack */
-		if (arg[1] == 'a') {
+	} else if (strncmp(arg, "stack", 5) == 0) {
+		if (arg[5] == '\0') {
 			ff->func = fetch_stack_address;
 			ff->data = NULL;
-		} else {
-			ret = strict_strtoul(arg + 1, 10, &param);
+		} else if (isdigit(arg[5])) {
+			ret = strict_strtoul(arg + 5, 10, &param);
 			if (ret || param > PARAM_MAX_STACK)
 				ret = -EINVAL;
 			else {
 				ff->func = fetch_stack;
 				ff->data = (void *)param;
 			}
+		} else
+			ret = -EINVAL;
+	} else if (strncmp(arg, "arg", 3) == 0 && isdigit(arg[3])) {
+		ret = strict_strtoul(arg + 3, 10, &param);
+		if (ret || param > PARAM_MAX_ARGS)
+			ret = -EINVAL;
+		else {
+			ff->func = fetch_argument;
+			ff->data = (void *)param;
 		}
-		break;
-	default:
+	} else
 		ret = -EINVAL;
-	}
 	return ret;
 }
 
@@ -548,10 +544,10 @@ static int create_trace_probe(int argc, char **argv)
 	 *  - Add kprobe: p[:[GRP/]EVENT] KSYM[+OFFS]|KADDR [FETCHARGS]
 	 *  - Add kretprobe: r[:[GRP/]EVENT] KSYM[+0] [FETCHARGS]
 	 * Fetch args:
-	 *  $aN	: fetch Nth of function argument. (N:0-)
-	 *  $rv	: fetch return value
-	 *  $sa	: fetch stack address
-	 *  $sN	: fetch Nth of stack (N:0-)
+	 *  $argN	: fetch Nth of function argument. (N:0-)
+	 *  $retval	: fetch return value
+	 *  $stack	: fetch stack address
+	 *  $stackN	: fetch Nth of stack (N:0-)
 	 *  @ADDR	: fetch memory at ADDR (ADDR should be in kernel)
 	 *  @SYM[+|-offs] : fetch memory at SYM +|- offs (SYM is a data symbol)
 	 *  %REG	: fetch register REG
-- 
1.6.2.3

[PATCH 4/7] tracing/kprobes: Avoid field name confliction

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Check whether the argument name is in conflict with other field names
while creating a kprobe through the debugfs interface.

Changes in v3:
 - Check strcmp() == 0 instead of !strcmp().

Changes in v2:
 - Add common_lock_depth to reserved name list.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222807.1684.26880.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 kernel/trace/trace_kprobe.c |   65 +++++++++++++++++++++++++++++++++++--------
 1 files changed, 53 insertions(+), 12 deletions(-)

diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index 3313fa7..bb6cb2b 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -38,6 +38,25 @@
 #define MAX_EVENT_NAME_LEN 64
 #define KPROBE_EVENT_SYSTEM "kprobes"
 
+/* Reserved field names */
+#define FIELD_STRING_IP "ip"
+#define FIELD_STRING_NARGS "nargs"
+#define FIELD_STRING_RETIP "ret_ip"
+#define FIELD_STRING_FUNC "func"
+
+const char *reserved_field_names[] = {
+	"common_type",
+	"common_flags",
+	"common_preempt_count",
+	"common_pid",
+	"common_tgid",
+	"common_lock_depth",
+	FIELD_STRING_IP,
+	FIELD_STRING_NARGS,
+	FIELD_STRING_RETIP,
+	FIELD_STRING_FUNC,
+};
+
 /* currently, trace_kprobe only supports X86. */
 
 struct fetch_func {
@@ -537,6 +556,20 @@ static int parse_probe_arg(char *arg, struct fetch_func *ff, int is_return)
 	return ret;
 }
 
+/* Return 1 if name is reserved or already used by another argument */
+static int conflict_field_name(const char *name,
+			       struct probe_arg *args, int narg)
+{
+	int i;
+	for (i = 0; i < ARRAY_SIZE(reserved_field_names); i++)
+		if (strcmp(reserved_field_names[i], name) == 0)
+			return 1;
+	for (i = 0; i < narg; i++)
+		if (strcmp(args[i].name, name) == 0)
+			return 1;
+	return 0;
+}
+
 static int create_trace_probe(int argc, char **argv)
 {
 	/*
@@ -637,6 +670,12 @@ static int create_trace_probe(int argc, char **argv)
 			*arg++ = '\0';
 		else
 			arg = argv[i];
+
+		if (conflict_field_name(argv[i], tp->args, i)) {
+			ret = -EINVAL;
+			goto error;
+		}
+
 		tp->args[i].name = kstrdup(argv[i], GFP_KERNEL);
 
 		/* Parse fetch argument */
@@ -1039,8 +1078,8 @@ static int kprobe_event_define_fields(struct ftrace_event_call *event_call)
 	if (!ret)
 		return ret;
 
-	DEFINE_FIELD(unsigned long, ip, "ip", 0);
-	DEFINE_FIELD(int, nargs, "nargs", 1);
+	DEFINE_FIELD(unsigned long, ip, FIELD_STRING_IP, 0);
+	DEFINE_FIELD(int, nargs, FIELD_STRING_NARGS, 1);
 	/* Set argument names as fields */
 	for (i = 0; i < tp->nr_args; i++)
 		DEFINE_FIELD(unsigned long, args[i], tp->args[i].name, 0);
@@ -1057,9 +1096,9 @@ static int kretprobe_event_define_fields(struct ftrace_event_call *event_call)
 	if (!ret)
 		return ret;
 
-	DEFINE_FIELD(unsigned long, func, "func", 0);
-	DEFINE_FIELD(unsigned long, ret_ip, "ret_ip", 0);
-	DEFINE_FIELD(int, nargs, "nargs", 1);
+	DEFINE_FIELD(unsigned long, func, FIELD_STRING_FUNC, 0);
+	DEFINE_FIELD(unsigned long, ret_ip, FIELD_STRING_RETIP, 0);
+	DEFINE_FIELD(int, nargs, FIELD_STRING_NARGS, 1);
 	/* Set argument names as fields */
 	for (i = 0; i < tp->nr_args; i++)
 		DEFINE_FIELD(unsigned long, args[i], tp->args[i].name, 0);
@@ -1108,15 +1147,16 @@ static int kprobe_event_show_format(struct ftrace_event_call *call,
 	int ret, i;
 	struct trace_probe *tp = (struct trace_probe *)call->data;
 
-	SHOW_FIELD(unsigned long, ip, "ip");
-	SHOW_FIELD(int, nargs, "nargs");
+	SHOW_FIELD(unsigned long, ip, FIELD_STRING_IP);
+	SHOW_FIELD(int, nargs, FIELD_STRING_NARGS);
 
 	/* Show fields */
 	for (i = 0; i < tp->nr_args; i++)
 		SHOW_FIELD(unsigned long, args[i], tp->args[i].name);
 	trace_seq_puts(s, "\n");
 
-	return __probe_event_show_format(s, tp, "(%lx)", "REC->ip");
+	return __probe_event_show_format(s, tp, "(%lx)",
+					 "REC->" FIELD_STRING_IP);
 }
 
 static int kretprobe_event_show_format(struct ftrace_event_call *call,
@@ -1126,9 +1166,9 @@ static int kretprobe_event_show_format(struct ftrace_event_call *call,
 	int ret, i;
 	struct trace_probe *tp = (struct trace_probe *)call->data;
 
-	SHOW_FIELD(unsigned long, func, "func");
-	SHOW_FIELD(unsigned long, ret_ip, "ret_ip");
-	SHOW_FIELD(int, nargs, "nargs");
+	SHOW_FIELD(unsigned long, func, FIELD_STRING_FUNC);
+	SHOW_FIELD(unsigned long, ret_ip, FIELD_STRING_RETIP);
+	SHOW_FIELD(int, nargs, FIELD_STRING_NARGS);
 
 	/* Show fields */
 	for (i = 0; i < tp->nr_args; i++)
@@ -1136,7 +1176,8 @@ static int kretprobe_event_show_format(struct ftrace_event_call *call,
 	trace_seq_puts(s, "\n");
 
 	return __probe_event_show_format(s, tp, "(%lx <- %lx)",
-					  "REC->func, REC->ret_ip");
+					 "REC->" FIELD_STRING_FUNC
+					 ", REC->" FIELD_STRING_RETIP);
 }
 
 #ifdef CONFIG_EVENT_PROFILE
-- 
1.6.2.3

[PATCH 5/7] tracing/kprobes: Robustify fixed field names against variable field names conflicts

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Rename probe-common fixed field names to harder conflictable names,
because current 'ip', 'func', and other probe field names are easily in
conflict with user-specified variable names.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222814.1684.407.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 kernel/trace/trace_kprobe.c |    8 ++++----
 1 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/kernel/trace/trace_kprobe.c b/kernel/trace/trace_kprobe.c
index bb6cb2b..739f70e 100644
--- a/kernel/trace/trace_kprobe.c
+++ b/kernel/trace/trace_kprobe.c
@@ -39,10 +39,10 @@
 #define KPROBE_EVENT_SYSTEM "kprobes"
 
 /* Reserved field names */
-#define FIELD_STRING_IP "ip"
-#define FIELD_STRING_NARGS "nargs"
-#define FIELD_STRING_RETIP "ret_ip"
-#define FIELD_STRING_FUNC "func"
+#define FIELD_STRING_IP "__probe_ip"
+#define FIELD_STRING_NARGS "__probe_nargs"
+#define FIELD_STRING_RETIP "__probe_ret_ip"
+#define FIELD_STRING_FUNC "__probe_func"
 
 const char *reserved_field_names[] = {
 	"common_type",
-- 
1.6.2.3

[PATCH 6/7] perf: Add perf probe subcommand, a kprobe-event setup helper

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Add perf probe subcommand that implements a kprobe-event setup helper
to the perf command.
This allows user to define kprobe events using C expressions (C line
numbers, C function names, and C local variables).

Usage
-----
 perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...]

    -k, --vmlinux <file>  vmlinux/module pathname
    -P, --probe <p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]>
                          probe point definition, where
		p:	kprobe probe
		r:	kretprobe probe
		GRP:	Group name (optional)
		NAME:	Event name
		FUNC:	Function name
		OFFS:	Offset from function entry (in byte)
		SRC:	Source code path
		LINE:	Line number
		ARG:	Probe argument (local variable name or
			kprobe-tracer argument format is supported.)

Changes in v4:
 - Add _GNU_SOURCE macro for strndup().

Changes in v3:
 - Remove -r option because perf always be used for online kernel.
 - Check malloc/calloc results.

Changes in v2:
 - Check synthesized string length.
 - Rename perf kprobe to perf probe.
 - Use spaces for separator and update usage comment.
 - Check error paths in parse_probepoint().
 - Check optimized-out variables.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091008211737.29299.14784.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 tools/perf/Makefile            |   10 +
 tools/perf/builtin-probe.c     |  358 +++++++++++++++++++++
 tools/perf/builtin.h           |    1 +
 tools/perf/perf.c              |    3 +
 tools/perf/util/probe-finder.c |  690 ++++++++++++++++++++++++++++++++++++++++
 tools/perf/util/probe-finder.h |   68 ++++
 6 files changed, 1130 insertions(+), 0 deletions(-)
 create mode 100644 tools/perf/builtin-probe.c
 create mode 100644 tools/perf/util/probe-finder.c
 create mode 100644 tools/perf/util/probe-finder.h

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index b5f1953..6dabcf1 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -419,6 +419,16 @@ ifneq ($(shell sh -c "(echo '\#include <libelf.h>'; echo 'int main(void) { Elf *
 	msg := $(error No libelf.h/libelf found, please install libelf-dev/elfutils-libelf-devel);
 endif
 
+ifneq ($(shell sh -c "(echo '\#include <libdwarf/dwarf.h>'; echo '\#include <libdwarf/libdwarf.h>'; echo 'int main(void) { Dwarf_Debug dbg; Dwarf_Error err; dwarf_init(0, DW_DLC_READ, 0, 0, &dbg, &err); return (long)dbg; }') | $(CC) -x c - $(ALL_CFLAGS) -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -ldwarf -lelf -o /dev/null $(ALL_LDFLAGS) > /dev/null 2>&1 && echo y"), y)
+	msg := $(warning No libdwarf.h found, disables probe subcommand. Please install libdwarf-dev/libdwarf-devel);
+else
+	EXTLIBS += -lelf -ldwarf
+	LIB_H += util/probe-finder.h
+	LIB_OBJS += util/probe-finder.o
+	BUILTIN_OBJS += builtin-probe.o
+	BASIC_CFLAGS += -DSUPPORT_DWARF
+endif
+
 ifdef NO_DEMANGLE
 	BASIC_CFLAGS += -DNO_DEMANGLE
 else
diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
new file mode 100644
index 0000000..24b64b5
--- /dev/null
+++ b/tools/perf/builtin-probe.c
@@ -0,0 +1,358 @@
+/*
+ * builtin-probe.c
+ *
+ * Builtin probe command: Set up probe events by C expression
+ *
+ * Written by Masami Hiramatsu <mhiramat@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+#define _GNU_SOURCE
+#include <sys/utsname.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <stdlib.h>
+#include <string.h>
+
+#undef _GNU_SOURCE
+#include "perf.h"
+#include "builtin.h"
+#include "util/util.h"
+#include "util/parse-options.h"
+#include "util/parse-events.h"	/* For debugfs_path */
+#include "util/probe-finder.h"
+
+/* Default vmlinux search paths */
+#define NR_SEARCH_PATH 3
+const char *default_search_path[NR_SEARCH_PATH] = {
+"/lib/modules/%s/build/vmlinux",		/* Custom build kernel */
+"/usr/lib/debug/lib/modules/%s/vmlinux",	/* Red Hat debuginfo */
+"/boot/vmlinux-debug-%s",			/* Ubuntu */
+};
+
+#define MAX_PATH_LEN 256
+#define MAX_PROBES 128
+
+/* Session management structure */
+static struct {
+	char *vmlinux;
+	char *release;
+	int nr_probe;
+	struct probe_point probes[MAX_PROBES];
+	char *events[MAX_PROBES];
+} session;
+
+static void semantic_error(const char *msg)
+{
+	fprintf(stderr, "Semantic error: %s\n", msg);
+	exit(1);
+}
+
+static void perror_exit(const char *msg)
+{
+	perror(msg);
+	exit(1);
+}
+
+#define MAX_PROBE_ARGS 128
+
+static int parse_probepoint(const struct option *opt __used,
+			    const char *str, int unset __used)
+{
+	char *argv[MAX_PROBE_ARGS + 2];	/* Event + probe + args */
+	int argc, i;
+	char *arg, *ptr;
+	struct probe_point *pp = &session.probes[session.nr_probe];
+	char **event = &session.events[session.nr_probe];
+	int retp = 0;
+
+	if (!str)	/* The end of probe points */
+		return 0;
+
+	debug("Probe-define(%d): %s\n", session.nr_probe, str);
+	if (++session.nr_probe == MAX_PROBES)
+		semantic_error("Too many probes");
+
+	/* Separate arguments, similar to argv_split */
+	argc = 0;
+	do {
+		/* Skip separators */
+		while (isspace(*str))
+			str++;
+
+		/* Add an argument */
+		if (*str != '\0') {
+			const char *s = str;
+
+			/* Skip the argument */
+			while (!isspace(*str) && *str != '\0')
+				str++;
+
+			/* Duplicate the argument */
+			argv[argc] = strndup(s, str - s);
+			if (argv[argc] == NULL)
+				perror_exit("strndup");
+			if (++argc == MAX_PROBE_ARGS)
+				semantic_error("Too many arguments");
+			debug("argv[%d]=%s\n", argc, argv[argc - 1]);
+		}
+	} while (*str != '\0');
+	if (argc < 2)
+		semantic_error("Need event-name and probe-point at least.");
+
+	/* Parse the event name */
+	if (argv[0][0] == 'r')
+		retp = 1;
+	else if (argv[0][0] != 'p')
+		semantic_error("You must specify 'p'(kprobe) or"
+				" 'r'(kretprobe) first.");
+	/* TODO: check event name */
+	*event = argv[0];
+
+	/* Parse probe point */
+	arg = argv[1];
+	if (arg[0] == '@') {
+		/* Source Line */
+		arg++;
+		ptr = strchr(arg, ':');
+		if (!ptr || !isdigit(ptr[1]))
+			semantic_error("Line number is required.");
+		*ptr++ = '\0';
+		if (strlen(arg) == 0)
+			semantic_error("No file name.");
+		pp->file = strdup(arg);
+		pp->line = atoi(ptr);
+		if (!pp->file || !pp->line)
+			semantic_error("Failed to parse line.");
+		debug("file:%s line:%d\n", pp->file, pp->line);
+	} else {
+		/* Function name */
+		ptr = strchr(arg, '+');
+		if (ptr) {
+			if (!isdigit(ptr[1]))
+				semantic_error("Offset is required.");
+			*ptr++ = '\0';
+			pp->offset = atoi(ptr);
+		} else
+			ptr = arg;
+		ptr = strchr(ptr, '@');
+		if (ptr) {
+			*ptr++ = '\0';
+			pp->file = strdup(ptr);
+		}
+		pp->function = strdup(arg);
+		debug("symbol:%s file:%s offset:%d\n",
+		      pp->function, pp->file, pp->offset);
+	}
+	free(argv[1]);
+
+	/* Copy arguments */
+	pp->nr_args = argc - 2;
+	if (pp->nr_args > 0) {
+		pp->args = (char **)malloc(sizeof(char *) * pp->nr_args);
+		if (!pp->args)
+			perror_exit("malloc");
+		memcpy(pp->args, &argv[2], sizeof(char *) * pp->nr_args);
+	}
+
+	/* Ensure return probe has no C argument */
+	if (retp)
+		for (i = 0; i < pp->nr_args; i++)
+			if (is_c_varname(pp->args[i]))
+				semantic_error("You can't specify local"
+						" variable for kretprobe");
+	debug("%d arguments\n", pp->nr_args);
+	return 0;
+}
+
+static int open_default_vmlinux(void)
+{
+	struct utsname uts;
+	char fname[MAX_PATH_LEN];
+	int fd, ret, i;
+
+	ret = uname(&uts);
+	if (ret) {
+		debug("uname() failed.\n");
+		return -errno;
+	}
+	session.release = uts.release;
+	for (i = 0; i < NR_SEARCH_PATH; i++) {
+		ret = snprintf(fname, MAX_PATH_LEN,
+			       default_search_path[i], session.release);
+		if (ret >= MAX_PATH_LEN || ret < 0) {
+			debug("Filename(%d,%s) is too long.\n", i, uts.release);
+			errno = E2BIG;
+			return -E2BIG;
+		}
+		debug("try to open %s\n", fname);
+		fd = open(fname, O_RDONLY);
+		if (fd >= 0)
+			break;
+	}
+	return fd;
+}
+
+static const char * const probe_usage[] = {
+	"perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...]",
+	NULL
+};
+
+static const struct option options[] = {
+	OPT_STRING('k', "vmlinux", &session.vmlinux, "file",
+		"vmlinux/module pathname"),
+	OPT_CALLBACK('P', "probe", NULL,
+		"p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]",
+		"probe point definition, where\n"
+		"\t\tp:\tkprobe probe\n"
+		"\t\tr:\tkretprobe probe\n"
+		"\t\tGRP:\tGroup name (optional)\n"
+		"\t\tNAME:\tEvent name\n"
+		"\t\tFUNC:\tFunction name\n"
+		"\t\tOFFS:\tOffset from function entry (in byte)\n"
+		"\t\tSRC:\tSource code path\n"
+		"\t\tLINE:\tLine number\n"
+		"\t\tARG:\tProbe argument (local variable name or\n"
+		"\t\t\tkprobe-tracer argument format is supported.)\n",
+		parse_probepoint),
+	OPT_END()
+};
+
+static int write_new_event(int fd, const char *buf)
+{
+	int ret;
+
+	printf("Adding new event: %s\n", buf);
+	ret = write(fd, buf, strlen(buf));
+	if (ret <= 0)
+		perror("Error: Failed to create event");
+
+	return ret;
+}
+
+#define MAX_CMDLEN 256
+
+static int synthesize_probepoint(struct probe_point *pp)
+{
+	char *buf;
+	int i, len, ret;
+	pp->probes[0] = buf = (char *)calloc(MAX_CMDLEN, sizeof(char));
+	if (!buf)
+		perror_exit("calloc");
+	ret = snprintf(buf, MAX_CMDLEN, "%s+%d", pp->function, pp->offset);
+	if (ret <= 0 || ret >= MAX_CMDLEN)
+		goto error;
+	len = ret;
+
+	for (i = 0; i < pp->nr_args; i++) {
+		ret = snprintf(&buf[len], MAX_CMDLEN - len, " %s",
+			       pp->args[i]);
+		if (ret <= 0 || ret >= MAX_CMDLEN - len)
+			goto error;
+		len += ret;
+	}
+	pp->found = 1;
+	return pp->found;
+error:
+	free(pp->probes[0]);
+	if (ret > 0)
+		ret = -E2BIG;
+	return ret;
+}
+
+int cmd_probe(int argc, const char **argv, const char *prefix __used)
+{
+	int i, j, fd, ret, need_dwarf = 0;
+	struct probe_point *pp;
+	char buf[MAX_CMDLEN];
+
+	argc = parse_options(argc, argv, options, probe_usage,
+		PARSE_OPT_STOP_AT_NON_OPTION);
+	if (argc || session.nr_probe == 0)
+		usage_with_options(probe_usage, options);
+
+	/* Synthesize return probes */
+	for (j = 0; j < session.nr_probe; j++) {
+		if (session.events[j][0] != 'r') {
+			need_dwarf = 1;
+			continue;
+		}
+		ret = synthesize_probepoint(&session.probes[j]);
+		if (ret == -E2BIG)
+			semantic_error("probe point is too long.");
+		else if (ret < 0) {
+			perror("snprintf");
+			return -1;
+		}
+	}
+
+	if (!need_dwarf)
+		goto setup_probes;
+
+	if (session.vmlinux)
+		fd = open(session.vmlinux, O_RDONLY);
+	else
+		fd = open_default_vmlinux();
+	if (fd < 0) {
+		perror("vmlinux/module file open");
+		return -1;
+	}
+
+	/* Searching probe points */
+	for (j = 0; j < session.nr_probe; j++) {
+		pp = &session.probes[j];
+		if (pp->found)
+			continue;
+
+		lseek(fd, SEEK_SET, 0);
+		ret = find_probepoint(fd, pp);
+		if (ret <= 0) {
+			fprintf(stderr, "Error: No probe point found.\n");
+			return -1;
+		}
+		debug("probe event %s found\n", session.events[j]);
+	}
+	close(fd);
+
+setup_probes:
+	/* Settng up probe points */
+	snprintf(buf, MAX_CMDLEN, "%s/../kprobe_events", debugfs_path);
+	fd = open(buf, O_WRONLY, O_APPEND);
+	if (fd < 0) {
+		perror("kprobe_events open");
+		return -1;
+	}
+	for (j = 0; j < session.nr_probe; j++) {
+		pp = &session.probes[j];
+		if (pp->found == 1) {
+			snprintf(buf, MAX_CMDLEN, "%s %s\n",
+				session.events[j], pp->probes[0]);
+			write_new_event(fd, buf);
+		} else
+			for (i = 0; i < pp->found; i++) {
+				snprintf(buf, MAX_CMDLEN, "%s%d %s\n",
+					session.events[j], i, pp->probes[i]);
+				write_new_event(fd, buf);
+			}
+	}
+	close(fd);
+	return 0;
+}
+
diff --git a/tools/perf/builtin.h b/tools/perf/builtin.h
index e11d8d2..ad5f0f4 100644
--- a/tools/perf/builtin.h
+++ b/tools/perf/builtin.h
@@ -25,5 +25,6 @@ extern int cmd_timechart(int argc, const char **argv, const char *prefix);
 extern int cmd_top(int argc, const char **argv, const char *prefix);
 extern int cmd_trace(int argc, const char **argv, const char *prefix);
 extern int cmd_version(int argc, const char **argv, const char *prefix);
+extern int cmd_probe(int argc, const char **argv, const char *prefix);
 
 #endif
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index 19fc7fe..f598120 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -295,6 +295,9 @@ static void handle_internal_command(int argc, const char **argv)
 		{ "version", cmd_version, 0 },
 		{ "trace", cmd_trace, 0 },
 		{ "sched", cmd_sched, 0 },
+#ifdef SUPPORT_DWARF
+		{ "probe", cmd_probe, 0 },
+#endif
 	};
 	unsigned int i;
 	static const char ext[] = STRIP_EXTENSION;
diff --git a/tools/perf/util/probe-finder.c b/tools/perf/util/probe-finder.c
new file mode 100644
index 0000000..ec6f53f
--- /dev/null
+++ b/tools/perf/util/probe-finder.c
@@ -0,0 +1,690 @@
+/*
+ * probe-finder.c : C expression to kprobe event converter
+ *
+ * Written by Masami Hiramatsu <mhiramat@redhat.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
+ *
+ */
+
+#include <sys/utsname.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <fcntl.h>
+#include <errno.h>
+#include <stdio.h>
+#include <unistd.h>
+#include <getopt.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdarg.h>
+#include <ctype.h>
+#include "probe-finder.h"
+
+
+/* Dwarf_Die Linkage to parent Die */
+struct die_link {
+	struct die_link *parent;	/* Parent die */
+	Dwarf_Die die;			/* Current die */
+};
+
+static Dwarf_Debug __dw_debug;
+static Dwarf_Error __dw_error;
+
+static void msg_exit(int ret, const char *fmt, ...)
+{
+	va_list ap;
+
+	va_start(ap, fmt);
+	fprintf(stderr, "Error:  ");
+	vfprintf(stderr, fmt, ap);
+	va_end(ap);
+
+	fprintf(stderr, "\n");
+	exit(ret);
+}
+
+
+/*
+ * Generic dwarf analysis helpers
+ */
+
+#define X86_32_MAX_REGS 8
+const char *x86_32_regs_table[X86_32_MAX_REGS] = {
+	"%ax",
+	"%cx",
+	"%dx",
+	"%bx",
+	"$stack",	/* Stack address instead of %sp */
+	"%bp",
+	"%si",
+	"%di",
+};
+
+#define X86_64_MAX_REGS 16
+const char *x86_64_regs_table[X86_64_MAX_REGS] = {
+	"%ax",
+	"%dx",
+	"%cx",
+	"%bx",
+	"%si",
+	"%di",
+	"%bp",
+	"%sp",
+	"%r8",
+	"%r9",
+	"%r10",
+	"%r11",
+	"%r12",
+	"%r13",
+	"%r14",
+	"%r15",
+};
+
+/* TODO: switching by dwarf address size */
+#ifdef __x86_64__
+#define ARCH_MAX_REGS X86_64_MAX_REGS
+#define arch_regs_table x86_64_regs_table
+#else
+#define ARCH_MAX_REGS X86_32_MAX_REGS
+#define arch_regs_table x86_32_regs_table
+#endif
+
+/* Return architecture dependent register string (for kprobe-tracer) */
+static const char *get_arch_regstr(unsigned int n)
+{
+	return (n <= ARCH_MAX_REGS) ? arch_regs_table[n] : NULL;
+}
+
+/*
+ * Compare the tail of two strings.
+ * Return 0 if whole of either string is same as another's tail part.
+ */
+static int strtailcmp(const char *s1, const char *s2)
+{
+	int i1 = strlen(s1);
+	int i2 = strlen(s2);
+	while (--i1 > 0 && --i2 > 0) {
+		if (s1[i1] != s2[i2])
+			return s1[i1] - s2[i2];
+	}
+	return 0;
+}
+
+/* Find the fileno of the target file. */
+static Dwarf_Unsigned die_get_fileno(Dwarf_Die cu_die, const char *fname)
+{
+	Dwarf_Signed cnt, i;
+	Dwarf_Unsigned found = 0;
+	char **srcs;
+	int ret;
+
+	if (!fname)
+		return 0;
+
+	ret = dwarf_srcfiles(cu_die, &srcs, &cnt, &__dw_error);
+	if (ret == DW_DLV_OK) {
+		for (i = 0; i < cnt && !found; i++) {
+			if (strtailcmp(srcs[i], fname) == 0)
+				found = i + 1;
+			dwarf_dealloc(__dw_debug, srcs[i], DW_DLA_STRING);
+		}
+		for (; i < cnt; i++)
+			dwarf_dealloc(__dw_debug, srcs[i], DW_DLA_STRING);
+		dwarf_dealloc(__dw_debug, srcs, DW_DLA_LIST);
+	}
+	if (found)
+		debug("found fno: %d\n", (int)found);
+	return found;
+}
+
+/* Compare diename and tname */
+static int die_compare_name(Dwarf_Die die, const char *tname)
+{
+	char *name;
+	int ret;
+	ret = dwarf_diename(die, &name, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_OK) {
+		ret = strcmp(tname, name);
+		dwarf_dealloc(__dw_debug, name, DW_DLA_STRING);
+	} else
+		ret = -1;
+	return ret;
+}
+
+/* Check the address is in the subprogram(function). */
+static int die_within_subprogram(Dwarf_Die sp_die, Dwarf_Addr addr,
+				 Dwarf_Signed *offs)
+{
+	Dwarf_Addr lopc, hipc;
+	int ret;
+
+	/* TODO: check ranges */
+	ret = dwarf_lowpc(sp_die, &lopc, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_NO_ENTRY)
+		return 0;
+	ret = dwarf_highpc(sp_die, &hipc, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	if (lopc <= addr && addr < hipc) {
+		*offs = addr - lopc;
+		return 1;
+	} else
+		return 0;
+}
+
+/* Check the die is inlined function */
+static Dwarf_Bool die_inlined_subprogram(Dwarf_Die die)
+{
+	/* TODO: check strictly */
+	Dwarf_Bool inl;
+	int ret;
+
+	ret = dwarf_hasattr(die, DW_AT_inline, &inl, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	return inl;
+}
+
+/* Get the offset of abstruct_origin */
+static Dwarf_Off die_get_abstract_origin(Dwarf_Die die)
+{
+	Dwarf_Attribute attr;
+	Dwarf_Off cu_offs;
+	int ret;
+
+	ret = dwarf_attr(die, DW_AT_abstract_origin, &attr, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	ret = dwarf_formref(attr, &cu_offs, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	dwarf_dealloc(__dw_debug, attr, DW_DLA_ATTR);
+	return cu_offs;
+}
+
+/* Get entry pc(or low pc, 1st entry of ranges)  of the die */
+static Dwarf_Addr die_get_entrypc(Dwarf_Die die)
+{
+	Dwarf_Attribute attr;
+	Dwarf_Addr addr;
+	Dwarf_Off offs;
+	Dwarf_Ranges *ranges;
+	Dwarf_Signed cnt;
+	int ret;
+
+	/* Try to get entry pc */
+	ret = dwarf_attr(die, DW_AT_entry_pc, &attr, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_OK) {
+		ret = dwarf_formaddr(attr, &addr, &__dw_error);
+		ERR_IF(ret != DW_DLV_OK);
+		dwarf_dealloc(__dw_debug, attr, DW_DLA_ATTR);
+		return addr;
+	}
+
+	/* Try to get low pc */
+	ret = dwarf_lowpc(die, &addr, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_OK)
+		return addr;
+
+	/* Try to get ranges */
+	ret = dwarf_attr(die, DW_AT_ranges, &attr, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	ret = dwarf_formref(attr, &offs, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	ret = dwarf_get_ranges(__dw_debug, offs, &ranges, &cnt, NULL,
+				&__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	addr = ranges[0].dwr_addr1;
+	dwarf_ranges_dealloc(__dw_debug, ranges, cnt);
+	return addr;
+}
+
+/*
+ * Search a Die from Die tree.
+ * Note: cur_link->die should be deallocated in this function.
+ */
+static int __search_die_tree(struct die_link *cur_link,
+			     int (*die_cb)(struct die_link *, void *),
+			     void *data)
+{
+	Dwarf_Die new_die;
+	struct die_link new_link;
+	int ret;
+
+	if (!die_cb)
+		return 0;
+
+	/* Check current die */
+	while (!(ret = die_cb(cur_link, data))) {
+		/* Check child die */
+		ret = dwarf_child(cur_link->die, &new_die, &__dw_error);
+		ERR_IF(ret == DW_DLV_ERROR);
+		if (ret == DW_DLV_OK) {
+			new_link.parent = cur_link;
+			new_link.die = new_die;
+			ret = __search_die_tree(&new_link, die_cb, data);
+			if (ret)
+				break;
+		}
+
+		/* Move to next sibling */
+		ret = dwarf_siblingof(__dw_debug, cur_link->die, &new_die,
+				      &__dw_error);
+		ERR_IF(ret == DW_DLV_ERROR);
+		dwarf_dealloc(__dw_debug, cur_link->die, DW_DLA_DIE);
+		cur_link->die = new_die;
+		if (ret == DW_DLV_NO_ENTRY)
+			return 0;
+	}
+	dwarf_dealloc(__dw_debug, cur_link->die, DW_DLA_DIE);
+	return ret;
+}
+
+/* Search a die in its children's die tree */
+static int search_die_from_children(Dwarf_Die parent_die,
+				    int (*die_cb)(struct die_link *, void *),
+				    void *data)
+{
+	struct die_link new_link;
+	int ret;
+
+	new_link.parent = NULL;
+	ret = dwarf_child(parent_die, &new_link.die, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_OK)
+		return __search_die_tree(&new_link, die_cb, data);
+	else
+		return 0;
+}
+
+/* Find a locdesc corresponding to the address */
+static int attr_get_locdesc(Dwarf_Attribute attr, Dwarf_Locdesc *desc,
+			    Dwarf_Addr addr)
+{
+	Dwarf_Signed lcnt;
+	Dwarf_Locdesc **llbuf;
+	int ret, i;
+
+	ret = dwarf_loclist_n(attr, &llbuf, &lcnt, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	ret = DW_DLV_NO_ENTRY;
+	for (i = 0; i < lcnt; ++i) {
+		if (llbuf[i]->ld_lopc <= addr &&
+		    llbuf[i]->ld_hipc > addr) {
+			memcpy(desc, llbuf[i], sizeof(Dwarf_Locdesc));
+			desc->ld_s =
+				malloc(sizeof(Dwarf_Loc) * llbuf[i]->ld_cents);
+			ERR_IF(desc->ld_s == NULL);
+			memcpy(desc->ld_s, llbuf[i]->ld_s,
+				sizeof(Dwarf_Loc) * llbuf[i]->ld_cents);
+			ret = DW_DLV_OK;
+			break;
+		}
+		dwarf_dealloc(__dw_debug, llbuf[i]->ld_s, DW_DLA_LOC_BLOCK);
+		dwarf_dealloc(__dw_debug, llbuf[i], DW_DLA_LOCDESC);
+	}
+	/* Releasing loop */
+	for (; i < lcnt; ++i) {
+		dwarf_dealloc(__dw_debug, llbuf[i]->ld_s, DW_DLA_LOC_BLOCK);
+		dwarf_dealloc(__dw_debug, llbuf[i], DW_DLA_LOCDESC);
+	}
+	dwarf_dealloc(__dw_debug, llbuf, DW_DLA_LIST);
+	return ret;
+}
+
+/*
+ * Probe finder related functions
+ */
+
+/* Show a location */
+static void show_location(Dwarf_Loc *loc, struct probe_finder *pf)
+{
+	Dwarf_Small op;
+	Dwarf_Unsigned regn;
+	Dwarf_Signed offs;
+	int deref = 0, ret;
+	const char *regs;
+
+	op = loc->lr_atom;
+
+	/* If this is based on frame buffer, set the offset */
+	if (op == DW_OP_fbreg) {
+		deref = 1;
+		offs = (Dwarf_Signed)loc->lr_number;
+		op = pf->fbloc.ld_s[0].lr_atom;
+		loc = &pf->fbloc.ld_s[0];
+	} else
+		offs = 0;
+
+	if (op >= DW_OP_breg0 && op <= DW_OP_breg31) {
+		regn = op - DW_OP_breg0;
+		offs += (Dwarf_Signed)loc->lr_number;
+		deref = 1;
+	} else if (op >= DW_OP_reg0 && op <= DW_OP_reg31) {
+		regn = op - DW_OP_reg0;
+	} else if (op == DW_OP_bregx) {
+		regn = loc->lr_number;
+		offs += (Dwarf_Signed)loc->lr_number2;
+		deref = 1;
+	} else if (op == DW_OP_regx) {
+		regn = loc->lr_number;
+	} else
+		msg_exit(-EINVAL, "Dwarf_OP %d is not supported.\n", op);
+
+	regs = get_arch_regstr(regn);
+	if (!regs)
+		msg_exit(-EINVAL, "%lld exceeds max register number.\n", regn);
+
+	if (deref)
+		ret = snprintf(pf->buf, pf->len,
+				 " %s=%+lld(%s)", pf->var, offs, regs);
+	else
+		ret = snprintf(pf->buf, pf->len, " %s=%s", pf->var, regs);
+	ERR_IF(ret < 0);
+	ERR_IF(ret >= pf->len);
+}
+
+/* Show a variables in kprobe event format */
+static void show_variable(Dwarf_Die vr_die, struct probe_finder *pf)
+{
+	Dwarf_Attribute attr;
+	Dwarf_Locdesc ld;
+	int ret;
+
+	ret = dwarf_attr(vr_die, DW_AT_location, &attr, &__dw_error);
+	if (ret != DW_DLV_OK)
+		goto error;
+	ret = attr_get_locdesc(attr, &ld, (pf->addr - pf->cu_base));
+	if (ret != DW_DLV_OK)
+		goto error;
+	/* TODO? */
+	ERR_IF(ld.ld_cents != 1);
+	show_location(&ld.ld_s[0], pf);
+	free(ld.ld_s);
+	dwarf_dealloc(__dw_debug, attr, DW_DLA_ATTR);
+	return ;
+error:
+	msg_exit(-1, "Failed to find the location of %s at this address.\n"
+		 " Perhaps, it was optimized out.\n", pf->var);
+}
+
+static int variable_callback(struct die_link *dlink, void *data)
+{
+	struct probe_finder *pf = (struct probe_finder *)data;
+	Dwarf_Half tag;
+	int ret;
+
+	ret = dwarf_tag(dlink->die, &tag, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if ((tag == DW_TAG_formal_parameter ||
+	     tag == DW_TAG_variable) &&
+	    (die_compare_name(dlink->die, pf->var) == 0)) {
+		show_variable(dlink->die, pf);
+		return 1;
+	}
+	/* TODO: Support struct members and arrays */
+	return 0;
+}
+
+/* Find a variable in a subprogram die */
+static void find_variable(Dwarf_Die sp_die, struct probe_finder *pf)
+{
+	int ret;
+
+	if (!is_c_varname(pf->var)) {
+		/* Output raw parameters */
+		ret = snprintf(pf->buf, pf->len, " %s", pf->var);
+		ERR_IF(ret < 0);
+		ERR_IF(ret >= pf->len);
+		return ;
+	}
+
+	debug("Searching '%s' variable in context.\n", pf->var);
+	/* Search child die for local variables and parameters. */
+	ret = search_die_from_children(sp_die, variable_callback, pf);
+	if (!ret)
+		msg_exit(-1, "Failed to find '%s' in this function.\n",
+			 pf->var);
+}
+
+/* Get a frame base on the address */
+static void get_current_frame_base(Dwarf_Die sp_die, struct probe_finder *pf)
+{
+	Dwarf_Attribute attr;
+	int ret;
+
+	ret = dwarf_attr(sp_die, DW_AT_frame_base, &attr, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+	ret = attr_get_locdesc(attr, &pf->fbloc, (pf->addr - pf->cu_base));
+	ERR_IF(ret != DW_DLV_OK);
+	dwarf_dealloc(__dw_debug, attr, DW_DLA_ATTR);
+}
+
+static void free_current_frame_base(struct probe_finder *pf)
+{
+	free(pf->fbloc.ld_s);
+	memset(&pf->fbloc, 0, sizeof(Dwarf_Locdesc));
+}
+
+/* Show a probe point to output buffer */
+static void show_probepoint(Dwarf_Die sp_die, Dwarf_Signed offs,
+			    struct probe_finder *pf)
+{
+	struct probe_point *pp = pf->pp;
+	char *name;
+	char tmp[MAX_PROBE_BUFFER];
+	int ret, i, len;
+
+	/* Output name of probe point */
+	ret = dwarf_diename(sp_die, &name, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (ret == DW_DLV_OK) {
+		ret = snprintf(tmp, MAX_PROBE_BUFFER, "%s+%u", name,
+				(unsigned int)offs);
+		dwarf_dealloc(__dw_debug, name, DW_DLA_STRING);
+	} else {
+		/* This function has no name. */
+		ret = snprintf(tmp, MAX_PROBE_BUFFER, "0x%llx", pf->addr);
+	}
+	ERR_IF(ret < 0);
+	ERR_IF(ret >= MAX_PROBE_BUFFER);
+	len = ret;
+
+	/* Find each argument */
+	get_current_frame_base(sp_die, pf);
+	for (i = 0; i < pp->nr_args; i++) {
+		pf->var = pp->args[i];
+		pf->buf = &tmp[len];
+		pf->len = MAX_PROBE_BUFFER - len;
+		find_variable(sp_die, pf);
+		len += strlen(pf->buf);
+	}
+	free_current_frame_base(pf);
+
+	pp->probes[pp->found] = strdup(tmp);
+	pp->found++;
+}
+
+static int probeaddr_callback(struct die_link *dlink, void *data)
+{
+	struct probe_finder *pf = (struct probe_finder *)data;
+	Dwarf_Half tag;
+	Dwarf_Signed offs;
+	int ret;
+
+	ret = dwarf_tag(dlink->die, &tag, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	/* Check the address is in this subprogram */
+	if (tag == DW_TAG_subprogram &&
+	    die_within_subprogram(dlink->die, pf->addr, &offs)) {
+		show_probepoint(dlink->die, offs, pf);
+		return 1;
+	}
+	return 0;
+}
+
+/* Find probe point from its line number */
+static void find_by_line(Dwarf_Die cu_die, struct probe_finder *pf)
+{
+	struct probe_point *pp = pf->pp;
+	Dwarf_Signed cnt, i;
+	Dwarf_Line *lines;
+	Dwarf_Unsigned lineno = 0;
+	Dwarf_Addr addr;
+	Dwarf_Unsigned fno;
+	int ret;
+
+	ret = dwarf_srclines(cu_die, &lines, &cnt, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+
+	for (i = 0; i < cnt; i++) {
+		ret = dwarf_line_srcfileno(lines[i], &fno, &__dw_error);
+		ERR_IF(ret != DW_DLV_OK);
+		if (fno != pf->fno)
+			continue;
+
+		ret = dwarf_lineno(lines[i], &lineno, &__dw_error);
+		ERR_IF(ret != DW_DLV_OK);
+		if (lineno != (Dwarf_Unsigned)pp->line)
+			continue;
+
+		ret = dwarf_lineaddr(lines[i], &addr, &__dw_error);
+		ERR_IF(ret != DW_DLV_OK);
+		debug("Probe point found: 0x%llx\n", addr);
+		pf->addr = addr;
+		/* Search a real subprogram including this line, */
+		ret = search_die_from_children(cu_die, probeaddr_callback, pf);
+		if (ret == 0)
+			msg_exit(-1,
+				 "Probe point is not found in subprograms.\n");
+		/* Continuing, because target line might be inlined. */
+	}
+	dwarf_srclines_dealloc(__dw_debug, lines, cnt);
+}
+
+/* Search function from function name */
+static int probefunc_callback(struct die_link *dlink, void *data)
+{
+	struct probe_finder *pf = (struct probe_finder *)data;
+	struct probe_point *pp = pf->pp;
+	struct die_link *lk;
+	Dwarf_Signed offs;
+	Dwarf_Half tag;
+	int ret;
+
+	ret = dwarf_tag(dlink->die, &tag, &__dw_error);
+	ERR_IF(ret == DW_DLV_ERROR);
+	if (tag == DW_TAG_subprogram) {
+		if (die_compare_name(dlink->die, pp->function) == 0) {
+			if (die_inlined_subprogram(dlink->die)) {
+				/* Inlined function, save it. */
+				ret = dwarf_die_CU_offset(dlink->die,
+							  &pf->inl_offs,
+							  &__dw_error);
+				ERR_IF(ret != DW_DLV_OK);
+				debug("inline definition offset %lld\n",
+					pf->inl_offs);
+				return 0;
+			}
+			/* Get probe address */
+			pf->addr = die_get_entrypc(dlink->die);
+			pf->addr += pp->offset;
+			/* TODO: Check the address in this function */
+			show_probepoint(dlink->die, pp->offset, pf);
+			/* Continue to search */
+		}
+	} else if (tag == DW_TAG_inlined_subroutine && pf->inl_offs) {
+		if (die_get_abstract_origin(dlink->die) == pf->inl_offs) {
+			/* Get probe address */
+			pf->addr = die_get_entrypc(dlink->die);
+			pf->addr += pp->offset;
+			debug("found inline addr: 0x%llx\n", pf->addr);
+			/* Inlined function. Get a real subprogram */
+			for (lk = dlink->parent; lk != NULL; lk = lk->parent) {
+				tag = 0;
+				dwarf_tag(lk->die, &tag, &__dw_error);
+				ERR_IF(ret == DW_DLV_ERROR);
+				if (tag == DW_TAG_subprogram &&
+				    !die_inlined_subprogram(lk->die))
+					goto found;
+			}
+			msg_exit(-1, "Failed to find real subprogram.\n");
+found:
+			/* Get offset from subprogram */
+			ret = die_within_subprogram(lk->die, pf->addr, &offs);
+			ERR_IF(!ret);
+			show_probepoint(lk->die, offs, pf);
+			/* Continue to search */
+		}
+	}
+	return 0;
+}
+
+static void find_by_func(Dwarf_Die cu_die, struct probe_finder *pf)
+{
+	search_die_from_children(cu_die, probefunc_callback, pf);
+}
+
+/* Find a probe point */
+int find_probepoint(int fd, struct probe_point *pp)
+{
+	Dwarf_Half addr_size = 0;
+	Dwarf_Unsigned next_cuh = 0;
+	Dwarf_Die cu_die = 0;
+	int cu_number = 0, ret;
+	struct probe_finder pf = {.pp = pp};
+
+	ret = dwarf_init(fd, DW_DLC_READ, 0, 0, &__dw_debug, &__dw_error);
+	if (ret != DW_DLV_OK)
+		msg_exit(-1, "Failed to call dwarf_init(). "
+			 "Maybe, not a dwarf file?\n");
+
+	pp->found = 0;
+	while (++cu_number) {
+		/* Search CU (Compilation Unit) */
+		ret = dwarf_next_cu_header(__dw_debug, NULL, NULL, NULL,
+			&addr_size, &next_cuh, &__dw_error);
+		ERR_IF(ret == DW_DLV_ERROR);
+		if (ret == DW_DLV_NO_ENTRY)
+			break;
+
+		/* Get the DIE(Debugging Information Entry) of this CU */
+		ret = dwarf_siblingof(__dw_debug, 0, &cu_die, &__dw_error);
+		ERR_IF(ret != DW_DLV_OK);
+
+		/* Check if target file is included. */
+		if (pp->file)
+			pf.fno = die_get_fileno(cu_die, pp->file);
+
+		if (!pp->file || pf.fno) {
+			/* Save CU base address (for frame_base) */
+			ret = dwarf_lowpc(cu_die, &pf.cu_base, &__dw_error);
+			ERR_IF(ret == DW_DLV_ERROR);
+			if (ret == DW_DLV_NO_ENTRY)
+				pf.cu_base = 0;
+			if (pp->line)
+				find_by_line(cu_die, &pf);
+			if (pp->function)
+				find_by_func(cu_die, &pf);
+		}
+		dwarf_dealloc(__dw_debug, cu_die, DW_DLA_DIE);
+	}
+	ret = dwarf_finish(__dw_debug, &__dw_error);
+	ERR_IF(ret != DW_DLV_OK);
+
+	return pp->found;
+}
+
diff --git a/tools/perf/util/probe-finder.h b/tools/perf/util/probe-finder.h
new file mode 100644
index 0000000..af920de
--- /dev/null
+++ b/tools/perf/util/probe-finder.h
@@ -0,0 +1,68 @@
+#ifndef _PROBE_FINDER_H
+#define _PROBE_FINDER_H
+
+#define _stringify(n)	#n
+#define stringify(n)	_stringify(n)
+
+#ifdef DEBUG
+#define debug(fmt ...)	\
+	fprintf(stderr, "DBG(" __FILE__ ":" stringify(__LINE__) "): " fmt)
+#else
+#define debug(fmt ...)	do {} while (0)
+#endif
+
+#define ERR_IF(cnd)	\
+	do { if (cnd) {	\
+		fprintf(stderr, "Error (" __FILE__ ":" stringify(__LINE__) \
+			"): " stringify(cnd) "\n");			\
+		exit(1);						\
+	} } while (0)
+
+#define MAX_PATH_LEN 256
+#define MAX_PROBE_BUFFER 1024
+#define MAX_PROBES 128
+
+static inline int is_c_varname(const char *name)
+{
+	/* TODO */
+	return isalpha(name[0]) || name[0] == '_';
+}
+
+struct probe_point {
+	/* Inputs */
+	char	*file;		/* File name */
+	int	line;		/* Line number */
+
+	char	*function;	/* Function name */
+	int	offset;		/* Offset bytes */
+
+	int	nr_args;	/* Number of arguments */
+	char	**args;		/* Arguments */
+
+	/* Output */
+	int	found;		/* Number of found probe points */
+	char	*probes[MAX_PROBES];	/* Output buffers (will be allocated)*/
+};
+
+extern int find_probepoint(int fd, struct probe_point *pp);
+
+#include <libdwarf/dwarf.h>
+#include <libdwarf/libdwarf.h>
+
+struct probe_finder {
+	struct probe_point	*pp;	/* Target probe point */
+
+	/* For function searching */
+	Dwarf_Addr	addr;		/* Address */
+	Dwarf_Unsigned	fno;		/* File number */
+	Dwarf_Off	inl_offs;	/* Inline offset */
+
+	/* For variable searching */
+	Dwarf_Addr	cu_base;	/* Current CU base address */
+	Dwarf_Locdesc	fbloc;		/* Location of Current Frame Base */
+	const char	*var;		/* Current variable name */
+	char		*buf;		/* Current output buffer */
+	int		len;		/* Length of output buffer */
+};
+
+#endif /*_PROBE_FINDER_H */
-- 
1.6.2.3

[PATCH 7/7] perf probe: Add perf probe command support without libdwarf

[Frederic Weisbecker]

From: Masami Hiramatsu <mhiramat@redhat.com>

Enables 'perf probe' even if libdwarf is not installed. If libdwarf is
not found, 'perf probe' just disables dwarf support. Users can use
'perf probe' to set up new events by using kprobe_events format.

Signed-off-by: Masami Hiramatsu <mhiramat@redhat.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Ananth N Mavinakayanahalli <ananth@in.ibm.com>
Cc: Jim Keniston <jkenisto@us.ibm.com>
Cc: Frank Ch. Eigler <fche@redhat.com>
LKML-Reference: <20091007222830.1684.25665.stgit@dhcp-100-2-132.bos.redhat.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
---
 tools/perf/Makefile            |    6 ++--
 tools/perf/builtin-probe.c     |   42 +++++++++++++++++++++++++++++++++------
 tools/perf/perf.c              |    2 -
 tools/perf/util/probe-finder.h |    2 +
 4 files changed, 40 insertions(+), 12 deletions(-)

diff --git a/tools/perf/Makefile b/tools/perf/Makefile
index 6dabcf1..52b1f43 100644
--- a/tools/perf/Makefile
+++ b/tools/perf/Makefile
@@ -385,6 +385,7 @@ BUILTIN_OBJS += builtin-stat.o
 BUILTIN_OBJS += builtin-timechart.o
 BUILTIN_OBJS += builtin-top.o
 BUILTIN_OBJS += builtin-trace.o
+BUILTIN_OBJS += builtin-probe.o
 
 PERFLIBS = $(LIB_FILE)
 
@@ -420,13 +421,12 @@ ifneq ($(shell sh -c "(echo '\#include <libelf.h>'; echo 'int main(void) { Elf *
 endif
 
 ifneq ($(shell sh -c "(echo '\#include <libdwarf/dwarf.h>'; echo '\#include <libdwarf/libdwarf.h>'; echo 'int main(void) { Dwarf_Debug dbg; Dwarf_Error err; dwarf_init(0, DW_DLC_READ, 0, 0, &dbg, &err); return (long)dbg; }') | $(CC) -x c - $(ALL_CFLAGS) -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -ldwarf -lelf -o /dev/null $(ALL_LDFLAGS) > /dev/null 2>&1 && echo y"), y)
-	msg := $(warning No libdwarf.h found, disables probe subcommand. Please install libdwarf-dev/libdwarf-devel);
+	msg := $(warning No libdwarf.h found, disables dwarf support. Please install libdwarf-dev/libdwarf-devel);
+	BASIC_CFLAGS += -DNO_LIBDWARF
 else
 	EXTLIBS += -lelf -ldwarf
 	LIB_H += util/probe-finder.h
 	LIB_OBJS += util/probe-finder.o
-	BUILTIN_OBJS += builtin-probe.o
-	BASIC_CFLAGS += -DSUPPORT_DWARF
 endif
 
 ifdef NO_DEMANGLE
diff --git a/tools/perf/builtin-probe.c b/tools/perf/builtin-probe.c
index 24b64b5..73c883b 100644
--- a/tools/perf/builtin-probe.c
+++ b/tools/perf/builtin-probe.c
@@ -54,6 +54,7 @@ const char *default_search_path[NR_SEARCH_PATH] = {
 static struct {
 	char *vmlinux;
 	char *release;
+	int need_dwarf;
 	int nr_probe;
 	struct probe_point probes[MAX_PROBES];
 	char *events[MAX_PROBES];
@@ -162,6 +163,8 @@ static int parse_probepoint(const struct option *opt __used,
 		      pp->function, pp->file, pp->offset);
 	}
 	free(argv[1]);
+	if (pp->file)
+		session.need_dwarf = 1;
 
 	/* Copy arguments */
 	pp->nr_args = argc - 2;
@@ -173,15 +176,19 @@ static int parse_probepoint(const struct option *opt __used,
 	}
 
 	/* Ensure return probe has no C argument */
-	if (retp)
-		for (i = 0; i < pp->nr_args; i++)
-			if (is_c_varname(pp->args[i]))
+	for (i = 0; i < pp->nr_args; i++)
+		if (is_c_varname(pp->args[i])) {
+			if (retp)
 				semantic_error("You can't specify local"
 						" variable for kretprobe");
+			session.need_dwarf = 1;
+		}
+
 	debug("%d arguments\n", pp->nr_args);
 	return 0;
 }
 
+#ifndef NO_LIBDWARF
 static int open_default_vmlinux(void)
 {
 	struct utsname uts;
@@ -209,6 +216,7 @@ static int open_default_vmlinux(void)
 	}
 	return fd;
 }
+#endif
 
 static const char * const probe_usage[] = {
 	"perf probe [<options>] -P 'PROBEDEF' [-P 'PROBEDEF' ...]",
@@ -216,10 +224,16 @@ static const char * const probe_usage[] = {
 };
 
 static const struct option options[] = {
+#ifndef NO_LIBDWARF
 	OPT_STRING('k', "vmlinux", &session.vmlinux, "file",
 		"vmlinux/module pathname"),
+#endif
 	OPT_CALLBACK('P', "probe", NULL,
+#ifdef NO_LIBDWARF
+		"p|r:[GRP/]NAME FUNC[+OFFS] [ARG ...]",
+#else
 		"p|r:[GRP/]NAME FUNC[+OFFS][@SRC]|@SRC:LINE [ARG ...]",
+#endif
 		"probe point definition, where\n"
 		"\t\tp:\tkprobe probe\n"
 		"\t\tr:\tkretprobe probe\n"
@@ -227,9 +241,13 @@ static const struct option options[] = {
 		"\t\tNAME:\tEvent name\n"
 		"\t\tFUNC:\tFunction name\n"
 		"\t\tOFFS:\tOffset from function entry (in byte)\n"
+#ifdef NO_LIBDWARF
+		"\t\tARG:\tProbe argument (only \n"
+#else
 		"\t\tSRC:\tSource code path\n"
 		"\t\tLINE:\tLine number\n"
 		"\t\tARG:\tProbe argument (local variable name or\n"
+#endif
 		"\t\t\tkprobe-tracer argument format is supported.)\n",
 		parse_probepoint),
 	OPT_END()
@@ -279,7 +297,7 @@ error:
 
 int cmd_probe(int argc, const char **argv, const char *prefix __used)
 {
-	int i, j, fd, ret, need_dwarf = 0;
+	int i, j, fd, ret;
 	struct probe_point *pp;
 	char buf[MAX_CMDLEN];
 
@@ -288,12 +306,19 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	if (argc || session.nr_probe == 0)
 		usage_with_options(probe_usage, options);
 
-	/* Synthesize return probes */
+#ifdef NO_LIBDWARF
+	if (session.need_dwarf)
+		semantic_error("Dwarf-analysis is not supported");
+#endif
+
+	/* Synthesize probes without dwarf */
 	for (j = 0; j < session.nr_probe; j++) {
+#ifndef NO_LIBDWARF
 		if (session.events[j][0] != 'r') {
-			need_dwarf = 1;
+			session.need_dwarf = 1;
 			continue;
 		}
+#endif
 		ret = synthesize_probepoint(&session.probes[j]);
 		if (ret == -E2BIG)
 			semantic_error("probe point is too long.");
@@ -303,7 +328,8 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 		}
 	}
 
-	if (!need_dwarf)
+#ifndef NO_LIBDWARF
+	if (!session.need_dwarf)
 		goto setup_probes;
 
 	if (session.vmlinux)
@@ -332,6 +358,8 @@ int cmd_probe(int argc, const char **argv, const char *prefix __used)
 	close(fd);
 
 setup_probes:
+#endif /* !NO_LIBDWARF */
+
 	/* Settng up probe points */
 	snprintf(buf, MAX_CMDLEN, "%s/../kprobe_events", debugfs_path);
 	fd = open(buf, O_WRONLY, O_APPEND);
diff --git a/tools/perf/perf.c b/tools/perf/perf.c
index f598120..c570d17 100644
--- a/tools/perf/perf.c
+++ b/tools/perf/perf.c
@@ -295,9 +295,7 @@ static void handle_internal_command(int argc, const char **argv)
 		{ "version", cmd_version, 0 },
 		{ "trace", cmd_trace, 0 },
 		{ "sched", cmd_sched, 0 },
-#ifdef SUPPORT_DWARF
 		{ "probe", cmd_probe, 0 },
-#endif
 	};
 	unsigned int i;
 	static const char ext[] = STRIP_EXTENSION;
diff --git a/tools/perf/util/probe-finder.h b/tools/perf/util/probe-finder.h
index af920de..306810c 100644
--- a/tools/perf/util/probe-finder.h
+++ b/tools/perf/util/probe-finder.h
@@ -44,6 +44,7 @@ struct probe_point {
 	char	*probes[MAX_PROBES];	/* Output buffers (will be allocated)*/
 };
 
+#ifndef NO_LIBDWARF
 extern int find_probepoint(int fd, struct probe_point *pp);
 
 #include <libdwarf/dwarf.h>
@@ -64,5 +65,6 @@ struct probe_finder {
 	char		*buf;		/* Current output buffer */
 	int		len;		/* Length of output buffer */
 };
+#endif /* NO_LIBDWARF */
 
 #endif /*_PROBE_FINDER_H */
-- 
1.6.2.3

Re: [GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> Ingo,
> 
> I've tested a little this set and it seems to work well. I can't test 
> the C expression parser (need some libdwarf cross-version 
> compatiblity), but I can create kprobes via low-level parameters, then 
> launch perf record and perf trace successfully. I'll need to update 
> perf trace to handle variables that start with '$' characters though.
> 
> Anyway, it's still unstable but I guess pulling it would help Masami 
> to continue without the need to handle the whole batch.
> 
> Please pull the tracing/kprobes branch that can be found at:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
> 	tracing/kprobes

Pulled, thanks Frederic.

>  Documentation/trace/kprobetrace.txt |   27 +-
>  kernel/trace/trace_kprobe.c         |  160 +++++---
>  tools/perf/Makefile                 |   10 +
>  tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
>  tools/perf/builtin.h                |    1 +
>  tools/perf/perf.c                   |    1 +
>  tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
>  tools/perf/util/probe-finder.h      |   70 ++++
>  8 files changed, 1271 insertions(+), 74 deletions(-)
>  create mode 100644 tools/perf/builtin-probe.c
>  create mode 100644 tools/perf/util/probe-finder.c
>  create mode 100644 tools/perf/util/probe-finder.h

Hm, there's no x86 decoder change here - does this lot fixes the 
instruction decoder bug i reported some time ago? That is which keeps 
tracing/kprobes out of tip:master.

	Ingo

Re: [GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> >  tools/perf/Makefile                 |   10 +
> >  tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
> >  tools/perf/builtin.h                |    1 +
> >  tools/perf/perf.c                   |    1 +
> >  tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
> >  tools/perf/util/probe-finder.h      |   70 ++++
> >  8 files changed, 1271 insertions(+), 74 deletions(-)
> >  create mode 100644 tools/perf/builtin-probe.c
> >  create mode 100644 tools/perf/util/probe-finder.c
> >  create mode 100644 tools/perf/util/probe-finder.h
> 
> Hm, there's no x86 decoder change here - does this lot fixes the 
> instruction decoder bug i reported some time ago? That is which keeps 
> tracing/kprobes out of tip:master.

See my previous report below.

	Ingo

----- Forwarded message from Ingo Molnar <mingo@elte.hu> -----

Date: Sat, 3 Oct 2009 12:49:23 +0200
From: Ingo Molnar <mingo@elte.hu>
To: Frederic Weisbecker <fweisbec@gmail.com>
Subject: Re: [GIT PULL] tracing/kprobes: kprobes fixes
Cc: LKML <linux-kernel@vger.kernel.org>,
	Steven Rostedt <rostedt@goodmis.org>,
	Li Zefan <lizf@cn.fujitsu.com>


ok, -tip testing found this new failure:

  TEST    posttest
Error: ffffffff8106ac00:        66 0f 73 fd 04          pslldq $0x4,%xmm5
Error: objdump says 5 bytes, but insn_get_length() says 4 (attr:8000)
make[1]: *** [posttest] Error 2
make: *** [bzImage] Error 2

	Ingo

----- End forwarded message -----

Re: [GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Masami Hiramatsu]

Ingo Molnar wrote:
> 
> * Ingo Molnar <mingo@elte.hu> wrote:
> 
>>>  tools/perf/Makefile                 |   10 +
>>>  tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
>>>  tools/perf/builtin.h                |    1 +
>>>  tools/perf/perf.c                   |    1 +
>>>  tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
>>>  tools/perf/util/probe-finder.h      |   70 ++++
>>>  8 files changed, 1271 insertions(+), 74 deletions(-)
>>>  create mode 100644 tools/perf/builtin-probe.c
>>>  create mode 100644 tools/perf/util/probe-finder.c
>>>  create mode 100644 tools/perf/util/probe-finder.h
>>
>> Hm, there's no x86 decoder change here - does this lot fixes the 
>> instruction decoder bug i reported some time ago? That is which keeps 
>> tracing/kprobes out of tip:master.
> 
> See my previous report below.
> 
> 	Ingo
> 
> ----- Forwarded message from Ingo Molnar <mingo@elte.hu> -----
> 
> Date: Sat, 3 Oct 2009 12:49:23 +0200
> From: Ingo Molnar <mingo@elte.hu>
> To: Frederic Weisbecker <fweisbec@gmail.com>
> Subject: Re: [GIT PULL] tracing/kprobes: kprobes fixes
> Cc: LKML <linux-kernel@vger.kernel.org>,
> 	Steven Rostedt <rostedt@goodmis.org>,
> 	Li Zefan <lizf@cn.fujitsu.com>
> 
> 
> ok, -tip testing found this new failure:
> 
>   TEST    posttest
> Error: ffffffff8106ac00:        66 0f 73 fd 04          pslldq $0x4,%xmm5
> Error: objdump says 5 bytes, but insn_get_length() says 4 (attr:8000)
> make[1]: *** [posttest] Error 2
> make: *** [bzImage] Error 2
> 
> 	Ingo
> 

Hmm, thank you for reporting, I've missed some group opcode...

Thank you again,

-- 
Masami Hiramatsu

Software Engineer
Hitachi Computer Products (America), Inc.
Software Solutions Division

e-mail: mhiramat@redhat.com

Re: [GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Frederic Weisbecker]

On Tue, Oct 13, 2009 at 08:13:23AM +0200, Ingo Molnar wrote:
> 
> * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> 
> > Ingo,
> > 
> > I've tested a little this set and it seems to work well. I can't test 
> > the C expression parser (need some libdwarf cross-version 
> > compatiblity), but I can create kprobes via low-level parameters, then 
> > launch perf record and perf trace successfully. I'll need to update 
> > perf trace to handle variables that start with '$' characters though.
> > 
> > Anyway, it's still unstable but I guess pulling it would help Masami 
> > to continue without the need to handle the whole batch.
> > 
> > Please pull the tracing/kprobes branch that can be found at:
> > 
> > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
> > 	tracing/kprobes
> 
> Pulled, thanks Frederic.
> 
> >  Documentation/trace/kprobetrace.txt |   27 +-
> >  kernel/trace/trace_kprobe.c         |  160 +++++---
> >  tools/perf/Makefile                 |   10 +
> >  tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
> >  tools/perf/builtin.h                |    1 +
> >  tools/perf/perf.c                   |    1 +
> >  tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
> >  tools/perf/util/probe-finder.h      |   70 ++++
> >  8 files changed, 1271 insertions(+), 74 deletions(-)
> >  create mode 100644 tools/perf/builtin-probe.c
> >  create mode 100644 tools/perf/util/probe-finder.c
> >  create mode 100644 tools/perf/util/probe-finder.h
> 
> Hm, there's no x86 decoder change here - does this lot fixes the 
> instruction decoder bug i reported some time ago? That is which keeps 
> tracing/kprobes out of tip:master.
> 
> 	Ingo


Not yet, I hope it's part of the last family of instruction set we
are missing.

Re: [GIT PULL] tracing/kprobes: Syntax updates, introduce perf probe

[Ingo Molnar]

* Frederic Weisbecker <fweisbec@gmail.com> wrote:

> On Tue, Oct 13, 2009 at 08:13:23AM +0200, Ingo Molnar wrote:
> > 
> > * Frederic Weisbecker <fweisbec@gmail.com> wrote:
> > 
> > > Ingo,
> > > 
> > > I've tested a little this set and it seems to work well. I can't test 
> > > the C expression parser (need some libdwarf cross-version 
> > > compatiblity), but I can create kprobes via low-level parameters, then 
> > > launch perf record and perf trace successfully. I'll need to update 
> > > perf trace to handle variables that start with '$' characters though.
> > > 
> > > Anyway, it's still unstable but I guess pulling it would help Masami 
> > > to continue without the need to handle the whole batch.
> > > 
> > > Please pull the tracing/kprobes branch that can be found at:
> > > 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
> > > 	tracing/kprobes
> > 
> > Pulled, thanks Frederic.
> > 
> > >  Documentation/trace/kprobetrace.txt |   27 +-
> > >  kernel/trace/trace_kprobe.c         |  160 +++++---
> > >  tools/perf/Makefile                 |   10 +
> > >  tools/perf/builtin-probe.c          |  386 +++++++++++++++++++
> > >  tools/perf/builtin.h                |    1 +
> > >  tools/perf/perf.c                   |    1 +
> > >  tools/perf/util/probe-finder.c      |  690 +++++++++++++++++++++++++++++++++++
> > >  tools/perf/util/probe-finder.h      |   70 ++++
> > >  8 files changed, 1271 insertions(+), 74 deletions(-)
> > >  create mode 100644 tools/perf/builtin-probe.c
> > >  create mode 100644 tools/perf/util/probe-finder.c
> > >  create mode 100644 tools/perf/util/probe-finder.h
> > 
> > Hm, there's no x86 decoder change here - does this lot fixes the 
> > instruction decoder bug i reported some time ago? That is which keeps 
> > tracing/kprobes out of tip:master.
> > 
> > 	Ingo
> 
> 
> Not yet, I hope it's part of the last family of instruction set we are 
> missing.

ok.

We dont want topics to not be in tip:master for too long - we dont grow 
conflict resolutions nor does it get tested widely.

	Ingo

[PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Frederic Weisbecker]

Hi,

This small set fixes some lock dependency inversions found in reiserfs
xattr and mmap paths.
I guess there are still some of them that I'll have to hunt, especially one
reported by Laurent Riffard and another one introduced by the reiserfs_readdir
path optimization (though I'm not sure about the latter, I have yet to find a
way to reproduce it properly).

As usual, these patches can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git \
	reiserfs/kill-bkl

Thanks,
Frederic.

Frederic Weisbecker (4):
  kill-the-bkl/reiserfs: fix "reiserfs lock" / "inode mutex" lock
    inversion dependency
  kill-the-bkl/reiserfs: fix recursive reiserfs lock in
    reiserfs_mkdir()
  kill-the-bkl/reiserfs: fix recursive reiserfs write lock in
    reiserfs_commit_write()
  kill-the-bkl/reiserfs: panic in case of lock imbalance

 fs/reiserfs/inode.c |   11 ++---------
 fs/reiserfs/lock.c  |    7 +++----
 fs/reiserfs/namei.c |    7 ++++---
 fs/reiserfs/xattr.c |    2 +-
 4 files changed, 10 insertions(+), 17 deletions(-)

Re: [PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Alexander Beregalov]

On Tue, Aug 25, 2009 at 04:32:46AM +0200, Frederic Weisbecker wrote:
> Hi,
> 
> This small set fixes some lock dependency inversions found in reiserfs
> xattr and mmap paths.
> I guess there are still some of them that I'll have to hunt, especially one
> reported by Laurent Riffard and another one introduced by the reiserfs_readdir
> path optimization (though I'm not sure about the latter, I have yet to find a
> way to reproduce it properly).
> 
> As usual, these patches can be found at:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git \
> 	reiserfs/kill-bkl

Hi,
possible circular locking dependency detected

Is it false positive?

REISERFS (device sda1): found reiserfs format "3.6" with standard journal
REISERFS (device sda1): using ordered data mode
REISERFS (device sda1): journal params: device sda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
REISERFS (device sda1): checking transaction log (sda1)
REISERFS debug (device sda1): journal-1153: found in header: first_unflushed_offset 6766, last_flushed_trans_id 1836992
REISERFS debug (device sda1): journal-1206: Starting replay from offset 7889824857987694, trans_id 18
REISERFS debug (device sda1): journal-1299: Setting newest_mount_id to 229
REISERFS (device sda1): Using r5 hash to sort names
VFS: Mounted root (reiserfs filesystem) readonly on device 8:1.

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.31-rc7-00135-g9399a4c #5
-------------------------------------------------------
init.sh/599 is trying to acquire lock:
 (&mm->mmap_sem){++++++}, at: [<c1071056>] might_fault+0x46/0xa0

but task is already holding lock:
 (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c10ff20e>] reiserfs_write_lock+0x1e/0x30

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
       [<c104dd18>] __lock_acquire+0xd28/0x1390
       [<c104e3ef>] lock_acquire+0x6f/0x90
       [<c133af96>] __mutex_lock_common+0x46/0x310
       [<c133b318>] mutex_lock_nested+0x38/0x40
       [<c10ff1ce>] reiserfs_write_lock_once+0x1e/0x40
       [<c10db74c>] reiserfs_get_block+0x5c/0x1440
       [<c10ae7a0>] do_mpage_readpage+0x120/0x4b0
       [<c10aec2f>] mpage_readpages+0x9f/0xe0
       [<c10d8c39>] reiserfs_readpages+0x19/0x20
       [<c10671f5>] __do_page_cache_readahead+0x195/0x210
       [<c1067291>] ra_submit+0x21/0x30
       [<c1061229>] filemap_fault+0x2e9/0x380
       [<c1072f08>] __do_fault+0x38/0x3b0
       [<c1073b5d>] handle_mm_fault+0xcd/0x550
       [<c101ad63>] do_page_fault+0xf3/0x240
       [<c133cd53>] error_code+0x63/0x68
       [<c10baa74>] padzero+0x24/0x40
       [<c10bc062>] load_elf_binary+0x632/0x1480
       [<c108aafa>] search_binary_handler+0x8a/0x270
       [<c108cb15>] do_execve+0x215/0x2a0
       [<c1001658>] sys_execve+0x28/0x60
       [<c1002d99>] syscall_call+0x7/0xb
       [<ffffffff>] 0xffffffff

-> #0 (&mm->mmap_sem){++++++}:
       [<c104dd99>] __lock_acquire+0xda9/0x1390
       [<c104e3ef>] lock_acquire+0x6f/0x90
       [<c1071087>] might_fault+0x77/0xa0
       [<c11c1886>] copy_to_user+0x36/0x130
       [<c1093c29>] filldir64+0xa9/0xf0
       [<c10deda1>] reiserfs_readdir_dentry+0x4a1/0x7b0
       [<c10df0c7>] reiserfs_readdir+0x17/0x20
       [<c1093eb5>] vfs_readdir+0x85/0xa0
       [<c1093f34>] sys_getdents64+0x64/0xb0
       [<c1002d18>] sysenter_do_call+0x12/0x36
       [<ffffffff>] 0xffffffff

other info that might help us debug this:

2 locks held by init.sh/599:
 #0:  (&sb->s_type->i_mutex_key#4){+.+.+.}, at: [<c1093e82>] vfs_readdir+0x52/0xa0
 #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c10ff20e>] reiserfs_write_lock+0x1e/0x30

stack backtrace:
Pid: 599, comm: init.sh Not tainted 2.6.31-rc7-00135-g9399a4c #5
Call Trace:
 [<c133a11a>] ? printk+0x18/0x1e
 [<c104bfdd>] print_circular_bug_tail+0x8d/0xd0
 [<c104dd99>] __lock_acquire+0xda9/0x1390
 [<c104e3ef>] lock_acquire+0x6f/0x90
 [<c1071056>] ? might_fault+0x46/0xa0
 [<c1071087>] might_fault+0x77/0xa0
 [<c1071056>] ? might_fault+0x46/0xa0
 [<c11c1886>] copy_to_user+0x36/0x130
 [<c1093c29>] filldir64+0xa9/0xf0
 [<c10ff20e>] ? reiserfs_write_lock+0x1e/0x30
 [<c10deda1>] reiserfs_readdir_dentry+0x4a1/0x7b0
 [<c1093b80>] ? filldir64+0x0/0xf0
 [<c104d453>] ? __lock_acquire+0x463/0x1390
 [<c104c62e>] ? trace_hardirqs_on_caller+0x7e/0x170
 [<c1093b80>] ? filldir64+0x0/0xf0
 [<c133b153>] ? __mutex_lock_common+0x203/0x310
 [<c1093b80>] ? filldir64+0x0/0xf0
 [<c10df0c7>] reiserfs_readdir+0x17/0x20
 [<c1093eb5>] vfs_readdir+0x85/0xa0
 [<c1093f34>] sys_getdents64+0x64/0xb0
 [<c1002d18>] sysenter_do_call+0x12/0x36

Re: [PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Frederic Weisbecker]

On Thu, Aug 27, 2009 at 12:13:30AM +0400, Alexander Beregalov wrote:
> Hi,
> possible circular locking dependency detected
> 
> Is it false positive?
> 
> REISERFS (device sda1): found reiserfs format "3.6" with standard journal
> REISERFS (device sda1): using ordered data mode
> REISERFS (device sda1): journal params: device sda1, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30
> REISERFS (device sda1): checking transaction log (sda1)
> REISERFS debug (device sda1): journal-1153: found in header: first_unflushed_offset 6766, last_flushed_trans_id 1836992
> REISERFS debug (device sda1): journal-1206: Starting replay from offset 7889824857987694, trans_id 18
> REISERFS debug (device sda1): journal-1299: Setting newest_mount_id to 229
> REISERFS (device sda1): Using r5 hash to sort names
> VFS: Mounted root (reiserfs filesystem) readonly on device 8:1.
> 
> =======================================================
> [ INFO: possible circular locking dependency detected ]
> 2.6.31-rc7-00135-g9399a4c #5
> -------------------------------------------------------
> init.sh/599 is trying to acquire lock:
>  (&mm->mmap_sem){++++++}, at: [<c1071056>] might_fault+0x46/0xa0
> 
> but task is already holding lock:
>  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c10ff20e>] reiserfs_write_lock+0x1e/0x30
> 
> which lock already depends on the new lock.
> 
> 
> the existing dependency chain (in reverse order) is:
> 
> -> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
>        [<c104dd18>] __lock_acquire+0xd28/0x1390
>        [<c104e3ef>] lock_acquire+0x6f/0x90
>        [<c133af96>] __mutex_lock_common+0x46/0x310
>        [<c133b318>] mutex_lock_nested+0x38/0x40
>        [<c10ff1ce>] reiserfs_write_lock_once+0x1e/0x40
>        [<c10db74c>] reiserfs_get_block+0x5c/0x1440
>        [<c10ae7a0>] do_mpage_readpage+0x120/0x4b0
>        [<c10aec2f>] mpage_readpages+0x9f/0xe0
>        [<c10d8c39>] reiserfs_readpages+0x19/0x20
>        [<c10671f5>] __do_page_cache_readahead+0x195/0x210
>        [<c1067291>] ra_submit+0x21/0x30
>        [<c1061229>] filemap_fault+0x2e9/0x380
>        [<c1072f08>] __do_fault+0x38/0x3b0
>        [<c1073b5d>] handle_mm_fault+0xcd/0x550
>        [<c101ad63>] do_page_fault+0xf3/0x240
>        [<c133cd53>] error_code+0x63/0x68
>        [<c10baa74>] padzero+0x24/0x40
>        [<c10bc062>] load_elf_binary+0x632/0x1480
>        [<c108aafa>] search_binary_handler+0x8a/0x270
>        [<c108cb15>] do_execve+0x215/0x2a0
>        [<c1001658>] sys_execve+0x28/0x60
>        [<c1002d99>] syscall_call+0x7/0xb
>        [<ffffffff>] 0xffffffff
> 
> -> #0 (&mm->mmap_sem){++++++}:
>        [<c104dd99>] __lock_acquire+0xda9/0x1390
>        [<c104e3ef>] lock_acquire+0x6f/0x90
>        [<c1071087>] might_fault+0x77/0xa0
>        [<c11c1886>] copy_to_user+0x36/0x130
>        [<c1093c29>] filldir64+0xa9/0xf0
>        [<c10deda1>] reiserfs_readdir_dentry+0x4a1/0x7b0
>        [<c10df0c7>] reiserfs_readdir+0x17/0x20
>        [<c1093eb5>] vfs_readdir+0x85/0xa0
>        [<c1093f34>] sys_getdents64+0x64/0xb0
>        [<c1002d18>] sysenter_do_call+0x12/0x36
>        [<ffffffff>] 0xffffffff
> 
> other info that might help us debug this:
> 
> 2 locks held by init.sh/599:
>  #0:  (&sb->s_type->i_mutex_key#4){+.+.+.}, at: [<c1093e82>] vfs_readdir+0x52/0xa0
>  #1:  (&REISERFS_SB(s)->lock){+.+.+.}, at: [<c10ff20e>] reiserfs_write_lock+0x1e/0x30
> 
> stack backtrace:
> Pid: 599, comm: init.sh Not tainted 2.6.31-rc7-00135-g9399a4c #5
> Call Trace:
>  [<c133a11a>] ? printk+0x18/0x1e
>  [<c104bfdd>] print_circular_bug_tail+0x8d/0xd0
>  [<c104dd99>] __lock_acquire+0xda9/0x1390
>  [<c104e3ef>] lock_acquire+0x6f/0x90
>  [<c1071056>] ? might_fault+0x46/0xa0
>  [<c1071087>] might_fault+0x77/0xa0
>  [<c1071056>] ? might_fault+0x46/0xa0
>  [<c11c1886>] copy_to_user+0x36/0x130
>  [<c1093c29>] filldir64+0xa9/0xf0
>  [<c10ff20e>] ? reiserfs_write_lock+0x1e/0x30
>  [<c10deda1>] reiserfs_readdir_dentry+0x4a1/0x7b0
>  [<c1093b80>] ? filldir64+0x0/0xf0
>  [<c104d453>] ? __lock_acquire+0x463/0x1390
>  [<c104c62e>] ? trace_hardirqs_on_caller+0x7e/0x170
>  [<c1093b80>] ? filldir64+0x0/0xf0
>  [<c133b153>] ? __mutex_lock_common+0x203/0x310
>  [<c1093b80>] ? filldir64+0x0/0xf0
>  [<c10df0c7>] reiserfs_readdir+0x17/0x20
>  [<c1093eb5>] vfs_readdir+0x85/0xa0
>  [<c1093f34>] sys_getdents64+0x64/0xb0
>  [<c1002d18>] sysenter_do_call+0x12/0x36



Hi Alexander,

It should be fixed now, still in the following tree:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
	reiserfs/kill-bkl


Don't hesistate to tell me if you see other problems.

Thanks a lot for your report!

Frederic.

Re: [PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Alexander Beregalov]

> Hi Alexander,
>
> It should be fixed now, still in the following tree:

Hi!
Another one, similar:
It is v2.6.31-3123-g99bc470 plus your 805031859(kill-the-bkl/reiserfs:
panic in case of lock imbalance), UP.

=======================================================
[ INFO: possible circular locking dependency detected ]
2.6.31-03149-gdcc030a #1
-------------------------------------------------------
udevadm/716 is trying to acquire lock:
 (&mm->mmap_sem){++++++}, at: [<c107249a>] might_fault+0x4a/0xa0

but task is already holding lock:
 (sysfs_mutex){+.+.+.}, at: [<c10cb9aa>] sysfs_readdir+0x5a/0x200

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #3 (sysfs_mutex){+.+.+.}:
       [<c104e08e>] __lock_acquire+0xd0e/0x15c0
       [<c104e9ba>] lock_acquire+0x7a/0xa0
       [<c13402c6>] __mutex_lock_common+0x46/0x310
       [<c134066a>] mutex_lock_nested+0x3a/0x50
       [<c10cbcdc>] sysfs_addrm_start+0x2c/0xa0
       [<c10cca50>] create_dir+0x40/0x90
       [<c10ccacb>] sysfs_create_dir+0x2b/0x40
       [<c11bdd6b>] kobject_add_internal+0x9b/0x250
       [<c11be01d>] kobject_add_varg+0x2d/0x50
       [<c11be09c>] kobject_add+0x2c/0x60
       [<c1230984>] device_add+0xf4/0x4a0
       [<c10c98df>] add_partition+0x13f/0x230
       [<c10c9fdb>] rescan_partitions+0x24b/0x3c0
       [<c10aeb80>] __blkdev_get+0x140/0x320
       [<c10aed6a>] blkdev_get+0xa/0x10
       [<c10c9787>] register_disk+0x127/0x140
       [<c11b7ff0>] add_disk+0x80/0x140
       [<c12473a8>] sd_probe_async+0xf8/0x1b0
       [<c1042371>] async_thread+0xd1/0x220
       [<c103c6cc>] kthread+0x6c/0x80
       [<c100355f>] kernel_thread_helper+0x7/0x18

-> #2 (&bdev->bd_mutex){+.+.+.}:
       [<c104e08e>] __lock_acquire+0xd0e/0x15c0
       [<c104e9ba>] lock_acquire+0x7a/0xa0
       [<c13402c6>] __mutex_lock_common+0x46/0x310
       [<c134066a>] mutex_lock_nested+0x3a/0x50
       [<c10aea6a>] __blkdev_get+0x2a/0x320
       [<c10aed6a>] blkdev_get+0xa/0x10
       [<c10aeed1>] open_by_devnum+0x21/0x50
       [<c10fb577>] journal_init+0x1d7/0xa10
       [<c10e8463>] reiserfs_fill_super+0x313/0xdb0
       [<c108a898>] get_sb_bdev+0x108/0x150
       [<c10e6381>] get_super_block+0x21/0x30
       [<c1089910>] vfs_kern_mount+0x40/0xa0
       [<c10899c9>] do_kern_mount+0x39/0xd0
       [<c109f319>] do_mount+0x309/0x700
       [<c109f776>] sys_mount+0x66/0xa0
       [<c15fe886>] mount_block_root+0xc4/0x245
       [<c15fea60>] mount_root+0x59/0x5f
       [<c15feb77>] prepare_namespace+0x111/0x14b
       [<c15fe23d>] kernel_init+0xca/0xd6
       [<c100355f>] kernel_thread_helper+0x7/0x18

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
       [<c104e08e>] __lock_acquire+0xd0e/0x15c0
       [<c104e9ba>] lock_acquire+0x7a/0xa0
       [<c13402c6>] __mutex_lock_common+0x46/0x310
       [<c134066a>] mutex_lock_nested+0x3a/0x50
       [<c11012d8>] reiserfs_write_lock_once+0x28/0x50
       [<c10dd7cc>] reiserfs_get_block+0x5c/0x1440
       [<c10b0660>] do_mpage_readpage+0x120/0x4b0
       [<c10b0aef>] mpage_readpages+0x9f/0xe0
       [<c10dacb9>] reiserfs_readpages+0x19/0x20
       [<c1067dc5>] __do_page_cache_readahead+0x195/0x210
       [<c1067e61>] ra_submit+0x21/0x30
       [<c1062119>] filemap_fault+0x2e9/0x380
       [<c1074388>] __do_fault+0x38/0x3b0
       [<c1074fdd>] handle_mm_fault+0xcd/0x550
       [<c101af55>] do_page_fault+0xf5/0x240
       [<c13420d3>] error_code+0x63/0x68
       [<c10bcae4>] padzero+0x24/0x40
       [<c10bde9a>] load_elf_binary+0x61a/0x1450
       [<c108c0e0>] search_binary_handler+0x90/0x270
       [<c108e152>] do_execve+0x1d2/0x240
       [<c1001658>] sys_execve+0x28/0x60
       [<c1002d59>] syscall_call+0x7/0xb

-> #0 (&mm->mmap_sem){++++++}:
       [<c104e4a5>] __lock_acquire+0x1125/0x15c0
       [<c104e9ba>] lock_acquire+0x7a/0xa0
       [<c10724cb>] might_fault+0x7b/0xa0
       [<c11c4c86>] copy_to_user+0x36/0x130
       [<c10951f9>] filldir64+0xa9/0xf0
       [<c10cba4e>] sysfs_readdir+0xfe/0x200
       [<c1095485>] vfs_readdir+0x85/0xa0
       [<c1095504>] sys_getdents64+0x64/0xb0
       [<c1002cd8>] sysenter_do_call+0x12/0x36

other info that might help us debug this:

2 locks held by udevadm/716:
 #0:  (&type->i_mutex_dir_key){+.+.+.}, at: [<c1095452>] vfs_readdir+0x52/0xa0
 #1:  (sysfs_mutex){+.+.+.}, at: [<c10cb9aa>] sysfs_readdir+0x5a/0x200

stack backtrace:
Pid: 716, comm: udevadm Tainted: G        W  2.6.31-03149-gdcc030a #1
Call Trace:
 [<c133f41a>] ? printk+0x18/0x1e
 [<c104c110>] print_circular_bug+0xc0/0xd0
 [<c104e4a5>] __lock_acquire+0x1125/0x15c0
 [<c104e9ba>] lock_acquire+0x7a/0xa0
 [<c107249a>] ? might_fault+0x4a/0xa0
 [<c10724cb>] might_fault+0x7b/0xa0
 [<c107249a>] ? might_fault+0x4a/0xa0
 [<c11c4c86>] copy_to_user+0x36/0x130
 [<c10951f9>] filldir64+0xa9/0xf0
 [<c1095150>] ? filldir64+0x0/0xf0
 [<c10cba4e>] sysfs_readdir+0xfe/0x200
 [<c1095150>] ? filldir64+0x0/0xf0
 [<c1095150>] ? filldir64+0x0/0xf0
 [<c1095485>] vfs_readdir+0x85/0xa0
 [<c1095504>] sys_getdents64+0x64/0xb0
 [<c1002cd8>] sysenter_do_call+0x12/0x36

Re: [PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Frederic Weisbecker]

On Tue, Sep 15, 2009 at 01:33:42AM +0400, Alexander Beregalov wrote:
> > Hi Alexander,
> >
> > It should be fixed now, still in the following tree:
> 
> Hi!
> Another one, similar:
> It is v2.6.31-3123-g99bc470 plus your 805031859(kill-the-bkl/reiserfs:
> panic in case of lock imbalance), UP.



Although I can't reproduce it, I think I see how that can happen.

On mount time, we have the following dependency:

reiserfs_lock -> bdev_mutex -> sysfs_mutex

which happens while calling journal_init_dev() because
we open the device there.

But also in case of mmap on a reiserfs filesystem we
may call reiserfs_readpages(), holding the reiserfs lock
while already holding mm->mmap_sem

The above dependency is then updated:

mmap_sem
     |
     |
     ------- reiserfs_lock -> bdev_mutex -> sysfs_mutex

And later, while doing a readdir() on a sysfs directory,
sysfs calls filldir, which might_fault, and then might grab
mmap_sem. filldir is called there while holding sys_mutex,
creating the new following dependency

sysfs_mutex -> mmap_sem

Hence the inversion.
It seems the deadlock can't ever happen, I even don't see
corner cases where it could happen.

But still, this dependency should disappear.

Could you please tell me if the following patch makes it shut down?

Otherwise, I may need your config. I don't know why, but I suspect
the lock_acquire(mmap_sem) in might_fault doesn't trigger needed the lockdep
check, although I have the appropriate debug config, at least it seems.
But anyway, it should also happen in my box but it doesn't...

Thanks!

The patch:

diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index d23d6d7..59f7a4c 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2801,11 +2801,14 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 		goto free_and_return;
 	}
 
+	reiserfs_write_unlock(sb);
 	if (journal_init_dev(sb, journal, j_dev_name) != 0) {
 		reiserfs_warning(sb, "sh-462",
 				 "unable to initialize jornal device");
+		reiserfs_write_lock(sb);
 		goto free_and_return;
 	}
+	reiserfs_write_lock(sb);
 
 	rs = SB_DISK_SUPER_BLOCK(sb);
 

Re: [PATCH 0/4] kill-the-bkl/reiserfs: fix some lock dependency inversions

[Alexander Beregalov]

2009/9/17 Frederic Weisbecker <fweisbec@gmail.com>:
> On Tue, Sep 15, 2009 at 01:33:42AM +0400, Alexander Beregalov wrote:
>> > Hi Alexander,
>> >
>> > It should be fixed now, still in the following tree:
>>
>> Hi!
>> Another one, similar:
>> It is v2.6.31-3123-g99bc470 plus your 805031859(kill-the-bkl/reiserfs:
>> panic in case of lock imbalance), UP.
>
>
>
> Although I can't reproduce it, I think I see how that can happen.
>
> On mount time, we have the following dependency:
>
> reiserfs_lock -> bdev_mutex -> sysfs_mutex
>
> which happens while calling journal_init_dev() because
> we open the device there.
>
> But also in case of mmap on a reiserfs filesystem we
> may call reiserfs_readpages(), holding the reiserfs lock
> while already holding mm->mmap_sem
>
> The above dependency is then updated:
>
> mmap_sem
>     |
>     |
>     ------- reiserfs_lock -> bdev_mutex -> sysfs_mutex
>
> And later, while doing a readdir() on a sysfs directory,
> sysfs calls filldir, which might_fault, and then might grab
> mmap_sem. filldir is called there while holding sys_mutex,
> creating the new following dependency
>
> sysfs_mutex -> mmap_sem
>
> Hence the inversion.
> It seems the deadlock can't ever happen, I even don't see
> corner cases where it could happen.
>
> But still, this dependency should disappear.
>
> Could you please tell me if the following patch makes it shut down?
>
> Otherwise, I may need your config. I don't know why, but I suspect
> the lock_acquire(mmap_sem) in might_fault doesn't trigger needed the lockdep
> check, although I have the appropriate debug config, at least it seems.
> But anyway, it should also happen in my box but it doesn't...
>
> Thanks!
>
> The patch:
Yes, it is working!

>
> diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
> index d23d6d7..59f7a4c 100644
> --- a/fs/reiserfs/journal.c
> +++ b/fs/reiserfs/journal.c
> @@ -2801,11 +2801,14 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
>                goto free_and_return;
>        }
>
> +       reiserfs_write_unlock(sb);
>        if (journal_init_dev(sb, journal, j_dev_name) != 0) {
>                reiserfs_warning(sb, "sh-462",
>                                 "unable to initialize jornal device");
> +               reiserfs_write_lock(sb);
>                goto free_and_return;
>        }
> +       reiserfs_write_lock(sb);
>
>        rs = SB_DISK_SUPER_BLOCK(sb);
>
>
>

[PATCH] kill-the-bkl/reiserfs: Fix induced mm->mmap_sem to sysfs_mutex dependency

[Frederic Weisbecker]

On Thu, Sep 17, 2009 at 03:37:22AM +0400, Alexander Beregalov wrote:
> 2009/9/17 Frederic Weisbecker <fweisbec@gmail.com>:
> > On Tue, Sep 15, 2009 at 01:33:42AM +0400, Alexander Beregalov wrote:
> >> > Hi Alexander,
> >> >
> >> > It should be fixed now, still in the following tree:
> >>
> >> Hi!
> >> Another one, similar:
> >> It is v2.6.31-3123-g99bc470 plus your 805031859(kill-the-bkl/reiserfs:
> >> panic in case of lock imbalance), UP.
> >
> >
> >
> > Although I can't reproduce it, I think I see how that can happen.
> >
> > On mount time, we have the following dependency:
> >
> > reiserfs_lock -> bdev_mutex -> sysfs_mutex
> >
> > which happens while calling journal_init_dev() because
> > we open the device there.
> >
> > But also in case of mmap on a reiserfs filesystem we
> > may call reiserfs_readpages(), holding the reiserfs lock
> > while already holding mm->mmap_sem
> >
> > The above dependency is then updated:
> >
> > mmap_sem
> >     |
> >     |
> >     ------- reiserfs_lock -> bdev_mutex -> sysfs_mutex
> >
> > And later, while doing a readdir() on a sysfs directory,
> > sysfs calls filldir, which might_fault, and then might grab
> > mmap_sem. filldir is called there while holding sys_mutex,
> > creating the new following dependency
> >
> > sysfs_mutex -> mmap_sem
> >
> > Hence the inversion.
> > It seems the deadlock can't ever happen, I even don't see
> > corner cases where it could happen.
> >
> > But still, this dependency should disappear.
> >
> > Could you please tell me if the following patch makes it shut down?
> >
> > Otherwise, I may need your config. I don't know why, but I suspect
> > the lock_acquire(mmap_sem) in might_fault doesn't trigger needed the lockdep
> > check, although I have the appropriate debug config, at least it seems.
> > But anyway, it should also happen in my box but it doesn't...
> >
> > Thanks!
> >
> > The patch:
> Yes, it is working!


Great, I've pushed the fix then, see the following patch.

Thanks a lot Alexander!

---
From: Frederic Weisbecker <fweisbec@gmail.com>
Date: Thu, 17 Sep 2009 05:31:37 +0200
Subject: [PATCH] kill-the-bkl/reiserfs: Fix induced mm->mmap_sem to sysfs_mutex dependency

Alexander Beregalov reported the following warning:

	=======================================================
	[ INFO: possible circular locking dependency detected ]
	2.6.31-03149-gdcc030a #1
	-------------------------------------------------------
	udevadm/716 is trying to acquire lock:
	 (&mm->mmap_sem){++++++}, at: [<c107249a>] might_fault+0x4a/0xa0

	but task is already holding lock:
	 (sysfs_mutex){+.+.+.}, at: [<c10cb9aa>] sysfs_readdir+0x5a/0x200

	which lock already depends on the new lock.

	the existing dependency chain (in reverse order) is:

	-> #3 (sysfs_mutex){+.+.+.}:
	       [...]

	-> #2 (&bdev->bd_mutex){+.+.+.}:
	       [...]

	-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
	       [...]

	-> #0 (&mm->mmap_sem){++++++}:
	       [...]

On reiserfs mount path, we take the reiserfs lock and while
initializing the journal, we open the device, taking the
bdev->bd_mutex. Then rescan_partition() may signal the change
to sysfs.

We have then the following dependency:

	reiserfs_lock -> bd_mutex -> sysfs_mutex

Later, while entering reiserfs_readpage() after a pagefault in an
mmaped reiserfs file, we are holding the mm->mmap_sem, and we are going
to take the reiserfs lock too.
We have then the following dependency:

	mm->mmap_sem -> reiserfs_lock

which, expanded with the previous dependency gives us:

	mm->mmap_sem -> reiserfs_lock -> bd_mutex -> sysfs_mutex

Now while entering the sysfs readdir path, we are holding the
sysfs_mutex. And when we copy a directory entry to the user buffer, we
might fault and then take the mm->mmap_sem lock. Which leads to the
circular locking dependency reported.

We can fix that by relaxing the reiserfs lock during the call to
journal_init_dev(), which is the place where we open the mounted
device.

This is fine to relax the lock here because we are in the begining of
the reiserfs mount path and there is nothing to protect at this time,
the journal is not intialized.
We just keep this lock around for paranoid reasons.

Reported-by: Alexander Beregalov <a.beregalov@gmail.com>
Tested-by: Alexander Beregalov <a.beregalov@gmail.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Alexander Beregalov <a.beregalov@gmail.com>
Cc: Laurent Riffard <laurent.riffard@free.fr>
---
 fs/reiserfs/journal.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index d23d6d7..04e3c42 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2801,11 +2801,27 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 		goto free_and_return;
 	}
 
+	/*
+	 * We need to unlock here to avoid creating the following
+	 * dependency:
+	 * reiserfs_lock -> sysfs_mutex
+	 * Because the reiserfs mmap path creates the following dependency:
+	 * mm->mmap -> reiserfs_lock, hence we have
+	 * mm->mmap -> reiserfs_lock ->sysfs_mutex
+	 * This would ends up in a circular dependency with sysfs readdir path
+	 * which does sysfs_mutex -> mm->mmap_sem
+	 * This is fine because the reiserfs lock is useless in mount path,
+	 * at least until we call journal_begin. We keep it for paranoid
+	 * reasons.
+	 */
+	reiserfs_write_unlock(sb);
 	if (journal_init_dev(sb, journal, j_dev_name) != 0) {
+		reiserfs_write_lock(sb);
 		reiserfs_warning(sb, "sh-462",
 				 "unable to initialize jornal device");
 		goto free_and_return;
 	}
+	reiserfs_write_lock(sb);
 
 	rs = SB_DISK_SUPER_BLOCK(sb);
 
-- 
1.6.2.3

Re: [PATCH] kill-the-bkl/reiserfs: Fix induced mm->mmap_sem to sysfs_mutex dependency

[Alexander Beregalov]

Hi Frederic.

Another very similar warning.
(smp 2*2core)
v2.6.31-7068-g43c1266 plus 193be0ee1 kill-the-bkl/reiserfs: Fix
induced mm->mmap_sem to sysfs_mutex dependency


[ INFO: possible circular locking dependency detected ]
2.6.31-07095-g25a3912 #4
-------------------------------------------------------
udevadm/790 is trying to acquire lock:
 (&mm->mmap_sem){++++++}, at: [<c1098942>] might_fault+0x72/0xc0

but task is already holding lock:
 (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260

which lock already depends on the new lock.


the existing dependency chain (in reverse order) is:

-> #5 (sysfs_mutex){+.+.+.}:
       [<c1069810>] __lock_acquire+0xc90/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c13ff805>] __mutex_lock_common+0x65/0x440
       [<c13ffce0>] mutex_lock_nested+0x40/0x60
       [<c1108535>] sysfs_addrm_start+0x35/0xc0
       [<c11094d1>] create_dir+0x51/0xb0
       [<c1109566>] sysfs_create_dir+0x36/0x60
       [<c12356d7>] kobject_add_internal+0xa7/0x270
       [<c12359da>] kobject_add_varg+0x3a/0x70
       [<c1235a4e>] kobject_init_and_add+0x3e/0x60
       [<c10af62d>] sysfs_slab_add+0x7d/0x1e0
       [<c10af7ef>] sysfs_add_func+0x5f/0xa0
       [<c104d16e>] worker_thread+0x16e/0x270
       [<c1051fac>] kthread+0x7c/0x90
       [<c1003f5b>] kernel_thread_helper+0x7/0x7c

-> #4 (slub_lock){+++++.}:
       [<c1069810>] __lock_acquire+0xc90/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c14000d2>] down_read+0x52/0xb0
       [<c15e2497>] slab_cpuup_callback+0x4a/0x196
       [<c140481d>] notifier_call_chain+0x4d/0x90
       [<c1058095>] __raw_notifier_call_chain+0x25/0x40
       [<c15e1c63>] cpu_up+0xdd/0x1a6
       [<c15b1403>] kernel_init+0xac/0x1a8
       [<c1003f5b>] kernel_thread_helper+0x7/0x7c

-> #3 (cpu_hotplug.lock){+.+.+.}:
       [<c1069810>] __lock_acquire+0xc90/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c13ff805>] __mutex_lock_common+0x65/0x440
       [<c13ffce0>] mutex_lock_nested+0x40/0x60
       [<c15e1c24>] cpu_up+0x9e/0x1a6
       [<c15b1403>] kernel_init+0xac/0x1a8
       [<c1003f5b>] kernel_thread_helper+0x7/0x7c

-> #2 (cpu_add_remove_lock){+.+.+.}:
       [<c1069810>] __lock_acquire+0xc90/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c13ff805>] __mutex_lock_common+0x65/0x440
       [<c13ffce0>] mutex_lock_nested+0x40/0x60
       [<c103ad9d>] cpu_maps_update_begin+0x1d/0x40
       [<c104d9cc>] __create_workqueue_key+0x9c/0x210
       [<c113be94>] journal_init+0x9a4/0xa60
       [<c112735f>] reiserfs_fill_super+0x35f/0xdf0
       [<c10b8618>] get_sb_bdev+0x138/0x180
       [<c1124f0d>] get_super_block+0x2d/0x50
       [<c10b72f1>] vfs_kern_mount+0x51/0xc0
       [<c10b73ed>] do_kern_mount+0x4d/0x100
       [<c10d115a>] do_mount+0x21a/0x720
       [<c10d16ec>] sys_mount+0x8c/0xe0
       [<c15b1d19>] mount_block_root+0xcf/0x26b
       [<c15b1f19>] mount_root+0x64/0x7b
       [<c15b204c>] prepare_namespace+0x11c/0x167
       [<c15b14df>] kernel_init+0x188/0x1a8
       [<c1003f5b>] kernel_thread_helper+0x7/0x7c

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
       [<c1069810>] __lock_acquire+0xc90/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c13ff805>] __mutex_lock_common+0x65/0x440
       [<c13ffce0>] mutex_lock_nested+0x40/0x60
       [<c1142282>] reiserfs_write_lock_once+0x32/0x70
       [<c111c1fb>] reiserfs_get_block+0x6b/0x1510
       [<c10e5e48>] do_mpage_readpage+0x168/0x510
       [<c10e631f>] mpage_readpages+0xaf/0x100
       [<c1118ef5>] reiserfs_readpages+0x25/0x40
       [<c108bc2d>] __do_page_cache_readahead+0x1fd/0x2a0
       [<c108bcfd>] ra_submit+0x2d/0x50
       [<c1083bf6>] filemap_fault+0x436/0x470
       [<c109af24>] __do_fault+0x54/0x410
       [<c109bdd1>] handle_mm_fault+0x1c1/0x680
       [<c14043e5>] do_page_fault+0x115/0x380
       [<c1401ff0>] error_code+0x78/0x80
       [<c10f45a7>] padzero+0x37/0x50
       [<c10f4d4d>] load_elf_binary+0x63d/0x1500
       [<c10bc461>] search_binary_handler+0x1a1/0x330
       [<c10bcb48>] do_execve+0x1f8/0x270
       [<c1001783>] sys_execve+0x33/0x80
       [<c10033dc>] syscall_call+0x7/0xb

-> #0 (&mm->mmap_sem){++++++}:
       [<c1069dc8>] __lock_acquire+0x1248/0x1290
       [<c1069eaa>] lock_acquire+0x9a/0xd0
       [<c1098973>] might_fault+0xa3/0xc0
       [<c123d611>] copy_to_user+0x41/0x130
       [<c10c4f9c>] filldir64+0xcc/0x120
       [<c11081ba>] sysfs_readdir+0xfa/0x260
       [<c10c525e>] vfs_readdir+0x9e/0xc0
       [<c10c52fa>] sys_getdents64+0x7a/0xe0
       [<c100334f>] sysenter_do_call+0x12/0x36

other info that might help us debug this:

2 locks held by udevadm/790:
 #0:  (&type->i_mutex_dir_key){+.+.+.}, at: [<c10c522a>] vfs_readdir+0x6a/0xc0
 #1:  (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260

stack backtrace:
Pid: 790, comm: udevadm Not tainted 2.6.31-07095-g25a3912 #4
Call Trace:
 [<c13fdd64>] ? printk+0x23/0x37
 [<c1067738>] print_circular_bug+0xe8/0x100
 [<c1069dc8>] __lock_acquire+0x1248/0x1290
 [<c1069eaa>] lock_acquire+0x9a/0xd0
 [<c1098942>] ? might_fault+0x72/0xc0
 [<c1098942>] ? might_fault+0x72/0xc0
 [<c1098973>] might_fault+0xa3/0xc0
 [<c1098942>] ? might_fault+0x72/0xc0
 [<c123d611>] copy_to_user+0x41/0x130
 [<c10c4f9c>] filldir64+0xcc/0x120
 [<c11081ba>] sysfs_readdir+0xfa/0x260
 [<c10c4ed0>] ? filldir64+0x0/0x120
 [<c10c525e>] vfs_readdir+0x9e/0xc0
 [<c10c4ed0>] ? filldir64+0x0/0x120
 [<c10c52fa>] sys_getdents64+0x7a/0xe0
 [<c100334f>] sysenter_do_call+0x12/0x36

Re: [PATCH] kill-the-bkl/reiserfs: Fix induced mm->mmap_sem to sysfs_mutex dependency

[Frederic Weisbecker]

On Tue, Sep 22, 2009 at 05:55:43PM +0400, Alexander Beregalov wrote:
> Hi Frederic.
> 
> Another very similar warning.
> (smp 2*2core)
> v2.6.31-7068-g43c1266 plus 193be0ee1 kill-the-bkl/reiserfs: Fix
> induced mm->mmap_sem to sysfs_mutex dependency
> 
> 
> [ INFO: possible circular locking dependency detected ]
> 2.6.31-07095-g25a3912 #4
> -------------------------------------------------------
> udevadm/790 is trying to acquire lock:
>  (&mm->mmap_sem){++++++}, at: [<c1098942>] might_fault+0x72/0xc0
> 
> but task is already holding lock:
>  (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260
> 
> which lock already depends on the new lock.



Yeah indeed, it's about the same kind of thing.
Could you please test the following patch?

Thanks!

diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index 04e3c42..2f8a7e7 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2933,8 +2933,11 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 	}
 
 	reiserfs_mounted_fs_count++;
-	if (reiserfs_mounted_fs_count <= 1)
+	if (reiserfs_mounted_fs_count <= 1) {
+		reiserfs_write_unlock(sb);
 		commit_wq = create_workqueue("reiserfs");
+		reiserfs_write_lock(sb);
+	}
 
 	INIT_DELAYED_WORK(&journal->j_work, flush_async_commits);
 	journal->j_work_sb = sb;

Re: [PATCH] kill-the-bkl/reiserfs: Fix induced mm->mmap_sem to sysfs_mutex dependency

[Alexander Beregalov]

2009/9/29 Frederic Weisbecker <fweisbec@gmail.com>:
> On Tue, Sep 22, 2009 at 05:55:43PM +0400, Alexander Beregalov wrote:
>> Hi Frederic.
>>
>> Another very similar warning.
>> (smp 2*2core)
>> v2.6.31-7068-g43c1266 plus 193be0ee1 kill-the-bkl/reiserfs: Fix
>> induced mm->mmap_sem to sysfs_mutex dependency
>>
>>
>> [ INFO: possible circular locking dependency detected ]
>> 2.6.31-07095-g25a3912 #4
>> -------------------------------------------------------
>> udevadm/790 is trying to acquire lock:
>>  (&mm->mmap_sem){++++++}, at: [<c1098942>] might_fault+0x72/0xc0
>>
>> but task is already holding lock:
>>  (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260
>>
>> which lock already depends on the new lock.
>
>
>
> Yeah indeed, it's about the same kind of thing.
> Could you please test the following patch?

Thanks, the warning has gone away.

[PATCH] kill-the-bkl/reiserfs: fix reiserfs lock to cpu_add_remove_lock dependency

[Frederic Weisbecker]

On Tue, Sep 29, 2009 at 02:22:42PM +0400, Alexander Beregalov wrote:
> 2009/9/29 Frederic Weisbecker <fweisbec@gmail.com>:
> > Yeah indeed, it's about the same kind of thing.
> > Could you please test the following patch?
> 
> Thanks, the warning has gone away.


Thanks a lot Alexander, your tests and reports are very precious!

I've pushed the commit below, as usual it can be found at:

git://git.kernel.org/pub/scm/linux/kernel/git/frederic/random-tracing.git
	reiserfs/kill-bkl

---
From: Frederic Weisbecker <fweisbec@gmail.com>
Date: Mon, 5 Oct 2009 16:31:37 +0200
Subject: [PATCH] kill-the-bkl/reiserfs: fix reiserfs lock to cpu_add_remove_lock dependency

While creating the reiserfs workqueue during the journal
initialization, we are holding the reiserfs lock, but
create_workqueue() also holds the cpu_add_remove_lock, creating
then the following dependency:

- reiserfs lock -> cpu_add_remove_lock

But we also have the following existing dependencies:

- mm->mmap_sem -> reiserfs lock
- cpu_add_remove_lock -> cpu_hotplug.lock -> slub_lock -> sysfs_mutex

The merged dependency chain then becomes:

- mm->mmap_sem -> reiserfs lock -> cpu_add_remove_lock ->
	cpu_hotplug.lock -> slub_lock -> sysfs_mutex

But when we fill a dir entry in sysfs_readir(), we are holding the
sysfs_mutex and we also might fault while copying the directory entry
to the user, leading to the following dependency:

- sysfs_mutex -> mm->mmap_sem

The end result is then a lock inversion between sysfs_mutex and
mm->mmap_sem, as reported in the following lockdep warning:

[ INFO: possible circular locking dependency detected ]
2.6.31-07095-g25a3912 #4
-------------------------------------------------------
udevadm/790 is trying to acquire lock:
 (&mm->mmap_sem){++++++}, at: [<c1098942>] might_fault+0x72/0xc0

but task is already holding lock:
 (sysfs_mutex){+.+.+.}, at: [<c110813c>] sysfs_readdir+0x7c/0x260

which lock already depends on the new lock.

the existing dependency chain (in reverse order) is:

-> #5 (sysfs_mutex){+.+.+.}:
      [...]

-> #4 (slub_lock){+++++.}:
      [...]

-> #3 (cpu_hotplug.lock){+.+.+.}:
      [...]

-> #2 (cpu_add_remove_lock){+.+.+.}:
      [...]

-> #1 (&REISERFS_SB(s)->lock){+.+.+.}:
      [...]

-> #0 (&mm->mmap_sem){++++++}:
      [...]

This can be fixed by relaxing the reiserfs lock while creating the
workqueue.
This is fine to relax the lock here, we just keep it around to pass
through reiserfs lock checks and for paranoid reasons.

Reported-by: Alexander Beregalov <a.beregalov@gmail.com>
Tested-by: Alexander Beregalov <a.beregalov@gmail.com>
Signed-off-by: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jeff Mahoney <jeffm@suse.com>
Cc: Chris Mason <chris.mason@oracle.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Alexander Beregalov <a.beregalov@gmail.com>
Cc: Laurent Riffard <laurent.riffard@free.fr>
---
 fs/reiserfs/journal.c |    5 ++++-
 1 files changed, 4 insertions(+), 1 deletions(-)

diff --git a/fs/reiserfs/journal.c b/fs/reiserfs/journal.c
index 04e3c42..2f8a7e7 100644
--- a/fs/reiserfs/journal.c
+++ b/fs/reiserfs/journal.c
@@ -2933,8 +2933,11 @@ int journal_init(struct super_block *sb, const char *j_dev_name,
 	}
 
 	reiserfs_mounted_fs_count++;
-	if (reiserfs_mounted_fs_count <= 1)
+	if (reiserfs_mounted_fs_count <= 1) {
+		reiserfs_write_unlock(sb);
 		commit_wq = create_workqueue("reiserfs");
+		reiserfs_write_lock(sb);
+	}
 
 	INIT_DELAYED_WORK(&journal->j_work, flush_async_commits);
 	journal->j_work_sb = sb;
-- 
1.6.2.3