[git pull] ftrace for v2.6.27

[git pull] ftrace for v2.6.27

[Ingo Molnar]

Linus,

Please pull the ftrace git tree from:

   git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip.git tracing/for-linus

[ NOTE: please pull this only after the x86/for-linus and
  sched/for-linus pulls, as it's a manual merge ontop of those branches. ]

This tree adds the ftrace framework and a couple of tracing plugins: 
ftrace, latency tracer, x86/mmiotrace, irqs-off tracer, sysprof, 
scheduler tracer.

Thanks,

	Ingo

------------------>
Abhishek Sagar (15):
      ftrace: safe traversal of ftrace_hash hlist
      ftrace: fix updating of ftrace_update_cnt
      ftrace: core support for ARM
      ftrace: distinguish kretprobe'd functions in trace logs
      ftrace: export kretprobe_trampoline for function tracer
      ftrace: prevent freeing of all failed updates
      ftrace: remove ftrace_ip_converted()
      ftrace: add debugfs entry 'failures'
      ftrace: prevent freeing of all failed updates
      ftrace: fix "notrace" filtering priority
      ftrace: store mcount address in rec->ip
      kprobes: enable clean usage of get_kprobe
      ftrace: freeze kprobe'd records
      ftrace: avoid modifying kprobe'd records
      ftrace: prevent ftrace modifications while being kprobe'd, v2

Ankita Garg (2):
      ftrace: fix conversion of task state to char in latency tracer
      ftrace: disable tracing when current_tracer is set to "none"

Arnaldo Carvalho de Melo (2):
      ftrace: annotate core code that should not be traced
      ftrace: add basic support for gcc profiler instrumentation

David Miller (2):
      ftrace: remove packed attribute on ftrace_page.
      sparc64: add ftrace support.

Ingo Molnar (70):
      sched: add latency tracer callbacks to the scheduler
      tracing: add notrace to linkage.h
      ftrace: fix kexec
      ftrace: cleanups
      ftrace: add README
      ftrace: fix time offset
      ftrace: fix locking
      ftrace: cleanups
      ftrace: timestamp syncing, prepare
      ftrace: fast, scalable, synchronized timestamps
      ftrace: remove-idx-sync
      ftrace: clean-up-pipe-iteration
      ftrace: add raw output
      ftrace: bin-output
      ftrace: add trace_special()
      ftrace, locking fix
      ftrace: use cpu clock again
      ftrace: build fix
      ftrace: introduce the "hex" output method
      ftrace: build fix
      ftrace: build fix
      ftrace: cleanups
      ftrace: disable -pg for the tracer itself
      ftrace: remove notrace
      ftrace: add wakeup events to sched tracer
      ftrace: add stack tracing
      ftrace: sched tracer fix
      ftrace: make nostacktrace the default
      ftrace: sched tracer, trace full rbtree
      ftrace: trace curr/next tasks
      ftrace: fix wakeups
      ftrace: fix __trace_special()
      ftrace: trace scheduler rbtree
      ftrace: add tracing_cpumask
      ftrace: make use of tracing_cpumask
      ftrace: fix cmdline tracing
      ftrace: iter ctrl fix
      ftrace: include cpu in stacktrace
      ftrace: sched tree fix
      ftrace: sched special
      ftrace: make it more available in the Kconfig
      ftrace: remove add-hoc code
      ftrace: restrict tracing to HAVE_FTRACE architectures
      ftrace: cleanups
      ftrace: trace faster
      ftrace: special stacktrace
      ftrace: stacktrace fix
      ftrace: fix mcount export bug
      ftrace: cleanups
      ftrace: fix include file dependency
      ftrace: fix build bug
      ftrace: add sysprof plugin
      ftrace: extend sysprof plugin
      ftrace: extend sysprof plugin some more
      ftrace: sysprof-plugin, add self-tests
      ftrace: sysprof plugin improvement
      ftrace: make sysprof dependent on x86 for now
      ftrace: sysprof fix
      ftrace: sysprof updates
      sysprof: update copyrights
      mmiotrace: ftrace fix
      mmiotrace: cleanup
      - fix mmioftrace + rcu merge interaction
      sched: do not trace sched_clock
      ftrace: fix merge
      sched: sched_clock() lockdep fix
      namespacecheck: fixes
      ftrace: build fix with gcc 4.3
      fix "ftrace: store mcount address in rec->ip"
      ftrace: build fix for ftraced_suspend

Jiri Slaby (1):
      ftrace: fix printout

Mathieu Desnoyers (3):
      Markers - remove extra format argument
      Markers - define non optimized marker
      Port ftrace to markers

Pekka Paalanen (30):
      x86: add a list for custom page fault handlers.
      x86_64: fix kernel rodata NX setting
      x86: fix SMP alternatives: use mutex instead of spinlock, text_poke is sleepable
      ftrace: add readpos to struct trace_seq; add trace_seq_to_user()
      x86: mmiotrace - trace memory mapped IO
      x86 mmiotrace: use lookup_address()
      x86 mmiotrace: fix relay-buffer-full flag for SMP
      x86 mmiotrace: comment about user space ABI
      x86: explicit call to mmiotrace in do_page_fault()
      x86 mmiotrace: Use percpu instead of arrays.
      x86: mmiotrace full patch, preview 1
      x86: mmiotrace, preview 2
      ftrace: add mmiotrace plugin
      ftrace: mmiotrace, updates
      ftrace: mmiotrace update, #2
      x86 mmiotrace: move files into arch/x86/mm/.
      x86 mmiotrace: remove ISA_trace parameter.
      x86 mmiotrace: Do not print bogus pid
      mmiotrace: add user documentation
      x86 mmiotrace: dynamically disable non-boot CPUs
      mmiotrace: print header using the read hook.
      mmiotrace: print overrun counts
      mmiotrace: do not print bogus pid for maps either
      doc: update mmiotrace doc to current status
      mmiotrace: code style cleanups
      x86 mmiotrace: fix page-unaligned ioremaps
      x86 mmiotrace: use resource_size_t for phys addresses
      mmiotrace: rename kmmio_probe::user_data to :private.
      x86 mmiotrace: page level is unsigned
      mmiotrace broken in linux-next (8-bit writes only)

Peter Zijlstra (2):
      ftrace: trace next state
      ftrace: fix wakeup callback

Randy Dunlap (2):
      mmiotrace: fix printk format
      x86/mmiotrace: uses/depends on PCI

Soeren Sandmann (1):
      ftrace/sysprof: don't trace the user stack if we are a kernel thread.

Soeren Sandmann Pedersen (2):
      ftrace: allow the event pipe to be polled
      sysprof: kernel trace

Steven Rostedt (90):
      ftrace: make the task state char-string visible to all
      ftrace: add preempt_enable/disable notrace macros
      x86: add notrace annotations to vsyscall.
      ftrace: latency tracer infrastructure
      ftrace: function tracer
      ftrace: add tracing of context switches
      ftrace: tracer for scheduler wakeup latency
      ftrace: trace irq disabled critical timings
      ftrace: trace preempt off critical timings
      ftrace: dynamic enabling/disabling of function calls
      ftrace: add ftrace_enabled sysctl to disable mcount function
      ftrace: use nops instead of jmp
      ftrace: move memory management out of arch code
      ftrace: use dynamic patching for updating mcount calls
      ftrace: add filter select functions to trace
      ftrace: convert single large buffer into single pages.
      ftrace: debug smp_processor_id, use notrace preempt disable
      ftrace: irqs off smp_processor_id() fix
      ftrace: lockdep notrace annotations
      ftrace: don't use raw_local_irq_save/restore
      ftrace: fix updates to max trace
      ftrace: fix max latency
      ftrace: force recording
      ftrace: add self-tests
      ftrace: startup tester on dynamic tracing.
      ftrace: disable all tracers on corrupted buffer
      ftrace: reset selftests
      ftrace: change buffers to producer consumer
      ftrace: add a buffer for output
      ftrace: user run time file reading
      ftrace: pipe fixes
      ftrace - fix dynamic ftrace memory leak
      ftrace: disable tracing on failure
      ftrace: enabled tracing by default
      ftrace: add trace_function api for other tracers to use
      ftrace: remove address of function names
      ftrace: do not profile lib/string.o
      ftrace: remove wakeup from function trace
      ftrace: printk and trace irqsoff and wakeups
      ftrace: add TRACE_STACK and TRACE_SPECIAL to selftest validation
      ftrace: fix dynamic ftrace selftest
      ftrace: irqsoff use raw_smp_processor_id
      ftrace: user raw_spin_lock in tracing
      ftrace: remove function tracing from spinlock debug
      ftrace: use Makefile to remove tracing from lockdep
      ftrace: add UNINTERRUPTIBLE state for kftraced on disable
      ftrace: fix mutex unlock in trace output
      ftrace: selftest protect againt max flip
      ftrace: fix the fault label in updating code
      ftrace: dont write protect kernel text
      ftrace: allow trace_pipe to block on all reads
      ftrace: restore iterator trace in pipe read
      ftrace: return EOF in trace_pipe on change of tracer
      ftrace: trace_pipe implement NONBLOCK
      ftrace: user proper API for setting RT prios in selftest
      ftrace: trace_entries to dynamically change trace buffer size
      lockdep: update lockdep_recursion on graph_lock
      ftrace: simple clean ups
      ftrace: replace simple_strtoul with strict_strtoul
      ftrace: modulize the number of CPU buffers
      ftrace: comment code
      ftrace: fix comm on function trace output
      ftrace: add logic to record overruns
      ftrace: add trace pipe header pluggin
      ftrace: fix setting of pos in read_pipe
      ftrace: use raw_smp_processor_id for mcount functions
      ftrace: limit trace entries
      ftrace, POWERPC: add irqs_disabled_flags to ppc
      ftrace: support for PowerPC
      kbuild: create new CFLAGS_REMOVE_(basename).o option
      ftrace: use the new kbuild CFLAGS_REMOVE for kernel directory
      ftrace: use the new kbuild CFLAGS_REMOVE for lib directory
      ftrace: use the new kbuild CFLAGS_REMOVE for x86/kernel directory
      ftrace: add have dynamic ftrace config for archs
      ftrace: limit use of check pages
      ftrace: move ftrace_special to trace.c
      ftrace: add function tracing to wake up tracing
      ftrace: remove printks from irqsoff trace
      ftrace: set_ftrace_notrace feature
      ftrace: fix up cmdline recording
      ftrace: powerpc clean ups
      ftrace: user update and disable dynamic ftrace daemon
      ftrace: move sched_switch enable after markers
      ftrace: define function trace nop
      ftrace: trace schedule
      ftrace: check proper config for preempt type
      ftrace: start wakeup tracing after setting function tracer
      ftrace: use current CPU for function startup
      ftrace: add ftrace_kill_atomic
      ftrace: separate out the function enabled variable

Thomas Gleixner (6):
      ftrace: move enums to ftrace.h and make helper function global
      ftrace: simplify hexprint
      ftrace: remove notrace
      ftrace: fix __trace_special()
      ftrace: make it more available in the Kconfig
      sysprof: make it depend on X86


 Documentation/tracers/mmiotrace.txt      |  164 ++
 Makefile                                 |    4 +
 arch/arm/Kconfig                         |    2 +
 arch/arm/boot/compressed/Makefile        |    6 +
 arch/arm/kernel/Makefile                 |    5 +
 arch/arm/kernel/armksyms.c               |    5 +
 arch/arm/kernel/entry-common.S           |   51 +
 arch/arm/kernel/ftrace.c                 |  116 ++
 arch/arm/kernel/kprobes.c                |    2 +-
 arch/powerpc/Kconfig                     |    4 +-
 arch/powerpc/kernel/Makefile             |   14 +
 arch/powerpc/kernel/entry_32.S           |  127 ++
 arch/powerpc/kernel/entry_64.S           |   65 +
 arch/powerpc/kernel/ftrace.c             |  154 ++
 arch/powerpc/kernel/io.c                 |    3 +-
 arch/powerpc/kernel/irq.c                |    6 +-
 arch/powerpc/kernel/ppc_ksyms.c          |    5 +
 arch/powerpc/kernel/setup_32.c           |    6 +-
 arch/powerpc/platforms/powermac/Makefile |    5 +
 arch/sparc64/Kconfig                     |    2 +
 arch/sparc64/Kconfig.debug               |    2 +-
 arch/sparc64/kernel/Makefile             |    1 +
 arch/sparc64/kernel/ftrace.c             |   94 +
 arch/sparc64/kernel/sparc64_ksyms.c      |    4 +-
 arch/sparc64/lib/mcount.S                |   58 +-
 arch/x86/Kconfig                         |    2 +
 arch/x86/Kconfig.debug                   |   28 +
 arch/x86/kernel/Makefile                 |    8 +
 arch/x86/kernel/alternative.c            |   22 +-
 arch/x86/kernel/entry_32.S               |   72 +
 arch/x86/kernel/entry_64.S               |  106 +
 arch/x86/kernel/ftrace.c                 |  141 ++
 arch/x86/kernel/i386_ksyms_32.c          |    9 +-
 arch/x86/kernel/machine_kexec_32.c       |    4 +
 arch/x86/kernel/machine_kexec_64.c       |    4 +
 arch/x86/kernel/process_32.c             |    3 +
 arch/x86/kernel/process_64.c             |    3 +
 arch/x86/kernel/vsyscall_64.c            |    3 +-
 arch/x86/kernel/x8664_ksyms_64.c         |   11 +-
 arch/x86/lib/Makefile                    |    1 +
 arch/x86/lib/thunk_32.S                  |   47 +
 arch/x86/lib/thunk_64.S                  |   19 +-
 arch/x86/mm/Makefile                     |    5 +
 arch/x86/mm/fault.c                      |   13 +
 arch/x86/mm/init_32.c                    |    4 +
 arch/x86/mm/init_64.c                    |   10 +-
 arch/x86/mm/ioremap.c                    |   11 +-
 arch/x86/mm/kmmio.c                      |  510 +++++
 arch/x86/mm/mmio-mod.c                   |  515 +++++
 arch/x86/mm/pageattr.c                   |    1 +
 arch/x86/mm/pf_in.c                      |  489 +++++
 arch/x86/mm/pf_in.h                      |   39 +
 arch/x86/mm/testmmiotrace.c              |   71 +
 arch/x86/vdso/vclock_gettime.c           |   15 +-
 arch/x86/vdso/vgetcpu.c                  |    3 +-
 include/asm-arm/ftrace.h                 |   14 +
 include/asm-arm/kprobes.h                |    1 +
 include/asm-powerpc/ftrace.h             |   14 +
 include/asm-powerpc/hw_irq.h             |   10 +
 include/asm-sparc64/ftrace.h             |   14 +
 include/asm-x86/alternative.h            |    2 +
 include/asm-x86/ftrace.h                 |   14 +
 include/asm-x86/irqflags.h               |   24 +-
 include/asm-x86/vsyscall.h               |    3 +-
 include/linux/ftrace.h                   |  144 ++
 include/linux/irqflags.h                 |   13 +-
 include/linux/kprobes.h                  |    4 +
 include/linux/linkage.h                  |    2 +
 include/linux/marker.h                   |   40 +-
 include/linux/mmiotrace.h                |   85 +
 include/linux/preempt.h                  |   34 +-
 include/linux/sched.h                    |   16 +
 include/linux/writeback.h                |    2 +
 kernel/Makefile                          |   14 +
 kernel/fork.c                            |    2 +-
 kernel/lockdep.c                         |   33 +-
 kernel/marker.c                          |   30 +-
 kernel/printk.c                          |    2 +
 kernel/sched.c                           |   55 +-
 kernel/semaphore.c                       |    1 +
 kernel/spinlock.c                        |    2 +-
 kernel/sysctl.c                          |   11 +
 kernel/trace/Kconfig                     |  135 ++
 kernel/trace/Makefile                    |   24 +
 kernel/trace/ftrace.c                    | 1727 ++++++++++++++++
 kernel/trace/trace.c                     | 3161 ++++++++++++++++++++++++++++++
 kernel/trace/trace.h                     |  339 ++++
 kernel/trace/trace_functions.c           |   81 +
 kernel/trace/trace_irqsoff.c             |  486 +++++
 kernel/trace/trace_mmiotrace.c           |  295 +++
 kernel/trace/trace_sched_switch.c        |  286 +++
 kernel/trace/trace_sched_wakeup.c        |  448 +++++
 kernel/trace/trace_selftest.c            |  563 ++++++
 kernel/trace/trace_selftest_dynamic.c    |    7 +
 kernel/trace/trace_sysprof.c             |  363 ++++
 lib/Kconfig.debug                        |    2 +
 lib/Makefile                             |    9 +
 lib/smp_processor_id.c                   |    6 +-
 mm/page-writeback.c                      |   10 +-
 scripts/Makefile.lib                     |    3 +-
 100 files changed, 11490 insertions(+), 115 deletions(-)
 create mode 100644 Documentation/tracers/mmiotrace.txt
 create mode 100644 arch/arm/kernel/ftrace.c
 create mode 100644 arch/powerpc/kernel/ftrace.c
 create mode 100644 arch/sparc64/kernel/ftrace.c
 create mode 100644 arch/x86/kernel/ftrace.c
 create mode 100644 arch/x86/lib/thunk_32.S
 create mode 100644 arch/x86/mm/kmmio.c
 create mode 100644 arch/x86/mm/mmio-mod.c
 create mode 100644 arch/x86/mm/pf_in.c
 create mode 100644 arch/x86/mm/pf_in.h
 create mode 100644 arch/x86/mm/testmmiotrace.c
 create mode 100644 include/asm-arm/ftrace.h
 create mode 100644 include/asm-powerpc/ftrace.h
 create mode 100644 include/asm-sparc64/ftrace.h
 create mode 100644 include/asm-x86/ftrace.h
 create mode 100644 include/linux/ftrace.h
 create mode 100644 include/linux/mmiotrace.h
 create mode 100644 kernel/trace/Kconfig
 create mode 100644 kernel/trace/Makefile
 create mode 100644 kernel/trace/ftrace.c
 create mode 100644 kernel/trace/trace.c
 create mode 100644 kernel/trace/trace.h
 create mode 100644 kernel/trace/trace_functions.c
 create mode 100644 kernel/trace/trace_irqsoff.c
 create mode 100644 kernel/trace/trace_mmiotrace.c
 create mode 100644 kernel/trace/trace_sched_switch.c
 create mode 100644 kernel/trace/trace_sched_wakeup.c
 create mode 100644 kernel/trace/trace_selftest.c
 create mode 100644 kernel/trace/trace_selftest_dynamic.c
 create mode 100644 kernel/trace/trace_sysprof.c

Re: [git pull] ftrace for v2.6.27

[Bruce Duncan]

[-- Attachment #1.1: Type: text/plain, Size: 639 bytes --]

Hi Ingo, Steven, everyone,

[ Please CC me in replies ]

I have just tracked down a regression in 2.6.27-rc2 wrt 2.6.26. Commit 
77a2b37d227483fe52aead242652aee406c25bf0 (ftrace: startup tester on dynamic 
tracing.) causes my laptop to fail to resume from S3 (it simply reboots about 
a second after the resume starts and the display never shows anything).

The patch doesn't revert with patch -R (I don't know if there's a cleverer way 
to ask git to revert it), but the problem goes away if I turn off 
CONFIG_DYNAMIC_FTRACE.

The commit and bisect log are attached. Please can you help me to debug this?

Cheers,
Bruce

[-- Attachment #1.2: bad-commit.txt --]
[-- Type: text/plain, Size: 6385 bytes --]

commit 77a2b37d227483fe52aead242652aee406c25bf0
Author: Steven Rostedt <srostedt@redhat.com>
Date:   Mon May 12 21:20:45 2008 +0200

    ftrace: startup tester on dynamic tracing.
    
    This patch adds a startup self test on dynamic code modification
    and filters. The test filters on a specific function, makes sure that
    no other function is traced, exectutes the function, then makes sure that
    the function is traced.
    
    This patch also fixes a slight bug with the ftrace selftest, where
    tracer_enabled was not being set.
    
    Signed-off-by: Steven Rostedt <srostedt@redhat.com>
    Signed-off-by: Ingo Molnar <mingo@elte.hu>
    Signed-off-by: Thomas Gleixner <tglx@linutronix.de>

diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index 953a36d..a842d96 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -55,6 +55,7 @@ struct dyn_ftrace {
 };
 
 int ftrace_force_update(void);
+void ftrace_set_filter(unsigned char *buf, int len, int reset);
 
 /* defined in arch */
 extern int ftrace_ip_converted(unsigned long ip);
@@ -70,6 +71,7 @@ extern void ftrace_call(void);
 extern void mcount_call(void);
 #else
 # define ftrace_force_update() ({ 0; })
+# define ftrace_set_filter(buf, len, reset) do { } while (0)
 #endif
 
 static inline void tracer_disable(void)
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index 6d4d2e8..5e9389f 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -1010,6 +1010,25 @@ ftrace_filter_write(struct file *file, const char __user *ubuf,
 	return ret;
 }
 
+/**
+ * ftrace_set_filter - set a function to filter on in ftrace
+ * @buf - the string that holds the function filter text.
+ * @len - the length of the string.
+ * @reset - non zero to reset all filters before applying this filter.
+ *
+ * Filters denote which functions should be enabled when tracing is enabled.
+ * If @buf is NULL and reset is set, all functions will be enabled for tracing.
+ */
+notrace void ftrace_set_filter(unsigned char *buf, int len, int reset)
+{
+	mutex_lock(&ftrace_filter_lock);
+	if (reset)
+		ftrace_filter_reset();
+	if (buf)
+		ftrace_match(buf, len);
+	mutex_unlock(&ftrace_filter_lock);
+}
+
 static int notrace
 ftrace_filter_release(struct inode *inode, struct file *file)
 {
diff --git a/kernel/trace/trace_selftest.c b/kernel/trace/trace_selftest.c
index c01874c..4c8a1b2 100644
--- a/kernel/trace/trace_selftest.c
+++ b/kernel/trace/trace_selftest.c
@@ -99,6 +99,100 @@ static int trace_test_buffer(struct trace_array *tr, unsigned long *count)
 }
 
 #ifdef CONFIG_FTRACE
+
+#ifdef CONFIG_DYNAMIC_FTRACE
+
+#define DYN_FTRACE_TEST_NAME trace_selftest_dynamic_test_func
+#define __STR(x) #x
+#define STR(x) __STR(x)
+static int DYN_FTRACE_TEST_NAME(void)
+{
+	/* used to call mcount */
+	return 0;
+}
+
+/* Test dynamic code modification and ftrace filters */
+int trace_selftest_startup_dynamic_tracing(struct tracer *trace,
+					   struct trace_array *tr,
+					   int (*func)(void))
+{
+	unsigned long count;
+	int ret;
+	int save_ftrace_enabled = ftrace_enabled;
+	int save_tracer_enabled = tracer_enabled;
+
+	/* The ftrace test PASSED */
+	printk(KERN_CONT "PASSED\n");
+	pr_info("Testing dynamic ftrace: ");
+
+	/* enable tracing, and record the filter function */
+	ftrace_enabled = 1;
+	tracer_enabled = 1;
+
+	/* passed in by parameter to fool gcc from optimizing */
+	func();
+
+	/* update the records */
+	ret = ftrace_force_update();
+	if (ret) {
+		printk(KERN_CONT ".. ftraced failed .. ");
+		return ret;
+	}
+
+	/* filter only on our function */
+	ftrace_set_filter(STR(DYN_FTRACE_TEST_NAME),
+			  sizeof(STR(DYN_FTRACE_TEST_NAME)), 1);
+
+	/* enable tracing */
+	tr->ctrl = 1;
+	trace->init(tr);
+	/* Sleep for a 1/10 of a second */
+	msleep(100);
+
+	/* we should have nothing in the buffer */
+	ret = trace_test_buffer(tr, &count);
+	if (ret)
+		goto out;
+
+	if (count) {
+		ret = -1;
+		printk(KERN_CONT ".. filter did not filter .. ");
+		goto out;
+	}
+
+	/* call our function again */
+	func();
+
+	/* sleep again */
+	msleep(100);
+
+	/* stop the tracing. */
+	tr->ctrl = 0;
+	trace->ctrl_update(tr);
+	ftrace_enabled = 0;
+
+	/* check the trace buffer */
+	ret = trace_test_buffer(tr, &count);
+	trace->reset(tr);
+
+	/* we should only have one item */
+	if (!ret && count != 1) {
+		printk(KERN_CONT ".. filter failed ..");
+		ret = -1;
+		goto out;
+	}
+ out:
+	ftrace_enabled = save_ftrace_enabled;
+	tracer_enabled = save_tracer_enabled;
+
+	/* Enable tracing on all functions again */
+	ftrace_set_filter(NULL, 0, 1);
+
+	return ret;
+}
+#else
+# define trace_selftest_startup_dynamic_tracing(trace, tr, func) ({ 0; })
+#endif /* CONFIG_DYNAMIC_FTRACE */
 /*
  * Simple verification test of ftrace function tracer.
  * Enable ftrace, sleep 1/10 second, and then read the trace
@@ -109,8 +203,13 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)
 {
 	unsigned long count;
 	int ret;
+	int save_ftrace_enabled = ftrace_enabled;
+	int save_tracer_enabled = tracer_enabled;
 
-	/* make sure functions have been recorded */
+	/* make sure msleep has been recorded */
+	msleep(1);
+
+	/* force the recorded functions to be traced */
 	ret = ftrace_force_update();
 	if (ret) {
 		printk(KERN_CONT ".. ftraced failed .. ");
@@ -119,6 +218,7 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)
 
 	/* start the tracing */
 	ftrace_enabled = 1;
+	tracer_enabled = 1;
 
 	tr->ctrl = 1;
 	trace->init(tr);
@@ -136,8 +236,16 @@ trace_selftest_startup_function(struct tracer *trace, struct trace_array *tr)
 	if (!ret && !count) {
 		printk(KERN_CONT ".. no entries found ..");
 		ret = -1;
+		goto out;
 	}
 
+	ret = trace_selftest_startup_dynamic_tracing(trace, tr,
+						     DYN_FTRACE_TEST_NAME);
+
+ out:
+	ftrace_enabled = save_ftrace_enabled;
+	tracer_enabled = save_tracer_enabled;
+
 	return ret;
 }
 #endif /* CONFIG_FTRACE */
@@ -415,6 +523,3 @@ trace_selftest_startup_sched_switch(struct tracer *trace, struct trace_array *tr
 	return ret;
 }
 #endif /* CONFIG_CONTEXT_SWITCH_TRACER */
-
-#ifdef CONFIG_DYNAMIC_FTRACE
-#endif /* CONFIG_DYNAMIC_FTRACE */

[-- Attachment #1.3: bisect.log --]
[-- Type: text/x-log, Size: 3192 bytes --]

# bad: [6e86841d05f371b5b9b86ce76c02aaee83352298] Linux 2.6.27-rc1
# good: [bce7f793daec3e65ec5c5705d2457b81fe7b5725] Linux 2.6.26
git-bisect start 'v2.6.27-rc1' 'v2.6.26'
# bad: [d20b27478d6ccf7c4c8de4f09db2bdbaec82a6c0] V4L/DVB (8415): gspca: Infinite loop in i2c_w() of etoms.
git-bisect bad d20b27478d6ccf7c4c8de4f09db2bdbaec82a6c0
# bad: [666484f0250db2e016948d63b3ef33e202e3b8d0] Merge branch 'core/softirq' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect bad 666484f0250db2e016948d63b3ef33e202e3b8d0
# good: [d59fdcf2ac501de99c3dfb452af5e254d4342886] Merge commit 'v2.6.26' into x86/core
git-bisect good d59fdcf2ac501de99c3dfb452af5e254d4342886
# good: [a3da5bf84a97d48cfaf66c6842470fc403da5121] Merge branch 'x86/for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip
git-bisect good a3da5bf84a97d48cfaf66c6842470fc403da5121
# bad: [6712e299b7dc78aa4971b85e803435ee6d49a9dd] Merge branch 'tracing/ftrace' into auto-ftrace-next
git-bisect bad 6712e299b7dc78aa4971b85e803435ee6d49a9dd
# bad: [4823ed7eadf35e4b57ce581327e21d39585f1f32] ftrace: fix setting of pos in read_pipe
git-bisect bad 4823ed7eadf35e4b57ce581327e21d39585f1f32
# bad: [9ff9cdb2d3b0971f89e899b3420aadd91bddc215] ftrace: cleanups
git-bisect bad 9ff9cdb2d3b0971f89e899b3420aadd91bddc215
# good: [c7aafc549766b87819285d3480648fc652a47bc4] ftrace: cleanups
git-bisect good c7aafc549766b87819285d3480648fc652a47bc4
# skip: [f9896bf30928922a3913a3810a4ab7908da6cfe7] ftrace: add raw output
git-bisect skip f9896bf30928922a3913a3810a4ab7908da6cfe7
# skip: [8c523a9d82dbc4f3f7d972df8c0f1eacd83d0d55] ftrace: clean-up-pipe-iteration
git-bisect skip 8c523a9d82dbc4f3f7d972df8c0f1eacd83d0d55
# skip: [cb0f12aae8d085140d37ada351aa5a8e76c3f9b0] ftrace: bin-output
git-bisect skip cb0f12aae8d085140d37ada351aa5a8e76c3f9b0
# skip: [cdd31cd2d7a0dcbec2cce3974f7129dd4fc8c879] ftrace: remove-idx-sync
git-bisect skip cdd31cd2d7a0dcbec2cce3974f7129dd4fc8c879
# skip: [f0a920d5752e1788c0cba2add103076bcc0f7a49] ftrace: add trace_special()
git-bisect skip f0a920d5752e1788c0cba2add103076bcc0f7a49
# skip: [53c37c17aafcf50f7c6fddaf01dda8f9d7e31ddf] ftrace: fast, scalable, synchronized timestamps
git-bisect skip 53c37c17aafcf50f7c6fddaf01dda8f9d7e31ddf
# skip: [dcb6308f2b56720599f6b9d5a01c33e67e69bde4] ftrace, locking fix
git-bisect skip dcb6308f2b56720599f6b9d5a01c33e67e69bde4
# skip: [088b1e427dbba2af93cb6a7d39258c10ff58dd27] ftrace: pipe fixes
git-bisect skip 088b1e427dbba2af93cb6a7d39258c10ff58dd27
# bad: [750ed1a40783432d0dcb0e6c2e813a12615d7664] ftrace: timestamp syncing, prepare
git-bisect bad 750ed1a40783432d0dcb0e6c2e813a12615d7664
# bad: [1d4db00a5e30c7b8f8dc2a1b19e886fd942be143] ftrace: reset selftests
git-bisect bad 1d4db00a5e30c7b8f8dc2a1b19e886fd942be143
# bad: [4e3c3333f3bd7eedfd21b1155b3c7cd24fc7f754] ftrace: fix time offset
git-bisect bad 4e3c3333f3bd7eedfd21b1155b3c7cd24fc7f754
# bad: [77a2b37d227483fe52aead242652aee406c25bf0] ftrace: startup tester on dynamic tracing.
git-bisect bad 77a2b37d227483fe52aead242652aee406c25bf0
# good: [7bd2f24c2f769e3f8f1d4fc8b9fddf689825f6a7] ftrace: add README
git-bisect good 7bd2f24c2f769e3f8f1d4fc8b9fddf689825f6a7

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1374 bytes --]

Re: [git pull] ftrace for v2.6.27

[Ingo Molnar]

* Bruce Duncan <bwduncan@gmail.com> wrote:

> Hi Ingo, Steven, everyone,
> 
> [ Please CC me in replies ]
> 
> I have just tracked down a regression in 2.6.27-rc2 wrt 2.6.26. Commit 
> 77a2b37d227483fe52aead242652aee406c25bf0 (ftrace: startup tester on 
> dynamic tracing.) causes my laptop to fail to resume from S3 (it 
> simply reboots about a second after the resume starts and the display 
> never shows anything).
> 
> The patch doesn't revert with patch -R (I don't know if there's a 
> cleverer way to ask git to revert it), but the problem goes away if I 
> turn off CONFIG_DYNAMIC_FTRACE.
> 
> The commit and bisect log are attached. Please can you help me to 
> debug this?

could you please send your .config file as well?

Suspend+resume bugs are notoriously hard to debug (there's no console 
output in most cases), so i guess the best way to debug this is for 
Steve or me to try to reproduce it. Could you send the output of this 
script:

  http://people.redhat.com/mingo/cfs-scheduler/tools/cfs-debug-info.sh

that will tell us what kind of hw you have.

and could you perhaps try the patch below? The patch disables on the fly 
dynamic code patching - which is the primary suspect here. If it makes a 
difference on your box then a theory would be that ftraced runs too soon 
after suspend+resume.

	Ingo

----------------------->
Subject: ftrace: debug
From: Ingo Molnar <mingo@elte.hu>
Date: Wed Aug 13 09:17:12 CEST 2008

Signed-off-by: Ingo Molnar <mingo@elte.hu>
---
 kernel/trace/ftrace.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Index: linux/kernel/trace/ftrace.c
===================================================================
--- linux.orig/kernel/trace/ftrace.c
+++ linux/kernel/trace/ftrace.c
@@ -811,7 +811,7 @@ static int ftraced(void *ignore)
 		/* check once a second */
 		schedule_timeout(HZ);
 
-		if (unlikely(ftrace_disabled))
+		if (1)
 			continue;
 
 		mutex_lock(&ftrace_sysctl_lock);

Re: [git pull] ftrace for v2.6.27

[Steven Rostedt]

Bruce Duncan wrote:
> Hi Ingo, Steven, everyone,
>
> [ Please CC me in replies ]
>
> I have just tracked down a regression in 2.6.27-rc2 wrt 2.6.26. Commit 
> 77a2b37d227483fe52aead242652aee406c25bf0 (ftrace: startup tester on dynamic 
> tracing.) causes my laptop to fail to resume from S3 (it simply reboots about 
> a second after the resume starts and the display never shows anything).
>
> The patch doesn't revert with patch -R (I don't know if there's a cleverer way 
> to ask git to revert it), but the problem goes away if I turn off 
> CONFIG_DYNAMIC_FTRACE.
>
> The commit and bisect log are attached. Please can you help me to debug this?
>
>
>   

I found that I have a box that displays this bug. Without FTRACE it 
suspends and resumes fine. With FTRACE enabled, it does not.

I'll be actively debugging it now. (unfortunately, as with all new 
laptops, I don't have a serial for it).

-- Steve

Re: [git pull] ftrace for v2.6.27

[Ingo Molnar]

* Steven Rostedt <srostedt@redhat.com> wrote:

> Bruce Duncan wrote:
>> Hi Ingo, Steven, everyone,
>>
>> [ Please CC me in replies ]
>>
>> I have just tracked down a regression in 2.6.27-rc2 wrt 2.6.26. Commit  
>> 77a2b37d227483fe52aead242652aee406c25bf0 (ftrace: startup tester on 
>> dynamic tracing.) causes my laptop to fail to resume from S3 (it simply 
>> reboots about a second after the resume starts and the display never 
>> shows anything).
>>
>> The patch doesn't revert with patch -R (I don't know if there's a 
>> cleverer way to ask git to revert it), but the problem goes away if I 
>> turn off CONFIG_DYNAMIC_FTRACE.
>>
>> The commit and bisect log are attached. Please can you help me to debug this?
>>
>>
>>   
>
> I found that I have a box that displays this bug. Without FTRACE it 
> suspends and resumes fine. With FTRACE enabled, it does not.
>
> I'll be actively debugging it now. (unfortunately, as with all new 
> laptops, I don't have a serial for it).

try latest tip/master plus an USB debug key, then enable 
CONFIG_EARLY_PRINTK_DBGP=y in your .config on the host, add the 
earlyprintk=dbgp,keep boot parameter and reboot [and disable USB for 
good measure]. On the other host you connect to via an USB cable, enable 
USB_SERIAL_DEBUG=y and use /dev/ttyUSB0 with the serial terminal app of 
your choice.

that, as long as the USB ports are operational, gives a working serial 
console on just about any laptop.

	Ingo

Re: [git pull] ftrace for v2.6.27

[Steven Rostedt]

[-- Attachment #1: Type: text/plain, Size: 1623 bytes --]

Bruce Duncan wrote:
> Hi Ingo, Steven, everyone,
>
> [ Please CC me in replies ]
>
> I have just tracked down a regression in 2.6.27-rc2 wrt 2.6.26. Commit 
> 77a2b37d227483fe52aead242652aee406c25bf0 (ftrace: startup tester on dynamic 
> tracing.) causes my laptop to fail to resume from S3 (it simply reboots about 
> a second after the resume starts and the display never shows anything).
>
> The patch doesn't revert with patch -R (I don't know if there's a cleverer way 
> to ask git to revert it), but the problem goes away if I turn off 
> CONFIG_DYNAMIC_FTRACE.
>
> The commit and bisect log are attached. Please can you help me to debug this?
>   
Hi Bruce,

Seems that the issue is with the "ftraced" daemon. The patch you 
reverted was just a fix in the ftrace startup test. If something fails 
in the start up test, that something is disabled.  There was a bug in 
the start up test that caused the dynamic ftrace to fail, and thus 
disabling dynamic ftrace. The patch you reverted, was the fix to that 
bug, so you indirectly disabled dynamic ftrace with that revert.

I've been developing a way to get rid of the "ftraced" daemon. Here's a 
big patch to do so against latest git (as of today). Can you go back to 
the original git, apply this patch and see if you can resume again.

After applying this patch you need to do a "chmod +x 
scripts/recordmcount.pl" before compiling.

Thanks,

-- Steve


Note: my goodmis.org account is currently offline :-( so I'm sending 
this from my redhat account, where I use TB. I have yet to figure out 
how to send a patch inline correctly with TB, so I'm attaching it.


[-- Attachment #2: ftrace-mcount-record-big.patch --]
[-- Type: text/x-patch, Size: 19952 bytes --]

Index: linux-compile.git/include/asm-generic/vmlinux.lds.h
===================================================================
--- linux-compile.git.orig/include/asm-generic/vmlinux.lds.h	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/include/asm-generic/vmlinux.lds.h	2008-08-13 12:25:44.000000000 -0400
@@ -37,6 +37,13 @@
 #define MEM_DISCARD(sec) *(.mem##sec)
 #endif
 
+#ifdef CONFIG_FTRACE_MCOUNT_RECORD
+#define MCOUNT_REC()	VMLINUX_SYMBOL(__start_mcount_loc) = .; \
+			*(__mcount_loc)				\
+			VMLINUX_SYMBOL(__stop_mcount_loc) = .;
+#else
+#define MCOUNT_REC()
+#endif
 
 /* .data section */
 #define DATA_DATA							\
@@ -188,6 +195,7 @@
 	/* __*init sections */						\
 	__init_rodata : AT(ADDR(__init_rodata) - LOAD_OFFSET) {		\
 		*(.ref.rodata)						\
+		MCOUNT_REC()						\
 		DEV_KEEP(init.rodata)					\
 		DEV_KEEP(exit.rodata)					\
 		CPU_KEEP(init.rodata)					\
Index: linux-compile.git/scripts/Makefile.build
===================================================================
--- linux-compile.git.orig/scripts/Makefile.build	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/scripts/Makefile.build	2008-08-13 12:25:44.000000000 -0400
@@ -198,10 +198,16 @@ cmd_modversions =							\
 	fi;
 endif
 
+ifdef CONFIG_FTRACE_MCOUNT_RECORD
+cmd_record_mcount = scripts/recordmcount.pl "$(ARCH)" \
+	"$(OBJDUMP)" "$(OBJCOPY)" "$(CC)" "$(LD)" "$(NM)" "$(RM)" "$(MV)" "$(@)";
+endif
+
 define rule_cc_o_c
 	$(call echo-cmd,checksrc) $(cmd_checksrc)			  \
 	$(call echo-cmd,cc_o_c) $(cmd_cc_o_c);				  \
 	$(cmd_modversions)						  \
+	$(cmd_record_mcount)						  \
 	scripts/basic/fixdep $(depfile) $@ '$(call make-cmd,cc_o_c)' >    \
 	                                              $(dot-target).tmp;  \
 	rm -f $(depfile);						  \
Index: linux-compile.git/scripts/recordmcount.pl
===================================================================
--- /dev/null	1970-01-01 00:00:00.000000000 +0000
+++ linux-compile.git/scripts/recordmcount.pl	2008-08-13 12:25:44.000000000 -0400
@@ -0,0 +1,280 @@
+#!/usr/bin/perl -w
+# (c) 2008, Steven Rostedt <srostedt@redhat.com>
+# Licensed under the terms of the GNU GPL License version 2
+#
+# recordmcount.pl - makes a section called __mcount_loc that holds
+#                   all the offsets to the calls to mcount.
+#
+#
+# What we want to end up with is a section in vmlinux called
+# __mcount_loc that contains a list of pointers to all the
+# call sites in the kernel that call mcount. Later on boot up, the kernel
+# will read this list, save the locations and turn them into nops.
+# When tracing or profiling is later enabled, these locations will then
+# be converted back to pointers to some function.
+#
+# This is no easy feat. This script is called just after the original
+# object is compiled and before it is linked.
+#
+# The references to the call sites are offsets from the section of text
+# that the call site is in. Hence, all functions in a section that
+# has a call site to mcount, will have the offset from the beginning of
+# the section and not the beginning of the function.
+#
+# The trick is to find a way to record the beginning of the section.
+# The way we do this is to look at the first function in the section
+# which will also be the location of that section after final link.
+# e.g.
+#
+#  .section ".text.sched"
+#  .globl my_func
+#  my_func:
+#        [...]
+#        call mcount  (offset: 0x5)
+#        [...]
+#        ret
+#  other_func:
+#        [...]
+#        call mcount (offset: 0x1b)
+#        [...]
+#
+# Both relocation offsets for the mcounts in the above example will be
+# offset from .text.sched. If we make another file called tmp.s with:
+#
+#  .section __mcount_loc
+#  .quad  my_func + 0x5
+#  .quad  my_func + 0x1b
+#
+# We can then compile this tmp.s into tmp.o, and link it to the original
+# object.
+#
+# But this gets hard if my_func is not globl (a static function).
+# In such a case we have:
+#
+#  .section ".text.sched"
+#  my_func:
+#        [...]
+#        call mcount  (offset: 0x5)
+#        [...]
+#        ret
+#  .globl my_func
+#  other_func:
+#        [...]
+#        call mcount (offset: 0x1b)
+#        [...]
+#
+# If we make the tmp.s the same as above, when we link together with
+# the original object, we will end up with two symbols for my_func:
+# one local, one global.  After final compile, we will end up with
+# an undefined reference to my_func.
+#
+# Since local objects can reference local variables, we need to find
+# a way to make tmp.o reference the local objects of the original object
+# file after it is linked together. To do this, we convert the my_func
+# into a global symbol before linking tmp.o. Then after we link tmp.o
+# we will only have a single symbol for my_func that is global.
+# We can convert my_func back into a local symbol and we are done.
+#
+# Here are the steps we take:
+#
+# 1) Record all the local symbols by using 'nm'
+# 2) Use objdump to find all the call site offsets and sections for
+#    mcount.
+# 3) Compile the list into its own object.
+# 4) Do we have to deal with local functions? If not, go to step 8.
+# 5) Make an object that converts these local functions to global symbols
+#    with objcopy.
+# 6) Link together this new object with the list object.
+# 7) Convert the local functions back to local symbols and rename
+#    the result as the original object.
+#    End.
+# 8) Link the object with the list object.
+# 9) Move the result back to the original object.
+#    End.
+#
+
+use strict;
+
+my $P = $0;
+$P =~ s@.*/@@g;
+
+my $V = '0.1';
+
+if ($#ARGV < 6) {
+	print "usage: $P arch objdump objcopy cc ld nm rm mv inputfile\n";
+	print "version: $V\n";
+	exit(1);
+}
+
+my ($arch, $objdump, $objcopy, $cc, $ld, $nm, $rm, $mv, $inputfile) = @ARGV;
+
+$objdump = "objdump" if ((length $objdump) == 0);
+$objcopy = "objcopy" if ((length $objcopy) == 0);
+$cc = "gcc" if ((length $cc) == 0);
+$ld = "ld" if ((length $ld) == 0);
+$nm = "nm" if ((length $nm) == 0);
+$rm = "rm" if ((length $rm) == 0);
+$mv = "mv" if ((length $mv) == 0);
+
+#print STDERR "running: $P '$arch' '$objdump' '$objcopy' '$cc' '$ld' " .
+#    "'$nm' '$rm' '$mv' '$inputfile'\n";
+
+my %locals;
+my %convert;
+
+my $type;
+my $section_regex;	# Find the start of a section
+my $function_regex;	# Find the name of a function (return func name)
+my $mcount_regex;	# Find the call site to mcount (return offset)
+
+if ($arch eq "x86_64") {
+    $section_regex = "Disassembly of section";
+    $function_regex = "<(.*?)>:";
+    $mcount_regex = "^\\s*([0-9a-fA-F]+):.*\\smcount\\+";
+    $type = ".quad";
+} elsif ($arch eq "i386") {
+    $section_regex = "Disassembly of section";
+    $function_regex = "<(.*?)>:";
+    $mcount_regex = "^\\s*([0-9a-fA-F]+):.*\\smcount\$";
+    $type = ".long";
+} else {
+    die "Arch $arch is not supported with CONFIG_FTRACE_MCOUNT_RECORD";
+}
+
+my $text_found = 0;
+my $read_function = 0;
+my $opened = 0;
+my $text = "";
+my $mcount_section = "__mcount_loc";
+
+my $dirname;
+my $filename;
+my $prefix;
+my $ext;
+
+if ($inputfile =~ m,^(.*)/([^/]*)$,) {
+    $dirname = $1;
+    $filename = $2;
+} else {
+    $dirname = ".";
+    $filename = $inputfile;
+}
+
+if ($filename =~ m,^(.*)(\.\S),) {
+    $prefix = $1;
+    $ext = $2;
+} else {
+    $prefix = $filename;
+    $ext = "";
+}
+
+my $mcount_s = $dirname . "/.tmp_mc_" . $prefix . ".s";
+my $mcount_o = $dirname . "/.tmp_mc_" . $prefix . ".o";
+
+#
+# Step 1: find all the local symbols (static functions).
+#
+open (IN, "$nm $inputfile|") || die "error running $nm";
+while (<IN>) {
+    if (/^[0-9a-fA-F]+\s+t\s+(\S+)/) {
+	$locals{$1} = 1;
+    }
+}
+close(IN);
+
+#
+# Step 2: find the sections and mcount call sites
+#
+open(IN, "$objdump -dr $inputfile|") || die "error running $objdump";
+
+while (<IN>) {
+    # is it a section?
+    if (/$section_regex/) {
+	$read_function = 1;
+	$text_found = 0;
+    # section found, now is this a start of a function?
+    } elsif ($read_function && /$function_regex/) {
+	$read_function = 0;
+	$text_found = 1;
+	$text = $1;
+	# is this function static? If so, note this fact.
+	if (defined $locals{$text}) {
+	    $convert{$text} = 1;
+	}
+    # is this a call site to mcount? If so, print the offset from the section
+    } elsif ($text_found && /$mcount_regex/) {
+	if (!$opened) {
+	    open(FILE, ">$mcount_s") || die "can't create $mcount_s\n";
+	    $opened = 1;
+	    print FILE "\t.section $mcount_section,\"a\",\@progbits\n";
+	}
+	print FILE "\t$type $text + 0x$1\n";
+    }
+}
+
+# If we did not find any mcount callers, we are done (do nothing).
+if (!$opened) {
+    exit(0);
+}
+
+close(FILE);
+
+#
+# Step 3: Compile the file that holds the list of call sites to mcount.
+#
+`$cc -o $mcount_o -c $mcount_s`;
+
+my @converts = keys %convert;
+
+#
+# Step 4: Do we have sections that started with local functions?
+#
+if ($#converts >= 0) {
+    my $globallist = "";
+    my $locallist = "";
+
+    foreach my $con (@converts) {
+	$globallist .= " --globalize-symbol $con";
+	$locallist .= " --localize-symbol $con";
+    }
+
+    my $globalobj = $dirname . "/.tmp_gl_" . $filename;
+    my $globalmix = $dirname . "/.tmp_mx_" . $filename;
+
+    #
+    # Step 5: set up each local function as a global
+    #
+    `$objcopy $globallist $inputfile $globalobj`;
+
+    #
+    # Step 6: Link the global version to our list.
+    #
+    `$ld -r $globalobj $mcount_o -o $globalmix`;
+
+    #
+    # Step 7: Convert the local functions back into local symbols
+    #
+    `$objcopy $locallist $globalmix $inputfile`;
+
+    # Remove the temp files
+    `$rm $globalobj $globalmix`;
+
+} else {
+
+    my $mix = $dirname . "/.tmp_mx_" . $filename;
+
+    #
+    # Step 8: Link the object with our list of call sites object.
+    #
+    `$ld -r $inputfile $mcount_o -o $mix`;
+
+    #
+    # Step 9: Move the result back to the original object.
+    #
+    `$mv $mix $inputfile`;
+}
+
+# Clean up the temp files
+`$rm $mcount_o $mcount_s`;
+
+exit(0);
Index: linux-compile.git/kernel/trace/Kconfig
===================================================================
--- linux-compile.git.orig/kernel/trace/Kconfig	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/kernel/trace/Kconfig	2008-08-13 12:25:44.000000000 -0400
@@ -7,6 +7,9 @@ config HAVE_FTRACE
 config HAVE_DYNAMIC_FTRACE
 	bool
 
+config HAVE_FTRACE_MCOUNT_RECORD
+	bool
+
 config TRACER_MAX_TRACE
 	bool
 
@@ -121,6 +124,11 @@ config DYNAMIC_FTRACE
 	 were made. If so, it runs stop_machine (stops all CPUS)
 	 and modifies the code to jump over the call to ftrace.
 
+config FTRACE_MCOUNT_RECORD
+	def_bool y
+	depends on DYNAMIC_FTRACE
+	depends on HAVE_FTRACE_MCOUNT_RECORD
+
 config FTRACE_SELFTEST
 	bool
 
Index: linux-compile.git/include/asm-x86/ftrace.h
===================================================================
--- linux-compile.git.orig/include/asm-x86/ftrace.h	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/include/asm-x86/ftrace.h	2008-08-13 12:25:44.000000000 -0400
@@ -7,6 +7,16 @@
 
 #ifndef __ASSEMBLY__
 extern void mcount(void);
+
+static inline unsigned long ftrace_call_adjust(unsigned long addr)
+{
+	/*
+	 * call mcount is "e8 <4 byte offset>"
+	 * The addr points to the 4 byte offset and the caller of this
+	 * function wants the pointer to e8. Simply subtract one.
+	 */
+	return addr - 1;
+}
 #endif
 
 #endif /* CONFIG_FTRACE */
Index: linux-compile.git/include/linux/ftrace.h
===================================================================
--- linux-compile.git.orig/include/linux/ftrace.h	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/include/linux/ftrace.h	2008-08-13 12:25:44.000000000 -0400
@@ -141,4 +141,13 @@ static inline void
 ftrace_special(unsigned long arg1, unsigned long arg2, unsigned long arg3) { }
 #endif
 
+#ifdef CONFIG_FTRACE_MCOUNT_RECORD
+extern void ftrace_init(void);
+extern void ftrace_init_module(unsigned long *start, unsigned long *end);
+#else
+static inline void ftrace_init(void) { }
+static inline void
+ftrace_init_module(unsigned long *start, unsigned long *end) { }
+#endif
+
 #endif /* _LINUX_FTRACE_H */
Index: linux-compile.git/init/main.c
===================================================================
--- linux-compile.git.orig/init/main.c	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/init/main.c	2008-08-13 12:25:44.000000000 -0400
@@ -60,6 +60,7 @@
 #include <linux/sched.h>
 #include <linux/signal.h>
 #include <linux/idr.h>
+#include <linux/ftrace.h>
 
 #include <asm/io.h>
 #include <asm/bugs.h>
@@ -687,6 +688,8 @@ asmlinkage void __init start_kernel(void
 
 	acpi_early_init(); /* before LAPIC and SMP init */
 
+	ftrace_init();
+
 	/* Do the rest non-__init'ed, we're now alive */
 	rest_init();
 }
Index: linux-compile.git/kernel/trace/ftrace.c
===================================================================
--- linux-compile.git.orig/kernel/trace/ftrace.c	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/kernel/trace/ftrace.c	2008-08-13 12:25:44.000000000 -0400
@@ -792,47 +792,7 @@ static int ftrace_update_code(void)
 	return 1;
 }
 
-static int ftraced(void *ignore)
-{
-	unsigned long usecs;
-
-	while (!kthread_should_stop()) {
-
-		set_current_state(TASK_INTERRUPTIBLE);
-
-		/* check once a second */
-		schedule_timeout(HZ);
-
-		if (unlikely(ftrace_disabled))
-			continue;
-
-		mutex_lock(&ftrace_sysctl_lock);
-		mutex_lock(&ftraced_lock);
-		if (!ftraced_suspend && !ftraced_stop &&
-		    ftrace_update_code()) {
-			usecs = nsecs_to_usecs(ftrace_update_time);
-			if (ftrace_update_tot_cnt > 100000) {
-				ftrace_update_tot_cnt = 0;
-				pr_info("hm, dftrace overflow: %lu change%s"
-					" (%lu total) in %lu usec%s\n",
-					ftrace_update_cnt,
-					ftrace_update_cnt != 1 ? "s" : "",
-					ftrace_update_tot_cnt,
-					usecs, usecs != 1 ? "s" : "");
-				ftrace_disabled = 1;
-				WARN_ON_ONCE(1);
-			}
-		}
-		mutex_unlock(&ftraced_lock);
-		mutex_unlock(&ftrace_sysctl_lock);
-
-		ftrace_shutdown_replenish();
-	}
-	__set_current_state(TASK_RUNNING);
-	return 0;
-}
-
-static int __init ftrace_dyn_table_alloc(void)
+static int __init ftrace_dyn_table_alloc(unsigned long num_to_init)
 {
 	struct ftrace_page *pg;
 	int cnt;
@@ -859,7 +819,9 @@ static int __init ftrace_dyn_table_alloc
 
 	pg = ftrace_pages = ftrace_pages_start;
 
-	cnt = NR_TO_INIT / ENTRIES_PER_PAGE;
+	cnt = num_to_init / ENTRIES_PER_PAGE;
+	pr_info("ftrace: allocating %ld hash entries in %d pages\n",
+		num_to_init, cnt);
 
 	for (i = 0; i < cnt; i++) {
 		pg->next = (void *)get_zeroed_page(GFP_KERNEL);
@@ -1556,6 +1518,109 @@ static __init int ftrace_init_debugfs(vo
 
 fs_initcall(ftrace_init_debugfs);
 
+#ifdef CONFIG_FTRACE_MCOUNT_RECORD
+static int ftrace_convert_nops(unsigned long *start,
+			       unsigned long *end)
+{
+	unsigned long *p;
+	unsigned long addr;
+	unsigned long flags;
+
+	p = start;
+	while (p < end) {
+		addr = ftrace_call_adjust(*p++);
+		ftrace_record_ip(addr);
+		ftrace_shutdown_replenish();
+	}
+
+	/* p is ignored */
+	local_irq_save(flags);
+	__ftrace_update_code(p);
+	local_irq_restore(flags);
+
+	return 0;
+}
+
+void ftrace_init_module(unsigned long *start, unsigned long *end)
+{
+	ftrace_convert_nops(start, end);
+}
+
+extern unsigned long __start_mcount_loc[];
+extern unsigned long __stop_mcount_loc[];
+
+void __init ftrace_init(void)
+{
+	unsigned long count, addr, flags;
+	int ret;
+
+	/* Keep the ftrace pointer to the stub */
+	addr = (unsigned long)ftrace_stub;
+
+	local_irq_save(flags);
+	ftrace_dyn_arch_init(&addr);
+	local_irq_restore(flags);
+
+	/* ftrace_dyn_arch_init places the return code in addr */
+	if (addr)
+		goto failed;
+
+	count = __stop_mcount_loc - __start_mcount_loc;
+
+	ret = ftrace_dyn_table_alloc(count);
+	if (ret)
+		goto failed;
+
+	last_ftrace_enabled = ftrace_enabled = 1;
+
+	ret = ftrace_convert_nops(__start_mcount_loc,
+				  __stop_mcount_loc);
+
+	return;
+ failed:
+	ftrace_disabled = 1;
+}
+#else /* CONFIG_FTRACE_MCOUNT_RECORD */
+static int ftraced(void *ignore)
+{
+	unsigned long usecs;
+
+	while (!kthread_should_stop()) {
+
+		set_current_state(TASK_INTERRUPTIBLE);
+
+		/* check once a second */
+		schedule_timeout(HZ);
+
+		if (unlikely(ftrace_disabled))
+			continue;
+
+		mutex_lock(&ftrace_sysctl_lock);
+		mutex_lock(&ftraced_lock);
+		if (!ftraced_suspend && !ftraced_stop &&
+		    ftrace_update_code()) {
+			usecs = nsecs_to_usecs(ftrace_update_time);
+			if (ftrace_update_tot_cnt > 100000) {
+				ftrace_update_tot_cnt = 0;
+				pr_info("hm, dftrace overflow: %lu change%s"
+					" (%lu total) in %lu usec%s\n",
+					ftrace_update_cnt,
+					ftrace_update_cnt != 1 ? "s" : "",
+					ftrace_update_tot_cnt,
+					usecs, usecs != 1 ? "s" : "");
+				ftrace_disabled = 1;
+				WARN_ON_ONCE(1);
+			}
+		}
+		mutex_unlock(&ftraced_lock);
+		mutex_unlock(&ftrace_sysctl_lock);
+
+		ftrace_shutdown_replenish();
+	}
+	__set_current_state(TASK_RUNNING);
+	return 0;
+}
+
 static int __init ftrace_dynamic_init(void)
 {
 	struct task_struct *p;
@@ -1572,7 +1637,7 @@ static int __init ftrace_dynamic_init(vo
 		goto failed;
 	}
 
-	ret = ftrace_dyn_table_alloc();
+	ret = ftrace_dyn_table_alloc(NR_TO_INIT);
 	if (ret)
 		goto failed;
 
@@ -1593,6 +1658,8 @@ static int __init ftrace_dynamic_init(vo
 }
 
 core_initcall(ftrace_dynamic_init);
+#endif /* CONFIG_FTRACE_MCOUNT_RECORD */
+
 #else
 # define ftrace_startup()		do { } while (0)
 # define ftrace_shutdown()		do { } while (0)
Index: linux-compile.git/kernel/module.c
===================================================================
--- linux-compile.git.orig/kernel/module.c	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/kernel/module.c	2008-08-13 12:25:44.000000000 -0400
@@ -46,6 +46,7 @@
 #include <asm/cacheflush.h>
 #include <linux/license.h>
 #include <asm/sections.h>
+#include <linux/ftrace.h>
 
 #if 0
 #define DEBUGP printk
@@ -1831,6 +1832,7 @@ static struct module *load_module(void _
 #endif
 	unsigned int markersindex;
 	unsigned int markersstringsindex;
+	unsigned int mcountindex;
 	struct module *mod;
 	long err = 0;
 	void *percpu = NULL, *ptr = NULL; /* Stops spurious gcc warning */
@@ -2118,6 +2120,9 @@ static struct module *load_module(void _
  	markersstringsindex = find_sec(hdr, sechdrs, secstrings,
 					"__markers_strings");
 
+	mcountindex = find_sec(hdr, sechdrs, secstrings,
+			       "__mcount_loc");
+
 	/* Now do relocations. */
 	for (i = 1; i < hdr->e_shnum; i++) {
 		const char *strtab = (char *)sechdrs[strindex].sh_addr;
@@ -2167,6 +2172,12 @@ static struct module *load_module(void _
 		marker_update_probe_range(mod->markers,
 			mod->markers + mod->num_markers);
 #endif
+
+	if (mcountindex) {
+		void *mseg = (void *)sechdrs[mcountindex].sh_addr;
+		ftrace_init_module(mseg, mseg + sechdrs[mcountindex].sh_size);
+	}
+
 	err = module_finalize(hdr, sechdrs, mod);
 	if (err < 0)
 		goto cleanup;
Index: linux-compile.git/include/linux/kernel.h
===================================================================
--- linux-compile.git.orig/include/linux/kernel.h	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/include/linux/kernel.h	2008-08-13 12:25:44.000000000 -0400
@@ -486,4 +486,9 @@ struct sysinfo {
 #define NUMA_BUILD 0
 #endif
 
+/* Rebuild everything on CONFIG_FTRACE_MCOUNT_RECORD */
+#ifdef CONFIG_FTRACE_MCOUNT_RECORD
+# define REBUILD_DUE_TO_FTRACE_MCOUNT_RECORD
+#endif
+
 #endif
Index: linux-compile.git/arch/x86/Kconfig
===================================================================
--- linux-compile.git.orig/arch/x86/Kconfig	2008-08-13 12:25:33.000000000 -0400
+++ linux-compile.git/arch/x86/Kconfig	2008-08-13 12:25:44.000000000 -0400
@@ -25,6 +25,7 @@ config X86
 	select HAVE_KPROBES
 	select ARCH_WANT_OPTIONAL_GPIOLIB
 	select HAVE_KRETPROBES
+	select HAVE_FTRACE_MCOUNT_RECORD
 	select HAVE_DYNAMIC_FTRACE
 	select HAVE_FTRACE
 	select HAVE_KVM if ((X86_32 && !X86_VOYAGER && !X86_VISWS && !X86_NUMAQ) || X86_64)

Re: [git pull] ftrace for v2.6.27

[Bruce Duncan]

[-- Attachment #1: Type: text/plain, Size: 1212 bytes --]

On Wednesday 13 August 2008 18:15:14 Steven Rostedt wrote:
> Seems that the issue is with the "ftraced" daemon. The patch you
> reverted was just a fix in the ftrace startup test. If something fails
> in the start up test, that something is disabled.  There was a bug in
> the start up test that caused the dynamic ftrace to fail, and thus
> disabling dynamic ftrace. The patch you reverted, was the fix to that
> bug, so you indirectly disabled dynamic ftrace with that revert.
>
> I've been developing a way to get rid of the "ftraced" daemon. Here's a
> big patch to do so against latest git (as of today). Can you go back to
> the original git, apply this patch and see if you can resume again.

I've been following that thread with great interest! Glad to see it's taking 
shape.

The patch works beautifully. Thanks very much!

Bruce

> After applying this patch you need to do a "chmod +x
> scripts/recordmcount.pl" before compiling.
>
> Thanks,
>
> -- Steve
>
>
> Note: my goodmis.org account is currently offline :-( so I'm sending
> this from my redhat account, where I use TB. I have yet to figure out
> how to send a patch inline correctly with TB, so I'm attaching it.

[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1374 bytes --]

Re: [git pull] ftrace for v2.6.27

[Andi Kleen]

Ingo Molnar <mingo@elte.hu> writes:
>
> try latest tip/master plus an USB debug key, then enable 
> CONFIG_EARLY_PRINTK_DBGP=y in your .config on the host, add the 
> earlyprintk=dbgp,keep boot parameter and reboot [and disable USB for 
> good measure]. On the other host you connect to via an USB cable, enable 
> USB_SERIAL_DEBUG=y and use /dev/ttyUSB0 with the serial terminal app of 
> your choice.
>
> that, as long as the USB ports are operational, gives a working serial 
> console on just about any laptop.

Unfortunately the usb debug dongle cables are hard to get and
expensive. A more available alternative would be to use firescope if
your laptop has firewire, because firewire cables are widely
available. The kernel supports early firewire since some time too, so
it can be even used for early debugging. Another advantage is that
firescope can even get data from a dead kernel as long as the firewire
hardware is not disabled.

For details see Documentation/debugging-via-ohci1394.txt

-Andi

Re: [git pull] ftrace for v2.6.27

[Ingo Molnar]

* Bruce Duncan <bwduncan@gmail.com> wrote:

> On Wednesday 13 August 2008 18:15:14 Steven Rostedt wrote:
> > Seems that the issue is with the "ftraced" daemon. The patch you
> > reverted was just a fix in the ftrace startup test. If something fails
> > in the start up test, that something is disabled.  There was a bug in
> > the start up test that caused the dynamic ftrace to fail, and thus
> > disabling dynamic ftrace. The patch you reverted, was the fix to that
> > bug, so you indirectly disabled dynamic ftrace with that revert.
> >
> > I've been developing a way to get rid of the "ftraced" daemon. Here's a
> > big patch to do so against latest git (as of today). Can you go back to
> > the original git, apply this patch and see if you can resume again.
> 
> I've been following that thread with great interest! Glad to see it's 
> taking shape.
> 
> The patch works beautifully. Thanks very much!

it's now integrated into tip/master (with a few other ftrace goodies), 
if you want to test the end result check out tip/master via:

  http://people.redhat.com/mingo/tip.git/README

	Ingo