[PATCH] markers-linker-generic

[PATCH] markers-linker-generic

[Mathieu Desnoyers]

Defines the linker marcro EXTRA_RWDATA for marker data section. It
puts the marker data in a separate section that will not pollute the
normal .data section, which minimize the cache impact. Markers need such
a special section because they define a lot of spreaded and small data
structures at multiple sites.

This patch also creates the __markers_strings section (ro marker
strings) and makes sure the __markers section is aligned by putting it
before the __ksymtab_strings (not after).

It applies on 2.6.21-rc6-mm1.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/include/asm-generic/vmlinux.lds.h
+++ b/include/asm-generic/vmlinux.lds.h
@@ -116,21 +116,19 @@
 		*(__kcrctab_gpl_future)					\
 		VMLINUX_SYMBOL(__stop___kcrctab_gpl_future) = .;	\
 	}								\
-									\
-	/* Kernel symbol table: strings */				\
-        __ksymtab_strings : AT(ADDR(__ksymtab_strings) - LOAD_OFFSET) {	\
-		*(__ksymtab_strings)					\
-	}								\
 	/* Kernel markers : pointers */					\
-	.markers : AT(ADDR(.markers) - LOAD_OFFSET) {			\
+	__markers : AT(ADDR(__markers) - LOAD_OFFSET) {			\
 		VMLINUX_SYMBOL(__start___markers) = .;			\
-		*(.markers)						\
+		*(__markers)						\
 		VMLINUX_SYMBOL(__stop___markers) = .;			\
 	}								\
-	.markers.c : AT(ADDR(.markers.c) - LOAD_OFFSET) {		\
-		VMLINUX_SYMBOL(__start___markers_c) = .;		\
-		*(.markers.c)						\
-		VMLINUX_SYMBOL(__stop___markers_c) = .;			\
+	/* Kernel symbol table: strings */				\
+        __ksymtab_strings : AT(ADDR(__ksymtab_strings) - LOAD_OFFSET) {	\
+		*(__ksymtab_strings)					\
+	}								\
+	/* Kernel markers : strings */					\
+	__markers_strings : AT(ADDR(__markers_strings) - LOAD_OFFSET) {	\
+		*(__markers_strings)					\
 	}								\
 	__end_rodata = .;						\
 	. = ALIGN(4096);						\
@@ -147,6 +145,10 @@
 									\
 	. = ALIGN(4096);
 
+#define EXTRA_RWDATA							\
+	. = ALIGN(8);							\
+	*(__markers_data)						\
+
 #define SECURITY_INIT							\
 	.security_initcall.init : AT(ADDR(.security_initcall.init) - LOAD_OFFSET) { \
 		VMLINUX_SYMBOL(__security_initcall_start) = .;		\
@@ -263,4 +265,3 @@
   	*(.initcall6s.init)						\
   	*(.initcall7.init)						\
   	*(.initcall7s.init)
-
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-m68knommu

[Mathieu Desnoyers]

Add EXTRA_RWDATA to m68knommu.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/m68knommu/kernel/vmlinux.lds.S
+++ b/arch/m68knommu/kernel/vmlinux.lds.S
@@ -134,6 +134,7 @@ SECTIONS {
 		. = ALIGN(4);
 		_sdata = . ;
 		*(.data)
+		EXTRA_RWDATA
 		. = ALIGN(8192) ;
 		*(.data.init_task)
 		_edata = . ;
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-mips

[Mathieu Desnoyers]

Add EXTRA_RWDATA to mips.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/mips/kernel/vmlinux.lds.S
+++ b/arch/mips/kernel/vmlinux.lds.S
@@ -64,6 +64,8 @@ SECTIONS
 
     *(.data)
 
+    EXTRA_RWDATA
+
     CONSTRUCTORS
   }
   _gp = . + 0x8000;
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-parisc

[Mathieu Desnoyers]

Add EXTRA_RWDATA to parisc.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/parisc/kernel/vmlinux.lds.S
+++ b/arch/parisc/kernel/vmlinux.lds.S
@@ -92,6 +92,7 @@ SECTIONS
   . = ALIGN(L1_CACHE_BYTES);
   .data : {			/* Data */
 	*(.data)
+	EXTRA_RWDATA
 	CONSTRUCTORS
 	}
 
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-powerpc

[Mathieu Desnoyers]

Add EXTRA_RWDATA to powerpc.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/powerpc/kernel/vmlinux.lds.S
+++ b/arch/powerpc/kernel/vmlinux.lds.S
@@ -170,11 +170,13 @@ SECTIONS
 		*(.data)
 		*(.sdata)
 		*(.got.plt) *(.got)
+		EXTRA_RWDATA
 	}
 #else
 	.data : {
 		*(.data .data.rel* .toc1)
 		*(.branch_lt)
+		EXTRA_RWDATA
 	}
 
 	.opd : {
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-ppc

[Mathieu Desnoyers]

Add EXTRA_RWDATA to ppc.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/ppc/kernel/vmlinux.lds.S
+++ b/arch/ppc/kernel/vmlinux.lds.S
@@ -73,6 +73,7 @@ SECTIONS
     *(.sdata2)
     *(.got.plt) *(.got)
     *(.dynamic)
+    EXTRA_RWDATA
     CONSTRUCTORS
   }
 
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-s390

[Mathieu Desnoyers]

Add EXTRA_RWDATA to s390.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/s390/kernel/vmlinux.lds.S
+++ b/arch/s390/kernel/vmlinux.lds.S
@@ -49,6 +49,7 @@ SECTIONS
 
   .data : {			/* Data */
 	*(.data)
+	EXTRA_RWDATA
 	CONSTRUCTORS
 	}
 
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-sh

[Mathieu Desnoyers]

Add EXTRA_RWDATA to sh.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/sh/kernel/vmlinux.lds.S
+++ b/arch/sh/kernel/vmlinux.lds.S
@@ -40,6 +40,7 @@ SECTIONS
 
   .data : {			/* Data */
 	*(.data)
+	EXTRA_RWDATA
 
  	 /* Align the initial ramdisk image (INITRD) on page boundaries. */
  	 . = ALIGN(PAGE_SIZE);
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-sh64

[Mathieu Desnoyers]

Add EXTRA_RWDATA to sh64.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/sh64/kernel/vmlinux.lds.S
+++ b/arch/sh64/kernel/vmlinux.lds.S
@@ -79,6 +79,7 @@ SECTIONS
 
   .data : C_PHYS(.data) {			/* Data */
 	*(.data)
+	EXTRA_RWDATA
 	CONSTRUCTORS
 	}
 
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-sparc

[Mathieu Desnoyers]

Add EXTRA_RWDATA to sparc.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/sparc/kernel/vmlinux.lds.S
+++ b/arch/sparc/kernel/vmlinux.lds.S
@@ -23,6 +23,7 @@ SECTIONS
   .data    :
   {
     *(.data)
+    EXTRA_RWDATA
     CONSTRUCTORS
   }
   .data1   : { *(.data1) }
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-sparc64

[Mathieu Desnoyers]

Add EXTRA_RWDATA to sparc64.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/sparc64/kernel/vmlinux.lds.S
+++ b/arch/sparc64/kernel/vmlinux.lds.S
@@ -28,6 +28,7 @@ SECTIONS
   .data    :
   {
     *(.data)
+    EXTRA_RWDATA
     CONSTRUCTORS
   }
   .data1   : { *(.data1) }
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-um

[Mathieu Desnoyers]

Add EXTRA_RWDATA to um.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/um/kernel/dyn.lds.S
+++ b/arch/um/kernel/dyn.lds.S
@@ -98,6 +98,7 @@ SECTIONS
     . = ALIGN(KERNEL_STACK_SIZE);		/* init_task */
     *(.data.init_task)
     *(.data .data.* .gnu.linkonce.d.*)
+    EXTRA_RWDATA
     SORT(CONSTRUCTORS)
   }
   .data1          : { *(.data1) }
--- a/arch/um/kernel/uml.lds.S
+++ b/arch/um/kernel/uml.lds.S
@@ -61,6 +61,7 @@ SECTIONS
     *(.data.init_task)
     *(.data)
     *(.gnu.linkonce.d*)
+    EXTRA_RWDATA
     CONSTRUCTORS
   }
   .data1   : { *(.data1) }
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-v850

[Mathieu Desnoyers]

Add EXTRA_RWDATA to v850.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/v850/kernel/vmlinux.lds.S
+++ b/arch/v850/kernel/vmlinux.lds.S
@@ -116,6 +116,7 @@
         	*(.data)						      \
 			*(.exit.data)	/* 2.5 convention */		      \
 			*(.data.exit)	/* 2.4 convention */		      \
+		EXTRA_RWDATA						      \
 		. = ALIGN (16) ;					      \
 		*(.data.cacheline_aligned)				      \
 		. = ALIGN (0x2000) ;					      \
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-x86_64

[Mathieu Desnoyers]

Add EXTRA_RWDATA to x86_64.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/x86_64/kernel/vmlinux.lds.S
+++ b/arch/x86_64/kernel/vmlinux.lds.S
@@ -59,6 +59,7 @@ SECTIONS
 				/* Data */
   .data : AT(ADDR(.data) - LOAD_OFFSET) {
 	*(.data)
+	EXTRA_RWDATA
 	CONSTRUCTORS
 	} :data
 
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

[PATCH] markers-linker-xtensa

[Mathieu Desnoyers]

Add EXTRA_RWDATA to xtensa.

Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>

--- a/arch/xtensa/kernel/vmlinux.lds.S
+++ b/arch/xtensa/kernel/vmlinux.lds.S
@@ -144,7 +144,7 @@ SECTIONS
   _fdata = .;
   .data :
   {
-    *(.data) CONSTRUCTORS
+    *(.data) EXTRA_RWDATA CONSTRUCTORS
     . = ALIGN(XCHAL_ICACHE_LINESIZE);
     *(.data.cacheline_aligned)
   }
-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH] markers-linker-generic

[Russell King]

On Tue, Apr 10, 2007 at 06:36:58PM -0400, Mathieu Desnoyers wrote:
> Defines the linker marcro EXTRA_RWDATA for marker data section. It
> puts the marker data in a separate section that will not pollute the
> normal .data section, which minimize the cache impact. Markers need such
> a special section because they define a lot of spreaded and small data
> structures at multiple sites.
> 
> This patch also creates the __markers_strings section (ro marker
> strings) and makes sure the __markers section is aligned by putting it
> before the __ksymtab_strings (not after).

What's this marker stuff about?

-- 
Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Re: [PATCH] markers-linker-generic

[Mathieu Desnoyers]

* Russell King (rmk+lkml@arm.linux.org.uk) wrote:
> On Tue, Apr 10, 2007 at 06:36:58PM -0400, Mathieu Desnoyers wrote:
> > Defines the linker marcro EXTRA_RWDATA for marker data section. It
> > puts the marker data in a separate section that will not pollute the
> > normal .data section, which minimize the cache impact. Markers need such
> > a special section because they define a lot of spreaded and small data
> > structures at multiple sites.
> > 
> > This patch also creates the __markers_strings section (ro marker
> > strings) and makes sure the __markers section is aligned by putting it
> > before the __ksymtab_strings (not after).
> 
> What's this marker stuff about?
> 

Hi Russel,

Here is an overview :

This marker infrastructure is a hook-callback mechanism. It is 
meant to have an impact as low as possible on the system performances
when no callback (probe) is connected so markers (hooks) can be
compiled into a production kernel without noticeable slowdown.

The rationale behind this mechanism the following :
1 - It makes sense to have instrumentation (for tracing, profiling)
    within the kernel source tree so that it can follow its evolution.
    Other options, such as kprobes, imply maintaining an external set of
    instrumentation that must be adapted to each kernel version.
    Although it may make sense for distributions, it is not well suited
    for kernel developers, since they rarely work on a major
    distribution image.
2 - kprobes, although being a very good attempt at providing a dynamic
    hooking mechanism that has no impact when disabled, suffers from
    important limitations :
  a - It cannot access local variables of a function at a particular
      point within its body that will be consistent thorough the kernel
      versions without involving a lot of recurrent hair-pulling.
  b - Kprobes is slow, since it involves going though a trap each time
      a probe site is executed. Even though the djprobes project made a
      good effort to make things faster, it cannot currently instrument
      fully-preemptible kernels and does not solve (1), (2a) and (2c).
  c - On the reentrancy side, going though a trap (thus playing with
      interrupt enable/disable) and taking spinlocks are not suited to
      some code paths, i.e. :
      kernel/lockdep.c, printk (within the lockdep_on()/lockdep_off()).
      It must be understood that some code paths interesting for
      instrumentation often present a particular reentrancy challenge.

Some more details :

The probe callback connection to its markers is done dynamically. A
predicted branch is used to skip the hook stack setup and function call
when the marker is "disabled" (no probe is connected). Further
optimizations can be implemented for each architecture to make this
branch faster.

Instrumentation of a subsystem becomes therefore a straightforward task.
One has to add instrumentation within the key locations of the kernel
code in the following form :

MARK(subsystem_event, "%d %p", myint, myptr);

I will be glad to discuss more specific questions that may arise about
the markers.

Regards,

Mathieu


-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH] markers-linker-generic

[Andrew Morton]

On Wed, 11 Apr 2007 13:51:11 -0400
Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:

> > What's this marker stuff about?
> > 
> 
> Hi Russel,
> 
> Here is an overview :

I am told that the systemtap developers plan to (or are) using this
infrastructure.

If correct: what is their reason for preferring it over kprobes?


Secondly, the code in -mm has no actual backend for delivering anything to
userspace or to anywhere else.  Why is this?  Which backends currently
exist, and which ones are envisaged?  Is there a plan to merge any backend
into the mainline kernel?

Re: [PATCH] markers-linker-generic

[Mathieu Desnoyers]

* Andrew Morton (akpm@linux-foundation.org) wrote:
> On Wed, 11 Apr 2007 13:51:11 -0400
> Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:
> 
> > > What's this marker stuff about?
> > > 
> > 
> > Hi Russel,
> > 
> > Here is an overview :
> 
> I am told that the systemtap developers plan to (or are) using this
> infrastructure.
> 

Quoting Frank Ch. Eigler, from the SystemTAP team :

"The LTTng user-space programs use it today.  Systemtap used to support
the earlier marker prototype and will be rapidly ported over to this
new API upon acceptance."


> If correct: what is their reason for preferring it over kprobes?
> 

I will let them answer on this one..

> 
> Secondly, the code in -mm has no actual backend for delivering anything to
> userspace or to anywhere else.  Why is this?

I wanted the community to come to an agreement regarding the
instrumentation mechanism before submitting a data transport backend.

> Which backends currently
> exist, and which ones are envisaged?  Is there a plan to merge any backend
> into the mainline kernel?
> 

Yes. The one I envisage to mainline is LTTng, available there :
http://ltt.polymtl.ca. It aims at providing the lowest system
disturbance and best performances possible, with reentrancy from almost
any kernel call site while keeping the task of adding new
instrumentation as easy as possible for the programmer. The current
patches are against 2.6.21-rc6-mm1. Is comes in small modules easy to
merge incrementally, which should make the whole process easier.

LKET, another tracer built on top of SystemTAP, can also be used to
extract the data to user space.

Mathieu


-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH] markers-linker-generic

[Jim Keniston]

On Wed, 2007-04-11 at 15:21 -0400, Mathieu Desnoyers wrote:
> * Andrew Morton (akpm@linux-foundation.org) wrote:
> > On Wed, 11 Apr 2007 13:51:11 -0400
> > Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:
> > 
> > > > What's this marker stuff about?
> > > > 
> > > 
> > > Hi Russel,
> > > 
> > > Here is an overview :
> > 
> > I am told that the systemtap developers plan to (or are) using this
> > infrastructure.
> > 
> 
> Quoting Frank Ch. Eigler, from the SystemTAP team :
> 
> "The LTTng user-space programs use it today.  Systemtap used to support
> the earlier marker prototype and will be rapidly ported over to this
> new API upon acceptance."
> 
> 
> > If correct: what is their reason for preferring it over kprobes?
> > 
> 
> I will let them answer on this one..
> 

I'll take a shot at this one.

First of all, kprobes remains a vital foundation for SystemTap.  But
markers are attactive as an alternate source of trace/debug info.
Here's why:

1. Markers will live in the kernel and presumably be kept up to date by
the maintainers of the enclosing code.  We have a growing set of tapsets
(probe libraries), each of which "knows" the source code for a certain
area of the kernel.  Whenever the underlying kernel code changes (e.g.,
a function or one of its args disappears or is renamed), there's a
chance that the tapset will become invalid until we bring it back in
sync with the kernel.  As you can imagine, maintaining tapsets separate
from the kernel source is a maintenance headache.  Markers could
mitigate this.

2. Because the kernel code is highly optimized, the kernel's dwarf info
doesn't always accurately reflect which variables have which values on
which lines (sometimes even upon entry to a function).  A marker is a
way to ensure that values of interest are available to SystemTap at
marked points.

3. Sometimes the overhead of a kprobe probepoint is too much (either in
terms of time or locking) for the particular hotspot we want to probe.

Jim

Re: [PATCH] markers-linker-generic

[Vara Prasad]

Jim Keniston wrote:

>On Wed, 2007-04-11 at 15:21 -0400, Mathieu Desnoyers wrote:
>  
>
>>* Andrew Morton (akpm@linux-foundation.org) wrote:
>>    
>>
>>>On Wed, 11 Apr 2007 13:51:11 -0400
>>>Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:
>>>
>>>      
>>>
>>>>>What's this marker stuff about?
>>>>>
>>>>>          
>>>>>
>>>>Hi Russel,
>>>>
>>>>Here is an overview :
>>>>        
>>>>
>>>I am told that the systemtap developers plan to (or are) using this
>>>infrastructure.
>>>
>>>      
>>>
>>Quoting Frank Ch. Eigler, from the SystemTAP team :
>>
>>"The LTTng user-space programs use it today.  Systemtap used to support
>>the earlier marker prototype and will be rapidly ported over to this
>>new API upon acceptance."
>>
>>
>>    
>>
>>>If correct: what is their reason for preferring it over kprobes?
>>>      
>>>
Markers are not a substitute or preference over kprobes, they augment 
kprobes by enabling additional functionality.

>>>      
>>>
>>I will let them answer on this one..
>>
>>    
>>
>
>I'll take a shot at this one.
>
>First of all, kprobes remains a vital foundation for SystemTap.  But
>markers are attactive as an alternate source of trace/debug info.
>Here's why:
>
>1. Markers will live in the kernel and presumably be kept up to date by
>the maintainers of the enclosing code.  We have a growing set of tapsets
>(probe libraries), each of which "knows" the source code for a certain
>area of the kernel.  Whenever the underlying kernel code changes (e.g.,
>a function or one of its args disappears or is renamed), there's a
>chance that the tapset will become invalid until we bring it back in
>sync with the kernel.  As you can imagine, maintaining tapsets separate
>from the kernel source is a maintenance headache.  Markers could
>mitigate this.
>  
>
Jim's above stated reason is not a consideration for markers. We don't 
plan to convert the current tapsets to use markers. We do need to 
augment tapsets with a few markers in the kernel code where it is not 
easy to put a kprobe in a maintainable fashion -- e.g in the middle of a 
function.

>2. Because the kernel code is highly optimized, the kernel's dwarf info
>doesn't always accurately reflect which variables have which values on
>which lines (sometimes even upon entry to a function).  A marker is a
>way to ensure that values of interest are available to SystemTap at
>marked points.
>  
>
Agreed

>3. Sometimes the overhead of a kprobe probepoint is too much (either in
>terms of time or locking) for the particular hotspot we want to probe.
>
>  
>
Agreed

>Jim
>
>  
>
bye,
Vara Prasad

Re: [PATCH] markers-linker-generic

[Mathieu Desnoyers]

* Andrew Morton (akpm@linux-foundation.org) wrote:
> On Wed, 11 Apr 2007 13:51:11 -0400
> Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> wrote:
> 
> > > What's this marker stuff about?
> > > 
> > 
> > Hi Russel,
> > 
> > Here is an overview :
> 
> I am told that the systemtap developers plan to (or are) using this
> infrastructure.
> 
> If correct: what is their reason for preferring it over kprobes?
> 
> 
> Secondly, the code in -mm has no actual backend for delivering anything to
> userspace or to anywhere else.  Why is this?  Which backends currently
> exist, and which ones are envisaged?  Is there a plan to merge any backend
> into the mainline kernel?
> 

Here is more context around this :


With the increasing complexity of today's user-space application and the
wide deployment of SMP systems, the users need an increasing
understanding of the behavior and performances of a system across
multiple processes/different execution contexts/multiple CPUs. In
applications such as large clusters (Google, IBM), video acquisition
(Autodesk), embedded real-time systems (Wind River, Monta Vista, Sony)
or sysadmin/programmer-type tasks (SystemTAP from Redhat), a tool that
permits tracing of kernel-user space interaction becomes necessary.

Usage of such tools have been made to successfully pinpoint problems such
as : latency issues in a user-space video acquisition application,
slowdown problems in large clusters due to a switch to a different
filesystems with a different cache size, abnormal Linux scheduler latency
(just to name a few that I have personally investigated).

The currently existing solutions does not give a system-wide overview of
what - and when - things are happening on the system. Ptracing a program
works with few processes, but quickly becomes useless when it
comes to keeping track of many processes.

Bugs occuring because of bad interaction of such complex systems can be
very hard to find due to the fact that they occur rarely (sometimes once
a week on hundreds of machines). One can therefore only hope at having
the best conditions to statistically reproduce the bug while extracting
information from the system. Some bugs have been successfully found at
Google using their ktrace tracer only because they could enable it on
production machines and therefore recreate the same context where the
bug happened.

Therefore, it makes sense to offer an instrumentation set of the most
relevant events occurring in the Linux that can have the smallest
performance cost possible when not active while not requiring a reboot
of a production system to activate. This is essentially what the markers
are providing.

Since we cannot limit the growth of the Linux kernel, nor can we
pre-determine each and every "interesting" instrumentation within each
subsystem and driver, it is sensible to let this task to the persons
who knows the best their code. Adding instrumentation should therefore
be as easy as adding and maintaining a "printk" in the kernel code from
the developer's point of view.

Towards a complete tracing mechanism in the Linux kernel, the markers
are only one step forward. The following step is to connect probes to
those markers that will record the tracing information in buffers
exported to user-space, organized in timestamped "events". Probe
callbacks are responsible for serializing the information passed as
parameter to the markers (described by the format string) into the
events. A control mechanism to activate/stop the tracing is required,
as well as a daemon that maps the buffers to write them to disk or send
them through the network.

Keeping track of the events also requires a centralized infrastructure :
the idea is to assign a unique ID to each event so they can be later
recognized in the trace. Keeping in mind that recording the complete
instrumentation site name string for each event would be more that
inefficient, assigning a numeric unique identifier makes sense.

Finally, support for gathering events coming from user-space, with a
minimal performance impact, is very useful to see the interaction
between the system's execution contexts.

The last steps are currently implemented in Linux Trace Toolkit Next
Generation (LTTng).

The SystemTAP project could clearly benefit from such an infrastructure
for tracing. In addition, they would be providing support for dynamic
addition of kernel probes through breakpoints/jumps when possible, with
the associated restrictions (accessing local variables, reentrancy,
speed).

Mathieu


-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

Re: [PATCH] markers-linker-generic

[Frank Ch. Eigler]

Andrew Morton <akpm@linux-foundation.org> writes:

> [...]  I am told that the systemtap developers plan to (or are)
> using this infrastructure.

Indeed.

> If correct: what is their reason for preferring it over kprobes?
> [...]

It's not a preference - it's more of a supplement.  It's helpful when
some combination of such factors exists:

- kprobe int3-fault dispatching overhead orders of magnitude too high
- fault dispatching not permissible in some areas
- local context variables not easily retrievable via dwarf information
- dwarf information not available at all
- costs of permanently placed but passive marker acceptable

>From systemtap's point of view, instrumentation hooked to markers,
kprobes, and other facilities like timers, coexist just fine.  A
greater number of probe-able event sources makes for a richer tool.


- FChE