[PATCH] documentation for basic guide to profiling

[PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

I was trying to write some simple docs on how to do profiling for people 
to use for really basic stuff. I got it all wrong, but John's kindly 
corrected  it ;-) Andrew asked me to do this as a patch for the 
documentation directory ... feedback would be much appreciated 
(yes, it's oversimplified - it's meant to be).

diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt	Wed Dec 31 16:00:00 1969
+++ oprofile_doc/Documentation/basic_profiling.txt	Fri Feb 28 00:05:59 2003
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    (mbligh@aracnet.com)
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear		echo 2 > /proc/profile
+		<test>
+dump output	readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+

Re: [PATCH] documentation for basic guide to profiling

[Andrew Morton]

"Martin J. Bligh" <mbligh@aracnet.com> wrote:
>
> ...
> +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
> +ftp://ftp.kernel.org/pub/linux/       people/mbligh/tools/readprofile/

                                  kernel/

> +add "profile=2" to the kernel command line.
> +
> +clear		echo 2 > /proc/profile
> +		<test>
> +dump output	readprofile -m /boot/System.map > catured_profile

util-linux-2.11z (at least) has the fixed readprofile.

Of course, installing standard util-linux turns your boot process
into a complete mess because vendors have added incompatible features
to their versions.  But it seems to struggle through.

> +Oprofile
> +--------
> +get source (I use 0.5) from http://oprofile.sourceforge.net/
> +add "poll=idle" to the kernel command line 
> +Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
> +./configure --with-kernel-support
> +make install
> +
> +One time setup (pick appropriate one for your CPU):
> +P3		opcontrol --setup --vmlinux=/boot/vmlinux \
> +		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
> +Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \

   Athlon

> +		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
> +P4		opcontrol --setup --vmlinux=/boot/vmlinux \
> +		--ctr0-event=GLOBAL_POWER_EVENTS \
> +		--ctr0-unit-mask=1 --ctr0-count=100000
> +
> +start daemon	opcontrol --start-daemon
> +clear		opcontrol --reset
> +start		opcontrol --start
> +		<test>
> +stop		opcontrol --stop
> +dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
> +

OK.  Might be worth adding a pointer to this in REPORTING-BUGS, but
nobody reads that anyway.

Re: [PATCH] documentation for basic guide to profiling

[Dave Jones]

On Fri, Feb 28, 2003 at 12:29:35AM -0800, Andrew Morton wrote:

 > > +Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \
 >    Athlon

Also x86-64 as of 0.5.1

        Dave

Re: [PATCH] documentation for basic guide to profiling

[Jeff Garzik]

On Fri, Feb 28, 2003 at 11:22:38AM +0000, Dave Jones wrote:
> On Fri, Feb 28, 2003 at 12:29:35AM -0800, Andrew Morton wrote:
> 
>  > > +Athalon		opcontrol --setup --vmlinux=/boot/vmlinux \
>  >    Athlon
> 
> Also x86-64 as of 0.5.1

Alpha is supported too... (as least I saw the kernel part go in)

Re: [PATCH] documentation for basic guide to profiling

[Anton Blanchard]

> Alpha is supported too... (as least I saw the kernel part go in)

sparc64 and ppc64 too, but only timer tick profiling so far.

Anton

Re: [PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

>> Alpha is supported too... (as least I saw the kernel part go in)
> 
> sparc64 and ppc64 too, but only timer tick profiling so far.

I need the magic incantation for oprofile for these then ...

Actually, better still, we need a wrapper script that works out this
from /proc/cpuinfo and auto-sets it up for you, if someone who knows
enough about different Pentium types knows how ... I'm happy to go
write it if it's easy to detect ...

My laptop says: 

model name      : Pentium III (Coppermine)

but a list of pattern matches from someone (or list of people) who
know how this works would be very helpful (ie matching up cpuinfo
to magic incantation to oprofile).

M.

Re: [PATCH] documentation for basic guide to profiling

[Dave Hansen]

Martin J. Bligh wrote:
>>>Alpha is supported too... (as least I saw the kernel part go in)
>>
>>sparc64 and ppc64 too, but only timer tick profiling so far.
> 
> I need the magic incantation for oprofile for these then ...
> 
> Actually, better still, we need a wrapper script that works out this
> from /proc/cpuinfo and auto-sets it up for you, if someone who knows
> enough about different Pentium types knows how ... I'm happy to go
> write it if it's easy to detect ...
> 
> My laptop says: 
> 
> model name      : Pentium III (Coppermine)
> 
> but a list of pattern matches from someone (or list of people) who
> know how this works would be very helpful (ie matching up cpuinfo
> to magic incantation to oprofile).

Actually, oprofile already does this somehow.  opcontrol --list-events
will tell you the perf counters for your particular CPU.  When my
scripts are deciding whether the machine is P3/4, they grep through this
list.

I think the proper way to do this is in oprofile itself.  John mentioned
earlier in this thread that they have considered doing default events,
but there may be a better way to do it.
-- 
Dave Hansen
haveblue@us.ibm.com

Re: [PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

OK, fixed a couple more things (spelling, and pointer to new tools).
I don't really want to add a list of 200 arches, but I'll do a script
with any info people email me the cpuinfo output and correct oprofile
setting for ... I'll do that for the existing ones instead of what's
there now as well.

diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt	Wed Dec 31 16:00:00 1969
+++ prof_docs/Documentation/basic_profiling.txt	Fri Feb 28 15:04:17 2003
@@ -0,0 +1,48 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    (mbligh@aracnet.com)
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+You need a fixed readprofile command for 2.5 ... either get hold of
+a current version from:
+http://www.kernel.org/pub/linux/utils/util-linux/
+or get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/readprofile/
+
+Add "profile=2" to the kernel command line.
+
+clear		readprofile -r
+		<test>
+dump output	readprofile -m /boot/System.map > captured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64	opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+

Re: [PATCH] documentation for basic guide to profiling

[Dave Jones]

On Fri, Feb 28, 2003 at 03:11:21PM -0800, Martin J. Bligh wrote:
 > +start daemon	opcontrol --start-daemon

--start implies starting the daemon if it isn't started
already.

 > +stop		opcontrol --stop

--stop unsupported. use "--shutdown"

 > +dump output	oprofpp -dl -i /boot/vmlinux  >  output_file

opcontrol --dump

op_time -l is also worth adding to this doc imo.

        Dave

Re: [PATCH] documentation for basic guide to profiling

[John Levon]

On Sat, Mar 01, 2003 at 05:51:14PM +0000, Dave Jones wrote:

>  > +start daemon	opcontrol --start-daemon
> 
> --start implies starting the daemon if it isn't started
> already.
> 
>  > +stop		opcontrol --stop
> 
> --stop unsupported. use "--shutdown"

Stop running 2.4 ! :)

--stop works on 2.5 only ...

regards
john

Re: [PATCH] documentation for basic guide to profiling

[Dave Jones]

On Sat, Mar 01, 2003 at 08:48:57PM +0000, John Levon wrote:

 > >  > +stop		opcontrol --stop
 > > --stop unsupported. use "--shutdown"
 > 
 > Stop running 2.4 ! :)
 > 
 > --stop works on 2.5 only ...

That seems quite confusing. Why not make --stop an
alias for --shutdown if running on 2.4 ?

        Dave

Re: [PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

> --start implies starting the daemon if it isn't started
> already.

I think John suggested to do those seperately, to minimise overhead
or something.
 
>  > +dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
> 
> opcontrol --dump
> 
> op_time -l is also worth adding to this doc imo.

OK, will add.

M.

Re: [PATCH] documentation for basic guide to profiling

[Dave Hansen]

Martin J. Bligh wrote:
>>--start implies starting the daemon if it isn't started
>>already.
> 
> I think John suggested to do those seperately, to minimise overhead
> or something.

On a heavily loaded system, --start can take quite a while if the daemon
isn't already running.  Before starting my machine-killing benchmark
(which I want to profile), I use --start-daemon which gets most of the
work out of the way.  The subsequent --start, which occurs while the
benchmark is already steaming along, will happen quickly and with much
less impact to the benchmark results.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [PATCH] documentation for basic guide to profiling

[Pavel Machek]

Hi!

> +Oprofile
> +--------
> +get source (I use 0.5) from http://oprofile.sourceforge.net/
> +add "poll=idle" to the kernel command line 

This should be idle=poll, AFAICS.
							Pavel

-- 
When do you have a heart between your knees?
[Johanka's followup: and *two* hearts?]

Re: [PATCH] documentation for basic guide to profiling

[Randy.Dunlap]

On Fri, 28 Feb 2003 00:12:42 -0800
"Martin J. Bligh" <mbligh@aracnet.com> wrote:

| I was trying to write some simple docs on how to do profiling for people 
| to use for really basic stuff. I got it all wrong, but John's kindly 
| corrected  it ;-) Andrew asked me to do this as a patch for the 
| documentation directory ... feedback would be much appreciated 
| (yes, it's oversimplified - it's meant to be).
| 
| diff -urpN -X /home/fletch/.diff.exclude virgin/Documentation/basic_profiling.txt oprofile_doc/Documentation/basic_profiling.txt
| --- virgin/Documentation/basic_profiling.txt	Wed Dec 31 16:00:00 1969
| +++ oprofile_doc/Documentation/basic_profiling.txt	Fri Feb 28 00:05:59 2003
| @@ -0,0 +1,44 @@
| +These instructions are deliberately very basic. If you want something clever,
| +go read the real docs ;-) Please don't add more stuff, but feel free to 
| +correct my mistakes ;-)    (mbligh@aracnet.com)
| +Thanks to John Levon and Dave Hansen for help writing this.
| +
| +<test> is the thing you're trying to measure.
| +Make sure you have the correct System.map / vmlinux referenced!
| +IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
| +to copy config files, system.map, vmlinux to /boot.
| +
| +Readprofile
| +-----------
| +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
| +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
| +add "profile=2" to the kernel command line.
These:          ^------------v  should be the same value (as you have it).
                             v
| +clear		echo 2 > /proc/profile
man page says to use "readprofile -r".  Doesn't that still work?

| +		<test>
| +dump output	readprofile -m /boot/System.map > catured_profile
                                                > captured_profile


--
~Randy

Re: [PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

> These:          ^------------v  should be the same value (as you have it).
>                              v
>| +clear		echo 2 > /proc/profile
> man page says to use "readprofile -r".  Doesn't that still work?

Dunno. I always have done the above ... have you been using -r with
success? If so, I'll change it.

M.

Re: [PATCH] documentation for basic guide to profiling

[Dave Hansen]

Martin J. Bligh wrote:
>>These:          ^------------v  should be the same value (as you have it).
>>                             v
>>| +clear		echo 2 > /proc/profile
>>man page says to use "readprofile -r".  Doesn't that still work?
> 
> Dunno. I always have done the above ... have you been using -r with
> success? If so, I'll change it.

It's what I've always used.

-- 
Dave Hansen
haveblue@us.ibm.com

Re: [PATCH] documentation for basic guide to profiling

[Randy.Dunlap]

On Fri, 28 Feb 2003 10:13:51 -0800
Dave Hansen <haveblue@us.ibm.com> wrote:

| Martin J. Bligh wrote:
| >>These:          ^------------v  should be the same value (as you have it).
| >>                             v
| >>| +clear		echo 2 > /proc/profile
| >>man page says to use "readprofile -r".  Doesn't that still work?
| > 
| > Dunno. I always have done the above ... have you been using -r with
| > success? If so, I'll change it.
| 
| It's what I've always used.

Same here (-r).

--
~Randy

Re: [PATCH] documentation for basic guide to profiling

[Martin J. Bligh]

OK, fixed a couple of things ... thanks for the feedback everyone.

diff -purN -X /home/mbligh/.diff.exclude virgin/Documentation/basic_profiling.txt prof_docs/Documentation/basic_profiling.txt
--- virgin/Documentation/basic_profiling.txt	Wed Dec 31 16:00:00 1969
+++ prof_docs/Documentation/basic_profiling.txt	Fri Feb 28 13:44:11 2003
@@ -0,0 +1,44 @@
+These instructions are deliberately very basic. If you want something clever,
+go read the real docs ;-) Please don't add more stuff, but feel free to 
+correct my mistakes ;-)    (mbligh@aracnet.com)
+Thanks to John Levon and Dave Hansen for help writing this.
+
+<test> is the thing you're trying to measure.
+Make sure you have the correct System.map / vmlinux referenced!
+IMHO it's easier to use "make install" for linux and hack /sbin/installkernel
+to copy config files, system.map, vmlinux to /boot.
+
+Readprofile
+-----------
+get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
+ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
+add "profile=2" to the kernel command line.
+
+clear		readprofile -r
+		<test>
+dump output	readprofile -m /boot/System.map > catured_profile
+
+Oprofile
+--------
+get source (I use 0.5) from http://oprofile.sourceforge.net/
+add "poll=idle" to the kernel command line 
+Configure with CONFIG_PROFILING=y and CONFIG_OPROFILE=y & reboot on new kernel
+./configure --with-kernel-support
+make install
+
+One time setup (pick appropriate one for your CPU):
+P3		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=CPU_CLK_UNHALTED --ctr0-count=100000
+Athlon/x86-64	opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=RETIRED_INSNS --ctr0-count=100000
+P4		opcontrol --setup --vmlinux=/boot/vmlinux \
+		--ctr0-event=GLOBAL_POWER_EVENTS \
+		--ctr0-unit-mask=1 --ctr0-count=100000
+
+start daemon	opcontrol --start-daemon
+clear		opcontrol --reset
+start		opcontrol --start
+		<test>
+stop		opcontrol --stop
+dump output	oprofpp -dl -i /boot/vmlinux  >  output_file
+

Re: [PATCH] documentation for basic guide to profiling

[Andries Brouwer]

On Fri, Feb 28, 2003 at 12:12:42AM -0800, Martin J. Bligh wrote:

> +Readprofile
> +-----------
> +get readprofile binary fixed for 2.5 / akpm's 2.5 patch from 
> +ftp://ftp.kernel.org/pub/linux/people/mbligh/tools/readprofile/
> +add "profile=2" to the kernel command line.
> +
> +clear		echo 2 > /proc/profile

As far as I can see, the 2 is meaningless here. This should just be

	echo > /proc/profile

(On SMP when writing sizeof(int) bytes, the value written
is significant. But 1 or 2 is not sizeof(int).)

Andries

-----
Fragment of some notes:

<sect1>Profiling the kernel<p>
There are several facilities to see where the kernel spends
its resources. A simple one is the profiling function, that
stores the current EIP (instruction pointer) at each clock tick.
<p>
Boot the kernel with command line option <tt>profile=2</tt>
(or some other number instead of 2). This will cause
a file <tt>/proc/profile</tt> to be created.
The number given after <tt>profile=</tt> is the number of positions
EIP is shifted right when profiling. So a large number gives a
coarse profile.
The counters are reset by writing to <tt>/proc/profile</tt>.
The utility <tt>readprofile</tt> will output statistics for you.
It does not sort - you have to invoke <tt>sort</tt> explicitly.
But given a memory map it will translate addresses to kernel symbols.
<p>
See <tt>kernel/profile.c</tt> and <tt>fs/proc/proc_misc.c</tt>
and <tt>readprofile(1)</tt>.
<p>
For example:
<verb>
# echo > /proc/profile
...
# readprofile -m System.map-2.5.59 | sort -nr | head -2
510502 total                                      0.1534
508548 default_idle                           10594.7500
</verb>