Re: Instruction Tracing for Linux

Re: Instruction Tracing for Linux

[Marcelo Tosatti]

On Fri, Jul 08, 2005 at 09:11:03PM +0200, Andi Kleen wrote:
> "Adnan Khaleel" <Adnan.Khaleel@newisys.com> writes:
> 
> > Hi there,
> > 
> > I'm a hardware designer and I'm interested in collecting dynamic execution traces in Linux. I've looked at several trace toolkits available for Linux currently but none of them offer the level of detail that I need. Ideally I would like to be able to record the instructions being executed on an SMP system along with markers for system or user space in addition to process id. I need these traces in order to evaluate the data sharing, coherence traffic etc in larger SMP systems. I've tried several other approaches to collecting execution traces namely via machine emulators etc but so far I've been dogged with the problem of trying to get any OS up and running stably on a multiprocessor configuration.
> > 
> > Is there a Linux kernel patch that will let me do this? I have considered using User Mode Linux but I'm not sure if this is the correct approach either - if any of you think that this is the easier path, I'd be interested in exploring this more. Other things that have crossed my mind is to use a gdb or the kernel debugger interface in order to collect the instructions but I'm not sure if this would be the correct path. Also I do require the tool/patch to be  stable enough so that I can run commercial benchmarks on it reliably.
> > 
> > I understand that recording every executed instruction can considerably slow down the application and may be considerably different from the freely running application but nevertheless I think that some trace is better than no trace and this is where I am at the moment.
> > 
> > If any of you have had experiences in profiling the kernel etc by collecting actual kernel instructions executed, I'd be interested in seeing if that may be extended for my purpose.
> 
> While some CPUs (like Intel P4) have ways to do such hardware
> tracing I know of no free tool that uses it. There are some user
> space tools to collect at user space, but they probably won't help you.
> 
> Your best bet is likely some full system simulator with tracing
> support like Simics, SimNow, bochs, qemu.  The first two might be able to 
> run enterprise benchmarks.

Laplace is a modified version of Bochs for load/store instruction tracing 
and more:

http://www.cs.amherst.edu/~sfkaplan/research/laplace/ 

Instruction Tracing for Linux

[Adnan Khaleel]

Hi there,

I'm a hardware designer and I'm interested in collecting dynamic execution traces in Linux. I've looked at several trace toolkits available for Linux currently but none of them offer the level of detail that I need. Ideally I would like to be able to record the instructions being executed on an SMP system along with markers for system or user space in addition to process id. I need these traces in order to evaluate the data sharing, coherence traffic etc in larger SMP systems. I've tried several other approaches to collecting execution traces namely via machine emulators etc but so far I've been dogged with the problem of trying to get any OS up and running stably on a multiprocessor configuration.

Is there a Linux kernel patch that will let me do this? I have considered using User Mode Linux but I'm not sure if this is the correct approach either - if any of you think that this is the easier path, I'd be interested in exploring this more. Other things that have crossed my mind is to use a gdb or the kernel debugger interface in order to collect the instructions but I'm not sure if this would be the correct path. Also I do require the tool/patch to be  stable enough so that I can run commercial benchmarks on it reliably.

I understand that recording every executed instruction can considerably slow down the application and may be considerably different from the freely running application but nevertheless I think that some trace is better than no trace and this is where I am at the moment.

If any of you have had experiences in profiling the kernel etc by collecting actual kernel instructions executed, I'd be interested in seeing if that may be extended for my purpose.

Thanks

Adnan

PS: I'm not subscribed to this mailing list so I'd appreciated if you would cc me on the responses.

Re: Instruction Tracing for Linux

[Richard B. Johnson]

Please read:
http://www.adapti.com/instruction-trace.html

It's part of an advertisement, but will tell you the problems
you will have with a "software solution". Basically, software
can't do it. Also, even hardware won't give you the real
picture because there is too much going on inside the CPUs
that you can't see. However a hardware/software solution
can produce some useful information but it's not very accurate.


On Fri, 8 Jul 2005, Adnan Khaleel wrote:

> Hi there,
>
> I'm a hardware designer and I'm interested in collecting dynamic execution traces in Linux. I've looked at several trace toolkits available for Linux currently but none of them offer the level of detail that I need. Ideally I would like to be able to record the instructions being executed on an SMP system along with markers for system or user space in addition to process id. I need these traces in order to evaluate the data sharing, coherence traffic etc in larger SMP systems. I've tried several other approaches to collecting execution traces namely via machine emulators etc but so far I've been dogged with the problem of trying to get any OS up and running stably on a multiprocessor configuration.
>
> Is there a Linux kernel patch that will let me do this? I have considered using User Mode Linux but I'm not sure if this is the correct approach either - if any of you think that this is the easier path, I'd be interested in exploring this more. Other things that have crossed my mind is to use a gdb or the kernel debugger interface in order to collect the instructions but I'm not sure if this would be the correct path. Also I do require the tool/patch to be  stable enough so that I can run commercial benchmarks on it reliably.
>
> I understand that recording every executed instruction can considerably slow down the application and may be considerably different from the freely running application but nevertheless I think that some trace is better than no trace and this is where I am at the moment.
>
> If any of you have had experiences in profiling the kernel etc by collecting actual kernel instructions executed, I'd be interested in seeing if that may be extended for my purpose.
>
> Thanks
>
> Adnan
>
> PS: I'm not subscribed to this mailing list so I'd appreciated if you would cc me on the responses.
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Cheers,
Dick Johnson
Penguin : Linux version 2.6.12 on an i686 machine (5537.79 BogoMips).
  Notice : All mail here is now cached for review by Dictator Bush.
                  98.36% of all statistics are fiction.

Re: Instruction Tracing for Linux

[Andi Kleen]

"Adnan Khaleel" <Adnan.Khaleel@newisys.com> writes:

> Hi there,
> 
> I'm a hardware designer and I'm interested in collecting dynamic execution traces in Linux. I've looked at several trace toolkits available for Linux currently but none of them offer the level of detail that I need. Ideally I would like to be able to record the instructions being executed on an SMP system along with markers for system or user space in addition to process id. I need these traces in order to evaluate the data sharing, coherence traffic etc in larger SMP systems. I've tried several other approaches to collecting execution traces namely via machine emulators etc but so far I've been dogged with the problem of trying to get any OS up and running stably on a multiprocessor configuration.
> 
> Is there a Linux kernel patch that will let me do this? I have considered using User Mode Linux but I'm not sure if this is the correct approach either - if any of you think that this is the easier path, I'd be interested in exploring this more. Other things that have crossed my mind is to use a gdb or the kernel debugger interface in order to collect the instructions but I'm not sure if this would be the correct path. Also I do require the tool/patch to be  stable enough so that I can run commercial benchmarks on it reliably.
> 
> I understand that recording every executed instruction can considerably slow down the application and may be considerably different from the freely running application but nevertheless I think that some trace is better than no trace and this is where I am at the moment.
> 
> If any of you have had experiences in profiling the kernel etc by collecting actual kernel instructions executed, I'd be interested in seeing if that may be extended for my purpose.

While some CPUs (like Intel P4) have ways to do such hardware
tracing I know of no free tool that uses it. There are some user
space tools to collect at user space, but they probably won't help you.

Your best bet is likely some full system simulator with tracing
support like Simics, SimNow, bochs, qemu.  The first two might be able to 
run enterprise benchmarks.

-Andi

Re: Instruction Tracing for Linux

[David S. Miller]

From: Andi Kleen <ak@suse.de>
Date: 08 Jul 2005 21:11:03 +0200

> While some CPUs (like Intel P4) have ways to do such hardware
> tracing I know of no free tool that uses it. There are some user
> space tools to collect at user space, but they probably won't help you.

FWIW, even without explicit tracing support in the CPU it
is possible to get these kinds of traces nontheless.

One great example is how they did this on Sparc sun4d machines
at Sun using SKY which was written by Gordon Irlam.  You can
read about it at:

     http://www.base.com/gordoni/sky.html

Basically, the "simulator" would take advantage of Sparc's
delay slot branching to explicitly execute one instruction
from the kernel's code stream at a time.

So it would branch to the kernel's current PC, and in the delay
slot branch back into the simulator.

The only special case is that it needs two temporary registers, global
registers %g2 and %g3, to pull off this trick.  So if the instruction
needed to actually use either register %g2 or %g3, it was emulated in
software instead of actually being executed.

Re: Instruction Tracing for Linux

[Andi Kleen]

On Fri, Jul 08, 2005 at 12:23:34PM -0700, David S. Miller wrote:
> From: Andi Kleen <ak@suse.de>
> Date: 08 Jul 2005 21:11:03 +0200
> 
> > While some CPUs (like Intel P4) have ways to do such hardware
> > tracing I know of no free tool that uses it. There are some user
> > space tools to collect at user space, but they probably won't help you.
> 
> FWIW, even without explicit tracing support in the CPU it
> is possible to get these kinds of traces nontheless.

x86 has single step by default, but doing full tracing with that 
would be difficult and slow. With the simulators it is really
easy though - i used that extensively while bringing up x86-64.

-Andi

RE: Instruction Tracing for Linux

[Adnan Khaleel]

Thanks for your suggestions. I have been working with Simics, SimNow and Bochs. I've had mixed luck with all of them. Although Simics should be the most promising, I've really had
an uphill struggle with it especially when it comes to x86-64. I've been playing around with Bochs and most likely will end up using that but it has its drawbacks as well. 

Even if I can't trace the kernel, is there anything available for just the user space stuff?

Thanks again,

Adnan


-----Original Message-----
From: ak@suse.de [mailto:ak@suse.de]
Sent: Friday, July 08, 2005 2:11 PM
To: Adnan Khaleel
Cc: linux-kernel@vger.kernel.org
Subject: Re: Instruction Tracing for Linux


"Adnan Khaleel" <Adnan.Khaleel@newisys.com> writes:

> Hi there,
> 
> I'm a hardware designer and I'm interested in collecting dynamic execution traces in Linux. I've looked at several trace toolkits available for Linux currently but none of them offer the level of detail that I need. Ideally I would like to be able to record the instructions being executed on an SMP system along with markers for system or user space in addition to process id. I need these traces in order to evaluate the data sharing, coherence traffic etc in larger SMP systems. I've tried several other approaches to collecting execution traces namely via machine emulators etc but so far I've been dogged with the problem of trying to get any OS up and running stably on a multiprocessor configuration.
> 
> Is there a Linux kernel patch that will let me do this? I have considered using User Mode Linux but I'm not sure if this is the correct approach either - if any of you think that this is the easier path, I'd be interested in exploring this more. Other things that have crossed my mind is to use a gdb or the kernel debugger interface in order to collect the instructions but I'm not sure if this would be the correct path. Also I do require the tool/patch to be  stable enough so that I can run commercial benchmarks on it reliably.
> 
> I understand that recording every executed instruction can considerably slow down the application and may be considerably different from the freely running application but nevertheless I think that some trace is better than no trace and this is where I am at the moment.
> 
> If any of you have had experiences in profiling the kernel etc by collecting actual kernel instructions executed, I'd be interested in seeing if that may be extended for my purpose.

While some CPUs (like Intel P4) have ways to do such hardware
tracing I know of no free tool that uses it. There are some user
space tools to collect at user space, but they probably won't help you.

Your best bet is likely some full system simulator with tracing
support like Simics, SimNow, bochs, qemu.  The first two might be able to 
run enterprise benchmarks.

-Andi

Re: Instruction Tracing for Linux

[Miquel van Smoorenburg]

In article <DC392CA07E5A5746837A411B4CA2B713010D7791@sekhmet.ad.newisys.com>,
Adnan Khaleel <Adnan.Khaleel@newisys.com> wrote:
>Thanks for your suggestions. I have been working with Simics, SimNow and
>Bochs. I've had mixed luck with all of them. Although Simics should be
>the most promising, I've really had
>an uphill struggle with it especially when it comes to x86-64. I've been
>playing around with Bochs and most likely will end up using that but it
>has its drawbacks as well. 
>
>Even if I can't trace the kernel, is there anything available for just
>the user space stuff?

You might be able to get valgrind or cachegrind to do what you need..

Mike.

Re: Instruction Tracing for Linux

[Andi Kleen]

On Fri, Jul 08, 2005 at 03:49:47PM -0500, Adnan Khaleel wrote:
> Thanks for your suggestions. I have been working with Simics, SimNow and Bochs. I've had mixed luck with all of them. Although Simics should be the most promising, I've really had
> an uphill struggle with it especially when it comes to x86-64. I've been playing around with Bochs and most likely will end up using that but it has its drawbacks as well. 

I haven't tested any recent versions, but the original development
was near mostly done with Simics and it did work well for me.

iirc there was a recent bug that the optimized memcpy or memset
in the glibc didn't like a CPU returning 0 bytes of cache size and
Simics did that. You might have run into that. It should be
fixed now.

Bochs used to be quite buggy on the x86-64 department and didn't do
multi processor, but that also might have changed. It is significantly
slower than the others.

> 
> Even if I can't trace the kernel, is there anything available for just the user space stuff?

The AMD CodeAnalyst for Linux has a simulator that first collects 
a trace and then runs that in a CPU model.  I assume it does
first single stepping. It's unfortunately binary only.

If slow single stepping is enough it's reasonably easy to write
something yourself too. I have old example source that implements
single stepping.

However I doubt any of them are capable of running the bigger
benchmarks.

-Andi

RE: Instruction Tracing for Linux

[Adnan Khaleel]

Andi,

Would it be possible for you to send me the source to the single stepping code. It may be
too slow but it might prove to be sufficient for some simple benchmarks I guess.

Thanks a million.

Adnan

-----Original Message-----
From: Andi Kleen [mailto:ak@suse.de]
Sent: Friday, July 08, 2005 5:12 PM
To: Adnan Khaleel
Cc: ak@suse.de; linux-kernel@vger.kernel.org
Subject: Re: Instruction Tracing for Linux


On Fri, Jul 08, 2005 at 03:49:47PM -0500, Adnan Khaleel wrote:
> Thanks for your suggestions. I have been working with Simics, SimNow and Bochs. I've had mixed luck with all of them. Although Simics should be the most promising, I've really had
> an uphill struggle with it especially when it comes to x86-64. I've been playing around with Bochs and most likely will end up using that but it has its drawbacks as well. 

I haven't tested any recent versions, but the original development
was near mostly done with Simics and it did work well for me.

iirc there was a recent bug that the optimized memcpy or memset
in the glibc didn't like a CPU returning 0 bytes of cache size and
Simics did that. You might have run into that. It should be
fixed now.

Bochs used to be quite buggy on the x86-64 department and didn't do
multi processor, but that also might have changed. It is significantly
slower than the others.

> 
> Even if I can't trace the kernel, is there anything available for just the user space stuff?

The AMD CodeAnalyst for Linux has a simulator that first collects 
a trace and then runs that in a CPU model.  I assume it does
first single stepping. It's unfortunately binary only.

If slow single stepping is enough it's reasonably easy to write
something yourself too. I have old example source that implements
single stepping.

However I doubt any of them are capable of running the bigger
benchmarks.

-Andi

Re: Instruction Tracing for Linux

[Andi Kleen]

On Fri, Jul 08, 2005 at 05:46:35PM -0500, Adnan Khaleel wrote:
> Andi,
> 
> Would it be possible for you to send me the source to the single stepping code. It may be
> too slow but it might prove to be sufficient for some simple benchmarks I guess.

ftp://ftp.firstfloor.org/pub/ak/smallsrc/stepper.c

-Andi