kernel 'simulator' and wave-form analysis tool?

kernel 'simulator' and wave-form analysis tool?

[Timothy Miller]

I wouldn't be surprised if someone's already done this, but...

I'm a chip designer, and when we design a chip, before we put it in 
silicon, we use simulator tools that emulate the logic we designed.  One 
of the most important parts of the simulator is the wave-form analyzer. 
  We run the simulator for some period of time, and then we can look at 
the history of every signal in the design.

Well, I've been looking at Bochs, and it has this 'instrumentation' 
facility which you can use to track everything that goes on in its 
simulation of an x86 processor.  If I were to put a hook in to track all 
memory writes, then I could record all memory activity (I could hook 
much more!).  When a crash occurs, someone could use the analogue to the 
wave-form tool to trace execution back to the event that caused the 
problem (because, for instance, heap corruption causes crashes much 
later than the bug).

Would it be a productive use of my time to work on this?

Re: kernel 'simulator' and wave-form analysis tool?

[Andi Kleen]

Timothy Miller <miller@techsource.com> writes:

> Would it be a productive use of my time to work on this?

Not sure what you mean with waveform tool in this context, but
simulator logs can be indeed very useful to track down kernel bugs. I
used this excessively in the early days of the x86-64 port first with
the SimNow and later with the commercial Virtutech Simics
simulator. They both can generate a log file with all instructions and
all memory accesses. We wrote some post processing tools that added
the kernel symbols to that.  While processing these files (which can
quickly become several GB) was quite a stress test for less and the VM
it was often very useful. I can also only recommend XFS for such
uses, it seems to be by far the best file system for such workloads.

For example to find out who previously changed something in memory you
just need to search backwards for the same physical address.  The real
work usually was just to get a small enough "window" into the problem
to keep the log log files manageable. With logging the simulator runs
extremly slow and fills up disks quite quickly. I don't think it's
useful for complicated problems like races that take longer to trigger
(for longer running tests full logging is not practical), only for
something small and tricky that is relatively easy to repeat.

Writing a post processing tool that adds the symbols is not that complicated,
you can probably do it easily with any simulator log file. While it
would be possible to write a frontend to look for memory addresses
and automate other tasks just grep and less already work reasonably well.
With the amount of data involved you would probably want to do most
analysis as "batch" jobs.

-Andi

Re: kernel 'simulator' and wave-form analysis tool?

[Richard B. Johnson]

On Fri, 5 Mar 2004, Timothy Miller wrote:

> I wouldn't be surprised if someone's already done this, but...
>
> I'm a chip designer, and when we design a chip, before we put it in
> silicon, we use simulator tools that emulate the logic we designed.  One
> of the most important parts of the simulator is the wave-form analyzer.
>   We run the simulator for some period of time, and then we can look at
> the history of every signal in the design.
>
> Well, I've been looking at Bochs, and it has this 'instrumentation'
> facility which you can use to track everything that goes on in its
> simulation of an x86 processor.  If I were to put a hook in to track all
> memory writes, then I could record all memory activity (I could hook
> much more!).  When a crash occurs, someone could use the analogue to the
> wave-form tool to trace execution back to the event that caused the
> problem (because, for instance, heap corruption causes crashes much
> later than the bug).
>
> Would it be a productive use of my time to work on this?
>

If you are making hardware that goes between the CPU and the
rest of the world, then you can keep track of anything that's
going on with some hardware-software combination, external
to the chip you are analyzing. These things exist and they
are called emulators, even though most don't emulate anything,
they use the real chip, but provide the physical and logical
connections to the user. However, in the case of an already-made
machine, you are limited in what you can do on the machine
with software. For instance, to trap every memory access, you
would need a trap-handler and set all the memory to trap
on an access. This would a bit hard to do within the kernel
because all the code on that page would trap as instructions
were fetched. So, some mere "hook" won't do it, you need
a kernel that executes a kernel and I think one for Linux
already exists. So, before you get too involved, you might
want to check that out.

Cheers,
Dick Johnson
Penguin : Linux version 2.4.24 on an i686 machine (797.90 BogoMips).
            Note 96.31% of all statistics are fiction.

Re: kernel 'simulator' and wave-form analysis tool?

[Timothy Miller]

Richard B. Johnson wrote:

> 
> 
> If you are making hardware that goes between the CPU and the
> rest of the world, then you can keep track of anything that's
> going on with some hardware-software combination, external
> to the chip you are analyzing. These things exist and they
> are called emulators, even though most don't emulate anything,
> they use the real chip, but provide the physical and logical
> connections to the user. However, in the case of an already-made
> machine, you are limited in what you can do on the machine
> with software. For instance, to trap every memory access, you
> would need a trap-handler and set all the memory to trap
> on an access. This would a bit hard to do within the kernel
> because all the code on that page would trap as instructions
> were fetched. So, some mere "hook" won't do it, you need
> a kernel that executes a kernel and I think one for Linux
> already exists. So, before you get too involved, you might
> want to check that out.


I must have been unclear.  I was not suggesting adding hardware.  I was 
suggesting that we could run Linux under Bochs, which is a software x86 
emulator.  Being what it is, hooks can be added to track "cpu activity" 
is it occurs within the emulator.  This is all a simulation.  The key 
idea I was suggesting was to log processor activity (of the emulator) 
and develop a viewer program which would help people visualize the activity.

Bochs already has hooks.  I could write a logger and a viewer.

Re: kernel 'simulator' and wave-form analysis tool?

[John Bradford]

> I must have been unclear.  I was not suggesting adding hardware.  I was 
> suggesting that we could run Linux under Bochs, which is a software x86 
> emulator.  Being what it is, hooks can be added to track "cpu activity" 
> is it occurs within the emulator.  This is all a simulation.  The key 
> idea I was suggesting was to log processor activity (of the emulator) 
> and develop a viewer program which would help people visualize the activity.

Doesn't Valgrind already do most of what you want?

John.

Re: kernel 'simulator' and wave-form analysis tool?

[Mike Fedyk]

John Bradford wrote:
>>I must have been unclear.  I was not suggesting adding hardware.  I was 
>>suggesting that we could run Linux under Bochs, which is a software x86 
>>emulator.  Being what it is, hooks can be added to track "cpu activity" 
>>is it occurs within the emulator.  This is all a simulation.  The key 
>>idea I was suggesting was to log processor activity (of the emulator) 
>>and develop a viewer program which would help people visualize the activity.
> 
> 
> Doesn't Valgrind already do most of what you want?

Can you valgrind a UML process?

Tim, what will this give you that a stack trace won't?

Re: kernel 'simulator' and wave-form analysis tool?

[Jeff Dike]

On Sat, Mar 06, 2004 at 06:59:23PM -0800, Mike Fedyk wrote:
> Can you valgrind a UML process?

Sorta.  I think that valgrind should be able to handle UML.  It just
needs to be told how the kernel memory allocators work before it will
provide any useful debugging information about the kernel.

				Jeff

Re: kernel 'simulator' and wave-form analysis tool?

[Timothy Miller]

Mike Fedyk wrote:
> John Bradford wrote:
> 
>>> I must have been unclear.  I was not suggesting adding hardware.  I 
>>> was suggesting that we could run Linux under Bochs, which is a 
>>> software x86 emulator.  Being what it is, hooks can be added to track 
>>> "cpu activity" is it occurs within the emulator.  This is all a 
>>> simulation.  The key idea I was suggesting was to log processor 
>>> activity (of the emulator) and develop a viewer program which would 
>>> help people visualize the activity.
>>
>>
>>
>> Doesn't Valgrind already do most of what you want?
> 
> 
> Can you valgrind a UML process?
> 
> Tim, what will this give you that a stack trace won't?
> 
> 

If your stack gets hosed by a bug, a simulator with a complete history 
of memory writes will help you discover the problem.

I know nothing about Valgrind.

Re: kernel 'simulator' and wave-form analysis tool?

[Herbert Poetzl]

On Mon, Mar 08, 2004 at 11:33:40AM -0500, Timothy Miller wrote:
> 
> 
> Mike Fedyk wrote:
> >John Bradford wrote:
> >
> >>>I must have been unclear.  I was not suggesting adding hardware.  I 
> >>>was suggesting that we could run Linux under Bochs, which is a 
> >>>software x86 emulator.  Being what it is, hooks can be added to track 
> >>>"cpu activity" is it occurs within the emulator.  This is all a 
> >>>simulation.  The key idea I was suggesting was to log processor 
> >>>activity (of the emulator) and develop a viewer program which would 
> >>>help people visualize the activity.

sounds good and useful to me, have a look at 
other simulators/emulators too, for example
I use QEMU[1] to test linux kernels, because
it is much simpler and faster than Bochs (YMMV)

best,
Herbert

> If your stack gets hosed by a bug, a simulator with a complete history 
> of memory writes will help you discover the problem.
> 
> I know nothing about Valgrind.

[1] http://fabrice.bellard.free.fr/qemu/