Re: Hi,something about the xentrace tool

[Rob Gardner <rob.gardner@hp.com>]

George Dunlap wrote:
> You misunderstand me. :-)  I meant to write out (via DMA) several
> pages at a time, straight from the HV trace buffers.  The default tbuf
> size in xentrace is 20 pages, so if (as the plan is) xentrace would be
> notified when it would be half full, we could easily write out 10
> pages in one transaction.  The tbuf size could be increased if DMA
> setup/teardown overhead were an issue on that scale.
> ...
> Some of my recent traces have been on the order of 10 gigabytes.  I
> haven't done much to modify xentrace, because I'm not worried about
> the trace overhead at this point.  But I've had to pull some tricks to
> get my analysis tools to run in anything like a reasonable amount of
> time.

I am glad to discover that I misunderstood you. ;) But I am still having 
trouble understanding what the actual problem is, or even if one exists.

If you have a trace that is 10 gigabytes, that's several days (maybe 
weeks) worth of trace records, depending on the rate they're generated.

A memory to memory copy of 10 gigabytes will take mere seconds on any 
modern machine, and amortized over a few days, I don't see how it's 
worth any work to further reduce that or eliminate it. Is the system so 
cpu-bound that the loss of a few seconds over several days is that 
serious? Even compared to the disk I/O to write out 10 gb, which is 
probably several minutes, I don't see how the memory copies are a big 
deal. Perhaps kernel buffer cache effects are noticeable, but again at 
the data rate you're talking about, the cache will only get completely 
purged once every 5 or 10 hours.

If your analysis tools take a long time to run, I'd guess it's because 
of the size of the data, not because system resources are being hogged 
by xentrace; If you are generating that much data, maybe you consider 
methods to reduce it. Take a look the the trace code 
(xen/common/trace.c) and you'll see that there is a facility to mask out 
tracing of certain events, classes of events, and cpu's. You might use 
this to drastically reduce the number of trace records generated. For 
instance, if you are not interested in tracing I/O related events, you 
don't want to be storing TRC_MEM records, which account for a large 
percentage of the trace records generated on a busy system.


Rob