Re: Hi,something about the xentrace tool

[Rob Gardner <rob.gardner@hp.com>]

George Dunlap wrote:
>
> There's no copying from the HV to the xentrace process.  But there is
> copying from xentrace to the dom0 kernel for the output file.  Some
> copying is necessary right now, because rather than writing out the
> pages verbatim, xentrace writes out the pcpu before writing out each
> record:
>
> void write_rec(unsigned int cpu, struct t_rec *rec, FILE *out)
> {
>    size_t written = 0;
>    written += fwrite(&cpu, sizeof(cpu), 1, out);
>    written += fwrite(rec, sizeof(*rec), 1, out);
>    if ( written != 2 )
>    {
>        PERROR("Failed to write trace record");
>        exit(EXIT_FAILURE);
>    }
> }
>
> If we wanted to make it zero copy all the way from the HV to the disk,
> we could have the xentrace process one stream per cpu, and do
> whatever's necessary to use DMA.  (Does anyone know if O_DIRECT will
> do direct DMA, or if one would have to use a raw disk?) 


So you're saying if we didn't have to write the cpu number, then we 
could bypass stdio, and directly do a write() using the trace buffer? 
And this would be better because it would avoid a memory to memory copy, 
and use DMA immediately on the trace buffer memory? Do I understand you 
correctly? Assuming this is what you mean, allow me to correct a slight 
logic flaw. Stdio is there for a reason; Doing lots of raw I/O using 
very small buffers is highly inefficient. There's the overhead of kernel 
entry/exit and of setting up and tearing down DMA transactions. And 
writing to a block device will result in I/O's that are multiples of the 
devices' block size, so writing a 32 byte trace record will probably 
cause a 512-byte block to actually be written to disk. So bypassing 
stdio in this case will result in lots more disk accesses, lots more dma 
setup/teardown, and lots more system calls. In other words, the 
performance is going to horrible. The Stdio library greatly reduces all 
this overhead by buffering stuff in memory until there's enough to make 
a genuine I/O relatively efficient. In this case, the memory copies are 
intentional and beneficial; We do not want to eliminate them in our 
quest for "zero copy".

Rob