unconditionally enable the trace buffer

unconditionally enable the trace buffer

[Rob Gardner]

Last month Mark Williamson sent out a patch to unconditionally enable 
the trace buffer. I'd like to suggest that his patch be accepted and 
merged. I have performed some simple benchmarks to quantify the overhead 
associated with the trace buffer and calls, and found a negligable 
performance loss due to them. The actual number was .069%, and was 
gotten by timing a simple cpu-intensive task running with trace buffers 
compiled in and again with trace buffers not compiled in. It's important 
for us to have the trace buffer functionality totally enabled by 
default, so we can depend on its availability for running XenMon without 
requiring recompiling and rebooting.

Thanks,
Rob Gardner (HP)

RE: unconditionally enable the trace buffer

[Ian Pratt]

> Last month Mark Williamson sent out a patch to 
> unconditionally enable the trace buffer. I'd like to suggest 
> that his patch be accepted and merged. I have performed some 
> simple benchmarks to quantify the overhead associated with 
> the trace buffer and calls, and found a negligable 
> performance loss due to them. The actual number was .069%, 
> and was gotten by timing a simple cpu-intensive task running 
> with trace buffers compiled in and again with trace buffers 
> not compiled in. 

Of course, a cpu intensive workload is likely to be generating fewest
trace events. I doubt the overhead is a big deal, though.

I haven't looked at xentrace in a while, but last time I did it could
seriously do with some tidying up. Here's a list of features I'd like to
see it have. I'd be grateful if you could tell me what the current state
is:

* ability to turn on/off via hypercall
* trace events grouped by type, with a bitmap to enable the event types
of interest
* ability to set the per CPU tracebuffer size when turning it on
* ability for the user-space reader to explicitly block (select on fd)
on an eventchn notification that the buffer is e.g. half full. (reader
should write out all the pages that are full of trace events)
* user space reader should log when it misses blocks of events
(overwrite last trace message in buffer with a special 'missed X events'
message) 

Thanks,
Ian

Re: unconditionally enable the trace buffer

[Rob Gardner]

Ian Pratt wrote:

>I haven't looked at xentrace in a while, but last time I did it could
>seriously do with some tidying up. Here's a list of features I'd like to
>see it have. I'd be grateful if you could tell me what the current state
>is:
>
>* ability to turn on/off via hypercall
>  
>

Not currently implemented, but would not be difficult to add.

>* trace events grouped by type, with a bitmap to enable the event types
>of interest
>  
>

This functionality is in there already.

>* ability to set the per CPU tracebuffer size when turning it on
>  
>

Partially; You can enable the trace buffer on the xen (boot) command 
line, and you can specify the trace buffer size there. You cannot change 
the size dynamically.

>* ability for the user-space reader to explicitly block (select on fd)
>on an eventchn notification that the buffer is e.g. half full. (reader
>should write out all the pages that are full of trace events)
>  
>
Not done.

>* user space reader should log when it misses blocks of events
>(overwrite last trace message in buffer with a special 'missed X events'
>message) 
>
>  
>
Not done, but my XenMon patch includes a change to the trace buffer code 
to help with this. I've added a "sequence number" to each trace record 
which can be used to detect when blocks of events have been missed.

Rob Gardner

Re: unconditionally enable the trace buffer

[Mark Williamson]

> >* ability to turn on/off via hypercall
>
> Not currently implemented, but would not be difficult to add.

Just as an aside I'm not sure this matters: From what Rob's told me, having 
the (inline) trace() calls in there produces the same overhead whether 
tracing is active or not.  I guess it makes sense; once you've incurred the 
overhead of having the function there and evaluating the "is tracing on" 
conditional, you might as well have stored a few values also ;-)

> >* ability to set the per CPU tracebuffer size when turning it on
>
> Partially; You can enable the trace buffer on the xen (boot) command
> line, and you can specify the trace buffer size there. You cannot change
> the size dynamically.

Aside #2: You can disable at boot time by setting size=0.

> >* ability for the user-space reader to explicitly block (select on fd)
> >on an eventchn notification that the buffer is e.g. half full. (reader
> >should write out all the pages that are full of trace events)
>
> Not done.

We can just block on /dev/evtchn, I guess.  Shouldn't be too hard; we just 
need to define a new "trace" virq.

> >* user space reader should log when it misses blocks of events
> >(overwrite last trace message in buffer with a special 'missed X events'
> >message)
>
> Not done, but my XenMon patch includes a change to the trace buffer code
> to help with this. I've added a "sequence number" to each trace record
> which can be used to detect when blocks of events have been missed.

Yep, sounds good to me.

Cheers,
Mark0

RE: unconditionally enable the trace buffer

[Ian Pratt]

 

> > >* ability to turn on/off via hypercall
> >
> > Not currently implemented, but would not be difficult to add.
> 
> Just as an aside I'm not sure this matters: From what Rob's 
> told me, having the (inline) trace() calls in there produces 
> the same overhead whether tracing is active or not.  I guess 
> it makes sense; once you've incurred the overhead of having 
> the function there and evaluating the "is tracing on" 
> conditional, you might as well have stored a few values also ;-)

We could cache miss on reading the tracebuffer producer counter and
tracebuffer base address, so its not totally a done deal.

I don't have a big worry about performance, but I'd feel more
comfortable to see a more realistic assesment of overhead. Comparing
results from ttcp in a domU with 128k sock buf and the MTU set to 552
bytes should do it.
 
> > >* ability to set the per CPU tracebuffer size when turning it on
> >
> > Partially; You can enable the trace buffer on the xen 
> (boot) command 
> > line, and you can specify the trace buffer size there. You cannot 
> > change the size dynamically.
> 
> Aside #2: You can disable at boot time by setting size=0.

Ideally, it should be set as part of the hypercall to turn tracing on.

> > >* ability for the user-space reader to explicitly block (select on 
> > >fd) on an eventchn notification that the buffer is e.g. half full. 
> > >(reader should write out all the pages that are full of 
> trace events)
> >
> > Not done.
> 
> We can just block on /dev/evtchn, I guess.  Shouldn't be too 
> hard; we just need to define a new "trace" virq.

The polling stuff totally sucks, and we definitely need the virq.

> > >* user space reader should log when it misses blocks of events 
> > >(overwrite last trace message in buffer with a special 
> 'missed X events'
> > >message)
> >
> > Not done, but my XenMon patch includes a change to the trace buffer 
> > code to help with this. I've added a "sequence number" to 
> each trace 
> > record which can be used to detect when blocks of events 
> have been missed.

My scheme avoids burning tracebuffer memory, but OK.

Ian

Re: unconditionally enable the trace buffer

[Rob Gardner]

Ian Pratt wrote:

>>>>* ability to turn on/off via hypercall
>>>>        
>>>>
>>>Not currently implemented, but would not be difficult to add.
>>>      
>>>
>>Just as an aside I'm not sure this matters: From what Rob's 
>>told me, having the (inline) trace() calls in there produces 
>>the same overhead whether tracing is active or not.  I guess 
>>it makes sense; once you've incurred the overhead of having 
>>the function there and evaluating the "is tracing on" 
>>conditional, you might as well have stored a few values also ;-)
>>    
>>
>We could cache miss on reading the tracebuffer producer counter and
>tracebuffer base address, so its not totally a done deal.
>
>I don't have a big worry about performance, but I'd feel more
>comfortable to see a more realistic assesment of overhead. Comparing
>results from ttcp in a domU with 128k sock buf and the MTU set to 552
>bytes should do it.
>  
>

I just completed some benchmarks using ttcp as Ian suggests. I was 
surprised by the results, but essentially, Ian's intuition n this case 
may be right on the money. I tested three cases:
  1. trace buffers turned off, that is not compiled in
  2. trace buffers compiled in and turned on
  3. trace buffers compiled in but disabled
Cases 1 and 3 show no performance difference at all, but case 2 does 
show a non-trivial degradation.

Details of the benchmarking: ttcp using 128k buffers to send data to a 
domU from another machine using a gigabit network. 2gb were transferred 
in each run, and 10 runs were performed for each case. Dom0 and domU 
together used up just about all of a cpu, and experienced 2000-2500 
context switches per second to handle 40-50,000 I/O's per second. Please 
note that the additional trace events for XenMon were enabled in this 
system (case 2) so each of those 50,000 I/O's each second generates a 
trace record!

I conclude the following from this:
 - there is no penalty for simply having the trace buffer code compiled 
into xen
 - there is a penalty for enabling tracing
 - therefore, the ability to turn tracing on/off via a hypercall is 
definitely important

Rob