Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[Steven Rostedt]

On Mon, 18 Dec 2017 12:24:12 +0100
David Hildenbrand <david@redhat.com> wrote:

> If recording a big number of events to a big buffer, one might want to
> minimize the trace-cmd activity for a certain period in time.
> Especially, don't see any trace-cmd activity for some time. If there are
> a lot of events, the loop might actually never sleep, resulting in the
> "-s" option never becomming active.
> 
> E.g. I am using scheduler events to trace thread activity per CPU. Esp.
> sched:sched_wakeup and sched:sched_stat_runtime. Even when setting the
> sleep time to something huge like 30 seconds, I can see constant activity
> from the trace-cmd tasks.

Thanks for reporting this. Honestly, I believe this is fixing a symptom
of a real problem. I'd like to investigate why trace-cmd is running
with a sleep time of 30 seconds. I think that should be fixed.

-- Steve

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[David Hildenbrand]

On 18.12.2017 16:41, Steven Rostedt wrote:
> On Mon, 18 Dec 2017 12:24:12 +0100
> David Hildenbrand <david@redhat.com> wrote:
> 
>> If recording a big number of events to a big buffer, one might want to
>> minimize the trace-cmd activity for a certain period in time.
>> Especially, don't see any trace-cmd activity for some time. If there are
>> a lot of events, the loop might actually never sleep, resulting in the
>> "-s" option never becomming active.
>>
>> E.g. I am using scheduler events to trace thread activity per CPU. Esp.
>> sched:sched_wakeup and sched:sched_stat_runtime. Even when setting the
>> sleep time to something huge like 30 seconds, I can see constant activity
>> from the trace-cmd tasks.
> 
> Thanks for reporting this. Honestly, I believe this is fixing a symptom
> of a real problem. I'd like to investigate why trace-cmd is running
> with a sleep time of 30 seconds. I think that should be fixed.
> 

The problem seems to be that we only sleep if we haven't read something
in the last iteration (!read), hindering us to sleep if there are many
events. Especially on the first run.

If tracing e.g. the scheduler, after each iteration there will already
be new data to pick up in the buffer, resulting in the thread never
going to sleep.

What I need: Start tracing and flush all buffers when exiting. (e.g.
after 30 seconds). Never wakeup in between, so the real trace overhead
in that period of time is purely storing the tracepoints to the buffer
in the kernel. Of course we could implement something like that ("copy
from the buffer only when exiting") or try to see if we can fix the
existing "-s" flag in a way that allows it.

Thanks!

> -- Steve
> 


-- 

Thanks,

David / dhildenb

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[Steven Rostedt]

On Mon, 18 Dec 2017 17:23:24 +0100
David Hildenbrand <david@redhat.com> wrote:

> What I need: Start tracing and flush all buffers when exiting. (e.g.

Why don't you use "trace-cmd start" and "trace-cmd extract"?

"trace-cmd record" is all about not losing events. If you are creating
a big buffer, then I think you want to use this.

# trace-cmd start -p <tracer> -e <events> -b <big-buffer-size>
# run test
# trace-cmd stop
# trace-cmd extract

Wouldn't that work for you?

-- Steve

> after 30 seconds). Never wakeup in between, so the real trace overhead
> in that period of time is purely storing the tracepoints to the buffer
> in the kernel. Of course we could implement something like that ("copy
> from the buffer only when exiting") or try to see if we can fix the
> existing "-s" flag in a way that allows it.

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[David Hildenbrand]

On 18.12.2017 18:52, Steven Rostedt wrote:
> On Mon, 18 Dec 2017 17:23:24 +0100
> David Hildenbrand <david@redhat.com> wrote:
> 
>> What I need: Start tracing and flush all buffers when exiting. (e.g.
> 
> Why don't you use "trace-cmd start" and "trace-cmd extract"?
> 
> "trace-cmd record" is all about not losing events. If you are creating
> a big buffer, then I think you want to use this.
> 
> # trace-cmd start -p <tracer> -e <events> -b <big-buffer-size>
> # run test
> # trace-cmd stop
> # trace-cmd extract
> 
> Wouldn't that work for you?

It works for some scenarios I have in mind. Especially when recording
long runs, it might be beneficial to e.g. wakeup every 30 seconds to
just write out a couple of MB of traces (compared to right now -s
30000000 waking up every couple of milliseconds).

With start/stop/extract the downside is, that buffers have to be huge
for longer runs.

I'll have a try tomorrow if I I''lose events with 20MB buffers per CPU
when recording more than 60 seconds (on a very active system with
mentioned scheduler rtaces being turned on).

Thanks!

> 
> -- Steve
> 
>> after 30 seconds). Never wakeup in between, so the real trace overhead
>> in that period of time is purely storing the tracepoints to the buffer
>> in the kernel. Of course we could implement something like that ("copy
>> from the buffer only when exiting") or try to see if we can fix the
>> existing "-s" flag in a way that allows it.


-- 

Thanks,

David / dhildenb

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[Steven Rostedt]

On Mon, 18 Dec 2017 21:56:33 +0100
David Hildenbrand <david@redhat.com> wrote:
 
> I'll have a try tomorrow if I I''lose events with 20MB buffers per CPU
> when recording more than 60 seconds (on a very active system with
> mentioned scheduler rtaces being turned on).
>

Another option is to add a '-w' option to record that causes the
recorders (what reads the data) to "wait" or force it to sleep even if
it did read something. I'm fine with that.

-- Steve

Re: [PATCH v1] trace-cmd: introduce --initital-delay for record command

[David Hildenbrand]

On 18.12.2017 22:43, Steven Rostedt wrote:
> On Mon, 18 Dec 2017 21:56:33 +0100
> David Hildenbrand <david@redhat.com> wrote:
>  
>> I'll have a try tomorrow if I I''lose events with 20MB buffers per CPU
>> when recording more than 60 seconds (on a very active system with
>> mentioned scheduler rtaces being turned on).
>>
> 
> Another option is to add a '-w' option to record that causes the
> recorders (what reads the data) to "wait" or force it to sleep even if
> it did read something. I'm fine with that.

Also thought about that. Might make sense (and makes -s actually do what
it advertises even when there are a lot of events coming in).

Thanks Steve

> 
> -- Steve
> 


-- 

Thanks,

David / dhildenb