Re: [PATCH v6 3/4] perf inject: add jitdump mmap injection support

Stephane Eranian
2015-04-13T00:37:27Z

On Fri, Apr 10, 2015 at 5:51 AM, Adrian Hunter <adrian.hunter@intel.com> wrote:
> On 08/04/15 17:12, Stephane Eranian wrote:
>> On Wed, Apr 8, 2015 at 5:15 AM, Adrian Hunter <adrian.hunter@intel.com> wrote:
>>> On 06/04/15 22:41, Stephane Eranian wrote:
>>>> > + if (inject.build_ids) {
>>>> > + /*
>>>> > + * to make sure the mmap records are ordered correctly
>>>> > + * and so that the correct especially due to jitted code
>>>> > + * mmaps. We cannot generate the buildid hit list and
>>>> > + * inject the jit mmaps at the same time for now.
>>>> > + */
>>>> > + inject.tool.ordered_events = true;
>>>> > + inject.tool.ordering_requires_timestamps = true;
>>>> > + }
>>>> > +
>>>> > + if (inject.jit_mode) {
>>>> > + inject.tool.mmap2 = perf_event__repipe_mmap2;
>>>> > + inject.tool.mmap = perf_event__repipe_mmap;
>>>>
>>>> As suggested above, why not make your own tool fns e.g.
>>>>
>>>> inject.tool.mmap2 = perf_event__jit_mode_mmap2;
>>>> inject.tool.mmap = perf_event__jit_mode_mmap;
>>>>
>>>>
>>>> > + inject.tool.ordered_events = true;
>>>> > + inject.tool.ordering_requires_timestamps = true;
>>>>
>>>> You are taking advantage of a bug in perf-inject, that is the
>>>> "finished_round" events are not being processed. Really they should be
>>>> processed and you should inject in time order.
>>>>
>>>> I am not sure I understand.
>>>> Yes, I am trying to have inject reorder the samples instead of perf report.
>>>> You are likely to run inject once, and report many times. Also avoids a
>>>> warning in report about out-of-order events.
>>>
>>> Well forgetting about "finished_round", it seems to me you need to intercept
>>> all the delivered events (which will be in time order) and inject your own
>>> events at the right time.
>>>
>>> At the moment it seems to me you are injecting all your events in one go
>>> when you see the special jitdump mmap. So I would not expect the injected
>>> events to be ordered with respect to other events that come later. But
>>> maybe I misunderstand that?
>>>
>> My understanding is that if I set ordered_events = true, then events
>> are not delivered immediately to the callbacks. They are queued and
>> sorted and then passed to the callbacks. And yes, there is the finished
>> round mechanism of which I don't quite fully understand the logic in this
>> case.
>>
>>> As I confusingly tried to suggest earlier, one way to see all the
>>> delivered events is to hook the ordered_events "deliver" callback. That
>>> will mean injecting one mmap event at a time.
>>>
>>> Here is just an idea.
>>>
>>> struct perf_inject {
>>> ...
>>> ordered_events__deliver_t deliver;
>>> };
>>>
>>> int cmd_inject(int argc, const char **argv, const char *prefix __maybe_unused)
>>> {
>>> ...
>>> inject.deliver = inject.session->ordered_events.deliver;
>>> inject.session->ordered_events.deliver = inject_jit_mmap;
>>> ...
>>> }
>>>
>> ok on that.
>>
>>> int inject_jit_mmap(struct ordered_events *oe, struct ordered_event *event)
>>> {
>>> struct perf_session *session = container_of(oe, struct perf_session, ordered_events);
>>> struct perf_inject *inject = container_of(session->tool, struct perf_inject, tool);
>>>
>>> /* Is it time to inject an event */
>>> if (jit_next_timestamp(inject) < event->timestamp) {
>>> /* Yes, so inject it by delivery */
>>> perf_session__deliver_synth_event(...);
>>> }
>>> inject->deliver(oe, event);
>>> }
>>>
>> That suggests I have buffered all the MMAPs synthesized from the jitdump and
>> I have some sorted queue based on jitdump timestamps.
>
> Yes, but won't the jitdump timestamps already be in order?
>
I looked at implementing your approach and it turns out to be much
more complicated
than what is already there.
I do not inject anything if I don't have to. I am waiting to see a
special MMAP of the
jitdump file to start inject jit mmaps. That could not change in your
scheme. I would
need to track MMAP records and check if they correspond to a jitdump. Note that
I may have to inject multiple jitdumps in one run of perf inject. In
that case, in the
deliver function, I would have to (1) either have all the jitdump MMAPs prepared
in order for all the jitdumps I have detected or (2) go through all
the jitdump current
head records to see which one matches. In (2) I would have to do this
not just when
I see an MMAP but for any event delivered: for each new event, compare its
timestamp to the timestamps of jitdump current record for ALL jitdump
files. For (1),
I need to build a global list of all jitdump records in for each
jitdump file, i.e, another
RB tree. And then pull them one by one out if I find an event with the
right timestamp.

I think (2) i more reasonable but requires quite some infrastructure
work in the jitdump
code. Most of the pieces are there, the RB tree needs to be added
obviously. I will
experiment with (2).

>> The test would have to
>> be more sophisticated that this. You'd want to insert at the right
>> time, i.e., you'd
>> have to track the previous and next timestamp in the stream:
>>
>> retry:
>> j = jit_next_timestamp(inject->jit);
>> if (prev_event->timestamp <= j && j > next_event->timestamp) {
>
> (prev_event->timestamp <= j) must be always true because otherwise you would
> have injected the jit mmap event already.
>
> So perf is about to deliver an event with timestamp 'event->timestamp', so
> you just need to deliver all your events that have not already been
> delivered but have a timestamp < event->timestamp
>
>
>> deliver_synth_event(inject->jit);
>> jit_next(inject->jit);
>> goto retry;
>> }
>>
>> All of this is required because the finished round logic is such that perf
>> cannot sort all records at once.
>
> It can sort them all at once, but finished_round is an optimization that
> starts delivering "early" events before having to sort the "later" events.
> The result is still that the delivered events are in sorted order.
>
>> It does sort them by chunks. Thus I need
>> to make sure I inject in the right chunk otherwise it won't work. Am I reading
>> this right?
>
> If you were to queue the events for sorting, yes. What I suggested was
> delivering your events directly at the "right time".
>
>>
>>> You would also need to inject any remaining events right at the end.
>>>
>>> Note the advantage of "delivering" events is that the same approach works whatever the tool.
>>>
>>>
>>
>>
>