sdt provider and access to the trace_event_raw_* struct

sdt provider and access to the trace_event_raw_* struct

[Alan Maguire]

hi folks

I've come across a case where I need to trace a kernel tracepoint with a
lot of associated trace info.  It seems that the current approach for
sdt probes looks at the "struct trace_event_raw_<tracepoint_name>"
structure and maps its fields into args[] values, translating each
member into a separate argument.  That works great for tracepoints with
a limited number of fields. However in the case of a tracepoint with a
lot of such fields (i.e. more than the number of args[] supported), it
would be useful to also have a convenient way to access the raw "struct
trace_event_raw_*" data, especially since we have access to it directly
via CTF. It's possible to do this via a hack, e.g. the following works:

#!/usr/sbin/dtrace -s

sdt:sched::sched_switch
{
        s = (struct trace_event_raw_sched_switch *)(arg0-8);
        print(s);
}


...but presumably that only works because the first arg value isn't
scalar. It would be good to have a helper or builtin variable to access
this pointer directly. Maybe there's a better way to do this, or maybe
we could add a helper/builtin to make this pointer accessible? What do
folks think?

Thanks!

Alan

Re: sdt provider and access to the trace_event_raw_* struct

[Kris Van Hees]

On Fri, Oct 04, 2024 at 12:29:35PM +0100, Alan Maguire wrote:
> hi folks
> 
> I've come across a case where I need to trace a kernel tracepoint with a
> lot of associated trace info.  It seems that the current approach for
> sdt probes looks at the "struct trace_event_raw_<tracepoint_name>"
> structure and maps its fields into args[] values, translating each
> member into a separate argument.  That works great for tracepoints with
> a limited number of fields. However in the case of a tracepoint with a
> lot of such fields (i.e. more than the number of args[] supported), it
> would be useful to also have a convenient way to access the raw "struct
> trace_event_raw_*" data, especially since we have access to it directly
> via CTF. It's possible to do this via a hack, e.g. the following works:

You should be able to use the raw tracepoint provider, rawtp,
e.g. rawtp:sched::sched_switch

> #!/usr/sbin/dtrace -s
> 
> sdt:sched::sched_switch
> {
>         s = (struct trace_event_raw_sched_switch *)(arg0-8);
>         print(s);
> }
> 
> 
> ...but presumably that only works because the first arg value isn't
> scalar. It would be good to have a helper or builtin variable to access
> this pointer directly. Maybe there's a better way to do this, or maybe
> we could add a helper/builtin to make this pointer accessible? What do
> folks think?
> 
> Thanks!
> 
> Alan

Re: sdt provider and access to the trace_event_raw_* struct

[Alan Maguire]

On 04/10/2024 15:29, Kris Van Hees wrote:
> On Fri, Oct 04, 2024 at 12:29:35PM +0100, Alan Maguire wrote:
>> hi folks
>>
>> I've come across a case where I need to trace a kernel tracepoint with a
>> lot of associated trace info.  It seems that the current approach for
>> sdt probes looks at the "struct trace_event_raw_<tracepoint_name>"
>> structure and maps its fields into args[] values, translating each
>> member into a separate argument.  That works great for tracepoints with
>> a limited number of fields. However in the case of a tracepoint with a
>> lot of such fields (i.e. more than the number of args[] supported), it
>> would be useful to also have a convenient way to access the raw "struct
>> trace_event_raw_*" data, especially since we have access to it directly
>> via CTF. It's possible to do this via a hack, e.g. the following works:
> 
> You should be able to use the raw tracepoint provider, rawtp,
> e.g. rawtp:sched::sched_switch
>

That's a good help, but I should have clarified that I was hoping for a
way to get the tracepoint data _after_ it has been massaged into the
tracepoint form; the above will give me access to the raw arguments that
are used in tracepoint data setup, but I was hoping to have a way to get
a pointer to the entire trace structure after it has been assigned. It's
doable in my case (since the first parameter is always a reference) so
not a massive deal, but it might be useful enhancement for others.

Thanks!

Alan

>> #!/usr/sbin/dtrace -s
>>
>> sdt:sched::sched_switch
>> {
>>         s = (struct trace_event_raw_sched_switch *)(arg0-8);
>>         print(s);
>> }
>>
>>
>> ...but presumably that only works because the first arg value isn't
>> scalar. It would be good to have a helper or builtin variable to access
>> this pointer directly. Maybe there's a better way to do this, or maybe
>> we could add a helper/builtin to make this pointer accessible? What do
>> folks think?
>>
>> Thanks!
>>
>> Alan

Re: sdt provider and access to the trace_event_raw_* struct

[Kris Van Hees]

On Fri, Oct 04, 2024 at 04:46:58PM +0100, Alan Maguire wrote:
> On 04/10/2024 15:29, Kris Van Hees wrote:
> > On Fri, Oct 04, 2024 at 12:29:35PM +0100, Alan Maguire wrote:
> >> hi folks
> >>
> >> I've come across a case where I need to trace a kernel tracepoint with a
> >> lot of associated trace info.  It seems that the current approach for
> >> sdt probes looks at the "struct trace_event_raw_<tracepoint_name>"
> >> structure and maps its fields into args[] values, translating each
> >> member into a separate argument.  That works great for tracepoints with
> >> a limited number of fields. However in the case of a tracepoint with a
> >> lot of such fields (i.e. more than the number of args[] supported), it
> >> would be useful to also have a convenient way to access the raw "struct
> >> trace_event_raw_*" data, especially since we have access to it directly
> >> via CTF. It's possible to do this via a hack, e.g. the following works:
> > 
> > You should be able to use the raw tracepoint provider, rawtp,
> > e.g. rawtp:sched::sched_switch
> >
> 
> That's a good help, but I should have clarified that I was hoping for a
> way to get the tracepoint data _after_ it has been massaged into the
> tracepoint form; the above will give me access to the raw arguments that
> are used in tracepoint data setup, but I was hoping to have a way to get
> a pointer to the entire trace structure after it has been assigned. It's
> doable in my case (since the first parameter is always a reference) so
> not a massive deal, but it might be useful enhancement for others.

Can you give an example of where it goes wrong?  I don't see a reason why we
wouldn't be able to support more than the number of arguuments that we store
by default.  I.e. I do think that there is a limitation roght now, but I don't
think there is a hard reason for that.  We ought to be able to support access
to all arguments of the probe without much extra effort.

> Thanks!
> 
> Alan
> 
> >> #!/usr/sbin/dtrace -s
> >>
> >> sdt:sched::sched_switch
> >> {
> >>         s = (struct trace_event_raw_sched_switch *)(arg0-8);
> >>         print(s);
> >> }
> >>
> >>
> >> ...but presumably that only works because the first arg value isn't
> >> scalar. It would be good to have a helper or builtin variable to access
> >> this pointer directly. Maybe there's a better way to do this, or maybe
> >> we could add a helper/builtin to make this pointer accessible? What do
> >> folks think?
> >>
> >> Thanks!
> >>
> >> Alan

Re: sdt provider and access to the trace_event_raw_* struct

[Alan Maguire]

On 04/10/2024 20:22, Kris Van Hees wrote:
> On Fri, Oct 04, 2024 at 04:46:58PM +0100, Alan Maguire wrote:
>> On 04/10/2024 15:29, Kris Van Hees wrote:
>>> On Fri, Oct 04, 2024 at 12:29:35PM +0100, Alan Maguire wrote:
>>>> hi folks
>>>>
>>>> I've come across a case where I need to trace a kernel tracepoint with a
>>>> lot of associated trace info.  It seems that the current approach for
>>>> sdt probes looks at the "struct trace_event_raw_<tracepoint_name>"
>>>> structure and maps its fields into args[] values, translating each
>>>> member into a separate argument.  That works great for tracepoints with
>>>> a limited number of fields. However in the case of a tracepoint with a
>>>> lot of such fields (i.e. more than the number of args[] supported), it
>>>> would be useful to also have a convenient way to access the raw "struct
>>>> trace_event_raw_*" data, especially since we have access to it directly
>>>> via CTF. It's possible to do this via a hack, e.g. the following works:
>>>
>>> You should be able to use the raw tracepoint provider, rawtp,
>>> e.g. rawtp:sched::sched_switch
>>>
>>
>> That's a good help, but I should have clarified that I was hoping for a
>> way to get the tracepoint data _after_ it has been massaged into the
>> tracepoint form; the above will give me access to the raw arguments that
>> are used in tracepoint data setup, but I was hoping to have a way to get
>> a pointer to the entire trace structure after it has been assigned. It's
>> doable in my case (since the first parameter is always a reference) so
>> not a massive deal, but it might be useful enhancement for others.
> 
> Can you give an example of where it goes wrong?  I don't see a reason why we
> wouldn't be able to support more than the number of arguuments that we store
> by default.  I.e. I do think that there is a limitation roght now, but I don't
> think there is a hard reason for that.  We ought to be able to support access
> to all arguments of the probe without much extra effort.
>

sure; the RDS tracepoints are one example where we have a lot of fields.
For example the RDS state change tracepoints have trace structures like
this:


struct trace_event_raw_rds_state {
        struct trace_entry ent;
        __u8 laddr[16];
        __u8 faddr[16];
        __u8 tos;
        unsigned int transport;
        __u16 lport;
        __u16 fport;
        __u64 netns_inum;
        __u32 qp_num;
        __u32 remote_qp_num;
        long unsigned int flags;
        int err;
        char reason[64];
        __u64 cgroup_id;
        void *cgroup;
        void *rm;
        void *rs;
        void *conn;
        void *cp;
        int last;
        int curr;
        char __data[0];
};

So there's 20 fields there, which is greater than the number of
currently supported args[]. I tried the following

$ sudo dtrace -n 'sdt:rds::rds_state_change { printf("state %d\n",
args[18]); }'
DTrace 2.0.0 [Pre-Release with limited functionality]
dtrace: description 'sdt:rds::rds_state_change ' matched 1 probe
dtrace: error on enabled probe ID 2 (ID 120521:
sdt:rds::rds_state_change): illegal operation in action #1 at BPF pc 348

Experimentation reveals args[0]..args[9] work, but anything beyond that
triggers the above. Thanks!

Alan

>> Thanks!
>>
>> Alan
>>
>>>> #!/usr/sbin/dtrace -s
>>>>
>>>> sdt:sched::sched_switch
>>>> {
>>>>         s = (struct trace_event_raw_sched_switch *)(arg0-8);
>>>>         print(s);
>>>> }
>>>>
>>>>
>>>> ...but presumably that only works because the first arg value isn't
>>>> scalar. It would be good to have a helper or builtin variable to access
>>>> this pointer directly. Maybe there's a better way to do this, or maybe
>>>> we could add a helper/builtin to make this pointer accessible? What do
>>>> folks think?
>>>>
>>>> Thanks!
>>>>
>>>> Alan