trace events

trace events

[Marek Bykowski]

Hello cxl mailing list,

I just started a project for validating the RTL of the CXL root bridge
from Intel. One of my first goals is to better understand the
SW/protocol flows, so that we better validate it. I'm going to use
the tracpoints for it and possibly lttng. Are you considering adding
some of it shortly? If not would you be interested into the patches for
it?

On the list I'm seeing include/trace/events/cxl.h but on the repository
kernel/git/cxl/cxl.git seeing none for example with:

 git cat-file -e origin:include/trace/events/cxl.h

Thanks,
Marek

Re: trace events

[Alison Schofield]

On Mon, Nov 14, 2022 at 08:27:57PM +0000, Marek Bykowski wrote:
> Hello cxl mailing list,
> 
> I just started a project for validating the RTL of the CXL root bridge
> from Intel. One of my first goals is to better understand the
> SW/protocol flows, so that we better validate it. I'm going to use
> the tracpoints for it and possibly lttng. Are you considering adding
> some of it shortly? If not would you be interested into the patches for
> it?
> 
> On the list I'm seeing include/trace/events/cxl.h but on the repository
> kernel/git/cxl/cxl.git seeing none for example with:
> 
>  git cat-file -e origin:include/trace/events/cxl.h

Hi Marek,

There are a few patchsets under review that are using trace events
in CXL. I'm working on one for CXL Poison Retrieval.

There is also tool support, the monitor patchset, making its way
upstream for ndctl that uses the trace events.

I haven't looked at lttng.

Can you share more about your project?

Thanks,
Alison

> 
> Thanks,
> Marek

Re: trace events

[Marek Bykowski]

On Thu, 17 Nov 2022 16:50:56 -0800
Alison Schofield <alison.schofield@intel.com> wrote:

> There are a few patchsets under review that are using trace events
> in CXL. I'm working on one for CXL Poison Retrieval.
> 
> There is also tool support, the monitor patchset, making its way
> upstream for ndctl that uses the trace events.
> 
> I haven't looked at lttng.
> 
> Can you share more about your project?

Hi Alison,

There should be already plenty of tools using the trace events,
libtraceevent, libtrracefs and trace-cmd, lttng, but I guess
another one may be of use too. When do you think it should be out
for testing?

For the project, it is strictly secret, and I shouldn't reveal too
much publicly, as I was told at least, but what we do we basically
test a CXL root complex in RTL/verilog that instead of being synthesized,
gets compiled to a VCS sim (a simulator) that in turn gets
connected through the BFM (bus-function-model) to another simulator,
simics or qemu. The connection to another simulator is for having
the root complex to test it against something.

So to serve the purpose I need a good grip at what is happening at
a very low level, ideally with the CXL operation taken down to the
register access/mnemonics, especially throughout the enumeration
phase, which we think will the primary source of all the defects to be
caught up and fixed in RTL.

I hope our paths would cross at some point in which the HW will be
finding its way to Linux with all the support it requires.

Any tips/suggestions how I can use Linux for now effectively to
achieve my goals is highly appreciated.

Thanks,
Marek

Re: trace events

[Dave Jiang]

On 11/18/2022 6:53 AM, Marek Bykowski wrote:
> On Thu, 17 Nov 2022 16:50:56 -0800
> Alison Schofield <alison.schofield@intel.com> wrote:
> 
>> There are a few patchsets under review that are using trace events
>> in CXL. I'm working on one for CXL Poison Retrieval.
>>
>> There is also tool support, the monitor patchset, making its way
>> upstream for ndctl that uses the trace events.
>>
>> I haven't looked at lttng.
>>
>> Can you share more about your project?
> 
> Hi Alison,
> 
> There should be already plenty of tools using the trace events,
> libtraceevent, libtrracefs and trace-cmd, lttng, but I guess
> another one may be of use too. When do you think it should be out
> for testing?
> 
> For the project, it is strictly secret, and I shouldn't reveal too
> much publicly, as I was told at least, but what we do we basically
> test a CXL root complex in RTL/verilog that instead of being synthesized,
> gets compiled to a VCS sim (a simulator) that in turn gets
> connected through the BFM (bus-function-model) to another simulator,
> simics or qemu. The connection to another simulator is for having
> the root complex to test it against something.
> 
> So to serve the purpose I need a good grip at what is happening at
> a very low level, ideally with the CXL operation taken down to the
> register access/mnemonics, especially throughout the enumeration
> phase, which we think will the primary source of all the defects to be
> caught up and fixed in RTL.
> 
> I hope our paths would cross at some point in which the HW will be
> finding its way to Linux with all the support it requires.
> 
> Any tips/suggestions how I can use Linux for now effectively to
> achieve my goals is highly appreciated.

Hi Marek. Have you looked into enabling ftrace? Maybe that would be the 
right starting point for what you are looking for?

https://lwn.net/Articles/365835/

> 
> Thanks,
> Marek

Re: trace events

[Marek Bykowski]

On Fri, 18 Nov 2022 08:18:43 -0700
Dave Jiang <dave.jiang@intel.com> wrote:


> 
> Hi Marek. Have you looked into enabling ftrace? Maybe that would be
> the right starting point for what you are looking for?
> 
> https://lwn.net/Articles/365835/
> 

That was my initial idea but if there are no trace events defined for
the CXL what am I to enable? At least I cannot find any events for CXL,
there is no file include/trace/events/cxl.h in kernel/git/cxl/cxl.git.
Also

  find /sys/kernel/debug/tracing/events/ -name "*cxl*"

on linux-cxl returns nothing.

Alison mentioned she is working on some but these are yet in review.

As putting the trace events throughput the CXL driver/s takes time I
took a shortcut and added trace_printk scattered around for now. With
libtraceevent, trace-cmd and kernelshark and it is looking fantastic,
except it is not the CXL events as per-se.

root@localhost:~# trace-cmd show |tail -n 2
   systemd-udevd-520 [004] ..... 21.236650: devm_cxl_enumerate_decoders: \
			success adding decoder to port endpoint5
   systemd-udevd-520 [004] ..... 21.240517: devm_cxl_enumerate_ports.cold: \
			find_cxl_port() returned port port2

Re: trace events

[Dave Jiang]

On 11/18/2022 3:39 PM, Marek Bykowski wrote:
> On Fri, 18 Nov 2022 08:18:43 -0700
> Dave Jiang <dave.jiang@intel.com> wrote:
> 
> 
>>
>> Hi Marek. Have you looked into enabling ftrace? Maybe that would be
>> the right starting point for what you are looking for?
>>
>> https://lwn.net/Articles/365835/
>>
> 
> That was my initial idea but if there are no trace events defined for
> the CXL what am I to enable? At least I cannot find any events for CXL,
> there is no file include/trace/events/cxl.h in kernel/git/cxl/cxl.git.
> Also
> 
>    find /sys/kernel/debug/tracing/events/ -name "*cxl*"
> 
> on linux-cxl returns nothing.
> 
> Alison mentioned she is working on some but these are yet in review.
> 
> As putting the trace events throughput the CXL driver/s takes time I
> took a shortcut and added trace_printk scattered around for now. With
> libtraceevent, trace-cmd and kernelshark and it is looking fantastic,
> except it is not the CXL events as per-se.
> 
> root@localhost:~# trace-cmd show |tail -n 2
>     systemd-udevd-520 [004] ..... 21.236650: devm_cxl_enumerate_decoders: \
> 			success adding decoder to port endpoint5
>     systemd-udevd-520 [004] ..... 21.240517: devm_cxl_enumerate_ports.cold: \
> 			find_cxl_port() returned port port2

For kernel we are enabling trace events for CXL. But those events are 
stemmed from hardware events. Ira is enabling CXL events defined by the 
CXL spec and emitted by the devices. Alison is enabling CXL events for 
poison data. And I'm enabling CXL events for PCIe AER errors.

But I think you want different type of events right? i.e. the flow of 
the driver? I believe ftrace can enable function names for ftrace. Is 
that what you are looking for?

https://opensource.com/article/21/7/linux-kernel-ftrace
https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/
https://lwn.net/Articles/370423/

Re: trace events

[Marek Bykowski]

On Fri, 18 Nov 2022 16:22:26 -0700
Dave Jiang <dave.jiang@intel.com> wrote:

> 
> For kernel we are enabling trace events for CXL. But those events are 
> stemmed from hardware events. Ira is enabling CXL events defined by
> the CXL spec and emitted by the devices. Alison is enabling CXL
> events for poison data. And I'm enabling CXL events for PCIe AER
> errors.
> 
> But I think you want different type of events right? i.e. the flow of 
> the driver? I believe ftrace can enable function names for ftrace. Is 
> that what you are looking for?

Apology Dave, my bad. I'm so used to into people saying ftrace
in which they really mean the trace events that I thought you referred
to the latter, which of course isn't true as the two are separate.

Honestly I've never found a real use of the ftrace, except exercising,
as to me it is more a call trace and how a developer designs the SW
rather than how the HW is managed/interogated, but in the absence of the
trace events I re-thought and went down that path.

After a several attempts and finally setting the 'trace_clock'
to global I succeeded. With the 'trace_clock' set local (default)
I had only the last ftrace events logged in - don't know really why.
Actually the ./tools/bootconfig/scripts/ftrace2bconf.sh figured that
out for me walking through the ftrace settings I had configured after
booting the Linux up.

So now with a bootconfig as this
  kernel { trace_options =  sym-addr, hash-ptr, nofunc_stack_trace,
                   \nofunc-no-repeats 
           #trace_event = "module:*"
           trace_clock = global
           trace_buf_size = 2M
           ftrace = function_graph
           max_graph_depth = 5
           ftrace_filter = "*cxl*"
  } 

I'm seeing all the functions with *cxl*.

Thanks again.
Marek