Issue with trace-cmd 3 and kernelshark

Issue with trace-cmd 3 and kernelshark

[Steven Rostedt]

I just tried the following with trace-cmd 3 on both the host and guest:

 guest> trace-cmd agent

 host> trace-cmd record -e kvm -A guest -e all ssh guest

[ logged in, did an 'ls' then logged out ]

I ended up with a trace.dat and trace-guest.dat

But when I did:

 kernelshark trace.dat -a trace-guest.dat

I ended up with a segfault here (in trace-input.c)

static struct page *allocate_page(struct tracecmd_input *handle,
				  int cpu, off64_t offset)
{
	struct cpu_data *cpu_data = &handle->cpu_data[cpu];
	struct page **pages;
	struct page *page;
	int index;

	index = (offset - cpu_data->file_offset) / handle->page_size;
	if (index >= cpu_data->nr_pages) {
		pages = realloc(cpu_data->pages, (index + 1) * sizeof(*cpu_data->pages));
		if (!pages)
			return NULL;
		memset(pages + cpu_data->nr_pages, 0,
		       (index + 1 - cpu_data->nr_pages) * sizeof(*cpu_data->pages));
		cpu_data->pages = pages;
		cpu_data->nr_pages = index + 1;
	}
	if (cpu_data->pages[index]) {
		cpu_data->pages[index]->ref_count++;  <<---- seg fault here!
		return cpu_data->pages[index];
	}

	page = malloc(sizeof(*page));
	if (!page)
		return NULL;


It appears that it doesn't like the fact that the file is compressed. This
was my worry about compressing date :-/

-- Steve

Re: Issue with trace-cmd 3 and kernelshark

[Steven Rostedt]

On Fri, 11 Mar 2022 22:18:25 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> It appears that it doesn't like the fact that the file is compressed. This
> was my worry about compressing date :-/

Actually, it doesn't look like this is an issue with compression, but the
fact that the cpu involved doesn't have any data, and we no longer save
anything for that case.

The problem data is here: 
  https://rostedt.org/private/trace-data/gentoo/trace.dat
  https://rostedt.org/private/trace-data/gentoo/trace-gentoo.dat

And do a: kernelshark trace.dat -a trace-gentoo.dat

-- Steve

Re: Issue with trace-cmd 3 and kernelshark

[Steven Rostedt]

On Fri, 11 Mar 2022 23:50:44 -0500
Steven Rostedt <rostedt@goodmis.org> wrote:

> On Fri, 11 Mar 2022 22:18:25 -0500
> Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > It appears that it doesn't like the fact that the file is compressed. This
> > was my worry about compressing date :-/  
> 
> Actually, it doesn't look like this is an issue with compression, but the
> fact that the cpu involved doesn't have any data, and we no longer save
> anything for that case.
> 
> The problem data is here: 
>   https://rostedt.org/private/trace-data/gentoo/trace.dat
>   https://rostedt.org/private/trace-data/gentoo/trace-gentoo.dat
> 
> And do a: kernelshark trace.dat -a trace-gentoo.dat
> 

Actually, it's just the trace.dat file that crashes in kernelshark.

As the number of CPUs that have data is only 35 (and thus handle->cpus is
just 35) but the box I ran this on has 128 CPUs.

There needs to be a generic way to map the actual CPU to the one saved in
the file. the cpu_data[cpu] won't work if the cpu_data array does not
contain CPUs that exist on the machine.

-- Steve