From: Phil Auld <pauld@redhat.com>
To: linux-trace-devel@vger.kernel.org
Cc: Yordan Karadzhov <ykaradzhov@vmware.com>,
Steven Rostedt <rostedt@goodmis.org>
Subject: trace-cmd fails with many cpus
Date: Fri, 12 Apr 2019 09:53:33 -0400 [thread overview]
Message-ID: <20190412135333.GB6201@pauld.bos.csb> (raw)
Hi,
I was trying to get some sched traces on a 160 cpu box yesterday. Trace-cmd
failed with
# ./tracecmd/trace-cmd record -e "sched:*" sleep 2
none
trace-cmd: Invalid argument
Failed filter of /sys/kernel/tracing/events/sched/sched_switch/filter
trace-cmd: No such file or directory
can not stat 'trace.dat.cpu0'
#
Which can be seen better with strace
[pid 97653] open("/sys/kernel/tracing/events/sched/sched_swap_numa/filter", O_WRONLY|O_TRUNC) = 5
[pid 97653] write(5, "(common_pid!=97652)&&(common_pid"..., 3358) = 3358
[pid 97653] close(5) = 0
[pid 97653] open("/sys/kernel/tracing/events/sched/sched_switch/filter", O_WRONLY|O_TRUNC) = 5
[pid 97653] write(5, "(common_pid!=97652)&&(common_pid"..., 6398) = -1 EINVAL (Invalid argument)
[pid 97653] close(5) = 0
The filter file can only take a max write of length PAGE_SIZE.
The extra pid filtering added for "next_pid" more or less doubles length
and pushes it over the 4k limit.
WRITE: /sys/kernel/tracing/events/sched/sched_switch/filter, len 6718, data "(common_pid!=100199)&&(common_pid!=100198)&&(common_pid!=100197)&&(common_pid!=100196)&&(common_pid!=100195)&&(common_pid!=100194)&&(common_pid!=100193)&&(common_pid!=100192)&&(common_pid!=100191) ... 160 of these ...
&&(common_pid!=100040)||(next_pid!=100199)&&(next_pid!=100198)&&(next_pid!=100197)&&(next_pid!=100196)&&(next_pid!=100195)&&(next_pid!=100194)&&(next_pid!=100193)&&(next_pid!=100192)... 160 of these...
I suppose the answer is don't run on a system with that many cpus :)
But I wonder if it would be possible to have the threads each handle say 8 cpu
files or something.
Or maybe have the kernel filter accept an "all_pid" that covered common_pid, next_pid, pid to reduce
the number of items needed in there?
Thanks,
Phil
--
next reply other threads:[~2019-04-12 13:53 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-12 13:53 Phil Auld [this message]
2019-04-12 15:15 ` trace-cmd fails with many cpus Steven Rostedt
2019-04-12 15:35 ` Steven Rostedt
2019-04-12 18:13 ` Phil Auld
2019-04-12 18:39 ` Steven Rostedt
2019-04-12 18:51 ` Steven Rostedt
2019-04-12 23:01 ` [PATCH] Slavomir Kaslev
2019-04-12 23:01 ` [PATCH] Slavomir Kaslev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190412135333.GB6201@pauld.bos.csb \
--to=pauld@redhat.com \
--cc=linux-trace-devel@vger.kernel.org \
--cc=rostedt@goodmis.org \
--cc=ykaradzhov@vmware.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).