From: Chun-Tse Shao <ctshao@google.com>
To: linux-kernel@vger.kernel.org
Cc: Chun-Tse Shao <ctshao@google.com>,
peterz@infradead.org, mingo@redhat.com, acme@kernel.org,
namhyung@kernel.org, mark.rutland@arm.com,
alexander.shishkin@linux.intel.com, jolsa@kernel.org,
irogers@google.com, adrian.hunter@intel.com,
kan.liang@linux.intel.com, linux-perf-users@vger.kernel.org,
bpf@vger.kernel.org
Subject: [PATCH v5 0/5] Tracing contention lock owner call stack
Date: Wed, 12 Feb 2025 14:24:51 -0800 [thread overview]
Message-ID: <20250212222859.2086080-1-ctshao@google.com> (raw)
For perf lock contention, the current owner tracking (-o option) only
works with per-thread mode (-t option). Enabling call stack mode for
owner can be useful for diagnosing why a system running slow in
lock contention.
Example output:
$ sudo ~/linux/tools/perf/perf lock con -abvo -Y mutex -E16 perf bench sched pipe
...
contended total wait max wait avg wait type caller
171 1.55 ms 20.26 us 9.06 us mutex pipe_read+0x57
0xffffffffac6318e7 pipe_read+0x57
0xffffffffac623862 vfs_read+0x332
0xffffffffac62434b ksys_read+0xbb
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
36 193.71 us 15.27 us 5.38 us mutex pipe_write+0x50
0xffffffffac631ee0 pipe_write+0x50
0xffffffffac6241db vfs_write+0x3bb
0xffffffffac6244ab ksys_write+0xbb
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
4 51.22 us 16.47 us 12.80 us mutex do_epoll_wait+0x24d
0xffffffffac691f0d do_epoll_wait+0x24d
0xffffffffac69249b do_epoll_pwait.part.0+0xb
0xffffffffac693ba5 __x64_sys_epoll_pwait+0x95
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
2 20.88 us 11.95 us 10.44 us mutex do_epoll_wait+0x24d
0xffffffffac691f0d do_epoll_wait+0x24d
0xffffffffac693943 __x64_sys_epoll_wait+0x73
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
1 7.33 us 7.33 us 7.33 us mutex do_epoll_ctl+0x6c1
0xffffffffac692e01 do_epoll_ctl+0x6c1
0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
1 6.64 us 6.64 us 6.64 us mutex do_epoll_ctl+0x3d4
0xffffffffac692b14 do_epoll_ctl+0x3d4
0xffffffffac6937e0 __x64_sys_epoll_ctl+0x70
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
=== owner stack trace ===
3 31.24 us 15.27 us 10.41 us mutex pipe_read+0x348
0xffffffffac631bd8 pipe_read+0x348
0xffffffffac623862 vfs_read+0x332
0xffffffffac62434b ksys_read+0xbb
0xfffffffface604b2 do_syscall_64+0x82
0xffffffffad00012f entry_SYSCALL_64_after_hwframe+0x76
...
v5:
Move duplicated code into function.
Remove code to retrieve undesired callstack at the end of `contention_end()`.
Other minor fix based on Namhyung's review.
v4: lore.kernel.org/20250130052510.860318-1-ctshao@google.com
Use `__sync_fetch_and_add()` to generate owner stackid automatically.
Use `__sync_fetch_and_add(..., -1)` to workaround compiler error from
`__sync_fetch_and_sub()`
Remove unnecessary include headers.
Dedicate on C-style comment.
Other minor fix based on Namhyung's review.
v3: lore.kernel.org/20250129001905.619859-1-ctshao@google.com
Rename/shorten multiple variables.
Implement owner stackid.
Add description for lock function return code in `contention_end()`.
Other minor fix based on Namhyung's review.
v2: lore.kernel.org/20250113052220.2105645-1-ctshao@google.com
Fix logic deficit in v1 patch 2/4.
v1: lore.kernel.org/20250110051346.1507178-1-ctshao@google.com
Chun-Tse Shao (5):
perf lock: Add bpf maps for owner stack tracing
perf lock: Retrieve owner callstack in bpf program
perf lock: Make rb_tree helper functions generic
perf lock: Report owner stack in usermode
perf lock: Update documentation for -o option in contention mode
tools/perf/builtin-lock.c | 59 +++-
tools/perf/util/bpf_lock_contention.c | 68 ++++-
.../perf/util/bpf_skel/lock_contention.bpf.c | 251 +++++++++++++++++-
tools/perf/util/bpf_skel/lock_data.h | 7 +
tools/perf/util/lock-contention.h | 7 +
5 files changed, 367 insertions(+), 25 deletions(-)
--
2.48.1.502.g6dc24dfdaf-goog
next reply other threads:[~2025-02-12 22:29 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-02-12 22:24 Chun-Tse Shao [this message]
2025-02-12 22:24 ` [PATCH v5 1/5] perf lock: Add bpf maps for owner stack tracing Chun-Tse Shao
2025-02-12 22:24 ` [PATCH v5 2/5] perf lock: Retrieve owner callstack in bpf program Chun-Tse Shao
2025-02-12 22:24 ` [PATCH v5 3/5] perf lock: Make rb_tree helper functions generic Chun-Tse Shao
2025-02-12 22:24 ` [PATCH v5 4/5] perf lock: Report owner stack in usermode Chun-Tse Shao
2025-02-13 23:24 ` Namhyung Kim
2025-02-12 22:24 ` [PATCH v5 5/5] perf lock: Update documentation for -o option in contention mode Chun-Tse Shao
2025-02-13 23:28 ` Namhyung Kim
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20250212222859.2086080-1-ctshao@google.com \
--to=ctshao@google.com \
--cc=acme@kernel.org \
--cc=adrian.hunter@intel.com \
--cc=alexander.shishkin@linux.intel.com \
--cc=bpf@vger.kernel.org \
--cc=irogers@google.com \
--cc=jolsa@kernel.org \
--cc=kan.liang@linux.intel.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-perf-users@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mingo@redhat.com \
--cc=namhyung@kernel.org \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.