[PATCH v2 0/5] perf: Add CoreSight branch history to existing samples

[PATCH v2 0/5] perf: Add CoreSight branch history to existing samples

[Amir Ayupov]

CoreSight ETM can synthesize branch samples from an instruction trace, but
context-sensitive PGO needs the branch history leading to an existing PMU
sample together with that sample's event identity and callchain. This series
implements that mode as --itrace=L, following the corresponding Intel PT
behavior.

The series first separates timestamped queue setup and teardown from the ETM
decode loop so decoding can stop at an existing sample's timestamp. It then
reconstructs branch history in the thread stack and attaches it to eligible
samples without replacing their IP, event, or callchain. The attached history
is consumed after use so a later sample cannot reuse an earlier trace window.

This enables a context-sensitive PGO workflow where a cycles event supplies a
frame-pointer callchain while duty-cycled ETM supplies the path leading to the
sample. A dlfilter removes samples for which no ETM history was available, and
the documentation describes the complete recording and decoding workflow.

Changes since v1:

  - Rebased onto perf-tools-next at d17c5b770972.
  - Dropped the HEADER_GROUP_DESC reader workaround. The issue is in the writer
    and should be fixed separately.
  - Dropped the branch-stack hw_idx patch after review established that zero is
    appropriate for age-ordered CoreSight branch stacks.
  - Dropped the local wrapped branch-stack copy fix in favor of upstream commit
    ab9c84d1cd59 ("perf thread-stack: Fix heap buffer overflow on branch stack
    wrap copy").
  - Added James Clark's Reviewed-by tag to the dlfilter patch.
  - Consume branch history after attaching it so samples with no newly decoded
    trace cannot reuse a window from before an untraced AUX pause interval.
  - Flush all trace-ID frontend thread stacks when their physical ETM queue is
    exhausted, preventing stale history from surviving a trace gap.
  - Check every matching CoreSight event when deciding whether kernel trace is
    enabled.
  - Reworked the shell test to use FIFO recording control, removed the invalid
    bare timestamp option, reduced the workload to 10000 iterations, and check
    proc1 and proc2 callchains independently with bounded failure diagnostics.
  - Renamed decoded test outputs to script-L4 and script-L64.

Amir Ayupov (5):
  perf dlfilter: Add non-empty branch stack filter
  perf cs-etm: Split up cs_etm__process_timestamped_queues()
  perf cs-etm: Add branch history to existing samples
  perf test cs-etm: Test branch history on existing samples
  Documentation: coresight: Document context-sensitive PGO workflow

 .../trace/coresight/coresight-perf.rst        |  62 +++++
 tools/perf/Makefile.perf                      |   1 +
 .../dlfilters/dlfilter-nonempty-brstack.c     |  26 ++
 .../tests/shell/coresight/add_last_branch.sh  | 203 ++++++++++++++
 tools/perf/util/cs-etm.c                      | 252 ++++++++++++++++--
 tools/perf/util/thread-stack.c                |  17 ++
 tools/perf/util/thread-stack.h                |   1 +
 7 files changed, 546 insertions(+), 16 deletions(-)
 create mode 100644 tools/perf/dlfilters/dlfilter-nonempty-brstack.c
 create mode 100755 tools/perf/tests/shell/coresight/add_last_branch.sh


base-commit: d17c5b770972854a4fe4cf5cc22e17eb21cdc787
-- 
2.52.0

[PATCH v2 1/5] perf dlfilter: Add non-empty branch stack filter

[Amir Ayupov]

--itrace=L adds decoded branch history to existing samples, but a sample
that was recorded while the decoder had no trace for that thread keeps an
empty branch stack. Consumers of the resulting perf script output, such
as profile generators for context-sensitive PGO, have no use for those
samples.

Add an opt-in dlfilter that drops samples whose parsed branch stack is
empty, so users can exclude them without changing default sample
semantics. Build and install it alongside perf's existing dlfilters.

Assisted-by: Devmate:GPT-5.6
Signed-off-by: Amir Ayupov <aaupov@fb.com>
Reviewed-by: James Clark <james.clark@linaro.org>
---
 tools/perf/Makefile.perf                      |  1 +
 .../dlfilters/dlfilter-nonempty-brstack.c     | 26 +++++++++++++++++++
 2 files changed, 27 insertions(+)
 create mode 100644 tools/perf/dlfilters/dlfilter-nonempty-brstack.c

diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 29cfd44c427f3..750bd1cce1287 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -425,6 +425,7 @@ PROGRAMS += $(OUTPUT)$(LIBJVMTI)
 endif
 
 DLFILTERS := dlfilter-test-api-v0.so dlfilter-test-api-v2.so dlfilter-show-cycles.so
+DLFILTERS += dlfilter-nonempty-brstack.so
 DLFILTERS := $(patsubst %,$(OUTPUT)dlfilters/%,$(DLFILTERS))
 
 # what 'all' will build and 'install' will install, in perfexecdir
diff --git a/tools/perf/dlfilters/dlfilter-nonempty-brstack.c b/tools/perf/dlfilters/dlfilter-nonempty-brstack.c
new file mode 100644
index 0000000000000..9e66205b841d5
--- /dev/null
+++ b/tools/perf/dlfilters/dlfilter-nonempty-brstack.c
@@ -0,0 +1,26 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * dlfilter-nonempty-brstack.c: Filter out samples with no branch stack
+ * Copyright (c) 2026, Meta Platforms, Inc.
+ */
+#include <stddef.h>
+
+#include <perf/perf_dlfilter.h>
+
+int filter_event(void *data, const struct perf_dlfilter_sample *sample, void *ctx)
+{
+	/* Return 1 to filter out the sample, 0 to keep it */
+	return !sample->brstack_nr;
+}
+
+const char *filter_description(const char **long_description)
+{
+	static char *long_desc =
+		"Instruction trace decoders can add branch history to existing "
+		"samples, but samples that were recorded while no trace was "
+		"being collected get an empty branch stack. Filter those out so "
+		"that only samples carrying branch history remain.";
+
+	*long_description = long_desc;
+	return "Keep only samples with a non-empty branch stack";
+}
-- 
2.52.0

[PATCH v2 2/5] perf cs-etm: Split up cs_etm__process_timestamped_queues()

[Amir Ayupov]

cs_etm__process_timestamped_queues() currently does three things: it seeds
the auxtrace heap with one entry per queue, it decodes until the heap is
empty, and it then walks every traceID queue to flush whatever is left in
the branch stacks. That is fine while the only caller is
cs_etm__flush_events(), which runs once, but it does not survive the
function being called repeatedly.

Seeding cannot be repeated because a queue that still holds a heap slot
would be seeded again, adding duplicate entries and growing the heap
without bound. Flushing cannot be repeated either, because ending a block
finalises state that later trace still needs.

Move both out. Seeding becomes cs_etm__update_queues(), gated on
queues.new_data so it only runs when new AUX data has been queued, with
etmq->on_heap tracking whether a queue currently occupies a heap slot;
this mirrors intel_pt_update_queues() and intel_pt_queue::on_heap.
Flushing becomes cs_etm__flush_timestamped_queues(). What remains is the
decode loop on its own, which a later patch can then drive incrementally.

No functional change: the sole caller performs the same three steps in the
same order.

Assisted-by: Devmate:GPT-5.6
Signed-off-by: Amir Ayupov <aaupov@fb.com>
---
 tools/perf/util/cs-etm.c | 71 ++++++++++++++++++++++++++++++++++------
 1 file changed, 61 insertions(+), 10 deletions(-)

diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c
index 114b3cd2da495..4d895f11deb7f 100644
--- a/tools/perf/util/cs-etm.c
+++ b/tools/perf/util/cs-etm.c
@@ -136,9 +136,13 @@ struct cs_etm_queue {
 	 */
 	struct intlist *own_traceid_list;
 	u32 sink_id;
+	/* Whether this queue currently occupies a slot in etm->heap */
+	bool on_heap;
 };
 
+static int cs_etm__update_queues(struct cs_etm_auxtrace *etm);
 static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm);
+static int cs_etm__flush_timestamped_queues(struct cs_etm_auxtrace *etm);
 static int cs_etm__process_timeless_queues(struct cs_etm_auxtrace *etm,
 					   pid_t tid);
 static int cs_etm__get_data_block(struct cs_etm_queue *etmq);
@@ -939,6 +943,8 @@ static int cs_etm__flush_events(struct perf_session *session,
 	struct cs_etm_auxtrace *etm = container_of(session->auxtrace,
 						   struct cs_etm_auxtrace,
 						   auxtrace);
+	int ret;
+
 	if (dump_trace)
 		return 0;
 
@@ -953,7 +959,15 @@ static int cs_etm__flush_events(struct perf_session *session,
 		return cs_etm__process_timeless_queues(etm, -1);
 	}
 
-	return cs_etm__process_timestamped_queues(etm);
+	ret = cs_etm__update_queues(etm);
+	if (ret)
+		return ret;
+
+	ret = cs_etm__process_timestamped_queues(etm);
+	if (ret)
+		return ret;
+
+	return cs_etm__flush_timestamped_queues(etm);
 }
 
 static void cs_etm__free_traceid_queues(struct cs_etm_queue *etmq)
@@ -1330,6 +1344,8 @@ static int cs_etm__queue_first_cs_timestamp(struct cs_etm_auxtrace *etm,
 	 */
 	cs_queue_nr = TO_CS_QUEUE_NR(queue_nr, trace_chan_id);
 	ret = auxtrace_heap__add(&etm->heap, cs_queue_nr, cs_timestamp);
+	if (!ret)
+		etmq->on_heap = true;
 out:
 	return ret;
 }
@@ -2767,23 +2783,30 @@ static int cs_etm__process_timeless_queues(struct cs_etm_auxtrace *etm,
 	return 0;
 }
 
-static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
+/*
+ * Seed the heap with one entry from each queue that is not already
+ * represented in it, so that decoding proceeds in time order across all
+ * queues. Only queues that have newly queued data need to be considered.
+ */
+static int cs_etm__update_queues(struct cs_etm_auxtrace *etm)
 {
 	int ret = 0;
-	unsigned int cs_queue_nr, queue_nr, i;
-	u8 trace_chan_id;
-	u64 cs_timestamp;
-	struct auxtrace_queue *queue;
+	unsigned int i;
 	struct cs_etm_queue *etmq;
-	struct cs_etm_traceid_queue *tidq;
+
+	if (!etm->queues.new_data)
+		return 0;
+
+	etm->queues.new_data = false;
 
 	/*
 	 * Pre-populate the heap with one entry from each queue so that we can
-	 * start processing in time order across all queues.
+	 * start processing in time order across all queues. Skip queues that
+	 * already occupy a heap slot, otherwise they would be added twice.
 	 */
 	for (i = 0; i < etm->queues.nr_queues; i++) {
 		etmq = etm->queues.queue_array[i].priv;
-		if (!etmq)
+		if (!etmq || etmq->on_heap)
 			continue;
 
 		ret = cs_etm__queue_first_cs_timestamp(etm, etmq, i);
@@ -2791,6 +2814,19 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 			return ret;
 	}
 
+	return ret;
+}
+
+static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
+{
+	int ret = 0;
+	unsigned int cs_queue_nr, queue_nr;
+	u8 trace_chan_id;
+	u64 cs_timestamp;
+	struct auxtrace_queue *queue;
+	struct cs_etm_queue *etmq;
+	struct cs_etm_traceid_queue *tidq;
+
 	while (1) {
 		if (!etm->heap.heap_cnt)
 			break;
@@ -2807,6 +2843,7 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 		 * to process it.
 		 */
 		auxtrace_heap__pop(&etm->heap);
+		etmq->on_heap = false;
 
 		tidq  = cs_etm__etmq_get_traceid_queue(etmq, trace_chan_id);
 		if (!tidq) {
@@ -2874,7 +2911,21 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 		 */
 		cs_queue_nr = TO_CS_QUEUE_NR(queue_nr, trace_chan_id);
 		ret = auxtrace_heap__add(&etm->heap, cs_queue_nr, cs_timestamp);
+		if (ret)
+			goto out;
+		etmq->on_heap = true;
 	}
+out:
+	return ret;
+}
+
+/* Flush any branch stack entries left over once all trace is decoded */
+static int cs_etm__flush_timestamped_queues(struct cs_etm_auxtrace *etm)
+{
+	int ret = 0;
+	unsigned int i;
+	struct cs_etm_queue *etmq;
+	struct cs_etm_traceid_queue *tidq;
 
 	for (i = 0; i < etm->queues.nr_queues; i++) {
 		struct int_node *inode;
@@ -2893,7 +2944,7 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 				return ret;
 		}
 	}
-out:
+
 	return ret;
 }
 
-- 
2.52.0

[PATCH v2 3/5] perf cs-etm: Add branch history to existing samples

[Amir Ayupov]

Implement --itrace=L for CoreSight ETM: decode timestamped trace up to
each existing PMU sample and attach the branch history that led to it.
The sample keeps its own ip, callchain and event identity, and a sample
that already carries a branch stack is left alone.

Samples are correlated with the trace by time, so this requires virtual
ETM timestamps that are correlated to perf time; timeless decoding is
rejected. The decode loop, which the previous patch left on its own in
cs_etm__process_timestamped_queues(), grows a timestamp argument and
stops once the decode frontier reaches it, so on return the
thread stack holds the branches that executed before the sample and none
that executed after. Attaching then reduces to the same
thread_stack__br_sample_late() call intel-pt uses.

No explicit sample-to-queue matching is needed:
thread_stack__br_sample_late() keys on the thread, and the thread stack
is already emptied whenever the decoder reports a discontinuity. The one
case that was not covered is a queue whose trace runs out: flush the
thread stack there too, otherwise samples recorded after the last trace
would pick up stale history.

Take the branch history when attaching it rather than leaving it in the
thread stack. With AUX pause and resume, a pause sample ends a completed
trace window and that window belongs to the sample. Execution while AUX
is paused is not traced, so retaining the window would let a later sample
reuse branches from before the untraced gap. Consuming it ensures that a
sample with no newly decoded trace gets an empty branch stack instead.

As with intel-pt, the internal reconstruction ring is kept deeper than
the requested output depth to cover branches decoded between the sampled
ip and the point at which the sample time was recorded, so --itrace=L<n>
can actually return n entries. Kernel-inclusive trace gets the same
conservative 1024-entry headroom that intel-pt uses.

Assisted-by: Devmate:GPT-5.6
Signed-off-by: Amir Ayupov <aaupov@fb.com>
---
 tools/perf/util/cs-etm.c       | 185 +++++++++++++++++++++++++++++++--
 tools/perf/util/thread-stack.c |  17 +++
 tools/perf/util/thread-stack.h |   1 +
 3 files changed, 195 insertions(+), 8 deletions(-)

diff --git a/tools/perf/util/cs-etm.c b/tools/perf/util/cs-etm.c
index 4d895f11deb7f..00407a80933e1 100644
--- a/tools/perf/util/cs-etm.c
+++ b/tools/perf/util/cs-etm.c
@@ -72,6 +72,11 @@ struct cs_etm_auxtrace {
 	bool use_callchain;
 
 	int num_cpu;
+	/* Output depth requested with --itrace=L<n> */
+	unsigned int br_stack_sz;
+	/* Internal reconstruction depth, see cs_etm__br_stack_init() */
+	unsigned int br_stack_sz_plus;
+	struct branch_stack *br_stack;
 	u64 latest_kernel_timestamp;
 	u32 auxtrace_type;
 	u32 branches_filter;
@@ -91,6 +96,7 @@ struct cs_etm_traceid_queue {
 	u64 kernel_start;
 	union perf_event *event_buf;
 	unsigned int br_stack_sz;
+	unsigned int br_stack_sz_plus;
 	struct branch_stack *last_branch;
 	struct ip_callchain *callchain;
 	struct cs_etm_packet *prev_packet;
@@ -141,7 +147,8 @@ struct cs_etm_queue {
 };
 
 static int cs_etm__update_queues(struct cs_etm_auxtrace *etm);
-static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm);
+static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm,
+					      u64 timestamp);
 static int cs_etm__flush_timestamped_queues(struct cs_etm_auxtrace *etm);
 static int cs_etm__process_timeless_queues(struct cs_etm_auxtrace *etm,
 					   pid_t tid);
@@ -165,6 +172,7 @@ static int cs_etm__metadata_set_trace_id(u8 trace_chan_id, u64 *cpu_metadata);
 #define TO_QUEUE_NR(cs_queue_nr) (cs_queue_nr >> 16)
 #define TO_TRACE_CHAN_ID(cs_queue_nr) (cs_queue_nr & 0x0000ffff)
 #define SINK_UNSET ((u32) -1)
+#define MAX_TIMESTAMP (~0ULL)
 
 static u32 cs_etm__get_v7_protocol_version(u32 etmidr)
 {
@@ -674,7 +682,8 @@ static int cs_etm__init_traceid_queue(struct cs_etm_queue *etmq,
 		if (!tidq->last_branch)
 			goto out_free;
 
-		tidq->br_stack_sz = etm->synth_opts.last_branch_sz;
+		tidq->br_stack_sz = etm->br_stack_sz;
+		tidq->br_stack_sz_plus = etm->br_stack_sz_plus;
 	}
 
 	if (etm->synth_opts.callchain) {
@@ -794,7 +803,7 @@ static void cs_etm__packet_swap(struct cs_etm_auxtrace *etm,
 	struct cs_etm_packet *tmp;
 
 	if (etm->synth_opts.branches || etm->synth_opts.last_branch ||
-	    etm->synth_opts.instructions) {
+	    etm->synth_opts.add_last_branch || etm->synth_opts.instructions) {
 		/*
 		 * Swap PACKET with PREV_PACKET: PACKET becomes PREV_PACKET for
 		 * the next incoming packet.
@@ -963,7 +972,7 @@ static int cs_etm__flush_events(struct perf_session *session,
 	if (ret)
 		return ret;
 
-	ret = cs_etm__process_timestamped_queues(etm);
+	ret = cs_etm__process_timestamped_queues(etm, MAX_TIMESTAMP);
 	if (ret)
 		return ret;
 
@@ -1060,6 +1069,7 @@ static void cs_etm__free(struct perf_session *session)
 		zfree(&aux->metadata[i]);
 
 	zfree(&aux->metadata);
+	zfree(&aux->br_stack);
 	zfree(&aux);
 }
 
@@ -1597,7 +1607,8 @@ static void cs_etm__add_stack_event(struct cs_etm_queue *etmq,
 	u64 from, to;
 	int size;
 
-	if (!etm->synth_opts.branches && !etm->synth_opts.instructions)
+	if (!etm->synth_opts.branches && !etm->synth_opts.instructions &&
+	    !etm->synth_opts.add_last_branch)
 		return;
 
 	if (!cs_etm__packet_has_taken_branch(tidq->prev_packet))
@@ -1614,7 +1625,7 @@ static void cs_etm__add_stack_event(struct cs_etm_queue *etmq,
 				    tidq->prev_packet->flags, from, to, size,
 				    etmq->buffer->buffer_nr + 1,
 				    etmq->etm->use_callchain,
-				    tidq->br_stack_sz, 0);
+				    tidq->br_stack_sz_plus, 0);
 	} else {
 		thread_stack__set_trace_nr(tidq->frontend_thread,
 					   tidq->prev_packet->cpu,
@@ -2817,7 +2828,8 @@ static int cs_etm__update_queues(struct cs_etm_auxtrace *etm)
 	return ret;
 }
 
-static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
+static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm,
+					      u64 timestamp)
 {
 	int ret = 0;
 	unsigned int cs_queue_nr, queue_nr;
@@ -2831,6 +2843,9 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 		if (!etm->heap.heap_cnt)
 			break;
 
+		if (etm->heap.heap_array[0].ordinal >= timestamp)
+			break;
+
 		/* Take the entry at the top of the min heap */
 		cs_queue_nr = etm->heap.heap_array[0].queue_nr;
 		queue_nr = TO_QUEUE_NR(cs_queue_nr);
@@ -2878,8 +2893,25 @@ static int cs_etm__process_timestamped_queues(struct cs_etm_auxtrace *etm)
 		 * No more auxtrace_buffers to process in this etmq, simply
 		 * move on to another entry in the auxtrace_heap.
 		 */
-		if (!ret)
+		if (!ret) {
+			/*
+			 * The trace for this physical queue is exhausted. Drop
+			 * branch history for every trace ID it carried so that
+			 * samples arriving later cannot pick up entries decoded
+			 * before the gap.
+			 */
+			if (etm->synth_opts.add_last_branch) {
+				struct int_node *inode;
+
+				intlist__for_each_entry(inode, etmq->traceid_queues_list) {
+					int idx = (int)(intptr_t)inode->priv;
+
+					tidq = etmq->traceid_queues[idx];
+					thread_stack__flush(tidq->frontend_thread);
+				}
+			}
 			continue;
+		}
 
 		ret = cs_etm__decode_data_block(etmq);
 		if (ret)
@@ -3011,6 +3043,116 @@ static int cs_etm__process_switch_cpu_wide(struct cs_etm_auxtrace *etm,
 	return 0;
 }
 
+static bool cs_etm__tracing_kernel(struct cs_etm_auxtrace *etm,
+				   struct perf_session *session)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(session->evlist, evsel) {
+		if (evsel->core.attr.type == etm->pmu_type &&
+		    !evsel->core.attr.exclude_kernel)
+			return true;
+	}
+
+	return false;
+}
+
+static int cs_etm__br_stack_init(struct cs_etm_auxtrace *etm,
+				 struct perf_session *session)
+{
+	struct evsel *evsel;
+
+	evlist__for_each_entry(session->evlist, evsel) {
+		/*
+		 * Only timestamped events can be matched against the decoded
+		 * trace, so do not advertise a branch stack on any other.
+		 */
+		if (!(evsel->core.attr.sample_type & PERF_SAMPLE_TIME))
+			continue;
+		if (!(evsel->core.attr.sample_type & PERF_SAMPLE_BRANCH_STACK))
+			evsel->synth_sample_type |= PERF_SAMPLE_BRANCH_STACK;
+	}
+
+	/*
+	 * Additional branch stack depth to cater for the branches decoded
+	 * between the sampled ip and the point at which the sample time was
+	 * recorded. Those are trimmed by thread_stack__br_sample_late(), so
+	 * the extra depth keeps the requested output depth achievable. If
+	 * kernel space is not traced, only the branch into the kernel needs
+	 * to be accounted for.
+	 */
+	if (cs_etm__tracing_kernel(etm, session))
+		etm->br_stack_sz_plus += 1024;
+	else
+		etm->br_stack_sz_plus += 1;
+
+	etm->br_stack = zalloc(sizeof(struct branch_stack) +
+			       etm->br_stack_sz * sizeof(struct branch_entry));
+	if (!etm->br_stack)
+		return -ENOMEM;
+
+	return 0;
+}
+
+/*
+ * Add decoded branch history to an existing sample. The sample keeps its own
+ * ip, callchain and event identity; only an absent branch stack is filled in.
+ */
+static int cs_etm__process_sample(struct cs_etm_auxtrace *etm,
+				  struct perf_session *session,
+				  struct perf_sample *sample)
+{
+	struct machine *machine = &session->machines.host;
+	struct thread *thread;
+	int err;
+
+	if (!etm->synth_opts.add_last_branch || sample->branch_stack ||
+	    !sample->ip || !sample->time || sample->time == (u64)-1)
+		return 0;
+
+	/* Adding branch history to existing samples supports the host only */
+	if (sample->cpumode == PERF_RECORD_MISC_GUEST_KERNEL ||
+	    sample->cpumode == PERF_RECORD_MISC_GUEST_USER)
+		return 0;
+
+	err = cs_etm__update_queues(etm);
+	if (err)
+		return err;
+
+	/*
+	 * Decode every queue up to this sample's time. Afterwards the thread
+	 * stack holds the branches that executed before the sample, and
+	 * nothing that executed after it.
+	 */
+	err = cs_etm__process_timestamped_queues(etm, sample->time);
+	if (err)
+		return err;
+
+	thread = machine__findnew_thread(machine, sample->pid, sample->tid);
+	if (!thread)
+		return -ENOMEM;
+
+	/*
+	 * Take the branch history rather than copying it. The trace window
+	 * belongs to the sample that ends it, so once it has been attached a
+	 * later sample with nothing newly decoded finds an empty stack rather
+	 * than being given an earlier window's branches. That is the common
+	 * case whenever the trace is duty cycled, by AUX pause/resume or by
+	 * ETM strobing.
+	 */
+	thread_stack__br_sample_late(thread, sample->cpu, etm->br_stack,
+				     etm->br_stack_sz, sample->ip,
+				     machine__kernel_start(machine));
+	thread_stack__br_stack_consume(thread, sample->cpu);
+
+	if (etm->br_stack->nr)
+		sample->branch_stack = etm->br_stack;
+
+	thread__put(thread);
+
+	return 0;
+}
+
 static int cs_etm__process_event(struct perf_session *session,
 				 union perf_event *event,
 				 struct perf_sample *sample,
@@ -3049,6 +3191,9 @@ static int cs_etm__process_event(struct perf_session *session,
 	case PERF_RECORD_SWITCH_CPU_WIDE:
 		return cs_etm__process_switch_cpu_wide(etm, event);
 
+	case PERF_RECORD_SAMPLE:
+		return cs_etm__process_sample(etm, session, sample);
+
 	case PERF_RECORD_AUX:
 		/*
 		 * Record the latest kernel timestamp available in the header
@@ -3752,11 +3897,34 @@ int cs_etm__process_auxtrace_info_full(union perf_event *event,
 
 	etm->use_thread_stack = etm->synth_opts.thread_stack ||
 				etm->synth_opts.last_branch ||
+				etm->synth_opts.add_last_branch ||
 				etm->synth_opts.callchain;
 
 	etm->use_callchain = etm->synth_opts.thread_stack ||
 			     etm->synth_opts.callchain;
 
+	if (etm->synth_opts.last_branch || etm->synth_opts.add_last_branch) {
+		etm->br_stack_sz = etm->synth_opts.last_branch_sz;
+		etm->br_stack_sz_plus = etm->br_stack_sz;
+	}
+
+	if (etm->synth_opts.add_last_branch) {
+		/*
+		 * Existing samples are matched to decoded trace by time, so
+		 * the trace must carry timestamps that are correlated to perf
+		 * time and the queues must be decoded in time order.
+		 */
+		if (etm->timeless_decoding || !etm->has_virtual_ts) {
+			pr_err("CS ETM Trace: --itrace=L requires virtual timestamped trace\n");
+			err = -EINVAL;
+			goto err_free_queues;
+		}
+
+		err = cs_etm__br_stack_init(etm, session);
+		if (err)
+			goto err_free_queues;
+	}
+
 	err = cs_etm__synth_events(etm, session);
 	if (err)
 		goto err_free_queues;
@@ -3812,6 +3980,7 @@ int cs_etm__process_auxtrace_info_full(union perf_event *event,
 	auxtrace_queues__free(&etm->queues);
 	session->auxtrace = NULL;
 err_free_etm:
+	zfree(&etm->br_stack);
 	zfree(&etm);
 err_free_metadata:
 	/* No need to check @metadata[j], free(NULL) is supported */
diff --git a/tools/perf/util/thread-stack.c b/tools/perf/util/thread-stack.c
index 1360f44421ef8..2713a2ad70b69 100644
--- a/tools/perf/util/thread-stack.c
+++ b/tools/perf/util/thread-stack.c
@@ -614,6 +614,23 @@ void thread_stack__sample_late(struct thread *thread, int cpu,
 	}
 }
 
+/*
+ * Branch history belongs to the sample that ends the trace window, so a
+ * decoder that attaches it to an existing sample should take it rather than
+ * copy it. A later sample with no newly decoded trace then finds an empty
+ * branch stack instead of the previous window's branches.
+ */
+void thread_stack__br_stack_consume(struct thread *thread, int cpu)
+{
+	struct thread_stack *ts = thread__stack(thread, cpu);
+
+	if (!ts || !ts->br_stack_rb)
+		return;
+
+	ts->br_stack_pos = 0;
+	ts->br_stack_rb->nr = 0;
+}
+
 void thread_stack__br_sample(struct thread *thread, int cpu,
 			     struct branch_stack *dst, unsigned int sz)
 {
diff --git a/tools/perf/util/thread-stack.h b/tools/perf/util/thread-stack.h
index b3cd09beb62f0..2aec292bd1bcb 100644
--- a/tools/perf/util/thread-stack.h
+++ b/tools/perf/util/thread-stack.h
@@ -88,6 +88,7 @@ void thread_stack__sample(struct thread *thread, int cpu, struct ip_callchain *c
 void thread_stack__sample_late(struct thread *thread, int cpu,
 			       struct ip_callchain *chain, size_t sz, u64 ip,
 			       u64 kernel_start);
+void thread_stack__br_stack_consume(struct thread *thread, int cpu);
 void thread_stack__br_sample(struct thread *thread, int cpu,
 			     struct branch_stack *dst, unsigned int sz);
 void thread_stack__br_sample_late(struct thread *thread, int cpu,
-- 
2.52.0

[PATCH v2 4/5] perf test cs-etm: Test branch history on existing samples

[Amir Ayupov]

Add a CoreSight shell test for --itrace=L. Record timestamped ETM trace
with explicit -T sample timestamps and AUX pause/resume events, then
check that the pause samples carry both a multi-frame callchain and a
non-empty branch stack for each of the workload's two processes.

Decode the same recording with L4 and L64 and reject any branch stack
deeper than the requested depth.

The test skips when cs_etm is absent, when not run as root, or when the
recording turns out to lack virtual timestamps. It exercises the
timestamp-gated path and the requested-depth bound; it does not attempt
to verify that the attached history is correlated to the sample.

Assisted-by: Devmate:GPT-5.6
Signed-off-by: Amir Ayupov <aaupov@fb.com>
---
 .../tests/shell/coresight/add_last_branch.sh  | 203 ++++++++++++++++++
 1 file changed, 203 insertions(+)
 create mode 100755 tools/perf/tests/shell/coresight/add_last_branch.sh

diff --git a/tools/perf/tests/shell/coresight/add_last_branch.sh b/tools/perf/tests/shell/coresight/add_last_branch.sh
new file mode 100755
index 0000000000000..6f09e720abe80
--- /dev/null
+++ b/tools/perf/tests/shell/coresight/add_last_branch.sh
@@ -0,0 +1,203 @@
+#!/bin/bash -e
+# SPDX-License-Identifier: GPL-2.0
+# CoreSight branch history on existing samples (exclusive)
+
+perf list pmu | grep -q 'cs_etm//' || exit 2
+
+if [ "$(id -u)" != 0 ]; then
+	echo "[Skip] No root permission"
+	exit 2
+fi
+
+tmpdir=$(mktemp -d /tmp/perf-cs-add-last-branch.XXXXX)
+
+cleanup()
+{
+	rm -rf "$tmpdir"
+	trap - EXIT TERM INT
+}
+
+# shellcheck disable=SC2317 # Called through trap.
+trap_cleanup()
+{
+	cleanup
+	exit 1
+}
+trap trap_cleanup EXIT TERM INT
+
+record_data()
+{
+	local cf="$tmpdir/ctl"
+	local af="$tmpdir/ack"
+
+	mkfifo "$cf" "$af"
+
+	# Pin to one CPU so proc1 and proc2 alternate in one per-CPU trace
+	# buffer. Start disabled and use the control FIFO to record only the
+	# workload, not perf test setup and teardown.
+	if perf record -T -o "$tmpdir/data" -C 0 -D -1 \
+		--control fifo:"$cf","$af" \
+		-e cs_etm/aux-action=start-paused/u \
+		-e cycles/aux-action=resume,period=550019/u \
+		-e cycles/aux-action=pause,period=100003,call-graph=fp/u -- \
+		taskset --cpu-list 0 perf test --record-ctl fifo:"$cf","$af" \
+		-w context_switch_loop 10000 \
+		>/dev/null 2>"$tmpdir/stderr"; then
+		return 0
+	fi
+
+	echo "Failed to record ETM trace with AUX pause/resume" >&2
+	cat "$tmpdir/stderr" >&2
+	return 1
+}
+
+decode()
+{
+	local size=$1
+	local output=$2
+
+	if perf script -i "$tmpdir/data" --itrace="L$size" \
+		-F comm,pid,tid,event,ip,brstack >"$output" \
+		2>"$tmpdir/stderr"; then
+		return 0
+	fi
+
+	if grep -q "itrace=L requires virtual timestamped trace" \
+		"$tmpdir/stderr"; then
+		echo "[Skip] Virtual CoreSight timestamps are not available"
+		cleanup
+		exit 2
+	fi
+
+	cat "$tmpdir/stderr" >&2
+	return 1
+}
+
+check_process_samples()
+{
+	local output=$1
+	local comm
+
+	# Expect each process to have a pause-event sample followed by at least
+	# one branch entry in 0xFROM/0xTO/... form.
+	for comm in proc1 proc2; do
+		awk -v comm="$comm" '
+			$1 == comm && /cycles\/aux-action=pause/ {
+				in_sample = 1
+				next
+			}
+			!NF {
+				in_sample = 0
+				next
+			}
+			in_sample && /0x[[:xdigit:]]+\/0x[[:xdigit:]]+\// {
+				found = 1
+			}
+			END { exit !found }
+		' "$output" || {
+			echo "No pause-event branch stack found for $comm" >&2
+			grep -A 4 "^$comm .*cycles/aux-action=pause" "$output" \
+				| head -n 20 >&2 || true
+			return 1
+		}
+	done
+}
+
+check_callchains()
+{
+	local output="$tmpdir/script-callchain"
+	local comm
+
+	if ! perf script -i "$tmpdir/data" -F comm,event,ip >"$output" \
+		2>"$tmpdir/stderr"; then
+		echo "Failed to dump pause-event callchains" >&2
+		cat "$tmpdir/stderr" >&2
+		return 1
+	fi
+
+	# Expect a pause-event header for each process followed by at least two
+	# indented instruction-pointer frames.
+	for comm in proc1 proc2; do
+		awk -v comm="$comm" '
+			$1 == comm && /cycles\/aux-action=pause/ {
+				in_sample = 1
+				frames = 0
+				next
+			}
+			!NF {
+				if (in_sample && frames >= 2)
+					found = 1
+				in_sample = 0
+				next
+			}
+			in_sample && /^[[:space:]]+[[:xdigit:]]+([[:space:]]|$)/ {
+				frames++
+			}
+			END {
+				if (in_sample && frames >= 2)
+					found = 1
+				exit !found
+			}
+		' "$output" || {
+			echo "No multi-frame pause-event callchain found for $comm" >&2
+			grep -A 8 "^$comm .*cycles/aux-action=pause" "$output" \
+				| head -n 40 >&2 || true
+			return 1
+		}
+	done
+}
+
+check_branch_stacks()
+{
+	local output=$1
+	local max_entries=$2
+
+	local ret
+
+	if awk -v max="$max_entries" '
+		/0x[[:xdigit:]]+\/0x[[:xdigit:]]+\// {
+			entries = 0
+			for (i = 1; i <= NF; i++)
+				if ($i ~ /^0x[[:xdigit:]]+\/0x[[:xdigit:]]+\//)
+					entries++
+			if (entries)
+				found = 1
+			if (entries > max) {
+				status = 2
+				exit
+			}
+		}
+		END {
+			if (status)
+				exit status
+			if (!found)
+				exit 1
+		}
+	' "$output"; then
+		return 0
+	else
+		ret=$?
+	fi
+
+	case $ret in
+	1) echo "No ETM branch stacks found" >&2 ;;
+	2) echo "Branch stack exceeds requested L$max_entries depth" >&2 ;;
+	esac
+	# Expected decoded pause-event lines contain at most L<n> branch entries.
+	grep 'cycles/aux-action=pause' "$output" | head -n 5 >&2 || true
+	return 1
+}
+
+record_data
+check_callchains
+
+decode 4 "$tmpdir/script-L4"
+check_process_samples "$tmpdir/script-L4"
+check_branch_stacks "$tmpdir/script-L4" 4
+
+decode 64 "$tmpdir/script-L64"
+check_process_samples "$tmpdir/script-L64"
+check_branch_stacks "$tmpdir/script-L64" 64
+
+cleanup
+exit 0
-- 
2.52.0

[PATCH v2 5/5] Documentation: coresight: Document context-sensitive PGO workflow

[Amir Ayupov]

Document combining callchain-bearing AUX pause samples with the ETM
branch history that precedes them, which is the pairing that
context-sensitive PGO tools consume, and the --itrace=L64 decode with the
non-empty branch-stack dlfilter.

Assisted-by: Devmate:GPT-5.6
Signed-off-by: Amir Ayupov <aaupov@fb.com>
---
 .../trace/coresight/coresight-perf.rst        | 62 +++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/Documentation/trace/coresight/coresight-perf.rst b/Documentation/trace/coresight/coresight-perf.rst
index 0a77741a431ef..c0c82b3d26ea8 100644
--- a/Documentation/trace/coresight/coresight-perf.rst
+++ b/Documentation/trace/coresight/coresight-perf.rst
@@ -109,6 +109,68 @@ Example for triggering AUX pause and resume with PMU event::
         -e cycles/aux-action=pause,period=10000000/ \
         -e cycles/aux-action=resume,period=1050000/ -- sleep 1
 
+Context-sensitive sampled PGO (CSSPGO) profiling
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A callchain-bearing pause event can be combined with the branch history from
+its preceding ETM trace window. This provides the synchronized callchain and
+branch stack consumed by context-sensitive PGO tools without continuously
+recording ETM trace for a long-running process.
+
+For example, record user-space ETM trace, resume it periodically, and pause it
+with a cycle event that also captures a frame-pointer callchain::
+
+  perf record -T \
+        -e cs_etm/aux-action=start-paused,timestamp/u \
+        -e cycles/aux-action=resume,period=8350251/u \
+        -e cycles/aux-action=pause,period=100003,call-graph=fp/u \
+        -- ./workload
+
+The two cycle events count independently. With pause period ``P`` and resume
+period ``R``, each trace window is approximately 0 to ``P`` cycles long, so the
+average duty cycle is ``P / (2 * R)``. The periods above give about 0.6% duty.
+
+Do not make ``R`` an integer multiple of ``P``: coincident pause and resume
+interrupts can produce zero-length windows. Choosing ``R`` near
+``(k + 1/2) * P``, as above, moves the resume phase across the pause interval.
+Pause events that fire while ETM is already paused have no branch history; the
+dlfilter below removes those samples.
+
+The ``-T`` option timestamps the pause samples, while ``timestamp`` enables
+ETM timestamp packets. Both are required to correlate the sample with ETM
+trace. This mode also requires virtual ETM timestamps correlated to perf time
+and a callchain on the pause event. Use ``call-graph=dwarf`` instead of
+``call-graph=fp`` when the workload does not preserve frame pointers.
+
+Tuning duty cycle
+^^^^^^^^^^^^^^^^^
+
+The example above favors low recording overhead for fleet collection. With
+independent counters, ``P / R`` is the nominal window ratio while
+``P / (2 * R)`` is the expected average ETM-on duty. For ``P = 100003``, two
+measured Neoverse V2 operating points are:
+
+- Fleet collection: ``R = 8350251``, 1.2% nominal ratio and 0.6% average duty.
+- Targeted profiling: ``R = 1050031``, 9.5% nominal ratio and 4.8% average duty.
+
+AUX buffer size must also scale with trace volume. A 128 KiB AUX buffer worked
+at 0.6% duty but was unstable at some higher-duty points, where it increased
+output size or overran and reduced useful-sample yield. Use 4 MiB as a
+conservative starting point around ``R/P = 6.7`` to ``12.5``. These values are
+workload and platform dependent; verify useful samples per MiB and lost AUX
+records when tuning another system.
+
+Add up to 64 decoded ETM branches to each existing pause sample and emit the
+hybrid samples in the regular perf-script format::
+
+  perf script -i perf.data --itrace=L64 \
+        --dlfilter=dlfilter-nonempty-brstack.so > perf.script
+
+The ``dlfilter-nonempty-brstack.so`` filter drops samples that ended up with
+no branch history at all, for example samples from a thread that was never
+traced, or samples recorded before the first or after the last trace window.
+It is built and installed with perf's other dlfilters.
+
 Perf test - Verify kernel and userspace perf CoreSight work
 -----------------------------------------------------------
 
-- 
2.52.0

Re: [PATCH v2 5/5] Documentation: coresight: Document context-sensitive PGO workflow

[sashiko-bot]

> Document combining callchain-bearing AUX pause samples with the ETM
> branch history that precedes them, which is the pairing that
> context-sensitive PGO tools consume, and the --itrace=L64 decode with the
> non-empty branch-stack dlfilter.
> 
> Assisted-by: Devmate:GPT-5.6
> Signed-off-by: Amir Ayupov <aaupov@fb.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787005265.git.aaupov@fb.com?part=5

Re: [PATCH v2 1/5] perf dlfilter: Add non-empty branch stack filter

[sashiko-bot]

> --itrace=L adds decoded branch history to existing samples, but a sample
> that was recorded while the decoder had no trace for that thread keeps an
> empty branch stack. Consumers of the resulting perf script output, such
> as profile generators for context-sensitive PGO, have no use for those
> samples.
> 
> Add an opt-in dlfilter that drops samples whose parsed branch stack is
> empty, so users can exclude them without changing default sample
> semantics. Build and install it alongside perf's existing dlfilters.
> 
> Assisted-by: Devmate:GPT-5.6
> Signed-off-by: Amir Ayupov <aaupov@fb.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787005265.git.aaupov@fb.com?part=1

Re: [PATCH v2 4/5] perf test cs-etm: Test branch history on existing samples

[sashiko-bot]

> Add a CoreSight shell test for --itrace=L. Record timestamped ETM trace
> with explicit -T sample timestamps and AUX pause/resume events, then
> check that the pause samples carry both a multi-frame callchain and a
> non-empty branch stack for each of the workload's two processes.
> 
> Decode the same recording with L4 and L64 and reject any branch stack
> deeper than the requested depth.
> 
> The test skips when cs_etm is absent, when not run as root, or when the
> recording turns out to lack virtual timestamps. It exercises the
> timestamp-gated path and the requested-depth bound; it does not attempt
> to verify that the attached history is correlated to the sample.
> 
> Assisted-by: Devmate:GPT-5.6
> Signed-off-by: Amir Ayupov <aaupov@fb.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787005265.git.aaupov@fb.com?part=4

Re: [PATCH v2 2/5] perf cs-etm: Split up cs_etm__process_timestamped_queues()

[sashiko-bot]

> cs_etm__process_timestamped_queues() currently does three things: it seeds
> the auxtrace heap with one entry per queue, it decodes until the heap is
> empty, and it then walks every traceID queue to flush whatever is left in
> the branch stacks. That is fine while the only caller is
> cs_etm__flush_events(), which runs once, but it does not survive the
> function being called repeatedly.
> 
> Seeding cannot be repeated because a queue that still holds a heap slot
> would be seeded again, adding duplicate entries and growing the heap
> without bound. Flushing cannot be repeated either, because ending a block
> finalises state that later trace still needs.
> 
> Move both out. Seeding becomes cs_etm__update_queues(), gated on
> queues.new_data so it only runs when new AUX data has been queued, with
> etmq->on_heap tracking whether a queue currently occupies a heap slot;
> this mirrors intel_pt_update_queues() and intel_pt_queue::on_heap.
> Flushing becomes cs_etm__flush_timestamped_queues(). What remains is the
> decode loop on its own, which a later patch can then drive incrementally.
> 
> No functional change: the sole caller performs the same three steps in the
> same order.
> 
> Assisted-by: Devmate:GPT-5.6
> Signed-off-by: Amir Ayupov <aaupov@fb.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787005265.git.aaupov@fb.com?part=2

Re: [PATCH v2 3/5] perf cs-etm: Add branch history to existing samples

[sashiko-bot]

> Implement --itrace=L for CoreSight ETM: decode timestamped trace up to
> each existing PMU sample and attach the branch history that led to it.
> The sample keeps its own ip, callchain and event identity, and a sample
> that already carries a branch stack is left alone.
> 
> Samples are correlated with the trace by time, so this requires virtual
> ETM timestamps that are correlated to perf time; timeless decoding is
> rejected. The decode loop, which the previous patch left on its own in
> cs_etm__process_timestamped_queues(), grows a timestamp argument and
> stops once the decode frontier reaches it, so on return the
> thread stack holds the branches that executed before the sample and none
> that executed after. Attaching then reduces to the same
> thread_stack__br_sample_late() call intel-pt uses.
> 
> No explicit sample-to-queue matching is needed:
> [ ... ]
> can actually return n entries. Kernel-inclusive trace gets the same
> conservative 1024-entry headroom that intel-pt uses.
> 
> Assisted-by: Devmate:GPT-5.6
> Signed-off-by: Amir Ayupov <aaupov@fb.com>

Sashiko has reviewed this patch and found no issues. It looks great!

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/cover.1787005265.git.aaupov@fb.com?part=3