[PATCH v5 0/8] Add support for Firefox's gecko profile format

[PATCH v5 0/8] Add support for Firefox's gecko profile format

[Anup Sharma]

This patch series adds support for Firefox's gecko profile format.
The format is documented here [1].

I have incorporated several changes based on feedback from the
previous version of the patch.

Changes in v5:
- Reorganized the patch series to ensure that each patch works independently.
- Renamed the file to perf-gecko.py.
- Added command exection support for the script. The script can now be
  executed as "perf script report perf-gecko > output.json."
- Simplified the usage information; previously it was "perf script
  perf-gecko.py > output.json," now it is "perf script report
  perf-gecko > output.json."
- Merged the test and command execution patches into this series,
  which were previously in a separate series.

Committer Testing:
- Tested with a perf.data file generated for single and multiple cpu
  cores enabled.
- Uploaded the stdout into profiler.firefox.com and verified the
  output.
- Verified the output with the output generated by the existing
  script as mentioned here [2].

Method:
- perf record -F 99 -a -g -- sleep 5
- perf script report perf-gecko > output.json
- upload output.json to profiler.firefox.com

[1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
[2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler

Anup Sharma (8):
  perf scripts python: Add initial script file with usage information
  perf scripts python: Extact necessary information from process event
  perf scripts python: Add classes and conversion functions
  perf scripts python: Add trace end processing and PRODUCT and
    CATEGORIES information
  perf scripts python: Implement add sample function and thread
    processing
  perf scripts python: Implement add sample function and thread
    processing
  perf scripts python: Add command execution for firefox gecko converter
    script
  perf test: Add support for testing firefox gecko converter script

 .../perf/scripts/python/bin/perf-gecko-record |   2 +
 .../perf/scripts/python/bin/perf-gecko-report |   3 +
 tools/perf/scripts/python/perf-gecko.py       | 339 ++++++++++++++++++
 tools/perf/tests/shell/test_perf_gecko.sh     | 188 ++++++++++
 4 files changed, 532 insertions(+)
 create mode 100644 tools/perf/scripts/python/bin/perf-gecko-record
 create mode 100644 tools/perf/scripts/python/bin/perf-gecko-report
 create mode 100644 tools/perf/scripts/python/perf-gecko.py
 create mode 100755 tools/perf/tests/shell/test_perf_gecko.sh

-- 
2.34.1

[PATCH v5 1/8] perf scripts python: Add initial script file with usage information

[Anup Sharma]

Added necessary modules, including the Perf-Trace-Util
library, and defines the required functions and variables
for using perf script python. The perf_trace_context and
Core modules for tracing and processing events has been
also imported. Added usage information.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 31 +++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 tools/perf/scripts/python/perf-gecko.py

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
new file mode 100644
index 000000000000..4e651adb6395
--- /dev/null
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -0,0 +1,31 @@
+# firefox-gecko-converter.py - Convert perf record output to Firefox's gecko profile format
+# SPDX-License-Identifier: GPL-2.0
+#
+# The script converts perf.data to Gecko Profile Format,
+# which can be read by https://profiler.firefox.com/.
+#
+# Usage:
+#
+#     perf record -a -g -F 99 sleep 60
+#     perf script report perf-gecko > output.json
+
+import os
+import sys
+from typing import Dict
+
+# Add the Perf-Trace-Util library to the Python path
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+	'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+from perf_trace_context import *
+from Core import *
+
+# Uses perf script python interface to parse each
+# event and store the data in the thread builder.
+def process_event(param_dict: Dict) -> None:
+	pass
+
+# Trace_end runs at the end and will be used to aggregate
+# the data into the final json object and print it out to stdout.
+def trace_end() -> None:
+	pass
-- 
2.34.1

[PATCH v5 2/8] perf scripts python: Extact necessary information from process event

[Anup Sharma]

The script takes in a sample event dictionary(param_dict) and retrieves
relevant data such as time stamp, PID, TID, and comm for each event.
Also start time is defined as a global variable as it need to be passed
to trace_end for gecko meta information field creation.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
index 4e651adb6395..02849d3c9e81 100644
--- a/tools/perf/scripts/python/perf-gecko.py
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -20,10 +20,22 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 from perf_trace_context import *
 from Core import *
 
+# start_time is intialiazed only once for the all event traces.
+start_time = None
+
 # Uses perf script python interface to parse each
 # event and store the data in the thread builder.
 def process_event(param_dict: Dict) -> None:
-	pass
+	global start_time
+	global tid_to_thread
+	time_stamp = (param_dict['sample']['time'] // 1000) / 1000
+	pid = param_dict['sample']['pid']
+	tid = param_dict['sample']['tid']
+	comm = param_dict['comm']
+
+	# Start time is the time of the first sample
+	if not start_time:
+		start_time = time_stamp
 
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
-- 
2.34.1

[PATCH v5 3/8] perf scripts python: Add classes and conversion functions

[Anup Sharma]

This commit introduces new classes and conversion functions to
facilitate the representation of Gecko profile information. The new
classes Frame, Stack, Sample, and Thread are added to handle specific
components of the profile data, also link to the origin docs has been
commented out.

Additionally, Inside the Thread class _to_json_dict() method has been
created that converts the current thread data into the corresponding
format expected by the GeckoThread JSON schema, as per the Gecko
profile format specification.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 133 +++++++++++++++++++++++-
 1 file changed, 132 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
index 02849d3c9e81..71e2b042e695 100644
--- a/tools/perf/scripts/python/perf-gecko.py
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -11,7 +11,8 @@
 
 import os
 import sys
-from typing import Dict
+from dataclasses import dataclass, field
+from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
 # Add the Perf-Trace-Util library to the Python path
 sys.path.append(os.environ['PERF_EXEC_PATH'] + \
@@ -20,9 +21,139 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 from perf_trace_context import *
 from Core import *
 
+StringID = int
+StackID = int
+FrameID = int
+CategoryID = int
+Milliseconds = float
+
 # start_time is intialiazed only once for the all event traces.
 start_time = None
 
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
+class Frame(NamedTuple):
+	string_id: StringID
+	relevantForJS: bool
+	innerWindowID: int
+	implementation: None
+	optimizations: None
+	line: None
+	column: None
+	category: CategoryID
+	subcategory: int
+
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L216
+class Stack(NamedTuple):
+	prefix_id: Optional[StackID]
+	frame_id: FrameID
+
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L90
+class Sample(NamedTuple):
+	stack_id: Optional[StackID]
+	time_ms: Milliseconds
+	responsiveness: int
+
+@dataclass
+class Thread:
+	"""A builder for a profile of the thread.
+
+	Attributes:
+		comm: Thread command-line (name).
+		pid: process ID of containing process.
+		tid: thread ID.
+		samples: Timeline of profile samples.
+		frameTable: interned stack frame ID -> stack frame.
+		stringTable: interned string ID -> string.
+		stringMap: interned string -> string ID.
+		stackTable: interned stack ID -> stack.
+		stackMap: (stack prefix ID, leaf stack frame ID) -> interned Stack ID.
+		frameMap: Stack Frame string -> interned Frame ID.
+		comm: str
+		pid: int
+		tid: int
+		samples: List[Sample] = field(default_factory=list)
+		frameTable: List[Frame] = field(default_factory=list)
+		stringTable: List[str] = field(default_factory=list)
+		stringMap: Dict[str, int] = field(default_factory=dict)
+		stackTable: List[Stack] = field(default_factory=list)
+		stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
+		frameMap: Dict[str, int] = field(default_factory=dict)
+	"""
+	comm: str
+	pid: int
+	tid: int
+	samples: List[Sample] = field(default_factory=list)
+	frameTable: List[Frame] = field(default_factory=list)
+	stringTable: List[str] = field(default_factory=list)
+	stringMap: Dict[str, int] = field(default_factory=dict)
+	stackTable: List[Stack] = field(default_factory=list)
+	stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
+	frameMap: Dict[str, int] = field(default_factory=dict)
+
+	def _to_json_dict(self) -> Dict:
+		"""Converts current Thread to GeckoThread JSON format."""
+		# Gecko profile format is row-oriented data as List[List],
+		# And a schema for interpreting each index.
+		# Schema:
+		# https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
+		# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L230
+		return {
+			"tid": self.tid,
+			"pid": self.pid,
+			"name": self.comm,
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L51
+			"markers": {
+				"schema": {
+					"name": 0,
+					"startTime": 1,
+					"endTime": 2,
+					"phase": 3,
+					"category": 4,
+					"data": 5,
+				},
+				"data": [],
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L90
+			"samples": {
+				"schema": {
+					"stack": 0,
+					"time": 1,
+					"responsiveness": 2,
+				},
+				"data": self.samples
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
+			"frameTable": {
+				"schema": {
+					"location": 0,
+					"relevantForJS": 1,
+					"innerWindowID": 2,
+					"implementation": 3,
+					"optimizations": 4,
+					"line": 5,
+					"column": 6,
+					"category": 7,
+					"subcategory": 8,
+				},
+				"data": self.frameTable,
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L216
+			"stackTable": {
+				"schema": {
+					"prefix": 0,
+					"frame": 1,
+				},
+				"data": self.stackTable,
+			},
+			"stringTable": self.stringTable,
+			"registerTime": 0,
+			"unregisterTime": None,
+			"processType": "default",
+		}
+
 # Uses perf script python interface to parse each
 # event and store the data in the thread builder.
 def process_event(param_dict: Dict) -> None:
-- 
2.34.1

[PATCH v5 4/8] perf scripts python: Add trace end processing and PRODUCT and CATEGORIES information

[Anup Sharma]

The final output will now be presented in JSON format following the Gecko
profile structure. Additionally, the inclusion of PRODUCT allows easy retrieval
of header information for UI.

Furthermore, CATEGORIES have been introduced to enable customization of
kernel and user colors using input arguments. To facilitate this functionality,
an argparse-based parser has been implemented.

Note: The implementation of threads will be addressed in subsequent commits
for now I have commented it out.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 65 ++++++++++++++++++++++++-
 1 file changed, 64 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
index 71e2b042e695..f0d29f48467e 100644
--- a/tools/perf/scripts/python/perf-gecko.py
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -11,6 +11,8 @@
 
 import os
 import sys
+import json
+import argparse
 from dataclasses import dataclass, field
 from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
@@ -30,6 +32,13 @@ Milliseconds = float
 # start_time is intialiazed only once for the all event traces.
 start_time = None
 
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/profile.js#L425
+# Follow Brendan Gregg's Flamegraph convention: orange for kernel and yellow for user space by default.
+CATEGORIES = None
+
+# The product name is used by the profiler UI to show the Operating system and Processor.
+PRODUCT = os.popen('uname -op').read().strip()
+
 # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
 class Frame(NamedTuple):
 	string_id: StringID
@@ -171,4 +180,58 @@ def process_event(param_dict: Dict) -> None:
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
 def trace_end() -> None:
-	pass
+	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
+	gecko_profile_with_meta = {
+		"meta": {
+			"interval": 1,
+			"processType": 0,
+			"product": PRODUCT,
+			"stackwalk": 1,
+			"debug": 0,
+			"gcpoison": 0,
+			"asyncstack": 1,
+			"startTime": start_time,
+			"shutdownTime": None,
+			"version": 24,
+			"presymbolicated": True,
+			"categories": CATEGORIES,
+			"markerSchema": [],
+			},
+		"libs": [],
+		# threads will be implemented in later commits.
+		# "threads": threads,
+		"processes": [],
+		"pausedRanges": [],
+	}
+	json.dump(gecko_profile_with_meta, sys.stdout, indent=2)
+
+def main() -> None:
+	global CATEGORIES
+	parser = argparse.ArgumentParser(description="Convert perf.data to Firefox\'s Gecko Profile format")
+
+	# Add the command-line options
+	# Colors must be defined according to this:
+	# https://github.com/firefox-devtools/profiler/blob/50124adbfa488adba6e2674a8f2618cf34b59cd2/res/css/categories.css
+	parser.add_argument('--user-color', default='yellow', help='Color for the User category')
+	parser.add_argument('--kernel-color', default='orange', help='Color for the Kernel category')
+	# Parse the command-line arguments
+	args = parser.parse_args()
+	# Access the values provided by the user
+	user_color = args.user_color
+	kernel_color = args.kernel_color
+
+	CATEGORIES = [
+		{
+			"name": 'User',
+			"color": user_color,
+			"subcategories": ['Other']
+		},
+		{
+			"name": 'Kernel',
+			"color": kernel_color,
+			"subcategories": ['Other']
+		},
+	]
+
+if __name__ == '__main__':
+    main()
-- 
2.34.1

[PATCH v5 5/8] perf scripts python: Implement add sample function and thread processing

[Anup Sharma]

The intern_stack function is responsible for retrieving
or creating a stack_id based on the provided frame_id and prefix_id.
It first generates a key using the frame_id and prefix_id values.
If the stack corresponding to the key is found in the stackMap,
it is returned. Otherwise, a new stack is created by appending
the prefix_id and frame_id to the stackTable. The key
and the index of the newly created stack are added to the
stackMap for future reference.

The _intern_frame function is responsible for retrieving or
creating a frame_id based on the provided frame string. If the frame_id
corresponding to the frameString is found in the frameMap, it is
returned. Otherwise, a new frame is created by appending relevant
information to the frameTable and adding the frameString to the string_id
through _intern_string.

The _intern_string function will gets a matching string, or saves the new
string and returns a String ID.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 54 +++++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
index f0d29f48467e..35e25e64d6d9 100644
--- a/tools/perf/scripts/python/perf-gecko.py
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -13,6 +13,7 @@ import os
 import sys
 import json
 import argparse
+from functools import reduce
 from dataclasses import dataclass, field
 from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
@@ -39,6 +40,10 @@ CATEGORIES = None
 # The product name is used by the profiler UI to show the Operating system and Processor.
 PRODUCT = os.popen('uname -op').read().strip()
 
+# The category index is used by the profiler UI to show the color of the flame graph.
+USER_CATEGORY_INDEX = 0
+KERNEL_CATEGORY_INDEX = 1
+
 # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
 class Frame(NamedTuple):
 	string_id: StringID
@@ -99,6 +104,55 @@ class Thread:
 	stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
 	frameMap: Dict[str, int] = field(default_factory=dict)
 
+	def _intern_stack(self, frame_id: int, prefix_id: Optional[int]) -> int:
+		"""Gets a matching stack, or saves the new stack. Returns a Stack ID."""
+		key = f"{frame_id}" if prefix_id is None else f"{frame_id},{prefix_id}"
+		# key = (prefix_id, frame_id)
+		stack_id = self.stackMap.get(key)
+		if stack_id is None:
+			# return stack_id
+			stack_id = len(self.stackTable)
+			self.stackTable.append(Stack(prefix_id=prefix_id, frame_id=frame_id))
+			self.stackMap[key] = stack_id
+		return stack_id
+
+	def _intern_string(self, string: str) -> int:
+		"""Gets a matching string, or saves the new string. Returns a String ID."""
+		string_id = self.stringMap.get(string)
+		if string_id is not None:
+			return string_id
+		string_id = len(self.stringTable)
+		self.stringTable.append(string)
+		self.stringMap[string] = string_id
+		return string_id
+
+	def _intern_frame(self, frame_str: str) -> int:
+		"""Gets a matching stack frame, or saves the new frame. Returns a Frame ID."""
+		frame_id = self.frameMap.get(frame_str)
+		if frame_id is not None:
+			return frame_id
+		frame_id = len(self.frameTable)
+		self.frameMap[frame_str] = frame_id
+		string_id = self._intern_string(frame_str)
+
+		symbol_name_to_category = KERNEL_CATEGORY_INDEX if frame_str.find('kallsyms') != -1 \
+		or frame_str.find('/vmlinux') != -1 \
+		or frame_str.endswith('.ko)') \
+		else USER_CATEGORY_INDEX
+
+		self.frameTable.append(Frame(
+			string_id=string_id,
+			relevantForJS=False,
+			innerWindowID=0,
+			implementation=None,
+			optimizations=None,
+			line=None,
+			column=None,
+			category=symbol_name_to_category,
+			subcategory=None,
+		))
+		return frame_id
+
 	def _to_json_dict(self) -> Dict:
 		"""Converts current Thread to GeckoThread JSON format."""
 		# Gecko profile format is row-oriented data as List[List],
-- 
2.34.1

[PATCH v5 6/8] perf scripts python: Implement add sample function and thread processing

[Anup Sharma]

The stack has been created for storing func and dso from the callchain.
The sample has been added to a specific thread. It first checks if the
thread exists in the Thread class. Then it call _add_sample function
which is responsible for appending a new entry to the samples list.

Also callchain parsing and storing part is implemented. Moreover removed
the comment from thread.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/perf-gecko.py | 52 ++++++++++++++++++++++++-
 1 file changed, 50 insertions(+), 2 deletions(-)

diff --git a/tools/perf/scripts/python/perf-gecko.py b/tools/perf/scripts/python/perf-gecko.py
index 35e25e64d6d9..29d10985e176 100644
--- a/tools/perf/scripts/python/perf-gecko.py
+++ b/tools/perf/scripts/python/perf-gecko.py
@@ -40,6 +40,9 @@ CATEGORIES = None
 # The product name is used by the profiler UI to show the Operating system and Processor.
 PRODUCT = os.popen('uname -op').read().strip()
 
+# Here key = tid, value = Thread
+tid_to_thread = dict()
+
 # The category index is used by the profiler UI to show the color of the flame graph.
 USER_CATEGORY_INDEX = 0
 KERNEL_CATEGORY_INDEX = 1
@@ -153,6 +156,25 @@ class Thread:
 		))
 		return frame_id
 
+	def _add_sample(self, comm: str, stack: List[str], time_ms: Milliseconds) -> None:
+		"""Add a timestamped stack trace sample to the thread builder.
+		Args:
+			comm: command-line (name) of the thread at this sample
+			stack: sampled stack frames. Root first, leaf last.
+			time_ms: timestamp of sample in milliseconds.
+		"""
+		# Ihreads may not set their names right after they are created.
+		# Instead, they might do it later. In such situations, to use the latest name they have set.
+		if self.comm != comm:
+			self.comm = comm
+
+		prefix_stack_id = reduce(lambda prefix_id, frame: self._intern_stack
+						(self._intern_frame(frame), prefix_id), stack, None)
+		if prefix_stack_id is not None:
+			self.samples.append(Sample(stack_id=prefix_stack_id,
+									time_ms=time_ms,
+									responsiveness=0))
+
 	def _to_json_dict(self) -> Dict:
 		"""Converts current Thread to GeckoThread JSON format."""
 		# Gecko profile format is row-oriented data as List[List],
@@ -231,9 +253,36 @@ def process_event(param_dict: Dict) -> None:
 	if not start_time:
 		start_time = time_stamp
 
+	# Parse and append the callchain of the current sample into a stack.
+	stack = []
+	if param_dict['callchain']:
+		for call in param_dict['callchain']:
+			if 'sym' not in call:
+				continue
+			stack.append(f'{call["sym"]["name"]} (in {call["dso"]})')
+		if len(stack) != 0:
+			# Reverse the stack, as root come first and the leaf at the end.
+			stack = stack[::-1]
+
+	# During perf record if -g is not used, the callchain is not available.
+	# In that case, the symbol and dso are available in the event parameters.
+	else:
+		func = param_dict['symbol'] if 'symbol' in param_dict else '[unknown]'
+		dso = param_dict['dso'] if 'dso' in param_dict else '[unknown]'
+		stack.append(f'{func} (in {dso})')
+
+	# Add sample to the specific thread.
+	thread = tid_to_thread.get(tid)
+	if thread is None:
+		thread = Thread(comm=comm, pid=pid, tid=tid)
+		tid_to_thread[tid] = thread
+	thread._add_sample(comm=comm, stack=stack, time_ms=time_stamp)
+
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
 def trace_end() -> None:
+	threads = [thread._to_json_dict() for thread in tid_to_thread.values()]
+
 	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
 	gecko_profile_with_meta = {
 		"meta": {
@@ -252,8 +301,7 @@ def trace_end() -> None:
 			"markerSchema": [],
 			},
 		"libs": [],
-		# threads will be implemented in later commits.
-		# "threads": threads,
+		"threads": threads,
 		"processes": [],
 		"pausedRanges": [],
 	}
-- 
2.34.1

[PATCH v5 7/8] perf scripts python: Add command execution for perf gecko script

[Anup Sharma]

This will enable the execution of perf-gecko.py script
using record and report commands in perf script.
And this will be also reflected at "perf script -l" command.

For Example:
    perf script record perf-gecko
    perf script report perf-gecko

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 tools/perf/scripts/python/bin/perf-gecko-record | 2 ++
 tools/perf/scripts/python/bin/perf-gecko-report | 3 +++
 2 files changed, 5 insertions(+)
 create mode 100644 tools/perf/scripts/python/bin/perf-gecko-record
 create mode 100644 tools/perf/scripts/python/bin/perf-gecko-report

diff --git a/tools/perf/scripts/python/bin/perf-gecko-record b/tools/perf/scripts/python/bin/perf-gecko-record
new file mode 100644
index 000000000000..7df5a19c0163
--- /dev/null
+++ b/tools/perf/scripts/python/bin/perf-gecko-record
@@ -0,0 +1,2 @@
+#!/bin/bash
+perf record -g "$@"
diff --git a/tools/perf/scripts/python/bin/perf-gecko-report b/tools/perf/scripts/python/bin/perf-gecko-report
new file mode 100644
index 000000000000..6781313dc862
--- /dev/null
+++ b/tools/perf/scripts/python/bin/perf-gecko-report
@@ -0,0 +1,3 @@
+#!/bin/bash
+# description: create firefox gecko profile json format from perf.data
+perf script -s "$PERF_EXEC_PATH"/scripts/python/perf-gecko.py -- "$@"
-- 
2.34.1

Re: [PATCH v5 7/8] perf scripts python: Add command execution for perf gecko script

[Arnaldo Carvalho de Melo]

Em Fri, Jul 21, 2023 at 11:27:46PM +0530, Anup Sharma escreveu:
> This will enable the execution of perf-gecko.py script
> using record and report commands in perf script.
> And this will be also reflected at "perf script -l" command.
> 
> For Example:
>     perf script record perf-gecko
>     perf script report perf-gecko

But this isn't working:

[root@five ~]# perf script gecko
usage: gecko.py [-h] [--user-color USER_COLOR] [--kernel-color KERNEL_COLOR]
gecko.py: error: unrecognized arguments: -i -
[root@five ~]#

See below how this pipes the 'record' script with its 'report'
counterpart.

[root@five ~]# strace -s200 -e execve perf script gecko
execve("/root/bin/perf", ["perf", "script", "gecko"], 0x7ffd0b021c18 /* 23 vars */) = 0
execve("/bin/sh", ["/bin/sh", "/var/home/acme/libexec/perf-core/scripts/python/bin/gecko-report", "-i", "-"], 0x24a5c20 /* 26 vars */) = 0
usage: gecko.py [-h] [--user-color USER_COLOR] [--kernel-color KERNEL_COLOR]
gecko.py: error: unrecognized arguments: -i -
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3275075, si_uid=0, si_status=2, si_utime=3 /* 0.03 s */, si_stime=0} ---
+++ exited with 2 +++
[root@five ~]#
[root@five ~]# perf record -F 99 -a -g -- sleep 5 -o - | perf script gecko
usage: gecko.py [-h] [--user-color USER_COLOR] [--kernel-color KERNEL_COLOR]
gecko.py: error: unrecognized arguments: -i -
[root@five ~]#

See it being used another script:

[root@five ~]# perf script -l
List of available trace scripts:
  failed-syscalls [comm]               system-wide failed syscalls
  rw-by-file <comm>                    r/w activity for a program, by file
  rw-by-pid                            system-wide r/w activity
  rwtop [interval]                     system-wide r/w top
  wakeup-latency                       system-wide min/max/avg wakeup latency
  compaction-times [-h] [-u] [-p|-pv] [-t | [-m] [-fs] [-ms]] [pid|pid-range|comm-regex] display time taken by mm compaction
  event_analyzing_sample               analyze all perf samples
  export-to-postgresql [database name] [columns] [calls] export perf data to a postgresql database
  export-to-sqlite [database name] [columns] [calls] export perf data to a sqlite3 database
  failed-syscalls-by-pid [comm]        system-wide failed syscalls, by pid
  flamegraph                           create flame graphs
  futex-contention                     futext contention measurement
  gecko                                create firefox gecko profile json format from perf.data
  intel-pt-events                      print Intel PT Events including Power Events and PTWRITE
  mem-phys-addr                        resolve physical address samples
  net_dropmonitor                      display a table of dropped frames
  netdev-times [tx] [rx] [dev=] [debug] display a process of packet and processing time
  powerpc-hcalls
  sched-migration                      sched migration overview
  sctop [comm] [interval]              syscall top
  stackcollapse                        produce callgraphs in short form for scripting use
  syscall-counts-by-pid [comm]         system-wide syscall counts, by pid
  syscall-counts [comm]                system-wide syscall counts
  task-analyzer                        analyze timings of tasks
[root@five ~]#

[root@five ~]# perf script futex-contention
Press control+C to stop and show the summary
^CSocket Thread[195353] lock 7f294b100018 contended 387 times, 579 avg ns [max: 32060 ns, min 390 ns]
WebrtcC~read #1[2866067] lock 7f294b100018 contended 390 times, 506 avg ns [max: 19130 ns, min 330 ns]
Socket Thread[12316] lock 7efd9479b8a0 contended 2 times, 785 avg ns [max: 910 ns, min 660 ns]
IPDL Background[12109] lock 7f7792b00018 contended 8 times, 6108 avg ns [max: 9060 ns, min 550 ns]
WebrtcWorker #3[2866076] lock 7f2922c074cc contended 5 times, 79002 avg ns [max: 133610 ns, min 22810 ns]
WebrtcWorker #4[2866078] lock 7f2922c07480 contended 19 times, 19540 avg ns [max: 57740 ns, min 4490 ns]
firefox[12080] lock 7f7792b00e18 contended 15 times, 86585 avg ns [max: 141290 ns, min 510 ns]
AudioIP~ent RPC[201969] lock 7f294b100018 contended 8 times, 1764 avg ns [max: 2440 ns, min 510 ns]
<SNIP>
TaskCon~ller #6[195531] lock 7f294b100dd8 contended 1 times, 10300 avg ns [max: 10300 ns, min 10300 ns]
Isolated Web Co[16799] lock 7ffb1d100018 contended 2 times, 7270 avg ns [max: 12450 ns, min 2090 ns]
[root@five ~]#

It will:

3275194 pts/2    S+     0:00 strace -s200 -e execve perf script futex-contention
3275197 pts/2    S+     0:00 /bin/sh /var/home/acme/libexec/perf-core/scripts/python/bin/futex-contention-report -i -
3275198 pts/2    S+     0:00 /bin/sh /var/home/acme/libexec/perf-core/scripts/python/bin/futex-contention-record -a -q -o -
3275199 pts/2    Sl+    0:00 perf record -e syscalls:sys_enter_futex -e syscalls:sys_exit_futex -a -q -o -
3275200 pts/2    S+     0:02 perf script -i - -s /var/home/acme/libexec/perf-core/scripts/python/futex-contention.py


Create a pipe, run
/var/home/acme/libexec/perf-core/scripts/python/bin/futex-contention-record
and pipe it into /var/home/acme/libexec/perf-core/scripts/python/bin/futex-contention-report

I also renamed perf-gecko to just gecko, no need to have the 'perf-'
prefix.

Try this other script, that updates the screen periodically:

[root@five ~]# perf script rwtop
read counts by pid:

   pid                  comm      # reads   bytes_req  bytes_read
------  --------------------   ----------  ----------  ----------
3275423  perf                      341653    16403072    16403816
3275448  Web Content                  615     2503477     1419444
3275431  DOMCacheThread               318     1234020     1232492
3275466  StreamTrans #1               141     1140597      518820
 12189  Cache2 I/O                    193      389798      376046
 12313  QuotaManager IO                81      126252      121668
3275432  StreamTrans #2                10       92305       92289
  9317  fuse-overlayfs               1078  1134780416       56796
3275414  StreamT~s #6488                5       45076       45068
3275446  IndexedDB #5496               25       33892       32364
3275461  pkla-check-auth               40       67235       28513
  1138  systemd-oomd                   40      106560       10520
  1170  polkitd                        51      110776        6951
  2227  pipewire                      370        2960        2960
  3137  gnome-terminal-                15       81104        2869
 12441  MemoryPoller                    3        3072        1559
  2425  pipewire-pulse                185        1480        1480
  2418  pipewire-pulse                185        1480        1480
 12751  threaded-ml                   563        5630         939
  2275  gnome-shell                    36      295488         888

write counts by pid:

   pid                  comm    # writes  bytes_written
------  --------------------  ----------  -------------
3275424  perf                        2440       68339552
 12518  mozStorage #4                 15         229545
 12521  LS Thread                    233         118752
 12189  Cache2 I/O                    11          21760
3275431  DOMCacheThread                4           4153
  2425  pipewire-pulse               370           2960
  2227  pipewire                     185           1480
 12751  threaded-ml                 1124           1124
 12080  firefox                      367            371
  1211  gdbus                         36            288
  1705  gdbus                         30            240
195348  Isolated Web Co              235            235
  1139  systemd-resolve                7            223
  2207  gnome-shell                   26            208
  1545  gdbus                         24            192
201970  AudioIPC0                    185            185
2866066  GraphRunner                 176            176
  1533  geoclue                       16            128
565667  gvfsd-dnssd                   16            128
3275460  pcscd                        15            120
[root@five ~]#

I put what I have on this branch:

git://git.kernel.org/pub/scm/linux/kernel/git/perf/perf-tools-next.git tmp.perf-tools-next

Later I'll try and work on have this working if you don't beat me to it
:-)

- Arnaldo

 
> Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
> ---
>  tools/perf/scripts/python/bin/perf-gecko-record | 2 ++
>  tools/perf/scripts/python/bin/perf-gecko-report | 3 +++
>  2 files changed, 5 insertions(+)
>  create mode 100644 tools/perf/scripts/python/bin/perf-gecko-record
>  create mode 100644 tools/perf/scripts/python/bin/perf-gecko-report
> 
> diff --git a/tools/perf/scripts/python/bin/perf-gecko-record b/tools/perf/scripts/python/bin/perf-gecko-record
> new file mode 100644
> index 000000000000..7df5a19c0163
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/perf-gecko-record
> @@ -0,0 +1,2 @@
> +#!/bin/bash
> +perf record -g "$@"
> diff --git a/tools/perf/scripts/python/bin/perf-gecko-report b/tools/perf/scripts/python/bin/perf-gecko-report
> new file mode 100644
> index 000000000000..6781313dc862
> --- /dev/null
> +++ b/tools/perf/scripts/python/bin/perf-gecko-report
> @@ -0,0 +1,3 @@
> +#!/bin/bash
> +# description: create firefox gecko profile json format from perf.data
> +perf script -s "$PERF_EXEC_PATH"/scripts/python/perf-gecko.py -- "$@"
> -- 
> 2.34.1
> 

-- 

- Arnaldo