[PATCH v4 0/6] Add support for Firefox's gecko profile format

[PATCH v4 0/6] Add support for Firefox's gecko profile format

[Anup Sharma]

This patch series adds support for Firefox's gecko profile format.
The format is documented here [1].

I have incorporated several changes based on feedback from the
previous version of the patch.

Changes in v3:
- Implemented object-oriented programming (OOP) approach for Thread and Sample
  to improve code organization and maintainability.
- Enhanced user experience by introducing argparse for changing color settings
  of user and kernel symbols during execution.
- Ensured proper module encapsulation by adding imports wherever necessary.
- Improved code readability by adding descriptive comments and type annotations.

These changes focus on adopting OOP principles, enhancing user interaction with argparse,
and making the code more readable with comments and type information.

TODO:
- use cpu_mode to determine the category instead of finding it from symbol name.
- Write a test.
- add direct execution command for this script under script/python/bin directory.

Committer Testing:
- Tested with a perf.data file generated for single and multiple cpu
  cores enabled.
- Uploaded the stdout into profiler.firefox.com and verified the
  output.
- Verified the output with the output generated by the existing
  script as mentioned here [2].
Method:
- perf record -F 99 -a -g -- sleep 5
- perf script firefox-gecko-converter.py > output.json
- upload output.json to profiler.firefox.com

[1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
[2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler

Anup Sharma (6):
  perf scripts python: Add initial script file with usage information
  perf scripts python: Extact necessary information from process event
  perf scripts python: Add classes and conversion functions
  perf scripts python: Add trace end processing and PRODUCT and
    CATEGORIES information
  perf scripts python: implement internal get or create frame, stack and
    string function
  perf scripts python: Implement add sample function and thread
    processing

 .../scripts/python/firefox-gecko-converter.py | 339 ++++++++++++++++++
 1 file changed, 339 insertions(+)
 create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py

-- 
2.34.1

[PATCH v4 1/6] perf scripts python: Add initial script file with usage information

[Anup Sharma]

Added necessary modules, including the Perf-Trace-Util
library, and defines the required functions and variables
for using perf script python. The perf_trace_context and
Core modules for tracing and processing events has been
also imported. Also added usage information.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../scripts/python/firefox-gecko-converter.py | 31 +++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
new file mode 100644
index 000000000000..63ecaf1abee8
--- /dev/null
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -0,0 +1,31 @@
+# firefox-gecko-converter.py - Convert perf record output to Firefox's gecko profile format
+# SPDX-License-Identifier: GPL-2.0
+#
+# The script converts perf.data to Gecko Profile Format,
+# which can be read by https://profiler.firefox.com/.
+#
+# Usage:
+#
+#     perf record -a -g -F 99 sleep 60
+#     perf script firefox-gecko-converter.py > output.json
+
+import os
+import sys
+from typing import Dict
+
+# Add the Perf-Trace-Util library to the Python path
+sys.path.append(os.environ['PERF_EXEC_PATH'] + \
+	'/scripts/python/Perf-Trace-Util/lib/Perf/Trace')
+
+from perf_trace_context import *
+from Core import *
+
+# Uses perf script python interface to parse each
+# event and store the data in the thread builder.
+def process_event(param_dict: Dict) -> None:
+	pass
+
+# Trace_end runs at the end and will be used to aggregate
+# the data into the final json object and print it out to stdout.
+def trace_end() -> None:
+	pass
-- 
2.34.1

[PATCH v4 2/6] perf scripts python: Extact necessary information from process event

[Anup Sharma]

The script takes in a sample event dictionary(param_dict) and retrieves
relevant data such as time stamp, PID, TID, and comm for each event.
Also start time is defined as a global variable as it need to be passed
to trace_end for gecko meta information field creation.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../perf/scripts/python/firefox-gecko-converter.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
index 63ecaf1abee8..bfc8c2b026bf 100644
--- a/tools/perf/scripts/python/firefox-gecko-converter.py
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -20,10 +20,22 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 from perf_trace_context import *
 from Core import *
 
+# start_time is intialiazed only once for the all event traces.
+start_time = None
+
 # Uses perf script python interface to parse each
 # event and store the data in the thread builder.
 def process_event(param_dict: Dict) -> None:
-	pass
+	global start_time
+	global tid_to_thread
+	time_stamp = (param_dict['sample']['time'] // 1000) / 1000
+	pid = param_dict['sample']['pid']
+	tid = param_dict['sample']['tid']
+	comm = param_dict['comm']
+
+	# Start time is the time of the first sample
+	if not start_time:
+		start_time = time_stamp
 
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
-- 
2.34.1

[PATCH v4 3/6] perf scripts python: Add classes and conversion functions

[Anup Sharma]

This commit introduces new classes and conversion functions to
facilitate the representation of Gecko profile information. The new
classes Frame, Stack, Sample, and Thread are added to handle specific
components of the profile data, also link to the origin docs has been
commented out.

Additionally, Inside the Thread class _to_json_dict() method has been
created that converts the current thread data into the corresponding
format expected by the GeckoThread JSON schema, as per the Gecko
profile format specification.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../scripts/python/firefox-gecko-converter.py | 134 +++++++++++++++++-
 1 file changed, 133 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
index bfc8c2b026bf..d9b1ec18997a 100644
--- a/tools/perf/scripts/python/firefox-gecko-converter.py
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -11,7 +11,8 @@
 
 import os
 import sys
-from typing import Dict
+from dataclasses import dataclass, field
+from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
 # Add the Perf-Trace-Util library to the Python path
 sys.path.append(os.environ['PERF_EXEC_PATH'] + \
@@ -20,9 +21,140 @@ sys.path.append(os.environ['PERF_EXEC_PATH'] + \
 from perf_trace_context import *
 from Core import *
 
+StringID = int
+StackID = int
+FrameID = int
+CategoryID = int
+Milliseconds = float
+
 # start_time is intialiazed only once for the all event traces.
 start_time = None
 
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
+class Frame(NamedTuple):
+	string_id: StringID
+	relevantForJS: bool
+	innerWindowID: int
+	implementation: None
+	optimizations: None
+	line: None
+	column: None
+	category: CategoryID
+	subcategory: int
+
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L216
+class Stack(NamedTuple):
+	prefix_id: Optional[StackID]
+	frame_id: FrameID
+
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L90
+class Sample(NamedTuple):
+	stack_id: Optional[StackID]
+	time_ms: Milliseconds
+	responsiveness: int
+
+@dataclass
+class Thread:
+	"""A builder for a profile of the thread.
+
+	Attributes:
+		comm: Thread command-line (name).
+		pid: process ID of containing process.
+		tid: thread ID.
+		samples: Timeline of profile samples.
+		frameTable: interned stack frame ID -> stack frame.
+		stringTable: interned string ID -> string.
+		stringMap: interned string -> string ID.
+		stackTable: interned stack ID -> stack.
+		stackMap: (stack prefix ID, leaf stack frame ID) -> interned Stack ID.
+		frameMap: Stack Frame string -> interned Frame ID.
+		comm: str
+		pid: int
+		tid: int
+		samples: List[Sample] = field(default_factory=list)
+		frameTable: List[Frame] = field(default_factory=list)
+		stringTable: List[str] = field(default_factory=list)
+		stringMap: Dict[str, int] = field(default_factory=dict)
+		stackTable: List[Stack] = field(default_factory=list)
+		stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
+		frameMap: Dict[str, int] = field(default_factory=dict)
+	"""
+	comm: str
+	pid: int
+	tid: int
+	samples: List[Sample] = field(default_factory=list)
+	frameTable: List[Frame] = field(default_factory=list)
+	stringTable: List[str] = field(default_factory=list)
+	stringMap: Dict[str, int] = field(default_factory=dict)
+	stackTable: List[Stack] = field(default_factory=list)
+	stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
+	frameMap: Dict[str, int] = field(default_factory=dict)
+
+	def _to_json_dict(self) -> Dict:
+		"""Converts current Thread to GeckoThread JSON format."""
+		# Gecko profile format is row-oriented data as List[List],
+		# And a schema for interpreting each index.
+		# Schema:
+		# https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
+		# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L230
+		return {
+			"tid": self.tid,
+			"pid": self.pid,
+			"name": self.comm,
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L51
+			"markers": {
+				"schema": {
+					"name": 0,
+					"startTime": 1,
+					"endTime": 2,
+					"phase": 3,
+					"category": 4,
+					"data": 5,
+				},
+				"data": [],
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L90
+			"samples": {
+				"schema": {
+					"stack": 0,
+					"time": 1,
+					"responsiveness": 2,
+				},
+				"data": self.samples
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
+			"frameTable": {
+				"schema": {
+					"location": 0,
+					"relevantForJS": 1,
+					"innerWindowID": 2,
+					"implementation": 3,
+					"optimizations": 4,
+					"line": 5,
+					"column": 6,
+					"category": 7,
+					"subcategory": 8,
+				},
+				"data": self.frameTable,
+			},
+
+			# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L216
+			"stackTable": {
+				"schema": {
+					"prefix": 0,
+					"frame": 1,
+				},
+				"data": self.stackTable,
+			},
+			"stringTable": self.stringTable,
+			"registerTime": 0,
+			"unregisterTime": None,
+			"processType": "default",
+		}
+
+
 # Uses perf script python interface to parse each
 # event and store the data in the thread builder.
 def process_event(param_dict: Dict) -> None:
-- 
2.34.1

[PATCH v4 4/6] perf scripts python: Add trace end processing and PRODUCT and CATEGORIES information

[Anup Sharma]

The final output will now be presented in JSON format following the Gecko
profile structure. Additionally, the inclusion of PRODUCT allows easy retrieval
of header information for UI.

Furthermore, CATEGORIES have been introduced to enable customization of
kernel and user colors using input arguments. To facilitate this functionality,
an argparse-based parser has been implemented.

Note that the implementation of threads will be addressed in subsequent commits."

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../scripts/python/firefox-gecko-converter.py | 64 ++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
index d9b1ec18997a..a0218e2245f2 100644
--- a/tools/perf/scripts/python/firefox-gecko-converter.py
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -11,6 +11,8 @@
 
 import os
 import sys
+import json
+import argparse
 from dataclasses import dataclass, field
 from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
@@ -30,6 +32,13 @@ Milliseconds = float
 # start_time is intialiazed only once for the all event traces.
 start_time = None
 
+# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/profile.js#L425
+# Follow Brendan Gregg's Flamegraph convention: orange for kernel and yellow for user space by default.
+CATEGORIES = None
+
+# The product name is used by the profiler UI to show the Operating system and Processor.
+PRODUCT = os.popen('uname -op').read().strip()
+
 # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
 class Frame(NamedTuple):
 	string_id: StringID
@@ -172,4 +181,57 @@ def process_event(param_dict: Dict) -> None:
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
 def trace_end() -> None:
-	pass
+	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
+	gecko_profile_with_meta = {
+		"meta": {
+			"interval": 1,
+			"processType": 0,
+			"product": PRODUCT,
+			"stackwalk": 1,
+			"debug": 0,
+			"gcpoison": 0,
+			"asyncstack": 1,
+			"startTime": start_time,
+			"shutdownTime": None,
+			"version": 24,
+			"presymbolicated": True,
+			"categories": CATEGORIES,
+			"markerSchema": [],
+			},
+		"libs": [],
+		"threads": threads,
+		"processes": [],
+		"pausedRanges": [],
+	}
+	json.dump(gecko_profile_with_meta, sys.stdout, indent=2)
+
+def main() -> None:
+	global CATEGORIES
+	parser = argparse.ArgumentParser(description="Convert perf.data to Firefox\'s Gecko Profile format")
+
+	# Add the command-line options
+	# Colors must be defined according to this:
+	# https://github.com/firefox-devtools/profiler/blob/50124adbfa488adba6e2674a8f2618cf34b59cd2/res/css/categories.css
+	parser.add_argument('--user-color', default='yellow', help='Color for the User category')
+	parser.add_argument('--kernel-color', default='orange', help='Color for the Kernel category')
+	# Parse the command-line arguments
+	args = parser.parse_args()
+	# Access the values provided by the user
+	user_color = args.user_color
+	kernel_color = args.kernel_color
+
+	CATEGORIES = [
+		{
+			"name": 'User',
+			"color": user_color,
+			"subcategories": ['Other']
+		},
+		{
+			"name": 'Kernel',
+			"color": kernel_color,
+			"subcategories": ['Other']
+		},
+	]
+
+if __name__ == '__main__':
+    main()
-- 
2.34.1

[PATCH v4 5/6] perf scripts python: implement internal get or create frame, stack and string function

[Anup Sharma]

The intern_stack function is responsible for retrieving
or creating a stack_id based on the provided frame_id and prefix_id.
It first generates a key using the frame_id and prefix_id values.
If the stack corresponding to the key is found in the stackMap,
it is returned. Otherwise, a new stack is created by appending
the prefix_id and frame_id to the stackTable. The key
and the index of the newly created stack are added to the
stackMap for future reference.

The _intern_frame function is responsible for retrieving or
creating a frame_id based on the provided frame string. If the frame_id
corresponding to the frameString is found in the frameMap, it is
returned. Otherwise, a new frame is created by appending relevant
information to the frameTable and adding the frameString to the string_id
through _intern_string.

The _intern_string function will gets a matching string, or saves the new
string and returns a String ID.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../scripts/python/firefox-gecko-converter.py | 58 ++++++++++++++++++-
 1 file changed, 57 insertions(+), 1 deletion(-)

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
index a0218e2245f2..ae69c0a4af13 100644
--- a/tools/perf/scripts/python/firefox-gecko-converter.py
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -13,6 +13,7 @@ import os
 import sys
 import json
 import argparse
+from functools import reduce
 from dataclasses import dataclass, field
 from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
 
@@ -39,6 +40,13 @@ CATEGORIES = None
 # The product name is used by the profiler UI to show the Operating system and Processor.
 PRODUCT = os.popen('uname -op').read().strip()
 
+# Here key = tid, value = Thread
+tid_to_thread = dict()
+
+# The category index is used by the profiler UI to show the color of the flame graph.
+USER_CATEGORY_INDEX = 0
+KERNEL_CATEGORY_INDEX = 1
+
 # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
 class Frame(NamedTuple):
 	string_id: StringID
@@ -99,6 +107,55 @@ class Thread:
 	stackMap: Dict[Tuple[Optional[int], int], int] = field(default_factory=dict)
 	frameMap: Dict[str, int] = field(default_factory=dict)
 
+	def _intern_stack(self, frame_id: int, prefix_id: Optional[int]) -> int:
+		"""Gets a matching stack, or saves the new stack. Returns a Stack ID."""
+		key = f"{frame_id}" if prefix_id is None else f"{frame_id},{prefix_id}"
+		# key = (prefix_id, frame_id)
+		stack_id = self.stackMap.get(key)
+		if stack_id is None:
+			# return stack_id
+			stack_id = len(self.stackTable)
+			self.stackTable.append(Stack(prefix_id=prefix_id, frame_id=frame_id))
+			self.stackMap[key] = stack_id
+		return stack_id
+
+	def _intern_string(self, string: str) -> int:
+		"""Gets a matching string, or saves the new string. Returns a String ID."""
+		string_id = self.stringMap.get(string)
+		if string_id is not None:
+			return string_id
+		string_id = len(self.stringTable)
+		self.stringTable.append(string)
+		self.stringMap[string] = string_id
+		return string_id
+
+	def _intern_frame(self, frame_str: str) -> int:
+		"""Gets a matching stack frame, or saves the new frame. Returns a Frame ID."""
+		frame_id = self.frameMap.get(frame_str)
+		if frame_id is not None:
+			return frame_id
+		frame_id = len(self.frameTable)
+		self.frameMap[frame_str] = frame_id
+		string_id = self._intern_string(frame_str)
+
+		symbol_name_to_category = KERNEL_CATEGORY_INDEX if frame_str.find('kallsyms') != -1 \
+		or frame_str.find('/vmlinux') != -1 \
+		or frame_str.endswith('.ko)') \
+		else USER_CATEGORY_INDEX
+
+		self.frameTable.append(Frame(
+			string_id=string_id,
+			relevantForJS=False,
+			innerWindowID=0,
+			implementation=None,
+			optimizations=None,
+			line=None,
+			column=None,
+			category=symbol_name_to_category,
+			subcategory=None,
+		))
+		return frame_id
+
 	def _to_json_dict(self) -> Dict:
 		"""Converts current Thread to GeckoThread JSON format."""
 		# Gecko profile format is row-oriented data as List[List],
@@ -163,7 +220,6 @@ class Thread:
 			"processType": "default",
 		}
 
-
 # Uses perf script python interface to parse each
 # event and store the data in the thread builder.
 def process_event(param_dict: Dict) -> None:
-- 
2.34.1

[PATCH v4 6/6] perf scripts python: Implement add sample function and thread processing

[Anup Sharma]

The stack has been created for storing func and dso from the callchain.
The sample has been added to a specific thread. It first checks if the
thread exists in the Thread class. Then it call _add_sample function
which is responsible for appending a new entry to the samples list.

Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
---
 .../scripts/python/firefox-gecko-converter.py | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
index ae69c0a4af13..9c4393787daa 100644
--- a/tools/perf/scripts/python/firefox-gecko-converter.py
+++ b/tools/perf/scripts/python/firefox-gecko-converter.py
@@ -156,6 +156,25 @@ class Thread:
 		))
 		return frame_id
 
+	def _add_sample(self, comm: str, stack: List[str], time_ms: Milliseconds) -> None:
+		"""Add a timestamped stack trace sample to the thread builder.
+		Args:
+			comm: command-line (name) of the thread at this sample
+			stack: sampled stack frames. Root first, leaf last.
+			time_ms: timestamp of sample in milliseconds.
+		"""
+		# Ihreads may not set their names right after they are created.
+		# Instead, they might do it later. In such situations, to use the latest name they have set.
+		if self.comm != comm:
+			self.comm = comm
+
+		prefix_stack_id = reduce(lambda prefix_id, frame: self._intern_stack
+						(self._intern_frame(frame), prefix_id), stack, None)
+		if prefix_stack_id is not None:
+			self.samples.append(Sample(stack_id=prefix_stack_id,
+									time_ms=time_ms,
+									responsiveness=0))
+
 	def _to_json_dict(self) -> Dict:
 		"""Converts current Thread to GeckoThread JSON format."""
 		# Gecko profile format is row-oriented data as List[List],
@@ -234,9 +253,36 @@ def process_event(param_dict: Dict) -> None:
 	if not start_time:
 		start_time = time_stamp
 
+	# Parse and append the callchain of the current sample into a stack.
+	stack = []
+	if param_dict['callchain']:
+		for call in param_dict['callchain']:
+			if 'sym' not in call:
+				continue
+			stack.append(f'{call["sym"]["name"]} (in {call["dso"]})')
+		if len(stack) != 0:
+			# Reverse the stack, as root come first and the leaf at the end.
+			stack = stack[::-1]
+
+	# During perf record if -g is not used, the callchain is not available.
+	# In that case, the symbol and dso are available in the event parameters.
+	else:
+		func = param_dict['symbol'] if 'symbol' in param_dict else '[unknown]'
+		dso = param_dict['dso'] if 'dso' in param_dict else '[unknown]'
+		stack.append(f'{func} (in {dso})')
+
+	# Add sample to the specific thread.
+	thread = tid_to_thread.get(tid)
+	if thread is None:
+		thread = Thread(comm=comm, pid=pid, tid=tid)
+		tid_to_thread[tid] = thread
+	thread._add_sample(comm=comm, stack=stack, time_ms=time_stamp)
+
 # Trace_end runs at the end and will be used to aggregate
 # the data into the final json object and print it out to stdout.
 def trace_end() -> None:
+	threads = [thread._to_json_dict() for thread in tid_to_thread.values()]
+
 	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
 	gecko_profile_with_meta = {
 		"meta": {
-- 
2.34.1

Re: [PATCH v4 0/6] Add support for Firefox's gecko profile format

[Anup Sharma]

On Wed, Jul 19, 2023 at 04:15:52AM +0530, Anup Sharma wrote:
> This patch series adds support for Firefox's gecko profile format.
> The format is documented here [1].
> 
> I have incorporated several changes based on feedback from the
> previous version of the patch.
> 
> Changes in v3:
a small typo here. It should be v4 instead of v3 (changes in v4).
> - Implemented object-oriented programming (OOP) approach for Thread and Sample
>   to improve code organization and maintainability.
> - Enhanced user experience by introducing argparse for changing color settings
>   of user and kernel symbols during execution.
> - Ensured proper module encapsulation by adding imports wherever necessary.
> - Improved code readability by adding descriptive comments and type annotations.
> 
> These changes focus on adopting OOP principles, enhancing user interaction with argparse,
> and making the code more readable with comments and type information.
> 
> TODO:
> - use cpu_mode to determine the category instead of finding it from symbol name.
> - Write a test.
> - add direct execution command for this script under script/python/bin directory.
> 
> Committer Testing:
> - Tested with a perf.data file generated for single and multiple cpu
>   cores enabled.
> - Uploaded the stdout into profiler.firefox.com and verified the
>   output.
> - Verified the output with the output generated by the existing
>   script as mentioned here [2].
> Method:
> - perf record -F 99 -a -g -- sleep 5
> - perf script firefox-gecko-converter.py > output.json
> - upload output.json to profiler.firefox.com
> 
> [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
> [2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler
> 
> Anup Sharma (6):
>   perf scripts python: Add initial script file with usage information
>   perf scripts python: Extact necessary information from process event
>   perf scripts python: Add classes and conversion functions
>   perf scripts python: Add trace end processing and PRODUCT and
>     CATEGORIES information
>   perf scripts python: implement internal get or create frame, stack and
>     string function
>   perf scripts python: Implement add sample function and thread
>     processing
> 
>  .../scripts/python/firefox-gecko-converter.py | 339 ++++++++++++++++++
>  1 file changed, 339 insertions(+)
>  create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py
> 
> -- 
> 2.34.1
> 

Re: [PATCH v4 0/6] Add support for Firefox's gecko profile format

[Ian Rogers]

On Wed, Jul 19, 2023 at 4:04 AM Anup Sharma <anupnewsmail@gmail.com> wrote:
>
> On Wed, Jul 19, 2023 at 04:15:52AM +0530, Anup Sharma wrote:
> > This patch series adds support for Firefox's gecko profile format.
> > The format is documented here [1].
> >
> > I have incorporated several changes based on feedback from the
> > previous version of the patch.
> >
> > Changes in v3:
> a small typo here. It should be v4 instead of v3 (changes in v4).
> > - Implemented object-oriented programming (OOP) approach for Thread and Sample
> >   to improve code organization and maintainability.
> > - Enhanced user experience by introducing argparse for changing color settings
> >   of user and kernel symbols during execution.
> > - Ensured proper module encapsulation by adding imports wherever necessary.
> > - Improved code readability by adding descriptive comments and type annotations.
> >
> > These changes focus on adopting OOP principles, enhancing user interaction with argparse,
> > and making the code more readable with comments and type information.
> >
> > TODO:
> > - use cpu_mode to determine the category instead of finding it from symbol name.
> > - Write a test.
> > - add direct execution command for this script under script/python/bin directory.
> >

Thanks Anup, is there a minimal TODO so that some code can be merged?
I didn't have any comments for this series so:
Acked-by: Ian Rogers <irogers@google.com>
I think even the test can be follow up as the code isn't on the critical path.

Thanks,
Ian

> > Committer Testing:
> > - Tested with a perf.data file generated for single and multiple cpu
> >   cores enabled.
> > - Uploaded the stdout into profiler.firefox.com and verified the
> >   output.
> > - Verified the output with the output generated by the existing
> >   script as mentioned here [2].
> > Method:
> > - perf record -F 99 -a -g -- sleep 5
> > - perf script firefox-gecko-converter.py > output.json
> > - upload output.json to profiler.firefox.com
> >
> > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
> > [2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler
> >
> > Anup Sharma (6):
> >   perf scripts python: Add initial script file with usage information
> >   perf scripts python: Extact necessary information from process event
> >   perf scripts python: Add classes and conversion functions
> >   perf scripts python: Add trace end processing and PRODUCT and
> >     CATEGORIES information
> >   perf scripts python: implement internal get or create frame, stack and
> >     string function
> >   perf scripts python: Implement add sample function and thread
> >     processing
> >
> >  .../scripts/python/firefox-gecko-converter.py | 339 ++++++++++++++++++
> >  1 file changed, 339 insertions(+)
> >  create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py
> >
> > --
> > 2.34.1
> >

Re: [PATCH v4 0/6] Add support for Firefox's gecko profile format

[Anup Sharma]

On Wed, Jul 19, 2023 at 08:22:24AM -0700, Ian Rogers wrote:
> On Wed, Jul 19, 2023 at 4:04 AM Anup Sharma <anupnewsmail@gmail.com> wrote:
> >
> > On Wed, Jul 19, 2023 at 04:15:52AM +0530, Anup Sharma wrote:
> > > This patch series adds support for Firefox's gecko profile format.
> > > The format is documented here [1].
> > >
> > > I have incorporated several changes based on feedback from the
> > > previous version of the patch.
> > >
> > > Changes in v3:
> > a small typo here. It should be v4 instead of v3 (changes in v4).
> > > - Implemented object-oriented programming (OOP) approach for Thread and Sample
> > >   to improve code organization and maintainability.
> > > - Enhanced user experience by introducing argparse for changing color settings
> > >   of user and kernel symbols during execution.
> > > - Ensured proper module encapsulation by adding imports wherever necessary.
> > > - Improved code readability by adding descriptive comments and type annotations.
> > >
> > > These changes focus on adopting OOP principles, enhancing user interaction with argparse,
> > > and making the code more readable with comments and type information.
> > >
> > > TODO:
> > > - use cpu_mode to determine the category instead of finding it from symbol name.
> > > - Write a test.
> > > - add direct execution command for this script under script/python/bin directory.
> > >
> 
> Thanks Anup, is there a minimal TODO so that some code can be merged?

Yaah, only the first TODO is left, will discuss during office hours.

> I didn't have any comments for this series so:
> Acked-by: Ian Rogers <irogers@google.com>

Thanks for the review.

> I think even the test can be follow up as the code isn't on the critical path.

By the way, I've finished writing the test and will send it very soon.

> Thanks,
> Ian
> 
> > > Committer Testing:
> > > - Tested with a perf.data file generated for single and multiple cpu
> > >   cores enabled.
> > > - Uploaded the stdout into profiler.firefox.com and verified the
> > >   output.
> > > - Verified the output with the output generated by the existing
> > >   script as mentioned here [2].
> > > Method:
> > > - perf record -F 99 -a -g -- sleep 5
> > > - perf script firefox-gecko-converter.py > output.json
> > > - upload output.json to profiler.firefox.com
> > >
> > > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
> > > [2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler
> > >
> > > Anup Sharma (6):
> > >   perf scripts python: Add initial script file with usage information
> > >   perf scripts python: Extact necessary information from process event
> > >   perf scripts python: Add classes and conversion functions
> > >   perf scripts python: Add trace end processing and PRODUCT and
> > >     CATEGORIES information
> > >   perf scripts python: implement internal get or create frame, stack and
> > >     string function
> > >   perf scripts python: Implement add sample function and thread
> > >     processing
> > >
> > >  .../scripts/python/firefox-gecko-converter.py | 339 ++++++++++++++++++
> > >  1 file changed, 339 insertions(+)
> > >  create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py
> > >
> > > --
> > > 2.34.1
> > >

Re: [PATCH v4 4/6] perf scripts python: Add trace end processing and PRODUCT and CATEGORIES information

[Arnaldo Carvalho de Melo]

Em Wed, Jul 19, 2023 at 04:20:08AM +0530, Anup Sharma escreveu:
> The final output will now be presented in JSON format following the Gecko
> profile structure. Additionally, the inclusion of PRODUCT allows easy retrieval
> of header information for UI.
> 
> Furthermore, CATEGORIES have been introduced to enable customization of
> kernel and user colors using input arguments. To facilitate this functionality,
> an argparse-based parser has been implemented.
> 
> Note that the implementation of threads will be addressed in subsequent commits."

It is failing here:

  [root@five ~]# perf script firefox-gecko-converter.py
  Traceback (most recent call last):
    File "/var/home/acme/libexec/perf-core/scripts/python/firefox-gecko-converter.py", line 202, in trace_end
      "threads": threads,
  NameError: name 'threads' is not defined. Did you mean: 'Thread'?
  Fatal Python error: handler_call_die: problem in Python trace event handler
  Python runtime state: initialized
  
  Current thread 0x00007fde43f98f40 (most recent call first):
    <no Python frame>
  
  Extension modules: perf_trace_context (total: 1)
  Aborted (core dumped)
  [root@five ~]#

We need to be able to test it at each changeset, not just at the end of
the patchkit, so please reorganize this so that running:

  perf script firefox-gecko-converter.py

After each patch works.

We need this to bisect problems, etc.

- Arnaldo
 
> Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
> ---
>  .../scripts/python/firefox-gecko-converter.py | 64 ++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
> index d9b1ec18997a..a0218e2245f2 100644
> --- a/tools/perf/scripts/python/firefox-gecko-converter.py
> +++ b/tools/perf/scripts/python/firefox-gecko-converter.py
> @@ -11,6 +11,8 @@
>  
>  import os
>  import sys
> +import json
> +import argparse
>  from dataclasses import dataclass, field
>  from typing import List, Dict, Optional, NamedTuple, Set, Tuple, Any
>  
> @@ -30,6 +32,13 @@ Milliseconds = float
>  # start_time is intialiazed only once for the all event traces.
>  start_time = None
>  
> +# https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/profile.js#L425
> +# Follow Brendan Gregg's Flamegraph convention: orange for kernel and yellow for user space by default.
> +CATEGORIES = None
> +
> +# The product name is used by the profiler UI to show the Operating system and Processor.
> +PRODUCT = os.popen('uname -op').read().strip()
> +
>  # https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L156
>  class Frame(NamedTuple):
>  	string_id: StringID
> @@ -172,4 +181,57 @@ def process_event(param_dict: Dict) -> None:
>  # Trace_end runs at the end and will be used to aggregate
>  # the data into the final json object and print it out to stdout.
>  def trace_end() -> None:
> -	pass
> +	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
> +	gecko_profile_with_meta = {
> +		"meta": {
> +			"interval": 1,
> +			"processType": 0,
> +			"product": PRODUCT,
> +			"stackwalk": 1,
> +			"debug": 0,
> +			"gcpoison": 0,
> +			"asyncstack": 1,
> +			"startTime": start_time,
> +			"shutdownTime": None,
> +			"version": 24,
> +			"presymbolicated": True,
> +			"categories": CATEGORIES,
> +			"markerSchema": [],
> +			},
> +		"libs": [],
> +		"threads": threads,
> +		"processes": [],
> +		"pausedRanges": [],
> +	}
> +	json.dump(gecko_profile_with_meta, sys.stdout, indent=2)
> +
> +def main() -> None:
> +	global CATEGORIES
> +	parser = argparse.ArgumentParser(description="Convert perf.data to Firefox\'s Gecko Profile format")
> +
> +	# Add the command-line options
> +	# Colors must be defined according to this:
> +	# https://github.com/firefox-devtools/profiler/blob/50124adbfa488adba6e2674a8f2618cf34b59cd2/res/css/categories.css
> +	parser.add_argument('--user-color', default='yellow', help='Color for the User category')
> +	parser.add_argument('--kernel-color', default='orange', help='Color for the Kernel category')
> +	# Parse the command-line arguments
> +	args = parser.parse_args()
> +	# Access the values provided by the user
> +	user_color = args.user_color
> +	kernel_color = args.kernel_color
> +
> +	CATEGORIES = [
> +		{
> +			"name": 'User',
> +			"color": user_color,
> +			"subcategories": ['Other']
> +		},
> +		{
> +			"name": 'Kernel',
> +			"color": kernel_color,
> +			"subcategories": ['Other']
> +		},
> +	]
> +
> +if __name__ == '__main__':
> +    main()
> -- 
> 2.34.1
> 

-- 

- Arnaldo

Re: [PATCH v4 6/6] perf scripts python: Implement add sample function and thread processing

[Arnaldo Carvalho de Melo]

Em Wed, Jul 19, 2023 at 04:22:13AM +0530, Anup Sharma escreveu:
> The stack has been created for storing func and dso from the callchain.
> The sample has been added to a specific thread. It first checks if the
> thread exists in the Thread class. Then it call _add_sample function
> which is responsible for appending a new entry to the samples list.

At the end of the series it produces JSON output, I was expecting that
at this point you would provide a way for us to use this output and see
it rendered somewhere, can you add instructions for that in this cset
comment?

Also it isn't working when I try to see just the start of the output:

[root@five ~]# perf script firefox-gecko-converter.py | head
{
  "meta": {
    "interval": 1,
    "processType": 0,
    "product": "x86_64 GNU/Linux",
    "stackwalk": 1,
    "debug": 0,
    "gcpoison": 0,
    "asyncstack": 1,
    "startTime": 6796435.569,
Traceback (most recent call last):
  File "/var/home/acme/libexec/perf-core/scripts/python/firefox-gecko-converter.py", line 308, in trace_end
    json.dump(gecko_profile_with_meta, sys.stdout, indent=2)
  File "/usr/lib64/python3.10/json/__init__.py", line 180, in dump
    fp.write(chunk)
BrokenPipeError: [Errno 32] Broken pipe
Fatal Python error: handler_call_die: problem in Python trace event handler
Python runtime state: initialized

Current thread 0x00007fc21c0ebf40 (most recent call first):
  <no Python frame>

Extension modules: perf_trace_context (total: 1)
[root@five ~]#
 
> Signed-off-by: Anup Sharma <anupnewsmail@gmail.com>
> ---
>  .../scripts/python/firefox-gecko-converter.py | 46 +++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/tools/perf/scripts/python/firefox-gecko-converter.py b/tools/perf/scripts/python/firefox-gecko-converter.py
> index ae69c0a4af13..9c4393787daa 100644
> --- a/tools/perf/scripts/python/firefox-gecko-converter.py
> +++ b/tools/perf/scripts/python/firefox-gecko-converter.py
> @@ -156,6 +156,25 @@ class Thread:
>  		))
>  		return frame_id
>  
> +	def _add_sample(self, comm: str, stack: List[str], time_ms: Milliseconds) -> None:
> +		"""Add a timestamped stack trace sample to the thread builder.
> +		Args:
> +			comm: command-line (name) of the thread at this sample
> +			stack: sampled stack frames. Root first, leaf last.
> +			time_ms: timestamp of sample in milliseconds.
> +		"""
> +		# Ihreads may not set their names right after they are created.
> +		# Instead, they might do it later. In such situations, to use the latest name they have set.
> +		if self.comm != comm:
> +			self.comm = comm
> +
> +		prefix_stack_id = reduce(lambda prefix_id, frame: self._intern_stack
> +						(self._intern_frame(frame), prefix_id), stack, None)
> +		if prefix_stack_id is not None:
> +			self.samples.append(Sample(stack_id=prefix_stack_id,
> +									time_ms=time_ms,
> +									responsiveness=0))
> +
>  	def _to_json_dict(self) -> Dict:
>  		"""Converts current Thread to GeckoThread JSON format."""
>  		# Gecko profile format is row-oriented data as List[List],
> @@ -234,9 +253,36 @@ def process_event(param_dict: Dict) -> None:
>  	if not start_time:
>  		start_time = time_stamp
>  
> +	# Parse and append the callchain of the current sample into a stack.
> +	stack = []
> +	if param_dict['callchain']:
> +		for call in param_dict['callchain']:
> +			if 'sym' not in call:
> +				continue
> +			stack.append(f'{call["sym"]["name"]} (in {call["dso"]})')
> +		if len(stack) != 0:
> +			# Reverse the stack, as root come first and the leaf at the end.
> +			stack = stack[::-1]
> +
> +	# During perf record if -g is not used, the callchain is not available.
> +	# In that case, the symbol and dso are available in the event parameters.
> +	else:
> +		func = param_dict['symbol'] if 'symbol' in param_dict else '[unknown]'
> +		dso = param_dict['dso'] if 'dso' in param_dict else '[unknown]'
> +		stack.append(f'{func} (in {dso})')
> +
> +	# Add sample to the specific thread.
> +	thread = tid_to_thread.get(tid)
> +	if thread is None:
> +		thread = Thread(comm=comm, pid=pid, tid=tid)
> +		tid_to_thread[tid] = thread
> +	thread._add_sample(comm=comm, stack=stack, time_ms=time_stamp)
> +
>  # Trace_end runs at the end and will be used to aggregate
>  # the data into the final json object and print it out to stdout.
>  def trace_end() -> None:
> +	threads = [thread._to_json_dict() for thread in tid_to_thread.values()]
> +
>  	# Schema: https://github.com/firefox-devtools/profiler/blob/53970305b51b9b472e26d7457fee1d66cd4e2737/src/types/gecko-profile.js#L305
>  	gecko_profile_with_meta = {
>  		"meta": {
> -- 
> 2.34.1
> 

-- 

- Arnaldo

Re: [PATCH v4 0/6] Add support for Firefox's gecko profile format

[Arnaldo Carvalho de Melo]

Em Wed, Jul 19, 2023 at 04:34:28PM +0530, Anup Sharma escreveu:
> On Wed, Jul 19, 2023 at 04:15:52AM +0530, Anup Sharma wrote:
> > This patch series adds support for Firefox's gecko profile format.
> > The format is documented here [1].
> > 
> > I have incorporated several changes based on feedback from the
> > previous version of the patch.
> > 
> > Changes in v3:
> a small typo here. It should be v4 instead of v3 (changes in v4).
> > - Implemented object-oriented programming (OOP) approach for Thread and Sample
> >   to improve code organization and maintainability.
> > - Enhanced user experience by introducing argparse for changing color settings
> >   of user and kernel symbols during execution.
> > - Ensured proper module encapsulation by adding imports wherever necessary.
> > - Improved code readability by adding descriptive comments and type annotations.
> > 
> > These changes focus on adopting OOP principles, enhancing user interaction with argparse,
> > and making the code more readable with comments and type information.
> > 
> > TODO:
> > - use cpu_mode to determine the category instead of finding it from symbol name.
> > - Write a test.
> > - add direct execution command for this script under script/python/bin directory.
> > 
> > Committer Testing:
> > - Tested with a perf.data file generated for single and multiple cpu
> >   cores enabled.
> > - Uploaded the stdout into profiler.firefox.com and verified the
> >   output.
> > - Verified the output with the output generated by the existing
> >   script as mentioned here [2].
> > Method:
> > - perf record -F 99 -a -g -- sleep 5
> > - perf script firefox-gecko-converter.py > output.json
> > - upload output.json to profiler.firefox.com

I see it in the cover letter, will try now.

> > [1] https://github.com/firefox-devtools/profiler/blob/main/docs-developer/gecko-profile-format.md
> > [2] https://perf.wiki.kernel.org/index.php/Tutorial#Firefox_Profiler
> > 
> > Anup Sharma (6):
> >   perf scripts python: Add initial script file with usage information
> >   perf scripts python: Extact necessary information from process event
> >   perf scripts python: Add classes and conversion functions
> >   perf scripts python: Add trace end processing and PRODUCT and
> >     CATEGORIES information
> >   perf scripts python: implement internal get or create frame, stack and
> >     string function
> >   perf scripts python: Implement add sample function and thread
> >     processing
> > 
> >  .../scripts/python/firefox-gecko-converter.py | 339 ++++++++++++++++++
> >  1 file changed, 339 insertions(+)
> >  create mode 100644 tools/perf/scripts/python/firefox-gecko-converter.py
> > 
> > -- 
> > 2.34.1
> > 

-- 

- Arnaldo

Re: [PATCH v4 0/6] Add support for Firefox's gecko profile format

[Arnaldo Carvalho de Melo]

Em Thu, Jul 20, 2023 at 12:14:10PM -0300, Arnaldo Carvalho de Melo escreveu:
> Em Wed, Jul 19, 2023 at 04:34:28PM +0530, Anup Sharma escreveu:
> > On Wed, Jul 19, 2023 at 04:15:52AM +0530, Anup Sharma wrote:
> > > This patch series adds support for Firefox's gecko profile format.
> > > The format is documented here [1].
> > > 
> > > I have incorporated several changes based on feedback from the
> > > previous version of the patch.
> > > 
> > > Changes in v3:
> > a small typo here. It should be v4 instead of v3 (changes in v4).
> > > - Implemented object-oriented programming (OOP) approach for Thread and Sample
> > >   to improve code organization and maintainability.
> > > - Enhanced user experience by introducing argparse for changing color settings
> > >   of user and kernel symbols during execution.
> > > - Ensured proper module encapsulation by adding imports wherever necessary.
> > > - Improved code readability by adding descriptive comments and type annotations.
> > > 
> > > These changes focus on adopting OOP principles, enhancing user interaction with argparse,
> > > and making the code more readable with comments and type information.
> > > 
> > > TODO:
> > > - use cpu_mode to determine the category instead of finding it from symbol name.
> > > - Write a test.
> > > - add direct execution command for this script under script/python/bin directory.
> > > 
> > > Committer Testing:
> > > - Tested with a perf.data file generated for single and multiple cpu
> > >   cores enabled.
> > > - Uploaded the stdout into profiler.firefox.com and verified the
> > >   output.
> > > - Verified the output with the output generated by the existing
> > >   script as mentioned here [2].
> > > Method:
> > > - perf record -F 99 -a -g -- sleep 5
> > > - perf script firefox-gecko-converter.py > output.json
> > > - upload output.json to profiler.firefox.com
> 
> I see it in the cover letter, will try now.

I followed the instructions, uploaded a output.json produced and it
seems to work, good job!

- Arnaldo