* Re: [question] smc: how to enable SMC_LO feature
[not found] <8ac15e20beb54acfae1a35d1603c1827@huawei.com>
@ 2023-11-23 2:21 ` Wen Gu
2023-11-23 6:15 ` shaozhengchao
0 siblings, 1 reply; 14+ messages in thread
From: Wen Gu @ 2023-11-23 2:21 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe, guwen
On 2023/11/21 20:14, shaozhengchao wrote:
> Hi Wen Gu:
> Currently, I am interested in the SMC_LOOPBACK feature proposed
> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
> The test result shows that the smc_lo feature cannot be enabled. Here's
> my analysis:
>
> 1. Run the following command to perform the test, and then capture
> packets on the lo device.
> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>
> 2. Use Wireshark to open packets. It is found that the VCE port replies with
> SMC-R-Deline packets.
> [cid:image001.png@01DA1CB4.F1052C30]
>
> 3. Rx
> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>
> 4. Tx
> In smc_clc_send_proposal,
> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>
> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
> always equal to 0.
>
> So, I must call smc_clc_ueid_add function to increase ueid count?
> But I don't see where operations can be added, may I missed something?
>
Hi Zhengchao Shao,
Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
UEID should be set. It can be set by following steps:
- Install smc-tools[1].
- Run # smcd ueid add <ueid> in loopback test environment.
EID works as an ID to indicate the max communication space of SMC. When SEID is
unavailable, an UEID is required.
- Then run the test.
Hope this works for you :)
[1] https://github.com/ibm-s390-linux/smc-tools
Regards,
Wen Gu
> Could you give me some advice? Thanks very much.
>
> Zhengchao Shao
>
>
> [1]link: https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-11-23 2:21 ` [question] smc: how to enable SMC_LO feature Wen Gu
@ 2023-11-23 6:15 ` shaozhengchao
2023-12-04 3:22 ` shaozhengchao
0 siblings, 1 reply; 14+ messages in thread
From: shaozhengchao @ 2023-11-23 6:15 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/11/23 10:21, Wen Gu wrote:
>
>
> On 2023/11/21 20:14, shaozhengchao wrote:
>> Hi Wen Gu:
>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>> my x86_64 environment and kernel is based on linux-next, commit:
>> 5ba73bec5e7b.
>> The test result shows that the smc_lo feature cannot be enabled. Here's
>> my analysis:
>>
>> 1. Run the following command to perform the test, and then capture
>> packets on the lo device.
>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000
>> -i 127.0.0.1 -t 30
>>
>> 2. Use Wireshark to open packets. It is found that the VCE port
>> replies with
>> SMC-R-Deline packets.
>> [cid:image001.png@01DA1CB4.F1052C30]
>>
>> 3. Rx
>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>> a Decline packet because eid_cnt and flag.seid in the received packet
>> are both 0.
>>
>> 4. Tx
>> In smc_clc_send_proposal,
>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>
>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>> seid_enabled is
>> always equal to 0.
>>
>> So, I must call smc_clc_ueid_add function to increase ueid count?
>> But I don't see where operations can be added, may I missed something?
>>
>
> Hi Zhengchao Shao,
>
> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
> UEID should be set. It can be set by following steps:
>
> - Install smc-tools[1].
>
> - Run # smcd ueid add <ueid> in loopback test environment.
>
> EID works as an ID to indicate the max communication space of SMC.
> When SEID is
> unavailable, an UEID is required.
>
Hi Wen Gu:
Thank you for your reply. This is very useful for me. And I will
be happy to learn from it.
Thanks
Zhengchao Shao
> - Then run the test.
>
> Hope this works for you :)
>
> [1] https://github.com/ibm-s390-linux/smc-tools
>
> Regards,
> Wen Gu
>
>> Could you give me some advice? Thanks very much.
>>
>> Zhengchao Shao
>>
>>
>> [1]link:
>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-11-23 6:15 ` shaozhengchao
@ 2023-12-04 3:22 ` shaozhengchao
2023-12-04 3:52 ` Wen Gu
0 siblings, 1 reply; 14+ messages in thread
From: shaozhengchao @ 2023-12-04 3:22 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
[-- Attachment #1: Type: text/plain, Size: 4687 bytes --]
On 2023/11/23 14:15, shaozhengchao wrote:
>
>
> On 2023/11/23 10:21, Wen Gu wrote:
>>
>>
>> On 2023/11/21 20:14, shaozhengchao wrote:
>>> Hi Wen Gu:
>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>> my x86_64 environment and kernel is based on linux-next, commit:
>>> 5ba73bec5e7b.
>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>> my analysis:
>>>
>>> 1. Run the following command to perform the test, and then capture
>>> packets on the lo device.
>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000
>>> -i 127.0.0.1 -t 30
>>>
>>> 2. Use Wireshark to open packets. It is found that the VCE port
>>> replies with
>>> SMC-R-Deline packets.
>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>
>>> 3. Rx
>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>> a Decline packet because eid_cnt and flag.seid in the received packet
>>> are both 0.
>>>
>>> 4. Tx
>>> In smc_clc_send_proposal,
>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>
>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>> seid_enabled is
>>> always equal to 0.
>>>
>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>> But I don't see where operations can be added, may I missed something?
>>>
>>
>> Hi Zhengchao Shao,
>>
>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>> UEID should be set. It can be set by following steps:
>>
>> - Install smc-tools[1].
>>
>> - Run # smcd ueid add <ueid> in loopback test environment.
>>
>> EID works as an ID to indicate the max communication space of SMC.
>> When SEID is
>> unavailable, an UEID is required.
>>
> Hi Wen Gu:
> Thank you for your reply. This is very useful for me. And I will
> be happy to learn from it.
>
> Thanks
>
> Zhengchao Shao
>> - Then run the test.
>>
>> Hope this works for you :)
>>
>> [1] https://github.com/ibm-s390-linux/smc-tools
>>
>> Regards,
>> Wen Gu
>>
>>> Could you give me some advice? Thanks very much.
>>>
>>> Zhengchao Shao
>>>
>>>
>>> [1]link:
>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>
>>
>
>
Hi Wen Gu:
I have test as following, but the performance is really
degraded. Now I have no idea.
1. add ueid
run: smcd ueid add 16
kernel message:
[ 5252.009133] NET: Registered PF_SMC protocol family
[ 5252.009233] smc: adding smcd device smc_lo with pnetid
2. start server
smc_run taskset -c 1 sockperf sr --tcp
3. start client
smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
The test results are as follows:
TCP SMC-lo
Bandwidth(MBps) 1890.56 1300.41(-31.22%)
I didn't find a better direction when I initially positioned it. No
error is recorded in the kernel log, and the smcd statistics are normal.
[root@localhost smc-tools]# smcd stats
SMC-D Connections Summary
Total connections handled 2
SMC connections 2
Handshake errors 0
Avg requests per SMC conn 1277462.0
TCP fallback 0
RX Stats
Data transmitted (Bytes) 40907328000 (40.91G)
Total requests 1277190
Buffer full 45 (0.00%)
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 0 0 2 0 0 0 0
Reqs 638.0K 0 0 639.2K 0 0 0 0
TX Stats
Data transmitted (Bytes) 40907328000 (40.91G)
Total requests 1277734
Buffer full 638239 (49.95%)
Buffer full (remote) 0 (0.00%)
Buffer too small 0 (0.00%)
Buffer too small (remote) 0 (0.00%)
8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
Bufs 0 0 0 0 0 0 0 0
Reqs 0 0 0 1.278M 0 0 0 0
Extras
Special socket calls 1
I captured the perf information and found that the percentage of
rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
during tx and rx.
36.12% [kernel] [k]rep_movs_alternative
14.23% [kernel] [k]_raw_spin_unlock_irqrestore
I've attached the flame map. Could you help analyze it? What I missed?
Thanks.
Zhengchao Shao
[-- Attachment #2: smc.svg --]
[-- Type: image/svg+xml, Size: 61577 bytes --]
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" width="1200" height="422" onload="init(evt)" viewBox="0 0 1200 422" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Flame graph stack visualization. See https://github.com/brendangregg/FlameGraph for latest version, and http://www.brendangregg.com/flamegraphs.html for examples. -->
<!-- NOTES: -->
<defs>
<linearGradient id="background" y1="0" y2="1" x1="0" x2="0" >
<stop stop-color="#eeeeee" offset="5%" />
<stop stop-color="#eeeeb0" offset="95%" />
</linearGradient>
</defs>
<style type="text/css">
text { font-family:Verdana; font-size:12px; fill:rgb(0,0,0); }
#search, #ignorecase { opacity:0.1; cursor:pointer; }
#search:hover, #search.show, #ignorecase:hover, #ignorecase.show { opacity:1; }
#subtitle { text-anchor:middle; font-color:rgb(160,160,160); }
#title { text-anchor:middle; font-size:17px}
#unzoom { cursor:pointer; }
#frames > *:hover { stroke:black; stroke-width:0.5; cursor:pointer; }
.hide { display:none; }
.parent { opacity:0.5; }
</style>
<script type="text/ecmascript">
<![CDATA[
"use strict";
var details, searchbtn, unzoombtn, matchedtxt, svg, searching, currentSearchTerm, ignorecase, ignorecaseBtn;
function init(evt) {
details = document.getElementById("details").firstChild;
searchbtn = document.getElementById("search");
ignorecaseBtn = document.getElementById("ignorecase");
unzoombtn = document.getElementById("unzoom");
matchedtxt = document.getElementById("matched");
svg = document.getElementsByTagName("svg")[0];
searching = 0;
currentSearchTerm = null;
// use GET parameters to restore a flamegraphs state.
var params = get_params();
if (params.x && params.y)
zoom(find_group(document.querySelector('[x="' + params.x + '"][y="' + params.y + '"]')));
if (params.s) search(params.s);
}
// event listeners
window.addEventListener("click", function(e) {
var target = find_group(e.target);
if (target) {
if (target.nodeName == "a") {
if (e.ctrlKey === false) return;
e.preventDefault();
}
if (target.classList.contains("parent")) unzoom();
zoom(target);
if (!document.querySelector('.parent')) {
clearzoom();
return;
}
// set parameters for zoom state
var el = target.querySelector("rect");
if (el && el.attributes && el.attributes.y && el.attributes._orig_x) {
var params = get_params()
params.x = el.attributes._orig_x.value;
params.y = el.attributes.y.value;
history.replaceState(null, null, parse_params(params));
}
}
else if (e.target.id == "unzoom") clearzoom();
else if (e.target.id == "search") search_prompt();
else if (e.target.id == "ignorecase") toggle_ignorecase();
}, false)
// mouse-over for info
// show
window.addEventListener("mouseover", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = "Function: " + g_to_text(target);
}, false)
// clear
window.addEventListener("mouseout", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = ' ';
}, false)
// ctrl-F for search
// ctrl-I to toggle case-sensitive search
window.addEventListener("keydown",function (e) {
if (e.keyCode === 114 || (e.ctrlKey && e.keyCode === 70)) {
e.preventDefault();
search_prompt();
}
else if (e.ctrlKey && e.keyCode === 73) {
e.preventDefault();
toggle_ignorecase();
}
}, false)
// functions
function get_params() {
var params = {};
var paramsarr = window.location.search.substr(1).split('&');
for (var i = 0; i < paramsarr.length; ++i) {
var tmp = paramsarr[i].split("=");
if (!tmp[0] || !tmp[1]) continue;
params[tmp[0]] = decodeURIComponent(tmp[1]);
}
return params;
}
function parse_params(params) {
var uri = "?";
for (var key in params) {
uri += key + '=' + encodeURIComponent(params[key]) + '&';
}
if (uri.slice(-1) == "&")
uri = uri.substring(0, uri.length - 1);
if (uri == '?')
uri = window.location.href.split('?')[0];
return uri;
}
function find_child(node, selector) {
var children = node.querySelectorAll(selector);
if (children.length) return children[0];
}
function find_group(node) {
var parent = node.parentElement;
if (!parent) return;
if (parent.id == "frames") return node;
return find_group(parent);
}
function orig_save(e, attr, val) {
if (e.attributes["_orig_" + attr] != undefined) return;
if (e.attributes[attr] == undefined) return;
if (val == undefined) val = e.attributes[attr].value;
e.setAttribute("_orig_" + attr, val);
}
function orig_load(e, attr) {
if (e.attributes["_orig_"+attr] == undefined) return;
e.attributes[attr].value = e.attributes["_orig_" + attr].value;
e.removeAttribute("_orig_"+attr);
}
function g_to_text(e) {
var text = find_child(e, "title").firstChild.nodeValue;
return (text)
}
function g_to_func(e) {
var func = g_to_text(e);
// if there's any manipulation we want to do to the function
// name before it's searched, do it here before returning.
return (func);
}
function update_text(e) {
var r = find_child(e, "rect");
var t = find_child(e, "text");
var w = parseFloat(r.attributes.width.value) -3;
var txt = find_child(e, "title").textContent.replace(/\([^(]*\)$/,"");
t.attributes.x.value = parseFloat(r.attributes.x.value) + 3;
// Smaller than this size won't fit anything
if (w < 2 * 12 * 0.59) {
t.textContent = "";
return;
}
t.textContent = txt;
// Fit in full text width
if (/^ *$/.test(txt) || t.getSubStringLength(0, txt.length) < w)
return;
for (var x = txt.length - 2; x > 0; x--) {
if (t.getSubStringLength(0, x + 2) <= w) {
t.textContent = txt.substring(0, x) + "..";
return;
}
}
t.textContent = "";
}
// zoom
function zoom_reset(e) {
if (e.attributes != undefined) {
orig_load(e, "x");
orig_load(e, "width");
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_reset(c[i]);
}
}
function zoom_child(e, x, ratio) {
if (e.attributes != undefined) {
if (e.attributes.x != undefined) {
orig_save(e, "x");
e.attributes.x.value = (parseFloat(e.attributes.x.value) - x - 10) * ratio + 10;
if (e.tagName == "text")
e.attributes.x.value = find_child(e.parentNode, "rect[x]").attributes.x.value + 3;
}
if (e.attributes.width != undefined) {
orig_save(e, "width");
e.attributes.width.value = parseFloat(e.attributes.width.value) * ratio;
}
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_child(c[i], x - 10, ratio);
}
}
function zoom_parent(e) {
if (e.attributes) {
if (e.attributes.x != undefined) {
orig_save(e, "x");
e.attributes.x.value = 10;
}
if (e.attributes.width != undefined) {
orig_save(e, "width");
e.attributes.width.value = parseInt(svg.width.baseVal.value) - (10 * 2);
}
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_parent(c[i]);
}
}
function zoom(node) {
var attr = find_child(node, "rect").attributes;
var width = parseFloat(attr.width.value);
var xmin = parseFloat(attr.x.value);
var xmax = parseFloat(xmin + width);
var ymin = parseFloat(attr.y.value);
var ratio = (svg.width.baseVal.value - 2 * 10) / width;
// XXX: Workaround for JavaScript float issues (fix me)
var fudge = 0.0001;
unzoombtn.classList.remove("hide");
var el = document.getElementById("frames").children;
for (var i = 0; i < el.length; i++) {
var e = el[i];
var a = find_child(e, "rect").attributes;
var ex = parseFloat(a.x.value);
var ew = parseFloat(a.width.value);
var upstack;
// Is it an ancestor
if (0 == 0) {
upstack = parseFloat(a.y.value) > ymin;
} else {
upstack = parseFloat(a.y.value) < ymin;
}
if (upstack) {
// Direct ancestor
if (ex <= xmin && (ex+ew+fudge) >= xmax) {
e.classList.add("parent");
zoom_parent(e);
update_text(e);
}
// not in current path
else
e.classList.add("hide");
}
// Children maybe
else {
// no common path
if (ex < xmin || ex + fudge >= xmax) {
e.classList.add("hide");
}
else {
zoom_child(e, xmin, ratio);
update_text(e);
}
}
}
search();
}
function unzoom() {
unzoombtn.classList.add("hide");
var el = document.getElementById("frames").children;
for(var i = 0; i < el.length; i++) {
el[i].classList.remove("parent");
el[i].classList.remove("hide");
zoom_reset(el[i]);
update_text(el[i]);
}
search();
}
function clearzoom() {
unzoom();
// remove zoom state
var params = get_params();
if (params.x) delete params.x;
if (params.y) delete params.y;
history.replaceState(null, null, parse_params(params));
}
// search
function toggle_ignorecase() {
ignorecase = !ignorecase;
if (ignorecase) {
ignorecaseBtn.classList.add("show");
} else {
ignorecaseBtn.classList.remove("show");
}
reset_search();
search();
}
function reset_search() {
var el = document.querySelectorAll("#frames rect");
for (var i = 0; i < el.length; i++) {
orig_load(el[i], "fill")
}
var params = get_params();
delete params.s;
history.replaceState(null, null, parse_params(params));
}
function search_prompt() {
if (!searching) {
var term = prompt("Enter a search term (regexp " +
"allowed, eg: ^ext4_)"
+ (ignorecase ? ", ignoring case" : "")
+ "\nPress Ctrl-i to toggle case sensitivity", "");
if (term != null) search(term);
} else {
reset_search();
searching = 0;
currentSearchTerm = null;
searchbtn.classList.remove("show");
searchbtn.firstChild.nodeValue = "Search"
matchedtxt.classList.add("hide");
matchedtxt.firstChild.nodeValue = ""
}
}
function search(term) {
if (term) currentSearchTerm = term;
var re = new RegExp(currentSearchTerm, ignorecase ? 'i' : '');
var el = document.getElementById("frames").children;
var matches = new Object();
var maxwidth = 0;
for (var i = 0; i < el.length; i++) {
var e = el[i];
var func = g_to_func(e);
var rect = find_child(e, "rect");
if (func == null || rect == null)
continue;
// Save max width. Only works as we have a root frame
var w = parseFloat(rect.attributes.width.value);
if (w > maxwidth)
maxwidth = w;
if (func.match(re)) {
// highlight
var x = parseFloat(rect.attributes.x.value);
orig_save(rect, "fill");
rect.attributes.fill.value = "rgb(230,0,230)";
// remember matches
if (matches[x] == undefined) {
matches[x] = w;
} else {
if (w > matches[x]) {
// overwrite with parent
matches[x] = w;
}
}
searching = 1;
}
}
if (!searching)
return;
var params = get_params();
params.s = currentSearchTerm;
history.replaceState(null, null, parse_params(params));
searchbtn.classList.add("show");
searchbtn.firstChild.nodeValue = "Reset Search";
// calculate percent matched, excluding vertical overlap
var count = 0;
var lastx = -1;
var lastw = 0;
var keys = Array();
for (k in matches) {
if (matches.hasOwnProperty(k))
keys.push(k);
}
// sort the matched frames by their x location
// ascending, then width descending
keys.sort(function(a, b){
return a - b;
});
// Step through frames saving only the biggest bottom-up frames
// thanks to the sort order. This relies on the tree property
// where children are always smaller than their parents.
var fudge = 0.0001; // JavaScript floating point
for (var k in keys) {
var x = parseFloat(keys[k]);
var w = matches[keys[k]];
if (x >= lastx + lastw - fudge) {
count += w;
lastx = x;
lastw = w;
}
}
// display matched percent
matchedtxt.classList.remove("hide");
var pct = 100 * count / maxwidth;
if (pct != 100) pct = pct.toFixed(1)
matchedtxt.firstChild.nodeValue = "Matched: " + pct + "%";
}
]]>
</script>
<rect x="0.0" y="0" width="1200.0" height="422.0" fill="url(#background)" />
<text id="title" x="600.00" y="24" >Flame Graph</text>
<text id="details" x="10.00" y="405" > </text>
<text id="unzoom" x="10.00" y="24" class="hide">Reset Zoom</text>
<text id="search" x="1090.00" y="24" >Search</text>
<text id="ignorecase" x="1174.00" y="24" >ic</text>
<text id="matched" x="1090.00" y="405" > </text>
<g id="frames">
<g >
<title>tick_nohz_idle_exit (54 samples, 0.04%)</title><rect x="1156.7" y="277" width="0.4" height="15.0" fill="rgb(217,101,1)" rx="2" ry="2" />
<text x="1159.71" y="287.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (785 samples, 0.52%)</title><rect x="36.5" y="149" width="6.1" height="15.0" fill="rgb(214,67,34)" rx="2" ry="2" />
<text x="39.48" y="159.5" ></text>
</g>
<g >
<title>copy_page_range (17 samples, 0.01%)</title><rect x="13.9" y="213" width="0.1" height="15.0" fill="rgb(205,176,12)" rx="2" ry="2" />
<text x="16.88" y="223.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (37 samples, 0.02%)</title><rect x="76.3" y="213" width="0.2" height="15.0" fill="rgb(238,95,49)" rx="2" ry="2" />
<text x="79.25" y="223.5" ></text>
</g>
<g >
<title>pmlogger_check (328 samples, 0.22%)</title><rect x="12.8" y="357" width="2.5" height="15.0" fill="rgb(237,214,8)" rx="2" ry="2" />
<text x="15.76" y="367.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (48 samples, 0.03%)</title><rect x="11.1" y="325" width="0.4" height="15.0" fill="rgb(205,158,0)" rx="2" ry="2" />
<text x="14.09" y="335.5" ></text>
</g>
<g >
<title>x86_64_start_kernel (4,168 samples, 2.76%)</title><rect x="1157.1" y="325" width="32.6" height="15.0" fill="rgb(213,82,52)" rx="2" ry="2" />
<text x="1160.13" y="335.5" >x8..</text>
</g>
<g >
<title>smc_sendmsg (3,915 samples, 2.60%)</title><rect x="47.2" y="261" width="30.7" height="15.0" fill="rgb(247,128,15)" rx="2" ry="2" />
<text x="50.24" y="271.5" >sm..</text>
</g>
<g >
<title>asm_sysvec_apic_timer_interrupt (28 samples, 0.02%)</title><rect x="1153.0" y="229" width="0.2" height="15.0" fill="rgb(253,25,3)" rx="2" ry="2" />
<text x="1155.99" y="239.5" ></text>
</g>
<g >
<title>do_syscall_64 (15 samples, 0.01%)</title><rect x="17.0" y="277" width="0.1" height="15.0" fill="rgb(251,24,38)" rx="2" ry="2" />
<text x="20.03" y="287.5" ></text>
</g>
<g >
<title>__pfx_kernel_init (4,168 samples, 2.76%)</title><rect x="1157.1" y="261" width="32.6" height="15.0" fill="rgb(212,121,45)" rx="2" ry="2" />
<text x="1160.13" y="271.5" >__..</text>
</g>
<g >
<title>sockperf (7,488 samples, 4.97%)</title><rect x="20.4" y="357" width="58.6" height="15.0" fill="rgb(207,36,54)" rx="2" ry="2" />
<text x="23.40" y="367.5" >sockperf</text>
</g>
<g >
<title>exc_page_fault (19 samples, 0.01%)</title><rect x="13.1" y="293" width="0.1" height="15.0" fill="rgb(253,131,2)" rx="2" ry="2" />
<text x="16.07" y="303.5" ></text>
</g>
<g >
<title>do_user_addr_fault (21 samples, 0.01%)</title><rect x="19.0" y="261" width="0.2" height="15.0" fill="rgb(248,155,7)" rx="2" ry="2" />
<text x="22.02" y="271.5" ></text>
</g>
<g >
<title>kernel_clone (47 samples, 0.03%)</title><rect x="13.8" y="277" width="0.4" height="15.0" fill="rgb(210,125,34)" rx="2" ry="2" />
<text x="16.82" y="287.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (55 samples, 0.04%)</title><rect x="18.6" y="293" width="0.4" height="15.0" fill="rgb(239,56,28)" rx="2" ry="2" />
<text x="21.58" y="303.5" ></text>
</g>
<g >
<title>_raw_spin_lock (53 samples, 0.04%)</title><rect x="49.2" y="181" width="0.4" height="15.0" fill="rgb(217,50,20)" rx="2" ry="2" />
<text x="52.16" y="191.5" ></text>
</g>
<g >
<title>handle_mm_fault (15 samples, 0.01%)</title><rect x="13.1" y="261" width="0.1" height="15.0" fill="rgb(217,131,32)" rx="2" ry="2" />
<text x="16.10" y="271.5" ></text>
</g>
<g >
<title>_Fork (58 samples, 0.04%)</title><rect x="13.8" y="341" width="0.5" height="15.0" fill="rgb(232,49,14)" rx="2" ry="2" />
<text x="16.81" y="351.5" ></text>
</g>
<g >
<title>_raw_spin_lock_bh (13 samples, 0.01%)</title><rect x="76.6" y="197" width="0.1" height="15.0" fill="rgb(215,190,51)" rx="2" ry="2" />
<text x="79.56" y="207.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (2,499 samples, 1.66%)</title><rect x="24.9" y="229" width="19.5" height="15.0" fill="rgb(206,191,16)" rx="2" ry="2" />
<text x="27.87" y="239.5" ></text>
</g>
<g >
<title>smc_lo_move_data (782 samples, 0.52%)</title><rect x="36.5" y="133" width="6.1" height="15.0" fill="rgb(220,34,13)" rx="2" ry="2" />
<text x="39.51" y="143.5" ></text>
</g>
<g >
<title>__errno_location (57 samples, 0.04%)</title><rect x="21.2" y="341" width="0.5" height="15.0" fill="rgb(230,1,43)" rx="2" ry="2" />
<text x="24.21" y="351.5" ></text>
</g>
<g >
<title>__do_softirq (28 samples, 0.02%)</title><rect x="1153.0" y="181" width="0.2" height="15.0" fill="rgb(216,120,32)" rx="2" ry="2" />
<text x="1155.99" y="191.5" ></text>
</g>
<g >
<title>load_elf_binary (25 samples, 0.02%)</title><rect x="14.7" y="213" width="0.2" height="15.0" fill="rgb(225,180,50)" rx="2" ry="2" />
<text x="17.71" y="223.5" ></text>
</g>
<g >
<title>__handle_mm_fault (15 samples, 0.01%)</title><rect x="13.1" y="245" width="0.1" height="15.0" fill="rgb(236,33,5)" rx="2" ry="2" />
<text x="16.10" y="255.5" ></text>
</g>
<g >
<title>do_execveat_common (15 samples, 0.01%)</title><rect x="13.5" y="261" width="0.1" height="15.0" fill="rgb(234,160,28)" rx="2" ry="2" />
<text x="16.45" y="271.5" ></text>
</g>
<g >
<title>ksys_mmap_pgoff (41 samples, 0.03%)</title><rect x="15.7" y="293" width="0.3" height="15.0" fill="rgb(231,60,28)" rx="2" ry="2" />
<text x="18.72" y="303.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (45 samples, 0.03%)</title><rect x="15.7" y="325" width="0.4" height="15.0" fill="rgb(224,204,38)" rx="2" ry="2" />
<text x="18.71" y="335.5" ></text>
</g>
<g >
<title>rcu_core (19 samples, 0.01%)</title><rect x="1153.0" y="165" width="0.1" height="15.0" fill="rgb(239,224,50)" rx="2" ry="2" />
<text x="1155.99" y="175.5" ></text>
</g>
<g >
<title>__x64_sys_sendto (4,045 samples, 2.68%)</title><rect x="46.7" y="293" width="31.7" height="15.0" fill="rgb(218,130,23)" rx="2" ry="2" />
<text x="49.74" y="303.5" >__..</text>
</g>
<g >
<title>security_socket_recvmsg (112 samples, 0.07%)</title><rect x="23.7" y="245" width="0.8" height="15.0" fill="rgb(205,107,18)" rx="2" ry="2" />
<text x="26.66" y="255.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (17 samples, 0.01%)</title><rect x="16.7" y="277" width="0.1" height="15.0" fill="rgb(211,162,47)" rx="2" ry="2" />
<text x="19.71" y="287.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (89 samples, 0.06%)</title><rect x="23.8" y="197" width="0.7" height="15.0" fill="rgb(206,85,41)" rx="2" ry="2" />
<text x="26.84" y="207.5" ></text>
</g>
<g >
<title>selinux_socket_recvmsg (111 samples, 0.07%)</title><rect x="23.7" y="229" width="0.8" height="15.0" fill="rgb(225,158,0)" rx="2" ry="2" />
<text x="26.67" y="239.5" ></text>
</g>
<g >
<title>move_addr_to_user (21 samples, 0.01%)</title><rect x="23.5" y="261" width="0.1" height="15.0" fill="rgb(240,76,11)" rx="2" ry="2" />
<text x="26.45" y="271.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv_action (711 samples, 0.47%)</title><rect x="37.1" y="85" width="5.5" height="15.0" fill="rgb(240,100,47)" rx="2" ry="2" />
<text x="40.06" y="95.5" ></text>
</g>
<g >
<title>schedule_timeout (146 samples, 0.10%)</title><rect x="76.7" y="197" width="1.2" height="15.0" fill="rgb(248,170,45)" rx="2" ry="2" />
<text x="79.74" y="207.5" ></text>
</g>
<g >
<title>__wake_up (660 samples, 0.44%)</title><rect x="37.5" y="53" width="5.1" height="15.0" fill="rgb(237,24,32)" rx="2" ry="2" />
<text x="40.46" y="63.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (16 samples, 0.01%)</title><rect x="19.6" y="341" width="0.1" height="15.0" fill="rgb(219,213,27)" rx="2" ry="2" />
<text x="22.60" y="351.5" ></text>
</g>
<g >
<title>search_binary_handler (25 samples, 0.02%)</title><rect x="14.7" y="229" width="0.2" height="15.0" fill="rgb(219,70,33)" rx="2" ry="2" />
<text x="17.71" y="239.5" ></text>
</g>
<g >
<title>exc_page_fault (23 samples, 0.02%)</title><rect x="16.8" y="277" width="0.2" height="15.0" fill="rgb(245,148,24)" rx="2" ry="2" />
<text x="19.85" y="287.5" ></text>
</g>
<g >
<title>wait_woken (154 samples, 0.10%)</title><rect x="76.7" y="213" width="1.2" height="15.0" fill="rgb(223,170,5)" rx="2" ry="2" />
<text x="79.67" y="223.5" ></text>
</g>
<g >
<title>lock_sock_nested (14 samples, 0.01%)</title><rect x="76.5" y="213" width="0.2" height="15.0" fill="rgb(243,105,42)" rx="2" ry="2" />
<text x="79.55" y="223.5" ></text>
</g>
<g >
<title>rep_movs_alternative (1,204 samples, 0.80%)</title><rect x="26.9" y="197" width="9.4" height="15.0" fill="rgb(220,75,34)" rx="2" ry="2" />
<text x="29.87" y="207.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (19 samples, 0.01%)</title><rect x="11.2" y="293" width="0.1" height="15.0" fill="rgb(248,33,18)" rx="2" ry="2" />
<text x="14.20" y="303.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (3,864 samples, 2.56%)</title><rect x="47.6" y="245" width="30.3" height="15.0" fill="rgb(231,171,54)" rx="2" ry="2" />
<text x="50.64" y="255.5" >sm..</text>
</g>
<g >
<title>do_syscall_64 (17 samples, 0.01%)</title><rect x="16.7" y="261" width="0.1" height="15.0" fill="rgb(211,75,30)" rx="2" ry="2" />
<text x="19.71" y="271.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (19 samples, 0.01%)</title><rect x="13.1" y="309" width="0.1" height="15.0" fill="rgb(226,110,28)" rx="2" ry="2" />
<text x="16.07" y="319.5" ></text>
</g>
<g >
<title>do_exit (16 samples, 0.01%)</title><rect x="19.6" y="277" width="0.1" height="15.0" fill="rgb(250,127,48)" rx="2" ry="2" />
<text x="22.60" y="287.5" ></text>
</g>
<g >
<title>do_group_exit (16 samples, 0.01%)</title><rect x="19.6" y="293" width="0.1" height="15.0" fill="rgb(205,11,45)" rx="2" ry="2" />
<text x="22.60" y="303.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (26 samples, 0.02%)</title><rect x="47.0" y="213" width="0.2" height="15.0" fill="rgb(234,32,13)" rx="2" ry="2" />
<text x="50.03" y="223.5" ></text>
</g>
<g >
<title>__mmput (17 samples, 0.01%)</title><rect x="17.6" y="245" width="0.1" height="15.0" fill="rgb(231,95,33)" rx="2" ry="2" />
<text x="20.59" y="255.5" ></text>
</g>
<g >
<title>do_mprotect_pkey (15 samples, 0.01%)</title><rect x="16.7" y="229" width="0.1" height="15.0" fill="rgb(210,13,43)" rx="2" ry="2" />
<text x="19.71" y="239.5" ></text>
</g>
<g >
<title>__x64_sys_execve (15 samples, 0.01%)</title><rect x="13.5" y="277" width="0.1" height="15.0" fill="rgb(206,143,4)" rx="2" ry="2" />
<text x="16.45" y="287.5" ></text>
</g>
<g >
<title>smc_cdc_get_slot_and_msg_send (804 samples, 0.53%)</title><rect x="36.3" y="181" width="6.3" height="15.0" fill="rgb(232,72,4)" rx="2" ry="2" />
<text x="39.34" y="191.5" ></text>
</g>
<g >
<title>smc_tx_rdma_writes (55 samples, 0.04%)</title><rect x="68.2" y="197" width="0.4" height="15.0" fill="rgb(218,55,26)" rx="2" ry="2" />
<text x="71.16" y="207.5" ></text>
</g>
<g >
<title>schedule (111 samples, 0.07%)</title><rect x="43.6" y="165" width="0.8" height="15.0" fill="rgb(218,94,18)" rx="2" ry="2" />
<text x="46.56" y="175.5" ></text>
</g>
<g >
<title>do_syscall_64 (16 samples, 0.01%)</title><rect x="19.6" y="325" width="0.1" height="15.0" fill="rgb(226,40,52)" rx="2" ry="2" />
<text x="22.60" y="335.5" ></text>
</g>
<g >
<title>x86_64_start_reservations (4,168 samples, 2.76%)</title><rect x="1157.1" y="309" width="32.6" height="15.0" fill="rgb(246,24,3)" rx="2" ry="2" />
<text x="1160.13" y="319.5" >x8..</text>
</g>
<g >
<title>[[kernel.vmlinux].init.text] (4,168 samples, 2.76%)</title><rect x="1157.1" y="277" width="32.6" height="15.0" fill="rgb(206,71,43)" rx="2" ry="2" />
<text x="1160.13" y="287.5" >[[..</text>
</g>
<g >
<title>sock_recvmsg (2,659 samples, 1.76%)</title><rect x="23.6" y="261" width="20.8" height="15.0" fill="rgb(225,162,25)" rx="2" ry="2" />
<text x="26.62" y="271.5" ></text>
</g>
<g >
<title>vm_mmap_pgoff (40 samples, 0.03%)</title><rect x="15.7" y="277" width="0.3" height="15.0" fill="rgb(237,5,0)" rx="2" ry="2" />
<text x="18.73" y="287.5" ></text>
</g>
<g >
<title>do_vmi_munmap (18 samples, 0.01%)</title><rect x="15.8" y="229" width="0.1" height="15.0" fill="rgb(225,69,35)" rx="2" ry="2" />
<text x="18.75" y="239.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (206 samples, 0.14%)</title><rect x="25.2" y="213" width="1.7" height="15.0" fill="rgb(252,217,15)" rx="2" ry="2" />
<text x="28.24" y="223.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (28 samples, 0.02%)</title><rect x="10.8" y="341" width="0.2" height="15.0" fill="rgb(221,215,39)" rx="2" ry="2" />
<text x="13.81" y="351.5" ></text>
</g>
<g >
<title>sed (26 samples, 0.02%)</title><rect x="20.2" y="357" width="0.2" height="15.0" fill="rgb(209,87,30)" rx="2" ry="2" />
<text x="23.20" y="367.5" ></text>
</g>
<g >
<title>schedule_idle (371 samples, 0.25%)</title><rect x="1153.4" y="277" width="2.9" height="15.0" fill="rgb(211,123,8)" rx="2" ry="2" />
<text x="1156.40" y="287.5" ></text>
</g>
<g >
<title>do_syscall_64 (48 samples, 0.03%)</title><rect x="13.8" y="309" width="0.4" height="15.0" fill="rgb(209,188,36)" rx="2" ry="2" />
<text x="16.81" y="319.5" ></text>
</g>
<g >
<title>__mmput (21 samples, 0.01%)</title><rect x="14.7" y="165" width="0.2" height="15.0" fill="rgb(249,117,24)" rx="2" ry="2" />
<text x="17.72" y="175.5" ></text>
</g>
<g >
<title>[unknown] (110 samples, 0.07%)</title><rect x="13.0" y="341" width="0.8" height="15.0" fill="rgb(208,113,30)" rx="2" ry="2" />
<text x="15.95" y="351.5" ></text>
</g>
<g >
<title>irq_exit_rcu (28 samples, 0.02%)</title><rect x="1153.0" y="197" width="0.2" height="15.0" fill="rgb(239,81,11)" rx="2" ry="2" />
<text x="1155.99" y="207.5" ></text>
</g>
<g >
<title>exc_page_fault (20 samples, 0.01%)</title><rect x="16.6" y="261" width="0.1" height="15.0" fill="rgb(208,181,33)" rx="2" ry="2" />
<text x="19.55" y="271.5" ></text>
</g>
<g >
<title>avc_lookup (26 samples, 0.02%)</title><rect x="47.0" y="197" width="0.2" height="15.0" fill="rgb(220,113,46)" rx="2" ry="2" />
<text x="50.03" y="207.5" ></text>
</g>
<g >
<title>client_handler<IoRecvfrom, SwitchOff, PongModeNever> (56 samples, 0.04%)</title><rect x="21.7" y="341" width="0.5" height="15.0" fill="rgb(218,10,35)" rx="2" ry="2" />
<text x="24.72" y="351.5" ></text>
</g>
<g >
<title>do_syscall_64 (17 samples, 0.01%)</title><rect x="17.6" y="325" width="0.1" height="15.0" fill="rgb(226,54,14)" rx="2" ry="2" />
<text x="20.59" y="335.5" ></text>
</g>
<g >
<title>start_secondary (137,750 samples, 91.36%)</title><rect x="79.0" y="325" width="1078.1" height="15.0" fill="rgb(231,220,45)" rx="2" ry="2" />
<text x="82.03" y="335.5" >start_secondary</text>
</g>
<g >
<title>sysvec_call_function_single (89 samples, 0.06%)</title><rect x="1189.1" y="149" width="0.6" height="15.0" fill="rgb(231,192,52)" rx="2" ry="2" />
<text x="1192.05" y="159.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv_action (836 samples, 0.55%)</title><rect x="69.6" y="117" width="6.6" height="15.0" fill="rgb(240,137,43)" rx="2" ry="2" />
<text x="72.63" y="127.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (23 samples, 0.02%)</title><rect x="16.8" y="293" width="0.2" height="15.0" fill="rgb(249,168,50)" rx="2" ry="2" />
<text x="19.85" y="303.5" ></text>
</g>
<g >
<title>lock_sock_nested (33 samples, 0.02%)</title><rect x="43.0" y="197" width="0.2" height="15.0" fill="rgb(218,40,37)" rx="2" ry="2" />
<text x="45.97" y="207.5" ></text>
</g>
<g >
<title>avc_has_perm (27 samples, 0.02%)</title><rect x="47.0" y="229" width="0.2" height="15.0" fill="rgb(211,188,10)" rx="2" ry="2" />
<text x="50.03" y="239.5" ></text>
</g>
<g >
<title>irq_exit_rcu (89 samples, 0.06%)</title><rect x="1189.1" y="133" width="0.6" height="15.0" fill="rgb(226,218,3)" rx="2" ry="2" />
<text x="1192.05" y="143.5" ></text>
</g>
<g >
<title>find_vmap_area (203 samples, 0.13%)</title><rect x="25.2" y="181" width="1.6" height="15.0" fill="rgb(230,27,8)" rx="2" ry="2" />
<text x="28.25" y="191.5" ></text>
</g>
<g >
<title>bprm_execve.part.0 (25 samples, 0.02%)</title><rect x="14.7" y="261" width="0.2" height="15.0" fill="rgb(248,30,21)" rx="2" ry="2" />
<text x="17.71" y="271.5" ></text>
</g>
<g >
<title>__get_user_4 (17 samples, 0.01%)</title><rect x="23.5" y="245" width="0.1" height="15.0" fill="rgb(222,221,19)" rx="2" ry="2" />
<text x="26.48" y="255.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (23 samples, 0.02%)</title><rect x="19.4" y="309" width="0.1" height="15.0" fill="rgb(205,209,34)" rx="2" ry="2" />
<text x="22.35" y="319.5" ></text>
</g>
<g >
<title>__sys_sendto (4,045 samples, 2.68%)</title><rect x="46.7" y="277" width="31.7" height="15.0" fill="rgb(219,128,42)" rx="2" ry="2" />
<text x="49.74" y="287.5" >__..</text>
</g>
<g >
<title>execve (25 samples, 0.02%)</title><rect x="14.7" y="341" width="0.2" height="15.0" fill="rgb(236,41,39)" rx="2" ry="2" />
<text x="17.71" y="351.5" ></text>
</g>
<g >
<title>exit_mmap (21 samples, 0.01%)</title><rect x="14.7" y="149" width="0.2" height="15.0" fill="rgb(232,219,46)" rx="2" ry="2" />
<text x="17.72" y="159.5" ></text>
</g>
<g >
<title>smc_rx_update_consumer (809 samples, 0.54%)</title><rect x="36.3" y="213" width="6.3" height="15.0" fill="rgb(217,223,51)" rx="2" ry="2" />
<text x="39.30" y="223.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (44 samples, 0.03%)</title><rect x="18.0" y="325" width="0.4" height="15.0" fill="rgb(219,64,26)" rx="2" ry="2" />
<text x="21.05" y="335.5" ></text>
</g>
<g >
<title>__fget_light (63 samples, 0.04%)</title><rect x="44.5" y="245" width="0.5" height="15.0" fill="rgb(244,117,36)" rx="2" ry="2" />
<text x="47.47" y="255.5" ></text>
</g>
<g >
<title>do_group_exit (17 samples, 0.01%)</title><rect x="17.6" y="293" width="0.1" height="15.0" fill="rgb(248,139,30)" rx="2" ry="2" />
<text x="20.59" y="303.5" ></text>
</g>
<g >
<title>do_idle (4,168 samples, 2.76%)</title><rect x="1157.1" y="229" width="32.6" height="15.0" fill="rgb(216,152,1)" rx="2" ry="2" />
<text x="1160.13" y="239.5" >do..</text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (102 samples, 0.07%)</title><rect x="16.4" y="309" width="0.8" height="15.0" fill="rgb(224,120,11)" rx="2" ry="2" />
<text x="19.36" y="319.5" ></text>
</g>
<g >
<title>__x64_sys_recvfrom (2,777 samples, 1.84%)</title><rect x="23.2" y="293" width="21.8" height="15.0" fill="rgb(240,207,44)" rx="2" ry="2" />
<text x="26.23" y="303.5" >_..</text>
</g>
<g >
<title>[libc.so.6] (26 samples, 0.02%)</title><rect x="13.0" y="325" width="0.2" height="15.0" fill="rgb(251,145,9)" rx="2" ry="2" />
<text x="16.01" y="335.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (2,894 samples, 1.92%)</title><rect x="23.2" y="325" width="22.7" height="15.0" fill="rgb(228,43,7)" rx="2" ry="2" />
<text x="26.23" y="335.5" >e..</text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (94 samples, 0.06%)</title><rect x="15.3" y="341" width="0.8" height="15.0" fill="rgb(211,107,7)" rx="2" ry="2" />
<text x="18.33" y="351.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (83 samples, 0.06%)</title><rect x="17.7" y="341" width="0.7" height="15.0" fill="rgb(242,178,12)" rx="2" ry="2" />
<text x="20.74" y="351.5" ></text>
</g>
<g >
<title>asm_sysvec_call_function_single (89 samples, 0.06%)</title><rect x="1189.1" y="165" width="0.6" height="15.0" fill="rgb(230,121,48)" rx="2" ry="2" />
<text x="1192.05" y="175.5" ></text>
</g>
<g >
<title>do_mmap (39 samples, 0.03%)</title><rect x="15.7" y="261" width="0.3" height="15.0" fill="rgb(215,110,53)" rx="2" ry="2" />
<text x="18.73" y="271.5" ></text>
</g>
<g >
<title>do_user_addr_fault (20 samples, 0.01%)</title><rect x="16.6" y="245" width="0.1" height="15.0" fill="rgb(206,120,26)" rx="2" ry="2" />
<text x="19.55" y="255.5" ></text>
</g>
<g >
<title>pmlogger (45 samples, 0.03%)</title><rect x="12.4" y="357" width="0.4" height="15.0" fill="rgb(252,102,30)" rx="2" ry="2" />
<text x="15.41" y="367.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (117 samples, 0.08%)</title><rect x="45.0" y="293" width="0.9" height="15.0" fill="rgb(232,93,5)" rx="2" ry="2" />
<text x="47.96" y="303.5" ></text>
</g>
<g >
<title>do_syscall_64 (4,123 samples, 2.73%)</title><rect x="46.7" y="309" width="32.3" height="15.0" fill="rgb(235,184,26)" rx="2" ry="2" />
<text x="49.74" y="319.5" >do..</text>
</g>
<g >
<title>[unknown] (24 samples, 0.02%)</title><rect x="17.2" y="325" width="0.2" height="15.0" fill="rgb(208,11,35)" rx="2" ry="2" />
<text x="20.18" y="335.5" ></text>
</g>
<g >
<title>__mmput (16 samples, 0.01%)</title><rect x="19.6" y="245" width="0.1" height="15.0" fill="rgb(222,76,2)" rx="2" ry="2" />
<text x="22.60" y="255.5" ></text>
</g>
<g >
<title>rcu_do_batch (19 samples, 0.01%)</title><rect x="1153.0" y="149" width="0.1" height="15.0" fill="rgb(220,2,22)" rx="2" ry="2" />
<text x="1155.99" y="159.5" ></text>
</g>
<g >
<title>do_syscall_64 (45 samples, 0.03%)</title><rect x="15.7" y="309" width="0.4" height="15.0" fill="rgb(209,75,24)" rx="2" ry="2" />
<text x="18.71" y="319.5" ></text>
</g>
<g >
<title>sysvec_apic_timer_interrupt (28 samples, 0.02%)</title><rect x="1153.0" y="213" width="0.2" height="15.0" fill="rgb(248,17,29)" rx="2" ry="2" />
<text x="1155.99" y="223.5" ></text>
</g>
<g >
<title>__sys_recvfrom (2,777 samples, 1.84%)</title><rect x="23.2" y="277" width="21.8" height="15.0" fill="rgb(249,88,23)" rx="2" ry="2" />
<text x="26.23" y="287.5" >_..</text>
</g>
<g >
<title>[vmlinux] (16 samples, 0.01%)</title><rect x="19.6" y="309" width="0.1" height="15.0" fill="rgb(231,131,52)" rx="2" ry="2" />
<text x="22.60" y="319.5" ></text>
</g>
<g >
<title>exit_mmap (17 samples, 0.01%)</title><rect x="17.6" y="229" width="0.1" height="15.0" fill="rgb(241,185,36)" rx="2" ry="2" />
<text x="20.59" y="239.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (34 samples, 0.02%)</title><rect x="10.3" y="325" width="0.3" height="15.0" fill="rgb(245,222,24)" rx="2" ry="2" />
<text x="13.33" y="335.5" ></text>
</g>
<g >
<title>do_exit (17 samples, 0.01%)</title><rect x="17.6" y="277" width="0.1" height="15.0" fill="rgb(219,141,0)" rx="2" ry="2" />
<text x="20.59" y="287.5" ></text>
</g>
<g >
<title>dup_mmap (25 samples, 0.02%)</title><rect x="13.9" y="229" width="0.2" height="15.0" fill="rgb(245,70,9)" rx="2" ry="2" />
<text x="16.87" y="239.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (25 samples, 0.02%)</title><rect x="10.4" y="309" width="0.2" height="15.0" fill="rgb(209,162,18)" rx="2" ry="2" />
<text x="13.38" y="319.5" ></text>
</g>
<g >
<title>grep (28 samples, 0.02%)</title><rect x="11.7" y="357" width="0.2" height="15.0" fill="rgb(217,46,1)" rx="2" ry="2" />
<text x="14.71" y="367.5" ></text>
</g>
<g >
<title>exc_page_fault (21 samples, 0.01%)</title><rect x="19.0" y="277" width="0.2" height="15.0" fill="rgb(254,197,42)" rx="2" ry="2" />
<text x="22.02" y="287.5" ></text>
</g>
<g >
<title>cpuidle_idle_call (4,168 samples, 2.76%)</title><rect x="1157.1" y="213" width="32.6" height="15.0" fill="rgb(226,34,50)" rx="2" ry="2" />
<text x="1160.13" y="223.5" >cp..</text>
</g>
<g >
<title>schedule (131 samples, 0.09%)</title><rect x="76.9" y="181" width="1.0" height="15.0" fill="rgb(224,184,18)" rx="2" ry="2" />
<text x="79.85" y="191.5" ></text>
</g>
<g >
<title>handle_mm_fault (20 samples, 0.01%)</title><rect x="16.9" y="245" width="0.1" height="15.0" fill="rgb(237,5,3)" rx="2" ry="2" />
<text x="19.86" y="255.5" ></text>
</g>
<g >
<title>do_user_addr_fault (19 samples, 0.01%)</title><rect x="13.1" y="277" width="0.1" height="15.0" fill="rgb(246,128,33)" rx="2" ry="2" />
<text x="16.07" y="287.5" ></text>
</g>
<g >
<title>_raw_spin_lock (32 samples, 0.02%)</title><rect x="26.5" y="165" width="0.3" height="15.0" fill="rgb(215,157,30)" rx="2" ry="2" />
<text x="29.54" y="175.5" ></text>
</g>
<g >
<title>mmap_region (35 samples, 0.02%)</title><rect x="15.8" y="245" width="0.2" height="15.0" fill="rgb(229,199,48)" rx="2" ry="2" />
<text x="18.75" y="255.5" ></text>
</g>
<g >
<title>handle_mm_fault (17 samples, 0.01%)</title><rect x="18.8" y="229" width="0.1" height="15.0" fill="rgb(213,8,18)" rx="2" ry="2" />
<text x="21.78" y="239.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (20 samples, 0.01%)</title><rect x="16.6" y="277" width="0.1" height="15.0" fill="rgb(232,10,47)" rx="2" ry="2" />
<text x="19.55" y="287.5" ></text>
</g>
<g >
<title>[libc.so.6] (17 samples, 0.01%)</title><rect x="12.8" y="341" width="0.2" height="15.0" fill="rgb(218,11,48)" rx="2" ry="2" />
<text x="15.82" y="351.5" ></text>
</g>
<g >
<title>[unknown] (30 samples, 0.02%)</title><rect x="19.4" y="325" width="0.2" height="15.0" fill="rgb(237,122,7)" rx="2" ry="2" />
<text x="22.35" y="335.5" ></text>
</g>
<g >
<title>_copy_to_iter (1,206 samples, 0.80%)</title><rect x="26.9" y="213" width="9.4" height="15.0" fill="rgb(229,17,18)" rx="2" ry="2" />
<text x="29.85" y="223.5" ></text>
</g>
<g >
<title>selinux_socket_sendmsg (44 samples, 0.03%)</title><rect x="46.9" y="245" width="0.3" height="15.0" fill="rgb(246,30,25)" rx="2" ry="2" />
<text x="49.89" y="255.5" ></text>
</g>
<g >
<title>do_vmi_align_munmap (18 samples, 0.01%)</title><rect x="15.8" y="213" width="0.1" height="15.0" fill="rgb(219,28,50)" rx="2" ry="2" />
<text x="18.75" y="223.5" ></text>
</g>
<g >
<title>ksys_mmap_pgoff (38 samples, 0.03%)</title><rect x="18.1" y="293" width="0.3" height="15.0" fill="rgb(221,134,30)" rx="2" ry="2" />
<text x="21.05" y="303.5" ></text>
</g>
<g >
<title>tick_nohz_idle_enter (51 samples, 0.03%)</title><rect x="1156.3" y="277" width="0.4" height="15.0" fill="rgb(223,197,49)" rx="2" ry="2" />
<text x="1159.31" y="287.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (109 samples, 0.07%)</title><rect x="77.0" y="149" width="0.9" height="15.0" fill="rgb(214,126,6)" rx="2" ry="2" />
<text x="80.03" y="159.5" ></text>
</g>
<g >
<title>__handle_mm_fault (17 samples, 0.01%)</title><rect x="18.8" y="213" width="0.1" height="15.0" fill="rgb(221,16,21)" rx="2" ry="2" />
<text x="21.78" y="223.5" ></text>
</g>
<g >
<title>smc_lo_move_data (950 samples, 0.63%)</title><rect x="68.7" y="165" width="7.5" height="15.0" fill="rgb(237,92,27)" rx="2" ry="2" />
<text x="71.74" y="175.5" ></text>
</g>
<g >
<title>do_execveat_common (25 samples, 0.02%)</title><rect x="14.7" y="277" width="0.2" height="15.0" fill="rgb(224,33,27)" rx="2" ry="2" />
<text x="17.71" y="287.5" ></text>
</g>
<g >
<title>check_heap_object (187 samples, 0.12%)</title><rect x="48.2" y="213" width="1.4" height="15.0" fill="rgb(211,193,31)" rx="2" ry="2" />
<text x="51.15" y="223.5" ></text>
</g>
<g >
<title>handle_mm_fault (15 samples, 0.01%)</title><rect x="16.6" y="229" width="0.1" height="15.0" fill="rgb(222,109,18)" rx="2" ry="2" />
<text x="19.57" y="239.5" ></text>
</g>
<g >
<title>schedule_timeout (115 samples, 0.08%)</title><rect x="43.5" y="181" width="0.9" height="15.0" fill="rgb(244,161,32)" rx="2" ry="2" />
<text x="46.53" y="191.5" ></text>
</g>
<g >
<title>__fget_light (65 samples, 0.04%)</title><rect x="77.9" y="245" width="0.5" height="15.0" fill="rgb(232,161,5)" rx="2" ry="2" />
<text x="80.89" y="255.5" ></text>
</g>
<g >
<title>[vmlinux] (17 samples, 0.01%)</title><rect x="17.6" y="309" width="0.1" height="15.0" fill="rgb(221,17,35)" rx="2" ry="2" />
<text x="20.59" y="319.5" ></text>
</g>
<g >
<title>exit_mmap (16 samples, 0.01%)</title><rect x="19.6" y="229" width="0.1" height="15.0" fill="rgb(220,78,50)" rx="2" ry="2" />
<text x="22.60" y="239.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (15 samples, 0.01%)</title><rect x="17.0" y="293" width="0.1" height="15.0" fill="rgb(247,46,4)" rx="2" ry="2" />
<text x="20.03" y="303.5" ></text>
</g>
<g >
<title>__schedule (102 samples, 0.07%)</title><rect x="43.6" y="149" width="0.8" height="15.0" fill="rgb(207,75,37)" rx="2" ry="2" />
<text x="46.63" y="159.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (25 samples, 0.02%)</title><rect x="14.7" y="325" width="0.2" height="15.0" fill="rgb(222,80,3)" rx="2" ry="2" />
<text x="17.71" y="335.5" ></text>
</g>
<g >
<title>gawk (116 samples, 0.08%)</title><rect x="10.8" y="357" width="0.9" height="15.0" fill="rgb(251,121,5)" rx="2" ry="2" />
<text x="13.81" y="367.5" ></text>
</g>
<g >
<title>avc_lookup (87 samples, 0.06%)</title><rect x="23.9" y="181" width="0.6" height="15.0" fill="rgb(232,93,37)" rx="2" ry="2" />
<text x="26.85" y="191.5" ></text>
</g>
<g >
<title>default_idle_call (137,179 samples, 90.99%)</title><rect x="79.6" y="261" width="1073.6" height="15.0" fill="rgb(251,150,9)" rx="2" ry="2" />
<text x="82.60" y="271.5" >default_idle_call</text>
</g>
<g >
<title>syscall_enter_from_user_mode (75 samples, 0.05%)</title><rect x="78.4" y="293" width="0.6" height="15.0" fill="rgb(236,51,46)" rx="2" ry="2" />
<text x="81.40" y="303.5" ></text>
</g>
<g >
<title>exc_page_fault (18 samples, 0.01%)</title><rect x="18.8" y="261" width="0.1" height="15.0" fill="rgb(223,135,16)" rx="2" ry="2" />
<text x="21.77" y="271.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (359 samples, 0.24%)</title><rect x="1153.5" y="245" width="2.8" height="15.0" fill="rgb(241,201,11)" rx="2" ry="2" />
<text x="1156.49" y="255.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (117 samples, 0.08%)</title><rect x="18.4" y="325" width="0.9" height="15.0" fill="rgb(208,159,35)" rx="2" ry="2" />
<text x="21.42" y="335.5" ></text>
</g>
<g >
<title>rep_movs_alternative (2,317 samples, 1.54%)</title><rect x="49.8" y="213" width="18.1" height="15.0" fill="rgb(254,169,49)" rx="2" ry="2" />
<text x="52.78" y="223.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (861 samples, 0.57%)</title><rect x="69.4" y="133" width="6.8" height="15.0" fill="rgb(247,63,45)" rx="2" ry="2" />
<text x="72.43" y="143.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (54 samples, 0.04%)</title><rect x="16.4" y="293" width="0.4" height="15.0" fill="rgb(215,182,49)" rx="2" ry="2" />
<text x="19.42" y="303.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (37 samples, 0.02%)</title><rect x="11.1" y="309" width="0.3" height="15.0" fill="rgb(211,30,36)" rx="2" ry="2" />
<text x="14.14" y="319.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (21 samples, 0.01%)</title><rect x="19.0" y="293" width="0.2" height="15.0" fill="rgb(234,101,11)" rx="2" ry="2" />
<text x="22.02" y="303.5" ></text>
</g>
<g >
<title>__schedule (114 samples, 0.08%)</title><rect x="77.0" y="165" width="0.9" height="15.0" fill="rgb(233,174,27)" rx="2" ry="2" />
<text x="79.99" y="175.5" ></text>
</g>
<g >
<title>_copy_from_iter (2,336 samples, 1.55%)</title><rect x="49.6" y="229" width="18.3" height="15.0" fill="rgb(222,39,46)" rx="2" ry="2" />
<text x="52.63" y="239.5" ></text>
</g>
<g >
<title>cpuidle_idle_call (137,224 samples, 91.02%)</title><rect x="79.2" y="277" width="1074.0" height="15.0" fill="rgb(249,165,52)" rx="2" ry="2" />
<text x="82.25" y="287.5" >cpuidle_idle_call</text>
</g>
<g >
<title>do_idle (137,746 samples, 91.36%)</title><rect x="79.1" y="293" width="1078.0" height="15.0" fill="rgb(232,160,31)" rx="2" ry="2" />
<text x="82.06" y="303.5" >do_idle</text>
</g>
<g >
<title>pmprobe (308 samples, 0.20%)</title><rect x="15.3" y="357" width="2.4" height="15.0" fill="rgb(231,146,38)" rx="2" ry="2" />
<text x="18.33" y="367.5" ></text>
</g>
<g >
<title>avc_has_perm (91 samples, 0.06%)</title><rect x="23.8" y="213" width="0.7" height="15.0" fill="rgb(208,215,18)" rx="2" ry="2" />
<text x="26.82" y="223.5" ></text>
</g>
<g >
<title>pmsleep (257 samples, 0.17%)</title><rect x="17.7" y="357" width="2.1" height="15.0" fill="rgb(208,30,12)" rx="2" ry="2" />
<text x="20.74" y="367.5" ></text>
</g>
<g >
<title>smc_recvmsg (2,542 samples, 1.69%)</title><rect x="24.5" y="245" width="19.9" height="15.0" fill="rgb(252,206,5)" rx="2" ry="2" />
<text x="27.53" y="255.5" ></text>
</g>
<g >
<title>__x64_sys_execve (25 samples, 0.02%)</title><rect x="14.7" y="293" width="0.2" height="15.0" fill="rgb(212,74,28)" rx="2" ry="2" />
<text x="17.71" y="303.5" ></text>
</g>
<g >
<title>wait_woken (143 samples, 0.09%)</title><rect x="43.3" y="197" width="1.1" height="15.0" fill="rgb(228,115,22)" rx="2" ry="2" />
<text x="46.31" y="207.5" ></text>
</g>
<g >
<title>default_idle_call (4,167 samples, 2.76%)</title><rect x="1157.1" y="197" width="32.6" height="15.0" fill="rgb(234,215,47)" rx="2" ry="2" />
<text x="1160.14" y="207.5" >de..</text>
</g>
<g >
<title>swapper (141,918 samples, 94.13%)</title><rect x="79.0" y="357" width="1110.7" height="15.0" fill="rgb(239,155,0)" rx="2" ry="2" />
<text x="82.03" y="367.5" >swapper</text>
</g>
<g >
<title>entry_SYSCALL_64 (4,123 samples, 2.73%)</title><rect x="46.7" y="325" width="32.3" height="15.0" fill="rgb(242,184,45)" rx="2" ry="2" />
<text x="49.74" y="335.5" >en..</text>
</g>
<g >
<title>mprotect_fixup (13 samples, 0.01%)</title><rect x="16.7" y="213" width="0.1" height="15.0" fill="rgb(242,63,35)" rx="2" ry="2" />
<text x="19.71" y="223.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (31 samples, 0.02%)</title><rect x="42.7" y="197" width="0.3" height="15.0" fill="rgb(254,67,14)" rx="2" ry="2" />
<text x="45.71" y="207.5" ></text>
</g>
<g >
<title>[unknown] (151 samples, 0.10%)</title><rect x="18.4" y="341" width="1.2" height="15.0" fill="rgb(230,226,10)" rx="2" ry="2" />
<text x="21.42" y="351.5" ></text>
</g>
<g >
<title>smc_tx_wait (218 samples, 0.14%)</title><rect x="76.2" y="229" width="1.7" height="15.0" fill="rgb(248,54,30)" rx="2" ry="2" />
<text x="79.17" y="239.5" ></text>
</g>
<g >
<title>copy_p4d_range (16 samples, 0.01%)</title><rect x="13.9" y="197" width="0.1" height="15.0" fill="rgb(208,52,53)" rx="2" ry="2" />
<text x="16.89" y="207.5" ></text>
</g>
<g >
<title>_raw_read_lock (17 samples, 0.01%)</title><rect x="36.6" y="117" width="0.1" height="15.0" fill="rgb(237,97,19)" rx="2" ry="2" />
<text x="39.58" y="127.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (789 samples, 0.52%)</title><rect x="36.5" y="165" width="6.1" height="15.0" fill="rgb(227,73,41)" rx="2" ry="2" />
<text x="39.45" y="175.5" ></text>
</g>
<g >
<title>systemd (13 samples, 0.01%)</title><rect x="1189.8" y="357" width="0.1" height="15.0" fill="rgb(248,134,53)" rx="2" ry="2" />
<text x="1192.84" y="367.5" ></text>
</g>
<g >
<title>_nohz_idle_balance.isra.0 (37 samples, 0.02%)</title><rect x="1189.5" y="101" width="0.2" height="15.0" fill="rgb(231,138,41)" rx="2" ry="2" />
<text x="1192.46" y="111.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (1,044 samples, 0.69%)</title><rect x="68.0" y="229" width="8.2" height="15.0" fill="rgb(234,215,10)" rx="2" ry="2" />
<text x="71.00" y="239.5" ></text>
</g>
<g >
<title>[unknown] (63 samples, 0.04%)</title><rect x="11.1" y="341" width="0.5" height="15.0" fill="rgb(222,14,30)" rx="2" ry="2" />
<text x="14.06" y="351.5" ></text>
</g>
<g >
<title>exec_mmap (23 samples, 0.02%)</title><rect x="14.7" y="181" width="0.2" height="15.0" fill="rgb(215,58,8)" rx="2" ry="2" />
<text x="17.71" y="191.5" ></text>
</g>
<g >
<title>__handle_mm_fault (14 samples, 0.01%)</title><rect x="16.6" y="213" width="0.1" height="15.0" fill="rgb(228,29,6)" rx="2" ry="2" />
<text x="19.57" y="223.5" ></text>
</g>
<g >
<title>exit_mm (17 samples, 0.01%)</title><rect x="17.6" y="261" width="0.1" height="15.0" fill="rgb(224,150,39)" rx="2" ry="2" />
<text x="20.59" y="271.5" ></text>
</g>
<g >
<title>sockfd_lookup_light (66 samples, 0.04%)</title><rect x="77.9" y="261" width="0.5" height="15.0" fill="rgb(227,127,48)" rx="2" ry="2" />
<text x="80.88" y="271.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (952 samples, 0.63%)</title><rect x="68.7" y="181" width="7.5" height="15.0" fill="rgb(216,56,26)" rx="2" ry="2" />
<text x="71.72" y="191.5" ></text>
</g>
<g >
<title>wp_page_copy (13 samples, 0.01%)</title><rect x="18.8" y="197" width="0.1" height="15.0" fill="rgb(231,142,49)" rx="2" ry="2" />
<text x="21.81" y="207.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (14 samples, 0.01%)</title><rect x="19.2" y="293" width="0.1" height="15.0" fill="rgb(225,196,5)" rx="2" ry="2" />
<text x="22.18" y="303.5" ></text>
</g>
<g >
<title>sockfd_lookup_light (68 samples, 0.05%)</title><rect x="44.4" y="261" width="0.6" height="15.0" fill="rgb(247,125,1)" rx="2" ry="2" />
<text x="47.43" y="271.5" ></text>
</g>
<g >
<title>check_heap_object (205 samples, 0.14%)</title><rect x="25.2" y="197" width="1.6" height="15.0" fill="rgb(215,38,12)" rx="2" ry="2" />
<text x="28.24" y="207.5" ></text>
</g>
<g >
<title>vm_mmap_pgoff (38 samples, 0.03%)</title><rect x="18.1" y="277" width="0.3" height="15.0" fill="rgb(227,122,0)" rx="2" ry="2" />
<text x="21.05" y="287.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (21 samples, 0.01%)</title><rect x="10.1" y="341" width="0.2" height="15.0" fill="rgb(214,204,51)" rx="2" ry="2" />
<text x="13.10" y="351.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (18 samples, 0.01%)</title><rect x="18.8" y="277" width="0.1" height="15.0" fill="rgb(250,135,17)" rx="2" ry="2" />
<text x="21.77" y="287.5" ></text>
</g>
<g >
<title>Server<IoRecvfrom, SwitchOff>::doLoop (102 samples, 0.07%)</title><rect x="20.4" y="341" width="0.8" height="15.0" fill="rgb(212,150,39)" rx="2" ry="2" />
<text x="23.40" y="351.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (969 samples, 0.64%)</title><rect x="68.6" y="197" width="7.6" height="15.0" fill="rgb(244,88,26)" rx="2" ry="2" />
<text x="71.59" y="207.5" ></text>
</g>
<g >
<title>find_vmap_area (185 samples, 0.12%)</title><rect x="48.2" y="197" width="1.4" height="15.0" fill="rgb(216,137,15)" rx="2" ry="2" />
<text x="51.17" y="207.5" ></text>
</g>
<g >
<title>start_kernel (4,168 samples, 2.76%)</title><rect x="1157.1" y="293" width="32.6" height="15.0" fill="rgb(224,187,26)" rx="2" ry="2" />
<text x="1160.13" y="303.5" >st..</text>
</g>
<g >
<title>__x64_sys_mprotect (15 samples, 0.01%)</title><rect x="16.7" y="245" width="0.1" height="15.0" fill="rgb(253,137,5)" rx="2" ry="2" />
<text x="19.71" y="255.5" ></text>
</g>
<g >
<title>smc_tx_consumer_update (808 samples, 0.54%)</title><rect x="36.3" y="197" width="6.3" height="15.0" fill="rgb(251,32,54)" rx="2" ry="2" />
<text x="39.30" y="207.5" ></text>
</g>
<g >
<title>__do_softirq (89 samples, 0.06%)</title><rect x="1189.1" y="117" width="0.6" height="15.0" fill="rgb(232,159,28)" rx="2" ry="2" />
<text x="1192.05" y="127.5" ></text>
</g>
<g >
<title>__schedule (361 samples, 0.24%)</title><rect x="1153.5" y="261" width="2.8" height="15.0" fill="rgb(237,95,25)" rx="2" ry="2" />
<text x="1156.48" y="271.5" ></text>
</g>
<g >
<title>recvfrom (3,030 samples, 2.01%)</title><rect x="22.2" y="341" width="23.7" height="15.0" fill="rgb(221,214,49)" rx="2" ry="2" />
<text x="25.16" y="351.5" >r..</text>
</g>
<g >
<title>flush_smp_call_function_queue (14 samples, 0.01%)</title><rect x="1153.2" y="277" width="0.1" height="15.0" fill="rgb(244,123,45)" rx="2" ry="2" />
<text x="1156.23" y="287.5" ></text>
</g>
<g >
<title>default_idle (137,179 samples, 90.99%)</title><rect x="79.6" y="245" width="1073.6" height="15.0" fill="rgb(233,84,13)" rx="2" ry="2" />
<text x="82.60" y="255.5" >default_idle</text>
</g>
<g >
<title>default_idle (4,167 samples, 2.76%)</title><rect x="1157.1" y="181" width="32.6" height="15.0" fill="rgb(237,61,50)" rx="2" ry="2" />
<text x="1160.14" y="191.5" >de..</text>
</g>
<g >
<title>[unknown] (172 samples, 0.11%)</title><rect x="16.2" y="341" width="1.3" height="15.0" fill="rgb(229,191,42)" rx="2" ry="2" />
<text x="19.18" y="351.5" ></text>
</g>
<g >
<title>_raw_read_lock (52 samples, 0.03%)</title><rect x="68.8" y="149" width="0.5" height="15.0" fill="rgb(233,48,21)" rx="2" ry="2" />
<text x="71.85" y="159.5" ></text>
</g>
<g >
<title>do_syscall_64 (2,894 samples, 1.92%)</title><rect x="23.2" y="309" width="22.7" height="15.0" fill="rgb(237,176,51)" rx="2" ry="2" />
<text x="26.23" y="319.5" >d..</text>
</g>
<g >
<title>do_mmap (30 samples, 0.02%)</title><rect x="18.1" y="261" width="0.2" height="15.0" fill="rgb(229,190,36)" rx="2" ry="2" />
<text x="21.05" y="271.5" ></text>
</g>
<g >
<title>dup_mm (27 samples, 0.02%)</title><rect x="13.9" y="245" width="0.2" height="15.0" fill="rgb(216,74,49)" rx="2" ry="2" />
<text x="16.87" y="255.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (15 samples, 0.01%)</title><rect x="13.5" y="309" width="0.1" height="15.0" fill="rgb(206,115,13)" rx="2" ry="2" />
<text x="16.45" y="319.5" ></text>
</g>
<g >
<title>begin_new_exec (25 samples, 0.02%)</title><rect x="14.7" y="197" width="0.2" height="15.0" fill="rgb(226,41,40)" rx="2" ry="2" />
<text x="17.71" y="207.5" ></text>
</g>
<g >
<title>smc_tx_write_space (687 samples, 0.46%)</title><rect x="37.3" y="69" width="5.3" height="15.0" fill="rgb(233,165,6)" rx="2" ry="2" />
<text x="40.25" y="79.5" ></text>
</g>
<g >
<title>malloc (16 samples, 0.01%)</title><rect x="15.0" y="341" width="0.1" height="15.0" fill="rgb(211,145,26)" rx="2" ry="2" />
<text x="18.02" y="351.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (94 samples, 0.06%)</title><rect x="43.7" y="133" width="0.7" height="15.0" fill="rgb(248,45,39)" rx="2" ry="2" />
<text x="46.65" y="143.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (774 samples, 0.51%)</title><rect x="70.1" y="85" width="6.1" height="15.0" fill="rgb(234,36,47)" rx="2" ry="2" />
<text x="73.12" y="95.5" ></text>
</g>
<g >
<title>security_socket_sendmsg (44 samples, 0.03%)</title><rect x="46.9" y="261" width="0.3" height="15.0" fill="rgb(240,172,35)" rx="2" ry="2" />
<text x="49.89" y="271.5" ></text>
</g>
<g >
<title>[unknown] (17 samples, 0.01%)</title><rect x="12.5" y="341" width="0.1" height="15.0" fill="rgb(249,99,14)" rx="2" ry="2" />
<text x="15.47" y="351.5" ></text>
</g>
<g >
<title>do_syscall_64 (15 samples, 0.01%)</title><rect x="13.5" y="293" width="0.1" height="15.0" fill="rgb(250,108,52)" rx="2" ry="2" />
<text x="16.45" y="303.5" ></text>
</g>
<g >
<title>__handle_mm_fault (20 samples, 0.01%)</title><rect x="16.9" y="229" width="0.1" height="15.0" fill="rgb(249,6,47)" rx="2" ry="2" />
<text x="19.86" y="239.5" ></text>
</g>
<g >
<title>do_user_addr_fault (18 samples, 0.01%)</title><rect x="18.8" y="245" width="0.1" height="15.0" fill="rgb(209,130,16)" rx="2" ry="2" />
<text x="21.77" y="255.5" ></text>
</g>
<g >
<title>execve (15 samples, 0.01%)</title><rect x="13.5" y="325" width="0.1" height="15.0" fill="rgb(205,187,39)" rx="2" ry="2" />
<text x="16.45" y="335.5" ></text>
</g>
<g >
<title>smc_rx_wait (230 samples, 0.15%)</title><rect x="42.6" y="213" width="1.8" height="15.0" fill="rgb(231,150,54)" rx="2" ry="2" />
<text x="45.63" y="223.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (100 samples, 0.07%)</title><rect x="18.5" y="309" width="0.8" height="15.0" fill="rgb(217,69,29)" rx="2" ry="2" />
<text x="21.52" y="319.5" ></text>
</g>
<g >
<title>do_user_addr_fault (23 samples, 0.02%)</title><rect x="16.8" y="261" width="0.2" height="15.0" fill="rgb(230,100,25)" rx="2" ry="2" />
<text x="19.85" y="271.5" ></text>
</g>
<g >
<title>cpu_startup_entry (137,750 samples, 91.36%)</title><rect x="79.0" y="309" width="1078.1" height="15.0" fill="rgb(240,196,18)" rx="2" ry="2" />
<text x="82.03" y="319.5" >cpu_startup_entry</text>
</g>
<g >
<title>ps (40 samples, 0.03%)</title><rect x="19.8" y="357" width="0.3" height="15.0" fill="rgb(215,26,44)" rx="2" ry="2" />
<text x="22.75" y="367.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (727 samples, 0.48%)</title><rect x="36.9" y="101" width="5.7" height="15.0" fill="rgb(251,38,39)" rx="2" ry="2" />
<text x="39.94" y="111.5" ></text>
</g>
<g >
<title>secondary_startup_64_no_verify (141,918 samples, 94.13%)</title><rect x="79.0" y="341" width="1110.7" height="15.0" fill="rgb(238,121,0)" rx="2" ry="2" />
<text x="82.03" y="351.5" >secondary_startup_64_no_verify</text>
</g>
<g >
<title>sendto (4,233 samples, 2.81%)</title><rect x="45.9" y="341" width="33.1" height="15.0" fill="rgb(213,36,7)" rx="2" ry="2" />
<text x="48.88" y="351.5" >se..</text>
</g>
<g >
<title>smcd_cdc_rx_handler (869 samples, 0.58%)</title><rect x="69.4" y="149" width="6.8" height="15.0" fill="rgb(220,89,7)" rx="2" ry="2" />
<text x="72.37" y="159.5" ></text>
</g>
<g >
<title>_raw_spin_lock_bh (28 samples, 0.02%)</title><rect x="43.0" y="181" width="0.2" height="15.0" fill="rgb(249,218,9)" rx="2" ry="2" />
<text x="46.01" y="191.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (48 samples, 0.03%)</title><rect x="13.8" y="325" width="0.4" height="15.0" fill="rgb(248,86,4)" rx="2" ry="2" />
<text x="16.81" y="335.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (189 samples, 0.13%)</title><rect x="48.2" y="229" width="1.4" height="15.0" fill="rgb(211,178,21)" rx="2" ry="2" />
<text x="51.15" y="239.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (17 samples, 0.01%)</title><rect x="17.6" y="341" width="0.1" height="15.0" fill="rgb(218,103,1)" rx="2" ry="2" />
<text x="20.59" y="351.5" ></text>
</g>
<g >
<title>[unknown] (17 samples, 0.01%)</title><rect x="13.2" y="325" width="0.1" height="15.0" fill="rgb(232,47,8)" rx="2" ry="2" />
<text x="16.22" y="335.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (125 samples, 0.08%)</title><rect x="16.2" y="325" width="1.0" height="15.0" fill="rgb(242,89,34)" rx="2" ry="2" />
<text x="19.18" y="335.5" ></text>
</g>
<g >
<title>__do_sys_clone (48 samples, 0.03%)</title><rect x="13.8" y="293" width="0.4" height="15.0" fill="rgb(230,46,52)" rx="2" ry="2" />
<text x="16.81" y="303.5" ></text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (1,040 samples, 0.69%)</title><rect x="68.0" y="213" width="8.2" height="15.0" fill="rgb(207,162,12)" rx="2" ry="2" />
<text x="71.03" y="223.5" ></text>
</g>
<g >
<title>exit_mm (16 samples, 0.01%)</title><rect x="19.6" y="261" width="0.1" height="15.0" fill="rgb(205,0,49)" rx="2" ry="2" />
<text x="22.60" y="271.5" ></text>
</g>
<g >
<title>mmap_region (29 samples, 0.02%)</title><rect x="18.1" y="245" width="0.2" height="15.0" fill="rgb(238,229,51)" rx="2" ry="2" />
<text x="21.06" y="255.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (20 samples, 0.01%)</title><rect x="17.2" y="309" width="0.1" height="15.0" fill="rgb(208,30,8)" rx="2" ry="2" />
<text x="20.18" y="319.5" ></text>
</g>
<g >
<title>smc_rx_wake_up (807 samples, 0.54%)</title><rect x="69.9" y="101" width="6.3" height="15.0" fill="rgb(215,142,42)" rx="2" ry="2" />
<text x="72.86" y="111.5" ></text>
</g>
<g >
<title>smc_lo_move_data (33 samples, 0.02%)</title><rect x="68.3" y="181" width="0.3" height="15.0" fill="rgb(209,197,28)" rx="2" ry="2" />
<text x="71.33" y="191.5" ></text>
</g>
<g >
<title>do_syscall_64 (25 samples, 0.02%)</title><rect x="14.7" y="309" width="0.2" height="15.0" fill="rgb(223,227,39)" rx="2" ry="2" />
<text x="17.71" y="319.5" ></text>
</g>
<g >
<title>exec_binprm (25 samples, 0.02%)</title><rect x="14.7" y="245" width="0.2" height="15.0" fill="rgb(249,95,25)" rx="2" ry="2" />
<text x="17.71" y="255.5" ></text>
</g>
<g >
<title>release_sock (15 samples, 0.01%)</title><rect x="24.8" y="229" width="0.1" height="15.0" fill="rgb(254,126,37)" rx="2" ry="2" />
<text x="27.75" y="239.5" ></text>
</g>
<g >
<title>[unknown] (43 samples, 0.03%)</title><rect x="10.3" y="341" width="0.4" height="15.0" fill="rgb(246,151,23)" rx="2" ry="2" />
<text x="13.32" y="351.5" ></text>
</g>
<g >
<title>cpu_startup_entry (4,168 samples, 2.76%)</title><rect x="1157.1" y="245" width="32.6" height="15.0" fill="rgb(223,209,44)" rx="2" ry="2" />
<text x="1160.13" y="255.5" >cp..</text>
</g>
<g >
<title>expr (92 samples, 0.06%)</title><rect x="10.1" y="357" width="0.7" height="15.0" fill="rgb(247,166,30)" rx="2" ry="2" />
<text x="13.09" y="367.5" ></text>
</g>
<g >
<title>do_syscall_64 (14 samples, 0.01%)</title><rect x="19.2" y="277" width="0.1" height="15.0" fill="rgb(212,109,0)" rx="2" ry="2" />
<text x="22.18" y="287.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (744 samples, 0.49%)</title><rect x="36.8" y="117" width="5.8" height="15.0" fill="rgb(252,102,43)" rx="2" ry="2" />
<text x="39.81" y="127.5" ></text>
</g>
<g >
<title>all (150,770 samples, 100%)</title><rect x="10.0" y="373" width="1180.0" height="15.0" fill="rgb(241,174,43)" rx="2" ry="2" />
<text x="13.00" y="383.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (659 samples, 0.44%)</title><rect x="37.5" y="37" width="5.1" height="15.0" fill="rgb(241,114,48)" rx="2" ry="2" />
<text x="40.47" y="47.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (16 samples, 0.01%)</title><rect x="43.1" y="165" width="0.1" height="15.0" fill="rgb(211,39,46)" rx="2" ry="2" />
<text x="46.11" y="175.5" ></text>
</g>
<g >
<title>copy_process (37 samples, 0.02%)</title><rect x="13.8" y="261" width="0.3" height="15.0" fill="rgb(246,194,16)" rx="2" ry="2" />
<text x="16.83" y="271.5" ></text>
</g>
<g >
<title>[unknown] (26 samples, 0.02%)</title><rect x="19.8" y="341" width="0.2" height="15.0" fill="rgb(209,8,15)" rx="2" ry="2" />
<text x="22.78" y="351.5" ></text>
</g>
<g >
<title>do_syscall_64 (44 samples, 0.03%)</title><rect x="18.0" y="309" width="0.4" height="15.0" fill="rgb(237,201,40)" rx="2" ry="2" />
<text x="21.05" y="319.5" ></text>
</g>
</g>
</svg>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-04 3:22 ` shaozhengchao
@ 2023-12-04 3:52 ` Wen Gu
2023-12-04 4:06 ` shaozhengchao
0 siblings, 1 reply; 14+ messages in thread
From: Wen Gu @ 2023-12-04 3:52 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/4 11:22, shaozhengchao wrote:
>
>
> On 2023/11/23 14:15, shaozhengchao wrote:
>>
>>
>> On 2023/11/23 10:21, Wen Gu wrote:
>>>
>>>
>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>> Hi Wen Gu:
>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>> my analysis:
>>>>
>>>> 1. Run the following command to perform the test, and then capture
>>>> packets on the lo device.
>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>
>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>> SMC-R-Deline packets.
>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>
>>>> 3. Rx
>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>
>>>> 4. Tx
>>>> In smc_clc_send_proposal,
>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>
>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>> always equal to 0.
>>>>
>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>> But I don't see where operations can be added, may I missed something?
>>>>
>>>
>>> Hi Zhengchao Shao,
>>>
>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>> UEID should be set. It can be set by following steps:
>>>
>>> - Install smc-tools[1].
>>>
>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>
>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>> unavailable, an UEID is required.
>>>
>> Hi Wen Gu:
>> Thank you for your reply. This is very useful for me. And I will
>> be happy to learn from it.
>>
>> Thanks
>>
>> Zhengchao Shao
>>> - Then run the test.
>>>
>>> Hope this works for you :)
>>>
>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>
>>> Regards,
>>> Wen Gu
>>>
>>>> Could you give me some advice? Thanks very much.
>>>>
>>>> Zhengchao Shao
>>>>
>>>>
>>>> [1]link:
>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>
>>>
>>
>>
> Hi Wen Gu:
> I have test as following, but the performance is really
> degraded. Now I have no idea.
> 1. add ueid
> run: smcd ueid add 16
> kernel message:
> [ 5252.009133] NET: Registered PF_SMC protocol family
> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
> 2. start server
> smc_run taskset -c 1 sockperf sr --tcp
> 3. start client
> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>
> The test results are as follows:
> TCP SMC-lo
> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>
> I didn't find a better direction when I initially positioned it. No
> error is recorded in the kernel log, and the smcd statistics are normal.
> [root@localhost smc-tools]# smcd stats
> SMC-D Connections Summary
> Total connections handled 2
> SMC connections 2
> Handshake errors 0
> Avg requests per SMC conn 1277462.0
> TCP fallback 0
>
> RX Stats
> Data transmitted (Bytes) 40907328000 (40.91G)
> Total requests 1277190
> Buffer full 45 (0.00%)
> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
> Bufs 0 0 0 2 0 0 0 0
> Reqs 638.0K 0 0 639.2K 0 0 0 0
>
> TX Stats
> Data transmitted (Bytes) 40907328000 (40.91G)
> Total requests 1277734
> Buffer full 638239 (49.95%)
> Buffer full (remote) 0 (0.00%)
> Buffer too small 0 (0.00%)
> Buffer too small (remote) 0 (0.00%)
> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
> Bufs 0 0 0 0 0 0 0 0
> Reqs 0 0 0 1.278M 0 0 0 0
>
> Extras
> Special socket calls 1
>
> I captured the perf information and found that the percentage of
> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
> during tx and rx.
> 36.12% [kernel] [k]rep_movs_alternative
> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>
> I've attached the flame map. Could you help analyze it? What I missed?
> Thanks.
Hi Zhengchao Shao,
Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
like TCP, it is necessary to appropriately increase the default value of smc
sk_sndbuf and sk_rcvbuf before testing throughput.
Set this and try again:
# sysctl -w net.smc.wmem=1048576
# sysctl -w net.smc.rmem=1048576
(The initial value of wmem and rmem are 64K)
Regards,
Wen Gu
>
> Zhengchao Shao
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-04 3:52 ` Wen Gu
@ 2023-12-04 4:06 ` shaozhengchao
2023-12-05 6:45 ` shaozhengchao
0 siblings, 1 reply; 14+ messages in thread
From: shaozhengchao @ 2023-12-04 4:06 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/4 11:52, Wen Gu wrote:
>
> On 2023/12/4 11:22, shaozhengchao wrote:
>>
>>
>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>
>>>
>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>
>>>>
>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>> Hi Wen Gu:
>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO
>>>>> feature on
>>>>> my x86_64 environment and kernel is based on linux-next, commit:
>>>>> 5ba73bec5e7b.
>>>>> The test result shows that the smc_lo feature cannot be enabled.
>>>>> Here's
>>>>> my analysis:
>>>>>
>>>>> 1. Run the following command to perform the test, and then capture
>>>>> packets on the lo device.
>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp
>>>>> --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>
>>>>> 2. Use Wireshark to open packets. It is found that the VCE port
>>>>> replies with
>>>>> SMC-R-Deline packets.
>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>
>>>>> 3. Rx
>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>> a Decline packet because eid_cnt and flag.seid in the received
>>>>> packet are both 0.
>>>>>
>>>>> 4. Tx
>>>>> In smc_clc_send_proposal,
>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>
>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>>>> seid_enabled is
>>>>> always equal to 0.
>>>>>
>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>> But I don't see where operations can be added, may I missed something?
>>>>>
>>>>
>>>> Hi Zhengchao Shao,
>>>>
>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A
>>>> common
>>>> UEID should be set. It can be set by following steps:
>>>>
>>>> - Install smc-tools[1].
>>>>
>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>
>>>> EID works as an ID to indicate the max communication space of
>>>> SMC. When SEID is
>>>> unavailable, an UEID is required.
>>>>
>>> Hi Wen Gu:
>>> Thank you for your reply. This is very useful for me. And I will
>>> be happy to learn from it.
>>>
>>> Thanks
>>>
>>> Zhengchao Shao
>>>> - Then run the test.
>>>>
>>>> Hope this works for you :)
>>>>
>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>
>>>> Regards,
>>>> Wen Gu
>>>>
>>>>> Could you give me some advice? Thanks very much.
>>>>>
>>>>> Zhengchao Shao
>>>>>
>>>>>
>>>>> [1]link:
>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>
>>>>
>>>
>>>
>> Hi Wen Gu:
>> I have test as following, but the performance is really
>> degraded. Now I have no idea.
>> 1. add ueid
>> run: smcd ueid add 16
>> kernel message:
>> [ 5252.009133] NET: Registered PF_SMC protocol family
>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>> 2. start server
>> smc_run taskset -c 1 sockperf sr --tcp
>> 3. start client
>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1
>> -t 30
>>
>> The test results are as follows:
>> TCP SMC-lo
>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>
>> I didn't find a better direction when I initially positioned it. No
>> error is recorded in the kernel log, and the smcd statistics are normal.
>> [root@localhost smc-tools]# smcd stats
>> SMC-D Connections Summary
>> Total connections handled 2
>> SMC connections 2
>> Handshake errors 0
>> Avg requests per SMC conn 1277462.0
>> TCP fallback 0
>>
>> RX Stats
>> Data transmitted (Bytes) 40907328000 (40.91G)
>> Total requests 1277190
>> Buffer full 45 (0.00%)
>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>> Bufs 0 0 0 2 0 0 0 0
>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>
>> TX Stats
>> Data transmitted (Bytes) 40907328000 (40.91G)
>> Total requests 1277734
>> Buffer full 638239 (49.95%)
>> Buffer full (remote) 0 (0.00%)
>> Buffer too small 0 (0.00%)
>> Buffer too small (remote) 0 (0.00%)
>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>> Bufs 0 0 0 0 0 0 0 0
>> Reqs 0 0 0 1.278M 0 0 0 0
>>
>> Extras
>> Special socket calls 1
>>
>> I captured the perf information and found that the percentage of
>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>> during tx and rx.
>> 36.12% [kernel] [k]rep_movs_alternative
>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>
>> I've attached the flame map. Could you help analyze it? What I missed?
>> Thanks.
>
> Hi Zhengchao Shao,
>
> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow
> dynamically
> like TCP, it is necessary to appropriately increase the default value of
> smc
> sk_sndbuf and sk_rcvbuf before testing throughput.
>
> Set this and try again:
>
> # sysctl -w net.smc.wmem=1048576
> # sysctl -w net.smc.rmem=1048576
>
> (The initial value of wmem and rmem are 64K)
>
> Regards,
> Wen Gu
>
>>
>> Zhengchao Shao
Hi Wen Gu:
It solves the issue. Thank you very much.
Zhengchao Shao
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-04 4:06 ` shaozhengchao
@ 2023-12-05 6:45 ` shaozhengchao
2023-12-13 9:00 ` shaozhengchao
2023-12-13 13:06 ` Wen Gu
0 siblings, 2 replies; 14+ messages in thread
From: shaozhengchao @ 2023-12-05 6:45 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
[-- Attachment #1: Type: text/plain, Size: 7443 bytes --]
On 2023/12/4 12:06, shaozhengchao wrote:
>
>
> On 2023/12/4 11:52, Wen Gu wrote:
>>
>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>
>>>
>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>
>>>>>
>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>> Hi Wen Gu:
>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO
>>>>>> feature on
>>>>>> my x86_64 environment and kernel is based on linux-next, commit:
>>>>>> 5ba73bec5e7b.
>>>>>> The test result shows that the smc_lo feature cannot be enabled.
>>>>>> Here's
>>>>>> my analysis:
>>>>>>
>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>> packets on the lo device.
>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp
>>>>>> --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>
>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port
>>>>>> replies with
>>>>>> SMC-R-Deline packets.
>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>
>>>>>> 3. Rx
>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port
>>>>>> returns
>>>>>> a Decline packet because eid_cnt and flag.seid in the received
>>>>>> packet are both 0.
>>>>>>
>>>>>> 4. Tx
>>>>>> In smc_clc_send_proposal,
>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>
>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>>>>> seid_enabled is
>>>>>> always equal to 0.
>>>>>>
>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>> But I don't see where operations can be added, may I missed
>>>>>> something?
>>>>>>
>>>>>
>>>>> Hi Zhengchao Shao,
>>>>>
>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A
>>>>> common
>>>>> UEID should be set. It can be set by following steps:
>>>>>
>>>>> - Install smc-tools[1].
>>>>>
>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>
>>>>> EID works as an ID to indicate the max communication space of
>>>>> SMC. When SEID is
>>>>> unavailable, an UEID is required.
>>>>>
>>>> Hi Wen Gu:
>>>> Thank you for your reply. This is very useful for me. And I will
>>>> be happy to learn from it.
>>>>
>>>> Thanks
>>>>
>>>> Zhengchao Shao
>>>>> - Then run the test.
>>>>>
>>>>> Hope this works for you :)
>>>>>
>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>
>>>>> Regards,
>>>>> Wen Gu
>>>>>
>>>>>> Could you give me some advice? Thanks very much.
>>>>>>
>>>>>> Zhengchao Shao
>>>>>>
>>>>>>
>>>>>> [1]link:
>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>
>>>>>
>>>>
>>>>
>>> Hi Wen Gu:
>>> I have test as following, but the performance is really
>>> degraded. Now I have no idea.
>>> 1. add ueid
>>> run: smcd ueid add 16
>>> kernel message:
>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>> 2. start server
>>> smc_run taskset -c 1 sockperf sr --tcp
>>> 3. start client
>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1
>>> -t 30
>>>
>>> The test results are as follows:
>>> TCP SMC-lo
>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>
>>> I didn't find a better direction when I initially positioned it. No
>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>> [root@localhost smc-tools]# smcd stats
>>> SMC-D Connections Summary
>>> Total connections handled 2
>>> SMC connections 2
>>> Handshake errors 0
>>> Avg requests per SMC conn 1277462.0
>>> TCP fallback 0
>>>
>>> RX Stats
>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>> Total requests 1277190
>>> Buffer full 45 (0.00%)
>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>> Bufs 0 0 0 2 0 0 0 0
>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>
>>> TX Stats
>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>> Total requests 1277734
>>> Buffer full 638239 (49.95%)
>>> Buffer full (remote) 0 (0.00%)
>>> Buffer too small 0 (0.00%)
>>> Buffer too small (remote) 0 (0.00%)
>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>> Bufs 0 0 0 0 0 0 0 0
>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>
>>> Extras
>>> Special socket calls 1
>>>
>>> I captured the perf information and found that the percentage of
>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>> during tx and rx.
>>> 36.12% [kernel] [k]rep_movs_alternative
>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>
>>> I've attached the flame map. Could you help analyze it? What I missed?
>>> Thanks.
>>
>> Hi Zhengchao Shao,
>>
>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow
>> dynamically
>> like TCP, it is necessary to appropriately increase the default value
>> of smc
>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>
>> Set this and try again:
>>
>> # sysctl -w net.smc.wmem=1048576
>> # sysctl -w net.smc.rmem=1048576
>>
>> (The initial value of wmem and rmem are 64K)
>>
>> Regards,
>> Wen Gu
>>
>>>
>>> Zhengchao Shao
> Hi Wen Gu:
> It solves the issue. Thank you very much.
>
> Zhengchao Shao
>
Hi Wen Gu:
I've tested all the performance test items in the patchset. The
performance improvement is to be expected, except for nignx.
My VM is configured with 48 cores and 32 GB memory. Therefore, run
the following command:
<smc_run> nignx
<smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
The test results are as follows:
TCP SMC_lo
Requests/s 309425.42 135547.25(-56.19%)
The performance decreases by 56.19%.
I capture packets and find that wrk can perform HTTP GET after each
connect when smc_loopback is disabled.
However, when smc_loopback is enabled, there is no HTTP GET behavior.
I wonder if there is some compatibility problem with the SMC protocol
when encapsulate packet? Could you give me some advice?
In the attachment, I captured some of the packets.
nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
enabled.
Thank you very much.
Zhengchao Shao
[-- Attachment #2: nosmc_nginx.pcap --]
[-- Type: application/octet-stream, Size: 2385131 bytes --]
[-- Attachment #3: smc_nginx.pcap --]
[-- Type: application/octet-stream, Size: 1258836 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-05 6:45 ` shaozhengchao
@ 2023-12-13 9:00 ` shaozhengchao
2023-12-13 9:41 ` Wen Gu
2023-12-13 12:59 ` Wen Gu
2023-12-13 13:06 ` Wen Gu
1 sibling, 2 replies; 14+ messages in thread
From: shaozhengchao @ 2023-12-13 9:00 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/5 14:45, shaozhengchao wrote:
>
>
> On 2023/12/4 12:06, shaozhengchao wrote:
>>
>>
>> On 2023/12/4 11:52, Wen Gu wrote:
>>>
>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>> Hi Wen Gu:
>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO
>>>>>>> feature on
>>>>>>> my x86_64 environment and kernel is based on linux-next, commit:
>>>>>>> 5ba73bec5e7b.
>>>>>>> The test result shows that the smc_lo feature cannot be enabled.
>>>>>>> Here's
>>>>>>> my analysis:
>>>>>>>
>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>> packets on the lo device.
>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp
>>>>>>> --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>
>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port
>>>>>>> replies with
>>>>>>> SMC-R-Deline packets.
>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>
>>>>>>> 3. Rx
>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port
>>>>>>> returns
>>>>>>> a Decline packet because eid_cnt and flag.seid in the received
>>>>>>> packet are both 0.
>>>>>>>
>>>>>>> 4. Tx
>>>>>>> In smc_clc_send_proposal,
>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>
>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>>>>>> seid_enabled is
>>>>>>> always equal to 0.
>>>>>>>
>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>> But I don't see where operations can be added, may I missed
>>>>>>> something?
>>>>>>>
>>>>>>
>>>>>> Hi Zhengchao Shao,
>>>>>>
>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A
>>>>>> common
>>>>>> UEID should be set. It can be set by following steps:
>>>>>>
>>>>>> - Install smc-tools[1].
>>>>>>
>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>
>>>>>> EID works as an ID to indicate the max communication space of
>>>>>> SMC. When SEID is
>>>>>> unavailable, an UEID is required.
>>>>>>
>>>>> Hi Wen Gu:
>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>> be happy to learn from it.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Zhengchao Shao
>>>>>> - Then run the test.
>>>>>>
>>>>>> Hope this works for you :)
>>>>>>
>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>
>>>>>> Regards,
>>>>>> Wen Gu
>>>>>>
>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>
>>>>>>> Zhengchao Shao
>>>>>>>
>>>>>>>
>>>>>>> [1]link:
>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> Hi Wen Gu:
>>>> I have test as following, but the performance is really
>>>> degraded. Now I have no idea.
>>>> 1. add ueid
>>>> run: smcd ueid add 16
>>>> kernel message:
>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>> 2. start server
>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>> 3. start client
>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i
>>>> 127.0.0.1 -t 30
>>>>
>>>> The test results are as follows:
>>>> TCP SMC-lo
>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>
>>>> I didn't find a better direction when I initially positioned it. No
>>>> error is recorded in the kernel log, and the smcd statistics are
>>>> normal.
>>>> [root@localhost smc-tools]# smcd stats
>>>> SMC-D Connections Summary
>>>> Total connections handled 2
>>>> SMC connections 2
>>>> Handshake errors 0
>>>> Avg requests per SMC conn 1277462.0
>>>> TCP fallback 0
>>>>
>>>> RX Stats
>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>> Total requests 1277190
>>>> Buffer full 45 (0.00%)
>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>> >512KB
>>>> Bufs 0 0 0 2 0 0
>>>> 0 0
>>>> Reqs 638.0K 0 0 639.2K 0 0
>>>> 0 0
>>>>
>>>> TX Stats
>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>> Total requests 1277734
>>>> Buffer full 638239 (49.95%)
>>>> Buffer full (remote) 0 (0.00%)
>>>> Buffer too small 0 (0.00%)
>>>> Buffer too small (remote) 0 (0.00%)
>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>> >512KB
>>>> Bufs 0 0 0 0 0 0
>>>> 0 0
>>>> Reqs 0 0 0 1.278M 0 0
>>>> 0 0
>>>>
>>>> Extras
>>>> Special socket calls 1
>>>>
>>>> I captured the perf information and found that the percentage of
>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>> during tx and rx.
>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>
>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>> Thanks.
>>>
>>> Hi Zhengchao Shao,
>>>
>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow
>>> dynamically
>>> like TCP, it is necessary to appropriately increase the default value
>>> of smc
>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>
>>> Set this and try again:
>>>
>>> # sysctl -w net.smc.wmem=1048576
>>> # sysctl -w net.smc.rmem=1048576
>>>
>>> (The initial value of wmem and rmem are 64K)
>>>
>>> Regards,
>>> Wen Gu
>>>
>>>>
>>>> Zhengchao Shao
>> Hi Wen Gu:
>> It solves the issue. Thank you very much.
>>
>> Zhengchao Shao
>>
> Hi Wen Gu:
> I've tested all the performance test items in the patchset. The
> performance improvement is to be expected, except for nignx.
> My VM is configured with 48 cores and 32 GB memory. Therefore, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>
> The test results are as follows:
> TCP SMC_lo
> Requests/s 309425.42 135547.25(-56.19%)
> The performance decreases by 56.19%.
>
> I capture packets and find that wrk can perform HTTP GET after each
> connect when smc_loopback is disabled.
> However, when smc_loopback is enabled, there is no HTTP GET behavior.
> I wonder if there is some compatibility problem with the SMC protocol
> when encapsulate packet? Could you give me some advice?
> In the attachment, I captured some of the packets.
> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
> enabled.
> Thank you very much.
>
> Zhengchao Shao
>
>
>
Hi Wen Gu:
When the VM is configured with 8 cores and 16 GB memory, run
the following command:
<smc_run> nignx
<smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
the test data is as follows:
TCP SMC_lo
Requests/s 66056.66 94526.66(43.10%)
But When the VM is configured with 48 cores and 32 GB memory, run
the following command:
<smc_run> nignx
<smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
the test data is as follows:
TCP SMC_lo
Requests/s 309425.42 135547.25(-56.19%)
It seems that in the scenario with a large number of CPU cores,
performance is not optimized, but performance deteriorates. What I
missed?
Thank you.
Zhengchao Shao
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-13 9:00 ` shaozhengchao
@ 2023-12-13 9:41 ` Wen Gu
2023-12-13 12:59 ` Wen Gu
1 sibling, 0 replies; 14+ messages in thread
From: Wen Gu @ 2023-12-13 9:41 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/13 17:00, shaozhengchao wrote:
>
>
> On 2023/12/5 14:45, shaozhengchao wrote:
>>
>>
>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>
>>>
>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>
>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>> Hi Wen Gu:
>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>>>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>>>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>>>>>> my analysis:
>>>>>>>>
>>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>>> packets on the lo device.
>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>
>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>>>>>> SMC-R-Deline packets.
>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>
>>>>>>>> 3. Rx
>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>>>>>
>>>>>>>> 4. Tx
>>>>>>>> In smc_clc_send_proposal,
>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>
>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>>>>>> always equal to 0.
>>>>>>>>
>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>>> But I don't see where operations can be added, may I missed something?
>>>>>>>>
>>>>>>>
>>>>>>> Hi Zhengchao Shao,
>>>>>>>
>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>
>>>>>>> - Install smc-tools[1].
>>>>>>>
>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>
>>>>>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>>>>>> unavailable, an UEID is required.
>>>>>>>
>>>>>> Hi Wen Gu:
>>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>>> be happy to learn from it.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Zhengchao Shao
>>>>>>> - Then run the test.
>>>>>>>
>>>>>>> Hope this works for you :)
>>>>>>>
>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>
>>>>>>> Regards,
>>>>>>> Wen Gu
>>>>>>>
>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>
>>>>>>>> Zhengchao Shao
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]link:
>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> Hi Wen Gu:
>>>>> I have test as following, but the performance is really
>>>>> degraded. Now I have no idea.
>>>>> 1. add ueid
>>>>> run: smcd ueid add 16
>>>>> kernel message:
>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>> 2. start server
>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>> 3. start client
>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>
>>>>> The test results are as follows:
>>>>> TCP SMC-lo
>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>
>>>>> I didn't find a better direction when I initially positioned it. No
>>>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>>>> [root@localhost smc-tools]# smcd stats
>>>>> SMC-D Connections Summary
>>>>> Total connections handled 2
>>>>> SMC connections 2
>>>>> Handshake errors 0
>>>>> Avg requests per SMC conn 1277462.0
>>>>> TCP fallback 0
>>>>>
>>>>> RX Stats
>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>> Total requests 1277190
>>>>> Buffer full 45 (0.00%)
>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>
>>>>> TX Stats
>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>> Total requests 1277734
>>>>> Buffer full 638239 (49.95%)
>>>>> Buffer full (remote) 0 (0.00%)
>>>>> Buffer too small 0 (0.00%)
>>>>> Buffer too small (remote) 0 (0.00%)
>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>
>>>>> Extras
>>>>> Special socket calls 1
>>>>>
>>>>> I captured the perf information and found that the percentage of
>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>>> during tx and rx.
>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>
>>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>>> Thanks.
>>>>
>>>> Hi Zhengchao Shao,
>>>>
>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
>>>> like TCP, it is necessary to appropriately increase the default value of smc
>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>
>>>> Set this and try again:
>>>>
>>>> # sysctl -w net.smc.wmem=1048576
>>>> # sysctl -w net.smc.rmem=1048576
>>>>
>>>> (The initial value of wmem and rmem are 64K)
>>>>
>>>> Regards,
>>>> Wen Gu
>>>>
>>>>>
>>>>> Zhengchao Shao
>>> Hi Wen Gu:
>>> It solves the issue. Thank you very much.
>>>
>>> Zhengchao Shao
>>>
>> Hi Wen Gu:
>> I've tested all the performance test items in the patchset. The
>> performance improvement is to be expected, except for nignx.
>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>> the following command:
>> <smc_run> nignx
>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>
>> The test results are as follows:
>> TCP SMC_lo
>> Requests/s 309425.42 135547.25(-56.19%)
>> The performance decreases by 56.19%.
>>
>> I capture packets and find that wrk can perform HTTP GET after each
>> connect when smc_loopback is disabled.
>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>> I wonder if there is some compatibility problem with the SMC protocol when encapsulate packet? Could you give me some
>> advice?
>> In the attachment, I captured some of the packets.
>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>> enabled.
>> Thank you very much.
>>
>> Zhengchao Shao
>>
>>
>>
> Hi Wen Gu:
> When the VM is configured with 8 cores and 16 GB memory, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
> the test data is as follows:
> TCP SMC_lo
> Requests/s 66056.66 94526.66(43.10%)
>
> But When the VM is configured with 48 cores and 32 GB memory, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
> the test data is as follows:
> TCP SMC_lo
> Requests/s 309425.42 135547.25(-56.19%)
>
> It seems that in the scenario with a large number of CPU cores,
> performance is not optimized, but performance deteriorates. What I
> missed?
> Thank you.
>
> Zhengchao Shao
Hi Zhengchao Shao,
Somehow I missed your email on 12/05. I will try to reproduce this
issue in my test environment and let you know if I find anything.
Best Regards,
Wen Gu
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-13 9:00 ` shaozhengchao
2023-12-13 9:41 ` Wen Gu
@ 2023-12-13 12:59 ` Wen Gu
2023-12-14 3:17 ` shaozhengchao
1 sibling, 1 reply; 14+ messages in thread
From: Wen Gu @ 2023-12-13 12:59 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/13 17:00, shaozhengchao wrote:
>
>
> On 2023/12/5 14:45, shaozhengchao wrote:
>>
>>
>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>
>>>
>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>
>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>> Hi Wen Gu:
>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>>>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>>>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>>>>>> my analysis:
>>>>>>>>
>>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>>> packets on the lo device.
>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>
>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>>>>>> SMC-R-Deline packets.
>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>
>>>>>>>> 3. Rx
>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>>>>>
>>>>>>>> 4. Tx
>>>>>>>> In smc_clc_send_proposal,
>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>
>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>>>>>> always equal to 0.
>>>>>>>>
>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>>> But I don't see where operations can be added, may I missed something?
>>>>>>>>
>>>>>>>
>>>>>>> Hi Zhengchao Shao,
>>>>>>>
>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>
>>>>>>> - Install smc-tools[1].
>>>>>>>
>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>
>>>>>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>>>>>> unavailable, an UEID is required.
>>>>>>>
>>>>>> Hi Wen Gu:
>>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>>> be happy to learn from it.
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Zhengchao Shao
>>>>>>> - Then run the test.
>>>>>>>
>>>>>>> Hope this works for you :)
>>>>>>>
>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>
>>>>>>> Regards,
>>>>>>> Wen Gu
>>>>>>>
>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>
>>>>>>>> Zhengchao Shao
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]link:
>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>> Hi Wen Gu:
>>>>> I have test as following, but the performance is really
>>>>> degraded. Now I have no idea.
>>>>> 1. add ueid
>>>>> run: smcd ueid add 16
>>>>> kernel message:
>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>> 2. start server
>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>> 3. start client
>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>
>>>>> The test results are as follows:
>>>>> TCP SMC-lo
>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>
>>>>> I didn't find a better direction when I initially positioned it. No
>>>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>>>> [root@localhost smc-tools]# smcd stats
>>>>> SMC-D Connections Summary
>>>>> Total connections handled 2
>>>>> SMC connections 2
>>>>> Handshake errors 0
>>>>> Avg requests per SMC conn 1277462.0
>>>>> TCP fallback 0
>>>>>
>>>>> RX Stats
>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>> Total requests 1277190
>>>>> Buffer full 45 (0.00%)
>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>
>>>>> TX Stats
>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>> Total requests 1277734
>>>>> Buffer full 638239 (49.95%)
>>>>> Buffer full (remote) 0 (0.00%)
>>>>> Buffer too small 0 (0.00%)
>>>>> Buffer too small (remote) 0 (0.00%)
>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>
>>>>> Extras
>>>>> Special socket calls 1
>>>>>
>>>>> I captured the perf information and found that the percentage of
>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>>> during tx and rx.
>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>
>>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>>> Thanks.
>>>>
>>>> Hi Zhengchao Shao,
>>>>
>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
>>>> like TCP, it is necessary to appropriately increase the default value of smc
>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>
>>>> Set this and try again:
>>>>
>>>> # sysctl -w net.smc.wmem=1048576
>>>> # sysctl -w net.smc.rmem=1048576
>>>>
>>>> (The initial value of wmem and rmem are 64K)
>>>>
>>>> Regards,
>>>> Wen Gu
>>>>
>>>>>
>>>>> Zhengchao Shao
>>> Hi Wen Gu:
>>> It solves the issue. Thank you very much.
>>>
>>> Zhengchao Shao
>>>
>> Hi Wen Gu:
>> I've tested all the performance test items in the patchset. The
>> performance improvement is to be expected, except for nignx.
>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>> the following command:
>> <smc_run> nignx
>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>
>> The test results are as follows:
>> TCP SMC_lo
>> Requests/s 309425.42 135547.25(-56.19%)
>> The performance decreases by 56.19%.
>>
>> I capture packets and find that wrk can perform HTTP GET after each
>> connect when smc_loopback is disabled.
>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>> I wonder if there is some compatibility problem with the SMC protocol when encapsulate packet? Could you give me some
>> advice?
>> In the attachment, I captured some of the packets.
>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>> enabled.
>> Thank you very much.
>>
>> Zhengchao Shao
>>
>>
>>
> Hi Wen Gu:
> When the VM is configured with 8 cores and 16 GB memory, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
> the test data is as follows:
> TCP SMC_lo
> Requests/s 66056.66 94526.66(43.10%)
>
> But When the VM is configured with 48 cores and 32 GB memory, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
> the test data is as follows:
> TCP SMC_lo
> Requests/s 309425.42 135547.25(-56.19%)
>
> It seems that in the scenario with a large number of CPU cores,
> performance is not optimized, but performance deteriorates. What I
> missed?
> Thank you.
>
> Zhengchao Shao
Hi Zhengchao,
I failed to reproduce this large regression. Could you please share some
information about your test environment?
- The nginx configure.
- The guest(VM) cpu topology.
- The host(physical machine) cpu topology.
- The mapping relationship between vcpu of guest(VM) and physical cpu of host.
- The cpu usage (top) when regression happens.
Thank you.
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-05 6:45 ` shaozhengchao
2023-12-13 9:00 ` shaozhengchao
@ 2023-12-13 13:06 ` Wen Gu
1 sibling, 0 replies; 14+ messages in thread
From: Wen Gu @ 2023-12-13 13:06 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/5 14:45, shaozhengchao wrote:
>
>
> On 2023/12/4 12:06, shaozhengchao wrote:
>>
>>
>> On 2023/12/4 11:52, Wen Gu wrote:
>>>
>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>> Hi Wen Gu:
>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>>>>> my analysis:
>>>>>>>
>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>> packets on the lo device.
>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>
>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>>>>> SMC-R-Deline packets.
>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>
>>>>>>> 3. Rx
>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>>>>
>>>>>>> 4. Tx
>>>>>>> In smc_clc_send_proposal,
>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>
>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>>>>> always equal to 0.
>>>>>>>
>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>> But I don't see where operations can be added, may I missed something?
>>>>>>>
>>>>>>
>>>>>> Hi Zhengchao Shao,
>>>>>>
>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>>>>> UEID should be set. It can be set by following steps:
>>>>>>
>>>>>> - Install smc-tools[1].
>>>>>>
>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>
>>>>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>>>>> unavailable, an UEID is required.
>>>>>>
>>>>> Hi Wen Gu:
>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>> be happy to learn from it.
>>>>>
>>>>> Thanks
>>>>>
>>>>> Zhengchao Shao
>>>>>> - Then run the test.
>>>>>>
>>>>>> Hope this works for you :)
>>>>>>
>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>
>>>>>> Regards,
>>>>>> Wen Gu
>>>>>>
>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>
>>>>>>> Zhengchao Shao
>>>>>>>
>>>>>>>
>>>>>>> [1]link:
>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>> Hi Wen Gu:
>>>> I have test as following, but the performance is really
>>>> degraded. Now I have no idea.
>>>> 1. add ueid
>>>> run: smcd ueid add 16
>>>> kernel message:
>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>> 2. start server
>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>> 3. start client
>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>
>>>> The test results are as follows:
>>>> TCP SMC-lo
>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>
>>>> I didn't find a better direction when I initially positioned it. No
>>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>>> [root@localhost smc-tools]# smcd stats
>>>> SMC-D Connections Summary
>>>> Total connections handled 2
>>>> SMC connections 2
>>>> Handshake errors 0
>>>> Avg requests per SMC conn 1277462.0
>>>> TCP fallback 0
>>>>
>>>> RX Stats
>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>> Total requests 1277190
>>>> Buffer full 45 (0.00%)
>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>> Bufs 0 0 0 2 0 0 0 0
>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>
>>>> TX Stats
>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>> Total requests 1277734
>>>> Buffer full 638239 (49.95%)
>>>> Buffer full (remote) 0 (0.00%)
>>>> Buffer too small 0 (0.00%)
>>>> Buffer too small (remote) 0 (0.00%)
>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>> Bufs 0 0 0 0 0 0 0 0
>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>
>>>> Extras
>>>> Special socket calls 1
>>>>
>>>> I captured the perf information and found that the percentage of
>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>> during tx and rx.
>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>
>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>> Thanks.
>>>
>>> Hi Zhengchao Shao,
>>>
>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
>>> like TCP, it is necessary to appropriately increase the default value of smc
>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>
>>> Set this and try again:
>>>
>>> # sysctl -w net.smc.wmem=1048576
>>> # sysctl -w net.smc.rmem=1048576
>>>
>>> (The initial value of wmem and rmem are 64K)
>>>
>>> Regards,
>>> Wen Gu
>>>
>>>>
>>>> Zhengchao Shao
>> Hi Wen Gu:
>> It solves the issue. Thank you very much.
>>
>> Zhengchao Shao
>>
> Hi Wen Gu:
> I've tested all the performance test items in the patchset. The
> performance improvement is to be expected, except for nignx.
> My VM is configured with 48 cores and 32 GB memory. Therefore, run
> the following command:
> <smc_run> nignx
> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>
> The test results are as follows:
> TCP SMC_lo
> Requests/s 309425.42 135547.25(-56.19%)
> The performance decreases by 56.19%.
>
> I capture packets and find that wrk can perform HTTP GET after each
> connect when smc_loopback is disabled.
> However, when smc_loopback is enabled, there is no HTTP GET behavior.
The HTTP GET is transfered by SMC stack through shared memory. So they
can't be captured by tcpdump.
> I wonder if there is some compatibility problem with the SMC protocol when encapsulate packet? Could you give me some
> advice?
> In the attachment, I captured some of the packets.
> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
> enabled.
> Thank you very much.
>
> Zhengchao Shao
>
>
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-13 12:59 ` Wen Gu
@ 2023-12-14 3:17 ` shaozhengchao
2023-12-26 11:52 ` Wen Gu
0 siblings, 1 reply; 14+ messages in thread
From: shaozhengchao @ 2023-12-14 3:17 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
[-- Attachment #1: Type: text/plain, Size: 9781 bytes --]
On 2023/12/13 20:59, Wen Gu wrote:
> On 2023/12/13 17:00, shaozhengchao wrote:
>>
>>
>> On 2023/12/5 14:45, shaozhengchao wrote:
>>>
>>>
>>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>>
>>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>>> Hi Wen Gu:
>>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO
>>>>>>>>> feature on
>>>>>>>>> my x86_64 environment and kernel is based on linux-next,
>>>>>>>>> commit: 5ba73bec5e7b.
>>>>>>>>> The test result shows that the smc_lo feature cannot be
>>>>>>>>> enabled. Here's
>>>>>>>>> my analysis:
>>>>>>>>>
>>>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>>>> packets on the lo device.
>>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp
>>>>>>>>> --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>>
>>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port
>>>>>>>>> replies with
>>>>>>>>> SMC-R-Deline packets.
>>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>>
>>>>>>>>> 3. Rx
>>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port
>>>>>>>>> returns
>>>>>>>>> a Decline packet because eid_cnt and flag.seid in the received
>>>>>>>>> packet are both 0.
>>>>>>>>>
>>>>>>>>> 4. Tx
>>>>>>>>> In smc_clc_send_proposal,
>>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>>
>>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>>>>>>>> seid_enabled is
>>>>>>>>> always equal to 0.
>>>>>>>>>
>>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>>>> But I don't see where operations can be added, may I missed
>>>>>>>>> something?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Zhengchao Shao,
>>>>>>>>
>>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here),
>>>>>>>> A common
>>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>>
>>>>>>>> - Install smc-tools[1].
>>>>>>>>
>>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>>
>>>>>>>> EID works as an ID to indicate the max communication space of
>>>>>>>> SMC. When SEID is
>>>>>>>> unavailable, an UEID is required.
>>>>>>>>
>>>>>>> Hi Wen Gu:
>>>>>>> Thank you for your reply. This is very useful for me. And I
>>>>>>> will
>>>>>>> be happy to learn from it.
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Zhengchao Shao
>>>>>>>> - Then run the test.
>>>>>>>>
>>>>>>>> Hope this works for you :)
>>>>>>>>
>>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Wen Gu
>>>>>>>>
>>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>>
>>>>>>>>> Zhengchao Shao
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [1]link:
>>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Hi Wen Gu:
>>>>>> I have test as following, but the performance is really
>>>>>> degraded. Now I have no idea.
>>>>>> 1. add ueid
>>>>>> run: smcd ueid add 16
>>>>>> kernel message:
>>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>>> 2. start server
>>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>>> 3. start client
>>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i
>>>>>> 127.0.0.1 -t 30
>>>>>>
>>>>>> The test results are as follows:
>>>>>> TCP SMC-lo
>>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>>
>>>>>> I didn't find a better direction when I initially positioned it. No
>>>>>> error is recorded in the kernel log, and the smcd statistics are
>>>>>> normal.
>>>>>> [root@localhost smc-tools]# smcd stats
>>>>>> SMC-D Connections Summary
>>>>>> Total connections handled 2
>>>>>> SMC connections 2
>>>>>> Handshake errors 0
>>>>>> Avg requests per SMC conn 1277462.0
>>>>>> TCP fallback 0
>>>>>>
>>>>>> RX Stats
>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>> Total requests 1277190
>>>>>> Buffer full 45 (0.00%)
>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>>>> >512KB
>>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>>
>>>>>> TX Stats
>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>> Total requests 1277734
>>>>>> Buffer full 638239 (49.95%)
>>>>>> Buffer full (remote) 0 (0.00%)
>>>>>> Buffer too small 0 (0.00%)
>>>>>> Buffer too small (remote) 0 (0.00%)
>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>>>> >512KB
>>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>>
>>>>>> Extras
>>>>>> Special socket calls 1
>>>>>>
>>>>>> I captured the perf information and found that the percentage of
>>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was
>>>>>> high
>>>>>> during tx and rx.
>>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>>
>>>>>> I've attached the flame map. Could you help analyze it? What I
>>>>>> missed?
>>>>>> Thanks.
>>>>>
>>>>> Hi Zhengchao Shao,
>>>>>
>>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow
>>>>> dynamically
>>>>> like TCP, it is necessary to appropriately increase the default
>>>>> value of smc
>>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>>
>>>>> Set this and try again:
>>>>>
>>>>> # sysctl -w net.smc.wmem=1048576
>>>>> # sysctl -w net.smc.rmem=1048576
>>>>>
>>>>> (The initial value of wmem and rmem are 64K)
>>>>>
>>>>> Regards,
>>>>> Wen Gu
>>>>>
>>>>>>
>>>>>> Zhengchao Shao
>>>> Hi Wen Gu:
>>>> It solves the issue. Thank you very much.
>>>>
>>>> Zhengchao Shao
>>>>
>>> Hi Wen Gu:
>>> I've tested all the performance test items in the patchset. The
>>> performance improvement is to be expected, except for nignx.
>>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>>> the following command:
>>> <smc_run> nignx
>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>
>>> The test results are as follows:
>>> TCP SMC_lo
>>> Requests/s 309425.42 135547.25(-56.19%)
>>> The performance decreases by 56.19%.
>>>
>>> I capture packets and find that wrk can perform HTTP GET after each
>>> connect when smc_loopback is disabled.
>>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>>> I wonder if there is some compatibility problem with the SMC protocol
>>> when encapsulate packet? Could you give me some advice?
>>> In the attachment, I captured some of the packets.
>>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>>> enabled.
>>> Thank you very much.
>>>
>>> Zhengchao Shao
>>>
>>>
>>>
>> Hi Wen Gu:
>> When the VM is configured with 8 cores and 16 GB memory, run
>> the following command:
>> <smc_run> nignx
>> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
>> the test data is as follows:
>> TCP SMC_lo
>> Requests/s 66056.66 94526.66(43.10%)
>>
>> But When the VM is configured with 48 cores and 32 GB memory, run
>> the following command:
>> <smc_run> nignx
>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>> the test data is as follows:
>> TCP SMC_lo
>> Requests/s 309425.42 135547.25(-56.19%)
>>
>> It seems that in the scenario with a large number of CPU cores,
>> performance is not optimized, but performance deteriorates. What I
>> missed?
>> Thank you.
>>
>> Zhengchao Shao
>
> Hi Zhengchao,
>
> I failed to reproduce this large regression. Could you please share some
> information about your test environment?
>
Hi Wen Gu:
> - The nginx configure.
See the nginx.conf file in the attachment.
> - The guest(VM) cpu topology.
See the vm_cpuinfo file in the attachment.
> - The host(physical machine) cpu topology.
See the host_cpuinfo file in the attachment.
> - The mapping relationship between vcpu of guest(VM) and physical cpu of
> host.
See the cpu_map file in the attachment.
> - The cpu usage (top) when regression happens.
>
See the perf_top and perf.svg file in the attachment.
> Thank you.
>
Thank you very much.
Zhengchao Shao
[-- Attachment #2: cpu_map --]
[-- Type: text/plain, Size: 7215 bytes --]
VCPU: 0
CPU: 5
State: running
CPU time: 274.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 1
CPU: 3
State: running
CPU time: 311.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 2
CPU: 27
State: running
CPU time: 287.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 3
CPU: 9
State: running
CPU time: 301.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 4
CPU: 7
State: running
CPU time: 271.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 5
CPU: 10
State: running
CPU time: 280.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 6
CPU: 17
State: running
CPU time: 259.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 7
CPU: 13
State: running
CPU time: 277.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 8
CPU: 29
State: running
CPU time: 263.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 9
CPU: 2
State: running
CPU time: 278.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 10
CPU: 11
State: running
CPU time: 256.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 11
CPU: 19
State: running
CPU time: 289.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 12
CPU: 6
State: running
CPU time: 254.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 13
CPU: 4
State: running
CPU time: 304.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 14
CPU: 10
State: running
CPU time: 269.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 15
CPU: 3
State: running
CPU time: 264.2s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 16
CPU: 2
State: running
CPU time: 277.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 17
CPU: 4
State: running
CPU time: 269.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 18
CPU: 9
State: running
CPU time: 248.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 19
CPU: 13
State: running
CPU time: 289.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 20
CPU: 8
State: running
CPU time: 280.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 21
CPU: 4
State: running
CPU time: 288.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 22
CPU: 5
State: running
CPU time: 290.7s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 23
CPU: 9
State: running
CPU time: 287.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 24
CPU: 3
State: running
CPU time: 292.8s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 25
CPU: 30
State: running
CPU time: 264.2s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 26
CPU: 12
State: running
CPU time: 287.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 27
CPU: 7
State: running
CPU time: 300.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 28
CPU: 4
State: running
CPU time: 257.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 29
CPU: 34
State: running
CPU time: 262.7s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 30
CPU: 31
State: running
CPU time: 299.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 31
CPU: 18
State: running
CPU time: 260.4s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 32
CPU: 3
State: running
CPU time: 263.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 33
CPU: 11
State: running
CPU time: 284.5s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 34
CPU: 9
State: running
CPU time: 254.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 35
CPU: 10
State: running
CPU time: 247.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 36
CPU: 3
State: running
CPU time: 277.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 37
CPU: 10
State: running
CPU time: 283.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 38
CPU: 32
State: running
CPU time: 266.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 39
CPU: 26
State: running
CPU time: 255.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 40
CPU: 24
State: running
CPU time: 240.1s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 41
CPU: 14
State: running
CPU time: 229.6s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 42
CPU: 0
State: running
CPU time: 251.5s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 43
CPU: 33
State: running
CPU time: 236.9s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 44
CPU: 2
State: running
CPU time: 262.3s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 45
CPU: 16
State: running
CPU time: 241.0s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 46
CPU: 15
State: running
CPU time: 261.2s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
VCPU: 47
CPU: 13
State: running
CPU time: 284.5s
CPU Affinity: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
[-- Attachment #3: host_cpuinfo --]
[-- Type: text/plain, Size: 54754 bytes --]
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.476
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 0
cpu cores : 12
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 1
cpu cores : 12
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 2
cpu cores : 12
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 3
cpu cores : 12
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 4
cpu cores : 12
apicid : 8
initial apicid : 8
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2602.539
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 5
cpu cores : 12
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.158
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 8
cpu cores : 12
apicid : 16
initial apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 9
cpu cores : 12
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 10
cpu cores : 12
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 11
cpu cores : 12
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2601.428
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 12
cpu cores : 12
apicid : 24
initial apicid : 24
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 13
cpu cores : 12
apicid : 26
initial apicid : 26
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 12
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 0
cpu cores : 12
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 13
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 1
cpu cores : 12
apicid : 34
initial apicid : 34
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 14
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2598.095
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 2
cpu cores : 12
apicid : 36
initial apicid : 36
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 3
cpu cores : 12
apicid : 38
initial apicid : 38
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 16
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 4
cpu cores : 12
apicid : 40
initial apicid : 40
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 17
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.523
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 5
cpu cores : 12
apicid : 42
initial apicid : 42
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 18
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 8
cpu cores : 12
apicid : 48
initial apicid : 48
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 19
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.682
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 9
cpu cores : 12
apicid : 50
initial apicid : 50
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 20
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 10
cpu cores : 12
apicid : 52
initial apicid : 52
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 21
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.158
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 11
cpu cores : 12
apicid : 54
initial apicid : 54
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 22
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.682
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 12
cpu cores : 12
apicid : 56
initial apicid : 56
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 13
cpu cores : 12
apicid : 58
initial apicid : 58
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 24
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 0
cpu cores : 12
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 25
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 1
cpu cores : 12
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 26
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2605.712
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 2
cpu cores : 12
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 27
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 3
cpu cores : 12
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 28
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2598.889
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 4
cpu cores : 12
apicid : 9
initial apicid : 9
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 29
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.158
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 5
cpu cores : 12
apicid : 11
initial apicid : 11
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 30
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 8
cpu cores : 12
apicid : 17
initial apicid : 17
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 31
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.047
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 9
cpu cores : 12
apicid : 19
initial apicid : 19
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 32
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 10
cpu cores : 12
apicid : 21
initial apicid : 21
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 33
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 11
cpu cores : 12
apicid : 23
initial apicid : 23
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 34
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2598.571
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 12
cpu cores : 12
apicid : 25
initial apicid : 25
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 35
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 0
siblings : 24
core id : 13
cpu cores : 12
apicid : 27
initial apicid : 27
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5188.46
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 36
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2601.269
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 0
cpu cores : 12
apicid : 33
initial apicid : 33
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 37
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.317
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 1
cpu cores : 12
apicid : 35
initial apicid : 35
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 38
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 2
cpu cores : 12
apicid : 37
initial apicid : 37
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 39
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2601.586
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 3
cpu cores : 12
apicid : 39
initial apicid : 39
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 40
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.000
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 4
cpu cores : 12
apicid : 41
initial apicid : 41
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 41
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.523
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 5
cpu cores : 12
apicid : 43
initial apicid : 43
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 42
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.158
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 8
cpu cores : 12
apicid : 49
initial apicid : 49
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 43
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.365
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 9
cpu cores : 12
apicid : 51
initial apicid : 51
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 44
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.365
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 10
cpu cores : 12
apicid : 53
initial apicid : 53
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 45
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2599.841
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 11
cpu cores : 12
apicid : 55
initial apicid : 55
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 46
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.158
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 12
cpu cores : 12
apicid : 57
initial apicid : 57
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 47
vendor_id : GenuineIntel
cpu family : 6
model : 63
model name : Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz
stepping : 2
microcode : 0x49
cpu MHz : 2600.317
cache size : 30720 KB
physical id : 1
siblings : 24
core id : 13
cpu cores : 12
apicid : 59
initial apicid : 59
fpu : yes
fpu_exception : yes
cpuid level : 15
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpcid_single intel_ppin ssbd rsb_ctxsw ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm arat pln pts md_clear spec_ctrl intel_stibp flush_l1d
bogomips : 5200.75
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
[-- Attachment #4: nginx.conf --]
[-- Type: text/plain, Size: 2342 bytes --]
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;
events {
worker_connections 1024;
}
http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 4096;
include /etc/nginx/mime.types;
default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;
server {
listen 80;
listen [::]:80;
server_name _;
root /usr/share/nginx/html;
# Load configuration files for the default server block.
include /etc/nginx/default.d/*.conf;
error_page 404 /404.html;
location = /40x.html {
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
}
}
# Settings for a TLS enabled server.
#
# server {
# listen 443 ssl http2;
# listen [::]:443 ssl http2;
# server_name _;
# root /usr/share/nginx/html;
#
# ssl_certificate "/etc/pki/nginx/server.crt";
# ssl_certificate_key "/etc/pki/nginx/private/server.key";
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout 10m;
# ssl_ciphers PROFILE=SYSTEM;
# ssl_prefer_server_ciphers on;
#
# # Load configuration files for the default server block.
# include /etc/nginx/default.d/*.conf;
#
# error_page 404 /404.html;
# location = /40x.html {
# }
#
# error_page 500 502 503 504 /50x.html;
# location = /50x.html {
# }
# }
}
[-- Attachment #5: perf.svg --]
[-- Type: image/svg+xml, Size: 267364 bytes --]
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" width="1200" height="2134" onload="init(evt)" viewBox="0 0 1200 2134" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Flame graph stack visualization. See https://github.com/brendangregg/FlameGraph for latest version, and http://www.brendangregg.com/flamegraphs.html for examples. -->
<!-- NOTES: -->
<defs>
<linearGradient id="background" y1="0" y2="1" x1="0" x2="0" >
<stop stop-color="#eeeeee" offset="5%" />
<stop stop-color="#eeeeb0" offset="95%" />
</linearGradient>
</defs>
<style type="text/css">
text { font-family:Verdana; font-size:12px; fill:rgb(0,0,0); }
#search, #ignorecase { opacity:0.1; cursor:pointer; }
#search:hover, #search.show, #ignorecase:hover, #ignorecase.show { opacity:1; }
#subtitle { text-anchor:middle; font-color:rgb(160,160,160); }
#title { text-anchor:middle; font-size:17px}
#unzoom { cursor:pointer; }
#frames > *:hover { stroke:black; stroke-width:0.5; cursor:pointer; }
.hide { display:none; }
.parent { opacity:0.5; }
</style>
<script type="text/ecmascript">
<![CDATA[
"use strict";
var details, searchbtn, unzoombtn, matchedtxt, svg, searching, currentSearchTerm, ignorecase, ignorecaseBtn;
function init(evt) {
details = document.getElementById("details").firstChild;
searchbtn = document.getElementById("search");
ignorecaseBtn = document.getElementById("ignorecase");
unzoombtn = document.getElementById("unzoom");
matchedtxt = document.getElementById("matched");
svg = document.getElementsByTagName("svg")[0];
searching = 0;
currentSearchTerm = null;
// use GET parameters to restore a flamegraphs state.
var params = get_params();
if (params.x && params.y)
zoom(find_group(document.querySelector('[x="' + params.x + '"][y="' + params.y + '"]')));
if (params.s) search(params.s);
}
// event listeners
window.addEventListener("click", function(e) {
var target = find_group(e.target);
if (target) {
if (target.nodeName == "a") {
if (e.ctrlKey === false) return;
e.preventDefault();
}
if (target.classList.contains("parent")) unzoom();
zoom(target);
if (!document.querySelector('.parent')) {
clearzoom();
return;
}
// set parameters for zoom state
var el = target.querySelector("rect");
if (el && el.attributes && el.attributes.y && el.attributes._orig_x) {
var params = get_params()
params.x = el.attributes._orig_x.value;
params.y = el.attributes.y.value;
history.replaceState(null, null, parse_params(params));
}
}
else if (e.target.id == "unzoom") clearzoom();
else if (e.target.id == "search") search_prompt();
else if (e.target.id == "ignorecase") toggle_ignorecase();
}, false)
// mouse-over for info
// show
window.addEventListener("mouseover", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = "Function: " + g_to_text(target);
}, false)
// clear
window.addEventListener("mouseout", function(e) {
var target = find_group(e.target);
if (target) details.nodeValue = ' ';
}, false)
// ctrl-F for search
// ctrl-I to toggle case-sensitive search
window.addEventListener("keydown",function (e) {
if (e.keyCode === 114 || (e.ctrlKey && e.keyCode === 70)) {
e.preventDefault();
search_prompt();
}
else if (e.ctrlKey && e.keyCode === 73) {
e.preventDefault();
toggle_ignorecase();
}
}, false)
// functions
function get_params() {
var params = {};
var paramsarr = window.location.search.substr(1).split('&');
for (var i = 0; i < paramsarr.length; ++i) {
var tmp = paramsarr[i].split("=");
if (!tmp[0] || !tmp[1]) continue;
params[tmp[0]] = decodeURIComponent(tmp[1]);
}
return params;
}
function parse_params(params) {
var uri = "?";
for (var key in params) {
uri += key + '=' + encodeURIComponent(params[key]) + '&';
}
if (uri.slice(-1) == "&")
uri = uri.substring(0, uri.length - 1);
if (uri == '?')
uri = window.location.href.split('?')[0];
return uri;
}
function find_child(node, selector) {
var children = node.querySelectorAll(selector);
if (children.length) return children[0];
}
function find_group(node) {
var parent = node.parentElement;
if (!parent) return;
if (parent.id == "frames") return node;
return find_group(parent);
}
function orig_save(e, attr, val) {
if (e.attributes["_orig_" + attr] != undefined) return;
if (e.attributes[attr] == undefined) return;
if (val == undefined) val = e.attributes[attr].value;
e.setAttribute("_orig_" + attr, val);
}
function orig_load(e, attr) {
if (e.attributes["_orig_"+attr] == undefined) return;
e.attributes[attr].value = e.attributes["_orig_" + attr].value;
e.removeAttribute("_orig_"+attr);
}
function g_to_text(e) {
var text = find_child(e, "title").firstChild.nodeValue;
return (text)
}
function g_to_func(e) {
var func = g_to_text(e);
// if there's any manipulation we want to do to the function
// name before it's searched, do it here before returning.
return (func);
}
function update_text(e) {
var r = find_child(e, "rect");
var t = find_child(e, "text");
var w = parseFloat(r.attributes.width.value) -3;
var txt = find_child(e, "title").textContent.replace(/\([^(]*\)$/,"");
t.attributes.x.value = parseFloat(r.attributes.x.value) + 3;
// Smaller than this size won't fit anything
if (w < 2 * 12 * 0.59) {
t.textContent = "";
return;
}
t.textContent = txt;
// Fit in full text width
if (/^ *$/.test(txt) || t.getSubStringLength(0, txt.length) < w)
return;
for (var x = txt.length - 2; x > 0; x--) {
if (t.getSubStringLength(0, x + 2) <= w) {
t.textContent = txt.substring(0, x) + "..";
return;
}
}
t.textContent = "";
}
// zoom
function zoom_reset(e) {
if (e.attributes != undefined) {
orig_load(e, "x");
orig_load(e, "width");
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_reset(c[i]);
}
}
function zoom_child(e, x, ratio) {
if (e.attributes != undefined) {
if (e.attributes.x != undefined) {
orig_save(e, "x");
e.attributes.x.value = (parseFloat(e.attributes.x.value) - x - 10) * ratio + 10;
if (e.tagName == "text")
e.attributes.x.value = find_child(e.parentNode, "rect[x]").attributes.x.value + 3;
}
if (e.attributes.width != undefined) {
orig_save(e, "width");
e.attributes.width.value = parseFloat(e.attributes.width.value) * ratio;
}
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_child(c[i], x - 10, ratio);
}
}
function zoom_parent(e) {
if (e.attributes) {
if (e.attributes.x != undefined) {
orig_save(e, "x");
e.attributes.x.value = 10;
}
if (e.attributes.width != undefined) {
orig_save(e, "width");
e.attributes.width.value = parseInt(svg.width.baseVal.value) - (10 * 2);
}
}
if (e.childNodes == undefined) return;
for (var i = 0, c = e.childNodes; i < c.length; i++) {
zoom_parent(c[i]);
}
}
function zoom(node) {
var attr = find_child(node, "rect").attributes;
var width = parseFloat(attr.width.value);
var xmin = parseFloat(attr.x.value);
var xmax = parseFloat(xmin + width);
var ymin = parseFloat(attr.y.value);
var ratio = (svg.width.baseVal.value - 2 * 10) / width;
// XXX: Workaround for JavaScript float issues (fix me)
var fudge = 0.0001;
unzoombtn.classList.remove("hide");
var el = document.getElementById("frames").children;
for (var i = 0; i < el.length; i++) {
var e = el[i];
var a = find_child(e, "rect").attributes;
var ex = parseFloat(a.x.value);
var ew = parseFloat(a.width.value);
var upstack;
// Is it an ancestor
if (0 == 0) {
upstack = parseFloat(a.y.value) > ymin;
} else {
upstack = parseFloat(a.y.value) < ymin;
}
if (upstack) {
// Direct ancestor
if (ex <= xmin && (ex+ew+fudge) >= xmax) {
e.classList.add("parent");
zoom_parent(e);
update_text(e);
}
// not in current path
else
e.classList.add("hide");
}
// Children maybe
else {
// no common path
if (ex < xmin || ex + fudge >= xmax) {
e.classList.add("hide");
}
else {
zoom_child(e, xmin, ratio);
update_text(e);
}
}
}
search();
}
function unzoom() {
unzoombtn.classList.add("hide");
var el = document.getElementById("frames").children;
for(var i = 0; i < el.length; i++) {
el[i].classList.remove("parent");
el[i].classList.remove("hide");
zoom_reset(el[i]);
update_text(el[i]);
}
search();
}
function clearzoom() {
unzoom();
// remove zoom state
var params = get_params();
if (params.x) delete params.x;
if (params.y) delete params.y;
history.replaceState(null, null, parse_params(params));
}
// search
function toggle_ignorecase() {
ignorecase = !ignorecase;
if (ignorecase) {
ignorecaseBtn.classList.add("show");
} else {
ignorecaseBtn.classList.remove("show");
}
reset_search();
search();
}
function reset_search() {
var el = document.querySelectorAll("#frames rect");
for (var i = 0; i < el.length; i++) {
orig_load(el[i], "fill")
}
var params = get_params();
delete params.s;
history.replaceState(null, null, parse_params(params));
}
function search_prompt() {
if (!searching) {
var term = prompt("Enter a search term (regexp " +
"allowed, eg: ^ext4_)"
+ (ignorecase ? ", ignoring case" : "")
+ "\nPress Ctrl-i to toggle case sensitivity", "");
if (term != null) search(term);
} else {
reset_search();
searching = 0;
currentSearchTerm = null;
searchbtn.classList.remove("show");
searchbtn.firstChild.nodeValue = "Search"
matchedtxt.classList.add("hide");
matchedtxt.firstChild.nodeValue = ""
}
}
function search(term) {
if (term) currentSearchTerm = term;
var re = new RegExp(currentSearchTerm, ignorecase ? 'i' : '');
var el = document.getElementById("frames").children;
var matches = new Object();
var maxwidth = 0;
for (var i = 0; i < el.length; i++) {
var e = el[i];
var func = g_to_func(e);
var rect = find_child(e, "rect");
if (func == null || rect == null)
continue;
// Save max width. Only works as we have a root frame
var w = parseFloat(rect.attributes.width.value);
if (w > maxwidth)
maxwidth = w;
if (func.match(re)) {
// highlight
var x = parseFloat(rect.attributes.x.value);
orig_save(rect, "fill");
rect.attributes.fill.value = "rgb(230,0,230)";
// remember matches
if (matches[x] == undefined) {
matches[x] = w;
} else {
if (w > matches[x]) {
// overwrite with parent
matches[x] = w;
}
}
searching = 1;
}
}
if (!searching)
return;
var params = get_params();
params.s = currentSearchTerm;
history.replaceState(null, null, parse_params(params));
searchbtn.classList.add("show");
searchbtn.firstChild.nodeValue = "Reset Search";
// calculate percent matched, excluding vertical overlap
var count = 0;
var lastx = -1;
var lastw = 0;
var keys = Array();
for (k in matches) {
if (matches.hasOwnProperty(k))
keys.push(k);
}
// sort the matched frames by their x location
// ascending, then width descending
keys.sort(function(a, b){
return a - b;
});
// Step through frames saving only the biggest bottom-up frames
// thanks to the sort order. This relies on the tree property
// where children are always smaller than their parents.
var fudge = 0.0001; // JavaScript floating point
for (var k in keys) {
var x = parseFloat(keys[k]);
var w = matches[keys[k]];
if (x >= lastx + lastw - fudge) {
count += w;
lastx = x;
lastw = w;
}
}
// display matched percent
matchedtxt.classList.remove("hide");
var pct = 100 * count / maxwidth;
if (pct != 100) pct = pct.toFixed(1)
matchedtxt.firstChild.nodeValue = "Matched: " + pct + "%";
}
]]>
</script>
<rect x="0.0" y="0" width="1200.0" height="2134.0" fill="url(#background)" />
<text id="title" x="600.00" y="24" >Flame Graph</text>
<text id="details" x="10.00" y="2117" > </text>
<text id="unzoom" x="10.00" y="24" class="hide">Reset Zoom</text>
<text id="search" x="1090.00" y="24" >Search</text>
<text id="ignorecase" x="1174.00" y="24" >ic</text>
<text id="matched" x="1090.00" y="2117" > </text>
<g id="frames">
<g >
<title>do_syscall_64 (4,494 samples, 1.58%)</title><rect x="650.2" y="2005" width="18.6" height="15.0" fill="rgb(212,19,39)" rx="2" ry="2" />
<text x="653.22" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1557" width="364.1" height="15.0" fill="rgb(236,142,39)" rx="2" ry="2" />
<text x="790.11" y="1567.5" >[unknown]</text>
</g>
<g >
<title>sock_write_iter (189 samples, 0.07%)</title><rect x="822.3" y="133" width="0.8" height="15.0" fill="rgb(217,12,31)" rx="2" ry="2" />
<text x="825.30" y="143.5" ></text>
</g>
<g >
<title>pv_wait_head_or_lock (59 samples, 0.02%)</title><rect x="491.6" y="1781" width="0.3" height="15.0" fill="rgb(227,70,6)" rx="2" ry="2" />
<text x="494.64" y="1791.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1077" width="364.1" height="15.0" fill="rgb(214,79,21)" rx="2" ry="2" />
<text x="790.11" y="1087.5" >[unknown]</text>
</g>
<g >
<title>alloc_thread_stack_node (40 samples, 0.01%)</title><rect x="716.9" y="1941" width="0.1" height="15.0" fill="rgb(209,215,24)" rx="2" ry="2" />
<text x="719.87" y="1951.5" ></text>
</g>
<g >
<title>__vmalloc_node_range (29 samples, 0.01%)</title><rect x="716.9" y="1925" width="0.1" height="15.0" fill="rgb(240,165,23)" rx="2" ry="2" />
<text x="719.87" y="1935.5" ></text>
</g>
<g >
<title>kmem_cache_free (133 samples, 0.05%)</title><rect x="262.1" y="1941" width="0.6" height="15.0" fill="rgb(213,209,40)" rx="2" ry="2" />
<text x="265.11" y="1951.5" ></text>
</g>
<g >
<title>vm_mmap_pgoff (41 samples, 0.01%)</title><rect x="717.7" y="1989" width="0.2" height="15.0" fill="rgb(253,53,49)" rx="2" ry="2" />
<text x="720.74" y="1999.5" ></text>
</g>
<g >
<title>vfs_write (253 samples, 0.09%)</title><rect x="811.4" y="117" width="1.0" height="15.0" fill="rgb(227,152,32)" rx="2" ry="2" />
<text x="814.40" y="127.5" ></text>
</g>
<g >
<title>vfs_read (458 samples, 0.16%)</title><rect x="807.6" y="101" width="1.9" height="15.0" fill="rgb(249,85,45)" rx="2" ry="2" />
<text x="810.61" y="111.5" ></text>
</g>
<g >
<title>write (253 samples, 0.09%)</title><rect x="811.4" y="181" width="1.0" height="15.0" fill="rgb(205,97,23)" rx="2" ry="2" />
<text x="814.40" y="191.5" ></text>
</g>
<g >
<title>__x64_sys_newfstatat (26 samples, 0.01%)</title><rect x="667.3" y="1989" width="0.1" height="15.0" fill="rgb(215,223,54)" rx="2" ry="2" />
<text x="670.34" y="1999.5" ></text>
</g>
<g >
<title>check_heap_object (602 samples, 0.21%)</title><rect x="819.8" y="53" width="2.5" height="15.0" fill="rgb(215,65,49)" rx="2" ry="2" />
<text x="822.81" y="63.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (63 samples, 0.02%)</title><rect x="799.3" y="85" width="0.3" height="15.0" fill="rgb(218,115,0)" rx="2" ry="2" />
<text x="802.29" y="95.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (43 samples, 0.02%)</title><rect x="78.6" y="1893" width="0.2" height="15.0" fill="rgb(232,141,52)" rx="2" ry="2" />
<text x="81.62" y="1903.5" ></text>
</g>
<g >
<title>rep_movs_alternative (65 samples, 0.02%)</title><rect x="54.8" y="1813" width="0.3" height="15.0" fill="rgb(236,158,48)" rx="2" ry="2" />
<text x="57.79" y="1823.5" ></text>
</g>
<g >
<title>inode_security (72 samples, 0.03%)</title><rect x="646.1" y="1893" width="0.3" height="15.0" fill="rgb(232,211,46)" rx="2" ry="2" />
<text x="649.14" y="1903.5" ></text>
</g>
<g >
<title>write (86 samples, 0.03%)</title><rect x="1148.3" y="293" width="0.3" height="15.0" fill="rgb(229,60,47)" rx="2" ry="2" />
<text x="1151.28" y="303.5" ></text>
</g>
<g >
<title>sock_recvmsg (888 samples, 0.31%)</title><rect x="838.9" y="165" width="3.7" height="15.0" fill="rgb(251,90,40)" rx="2" ry="2" />
<text x="841.89" y="175.5" ></text>
</g>
<g >
<title>sock_recvmsg (880 samples, 0.31%)</title><rect x="812.9" y="101" width="3.7" height="15.0" fill="rgb(239,176,39)" rx="2" ry="2" />
<text x="815.91" y="111.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (28 samples, 0.01%)</title><rect x="719.6" y="1989" width="0.1" height="15.0" fill="rgb(254,22,28)" rx="2" ry="2" />
<text x="722.57" y="1999.5" ></text>
</g>
<g >
<title>ksys_read (36,893 samples, 12.93%)</title><rect x="992.2" y="229" width="152.5" height="15.0" fill="rgb(230,201,5)" rx="2" ry="2" />
<text x="995.15" y="239.5" >ksys_read</text>
</g>
<g >
<title>do_sys_openat2 (3,958 samples, 1.39%)</title><rect x="685.3" y="1973" width="16.3" height="15.0" fill="rgb(243,214,22)" rx="2" ry="2" />
<text x="688.26" y="1983.5" ></text>
</g>
<g >
<title>__handle_mm_fault (26 samples, 0.01%)</title><rect x="716.2" y="1957" width="0.1" height="15.0" fill="rgb(217,6,41)" rx="2" ry="2" />
<text x="719.20" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1189" width="364.1" height="15.0" fill="rgb(228,215,26)" rx="2" ry="2" />
<text x="790.11" y="1199.5" >[unknown]</text>
</g>
<g >
<title>ngx_http_core_run_phases (261 samples, 0.09%)</title><rect x="41.1" y="2021" width="1.1" height="15.0" fill="rgb(228,61,29)" rx="2" ry="2" />
<text x="44.08" y="2031.5" ></text>
</g>
<g >
<title>__virt_addr_valid (150 samples, 0.05%)</title><rect x="653.2" y="1893" width="0.6" height="15.0" fill="rgb(225,151,47)" rx="2" ry="2" />
<text x="656.21" y="1903.5" ></text>
</g>
<g >
<title>ngx_create_pool (51 samples, 0.02%)</title><rect x="669.3" y="2037" width="0.2" height="15.0" fill="rgb(234,6,6)" rx="2" ry="2" />
<text x="672.30" y="2047.5" ></text>
</g>
<g >
<title>[perf] (43 samples, 0.02%)</title><rect x="715.5" y="2037" width="0.2" height="15.0" fill="rgb(236,204,26)" rx="2" ry="2" />
<text x="718.50" y="2047.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (29 samples, 0.01%)</title><rect x="54.3" y="1829" width="0.1" height="15.0" fill="rgb(240,159,11)" rx="2" ry="2" />
<text x="57.29" y="1839.5" ></text>
</g>
<g >
<title>filename_lookup (2,614 samples, 0.92%)</title><rect x="655.0" y="1941" width="10.8" height="15.0" fill="rgb(234,226,21)" rx="2" ry="2" />
<text x="658.00" y="1951.5" ></text>
</g>
<g >
<title>ext4_file_getattr (201 samples, 0.07%)</title><rect x="654.2" y="1941" width="0.8" height="15.0" fill="rgb(217,220,16)" rx="2" ry="2" />
<text x="657.17" y="1951.5" ></text>
</g>
<g >
<title>aeProcessEvents (540 samples, 0.19%)</title><rect x="1163.0" y="2053" width="2.2" height="15.0" fill="rgb(248,29,44)" rx="2" ry="2" />
<text x="1166.00" y="2063.5" ></text>
</g>
<g >
<title>sock_write_iter (131 samples, 0.05%)</title><rect x="809.7" y="85" width="0.5" height="15.0" fill="rgb(228,129,32)" rx="2" ry="2" />
<text x="812.67" y="95.5" ></text>
</g>
<g >
<title>selinux_file_permission (200 samples, 0.07%)</title><rect x="802.5" y="37" width="0.8" height="15.0" fill="rgb(246,178,2)" rx="2" ry="2" />
<text x="805.51" y="47.5" ></text>
</g>
<g >
<title>find_vmap_area (134 samples, 0.05%)</title><rect x="822.3" y="53" width="0.6" height="15.0" fill="rgb(247,83,15)" rx="2" ry="2" />
<text x="825.30" y="63.5" ></text>
</g>
<g >
<title>ngx_rbtree_insert_timer_value (105 samples, 0.04%)</title><rect x="270.9" y="2021" width="0.4" height="15.0" fill="rgb(242,214,42)" rx="2" ry="2" />
<text x="273.90" y="2031.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (253 samples, 0.09%)</title><rect x="811.4" y="165" width="1.0" height="15.0" fill="rgb(245,202,0)" rx="2" ry="2" />
<text x="814.40" y="175.5" ></text>
</g>
<g >
<title>[unknown] (39 samples, 0.01%)</title><rect x="720.1" y="2053" width="0.2" height="15.0" fill="rgb(226,182,16)" rx="2" ry="2" />
<text x="723.13" y="2063.5" ></text>
</g>
<g >
<title>[unknown] (86,943 samples, 30.47%)</title><rect x="787.1" y="293" width="359.6" height="15.0" fill="rgb(240,226,20)" rx="2" ry="2" />
<text x="790.11" y="303.5" >[unknown]</text>
</g>
<g >
<title>smc_cdc_msg_recv_action (473 samples, 0.17%)</title><rect x="1148.7" y="69" width="1.9" height="15.0" fill="rgb(234,94,22)" rx="2" ry="2" />
<text x="1151.67" y="79.5" ></text>
</g>
<g >
<title>do_syscall_64 (36,168 samples, 12.68%)</title><rect x="842.6" y="229" width="149.6" height="15.0" fill="rgb(235,221,44)" rx="2" ry="2" />
<text x="845.56" y="239.5" >do_syscall_64</text>
</g>
<g >
<title>do_syscall_64 (208 samples, 0.07%)</title><rect x="804.8" y="117" width="0.8" height="15.0" fill="rgb(220,188,0)" rx="2" ry="2" />
<text x="807.76" y="127.5" ></text>
</g>
<g >
<title>vfs_write (86 samples, 0.03%)</title><rect x="1148.3" y="229" width="0.3" height="15.0" fill="rgb(240,131,9)" rx="2" ry="2" />
<text x="1151.28" y="239.5" ></text>
</g>
<g >
<title>[unknown] (88,032 samples, 30.86%)</title><rect x="787.1" y="709" width="364.1" height="15.0" fill="rgb(226,42,29)" rx="2" ry="2" />
<text x="790.11" y="719.5" >[unknown]</text>
</g>
<g >
<title>connect (66 samples, 0.02%)</title><rect x="1165.2" y="2053" width="0.3" height="15.0" fill="rgb(236,161,17)" rx="2" ry="2" />
<text x="1168.24" y="2063.5" ></text>
</g>
<g >
<title>[ngx_http_xslt_filter_module.so] (212 samples, 0.07%)</title><rect x="42.9" y="2037" width="0.9" height="15.0" fill="rgb(240,141,49)" rx="2" ry="2" />
<text x="45.95" y="2047.5" ></text>
</g>
<g >
<title>ngx_http_set_etag (112 samples, 0.04%)</title><rect x="677.1" y="2037" width="0.5" height="15.0" fill="rgb(236,0,50)" rx="2" ry="2" />
<text x="680.12" y="2047.5" ></text>
</g>
<g >
<title>vfs_read (381 samples, 0.13%)</title><rect x="1146.7" y="229" width="1.6" height="15.0" fill="rgb(254,149,34)" rx="2" ry="2" />
<text x="1149.71" y="239.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (40,843 samples, 14.32%)</title><rect x="479.3" y="2005" width="168.9" height="15.0" fill="rgb(234,207,43)" rx="2" ry="2" />
<text x="482.27" y="2015.5" >entry_SYSCALL_64</text>
</g>
<g >
<title>__x64_sys_connect (62 samples, 0.02%)</title><rect x="1165.3" y="2005" width="0.2" height="15.0" fill="rgb(210,34,27)" rx="2" ry="2" />
<text x="1168.25" y="2015.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (1,530 samples, 0.54%)</title><rect x="260.2" y="2005" width="6.3" height="15.0" fill="rgb(251,11,3)" rx="2" ry="2" />
<text x="263.20" y="2015.5" ></text>
</g>
<g >
<title>sock_recvmsg (38,591 samples, 13.53%)</title><rect x="83.3" y="1893" width="159.6" height="15.0" fill="rgb(240,9,10)" rx="2" ry="2" />
<text x="86.29" y="1903.5" >sock_recvmsg</text>
</g>
<g >
<title>alloc_empty_file (634 samples, 0.22%)</title><rect x="686.4" y="1925" width="2.6" height="15.0" fill="rgb(250,199,43)" rx="2" ry="2" />
<text x="689.42" y="1935.5" ></text>
</g>
<g >
<title>may_open (172 samples, 0.06%)</title><rect x="693.3" y="1909" width="0.7" height="15.0" fill="rgb(230,97,25)" rx="2" ry="2" />
<text x="696.33" y="1919.5" ></text>
</g>
<g >
<title>smc_recvmsg (1,201 samples, 0.42%)</title><rect x="829.1" y="133" width="5.0" height="15.0" fill="rgb(232,98,21)" rx="2" ry="2" />
<text x="832.08" y="143.5" ></text>
</g>
<g >
<title>vfs_fstatat (105 samples, 0.04%)</title><rect x="708.4" y="1989" width="0.5" height="15.0" fill="rgb(210,131,8)" rx="2" ry="2" />
<text x="711.44" y="1999.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1301" width="364.1" height="15.0" fill="rgb(229,144,9)" rx="2" ry="2" />
<text x="790.11" y="1311.5" >[unknown]</text>
</g>
<g >
<title>cfree (118 samples, 0.04%)</title><rect x="266.5" y="2021" width="0.5" height="15.0" fill="rgb(238,211,14)" rx="2" ry="2" />
<text x="269.53" y="2031.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1125" width="364.1" height="15.0" fill="rgb(226,69,34)" rx="2" ry="2" />
<text x="790.11" y="1135.5" >[unknown]</text>
</g>
<g >
<title>ksys_read (254 samples, 0.09%)</title><rect x="810.3" y="133" width="1.1" height="15.0" fill="rgb(229,123,28)" rx="2" ry="2" />
<text x="813.35" y="143.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (190 samples, 0.07%)</title><rect x="822.3" y="197" width="0.8" height="15.0" fill="rgb(245,62,37)" rx="2" ry="2" />
<text x="825.30" y="207.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (86 samples, 0.03%)</title><rect x="1148.3" y="165" width="0.3" height="15.0" fill="rgb(216,113,30)" rx="2" ry="2" />
<text x="1151.28" y="175.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="821" width="364.1" height="15.0" fill="rgb(216,85,46)" rx="2" ry="2" />
<text x="790.11" y="831.5" >[unknown]</text>
</g>
<g >
<title>__check_object_size.part.0 (132 samples, 0.05%)</title><rect x="701.0" y="1925" width="0.5" height="15.0" fill="rgb(244,10,26)" rx="2" ry="2" />
<text x="703.95" y="1935.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1605" width="364.1" height="15.0" fill="rgb(215,60,44)" rx="2" ry="2" />
<text x="790.11" y="1615.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1205" width="364.1" height="15.0" fill="rgb(218,10,6)" rx="2" ry="2" />
<text x="790.11" y="1215.5" >[unknown]</text>
</g>
<g >
<title>cpu_startup_entry (14,885 samples, 5.22%)</title><rect x="720.6" y="2021" width="61.6" height="15.0" fill="rgb(250,93,27)" rx="2" ry="2" />
<text x="723.60" y="2031.5" >cpu_st..</text>
</g>
<g >
<title>do_syscall_64 (64 samples, 0.02%)</title><rect x="648.3" y="2005" width="0.3" height="15.0" fill="rgb(208,100,52)" rx="2" ry="2" />
<text x="651.29" y="2015.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (303 samples, 0.11%)</title><rect x="652.6" y="1925" width="1.3" height="15.0" fill="rgb(233,213,54)" rx="2" ry="2" />
<text x="655.62" y="1935.5" ></text>
</g>
<g >
<title>alloc_fd (177 samples, 0.06%)</title><rect x="685.3" y="1957" width="0.8" height="15.0" fill="rgb(243,135,41)" rx="2" ry="2" />
<text x="688.34" y="1967.5" ></text>
</g>
<g >
<title>selinux_file_permission (216 samples, 0.08%)</title><rect x="645.5" y="1909" width="0.9" height="15.0" fill="rgb(211,60,14)" rx="2" ry="2" />
<text x="648.55" y="1919.5" ></text>
</g>
<g >
<title>ksys_read (131 samples, 0.05%)</title><rect x="823.1" y="181" width="0.5" height="15.0" fill="rgb(212,45,35)" rx="2" ry="2" />
<text x="826.08" y="191.5" ></text>
</g>
<g >
<title>tcp_v4_connect (33 samples, 0.01%)</title><rect x="1165.3" y="1909" width="0.1" height="15.0" fill="rgb(247,37,5)" rx="2" ry="2" />
<text x="1168.26" y="1919.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (122 samples, 0.04%)</title><rect x="828.6" y="101" width="0.5" height="15.0" fill="rgb(247,62,44)" rx="2" ry="2" />
<text x="831.58" y="111.5" ></text>
</g>
<g >
<title>smc_sendmsg (86 samples, 0.03%)</title><rect x="1148.3" y="197" width="0.3" height="15.0" fill="rgb(235,68,36)" rx="2" ry="2" />
<text x="1151.28" y="207.5" ></text>
</g>
<g >
<title>ngx_reusable_connection (42 samples, 0.01%)</title><rect x="49.0" y="1957" width="0.2" height="15.0" fill="rgb(215,102,33)" rx="2" ry="2" />
<text x="51.99" y="1967.5" ></text>
</g>
<g >
<title>smc_rx_wake_up (3,785 samples, 1.33%)</title><rect x="60.8" y="1701" width="15.7" height="15.0" fill="rgb(211,186,25)" rx="2" ry="2" />
<text x="63.83" y="1711.5" ></text>
</g>
<g >
<title>response_complete (114 samples, 0.04%)</title><rect x="791.2" y="37" width="0.4" height="15.0" fill="rgb(219,15,41)" rx="2" ry="2" />
<text x="794.17" y="47.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (308 samples, 0.11%)</title><rect x="837.6" y="117" width="1.3" height="15.0" fill="rgb(251,142,31)" rx="2" ry="2" />
<text x="840.61" y="127.5" ></text>
</g>
<g >
<title>ksys_mmap_pgoff (41 samples, 0.01%)</title><rect x="717.7" y="2005" width="0.2" height="15.0" fill="rgb(242,59,37)" rx="2" ry="2" />
<text x="720.74" y="2015.5" ></text>
</g>
<g >
<title>__x64_sys_sendfile64 (48,637 samples, 17.05%)</title><rect x="274.5" y="1973" width="201.2" height="15.0" fill="rgb(215,91,47)" rx="2" ry="2" />
<text x="277.52" y="1983.5" >__x64_sys_sendfile64</text>
</g>
<g >
<title>_raw_spin_lock (1,116 samples, 0.39%)</title><rect x="829.1" y="53" width="4.6" height="15.0" fill="rgb(224,193,32)" rx="2" ry="2" />
<text x="832.08" y="63.5" ></text>
</g>
<g >
<title>read (211 samples, 0.07%)</title><rect x="804.7" y="149" width="0.9" height="15.0" fill="rgb(220,110,2)" rx="2" ry="2" />
<text x="807.75" y="159.5" ></text>
</g>
<g >
<title>ktime_get_ts64 (248 samples, 0.09%)</title><rect x="1186.6" y="1989" width="1.1" height="15.0" fill="rgb(226,220,2)" rx="2" ry="2" />
<text x="1189.65" y="1999.5" ></text>
</g>
<g >
<title>selinux_socket_setsockopt (309 samples, 0.11%)</title><rect x="251.7" y="1893" width="1.3" height="15.0" fill="rgb(232,42,39)" rx="2" ry="2" />
<text x="254.75" y="1903.5" ></text>
</g>
<g >
<title>ngx_destroy_pool (130 samples, 0.05%)</title><rect x="267.7" y="2021" width="0.5" height="15.0" fill="rgb(247,79,49)" rx="2" ry="2" />
<text x="270.69" y="2031.5" ></text>
</g>
<g >
<title>sock_write_iter (61 samples, 0.02%)</title><rect x="801.7" y="37" width="0.3" height="15.0" fill="rgb(251,37,44)" rx="2" ry="2" />
<text x="804.71" y="47.5" ></text>
</g>
<g >
<title>copy_iovec_from_user (59 samples, 0.02%)</title><rect x="646.5" y="1909" width="0.2" height="15.0" fill="rgb(218,200,42)" rx="2" ry="2" />
<text x="649.46" y="1919.5" ></text>
</g>
<g >
<title>do_mmap (39 samples, 0.01%)</title><rect x="717.7" y="1973" width="0.2" height="15.0" fill="rgb(216,220,1)" rx="2" ry="2" />
<text x="720.74" y="1983.5" ></text>
</g>
<g >
<title>ksys_write (1,319 samples, 0.46%)</title><rect x="823.6" y="181" width="5.5" height="15.0" fill="rgb(231,0,5)" rx="2" ry="2" />
<text x="826.63" y="191.5" ></text>
</g>
<g >
<title>smc_sendmsg (189 samples, 0.07%)</title><rect x="822.3" y="117" width="0.8" height="15.0" fill="rgb(212,38,20)" rx="2" ry="2" />
<text x="825.30" y="127.5" ></text>
</g>
<g >
<title>ngx_http_process_request (88 samples, 0.03%)</title><rect x="675.9" y="2037" width="0.3" height="15.0" fill="rgb(227,124,1)" rx="2" ry="2" />
<text x="678.86" y="2047.5" ></text>
</g>
<g >
<title>__x64_sys_epoll_ctl (773 samples, 0.27%)</title><rect x="1167.2" y="2005" width="3.2" height="15.0" fill="rgb(227,193,18)" rx="2" ry="2" />
<text x="1170.17" y="2015.5" ></text>
</g>
<g >
<title>lru_add_drain_cpu (31 samples, 0.01%)</title><rect x="470.6" y="1877" width="0.1" height="15.0" fill="rgb(238,155,33)" rx="2" ry="2" />
<text x="473.60" y="1887.5" ></text>
</g>
<g >
<title>[unknown] (87,988 samples, 30.84%)</title><rect x="787.1" y="421" width="363.9" height="15.0" fill="rgb(231,15,47)" rx="2" ry="2" />
<text x="790.11" y="431.5" >[unknown]</text>
</g>
<g >
<title>smc_recvmsg (36,893 samples, 12.93%)</title><rect x="992.2" y="165" width="152.5" height="15.0" fill="rgb(209,15,2)" rx="2" ry="2" />
<text x="995.15" y="175.5" >smc_recvmsg</text>
</g>
<g >
<title>entry_SYSCALL_64 (614 samples, 0.22%)</title><rect x="819.8" y="197" width="2.5" height="15.0" fill="rgb(225,119,31)" rx="2" ry="2" />
<text x="822.76" y="207.5" ></text>
</g>
<g >
<title>try_to_unlazy (119 samples, 0.04%)</title><rect x="689.1" y="1893" width="0.5" height="15.0" fill="rgb(205,34,37)" rx="2" ry="2" />
<text x="692.14" y="1903.5" ></text>
</g>
<g >
<title>smc_sendmsg (45,912 samples, 16.09%)</title><rect x="279.9" y="1861" width="189.9" height="15.0" fill="rgb(231,11,48)" rx="2" ry="2" />
<text x="282.87" y="1871.5" >smc_sendmsg</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="805" width="364.1" height="15.0" fill="rgb(214,186,34)" rx="2" ry="2" />
<text x="790.11" y="815.5" >[unknown]</text>
</g>
<g >
<title>read (1,201 samples, 0.42%)</title><rect x="829.1" y="245" width="5.0" height="15.0" fill="rgb(232,153,1)" rx="2" ry="2" />
<text x="832.08" y="255.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (473 samples, 0.17%)</title><rect x="1148.7" y="181" width="1.9" height="15.0" fill="rgb(216,12,1)" rx="2" ry="2" />
<text x="1151.67" y="191.5" ></text>
</g>
<g >
<title>splice_direct_to_actor (47,719 samples, 16.73%)</title><rect x="276.9" y="1925" width="197.4" height="15.0" fill="rgb(210,19,22)" rx="2" ry="2" />
<text x="279.95" y="1935.5" >splice_direct_to_actor</text>
</g>
<g >
<title>syscall_exit_to_user_mode (115 samples, 0.04%)</title><rect x="647.7" y="1973" width="0.5" height="15.0" fill="rgb(240,65,24)" rx="2" ry="2" />
<text x="650.72" y="1983.5" ></text>
</g>
<g >
<title>[unknown] (88,001 samples, 30.85%)</title><rect x="787.1" y="469" width="364.0" height="15.0" fill="rgb(232,81,28)" rx="2" ry="2" />
<text x="790.11" y="479.5" >[unknown]</text>
</g>
<g >
<title>irq_exit_rcu (30 samples, 0.01%)</title><rect x="642.0" y="1717" width="0.1" height="15.0" fill="rgb(236,172,9)" rx="2" ry="2" />
<text x="645.03" y="1727.5" ></text>
</g>
<g >
<title>read (614 samples, 0.22%)</title><rect x="819.8" y="213" width="2.5" height="15.0" fill="rgb(222,182,9)" rx="2" ry="2" />
<text x="822.76" y="223.5" ></text>
</g>
<g >
<title>check_heap_object (38,844 samples, 13.62%)</title><rect x="483.4" y="1845" width="160.7" height="15.0" fill="rgb(207,85,13)" rx="2" ry="2" />
<text x="486.39" y="1855.5" >check_heap_object</text>
</g>
<g >
<title>inode_security (78 samples, 0.03%)</title><rect x="805.7" y="37" width="0.3" height="15.0" fill="rgb(219,154,43)" rx="2" ry="2" />
<text x="808.70" y="47.5" ></text>
</g>
<g >
<title>ngx_http_set_disable_symlinks (32 samples, 0.01%)</title><rect x="677.0" y="2037" width="0.1" height="15.0" fill="rgb(252,54,19)" rx="2" ry="2" />
<text x="679.99" y="2047.5" ></text>
</g>
<g >
<title>set_root (226 samples, 0.08%)</title><rect x="664.2" y="1877" width="1.0" height="15.0" fill="rgb(213,93,29)" rx="2" ry="2" />
<text x="667.22" y="1887.5" ></text>
</g>
<g >
<title>memcpy_orig (92 samples, 0.03%)</title><rect x="59.0" y="1749" width="0.4" height="15.0" fill="rgb(251,95,16)" rx="2" ry="2" />
<text x="62.02" y="1759.5" ></text>
</g>
<g >
<title>fsnotify_perm.part.0 (69 samples, 0.02%)</title><rect x="474.3" y="1941" width="0.3" height="15.0" fill="rgb(234,46,46)" rx="2" ry="2" />
<text x="477.32" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (4,954 samples, 1.74%)</title><rect x="787.1" y="165" width="20.5" height="15.0" fill="rgb(210,97,39)" rx="2" ry="2" />
<text x="790.11" y="175.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (118 samples, 0.04%)</title><rect x="828.6" y="53" width="0.5" height="15.0" fill="rgb(244,197,48)" rx="2" ry="2" />
<text x="831.59" y="63.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (602 samples, 0.21%)</title><rect x="819.8" y="85" width="2.5" height="15.0" fill="rgb(244,10,38)" rx="2" ry="2" />
<text x="822.81" y="95.5" ></text>
</g>
<g >
<title>sock_close (31 samples, 0.01%)</title><rect x="703.7" y="1973" width="0.1" height="15.0" fill="rgb(205,124,39)" rx="2" ry="2" />
<text x="706.71" y="1983.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (872 samples, 0.31%)</title><rect x="812.9" y="69" width="3.7" height="15.0" fill="rgb(233,87,29)" rx="2" ry="2" />
<text x="815.95" y="79.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (134 samples, 0.05%)</title><rect x="1146.2" y="149" width="0.5" height="15.0" fill="rgb(212,97,33)" rx="2" ry="2" />
<text x="1149.15" y="159.5" ></text>
</g>
<g >
<title>security_socket_recvmsg (29 samples, 0.01%)</title><rect x="805.1" y="37" width="0.1" height="15.0" fill="rgb(253,28,48)" rx="2" ry="2" />
<text x="808.08" y="47.5" ></text>
</g>
<g >
<title>tcp_v4_do_rcv (25 samples, 0.01%)</title><rect x="1165.4" y="1893" width="0.1" height="15.0" fill="rgb(253,122,39)" rx="2" ry="2" />
<text x="1168.40" y="1903.5" ></text>
</g>
<g >
<title>check_heap_object (37,616 samples, 13.18%)</title><rect x="86.3" y="1829" width="155.6" height="15.0" fill="rgb(211,166,49)" rx="2" ry="2" />
<text x="89.28" y="1839.5" >check_heap_object</text>
</g>
<g >
<title>__x64_sys_epoll_ctl (561 samples, 0.20%)</title><rect x="1153.7" y="1989" width="2.3" height="15.0" fill="rgb(211,172,31)" rx="2" ry="2" />
<text x="1156.69" y="1999.5" ></text>
</g>
<g >
<title>ngx_http_output_filter (55 samples, 0.02%)</title><rect x="674.0" y="2037" width="0.3" height="15.0" fill="rgb(247,198,50)" rx="2" ry="2" />
<text x="677.03" y="2047.5" ></text>
</g>
<g >
<title>smc_recvmsg (872 samples, 0.31%)</title><rect x="812.9" y="85" width="3.7" height="15.0" fill="rgb(228,96,54)" rx="2" ry="2" />
<text x="815.95" y="95.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="757" width="364.1" height="15.0" fill="rgb(248,170,15)" rx="2" ry="2" />
<text x="790.11" y="767.5" >[unknown]</text>
</g>
<g >
<title>smc_close_wake_tx_prepared (37 samples, 0.01%)</title><rect x="819.1" y="37" width="0.2" height="15.0" fill="rgb(246,19,5)" rx="2" ry="2" />
<text x="822.15" y="47.5" ></text>
</g>
<g >
<title>ngx_http_process_request_uri (29 samples, 0.01%)</title><rect x="676.3" y="2037" width="0.1" height="15.0" fill="rgb(223,152,21)" rx="2" ry="2" />
<text x="679.31" y="2047.5" ></text>
</g>
<g >
<title>avc_lookup (33 samples, 0.01%)</title><rect x="481.5" y="1829" width="0.1" height="15.0" fill="rgb(218,191,11)" rx="2" ry="2" />
<text x="484.49" y="1839.5" ></text>
</g>
<g >
<title>filemap_splice_read (847 samples, 0.30%)</title><rect x="470.3" y="1909" width="3.5" height="15.0" fill="rgb(238,218,47)" rx="2" ry="2" />
<text x="473.26" y="1919.5" ></text>
</g>
<g >
<title>ima_file_check (98 samples, 0.03%)</title><rect x="692.9" y="1909" width="0.4" height="15.0" fill="rgb(209,130,48)" rx="2" ry="2" />
<text x="695.92" y="1919.5" ></text>
</g>
<g >
<title>write (254 samples, 0.09%)</title><rect x="798.2" y="85" width="1.1" height="15.0" fill="rgb(239,11,39)" rx="2" ry="2" />
<text x="801.22" y="95.5" ></text>
</g>
<g >
<title>lookup_fast (316 samples, 0.11%)</title><rect x="662.6" y="1877" width="1.3" height="15.0" fill="rgb(211,115,7)" rx="2" ry="2" />
<text x="665.59" y="1887.5" ></text>
</g>
<g >
<title>sock_write_iter (772 samples, 0.27%)</title><rect x="816.6" y="117" width="3.1" height="15.0" fill="rgb(244,0,35)" rx="2" ry="2" />
<text x="819.55" y="127.5" ></text>
</g>
<g >
<title>[libc.so.6] (260 samples, 0.09%)</title><rect x="786.0" y="2053" width="1.1" height="15.0" fill="rgb(215,98,3)" rx="2" ry="2" />
<text x="789.03" y="2063.5" ></text>
</g>
<g >
<title>check_heap_object (134 samples, 0.05%)</title><rect x="822.3" y="69" width="0.6" height="15.0" fill="rgb(244,160,31)" rx="2" ry="2" />
<text x="825.30" y="79.5" ></text>
</g>
<g >
<title>kvm_kick_cpu (427 samples, 0.15%)</title><rect x="463.1" y="1717" width="1.8" height="15.0" fill="rgb(240,9,31)" rx="2" ry="2" />
<text x="466.11" y="1727.5" ></text>
</g>
<g >
<title>complete_walk (128 samples, 0.04%)</title><rect x="689.1" y="1909" width="0.5" height="15.0" fill="rgb(215,53,50)" rx="2" ry="2" />
<text x="692.11" y="1919.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="901" width="364.1" height="15.0" fill="rgb(207,179,31)" rx="2" ry="2" />
<text x="790.11" y="911.5" >[unknown]</text>
</g>
<g >
<title>ksys_write (301 samples, 0.11%)</title><rect x="803.5" y="85" width="1.2" height="15.0" fill="rgb(225,147,20)" rx="2" ry="2" />
<text x="806.50" y="95.5" ></text>
</g>
<g >
<title>read (36,893 samples, 12.93%)</title><rect x="992.2" y="277" width="152.5" height="15.0" fill="rgb(246,163,42)" rx="2" ry="2" />
<text x="995.15" y="287.5" >read</text>
</g>
<g >
<title>entry_SYSCALL_64 (143 samples, 0.05%)</title><rect x="801.4" y="101" width="0.6" height="15.0" fill="rgb(243,98,15)" rx="2" ry="2" />
<text x="804.38" y="111.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1893" width="364.1" height="15.0" fill="rgb(252,153,8)" rx="2" ry="2" />
<text x="790.11" y="1903.5" >[unknown]</text>
</g>
<g >
<title>__pv_queued_spin_unlock_slowpath (381 samples, 0.13%)</title><rect x="1146.7" y="53" width="1.6" height="15.0" fill="rgb(232,213,38)" rx="2" ry="2" />
<text x="1149.71" y="63.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (47 samples, 0.02%)</title><rect x="719.1" y="2037" width="0.2" height="15.0" fill="rgb(233,114,52)" rx="2" ry="2" />
<text x="722.15" y="2047.5" ></text>
</g>
<g >
<title>nd_jump_root (249 samples, 0.09%)</title><rect x="664.1" y="1893" width="1.1" height="15.0" fill="rgb(208,139,26)" rx="2" ry="2" />
<text x="667.12" y="1903.5" ></text>
</g>
<g >
<title>do_syscall_64 (475 samples, 0.17%)</title><rect x="1144.7" y="245" width="2.0" height="15.0" fill="rgb(241,91,17)" rx="2" ry="2" />
<text x="1147.74" y="255.5" ></text>
</g>
<g >
<title>vfs_write (1,319 samples, 0.46%)</title><rect x="823.6" y="165" width="5.5" height="15.0" fill="rgb(226,69,52)" rx="2" ry="2" />
<text x="826.63" y="175.5" ></text>
</g>
<g >
<title>kvm_wait.part.0 (36,139 samples, 12.67%)</title><rect x="842.6" y="37" width="149.4" height="15.0" fill="rgb(211,61,0)" rx="2" ry="2" />
<text x="845.58" y="47.5" >kvm_wait.part.0</text>
</g>
<g >
<title>lock_sock_nested (77 samples, 0.03%)</title><rect x="810.7" y="53" width="0.3" height="15.0" fill="rgb(253,222,4)" rx="2" ry="2" />
<text x="813.65" y="63.5" ></text>
</g>
<g >
<title>ngx_http_update_location_config (176 samples, 0.06%)</title><rect x="677.9" y="2037" width="0.8" height="15.0" fill="rgb(249,145,42)" rx="2" ry="2" />
<text x="680.93" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (360 samples, 0.13%)</title><rect x="245.6" y="1973" width="1.5" height="15.0" fill="rgb(209,152,1)" rx="2" ry="2" />
<text x="248.62" y="1983.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1381" width="364.1" height="15.0" fill="rgb(213,43,46)" rx="2" ry="2" />
<text x="790.11" y="1391.5" >[unknown]</text>
</g>
<g >
<title>security_file_open (85 samples, 0.03%)</title><rect x="692.6" y="1893" width="0.3" height="15.0" fill="rgb(246,75,41)" rx="2" ry="2" />
<text x="695.56" y="1903.5" ></text>
</g>
<g >
<title>do_filp_open (3,221 samples, 1.13%)</title><rect x="686.3" y="1957" width="13.3" height="15.0" fill="rgb(230,219,39)" rx="2" ry="2" />
<text x="689.26" y="1967.5" ></text>
</g>
<g >
<title>__pv_queued_spin_unlock_slowpath (341 samples, 0.12%)</title><rect x="1144.7" y="53" width="1.5" height="15.0" fill="rgb(235,139,32)" rx="2" ry="2" />
<text x="1147.74" y="63.5" ></text>
</g>
<g >
<title>__get_file_rcu (77 samples, 0.03%)</title><rect x="803.5" y="37" width="0.3" height="15.0" fill="rgb(232,196,8)" rx="2" ry="2" />
<text x="806.50" y="47.5" ></text>
</g>
<g >
<title>cpu_startup_entry (892 samples, 0.31%)</title><rect x="782.2" y="1957" width="3.7" height="15.0" fill="rgb(239,138,37)" rx="2" ry="2" />
<text x="785.17" y="1967.5" ></text>
</g>
<g >
<title>_copy_to_iter (186 samples, 0.07%)</title><rect x="241.9" y="1845" width="0.7" height="15.0" fill="rgb(250,137,48)" rx="2" ry="2" />
<text x="244.87" y="1855.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (1,681 samples, 0.59%)</title><rect x="251.4" y="1989" width="6.9" height="15.0" fill="rgb(221,16,47)" rx="2" ry="2" />
<text x="254.36" y="1999.5" ></text>
</g>
<g >
<title>schedule (31 samples, 0.01%)</title><rect x="810.2" y="69" width="0.1" height="15.0" fill="rgb(216,22,39)" rx="2" ry="2" />
<text x="813.21" y="79.5" ></text>
</g>
<g >
<title>aeDeleteFileEvent (170 samples, 0.06%)</title><rect x="1151.2" y="2037" width="0.7" height="15.0" fill="rgb(234,2,51)" rx="2" ry="2" />
<text x="1154.21" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,681 samples, 0.59%)</title><rect x="251.4" y="1973" width="6.9" height="15.0" fill="rgb(211,177,28)" rx="2" ry="2" />
<text x="254.36" y="1983.5" ></text>
</g>
<g >
<title>wrk (97,675 samples, 34.24%)</title><rect x="786.0" y="2069" width="404.0" height="15.0" fill="rgb(215,221,5)" rx="2" ry="2" />
<text x="789.02" y="2079.5" >wrk</text>
</g>
<g >
<title>smcd_cdc_msg_send (121 samples, 0.04%)</title><rect x="828.6" y="69" width="0.5" height="15.0" fill="rgb(247,123,29)" rx="2" ry="2" />
<text x="831.58" y="79.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1749" width="364.1" height="15.0" fill="rgb(228,205,19)" rx="2" ry="2" />
<text x="790.11" y="1759.5" >[unknown]</text>
</g>
<g >
<title>syscall_enter_from_user_mode (238 samples, 0.08%)</title><rect x="1156.0" y="1989" width="1.0" height="15.0" fill="rgb(205,81,1)" rx="2" ry="2" />
<text x="1159.01" y="1999.5" ></text>
</g>
<g >
<title>avc_lookup (66 samples, 0.02%)</title><rect x="645.3" y="1861" width="0.2" height="15.0" fill="rgb(253,92,48)" rx="2" ry="2" />
<text x="648.25" y="1871.5" ></text>
</g>
<g >
<title>__close (40 samples, 0.01%)</title><rect x="703.7" y="2053" width="0.1" height="15.0" fill="rgb(243,3,23)" rx="2" ry="2" />
<text x="706.67" y="2063.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (29 samples, 0.01%)</title><rect x="715.5" y="1925" width="0.2" height="15.0" fill="rgb(223,43,21)" rx="2" ry="2" />
<text x="718.53" y="1935.5" ></text>
</g>
<g >
<title>vfs_read (131 samples, 0.05%)</title><rect x="823.1" y="165" width="0.5" height="15.0" fill="rgb(223,81,20)" rx="2" ry="2" />
<text x="826.08" y="175.5" ></text>
</g>
<g >
<title>tcp_connect (29 samples, 0.01%)</title><rect x="1165.3" y="1893" width="0.1" height="15.0" fill="rgb(208,112,54)" rx="2" ry="2" />
<text x="1168.28" y="1903.5" ></text>
</g>
<g >
<title>inet_stream_connect (58 samples, 0.02%)</title><rect x="1165.3" y="1941" width="0.2" height="15.0" fill="rgb(249,48,22)" rx="2" ry="2" />
<text x="1168.26" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (88,030 samples, 30.86%)</title><rect x="787.1" y="629" width="364.1" height="15.0" fill="rgb(217,136,10)" rx="2" ry="2" />
<text x="790.11" y="639.5" >[unknown]</text>
</g>
<g >
<title>kvm_clock_get_cycles (36 samples, 0.01%)</title><rect x="1186.2" y="1925" width="0.1" height="15.0" fill="rgb(249,10,2)" rx="2" ry="2" />
<text x="1189.18" y="1935.5" ></text>
</g>
<g >
<title>ngx_http_core_generic_phase (49 samples, 0.02%)</title><rect x="671.8" y="2037" width="0.2" height="15.0" fill="rgb(212,227,16)" rx="2" ry="2" />
<text x="674.83" y="2047.5" ></text>
</g>
<g >
<title>[unknown] (88,032 samples, 30.86%)</title><rect x="787.1" y="693" width="364.1" height="15.0" fill="rgb(247,12,28)" rx="2" ry="2" />
<text x="790.11" y="703.5" >[unknown]</text>
</g>
<g >
<title>security_inode_permission (77 samples, 0.03%)</title><rect x="693.7" y="1893" width="0.3" height="15.0" fill="rgb(242,225,22)" rx="2" ry="2" />
<text x="696.72" y="1903.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (228 samples, 0.08%)</title><rect x="808.6" y="37" width="0.9" height="15.0" fill="rgb(221,82,5)" rx="2" ry="2" />
<text x="811.56" y="47.5" ></text>
</g>
<g >
<title>bprm_execve.part.0 (29 samples, 0.01%)</title><rect x="717.3" y="1973" width="0.1" height="15.0" fill="rgb(252,17,7)" rx="2" ry="2" />
<text x="720.30" y="1983.5" ></text>
</g>
<g >
<title>ksys_write (475 samples, 0.17%)</title><rect x="1144.7" y="229" width="2.0" height="15.0" fill="rgb(211,215,13)" rx="2" ry="2" />
<text x="1147.74" y="239.5" ></text>
</g>
<g >
<title>sysvec_apic_timer_interrupt (29 samples, 0.01%)</title><rect x="240.0" y="1717" width="0.1" height="15.0" fill="rgb(250,111,4)" rx="2" ry="2" />
<text x="242.96" y="1727.5" ></text>
</g>
<g >
<title>[unknown] (7,891 samples, 2.77%)</title><rect x="787.1" y="213" width="32.6" height="15.0" fill="rgb(221,141,0)" rx="2" ry="2" />
<text x="790.11" y="223.5" >[u..</text>
</g>
<g >
<title>_copy_from_user (45 samples, 0.02%)</title><rect x="274.7" y="1957" width="0.2" height="15.0" fill="rgb(212,228,43)" rx="2" ry="2" />
<text x="277.72" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1877" width="364.1" height="15.0" fill="rgb(209,127,17)" rx="2" ry="2" />
<text x="790.11" y="1887.5" >[unknown]</text>
</g>
<g >
<title>sysvec_apic_timer_interrupt (30 samples, 0.01%)</title><rect x="642.0" y="1733" width="0.1" height="15.0" fill="rgb(231,90,27)" rx="2" ry="2" />
<text x="645.03" y="1743.5" ></text>
</g>
<g >
<title>do_syscall_64 (293 samples, 0.10%)</title><rect x="800.2" y="85" width="1.2" height="15.0" fill="rgb(247,104,41)" rx="2" ry="2" />
<text x="803.15" y="95.5" ></text>
</g>
<g >
<title>vfs_write (137 samples, 0.05%)</title><rect x="809.6" y="101" width="0.6" height="15.0" fill="rgb(205,50,44)" rx="2" ry="2" />
<text x="812.65" y="111.5" ></text>
</g>
<g >
<title>hrtimer_init_sleeper (32 samples, 0.01%)</title><rect x="1180.8" y="1941" width="0.2" height="15.0" fill="rgb(220,20,35)" rx="2" ry="2" />
<text x="1183.83" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (196 samples, 0.07%)</title><rect x="718.0" y="2053" width="0.8" height="15.0" fill="rgb(234,53,12)" rx="2" ry="2" />
<text x="721.01" y="2063.5" ></text>
</g>
<g >
<title>vfs_write (1,169 samples, 0.41%)</title><rect x="834.1" y="181" width="4.8" height="15.0" fill="rgb(227,175,19)" rx="2" ry="2" />
<text x="837.05" y="191.5" ></text>
</g>
<g >
<title>sock_read_iter (129 samples, 0.05%)</title><rect x="805.1" y="69" width="0.5" height="15.0" fill="rgb(211,49,50)" rx="2" ry="2" />
<text x="808.08" y="79.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock_slowpath (431 samples, 0.15%)</title><rect x="463.1" y="1749" width="1.8" height="15.0" fill="rgb(222,219,22)" rx="2" ry="2" />
<text x="466.09" y="1759.5" ></text>
</g>
<g >
<title>ksys_write (36,168 samples, 12.68%)</title><rect x="842.6" y="213" width="149.6" height="15.0" fill="rgb(211,83,50)" rx="2" ry="2" />
<text x="845.56" y="223.5" >ksys_write</text>
</g>
<g >
<title>__fstat64 (53 samples, 0.02%)</title><rect x="703.8" y="2053" width="0.3" height="15.0" fill="rgb(206,79,40)" rx="2" ry="2" />
<text x="706.84" y="2063.5" ></text>
</g>
<g >
<title>path_init (82 samples, 0.03%)</title><rect x="698.8" y="1925" width="0.4" height="15.0" fill="rgb(219,98,28)" rx="2" ry="2" />
<text x="701.83" y="1935.5" ></text>
</g>
<g >
<title>[unknown] (49,575 samples, 17.38%)</title><rect x="787.1" y="277" width="205.1" height="15.0" fill="rgb(224,147,18)" rx="2" ry="2" />
<text x="790.11" y="287.5" >[unknown]</text>
</g>
<g >
<title>__x64_sys_setsockopt (6,219 samples, 2.18%)</title><rect x="51.9" y="1909" width="25.7" height="15.0" fill="rgb(253,229,1)" rx="2" ry="2" />
<text x="54.86" y="1919.5" >_..</text>
</g>
<g >
<title>__check_object_size.part.0 (44,458 samples, 15.58%)</title><rect x="281.3" y="1829" width="183.9" height="15.0" fill="rgb(209,2,26)" rx="2" ry="2" />
<text x="284.34" y="1839.5" >__check_object_size.par..</text>
</g>
<g >
<title>ngx_unix_recv (304 samples, 0.11%)</title><rect x="79.6" y="1973" width="1.3" height="15.0" fill="rgb(232,179,38)" rx="2" ry="2" />
<text x="82.61" y="1983.5" ></text>
</g>
<g >
<title>ep_item_poll.isra.0 (152 samples, 0.05%)</title><rect x="1155.2" y="1941" width="0.6" height="15.0" fill="rgb(225,128,5)" rx="2" ry="2" />
<text x="1158.17" y="1951.5" ></text>
</g>
<g >
<title>_copy_from_user (62 samples, 0.02%)</title><rect x="1167.2" y="1989" width="0.3" height="15.0" fill="rgb(226,55,42)" rx="2" ry="2" />
<text x="1170.22" y="1999.5" ></text>
</g>
<g >
<title>release_sock (25 samples, 0.01%)</title><rect x="809.8" y="53" width="0.1" height="15.0" fill="rgb(244,123,52)" rx="2" ry="2" />
<text x="812.82" y="63.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (31 samples, 0.01%)</title><rect x="810.2" y="101" width="0.1" height="15.0" fill="rgb(245,81,26)" rx="2" ry="2" />
<text x="813.21" y="111.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="773" width="364.1" height="15.0" fill="rgb(209,102,18)" rx="2" ry="2" />
<text x="790.11" y="783.5" >[unknown]</text>
</g>
<g >
<title>syscall_exit_to_user_mode (119 samples, 0.04%)</title><rect x="244.9" y="1925" width="0.5" height="15.0" fill="rgb(252,193,23)" rx="2" ry="2" />
<text x="247.92" y="1935.5" ></text>
</g>
<g >
<title>__fsnotify_parent (28 samples, 0.01%)</title><rect x="260.8" y="1941" width="0.1" height="15.0" fill="rgb(249,37,8)" rx="2" ry="2" />
<text x="263.83" y="1951.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (86 samples, 0.03%)</title><rect x="1148.3" y="133" width="0.3" height="15.0" fill="rgb(227,20,32)" rx="2" ry="2" />
<text x="1151.28" y="143.5" ></text>
</g>
<g >
<title>do_syscall_64 (131 samples, 0.05%)</title><rect x="823.1" y="197" width="0.5" height="15.0" fill="rgb(238,213,4)" rx="2" ry="2" />
<text x="826.08" y="207.5" ></text>
</g>
<g >
<title>_raw_spin_lock (36,448 samples, 12.78%)</title><rect x="89.3" y="1797" width="150.8" height="15.0" fill="rgb(210,172,8)" rx="2" ry="2" />
<text x="92.33" y="1807.5" >_raw_spin_lock</text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock_slowpath (333 samples, 0.12%)</title><rect x="240.1" y="1765" width="1.4" height="15.0" fill="rgb(225,206,32)" rx="2" ry="2" />
<text x="243.10" y="1775.5" ></text>
</g>
<g >
<title>write (36,168 samples, 12.68%)</title><rect x="842.6" y="261" width="149.6" height="15.0" fill="rgb(242,89,22)" rx="2" ry="2" />
<text x="845.56" y="271.5" >write</text>
</g>
<g >
<title>__d_lookup_rcu (303 samples, 0.11%)</title><rect x="662.6" y="1861" width="1.3" height="15.0" fill="rgb(235,188,26)" rx="2" ry="2" />
<text x="665.64" y="1871.5" ></text>
</g>
<g >
<title>ksys_write (86 samples, 0.03%)</title><rect x="1148.3" y="245" width="0.3" height="15.0" fill="rgb(207,165,41)" rx="2" ry="2" />
<text x="1151.28" y="255.5" ></text>
</g>
<g >
<title>__get_file_rcu (98 samples, 0.03%)</title><rect x="1154.7" y="1941" width="0.4" height="15.0" fill="rgb(223,76,51)" rx="2" ry="2" />
<text x="1157.67" y="1951.5" ></text>
</g>
<g >
<title>do_epoll_wait (309 samples, 0.11%)</title><rect x="245.6" y="1941" width="1.3" height="15.0" fill="rgb(225,224,5)" rx="2" ry="2" />
<text x="248.64" y="1951.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (610 samples, 0.21%)</title><rect x="816.6" y="69" width="2.5" height="15.0" fill="rgb(236,2,52)" rx="2" ry="2" />
<text x="819.56" y="79.5" ></text>
</g>
<g >
<title>rep_movs_alternative (193 samples, 0.07%)</title><rect x="811.6" y="37" width="0.8" height="15.0" fill="rgb(206,155,7)" rx="2" ry="2" />
<text x="814.60" y="47.5" ></text>
</g>
<g >
<title>do_syscall_64 (493 samples, 0.17%)</title><rect x="807.6" y="133" width="2.0" height="15.0" fill="rgb(208,170,37)" rx="2" ry="2" />
<text x="810.61" y="143.5" ></text>
</g>
<g >
<title>generic_permission (93 samples, 0.03%)</title><rect x="694.8" y="1893" width="0.4" height="15.0" fill="rgb(238,213,28)" rx="2" ry="2" />
<text x="697.84" y="1903.5" ></text>
</g>
<g >
<title>selinux_file_permission (113 samples, 0.04%)</title><rect x="803.8" y="37" width="0.5" height="15.0" fill="rgb(216,17,20)" rx="2" ry="2" />
<text x="806.85" y="47.5" ></text>
</g>
<g >
<title>cpuidle_idle_call (876 samples, 0.31%)</title><rect x="782.2" y="1925" width="3.6" height="15.0" fill="rgb(233,160,40)" rx="2" ry="2" />
<text x="785.17" y="1935.5" ></text>
</g>
<g >
<title>cfree (57 samples, 0.02%)</title><rect x="704.1" y="2053" width="0.2" height="15.0" fill="rgb(252,5,39)" rx="2" ry="2" />
<text x="707.09" y="2063.5" ></text>
</g>
<g >
<title>sendfile (527 samples, 0.18%)</title><rect x="272.2" y="2021" width="2.2" height="15.0" fill="rgb(239,214,4)" rx="2" ry="2" />
<text x="275.18" y="2031.5" ></text>
</g>
<g >
<title>do_sock_setsockopt (1,170 samples, 0.41%)</title><rect x="251.4" y="1925" width="4.9" height="15.0" fill="rgb(205,161,3)" rx="2" ry="2" />
<text x="254.44" y="1935.5" ></text>
</g>
<g >
<title>security_file_permission (210 samples, 0.07%)</title><rect x="802.5" y="53" width="0.8" height="15.0" fill="rgb(227,130,13)" rx="2" ry="2" />
<text x="805.47" y="63.5" ></text>
</g>
<g >
<title>__schedule (35 samples, 0.01%)</title><rect x="809.5" y="53" width="0.1" height="15.0" fill="rgb(245,11,38)" rx="2" ry="2" />
<text x="812.50" y="63.5" ></text>
</g>
<g >
<title>sock_recvmsg (40 samples, 0.01%)</title><rect x="803.3" y="37" width="0.2" height="15.0" fill="rgb(227,49,10)" rx="2" ry="2" />
<text x="806.33" y="47.5" ></text>
</g>
<g >
<title>walk_component (326 samples, 0.11%)</title><rect x="662.5" y="1893" width="1.4" height="15.0" fill="rgb(225,19,9)" rx="2" ry="2" />
<text x="665.55" y="1903.5" ></text>
</g>
<g >
<title>import_iovec (74 samples, 0.03%)</title><rect x="646.4" y="1941" width="0.3" height="15.0" fill="rgb(235,123,21)" rx="2" ry="2" />
<text x="649.44" y="1951.5" ></text>
</g>
<g >
<title>security_file_permission (254 samples, 0.09%)</title><rect x="474.6" y="1941" width="1.1" height="15.0" fill="rgb(211,3,40)" rx="2" ry="2" />
<text x="477.63" y="1951.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (4,555 samples, 1.60%)</title><rect x="57.7" y="1781" width="18.8" height="15.0" fill="rgb(234,197,11)" rx="2" ry="2" />
<text x="60.70" y="1791.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1173" width="364.1" height="15.0" fill="rgb(222,2,30)" rx="2" ry="2" />
<text x="790.11" y="1183.5" >[unknown]</text>
</g>
<g >
<title>__schedule (100 samples, 0.04%)</title><rect x="647.7" y="1909" width="0.5" height="15.0" fill="rgb(219,136,8)" rx="2" ry="2" />
<text x="650.74" y="1919.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (35 samples, 0.01%)</title><rect x="811.4" y="53" width="0.1" height="15.0" fill="rgb(213,156,51)" rx="2" ry="2" />
<text x="814.40" y="63.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (144 samples, 0.05%)</title><rect x="719.4" y="2037" width="0.6" height="15.0" fill="rgb(251,101,38)" rx="2" ry="2" />
<text x="722.38" y="2047.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (1,201 samples, 0.42%)</title><rect x="829.1" y="229" width="5.0" height="15.0" fill="rgb(208,54,36)" rx="2" ry="2" />
<text x="832.08" y="239.5" ></text>
</g>
<g >
<title>ngx_http_create_request (74 samples, 0.03%)</title><rect x="79.1" y="1973" width="0.4" height="15.0" fill="rgb(213,79,8)" rx="2" ry="2" />
<text x="82.15" y="1983.5" ></text>
</g>
<g >
<title>kvm_wait.part.0 (41,216 samples, 14.45%)</title><rect x="292.6" y="1733" width="170.5" height="15.0" fill="rgb(241,116,36)" rx="2" ry="2" />
<text x="295.61" y="1743.5" >kvm_wait.part.0</text>
</g>
<g >
<title>secondary_startup_64_no_verify (15,777 samples, 5.53%)</title><rect x="720.6" y="2053" width="65.3" height="15.0" fill="rgb(233,228,1)" rx="2" ry="2" />
<text x="723.60" y="2063.5" >seconda..</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1925" width="364.1" height="15.0" fill="rgb(217,112,40)" rx="2" ry="2" />
<text x="790.11" y="1935.5" >[unknown]</text>
</g>
<g >
<title>kmem_cache_alloc (214 samples, 0.08%)</title><rect x="651.4" y="1941" width="0.9" height="15.0" fill="rgb(244,126,19)" rx="2" ry="2" />
<text x="654.38" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (3,151 samples, 1.10%)</title><rect x="787.1" y="117" width="13.0" height="15.0" fill="rgb(213,127,1)" rx="2" ry="2" />
<text x="790.11" y="127.5" ></text>
</g>
<g >
<title>__d_lookup_rcu (121 samples, 0.04%)</title><rect x="698.3" y="1893" width="0.5" height="15.0" fill="rgb(207,149,36)" rx="2" ry="2" />
<text x="701.31" y="1903.5" ></text>
</g>
<g >
<title>smc_sendmsg (286 samples, 0.10%)</title><rect x="806.4" y="53" width="1.2" height="15.0" fill="rgb(254,125,28)" rx="2" ry="2" />
<text x="809.41" y="63.5" ></text>
</g>
<g >
<title>sock_write_iter (104 samples, 0.04%)</title><rect x="804.3" y="53" width="0.4" height="15.0" fill="rgb(216,28,25)" rx="2" ry="2" />
<text x="807.31" y="63.5" ></text>
</g>
<g >
<title>smc_rx_wake_up (472 samples, 0.17%)</title><rect x="1148.7" y="53" width="1.9" height="15.0" fill="rgb(236,169,26)" rx="2" ry="2" />
<text x="1151.67" y="63.5" ></text>
</g>
<g >
<title>_copy_to_iter (820 samples, 0.29%)</title><rect x="813.1" y="53" width="3.3" height="15.0" fill="rgb(249,104,34)" rx="2" ry="2" />
<text x="816.05" y="63.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (473 samples, 0.17%)</title><rect x="1148.7" y="85" width="1.9" height="15.0" fill="rgb(210,194,19)" rx="2" ry="2" />
<text x="1151.67" y="95.5" ></text>
</g>
<g >
<title>do_user_addr_fault (27 samples, 0.01%)</title><rect x="719.8" y="1973" width="0.1" height="15.0" fill="rgb(211,106,25)" rx="2" ry="2" />
<text x="722.78" y="1983.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (888 samples, 0.31%)</title><rect x="838.9" y="245" width="3.7" height="15.0" fill="rgb(216,158,14)" rx="2" ry="2" />
<text x="841.89" y="255.5" ></text>
</g>
<g >
<title>__fget_light (218 samples, 0.08%)</title><rect x="275.5" y="1941" width="0.9" height="15.0" fill="rgb(207,193,45)" rx="2" ry="2" />
<text x="278.51" y="1951.5" ></text>
</g>
<g >
<title>exc_page_fault (27 samples, 0.01%)</title><rect x="718.3" y="1973" width="0.1" height="15.0" fill="rgb(242,186,50)" rx="2" ry="2" />
<text x="721.26" y="1983.5" ></text>
</g>
<g >
<title>smc_sndbuf_sync_sg_for_device (32 samples, 0.01%)</title><rect x="810.0" y="37" width="0.1" height="15.0" fill="rgb(253,77,40)" rx="2" ry="2" />
<text x="813.01" y="47.5" ></text>
</g>
<g >
<title>ngx_http_internal_redirect (233 samples, 0.08%)</title><rect x="673.0" y="2037" width="0.9" height="15.0" fill="rgb(237,102,14)" rx="2" ry="2" />
<text x="675.97" y="2047.5" ></text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (308 samples, 0.11%)</title><rect x="837.6" y="101" width="1.3" height="15.0" fill="rgb(215,29,35)" rx="2" ry="2" />
<text x="840.61" y="111.5" ></text>
</g>
<g >
<title>sock_read_iter (207 samples, 0.07%)</title><rect x="810.5" y="101" width="0.9" height="15.0" fill="rgb(211,70,45)" rx="2" ry="2" />
<text x="813.54" y="111.5" ></text>
</g>
<g >
<title>__get_file_rcu (105 samples, 0.04%)</title><rect x="802.0" y="37" width="0.4" height="15.0" fill="rgb(225,152,37)" rx="2" ry="2" />
<text x="804.97" y="47.5" ></text>
</g>
<g >
<title>link_path_walk.part.0 (1,916 samples, 0.67%)</title><rect x="656.0" y="1909" width="7.9" height="15.0" fill="rgb(241,86,28)" rx="2" ry="2" />
<text x="658.97" y="1919.5" ></text>
</g>
<g >
<title>xas_load (47 samples, 0.02%)</title><rect x="472.4" y="1861" width="0.2" height="15.0" fill="rgb(225,201,38)" rx="2" ry="2" />
<text x="475.44" y="1871.5" ></text>
</g>
<g >
<title>mmap_region (40 samples, 0.01%)</title><rect x="719.1" y="1957" width="0.2" height="15.0" fill="rgb(219,40,36)" rx="2" ry="2" />
<text x="722.15" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1973" width="364.1" height="15.0" fill="rgb(248,97,8)" rx="2" ry="2" />
<text x="790.11" y="1983.5" >[unknown]</text>
</g>
<g >
<title>current_time (93 samples, 0.03%)</title><rect x="473.3" y="1861" width="0.4" height="15.0" fill="rgb(240,154,50)" rx="2" ry="2" />
<text x="476.29" y="1871.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,034 samples, 0.36%)</title><rect x="1167.2" y="2021" width="4.2" height="15.0" fill="rgb(222,175,30)" rx="2" ry="2" />
<text x="1170.17" y="2031.5" ></text>
</g>
<g >
<title>tcp_rcv_state_process (25 samples, 0.01%)</title><rect x="1165.4" y="1877" width="0.1" height="15.0" fill="rgb(242,88,2)" rx="2" ry="2" />
<text x="1168.40" y="1887.5" ></text>
</g>
<g >
<title>.slowpath (341 samples, 0.12%)</title><rect x="1144.7" y="85" width="1.5" height="15.0" fill="rgb(242,150,0)" rx="2" ry="2" />
<text x="1147.74" y="95.5" ></text>
</g>
<g >
<title>.slowpath (352 samples, 0.12%)</title><rect x="642.2" y="1797" width="1.4" height="15.0" fill="rgb(239,27,35)" rx="2" ry="2" />
<text x="645.16" y="1807.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (473 samples, 0.17%)</title><rect x="1148.7" y="149" width="1.9" height="15.0" fill="rgb(213,170,54)" rx="2" ry="2" />
<text x="1151.67" y="159.5" ></text>
</g>
<g >
<title>release_sock (65 samples, 0.02%)</title><rect x="55.5" y="1845" width="0.3" height="15.0" fill="rgb(252,91,35)" rx="2" ry="2" />
<text x="58.49" y="1855.5" ></text>
</g>
<g >
<title>[unknown] (88,016 samples, 30.85%)</title><rect x="787.1" y="549" width="364.0" height="15.0" fill="rgb(215,41,49)" rx="2" ry="2" />
<text x="790.11" y="559.5" >[unknown]</text>
</g>
<g >
<title>schedule (39 samples, 0.01%)</title><rect x="78.6" y="1861" width="0.2" height="15.0" fill="rgb(205,11,32)" rx="2" ry="2" />
<text x="81.63" y="1871.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="837" width="364.1" height="15.0" fill="rgb(213,99,29)" rx="2" ry="2" />
<text x="790.11" y="847.5" >[unknown]</text>
</g>
<g >
<title>do_syscall_64 (772 samples, 0.27%)</title><rect x="816.6" y="165" width="3.1" height="15.0" fill="rgb(238,61,16)" rx="2" ry="2" />
<text x="819.55" y="175.5" ></text>
</g>
<g >
<title>security_inode_getattr (137 samples, 0.05%)</title><rect x="707.5" y="1973" width="0.6" height="15.0" fill="rgb(252,202,30)" rx="2" ry="2" />
<text x="710.52" y="1983.5" ></text>
</g>
<g >
<title>avc_has_perm (38 samples, 0.01%)</title><rect x="666.7" y="1909" width="0.2" height="15.0" fill="rgb(226,212,3)" rx="2" ry="2" />
<text x="669.73" y="1919.5" ></text>
</g>
<g >
<title>ima_file_free (37 samples, 0.01%)</title><rect x="262.0" y="1941" width="0.1" height="15.0" fill="rgb(227,19,11)" rx="2" ry="2" />
<text x="264.95" y="1951.5" ></text>
</g>
<g >
<title>pmprobe (335 samples, 0.12%)</title><rect x="717.6" y="2069" width="1.3" height="15.0" fill="rgb(227,133,11)" rx="2" ry="2" />
<text x="720.56" y="2079.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (63 samples, 0.02%)</title><rect x="1165.3" y="2037" width="0.2" height="15.0" fill="rgb(212,82,16)" rx="2" ry="2" />
<text x="1168.25" y="2047.5" ></text>
</g>
<g >
<title>sock_recvmsg (381 samples, 0.13%)</title><rect x="1146.7" y="197" width="1.6" height="15.0" fill="rgb(246,61,14)" rx="2" ry="2" />
<text x="1149.71" y="207.5" ></text>
</g>
<g >
<title>kmem_cache_alloc (54 samples, 0.02%)</title><rect x="686.8" y="1877" width="0.2" height="15.0" fill="rgb(251,216,33)" rx="2" ry="2" />
<text x="689.79" y="1887.5" ></text>
</g>
<g >
<title>do_syscall_64 (4,279 samples, 1.50%)</title><rect x="685.1" y="2005" width="17.7" height="15.0" fill="rgb(252,25,9)" rx="2" ry="2" />
<text x="688.13" y="2015.5" ></text>
</g>
<g >
<title>find_vmap_area (38,824 samples, 13.61%)</title><rect x="483.4" y="1829" width="160.6" height="15.0" fill="rgb(246,194,54)" rx="2" ry="2" />
<text x="486.44" y="1839.5" >find_vmap_area</text>
</g>
<g >
<title>pv_wait_head_or_lock (61 samples, 0.02%)</title><rect x="834.1" y="37" width="0.2" height="15.0" fill="rgb(241,53,14)" rx="2" ry="2" />
<text x="837.07" y="47.5" ></text>
</g>
<g >
<title>_raw_spin_lock (34 samples, 0.01%)</title><rect x="685.8" y="1941" width="0.2" height="15.0" fill="rgb(208,53,8)" rx="2" ry="2" />
<text x="688.85" y="1951.5" ></text>
</g>
<g >
<title>inode_security (141 samples, 0.05%)</title><rect x="475.1" y="1909" width="0.6" height="15.0" fill="rgb(251,13,15)" rx="2" ry="2" />
<text x="478.10" y="1919.5" ></text>
</g>
<g >
<title>do_syscall_64 (48,998 samples, 17.17%)</title><rect x="274.5" y="1989" width="202.6" height="15.0" fill="rgb(232,142,47)" rx="2" ry="2" />
<text x="277.49" y="1999.5" >do_syscall_64</text>
</g>
<g >
<title>__pv_queued_spin_unlock_slowpath (429 samples, 0.15%)</title><rect x="463.1" y="1733" width="1.8" height="15.0" fill="rgb(226,115,42)" rx="2" ry="2" />
<text x="466.10" y="1743.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1317" width="364.1" height="15.0" fill="rgb(226,74,41)" rx="2" ry="2" />
<text x="790.11" y="1327.5" >[unknown]</text>
</g>
<g >
<title>sock_poll (103 samples, 0.04%)</title><rect x="1169.7" y="1941" width="0.4" height="15.0" fill="rgb(230,227,46)" rx="2" ry="2" />
<text x="1172.67" y="1951.5" ></text>
</g>
<g >
<title>syscall_exit_to_user_mode (35 samples, 0.01%)</title><rect x="809.5" y="117" width="0.1" height="15.0" fill="rgb(245,105,11)" rx="2" ry="2" />
<text x="812.50" y="127.5" ></text>
</g>
<g >
<title>smc_tx_rdma_writes (196 samples, 0.07%)</title><rect x="56.6" y="1797" width="0.8" height="15.0" fill="rgb(207,29,42)" rx="2" ry="2" />
<text x="59.60" y="1807.5" ></text>
</g>
<g >
<title>[nginx] (221 samples, 0.08%)</title><rect x="47.6" y="1957" width="1.0" height="15.0" fill="rgb(238,47,32)" rx="2" ry="2" />
<text x="50.65" y="1967.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (36,893 samples, 12.93%)</title><rect x="992.2" y="133" width="152.5" height="15.0" fill="rgb(226,93,40)" rx="2" ry="2" />
<text x="995.15" y="143.5" >__check_object_size..</text>
</g>
<g >
<title>ngx_output_chain_to_iovec (53 samples, 0.02%)</title><rect x="714.0" y="2053" width="0.3" height="15.0" fill="rgb(220,228,49)" rx="2" ry="2" />
<text x="717.04" y="2063.5" ></text>
</g>
<g >
<title>[libc.so.6] (509 samples, 0.18%)</title><rect x="19.1" y="2037" width="2.1" height="15.0" fill="rgb(214,131,17)" rx="2" ry="2" />
<text x="22.11" y="2047.5" ></text>
</g>
<g >
<title>ksys_write (190 samples, 0.07%)</title><rect x="822.3" y="165" width="0.8" height="15.0" fill="rgb(211,222,9)" rx="2" ry="2" />
<text x="825.30" y="175.5" ></text>
</g>
<g >
<title>do_splice_direct (47,807 samples, 16.76%)</title><rect x="276.6" y="1941" width="197.7" height="15.0" fill="rgb(224,84,21)" rx="2" ry="2" />
<text x="279.59" y="1951.5" >do_splice_direct</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="2021" width="364.1" height="15.0" fill="rgb(253,89,10)" rx="2" ry="2" />
<text x="790.11" y="2031.5" >[unknown]</text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (55 samples, 0.02%)</title><rect x="822.9" y="69" width="0.2" height="15.0" fill="rgb(205,154,26)" rx="2" ry="2" />
<text x="825.86" y="79.5" ></text>
</g>
<g >
<title>__fget_light (180 samples, 0.06%)</title><rect x="256.4" y="1909" width="0.7" height="15.0" fill="rgb(226,226,52)" rx="2" ry="2" />
<text x="259.37" y="1919.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (1,088 samples, 0.38%)</title><rect x="823.6" y="37" width="4.5" height="15.0" fill="rgb(217,124,54)" rx="2" ry="2" />
<text x="826.63" y="47.5" ></text>
</g>
<g >
<title>__do_softirq (26 samples, 0.01%)</title><rect x="785.7" y="1829" width="0.1" height="15.0" fill="rgb(238,125,39)" rx="2" ry="2" />
<text x="788.68" y="1839.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (33 samples, 0.01%)</title><rect x="716.2" y="2021" width="0.1" height="15.0" fill="rgb(209,21,7)" rx="2" ry="2" />
<text x="719.17" y="2031.5" ></text>
</g>
<g >
<title>ksys_read (381 samples, 0.13%)</title><rect x="1146.7" y="245" width="1.6" height="15.0" fill="rgb(221,191,51)" rx="2" ry="2" />
<text x="1149.71" y="255.5" ></text>
</g>
<g >
<title>_raw_read_lock (122 samples, 0.04%)</title><rect x="58.5" y="1749" width="0.5" height="15.0" fill="rgb(232,38,26)" rx="2" ry="2" />
<text x="61.46" y="1759.5" ></text>
</g>
<g >
<title>sock_write_iter (218 samples, 0.08%)</title><rect x="811.5" y="101" width="0.9" height="15.0" fill="rgb(227,139,51)" rx="2" ry="2" />
<text x="814.54" y="111.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (5,009 samples, 1.76%)</title><rect x="55.8" y="1829" width="20.7" height="15.0" fill="rgb(245,3,47)" rx="2" ry="2" />
<text x="58.83" y="1839.5" ></text>
</g>
<g >
<title>sockfd_lookup_light (202 samples, 0.07%)</title><rect x="256.3" y="1925" width="0.8" height="15.0" fill="rgb(205,137,9)" rx="2" ry="2" />
<text x="259.28" y="1935.5" ></text>
</g>
<g >
<title>ngx_vslprintf (347 samples, 0.12%)</title><rect x="681.1" y="2037" width="1.4" height="15.0" fill="rgb(228,216,50)" rx="2" ry="2" />
<text x="684.10" y="2047.5" ></text>
</g>
<g >
<title>do_execveat_common (29 samples, 0.01%)</title><rect x="716.5" y="1973" width="0.1" height="15.0" fill="rgb(215,138,11)" rx="2" ry="2" />
<text x="719.45" y="1983.5" ></text>
</g>
<g >
<title>ngx_http_finalize_request (105 samples, 0.04%)</title><rect x="711.5" y="2053" width="0.4" height="15.0" fill="rgb(208,94,13)" rx="2" ry="2" />
<text x="714.51" y="2063.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (31 samples, 0.01%)</title><rect x="810.2" y="37" width="0.1" height="15.0" fill="rgb(227,176,24)" rx="2" ry="2" />
<text x="813.21" y="47.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (1,049 samples, 0.37%)</title><rect x="1181.5" y="1909" width="4.4" height="15.0" fill="rgb(224,160,36)" rx="2" ry="2" />
<text x="1184.55" y="1919.5" ></text>
</g>
<g >
<title>do_syscall_64 (3,503 samples, 1.23%)</title><rect x="1174.3" y="2021" width="14.5" height="15.0" fill="rgb(212,124,15)" rx="2" ry="2" />
<text x="1177.32" y="2031.5" ></text>
</g>
<g >
<title>check_heap_object (341 samples, 0.12%)</title><rect x="1144.7" y="133" width="1.5" height="15.0" fill="rgb(217,120,41)" rx="2" ry="2" />
<text x="1147.74" y="143.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (807 samples, 0.28%)</title><rect x="1153.7" y="2021" width="3.3" height="15.0" fill="rgb(221,127,21)" rx="2" ry="2" />
<text x="1156.67" y="2031.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (96 samples, 0.03%)</title><rect x="718.9" y="2053" width="0.4" height="15.0" fill="rgb(226,39,26)" rx="2" ry="2" />
<text x="721.94" y="2063.5" ></text>
</g>
<g >
<title>selinux_file_permission (63 samples, 0.02%)</title><rect x="804.8" y="53" width="0.3" height="15.0" fill="rgb(221,29,40)" rx="2" ry="2" />
<text x="807.82" y="63.5" ></text>
</g>
<g >
<title>response_complete (100 samples, 0.04%)</title><rect x="796.4" y="53" width="0.4" height="15.0" fill="rgb(240,198,22)" rx="2" ry="2" />
<text x="799.41" y="63.5" ></text>
</g>
<g >
<title>security_file_alloc (136 samples, 0.05%)</title><rect x="686.7" y="1893" width="0.6" height="15.0" fill="rgb(216,2,10)" rx="2" ry="2" />
<text x="689.72" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1093" width="364.1" height="15.0" fill="rgb(244,208,32)" rx="2" ry="2" />
<text x="790.11" y="1103.5" >[unknown]</text>
</g>
<g >
<title>file_has_perm (31 samples, 0.01%)</title><rect x="276.8" y="1909" width="0.1" height="15.0" fill="rgb(247,147,17)" rx="2" ry="2" />
<text x="279.75" y="1919.5" ></text>
</g>
<g >
<title>selinux_inode_permission (73 samples, 0.03%)</title><rect x="693.7" y="1877" width="0.3" height="15.0" fill="rgb(227,92,30)" rx="2" ry="2" />
<text x="696.74" y="1887.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (43,385 samples, 15.21%)</title><rect x="283.6" y="1765" width="179.5" height="15.0" fill="rgb(214,175,14)" rx="2" ry="2" />
<text x="286.64" y="1775.5" >__pv_queued_spin_lock_s..</text>
</g>
<g >
<title>ngx_http_write_filter (222 samples, 0.08%)</title><rect x="712.6" y="2053" width="1.0" height="15.0" fill="rgb(239,148,51)" rx="2" ry="2" />
<text x="715.64" y="2063.5" ></text>
</g>
<g >
<title>release_sock (34 samples, 0.01%)</title><rect x="482.1" y="1877" width="0.1" height="15.0" fill="rgb(242,65,17)" rx="2" ry="2" />
<text x="485.09" y="1887.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1573" width="364.1" height="15.0" fill="rgb(220,8,38)" rx="2" ry="2" />
<text x="790.11" y="1583.5" >[unknown]</text>
</g>
<g >
<title>entry_SYSCALL_64 (1,319 samples, 0.46%)</title><rect x="823.6" y="213" width="5.5" height="15.0" fill="rgb(234,172,40)" rx="2" ry="2" />
<text x="826.63" y="223.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (144 samples, 0.05%)</title><rect x="838.3" y="37" width="0.6" height="15.0" fill="rgb(232,74,20)" rx="2" ry="2" />
<text x="841.29" y="47.5" ></text>
</g>
<g >
<title>avc_has_perm (40 samples, 0.01%)</title><rect x="481.5" y="1861" width="0.1" height="15.0" fill="rgb(208,209,25)" rx="2" ry="2" />
<text x="484.46" y="1871.5" ></text>
</g>
<g >
<title>__close (1,980 samples, 0.69%)</title><rect x="258.3" y="2021" width="8.2" height="15.0" fill="rgb(252,93,23)" rx="2" ry="2" />
<text x="261.34" y="2031.5" ></text>
</g>
<g >
<title>hrtimer_try_to_cancel.part.0 (105 samples, 0.04%)</title><rect x="1181.0" y="1941" width="0.4" height="15.0" fill="rgb(246,11,13)" rx="2" ry="2" />
<text x="1183.99" y="1951.5" ></text>
</g>
<g >
<title>selinux_file_alloc_security (63 samples, 0.02%)</title><rect x="687.0" y="1877" width="0.3" height="15.0" fill="rgb(236,189,0)" rx="2" ry="2" />
<text x="690.02" y="1887.5" ></text>
</g>
<g >
<title>avc_has_perm (77 samples, 0.03%)</title><rect x="645.2" y="1893" width="0.3" height="15.0" fill="rgb(218,215,31)" rx="2" ry="2" />
<text x="648.21" y="1903.5" ></text>
</g>
<g >
<title>sock_close (112 samples, 0.04%)</title><rect x="812.4" y="117" width="0.5" height="15.0" fill="rgb(238,202,38)" rx="2" ry="2" />
<text x="815.45" y="127.5" ></text>
</g>
<g >
<title>_raw_spin_lock (860 samples, 0.30%)</title><rect x="834.1" y="69" width="3.5" height="15.0" fill="rgb(213,173,3)" rx="2" ry="2" />
<text x="837.06" y="79.5" ></text>
</g>
<g >
<title>find_vmap_area (606 samples, 0.21%)</title><rect x="816.6" y="37" width="2.5" height="15.0" fill="rgb(214,106,48)" rx="2" ry="2" />
<text x="819.56" y="47.5" ></text>
</g>
<g >
<title>do_syscall_64 (614 samples, 0.22%)</title><rect x="819.8" y="181" width="2.5" height="15.0" fill="rgb(243,19,54)" rx="2" ry="2" />
<text x="822.76" y="191.5" ></text>
</g>
<g >
<title>find_vmap_area (1,201 samples, 0.42%)</title><rect x="829.1" y="69" width="5.0" height="15.0" fill="rgb(223,178,17)" rx="2" ry="2" />
<text x="832.08" y="79.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (60 samples, 0.02%)</title><rect x="84.2" y="1829" width="0.3" height="15.0" fill="rgb(244,77,40)" rx="2" ry="2" />
<text x="87.22" y="1839.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1781" width="364.1" height="15.0" fill="rgb(205,162,8)" rx="2" ry="2" />
<text x="790.11" y="1791.5" >[unknown]</text>
</g>
<g >
<title>__sys_setsockopt (6,209 samples, 2.18%)</title><rect x="51.9" y="1893" width="25.7" height="15.0" fill="rgb(222,39,11)" rx="2" ry="2" />
<text x="54.90" y="1903.5" >_..</text>
</g>
<g >
<title>sysvec_call_function_single (26 samples, 0.01%)</title><rect x="785.7" y="1861" width="0.1" height="15.0" fill="rgb(233,213,18)" rx="2" ry="2" />
<text x="788.68" y="1871.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (477 samples, 0.17%)</title><rect x="805.6" y="133" width="2.0" height="15.0" fill="rgb(247,59,47)" rx="2" ry="2" />
<text x="808.63" y="143.5" ></text>
</g>
<g >
<title>[unknown] (6,126 samples, 2.15%)</title><rect x="787.1" y="197" width="25.3" height="15.0" fill="rgb(219,28,22)" rx="2" ry="2" />
<text x="790.11" y="207.5" >[..</text>
</g>
<g >
<title>smc_poll (115 samples, 0.04%)</title><rect x="1179.0" y="1909" width="0.5" height="15.0" fill="rgb(254,138,14)" rx="2" ry="2" />
<text x="1181.99" y="1919.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1653" width="364.1" height="15.0" fill="rgb(205,45,45)" rx="2" ry="2" />
<text x="790.11" y="1663.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1957" width="364.1" height="15.0" fill="rgb(225,139,11)" rx="2" ry="2" />
<text x="790.11" y="1967.5" >[unknown]</text>
</g>
<g >
<title>selinux_socket_sendmsg (94 samples, 0.03%)</title><rect x="806.0" y="37" width="0.4" height="15.0" fill="rgb(206,136,21)" rx="2" ry="2" />
<text x="809.02" y="47.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (31 samples, 0.01%)</title><rect x="810.2" y="85" width="0.1" height="15.0" fill="rgb(252,56,39)" rx="2" ry="2" />
<text x="813.21" y="95.5" ></text>
</g>
<g >
<title>__legitimize_path (149 samples, 0.05%)</title><rect x="655.3" y="1877" width="0.6" height="15.0" fill="rgb(233,154,53)" rx="2" ry="2" />
<text x="658.31" y="1887.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (1,319 samples, 0.46%)</title><rect x="823.6" y="117" width="5.5" height="15.0" fill="rgb(254,109,35)" rx="2" ry="2" />
<text x="826.63" y="127.5" ></text>
</g>
<g >
<title>vfs_write (190 samples, 0.07%)</title><rect x="822.3" y="149" width="0.8" height="15.0" fill="rgb(229,86,41)" rx="2" ry="2" />
<text x="825.30" y="159.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (86 samples, 0.03%)</title><rect x="1148.3" y="69" width="0.3" height="15.0" fill="rgb(245,61,3)" rx="2" ry="2" />
<text x="1151.28" y="79.5" ></text>
</g>
<g >
<title>__do_softirq (30 samples, 0.01%)</title><rect x="642.0" y="1701" width="0.1" height="15.0" fill="rgb(245,127,24)" rx="2" ry="2" />
<text x="645.03" y="1711.5" ></text>
</g>
<g >
<title>__mmput (27 samples, 0.01%)</title><rect x="717.3" y="1877" width="0.1" height="15.0" fill="rgb(223,218,25)" rx="2" ry="2" />
<text x="720.31" y="1887.5" ></text>
</g>
<g >
<title>do_syscall_64 (254 samples, 0.09%)</title><rect x="798.2" y="53" width="1.1" height="15.0" fill="rgb(217,187,24)" rx="2" ry="2" />
<text x="801.22" y="63.5" ></text>
</g>
<g >
<title>smc_lo_move_data (308 samples, 0.11%)</title><rect x="837.6" y="53" width="1.3" height="15.0" fill="rgb(221,79,16)" rx="2" ry="2" />
<text x="840.61" y="63.5" ></text>
</g>
<g >
<title>[unknown] (87,411 samples, 30.64%)</title><rect x="787.1" y="309" width="361.5" height="15.0" fill="rgb(245,221,26)" rx="2" ry="2" />
<text x="790.11" y="319.5" >[unknown]</text>
</g>
<g >
<title>security_socket_sendmsg (135 samples, 0.05%)</title><rect x="481.1" y="1893" width="0.5" height="15.0" fill="rgb(220,100,18)" rx="2" ry="2" />
<text x="484.07" y="1903.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (475 samples, 0.17%)</title><rect x="1144.7" y="261" width="2.0" height="15.0" fill="rgb(241,167,10)" rx="2" ry="2" />
<text x="1147.74" y="271.5" ></text>
</g>
<g >
<title>ngx_tcp_nopush (122 samples, 0.04%)</title><rect x="248.9" y="2005" width="0.5" height="15.0" fill="rgb(243,175,32)" rx="2" ry="2" />
<text x="251.94" y="2015.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock_slowpath (350 samples, 0.12%)</title><rect x="642.2" y="1781" width="1.4" height="15.0" fill="rgb(218,53,31)" rx="2" ry="2" />
<text x="645.17" y="1791.5" ></text>
</g>
<g >
<title>__schedule (511 samples, 0.18%)</title><rect x="779.7" y="1973" width="2.1" height="15.0" fill="rgb(236,67,3)" rx="2" ry="2" />
<text x="782.71" y="1983.5" ></text>
</g>
<g >
<title>read (369 samples, 0.13%)</title><rect x="802.0" y="133" width="1.5" height="15.0" fill="rgb(217,17,30)" rx="2" ry="2" />
<text x="804.97" y="143.5" ></text>
</g>
<g >
<title>sock_read_iter (381 samples, 0.13%)</title><rect x="1146.7" y="213" width="1.6" height="15.0" fill="rgb(251,157,18)" rx="2" ry="2" />
<text x="1149.71" y="223.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (45,788 samples, 16.05%)</title><rect x="280.4" y="1845" width="189.4" height="15.0" fill="rgb(221,93,41)" rx="2" ry="2" />
<text x="283.38" y="1855.5" >smc_tx_sendmsg</text>
</g>
<g >
<title>vfs_fstat (507 samples, 0.18%)</title><rect x="706.3" y="1989" width="2.1" height="15.0" fill="rgb(223,92,48)" rx="2" ry="2" />
<text x="709.34" y="1999.5" ></text>
</g>
<g >
<title>cfree (57 samples, 0.02%)</title><rect x="48.6" y="1957" width="0.2" height="15.0" fill="rgb(227,41,7)" rx="2" ry="2" />
<text x="51.56" y="1967.5" ></text>
</g>
<g >
<title>do_syscall_64 (63 samples, 0.02%)</title><rect x="799.3" y="69" width="0.3" height="15.0" fill="rgb(208,89,53)" rx="2" ry="2" />
<text x="802.29" y="79.5" ></text>
</g>
<g >
<title>[unknown] (87,971 samples, 30.83%)</title><rect x="787.1" y="389" width="363.9" height="15.0" fill="rgb(213,67,3)" rx="2" ry="2" />
<text x="790.11" y="399.5" >[unknown]</text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (148 samples, 0.05%)</title><rect x="1180.1" y="1941" width="0.6" height="15.0" fill="rgb(243,153,46)" rx="2" ry="2" />
<text x="1183.10" y="1951.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (381 samples, 0.13%)</title><rect x="1146.7" y="101" width="1.6" height="15.0" fill="rgb(253,176,19)" rx="2" ry="2" />
<text x="1149.71" y="111.5" ></text>
</g>
<g >
<title>ksys_read (1,201 samples, 0.42%)</title><rect x="829.1" y="197" width="5.0" height="15.0" fill="rgb(230,213,12)" rx="2" ry="2" />
<text x="832.08" y="207.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (473 samples, 0.17%)</title><rect x="1148.7" y="133" width="1.9" height="15.0" fill="rgb(214,33,8)" rx="2" ry="2" />
<text x="1151.67" y="143.5" ></text>
</g>
<g >
<title>vfs_write (475 samples, 0.17%)</title><rect x="1144.7" y="213" width="2.0" height="15.0" fill="rgb(234,94,53)" rx="2" ry="2" />
<text x="1147.74" y="223.5" ></text>
</g>
<g >
<title>_raw_spin_lock (43,561 samples, 15.27%)</title><rect x="282.9" y="1781" width="180.2" height="15.0" fill="rgb(241,114,40)" rx="2" ry="2" />
<text x="285.91" y="1791.5" >_raw_spin_lock</text>
</g>
<g >
<title>_copy_from_iter (193 samples, 0.07%)</title><rect x="811.6" y="53" width="0.8" height="15.0" fill="rgb(248,152,7)" rx="2" ry="2" />
<text x="814.60" y="63.5" ></text>
</g>
<g >
<title>do_syscall_64 (190 samples, 0.07%)</title><rect x="822.3" y="181" width="0.8" height="15.0" fill="rgb(254,37,29)" rx="2" ry="2" />
<text x="825.30" y="191.5" ></text>
</g>
<g >
<title>[unknown] (48 samples, 0.02%)</title><rect x="10.5" y="2053" width="0.2" height="15.0" fill="rgb(248,94,32)" rx="2" ry="2" />
<text x="13.50" y="2063.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (36,890 samples, 12.93%)</title><rect x="992.2" y="69" width="152.5" height="15.0" fill="rgb(227,109,13)" rx="2" ry="2" />
<text x="995.15" y="79.5" >__pv_queued_spin_lo..</text>
</g>
<g >
<title>build_open_flags (45 samples, 0.02%)</title><rect x="686.1" y="1957" width="0.2" height="15.0" fill="rgb(233,222,9)" rx="2" ry="2" />
<text x="689.07" y="1967.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (36 samples, 0.01%)</title><rect x="10.1" y="2037" width="0.2" height="15.0" fill="rgb(230,188,48)" rx="2" ry="2" />
<text x="13.13" y="2047.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (35 samples, 0.01%)</title><rect x="809.5" y="37" width="0.1" height="15.0" fill="rgb(219,98,28)" rx="2" ry="2" />
<text x="812.50" y="47.5" ></text>
</g>
<g >
<title>sockopt_lock_sock (36 samples, 0.01%)</title><rect x="254.8" y="1877" width="0.1" height="15.0" fill="rgb(242,53,0)" rx="2" ry="2" />
<text x="257.77" y="1887.5" ></text>
</g>
<g >
<title>do_writev (40,483 samples, 14.19%)</title><rect x="479.3" y="1973" width="167.5" height="15.0" fill="rgb(207,23,40)" rx="2" ry="2" />
<text x="482.32" y="1983.5" >do_writev</text>
</g>
<g >
<title>smc_sendmsg (1,319 samples, 0.46%)</title><rect x="823.6" y="133" width="5.5" height="15.0" fill="rgb(231,117,42)" rx="2" ry="2" />
<text x="826.63" y="143.5" ></text>
</g>
<g >
<title>fstatat64 (1,312 samples, 0.46%)</title><rect x="704.5" y="2053" width="5.4" height="15.0" fill="rgb(213,41,11)" rx="2" ry="2" />
<text x="707.47" y="2063.5" ></text>
</g>
<g >
<title>[unknown] (87,984 samples, 30.84%)</title><rect x="787.1" y="405" width="363.9" height="15.0" fill="rgb(234,141,10)" rx="2" ry="2" />
<text x="790.11" y="415.5" >[unknown]</text>
</g>
<g >
<title>smc_rx_recvmsg (381 samples, 0.13%)</title><rect x="1146.7" y="165" width="1.6" height="15.0" fill="rgb(227,23,18)" rx="2" ry="2" />
<text x="1149.71" y="175.5" ></text>
</g>
<g >
<title>smc_lo_move_data (134 samples, 0.05%)</title><rect x="1146.2" y="85" width="0.5" height="15.0" fill="rgb(251,112,38)" rx="2" ry="2" />
<text x="1149.15" y="95.5" ></text>
</g>
<g >
<title>_raw_spin_lock (59 samples, 0.02%)</title><rect x="263.1" y="1941" width="0.2" height="15.0" fill="rgb(208,207,10)" rx="2" ry="2" />
<text x="266.08" y="1951.5" ></text>
</g>
<g >
<title>smc_lo_move_data (4,534 samples, 1.59%)</title><rect x="57.8" y="1765" width="18.7" height="15.0" fill="rgb(249,131,54)" rx="2" ry="2" />
<text x="60.79" y="1775.5" ></text>
</g>
<g >
<title>smc_recvmsg (100 samples, 0.04%)</title><rect x="805.2" y="37" width="0.4" height="15.0" fill="rgb(218,101,46)" rx="2" ry="2" />
<text x="808.20" y="47.5" ></text>
</g>
<g >
<title>read (245 samples, 0.09%)</title><rect x="797.2" y="85" width="1.0" height="15.0" fill="rgb(206,177,32)" rx="2" ry="2" />
<text x="800.21" y="95.5" ></text>
</g>
<g >
<title>ngx_memalign (52 samples, 0.02%)</title><rect x="678.8" y="2037" width="0.2" height="15.0" fill="rgb(233,75,37)" rx="2" ry="2" />
<text x="681.79" y="2047.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (111 samples, 0.04%)</title><rect x="52.9" y="1813" width="0.4" height="15.0" fill="rgb(241,185,30)" rx="2" ry="2" />
<text x="55.86" y="1823.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1269" width="364.1" height="15.0" fill="rgb(211,96,12)" rx="2" ry="2" />
<text x="790.11" y="1279.5" >[unknown]</text>
</g>
<g >
<title>sock_read_iter (613 samples, 0.21%)</title><rect x="819.8" y="133" width="2.5" height="15.0" fill="rgb(230,5,8)" rx="2" ry="2" />
<text x="822.76" y="143.5" ></text>
</g>
<g >
<title>rep_movs_alternative (47 samples, 0.02%)</title><rect x="1167.3" y="1973" width="0.2" height="15.0" fill="rgb(223,182,6)" rx="2" ry="2" />
<text x="1170.29" y="1983.5" ></text>
</g>
<g >
<title>security_socket_sendmsg (94 samples, 0.03%)</title><rect x="806.0" y="53" width="0.4" height="15.0" fill="rgb(209,60,39)" rx="2" ry="2" />
<text x="809.02" y="63.5" ></text>
</g>
<g >
<title>hrtimer_active (27 samples, 0.01%)</title><rect x="1180.7" y="1941" width="0.1" height="15.0" fill="rgb(214,23,28)" rx="2" ry="2" />
<text x="1183.71" y="1951.5" ></text>
</g>
<g >
<title>ksys_write (474 samples, 0.17%)</title><rect x="805.6" y="101" width="2.0" height="15.0" fill="rgb(224,40,30)" rx="2" ry="2" />
<text x="808.63" y="111.5" ></text>
</g>
<g >
<title>vfs_read (1,201 samples, 0.42%)</title><rect x="829.1" y="181" width="5.0" height="15.0" fill="rgb(230,206,4)" rx="2" ry="2" />
<text x="832.08" y="191.5" ></text>
</g>
<g >
<title>epoll_wait (395 samples, 0.14%)</title><rect x="245.5" y="2005" width="1.6" height="15.0" fill="rgb(234,89,36)" rx="2" ry="2" />
<text x="248.47" y="2015.5" ></text>
</g>
<g >
<title>smc_conn_free (108 samples, 0.04%)</title><rect x="812.5" y="53" width="0.4" height="15.0" fill="rgb(241,214,52)" rx="2" ry="2" />
<text x="815.45" y="63.5" ></text>
</g>
<g >
<title>writev (41,332 samples, 14.49%)</title><rect x="477.2" y="2021" width="171.0" height="15.0" fill="rgb(230,207,47)" rx="2" ry="2" />
<text x="480.24" y="2031.5" >writev</text>
</g>
<g >
<title>syscall_enter_from_user_mode (259 samples, 0.09%)</title><rect x="243.8" y="1925" width="1.1" height="15.0" fill="rgb(224,130,39)" rx="2" ry="2" />
<text x="246.85" y="1935.5" ></text>
</g>
<g >
<title>smc_rx_wake_up (52 samples, 0.02%)</title><rect x="1148.3" y="37" width="0.3" height="15.0" fill="rgb(244,52,51)" rx="2" ry="2" />
<text x="1151.34" y="47.5" ></text>
</g>
<g >
<title>exit_mmap (26 samples, 0.01%)</title><rect x="717.3" y="1861" width="0.1" height="15.0" fill="rgb(227,31,11)" rx="2" ry="2" />
<text x="720.31" y="1871.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (36,893 samples, 12.93%)</title><rect x="992.2" y="149" width="152.5" height="15.0" fill="rgb(219,100,10)" rx="2" ry="2" />
<text x="995.15" y="159.5" >smc_rx_recvmsg</text>
</g>
<g >
<title>ksys_read (614 samples, 0.22%)</title><rect x="819.8" y="165" width="2.5" height="15.0" fill="rgb(209,63,12)" rx="2" ry="2" />
<text x="822.76" y="175.5" ></text>
</g>
<g >
<title>selinux_inode_getattr (144 samples, 0.05%)</title><rect x="666.3" y="1925" width="0.6" height="15.0" fill="rgb(242,225,31)" rx="2" ry="2" />
<text x="669.30" y="1935.5" ></text>
</g>
<g >
<title>ngx_chain_update_sent (36 samples, 0.01%)</title><rect x="267.5" y="2021" width="0.2" height="15.0" fill="rgb(222,90,35)" rx="2" ry="2" />
<text x="270.54" y="2031.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (309 samples, 0.11%)</title><rect x="667.4" y="1989" width="1.3" height="15.0" fill="rgb(237,157,16)" rx="2" ry="2" />
<text x="670.45" y="1999.5" ></text>
</g>
<g >
<title>ksys_read (458 samples, 0.16%)</title><rect x="807.6" y="117" width="1.9" height="15.0" fill="rgb(219,11,15)" rx="2" ry="2" />
<text x="810.61" y="127.5" ></text>
</g>
<g >
<title>[unknown] (88,026 samples, 30.85%)</title><rect x="787.1" y="597" width="364.1" height="15.0" fill="rgb(211,168,15)" rx="2" ry="2" />
<text x="790.11" y="607.5" >[unknown]</text>
</g>
<g >
<title>__check_object_size.part.0 (26 samples, 0.01%)</title><rect x="812.9" y="53" width="0.2" height="15.0" fill="rgb(215,60,15)" rx="2" ry="2" />
<text x="815.95" y="63.5" ></text>
</g>
<g >
<title>obj_cgroup_uncharge_pages (30 samples, 0.01%)</title><rect x="265.1" y="1941" width="0.2" height="15.0" fill="rgb(210,17,46)" rx="2" ry="2" />
<text x="268.13" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="933" width="364.1" height="15.0" fill="rgb(234,137,17)" rx="2" ry="2" />
<text x="790.11" y="943.5" >[unknown]</text>
</g>
<g >
<title>avc_has_perm (74 samples, 0.03%)</title><rect x="279.6" y="1829" width="0.3" height="15.0" fill="rgb(209,172,32)" rx="2" ry="2" />
<text x="282.56" y="1839.5" ></text>
</g>
<g >
<title>[unknown] (48,092 samples, 16.86%)</title><rect x="46.6" y="2005" width="198.9" height="15.0" fill="rgb(238,52,41)" rx="2" ry="2" />
<text x="49.56" y="2015.5" >[unknown]</text>
</g>
<g >
<title>search_binary_handler (29 samples, 0.01%)</title><rect x="717.3" y="1941" width="0.1" height="15.0" fill="rgb(241,11,46)" rx="2" ry="2" />
<text x="720.30" y="1951.5" ></text>
</g>
<g >
<title>__sys_connect (62 samples, 0.02%)</title><rect x="1165.3" y="1989" width="0.2" height="15.0" fill="rgb(212,180,44)" rx="2" ry="2" />
<text x="1168.25" y="1999.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (43 samples, 0.02%)</title><rect x="78.6" y="1877" width="0.2" height="15.0" fill="rgb(244,51,22)" rx="2" ry="2" />
<text x="81.62" y="1887.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (39 samples, 0.01%)</title><rect x="703.7" y="2037" width="0.1" height="15.0" fill="rgb(210,152,11)" rx="2" ry="2" />
<text x="706.68" y="2047.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (61 samples, 0.02%)</title><rect x="811.1" y="53" width="0.3" height="15.0" fill="rgb(209,10,22)" rx="2" ry="2" />
<text x="814.15" y="63.5" ></text>
</g>
<g >
<title>vfs_read (614 samples, 0.22%)</title><rect x="819.8" y="149" width="2.5" height="15.0" fill="rgb(250,176,30)" rx="2" ry="2" />
<text x="822.76" y="159.5" ></text>
</g>
<g >
<title>release_sock (27 samples, 0.01%)</title><rect x="55.1" y="1829" width="0.1" height="15.0" fill="rgb(252,139,26)" rx="2" ry="2" />
<text x="58.10" y="1839.5" ></text>
</g>
<g >
<title>malloc (40 samples, 0.01%)</title><rect x="668.8" y="2037" width="0.2" height="15.0" fill="rgb(233,222,35)" rx="2" ry="2" />
<text x="671.80" y="2047.5" ></text>
</g>
<g >
<title>__close (112 samples, 0.04%)</title><rect x="812.4" y="197" width="0.5" height="15.0" fill="rgb(236,48,32)" rx="2" ry="2" />
<text x="815.45" y="207.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (132 samples, 0.05%)</title><rect x="476.6" y="1957" width="0.5" height="15.0" fill="rgb(223,10,53)" rx="2" ry="2" />
<text x="479.56" y="1967.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (308 samples, 0.11%)</title><rect x="837.6" y="85" width="1.3" height="15.0" fill="rgb(249,23,54)" rx="2" ry="2" />
<text x="840.61" y="95.5" ></text>
</g>
<g >
<title>smc_recvmsg (204 samples, 0.07%)</title><rect x="810.6" y="69" width="0.8" height="15.0" fill="rgb(244,56,41)" rx="2" ry="2" />
<text x="813.56" y="79.5" ></text>
</g>
<g >
<title>[unknown] (88,006 samples, 30.85%)</title><rect x="787.1" y="501" width="364.0" height="15.0" fill="rgb(219,138,42)" rx="2" ry="2" />
<text x="790.11" y="511.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1861" width="364.1" height="15.0" fill="rgb(246,82,44)" rx="2" ry="2" />
<text x="790.11" y="1871.5" >[unknown]</text>
</g>
<g >
<title>sock_write_iter (380 samples, 0.13%)</title><rect x="806.0" y="69" width="1.6" height="15.0" fill="rgb(205,227,0)" rx="2" ry="2" />
<text x="809.02" y="79.5" ></text>
</g>
<g >
<title>release_sock (36 samples, 0.01%)</title><rect x="85.3" y="1861" width="0.2" height="15.0" fill="rgb(205,212,18)" rx="2" ry="2" />
<text x="88.31" y="1871.5" ></text>
</g>
<g >
<title>ksys_write (128 samples, 0.04%)</title><rect x="799.6" y="53" width="0.5" height="15.0" fill="rgb(234,134,7)" rx="2" ry="2" />
<text x="802.59" y="63.5" ></text>
</g>
<g >
<title>pick_file (71 samples, 0.02%)</title><rect x="263.4" y="1941" width="0.3" height="15.0" fill="rgb(215,135,14)" rx="2" ry="2" />
<text x="266.37" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1413" width="364.1" height="15.0" fill="rgb(213,210,19)" rx="2" ry="2" />
<text x="790.11" y="1423.5" >[unknown]</text>
</g>
<g >
<title>selinux_socket_sendmsg (109 samples, 0.04%)</title><rect x="481.2" y="1877" width="0.4" height="15.0" fill="rgb(206,199,23)" rx="2" ry="2" />
<text x="484.17" y="1887.5" ></text>
</g>
<g >
<title>vfs_write (28 samples, 0.01%)</title><rect x="715.5" y="1877" width="0.2" height="15.0" fill="rgb(230,125,46)" rx="2" ry="2" />
<text x="718.53" y="1887.5" ></text>
</g>
<g >
<title>[unknown] (88,032 samples, 30.86%)</title><rect x="787.1" y="645" width="364.1" height="15.0" fill="rgb(208,129,20)" rx="2" ry="2" />
<text x="790.11" y="655.5" >[unknown]</text>
</g>
<g >
<title>[nginx] (386 samples, 0.14%)</title><rect x="44.8" y="2021" width="1.6" height="15.0" fill="rgb(252,152,33)" rx="2" ry="2" />
<text x="47.83" y="2031.5" ></text>
</g>
<g >
<title>stats_record (208 samples, 0.07%)</title><rect x="793.6" y="37" width="0.9" height="15.0" fill="rgb(230,86,31)" rx="2" ry="2" />
<text x="796.60" y="47.5" ></text>
</g>
<g >
<title>do_sendfile (48,544 samples, 17.02%)</title><rect x="274.9" y="1957" width="200.8" height="15.0" fill="rgb(235,202,3)" rx="2" ry="2" />
<text x="277.91" y="1967.5" >do_sendfile</text>
</g>
<g >
<title>smc_tx_sendmsg (475 samples, 0.17%)</title><rect x="1144.7" y="165" width="2.0" height="15.0" fill="rgb(232,210,25)" rx="2" ry="2" />
<text x="1147.74" y="175.5" ></text>
</g>
<g >
<title>mutex_lock (102 samples, 0.04%)</title><rect x="1179.5" y="1941" width="0.4" height="15.0" fill="rgb(252,129,54)" rx="2" ry="2" />
<text x="1182.47" y="1951.5" ></text>
</g>
<g >
<title>smc_close_active (30 samples, 0.01%)</title><rect x="703.7" y="1909" width="0.1" height="15.0" fill="rgb(238,174,3)" rx="2" ry="2" />
<text x="706.71" y="1919.5" ></text>
</g>
<g >
<title>kvm_kick_cpu (332 samples, 0.12%)</title><rect x="240.1" y="1733" width="1.4" height="15.0" fill="rgb(229,143,39)" rx="2" ry="2" />
<text x="243.11" y="1743.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (69 samples, 0.02%)</title><rect x="809.9" y="53" width="0.3" height="15.0" fill="rgb(215,197,8)" rx="2" ry="2" />
<text x="812.93" y="63.5" ></text>
</g>
<g >
<title>sock_recvmsg (452 samples, 0.16%)</title><rect x="807.6" y="69" width="1.9" height="15.0" fill="rgb(232,47,23)" rx="2" ry="2" />
<text x="810.63" y="79.5" ></text>
</g>
<g >
<title>_copy_to_user (99 samples, 0.03%)</title><rect x="650.4" y="1957" width="0.4" height="15.0" fill="rgb(238,132,16)" rx="2" ry="2" />
<text x="653.42" y="1967.5" ></text>
</g>
<g >
<title>smc_rx_update_consumer (26 samples, 0.01%)</title><rect x="816.4" y="53" width="0.2" height="15.0" fill="rgb(220,63,31)" rx="2" ry="2" />
<text x="819.45" y="63.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (362 samples, 0.13%)</title><rect x="696.0" y="1877" width="1.5" height="15.0" fill="rgb(223,1,29)" rx="2" ry="2" />
<text x="698.99" y="1887.5" ></text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (134 samples, 0.05%)</title><rect x="1146.2" y="133" width="0.5" height="15.0" fill="rgb(235,55,17)" rx="2" ry="2" />
<text x="1149.15" y="143.5" ></text>
</g>
<g >
<title>ep_modify (241 samples, 0.08%)</title><rect x="1169.1" y="1973" width="1.0" height="15.0" fill="rgb(226,184,43)" rx="2" ry="2" />
<text x="1172.10" y="1983.5" ></text>
</g>
<g >
<title>_copy_from_iter (201 samples, 0.07%)</title><rect x="644.1" y="1861" width="0.8" height="15.0" fill="rgb(206,104,40)" rx="2" ry="2" />
<text x="647.07" y="1871.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (240 samples, 0.08%)</title><rect x="708.9" y="2005" width="1.0" height="15.0" fill="rgb(253,82,41)" rx="2" ry="2" />
<text x="711.88" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (5,616 samples, 1.97%)</title><rect x="787.1" y="181" width="23.2" height="15.0" fill="rgb(227,108,22)" rx="2" ry="2" />
<text x="790.11" y="191.5" >[..</text>
</g>
<g >
<title>__do_softirq (120 samples, 0.04%)</title><rect x="779.2" y="1893" width="0.5" height="15.0" fill="rgb(221,30,28)" rx="2" ry="2" />
<text x="782.17" y="1903.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (417 samples, 0.15%)</title><rect x="240.1" y="1797" width="1.7" height="15.0" fill="rgb(243,35,15)" rx="2" ry="2" />
<text x="243.08" y="1807.5" ></text>
</g>
<g >
<title>irqentry_exit_to_user_mode (43 samples, 0.02%)</title><rect x="40.6" y="2005" width="0.1" height="15.0" fill="rgb(213,69,21)" rx="2" ry="2" />
<text x="43.56" y="2015.5" ></text>
</g>
<g >
<title>mlock_drain_local (34 samples, 0.01%)</title><rect x="470.7" y="1877" width="0.2" height="15.0" fill="rgb(217,203,49)" rx="2" ry="2" />
<text x="473.73" y="1887.5" ></text>
</g>
<g >
<title>security_socket_recvmsg (190 samples, 0.07%)</title><rect x="807.6" y="53" width="0.8" height="15.0" fill="rgb(214,177,38)" rx="2" ry="2" />
<text x="810.63" y="63.5" ></text>
</g>
<g >
<title>__vdso_time (112 samples, 0.04%)</title><rect x="1161.9" y="2053" width="0.4" height="15.0" fill="rgb(239,127,12)" rx="2" ry="2" />
<text x="1164.85" y="2063.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (131 samples, 0.05%)</title><rect x="823.1" y="85" width="0.5" height="15.0" fill="rgb(227,66,25)" rx="2" ry="2" />
<text x="826.08" y="95.5" ></text>
</g>
<g >
<title>perf (44 samples, 0.02%)</title><rect x="715.5" y="2069" width="0.2" height="15.0" fill="rgb(241,168,10)" rx="2" ry="2" />
<text x="718.50" y="2079.5" ></text>
</g>
<g >
<title>generic_permission (63 samples, 0.02%)</title><rect x="693.5" y="1877" width="0.2" height="15.0" fill="rgb(243,17,6)" rx="2" ry="2" />
<text x="696.46" y="1887.5" ></text>
</g>
<g >
<title>release_pages (130 samples, 0.05%)</title><rect x="470.9" y="1877" width="0.5" height="15.0" fill="rgb(223,180,10)" rx="2" ry="2" />
<text x="473.87" y="1887.5" ></text>
</g>
<g >
<title>ngx_linux_sendfile_chain (247 samples, 0.09%)</title><rect x="268.4" y="2021" width="1.0" height="15.0" fill="rgb(238,11,48)" rx="2" ry="2" />
<text x="271.42" y="2031.5" ></text>
</g>
<g >
<title>smc_release (112 samples, 0.04%)</title><rect x="812.4" y="85" width="0.5" height="15.0" fill="rgb(250,220,11)" rx="2" ry="2" />
<text x="815.45" y="95.5" ></text>
</g>
<g >
<title>avc_lookup (117 samples, 0.04%)</title><rect x="252.5" y="1845" width="0.5" height="15.0" fill="rgb(247,183,36)" rx="2" ry="2" />
<text x="255.54" y="1855.5" ></text>
</g>
<g >
<title>.slowpath (381 samples, 0.13%)</title><rect x="1146.7" y="85" width="1.6" height="15.0" fill="rgb(222,224,6)" rx="2" ry="2" />
<text x="1149.71" y="95.5" ></text>
</g>
<g >
<title>default_idle (14,242 samples, 4.99%)</title><rect x="720.8" y="1957" width="58.9" height="15.0" fill="rgb(254,139,25)" rx="2" ry="2" />
<text x="723.76" y="1967.5" >defaul..</text>
</g>
<g >
<title>__do_softirq (25 samples, 0.01%)</title><rect x="463.0" y="1669" width="0.1" height="15.0" fill="rgb(221,131,17)" rx="2" ry="2" />
<text x="465.97" y="1679.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1989" width="364.1" height="15.0" fill="rgb(249,187,48)" rx="2" ry="2" />
<text x="790.11" y="1999.5" >[unknown]</text>
</g>
<g >
<title>obj_cgroup_charge (56 samples, 0.02%)</title><rect x="688.7" y="1893" width="0.3" height="15.0" fill="rgb(237,113,43)" rx="2" ry="2" />
<text x="691.73" y="1903.5" ></text>
</g>
<g >
<title>[libc.so.6] (243 samples, 0.09%)</title><rect x="43.8" y="2021" width="1.0" height="15.0" fill="rgb(226,64,0)" rx="2" ry="2" />
<text x="46.83" y="2031.5" ></text>
</g>
<g >
<title>release_sock (53 samples, 0.02%)</title><rect x="254.6" y="1877" width="0.2" height="15.0" fill="rgb(250,92,28)" rx="2" ry="2" />
<text x="257.55" y="1887.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (241 samples, 0.08%)</title><rect x="77.6" y="1909" width="1.0" height="15.0" fill="rgb(223,85,37)" rx="2" ry="2" />
<text x="80.58" y="1919.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (134 samples, 0.05%)</title><rect x="1146.2" y="53" width="0.5" height="15.0" fill="rgb(232,169,32)" rx="2" ry="2" />
<text x="1149.15" y="63.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="2037" width="364.1" height="15.0" fill="rgb(208,112,16)" rx="2" ry="2" />
<text x="790.11" y="2047.5" >[unknown]</text>
</g>
<g >
<title>ksys_read (888 samples, 0.31%)</title><rect x="838.9" y="213" width="3.7" height="15.0" fill="rgb(219,227,18)" rx="2" ry="2" />
<text x="841.89" y="223.5" ></text>
</g>
<g >
<title>__import_iovec (73 samples, 0.03%)</title><rect x="646.4" y="1925" width="0.3" height="15.0" fill="rgb(251,142,32)" rx="2" ry="2" />
<text x="649.44" y="1935.5" ></text>
</g>
<g >
<title>cpuidle_idle_call (14,264 samples, 5.00%)</title><rect x="720.7" y="1989" width="59.0" height="15.0" fill="rgb(209,64,6)" rx="2" ry="2" />
<text x="723.67" y="1999.5" >cpuidl..</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="997" width="364.1" height="15.0" fill="rgb(219,3,23)" rx="2" ry="2" />
<text x="790.11" y="1007.5" >[unknown]</text>
</g>
<g >
<title>ep_send_events (1,047 samples, 0.37%)</title><rect x="1175.6" y="1957" width="4.4" height="15.0" fill="rgb(244,146,24)" rx="2" ry="2" />
<text x="1178.64" y="1967.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,530 samples, 0.54%)</title><rect x="260.2" y="1989" width="6.3" height="15.0" fill="rgb(253,23,12)" rx="2" ry="2" />
<text x="263.20" y="1999.5" ></text>
</g>
<g >
<title>ngx_http_time (64 samples, 0.02%)</title><rect x="677.7" y="2037" width="0.2" height="15.0" fill="rgb(220,132,41)" rx="2" ry="2" />
<text x="680.67" y="2047.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1477" width="364.1" height="15.0" fill="rgb(225,26,38)" rx="2" ry="2" />
<text x="790.11" y="1487.5" >[unknown]</text>
</g>
<g >
<title>__fput (637 samples, 0.22%)</title><rect x="260.3" y="1957" width="2.6" height="15.0" fill="rgb(223,113,3)" rx="2" ry="2" />
<text x="263.29" y="1967.5" ></text>
</g>
<g >
<title>smc_sendmsg (36,168 samples, 12.68%)</title><rect x="842.6" y="165" width="149.6" height="15.0" fill="rgb(206,203,5)" rx="2" ry="2" />
<text x="845.56" y="175.5" >smc_sendmsg</text>
</g>
<g >
<title>__fget_light (177 samples, 0.06%)</title><rect x="1154.4" y="1957" width="0.7" height="15.0" fill="rgb(208,118,50)" rx="2" ry="2" />
<text x="1157.36" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="965" width="364.1" height="15.0" fill="rgb(244,175,21)" rx="2" ry="2" />
<text x="790.11" y="975.5" >[unknown]</text>
</g>
<g >
<title>do_idle (14,884 samples, 5.22%)</title><rect x="720.6" y="2005" width="61.6" height="15.0" fill="rgb(232,63,51)" rx="2" ry="2" />
<text x="723.61" y="2015.5" >do_idle</text>
</g>
<g >
<title>vfs_read (254 samples, 0.09%)</title><rect x="810.3" y="117" width="1.1" height="15.0" fill="rgb(238,39,13)" rx="2" ry="2" />
<text x="813.35" y="127.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="981" width="364.1" height="15.0" fill="rgb(225,169,27)" rx="2" ry="2" />
<text x="790.11" y="991.5" >[unknown]</text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (4,927 samples, 1.73%)</title><rect x="56.2" y="1813" width="20.3" height="15.0" fill="rgb(243,6,16)" rx="2" ry="2" />
<text x="59.17" y="1823.5" ></text>
</g>
<g >
<title>[unknown] (87,995 samples, 30.84%)</title><rect x="787.1" y="453" width="364.0" height="15.0" fill="rgb(216,86,9)" rx="2" ry="2" />
<text x="790.11" y="463.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1461" width="364.1" height="15.0" fill="rgb(211,119,53)" rx="2" ry="2" />
<text x="790.11" y="1471.5" >[unknown]</text>
</g>
<g >
<title>__d_lookup_rcu (130 samples, 0.05%)</title><rect x="665.3" y="1877" width="0.5" height="15.0" fill="rgb(239,11,52)" rx="2" ry="2" />
<text x="668.27" y="1887.5" ></text>
</g>
<g >
<title>_raw_spin_lock (129 samples, 0.05%)</title><rect x="822.3" y="37" width="0.5" height="15.0" fill="rgb(222,214,31)" rx="2" ry="2" />
<text x="825.30" y="47.5" ></text>
</g>
<g >
<title>__fget_light (78 samples, 0.03%)</title><rect x="803.5" y="53" width="0.3" height="15.0" fill="rgb(254,83,17)" rx="2" ry="2" />
<text x="806.50" y="63.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1685" width="364.1" height="15.0" fill="rgb(220,11,43)" rx="2" ry="2" />
<text x="790.11" y="1695.5" >[unknown]</text>
</g>
<g >
<title>__legitimize_mnt (69 samples, 0.02%)</title><rect x="689.2" y="1861" width="0.3" height="15.0" fill="rgb(207,38,16)" rx="2" ry="2" />
<text x="692.17" y="1871.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (1,116 samples, 0.39%)</title><rect x="829.1" y="37" width="4.6" height="15.0" fill="rgb(235,88,4)" rx="2" ry="2" />
<text x="832.08" y="47.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (37 samples, 0.01%)</title><rect x="666.7" y="1893" width="0.2" height="15.0" fill="rgb(249,102,45)" rx="2" ry="2" />
<text x="669.73" y="1903.5" ></text>
</g>
<g >
<title>getname_flags.part.0 (635 samples, 0.22%)</title><rect x="651.2" y="1957" width="2.7" height="15.0" fill="rgb(218,29,2)" rx="2" ry="2" />
<text x="654.25" y="1967.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (189 samples, 0.07%)</title><rect x="822.3" y="101" width="0.8" height="15.0" fill="rgb(236,63,37)" rx="2" ry="2" />
<text x="825.30" y="111.5" ></text>
</g>
<g >
<title>execve (29 samples, 0.01%)</title><rect x="716.5" y="2037" width="0.1" height="15.0" fill="rgb(252,60,26)" rx="2" ry="2" />
<text x="719.45" y="2047.5" ></text>
</g>
<g >
<title>ep_poll (2,710 samples, 0.95%)</title><rect x="1175.2" y="1973" width="11.2" height="15.0" fill="rgb(211,137,52)" rx="2" ry="2" />
<text x="1178.19" y="1983.5" ></text>
</g>
<g >
<title>smc_recvmsg (38,305 samples, 13.43%)</title><rect x="84.5" y="1877" width="158.4" height="15.0" fill="rgb(237,78,4)" rx="2" ry="2" />
<text x="87.47" y="1887.5" >smc_recvmsg</text>
</g>
<g >
<title>vfs_getattr_nosec (85 samples, 0.03%)</title><rect x="708.1" y="1973" width="0.3" height="15.0" fill="rgb(221,57,51)" rx="2" ry="2" />
<text x="711.09" y="1983.5" ></text>
</g>
<g >
<title>__pv_queued_spin_unlock_slowpath (349 samples, 0.12%)</title><rect x="642.2" y="1765" width="1.4" height="15.0" fill="rgb(239,72,9)" rx="2" ry="2" />
<text x="645.17" y="1775.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (473 samples, 0.17%)</title><rect x="1148.7" y="293" width="1.9" height="15.0" fill="rgb(226,217,27)" rx="2" ry="2" />
<text x="1151.67" y="303.5" ></text>
</g>
<g >
<title>do_epoll_wait (2,947 samples, 1.03%)</title><rect x="1174.5" y="1989" width="12.1" height="15.0" fill="rgb(232,104,41)" rx="2" ry="2" />
<text x="1177.46" y="1999.5" ></text>
</g>
<g >
<title>lockref_get_not_dead (40 samples, 0.01%)</title><rect x="655.8" y="1861" width="0.1" height="15.0" fill="rgb(242,181,48)" rx="2" ry="2" />
<text x="658.76" y="1871.5" ></text>
</g>
<g >
<title>sock_recvmsg (131 samples, 0.05%)</title><rect x="823.1" y="133" width="0.5" height="15.0" fill="rgb(210,64,44)" rx="2" ry="2" />
<text x="826.08" y="143.5" ></text>
</g>
<g >
<title>ngx_list_push (28 samples, 0.01%)</title><rect x="678.7" y="2037" width="0.1" height="15.0" fill="rgb(211,27,11)" rx="2" ry="2" />
<text x="681.68" y="2047.5" ></text>
</g>
<g >
<title>acl_permission_check (83 samples, 0.03%)</title><rect x="694.9" y="1877" width="0.3" height="15.0" fill="rgb(235,18,4)" rx="2" ry="2" />
<text x="697.88" y="1887.5" ></text>
</g>
<g >
<title>write (1,169 samples, 0.41%)</title><rect x="834.1" y="245" width="4.8" height="15.0" fill="rgb(254,52,45)" rx="2" ry="2" />
<text x="837.05" y="255.5" ></text>
</g>
<g >
<title>asm_sysvec_call_function_single (43 samples, 0.02%)</title><rect x="40.6" y="2021" width="0.1" height="15.0" fill="rgb(248,193,15)" rx="2" ry="2" />
<text x="43.56" y="2031.5" ></text>
</g>
<g >
<title>smc_poll (58 samples, 0.02%)</title><rect x="1169.9" y="1925" width="0.2" height="15.0" fill="rgb(222,106,14)" rx="2" ry="2" />
<text x="1172.86" y="1935.5" ></text>
</g>
<g >
<title>touch_atime (164 samples, 0.06%)</title><rect x="473.1" y="1893" width="0.7" height="15.0" fill="rgb(235,111,8)" rx="2" ry="2" />
<text x="476.09" y="1903.5" ></text>
</g>
<g >
<title>__schedule (1,077 samples, 0.38%)</title><rect x="1181.5" y="1925" width="4.4" height="15.0" fill="rgb(210,222,40)" rx="2" ry="2" />
<text x="1184.47" y="1935.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (451 samples, 0.16%)</title><rect x="642.1" y="1813" width="1.9" height="15.0" fill="rgb(242,88,50)" rx="2" ry="2" />
<text x="645.15" y="1823.5" ></text>
</g>
<g >
<title>cp_new_stat (90 samples, 0.03%)</title><rect x="706.0" y="1989" width="0.3" height="15.0" fill="rgb(250,216,35)" rx="2" ry="2" />
<text x="708.97" y="1999.5" ></text>
</g>
<g >
<title>ngx_http_run_posted_requests (26 samples, 0.01%)</title><rect x="676.4" y="2037" width="0.1" height="15.0" fill="rgb(217,160,18)" rx="2" ry="2" />
<text x="679.42" y="2047.5" ></text>
</g>
<g >
<title>walk_component (134 samples, 0.05%)</title><rect x="665.3" y="1909" width="0.5" height="15.0" fill="rgb(209,144,53)" rx="2" ry="2" />
<text x="668.25" y="1919.5" ></text>
</g>
<g >
<title>check_heap_object (36,893 samples, 12.93%)</title><rect x="992.2" y="117" width="152.5" height="15.0" fill="rgb(217,8,53)" rx="2" ry="2" />
<text x="995.15" y="127.5" >check_heap_object</text>
</g>
<g >
<title>syscall_exit_to_user_mode (31 samples, 0.01%)</title><rect x="810.2" y="117" width="0.1" height="15.0" fill="rgb(236,201,22)" rx="2" ry="2" />
<text x="813.21" y="127.5" ></text>
</g>
<g >
<title>ngx_hash_find (233 samples, 0.08%)</title><rect x="670.2" y="2037" width="0.9" height="15.0" fill="rgb(246,40,40)" rx="2" ry="2" />
<text x="673.18" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (168 samples, 0.06%)</title><rect x="809.6" y="133" width="0.7" height="15.0" fill="rgb(253,67,14)" rx="2" ry="2" />
<text x="812.65" y="143.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1445" width="364.1" height="15.0" fill="rgb(236,69,20)" rx="2" ry="2" />
<text x="790.11" y="1455.5" >[unknown]</text>
</g>
<g >
<title>link_path_walk.part.0 (998 samples, 0.35%)</title><rect x="694.1" y="1925" width="4.1" height="15.0" fill="rgb(205,64,35)" rx="2" ry="2" />
<text x="697.09" y="1935.5" ></text>
</g>
<g >
<title>smc_lo_move_data (86 samples, 0.03%)</title><rect x="1148.3" y="101" width="0.3" height="15.0" fill="rgb(211,10,33)" rx="2" ry="2" />
<text x="1151.28" y="111.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (473 samples, 0.17%)</title><rect x="1148.7" y="101" width="1.9" height="15.0" fill="rgb(216,126,44)" rx="2" ry="2" />
<text x="1151.67" y="111.5" ></text>
</g>
<g >
<title>pvclock_clocksource_read_nowd (27 samples, 0.01%)</title><rect x="1186.2" y="1909" width="0.1" height="15.0" fill="rgb(236,77,51)" rx="2" ry="2" />
<text x="1189.22" y="1919.5" ></text>
</g>
<g >
<title>__fget_light (153 samples, 0.05%)</title><rect x="76.9" y="1861" width="0.7" height="15.0" fill="rgb(229,127,19)" rx="2" ry="2" />
<text x="79.95" y="1871.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1669" width="364.1" height="15.0" fill="rgb(250,90,7)" rx="2" ry="2" />
<text x="790.11" y="1679.5" >[unknown]</text>
</g>
<g >
<title>syscall_enter_from_user_mode (232 samples, 0.08%)</title><rect x="646.8" y="1973" width="0.9" height="15.0" fill="rgb(217,88,34)" rx="2" ry="2" />
<text x="649.76" y="1983.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (381 samples, 0.13%)</title><rect x="1146.7" y="277" width="1.6" height="15.0" fill="rgb(222,88,22)" rx="2" ry="2" />
<text x="1149.71" y="287.5" ></text>
</g>
<g >
<title>security_inode_permission (548 samples, 0.19%)</title><rect x="695.2" y="1909" width="2.3" height="15.0" fill="rgb(230,94,0)" rx="2" ry="2" />
<text x="698.22" y="1919.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1397" width="364.1" height="15.0" fill="rgb(233,106,32)" rx="2" ry="2" />
<text x="790.11" y="1407.5" >[unknown]</text>
</g>
<g >
<title>smc_rx_recvmsg (1,201 samples, 0.42%)</title><rect x="829.1" y="117" width="5.0" height="15.0" fill="rgb(226,56,14)" rx="2" ry="2" />
<text x="832.08" y="127.5" ></text>
</g>
<g >
<title>read (888 samples, 0.31%)</title><rect x="838.9" y="261" width="3.7" height="15.0" fill="rgb(228,99,37)" rx="2" ry="2" />
<text x="841.89" y="271.5" ></text>
</g>
<g >
<title>ngx_rbtree_insert (68 samples, 0.02%)</title><rect x="270.6" y="2021" width="0.3" height="15.0" fill="rgb(230,14,45)" rx="2" ry="2" />
<text x="273.62" y="2031.5" ></text>
</g>
<g >
<title>__sock_release (112 samples, 0.04%)</title><rect x="812.4" y="101" width="0.5" height="15.0" fill="rgb(250,189,3)" rx="2" ry="2" />
<text x="815.45" y="111.5" ></text>
</g>
<g >
<title>ngx_tcp_push (192 samples, 0.07%)</title><rect x="49.2" y="1957" width="0.8" height="15.0" fill="rgb(217,149,24)" rx="2" ry="2" />
<text x="52.16" y="1967.5" ></text>
</g>
<g >
<title>vm_mmap_pgoff (45 samples, 0.02%)</title><rect x="719.1" y="1989" width="0.2" height="15.0" fill="rgb(236,127,51)" rx="2" ry="2" />
<text x="722.15" y="1999.5" ></text>
</g>
<g >
<title>pv_wait_node (42,110 samples, 14.76%)</title><rect x="288.9" y="1749" width="174.2" height="15.0" fill="rgb(210,176,31)" rx="2" ry="2" />
<text x="291.92" y="1759.5" >pv_wait_node</text>
</g>
<g >
<title>__lookup_mnt (38 samples, 0.01%)</title><rect x="662.4" y="1861" width="0.1" height="15.0" fill="rgb(237,152,8)" rx="2" ry="2" />
<text x="665.39" y="1871.5" ></text>
</g>
<g >
<title>smc_sendmsg (39,519 samples, 13.85%)</title><rect x="481.6" y="1893" width="163.5" height="15.0" fill="rgb(227,85,9)" rx="2" ry="2" />
<text x="484.62" y="1903.5" >smc_sendmsg</text>
</g>
<g >
<title>smc_sendmsg (769 samples, 0.27%)</title><rect x="816.6" y="101" width="3.1" height="15.0" fill="rgb(233,148,24)" rx="2" ry="2" />
<text x="819.56" y="111.5" ></text>
</g>
<g >
<title>sock_write_iter (475 samples, 0.17%)</title><rect x="1144.7" y="197" width="2.0" height="15.0" fill="rgb(244,192,29)" rx="2" ry="2" />
<text x="1147.74" y="207.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (48,998 samples, 17.17%)</title><rect x="274.5" y="2005" width="202.6" height="15.0" fill="rgb(215,226,41)" rx="2" ry="2" />
<text x="277.49" y="2015.5" >entry_SYSCALL_64</text>
</g>
<g >
<title>vfs_write (105 samples, 0.04%)</title><rect x="799.7" y="37" width="0.4" height="15.0" fill="rgb(247,217,51)" rx="2" ry="2" />
<text x="802.68" y="47.5" ></text>
</g>
<g >
<title>_raw_spin_lock (37,693 samples, 13.21%)</title><rect x="486.3" y="1813" width="155.8" height="15.0" fill="rgb(222,152,40)" rx="2" ry="2" />
<text x="489.25" y="1823.5" >_raw_spin_lock</text>
</g>
<g >
<title>_raw_spin_lock (36,142 samples, 12.67%)</title><rect x="842.6" y="85" width="149.4" height="15.0" fill="rgb(245,86,27)" rx="2" ry="2" />
<text x="845.56" y="95.5" >_raw_spin_lock</text>
</g>
<g >
<title>do_syscall_64 (888 samples, 0.31%)</title><rect x="838.9" y="229" width="3.7" height="15.0" fill="rgb(254,143,38)" rx="2" ry="2" />
<text x="841.89" y="239.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="917" width="364.1" height="15.0" fill="rgb(213,61,12)" rx="2" ry="2" />
<text x="790.11" y="927.5" >[unknown]</text>
</g>
<g >
<title>pv_wait_head_or_lock (58 samples, 0.02%)</title><rect x="838.9" y="37" width="0.3" height="15.0" fill="rgb(218,53,49)" rx="2" ry="2" />
<text x="841.92" y="47.5" ></text>
</g>
<g >
<title>[unknown] (50 samples, 0.02%)</title><rect x="10.1" y="2053" width="0.2" height="15.0" fill="rgb(225,64,23)" rx="2" ry="2" />
<text x="13.13" y="2063.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="741" width="364.1" height="15.0" fill="rgb(238,79,9)" rx="2" ry="2" />
<text x="790.11" y="751.5" >[unknown]</text>
</g>
<g >
<title>do_syscall_64 (473 samples, 0.17%)</title><rect x="1148.7" y="277" width="1.9" height="15.0" fill="rgb(235,59,12)" rx="2" ry="2" />
<text x="1151.67" y="287.5" ></text>
</g>
<g >
<title>[unknown] (88,026 samples, 30.85%)</title><rect x="787.1" y="613" width="364.1" height="15.0" fill="rgb(217,119,20)" rx="2" ry="2" />
<text x="790.11" y="623.5" >[unknown]</text>
</g>
<g >
<title>selinux_socket_setsockopt (264 samples, 0.09%)</title><rect x="52.2" y="1845" width="1.1" height="15.0" fill="rgb(246,161,26)" rx="2" ry="2" />
<text x="55.23" y="1855.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (38,066 samples, 13.34%)</title><rect x="85.5" y="1861" width="157.4" height="15.0" fill="rgb(216,7,17)" rx="2" ry="2" />
<text x="88.46" y="1871.5" >smc_rx_recvmsg</text>
</g>
<g >
<title>filemap_get_pages (297 samples, 0.10%)</title><rect x="471.4" y="1893" width="1.2" height="15.0" fill="rgb(227,37,40)" rx="2" ry="2" />
<text x="474.41" y="1903.5" ></text>
</g>
<g >
<title>schedule (100 samples, 0.04%)</title><rect x="647.7" y="1925" width="0.5" height="15.0" fill="rgb(239,199,1)" rx="2" ry="2" />
<text x="650.74" y="1935.5" ></text>
</g>
<g >
<title>ksys_write (1,169 samples, 0.41%)</title><rect x="834.1" y="197" width="4.8" height="15.0" fill="rgb(211,166,40)" rx="2" ry="2" />
<text x="837.05" y="207.5" ></text>
</g>
<g >
<title>avc_lookup (102 samples, 0.04%)</title><rect x="52.9" y="1797" width="0.4" height="15.0" fill="rgb(207,140,16)" rx="2" ry="2" />
<text x="55.90" y="1807.5" ></text>
</g>
<g >
<title>_Fork (84 samples, 0.03%)</title><rect x="716.7" y="2053" width="0.4" height="15.0" fill="rgb(252,12,54)" rx="2" ry="2" />
<text x="719.73" y="2063.5" ></text>
</g>
<g >
<title>pmsleep (284 samples, 0.10%)</title><rect x="718.9" y="2069" width="1.2" height="15.0" fill="rgb(205,4,37)" rx="2" ry="2" />
<text x="721.94" y="2079.5" ></text>
</g>
<g >
<title>[unknown] (87,993 samples, 30.84%)</title><rect x="787.1" y="437" width="363.9" height="15.0" fill="rgb(229,74,50)" rx="2" ry="2" />
<text x="790.11" y="447.5" >[unknown]</text>
</g>
<g >
<title>kernel_clone (74 samples, 0.03%)</title><rect x="716.7" y="1989" width="0.3" height="15.0" fill="rgb(224,222,13)" rx="2" ry="2" />
<text x="719.73" y="1999.5" ></text>
</g>
<g >
<title>[perf] (35 samples, 0.01%)</title><rect x="715.5" y="1973" width="0.2" height="15.0" fill="rgb(212,105,24)" rx="2" ry="2" />
<text x="718.51" y="1983.5" ></text>
</g>
<g >
<title>kvm_clock_get_cycles (131 samples, 0.05%)</title><rect x="1187.1" y="1973" width="0.6" height="15.0" fill="rgb(207,63,48)" rx="2" ry="2" />
<text x="1190.13" y="1983.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (27 samples, 0.01%)</title><rect x="719.8" y="2005" width="0.1" height="15.0" fill="rgb(222,184,22)" rx="2" ry="2" />
<text x="722.78" y="2015.5" ></text>
</g>
<g >
<title>__legitimize_path (107 samples, 0.04%)</title><rect x="689.2" y="1877" width="0.4" height="15.0" fill="rgb(222,154,46)" rx="2" ry="2" />
<text x="692.17" y="1887.5" ></text>
</g>
<g >
<title>release_sock (42 samples, 0.01%)</title><rect x="811.0" y="53" width="0.1" height="15.0" fill="rgb(233,39,45)" rx="2" ry="2" />
<text x="813.97" y="63.5" ></text>
</g>
<g >
<title>find_vmap_area (381 samples, 0.13%)</title><rect x="1146.7" y="117" width="1.6" height="15.0" fill="rgb(250,90,21)" rx="2" ry="2" />
<text x="1149.71" y="127.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv_action (3,981 samples, 1.40%)</title><rect x="60.1" y="1717" width="16.4" height="15.0" fill="rgb(229,156,53)" rx="2" ry="2" />
<text x="63.08" y="1727.5" ></text>
</g>
<g >
<title>[perf] (38 samples, 0.01%)</title><rect x="715.5" y="2005" width="0.2" height="15.0" fill="rgb(236,126,5)" rx="2" ry="2" />
<text x="718.50" y="2015.5" ></text>
</g>
<g >
<title>sock_write_iter (86 samples, 0.03%)</title><rect x="1148.3" y="213" width="0.3" height="15.0" fill="rgb(243,65,32)" rx="2" ry="2" />
<text x="1151.28" y="223.5" ></text>
</g>
<g >
<title>http_parser_init (32 samples, 0.01%)</title><rect x="788.8" y="37" width="0.2" height="15.0" fill="rgb(210,195,50)" rx="2" ry="2" />
<text x="791.82" y="47.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (245 samples, 0.09%)</title><rect x="797.2" y="69" width="1.0" height="15.0" fill="rgb(209,150,2)" rx="2" ry="2" />
<text x="800.21" y="79.5" ></text>
</g>
<g >
<title>epoll_ctl (1,232 samples, 0.43%)</title><rect x="1151.9" y="2037" width="5.1" height="15.0" fill="rgb(232,154,19)" rx="2" ry="2" />
<text x="1154.92" y="2047.5" ></text>
</g>
<g >
<title>sysvec_call_function_single (120 samples, 0.04%)</title><rect x="779.2" y="1925" width="0.5" height="15.0" fill="rgb(229,37,14)" rx="2" ry="2" />
<text x="782.17" y="1935.5" ></text>
</g>
<g >
<title>__fget_light (119 samples, 0.04%)</title><rect x="802.0" y="53" width="0.5" height="15.0" fill="rgb(232,18,49)" rx="2" ry="2" />
<text x="804.97" y="63.5" ></text>
</g>
<g >
<title>ngx_output_chain (45 samples, 0.02%)</title><rect x="269.5" y="2021" width="0.1" height="15.0" fill="rgb(209,138,20)" rx="2" ry="2" />
<text x="272.46" y="2031.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (39,368 samples, 13.80%)</title><rect x="82.6" y="1957" width="162.8" height="15.0" fill="rgb(245,44,42)" rx="2" ry="2" />
<text x="85.59" y="1967.5" >entry_SYSCALL_64</text>
</g>
<g >
<title>do_syscall_64 (477 samples, 0.17%)</title><rect x="805.6" y="117" width="2.0" height="15.0" fill="rgb(253,39,34)" rx="2" ry="2" />
<text x="808.63" y="127.5" ></text>
</g>
<g >
<title>__x64_sys_close (39 samples, 0.01%)</title><rect x="703.7" y="2005" width="0.1" height="15.0" fill="rgb(225,30,17)" rx="2" ry="2" />
<text x="706.68" y="2015.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (97 samples, 0.03%)</title><rect x="1181.0" y="1925" width="0.4" height="15.0" fill="rgb(207,86,21)" rx="2" ry="2" />
<text x="1184.02" y="1935.5" ></text>
</g>
<g >
<title>socket_writeable (79 samples, 0.03%)</title><rect x="796.8" y="53" width="0.3" height="15.0" fill="rgb(206,143,49)" rx="2" ry="2" />
<text x="799.82" y="63.5" ></text>
</g>
<g >
<title>ps (64 samples, 0.02%)</title><rect x="720.1" y="2069" width="0.3" height="15.0" fill="rgb(246,148,37)" rx="2" ry="2" />
<text x="723.12" y="2079.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (506 samples, 0.18%)</title><rect x="463.1" y="1781" width="2.1" height="15.0" fill="rgb(226,100,16)" rx="2" ry="2" />
<text x="466.08" y="1791.5" ></text>
</g>
<g >
<title>setsockopt (2,132 samples, 0.75%)</title><rect x="249.5" y="2005" width="8.8" height="15.0" fill="rgb(214,214,45)" rx="2" ry="2" />
<text x="252.49" y="2015.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (74 samples, 0.03%)</title><rect x="645.2" y="1877" width="0.3" height="15.0" fill="rgb(238,150,1)" rx="2" ry="2" />
<text x="648.22" y="1887.5" ></text>
</g>
<g >
<title>check_heap_object (861 samples, 0.30%)</title><rect x="834.1" y="101" width="3.5" height="15.0" fill="rgb(216,65,51)" rx="2" ry="2" />
<text x="837.05" y="111.5" ></text>
</g>
<g >
<title>read (131 samples, 0.05%)</title><rect x="823.1" y="229" width="0.5" height="15.0" fill="rgb(228,22,42)" rx="2" ry="2" />
<text x="826.08" y="239.5" ></text>
</g>
<g >
<title>accept4 (68 samples, 0.02%)</title><rect x="648.3" y="2037" width="0.3" height="15.0" fill="rgb(207,221,17)" rx="2" ry="2" />
<text x="651.28" y="2047.5" ></text>
</g>
<g >
<title>avc_lookup (27 samples, 0.01%)</title><rect x="474.7" y="1877" width="0.1" height="15.0" fill="rgb(240,153,13)" rx="2" ry="2" />
<text x="477.72" y="1887.5" ></text>
</g>
<g >
<title>step_into (29 samples, 0.01%)</title><rect x="697.5" y="1909" width="0.1" height="15.0" fill="rgb(213,181,31)" rx="2" ry="2" />
<text x="700.49" y="1919.5" ></text>
</g>
<g >
<title>vfs_read (207 samples, 0.07%)</title><rect x="804.8" y="85" width="0.8" height="15.0" fill="rgb(206,43,53)" rx="2" ry="2" />
<text x="807.76" y="95.5" ></text>
</g>
<g >
<title>ngx_http_core_generic_phase (42 samples, 0.01%)</title><rect x="40.9" y="2021" width="0.2" height="15.0" fill="rgb(210,221,53)" rx="2" ry="2" />
<text x="43.88" y="2031.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (254 samples, 0.09%)</title><rect x="810.3" y="165" width="1.1" height="15.0" fill="rgb(222,185,14)" rx="2" ry="2" />
<text x="813.35" y="175.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1061" width="364.1" height="15.0" fill="rgb(247,139,0)" rx="2" ry="2" />
<text x="790.11" y="1071.5" >[unknown]</text>
</g>
<g >
<title>smc_tx_sendmsg (214 samples, 0.08%)</title><rect x="811.6" y="69" width="0.8" height="15.0" fill="rgb(210,53,30)" rx="2" ry="2" />
<text x="814.56" y="79.5" ></text>
</g>
<g >
<title>sendfile (49,014 samples, 17.18%)</title><rect x="274.4" y="2021" width="202.7" height="15.0" fill="rgb(222,219,26)" rx="2" ry="2" />
<text x="277.42" y="2031.5" >sendfile</text>
</g>
<g >
<title>entry_SYSCALL_64 (4,494 samples, 1.58%)</title><rect x="650.2" y="2021" width="18.6" height="15.0" fill="rgb(234,173,9)" rx="2" ry="2" />
<text x="653.22" y="2031.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1541" width="364.1" height="15.0" fill="rgb(232,203,12)" rx="2" ry="2" />
<text x="790.11" y="1551.5" >[unknown]</text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (36,142 samples, 12.67%)</title><rect x="842.6" y="69" width="149.4" height="15.0" fill="rgb(212,94,33)" rx="2" ry="2" />
<text x="845.56" y="79.5" >__pv_queued_spin_lo..</text>
</g>
<g >
<title>memcpy_orig (36 samples, 0.01%)</title><rect x="838.1" y="37" width="0.2" height="15.0" fill="rgb(206,17,12)" rx="2" ry="2" />
<text x="841.14" y="47.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (860 samples, 0.30%)</title><rect x="834.1" y="53" width="3.5" height="15.0" fill="rgb(247,66,42)" rx="2" ry="2" />
<text x="837.06" y="63.5" ></text>
</g>
<g >
<title>start_kernel (892 samples, 0.31%)</title><rect x="782.2" y="2005" width="3.7" height="15.0" fill="rgb(213,129,22)" rx="2" ry="2" />
<text x="785.17" y="2015.5" ></text>
</g>
<g >
<title>do_syscall_64 (40,843 samples, 14.32%)</title><rect x="479.3" y="1989" width="168.9" height="15.0" fill="rgb(211,197,31)" rx="2" ry="2" />
<text x="482.27" y="1999.5" >do_syscall_64</text>
</g>
<g >
<title>write (135 samples, 0.05%)</title><rect x="799.6" y="101" width="0.5" height="15.0" fill="rgb(220,181,43)" rx="2" ry="2" />
<text x="802.59" y="111.5" ></text>
</g>
<g >
<title>page_cache_pipe_buf_confirm (54 samples, 0.02%)</title><rect x="278.2" y="1877" width="0.2" height="15.0" fill="rgb(225,208,23)" rx="2" ry="2" />
<text x="281.19" y="1887.5" ></text>
</g>
<g >
<title>__inode_security_revalidate (212 samples, 0.07%)</title><rect x="658.6" y="1861" width="0.9" height="15.0" fill="rgb(253,76,41)" rx="2" ry="2" />
<text x="661.63" y="1871.5" ></text>
</g>
<g >
<title>cp_new_stat (179 samples, 0.06%)</title><rect x="650.3" y="1973" width="0.7" height="15.0" fill="rgb(238,152,18)" rx="2" ry="2" />
<text x="653.30" y="1983.5" ></text>
</g>
<g >
<title>ksys_write (137 samples, 0.05%)</title><rect x="809.6" y="117" width="0.6" height="15.0" fill="rgb(241,39,8)" rx="2" ry="2" />
<text x="812.65" y="127.5" ></text>
</g>
<g >
<title>[unknown] (7,564 samples, 2.65%)</title><rect x="47.6" y="1973" width="31.3" height="15.0" fill="rgb(242,131,20)" rx="2" ry="2" />
<text x="50.65" y="1983.5" >[u..</text>
</g>
<g >
<title>smcd_cdc_msg_send (134 samples, 0.05%)</title><rect x="1146.2" y="117" width="0.5" height="15.0" fill="rgb(214,220,27)" rx="2" ry="2" />
<text x="1149.15" y="127.5" ></text>
</g>
<g >
<title>do_iter_write (40,217 samples, 14.10%)</title><rect x="480.1" y="1941" width="166.3" height="15.0" fill="rgb(215,75,3)" rx="2" ry="2" />
<text x="483.10" y="1951.5" >do_iter_write</text>
</g>
<g >
<title>aeDeleteFileEvent (58 samples, 0.02%)</title><rect x="788.1" y="37" width="0.3" height="15.0" fill="rgb(254,142,54)" rx="2" ry="2" />
<text x="791.13" y="47.5" ></text>
</g>
<g >
<title>asm_sysvec_apic_timer_interrupt (30 samples, 0.01%)</title><rect x="642.0" y="1749" width="0.1" height="15.0" fill="rgb(235,127,44)" rx="2" ry="2" />
<text x="645.03" y="1759.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (37,539 samples, 13.16%)</title><rect x="486.9" y="1797" width="155.2" height="15.0" fill="rgb(227,78,36)" rx="2" ry="2" />
<text x="489.89" y="1807.5" >__pv_queued_spin_lo..</text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (159 samples, 0.06%)</title><rect x="819.1" y="53" width="0.6" height="15.0" fill="rgb(245,21,48)" rx="2" ry="2" />
<text x="822.09" y="63.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (1,201 samples, 0.42%)</title><rect x="829.1" y="101" width="5.0" height="15.0" fill="rgb(219,43,3)" rx="2" ry="2" />
<text x="832.08" y="111.5" ></text>
</g>
<g >
<title>step_into (68 samples, 0.02%)</title><rect x="662.3" y="1893" width="0.2" height="15.0" fill="rgb(217,212,47)" rx="2" ry="2" />
<text x="665.27" y="1903.5" ></text>
</g>
<g >
<title>from_kuid_munged (39 samples, 0.01%)</title><rect x="650.9" y="1957" width="0.1" height="15.0" fill="rgb(243,148,8)" rx="2" ry="2" />
<text x="653.88" y="1967.5" ></text>
</g>
<g >
<title>asm_sysvec_apic_timer_interrupt (25 samples, 0.01%)</title><rect x="463.0" y="1717" width="0.1" height="15.0" fill="rgb(223,52,27)" rx="2" ry="2" />
<text x="465.97" y="1727.5" ></text>
</g>
<g >
<title>dput (63 samples, 0.02%)</title><rect x="665.8" y="1925" width="0.3" height="15.0" fill="rgb(242,56,33)" rx="2" ry="2" />
<text x="668.85" y="1935.5" ></text>
</g>
<g >
<title>smc_lo_move_data (32 samples, 0.01%)</title><rect x="822.9" y="37" width="0.1" height="15.0" fill="rgb(210,97,6)" rx="2" ry="2" />
<text x="825.89" y="47.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (109 samples, 0.04%)</title><rect x="828.1" y="53" width="0.5" height="15.0" fill="rgb(227,119,21)" rx="2" ry="2" />
<text x="831.13" y="63.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv (4,032 samples, 1.41%)</title><rect x="59.9" y="1733" width="16.6" height="15.0" fill="rgb(215,60,8)" rx="2" ry="2" />
<text x="62.87" y="1743.5" ></text>
</g>
<g >
<title>smc_tx_rdma_writes (87 samples, 0.03%)</title><rect x="819.3" y="37" width="0.4" height="15.0" fill="rgb(234,21,38)" rx="2" ry="2" />
<text x="822.30" y="47.5" ></text>
</g>
<g >
<title>find_vmap_area (341 samples, 0.12%)</title><rect x="1144.7" y="117" width="1.5" height="15.0" fill="rgb(233,172,33)" rx="2" ry="2" />
<text x="1147.74" y="127.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (28 samples, 0.01%)</title><rect x="10.1" y="2021" width="0.2" height="15.0" fill="rgb(251,33,31)" rx="2" ry="2" />
<text x="13.14" y="2031.5" ></text>
</g>
<g >
<title>ext4_release_file (173 samples, 0.06%)</title><rect x="261.2" y="1941" width="0.8" height="15.0" fill="rgb(222,74,45)" rx="2" ry="2" />
<text x="264.24" y="1951.5" ></text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (122 samples, 0.04%)</title><rect x="828.6" y="85" width="0.5" height="15.0" fill="rgb(248,117,35)" rx="2" ry="2" />
<text x="831.58" y="95.5" ></text>
</g>
<g >
<title>security_inode_permission (1,051 samples, 0.37%)</title><rect x="657.9" y="1893" width="4.4" height="15.0" fill="rgb(226,171,53)" rx="2" ry="2" />
<text x="660.92" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1797" width="364.1" height="15.0" fill="rgb(228,24,53)" rx="2" ry="2" />
<text x="790.11" y="1807.5" >[unknown]</text>
</g>
<g >
<title>nd_jump_root (71 samples, 0.02%)</title><rect x="698.9" y="1909" width="0.3" height="15.0" fill="rgb(213,184,7)" rx="2" ry="2" />
<text x="701.88" y="1919.5" ></text>
</g>
<g >
<title>find_vmap_area (44,414 samples, 15.57%)</title><rect x="281.5" y="1797" width="183.7" height="15.0" fill="rgb(237,81,39)" rx="2" ry="2" />
<text x="284.48" y="1807.5" >find_vmap_area</text>
</g>
<g >
<title>exit_to_user_mode_loop (101 samples, 0.04%)</title><rect x="244.9" y="1893" width="0.5" height="15.0" fill="rgb(222,132,39)" rx="2" ry="2" />
<text x="247.94" y="1903.5" ></text>
</g>
<g >
<title>smc_tx_rdma_writes (32 samples, 0.01%)</title><rect x="822.9" y="53" width="0.1" height="15.0" fill="rgb(253,89,2)" rx="2" ry="2" />
<text x="825.89" y="63.5" ></text>
</g>
<g >
<title>sock_write_iter (1,169 samples, 0.41%)</title><rect x="834.1" y="165" width="4.8" height="15.0" fill="rgb(229,114,48)" rx="2" ry="2" />
<text x="837.05" y="175.5" ></text>
</g>
<g >
<title>ngx_http_parse_request_line (375 samples, 0.13%)</title><rect x="674.3" y="2037" width="1.6" height="15.0" fill="rgb(238,30,52)" rx="2" ry="2" />
<text x="677.31" y="2047.5" ></text>
</g>
<g >
<title>avc_lookup (31 samples, 0.01%)</title><rect x="811.4" y="37" width="0.1" height="15.0" fill="rgb(240,199,29)" rx="2" ry="2" />
<text x="814.42" y="47.5" ></text>
</g>
<g >
<title>ngx_chain_coalesce_file (34 samples, 0.01%)</title><rect x="267.4" y="2021" width="0.1" height="15.0" fill="rgb(222,153,46)" rx="2" ry="2" />
<text x="270.40" y="2031.5" ></text>
</g>
<g >
<title>ep_item_poll.isra.0 (180 samples, 0.06%)</title><rect x="246.1" y="1893" width="0.7" height="15.0" fill="rgb(205,42,44)" rx="2" ry="2" />
<text x="249.10" y="1903.5" ></text>
</g>
<g >
<title>sock_write_iter (36,168 samples, 12.68%)</title><rect x="842.6" y="181" width="149.6" height="15.0" fill="rgb(214,4,7)" rx="2" ry="2" />
<text x="845.56" y="191.5" >sock_write_iter</text>
</g>
<g >
<title>do_iter_readv_writev (39,740 samples, 13.93%)</title><rect x="480.7" y="1925" width="164.4" height="15.0" fill="rgb(249,107,13)" rx="2" ry="2" />
<text x="483.71" y="1935.5" >do_iter_readv_writev</text>
</g>
<g >
<title>entry_SYSCALL_64 (29 samples, 0.01%)</title><rect x="717.3" y="2037" width="0.1" height="15.0" fill="rgb(224,25,31)" rx="2" ry="2" />
<text x="720.30" y="2047.5" ></text>
</g>
<g >
<title>lockref_get (87 samples, 0.03%)</title><rect x="692.1" y="1893" width="0.3" height="15.0" fill="rgb(231,174,8)" rx="2" ry="2" />
<text x="695.06" y="1903.5" ></text>
</g>
<g >
<title>sock_read_iter (40 samples, 0.01%)</title><rect x="803.3" y="53" width="0.2" height="15.0" fill="rgb(218,188,9)" rx="2" ry="2" />
<text x="806.33" y="63.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (4,279 samples, 1.50%)</title><rect x="685.1" y="2021" width="17.7" height="15.0" fill="rgb(219,136,34)" rx="2" ry="2" />
<text x="688.13" y="2031.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (108 samples, 0.04%)</title><rect x="647.7" y="1941" width="0.5" height="15.0" fill="rgb(254,49,4)" rx="2" ry="2" />
<text x="650.73" y="1951.5" ></text>
</g>
<g >
<title>ep_item_poll.isra.0 (108 samples, 0.04%)</title><rect x="1169.7" y="1957" width="0.4" height="15.0" fill="rgb(249,102,44)" rx="2" ry="2" />
<text x="1172.65" y="1967.5" ></text>
</g>
<g >
<title>sock_poll (173 samples, 0.06%)</title><rect x="246.1" y="1877" width="0.7" height="15.0" fill="rgb(226,227,53)" rx="2" ry="2" />
<text x="249.13" y="1887.5" ></text>
</g>
<g >
<title>ksys_write (772 samples, 0.27%)</title><rect x="816.6" y="149" width="3.1" height="15.0" fill="rgb(219,47,50)" rx="2" ry="2" />
<text x="819.55" y="159.5" ></text>
</g>
<g >
<title>do_idle (892 samples, 0.31%)</title><rect x="782.2" y="1941" width="3.7" height="15.0" fill="rgb(214,156,36)" rx="2" ry="2" />
<text x="785.17" y="1951.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (74 samples, 0.03%)</title><rect x="718.1" y="2005" width="0.3" height="15.0" fill="rgb(223,38,40)" rx="2" ry="2" />
<text x="721.14" y="2015.5" ></text>
</g>
<g >
<title>acl_permission_check (144 samples, 0.05%)</title><rect x="657.3" y="1861" width="0.6" height="15.0" fill="rgb(252,222,53)" rx="2" ry="2" />
<text x="660.32" y="1871.5" ></text>
</g>
<g >
<title>kvm_wait.part.0 (36,890 samples, 12.93%)</title><rect x="992.2" y="37" width="152.5" height="15.0" fill="rgb(241,118,18)" rx="2" ry="2" />
<text x="995.15" y="47.5" >kvm_wait.part.0</text>
</g>
<g >
<title>[[vdso]] (87 samples, 0.03%)</title><rect x="1164.9" y="2037" width="0.3" height="15.0" fill="rgb(234,64,33)" rx="2" ry="2" />
<text x="1167.86" y="2047.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (229 samples, 0.08%)</title><rect x="797.3" y="37" width="0.9" height="15.0" fill="rgb(243,5,52)" rx="2" ry="2" />
<text x="800.26" y="47.5" ></text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (86 samples, 0.03%)</title><rect x="1148.3" y="149" width="0.3" height="15.0" fill="rgb(252,39,41)" rx="2" ry="2" />
<text x="1151.28" y="159.5" ></text>
</g>
<g >
<title>syscall_exit_to_user_mode (74 samples, 0.03%)</title><rect x="78.6" y="1909" width="0.3" height="15.0" fill="rgb(224,193,37)" rx="2" ry="2" />
<text x="81.58" y="1919.5" ></text>
</g>
<g >
<title>avc_has_perm (47 samples, 0.02%)</title><rect x="810.3" y="69" width="0.2" height="15.0" fill="rgb(213,222,28)" rx="2" ry="2" />
<text x="813.35" y="79.5" ></text>
</g>
<g >
<title>smc_sendmsg (214 samples, 0.08%)</title><rect x="811.6" y="85" width="0.8" height="15.0" fill="rgb(238,109,42)" rx="2" ry="2" />
<text x="814.56" y="95.5" ></text>
</g>
<g >
<title>read (490 samples, 0.17%)</title><rect x="789.1" y="37" width="2.0" height="15.0" fill="rgb(215,9,31)" rx="2" ry="2" />
<text x="792.11" y="47.5" ></text>
</g>
<g >
<title>[unknown] (87,948 samples, 30.83%)</title><rect x="787.1" y="373" width="363.8" height="15.0" fill="rgb(209,98,48)" rx="2" ry="2" />
<text x="790.11" y="383.5" >[unknown]</text>
</g>
<g >
<title>current_obj_cgroup (26 samples, 0.01%)</title><rect x="687.5" y="1893" width="0.2" height="15.0" fill="rgb(233,207,51)" rx="2" ry="2" />
<text x="690.55" y="1903.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (85 samples, 0.03%)</title><rect x="833.7" y="37" width="0.4" height="15.0" fill="rgb(207,140,0)" rx="2" ry="2" />
<text x="836.70" y="47.5" ></text>
</g>
<g >
<title>lookup_fast (127 samples, 0.04%)</title><rect x="698.3" y="1909" width="0.5" height="15.0" fill="rgb(232,136,22)" rx="2" ry="2" />
<text x="701.28" y="1919.5" ></text>
</g>
<g >
<title>smc_recvmsg (262 samples, 0.09%)</title><rect x="808.4" y="53" width="1.1" height="15.0" fill="rgb(247,68,37)" rx="2" ry="2" />
<text x="811.42" y="63.5" ></text>
</g>
<g >
<title>do_syscall_64 (245 samples, 0.09%)</title><rect x="797.2" y="53" width="1.0" height="15.0" fill="rgb(243,147,8)" rx="2" ry="2" />
<text x="800.21" y="63.5" ></text>
</g>
<g >
<title>pvclock_clocksource_read_nowd (98 samples, 0.03%)</title><rect x="1187.3" y="1957" width="0.4" height="15.0" fill="rgb(208,55,54)" rx="2" ry="2" />
<text x="1190.27" y="1967.5" ></text>
</g>
<g >
<title>file_has_perm (47 samples, 0.02%)</title><rect x="810.3" y="85" width="0.2" height="15.0" fill="rgb(236,36,33)" rx="2" ry="2" />
<text x="813.35" y="95.5" ></text>
</g>
<g >
<title>security_file_permission (50 samples, 0.02%)</title><rect x="276.7" y="1925" width="0.2" height="15.0" fill="rgb(230,162,50)" rx="2" ry="2" />
<text x="279.74" y="1935.5" ></text>
</g>
<g >
<title>file_has_perm (47 samples, 0.02%)</title><rect x="474.6" y="1925" width="0.2" height="15.0" fill="rgb(251,130,31)" rx="2" ry="2" />
<text x="477.65" y="1935.5" ></text>
</g>
<g >
<title>[[vdso]] (79 samples, 0.03%)</title><rect x="796.8" y="37" width="0.3" height="15.0" fill="rgb(207,106,6)" rx="2" ry="2" />
<text x="799.82" y="47.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (66 samples, 0.02%)</title><rect x="719.5" y="2005" width="0.3" height="15.0" fill="rgb(223,141,36)" rx="2" ry="2" />
<text x="722.48" y="2015.5" ></text>
</g>
<g >
<title>epoll_ctl (1,434 samples, 0.50%)</title><rect x="1165.5" y="2053" width="5.9" height="15.0" fill="rgb(232,185,33)" rx="2" ry="2" />
<text x="1168.51" y="2063.5" ></text>
</g>
<g >
<title>ngx_palloc (41 samples, 0.01%)</title><rect x="248.7" y="2005" width="0.2" height="15.0" fill="rgb(244,104,54)" rx="2" ry="2" />
<text x="251.70" y="2015.5" ></text>
</g>
<g >
<title>__sock_release (31 samples, 0.01%)</title><rect x="703.7" y="1957" width="0.1" height="15.0" fill="rgb(209,71,53)" rx="2" ry="2" />
<text x="706.71" y="1967.5" ></text>
</g>
<g >
<title>[nginx] (5,064 samples, 1.77%)</title><rect x="21.2" y="2037" width="21.0" height="15.0" fill="rgb(236,26,21)" rx="2" ry="2" />
<text x="24.22" y="2047.5" ></text>
</g>
<g >
<title>smc_lo_move_data (473 samples, 0.17%)</title><rect x="1148.7" y="117" width="1.9" height="15.0" fill="rgb(223,200,3)" rx="2" ry="2" />
<text x="1151.67" y="127.5" ></text>
</g>
<g >
<title>vfs_read (888 samples, 0.31%)</title><rect x="838.9" y="197" width="3.7" height="15.0" fill="rgb(241,26,26)" rx="2" ry="2" />
<text x="841.89" y="207.5" ></text>
</g>
<g >
<title>[unknown] (29 samples, 0.01%)</title><rect x="715.7" y="2053" width="0.2" height="15.0" fill="rgb(218,128,28)" rx="2" ry="2" />
<text x="718.74" y="2063.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1845" width="364.1" height="15.0" fill="rgb(232,156,6)" rx="2" ry="2" />
<text x="790.11" y="1855.5" >[unknown]</text>
</g>
<g >
<title>__do_softirq (29 samples, 0.01%)</title><rect x="240.0" y="1685" width="0.1" height="15.0" fill="rgb(254,157,48)" rx="2" ry="2" />
<text x="242.96" y="1695.5" ></text>
</g>
<g >
<title>default_idle_call (14,242 samples, 4.99%)</title><rect x="720.8" y="1973" width="58.9" height="15.0" fill="rgb(220,70,54)" rx="2" ry="2" />
<text x="723.76" y="1983.5" >defaul..</text>
</g>
<g >
<title>getname_flags (42 samples, 0.01%)</title><rect x="651.1" y="1957" width="0.1" height="15.0" fill="rgb(221,182,53)" rx="2" ry="2" />
<text x="654.08" y="1967.5" ></text>
</g>
<g >
<title>path_openat (3,204 samples, 1.12%)</title><rect x="686.3" y="1941" width="13.3" height="15.0" fill="rgb(232,115,22)" rx="2" ry="2" />
<text x="689.33" y="1951.5" ></text>
</g>
<g >
<title>fsnotify_perm.part.0 (36 samples, 0.01%)</title><rect x="691.9" y="1893" width="0.2" height="15.0" fill="rgb(250,55,53)" rx="2" ry="2" />
<text x="694.91" y="1903.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (493 samples, 0.17%)</title><rect x="807.6" y="149" width="2.0" height="15.0" fill="rgb(244,197,10)" rx="2" ry="2" />
<text x="810.61" y="159.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1493" width="364.1" height="15.0" fill="rgb(220,142,21)" rx="2" ry="2" />
<text x="790.11" y="1503.5" >[unknown]</text>
</g>
<g >
<title>pv_wait_head_or_lock (64 samples, 0.02%)</title><rect x="94.5" y="1765" width="0.2" height="15.0" fill="rgb(226,4,32)" rx="2" ry="2" />
<text x="97.47" y="1775.5" ></text>
</g>
<g >
<title>find_vmap_area (860 samples, 0.30%)</title><rect x="834.1" y="85" width="3.5" height="15.0" fill="rgb(246,160,14)" rx="2" ry="2" />
<text x="837.06" y="95.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1013" width="364.1" height="15.0" fill="rgb(236,114,49)" rx="2" ry="2" />
<text x="790.11" y="1023.5" >[unknown]</text>
</g>
<g >
<title>vfs_read (181 samples, 0.06%)</title><rect x="800.6" y="53" width="0.8" height="15.0" fill="rgb(226,186,1)" rx="2" ry="2" />
<text x="803.60" y="63.5" ></text>
</g>
<g >
<title>__schedule (31 samples, 0.01%)</title><rect x="810.2" y="53" width="0.1" height="15.0" fill="rgb(230,29,54)" rx="2" ry="2" />
<text x="813.21" y="63.5" ></text>
</g>
<g >
<title>__pfx_kernel_init (892 samples, 0.31%)</title><rect x="782.2" y="1973" width="3.7" height="15.0" fill="rgb(236,89,28)" rx="2" ry="2" />
<text x="785.17" y="1983.5" ></text>
</g>
<g >
<title>__x64_sys_recvfrom (38,984 samples, 13.66%)</title><rect x="82.6" y="1925" width="161.2" height="15.0" fill="rgb(227,207,13)" rx="2" ry="2" />
<text x="85.61" y="1935.5" >__x64_sys_recvfrom</text>
</g>
<g >
<title>check_heap_object (260 samples, 0.09%)</title><rect x="652.8" y="1909" width="1.1" height="15.0" fill="rgb(227,61,46)" rx="2" ry="2" />
<text x="655.80" y="1919.5" ></text>
</g>
<g >
<title>smc_recvmsg (131 samples, 0.05%)</title><rect x="823.1" y="117" width="0.5" height="15.0" fill="rgb(252,10,3)" rx="2" ry="2" />
<text x="826.08" y="127.5" ></text>
</g>
<g >
<title>getname_flags.part.0 (304 samples, 0.11%)</title><rect x="700.2" y="1957" width="1.3" height="15.0" fill="rgb(250,175,7)" rx="2" ry="2" />
<text x="703.24" y="1967.5" ></text>
</g>
<g >
<title>dup_task_struct (41 samples, 0.01%)</title><rect x="716.9" y="1957" width="0.1" height="15.0" fill="rgb(211,61,15)" rx="2" ry="2" />
<text x="719.87" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (156 samples, 0.05%)</title><rect x="716.1" y="2053" width="0.6" height="15.0" fill="rgb(209,209,52)" rx="2" ry="2" />
<text x="719.08" y="2063.5" ></text>
</g>
<g >
<title>ksys_write (141 samples, 0.05%)</title><rect x="801.4" y="69" width="0.6" height="15.0" fill="rgb(240,89,36)" rx="2" ry="2" />
<text x="804.38" y="79.5" ></text>
</g>
<g >
<title>sock_poll (150 samples, 0.05%)</title><rect x="1155.2" y="1925" width="0.6" height="15.0" fill="rgb(247,143,17)" rx="2" ry="2" />
<text x="1158.17" y="1935.5" ></text>
</g>
<g >
<title>__folio_batch_release (206 samples, 0.07%)</title><rect x="470.6" y="1893" width="0.8" height="15.0" fill="rgb(207,24,4)" rx="2" ry="2" />
<text x="473.55" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (51,236 samples, 17.96%)</title><rect x="46.4" y="2021" width="211.9" height="15.0" fill="rgb(235,78,38)" rx="2" ry="2" />
<text x="49.43" y="2031.5" >[unknown]</text>
</g>
<g >
<title>[ngx_http_image_filter_module.so] (190 samples, 0.07%)</title><rect x="42.2" y="2037" width="0.7" height="15.0" fill="rgb(208,0,12)" rx="2" ry="2" />
<text x="45.16" y="2047.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (769 samples, 0.27%)</title><rect x="816.6" y="85" width="3.1" height="15.0" fill="rgb(212,9,28)" rx="2" ry="2" />
<text x="819.56" y="95.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (125 samples, 0.04%)</title><rect x="718.1" y="2021" width="0.5" height="15.0" fill="rgb(225,181,25)" rx="2" ry="2" />
<text x="721.10" y="2031.5" ></text>
</g>
<g >
<title>vfs_write (474 samples, 0.17%)</title><rect x="805.6" y="85" width="2.0" height="15.0" fill="rgb(247,57,22)" rx="2" ry="2" />
<text x="808.63" y="95.5" ></text>
</g>
<g >
<title>smc_tx_pending (5,025 samples, 1.76%)</title><rect x="55.8" y="1845" width="20.7" height="15.0" fill="rgb(220,160,29)" rx="2" ry="2" />
<text x="58.76" y="1855.5" ></text>
</g>
<g >
<title>selinux_inode_permission (1,001 samples, 0.35%)</title><rect x="658.1" y="1877" width="4.2" height="15.0" fill="rgb(239,48,52)" rx="2" ry="2" />
<text x="661.13" y="1887.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (159 samples, 0.06%)</title><rect x="819.1" y="69" width="0.6" height="15.0" fill="rgb(211,88,43)" rx="2" ry="2" />
<text x="822.09" y="79.5" ></text>
</g>
<g >
<title>get_unused_fd_flags (115 samples, 0.04%)</title><rect x="699.7" y="1957" width="0.5" height="15.0" fill="rgb(224,191,20)" rx="2" ry="2" />
<text x="702.71" y="1967.5" ></text>
</g>
<g >
<title>close@plt (32 samples, 0.01%)</title><rect x="267.0" y="2021" width="0.2" height="15.0" fill="rgb(249,161,26)" rx="2" ry="2" />
<text x="270.02" y="2031.5" ></text>
</g>
<g >
<title>ext4_file_open (110 samples, 0.04%)</title><rect x="690.9" y="1893" width="0.5" height="15.0" fill="rgb(248,175,34)" rx="2" ry="2" />
<text x="693.91" y="1903.5" ></text>
</g>
<g >
<title>mutex_lock (53 samples, 0.02%)</title><rect x="255.0" y="1893" width="0.2" height="15.0" fill="rgb(209,185,5)" rx="2" ry="2" />
<text x="258.02" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="949" width="364.1" height="15.0" fill="rgb(232,105,5)" rx="2" ry="2" />
<text x="790.11" y="959.5" >[unknown]</text>
</g>
<g >
<title>atime_needs_update (39 samples, 0.01%)</title><rect x="473.8" y="1893" width="0.2" height="15.0" fill="rgb(222,96,18)" rx="2" ry="2" />
<text x="476.79" y="1903.5" ></text>
</g>
<g >
<title>lockref_get_not_dead (37 samples, 0.01%)</title><rect x="689.5" y="1861" width="0.1" height="15.0" fill="rgb(249,39,52)" rx="2" ry="2" />
<text x="692.46" y="1871.5" ></text>
</g>
<g >
<title>__x64_sys_accept4 (47 samples, 0.02%)</title><rect x="648.3" y="1989" width="0.2" height="15.0" fill="rgb(250,209,46)" rx="2" ry="2" />
<text x="651.29" y="1999.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (3,503 samples, 1.23%)</title><rect x="1174.3" y="2037" width="14.5" height="15.0" fill="rgb(239,203,29)" rx="2" ry="2" />
<text x="1177.32" y="2047.5" ></text>
</g>
<g >
<title>sock_poll (464 samples, 0.16%)</title><rect x="1177.6" y="1925" width="1.9" height="15.0" fill="rgb(251,8,30)" rx="2" ry="2" />
<text x="1180.55" y="1935.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (462 samples, 0.16%)</title><rect x="1148.7" y="37" width="1.9" height="15.0" fill="rgb(225,37,25)" rx="2" ry="2" />
<text x="1151.71" y="47.5" ></text>
</g>
<g >
<title>_copy_from_user (73 samples, 0.03%)</title><rect x="254.2" y="1877" width="0.3" height="15.0" fill="rgb(244,170,11)" rx="2" ry="2" />
<text x="257.18" y="1887.5" ></text>
</g>
<g >
<title>pv_wait_node (36,890 samples, 12.93%)</title><rect x="992.2" y="53" width="152.5" height="15.0" fill="rgb(221,130,15)" rx="2" ry="2" />
<text x="995.15" y="63.5" >pv_wait_node</text>
</g>
<g >
<title>security_socket_recvmsg (249 samples, 0.09%)</title><rect x="83.4" y="1877" width="1.1" height="15.0" fill="rgb(223,61,6)" rx="2" ry="2" />
<text x="86.44" y="1887.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (36,893 samples, 12.93%)</title><rect x="992.2" y="261" width="152.5" height="15.0" fill="rgb(215,171,29)" rx="2" ry="2" />
<text x="995.15" y="271.5" >entry_SYSCALL_64</text>
</g>
<g >
<title>__fget_light (196 samples, 0.07%)</title><rect x="1168.3" y="1973" width="0.8" height="15.0" fill="rgb(227,110,20)" rx="2" ry="2" />
<text x="1171.28" y="1983.5" ></text>
</g>
<g >
<title>asm_sysvec_call_function_single (120 samples, 0.04%)</title><rect x="779.2" y="1941" width="0.5" height="15.0" fill="rgb(208,198,5)" rx="2" ry="2" />
<text x="782.17" y="1951.5" ></text>
</g>
<g >
<title>find_vmap_area (36,893 samples, 12.93%)</title><rect x="992.2" y="101" width="152.5" height="15.0" fill="rgb(236,9,25)" rx="2" ry="2" />
<text x="995.15" y="111.5" >find_vmap_area</text>
</g>
<g >
<title>sockfd_lookup_light (172 samples, 0.06%)</title><rect x="76.9" y="1877" width="0.7" height="15.0" fill="rgb(210,156,43)" rx="2" ry="2" />
<text x="79.87" y="1887.5" ></text>
</g>
<g >
<title>find_vmap_area (590 samples, 0.21%)</title><rect x="819.8" y="37" width="2.4" height="15.0" fill="rgb(215,112,53)" rx="2" ry="2" />
<text x="822.81" y="47.5" ></text>
</g>
<g >
<title>sock_write_iter (473 samples, 0.17%)</title><rect x="1148.7" y="229" width="1.9" height="15.0" fill="rgb(212,76,26)" rx="2" ry="2" />
<text x="1151.67" y="239.5" ></text>
</g>
<g >
<title>_raw_spin_lock (127 samples, 0.04%)</title><rect x="823.1" y="37" width="0.5" height="15.0" fill="rgb(252,57,52)" rx="2" ry="2" />
<text x="826.08" y="47.5" ></text>
</g>
<g >
<title>ext4_getattr (66 samples, 0.02%)</title><rect x="654.7" y="1925" width="0.3" height="15.0" fill="rgb(221,81,35)" rx="2" ry="2" />
<text x="657.72" y="1935.5" ></text>
</g>
<g >
<title>[libc.so.6] (41 samples, 0.01%)</title><rect x="716.1" y="2037" width="0.2" height="15.0" fill="rgb(248,161,12)" rx="2" ry="2" />
<text x="719.14" y="2047.5" ></text>
</g>
<g >
<title>check_heap_object (106 samples, 0.04%)</title><rect x="701.1" y="1909" width="0.4" height="15.0" fill="rgb(240,169,22)" rx="2" ry="2" />
<text x="704.05" y="1919.5" ></text>
</g>
<g >
<title>load_elf_binary (29 samples, 0.01%)</title><rect x="717.3" y="1925" width="0.1" height="15.0" fill="rgb(222,24,2)" rx="2" ry="2" />
<text x="720.30" y="1935.5" ></text>
</g>
<g >
<title>rep_movs_alternative (61 samples, 0.02%)</title><rect x="706.0" y="1957" width="0.3" height="15.0" fill="rgb(231,117,5)" rx="2" ry="2" />
<text x="709.03" y="1967.5" ></text>
</g>
<g >
<title>_raw_write_unlock_irq (116 samples, 0.04%)</title><rect x="1176.6" y="1941" width="0.4" height="15.0" fill="rgb(207,46,11)" rx="2" ry="2" />
<text x="1179.56" y="1951.5" ></text>
</g>
<g >
<title>selinux_file_open (80 samples, 0.03%)</title><rect x="692.6" y="1877" width="0.3" height="15.0" fill="rgb(234,213,26)" rx="2" ry="2" />
<text x="695.58" y="1887.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (86 samples, 0.03%)</title><rect x="1148.3" y="277" width="0.3" height="15.0" fill="rgb(212,187,21)" rx="2" ry="2" />
<text x="1151.28" y="287.5" ></text>
</g>
<g >
<title>sock_common_setsockopt (135 samples, 0.05%)</title><rect x="255.6" y="1893" width="0.6" height="15.0" fill="rgb(218,218,8)" rx="2" ry="2" />
<text x="258.65" y="1903.5" ></text>
</g>
<g >
<title>exc_page_fault (33 samples, 0.01%)</title><rect x="716.2" y="2005" width="0.1" height="15.0" fill="rgb(212,131,17)" rx="2" ry="2" />
<text x="719.17" y="2015.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (507 samples, 0.18%)</title><rect x="779.7" y="1957" width="2.1" height="15.0" fill="rgb(211,165,47)" rx="2" ry="2" />
<text x="782.73" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1429" width="364.1" height="15.0" fill="rgb(226,17,22)" rx="2" ry="2" />
<text x="790.11" y="1439.5" >[unknown]</text>
</g>
<g >
<title>release_sock (25 samples, 0.01%)</title><rect x="1165.4" y="1925" width="0.1" height="15.0" fill="rgb(252,226,14)" rx="2" ry="2" />
<text x="1168.40" y="1935.5" ></text>
</g>
<g >
<title>smc_poll (54 samples, 0.02%)</title><rect x="1155.6" y="1909" width="0.2" height="15.0" fill="rgb(237,50,36)" rx="2" ry="2" />
<text x="1158.57" y="1919.5" ></text>
</g>
<g >
<title>splice_folio_into_pipe (86 samples, 0.03%)</title><rect x="472.7" y="1893" width="0.4" height="15.0" fill="rgb(243,30,26)" rx="2" ry="2" />
<text x="475.73" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1829" width="364.1" height="15.0" fill="rgb(250,202,10)" rx="2" ry="2" />
<text x="790.11" y="1839.5" >[unknown]</text>
</g>
<g >
<title>default_idle_call (875 samples, 0.31%)</title><rect x="782.2" y="1909" width="3.6" height="15.0" fill="rgb(238,207,48)" rx="2" ry="2" />
<text x="785.17" y="1919.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (1,197 samples, 0.42%)</title><rect x="823.6" y="101" width="5.0" height="15.0" fill="rgb(237,22,32)" rx="2" ry="2" />
<text x="826.63" y="111.5" ></text>
</g>
<g >
<title>[unknown] (4,263 samples, 1.49%)</title><rect x="787.1" y="149" width="17.6" height="15.0" fill="rgb(208,125,6)" rx="2" ry="2" />
<text x="790.11" y="159.5" ></text>
</g>
<g >
<title>_raw_spin_lock_bh (52 samples, 0.02%)</title><rect x="85.1" y="1845" width="0.2" height="15.0" fill="rgb(252,127,5)" rx="2" ry="2" />
<text x="88.10" y="1855.5" ></text>
</g>
<g >
<title>vfs_read (38 samples, 0.01%)</title><rect x="799.4" y="37" width="0.1" height="15.0" fill="rgb(206,185,52)" rx="2" ry="2" />
<text x="802.38" y="47.5" ></text>
</g>
<g >
<title>syscall_exit_to_user_mode (142 samples, 0.05%)</title><rect x="476.6" y="1973" width="0.5" height="15.0" fill="rgb(230,172,39)" rx="2" ry="2" />
<text x="479.56" y="1983.5" ></text>
</g>
<g >
<title>strncpy_from_user (389 samples, 0.14%)</title><rect x="652.3" y="1941" width="1.6" height="15.0" fill="rgb(254,5,50)" rx="2" ry="2" />
<text x="655.27" y="1951.5" ></text>
</g>
<g >
<title>do_syscall_64 (807 samples, 0.28%)</title><rect x="1153.7" y="2005" width="3.3" height="15.0" fill="rgb(224,193,23)" rx="2" ry="2" />
<text x="1156.67" y="2015.5" ></text>
</g>
<g >
<title>rep_movs_alternative (177 samples, 0.06%)</title><rect x="241.9" y="1829" width="0.7" height="15.0" fill="rgb(233,97,52)" rx="2" ry="2" />
<text x="244.91" y="1839.5" ></text>
</g>
<g >
<title>ngx_http_set_content_type (80 samples, 0.03%)</title><rect x="676.7" y="2037" width="0.3" height="15.0" fill="rgb(242,184,21)" rx="2" ry="2" />
<text x="679.66" y="2047.5" ></text>
</g>
<g >
<title>security_socket_setsockopt (320 samples, 0.11%)</title><rect x="251.7" y="1909" width="1.3" height="15.0" fill="rgb(247,157,52)" rx="2" ry="2" />
<text x="254.70" y="1919.5" ></text>
</g>
<g >
<title>ngx_time_update (25 samples, 0.01%)</title><rect x="715.4" y="2053" width="0.1" height="15.0" fill="rgb(207,204,45)" rx="2" ry="2" />
<text x="718.37" y="2063.5" ></text>
</g>
<g >
<title>copy_process (73 samples, 0.03%)</title><rect x="716.7" y="1973" width="0.3" height="15.0" fill="rgb(230,80,12)" rx="2" ry="2" />
<text x="719.73" y="1983.5" ></text>
</g>
<g >
<title>__list_del_entry_valid_or_report (31 samples, 0.01%)</title><rect x="245.9" y="1893" width="0.1" height="15.0" fill="rgb(219,62,33)" rx="2" ry="2" />
<text x="248.90" y="1903.5" ></text>
</g>
<g >
<title>memset_orig (106 samples, 0.04%)</title><rect x="812.5" y="37" width="0.4" height="15.0" fill="rgb(241,89,46)" rx="2" ry="2" />
<text x="815.46" y="47.5" ></text>
</g>
<g >
<title>[unknown] (88,025 samples, 30.85%)</title><rect x="787.1" y="581" width="364.1" height="15.0" fill="rgb(209,186,45)" rx="2" ry="2" />
<text x="790.11" y="591.5" >[unknown]</text>
</g>
<g >
<title>ngx_reusable_connection (30 samples, 0.01%)</title><rect x="79.5" y="1973" width="0.1" height="15.0" fill="rgb(230,125,37)" rx="2" ry="2" />
<text x="82.49" y="1983.5" ></text>
</g>
<g >
<title>do_user_addr_fault (33 samples, 0.01%)</title><rect x="716.2" y="1989" width="0.1" height="15.0" fill="rgb(234,125,34)" rx="2" ry="2" />
<text x="719.17" y="1999.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (92 samples, 0.03%)</title><rect x="717.6" y="2053" width="0.3" height="15.0" fill="rgb(213,85,43)" rx="2" ry="2" />
<text x="720.56" y="2063.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (263 samples, 0.09%)</title><rect x="701.6" y="1989" width="1.1" height="15.0" fill="rgb(212,214,2)" rx="2" ry="2" />
<text x="704.63" y="1999.5" ></text>
</g>
<g >
<title>sockfd_lookup_light (229 samples, 0.08%)</title><rect x="242.9" y="1893" width="0.9" height="15.0" fill="rgb(247,102,45)" rx="2" ry="2" />
<text x="245.90" y="1903.5" ></text>
</g>
<g >
<title>__get_user_1 (100 samples, 0.04%)</title><rect x="708.5" y="1973" width="0.4" height="15.0" fill="rgb(250,125,50)" rx="2" ry="2" />
<text x="711.46" y="1983.5" ></text>
</g>
<g >
<title>atime_needs_update (143 samples, 0.05%)</title><rect x="473.2" y="1877" width="0.6" height="15.0" fill="rgb(209,78,52)" rx="2" ry="2" />
<text x="476.17" y="1887.5" ></text>
</g>
<g >
<title>.slowpath (431 samples, 0.15%)</title><rect x="463.1" y="1765" width="1.8" height="15.0" fill="rgb(232,124,52)" rx="2" ry="2" />
<text x="466.09" y="1775.5" ></text>
</g>
<g >
<title>generic_fillattr (29 samples, 0.01%)</title><rect x="707.4" y="1941" width="0.1" height="15.0" fill="rgb(251,9,17)" rx="2" ry="2" />
<text x="710.40" y="1951.5" ></text>
</g>
<g >
<title>ngx_http_handler (77 samples, 0.03%)</title><rect x="672.6" y="2037" width="0.4" height="15.0" fill="rgb(251,159,37)" rx="2" ry="2" />
<text x="675.65" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (39 samples, 0.01%)</title><rect x="703.7" y="2021" width="0.1" height="15.0" fill="rgb(246,103,11)" rx="2" ry="2" />
<text x="706.68" y="2031.5" ></text>
</g>
<g >
<title>write (1,319 samples, 0.46%)</title><rect x="823.6" y="229" width="5.5" height="15.0" fill="rgb(233,129,50)" rx="2" ry="2" />
<text x="826.63" y="239.5" ></text>
</g>
<g >
<title>filemap_get_read_batch (249 samples, 0.09%)</title><rect x="471.6" y="1877" width="1.0" height="15.0" fill="rgb(252,201,21)" rx="2" ry="2" />
<text x="474.61" y="1887.5" ></text>
</g>
<g >
<title>__list_del_entry_valid_or_report (36 samples, 0.01%)</title><rect x="1176.0" y="1941" width="0.2" height="15.0" fill="rgb(244,179,48)" rx="2" ry="2" />
<text x="1179.03" y="1951.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock_slowpath (381 samples, 0.13%)</title><rect x="1146.7" y="69" width="1.6" height="15.0" fill="rgb(229,181,35)" rx="2" ry="2" />
<text x="1149.71" y="79.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (293 samples, 0.10%)</title><rect x="800.2" y="101" width="1.2" height="15.0" fill="rgb(208,140,19)" rx="2" ry="2" />
<text x="803.15" y="111.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (1,169 samples, 0.41%)</title><rect x="834.1" y="133" width="4.8" height="15.0" fill="rgb(211,101,35)" rx="2" ry="2" />
<text x="837.05" y="143.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1253" width="364.1" height="15.0" fill="rgb(244,161,35)" rx="2" ry="2" />
<text x="790.11" y="1263.5" >[unknown]</text>
</g>
<g >
<title>__fdget_pos (78 samples, 0.03%)</title><rect x="803.5" y="69" width="0.3" height="15.0" fill="rgb(240,184,15)" rx="2" ry="2" />
<text x="806.50" y="79.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (47 samples, 0.02%)</title><rect x="810.3" y="53" width="0.2" height="15.0" fill="rgb(246,159,12)" rx="2" ry="2" />
<text x="813.35" y="63.5" ></text>
</g>
<g >
<title>do_epoll_ctl (514 samples, 0.18%)</title><rect x="1153.9" y="1973" width="2.1" height="15.0" fill="rgb(226,175,7)" rx="2" ry="2" />
<text x="1156.88" y="1983.5" ></text>
</g>
<g >
<title>x86_64_start_reservations (892 samples, 0.31%)</title><rect x="782.2" y="2021" width="3.7" height="15.0" fill="rgb(254,226,30)" rx="2" ry="2" />
<text x="785.17" y="2031.5" ></text>
</g>
<g >
<title>file_has_perm (35 samples, 0.01%)</title><rect x="811.4" y="85" width="0.1" height="15.0" fill="rgb(213,5,26)" rx="2" ry="2" />
<text x="814.40" y="95.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (36,168 samples, 12.68%)</title><rect x="842.6" y="245" width="149.6" height="15.0" fill="rgb(220,132,48)" rx="2" ry="2" />
<text x="845.56" y="255.5" >entry_SYSCALL_64</text>
</g>
<g >
<title>entry_SYSCALL_64 (772 samples, 0.27%)</title><rect x="816.6" y="181" width="3.1" height="15.0" fill="rgb(254,108,27)" rx="2" ry="2" />
<text x="819.55" y="191.5" ></text>
</g>
<g >
<title>avc_lookup (29 samples, 0.01%)</title><rect x="666.8" y="1877" width="0.1" height="15.0" fill="rgb(235,51,0)" rx="2" ry="2" />
<text x="669.76" y="1887.5" ></text>
</g>
<g >
<title>[unknown] (10,148 samples, 3.56%)</title><rect x="787.1" y="245" width="42.0" height="15.0" fill="rgb(211,114,27)" rx="2" ry="2" />
<text x="790.11" y="255.5" >[un..</text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock_slowpath (341 samples, 0.12%)</title><rect x="1144.7" y="69" width="1.5" height="15.0" fill="rgb(232,51,1)" rx="2" ry="2" />
<text x="1147.74" y="79.5" ></text>
</g>
<g >
<title>__sys_recvfrom (38,960 samples, 13.66%)</title><rect x="82.7" y="1909" width="161.1" height="15.0" fill="rgb(205,136,17)" rx="2" ry="2" />
<text x="85.71" y="1919.5" >__sys_recvfrom</text>
</g>
<g >
<title>__sys_accept4 (47 samples, 0.02%)</title><rect x="648.3" y="1973" width="0.2" height="15.0" fill="rgb(207,147,27)" rx="2" ry="2" />
<text x="651.29" y="1983.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_bh (32 samples, 0.01%)</title><rect x="54.3" y="1845" width="0.1" height="15.0" fill="rgb(226,13,25)" rx="2" ry="2" />
<text x="57.28" y="1855.5" ></text>
</g>
<g >
<title>sock_read_iter (36,893 samples, 12.93%)</title><rect x="992.2" y="197" width="152.5" height="15.0" fill="rgb(220,217,33)" rx="2" ry="2" />
<text x="995.15" y="207.5" >sock_read_iter</text>
</g>
<g >
<title>__smc_release (31 samples, 0.01%)</title><rect x="703.7" y="1925" width="0.1" height="15.0" fill="rgb(219,130,28)" rx="2" ry="2" />
<text x="706.71" y="1935.5" ></text>
</g>
<g >
<title>set_root (67 samples, 0.02%)</title><rect x="698.9" y="1893" width="0.3" height="15.0" fill="rgb(248,65,40)" rx="2" ry="2" />
<text x="701.90" y="1903.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,319 samples, 0.46%)</title><rect x="823.6" y="197" width="5.5" height="15.0" fill="rgb(211,135,16)" rx="2" ry="2" />
<text x="826.63" y="207.5" ></text>
</g>
<g >
<title>security_file_permission (78 samples, 0.03%)</title><rect x="804.8" y="69" width="0.3" height="15.0" fill="rgb(243,10,20)" rx="2" ry="2" />
<text x="807.76" y="79.5" ></text>
</g>
<g >
<title>schedule (35 samples, 0.01%)</title><rect x="809.5" y="69" width="0.1" height="15.0" fill="rgb(245,10,22)" rx="2" ry="2" />
<text x="812.50" y="79.5" ></text>
</g>
<g >
<title>avc_lookup (44 samples, 0.02%)</title><rect x="810.4" y="37" width="0.1" height="15.0" fill="rgb(205,143,25)" rx="2" ry="2" />
<text x="813.36" y="47.5" ></text>
</g>
<g >
<title>do_mmap (42 samples, 0.01%)</title><rect x="719.1" y="1973" width="0.2" height="15.0" fill="rgb(237,0,25)" rx="2" ry="2" />
<text x="722.15" y="1983.5" ></text>
</g>
<g >
<title>__inode_security_revalidate (32 samples, 0.01%)</title><rect x="693.9" y="1861" width="0.1" height="15.0" fill="rgb(238,189,2)" rx="2" ry="2" />
<text x="696.87" y="1871.5" ></text>
</g>
<g >
<title>ksys_read (290 samples, 0.10%)</title><rect x="800.2" y="69" width="1.2" height="15.0" fill="rgb(214,214,21)" rx="2" ry="2" />
<text x="803.15" y="79.5" ></text>
</g>
<g >
<title>ngx_linux_sendfile_chain (111 samples, 0.04%)</title><rect x="713.6" y="2053" width="0.4" height="15.0" fill="rgb(228,148,3)" rx="2" ry="2" />
<text x="716.55" y="2063.5" ></text>
</g>
<g >
<title>avc_lookup (54 samples, 0.02%)</title><rect x="84.2" y="1813" width="0.3" height="15.0" fill="rgb(219,58,43)" rx="2" ry="2" />
<text x="87.25" y="1823.5" ></text>
</g>
<g >
<title>smc_release_cb (34 samples, 0.01%)</title><rect x="55.6" y="1829" width="0.2" height="15.0" fill="rgb(233,10,20)" rx="2" ry="2" />
<text x="58.62" y="1839.5" ></text>
</g>
<g >
<title>__check_heap_object (26 samples, 0.01%)</title><rect x="652.7" y="1909" width="0.1" height="15.0" fill="rgb(215,67,31)" rx="2" ry="2" />
<text x="655.68" y="1919.5" ></text>
</g>
<g >
<title>ngx_http_send_header (31 samples, 0.01%)</title><rect x="676.5" y="2037" width="0.2" height="15.0" fill="rgb(215,217,15)" rx="2" ry="2" />
<text x="679.53" y="2047.5" ></text>
</g>
<g >
<title>lookup_fast (137 samples, 0.05%)</title><rect x="697.7" y="1893" width="0.5" height="15.0" fill="rgb(235,100,23)" rx="2" ry="2" />
<text x="700.66" y="1903.5" ></text>
</g>
<g >
<title>__schedule (37 samples, 0.01%)</title><rect x="78.6" y="1845" width="0.2" height="15.0" fill="rgb(242,19,40)" rx="2" ry="2" />
<text x="81.64" y="1855.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1637" width="364.1" height="15.0" fill="rgb(206,180,33)" rx="2" ry="2" />
<text x="790.11" y="1647.5" >[unknown]</text>
</g>
<g >
<title>inode_security (63 samples, 0.02%)</title><rect x="804.8" y="37" width="0.3" height="15.0" fill="rgb(226,52,53)" rx="2" ry="2" />
<text x="807.82" y="47.5" ></text>
</g>
<g >
<title>pmlogger (66 samples, 0.02%)</title><rect x="715.7" y="2069" width="0.3" height="15.0" fill="rgb(217,187,18)" rx="2" ry="2" />
<text x="718.71" y="2079.5" ></text>
</g>
<g >
<title>vfs_getattr_nosec (107 samples, 0.04%)</title><rect x="666.9" y="1941" width="0.4" height="15.0" fill="rgb(230,85,3)" rx="2" ry="2" />
<text x="669.90" y="1951.5" ></text>
</g>
<g >
<title>do_syscall_64 (29 samples, 0.01%)</title><rect x="716.5" y="2005" width="0.1" height="15.0" fill="rgb(230,101,44)" rx="2" ry="2" />
<text x="719.45" y="2015.5" ></text>
</g>
<g >
<title>__fsnotify_parent (41 samples, 0.01%)</title><rect x="276.4" y="1941" width="0.2" height="15.0" fill="rgb(239,39,22)" rx="2" ry="2" />
<text x="279.42" y="1951.5" ></text>
</g>
<g >
<title>security_file_permission (322 samples, 0.11%)</title><rect x="645.1" y="1925" width="1.3" height="15.0" fill="rgb(207,132,17)" rx="2" ry="2" />
<text x="648.11" y="1935.5" ></text>
</g>
<g >
<title>inode_to_bdi (42 samples, 0.01%)</title><rect x="691.7" y="1877" width="0.2" height="15.0" fill="rgb(215,8,21)" rx="2" ry="2" />
<text x="694.74" y="1887.5" ></text>
</g>
<g >
<title>memcg_slab_post_alloc_hook (237 samples, 0.08%)</title><rect x="687.7" y="1893" width="0.9" height="15.0" fill="rgb(236,205,3)" rx="2" ry="2" />
<text x="690.65" y="1903.5" ></text>
</g>
<g >
<title>selinux_socket_recvmsg (190 samples, 0.07%)</title><rect x="807.6" y="37" width="0.8" height="15.0" fill="rgb(214,131,21)" rx="2" ry="2" />
<text x="810.63" y="47.5" ></text>
</g>
<g >
<title>smc_tx_consumer_update (26 samples, 0.01%)</title><rect x="816.4" y="37" width="0.2" height="15.0" fill="rgb(243,23,23)" rx="2" ry="2" />
<text x="819.45" y="47.5" ></text>
</g>
<g >
<title>http_parser_execute (204 samples, 0.07%)</title><rect x="1188.8" y="2053" width="0.9" height="15.0" fill="rgb(246,63,11)" rx="2" ry="2" />
<text x="1191.82" y="2063.5" ></text>
</g>
<g >
<title>lock_sock_nested (54 samples, 0.02%)</title><rect x="85.1" y="1861" width="0.2" height="15.0" fill="rgb(214,200,0)" rx="2" ry="2" />
<text x="88.09" y="1871.5" ></text>
</g>
<g >
<title>find_vmap_area (1,197 samples, 0.42%)</title><rect x="823.6" y="69" width="5.0" height="15.0" fill="rgb(239,60,46)" rx="2" ry="2" />
<text x="826.63" y="79.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (208 samples, 0.07%)</title><rect x="804.8" y="133" width="0.8" height="15.0" fill="rgb(238,199,30)" rx="2" ry="2" />
<text x="807.76" y="143.5" ></text>
</g>
<g >
<title>exc_page_fault (27 samples, 0.01%)</title><rect x="719.8" y="1989" width="0.1" height="15.0" fill="rgb(246,86,43)" rx="2" ry="2" />
<text x="722.78" y="1999.5" ></text>
</g>
<g >
<title>__fdget_pos (61 samples, 0.02%)</title><rect x="801.4" y="53" width="0.2" height="15.0" fill="rgb(242,49,20)" rx="2" ry="2" />
<text x="804.38" y="63.5" ></text>
</g>
<g >
<title>recv (39,783 samples, 13.94%)</title><rect x="80.9" y="1973" width="164.5" height="15.0" fill="rgb(230,99,11)" rx="2" ry="2" />
<text x="83.87" y="1983.5" >recv</text>
</g>
<g >
<title>inode_permission.part.0 (167 samples, 0.06%)</title><rect x="694.5" y="1909" width="0.7" height="15.0" fill="rgb(217,193,17)" rx="2" ry="2" />
<text x="697.53" y="1919.5" ></text>
</g>
<g >
<title>smc_sendmsg (1,169 samples, 0.41%)</title><rect x="834.1" y="149" width="4.8" height="15.0" fill="rgb(248,214,12)" rx="2" ry="2" />
<text x="837.05" y="159.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (114 samples, 0.04%)</title><rect x="476.6" y="1893" width="0.5" height="15.0" fill="rgb(230,34,54)" rx="2" ry="2" />
<text x="479.61" y="1903.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (112 samples, 0.04%)</title><rect x="812.4" y="181" width="0.5" height="15.0" fill="rgb(217,116,40)" rx="2" ry="2" />
<text x="815.45" y="191.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (665 samples, 0.23%)</title><rect x="659.5" y="1861" width="2.8" height="15.0" fill="rgb(222,140,30)" rx="2" ry="2" />
<text x="662.52" y="1871.5" ></text>
</g>
<g >
<title>avc_has_perm (129 samples, 0.05%)</title><rect x="252.5" y="1877" width="0.5" height="15.0" fill="rgb(227,66,52)" rx="2" ry="2" />
<text x="255.49" y="1887.5" ></text>
</g>
<g >
<title>do_syscall_64 (381 samples, 0.13%)</title><rect x="1146.7" y="261" width="1.6" height="15.0" fill="rgb(218,172,8)" rx="2" ry="2" />
<text x="1149.71" y="271.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (250 samples, 0.09%)</title><rect x="1187.7" y="2005" width="1.1" height="15.0" fill="rgb(234,49,14)" rx="2" ry="2" />
<text x="1190.74" y="2015.5" ></text>
</g>
<g >
<title>tick_nohz_idle_exit (67 samples, 0.02%)</title><rect x="781.9" y="1989" width="0.3" height="15.0" fill="rgb(253,141,7)" rx="2" ry="2" />
<text x="784.89" y="1999.5" ></text>
</g>
<g >
<title>[perf] (38 samples, 0.01%)</title><rect x="715.5" y="1989" width="0.2" height="15.0" fill="rgb(211,145,11)" rx="2" ry="2" />
<text x="718.50" y="1999.5" ></text>
</g>
<g >
<title>do_syscall_64 (29 samples, 0.01%)</title><rect x="717.3" y="2021" width="0.1" height="15.0" fill="rgb(241,174,9)" rx="2" ry="2" />
<text x="720.30" y="2031.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (602 samples, 0.21%)</title><rect x="819.8" y="69" width="2.5" height="15.0" fill="rgb(211,102,6)" rx="2" ry="2" />
<text x="822.81" y="79.5" ></text>
</g>
<g >
<title>path_put (66 samples, 0.02%)</title><rect x="665.8" y="1941" width="0.3" height="15.0" fill="rgb(224,86,2)" rx="2" ry="2" />
<text x="668.84" y="1951.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (155 samples, 0.05%)</title><rect x="718.0" y="2037" width="0.6" height="15.0" fill="rgb(219,71,36)" rx="2" ry="2" />
<text x="721.01" y="2047.5" ></text>
</g>
<g >
<title>[unknown] (87,891 samples, 30.81%)</title><rect x="787.1" y="325" width="363.5" height="15.0" fill="rgb(220,37,39)" rx="2" ry="2" />
<text x="790.11" y="335.5" >[unknown]</text>
</g>
<g >
<title>kmem_cache_free (225 samples, 0.08%)</title><rect x="264.4" y="1957" width="1.0" height="15.0" fill="rgb(221,109,3)" rx="2" ry="2" />
<text x="267.42" y="1967.5" ></text>
</g>
<g >
<title>asm_sysvec_apic_timer_interrupt (29 samples, 0.01%)</title><rect x="240.0" y="1733" width="0.1" height="15.0" fill="rgb(229,116,37)" rx="2" ry="2" />
<text x="242.96" y="1743.5" ></text>
</g>
<g >
<title>__x64_sys_close (1,243 samples, 0.44%)</title><rect x="260.2" y="1973" width="5.2" height="15.0" fill="rgb(224,199,36)" rx="2" ry="2" />
<text x="263.21" y="1983.5" ></text>
</g>
<g >
<title>exc_page_fault (28 samples, 0.01%)</title><rect x="719.6" y="1973" width="0.1" height="15.0" fill="rgb(218,124,29)" rx="2" ry="2" />
<text x="722.57" y="1983.5" ></text>
</g>
<g >
<title>rep_movs_alternative (41 samples, 0.01%)</title><rect x="274.7" y="1941" width="0.2" height="15.0" fill="rgb(215,229,46)" rx="2" ry="2" />
<text x="277.74" y="1951.5" ></text>
</g>
<g >
<title>syscall_exit_to_user_mode (27 samples, 0.01%)</title><rect x="702.7" y="1989" width="0.1" height="15.0" fill="rgb(238,227,0)" rx="2" ry="2" />
<text x="705.72" y="1999.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (888 samples, 0.31%)</title><rect x="838.9" y="117" width="3.7" height="15.0" fill="rgb(219,142,43)" rx="2" ry="2" />
<text x="841.89" y="127.5" ></text>
</g>
<g >
<title>ngx_handle_read_event (78 samples, 0.03%)</title><rect x="669.9" y="2037" width="0.3" height="15.0" fill="rgb(231,122,8)" rx="2" ry="2" />
<text x="672.86" y="2047.5" ></text>
</g>
<g >
<title>kmem_cache_alloc (406 samples, 0.14%)</title><rect x="687.3" y="1909" width="1.7" height="15.0" fill="rgb(236,173,45)" rx="2" ry="2" />
<text x="690.28" y="1919.5" ></text>
</g>
<g >
<title>sock_sendmsg (46,114 samples, 16.16%)</title><rect x="279.0" y="1877" width="190.8" height="15.0" fill="rgb(212,143,3)" rx="2" ry="2" />
<text x="282.03" y="1887.5" >sock_sendmsg</text>
</g>
<g >
<title>__schedule (121 samples, 0.04%)</title><rect x="476.6" y="1909" width="0.5" height="15.0" fill="rgb(235,153,11)" rx="2" ry="2" />
<text x="479.58" y="1919.5" ></text>
</g>
<g >
<title>avc_has_perm (66 samples, 0.02%)</title><rect x="84.2" y="1845" width="0.3" height="15.0" fill="rgb(246,38,42)" rx="2" ry="2" />
<text x="87.20" y="1855.5" ></text>
</g>
<g >
<title>__x64_sys_epoll_wait (3,234 samples, 1.13%)</title><rect x="1174.4" y="2005" width="13.3" height="15.0" fill="rgb(251,189,47)" rx="2" ry="2" />
<text x="1177.37" y="2015.5" ></text>
</g>
<g >
<title>__fget_light (107 samples, 0.04%)</title><rect x="479.5" y="1941" width="0.4" height="15.0" fill="rgb(213,64,14)" rx="2" ry="2" />
<text x="482.48" y="1951.5" ></text>
</g>
<g >
<title>ngx_linux_sendfile_chain (167 samples, 0.06%)</title><rect x="247.8" y="2005" width="0.7" height="15.0" fill="rgb(253,18,0)" rx="2" ry="2" />
<text x="250.80" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1621" width="364.1" height="15.0" fill="rgb(221,190,48)" rx="2" ry="2" />
<text x="790.11" y="1631.5" >[unknown]</text>
</g>
<g >
<title>__smc_tx_sndbuf_nonempty (473 samples, 0.17%)</title><rect x="1148.7" y="165" width="1.9" height="15.0" fill="rgb(228,228,25)" rx="2" ry="2" />
<text x="1151.67" y="175.5" ></text>
</g>
<g >
<title>ngx_http_complex_value_size (54 samples, 0.02%)</title><rect x="711.3" y="2053" width="0.2" height="15.0" fill="rgb(240,61,54)" rx="2" ry="2" />
<text x="714.29" y="2063.5" ></text>
</g>
<g >
<title>security_file_permission (119 samples, 0.04%)</title><rect x="803.8" y="53" width="0.5" height="15.0" fill="rgb(211,148,10)" rx="2" ry="2" />
<text x="806.82" y="63.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (96 samples, 0.03%)</title><rect x="643.6" y="1797" width="0.4" height="15.0" fill="rgb(248,6,41)" rx="2" ry="2" />
<text x="646.62" y="1807.5" ></text>
</g>
<g >
<title>locks_remove_posix (68 samples, 0.02%)</title><rect x="264.1" y="1941" width="0.3" height="15.0" fill="rgb(216,129,7)" rx="2" ry="2" />
<text x="267.14" y="1951.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (37 samples, 0.01%)</title><rect x="481.5" y="1845" width="0.1" height="15.0" fill="rgb(227,11,17)" rx="2" ry="2" />
<text x="484.47" y="1855.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (73 samples, 0.03%)</title><rect x="464.9" y="1765" width="0.3" height="15.0" fill="rgb(254,72,11)" rx="2" ry="2" />
<text x="467.87" y="1775.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1813" width="364.1" height="15.0" fill="rgb(228,58,17)" rx="2" ry="2" />
<text x="790.11" y="1823.5" >[unknown]</text>
</g>
<g >
<title>syscall_enter_from_user_mode (273 samples, 0.10%)</title><rect x="257.1" y="1957" width="1.1" height="15.0" fill="rgb(246,47,10)" rx="2" ry="2" />
<text x="260.12" y="1967.5" ></text>
</g>
<g >
<title>ngx_filename_cmp (44 samples, 0.02%)</title><rect x="669.6" y="2037" width="0.2" height="15.0" fill="rgb(228,223,41)" rx="2" ry="2" />
<text x="672.59" y="2047.5" ></text>
</g>
<g >
<title>do_epoll_ctl (698 samples, 0.24%)</title><rect x="1167.5" y="1989" width="2.9" height="15.0" fill="rgb(249,217,22)" rx="2" ry="2" />
<text x="1170.48" y="1999.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (43 samples, 0.02%)</title><rect x="40.6" y="1989" width="0.1" height="15.0" fill="rgb(238,46,24)" rx="2" ry="2" />
<text x="43.56" y="1999.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (887 samples, 0.31%)</title><rect x="838.9" y="53" width="3.7" height="15.0" fill="rgb(244,201,40)" rx="2" ry="2" />
<text x="841.89" y="63.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1045" width="364.1" height="15.0" fill="rgb(250,30,22)" rx="2" ry="2" />
<text x="790.11" y="1055.5" >[unknown]</text>
</g>
<g >
<title>direct_splice_actor (46,511 samples, 16.30%)</title><rect x="277.4" y="1909" width="192.4" height="15.0" fill="rgb(205,179,28)" rx="2" ry="2" />
<text x="280.39" y="1919.5" >direct_splice_actor</text>
</g>
<g >
<title>ngx_sprintf (67 samples, 0.02%)</title><rect x="680.8" y="2037" width="0.3" height="15.0" fill="rgb(210,91,1)" rx="2" ry="2" />
<text x="683.82" y="2047.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1285" width="364.1" height="15.0" fill="rgb(210,147,26)" rx="2" ry="2" />
<text x="790.11" y="1295.5" >[unknown]</text>
</g>
<g >
<title>get_inode_acl (42 samples, 0.01%)</title><rect x="693.5" y="1845" width="0.2" height="15.0" fill="rgb(242,74,30)" rx="2" ry="2" />
<text x="696.54" y="1855.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1525" width="364.1" height="15.0" fill="rgb(247,105,53)" rx="2" ry="2" />
<text x="790.11" y="1535.5" >[unknown]</text>
</g>
<g >
<title>smcd_tx_ism_write (308 samples, 0.11%)</title><rect x="837.6" y="69" width="1.3" height="15.0" fill="rgb(242,220,31)" rx="2" ry="2" />
<text x="840.61" y="79.5" ></text>
</g>
<g >
<title>ep_modify (169 samples, 0.06%)</title><rect x="1155.1" y="1957" width="0.7" height="15.0" fill="rgb(207,71,18)" rx="2" ry="2" />
<text x="1158.10" y="1967.5" ></text>
</g>
<g >
<title>__do_sys_newfstatat (702 samples, 0.25%)</title><rect x="706.0" y="2005" width="2.9" height="15.0" fill="rgb(242,21,51)" rx="2" ry="2" />
<text x="708.97" y="2015.5" ></text>
</g>
<g >
<title>dput (70 samples, 0.02%)</title><rect x="260.9" y="1941" width="0.3" height="15.0" fill="rgb(231,139,7)" rx="2" ry="2" />
<text x="263.95" y="1951.5" ></text>
</g>
<g >
<title>do_syscall_64 (36,893 samples, 12.93%)</title><rect x="992.2" y="245" width="152.5" height="15.0" fill="rgb(242,108,31)" rx="2" ry="2" />
<text x="995.15" y="255.5" >do_syscall_64</text>
</g>
<g >
<title>read (493 samples, 0.17%)</title><rect x="807.6" y="165" width="2.0" height="15.0" fill="rgb(237,67,19)" rx="2" ry="2" />
<text x="810.61" y="175.5" ></text>
</g>
<g >
<title>[unknown] (2,939 samples, 1.03%)</title><rect x="787.1" y="101" width="12.2" height="15.0" fill="rgb(253,12,51)" rx="2" ry="2" />
<text x="790.12" y="111.5" ></text>
</g>
<g >
<title>schedule_idle (514 samples, 0.18%)</title><rect x="779.7" y="1989" width="2.1" height="15.0" fill="rgb(239,23,9)" rx="2" ry="2" />
<text x="782.70" y="1999.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (168 samples, 0.06%)</title><rect x="809.6" y="149" width="0.7" height="15.0" fill="rgb(207,11,21)" rx="2" ry="2" />
<text x="812.65" y="159.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1237" width="364.1" height="15.0" fill="rgb(241,174,48)" rx="2" ry="2" />
<text x="790.11" y="1247.5" >[unknown]</text>
</g>
<g >
<title>ksys_read (60 samples, 0.02%)</title><rect x="799.3" y="53" width="0.2" height="15.0" fill="rgb(240,134,11)" rx="2" ry="2" />
<text x="802.29" y="63.5" ></text>
</g>
<g >
<title>ngx_pool_cleanup_file (93 samples, 0.03%)</title><rect x="269.9" y="2021" width="0.4" height="15.0" fill="rgb(212,149,35)" rx="2" ry="2" />
<text x="272.94" y="2031.5" ></text>
</g>
<g >
<title>fd_install (31 samples, 0.01%)</title><rect x="699.6" y="1957" width="0.1" height="15.0" fill="rgb(216,77,40)" rx="2" ry="2" />
<text x="702.58" y="1967.5" ></text>
</g>
<g >
<title>fput (61 samples, 0.02%)</title><rect x="1186.4" y="1973" width="0.2" height="15.0" fill="rgb(208,198,31)" rx="2" ry="2" />
<text x="1189.40" y="1983.5" ></text>
</g>
<g >
<title>ngx_http_discard_request_body (94 samples, 0.03%)</title><rect x="247.4" y="2005" width="0.4" height="15.0" fill="rgb(206,210,36)" rx="2" ry="2" />
<text x="250.41" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1509" width="364.1" height="15.0" fill="rgb(225,69,2)" rx="2" ry="2" />
<text x="790.11" y="1519.5" >[unknown]</text>
</g>
<g >
<title>sock_write_iter (1,319 samples, 0.46%)</title><rect x="823.6" y="149" width="5.5" height="15.0" fill="rgb(225,170,48)" rx="2" ry="2" />
<text x="826.63" y="159.5" ></text>
</g>
<g >
<title>smc_release (31 samples, 0.01%)</title><rect x="703.7" y="1941" width="0.1" height="15.0" fill="rgb(235,148,12)" rx="2" ry="2" />
<text x="706.71" y="1951.5" ></text>
</g>
<g >
<title>[[vdso]] (100 samples, 0.04%)</title><rect x="796.4" y="37" width="0.4" height="15.0" fill="rgb(230,49,52)" rx="2" ry="2" />
<text x="799.41" y="47.5" ></text>
</g>
<g >
<title>sysvec_apic_timer_interrupt (25 samples, 0.01%)</title><rect x="463.0" y="1701" width="0.1" height="15.0" fill="rgb(235,206,31)" rx="2" ry="2" />
<text x="465.97" y="1711.5" ></text>
</g>
<g >
<title>sock_read_iter (131 samples, 0.05%)</title><rect x="823.1" y="149" width="0.5" height="15.0" fill="rgb(229,17,40)" rx="2" ry="2" />
<text x="826.08" y="159.5" ></text>
</g>
<g >
<title>__fget_light (61 samples, 0.02%)</title><rect x="801.4" y="37" width="0.2" height="15.0" fill="rgb(242,43,4)" rx="2" ry="2" />
<text x="804.38" y="47.5" ></text>
</g>
<g >
<title>avc_has_perm (30 samples, 0.01%)</title><rect x="707.9" y="1941" width="0.1" height="15.0" fill="rgb(252,112,24)" rx="2" ry="2" />
<text x="710.92" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1029" width="364.1" height="15.0" fill="rgb(253,30,49)" rx="2" ry="2" />
<text x="790.11" y="1039.5" >[unknown]</text>
</g>
<g >
<title>http_parser_execute (63 samples, 0.02%)</title><rect x="788.6" y="37" width="0.2" height="15.0" fill="rgb(218,178,40)" rx="2" ry="2" />
<text x="791.56" y="47.5" ></text>
</g>
<g >
<title>fstatat64 (4,895 samples, 1.72%)</title><rect x="648.6" y="2037" width="20.2" height="15.0" fill="rgb(213,84,27)" rx="2" ry="2" />
<text x="651.56" y="2047.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (1,169 samples, 0.41%)</title><rect x="834.1" y="229" width="4.8" height="15.0" fill="rgb(235,135,21)" rx="2" ry="2" />
<text x="837.05" y="239.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (258 samples, 0.09%)</title><rect x="1170.4" y="2005" width="1.0" height="15.0" fill="rgb(229,11,33)" rx="2" ry="2" />
<text x="1173.37" y="2015.5" ></text>
</g>
<g >
<title>kvm_wait.part.0 (35,445 samples, 12.42%)</title><rect x="495.5" y="1765" width="146.6" height="15.0" fill="rgb(219,224,14)" rx="2" ry="2" />
<text x="498.55" y="1775.5" >kvm_wait.part.0</text>
</g>
<g >
<title>sock_read_iter (888 samples, 0.31%)</title><rect x="838.9" y="181" width="3.7" height="15.0" fill="rgb(237,197,27)" rx="2" ry="2" />
<text x="841.89" y="191.5" ></text>
</g>
<g >
<title>vfs_write (80 samples, 0.03%)</title><rect x="801.6" y="53" width="0.4" height="15.0" fill="rgb(245,70,6)" rx="2" ry="2" />
<text x="804.63" y="63.5" ></text>
</g>
<g >
<title>[unknown] (88,015 samples, 30.85%)</title><rect x="787.1" y="533" width="364.0" height="15.0" fill="rgb(248,148,4)" rx="2" ry="2" />
<text x="790.11" y="543.5" >[unknown]</text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (37 samples, 0.01%)</title><rect x="10.5" y="2037" width="0.2" height="15.0" fill="rgb(220,4,31)" rx="2" ry="2" />
<text x="13.52" y="2047.5" ></text>
</g>
<g >
<title>irq_exit_rcu (25 samples, 0.01%)</title><rect x="463.0" y="1685" width="0.1" height="15.0" fill="rgb(230,44,0)" rx="2" ry="2" />
<text x="465.97" y="1695.5" ></text>
</g>
<g >
<title>selinux_file_permission (204 samples, 0.07%)</title><rect x="474.8" y="1925" width="0.9" height="15.0" fill="rgb(234,7,40)" rx="2" ry="2" />
<text x="477.84" y="1935.5" ></text>
</g>
<g >
<title>selinux_socket_sendmsg (159 samples, 0.06%)</title><rect x="279.2" y="1845" width="0.7" height="15.0" fill="rgb(214,157,22)" rx="2" ry="2" />
<text x="282.21" y="1855.5" ></text>
</g>
<g >
<title>__fput (38 samples, 0.01%)</title><rect x="703.7" y="1989" width="0.1" height="15.0" fill="rgb(217,113,54)" rx="2" ry="2" />
<text x="706.68" y="1999.5" ></text>
</g>
<g >
<title>ep_send_events (258 samples, 0.09%)</title><rect x="245.8" y="1909" width="1.1" height="15.0" fill="rgb(224,34,4)" rx="2" ry="2" />
<text x="248.81" y="1919.5" ></text>
</g>
<g >
<title>[libc.so.6] (44 samples, 0.02%)</title><rect x="715.5" y="2053" width="0.2" height="15.0" fill="rgb(213,130,35)" rx="2" ry="2" />
<text x="718.50" y="2063.5" ></text>
</g>
<g >
<title>memcpy (39 samples, 0.01%)</title><rect x="669.0" y="2037" width="0.2" height="15.0" fill="rgb(205,121,8)" rx="2" ry="2" />
<text x="672.03" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (47 samples, 0.02%)</title><rect x="719.1" y="2021" width="0.2" height="15.0" fill="rgb(208,117,24)" rx="2" ry="2" />
<text x="722.15" y="2031.5" ></text>
</g>
<g >
<title>security_current_getsecid_subj (62 samples, 0.02%)</title><rect x="693.1" y="1893" width="0.2" height="15.0" fill="rgb(208,162,24)" rx="2" ry="2" />
<text x="696.07" y="1903.5" ></text>
</g>
<g >
<title>__tcp_sock_set_cork.part.0 (27 samples, 0.01%)</title><rect x="54.6" y="1829" width="0.1" height="15.0" fill="rgb(252,141,19)" rx="2" ry="2" />
<text x="57.62" y="1839.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (86 samples, 0.03%)</title><rect x="1148.3" y="117" width="0.3" height="15.0" fill="rgb(217,0,21)" rx="2" ry="2" />
<text x="1151.28" y="127.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (132 samples, 0.05%)</title><rect x="476.6" y="1941" width="0.5" height="15.0" fill="rgb(212,156,54)" rx="2" ry="2" />
<text x="479.56" y="1951.5" ></text>
</g>
<g >
<title>asm_exc_page_fault (27 samples, 0.01%)</title><rect x="718.3" y="1989" width="0.1" height="15.0" fill="rgb(230,103,10)" rx="2" ry="2" />
<text x="721.26" y="1999.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1701" width="364.1" height="15.0" fill="rgb(235,25,28)" rx="2" ry="2" />
<text x="790.11" y="1711.5" >[unknown]</text>
</g>
<g >
<title>aeCreateFileEvent (167 samples, 0.06%)</title><rect x="1162.3" y="2053" width="0.7" height="15.0" fill="rgb(242,217,50)" rx="2" ry="2" />
<text x="1165.31" y="2063.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (6,552 samples, 2.30%)</title><rect x="51.8" y="1941" width="27.1" height="15.0" fill="rgb(232,143,1)" rx="2" ry="2" />
<text x="54.79" y="1951.5" >e..</text>
</g>
<g >
<title>file_has_perm (94 samples, 0.03%)</title><rect x="645.2" y="1909" width="0.3" height="15.0" fill="rgb(231,219,4)" rx="2" ry="2" />
<text x="648.16" y="1919.5" ></text>
</g>
<g >
<title>ngx_http_discard_request_body (48 samples, 0.02%)</title><rect x="672.2" y="2037" width="0.2" height="15.0" fill="rgb(249,26,44)" rx="2" ry="2" />
<text x="675.16" y="2047.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (29 samples, 0.01%)</title><rect x="716.5" y="2021" width="0.1" height="15.0" fill="rgb(219,175,3)" rx="2" ry="2" />
<text x="719.45" y="2031.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (36,143 samples, 12.67%)</title><rect x="842.6" y="133" width="149.5" height="15.0" fill="rgb(213,172,12)" rx="2" ry="2" />
<text x="845.56" y="143.5" >__check_object_size..</text>
</g>
<g >
<title>_raw_spin_lock (1,088 samples, 0.38%)</title><rect x="823.6" y="53" width="4.5" height="15.0" fill="rgb(211,78,11)" rx="2" ry="2" />
<text x="826.63" y="63.5" ></text>
</g>
<g >
<title>ext4_file_getattr (129 samples, 0.05%)</title><rect x="707.0" y="1973" width="0.5" height="15.0" fill="rgb(249,65,25)" rx="2" ry="2" />
<text x="709.99" y="1983.5" ></text>
</g>
<g >
<title>[unknown] (48,081 samples, 16.85%)</title><rect x="46.6" y="1989" width="198.8" height="15.0" fill="rgb(223,13,27)" rx="2" ry="2" />
<text x="49.57" y="1999.5" >[unknown]</text>
</g>
<g >
<title>init_file (185 samples, 0.06%)</title><rect x="686.5" y="1909" width="0.8" height="15.0" fill="rgb(213,6,15)" rx="2" ry="2" />
<text x="689.51" y="1919.5" ></text>
</g>
<g >
<title>[unknown] (90,585 samples, 31.75%)</title><rect x="787.1" y="2053" width="374.7" height="15.0" fill="rgb(235,181,46)" rx="2" ry="2" />
<text x="790.11" y="2063.5" >[unknown]</text>
</g>
<g >
<title>complete_walk (180 samples, 0.06%)</title><rect x="655.2" y="1909" width="0.8" height="15.0" fill="rgb(230,179,25)" rx="2" ry="2" />
<text x="658.23" y="1919.5" ></text>
</g>
<g >
<title>dput (78 samples, 0.03%)</title><rect x="699.2" y="1909" width="0.3" height="15.0" fill="rgb(254,113,47)" rx="2" ry="2" />
<text x="702.23" y="1919.5" ></text>
</g>
<g >
<title>ksys_read (881 samples, 0.31%)</title><rect x="812.9" y="149" width="3.7" height="15.0" fill="rgb(234,54,40)" rx="2" ry="2" />
<text x="815.91" y="159.5" ></text>
</g>
<g >
<title>avc_has_perm (43 samples, 0.02%)</title><rect x="474.7" y="1909" width="0.1" height="15.0" fill="rgb(222,179,38)" rx="2" ry="2" />
<text x="477.66" y="1919.5" ></text>
</g>
<g >
<title>read (381 samples, 0.13%)</title><rect x="1146.7" y="293" width="1.6" height="15.0" fill="rgb(211,19,16)" rx="2" ry="2" />
<text x="1149.71" y="303.5" ></text>
</g>
<g >
<title>__legitimize_mnt (98 samples, 0.03%)</title><rect x="655.4" y="1861" width="0.4" height="15.0" fill="rgb(217,109,26)" rx="2" ry="2" />
<text x="658.35" y="1871.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (861 samples, 0.30%)</title><rect x="834.1" y="117" width="3.5" height="15.0" fill="rgb(215,130,44)" rx="2" ry="2" />
<text x="837.05" y="127.5" ></text>
</g>
<g >
<title>__inet_stream_connect (33 samples, 0.01%)</title><rect x="1165.3" y="1925" width="0.1" height="15.0" fill="rgb(216,111,47)" rx="2" ry="2" />
<text x="1168.26" y="1935.5" ></text>
</g>
<g >
<title>[nginx] (770 samples, 0.27%)</title><rect x="15.9" y="2053" width="3.2" height="15.0" fill="rgb(217,23,46)" rx="2" ry="2" />
<text x="18.93" y="2063.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (79 samples, 0.03%)</title><rect x="241.5" y="1781" width="0.3" height="15.0" fill="rgb(210,119,29)" rx="2" ry="2" />
<text x="244.48" y="1791.5" ></text>
</g>
<g >
<title>__fdget_pos (109 samples, 0.04%)</title><rect x="800.2" y="53" width="0.4" height="15.0" fill="rgb(248,15,46)" rx="2" ry="2" />
<text x="803.15" y="63.5" ></text>
</g>
<g >
<title>sock_read (220 samples, 0.08%)</title><rect x="791.6" y="37" width="1.0" height="15.0" fill="rgb(247,105,28)" rx="2" ry="2" />
<text x="794.65" y="47.5" ></text>
</g>
<g >
<title>vfs_splice_read.part.0 (87 samples, 0.03%)</title><rect x="474.0" y="1909" width="0.3" height="15.0" fill="rgb(235,18,17)" rx="2" ry="2" />
<text x="476.96" y="1919.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (31 samples, 0.01%)</title><rect x="10.5" y="2021" width="0.2" height="15.0" fill="rgb(214,67,51)" rx="2" ry="2" />
<text x="13.53" y="2031.5" ></text>
</g>
<g >
<title>check_heap_object (381 samples, 0.13%)</title><rect x="1146.7" y="133" width="1.6" height="15.0" fill="rgb(235,220,9)" rx="2" ry="2" />
<text x="1149.71" y="143.5" ></text>
</g>
<g >
<title>do_syscall_64 (143 samples, 0.05%)</title><rect x="801.4" y="85" width="0.6" height="15.0" fill="rgb(217,78,28)" rx="2" ry="2" />
<text x="804.38" y="95.5" ></text>
</g>
<g >
<title>security_file_permission (47 samples, 0.02%)</title><rect x="810.3" y="101" width="0.2" height="15.0" fill="rgb(221,10,5)" rx="2" ry="2" />
<text x="813.35" y="111.5" ></text>
</g>
<g >
<title>vfs_read (881 samples, 0.31%)</title><rect x="812.9" y="133" width="3.7" height="15.0" fill="rgb(252,226,20)" rx="2" ry="2" />
<text x="815.91" y="143.5" ></text>
</g>
<g >
<title>do_syscall_64 (112 samples, 0.04%)</title><rect x="812.4" y="165" width="0.5" height="15.0" fill="rgb(227,41,30)" rx="2" ry="2" />
<text x="815.45" y="175.5" ></text>
</g>
<g >
<title>inode_permission.part.0 (336 samples, 0.12%)</title><rect x="656.5" y="1893" width="1.4" height="15.0" fill="rgb(240,7,50)" rx="2" ry="2" />
<text x="659.53" y="1903.5" ></text>
</g>
<g >
<title>.slowpath (333 samples, 0.12%)</title><rect x="240.1" y="1781" width="1.4" height="15.0" fill="rgb(215,97,30)" rx="2" ry="2" />
<text x="243.10" y="1791.5" ></text>
</g>
<g >
<title>check_heap_object (888 samples, 0.31%)</title><rect x="838.9" y="101" width="3.7" height="15.0" fill="rgb(236,38,0)" rx="2" ry="2" />
<text x="841.89" y="111.5" ></text>
</g>
<g >
<title>_raw_spin_lock_bh (30 samples, 0.01%)</title><rect x="811.0" y="37" width="0.1" height="15.0" fill="rgb(234,81,0)" rx="2" ry="2" />
<text x="813.97" y="47.5" ></text>
</g>
<g >
<title>vfs_write (772 samples, 0.27%)</title><rect x="816.6" y="133" width="3.1" height="15.0" fill="rgb(217,19,42)" rx="2" ry="2" />
<text x="819.55" y="143.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1941" width="364.1" height="15.0" fill="rgb(240,196,22)" rx="2" ry="2" />
<text x="790.11" y="1951.5" >[unknown]</text>
</g>
<g >
<title>lockref_put_return (67 samples, 0.02%)</title><rect x="261.0" y="1925" width="0.2" height="15.0" fill="rgb(253,57,7)" rx="2" ry="2" />
<text x="263.96" y="1935.5" ></text>
</g>
<g >
<title>page_cache_pipe_buf_release (106 samples, 0.04%)</title><rect x="278.4" y="1877" width="0.5" height="15.0" fill="rgb(218,4,18)" rx="2" ry="2" />
<text x="281.42" y="1887.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (47 samples, 0.02%)</title><rect x="717.7" y="2037" width="0.2" height="15.0" fill="rgb(232,210,16)" rx="2" ry="2" />
<text x="720.74" y="2047.5" ></text>
</g>
<g >
<title>__tcp_transmit_skb (26 samples, 0.01%)</title><rect x="1165.3" y="1877" width="0.1" height="15.0" fill="rgb(206,155,10)" rx="2" ry="2" />
<text x="1168.29" y="1887.5" ></text>
</g>
<g >
<title>strncpy_from_user (194 samples, 0.07%)</title><rect x="700.7" y="1941" width="0.8" height="15.0" fill="rgb(234,57,54)" rx="2" ry="2" />
<text x="703.69" y="1951.5" ></text>
</g>
<g >
<title>smc_setsockopt (5,694 samples, 2.00%)</title><rect x="53.3" y="1861" width="23.6" height="15.0" fill="rgb(252,35,49)" rx="2" ry="2" />
<text x="56.32" y="1871.5" >s..</text>
</g>
<g >
<title>read (254 samples, 0.09%)</title><rect x="810.3" y="181" width="1.1" height="15.0" fill="rgb(242,3,36)" rx="2" ry="2" />
<text x="813.35" y="191.5" ></text>
</g>
<g >
<title>release_sock (90 samples, 0.03%)</title><rect x="255.3" y="1893" width="0.3" height="15.0" fill="rgb(229,217,3)" rx="2" ry="2" />
<text x="258.27" y="1903.5" ></text>
</g>
<g >
<title>__follow_mount_rcu (49 samples, 0.02%)</title><rect x="662.3" y="1877" width="0.2" height="15.0" fill="rgb(233,150,33)" rx="2" ry="2" />
<text x="665.35" y="1887.5" ></text>
</g>
<g >
<title>exec_mmap (29 samples, 0.01%)</title><rect x="717.3" y="1893" width="0.1" height="15.0" fill="rgb(218,59,19)" rx="2" ry="2" />
<text x="720.30" y="1903.5" ></text>
</g>
<g >
<title>try_to_unlazy (173 samples, 0.06%)</title><rect x="655.3" y="1893" width="0.7" height="15.0" fill="rgb(232,135,29)" rx="2" ry="2" />
<text x="658.26" y="1903.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (341 samples, 0.12%)</title><rect x="1144.7" y="101" width="1.5" height="15.0" fill="rgb(210,53,54)" rx="2" ry="2" />
<text x="1147.74" y="111.5" ></text>
</g>
<g >
<title>__get_vm_area_node (28 samples, 0.01%)</title><rect x="716.9" y="1909" width="0.1" height="15.0" fill="rgb(249,98,41)" rx="2" ry="2" />
<text x="719.87" y="1919.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (37 samples, 0.01%)</title><rect x="78.6" y="1829" width="0.2" height="15.0" fill="rgb(224,86,10)" rx="2" ry="2" />
<text x="81.64" y="1839.5" ></text>
</g>
<g >
<title>__sys_setsockopt (1,386 samples, 0.49%)</title><rect x="251.4" y="1941" width="5.7" height="15.0" fill="rgb(213,52,48)" rx="2" ry="2" />
<text x="254.39" y="1951.5" ></text>
</g>
<g >
<title>[ld-linux-x86-64.so.2] (124 samples, 0.04%)</title><rect x="719.5" y="2021" width="0.5" height="15.0" fill="rgb(207,40,50)" rx="2" ry="2" />
<text x="722.46" y="2031.5" ></text>
</g>
<g >
<title>smc_close_wake_tx_prepared (33 samples, 0.01%)</title><rect x="56.5" y="1797" width="0.1" height="15.0" fill="rgb(236,197,48)" rx="2" ry="2" />
<text x="59.46" y="1807.5" ></text>
</g>
<g >
<title>ep_poll (285 samples, 0.10%)</title><rect x="245.7" y="1925" width="1.2" height="15.0" fill="rgb(213,79,3)" rx="2" ry="2" />
<text x="248.74" y="1935.5" ></text>
</g>
<g >
<title>kvm_kick_cpu (349 samples, 0.12%)</title><rect x="642.2" y="1749" width="1.4" height="15.0" fill="rgb(230,145,31)" rx="2" ry="2" />
<text x="645.17" y="1759.5" ></text>
</g>
<g >
<title>do_syscall_64 (881 samples, 0.31%)</title><rect x="812.9" y="165" width="3.7" height="15.0" fill="rgb(240,126,13)" rx="2" ry="2" />
<text x="815.91" y="175.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (1,034 samples, 0.36%)</title><rect x="1167.2" y="2037" width="4.2" height="15.0" fill="rgb(208,161,37)" rx="2" ry="2" />
<text x="1170.17" y="2047.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="725" width="364.1" height="15.0" fill="rgb(253,229,54)" rx="2" ry="2" />
<text x="790.11" y="735.5" >[unknown]</text>
</g>
<g >
<title>__x64_sys_openat (3,984 samples, 1.40%)</title><rect x="685.2" y="1989" width="16.4" height="15.0" fill="rgb(218,215,41)" rx="2" ry="2" />
<text x="688.15" y="1999.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (131 samples, 0.05%)</title><rect x="823.1" y="101" width="0.5" height="15.0" fill="rgb(206,115,3)" rx="2" ry="2" />
<text x="826.08" y="111.5" ></text>
</g>
<g >
<title>[unknown] (153 samples, 0.05%)</title><rect x="719.4" y="2053" width="0.6" height="15.0" fill="rgb(229,96,8)" rx="2" ry="2" />
<text x="722.38" y="2063.5" ></text>
</g>
<g >
<title>close_fd_get_file (168 samples, 0.06%)</title><rect x="263.0" y="1957" width="0.7" height="15.0" fill="rgb(228,63,26)" rx="2" ry="2" />
<text x="265.97" y="1967.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (88 samples, 0.03%)</title><rect x="245.0" y="1845" width="0.3" height="15.0" fill="rgb(216,80,36)" rx="2" ry="2" />
<text x="247.98" y="1855.5" ></text>
</g>
<g >
<title>__raw_callee_save___pv_queued_spin_unlock (108 samples, 0.04%)</title><rect x="828.1" y="37" width="0.5" height="15.0" fill="rgb(252,177,42)" rx="2" ry="2" />
<text x="831.13" y="47.5" ></text>
</g>
<g >
<title>file_ra_state_init (132 samples, 0.05%)</title><rect x="691.4" y="1893" width="0.5" height="15.0" fill="rgb(220,32,33)" rx="2" ry="2" />
<text x="694.37" y="1903.5" ></text>
</g>
<g >
<title>vfs_write (36,168 samples, 12.68%)</title><rect x="842.6" y="197" width="149.6" height="15.0" fill="rgb(250,27,5)" rx="2" ry="2" />
<text x="845.56" y="207.5" >vfs_write</text>
</g>
<g >
<title>ngx_output_chain_to_iovec (29 samples, 0.01%)</title><rect x="269.6" y="2021" width="0.2" height="15.0" fill="rgb(207,171,26)" rx="2" ry="2" />
<text x="272.65" y="2031.5" ></text>
</g>
<g >
<title>smc_tx_sndbuf_nonempty (55 samples, 0.02%)</title><rect x="822.9" y="85" width="0.2" height="15.0" fill="rgb(217,200,2)" rx="2" ry="2" />
<text x="825.86" y="95.5" ></text>
</g>
<g >
<title>vfs_write (223 samples, 0.08%)</title><rect x="803.8" y="69" width="0.9" height="15.0" fill="rgb(252,107,4)" rx="2" ry="2" />
<text x="806.82" y="79.5" ></text>
</g>
<g >
<title>expr (93 samples, 0.03%)</title><rect x="10.0" y="2069" width="0.4" height="15.0" fill="rgb(231,218,48)" rx="2" ry="2" />
<text x="13.03" y="2079.5" ></text>
</g>
<g >
<title>ngx_reusable_connection (229 samples, 0.08%)</title><rect x="714.4" y="2053" width="1.0" height="15.0" fill="rgb(247,190,24)" rx="2" ry="2" />
<text x="717.42" y="2063.5" ></text>
</g>
<g >
<title>write (190 samples, 0.07%)</title><rect x="822.3" y="213" width="0.8" height="15.0" fill="rgb(216,217,38)" rx="2" ry="2" />
<text x="825.30" y="223.5" ></text>
</g>
<g >
<title>vfs_read (250 samples, 0.09%)</title><rect x="802.5" y="69" width="1.0" height="15.0" fill="rgb(253,2,46)" rx="2" ry="2" />
<text x="805.47" y="79.5" ></text>
</g>
<g >
<title>open_last_lookups (148 samples, 0.05%)</title><rect x="698.2" y="1925" width="0.6" height="15.0" fill="rgb(219,147,13)" rx="2" ry="2" />
<text x="701.22" y="1935.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="869" width="364.1" height="15.0" fill="rgb(223,164,53)" rx="2" ry="2" />
<text x="790.11" y="879.5" >[unknown]</text>
</g>
<g >
<title>__check_object_size.part.0 (37,631 samples, 13.19%)</title><rect x="86.2" y="1845" width="155.7" height="15.0" fill="rgb(209,43,28)" rx="2" ry="2" />
<text x="89.23" y="1855.5" >__check_object_size..</text>
</g>
<g >
<title>__fget_light (147 samples, 0.05%)</title><rect x="1174.6" y="1973" width="0.6" height="15.0" fill="rgb(221,180,19)" rx="2" ry="2" />
<text x="1177.58" y="1983.5" ></text>
</g>
<g >
<title>[unknown] (2,240 samples, 0.79%)</title><rect x="787.1" y="53" width="9.3" height="15.0" fill="rgb(222,203,52)" rx="2" ry="2" />
<text x="790.12" y="63.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (41 samples, 0.01%)</title><rect x="474.7" y="1893" width="0.1" height="15.0" fill="rgb(248,13,43)" rx="2" ry="2" />
<text x="477.67" y="1903.5" ></text>
</g>
<g >
<title>terminate_walk (99 samples, 0.03%)</title><rect x="699.2" y="1925" width="0.4" height="15.0" fill="rgb(216,115,53)" rx="2" ry="2" />
<text x="702.17" y="1935.5" ></text>
</g>
<g >
<title>socket_writeable (141 samples, 0.05%)</title><rect x="793.0" y="37" width="0.6" height="15.0" fill="rgb(247,86,12)" rx="2" ry="2" />
<text x="796.02" y="47.5" ></text>
</g>
<g >
<title>[[kernel.vmlinux].init.text] (892 samples, 0.31%)</title><rect x="782.2" y="1989" width="3.7" height="15.0" fill="rgb(228,105,35)" rx="2" ry="2" />
<text x="785.17" y="1999.5" ></text>
</g>
<g >
<title>smc_lo_move_data (118 samples, 0.04%)</title><rect x="828.6" y="37" width="0.5" height="15.0" fill="rgb(253,120,33)" rx="2" ry="2" />
<text x="831.59" y="47.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,201 samples, 0.42%)</title><rect x="829.1" y="213" width="5.0" height="15.0" fill="rgb(254,44,20)" rx="2" ry="2" />
<text x="832.08" y="223.5" ></text>
</g>
<g >
<title>ksys_write (253 samples, 0.09%)</title><rect x="811.4" y="133" width="1.0" height="15.0" fill="rgb(239,142,0)" rx="2" ry="2" />
<text x="814.40" y="143.5" ></text>
</g>
<g >
<title>write (477 samples, 0.17%)</title><rect x="805.6" y="149" width="2.0" height="15.0" fill="rgb(217,170,3)" rx="2" ry="2" />
<text x="808.63" y="159.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (64 samples, 0.02%)</title><rect x="648.3" y="2021" width="0.3" height="15.0" fill="rgb(228,62,33)" rx="2" ry="2" />
<text x="651.29" y="2031.5" ></text>
</g>
<g >
<title>sock_write_iter (39,706 samples, 13.92%)</title><rect x="480.9" y="1909" width="164.2" height="15.0" fill="rgb(252,126,13)" rx="2" ry="2" />
<text x="483.85" y="1919.5" >sock_write_iter</text>
</g>
<g >
<title>ksys_read (369 samples, 0.13%)</title><rect x="802.0" y="85" width="1.5" height="15.0" fill="rgb(241,131,49)" rx="2" ry="2" />
<text x="804.97" y="95.5" ></text>
</g>
<g >
<title>execve (29 samples, 0.01%)</title><rect x="717.3" y="2053" width="0.1" height="15.0" fill="rgb(230,92,18)" rx="2" ry="2" />
<text x="720.30" y="2063.5" ></text>
</g>
<g >
<title>rep_movs_alternative (820 samples, 0.29%)</title><rect x="813.1" y="37" width="3.3" height="15.0" fill="rgb(252,66,31)" rx="2" ry="2" />
<text x="816.05" y="47.5" ></text>
</g>
<g >
<title>do_syscall_64 (63 samples, 0.02%)</title><rect x="1165.3" y="2021" width="0.2" height="15.0" fill="rgb(215,47,46)" rx="2" ry="2" />
<text x="1168.25" y="2031.5" ></text>
</g>
<g >
<title>_copy_from_iter (1,057 samples, 0.37%)</title><rect x="465.2" y="1829" width="4.4" height="15.0" fill="rgb(220,225,22)" rx="2" ry="2" />
<text x="468.22" y="1839.5" ></text>
</g>
<g >
<title>rep_movs_alternative (36 samples, 0.01%)</title><rect x="1153.7" y="1957" width="0.2" height="15.0" fill="rgb(220,51,16)" rx="2" ry="2" />
<text x="1156.74" y="1967.5" ></text>
</g>
<g >
<title>_raw_spin_unlock (85 samples, 0.03%)</title><rect x="833.7" y="53" width="0.4" height="15.0" fill="rgb(241,220,23)" rx="2" ry="2" />
<text x="836.70" y="63.5" ></text>
</g>
<g >
<title>kvm_kick_cpu (381 samples, 0.13%)</title><rect x="1146.7" y="37" width="1.6" height="15.0" fill="rgb(225,41,44)" rx="2" ry="2" />
<text x="1149.71" y="47.5" ></text>
</g>
<g >
<title>[unknown] (88,032 samples, 30.86%)</title><rect x="787.1" y="661" width="364.1" height="15.0" fill="rgb(217,96,41)" rx="2" ry="2" />
<text x="790.11" y="671.5" >[unknown]</text>
</g>
<g >
<title>epoll_wait (4,199 samples, 1.47%)</title><rect x="1171.4" y="2053" width="17.4" height="15.0" fill="rgb(254,84,25)" rx="2" ry="2" />
<text x="1174.44" y="2063.5" ></text>
</g>
<g >
<title>posix_memalign (82 samples, 0.03%)</title><rect x="702.9" y="2037" width="0.4" height="15.0" fill="rgb(216,149,42)" rx="2" ry="2" />
<text x="705.92" y="2047.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (360 samples, 0.13%)</title><rect x="245.6" y="1989" width="1.5" height="15.0" fill="rgb(207,3,12)" rx="2" ry="2" />
<text x="248.62" y="1999.5" ></text>
</g>
<g >
<title>mmap_region (37 samples, 0.01%)</title><rect x="717.8" y="1957" width="0.1" height="15.0" fill="rgb(224,54,18)" rx="2" ry="2" />
<text x="720.75" y="1967.5" ></text>
</g>
<g >
<title>__fdget_pos (118 samples, 0.04%)</title><rect x="479.4" y="1957" width="0.5" height="15.0" fill="rgb(243,67,26)" rx="2" ry="2" />
<text x="482.44" y="1967.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1717" width="364.1" height="15.0" fill="rgb(254,196,36)" rx="2" ry="2" />
<text x="790.11" y="1727.5" >[unknown]</text>
</g>
<g >
<title>find_vmap_area (888 samples, 0.31%)</title><rect x="838.9" y="85" width="3.7" height="15.0" fill="rgb(243,19,39)" rx="2" ry="2" />
<text x="841.89" y="95.5" ></text>
</g>
<g >
<title>do_syscall_64 (254 samples, 0.09%)</title><rect x="810.3" y="149" width="1.1" height="15.0" fill="rgb(217,55,41)" rx="2" ry="2" />
<text x="813.35" y="159.5" ></text>
</g>
<g >
<title>selinux_inode_permission (537 samples, 0.19%)</title><rect x="695.3" y="1893" width="2.2" height="15.0" fill="rgb(215,215,8)" rx="2" ry="2" />
<text x="698.27" y="1903.5" ></text>
</g>
<g >
<title>ngx_output_chain_to_iovec (50 samples, 0.02%)</title><rect x="248.5" y="2005" width="0.2" height="15.0" fill="rgb(222,54,47)" rx="2" ry="2" />
<text x="251.49" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1349" width="364.1" height="15.0" fill="rgb(231,89,39)" rx="2" ry="2" />
<text x="790.11" y="1359.5" >[unknown]</text>
</g>
<g >
<title>read (65 samples, 0.02%)</title><rect x="799.3" y="101" width="0.3" height="15.0" fill="rgb(223,1,33)" rx="2" ry="2" />
<text x="802.28" y="111.5" ></text>
</g>
<g >
<title>lookup_fast (132 samples, 0.05%)</title><rect x="665.3" y="1893" width="0.5" height="15.0" fill="rgb(224,91,37)" rx="2" ry="2" />
<text x="668.26" y="1903.5" ></text>
</g>
<g >
<title>pv_wait_node (35,141 samples, 12.32%)</title><rect x="94.7" y="1765" width="145.4" height="15.0" fill="rgb(235,11,9)" rx="2" ry="2" />
<text x="97.74" y="1775.5" >pv_wait_node</text>
</g>
<g >
<title>ngx_pnalloc (40 samples, 0.01%)</title><rect x="680.5" y="2037" width="0.2" height="15.0" fill="rgb(223,102,8)" rx="2" ry="2" />
<text x="683.52" y="2047.5" ></text>
</g>
<g >
<title>gawk (91 samples, 0.03%)</title><rect x="10.4" y="2069" width="0.4" height="15.0" fill="rgb(219,39,0)" rx="2" ry="2" />
<text x="13.42" y="2079.5" ></text>
</g>
<g >
<title>do_tcp_setsockopt (265 samples, 0.09%)</title><rect x="253.8" y="1893" width="1.1" height="15.0" fill="rgb(211,152,30)" rx="2" ry="2" />
<text x="256.84" y="1903.5" ></text>
</g>
<g >
<title>pv_wait_head_or_lock (83 samples, 0.03%)</title><rect x="288.6" y="1749" width="0.3" height="15.0" fill="rgb(244,78,30)" rx="2" ry="2" />
<text x="291.57" y="1759.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1909" width="364.1" height="15.0" fill="rgb(247,43,41)" rx="2" ry="2" />
<text x="790.11" y="1919.5" >[unknown]</text>
</g>
<g >
<title>path_lookupat (2,579 samples, 0.90%)</title><rect x="655.1" y="1925" width="10.7" height="15.0" fill="rgb(218,116,51)" rx="2" ry="2" />
<text x="658.14" y="1935.5" ></text>
</g>
<g >
<title>vfs_fstatat (3,941 samples, 1.38%)</title><rect x="651.0" y="1973" width="16.3" height="15.0" fill="rgb(231,114,36)" rx="2" ry="2" />
<text x="654.04" y="1983.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (211 samples, 0.07%)</title><rect x="475.7" y="1973" width="0.9" height="15.0" fill="rgb(253,120,4)" rx="2" ry="2" />
<text x="478.68" y="1983.5" ></text>
</g>
<g >
<title>kmem_cache_alloc (102 samples, 0.04%)</title><rect x="700.3" y="1941" width="0.4" height="15.0" fill="rgb(223,135,9)" rx="2" ry="2" />
<text x="703.27" y="1951.5" ></text>
</g>
<g >
<title>schedule_hrtimeout_range_clock (1,435 samples, 0.50%)</title><rect x="1180.0" y="1957" width="5.9" height="15.0" fill="rgb(232,53,15)" rx="2" ry="2" />
<text x="1182.99" y="1967.5" ></text>
</g>
<g >
<title>smc_ism_dmb_mappable (26 samples, 0.01%)</title><rect x="57.6" y="1781" width="0.1" height="15.0" fill="rgb(214,184,15)" rx="2" ry="2" />
<text x="60.60" y="1791.5" ></text>
</g>
<g >
<title>__fget_light (195 samples, 0.07%)</title><rect x="243.0" y="1877" width="0.8" height="15.0" fill="rgb(249,93,36)" rx="2" ry="2" />
<text x="246.04" y="1887.5" ></text>
</g>
<g >
<title>__x64_sys_execve (29 samples, 0.01%)</title><rect x="717.3" y="2005" width="0.1" height="15.0" fill="rgb(205,117,25)" rx="2" ry="2" />
<text x="720.30" y="2015.5" ></text>
</g>
<g >
<title>write (475 samples, 0.17%)</title><rect x="1144.7" y="277" width="2.0" height="15.0" fill="rgb(216,117,23)" rx="2" ry="2" />
<text x="1147.74" y="287.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1765" width="364.1" height="15.0" fill="rgb(244,36,13)" rx="2" ry="2" />
<text x="790.11" y="1775.5" >[unknown]</text>
</g>
<g >
<title>security_file_permission (94 samples, 0.03%)</title><rect x="805.6" y="69" width="0.4" height="15.0" fill="rgb(229,181,11)" rx="2" ry="2" />
<text x="808.63" y="79.5" ></text>
</g>
<g >
<title>security_socket_sendmsg (174 samples, 0.06%)</title><rect x="279.1" y="1861" width="0.8" height="15.0" fill="rgb(213,148,37)" rx="2" ry="2" />
<text x="282.15" y="1871.5" ></text>
</g>
<g >
<title>__d_lookup_rcu (125 samples, 0.04%)</title><rect x="697.7" y="1877" width="0.5" height="15.0" fill="rgb(238,25,1)" rx="2" ry="2" />
<text x="700.70" y="1887.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="885" width="364.1" height="15.0" fill="rgb(236,162,18)" rx="2" ry="2" />
<text x="790.11" y="895.5" >[unknown]</text>
</g>
<g >
<title>selinux_current_getsecid_subj (41 samples, 0.01%)</title><rect x="693.2" y="1877" width="0.1" height="15.0" fill="rgb(227,19,35)" rx="2" ry="2" />
<text x="696.16" y="1887.5" ></text>
</g>
<g >
<title>avc_lookup (636 samples, 0.22%)</title><rect x="659.6" y="1845" width="2.7" height="15.0" fill="rgb(208,220,43)" rx="2" ry="2" />
<text x="662.64" y="1855.5" ></text>
</g>
<g >
<title>ngx_rbtree_delete (71 samples, 0.02%)</title><rect x="270.3" y="2021" width="0.3" height="15.0" fill="rgb(242,62,9)" rx="2" ry="2" />
<text x="273.32" y="2031.5" ></text>
</g>
<g >
<title>smc_sendmsg (473 samples, 0.17%)</title><rect x="1148.7" y="213" width="1.9" height="15.0" fill="rgb(231,154,31)" rx="2" ry="2" />
<text x="1151.67" y="223.5" ></text>
</g>
<g >
<title>memcpy (46 samples, 0.02%)</title><rect x="710.8" y="2053" width="0.1" height="15.0" fill="rgb(249,212,1)" rx="2" ry="2" />
<text x="713.76" y="2063.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1141" width="364.1" height="15.0" fill="rgb(238,12,28)" rx="2" ry="2" />
<text x="790.11" y="1151.5" >[unknown]</text>
</g>
<g >
<title>avc_has_perm_noaudit (65 samples, 0.02%)</title><rect x="279.6" y="1813" width="0.3" height="15.0" fill="rgb(250,114,1)" rx="2" ry="2" />
<text x="282.60" y="1823.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (36,167 samples, 12.68%)</title><rect x="842.6" y="149" width="149.6" height="15.0" fill="rgb(206,129,20)" rx="2" ry="2" />
<text x="845.56" y="159.5" >smc_tx_sendmsg</text>
</g>
<g >
<title>[unknown] (87,936 samples, 30.82%)</title><rect x="787.1" y="357" width="363.7" height="15.0" fill="rgb(243,0,38)" rx="2" ry="2" />
<text x="790.11" y="367.5" >[unknown]</text>
</g>
<g >
<title>do_syscall_64 (39,368 samples, 13.80%)</title><rect x="82.6" y="1941" width="162.8" height="15.0" fill="rgb(215,181,6)" rx="2" ry="2" />
<text x="85.59" y="1951.5" >do_syscall_64</text>
</g>
<g >
<title>do_user_addr_fault (28 samples, 0.01%)</title><rect x="719.6" y="1957" width="0.1" height="15.0" fill="rgb(251,145,31)" rx="2" ry="2" />
<text x="722.57" y="1967.5" ></text>
</g>
<g >
<title>ksys_write (473 samples, 0.17%)</title><rect x="1148.7" y="261" width="1.9" height="15.0" fill="rgb(205,87,39)" rx="2" ry="2" />
<text x="1151.67" y="271.5" ></text>
</g>
<g >
<title>__x64_sys_execve (29 samples, 0.01%)</title><rect x="716.5" y="1989" width="0.1" height="15.0" fill="rgb(211,6,37)" rx="2" ry="2" />
<text x="719.45" y="1999.5" ></text>
</g>
<g >
<title>__pv_queued_spin_unlock_slowpath (332 samples, 0.12%)</title><rect x="240.1" y="1749" width="1.4" height="15.0" fill="rgb(211,206,41)" rx="2" ry="2" />
<text x="243.11" y="1759.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1365" width="364.1" height="15.0" fill="rgb(220,58,0)" rx="2" ry="2" />
<text x="790.11" y="1375.5" >[unknown]</text>
</g>
<g >
<title>smc_connect (59 samples, 0.02%)</title><rect x="1165.3" y="1973" width="0.2" height="15.0" fill="rgb(233,39,24)" rx="2" ry="2" />
<text x="1168.26" y="1983.5" ></text>
</g>
<g >
<title>memcpy (46 samples, 0.02%)</title><rect x="267.2" y="2021" width="0.2" height="15.0" fill="rgb(240,170,3)" rx="2" ry="2" />
<text x="270.21" y="2031.5" ></text>
</g>
<g >
<title>__fget_light (109 samples, 0.04%)</title><rect x="800.2" y="37" width="0.4" height="15.0" fill="rgb(208,99,13)" rx="2" ry="2" />
<text x="803.15" y="47.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (280 samples, 0.10%)</title><rect x="265.4" y="1973" width="1.1" height="15.0" fill="rgb(254,157,36)" rx="2" ry="2" />
<text x="268.35" y="1983.5" ></text>
</g>
<g >
<title>pv_wait_node (821 samples, 0.29%)</title><rect x="839.2" y="37" width="3.4" height="15.0" fill="rgb(223,189,19)" rx="2" ry="2" />
<text x="842.16" y="47.5" ></text>
</g>
<g >
<title>smc_recvmsg (381 samples, 0.13%)</title><rect x="1146.7" y="181" width="1.6" height="15.0" fill="rgb(220,52,19)" rx="2" ry="2" />
<text x="1149.71" y="191.5" ></text>
</g>
<g >
<title>[unknown] (146,141 samples, 51.22%)</title><rect x="43.8" y="2037" width="604.5" height="15.0" fill="rgb(249,158,7)" rx="2" ry="2" />
<text x="46.82" y="2047.5" >[unknown]</text>
</g>
<g >
<title>walk_component (148 samples, 0.05%)</title><rect x="697.6" y="1909" width="0.6" height="15.0" fill="rgb(238,108,40)" rx="2" ry="2" />
<text x="700.61" y="1919.5" ></text>
</g>
<g >
<title>start_secondary (14,885 samples, 5.22%)</title><rect x="720.6" y="2037" width="61.6" height="15.0" fill="rgb(218,138,23)" rx="2" ry="2" />
<text x="723.60" y="2047.5" >start_..</text>
</g>
<g >
<title>__do_sys_clone (74 samples, 0.03%)</title><rect x="716.7" y="2005" width="0.3" height="15.0" fill="rgb(215,114,5)" rx="2" ry="2" />
<text x="719.73" y="2015.5" ></text>
</g>
<g >
<title>vfs_read (36,893 samples, 12.93%)</title><rect x="992.2" y="213" width="152.5" height="15.0" fill="rgb(206,206,52)" rx="2" ry="2" />
<text x="995.15" y="223.5" >vfs_read</text>
</g>
<g >
<title>[unknown] (8,698 samples, 3.05%)</title><rect x="787.1" y="229" width="36.0" height="15.0" fill="rgb(225,50,18)" rx="2" ry="2" />
<text x="790.11" y="239.5" >[un..</text>
</g>
<g >
<title>write (301 samples, 0.11%)</title><rect x="803.5" y="133" width="1.2" height="15.0" fill="rgb(249,21,29)" rx="2" ry="2" />
<text x="806.50" y="143.5" ></text>
</g>
<g >
<title>default_idle (875 samples, 0.31%)</title><rect x="782.2" y="1893" width="3.6" height="15.0" fill="rgb(217,109,7)" rx="2" ry="2" />
<text x="785.17" y="1903.5" ></text>
</g>
<g >
<title>fsnotify_perm.part.0 (134 samples, 0.05%)</title><rect x="800.6" y="37" width="0.6" height="15.0" fill="rgb(230,139,3)" rx="2" ry="2" />
<text x="803.63" y="47.5" ></text>
</g>
<g >
<title>dquot_file_open (25 samples, 0.01%)</title><rect x="690.7" y="1893" width="0.1" height="15.0" fill="rgb(218,200,9)" rx="2" ry="2" />
<text x="693.73" y="1903.5" ></text>
</g>
<g >
<title>__fput (112 samples, 0.04%)</title><rect x="812.4" y="133" width="0.5" height="15.0" fill="rgb(250,73,5)" rx="2" ry="2" />
<text x="815.45" y="143.5" ></text>
</g>
<g >
<title>[unknown] (87,908 samples, 30.81%)</title><rect x="787.1" y="341" width="363.6" height="15.0" fill="rgb(215,152,34)" rx="2" ry="2" />
<text x="790.11" y="351.5" >[unknown]</text>
</g>
<g >
<title>ktime_get_coarse_real_ts64 (64 samples, 0.02%)</title><rect x="473.4" y="1845" width="0.3" height="15.0" fill="rgb(244,93,42)" rx="2" ry="2" />
<text x="476.41" y="1855.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (108 samples, 0.04%)</title><rect x="647.7" y="1957" width="0.5" height="15.0" fill="rgb(225,37,19)" rx="2" ry="2" />
<text x="650.73" y="1967.5" ></text>
</g>
<g >
<title>acl_permission_check (57 samples, 0.02%)</title><rect x="693.5" y="1861" width="0.2" height="15.0" fill="rgb(223,4,38)" rx="2" ry="2" />
<text x="696.49" y="1871.5" ></text>
</g>
<g >
<title>free_unref_page_list (32 samples, 0.01%)</title><rect x="471.3" y="1861" width="0.1" height="15.0" fill="rgb(229,11,27)" rx="2" ry="2" />
<text x="474.27" y="1871.5" ></text>
</g>
<g >
<title>pmlogger_check (380 samples, 0.13%)</title><rect x="716.0" y="2069" width="1.6" height="15.0" fill="rgb(214,11,18)" rx="2" ry="2" />
<text x="718.99" y="2079.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (86 samples, 0.03%)</title><rect x="1148.3" y="85" width="0.3" height="15.0" fill="rgb(241,167,47)" rx="2" ry="2" />
<text x="1151.28" y="95.5" ></text>
</g>
<g >
<title>do_syscall_64 (29 samples, 0.01%)</title><rect x="715.5" y="1909" width="0.2" height="15.0" fill="rgb(224,47,24)" rx="2" ry="2" />
<text x="718.53" y="1919.5" ></text>
</g>
<g >
<title>__x64_sys_close (112 samples, 0.04%)</title><rect x="812.4" y="149" width="0.5" height="15.0" fill="rgb(207,223,43)" rx="2" ry="2" />
<text x="815.45" y="159.5" ></text>
</g>
<g >
<title>__fget_light (148 samples, 0.05%)</title><rect x="706.4" y="1973" width="0.6" height="15.0" fill="rgb(229,175,38)" rx="2" ry="2" />
<text x="709.38" y="1983.5" ></text>
</g>
<g >
<title>splice_to_socket (46,485 samples, 16.29%)</title><rect x="277.5" y="1893" width="192.3" height="15.0" fill="rgb(217,23,0)" rx="2" ry="2" />
<text x="280.50" y="1903.5" >splice_to_socket</text>
</g>
<g >
<title>write (168 samples, 0.06%)</title><rect x="809.6" y="165" width="0.7" height="15.0" fill="rgb(234,52,10)" rx="2" ry="2" />
<text x="812.65" y="175.5" ></text>
</g>
<g >
<title>http_should_keep_alive (36 samples, 0.01%)</title><rect x="789.0" y="37" width="0.1" height="15.0" fill="rgb(222,13,9)" rx="2" ry="2" />
<text x="791.96" y="47.5" ></text>
</g>
<g >
<title>__put_user_nocheck_4 (88 samples, 0.03%)</title><rect x="1176.2" y="1941" width="0.3" height="15.0" fill="rgb(210,199,27)" rx="2" ry="2" />
<text x="1179.18" y="1951.5" ></text>
</g>
<g >
<title>__smc_release (109 samples, 0.04%)</title><rect x="812.4" y="69" width="0.5" height="15.0" fill="rgb(231,203,20)" rx="2" ry="2" />
<text x="815.45" y="79.5" ></text>
</g>
<g >
<title>ngx_chain_update_sent (61 samples, 0.02%)</title><rect x="247.1" y="2005" width="0.3" height="15.0" fill="rgb(225,71,32)" rx="2" ry="2" />
<text x="250.12" y="2015.5" ></text>
</g>
<g >
<title>irq_exit_rcu (26 samples, 0.01%)</title><rect x="785.7" y="1845" width="0.1" height="15.0" fill="rgb(224,0,4)" rx="2" ry="2" />
<text x="788.68" y="1855.5" ></text>
</g>
<g >
<title>_copy_to_user (67 samples, 0.02%)</title><rect x="706.0" y="1973" width="0.3" height="15.0" fill="rgb(212,23,2)" rx="2" ry="2" />
<text x="709.01" y="1983.5" ></text>
</g>
<g >
<title>begin_new_exec (29 samples, 0.01%)</title><rect x="717.3" y="1909" width="0.1" height="15.0" fill="rgb(222,211,12)" rx="2" ry="2" />
<text x="720.30" y="1919.5" ></text>
</g>
<g >
<title>socket_readable (50 samples, 0.02%)</title><rect x="792.8" y="37" width="0.2" height="15.0" fill="rgb(242,10,13)" rx="2" ry="2" />
<text x="795.81" y="47.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (101 samples, 0.04%)</title><rect x="244.9" y="1909" width="0.5" height="15.0" fill="rgb(208,103,1)" rx="2" ry="2" />
<text x="247.94" y="1919.5" ></text>
</g>
<g >
<title>nginx (169,657 samples, 59.47%)</title><rect x="13.8" y="2069" width="701.7" height="15.0" fill="rgb(238,161,41)" rx="2" ry="2" />
<text x="16.80" y="2079.5" >nginx</text>
</g>
<g >
<title>[perf] (34 samples, 0.01%)</title><rect x="715.5" y="1957" width="0.2" height="15.0" fill="rgb(226,219,20)" rx="2" ry="2" />
<text x="718.51" y="1967.5" ></text>
</g>
<g >
<title>avc_has_perm (35 samples, 0.01%)</title><rect x="811.4" y="69" width="0.1" height="15.0" fill="rgb(254,148,30)" rx="2" ry="2" />
<text x="814.40" y="79.5" ></text>
</g>
<g >
<title>x86_64_start_kernel (892 samples, 0.31%)</title><rect x="782.2" y="2037" width="3.7" height="15.0" fill="rgb(217,180,48)" rx="2" ry="2" />
<text x="785.17" y="2047.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (881 samples, 0.31%)</title><rect x="812.9" y="181" width="3.7" height="15.0" fill="rgb(236,210,25)" rx="2" ry="2" />
<text x="815.91" y="191.5" ></text>
</g>
<g >
<title>smc_tx_consumer_update (31 samples, 0.01%)</title><rect x="242.8" y="1829" width="0.1" height="15.0" fill="rgb(246,60,0)" rx="2" ry="2" />
<text x="245.77" y="1839.5" ></text>
</g>
<g >
<title>_nohz_idle_balance.isra.0 (26 samples, 0.01%)</title><rect x="779.6" y="1877" width="0.1" height="15.0" fill="rgb(219,73,27)" rx="2" ry="2" />
<text x="782.56" y="1887.5" ></text>
</g>
<g >
<title>smc_recvmsg (888 samples, 0.31%)</title><rect x="838.9" y="149" width="3.7" height="15.0" fill="rgb(224,204,45)" rx="2" ry="2" />
<text x="841.89" y="159.5" ></text>
</g>
<g >
<title>ngx_create_temp_buf (34 samples, 0.01%)</title><rect x="711.1" y="2053" width="0.1" height="15.0" fill="rgb(234,138,34)" rx="2" ry="2" />
<text x="714.06" y="2063.5" ></text>
</g>
<g >
<title>mutex_lock (40 samples, 0.01%)</title><rect x="1170.2" y="1973" width="0.1" height="15.0" fill="rgb(252,49,44)" rx="2" ry="2" />
<text x="1173.16" y="1983.5" ></text>
</g>
<g >
<title>pv_wait_node (795 samples, 0.28%)</title><rect x="834.3" y="37" width="3.3" height="15.0" fill="rgb(232,130,20)" rx="2" ry="2" />
<text x="837.32" y="47.5" ></text>
</g>
<g >
<title>do_syscall_64 (6,552 samples, 2.30%)</title><rect x="51.8" y="1925" width="27.1" height="15.0" fill="rgb(229,171,24)" rx="2" ry="2" />
<text x="54.79" y="1935.5" >d..</text>
</g>
<g >
<title>do_tcp_setsockopt (196 samples, 0.07%)</title><rect x="54.5" y="1845" width="0.8" height="15.0" fill="rgb(220,138,1)" rx="2" ry="2" />
<text x="57.47" y="1855.5" ></text>
</g>
<g >
<title>selinux_socket_recvmsg (248 samples, 0.09%)</title><rect x="83.4" y="1861" width="1.1" height="15.0" fill="rgb(217,136,2)" rx="2" ry="2" />
<text x="86.45" y="1871.5" ></text>
</g>
<g >
<title>ngx_writev (193 samples, 0.07%)</title><rect x="271.4" y="2021" width="0.8" height="15.0" fill="rgb(253,191,13)" rx="2" ry="2" />
<text x="274.38" y="2031.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (135 samples, 0.05%)</title><rect x="799.6" y="85" width="0.5" height="15.0" fill="rgb(248,81,7)" rx="2" ry="2" />
<text x="802.59" y="95.5" ></text>
</g>
<g >
<title>touch_atime (46 samples, 0.02%)</title><rect x="473.8" y="1909" width="0.2" height="15.0" fill="rgb(228,61,29)" rx="2" ry="2" />
<text x="476.77" y="1919.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (134 samples, 0.05%)</title><rect x="1146.2" y="69" width="0.5" height="15.0" fill="rgb(209,52,13)" rx="2" ry="2" />
<text x="1149.15" y="79.5" ></text>
</g>
<g >
<title>[libc.so.6] (114 samples, 0.04%)</title><rect x="787.2" y="37" width="0.4" height="15.0" fill="rgb(252,69,1)" rx="2" ry="2" />
<text x="790.16" y="47.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (254 samples, 0.09%)</title><rect x="798.2" y="69" width="1.1" height="15.0" fill="rgb(210,151,52)" rx="2" ry="2" />
<text x="801.22" y="79.5" ></text>
</g>
<g >
<title>write (772 samples, 0.27%)</title><rect x="816.6" y="197" width="3.1" height="15.0" fill="rgb(246,64,14)" rx="2" ry="2" />
<text x="819.55" y="207.5" ></text>
</g>
<g >
<title>asm_sysvec_call_function_single (26 samples, 0.01%)</title><rect x="785.7" y="1877" width="0.1" height="15.0" fill="rgb(224,24,52)" rx="2" ry="2" />
<text x="788.68" y="1887.5" ></text>
</g>
<g >
<title>__get_file_rcu (69 samples, 0.02%)</title><rect x="1174.9" y="1957" width="0.3" height="15.0" fill="rgb(208,186,11)" rx="2" ry="2" />
<text x="1177.89" y="1967.5" ></text>
</g>
<g >
<title>__ip_queue_xmit (25 samples, 0.01%)</title><rect x="1165.3" y="1861" width="0.1" height="15.0" fill="rgb(210,179,34)" rx="2" ry="2" />
<text x="1168.29" y="1871.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (86 samples, 0.03%)</title><rect x="1148.3" y="181" width="0.3" height="15.0" fill="rgb(225,62,52)" rx="2" ry="2" />
<text x="1151.28" y="191.5" ></text>
</g>
<g >
<title>filp_flush (184 samples, 0.06%)</title><rect x="263.7" y="1957" width="0.7" height="15.0" fill="rgb(239,104,50)" rx="2" ry="2" />
<text x="266.66" y="1967.5" ></text>
</g>
<g >
<title>open64 (4,907 samples, 1.72%)</title><rect x="682.5" y="2037" width="20.3" height="15.0" fill="rgb(236,50,2)" rx="2" ry="2" />
<text x="685.53" y="2047.5" ></text>
</g>
<g >
<title>generic_permission (155 samples, 0.05%)</title><rect x="657.3" y="1877" width="0.6" height="15.0" fill="rgb(224,48,17)" rx="2" ry="2" />
<text x="660.28" y="1887.5" ></text>
</g>
<g >
<title>sock_recvmsg (613 samples, 0.21%)</title><rect x="819.8" y="117" width="2.5" height="15.0" fill="rgb(208,5,48)" rx="2" ry="2" />
<text x="822.76" y="127.5" ></text>
</g>
<g >
<title>avc_has_perm_noaudit (125 samples, 0.04%)</title><rect x="252.5" y="1861" width="0.5" height="15.0" fill="rgb(246,175,48)" rx="2" ry="2" />
<text x="255.51" y="1871.5" ></text>
</g>
<g >
<title>_raw_spin_lock_bh (77 samples, 0.03%)</title><rect x="810.7" y="37" width="0.3" height="15.0" fill="rgb(247,70,27)" rx="2" ry="2" />
<text x="813.65" y="47.5" ></text>
</g>
<g >
<title>__pv_queued_spin_lock_slowpath (36,311 samples, 12.73%)</title><rect x="89.9" y="1781" width="150.2" height="15.0" fill="rgb(249,211,19)" rx="2" ry="2" />
<text x="92.90" y="1791.5" >__pv_queued_spin_lo..</text>
</g>
<g >
<title>__check_object_size.part.0 (341 samples, 0.12%)</title><rect x="1144.7" y="149" width="1.5" height="15.0" fill="rgb(210,207,17)" rx="2" ry="2" />
<text x="1147.74" y="159.5" ></text>
</g>
<g >
<title>vfs_statx (3,224 samples, 1.13%)</title><rect x="654.0" y="1957" width="13.3" height="15.0" fill="rgb(230,136,38)" rx="2" ry="2" />
<text x="657.00" y="1967.5" ></text>
</g>
<g >
<title>sock_recvmsg (36,893 samples, 12.93%)</title><rect x="992.2" y="181" width="152.5" height="15.0" fill="rgb(235,14,26)" rx="2" ry="2" />
<text x="995.15" y="191.5" >sock_recvmsg</text>
</g>
<g >
<title>find_vmap_area (37,595 samples, 13.18%)</title><rect x="86.3" y="1813" width="155.5" height="15.0" fill="rgb(253,137,5)" rx="2" ry="2" />
<text x="89.32" y="1823.5" >find_vmap_area</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1221" width="364.1" height="15.0" fill="rgb(226,124,33)" rx="2" ry="2" />
<text x="790.11" y="1231.5" >[unknown]</text>
</g>
<g >
<title>smc_sndbuf_sync_sg_for_device (42 samples, 0.01%)</title><rect x="644.9" y="1861" width="0.2" height="15.0" fill="rgb(237,53,48)" rx="2" ry="2" />
<text x="647.90" y="1871.5" ></text>
</g>
<g >
<title>smc_release_cb (57 samples, 0.02%)</title><rect x="255.4" y="1877" width="0.2" height="15.0" fill="rgb(229,165,39)" rx="2" ry="2" />
<text x="258.41" y="1887.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (75 samples, 0.03%)</title><rect x="716.7" y="2037" width="0.3" height="15.0" fill="rgb(221,102,1)" rx="2" ry="2" />
<text x="719.73" y="2047.5" ></text>
</g>
<g >
<title>stat64 (65 samples, 0.02%)</title><rect x="703.3" y="2037" width="0.3" height="15.0" fill="rgb(234,116,49)" rx="2" ry="2" />
<text x="706.31" y="2047.5" ></text>
</g>
<g >
<title>mntput_no_expire (29 samples, 0.01%)</title><rect x="262.7" y="1941" width="0.1" height="15.0" fill="rgb(242,158,45)" rx="2" ry="2" />
<text x="265.71" y="1951.5" ></text>
</g>
<g >
<title>do_accept (40 samples, 0.01%)</title><rect x="648.3" y="1957" width="0.2" height="15.0" fill="rgb(240,23,47)" rx="2" ry="2" />
<text x="651.32" y="1967.5" ></text>
</g>
<g >
<title>[perf] (43 samples, 0.02%)</title><rect x="715.5" y="2021" width="0.2" height="15.0" fill="rgb(239,5,9)" rx="2" ry="2" />
<text x="718.50" y="2031.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (381 samples, 0.13%)</title><rect x="1146.7" y="149" width="1.6" height="15.0" fill="rgb(226,163,31)" rx="2" ry="2" />
<text x="1149.71" y="159.5" ></text>
</g>
<g >
<title>do_syscall_64 (86 samples, 0.03%)</title><rect x="1148.3" y="261" width="0.3" height="15.0" fill="rgb(219,82,32)" rx="2" ry="2" />
<text x="1151.28" y="271.5" ></text>
</g>
<g >
<title>read (293 samples, 0.10%)</title><rect x="800.2" y="117" width="1.2" height="15.0" fill="rgb(245,155,15)" rx="2" ry="2" />
<text x="803.15" y="127.5" ></text>
</g>
<g >
<title>ext4_getattr (49 samples, 0.02%)</title><rect x="707.3" y="1957" width="0.2" height="15.0" fill="rgb(241,166,10)" rx="2" ry="2" />
<text x="710.32" y="1967.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (301 samples, 0.11%)</title><rect x="803.5" y="117" width="1.2" height="15.0" fill="rgb(224,155,20)" rx="2" ry="2" />
<text x="806.50" y="127.5" ></text>
</g>
<g >
<title>ngx_http_time (42 samples, 0.01%)</title><rect x="268.2" y="2021" width="0.2" height="15.0" fill="rgb(229,70,50)" rx="2" ry="2" />
<text x="271.23" y="2031.5" ></text>
</g>
<g >
<title>smc_rx_update_consumer (50 samples, 0.02%)</title><rect x="242.7" y="1845" width="0.2" height="15.0" fill="rgb(209,125,13)" rx="2" ry="2" />
<text x="245.70" y="1855.5" ></text>
</g>
<g >
<title>avc_lookup (325 samples, 0.11%)</title><rect x="696.1" y="1861" width="1.4" height="15.0" fill="rgb(246,44,46)" rx="2" ry="2" />
<text x="699.15" y="1871.5" ></text>
</g>
<g >
<title>swapper (15,777 samples, 5.53%)</title><rect x="720.6" y="2069" width="65.3" height="15.0" fill="rgb(213,131,30)" rx="2" ry="2" />
<text x="723.60" y="2079.5" >swapper</text>
</g>
<g >
<title>smc_tx_sendmsg (261 samples, 0.09%)</title><rect x="806.5" y="37" width="1.1" height="15.0" fill="rgb(209,71,33)" rx="2" ry="2" />
<text x="809.51" y="47.5" ></text>
</g>
<g >
<title>check_heap_object (44,433 samples, 15.57%)</title><rect x="281.4" y="1813" width="183.8" height="15.0" fill="rgb(240,183,9)" rx="2" ry="2" />
<text x="284.42" y="1823.5" >check_heap_object</text>
</g>
<g >
<title>smc_poll (81 samples, 0.03%)</title><rect x="246.5" y="1861" width="0.3" height="15.0" fill="rgb(243,205,39)" rx="2" ry="2" />
<text x="249.51" y="1871.5" ></text>
</g>
<g >
<title>select_estimate_accuracy (114 samples, 0.04%)</title><rect x="1185.9" y="1957" width="0.5" height="15.0" fill="rgb(245,17,48)" rx="2" ry="2" />
<text x="1188.93" y="1967.5" ></text>
</g>
<g >
<title>smc_sendmsg (475 samples, 0.17%)</title><rect x="1144.7" y="181" width="2.0" height="15.0" fill="rgb(208,79,52)" rx="2" ry="2" />
<text x="1147.74" y="191.5" ></text>
</g>
<g >
<title>check_heap_object (1,197 samples, 0.42%)</title><rect x="823.6" y="85" width="5.0" height="15.0" fill="rgb(207,200,33)" rx="2" ry="2" />
<text x="826.63" y="95.5" ></text>
</g>
<g >
<title>irq_exit_rcu (120 samples, 0.04%)</title><rect x="779.2" y="1909" width="0.5" height="15.0" fill="rgb(247,173,19)" rx="2" ry="2" />
<text x="782.17" y="1919.5" ></text>
</g>
<g >
<title>rep_movs_alternative (65 samples, 0.02%)</title><rect x="254.2" y="1861" width="0.3" height="15.0" fill="rgb(212,2,13)" rx="2" ry="2" />
<text x="257.21" y="1871.5" ></text>
</g>
<g >
<title>ktime_get_ts64 (46 samples, 0.02%)</title><rect x="1186.1" y="1941" width="0.2" height="15.0" fill="rgb(237,61,53)" rx="2" ry="2" />
<text x="1189.14" y="1951.5" ></text>
</g>
<g >
<title>exec_binprm (29 samples, 0.01%)</title><rect x="717.3" y="1957" width="0.1" height="15.0" fill="rgb(228,65,30)" rx="2" ry="2" />
<text x="720.30" y="1967.5" ></text>
</g>
<g >
<title>__fsnotify_parent (40 samples, 0.01%)</title><rect x="690.6" y="1893" width="0.1" height="15.0" fill="rgb(229,173,21)" rx="2" ry="2" />
<text x="693.56" y="1903.5" ></text>
</g>
<g >
<title>__schedule (93 samples, 0.03%)</title><rect x="245.0" y="1861" width="0.3" height="15.0" fill="rgb(214,73,53)" rx="2" ry="2" />
<text x="247.96" y="1871.5" ></text>
</g>
<g >
<title>avc_lookup (50 samples, 0.02%)</title><rect x="279.7" y="1797" width="0.2" height="15.0" fill="rgb(241,84,39)" rx="2" ry="2" />
<text x="282.66" y="1807.5" ></text>
</g>
<g >
<title>sock_read_iter (1,201 samples, 0.42%)</title><rect x="829.1" y="165" width="5.0" height="15.0" fill="rgb(222,197,43)" rx="2" ry="2" />
<text x="832.08" y="175.5" ></text>
</g>
<g >
<title>security_file_permission (35 samples, 0.01%)</title><rect x="811.4" y="101" width="0.1" height="15.0" fill="rgb(218,63,32)" rx="2" ry="2" />
<text x="814.40" y="111.5" ></text>
</g>
<g >
<title>all (285,300 samples, 100%)</title><rect x="10.0" y="2085" width="1180.0" height="15.0" fill="rgb(251,117,13)" rx="2" ry="2" />
<text x="13.00" y="2095.5" ></text>
</g>
<g >
<title>_raw_read_lock (119 samples, 0.04%)</title><rect x="837.6" y="37" width="0.5" height="15.0" fill="rgb(208,9,22)" rx="2" ry="2" />
<text x="840.62" y="47.5" ></text>
</g>
<g >
<title>__vdso_gettimeofday (106 samples, 0.04%)</title><rect x="787.6" y="37" width="0.5" height="15.0" fill="rgb(252,109,6)" rx="2" ry="2" />
<text x="790.64" y="47.5" ></text>
</g>
<g >
<title>aeMain (26 samples, 0.01%)</title><rect x="788.4" y="37" width="0.1" height="15.0" fill="rgb(211,143,17)" rx="2" ry="2" />
<text x="791.37" y="47.5" ></text>
</g>
<g >
<title>ksys_write (28 samples, 0.01%)</title><rect x="715.5" y="1893" width="0.2" height="15.0" fill="rgb(227,212,11)" rx="2" ry="2" />
<text x="718.53" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1589" width="364.1" height="15.0" fill="rgb(246,78,1)" rx="2" ry="2" />
<text x="790.11" y="1599.5" >[unknown]</text>
</g>
<g >
<title>selinux_file_permission (78 samples, 0.03%)</title><rect x="805.7" y="53" width="0.3" height="15.0" fill="rgb(228,37,9)" rx="2" ry="2" />
<text x="808.70" y="63.5" ></text>
</g>
<g >
<title>smc_sendmsg (130 samples, 0.05%)</title><rect x="809.7" y="69" width="0.5" height="15.0" fill="rgb(241,186,35)" rx="2" ry="2" />
<text x="812.67" y="79.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (39,373 samples, 13.80%)</title><rect x="482.2" y="1877" width="162.9" height="15.0" fill="rgb(239,101,37)" rx="2" ry="2" />
<text x="485.23" y="1887.5" >smc_tx_sendmsg</text>
</g>
<g >
<title>mutex_lock (29 samples, 0.01%)</title><rect x="1155.9" y="1957" width="0.1" height="15.0" fill="rgb(228,147,46)" rx="2" ry="2" />
<text x="1158.87" y="1967.5" ></text>
</g>
<g >
<title>sock_recvmsg (207 samples, 0.07%)</title><rect x="810.5" y="85" width="0.9" height="15.0" fill="rgb(234,119,1)" rx="2" ry="2" />
<text x="813.54" y="95.5" ></text>
</g>
<g >
<title>smc_sndbuf_sync_sg_for_device (40 samples, 0.01%)</title><rect x="469.6" y="1829" width="0.2" height="15.0" fill="rgb(236,130,25)" rx="2" ry="2" />
<text x="472.59" y="1839.5" ></text>
</g>
<g >
<title>write (473 samples, 0.17%)</title><rect x="1148.7" y="309" width="1.9" height="15.0" fill="rgb(229,69,30)" rx="2" ry="2" />
<text x="1151.67" y="319.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1109" width="364.1" height="15.0" fill="rgb(247,30,46)" rx="2" ry="2" />
<text x="790.11" y="1119.5" >[unknown]</text>
</g>
<g >
<title>pv_wait_node (36,331 samples, 12.73%)</title><rect x="491.9" y="1781" width="150.2" height="15.0" fill="rgb(228,216,32)" rx="2" ry="2" />
<text x="494.89" y="1791.5" >pv_wait_node</text>
</g>
<g >
<title>__inode_security_revalidate (90 samples, 0.03%)</title><rect x="695.6" y="1877" width="0.4" height="15.0" fill="rgb(217,215,7)" rx="2" ry="2" />
<text x="698.61" y="1887.5" ></text>
</g>
<g >
<title>ngx_http_core_find_config_phase (43 samples, 0.02%)</title><rect x="671.7" y="2037" width="0.1" height="15.0" fill="rgb(251,111,40)" rx="2" ry="2" />
<text x="674.66" y="2047.5" ></text>
</g>
<g >
<title>smc_tx_sendmsg (473 samples, 0.17%)</title><rect x="1148.7" y="197" width="1.9" height="15.0" fill="rgb(225,60,54)" rx="2" ry="2" />
<text x="1151.67" y="207.5" ></text>
</g>
<g >
<title>__x64_sys_setsockopt (1,390 samples, 0.49%)</title><rect x="251.4" y="1957" width="5.7" height="15.0" fill="rgb(205,37,21)" rx="2" ry="2" />
<text x="254.37" y="1967.5" ></text>
</g>
<g >
<title>_raw_spin_lock (36,890 samples, 12.93%)</title><rect x="992.2" y="85" width="152.5" height="15.0" fill="rgb(231,140,54)" rx="2" ry="2" />
<text x="995.15" y="95.5" >_raw_spin_lock</text>
</g>
<g >
<title>irq_exit_rcu (29 samples, 0.01%)</title><rect x="240.0" y="1701" width="0.1" height="15.0" fill="rgb(230,71,30)" rx="2" ry="2" />
<text x="242.96" y="1711.5" ></text>
</g>
<g >
<title>smc_rx_recvmsg (888 samples, 0.31%)</title><rect x="838.9" y="133" width="3.7" height="15.0" fill="rgb(235,214,3)" rx="2" ry="2" />
<text x="841.89" y="143.5" ></text>
</g>
<g >
<title>__get_file_rcu (114 samples, 0.04%)</title><rect x="1168.6" y="1957" width="0.5" height="15.0" fill="rgb(240,216,2)" rx="2" ry="2" />
<text x="1171.61" y="1967.5" ></text>
</g>
<g >
<title>sock_read_iter (880 samples, 0.31%)</title><rect x="812.9" y="117" width="3.7" height="15.0" fill="rgb(234,54,5)" rx="2" ry="2" />
<text x="815.91" y="127.5" ></text>
</g>
<g >
<title>do_syscall_64 (1,169 samples, 0.41%)</title><rect x="834.1" y="213" width="4.8" height="15.0" fill="rgb(250,50,54)" rx="2" ry="2" />
<text x="837.05" y="223.5" ></text>
</g>
<g >
<title>sock_recvmsg (129 samples, 0.05%)</title><rect x="805.1" y="53" width="0.5" height="15.0" fill="rgb(236,161,49)" rx="2" ry="2" />
<text x="808.08" y="63.5" ></text>
</g>
<g >
<title>[unknown] (165,513 samples, 58.01%)</title><rect x="19.1" y="2053" width="684.6" height="15.0" fill="rgb(240,109,34)" rx="2" ry="2" />
<text x="22.11" y="2063.5" >[unknown]</text>
</g>
<g >
<title>ngx_open_cached_file (339 samples, 0.12%)</title><rect x="679.0" y="2037" width="1.4" height="15.0" fill="rgb(215,28,3)" rx="2" ry="2" />
<text x="682.01" y="2047.5" ></text>
</g>
<g >
<title>finish_task_switch.isra.0 (98 samples, 0.03%)</title><rect x="647.8" y="1893" width="0.4" height="15.0" fill="rgb(214,71,30)" rx="2" ry="2" />
<text x="650.75" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (3,592 samples, 1.26%)</title><rect x="787.1" y="133" width="14.9" height="15.0" fill="rgb(233,27,14)" rx="2" ry="2" />
<text x="790.11" y="143.5" ></text>
</g>
<g >
<title>__fdget_pos (119 samples, 0.04%)</title><rect x="802.0" y="69" width="0.5" height="15.0" fill="rgb(238,170,45)" rx="2" ry="2" />
<text x="804.97" y="79.5" ></text>
</g>
<g >
<title>_raw_spin_lock (887 samples, 0.31%)</title><rect x="838.9" y="69" width="3.7" height="15.0" fill="rgb(207,86,11)" rx="2" ry="2" />
<text x="841.89" y="79.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (43 samples, 0.02%)</title><rect x="40.6" y="1973" width="0.1" height="15.0" fill="rgb(215,16,32)" rx="2" ry="2" />
<text x="43.56" y="1983.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv_action (78 samples, 0.03%)</title><rect x="1148.3" y="53" width="0.3" height="15.0" fill="rgb(205,160,11)" rx="2" ry="2" />
<text x="1151.32" y="63.5" ></text>
</g>
<g >
<title>check_heap_object (131 samples, 0.05%)</title><rect x="823.1" y="69" width="0.5" height="15.0" fill="rgb(249,169,2)" rx="2" ry="2" />
<text x="826.08" y="79.5" ></text>
</g>
<g >
<title>release_sock (37 samples, 0.01%)</title><rect x="280.2" y="1845" width="0.2" height="15.0" fill="rgb(229,54,17)" rx="2" ry="2" />
<text x="283.23" y="1855.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="853" width="364.1" height="15.0" fill="rgb(242,45,9)" rx="2" ry="2" />
<text x="790.11" y="863.5" >[unknown]</text>
</g>
<g >
<title>vfs_writev (40,337 samples, 14.14%)</title><rect x="479.9" y="1957" width="166.9" height="15.0" fill="rgb(237,13,25)" rx="2" ry="2" />
<text x="482.92" y="1967.5" >vfs_writev</text>
</g>
<g >
<title>ngx_pfree (47 samples, 0.02%)</title><rect x="48.8" y="1957" width="0.2" height="15.0" fill="rgb(237,89,28)" rx="2" ry="2" />
<text x="51.80" y="1967.5" ></text>
</g>
<g >
<title>ksys_mmap_pgoff (45 samples, 0.02%)</title><rect x="719.1" y="2005" width="0.2" height="15.0" fill="rgb(222,110,17)" rx="2" ry="2" />
<text x="722.15" y="2015.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1333" width="364.1" height="15.0" fill="rgb(223,125,41)" rx="2" ry="2" />
<text x="790.11" y="1343.5" >[unknown]</text>
</g>
<g >
<title>__cancel_work (87 samples, 0.03%)</title><rect x="53.9" y="1845" width="0.3" height="15.0" fill="rgb(218,12,11)" rx="2" ry="2" />
<text x="56.87" y="1855.5" ></text>
</g>
<g >
<title>__release_sock (25 samples, 0.01%)</title><rect x="1165.4" y="1909" width="0.1" height="15.0" fill="rgb(226,70,21)" rx="2" ry="2" />
<text x="1168.40" y="1919.5" ></text>
</g>
<g >
<title>lockref_put_return (61 samples, 0.02%)</title><rect x="665.9" y="1909" width="0.2" height="15.0" fill="rgb(245,42,37)" rx="2" ry="2" />
<text x="668.86" y="1919.5" ></text>
</g>
<g >
<title>http_parser_execute (1,116 samples, 0.39%)</title><rect x="1157.0" y="2037" width="4.7" height="15.0" fill="rgb(217,110,19)" rx="2" ry="2" />
<text x="1160.04" y="2047.5" ></text>
</g>
<g >
<title>pv_wait_node (36,141 samples, 12.67%)</title><rect x="842.6" y="53" width="149.4" height="15.0" fill="rgb(254,79,52)" rx="2" ry="2" />
<text x="845.57" y="63.5" >pv_wait_node</text>
</g>
<g >
<title>schedule (97 samples, 0.03%)</title><rect x="244.9" y="1877" width="0.4" height="15.0" fill="rgb(230,172,46)" rx="2" ry="2" />
<text x="247.95" y="1887.5" ></text>
</g>
<g >
<title>sock_write (59 samples, 0.02%)</title><rect x="792.6" y="37" width="0.2" height="15.0" fill="rgb(215,32,19)" rx="2" ry="2" />
<text x="795.56" y="47.5" ></text>
</g>
<g >
<title>__do_sys_newfstatat (4,139 samples, 1.45%)</title><rect x="650.2" y="1989" width="17.1" height="15.0" fill="rgb(206,0,45)" rx="2" ry="2" />
<text x="653.22" y="1999.5" ></text>
</g>
<g >
<title>[unknown] (12,519 samples, 4.39%)</title><rect x="787.1" y="261" width="51.8" height="15.0" fill="rgb(226,210,50)" rx="2" ry="2" />
<text x="790.11" y="271.5" >[unkn..</text>
</g>
<g >
<title>do_syscall_64 (949 samples, 0.33%)</title><rect x="706.0" y="2021" width="3.9" height="15.0" fill="rgb(242,74,30)" rx="2" ry="2" />
<text x="708.97" y="2031.5" ></text>
</g>
<g >
<title>[libc.so.6] (511 samples, 0.18%)</title><rect x="13.8" y="2053" width="2.1" height="15.0" fill="rgb(210,139,19)" rx="2" ry="2" />
<text x="16.81" y="2063.5" ></text>
</g>
<g >
<title>ngx_http_finalize_request (56 samples, 0.02%)</title><rect x="672.4" y="2037" width="0.2" height="15.0" fill="rgb(223,103,15)" rx="2" ry="2" />
<text x="675.36" y="2047.5" ></text>
</g>
<g >
<title>smc_setsockopt (787 samples, 0.28%)</title><rect x="253.0" y="1909" width="3.3" height="15.0" fill="rgb(216,21,14)" rx="2" ry="2" />
<text x="256.03" y="1919.5" ></text>
</g>
<g >
<title>check_heap_object (1,201 samples, 0.42%)</title><rect x="829.1" y="85" width="5.0" height="15.0" fill="rgb(248,2,19)" rx="2" ry="2" />
<text x="832.08" y="95.5" ></text>
</g>
<g >
<title>do_syscall_64 (75 samples, 0.03%)</title><rect x="716.7" y="2021" width="0.3" height="15.0" fill="rgb(240,77,15)" rx="2" ry="2" />
<text x="719.73" y="2031.5" ></text>
</g>
<g >
<title>path_init (303 samples, 0.11%)</title><rect x="663.9" y="1909" width="1.3" height="15.0" fill="rgb(221,24,31)" rx="2" ry="2" />
<text x="666.90" y="1919.5" ></text>
</g>
<g >
<title>[unknown] (88,003 samples, 30.85%)</title><rect x="787.1" y="485" width="364.0" height="15.0" fill="rgb(228,182,39)" rx="2" ry="2" />
<text x="790.11" y="495.5" >[unknown]</text>
</g>
<g >
<title>setsockopt (6,994 samples, 2.45%)</title><rect x="50.0" y="1957" width="28.9" height="15.0" fill="rgb(206,108,6)" rx="2" ry="2" />
<text x="52.96" y="1967.5" >se..</text>
</g>
<g >
<title>vfs_write (473 samples, 0.17%)</title><rect x="1148.7" y="245" width="1.9" height="15.0" fill="rgb(245,50,51)" rx="2" ry="2" />
<text x="1151.67" y="255.5" ></text>
</g>
<g >
<title>schedule (1,088 samples, 0.38%)</title><rect x="1181.4" y="1941" width="4.5" height="15.0" fill="rgb(248,217,39)" rx="2" ry="2" />
<text x="1184.43" y="1951.5" ></text>
</g>
<g >
<title>inode_permission.part.0 (71 samples, 0.02%)</title><rect x="693.4" y="1893" width="0.3" height="15.0" fill="rgb(238,181,11)" rx="2" ry="2" />
<text x="696.43" y="1903.5" ></text>
</g>
<g >
<title>do_user_addr_fault (27 samples, 0.01%)</title><rect x="718.3" y="1957" width="0.1" height="15.0" fill="rgb(231,145,18)" rx="2" ry="2" />
<text x="721.26" y="1967.5" ></text>
</g>
<g >
<title>check_heap_object (36,143 samples, 12.67%)</title><rect x="842.6" y="117" width="149.5" height="15.0" fill="rgb(218,152,24)" rx="2" ry="2" />
<text x="845.56" y="127.5" >check_heap_object</text>
</g>
<g >
<title>sock_recvmsg (1,201 samples, 0.42%)</title><rect x="829.1" y="149" width="5.0" height="15.0" fill="rgb(236,155,43)" rx="2" ry="2" />
<text x="832.08" y="159.5" ></text>
</g>
<g >
<title>__x64_sys_epoll_wait (338 samples, 0.12%)</title><rect x="245.6" y="1957" width="1.4" height="15.0" fill="rgb(217,106,11)" rx="2" ry="2" />
<text x="248.62" y="1967.5" ></text>
</g>
<g >
<title>_raw_spin_unlock_irqrestore (3,724 samples, 1.31%)</title><rect x="61.1" y="1685" width="15.4" height="15.0" fill="rgb(205,143,11)" rx="2" ry="2" />
<text x="64.08" y="1695.5" ></text>
</g>
<g >
<title>kvm_wait.part.0 (34,358 samples, 12.04%)</title><rect x="98.0" y="1749" width="142.1" height="15.0" fill="rgb(252,215,41)" rx="2" ry="2" />
<text x="100.98" y="1759.5" >kvm_wait.part.0</text>
</g>
<g >
<title>do_open (1,221 samples, 0.43%)</title><rect x="689.0" y="1925" width="5.1" height="15.0" fill="rgb(250,122,10)" rx="2" ry="2" />
<text x="692.04" y="1935.5" ></text>
</g>
<g >
<title>rep_movs_alternative (85 samples, 0.03%)</title><rect x="650.5" y="1941" width="0.3" height="15.0" fill="rgb(210,47,50)" rx="2" ry="2" />
<text x="653.48" y="1951.5" ></text>
</g>
<g >
<title>ngx_http_parse_header_line (167 samples, 0.06%)</title><rect x="711.9" y="2053" width="0.7" height="15.0" fill="rgb(213,184,14)" rx="2" ry="2" />
<text x="714.94" y="2063.5" ></text>
</g>
<g >
<title>do_syscall_64 (369 samples, 0.13%)</title><rect x="802.0" y="101" width="1.5" height="15.0" fill="rgb(234,136,37)" rx="2" ry="2" />
<text x="804.97" y="111.5" ></text>
</g>
<g >
<title>smc_cdc_msg_recv_action (129 samples, 0.05%)</title><rect x="1146.2" y="37" width="0.5" height="15.0" fill="rgb(229,141,42)" rx="2" ry="2" />
<text x="1149.17" y="47.5" ></text>
</g>
<g >
<title>do_execveat_common (29 samples, 0.01%)</title><rect x="717.3" y="1989" width="0.1" height="15.0" fill="rgb(240,218,4)" rx="2" ry="2" />
<text x="720.30" y="1999.5" ></text>
</g>
<g >
<title>ngx_pool_cleanup_add (31 samples, 0.01%)</title><rect x="680.7" y="2037" width="0.1" height="15.0" fill="rgb(231,76,0)" rx="2" ry="2" />
<text x="683.69" y="2047.5" ></text>
</g>
<g >
<title>kvm_kick_cpu (341 samples, 0.12%)</title><rect x="1144.7" y="37" width="1.5" height="15.0" fill="rgb(246,59,54)" rx="2" ry="2" />
<text x="1147.74" y="47.5" ></text>
</g>
<g >
<title>[unknown] (2,440 samples, 0.86%)</title><rect x="787.1" y="85" width="10.1" height="15.0" fill="rgb(215,153,26)" rx="2" ry="2" />
<text x="790.12" y="95.5" ></text>
</g>
<g >
<title>ngx_http_core_content_phase (103 samples, 0.04%)</title><rect x="671.2" y="2037" width="0.5" height="15.0" fill="rgb(251,185,6)" rx="2" ry="2" />
<text x="674.23" y="2047.5" ></text>
</g>
<g >
<title>do_syscall_64 (253 samples, 0.09%)</title><rect x="811.4" y="149" width="1.0" height="15.0" fill="rgb(231,225,24)" rx="2" ry="2" />
<text x="814.40" y="159.5" ></text>
</g>
<g >
<title>find_vmap_area (131 samples, 0.05%)</title><rect x="823.1" y="53" width="0.5" height="15.0" fill="rgb(243,38,38)" rx="2" ry="2" />
<text x="826.08" y="63.5" ></text>
</g>
<g >
<title>write (32 samples, 0.01%)</title><rect x="715.5" y="1941" width="0.2" height="15.0" fill="rgb(230,114,22)" rx="2" ry="2" />
<text x="718.52" y="1951.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1157" width="364.1" height="15.0" fill="rgb(242,69,34)" rx="2" ry="2" />
<text x="790.11" y="1167.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (2,429 samples, 0.85%)</title><rect x="787.1" y="69" width="10.1" height="15.0" fill="rgb(247,102,51)" rx="2" ry="2" />
<text x="790.12" y="79.5" ></text>
</g>
<g >
<title>read (881 samples, 0.31%)</title><rect x="812.9" y="197" width="3.7" height="15.0" fill="rgb(211,218,14)" rx="2" ry="2" />
<text x="815.91" y="207.5" ></text>
</g>
<g >
<title>do_syscall_64 (301 samples, 0.11%)</title><rect x="803.5" y="101" width="1.2" height="15.0" fill="rgb(231,157,48)" rx="2" ry="2" />
<text x="806.50" y="111.5" ></text>
</g>
<g >
<title>check_heap_object (610 samples, 0.21%)</title><rect x="816.6" y="53" width="2.5" height="15.0" fill="rgb(233,198,16)" rx="2" ry="2" />
<text x="819.56" y="63.5" ></text>
</g>
<g >
<title>[nginx] (259 samples, 0.09%)</title><rect x="46.6" y="1973" width="1.0" height="15.0" fill="rgb(250,147,19)" rx="2" ry="2" />
<text x="49.57" y="1983.5" ></text>
</g>
<g >
<title>do_sock_setsockopt (6,018 samples, 2.11%)</title><rect x="52.0" y="1877" width="24.9" height="15.0" fill="rgb(242,19,7)" rx="2" ry="2" />
<text x="54.98" y="1887.5" >d..</text>
</g>
<g >
<title>malloc (206 samples, 0.07%)</title><rect x="709.9" y="2053" width="0.9" height="15.0" fill="rgb(213,104,26)" rx="2" ry="2" />
<text x="712.91" y="2063.5" ></text>
</g>
<g >
<title>path_get (33 samples, 0.01%)</title><rect x="692.4" y="1893" width="0.2" height="15.0" fill="rgb(212,227,46)" rx="2" ry="2" />
<text x="695.42" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="2005" width="364.1" height="15.0" fill="rgb(250,184,16)" rx="2" ry="2" />
<text x="790.11" y="2015.5" >[unknown]</text>
</g>
<g >
<title>_copy_from_user (78 samples, 0.03%)</title><rect x="54.7" y="1829" width="0.4" height="15.0" fill="rgb(244,27,6)" rx="2" ry="2" />
<text x="57.73" y="1839.5" ></text>
</g>
<g >
<title>sock_common_setsockopt (57 samples, 0.02%)</title><rect x="76.5" y="1845" width="0.3" height="15.0" fill="rgb(209,126,33)" rx="2" ry="2" />
<text x="79.54" y="1855.5" ></text>
</g>
<g >
<title>security_socket_setsockopt (274 samples, 0.10%)</title><rect x="52.2" y="1861" width="1.1" height="15.0" fill="rgb(237,138,51)" rx="2" ry="2" />
<text x="55.19" y="1871.5" ></text>
</g>
<g >
<title>exit_to_user_mode_loop (35 samples, 0.01%)</title><rect x="809.5" y="85" width="0.1" height="15.0" fill="rgb(245,103,28)" rx="2" ry="2" />
<text x="812.50" y="95.5" ></text>
</g>
<g >
<title>do_syscall_64 (47 samples, 0.02%)</title><rect x="717.7" y="2021" width="0.2" height="15.0" fill="rgb(235,95,48)" rx="2" ry="2" />
<text x="720.74" y="2031.5" ></text>
</g>
<g >
<title>generic_fillattr (50 samples, 0.02%)</title><rect x="654.8" y="1909" width="0.2" height="15.0" fill="rgb(249,30,35)" rx="2" ry="2" />
<text x="657.79" y="1919.5" ></text>
</g>
<g >
<title>write (462 samples, 0.16%)</title><rect x="794.5" y="37" width="1.9" height="15.0" fill="rgb(235,214,14)" rx="2" ry="2" />
<text x="797.46" y="47.5" ></text>
</g>
<g >
<title>find_vmap_area (36,143 samples, 12.67%)</title><rect x="842.6" y="101" width="149.5" height="15.0" fill="rgb(251,44,39)" rx="2" ry="2" />
<text x="845.56" y="111.5" >find_vmap_area</text>
</g>
<g >
<title>do_syscall_64 (135 samples, 0.05%)</title><rect x="799.6" y="69" width="0.5" height="15.0" fill="rgb(221,103,6)" rx="2" ry="2" />
<text x="802.59" y="79.5" ></text>
</g>
<g >
<title>ksys_read (207 samples, 0.07%)</title><rect x="804.8" y="101" width="0.8" height="15.0" fill="rgb(233,185,31)" rx="2" ry="2" />
<text x="807.76" y="111.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (369 samples, 0.13%)</title><rect x="802.0" y="117" width="1.5" height="15.0" fill="rgb(208,26,2)" rx="2" ry="2" />
<text x="804.97" y="127.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (949 samples, 0.33%)</title><rect x="706.0" y="2037" width="3.9" height="15.0" fill="rgb(209,74,27)" rx="2" ry="2" />
<text x="708.97" y="2047.5" ></text>
</g>
<g >
<title>selinux_inode_getattr (90 samples, 0.03%)</title><rect x="707.7" y="1957" width="0.4" height="15.0" fill="rgb(237,46,43)" rx="2" ry="2" />
<text x="710.72" y="1967.5" ></text>
</g>
<g >
<title>__virt_addr_valid (99 samples, 0.03%)</title><rect x="701.1" y="1893" width="0.4" height="15.0" fill="rgb(239,156,39)" rx="2" ry="2" />
<text x="704.08" y="1903.5" ></text>
</g>
<g >
<title>[unknown] (88,021 samples, 30.85%)</title><rect x="787.1" y="565" width="364.1" height="15.0" fill="rgb(206,116,17)" rx="2" ry="2" />
<text x="790.11" y="575.5" >[unknown]</text>
</g>
<g >
<title>smc_sendmsg (87 samples, 0.03%)</title><rect x="804.4" y="37" width="0.3" height="15.0" fill="rgb(245,216,26)" rx="2" ry="2" />
<text x="807.38" y="47.5" ></text>
</g>
<g >
<title>syscall_enter_from_user_mode (231 samples, 0.08%)</title><rect x="798.3" y="37" width="1.0" height="15.0" fill="rgb(231,217,27)" rx="2" ry="2" />
<text x="801.30" y="47.5" ></text>
</g>
<g >
<title>entry_SYSCALL_64 (131 samples, 0.05%)</title><rect x="823.1" y="213" width="0.5" height="15.0" fill="rgb(235,50,30)" rx="2" ry="2" />
<text x="826.08" y="223.5" ></text>
</g>
<g >
<title>[unknown] (88,032 samples, 30.86%)</title><rect x="787.1" y="677" width="364.1" height="15.0" fill="rgb(221,20,26)" rx="2" ry="2" />
<text x="790.11" y="687.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="1733" width="364.1" height="15.0" fill="rgb(214,182,53)" rx="2" ry="2" />
<text x="790.11" y="1743.5" >[unknown]</text>
</g>
<g >
<title>ext4_file_splice_read (122 samples, 0.04%)</title><rect x="469.8" y="1909" width="0.5" height="15.0" fill="rgb(252,174,19)" rx="2" ry="2" />
<text x="472.76" y="1919.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (134 samples, 0.05%)</title><rect x="822.3" y="85" width="0.6" height="15.0" fill="rgb(223,34,11)" rx="2" ry="2" />
<text x="825.30" y="95.5" ></text>
</g>
<g >
<title>ep_item_poll.isra.0 (579 samples, 0.20%)</title><rect x="1177.1" y="1941" width="2.4" height="15.0" fill="rgb(252,202,42)" rx="2" ry="2" />
<text x="1180.08" y="1951.5" ></text>
</g>
<g >
<title>smcd_tx_ism_write (134 samples, 0.05%)</title><rect x="1146.2" y="101" width="0.5" height="15.0" fill="rgb(236,98,15)" rx="2" ry="2" />
<text x="1149.15" y="111.5" ></text>
</g>
<g >
<title>rep_movs_alternative (188 samples, 0.07%)</title><rect x="644.1" y="1845" width="0.8" height="15.0" fill="rgb(244,122,5)" rx="2" ry="2" />
<text x="647.12" y="1855.5" ></text>
</g>
<g >
<title>exit_to_user_mode_prepare (35 samples, 0.01%)</title><rect x="809.5" y="101" width="0.1" height="15.0" fill="rgb(254,130,11)" rx="2" ry="2" />
<text x="812.50" y="111.5" ></text>
</g>
<g >
<title>smc_recvmsg (603 samples, 0.21%)</title><rect x="819.8" y="101" width="2.5" height="15.0" fill="rgb(210,149,3)" rx="2" ry="2" />
<text x="822.80" y="111.5" ></text>
</g>
<g >
<title>_copy_from_user (41 samples, 0.01%)</title><rect x="1153.7" y="1973" width="0.2" height="15.0" fill="rgb(248,176,8)" rx="2" ry="2" />
<text x="1156.71" y="1983.5" ></text>
</g>
<g >
<title>lockref_put_return (71 samples, 0.02%)</title><rect x="699.3" y="1893" width="0.2" height="15.0" fill="rgb(207,12,12)" rx="2" ry="2" />
<text x="702.26" y="1903.5" ></text>
</g>
<g >
<title>kernel_connect (58 samples, 0.02%)</title><rect x="1165.3" y="1957" width="0.2" height="15.0" fill="rgb(240,217,32)" rx="2" ry="2" />
<text x="1168.26" y="1967.5" ></text>
</g>
<g >
<title>smc_lo_move_data (38 samples, 0.01%)</title><rect x="57.2" y="1781" width="0.2" height="15.0" fill="rgb(252,55,23)" rx="2" ry="2" />
<text x="60.25" y="1791.5" ></text>
</g>
<g >
<title>[unknown] (88,010 samples, 30.85%)</title><rect x="787.1" y="517" width="364.0" height="15.0" fill="rgb(251,187,46)" rx="2" ry="2" />
<text x="790.11" y="527.5" >[unknown]</text>
</g>
<g >
<title>[unknown] (88,033 samples, 30.86%)</title><rect x="787.1" y="789" width="364.1" height="15.0" fill="rgb(207,0,34)" rx="2" ry="2" />
<text x="790.11" y="799.5" >[unknown]</text>
</g>
<g >
<title>write (143 samples, 0.05%)</title><rect x="801.4" y="117" width="0.6" height="15.0" fill="rgb(238,24,47)" rx="2" ry="2" />
<text x="804.38" y="127.5" ></text>
</g>
<g >
<title>schedule (123 samples, 0.04%)</title><rect x="476.6" y="1925" width="0.5" height="15.0" fill="rgb(213,169,40)" rx="2" ry="2" />
<text x="479.57" y="1935.5" ></text>
</g>
<g >
<title>avc_has_perm (115 samples, 0.04%)</title><rect x="52.8" y="1829" width="0.5" height="15.0" fill="rgb(249,127,41)" rx="2" ry="2" />
<text x="55.84" y="1839.5" ></text>
</g>
<g >
<title>do_dentry_open (795 samples, 0.28%)</title><rect x="689.6" y="1909" width="3.3" height="15.0" fill="rgb(213,125,42)" rx="2" ry="2" />
<text x="692.64" y="1919.5" ></text>
</g>
<g >
<title>ext4_buffered_write_iter (27 samples, 0.01%)</title><rect x="715.5" y="1861" width="0.1" height="15.0" fill="rgb(243,156,50)" rx="2" ry="2" />
<text x="718.53" y="1871.5" ></text>
</g>
<g >
<title>smcd_cdc_msg_send (4,627 samples, 1.62%)</title><rect x="57.4" y="1797" width="19.1" height="15.0" fill="rgb(222,108,41)" rx="2" ry="2" />
<text x="60.41" y="1807.5" ></text>
</g>
<g >
<title>__check_object_size.part.0 (38,856 samples, 13.62%)</title><rect x="483.4" y="1861" width="160.7" height="15.0" fill="rgb(233,186,50)" rx="2" ry="2" />
<text x="486.36" y="1871.5" >__check_object_size...</text>
</g>
<g >
<title>security_inode_getattr (190 samples, 0.07%)</title><rect x="666.1" y="1941" width="0.8" height="15.0" fill="rgb(231,17,11)" rx="2" ry="2" />
<text x="669.11" y="1951.5" ></text>
</g>
<g >
<title>sock_read_iter (452 samples, 0.16%)</title><rect x="807.6" y="85" width="1.9" height="15.0" fill="rgb(229,96,36)" rx="2" ry="2" />
<text x="810.63" y="95.5" ></text>
</g>
<g >
<title>smcd_cdc_rx_handler (4,144 samples, 1.45%)</title><rect x="59.4" y="1749" width="17.1" height="15.0" fill="rgb(235,205,48)" rx="2" ry="2" />
<text x="62.40" y="1759.5" ></text>
</g>
<g >
<title>handle_mm_fault (27 samples, 0.01%)</title><rect x="716.2" y="1973" width="0.1" height="15.0" fill="rgb(243,100,16)" rx="2" ry="2" />
<text x="719.20" y="1983.5" ></text>
</g>
<g >
<title>memchr@plt (31 samples, 0.01%)</title><rect x="1189.7" y="2053" width="0.1" height="15.0" fill="rgb(236,188,34)" rx="2" ry="2" />
<text x="1192.66" y="2063.5" ></text>
</g>
</g>
</svg>
[-- Attachment #6: perf_top --]
[-- Type: text/plain, Size: 1370 bytes --]
^[[H^[[2J
PerfTop: 0 irqs/sec kernel: 0.0% exact: 0.0% lost: 0/0 drop: 0/0 [4000Hz cpu-clock:pppH], (all, 48 CPUs)
-------------------------------------------------------------------------------
^[[H^[[2J
PerfTop: 124055 irqs/sec kernel:89.5% exact: 0.0% lost: 0/0 drop: 0/0 [4000Hz cpu-clock:pppH], (all, 48 CPUs)
-------------------------------------------------------------------------------
66.85% [kernel] [k] kvm_wait.part.0
2.53% [kernel] [k] __pv_queued_spin_lock_slowpath
1.75% [kernel] [k] _raw_spin_unlock_irqrestore
1.35% [kernel] [k] syscall_enter_from_user_mode
1.04% [kernel] [k] finish_task_switch.isra.0
1.00% [kernel] [k] find_vmap_area
0.77% [kernel] [k] pv_wait_node
0.71% [kernel] [k] avc_lookup
0.62% [kernel] [k] rep_movs_alternative
0.55% [kernel] [k] __fget_light
0.50% wrk [.] http_parser_execute
0.46% [kernel] [k] kvm_kick_cpu
0.33% [kernel] [k] memset_orig
0.33% libc.so.6 [.] setsockopt
0.31% [kernel] [k] _copy_from_iter
0.31% libc.so.6 [.] epoll_wait
0.30% [kernel] [k] _raw_spin_lock
0.29% libc.so.6 [.] fstatat64
0.28% [kernel] [k] __d_lookup_rcu
0.27% libc.so.6 [.] epoll_ctl
[-- Attachment #7: vm_cpuinfo --]
[-- Type: text/plain, Size: 35624 bytes --]
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 4
siblings : 1
core id : 0
cpu cores : 1
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 5
siblings : 1
core id : 0
cpu cores : 1
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 6
siblings : 1
core id : 0
cpu cores : 1
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 7
siblings : 1
core id : 0
cpu cores : 1
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 8
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 8
siblings : 1
core id : 0
cpu cores : 1
apicid : 8
initial apicid : 8
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 9
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 9
siblings : 1
core id : 0
cpu cores : 1
apicid : 9
initial apicid : 9
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 10
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 10
siblings : 1
core id : 0
cpu cores : 1
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 11
siblings : 1
core id : 0
cpu cores : 1
apicid : 11
initial apicid : 11
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 12
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 12
siblings : 1
core id : 0
cpu cores : 1
apicid : 12
initial apicid : 12
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 13
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 13
siblings : 1
core id : 0
cpu cores : 1
apicid : 13
initial apicid : 13
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 14
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 14
siblings : 1
core id : 0
cpu cores : 1
apicid : 14
initial apicid : 14
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 15
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 15
siblings : 1
core id : 0
cpu cores : 1
apicid : 15
initial apicid : 15
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 16
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 16
siblings : 1
core id : 0
cpu cores : 1
apicid : 16
initial apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 17
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 17
siblings : 1
core id : 0
cpu cores : 1
apicid : 17
initial apicid : 17
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 18
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 18
siblings : 1
core id : 0
cpu cores : 1
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 19
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 19
siblings : 1
core id : 0
cpu cores : 1
apicid : 19
initial apicid : 19
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 20
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 20
siblings : 1
core id : 0
cpu cores : 1
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 21
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 21
siblings : 1
core id : 0
cpu cores : 1
apicid : 21
initial apicid : 21
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 22
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 22
siblings : 1
core id : 0
cpu cores : 1
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 23
siblings : 1
core id : 0
cpu cores : 1
apicid : 23
initial apicid : 23
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 24
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 24
siblings : 1
core id : 0
cpu cores : 1
apicid : 24
initial apicid : 24
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 25
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 25
siblings : 1
core id : 0
cpu cores : 1
apicid : 25
initial apicid : 25
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 26
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 26
siblings : 1
core id : 0
cpu cores : 1
apicid : 26
initial apicid : 26
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 27
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 27
siblings : 1
core id : 0
cpu cores : 1
apicid : 27
initial apicid : 27
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 28
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 28
siblings : 1
core id : 0
cpu cores : 1
apicid : 28
initial apicid : 28
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 29
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 29
siblings : 1
core id : 0
cpu cores : 1
apicid : 29
initial apicid : 29
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 30
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 30
siblings : 1
core id : 0
cpu cores : 1
apicid : 30
initial apicid : 30
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 31
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 31
siblings : 1
core id : 0
cpu cores : 1
apicid : 31
initial apicid : 31
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 32
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 32
siblings : 1
core id : 0
cpu cores : 1
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 33
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 33
siblings : 1
core id : 0
cpu cores : 1
apicid : 33
initial apicid : 33
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 34
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 34
siblings : 1
core id : 0
cpu cores : 1
apicid : 34
initial apicid : 34
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 35
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 35
siblings : 1
core id : 0
cpu cores : 1
apicid : 35
initial apicid : 35
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 36
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 36
siblings : 1
core id : 0
cpu cores : 1
apicid : 36
initial apicid : 36
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 37
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 37
siblings : 1
core id : 0
cpu cores : 1
apicid : 37
initial apicid : 37
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 38
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 38
siblings : 1
core id : 0
cpu cores : 1
apicid : 38
initial apicid : 38
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 39
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 39
siblings : 1
core id : 0
cpu cores : 1
apicid : 39
initial apicid : 39
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 40
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 40
siblings : 1
core id : 0
cpu cores : 1
apicid : 40
initial apicid : 40
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 41
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 41
siblings : 1
core id : 0
cpu cores : 1
apicid : 41
initial apicid : 41
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 42
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 42
siblings : 1
core id : 0
cpu cores : 1
apicid : 42
initial apicid : 42
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 43
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 43
siblings : 1
core id : 0
cpu cores : 1
apicid : 43
initial apicid : 43
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 44
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 44
siblings : 1
core id : 0
cpu cores : 1
apicid : 44
initial apicid : 44
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 45
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 45
siblings : 1
core id : 0
cpu cores : 1
apicid : 45
initial apicid : 45
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 46
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 46
siblings : 1
core id : 0
cpu cores : 1
apicid : 46
initial apicid : 46
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 47
vendor_id : GenuineIntel
cpu family : 6
model : 13
model name : QEMU Virtual CPU version 1.5.3
stepping : 3
microcode : 0x1
cpu MHz : 2593.992
cache size : 4096 KB
physical id : 47
siblings : 1
core id : 0
cpu cores : 1
apicid : 47
initial apicid : 47
fpu : yes
fpu_exception : yes
cpuid level : 4
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pse36 clflush mmx fxsr sse sse2 syscall nx lm rep_good nopl cpuid tsc_known_freq pni cx16 hypervisor lahf_lm abm pti
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_unknown
bogomips : 5187.98
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-14 3:17 ` shaozhengchao
@ 2023-12-26 11:52 ` Wen Gu
2023-12-27 1:12 ` shaozhengchao
0 siblings, 1 reply; 14+ messages in thread
From: Wen Gu @ 2023-12-26 11:52 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
[-- Attachment #1: Type: text/plain, Size: 12358 bytes --]
On 2023/12/14 11:17, shaozhengchao wrote:
>
>
> On 2023/12/13 20:59, Wen Gu wrote:
>> On 2023/12/13 17:00, shaozhengchao wrote:
>>>
>>>
>>> On 2023/12/5 14:45, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>>>
>>>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>>>> Hi Wen Gu:
>>>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>>>>>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>>>>>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>>>>>>>> my analysis:
>>>>>>>>>>
>>>>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>>>>> packets on the lo device.
>>>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>>>
>>>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>>>>>>>> SMC-R-Deline packets.
>>>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>>>
>>>>>>>>>> 3. Rx
>>>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>>>>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>>>>>>>
>>>>>>>>>> 4. Tx
>>>>>>>>>> In smc_clc_send_proposal,
>>>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>>>
>>>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>>>>>>>> always equal to 0.
>>>>>>>>>>
>>>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>>>>> But I don't see where operations can be added, may I missed something?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Zhengchao Shao,
>>>>>>>>>
>>>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>>>
>>>>>>>>> - Install smc-tools[1].
>>>>>>>>>
>>>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>>>
>>>>>>>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>>>>>>>> unavailable, an UEID is required.
>>>>>>>>>
>>>>>>>> Hi Wen Gu:
>>>>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>>>>> be happy to learn from it.
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Zhengchao Shao
>>>>>>>>> - Then run the test.
>>>>>>>>>
>>>>>>>>> Hope this works for you :)
>>>>>>>>>
>>>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Wen Gu
>>>>>>>>>
>>>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>>>
>>>>>>>>>> Zhengchao Shao
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1]link:
>>>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> Hi Wen Gu:
>>>>>>> I have test as following, but the performance is really
>>>>>>> degraded. Now I have no idea.
>>>>>>> 1. add ueid
>>>>>>> run: smcd ueid add 16
>>>>>>> kernel message:
>>>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>>>> 2. start server
>>>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>>>> 3. start client
>>>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>
>>>>>>> The test results are as follows:
>>>>>>> TCP SMC-lo
>>>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>>>
>>>>>>> I didn't find a better direction when I initially positioned it. No
>>>>>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>>>>>> [root@localhost smc-tools]# smcd stats
>>>>>>> SMC-D Connections Summary
>>>>>>> Total connections handled 2
>>>>>>> SMC connections 2
>>>>>>> Handshake errors 0
>>>>>>> Avg requests per SMC conn 1277462.0
>>>>>>> TCP fallback 0
>>>>>>>
>>>>>>> RX Stats
>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>> Total requests 1277190
>>>>>>> Buffer full 45 (0.00%)
>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>>>
>>>>>>> TX Stats
>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>> Total requests 1277734
>>>>>>> Buffer full 638239 (49.95%)
>>>>>>> Buffer full (remote) 0 (0.00%)
>>>>>>> Buffer too small 0 (0.00%)
>>>>>>> Buffer too small (remote) 0 (0.00%)
>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>>>
>>>>>>> Extras
>>>>>>> Special socket calls 1
>>>>>>>
>>>>>>> I captured the perf information and found that the percentage of
>>>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>>>>> during tx and rx.
>>>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>>>
>>>>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>>>>> Thanks.
>>>>>>
>>>>>> Hi Zhengchao Shao,
>>>>>>
>>>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
>>>>>> like TCP, it is necessary to appropriately increase the default value of smc
>>>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>>>
>>>>>> Set this and try again:
>>>>>>
>>>>>> # sysctl -w net.smc.wmem=1048576
>>>>>> # sysctl -w net.smc.rmem=1048576
>>>>>>
>>>>>> (The initial value of wmem and rmem are 64K)
>>>>>>
>>>>>> Regards,
>>>>>> Wen Gu
>>>>>>
>>>>>>>
>>>>>>> Zhengchao Shao
>>>>> Hi Wen Gu:
>>>>> It solves the issue. Thank you very much.
>>>>>
>>>>> Zhengchao Shao
>>>>>
>>>> Hi Wen Gu:
>>>> I've tested all the performance test items in the patchset. The
>>>> performance improvement is to be expected, except for nignx.
>>>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>>>> the following command:
>>>> <smc_run> nignx
>>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>>
>>>> The test results are as follows:
>>>> TCP SMC_lo
>>>> Requests/s 309425.42 135547.25(-56.19%)
>>>> The performance decreases by 56.19%.
>>>>
>>>> I capture packets and find that wrk can perform HTTP GET after each
>>>> connect when smc_loopback is disabled.
>>>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>>>> I wonder if there is some compatibility problem with the SMC protocol when encapsulate packet? Could you give me
>>>> some advice?
>>>> In the attachment, I captured some of the packets.
>>>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>>>> enabled.
>>>> Thank you very much.
>>>>
>>>> Zhengchao Shao
>>>>
>>>>
>>>>
>>> Hi Wen Gu:
>>> When the VM is configured with 8 cores and 16 GB memory, run
>>> the following command:
>>> <smc_run> nignx
>>> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
>>> the test data is as follows:
>>> TCP SMC_lo
>>> Requests/s 66056.66 94526.66(43.10%)
>>>
>>> But When the VM is configured with 48 cores and 32 GB memory, run
>>> the following command:
>>> <smc_run> nignx
>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>> the test data is as follows:
>>> TCP SMC_lo
>>> Requests/s 309425.42 135547.25(-56.19%)
>>>
>>> It seems that in the scenario with a large number of CPU cores,
>>> performance is not optimized, but performance deteriorates. What I
>>> missed?
>>> Thank you.
>>>
>>> Zhengchao Shao
>>
>> Hi Zhengchao,
>>
>> I failed to reproduce this large regression. Could you please share some
>> information about your test environment?
>>
> Hi Wen Gu:
>> - The nginx configure.
> See the nginx.conf file in the attachment.
>> - The guest(VM) cpu topology.
> See the vm_cpuinfo file in the attachment.
>> - The host(physical machine) cpu topology.
> See the host_cpuinfo file in the attachment.
>> - The mapping relationship between vcpu of guest(VM) and physical cpu of host.
> See the cpu_map file in the attachment.
>> - The cpu usage (top) when regression happens.
>>
> See the perf_top and perf.svg file in the attachment.
Hi Zhengchao,
Thank you for the detailed information.
In the flame graph you provided, there are clearly prolonged spin-wait in
both smc_rx_recvmsg and smc_tx_sendmsg. The footprint involves the
__check_object_size() and find_vmap_area(). I think the regression relates
to the spin lock contention when CONFIG_HARDENED_USERCOPY is set and Tx / Rx
concurrently copy data between userspace and kernel vzalloced DMB.
App1 App2
| ^
| | userspace
----------------------
| +-----+ | kernel
+--->| DMB |---+
+-----+
- smc_tx_sendmsg -> memcpy_from_msg -> copy_from_iter -> check_copy_size ->
check_object_size -> check_heap_object -> if(vm) find_vmap_area -> try to hold spin lock vmap_area_lock
- smc_rx_recvmsg -> memcpy_to_msg -> copy_to_iter -> check_copy_size ->
check_object_size -> check_heap_object -> if(vm) find_vmap_area -> try to hold spin lock vmap_area_lock
So I reproduced your test (thanks again for the details) and changed the DMB
creation from vzalloc to kzalloc(or alloc_page), thereby avoiding the spin lock
contention in find_vmap_area. Then the regression disappears.
(The attachments include flame graphs that use vzalloc and kzalloc respectively.)
SMC
-c1000 -t8 615397.66
-c1000 -t96 625627.69
diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
index 909c0699e91c..d6c9cd1a2f5b 100644
--- a/net/smc/smc_loopback.c
+++ b/net/smc/smc_loopback.c
@@ -191,7 +191,9 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
}
dmb_node->sba_idx = sba_idx;
- dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
+ dmb_node->cpu_addr = kzalloc(dmb->dmb_len, GFP_KERNEL |
+ __GFP_NOWARN | __GFP_NORETRY |
+ __GFP_NOMEMALLOC);
if (!dmb_node->cpu_addr) {
rc = -ENOMEM;
goto err_node;
@@ -260,7 +262,7 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
write_unlock(&ldev->dmb_ht_lock);
clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
- vfree(dmb_node->cpu_addr);
+ kfree(dmb_node->cpu_addr);
kfree(dmb_node);
SMC_LO_STAT_DMBS_DEC(ldev);
Hope this works for you. And it needs to reconsider if the virtual alloced
memory is the right way for DMB in loopback-ism.
Best regards,
Wen Gu
>> Thank you.
>>
>
> Thank you very much.
>
> Zhengchao Shao
[-- Attachment #2: smc_vzalloc_t96.svg.svg --]
[-- Type: image/svg+xml, Size: 202182 bytes --]
[-- Attachment #3: smc_kzalloc_t96.svg.svg --]
[-- Type: image/svg+xml, Size: 296582 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-26 11:52 ` Wen Gu
@ 2023-12-27 1:12 ` shaozhengchao
2023-12-27 1:58 ` Wen Gu
0 siblings, 1 reply; 14+ messages in thread
From: shaozhengchao @ 2023-12-27 1:12 UTC (permalink / raw)
To: Wen Gu
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/26 19:52, Wen Gu wrote:
>
>
> On 2023/12/14 11:17, shaozhengchao wrote:
>>
>>
>> On 2023/12/13 20:59, Wen Gu wrote:
>>> On 2023/12/13 17:00, shaozhengchao wrote:
>>>>
>>>>
>>>> On 2023/12/5 14:45, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>>>>
>>>>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>>>>> Hi Wen Gu:
>>>>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO
>>>>>>>>>>> feature on
>>>>>>>>>>> my x86_64 environment and kernel is based on linux-next,
>>>>>>>>>>> commit: 5ba73bec5e7b.
>>>>>>>>>>> The test result shows that the smc_lo feature cannot be
>>>>>>>>>>> enabled. Here's
>>>>>>>>>>> my analysis:
>>>>>>>>>>>
>>>>>>>>>>> 1. Run the following command to perform the test, and then
>>>>>>>>>>> capture
>>>>>>>>>>> packets on the lo device.
>>>>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp
>>>>>>>>>>> --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>>>>
>>>>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE
>>>>>>>>>>> port replies with
>>>>>>>>>>> SMC-R-Deline packets.
>>>>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>>>>
>>>>>>>>>>> 3. Rx
>>>>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE
>>>>>>>>>>> port returns
>>>>>>>>>>> a Decline packet because eid_cnt and flag.seid in the
>>>>>>>>>>> received packet are both 0.
>>>>>>>>>>>
>>>>>>>>>>> 4. Tx
>>>>>>>>>>> In smc_clc_send_proposal,
>>>>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>>>>
>>>>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment,
>>>>>>>>>>> seid_enabled is
>>>>>>>>>>> always equal to 0.
>>>>>>>>>>>
>>>>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid
>>>>>>>>>>> count?
>>>>>>>>>>> But I don't see where operations can be added, may I missed
>>>>>>>>>>> something?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi Zhengchao Shao,
>>>>>>>>>>
>>>>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86
>>>>>>>>>> here), A common
>>>>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>>>>
>>>>>>>>>> - Install smc-tools[1].
>>>>>>>>>>
>>>>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>>>>
>>>>>>>>>> EID works as an ID to indicate the max communication space
>>>>>>>>>> of SMC. When SEID is
>>>>>>>>>> unavailable, an UEID is required.
>>>>>>>>>>
>>>>>>>>> Hi Wen Gu:
>>>>>>>>> Thank you for your reply. This is very useful for me. And
>>>>>>>>> I will
>>>>>>>>> be happy to learn from it.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>>
>>>>>>>>> Zhengchao Shao
>>>>>>>>>> - Then run the test.
>>>>>>>>>>
>>>>>>>>>> Hope this works for you :)
>>>>>>>>>>
>>>>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Wen Gu
>>>>>>>>>>
>>>>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>>>>
>>>>>>>>>>> Zhengchao Shao
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1]link:
>>>>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Hi Wen Gu:
>>>>>>>> I have test as following, but the performance is really
>>>>>>>> degraded. Now I have no idea.
>>>>>>>> 1. add ueid
>>>>>>>> run: smcd ueid add 16
>>>>>>>> kernel message:
>>>>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>>>>> 2. start server
>>>>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>>>>> 3. start client
>>>>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i
>>>>>>>> 127.0.0.1 -t 30
>>>>>>>>
>>>>>>>> The test results are as follows:
>>>>>>>> TCP SMC-lo
>>>>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>>>>
>>>>>>>> I didn't find a better direction when I initially positioned it. No
>>>>>>>> error is recorded in the kernel log, and the smcd statistics are
>>>>>>>> normal.
>>>>>>>> [root@localhost smc-tools]# smcd stats
>>>>>>>> SMC-D Connections Summary
>>>>>>>> Total connections handled 2
>>>>>>>> SMC connections 2
>>>>>>>> Handshake errors 0
>>>>>>>> Avg requests per SMC conn 1277462.0
>>>>>>>> TCP fallback 0
>>>>>>>>
>>>>>>>> RX Stats
>>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>>> Total requests 1277190
>>>>>>>> Buffer full 45 (0.00%)
>>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>>>>>> >512KB
>>>>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>>>>
>>>>>>>> TX Stats
>>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>>> Total requests 1277734
>>>>>>>> Buffer full 638239 (49.95%)
>>>>>>>> Buffer full (remote) 0 (0.00%)
>>>>>>>> Buffer too small 0 (0.00%)
>>>>>>>> Buffer too small (remote) 0 (0.00%)
>>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB
>>>>>>>> >512KB
>>>>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>>>>
>>>>>>>> Extras
>>>>>>>> Special socket calls 1
>>>>>>>>
>>>>>>>> I captured the perf information and found that the percentage of
>>>>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions
>>>>>>>> was high
>>>>>>>> during tx and rx.
>>>>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>>>>
>>>>>>>> I've attached the flame map. Could you help analyze it? What I
>>>>>>>> missed?
>>>>>>>> Thanks.
>>>>>>>
>>>>>>> Hi Zhengchao Shao,
>>>>>>>
>>>>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't
>>>>>>> grow dynamically
>>>>>>> like TCP, it is necessary to appropriately increase the default
>>>>>>> value of smc
>>>>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>>>>
>>>>>>> Set this and try again:
>>>>>>>
>>>>>>> # sysctl -w net.smc.wmem=1048576
>>>>>>> # sysctl -w net.smc.rmem=1048576
>>>>>>>
>>>>>>> (The initial value of wmem and rmem are 64K)
>>>>>>>
>>>>>>> Regards,
>>>>>>> Wen Gu
>>>>>>>
>>>>>>>>
>>>>>>>> Zhengchao Shao
>>>>>> Hi Wen Gu:
>>>>>> It solves the issue. Thank you very much.
>>>>>>
>>>>>> Zhengchao Shao
>>>>>>
>>>>> Hi Wen Gu:
>>>>> I've tested all the performance test items in the patchset. The
>>>>> performance improvement is to be expected, except for nignx.
>>>>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>>>>> the following command:
>>>>> <smc_run> nignx
>>>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>>>
>>>>> The test results are as follows:
>>>>> TCP SMC_lo
>>>>> Requests/s 309425.42 135547.25(-56.19%)
>>>>> The performance decreases by 56.19%.
>>>>>
>>>>> I capture packets and find that wrk can perform HTTP GET after each
>>>>> connect when smc_loopback is disabled.
>>>>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>>>>> I wonder if there is some compatibility problem with the SMC
>>>>> protocol when encapsulate packet? Could you give me some advice?
>>>>> In the attachment, I captured some of the packets.
>>>>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>>>>> enabled.
>>>>> Thank you very much.
>>>>>
>>>>> Zhengchao Shao
>>>>>
>>>>>
>>>>>
>>>> Hi Wen Gu:
>>>> When the VM is configured with 8 cores and 16 GB memory, run
>>>> the following command:
>>>> <smc_run> nignx
>>>> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
>>>> the test data is as follows:
>>>> TCP SMC_lo
>>>> Requests/s 66056.66 94526.66(43.10%)
>>>>
>>>> But When the VM is configured with 48 cores and 32 GB memory, run
>>>> the following command:
>>>> <smc_run> nignx
>>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>> the test data is as follows:
>>>> TCP SMC_lo
>>>> Requests/s 309425.42 135547.25(-56.19%)
>>>>
>>>> It seems that in the scenario with a large number of CPU cores,
>>>> performance is not optimized, but performance deteriorates. What I
>>>> missed?
>>>> Thank you.
>>>>
>>>> Zhengchao Shao
>>>
>>> Hi Zhengchao,
>>>
>>> I failed to reproduce this large regression. Could you please share some
>>> information about your test environment?
>>>
>> Hi Wen Gu:
>>> - The nginx configure.
>> See the nginx.conf file in the attachment.
>>> - The guest(VM) cpu topology.
>> See the vm_cpuinfo file in the attachment.
>>> - The host(physical machine) cpu topology.
>> See the host_cpuinfo file in the attachment.
>>> - The mapping relationship between vcpu of guest(VM) and physical cpu
>>> of host.
>> See the cpu_map file in the attachment.
>>> - The cpu usage (top) when regression happens.
>>>
>> See the perf_top and perf.svg file in the attachment.
>
> Hi Zhengchao,
>
> Thank you for the detailed information.
>
> In the flame graph you provided, there are clearly prolonged spin-wait in
> both smc_rx_recvmsg and smc_tx_sendmsg. The footprint involves the
> __check_object_size() and find_vmap_area(). I think the regression relates
> to the spin lock contention when CONFIG_HARDENED_USERCOPY is set and Tx
> / Rx
> concurrently copy data between userspace and kernel vzalloced DMB.
>
> App1 App2
> | ^
> | | userspace
> ----------------------
> | +-----+ | kernel
> +--->| DMB |---+
> +-----+
>
> - smc_tx_sendmsg -> memcpy_from_msg -> copy_from_iter -> check_copy_size ->
> check_object_size -> check_heap_object -> if(vm) find_vmap_area ->
> try to hold spin lock vmap_area_lock
>
> - smc_rx_recvmsg -> memcpy_to_msg -> copy_to_iter -> check_copy_size ->
> check_object_size -> check_heap_object -> if(vm) find_vmap_area ->
> try to hold spin lock vmap_area_lock
>
> So I reproduced your test (thanks again for the details) and changed the
> DMB
> creation from vzalloc to kzalloc(or alloc_page), thereby avoiding the
> spin lock
> contention in find_vmap_area. Then the regression disappears.
> (The attachments include flame graphs that use vzalloc and kzalloc
> respectively.)
>
> SMC
> -c1000 -t8 615397.66
> -c1000 -t96 625627.69
>
> diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
> index 909c0699e91c..d6c9cd1a2f5b 100644
> --- a/net/smc/smc_loopback.c
> +++ b/net/smc/smc_loopback.c
> @@ -191,7 +191,9 @@ static int smc_lo_register_dmb(struct smcd_dev
> *smcd, struct smcd_dmb *dmb,
> }
>
> dmb_node->sba_idx = sba_idx;
> - dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
> + dmb_node->cpu_addr = kzalloc(dmb->dmb_len, GFP_KERNEL |
> + __GFP_NOWARN | __GFP_NORETRY |
> + __GFP_NOMEMALLOC);
> if (!dmb_node->cpu_addr) {
> rc = -ENOMEM;
> goto err_node;
> @@ -260,7 +262,7 @@ static int smc_lo_unregister_dmb(struct smcd_dev
> *smcd, struct smcd_dmb *dmb)
> write_unlock(&ldev->dmb_ht_lock);
>
> clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
> - vfree(dmb_node->cpu_addr);
> + kfree(dmb_node->cpu_addr);
> kfree(dmb_node);
> SMC_LO_STAT_DMBS_DEC(ldev);
>
>
> Hope this works for you. And it needs to reconsider if the virtual alloced
> memory is the right way for DMB in loopback-ism.
>
Hi Wen Gu：
This patch works in my VM. Thank you for your support. And,
will this change take into your patchset?
Thanks very much.
Zhengchao Shao
> Best regards,
> Wen Gu
>
>>> Thank you.
>>>
>>
>> Thank you very much.
>>
>> Zhengchao Shao
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [question] smc: how to enable SMC_LO feature
2023-12-27 1:12 ` shaozhengchao
@ 2023-12-27 1:58 ` Wen Gu
0 siblings, 0 replies; 14+ messages in thread
From: Wen Gu @ 2023-12-27 1:58 UTC (permalink / raw)
To: shaozhengchao
Cc: netdev@vger.kernel.org, yuehaibing, Libin (Huawei), Dust Li,
tonylu_linux, D. Wythe
On 2023/12/27 09:12, shaozhengchao wrote:
>
>
> On 2023/12/26 19:52, Wen Gu wrote:
>>
>>
>> On 2023/12/14 11:17, shaozhengchao wrote:
>>>
>>>
>>> On 2023/12/13 20:59, Wen Gu wrote:
>>>> On 2023/12/13 17:00, shaozhengchao wrote:
>>>>>
>>>>>
>>>>> On 2023/12/5 14:45, shaozhengchao wrote:
>>>>>>
>>>>>>
>>>>>> On 2023/12/4 12:06, shaozhengchao wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2023/12/4 11:52, Wen Gu wrote:
>>>>>>>>
>>>>>>>> On 2023/12/4 11:22, shaozhengchao wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2023/11/23 14:15, shaozhengchao wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2023/11/23 10:21, Wen Gu wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2023/11/21 20:14, shaozhengchao wrote:
>>>>>>>>>>>> Hi Wen Gu:
>>>>>>>>>>>> Currently, I am interested in the SMC_LOOPBACK feature proposed
>>>>>>>>>>>> by you. Therefore, I use your patchset[1] to test the SMC_LO feature on
>>>>>>>>>>>> my x86_64 environment and kernel is based on linux-next, commit: 5ba73bec5e7b.
>>>>>>>>>>>> The test result shows that the smc_lo feature cannot be enabled. Here's
>>>>>>>>>>>> my analysis:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Run the following command to perform the test, and then capture
>>>>>>>>>>>> packets on the lo device.
>>>>>>>>>>>> - serv: smc_run taskset -c <cpu> sockperf sr --tcp
>>>>>>>>>>>> - clnt: smc_run taskset -c <cpu> sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>>>>>
>>>>>>>>>>>> 2. Use Wireshark to open packets. It is found that the VCE port replies with
>>>>>>>>>>>> SMC-R-Deline packets.
>>>>>>>>>>>> [cid:image001.png@01DA1CB4.F1052C30]
>>>>>>>>>>>>
>>>>>>>>>>>> 3. Rx
>>>>>>>>>>>> When smc_listen_work invokes smc_listen_v2_check, the VCE port returns
>>>>>>>>>>>> a Decline packet because eid_cnt and flag.seid in the received packet are both 0.
>>>>>>>>>>>>
>>>>>>>>>>>> 4. Tx
>>>>>>>>>>>> In smc_clc_send_proposal,
>>>>>>>>>>>> v2_ext->hdr.eid_cnt = smc_clc_eid_table.ueid_cnt;
>>>>>>>>>>>> v2_ext->hdr.flag.seid = smc_clc_eid_table.seid_enabled;
>>>>>>>>>>>>
>>>>>>>>>>>> When smc_clc_init, ueid_cnt=0, and in the x86_64 environment, seid_enabled is
>>>>>>>>>>>> always equal to 0.
>>>>>>>>>>>>
>>>>>>>>>>>> So, I must call smc_clc_ueid_add function to increase ueid count?
>>>>>>>>>>>> But I don't see where operations can be added, may I missed something?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Hi Zhengchao Shao,
>>>>>>>>>>>
>>>>>>>>>>> Yes. When using SMC-D in non-s390 architecture (like x86 here), A common
>>>>>>>>>>> UEID should be set. It can be set by following steps:
>>>>>>>>>>>
>>>>>>>>>>> - Install smc-tools[1].
>>>>>>>>>>>
>>>>>>>>>>> - Run # smcd ueid add <ueid> in loopback test environment.
>>>>>>>>>>>
>>>>>>>>>>> EID works as an ID to indicate the max communication space of SMC. When SEID is
>>>>>>>>>>> unavailable, an UEID is required.
>>>>>>>>>>>
>>>>>>>>>> Hi Wen Gu:
>>>>>>>>>> Thank you for your reply. This is very useful for me. And I will
>>>>>>>>>> be happy to learn from it.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Zhengchao Shao
>>>>>>>>>>> - Then run the test.
>>>>>>>>>>>
>>>>>>>>>>> Hope this works for you :)
>>>>>>>>>>>
>>>>>>>>>>> [1] https://github.com/ibm-s390-linux/smc-tools
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>> Wen Gu
>>>>>>>>>>>
>>>>>>>>>>>> Could you give me some advice? Thanks very much.
>>>>>>>>>>>>
>>>>>>>>>>>> Zhengchao Shao
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1]link:
>>>>>>>>>>>> https://patchwork.kernel.org/project/netdevbpf/cover/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> Hi Wen Gu:
>>>>>>>>> I have test as following, but the performance is really
>>>>>>>>> degraded. Now I have no idea.
>>>>>>>>> 1. add ueid
>>>>>>>>> run: smcd ueid add 16
>>>>>>>>> kernel message:
>>>>>>>>> [ 5252.009133] NET: Registered PF_SMC protocol family
>>>>>>>>> [ 5252.009233] smc: adding smcd device smc_lo with pnetid
>>>>>>>>> 2. start server
>>>>>>>>> smc_run taskset -c 1 sockperf sr --tcp
>>>>>>>>> 3. start client
>>>>>>>>> smc_run taskset -c 3 sockperf tp --tcp --msg-size=64000 -i 127.0.0.1 -t 30
>>>>>>>>>
>>>>>>>>> The test results are as follows:
>>>>>>>>> TCP SMC-lo
>>>>>>>>> Bandwidth(MBps) 1890.56 1300.41(-31.22%)
>>>>>>>>>
>>>>>>>>> I didn't find a better direction when I initially positioned it. No
>>>>>>>>> error is recorded in the kernel log, and the smcd statistics are normal.
>>>>>>>>> [root@localhost smc-tools]# smcd stats
>>>>>>>>> SMC-D Connections Summary
>>>>>>>>> Total connections handled 2
>>>>>>>>> SMC connections 2
>>>>>>>>> Handshake errors 0
>>>>>>>>> Avg requests per SMC conn 1277462.0
>>>>>>>>> TCP fallback 0
>>>>>>>>>
>>>>>>>>> RX Stats
>>>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>>>> Total requests 1277190
>>>>>>>>> Buffer full 45 (0.00%)
>>>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>>>>>> Bufs 0 0 0 2 0 0 0 0
>>>>>>>>> Reqs 638.0K 0 0 639.2K 0 0 0 0
>>>>>>>>>
>>>>>>>>> TX Stats
>>>>>>>>> Data transmitted (Bytes) 40907328000 (40.91G)
>>>>>>>>> Total requests 1277734
>>>>>>>>> Buffer full 638239 (49.95%)
>>>>>>>>> Buffer full (remote) 0 (0.00%)
>>>>>>>>> Buffer too small 0 (0.00%)
>>>>>>>>> Buffer too small (remote) 0 (0.00%)
>>>>>>>>> 8KB 16KB 32KB 64KB 128KB 256KB 512KB >512KB
>>>>>>>>> Bufs 0 0 0 0 0 0 0 0
>>>>>>>>> Reqs 0 0 0 1.278M 0 0 0 0
>>>>>>>>>
>>>>>>>>> Extras
>>>>>>>>> Special socket calls 1
>>>>>>>>>
>>>>>>>>> I captured the perf information and found that the percentage of
>>>>>>>>> rep_movs_alternative and _raw_spin_unlock_irqrestore functions was high
>>>>>>>>> during tx and rx.
>>>>>>>>> 36.12% [kernel] [k]rep_movs_alternative
>>>>>>>>> 14.23% [kernel] [k]_raw_spin_unlock_irqrestore
>>>>>>>>>
>>>>>>>>> I've attached the flame map. Could you help analyze it? What I missed?
>>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Hi Zhengchao Shao,
>>>>>>>>
>>>>>>>> Since sndbuf and RMB in SMC are pre-alloced ringbuf and won't grow dynamically
>>>>>>>> like TCP, it is necessary to appropriately increase the default value of smc
>>>>>>>> sk_sndbuf and sk_rcvbuf before testing throughput.
>>>>>>>>
>>>>>>>> Set this and try again:
>>>>>>>>
>>>>>>>> # sysctl -w net.smc.wmem=1048576
>>>>>>>> # sysctl -w net.smc.rmem=1048576
>>>>>>>>
>>>>>>>> (The initial value of wmem and rmem are 64K)
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Wen Gu
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Zhengchao Shao
>>>>>>> Hi Wen Gu:
>>>>>>> It solves the issue. Thank you very much.
>>>>>>>
>>>>>>> Zhengchao Shao
>>>>>>>
>>>>>> Hi Wen Gu:
>>>>>> I've tested all the performance test items in the patchset. The
>>>>>> performance improvement is to be expected, except for nignx.
>>>>>> My VM is configured with 48 cores and 32 GB memory. Therefore, run
>>>>>> the following command:
>>>>>> <smc_run> nignx
>>>>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>>>>
>>>>>> The test results are as follows:
>>>>>> TCP SMC_lo
>>>>>> Requests/s 309425.42 135547.25(-56.19%)
>>>>>> The performance decreases by 56.19%.
>>>>>>
>>>>>> I capture packets and find that wrk can perform HTTP GET after each
>>>>>> connect when smc_loopback is disabled.
>>>>>> However, when smc_loopback is enabled, there is no HTTP GET behavior.
>>>>>> I wonder if there is some compatibility problem with the SMC protocol when encapsulate packet? Could you give me
>>>>>> some advice?
>>>>>> In the attachment, I captured some of the packets.
>>>>>> nosmc_nginx.pcap is for SMC disabled and smc_nginx.pcap is for SMC
>>>>>> enabled.
>>>>>> Thank you very much.
>>>>>>
>>>>>> Zhengchao Shao
>>>>>>
>>>>>>
>>>>>>
>>>>> Hi Wen Gu:
>>>>> When the VM is configured with 8 cores and 16 GB memory, run
>>>>> the following command:
>>>>> <smc_run> nignx
>>>>> <smc_run>./wrk -t 8 -c 1000 -d 30 http://127.0.0.1:80
>>>>> the test data is as follows:
>>>>> TCP SMC_lo
>>>>> Requests/s 66056.66 94526.66(43.10%)
>>>>>
>>>>> But When the VM is configured with 48 cores and 32 GB memory, run
>>>>> the following command:
>>>>> <smc_run> nignx
>>>>> <smc_run>./wrk -t 96 -c 1000 -d 30 http://127.0.0.1:80
>>>>> the test data is as follows:
>>>>> TCP SMC_lo
>>>>> Requests/s 309425.42 135547.25(-56.19%)
>>>>>
>>>>> It seems that in the scenario with a large number of CPU cores,
>>>>> performance is not optimized, but performance deteriorates. What I
>>>>> missed?
>>>>> Thank you.
>>>>>
>>>>> Zhengchao Shao
>>>>
>>>> Hi Zhengchao,
>>>>
>>>> I failed to reproduce this large regression. Could you please share some
>>>> information about your test environment?
>>>>
>>> Hi Wen Gu:
>>>> - The nginx configure.
>>> See the nginx.conf file in the attachment.
>>>> - The guest(VM) cpu topology.
>>> See the vm_cpuinfo file in the attachment.
>>>> - The host(physical machine) cpu topology.
>>> See the host_cpuinfo file in the attachment.
>>>> - The mapping relationship between vcpu of guest(VM) and physical cpu of host.
>>> See the cpu_map file in the attachment.
>>>> - The cpu usage (top) when regression happens.
>>>>
>>> See the perf_top and perf.svg file in the attachment.
>>
>> Hi Zhengchao,
>>
>> Thank you for the detailed information.
>>
>> In the flame graph you provided, there are clearly prolonged spin-wait in
>> both smc_rx_recvmsg and smc_tx_sendmsg. The footprint involves the
>> __check_object_size() and find_vmap_area(). I think the regression relates
>> to the spin lock contention when CONFIG_HARDENED_USERCOPY is set and Tx / Rx
>> concurrently copy data between userspace and kernel vzalloced DMB.
>>
>> App1 App2
>> | ^
>> | | userspace
>> ----------------------
>> | +-----+ | kernel
>> +--->| DMB |---+
>> +-----+
>>
>> - smc_tx_sendmsg -> memcpy_from_msg -> copy_from_iter -> check_copy_size ->
>> check_object_size -> check_heap_object -> if(vm) find_vmap_area -> try to hold spin lock vmap_area_lock
>>
>> - smc_rx_recvmsg -> memcpy_to_msg -> copy_to_iter -> check_copy_size ->
>> check_object_size -> check_heap_object -> if(vm) find_vmap_area -> try to hold spin lock vmap_area_lock
>>
>> So I reproduced your test (thanks again for the details) and changed the DMB
>> creation from vzalloc to kzalloc(or alloc_page), thereby avoiding the spin lock
>> contention in find_vmap_area. Then the regression disappears.
>> (The attachments include flame graphs that use vzalloc and kzalloc respectively.)
>>
>> SMC
>> -c1000 -t8 615397.66
>> -c1000 -t96 625627.69
>>
>> diff --git a/net/smc/smc_loopback.c b/net/smc/smc_loopback.c
>> index 909c0699e91c..d6c9cd1a2f5b 100644
>> --- a/net/smc/smc_loopback.c
>> +++ b/net/smc/smc_loopback.c
>> @@ -191,7 +191,9 @@ static int smc_lo_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb,
>> }
>>
>> dmb_node->sba_idx = sba_idx;
>> - dmb_node->cpu_addr = vzalloc(dmb->dmb_len);
>> + dmb_node->cpu_addr = kzalloc(dmb->dmb_len, GFP_KERNEL |
>> + __GFP_NOWARN | __GFP_NORETRY |
>> + __GFP_NOMEMALLOC);
>> if (!dmb_node->cpu_addr) {
>> rc = -ENOMEM;
>> goto err_node;
>> @@ -260,7 +262,7 @@ static int smc_lo_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
>> write_unlock(&ldev->dmb_ht_lock);
>>
>> clear_bit(dmb_node->sba_idx, ldev->sba_idx_mask);
>> - vfree(dmb_node->cpu_addr);
>> + kfree(dmb_node->cpu_addr);
>> kfree(dmb_node);
>> SMC_LO_STAT_DMBS_DEC(ldev);
>>
>>
>> Hope this works for you. And it needs to reconsider if the virtual alloced
>> memory is the right way for DMB in loopback-ism.
>>
> Hi Wen Gu：
> This patch works in my VM. Thank you for your support. And,
> will this change take into your patchset?
> Thanks very much.
>
Yes, the regression caused by vzalloced DMB in concurrency scenarios will
be handled in my following patchset. But whether in this way I need to think
about it, since physical contiguous memory is expensive.
Best regards,
Wen Gu
> Zhengchao Shao
>> Best regards,
>> Wen Gu
>>
>>>> Thank you.
>>>>
>>>
>>> Thank you very much.
>>>
>>> Zhengchao Shao
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2023-12-27 1:58 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <8ac15e20beb54acfae1a35d1603c1827@huawei.com>
2023-11-23 2:21 ` [question] smc: how to enable SMC_LO feature Wen Gu
2023-11-23 6:15 ` shaozhengchao
2023-12-04 3:22 ` shaozhengchao
2023-12-04 3:52 ` Wen Gu
2023-12-04 4:06 ` shaozhengchao
2023-12-05 6:45 ` shaozhengchao
2023-12-13 9:00 ` shaozhengchao
2023-12-13 9:41 ` Wen Gu
2023-12-13 12:59 ` Wen Gu
2023-12-14 3:17 ` shaozhengchao
2023-12-26 11:52 ` Wen Gu
2023-12-27 1:12 ` shaozhengchao
2023-12-27 1:58 ` Wen Gu
2023-12-13 13:06 ` Wen Gu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).