Re: [LKP] [lkp] [sctp] a6c2f79287: netperf.Throughput_Mbps -37.2% regression

[Aaron Lu <aaron.lu@intel.com>]

On Wed, Aug 17, 2016 at 02:37:19PM +0800, Aaron Lu wrote:
> On Wed, Aug 17, 2016 at 02:14:05PM +0800, Aaron Lu wrote:
> > On Wed, Aug 17, 2016 at 01:41:04PM +0800, Xin Long wrote:
> > > > The perf-profile data for the two commits are attached(for the case of
> > > > prsctp_enable=1, the perf-profile data doesn't get collected for the 0
> > > > case for some reason, I'm checking the problem now).
> > > >
> > > > The CPU gets much more idle time in the bisected commit a6c2f79287:
> > > >
> > > >     68.89%     0.70%  [kernel.kallsyms]   [k] entry_SYSCALL_64_fastpath
> > > >     49.32%     0.12%  [kernel.kallsyms]   [k] sys_sendmsg
> > > >     49.17%     0.12%  [kernel.kallsyms]   [k] __sys_sendmsg
> > > >     48.58%     0.22%  [kernel.kallsyms]   [k] ___sys_sendmsg
> > > >     46.69%     0.06%  [kernel.kallsyms]   [k] sock_sendmsg
> > > >     46.31%     0.16%  [kernel.kallsyms]   [k] inet_sendmsg
> > > >     45.90%     0.98%  [kernel.kallsyms]   [k] sctp_sendmsg
> > > >     29.66%     0.45%  [kernel.kallsyms]   [k] sctp_do_sm
> > > >     29.54%     0.23%  [kernel.kallsyms]   [k] cpu_startup_entry
> > > >     28.81%     0.68%  [kernel.kallsyms]   [k] sctp_cmd_interpreter.isra.24
> > > >     26.20%     0.00%  [kernel.kallsyms]   [k] start_secondary
> > > >     23.04%     0.09%  [kernel.kallsyms]   [k] sctp_inq_push
> > > >     23.03%     0.08%  [kernel.kallsyms]   [k] call_cpuidle
> > > >     22.94%     0.00%  [kernel.kallsyms]   [k] cpuidle_enter
> > > >     22.60%     0.18%  [kernel.kallsyms]   [k] cpuidle_enter_state
> > > >     21.99%    21.99%  [kernel.kallsyms]   [k] intel_idle
> > > > ... ...
> > > >
> > > > While its immediate parent commit 826d253d57 is mostly busy working:
> > > >
> > > >     98.53%     0.83%  [kernel.kallsyms]   [k] entry_SYSCALL_64_fastpath
> > > >     78.13%     0.12%  [kernel.kallsyms]   [k] sys_sendmsg
> > > >     78.03%     0.16%  [kernel.kallsyms]   [k] __sys_sendmsg
> > > >     77.08%     0.28%  [kernel.kallsyms]   [k] ___sys_sendmsg
> > > >     74.44%     0.08%  [kernel.kallsyms]   [k] sock_sendmsg
> > > >     73.82%     0.13%  [kernel.kallsyms]   [k] inet_sendmsg
> > > >     73.34%     1.44%  [kernel.kallsyms]   [k] sctp_sendmsg
> > > >     47.52%     0.75%  [kernel.kallsyms]   [k] sctp_do_sm
> > > >     46.19%     0.90%  [kernel.kallsyms]   [k] sctp_cmd_interpreter.isra.24
> > > >     37.17%     1.43%  [kernel.kallsyms]   [k] sctp_outq_flush
> > > >     36.93%     0.08%  [kernel.kallsyms]   [k] sctp_outq_uncork
> > > >     34.24%     0.15%  [kernel.kallsyms]   [k] sctp_inq_push
> > > > ... ...
> > > > No idle related function above 1%.
> > > >
> > > > Will the bisected commit make the idle possible?
> > > No, not at all. :)
> > > 
> > > pls help to debug as I said in the last reply.
> > 
> > OK, will see how to do that.
> > 
> > In the meantime, I just tried to reproduce on my own desktop:
> > Sandybridge i7-2600 CPU @ 3.40GHz and it reproduced:
> > $ cat 4.7.0-rc6-01198-ga6c2f792873a/0/netperf.json
> > {
> >   "netperf.Throughput_Mbps": [
> >    752.9450000000002
> >   ]
> > }
> > $ cat 4.7.0-rc6-01197-g826d253d57b1/0/netperf.json
> > {
> >   "netperf.Throughput_Mbps": [
> >    1068.5556249999997
> >   ]
> > }
> 
> On top of
> commit 826d253d57b1 ("sctp: add SCTP_PR_ASSOC_STATUS on sctp sockopt")
> I applied the below commit:
> 
> From 98dd2532b14e29dcc2ab40a7348755531afa79e4 Mon Sep 17 00:00:00 2001
> From: Aaron Lu <aaron.lu@intel.com>
> Date: Wed, 17 Aug 2016 14:20:00 +0800
> Subject: [PATCH] sctp: test
> 
> ---
>  include/net/sctp/structs.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/net/sctp/structs.h b/include/net/sctp/structs.h
> index d8e464aacb20..932f2780d3a4 100644
> --- a/include/net/sctp/structs.h
> +++ b/include/net/sctp/structs.h
> @@ -602,6 +602,9 @@ struct sctp_chunk {
>  	/* This needs to be recoverable for SCTP_SEND_FAILED events. */
>  	struct sctp_sndrcvinfo sinfo;
>  
> +	unsigned long prsctp_param;
> +	int sent_count;
> +
>  	/* Which association does this belong to?  */
>  	struct sctp_association *asoc;
>  
> -- 
> 2.5.5
> 
> Then the performance dropped to the same as the bisected commit
> a6c2f792873a:
> $ cat 4.7.0-rc6-01198-g98dd2532b14e/0/netperf.json
> {
>   "netperf.Throughput_Mbps": [
>    754.494375
>   ]
> }
> 
> I think this agrees with the perf data in that the newly added function

Actually, I mean the modified functions like sctp_chunk_abandoned and
__sctp_packet_append_chunk, etc.

> doesn't show up in the perf-profile but still, the performance drops.
> So the only possible reason is the newly added fields to the sctp_chunk
> structure.
> 
> Is this expected?
> 
> Thanks,
> Aaron