From: Hyeonggon Yoo <42.hyeyoo@gmail.com>
To: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Ingo Molnar <mingo@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Shakeel Butt <shakeelb@google.com>,
Muchun Song <muchun.song@linux.dev>,
Matthew Wilcox <willy@infradead.org>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>,
Mel Gorman <mgorman@techsingularity.net>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Dennis Zhou <dennis@kernel.org>, Tejun Heo <tj@kernel.org>
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default
Date: Thu, 26 Jan 2023 00:41:15 +0900 [thread overview]
Message-ID: <Y9FNm50H+w1B26yR@hyeyoo> (raw)
In-Reply-To: <Y85MNmZDc5czMRUJ@dhcp22.suse.cz>
[-- Attachment #1: Type: text/plain, Size: 10317 bytes --]
From: Hyeonggon Yoo <42.hyeyoo@gmail.com>
To: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>, Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Ingo Molnar <mingo@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Shakeel Butt <shakeelb@google.com>,
Muchun Song <muchun.song@linux.dev>,
Matthew Wilcox <willy@infradead.org>, linux-mm@kvack.org,
linux-kernel@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>,
Mel Gorman <mgorman@techsingularity.net>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Dennis Zhou <dennis@kernel.org>, Tejun Heo <tj@kernel.org>
Bcc:
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable
DEBUG_PREEMPT by default
Reply-To:
In-Reply-To: <Y85MNmZDc5czMRUJ@dhcp22.suse.cz>
On Mon, Jan 23, 2023 at 09:58:30AM +0100, Michal Hocko wrote:
> On Sat 21-01-23 20:54:15, Hyeonggon Yoo wrote:
> > On Sat, Jan 21, 2023 at 12:29:44PM +0100, Vlastimil Babka wrote:
> > > On 1/21/23 04:39, Hyeonggon Yoo wrote:
> > > > In workloads where this_cpu operations are frequently performed,
> > > > enabling DEBUG_PREEMPT may result in significant increase in
> > > > runtime overhead due to frequent invocation of
> > > > __this_cpu_preempt_check() function.
> > > >
> > > > This can be demonstrated through benchmarks such as hackbench where this
> > > > configuration results in a 10% reduction in performance, primarily due to
> > > > the added overhead within memcg charging path.
> > > >
> > > > Therefore, do not to enable DEBUG_PREEMPT by default and make users aware
> > > > of its potential impact on performance in some workloads.
> > > >
> > > > hackbench-process-sockets
> > > > debug_preempt no_debug_preempt
> > > > Amean 1 0.4743 ( 0.00%) 0.4295 * 9.45%*
> > > > Amean 4 1.4191 ( 0.00%) 1.2650 * 10.86%*
> > > > Amean 7 2.2677 ( 0.00%) 2.0094 * 11.39%*
> > > > Amean 12 3.6821 ( 0.00%) 3.2115 * 12.78%*
> > > > Amean 21 6.6752 ( 0.00%) 5.7956 * 13.18%*
> > > > Amean 30 9.6646 ( 0.00%) 8.5197 * 11.85%*
> > > > Amean 48 15.3363 ( 0.00%) 13.5559 * 11.61%*
> > > > Amean 79 24.8603 ( 0.00%) 22.0597 * 11.27%*
> > > > Amean 96 30.1240 ( 0.00%) 26.8073 * 11.01%*
Hello Michal, thanks for looking at this.
> Do you happen to have any perf data collected during those runs? I
> would be interested in the memcg side of things. Maybe we can do
> something better there.
Yes, below is performance data I've collected.
6.1.8-debug-preempt-dirty
=========================
Overhead Command Shared Object Symbol
+ 9.14% hackbench [kernel.vmlinux] [k] check_preemption_disabled
+ 7.33% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
+ 7.32% hackbench [kernel.vmlinux] [k] mod_objcg_state
3.55% hackbench [kernel.vmlinux] [k] refill_obj_stock
3.39% hackbench [kernel.vmlinux] [k] debug_smp_processor_id
2.97% hackbench [kernel.vmlinux] [k] memset_erms
2.55% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.36% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.76% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.64% hackbench [kernel.vmlinux] [k] __slab_free
1.58% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.35% hackbench [kernel.vmlinux] [k] vfs_write
1.33% hackbench [kernel.vmlinux] [k] vfs_read
1.28% hackbench [kernel.vmlinux] [k] __alloc_skb
1.18% hackbench [kernel.vmlinux] [k] sock_read_iter
1.16% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
1.16% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64
1.14% hackbench [kernel.vmlinux] [k] sock_write_iter
1.12% hackbench [kernel.vmlinux] [k] skb_release_data
1.08% hackbench [kernel.vmlinux] [k] sock_wfree
1.07% hackbench [kernel.vmlinux] [k] cache_from_obj
0.96% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.95% hackbench [kernel.vmlinux] [k] kmem_cache_free
0.94% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.92% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
0.89% hackbench [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.84% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_array
0.84% hackbench libc.so.6 [.] write
0.81% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.76% hackbench libc.so.6 [.] read
0.75% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.75% hackbench [kernel.vmlinux] [k] preempt_count_add
0.74% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.69% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
0.65% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.63% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.62% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.60% hackbench [kernel.vmlinux] [k] __rcu_read_lock
0.59% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.54% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.51% hackbench [kernel.vmlinux] [k] security_file_permission
0.50% hackbench [kernel.vmlinux] [k] __x64_sys_write
0.49% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
0.48% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.46% hackbench [kernel.vmlinux] [k] __check_heap_object
and attached flamegraph-6.1.8-debug-preempt-dirty.svg.
6.1.8 (no debug preempt)
========================
Overhead Command Shared Object Symbol
+ 10.96% hackbench [kernel.vmlinux] [k] mod_objcg_state
+ 8.16% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
3.29% hackbench [kernel.vmlinux] [k] memset_erms
3.07% hackbench [kernel.vmlinux] [k] __slab_free
2.89% hackbench [kernel.vmlinux] [k] refill_obj_stock
2.82% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.72% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.96% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_rax
1.88% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
1.54% hackbench [kernel.vmlinux] [k] __alloc_skb
1.53% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] kmem_cache_free
1.44% hackbench [kernel.vmlinux] [k] vfs_write
1.43% hackbench [kernel.vmlinux] [k] vfs_read
1.33% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.31% hackbench [kernel.vmlinux] [k] sock_write_iter
1.27% hackbench [kernel.vmlinux] [k] kmalloc_slab
1.22% hackbench [kernel.vmlinux] [k] __rcu_read_lock
1.20% hackbench [kernel.vmlinux] [k] sock_read_iter
1.18% hackbench [kernel.vmlinux] [k] __entry_text_start
1.15% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
1.12% hackbench [kernel.vmlinux] [k] unix_stream_recvmsg
1.10% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
0.98% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.97% hackbench libc.so.6 [.] write
0.91% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.88% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.87% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.86% hackbench [kernel.vmlinux] [k] __kmalloc_node_track_caller
0.84% hackbench libc.so.6 [.] read
0.81% hackbench [kernel.vmlinux] [k] __lock_text_start
0.80% hackbench [kernel.vmlinux] [k] cache_from_obj
0.74% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.73% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.72% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.70% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.65% hackbench [kernel.vmlinux] [k] kfree
0.63% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.60% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.58% hackbench [kernel.vmlinux] [k] skb_release_data
0.56% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.56% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.56% hackbench [kernel.vmlinux] [k] __check_heap_object
0.55% hackbench [kernel.vmlinux] [k] sock_wfree
0.54% hackbench [kernel.vmlinux] [k] __audit_syscall_entry
0.53% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
and attached flamegraph-6.1.8.svg.
If you need more information, feel free to ask.
--
Thanks,
Hyeonggon
> --
> Michal Hocko
> SUSE Labs
[-- Attachment #2: flamegraph-6.1.8-debug-preempt-dirty.svg --]
[-- Type: image/svg+xml, Size: 129310 bytes --]
[-- Attachment #3: flamegraph-6.1.8.svg --]
[-- Type: image/svg+xml, Size: 115149 bytes --]
next prev parent reply other threads:[~2023-01-25 6:41 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-21 3:39 [PATCH mm-unstable] lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default Hyeonggon Yoo
2023-01-21 11:29 ` Vlastimil Babka
2023-01-21 11:54 ` Hyeonggon Yoo
2023-01-23 8:58 ` Michal Hocko
2023-01-25 15:41 ` Hyeonggon Yoo [this message]
2023-01-25 9:51 ` Michal Hocko
2023-01-27 11:43 ` Hyeonggon Yoo
2023-01-27 12:33 ` Michal Hocko
2023-01-23 11:05 ` Christoph Lameter
2023-01-24 16:34 ` Hyeonggon Yoo
2023-01-23 14:01 ` Mel Gorman
2023-01-26 2:02 ` Roman Gushchin
2023-01-27 11:45 ` Hyeonggon Yoo
2023-02-02 3:09 ` Davidlohr Bueso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y9FNm50H+w1B26yR@hyeyoo \
--to=42.hyeyoo@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=cl@linux.com \
--cc=dennis@kernel.org \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=iamjoonsoo.kim@lge.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=muchun.song@linux.dev \
--cc=penberg@kernel.org \
--cc=peterz@infradead.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=rostedt@goodmis.org \
--cc=shakeelb@google.com \
--cc=tj@kernel.org \
--cc=vbabka@suse.cz \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).