From: Hyeonggon Yoo <42.hyeyoo@gmail.com>
To: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>,
Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Ingo Molnar <mingo@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Shakeel Butt <shakeelb@google.com>,
Muchun Song <muchun.song@linux.dev>,
Matthew Wilcox <willy@infradead.org>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org,
Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>,
Mel Gorman <mgorman@techsingularity.net>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Dennis Zhou <dennis@kernel.org>, Tejun Heo <tj@kernel.org>
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default
Date: Thu, 26 Jan 2023 00:41:15 +0900 [thread overview]
Message-ID: <Y9FNm50H+w1B26yR@hyeyoo> (raw)
In-Reply-To: <Y85MNmZDc5czMRUJ@dhcp22.suse.cz>
[-- Attachment #1: Type: text/plain, Size: 10317 bytes --]
From: Hyeonggon Yoo <42.hyeyoo@gmail.com>
To: Michal Hocko <mhocko@suse.com>
Cc: Vlastimil Babka <vbabka@suse.cz>,
Andrew Morton <akpm@linux-foundation.org>,
Christoph Lameter <cl@linux.com>, Pekka Enberg <penberg@kernel.org>,
David Rientjes <rientjes@google.com>,
Joonsoo Kim <iamjoonsoo.kim@lge.com>,
Roman Gushchin <roman.gushchin@linux.dev>,
Ingo Molnar <mingo@redhat.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Shakeel Butt <shakeelb@google.com>,
Muchun Song <muchun.song@linux.dev>,
Matthew Wilcox <willy@infradead.org>, linux-mm@kvack.org,
linux-kernel@vger.kernel.org, Peter Zijlstra <peterz@infradead.org>,
Juri Lelli <juri.lelli@redhat.com>,
Vincent Guittot <vincent.guittot@linaro.org>,
Dietmar Eggemann <dietmar.eggemann@arm.com>,
Steven Rostedt <rostedt@goodmis.org>,
Ben Segall <bsegall@google.com>,
Mel Gorman <mgorman@techsingularity.net>,
Daniel Bristot de Oliveira <bristot@redhat.com>,
Valentin Schneider <vschneid@redhat.com>,
Dennis Zhou <dennis@kernel.org>, Tejun Heo <tj@kernel.org>
Bcc:
Subject: Re: [PATCH mm-unstable] lib/Kconfig.debug: do not enable
DEBUG_PREEMPT by default
Reply-To:
In-Reply-To: <Y85MNmZDc5czMRUJ@dhcp22.suse.cz>
On Mon, Jan 23, 2023 at 09:58:30AM +0100, Michal Hocko wrote:
> On Sat 21-01-23 20:54:15, Hyeonggon Yoo wrote:
> > On Sat, Jan 21, 2023 at 12:29:44PM +0100, Vlastimil Babka wrote:
> > > On 1/21/23 04:39, Hyeonggon Yoo wrote:
> > > > In workloads where this_cpu operations are frequently performed,
> > > > enabling DEBUG_PREEMPT may result in significant increase in
> > > > runtime overhead due to frequent invocation of
> > > > __this_cpu_preempt_check() function.
> > > >
> > > > This can be demonstrated through benchmarks such as hackbench where this
> > > > configuration results in a 10% reduction in performance, primarily due to
> > > > the added overhead within memcg charging path.
> > > >
> > > > Therefore, do not to enable DEBUG_PREEMPT by default and make users aware
> > > > of its potential impact on performance in some workloads.
> > > >
> > > > hackbench-process-sockets
> > > > debug_preempt no_debug_preempt
> > > > Amean 1 0.4743 ( 0.00%) 0.4295 * 9.45%*
> > > > Amean 4 1.4191 ( 0.00%) 1.2650 * 10.86%*
> > > > Amean 7 2.2677 ( 0.00%) 2.0094 * 11.39%*
> > > > Amean 12 3.6821 ( 0.00%) 3.2115 * 12.78%*
> > > > Amean 21 6.6752 ( 0.00%) 5.7956 * 13.18%*
> > > > Amean 30 9.6646 ( 0.00%) 8.5197 * 11.85%*
> > > > Amean 48 15.3363 ( 0.00%) 13.5559 * 11.61%*
> > > > Amean 79 24.8603 ( 0.00%) 22.0597 * 11.27%*
> > > > Amean 96 30.1240 ( 0.00%) 26.8073 * 11.01%*
Hello Michal, thanks for looking at this.
> Do you happen to have any perf data collected during those runs? I
> would be interested in the memcg side of things. Maybe we can do
> something better there.
Yes, below is performance data I've collected.
6.1.8-debug-preempt-dirty
=========================
Overhead Command Shared Object Symbol
+ 9.14% hackbench [kernel.vmlinux] [k] check_preemption_disabled
+ 7.33% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
+ 7.32% hackbench [kernel.vmlinux] [k] mod_objcg_state
3.55% hackbench [kernel.vmlinux] [k] refill_obj_stock
3.39% hackbench [kernel.vmlinux] [k] debug_smp_processor_id
2.97% hackbench [kernel.vmlinux] [k] memset_erms
2.55% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.36% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.76% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.64% hackbench [kernel.vmlinux] [k] __slab_free
1.58% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.35% hackbench [kernel.vmlinux] [k] vfs_write
1.33% hackbench [kernel.vmlinux] [k] vfs_read
1.28% hackbench [kernel.vmlinux] [k] __alloc_skb
1.18% hackbench [kernel.vmlinux] [k] sock_read_iter
1.16% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
1.16% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64
1.14% hackbench [kernel.vmlinux] [k] sock_write_iter
1.12% hackbench [kernel.vmlinux] [k] skb_release_data
1.08% hackbench [kernel.vmlinux] [k] sock_wfree
1.07% hackbench [kernel.vmlinux] [k] cache_from_obj
0.96% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.95% hackbench [kernel.vmlinux] [k] kmem_cache_free
0.94% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.92% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
0.89% hackbench [kernel.vmlinux] [k] _raw_spin_lock_irqsave
0.84% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_array
0.84% hackbench libc.so.6 [.] write
0.81% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.76% hackbench libc.so.6 [.] read
0.75% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.75% hackbench [kernel.vmlinux] [k] preempt_count_add
0.74% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.69% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
0.65% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.63% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.62% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.60% hackbench [kernel.vmlinux] [k] __rcu_read_lock
0.59% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.54% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.51% hackbench [kernel.vmlinux] [k] security_file_permission
0.50% hackbench [kernel.vmlinux] [k] __x64_sys_write
0.49% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
0.48% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.46% hackbench [kernel.vmlinux] [k] __check_heap_object
and attached flamegraph-6.1.8-debug-preempt-dirty.svg.
6.1.8 (no debug preempt)
========================
Overhead Command Shared Object Symbol
+ 10.96% hackbench [kernel.vmlinux] [k] mod_objcg_state
+ 8.16% hackbench [kernel.vmlinux] [k] copy_user_enhanced_fast_string
3.29% hackbench [kernel.vmlinux] [k] memset_erms
3.07% hackbench [kernel.vmlinux] [k] __slab_free
2.89% hackbench [kernel.vmlinux] [k] refill_obj_stock
2.82% hackbench [kernel.vmlinux] [k] __check_object_size
+ 2.72% hackbench [kernel.vmlinux] [k] native_queued_spin_lock_slowpath
1.96% hackbench [kernel.vmlinux] [k] __x86_indirect_thunk_rax
1.88% hackbench [kernel.vmlinux] [k] memcg_slab_post_alloc_hook
1.69% hackbench [kernel.vmlinux] [k] __rcu_read_unlock
1.54% hackbench [kernel.vmlinux] [k] __alloc_skb
1.53% hackbench [kernel.vmlinux] [k] unix_stream_sendmsg
1.46% hackbench [kernel.vmlinux] [k] kmem_cache_free
1.44% hackbench [kernel.vmlinux] [k] vfs_write
1.43% hackbench [kernel.vmlinux] [k] vfs_read
1.33% hackbench [kernel.vmlinux] [k] unix_stream_read_generic
1.31% hackbench [kernel.vmlinux] [k] sock_write_iter
1.27% hackbench [kernel.vmlinux] [k] kmalloc_slab
1.22% hackbench [kernel.vmlinux] [k] __rcu_read_lock
1.20% hackbench [kernel.vmlinux] [k] sock_read_iter
1.18% hackbench [kernel.vmlinux] [k] __entry_text_start
1.15% hackbench [kernel.vmlinux] [k] kmem_cache_alloc_node
1.12% hackbench [kernel.vmlinux] [k] unix_stream_recvmsg
1.10% hackbench [kernel.vmlinux] [k] obj_cgroup_charge
0.98% hackbench [kernel.vmlinux] [k] __kmem_cache_alloc_node
0.97% hackbench libc.so.6 [.] write
0.91% hackbench [kernel.vmlinux] [k] exit_to_user_mode_prepare
0.88% hackbench [kernel.vmlinux] [k] __kmem_cache_free
0.87% hackbench [kernel.vmlinux] [k] syscall_trace_enter.constprop.0
0.86% hackbench [kernel.vmlinux] [k] __kmalloc_node_track_caller
0.84% hackbench libc.so.6 [.] read
0.81% hackbench [kernel.vmlinux] [k] __lock_text_start
0.80% hackbench [kernel.vmlinux] [k] cache_from_obj
0.74% hackbench [kernel.vmlinux] [k] get_obj_cgroup_from_current
0.73% hackbench [kernel.vmlinux] [k] entry_SYSRETQ_unsafe_stack
0.72% hackbench [kernel.vmlinux] [k] unix_destruct_scm
0.70% hackbench [kernel.vmlinux] [k] get_partial_node
0.69% hackbench [kernel.vmlinux] [k] syscall_exit_to_user_mode_prepare
0.65% hackbench [kernel.vmlinux] [k] kfree
0.63% hackbench [kernel.vmlinux] [k] __unfreeze_partials
0.60% hackbench [kernel.vmlinux] [k] cmpxchg_double_slab.constprop.0.isra.0
0.58% hackbench [kernel.vmlinux] [k] skb_release_data
0.56% hackbench [kernel.vmlinux] [k] __virt_addr_valid
0.56% hackbench [kernel.vmlinux] [k] entry_SYSCALL_64_after_hwframe
0.56% hackbench [kernel.vmlinux] [k] __check_heap_object
0.55% hackbench [kernel.vmlinux] [k] sock_wfree
0.54% hackbench [kernel.vmlinux] [k] __audit_syscall_entry
0.53% hackbench [kernel.vmlinux] [k] ___slab_alloc
0.53% hackbench [kernel.vmlinux] [k] check_stack_object
0.52% hackbench [kernel.vmlinux] [k] bpf_lsm_file_permission
and attached flamegraph-6.1.8.svg.
If you need more information, feel free to ask.
--
Thanks,
Hyeonggon
> --
> Michal Hocko
> SUSE Labs
[-- Attachment #2: flamegraph-6.1.8-debug-preempt-dirty.svg --]
[-- Type: image/svg+xml, Size: 129310 bytes --]
[-- Attachment #3: flamegraph-6.1.8.svg --]
[-- Type: image/svg+xml, Size: 115149 bytes --]
next prev parent reply other threads:[~2023-01-25 6:41 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-01-21 3:39 [PATCH mm-unstable] lib/Kconfig.debug: do not enable DEBUG_PREEMPT by default Hyeonggon Yoo
2023-01-21 11:29 ` Vlastimil Babka
2023-01-21 11:54 ` Hyeonggon Yoo
2023-01-23 8:58 ` Michal Hocko
2023-01-25 15:41 ` Hyeonggon Yoo [this message]
2023-01-25 9:51 ` Michal Hocko
2023-01-27 11:43 ` Hyeonggon Yoo
2023-01-27 12:33 ` Michal Hocko
2023-01-23 11:05 ` Christoph Lameter
2023-01-24 16:34 ` Hyeonggon Yoo
2023-01-23 14:01 ` Mel Gorman
2023-01-26 2:02 ` Roman Gushchin
2023-01-27 11:45 ` Hyeonggon Yoo
2023-02-02 3:09 ` Davidlohr Bueso
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=Y9FNm50H+w1B26yR@hyeyoo \
--to=42.hyeyoo@gmail.com \
--cc=akpm@linux-foundation.org \
--cc=bristot@redhat.com \
--cc=bsegall@google.com \
--cc=cl@linux.com \
--cc=dennis@kernel.org \
--cc=dietmar.eggemann@arm.com \
--cc=hannes@cmpxchg.org \
--cc=iamjoonsoo.kim@lge.com \
--cc=juri.lelli@redhat.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=mingo@redhat.com \
--cc=muchun.song@linux.dev \
--cc=penberg@kernel.org \
--cc=peterz@infradead.org \
--cc=rientjes@google.com \
--cc=roman.gushchin@linux.dev \
--cc=rostedt@goodmis.org \
--cc=shakeelb@google.com \
--cc=tj@kernel.org \
--cc=vbabka@suse.cz \
--cc=vincent.guittot@linaro.org \
--cc=vschneid@redhat.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.