From: Dave Chinner <david@fromorbit.com>
To: Mel Gorman <mgorman@suse.de>
Cc: linuxppc-dev@lists.ozlabs.org,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
xfs@oss.sgi.com, Linux-MM <linux-mm@kvack.org>,
Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Ingo Molnar <mingo@kernel.org>
Subject: Re: [PATCH 2/2] mm: numa: Do not clear PTEs or PMDs for NUMA hinting faults
Date: Fri, 6 Mar 2015 14:59:16 +1100 [thread overview]
Message-ID: <20150306035916.GD4251@dastard> (raw)
In-Reply-To: <1425599692-32445-3-git-send-email-mgorman@suse.de>
On Thu, Mar 05, 2015 at 11:54:52PM +0000, Mel Gorman wrote:
> Dave Chinner reported the following on https://lkml.org/lkml/2015/3/1/226
>
> Across the board the 4.0-rc1 numbers are much slower, and the
> degradation is far worse when using the large memory footprint
> configs. Perf points straight at the cause - this is from 4.0-rc1
> on the "-o bhash=101073" config:
>
> - 56.07% 56.07% [kernel] [k] default_send_IPI_mask_sequence_phys
> - default_send_IPI_mask_sequence_phys
> - 99.99% physflat_send_IPI_mask
> - 99.37% native_send_call_func_ipi
> smp_call_function_many
> - native_flush_tlb_others
> - 99.85% flush_tlb_page
> ptep_clear_flush
> try_to_unmap_one
> rmap_walk
> try_to_unmap
> migrate_pages
> migrate_misplaced_page
> - handle_mm_fault
> - 99.73% __do_page_fault
> trace_do_page_fault
> do_async_page_fault
> + async_page_fault
> 0.63% native_send_call_func_single_ipi
> generic_exec_single
> smp_call_function_single
>
> This was bisected to commit 4d9424669946 ("mm: convert p[te|md]_mknonnuma
> and remaining page table manipulations") which clears PTEs and PMDs to make
> them PROT_NONE. This is tidy but tests on some benchmarks indicate that
> there are many more hinting faults trapped resulting in excessive migration.
> This is the result for the old autonuma benchmark for example.
[snip]
Doesn't fix the problem. Runtime is slightly improved (16m45s vs 17m35)
but it's still much slower that 3.19 (6m5s).
Stats and profiles still roughly the same:
360,228 migrate:mm_migrate_pages ( +- 4.28% )
- 52.69% 52.69% [kernel] [k] default_send_IPI_mask_sequence_phys
default_send_IPI_mask_sequence_phys
- physflat_send_IPI_mask
- 97.28% native_send_call_func_ipi
smp_call_function_many
native_flush_tlb_others
flush_tlb_page
ptep_clear_flush
try_to_unmap_one
rmap_walk
try_to_unmap
migrate_pages
migrate_misplaced_page
- handle_mm_fault
- 99.59% __do_page_fault
trace_do_page_fault
do_async_page_fault
+ async_page_fault
+ 2.72% native_send_call_func_single_ipi
numa_hit 36678767
numa_miss 905234
numa_foreign 905234
numa_interleave 14802
numa_local 36656791
numa_other 927210
numa_pte_updates 92168450
numa_huge_pte_updates 0
numa_hint_faults 87573926
numa_hint_faults_local 29730293
numa_pages_migrated 30195890
pgmigrate_success 30195890
pgmigrate_fail 0
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com>
To: Mel Gorman <mgorman@suse.de>
Cc: linuxppc-dev@lists.ozlabs.org,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
xfs@oss.sgi.com, Linux-MM <linux-mm@kvack.org>,
Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>,
Andrew Morton <akpm@linux-foundation.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Ingo Molnar <mingo@kernel.org>
Subject: Re: [PATCH 2/2] mm: numa: Do not clear PTEs or PMDs for NUMA hinting faults
Date: Fri, 6 Mar 2015 14:59:16 +1100 [thread overview]
Message-ID: <20150306035916.GD4251@dastard> (raw)
In-Reply-To: <1425599692-32445-3-git-send-email-mgorman@suse.de>
On Thu, Mar 05, 2015 at 11:54:52PM +0000, Mel Gorman wrote:
> Dave Chinner reported the following on https://lkml.org/lkml/2015/3/1/226
>
> Across the board the 4.0-rc1 numbers are much slower, and the
> degradation is far worse when using the large memory footprint
> configs. Perf points straight at the cause - this is from 4.0-rc1
> on the "-o bhash=101073" config:
>
> - 56.07% 56.07% [kernel] [k] default_send_IPI_mask_sequence_phys
> - default_send_IPI_mask_sequence_phys
> - 99.99% physflat_send_IPI_mask
> - 99.37% native_send_call_func_ipi
> smp_call_function_many
> - native_flush_tlb_others
> - 99.85% flush_tlb_page
> ptep_clear_flush
> try_to_unmap_one
> rmap_walk
> try_to_unmap
> migrate_pages
> migrate_misplaced_page
> - handle_mm_fault
> - 99.73% __do_page_fault
> trace_do_page_fault
> do_async_page_fault
> + async_page_fault
> 0.63% native_send_call_func_single_ipi
> generic_exec_single
> smp_call_function_single
>
> This was bisected to commit 4d9424669946 ("mm: convert p[te|md]_mknonnuma
> and remaining page table manipulations") which clears PTEs and PMDs to make
> them PROT_NONE. This is tidy but tests on some benchmarks indicate that
> there are many more hinting faults trapped resulting in excessive migration.
> This is the result for the old autonuma benchmark for example.
[snip]
Doesn't fix the problem. Runtime is slightly improved (16m45s vs 17m35)
but it's still much slower that 3.19 (6m5s).
Stats and profiles still roughly the same:
360,228 migrate:mm_migrate_pages ( +- 4.28% )
- 52.69% 52.69% [kernel] [k] default_send_IPI_mask_sequence_phys
default_send_IPI_mask_sequence_phys
- physflat_send_IPI_mask
- 97.28% native_send_call_func_ipi
smp_call_function_many
native_flush_tlb_others
flush_tlb_page
ptep_clear_flush
try_to_unmap_one
rmap_walk
try_to_unmap
migrate_pages
migrate_misplaced_page
- handle_mm_fault
- 99.59% __do_page_fault
trace_do_page_fault
do_async_page_fault
+ async_page_fault
+ 2.72% native_send_call_func_single_ipi
numa_hit 36678767
numa_miss 905234
numa_foreign 905234
numa_interleave 14802
numa_local 36656791
numa_other 927210
numa_pte_updates 92168450
numa_huge_pte_updates 0
numa_hint_faults 87573926
numa_hint_faults_local 29730293
numa_pages_migrated 30195890
pgmigrate_success 30195890
pgmigrate_fail 0
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com>
To: Mel Gorman <mgorman@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Ingo Molnar <mingo@kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>,
xfs@oss.sgi.com, linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH 2/2] mm: numa: Do not clear PTEs or PMDs for NUMA hinting faults
Date: Fri, 6 Mar 2015 14:59:16 +1100 [thread overview]
Message-ID: <20150306035916.GD4251@dastard> (raw)
In-Reply-To: <1425599692-32445-3-git-send-email-mgorman@suse.de>
On Thu, Mar 05, 2015 at 11:54:52PM +0000, Mel Gorman wrote:
> Dave Chinner reported the following on https://lkml.org/lkml/2015/3/1/226
>
> Across the board the 4.0-rc1 numbers are much slower, and the
> degradation is far worse when using the large memory footprint
> configs. Perf points straight at the cause - this is from 4.0-rc1
> on the "-o bhash=101073" config:
>
> - 56.07% 56.07% [kernel] [k] default_send_IPI_mask_sequence_phys
> - default_send_IPI_mask_sequence_phys
> - 99.99% physflat_send_IPI_mask
> - 99.37% native_send_call_func_ipi
> smp_call_function_many
> - native_flush_tlb_others
> - 99.85% flush_tlb_page
> ptep_clear_flush
> try_to_unmap_one
> rmap_walk
> try_to_unmap
> migrate_pages
> migrate_misplaced_page
> - handle_mm_fault
> - 99.73% __do_page_fault
> trace_do_page_fault
> do_async_page_fault
> + async_page_fault
> 0.63% native_send_call_func_single_ipi
> generic_exec_single
> smp_call_function_single
>
> This was bisected to commit 4d9424669946 ("mm: convert p[te|md]_mknonnuma
> and remaining page table manipulations") which clears PTEs and PMDs to make
> them PROT_NONE. This is tidy but tests on some benchmarks indicate that
> there are many more hinting faults trapped resulting in excessive migration.
> This is the result for the old autonuma benchmark for example.
[snip]
Doesn't fix the problem. Runtime is slightly improved (16m45s vs 17m35)
but it's still much slower that 3.19 (6m5s).
Stats and profiles still roughly the same:
360,228 migrate:mm_migrate_pages ( +- 4.28% )
- 52.69% 52.69% [kernel] [k] default_send_IPI_mask_sequence_phys
default_send_IPI_mask_sequence_phys
- physflat_send_IPI_mask
- 97.28% native_send_call_func_ipi
smp_call_function_many
native_flush_tlb_others
flush_tlb_page
ptep_clear_flush
try_to_unmap_one
rmap_walk
try_to_unmap
migrate_pages
migrate_misplaced_page
- handle_mm_fault
- 99.59% __do_page_fault
trace_do_page_fault
do_async_page_fault
+ async_page_fault
+ 2.72% native_send_call_func_single_ipi
numa_hit 36678767
numa_miss 905234
numa_foreign 905234
numa_interleave 14802
numa_local 36656791
numa_other 927210
numa_pte_updates 92168450
numa_huge_pte_updates 0
numa_hint_faults 87573926
numa_hint_faults_local 29730293
numa_pages_migrated 30195890
pgmigrate_success 30195890
pgmigrate_fail 0
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com>
To: Mel Gorman <mgorman@suse.de>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Ingo Molnar <mingo@kernel.org>,
Linus Torvalds <torvalds@linux-foundation.org>,
Aneesh Kumar <aneesh.kumar@linux.vnet.ibm.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Linux-MM <linux-mm@kvack.org>,
xfs@oss.sgi.com, linuxppc-dev@lists.ozlabs.org
Subject: Re: [PATCH 2/2] mm: numa: Do not clear PTEs or PMDs for NUMA hinting faults
Date: Fri, 6 Mar 2015 14:59:16 +1100 [thread overview]
Message-ID: <20150306035916.GD4251@dastard> (raw)
In-Reply-To: <1425599692-32445-3-git-send-email-mgorman@suse.de>
On Thu, Mar 05, 2015 at 11:54:52PM +0000, Mel Gorman wrote:
> Dave Chinner reported the following on https://lkml.org/lkml/2015/3/1/226
>
> Across the board the 4.0-rc1 numbers are much slower, and the
> degradation is far worse when using the large memory footprint
> configs. Perf points straight at the cause - this is from 4.0-rc1
> on the "-o bhash=101073" config:
>
> - 56.07% 56.07% [kernel] [k] default_send_IPI_mask_sequence_phys
> - default_send_IPI_mask_sequence_phys
> - 99.99% physflat_send_IPI_mask
> - 99.37% native_send_call_func_ipi
> smp_call_function_many
> - native_flush_tlb_others
> - 99.85% flush_tlb_page
> ptep_clear_flush
> try_to_unmap_one
> rmap_walk
> try_to_unmap
> migrate_pages
> migrate_misplaced_page
> - handle_mm_fault
> - 99.73% __do_page_fault
> trace_do_page_fault
> do_async_page_fault
> + async_page_fault
> 0.63% native_send_call_func_single_ipi
> generic_exec_single
> smp_call_function_single
>
> This was bisected to commit 4d9424669946 ("mm: convert p[te|md]_mknonnuma
> and remaining page table manipulations") which clears PTEs and PMDs to make
> them PROT_NONE. This is tidy but tests on some benchmarks indicate that
> there are many more hinting faults trapped resulting in excessive migration.
> This is the result for the old autonuma benchmark for example.
[snip]
Doesn't fix the problem. Runtime is slightly improved (16m45s vs 17m35)
but it's still much slower that 3.19 (6m5s).
Stats and profiles still roughly the same:
360,228 migrate:mm_migrate_pages ( +- 4.28% )
- 52.69% 52.69% [kernel] [k] default_send_IPI_mask_sequence_phys
default_send_IPI_mask_sequence_phys
- physflat_send_IPI_mask
- 97.28% native_send_call_func_ipi
smp_call_function_many
native_flush_tlb_others
flush_tlb_page
ptep_clear_flush
try_to_unmap_one
rmap_walk
try_to_unmap
migrate_pages
migrate_misplaced_page
- handle_mm_fault
- 99.59% __do_page_fault
trace_do_page_fault
do_async_page_fault
+ async_page_fault
+ 2.72% native_send_call_func_single_ipi
numa_hit 36678767
numa_miss 905234
numa_foreign 905234
numa_interleave 14802
numa_local 36656791
numa_other 927210
numa_pte_updates 92168450
numa_huge_pte_updates 0
numa_hint_faults 87573926
numa_hint_faults_local 29730293
numa_pages_migrated 30195890
pgmigrate_success 30195890
pgmigrate_fail 0
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
next prev parent reply other threads:[~2015-03-06 3:59 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-05 23:54 [RFC PATCH 0/2] Automatic NUMA balancing and PROT_NONE handling followup Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` [PATCH 1/2] mm: thp: Return the correct value for change_huge_pmd Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` [PATCH 2/2] mm: numa: Do not clear PTEs or PMDs for NUMA hinting faults Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-05 23:54 ` Mel Gorman
2015-03-06 0:53 ` Linus Torvalds
2015-03-06 0:53 ` Linus Torvalds
2015-03-06 0:53 ` Linus Torvalds
2015-03-06 0:53 ` Linus Torvalds
2015-03-06 3:59 ` Dave Chinner [this message]
2015-03-06 3:59 ` Dave Chinner
2015-03-06 3:59 ` Dave Chinner
2015-03-06 3:59 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150306035916.GD4251@dastard \
--to=david@fromorbit.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.vnet.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linuxppc-dev@lists.ozlabs.org \
--cc=mgorman@suse.de \
--cc=mingo@kernel.org \
--cc=torvalds@linux-foundation.org \
--cc=xfs@oss.sgi.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.