From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
To: Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux-arch@vger.kernel.org,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Subject: [PATCH 00/34] dynamically allocate split ptl if it cannot be embedded to struct page
Date: Thu, 10 Oct 2013 21:05:25 +0300 [thread overview]
Message-ID: <1381428359-14843-1-git-send-email-kirill.shutemov@linux.intel.com> (raw)
In split page table lock case, we embed spinlock_t into struct page. For
obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or on
-rt kernel. So we disble split page table lock, if spinlock_t is too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access, but
fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
Any comments?
Kirill A. Shutemov (34):
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
cris: fix potential NULL-pointer dereference
m32r: fix potential NULL-pointer dereference
xtensa: fix potential NULL-pointer dereference
mm: allow pgtable_page_ctor() to fail
microblaze: add missing pgtable_page_ctor/dtor calls
mn10300: add missing pgtable_page_ctor/dtor calls
openrisc: add missing pgtable_page_ctor/dtor calls
alpha: handle pgtable_page_ctor() fail
arc: handle pgtable_page_ctor() fail
arm: handle pgtable_page_ctor() fail
arm64: handle pgtable_page_ctor() fail
avr32: handle pgtable_page_ctor() fail
cris: handle pgtable_page_ctor() fail
frv: handle pgtable_page_ctor() fail
hexagon: handle pgtable_page_ctor() fail
ia64: handle pgtable_page_ctor() fail
m32r: handle pgtable_page_ctor() fail
m68k: handle pgtable_page_ctor() fail
metag: handle pgtable_page_ctor() fail
mips: handle pgtable_page_ctor() fail
parisc: handle pgtable_page_ctor() fail
powerpc: handle pgtable_page_ctor() fail
s390: handle pgtable_page_ctor() fail
score: handle pgtable_page_ctor() fail
sh: handle pgtable_page_ctor() fail
sparc: handle pgtable_page_ctor() fail
tile: handle pgtable_page_ctor() fail
um: handle pgtable_page_ctor() fail
unicore32: handle pgtable_page_ctor() fail
x86: handle pgtable_page_ctor() fail
xtensa: handle pgtable_page_ctor() fail
iommu/arm-smmu: handle pgtable_page_ctor() fail
mm: dynamically allocate page->ptl if it cannot be embedded to struct
page
Documentation/vm/split_page_table_lock | 90 ++++++++++++++++++++++++++++++++
arch/alpha/include/asm/pgalloc.h | 5 +-
arch/arc/include/asm/pgalloc.h | 11 ++--
arch/arm/include/asm/pgalloc.h | 12 +++--
arch/arm64/include/asm/pgalloc.h | 9 ++--
arch/avr32/include/asm/pgalloc.h | 5 +-
arch/cris/include/asm/pgalloc.h | 7 ++-
arch/frv/mm/pgalloc.c | 12 +++--
arch/hexagon/include/asm/pgalloc.h | 10 ++--
arch/ia64/include/asm/pgalloc.h | 5 +-
arch/m32r/include/asm/pgalloc.h | 7 ++-
arch/m68k/include/asm/motorola_pgalloc.h | 5 +-
arch/m68k/include/asm/sun3_pgalloc.h | 5 +-
arch/metag/include/asm/pgalloc.h | 8 ++-
arch/microblaze/include/asm/pgalloc.h | 12 +++--
arch/mips/include/asm/pgalloc.h | 9 ++--
arch/mn10300/include/asm/pgalloc.h | 1 +
arch/mn10300/mm/pgtable.c | 9 +++-
arch/openrisc/include/asm/pgalloc.h | 10 +++-
arch/parisc/include/asm/pgalloc.h | 8 ++-
arch/powerpc/include/asm/pgalloc-64.h | 5 +-
arch/powerpc/mm/pgtable_32.c | 5 +-
arch/powerpc/mm/pgtable_64.c | 7 +--
arch/s390/mm/pgtable.c | 11 +++-
arch/score/include/asm/pgalloc.h | 9 ++--
arch/sh/include/asm/pgalloc.h | 5 +-
arch/sparc/mm/init_64.c | 11 ++--
arch/sparc/mm/srmmu.c | 5 +-
arch/tile/mm/pgtable.c | 6 ++-
arch/um/kernel/mem.c | 8 ++-
arch/unicore32/include/asm/pgalloc.h | 14 ++---
arch/x86/mm/pgtable.c | 19 +++++--
arch/x86/xen/mmu.c | 2 +-
arch/xtensa/include/asm/pgalloc.h | 11 +++-
drivers/iommu/arm-smmu.c | 5 +-
include/linux/mm.h | 73 +++++++++++++++++++-------
include/linux/mm_types.h | 5 +-
mm/Kconfig | 2 -
mm/memory.c | 19 +++++++
39 files changed, 365 insertions(+), 97 deletions(-)
create mode 100644 Documentation/vm/split_page_table_lock
--
1.8.4.rc3
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
WARNING: multiple messages have this Message-ID (diff)
From: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
To: Andrew Morton <akpm@linux-foundation.org>,
Peter Zijlstra <peterz@infradead.org>,
Ingo Molnar <mingo@redhat.com>
Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org,
linux-arch@vger.kernel.org,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Subject: [PATCH 00/34] dynamically allocate split ptl if it cannot be embedded to struct page
Date: Thu, 10 Oct 2013 21:05:25 +0300 [thread overview]
Message-ID: <1381428359-14843-1-git-send-email-kirill.shutemov@linux.intel.com> (raw)
Message-ID: <20131010180525.daM6jMAmDf8d2-0R_--a4WHi6MHkAKTXsakMJwCunUs@z> (raw)
In split page table lock case, we embed spinlock_t into struct page. For
obvious reason, we don't want to increase size of struct page if
spinlock_t is too big, like with DEBUG_SPINLOCK or DEBUG_LOCK_ALLOC or on
-rt kernel. So we disble split page table lock, if spinlock_t is too big.
This patchset allows to allocate the lock dynamically if spinlock_t is
big. In this page->ptl is used to store pointer to spinlock instead of
spinlock itself. It costs additional cache line for indirect access, but
fix page fault scalability for multi-threaded applications.
LOCK_STAT depends on DEBUG_SPINLOCK, so on current kernel enabling
LOCK_STAT to analyse scalability issues breaks scalability. ;)
The patchset mostly fixes this. Results for ./thp_memscale -c 80 -b 512M
on 4-socket machine:
baseline, no CONFIG_LOCK_STAT: 9.115460703 seconds time elapsed
baseline, CONFIG_LOCK_STAT=y: 53.890567123 seconds time elapsed
patched, no CONFIG_LOCK_STAT: 8.852250368 seconds time elapsed
patched, CONFIG_LOCK_STAT=y: 11.069770759 seconds time elapsed
Patch count is scary, but most of them trivial. Overview:
Patches 1-4 Few bug fixes. No dependencies to other patches.
Probably should applied as soon as possible.
Patch 5 Changes signature of pgtable_page_ctor(). We will use it
for dynamic lock allocation, so it can fail.
Patches 6-8 Add missing constructor/destructor calls on few archs.
It's fixes NR_PAGETABLE accounting and prepare to use
split ptl.
Patches 9-33 Add pgtable_page_ctor() fail handling to all archs.
Patches 34 Finally adds support of dynamically-allocated page->pte.
Also contains documentation for split page table lock.
Any comments?
Kirill A. Shutemov (34):
x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds
cris: fix potential NULL-pointer dereference
m32r: fix potential NULL-pointer dereference
xtensa: fix potential NULL-pointer dereference
mm: allow pgtable_page_ctor() to fail
microblaze: add missing pgtable_page_ctor/dtor calls
mn10300: add missing pgtable_page_ctor/dtor calls
openrisc: add missing pgtable_page_ctor/dtor calls
alpha: handle pgtable_page_ctor() fail
arc: handle pgtable_page_ctor() fail
arm: handle pgtable_page_ctor() fail
arm64: handle pgtable_page_ctor() fail
avr32: handle pgtable_page_ctor() fail
cris: handle pgtable_page_ctor() fail
frv: handle pgtable_page_ctor() fail
hexagon: handle pgtable_page_ctor() fail
ia64: handle pgtable_page_ctor() fail
m32r: handle pgtable_page_ctor() fail
m68k: handle pgtable_page_ctor() fail
metag: handle pgtable_page_ctor() fail
mips: handle pgtable_page_ctor() fail
parisc: handle pgtable_page_ctor() fail
powerpc: handle pgtable_page_ctor() fail
s390: handle pgtable_page_ctor() fail
score: handle pgtable_page_ctor() fail
sh: handle pgtable_page_ctor() fail
sparc: handle pgtable_page_ctor() fail
tile: handle pgtable_page_ctor() fail
um: handle pgtable_page_ctor() fail
unicore32: handle pgtable_page_ctor() fail
x86: handle pgtable_page_ctor() fail
xtensa: handle pgtable_page_ctor() fail
iommu/arm-smmu: handle pgtable_page_ctor() fail
mm: dynamically allocate page->ptl if it cannot be embedded to struct
page
Documentation/vm/split_page_table_lock | 90 ++++++++++++++++++++++++++++++++
arch/alpha/include/asm/pgalloc.h | 5 +-
arch/arc/include/asm/pgalloc.h | 11 ++--
arch/arm/include/asm/pgalloc.h | 12 +++--
arch/arm64/include/asm/pgalloc.h | 9 ++--
arch/avr32/include/asm/pgalloc.h | 5 +-
arch/cris/include/asm/pgalloc.h | 7 ++-
arch/frv/mm/pgalloc.c | 12 +++--
arch/hexagon/include/asm/pgalloc.h | 10 ++--
arch/ia64/include/asm/pgalloc.h | 5 +-
arch/m32r/include/asm/pgalloc.h | 7 ++-
arch/m68k/include/asm/motorola_pgalloc.h | 5 +-
arch/m68k/include/asm/sun3_pgalloc.h | 5 +-
arch/metag/include/asm/pgalloc.h | 8 ++-
arch/microblaze/include/asm/pgalloc.h | 12 +++--
arch/mips/include/asm/pgalloc.h | 9 ++--
arch/mn10300/include/asm/pgalloc.h | 1 +
arch/mn10300/mm/pgtable.c | 9 +++-
arch/openrisc/include/asm/pgalloc.h | 10 +++-
arch/parisc/include/asm/pgalloc.h | 8 ++-
arch/powerpc/include/asm/pgalloc-64.h | 5 +-
arch/powerpc/mm/pgtable_32.c | 5 +-
arch/powerpc/mm/pgtable_64.c | 7 +--
arch/s390/mm/pgtable.c | 11 +++-
arch/score/include/asm/pgalloc.h | 9 ++--
arch/sh/include/asm/pgalloc.h | 5 +-
arch/sparc/mm/init_64.c | 11 ++--
arch/sparc/mm/srmmu.c | 5 +-
arch/tile/mm/pgtable.c | 6 ++-
arch/um/kernel/mem.c | 8 ++-
arch/unicore32/include/asm/pgalloc.h | 14 ++---
arch/x86/mm/pgtable.c | 19 +++++--
arch/x86/xen/mmu.c | 2 +-
arch/xtensa/include/asm/pgalloc.h | 11 +++-
drivers/iommu/arm-smmu.c | 5 +-
include/linux/mm.h | 73 +++++++++++++++++++-------
include/linux/mm_types.h | 5 +-
mm/Kconfig | 2 -
mm/memory.c | 19 +++++++
39 files changed, 365 insertions(+), 97 deletions(-)
create mode 100644 Documentation/vm/split_page_table_lock
--
1.8.4.rc3
next reply other threads:[~2013-10-10 18:05 UTC|newest]
Thread overview: 117+ messages / expand[flat|nested] mbox.gz Atom feed top
2013-10-10 18:05 Kirill A. Shutemov [this message]
2013-10-10 18:05 ` [PATCH 00/34] dynamically allocate split ptl if it cannot be embedded to struct page Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 01/34] x86: add missed pgtable_pmd_page_ctor/dtor calls for preallocated pmds Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-11 5:50 ` Ingo Molnar
2013-10-11 5:50 ` Ingo Molnar
2013-10-10 18:05 ` [PATCH 02/34] cris: fix potential NULL-pointer dereference Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-11 9:58 ` Jesper Nilsson
2013-10-11 9:58 ` Jesper Nilsson
2013-10-10 18:05 ` [PATCH 03/34] m32r: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 04/34] xtensa: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 05/34] mm: allow pgtable_page_ctor() to fail Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 06/34] microblaze: add missing pgtable_page_ctor/dtor calls Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 07/34] mn10300: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 08/34] openrisc: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 09/34] alpha: handle pgtable_page_ctor() fail Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 10/34] arc: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-16 7:55 ` Vineet Gupta
2013-10-10 18:05 ` [PATCH 11/34] arm: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 20:18 ` Russell King - ARM Linux
2013-10-10 21:53 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 12/34] arm64: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 13/34] avr32: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 19:14 ` Hans-Christian Egtvedt
2013-10-10 19:14 ` Hans-Christian Egtvedt
2013-10-10 18:05 ` [PATCH 14/34] cris: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-11 9:57 ` Jesper Nilsson
2013-10-11 9:57 ` Jesper Nilsson
2013-10-10 18:05 ` [PATCH 15/34] frv: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 16/34] hexagon: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 17/34] ia64: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 18/34] m32r: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 19/34] m68k: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-16 8:07 ` Geert Uytterhoeven
2013-10-16 8:07 ` Geert Uytterhoeven
2013-10-16 9:56 ` Kirill A. Shutemov
2013-10-16 9:56 ` Kirill A. Shutemov
2013-10-16 10:29 ` Kirill A. Shutemov
2013-10-16 10:29 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 20/34] metag: " Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 21/34] mips: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 22/34] parisc: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 23/34] powerpc: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 24/34] s390: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 25/34] score: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-13 10:07 ` Lennox Wu
2013-10-10 18:05 ` [PATCH 26/34] sh: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 27/34] sparc: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:33 ` David Miller
2013-10-10 18:05 ` [PATCH 28/34] tile: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 19:50 ` Chris Metcalf
2013-10-10 19:50 ` Chris Metcalf
2013-10-10 18:05 ` [PATCH 29/34] um: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 30/34] unicore32: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 31/34] x86: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 32/34] xtensa: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 18:05 ` [PATCH 33/34] iommu/arm-smmu: " Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-11 10:26 ` Will Deacon
2013-10-11 10:26 ` Will Deacon
2013-10-10 18:05 ` [PATCH 34/34] mm: dynamically allocate page->ptl if it cannot be embedded to struct page Kirill A. Shutemov
2013-10-10 18:05 ` Kirill A. Shutemov
2013-10-10 20:00 ` Christoph Lameter
2013-10-10 20:09 ` Kirill A. Shutemov
2013-10-10 20:09 ` Kirill A. Shutemov
2013-10-11 14:01 ` Christoph Lameter
2013-10-11 14:01 ` Christoph Lameter
2013-10-14 9:04 ` Kirill A. Shutemov
2013-10-14 19:04 ` Christoph Lameter
2013-10-14 19:04 ` Christoph Lameter
2013-10-14 21:25 ` Kirill A. Shutemov
2013-10-14 22:13 ` Christoph Lameter
2013-10-14 22:13 ` Christoph Lameter
2013-10-14 22:26 ` Kirill A. Shutemov
2013-11-20 8:44 ` Andrey Wagin
2013-11-20 8:44 ` Andrey Wagin
2013-11-20 12:53 ` Kirill A. Shutemov
2013-11-20 12:56 ` Andrey Wagin
2013-11-20 12:56 ` Andrey Wagin
2013-11-20 13:06 ` Kirill A. Shutemov
2013-11-20 13:06 ` Kirill A. Shutemov
2013-11-20 13:29 ` Andrey Wagin
2013-10-10 19:47 ` [PATCH 00/34] dynamically allocate split ptl " Peter Zijlstra
2013-10-11 16:30 ` [PATCH 15/34] frv: handle pgtable_page_ctor() fail David Howells
2013-10-11 16:30 ` David Howells
2013-10-14 9:12 ` Kirill A. Shutemov
2013-10-14 9:12 ` Kirill A. Shutemov
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1381428359-14843-1-git-send-email-kirill.shutemov@linux.intel.com \
--to=kirill.shutemov@linux.intel.com \
--cc=akpm@linux-foundation.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mingo@redhat.com \
--cc=peterz@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).