* [PATCH 1/7] percpu: use dynamic percpu allocator as the default percpu allocator
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 7:37 ` [PATCH 2/7] linker script: throw away .discard section Tejun Heo
` (8 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo, Russell King, Matthew Wilcox, Grant Grundler
This patch makes most !CONFIG_HAVE_SETUP_PER_CPU_AREA archs use
dynamic percpu allocator. The first chunk is allocated using
embedding helper and 8k is reserved for modules. This ensures that
the new allocator behaves almost identically to the original allocator
as long as static percpu variables are concerned, so it shouldn't
introduce much breakage.
s390 and alpha use custom SHIFT_PERCPU_PTR() to work around addressing
range limit the addressing model imposes. Unfortunately, this breaks
if the address is specified using a variable, so for now, the two
archs aren't converted.
The following architectures are affected by this change.
* sh
* arm
* cris
* mips
* sparc(32)
* blackfin
* avr32
* parisc
* m32r
* powerpc(32)
As this change makes the dynamic allocator the default one,
CONFIG_HAVE_DYNAMIC_PER_CPU_AREA is replaced with its invert -
CONFIG_HAVE_LEGACY_PER_CPU_AREA, which is added to yet-to-be converted
archs. These archs implement their own setup_per_cpu_areas() and the
conversion is not trivial.
* powerpc(64)
* sparc(64)
* ia64
* alpha
* s390
Boot and batch alloc/free tests on x86_32 with debug code (x86_32
doesn't use default first chunk initialization). Compile tested on
sparc(32), powerpc(32), arm and alpha.
[ Impact: use dynamic allocator for most archs w/o custom percpu setup ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Acked-by: Rusty Russell <rusty@rustcorp.com.au>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Reviewed-by: Christoph Lameter <cl@linux.com>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Russell King <rmk@arm.linux.org.uk>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Bryan Wu <cooloney@kernel.org>
Cc: Kyle McMartin <kyle@mcmartin.ca>
Cc: Matthew Wilcox <matthew@wil.cx>
Cc: Grant Grundler <grundler@parisc-linux.org>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Ingo Molnar <mingo@elte.hu>
---
arch/alpha/Kconfig | 3 +++
arch/ia64/Kconfig | 3 +++
arch/powerpc/Kconfig | 3 +++
arch/s390/Kconfig | 3 +++
arch/sparc/Kconfig | 3 +++
arch/x86/Kconfig | 3 ---
include/linux/percpu.h | 12 +++++++++---
init/main.c | 24 ------------------------
kernel/module.c | 6 +++---
mm/Makefile | 2 +-
mm/allocpercpu.c | 28 ++++++++++++++++++++++++++++
mm/percpu.c | 40 +++++++++++++++++++++++++++++++++++++++-
12 files changed, 95 insertions(+), 35 deletions(-)
diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index 9fb8aae..05d8640 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -70,6 +70,9 @@ config AUTO_IRQ_AFFINITY
depends on SMP
default y
+config HAVE_LEGACY_PER_CPU_AREA
+ def_bool y
+
source "init/Kconfig"
source "kernel/Kconfig.freezer"
diff --git a/arch/ia64/Kconfig b/arch/ia64/Kconfig
index 294a3b1..8e88df7 100644
--- a/arch/ia64/Kconfig
+++ b/arch/ia64/Kconfig
@@ -88,6 +88,9 @@ config GENERIC_TIME_VSYSCALL
bool
default y
+config HAVE_LEGACY_PER_CPU_AREA
+ def_bool y
+
config HAVE_SETUP_PER_CPU_AREA
def_bool y
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index a0d1146..828e60a 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -42,6 +42,9 @@ config GENERIC_HARDIRQS
bool
default y
+config HAVE_LEGACY_PER_CPU_AREA
+ def_bool PPC64
+
config HAVE_SETUP_PER_CPU_AREA
def_bool PPC64
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 2eca5fe..686909a 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -75,6 +75,9 @@ config VIRT_CPU_ACCOUNTING
config ARCH_SUPPORTS_DEBUG_PAGEALLOC
def_bool y
+config HAVE_LEGACY_PER_CPU_AREA
+ def_bool y
+
mainmenu "Linux Kernel Configuration"
config S390
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index cc12cd4..2e7f019 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -90,6 +90,9 @@ config AUDIT_ARCH
bool
default y
+config HAVE_LEGACY_PER_CPU_AREA
+ def_bool y if SPARC64
+
config HAVE_SETUP_PER_CPU_AREA
def_bool y if SPARC64
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index a6efe0a..52b17c2 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -141,9 +141,6 @@ config ARCH_HAS_CACHE_LINE_SIZE
config HAVE_SETUP_PER_CPU_AREA
def_bool y
-config HAVE_DYNAMIC_PER_CPU_AREA
- def_bool y
-
config HAVE_CPUMASK_OF_CPU_MAP
def_bool X86_64_SMP
diff --git a/include/linux/percpu.h b/include/linux/percpu.h
index 1581ff2..bbe5b2c 100644
--- a/include/linux/percpu.h
+++ b/include/linux/percpu.h
@@ -34,7 +34,7 @@
#ifdef CONFIG_SMP
-#ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
+#ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA
/* minimum unit size, also is the maximum supported allocation size */
#define PCPU_MIN_UNIT_SIZE PFN_ALIGN(64 << 10)
@@ -80,7 +80,7 @@ extern ssize_t __init pcpu_embed_first_chunk(
extern void *__alloc_reserved_percpu(size_t size, size_t align);
-#else /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
+#else /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */
struct percpu_data {
void *ptrs[1];
@@ -94,11 +94,15 @@ struct percpu_data {
(__typeof__(ptr))__p->ptrs[(cpu)]; \
})
-#endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
+#endif /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */
extern void *__alloc_percpu(size_t size, size_t align);
extern void free_percpu(void *__pdata);
+#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
+extern void __init setup_per_cpu_areas(void);
+#endif
+
#else /* CONFIG_SMP */
#define per_cpu_ptr(ptr, cpu) ({ (void)(cpu); (ptr); })
@@ -119,6 +123,8 @@ static inline void free_percpu(void *p)
kfree(p);
}
+static inline void __init setup_per_cpu_areas(void) { }
+
#endif /* CONFIG_SMP */
#define alloc_percpu(type) (type *)__alloc_percpu(sizeof(type), \
diff --git a/init/main.c b/init/main.c
index 3bbf93b..18d2dea 100644
--- a/init/main.c
+++ b/init/main.c
@@ -355,7 +355,6 @@ static void __init smp_init(void)
#define smp_init() do { } while (0)
#endif
-static inline void setup_per_cpu_areas(void) { }
static inline void setup_nr_cpu_ids(void) { }
static inline void smp_prepare_cpus(unsigned int maxcpus) { }
@@ -376,29 +375,6 @@ static void __init setup_nr_cpu_ids(void)
nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
}
-#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
-unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
-
-EXPORT_SYMBOL(__per_cpu_offset);
-
-static void __init setup_per_cpu_areas(void)
-{
- unsigned long size, i;
- char *ptr;
- unsigned long nr_possible_cpus = num_possible_cpus();
-
- /* Copy section for each CPU (we discard the original) */
- size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
- ptr = alloc_bootmem_pages(size * nr_possible_cpus);
-
- for_each_possible_cpu(i) {
- __per_cpu_offset[i] = ptr - __per_cpu_start;
- memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
- ptr += size;
- }
-}
-#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
-
/* Called by boot processor to activate the rest. */
static void __init smp_init(void)
{
diff --git a/kernel/module.c b/kernel/module.c
index e797812..1559bd0 100644
--- a/kernel/module.c
+++ b/kernel/module.c
@@ -359,7 +359,7 @@ EXPORT_SYMBOL_GPL(find_module);
#ifdef CONFIG_SMP
-#ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
+#ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA
static void *percpu_modalloc(unsigned long size, unsigned long align,
const char *name)
@@ -384,7 +384,7 @@ static void percpu_modfree(void *freeme)
free_percpu(freeme);
}
-#else /* ... !CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
+#else /* ... CONFIG_HAVE_LEGACY_PER_CPU_AREA */
/* Number of blocks used and allocated. */
static unsigned int pcpu_num_used, pcpu_num_allocated;
@@ -519,7 +519,7 @@ static int percpu_modinit(void)
}
__initcall(percpu_modinit);
-#endif /* CONFIG_HAVE_DYNAMIC_PER_CPU_AREA */
+#endif /* CONFIG_HAVE_LEGACY_PER_CPU_AREA */
static unsigned int find_pcpusec(Elf_Ehdr *hdr,
Elf_Shdr *sechdrs,
diff --git a/mm/Makefile b/mm/Makefile
index ec73c68..67838bd 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -31,7 +31,7 @@ obj-$(CONFIG_FAILSLAB) += failslab.o
obj-$(CONFIG_MEMORY_HOTPLUG) += memory_hotplug.o
obj-$(CONFIG_FS_XIP) += filemap_xip.o
obj-$(CONFIG_MIGRATION) += migrate.o
-ifdef CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
+ifndef CONFIG_HAVE_LEGACY_PER_CPU_AREA
obj-$(CONFIG_SMP) += percpu.o
else
obj-$(CONFIG_SMP) += allocpercpu.o
diff --git a/mm/allocpercpu.c b/mm/allocpercpu.c
index dfdee6a..df34cea 100644
--- a/mm/allocpercpu.c
+++ b/mm/allocpercpu.c
@@ -5,6 +5,8 @@
*/
#include <linux/mm.h>
#include <linux/module.h>
+#include <linux/bootmem.h>
+#include <asm/sections.h>
#ifndef cache_line_size
#define cache_line_size() L1_CACHE_BYTES
@@ -147,3 +149,29 @@ void free_percpu(void *__pdata)
kfree(__percpu_disguise(__pdata));
}
EXPORT_SYMBOL_GPL(free_percpu);
+
+/*
+ * Generic percpu area setup.
+ */
+#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
+unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
+
+EXPORT_SYMBOL(__per_cpu_offset);
+
+void __init setup_per_cpu_areas(void)
+{
+ unsigned long size, i;
+ char *ptr;
+ unsigned long nr_possible_cpus = num_possible_cpus();
+
+ /* Copy section for each CPU (we discard the original) */
+ size = ALIGN(PERCPU_ENOUGH_ROOM, PAGE_SIZE);
+ ptr = alloc_bootmem_pages(size * nr_possible_cpus);
+
+ for_each_possible_cpu(i) {
+ __per_cpu_offset[i] = ptr - __per_cpu_start;
+ memcpy(ptr, __per_cpu_start, __per_cpu_end - __per_cpu_start);
+ ptr += size;
+ }
+}
+#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
diff --git a/mm/percpu.c b/mm/percpu.c
index c0b2c1a..0f16180 100644
--- a/mm/percpu.c
+++ b/mm/percpu.c
@@ -43,7 +43,7 @@
*
* To use this allocator, arch code should do the followings.
*
- * - define CONFIG_HAVE_DYNAMIC_PER_CPU_AREA
+ * - drop CONFIG_HAVE_LEGACY_PER_CPU_AREA
*
* - define __addr_to_pcpu_ptr() and __pcpu_ptr_to_addr() to translate
* regular address to percpu pointer and back if they need to be
@@ -1271,3 +1271,41 @@ ssize_t __init pcpu_embed_first_chunk(size_t static_size, size_t reserved_size,
reserved_size, dyn_size,
pcpue_unit_size, pcpue_ptr, NULL);
}
+
+/*
+ * Generic percpu area setup.
+ *
+ * The embedding helper is used because its behavior closely resembles
+ * the original non-dynamic generic percpu area setup. This is
+ * important because many archs have addressing restrictions and might
+ * fail if the percpu area is located far away from the previous
+ * location. As an added bonus, in non-NUMA cases, embedding is
+ * generally a good idea TLB-wise because percpu area can piggy back
+ * on the physical linear memory mapping which uses large page
+ * mappings on applicable archs.
+ */
+#ifndef CONFIG_HAVE_SETUP_PER_CPU_AREA
+unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
+EXPORT_SYMBOL(__per_cpu_offset);
+
+void __init setup_per_cpu_areas(void)
+{
+ size_t static_size = __per_cpu_end - __per_cpu_start;
+ ssize_t unit_size;
+ unsigned long delta;
+ unsigned int cpu;
+
+ /*
+ * Always reserve area for module percpu variables. That's
+ * what the legacy allocator did.
+ */
+ unit_size = pcpu_embed_first_chunk(static_size, PERCPU_MODULE_RESERVE,
+ PERCPU_DYNAMIC_RESERVE, -1);
+ if (unit_size < 0)
+ panic("Failed to initialized percpu areas.");
+
+ delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
+ for_each_possible_cpu(cpu)
+ __per_cpu_offset[cpu] = delta + cpu * unit_size;
+}
+#endif /* CONFIG_HAVE_SETUP_PER_CPU_AREA */
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 2/7] linker script: throw away .discard section
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
2009-05-20 7:37 ` [PATCH 1/7] percpu: use dynamic percpu allocator as the default percpu allocator Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 7:59 ` Mike Frysinger
2009-05-20 7:37 ` [PATCH 3/7] percpu: clean up percpu variable definitions Tejun Heo
` (7 subsequent siblings)
9 siblings, 1 reply; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo
x86 throws away .discard section but no other archs do. Also,
.discard is not thrown away while linking modules. Make every arch
and module linking throw it away. This will be used to define dummy
variables for percpu declarations and definitions.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: always throw away everything in .discard ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Richard Henderson <rth@twiddle.net>
Cc: Russell King <linux@arm.linux.org.uk>
Cc: Haavard Skinnemoen <hskinnemoen@atmel.com>
Cc: Bryan Wu <cooloney@kernel.org>
Cc: Mikael Starvik <starvik@axis.com>
Cc: Jesper Nilsson <jesper.nilsson@axis.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Tony Luck <tony.luck@intel.com>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Kyle McMartin <kyle@mcmartin.ca>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: David S. Miller <davem@davemloft.net>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Chris Zankel <chris@zankel.net>
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Ingo Molnar <mingo@elte.hu>
---
Makefile | 2 +-
arch/alpha/kernel/vmlinux.lds.S | 1 +
arch/arm/kernel/vmlinux.lds.S | 1 +
arch/avr32/kernel/vmlinux.lds.S | 1 +
arch/blackfin/kernel/vmlinux.lds.S | 1 +
arch/cris/kernel/vmlinux.lds.S | 1 +
arch/frv/kernel/vmlinux.lds.S | 2 ++
arch/h8300/kernel/vmlinux.lds.S | 1 +
arch/ia64/kernel/vmlinux.lds.S | 1 +
arch/m32r/kernel/vmlinux.lds.S | 1 +
arch/m68k/kernel/vmlinux-std.lds | 1 +
arch/m68k/kernel/vmlinux-sun3.lds | 1 +
arch/m68knommu/kernel/vmlinux.lds.S | 1 +
arch/microblaze/kernel/vmlinux.lds.S | 2 ++
arch/mips/kernel/vmlinux.lds.S | 1 +
arch/mn10300/kernel/vmlinux.lds.S | 1 +
arch/parisc/kernel/vmlinux.lds.S | 1 +
arch/powerpc/kernel/vmlinux.lds.S | 1 +
arch/s390/kernel/vmlinux.lds.S | 1 +
arch/sh/kernel/vmlinux_32.lds.S | 1 +
arch/sh/kernel/vmlinux_64.lds.S | 1 +
arch/sparc/kernel/vmlinux.lds.S | 1 +
arch/um/kernel/dyn.lds.S | 2 ++
arch/um/kernel/uml.lds.S | 2 ++
arch/xtensa/kernel/vmlinux.lds.S | 1 +
scripts/module-common.lds | 8 ++++++++
26 files changed, 37 insertions(+), 1 deletions(-)
create mode 100644 scripts/module-common.lds
diff --git a/Makefile b/Makefile
index b57e1f5..355b449 100644
--- a/Makefile
+++ b/Makefile
@@ -335,7 +335,7 @@ CHECKFLAGS := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
MODFLAGS = -DMODULE
CFLAGS_MODULE = $(MODFLAGS)
AFLAGS_MODULE = $(MODFLAGS)
-LDFLAGS_MODULE =
+LDFLAGS_MODULE = -T $(srctree)/scripts/module-common.lds
CFLAGS_KERNEL =
AFLAGS_KERNEL =
diff --git a/arch/alpha/kernel/vmlinux.lds.S b/arch/alpha/kernel/vmlinux.lds.S
index b9d6568..75fe1d6 100644
--- a/arch/alpha/kernel/vmlinux.lds.S
+++ b/arch/alpha/kernel/vmlinux.lds.S
@@ -139,6 +139,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
.mdebug 0 : {
diff --git a/arch/arm/kernel/vmlinux.lds.S b/arch/arm/kernel/vmlinux.lds.S
index c90f272..be63bee 100644
--- a/arch/arm/kernel/vmlinux.lds.S
+++ b/arch/arm/kernel/vmlinux.lds.S
@@ -82,6 +82,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
*(.ARM.exidx.exit.text)
*(.ARM.extab.exit.text)
#ifndef CONFIG_MMU
diff --git a/arch/avr32/kernel/vmlinux.lds.S b/arch/avr32/kernel/vmlinux.lds.S
index 7910d41..b832460 100644
--- a/arch/avr32/kernel/vmlinux.lds.S
+++ b/arch/avr32/kernel/vmlinux.lds.S
@@ -131,6 +131,7 @@ SECTIONS
/DISCARD/ : {
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
DWARF_DEBUG
diff --git a/arch/blackfin/kernel/vmlinux.lds.S b/arch/blackfin/kernel/vmlinux.lds.S
index 27952ae..1fa9491 100644
--- a/arch/blackfin/kernel/vmlinux.lds.S
+++ b/arch/blackfin/kernel/vmlinux.lds.S
@@ -262,5 +262,6 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
}
diff --git a/arch/cris/kernel/vmlinux.lds.S b/arch/cris/kernel/vmlinux.lds.S
index 0d2adfc..a3175eb 100644
--- a/arch/cris/kernel/vmlinux.lds.S
+++ b/arch/cris/kernel/vmlinux.lds.S
@@ -145,6 +145,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
dram_end = dram_start + (CONFIG_ETRAX_DRAM_SIZE - __CONFIG_ETRAX_VMEM_SIZE)*1024*1024;
diff --git a/arch/frv/kernel/vmlinux.lds.S b/arch/frv/kernel/vmlinux.lds.S
index 22d9787..64b5a5e 100644
--- a/arch/frv/kernel/vmlinux.lds.S
+++ b/arch/frv/kernel/vmlinux.lds.S
@@ -177,6 +177,8 @@ SECTIONS
.debug_ranges 0 : { *(.debug_ranges) }
.comment 0 : { *(.comment) }
+
+ /DISCARD/ : { *(.discard) }
}
__kernel_image_size_no_bss = __bss_start - __kernel_image_start;
diff --git a/arch/h8300/kernel/vmlinux.lds.S b/arch/h8300/kernel/vmlinux.lds.S
index 43a87b9..03d6c0d 100644
--- a/arch/h8300/kernel/vmlinux.lds.S
+++ b/arch/h8300/kernel/vmlinux.lds.S
@@ -154,6 +154,7 @@ SECTIONS
}
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
}
.romfs :
{
diff --git a/arch/ia64/kernel/vmlinux.lds.S b/arch/ia64/kernel/vmlinux.lds.S
index 4a95e86..13d9589 100644
--- a/arch/ia64/kernel/vmlinux.lds.S
+++ b/arch/ia64/kernel/vmlinux.lds.S
@@ -29,6 +29,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
*(.IA_64.unwind.exit.text)
*(.IA_64.unwind_info.exit.text)
}
diff --git a/arch/m32r/kernel/vmlinux.lds.S b/arch/m32r/kernel/vmlinux.lds.S
index 4179adf..480a499 100644
--- a/arch/m32r/kernel/vmlinux.lds.S
+++ b/arch/m32r/kernel/vmlinux.lds.S
@@ -125,6 +125,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
/* Stabs debugging sections. */
diff --git a/arch/m68k/kernel/vmlinux-std.lds b/arch/m68k/kernel/vmlinux-std.lds
index 01d212b..905a797 100644
--- a/arch/m68k/kernel/vmlinux-std.lds
+++ b/arch/m68k/kernel/vmlinux-std.lds
@@ -87,6 +87,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
/* Stabs debugging sections. */
diff --git a/arch/m68k/kernel/vmlinux-sun3.lds b/arch/m68k/kernel/vmlinux-sun3.lds
index c192f77..47d04be 100644
--- a/arch/m68k/kernel/vmlinux-sun3.lds
+++ b/arch/m68k/kernel/vmlinux-sun3.lds
@@ -82,6 +82,7 @@ __init_begin = .;
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
.crap : {
diff --git a/arch/m68knommu/kernel/vmlinux.lds.S b/arch/m68knommu/kernel/vmlinux.lds.S
index b7fe505..68111a6 100644
--- a/arch/m68knommu/kernel/vmlinux.lds.S
+++ b/arch/m68knommu/kernel/vmlinux.lds.S
@@ -188,6 +188,7 @@ SECTIONS {
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
.bss : {
diff --git a/arch/microblaze/kernel/vmlinux.lds.S b/arch/microblaze/kernel/vmlinux.lds.S
index 840385e..4547363 100644
--- a/arch/microblaze/kernel/vmlinux.lds.S
+++ b/arch/microblaze/kernel/vmlinux.lds.S
@@ -160,4 +160,6 @@ SECTIONS {
}
. = ALIGN(4096);
_end = .;
+
+ /DISCARD/ : { *(.discard) }
}
diff --git a/arch/mips/kernel/vmlinux.lds.S b/arch/mips/kernel/vmlinux.lds.S
index 58738c8..4590160 100644
--- a/arch/mips/kernel/vmlinux.lds.S
+++ b/arch/mips/kernel/vmlinux.lds.S
@@ -179,6 +179,7 @@ SECTIONS
/* Sections to be discarded */
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
/* ABI crap starts here */
*(.MIPS.options)
diff --git a/arch/mn10300/kernel/vmlinux.lds.S b/arch/mn10300/kernel/vmlinux.lds.S
index 24de6b9..5d9f2f9 100644
--- a/arch/mn10300/kernel/vmlinux.lds.S
+++ b/arch/mn10300/kernel/vmlinux.lds.S
@@ -146,6 +146,7 @@ SECTIONS
/* Sections to be discarded */
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
}
STABS_DEBUG
diff --git a/arch/parisc/kernel/vmlinux.lds.S b/arch/parisc/kernel/vmlinux.lds.S
index fd2cc4f..ccf5834 100644
--- a/arch/parisc/kernel/vmlinux.lds.S
+++ b/arch/parisc/kernel/vmlinux.lds.S
@@ -240,6 +240,7 @@ SECTIONS
/* Sections to be discarded */
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
#ifdef CONFIG_64BIT
/* temporary hack until binutils is fixed to not emit these
* for static binaries
diff --git a/arch/powerpc/kernel/vmlinux.lds.S b/arch/powerpc/kernel/vmlinux.lds.S
index 8ef8a14..7fca935 100644
--- a/arch/powerpc/kernel/vmlinux.lds.S
+++ b/arch/powerpc/kernel/vmlinux.lds.S
@@ -40,6 +40,7 @@ SECTIONS
/* Sections to be discarded. */
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
EXIT_DATA
}
diff --git a/arch/s390/kernel/vmlinux.lds.S b/arch/s390/kernel/vmlinux.lds.S
index 89399b8..efa7a89 100644
--- a/arch/s390/kernel/vmlinux.lds.S
+++ b/arch/s390/kernel/vmlinux.lds.S
@@ -160,6 +160,7 @@ SECTIONS
/DISCARD/ : {
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
/* Debugging sections. */
diff --git a/arch/sh/kernel/vmlinux_32.lds.S b/arch/sh/kernel/vmlinux_32.lds.S
index dd9b2ee..5eb6856 100644
--- a/arch/sh/kernel/vmlinux_32.lds.S
+++ b/arch/sh/kernel/vmlinux_32.lds.S
@@ -147,6 +147,7 @@ SECTIONS
*/
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
}
STABS_DEBUG
diff --git a/arch/sh/kernel/vmlinux_64.lds.S b/arch/sh/kernel/vmlinux_64.lds.S
index 6966446..afc515a 100644
--- a/arch/sh/kernel/vmlinux_64.lds.S
+++ b/arch/sh/kernel/vmlinux_64.lds.S
@@ -156,6 +156,7 @@ SECTIONS
*/
/DISCARD/ : {
*(.exitcall.exit)
+ *(.discard)
}
STABS_DEBUG
diff --git a/arch/sparc/kernel/vmlinux.lds.S b/arch/sparc/kernel/vmlinux.lds.S
index fcbbd00..d63cf91 100644
--- a/arch/sparc/kernel/vmlinux.lds.S
+++ b/arch/sparc/kernel/vmlinux.lds.S
@@ -175,6 +175,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
STABS_DEBUG
diff --git a/arch/um/kernel/dyn.lds.S b/arch/um/kernel/dyn.lds.S
index 9975e1a..2916d6e 100644
--- a/arch/um/kernel/dyn.lds.S
+++ b/arch/um/kernel/dyn.lds.S
@@ -156,4 +156,6 @@ SECTIONS
STABS_DEBUG
DWARF_DEBUG
+
+ /DISCARD/ : { *(.discard) }
}
diff --git a/arch/um/kernel/uml.lds.S b/arch/um/kernel/uml.lds.S
index 11b8352..1f8a622 100644
--- a/arch/um/kernel/uml.lds.S
+++ b/arch/um/kernel/uml.lds.S
@@ -100,4 +100,6 @@ SECTIONS
STABS_DEBUG
DWARF_DEBUG
+
+ /DISCARD/ : { *(.discard) }
}
diff --git a/arch/xtensa/kernel/vmlinux.lds.S b/arch/xtensa/kernel/vmlinux.lds.S
index 41c159c..b1e2463 100644
--- a/arch/xtensa/kernel/vmlinux.lds.S
+++ b/arch/xtensa/kernel/vmlinux.lds.S
@@ -287,6 +287,7 @@ SECTIONS
EXIT_TEXT
EXIT_DATA
*(.exitcall.exit)
+ *(.discard)
}
.xt.lit : { *(.xt.lit) }
diff --git a/scripts/module-common.lds b/scripts/module-common.lds
new file mode 100644
index 0000000..47a1f9a
--- /dev/null
+++ b/scripts/module-common.lds
@@ -0,0 +1,8 @@
+/*
+ * Common module linker script, always used when linking a module.
+ * Archs are free to supply their own linker scripts. ld will
+ * combine them automatically.
+ */
+SECTIONS {
+ /DISCARD/ : { *(.discard) }
+}
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 2/7] linker script: throw away .discard section
2009-05-20 7:37 ` [PATCH 2/7] linker script: throw away .discard section Tejun Heo
@ 2009-05-20 7:59 ` Mike Frysinger
2009-05-20 8:06 ` Tejun Heo
0 siblings, 1 reply; 19+ messages in thread
From: Mike Frysinger @ 2009-05-20 7:59 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
On Wed, May 20, 2009 at 03:37, Tejun Heo wrote:
> x86 throws away .discard section but no other archs do. Also,
> .discard is not thrown away while linking modules. Make every arch
> and module linking throw it away. This will be used to define dummy
> variables for percpu declarations and definitions.
seems like a good time to introduce a common DISCARD define to
include/asm-generic/vmlinux.lds.h. considering multiple /DISCARD/
entries are allowed in a linker script, the define should be pretty
straightforward:
#define DISCARDS \
/DISCARD/ : { \
EXIT_TEXT \
EXIT_DATA \
*(.exitcall.exit) \
*(.discard) \
}
-mike
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 2/7] linker script: throw away .discard section
2009-05-20 7:59 ` Mike Frysinger
@ 2009-05-20 8:06 ` Tejun Heo
2009-05-20 8:18 ` Tejun Heo
0 siblings, 1 reply; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 8:06 UTC (permalink / raw)
To: Mike Frysinger
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Mike Frysinger wrote:
> On Wed, May 20, 2009 at 03:37, Tejun Heo wrote:
>> x86 throws away .discard section but no other archs do. Also,
>> .discard is not thrown away while linking modules. Make every arch
>> and module linking throw it away. This will be used to define dummy
>> variables for percpu declarations and definitions.
>
> seems like a good time to introduce a common DISCARD define to
> include/asm-generic/vmlinux.lds.h. considering multiple /DISCARD/
> entries are allowed in a linker script, the define should be pretty
> straightforward:
> #define DISCARDS \
> /DISCARD/ : { \
> EXIT_TEXT \
> EXIT_DATA \
> *(.exitcall.exit) \
> *(.discard) \
> }
Hmmm... indeed, that would be nice. I'll update the patch.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 2/7] linker script: throw away .discard section
2009-05-20 8:06 ` Tejun Heo
@ 2009-05-20 8:18 ` Tejun Heo
2009-05-20 8:28 ` Mike Frysinger
0 siblings, 1 reply; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 8:18 UTC (permalink / raw)
To: Mike Frysinger
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Tejun Heo wrote:
> Mike Frysinger wrote:
>> On Wed, May 20, 2009 at 03:37, Tejun Heo wrote:
>>> x86 throws away .discard section but no other archs do. Also,
>>> .discard is not thrown away while linking modules. Make every arch
>>> and module linking throw it away. This will be used to define dummy
>>> variables for percpu declarations and definitions.
>> seems like a good time to introduce a common DISCARD define to
>> include/asm-generic/vmlinux.lds.h. considering multiple /DISCARD/
>> entries are allowed in a linker script, the define should be pretty
>> straightforward:
>> #define DISCARDS \
>> /DISCARD/ : { \
>> EXIT_TEXT \
>> EXIT_DATA \
>> *(.exitcall.exit) \
>> *(.discard) \
>> }
>
> Hmmm... indeed, that would be nice. I'll update the patch.
Eh... handling of EXIT_TEXT and DATA aren't uniform across archs. I
think I'll leave the /DISCARD/ unification for the next time.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 2/7] linker script: throw away .discard section
2009-05-20 8:18 ` Tejun Heo
@ 2009-05-20 8:28 ` Mike Frysinger
0 siblings, 0 replies; 19+ messages in thread
From: Mike Frysinger @ 2009-05-20 8:28 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
On Wed, May 20, 2009 at 04:18, Tejun Heo wrote:
> Tejun Heo wrote:
>> Mike Frysinger wrote:
>>> On Wed, May 20, 2009 at 03:37, Tejun Heo wrote:
>>>> x86 throws away .discard section but no other archs do. Also,
>>>> .discard is not thrown away while linking modules. Make every arch
>>>> and module linking throw it away. This will be used to define dummy
>>>> variables for percpu declarations and definitions.
>>> seems like a good time to introduce a common DISCARD define to
>>> include/asm-generic/vmlinux.lds.h. considering multiple /DISCARD/
>>> entries are allowed in a linker script, the define should be pretty
>>> straightforward:
>>> #define DISCARDS \
>>> /DISCARD/ : { \
>>> EXIT_TEXT \
>>> EXIT_DATA \
>>> *(.exitcall.exit) \
>>> *(.discard) \
>>> }
>>
>> Hmmm... indeed, that would be nice. I'll update the patch.
>
> Eh... handling of EXIT_TEXT and DATA aren't uniform across archs. I
> think I'll leave the /DISCARD/ unification for the next time.
right, i think that's a bug where people simply forgot to declare
them. if you stick DISCARDS at the end (and only replace when an arch
already has it at the end), there should be no change in behavior.
-mike
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 3/7] percpu: clean up percpu variable definitions
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
2009-05-20 7:37 ` [PATCH 1/7] percpu: use dynamic percpu allocator as the default percpu allocator Tejun Heo
2009-05-20 7:37 ` [PATCH 2/7] linker script: throw away .discard section Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 9:17 ` Jens Axboe
2009-05-25 6:07 ` Rusty Russell
2009-05-20 7:37 ` [PATCH 4/7] percpu: enforce global uniqueness and disallow in-function statics Tejun Heo
` (6 subsequent siblings)
9 siblings, 2 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo, Jens Axboe, Dave Jones, Jeremy Fitzhardinge, linux-mm
Percpu variable definition is about to be updated such that
* percpu symbols must be unique even the static ones
* in-function static definition is not allowed
Update percpu variable definitions accoringly.
* as,cfq: rename ioc_count uniquely
* cpufreq: rename cpu_dbs_info uniquely
* xen: move nesting_count out of xen_evtchn_do_upcall() and rename it
* mm: move ratelimits out of balance_dirty_pages_ratelimited_nr() and
rename it
* ipv4,6: rename cookie_scratch uniquely
While at it, make cris:use DECLARE_PER_CPU() instead of extern
volatile DEFINE_PER_CPU() for declaration.
[ Impact: percpu usage cleanups, no duplicate static percpu var names ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Jens Axboe <jens.axboe@oracle.com>
Cc: Dave Jones <davej@redhat.com>
Cc: Jeremy Fitzhardinge <jeremy@xensource.com>
Cc: linux-mm <linux-mm@kvack.org>
Cc: David S. Miller <davem@davemloft.net>
---
arch/cris/include/asm/mmu_context.h | 2 +-
block/as-iosched.c | 10 +++++-----
block/cfq-iosched.c | 10 +++++-----
drivers/cpufreq/cpufreq_conservative.c | 12 ++++++------
drivers/cpufreq/cpufreq_ondemand.c | 15 ++++++++-------
drivers/xen/events.c | 9 +++++----
mm/page-writeback.c | 5 +++--
net/ipv4/syncookies.c | 4 ++--
net/ipv6/syncookies.c | 4 ++--
9 files changed, 37 insertions(+), 34 deletions(-)
diff --git a/arch/cris/include/asm/mmu_context.h b/arch/cris/include/asm/mmu_context.h
index 72ba08d..00de1a0 100644
--- a/arch/cris/include/asm/mmu_context.h
+++ b/arch/cris/include/asm/mmu_context.h
@@ -17,7 +17,7 @@ extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
* registers like cr3 on the i386
*/
-extern volatile DEFINE_PER_CPU(pgd_t *,current_pgd); /* defined in arch/cris/mm/fault.c */
+DECLARE_PER_CPU(pgd_t *,current_pgd); /* defined in arch/cris/mm/fault.c */
static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
{
diff --git a/block/as-iosched.c b/block/as-iosched.c
index c48fa67..96ff4d1 100644
--- a/block/as-iosched.c
+++ b/block/as-iosched.c
@@ -146,7 +146,7 @@ enum arq_state {
#define RQ_STATE(rq) ((enum arq_state)(rq)->elevator_private2)
#define RQ_SET_STATE(rq, state) ((rq)->elevator_private2 = (void *) state)
-static DEFINE_PER_CPU(unsigned long, ioc_count);
+static DEFINE_PER_CPU(unsigned long, as_ioc_count);
static struct completion *ioc_gone;
static DEFINE_SPINLOCK(ioc_gone_lock);
@@ -161,7 +161,7 @@ static void as_antic_stop(struct as_data *ad);
static void free_as_io_context(struct as_io_context *aic)
{
kfree(aic);
- elv_ioc_count_dec(ioc_count);
+ elv_ioc_count_dec(as_ioc_count);
if (ioc_gone) {
/*
* AS scheduler is exiting, grab exit lock and check
@@ -169,7 +169,7 @@ static void free_as_io_context(struct as_io_context *aic)
* complete ioc_gone and set it back to NULL.
*/
spin_lock(&ioc_gone_lock);
- if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
+ if (ioc_gone && !elv_ioc_count_read(as_ioc_count)) {
complete(ioc_gone);
ioc_gone = NULL;
}
@@ -211,7 +211,7 @@ static struct as_io_context *alloc_as_io_context(void)
ret->seek_total = 0;
ret->seek_samples = 0;
ret->seek_mean = 0;
- elv_ioc_count_inc(ioc_count);
+ elv_ioc_count_inc(as_ioc_count);
}
return ret;
@@ -1509,7 +1509,7 @@ static void __exit as_exit(void)
ioc_gone = &all_gone;
/* ioc_gone's update must be visible before reading ioc_count */
smp_wmb();
- if (elv_ioc_count_read(ioc_count))
+ if (elv_ioc_count_read(as_ioc_count))
wait_for_completion(&all_gone);
synchronize_rcu();
}
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index a55a9bd..deea748 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -48,7 +48,7 @@ static int cfq_slice_idle = HZ / 125;
static struct kmem_cache *cfq_pool;
static struct kmem_cache *cfq_ioc_pool;
-static DEFINE_PER_CPU(unsigned long, ioc_count);
+static DEFINE_PER_CPU(unsigned long, cfq_ioc_count);
static struct completion *ioc_gone;
static DEFINE_SPINLOCK(ioc_gone_lock);
@@ -1423,7 +1423,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
cic = container_of(head, struct cfq_io_context, rcu_head);
kmem_cache_free(cfq_ioc_pool, cic);
- elv_ioc_count_dec(ioc_count);
+ elv_ioc_count_dec(cfq_ioc_count);
if (ioc_gone) {
/*
@@ -1432,7 +1432,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
* complete ioc_gone and set it back to NULL
*/
spin_lock(&ioc_gone_lock);
- if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
+ if (ioc_gone && !elv_ioc_count_read(cfq_ioc_count)) {
complete(ioc_gone);
ioc_gone = NULL;
}
@@ -1558,7 +1558,7 @@ cfq_alloc_io_context(struct cfq_data *cfqd, gfp_t gfp_mask)
INIT_HLIST_NODE(&cic->cic_list);
cic->dtor = cfq_free_io_context;
cic->exit = cfq_exit_io_context;
- elv_ioc_count_inc(ioc_count);
+ elv_ioc_count_inc(cfq_ioc_count);
}
return cic;
@@ -2663,7 +2663,7 @@ static void __exit cfq_exit(void)
* this also protects us from entering cfq_slab_kill() with
* pending RCU callbacks
*/
- if (elv_ioc_count_read(ioc_count))
+ if (elv_ioc_count_read(cfq_ioc_count))
wait_for_completion(&all_gone);
cfq_slab_kill();
}
diff --git a/drivers/cpufreq/cpufreq_conservative.c b/drivers/cpufreq/cpufreq_conservative.c
index 2ecd95e..e0faa3e 100644
--- a/drivers/cpufreq/cpufreq_conservative.c
+++ b/drivers/cpufreq/cpufreq_conservative.c
@@ -80,7 +80,7 @@ struct cpu_dbs_info_s {
int cpu;
unsigned int enable:1;
};
-static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info);
+static DEFINE_PER_CPU(struct cpu_dbs_info_s, cs_cpu_dbs_info);
static unsigned int dbs_enable; /* number of CPUs using this policy */
@@ -150,7 +150,7 @@ dbs_cpufreq_notifier(struct notifier_block *nb, unsigned long val,
void *data)
{
struct cpufreq_freqs *freq = data;
- struct cpu_dbs_info_s *this_dbs_info = &per_cpu(cpu_dbs_info,
+ struct cpu_dbs_info_s *this_dbs_info = &per_cpu(cs_cpu_dbs_info,
freq->cpu);
struct cpufreq_policy *policy;
@@ -323,7 +323,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,
/* we need to re-evaluate prev_cpu_idle */
for_each_online_cpu(j) {
struct cpu_dbs_info_s *dbs_info;
- dbs_info = &per_cpu(cpu_dbs_info, j);
+ dbs_info = &per_cpu(cs_cpu_dbs_info, j);
dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
&dbs_info->prev_cpu_wall);
if (dbs_tuners_ins.ignore_nice)
@@ -413,7 +413,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)
cputime64_t cur_wall_time, cur_idle_time;
unsigned int idle_time, wall_time;
- j_dbs_info = &per_cpu(cpu_dbs_info, j);
+ j_dbs_info = &per_cpu(cs_cpu_dbs_info, j);
cur_idle_time = get_cpu_idle_time(j, &cur_wall_time);
@@ -553,7 +553,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
unsigned int j;
int rc;
- this_dbs_info = &per_cpu(cpu_dbs_info, cpu);
+ this_dbs_info = &per_cpu(cs_cpu_dbs_info, cpu);
switch (event) {
case CPUFREQ_GOV_START:
@@ -573,7 +573,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
for_each_cpu(j, policy->cpus) {
struct cpu_dbs_info_s *j_dbs_info;
- j_dbs_info = &per_cpu(cpu_dbs_info, j);
+ j_dbs_info = &per_cpu(cs_cpu_dbs_info, j);
j_dbs_info->cur_policy = policy;
j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
diff --git a/drivers/cpufreq/cpufreq_ondemand.c b/drivers/cpufreq/cpufreq_ondemand.c
index 338f428..2eaf88f 100644
--- a/drivers/cpufreq/cpufreq_ondemand.c
+++ b/drivers/cpufreq/cpufreq_ondemand.c
@@ -87,7 +87,7 @@ struct cpu_dbs_info_s {
unsigned int enable:1,
sample_type:1;
};
-static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info);
+static DEFINE_PER_CPU(struct cpu_dbs_info_s, od_cpu_dbs_info);
static unsigned int dbs_enable; /* number of CPUs using this policy */
@@ -162,7 +162,8 @@ static unsigned int powersave_bias_target(struct cpufreq_policy *policy,
unsigned int freq_hi, freq_lo;
unsigned int index = 0;
unsigned int jiffies_total, jiffies_hi, jiffies_lo;
- struct cpu_dbs_info_s *dbs_info = &per_cpu(cpu_dbs_info, policy->cpu);
+ struct cpu_dbs_info_s *dbs_info = &per_cpu(od_cpu_dbs_info,
+ policy->cpu);
if (!dbs_info->freq_table) {
dbs_info->freq_lo = 0;
@@ -207,7 +208,7 @@ static void ondemand_powersave_bias_init(void)
{
int i;
for_each_online_cpu(i) {
- struct cpu_dbs_info_s *dbs_info = &per_cpu(cpu_dbs_info, i);
+ struct cpu_dbs_info_s *dbs_info = &per_cpu(od_cpu_dbs_info, i);
dbs_info->freq_table = cpufreq_frequency_get_table(i);
dbs_info->freq_lo = 0;
}
@@ -322,7 +323,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,
/* we need to re-evaluate prev_cpu_idle */
for_each_online_cpu(j) {
struct cpu_dbs_info_s *dbs_info;
- dbs_info = &per_cpu(cpu_dbs_info, j);
+ dbs_info = &per_cpu(od_cpu_dbs_info, j);
dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
&dbs_info->prev_cpu_wall);
if (dbs_tuners_ins.ignore_nice)
@@ -416,7 +417,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)
unsigned int load, load_freq;
int freq_avg;
- j_dbs_info = &per_cpu(cpu_dbs_info, j);
+ j_dbs_info = &per_cpu(od_cpu_dbs_info, j);
cur_idle_time = get_cpu_idle_time(j, &cur_wall_time);
@@ -573,7 +574,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
unsigned int j;
int rc;
- this_dbs_info = &per_cpu(cpu_dbs_info, cpu);
+ this_dbs_info = &per_cpu(od_cpu_dbs_info, cpu);
switch (event) {
case CPUFREQ_GOV_START:
@@ -595,7 +596,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
for_each_cpu(j, policy->cpus) {
struct cpu_dbs_info_s *j_dbs_info;
- j_dbs_info = &per_cpu(cpu_dbs_info, j);
+ j_dbs_info = &per_cpu(od_cpu_dbs_info, j);
j_dbs_info->cur_policy = policy;
j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
diff --git a/drivers/xen/events.c b/drivers/xen/events.c
index 30963af..4dbe5c0 100644
--- a/drivers/xen/events.c
+++ b/drivers/xen/events.c
@@ -596,6 +596,8 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id)
return IRQ_HANDLED;
}
+static DEFINE_PER_CPU(unsigned, xed_nesting_count);
+
/*
* Search the CPUs pending events bitmasks. For each one found, map
* the event number to an irq, and feed it into do_IRQ() for
@@ -611,7 +613,6 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
struct pt_regs *old_regs = set_irq_regs(regs);
struct shared_info *s = HYPERVISOR_shared_info;
struct vcpu_info *vcpu_info = __get_cpu_var(xen_vcpu);
- static DEFINE_PER_CPU(unsigned, nesting_count);
unsigned count;
exit_idle();
@@ -622,7 +623,7 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
vcpu_info->evtchn_upcall_pending = 0;
- if (__get_cpu_var(nesting_count)++)
+ if (__get_cpu_var(xed_nesting_count)++)
goto out;
#ifndef CONFIG_X86 /* No need for a barrier -- XCHG is a barrier on x86. */
@@ -647,8 +648,8 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
BUG_ON(!irqs_disabled());
- count = __get_cpu_var(nesting_count);
- __get_cpu_var(nesting_count) = 0;
+ count = __get_cpu_var(xed_nesting_count);
+ __get_cpu_var(xed_nesting_count) = 0;
} while(count != 1);
out:
diff --git a/mm/page-writeback.c b/mm/page-writeback.c
index bb553c3..0e0c9de 100644
--- a/mm/page-writeback.c
+++ b/mm/page-writeback.c
@@ -606,6 +606,8 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite)
}
}
+static DEFINE_PER_CPU(unsigned long, bdp_ratelimits) = 0;
+
/**
* balance_dirty_pages_ratelimited_nr - balance dirty memory state
* @mapping: address_space which was dirtied
@@ -623,7 +625,6 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite)
void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
unsigned long nr_pages_dirtied)
{
- static DEFINE_PER_CPU(unsigned long, ratelimits) = 0;
unsigned long ratelimit;
unsigned long *p;
@@ -636,7 +637,7 @@ void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
* tasks in balance_dirty_pages(). Period.
*/
preempt_disable();
- p = &__get_cpu_var(ratelimits);
+ p = &__get_cpu_var(bdp_ratelimits);
*p += nr_pages_dirtied;
if (unlikely(*p >= ratelimit)) {
*p = 0;
diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c
index b35a950..70ee18c 100644
--- a/net/ipv4/syncookies.c
+++ b/net/ipv4/syncookies.c
@@ -37,12 +37,12 @@ __initcall(init_syncookies);
#define COOKIEBITS 24 /* Upper bits store count */
#define COOKIEMASK (((__u32)1 << COOKIEBITS) - 1)
-static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
+static DEFINE_PER_CPU(__u32, ipv4_cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
static u32 cookie_hash(__be32 saddr, __be32 daddr, __be16 sport, __be16 dport,
u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(cookie_scratch);
+ __u32 *tmp = __get_cpu_var(ipv4_cookie_scratch);
memcpy(tmp + 4, syncookie_secret[c], sizeof(syncookie_secret[c]));
tmp[0] = (__force u32)saddr;
diff --git a/net/ipv6/syncookies.c b/net/ipv6/syncookies.c
index 711175e..348e38c 100644
--- a/net/ipv6/syncookies.c
+++ b/net/ipv6/syncookies.c
@@ -74,12 +74,12 @@ static inline struct sock *get_cookie_sock(struct sock *sk, struct sk_buff *skb,
return child;
}
-static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
+static DEFINE_PER_CPU(__u32, ipv6_cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
static u32 cookie_hash(struct in6_addr *saddr, struct in6_addr *daddr,
__be16 sport, __be16 dport, u32 count, int c)
{
- __u32 *tmp = __get_cpu_var(cookie_scratch);
+ __u32 *tmp = __get_cpu_var(ipv6_cookie_scratch);
/*
* we have 320 bits of information to hash, copy in the remaining
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 3/7] percpu: clean up percpu variable definitions
2009-05-20 7:37 ` [PATCH 3/7] percpu: clean up percpu variable definitions Tejun Heo
@ 2009-05-20 9:17 ` Jens Axboe
2009-05-25 6:07 ` Rusty Russell
1 sibling, 0 replies; 19+ messages in thread
From: Jens Axboe @ 2009-05-20 9:17 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty, Dave Jones,
Jeremy Fitzhardinge, linux-mm
On Wed, May 20 2009, Tejun Heo wrote:
> Percpu variable definition is about to be updated such that
>
> * percpu symbols must be unique even the static ones
>
> * in-function static definition is not allowed
>
> Update percpu variable definitions accoringly.
>
> * as,cfq: rename ioc_count uniquely
>
> * cpufreq: rename cpu_dbs_info uniquely
>
> * xen: move nesting_count out of xen_evtchn_do_upcall() and rename it
>
> * mm: move ratelimits out of balance_dirty_pages_ratelimited_nr() and
> rename it
>
> * ipv4,6: rename cookie_scratch uniquely
>
> While at it, make cris:use DECLARE_PER_CPU() instead of extern
> volatile DEFINE_PER_CPU() for declaration.
>
> [ Impact: percpu usage cleanups, no duplicate static percpu var names ]
The block bits looks fine.
Acked-by: Jens Axboe <jens.axboe@oracle.com>
>
> Signed-off-by: Tejun Heo <tj@kernel.org>
> Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
> Cc: Jens Axboe <jens.axboe@oracle.com>
> Cc: Dave Jones <davej@redhat.com>
> Cc: Jeremy Fitzhardinge <jeremy@xensource.com>
> Cc: linux-mm <linux-mm@kvack.org>
> Cc: David S. Miller <davem@davemloft.net>
> ---
> arch/cris/include/asm/mmu_context.h | 2 +-
> block/as-iosched.c | 10 +++++-----
> block/cfq-iosched.c | 10 +++++-----
> drivers/cpufreq/cpufreq_conservative.c | 12 ++++++------
> drivers/cpufreq/cpufreq_ondemand.c | 15 ++++++++-------
> drivers/xen/events.c | 9 +++++----
> mm/page-writeback.c | 5 +++--
> net/ipv4/syncookies.c | 4 ++--
> net/ipv6/syncookies.c | 4 ++--
> 9 files changed, 37 insertions(+), 34 deletions(-)
>
> diff --git a/arch/cris/include/asm/mmu_context.h b/arch/cris/include/asm/mmu_context.h
> index 72ba08d..00de1a0 100644
> --- a/arch/cris/include/asm/mmu_context.h
> +++ b/arch/cris/include/asm/mmu_context.h
> @@ -17,7 +17,7 @@ extern void switch_mm(struct mm_struct *prev, struct mm_struct *next,
> * registers like cr3 on the i386
> */
>
> -extern volatile DEFINE_PER_CPU(pgd_t *,current_pgd); /* defined in arch/cris/mm/fault.c */
> +DECLARE_PER_CPU(pgd_t *,current_pgd); /* defined in arch/cris/mm/fault.c */
>
> static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk)
> {
> diff --git a/block/as-iosched.c b/block/as-iosched.c
> index c48fa67..96ff4d1 100644
> --- a/block/as-iosched.c
> +++ b/block/as-iosched.c
> @@ -146,7 +146,7 @@ enum arq_state {
> #define RQ_STATE(rq) ((enum arq_state)(rq)->elevator_private2)
> #define RQ_SET_STATE(rq, state) ((rq)->elevator_private2 = (void *) state)
>
> -static DEFINE_PER_CPU(unsigned long, ioc_count);
> +static DEFINE_PER_CPU(unsigned long, as_ioc_count);
> static struct completion *ioc_gone;
> static DEFINE_SPINLOCK(ioc_gone_lock);
>
> @@ -161,7 +161,7 @@ static void as_antic_stop(struct as_data *ad);
> static void free_as_io_context(struct as_io_context *aic)
> {
> kfree(aic);
> - elv_ioc_count_dec(ioc_count);
> + elv_ioc_count_dec(as_ioc_count);
> if (ioc_gone) {
> /*
> * AS scheduler is exiting, grab exit lock and check
> @@ -169,7 +169,7 @@ static void free_as_io_context(struct as_io_context *aic)
> * complete ioc_gone and set it back to NULL.
> */
> spin_lock(&ioc_gone_lock);
> - if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
> + if (ioc_gone && !elv_ioc_count_read(as_ioc_count)) {
> complete(ioc_gone);
> ioc_gone = NULL;
> }
> @@ -211,7 +211,7 @@ static struct as_io_context *alloc_as_io_context(void)
> ret->seek_total = 0;
> ret->seek_samples = 0;
> ret->seek_mean = 0;
> - elv_ioc_count_inc(ioc_count);
> + elv_ioc_count_inc(as_ioc_count);
> }
>
> return ret;
> @@ -1509,7 +1509,7 @@ static void __exit as_exit(void)
> ioc_gone = &all_gone;
> /* ioc_gone's update must be visible before reading ioc_count */
> smp_wmb();
> - if (elv_ioc_count_read(ioc_count))
> + if (elv_ioc_count_read(as_ioc_count))
> wait_for_completion(&all_gone);
> synchronize_rcu();
> }
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index a55a9bd..deea748 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -48,7 +48,7 @@ static int cfq_slice_idle = HZ / 125;
> static struct kmem_cache *cfq_pool;
> static struct kmem_cache *cfq_ioc_pool;
>
> -static DEFINE_PER_CPU(unsigned long, ioc_count);
> +static DEFINE_PER_CPU(unsigned long, cfq_ioc_count);
> static struct completion *ioc_gone;
> static DEFINE_SPINLOCK(ioc_gone_lock);
>
> @@ -1423,7 +1423,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> cic = container_of(head, struct cfq_io_context, rcu_head);
>
> kmem_cache_free(cfq_ioc_pool, cic);
> - elv_ioc_count_dec(ioc_count);
> + elv_ioc_count_dec(cfq_ioc_count);
>
> if (ioc_gone) {
> /*
> @@ -1432,7 +1432,7 @@ static void cfq_cic_free_rcu(struct rcu_head *head)
> * complete ioc_gone and set it back to NULL
> */
> spin_lock(&ioc_gone_lock);
> - if (ioc_gone && !elv_ioc_count_read(ioc_count)) {
> + if (ioc_gone && !elv_ioc_count_read(cfq_ioc_count)) {
> complete(ioc_gone);
> ioc_gone = NULL;
> }
> @@ -1558,7 +1558,7 @@ cfq_alloc_io_context(struct cfq_data *cfqd, gfp_t gfp_mask)
> INIT_HLIST_NODE(&cic->cic_list);
> cic->dtor = cfq_free_io_context;
> cic->exit = cfq_exit_io_context;
> - elv_ioc_count_inc(ioc_count);
> + elv_ioc_count_inc(cfq_ioc_count);
> }
>
> return cic;
> @@ -2663,7 +2663,7 @@ static void __exit cfq_exit(void)
> * this also protects us from entering cfq_slab_kill() with
> * pending RCU callbacks
> */
> - if (elv_ioc_count_read(ioc_count))
> + if (elv_ioc_count_read(cfq_ioc_count))
> wait_for_completion(&all_gone);
> cfq_slab_kill();
> }
> diff --git a/drivers/cpufreq/cpufreq_conservative.c b/drivers/cpufreq/cpufreq_conservative.c
> index 2ecd95e..e0faa3e 100644
> --- a/drivers/cpufreq/cpufreq_conservative.c
> +++ b/drivers/cpufreq/cpufreq_conservative.c
> @@ -80,7 +80,7 @@ struct cpu_dbs_info_s {
> int cpu;
> unsigned int enable:1;
> };
> -static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info);
> +static DEFINE_PER_CPU(struct cpu_dbs_info_s, cs_cpu_dbs_info);
>
> static unsigned int dbs_enable; /* number of CPUs using this policy */
>
> @@ -150,7 +150,7 @@ dbs_cpufreq_notifier(struct notifier_block *nb, unsigned long val,
> void *data)
> {
> struct cpufreq_freqs *freq = data;
> - struct cpu_dbs_info_s *this_dbs_info = &per_cpu(cpu_dbs_info,
> + struct cpu_dbs_info_s *this_dbs_info = &per_cpu(cs_cpu_dbs_info,
> freq->cpu);
>
> struct cpufreq_policy *policy;
> @@ -323,7 +323,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,
> /* we need to re-evaluate prev_cpu_idle */
> for_each_online_cpu(j) {
> struct cpu_dbs_info_s *dbs_info;
> - dbs_info = &per_cpu(cpu_dbs_info, j);
> + dbs_info = &per_cpu(cs_cpu_dbs_info, j);
> dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
> &dbs_info->prev_cpu_wall);
> if (dbs_tuners_ins.ignore_nice)
> @@ -413,7 +413,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)
> cputime64_t cur_wall_time, cur_idle_time;
> unsigned int idle_time, wall_time;
>
> - j_dbs_info = &per_cpu(cpu_dbs_info, j);
> + j_dbs_info = &per_cpu(cs_cpu_dbs_info, j);
>
> cur_idle_time = get_cpu_idle_time(j, &cur_wall_time);
>
> @@ -553,7 +553,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
> unsigned int j;
> int rc;
>
> - this_dbs_info = &per_cpu(cpu_dbs_info, cpu);
> + this_dbs_info = &per_cpu(cs_cpu_dbs_info, cpu);
>
> switch (event) {
> case CPUFREQ_GOV_START:
> @@ -573,7 +573,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
>
> for_each_cpu(j, policy->cpus) {
> struct cpu_dbs_info_s *j_dbs_info;
> - j_dbs_info = &per_cpu(cpu_dbs_info, j);
> + j_dbs_info = &per_cpu(cs_cpu_dbs_info, j);
> j_dbs_info->cur_policy = policy;
>
> j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
> diff --git a/drivers/cpufreq/cpufreq_ondemand.c b/drivers/cpufreq/cpufreq_ondemand.c
> index 338f428..2eaf88f 100644
> --- a/drivers/cpufreq/cpufreq_ondemand.c
> +++ b/drivers/cpufreq/cpufreq_ondemand.c
> @@ -87,7 +87,7 @@ struct cpu_dbs_info_s {
> unsigned int enable:1,
> sample_type:1;
> };
> -static DEFINE_PER_CPU(struct cpu_dbs_info_s, cpu_dbs_info);
> +static DEFINE_PER_CPU(struct cpu_dbs_info_s, od_cpu_dbs_info);
>
> static unsigned int dbs_enable; /* number of CPUs using this policy */
>
> @@ -162,7 +162,8 @@ static unsigned int powersave_bias_target(struct cpufreq_policy *policy,
> unsigned int freq_hi, freq_lo;
> unsigned int index = 0;
> unsigned int jiffies_total, jiffies_hi, jiffies_lo;
> - struct cpu_dbs_info_s *dbs_info = &per_cpu(cpu_dbs_info, policy->cpu);
> + struct cpu_dbs_info_s *dbs_info = &per_cpu(od_cpu_dbs_info,
> + policy->cpu);
>
> if (!dbs_info->freq_table) {
> dbs_info->freq_lo = 0;
> @@ -207,7 +208,7 @@ static void ondemand_powersave_bias_init(void)
> {
> int i;
> for_each_online_cpu(i) {
> - struct cpu_dbs_info_s *dbs_info = &per_cpu(cpu_dbs_info, i);
> + struct cpu_dbs_info_s *dbs_info = &per_cpu(od_cpu_dbs_info, i);
> dbs_info->freq_table = cpufreq_frequency_get_table(i);
> dbs_info->freq_lo = 0;
> }
> @@ -322,7 +323,7 @@ static ssize_t store_ignore_nice_load(struct cpufreq_policy *policy,
> /* we need to re-evaluate prev_cpu_idle */
> for_each_online_cpu(j) {
> struct cpu_dbs_info_s *dbs_info;
> - dbs_info = &per_cpu(cpu_dbs_info, j);
> + dbs_info = &per_cpu(od_cpu_dbs_info, j);
> dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
> &dbs_info->prev_cpu_wall);
> if (dbs_tuners_ins.ignore_nice)
> @@ -416,7 +417,7 @@ static void dbs_check_cpu(struct cpu_dbs_info_s *this_dbs_info)
> unsigned int load, load_freq;
> int freq_avg;
>
> - j_dbs_info = &per_cpu(cpu_dbs_info, j);
> + j_dbs_info = &per_cpu(od_cpu_dbs_info, j);
>
> cur_idle_time = get_cpu_idle_time(j, &cur_wall_time);
>
> @@ -573,7 +574,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
> unsigned int j;
> int rc;
>
> - this_dbs_info = &per_cpu(cpu_dbs_info, cpu);
> + this_dbs_info = &per_cpu(od_cpu_dbs_info, cpu);
>
> switch (event) {
> case CPUFREQ_GOV_START:
> @@ -595,7 +596,7 @@ static int cpufreq_governor_dbs(struct cpufreq_policy *policy,
>
> for_each_cpu(j, policy->cpus) {
> struct cpu_dbs_info_s *j_dbs_info;
> - j_dbs_info = &per_cpu(cpu_dbs_info, j);
> + j_dbs_info = &per_cpu(od_cpu_dbs_info, j);
> j_dbs_info->cur_policy = policy;
>
> j_dbs_info->prev_cpu_idle = get_cpu_idle_time(j,
> diff --git a/drivers/xen/events.c b/drivers/xen/events.c
> index 30963af..4dbe5c0 100644
> --- a/drivers/xen/events.c
> +++ b/drivers/xen/events.c
> @@ -596,6 +596,8 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id)
> return IRQ_HANDLED;
> }
>
> +static DEFINE_PER_CPU(unsigned, xed_nesting_count);
> +
> /*
> * Search the CPUs pending events bitmasks. For each one found, map
> * the event number to an irq, and feed it into do_IRQ() for
> @@ -611,7 +613,6 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
> struct pt_regs *old_regs = set_irq_regs(regs);
> struct shared_info *s = HYPERVISOR_shared_info;
> struct vcpu_info *vcpu_info = __get_cpu_var(xen_vcpu);
> - static DEFINE_PER_CPU(unsigned, nesting_count);
> unsigned count;
>
> exit_idle();
> @@ -622,7 +623,7 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
>
> vcpu_info->evtchn_upcall_pending = 0;
>
> - if (__get_cpu_var(nesting_count)++)
> + if (__get_cpu_var(xed_nesting_count)++)
> goto out;
>
> #ifndef CONFIG_X86 /* No need for a barrier -- XCHG is a barrier on x86. */
> @@ -647,8 +648,8 @@ void xen_evtchn_do_upcall(struct pt_regs *regs)
>
> BUG_ON(!irqs_disabled());
>
> - count = __get_cpu_var(nesting_count);
> - __get_cpu_var(nesting_count) = 0;
> + count = __get_cpu_var(xed_nesting_count);
> + __get_cpu_var(xed_nesting_count) = 0;
> } while(count != 1);
>
> out:
> diff --git a/mm/page-writeback.c b/mm/page-writeback.c
> index bb553c3..0e0c9de 100644
> --- a/mm/page-writeback.c
> +++ b/mm/page-writeback.c
> @@ -606,6 +606,8 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite)
> }
> }
>
> +static DEFINE_PER_CPU(unsigned long, bdp_ratelimits) = 0;
> +
> /**
> * balance_dirty_pages_ratelimited_nr - balance dirty memory state
> * @mapping: address_space which was dirtied
> @@ -623,7 +625,6 @@ void set_page_dirty_balance(struct page *page, int page_mkwrite)
> void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
> unsigned long nr_pages_dirtied)
> {
> - static DEFINE_PER_CPU(unsigned long, ratelimits) = 0;
> unsigned long ratelimit;
> unsigned long *p;
>
> @@ -636,7 +637,7 @@ void balance_dirty_pages_ratelimited_nr(struct address_space *mapping,
> * tasks in balance_dirty_pages(). Period.
> */
> preempt_disable();
> - p = &__get_cpu_var(ratelimits);
> + p = &__get_cpu_var(bdp_ratelimits);
> *p += nr_pages_dirtied;
> if (unlikely(*p >= ratelimit)) {
> *p = 0;
> diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c
> index b35a950..70ee18c 100644
> --- a/net/ipv4/syncookies.c
> +++ b/net/ipv4/syncookies.c
> @@ -37,12 +37,12 @@ __initcall(init_syncookies);
> #define COOKIEBITS 24 /* Upper bits store count */
> #define COOKIEMASK (((__u32)1 << COOKIEBITS) - 1)
>
> -static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
> +static DEFINE_PER_CPU(__u32, ipv4_cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
>
> static u32 cookie_hash(__be32 saddr, __be32 daddr, __be16 sport, __be16 dport,
> u32 count, int c)
> {
> - __u32 *tmp = __get_cpu_var(cookie_scratch);
> + __u32 *tmp = __get_cpu_var(ipv4_cookie_scratch);
>
> memcpy(tmp + 4, syncookie_secret[c], sizeof(syncookie_secret[c]));
> tmp[0] = (__force u32)saddr;
> diff --git a/net/ipv6/syncookies.c b/net/ipv6/syncookies.c
> index 711175e..348e38c 100644
> --- a/net/ipv6/syncookies.c
> +++ b/net/ipv6/syncookies.c
> @@ -74,12 +74,12 @@ static inline struct sock *get_cookie_sock(struct sock *sk, struct sk_buff *skb,
> return child;
> }
>
> -static DEFINE_PER_CPU(__u32, cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
> +static DEFINE_PER_CPU(__u32, ipv6_cookie_scratch)[16 + 5 + SHA_WORKSPACE_WORDS];
>
> static u32 cookie_hash(struct in6_addr *saddr, struct in6_addr *daddr,
> __be16 sport, __be16 dport, u32 count, int c)
> {
> - __u32 *tmp = __get_cpu_var(cookie_scratch);
> + __u32 *tmp = __get_cpu_var(ipv6_cookie_scratch);
>
> /*
> * we have 320 bits of information to hash, copy in the remaining
> --
> 1.6.0.2
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
--
Jens Axboe
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 3/7] percpu: clean up percpu variable definitions
2009-05-20 7:37 ` [PATCH 3/7] percpu: clean up percpu variable definitions Tejun Heo
2009-05-20 9:17 ` Jens Axboe
@ 2009-05-25 6:07 ` Rusty Russell
2009-05-25 16:07 ` Tejun Heo
1 sibling, 1 reply; 19+ messages in thread
From: Rusty Russell @ 2009-05-25 6:07 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, Jens Axboe,
Dave Jones, Jeremy Fitzhardinge, linux-mm
On Wed, 20 May 2009 05:07:35 pm Tejun Heo wrote:
> Percpu variable definition is about to be updated such that
>
> * percpu symbols must be unique even the static ones
>
> * in-function static definition is not allowed
That spluttering noise is be choking on the title of this patch :)
Making these pseudo statics is in no way a cleanup. How about we just
say "they can't be static" and do something like:
/* Sorry, can't be static: that breaks archs which need these weak. */
#define DEFINE_PER_CPU(type, var) \
extern typeof(type) var; DEFINE_PER_CPU_SECTION(type, name, "")
Thanks,
Rusty.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 3/7] percpu: clean up percpu variable definitions
2009-05-25 6:07 ` Rusty Russell
@ 2009-05-25 16:07 ` Tejun Heo
0 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-25 16:07 UTC (permalink / raw)
To: Rusty Russell
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, Jens Axboe,
Dave Jones, Jeremy Fitzhardinge, linux-mm
Rusty Russell wrote:
> On Wed, 20 May 2009 05:07:35 pm Tejun Heo wrote:
>> Percpu variable definition is about to be updated such that
>>
>> * percpu symbols must be unique even the static ones
>>
>> * in-function static definition is not allowed
>
> That spluttering noise is be choking on the title of this patch :)
>
> Making these pseudo statics is in no way a cleanup. How about we just
> say "they can't be static" and do something like:
>
> /* Sorry, can't be static: that breaks archs which need these weak. */
> #define DEFINE_PER_CPU(type, var) \
> extern typeof(type) var; DEFINE_PER_CPU_SECTION(type, name, "")
Heh... well, even though I authored the patch, I kind of agree with
you. Maybe it would be better to simply disallow static declaration /
definition at all. I wanted to give a go at the original idea as it
seemed to have some potential. The result isn't too disappointing but
I can't really say there are distinctively compelling advantages to
justify the added complexity and subtlety.
What do others think? Is everyone happy with going extern only?
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 4/7] percpu: enforce global uniqueness and disallow in-function statics
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (2 preceding siblings ...)
2009-05-20 7:37 ` [PATCH 3/7] percpu: clean up percpu variable definitions Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 7:37 ` [PATCH 5/7] alpha: kill unnecessary __used attribute in PER_CPU_ATTRIBUTES Tejun Heo
` (5 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo
Some archs (alpha and s390) need to add 'weak' attribute to percpu
variable definitions so that the compiler generates external
references for them. To allow this, all percpu variables should be
unique regardless of scope and in-function static definitions can't be
allowed.
This patch updates DECLARE_PER_CPU_SECTION() and
DEFINE_PER_CPU_SECTION() such that percpu symbol itself is always
defined global so that weak can be added via PER_CPU_ATTRIBUTES. The
scope and uniqueness rules are enforced using two dummy variables,
both of which are thrown away during linking. The multiline
definition combined with section specification prevents in-function
static definitions.
This patch is inspired by Ivan Kokshaysky's alpha percpu patch.
[ Impact: stricter rules for percpu variable definitions ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: David Howells <dhowells@redhat.com>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
---
include/linux/percpu-defs.h | 34 +++++++++++++++++++++++++---------
1 files changed, 25 insertions(+), 9 deletions(-)
diff --git a/include/linux/percpu-defs.h b/include/linux/percpu-defs.h
index 8f921d7..6ff4fc1 100644
--- a/include/linux/percpu-defs.h
+++ b/include/linux/percpu-defs.h
@@ -10,21 +10,37 @@
/*
* Base implementations of per-CPU variable declarations and definitions, where
* the section in which the variable is to be placed is provided by the
- * 'section' argument. This may be used to affect the parameters governing the
+ * 'sec' argument. This may be used to affect the parameters governing the
* variable's storage.
*
+ * Percpu variable declaration and definition decouple scope (static or extern)
+ * specification from the actual percpu variable and use two dummy variables
+ * to ensure DECLARE_PER_CPU_SECTION() is not coupled with static definition
+ * and each percpu symbol is unique regardless of scope. This is for archs
+ * which need to add 'weak' attribute to per cpu variables to force the
+ * compiler to generate external references for percpu variables (alpha and
+ * s390). This also means that in-function static percpu variables aren't
+ * allowed.
+ *
* NOTE! The sections for the DECLARE and for the DEFINE must match, lest
* linkage errors occur due the compiler generating the wrong code to access
* that section.
*/
-#define DECLARE_PER_CPU_SECTION(type, name, section) \
- extern \
- __attribute__((__section__(PER_CPU_BASE_SECTION section))) \
- PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
-
-#define DEFINE_PER_CPU_SECTION(type, name, section) \
- __attribute__((__section__(PER_CPU_BASE_SECTION section))) \
- PER_CPU_ATTRIBUTES __typeof__(type) per_cpu__##name
+#define __PCPU_ATTRS(sec) \
+ __attribute__((section(PER_CPU_BASE_SECTION sec))) \
+ PER_CPU_ATTRIBUTES
+
+#define __PCPU_DUMMY_ATTRS \
+ __attribute__((section(".discard"), unused))
+
+#define DECLARE_PER_CPU_SECTION(type, name, sec) \
+ extern __PCPU_DUMMY_ATTRS char __pcpu_scope_##name; \
+ extern __PCPU_ATTRS(sec) __typeof__(type) per_cpu__##name
+
+#define DEFINE_PER_CPU_SECTION(type, name, sec) \
+ __PCPU_DUMMY_ATTRS char __pcpu_scope_##name; \
+ __PCPU_DUMMY_ATTRS char __pcpu_unique_##name; \
+ __PCPU_ATTRS(sec) __typeof__(type) per_cpu__##name
/*
* Variant on the per-CPU variable declaration/definition theme used for
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 5/7] alpha: kill unnecessary __used attribute in PER_CPU_ATTRIBUTES
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (3 preceding siblings ...)
2009-05-20 7:37 ` [PATCH 4/7] percpu: enforce global uniqueness and disallow in-function statics Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 7:37 ` [PATCH 6/7] alpha: switch to dynamic percpu allocator Tejun Heo
` (4 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo
With the previous percpu variable definition change, all percpu
variables are global and there's no need to specify __used, which only
triggers on recent compilers anyway. Kill it.
[ Impact: remove unnecessary percpu attribute ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Richard Henderson <rth@twiddle.net>
---
arch/alpha/include/asm/percpu.h | 5 -----
1 files changed, 0 insertions(+), 5 deletions(-)
diff --git a/arch/alpha/include/asm/percpu.h b/arch/alpha/include/asm/percpu.h
index 06c5c7a..7f0a9c4 100644
--- a/arch/alpha/include/asm/percpu.h
+++ b/arch/alpha/include/asm/percpu.h
@@ -30,7 +30,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS];
#ifndef MODULE
#define SHIFT_PERCPU_PTR(var, offset) RELOC_HIDE(&per_cpu_var(var), (offset))
-#define PER_CPU_ATTRIBUTES
#else
/*
* To calculate addresses of locally defined variables, GCC uses 32-bit
@@ -49,8 +48,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS];
: "=&r"(__ptr), "=&r"(tmp_gp)); \
(typeof(&per_cpu_var(var)))(__ptr + (offset)); })
-#define PER_CPU_ATTRIBUTES __used
-
#endif /* MODULE */
/*
@@ -71,8 +68,6 @@ extern unsigned long __per_cpu_offset[NR_CPUS];
#define __get_cpu_var(var) per_cpu_var(var)
#define __raw_get_cpu_var(var) per_cpu_var(var)
-#define PER_CPU_ATTRIBUTES
-
#endif /* SMP */
#ifdef CONFIG_SMP
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 6/7] alpha: switch to dynamic percpu allocator
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (4 preceding siblings ...)
2009-05-20 7:37 ` [PATCH 5/7] alpha: kill unnecessary __used attribute in PER_CPU_ATTRIBUTES Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 7:37 ` [PATCH 7/7] s390: " Tejun Heo
` (3 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo
Alpha implements custom SHIFT_PERCPU_PTR for modules because percpu
area can be located far away from the 4G area where the module text is
located. The custom SHIFT_PERCPU_PTR forces GOT usage using ldq
instruction with literal relocation; however, the relocation can't be
used with dynamically allocatd percpu variables. Fortunately, similar
result can be achieved using weak attribute on percpu variable
declarations and definitions, which is allowed with previous changes.
This patch makes alpha use weak attribute instead and switch to
dynamic percpu allocator.
asm/tlbflush.h was getting linux/sched.h via asm/percpu.h which no
longer needs it. Include linux/sched.h directly in asm/tlbflush.h.
Compile tested. Generation of litereal relocation verified.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: use dynamic percpu allocator ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Richard Henderson <rth@twiddle.net>
---
arch/alpha/Kconfig | 3 -
arch/alpha/include/asm/percpu.h | 96 ++++---------------------------------
arch/alpha/include/asm/tlbflush.h | 1 +
3 files changed, 10 insertions(+), 90 deletions(-)
diff --git a/arch/alpha/Kconfig b/arch/alpha/Kconfig
index 05d8640..9fb8aae 100644
--- a/arch/alpha/Kconfig
+++ b/arch/alpha/Kconfig
@@ -70,9 +70,6 @@ config AUTO_IRQ_AFFINITY
depends on SMP
default y
-config HAVE_LEGACY_PER_CPU_AREA
- def_bool y
-
source "init/Kconfig"
source "kernel/Kconfig.freezer"
diff --git a/arch/alpha/include/asm/percpu.h b/arch/alpha/include/asm/percpu.h
index 7f0a9c4..2b0c79c 100644
--- a/arch/alpha/include/asm/percpu.h
+++ b/arch/alpha/include/asm/percpu.h
@@ -1,97 +1,19 @@
#ifndef __ALPHA_PERCPU_H
#define __ALPHA_PERCPU_H
-#include <linux/compiler.h>
-#include <linux/threads.h>
-#include <linux/percpu-defs.h>
-
/*
- * Determine the real variable name from the name visible in the
- * kernel sources.
- */
-#define per_cpu_var(var) per_cpu__##var
-
-#ifdef CONFIG_SMP
-
-/*
- * per_cpu_offset() is the offset that has to be added to a
- * percpu variable to get to the instance for a certain processor.
- */
-extern unsigned long __per_cpu_offset[NR_CPUS];
-
-#define per_cpu_offset(x) (__per_cpu_offset[x])
-
-#define __my_cpu_offset per_cpu_offset(raw_smp_processor_id())
-#ifdef CONFIG_DEBUG_PREEMPT
-#define my_cpu_offset per_cpu_offset(smp_processor_id())
-#else
-#define my_cpu_offset __my_cpu_offset
-#endif
-
-#ifndef MODULE
-#define SHIFT_PERCPU_PTR(var, offset) RELOC_HIDE(&per_cpu_var(var), (offset))
-#else
-/*
- * To calculate addresses of locally defined variables, GCC uses 32-bit
- * displacement from the GP. Which doesn't work for per cpu variables in
- * modules, as an offset to the kernel per cpu area is way above 4G.
+ * To calculate addresses of locally defined variables, GCC uses
+ * 32-bit displacement from the GP. Which doesn't work for per cpu
+ * variables in modules, as an offset to the kernel per cpu area is
+ * way above 4G.
*
- * This forces allocation of a GOT entry for per cpu variable using
- * ldq instruction with a 'literal' relocation.
- */
-#define SHIFT_PERCPU_PTR(var, offset) ({ \
- extern int simple_identifier_##var(void); \
- unsigned long __ptr, tmp_gp; \
- asm ( "br %1, 1f \n\
- 1: ldgp %1, 0(%1) \n\
- ldq %0, per_cpu__" #var"(%1)\t!literal" \
- : "=&r"(__ptr), "=&r"(tmp_gp)); \
- (typeof(&per_cpu_var(var)))(__ptr + (offset)); })
-
-#endif /* MODULE */
-
-/*
- * A percpu variable may point to a discarded regions. The following are
- * established ways to produce a usable pointer from the percpu variable
- * offset.
+ * Use "weak" attribute to force the compiler to generate external
+ * reference.
*/
-#define per_cpu(var, cpu) \
- (*SHIFT_PERCPU_PTR(var, per_cpu_offset(cpu)))
-#define __get_cpu_var(var) \
- (*SHIFT_PERCPU_PTR(var, my_cpu_offset))
-#define __raw_get_cpu_var(var) \
- (*SHIFT_PERCPU_PTR(var, __my_cpu_offset))
-
-#else /* ! SMP */
-
-#define per_cpu(var, cpu) (*((void)(cpu), &per_cpu_var(var)))
-#define __get_cpu_var(var) per_cpu_var(var)
-#define __raw_get_cpu_var(var) per_cpu_var(var)
-
-#endif /* SMP */
-
-#ifdef CONFIG_SMP
-#define PER_CPU_BASE_SECTION ".data.percpu"
-#else
-#define PER_CPU_BASE_SECTION ".data"
-#endif
-
-#ifdef CONFIG_SMP
-
-#ifdef MODULE
-#define PER_CPU_SHARED_ALIGNED_SECTION ""
-#else
-#define PER_CPU_SHARED_ALIGNED_SECTION ".shared_aligned"
-#endif
-#define PER_CPU_FIRST_SECTION ".first"
-
-#else
-
-#define PER_CPU_SHARED_ALIGNED_SECTION ""
-#define PER_CPU_FIRST_SECTION ""
-
+#if defined(MODULE) && defined(CONFIG_SMP)
+#define PER_CPU_ATTRIBUTES __attribute__((weak))
#endif
-#define PER_CPU_ATTRIBUTES
+#include <asm-generic/percpu.h>
#endif /* __ALPHA_PERCPU_H */
diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h
index 9d87aaa..e89e0c2 100644
--- a/arch/alpha/include/asm/tlbflush.h
+++ b/arch/alpha/include/asm/tlbflush.h
@@ -2,6 +2,7 @@
#define _ALPHA_TLBFLUSH_H
#include <linux/mm.h>
+#include <linux/sched.h>
#include <asm/compiler.h>
#include <asm/pgalloc.h>
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 7/7] s390: switch to dynamic percpu allocator
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (5 preceding siblings ...)
2009-05-20 7:37 ` [PATCH 6/7] alpha: switch to dynamic percpu allocator Tejun Heo
@ 2009-05-20 7:37 ` Tejun Heo
2009-05-20 8:41 ` [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Martin Schwidefsky
` (2 subsequent siblings)
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-20 7:37 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Cc: Tejun Heo
64bit s390 shares the same problem with percpu symbol addressing from
modules. It needs assembly magic to force GOTENT reference when
building module as the percpu address will be outside the usual 4G
range from the module text. Simiarly to alpha, this can be solved by
using the weak attribute.
This patch makes s390 use weak attribute instead and switch to dynamic
percpu allocator. Please note that weak attribute is not added if
!SMP as percpu variables behave exactly the same as normal variables
on UP.
Compile tested. Generation of GOTENT reference verified.
This patch is based on Ivan Kokshaysky's alpha percpu patch.
[ Impact: use dynamic percpu allocator ]
Signed-off-by: Tejun Heo <tj@kernel.org>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
---
arch/s390/Kconfig | 3 ---
arch/s390/include/asm/percpu.h | 32 ++++++++------------------------
2 files changed, 8 insertions(+), 27 deletions(-)
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 686909a..2eca5fe 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -75,9 +75,6 @@ config VIRT_CPU_ACCOUNTING
config ARCH_SUPPORTS_DEBUG_PAGEALLOC
def_bool y
-config HAVE_LEGACY_PER_CPU_AREA
- def_bool y
-
mainmenu "Linux Kernel Configuration"
config S390
diff --git a/arch/s390/include/asm/percpu.h b/arch/s390/include/asm/percpu.h
index 408d60b..36672ff 100644
--- a/arch/s390/include/asm/percpu.h
+++ b/arch/s390/include/asm/percpu.h
@@ -1,37 +1,21 @@
#ifndef __ARCH_S390_PERCPU__
#define __ARCH_S390_PERCPU__
-#include <linux/compiler.h>
-#include <asm/lowcore.h>
-
/*
* s390 uses its own implementation for per cpu data, the offset of
* the cpu local data area is cached in the cpu's lowcore memory.
- * For 64 bit module code s390 forces the use of a GOT slot for the
- * address of the per cpu variable. This is needed because the module
- * may be more than 4G above the per cpu area.
*/
-#if defined(__s390x__) && defined(MODULE)
-
-#define SHIFT_PERCPU_PTR(ptr,offset) (({ \
- extern int simple_identifier_##var(void); \
- unsigned long *__ptr; \
- asm ( "larl %0, %1@GOTENT" \
- : "=a" (__ptr) : "X" (ptr) ); \
- (typeof(ptr))((*__ptr) + (offset)); }))
-
-#else
-
-#define SHIFT_PERCPU_PTR(ptr, offset) (({ \
- extern int simple_identifier_##var(void); \
- unsigned long __ptr; \
- asm ( "" : "=a" (__ptr) : "0" (ptr) ); \
- (typeof(ptr)) (__ptr + (offset)); }))
+#define __my_cpu_offset S390_lowcore.percpu_offset
+/*
+ * For 64 bit module code, the module may be more than 4G above the
+ * per cpu area, use "weak" attribute to force the compiler to
+ * generate an external reference.
+ */
+#if defined(CONFIG_SMP) && defined(__s390x__) && defined(MODULE)
+#define PER_CPU_ATTRIBUTES __attribute__((weak))
#endif
-#define __my_cpu_offset S390_lowcore.percpu_offset
-
#include <asm-generic/percpu.h>
#endif /* __ARCH_S390_PERCPU__ */
--
1.6.0.2
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (6 preceding siblings ...)
2009-05-20 7:37 ` [PATCH 7/7] s390: " Tejun Heo
@ 2009-05-20 8:41 ` Martin Schwidefsky
2009-05-20 13:49 ` Ivan Kokshaysky
2009-05-21 1:21 ` Tejun Heo
9 siblings, 0 replies; 19+ messages in thread
From: Martin Schwidefsky @ 2009-05-20 8:41 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, heiko.carstens, lethal,
davem, jdike, chris, rusty
On Wed, 20 May 2009 16:37:32 +0900
Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> Please do NOT pull before maintainers of alpha and s390 ack. Upon
> ack, please pull from the following git tree.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu
>
> This patchset is combination of the last patch of
> percpu-use-dynamic-percpu-allocator-as-default patchset[1] and patches
> inspired by alpha conversion patch[2] posted by Ivan Kokshaysky.
>
> 0001-percpu-use-dynamic-percpu-allocator-as-the-default.patch
> 0002-linker-script-throw-away-.discard-section.patch
> 0003-percpu-clean-up-percpu-variable-definitions.patch
> 0004-percpu-enforce-global-uniqueness-and-disallow-in-fu.patch
> 0005-alpha-kill-unnecessary-__used-attribute-in-PER_CPU_.patch
> 0006-alpha-switch-to-dynamic-percpu-allocator.patch
> 0007-s390-switch-to-dynamic-percpu-allocator.patch
>
> 0001 is mostly unchanged except for added Acked-by's.
>
> 0002-0003 prepare for percpu variable definition update - .discard is
> thrown away on all archs when linking kernel and modules, static
> percpu variables in functions are moved outside of the function and
> all static percpu variables are given unique names.
>
> 0004 updates percpu variable definition such that all percpu variables
> in the same link unit have unique names regardless of its scope
> (static or global) and disallow in-function static definitions. The
> new definition uses dummy variables in .discard to enforce the
> declaration/definition semantics and always defines percpu variables
> as globals. This allows archs to add 'weak' attribute if necessary.
>
> 0005-0006 converts alpha to use 'weak' attribute instead of inline
> assembly to generate GOT based reference when referencing percpu
> variables from modules, which in turn allows conversion to dynamic
> percpu allocator.
>
> 0007 does the same for s390.
>
> The new DECLARE/DEFINE macros implement the following behaviors.
>
> a. DECLARE + static DEFINE -> compile error
>
> b. multiple non-static DEFINEs with the same name -> compile error
>
> c. multiple static/global DEFINEs with the same name -> compile error
>
> d. in function static DEFINEs -> compile error
>
> #a and #b are expected behaviors on DECLARE/DEFINE macros in general.
> #c and #d are necessary to allow 'weak' attribute on certain archs.
> Please note that the rules are applied consistently whether 'weak'
> attribute is specified or not. This eases catching mistakes and makes
> things less confusing.
>
> Test config boots and works fine on x86_64. alpha and s390 are
> compile tested and verified to generate necessary code for external
> references but I don't have any way to actually verify them, so please
> test them. :-)
>
> all-yes-config on x86_64 is building now. Ergggh... I want more
> memory. Okay, completed successfully.
Just tested the patch-series on s390. Works fine:
Acked-by: Martin Schwidefsky <schwidefsky@de.ibm.com>
Kudos to Tejun and Ivan :-)
--
blue skies,
Martin.
"Reality continues to ruin my life." - Calvin.
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (7 preceding siblings ...)
2009-05-20 8:41 ` [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Martin Schwidefsky
@ 2009-05-20 13:49 ` Ivan Kokshaysky
2009-05-21 1:21 ` Tejun Heo
9 siblings, 0 replies; 19+ messages in thread
From: Ivan Kokshaysky @ 2009-05-20 13:49 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, linux-kernel, x86, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
On Wed, May 20, 2009 at 04:37:32PM +0900, Tejun Heo wrote:
> Test config boots and works fine on x86_64. alpha and s390 are
> compile tested and verified to generate necessary code for external
> references but I don't have any way to actually verify them, so please
> test them. :-)
Works fine on alpha, thanks!
Acked-by: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Ivan.
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu
2009-05-20 7:37 [PATCHSET core/percpu] percpu: convert most archs to dynamic percpu Tejun Heo
` (8 preceding siblings ...)
2009-05-20 13:49 ` Ivan Kokshaysky
@ 2009-05-21 1:21 ` Tejun Heo
9 siblings, 0 replies; 19+ messages in thread
From: Tejun Heo @ 2009-05-21 1:21 UTC (permalink / raw)
To: mingo, linux-kernel, x86, ink, rth, linux, hskinnemoen, cooloney,
starvik, jesper.nilsson, dhowells, ysato, tony.luck, takata,
geert, monstr, ralf, kyle, benh, paulus, schwidefsky,
heiko.carstens, lethal, davem, jdike, chris, rusty
Tejun Heo wrote:
> Hello,
>
> Please do NOT pull before maintainers of alpha and s390 ack. Upon
> ack, please pull from the following git tree.
>
> git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu
Ingo, I added Martin's and Ivan's acks. Please go ahead and pull into
core/percpu.
Thanks.
--
tejun
^ permalink raw reply [flat|nested] 19+ messages in thread