* Re: [PATCH v2 2/2] perf tools: Make Power7 events available for perf
From: Vince Weaver @ 2013-06-25 16:46 UTC (permalink / raw)
To: Runzhen Wang
Cc: acme, vincent.weaver, Peter Zijlstra, linux-kernel,
Stephane Eranian, xiaoguangrong, paulus, sukadev, linuxppc-dev,
mingo
In-Reply-To: <1372170933-4538-3-git-send-email-runzhen@linux.vnet.ibm.com>
On Tue, 25 Jun 2013, Runzhen Wang wrote:
> This patch makes all the POWER7 events available in sysfs.
>
> ...
>
> $ size arch/powerpc/perf/power7-pmu.o
> text data bss dec hex filename
> 3073 2720 0 5793 16a1 arch/powerpc/perf/power7-pmu.o
>
> and after the patch is applied, it is:
>
> $ size arch/powerpc/perf/power7-pmu.o
> text data bss dec hex filename
> 15950 31112 0 47062 b7d6 arch/powerpc/perf/power7-pmu.o
So if I'm reading this right, there's 45k of overhead for just one cpu
type?
What happens if we do this on x86?
If we have similar for p6/p4/core2/nehalem/ivb/snb/amd10h/amd15h/amd16h/knb
that's 450k of event defintions in the kernel. And may I remind everyone
that you can't compile perf_event support as a module, nor can you
unconfigure it on x86 (it's always built in, no option to disable).
I'd like to repeat my unpopular position that we just link perf against
libpfm4 and keep event tables in userspace where they belong.
Vince
^ permalink raw reply
* BUG: no PCI/PCIe devices found in 85xx architecture
From: Stefani Seibold @ 2013-06-25 18:40 UTC (permalink / raw)
To: linuxppc-dev
Hi,
there is a bug in kernel 3.9 which the new fsl_pci platform driver. The
pcibios_init in pci_32.c will be called before the platform driver probe
will be invoked.
The call order for a p2020 board with linux 3.9 is currently:
fsl_pci_init
pcibios_init
fsl_pci_probe
fsl_pci_probe
fsl_pci_probe
Therefore the PCI/PCIe bridge will be added after the PCI/PCIe busses
was scanned for devices. So no PCI/PCIe devices are available.
Everything works fine by reverting the fsl_pci.[ch] to the version in
linux 3.4, because the PCI/PCIe bridges will be added in
the ..._setup_arch() function, before the pcibios_init function is
called.
Any solution for this issue?
- Stefani
^ permalink raw reply
* Re: Inbound PCI and Memory Corruption
From: Peter LaDow @ 2013-06-25 18:44 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev@lists.ozlabs.org
In-Reply-To: <1372036573.3944.115.camel@pasglop>
On Sat, Jun 22, 2013 at 5:00 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> Afaik e300 is slightly out of order, maybe it's missing a memory barrier
> somewhere.... One thing to try is to add some to the dma_map/unmap ops.
I went through the driver and added memory barriers to the
dma_map_page/dma_unmap_page and dma_alloc_coherent/dma_free_coherent
calls (wmb() calls after each, which resolves to a sync instruction).
I still get a kernel panic.
I did turn on DEBUG_PAGE_ALLOC to try and get more information, but
I'm not finding anything new. However, with the SLAB debugging I do
find SLAB corruption, e.g.:
Slab corruption: fib6_nodes start=e900c7f8, len=32
Redzone: 0x9f911029d74e35b/0x30a706a6050806.
Last user: [<06040001>](0x6040001)
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b ff ff ff ff ff ff
Prev obj: start=e900c7c0, len=32
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [< (null)>](0x0)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5
Next obj: start=e900c830, len=32
Redzone: 0x30a706a6050aca/0xc8be11029d74e35b.
Last user: [< (null)>](0x0)
000: 0d aa 00 00 00 00 00 00 0a ca 0d 49 00 00 00 00
010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 75 8b
Which is clearly corrupted with ethernet frames. The only interface
connected is the e1000. Eventually this corruption leads to a kernel
panic.
I'm completely confused on how this could happen. Given the M bit is
set for all pages (see below), and with memory barriers on the DMA
map/unmap and register operations, the only thing I can think of is
something in the IO sequencer (which was suggested in the link I gave
earlier). Yet the patch mentioned is in place.
> Also audit the driver to ensure that it properly uses barriers when
> populating descriptors (and maybe compare to a more recent version of
> the driver upstream).
I've gone through the driver and didn't see anything missing. And the
upstream (v3.10-rc5) driver is the same version (7.3.21-k8-NAPI). And
I've used the latest from the e1000 release (8.0.35-NAPI), and I get
the same problem.
On Sun, Jun 23, 2013 at 6:16 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> Also dbl check that the MMU is indeed mapping all these pages with the
> "M" bit.
The DBAT's have the M bit set (both have 0x12 in the DBATxL
registers)...sometimes. Usually when I halt the CPU and dumps the
BAT's, all the IBAT's and DBAT's have zeros. But occasionally I see
DBAT2 and DBAT3 with values and the M bit set.
I also dumped all the TLB entries, and every one of them has the M bit
set (see below).
TLB dump:
BDI>dtlb 0 63
IDX V RC VSID VPI RPN WIMG PP
0: V 0C 000eee_e9a0000 -> 2e9a0000 --M- 00
1: V 0C 000eee_f401000 -> 2f401000 --M- 00
2: V 1C 000ccc_0502000 -> 00502000 --M- 00
3: V 0C 000eee_f403000 -> 2f403000 --M- 00
4: V 0C 000eee_c124000 -> 2c124000 --M- 00
5: V 0C 000eee_f405000 -> 2f405000 --M- 00
6: V 0C 000eee_e9e6000 -> 2e9e6000 --M- 00
7: V 0C 33afd1_0427000 -> 005f8000 --M- 10
8: V 0C 33afd1_0428000 -> 2ff63000 --M- 10
9: V 0C 000ccc_0349000 -> 00349000 --M- 00
10: V 1C 000ccc_03ca000 -> 003ca000 --M- 00
11: V 1C 000ccc_03cb000 -> 003cb000 --M- 00
12: V 0C 33afd1_040c000 -> 003b4000 --M- 11
13: V 0C 000eee_f40d000 -> 2f40d000 --M- 00
14: V 1C 000eee_fa8e000 -> 2fa8e000 --M- 00
15: V 0- 33afd1_034f000 -> 2e6b1000 --M- 11
16: V 0C 000eee_f470000 -> 2f470000 --M- 00
17: V 0C 33afd1_0411000 -> 2fe54000 --M- 10
18: V 0C 000eee_f4b2000 -> 2f4b2000 --M- 00
19: V 1C 33eb14_8073000 -> 00462000 --M- 10
20: V 0C 000ccc_02f4000 -> 002f4000 --M- 00
21: V 0C 000eee_f415000 -> 2f415000 --M- 00
22: V 1C 000ccc_03f6000 -> 003f6000 --M- 00
23: V 0C 000ccc_02f7000 -> 002f7000 --M- 00
24: V 1C 000ccc_03f8000 -> 003f8000 --M- 00
25: V 0C 000ccc_03d9000 -> 003d9000 --M- 00
26: V 1C 33b304_a31a000 -> 007f4000 --M- 10
27: V 1C 000ccc_03fb000 -> 003fb000 --M- 00
28: V 1C 000ccc_03fc000 -> 003fc000 --M- 00
29: V 0C 000eee_f41d000 -> 2f41d000 --M- 00
30: V 1C 000eee_e87e000 -> 2e87e000 --M- 00
31: V 1C 33afd1_045f000 -> 2fe52000 --M- 10
32: V 0C 000ccc_0000000 -> 00000000 --M- 00
33: V 0C 000eee_e9a1000 -> 2e9a1000 --M- 00
34: V 1C 33b304_8022000 -> 00f44000 --M- 10
35: V 0C 000ccc_0503000 -> 00503000 --M- 00
36: V 0C 33afd1_0744000 -> 2fe17000 --M- 10
37: V 0C 000eee_c125000 -> 2c125000 --M- 00
38: V 0C 33e7e1_0406000 -> 0078e000 --M- 11
39: V 0C 000eee_e987000 -> 2e987000 --M- 00
40: V 0C 000ccc_0008000 -> 00008000 --M- 00
41: V 0C 000ccc_03c9000 -> 003c9000 --M- 00
42: V 1C 33ba7b_f8ea000 -> 005f9000 --M- 10
43: V 1C 33afd1_040b000 -> 2ffe0000 --M- 11
44: V 0C 000ccc_03cc000 -> 003cc000 --M- 00
45: V 0C 000eee_b68d000 -> 2b68d000 --M- 00
46: V 1C 000eee_f40e000 -> 2f40e000 --M- 00
47: V 0C 000eee_fa8f000 -> 2fa8f000 --M- 00
48: V 0C 33afd1_0410000 -> 2fe4a000 --M- 10
49: V 0C 000eee_f471000 -> 2f471000 --M- 00
50: V 0C 000ccc_03f2000 -> 003f2000 --M- 00
51: V 1C 000eee_f473000 -> 2f473000 --M- 00
52: V 0C 000ccc_03f4000 -> 003f4000 --M- 00
53: V 0C 000ccc_03f5000 -> 003f5000 --M- 00
54: V 1C 000eee_f456000 -> 2f456000 --M- 00
55: V 0C 000eee_d2f7000 -> 2d2f7000 --M- 00
56: V 1C 000ccc_03d8000 -> 003d8000 --M- 00
57: V 0C 000eee_e879000 -> 2e879000 --M- 00
58: V 1C 000eee_f41a000 -> 2f41a000 --M- 00
59: V 1C 000ccc_03db000 -> 003db000 --M- 00
60: V 1C 000eee_f43c000 -> 2f43c000 --M- 00
61: V 0C 000ccc_04fd000 -> 004fd000 --M- 00
62: V 1C 000eee_f43e000 -> 2f43e000 --M- 00
63: V 1C 000eee_e93f000 -> 2e93f000 --M- 00
^ permalink raw reply
* Re: [PATCH 04/45] CPU hotplug: Add infrastructure to check lacking hotplug synchronization
From: Srivatsa S. Bhat @ 2013-06-25 18:49 UTC (permalink / raw)
To: Steven Rostedt
Cc: peterz, fweisbec, linux-kernel, walken, mingo, linux-arch,
Alex Shi, vincent.guittot, Rafael J. Wysocki, xiaoguangrong,
Yasuaki Ishimatsu, wangyun, KOSAKI Motohiro, paulmck, nikunj,
linux-pm, rusty, namhyung, tglx, laijs, zhong, netdev, oleg, sbw,
tj, akpm, linuxppc-dev
In-Reply-To: <1372116363.18733.189.camel@gandalf.local.home>
On 06/25/2013 04:56 AM, Steven Rostedt wrote:
> On Sun, 2013-06-23 at 19:08 +0530, Srivatsa S. Bhat wrote:
>
>
> Just to make the code a little cleaner, can you add:
>
>> diff --git a/kernel/cpu.c b/kernel/cpu.c
>> index 860f51a..e90d9d7 100644
>> --- a/kernel/cpu.c
>> +++ b/kernel/cpu.c
>> @@ -63,6 +63,72 @@ static struct {
>> .refcount = 0,
>> };
>>
>> +#ifdef CONFIG_DEBUG_HOTPLUG_CPU
>> +
[..]
>
> static inline void atomic_reader_refcnt_inc(void)
> {
> this_cpu_inc(atomic_reader_refcnt);
> }
> static inline void atomic_reader_refcnt_dec(void)
> {
> this_cpu_dec(atomic_reader_refcnt);
> }
>
> #else
> static inline void atomic_reader_refcnt_inc(void)
> {
> }
> static inline void atomic_reader_refcnt_dec(void)
> {
> }
> #endif
>
>> +#endif
>> +
>> void get_online_cpus(void)
>> {
>> might_sleep();
>> @@ -189,13 +255,22 @@ unsigned int get_online_cpus_atomic(void)
>> * from going offline.
>> */
>> preempt_disable();
>> +
>> +#ifdef CONFIG_DEBUG_HOTPLUG_CPU
>> + this_cpu_inc(atomic_reader_refcnt);
>> +#endif
>
> Replace the #ifdef with just:
>
> atomic_reader_refcnt_inc();
>
>> return smp_processor_id();
>> }
>> EXPORT_SYMBOL_GPL(get_online_cpus_atomic);
>>
>> void put_online_cpus_atomic(void)
>> {
>> +
>> +#ifdef CONFIG_DEBUG_HOTPLUG_CPU
>> + this_cpu_dec(atomic_reader_refcnt);
>> +#endif
>
> And
>
> atomic_reader_refcnt_dec();
>
This makes the code look much better. Thank you!
I'll make that change in my v2.
Regards,
Srivatsa S. Bhat
^ permalink raw reply
* Re: [PATCH 40/45] powerpc, irq: Use GFP_ATOMIC allocations in atomic context
From: Srivatsa S. Bhat @ 2013-06-25 19:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt
Cc: peterz, fweisbec, linux-kernel, Paul Mackerras, walken, mingo,
linux-arch, vincent.guittot, xiaoguangrong, wangyun, paulmck,
nikunj, linux-pm, rusty, rostedt, Ian Munsie, namhyung, tglx,
laijs, zhong, netdev, oleg, sbw, tj, akpm, linuxppc-dev
In-Reply-To: <1372130007.3944.185.camel@pasglop>
On 06/25/2013 08:43 AM, Benjamin Herrenschmidt wrote:
> On Tue, 2013-06-25 at 12:58 +1000, Michael Ellerman wrote:
>> On Tue, Jun 25, 2013 at 12:13:04PM +1000, Benjamin Herrenschmidt wrote:
>>> On Tue, 2013-06-25 at 12:08 +1000, Michael Ellerman wrote:
>>>> We're not checking for allocation failure, which we should be.
>>>>
>>>> But this code is only used on powermac and 85xx, so it should probably
>>>> just be a TODO to fix this up to handle the failure.
>>>
>>> And what can we do if they fail ?
>>
>> Fail up the chain and not unplug the CPU presumably.
>
> BTW. Isn't Srivatsa series removing the need to stop_machine() for
> unplug ?
Yes.
That should mean we should be able to use GFP_KERNEL no ?
No, because whatever code was being executed in stop_machine() context
would still be executed with interrupts disabled. So allocations that
can sleep would continue to be forbidden in this path.
In the CPU unplug sequence, the CPU_DYING notifications (and the surrounding
code) is guaranteed to be run:
a. _on_ the CPU going offline
b. with interrupts disabled on that CPU.
My patchset will retain these guarantees even after removing stop_machine().
And these are required for the correct execution of the code in this path,
since they rely on these semantics.
So I guess I'll retain the patch as it is. Thank you!
Regards,
Srivatsa S. Bhat
^ permalink raw reply
* [PATCH v2 00/45] CPU hotplug: stop_machine()-free CPU hotplug, part 1
From: Srivatsa S. Bhat @ 2013-06-25 20:25 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, wangyun, srivatsa.bhat, netdev,
linuxppc-dev
Hi,
This patchset is a first step towards removing stop_machine() from the
CPU hotplug offline path. It introduces a set of APIs (as a replacement to
preempt_disable()/preempt_enable()) to synchronize with CPU hotplug from
atomic contexts.
The motivation behind getting rid of stop_machine() is to avoid its
ill-effects, such as performance penalties[1] and the real-time latencies it
inflicts on the system (and also things involving stop_machine() have often
been notoriously hard to debug). And in general, getting rid of stop_machine()
from CPU hotplug also greatly enhances the elegance of the CPU hotplug design
itself.
Getting rid of stop_machine() involves building the corresponding
infrastructure in the core CPU hotplug code and converting all places which
used to depend on the semantics of stop_machine() to synchronize with CPU
hotplug.
This patchset builds a first-level base infrastructure on which tree-wide
conversions can be built upon, and also includes the conversions themselves.
We certainly need a few more careful tree-sweeps to complete the conversion,
but the goal of this patchset is to introduce the core pieces and to get the
first batch of conversions in, while covering a reasonable bulk among them.
This patchset also has a debug infrastructure to help with the conversions -
with the newly introduced CONFIG_DEBUG_HOTPLUG_CPU option turned on, it
prints warnings whenever the need for a conversion is detected. Patches 4-7
build this framework. Needless to say, I'd really appreciate if people could
test kernels with this option turned on and report omissions or better yet,
send patches to contribute to this effort.
[It is to be noted that this patchset doesn't replace stop_machine() yet,
so the immediate risk in having an unconverted (or converted) call-site
is nil, since there is no major functional change involved.]
Once the conversion gets completed, we can finalize on the design of the
stop_machine() replacement and use that in the core CPU hotplug code. We have
had some discussions in the past where we debated several different
designs[2]. We'll revisit that with more ideas once this conversion gets over.
This patchset applies on current tip:master. It is also available in the
following git branch:
git://github.com/srivatsabhat/linux.git stop-mch-free-cpuhp-part1-v2
Thank you very much!
Changes in v2:
-------------
* Build-fix for !HOTPLUG_CPU case.
* Minor code cleanups, and added a few comments.
References:
----------
1. Performance difference between CPU Hotplug with and without
stop_machine():
http://article.gmane.org/gmane.linux.kernel/1435249
2. Links to discussions around alternative synchronization schemes to
replace stop_machine() in the CPU Hotplug code:
v6: http://lwn.net/Articles/538819/
v5: http://lwn.net/Articles/533553/
v4: https://lkml.org/lkml/2012/12/11/209
v3: https://lkml.org/lkml/2012/12/7/287
v2: https://lkml.org/lkml/2012/12/5/322
v1: https://lkml.org/lkml/2012/12/4/88
3. Links to previous versions of this patchset:
v1: http://lwn.net/Articles/556138/
--
Srivatsa S. Bhat (45):
CPU hotplug: Provide APIs to prevent CPU offline from atomic context
CPU hotplug: Clarify the usage of different synchronization APIs
Documentation, CPU hotplug: Recommend usage of get/put_online_cpus_atomic()
CPU hotplug: Add infrastructure to check lacking hotplug synchronization
CPU hotplug: Protect set_cpu_online() to avoid false-positives
CPU hotplug: Sprinkle debugging checks to catch locking bugs
CPU hotplug: Expose the new debug config option
CPU hotplug: Convert preprocessor macros to static inline functions
smp: Use get/put_online_cpus_atomic() to prevent CPU offline
sched/core: Use get/put_online_cpus_atomic() to prevent CPU offline
migration: Use raw_spin_lock/unlock since interrupts are already disabled
sched/fair: Use get/put_online_cpus_atomic() to prevent CPU offline
timer: Use get/put_online_cpus_atomic() to prevent CPU offline
sched/rt: Use get/put_online_cpus_atomic() to prevent CPU offline
rcu: Use get/put_online_cpus_atomic() to prevent CPU offline
tick-broadcast: Use get/put_online_cpus_atomic() to prevent CPU offline
time/clocksource: Use get/put_online_cpus_atomic() to prevent CPU offline
softirq: Use get/put_online_cpus_atomic() to prevent CPU offline
irq: Use get/put_online_cpus_atomic() to prevent CPU offline
net: Use get/put_online_cpus_atomic() to prevent CPU offline
block: Use get/put_online_cpus_atomic() to prevent CPU offline
percpu_counter: Use get/put_online_cpus_atomic() to prevent CPU offline
infiniband: ehca: Use get/put_online_cpus_atomic() to prevent CPU offline
[SCSI] fcoe: Use get/put_online_cpus_atomic() to prevent CPU offline
staging/octeon: Use get/put_online_cpus_atomic() to prevent CPU offline
x86: Use get/put_online_cpus_atomic() to prevent CPU offline
perf/x86: Use get/put_online_cpus_atomic() to prevent CPU offline
KVM: Use get/put_online_cpus_atomic() to prevent CPU offline
kvm/vmx: Use get/put_online_cpus_atomic() to prevent CPU offline
x86/xen: Use get/put_online_cpus_atomic() to prevent CPU offline
alpha/smp: Use get/put_online_cpus_atomic() to prevent CPU offline
blackfin/smp: Use get/put_online_cpus_atomic() to prevent CPU offline
cris/smp: Use get/put_online_cpus_atomic() to prevent CPU offline
hexagon/smp: Use get/put_online_cpus_atomic() to prevent CPU offline
ia64: irq, perfmon: Use get/put_online_cpus_atomic() to prevent CPU offline
ia64: smp, tlb: Use get/put_online_cpus_atomic() to prevent CPU offline
m32r: Use get/put_online_cpus_atomic() to prevent CPU offline
MIPS: Use get/put_online_cpus_atomic() to prevent CPU offline
mn10300: Use get/put_online_cpus_atomic() to prevent CPU offline
powerpc, irq: Use GFP_ATOMIC allocations in atomic context
powerpc: Use get/put_online_cpus_atomic() to prevent CPU offline
powerpc: Use get/put_online_cpus_atomic() to avoid false-positive warning
sh: Use get/put_online_cpus_atomic() to prevent CPU offline
sparc: Use get/put_online_cpus_atomic() to prevent CPU offline
tile: Use get/put_online_cpus_atomic() to prevent CPU offline
Documentation/cpu-hotplug.txt | 20 +++-
arch/alpha/kernel/smp.c | 19 ++-
arch/blackfin/mach-common/smp.c | 4 -
arch/cris/arch-v32/kernel/smp.c | 5 +
arch/hexagon/kernel/smp.c | 3 +
arch/ia64/kernel/irq_ia64.c | 15 +++
arch/ia64/kernel/perfmon.c | 8 +
arch/ia64/kernel/smp.c | 12 +-
arch/ia64/mm/tlb.c | 4 -
arch/m32r/kernel/smp.c | 16 +--
arch/mips/kernel/cevt-smtc.c | 7 +
arch/mips/kernel/smp.c | 16 +--
arch/mips/kernel/smtc.c | 12 ++
arch/mips/mm/c-octeon.c | 4 -
arch/mn10300/mm/cache-smp.c | 3 +
arch/mn10300/mm/tlb-smp.c | 17 ++-
arch/powerpc/kernel/irq.c | 9 +-
arch/powerpc/kernel/machine_kexec_64.c | 4 -
arch/powerpc/kernel/smp.c | 4 +
arch/powerpc/kvm/book3s_hv.c | 5 +
arch/powerpc/mm/mmu_context_nohash.c | 3 +
arch/powerpc/oprofile/cell/spu_profiler.c | 3 +
arch/powerpc/oprofile/cell/spu_task_sync.c | 4 +
arch/powerpc/oprofile/op_model_cell.c | 6 +
arch/sh/kernel/smp.c | 12 +-
arch/sparc/kernel/smp_64.c | 12 +-
arch/tile/kernel/module.c | 3 +
arch/tile/kernel/tlb.c | 15 +++
arch/tile/mm/homecache.c | 3 +
arch/x86/kernel/apic/io_apic.c | 21 +++-
arch/x86/kernel/cpu/mcheck/therm_throt.c | 4 -
arch/x86/kernel/cpu/perf_event_intel_uncore.c | 6 +
arch/x86/kvm/vmx.c | 13 +-
arch/x86/mm/tlb.c | 14 +--
arch/x86/xen/mmu.c | 9 +-
block/blk-softirq.c | 3 +
drivers/infiniband/hw/ehca/ehca_irq.c | 5 +
drivers/scsi/fcoe/fcoe.c | 7 +
drivers/staging/octeon/ethernet-rx.c | 3 +
include/linux/cpu.h | 29 +++++
include/linux/cpumask.h | 59 ++++++++++-
kernel/cpu.c | 138 +++++++++++++++++++++++++
kernel/irq/manage.c | 7 +
kernel/irq/proc.c | 3 +
kernel/rcutree.c | 4 +
kernel/sched/core.c | 27 ++++-
kernel/sched/fair.c | 14 ++-
kernel/sched/rt.c | 14 +++
kernel/smp.c | 52 +++++----
kernel/softirq.c | 3 +
kernel/time/clocksource.c | 5 +
kernel/time/tick-broadcast.c | 8 +
kernel/timer.c | 4 +
lib/Kconfig.debug | 8 +
lib/cpumask.c | 8 +
lib/percpu_counter.c | 10 ++
net/core/dev.c | 9 +-
virt/kvm/kvm_main.c | 8 +
58 files changed, 614 insertions(+), 129 deletions(-)
Regards,
Srivatsa S. Bhat
IBM Linux Technology Center
^ permalink raw reply
* [PATCH v2 01/45] CPU hotplug: Provide APIs to prevent CPU offline from atomic context
From: Srivatsa S. Bhat @ 2013-06-25 20:25 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, Rafael J. Wysocki, sbw, Yasuaki Ishimatsu,
wangyun, Srivatsa S. Bhat, netdev, Tejun Heo, Thomas Gleixner,
linuxppc-dev, Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
The current CPU offline code uses stop_machine() internally. And disabling
preemption prevents stop_machine() from taking effect, thus also preventing
CPUs from going offline, as a side effect.
There are places where this side-effect of preempt_disable() (or equivalent)
is used to synchronize with CPU hotplug. Typically these are in atomic
sections of code, where they can't make use of get/put_online_cpus(), because
the latter set of APIs can sleep.
Going forward, we want to get rid of stop_machine() from the CPU hotplug
offline path. And then, with stop_machine() gone, disabling preemption will
no longer prevent CPUs from going offline.
So provide a set of APIs for such atomic hotplug readers, to prevent (any)
CPUs from going offline. For now, they will default to preempt_disable()
and preempt_enable() itself, but this will help us do the tree-wide conversion,
as a preparatory step to remove stop_machine() from CPU hotplug.
(Besides, it is good documentation as well, since it clearly marks places
where we synchronize with CPU hotplug, instead of combining it subtly with
disabling preemption).
In future, when actually removing stop_machine(), we will alter the
implementation of these APIs to a suitable synchronization scheme.
Reviewed-by: Steven Rostedt <rostedt@goodmis.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
include/linux/cpu.h | 20 ++++++++++++++++++++
kernel/cpu.c | 38 ++++++++++++++++++++++++++++++++++++++
2 files changed, 58 insertions(+)
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index 9f3c7e8..a57b25a 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -17,6 +17,8 @@
#include <linux/node.h>
#include <linux/compiler.h>
#include <linux/cpumask.h>
+#include <linux/percpu.h>
+#include <linux/smp.h>
struct device;
@@ -175,6 +177,8 @@ extern struct bus_type cpu_subsys;
extern void get_online_cpus(void);
extern void put_online_cpus(void);
+extern unsigned int get_online_cpus_atomic(void);
+extern void put_online_cpus_atomic(void);
extern void cpu_hotplug_disable(void);
extern void cpu_hotplug_enable(void);
#define hotcpu_notifier(fn, pri) cpu_notifier(fn, pri)
@@ -202,6 +206,22 @@ static inline void cpu_hotplug_driver_unlock(void)
#define put_online_cpus() do { } while (0)
#define cpu_hotplug_disable() do { } while (0)
#define cpu_hotplug_enable() do { } while (0)
+
+static inline unsigned int get_online_cpus_atomic(void)
+{
+ /*
+ * Disable preemption to avoid getting complaints from the
+ * debug_smp_processor_id() code.
+ */
+ preempt_disable();
+ return smp_processor_id();
+}
+
+static inline void put_online_cpus_atomic(void)
+{
+ preempt_enable();
+}
+
#define hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
/* These aren't inline functions due to a GCC bug. */
#define register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 198a388..2d03398 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -154,6 +154,44 @@ void cpu_hotplug_enable(void)
cpu_maps_update_done();
}
+/*
+ * get_online_cpus_atomic - Prevent any CPU from going offline
+ *
+ * Atomic hotplug readers (tasks which wish to prevent CPUs from going
+ * offline during their critical section, but can't afford to sleep)
+ * can invoke this function to synchronize with CPU offline. This function
+ * can be called recursively, provided it is matched with an equal number
+ * of calls to put_online_cpus_atomic().
+ *
+ * Note: This does NOT prevent CPUs from coming online! It only prevents
+ * CPUs from going offline.
+ *
+ * Lock ordering rule: Strictly speaking, there is no lock ordering
+ * requirement here, but it is advisable to keep the locking consistent.
+ * As a simple rule-of-thumb, use these functions in the outer-most blocks
+ * of your critical sections, outside of other locks.
+ *
+ * Returns the current CPU number, with preemption disabled.
+ */
+unsigned int get_online_cpus_atomic(void)
+{
+ /*
+ * The current CPU hotplug implementation uses stop_machine() in
+ * the CPU offline path. And disabling preemption prevents
+ * stop_machine() from taking effect. Thus, this prevents any CPU
+ * from going offline.
+ */
+ preempt_disable();
+ return smp_processor_id();
+}
+EXPORT_SYMBOL_GPL(get_online_cpus_atomic);
+
+void put_online_cpus_atomic(void)
+{
+ preempt_enable();
+}
+EXPORT_SYMBOL_GPL(put_online_cpus_atomic);
+
#else /* #if CONFIG_HOTPLUG_CPU */
static void cpu_hotplug_begin(void) {}
static void cpu_hotplug_done(void) {}
^ permalink raw reply related
* [PATCH v2 02/45] CPU hotplug: Clarify the usage of different synchronization APIs
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, Rafael J. Wysocki,
linux-kernel, rostedt, xiaoguangrong, sbw, Yasuaki Ishimatsu,
wangyun, Srivatsa S. Bhat, netdev, Thomas Gleixner, linuxppc-dev,
Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
We have quite a few APIs now which help synchronize with CPU hotplug.
Among them, get/put_online_cpus() is the oldest and the most well-known,
so no problems there. By extension, its easy to comprehend the new
set : get/put_online_cpus_atomic().
But there is yet another set, which might appear tempting to use:
cpu_hotplug_disable()/cpu_hotplug_enable(). Add comments to clarify
that this latter set is NOT for general use and must be used only in
specific cases where the requirement is really to _disable_ hotplug
and not just to synchronize with it.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/cpu.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 2d03398..860f51a 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -139,6 +139,13 @@ static void cpu_hotplug_done(void)
* the 'cpu_hotplug_disabled' flag. The same lock is also acquired by the
* hotplug path before performing hotplug operations. So acquiring that lock
* guarantees mutual exclusion from any currently running hotplug operations.
+ *
+ * Note: In most cases, this is *NOT* the function you need. If you simply
+ * want to avoid racing with CPU hotplug operations, use get/put_online_cpus()
+ * or get/put_online_cpus_atomic(), depending on the situation.
+ *
+ * This set of functions is reserved for cases where you really wish to
+ * _disable_ CPU hotplug and not just synchronize with it.
*/
void cpu_hotplug_disable(void)
{
^ permalink raw reply related
* [PATCH v2 03/45] Documentation, CPU hotplug: Recommend usage of get/put_online_cpus_atomic()
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-doc,
linux-kernel, rostedt, xiaoguangrong, sbw, wangyun,
Srivatsa S. Bhat, netdev, Rob Landley, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
So add documentation to recommend using the new get/put_online_cpus_atomic()
APIs to prevent CPUs from going offline, while invoking from atomic context.
Cc: Rob Landley <rob@landley.net>
Cc: linux-doc@vger.kernel.org
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
Documentation/cpu-hotplug.txt | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/Documentation/cpu-hotplug.txt b/Documentation/cpu-hotplug.txt
index 9f40135..7b3ca60 100644
--- a/Documentation/cpu-hotplug.txt
+++ b/Documentation/cpu-hotplug.txt
@@ -113,13 +113,18 @@ Never use anything other than cpumask_t to represent bitmap of CPUs.
#include <linux/cpu.h>
get_online_cpus() and put_online_cpus():
-The above calls are used to inhibit cpu hotplug operations. While the
+The above calls are used to inhibit cpu hotplug operations, when invoked from
+non-atomic contexts (because the above functions can sleep). While the
cpu_hotplug.refcount is non zero, the cpu_online_mask will not change.
-If you merely need to avoid cpus going away, you could also use
-preempt_disable() and preempt_enable() for those sections.
-Just remember the critical section cannot call any
-function that can sleep or schedule this process away. The preempt_disable()
-will work as long as stop_machine_run() is used to take a cpu down.
+
+However, if you are executing in atomic context (ie., you can't afford to
+sleep), and you merely need to avoid cpus going offline, you can use
+get_online_cpus_atomic() and put_online_cpus_atomic() for those sections.
+Just remember the critical section cannot call any function that can sleep or
+schedule this process away. Using preempt_disable() will also work, as long
+as stop_machine() is used to take a CPU down. But we are going to get rid of
+stop_machine() in the CPU offline path soon, so it is strongly recommended
+to use the APIs mentioned above.
CPU Hotplug - Frequently Asked Questions.
@@ -360,6 +365,9 @@ A: There are two ways. If your code can be run in interrupt context, use
return err;
}
+ If my_func_on_cpu() itself cannot block, use get/put_online_cpus_atomic()
+ instead of get/put_online_cpus(), to prevent CPUs from going offline.
+
Q: How do we determine how many CPUs are available for hotplug.
A: There is no clear spec defined way from ACPI that can give us that
information today. Based on some input from Natalie of Unisys,
^ permalink raw reply related
* [PATCH v2 04/45] CPU hotplug: Add infrastructure to check lacking hotplug synchronization
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, Alex Shi, nikunj, zhong, linux-pm, fweisbec,
Rusty Russell, linux-kernel, rostedt, xiaoguangrong,
Rafael J. Wysocki, sbw, Yasuaki Ishimatsu, wangyun,
Srivatsa S. Bhat, netdev, Tejun Heo, Thomas Gleixner,
KOSAKI Motohiro, linuxppc-dev, Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Add a debugging infrastructure to warn if an atomic hotplug reader has not
invoked get_online_cpus_atomic() before traversing/accessing the
cpu_online_mask. Encapsulate these checks under a new debug config option
DEBUG_HOTPLUG_CPU.
This debugging infrastructure proves useful in the tree-wide conversion
of atomic hotplug readers from preempt_disable() to the new APIs, and
help us catch the places we missed, much before we actually get rid of
stop_machine(). We can perhaps remove the debugging checks later on.
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Alex Shi <alex.shi@intel.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
include/linux/cpumask.h | 12 ++++++
kernel/cpu.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 101 insertions(+)
diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index d08e4d2..9197ca4 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -101,6 +101,18 @@ extern const struct cpumask *const cpu_active_mask;
#define cpu_active(cpu) ((cpu) == 0)
#endif
+#ifdef CONFIG_DEBUG_HOTPLUG_CPU
+extern void check_hotplug_safe_cpumask(const struct cpumask *mask);
+extern void check_hotplug_safe_cpu(unsigned int cpu,
+ const struct cpumask *mask);
+#else
+static inline void check_hotplug_safe_cpumask(const struct cpumask *mask) { }
+static inline void check_hotplug_safe_cpu(unsigned int cpu,
+ const struct cpumask *mask)
+{
+}
+#endif
+
/* verify cpu argument to cpumask_* operators */
static inline unsigned int cpumask_check(unsigned int cpu)
{
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 860f51a..5297ec1 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -63,6 +63,92 @@ static struct {
.refcount = 0,
};
+#ifdef CONFIG_DEBUG_HOTPLUG_CPU
+
+static DEFINE_PER_CPU(unsigned long, atomic_reader_refcnt);
+
+static int current_is_hotplug_safe(const struct cpumask *mask)
+{
+
+ /* If we are not dealing with cpu_online_mask, don't complain. */
+ if (mask != cpu_online_mask)
+ return 1;
+
+ /* If this is the task doing hotplug, don't complain. */
+ if (unlikely(current == cpu_hotplug.active_writer))
+ return 1;
+
+ /* If we are in early boot, don't complain. */
+ if (system_state != SYSTEM_RUNNING)
+ return 1;
+
+ /*
+ * Check if the current task is in atomic context and it has
+ * invoked get_online_cpus_atomic() to synchronize with
+ * CPU Hotplug.
+ */
+ if (preempt_count() || irqs_disabled())
+ return this_cpu_read(atomic_reader_refcnt);
+ else
+ return 1; /* No checks for non-atomic contexts for now */
+}
+
+static inline void warn_hotplug_unsafe(void)
+{
+ WARN_ONCE(1, "Must use get/put_online_cpus_atomic() to synchronize"
+ " with CPU hotplug\n");
+}
+
+/*
+ * Check if the task (executing in atomic context) has the required protection
+ * against CPU hotplug, while accessing the specified cpumask.
+ */
+void check_hotplug_safe_cpumask(const struct cpumask *mask)
+{
+ if (!current_is_hotplug_safe(mask))
+ warn_hotplug_unsafe();
+}
+EXPORT_SYMBOL_GPL(check_hotplug_safe_cpumask);
+
+/*
+ * Similar to check_hotplug_safe_cpumask(), except that we don't complain
+ * if the task (executing in atomic context) is testing whether the CPU it
+ * is executing on is online or not.
+ *
+ * (A task executing with preemption disabled on a CPU, automatically prevents
+ * offlining that CPU, irrespective of the actual implementation of CPU
+ * offline. So we don't enforce holding of get_online_cpus_atomic() for that
+ * case).
+ */
+void check_hotplug_safe_cpu(unsigned int cpu, const struct cpumask *mask)
+{
+ if(!current_is_hotplug_safe(mask) && cpu != smp_processor_id())
+ warn_hotplug_unsafe();
+}
+EXPORT_SYMBOL_GPL(check_hotplug_safe_cpu);
+
+static inline void atomic_reader_refcnt_inc(void)
+{
+ this_cpu_inc(atomic_reader_refcnt);
+}
+
+static inline void atomic_reader_refcnt_dec(void)
+{
+ this_cpu_dec(atomic_reader_refcnt);
+}
+
+#else
+
+static inline void atomic_reader_refcnt_inc(void)
+{
+}
+
+static inline void atomic_reader_refcnt_dec(void)
+{
+}
+
+#endif
+
void get_online_cpus(void)
{
might_sleep();
@@ -189,12 +275,15 @@ unsigned int get_online_cpus_atomic(void)
* from going offline.
*/
preempt_disable();
+ atomic_reader_refcnt_inc();
+
return smp_processor_id();
}
EXPORT_SYMBOL_GPL(get_online_cpus_atomic);
void put_online_cpus_atomic(void)
{
+ atomic_reader_refcnt_dec();
preempt_enable();
}
EXPORT_SYMBOL_GPL(put_online_cpus_atomic);
^ permalink raw reply related
* [PATCH v2 05/45] CPU hotplug: Protect set_cpu_online() to avoid false-positives
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, Rafael J. Wysocki,
linux-kernel, rostedt, xiaoguangrong, sbw, Yasuaki Ishimatsu,
wangyun, Srivatsa S. Bhat, netdev, Thomas Gleixner, linuxppc-dev,
Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
When bringing a secondary CPU online, the task running on the CPU coming up
sets itself in the cpu_online_mask. This is safe even though this task is not
the hotplug writer task.
But it is kinda hard to teach this to the CPU hotplug debug infrastructure,
and if we get it wrong, we risk making the debug code too lenient, risking
false-negatives.
Luckily, all architectures use set_cpu_online() to manipulate the
cpu_online_mask. So, to avoid false-positive warnings from the CPU hotplug
debug code, encapsulate the body of set_cpu_online() within
get/put_online_cpus_atomic().
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Yasuaki Ishimatsu <isimatu.yasuaki@jp.fujitsu.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/cpu.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/cpu.c b/kernel/cpu.c
index 5297ec1..35e7115 100644
--- a/kernel/cpu.c
+++ b/kernel/cpu.c
@@ -832,10 +832,14 @@ void set_cpu_present(unsigned int cpu, bool present)
void set_cpu_online(unsigned int cpu, bool online)
{
+ get_online_cpus_atomic();
+
if (online)
cpumask_set_cpu(cpu, to_cpumask(cpu_online_bits));
else
cpumask_clear_cpu(cpu, to_cpumask(cpu_online_bits));
+
+ put_online_cpus_atomic();
}
void set_cpu_active(unsigned int cpu, bool active)
^ permalink raw reply related
* [PATCH v2 06/45] CPU hotplug: Sprinkle debugging checks to catch locking bugs
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, Alex Shi, nikunj, zhong, linux-pm, fweisbec,
Rusty Russell, linux-kernel, rostedt, xiaoguangrong, sbw,
Joonsoo Kim, wangyun, Srivatsa S. Bhat, netdev, Tejun Heo,
Andrew Morton, KOSAKI Motohiro, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Now that we have a debug infrastructure in place to detect cases where
get/put_online_cpus_atomic() had to be used, add these checks at the
right spots to help catch places where we missed converting to the new
APIs.
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Alex Shi <alex.shi@intel.com>
Cc: KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Joonsoo Kim <js1304@gmail.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
include/linux/cpumask.h | 47 +++++++++++++++++++++++++++++++++++++++++++++--
lib/cpumask.c | 8 ++++++++
2 files changed, 53 insertions(+), 2 deletions(-)
diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h
index 9197ca4..06d2c36 100644
--- a/include/linux/cpumask.h
+++ b/include/linux/cpumask.h
@@ -169,6 +169,7 @@ static inline unsigned int cpumask_any_but(const struct cpumask *mask,
*/
static inline unsigned int cpumask_first(const struct cpumask *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
return find_first_bit(cpumask_bits(srcp), nr_cpumask_bits);
}
@@ -184,6 +185,8 @@ static inline unsigned int cpumask_next(int n, const struct cpumask *srcp)
/* -1 is a legal arg here. */
if (n != -1)
cpumask_check(n);
+
+ check_hotplug_safe_cpumask(srcp);
return find_next_bit(cpumask_bits(srcp), nr_cpumask_bits, n+1);
}
@@ -199,6 +202,8 @@ static inline unsigned int cpumask_next_zero(int n, const struct cpumask *srcp)
/* -1 is a legal arg here. */
if (n != -1)
cpumask_check(n);
+
+ check_hotplug_safe_cpumask(srcp);
return find_next_zero_bit(cpumask_bits(srcp), nr_cpumask_bits, n+1);
}
@@ -288,8 +293,15 @@ static inline void cpumask_clear_cpu(int cpu, struct cpumask *dstp)
*
* No static inline type checking - see Subtlety (1) above.
*/
-#define cpumask_test_cpu(cpu, cpumask) \
- test_bit(cpumask_check(cpu), cpumask_bits((cpumask)))
+#define cpumask_test_cpu(cpu, cpumask) \
+({ \
+ int __ret; \
+ \
+ check_hotplug_safe_cpu(cpu, cpumask); \
+ __ret = test_bit(cpumask_check(cpu), \
+ cpumask_bits((cpumask))); \
+ __ret; \
+})
/**
* cpumask_test_and_set_cpu - atomically test and set a cpu in a cpumask
@@ -349,6 +361,9 @@ static inline int cpumask_and(struct cpumask *dstp,
const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
return bitmap_and(cpumask_bits(dstp), cpumask_bits(src1p),
cpumask_bits(src2p), nr_cpumask_bits);
}
@@ -362,6 +377,9 @@ static inline int cpumask_and(struct cpumask *dstp,
static inline void cpumask_or(struct cpumask *dstp, const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
bitmap_or(cpumask_bits(dstp), cpumask_bits(src1p),
cpumask_bits(src2p), nr_cpumask_bits);
}
@@ -376,6 +394,9 @@ static inline void cpumask_xor(struct cpumask *dstp,
const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
bitmap_xor(cpumask_bits(dstp), cpumask_bits(src1p),
cpumask_bits(src2p), nr_cpumask_bits);
}
@@ -392,6 +413,9 @@ static inline int cpumask_andnot(struct cpumask *dstp,
const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
return bitmap_andnot(cpumask_bits(dstp), cpumask_bits(src1p),
cpumask_bits(src2p), nr_cpumask_bits);
}
@@ -404,6 +428,8 @@ static inline int cpumask_andnot(struct cpumask *dstp,
static inline void cpumask_complement(struct cpumask *dstp,
const struct cpumask *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
+
bitmap_complement(cpumask_bits(dstp), cpumask_bits(srcp),
nr_cpumask_bits);
}
@@ -416,6 +442,9 @@ static inline void cpumask_complement(struct cpumask *dstp,
static inline bool cpumask_equal(const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
return bitmap_equal(cpumask_bits(src1p), cpumask_bits(src2p),
nr_cpumask_bits);
}
@@ -428,6 +457,10 @@ static inline bool cpumask_equal(const struct cpumask *src1p,
static inline bool cpumask_intersects(const struct cpumask *src1p,
const struct cpumask *src2p)
{
+
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
return bitmap_intersects(cpumask_bits(src1p), cpumask_bits(src2p),
nr_cpumask_bits);
}
@@ -442,6 +475,9 @@ static inline bool cpumask_intersects(const struct cpumask *src1p,
static inline int cpumask_subset(const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
return bitmap_subset(cpumask_bits(src1p), cpumask_bits(src2p),
nr_cpumask_bits);
}
@@ -470,6 +506,12 @@ static inline bool cpumask_full(const struct cpumask *srcp)
*/
static inline unsigned int cpumask_weight(const struct cpumask *srcp)
{
+ /*
+ * Often, we just want to have a rough estimate of the number of
+ * online CPUs, without going to the trouble of synchronizing with
+ * CPU hotplug. So don't invoke check_hotplug_safe_cpumask() here.
+ */
+
return bitmap_weight(cpumask_bits(srcp), nr_cpumask_bits);
}
@@ -507,6 +549,7 @@ static inline void cpumask_shift_left(struct cpumask *dstp,
static inline void cpumask_copy(struct cpumask *dstp,
const struct cpumask *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
bitmap_copy(cpumask_bits(dstp), cpumask_bits(srcp), nr_cpumask_bits);
}
diff --git a/lib/cpumask.c b/lib/cpumask.c
index d327b87..481df57 100644
--- a/lib/cpumask.c
+++ b/lib/cpumask.c
@@ -7,12 +7,14 @@
int __first_cpu(const cpumask_t *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
return min_t(int, NR_CPUS, find_first_bit(srcp->bits, NR_CPUS));
}
EXPORT_SYMBOL(__first_cpu);
int __next_cpu(int n, const cpumask_t *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
return min_t(int, NR_CPUS, find_next_bit(srcp->bits, NR_CPUS, n+1));
}
EXPORT_SYMBOL(__next_cpu);
@@ -20,6 +22,7 @@ EXPORT_SYMBOL(__next_cpu);
#if NR_CPUS > 64
int __next_cpu_nr(int n, const cpumask_t *srcp)
{
+ check_hotplug_safe_cpumask(srcp);
return min_t(int, nr_cpu_ids,
find_next_bit(srcp->bits, nr_cpu_ids, n+1));
}
@@ -37,6 +40,9 @@ EXPORT_SYMBOL(__next_cpu_nr);
int cpumask_next_and(int n, const struct cpumask *src1p,
const struct cpumask *src2p)
{
+ check_hotplug_safe_cpumask(src1p);
+ check_hotplug_safe_cpumask(src2p);
+
while ((n = cpumask_next(n, src1p)) < nr_cpu_ids)
if (cpumask_test_cpu(n, src2p))
break;
@@ -57,6 +63,8 @@ int cpumask_any_but(const struct cpumask *mask, unsigned int cpu)
unsigned int i;
cpumask_check(cpu);
+ check_hotplug_safe_cpumask(mask);
+
for_each_cpu(i, mask)
if (i != cpu)
break;
^ permalink raw reply related
* [PATCH v2 07/45] CPU hotplug: Expose the new debug config option
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, Michel Lespinasse, Catalin Marinas, nikunj, zhong,
linux-pm, fweisbec, linux-kernel, rostedt, xiaoguangrong, sbw,
Akinobu Mita, wangyun, Srivatsa S. Bhat, netdev, Andrew Morton,
Paul E. McKenney, linuxppc-dev, Sergei Shtylyov
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Now that we have all the pieces of the CPU hotplug debug infrastructure
in place, expose the feature by growing a new Kconfig option,
CONFIG_DEBUG_HOTPLUG_CPU.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: "Paul E. McKenney" <paul.mckenney@linaro.org>
Cc: Akinobu Mita <akinobu.mita@gmail.com>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Michel Lespinasse <walken@google.com>
Cc: Sergei Shtylyov <sergei.shtylyov@cogentembedded.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
lib/Kconfig.debug | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index 566cf2b..ec6be74 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -512,6 +512,14 @@ config DEBUG_PREEMPT
if kernel code uses it in a preemption-unsafe way. Also, the kernel
will detect preemption count underflows.
+config DEBUG_HOTPLUG_CPU
+ bool "Debug CPU hotplug"
+ depends on HOTPLUG_CPU
+ help
+ If you say Y here, the kernel will check all the accesses of
+ cpu_online_mask from atomic contexts, and will print warnings if
+ the task lacks appropriate synchronization with CPU hotplug.
+
config DEBUG_RT_MUTEXES
bool "RT Mutex debugging, deadlock detection"
depends on DEBUG_KERNEL && RT_MUTEXES
^ permalink raw reply related
* [PATCH v2 08/45] CPU hotplug: Convert preprocessor macros to static inline functions
From: Srivatsa S. Bhat @ 2013-06-25 20:26 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, Rafael J. Wysocki, sbw, wangyun,
Srivatsa S. Bhat, netdev, Tejun Heo, Thomas Gleixner,
linuxppc-dev, Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Convert the macros in the CPU hotplug code to static inline C functions.
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: "Rafael J. Wysocki" <rjw@sisk.pl>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
include/linux/cpu.h | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/include/linux/cpu.h b/include/linux/cpu.h
index a57b25a..85431a0 100644
--- a/include/linux/cpu.h
+++ b/include/linux/cpu.h
@@ -202,10 +202,8 @@ static inline void cpu_hotplug_driver_unlock(void)
#else /* CONFIG_HOTPLUG_CPU */
-#define get_online_cpus() do { } while (0)
-#define put_online_cpus() do { } while (0)
-#define cpu_hotplug_disable() do { } while (0)
-#define cpu_hotplug_enable() do { } while (0)
+static inline void get_online_cpus(void) {}
+static inline void put_online_cpus(void) {}
static inline unsigned int get_online_cpus_atomic(void)
{
@@ -222,6 +220,9 @@ static inline void put_online_cpus_atomic(void)
preempt_enable();
}
+static inline void cpu_hotplug_disable(void) {}
+static inline void cpu_hotplug_enable(void) {}
+
#define hotcpu_notifier(fn, pri) do { (void)(fn); } while (0)
/* These aren't inline functions due to a GCC bug. */
#define register_hotcpu_notifier(nb) ({ (void)(nb); 0; })
^ permalink raw reply related
* [PATCH v2 09/45] smp: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, Shaohua Li,
linux-kernel, rostedt, xiaoguangrong, sbw, Jan Beulich, wangyun,
Srivatsa S. Bhat, netdev, Andrew Morton, linuxppc-dev, liguang,
Wang YanQing
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Wang YanQing <udknight@gmail.com>
Cc: Shaohua Li <shli@fusionio.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: liguang <lig.fnst@cn.fujitsu.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/smp.c | 52 ++++++++++++++++++++++++++++++----------------------
1 file changed, 30 insertions(+), 22 deletions(-)
diff --git a/kernel/smp.c b/kernel/smp.c
index 4dba0f7..1f36d6d 100644
--- a/kernel/smp.c
+++ b/kernel/smp.c
@@ -232,7 +232,7 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
* prevent preemption and reschedule on another processor,
* as well as CPU removal
*/
- this_cpu = get_cpu();
+ this_cpu = get_online_cpus_atomic();
/*
* Can deadlock when called with interrupts disabled.
@@ -264,7 +264,7 @@ int smp_call_function_single(int cpu, smp_call_func_t func, void *info,
}
}
- put_cpu();
+ put_online_cpus_atomic();
return err;
}
@@ -294,7 +294,7 @@ int smp_call_function_any(const struct cpumask *mask,
int ret;
/* Try for same CPU (cheapest) */
- cpu = get_cpu();
+ cpu = get_online_cpus_atomic();
if (cpumask_test_cpu(cpu, mask))
goto call;
@@ -310,7 +310,7 @@ int smp_call_function_any(const struct cpumask *mask,
cpu = cpumask_any_and(mask, cpu_online_mask);
call:
ret = smp_call_function_single(cpu, func, info, wait);
- put_cpu();
+ put_online_cpus_atomic();
return ret;
}
EXPORT_SYMBOL_GPL(smp_call_function_any);
@@ -331,7 +331,8 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd,
unsigned int this_cpu;
unsigned long flags;
- this_cpu = get_cpu();
+ this_cpu = get_online_cpus_atomic();
+
/*
* Can deadlock when called with interrupts disabled.
* We allow cpu's that are not yet online though, as no one else can
@@ -349,7 +350,8 @@ void __smp_call_function_single(int cpu, struct call_single_data *csd,
csd_lock(csd);
generic_exec_single(cpu, csd, wait);
}
- put_cpu();
+
+ put_online_cpus_atomic();
}
/**
@@ -370,7 +372,9 @@ void smp_call_function_many(const struct cpumask *mask,
smp_call_func_t func, void *info, bool wait)
{
struct call_function_data *cfd;
- int cpu, next_cpu, this_cpu = smp_processor_id();
+ int cpu, next_cpu, this_cpu;
+
+ this_cpu = get_online_cpus_atomic();
/*
* Can deadlock when called with interrupts disabled.
@@ -388,7 +392,7 @@ void smp_call_function_many(const struct cpumask *mask,
/* No online cpus? We're done. */
if (cpu >= nr_cpu_ids)
- return;
+ goto out;
/* Do we have another CPU which isn't us? */
next_cpu = cpumask_next_and(cpu, mask, cpu_online_mask);
@@ -398,7 +402,7 @@ void smp_call_function_many(const struct cpumask *mask,
/* Fastpath: do that cpu by itself. */
if (next_cpu >= nr_cpu_ids) {
smp_call_function_single(cpu, func, info, wait);
- return;
+ goto out;
}
cfd = &__get_cpu_var(cfd_data);
@@ -408,7 +412,7 @@ void smp_call_function_many(const struct cpumask *mask,
/* Some callers race with other cpus changing the passed mask */
if (unlikely(!cpumask_weight(cfd->cpumask)))
- return;
+ goto out;
/*
* After we put an entry into the list, cfd->cpumask may be cleared
@@ -443,6 +447,9 @@ void smp_call_function_many(const struct cpumask *mask,
csd_lock_wait(csd);
}
}
+
+out:
+ put_online_cpus_atomic();
}
EXPORT_SYMBOL(smp_call_function_many);
@@ -463,9 +470,9 @@ EXPORT_SYMBOL(smp_call_function_many);
*/
int smp_call_function(smp_call_func_t func, void *info, int wait)
{
- preempt_disable();
+ get_online_cpus_atomic();
smp_call_function_many(cpu_online_mask, func, info, wait);
- preempt_enable();
+ put_online_cpus_atomic();
return 0;
}
@@ -565,12 +572,12 @@ int on_each_cpu(void (*func) (void *info), void *info, int wait)
unsigned long flags;
int ret = 0;
- preempt_disable();
+ get_online_cpus_atomic();
ret = smp_call_function(func, info, wait);
local_irq_save(flags);
func(info);
local_irq_restore(flags);
- preempt_enable();
+ put_online_cpus_atomic();
return ret;
}
EXPORT_SYMBOL(on_each_cpu);
@@ -592,7 +599,7 @@ EXPORT_SYMBOL(on_each_cpu);
void on_each_cpu_mask(const struct cpumask *mask, smp_call_func_t func,
void *info, bool wait)
{
- int cpu = get_cpu();
+ unsigned int cpu = get_online_cpus_atomic();
smp_call_function_many(mask, func, info, wait);
if (cpumask_test_cpu(cpu, mask)) {
@@ -600,7 +607,7 @@ void on_each_cpu_mask(const struct cpumask *mask, smp_call_func_t func,
func(info);
local_irq_enable();
}
- put_cpu();
+ put_online_cpus_atomic();
}
EXPORT_SYMBOL(on_each_cpu_mask);
@@ -625,8 +632,9 @@ EXPORT_SYMBOL(on_each_cpu_mask);
* The function might sleep if the GFP flags indicates a non
* atomic allocation is allowed.
*
- * Preemption is disabled to protect against CPUs going offline but not online.
- * CPUs going online during the call will not be seen or sent an IPI.
+ * We use get/put_online_cpus_atomic() to protect against CPUs going
+ * offline but not online. CPUs going online during the call will
+ * not be seen or sent an IPI.
*
* You must not call this function with disabled interrupts or
* from a hardware interrupt handler or from a bottom half handler.
@@ -641,26 +649,26 @@ void on_each_cpu_cond(bool (*cond_func)(int cpu, void *info),
might_sleep_if(gfp_flags & __GFP_WAIT);
if (likely(zalloc_cpumask_var(&cpus, (gfp_flags|__GFP_NOWARN)))) {
- preempt_disable();
+ get_online_cpus_atomic();
for_each_online_cpu(cpu)
if (cond_func(cpu, info))
cpumask_set_cpu(cpu, cpus);
on_each_cpu_mask(cpus, func, info, wait);
- preempt_enable();
+ put_online_cpus_atomic();
free_cpumask_var(cpus);
} else {
/*
* No free cpumask, bother. No matter, we'll
* just have to IPI them one by one.
*/
- preempt_disable();
+ get_online_cpus_atomic();
for_each_online_cpu(cpu)
if (cond_func(cpu, info)) {
ret = smp_call_function_single(cpu, func,
info, wait);
WARN_ON_ONCE(!ret);
}
- preempt_enable();
+ put_online_cpus_atomic();
}
}
EXPORT_SYMBOL(on_each_cpu_cond);
^ permalink raw reply related
* [PATCH v2 10/45] sched/core: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, Peter Zijlstra, fweisbec,
linux-kernel, rostedt, xiaoguangrong, sbw, Ingo Molnar, wangyun,
Srivatsa S. Bhat, netdev, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/sched/core.c | 23 +++++++++++++++++++++--
1 file changed, 21 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 195658b..accd550 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -1160,11 +1160,11 @@ void kick_process(struct task_struct *p)
{
int cpu;
- preempt_disable();
+ get_online_cpus_atomic();
cpu = task_cpu(p);
if ((cpu != smp_processor_id()) && task_curr(p))
smp_send_reschedule(cpu);
- preempt_enable();
+ put_online_cpus_atomic();
}
EXPORT_SYMBOL_GPL(kick_process);
#endif /* CONFIG_SMP */
@@ -1172,6 +1172,9 @@ EXPORT_SYMBOL_GPL(kick_process);
#ifdef CONFIG_SMP
/*
* ->cpus_allowed is protected by both rq->lock and p->pi_lock
+ *
+ * Must be called within get/put_online_cpus_atomic(), to prevent
+ * CPUs from going offline from under us.
*/
static int select_fallback_rq(int cpu, struct task_struct *p)
{
@@ -1245,6 +1248,9 @@ out:
/*
* The caller (fork, wakeup) owns p->pi_lock, ->cpus_allowed is stable.
+ *
+ * Must be called within get/put_online_cpus_atomic(), to prevent
+ * CPUs from going offline from under us.
*/
static inline
int select_task_rq(struct task_struct *p, int sd_flags, int wake_flags)
@@ -1489,6 +1495,8 @@ try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags)
unsigned long flags;
int cpu, success = 0;
+ get_online_cpus_atomic();
+
smp_wmb();
raw_spin_lock_irqsave(&p->pi_lock, flags);
if (!(p->state & state))
@@ -1531,6 +1539,7 @@ stat:
out:
raw_spin_unlock_irqrestore(&p->pi_lock, flags);
+ put_online_cpus_atomic();
return success;
}
@@ -1753,6 +1762,8 @@ void wake_up_new_task(struct task_struct *p)
unsigned long flags;
struct rq *rq;
+ get_online_cpus_atomic();
+
raw_spin_lock_irqsave(&p->pi_lock, flags);
#ifdef CONFIG_SMP
/*
@@ -1773,6 +1784,8 @@ void wake_up_new_task(struct task_struct *p)
p->sched_class->task_woken(rq, p);
#endif
task_rq_unlock(rq, p, &flags);
+
+ put_online_cpus_atomic();
}
#ifdef CONFIG_PREEMPT_NOTIFIERS
@@ -3886,6 +3899,8 @@ bool __sched yield_to(struct task_struct *p, bool preempt)
unsigned long flags;
int yielded = 0;
+ get_online_cpus_atomic();
+
local_irq_save(flags);
rq = this_rq();
@@ -3931,6 +3946,8 @@ out_unlock:
out_irq:
local_irq_restore(flags);
+ put_online_cpus_atomic();
+
if (yielded > 0)
schedule();
@@ -4331,9 +4348,11 @@ static int migration_cpu_stop(void *data)
* The original target cpu might have gone down and we might
* be on another cpu but it doesn't matter.
*/
+ get_online_cpus_atomic();
local_irq_disable();
__migrate_task(arg->task, raw_smp_processor_id(), arg->dest_cpu);
local_irq_enable();
+ put_online_cpus_atomic();
return 0;
}
^ permalink raw reply related
* [PATCH v2 11/45] migration: Use raw_spin_lock/unlock since interrupts are already disabled
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, Peter Zijlstra, fweisbec,
linux-kernel, rostedt, xiaoguangrong, sbw, Ingo Molnar, wangyun,
Srivatsa S. Bhat, netdev, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
We need not use the raw_spin_lock_irqsave/restore primitives because
all CPU_DYING notifiers run with interrupts disabled. So just use
raw_spin_lock/unlock.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/sched/core.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index accd550..ff26f54 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -4682,14 +4682,14 @@ migration_call(struct notifier_block *nfb, unsigned long action, void *hcpu)
case CPU_DYING:
sched_ttwu_pending();
/* Update our root-domain */
- raw_spin_lock_irqsave(&rq->lock, flags);
+ raw_spin_lock(&rq->lock); /* IRQs already disabled */
if (rq->rd) {
BUG_ON(!cpumask_test_cpu(cpu, rq->rd->span));
set_rq_offline(rq);
}
migrate_tasks(cpu);
BUG_ON(rq->nr_running != 1); /* the migration thread */
- raw_spin_unlock_irqrestore(&rq->lock, flags);
+ raw_spin_unlock(&rq->lock);
break;
case CPU_DEAD:
^ permalink raw reply related
* [PATCH v2 12/45] sched/fair: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, Peter Zijlstra, fweisbec,
linux-kernel, rostedt, xiaoguangrong, sbw, Ingo Molnar, wangyun,
Srivatsa S. Bhat, netdev, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/sched/fair.c | 14 +++++++++++++-
1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
index c0ac2c3..88f056e 100644
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -3338,7 +3338,8 @@ done:
*
* Returns the target CPU number, or the same CPU if no balancing is needed.
*
- * preempt must be disabled.
+ * Must be called within get/put_online_cpus_atomic(), to prevent CPUs
+ * from going offline from under us.
*/
static int
select_task_rq_fair(struct task_struct *p, int sd_flag, int wake_flags)
@@ -5267,6 +5268,8 @@ void idle_balance(int this_cpu, struct rq *this_rq)
raw_spin_unlock(&this_rq->lock);
update_blocked_averages(this_cpu);
+
+ get_online_cpus_atomic();
rcu_read_lock();
for_each_domain(this_cpu, sd) {
unsigned long interval;
@@ -5290,6 +5293,7 @@ void idle_balance(int this_cpu, struct rq *this_rq)
}
}
rcu_read_unlock();
+ put_online_cpus_atomic();
raw_spin_lock(&this_rq->lock);
@@ -5316,6 +5320,7 @@ static int active_load_balance_cpu_stop(void *data)
struct rq *target_rq = cpu_rq(target_cpu);
struct sched_domain *sd;
+ get_online_cpus_atomic();
raw_spin_lock_irq(&busiest_rq->lock);
/* make sure the requested cpu hasn't gone down in the meantime */
@@ -5367,6 +5372,7 @@ static int active_load_balance_cpu_stop(void *data)
out_unlock:
busiest_rq->active_balance = 0;
raw_spin_unlock_irq(&busiest_rq->lock);
+ put_online_cpus_atomic();
return 0;
}
@@ -5527,6 +5533,7 @@ static void rebalance_domains(int cpu, enum cpu_idle_type idle)
update_blocked_averages(cpu);
+ get_online_cpus_atomic();
rcu_read_lock();
for_each_domain(cpu, sd) {
if (!(sd->flags & SD_LOAD_BALANCE))
@@ -5575,6 +5582,7 @@ out:
break;
}
rcu_read_unlock();
+ put_online_cpus_atomic();
/*
* next_balance will be updated only when there is a need.
@@ -5706,6 +5714,7 @@ static void run_rebalance_domains(struct softirq_action *h)
enum cpu_idle_type idle = this_rq->idle_balance ?
CPU_IDLE : CPU_NOT_IDLE;
+ get_online_cpus_atomic();
rebalance_domains(this_cpu, idle);
/*
@@ -5714,6 +5723,7 @@ static void run_rebalance_domains(struct softirq_action *h)
* stopped.
*/
nohz_idle_balance(this_cpu, idle);
+ put_online_cpus_atomic();
}
static inline int on_null_domain(int cpu)
@@ -5731,8 +5741,10 @@ void trigger_load_balance(struct rq *rq, int cpu)
likely(!on_null_domain(cpu)))
raise_softirq(SCHED_SOFTIRQ);
#ifdef CONFIG_NO_HZ_COMMON
+ get_online_cpus_atomic();
if (nohz_kick_needed(rq, cpu) && likely(!on_null_domain(cpu)))
nohz_balancer_kick(cpu);
+ put_online_cpus_atomic();
#endif
}
^ permalink raw reply related
* [PATCH v2 13/45] timer: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, wangyun, Srivatsa S. Bhat, netdev,
Thomas Gleixner, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/timer.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/timer.c b/kernel/timer.c
index 15ffdb3..5db594c 100644
--- a/kernel/timer.c
+++ b/kernel/timer.c
@@ -729,6 +729,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires,
timer_stats_timer_set_start_info(timer);
BUG_ON(!timer->function);
+ get_online_cpus_atomic();
base = lock_timer_base(timer, &flags);
ret = detach_if_pending(timer, base, false);
@@ -768,6 +769,7 @@ __mod_timer(struct timer_list *timer, unsigned long expires,
out_unlock:
spin_unlock_irqrestore(&base->lock, flags);
+ put_online_cpus_atomic();
return ret;
}
@@ -926,6 +928,7 @@ void add_timer_on(struct timer_list *timer, int cpu)
timer_stats_timer_set_start_info(timer);
BUG_ON(timer_pending(timer) || !timer->function);
+ get_online_cpus_atomic();
spin_lock_irqsave(&base->lock, flags);
timer_set_base(timer, base);
debug_activate(timer, timer->expires);
@@ -940,6 +943,7 @@ void add_timer_on(struct timer_list *timer, int cpu)
*/
wake_up_nohz_cpu(cpu);
spin_unlock_irqrestore(&base->lock, flags);
+ put_online_cpus_atomic();
}
EXPORT_SYMBOL_GPL(add_timer_on);
^ permalink raw reply related
* [PATCH v2 14/45] sched/rt: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, Peter Zijlstra, fweisbec,
linux-kernel, rostedt, xiaoguangrong, sbw, Ingo Molnar, wangyun,
Srivatsa S. Bhat, netdev, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/sched/rt.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 01970c8..03d9f38 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -6,6 +6,7 @@
#include "sched.h"
#include <linux/slab.h>
+#include <linux/cpu.h>
int sched_rr_timeslice = RR_TIMESLICE;
@@ -28,7 +29,9 @@ static enum hrtimer_restart sched_rt_period_timer(struct hrtimer *timer)
if (!overrun)
break;
+ get_online_cpus_atomic();
idle = do_sched_rt_period_timer(rt_b, overrun);
+ put_online_cpus_atomic();
}
return idle ? HRTIMER_NORESTART : HRTIMER_RESTART;
@@ -547,6 +550,7 @@ static int do_balance_runtime(struct rt_rq *rt_rq)
int i, weight, more = 0;
u64 rt_period;
+ get_online_cpus_atomic();
weight = cpumask_weight(rd->span);
raw_spin_lock(&rt_b->rt_runtime_lock);
@@ -588,6 +592,7 @@ next:
raw_spin_unlock(&iter->rt_runtime_lock);
}
raw_spin_unlock(&rt_b->rt_runtime_lock);
+ put_online_cpus_atomic();
return more;
}
@@ -1168,6 +1173,10 @@ static void yield_task_rt(struct rq *rq)
#ifdef CONFIG_SMP
static int find_lowest_rq(struct task_struct *task);
+/*
+ * Must be called within get/put_online_cpus_atomic(), to prevent CPUs
+ * from going offline from under us.
+ */
static int
select_task_rq_rt(struct task_struct *p, int sd_flag, int flags)
{
@@ -1561,6 +1570,8 @@ retry:
return 0;
}
+ get_online_cpus_atomic();
+
/* We might release rq lock */
get_task_struct(next_task);
@@ -1611,6 +1622,7 @@ retry:
out:
put_task_struct(next_task);
+ put_online_cpus_atomic();
return ret;
}
@@ -1630,6 +1642,7 @@ static int pull_rt_task(struct rq *this_rq)
if (likely(!rt_overloaded(this_rq)))
return 0;
+ get_online_cpus_atomic();
for_each_cpu(cpu, this_rq->rd->rto_mask) {
if (this_cpu == cpu)
continue;
@@ -1695,6 +1708,7 @@ skip:
double_unlock_balance(this_rq, src_rq);
}
+ put_online_cpus_atomic();
return ret;
}
^ permalink raw reply related
* [PATCH v2 15/45] rcu: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:27 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, wangyun, Srivatsa S. Bhat, netdev,
Paul E. McKenney, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
In RCU code, rcu_implicit_dynticks_qs() checks if a CPU is offline,
while being protected by a spinlock. Use the get/put_online_cpus_atomic()
APIs to prevent CPUs from going offline, while invoking from atomic context.
Cc: Dipankar Sarma <dipankar@in.ibm.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/rcutree.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/kernel/rcutree.c b/kernel/rcutree.c
index cf3adc6..caeed1a 100644
--- a/kernel/rcutree.c
+++ b/kernel/rcutree.c
@@ -2107,6 +2107,8 @@ static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *))
rcu_initiate_boost(rnp, flags); /* releases rnp->lock */
continue;
}
+
+ get_online_cpus_atomic();
cpu = rnp->grplo;
bit = 1;
for (; cpu <= rnp->grphi; cpu++, bit <<= 1) {
@@ -2114,6 +2116,8 @@ static void force_qs_rnp(struct rcu_state *rsp, int (*f)(struct rcu_data *))
f(per_cpu_ptr(rsp->rda, cpu)))
mask |= bit;
}
+ put_online_cpus_atomic();
+
if (mask != 0) {
/* rcu_report_qs_rnp() releases rnp->lock. */
^ permalink raw reply related
* [PATCH v2 16/45] tick-broadcast: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:28 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, wangyun, Srivatsa S. Bhat, netdev,
Thomas Gleixner, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/time/tick-broadcast.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
index 9d96a54..1e4ac45 100644
--- a/kernel/time/tick-broadcast.c
+++ b/kernel/time/tick-broadcast.c
@@ -227,12 +227,14 @@ static void tick_do_broadcast(struct cpumask *mask)
*/
static void tick_do_periodic_broadcast(void)
{
+ get_online_cpus_atomic();
raw_spin_lock(&tick_broadcast_lock);
cpumask_and(tmpmask, cpu_online_mask, tick_broadcast_mask);
tick_do_broadcast(tmpmask);
raw_spin_unlock(&tick_broadcast_lock);
+ put_online_cpus_atomic();
}
/*
@@ -335,11 +337,13 @@ out:
*/
void tick_broadcast_on_off(unsigned long reason, int *oncpu)
{
+ get_online_cpus_atomic();
if (!cpumask_test_cpu(*oncpu, cpu_online_mask))
printk(KERN_ERR "tick-broadcast: ignoring broadcast for "
"offline CPU #%d\n", *oncpu);
else
tick_do_broadcast_on_off(&reason);
+ put_online_cpus_atomic();
}
/*
@@ -505,6 +509,7 @@ static void tick_handle_oneshot_broadcast(struct clock_event_device *dev)
ktime_t now, next_event;
int cpu, next_cpu = 0;
+ get_online_cpus_atomic();
raw_spin_lock(&tick_broadcast_lock);
again:
dev->next_event.tv64 = KTIME_MAX;
@@ -562,6 +567,7 @@ again:
goto again;
}
raw_spin_unlock(&tick_broadcast_lock);
+ put_online_cpus_atomic();
}
/*
@@ -756,6 +762,7 @@ void tick_broadcast_switch_to_oneshot(void)
struct clock_event_device *bc;
unsigned long flags;
+ get_online_cpus_atomic();
raw_spin_lock_irqsave(&tick_broadcast_lock, flags);
tick_broadcast_device.mode = TICKDEV_MODE_ONESHOT;
@@ -764,6 +771,7 @@ void tick_broadcast_switch_to_oneshot(void)
tick_broadcast_setup_oneshot(bc);
raw_spin_unlock_irqrestore(&tick_broadcast_lock, flags);
+ put_online_cpus_atomic();
}
^ permalink raw reply related
* [PATCH v2 17/45] time/clocksource: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:28 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, John Stultz, wangyun,
Srivatsa S. Bhat, netdev, Thomas Gleixner, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: John Stultz <john.stultz@linaro.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/time/clocksource.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/kernel/time/clocksource.c b/kernel/time/clocksource.c
index e713ef7..c4bbc25 100644
--- a/kernel/time/clocksource.c
+++ b/kernel/time/clocksource.c
@@ -30,6 +30,7 @@
#include <linux/sched.h> /* for spin_unlock_irq() using preempt_count() m68k */
#include <linux/tick.h>
#include <linux/kthread.h>
+#include <linux/cpu.h>
#include "tick-internal.h"
@@ -252,6 +253,7 @@ static void clocksource_watchdog(unsigned long data)
int64_t wd_nsec, cs_nsec;
int next_cpu, reset_pending;
+ get_online_cpus_atomic();
spin_lock(&watchdog_lock);
if (!watchdog_running)
goto out;
@@ -329,6 +331,7 @@ static void clocksource_watchdog(unsigned long data)
add_timer_on(&watchdog_timer, next_cpu);
out:
spin_unlock(&watchdog_lock);
+ put_online_cpus_atomic();
}
static inline void clocksource_start_watchdog(void)
@@ -367,6 +370,7 @@ static void clocksource_enqueue_watchdog(struct clocksource *cs)
{
unsigned long flags;
+ get_online_cpus_atomic();
spin_lock_irqsave(&watchdog_lock, flags);
if (cs->flags & CLOCK_SOURCE_MUST_VERIFY) {
/* cs is a clocksource to be watched. */
@@ -386,6 +390,7 @@ static void clocksource_enqueue_watchdog(struct clocksource *cs)
/* Check if the watchdog timer needs to be started. */
clocksource_start_watchdog();
spin_unlock_irqrestore(&watchdog_lock, flags);
+ put_online_cpus_atomic();
}
static void clocksource_dequeue_watchdog(struct clocksource *cs)
^ permalink raw reply related
* [PATCH v2 18/45] softirq: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:28 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, Frederic Weisbecker,
linux-kernel, rostedt, xiaoguangrong, sbw, wangyun,
Srivatsa S. Bhat, netdev, Sedat Dilek, Thomas Gleixner,
Paul E. McKenney, linuxppc-dev, Andrew Morton
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Sedat Dilek <sedat.dilek@gmail.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/softirq.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/kernel/softirq.c b/kernel/softirq.c
index 3d6833f..c289722 100644
--- a/kernel/softirq.c
+++ b/kernel/softirq.c
@@ -644,14 +644,17 @@ static void remote_softirq_receive(void *data)
static int __try_remote_softirq(struct call_single_data *cp, int cpu, int softirq)
{
+ get_online_cpus_atomic();
if (cpu_online(cpu)) {
cp->func = remote_softirq_receive;
cp->info = &softirq;
cp->flags = 0;
__smp_call_function_single(cpu, cp, 0);
+ put_online_cpus_atomic();
return 0;
}
+ put_online_cpus_atomic();
return 1;
}
#else /* CONFIG_USE_GENERIC_SMP_HELPERS */
^ permalink raw reply related
* [PATCH v2 19/45] irq: Use get/put_online_cpus_atomic() to prevent CPU offline
From: Srivatsa S. Bhat @ 2013-06-25 20:28 UTC (permalink / raw)
To: tglx, peterz, tj, oleg, paulmck, rusty, mingo, akpm, namhyung,
walken, vincent.guittot, laijs
Cc: linux-arch, nikunj, zhong, linux-pm, fweisbec, linux-kernel,
rostedt, xiaoguangrong, sbw, wangyun, Srivatsa S. Bhat, netdev,
Thomas Gleixner, linuxppc-dev
In-Reply-To: <20130625202452.16593.22810.stgit@srivatsabhat.in.ibm.com>
Once stop_machine() is gone from the CPU offline path, we won't be able
to depend on disabling preemption to prevent CPUs from going offline
from under us.
Use the get/put_online_cpus_atomic() APIs to prevent CPUs from going
offline, while invoking from atomic context.
Cc: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Srivatsa S. Bhat <srivatsa.bhat@linux.vnet.ibm.com>
---
kernel/irq/manage.c | 7 +++++++
kernel/irq/proc.c | 3 +++
2 files changed, 10 insertions(+)
diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c
index e16caa8..4d89f19 100644
--- a/kernel/irq/manage.c
+++ b/kernel/irq/manage.c
@@ -18,6 +18,7 @@
#include <linux/sched.h>
#include <linux/sched/rt.h>
#include <linux/task_work.h>
+#include <linux/cpu.h>
#include "internals.h"
@@ -202,9 +203,11 @@ int irq_set_affinity(unsigned int irq, const struct cpumask *mask)
if (!desc)
return -EINVAL;
+ get_online_cpus_atomic();
raw_spin_lock_irqsave(&desc->lock, flags);
ret = __irq_set_affinity_locked(irq_desc_get_irq_data(desc), mask);
raw_spin_unlock_irqrestore(&desc->lock, flags);
+ put_online_cpus_atomic();
return ret;
}
@@ -343,9 +346,11 @@ int irq_select_affinity_usr(unsigned int irq, struct cpumask *mask)
unsigned long flags;
int ret;
+ get_online_cpus_atomic();
raw_spin_lock_irqsave(&desc->lock, flags);
ret = setup_affinity(irq, desc, mask);
raw_spin_unlock_irqrestore(&desc->lock, flags);
+ put_online_cpus_atomic();
return ret;
}
@@ -1128,7 +1133,9 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new)
}
/* Set default affinity mask once everything is setup */
+ get_online_cpus_atomic();
setup_affinity(irq, desc, mask);
+ put_online_cpus_atomic();
} else if (new->flags & IRQF_TRIGGER_MASK) {
unsigned int nmsk = new->flags & IRQF_TRIGGER_MASK;
diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c
index 19ed5c4..47f9a74 100644
--- a/kernel/irq/proc.c
+++ b/kernel/irq/proc.c
@@ -7,6 +7,7 @@
*/
#include <linux/irq.h>
+#include <linux/cpu.h>
#include <linux/gfp.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>
@@ -441,6 +442,7 @@ int show_interrupts(struct seq_file *p, void *v)
if (!desc)
return 0;
+ get_online_cpus_atomic();
raw_spin_lock_irqsave(&desc->lock, flags);
for_each_online_cpu(j)
any_count |= kstat_irqs_cpu(i, j);
@@ -477,6 +479,7 @@ int show_interrupts(struct seq_file *p, void *v)
seq_putc(p, '\n');
out:
raw_spin_unlock_irqrestore(&desc->lock, flags);
+ put_online_cpus_atomic();
return 0;
}
#endif
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox