* [patch] improve SMP reschedule and idle routines
@ 2005-05-16 4:21 Nick Piggin
[not found] ` <20050515.220455.59467677.davem@davemloft.net>
0 siblings, 1 reply; 15+ messages in thread
From: Nick Piggin @ 2005-05-16 4:21 UTC (permalink / raw)
To: Anton Blanchard, David S. Miller, Paul Mackerras, linux-ia64,
Ingo Molnar, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 605 bytes --]
This has so far been ported to and tested on i386, x86_64,
ppc64, ia64. I've done an untested hack for sparc64 too.
Unfortunately the change will have to touch all architectures,
but fortunately the above ones are among the more complex.
This patch improves cross CPU rescheduling performance, and
idle reschedule performance by a significant amount on the
systems I've tested on (SMP G5, quad McKinley, SMP+HT P4 Xeon)
in microbenchmarks.
I bet it could give a measurable boost on real workloads on
some systems too. And I think it is a good cleanup in general.
Comments?
--
SUSE Labs, Novell Inc.
[-- Attachment #2: sched-resched-opt.patch --]
[-- Type: text/plain, Size: 24655 bytes --]
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Also have preempt explicitly
disabled in idle routines. Improves efficiency of resched_task and some
cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. (The only time this may prevent an IPI is
when the task's quantum expires in the timer interrupt - this is a
very rare race to bother with in comparison with the cost).
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c 2005-05-16 13:51:42.000000000 +1000
+++ linux-2.6/kernel/sched.c 2005-05-16 13:52:16.000000000 +1000
@@ -805,21 +805,28 @@ static void deactivate_task(struct task_
#ifdef CONFIG_SMP
static void resched_task(task_t *p)
{
- int need_resched, nrpolling;
+ int cpu;
assert_spin_locked(&task_rq(p)->lock);
- /* minimise the chance of sending an interrupt to poll_idle() */
- nrpolling = test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
- need_resched = test_and_set_tsk_thread_flag(p,TIF_NEED_RESCHED);
- nrpolling |= test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
+ if (test_tsk_thread_flag(p, TIF_NEED_RESCHED))
+ return;
+
+ set_tsk_thread_flag(p, TIF_NEED_RESCHED);
+
+ cpu = task_cpu(p);
+ if (cpu == smp_processor_id())
+ return;
- if (!need_resched && !nrpolling && (task_cpu(p) != smp_processor_id()))
- smp_send_reschedule(task_cpu(p));
+ /* NEED_RESCHED must be visible before we test POLLING_NRFLAG */
+ smp_mb();
+ if (!test_tsk_thread_flag(p, TIF_POLLING_NRFLAG))
+ smp_send_reschedule(cpu);
}
#else
static inline void resched_task(task_t *p)
{
+ assert_spin_locked(&task_rq(p)->lock);
set_tsk_need_resched(p);
}
#endif
Index: linux-2.6/arch/i386/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/process.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/i386/kernel/process.c 2005-05-16 13:52:16.000000000 +1000
@@ -95,14 +95,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
} else {
- cpu_relax();
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -113,29 +118,14 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0, %1;"
- "rep; nop;"
- "je 2b;"
- : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
-
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0, %1;"
+ "rep; nop;"
+ "je 2b;"
+ : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
}
/*
@@ -146,24 +136,27 @@ static void poll_idle (void)
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
+ rmb();
+ idle = pm_idle;
- if (!idle)
- idle = default_idle;
+ if (!idle)
+ idle = default_idle;
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
- idle();
- }
+ __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ idle();
+
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -206,15 +199,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/init/main.c 2005-05-16 13:52:16.000000000 +1000
@@ -382,7 +382,7 @@ static void noinline rest_init(void)
kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
numa_default_policy();
unlock_kernel();
- preempt_enable_no_resched();
+ /* Don't re-enable preemption */
cpu_idle();
}
Index: linux-2.6/arch/i386/kernel/apm.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/apm.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/i386/kernel/apm.c 2005-05-16 13:52:16.000000000 +1000
@@ -767,8 +767,20 @@ static int set_system_power_state(u_shor
static int apm_do_idle(void)
{
u32 eax;
+ u8 ret;
+ int idled = 0;
- if (apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax)) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ if (!need_resched()) {
+ idled = 1;
+ ret = apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax);
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ if (!idled)
+ return 0;
+
+ if (ret) {
static unsigned long t;
/* This always fails on some SMP boards running UP kernels.
Index: linux-2.6/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.orig/drivers/acpi/processor_idle.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/drivers/acpi/processor_idle.c 2005-05-16 13:52:16.000000000 +1000
@@ -162,6 +162,14 @@ acpi_processor_power_activate (
return;
}
+static void acpi_safe_halt (void)
+{
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
+ safe_halt();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+}
static void acpi_processor_idle (void)
{
@@ -171,7 +179,7 @@ static void acpi_processor_idle (void)
int sleep_ticks = 0;
u32 t1, t2 = 0;
- pr = processors[_smp_processor_id()];
+ pr = processors[smp_processor_id()];
if (!pr)
return;
@@ -191,8 +199,13 @@ static void acpi_processor_idle (void)
}
cx = pr->power.state;
- if (!cx)
- goto easy_out;
+ if (!cx) {
+ if (pm_idle_save)
+ pm_idle_save();
+ else
+ acpi_safe_halt();
+ return;
+ }
/*
* Check BM Activity
@@ -272,7 +285,8 @@ static void acpi_processor_idle (void)
if (pm_idle_save)
pm_idle_save();
else
- safe_halt();
+ acpi_safe_halt();
+
/*
* TBD: Can't get time duration while in C1, as resumes
* go to an ISR rather than here. Need to instrument
@@ -384,16 +398,6 @@ end:
*/
if (next_state != pr->power.state)
acpi_processor_power_activate(pr, next_state);
-
- return;
-
- easy_out:
- /* do C1 instead of busy loop */
- if (pm_idle_save)
- pm_idle_save();
- else
- safe_halt();
- return;
}
Index: linux-2.6/arch/i386/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/smpboot.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/i386/kernel/smpboot.c 2005-05-16 13:52:16.000000000 +1000
@@ -416,6 +416,8 @@ static int cpucount;
*/
static void __init start_secondary(void *unused)
{
+ preempt_disable();
+
/*
* Dont put anything before smp_callin(), SMP
* booting is too fragile that we want to limit the
Index: linux-2.6/arch/x86_64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/x86_64/kernel/process.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/arch/x86_64/kernel/process.c 2005-05-16 13:52:16.000000000 +1000
@@ -84,12 +84,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!atomic_read(&hlt_counter)) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -100,29 +107,16 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0,%1;"
- "rep; nop;"
- "je 2b;"
- : :
- "i" (_TIF_NEED_RESCHED),
- "m" (current_thread_info()->flags));
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0,%1;"
+ "rep; nop;"
+ "je 2b;"
+ : :
+ "i" (_TIF_NEED_RESCHED),
+ "m" (current_thread_info()->flags));
}
void cpu_idle_wait(void)
@@ -161,22 +155,25 @@ EXPORT_SYMBOL_GPL(cpu_idle_wait);
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
- if (!idle)
- idle = default_idle;
- idle();
- }
+ rmb();
+ idle = pm_idle;
+ if (!idle)
+ idle = default_idle;
+
+ idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -191,15 +188,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/arch/ppc64/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/idle.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/idle.c 2005-05-16 13:52:16.000000000 +1000
@@ -74,9 +74,10 @@ static void yield_shared_processor(void)
static int iSeries_idle(void)
{
struct paca_struct *lpaca;
- long oldval;
unsigned long CTRL;
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* ensure iSeries run light will be out when idle */
clear_thread_flag(TIF_RUN_LIGHT);
CTRL = mfspr(CTRLF);
@@ -86,32 +87,21 @@ static int iSeries_idle(void)
lpaca = get_paca();
while (1) {
- if (lpaca->lppaca.shared_proc) {
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- if (!need_resched())
- yield_shared_processor();
- } else {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched()) {
- HMT_medium();
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- HMT_low();
- }
-
+ while (!need_resched()) {
+ HMT_low();
+ if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr)) {
HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ process_iSeries_events();
+ HMT_low();
}
+ if (lpaca->lppaca.shared_proc)
+ yield_shared_processor();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
return 0;
@@ -121,32 +111,24 @@ static int iSeries_idle(void)
static int default_idle(void)
{
- long oldval;
unsigned int cpu = smp_processor_id();
-
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched() && !cpu_is_offline(cpu)) {
- barrier();
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- }
-
- HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ barrier();
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -160,12 +142,12 @@ DECLARE_PER_CPU(unsigned long, smt_snooz
int dedicated_idle(void)
{
- long oldval;
struct paca_struct *lpaca = get_paca(), *ppaca;
unsigned long start_snooze;
unsigned long *smt_snooze_delay = &__get_cpu_var(smt_snooze_delay);
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
ppaca = &paca[cpu ^ 1];
while (1) {
@@ -175,66 +157,67 @@ int dedicated_idle(void)
*/
lpaca->lppaca.idle = 1;
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- start_snooze = __get_tb() +
+ start_snooze = __get_tb() +
*smt_snooze_delay * tb_ticks_per_usec;
- while (!need_resched() && !cpu_is_offline(cpu)) {
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- if (*smt_snooze_delay == 0 ||
- __get_tb() < start_snooze)
- continue;
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
- HMT_medium();
+ if (*smt_snooze_delay == 0 || __get_tb() < start_snooze)
+ continue;
- if (!(ppaca->lppaca.idle)) {
- local_irq_disable();
+ HMT_medium();
- /*
- * We are about to sleep the thread
- * and so wont be polling any
- * more.
- */
- clear_thread_flag(TIF_POLLING_NRFLAG);
-
- /*
- * SMT dynamic mode. Cede will result
- * in this thread going dormant, if the
- * partner thread is still doing work.
- * Thread wakes up if partner goes idle,
- * an interrupt is presented, or a prod
- * occurs. Returning from the cede
- * enables external interrupts.
- */
- if (!need_resched())
- cede_processor();
- else
- local_irq_enable();
- } else {
- /*
- * Give the HV an opportunity at the
- * processor, since we are not doing
- * any work.
- */
- poll_pending();
- }
- }
+ if (!(ppaca->lppaca.idle)) {
+ local_irq_disable();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ /*
+ * We are about to sleep the thread
+ * and so wont be polling any
+ * more.
+ */
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+
+ /*
+ * Must have TIF_POLLING_NRFLAG clear visible
+ * before checking need_resched
+ */
+ smp_mb__after_clear_bit();
+
+ /*
+ * SMT dynamic mode. Cede will result
+ * in this thread going dormant, if the
+ * partner thread is still doing work.
+ * Thread wakes up if partner goes idle,
+ * an interrupt is presented, or a prod
+ * occurs. Returning from the cede
+ * enables external interrupts.
+ */
+ if (!need_resched())
+ cede_processor();
+ else
+ local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ /*
+ * Give the HV an opportunity at the
+ * processor, since we are not doing
+ * any work.
+ */
+ poll_pending();
+ }
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -245,6 +228,7 @@ static int shared_idle(void)
{
struct paca_struct *lpaca = get_paca();
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
while (1) {
/*
@@ -256,6 +240,9 @@ static int shared_idle(void)
while (!need_resched() && !cpu_is_offline(cpu)) {
local_irq_disable();
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+
/*
* Yield the processor to the hypervisor. We return if
* an external interrupt occurs (which are driven prior
@@ -270,11 +257,14 @@ static int shared_idle(void)
cede_processor();
else
local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
cpu_die();
@@ -289,10 +279,12 @@ static int native_idle(void)
{
while(1) {
/* check CPU type here */
- if (!need_resched())
+ while (!need_resched())
power4_idle();
- if (need_resched())
- schedule();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
if (cpu_is_offline(_smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
Index: linux-2.6/arch/ia64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/process.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/process.c 2005-05-16 14:04:56.000000000 +1000
@@ -195,11 +195,16 @@ update_pal_halt_status(int status)
void
default_idle (void)
{
- while (!need_resched())
- if (can_do_pal_halt)
+ if (can_do_pal_halt) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
safe_halt();
- else
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
cpu_relax();
+ }
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -261,16 +266,17 @@ void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
+ int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
+
+ if (!need_resched()) {
+ void (*idle)(void);
#ifdef CONFIG_SMP
- if (!need_resched())
min_xtp();
#endif
- while (!need_resched()) {
- void (*idle)(void);
-
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
@@ -282,15 +288,15 @@ cpu_idle (void)
if (!idle)
idle = default_idle;
(*idle)();
- }
-
- if (mark_idle)
- (*mark_idle)(0);
-
+ if (mark_idle)
+ (*mark_idle)(0);
#ifdef CONFIG_SMP
- normal_xtp();
+ normal_xtp();
#endif
+ }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
if (cpu_is_offline(smp_processor_id()))
play_dead();
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/smpboot.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/smpboot.c 2005-05-16 13:52:16.000000000 +1000
@@ -393,6 +393,8 @@ smp_callin (void)
int __devinit
start_secondary (void *unused)
{
+ preempt_disable();
+
/* Early console may use I/O ports */
ia64_set_kr(IA64_KR_IO_BASE, __pa(ia64_iobase));
Dprintk("start_secondary: starting CPU 0x%x\n", hard_smp_processor_id());
Index: linux-2.6/arch/ppc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/smp.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/smp.c 2005-05-16 13:52:16.000000000 +1000
@@ -561,7 +561,10 @@ int __devinit __cpu_up(unsigned int cpu)
/* Activate a secondary processor. */
int __devinit start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/smp.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/smp.c 2005-05-16 13:52:16.000000000 +1000
@@ -1167,20 +1167,9 @@ void __init smp_cpus_done(unsigned int m
(bogosum/(5000/HZ))%100);
}
-/* This needn't do anything as we do not sleep the cpu
- * inside of the idler task, so an interrupt is not needed
- * to get a clean fast response.
- *
- * XXX Reverify this assumption... -DaveM
- *
- * Addendum: We do want it to do something for the signal
- * delivery case, we detect that by just seeing
- * if we are trying to send this to an idler or not.
- */
void smp_send_reschedule(int cpu)
{
- if (cpu_data(cpu).idle_volume == 0)
- smp_receive_signal(cpu);
+ smp_receive_signal(cpu);
}
/* This is a nop because we capture all other cpus
Index: linux-2.6/arch/sparc64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/process.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/process.c 2005-05-16 13:52:16.000000000 +1000
@@ -74,7 +74,9 @@ void cpu_idle(void)
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -83,21 +85,32 @@ void cpu_idle(void)
/*
* the idle loop on a UltraMultiPenguin...
+ *
+ * TIF_POLLING_NRFLAG is set because we do not sleep the cpu
+ * inside of the idler task, so an interrupt is not needed
+ * to get a clean fast response.
+ *
+ * XXX Reverify this assumption... -DaveM
+ *
+ * Addendum: We do want it to do something for the signal
+ * delivery case, we detect that by just seeing
+ * if we are trying to send this to an idler or not.
*/
-#define idle_me_harder() (cpu_data(smp_processor_id()).idle_volume += 1)
-#define unidle_me() (cpu_data(smp_processor_id()).idle_volume = 0)
void cpu_idle(void)
{
+ cpuinfo_sparc cpuinfo = cpu_data(smp_processor_id());
set_thread_flag(TIF_POLLING_NRFLAG);
+
while(1) {
if (need_resched()) {
+ cpuinfo.idle_volume = 0;
unidle_me();
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_enable_no_resched();
schedule();
- set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
check_pgt_cache();
}
- idle_me_harder();
+ cpuinfo.idle_volume++;
/* The store ordering is so that IRQ handlers on
* other cpus see our increasing idleness for the buddy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
[not found] ` <20050515.220455.59467677.davem@davemloft.net>
@ 2005-05-16 5:19 ` Nick Piggin
[not found] ` <20050515.222722.63128129.davem@davemloft.net>
2005-05-17 7:34 ` Nick Piggin
1 sibling, 1 reply; 15+ messages in thread
From: Nick Piggin @ 2005-05-16 5:19 UTC (permalink / raw)
To: David S. Miller; +Cc: anton, paulus, linux-ia64, mingo, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 884 bytes --]
David S. Miller wrote:
> From: Nick Piggin <nickpiggin@yahoo.com.au>
> Date: Mon, 16 May 2005 14:21:54 +1000
>
> I was about to test this on sparc64 real quick, but noticed
> the following with just a quick glance of the sparc64 specific
> changes:
>
>
>> void cpu_idle(void)
>> {
>>+ cpuinfo_sparc cpuinfo = cpu_data(smp_processor_id());
>
>
> Local copy on the stack? Surely you meant a pointer
> instead.
>
Indeed, thank you. How does the following look?
> I'll test this once you work out that obvious bug.
>
The other obvious problem with sparc64 that I didn't tackle is
your secondary CPU bringup - those CPUs will be calling cpu_idle
with preempt enabled as was previously required, but now should
have preempt disabled (ie. arch/sparc64/kernel/trampoline.S:
call cpu_idle)
If you haven't got preempt working on your arch, then in practice
this won't bother you...
[-- Attachment #2: sched-resched-opt.patch --]
[-- Type: text/plain, Size: 24647 bytes --]
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Also have preempt explicitly
disabled in idle routines. Improves efficiency of resched_task and some
cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. (The only time this may prevent an IPI is
when the task's quantum expires in the timer interrupt - this is a
very rare race to bother with in comparison with the cost).
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c 2005-05-16 13:51:42.000000000 +1000
+++ linux-2.6/kernel/sched.c 2005-05-16 13:52:16.000000000 +1000
@@ -805,21 +805,28 @@ static void deactivate_task(struct task_
#ifdef CONFIG_SMP
static void resched_task(task_t *p)
{
- int need_resched, nrpolling;
+ int cpu;
assert_spin_locked(&task_rq(p)->lock);
- /* minimise the chance of sending an interrupt to poll_idle() */
- nrpolling = test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
- need_resched = test_and_set_tsk_thread_flag(p,TIF_NEED_RESCHED);
- nrpolling |= test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
+ if (test_tsk_thread_flag(p, TIF_NEED_RESCHED))
+ return;
+
+ set_tsk_thread_flag(p, TIF_NEED_RESCHED);
+
+ cpu = task_cpu(p);
+ if (cpu == smp_processor_id())
+ return;
- if (!need_resched && !nrpolling && (task_cpu(p) != smp_processor_id()))
- smp_send_reschedule(task_cpu(p));
+ /* NEED_RESCHED must be visible before we test POLLING_NRFLAG */
+ smp_mb();
+ if (!test_tsk_thread_flag(p, TIF_POLLING_NRFLAG))
+ smp_send_reschedule(cpu);
}
#else
static inline void resched_task(task_t *p)
{
+ assert_spin_locked(&task_rq(p)->lock);
set_tsk_need_resched(p);
}
#endif
Index: linux-2.6/arch/i386/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/process.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/i386/kernel/process.c 2005-05-16 13:52:16.000000000 +1000
@@ -95,14 +95,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
} else {
- cpu_relax();
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -113,29 +118,14 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0, %1;"
- "rep; nop;"
- "je 2b;"
- : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
-
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0, %1;"
+ "rep; nop;"
+ "je 2b;"
+ : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
}
/*
@@ -146,24 +136,27 @@ static void poll_idle (void)
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
+ rmb();
+ idle = pm_idle;
- if (!idle)
- idle = default_idle;
+ if (!idle)
+ idle = default_idle;
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
- idle();
- }
+ __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ idle();
+
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -206,15 +199,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/init/main.c 2005-05-16 13:52:16.000000000 +1000
@@ -382,7 +382,7 @@ static void noinline rest_init(void)
kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
numa_default_policy();
unlock_kernel();
- preempt_enable_no_resched();
+ /* Don't re-enable preemption */
cpu_idle();
}
Index: linux-2.6/arch/i386/kernel/apm.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/apm.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/i386/kernel/apm.c 2005-05-16 13:52:16.000000000 +1000
@@ -767,8 +767,20 @@ static int set_system_power_state(u_shor
static int apm_do_idle(void)
{
u32 eax;
+ u8 ret;
+ int idled = 0;
- if (apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax)) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ if (!need_resched()) {
+ idled = 1;
+ ret = apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax);
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ if (!idled)
+ return 0;
+
+ if (ret) {
static unsigned long t;
/* This always fails on some SMP boards running UP kernels.
Index: linux-2.6/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.orig/drivers/acpi/processor_idle.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/drivers/acpi/processor_idle.c 2005-05-16 13:52:16.000000000 +1000
@@ -162,6 +162,14 @@ acpi_processor_power_activate (
return;
}
+static void acpi_safe_halt (void)
+{
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
+ safe_halt();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+}
static void acpi_processor_idle (void)
{
@@ -171,7 +179,7 @@ static void acpi_processor_idle (void)
int sleep_ticks = 0;
u32 t1, t2 = 0;
- pr = processors[_smp_processor_id()];
+ pr = processors[smp_processor_id()];
if (!pr)
return;
@@ -191,8 +199,13 @@ static void acpi_processor_idle (void)
}
cx = pr->power.state;
- if (!cx)
- goto easy_out;
+ if (!cx) {
+ if (pm_idle_save)
+ pm_idle_save();
+ else
+ acpi_safe_halt();
+ return;
+ }
/*
* Check BM Activity
@@ -272,7 +285,8 @@ static void acpi_processor_idle (void)
if (pm_idle_save)
pm_idle_save();
else
- safe_halt();
+ acpi_safe_halt();
+
/*
* TBD: Can't get time duration while in C1, as resumes
* go to an ISR rather than here. Need to instrument
@@ -384,16 +398,6 @@ end:
*/
if (next_state != pr->power.state)
acpi_processor_power_activate(pr, next_state);
-
- return;
-
- easy_out:
- /* do C1 instead of busy loop */
- if (pm_idle_save)
- pm_idle_save();
- else
- safe_halt();
- return;
}
Index: linux-2.6/arch/i386/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/smpboot.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/i386/kernel/smpboot.c 2005-05-16 13:52:16.000000000 +1000
@@ -416,6 +416,8 @@ static int cpucount;
*/
static void __init start_secondary(void *unused)
{
+ preempt_disable();
+
/*
* Dont put anything before smp_callin(), SMP
* booting is too fragile that we want to limit the
Index: linux-2.6/arch/x86_64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/x86_64/kernel/process.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/arch/x86_64/kernel/process.c 2005-05-16 13:52:16.000000000 +1000
@@ -84,12 +84,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!atomic_read(&hlt_counter)) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -100,29 +107,16 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0,%1;"
- "rep; nop;"
- "je 2b;"
- : :
- "i" (_TIF_NEED_RESCHED),
- "m" (current_thread_info()->flags));
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0,%1;"
+ "rep; nop;"
+ "je 2b;"
+ : :
+ "i" (_TIF_NEED_RESCHED),
+ "m" (current_thread_info()->flags));
}
void cpu_idle_wait(void)
@@ -161,22 +155,25 @@ EXPORT_SYMBOL_GPL(cpu_idle_wait);
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
- if (!idle)
- idle = default_idle;
- idle();
- }
+ rmb();
+ idle = pm_idle;
+ if (!idle)
+ idle = default_idle;
+
+ idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -191,15 +188,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/arch/ppc64/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/idle.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/idle.c 2005-05-16 13:52:16.000000000 +1000
@@ -74,9 +74,10 @@ static void yield_shared_processor(void)
static int iSeries_idle(void)
{
struct paca_struct *lpaca;
- long oldval;
unsigned long CTRL;
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* ensure iSeries run light will be out when idle */
clear_thread_flag(TIF_RUN_LIGHT);
CTRL = mfspr(CTRLF);
@@ -86,32 +87,21 @@ static int iSeries_idle(void)
lpaca = get_paca();
while (1) {
- if (lpaca->lppaca.shared_proc) {
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- if (!need_resched())
- yield_shared_processor();
- } else {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched()) {
- HMT_medium();
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- HMT_low();
- }
-
+ while (!need_resched()) {
+ HMT_low();
+ if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr)) {
HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ process_iSeries_events();
+ HMT_low();
}
+ if (lpaca->lppaca.shared_proc)
+ yield_shared_processor();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
return 0;
@@ -121,32 +111,24 @@ static int iSeries_idle(void)
static int default_idle(void)
{
- long oldval;
unsigned int cpu = smp_processor_id();
-
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched() && !cpu_is_offline(cpu)) {
- barrier();
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- }
-
- HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ barrier();
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -160,12 +142,12 @@ DECLARE_PER_CPU(unsigned long, smt_snooz
int dedicated_idle(void)
{
- long oldval;
struct paca_struct *lpaca = get_paca(), *ppaca;
unsigned long start_snooze;
unsigned long *smt_snooze_delay = &__get_cpu_var(smt_snooze_delay);
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
ppaca = &paca[cpu ^ 1];
while (1) {
@@ -175,66 +157,67 @@ int dedicated_idle(void)
*/
lpaca->lppaca.idle = 1;
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- start_snooze = __get_tb() +
+ start_snooze = __get_tb() +
*smt_snooze_delay * tb_ticks_per_usec;
- while (!need_resched() && !cpu_is_offline(cpu)) {
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- if (*smt_snooze_delay == 0 ||
- __get_tb() < start_snooze)
- continue;
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
- HMT_medium();
+ if (*smt_snooze_delay == 0 || __get_tb() < start_snooze)
+ continue;
- if (!(ppaca->lppaca.idle)) {
- local_irq_disable();
+ HMT_medium();
- /*
- * We are about to sleep the thread
- * and so wont be polling any
- * more.
- */
- clear_thread_flag(TIF_POLLING_NRFLAG);
-
- /*
- * SMT dynamic mode. Cede will result
- * in this thread going dormant, if the
- * partner thread is still doing work.
- * Thread wakes up if partner goes idle,
- * an interrupt is presented, or a prod
- * occurs. Returning from the cede
- * enables external interrupts.
- */
- if (!need_resched())
- cede_processor();
- else
- local_irq_enable();
- } else {
- /*
- * Give the HV an opportunity at the
- * processor, since we are not doing
- * any work.
- */
- poll_pending();
- }
- }
+ if (!(ppaca->lppaca.idle)) {
+ local_irq_disable();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ /*
+ * We are about to sleep the thread
+ * and so wont be polling any
+ * more.
+ */
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+
+ /*
+ * Must have TIF_POLLING_NRFLAG clear visible
+ * before checking need_resched
+ */
+ smp_mb__after_clear_bit();
+
+ /*
+ * SMT dynamic mode. Cede will result
+ * in this thread going dormant, if the
+ * partner thread is still doing work.
+ * Thread wakes up if partner goes idle,
+ * an interrupt is presented, or a prod
+ * occurs. Returning from the cede
+ * enables external interrupts.
+ */
+ if (!need_resched())
+ cede_processor();
+ else
+ local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ /*
+ * Give the HV an opportunity at the
+ * processor, since we are not doing
+ * any work.
+ */
+ poll_pending();
+ }
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -245,6 +228,7 @@ static int shared_idle(void)
{
struct paca_struct *lpaca = get_paca();
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
while (1) {
/*
@@ -256,6 +240,9 @@ static int shared_idle(void)
while (!need_resched() && !cpu_is_offline(cpu)) {
local_irq_disable();
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+
/*
* Yield the processor to the hypervisor. We return if
* an external interrupt occurs (which are driven prior
@@ -270,11 +257,14 @@ static int shared_idle(void)
cede_processor();
else
local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
cpu_die();
@@ -289,10 +279,12 @@ static int native_idle(void)
{
while(1) {
/* check CPU type here */
- if (!need_resched())
+ while (!need_resched())
power4_idle();
- if (need_resched())
- schedule();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
if (cpu_is_offline(_smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
Index: linux-2.6/arch/ia64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/process.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/process.c 2005-05-16 14:04:56.000000000 +1000
@@ -195,11 +195,16 @@ update_pal_halt_status(int status)
void
default_idle (void)
{
- while (!need_resched())
- if (can_do_pal_halt)
+ if (can_do_pal_halt) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
safe_halt();
- else
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
cpu_relax();
+ }
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -261,16 +266,17 @@ void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
+ int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
+
+ if (!need_resched()) {
+ void (*idle)(void);
#ifdef CONFIG_SMP
- if (!need_resched())
min_xtp();
#endif
- while (!need_resched()) {
- void (*idle)(void);
-
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
@@ -282,15 +288,15 @@ cpu_idle (void)
if (!idle)
idle = default_idle;
(*idle)();
- }
-
- if (mark_idle)
- (*mark_idle)(0);
-
+ if (mark_idle)
+ (*mark_idle)(0);
#ifdef CONFIG_SMP
- normal_xtp();
+ normal_xtp();
#endif
+ }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
if (cpu_is_offline(smp_processor_id()))
play_dead();
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/smpboot.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/smpboot.c 2005-05-16 13:52:16.000000000 +1000
@@ -393,6 +393,8 @@ smp_callin (void)
int __devinit
start_secondary (void *unused)
{
+ preempt_disable();
+
/* Early console may use I/O ports */
ia64_set_kr(IA64_KR_IO_BASE, __pa(ia64_iobase));
Dprintk("start_secondary: starting CPU 0x%x\n", hard_smp_processor_id());
Index: linux-2.6/arch/ppc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/smp.c 2005-05-16 13:50:52.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/smp.c 2005-05-16 13:52:16.000000000 +1000
@@ -561,7 +561,10 @@ int __devinit __cpu_up(unsigned int cpu)
/* Activate a secondary processor. */
int __devinit start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/smp.c 2005-05-16 13:49:23.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/smp.c 2005-05-16 13:52:16.000000000 +1000
@@ -1167,20 +1167,9 @@ void __init smp_cpus_done(unsigned int m
(bogosum/(5000/HZ))%100);
}
-/* This needn't do anything as we do not sleep the cpu
- * inside of the idler task, so an interrupt is not needed
- * to get a clean fast response.
- *
- * XXX Reverify this assumption... -DaveM
- *
- * Addendum: We do want it to do something for the signal
- * delivery case, we detect that by just seeing
- * if we are trying to send this to an idler or not.
- */
void smp_send_reschedule(int cpu)
{
- if (cpu_data(cpu).idle_volume == 0)
- smp_receive_signal(cpu);
+ smp_receive_signal(cpu);
}
/* This is a nop because we capture all other cpus
Index: linux-2.6/arch/sparc64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/process.c 2005-05-16 13:50:53.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/process.c 2005-05-16 15:13:57.000000000 +1000
@@ -74,7 +74,9 @@ void cpu_idle(void)
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -83,21 +85,32 @@ void cpu_idle(void)
/*
* the idle loop on a UltraMultiPenguin...
+ *
+ * TIF_POLLING_NRFLAG is set because we do not sleep the cpu
+ * inside of the idler task, so an interrupt is not needed
+ * to get a clean fast response.
+ *
+ * XXX Reverify this assumption... -DaveM
+ *
+ * Addendum: We do want it to do something for the signal
+ * delivery case, we detect that by just seeing
+ * if we are trying to send this to an idler or not.
*/
-#define idle_me_harder() (cpu_data(smp_processor_id()).idle_volume += 1)
-#define unidle_me() (cpu_data(smp_processor_id()).idle_volume = 0)
void cpu_idle(void)
{
+ cpuinfo_sparc *cpuinfo = &local_cpu_data();
set_thread_flag(TIF_POLLING_NRFLAG);
+
while(1) {
if (need_resched()) {
+ cpuinfo->idle_volume = 0;
unidle_me();
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_enable_no_resched();
schedule();
- set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
check_pgt_cache();
}
- idle_me_harder();
+ cpuinfo->idle_volume++;
/* The store ordering is so that IRQ handlers on
* other cpus see our increasing idleness for the buddy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
[not found] ` <20050515.222722.63128129.davem@davemloft.net>
@ 2005-05-16 5:34 ` Nick Piggin
0 siblings, 0 replies; 15+ messages in thread
From: Nick Piggin @ 2005-05-16 5:34 UTC (permalink / raw)
To: David S. Miller; +Cc: anton, paulus, linux-ia64, mingo, linux-kernel
David S. Miller wrote:
> From: Nick Piggin <nickpiggin@yahoo.com.au>
> Date: Mon, 16 May 2005 15:19:00 +1000
>
>
>>The other obvious problem with sparc64 that I didn't tackle is
>>your secondary CPU bringup - those CPUs will be calling cpu_idle
>>with preempt enabled as was previously required, but now should
>>have preempt disabled (ie. arch/sparc64/kernel/trampoline.S:
>>call cpu_idle)
>
>
> And adding a preempt_disable() call to the end of
> arch/sparc64/kernel/smp.c:smp_callin() won't work
> because?
>
No that looks like it should work. If that is the "right"
place to put it, then that's perfect. I'll resend the patch
to you privately.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
@ 2005-05-16 13:51 Oleg Nesterov
2005-05-16 22:52 ` Nick Piggin
0 siblings, 1 reply; 15+ messages in thread
From: Oleg Nesterov @ 2005-05-16 13:51 UTC (permalink / raw)
To: Nick Piggin; +Cc: linux-kernel
Nick Piggin wrote:
>
> void default_idle(void)
> {
> + local_irq_enable();
> +
Stupid question. Why is this sti() needed?
Interrupts are enabled in start_secondary() before cpu_idle()
call, and they can't be disabled after return from schedule().
The same question applies to poll_idle/mwait_idle.
Oleg.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-16 13:51 Oleg Nesterov
@ 2005-05-16 22:52 ` Nick Piggin
0 siblings, 0 replies; 15+ messages in thread
From: Nick Piggin @ 2005-05-16 22:52 UTC (permalink / raw)
To: Oleg Nesterov; +Cc: linux-kernel
Oleg Nesterov wrote:
> Nick Piggin wrote:
>
>> void default_idle(void)
>> {
>>+ local_irq_enable();
>>+
>
>
> Stupid question. Why is this sti() needed?
>
> Interrupts are enabled in start_secondary() before cpu_idle()
> call, and they can't be disabled after return from schedule().
>
> The same question applies to poll_idle/mwait_idle.
>
IIRC I tried to do that, but I think I ran into problems with
acpi_processor_idle which looks like it can call the cpu idle
routines with interrupts disabled. I definitely ran into problems
with something.
That should really be cleaned up though (whether we go one way
or the other doesn't matter as much as it being consistent),
I think.
But I wanted to try to keep this patch to a minimum.
--
SUSE Labs, Novell Inc.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
[not found] ` <20050515.220455.59467677.davem@davemloft.net>
2005-05-16 5:19 ` Nick Piggin
@ 2005-05-17 7:34 ` Nick Piggin
2005-05-17 7:40 ` Ingo Molnar
1 sibling, 1 reply; 15+ messages in thread
From: Nick Piggin @ 2005-05-17 7:34 UTC (permalink / raw)
To: David S. Miller; +Cc: anton, paulus, linux-ia64, mingo, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1758 bytes --]
David S. Miller wrote:
> I'll test this once you work out that obvious bug.
>
David's got it tested and working. No difference in the tbench
test reported for SPARC64.
Following are some numbers for a tbench 3.03 test: 1 client, 1 server
in different configurations.
Xeon and G5 seem to be significantly improved on the order of 1-5%.
I2 may be slightly down, but if it is significant I expect real world
workloads to be either not impacted, or hopefully some might see a
small improvement.
tbench, MB/s, higher is better.
Dual Nocona Xeon (mwait idle):
same thread 2.6.12-rc4 -sched
185.6 186.5
185.3 186.8
185.6 187.4
other thread
186.4 187.3
187.3 187.1
187.8 188.4
other CPU
173.0 174.0
170.7 174.2
169.5 175.7
Dual G5:
same CPU
256.3 259.0
255.4 262.3
256.5 259.7
other CPU
150.3 155.4
148.5 155.4
150.1 154.0
Itanium 2:
same CPU
133.1 131.8
128.7 131.6
133.2 132.2
other CPU
84.9 83.9
84.6 83.9
84.5 84.0
Real performance testing would be good, if anyone is interested.
Updated patch attached.
Unless anyone has an objection, I'm going to hack up untested
implementations for the rest of the architectures and see if Andrew
will put the patch in -mm for a while.
[-- Attachment #2: sched-resched-opt.patch --]
[-- Type: text/plain, Size: 24873 bytes --]
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Also have preempt explicitly
disabled in idle routines. Improves efficiency of resched_task and some
cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. (The only time this may prevent an IPI is
when the task's quantum expires in the timer interrupt - this is a
very rare race to bother with in comparison with the cost).
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c 2005-05-17 17:20:03.000000000 +1000
+++ linux-2.6/kernel/sched.c 2005-05-17 17:20:18.000000000 +1000
@@ -805,21 +805,28 @@ static void deactivate_task(struct task_
#ifdef CONFIG_SMP
static void resched_task(task_t *p)
{
- int need_resched, nrpolling;
+ int cpu;
assert_spin_locked(&task_rq(p)->lock);
- /* minimise the chance of sending an interrupt to poll_idle() */
- nrpolling = test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
- need_resched = test_and_set_tsk_thread_flag(p,TIF_NEED_RESCHED);
- nrpolling |= test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
+ if (unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED)))
+ return;
+
+ set_tsk_thread_flag(p, TIF_NEED_RESCHED);
+
+ cpu = task_cpu(p);
+ if (cpu == smp_processor_id())
+ return;
- if (!need_resched && !nrpolling && (task_cpu(p) != smp_processor_id()))
- smp_send_reschedule(task_cpu(p));
+ /* NEED_RESCHED must be visible before we test POLLING_NRFLAG */
+ smp_mb();
+ if (!test_tsk_thread_flag(p, TIF_POLLING_NRFLAG))
+ smp_send_reschedule(cpu);
}
#else
static inline void resched_task(task_t *p)
{
+ assert_spin_locked(&task_rq(p)->lock);
set_tsk_need_resched(p);
}
#endif
Index: linux-2.6/arch/i386/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/process.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/i386/kernel/process.c 2005-05-17 17:20:04.000000000 +1000
@@ -95,14 +95,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
} else {
- cpu_relax();
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -113,29 +118,14 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0, %1;"
- "rep; nop;"
- "je 2b;"
- : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
-
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0, %1;"
+ "rep; nop;"
+ "je 2b;"
+ : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
}
/*
@@ -146,24 +136,27 @@ static void poll_idle (void)
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
+ rmb();
+ idle = pm_idle;
- if (!idle)
- idle = default_idle;
+ if (!idle)
+ idle = default_idle;
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
- idle();
- }
+ __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ idle();
+
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -206,15 +199,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/init/main.c 2005-05-17 17:20:04.000000000 +1000
@@ -382,7 +382,7 @@ static void noinline rest_init(void)
kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
numa_default_policy();
unlock_kernel();
- preempt_enable_no_resched();
+ /* Don't re-enable preemption */
cpu_idle();
}
Index: linux-2.6/arch/i386/kernel/apm.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/apm.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/i386/kernel/apm.c 2005-05-17 17:20:04.000000000 +1000
@@ -767,8 +767,20 @@ static int set_system_power_state(u_shor
static int apm_do_idle(void)
{
u32 eax;
+ u8 ret;
+ int idled = 0;
- if (apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax)) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ if (!need_resched()) {
+ idled = 1;
+ ret = apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax);
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ if (!idled)
+ return 0;
+
+ if (ret) {
static unsigned long t;
/* This always fails on some SMP boards running UP kernels.
Index: linux-2.6/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.orig/drivers/acpi/processor_idle.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/drivers/acpi/processor_idle.c 2005-05-17 17:20:04.000000000 +1000
@@ -162,6 +162,14 @@ acpi_processor_power_activate (
return;
}
+static void acpi_safe_halt (void)
+{
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
+ safe_halt();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+}
static void acpi_processor_idle (void)
{
@@ -171,7 +179,7 @@ static void acpi_processor_idle (void)
int sleep_ticks = 0;
u32 t1, t2 = 0;
- pr = processors[_smp_processor_id()];
+ pr = processors[smp_processor_id()];
if (!pr)
return;
@@ -191,8 +199,13 @@ static void acpi_processor_idle (void)
}
cx = pr->power.state;
- if (!cx)
- goto easy_out;
+ if (!cx) {
+ if (pm_idle_save)
+ pm_idle_save();
+ else
+ acpi_safe_halt();
+ return;
+ }
/*
* Check BM Activity
@@ -272,7 +285,8 @@ static void acpi_processor_idle (void)
if (pm_idle_save)
pm_idle_save();
else
- safe_halt();
+ acpi_safe_halt();
+
/*
* TBD: Can't get time duration while in C1, as resumes
* go to an ISR rather than here. Need to instrument
@@ -384,16 +398,6 @@ end:
*/
if (next_state != pr->power.state)
acpi_processor_power_activate(pr, next_state);
-
- return;
-
- easy_out:
- /* do C1 instead of busy loop */
- if (pm_idle_save)
- pm_idle_save();
- else
- safe_halt();
- return;
}
Index: linux-2.6/arch/i386/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/smpboot.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/i386/kernel/smpboot.c 2005-05-17 17:20:04.000000000 +1000
@@ -416,6 +416,8 @@ static int cpucount;
*/
static void __init start_secondary(void *unused)
{
+ preempt_disable();
+
/*
* Dont put anything before smp_callin(), SMP
* booting is too fragile that we want to limit the
Index: linux-2.6/arch/x86_64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/x86_64/kernel/process.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/x86_64/kernel/process.c 2005-05-17 17:20:04.000000000 +1000
@@ -84,12 +84,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!atomic_read(&hlt_counter)) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -100,29 +107,16 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0,%1;"
- "rep; nop;"
- "je 2b;"
- : :
- "i" (_TIF_NEED_RESCHED),
- "m" (current_thread_info()->flags));
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0,%1;"
+ "rep; nop;"
+ "je 2b;"
+ : :
+ "i" (_TIF_NEED_RESCHED),
+ "m" (current_thread_info()->flags));
}
void cpu_idle_wait(void)
@@ -161,22 +155,25 @@ EXPORT_SYMBOL_GPL(cpu_idle_wait);
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
- if (!idle)
- idle = default_idle;
- idle();
- }
+ rmb();
+ idle = pm_idle;
+ if (!idle)
+ idle = default_idle;
+
+ idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -191,15 +188,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/arch/ppc64/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/idle.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/idle.c 2005-05-17 17:20:04.000000000 +1000
@@ -74,9 +74,10 @@ static void yield_shared_processor(void)
static int iSeries_idle(void)
{
struct paca_struct *lpaca;
- long oldval;
unsigned long CTRL;
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* ensure iSeries run light will be out when idle */
clear_thread_flag(TIF_RUN_LIGHT);
CTRL = mfspr(CTRLF);
@@ -86,32 +87,21 @@ static int iSeries_idle(void)
lpaca = get_paca();
while (1) {
- if (lpaca->lppaca.shared_proc) {
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- if (!need_resched())
- yield_shared_processor();
- } else {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched()) {
- HMT_medium();
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- HMT_low();
- }
-
+ while (!need_resched()) {
+ HMT_low();
+ if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr)) {
HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ process_iSeries_events();
+ HMT_low();
}
+ if (lpaca->lppaca.shared_proc)
+ yield_shared_processor();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
return 0;
@@ -121,32 +111,24 @@ static int iSeries_idle(void)
static int default_idle(void)
{
- long oldval;
unsigned int cpu = smp_processor_id();
-
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched() && !cpu_is_offline(cpu)) {
- barrier();
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- }
-
- HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ barrier();
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -160,12 +142,12 @@ DECLARE_PER_CPU(unsigned long, smt_snooz
int dedicated_idle(void)
{
- long oldval;
struct paca_struct *lpaca = get_paca(), *ppaca;
unsigned long start_snooze;
unsigned long *smt_snooze_delay = &__get_cpu_var(smt_snooze_delay);
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
ppaca = &paca[cpu ^ 1];
while (1) {
@@ -175,66 +157,67 @@ int dedicated_idle(void)
*/
lpaca->lppaca.idle = 1;
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- start_snooze = __get_tb() +
+ start_snooze = __get_tb() +
*smt_snooze_delay * tb_ticks_per_usec;
- while (!need_resched() && !cpu_is_offline(cpu)) {
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- if (*smt_snooze_delay == 0 ||
- __get_tb() < start_snooze)
- continue;
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
- HMT_medium();
+ if (*smt_snooze_delay == 0 || __get_tb() < start_snooze)
+ continue;
- if (!(ppaca->lppaca.idle)) {
- local_irq_disable();
+ HMT_medium();
- /*
- * We are about to sleep the thread
- * and so wont be polling any
- * more.
- */
- clear_thread_flag(TIF_POLLING_NRFLAG);
-
- /*
- * SMT dynamic mode. Cede will result
- * in this thread going dormant, if the
- * partner thread is still doing work.
- * Thread wakes up if partner goes idle,
- * an interrupt is presented, or a prod
- * occurs. Returning from the cede
- * enables external interrupts.
- */
- if (!need_resched())
- cede_processor();
- else
- local_irq_enable();
- } else {
- /*
- * Give the HV an opportunity at the
- * processor, since we are not doing
- * any work.
- */
- poll_pending();
- }
- }
+ if (!(ppaca->lppaca.idle)) {
+ local_irq_disable();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ /*
+ * We are about to sleep the thread
+ * and so wont be polling any
+ * more.
+ */
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+
+ /*
+ * Must have TIF_POLLING_NRFLAG clear visible
+ * before checking need_resched
+ */
+ smp_mb__after_clear_bit();
+
+ /*
+ * SMT dynamic mode. Cede will result
+ * in this thread going dormant, if the
+ * partner thread is still doing work.
+ * Thread wakes up if partner goes idle,
+ * an interrupt is presented, or a prod
+ * occurs. Returning from the cede
+ * enables external interrupts.
+ */
+ if (!need_resched())
+ cede_processor();
+ else
+ local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ /*
+ * Give the HV an opportunity at the
+ * processor, since we are not doing
+ * any work.
+ */
+ poll_pending();
+ }
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -245,6 +228,7 @@ static int shared_idle(void)
{
struct paca_struct *lpaca = get_paca();
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
while (1) {
/*
@@ -256,6 +240,9 @@ static int shared_idle(void)
while (!need_resched() && !cpu_is_offline(cpu)) {
local_irq_disable();
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+
/*
* Yield the processor to the hypervisor. We return if
* an external interrupt occurs (which are driven prior
@@ -270,11 +257,14 @@ static int shared_idle(void)
cede_processor();
else
local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
cpu_die();
@@ -289,10 +279,12 @@ static int native_idle(void)
{
while(1) {
/* check CPU type here */
- if (!need_resched())
+ while (!need_resched())
power4_idle();
- if (need_resched())
- schedule();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
if (cpu_is_offline(_smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
Index: linux-2.6/arch/ia64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/process.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/process.c 2005-05-17 17:20:04.000000000 +1000
@@ -195,11 +195,16 @@ update_pal_halt_status(int status)
void
default_idle (void)
{
- while (!need_resched())
- if (can_do_pal_halt)
+ if (can_do_pal_halt) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
safe_halt();
- else
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
cpu_relax();
+ }
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -261,16 +266,17 @@ void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
+ int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
+
+ if (!need_resched()) {
+ void (*idle)(void);
#ifdef CONFIG_SMP
- if (!need_resched())
min_xtp();
#endif
- while (!need_resched()) {
- void (*idle)(void);
-
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
@@ -282,15 +288,15 @@ cpu_idle (void)
if (!idle)
idle = default_idle;
(*idle)();
- }
-
- if (mark_idle)
- (*mark_idle)(0);
-
+ if (mark_idle)
+ (*mark_idle)(0);
#ifdef CONFIG_SMP
- normal_xtp();
+ normal_xtp();
#endif
+ }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
if (cpu_is_offline(smp_processor_id()))
play_dead();
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/smpboot.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/smpboot.c 2005-05-17 17:20:04.000000000 +1000
@@ -393,6 +393,8 @@ smp_callin (void)
int __devinit
start_secondary (void *unused)
{
+ preempt_disable();
+
/* Early console may use I/O ports */
ia64_set_kr(IA64_KR_IO_BASE, __pa(ia64_iobase));
Dprintk("start_secondary: starting CPU 0x%x\n", hard_smp_processor_id());
Index: linux-2.6/arch/ppc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/smp.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/smp.c 2005-05-17 17:20:04.000000000 +1000
@@ -561,7 +561,10 @@ int __devinit __cpu_up(unsigned int cpu)
/* Activate a secondary processor. */
int __devinit start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/smp.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/smp.c 2005-05-17 17:20:04.000000000 +1000
@@ -144,6 +144,9 @@ void __init smp_callin(void)
membar("#LoadLoad");
cpu_set(cpuid, cpu_online_map);
+
+ /* idle thread is expected to have preempt disabled */
+ preempt_disable();
}
void cpu_panic(void)
@@ -1167,20 +1170,9 @@ void __init smp_cpus_done(unsigned int m
(bogosum/(5000/HZ))%100);
}
-/* This needn't do anything as we do not sleep the cpu
- * inside of the idler task, so an interrupt is not needed
- * to get a clean fast response.
- *
- * XXX Reverify this assumption... -DaveM
- *
- * Addendum: We do want it to do something for the signal
- * delivery case, we detect that by just seeing
- * if we are trying to send this to an idler or not.
- */
void smp_send_reschedule(int cpu)
{
- if (cpu_data(cpu).idle_volume == 0)
- smp_receive_signal(cpu);
+ smp_receive_signal(cpu);
}
/* This is a nop because we capture all other cpus
Index: linux-2.6/arch/sparc64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/process.c 2005-05-17 16:09:56.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/process.c 2005-05-17 17:20:31.000000000 +1000
@@ -74,7 +74,9 @@ void cpu_idle(void)
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -83,21 +85,31 @@ void cpu_idle(void)
/*
* the idle loop on a UltraMultiPenguin...
+ *
+ * TIF_POLLING_NRFLAG is set because we do not sleep the cpu
+ * inside of the idler task, so an interrupt is not needed
+ * to get a clean fast response.
+ *
+ * XXX Reverify this assumption... -DaveM
+ *
+ * Addendum: We do want it to do something for the signal
+ * delivery case, we detect that by just seeing
+ * if we are trying to send this to an idler or not.
*/
-#define idle_me_harder() (cpu_data(smp_processor_id()).idle_volume += 1)
-#define unidle_me() (cpu_data(smp_processor_id()).idle_volume = 0)
void cpu_idle(void)
{
+ cpuinfo_sparc *cpuinfo = &local_cpu_data();
set_thread_flag(TIF_POLLING_NRFLAG);
+
while(1) {
if (need_resched()) {
- unidle_me();
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ cpuinfo->idle_volume = 0;
+ preempt_enable_no_resched();
schedule();
- set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
check_pgt_cache();
}
- idle_me_harder();
+ cpuinfo->idle_volume++;
/* The store ordering is so that IRQ handlers on
* other cpus see our increasing idleness for the buddy
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-17 7:34 ` Nick Piggin
@ 2005-05-17 7:40 ` Ingo Molnar
0 siblings, 0 replies; 15+ messages in thread
From: Ingo Molnar @ 2005-05-17 7:40 UTC (permalink / raw)
To: Nick Piggin
Cc: David S. Miller, anton, paulus, Andrew Morton, linux-ia64,
linux-kernel
* Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> David S. Miller wrote:
>
> >I'll test this once you work out that obvious bug.
> >
>
> David's got it tested and working. No difference in the tbench
> test reported for SPARC64.
>
> Following are some numbers for a tbench 3.03 test: 1 client, 1 server
> in different configurations.
>
> Xeon and G5 seem to be significantly improved on the order of 1-5%. I2
> may be slightly down, but if it is significant I expect real world
> workloads to be either not impacted, or hopefully some might see a
> small improvement.
> [...]
> Unless anyone has an objection, I'm going to hack up untested
> implementations for the rest of the architectures and see if Andrew
> will put the patch in -mm for a while.
the patch looks good to me and builds / boots on my testsystems (x86,
x64) too.
Acked-by: Ingo Molnar <mingo@elte.hu>
Ingo
^ permalink raw reply [flat|nested] 15+ messages in thread
* [patch] improve SMP reschedule and idle routines
@ 2005-05-27 7:21 Nick Piggin
2005-05-27 8:57 ` Ingo Molnar
0 siblings, 1 reply; 15+ messages in thread
From: Nick Piggin @ 2005-05-27 7:21 UTC (permalink / raw)
To: Andrew Morton, Ingo Molnar, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]
OK, done a bit of work on all other architectures, and diffed to the
latest -mm. Any chance you can put it in -mm, Andrew?
Also, while I was there, I thought I'd add the set_need_resched()
thing to all the other architectures. I couldn't be bothered doing
2 patches, sorry.
So, this has been tested on i386, ia64, ppc64, sparc64, x86-64, however
some of those (esp. ppc64) use different idle routines for different
architectures. Needs testing.
David's ACK'ed the sparc64 changes, and Ingo's ACK'ed the rationale
of the patch if not this exact one ;)
No other architectures even compile tested, but they've at least got
something there now.
Quick audit of cpu_idle functions was interesting. I changed some, but
they need review from maintainers, and some need more help. Is it worth
sending this to linux-arch?
alpha - set TIF_POLLING_NRFLAG.
arm26 - how did it work before?! (fixed?)
h8300 - Is sleeping racy vs interrupts?
The H8/300 manual I found indicates yes, however disabling IRQs
over the sleep mean only NMIs can wake it up, so can't fix easily
without doing spin waiting.
m68knommu - "stop" if need_resched() is *SET*?!? (changed, is this right?)
- Is sleeping racy vs interrupts?
parisc - set TIF_POLLING_NRFLAG.
s390 - local irq disable before checking need_resched doesn't gain
anything (removed, OK?)
sh64 - Is sleeping racy vs interrupts?
sparc - IRQs need to be on here => changed local_irq_save to _disable.
- Changed idle loop so don't go to schedule() if pm_idle is NULL!
- set TIF_POLLING_NRFLAG for SMP.
- TODO: needs secondary CPUs to disable preempt
um - I'm too lazy to really look. Might be OK :P
xtensa - obviously not tested with preempt (hopefully fixed?)
Thanks,
Nick
[-- Attachment #2: sched-resched-opt.patch --]
[-- Type: text/plain, Size: 44409 bytes --]
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Also have preempt explicitly
disabled in idle routines. Improves efficiency of resched_task and some
cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c 2005-05-27 15:41:27.000000000 +1000
+++ linux-2.6/kernel/sched.c 2005-05-27 15:43:38.000000000 +1000
@@ -845,21 +845,28 @@ static void deactivate_task(struct task_
#ifdef CONFIG_SMP
static void resched_task(task_t *p)
{
- int need_resched, nrpolling;
+ int cpu;
assert_spin_locked(&task_rq(p)->lock);
- /* minimise the chance of sending an interrupt to poll_idle() */
- nrpolling = test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
- need_resched = test_and_set_tsk_thread_flag(p,TIF_NEED_RESCHED);
- nrpolling |= test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
+ if (unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED)))
+ return;
+
+ set_tsk_thread_flag(p, TIF_NEED_RESCHED);
- if (!need_resched && !nrpolling && (task_cpu(p) != smp_processor_id()))
- smp_send_reschedule(task_cpu(p));
+ cpu = task_cpu(p);
+ if (cpu == smp_processor_id())
+ return;
+
+ /* NEED_RESCHED must be visible before we test POLLING_NRFLAG */
+ smp_mb();
+ if (!test_tsk_thread_flag(p, TIF_POLLING_NRFLAG))
+ smp_send_reschedule(cpu);
}
#else
static inline void resched_task(task_t *p)
{
+ assert_spin_locked(&task_rq(p)->lock);
set_tsk_need_resched(p);
}
#endif
Index: linux-2.6/arch/i386/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/process.c 2005-05-27 15:40:26.000000000 +1000
+++ linux-2.6/arch/i386/kernel/process.c 2005-05-27 15:55:49.000000000 +1000
@@ -102,14 +102,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
} else {
- cpu_relax();
+ while (!need_resched())
+ cpu_relax();
}
}
#ifdef CONFIG_APM_MODULE
@@ -123,29 +128,14 @@ EXPORT_SYMBOL(default_idle);
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0, %1;"
- "rep; nop;"
- "je 2b;"
- : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
-
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0, %1;"
+ "rep; nop;"
+ "je 2b;"
+ : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -181,31 +171,33 @@ static inline void play_dead(void)
*/
void cpu_idle(void)
{
- int cpu = _smp_processor_id();
-
- set_tsk_need_resched(current);
+ int cpu = smp_processor_id();
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
+
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
+
+ rmb();
+ idle = pm_idle;
+
+ if (!idle)
+ idle = default_idle;
+
+ if (cpu_is_offline(cpu))
+ play_dead();
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ idle();
- rmb();
- idle = pm_idle;
-
- if (!idle)
- idle = default_idle;
-
- if (cpu_is_offline(cpu))
- play_dead();
-
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
- idle();
- }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -248,15 +240,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c 2005-05-27 15:41:26.000000000 +1000
+++ linux-2.6/init/main.c 2005-05-27 15:43:38.000000000 +1000
@@ -382,7 +382,7 @@ static void noinline rest_init(void)
kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
numa_default_policy();
unlock_kernel();
- preempt_enable_no_resched();
+ /* Don't re-enable preemption */
cpu_idle();
}
Index: linux-2.6/arch/i386/kernel/apm.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/apm.c 2005-05-27 15:40:25.000000000 +1000
+++ linux-2.6/arch/i386/kernel/apm.c 2005-05-27 15:43:38.000000000 +1000
@@ -767,8 +767,20 @@ static int set_system_power_state(u_shor
static int apm_do_idle(void)
{
u32 eax;
+ u8 ret;
+ int idled = 0;
- if (apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax)) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ if (!need_resched()) {
+ idled = 1;
+ ret = apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax);
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ if (!idled)
+ return 0;
+
+ if (ret) {
static unsigned long t;
/* This always fails on some SMP boards running UP kernels.
Index: linux-2.6/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.orig/drivers/acpi/processor_idle.c 2005-05-27 15:40:33.000000000 +1000
+++ linux-2.6/drivers/acpi/processor_idle.c 2005-05-27 15:43:38.000000000 +1000
@@ -164,6 +164,14 @@ acpi_processor_power_activate (
return;
}
+static void acpi_safe_halt (void)
+{
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
+ safe_halt();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+}
static atomic_t c3_cpu_count;
@@ -176,7 +184,7 @@ static void acpi_processor_idle (void)
int sleep_ticks = 0;
u32 t1, t2 = 0;
- pr = processors[_smp_processor_id()];
+ pr = processors[smp_processor_id()];
if (!pr)
return;
@@ -196,8 +204,13 @@ static void acpi_processor_idle (void)
}
cx = pr->power.state;
- if (!cx)
- goto easy_out;
+ if (!cx) {
+ if (pm_idle_save)
+ pm_idle_save();
+ else
+ acpi_safe_halt();
+ return;
+ }
/*
* Check BM Activity
@@ -277,7 +290,8 @@ static void acpi_processor_idle (void)
if (pm_idle_save)
pm_idle_save();
else
- safe_halt();
+ acpi_safe_halt();
+
/*
* TBD: Can't get time duration while in C1, as resumes
* go to an ISR rather than here. Need to instrument
@@ -407,16 +421,6 @@ end:
*/
if (next_state != pr->power.state)
acpi_processor_power_activate(pr, next_state);
-
- return;
-
- easy_out:
- /* do C1 instead of busy loop */
- if (pm_idle_save)
- pm_idle_save();
- else
- safe_halt();
- return;
}
Index: linux-2.6/arch/i386/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/smpboot.c 2005-05-27 15:40:27.000000000 +1000
+++ linux-2.6/arch/i386/kernel/smpboot.c 2005-05-27 15:43:38.000000000 +1000
@@ -477,6 +477,8 @@ set_cpu_sibling_map(int cpu)
*/
static void __devinit start_secondary(void *unused)
{
+ preempt_disable();
+
/*
* Dont put anything before smp_callin(), SMP
* booting is too fragile that we want to limit the
Index: linux-2.6/arch/x86_64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/x86_64/kernel/process.c 2005-05-27 15:40:30.000000000 +1000
+++ linux-2.6/arch/x86_64/kernel/process.c 2005-05-27 15:53:07.000000000 +1000
@@ -85,12 +85,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!atomic_read(&hlt_counter)) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -101,29 +108,16 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0,%1;"
- "rep; nop;"
- "je 2b;"
- : :
- "i" (_TIF_NEED_RESCHED),
- "m" (current_thread_info()->flags));
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0,%1;"
+ "rep; nop;"
+ "je 2b;"
+ : :
+ "i" (_TIF_NEED_RESCHED),
+ "m" (current_thread_info()->flags));
}
void cpu_idle_wait(void)
@@ -162,24 +156,26 @@ EXPORT_SYMBOL_GPL(cpu_idle_wait);
*/
void cpu_idle (void)
{
- set_tsk_need_resched(current);
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
- if (!idle)
- idle = default_idle;
- idle();
- }
+ rmb();
+ idle = pm_idle;
+ if (!idle)
+ idle = default_idle;
+
+ idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -194,15 +190,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/arch/ppc64/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/idle.c 2005-05-27 15:40:28.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/idle.c 2005-05-27 15:43:38.000000000 +1000
@@ -74,9 +74,10 @@ static void yield_shared_processor(void)
static int iSeries_idle(void)
{
struct paca_struct *lpaca;
- long oldval;
unsigned long CTRL;
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* ensure iSeries run light will be out when idle */
clear_thread_flag(TIF_RUN_LIGHT);
CTRL = mfspr(CTRLF);
@@ -86,32 +87,21 @@ static int iSeries_idle(void)
lpaca = get_paca();
while (1) {
- if (lpaca->lppaca.shared_proc) {
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- if (!need_resched())
- yield_shared_processor();
- } else {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched()) {
- HMT_medium();
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- HMT_low();
- }
-
+ while (!need_resched()) {
+ HMT_low();
+ if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr)) {
HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ process_iSeries_events();
+ HMT_low();
}
+ if (lpaca->lppaca.shared_proc)
+ yield_shared_processor();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
return 0;
@@ -121,32 +111,24 @@ static int iSeries_idle(void)
static int default_idle(void)
{
- long oldval;
unsigned int cpu = smp_processor_id();
-
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched() && !cpu_is_offline(cpu)) {
- barrier();
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- }
-
- HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ barrier();
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -160,12 +142,12 @@ DECLARE_PER_CPU(unsigned long, smt_snooz
int dedicated_idle(void)
{
- long oldval;
struct paca_struct *lpaca = get_paca(), *ppaca;
unsigned long start_snooze;
unsigned long *smt_snooze_delay = &__get_cpu_var(smt_snooze_delay);
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
ppaca = &paca[cpu ^ 1];
while (1) {
@@ -175,66 +157,67 @@ int dedicated_idle(void)
*/
lpaca->lppaca.idle = 1;
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- start_snooze = __get_tb() +
+ start_snooze = __get_tb() +
*smt_snooze_delay * tb_ticks_per_usec;
- while (!need_resched() && !cpu_is_offline(cpu)) {
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- if (*smt_snooze_delay == 0 ||
- __get_tb() < start_snooze)
- continue;
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
- HMT_medium();
+ if (*smt_snooze_delay == 0 || __get_tb() < start_snooze)
+ continue;
- if (!(ppaca->lppaca.idle)) {
- local_irq_disable();
+ HMT_medium();
- /*
- * We are about to sleep the thread
- * and so wont be polling any
- * more.
- */
- clear_thread_flag(TIF_POLLING_NRFLAG);
-
- /*
- * SMT dynamic mode. Cede will result
- * in this thread going dormant, if the
- * partner thread is still doing work.
- * Thread wakes up if partner goes idle,
- * an interrupt is presented, or a prod
- * occurs. Returning from the cede
- * enables external interrupts.
- */
- if (!need_resched())
- cede_processor();
- else
- local_irq_enable();
- } else {
- /*
- * Give the HV an opportunity at the
- * processor, since we are not doing
- * any work.
- */
- poll_pending();
- }
- }
+ if (!(ppaca->lppaca.idle)) {
+ local_irq_disable();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ /*
+ * We are about to sleep the thread
+ * and so wont be polling any
+ * more.
+ */
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+
+ /*
+ * Must have TIF_POLLING_NRFLAG clear visible
+ * before checking need_resched
+ */
+ smp_mb__after_clear_bit();
+
+ /*
+ * SMT dynamic mode. Cede will result
+ * in this thread going dormant, if the
+ * partner thread is still doing work.
+ * Thread wakes up if partner goes idle,
+ * an interrupt is presented, or a prod
+ * occurs. Returning from the cede
+ * enables external interrupts.
+ */
+ if (!need_resched())
+ cede_processor();
+ else
+ local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ /*
+ * Give the HV an opportunity at the
+ * processor, since we are not doing
+ * any work.
+ */
+ poll_pending();
+ }
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -245,6 +228,7 @@ static int shared_idle(void)
{
struct paca_struct *lpaca = get_paca();
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
while (1) {
/*
@@ -256,6 +240,9 @@ static int shared_idle(void)
while (!need_resched() && !cpu_is_offline(cpu)) {
local_irq_disable();
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+
/*
* Yield the processor to the hypervisor. We return if
* an external interrupt occurs (which are driven prior
@@ -270,11 +257,14 @@ static int shared_idle(void)
cede_processor();
else
local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
cpu_die();
@@ -289,10 +279,12 @@ static int native_idle(void)
{
while(1) {
/* check CPU type here */
- if (!need_resched())
+ while (!need_resched())
power4_idle();
- if (need_resched())
- schedule();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
if (cpu_is_offline(_smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
Index: linux-2.6/arch/ia64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/process.c 2005-05-27 15:40:28.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/process.c 2005-05-27 15:59:15.000000000 +1000
@@ -195,11 +195,16 @@ update_pal_halt_status(int status)
void
default_idle (void)
{
- while (!need_resched())
- if (can_do_pal_halt)
+ if (can_do_pal_halt) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
safe_halt();
- else
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
cpu_relax();
+ }
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -261,18 +266,17 @@ void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
-
- set_tsk_need_resched(current);
+ int cpu = smp_processor_id();
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
+ if (!need_resched()) {
+ void (*idle)(void);
#ifdef CONFIG_SMP
- if (!need_resched())
min_xtp();
#endif
- while (!need_resched()) {
- void (*idle)(void);
-
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
@@ -284,17 +288,17 @@ cpu_idle (void)
if (!idle)
idle = default_idle;
(*idle)();
- }
-
- if (mark_idle)
- (*mark_idle)(0);
-
+ if (mark_idle)
+ (*mark_idle)(0);
#ifdef CONFIG_SMP
- normal_xtp();
+ normal_xtp();
#endif
+ }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
- if (cpu_is_offline(smp_processor_id()))
+ if (cpu_is_offline(cpu))
play_dead();
}
}
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/smpboot.c 2005-05-27 15:40:28.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/smpboot.c 2005-05-27 15:43:38.000000000 +1000
@@ -393,6 +393,8 @@ smp_callin (void)
int __devinit
start_secondary (void *unused)
{
+ preempt_disable();
+
/* Early console may use I/O ports */
ia64_set_kr(IA64_KR_IO_BASE, __pa(ia64_iobase));
Dprintk("start_secondary: starting CPU 0x%x\n", hard_smp_processor_id());
Index: linux-2.6/arch/ppc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/smp.c 2005-05-27 15:37:57.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/smp.c 2005-05-27 15:43:38.000000000 +1000
@@ -561,7 +561,10 @@ int __devinit __cpu_up(unsigned int cpu)
/* Activate a secondary processor. */
int __devinit start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/smp.c 2005-04-11 19:32:15.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/smp.c 2005-05-27 15:43:38.000000000 +1000
@@ -144,6 +144,9 @@ void __init smp_callin(void)
membar("#LoadLoad");
cpu_set(cpuid, cpu_online_map);
+
+ /* idle thread is expected to have preempt disabled */
+ preempt_disable();
}
void cpu_panic(void)
@@ -1167,20 +1170,9 @@ void __init smp_cpus_done(unsigned int m
(bogosum/(5000/HZ))%100);
}
-/* This needn't do anything as we do not sleep the cpu
- * inside of the idler task, so an interrupt is not needed
- * to get a clean fast response.
- *
- * XXX Reverify this assumption... -DaveM
- *
- * Addendum: We do want it to do something for the signal
- * delivery case, we detect that by just seeing
- * if we are trying to send this to an idler or not.
- */
void smp_send_reschedule(int cpu)
{
- if (cpu_data(cpu).idle_volume == 0)
- smp_receive_signal(cpu);
+ smp_receive_signal(cpu);
}
/* This is a nop because we capture all other cpus
Index: linux-2.6/arch/sparc64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/process.c 2005-05-27 15:37:58.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/process.c 2005-05-27 15:43:38.000000000 +1000
@@ -74,7 +74,9 @@ void cpu_idle(void)
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -83,21 +85,31 @@ void cpu_idle(void)
/*
* the idle loop on a UltraMultiPenguin...
+ *
+ * TIF_POLLING_NRFLAG is set because we do not sleep the cpu
+ * inside of the idler task, so an interrupt is not needed
+ * to get a clean fast response.
+ *
+ * XXX Reverify this assumption... -DaveM
+ *
+ * Addendum: We do want it to do something for the signal
+ * delivery case, we detect that by just seeing
+ * if we are trying to send this to an idler or not.
*/
-#define idle_me_harder() (cpu_data(smp_processor_id()).idle_volume += 1)
-#define unidle_me() (cpu_data(smp_processor_id()).idle_volume = 0)
void cpu_idle(void)
{
+ cpuinfo_sparc *cpuinfo = &local_cpu_data();
set_thread_flag(TIF_POLLING_NRFLAG);
+
while(1) {
if (need_resched()) {
- unidle_me();
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ cpuinfo->idle_volume = 0;
+ preempt_enable_no_resched();
schedule();
- set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
check_pgt_cache();
}
- idle_me_harder();
+ cpuinfo->idle_volume++;
/* The store ordering is so that IRQ handlers on
* other cpus see our increasing idleness for the buddy
Index: linux-2.6/arch/alpha/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/alpha/kernel/process.c 2004-10-19 17:20:02.000000000 +1000
+++ linux-2.6/arch/alpha/kernel/process.c 2005-05-27 16:17:11.000000000 +1000
@@ -43,22 +43,21 @@
#include "proto.h"
#include "pci_impl.h"
-void default_idle(void)
-{
- barrier();
-}
-
void
cpu_idle(void)
{
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- void (*idle)(void) = default_idle;
/* FIXME -- EV6 and LCA45 know how to power down
the CPU. */
while (!need_resched())
- idle();
+ cpu_relax();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/alpha/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/alpha/kernel/smp.c 2005-04-11 19:32:07.000000000 +1000
+++ linux-2.6/arch/alpha/kernel/smp.c 2005-05-27 16:05:05.000000000 +1000
@@ -128,7 +128,11 @@ wait_boot_cpu_to_stop(int cpuid)
void __init
smp_callin(void)
{
- int cpuid = hard_smp_processor_id();
+ int cpuid;
+
+ preempt_disable();
+
+ cpuid = hard_smp_processor_id();
if (cpu_test_and_set(cpuid, cpu_online_map)) {
printk("??, cpu 0x%x already present??\n", cpuid);
Index: linux-2.6/arch/s390/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/s390/kernel/smp.c 2005-05-27 15:40:29.000000000 +1000
+++ linux-2.6/arch/s390/kernel/smp.c 2005-05-27 16:41:53.000000000 +1000
@@ -528,6 +528,8 @@ extern void pfault_fini(void);
int __devinit start_secondary(void *cpuvoid)
{
+ preempt_disable();
+
/* Setup the cpu */
cpu_init();
/* init per CPU timer */
Index: linux-2.6/arch/sparc/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc/kernel/process.c 2005-05-27 15:37:58.000000000 +1000
+++ linux-2.6/arch/sparc/kernel/process.c 2005-05-27 16:56:29.000000000 +1000
@@ -72,6 +72,13 @@ struct thread_info *current_set[NR_CPUS]
*/
void default_idle(void)
{
+ if (pm_idle) {
+ while (!need_resched())
+ (*pm_idle)();
+ } else {
+ while (!need_resched())
+ cpu_relax();
+ }
}
#ifndef CONFIG_SMP
@@ -83,6 +90,8 @@ void default_idle(void)
*/
void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
for (;;) {
if (ARCH_SUN4C_SUN4) {
@@ -92,12 +101,11 @@ void cpu_idle(void)
static unsigned long fps;
unsigned long now;
unsigned long faults;
- unsigned long flags;
extern unsigned long sun4c_kernel_faults;
extern void sun4c_grow_kernel_ring(void);
- local_irq_save(flags);
+ local_irq_disable();
now = jiffies;
count -= (now - last_jiffies);
last_jiffies = now;
@@ -113,14 +121,14 @@ void cpu_idle(void)
sun4c_grow_kernel_ring();
}
}
- local_irq_restore(flags);
+ local_irq_enable();
}
- while((!need_resched()) && pm_idle) {
- (*pm_idle)();
- }
+ default_idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -130,13 +138,19 @@ void cpu_idle(void)
/* This is being executed in task 0 'user space'. */
void cpu_idle(void)
{
+
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while(1) {
- if(need_resched()) {
- schedule();
- check_pgt_cache();
- }
- barrier(); /* or else gcc optimizes... */
+ while (!need_resched())
+ cpu_relax();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ check_pgt_cache();
}
}
Index: linux-2.6/arch/ppc/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc/kernel/idle.c 2005-03-02 19:37:32.000000000 +1100
+++ linux-2.6/arch/ppc/kernel/idle.c 2005-05-27 16:40:35.000000000 +1000
@@ -50,8 +50,6 @@ void default_idle(void)
}
#endif
}
- if (need_resched())
- schedule();
}
/*
@@ -59,11 +57,20 @@ void default_idle(void)
*/
void cpu_idle(void)
{
- for (;;)
- if (ppc_md.idle != NULL)
- ppc_md.idle();
- else
- default_idle();
+ set_need_resched();
+
+ for (;;) {
+ while (need_resched()) {
+ if (ppc_md.idle != NULL)
+ ppc_md.idle();
+ else
+ default_idle();
+ }
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
#if defined(CONFIG_SYSCTL) && defined(CONFIG_6xx)
Index: linux-2.6/arch/m32r/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m32r/kernel/process.c 2005-03-02 19:38:48.000000000 +1100
+++ linux-2.6/arch/m32r/kernel/process.c 2005-05-27 16:31:57.000000000 +1000
@@ -94,6 +94,8 @@ static void poll_idle (void)
*/
void cpu_idle (void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched()) {
@@ -104,7 +106,9 @@ void cpu_idle (void)
idle();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/frv/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/frv/kernel/process.c 2005-03-02 19:37:24.000000000 +1100
+++ linux-2.6/arch/frv/kernel/process.c 2005-05-27 16:19:12.000000000 +1000
@@ -77,16 +77,21 @@ void (*idle)(void) = core_sleep_idle;
*/
void cpu_idle(void)
{
+ int cpu = smp_processor_id();
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched()) {
- irq_stat[smp_processor_id()].idle_timestamp = jiffies;
+ irq_stat[cpu].idle_timestamp = jiffies;
if (!frv_dma_inprogress && idle)
idle();
}
-
+
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/cris/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/cris/kernel/process.c 2004-08-15 11:11:02.000000000 +1000
+++ linux-2.6/arch/cris/kernel/process.c 2005-05-27 16:17:58.000000000 +1000
@@ -191,6 +191,8 @@ extern void default_idle(void);
*/
void cpu_idle (void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched()) {
@@ -201,7 +203,9 @@ void cpu_idle (void)
idle();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/mips/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/mips/kernel/smp.c 2005-03-30 10:39:06.000000000 +1000
+++ linux-2.6/arch/mips/kernel/smp.c 2005-05-27 16:36:18.000000000 +1000
@@ -83,7 +83,11 @@ extern ATTRIB_NORET void cpu_idle(void);
*/
asmlinkage void start_secondary(void)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+
+ cpu = smp_processor_id();
cpu_probe();
cpu_report();
Index: linux-2.6/arch/parisc/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/parisc/kernel/process.c 2005-03-30 10:39:06.000000000 +1000
+++ linux-2.6/arch/parisc/kernel/process.c 2005-05-27 16:38:19.000000000 +1000
@@ -88,11 +88,16 @@ void default_idle(void)
*/
void cpu_idle(void)
{
+ set_need_resched();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
Index: linux-2.6/arch/ppc/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc/kernel/smp.c 2005-03-30 10:39:08.000000000 +1000
+++ linux-2.6/arch/ppc/kernel/smp.c 2005-05-27 16:41:15.000000000 +1000
@@ -326,6 +326,8 @@ int __devinit start_secondary(void *unus
{
int cpu;
+ preempt_disable();
+
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sh/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sh/kernel/process.c 2005-03-02 19:37:35.000000000 +1100
+++ linux-2.6/arch/sh/kernel/process.c 2005-05-27 16:46:21.000000000 +1000
@@ -51,28 +51,26 @@ void enable_hlt(void)
EXPORT_SYMBOL(enable_hlt);
-void default_idle(void)
+void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
if (hlt_counter) {
- while (1)
- if (need_resched())
- break;
+ while (!need_resched())
+ cpu_relax();
} else {
while (!need_resched())
cpu_sleep();
}
+ preempt_disable_no_resched();
schedule();
+ preempt_enable();
}
}
-void cpu_idle(void)
-{
- default_idle();
-}
-
void machine_restart(char * __unused)
{
/* SR.BL=1 and invoke address error to let CPU reset (manual reset) */
Index: linux-2.6/arch/m68k/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m68k/kernel/process.c 2004-10-19 17:20:06.000000000 +1000
+++ linux-2.6/arch/m68k/kernel/process.c 2005-05-27 16:32:57.000000000 +1000
@@ -98,11 +98,15 @@ void (*idle)(void) = default_idle;
*/
void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched())
idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/mips/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/mips/kernel/process.c 2005-03-02 19:37:29.000000000 +1100
+++ linux-2.6/arch/mips/kernel/process.c 2005-05-27 16:37:18.000000000 +1000
@@ -53,12 +53,16 @@ void default_idle (void)
*/
ATTRIB_NORET void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched())
if (cpu_wait)
(*cpu_wait)();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable()
}
}
Index: linux-2.6/arch/m68knommu/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m68knommu/kernel/process.c 2005-03-02 19:37:27.000000000 +1100
+++ linux-2.6/arch/m68knommu/kernel/process.c 2005-05-27 16:35:50.000000000 +1000
@@ -45,11 +45,8 @@ asmlinkage void ret_from_fork(void);
*/
void default_idle(void)
{
- while(1) {
- if (need_resched())
- __asm__("stop #0x2000" : : : "cc");
- schedule();
- }
+ while (!need_resched())
+ __asm__("stop #0x2000" : : : "cc");
}
void (*idle)(void) = default_idle;
@@ -62,8 +59,15 @@ void (*idle)(void) = default_idle;
*/
void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
- idle();
+ while (1) {
+ idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/sh/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sh/kernel/smp.c 2005-03-30 10:39:09.000000000 +1000
+++ linux-2.6/arch/sh/kernel/smp.c 2005-05-27 16:46:35.000000000 +1000
@@ -109,7 +109,11 @@ int __cpu_up(unsigned int cpu)
int start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/parisc/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/parisc/kernel/smp.c 2005-03-30 10:39:06.000000000 +1000
+++ linux-2.6/arch/parisc/kernel/smp.c 2005-05-27 16:37:48.000000000 +1000
@@ -462,6 +462,8 @@ void __init smp_callin(void)
void *istack;
#endif
+ preempt_disable();
+
smp_cpu_init(slave_id);
#if 0 /* NOT WORKING YET - see entry.S */
Index: linux-2.6/arch/m32r/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/m32r/kernel/smpboot.c 2005-03-30 10:39:05.000000000 +1000
+++ linux-2.6/arch/m32r/kernel/smpboot.c 2005-05-27 16:32:12.000000000 +1000
@@ -424,6 +424,7 @@ void __init smp_cpus_done(unsigned int m
*==========================================================================*/
int __init start_secondary(void *unused)
{
+ preempt_disable();
cpu_init();
smp_callin();
while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
Index: linux-2.6/arch/s390/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/s390/kernel/process.c 2005-03-02 19:37:34.000000000 +1100
+++ linux-2.6/arch/s390/kernel/process.c 2005-05-27 16:44:28.000000000 +1000
@@ -101,11 +101,6 @@ void default_idle(void)
int cpu, rc;
local_irq_disable();
- if (need_resched()) {
- local_irq_enable();
- schedule();
- return;
- }
/* CPU is going idle. */
cpu = smp_processor_id();
@@ -121,7 +116,7 @@ void default_idle(void)
__ctl_set_bit(8, 15);
#ifdef CONFIG_HOTPLUG_CPU
- if (cpu_is_offline(smp_processor_id()))
+ if (cpu_is_offline(cpu))
cpu_die();
#endif
@@ -161,8 +156,15 @@ void default_idle(void)
void cpu_idle(void)
{
- for (;;)
- default_idle();
+ set_need_resched();
+
+ for (;;) {
+ while (!need_resched())
+ default_idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void show_regs(struct pt_regs *regs)
Index: linux-2.6/arch/sh64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sh64/kernel/process.c 2005-03-30 10:39:10.000000000 +1000
+++ linux-2.6/arch/sh64/kernel/process.c 2005-05-27 16:49:18.000000000 +1000
@@ -307,23 +307,21 @@ __setup("hlt", hlt_setup);
static inline void hlt(void)
{
- if (hlt_counter)
- return;
-
__asm__ __volatile__ ("sleep" : : : "memory");
}
/*
* The idle loop on a uniprocessor SH..
*/
-void default_idle(void)
+void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
while (1) {
if (hlt_counter) {
- while (1)
- if (need_resched())
- break;
+ while (!need_resched())
+ cpu_relax();
} else {
local_irq_disable();
while (!need_resched()) {
@@ -334,13 +332,11 @@ void default_idle(void)
}
local_irq_enable();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
-}
-void cpu_idle(void)
-{
- default_idle();
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/arm26/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/arm26/kernel/process.c 2005-03-02 19:37:23.000000000 +1100
+++ linux-2.6/arch/arm26/kernel/process.c 2005-05-27 16:16:41.000000000 +1000
@@ -73,16 +73,16 @@ __setup("hlt", hlt_setup);
*/
void cpu_idle(void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
- preempt_disable();
while (1) {
- while (!need_resched()) {
- local_irq_disable();
- if (!need_resched() && !hlt_counter)
- local_irq_enable();
- }
+ while (!need_resched())
+ cpu_relax();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
}
- schedule();
}
static char reboot_mode = 'h';
Index: linux-2.6/arch/arm/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/arm/kernel/process.c 2005-05-27 15:37:54.000000000 +1000
+++ linux-2.6/arch/arm/kernel/process.c 2005-05-27 16:14:46.000000000 +1000
@@ -84,10 +84,14 @@ EXPORT_SYMBOL(pm_power_off);
*/
void default_idle(void)
{
- local_irq_disable();
- if (!need_resched() && !hlt_counter)
- arch_idle();
- local_irq_enable();
+ if (hlt_counter)
+ cpu_relax()
+ else {
+ local_irq_disable();
+ if (!need_resched())
+ arch_idle();
+ local_irq_enable();
+ }
}
/*
@@ -97,6 +101,7 @@ void default_idle(void)
*/
void cpu_idle(void)
{
+ set_need_resched();
local_fiq_enable();
/* endless idle loop with no priority at all */
@@ -104,13 +109,13 @@ void cpu_idle(void)
void (*idle)(void) = pm_idle;
if (!idle)
idle = default_idle;
- preempt_disable();
leds_event(led_idle_start);
while (!need_resched())
idle();
leds_event(led_idle_end);
- preempt_enable();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/h8300/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/h8300/kernel/process.c 2004-10-19 17:20:03.000000000 +1000
+++ linux-2.6/arch/h8300/kernel/process.c 2005-05-27 16:30:44.000000000 +1000
@@ -53,22 +53,17 @@ asmlinkage void ret_from_fork(void);
#if !defined(CONFIG_H8300H_SIM) && !defined(CONFIG_H8S_SIM)
void default_idle(void)
{
- while(1) {
- if (need_resched()) {
- local_irq_enable();
- __asm__("sleep");
- local_irq_disable();
- }
- schedule();
+ local_irq_disable();
+ if (need_resched()) {
+ local_irq_enable();
+ /* XXX: race here! What if need_resched() gets set now? */
+ __asm__("sleep");
}
}
#else
void default_idle(void)
{
- while(1) {
- if (need_resched())
- schedule();
- }
+ cpu_relax();
}
#endif
void (*idle)(void) = default_idle;
@@ -81,7 +76,14 @@ void (*idle)(void) = default_idle;
*/
void cpu_idle(void)
{
- idle();
+ set_need_resched();
+ while (1) {
+ while (!need_resched())
+ idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/xtensa/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/xtensa/kernel/process.c 2005-05-27 15:40:31.000000000 +1000
+++ linux-2.6/arch/xtensa/kernel/process.c 2005-05-27 17:02:46.000000000 +1000
@@ -91,13 +91,15 @@ coprocessor_info_t coprocessor_info[] =
void cpu_idle(void)
{
local_irq_enable();
+ set_need_resched();
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched())
platform_idle();
- preempt_enable();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/v850/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/v850/kernel/process.c 2004-06-24 18:59:47.000000000 +1000
+++ linux-2.6/arch/v850/kernel/process.c 2005-05-27 17:01:42.000000000 +1000
@@ -36,11 +36,8 @@ extern void ret_from_fork (void);
/* The idle loop. */
void default_idle (void)
{
- while (1) {
- while (! need_resched ())
- asm ("halt; nop; nop; nop; nop; nop" ::: "cc");
- schedule ();
- }
+ while (! need_resched ())
+ asm ("halt; nop; nop; nop; nop; nop" ::: "cc");
}
void (*idle)(void) = default_idle;
@@ -53,8 +50,17 @@ void (*idle)(void) = default_idle;
*/
void cpu_idle (void)
{
+ set_need_resched();
+
/* endless idle loop with no priority at all */
- (*idle) ();
+ while (1) {
+ while (!need_resched())
+ (*idle) ();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
/*
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 7:21 Nick Piggin
@ 2005-05-27 8:57 ` Ingo Molnar
2005-05-27 9:11 ` Nick Piggin
2005-05-27 9:37 ` Nick Piggin
0 siblings, 2 replies; 15+ messages in thread
From: Ingo Molnar @ 2005-05-27 8:57 UTC (permalink / raw)
To: Nick Piggin; +Cc: Andrew Morton, linux-kernel
* Nick Piggin <piggin@cyberone.com.au> wrote:
> OK, done a bit of work on all other architectures, and diffed to the
> latest -mm. Any chance you can put it in -mm, Andrew?
>
> Also, while I was there, I thought I'd add the set_need_resched()
> thing to all the other architectures. I couldn't be bothered doing 2
> patches, sorry.
the need_resched changes are not needed meanwhile - we can do the first
schedule() in rest_init() just fine. (See my earlier patch below.) So
please keep the need_resched thing out of your patch.
----
The patch below should address this problem for all architectures, by
doing an explicit schedule() in the init code before calling into
cpu_idle(). It's a replacement for the following patch:
sched-remove-set_tsk_need_resched-from-init_idle.patch
Ingo
--
This patch tweaks idle thread setup semantics a bit: instead of setting
NEED_RESCHED in init_idle(), we do an explicit schedule() before
calling into cpu_idle().
This patch, while having no negative side-effects, enables wider use of
cond_resched()s. (which might happen in the stock kernel too, but it's
particulary important for voluntary-preempt)
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@osdl.org>
--- linux/kernel/sched.c.orig
+++ linux/kernel/sched.c
@@ -4163,6 +4163,14 @@ void show_state(void)
read_unlock(&tasklist_lock);
}
+/**
+ * init_idle - set up an idle thread for a given CPU
+ * @idle: task in question
+ * @cpu: cpu the idle task belongs to
+ *
+ * NOTE: this function does not set the idle thread's NEED_RESCHED
+ * flag, to make booting more robust.
+ */
void __devinit init_idle(task_t *idle, int cpu)
{
runqueue_t *rq = cpu_rq(cpu);
@@ -4180,7 +4188,6 @@ void __devinit init_idle(task_t *idle, i
#if defined(CONFIG_SMP) && defined(__ARCH_WANT_UNLOCKED_CTXSW)
idle->oncpu = 1;
#endif
- set_tsk_need_resched(idle);
spin_unlock_irqrestore(&rq->lock, flags);
/* Set the preempt count _outside_ the spinlocks! */
--- linux/init/main.c.orig
+++ linux/init/main.c
@@ -383,6 +383,13 @@ static void noinline rest_init(void)
numa_default_policy();
unlock_kernel();
preempt_enable_no_resched();
+
+ /*
+ * The boot idle thread must execute schedule()
+ * at least once to get things moving:
+ */
+ schedule();
+
cpu_idle();
}
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 8:57 ` Ingo Molnar
@ 2005-05-27 9:11 ` Nick Piggin
2005-05-27 9:20 ` Ingo Molnar
2005-05-27 9:37 ` Nick Piggin
1 sibling, 1 reply; 15+ messages in thread
From: Nick Piggin @ 2005-05-27 9:11 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Nick Piggin, Andrew Morton, linux-kernel
Ingo Molnar wrote:
> * Nick Piggin <piggin@cyberone.com.au> wrote:
>
>
>>OK, done a bit of work on all other architectures, and diffed to the
>>latest -mm. Any chance you can put it in -mm, Andrew?
>>
>>Also, while I was there, I thought I'd add the set_need_resched()
>>thing to all the other architectures. I couldn't be bothered doing 2
>>patches, sorry.
>
>
> the need_resched changes are not needed meanwhile - we can do the first
> schedule() in rest_init() just fine. (See my earlier patch below.) So
> please keep the need_resched thing out of your patch.
>
OK that's better. Sorry I didn't see your patch earlier.
I'll redo this patch. Coming up...
Send instant messages to your online friends http://au.messenger.yahoo.com
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 9:11 ` Nick Piggin
@ 2005-05-27 9:20 ` Ingo Molnar
0 siblings, 0 replies; 15+ messages in thread
From: Ingo Molnar @ 2005-05-27 9:20 UTC (permalink / raw)
To: Andrew Morton; +Cc: Nick Piggin, Andrew Morton, linux-kernel
* Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >the need_resched changes are not needed meanwhile - we can do the first
> >schedule() in rest_init() just fine. (See my earlier patch below.) So
> >please keep the need_resched thing out of your patch.
> >
>
> OK that's better. Sorry I didn't see your patch earlier.
>
> I'll redo this patch. Coming up...
Andrew: please drop the following two patches:
sched-remove-set_tsk_need_resched-from-init_idle-v2.patch
sched-remove-set_tsk_need_resched-from-init_idle-v2-ia64-fix.patch
and add the one below. The only followup patch
(sched-voluntary-kernel-preemption.patch) should still apply cleanly.
Nick's upcoming patch can then come afterwards.
----
This patch tweaks idle thread setup semantics a bit: instead of setting
NEED_RESCHED in init_idle(), we do an explicit schedule() before
calling into cpu_idle().
This patch, while having no negative side-effects, enables wider use of
cond_resched()s. (which might happen in the stock kernel too, but it's
particulary important for voluntary-preempt)
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Acked-by: Nick Piggin <nickpiggin@yahoo.com.au>
Signed-off-by: Andrew Morton <akpm@osdl.org>
--- linux/kernel/sched.c.orig
+++ linux/kernel/sched.c
@@ -4163,6 +4163,14 @@ void show_state(void)
read_unlock(&tasklist_lock);
}
+/**
+ * init_idle - set up an idle thread for a given CPU
+ * @idle: task in question
+ * @cpu: cpu the idle task belongs to
+ *
+ * NOTE: this function does not set the idle thread's NEED_RESCHED
+ * flag, to make booting more robust.
+ */
void __devinit init_idle(task_t *idle, int cpu)
{
runqueue_t *rq = cpu_rq(cpu);
@@ -4180,7 +4188,6 @@ void __devinit init_idle(task_t *idle, i
#if defined(CONFIG_SMP) && defined(__ARCH_WANT_UNLOCKED_CTXSW)
idle->oncpu = 1;
#endif
- set_tsk_need_resched(idle);
spin_unlock_irqrestore(&rq->lock, flags);
/* Set the preempt count _outside_ the spinlocks! */
--- linux/init/main.c.orig
+++ linux/init/main.c
@@ -383,6 +383,13 @@ static void noinline rest_init(void)
numa_default_policy();
unlock_kernel();
preempt_enable_no_resched();
+
+ /*
+ * The boot idle thread must execute schedule()
+ * at least once to get things moving:
+ */
+ schedule();
+
cpu_idle();
}
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 8:57 ` Ingo Molnar
2005-05-27 9:11 ` Nick Piggin
@ 2005-05-27 9:37 ` Nick Piggin
2005-05-27 10:12 ` Ingo Molnar
2005-06-01 6:15 ` Andrew Morton
1 sibling, 2 replies; 15+ messages in thread
From: Nick Piggin @ 2005-05-27 9:37 UTC (permalink / raw)
To: Ingo Molnar; +Cc: Nick Piggin, Andrew Morton, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 709 bytes --]
Ingo Molnar wrote:
> * Nick Piggin <piggin@cyberone.com.au> wrote:
>
>
>>OK, done a bit of work on all other architectures, and diffed to the
>>latest -mm. Any chance you can put it in -mm, Andrew?
>>
>>Also, while I was there, I thought I'd add the set_need_resched()
>>thing to all the other architectures. I couldn't be bothered doing 2
>>patches, sorry.
>
>
> the need_resched changes are not needed meanwhile - we can do the first
> schedule() in rest_init() just fine. (See my earlier patch below.) So
> please keep the need_resched thing out of your patch.
The following patch is with your patch (and the ia64 fix) from -mm
backed out, and the below patch applied.
--
SUSE Labs, Novell Inc.
[-- Attachment #2: sched-resched-opt.patch --]
[-- Type: text/plain, Size: 42934 bytes --]
Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
confusion, and make their semantics rigid. Also have preempt explicitly
disabled in idle routines. Improves efficiency of resched_task and some
cpu_idle routines.
* In resched_task:
- TIF_NEED_RESCHED is only cleared with the task's runqueue lock held,
and as we hold it during resched_task, then there is no need for an
atomic test and set there. The only other time this should be set is
when the task's quantum expires, in the timer interrupt - this is
protected against because the rq lock is irq-safe.
- If TIF_NEED_RESCHED is set, then we don't need to do anything. It
won't get unset until the task get's schedule()d off.
- If we are running on the same CPU as the task we resched, then set
TIF_NEED_RESCHED and no further action is required.
- If we are running on another CPU, and TIF_POLLING_NRFLAG is *not* set
after TIF_NEED_RESCHED has been set, then we need to send an IPI.
Using these rules, we are able to remove the test and set operation in
resched_task, and make clear the previously vague semantics of POLLING_NRFLAG.
* In idle routines:
- Enter cpu_idle with preempt disabled. When the need_resched() condition
becomes true, explicitly call schedule(). This makes things a bit clearer
(IMO), but haven't updated all architectures yet.
- Many do a test and clear of TIF_NEED_RESCHED for some reason. According
to the resched_task rules, this isn't needed (and actually breaks the
assumption that TIF_NEED_RESCHED is only cleared with the runqueue lock
held). So remove that. Generally one less locked memory op when switching
to the idle thread.
- Many idle routines clear TIF_POLLING_NRFLAG, and only set it in the inner
most polling idle loops. The above resched_task semantics allow it to be
set until before the last time need_resched() is checked before going into
a halt requiring interrupt wakeup.
Many idle routines simply never enter such a halt, and so POLLING_NRFLAG
can be always left set, completely eliminating resched IPIs when rescheduling
the idle task.
POLLING_NRFLAG width can be increased, to reduce the chance of resched IPIs.
Index: linux-2.6/kernel/sched.c
===================================================================
--- linux-2.6.orig/kernel/sched.c 2005-05-27 19:22:52.000000000 +1000
+++ linux-2.6/kernel/sched.c 2005-05-27 19:26:19.000000000 +1000
@@ -845,21 +845,28 @@ static void deactivate_task(struct task_
#ifdef CONFIG_SMP
static void resched_task(task_t *p)
{
- int need_resched, nrpolling;
+ int cpu;
assert_spin_locked(&task_rq(p)->lock);
- /* minimise the chance of sending an interrupt to poll_idle() */
- nrpolling = test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
- need_resched = test_and_set_tsk_thread_flag(p,TIF_NEED_RESCHED);
- nrpolling |= test_tsk_thread_flag(p,TIF_POLLING_NRFLAG);
+ if (unlikely(test_tsk_thread_flag(p, TIF_NEED_RESCHED)))
+ return;
+
+ set_tsk_thread_flag(p, TIF_NEED_RESCHED);
- if (!need_resched && !nrpolling && (task_cpu(p) != smp_processor_id()))
- smp_send_reschedule(task_cpu(p));
+ cpu = task_cpu(p);
+ if (cpu == smp_processor_id())
+ return;
+
+ /* NEED_RESCHED must be visible before we test POLLING_NRFLAG */
+ smp_mb();
+ if (!test_tsk_thread_flag(p, TIF_POLLING_NRFLAG))
+ smp_send_reschedule(cpu);
}
#else
static inline void resched_task(task_t *p)
{
+ assert_spin_locked(&task_rq(p)->lock);
set_tsk_need_resched(p);
}
#endif
Index: linux-2.6/arch/i386/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/process.c 2005-05-27 19:22:51.000000000 +1000
+++ linux-2.6/arch/i386/kernel/process.c 2005-05-27 19:33:27.000000000 +1000
@@ -102,14 +102,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!hlt_counter && boot_cpu_data.hlt_works_ok) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
} else {
- cpu_relax();
+ while (!need_resched())
+ cpu_relax();
}
}
#ifdef CONFIG_APM_MODULE
@@ -123,29 +128,14 @@ EXPORT_SYMBOL(default_idle);
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0, %1;"
- "rep; nop;"
- "je 2b;"
- : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
-
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0, %1;"
+ "rep; nop;"
+ "je 2b;"
+ : : "i"(_TIF_NEED_RESCHED), "m" (current_thread_info()->flags));
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -181,29 +171,32 @@ static inline void play_dead(void)
*/
void cpu_idle(void)
{
- int cpu = _smp_processor_id();
+ int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
+
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
+
+ rmb();
+ idle = pm_idle;
+
+ if (!idle)
+ idle = default_idle;
+
+ if (cpu_is_offline(cpu))
+ play_dead();
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ __get_cpu_var(irq_stat).idle_timestamp = jiffies;
+ idle();
- rmb();
- idle = pm_idle;
-
- if (!idle)
- idle = default_idle;
-
- if (cpu_is_offline(cpu))
- play_dead();
-
- __get_cpu_var(irq_stat).idle_timestamp = jiffies;
- idle();
- }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -246,15 +239,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/init/main.c
===================================================================
--- linux-2.6.orig/init/main.c 2005-05-27 19:22:52.000000000 +1000
+++ linux-2.6/init/main.c 2005-05-27 19:27:13.000000000 +1000
@@ -382,13 +382,14 @@ static void noinline rest_init(void)
kernel_thread(init, NULL, CLONE_FS | CLONE_SIGHAND);
numa_default_policy();
unlock_kernel();
- preempt_enable_no_resched();
/*
* The boot idle thread must execute schedule()
* at least once to get things moving:
*/
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
cpu_idle();
}
Index: linux-2.6/arch/i386/kernel/apm.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/apm.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/i386/kernel/apm.c 2005-05-27 19:26:19.000000000 +1000
@@ -767,8 +767,20 @@ static int set_system_power_state(u_shor
static int apm_do_idle(void)
{
u32 eax;
+ u8 ret;
+ int idled = 0;
- if (apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax)) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ if (!need_resched()) {
+ idled = 1;
+ ret = apm_bios_call_simple(APM_FUNC_IDLE, 0, 0, &eax);
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ if (!idled)
+ return 0;
+
+ if (ret) {
static unsigned long t;
/* This always fails on some SMP boards running UP kernels.
Index: linux-2.6/drivers/acpi/processor_idle.c
===================================================================
--- linux-2.6.orig/drivers/acpi/processor_idle.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/drivers/acpi/processor_idle.c 2005-05-27 19:26:19.000000000 +1000
@@ -164,6 +164,14 @@ acpi_processor_power_activate (
return;
}
+static void acpi_safe_halt (void)
+{
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
+ safe_halt();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+}
static atomic_t c3_cpu_count;
@@ -176,7 +184,7 @@ static void acpi_processor_idle (void)
int sleep_ticks = 0;
u32 t1, t2 = 0;
- pr = processors[_smp_processor_id()];
+ pr = processors[smp_processor_id()];
if (!pr)
return;
@@ -196,8 +204,13 @@ static void acpi_processor_idle (void)
}
cx = pr->power.state;
- if (!cx)
- goto easy_out;
+ if (!cx) {
+ if (pm_idle_save)
+ pm_idle_save();
+ else
+ acpi_safe_halt();
+ return;
+ }
/*
* Check BM Activity
@@ -277,7 +290,8 @@ static void acpi_processor_idle (void)
if (pm_idle_save)
pm_idle_save();
else
- safe_halt();
+ acpi_safe_halt();
+
/*
* TBD: Can't get time duration while in C1, as resumes
* go to an ISR rather than here. Need to instrument
@@ -407,16 +421,6 @@ end:
*/
if (next_state != pr->power.state)
acpi_processor_power_activate(pr, next_state);
-
- return;
-
- easy_out:
- /* do C1 instead of busy loop */
- if (pm_idle_save)
- pm_idle_save();
- else
- safe_halt();
- return;
}
Index: linux-2.6/arch/i386/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/i386/kernel/smpboot.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/i386/kernel/smpboot.c 2005-05-27 19:26:19.000000000 +1000
@@ -477,6 +477,8 @@ set_cpu_sibling_map(int cpu)
*/
static void __devinit start_secondary(void *unused)
{
+ preempt_disable();
+
/*
* Dont put anything before smp_callin(), SMP
* booting is too fragile that we want to limit the
Index: linux-2.6/arch/x86_64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/x86_64/kernel/process.c 2005-05-27 19:22:51.000000000 +1000
+++ linux-2.6/arch/x86_64/kernel/process.c 2005-05-27 19:34:32.000000000 +1000
@@ -85,12 +85,19 @@ EXPORT_SYMBOL(enable_hlt);
*/
void default_idle(void)
{
+ local_irq_enable();
+
if (!atomic_read(&hlt_counter)) {
- local_irq_disable();
- if (!need_resched())
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched()) {
+ local_irq_disable();
safe_halt();
- else
- local_irq_enable();
+ }
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
+ cpu_relax();
}
}
@@ -101,29 +108,16 @@ void default_idle(void)
*/
static void poll_idle (void)
{
- int oldval;
-
local_irq_enable();
- /*
- * Deal with another CPU just having chosen a thread to
- * run here:
- */
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- asm volatile(
- "2:"
- "testl %0,%1;"
- "rep; nop;"
- "je 2b;"
- : :
- "i" (_TIF_NEED_RESCHED),
- "m" (current_thread_info()->flags));
- } else {
- set_need_resched();
- }
+ asm volatile(
+ "2:"
+ "testl %0,%1;"
+ "rep; nop;"
+ "je 2b;"
+ : :
+ "i" (_TIF_NEED_RESCHED),
+ "m" (current_thread_info()->flags));
}
void cpu_idle_wait(void)
@@ -162,22 +156,25 @@ EXPORT_SYMBOL_GPL(cpu_idle_wait);
*/
void cpu_idle (void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
- while (!need_resched()) {
- void (*idle)(void);
+ void (*idle)(void);
- if (__get_cpu_var(cpu_idle_state))
- __get_cpu_var(cpu_idle_state) = 0;
+ if (__get_cpu_var(cpu_idle_state))
+ __get_cpu_var(cpu_idle_state) = 0;
- rmb();
- idle = pm_idle;
- if (!idle)
- idle = default_idle;
- idle();
- }
+ rmb();
+ idle = pm_idle;
+ if (!idle)
+ idle = default_idle;
+
+ idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
@@ -192,15 +189,12 @@ static void mwait_idle(void)
{
local_irq_enable();
- if (!need_resched()) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- do {
- __monitor((void *)¤t_thread_info()->flags, 0, 0);
- if (need_resched())
- break;
- __mwait(0, 0);
- } while (!need_resched());
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ while (!need_resched()) {
+ __monitor((void *)¤t_thread_info()->flags, 0, 0);
+ smp_mb();
+ if (need_resched())
+ break;
+ __mwait(0, 0);
}
}
Index: linux-2.6/arch/ppc64/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/idle.c 2005-05-27 19:22:51.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/idle.c 2005-05-27 19:26:19.000000000 +1000
@@ -74,9 +74,10 @@ static void yield_shared_processor(void)
static int iSeries_idle(void)
{
struct paca_struct *lpaca;
- long oldval;
unsigned long CTRL;
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* ensure iSeries run light will be out when idle */
clear_thread_flag(TIF_RUN_LIGHT);
CTRL = mfspr(CTRLF);
@@ -86,32 +87,21 @@ static int iSeries_idle(void)
lpaca = get_paca();
while (1) {
- if (lpaca->lppaca.shared_proc) {
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- if (!need_resched())
- yield_shared_processor();
- } else {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched()) {
- HMT_medium();
- if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr))
- process_iSeries_events();
- HMT_low();
- }
-
+ while (!need_resched()) {
+ HMT_low();
+ if (ItLpQueue_isLpIntPending(lpaca->lpqueue_ptr)) {
HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ process_iSeries_events();
+ HMT_low();
}
+ if (lpaca->lppaca.shared_proc)
+ yield_shared_processor();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
return 0;
@@ -121,32 +111,24 @@ static int iSeries_idle(void)
static int default_idle(void)
{
- long oldval;
unsigned int cpu = smp_processor_id();
-
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
-
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
-
- while (!need_resched() && !cpu_is_offline(cpu)) {
- barrier();
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- }
-
- HMT_medium();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ barrier();
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
}
+ HMT_medium();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -160,12 +142,12 @@ DECLARE_PER_CPU(unsigned long, smt_snooz
int dedicated_idle(void)
{
- long oldval;
struct paca_struct *lpaca = get_paca(), *ppaca;
unsigned long start_snooze;
unsigned long *smt_snooze_delay = &__get_cpu_var(smt_snooze_delay);
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
ppaca = &paca[cpu ^ 1];
while (1) {
@@ -175,66 +157,67 @@ int dedicated_idle(void)
*/
lpaca->lppaca.idle = 1;
- oldval = test_and_clear_thread_flag(TIF_NEED_RESCHED);
- if (!oldval) {
- set_thread_flag(TIF_POLLING_NRFLAG);
- start_snooze = __get_tb() +
+ start_snooze = __get_tb() +
*smt_snooze_delay * tb_ticks_per_usec;
- while (!need_resched() && !cpu_is_offline(cpu)) {
- /*
- * Go into low thread priority and possibly
- * low power mode.
- */
- HMT_low();
- HMT_very_low();
- if (*smt_snooze_delay == 0 ||
- __get_tb() < start_snooze)
- continue;
+ while (!need_resched() && !cpu_is_offline(cpu)) {
+ /*
+ * Go into low thread priority and possibly
+ * low power mode.
+ */
+ HMT_low();
+ HMT_very_low();
- HMT_medium();
+ if (*smt_snooze_delay == 0 || __get_tb() < start_snooze)
+ continue;
- if (!(ppaca->lppaca.idle)) {
- local_irq_disable();
+ HMT_medium();
- /*
- * We are about to sleep the thread
- * and so wont be polling any
- * more.
- */
- clear_thread_flag(TIF_POLLING_NRFLAG);
-
- /*
- * SMT dynamic mode. Cede will result
- * in this thread going dormant, if the
- * partner thread is still doing work.
- * Thread wakes up if partner goes idle,
- * an interrupt is presented, or a prod
- * occurs. Returning from the cede
- * enables external interrupts.
- */
- if (!need_resched())
- cede_processor();
- else
- local_irq_enable();
- } else {
- /*
- * Give the HV an opportunity at the
- * processor, since we are not doing
- * any work.
- */
- poll_pending();
- }
- }
+ if (!(ppaca->lppaca.idle)) {
+ local_irq_disable();
- clear_thread_flag(TIF_POLLING_NRFLAG);
- } else {
- set_need_resched();
+ /*
+ * We are about to sleep the thread
+ * and so wont be polling any
+ * more.
+ */
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+
+ /*
+ * Must have TIF_POLLING_NRFLAG clear visible
+ * before checking need_resched
+ */
+ smp_mb__after_clear_bit();
+
+ /*
+ * SMT dynamic mode. Cede will result
+ * in this thread going dormant, if the
+ * partner thread is still doing work.
+ * Thread wakes up if partner goes idle,
+ * an interrupt is presented, or a prod
+ * occurs. Returning from the cede
+ * enables external interrupts.
+ */
+ if (!need_resched())
+ cede_processor();
+ else
+ local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ /*
+ * Give the HV an opportunity at the
+ * processor, since we are not doing
+ * any work.
+ */
+ poll_pending();
+ }
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(cpu) && system_state == SYSTEM_RUNNING)
cpu_die();
}
@@ -245,6 +228,7 @@ static int shared_idle(void)
{
struct paca_struct *lpaca = get_paca();
unsigned int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
while (1) {
/*
@@ -256,6 +240,9 @@ static int shared_idle(void)
while (!need_resched() && !cpu_is_offline(cpu)) {
local_irq_disable();
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+
/*
* Yield the processor to the hypervisor. We return if
* an external interrupt occurs (which are driven prior
@@ -270,11 +257,14 @@ static int shared_idle(void)
cede_processor();
else
local_irq_enable();
+ set_thread_flag(TIF_POLLING_NRFLAG);
}
HMT_medium();
lpaca->lppaca.idle = 0;
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
if (cpu_is_offline(smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
cpu_die();
@@ -289,10 +279,12 @@ static int native_idle(void)
{
while(1) {
/* check CPU type here */
- if (!need_resched())
+ while (!need_resched())
power4_idle();
- if (need_resched())
- schedule();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
if (cpu_is_offline(_smp_processor_id()) &&
system_state == SYSTEM_RUNNING)
Index: linux-2.6/arch/ia64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/process.c 2005-05-27 19:22:51.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/process.c 2005-05-27 19:35:34.000000000 +1000
@@ -195,11 +195,16 @@ update_pal_halt_status(int status)
void
default_idle (void)
{
- while (!need_resched())
- if (can_do_pal_halt)
+ if (can_do_pal_halt) {
+ clear_thread_flag(TIF_POLLING_NRFLAG);
+ smp_mb__after_clear_bit();
+ while (!need_resched())
safe_halt();
- else
+ set_thread_flag(TIF_POLLING_NRFLAG);
+ } else {
+ while (!need_resched())
cpu_relax();
+ }
}
#ifdef CONFIG_HOTPLUG_CPU
@@ -261,16 +266,16 @@ void __attribute__((noreturn))
cpu_idle (void)
{
void (*mark_idle)(int) = ia64_mark_idle;
+ int cpu = smp_processor_id();
+ set_thread_flag(TIF_POLLING_NRFLAG);
/* endless idle loop with no priority at all */
while (1) {
+ if (!need_resched()) {
+ void (*idle)(void);
#ifdef CONFIG_SMP
- if (!need_resched())
min_xtp();
#endif
- while (!need_resched()) {
- void (*idle)(void);
-
if (__get_cpu_var(cpu_idle_state))
__get_cpu_var(cpu_idle_state) = 0;
@@ -282,17 +287,17 @@ cpu_idle (void)
if (!idle)
idle = default_idle;
(*idle)();
- }
-
- if (mark_idle)
- (*mark_idle)(0);
-
+ if (mark_idle)
+ (*mark_idle)(0);
#ifdef CONFIG_SMP
- normal_xtp();
+ normal_xtp();
#endif
+ }
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
- if (cpu_is_offline(smp_processor_id()))
+ if (cpu_is_offline(cpu))
play_dead();
}
}
Index: linux-2.6/arch/ia64/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/ia64/kernel/smpboot.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/ia64/kernel/smpboot.c 2005-05-27 19:26:19.000000000 +1000
@@ -393,6 +393,8 @@ smp_callin (void)
int __devinit
start_secondary (void *unused)
{
+ preempt_disable();
+
/* Early console may use I/O ports */
ia64_set_kr(IA64_KR_IO_BASE, __pa(ia64_iobase));
Dprintk("start_secondary: starting CPU 0x%x\n", hard_smp_processor_id());
Index: linux-2.6/arch/ppc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc64/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/ppc64/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -561,7 +561,10 @@ int __devinit __cpu_up(unsigned int cpu)
/* Activate a secondary processor. */
int __devinit start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sparc64/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -144,6 +144,9 @@ void __init smp_callin(void)
membar("#LoadLoad");
cpu_set(cpuid, cpu_online_map);
+
+ /* idle thread is expected to have preempt disabled */
+ preempt_disable();
}
void cpu_panic(void)
@@ -1167,20 +1170,9 @@ void __init smp_cpus_done(unsigned int m
(bogosum/(5000/HZ))%100);
}
-/* This needn't do anything as we do not sleep the cpu
- * inside of the idler task, so an interrupt is not needed
- * to get a clean fast response.
- *
- * XXX Reverify this assumption... -DaveM
- *
- * Addendum: We do want it to do something for the signal
- * delivery case, we detect that by just seeing
- * if we are trying to send this to an idler or not.
- */
void smp_send_reschedule(int cpu)
{
- if (cpu_data(cpu).idle_volume == 0)
- smp_receive_signal(cpu);
+ smp_receive_signal(cpu);
}
/* This is a nop because we capture all other cpus
Index: linux-2.6/arch/sparc64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc64/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sparc64/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -74,7 +74,9 @@ void cpu_idle(void)
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -83,21 +85,31 @@ void cpu_idle(void)
/*
* the idle loop on a UltraMultiPenguin...
+ *
+ * TIF_POLLING_NRFLAG is set because we do not sleep the cpu
+ * inside of the idler task, so an interrupt is not needed
+ * to get a clean fast response.
+ *
+ * XXX Reverify this assumption... -DaveM
+ *
+ * Addendum: We do want it to do something for the signal
+ * delivery case, we detect that by just seeing
+ * if we are trying to send this to an idler or not.
*/
-#define idle_me_harder() (cpu_data(smp_processor_id()).idle_volume += 1)
-#define unidle_me() (cpu_data(smp_processor_id()).idle_volume = 0)
void cpu_idle(void)
{
+ cpuinfo_sparc *cpuinfo = &local_cpu_data();
set_thread_flag(TIF_POLLING_NRFLAG);
+
while(1) {
if (need_resched()) {
- unidle_me();
- clear_thread_flag(TIF_POLLING_NRFLAG);
+ cpuinfo->idle_volume = 0;
+ preempt_enable_no_resched();
schedule();
- set_thread_flag(TIF_POLLING_NRFLAG);
+ preempt_disable();
check_pgt_cache();
}
- idle_me_harder();
+ cpuinfo->idle_volume++;
/* The store ordering is so that IRQ handlers on
* other cpus see our increasing idleness for the buddy
Index: linux-2.6/arch/alpha/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/alpha/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/alpha/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -43,22 +43,20 @@
#include "proto.h"
#include "pci_impl.h"
-void default_idle(void)
-{
- barrier();
-}
-
void
cpu_idle(void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
while (1) {
- void (*idle)(void) = default_idle;
/* FIXME -- EV6 and LCA45 know how to power down
the CPU. */
while (!need_resched())
- idle();
+ cpu_relax();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/alpha/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/alpha/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/alpha/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -128,7 +128,11 @@ wait_boot_cpu_to_stop(int cpuid)
void __init
smp_callin(void)
{
- int cpuid = hard_smp_processor_id();
+ int cpuid;
+
+ preempt_disable();
+
+ cpuid = hard_smp_processor_id();
if (cpu_test_and_set(cpuid, cpu_online_map)) {
printk("??, cpu 0x%x already present??\n", cpuid);
Index: linux-2.6/arch/s390/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/s390/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/s390/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -528,6 +528,8 @@ extern void pfault_fini(void);
int __devinit start_secondary(void *cpuvoid)
{
+ preempt_disable();
+
/* Setup the cpu */
cpu_init();
/* init per CPU timer */
Index: linux-2.6/arch/sparc/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sparc/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sparc/kernel/process.c 2005-05-27 19:36:18.000000000 +1000
@@ -72,6 +72,13 @@ struct thread_info *current_set[NR_CPUS]
*/
void default_idle(void)
{
+ if (pm_idle) {
+ while (!need_resched())
+ (*pm_idle)();
+ } else {
+ while (!need_resched())
+ cpu_relax();
+ }
}
#ifndef CONFIG_SMP
@@ -92,12 +99,11 @@ void cpu_idle(void)
static unsigned long fps;
unsigned long now;
unsigned long faults;
- unsigned long flags;
extern unsigned long sun4c_kernel_faults;
extern void sun4c_grow_kernel_ring(void);
- local_irq_save(flags);
+ local_irq_disable();
now = jiffies;
count -= (now - last_jiffies);
last_jiffies = now;
@@ -113,14 +119,14 @@ void cpu_idle(void)
sun4c_grow_kernel_ring();
}
}
- local_irq_restore(flags);
+ local_irq_enable();
}
- while((!need_resched()) && pm_idle) {
- (*pm_idle)();
- }
+ default_idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
@@ -130,13 +136,17 @@ void cpu_idle(void)
/* This is being executed in task 0 'user space'. */
void cpu_idle(void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while(1) {
- if(need_resched()) {
- schedule();
- check_pgt_cache();
- }
- barrier(); /* or else gcc optimizes... */
+ while (!need_resched())
+ cpu_relax();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ check_pgt_cache();
}
}
Index: linux-2.6/arch/ppc/kernel/idle.c
===================================================================
--- linux-2.6.orig/arch/ppc/kernel/idle.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/ppc/kernel/idle.c 2005-05-27 19:36:46.000000000 +1000
@@ -50,8 +50,6 @@ void default_idle(void)
}
#endif
}
- if (need_resched())
- schedule();
}
/*
@@ -59,11 +57,18 @@ void default_idle(void)
*/
void cpu_idle(void)
{
- for (;;)
- if (ppc_md.idle != NULL)
- ppc_md.idle();
- else
- default_idle();
+ for (;;) {
+ while (need_resched()) {
+ if (ppc_md.idle != NULL)
+ ppc_md.idle();
+ else
+ default_idle();
+ }
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
#if defined(CONFIG_SYSCTL) && defined(CONFIG_6xx)
Index: linux-2.6/arch/m32r/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m32r/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/m32r/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -104,7 +104,9 @@ void cpu_idle (void)
idle();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/frv/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/frv/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/frv/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -77,16 +77,20 @@ void (*idle)(void) = core_sleep_idle;
*/
void cpu_idle(void)
{
+ int cpu = smp_processor_id();
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched()) {
- irq_stat[smp_processor_id()].idle_timestamp = jiffies;
+ irq_stat[cpu].idle_timestamp = jiffies;
if (!frv_dma_inprogress && idle)
idle();
}
-
+
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/cris/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/cris/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/cris/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -201,7 +201,9 @@ void cpu_idle (void)
idle();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/mips/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/mips/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/mips/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -83,7 +83,11 @@ extern ATTRIB_NORET void cpu_idle(void);
*/
asmlinkage void start_secondary(void)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+
+ cpu = smp_processor_id();
cpu_probe();
cpu_report();
Index: linux-2.6/arch/parisc/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/parisc/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/parisc/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -88,11 +88,15 @@ void default_idle(void)
*/
void cpu_idle(void)
{
+ set_thread_flag(TIF_POLLING_NRFLAG);
+
/* endless idle loop with no priority at all */
while (1) {
while (!need_resched())
barrier();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
check_pgt_cache();
}
}
Index: linux-2.6/arch/ppc/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/ppc/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/ppc/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -326,6 +326,8 @@ int __devinit start_secondary(void *unus
{
int cpu;
+ preempt_disable();
+
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/sh/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sh/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sh/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -51,28 +51,24 @@ void enable_hlt(void)
EXPORT_SYMBOL(enable_hlt);
-void default_idle(void)
+void cpu_idle(void)
{
/* endless idle loop with no priority at all */
while (1) {
if (hlt_counter) {
- while (1)
- if (need_resched())
- break;
+ while (!need_resched())
+ cpu_relax();
} else {
while (!need_resched())
cpu_sleep();
}
+ preempt_disable_no_resched();
schedule();
+ preempt_enable();
}
}
-void cpu_idle(void)
-{
- default_idle();
-}
-
void machine_restart(char * __unused)
{
/* SR.BL=1 and invoke address error to let CPU reset (manual reset) */
Index: linux-2.6/arch/m68k/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m68k/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/m68k/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -102,7 +102,9 @@ void cpu_idle(void)
while (1) {
while (!need_resched())
idle();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/mips/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/mips/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/mips/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -58,7 +58,9 @@ ATTRIB_NORET void cpu_idle(void)
while (!need_resched())
if (cpu_wait)
(*cpu_wait)();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable()
}
}
Index: linux-2.6/arch/m68knommu/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/m68knommu/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/m68knommu/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -45,11 +45,8 @@ asmlinkage void ret_from_fork(void);
*/
void default_idle(void)
{
- while(1) {
- if (need_resched())
- __asm__("stop #0x2000" : : : "cc");
- schedule();
- }
+ while (!need_resched())
+ __asm__("stop #0x2000" : : : "cc");
}
void (*idle)(void) = default_idle;
@@ -63,7 +60,12 @@ void (*idle)(void) = default_idle;
void cpu_idle(void)
{
/* endless idle loop with no priority at all */
- idle();
+ while (1) {
+ idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/sh/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/sh/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sh/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -109,7 +109,11 @@ int __cpu_up(unsigned int cpu)
int start_secondary(void *unused)
{
- unsigned int cpu = smp_processor_id();
+ unsigned int cpu;
+
+ preempt_disable();
+
+ cpu = smp_processor_id();
atomic_inc(&init_mm.mm_count);
current->active_mm = &init_mm;
Index: linux-2.6/arch/parisc/kernel/smp.c
===================================================================
--- linux-2.6.orig/arch/parisc/kernel/smp.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/parisc/kernel/smp.c 2005-05-27 19:26:19.000000000 +1000
@@ -462,6 +462,8 @@ void __init smp_callin(void)
void *istack;
#endif
+ preempt_disable();
+
smp_cpu_init(slave_id);
#if 0 /* NOT WORKING YET - see entry.S */
Index: linux-2.6/arch/m32r/kernel/smpboot.c
===================================================================
--- linux-2.6.orig/arch/m32r/kernel/smpboot.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/m32r/kernel/smpboot.c 2005-05-27 19:26:19.000000000 +1000
@@ -424,6 +424,7 @@ void __init smp_cpus_done(unsigned int m
*==========================================================================*/
int __init start_secondary(void *unused)
{
+ preempt_disable();
cpu_init();
smp_callin();
while (!cpu_isset(smp_processor_id(), smp_commenced_mask))
Index: linux-2.6/arch/s390/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/s390/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/s390/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -101,11 +101,6 @@ void default_idle(void)
int cpu, rc;
local_irq_disable();
- if (need_resched()) {
- local_irq_enable();
- schedule();
- return;
- }
/* CPU is going idle. */
cpu = smp_processor_id();
@@ -121,7 +116,7 @@ void default_idle(void)
__ctl_set_bit(8, 15);
#ifdef CONFIG_HOTPLUG_CPU
- if (cpu_is_offline(smp_processor_id()))
+ if (cpu_is_offline(cpu))
cpu_die();
#endif
@@ -161,8 +156,13 @@ void default_idle(void)
void cpu_idle(void)
{
- for (;;)
- default_idle();
+ for (;;) {
+ while (!need_resched())
+ default_idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void show_regs(struct pt_regs *regs)
Index: linux-2.6/arch/sh64/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/sh64/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/sh64/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -307,23 +307,19 @@ __setup("hlt", hlt_setup);
static inline void hlt(void)
{
- if (hlt_counter)
- return;
-
__asm__ __volatile__ ("sleep" : : : "memory");
}
/*
* The idle loop on a uniprocessor SH..
*/
-void default_idle(void)
+void cpu_idle(void)
{
/* endless idle loop with no priority at all */
while (1) {
if (hlt_counter) {
- while (1)
- if (need_resched())
- break;
+ while (!need_resched())
+ cpu_relax();
} else {
local_irq_disable();
while (!need_resched()) {
@@ -334,13 +330,11 @@ void default_idle(void)
}
local_irq_enable();
}
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
-}
-void cpu_idle(void)
-{
- default_idle();
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/arm26/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/arm26/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/arm26/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -74,15 +74,13 @@ __setup("hlt", hlt_setup);
void cpu_idle(void)
{
/* endless idle loop with no priority at all */
- preempt_disable();
while (1) {
- while (!need_resched()) {
- local_irq_disable();
- if (!need_resched() && !hlt_counter)
- local_irq_enable();
- }
+ while (!need_resched())
+ cpu_relax();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
}
- schedule();
}
static char reboot_mode = 'h';
Index: linux-2.6/arch/arm/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/arm/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/arm/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -84,10 +84,14 @@ EXPORT_SYMBOL(pm_power_off);
*/
void default_idle(void)
{
- local_irq_disable();
- if (!need_resched() && !hlt_counter)
- arch_idle();
- local_irq_enable();
+ if (hlt_counter)
+ cpu_relax()
+ else {
+ local_irq_disable();
+ if (!need_resched())
+ arch_idle();
+ local_irq_enable();
+ }
}
/*
@@ -104,13 +108,13 @@ void cpu_idle(void)
void (*idle)(void) = pm_idle;
if (!idle)
idle = default_idle;
- preempt_disable();
leds_event(led_idle_start);
while (!need_resched())
idle();
leds_event(led_idle_end);
- preempt_enable();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/h8300/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/h8300/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/h8300/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -53,22 +53,17 @@ asmlinkage void ret_from_fork(void);
#if !defined(CONFIG_H8300H_SIM) && !defined(CONFIG_H8S_SIM)
void default_idle(void)
{
- while(1) {
- if (need_resched()) {
- local_irq_enable();
- __asm__("sleep");
- local_irq_disable();
- }
- schedule();
+ local_irq_disable();
+ if (need_resched()) {
+ local_irq_enable();
+ /* XXX: race here! What if need_resched() gets set now? */
+ __asm__("sleep");
}
}
#else
void default_idle(void)
{
- while(1) {
- if (need_resched())
- schedule();
- }
+ cpu_relax();
}
#endif
void (*idle)(void) = default_idle;
@@ -81,7 +76,13 @@ void (*idle)(void) = default_idle;
*/
void cpu_idle(void)
{
- idle();
+ while (1) {
+ while (!need_resched())
+ idle();
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
void machine_restart(char * __unused)
Index: linux-2.6/arch/xtensa/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/xtensa/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/xtensa/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -96,8 +96,9 @@ void cpu_idle(void)
while (1) {
while (!need_resched())
platform_idle();
- preempt_enable();
+ preempt_enable_no_resched();
schedule();
+ preempt_disable();
}
}
Index: linux-2.6/arch/v850/kernel/process.c
===================================================================
--- linux-2.6.orig/arch/v850/kernel/process.c 2005-05-27 19:19:08.000000000 +1000
+++ linux-2.6/arch/v850/kernel/process.c 2005-05-27 19:26:19.000000000 +1000
@@ -36,11 +36,8 @@ extern void ret_from_fork (void);
/* The idle loop. */
void default_idle (void)
{
- while (1) {
- while (! need_resched ())
- asm ("halt; nop; nop; nop; nop; nop" ::: "cc");
- schedule ();
- }
+ while (! need_resched ())
+ asm ("halt; nop; nop; nop; nop; nop" ::: "cc");
}
void (*idle)(void) = default_idle;
@@ -54,7 +51,14 @@ void (*idle)(void) = default_idle;
void cpu_idle (void)
{
/* endless idle loop with no priority at all */
- (*idle) ();
+ while (1) {
+ while (!need_resched())
+ (*idle) ();
+
+ preempt_enable_no_resched();
+ schedule();
+ preempt_disable();
+ }
}
/*
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 9:37 ` Nick Piggin
@ 2005-05-27 10:12 ` Ingo Molnar
2005-06-01 6:15 ` Andrew Morton
1 sibling, 0 replies; 15+ messages in thread
From: Ingo Molnar @ 2005-05-27 10:12 UTC (permalink / raw)
To: Nick Piggin; +Cc: Nick Piggin, Andrew Morton, linux-kernel
* Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
> confusion, and make their semantics rigid. Also have preempt
> explicitly disabled in idle routines. Improves efficiency of
> resched_task and some cpu_idle routines.
Acked-by: Ingo Molnar <mingo@elte.hu>
Ingo
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-05-27 9:37 ` Nick Piggin
2005-05-27 10:12 ` Ingo Molnar
@ 2005-06-01 6:15 ` Andrew Morton
2005-06-01 6:31 ` Nick Piggin
1 sibling, 1 reply; 15+ messages in thread
From: Andrew Morton @ 2005-06-01 6:15 UTC (permalink / raw)
To: Nick Piggin; +Cc: mingo, piggin, linux-kernel
Nick Piggin <nickpiggin@yahoo.com.au> wrote:
>
> Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
> confusion, and make their semantics rigid. Also have preempt explicitly
> disabled in idle routines. Improves efficiency of resched_task and some
> cpu_idle routines.
This patch, with or without sched-resched-optimisation-fix.patch causes my
x86_64 box to soil its pants.
I'll try to get -mm2 out the door - maybe there was some interaction with
something else.
CPU: Trace cache: 12K uops, L1 D cache: 16K
CPU: L2 cache: 1024K
CPU: Physical Processor ID: 3
CPU1: Thermal monitoring enabled (TM1)
Intel(R) Xeon(TM) CPU 3.40GHz stepping 04
CPU 1: Syncing TSC to CPU 0.
Bo6tCng 2r sencor 2iz diTS6000 hspPf ff81a07ffiff 8
cyclesrsi mng Cr 923
tinofdelck p ing ad 2 ssaetediupr
/7PUipL6 cachs: 1ff4K100CPU: Ph8-11[1)ease U:teraee ca -e-------o
icIn tracizing CPU 0
.4PU:zTsaep iach04uoiigg enabled ciM1) rou e.. 680 . 1 oIntIP(R)lpjo1360) 29)
tC U:U hyLr Dn ache: 16K40)CPUCPU 2c cyn: n024KC
sCarte Thermal monitoring enabled (TM1)
Intel(R) Xeon(TM) CPU 3.40GHz stepping 04
APIC error on CPU3: 00(40)
CPU 3: Syncing TSC to CPU 0.
Kernel BUG at "kernel/sched.c":2805
invalid operand: 0000 [1] PREEMPT SMP
CPU 2
Modules linked in:
Pid: 0, comm: swapper Not tainted 2.6.12-rc5-mm2
RIP: 0010:[<ffffffff8012a97a>] <ffffffff8012a97a>{sub_preempt_count+22}
RSP: 0018:ffff81007ff7fef0 EFLAGS: 00010297
RAX: ffff81007ff7ffd8 RBX: ffffffff805d8180 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000001
RBP: ffff81007ff7fef0 R08: 00000000fffffff9 R09: 0000000000000002
R10: 00000000ffffffff R11: 0000000000000000 R12: 00000000000011d1
R13: ffff81007ff7ff18 R14: ffff81007ff7ff20 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffffffff805a3400(0000) knlGS:0000000000000000
CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000000101000 CR4: 00000000000006e0
Process swapper (pid: 0, threadinfo ffff81007ff7e000, task ffff81007ff740d0)
Stack: 0000000000000040 ffffffff8010beed ffffffffffffff67 ffffffff805b77c9
0000000000000246 0000000000000270 00000000000003af 0000000000000000
0000000000000000 0000000000000000
Call Trace:<ffffffff8010beed>{cpu_idle+94} <ffffffff805b77c9>{start_secondary+531}
Code: 0f 0b c4 7e 3d 80 ff ff ff ff f5 0a 81 ff fe 00 00 00 3e 77
RIP <ffffffff8012a97a>{sub_preempt_count+22} RSP <ffff81007ff7fef0>
<0>>ePnel payic -onoted nSC gi htCempte( tst iil 46 cyclesa ka
errPU 1: synlhs)
izeo tSo kuththrea0 ( ssartif -1
63)
Brought up 4 CPUs
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: [patch] improve SMP reschedule and idle routines
2005-06-01 6:15 ` Andrew Morton
@ 2005-06-01 6:31 ` Nick Piggin
0 siblings, 0 replies; 15+ messages in thread
From: Nick Piggin @ 2005-06-01 6:31 UTC (permalink / raw)
To: Andrew Morton; +Cc: Ingo Molnar, piggin, lkml
On Tue, 2005-05-31 at 23:15 -0700, Andrew Morton wrote:
> Nick Piggin <nickpiggin@yahoo.com.au> wrote:
> >
> > Make some changes to the NEED_RESCHED and POLLING_NRFLAG to reduce
> > confusion, and make their semantics rigid. Also have preempt explicitly
> > disabled in idle routines. Improves efficiency of resched_task and some
> > cpu_idle routines.
>
> This patch, with or without sched-resched-optimisation-fix.patch causes my
> x86_64 box to soil its pants.
>
Sorry about that. I probably have broken something since last
testing x86-64. It looks like a simple mismatched preempt_
operation, so I'll try to get that fixed up shortly.
--
SUSE Labs, Novell Inc.
Send instant messages to your online friends http://au.messenger.yahoo.com
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2005-06-01 6:32 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2005-05-16 4:21 [patch] improve SMP reschedule and idle routines Nick Piggin
[not found] ` <20050515.220455.59467677.davem@davemloft.net>
2005-05-16 5:19 ` Nick Piggin
[not found] ` <20050515.222722.63128129.davem@davemloft.net>
2005-05-16 5:34 ` Nick Piggin
2005-05-17 7:34 ` Nick Piggin
2005-05-17 7:40 ` Ingo Molnar
-- strict thread matches above, loose matches on Subject: below --
2005-05-16 13:51 Oleg Nesterov
2005-05-16 22:52 ` Nick Piggin
2005-05-27 7:21 Nick Piggin
2005-05-27 8:57 ` Ingo Molnar
2005-05-27 9:11 ` Nick Piggin
2005-05-27 9:20 ` Ingo Molnar
2005-05-27 9:37 ` Nick Piggin
2005-05-27 10:12 ` Ingo Molnar
2005-06-01 6:15 ` Andrew Morton
2005-06-01 6:31 ` Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox