* [PATCH 0/5] call_rcu_sched() series
@ 2008-04-14 3:58 Paul E. McKenney
2008-04-14 4:00 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney
` (5 more replies)
0 siblings, 6 replies; 12+ messages in thread
From: Paul E. McKenney @ 2008-04-14 3:58 UTC (permalink / raw)
To: linux-kernel
Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith,
rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty,
jkenisto, oleg, jamesclhuang
This series adds call_rcu_sched(), applies some fixes to RCU Classic,
adds rcu_barrier_sched(), updates rcutorture, and updates the RCU
documentation.
^ permalink raw reply [flat|nested] 12+ messages in thread* [PATCH 1/5] Add call_rcu_sched() 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney @ 2008-04-14 4:00 ` Paul E. McKenney 2008-04-14 4:01 ` [PATCH 2/5] Add memory barriers and comments to rcu_check_callbacks() Paul E. McKenney ` (4 subsequent siblings) 5 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-04-14 4:00 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Fourth cut of patch to provide the call_rcu_sched(). This is again to synchronize_sched() as call_rcu() is to synchronize_rcu(). Should be fine for experimental and -rt use, but not ready for inclusion. With some luck, I will be able to tell Andrew to come out of hiding on the next round. Passes multi-day rcutorture sessions with concurrent CPU hotplugging. Fixes since the first version include a bug that could result in indefinite blocking (spotted by Gautham Shenoy), better resiliency against CPU-hotplug operations, and other minor fixes. Fixes since the second version include reworking grace-period detection to avoid deadlocks that could happen when running concurrently with CPU hotplug, adding Mathieu's fix to avoid the softlockup messages, as well as Mathieu's fix to allow use earlier in boot. Fixes since the third version include a wrong-CPU bug spotted by Andrew, getting rid of the obsolete synchronize_kernel API that somehow snuck back in, merging spin_unlock() and local_irq_restore() in a few places, commenting the code that checks for quiescent states based on interrupting from user-mode execution or the idle loop, removing some inline attributes, and some code-style changes. Known/suspected shortcomings: o I still do not entirely trust the sleep/wakeup logic. Next step will be to use a private snapshot of the CPU online mask in rcu_sched_grace_period() -- if the CPU wasn't there at the start of the grace period, we don't need to hear from it. And the bit about accounting for changes in online CPUs inside of rcu_sched_grace_period() is ugly anyway. o It might be good for rcu_sched_grace_period() to invoke resched_cpu() when a given CPU wasn't responding quickly, but resched_cpu() is declared static... This patch also fixes a long-standing bug in the earlier preemptable-RCU implementation of synchronize_rcu() that could result in loss of concurrent external changes to a task's CPU affinity mask. I still cannot remember who reported this... Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> --- include/linux/rcuclassic.h | 3 include/linux/rcupdate.h | 22 ++ include/linux/rcupreempt.h | 42 +++- init/main.c | 1 kernel/rcupdate.c | 20 -- kernel/rcupreempt.c | 414 ++++++++++++++++++++++++++++++++++++++++----- 6 files changed, 434 insertions(+), 68 deletions(-) diff -urpNa -X dontdiff linux-2.6.25-rc6/include/linux/rcuclassic.h linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcuclassic.h --- linux-2.6.25-rc6/include/linux/rcuclassic.h 2008-03-16 17:45:16.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcuclassic.h 2008-03-21 04:27:31.000000000 -0700 @@ -153,7 +153,10 @@ extern struct lockdep_map rcu_lock_map; #define __synchronize_sched() synchronize_rcu() +#define call_rcu_sched(head, func) call_rcu(head, func) + extern void __rcu_init(void); +#define rcu_init_sched() do { } while (0) extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); diff -urpNa -X dontdiff linux-2.6.25-rc6/include/linux/rcupdate.h linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcupdate.h --- linux-2.6.25-rc6/include/linux/rcupdate.h 2008-03-16 17:45:16.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcupdate.h 2008-03-20 21:10:42.000000000 -0700 @@ -42,6 +42,7 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> #include <linux/lockdep.h> +#include <linux/completion.h> /** * struct rcu_head - callback structure for use with RCU @@ -182,6 +183,27 @@ struct rcu_head { (p) = (v); \ }) +/* Infrastructure to implement the synchronize_() primitives. */ + +struct rcu_synchronize { + struct rcu_head head; + struct completion completion; +}; + +extern void wakeme_after_rcu(struct rcu_head *head); + +#define synchronize_rcu_xxx(name, func) \ +void name(void) \ +{ \ + struct rcu_synchronize rcu; \ + \ + init_completion(&rcu.completion); \ + /* Will wake me after RCU finished. */ \ + func(&rcu.head, wakeme_after_rcu); \ + /* Wait for it. */ \ + wait_for_completion(&rcu.completion); \ +} + /** * synchronize_sched - block until all CPUs have exited any non-preemptive * kernel code sequences. diff -urpNa -X dontdiff linux-2.6.25-rc6/include/linux/rcupreempt.h linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcupreempt.h --- linux-2.6.25-rc6/include/linux/rcupreempt.h 2008-03-16 17:45:16.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/include/linux/rcupreempt.h 2008-04-04 18:17:54.000000000 -0700 @@ -42,10 +42,39 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> -#define rcu_qsctr_inc(cpu) +struct rcu_dyntick_sched { + int dynticks; + int dynticks_snap; + int sched_qs; + int sched_qs_snap; + int sched_dynticks_snap; +}; + +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); + +static inline void rcu_qsctr_inc(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs++; +} #define rcu_bh_qsctr_inc(cpu) #define call_rcu_bh(head, rcu) call_rcu(head, rcu) +/** + * call_rcu_sched - Queue RCU callback for invocation after sched grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual update function to be invoked after the grace period + * + * The update function will be invoked some time after a full + * synchronize_sched()-style grace period elapses, in other words after + * all currently executing preempt-disabled sections of code (including + * hardirq handlers, NMI handlers, and local_irq_save() blocks) have + * completed. + */ +extern void call_rcu_sched(struct rcu_head *head, + void (*func)(struct rcu_head *head)); + extern void __rcu_read_lock(void) __acquires(RCU); extern void __rcu_read_unlock(void) __releases(RCU); extern int rcu_pending(int cpu); @@ -57,6 +86,7 @@ extern int rcu_needs_cpu(int cpu); extern void __synchronize_sched(void); extern void __rcu_init(void); +extern void rcu_init_sched(void); extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); extern long rcu_batches_completed(void); @@ -83,20 +113,20 @@ extern struct rcupreempt_trace *rcupreem struct softirq_action; #ifdef CONFIG_NO_HZ -DECLARE_PER_CPU(long, dynticks_progress_counter); +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); static inline void rcu_enter_nohz(void) { - __get_cpu_var(dynticks_progress_counter)++; - WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1); + __get_cpu_var(rcu_dyntick_sched).dynticks++; + WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); mb(); } static inline void rcu_exit_nohz(void) { mb(); - __get_cpu_var(dynticks_progress_counter)++; - WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1)); + __get_cpu_var(rcu_dyntick_sched).dynticks++; + WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); } #else /* CONFIG_NO_HZ */ diff -urpNa -X dontdiff linux-2.6.25-rc6/init/main.c linux-2.6.25-rc6-C1-call_rcu_sched/init/main.c --- linux-2.6.25-rc6/init/main.c 2008-03-16 17:45:17.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/init/main.c 2008-03-25 08:31:17.000000000 -0700 @@ -730,6 +730,7 @@ static void __init do_initcalls(void) */ static void __init do_basic_setup(void) { + rcu_init_sched(); /* needed by module_init stage. */ /* drivers will send hotplug events */ init_workqueues(); usermodehelper_init(); diff -urpNa -X dontdiff linux-2.6.25-rc6/kernel/rcupdate.c linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcupdate.c --- linux-2.6.25-rc6/kernel/rcupdate.c 2008-03-16 17:45:17.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcupdate.c 2008-03-20 21:10:39.000000000 -0700 @@ -39,18 +39,12 @@ #include <linux/sched.h> #include <asm/atomic.h> #include <linux/bitops.h> -#include <linux/completion.h> #include <linux/percpu.h> #include <linux/notifier.h> #include <linux/cpu.h> #include <linux/mutex.h> #include <linux/module.h> -struct rcu_synchronize { - struct rcu_head head; - struct completion completion; -}; - static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; static atomic_t rcu_barrier_cpu_count; static DEFINE_MUTEX(rcu_barrier_mutex); @@ -60,7 +54,7 @@ static struct completion rcu_barrier_com * Awaken the corresponding synchronize_rcu() instance now that a * grace period has elapsed. */ -static void wakeme_after_rcu(struct rcu_head *head) +void wakeme_after_rcu(struct rcu_head *head) { struct rcu_synchronize *rcu; @@ -77,17 +71,7 @@ static void wakeme_after_rcu(struct rcu_ * sections are delimited by rcu_read_lock() and rcu_read_unlock(), * and may be nested. */ -void synchronize_rcu(void) -{ - struct rcu_synchronize rcu; - - init_completion(&rcu.completion); - /* Will wake me after RCU finished */ - call_rcu(&rcu.head, wakeme_after_rcu); - - /* Wait for it */ - wait_for_completion(&rcu.completion); -} +synchronize_rcu_xxx(synchronize_rcu, call_rcu) EXPORT_SYMBOL_GPL(synchronize_rcu); static void rcu_barrier_callback(struct rcu_head *notused) diff -urpNa -X dontdiff linux-2.6.25-rc6/kernel/rcupreempt.c linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcupreempt.c --- linux-2.6.25-rc6/kernel/rcupreempt.c 2008-03-16 17:45:17.000000000 -0700 +++ linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcupreempt.c 2008-04-08 22:14:55.000000000 -0700 @@ -46,6 +46,7 @@ #include <asm/atomic.h> #include <linux/bitops.h> #include <linux/module.h> +#include <linux/kthread.h> #include <linux/completion.h> #include <linux/moduleparam.h> #include <linux/percpu.h> @@ -87,9 +88,14 @@ struct rcu_data { struct rcu_head **nexttail; struct rcu_head *waitlist[GP_STAGES]; struct rcu_head **waittail[GP_STAGES]; - struct rcu_head *donelist; + struct rcu_head *donelist; /* from waitlist & waitschedlist */ struct rcu_head **donetail; long rcu_flipctr[2]; + struct rcu_head *nextschedlist; + struct rcu_head **nextschedtail; + struct rcu_head *waitschedlist; + struct rcu_head **waitschedtail; + int rcu_sched_sleeping; #ifdef CONFIG_RCU_TRACE struct rcupreempt_trace trace; #endif /* #ifdef CONFIG_RCU_TRACE */ @@ -131,11 +137,24 @@ enum rcu_try_flip_states { rcu_try_flip_waitmb_state, }; +/* + * States for rcu_ctrlblk.rcu_sched_sleep. + */ + +enum rcu_sched_sleep_states { + rcu_sched_not_sleeping, /* Not sleeping, callbacks need GP. */ + rcu_sched_sleep_prep, /* Thinking of sleeping, rechecking. */ + rcu_sched_sleeping, /* Sleeping, awaken if GP needed. */ +}; + struct rcu_ctrlblk { spinlock_t fliplock; /* Protect state-machine transitions. */ long completed; /* Number of last completed batch. */ enum rcu_try_flip_states rcu_try_flip_state; /* The current state of the rcu state machine */ + spinlock_t schedlock; /* Protect rcu_sched sleep state. */ + enum rcu_sched_sleep_states sched_sleep; /* rcu_sched state. */ + wait_queue_head_t sched_wq; /* Place for rcu_sched to sleep. */ }; static DEFINE_PER_CPU(struct rcu_data, rcu_data); @@ -143,8 +162,12 @@ static struct rcu_ctrlblk rcu_ctrlblk = .fliplock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.fliplock), .completed = 0, .rcu_try_flip_state = rcu_try_flip_idle_state, + .schedlock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.schedlock), + .sched_sleep = rcu_sched_not_sleeping, + .sched_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rcu_ctrlblk.sched_wq), }; +static struct task_struct *rcu_sched_grace_period_task; #ifdef CONFIG_RCU_TRACE static char *rcu_try_flip_state_names[] = @@ -207,6 +230,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu */ #define RCU_TRACE_RDP(f, rdp) RCU_TRACE(f, &((rdp)->trace)); +#define RCU_SCHED_BATCH_TIME (HZ / 50) + /* * Return the number of RCU batches processed thus far. Useful * for debug and statistics. @@ -413,32 +438,34 @@ static void __rcu_advance_callbacks(stru } } -#ifdef CONFIG_NO_HZ +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_dyntick_sched, rcu_dyntick_sched) = { + .dynticks = 1, +}; -DEFINE_PER_CPU(long, dynticks_progress_counter) = 1; -static DEFINE_PER_CPU(long, rcu_dyntick_snapshot); +#ifdef CONFIG_NO_HZ static DEFINE_PER_CPU(int, rcu_update_flag); /** * rcu_irq_enter - Called from Hard irq handlers and NMI/SMI. * * If the CPU was idle with dynamic ticks active, this updates the - * dynticks_progress_counter to let the RCU handling know that the + * rcu_dyntick_sched.dynticks to let the RCU handling know that the * CPU is active. */ void rcu_irq_enter(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); if (per_cpu(rcu_update_flag, cpu)) per_cpu(rcu_update_flag, cpu)++; /* * Only update if we are coming from a stopped ticks mode - * (dynticks_progress_counter is even). + * (rcu_dyntick_sched.dynticks is even). */ if (!in_interrupt() && - (per_cpu(dynticks_progress_counter, cpu) & 0x1) == 0) { + (rdssp->dynticks & 0x1) == 0) { /* * The following might seem like we could have a race * with NMI/SMIs. But this really isn't a problem. @@ -461,12 +488,12 @@ void rcu_irq_enter(void) * RCU read-side critical sections on this CPU would * have already completed. */ - per_cpu(dynticks_progress_counter, cpu)++; + rdssp->dynticks++; /* * The following memory barrier ensures that any * rcu_read_lock() primitives in the irq handler * are seen by other CPUs to follow the above - * increment to dynticks_progress_counter. This is + * increment to rcu_dyntick_sched.dynticks. This is * required in order for other CPUs to correctly * determine when it is safe to advance the RCU * grace-period state machine. @@ -474,7 +501,7 @@ void rcu_irq_enter(void) smp_mb(); /* see above block comment. */ /* * Since we can't determine the dynamic tick mode from - * the dynticks_progress_counter after this routine, + * the rcu_dyntick_sched.dynticks after this routine, * we use a second flag to acknowledge that we came * from an idle state with ticks stopped. */ @@ -482,7 +509,7 @@ void rcu_irq_enter(void) /* * If we take an NMI/SMI now, they will also increment * the rcu_update_flag, and will not update the - * dynticks_progress_counter on exit. That is for + * rcu_dyntick_sched.dynticks on exit. That is for * this IRQ to do. */ } @@ -492,12 +519,13 @@ void rcu_irq_enter(void) * rcu_irq_exit - Called from exiting Hard irq context. * * If the CPU was idle with dynamic ticks active, update the - * dynticks_progress_counter to put let the RCU handling be + * rcu_dyntick_sched.dynticks to put let the RCU handling be * aware that the CPU is going back to idle with no ticks. */ void rcu_irq_exit(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); /* * rcu_update_flag is set if we interrupted the CPU @@ -505,7 +533,7 @@ void rcu_irq_exit(void) * Once this occurs, we keep track of interrupt nesting * because a NMI/SMI could also come in, and we still * only want the IRQ that started the increment of the - * dynticks_progress_counter to be the one that modifies + * rcu_dyntick_sched.dynticks to be the one that modifies * it on exit. */ if (per_cpu(rcu_update_flag, cpu)) { @@ -517,28 +545,29 @@ void rcu_irq_exit(void) /* * If an NMI/SMI happens now we are still - * protected by the dynticks_progress_counter being odd. + * protected by the rcu_dyntick_sched.dynticks being odd. */ /* * The following memory barrier ensures that any * rcu_read_unlock() primitives in the irq handler * are seen by other CPUs to preceed the following - * increment to dynticks_progress_counter. This + * increment to rcu_dyntick_sched.dynticks. This * is required in order for other CPUs to determine * when it is safe to advance the RCU grace-period * state machine. */ smp_mb(); /* see above block comment. */ - per_cpu(dynticks_progress_counter, cpu)++; - WARN_ON(per_cpu(dynticks_progress_counter, cpu) & 0x1); + rdssp->dynticks++; + WARN_ON(rdssp->dynticks & 0x1); } } static void dyntick_save_progress_counter(int cpu) { - per_cpu(rcu_dyntick_snapshot, cpu) = - per_cpu(dynticks_progress_counter, cpu); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->dynticks_snap = rdssp->dynticks; } static inline int @@ -546,9 +575,10 @@ rcu_try_flip_waitack_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -582,9 +612,10 @@ rcu_try_flip_waitmb_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -611,14 +642,86 @@ rcu_try_flip_waitmb_needed(int cpu) return 1; } +static void dyntick_save_progress_counter_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_dynticks_snap = rdssp->dynticks; +} + +static int rcu_qsctr_inc_needed_dyntick(int cpu) +{ + long curr; + long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + curr = rdssp->dynticks; + snap = rdssp->sched_dynticks_snap; + smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ + + /* + * If the CPU remained in dynticks mode for the entire time + * and didn't take any interrupts, NMIs, SMIs, or whatever, + * then it cannot be in the middle of an rcu_read_lock(), so + * the next rcu_read_lock() it executes must use the new value + * of the counter. Therefore, this CPU has been in a quiescent + * state the entire time, and we don't need to wait for it. + */ + + if ((curr == snap) && ((curr & 0x1) == 0)) + return 0; + + /* + * If the CPU passed through or entered a dynticks idle phase with + * no active irq handlers, then, as above, this CPU has already + * passed through a quiescent state. + */ + + if ((curr - snap) > 2 || (snap & 0x1) == 0) + return 0; + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + #else /* !CONFIG_NO_HZ */ -# define dyntick_save_progress_counter(cpu) do { } while (0) -# define rcu_try_flip_waitack_needed(cpu) (1) -# define rcu_try_flip_waitmb_needed(cpu) (1) +# define dyntick_save_progress_counter(cpu) do { } while (0) +# define rcu_try_flip_waitack_needed(cpu) (1) +# define rcu_try_flip_waitmb_needed(cpu) (1) + +# define dyntick_save_progress_counter_sched(cpu) do { } while (0) +# define rcu_qsctr_inc_needed_dyntick(cpu) (1) #endif /* CONFIG_NO_HZ */ +static void save_qsctr_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs_snap = rdssp->sched_qs; +} + +static inline int rcu_qsctr_inc_needed(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + /* + * If there has been a quiescent state, no more need to wait + * on this CPU. + */ + + if (rdssp->sched_qs != rdssp->sched_qs_snap) { + smp_mb(); /* force ordering with cpu entering schedule(). */ + return 0; + } + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + /* * Get here when RCU is idle. Decide whether we need to * move out of idle state, and return non-zero if so. @@ -821,6 +924,26 @@ void rcu_check_callbacks(int cpu, int us unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + /* + * If this CPU took its interrupt from user mode or from the + * idle loop, and this is not a nested interrupt, then + * this CPU has to have exited all prior preept-disable + * sections of code. So increment the counter to note this. + * + * The memory barrier is needed to handle the case where + * writes from a preempt-disable section of code get reordered + * into schedule() by this CPU's write buffer. So the memory + * barrier makes sure that the rcu_qsctr_inc() is seen by other + * CPUs to happen after any such write. + */ + + if (user || + (idle_cpu(cpu) && !in_softirq() && + hardirq_count() <= (1 << HARDIRQ_SHIFT))) { + smp_mb(); /* Guard against aggressive schedule(). */ + rcu_qsctr_inc(cpu); + } + rcu_check_mb(cpu); if (rcu_ctrlblk.completed == rdp->completed) rcu_try_flip(); @@ -871,6 +994,8 @@ void rcu_offline_cpu(int cpu) struct rcu_head *list = NULL; unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + struct rcu_head *schedlist = NULL; + struct rcu_head **schedtail = &schedlist; struct rcu_head **tail = &list; /* @@ -884,6 +1009,11 @@ void rcu_offline_cpu(int cpu) rcu_offline_cpu_enqueue(rdp->waitlist[i], rdp->waittail[i], list, tail); rcu_offline_cpu_enqueue(rdp->nextlist, rdp->nexttail, list, tail); + rcu_offline_cpu_enqueue(rdp->waitschedlist, rdp->waitschedtail, + schedlist, schedtail); + rcu_offline_cpu_enqueue(rdp->nextschedlist, rdp->nextschedtail, + schedlist, schedtail); + rdp->rcu_sched_sleeping = 0; spin_unlock_irqrestore(&rdp->lock, flags); rdp->waitlistcount = 0; @@ -918,22 +1048,40 @@ void rcu_offline_cpu(int cpu) * fix. */ - local_irq_save(flags); + local_irq_save(flags); /* disable preempt till we know what lock. */ rdp = RCU_DATA_ME(); spin_lock(&rdp->lock); *rdp->nexttail = list; if (list) rdp->nexttail = tail; + *rdp->nextschedtail = schedlist; + if (schedlist) + rdp->nextschedtail = schedtail; spin_unlock_irqrestore(&rdp->lock, flags); } void __devinit rcu_online_cpu(int cpu) { unsigned long flags; + struct rcu_data *rdp; spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); cpu_set(cpu, rcu_cpu_online_map); spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); + + /* + * The rcu_sched grace-period processing might have bypassed + * this CPU, given that it was not in the rcu_cpu_online_map + * when the grace-period scan started. This means that the + * grace-period task might sleep. So make sure that if this + * should happen, the first callback posted to this CPU will + * wake up the grace-period task if need be. + */ + + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + rdp->rcu_sched_sleeping = 1; + spin_unlock_irqrestore(&rdp->lock, flags); } #else /* #ifdef CONFIG_HOTPLUG_CPU */ @@ -988,31 +1136,196 @@ void call_rcu(struct rcu_head *head, voi *rdp->nexttail = head; rdp->nexttail = &head->next; RCU_TRACE_RDP(rcupreempt_trace_next_add, rdp); - spin_unlock(&rdp->lock); - local_irq_restore(flags); + spin_unlock_irqrestore(&rdp->lock, flags); } EXPORT_SYMBOL_GPL(call_rcu); +void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) +{ + unsigned long flags; + struct rcu_data *rdp; + int wake_gp = 0; + + head->func = func; + head->next = NULL; + local_irq_save(flags); + rdp = RCU_DATA_ME(); + spin_lock(&rdp->lock); + *rdp->nextschedtail = head; + rdp->nextschedtail = &head->next; + if (rdp->rcu_sched_sleeping) { + + /* Grace-period processing might be sleeping... */ + + rdp->rcu_sched_sleeping = 0; + wake_gp = 1; + } + spin_unlock_irqrestore(&rdp->lock, flags); + if (wake_gp) { + + /* Wake up grace-period processing, unless someone beat us. */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleeping) + wake_gp = 0; + rcu_ctrlblk.sched_sleep = rcu_sched_not_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + if (wake_gp) + wake_up_interruptible(&rcu_ctrlblk.sched_wq); + } +} +EXPORT_SYMBOL_GPL(call_rcu_sched); + /* * Wait until all currently running preempt_disable() code segments * (including hardware-irq-disable segments) complete. Note that * in -rt this does -not- necessarily result in all currently executing * interrupt -handlers- having completed. */ -void __synchronize_sched(void) +synchronize_rcu_xxx(__synchronize_sched, call_rcu_sched) +EXPORT_SYMBOL_GPL(__synchronize_sched); + +/* + * kthread function that manages call_rcu_sched grace periods. + */ +static int rcu_sched_grace_period(void *arg) { - cpumask_t oldmask; + int couldsleep; /* might sleep after current pass. */ + int couldsleepnext = 0; /* might sleep after next pass. */ int cpu; + unsigned long flags; + struct rcu_data *rdp; + int ret; - if (sched_getaffinity(0, &oldmask) < 0) - oldmask = cpu_possible_map; - for_each_online_cpu(cpu) { - sched_setaffinity(0, cpumask_of_cpu(cpu)); - schedule(); - } - sched_setaffinity(0, oldmask); + /* + * Each pass through the following loop handles one + * rcu_sched grace period cycle. + */ + do { + /* Save each CPU's current state. */ + + for_each_online_cpu(cpu) { + dyntick_save_progress_counter_sched(cpu); + save_qsctr_sched(cpu); + } + + /* + * Sleep for about an RCU grace-period's worth to + * allow better batching and to consume less CPU. + */ + schedule_timeout_interruptible(RCU_SCHED_BATCH_TIME); + + /* + * If there was nothing to do last time, prepare to + * sleep at the end of the current grace period cycle. + */ + couldsleep = couldsleepnext; + couldsleepnext = 1; + if (couldsleep) { + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + rcu_ctrlblk.sched_sleep = rcu_sched_sleep_prep; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + } + + /* + * Wait on each CPU in turn to have either visited + * a quiescent state or been in dynticks-idle mode. + */ + for_each_online_cpu(cpu) { + while (rcu_qsctr_inc_needed(cpu) && + rcu_qsctr_inc_needed_dyntick(cpu)) { + /* resched_cpu(cpu); @@@ */ + schedule_timeout_interruptible(1); + } + } + + /* Advance callbacks for each CPU. */ + + for_each_online_cpu(cpu) { + + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + + /* + * We are running on this CPU irq-disabled, so no + * CPU can go offline until we re-enable irqs. + * The current CPU might have already gone + * offline (between the for_each_offline_cpu and + * the spin_lock_irqsave), but in that case all its + * callback lists will be empty, so no harm done. + * + * Advance the callbacks! We share normal RCU's + * donelist, since callbacks are invoked the + * same way in either case. + */ + if (rdp->waitschedlist != NULL) { + *rdp->donetail = rdp->waitschedlist; + rdp->donetail = rdp->waitschedtail; + + /* + * Next rcu_check_callbacks() will + * do the required raise_softirq(). + */ + } + if (rdp->nextschedlist != NULL) { + rdp->waitschedlist = rdp->nextschedlist; + rdp->waitschedtail = rdp->nextschedtail; + couldsleep = 0; + couldsleepnext = 0; + } else { + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + } + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + + /* Mark sleep intention. */ + + rdp->rcu_sched_sleeping = couldsleep; + + spin_unlock_irqrestore(&rdp->lock, flags); + } + + /* If we saw callbacks on the last scan, go deal with them. */ + + if (!couldsleep) + continue; + + /* Attempt to block... */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleep_prep) { + + /* + * Someone posted a callback after we scanned. + * Go take care of it. + */ + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + couldsleepnext = 0; + continue; + } + + /* Block until the next person posts a callback. */ + + rcu_ctrlblk.sched_sleep = rcu_sched_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + ret = 0; + __wait_event_interruptible(rcu_ctrlblk.sched_wq, + rcu_ctrlblk.sched_sleep != rcu_sched_sleeping, + ret); + + /* + * Signals would prevent us from sleeping, and we cannot + * do much with them in any case. So flush them. + */ + if (ret) + flush_signals(current); + couldsleepnext = 0; + + } while (!kthread_should_stop()); + + return (0); } -EXPORT_SYMBOL_GPL(__synchronize_sched); /* * Check to see if any future RCU-related work will need to be done @@ -1029,7 +1342,9 @@ int rcu_needs_cpu(int cpu) return (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL); + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL); } int rcu_pending(int cpu) @@ -1040,7 +1355,9 @@ int rcu_pending(int cpu) if (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL) + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL) return 1; /* The RCU core needs an acknowledgement from this CPU. */ @@ -1107,6 +1424,11 @@ void __init __rcu_init(void) rdp->donetail = &rdp->donelist; rdp->rcu_flipctr[0] = 0; rdp->rcu_flipctr[1] = 0; + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + rdp->rcu_sched_sleeping = 0; } register_cpu_notifier(&rcu_nb); @@ -1129,11 +1451,15 @@ void __init __rcu_init(void) } /* - * Deprecated, use synchronize_rcu() or synchronize_sched() instead. + * Late-boot-time RCU initialization that must wait until after scheduler + * has been initialized. */ -void synchronize_kernel(void) +void __init rcu_init_sched(void) { - synchronize_rcu(); + rcu_sched_grace_period_task = kthread_run(rcu_sched_grace_period, + NULL, + "rcu_sched_grace_period"); + WARN_ON(IS_ERR(rcu_sched_grace_period_task)); } #ifdef CONFIG_RCU_TRACE ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 2/5] Add memory barriers and comments to rcu_check_callbacks() 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney 2008-04-14 4:00 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney @ 2008-04-14 4:01 ` Paul E. McKenney 2008-04-14 4:02 ` [PATCH 3/5] Add rcu_barrier_sched() and rcu_barrier_bh() Paul E. McKenney ` (3 subsequent siblings) 5 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-04-14 4:01 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Add comments to the logic that infers quiescent states when interrupting from either user mode or the idle loop. Also add a memory barrier: it appears that James Huang was in fact onto something, as the scheduler is much less synchronization happy than it once was, so we can no longer rely on its memory barriers in all cases. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Reported-by: James Huang <jamesclhuang@yahoo.com> --- rcuclassic.c | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff -urpNa -X dontdiff linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcuclassic.c linux-2.6.25-rc6-C2-rcuclassic-fixes/kernel/rcuclassic.c --- linux-2.6.25-rc6-C1-call_rcu_sched/kernel/rcuclassic.c 2008-03-16 17:45:17.000000000 -0700 +++ linux-2.6.25-rc6-C2-rcuclassic-fixes/kernel/rcuclassic.c 2008-04-08 09:30:16.000000000 -0700 @@ -502,10 +502,38 @@ void rcu_check_callbacks(int cpu, int us if (user || (idle_cpu(cpu) && !in_softirq() && hardirq_count() <= (1 << HARDIRQ_SHIFT))) { + + /* + * Get here if this CPU took its interrupt from user + * mode or from the idle loop, and if this is not a + * nested interrupt. In this case, the CPU is in + * a quiescent state, so count it. + * + * Also do a memory barrier. This is needed to handle + * the case where writes from a preempt-disable section + * of code get reordered into schedule() by this CPU's + * write buffer. The memory barrier makes sure that + * the rcu_qsctr_inc() and rcu_bh_qsctr_inc() are see + * by other CPUs to happen after any such write. + */ + + smp_mb(); /* See above block comment. */ rcu_qsctr_inc(cpu); rcu_bh_qsctr_inc(cpu); - } else if (!in_softirq()) + + } else if (!in_softirq()) { + + /* + * Get here if this CPU did not take its interrupt from + * softirq, in other words, if it is not interrupting + * a rcu_bh read-side critical section. This is an _bh + * critical section, so count it. The memory barrier + * is needed for the same reason as is the above one. + */ + + smp_mb(); /* See above block comment. */ rcu_bh_qsctr_inc(cpu); + } raise_rcu_softirq(); } ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 3/5] Add rcu_barrier_sched() and rcu_barrier_bh() 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney 2008-04-14 4:00 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney 2008-04-14 4:01 ` [PATCH 2/5] Add memory barriers and comments to rcu_check_callbacks() Paul E. McKenney @ 2008-04-14 4:02 ` Paul E. McKenney 2008-04-14 4:03 ` [PATCH 4/5] Add call_rcu_sched() and friends to rcutorture Paul E. McKenney ` (2 subsequent siblings) 5 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-04-14 4:02 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Add rcu_barrier_sched() and rcu_barrier_bh(). With these in place, rcutorture no longer gives the occasional oops when repeatedly starting and stopping torturing rcu_bh. Also adds the API needed to flush out pre-existing call_rcu_sched() callbacks. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> --- include/linux/rcupdate.h | 2 + kernel/rcupdate.c | 55 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 51 insertions(+), 6 deletions(-) diff -urpNa -X dontdiff linux-2.6.25-rc6-C2-rcuclassic-fixes/include/linux/rcupdate.h linux-2.6.25-rc6-C3-rcu_barrier_sched/include/linux/rcupdate.h --- linux-2.6.25-rc6-C2-rcuclassic-fixes/include/linux/rcupdate.h 2008-03-20 21:10:42.000000000 -0700 +++ linux-2.6.25-rc6-C3-rcu_barrier_sched/include/linux/rcupdate.h 2008-04-09 13:40:53.000000000 -0700 @@ -260,6 +260,8 @@ extern void call_rcu_bh(struct rcu_head /* Exported common interfaces */ extern void synchronize_rcu(void); extern void rcu_barrier(void); +extern void rcu_barrier_bh(void); +extern void rcu_barrier_sched(void); extern long rcu_batches_completed(void); extern long rcu_batches_completed_bh(void); diff -urpNa -X dontdiff linux-2.6.25-rc6-C2-rcuclassic-fixes/kernel/rcupdate.c linux-2.6.25-rc6-C3-rcu_barrier_sched/kernel/rcupdate.c --- linux-2.6.25-rc6-C2-rcuclassic-fixes/kernel/rcupdate.c 2008-03-20 21:10:39.000000000 -0700 +++ linux-2.6.25-rc6-C3-rcu_barrier_sched/kernel/rcupdate.c 2008-04-09 13:54:53.000000000 -0700 @@ -45,6 +45,12 @@ #include <linux/mutex.h> #include <linux/module.h> +enum rcu_barrier { + RCU_BARRIER_STD, + RCU_BARRIER_BH, + RCU_BARRIER_SCHED, +}; + static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; static atomic_t rcu_barrier_cpu_count; static DEFINE_MUTEX(rcu_barrier_mutex); @@ -83,19 +89,30 @@ static void rcu_barrier_callback(struct /* * Called with preemption disabled, and from cross-cpu IRQ context. */ -static void rcu_barrier_func(void *notused) +static void rcu_barrier_func(void *type) { int cpu = smp_processor_id(); struct rcu_head *head = &per_cpu(rcu_barrier_head, cpu); atomic_inc(&rcu_barrier_cpu_count); - call_rcu(head, rcu_barrier_callback); + switch ((enum rcu_barrier)type) { + case RCU_BARRIER_STD: + call_rcu(head, rcu_barrier_callback); + break; + case RCU_BARRIER_BH: + call_rcu_bh(head, rcu_barrier_callback); + break; + case RCU_BARRIER_SCHED: + call_rcu_sched(head, rcu_barrier_callback); + break; + } } -/** - * rcu_barrier - Wait until all the in-flight RCUs are complete. +/* + * Orchestrate the specified type of RCU barrier, waiting for all + * RCU callbacks of the specified type to complete. */ -void rcu_barrier(void) +static void _rcu_barrier(enum rcu_barrier type) { BUG_ON(in_interrupt()); /* Take cpucontrol mutex to protect against CPU hotplug */ @@ -111,13 +128,39 @@ void rcu_barrier(void) * until all the callbacks are queued. */ rcu_read_lock(); - on_each_cpu(rcu_barrier_func, NULL, 0, 1); + on_each_cpu(rcu_barrier_func, (void *)type, 0, 1); rcu_read_unlock(); wait_for_completion(&rcu_barrier_completion); mutex_unlock(&rcu_barrier_mutex); } + +/** + * rcu_barrier - Wait until all in-flight call_rcu() callbacks complete. + */ +void rcu_barrier(void) +{ + _rcu_barrier(RCU_BARRIER_STD); +} EXPORT_SYMBOL_GPL(rcu_barrier); +/** + * rcu_barrier_bh - Wait until all in-flight call_rcu_bh() callbacks complete. + */ +void rcu_barrier_bh(void) +{ + _rcu_barrier(RCU_BARRIER_BH); +} +EXPORT_SYMBOL_GPL(rcu_barrier_bh); + +/** + * rcu_barrier_sched - Wait for in-flight call_rcu_sched() callbacks. + */ +void rcu_barrier_sched(void) +{ + _rcu_barrier(RCU_BARRIER_SCHED); +} +EXPORT_SYMBOL_GPL(rcu_barrier_sched); + void __init rcu_init(void) { __rcu_init(); ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 4/5] Add call_rcu_sched() and friends to rcutorture 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney ` (2 preceding siblings ...) 2008-04-14 4:02 ` [PATCH 3/5] Add rcu_barrier_sched() and rcu_barrier_bh() Paul E. McKenney @ 2008-04-14 4:03 ` Paul E. McKenney 2008-04-14 4:04 ` [PATCH 5/5] 1Q08 RCU doc update, add call_rcu_sched() Paul E. McKenney 2008-04-15 1:33 ` [PATCH 0/5] call_rcu_sched() series Andrew Morton 5 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-04-14 4:03 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Add entry to rcu_torture_ops allowing the correct barrier function to be used upon exit from rcutorture. Also add torture options for the new call_rcu_sched() API. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- rcutorture.c | 34 +++++++++++++++++++++++++++++++--- 1 file changed, 31 insertions(+), 3 deletions(-) diff -urpNa -X dontdiff linux-2.6.25-rc6-C3-rcu_barrier_sched/kernel/rcutorture.c linux-2.6.25-rc6-C4-rcutorture/kernel/rcutorture.c --- linux-2.6.25-rc6-C3-rcu_barrier_sched/kernel/rcutorture.c 2008-03-16 17:45:17.000000000 -0700 +++ linux-2.6.25-rc6-C4-rcutorture/kernel/rcutorture.c 2008-04-09 16:39:14.000000000 -0700 @@ -191,6 +191,7 @@ struct rcu_torture_ops { int (*completed)(void); void (*deferredfree)(struct rcu_torture *p); void (*sync)(void); + void (*cb_barrier)(void); int (*stats)(char *page); char *name; }; @@ -264,6 +265,7 @@ static struct rcu_torture_ops rcu_ops = .completed = rcu_torture_completed, .deferredfree = rcu_torture_deferred_free, .sync = synchronize_rcu, + .cb_barrier = rcu_barrier, .stats = NULL, .name = "rcu" }; @@ -303,6 +305,7 @@ static struct rcu_torture_ops rcu_sync_o .completed = rcu_torture_completed, .deferredfree = rcu_sync_torture_deferred_free, .sync = synchronize_rcu, + .cb_barrier = NULL, .stats = NULL, .name = "rcu_sync" }; @@ -363,6 +366,7 @@ static struct rcu_torture_ops rcu_bh_ops .completed = rcu_bh_torture_completed, .deferredfree = rcu_bh_torture_deferred_free, .sync = rcu_bh_torture_synchronize, + .cb_barrier = rcu_barrier_bh, .stats = NULL, .name = "rcu_bh" }; @@ -376,6 +380,7 @@ static struct rcu_torture_ops rcu_bh_syn .completed = rcu_bh_torture_completed, .deferredfree = rcu_sync_torture_deferred_free, .sync = rcu_bh_torture_synchronize, + .cb_barrier = NULL, .stats = NULL, .name = "rcu_bh_sync" }; @@ -457,6 +462,7 @@ static struct rcu_torture_ops srcu_ops = .completed = srcu_torture_completed, .deferredfree = rcu_sync_torture_deferred_free, .sync = srcu_torture_synchronize, + .cb_barrier = NULL, .stats = srcu_torture_stats, .name = "srcu" }; @@ -481,6 +487,11 @@ static int sched_torture_completed(void) return 0; } +static void rcu_sched_torture_deferred_free(struct rcu_torture *p) +{ + call_rcu_sched(&p->rtort_rcu, rcu_torture_cb); +} + static void sched_torture_synchronize(void) { synchronize_sched(); @@ -493,12 +504,27 @@ static struct rcu_torture_ops sched_ops .readdelay = rcu_read_delay, /* just reuse rcu's version. */ .readunlock = sched_torture_read_unlock, .completed = sched_torture_completed, - .deferredfree = rcu_sync_torture_deferred_free, + .deferredfree = rcu_sched_torture_deferred_free, .sync = sched_torture_synchronize, + .cb_barrier = rcu_barrier_sched, .stats = NULL, .name = "sched" }; +static struct rcu_torture_ops sched_ops_sync = { + .init = rcu_sync_torture_init, + .cleanup = NULL, + .readlock = sched_torture_read_lock, + .readdelay = rcu_read_delay, /* just reuse rcu's version. */ + .readunlock = sched_torture_read_unlock, + .completed = sched_torture_completed, + .deferredfree = rcu_sync_torture_deferred_free, + .sync = sched_torture_synchronize, + .cb_barrier = NULL, + .stats = NULL, + .name = "sched_sync" +}; + /* * RCU torture writer kthread. Repeatedly substitutes a new structure * for that pointed to by rcu_torture_current, freeing the old structure @@ -844,7 +870,9 @@ rcu_torture_cleanup(void) stats_task = NULL; /* Wait for all RCU callbacks to fire. */ - rcu_barrier(); + + if (cur_ops->cb_barrier != NULL) + cur_ops->cb_barrier(); rcu_torture_stats_print(); /* -After- the stats thread is stopped! */ @@ -864,7 +892,7 @@ rcu_torture_init(void) int firsterr = 0; static struct rcu_torture_ops *torture_ops[] = { &rcu_ops, &rcu_sync_ops, &rcu_bh_ops, &rcu_bh_sync_ops, - &srcu_ops, &sched_ops, }; + &srcu_ops, &sched_ops, &sched_ops_sync, }; /* Process args and tell the world that the torturer is on the job. */ for (i = 0; i < ARRAY_SIZE(torture_ops); i++) { ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 5/5] 1Q08 RCU doc update, add call_rcu_sched() 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney ` (3 preceding siblings ...) 2008-04-14 4:03 ` [PATCH 4/5] Add call_rcu_sched() and friends to rcutorture Paul E. McKenney @ 2008-04-14 4:04 ` Paul E. McKenney 2008-04-15 1:33 ` [PATCH 0/5] call_rcu_sched() series Andrew Morton 5 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-04-14 4:04 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Long-delayed update to the RCU documentation, including adding the new call_rcu_sched() and rcu_barrier_sched() APIs. Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> --- NMI-RCU.txt | 3 + RTFP.txt | 108 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ checklist.txt | 89 ++++++++++++++++++++++++++++++++--------------- whatisRCU.txt | 58 ++++++++++++++++++++----------- 4 files changed, 210 insertions(+), 48 deletions(-) diff -urpNa -X dontdiff linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/checklist.txt linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/checklist.txt --- linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/checklist.txt 2008-01-24 14:58:37.000000000 -0800 +++ linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/checklist.txt 2008-04-10 21:38:21.000000000 -0700 @@ -13,10 +13,13 @@ over a rather long period of time, but i detailed performance measurements show that RCU is nonetheless the right tool for the job. - The other exception would be where performance is not an issue, - and RCU provides a simpler implementation. An example of this - situation is the dynamic NMI code in the Linux 2.6 kernel, - at least on architectures where NMIs are rare. + Another exception is where performance is not an issue, and RCU + provides a simpler implementation. An example of this situation + is the dynamic NMI code in the Linux 2.6 kernel, at least on + architectures where NMIs are rare. + + Yet another exception is where the low real-time latency of RCU's + read-side primitives is critically important. 1. Does the update code have proper mutual exclusion? @@ -39,9 +42,10 @@ over a rather long period of time, but i 2. Do the RCU read-side critical sections make proper use of rcu_read_lock() and friends? These primitives are needed - to suppress preemption (or bottom halves, in the case of - rcu_read_lock_bh()) in the read-side critical sections, - and are also an excellent aid to readability. + to prevent grace periods from ending prematurely, which + could result in data being unceremoniously freed out from + under your read-side code, which can greatly increase the + actuarial risk of your kernel. As a rough rule of thumb, any dereference of an RCU-protected pointer must be covered by rcu_read_lock() or rcu_read_lock_bh() @@ -54,15 +58,30 @@ over a rather long period of time, but i be running while updates are in progress. There are a number of ways to handle this concurrency, depending on the situation: - a. Make updates appear atomic to readers. For example, + a. Use the RCU variants of the list and hlist update + primitives to add, remove, and replace elements on an + RCU-protected list. Alternatively, use the RCU-protected + trees that have been added to the Linux kernel. + + This is almost always the best approach. + + b. Proceed as in (a) above, but also maintain per-element + locks (that are acquired by both readers and writers) + that guard per-element state. Of course, fields that + the readers refrain from accessing can be guarded by the + update-side lock. + + This works quite well, also. + + c. Make updates appear atomic to readers. For example, pointer updates to properly aligned fields will appear atomic, as will individual atomic primitives. Operations performed under a lock and sequences of multiple atomic primitives will -not- appear to be atomic. - This is almost always the best approach. + This can work, but is starting to get a bit tricky. - b. Carefully order the updates and the reads so that + d. Carefully order the updates and the reads so that readers see valid data at all phases of the update. This is often more difficult than it sounds, especially given modern CPUs' tendency to reorder memory references. @@ -123,18 +142,22 @@ over a rather long period of time, but i when publicizing a pointer to a structure that can be traversed by an RCU read-side critical section. -5. If call_rcu(), or a related primitive such as call_rcu_bh(), - is used, the callback function must be written to be called - from softirq context. In particular, it cannot block. +5. If call_rcu(), or a related primitive such as call_rcu_bh() or + call_rcu_sched(), is used, the callback function must be + written to be called from softirq context. In particular, + it cannot block. 6. Since synchronize_rcu() can block, it cannot be called from - any sort of irq context. + any sort of irq context. Ditto for synchronize_sched() and + synchronize_srcu(). 7. If the updater uses call_rcu(), then the corresponding readers must use rcu_read_lock() and rcu_read_unlock(). If the updater uses call_rcu_bh(), then the corresponding readers must use - rcu_read_lock_bh() and rcu_read_unlock_bh(). Mixing things up - will result in confusion and broken kernels. + rcu_read_lock_bh() and rcu_read_unlock_bh(). If the updater + uses call_rcu_sched(), then the corresponding readers must + disable preemption. Mixing things up will result in confusion + and broken kernels. One exception to this rule: rcu_read_lock() and rcu_read_unlock() may be substituted for rcu_read_lock_bh() and rcu_read_unlock_bh() @@ -143,9 +166,9 @@ over a rather long period of time, but i such cases is a must, of course! And the jury is still out on whether the increased speed is worth it. -8. Although synchronize_rcu() is a bit slower than is call_rcu(), - it usually results in simpler code. So, unless update - performance is critically important or the updaters cannot block, +8. Although synchronize_rcu() is slower than is call_rcu(), it + usually results in simpler code. So, unless update performance + is critically important or the updaters cannot block, synchronize_rcu() should be used in preference to call_rcu(). An especially important property of the synchronize_rcu() @@ -187,23 +210,23 @@ over a rather long period of time, but i number of updates per grace period. 9. All RCU list-traversal primitives, which include - list_for_each_rcu(), list_for_each_entry_rcu(), + rcu_dereference(), list_for_each_rcu(), list_for_each_entry_rcu(), list_for_each_continue_rcu(), and list_for_each_safe_rcu(), - must be within an RCU read-side critical section. RCU + must be either within an RCU read-side critical section or + must be protected by appropriate update-side locks. RCU read-side critical sections are delimited by rcu_read_lock() and rcu_read_unlock(), or by similar primitives such as rcu_read_lock_bh() and rcu_read_unlock_bh(). - Use of the _rcu() list-traversal primitives outside of an - RCU read-side critical section causes no harm other than - a slight performance degradation on Alpha CPUs. It can - also be quite helpful in reducing code bloat when common - code is shared between readers and updaters. + The reason that it is permissible to use RCU list-traversal + primitives when the update-side lock is held is that doing so + can be quite helpful in reducing code bloat when common code is + shared between readers and updaters. 10. Conversely, if you are in an RCU read-side critical section, - you -must- use the "_rcu()" variants of the list macros. - Failing to do so will break Alpha and confuse people reading - your code. + and you don't hold the appropriate update-side lock, you -must- + use the "_rcu()" variants of the list macros. Failing to do so + will break Alpha and confuse people reading your code. 11. Note that synchronize_rcu() -only- guarantees to wait until all currently executing rcu_read_lock()-protected RCU read-side @@ -230,6 +253,14 @@ over a rather long period of time, but i must use whatever locking or other synchronization is required to safely access and/or modify that data structure. + RCU callbacks are -usually- executed on the same CPU that executed + the corresponding call_rcu(), call_rcu_bh(), or call_rcu_sched(), + but are by -no- means guaranteed to be. For example, if a given + CPU goes offline while having an RCU callback pending, then that + RCU callback will execute on some surviving CPU. (If this was + not the case, a self-spawning RCU callback would prevent the + victim CPU from ever going offline.) + 14. SRCU (srcu_read_lock(), srcu_read_unlock(), and synchronize_srcu()) may only be invoked from process context. Unlike other forms of RCU, it -is- permissible to block in an SRCU read-side critical diff -urpNa -X dontdiff linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/NMI-RCU.txt linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/NMI-RCU.txt --- linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/NMI-RCU.txt 2008-03-16 17:44:58.000000000 -0700 +++ linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/NMI-RCU.txt 2008-04-10 21:39:12.000000000 -0700 @@ -93,6 +93,9 @@ Since NMI handlers disable preemption, s not to return until all ongoing NMI handlers exit. It is therefore safe to free up the handler's data as soon as synchronize_sched() returns. +Important note: for this to work, the architecture in question must +invoke irq_enter() and irq_exit() on NMI entry and exit, respectively. + Answer to Quick Quiz diff -urpNa -X dontdiff linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/RTFP.txt linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/RTFP.txt --- linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/RTFP.txt 2008-03-16 17:44:58.000000000 -0700 +++ linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/RTFP.txt 2008-04-11 17:17:09.000000000 -0700 @@ -52,6 +52,10 @@ of each iteration. Unfortunately, chaot structured data, such as the matrices used in scientific programs, and is thus inapplicable to most data structures in operating-system kernels. +In 1992, Henry (now Alexia) Massalin completed a dissertation advising +parallel programmers to defer processing when feasible to simplify +synchronization. RCU makes extremely heavy use of this advice. + In 1993, Jacobson [Jacobson93] verbally described what is perhaps the simplest deferred-free technique: simply waiting a fixed amount of time before freeing blocks awaiting deferred free. Jacobson did not describe @@ -138,6 +142,13 @@ blocking in read-side critical sections Robert Olsson described an RCU-protected trie-hash combination [RobertOlsson2006a]. +2007 saw the journal version of the award-winning RCU paper from 2006 +[ThomasEHart2007a], as well as a paper demonstrating use of Promela +and Spin to mechanically verify an optimization to Oleg Nesterov's +QRCU [PaulEMcKenney2007QRCUspin], a design document describing +preemptible RCU [PaulEMcKenney2007PreemptibleRCU], and the three-part +LWN "What is RCU?" series [PaulEMcKenney2007WhatIsRCUFundamentally, +PaulEMcKenney2008WhatIsRCUUsage, and PaulEMcKenney2008WhatIsRCUAPI]. Bibtex Entries @@ -202,6 +213,20 @@ Bibtex Entries ,Year="1991" } +@phdthesis{HMassalinPhD +,author="H. Massalin" +,title="Synthesis: An Efficient Implementation of Fundamental Operating +System Services" +,school="Columbia University" +,address="New York, NY" +,year="1992" +,annotation=" + Mondo optimizing compiler. + Wait-free stuff. + Good advice: defer work to avoid synchronization. +" +} + @unpublished{Jacobson93 ,author="Van Jacobson" ,title="Avoid Read-Side Locking Via Delayed Free" @@ -635,3 +660,86 @@ Revised: " } +@unpublished{PaulEMcKenney2007PreemptibleRCU +,Author="Paul E. McKenney" +,Title="The design of preemptible read-copy-update" +,month="October" +,day="8" +,year="2007" +,note="Available: +\url{http://lwn.net/Articles/253651/} +[Viewed October 25, 2007]" +,annotation=" + LWN article describing the design of preemptible RCU. +" +} + +######################################################################## +# +# "What is RCU?" LWN series. +# + +@unpublished{PaulEMcKenney2007WhatIsRCUFundamentally +,Author="Paul E. McKenney and Jonathan Walpole" +,Title="What is {RCU}, Fundamentally?" +,month="December" +,day="17" +,year="2007" +,note="Available: +\url{http://lwn.net/Articles/262464/} +[Viewed December 27, 2007]" +,annotation=" + Lays out the three basic components of RCU: (1) publish-subscribe, + (2) wait for pre-existing readers to complete, and (2) maintain + multiple versions. +" +} + +@unpublished{PaulEMcKenney2008WhatIsRCUUsage +,Author="Paul E. McKenney" +,Title="What is {RCU}? Part 2: Usage" +,month="January" +,day="4" +,year="2008" +,note="Available: +\url{http://lwn.net/Articles/263130/} +[Viewed January 4, 2008]" +,annotation=" + Lays out six uses of RCU: + 1. RCU is a Reader-Writer Lock Replacement + 2. RCU is a Restricted Reference-Counting Mechanism + 3. RCU is a Bulk Reference-Counting Mechanism + 4. RCU is a Poor Man's Garbage Collector + 5. RCU is a Way of Providing Existence Guarantees + 6. RCU is a Way of Waiting for Things to Finish +" +} + +@unpublished{PaulEMcKenney2008WhatIsRCUAPI +,Author="Paul E. McKenney" +,Title="{RCU} part 3: the {RCU} {API}" +,month="January" +,day="17" +,year="2008" +,note="Available: +\url{http://lwn.net/Articles/264090/} +[Viewed January 10, 2008]" +,annotation=" + Gives an overview of the Linux-kernel RCU API and a brief annotated RCU + bibliography. +" +} + +@article{DinakarGuniguntala2008IBMSysJ +,author="D. Guniguntala and P. E. McKenney and J. Triplett and J. Walpole" +,title="The read-copy-update mechanism for supporting real-time applications on shared-memory multiprocessor systems with {Linux}" +,Year="2008" +,Month="April" +,journal="IBM Systems Journal" +,volume="47" +,number="2" +,pages="@@-@@" +,annotation=" + RCU, realtime RCU, sleepable RCU, performance. +" +} diff -urpNa -X dontdiff linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/whatisRCU.txt linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/whatisRCU.txt --- linux-2.6.25-rc6-C4-rcutorture/Documentation/RCU/whatisRCU.txt 2008-01-24 14:58:37.000000000 -0800 +++ linux-2.6.25-rc6-C5-rcudoc/Documentation/RCU/whatisRCU.txt 2008-04-13 10:33:57.000000000 -0700 @@ -1,3 +1,11 @@ +Please note that the "What is RCU?" LWN series is an excellent place +to start learning about RCU: + +1. What is RCU, Fundamentally? http://lwn.net/Articles/262464/ +2. What is RCU? Part 2: Usage http://lwn.net/Articles/263130/ +3. RCU part 3: the RCU API http://lwn.net/Articles/264090/ + + What is RCU? RCU is a synchronization mechanism that was added to the Linux kernel @@ -772,26 +780,18 @@ Linux-kernel source code, but it helps t APIs, since there does not appear to be a way to categorize them in docbook. Here is the list, by category. -Markers for RCU read-side critical sections: - - rcu_read_lock - rcu_read_unlock - rcu_read_lock_bh - rcu_read_unlock_bh - srcu_read_lock - srcu_read_unlock - RCU pointer/list traversal: rcu_dereference + list_for_each_entry_rcu + hlist_for_each_entry_rcu + list_for_each_rcu (to be deprecated in favor of list_for_each_entry_rcu) - list_for_each_entry_rcu list_for_each_continue_rcu (to be deprecated in favor of new list_for_each_entry_continue_rcu) - hlist_for_each_entry_rcu -RCU pointer update: +RCU pointer/list update: rcu_assign_pointer list_add_rcu @@ -799,16 +799,36 @@ RCU pointer update: list_del_rcu list_replace_rcu hlist_del_rcu + hlist_add_after_rcu + hlist_add_before_rcu hlist_add_head_rcu + hlist_replace_rcu + list_splice_init_rcu() -RCU grace period: +RCU: Critical sections Grace period Barrier + + rcu_read_lock synchronize_net rcu_barrier + rcu_read_unlock synchronize_rcu + call_rcu + + +bh: Critical sections Grace period Barrier + + rcu_read_lock_bh call_rcu_bh rcu_barrier_bh + rcu_read_unlock_bh + + +sched: Critical sections Grace period Barrier + + [preempt_disable] synchronize_sched rcu_barrier_sched + [and friends] call_rcu_sched + + +SRCU: Critical sections Grace period Barrier + + srcu_read_lock synchronize_srcu N/A + srcu_read_unlock - synchronize_net - synchronize_sched - synchronize_rcu - synchronize_srcu - call_rcu - call_rcu_bh See the comment headers in the source code (or the docbook generated from them) for more information. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/5] call_rcu_sched() series 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney ` (4 preceding siblings ...) 2008-04-14 4:04 ` [PATCH 5/5] 1Q08 RCU doc update, add call_rcu_sched() Paul E. McKenney @ 2008-04-15 1:33 ` Andrew Morton 2008-04-15 1:45 ` Andrew Morton 5 siblings, 1 reply; 12+ messages in thread From: Andrew Morton @ 2008-04-15 1:33 UTC (permalink / raw) To: paulmck Cc: linux-kernel, mathieu.desnoyers, mingo, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang On Sun, 13 Apr 2008 20:58:48 -0700 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > This series adds call_rcu_sched(), applies some fixes to RCU Classic, > adds rcu_barrier_sched(), updates rcutorture, and updates the RCU > documentation. Some massaging was needed here to make these apply against the RCU changes in git-sched. All fairly simple, except I don't know if these: static inline void rcu_enter_nohz(void) { smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ __get_cpu_var(rcu_dyntick_sched).dynticks++; WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); } static inline void rcu_exit_nohz(void) { __get_cpu_var(rcu_dyntick_sched).dynticks++; smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); } in include/linux/rcupreempt.h ended up the way we want them? ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/5] call_rcu_sched() series 2008-04-15 1:33 ` [PATCH 0/5] call_rcu_sched() series Andrew Morton @ 2008-04-15 1:45 ` Andrew Morton 2008-04-15 1:48 ` Andrew Morton 0 siblings, 1 reply; 12+ messages in thread From: Andrew Morton @ 2008-04-15 1:45 UTC (permalink / raw) To: paulmck, linux-kernel, mathieu.desnoyers, mingo, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang On Mon, 14 Apr 2008 18:33:12 -0700 Andrew Morton <akpm@linux-foundation.org> wrote: > On Sun, 13 Apr 2008 20:58:48 -0700 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > > > This series adds call_rcu_sched(), applies some fixes to RCU Classic, > > adds rcu_barrier_sched(), updates rcutorture, and updates the RCU > > documentation. > > Some massaging was needed here to make these apply against the > RCU changes in git-sched. > > All fairly simple, except I don't know if these: > > static inline void rcu_enter_nohz(void) > { > smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ > __get_cpu_var(rcu_dyntick_sched).dynticks++; > WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); > } > > static inline void rcu_exit_nohz(void) > { > __get_cpu_var(rcu_dyntick_sched).dynticks++; > smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ > WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); > } > > in include/linux/rcupreempt.h ended up the way we want them? argh, I see. It's all tangled up with the WARN_ON_SECS stuff. Let me try to straighten it all out. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/5] call_rcu_sched() series 2008-04-15 1:45 ` Andrew Morton @ 2008-04-15 1:48 ` Andrew Morton 0 siblings, 0 replies; 12+ messages in thread From: Andrew Morton @ 2008-04-15 1:48 UTC (permalink / raw) To: paulmck, linux-kernel, mathieu.desnoyers, mingo, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang On Mon, 14 Apr 2008 18:45:44 -0700 Andrew Morton <akpm@linux-foundation.org> wrote: > On Mon, 14 Apr 2008 18:33:12 -0700 Andrew Morton <akpm@linux-foundation.org> wrote: > > > On Sun, 13 Apr 2008 20:58:48 -0700 "Paul E. McKenney" <paulmck@linux.vnet.ibm.com> wrote: > > > > > This series adds call_rcu_sched(), applies some fixes to RCU Classic, > > > adds rcu_barrier_sched(), updates rcutorture, and updates the RCU > > > documentation. > > > > Some massaging was needed here to make these apply against the > > RCU changes in git-sched. > > > > All fairly simple, except I don't know if these: > > > > static inline void rcu_enter_nohz(void) > > { > > smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ > > __get_cpu_var(rcu_dyntick_sched).dynticks++; > > WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); > > } > > > > static inline void rcu_exit_nohz(void) > > { > > __get_cpu_var(rcu_dyntick_sched).dynticks++; > > smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ > > WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); > > } > > > > in include/linux/rcupreempt.h ended up the way we want them? > > argh, I see. It's all tangled up with the WARN_ON_SECS stuff. No it isn't. Don't mind me. ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 0/5] call_rcu_sched() series @ 2008-04-22 0:44 Paul E. McKenney 2008-04-22 0:45 ` Paul E. McKenney 0 siblings, 1 reply; 12+ messages in thread From: Paul E. McKenney @ 2008-04-22 0:44 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang This series adds call_rcu_sched() and related tracing, applies some fixes to RCU Classic, adds rcu_barrier_sched(), updates rcutorture, and updates the RCU documentation. Drop-in replacement for patches in http://lkml.org/lkml/2008/4/13/375, but ported to 2.6.25. ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 0/5] call_rcu_sched() series 2008-04-22 0:44 Paul E. McKenney @ 2008-04-22 0:45 ` Paul E. McKenney 2008-04-22 0:50 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney 0 siblings, 1 reply; 12+ messages in thread From: Paul E. McKenney @ 2008-04-22 0:45 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang Fifth cut of patch to provide the call_rcu_sched(). This is again to synchronize_sched() as call_rcu() is to synchronize_rcu(). OK, Andrew, it finally appears safe for you to come out of hiding! ;-) (Although I still don't totally trust this, I do trust it more than I do what is currently in mainline.) Passes multi-day rcutorture sessions with concurrent CPU hotplugging. Fixes since the first version include a bug that could result in indefinite blocking (spotted by Gautham Shenoy), better resiliency against CPU-hotplug operations, and other minor fixes. Fixes since the second version include reworking grace-period detection to avoid deadlocks that could happen when running concurrently with CPU hotplug, adding Mathieu's fix to avoid the softlockup messages, as well as Mathieu's fix to allow use earlier in boot. Fixes since the third version include a wrong-CPU bug spotted by Andrew, getting rid of the obsolete synchronize_kernel API that somehow snuck back in, merging spin_unlock() and local_irq_restore() in a few places, commenting the code that checks for quiescent states based on interrupting from user-mode execution or the idle loop, removing some inline attributes, and some code-style changes. Fixes since the fourth version include snapshotting the online CPUs inside of rcu_sched_grace_period() to avoid races with CPUs onlining and offlining, adding rudimentary tracing for rcu_sched, and pulling in a RCU-trace fix from -rt (rcu-trace-fix-free.patch). Known/suspected shortcomings: o Torturing rcu_sched while onlining and offlining CPUs with CONFIG_NO_HZ and CONFIG_PREEMPT_RCU can provoke a race where the event kthread spins on a given CPU attempting to run workqueues, and blocking rcu_sched grace periods, in turn blocking attempts to hotplug CPUs. But this can also happen given the code currently in mainline. o It might be good for rcu_sched_grace_period() to invoke resched_cpu() when a given CPU wasn't responding quickly, but resched_cpu() is declared static... But this is a minor optimization, so dispensing with this unless and until it proves to be a problem. This patch also fixes a long-standing bug in the earlier preemptable-RCU implementation of synchronize_rcu() that could result in loss of concurrent external changes to a task's CPU affinity mask. I still cannot remember who reported this... Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> --- include/linux/rcuclassic.h | 3 include/linux/rcupdate.h | 22 + include/linux/rcupreempt.h | 42 +++ include/linux/rcupreempt_trace.h | 6 init/main.c | 1 kernel/rcupdate.c | 20 - kernel/rcupreempt.c | 433 +++++++++++++++++++++++++++++++++++---- kernel/rcupreempt_trace.c | 58 +++++ 8 files changed, 514 insertions(+), 71 deletions(-) diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcuclassic.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h --- linux-2.6.25/include/linux/rcuclassic.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h 2008-04-17 08:10:33.000000000 -0700 @@ -153,7 +153,10 @@ extern struct lockdep_map rcu_lock_map; #define __synchronize_sched() synchronize_rcu() +#define call_rcu_sched(head, func) call_rcu(head, func) + extern void __rcu_init(void); +#define rcu_init_sched() do { } while (0) extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupdate.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h --- linux-2.6.25/include/linux/rcupdate.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h 2008-04-17 08:10:33.000000000 -0700 @@ -42,6 +42,7 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> #include <linux/lockdep.h> +#include <linux/completion.h> /** * struct rcu_head - callback structure for use with RCU @@ -182,6 +183,27 @@ struct rcu_head { (p) = (v); \ }) +/* Infrastructure to implement the synchronize_() primitives. */ + +struct rcu_synchronize { + struct rcu_head head; + struct completion completion; +}; + +extern void wakeme_after_rcu(struct rcu_head *head); + +#define synchronize_rcu_xxx(name, func) \ +void name(void) \ +{ \ + struct rcu_synchronize rcu; \ + \ + init_completion(&rcu.completion); \ + /* Will wake me after RCU finished. */ \ + func(&rcu.head, wakeme_after_rcu); \ + /* Wait for it. */ \ + wait_for_completion(&rcu.completion); \ +} + /** * synchronize_sched - block until all CPUs have exited any non-preemptive * kernel code sequences. diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h --- linux-2.6.25/include/linux/rcupreempt.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h 2008-04-19 22:44:18.000000000 -0700 @@ -42,10 +42,39 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> -#define rcu_qsctr_inc(cpu) +struct rcu_dyntick_sched { + int dynticks; + int dynticks_snap; + int sched_qs; + int sched_qs_snap; + int sched_dynticks_snap; +}; + +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); + +static inline void rcu_qsctr_inc(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs++; +} #define rcu_bh_qsctr_inc(cpu) #define call_rcu_bh(head, rcu) call_rcu(head, rcu) +/** + * call_rcu_sched - Queue RCU callback for invocation after sched grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual update function to be invoked after the grace period + * + * The update function will be invoked some time after a full + * synchronize_sched()-style grace period elapses, in other words after + * all currently executing preempt-disabled sections of code (including + * hardirq handlers, NMI handlers, and local_irq_save() blocks) have + * completed. + */ +extern void call_rcu_sched(struct rcu_head *head, + void (*func)(struct rcu_head *head)); + extern void __rcu_read_lock(void) __acquires(RCU); extern void __rcu_read_unlock(void) __releases(RCU); extern int rcu_pending(int cpu); @@ -57,6 +86,7 @@ extern int rcu_needs_cpu(int cpu); extern void __synchronize_sched(void); extern void __rcu_init(void); +extern void rcu_init_sched(void); extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); extern long rcu_batches_completed(void); @@ -83,20 +113,20 @@ extern struct rcupreempt_trace *rcupreem struct softirq_action; #ifdef CONFIG_NO_HZ -DECLARE_PER_CPU(long, dynticks_progress_counter); +DECLARE_PER_CPU(struct rcu_dyntick_sched, dynticks_progress_counter); static inline void rcu_enter_nohz(void) { smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ - __get_cpu_var(dynticks_progress_counter)++; - WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1); + __get_cpu_var(rcu_dyntick_sched).dynticks++; + WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); } static inline void rcu_exit_nohz(void) { - __get_cpu_var(dynticks_progress_counter)++; + __get_cpu_var(rcu_dyntick_sched).dynticks++; smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ - WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1)); + WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); } #else /* CONFIG_NO_HZ */ diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt_trace.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h --- linux-2.6.25/include/linux/rcupreempt_trace.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h 2008-04-20 07:44:41.000000000 -0700 @@ -66,12 +66,15 @@ struct rcupreempt_trace { long rcu_try_flip_m1; long rcu_try_flip_me1; long rcu_try_flip_m2; + int rcu_sched_cpu; }; #ifdef CONFIG_RCU_TRACE -#define RCU_TRACE(fn, arg) fn(arg); +#define RCU_TRACE(fn, arg) fn(arg) +#define RCU_TRACE_1ARG(fn, arg, arg1) fn(arg, arg1) #else #define RCU_TRACE(fn, arg) +#define RCU_TRACE_1ARG(fn, arg, arg1) #endif extern void rcupreempt_trace_move2done(struct rcupreempt_trace *trace); @@ -94,6 +97,7 @@ extern void rcupreempt_trace_check_callb extern void rcupreempt_trace_done_remove(struct rcupreempt_trace *trace); extern void rcupreempt_trace_invoke(struct rcupreempt_trace *trace); extern void rcupreempt_trace_next_add(struct rcupreempt_trace *trace); +extern void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu); #endif /* __KERNEL__ */ #endif /* __LINUX_RCUPREEMPT_TRACE_H */ diff -urpNa -X dontdiff linux-2.6.25/init/main.c linux-2.6.25-C1-call_rcu_sched/init/main.c --- linux-2.6.25/init/main.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/init/main.c 2008-04-17 08:10:33.000000000 -0700 @@ -730,6 +730,7 @@ static void __init do_initcalls(void) */ static void __init do_basic_setup(void) { + rcu_init_sched(); /* needed by module_init stage. */ /* drivers will send hotplug events */ init_workqueues(); usermodehelper_init(); diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupdate.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c --- linux-2.6.25/kernel/rcupdate.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c 2008-04-17 08:10:33.000000000 -0700 @@ -39,18 +39,12 @@ #include <linux/sched.h> #include <asm/atomic.h> #include <linux/bitops.h> -#include <linux/completion.h> #include <linux/percpu.h> #include <linux/notifier.h> #include <linux/cpu.h> #include <linux/mutex.h> #include <linux/module.h> -struct rcu_synchronize { - struct rcu_head head; - struct completion completion; -}; - static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; static atomic_t rcu_barrier_cpu_count; static DEFINE_MUTEX(rcu_barrier_mutex); @@ -60,7 +54,7 @@ static struct completion rcu_barrier_com * Awaken the corresponding synchronize_rcu() instance now that a * grace period has elapsed. */ -static void wakeme_after_rcu(struct rcu_head *head) +void wakeme_after_rcu(struct rcu_head *head) { struct rcu_synchronize *rcu; @@ -77,17 +71,7 @@ static void wakeme_after_rcu(struct rcu_ * sections are delimited by rcu_read_lock() and rcu_read_unlock(), * and may be nested. */ -void synchronize_rcu(void) -{ - struct rcu_synchronize rcu; - - init_completion(&rcu.completion); - /* Will wake me after RCU finished */ - call_rcu(&rcu.head, wakeme_after_rcu); - - /* Wait for it */ - wait_for_completion(&rcu.completion); -} +synchronize_rcu_xxx(synchronize_rcu, call_rcu) EXPORT_SYMBOL_GPL(synchronize_rcu); static void rcu_barrier_callback(struct rcu_head *notused) diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c --- linux-2.6.25/kernel/rcupreempt.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c 2008-04-20 07:58:31.000000000 -0700 @@ -46,6 +46,7 @@ #include <asm/atomic.h> #include <linux/bitops.h> #include <linux/module.h> +#include <linux/kthread.h> #include <linux/completion.h> #include <linux/moduleparam.h> #include <linux/percpu.h> @@ -87,9 +88,14 @@ struct rcu_data { struct rcu_head **nexttail; struct rcu_head *waitlist[GP_STAGES]; struct rcu_head **waittail[GP_STAGES]; - struct rcu_head *donelist; + struct rcu_head *donelist; /* from waitlist & waitschedlist */ struct rcu_head **donetail; long rcu_flipctr[2]; + struct rcu_head *nextschedlist; + struct rcu_head **nextschedtail; + struct rcu_head *waitschedlist; + struct rcu_head **waitschedtail; + int rcu_sched_sleeping; #ifdef CONFIG_RCU_TRACE struct rcupreempt_trace trace; #endif /* #ifdef CONFIG_RCU_TRACE */ @@ -131,11 +137,24 @@ enum rcu_try_flip_states { rcu_try_flip_waitmb_state, }; +/* + * States for rcu_ctrlblk.rcu_sched_sleep. + */ + +enum rcu_sched_sleep_states { + rcu_sched_not_sleeping, /* Not sleeping, callbacks need GP. */ + rcu_sched_sleep_prep, /* Thinking of sleeping, rechecking. */ + rcu_sched_sleeping, /* Sleeping, awaken if GP needed. */ +}; + struct rcu_ctrlblk { spinlock_t fliplock; /* Protect state-machine transitions. */ long completed; /* Number of last completed batch. */ enum rcu_try_flip_states rcu_try_flip_state; /* The current state of the rcu state machine */ + spinlock_t schedlock; /* Protect rcu_sched sleep state. */ + enum rcu_sched_sleep_states sched_sleep; /* rcu_sched state. */ + wait_queue_head_t sched_wq; /* Place for rcu_sched to sleep. */ }; static DEFINE_PER_CPU(struct rcu_data, rcu_data); @@ -143,8 +162,12 @@ static struct rcu_ctrlblk rcu_ctrlblk = .fliplock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.fliplock), .completed = 0, .rcu_try_flip_state = rcu_try_flip_idle_state, + .schedlock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.schedlock), + .sched_sleep = rcu_sched_not_sleeping, + .sched_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rcu_ctrlblk.sched_wq), }; +static struct task_struct *rcu_sched_grace_period_task; #ifdef CONFIG_RCU_TRACE static char *rcu_try_flip_state_names[] = @@ -194,6 +217,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu * cached in a local variable, but where the CPU number is so cached. */ #define RCU_TRACE_CPU(f, cpu) RCU_TRACE(f, &(RCU_DATA_CPU(cpu)->trace)); +#define RCU_TRACE_1ARG_CPU(f, cpu, a) RCU_TRACE_1ARG(f, &(RCU_DATA_CPU(cpu)->trace), a); /* * Helper macro for tracing when the appropriate rcu_data is not @@ -207,6 +231,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu */ #define RCU_TRACE_RDP(f, rdp) RCU_TRACE(f, &((rdp)->trace)); +#define RCU_SCHED_BATCH_TIME (HZ / 50) + /* * Return the number of RCU batches processed thus far. Useful * for debug and statistics. @@ -413,32 +439,34 @@ static void __rcu_advance_callbacks(stru } } -#ifdef CONFIG_NO_HZ +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_dyntick_sched, rcu_dyntick_sched) = { + .dynticks = 1, +}; -DEFINE_PER_CPU(long, dynticks_progress_counter) = 1; -static DEFINE_PER_CPU(long, rcu_dyntick_snapshot); +#ifdef CONFIG_NO_HZ static DEFINE_PER_CPU(int, rcu_update_flag); /** * rcu_irq_enter - Called from Hard irq handlers and NMI/SMI. * * If the CPU was idle with dynamic ticks active, this updates the - * dynticks_progress_counter to let the RCU handling know that the + * rcu_dyntick_sched.dynticks to let the RCU handling know that the * CPU is active. */ void rcu_irq_enter(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); if (per_cpu(rcu_update_flag, cpu)) per_cpu(rcu_update_flag, cpu)++; /* * Only update if we are coming from a stopped ticks mode - * (dynticks_progress_counter is even). + * (rcu_dyntick_sched.dynticks is even). */ if (!in_interrupt() && - (per_cpu(dynticks_progress_counter, cpu) & 0x1) == 0) { + (rdssp->dynticks & 0x1) == 0) { /* * The following might seem like we could have a race * with NMI/SMIs. But this really isn't a problem. @@ -461,12 +489,12 @@ void rcu_irq_enter(void) * RCU read-side critical sections on this CPU would * have already completed. */ - per_cpu(dynticks_progress_counter, cpu)++; + rdssp->dynticks++; /* * The following memory barrier ensures that any * rcu_read_lock() primitives in the irq handler * are seen by other CPUs to follow the above - * increment to dynticks_progress_counter. This is + * increment to rcu_dyntick_sched.dynticks. This is * required in order for other CPUs to correctly * determine when it is safe to advance the RCU * grace-period state machine. @@ -474,7 +502,7 @@ void rcu_irq_enter(void) smp_mb(); /* see above block comment. */ /* * Since we can't determine the dynamic tick mode from - * the dynticks_progress_counter after this routine, + * the rcu_dyntick_sched.dynticks after this routine, * we use a second flag to acknowledge that we came * from an idle state with ticks stopped. */ @@ -482,7 +510,7 @@ void rcu_irq_enter(void) /* * If we take an NMI/SMI now, they will also increment * the rcu_update_flag, and will not update the - * dynticks_progress_counter on exit. That is for + * rcu_dyntick_sched.dynticks on exit. That is for * this IRQ to do. */ } @@ -492,12 +520,13 @@ void rcu_irq_enter(void) * rcu_irq_exit - Called from exiting Hard irq context. * * If the CPU was idle with dynamic ticks active, update the - * dynticks_progress_counter to put let the RCU handling be + * rcu_dyntick_sched.dynticks to put let the RCU handling be * aware that the CPU is going back to idle with no ticks. */ void rcu_irq_exit(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); /* * rcu_update_flag is set if we interrupted the CPU @@ -505,7 +534,7 @@ void rcu_irq_exit(void) * Once this occurs, we keep track of interrupt nesting * because a NMI/SMI could also come in, and we still * only want the IRQ that started the increment of the - * dynticks_progress_counter to be the one that modifies + * rcu_dyntick_sched.dynticks to be the one that modifies * it on exit. */ if (per_cpu(rcu_update_flag, cpu)) { @@ -517,28 +546,29 @@ void rcu_irq_exit(void) /* * If an NMI/SMI happens now we are still - * protected by the dynticks_progress_counter being odd. + * protected by the rcu_dyntick_sched.dynticks being odd. */ /* * The following memory barrier ensures that any * rcu_read_unlock() primitives in the irq handler * are seen by other CPUs to preceed the following - * increment to dynticks_progress_counter. This + * increment to rcu_dyntick_sched.dynticks. This * is required in order for other CPUs to determine * when it is safe to advance the RCU grace-period * state machine. */ smp_mb(); /* see above block comment. */ - per_cpu(dynticks_progress_counter, cpu)++; - WARN_ON(per_cpu(dynticks_progress_counter, cpu) & 0x1); + rdssp->dynticks++; + WARN_ON(rdssp->dynticks & 0x1); } } static void dyntick_save_progress_counter(int cpu) { - per_cpu(rcu_dyntick_snapshot, cpu) = - per_cpu(dynticks_progress_counter, cpu); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->dynticks_snap = rdssp->dynticks; } static inline int @@ -546,9 +576,10 @@ rcu_try_flip_waitack_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -582,9 +613,10 @@ rcu_try_flip_waitmb_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -611,14 +643,85 @@ rcu_try_flip_waitmb_needed(int cpu) return 1; } +static void dyntick_save_progress_counter_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_dynticks_snap = rdssp->dynticks; +} + +static int rcu_qsctr_inc_needed_dyntick(int cpu) +{ + long curr; + long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + curr = rdssp->dynticks; + snap = rdssp->sched_dynticks_snap; + smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ + + /* + * If the CPU remained in dynticks mode for the entire time + * and didn't take any interrupts, NMIs, SMIs, or whatever, + * then it cannot be in the middle of an preempt-disabled + * region of code. Therefore, this CPU has been in a quiescent + * state the entire time, and we don't need to wait for it. + */ + + if ((curr == snap) && ((curr & 0x1) == 0)) + return 0; + + /* + * If the CPU passed through or entered a dynticks idle phase with + * no active irq handlers, then, as above, this CPU has already + * passed through a quiescent state. + */ + + if ((curr - snap) > 2 || (snap & 0x1) == 0) + return 0; + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + #else /* !CONFIG_NO_HZ */ -# define dyntick_save_progress_counter(cpu) do { } while (0) -# define rcu_try_flip_waitack_needed(cpu) (1) -# define rcu_try_flip_waitmb_needed(cpu) (1) +# define dyntick_save_progress_counter(cpu) do { } while (0) +# define rcu_try_flip_waitack_needed(cpu) (1) +# define rcu_try_flip_waitmb_needed(cpu) (1) + +# define dyntick_save_progress_counter_sched(cpu) do { } while (0) +# define rcu_qsctr_inc_needed_dyntick(cpu) (1) #endif /* CONFIG_NO_HZ */ +static void save_qsctr_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs_snap = rdssp->sched_qs; +} + +static inline int rcu_qsctr_inc_needed(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + /* + * If there has been a quiescent state, no more need to wait + * on this CPU. + */ + + if (rdssp->sched_qs != rdssp->sched_qs_snap) { + smp_mb(); /* force ordering with cpu entering schedule(). */ + return 0; + } + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + /* * Get here when RCU is idle. Decide whether we need to * move out of idle state, and return non-zero if so. @@ -821,6 +924,26 @@ void rcu_check_callbacks(int cpu, int us unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + /* + * If this CPU took its interrupt from user mode or from the + * idle loop, and this is not a nested interrupt, then + * this CPU has to have exited all prior preept-disable + * sections of code. So increment the counter to note this. + * + * The memory barrier is needed to handle the case where + * writes from a preempt-disable section of code get reordered + * into schedule() by this CPU's write buffer. So the memory + * barrier makes sure that the rcu_qsctr_inc() is seen by other + * CPUs to happen after any such write. + */ + + if (user || + (idle_cpu(cpu) && !in_softirq() && + hardirq_count() <= (1 << HARDIRQ_SHIFT))) { + smp_mb(); /* Guard against aggressive schedule(). */ + rcu_qsctr_inc(cpu); + } + rcu_check_mb(cpu); if (rcu_ctrlblk.completed == rdp->completed) rcu_try_flip(); @@ -871,6 +994,8 @@ void rcu_offline_cpu(int cpu) struct rcu_head *list = NULL; unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + struct rcu_head *schedlist = NULL; + struct rcu_head **schedtail = &schedlist; struct rcu_head **tail = &list; /* @@ -884,6 +1009,11 @@ void rcu_offline_cpu(int cpu) rcu_offline_cpu_enqueue(rdp->waitlist[i], rdp->waittail[i], list, tail); rcu_offline_cpu_enqueue(rdp->nextlist, rdp->nexttail, list, tail); + rcu_offline_cpu_enqueue(rdp->waitschedlist, rdp->waitschedtail, + schedlist, schedtail); + rcu_offline_cpu_enqueue(rdp->nextschedlist, rdp->nextschedtail, + schedlist, schedtail); + rdp->rcu_sched_sleeping = 0; spin_unlock_irqrestore(&rdp->lock, flags); rdp->waitlistcount = 0; @@ -918,22 +1048,40 @@ void rcu_offline_cpu(int cpu) * fix. */ - local_irq_save(flags); + local_irq_save(flags); /* disable preempt till we know what lock. */ rdp = RCU_DATA_ME(); spin_lock(&rdp->lock); *rdp->nexttail = list; if (list) rdp->nexttail = tail; + *rdp->nextschedtail = schedlist; + if (schedlist) + rdp->nextschedtail = schedtail; spin_unlock_irqrestore(&rdp->lock, flags); } void __devinit rcu_online_cpu(int cpu) { unsigned long flags; + struct rcu_data *rdp; spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); cpu_set(cpu, rcu_cpu_online_map); spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); + + /* + * The rcu_sched grace-period processing might have bypassed + * this CPU, given that it was not in the rcu_cpu_online_map + * when the grace-period scan started. This means that the + * grace-period task might sleep. So make sure that if this + * should happen, the first callback posted to this CPU will + * wake up the grace-period task if need be. + */ + + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + rdp->rcu_sched_sleeping = 1; + spin_unlock_irqrestore(&rdp->lock, flags); } #else /* #ifdef CONFIG_HOTPLUG_CPU */ @@ -988,31 +1136,215 @@ void call_rcu(struct rcu_head *head, voi *rdp->nexttail = head; rdp->nexttail = &head->next; RCU_TRACE_RDP(rcupreempt_trace_next_add, rdp); - spin_unlock(&rdp->lock); - local_irq_restore(flags); + spin_unlock_irqrestore(&rdp->lock, flags); } EXPORT_SYMBOL_GPL(call_rcu); +void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) +{ + unsigned long flags; + struct rcu_data *rdp; + int wake_gp = 0; + + head->func = func; + head->next = NULL; + local_irq_save(flags); + rdp = RCU_DATA_ME(); + spin_lock(&rdp->lock); + *rdp->nextschedtail = head; + rdp->nextschedtail = &head->next; + if (rdp->rcu_sched_sleeping) { + + /* Grace-period processing might be sleeping... */ + + rdp->rcu_sched_sleeping = 0; + wake_gp = 1; + } + spin_unlock_irqrestore(&rdp->lock, flags); + if (wake_gp) { + + /* Wake up grace-period processing, unless someone beat us. */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleeping) + wake_gp = 0; + rcu_ctrlblk.sched_sleep = rcu_sched_not_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + if (wake_gp) + wake_up_interruptible(&rcu_ctrlblk.sched_wq); + } +} +EXPORT_SYMBOL_GPL(call_rcu_sched); + /* * Wait until all currently running preempt_disable() code segments * (including hardware-irq-disable segments) complete. Note that * in -rt this does -not- necessarily result in all currently executing * interrupt -handlers- having completed. */ -void __synchronize_sched(void) +synchronize_rcu_xxx(__synchronize_sched, call_rcu_sched) +EXPORT_SYMBOL_GPL(__synchronize_sched); + +/* + * kthread function that manages call_rcu_sched grace periods. + */ +static int rcu_sched_grace_period(void *arg) { - cpumask_t oldmask; + int couldsleep; /* might sleep after current pass. */ + int couldsleepnext = 0; /* might sleep after next pass. */ int cpu; + unsigned long flags; + cpumask_t cpu_snap; + struct rcu_data *rdp; + int ret; - if (sched_getaffinity(0, &oldmask) < 0) - oldmask = cpu_possible_map; - for_each_online_cpu(cpu) { - sched_setaffinity(0, cpumask_of_cpu(cpu)); - schedule(); - } - sched_setaffinity(0, oldmask); + /* + * Each pass through the following loop handles one + * rcu_sched grace period cycle. + */ + do { + /* Save each CPU's current state. */ + + spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); + cpu_snap = cpu_online_map; + spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); + for_each_cpu_mask(cpu, cpu_snap) { + if (cpu_is_offline(cpu)) { + cpu_clear(cpu, cpu_snap); + continue; + } + dyntick_save_progress_counter_sched(cpu); + save_qsctr_sched(cpu); + if (cpu_is_offline(cpu)) + cpu_clear(cpu, cpu_snap); + } + + /* + * Sleep for about an RCU grace-period's worth to + * allow better batching and to consume less CPU. + */ + schedule_timeout_interruptible(RCU_SCHED_BATCH_TIME); + + /* + * If there was nothing to do last time, prepare to + * sleep at the end of the current grace period cycle. + */ + couldsleep = couldsleepnext; + couldsleepnext = 1; + if (couldsleep) { + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + rcu_ctrlblk.sched_sleep = rcu_sched_sleep_prep; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + } + + /* + * Wait on each CPU in turn to have either visited + * a quiescent state or been in dynticks-idle mode. + */ + for_each_cpu_mask(cpu, cpu_snap) { + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, cpu); + while (rcu_qsctr_inc_needed(cpu) && + rcu_qsctr_inc_needed_dyntick(cpu)) { + if (cpu_is_offline(cpu)) { + cpu_clear(cpu, cpu_snap); + break; + } + /* resched_cpu(cpu); @@@ */ + schedule_timeout_interruptible(1); + } + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, -1); + } + + /* Advance callbacks for each CPU. */ + + for_each_cpu_mask(cpu, cpu_snap) { + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + if (cpu_is_offline(cpu)) { + spin_unlock_irqrestore(&rdp->lock, flags); + continue; + } + + /* + * We are running on this CPU irq-disabled, so no + * CPU can go offline until we re-enable irqs. + * The current CPU might have already gone + * offline (between the for_each_offline_cpu and + * the spin_lock_irqsave), but in that case all its + * callback lists will be empty, so no harm done. + * + * Advance the callbacks! We share normal RCU's + * donelist, since callbacks are invoked the + * same way in either case. + */ + if (rdp->waitschedlist != NULL) { + *rdp->donetail = rdp->waitschedlist; + rdp->donetail = rdp->waitschedtail; + + /* + * Next rcu_check_callbacks() will + * do the required raise_softirq(). + */ + } + if (rdp->nextschedlist != NULL) { + rdp->waitschedlist = rdp->nextschedlist; + rdp->waitschedtail = rdp->nextschedtail; + couldsleep = 0; + couldsleepnext = 0; + } else { + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + } + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + + /* Mark sleep intention. */ + + rdp->rcu_sched_sleeping = couldsleep; + + spin_unlock_irqrestore(&rdp->lock, flags); + } + + /* If we saw callbacks on the last scan, go deal with them. */ + + if (!couldsleep) + continue; + + /* Attempt to block... */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleep_prep) { + + /* + * Someone posted a callback after we scanned. + * Go take care of it. + */ + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + couldsleepnext = 0; + continue; + } + + /* Block until the next person posts a callback. */ + + rcu_ctrlblk.sched_sleep = rcu_sched_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + ret = 0; + __wait_event_interruptible(rcu_ctrlblk.sched_wq, + rcu_ctrlblk.sched_sleep != rcu_sched_sleeping, + ret); + + /* + * Signals would prevent us from sleeping, and we cannot + * do much with them in any case. So flush them. + */ + if (ret) + flush_signals(current); + couldsleepnext = 0; + + } while (!kthread_should_stop()); + + return (0); } -EXPORT_SYMBOL_GPL(__synchronize_sched); /* * Check to see if any future RCU-related work will need to be done @@ -1029,7 +1361,9 @@ int rcu_needs_cpu(int cpu) return (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL); + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL); } int rcu_pending(int cpu) @@ -1040,7 +1374,9 @@ int rcu_pending(int cpu) if (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL) + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL) return 1; /* The RCU core needs an acknowledgement from this CPU. */ @@ -1107,6 +1443,11 @@ void __init __rcu_init(void) rdp->donetail = &rdp->donelist; rdp->rcu_flipctr[0] = 0; rdp->rcu_flipctr[1] = 0; + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + rdp->rcu_sched_sleeping = 0; } register_cpu_notifier(&rcu_nb); @@ -1129,11 +1470,15 @@ void __init __rcu_init(void) } /* - * Deprecated, use synchronize_rcu() or synchronize_sched() instead. + * Late-boot-time RCU initialization that must wait until after scheduler + * has been initialized. */ -void synchronize_kernel(void) +void __init rcu_init_sched(void) { - synchronize_rcu(); + rcu_sched_grace_period_task = kthread_run(rcu_sched_grace_period, + NULL, + "rcu_sched_grace_period"); + WARN_ON(IS_ERR(rcu_sched_grace_period_task)); } #ifdef CONFIG_RCU_TRACE diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt_trace.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c --- linux-2.6.25/kernel/rcupreempt_trace.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c 2008-04-21 15:45:07.000000000 -0700 @@ -134,6 +134,10 @@ void rcupreempt_trace_next_add(struct rc trace->next_add++; trace->next_length++; } +void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu) +{ + trace->rcu_sched_cpu = cpu; +} static void rcupreempt_trace_sum(struct rcupreempt_trace *sp) { @@ -262,6 +266,38 @@ static ssize_t rcuctrs_read(struct file return bcount; } +static ssize_t rcuschedstat_read(struct file *filp, char __user *buffer, + size_t count, loff_t *ppos) +{ + int cnt = 0; + int cpu = rcupreempt_trace_cpu(0)->rcu_sched_cpu; + ssize_t bcount; + + mutex_lock(&rcupreempt_trace_mutex); + + cnt += snprintf(&rcupreempt_trace_buf[cnt], RCUPREEMPT_TRACE_BUF_SIZE, + "CPU: %d ", cpu); + if (cpu != -1) { + struct rcu_dyntick_sched *rdsp; + + rdsp = &per_cpu(rcu_dyntick_sched, cpu); + cnt += snprintf(&rcupreempt_trace_buf[cnt], + RCUPREEMPT_TRACE_BUF_SIZE - cnt, + "dynticks: %d %d qs %d %d", + rdsp->dynticks, + rdsp->sched_dynticks_snap, + rdsp->sched_qs, + rdsp->sched_qs_snap); + } + cnt += snprintf(&rcupreempt_trace_buf[cnt], + RCUPREEMPT_TRACE_BUF_SIZE - cnt, + "\n"); + bcount = simple_read_from_buffer(buffer, count, ppos, + rcupreempt_trace_buf, strlen(rcupreempt_trace_buf)); + mutex_unlock(&rcupreempt_trace_mutex); + return bcount; +} + static struct file_operations rcustats_fops = { .owner = THIS_MODULE, .read = rcustats_read, @@ -277,7 +313,12 @@ static struct file_operations rcuctrs_fo .read = rcuctrs_read, }; -static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir; +static struct file_operations rcuschedstat_fops = { + .owner = THIS_MODULE, + .read = rcuschedstat_read, +}; + +static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir, *schedstatdir; static int rcupreempt_debugfs_init(void) { rcudir = debugfs_create_dir("rcu", NULL); @@ -296,12 +337,19 @@ static int rcupreempt_debugfs_init(void) NULL, &rcuctrs_fops); if (!ctrsdir) goto free_out; + + schedstatdir = debugfs_create_file("schedstat", 0444, rcudir, + NULL, &rcuschedstat_fops); + if (!schedstatdir) + goto free_out; return 0; free_out: if (statdir) debugfs_remove(statdir); if (gpdir) debugfs_remove(gpdir); + if (ctrsdir) + debugfs_remove(ctrsdir); debugfs_remove(rcudir); out: return 1; @@ -309,11 +357,16 @@ out: static int __init rcupreempt_trace_init(void) { + int ret; + mutex_init(&rcupreempt_trace_mutex); rcupreempt_trace_buf = kmalloc(RCUPREEMPT_TRACE_BUF_SIZE, GFP_KERNEL); if (!rcupreempt_trace_buf) return 1; - return rcupreempt_debugfs_init(); + ret = rcupreempt_debugfs_init(); + if (ret) + kfree(rcupreempt_trace_buf); + return ret; } static void __exit rcupreempt_trace_cleanup(void) @@ -322,6 +375,7 @@ static void __exit rcupreempt_trace_clea debugfs_remove(gpdir); debugfs_remove(ctrsdir); debugfs_remove(rcudir); + debugfs_remove(schedstatdir); kfree(rcupreempt_trace_buf); } ^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/5] Add call_rcu_sched() 2008-04-22 0:45 ` Paul E. McKenney @ 2008-04-22 0:50 ` Paul E. McKenney 2008-05-05 9:14 ` Gautham R Shenoy 0 siblings, 1 reply; 12+ messages in thread From: Paul E. McKenney @ 2008-04-22 0:50 UTC (permalink / raw) To: linux-kernel Cc: mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, ego, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang [Resend, this time with correct subject: line.] Fifth cut of patch to provide the call_rcu_sched(). This is again to synchronize_sched() as call_rcu() is to synchronize_rcu(). OK, Andrew, it finally appears safe for you to come out of hiding! ;-) (Although I still don't totally trust this, I do trust it more than I do what is currently in mainline.) Passes multi-day rcutorture sessions with concurrent CPU hotplugging. Fixes since the first version include a bug that could result in indefinite blocking (spotted by Gautham Shenoy), better resiliency against CPU-hotplug operations, and other minor fixes. Fixes since the second version include reworking grace-period detection to avoid deadlocks that could happen when running concurrently with CPU hotplug, adding Mathieu's fix to avoid the softlockup messages, as well as Mathieu's fix to allow use earlier in boot. Fixes since the third version include a wrong-CPU bug spotted by Andrew, getting rid of the obsolete synchronize_kernel API that somehow snuck back in, merging spin_unlock() and local_irq_restore() in a few places, commenting the code that checks for quiescent states based on interrupting from user-mode execution or the idle loop, removing some inline attributes, and some code-style changes. Fixes since the fourth version include snapshotting the online CPUs inside of rcu_sched_grace_period() to avoid races with CPUs onlining and offlining, adding rudimentary tracing for rcu_sched, and pulling in a RCU-trace fix from -rt (rcu-trace-fix-free.patch). Known/suspected shortcomings: o Torturing rcu_sched while onlining and offlining CPUs with CONFIG_NO_HZ and CONFIG_PREEMPT_RCU can provoke a race where the event kthread spins on a given CPU attempting to run workqueues, and blocking rcu_sched grace periods, in turn blocking attempts to hotplug CPUs. But this can also happen given the code currently in mainline. o It might be good for rcu_sched_grace_period() to invoke resched_cpu() when a given CPU wasn't responding quickly, but resched_cpu() is declared static... But this is a minor optimization, so dispensing with this unless and until it proves to be a problem. This patch also fixes a long-standing bug in the earlier preemptable-RCU implementation of synchronize_rcu() that could result in loss of concurrent external changes to a task's CPU affinity mask. I still cannot remember who reported this... Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> --- include/linux/rcuclassic.h | 3 include/linux/rcupdate.h | 22 + include/linux/rcupreempt.h | 42 +++ include/linux/rcupreempt_trace.h | 6 init/main.c | 1 kernel/rcupdate.c | 20 - kernel/rcupreempt.c | 433 +++++++++++++++++++++++++++++++++++---- kernel/rcupreempt_trace.c | 58 +++++ 8 files changed, 514 insertions(+), 71 deletions(-) diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcuclassic.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h --- linux-2.6.25/include/linux/rcuclassic.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h 2008-04-17 08:10:33.000000000 -0700 @@ -153,7 +153,10 @@ extern struct lockdep_map rcu_lock_map; #define __synchronize_sched() synchronize_rcu() +#define call_rcu_sched(head, func) call_rcu(head, func) + extern void __rcu_init(void); +#define rcu_init_sched() do { } while (0) extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupdate.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h --- linux-2.6.25/include/linux/rcupdate.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h 2008-04-17 08:10:33.000000000 -0700 @@ -42,6 +42,7 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> #include <linux/lockdep.h> +#include <linux/completion.h> /** * struct rcu_head - callback structure for use with RCU @@ -182,6 +183,27 @@ struct rcu_head { (p) = (v); \ }) +/* Infrastructure to implement the synchronize_() primitives. */ + +struct rcu_synchronize { + struct rcu_head head; + struct completion completion; +}; + +extern void wakeme_after_rcu(struct rcu_head *head); + +#define synchronize_rcu_xxx(name, func) \ +void name(void) \ +{ \ + struct rcu_synchronize rcu; \ + \ + init_completion(&rcu.completion); \ + /* Will wake me after RCU finished. */ \ + func(&rcu.head, wakeme_after_rcu); \ + /* Wait for it. */ \ + wait_for_completion(&rcu.completion); \ +} + /** * synchronize_sched - block until all CPUs have exited any non-preemptive * kernel code sequences. diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h --- linux-2.6.25/include/linux/rcupreempt.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h 2008-04-19 22:44:18.000000000 -0700 @@ -42,10 +42,39 @@ #include <linux/cpumask.h> #include <linux/seqlock.h> -#define rcu_qsctr_inc(cpu) +struct rcu_dyntick_sched { + int dynticks; + int dynticks_snap; + int sched_qs; + int sched_qs_snap; + int sched_dynticks_snap; +}; + +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); + +static inline void rcu_qsctr_inc(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs++; +} #define rcu_bh_qsctr_inc(cpu) #define call_rcu_bh(head, rcu) call_rcu(head, rcu) +/** + * call_rcu_sched - Queue RCU callback for invocation after sched grace period. + * @head: structure to be used for queueing the RCU updates. + * @func: actual update function to be invoked after the grace period + * + * The update function will be invoked some time after a full + * synchronize_sched()-style grace period elapses, in other words after + * all currently executing preempt-disabled sections of code (including + * hardirq handlers, NMI handlers, and local_irq_save() blocks) have + * completed. + */ +extern void call_rcu_sched(struct rcu_head *head, + void (*func)(struct rcu_head *head)); + extern void __rcu_read_lock(void) __acquires(RCU); extern void __rcu_read_unlock(void) __releases(RCU); extern int rcu_pending(int cpu); @@ -57,6 +86,7 @@ extern int rcu_needs_cpu(int cpu); extern void __synchronize_sched(void); extern void __rcu_init(void); +extern void rcu_init_sched(void); extern void rcu_check_callbacks(int cpu, int user); extern void rcu_restart_cpu(int cpu); extern long rcu_batches_completed(void); @@ -83,20 +113,20 @@ extern struct rcupreempt_trace *rcupreem struct softirq_action; #ifdef CONFIG_NO_HZ -DECLARE_PER_CPU(long, dynticks_progress_counter); +DECLARE_PER_CPU(struct rcu_dyntick_sched, dynticks_progress_counter); static inline void rcu_enter_nohz(void) { smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ - __get_cpu_var(dynticks_progress_counter)++; - WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1); + __get_cpu_var(rcu_dyntick_sched).dynticks++; + WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); } static inline void rcu_exit_nohz(void) { - __get_cpu_var(dynticks_progress_counter)++; + __get_cpu_var(rcu_dyntick_sched).dynticks++; smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ - WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1)); + WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); } #else /* CONFIG_NO_HZ */ diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt_trace.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h --- linux-2.6.25/include/linux/rcupreempt_trace.h 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h 2008-04-20 07:44:41.000000000 -0700 @@ -66,12 +66,15 @@ struct rcupreempt_trace { long rcu_try_flip_m1; long rcu_try_flip_me1; long rcu_try_flip_m2; + int rcu_sched_cpu; }; #ifdef CONFIG_RCU_TRACE -#define RCU_TRACE(fn, arg) fn(arg); +#define RCU_TRACE(fn, arg) fn(arg) +#define RCU_TRACE_1ARG(fn, arg, arg1) fn(arg, arg1) #else #define RCU_TRACE(fn, arg) +#define RCU_TRACE_1ARG(fn, arg, arg1) #endif extern void rcupreempt_trace_move2done(struct rcupreempt_trace *trace); @@ -94,6 +97,7 @@ extern void rcupreempt_trace_check_callb extern void rcupreempt_trace_done_remove(struct rcupreempt_trace *trace); extern void rcupreempt_trace_invoke(struct rcupreempt_trace *trace); extern void rcupreempt_trace_next_add(struct rcupreempt_trace *trace); +extern void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu); #endif /* __KERNEL__ */ #endif /* __LINUX_RCUPREEMPT_TRACE_H */ diff -urpNa -X dontdiff linux-2.6.25/init/main.c linux-2.6.25-C1-call_rcu_sched/init/main.c --- linux-2.6.25/init/main.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/init/main.c 2008-04-17 08:10:33.000000000 -0700 @@ -730,6 +730,7 @@ static void __init do_initcalls(void) */ static void __init do_basic_setup(void) { + rcu_init_sched(); /* needed by module_init stage. */ /* drivers will send hotplug events */ init_workqueues(); usermodehelper_init(); diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupdate.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c --- linux-2.6.25/kernel/rcupdate.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c 2008-04-17 08:10:33.000000000 -0700 @@ -39,18 +39,12 @@ #include <linux/sched.h> #include <asm/atomic.h> #include <linux/bitops.h> -#include <linux/completion.h> #include <linux/percpu.h> #include <linux/notifier.h> #include <linux/cpu.h> #include <linux/mutex.h> #include <linux/module.h> -struct rcu_synchronize { - struct rcu_head head; - struct completion completion; -}; - static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; static atomic_t rcu_barrier_cpu_count; static DEFINE_MUTEX(rcu_barrier_mutex); @@ -60,7 +54,7 @@ static struct completion rcu_barrier_com * Awaken the corresponding synchronize_rcu() instance now that a * grace period has elapsed. */ -static void wakeme_after_rcu(struct rcu_head *head) +void wakeme_after_rcu(struct rcu_head *head) { struct rcu_synchronize *rcu; @@ -77,17 +71,7 @@ static void wakeme_after_rcu(struct rcu_ * sections are delimited by rcu_read_lock() and rcu_read_unlock(), * and may be nested. */ -void synchronize_rcu(void) -{ - struct rcu_synchronize rcu; - - init_completion(&rcu.completion); - /* Will wake me after RCU finished */ - call_rcu(&rcu.head, wakeme_after_rcu); - - /* Wait for it */ - wait_for_completion(&rcu.completion); -} +synchronize_rcu_xxx(synchronize_rcu, call_rcu) EXPORT_SYMBOL_GPL(synchronize_rcu); static void rcu_barrier_callback(struct rcu_head *notused) diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c --- linux-2.6.25/kernel/rcupreempt.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c 2008-04-20 07:58:31.000000000 -0700 @@ -46,6 +46,7 @@ #include <asm/atomic.h> #include <linux/bitops.h> #include <linux/module.h> +#include <linux/kthread.h> #include <linux/completion.h> #include <linux/moduleparam.h> #include <linux/percpu.h> @@ -87,9 +88,14 @@ struct rcu_data { struct rcu_head **nexttail; struct rcu_head *waitlist[GP_STAGES]; struct rcu_head **waittail[GP_STAGES]; - struct rcu_head *donelist; + struct rcu_head *donelist; /* from waitlist & waitschedlist */ struct rcu_head **donetail; long rcu_flipctr[2]; + struct rcu_head *nextschedlist; + struct rcu_head **nextschedtail; + struct rcu_head *waitschedlist; + struct rcu_head **waitschedtail; + int rcu_sched_sleeping; #ifdef CONFIG_RCU_TRACE struct rcupreempt_trace trace; #endif /* #ifdef CONFIG_RCU_TRACE */ @@ -131,11 +137,24 @@ enum rcu_try_flip_states { rcu_try_flip_waitmb_state, }; +/* + * States for rcu_ctrlblk.rcu_sched_sleep. + */ + +enum rcu_sched_sleep_states { + rcu_sched_not_sleeping, /* Not sleeping, callbacks need GP. */ + rcu_sched_sleep_prep, /* Thinking of sleeping, rechecking. */ + rcu_sched_sleeping, /* Sleeping, awaken if GP needed. */ +}; + struct rcu_ctrlblk { spinlock_t fliplock; /* Protect state-machine transitions. */ long completed; /* Number of last completed batch. */ enum rcu_try_flip_states rcu_try_flip_state; /* The current state of the rcu state machine */ + spinlock_t schedlock; /* Protect rcu_sched sleep state. */ + enum rcu_sched_sleep_states sched_sleep; /* rcu_sched state. */ + wait_queue_head_t sched_wq; /* Place for rcu_sched to sleep. */ }; static DEFINE_PER_CPU(struct rcu_data, rcu_data); @@ -143,8 +162,12 @@ static struct rcu_ctrlblk rcu_ctrlblk = .fliplock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.fliplock), .completed = 0, .rcu_try_flip_state = rcu_try_flip_idle_state, + .schedlock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.schedlock), + .sched_sleep = rcu_sched_not_sleeping, + .sched_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rcu_ctrlblk.sched_wq), }; +static struct task_struct *rcu_sched_grace_period_task; #ifdef CONFIG_RCU_TRACE static char *rcu_try_flip_state_names[] = @@ -194,6 +217,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu * cached in a local variable, but where the CPU number is so cached. */ #define RCU_TRACE_CPU(f, cpu) RCU_TRACE(f, &(RCU_DATA_CPU(cpu)->trace)); +#define RCU_TRACE_1ARG_CPU(f, cpu, a) RCU_TRACE_1ARG(f, &(RCU_DATA_CPU(cpu)->trace), a); /* * Helper macro for tracing when the appropriate rcu_data is not @@ -207,6 +231,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu */ #define RCU_TRACE_RDP(f, rdp) RCU_TRACE(f, &((rdp)->trace)); +#define RCU_SCHED_BATCH_TIME (HZ / 50) + /* * Return the number of RCU batches processed thus far. Useful * for debug and statistics. @@ -413,32 +439,34 @@ static void __rcu_advance_callbacks(stru } } -#ifdef CONFIG_NO_HZ +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_dyntick_sched, rcu_dyntick_sched) = { + .dynticks = 1, +}; -DEFINE_PER_CPU(long, dynticks_progress_counter) = 1; -static DEFINE_PER_CPU(long, rcu_dyntick_snapshot); +#ifdef CONFIG_NO_HZ static DEFINE_PER_CPU(int, rcu_update_flag); /** * rcu_irq_enter - Called from Hard irq handlers and NMI/SMI. * * If the CPU was idle with dynamic ticks active, this updates the - * dynticks_progress_counter to let the RCU handling know that the + * rcu_dyntick_sched.dynticks to let the RCU handling know that the * CPU is active. */ void rcu_irq_enter(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); if (per_cpu(rcu_update_flag, cpu)) per_cpu(rcu_update_flag, cpu)++; /* * Only update if we are coming from a stopped ticks mode - * (dynticks_progress_counter is even). + * (rcu_dyntick_sched.dynticks is even). */ if (!in_interrupt() && - (per_cpu(dynticks_progress_counter, cpu) & 0x1) == 0) { + (rdssp->dynticks & 0x1) == 0) { /* * The following might seem like we could have a race * with NMI/SMIs. But this really isn't a problem. @@ -461,12 +489,12 @@ void rcu_irq_enter(void) * RCU read-side critical sections on this CPU would * have already completed. */ - per_cpu(dynticks_progress_counter, cpu)++; + rdssp->dynticks++; /* * The following memory barrier ensures that any * rcu_read_lock() primitives in the irq handler * are seen by other CPUs to follow the above - * increment to dynticks_progress_counter. This is + * increment to rcu_dyntick_sched.dynticks. This is * required in order for other CPUs to correctly * determine when it is safe to advance the RCU * grace-period state machine. @@ -474,7 +502,7 @@ void rcu_irq_enter(void) smp_mb(); /* see above block comment. */ /* * Since we can't determine the dynamic tick mode from - * the dynticks_progress_counter after this routine, + * the rcu_dyntick_sched.dynticks after this routine, * we use a second flag to acknowledge that we came * from an idle state with ticks stopped. */ @@ -482,7 +510,7 @@ void rcu_irq_enter(void) /* * If we take an NMI/SMI now, they will also increment * the rcu_update_flag, and will not update the - * dynticks_progress_counter on exit. That is for + * rcu_dyntick_sched.dynticks on exit. That is for * this IRQ to do. */ } @@ -492,12 +520,13 @@ void rcu_irq_enter(void) * rcu_irq_exit - Called from exiting Hard irq context. * * If the CPU was idle with dynamic ticks active, update the - * dynticks_progress_counter to put let the RCU handling be + * rcu_dyntick_sched.dynticks to put let the RCU handling be * aware that the CPU is going back to idle with no ticks. */ void rcu_irq_exit(void) { int cpu = smp_processor_id(); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); /* * rcu_update_flag is set if we interrupted the CPU @@ -505,7 +534,7 @@ void rcu_irq_exit(void) * Once this occurs, we keep track of interrupt nesting * because a NMI/SMI could also come in, and we still * only want the IRQ that started the increment of the - * dynticks_progress_counter to be the one that modifies + * rcu_dyntick_sched.dynticks to be the one that modifies * it on exit. */ if (per_cpu(rcu_update_flag, cpu)) { @@ -517,28 +546,29 @@ void rcu_irq_exit(void) /* * If an NMI/SMI happens now we are still - * protected by the dynticks_progress_counter being odd. + * protected by the rcu_dyntick_sched.dynticks being odd. */ /* * The following memory barrier ensures that any * rcu_read_unlock() primitives in the irq handler * are seen by other CPUs to preceed the following - * increment to dynticks_progress_counter. This + * increment to rcu_dyntick_sched.dynticks. This * is required in order for other CPUs to determine * when it is safe to advance the RCU grace-period * state machine. */ smp_mb(); /* see above block comment. */ - per_cpu(dynticks_progress_counter, cpu)++; - WARN_ON(per_cpu(dynticks_progress_counter, cpu) & 0x1); + rdssp->dynticks++; + WARN_ON(rdssp->dynticks & 0x1); } } static void dyntick_save_progress_counter(int cpu) { - per_cpu(rcu_dyntick_snapshot, cpu) = - per_cpu(dynticks_progress_counter, cpu); + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->dynticks_snap = rdssp->dynticks; } static inline int @@ -546,9 +576,10 @@ rcu_try_flip_waitack_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -582,9 +613,10 @@ rcu_try_flip_waitmb_needed(int cpu) { long curr; long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); - curr = per_cpu(dynticks_progress_counter, cpu); - snap = per_cpu(rcu_dyntick_snapshot, cpu); + curr = rdssp->dynticks; + snap = rdssp->dynticks_snap; smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ /* @@ -611,14 +643,85 @@ rcu_try_flip_waitmb_needed(int cpu) return 1; } +static void dyntick_save_progress_counter_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_dynticks_snap = rdssp->dynticks; +} + +static int rcu_qsctr_inc_needed_dyntick(int cpu) +{ + long curr; + long snap; + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + curr = rdssp->dynticks; + snap = rdssp->sched_dynticks_snap; + smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ + + /* + * If the CPU remained in dynticks mode for the entire time + * and didn't take any interrupts, NMIs, SMIs, or whatever, + * then it cannot be in the middle of an preempt-disabled + * region of code. Therefore, this CPU has been in a quiescent + * state the entire time, and we don't need to wait for it. + */ + + if ((curr == snap) && ((curr & 0x1) == 0)) + return 0; + + /* + * If the CPU passed through or entered a dynticks idle phase with + * no active irq handlers, then, as above, this CPU has already + * passed through a quiescent state. + */ + + if ((curr - snap) > 2 || (snap & 0x1) == 0) + return 0; + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + #else /* !CONFIG_NO_HZ */ -# define dyntick_save_progress_counter(cpu) do { } while (0) -# define rcu_try_flip_waitack_needed(cpu) (1) -# define rcu_try_flip_waitmb_needed(cpu) (1) +# define dyntick_save_progress_counter(cpu) do { } while (0) +# define rcu_try_flip_waitack_needed(cpu) (1) +# define rcu_try_flip_waitmb_needed(cpu) (1) + +# define dyntick_save_progress_counter_sched(cpu) do { } while (0) +# define rcu_qsctr_inc_needed_dyntick(cpu) (1) #endif /* CONFIG_NO_HZ */ +static void save_qsctr_sched(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + rdssp->sched_qs_snap = rdssp->sched_qs; +} + +static inline int rcu_qsctr_inc_needed(int cpu) +{ + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); + + /* + * If there has been a quiescent state, no more need to wait + * on this CPU. + */ + + if (rdssp->sched_qs != rdssp->sched_qs_snap) { + smp_mb(); /* force ordering with cpu entering schedule(). */ + return 0; + } + + /* We need this CPU to go through a quiescent state. */ + + return 1; +} + /* * Get here when RCU is idle. Decide whether we need to * move out of idle state, and return non-zero if so. @@ -821,6 +924,26 @@ void rcu_check_callbacks(int cpu, int us unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + /* + * If this CPU took its interrupt from user mode or from the + * idle loop, and this is not a nested interrupt, then + * this CPU has to have exited all prior preept-disable + * sections of code. So increment the counter to note this. + * + * The memory barrier is needed to handle the case where + * writes from a preempt-disable section of code get reordered + * into schedule() by this CPU's write buffer. So the memory + * barrier makes sure that the rcu_qsctr_inc() is seen by other + * CPUs to happen after any such write. + */ + + if (user || + (idle_cpu(cpu) && !in_softirq() && + hardirq_count() <= (1 << HARDIRQ_SHIFT))) { + smp_mb(); /* Guard against aggressive schedule(). */ + rcu_qsctr_inc(cpu); + } + rcu_check_mb(cpu); if (rcu_ctrlblk.completed == rdp->completed) rcu_try_flip(); @@ -871,6 +994,8 @@ void rcu_offline_cpu(int cpu) struct rcu_head *list = NULL; unsigned long flags; struct rcu_data *rdp = RCU_DATA_CPU(cpu); + struct rcu_head *schedlist = NULL; + struct rcu_head **schedtail = &schedlist; struct rcu_head **tail = &list; /* @@ -884,6 +1009,11 @@ void rcu_offline_cpu(int cpu) rcu_offline_cpu_enqueue(rdp->waitlist[i], rdp->waittail[i], list, tail); rcu_offline_cpu_enqueue(rdp->nextlist, rdp->nexttail, list, tail); + rcu_offline_cpu_enqueue(rdp->waitschedlist, rdp->waitschedtail, + schedlist, schedtail); + rcu_offline_cpu_enqueue(rdp->nextschedlist, rdp->nextschedtail, + schedlist, schedtail); + rdp->rcu_sched_sleeping = 0; spin_unlock_irqrestore(&rdp->lock, flags); rdp->waitlistcount = 0; @@ -918,22 +1048,40 @@ void rcu_offline_cpu(int cpu) * fix. */ - local_irq_save(flags); + local_irq_save(flags); /* disable preempt till we know what lock. */ rdp = RCU_DATA_ME(); spin_lock(&rdp->lock); *rdp->nexttail = list; if (list) rdp->nexttail = tail; + *rdp->nextschedtail = schedlist; + if (schedlist) + rdp->nextschedtail = schedtail; spin_unlock_irqrestore(&rdp->lock, flags); } void __devinit rcu_online_cpu(int cpu) { unsigned long flags; + struct rcu_data *rdp; spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); cpu_set(cpu, rcu_cpu_online_map); spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); + + /* + * The rcu_sched grace-period processing might have bypassed + * this CPU, given that it was not in the rcu_cpu_online_map + * when the grace-period scan started. This means that the + * grace-period task might sleep. So make sure that if this + * should happen, the first callback posted to this CPU will + * wake up the grace-period task if need be. + */ + + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + rdp->rcu_sched_sleeping = 1; + spin_unlock_irqrestore(&rdp->lock, flags); } #else /* #ifdef CONFIG_HOTPLUG_CPU */ @@ -988,31 +1136,215 @@ void call_rcu(struct rcu_head *head, voi *rdp->nexttail = head; rdp->nexttail = &head->next; RCU_TRACE_RDP(rcupreempt_trace_next_add, rdp); - spin_unlock(&rdp->lock); - local_irq_restore(flags); + spin_unlock_irqrestore(&rdp->lock, flags); } EXPORT_SYMBOL_GPL(call_rcu); +void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) +{ + unsigned long flags; + struct rcu_data *rdp; + int wake_gp = 0; + + head->func = func; + head->next = NULL; + local_irq_save(flags); + rdp = RCU_DATA_ME(); + spin_lock(&rdp->lock); + *rdp->nextschedtail = head; + rdp->nextschedtail = &head->next; + if (rdp->rcu_sched_sleeping) { + + /* Grace-period processing might be sleeping... */ + + rdp->rcu_sched_sleeping = 0; + wake_gp = 1; + } + spin_unlock_irqrestore(&rdp->lock, flags); + if (wake_gp) { + + /* Wake up grace-period processing, unless someone beat us. */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleeping) + wake_gp = 0; + rcu_ctrlblk.sched_sleep = rcu_sched_not_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + if (wake_gp) + wake_up_interruptible(&rcu_ctrlblk.sched_wq); + } +} +EXPORT_SYMBOL_GPL(call_rcu_sched); + /* * Wait until all currently running preempt_disable() code segments * (including hardware-irq-disable segments) complete. Note that * in -rt this does -not- necessarily result in all currently executing * interrupt -handlers- having completed. */ -void __synchronize_sched(void) +synchronize_rcu_xxx(__synchronize_sched, call_rcu_sched) +EXPORT_SYMBOL_GPL(__synchronize_sched); + +/* + * kthread function that manages call_rcu_sched grace periods. + */ +static int rcu_sched_grace_period(void *arg) { - cpumask_t oldmask; + int couldsleep; /* might sleep after current pass. */ + int couldsleepnext = 0; /* might sleep after next pass. */ int cpu; + unsigned long flags; + cpumask_t cpu_snap; + struct rcu_data *rdp; + int ret; - if (sched_getaffinity(0, &oldmask) < 0) - oldmask = cpu_possible_map; - for_each_online_cpu(cpu) { - sched_setaffinity(0, cpumask_of_cpu(cpu)); - schedule(); - } - sched_setaffinity(0, oldmask); + /* + * Each pass through the following loop handles one + * rcu_sched grace period cycle. + */ + do { + /* Save each CPU's current state. */ + + spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); + cpu_snap = cpu_online_map; + spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); + for_each_cpu_mask(cpu, cpu_snap) { + if (cpu_is_offline(cpu)) { + cpu_clear(cpu, cpu_snap); + continue; + } + dyntick_save_progress_counter_sched(cpu); + save_qsctr_sched(cpu); + if (cpu_is_offline(cpu)) + cpu_clear(cpu, cpu_snap); + } + + /* + * Sleep for about an RCU grace-period's worth to + * allow better batching and to consume less CPU. + */ + schedule_timeout_interruptible(RCU_SCHED_BATCH_TIME); + + /* + * If there was nothing to do last time, prepare to + * sleep at the end of the current grace period cycle. + */ + couldsleep = couldsleepnext; + couldsleepnext = 1; + if (couldsleep) { + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + rcu_ctrlblk.sched_sleep = rcu_sched_sleep_prep; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + } + + /* + * Wait on each CPU in turn to have either visited + * a quiescent state or been in dynticks-idle mode. + */ + for_each_cpu_mask(cpu, cpu_snap) { + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, cpu); + while (rcu_qsctr_inc_needed(cpu) && + rcu_qsctr_inc_needed_dyntick(cpu)) { + if (cpu_is_offline(cpu)) { + cpu_clear(cpu, cpu_snap); + break; + } + /* resched_cpu(cpu); @@@ */ + schedule_timeout_interruptible(1); + } + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, -1); + } + + /* Advance callbacks for each CPU. */ + + for_each_cpu_mask(cpu, cpu_snap) { + rdp = RCU_DATA_CPU(cpu); + spin_lock_irqsave(&rdp->lock, flags); + if (cpu_is_offline(cpu)) { + spin_unlock_irqrestore(&rdp->lock, flags); + continue; + } + + /* + * We are running on this CPU irq-disabled, so no + * CPU can go offline until we re-enable irqs. + * The current CPU might have already gone + * offline (between the for_each_offline_cpu and + * the spin_lock_irqsave), but in that case all its + * callback lists will be empty, so no harm done. + * + * Advance the callbacks! We share normal RCU's + * donelist, since callbacks are invoked the + * same way in either case. + */ + if (rdp->waitschedlist != NULL) { + *rdp->donetail = rdp->waitschedlist; + rdp->donetail = rdp->waitschedtail; + + /* + * Next rcu_check_callbacks() will + * do the required raise_softirq(). + */ + } + if (rdp->nextschedlist != NULL) { + rdp->waitschedlist = rdp->nextschedlist; + rdp->waitschedtail = rdp->nextschedtail; + couldsleep = 0; + couldsleepnext = 0; + } else { + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + } + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + + /* Mark sleep intention. */ + + rdp->rcu_sched_sleeping = couldsleep; + + spin_unlock_irqrestore(&rdp->lock, flags); + } + + /* If we saw callbacks on the last scan, go deal with them. */ + + if (!couldsleep) + continue; + + /* Attempt to block... */ + + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleep_prep) { + + /* + * Someone posted a callback after we scanned. + * Go take care of it. + */ + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + couldsleepnext = 0; + continue; + } + + /* Block until the next person posts a callback. */ + + rcu_ctrlblk.sched_sleep = rcu_sched_sleeping; + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); + ret = 0; + __wait_event_interruptible(rcu_ctrlblk.sched_wq, + rcu_ctrlblk.sched_sleep != rcu_sched_sleeping, + ret); + + /* + * Signals would prevent us from sleeping, and we cannot + * do much with them in any case. So flush them. + */ + if (ret) + flush_signals(current); + couldsleepnext = 0; + + } while (!kthread_should_stop()); + + return (0); } -EXPORT_SYMBOL_GPL(__synchronize_sched); /* * Check to see if any future RCU-related work will need to be done @@ -1029,7 +1361,9 @@ int rcu_needs_cpu(int cpu) return (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL); + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL); } int rcu_pending(int cpu) @@ -1040,7 +1374,9 @@ int rcu_pending(int cpu) if (rdp->donelist != NULL || !!rdp->waitlistcount || - rdp->nextlist != NULL) + rdp->nextlist != NULL || + rdp->nextschedlist != NULL || + rdp->waitschedlist != NULL) return 1; /* The RCU core needs an acknowledgement from this CPU. */ @@ -1107,6 +1443,11 @@ void __init __rcu_init(void) rdp->donetail = &rdp->donelist; rdp->rcu_flipctr[0] = 0; rdp->rcu_flipctr[1] = 0; + rdp->nextschedlist = NULL; + rdp->nextschedtail = &rdp->nextschedlist; + rdp->waitschedlist = NULL; + rdp->waitschedtail = &rdp->waitschedlist; + rdp->rcu_sched_sleeping = 0; } register_cpu_notifier(&rcu_nb); @@ -1129,11 +1470,15 @@ void __init __rcu_init(void) } /* - * Deprecated, use synchronize_rcu() or synchronize_sched() instead. + * Late-boot-time RCU initialization that must wait until after scheduler + * has been initialized. */ -void synchronize_kernel(void) +void __init rcu_init_sched(void) { - synchronize_rcu(); + rcu_sched_grace_period_task = kthread_run(rcu_sched_grace_period, + NULL, + "rcu_sched_grace_period"); + WARN_ON(IS_ERR(rcu_sched_grace_period_task)); } #ifdef CONFIG_RCU_TRACE diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt_trace.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c --- linux-2.6.25/kernel/rcupreempt_trace.c 2008-04-16 19:49:44.000000000 -0700 +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c 2008-04-21 15:45:07.000000000 -0700 @@ -134,6 +134,10 @@ void rcupreempt_trace_next_add(struct rc trace->next_add++; trace->next_length++; } +void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu) +{ + trace->rcu_sched_cpu = cpu; +} static void rcupreempt_trace_sum(struct rcupreempt_trace *sp) { @@ -262,6 +266,38 @@ static ssize_t rcuctrs_read(struct file return bcount; } +static ssize_t rcuschedstat_read(struct file *filp, char __user *buffer, + size_t count, loff_t *ppos) +{ + int cnt = 0; + int cpu = rcupreempt_trace_cpu(0)->rcu_sched_cpu; + ssize_t bcount; + + mutex_lock(&rcupreempt_trace_mutex); + + cnt += snprintf(&rcupreempt_trace_buf[cnt], RCUPREEMPT_TRACE_BUF_SIZE, + "CPU: %d ", cpu); + if (cpu != -1) { + struct rcu_dyntick_sched *rdsp; + + rdsp = &per_cpu(rcu_dyntick_sched, cpu); + cnt += snprintf(&rcupreempt_trace_buf[cnt], + RCUPREEMPT_TRACE_BUF_SIZE - cnt, + "dynticks: %d %d qs %d %d", + rdsp->dynticks, + rdsp->sched_dynticks_snap, + rdsp->sched_qs, + rdsp->sched_qs_snap); + } + cnt += snprintf(&rcupreempt_trace_buf[cnt], + RCUPREEMPT_TRACE_BUF_SIZE - cnt, + "\n"); + bcount = simple_read_from_buffer(buffer, count, ppos, + rcupreempt_trace_buf, strlen(rcupreempt_trace_buf)); + mutex_unlock(&rcupreempt_trace_mutex); + return bcount; +} + static struct file_operations rcustats_fops = { .owner = THIS_MODULE, .read = rcustats_read, @@ -277,7 +313,12 @@ static struct file_operations rcuctrs_fo .read = rcuctrs_read, }; -static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir; +static struct file_operations rcuschedstat_fops = { + .owner = THIS_MODULE, + .read = rcuschedstat_read, +}; + +static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir, *schedstatdir; static int rcupreempt_debugfs_init(void) { rcudir = debugfs_create_dir("rcu", NULL); @@ -296,12 +337,19 @@ static int rcupreempt_debugfs_init(void) NULL, &rcuctrs_fops); if (!ctrsdir) goto free_out; + + schedstatdir = debugfs_create_file("schedstat", 0444, rcudir, + NULL, &rcuschedstat_fops); + if (!schedstatdir) + goto free_out; return 0; free_out: if (statdir) debugfs_remove(statdir); if (gpdir) debugfs_remove(gpdir); + if (ctrsdir) + debugfs_remove(ctrsdir); debugfs_remove(rcudir); out: return 1; @@ -309,11 +357,16 @@ out: static int __init rcupreempt_trace_init(void) { + int ret; + mutex_init(&rcupreempt_trace_mutex); rcupreempt_trace_buf = kmalloc(RCUPREEMPT_TRACE_BUF_SIZE, GFP_KERNEL); if (!rcupreempt_trace_buf) return 1; - return rcupreempt_debugfs_init(); + ret = rcupreempt_debugfs_init(); + if (ret) + kfree(rcupreempt_trace_buf); + return ret; } static void __exit rcupreempt_trace_cleanup(void) @@ -322,6 +375,7 @@ static void __exit rcupreempt_trace_clea debugfs_remove(gpdir); debugfs_remove(ctrsdir); debugfs_remove(rcudir); + debugfs_remove(schedstatdir); kfree(rcupreempt_trace_buf); } ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/5] Add call_rcu_sched() 2008-04-22 0:50 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney @ 2008-05-05 9:14 ` Gautham R Shenoy 2008-05-05 14:43 ` Paul E. McKenney 0 siblings, 1 reply; 12+ messages in thread From: Gautham R Shenoy @ 2008-05-05 9:14 UTC (permalink / raw) To: Paul E. McKenney Cc: linux-kernel, mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang On Mon, Apr 21, 2008 at 05:50:31PM -0700, Paul E. McKenney wrote: > Yet to review the complete series. > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> > --- > > include/linux/rcuclassic.h | 3 > include/linux/rcupdate.h | 22 + > include/linux/rcupreempt.h | 42 +++ > include/linux/rcupreempt_trace.h | 6 > init/main.c | 1 > kernel/rcupdate.c | 20 - > kernel/rcupreempt.c | 433 +++++++++++++++++++++++++++++++++++---- > kernel/rcupreempt_trace.c | 58 +++++ > 8 files changed, 514 insertions(+), 71 deletions(-) > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcuclassic.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h > --- linux-2.6.25/include/linux/rcuclassic.h 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h 2008-04-17 08:10:33.000000000 -0700 > @@ -153,7 +153,10 @@ extern struct lockdep_map rcu_lock_map; > > #define __synchronize_sched() synchronize_rcu() > > +#define call_rcu_sched(head, func) call_rcu(head, func) > + > extern void __rcu_init(void); > +#define rcu_init_sched() do { } while (0) > extern void rcu_check_callbacks(int cpu, int user); > extern void rcu_restart_cpu(int cpu); > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupdate.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h > --- linux-2.6.25/include/linux/rcupdate.h 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h 2008-04-17 08:10:33.000000000 -0700 > @@ -42,6 +42,7 @@ > #include <linux/cpumask.h> > #include <linux/seqlock.h> > #include <linux/lockdep.h> > +#include <linux/completion.h> > > /** > * struct rcu_head - callback structure for use with RCU > @@ -182,6 +183,27 @@ struct rcu_head { > (p) = (v); \ > }) > > +/* Infrastructure to implement the synchronize_() primitives. */ > + > +struct rcu_synchronize { > + struct rcu_head head; > + struct completion completion; > +}; > + > +extern void wakeme_after_rcu(struct rcu_head *head); > + > +#define synchronize_rcu_xxx(name, func) \ > +void name(void) \ > +{ \ > + struct rcu_synchronize rcu; \ > + \ > + init_completion(&rcu.completion); \ > + /* Will wake me after RCU finished. */ \ > + func(&rcu.head, wakeme_after_rcu); \ > + /* Wait for it. */ \ > + wait_for_completion(&rcu.completion); \ > +} > + > /** > * synchronize_sched - block until all CPUs have exited any non-preemptive > * kernel code sequences. > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h > --- linux-2.6.25/include/linux/rcupreempt.h 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h 2008-04-19 22:44:18.000000000 -0700 > @@ -42,10 +42,39 @@ > #include <linux/cpumask.h> > #include <linux/seqlock.h> > > -#define rcu_qsctr_inc(cpu) > +struct rcu_dyntick_sched { > + int dynticks; > + int dynticks_snap; > + int sched_qs; > + int sched_qs_snap; > + int sched_dynticks_snap; > +}; > + > +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); > + > +static inline void rcu_qsctr_inc(int cpu) > +{ > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + rdssp->sched_qs++; > +} > #define rcu_bh_qsctr_inc(cpu) > #define call_rcu_bh(head, rcu) call_rcu(head, rcu) > > +/** > + * call_rcu_sched - Queue RCU callback for invocation after sched grace period. > + * @head: structure to be used for queueing the RCU updates. > + * @func: actual update function to be invoked after the grace period > + * > + * The update function will be invoked some time after a full > + * synchronize_sched()-style grace period elapses, in other words after > + * all currently executing preempt-disabled sections of code (including > + * hardirq handlers, NMI handlers, and local_irq_save() blocks) have > + * completed. > + */ > +extern void call_rcu_sched(struct rcu_head *head, > + void (*func)(struct rcu_head *head)); > + > extern void __rcu_read_lock(void) __acquires(RCU); > extern void __rcu_read_unlock(void) __releases(RCU); > extern int rcu_pending(int cpu); > @@ -57,6 +86,7 @@ extern int rcu_needs_cpu(int cpu); > extern void __synchronize_sched(void); > > extern void __rcu_init(void); > +extern void rcu_init_sched(void); > extern void rcu_check_callbacks(int cpu, int user); > extern void rcu_restart_cpu(int cpu); > extern long rcu_batches_completed(void); > @@ -83,20 +113,20 @@ extern struct rcupreempt_trace *rcupreem > struct softirq_action; > > #ifdef CONFIG_NO_HZ > -DECLARE_PER_CPU(long, dynticks_progress_counter); > +DECLARE_PER_CPU(struct rcu_dyntick_sched, dynticks_progress_counter); ^^^^^^^^^^^^^^^^^^^^^^^^^ Where is this defined? And why do we need it? We already have rcu_dyntick_sched.dynticks for this purpose, no? > > static inline void rcu_enter_nohz(void) > { > smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ > - __get_cpu_var(dynticks_progress_counter)++; > - WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1); > + __get_cpu_var(rcu_dyntick_sched).dynticks++; > + WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); > } > > static inline void rcu_exit_nohz(void) > { > - __get_cpu_var(dynticks_progress_counter)++; > + __get_cpu_var(rcu_dyntick_sched).dynticks++; > smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ > - WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1)); > + WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); > } > > #else /* CONFIG_NO_HZ */ > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt_trace.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h > --- linux-2.6.25/include/linux/rcupreempt_trace.h 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h 2008-04-20 07:44:41.000000000 -0700 > @@ -66,12 +66,15 @@ struct rcupreempt_trace { > long rcu_try_flip_m1; > long rcu_try_flip_me1; > long rcu_try_flip_m2; > + int rcu_sched_cpu; > }; > > #ifdef CONFIG_RCU_TRACE > -#define RCU_TRACE(fn, arg) fn(arg); > +#define RCU_TRACE(fn, arg) fn(arg) > +#define RCU_TRACE_1ARG(fn, arg, arg1) fn(arg, arg1) > #else > #define RCU_TRACE(fn, arg) > +#define RCU_TRACE_1ARG(fn, arg, arg1) > #endif > > extern void rcupreempt_trace_move2done(struct rcupreempt_trace *trace); > @@ -94,6 +97,7 @@ extern void rcupreempt_trace_check_callb > extern void rcupreempt_trace_done_remove(struct rcupreempt_trace *trace); > extern void rcupreempt_trace_invoke(struct rcupreempt_trace *trace); > extern void rcupreempt_trace_next_add(struct rcupreempt_trace *trace); > +extern void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu); > > #endif /* __KERNEL__ */ > #endif /* __LINUX_RCUPREEMPT_TRACE_H */ > diff -urpNa -X dontdiff linux-2.6.25/init/main.c linux-2.6.25-C1-call_rcu_sched/init/main.c > --- linux-2.6.25/init/main.c 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/init/main.c 2008-04-17 08:10:33.000000000 -0700 > @@ -730,6 +730,7 @@ static void __init do_initcalls(void) > */ > static void __init do_basic_setup(void) > { > + rcu_init_sched(); /* needed by module_init stage. */ > /* drivers will send hotplug events */ > init_workqueues(); > usermodehelper_init(); > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupdate.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c > --- linux-2.6.25/kernel/rcupdate.c 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c 2008-04-17 08:10:33.000000000 -0700 > @@ -39,18 +39,12 @@ > #include <linux/sched.h> > #include <asm/atomic.h> > #include <linux/bitops.h> > -#include <linux/completion.h> > #include <linux/percpu.h> > #include <linux/notifier.h> > #include <linux/cpu.h> > #include <linux/mutex.h> > #include <linux/module.h> > > -struct rcu_synchronize { > - struct rcu_head head; > - struct completion completion; > -}; > - > static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; > static atomic_t rcu_barrier_cpu_count; > static DEFINE_MUTEX(rcu_barrier_mutex); > @@ -60,7 +54,7 @@ static struct completion rcu_barrier_com > * Awaken the corresponding synchronize_rcu() instance now that a > * grace period has elapsed. > */ > -static void wakeme_after_rcu(struct rcu_head *head) > +void wakeme_after_rcu(struct rcu_head *head) > { > struct rcu_synchronize *rcu; > > @@ -77,17 +71,7 @@ static void wakeme_after_rcu(struct rcu_ > * sections are delimited by rcu_read_lock() and rcu_read_unlock(), > * and may be nested. > */ > -void synchronize_rcu(void) > -{ > - struct rcu_synchronize rcu; > - > - init_completion(&rcu.completion); > - /* Will wake me after RCU finished */ > - call_rcu(&rcu.head, wakeme_after_rcu); > - > - /* Wait for it */ > - wait_for_completion(&rcu.completion); > -} > +synchronize_rcu_xxx(synchronize_rcu, call_rcu) > EXPORT_SYMBOL_GPL(synchronize_rcu); > > static void rcu_barrier_callback(struct rcu_head *notused) > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c > --- linux-2.6.25/kernel/rcupreempt.c 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c 2008-04-20 07:58:31.000000000 -0700 > @@ -46,6 +46,7 @@ > #include <asm/atomic.h> > #include <linux/bitops.h> > #include <linux/module.h> > +#include <linux/kthread.h> > #include <linux/completion.h> > #include <linux/moduleparam.h> > #include <linux/percpu.h> > @@ -87,9 +88,14 @@ struct rcu_data { > struct rcu_head **nexttail; > struct rcu_head *waitlist[GP_STAGES]; > struct rcu_head **waittail[GP_STAGES]; > - struct rcu_head *donelist; > + struct rcu_head *donelist; /* from waitlist & waitschedlist */ > struct rcu_head **donetail; > long rcu_flipctr[2]; > + struct rcu_head *nextschedlist; > + struct rcu_head **nextschedtail; > + struct rcu_head *waitschedlist; > + struct rcu_head **waitschedtail; > + int rcu_sched_sleeping; > #ifdef CONFIG_RCU_TRACE > struct rcupreempt_trace trace; > #endif /* #ifdef CONFIG_RCU_TRACE */ > @@ -131,11 +137,24 @@ enum rcu_try_flip_states { > rcu_try_flip_waitmb_state, > }; > > +/* > + * States for rcu_ctrlblk.rcu_sched_sleep. ^^^^^^^^^^^^^^^^^^^^^^^^^^^ should be rcu_ctrlblk.sched_sleep > + */ > + > +enum rcu_sched_sleep_states { > + rcu_sched_not_sleeping, /* Not sleeping, callbacks need GP. */ > + rcu_sched_sleep_prep, /* Thinking of sleeping, rechecking. */ > + rcu_sched_sleeping, /* Sleeping, awaken if GP needed. */ > +}; > + > struct rcu_ctrlblk { > spinlock_t fliplock; /* Protect state-machine transitions. */ > long completed; /* Number of last completed batch. */ > enum rcu_try_flip_states rcu_try_flip_state; /* The current state of > the rcu state machine */ > + spinlock_t schedlock; /* Protect rcu_sched sleep state. */ > + enum rcu_sched_sleep_states sched_sleep; /* rcu_sched state. */ > + wait_queue_head_t sched_wq; /* Place for rcu_sched to sleep. */ > }; > > static DEFINE_PER_CPU(struct rcu_data, rcu_data); > @@ -143,8 +162,12 @@ static struct rcu_ctrlblk rcu_ctrlblk = > .fliplock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.fliplock), > .completed = 0, > .rcu_try_flip_state = rcu_try_flip_idle_state, > + .schedlock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.schedlock), > + .sched_sleep = rcu_sched_not_sleeping, > + .sched_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rcu_ctrlblk.sched_wq), > }; > > +static struct task_struct *rcu_sched_grace_period_task; > > #ifdef CONFIG_RCU_TRACE > static char *rcu_try_flip_state_names[] = > @@ -194,6 +217,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu > * cached in a local variable, but where the CPU number is so cached. > */ > #define RCU_TRACE_CPU(f, cpu) RCU_TRACE(f, &(RCU_DATA_CPU(cpu)->trace)); > +#define RCU_TRACE_1ARG_CPU(f, cpu, a) RCU_TRACE_1ARG(f, &(RCU_DATA_CPU(cpu)->trace), a); > > /* > * Helper macro for tracing when the appropriate rcu_data is not > @@ -207,6 +231,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu > */ > #define RCU_TRACE_RDP(f, rdp) RCU_TRACE(f, &((rdp)->trace)); > > +#define RCU_SCHED_BATCH_TIME (HZ / 50) > + > /* > * Return the number of RCU batches processed thus far. Useful > * for debug and statistics. > @@ -413,32 +439,34 @@ static void __rcu_advance_callbacks(stru > } > } > > -#ifdef CONFIG_NO_HZ > +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_dyntick_sched, rcu_dyntick_sched) = { > + .dynticks = 1, > +}; > > -DEFINE_PER_CPU(long, dynticks_progress_counter) = 1; > -static DEFINE_PER_CPU(long, rcu_dyntick_snapshot); > +#ifdef CONFIG_NO_HZ > static DEFINE_PER_CPU(int, rcu_update_flag); > > /** > * rcu_irq_enter - Called from Hard irq handlers and NMI/SMI. > * > * If the CPU was idle with dynamic ticks active, this updates the > - * dynticks_progress_counter to let the RCU handling know that the > + * rcu_dyntick_sched.dynticks to let the RCU handling know that the > * CPU is active. > */ > void rcu_irq_enter(void) > { > int cpu = smp_processor_id(); > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > if (per_cpu(rcu_update_flag, cpu)) > per_cpu(rcu_update_flag, cpu)++; > > /* > * Only update if we are coming from a stopped ticks mode > - * (dynticks_progress_counter is even). > + * (rcu_dyntick_sched.dynticks is even). > */ > if (!in_interrupt() && > - (per_cpu(dynticks_progress_counter, cpu) & 0x1) == 0) { > + (rdssp->dynticks & 0x1) == 0) { > /* > * The following might seem like we could have a race > * with NMI/SMIs. But this really isn't a problem. > @@ -461,12 +489,12 @@ void rcu_irq_enter(void) > * RCU read-side critical sections on this CPU would > * have already completed. > */ > - per_cpu(dynticks_progress_counter, cpu)++; > + rdssp->dynticks++; > /* > * The following memory barrier ensures that any > * rcu_read_lock() primitives in the irq handler > * are seen by other CPUs to follow the above > - * increment to dynticks_progress_counter. This is > + * increment to rcu_dyntick_sched.dynticks. This is > * required in order for other CPUs to correctly > * determine when it is safe to advance the RCU > * grace-period state machine. > @@ -474,7 +502,7 @@ void rcu_irq_enter(void) > smp_mb(); /* see above block comment. */ > /* > * Since we can't determine the dynamic tick mode from > - * the dynticks_progress_counter after this routine, > + * the rcu_dyntick_sched.dynticks after this routine, > * we use a second flag to acknowledge that we came > * from an idle state with ticks stopped. > */ > @@ -482,7 +510,7 @@ void rcu_irq_enter(void) > /* > * If we take an NMI/SMI now, they will also increment > * the rcu_update_flag, and will not update the > - * dynticks_progress_counter on exit. That is for > + * rcu_dyntick_sched.dynticks on exit. That is for > * this IRQ to do. > */ > } > @@ -492,12 +520,13 @@ void rcu_irq_enter(void) > * rcu_irq_exit - Called from exiting Hard irq context. > * > * If the CPU was idle with dynamic ticks active, update the > - * dynticks_progress_counter to put let the RCU handling be > + * rcu_dyntick_sched.dynticks to put let the RCU handling be > * aware that the CPU is going back to idle with no ticks. > */ > void rcu_irq_exit(void) > { > int cpu = smp_processor_id(); > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > /* > * rcu_update_flag is set if we interrupted the CPU > @@ -505,7 +534,7 @@ void rcu_irq_exit(void) > * Once this occurs, we keep track of interrupt nesting > * because a NMI/SMI could also come in, and we still > * only want the IRQ that started the increment of the > - * dynticks_progress_counter to be the one that modifies > + * rcu_dyntick_sched.dynticks to be the one that modifies > * it on exit. > */ > if (per_cpu(rcu_update_flag, cpu)) { > @@ -517,28 +546,29 @@ void rcu_irq_exit(void) > > /* > * If an NMI/SMI happens now we are still > - * protected by the dynticks_progress_counter being odd. > + * protected by the rcu_dyntick_sched.dynticks being odd. > */ > > /* > * The following memory barrier ensures that any > * rcu_read_unlock() primitives in the irq handler > * are seen by other CPUs to preceed the following > - * increment to dynticks_progress_counter. This > + * increment to rcu_dyntick_sched.dynticks. This > * is required in order for other CPUs to determine > * when it is safe to advance the RCU grace-period > * state machine. > */ > smp_mb(); /* see above block comment. */ > - per_cpu(dynticks_progress_counter, cpu)++; > - WARN_ON(per_cpu(dynticks_progress_counter, cpu) & 0x1); > + rdssp->dynticks++; > + WARN_ON(rdssp->dynticks & 0x1); > } > } > > static void dyntick_save_progress_counter(int cpu) > { > - per_cpu(rcu_dyntick_snapshot, cpu) = > - per_cpu(dynticks_progress_counter, cpu); > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + rdssp->dynticks_snap = rdssp->dynticks; > } > > static inline int > @@ -546,9 +576,10 @@ rcu_try_flip_waitack_needed(int cpu) > { > long curr; > long snap; > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > - curr = per_cpu(dynticks_progress_counter, cpu); > - snap = per_cpu(rcu_dyntick_snapshot, cpu); > + curr = rdssp->dynticks; > + snap = rdssp->dynticks_snap; > smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > > /* > @@ -582,9 +613,10 @@ rcu_try_flip_waitmb_needed(int cpu) > { > long curr; > long snap; > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > - curr = per_cpu(dynticks_progress_counter, cpu); > - snap = per_cpu(rcu_dyntick_snapshot, cpu); > + curr = rdssp->dynticks; > + snap = rdssp->dynticks_snap; > smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > > /* > @@ -611,14 +643,85 @@ rcu_try_flip_waitmb_needed(int cpu) > return 1; > } > > +static void dyntick_save_progress_counter_sched(int cpu) > +{ > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + rdssp->sched_dynticks_snap = rdssp->dynticks; > +} > + > +static int rcu_qsctr_inc_needed_dyntick(int cpu) > +{ > + long curr; > + long snap; > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + curr = rdssp->dynticks; > + snap = rdssp->sched_dynticks_snap; > + smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > + > + /* > + * If the CPU remained in dynticks mode for the entire time > + * and didn't take any interrupts, NMIs, SMIs, or whatever, > + * then it cannot be in the middle of an preempt-disabled > + * region of code. Therefore, this CPU has been in a quiescent > + * state the entire time, and we don't need to wait for it. > + */ > + > + if ((curr == snap) && ((curr & 0x1) == 0)) > + return 0; > + > + /* > + * If the CPU passed through or entered a dynticks idle phase with > + * no active irq handlers, then, as above, this CPU has already > + * passed through a quiescent state. > + */ > + > + if ((curr - snap) > 2 || (snap & 0x1) == 0) > + return 0; > + > + /* We need this CPU to go through a quiescent state. */ > + > + return 1; > +} > + > #else /* !CONFIG_NO_HZ */ > > -# define dyntick_save_progress_counter(cpu) do { } while (0) > -# define rcu_try_flip_waitack_needed(cpu) (1) > -# define rcu_try_flip_waitmb_needed(cpu) (1) > +# define dyntick_save_progress_counter(cpu) do { } while (0) > +# define rcu_try_flip_waitack_needed(cpu) (1) > +# define rcu_try_flip_waitmb_needed(cpu) (1) > + > +# define dyntick_save_progress_counter_sched(cpu) do { } while (0) > +# define rcu_qsctr_inc_needed_dyntick(cpu) (1) > > #endif /* CONFIG_NO_HZ */ > > +static void save_qsctr_sched(int cpu) > +{ > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + rdssp->sched_qs_snap = rdssp->sched_qs; > +} > + > +static inline int rcu_qsctr_inc_needed(int cpu) > +{ > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > + > + /* > + * If there has been a quiescent state, no more need to wait > + * on this CPU. > + */ > + > + if (rdssp->sched_qs != rdssp->sched_qs_snap) { > + smp_mb(); /* force ordering with cpu entering schedule(). */ > + return 0; > + } > + > + /* We need this CPU to go through a quiescent state. */ > + > + return 1; > +} > + > /* > * Get here when RCU is idle. Decide whether we need to > * move out of idle state, and return non-zero if so. > @@ -821,6 +924,26 @@ void rcu_check_callbacks(int cpu, int us > unsigned long flags; > struct rcu_data *rdp = RCU_DATA_CPU(cpu); > > + /* > + * If this CPU took its interrupt from user mode or from the > + * idle loop, and this is not a nested interrupt, then > + * this CPU has to have exited all prior preept-disable > + * sections of code. So increment the counter to note this. > + * > + * The memory barrier is needed to handle the case where > + * writes from a preempt-disable section of code get reordered > + * into schedule() by this CPU's write buffer. So the memory > + * barrier makes sure that the rcu_qsctr_inc() is seen by other > + * CPUs to happen after any such write. > + */ > + > + if (user || > + (idle_cpu(cpu) && !in_softirq() && > + hardirq_count() <= (1 << HARDIRQ_SHIFT))) { > + smp_mb(); /* Guard against aggressive schedule(). */ > + rcu_qsctr_inc(cpu); > + } > + > rcu_check_mb(cpu); > if (rcu_ctrlblk.completed == rdp->completed) > rcu_try_flip(); > @@ -871,6 +994,8 @@ void rcu_offline_cpu(int cpu) > struct rcu_head *list = NULL; > unsigned long flags; > struct rcu_data *rdp = RCU_DATA_CPU(cpu); > + struct rcu_head *schedlist = NULL; > + struct rcu_head **schedtail = &schedlist; > struct rcu_head **tail = &list; > > /* > @@ -884,6 +1009,11 @@ void rcu_offline_cpu(int cpu) > rcu_offline_cpu_enqueue(rdp->waitlist[i], rdp->waittail[i], > list, tail); > rcu_offline_cpu_enqueue(rdp->nextlist, rdp->nexttail, list, tail); > + rcu_offline_cpu_enqueue(rdp->waitschedlist, rdp->waitschedtail, > + schedlist, schedtail); > + rcu_offline_cpu_enqueue(rdp->nextschedlist, rdp->nextschedtail, > + schedlist, schedtail); > + rdp->rcu_sched_sleeping = 0; > spin_unlock_irqrestore(&rdp->lock, flags); > rdp->waitlistcount = 0; > > @@ -918,22 +1048,40 @@ void rcu_offline_cpu(int cpu) > * fix. > */ > > - local_irq_save(flags); > + local_irq_save(flags); /* disable preempt till we know what lock. */ > rdp = RCU_DATA_ME(); > spin_lock(&rdp->lock); > *rdp->nexttail = list; > if (list) > rdp->nexttail = tail; > + *rdp->nextschedtail = schedlist; > + if (schedlist) > + rdp->nextschedtail = schedtail; > spin_unlock_irqrestore(&rdp->lock, flags); > } > > void __devinit rcu_online_cpu(int cpu) > { > unsigned long flags; > + struct rcu_data *rdp; > > spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); > cpu_set(cpu, rcu_cpu_online_map); > spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); > + > + /* > + * The rcu_sched grace-period processing might have bypassed > + * this CPU, given that it was not in the rcu_cpu_online_map > + * when the grace-period scan started. This means that the > + * grace-period task might sleep. So make sure that if this > + * should happen, the first callback posted to this CPU will > + * wake up the grace-period task if need be. > + */ > + > + rdp = RCU_DATA_CPU(cpu); > + spin_lock_irqsave(&rdp->lock, flags); > + rdp->rcu_sched_sleeping = 1; > + spin_unlock_irqrestore(&rdp->lock, flags); > } > > #else /* #ifdef CONFIG_HOTPLUG_CPU */ > @@ -988,31 +1136,215 @@ void call_rcu(struct rcu_head *head, voi > *rdp->nexttail = head; > rdp->nexttail = &head->next; > RCU_TRACE_RDP(rcupreempt_trace_next_add, rdp); > - spin_unlock(&rdp->lock); > - local_irq_restore(flags); > + spin_unlock_irqrestore(&rdp->lock, flags); > } > EXPORT_SYMBOL_GPL(call_rcu); > > +void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) > +{ > + unsigned long flags; > + struct rcu_data *rdp; > + int wake_gp = 0; > + > + head->func = func; > + head->next = NULL; > + local_irq_save(flags); > + rdp = RCU_DATA_ME(); > + spin_lock(&rdp->lock); > + *rdp->nextschedtail = head; > + rdp->nextschedtail = &head->next; > + if (rdp->rcu_sched_sleeping) { > + > + /* Grace-period processing might be sleeping... */ > + > + rdp->rcu_sched_sleeping = 0; > + wake_gp = 1; > + } > + spin_unlock_irqrestore(&rdp->lock, flags); > + if (wake_gp) { > + > + /* Wake up grace-period processing, unless someone beat us. */ > + > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleeping) > + wake_gp = 0; > + rcu_ctrlblk.sched_sleep = rcu_sched_not_sleeping; > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > + if (wake_gp) > + wake_up_interruptible(&rcu_ctrlblk.sched_wq); > + } > +} > +EXPORT_SYMBOL_GPL(call_rcu_sched); > + > /* > * Wait until all currently running preempt_disable() code segments > * (including hardware-irq-disable segments) complete. Note that > * in -rt this does -not- necessarily result in all currently executing > * interrupt -handlers- having completed. > */ > -void __synchronize_sched(void) > +synchronize_rcu_xxx(__synchronize_sched, call_rcu_sched) > +EXPORT_SYMBOL_GPL(__synchronize_sched); > + > +/* > + * kthread function that manages call_rcu_sched grace periods. > + */ > +static int rcu_sched_grace_period(void *arg) > { > - cpumask_t oldmask; > + int couldsleep; /* might sleep after current pass. */ > + int couldsleepnext = 0; /* might sleep after next pass. */ > int cpu; > + unsigned long flags; > + cpumask_t cpu_snap; > + struct rcu_data *rdp; > + int ret; > > - if (sched_getaffinity(0, &oldmask) < 0) > - oldmask = cpu_possible_map; > - for_each_online_cpu(cpu) { > - sched_setaffinity(0, cpumask_of_cpu(cpu)); > - schedule(); > - } > - sched_setaffinity(0, oldmask); > + /* > + * Each pass through the following loop handles one > + * rcu_sched grace period cycle. > + */ > + do { > + /* Save each CPU's current state. */ > + > + spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); > + cpu_snap = cpu_online_map; > + spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); > + for_each_cpu_mask(cpu, cpu_snap) { > + if (cpu_is_offline(cpu)) { > + cpu_clear(cpu, cpu_snap); > + continue; > + } > + dyntick_save_progress_counter_sched(cpu); > + save_qsctr_sched(cpu); > + if (cpu_is_offline(cpu)) > + cpu_clear(cpu, cpu_snap); > + } > + > + /* > + * Sleep for about an RCU grace-period's worth to > + * allow better batching and to consume less CPU. > + */ > + schedule_timeout_interruptible(RCU_SCHED_BATCH_TIME); > + > + /* > + * If there was nothing to do last time, prepare to > + * sleep at the end of the current grace period cycle. > + */ > + couldsleep = couldsleepnext; > + couldsleepnext = 1; > + if (couldsleep) { > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > + rcu_ctrlblk.sched_sleep = rcu_sched_sleep_prep; > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > + } > + > + /* > + * Wait on each CPU in turn to have either visited > + * a quiescent state or been in dynticks-idle mode. > + */ > + for_each_cpu_mask(cpu, cpu_snap) { > + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, cpu); > + while (rcu_qsctr_inc_needed(cpu) && > + rcu_qsctr_inc_needed_dyntick(cpu)) { > + if (cpu_is_offline(cpu)) { > + cpu_clear(cpu, cpu_snap); > + break; > + } > + /* resched_cpu(cpu); @@@ */ > + schedule_timeout_interruptible(1); > + } > + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, -1); > + } > + > + /* Advance callbacks for each CPU. */ > + > + for_each_cpu_mask(cpu, cpu_snap) { > + rdp = RCU_DATA_CPU(cpu); > + spin_lock_irqsave(&rdp->lock, flags); > + if (cpu_is_offline(cpu)) { > + spin_unlock_irqrestore(&rdp->lock, flags); > + continue; > + } > + > + /* > + * We are running on this CPU irq-disabled, so no > + * CPU can go offline until we re-enable irqs. > + * The current CPU might have already gone > + * offline (between the for_each_offline_cpu and ^^^^^^^^^^^^^^^^^^^^ for_each_cpu_mask(). Infact, for_each_offline_cpu() doesn't even exist ;-) I guess doing the multiple cpu_is_offline() checks is unavoidable, unless we use get_online_cpus()/put_online_cpus(). But that's an overkill in this context. So, looks okay to me. > + * the spin_lock_irqsave), but in that case all its > + * callback lists will be empty, so no harm done. > + * > + * Advance the callbacks! We share normal RCU's > + * donelist, since callbacks are invoked the > + * same way in either case. > + */ > + if (rdp->waitschedlist != NULL) { > + *rdp->donetail = rdp->waitschedlist; > + rdp->donetail = rdp->waitschedtail; > + > + /* > + * Next rcu_check_callbacks() will > + * do the required raise_softirq(). > + */ > + } > + if (rdp->nextschedlist != NULL) { > + rdp->waitschedlist = rdp->nextschedlist; > + rdp->waitschedtail = rdp->nextschedtail; > + couldsleep = 0; > + couldsleepnext = 0; > + } else { > + rdp->waitschedlist = NULL; > + rdp->waitschedtail = &rdp->waitschedlist; > + } > + rdp->nextschedlist = NULL; > + rdp->nextschedtail = &rdp->nextschedlist; > + > + /* Mark sleep intention. */ > + > + rdp->rcu_sched_sleeping = couldsleep; > + > + spin_unlock_irqrestore(&rdp->lock, flags); > + } > + > + /* If we saw callbacks on the last scan, go deal with them. */ > + > + if (!couldsleep) > + continue; > + > + /* Attempt to block... */ > + > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleep_prep) { > + > + /* > + * Someone posted a callback after we scanned. > + * Go take care of it. > + */ > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > + couldsleepnext = 0; > + continue; > + } > + > + /* Block until the next person posts a callback. */ > + > + rcu_ctrlblk.sched_sleep = rcu_sched_sleeping; > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > + ret = 0; > + __wait_event_interruptible(rcu_ctrlblk.sched_wq, > + rcu_ctrlblk.sched_sleep != rcu_sched_sleeping, > + ret); > + > + /* > + * Signals would prevent us from sleeping, and we cannot > + * do much with them in any case. So flush them. > + */ > + if (ret) > + flush_signals(current); > + couldsleepnext = 0; > + > + } while (!kthread_should_stop()); > + > + return (0); > } > -EXPORT_SYMBOL_GPL(__synchronize_sched); > > /* > * Check to see if any future RCU-related work will need to be done > @@ -1029,7 +1361,9 @@ int rcu_needs_cpu(int cpu) > > return (rdp->donelist != NULL || > !!rdp->waitlistcount || > - rdp->nextlist != NULL); > + rdp->nextlist != NULL || > + rdp->nextschedlist != NULL || > + rdp->waitschedlist != NULL); > } > > int rcu_pending(int cpu) > @@ -1040,7 +1374,9 @@ int rcu_pending(int cpu) > > if (rdp->donelist != NULL || > !!rdp->waitlistcount || > - rdp->nextlist != NULL) > + rdp->nextlist != NULL || > + rdp->nextschedlist != NULL || > + rdp->waitschedlist != NULL) > return 1; > > /* The RCU core needs an acknowledgement from this CPU. */ > @@ -1107,6 +1443,11 @@ void __init __rcu_init(void) > rdp->donetail = &rdp->donelist; > rdp->rcu_flipctr[0] = 0; > rdp->rcu_flipctr[1] = 0; > + rdp->nextschedlist = NULL; > + rdp->nextschedtail = &rdp->nextschedlist; > + rdp->waitschedlist = NULL; > + rdp->waitschedtail = &rdp->waitschedlist; > + rdp->rcu_sched_sleeping = 0; > } > register_cpu_notifier(&rcu_nb); > > @@ -1129,11 +1470,15 @@ void __init __rcu_init(void) > } > > /* > - * Deprecated, use synchronize_rcu() or synchronize_sched() instead. > + * Late-boot-time RCU initialization that must wait until after scheduler > + * has been initialized. > */ > -void synchronize_kernel(void) > +void __init rcu_init_sched(void) > { > - synchronize_rcu(); > + rcu_sched_grace_period_task = kthread_run(rcu_sched_grace_period, > + NULL, > + "rcu_sched_grace_period"); > + WARN_ON(IS_ERR(rcu_sched_grace_period_task)); > } > > #ifdef CONFIG_RCU_TRACE > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt_trace.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c > --- linux-2.6.25/kernel/rcupreempt_trace.c 2008-04-16 19:49:44.000000000 -0700 > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c 2008-04-21 15:45:07.000000000 -0700 > @@ -134,6 +134,10 @@ void rcupreempt_trace_next_add(struct rc > trace->next_add++; > trace->next_length++; > } > +void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu) > +{ > + trace->rcu_sched_cpu = cpu; > +} > > static void rcupreempt_trace_sum(struct rcupreempt_trace *sp) > { > @@ -262,6 +266,38 @@ static ssize_t rcuctrs_read(struct file > return bcount; > } > > +static ssize_t rcuschedstat_read(struct file *filp, char __user *buffer, > + size_t count, loff_t *ppos) > +{ > + int cnt = 0; > + int cpu = rcupreempt_trace_cpu(0)->rcu_sched_cpu; > + ssize_t bcount; > + > + mutex_lock(&rcupreempt_trace_mutex); > + > + cnt += snprintf(&rcupreempt_trace_buf[cnt], RCUPREEMPT_TRACE_BUF_SIZE, > + "CPU: %d ", cpu); > + if (cpu != -1) { > + struct rcu_dyntick_sched *rdsp; > + > + rdsp = &per_cpu(rcu_dyntick_sched, cpu); > + cnt += snprintf(&rcupreempt_trace_buf[cnt], > + RCUPREEMPT_TRACE_BUF_SIZE - cnt, > + "dynticks: %d %d qs %d %d", > + rdsp->dynticks, > + rdsp->sched_dynticks_snap, > + rdsp->sched_qs, > + rdsp->sched_qs_snap); > + } > + cnt += snprintf(&rcupreempt_trace_buf[cnt], > + RCUPREEMPT_TRACE_BUF_SIZE - cnt, > + "\n"); > + bcount = simple_read_from_buffer(buffer, count, ppos, > + rcupreempt_trace_buf, strlen(rcupreempt_trace_buf)); > + mutex_unlock(&rcupreempt_trace_mutex); > + return bcount; > +} > + > static struct file_operations rcustats_fops = { > .owner = THIS_MODULE, > .read = rcustats_read, > @@ -277,7 +313,12 @@ static struct file_operations rcuctrs_fo > .read = rcuctrs_read, > }; > > -static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir; > +static struct file_operations rcuschedstat_fops = { > + .owner = THIS_MODULE, > + .read = rcuschedstat_read, > +}; > + > +static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir, *schedstatdir; > static int rcupreempt_debugfs_init(void) > { > rcudir = debugfs_create_dir("rcu", NULL); > @@ -296,12 +337,19 @@ static int rcupreempt_debugfs_init(void) > NULL, &rcuctrs_fops); > if (!ctrsdir) > goto free_out; > + > + schedstatdir = debugfs_create_file("schedstat", 0444, rcudir, > + NULL, &rcuschedstat_fops); > + if (!schedstatdir) > + goto free_out; > return 0; > free_out: > if (statdir) > debugfs_remove(statdir); > if (gpdir) > debugfs_remove(gpdir); > + if (ctrsdir) > + debugfs_remove(ctrsdir); > debugfs_remove(rcudir); > out: > return 1; > @@ -309,11 +357,16 @@ out: > > static int __init rcupreempt_trace_init(void) > { > + int ret; > + > mutex_init(&rcupreempt_trace_mutex); > rcupreempt_trace_buf = kmalloc(RCUPREEMPT_TRACE_BUF_SIZE, GFP_KERNEL); > if (!rcupreempt_trace_buf) > return 1; > - return rcupreempt_debugfs_init(); > + ret = rcupreempt_debugfs_init(); > + if (ret) > + kfree(rcupreempt_trace_buf); > + return ret; > } > > static void __exit rcupreempt_trace_cleanup(void) > @@ -322,6 +375,7 @@ static void __exit rcupreempt_trace_clea > debugfs_remove(gpdir); > debugfs_remove(ctrsdir); > debugfs_remove(rcudir); > + debugfs_remove(schedstatdir); > kfree(rcupreempt_trace_buf); > } -- Thanks and Regards gautham ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 1/5] Add call_rcu_sched() 2008-05-05 9:14 ` Gautham R Shenoy @ 2008-05-05 14:43 ` Paul E. McKenney 0 siblings, 0 replies; 12+ messages in thread From: Paul E. McKenney @ 2008-05-05 14:43 UTC (permalink / raw) To: Gautham R Shenoy Cc: linux-kernel, mathieu.desnoyers, mingo, akpm, hch, mmlnx, dipankar, dsmith, rostedt, adrian.bunk, a.p.zijlstra, niv, dvhltc, rusty, jkenisto, oleg, jamesclhuang On Mon, May 05, 2008 at 02:44:50PM +0530, Gautham R Shenoy wrote: > On Mon, Apr 21, 2008 at 05:50:31PM -0700, Paul E. McKenney wrote: Thank you for looking this over! > Yet to review the complete series. > > > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> > > Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca> > > --- > > > > include/linux/rcuclassic.h | 3 > > include/linux/rcupdate.h | 22 + > > include/linux/rcupreempt.h | 42 +++ > > include/linux/rcupreempt_trace.h | 6 > > init/main.c | 1 > > kernel/rcupdate.c | 20 - > > kernel/rcupreempt.c | 433 +++++++++++++++++++++++++++++++++++---- > > kernel/rcupreempt_trace.c | 58 +++++ > > 8 files changed, 514 insertions(+), 71 deletions(-) > > > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcuclassic.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h > > --- linux-2.6.25/include/linux/rcuclassic.h 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcuclassic.h 2008-04-17 08:10:33.000000000 -0700 > > @@ -153,7 +153,10 @@ extern struct lockdep_map rcu_lock_map; > > > > #define __synchronize_sched() synchronize_rcu() > > > > +#define call_rcu_sched(head, func) call_rcu(head, func) > > + > > extern void __rcu_init(void); > > +#define rcu_init_sched() do { } while (0) > > extern void rcu_check_callbacks(int cpu, int user); > > extern void rcu_restart_cpu(int cpu); > > > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupdate.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h > > --- linux-2.6.25/include/linux/rcupdate.h 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupdate.h 2008-04-17 08:10:33.000000000 -0700 > > @@ -42,6 +42,7 @@ > > #include <linux/cpumask.h> > > #include <linux/seqlock.h> > > #include <linux/lockdep.h> > > +#include <linux/completion.h> > > > > /** > > * struct rcu_head - callback structure for use with RCU > > @@ -182,6 +183,27 @@ struct rcu_head { > > (p) = (v); \ > > }) > > > > +/* Infrastructure to implement the synchronize_() primitives. */ > > + > > +struct rcu_synchronize { > > + struct rcu_head head; > > + struct completion completion; > > +}; > > + > > +extern void wakeme_after_rcu(struct rcu_head *head); > > + > > +#define synchronize_rcu_xxx(name, func) \ > > +void name(void) \ > > +{ \ > > + struct rcu_synchronize rcu; \ > > + \ > > + init_completion(&rcu.completion); \ > > + /* Will wake me after RCU finished. */ \ > > + func(&rcu.head, wakeme_after_rcu); \ > > + /* Wait for it. */ \ > > + wait_for_completion(&rcu.completion); \ > > +} > > + > > /** > > * synchronize_sched - block until all CPUs have exited any non-preemptive > > * kernel code sequences. > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h > > --- linux-2.6.25/include/linux/rcupreempt.h 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt.h 2008-04-19 22:44:18.000000000 -0700 > > @@ -42,10 +42,39 @@ > > #include <linux/cpumask.h> > > #include <linux/seqlock.h> > > > > -#define rcu_qsctr_inc(cpu) > > +struct rcu_dyntick_sched { > > + int dynticks; > > + int dynticks_snap; > > + int sched_qs; > > + int sched_qs_snap; > > + int sched_dynticks_snap; > > +}; > > + > > +DECLARE_PER_CPU(struct rcu_dyntick_sched, rcu_dyntick_sched); > > + > > +static inline void rcu_qsctr_inc(int cpu) > > +{ > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + rdssp->sched_qs++; > > +} > > #define rcu_bh_qsctr_inc(cpu) > > #define call_rcu_bh(head, rcu) call_rcu(head, rcu) > > > > +/** > > + * call_rcu_sched - Queue RCU callback for invocation after sched grace period. > > + * @head: structure to be used for queueing the RCU updates. > > + * @func: actual update function to be invoked after the grace period > > + * > > + * The update function will be invoked some time after a full > > + * synchronize_sched()-style grace period elapses, in other words after > > + * all currently executing preempt-disabled sections of code (including > > + * hardirq handlers, NMI handlers, and local_irq_save() blocks) have > > + * completed. > > + */ > > +extern void call_rcu_sched(struct rcu_head *head, > > + void (*func)(struct rcu_head *head)); > > + > > extern void __rcu_read_lock(void) __acquires(RCU); > > extern void __rcu_read_unlock(void) __releases(RCU); > > extern int rcu_pending(int cpu); > > @@ -57,6 +86,7 @@ extern int rcu_needs_cpu(int cpu); > > extern void __synchronize_sched(void); > > > > extern void __rcu_init(void); > > +extern void rcu_init_sched(void); > > extern void rcu_check_callbacks(int cpu, int user); > > extern void rcu_restart_cpu(int cpu); > > extern long rcu_batches_completed(void); > > @@ -83,20 +113,20 @@ extern struct rcupreempt_trace *rcupreem > > struct softirq_action; > > > > #ifdef CONFIG_NO_HZ > > -DECLARE_PER_CPU(long, dynticks_progress_counter); > > +DECLARE_PER_CPU(struct rcu_dyntick_sched, dynticks_progress_counter); > ^^^^^^^^^^^^^^^^^^^^^^^^^ > Where is this defined? And why do we need it? We > already have rcu_dyntick_sched.dynticks for this purpose, no? Good catch, leftover line. I will remove it. > > > > static inline void rcu_enter_nohz(void) > > { > > smp_mb(); /* CPUs seeing ++ must see prior RCU read-side crit sects */ > > - __get_cpu_var(dynticks_progress_counter)++; > > - WARN_ON(__get_cpu_var(dynticks_progress_counter) & 0x1); > > + __get_cpu_var(rcu_dyntick_sched).dynticks++; > > + WARN_ON(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1); > > } > > > > static inline void rcu_exit_nohz(void) > > { > > - __get_cpu_var(dynticks_progress_counter)++; > > + __get_cpu_var(rcu_dyntick_sched).dynticks++; > > smp_mb(); /* CPUs seeing ++ must see later RCU read-side crit sects */ > > - WARN_ON(!(__get_cpu_var(dynticks_progress_counter) & 0x1)); > > + WARN_ON(!(__get_cpu_var(rcu_dyntick_sched).dynticks & 0x1)); > > } > > > > #else /* CONFIG_NO_HZ */ > > diff -urpNa -X dontdiff linux-2.6.25/include/linux/rcupreempt_trace.h linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h > > --- linux-2.6.25/include/linux/rcupreempt_trace.h 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/include/linux/rcupreempt_trace.h 2008-04-20 07:44:41.000000000 -0700 > > @@ -66,12 +66,15 @@ struct rcupreempt_trace { > > long rcu_try_flip_m1; > > long rcu_try_flip_me1; > > long rcu_try_flip_m2; > > + int rcu_sched_cpu; > > }; > > > > #ifdef CONFIG_RCU_TRACE > > -#define RCU_TRACE(fn, arg) fn(arg); > > +#define RCU_TRACE(fn, arg) fn(arg) > > +#define RCU_TRACE_1ARG(fn, arg, arg1) fn(arg, arg1) > > #else > > #define RCU_TRACE(fn, arg) > > +#define RCU_TRACE_1ARG(fn, arg, arg1) > > #endif > > > > extern void rcupreempt_trace_move2done(struct rcupreempt_trace *trace); > > @@ -94,6 +97,7 @@ extern void rcupreempt_trace_check_callb > > extern void rcupreempt_trace_done_remove(struct rcupreempt_trace *trace); > > extern void rcupreempt_trace_invoke(struct rcupreempt_trace *trace); > > extern void rcupreempt_trace_next_add(struct rcupreempt_trace *trace); > > +extern void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu); > > > > #endif /* __KERNEL__ */ > > #endif /* __LINUX_RCUPREEMPT_TRACE_H */ > > diff -urpNa -X dontdiff linux-2.6.25/init/main.c linux-2.6.25-C1-call_rcu_sched/init/main.c > > --- linux-2.6.25/init/main.c 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/init/main.c 2008-04-17 08:10:33.000000000 -0700 > > @@ -730,6 +730,7 @@ static void __init do_initcalls(void) > > */ > > static void __init do_basic_setup(void) > > { > > + rcu_init_sched(); /* needed by module_init stage. */ > > /* drivers will send hotplug events */ > > init_workqueues(); > > usermodehelper_init(); > > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupdate.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c > > --- linux-2.6.25/kernel/rcupdate.c 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupdate.c 2008-04-17 08:10:33.000000000 -0700 > > @@ -39,18 +39,12 @@ > > #include <linux/sched.h> > > #include <asm/atomic.h> > > #include <linux/bitops.h> > > -#include <linux/completion.h> > > #include <linux/percpu.h> > > #include <linux/notifier.h> > > #include <linux/cpu.h> > > #include <linux/mutex.h> > > #include <linux/module.h> > > > > -struct rcu_synchronize { > > - struct rcu_head head; > > - struct completion completion; > > -}; > > - > > static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL}; > > static atomic_t rcu_barrier_cpu_count; > > static DEFINE_MUTEX(rcu_barrier_mutex); > > @@ -60,7 +54,7 @@ static struct completion rcu_barrier_com > > * Awaken the corresponding synchronize_rcu() instance now that a > > * grace period has elapsed. > > */ > > -static void wakeme_after_rcu(struct rcu_head *head) > > +void wakeme_after_rcu(struct rcu_head *head) > > { > > struct rcu_synchronize *rcu; > > > > @@ -77,17 +71,7 @@ static void wakeme_after_rcu(struct rcu_ > > * sections are delimited by rcu_read_lock() and rcu_read_unlock(), > > * and may be nested. > > */ > > -void synchronize_rcu(void) > > -{ > > - struct rcu_synchronize rcu; > > - > > - init_completion(&rcu.completion); > > - /* Will wake me after RCU finished */ > > - call_rcu(&rcu.head, wakeme_after_rcu); > > - > > - /* Wait for it */ > > - wait_for_completion(&rcu.completion); > > -} > > +synchronize_rcu_xxx(synchronize_rcu, call_rcu) > > EXPORT_SYMBOL_GPL(synchronize_rcu); > > > > static void rcu_barrier_callback(struct rcu_head *notused) > > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c > > --- linux-2.6.25/kernel/rcupreempt.c 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt.c 2008-04-20 07:58:31.000000000 -0700 > > @@ -46,6 +46,7 @@ > > #include <asm/atomic.h> > > #include <linux/bitops.h> > > #include <linux/module.h> > > +#include <linux/kthread.h> > > #include <linux/completion.h> > > #include <linux/moduleparam.h> > > #include <linux/percpu.h> > > @@ -87,9 +88,14 @@ struct rcu_data { > > struct rcu_head **nexttail; > > struct rcu_head *waitlist[GP_STAGES]; > > struct rcu_head **waittail[GP_STAGES]; > > - struct rcu_head *donelist; > > + struct rcu_head *donelist; /* from waitlist & waitschedlist */ > > struct rcu_head **donetail; > > long rcu_flipctr[2]; > > + struct rcu_head *nextschedlist; > > + struct rcu_head **nextschedtail; > > + struct rcu_head *waitschedlist; > > + struct rcu_head **waitschedtail; > > + int rcu_sched_sleeping; > > #ifdef CONFIG_RCU_TRACE > > struct rcupreempt_trace trace; > > #endif /* #ifdef CONFIG_RCU_TRACE */ > > @@ -131,11 +137,24 @@ enum rcu_try_flip_states { > > rcu_try_flip_waitmb_state, > > }; > > > > +/* > > + * States for rcu_ctrlblk.rcu_sched_sleep. > ^^^^^^^^^^^^^^^^^^^^^^^^^^^ should be > rcu_ctrlblk.sched_sleep > > + */ > > + > > +enum rcu_sched_sleep_states { > > + rcu_sched_not_sleeping, /* Not sleeping, callbacks need GP. */ > > + rcu_sched_sleep_prep, /* Thinking of sleeping, rechecking. */ > > + rcu_sched_sleeping, /* Sleeping, awaken if GP needed. */ > > +}; > > + > > struct rcu_ctrlblk { > > spinlock_t fliplock; /* Protect state-machine transitions. */ > > long completed; /* Number of last completed batch. */ > > enum rcu_try_flip_states rcu_try_flip_state; /* The current state of > > the rcu state machine */ > > + spinlock_t schedlock; /* Protect rcu_sched sleep state. */ > > + enum rcu_sched_sleep_states sched_sleep; /* rcu_sched state. */ > > + wait_queue_head_t sched_wq; /* Place for rcu_sched to sleep. */ > > }; > > > > static DEFINE_PER_CPU(struct rcu_data, rcu_data); > > @@ -143,8 +162,12 @@ static struct rcu_ctrlblk rcu_ctrlblk = > > .fliplock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.fliplock), > > .completed = 0, > > .rcu_try_flip_state = rcu_try_flip_idle_state, > > + .schedlock = __SPIN_LOCK_UNLOCKED(rcu_ctrlblk.schedlock), > > + .sched_sleep = rcu_sched_not_sleeping, > > + .sched_wq = __WAIT_QUEUE_HEAD_INITIALIZER(rcu_ctrlblk.sched_wq), > > }; > > > > +static struct task_struct *rcu_sched_grace_period_task; > > > > #ifdef CONFIG_RCU_TRACE > > static char *rcu_try_flip_state_names[] = > > @@ -194,6 +217,7 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu > > * cached in a local variable, but where the CPU number is so cached. > > */ > > #define RCU_TRACE_CPU(f, cpu) RCU_TRACE(f, &(RCU_DATA_CPU(cpu)->trace)); > > +#define RCU_TRACE_1ARG_CPU(f, cpu, a) RCU_TRACE_1ARG(f, &(RCU_DATA_CPU(cpu)->trace), a); > > > > /* > > * Helper macro for tracing when the appropriate rcu_data is not > > @@ -207,6 +231,8 @@ static DEFINE_PER_CPU_SHARED_ALIGNED(enu > > */ > > #define RCU_TRACE_RDP(f, rdp) RCU_TRACE(f, &((rdp)->trace)); > > > > +#define RCU_SCHED_BATCH_TIME (HZ / 50) > > + > > /* > > * Return the number of RCU batches processed thus far. Useful > > * for debug and statistics. > > @@ -413,32 +439,34 @@ static void __rcu_advance_callbacks(stru > > } > > } > > > > -#ifdef CONFIG_NO_HZ > > +DEFINE_PER_CPU_SHARED_ALIGNED(struct rcu_dyntick_sched, rcu_dyntick_sched) = { > > + .dynticks = 1, > > +}; > > > > -DEFINE_PER_CPU(long, dynticks_progress_counter) = 1; > > -static DEFINE_PER_CPU(long, rcu_dyntick_snapshot); > > +#ifdef CONFIG_NO_HZ > > static DEFINE_PER_CPU(int, rcu_update_flag); > > > > /** > > * rcu_irq_enter - Called from Hard irq handlers and NMI/SMI. > > * > > * If the CPU was idle with dynamic ticks active, this updates the > > - * dynticks_progress_counter to let the RCU handling know that the > > + * rcu_dyntick_sched.dynticks to let the RCU handling know that the > > * CPU is active. > > */ > > void rcu_irq_enter(void) > > { > > int cpu = smp_processor_id(); > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > > > if (per_cpu(rcu_update_flag, cpu)) > > per_cpu(rcu_update_flag, cpu)++; > > > > /* > > * Only update if we are coming from a stopped ticks mode > > - * (dynticks_progress_counter is even). > > + * (rcu_dyntick_sched.dynticks is even). > > */ > > if (!in_interrupt() && > > - (per_cpu(dynticks_progress_counter, cpu) & 0x1) == 0) { > > + (rdssp->dynticks & 0x1) == 0) { > > /* > > * The following might seem like we could have a race > > * with NMI/SMIs. But this really isn't a problem. > > @@ -461,12 +489,12 @@ void rcu_irq_enter(void) > > * RCU read-side critical sections on this CPU would > > * have already completed. > > */ > > - per_cpu(dynticks_progress_counter, cpu)++; > > + rdssp->dynticks++; > > /* > > * The following memory barrier ensures that any > > * rcu_read_lock() primitives in the irq handler > > * are seen by other CPUs to follow the above > > - * increment to dynticks_progress_counter. This is > > + * increment to rcu_dyntick_sched.dynticks. This is > > * required in order for other CPUs to correctly > > * determine when it is safe to advance the RCU > > * grace-period state machine. > > @@ -474,7 +502,7 @@ void rcu_irq_enter(void) > > smp_mb(); /* see above block comment. */ > > /* > > * Since we can't determine the dynamic tick mode from > > - * the dynticks_progress_counter after this routine, > > + * the rcu_dyntick_sched.dynticks after this routine, > > * we use a second flag to acknowledge that we came > > * from an idle state with ticks stopped. > > */ > > @@ -482,7 +510,7 @@ void rcu_irq_enter(void) > > /* > > * If we take an NMI/SMI now, they will also increment > > * the rcu_update_flag, and will not update the > > - * dynticks_progress_counter on exit. That is for > > + * rcu_dyntick_sched.dynticks on exit. That is for > > * this IRQ to do. > > */ > > } > > @@ -492,12 +520,13 @@ void rcu_irq_enter(void) > > * rcu_irq_exit - Called from exiting Hard irq context. > > * > > * If the CPU was idle with dynamic ticks active, update the > > - * dynticks_progress_counter to put let the RCU handling be > > + * rcu_dyntick_sched.dynticks to put let the RCU handling be > > * aware that the CPU is going back to idle with no ticks. > > */ > > void rcu_irq_exit(void) > > { > > int cpu = smp_processor_id(); > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > > > /* > > * rcu_update_flag is set if we interrupted the CPU > > @@ -505,7 +534,7 @@ void rcu_irq_exit(void) > > * Once this occurs, we keep track of interrupt nesting > > * because a NMI/SMI could also come in, and we still > > * only want the IRQ that started the increment of the > > - * dynticks_progress_counter to be the one that modifies > > + * rcu_dyntick_sched.dynticks to be the one that modifies > > * it on exit. > > */ > > if (per_cpu(rcu_update_flag, cpu)) { > > @@ -517,28 +546,29 @@ void rcu_irq_exit(void) > > > > /* > > * If an NMI/SMI happens now we are still > > - * protected by the dynticks_progress_counter being odd. > > + * protected by the rcu_dyntick_sched.dynticks being odd. > > */ > > > > /* > > * The following memory barrier ensures that any > > * rcu_read_unlock() primitives in the irq handler > > * are seen by other CPUs to preceed the following > > - * increment to dynticks_progress_counter. This > > + * increment to rcu_dyntick_sched.dynticks. This > > * is required in order for other CPUs to determine > > * when it is safe to advance the RCU grace-period > > * state machine. > > */ > > smp_mb(); /* see above block comment. */ > > - per_cpu(dynticks_progress_counter, cpu)++; > > - WARN_ON(per_cpu(dynticks_progress_counter, cpu) & 0x1); > > + rdssp->dynticks++; > > + WARN_ON(rdssp->dynticks & 0x1); > > } > > } > > > > static void dyntick_save_progress_counter(int cpu) > > { > > - per_cpu(rcu_dyntick_snapshot, cpu) = > > - per_cpu(dynticks_progress_counter, cpu); > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + rdssp->dynticks_snap = rdssp->dynticks; > > } > > > > static inline int > > @@ -546,9 +576,10 @@ rcu_try_flip_waitack_needed(int cpu) > > { > > long curr; > > long snap; > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > > > - curr = per_cpu(dynticks_progress_counter, cpu); > > - snap = per_cpu(rcu_dyntick_snapshot, cpu); > > + curr = rdssp->dynticks; > > + snap = rdssp->dynticks_snap; > > smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > > > > /* > > @@ -582,9 +613,10 @@ rcu_try_flip_waitmb_needed(int cpu) > > { > > long curr; > > long snap; > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > > > - curr = per_cpu(dynticks_progress_counter, cpu); > > - snap = per_cpu(rcu_dyntick_snapshot, cpu); > > + curr = rdssp->dynticks; > > + snap = rdssp->dynticks_snap; > > smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > > > > /* > > @@ -611,14 +643,85 @@ rcu_try_flip_waitmb_needed(int cpu) > > return 1; > > } > > > > +static void dyntick_save_progress_counter_sched(int cpu) > > +{ > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + rdssp->sched_dynticks_snap = rdssp->dynticks; > > +} > > + > > +static int rcu_qsctr_inc_needed_dyntick(int cpu) > > +{ > > + long curr; > > + long snap; > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + curr = rdssp->dynticks; > > + snap = rdssp->sched_dynticks_snap; > > + smp_mb(); /* force ordering with cpu entering/leaving dynticks. */ > > + > > + /* > > + * If the CPU remained in dynticks mode for the entire time > > + * and didn't take any interrupts, NMIs, SMIs, or whatever, > > + * then it cannot be in the middle of an preempt-disabled > > + * region of code. Therefore, this CPU has been in a quiescent > > + * state the entire time, and we don't need to wait for it. > > + */ > > + > > + if ((curr == snap) && ((curr & 0x1) == 0)) > > + return 0; > > + > > + /* > > + * If the CPU passed through or entered a dynticks idle phase with > > + * no active irq handlers, then, as above, this CPU has already > > + * passed through a quiescent state. > > + */ > > + > > + if ((curr - snap) > 2 || (snap & 0x1) == 0) > > + return 0; > > + > > + /* We need this CPU to go through a quiescent state. */ > > + > > + return 1; > > +} > > + > > #else /* !CONFIG_NO_HZ */ > > > > -# define dyntick_save_progress_counter(cpu) do { } while (0) > > -# define rcu_try_flip_waitack_needed(cpu) (1) > > -# define rcu_try_flip_waitmb_needed(cpu) (1) > > +# define dyntick_save_progress_counter(cpu) do { } while (0) > > +# define rcu_try_flip_waitack_needed(cpu) (1) > > +# define rcu_try_flip_waitmb_needed(cpu) (1) > > + > > +# define dyntick_save_progress_counter_sched(cpu) do { } while (0) > > +# define rcu_qsctr_inc_needed_dyntick(cpu) (1) > > > > #endif /* CONFIG_NO_HZ */ > > > > +static void save_qsctr_sched(int cpu) > > +{ > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + rdssp->sched_qs_snap = rdssp->sched_qs; > > +} > > + > > +static inline int rcu_qsctr_inc_needed(int cpu) > > +{ > > + struct rcu_dyntick_sched *rdssp = &per_cpu(rcu_dyntick_sched, cpu); > > + > > + /* > > + * If there has been a quiescent state, no more need to wait > > + * on this CPU. > > + */ > > + > > + if (rdssp->sched_qs != rdssp->sched_qs_snap) { > > + smp_mb(); /* force ordering with cpu entering schedule(). */ > > + return 0; > > + } > > + > > + /* We need this CPU to go through a quiescent state. */ > > + > > + return 1; > > +} > > + > > /* > > * Get here when RCU is idle. Decide whether we need to > > * move out of idle state, and return non-zero if so. > > @@ -821,6 +924,26 @@ void rcu_check_callbacks(int cpu, int us > > unsigned long flags; > > struct rcu_data *rdp = RCU_DATA_CPU(cpu); > > > > + /* > > + * If this CPU took its interrupt from user mode or from the > > + * idle loop, and this is not a nested interrupt, then > > + * this CPU has to have exited all prior preept-disable > > + * sections of code. So increment the counter to note this. > > + * > > + * The memory barrier is needed to handle the case where > > + * writes from a preempt-disable section of code get reordered > > + * into schedule() by this CPU's write buffer. So the memory > > + * barrier makes sure that the rcu_qsctr_inc() is seen by other > > + * CPUs to happen after any such write. > > + */ > > + > > + if (user || > > + (idle_cpu(cpu) && !in_softirq() && > > + hardirq_count() <= (1 << HARDIRQ_SHIFT))) { > > + smp_mb(); /* Guard against aggressive schedule(). */ > > + rcu_qsctr_inc(cpu); > > + } > > + > > rcu_check_mb(cpu); > > if (rcu_ctrlblk.completed == rdp->completed) > > rcu_try_flip(); > > @@ -871,6 +994,8 @@ void rcu_offline_cpu(int cpu) > > struct rcu_head *list = NULL; > > unsigned long flags; > > struct rcu_data *rdp = RCU_DATA_CPU(cpu); > > + struct rcu_head *schedlist = NULL; > > + struct rcu_head **schedtail = &schedlist; > > struct rcu_head **tail = &list; > > > > /* > > @@ -884,6 +1009,11 @@ void rcu_offline_cpu(int cpu) > > rcu_offline_cpu_enqueue(rdp->waitlist[i], rdp->waittail[i], > > list, tail); > > rcu_offline_cpu_enqueue(rdp->nextlist, rdp->nexttail, list, tail); > > + rcu_offline_cpu_enqueue(rdp->waitschedlist, rdp->waitschedtail, > > + schedlist, schedtail); > > + rcu_offline_cpu_enqueue(rdp->nextschedlist, rdp->nextschedtail, > > + schedlist, schedtail); > > + rdp->rcu_sched_sleeping = 0; > > spin_unlock_irqrestore(&rdp->lock, flags); > > rdp->waitlistcount = 0; > > > > @@ -918,22 +1048,40 @@ void rcu_offline_cpu(int cpu) > > * fix. > > */ > > > > - local_irq_save(flags); > > + local_irq_save(flags); /* disable preempt till we know what lock. */ > > rdp = RCU_DATA_ME(); > > spin_lock(&rdp->lock); > > *rdp->nexttail = list; > > if (list) > > rdp->nexttail = tail; > > + *rdp->nextschedtail = schedlist; > > + if (schedlist) > > + rdp->nextschedtail = schedtail; > > spin_unlock_irqrestore(&rdp->lock, flags); > > } > > > > void __devinit rcu_online_cpu(int cpu) > > { > > unsigned long flags; > > + struct rcu_data *rdp; > > > > spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); > > cpu_set(cpu, rcu_cpu_online_map); > > spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); > > + > > + /* > > + * The rcu_sched grace-period processing might have bypassed > > + * this CPU, given that it was not in the rcu_cpu_online_map > > + * when the grace-period scan started. This means that the > > + * grace-period task might sleep. So make sure that if this > > + * should happen, the first callback posted to this CPU will > > + * wake up the grace-period task if need be. > > + */ > > + > > + rdp = RCU_DATA_CPU(cpu); > > + spin_lock_irqsave(&rdp->lock, flags); > > + rdp->rcu_sched_sleeping = 1; > > + spin_unlock_irqrestore(&rdp->lock, flags); > > } > > > > #else /* #ifdef CONFIG_HOTPLUG_CPU */ > > @@ -988,31 +1136,215 @@ void call_rcu(struct rcu_head *head, voi > > *rdp->nexttail = head; > > rdp->nexttail = &head->next; > > RCU_TRACE_RDP(rcupreempt_trace_next_add, rdp); > > - spin_unlock(&rdp->lock); > > - local_irq_restore(flags); > > + spin_unlock_irqrestore(&rdp->lock, flags); > > } > > EXPORT_SYMBOL_GPL(call_rcu); > > > > +void call_rcu_sched(struct rcu_head *head, void (*func)(struct rcu_head *rcu)) > > +{ > > + unsigned long flags; > > + struct rcu_data *rdp; > > + int wake_gp = 0; > > + > > + head->func = func; > > + head->next = NULL; > > + local_irq_save(flags); > > + rdp = RCU_DATA_ME(); > > + spin_lock(&rdp->lock); > > + *rdp->nextschedtail = head; > > + rdp->nextschedtail = &head->next; > > + if (rdp->rcu_sched_sleeping) { > > + > > + /* Grace-period processing might be sleeping... */ > > + > > + rdp->rcu_sched_sleeping = 0; > > + wake_gp = 1; > > + } > > + spin_unlock_irqrestore(&rdp->lock, flags); > > + if (wake_gp) { > > + > > + /* Wake up grace-period processing, unless someone beat us. */ > > + > > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > > + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleeping) > > + wake_gp = 0; > > + rcu_ctrlblk.sched_sleep = rcu_sched_not_sleeping; > > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > > + if (wake_gp) > > + wake_up_interruptible(&rcu_ctrlblk.sched_wq); > > + } > > +} > > +EXPORT_SYMBOL_GPL(call_rcu_sched); > > + > > /* > > * Wait until all currently running preempt_disable() code segments > > * (including hardware-irq-disable segments) complete. Note that > > * in -rt this does -not- necessarily result in all currently executing > > * interrupt -handlers- having completed. > > */ > > -void __synchronize_sched(void) > > +synchronize_rcu_xxx(__synchronize_sched, call_rcu_sched) > > +EXPORT_SYMBOL_GPL(__synchronize_sched); > > + > > +/* > > + * kthread function that manages call_rcu_sched grace periods. > > + */ > > +static int rcu_sched_grace_period(void *arg) > > { > > - cpumask_t oldmask; > > + int couldsleep; /* might sleep after current pass. */ > > + int couldsleepnext = 0; /* might sleep after next pass. */ > > int cpu; > > + unsigned long flags; > > + cpumask_t cpu_snap; > > + struct rcu_data *rdp; > > + int ret; > > > > - if (sched_getaffinity(0, &oldmask) < 0) > > - oldmask = cpu_possible_map; > > - for_each_online_cpu(cpu) { > > - sched_setaffinity(0, cpumask_of_cpu(cpu)); > > - schedule(); > > - } > > - sched_setaffinity(0, oldmask); > > + /* > > + * Each pass through the following loop handles one > > + * rcu_sched grace period cycle. > > + */ > > + do { > > + /* Save each CPU's current state. */ > > + > > + spin_lock_irqsave(&rcu_ctrlblk.fliplock, flags); > > + cpu_snap = cpu_online_map; > > + spin_unlock_irqrestore(&rcu_ctrlblk.fliplock, flags); > > + for_each_cpu_mask(cpu, cpu_snap) { > > + if (cpu_is_offline(cpu)) { > > + cpu_clear(cpu, cpu_snap); > > + continue; > > + } > > + dyntick_save_progress_counter_sched(cpu); > > + save_qsctr_sched(cpu); > > + if (cpu_is_offline(cpu)) > > + cpu_clear(cpu, cpu_snap); > > + } > > + > > + /* > > + * Sleep for about an RCU grace-period's worth to > > + * allow better batching and to consume less CPU. > > + */ > > + schedule_timeout_interruptible(RCU_SCHED_BATCH_TIME); > > + > > + /* > > + * If there was nothing to do last time, prepare to > > + * sleep at the end of the current grace period cycle. > > + */ > > + couldsleep = couldsleepnext; > > + couldsleepnext = 1; > > + if (couldsleep) { > > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > > + rcu_ctrlblk.sched_sleep = rcu_sched_sleep_prep; > > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > > + } > > + > > + /* > > + * Wait on each CPU in turn to have either visited > > + * a quiescent state or been in dynticks-idle mode. > > + */ > > + for_each_cpu_mask(cpu, cpu_snap) { > > + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, cpu); > > + while (rcu_qsctr_inc_needed(cpu) && > > + rcu_qsctr_inc_needed_dyntick(cpu)) { > > + if (cpu_is_offline(cpu)) { > > + cpu_clear(cpu, cpu_snap); > > + break; > > + } > > + /* resched_cpu(cpu); @@@ */ > > + schedule_timeout_interruptible(1); > > + } > > + RCU_TRACE_1ARG_CPU(rcusched_trace_grace_wait, 0, -1); > > + } > > + > > + /* Advance callbacks for each CPU. */ > > + > > + for_each_cpu_mask(cpu, cpu_snap) { > > + rdp = RCU_DATA_CPU(cpu); > > + spin_lock_irqsave(&rdp->lock, flags); > > + if (cpu_is_offline(cpu)) { > > + spin_unlock_irqrestore(&rdp->lock, flags); > > + continue; > > + } > > + > > + /* > > + * We are running on this CPU irq-disabled, so no > > + * CPU can go offline until we re-enable irqs. > > + * The current CPU might have already gone > > + * offline (between the for_each_offline_cpu and > ^^^^^^^^^^^^^^^^^^^^ > for_each_cpu_mask(). > > Infact, for_each_offline_cpu() doesn't even exist ;-) Good eyes, fixed! > I guess doing the multiple cpu_is_offline() checks is unavoidable, > unless we use get_online_cpus()/put_online_cpus(). But that's an > overkill in this context. Yeah, I didn't see a nicer way to do this, but if you think of one... > So, looks okay to me. > > > > + * the spin_lock_irqsave), but in that case all its > > + * callback lists will be empty, so no harm done. > > + * > > + * Advance the callbacks! We share normal RCU's > > + * donelist, since callbacks are invoked the > > + * same way in either case. > > + */ > > + if (rdp->waitschedlist != NULL) { > > + *rdp->donetail = rdp->waitschedlist; > > + rdp->donetail = rdp->waitschedtail; > > + > > + /* > > + * Next rcu_check_callbacks() will > > + * do the required raise_softirq(). > > + */ > > + } > > + if (rdp->nextschedlist != NULL) { > > + rdp->waitschedlist = rdp->nextschedlist; > > + rdp->waitschedtail = rdp->nextschedtail; > > + couldsleep = 0; > > + couldsleepnext = 0; > > + } else { > > + rdp->waitschedlist = NULL; > > + rdp->waitschedtail = &rdp->waitschedlist; > > + } > > + rdp->nextschedlist = NULL; > > + rdp->nextschedtail = &rdp->nextschedlist; > > + > > + /* Mark sleep intention. */ > > + > > + rdp->rcu_sched_sleeping = couldsleep; > > + > > + spin_unlock_irqrestore(&rdp->lock, flags); > > + } > > + > > + /* If we saw callbacks on the last scan, go deal with them. */ > > + > > + if (!couldsleep) > > + continue; > > + > > + /* Attempt to block... */ > > + > > + spin_lock_irqsave(&rcu_ctrlblk.schedlock, flags); > > + if (rcu_ctrlblk.sched_sleep != rcu_sched_sleep_prep) { > > + > > + /* > > + * Someone posted a callback after we scanned. > > + * Go take care of it. > > + */ > > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > > + couldsleepnext = 0; > > + continue; > > + } > > + > > + /* Block until the next person posts a callback. */ > > + > > + rcu_ctrlblk.sched_sleep = rcu_sched_sleeping; > > + spin_unlock_irqrestore(&rcu_ctrlblk.schedlock, flags); > > + ret = 0; > > + __wait_event_interruptible(rcu_ctrlblk.sched_wq, > > + rcu_ctrlblk.sched_sleep != rcu_sched_sleeping, > > + ret); > > + > > + /* > > + * Signals would prevent us from sleeping, and we cannot > > + * do much with them in any case. So flush them. > > + */ > > + if (ret) > > + flush_signals(current); > > + couldsleepnext = 0; > > + > > + } while (!kthread_should_stop()); > > + > > + return (0); > > } > > -EXPORT_SYMBOL_GPL(__synchronize_sched); > > > > /* > > * Check to see if any future RCU-related work will need to be done > > @@ -1029,7 +1361,9 @@ int rcu_needs_cpu(int cpu) > > > > return (rdp->donelist != NULL || > > !!rdp->waitlistcount || > > - rdp->nextlist != NULL); > > + rdp->nextlist != NULL || > > + rdp->nextschedlist != NULL || > > + rdp->waitschedlist != NULL); > > } > > > > int rcu_pending(int cpu) > > @@ -1040,7 +1374,9 @@ int rcu_pending(int cpu) > > > > if (rdp->donelist != NULL || > > !!rdp->waitlistcount || > > - rdp->nextlist != NULL) > > + rdp->nextlist != NULL || > > + rdp->nextschedlist != NULL || > > + rdp->waitschedlist != NULL) > > return 1; > > > > /* The RCU core needs an acknowledgement from this CPU. */ > > @@ -1107,6 +1443,11 @@ void __init __rcu_init(void) > > rdp->donetail = &rdp->donelist; > > rdp->rcu_flipctr[0] = 0; > > rdp->rcu_flipctr[1] = 0; > > + rdp->nextschedlist = NULL; > > + rdp->nextschedtail = &rdp->nextschedlist; > > + rdp->waitschedlist = NULL; > > + rdp->waitschedtail = &rdp->waitschedlist; > > + rdp->rcu_sched_sleeping = 0; > > } > > register_cpu_notifier(&rcu_nb); > > > > @@ -1129,11 +1470,15 @@ void __init __rcu_init(void) > > } > > > > /* > > - * Deprecated, use synchronize_rcu() or synchronize_sched() instead. > > + * Late-boot-time RCU initialization that must wait until after scheduler > > + * has been initialized. > > */ > > -void synchronize_kernel(void) > > +void __init rcu_init_sched(void) > > { > > - synchronize_rcu(); > > + rcu_sched_grace_period_task = kthread_run(rcu_sched_grace_period, > > + NULL, > > + "rcu_sched_grace_period"); > > + WARN_ON(IS_ERR(rcu_sched_grace_period_task)); > > } > > > > #ifdef CONFIG_RCU_TRACE > > diff -urpNa -X dontdiff linux-2.6.25/kernel/rcupreempt_trace.c linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c > > --- linux-2.6.25/kernel/rcupreempt_trace.c 2008-04-16 19:49:44.000000000 -0700 > > +++ linux-2.6.25-C1-call_rcu_sched/kernel/rcupreempt_trace.c 2008-04-21 15:45:07.000000000 -0700 > > @@ -134,6 +134,10 @@ void rcupreempt_trace_next_add(struct rc > > trace->next_add++; > > trace->next_length++; > > } > > +void rcusched_trace_grace_wait(struct rcupreempt_trace *trace, int cpu) > > +{ > > + trace->rcu_sched_cpu = cpu; > > +} > > > > static void rcupreempt_trace_sum(struct rcupreempt_trace *sp) > > { > > @@ -262,6 +266,38 @@ static ssize_t rcuctrs_read(struct file > > return bcount; > > } > > > > +static ssize_t rcuschedstat_read(struct file *filp, char __user *buffer, > > + size_t count, loff_t *ppos) > > +{ > > + int cnt = 0; > > + int cpu = rcupreempt_trace_cpu(0)->rcu_sched_cpu; > > + ssize_t bcount; > > + > > + mutex_lock(&rcupreempt_trace_mutex); > > + > > + cnt += snprintf(&rcupreempt_trace_buf[cnt], RCUPREEMPT_TRACE_BUF_SIZE, > > + "CPU: %d ", cpu); > > + if (cpu != -1) { > > + struct rcu_dyntick_sched *rdsp; > > + > > + rdsp = &per_cpu(rcu_dyntick_sched, cpu); > > + cnt += snprintf(&rcupreempt_trace_buf[cnt], > > + RCUPREEMPT_TRACE_BUF_SIZE - cnt, > > + "dynticks: %d %d qs %d %d", > > + rdsp->dynticks, > > + rdsp->sched_dynticks_snap, > > + rdsp->sched_qs, > > + rdsp->sched_qs_snap); > > + } > > + cnt += snprintf(&rcupreempt_trace_buf[cnt], > > + RCUPREEMPT_TRACE_BUF_SIZE - cnt, > > + "\n"); > > + bcount = simple_read_from_buffer(buffer, count, ppos, > > + rcupreempt_trace_buf, strlen(rcupreempt_trace_buf)); > > + mutex_unlock(&rcupreempt_trace_mutex); > > + return bcount; > > +} > > + > > static struct file_operations rcustats_fops = { > > .owner = THIS_MODULE, > > .read = rcustats_read, > > @@ -277,7 +313,12 @@ static struct file_operations rcuctrs_fo > > .read = rcuctrs_read, > > }; > > > > -static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir; > > +static struct file_operations rcuschedstat_fops = { > > + .owner = THIS_MODULE, > > + .read = rcuschedstat_read, > > +}; > > + > > +static struct dentry *rcudir, *statdir, *ctrsdir, *gpdir, *schedstatdir; > > static int rcupreempt_debugfs_init(void) > > { > > rcudir = debugfs_create_dir("rcu", NULL); > > @@ -296,12 +337,19 @@ static int rcupreempt_debugfs_init(void) > > NULL, &rcuctrs_fops); > > if (!ctrsdir) > > goto free_out; > > + > > + schedstatdir = debugfs_create_file("schedstat", 0444, rcudir, > > + NULL, &rcuschedstat_fops); > > + if (!schedstatdir) > > + goto free_out; > > return 0; > > free_out: > > if (statdir) > > debugfs_remove(statdir); > > if (gpdir) > > debugfs_remove(gpdir); > > + if (ctrsdir) > > + debugfs_remove(ctrsdir); > > debugfs_remove(rcudir); > > out: > > return 1; > > @@ -309,11 +357,16 @@ out: > > > > static int __init rcupreempt_trace_init(void) > > { > > + int ret; > > + > > mutex_init(&rcupreempt_trace_mutex); > > rcupreempt_trace_buf = kmalloc(RCUPREEMPT_TRACE_BUF_SIZE, GFP_KERNEL); > > if (!rcupreempt_trace_buf) > > return 1; > > - return rcupreempt_debugfs_init(); > > + ret = rcupreempt_debugfs_init(); > > + if (ret) > > + kfree(rcupreempt_trace_buf); > > + return ret; > > } > > > > static void __exit rcupreempt_trace_cleanup(void) > > @@ -322,6 +375,7 @@ static void __exit rcupreempt_trace_clea > > debugfs_remove(gpdir); > > debugfs_remove(ctrsdir); > > debugfs_remove(rcudir); > > + debugfs_remove(schedstatdir); > > kfree(rcupreempt_trace_buf); > > } > > -- > Thanks and Regards > gautham ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2008-05-05 14:43 UTC | newest] Thread overview: 12+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2008-04-14 3:58 [PATCH 0/5] call_rcu_sched() series Paul E. McKenney 2008-04-14 4:00 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney 2008-04-14 4:01 ` [PATCH 2/5] Add memory barriers and comments to rcu_check_callbacks() Paul E. McKenney 2008-04-14 4:02 ` [PATCH 3/5] Add rcu_barrier_sched() and rcu_barrier_bh() Paul E. McKenney 2008-04-14 4:03 ` [PATCH 4/5] Add call_rcu_sched() and friends to rcutorture Paul E. McKenney 2008-04-14 4:04 ` [PATCH 5/5] 1Q08 RCU doc update, add call_rcu_sched() Paul E. McKenney 2008-04-15 1:33 ` [PATCH 0/5] call_rcu_sched() series Andrew Morton 2008-04-15 1:45 ` Andrew Morton 2008-04-15 1:48 ` Andrew Morton -- strict thread matches above, loose matches on Subject: below -- 2008-04-22 0:44 Paul E. McKenney 2008-04-22 0:45 ` Paul E. McKenney 2008-04-22 0:50 ` [PATCH 1/5] Add call_rcu_sched() Paul E. McKenney 2008-05-05 9:14 ` Gautham R Shenoy 2008-05-05 14:43 ` Paul E. McKenney
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).