[RFC PATCH 0/8] Convert all tasklets to workqueues V3

[RFC PATCH 0/8] Convert all tasklets to workqueues V3

[Steven Rostedt]

-- 

This is Version 3 of tasklets to work queues conversion.

Changes from this and version 2:

 - Removed DECLARE_TASKLET and DECLARE_TASKLET_DISABLED macros
   Instead make all instances use tasklet_init.
   (Recommended by Arnd Bergmann)

 - Converted the net wireless nlevent_tasket into a work queue
   (there was no good way to use tasklet_init, that I know of).

 - Removed fastcall from tasklet_enable (why was that needed?)

 - Removed tasklet_kill_immediate.
   (suggested by Oleg Nesterov)

 - Removed takeover_tasklet
    (also suggested by Oleg Nesterov)

 - Removed smp_mb from tasklet_disable since the
   flush_workqueue should provide us a barrier.

 - Converted most the functions to static inline.
   (Recommended by Arjan van de Ven)

 - Removed unused tasklet state flags.

 - Put in stub for tasklet_unlock since nothing should call it
   (the only user should be fixed in 2.6.22)

 - I didn't add cleanups to make the moving of code better for
   checkpatch.pl. But I did fix the spots it flagged for added
   code.

Prologue from Version 1, you may skip if you already read this.
Added at Andrew Morton's request.  Slightly modified from V1 to
accomodate changes.


There's a very nice paper by Matthew Willcox that describes Softirqs,
Tasklets, Bottom Halves, Task Queues, Work Queues and Timers[1].
In the paper it describes the history of these items.  Softirqs and
tasklets were created to replace bottom halves after a company (Mindcraft)
showed that Microsoft on a 4x SMP box would out do Linux. It was discovered
that this was due to a bottle neck caused by the design of Bottom Halves.
So Alexey Kuznetsov and Dave Miller [1] (and I'm sure others) created
softirqs and tasklets to multithread the bottom halves.

This worked well, and for the time it shut-up Microsoft^WMindcraft from
saying Linux was slow at networking.

Time passed, and Linux developed other nifty tools, like kthreads and
work queues. These run in a process context and are not as menacing to
latencies as softirqs and tasklets are.  Specifically, a tasklet,
acts as a task by only being able to run the function on one CPU
at a time. The same tasklet can not run on multiple CPUS.  So in that
aspect it is like a task (a task can only exist on one CPU at a time).
But a tasklet is much harder on the rest of the system because it
runs in interrupt context.  This means that if a higher priority process
wants to run, it must wait for the tasklet to finish before doing so.

The most part, tasklets today are not used for time critical functions.
Running tasklets in thread context is not harmful to performance of
the overall system. But running them in interrupt context is, since
they increase the overall latency for high priority tasks.

Even in Matthew's paper, he says that work queues have replaced tasklets.
But this is not truly the case.  Tasklets are common and plentiful.
But to go and replace each driver that uses a tasklet with a work queue
would be very painful.

I've developed this way to replace all tasklets with work queues without
having to change all the drivers that use them.  I created an API that
uses the tasklet API as a wrapper to a work queue.  This API doesn't need
to be permanent. It shows 1) that work queues can replace tasklets, and
2) we can remove a duplicate functionality from the kernel.  This API
only needs to be around until we removed all uses of tasklets from
all drivers.

I just want to state that tasklets served their time well. But it's time
to give them an honorable discharge.  So lets get rid of tasklets and
given them a standing salute as they leave :-)

I've booted these patches on 6 machines, i386, x86_64, and my
PowerPC Powerbook.

I'd like to give thanks to Ingo Molnar and Oleg Nesterov for reviewing 
my initial patch series and giving me some pointers.

Also thanks go to (random order) Arjan van de Ven, Arnd Bergmann,
Christoph Hellwig, Linus Torvalds, Andrew Morton, Thomas Gleixner,
Daniel Walker, and others that have read the patches and offered
advice and criticism.


[1] www.wil.cx/matthew/lca2003/paper.pdf

[RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Steven Rostedt]

[-- Attachment #1: convert-rcu-tasklet-to-softirq.patch --]
[-- Type: text/plain, Size: 3016 bytes --]

I believe this was originally done by Dipankar Sarma. I pulled these
changes from the -rt kernel.

For better preformance, RCU should use a softirq instead of a
tasklet.

From: Dipankar Sarma <dipankar@in.ibm.com>
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux-2.6-test/include/linux/interrupt.h
===================================================================
--- linux-2.6-test.orig/include/linux/interrupt.h
+++ linux-2.6-test/include/linux/interrupt.h
@@ -245,6 +245,9 @@ enum
 #ifdef CONFIG_HIGH_RES_TIMERS
 	HRTIMER_SOFTIRQ,
 #endif
+	RCU_SOFTIRQ,	/* Preferable RCU should always be the last softirq */
+	/* Entries after this are ignored in split softirq mode */
+	MAX_SOFTIRQ,
 };
 
 /* softirq mask and active fields moved to irq_cpustat_t in
Index: linux-2.6-test/kernel/rcupdate.c
===================================================================
--- linux-2.6-test.orig/kernel/rcupdate.c
+++ linux-2.6-test/kernel/rcupdate.c
@@ -67,7 +67,6 @@ DEFINE_PER_CPU(struct rcu_data, rcu_data
 DEFINE_PER_CPU(struct rcu_data, rcu_bh_data) = { 0L };
 
 /* Fake initialization required by compiler */
-static DEFINE_PER_CPU(struct tasklet_struct, rcu_tasklet) = {NULL};
 static int blimit = 10;
 static int qhimark = 10000;
 static int qlowmark = 100;
@@ -253,7 +252,7 @@ static void rcu_do_batch(struct rcu_data
 	if (!rdp->donelist)
 		rdp->donetail = &rdp->donelist;
 	else
-		tasklet_schedule(&per_cpu(rcu_tasklet, rdp->cpu));
+		raise_softirq(RCU_SOFTIRQ);
 }
 
 /*
@@ -405,7 +404,6 @@ static void rcu_offline_cpu(int cpu)
 					&per_cpu(rcu_bh_data, cpu));
 	put_cpu_var(rcu_data);
 	put_cpu_var(rcu_bh_data);
-	tasklet_kill_immediate(&per_cpu(rcu_tasklet, cpu), cpu);
 }
 
 #else
@@ -417,7 +415,7 @@ static void rcu_offline_cpu(int cpu)
 #endif
 
 /*
- * This does the RCU processing work from tasklet context. 
+ * This does the RCU processing work from softirq context.
  */
 static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp,
 					struct rcu_data *rdp)
@@ -462,7 +460,7 @@ static void __rcu_process_callbacks(stru
 		rcu_do_batch(rdp);
 }
 
-static void rcu_process_callbacks(unsigned long unused)
+static void rcu_process_callbacks(struct softirq_action *unused)
 {
 	__rcu_process_callbacks(&rcu_ctrlblk, &__get_cpu_var(rcu_data));
 	__rcu_process_callbacks(&rcu_bh_ctrlblk, &__get_cpu_var(rcu_bh_data));
@@ -526,7 +524,7 @@ void rcu_check_callbacks(int cpu, int us
 		rcu_bh_qsctr_inc(cpu);
 	} else if (!in_softirq())
 		rcu_bh_qsctr_inc(cpu);
-	tasklet_schedule(&per_cpu(rcu_tasklet, cpu));
+	raise_softirq(RCU_SOFTIRQ);
 }
 
 static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp,
@@ -549,7 +547,7 @@ static void __devinit rcu_online_cpu(int
 
 	rcu_init_percpu_data(cpu, &rcu_ctrlblk, rdp);
 	rcu_init_percpu_data(cpu, &rcu_bh_ctrlblk, bh_rdp);
-	tasklet_init(&per_cpu(rcu_tasklet, cpu), rcu_process_callbacks, 0UL);
+	open_softirq(RCU_SOFTIRQ, rcu_process_callbacks, NULL);
 }
 
 static int __cpuinit rcu_cpu_notify(struct notifier_block *self,

-- 

[RFC PATCH 2/8] Split out tasklets from softirq.c

[Steven Rostedt]

[-- Attachment #1: split-out-tasklet.patch --]
[-- Type: text/plain, Size: 18098 bytes --]

Tasklets are really a separate entity from softirqs, so they
deserve their own file. Also this allows us to easily replace
tasklets for something else ;-)

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux-2.6-test/include/linux/interrupt.h
===================================================================
--- linux-2.6-test.orig/include/linux/interrupt.h
+++ linux-2.6-test/include/linux/interrupt.h
@@ -13,6 +13,7 @@
 #include <linux/irqflags.h>
 #include <linux/bottom_half.h>
 #include <linux/device.h>
+#include <linux/tasklet.h>
 #include <asm/atomic.h>
 #include <asm/ptrace.h>
 #include <asm/system.h>
@@ -268,117 +269,6 @@ extern void FASTCALL(raise_softirq_irqof
 extern void FASTCALL(raise_softirq(unsigned int nr));
 
 
-/* Tasklets --- multithreaded analogue of BHs.
-
-   Main feature differing them of generic softirqs: tasklet
-   is running only on one CPU simultaneously.
-
-   Main feature differing them of BHs: different tasklets
-   may be run simultaneously on different CPUs.
-
-   Properties:
-   * If tasklet_schedule() is called, then tasklet is guaranteed
-     to be executed on some cpu at least once after this.
-   * If the tasklet is already scheduled, but its excecution is still not
-     started, it will be executed only once.
-   * If this tasklet is already running on another CPU (or schedule is called
-     from tasklet itself), it is rescheduled for later.
-   * Tasklet is strictly serialized wrt itself, but not
-     wrt another tasklets. If client needs some intertask synchronization,
-     he makes it with spinlocks.
- */
-
-struct tasklet_struct
-{
-	struct tasklet_struct *next;
-	unsigned long state;
-	atomic_t count;
-	void (*func)(unsigned long);
-	unsigned long data;
-};
-
-#define DECLARE_TASKLET(name, func, data) \
-struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(0), func, data }
-
-#define DECLARE_TASKLET_DISABLED(name, func, data) \
-struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(1), func, data }
-
-
-enum
-{
-	TASKLET_STATE_SCHED,	/* Tasklet is scheduled for execution */
-	TASKLET_STATE_RUN	/* Tasklet is running (SMP only) */
-};
-
-#ifdef CONFIG_SMP
-static inline int tasklet_trylock(struct tasklet_struct *t)
-{
-	return !test_and_set_bit(TASKLET_STATE_RUN, &(t)->state);
-}
-
-static inline void tasklet_unlock(struct tasklet_struct *t)
-{
-	smp_mb__before_clear_bit(); 
-	clear_bit(TASKLET_STATE_RUN, &(t)->state);
-}
-
-static inline void tasklet_unlock_wait(struct tasklet_struct *t)
-{
-	while (test_bit(TASKLET_STATE_RUN, &(t)->state)) { barrier(); }
-}
-#else
-#define tasklet_trylock(t) 1
-#define tasklet_unlock_wait(t) do { } while (0)
-#define tasklet_unlock(t) do { } while (0)
-#endif
-
-extern void FASTCALL(__tasklet_schedule(struct tasklet_struct *t));
-
-static inline void tasklet_schedule(struct tasklet_struct *t)
-{
-	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
-		__tasklet_schedule(t);
-}
-
-extern void FASTCALL(__tasklet_hi_schedule(struct tasklet_struct *t));
-
-static inline void tasklet_hi_schedule(struct tasklet_struct *t)
-{
-	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
-		__tasklet_hi_schedule(t);
-}
-
-
-static inline void tasklet_disable_nosync(struct tasklet_struct *t)
-{
-	atomic_inc(&t->count);
-	smp_mb__after_atomic_inc();
-}
-
-static inline void tasklet_disable(struct tasklet_struct *t)
-{
-	tasklet_disable_nosync(t);
-	tasklet_unlock_wait(t);
-	smp_mb();
-}
-
-static inline void tasklet_enable(struct tasklet_struct *t)
-{
-	smp_mb__before_atomic_dec();
-	atomic_dec(&t->count);
-}
-
-static inline void tasklet_hi_enable(struct tasklet_struct *t)
-{
-	smp_mb__before_atomic_dec();
-	atomic_dec(&t->count);
-}
-
-extern void tasklet_kill(struct tasklet_struct *t);
-extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu);
-extern void tasklet_init(struct tasklet_struct *t,
-			 void (*func)(unsigned long), unsigned long data);
-
 /*
  * Autoprobing for irqs:
  *
Index: linux-2.6-test/include/linux/tasklet.h
===================================================================
--- /dev/null
+++ linux-2.6-test/include/linux/tasklet.h
@@ -0,0 +1,120 @@
+#ifndef _LINUX_TASKLET_H
+#define _LINUX_TASKLET_H
+
+/* Tasklets --- multithreaded analogue of BHs.
+
+   Main feature differing them of generic softirqs: tasklet
+   is running only on one CPU simultaneously.
+
+   Main feature differing them of BHs: different tasklets
+   may be run simultaneously on different CPUs.
+
+   Properties:
+   * If tasklet_schedule() is called, then tasklet is guaranteed
+     to be executed on some cpu at least once after this.
+   * If the tasklet is already scheduled, but its excecution is still not
+     started, it will be executed only once.
+   * If this tasklet is already running on another CPU (or schedule is called
+     from tasklet itself), it is rescheduled for later.
+   * Tasklet is strictly serialized wrt itself, but not
+     wrt another tasklets. If client needs some intertask synchronization,
+     he makes it with spinlocks.
+ */
+
+struct tasklet_struct
+{
+	struct tasklet_struct *next;
+	unsigned long state;
+	atomic_t count;
+	void (*func)(unsigned long);
+	unsigned long data;
+};
+
+#define DECLARE_TASKLET(name, func, data) \
+struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(0), func, data }
+
+#define DECLARE_TASKLET_DISABLED(name, func, data) \
+struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(1), func, data }
+
+
+enum
+{
+	TASKLET_STATE_SCHED,	/* Tasklet is scheduled for execution */
+	TASKLET_STATE_RUN	/* Tasklet is running (SMP only) */
+};
+
+#ifdef CONFIG_SMP
+static inline int tasklet_trylock(struct tasklet_struct *t)
+{
+	return !test_and_set_bit(TASKLET_STATE_RUN, &(t)->state);
+}
+
+static inline void tasklet_unlock(struct tasklet_struct *t)
+{
+	smp_mb__before_clear_bit(); 
+	clear_bit(TASKLET_STATE_RUN, &(t)->state);
+}
+
+static inline void tasklet_unlock_wait(struct tasklet_struct *t)
+{
+	while (test_bit(TASKLET_STATE_RUN, &(t)->state)) { barrier(); }
+}
+#else
+#define tasklet_trylock(t) 1
+#define tasklet_unlock_wait(t) do { } while (0)
+#define tasklet_unlock(t) do { } while (0)
+#endif
+
+extern void FASTCALL(__tasklet_schedule(struct tasklet_struct *t));
+
+static inline void tasklet_schedule(struct tasklet_struct *t)
+{
+	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
+		__tasklet_schedule(t);
+}
+
+extern void FASTCALL(__tasklet_hi_schedule(struct tasklet_struct *t));
+
+static inline void tasklet_hi_schedule(struct tasklet_struct *t)
+{
+	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
+		__tasklet_hi_schedule(t);
+}
+
+
+static inline void tasklet_disable_nosync(struct tasklet_struct *t)
+{
+	atomic_inc(&t->count);
+	smp_mb__after_atomic_inc();
+}
+
+static inline void tasklet_disable(struct tasklet_struct *t)
+{
+	tasklet_disable_nosync(t);
+	tasklet_unlock_wait(t);
+	smp_mb();
+}
+
+static inline void tasklet_enable(struct tasklet_struct *t)
+{
+	smp_mb__before_atomic_dec();
+	atomic_dec(&t->count);
+}
+
+static inline void tasklet_hi_enable(struct tasklet_struct *t)
+{
+	smp_mb__before_atomic_dec();
+	atomic_dec(&t->count);
+}
+
+extern void tasklet_kill(struct tasklet_struct *t);
+extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu);
+extern void tasklet_init(struct tasklet_struct *t,
+			 void (*func)(unsigned long), unsigned long data);
+
+#ifdef CONFIG_HOTPLUG_CPU
+void takeover_tasklets(unsigned int cpu);
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#endif /* _LINUX_TASKLET_H */
+
Index: linux-2.6-test/kernel/Makefile
===================================================================
--- linux-2.6-test.orig/kernel/Makefile
+++ linux-2.6-test/kernel/Makefile
@@ -21,6 +21,7 @@ obj-$(CONFIG_FUTEX) += futex.o
 ifeq ($(CONFIG_COMPAT),y)
 obj-$(CONFIG_FUTEX) += futex_compat.o
 endif
+obj-y += tasklet.o
 obj-$(CONFIG_RT_MUTEXES) += rtmutex.o
 obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o
 obj-$(CONFIG_RT_MUTEX_TESTER) += rtmutex-tester.o
Index: linux-2.6-test/kernel/softirq.c
===================================================================
--- linux-2.6-test.orig/kernel/softirq.c
+++ linux-2.6-test/kernel/softirq.c
@@ -348,144 +348,6 @@ void open_softirq(int nr, void (*action)
 	softirq_vec[nr].action = action;
 }
 
-/* Tasklets */
-struct tasklet_head
-{
-	struct tasklet_struct *list;
-};
-
-/* Some compilers disobey section attribute on statics when not
-   initialized -- RR */
-static DEFINE_PER_CPU(struct tasklet_head, tasklet_vec) = { NULL };
-static DEFINE_PER_CPU(struct tasklet_head, tasklet_hi_vec) = { NULL };
-
-void fastcall __tasklet_schedule(struct tasklet_struct *t)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	t->next = __get_cpu_var(tasklet_vec).list;
-	__get_cpu_var(tasklet_vec).list = t;
-	raise_softirq_irqoff(TASKLET_SOFTIRQ);
-	local_irq_restore(flags);
-}
-
-EXPORT_SYMBOL(__tasklet_schedule);
-
-void fastcall __tasklet_hi_schedule(struct tasklet_struct *t)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	t->next = __get_cpu_var(tasklet_hi_vec).list;
-	__get_cpu_var(tasklet_hi_vec).list = t;
-	raise_softirq_irqoff(HI_SOFTIRQ);
-	local_irq_restore(flags);
-}
-
-EXPORT_SYMBOL(__tasklet_hi_schedule);
-
-static void tasklet_action(struct softirq_action *a)
-{
-	struct tasklet_struct *list;
-
-	local_irq_disable();
-	list = __get_cpu_var(tasklet_vec).list;
-	__get_cpu_var(tasklet_vec).list = NULL;
-	local_irq_enable();
-
-	while (list) {
-		struct tasklet_struct *t = list;
-
-		list = list->next;
-
-		if (tasklet_trylock(t)) {
-			if (!atomic_read(&t->count)) {
-				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
-					BUG();
-				t->func(t->data);
-				tasklet_unlock(t);
-				continue;
-			}
-			tasklet_unlock(t);
-		}
-
-		local_irq_disable();
-		t->next = __get_cpu_var(tasklet_vec).list;
-		__get_cpu_var(tasklet_vec).list = t;
-		__raise_softirq_irqoff(TASKLET_SOFTIRQ);
-		local_irq_enable();
-	}
-}
-
-static void tasklet_hi_action(struct softirq_action *a)
-{
-	struct tasklet_struct *list;
-
-	local_irq_disable();
-	list = __get_cpu_var(tasklet_hi_vec).list;
-	__get_cpu_var(tasklet_hi_vec).list = NULL;
-	local_irq_enable();
-
-	while (list) {
-		struct tasklet_struct *t = list;
-
-		list = list->next;
-
-		if (tasklet_trylock(t)) {
-			if (!atomic_read(&t->count)) {
-				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
-					BUG();
-				t->func(t->data);
-				tasklet_unlock(t);
-				continue;
-			}
-			tasklet_unlock(t);
-		}
-
-		local_irq_disable();
-		t->next = __get_cpu_var(tasklet_hi_vec).list;
-		__get_cpu_var(tasklet_hi_vec).list = t;
-		__raise_softirq_irqoff(HI_SOFTIRQ);
-		local_irq_enable();
-	}
-}
-
-
-void tasklet_init(struct tasklet_struct *t,
-		  void (*func)(unsigned long), unsigned long data)
-{
-	t->next = NULL;
-	t->state = 0;
-	atomic_set(&t->count, 0);
-	t->func = func;
-	t->data = data;
-}
-
-EXPORT_SYMBOL(tasklet_init);
-
-void tasklet_kill(struct tasklet_struct *t)
-{
-	if (in_interrupt())
-		printk("Attempt to kill tasklet from interrupt\n");
-
-	while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
-		do
-			yield();
-		while (test_bit(TASKLET_STATE_SCHED, &t->state));
-	}
-	tasklet_unlock_wait(t);
-	clear_bit(TASKLET_STATE_SCHED, &t->state);
-}
-
-EXPORT_SYMBOL(tasklet_kill);
-
-void __init softirq_init(void)
-{
-	open_softirq(TASKLET_SOFTIRQ, tasklet_action, NULL);
-	open_softirq(HI_SOFTIRQ, tasklet_hi_action, NULL);
-}
-
 static int ksoftirqd(void * __bind_cpu)
 {
 	set_user_nice(current, 19);
@@ -532,58 +394,6 @@ wait_to_die:
 	return 0;
 }
 
-#ifdef CONFIG_HOTPLUG_CPU
-/*
- * tasklet_kill_immediate is called to remove a tasklet which can already be
- * scheduled for execution on @cpu.
- *
- * Unlike tasklet_kill, this function removes the tasklet
- * _immediately_, even if the tasklet is in TASKLET_STATE_SCHED state.
- *
- * When this function is called, @cpu must be in the CPU_DEAD state.
- */
-void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu)
-{
-	struct tasklet_struct **i;
-
-	BUG_ON(cpu_online(cpu));
-	BUG_ON(test_bit(TASKLET_STATE_RUN, &t->state));
-
-	if (!test_bit(TASKLET_STATE_SCHED, &t->state))
-		return;
-
-	/* CPU is dead, so no lock needed. */
-	for (i = &per_cpu(tasklet_vec, cpu).list; *i; i = &(*i)->next) {
-		if (*i == t) {
-			*i = t->next;
-			return;
-		}
-	}
-	BUG();
-}
-
-static void takeover_tasklets(unsigned int cpu)
-{
-	struct tasklet_struct **i;
-
-	/* CPU is dead, so no lock needed. */
-	local_irq_disable();
-
-	/* Find end, append list for that CPU. */
-	for (i = &__get_cpu_var(tasklet_vec).list; *i; i = &(*i)->next);
-	*i = per_cpu(tasklet_vec, cpu).list;
-	per_cpu(tasklet_vec, cpu).list = NULL;
-	raise_softirq_irqoff(TASKLET_SOFTIRQ);
-
-	for (i = &__get_cpu_var(tasklet_hi_vec).list; *i; i = &(*i)->next);
-	*i = per_cpu(tasklet_hi_vec, cpu).list;
-	per_cpu(tasklet_hi_vec, cpu).list = NULL;
-	raise_softirq_irqoff(HI_SOFTIRQ);
-
-	local_irq_enable();
-}
-#endif /* CONFIG_HOTPLUG_CPU */
-
 static int __cpuinit cpu_callback(struct notifier_block *nfb,
 				  unsigned long action,
 				  void *hcpu)
Index: linux-2.6-test/kernel/tasklet.c
===================================================================
--- /dev/null
+++ linux-2.6-test/kernel/tasklet.c
@@ -0,0 +1,203 @@
+/*
+ *	linux/kernel/softirq.c
+ *
+ *	Copyright (C) 1992 Linus Torvalds
+ *
+ * Rewritten. Old one was good in 2.2, but in 2.3 it was immoral. --ANK (990903)
+ *
+ *      Removed from softirq.c by Steven Rostedt
+ */
+#include <linux/interrupt.h>
+#include <linux/cpu.h>
+
+/* Tasklets */
+struct tasklet_head
+{
+	struct tasklet_struct *list;
+};
+
+/* Some compilers disobey section attribute on statics when not
+   initialized -- RR */
+static DEFINE_PER_CPU(struct tasklet_head, tasklet_vec) = { NULL };
+static DEFINE_PER_CPU(struct tasklet_head, tasklet_hi_vec) = { NULL };
+
+void fastcall __tasklet_schedule(struct tasklet_struct *t)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	t->next = __get_cpu_var(tasklet_vec).list;
+	__get_cpu_var(tasklet_vec).list = t;
+	raise_softirq_irqoff(TASKLET_SOFTIRQ);
+	local_irq_restore(flags);
+}
+
+EXPORT_SYMBOL(__tasklet_schedule);
+
+void fastcall __tasklet_hi_schedule(struct tasklet_struct *t)
+{
+	unsigned long flags;
+
+	local_irq_save(flags);
+	t->next = __get_cpu_var(tasklet_hi_vec).list;
+	__get_cpu_var(tasklet_hi_vec).list = t;
+	raise_softirq_irqoff(HI_SOFTIRQ);
+	local_irq_restore(flags);
+}
+
+EXPORT_SYMBOL(__tasklet_hi_schedule);
+
+static void tasklet_action(struct softirq_action *a)
+{
+	struct tasklet_struct *list;
+
+	local_irq_disable();
+	list = __get_cpu_var(tasklet_vec).list;
+	__get_cpu_var(tasklet_vec).list = NULL;
+	local_irq_enable();
+
+	while (list) {
+		struct tasklet_struct *t = list;
+
+		list = list->next;
+
+		if (tasklet_trylock(t)) {
+			if (!atomic_read(&t->count)) {
+				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
+					BUG();
+				t->func(t->data);
+				tasklet_unlock(t);
+				continue;
+			}
+			tasklet_unlock(t);
+		}
+
+		local_irq_disable();
+		t->next = __get_cpu_var(tasklet_vec).list;
+		__get_cpu_var(tasklet_vec).list = t;
+		__raise_softirq_irqoff(TASKLET_SOFTIRQ);
+		local_irq_enable();
+	}
+}
+
+static void tasklet_hi_action(struct softirq_action *a)
+{
+	struct tasklet_struct *list;
+
+	local_irq_disable();
+	list = __get_cpu_var(tasklet_hi_vec).list;
+	__get_cpu_var(tasklet_hi_vec).list = NULL;
+	local_irq_enable();
+
+	while (list) {
+		struct tasklet_struct *t = list;
+
+		list = list->next;
+
+		if (tasklet_trylock(t)) {
+			if (!atomic_read(&t->count)) {
+				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
+					BUG();
+				t->func(t->data);
+				tasklet_unlock(t);
+				continue;
+			}
+			tasklet_unlock(t);
+		}
+
+		local_irq_disable();
+		t->next = __get_cpu_var(tasklet_hi_vec).list;
+		__get_cpu_var(tasklet_hi_vec).list = t;
+		__raise_softirq_irqoff(HI_SOFTIRQ);
+		local_irq_enable();
+	}
+}
+
+
+void tasklet_init(struct tasklet_struct *t,
+		  void (*func)(unsigned long), unsigned long data)
+{
+	t->next = NULL;
+	t->state = 0;
+	atomic_set(&t->count, 0);
+	t->func = func;
+	t->data = data;
+}
+
+EXPORT_SYMBOL(tasklet_init);
+
+void tasklet_kill(struct tasklet_struct *t)
+{
+	if (in_interrupt())
+		printk("Attempt to kill tasklet from interrupt\n");
+
+	while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
+		do
+			yield();
+		while (test_bit(TASKLET_STATE_SCHED, &t->state));
+	}
+	tasklet_unlock_wait(t);
+	clear_bit(TASKLET_STATE_SCHED, &t->state);
+}
+
+EXPORT_SYMBOL(tasklet_kill);
+
+void __init softirq_init(void)
+{
+	open_softirq(TASKLET_SOFTIRQ, tasklet_action, NULL);
+	open_softirq(HI_SOFTIRQ, tasklet_hi_action, NULL);
+}
+
+#ifdef CONFIG_HOTPLUG_CPU
+/*
+ * tasklet_kill_immediate is called to remove a tasklet which can already be
+ * scheduled for execution on @cpu.
+ *
+ * Unlike tasklet_kill, this function removes the tasklet
+ * _immediately_, even if the tasklet is in TASKLET_STATE_SCHED state.
+ *
+ * When this function is called, @cpu must be in the CPU_DEAD state.
+ */
+void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu)
+{
+	struct tasklet_struct **i;
+
+	BUG_ON(cpu_online(cpu));
+	BUG_ON(test_bit(TASKLET_STATE_RUN, &t->state));
+
+	if (!test_bit(TASKLET_STATE_SCHED, &t->state))
+		return;
+
+	/* CPU is dead, so no lock needed. */
+	for (i = &per_cpu(tasklet_vec, cpu).list; *i; i = &(*i)->next) {
+		if (*i == t) {
+			*i = t->next;
+			return;
+		}
+	}
+	BUG();
+}
+
+void takeover_tasklets(unsigned int cpu)
+{
+	struct tasklet_struct **i;
+
+	/* CPU is dead, so no lock needed. */
+	local_irq_disable();
+
+	/* Find end, append list for that CPU. */
+	for (i = &__get_cpu_var(tasklet_vec).list; *i; i = &(*i)->next);
+	*i = per_cpu(tasklet_vec, cpu).list;
+	per_cpu(tasklet_vec, cpu).list = NULL;
+	raise_softirq_irqoff(TASKLET_SOFTIRQ);
+
+	for (i = &__get_cpu_var(tasklet_hi_vec).list; *i; i = &(*i)->next);
+	*i = per_cpu(tasklet_hi_vec, cpu).list;
+	per_cpu(tasklet_hi_vec, cpu).list = NULL;
+	raise_softirq_irqoff(HI_SOFTIRQ);
+
+	local_irq_enable();
+}
+#endif /* CONFIG_HOTPLUG_CPU */
+
+

-- 

[RFC PATCH 3/8] Add a tasklet is-scheduled API

[Steven Rostedt]

[-- Attachment #1: tasklet-state-api.patch --]
[-- Type: text/plain, Size: 773 bytes --]

This patch adds a tasklet_is_scheduled API to allow a driver
to know if its tasklet is already scheduled.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>

Index: linux-2.6-test/include/linux/tasklet.h
===================================================================
--- linux-2.6-test.orig/include/linux/tasklet.h
+++ linux-2.6-test/include/linux/tasklet.h
@@ -107,6 +107,11 @@ static inline void tasklet_hi_enable(str
 	atomic_dec(&t->count);
 }
 
+static inline int tasklet_is_scheduled(struct tasklet_struct *t)
+{
+	return test_bit(TASKLET_STATE_SCHED, &t->state);
+}
+
 extern void tasklet_kill(struct tasklet_struct *t);
 extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu);
 extern void tasklet_init(struct tasklet_struct *t,

-- 

[RFC PATCH 4/8] Make DRM use the tasklet is-sched API

[Steven Rostedt]

[-- Attachment #1: tasklet-driver-hacks.patch --]
[-- Type: text/plain, Size: 737 bytes --]

Update the DRM driver to use the new tasklet API, which does not rely
on the tasklet implementation details.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux-2.6.21-rt9/drivers/char/drm/drm_irq.c
===================================================================
--- linux-2.6.21-rt9.orig/drivers/char/drm/drm_irq.c
+++ linux-2.6.21-rt9/drivers/char/drm/drm_irq.c
@@ -461,7 +461,7 @@ void drm_locked_tasklet(drm_device_t *de
 	static DECLARE_TASKLET(drm_tasklet, drm_locked_tasklet_func, 0);
 
 	if (!drm_core_check_feature(dev, DRIVER_HAVE_IRQ) ||
-	    test_bit(TASKLET_STATE_SCHED, &drm_tasklet.state))
+	    tasklet_is_scheduled(&drm_tasklet))
 		return;
 
 	spin_lock_irqsave(&dev->tasklet_lock, irqflags);

-- 

[RFC PATCH 5/8] Convert net wireless tasklet to workqueue

[Steven Rostedt]

[-- Attachment #1: convert-net-wireless-tasklet-workqueue.patch --]
[-- Type: text/plain, Size: 1332 bytes --]

Since we want to remove DECLARE_TASKLET API, there wasn't a simple way
to do that elegantly with net/core/wireless.c, so this patch just
replaces that tasklet with a work queue directly.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux-2.6-test/net/core/wireless.c
===================================================================
--- linux-2.6-test.orig/net/core/wireless.c
+++ linux-2.6-test/net/core/wireless.c
@@ -1916,7 +1916,7 @@ static int __init wireless_nlevent_init(
 
 subsys_initcall(wireless_nlevent_init);
 
-static void wireless_nlevent_process(unsigned long data)
+static void wireless_nlevent_process(struct work_struct *work)
 {
 	struct sk_buff *skb;
 
@@ -1924,7 +1924,7 @@ static void wireless_nlevent_process(uns
 		rtnl_notify(skb, 0, RTNLGRP_LINK, NULL, GFP_ATOMIC);
 }
 
-static DECLARE_TASKLET(wireless_nlevent_tasklet, wireless_nlevent_process, 0);
+static DECLARE_WORK(wireless_nlevent_wq, wireless_nlevent_process);
 
 /* ---------------------------------------------------------------- */
 /*
@@ -1989,7 +1989,7 @@ static inline void rtmsg_iwinfo(struct n
 	}
 	NETLINK_CB(skb).dst_group = RTNLGRP_LINK;
 	skb_queue_tail(&wireless_nlevent_queue, skb);
-	tasklet_schedule(&wireless_nlevent_tasklet);
+	schedule_work(&wireless_nlevent_wq);
 }
 
 #endif	/* WE_EVENT_RTNETLINK */

-- 

[RFC PATCH 6/8] Remove DECLARE_TASKLET_DISABLE API

[Steven Rostedt]

[-- Attachment #1: remove-declare-tasklet-disable.patch --]
[-- Type: text/plain, Size: 3149 bytes --]

There's three uses of DECLARE_TASKLET_DISABLE and this patch
removes those uses.  Objective is to remove the declaration completely.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux-2.6-test/drivers/input/serio/hil_mlc.c
===================================================================
--- linux-2.6-test.orig/drivers/input/serio/hil_mlc.c
+++ linux-2.6-test/drivers/input/serio/hil_mlc.c
@@ -76,7 +76,7 @@ static struct timer_list	hil_mlcs_kicker
 static int			hil_mlcs_probe;
 
 static void hil_mlcs_process(unsigned long unused);
-DECLARE_TASKLET_DISABLED(hil_mlcs_tasklet, hil_mlcs_process, 0);
+struct tasklet_struct hil_mlcs_tasklet;
 
 
 /* #define HIL_MLC_DEBUG */
@@ -933,12 +933,13 @@ int hil_mlc_unregister(hil_mlc *mlc) {
 
 static int __init hil_mlc_init(void)
 {
+	tasklet_init(&hil_mlcs_tasklet, hil_mlcs_process, 0);
+
 	init_timer(&hil_mlcs_kicker);
 	hil_mlcs_kicker.expires = jiffies + HZ;
 	hil_mlcs_kicker.function = &hil_mlcs_timer;
 	add_timer(&hil_mlcs_kicker);
 
-	tasklet_enable(&hil_mlcs_tasklet);
 
 	return 0;
 }
Index: linux-2.6-test/drivers/char/keyboard.c
===================================================================
--- linux-2.6-test.orig/drivers/char/keyboard.c
+++ linux-2.6-test/drivers/char/keyboard.c
@@ -1011,7 +1011,7 @@ static void kbd_bh(unsigned long dummy)
 	ledstate = leds;
 }
 
-DECLARE_TASKLET_DISABLED(keyboard_tasklet, kbd_bh, 0);
+struct tasklet_struct keyboard_tasklet;
 
 #if defined(CONFIG_X86) || defined(CONFIG_IA64) || defined(CONFIG_ALPHA) ||\
     defined(CONFIG_MIPS) || defined(CONFIG_PPC) || defined(CONFIG_SPARC) ||\
@@ -1376,11 +1376,12 @@ int __init kbd_init(void)
 		kbd_table[i].kbdmode = VC_XLATE;
 	}
 
+	tasklet_init(&keyboard_tasklet, kbd_bh, 0);
+
 	error = input_register_handler(&kbd_handler);
 	if (error)
 		return error;
 
-	tasklet_enable(&keyboard_tasklet);
 	tasklet_schedule(&keyboard_tasklet);
 
 	return 0;
Index: linux-2.6-test/net/tipc/handler.c
===================================================================
--- linux-2.6-test.orig/net/tipc/handler.c
+++ linux-2.6-test/net/tipc/handler.c
@@ -49,7 +49,7 @@ static int handler_enabled = 0;
 
 static void process_signal_queue(unsigned long dummy);
 
-static DECLARE_TASKLET_DISABLED(tipc_tasklet, process_signal_queue, 0);
+static struct tasklet_struct tipc_tasklet;
 
 
 unsigned int tipc_k_signal(Handler routine, unsigned long argument)
@@ -102,7 +102,7 @@ int tipc_handler_start(void)
 		return -ENOMEM;
 
 	INIT_LIST_HEAD(&signal_queue_head);
-	tasklet_enable(&tipc_tasklet);
+	tasklet_init(&tipc_tasklet, process_signal_queue, 0);
 	handler_enabled = 1;
 	return 0;
 }
Index: linux-2.6-test/include/linux/tasklet.h
===================================================================
--- linux-2.6-test.orig/include/linux/tasklet.h
+++ linux-2.6-test/include/linux/tasklet.h
@@ -33,9 +33,6 @@ struct tasklet_struct
 #define DECLARE_TASKLET(name, func, data) \
 struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(0), func, data }
 
-#define DECLARE_TASKLET_DISABLED(name, func, data) \
-struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(1), func, data }
-
 
 enum
 {

-- 

[RFC PATCH 7/8] remove DECLARE_TASKLET API

[Steven Rostedt]

[-- Attachment #1: remove-declare-tasklet.patch --]
[-- Type: text/plain, Size: 12280 bytes --]

This patch replaces the DECLARE_TASKLET for tasklet_init.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux-2.6-test/arch/cris/arch-v32/drivers/cryptocop.c
===================================================================
--- linux-2.6-test.orig/arch/cris/arch-v32/drivers/cryptocop.c
+++ linux-2.6-test/arch/cris/arch-v32/drivers/cryptocop.c
@@ -1849,7 +1849,7 @@ static int cryptocop_job_queue_insert(cr
 }
 
 static void cryptocop_do_tasklet(unsigned long unused);
-DECLARE_TASKLET (cryptocop_tasklet, cryptocop_do_tasklet, 0);
+struct tasklet_struct cryptocop_tasklet;
 
 static void cryptocop_do_tasklet(unsigned long unused)
 {
@@ -3467,6 +3467,8 @@ static int init_stream_coprocessor(void)
 
 	printk("ETRAX FS stream co-processor driver v0.01, (c) 2003 Axis Communications AB\n");
 
+	tasklet_init(&cryptocop_tasklet, cryptocop_do_tasklet, 0);
+
 	err = register_chrdev(CRYPTOCOP_MAJOR, cryptocop_name, &cryptocop_fops);
 	if (err < 0) {
 		printk(KERN_ERR "stream co-processor: could not get major number.\n");
Index: linux-2.6-test/arch/ia64/hp/sim/simscsi.c
===================================================================
--- linux-2.6-test.orig/arch/ia64/hp/sim/simscsi.c
+++ linux-2.6-test/arch/ia64/hp/sim/simscsi.c
@@ -47,7 +47,7 @@
 static struct Scsi_Host *host;
 
 static void simscsi_interrupt (unsigned long val);
-static DECLARE_TASKLET(simscsi_tasklet, simscsi_interrupt, 0);
+static struct tasklet_struct simscsi_tasklet;
 
 struct disk_req {
 	unsigned long addr;
@@ -404,6 +404,8 @@ simscsi_init(void)
 {
 	int error;
 
+	tasklet_init(&simscsi_tasklet, simscsi_interrupt, 0);
+
 	host = scsi_host_alloc(&driver_template, 0);
 	if (!host)
 		return -ENOMEM;
Index: linux-2.6-test/arch/mips/lasat/picvue_proc.c
===================================================================
--- linux-2.6-test.orig/arch/mips/lasat/picvue_proc.c
+++ linux-2.6-test/arch/mips/lasat/picvue_proc.c
@@ -33,7 +33,7 @@ static void pvc_display(unsigned long da
 		pvc_write_string(pvc_lines[i], 0, i);
 }
 
-static DECLARE_TASKLET(pvc_display_tasklet, &pvc_display, 0);
+static struct tasklet_struct pvc_display_tasklet;
 
 static int pvc_proc_read_line(char *page, char **start,
                              off_t off, int count,
@@ -172,6 +172,8 @@ static int __init pvc_proc_init(void)
 	proc_entry->write_proc = pvc_proc_write_scroll;
 	proc_entry->read_proc = pvc_proc_read_scroll;
 
+	tasklet_init(&pvc_display_tasklet, &pvc_display, 0);
+
 	init_timer(&timer);
 	timer.function = pvc_proc_timerfunc;
 
Index: linux-2.6-test/drivers/cdrom/cm206.c
===================================================================
--- linux-2.6-test.orig/drivers/cdrom/cm206.c
+++ linux-2.6-test/drivers/cdrom/cm206.c
@@ -740,8 +740,6 @@ static void cm206_tasklet_func(unsigned 
 	}
 }
 
-static DECLARE_TASKLET(cm206_tasklet, cm206_tasklet_func, 0);
-
 /* This command clears the dsb_possible_media_change flag, so we must 
  * retain it.
  */
@@ -1413,6 +1411,9 @@ int __init cm206_init(void)
 	struct gendisk *disk;
 
 	printk(KERN_INFO "cm206 cdrom driver " REVISION);
+
+	tasklet_init(&cm206_tasklet, cm206_tasklet_func, 0);
+
 	cm206_base = probe_base_port(auto_probe ? 0 : cm206_base);
 	if (!cm206_base) {
 		printk(" can't find adapter!\n");
Index: linux-2.6-test/drivers/char/drm/drm_irq.c
===================================================================
--- linux-2.6-test.orig/drivers/char/drm/drm_irq.c
+++ linux-2.6-test/drivers/char/drm/drm_irq.c
@@ -458,7 +458,13 @@ static void drm_locked_tasklet_func(unsi
 void drm_locked_tasklet(drm_device_t *dev, void (*func)(drm_device_t*))
 {
 	unsigned long irqflags;
-	static DECLARE_TASKLET(drm_tasklet, drm_locked_tasklet_func, 0);
+	static struct tasklet_struct drm_tasklet;
+	static int drm_tasklet_init;
+
+	if (!drm_tasklet_init) {
+		tasklet_init(&drm_tasklet, drm_locked_tasklet_func, 0);
+		drm_tasklet_init = 1;
+	}
 
 	if (!drm_core_check_feature(dev, DRIVER_HAVE_IRQ) ||
 	    tasklet_is_scheduled(&drm_tasklet))
Index: linux-2.6-test/drivers/char/snsc_event.c
===================================================================
--- linux-2.6-test.orig/drivers/char/snsc_event.c
+++ linux-2.6-test/drivers/char/snsc_event.c
@@ -25,7 +25,7 @@
 static struct subch_data_s *event_sd;
 
 void scdrv_event(unsigned long);
-DECLARE_TASKLET(sn_sysctl_event, scdrv_event, 0);
+struct tasklet_struct sn_sysctl_event;
 
 /*
  * scdrv_event_interrupt
@@ -285,6 +285,9 @@ scdrv_event_init(struct sysctl_data_s *s
 	event_sd->sd_nasid = scd->scd_nasid;
 	spin_lock_init(&event_sd->sd_rlock);
 
+
+	tasklet_init(&sn_sysctl_event, scdrv_event, 0);
+
 	/* ask the system controllers to send events to this node */
 	event_sd->sd_subch = ia64_sn_sysctl_event_init(scd->scd_nasid);
 
Index: linux-2.6-test/drivers/media/video/vino.c
===================================================================
--- linux-2.6-test.orig/drivers/media/video/vino.c
+++ linux-2.6-test/drivers/media/video/vino.c
@@ -355,8 +355,8 @@ static const char *vino_v4l_device_name_
 
 static void vino_capture_tasklet(unsigned long channel);
 
-DECLARE_TASKLET(vino_tasklet_a, vino_capture_tasklet, VINO_CHANNEL_A);
-DECLARE_TASKLET(vino_tasklet_b, vino_capture_tasklet, VINO_CHANNEL_B);
+struct tasklet_struct vino_tasklet_a;
+struct tasklet_struct vino_tasklet_b;
 
 static const struct vino_input vino_inputs[] = {
 	{
@@ -4605,6 +4605,9 @@ static int __init vino_module_init(void)
 
 	/* initialize data structures */
 
+	tasklet_init(&vino_tasklet_a, vino_capture_tasklet, VINO_CHANNEL_A);
+	tasklet_init(&vino_tasklet_b, vino_capture_tasklet, VINO_CHANNEL_B);
+
 	spin_lock_init(&vino_drvdata->vino_lock);
 	spin_lock_init(&vino_drvdata->input_lock);
 
Index: linux-2.6-test/drivers/net/wan/farsync.c
===================================================================
--- linux-2.6-test.orig/drivers/net/wan/farsync.c
+++ linux-2.6-test/drivers/net/wan/farsync.c
@@ -572,8 +572,8 @@ static void do_bottom_half_rx(struct fst
 static void fst_process_tx_work_q(unsigned long work_q);
 static void fst_process_int_work_q(unsigned long work_q);
 
-static DECLARE_TASKLET(fst_tx_task, fst_process_tx_work_q, 0);
-static DECLARE_TASKLET(fst_int_task, fst_process_int_work_q, 0);
+struct tasklet_struct fst_tx_task;
+struct tasklet_struct fst_int_task;
 
 static struct fst_card_info *fst_card_array[FST_MAX_CARDS];
 static spinlock_t fst_work_q_lock;
@@ -2697,6 +2697,8 @@ fst_init(void)
 	for (i = 0; i < FST_MAX_CARDS; i++)
 		fst_card_array[i] = NULL;
 	spin_lock_init(&fst_work_q_lock);
+	tasklet_init(&fst_tx_task, fst_process_tx_work_q, 0);
+	tasklet_init(&fst_int_task, fst_process_int_work_q, 0);
 	return pci_register_driver(&fst_driver);
 }
 
Index: linux-2.6-test/drivers/s390/cio/qdio.c
===================================================================
--- linux-2.6-test.orig/drivers/s390/cio/qdio.c
+++ linux-2.6-test/drivers/s390/cio/qdio.c
@@ -98,7 +98,7 @@ static volatile struct qdio_q *tiq_list=
 static DEFINE_SPINLOCK(ttiq_list_lock);
 static int register_thinint_result;
 static void tiqdio_tl(unsigned long);
-static DECLARE_TASKLET(tiqdio_tasklet,tiqdio_tl,0);
+static struct tasklet_struct tiqdio_tasklet;
 
 /* not a macro, as one of the arguments is atomic_read */
 static inline int 
@@ -3839,6 +3839,8 @@ init_QDIO(void)
 
 	printk("qdio: loading %s\n",version);
 
+	tasklet_init(&tiqdio_tasklet, tiqdio_tl, 0);
+
 	res=qdio_get_qdio_memory();
 	if (res)
 		return res;
Index: linux-2.6-test/drivers/s390/crypto/ap_bus.c
===================================================================
--- linux-2.6-test.orig/drivers/s390/crypto/ap_bus.c
+++ linux-2.6-test/drivers/s390/crypto/ap_bus.c
@@ -80,7 +80,7 @@ static DECLARE_WORK(ap_config_work, ap_s
  * Tasklet & timer for AP request polling.
  */
 static struct timer_list ap_poll_timer = TIMER_INITIALIZER(ap_poll_timeout,0,0);
-static DECLARE_TASKLET(ap_tasklet, ap_poll_all, 0);
+static struct tasklet_struct ap_tasklet;
 static atomic_t ap_poll_requests = ATOMIC_INIT(0);
 static DECLARE_WAIT_QUEUE_HEAD(ap_poll_wait);
 static struct task_struct *ap_poll_kthread = NULL;
@@ -1188,6 +1188,8 @@ int __init ap_module_init(void)
 	}
 	register_reset_call(&ap_reset_call);
 
+	tasklet_init(&ap_tasklet, ap_poll_all, 0);
+
 	/* Create /sys/bus/ap. */
 	rc = bus_register(&ap_bus_type);
 	if (rc)
Index: linux-2.6-test/drivers/usb/gadget/net2280.c
===================================================================
--- linux-2.6-test.orig/drivers/usb/gadget/net2280.c
+++ linux-2.6-test/drivers/usb/gadget/net2280.c
@@ -511,7 +511,7 @@ static void do_free(unsigned long ignore
 	spin_unlock_irq(&buflock);
 }
 
-static DECLARE_TASKLET(deferred_free, do_free, 0);
+static struct tasklet_struct deferred_free;
 
 static void
 net2280_free_buffer (
@@ -3051,6 +3051,9 @@ static int __init init (void)
 {
 	if (!use_dma)
 		use_dma_chaining = 0;
+
+	tasklet_init(&deferred_free, do_free, 0);
+
 	return pci_register_driver (&net2280_pci_driver);
 }
 module_init (init);
Index: linux-2.6-test/drivers/usb/gadget/omap_udc.c
===================================================================
--- linux-2.6-test.orig/drivers/usb/gadget/omap_udc.c
+++ linux-2.6-test/drivers/usb/gadget/omap_udc.c
@@ -365,7 +365,7 @@ static void do_free(unsigned long ignore
 	spin_unlock_irq(&buflock);
 }
 
-static DECLARE_TASKLET(deferred_free, do_free, 0);
+static struct tasklet_struct deferred_free;
 
 static void omap_free_buffer(
 	struct usb_ep	*_ep,
@@ -3182,6 +3182,9 @@ static int __init udc_init(void)
 #endif
 		"%s\n", driver_desc,
 		use_dma ?  " (dma)" : "");
+
+	tasklet_init(&deferred_free, do_free, 0);
+
 	return platform_driver_register(&udc_driver);
 }
 module_init(udc_init);
Index: linux-2.6-test/kernel/irq/resend.c
===================================================================
--- linux-2.6-test.orig/kernel/irq/resend.c
+++ linux-2.6-test/kernel/irq/resend.c
@@ -44,7 +44,7 @@ static void resend_irqs(unsigned long ar
 }
 
 /* Tasklet to handle resend: */
-static DECLARE_TASKLET(resend_tasklet, resend_irqs, 0);
+static struct tasklet_struct resend_tasklet;
 
 #endif
 
@@ -75,3 +75,12 @@ void check_irq_resend(struct irq_desc *d
 		}
 	}
 }
+
+#ifdef CONFIG_HARDIRQS_SW_RESEND
+static int __init irq_resend_init(void)
+{
+	tasklet_init(&resend_tasklet, resend_irqs, 0);
+}
+
+core_initcall(irq_resend_init);
+#endif
Index: linux-2.6-test/net/atm/pppoatm.c
===================================================================
--- linux-2.6-test.orig/net/atm/pppoatm.c
+++ linux-2.6-test/net/atm/pppoatm.c
@@ -281,7 +281,14 @@ static int pppoatm_assign_vcc(struct atm
 	 * Each PPPoATM instance has its own tasklet - this is just a
 	 * prototypical one used to initialize them
 	 */
-	static const DECLARE_TASKLET(tasklet_proto, pppoatm_wakeup_sender, 0);
+	static struct tasklet_struct tasklet_proto;
+	static int tasklet_proto_init;
+
+	if (!tasklet_proto_init) {
+		tasklet_init(&tasklet_proto, pppoatm_wakeup_sender, 0);
+		tasklet_proto_init = 1;
+	}
+
 	if (copy_from_user(&be, arg, sizeof be))
 		return -EFAULT;
 	if (be.encaps != PPPOATM_ENCAPS_AUTODETECT &&
Index: linux-2.6-test/net/iucv/iucv.c
===================================================================
--- linux-2.6-test.orig/net/iucv/iucv.c
+++ linux-2.6-test/net/iucv/iucv.c
@@ -103,7 +103,7 @@ static cpumask_t iucv_buffer_cpumask = C
 static cpumask_t iucv_irq_cpumask = CPU_MASK_NONE;
 
 static void iucv_tasklet_handler(unsigned long);
-static DECLARE_TASKLET(iucv_tasklet, iucv_tasklet_handler,0);
+static struct tasklet_struct iucv_tasklet;
 
 enum iucv_command_codes {
 	IUCV_QUERY = 0,
@@ -1525,6 +1525,7 @@ static int iucv_init(void)
 		rc = -EPROTONOSUPPORT;
 		goto out;
 	}
+	tasklet_init(&iucv_tasklet, iucv_tasklet_handler,0);
 	rc = iucv_query_maxconn();
 	if (rc)
 		goto out;
Index: linux-2.6-test/include/linux/tasklet.h
===================================================================
--- linux-2.6-test.orig/include/linux/tasklet.h
+++ linux-2.6-test/include/linux/tasklet.h
@@ -30,10 +30,6 @@ struct tasklet_struct
 	unsigned long data;
 };
 
-#define DECLARE_TASKLET(name, func, data) \
-struct tasklet_struct name = { NULL, 0, ATOMIC_INIT(0), func, data }
-
-
 enum
 {
 	TASKLET_STATE_SCHED,	/* Tasklet is scheduled for execution */

-- 

[RFC PATCH 8/8] Convert tasklets to work queues

[Steven Rostedt]

[-- Attachment #1: tasklets-to-workqueues.patch --]
[-- Type: text/plain, Size: 13695 bytes --]

This patch replaces the tasklet implementation with a work
queue implementation while keeping the tasklet API.
The API is still the same, and the drivers don't know that a
work queue is being used.

This is (for now) a proof of concept approach to using work queues
instead of tasklets.  More can be done. For one thing, we could make
individual work queues for each tasklet instead of using the global
ktasklet_wq.  But for now it's just easier to do this.

Signed-off-by: Steven Rostedt <rostedt@goodmis.org>


Index: linux-2.6-test/kernel/Makefile
===================================================================
--- linux-2.6-test.orig/kernel/Makefile
+++ linux-2.6-test/kernel/Makefile
@@ -21,7 +21,7 @@ obj-$(CONFIG_FUTEX) += futex.o
 ifeq ($(CONFIG_COMPAT),y)
 obj-$(CONFIG_FUTEX) += futex_compat.o
 endif
-obj-y += tasklet.o
+obj-y += tasklet_work.o
 obj-$(CONFIG_RT_MUTEXES) += rtmutex.o
 obj-$(CONFIG_DEBUG_RT_MUTEXES) += rtmutex-debug.o
 obj-$(CONFIG_RT_MUTEX_TESTER) += rtmutex-tester.o
Index: linux-2.6-test/init/main.c
===================================================================
--- linux-2.6-test.orig/init/main.c
+++ linux-2.6-test/init/main.c
@@ -120,7 +120,7 @@ EXPORT_SYMBOL(system_state);
 extern void time_init(void);
 /* Default late time init is NULL. archs can override this later. */
 void (*late_time_init)(void);
-extern void softirq_init(void);
+extern void init_tasklets(void);
 
 /* Untouched command line saved by arch-specific code. */
 char __initdata boot_command_line[COMMAND_LINE_SIZE];
@@ -563,7 +563,6 @@ asmlinkage void __init start_kernel(void
 	pidhash_init();
 	init_timers();
 	hrtimers_init();
-	softirq_init();
 	timekeeping_init();
 	time_init();
 	profile_init();
@@ -706,6 +705,7 @@ static void __init do_basic_setup(void)
 {
 	/* drivers will send hotplug events */
 	init_workqueues();
+	init_tasklets();
 	usermodehelper_init();
 	driver_init();
 	init_irq_proc();
Index: linux-2.6-test/include/linux/tasklet.h
===================================================================
--- linux-2.6-test.orig/include/linux/tasklet.h
+++ linux-2.6-test/include/linux/tasklet.h
@@ -1,79 +1,45 @@
-#ifndef _LINUX_TASKLET_H
-#define _LINUX_TASKLET_H
+#ifndef _LINUX_WORK_TASKLET_H
+#define _LINUX_WORK_TASKLET_H
 
-/* Tasklets --- multithreaded analogue of BHs.
+#include <linux/workqueue.h>
 
-   Main feature differing them of generic softirqs: tasklet
-   is running only on one CPU simultaneously.
+extern void work_tasklet_exec(struct work_struct *work);
 
-   Main feature differing them of BHs: different tasklets
-   may be run simultaneously on different CPUs.
-
-   Properties:
-   * If tasklet_schedule() is called, then tasklet is guaranteed
-     to be executed on some cpu at least once after this.
-   * If the tasklet is already scheduled, but its excecution is still not
-     started, it will be executed only once.
-   * If this tasklet is already running on another CPU (or schedule is called
-     from tasklet itself), it is rescheduled for later.
-   * Tasklet is strictly serialized wrt itself, but not
-     wrt another tasklets. If client needs some intertask synchronization,
-     he makes it with spinlocks.
- */
+enum
+{
+	TASKLET_STATE_PENDING	/* Tasklet is pending */
+};
+
+#define TASKLET_STATEF_PENDING	(1 << TASKLET_STATE_PENDING)
 
 struct tasklet_struct
 {
-	struct tasklet_struct *next;
+	struct work_struct work;
+	struct list_head list;
 	unsigned long state;
 	atomic_t count;
 	void (*func)(unsigned long);
 	unsigned long data;
+	char *n;
 };
 
-enum
-{
-	TASKLET_STATE_SCHED,	/* Tasklet is scheduled for execution */
-	TASKLET_STATE_RUN	/* Tasklet is running (SMP only) */
-};
-
-#ifdef CONFIG_SMP
-static inline int tasklet_trylock(struct tasklet_struct *t)
-{
-	return !test_and_set_bit(TASKLET_STATE_RUN, &(t)->state);
-}
-
-static inline void tasklet_unlock(struct tasklet_struct *t)
-{
-	smp_mb__before_clear_bit(); 
-	clear_bit(TASKLET_STATE_RUN, &(t)->state);
-}
-
-static inline void tasklet_unlock_wait(struct tasklet_struct *t)
-{
-	while (test_bit(TASKLET_STATE_RUN, &(t)->state)) { barrier(); }
-}
-#else
-#define tasklet_trylock(t) 1
-#define tasklet_unlock_wait(t) do { } while (0)
-#define tasklet_unlock(t) do { } while (0)
-#endif
-
-extern void FASTCALL(__tasklet_schedule(struct tasklet_struct *t));
+extern struct workqueue_struct *ktaskletd_wq;
 
 static inline void tasklet_schedule(struct tasklet_struct *t)
 {
-	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
-		__tasklet_schedule(t);
+	WARN_ON_ONCE(!ktaskletd_wq);
+	queue_work(ktaskletd_wq, &t->work);
 }
+#define tasklet_hi_schedule tasklet_schedule
 
-extern void FASTCALL(__tasklet_hi_schedule(struct tasklet_struct *t));
-
-static inline void tasklet_hi_schedule(struct tasklet_struct *t)
+static inline void fastcall tasklet_enable(struct tasklet_struct *t)
 {
-	if (!test_and_set_bit(TASKLET_STATE_SCHED, &t->state))
-		__tasklet_hi_schedule(t);
+	if (!atomic_dec_and_test(&t->count))
+		return;
+	if (test_and_clear_bit(TASKLET_STATE_PENDING, &t->state))
+		tasklet_schedule(t);
 }
-
+#define tasklet_hi_enable tasklet_enable
 
 static inline void tasklet_disable_nosync(struct tasklet_struct *t)
 {
@@ -84,35 +50,35 @@ static inline void tasklet_disable_nosyn
 static inline void tasklet_disable(struct tasklet_struct *t)
 {
 	tasklet_disable_nosync(t);
-	tasklet_unlock_wait(t);
-	smp_mb();
-}
-
-static inline void tasklet_enable(struct tasklet_struct *t)
-{
-	smp_mb__before_atomic_dec();
-	atomic_dec(&t->count);
+	flush_workqueue(ktaskletd_wq);
+	/* flush_workqueue should provide us a barrier */
 }
 
-static inline void tasklet_hi_enable(struct tasklet_struct *t)
+static inline int tasklet_is_scheduled(struct tasklet_struct *t)
 {
-	smp_mb__before_atomic_dec();
-	atomic_dec(&t->count);
+	int ret;
+	ret = work_pending(&t->work);
+	return ret;
 }
 
-static inline int tasklet_is_scheduled(struct tasklet_struct *t)
+static inline void tasklet_kill(struct tasklet_struct *t)
 {
-	return test_bit(TASKLET_STATE_SCHED, &t->state);
+	flush_workqueue(ktaskletd_wq);
 }
 
-extern void tasklet_kill(struct tasklet_struct *t);
-extern void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu);
 extern void tasklet_init(struct tasklet_struct *t,
 			 void (*func)(unsigned long), unsigned long data);
 
-#ifdef CONFIG_HOTPLUG_CPU
-void takeover_tasklets(unsigned int cpu);
-#endif /* CONFIG_HOTPLUG_CPU */
+static inline void tasklet_unlock(struct tasklet_struct *t)
+{
+	static int once;
 
-#endif /* _LINUX_TASKLET_H */
+	if (!once) {
+		once = 1;
+		printk(KERN_ERR
+		       "tasklet_unlocked called - sign of buggy code\n");
+		dump_stack();
+	}
+}
 
+#endif /* _LINUX_WORK_TASKLET_H */
Index: linux-2.6-test/kernel/tasklet_work.c
===================================================================
--- /dev/null
+++ linux-2.6-test/kernel/tasklet_work.c
@@ -0,0 +1,59 @@
+/*
+ *	linux/kernel/tasklet_work.c
+ *
+ *	Copyright (C) 2007 Steven Rostedt, Red Hat
+ *
+ */
+
+#include <linux/module.h>
+#include <linux/tasklet.h>
+
+struct workqueue_struct *ktaskletd_wq;
+EXPORT_SYMBOL(ktaskletd_wq);
+
+void work_tasklet_exec(struct work_struct *work)
+{
+	struct tasklet_struct *t =
+		container_of(work, struct tasklet_struct, work);
+
+	if (unlikely(atomic_read(&t->count))) {
+		set_bit(TASKLET_STATE_PENDING, &t->state);
+		/* Make sure bit is visible before checking count */
+		smp_mb();
+		/* make sure we were not just enabled (count == 0) */
+		if (likely(atomic_read(&t->count)))
+			goto out;
+		/* did the enable schedule us? */
+		if (!test_and_clear_bit(TASKLET_STATE_PENDING, &t->state))
+			goto out;
+	}
+
+	local_bh_disable();
+	t->func(t->data);
+	local_bh_enable();
+
+out:
+	return;
+}
+EXPORT_SYMBOL(work_tasklet_exec);
+
+void __init init_tasklets(void)
+{
+	ktaskletd_wq = create_workqueue("tasklets");
+	BUG_ON(!ktaskletd_wq);
+}
+
+void tasklet_init(struct tasklet_struct *t,
+		  void (*func)(unsigned long), unsigned long data)
+{
+	INIT_WORK(&t->work, work_tasklet_exec);
+	INIT_LIST_HEAD(&t->list);
+	t->state = 0;
+	atomic_set(&t->count, 0);
+	t->func = func;
+	t->data = data;
+	t->n = "anonymous";
+	pr_debug("anonymous tasklet %p set at %p\n",
+		t, __builtin_return_address(0));
+}
+EXPORT_SYMBOL(tasklet_init);
Index: linux-2.6-test/kernel/tasklet.c
===================================================================
--- linux-2.6-test.orig/kernel/tasklet.c
+++ /dev/null
@@ -1,203 +0,0 @@
-/*
- *	linux/kernel/softirq.c
- *
- *	Copyright (C) 1992 Linus Torvalds
- *
- * Rewritten. Old one was good in 2.2, but in 2.3 it was immoral. --ANK (990903)
- *
- *      Removed from softirq.c by Steven Rostedt
- */
-#include <linux/interrupt.h>
-#include <linux/cpu.h>
-
-/* Tasklets */
-struct tasklet_head
-{
-	struct tasklet_struct *list;
-};
-
-/* Some compilers disobey section attribute on statics when not
-   initialized -- RR */
-static DEFINE_PER_CPU(struct tasklet_head, tasklet_vec) = { NULL };
-static DEFINE_PER_CPU(struct tasklet_head, tasklet_hi_vec) = { NULL };
-
-void fastcall __tasklet_schedule(struct tasklet_struct *t)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	t->next = __get_cpu_var(tasklet_vec).list;
-	__get_cpu_var(tasklet_vec).list = t;
-	raise_softirq_irqoff(TASKLET_SOFTIRQ);
-	local_irq_restore(flags);
-}
-
-EXPORT_SYMBOL(__tasklet_schedule);
-
-void fastcall __tasklet_hi_schedule(struct tasklet_struct *t)
-{
-	unsigned long flags;
-
-	local_irq_save(flags);
-	t->next = __get_cpu_var(tasklet_hi_vec).list;
-	__get_cpu_var(tasklet_hi_vec).list = t;
-	raise_softirq_irqoff(HI_SOFTIRQ);
-	local_irq_restore(flags);
-}
-
-EXPORT_SYMBOL(__tasklet_hi_schedule);
-
-static void tasklet_action(struct softirq_action *a)
-{
-	struct tasklet_struct *list;
-
-	local_irq_disable();
-	list = __get_cpu_var(tasklet_vec).list;
-	__get_cpu_var(tasklet_vec).list = NULL;
-	local_irq_enable();
-
-	while (list) {
-		struct tasklet_struct *t = list;
-
-		list = list->next;
-
-		if (tasklet_trylock(t)) {
-			if (!atomic_read(&t->count)) {
-				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
-					BUG();
-				t->func(t->data);
-				tasklet_unlock(t);
-				continue;
-			}
-			tasklet_unlock(t);
-		}
-
-		local_irq_disable();
-		t->next = __get_cpu_var(tasklet_vec).list;
-		__get_cpu_var(tasklet_vec).list = t;
-		__raise_softirq_irqoff(TASKLET_SOFTIRQ);
-		local_irq_enable();
-	}
-}
-
-static void tasklet_hi_action(struct softirq_action *a)
-{
-	struct tasklet_struct *list;
-
-	local_irq_disable();
-	list = __get_cpu_var(tasklet_hi_vec).list;
-	__get_cpu_var(tasklet_hi_vec).list = NULL;
-	local_irq_enable();
-
-	while (list) {
-		struct tasklet_struct *t = list;
-
-		list = list->next;
-
-		if (tasklet_trylock(t)) {
-			if (!atomic_read(&t->count)) {
-				if (!test_and_clear_bit(TASKLET_STATE_SCHED, &t->state))
-					BUG();
-				t->func(t->data);
-				tasklet_unlock(t);
-				continue;
-			}
-			tasklet_unlock(t);
-		}
-
-		local_irq_disable();
-		t->next = __get_cpu_var(tasklet_hi_vec).list;
-		__get_cpu_var(tasklet_hi_vec).list = t;
-		__raise_softirq_irqoff(HI_SOFTIRQ);
-		local_irq_enable();
-	}
-}
-
-
-void tasklet_init(struct tasklet_struct *t,
-		  void (*func)(unsigned long), unsigned long data)
-{
-	t->next = NULL;
-	t->state = 0;
-	atomic_set(&t->count, 0);
-	t->func = func;
-	t->data = data;
-}
-
-EXPORT_SYMBOL(tasklet_init);
-
-void tasklet_kill(struct tasklet_struct *t)
-{
-	if (in_interrupt())
-		printk("Attempt to kill tasklet from interrupt\n");
-
-	while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
-		do
-			yield();
-		while (test_bit(TASKLET_STATE_SCHED, &t->state));
-	}
-	tasklet_unlock_wait(t);
-	clear_bit(TASKLET_STATE_SCHED, &t->state);
-}
-
-EXPORT_SYMBOL(tasklet_kill);
-
-void __init softirq_init(void)
-{
-	open_softirq(TASKLET_SOFTIRQ, tasklet_action, NULL);
-	open_softirq(HI_SOFTIRQ, tasklet_hi_action, NULL);
-}
-
-#ifdef CONFIG_HOTPLUG_CPU
-/*
- * tasklet_kill_immediate is called to remove a tasklet which can already be
- * scheduled for execution on @cpu.
- *
- * Unlike tasklet_kill, this function removes the tasklet
- * _immediately_, even if the tasklet is in TASKLET_STATE_SCHED state.
- *
- * When this function is called, @cpu must be in the CPU_DEAD state.
- */
-void tasklet_kill_immediate(struct tasklet_struct *t, unsigned int cpu)
-{
-	struct tasklet_struct **i;
-
-	BUG_ON(cpu_online(cpu));
-	BUG_ON(test_bit(TASKLET_STATE_RUN, &t->state));
-
-	if (!test_bit(TASKLET_STATE_SCHED, &t->state))
-		return;
-
-	/* CPU is dead, so no lock needed. */
-	for (i = &per_cpu(tasklet_vec, cpu).list; *i; i = &(*i)->next) {
-		if (*i == t) {
-			*i = t->next;
-			return;
-		}
-	}
-	BUG();
-}
-
-void takeover_tasklets(unsigned int cpu)
-{
-	struct tasklet_struct **i;
-
-	/* CPU is dead, so no lock needed. */
-	local_irq_disable();
-
-	/* Find end, append list for that CPU. */
-	for (i = &__get_cpu_var(tasklet_vec).list; *i; i = &(*i)->next);
-	*i = per_cpu(tasklet_vec, cpu).list;
-	per_cpu(tasklet_vec, cpu).list = NULL;
-	raise_softirq_irqoff(TASKLET_SOFTIRQ);
-
-	for (i = &__get_cpu_var(tasklet_hi_vec).list; *i; i = &(*i)->next);
-	*i = per_cpu(tasklet_hi_vec, cpu).list;
-	per_cpu(tasklet_hi_vec, cpu).list = NULL;
-	raise_softirq_irqoff(HI_SOFTIRQ);
-
-	local_irq_enable();
-}
-#endif /* CONFIG_HOTPLUG_CPU */
-
-
Index: linux-2.6-test/kernel/softirq.c
===================================================================
--- linux-2.6-test.orig/kernel/softirq.c
+++ linux-2.6-test/kernel/softirq.c
@@ -425,7 +425,6 @@ static int __cpuinit cpu_callback(struct
 		p = per_cpu(ksoftirqd, hotcpu);
 		per_cpu(ksoftirqd, hotcpu) = NULL;
 		kthread_stop(p);
-		takeover_tasklets(hotcpu);
 		break;
 #endif /* CONFIG_HOTPLUG_CPU */
  	}

-- 

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Paul E. McKenney]

On Wed, Jun 27, 2007 at 03:36:34PM -0400, Steven Rostedt wrote:
> I believe this was originally done by Dipankar Sarma. I pulled these
> changes from the -rt kernel.
> 
> For better preformance, RCU should use a softirq instead of a
> tasklet.

Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Tested with both kernbench and rcutorture on i386, x86_64, and ppc64.

						Thanx, Paul

> From: Dipankar Sarma <dipankar@in.ibm.com>
> Signed-off-by: Steven Rostedt <rostedt@goodmis.org>
> 
> Index: linux-2.6-test/include/linux/interrupt.h
> ===================================================================
> --- linux-2.6-test.orig/include/linux/interrupt.h
> +++ linux-2.6-test/include/linux/interrupt.h
> @@ -245,6 +245,9 @@ enum
>  #ifdef CONFIG_HIGH_RES_TIMERS
>  	HRTIMER_SOFTIRQ,
>  #endif
> +	RCU_SOFTIRQ,	/* Preferable RCU should always be the last softirq */
> +	/* Entries after this are ignored in split softirq mode */
> +	MAX_SOFTIRQ,
>  };
> 
>  /* softirq mask and active fields moved to irq_cpustat_t in
> Index: linux-2.6-test/kernel/rcupdate.c
> ===================================================================
> --- linux-2.6-test.orig/kernel/rcupdate.c
> +++ linux-2.6-test/kernel/rcupdate.c
> @@ -67,7 +67,6 @@ DEFINE_PER_CPU(struct rcu_data, rcu_data
>  DEFINE_PER_CPU(struct rcu_data, rcu_bh_data) = { 0L };
> 
>  /* Fake initialization required by compiler */
> -static DEFINE_PER_CPU(struct tasklet_struct, rcu_tasklet) = {NULL};
>  static int blimit = 10;
>  static int qhimark = 10000;
>  static int qlowmark = 100;
> @@ -253,7 +252,7 @@ static void rcu_do_batch(struct rcu_data
>  	if (!rdp->donelist)
>  		rdp->donetail = &rdp->donelist;
>  	else
> -		tasklet_schedule(&per_cpu(rcu_tasklet, rdp->cpu));
> +		raise_softirq(RCU_SOFTIRQ);
>  }
> 
>  /*
> @@ -405,7 +404,6 @@ static void rcu_offline_cpu(int cpu)
>  					&per_cpu(rcu_bh_data, cpu));
>  	put_cpu_var(rcu_data);
>  	put_cpu_var(rcu_bh_data);
> -	tasklet_kill_immediate(&per_cpu(rcu_tasklet, cpu), cpu);
>  }
> 
>  #else
> @@ -417,7 +415,7 @@ static void rcu_offline_cpu(int cpu)
>  #endif
> 
>  /*
> - * This does the RCU processing work from tasklet context. 
> + * This does the RCU processing work from softirq context.
>   */
>  static void __rcu_process_callbacks(struct rcu_ctrlblk *rcp,
>  					struct rcu_data *rdp)
> @@ -462,7 +460,7 @@ static void __rcu_process_callbacks(stru
>  		rcu_do_batch(rdp);
>  }
> 
> -static void rcu_process_callbacks(unsigned long unused)
> +static void rcu_process_callbacks(struct softirq_action *unused)
>  {
>  	__rcu_process_callbacks(&rcu_ctrlblk, &__get_cpu_var(rcu_data));
>  	__rcu_process_callbacks(&rcu_bh_ctrlblk, &__get_cpu_var(rcu_bh_data));
> @@ -526,7 +524,7 @@ void rcu_check_callbacks(int cpu, int us
>  		rcu_bh_qsctr_inc(cpu);
>  	} else if (!in_softirq())
>  		rcu_bh_qsctr_inc(cpu);
> -	tasklet_schedule(&per_cpu(rcu_tasklet, cpu));
> +	raise_softirq(RCU_SOFTIRQ);
>  }
> 
>  static void rcu_init_percpu_data(int cpu, struct rcu_ctrlblk *rcp,
> @@ -549,7 +547,7 @@ static void __devinit rcu_online_cpu(int
> 
>  	rcu_init_percpu_data(cpu, &rcu_ctrlblk, rdp);
>  	rcu_init_percpu_data(cpu, &rcu_bh_ctrlblk, bh_rdp);
> -	tasklet_init(&per_cpu(rcu_tasklet, cpu), rcu_process_callbacks, 0UL);
> +	open_softirq(RCU_SOFTIRQ, rcu_process_callbacks, NULL);
>  }
> 
>  static int __cpuinit rcu_cpu_notify(struct notifier_block *self,
> 
> -- 

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Linus Torvalds]

On Mon, 16 Jul 2007, Paul E. McKenney wrote:
>
> On Wed, Jun 27, 2007 at 03:36:34PM -0400, Steven Rostedt wrote:
> > I believe this was originally done by Dipankar Sarma. I pulled these
> > changes from the -rt kernel.
> > 
> > For better preformance, RCU should use a softirq instead of a
> > tasklet.
> 
> Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> 
> Tested with both kernbench and rcutorture on i386, x86_64, and ppc64.

Btw, are there any numbers on this? Is this whole thing actually 
noticeable?

		Linus

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Steven Rostedt]

On Mon, 16 Jul 2007, Linus Torvalds wrote:

>
> Btw, are there any numbers on this? Is this whole thing actually
> noticeable?

Besides the obvious removal of code?

The old way actually made an effort to create per_cpu tasklets!  So that
the tasklets *can* run simultaneously, making it in essense a softirq.
Since tasklets are implemented by the softirq, with a lot of code to make
a tasklet function run only on one CPU at a time, and it can run on any
CPU.  By converting the rcu tasklet (which had work to make it act like a
softirq) to just be a softirq, we removed all the extra overhead of a
tasklet.

But to answer your question.  No, I didn't take any actual measurements.
The changes just seemed obvious to me (and others).

-- Steve

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Ingo Molnar]

* Linus Torvalds <torvalds@linux-foundation.org> wrote:

> On Mon, 16 Jul 2007, Paul E. McKenney wrote:
> >
> > On Wed, Jun 27, 2007 at 03:36:34PM -0400, Steven Rostedt wrote:
> > > I believe this was originally done by Dipankar Sarma. I pulled these
> > > changes from the -rt kernel.
> > > 
> > > For better preformance, RCU should use a softirq instead of a
> > > tasklet.
> > 
> > Acked-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > 
> > Tested with both kernbench and rcutorture on i386, x86_64, and ppc64.
> 
> Btw, are there any numbers on this? Is this whole thing actually 
> noticeable?

i doubt it's noticeable, because the RCU tasklet was not a "true" global 
tasklet, it was an array of tasklets put into a PER_CPU array. (which 
kind of defeats the purpose of tasklets)

So there should be no/little scalability disadvantage from this type of 
use of tasklets, and the tasklet use was at most a wart and a small, 
constant micro-cost that is likely masked by other costs. (and rcu never 
used any of the fancier tasklet ops like tasklet_disable() either)

	Ingo

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Ingo Molnar]

* Steven Rostedt <rostedt@goodmis.org> wrote:

> But to answer your question.  No, I didn't take any actual 
> measurements. The changes just seemed obvious to me (and others).

btw., does anyone know about some reliable way to stress and measure RCU 
completion performance, via some real userspace app? I think there used 
to be an open-tons-of-files thing that unearthed the 
RCU-completion-single-threadness bottleneck a year or two ago? Anyone 
got a link to it (or to some other performance tool for RCU)?

	Ingo

Re: [RFC PATCH 1/8] Convert the RCU tasklet into a softirq

[Paul E. McKenney]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

On Mon, Jul 16, 2007 at 10:53:23PM +0200, Ingo Molnar wrote:
> 
> * Steven Rostedt <rostedt@goodmis.org> wrote:
> 
> > But to answer your question.  No, I didn't take any actual 
> > measurements. The changes just seemed obvious to me (and others).
> 
> btw., does anyone know about some reliable way to stress and measure RCU 
> completion performance, via some real userspace app? I think there used 
> to be an open-tons-of-files thing that unearthed the 
> RCU-completion-single-threadness bottleneck a year or two ago? Anyone 
> got a link to it (or to some other performance tool for RCU)?

It was more of a stress test -- it simply opened a file and immediately
closes it in a tight loop.  2.6.22 survives four of the following running
on a 4-CPU machine.

Attached are the kernel modules I use to measure RCU performance
(and to torture RCU), but they don't really measure completion
performance.  (All licensed under GPL, FWIW.)

						Thanx, Paul

(The following is my reconstruction of the open/close stress test.)

#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main(int argc, char *argv[])
{
	int fd;

	for (;;) {
		fd = open("/dev/null", O_RDONLY);
		if (fd >= 0) {
			close(fd);
		}
	}
}

[-- Attachment #2: RCUtorture.2007.07.17a.tgz --]
[-- Type: application/x-gtar, Size: 11758 bytes --]