[PATCH] IPMI: Fix some RCU problems

[PATCH] IPMI: Fix some RCU problems

[Corey Minyard]

Fix some RCU problem pointed out by Paul McKenney of IBM.  These are:

The wholesale move of the command receivers list into a new list was
not safe because the list will point to the new tail during a
traversal, so the traversal will never end on a reader if this happens
during a read.

Memory barriers were needed to handle proper ordering of the setting
of the IPMI interface as valid.  Readers might not see proper ordering
of data otherwise.

In ipmi_smi_watcher_register(), the use of the _rcu suffix on the list
is unnecessary.

This require the list_splice_init_rcu() patch previously posted.

Signed-off-by: Corey Minyard <minyard@acm.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>

Index: linux-2.6.19/drivers/char/ipmi/ipmi_msghandler.c
===================================================================
--- linux-2.6.19.orig/drivers/char/ipmi/ipmi_msghandler.c	2006-12-30 12:41:15.000000000 -0600
+++ linux-2.6.19/drivers/char/ipmi/ipmi_msghandler.c	2006-12-30 12:43:50.000000000 -0600
@@ -406,13 +406,14 @@
 	free_smi_msg_list(&intf->waiting_msgs);
 	free_recv_msg_list(&intf->waiting_events);
 
-	/* Wholesale remove all the entries from the list in the
-	 * interface and wait for RCU to know that none are in use. */
+	/*
+	 * Wholesale remove all the entries from the list in the
+	 * interface and wait for RCU to know that none are in use.
+	 */
 	mutex_lock(&intf->cmd_rcvrs_mutex);
-	list_add_rcu(&list, &intf->cmd_rcvrs);
-	list_del_rcu(&intf->cmd_rcvrs);
+	INIT_LIST_HEAD(&list);
+	list_splice_init_rcu(&intf->cmd_rcvrs, &list, synchronize_rcu);
 	mutex_unlock(&intf->cmd_rcvrs_mutex);
-	synchronize_rcu();
 
 	list_for_each_entry_safe(rcvr, rcvr2, &list, link)
 		kfree(rcvr);
@@ -451,7 +452,7 @@
 	mutex_lock(&ipmi_interfaces_mutex);
 
 	/* Build a list of things to deliver. */
-	list_for_each_entry_rcu(intf, &ipmi_interfaces, link) {
+	list_for_each_entry(intf, &ipmi_interfaces, link) {
 		if (intf->intf_num == -1)
 			continue;
 		e = kmalloc(sizeof(*e), GFP_KERNEL);
@@ -838,6 +839,7 @@
 	goto out_kfree;
 
  found:
+	smp_rmb();
 	/* Note that each existing user holds a refcount to the interface. */
 	kref_get(&intf->refcount);
 
@@ -2761,6 +2763,7 @@
 		kref_put(&intf->refcount, intf_free);
 	} else {
 		/* After this point the interface is legal to use. */
+		smp_wmb(); /* Keep memory order straight for RCU readers. */
 		intf->intf_num = i;
 		mutex_unlock(&ipmi_interfaces_mutex);
 		call_smi_watchers(i, intf->si_dev);
@@ -3924,6 +3927,8 @@
 			/* Interface was not ready yet. */
 			continue;
 
+		smp_rmb();
+
 		/* First job here is to figure out where to send the
 		   OEM events.  There's no way in IPMI to send OEM
 		   events using an event send command, so we have to

Re: [PATCH] IPMI: Fix some RCU problems

[Andrew Morton]

On Wed, 3 Jan 2007 09:31:30 -0600
Corey Minyard <minyard@acm.org> wrote:

>   found:
> +	smp_rmb();
>  	/* Note that each existing user holds a refcount to the interface. */
>  	kref_get(&intf->refcount);
>  
> @@ -2761,6 +2763,7 @@
>  		kref_put(&intf->refcount, intf_free);
>  	} else {
>  		/* After this point the interface is legal to use. */
> +		smp_wmb(); /* Keep memory order straight for RCU readers. */
>  		intf->intf_num = i;
>  		mutex_unlock(&ipmi_interfaces_mutex);
>  		call_smi_watchers(i, intf->si_dev);
> @@ -3924,6 +3927,8 @@
>  			/* Interface was not ready yet. */
>  			continue;
>  
> +		smp_rmb();
> +

It's nice to always have a comment explaining the use of open-coded
barriers.  Because often the reader is left wondered what on earth it's
barriering against what on earth else.

Re: [Openipmi-developer] [PATCH] IPMI: Fix some RCU problems

[Corey Minyard]

On Wed, Jan 03, 2007 at 01:22:32PM -0800, Andrew Morton wrote:
> It's nice to always have a comment explaining the use of open-coded
> barriers.  Because often the reader is left wondered what on earth it's
> barriering against what on earth else.
> 

Ok, here it is...


Andrew asked that the open-coded barriers be commented, so here it
is.  I also realized that one of the read barriers was in an area
where the protecting mutex was held, so no read barrier was needed.

Signed-off-by: Corey Minyard <minyard@acm.org>

Index: linux-2.6.19/drivers/char/ipmi/ipmi_msghandler.c
===================================================================
--- linux-2.6.19.orig/drivers/char/ipmi/ipmi_msghandler.c
+++ linux-2.6.19/drivers/char/ipmi/ipmi_msghandler.c
@@ -839,7 +839,6 @@ int ipmi_create_user(unsigned int       
 	goto out_kfree;
 
  found:
-	smp_rmb();
 	/* Note that each existing user holds a refcount to the interface. */
 	kref_get(&intf->refcount);
 
@@ -2762,10 +2761,15 @@ int ipmi_register_smi(struct ipmi_smi_ha
 		synchronize_rcu();
 		kref_put(&intf->refcount, intf_free);
 	} else {
-		/* After this point the interface is legal to use. */
-		smp_wmb(); /* Keep memory order straight for RCU readers. */
+		/*
+		 * Keep memory order straight for RCU readers.  Make
+		 * sure everything else is committed to memory before
+		 * setting intf_num to mark the interface valid.
+		 */
+		smp_wmb();
 		intf->intf_num = i;
 		mutex_unlock(&ipmi_interfaces_mutex);
+		/* After this point the interface is legal to use. */
 		call_smi_watchers(i, intf->si_dev);
 		mutex_unlock(&smi_watchers_mutex);
 	}
@@ -3927,6 +3931,12 @@ static void send_panic_events(char *str)
 			/* Interface was not ready yet. */
 			continue;
 
+		/*
+		 * intf_num is used as an marker to tell if the
+		 * interface is valid.  Thus we need a read barrier to
+		 * make sure data fetched before checking intf_num
+		 * won't be used.
+		 */
 		smp_rmb();
 
 		/* First job here is to figure out where to send the