[PATCH] Exiting queue and task might race to free cic

[PATCH] Exiting queue and task might race to free cic

[Nikanth Karthikesan]

Hi Jens

Looking at the bug reported here
http://thread.gmane.org/gmane.linux.kernel/722539
it looks like an exiting queue can race with an exiting task.

When a queue exits the queue lock is taken and cfq_exit_queue() would free all 
the cic's associated with the queue.

But when a task exits, cfq_exit_io_context() gets cic one by one and then 
locks the associated queue to call __cfq_exit_single_io_context. It looks like 
between getting a cic from the ioc and locking the queue, the queue might have 
exited on another cpu. Isn't this possible?

If possible, either verifying whether cic->key is still not null or q->flags 
does not have QUEUE_FLAG_DEAD set would fix this.

Thanks
Nikanth Karthikesan

Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>

---
diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..b9b627a 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1318,7 +1318,12 @@ static void cfq_exit_single_io_context(struct 
io_context *ioc,
 		unsigned long flags;
 
 		spin_lock_irqsave(q->queue_lock, flags);
-		__cfq_exit_single_io_context(cfqd, cic);
+		/*
+		 * cic might have been already exited when an exiting task
+		 * races with an exiting queue.
+		 */
+		if (likely(cic->key))
+			__cfq_exit_single_io_context(cfqd, cic);
 		spin_unlock_irqrestore(q->queue_lock, flags);
 	}
 }

Or this would also work

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..7a068bd 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1318,7 +1318,11 @@ static void cfq_exit_single_io_context(struct 
io_context *ioc,
 		unsigned long flags;
 
 		spin_lock_irqsave(q->queue_lock, flags);
-		__cfq_exit_single_io_context(cfqd, cic);
+		/*
+		 * Make sure the queue is not dead.
+		 */
+		if (likely(!test_bit(QUEUE_FLAG_DEAD, &q->queue_flags)))
+			__cfq_exit_single_io_context(cfqd, cic);
 		spin_unlock_irqrestore(q->queue_lock, flags);
 	}
 }

Re: [PATCH] Exiting queue and task might race to free cic

[Jens Axboe]

On Wed, Nov 19 2008, Nikanth Karthikesan wrote:
> Hi Jens
> 
> Looking at the bug reported here
> http://thread.gmane.org/gmane.linux.kernel/722539
> it looks like an exiting queue can race with an exiting task.
> 
> When a queue exits the queue lock is taken and cfq_exit_queue() would free all 
> the cic's associated with the queue.
> 
> But when a task exits, cfq_exit_io_context() gets cic one by one and then 
> locks the associated queue to call __cfq_exit_single_io_context. It looks like 
> between getting a cic from the ioc and locking the queue, the queue might have 
> exited on another cpu. Isn't this possible?
> 
> If possible, either verifying whether cic->key is still not null or q->flags 
> does not have QUEUE_FLAG_DEAD set would fix this.
> 
> Thanks
> Nikanth Karthikesan
> 
> Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> 
> ---
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 6a062ee..b9b627a 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1318,7 +1318,12 @@ static void cfq_exit_single_io_context(struct 
> io_context *ioc,
>  		unsigned long flags;
>  
>  		spin_lock_irqsave(q->queue_lock, flags);
> -		__cfq_exit_single_io_context(cfqd, cic);
> +		/*
> +		 * cic might have been already exited when an exiting task
> +		 * races with an exiting queue.
> +		 */
> +		if (likely(cic->key))
> +			__cfq_exit_single_io_context(cfqd, cic);
>  		spin_unlock_irqrestore(q->queue_lock, flags);
>  	}
>  }

Not sure this is enough, we probably need to copy the key to ensure that
we get a fresh value. How does this look?

Did you actually trigger this, or is it just from code inspection?

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..560cd1c 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1318,7 +1318,14 @@ static void cfq_exit_single_io_context(struct io_context *ioc,
 		unsigned long flags;
 
 		spin_lock_irqsave(q->queue_lock, flags);
-		__cfq_exit_single_io_context(cfqd, cic);
+
+		/*
+		 * Ensure we get a fresh copy of the ->key to prevent
+		 * race between exiting task and queue
+		 */
+		smp_read_barrier_depends();
+		if (cic->key)
+			__cfq_exit_single_io_context(cfqd, cic);
 		spin_unlock_irqrestore(q->queue_lock, flags);
 	}
 }

-- 
Jens Axboe

Re: [PATCH] Exiting queue and task might race to free cic

[Fabio Checconi]

> From: Jens Axboe <jens.axboe@oracle.com>
> Date: Wed, Nov 19, 2008 03:15:31PM +0100
>
> On Wed, Nov 19 2008, Nikanth Karthikesan wrote:
> > Hi Jens
> > 
> > Looking at the bug reported here
> > http://thread.gmane.org/gmane.linux.kernel/722539
> > it looks like an exiting queue can race with an exiting task.
> > 
> > When a queue exits the queue lock is taken and cfq_exit_queue() would free all 
> > the cic's associated with the queue.
> > 
> > But when a task exits, cfq_exit_io_context() gets cic one by one and then 
> > locks the associated queue to call __cfq_exit_single_io_context. It looks like 
> > between getting a cic from the ioc and locking the queue, the queue might have 
> > exited on another cpu. Isn't this possible?
> > 
> > If possible, either verifying whether cic->key is still not null or q->flags 
> > does not have QUEUE_FLAG_DEAD set would fix this.
> > 
> > Thanks
> > Nikanth Karthikesan
> > 
> > Signed-off-by: Nikanth Karthikesan <knikanth@suse.de>
> > 
> > ---
> > diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> > index 6a062ee..b9b627a 100644
> > --- a/block/cfq-iosched.c
> > +++ b/block/cfq-iosched.c
> > @@ -1318,7 +1318,12 @@ static void cfq_exit_single_io_context(struct 
> > io_context *ioc,
> >  		unsigned long flags;
> >  
> >  		spin_lock_irqsave(q->queue_lock, flags);
> > -		__cfq_exit_single_io_context(cfqd, cic);
> > +		/*
> > +		 * cic might have been already exited when an exiting task
> > +		 * races with an exiting queue.
> > +		 */
> > +		if (likely(cic->key))
> > +			__cfq_exit_single_io_context(cfqd, cic);
> >  		spin_unlock_irqrestore(q->queue_lock, flags);
> >  	}
> >  }
> 
> Not sure this is enough, we probably need to copy the key to ensure that
> we get a fresh value. How does this look?
> 
> Did you actually trigger this, or is it just from code inspection?
> 
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 6a062ee..560cd1c 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1318,7 +1318,14 @@ static void cfq_exit_single_io_context(struct io_context *ioc,
>  		unsigned long flags;
>  
>  		spin_lock_irqsave(q->queue_lock, flags);
> -		__cfq_exit_single_io_context(cfqd, cic);
> +
> +		/*
> +		 * Ensure we get a fresh copy of the ->key to prevent
> +		 * race between exiting task and queue
> +		 */
> +		smp_read_barrier_depends();
> +		if (cic->key)
> +			__cfq_exit_single_io_context(cfqd, cic);
>  		spin_unlock_irqrestore(q->queue_lock, flags);
>  	}
>  }
> 

I've seen once the oops reported (the BUG() now @ line 1247), but I've
never been able to reproduce it afterwards.  I think that there still
is a window open for a race here:

1314 struct cfq_data *cfqd = cic->key;
1315

=====> here cfq_exit_queue() can free cfqd and assign cic->key = NULL,
       and accessing cfqd->queue is not safe.  [ If I'm not wrong :) ]

1316 if (cfqd) {
1317         struct request_queue *q = cfqd->queue;

Re: [PATCH] Exiting queue and task might race to free cic

[Nikanth Karthikesan]

On Wednesday 19 November 2008 19:45:31 Jens Axboe wrote:
> On Wed, Nov 19 2008, Nikanth Karthikesan wrote:

> Not sure this is enough, we probably need to copy the key to ensure that
> we get a fresh value. How does this look?
>

Agreed. Read barrier required. But the compiler hint, "likely" can stay?

diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
index 6a062ee..4504b94 100644
--- a/block/cfq-iosched.c
+++ b/block/cfq-iosched.c
@@ -1318,7 +1318,15 @@ static void cfq_exit_single_io_context(struct 
io_context *ioc,
 		unsigned long flags;
 
 		spin_lock_irqsave(q->queue_lock, flags);
-		__cfq_exit_single_io_context(cfqd, cic);
+
+		/*
+		 * Ensure we get a fresh copy of the ->key to prevent
+		 * race between exiting task and queue
+		 */
+		smp_read_barrier_depends();
+		if (likely(cic->key))
+			__cfq_exit_single_io_context(cfqd, cic);
+
 		spin_unlock_irqrestore(q->queue_lock, flags);
 	}
 }


> Did you actually trigger this, or is it just from code inspection?
>

No. But I am looking at another bug report on Suse Kernel where the bug is 
triggered during reboot when the kernel thread usb_stor_scan_thread exits.

Thanks
Nikanth Karthikesan

Re: [PATCH] Exiting queue and task might race to free cic

[Nikanth Karthikesan]

Hi Jens

On Thursday 20 November 2008 10:27:06 Nikanth Karthikesan wrote:
> On Wednesday 19 November 2008 19:45:31 Jens Axboe wrote:
> > On Wed, Nov 19 2008, Nikanth Karthikesan wrote:
> >
> > Not sure this is enough, we probably need to copy the key to ensure that
> > we get a fresh value. How does this look?
>
> Agreed. Read barrier required. But the compiler hint, "likely" can stay?
>
> diff --git a/block/cfq-iosched.c b/block/cfq-iosched.c
> index 6a062ee..4504b94 100644
> --- a/block/cfq-iosched.c
> +++ b/block/cfq-iosched.c
> @@ -1318,7 +1318,15 @@ static void cfq_exit_single_io_context(struct
> io_context *ioc,
>  		unsigned long flags;
>
>  		spin_lock_irqsave(q->queue_lock, flags);
> -		__cfq_exit_single_io_context(cfqd, cic);
> +
> +		/*
> +		 * Ensure we get a fresh copy of the ->key to prevent
> +		 * race between exiting task and queue
> +		 */
> +		smp_read_barrier_depends();
> +		if (likely(cic->key))
> +			__cfq_exit_single_io_context(cfqd, cic);
> +
>  		spin_unlock_irqrestore(q->queue_lock, flags);
>  	}
>  }
>
> > Did you actually trigger this, or is it just from code inspection?
>
> No. But I am looking at another bug report on Suse Kernel where the bug is
> triggered during reboot when the kernel thread usb_stor_scan_thread exits.
>

This patch seems to solve the above said bug report on Suse kernel. So, yes it 
is reproducible! Can this be merged?

Thanks
Nikanth