Re: [PATCH 3/6] drm/ttm: call ttm_bo_cleanup_refs with reservation and lru lock held

[Maarten Lankhorst <maarten.lankhorst@canonical.com>]

Op 28-11-12 16:10, Thomas Hellstrom schreef:
> On 11/28/2012 03:46 PM, Maarten Lankhorst wrote:
>> Op 28-11-12 15:23, Thomas Hellstrom schreef:
>>> On 11/28/2012 02:55 PM, Maarten Lankhorst wrote:
>>>> Op 28-11-12 14:21, Thomas Hellstrom schreef:
>>>>> On 11/28/2012 01:15 PM, Maarten Lankhorst wrote:
>>>>>> Op 28-11-12 12:54, Thomas Hellstrom schreef:
>>>>>>> On 11/28/2012 12:25 PM, Maarten Lankhorst wrote:
>>>>>>>> By removing the unlocking of lru and retaking it immediately, a race is
>>>>>>>> removed where the bo is taken off the swap list or the lru list between
>>>>>>>> the unlock and relock. As such the cleanup_refs code can be simplified,
>>>>>>>> it will attempt to call ttm_bo_wait non-blockingly, and if it fails
>>>>>>>> it will drop the locks and perform a blocking wait, or return an error
>>>>>>>> if no_wait_gpu was set.
>>>>>>>>
>>>>>>>> The need for looping is also eliminated, since swapout and evict_mem_first
>>>>>>>> will always follow the destruction path, so no new fence is allowed
>>>>>>>> to be attached. As far as I can see this may already have been the case,
>>>>>>>> but the unlocking / relocking required a complicated loop to deal with
>>>>>>>> re-reservation.
>>>>>>>>
>>>>>>>> The downside is that ttm_bo_cleanup_memtype_use is no longer called with
>>>>>>>> reservation held, so drivers must be aware that move_notify with a null
>>>>>>>> parameter doesn't require a reservation.
>>>>>>> Why can't we unreserve *after* ttm_bo_cleanup_memtype_use? That's not
>>>>>>> immediately clear from this patch.
>>>>>> Because we would hold the reservation while waiting and with the object still
>>>>>> on swap and lru lists still, that would defeat the whole purpose of keeping
>>>>>> the object on multiple lists, plus break current code that assumes bo's on the
>>>>>> those lists can always be reserved.
>>>>>>
>>>>>> the if (ret && !no_wait_gpu) path has to drop the reservation and lru lock, and
>>>>>> isn't guaranteed to be able to retake it. Maybe it could be guaranteed now, but
>>>>>> I'm sure that would not be the case if the reservations were shared across
>>>>>> devices.
>>>>> The evict path removes the BO from the LRU lists, drops the LRU lock but hangs on to the reservation,
>>>>> and in case the wait goes wrong, re-adds the bo to the LRU lists and returns an error.
>>>> If you really want to, we could hang on to the !no_wait_gpu path, wait shouldn't ever fail there, so I suppose
>>>> leaving it off the lru lists and not re-add on any list in case of wait fail is fine. It's still on the ddestroy list in that
>>>> case, so not adding it back to the other lists is harmless.
>>>>
>>> Well I'm a bit afraid that theoretically, other callers may have a bo reserved, while cleanup_refs_and_unlock
>>> more or less runs the whole destroy path on that buffer. Sure, we have control over those other reservers,
>>> but it may come back and bite us.
>> That's why initially I moved all the destruction to ttm_bo_release_list, to have all destruction in
>> only 1 place. But even now it's serialized with the lru lock, while the destruction may not happen
>> right away, it still happens before last list ref to the bo is dropped.
>>
>> But it's your call, just choose the approach you want and I'll resubmit this. :-)
>>
>>> Also the wait might fail if a signal is hit, so it's definitely possible, and even likely in the case of the X server process.
>>>
>>> Anyway, I prefer if we could try to keep the reservation across the ttm_cleanup_memtype_use  function, and as far
>>> as I can tell, the only thing preventing that is the reservation release in the (!no_wait_gpu) path. So if we alter that to
>>> do the same as the evict path I think without looking to deeply into the consequences that we should be safe.
>> I think returning success early without removing off ddestroy list if re-reserving fails
>> with lru lock held would be better.
>>
>> We completed the wait and attempt to reserve the bo, which failed. Without the lru
>> lock atomicity, this can't happen since the only places that would do it call this with
>> the lru lock held.
>>
>> With the atomicity removal, the only place that could do this is ttm_bo_delayed_delete
>> with remove_all set to true. And even if that happened the destruction code would run
>> *anyway* since we completed the waiting part already, it would just not necessarily be
>> run from this thread, but that guarantee didn't exist anyway.
>>> Then we should be able to skip patch 2 as well.
>> If my tryreserve approach sounds sane, second patch should still be skippable. :-)
>
> Sure, Lets go for that approach.
Ok updated patch below,  you only need to check if you like the approach or not,
since I haven't tested it yet beyond compiling. Rest of series (minus patch 2)
should still apply without modification.

    drm/ttm: call ttm_bo_cleanup_refs with reservation and lru lock held, v2
    
    By removing the unlocking of lru and retaking it immediately, a race is
    removed where the bo is taken off the swap list or the lru list between
    the unlock and relock. As such the cleanup_refs code can be simplified,
    it will attempt to call ttm_bo_wait non-blockingly, and if it fails
    it will drop the locks and perform a blocking wait, or return an error
    if no_wait_gpu was set.
    
    The need for looping is also eliminated, since swapout and evict_mem_first
    will always follow the destruction path, no new fence is allowed
    to be attached. As far as I can see this may already have been the case,
    but the unlocking / relocking required a complicated loop to deal with
    re-reservation.
    
    Changes since v1:
     - Simplify no_wait_gpu case by folding it in with empty ddestroy.
     - Hold a reservation while calling ttm_bo_cleanup_memtype_use again.
    
    Signed-off-by: Maarten Lankhorst <maarten.lankhorst@canonical.com>

diff --git a/drivers/gpu/drm/ttm/ttm_bo.c b/drivers/gpu/drm/ttm/ttm_bo.c
index 202fc20..e9f01fe 100644
--- a/drivers/gpu/drm/ttm/ttm_bo.c
+++ b/drivers/gpu/drm/ttm/ttm_bo.c
@@ -488,12 +488,16 @@ static void ttm_bo_cleanup_memtype_use(struct ttm_buffer_object *bo)
 	ttm_bo_mem_put(bo, &bo->mem);
 
 	atomic_set(&bo->reserved, 0);
+	wake_up_all(&bo->event_queue);
 
 	/*
-	 * Make processes trying to reserve really pick it up.
+	 * Since the final reference to this bo may not be dropped by
+	 * the current task we have to put a memory barrier here to make
+	 * sure the changes done in this function are always visible.
+	 *
+	 * This function only needs protection against the final kref_put.
 	 */
-	smp_mb__after_atomic_dec();
-	wake_up_all(&bo->event_queue);
+	smp_mb__before_atomic_dec();
 }
 
 static void ttm_bo_cleanup_refs_or_queue(struct ttm_buffer_object *bo)
@@ -543,68 +547,95 @@ static void ttm_bo_cleanup_refs_or_queue(struct ttm_buffer_object *bo)
 }
 
 /**
- * function ttm_bo_cleanup_refs
+ * function ttm_bo_cleanup_refs_and_unlock
  * If bo idle, remove from delayed- and lru lists, and unref.
  * If not idle, do nothing.
  *
+ * Must be called with lru_lock and reservation held, this function
+ * will drop both before returning.
+ *
  * @interruptible         Any sleeps should occur interruptibly.
- * @no_wait_reserve       Never wait for reserve. Return -EBUSY instead.
  * @no_wait_gpu           Never wait for gpu. Return -EBUSY instead.
  */
 
-static int ttm_bo_cleanup_refs(struct ttm_buffer_object *bo,
-			       bool interruptible,
-			       bool no_wait_reserve,
-			       bool no_wait_gpu)
+static int ttm_bo_cleanup_refs_and_unlock(struct ttm_buffer_object *bo,
+					  bool interruptible,
+					  bool no_wait_gpu)
 {
 	struct ttm_bo_device *bdev = bo->bdev;
+	struct ttm_bo_driver *driver = bdev->driver;
 	struct ttm_bo_global *glob = bo->glob;
 	int put_count;
-	int ret = 0;
+	int ret;
 
-retry:
 	spin_lock(&bdev->fence_lock);
-	ret = ttm_bo_wait(bo, false, interruptible, no_wait_gpu);
-	spin_unlock(&bdev->fence_lock);
+	ret = ttm_bo_wait(bo, false, false, true);
 
-	if (unlikely(ret != 0))
-		return ret;
+	if (ret && !no_wait_gpu) {
+		void *sync_obj;
 
-retry_reserve:
-	spin_lock(&glob->lru_lock);
+		/*
+		 * Take a reference to the fence and unreserve,
+		 * at this point the buffer should be dead, so
+		 * no new sync objects can be attached.
+		 */
+		sync_obj = driver->sync_obj_ref(&bo->sync_obj);
+		spin_unlock(&bdev->fence_lock);
 
-	if (unlikely(list_empty(&bo->ddestroy))) {
+		put_count = ttm_bo_del_from_lru(bo);
+
+		atomic_set(&bo->reserved, 0);
+		wake_up_all(&bo->event_queue);
 		spin_unlock(&glob->lru_lock);
-		return 0;
-	}
 
-	ret = ttm_bo_reserve_locked(bo, false, true, false, 0);
+		ttm_bo_list_ref_sub(bo, put_count, true);
 
-	if (unlikely(ret == -EBUSY)) {
-		spin_unlock(&glob->lru_lock);
-		if (likely(!no_wait_reserve))
-			ret = ttm_bo_wait_unreserved(bo, interruptible);
-		if (unlikely(ret != 0))
+		ret = driver->sync_obj_wait(sync_obj, false, interruptible);
+		driver->sync_obj_unref(&sync_obj);
+		if (ret) {
+			/*
+			 * Either the wait returned -ERESTARTSYS, or -EDEADLK
+			 * (radeon lockup) here. No effort is made to re-add
+			 * this bo to any lru list. Instead the bo only
+			 * appears on the delayed destroy list.
+			 */
 			return ret;
+		}
 
-		goto retry_reserve;
-	}
+		/*
+		 * remove sync_obj with ttm_bo_wait, the wait should be
+		 * finished, and no new wait object should have been added.
+		 */
+		spin_lock(&bdev->fence_lock);
+		ret = ttm_bo_wait(bo, false, false, true);
+		WARN_ON(ret);
+		spin_unlock(&bdev->fence_lock);
+		if (ret)
+			return ret;
 
-	BUG_ON(ret != 0);
+		spin_lock(&glob->lru_lock);
+		ret = ttm_bo_reserve_locked(bo, false, true, false, 0);
 
-	/**
-	 * We can re-check for sync object without taking
-	 * the bo::lock since setting the sync object requires
-	 * also bo::reserved. A busy object at this point may
-	 * be caused by another thread recently starting an accelerated
-	 * eviction.
-	 */
+		/*
+		 * We raced, and lost, someone else holds the reservation now,
+		 * and is probably busy in ttm_bo_cleanup_memtype_use.
+		 *
+		 * Even if it's not the case, because we finished waiting any
+		 * delayed destruction would succeed, so just return success
+		 * here.
+		 */
+		if (ret) {
+			spin_unlock(&glob->lru_lock);
+			return 0;
+		}
+	} else
+		spin_unlock(&bdev->fence_lock);
 
-	if (unlikely(bo->sync_obj)) {
+	if (ret || unlikely(list_empty(&bo->ddestroy))) {
 		atomic_set(&bo->reserved, 0);
 		wake_up_all(&bo->event_queue);
 		spin_unlock(&glob->lru_lock);
-		goto retry;
+		return ret;
 	}
 
 	put_count = ttm_bo_del_from_lru(bo);
@@ -647,9 +678,13 @@ static int ttm_bo_delayed_delete(struct ttm_bo_device *bdev, bool remove_all)
 			kref_get(&nentry->list_kref);
 		}
 
-		spin_unlock(&glob->lru_lock);
-		ret = ttm_bo_cleanup_refs(entry, false, !remove_all,
-					  !remove_all);
+		ret = ttm_bo_reserve_locked(entry, false, !remove_all, false, 0);
+		if (!ret)
+			ret = ttm_bo_cleanup_refs_and_unlock(entry, false,
+							     !remove_all);
+		else
+			spin_unlock(&glob->lru_lock);
+
 		kref_put(&entry->list_kref, ttm_bo_release_list);
 		entry = nentry;
 
@@ -803,9 +838,13 @@ retry:
 	kref_get(&bo->list_kref);
 
 	if (!list_empty(&bo->ddestroy)) {
-		spin_unlock(&glob->lru_lock);
-		ret = ttm_bo_cleanup_refs(bo, interruptible,
-					  no_wait_reserve, no_wait_gpu);
+		ret = ttm_bo_reserve_locked(bo, interruptible, no_wait_reserve, false, 0);
+		if (!ret)
+			ret = ttm_bo_cleanup_refs_and_unlock(bo, interruptible,
+							     no_wait_gpu);
+		else
+			spin_unlock(&glob->lru_lock);
+
 		kref_put(&bo->list_kref, ttm_bo_release_list);
 
 		return ret;
@@ -1799,8 +1838,9 @@ static int ttm_bo_swapout(struct ttm_mem_shrink *shrink)
 		kref_get(&bo->list_kref);
 
 		if (!list_empty(&bo->ddestroy)) {
-			spin_unlock(&glob->lru_lock);
-			(void) ttm_bo_cleanup_refs(bo, false, false, false);
+			ttm_bo_reserve_locked(bo, false, false, false, 0);
+			ttm_bo_cleanup_refs_and_unlock(bo, false, false);
+
 			kref_put(&bo->list_kref, ttm_bo_release_list);
 			spin_lock(&glob->lru_lock);
 			continue;