TTM Locking order of bo::reserve -> vm::mmap_sem

[Thomas Hellstrom <thellstrom@vmware.com>]

[-- Attachment #1: Type: text/plain, Size: 1771 bytes --]

Hi!

As discussed previously the current locking order in TTM of these locks 
is bo::reserve -> vm::mmap_sem. This leads to a hack in
the TTM fault() handle to try and revert the locking order. If a 
tryreserve failed, we tried to have the vm code release the mmap_sem() 
and then schedule, to give the holder of bo::reserve a chance to release 
the lock. This solution is no longer legal, since we've been more or 
less kindly asked to remove the set_need_resched() call.

Maarten has proposed to invert the locking order. I've previously said I 
had no strong preference. The current locking order dates back from the 
time when TTM wasn't using unmap_mapping_range() but walked the page 
tables itself, updating PTEs as needed. Furthermore it was needed for 
user bos that used get_user_pages() in the TTM populate and swap-in 
methods. User-bos were removed some time ago but I'm looking at 
re-adding them. They would suite the VMware model of cached-only pages 
very well. I see uses both in the gallium API, XA's DMA functionality 
and openCL.

We would then need a somewhat nicer way to invert the locking order. 
I've attached a solution that ups the mmap_sem and then reserves, but 
due to how the fault API is done, we then need to release the reserve 
and retry the fault. This of course opens up for starvation, but I don't 
think starvation at this point is very likely: One thread being refused 
to write or read from a buffer object because the GPU is continously 
busy with it. If this *would* become a problem, it's probably possible 
to modify the fault code to allow us to hold locks until the retried 
fault, but that would be a bit invasive, since it touches the arch code....

Basically I'm proposing to keep the current locking order.


/Thomas



[-- Attachment #2: vm_lock.diff --]
[-- Type: text/x-patch, Size: 1050 bytes --]

diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c
index 1006c15..55c487d 100644
--- a/drivers/gpu/drm/ttm/ttm_bo_vm.c
+++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c
@@ -61,13 +61,22 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf)
 	/*
 	 * Work around locking order reversal in fault / nopfn
 	 * between mmap_sem and bo_reserve: Perform a trylock operation
-	 * for reserve, and if it fails, retry the fault after scheduling.
+	 * for reserve, and if it fails, retry the fault after releasing
+	 * the mmap_sem and waiting.
 	 */
 
 	ret = ttm_bo_reserve(bo, true, true, false, 0);
 	if (unlikely(ret != 0)) {
-		if (ret == -EBUSY)
-			set_need_resched();
+		if (ret == -EBUSY) {
+			if ((vmf->flags & FAULT_FLAG_ALLOW_RETRY) &&
+			    !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) {
+				up_read(&vma->vm_mm->mmap_sem);
+				ttm_bo_reserve_nolru(bo, true, false, false,
+						     NULL);
+				ww_mutex_unlock(&bo->resv->lock);
+				return VM_FAULT_RETRY;
+			}
+		}
 		return VM_FAULT_NOPAGE;
 	}
 

[-- Attachment #3: Type: text/plain, Size: 159 bytes --]

_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/dri-devel