From mboxrd@z Thu Jan 1 00:00:00 1970 From: Thomas Hellstrom Subject: TTM Locking order of bo::reserve -> vm::mmap_sem Date: Mon, 21 Oct 2013 10:48:57 +0200 Message-ID: <5264EA79.3050505@vmware.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------070303050700000509040803" Return-path: Received: from smtp-outbound-2.vmware.com (smtp-outbound-2.vmware.com [208.91.2.13]) by gabe.freedesktop.org (Postfix) with ESMTP id 4951BE5E10 for ; Mon, 21 Oct 2013 01:49:02 -0700 (PDT) List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org Errors-To: dri-devel-bounces+sf-dri-devel=m.gmane.org@lists.freedesktop.org To: Maarten Lankhorst Cc: "dri-devel@lists.freedesktop.org" List-Id: dri-devel@lists.freedesktop.org This is a multi-part message in MIME format. --------------070303050700000509040803 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi! As discussed previously the current locking order in TTM of these locks is bo::reserve -> vm::mmap_sem. This leads to a hack in the TTM fault() handle to try and revert the locking order. If a tryreserve failed, we tried to have the vm code release the mmap_sem() and then schedule, to give the holder of bo::reserve a chance to release the lock. This solution is no longer legal, since we've been more or less kindly asked to remove the set_need_resched() call. Maarten has proposed to invert the locking order. I've previously said I had no strong preference. The current locking order dates back from the time when TTM wasn't using unmap_mapping_range() but walked the page tables itself, updating PTEs as needed. Furthermore it was needed for user bos that used get_user_pages() in the TTM populate and swap-in methods. User-bos were removed some time ago but I'm looking at re-adding them. They would suite the VMware model of cached-only pages very well. I see uses both in the gallium API, XA's DMA functionality and openCL. We would then need a somewhat nicer way to invert the locking order. I've attached a solution that ups the mmap_sem and then reserves, but due to how the fault API is done, we then need to release the reserve and retry the fault. This of course opens up for starvation, but I don't think starvation at this point is very likely: One thread being refused to write or read from a buffer object because the GPU is continously busy with it. If this *would* become a problem, it's probably possible to modify the fault code to allow us to hold locks until the retried fault, but that would be a bit invasive, since it touches the arch code.... Basically I'm proposing to keep the current locking order. /Thomas --------------070303050700000509040803 Content-Type: text/x-patch; name="vm_lock.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="vm_lock.diff" diff --git a/drivers/gpu/drm/ttm/ttm_bo_vm.c b/drivers/gpu/drm/ttm/ttm_bo_vm.c index 1006c15..55c487d 100644 --- a/drivers/gpu/drm/ttm/ttm_bo_vm.c +++ b/drivers/gpu/drm/ttm/ttm_bo_vm.c @@ -61,13 +61,22 @@ static int ttm_bo_vm_fault(struct vm_area_struct *vma, struct vm_fault *vmf) /* * Work around locking order reversal in fault / nopfn * between mmap_sem and bo_reserve: Perform a trylock operation - * for reserve, and if it fails, retry the fault after scheduling. + * for reserve, and if it fails, retry the fault after releasing + * the mmap_sem and waiting. */ ret = ttm_bo_reserve(bo, true, true, false, 0); if (unlikely(ret != 0)) { - if (ret == -EBUSY) - set_need_resched(); + if (ret == -EBUSY) { + if ((vmf->flags & FAULT_FLAG_ALLOW_RETRY) && + !(vmf->flags & FAULT_FLAG_RETRY_NOWAIT)) { + up_read(&vma->vm_mm->mmap_sem); + ttm_bo_reserve_nolru(bo, true, false, false, + NULL); + ww_mutex_unlock(&bo->resv->lock); + return VM_FAULT_RETRY; + } + } return VM_FAULT_NOPAGE; } --------------070303050700000509040803 Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ dri-devel mailing list dri-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/dri-devel --------------070303050700000509040803--