From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D488DC2D0A3 for ; Thu, 5 Nov 2020 00:07:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6169D20867 for ; Thu, 5 Nov 2020 00:07:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="GYL7zB9k" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730241AbgKEAHY (ORCPT ); Wed, 4 Nov 2020 19:07:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728323AbgKEAHY (ORCPT ); Wed, 4 Nov 2020 19:07:24 -0500 Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 09947C0613CF for ; Wed, 4 Nov 2020 16:07:24 -0800 (PST) Received: by mail-qk1-x744.google.com with SMTP id s14so207367qkg.11 for ; Wed, 04 Nov 2020 16:07:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Ngyyrd+7YeTNUGD6RWJvWNxN+aS5ZNXXD5GfmJI2C9s=; b=GYL7zB9kVnVbLQLfDQ7CNvcYkF80/hffFnJdGh/buXrdEBiQIoNyV5ftYv4B945PWp texok6bTLa0kCiwJq+Cthi41KeaJ5RcS9Z+VTq2aLUtb9D9MRdwtCUaB9fY0dTgyZsIV 7ISBqrMcwMO4xPZuveqJcG8EXqzfgERI4tSQiOfpwnGdTuehBS3je4ko+nA5bwX5QFkw sL1tyX8Ru6jb/558l/WcK7QZ4MSVQCyDEjEI6Zl+7PdmjCd6G26+QRi/Pbd2sdBFkJ+P j5gIBIaQNzHPdpgOKgGX8enUFCck1hoJG8Y7yzTGpdkTJZmKg4hjq2HXs+S4D7O2K+qK qvzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Ngyyrd+7YeTNUGD6RWJvWNxN+aS5ZNXXD5GfmJI2C9s=; b=CPZSgDfuXU1PKmcTullX5Zx1d9zS7vTmujcW1QqVr+6fGHXvMHXzvz1VMRcx30YRLX c3M7VQF5nu60k/Fs/GxhAopp/c0lFgdppDBMPzfYRtXt/TO0gxr6lprnhU8gsX+f+Z63 vkzVolUxJomx2pY89x6JU8ReumhtxOyMEZY/r/Mq6f94ahD8G0Cfi9xAO+Sld8kQFyye 6mU2X/vKbtw9ERiRI6c1ogjG4ot1zCmU57mKC2IiDfTsx+L6JUiLp4TXigGrY/GgKgxx O7TEElq5+IvLfbSVZDwOA2qS3TGgbwwGwAeUTJH6VEHoC4HiSbBO/tPHTGmCbUtwq1MK QGRQ== X-Gm-Message-State: AOAM531DlewqzHHO0TaKFEdq7EajhjOOLckJIej4wArRJW/QiJLk6r5F rfT80ezUv3LhqEiPpeCtqtKOSw== X-Google-Smtp-Source: ABdhPJwZ2zq4kbXGpuRR9Mw0TOTWAdjlsCRldJcSUyYR7qPJ2IsfD0KFKtR9c++Nf7n/gtYO2KsuBQ== X-Received: by 2002:a37:a642:: with SMTP id p63mr749300qke.5.1604534843136; Wed, 04 Nov 2020 16:07:23 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id y3sm1433098qto.2.2020.11.04.16.07.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Nov 2020 16:07:22 -0800 (PST) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kaSoD-00Gl4C-NS; Wed, 04 Nov 2020 20:07:21 -0400 Date: Wed, 4 Nov 2020 20:07:21 -0400 From: Jason Gunthorpe To: Jianxin Xiong Cc: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org, Doug Ledford , Leon Romanovsky , Sumit Semwal , Christian Koenig , Daniel Vetter Subject: Re: [PATCH v7 4/5] RDMA/mlx5: Support dma-buf based userspace memory region Message-ID: <20201105000721.GY36674@ziepe.ca> References: <1604527595-39736-1-git-send-email-jianxin.xiong@intel.com> <1604527595-39736-5-git-send-email-jianxin.xiong@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1604527595-39736-5-git-send-email-jianxin.xiong@intel.com> Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Wed, Nov 04, 2020 at 02:06:34PM -0800, Jianxin Xiong wrote: > + umem = ib_umem_dmabuf_get(&dev->ib_dev, offset, length, fd, access_flags, > + &mlx5_ib_dmabuf_attach_ops); > + if (IS_ERR(umem)) { > + mlx5_ib_dbg(dev, "umem get failed (%ld)\n", PTR_ERR(umem)); > + return ERR_PTR(PTR_ERR(umem)); > + } > + > + mr = alloc_mr_from_cache(pd, umem, virt_addr, access_flags); It is very subtle, but this calls mlx5_umem_find_best_pgsz() which calls ib_umem_find_best_pgsz() which goes over the SGL to determine the page size to use. As part of this it does validation of the IOVA vs first page offset vs first page dma address. These little details come into play if the IOVA and offset are not PAGE_SIZE aligned, which is very possible if the dma buf exporter or system PAGE_SIZE is over 4k. In other words, the dma_address of the first SGL must be the page aligned starting point of the MR. Since the 'skip' approach is being done when breaking the SGL into blocks the ib_umem_find_best_pgsz() sees an invalid page size. Slicing it has to be done in a way that gives a properly formed SGL. My suggestion is to just change the SGL in place. Iterate to the starting SGE in the SGL and assign it to the sg table, modify it to have a offset dma_address and reduced length Count the number of SGEs to span the remaning range and use that as the new nmaped Edit the last SGE to have a reduced length Upon unmap undo the edits so the exporter doesn't see the mangled SGL. It would be saner if the exporter could do this, but oh well. Approximately like this: struct ib_umem *umem = &umem_p->umem; struct scatterlist *sg; int i; for_each_sg(umem_p->umem.sg_head.sgl, sg, umem_p->umem.nmap, i) { if (cur + sg_dma_len(sg) > ALIGN_DOWN(umem->address, PAGE_SIZE)) { unsigned long offset; umem_p->first_sg = sg; umem_p->first_dma_address = sg->dma_address; umem_p->first_dma_length = sg_dma_len(sg); umem_p->first_length = sg->length; offset = ALIGN_DOWN(umem->addressm PAGE_SIZE) - cur; sg->dma_address += offset; sg_dma_len(sg) -= offset; sg->length -= offset; } if (ALIGN(umem->address + umem->length, PAGE_SIZE) < cur + sg_dma_len(sg)) { unsigned long trim; umem_p->last_sg = sg; umem_p->last_dma_length = sg_dma_len(sg); umem_p->last_length = sg->length; trim = cur + sg_dma_len(sg) - ALIGN(umem->address + umem->length, PAGE_SIZE); sg_dma_len(sg) -= trim; sg->length -= trim; return npages; } cur += sg_dma_len(sg); } /* It is essential that the length of the SGL exactly match the adjusted page aligned length of umem->length */ return -EINVAL; Further, this really only works if the umem->page_size is locked to 4k because this doesn't have code to resize the MKEY, or change the underlying page size when the SGL changes. So, I'd say put something like the above in the core code to validate and properly form the umem->sgl Then modify the alloc_mr_from_cache to use only PAGE_SIZE: if (umem->is_dma_buf) page_size = ib_umem_find_best_pgsz(umem, PAGE_SIZE, iova); else page_size = mlx5_umem_find_best_pgsz(umem, mkc, log_page_size, 0, iova); Jason