From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C27A3C43464 for ; Fri, 18 Sep 2020 13:59:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8814821D42 for ; Fri, 18 Sep 2020 13:59:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="dZS+Pz8u" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726849AbgIRN7R (ORCPT ); Fri, 18 Sep 2020 09:59:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726696AbgIRN7R (ORCPT ); Fri, 18 Sep 2020 09:59:17 -0400 Received: from mail-qt1-x842.google.com (mail-qt1-x842.google.com [IPv6:2607:f8b0:4864:20::842]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F05CC0613D0 for ; Fri, 18 Sep 2020 06:59:17 -0700 (PDT) Received: by mail-qt1-x842.google.com with SMTP id r8so4963250qtp.13 for ; Fri, 18 Sep 2020 06:59:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=iDzy1sMgIGxVws9oU0uRMBwks6chpSBoUJBA5ZAPs6Y=; b=dZS+Pz8uPa7xZedC6ywyuba3H9rxNa8iD2FLUWjM9LYjV/vp0Nnwuxl4sIAH/WzJmo Z71YmWaPcUeza3JssOVlprHU4cGQ3yjZxah7wyneWO2mh86SvelVUNGd45Cqk9Yie1oG 617Nev2Z7twgiRUh132ng7fDDjipACTM4bU0rYD4M+AA9ZUYUyf0U4tF3fwh8UPfh7cR ijQl8xM6r/btbx2DzAHfRisLO48r+cfseEBb/dQ/NLLKA9FIWvNEWjokBXGDc2fndUqW ScKssfiqwnp2XUoY0uD9YyQ4SnhhADwwA+/HFxDHtK2d76LmBhaSyZ9tPLqOZUnoOxnr 5sUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=iDzy1sMgIGxVws9oU0uRMBwks6chpSBoUJBA5ZAPs6Y=; b=I4ay766Hv7MQ0deuYgg1OZn6S+zjn7WD3KfVdUKFrkRIHGqExNI7oHGjMKh49DvXuR jgG6rmKD+fnLhLsDn+K1mTWzTLhKvjkZy427LfFoZNRCzX1K09QkAFn1AJY+h+q96zDX 8LL6bETf0ql04P2fchZJ7ea4GTTchtbb3mVKefjhuL0pj1KbrD+cPzjhaOktCWm0I3XC MH5i0geF89HveWQcZ5oQgkrnqvPJUksvzn9QJ7Yxk5oSzOYOZ5ckaZipj+YjHRX36lPy hr4WLxa9/saxzzf7ndYqK+4UuBjR/5hFRsd+AbHudeKTTyj+fAWBTtb2pp3SLkA4m6KR xJAA== X-Gm-Message-State: AOAM531VoSWIII2p+YmsgRWBe6m7bCfYepSqR4HNZpYQVnL1oYEFPKjN o6r7PAHPlQfJNiIDXEdzaO12cw== X-Google-Smtp-Source: ABdhPJxJEjIjl7tbKTQubndOek+3OibCxEDhsqhs1uDEM/H/kHLq/g77HjdghQ/1oNTy7EDlsPqX1w== X-Received: by 2002:ac8:6e90:: with SMTP id c16mr33479531qtv.17.1600437556361; Fri, 18 Sep 2020 06:59:16 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id v18sm2144966qtq.15.2020.09.18.06.59.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2020 06:59:15 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kJGux-001HHG-3h; Fri, 18 Sep 2020 10:59:15 -0300 Date: Fri, 18 Sep 2020 10:59:15 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Greg Kroah-Hartman , izur@habana.ai, Gal Pressman , Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org, Olof Johansson Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200918135915.GT8409@ziepe.ca> References: <20200917171833.GJ8409@ziepe.ca> <0b21db8d-1061-6453-960b-8043951b3bad@amazon.com> <20200918115601.GP8409@ziepe.ca> <20200918121621.GQ8409@ziepe.ca> <20200918125014.GR8409@ziepe.ca> <20200918132645.GS8409@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Fri, Sep 18, 2020 at 04:49:25PM +0300, Oded Gabbay wrote: > On Fri, Sep 18, 2020 at 4:26 PM Jason Gunthorpe wrote: > > > > On Fri, Sep 18, 2020 at 04:02:24PM +0300, Oded Gabbay wrote: > > > > > The problem with MR is that the API doesn't let us return a new VA. It > > > forces us to use the original VA that the Host OS allocated. > > > > If using the common MR API you'd have to assign a unique linear range > > in the single device address map and record both the IOVA and the MMU > > VA in the kernel struct. > > > > Then when submitting work using that MR lkey the kernel will adjust > > the work VA using the equation (WORK_VA - IOVA) + MMU_VA before > > forwarding to HW. > > > We can't do that. That will kill the performance. If for every > submission I need to modify the packet's contents, the throughput will > go downhill. You clearly didn't read where I explained there is a fast path and slow path expectation. > Also, submissions to our RDMA qmans are coupled with submissions to > our DMA/Compute QMANs. We can't separate those to different API calls. > That will also kill performance and in addition, will prevent us from > synchronizing all the engines. Not sure I see why this is a problem. I already explained the fast device specific path. As long as the kernel maintains proper security when it processes submissions the driver can allow objects to cross between the two domains. Jason