From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF79CC43463 for ; Fri, 18 Sep 2020 12:50:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5C79C21D7B for ; Fri, 18 Sep 2020 12:50:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="QyJDbsqV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726707AbgIRMuR (ORCPT ); Fri, 18 Sep 2020 08:50:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40564 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726239AbgIRMuQ (ORCPT ); Fri, 18 Sep 2020 08:50:16 -0400 Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E5E3AC061756 for ; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) Received: by mail-qk1-x743.google.com with SMTP id d20so5884718qka.5 for ; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=KAoPuTEzUmIe2wdxaRxMUqqp3YdtmLOZgXAXpjKRNsk=; b=QyJDbsqV+Clftle8r/t/pW701Y/xrp6nUJ3MlzmLFCL32GerNaErkVHXxBR5FtLcuE DjRIlx9mDzvwBV1qDo2wZUwIjWI27lrH2c1lCsovl4kYcbSdvNnau+k8cP18rJSDvG1L sGi0QcjouM6J0NHgJEdxsmuLb3yLqhb99Dq2cRUle5JcDjD22B1rCwyCq9YY86oIVbCw QqJzm+r2mcrE75S6SO/bTsMGVaezkZTEKr9uRuGfKtn3NSzQPYnRQGxjpOKopN4HgsfS kFr7dzmb/BHaUb7btWWeU259YSwwO8x+I1AuoJLMaULNgrqkMdz0EImj6xmYaymMrEzv g1CA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=KAoPuTEzUmIe2wdxaRxMUqqp3YdtmLOZgXAXpjKRNsk=; b=VCJe+zC/C6FaiepL9PRTHJTQvjKlDaY+iVajMobpyw0ZIum99K5gboQwiCpNXnxic9 HIgPblwqsaeXDjFqp+wgGryI3EJ7WmymD4oDb+bqdtaT3tu8bfwXImUUqQNqt+ZCFmxh bsvCDjFEvOWKSM4wSCnLIhGHQllY/TcZCE8ol1NZ5e3B24CG2SO3NjJrYig/gn9Spfoq IAJ41GzriLPp77oZpLkDxgbyss6Vue3d68nsGCrXwsCxblGaiTu2nYsZ7Fz9XiiNPBQo pK0uk254ldIxZnhVNlaVXB6WOE98jxBFkQ69Wb+3fRwmu+d40emuGUk8CYfD6G311ciY txug== X-Gm-Message-State: AOAM5318qhVNVNez9Z40RoNqWL6WyqUBCzSmvdQq5a0fUqfFYP1tBsNU M/jxLmM/C3yF5kFw0mPj2sYHeQ== X-Google-Smtp-Source: ABdhPJw907l05mss1ldyLf/9wiOK6C5YwGtGpEhTGyHGuXCXNATWIp2kIEIi+FlDBdV5O02aKx9WRQ== X-Received: by 2002:a37:4a57:: with SMTP id x84mr32144214qka.17.1600433416085; Fri, 18 Sep 2020 05:50:16 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id i5sm1927930qko.86.2020.09.18.05.50.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Sep 2020 05:50:15 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kJFqA-000z9a-Ua; Fri, 18 Sep 2020 09:50:14 -0300 Date: Fri, 18 Sep 2020 09:50:14 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: izur@habana.ai, Gal Pressman , Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , Greg Kroah-Hartman , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200918125014.GR8409@ziepe.ca> References: <20200915171022.10561-1-oded.gabbay@gmail.com> <20200915133556.21268811@kicinski-fedora-pc1c0hjn.dhcp.thefacebook.com> <20200917171833.GJ8409@ziepe.ca> <0b21db8d-1061-6453-960b-8043951b3bad@amazon.com> <20200918115601.GP8409@ziepe.ca> <20200918121621.GQ8409@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Fri, Sep 18, 2020 at 03:34:54PM +0300, Oded Gabbay wrote: > > > Another example is that the submission of WQ is done through our QMAN > > > mechanism and is NOT mapped to userspace (due to the restrictions you > > > mentioned above and other restrictions). > > > > Sure, other RDMA drivers also require a kernel ioctl for command > > execution. > > > > In this model the MR can be a software construct, again representing a > > security authorization: > > > > - A 'full process' MR, in which case the kernel command excution > > handles dma map and pinning at command execution time > > - A 'normal' MR, in which case the DMA list is pre-created and the > > command execution just re-uses this data > > > > The general requirement for RDMA is the same as DRM, you must provide > > enough code in rdma-core to show how the device works, and minimally > > test it. EFA uses ibv_ud_pingpong, and some pyverbs tests IIRC. > > > > So you'll want to arrange something where the default MR and PD > > mechanisms do something workable on this device, like auto-open the > > misc FD when building the PD, and support the 'normal' MR flow for > > command execution. > > I don't know how we can support MR because we can't support any > virtual address on the host. Our internal MMU doesn't support 64-bits. > We investigated in the past, very much wanted to use IBverbs but > didn't figure out how to make it work. > I'm adding Itay here and he can also shed more details on that. I'm not sure what that means, if the driver intends to DMA from process memory then it certainly has a MR concept. MRs can control the IOVA directly so if you say the HW needs a MR IOVA < 2**32 then that is still OK. Jason