From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB6E6C2D0E2 for ; Tue, 22 Sep 2020 11:42:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4ED72214F1 for ; Tue, 22 Sep 2020 11:42:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ziepe.ca header.i=@ziepe.ca header.b="oLM4ZSFX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726554AbgIVLmN (ORCPT ); Tue, 22 Sep 2020 07:42:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40058 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726633AbgIVLlb (ORCPT ); Tue, 22 Sep 2020 07:41:31 -0400 Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B7D7BC061755 for ; Tue, 22 Sep 2020 04:41:03 -0700 (PDT) Received: by mail-qk1-x741.google.com with SMTP id v123so18573897qkd.9 for ; Tue, 22 Sep 2020 04:41:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=yhD10jJduf1dgKkP68ou+U9mMDBt7tD8b18c/5OZcv0=; b=oLM4ZSFXdQRQNrMGFA4YSb6qTb/uhf0RGHa3k3SHj0ZvzobirHOgZvwEubXTdnmD8X g0x2RU5kwhc8KV33C74sw3XINlT28bOzpw2rBecqgyf3FXT7ZXS47XhIkkoeQL9f2Jfe EIQqCmlHxu3fRAjdPUH9yINPWViY4hmkzh5lHkLBKDKogJ630hwcVUz6cn09PmzhnOO/ KL+VM28MhtYEZFAeBEVy/+CFbM/Pi3l7fJJG1rbNdVoWuN35JeMvqO0PShhsgCysTdlE HRLoZtFkdaOnoxgQxky1XAaqQenuLuplSVwOQa/0fViN1SStlWfN6no7PzOe/A2XUkfw P8Pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=yhD10jJduf1dgKkP68ou+U9mMDBt7tD8b18c/5OZcv0=; b=pyy88w2Rqx30ImXPBX0MvvEODszD0HLu7lLrHiOfQ+k4KaZ6MmpRoO9eip6omNKRk1 6uGuu2IYVMbWpvEdJDaEEA+pGRbOR1vJhYvP99xnyfEjQPtXQ6IqQlMbYQdzpF0Ka6Fn G06uCILJ4ZN78uuayT/GazUFAW8Fsfq+ixDCgJWc1oZlTOUPYLdtoVJHGC8prhRDhP44 wHtVY4aVgzanHxWg5gksW33kXe01+mP9ysQbWzTVtwinq6ELCgH9boRNNnt88Jezq5OE DMUTaR6nNwzmnpHX25oMgo3bTO94OdYUR2WHByxRQBqVPF4lBTIswDfHb+mw/GRCW3+2 aKLw== X-Gm-Message-State: AOAM530QSVv3M9kdSt1DRj70LMEu8VcwoTPiyfFRETsJXRrRW/uWQC1U j12BdduWyG6DMe6eb4o315t7cw== X-Google-Smtp-Source: ABdhPJxoKo95FitIWDwXAcSfyfcOjyXkE53rqNXh2wpAj0K7EF3gP3JDLJjqBxC4M8Uz6fncjY2f4g== X-Received: by 2002:a37:7f82:: with SMTP id a124mr4296755qkd.70.1600774862931; Tue, 22 Sep 2020 04:41:02 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-156-34-48-30.dhcp-dynamic.fibreop.ns.bellaliant.net. [156.34.48.30]) by smtp.gmail.com with ESMTPSA id 192sm11863559qkn.9.2020.09.22.04.41.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 22 Sep 2020 04:41:02 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1kKgfN-0030RG-Hf; Tue, 22 Sep 2020 08:41:01 -0300 Date: Tue, 22 Sep 2020 08:41:01 -0300 From: Jason Gunthorpe To: Gal Pressman Cc: Oded Gabbay , Greg Kroah-Hartman , izur@habana.ai, Jakub Kicinski , "Linux-Kernel@Vger. Kernel. Org" , netdev@vger.kernel.org, SW_Drivers , "David S. Miller" , Andrew Lunn , Florian Fainelli , linux-rdma@vger.kernel.org, Olof Johansson Subject: Re: [PATCH v3 00/14] Adding GAUDI NIC code to habanalabs driver Message-ID: <20200922114101.GE8409@ziepe.ca> References: <20200918132645.GS8409@ziepe.ca> <20200918135915.GT8409@ziepe.ca> <20200918141909.GU8409@ziepe.ca> <20200918150735.GV8409@ziepe.ca> <20200918152852.GW8409@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Mon, Sep 21, 2020 at 02:22:02PM +0300, Gal Pressman wrote: > What is considered a RoCE port in this case if it's not compliant with RoCE? > Sounds like it's an implementation of RDMA over ethernet, not RoCE. > Does GAUDI support UD/RC/.. QPs? Is it using a proprietary wire protocol? > (BTW, Oded claims it's similar to nvlink, how is nvlink's implementation > exposed? Or is it closed source?) I think Oded was drawing a parallel to how nvlink is integral with the compute element. From Oded's descriptions I don't think it is much like nvlink at all. > Jason, how do you imagine GAUDI in the RDMA subsystem? Userspace control path > verbs (used by hl-thunk?) and all data path verbs exposed as kverbs (used by > habanalabs driver)? > So neither any userspace verbs apps could use it nor kernel ULPs? Based on what Oded described it seems like a reasonable RDMA device with some limitations around MR IOVA. Looks like the desire is to create a RDMA WR and CQ ring in userspace, and then co-mingle that with the compute side of the device. So instead of doing the special IOCTL and mmap against the compute FD it would create a RDMA QP and RDMA CQ, use dv to access the raw internals, and the propritary stack would have exactly the same stuff it would have had with the misc ioctl. But, completely separately, they'd also have to implement some of verbs which serves as the open source userspace showing how this HW works. What that is depends largely on what their HW can do, and if they want to connect to UCX/mpi/libfabric/etc A bunch of ioctl stubs or a few tests is far below our standard in RDMA. There may have been some argument that the compute side of this device has no industry standards so should be a drivers/misc, but HPC networking *does* have extensive standards and extensive open source software stacks. It is very hard for me to see how a device in this market could be competitive without integrating with that stuff. Jason