From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C098DC433E0 for ; Thu, 23 Jul 2020 06:58:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D00220709 for ; Thu, 23 Jul 2020 06:58:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cAzIUAHn" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1725849AbgGWG6s (ORCPT ); Thu, 23 Jul 2020 02:58:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725774AbgGWG6r (ORCPT ); Thu, 23 Jul 2020 02:58:47 -0400 Received: from mail-pl1-x641.google.com (mail-pl1-x641.google.com [IPv6:2607:f8b0:4864:20::641]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58759C0619DC for ; Wed, 22 Jul 2020 23:58:47 -0700 (PDT) Received: by mail-pl1-x641.google.com with SMTP id o1so2145859plk.1 for ; Wed, 22 Jul 2020 23:58:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=LrTc8Zt31PW3XHmOCyLW32ACrnszH9oJOXzPvegPeqk=; b=cAzIUAHnVv/5g4mamnTD232kAyVtHmbSh7RjCNqiQfc2sYeE8/q1Ydm/quR7WKythl oA449QYXIcUdh96lhMWq27jkq6hbLCVVyDPuhUfTcwZRgSf3+2pqiSH/X/sRw2ijKO0m s8+juGFJzv5AfP5rz1PPaHMfRWilhJ+D7KI7vKgrQR5PDDeok3zrNt9dGixZc+KjN0N1 M1cu3Mwvifmz4Dh9YFVSTrNXdDkQ8oAGEOxmBaGUeOsh4WMhkuaE9FtafjWu45cxPVUt Ov3e9B0bf5d35MCF3YwQigkPPWH5TlVIwAShqk/1lzUehSB7R6B75FKXWYHXMQXBsO+4 OYmA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=LrTc8Zt31PW3XHmOCyLW32ACrnszH9oJOXzPvegPeqk=; b=KR60gP98mcjWJP5JP2Ze8TXT9rMB7iLCyv6qWDcZSxnL21fD1UERWsnAXV+WJPvqrG TMzwMEv5FyXHci62SvVeghzDICaQnuAj9n8dEu4nFIjfYjS2kc9JHaNgtsRevFXqS3+C B7nK0EIbnThtNTlzMtYSriq1n9xh7jLp3uATmHvDweVgvnMsVVbwoSZPDPh0VGf1vAfa nJvGpg9t3PAZNLU/GgHbj3V91rw1Jixv+Dkjzwf9gOIl0XwacZgzMF6chjLwg33IfBGA 7l9hJoffNuNU2z/ppvoEs7+dpKm7087wf5eOn2z7MJLqTwSaTx54zD9XN9upvX1C32MQ 33tA== X-Gm-Message-State: AOAM530iMHrplFH4FysTx7m7aTnlkcR5kbXKkVY62Sqno3OuaoyW4iBu /gq/EsfN+mloBLSzvYijQps= X-Google-Smtp-Source: ABdhPJx+m5PY36qke8pqnITKatfwO7LUZBUAb0bFuKeFVlqxPIZLpaHaagEhHTJ9q7bCMi+n0E5z2Q== X-Received: by 2002:a17:902:fe11:: with SMTP id g17mr2786398plj.145.1595487526697; Wed, 22 Jul 2020 23:58:46 -0700 (PDT) Received: from [10.75.65.24] ([118.201.220.138]) by smtp.gmail.com with ESMTPSA id p11sm1711061pjb.3.2020.07.22.23.58.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 22 Jul 2020 23:58:46 -0700 (PDT) Subject: Re: FW: [PATCH for-next] RDMA/rxe: Remove pkey table To: Kamal Heib Cc: Yanjun Zhu , linux-rdma@vger.kernel.org, Doug Ledford , Jason Gunthorpe References: <20200721101618.686110-1-kamalheib1@gmail.com> <20200723055723.GA828525@kheib-workstation> From: Zhu Yanjun Message-ID: <7a6d602f-1adc-cc36-5a11-e0beb6e31cec@gmail.com> Date: Thu, 23 Jul 2020 14:58:41 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.0 MIME-Version: 1.0 In-Reply-To: <20200723055723.GA828525@kheib-workstation> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 7/23/2020 1:57 PM, Kamal Heib wrote: > On Wed, Jul 22, 2020 at 10:09:04AM +0800, Zhu Yanjun wrote: >> On Tue, Jul 21, 2020 at 7:28 PM Yanjun Zhu wrote: >>> >>> >>> -----Original Message----- >>> From: Kamal Heib >>> Sent: Tuesday, July 21, 2020 6:16 PM >>> To: linux-rdma@vger.kernel.org >>> Cc: Yanjun Zhu ; Doug Ledford ; Jason Gunthorpe ; Kamal Heib >>> Subject: [PATCH for-next] RDMA/rxe: Remove pkey table >>> >>> The RoCE spec require from RoCE devices to support only the defualt pkey, While the rxe driver maintain a 64 enties pkey table and use only the first entry. With that said remove the maintaing of the pkey table and used the default pkey when needed. >>> >> Hi Kamal >> >> After this patch is applied, do you make tests with SoftRoCE and mlx hardware? >> >> The SoftRoCE should work well with the mlx hardware. >> >> Zhu Yanjun >> > Hi Zhu, > > Yes, please see below: > > $ ibv_rc_pingpong -d mlx5_0 -g 11 > local address: LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121 Can you make tests with GSI QP? Zhu Yanjun > remote address: LID 0x0000, QPN 0x000011, PSN 0xd67210, GID ::ffff:172.31.40.122 > 8192000 bytes in 0.03 seconds = 2194.56 Mbit/sec > 1000 iters in 0.03 seconds = 29.86 usec/iter > > $ ibv_rc_pingpong -d rxe0 -g 1 rdma-dev-21 > local address: LID 0x0000, QPN 0x000011, PSN 0xd67210, GID ::ffff:172.31.40.122 > remote address: LID 0x0000, QPN 0x0000e3, PSN 0x728a4f, GID ::ffff:172.31.40.121 > 8192000 bytes in 0.03 seconds = 2192.72 Mbit/sec > 1000 iters in 0.03 seconds = 29.89 usec/iter > > Thanks, > Kamal > >>> Fixes: 8700e3e7c485 ("Soft RoCE driver") >>> Signed-off-by: Kamal Heib >>> --- >>> drivers/infiniband/sw/rxe/rxe.c | 34 +++------------------------ >>> drivers/infiniband/sw/rxe/rxe_param.h | 4 ++-- drivers/infiniband/sw/rxe/rxe_recv.c | 29 ++++------------------- >>> drivers/infiniband/sw/rxe/rxe_req.c | 5 +--- >>> drivers/infiniband/sw/rxe/rxe_verbs.c | 17 +++----------- drivers/infiniband/sw/rxe/rxe_verbs.h | 1 - >>> 6 files changed, 13 insertions(+), 77 deletions(-) >>> >>> diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index efcb72c92be6..907203afbd99 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe.c >>> +++ b/drivers/infiniband/sw/rxe/rxe.c >>> @@ -40,14 +40,6 @@ MODULE_AUTHOR("Bob Pearson, Frank Zago, John Groves, Kamal Heib"); MODULE_DESCRIPTION("Soft RDMA transport"); MODULE_LICENSE("Dual BSD/GPL"); >>> >>> -/* free resources for all ports on a device */ -static void rxe_cleanup_ports(struct rxe_dev *rxe) -{ >>> - kfree(rxe->port.pkey_tbl); >>> - rxe->port.pkey_tbl = NULL; >>> - >>> -} >>> - >>> /* free resources for a rxe device all objects created for this device must >>> * have been destroyed >>> */ >>> @@ -66,8 +58,6 @@ void rxe_dealloc(struct ib_device *ib_dev) >>> rxe_pool_cleanup(&rxe->mc_grp_pool); >>> rxe_pool_cleanup(&rxe->mc_elem_pool); >>> >>> - rxe_cleanup_ports(rxe); >>> - >>> if (rxe->tfm) >>> crypto_free_shash(rxe->tfm); >>> } >>> @@ -139,25 +129,14 @@ static void rxe_init_port_param(struct rxe_port *port) >>> /* initialize port state, note IB convention that HCA ports are always >>> * numbered from 1 >>> */ >>> -static int rxe_init_ports(struct rxe_dev *rxe) >>> +static void rxe_init_ports(struct rxe_dev *rxe) >>> { >>> struct rxe_port *port = &rxe->port; >>> >>> rxe_init_port_param(port); >>> - >>> - port->pkey_tbl = kcalloc(port->attr.pkey_tbl_len, >>> - sizeof(*port->pkey_tbl), GFP_KERNEL); >>> - >>> - if (!port->pkey_tbl) >>> - return -ENOMEM; >>> - >>> - port->pkey_tbl[0] = 0xffff; >>> addrconf_addr_eui48((unsigned char *)&port->port_guid, >>> rxe->ndev->dev_addr); >>> - >>> spin_lock_init(&port->port_lock); >>> - >>> - return 0; >>> } >>> >>> /* init pools of managed objects */ >>> @@ -247,13 +226,11 @@ static int rxe_init(struct rxe_dev *rxe) >>> /* init default device parameters */ >>> rxe_init_device_param(rxe); >>> >>> - err = rxe_init_ports(rxe); >>> - if (err) >>> - goto err1; >>> + rxe_init_ports(rxe); >>> >>> err = rxe_init_pools(rxe); >>> if (err) >>> - goto err2; >>> + return err; >>> >>> /* init pending mmap list */ >>> spin_lock_init(&rxe->mmap_offset_lock); >>> @@ -263,11 +240,6 @@ static int rxe_init(struct rxe_dev *rxe) >>> mutex_init(&rxe->usdev_lock); >>> >>> return 0; >>> - >>> -err2: >>> - rxe_cleanup_ports(rxe); >>> -err1: >>> - return err; >>> } >>> >>> void rxe_set_mtu(struct rxe_dev *rxe, unsigned int ndev_mtu) diff --git a/drivers/infiniband/sw/rxe/rxe_param.h b/drivers/infiniband/sw/rxe/rxe_param.h >>> index 99e9d8ba9767..2f381aeafcb5 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_param.h >>> +++ b/drivers/infiniband/sw/rxe/rxe_param.h >>> @@ -100,7 +100,7 @@ enum rxe_device_param { >>> RXE_MAX_SRQ_SGE = 27, >>> RXE_MIN_SRQ_SGE = 1, >>> RXE_MAX_FMR_PAGE_LIST_LEN = 512, >>> - RXE_MAX_PKEYS = 64, >>> + RXE_MAX_PKEYS = 1, >>> RXE_LOCAL_CA_ACK_DELAY = 15, >>> >>> RXE_MAX_UCONTEXT = 512, >>> @@ -148,7 +148,7 @@ enum rxe_port_param { >>> RXE_PORT_INIT_TYPE_REPLY = 0, >>> RXE_PORT_ACTIVE_WIDTH = IB_WIDTH_1X, >>> RXE_PORT_ACTIVE_SPEED = 1, >>> - RXE_PORT_PKEY_TBL_LEN = 64, >>> + RXE_PORT_PKEY_TBL_LEN = 1, >>> RXE_PORT_PHYS_STATE = IB_PORT_PHYS_STATE_POLLING, >>> RXE_PORT_SUBNET_PREFIX = 0xfe80000000000000ULL, >>> }; >>> diff --git a/drivers/infiniband/sw/rxe/rxe_recv.c b/drivers/infiniband/sw/rxe/rxe_recv.c >>> index 46e111c218fd..7e123d3c4d09 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_recv.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_recv.c >>> @@ -101,36 +101,15 @@ static void set_qkey_viol_cntr(struct rxe_port *port) static int check_keys(struct rxe_dev *rxe, struct rxe_pkt_info *pkt, >>> u32 qpn, struct rxe_qp *qp) >>> { >>> - int i; >>> - int found_pkey = 0; >>> struct rxe_port *port = &rxe->port; >>> u16 pkey = bth_pkey(pkt); >>> >>> pkt->pkey_index = 0; >>> >>> - if (qpn == 1) { >>> - for (i = 0; i < port->attr.pkey_tbl_len; i++) { >>> - if (pkey_match(pkey, port->pkey_tbl[i])) { >>> - pkt->pkey_index = i; >>> - found_pkey = 1; >>> - break; >>> - } >>> - } >>> - >>> - if (!found_pkey) { >>> - pr_warn_ratelimited("bad pkey = 0x%x\n", pkey); >>> - set_bad_pkey_cntr(port); >>> - goto err1; >>> - } >>> - } else { >>> - if (unlikely(!pkey_match(pkey, >>> - port->pkey_tbl[qp->attr.pkey_index] >>> - ))) { >>> - pr_warn_ratelimited("bad pkey = 0x%0x\n", pkey); >>> - set_bad_pkey_cntr(port); >>> - goto err1; >>> - } >>> - pkt->pkey_index = qp->attr.pkey_index; >>> + if (!pkey_match(pkey, IB_DEFAULT_PKEY_FULL)) { >>> + pr_warn_ratelimited("bad pkey = 0x%x\n", pkey); >>> + set_bad_pkey_cntr(port); >>> + goto err1; >>> } >>> >>> if ((qp_type(qp) == IB_QPT_UD || qp_type(qp) == IB_QPT_GSI) && diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c >>> index e5031172c019..34df2b55e650 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_req.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_req.c >>> @@ -381,7 +381,6 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, >>> struct rxe_pkt_info *pkt) >>> { >>> struct rxe_dev *rxe = to_rdev(qp->ibqp.device); >>> - struct rxe_port *port = &rxe->port; >>> struct sk_buff *skb; >>> struct rxe_send_wr *ibwr = &wqe->wr; >>> struct rxe_av *av; >>> @@ -419,9 +418,7 @@ static struct sk_buff *init_req_packet(struct rxe_qp *qp, >>> (pkt->mask & (RXE_WRITE_MASK | RXE_IMMDT_MASK)) == >>> (RXE_WRITE_MASK | RXE_IMMDT_MASK)); >>> >>> - pkey = (qp_type(qp) == IB_QPT_GSI) ? >>> - port->pkey_tbl[ibwr->wr.ud.pkey_index] : >>> - port->pkey_tbl[qp->attr.pkey_index]; >>> + pkey = IB_DEFAULT_PKEY_FULL; >>> >>> qp_num = (pkt->mask & RXE_DETH_MASK) ? ibwr->wr.ud.remote_qpn : >>> qp->attr.dest_qp_num; >>> diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c >>> index 74f071003690..779458ddd422 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_verbs.c >>> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c >>> @@ -83,22 +83,11 @@ static int rxe_query_port(struct ib_device *dev, static int rxe_query_pkey(struct ib_device *device, >>> u8 port_num, u16 index, u16 *pkey) { >>> - struct rxe_dev *rxe = to_rdev(device); >>> - struct rxe_port *port; >>> - >>> - port = &rxe->port; >>> - >>> - if (unlikely(index >= port->attr.pkey_tbl_len)) { >>> - dev_warn(device->dev.parent, "invalid index = %d\n", >>> - index); >>> - goto err1; >>> - } >>> + if (index > 0) >>> + return -EINVAL; >>> >>> - *pkey = port->pkey_tbl[index]; >>> + *pkey = IB_DEFAULT_PKEY_FULL; >>> return 0; >>> - >>> -err1: >>> - return -EINVAL; >>> } >>> >>> static int rxe_modify_device(struct ib_device *dev, diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h >>> index 92de39c4a7c1..c664c7f36ab5 100644 >>> --- a/drivers/infiniband/sw/rxe/rxe_verbs.h >>> +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h >>> @@ -371,7 +371,6 @@ struct rxe_mc_elem { >>> >>> struct rxe_port { >>> struct ib_port_attr attr; >>> - u16 *pkey_tbl; >>> __be64 port_guid; >>> __be64 subnet_prefix; >>> spinlock_t port_lock; /* guard port */ >>> -- >>> 2.25.4 >>>