From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 531BBC169C4 for ; Wed, 6 Feb 2019 23:36:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 143AF2070B for ; Wed, 6 Feb 2019 23:36:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tUTp1Wuc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726448AbfBFXgc (ORCPT ); Wed, 6 Feb 2019 18:36:32 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:35310 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725959AbfBFXgb (ORCPT ); Wed, 6 Feb 2019 18:36:31 -0500 Received: by mail-pg1-f195.google.com with SMTP id s198so3659077pgs.2 for ; Wed, 06 Feb 2019 15:36:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=bu0RI/wB85tfI/2aVxxiZX2BcUscKA5gdLLm3MqjKWQ=; b=tUTp1WucxcQQEbPDXKY6ugyYpApWgSNSQ7k+6sNfVjaBBm/ovhB2iJBGXxxBPZRWK0 CrZpeYUNdjhkTJlLJltSf8Sn4oS+wVysSN8+aTdLgPd5R0IIgR3VNGFDVVPnHokPzm86 rCwSAedpJvvCFq24jm11SiayZtOVenTvycvDROR9SdVgN6zH03OW45AtRJ8nsiyU6VIT Dw+Q7+Es5HIi8I4iQSP+xLAPxwC67ZCo8Q2B50eDDvfRDzYelMwyBHyE1zLXpwG6O0+l lYXQ08zIbyz1A2GLpu1o+mUtGKB22I3DnLeC2Ugmkv68Bm6ropvXEsP3oSS8lgemHBNW uOwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=bu0RI/wB85tfI/2aVxxiZX2BcUscKA5gdLLm3MqjKWQ=; b=i6U66asG2Qp1wtnONEap3+BOSty7NeiR7cNXXhQOR4622kFC+WTKtpIUBhmLv2Sq2n qffZ1AkJQIo3TMupGnx88vdY7GFhuOESjusr55nC3eQhvT/CyJWOecuFogRQ/vPfQrZH 88BQtNbk5QZHPTKHVv16hDudnffVZqz5u44MizxxOdzz+c13CzQq3XOFFouYknElrgP8 PdcwTJ3/eICRjaMaS+4hGFgAi9h9bF+4cxqDfKmZbh6kSryoKKtZoAoM1yXv5OZqkio/ fLsFtXWlreynKmoFHfgz3Ao/Xkudcp8G3cQp5ZqoYfIIv73YOWCkldluwx3/x61Rzwxt krzg== X-Gm-Message-State: AHQUAubeuCltAALsryWQuRF8MFm1ulyTzpTm+CQ+BcKVuT18KNbGcecI WzodN4p0rIYTY764W0EPO6mYv3Ui X-Google-Smtp-Source: AHgI3IY/TJVl377M0XDEAwVE0RHyavSIpBp2m51Ci6E0Xixz758e2RY4vHhAwOPpPjN1m8fwVyqgDg== X-Received: by 2002:a63:1f4e:: with SMTP id q14mr11650037pgm.88.1549496190996; Wed, 06 Feb 2019 15:36:30 -0800 (PST) Received: from ?IPv6:2620:15c:2c1:200:55c7:81e6:c7d8:94b? ([2620:15c:2c1:200:55c7:81e6:c7d8:94b]) by smtp.gmail.com with ESMTPSA id n186sm9390320pfn.137.2019.02.06.15.36.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Feb 2019 15:36:29 -0800 (PST) Subject: Re: [Patch net-next v2] mlx5: use RCU lock in mlx5_eq_cq_get() To: Cong Wang , netdev@vger.kernel.org Cc: Saeed Mahameed , Tariq Toukan References: <20190206230019.1303-1-xiyou.wangcong@gmail.com> From: Eric Dumazet Message-ID: <52ed9806-310d-bfeb-9610-0daefc1e66fa@gmail.com> Date: Wed, 6 Feb 2019 15:36:28 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190206230019.1303-1-xiyou.wangcong@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 02/06/2019 03:00 PM, Cong Wang wrote: > mlx5_eq_cq_get() is called in IRQ handler, the spinlock inside > gets a lot of contentions when we test some heavy workload > with 60 RX queues and 80 CPU's, and it is clearly shown in the > flame graph. > > In fact, radix_tree_lookup() is perfectly fine with RCU read lock, > we don't have to take a spinlock on this hot path. This is pretty > much similar to commit 291c566a2891 > ("net/mlx4_core: Fix racy CQ (Completion Queue) free"). Slow paths > are still serialized with the spinlock, and with synchronize_irq() > it should be safe to just move the fast path to RCU read lock. > > This patch itself reduces the latency by about 50% for our memcached > workload on a 4.14 kernel we test. In upstream, as pointed out by Saeed, > this spinlock gets some rework in commit 02d92f790364 > ("net/mlx5: CQ Database per EQ"), so the difference could be smaller. > > Cc: Saeed Mahameed > Cc: Tariq Toukan > Acked-by: Saeed Mahameed > Signed-off-by: Cong Wang > --- > drivers/net/ethernet/mellanox/mlx5/core/eq.c | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eq.c b/drivers/net/ethernet/mellanox/mlx5/core/eq.c > index ee04aab65a9f..7092457705a2 100644 > --- a/drivers/net/ethernet/mellanox/mlx5/core/eq.c > +++ b/drivers/net/ethernet/mellanox/mlx5/core/eq.c > @@ -114,11 +114,11 @@ static struct mlx5_core_cq *mlx5_eq_cq_get(struct mlx5_eq *eq, u32 cqn) > struct mlx5_cq_table *table = &eq->cq_table; > struct mlx5_core_cq *cq = NULL; > > - spin_lock(&table->lock); > + rcu_read_lock(); > cq = radix_tree_lookup(&table->tree, cqn); > if (likely(cq)) > mlx5_cq_hold(cq); I suspect that you need a variant that makes sure refcount is not zero. ( Typical RCU rules apply ) if (cq && !refcount_inc_not_zero(&cq->refcount)) cq = NULL; See commit 6fa19f5637a6c22bc0999596bcc83bdcac8a4fa6 rds: fix refcount bug in rds_sock_addref for a similar issue I fixed recently.