From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 902DDC169C4 for ; Wed, 6 Feb 2019 17:17:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 52D7B2175B for ; Wed, 6 Feb 2019 17:17:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="k+E1VEAz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729566AbfBFRRU (ORCPT ); Wed, 6 Feb 2019 12:17:20 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:42690 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728673AbfBFRRU (ORCPT ); Wed, 6 Feb 2019 12:17:20 -0500 Received: by mail-pl1-f194.google.com with SMTP id s1so3375074plp.9 for ; Wed, 06 Feb 2019 09:17:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=JJHYK6WYDV+y+8kEXdnqQhIqCj7gplM8Nj/zeRQoyjY=; b=k+E1VEAzH5qVjTwj7QBY7HPhKu082mC8DqKm8uywhvXOS9nzVekqOc0D1FfOTTqROC XeYZzb2ex1XsLJyyLK1bbTOgpce0qjEsMXWA4cOMzkvYeRmSK+aXvnT6XbFvXaczjjtJ XZWgrz4PP//r4uS79KeUbh35DTz/p3mTzVJqjyJl5hC65x+qwSyHwHv3tW25efsoQE7J 14XElLY8C+uljVCj2TjkYa068CSRu/qyWFBqDi9269xztKYJt+p4VCKpTFHP7RFS88di xgDBr/ZnaAnZcYef8DiFztjKj4kaoFvs/0VItkVB3C5g2HMpWHYB9mlsvOxxfUWSNQp1 KdiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=JJHYK6WYDV+y+8kEXdnqQhIqCj7gplM8Nj/zeRQoyjY=; b=YjcAdzEde7OvG36VDPC3ZdrfJxMEhxtn4waoGRV/mO1UNQdT8xfzKoDx08U56zzLSC IFp51ua8M7sVfbUJjYjWiAGbxk+eC0j9yZLQDZI5k6TK4K10vmp0WJWI21m/S4sxxR0A xrPk2GTLgfpsangCJb46dv2jeAUwq/E+JP/qDmLpcaBYGuBtjzcXV4ahQ1Zfw1C26Xt/ UQ1SWUr35uzO/QlsgSNFPyEihwNnGiT3A6xR9KiRduMeg/TW7xz3Ko3F7WWPREqaAz91 I5tleebnUTlS9tg0AH0zcBXZCYmn6lsxSjIOLYcxqyRrqmgrQFkpIL4enU4LiexgGSM4 g/6Q== X-Gm-Message-State: AHQUAubgLsT3iM6awv4MElasNErVJUHT/qMfjkzAxDjXruNanoDVRfhK bWyj1pHB1dHd6tp+GuBWBngIFTWG X-Google-Smtp-Source: AHgI3IaVW9Sppkgoh0h3K0SA0ELbouiUXs3q9ZQnuScTd9Pw82RZPUz9GwswlttGTLL3h3zWU8fiUQ== X-Received: by 2002:a17:902:7805:: with SMTP id p5mr9678088pll.261.1549473439911; Wed, 06 Feb 2019 09:17:19 -0800 (PST) Received: from ?IPv6:2620:15c:2c1:200:55c7:81e6:c7d8:94b? ([2620:15c:2c1:200:55c7:81e6:c7d8:94b]) by smtp.gmail.com with ESMTPSA id m198sm12294080pga.10.2019.02.06.09.17.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 06 Feb 2019 09:17:18 -0800 (PST) Subject: Re: [Patch net-next] mlx5: use RCU lock in mlx5_eq_cq_get() To: Saeed Mahameed , "netdev@vger.kernel.org" , Tariq Toukan , "xiyou.wangcong@gmail.com" References: <20190206003525.5041-1-xiyou.wangcong@gmail.com> <1620af96-f60c-9ec2-dd71-7c9081828bce@mellanox.com> From: Eric Dumazet Message-ID: <8df59337-8c20-de92-6088-d008031da6f2@gmail.com> Date: Wed, 6 Feb 2019 09:17:17 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 02/06/2019 08:55 AM, Saeed Mahameed wrote: > On Wed, 2019-02-06 at 12:02 +0000, Tariq Toukan wrote: >> >> On 2/6/2019 2:35 AM, Cong Wang wrote: >>> mlx5_eq_cq_get() is called in IRQ handler, the spinlock inside >>> gets a lot of contentions when we test some heavy workload >>> with 60 RX queues and 80 CPU's, and it is clearly shown in the >>> flame graph. >>> > > > Hi Cong, > > The patch is ok to me, but i really doubt that you can hit a contention > on latest upstream driver, since we already have spinlock per EQ, which > means spinlock per core, each EQ (core) msix handler can only access > one spinlock (its own), so I am surprised how you got the contention, > Maybe you are not running on latest upstream driver ? > > what is the workload ? Surprisingly (or not), atomic operations, even on _not_ contended cache lines can stall the cpu enough for perf tools to notice... If the atomic operation can be trivially replaced by RCU, then do it by any mean.