From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B0DFC3F2D1 for ; Tue, 3 Mar 2020 00:40:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 26341217F4 for ; Tue, 3 Mar 2020 00:40:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726979AbgCCAkA (ORCPT ); Mon, 2 Mar 2020 19:40:00 -0500 Received: from relay8-d.mail.gandi.net ([217.70.183.201]:39001 "EHLO relay8-d.mail.gandi.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726773AbgCCAj7 (ORCPT ); Mon, 2 Mar 2020 19:39:59 -0500 X-Originating-IP: 50.39.173.182 Received: from localhost (50-39-173-182.bvtn.or.frontiernet.net [50.39.173.182]) (Authenticated sender: josh@joshtriplett.org) by relay8-d.mail.gandi.net (Postfix) with ESMTPSA id 773441BF206; Tue, 3 Mar 2020 00:39:55 +0000 (UTC) Date: Mon, 2 Mar 2020 16:39:52 -0800 From: Josh Triplett To: "Machulsky, Zorik" Cc: "Belgazal, Netanel" , "Kiyanovski, Arthur" , "Tzalik, Guy" , "Bshara, Saeed" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: Re: [PATCH] ena: Speed up initialization 90x by reducing poll delays Message-ID: <20200303003952.GA264245@localhost> References: <20200229002813.GA177044@localhost> <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <8B4A52CD-FC5A-4256-B7DE-A659B50654CE@amazon.com> Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Mon, Mar 02, 2020 at 11:16:32PM +0000, Machulsky, Zorik wrote: > > On 2/28/20, 4:29 PM, "Josh Triplett" wrote: > > Before initializing completion queue interrupts, the ena driver uses > polling to wait for responses on the admin command queue. The ena driver > waits 5ms between polls, but the hardware has generally finished long > before that. Reduce the poll time to 10us. > > On a c5.12xlarge, this improves ena initialization time from 173.6ms to > 1.920ms, an improvement of more than 90x. This improves server boot time > and time to network bringup. > > Thanks Josh, > We agree that polling rate should be increased, but prefer not to do it aggressively and blindly. > For example linear backoff approach might be a better choice. Please let us re-work a little this > patch and bring it to review. Thanks! That's fine, as long as it has the same net improvement on boot time. I'd appreciate the opportunity to test any alternate approach you might have. (Also, as long as you're working on this, you might wish to make a similar change to the EFA driver, and to the FreeBSD drivers.) > Before: > [ 0.531722] calling ena_init+0x0/0x63 @ 1 > [ 0.531722] ena: Elastic Network Adapter (ENA) v2.1.0K > [ 0.531751] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K > [ 0.531946] PCI Interrupt Link [LNKD] enabled at IRQ 11 > [ 0.547425] ena: ena device version: 0.10 > [ 0.547427] ena: ena controller version: 0.0.1 implementation version 1 > [ 0.709497] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency > [ 0.709508] initcall ena_init+0x0/0x63 returned 0 after 173616 usecs > > After: > [ 0.526965] calling ena_init+0x0/0x63 @ 1 > [ 0.526966] ena: Elastic Network Adapter (ENA) v2.1.0K > [ 0.527056] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K > [ 0.527196] PCI Interrupt Link [LNKD] enabled at IRQ 11 > [ 0.527211] ena: ena device version: 0.10 > [ 0.527212] ena: ena controller version: 0.0.1 implementation version 1 > [ 0.528925] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency > [ 0.528934] initcall ena_init+0x0/0x63 returned 0 after 1920 usecs