From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D7DC8C433EF for ; Fri, 6 May 2022 07:48:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To:Subject: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=lYf/KfLFWv3aPmHd9PAYM4OAPjcbvGihDonKI+kvBxw=; b=uU+adtwxbdNP/L vDx1J9JCp/r7mH30yjzH2T0eJzitozYFUaU0hvlSMRYPOTKZ3kXiC66pNaZb0gtkbJlgspaQsnOmP Nh6V3bCXaGqjzHOyG+PFAwfm2VBbXTcmJZM5rDPwVhkYMyVJ4j6agRLP59SFO3cWIbC9c7FpPguFS pLc6+Dl1AQB9kh46luoa/4VVX0mR/vpLw0URE4hKQByHKMx0CGJxOeCLqMgGbD466420tq1xlfD+n K0qOx//32i773/KBhAr9/c6i55tFjzt62ERtXMxXv8ELgVJs8Ep/MwqE0T/e5wv5oBO/oMC4PeTn0 t7oqvHFGVaXeS4eIWveA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmsgn-001uVD-3J; Fri, 06 May 2022 07:47:49 +0000 Received: from mail-ej1-x634.google.com ([2a00:1450:4864:20::634]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nmsgi-001uSy-J4 for linux-arm-kernel@lists.infradead.org; Fri, 06 May 2022 07:47:46 +0000 Received: by mail-ej1-x634.google.com with SMTP id g6so12903815ejw.1 for ; Fri, 06 May 2022 00:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:to:cc:references :from:in-reply-to:content-transfer-encoding; bh=zKGL4m4tOsAY5uuE8MMLxzwFEWALxOxUgmZN40z4JCE=; b=WVSh85cJQd9BJ8bKwXxaL4eAkOYvBpq/3esrh8Zo9N7uU9WqmGlhWkPU7BjGTpY1Xy nh9ZL+PLQXVlNXQ7/T92xH3RBPrvqnFdwvX4c3ARKhWiz2RLSyKFbw2Xek2fCFoW+GRV 38INwTdx52QZhX9doP42y6ZpkG5/s5t7EF3SBwkw6fl1DCTsICnWtfrxNZyclDi5EuzL iZKu2OfpqCxsMvQEt/GZqXAdsAeqBNDRkX0GBxzCBl646EVAuRKwzueAaGUR7HnoLpve U5tHOc3fSUlsq8PfXquAiX0mTm1b25jttf8eaqgdZQL/CSQfR3OBfrjrme6MTfC7SmD1 BGgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :to:cc:references:from:in-reply-to:content-transfer-encoding; bh=zKGL4m4tOsAY5uuE8MMLxzwFEWALxOxUgmZN40z4JCE=; b=oGO9lDu7ANDe+3cOniKwR3vNrucKbZzZCP5wU832/wyJbZVMGj/qOuTVgx864h2PIG ST5opZX7Lp9SCfWrEvODzQMGRtaT4ornRKbsDMaRSCYobE2WChJ8TzyJfp4k4k2y1tXu cPPEug3FjOKS1NVNf430G7A9ppWS2/zoOlrERn9DcYYackYa7HzoXej5XkL3jBGifOH2 0cSLqh+ETqp0k+6XZA/ZqPs9FcUa6Sjd6E1YhsGegUxr819U3NUUjWfYPP5wYP3KdOS6 EApDnQ4WYGZ66usiBkOAlFtmijEIHyBBcBGtX4/ex8a18OB5tGHsrpXdIgLPbhr/wP4b Yy4Q== X-Gm-Message-State: AOAM531IL9rUm3UVkf7Joe8+i8Ix9lqPyfmO1vk8gLi/IdBx1HR7sD7l tYieyn0t/MbB/IOuKxa4hCTxVcmgRaQ= X-Google-Smtp-Source: ABdhPJwxVq9/G6zl0oAcrCmzq0FulenswlEYx3KJU3yp/+gzaeWU9NKPCEdmJTxFY+ZBtdC4jFleHw== X-Received: by 2002:a17:907:7247:b0:6f4:ed49:cd3f with SMTP id ds7-20020a170907724700b006f4ed49cd3fmr1991725ejc.172.1651823262528; Fri, 06 May 2022 00:47:42 -0700 (PDT) Received: from [192.168.26.149] (ip-194-187-74-233.konfederacka.maverick.com.pl. [194.187.74.233]) by smtp.googlemail.com with ESMTPSA id d2-20020a170907272200b006f3ef214e21sm1600452ejl.135.2022.05.06.00.47.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 May 2022 00:47:42 -0700 (PDT) Message-ID: <04fa6560-e6f4-005f-cddb-7bc9b4859ba2@gmail.com> Date: Fri, 6 May 2022 09:47:41 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:96.0) Gecko/20100101 Thunderbird/96.0 Subject: Re: Optimizing kernel compilation / alignments for network performance To: Felix Fietkau , Andrew Lunn Cc: Arnd Bergmann , Alexander Lobakin , Network Development , linux-arm-kernel , Russell King , "openwrt-devel@lists.openwrt.org" , Florian Fainelli References: <84f25f73-1fab-fe43-70eb-45d25b614b4c@gmail.com> <20220427125658.3127816-1-alexandr.lobakin@intel.com> <066fc320-dc04-11a4-476e-b0d11f3b17e6@gmail.com> <2a338e8e-3288-859c-d2e8-26c5712d3d06@nbd.name> From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= In-Reply-To: <2a338e8e-3288-859c-d2e8-26c5712d3d06@nbd.name> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220506_004744_700896_342E60FC X-CRM114-Status: GOOD ( 19.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: base64 Content-Type: text/plain; charset="utf-8"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org T24gNS4wNS4yMDIyIDE4OjQ2LCBGZWxpeCBGaWV0a2F1IHdyb3RlOgo+IAo+IE9uIDA1LjA1LjIy IDE4OjA0LCBBbmRyZXcgTHVubiB3cm90ZToKPj4+IHlvdSdsbCBzZWUgdGhhdCBtb3N0IHVzZWQg ZnVuY3Rpb25zIGFyZToKPj4+IHY3X2RtYV9pbnZfcmFuZ2UKPj4+IF9faXJxZW50cnlfdGV4dF9l bmQKPj4+IGwyYzIxMF9pbnZfcmFuZ2UKPj4+IHY3X2RtYV9jbGVhbl9yYW5nZQo+Pj4gYmNtYV9o b3N0X3NvY19yZWFkMzIKPj4+IF9fbmV0aWZfcmVjZWl2ZV9za2JfY29yZQo+Pj4gYXJjaF9jcHVf aWRsZQo+Pj4gbDJjMjEwX2NsZWFuX3JhbmdlCj4+PiBmaWJfdGFibGVfbG9va3VwCj4+Cj4+IFRo ZXJlIGlzIGEgbG90IG9mIGNhY2hlIG1hbmFnZW1lbnQgZnVuY3Rpb25zIGhlcmUuIE1pZ2h0IHNv dW5kIG9kZCwKPj4gYnV0IGhhdmUgeW91IHRyaWVkIGRpc2FibGluZyBTTVA/IFRoZXNlIGNhY2hl IGZ1bmN0aW9ucyBuZWVkIHRvCj4+IG9wZXJhdGUgYWNyb3NzIGFsbCBDUFVzLCBhbmQgdGhlIGNv bW11bmljYXRpb24gYmV0d2VlbiBDUFVzIGNhbiBzbG93Cj4+IHRoZW0gZG93bi4gSWYgdGhlcmUg aXMgb25seSBvbmUgQ1BVLCB0aGVzZSBjYWNoZSBmdW5jdGlvbnMgZ2V0IHNpbXBsZXIKPj4gYW5k IGZhc3Rlci4KPj4KPj4gSXQganVzdCBkZXBlbmRzIG9uIHlvdXIgd29ya2xvYWQuIElmIHlvdSBo YXZlIDEgQ1BVIGxvYWRlZCB0byAxMDAlIGFuZAo+PiB0aGUgb3RoZXIgMyBpZGxlLCB5b3UgbWln aHQgc2VlIGFuIGltcHJvdmVtZW50LiBJZiB5b3UgYWN0dWFsbHkgbmVlZAo+PiBtb3JlIHRoYW4g b25lIENQVSwgaXQgd2lsbCBwcm9iYWJseSBiZSB3b3JzZS4KPj4KPj4gSSd2ZSBhbHNvIGZvdW5k IHRoYXQgc29tZSBFdGhlcm5ldCBkcml2ZXJzIGludmFsaWRhdGUgb3IgZmx1c2ggdG9vCj4+IG11 Y2guIElmIHlvdSBhcmUgc2VuZGluZyBhIDY0IGJ5dGUgVENQIEFDSywgYWxsIHlvdSBuZWVkIHRv IGZsdXNoIGlzCj4+IDY0IGJ5dGVzLCBub3QgdGhlIGZ1bGwgMTUwMCBNVFUuIElmIHlvdSByZWNl aXZlIGEgVENQIEFDSywgYW5kIHRoZW4KPj4gcmVjeWNsZSB0aGUgYnVmZmVyLCBhbGwgeW91IG5l ZWQgdG8gaW52YWxpZGF0ZSBpcyB0aGUgc2l6ZSBvZiB0aGUgQUNLLAo+PiBzbyBsb25nIGFzIHlv dSBjYW4gZ3VhcmFudGVlIG5vdGhpbmcgaGFzIHRvdWNoZWQgdGhlIG1lbW9yeSBhYm92ZSBpdC4K Pj4gQnV0IHlvdSBuZWVkIHRvIGJlIGNhcmVmdWwgd2hlbiBpbXBsZW1lbnRpbmcgdHJpY2tzIGxp a2UgdGhpcywgb3IgeW91Cj4+IGNhbiBnZXQgc3VidGxlIGNvcnJ1cHRpb24gYnVncyB3aGVuIHlv dSBnZXQgaXQgd3JvbmcuCj4gSSBqdXN0IHRvb2sgYSBxdWljayBsb29rIGF0IHRoZSBkcml2ZXIu IEl0IGFsbG9jYXRlcyBhbmQgbWFwcyByeCBidWZmZXJzIHRoYXQgY2FuIGNvdmVyIGEgcGFja2V0 IHNpemUgb2YgQkdNQUNfUlhfTUFYX0ZSQU1FX1NJWkUgPSA5NzI0Lgo+IFRoaXMgc2VlbXMgcmF0 aGVyIGV4Y2Vzc2l2ZSwgZXNwZWNpYWxseSBzaW5jZSBtb3N0IHBlb3BsZSBhcmUgZ29pbmcgdG8g dXNlIGEgTVRVIG9mIDE1MDAuCj4gTXkgcHJvcG9zYWwgd291bGQgYmUgdG8gYWRkIHN1cHBvcnQg Zm9yIG1ha2luZyByeCBidWZmZXIgc2l6ZSBkZXBlbmRlbnQgb24gTVRVLCByZWFsbG9jYXRpbmcg dGhlIHJpbmcgb24gTVRVIGNoYW5nZXMuCj4gVGhpcyBzaG91bGQgc2lnbmlmaWNhbnRseSByZWR1 Y2UgdGhlIHRpbWUgc3BlbnQgb24gZmx1c2hpbmcgY2FjaGVzLgoKT2gsIHRoYXQncyBpbXBvcnRh bnQgdG9vLCBpdCB3YXMgY2hhbmdlZCBieSBjb21taXQgOGM3ZGE2Mzk3OGYxICgiYmdtYWM6CmNv bmZpZ3VyZSBNVFUgYW5kIGFkZCBzdXBwb3J0IGZvciBmcmFtZXMgYmV5b25kIDgxOTIgYnl0ZSBz aXplIik6Cmh0dHBzOi8vZ2l0Lmtlcm5lbC5vcmcvcHViL3NjbS9saW51eC9rZXJuZWwvZ2l0L3Rv cnZhbGRzL2xpbnV4LmdpdC9jb21taXQvP2lkPThjN2RhNjM5NzhmMTY3MmViNDAzN2JiY2E2ZTdl YWM3M2Y5MDhmMDMKCkl0IGxvd2VyZWQgTkFUIHNwZWVkIHdpdGggYmdtYWMgYnkgNjAlICgzNjIg TWJwcyDihpIgMTQwIE1icHMpLgoKSSBkbyBhbGwgbXkgdGVzdGluZyB3aXRoCiNkZWZpbmUgQkdN QUNfUlhfTUFYX0ZSQU1FX1NJWkUJCQkxNTM2CgpfX19fX19fX19fX19fX19fX19fX19fX19fX19f X19fX19fX19fX19fX19fX19fXwpsaW51eC1hcm0ta2VybmVsIG1haWxpbmcgbGlzdApsaW51eC1h cm0ta2VybmVsQGxpc3RzLmluZnJhZGVhZC5vcmcKaHR0cDovL2xpc3RzLmluZnJhZGVhZC5vcmcv bWFpbG1hbi9saXN0aW5mby9saW51eC1hcm0ta2VybmVsCg== From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E699C433F5 for ; Fri, 6 May 2022 07:47:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1389682AbiEFHv2 (ORCPT ); Fri, 6 May 2022 03:51:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1389689AbiEFHv0 (ORCPT ); Fri, 6 May 2022 03:51:26 -0400 Received: from mail-ej1-x62e.google.com (mail-ej1-x62e.google.com [IPv6:2a00:1450:4864:20::62e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 46941674F6 for ; Fri, 6 May 2022 00:47:43 -0700 (PDT) Received: by mail-ej1-x62e.google.com with SMTP id y3so12834266ejo.12 for ; Fri, 06 May 2022 00:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:to:cc:references :from:in-reply-to:content-transfer-encoding; bh=zKGL4m4tOsAY5uuE8MMLxzwFEWALxOxUgmZN40z4JCE=; b=WVSh85cJQd9BJ8bKwXxaL4eAkOYvBpq/3esrh8Zo9N7uU9WqmGlhWkPU7BjGTpY1Xy nh9ZL+PLQXVlNXQ7/T92xH3RBPrvqnFdwvX4c3ARKhWiz2RLSyKFbw2Xek2fCFoW+GRV 38INwTdx52QZhX9doP42y6ZpkG5/s5t7EF3SBwkw6fl1DCTsICnWtfrxNZyclDi5EuzL iZKu2OfpqCxsMvQEt/GZqXAdsAeqBNDRkX0GBxzCBl646EVAuRKwzueAaGUR7HnoLpve U5tHOc3fSUlsq8PfXquAiX0mTm1b25jttf8eaqgdZQL/CSQfR3OBfrjrme6MTfC7SmD1 BGgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :to:cc:references:from:in-reply-to:content-transfer-encoding; bh=zKGL4m4tOsAY5uuE8MMLxzwFEWALxOxUgmZN40z4JCE=; b=TlPtCzKMR2i51F1awpLtuRI2OoYYv5GYxm6FQZVKkg/ucPTJ4cqmN1kr1P6/jeSZ6t /xYdUs7KQ/cIPNNV9GUtv1NXpXKsssQUdGlrlVTSch8tCXExavA00igUvCr92D1Tk5NL 2QMdQYSy6nkiyt40OXqs4+QTqJhVQxAfq/lB0NNFoCqx7XToSHeS1dbApASw92AB0m1q G0sGgNJD3gp93h8Qqj71Bho5KG8URxDQoSVHZfjqDXThVitiq75Sbe/xrRthzW2UxCw9 VG8gD0PqIQMwg3Y6vlAteTwK20jcUOiVurflgZcAmc8NS9/1uW/Z905rF2X0AEc+AecW +dkA== X-Gm-Message-State: AOAM532kN4l54uCYoUNH3J6GNsJBSc4P/jN8cm015xyy+Sg1+RALkSpM 596JIQyJiqMDsv9YGAwUKzg= X-Google-Smtp-Source: ABdhPJwxVq9/G6zl0oAcrCmzq0FulenswlEYx3KJU3yp/+gzaeWU9NKPCEdmJTxFY+ZBtdC4jFleHw== X-Received: by 2002:a17:907:7247:b0:6f4:ed49:cd3f with SMTP id ds7-20020a170907724700b006f4ed49cd3fmr1991725ejc.172.1651823262528; Fri, 06 May 2022 00:47:42 -0700 (PDT) Received: from [192.168.26.149] (ip-194-187-74-233.konfederacka.maverick.com.pl. [194.187.74.233]) by smtp.googlemail.com with ESMTPSA id d2-20020a170907272200b006f3ef214e21sm1600452ejl.135.2022.05.06.00.47.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 06 May 2022 00:47:42 -0700 (PDT) Message-ID: <04fa6560-e6f4-005f-cddb-7bc9b4859ba2@gmail.com> Date: Fri, 6 May 2022 09:47:41 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:96.0) Gecko/20100101 Thunderbird/96.0 Subject: Re: Optimizing kernel compilation / alignments for network performance To: Felix Fietkau , Andrew Lunn Cc: Arnd Bergmann , Alexander Lobakin , Network Development , linux-arm-kernel , Russell King , "openwrt-devel@lists.openwrt.org" , Florian Fainelli References: <84f25f73-1fab-fe43-70eb-45d25b614b4c@gmail.com> <20220427125658.3127816-1-alexandr.lobakin@intel.com> <066fc320-dc04-11a4-476e-b0d11f3b17e6@gmail.com> <2a338e8e-3288-859c-d2e8-26c5712d3d06@nbd.name> From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= In-Reply-To: <2a338e8e-3288-859c-d2e8-26c5712d3d06@nbd.name> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 5.05.2022 18:46, Felix Fietkau wrote: > > On 05.05.22 18:04, Andrew Lunn wrote: >>> you'll see that most used functions are: >>> v7_dma_inv_range >>> __irqentry_text_end >>> l2c210_inv_range >>> v7_dma_clean_range >>> bcma_host_soc_read32 >>> __netif_receive_skb_core >>> arch_cpu_idle >>> l2c210_clean_range >>> fib_table_lookup >> >> There is a lot of cache management functions here. Might sound odd, >> but have you tried disabling SMP? These cache functions need to >> operate across all CPUs, and the communication between CPUs can slow >> them down. If there is only one CPU, these cache functions get simpler >> and faster. >> >> It just depends on your workload. If you have 1 CPU loaded to 100% and >> the other 3 idle, you might see an improvement. If you actually need >> more than one CPU, it will probably be worse. >> >> I've also found that some Ethernet drivers invalidate or flush too >> much. If you are sending a 64 byte TCP ACK, all you need to flush is >> 64 bytes, not the full 1500 MTU. If you receive a TCP ACK, and then >> recycle the buffer, all you need to invalidate is the size of the ACK, >> so long as you can guarantee nothing has touched the memory above it. >> But you need to be careful when implementing tricks like this, or you >> can get subtle corruption bugs when you get it wrong. > I just took a quick look at the driver. It allocates and maps rx buffers that can cover a packet size of BGMAC_RX_MAX_FRAME_SIZE = 9724. > This seems rather excessive, especially since most people are going to use a MTU of 1500. > My proposal would be to add support for making rx buffer size dependent on MTU, reallocating the ring on MTU changes. > This should significantly reduce the time spent on flushing caches. Oh, that's important too, it was changed by commit 8c7da63978f1 ("bgmac: configure MTU and add support for frames beyond 8192 byte size"): https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=8c7da63978f1672eb4037bbca6e7eac73f908f03 It lowered NAT speed with bgmac by 60% (362 Mbps → 140 Mbps). I do all my testing with #define BGMAC_RX_MAX_FRAME_SIZE 1536