From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EABADC169C4 for ; Tue, 29 Jan 2019 19:22:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B348A20882 for ; Tue, 29 Jan 2019 19:22:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="rUxD9f7t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727531AbfA2TWO (ORCPT ); Tue, 29 Jan 2019 14:22:14 -0500 Received: from mail-eopbgr50076.outbound.protection.outlook.com ([40.107.5.76]:22112 "EHLO EUR03-VE1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726852AbfA2TWO (ORCPT ); Tue, 29 Jan 2019 14:22:14 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cnwZs1csAw+plN6a0wCs+Vq/GB5+/X9RZaz+oyFgVNg=; b=rUxD9f7tUdbtHD06rVx7ZUWgFdZbYAD4f5OZBBA082nGv2AClXDiHw2zCPbhIh6SXvIh3l3osCWu0spJOeoiJCao/hIy5z2R4vfh7O3+xWtBN8rikUNPvKRh7dOCma0AovslsXdHAIqZO6GNjtzMtLYInwKzM7jjQbmdhpiS0pY= Received: from HE1PR0502MB3641.eurprd05.prod.outlook.com (10.167.127.11) by HE1PR0502MB3067.eurprd05.prod.outlook.com (10.175.29.137) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1558.21; Tue, 29 Jan 2019 19:22:10 +0000 Received: from HE1PR0502MB3641.eurprd05.prod.outlook.com ([fe80::4041:bb68:2af3:eab8]) by HE1PR0502MB3641.eurprd05.prod.outlook.com ([fe80::4041:bb68:2af3:eab8%5]) with mapi id 15.20.1558.023; Tue, 29 Jan 2019 19:22:10 +0000 From: Vlad Buslov To: Dennis Zhou CC: Eric Dumazet , Tejun Heo , Linux Kernel Network Developers , Yevgeny Kliteynik , Yossef Efraim , Maor Gottlieb Subject: Re: tc filter insertion rate degradation Thread-Topic: tc filter insertion rate degradation Thread-Index: AQHUsXvoWpbt+qtyHk2zMiwViDqQPKW7jh4AgAMhYgCAB/1dAA== Date: Tue, 29 Jan 2019 19:22:10 +0000 Message-ID: References: <20190124172126.GA66944@dennisz-mbp.dhcp.thefacebook.com> In-Reply-To: <20190124172126.GA66944@dennisz-mbp.dhcp.thefacebook.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: LO2P265CA0077.GBRP265.PROD.OUTLOOK.COM (2603:10a6:600:8::17) To HE1PR0502MB3641.eurprd05.prod.outlook.com (2603:10a6:7:85::11) authentication-results: spf=none (sender IP is ) smtp.mailfrom=vladbu@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [37.142.13.130] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;HE1PR0502MB3067;6:fg3gvUFiDjCPyjGBb6NJcaxjcax9l4YLbl0djRuz7ZMG0jxrFm4JYb0Lu1aoetAo7p84kmpr2w/WLXZAd8G56w69BC0DGx/ne+HmlSoGlzvPbqbFUDmP5zO/xMj2CEG7ZpdnMPcX3TNsNb2bM06Bwdjn5Rp/H7F8jCS3qJWLtuUARJf1cskgAHDrz9PpJCOF4+aV1jMqZ52M9L9miODFZRSUM8BxcCQO+N8v81phpsN3eZoaLNDcnjGHvtAGnC2K65KcOBP4rogmjmmgtNNhEozoollsrLVhPbZ2rr0jrsJYsFKnBUz7V8JykNF0dQ7Ogy4+Xuplgjt/IZSCmEFpEUZgDFmRO/fh5MPfCzbeAmVSX8SJgYvVrIV6SFR9R7AbS6hZ6jEqMKFp+bYc5vugRf8yquxD7sTaah9p9OYIxzbmarzWg8iwyvRjR8+ifRYi+njFTKJECErXgAi5eXhSdA==;5:bqb/TXbYxefCqMtBYIdr9MSUBhDOpU/fMZ0rPOLlxcz+r/CJPqZ6tZ7jTBu334TO5JGeD/uzqhKzNzy4cGs/ES8XhslEUvEN6v1A6r4XLfRtepBPaFmupKBVDm8A5ArVtyG4ipA7jCRVXUSw1Dgj640M88v6XYWtcsZZvu7DcugrH0czLMLGVnsdyzG3DY89+nFOgmb+ChfYjWafDF+iwg==;7:D67vbpLqMZk/W4sNPy92eNRDM9jvq8cbFMSTKtF23lMlsqan0p2wPDmEv7+NsRUFIP7slMaN5idkP19pRarDyMHakioEJcLRclOvHj0QHynbX7mScgTg7dkfiX6aUnO/PHrWzRg5s8SE1nJbDoAXzw== x-ms-office365-filtering-correlation-id: b1e72676-a9f5-47c8-170d-08d6861f1074 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600110)(711020)(4605077)(4618075)(2017052603328)(7153060)(7193020);SRVR:HE1PR0502MB3067; x-ms-traffictypediagnostic: HE1PR0502MB3067: x-microsoft-antispam-prvs: x-forefront-prvs: 093290AD39 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(366004)(346002)(376002)(396003)(39860400002)(136003)(189003)(199004)(316002)(102836004)(966005)(486006)(2616005)(8936002)(26005)(478600001)(6436002)(81166006)(54906003)(476003)(81156014)(8676002)(106356001)(2906002)(99286004)(305945005)(105586002)(11346002)(186003)(7736002)(76176011)(446003)(97736004)(6506007)(14454004)(229853002)(386003)(6486002)(53546011)(3846002)(6116002)(66066001)(71200400001)(71190400001)(4326008)(86362001)(6246003)(6306002)(53936002)(52116002)(256004)(36756003)(68736007)(6512007)(6916009)(25786009)(107886003);DIR:OUT;SFP:1101;SCL:1;SRVR:HE1PR0502MB3067;H:HE1PR0502MB3641.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: t+lPpwNUNm6Blwyre1WsjzSTriOHSo9tkWOO54um+VMULRMZbELIaQgdfDaUOsWLG4TBm+0mY2C0ywytzOzkAjAYoOKZMu4sVKrRXaP22khdHIAVSp3Xhq7oWD1PgiifnTr787B3v2F3wSjk/I1LXSN69xwLC4z2JW8biVB/bL6xgyzB7nUZdn9jmwKooo+KHlfFIKp27vNibkCBlWx2GBUoRCshGOWX7kSQYC+tYO7BiUzgmsMjymXx9aF52ckI7i+2YfnrJ0NQwfwRn5trNPJgRTFYICV8IrANofIm6Da3eDXmubVV3HoPeDoN0wgNlEp70RFZg+9iEPC0Dp7i3oFMS4l4f84m3JgrZHxxumLJPm5sIJOqMg8b9cz6uZjo3f/E6noFvM8KlHCbSf/g9Wy48O0fZuuTKmCaAqwK3+Y= Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: b1e72676-a9f5-47c8-170d-08d6861f1074 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Jan 2019 19:22:08.8028 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: HE1PR0502MB3067 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On Thu 24 Jan 2019 at 17:21, Dennis Zhou wrote: > Hi Vlad and Eric, > > On Tue, Jan 22, 2019 at 09:33:10AM -0800, Eric Dumazet wrote: >> On Mon, Jan 21, 2019 at 3:24 AM Vlad Buslov wrote: >> > >> > Hi Eric, >> > >> > I've been investigating significant tc filter insertion rate degradati= on >> > and it seems it is caused by your commit 001c96db0181 ("net: align >> > gnet_stats_basic_cpu struct"). With this commit insertion rate is >> > reduced from ~65k rules/sec to ~43k rules/sec when inserting 1m rules >> > from file in tc batch mode on my machine. >> > >> > Tc perf profile indicates that pcpu allocator now consumes 2x CPU: >> > >> > 1) Before: >> > >> > Samples: 63K of event 'cycles:ppp', Event count (approx.): 48796480071 >> > Children Self Co Shared Object Symbol >> > + 21.19% 3.38% tc [kernel.vmlinux] [k] pcpu_alloc >> > + 3.45% 0.25% tc [kernel.vmlinux] [k] pcpu_alloc_area >> > >> > 2) After: >> > >> > Samples1: 92K of event 'cycles:ppp', Event count (approx.): 7144680655= 0 >> > Children Self Co Shared Object Symbol >> > + 44.67% 3.99% tc [kernel.vmlinux] [k] pcpu_alloc >> > + 19.25% 0.22% tc [kernel.vmlinux] [k] pcpu_alloc_area >> > >> > It seems that it takes much more work for pcpu allocator to perform >> > allocation with new stricter alignment requirements. Not sure if it is >> > expected behavior or not in this case. >> > >> > Regards, >> > Vlad > > Would you mind sharing a little more information with me: > 1) output before and after a run of /sys/kernel/debug/percpu_stats Hi Dennis, Some of these files are quite large, so I put them to my Dropbox. Output before: Percpu Memory Statistics Allocation Info: ---------------------------------------- unit_size : 262144 static_size : 139160 reserved_size : 8192 dyn_size : 28776 atom_size : 2097152 alloc_size : 2097152 Global Stats: ---------------------------------------- nr_alloc : 3343 nr_dealloc : 752 nr_cur_alloc : 2591 nr_max_alloc : 2598 nr_chunks : 3 nr_max_chunks : 3 min_alloc_size : 4 max_alloc_size : 8208 empty_pop_pages : 3 Per Chunk Stats: ---------------------------------------- Chunk: <- Reserved Chunk nr_alloc : 5 max_alloc_size : 320 empty_pop_pages : 0 first_bit : 1002 free_bytes : 7448 contig_bytes : 7424 sum_frag : 24 max_frag : 24 cur_min_alloc : 16 cur_med_alloc : 64 cur_max_alloc : 320 Chunk: <- First Chunk nr_alloc : 479 max_alloc_size : 8208 empty_pop_pages : 0 first_bit : 8192 free_bytes : 0 contig_bytes : 0 sum_frag : 0 max_frag : 0 cur_min_alloc : 4 cur_med_alloc : 24 cur_max_alloc : 8208 Chunk: nr_alloc : 1925 max_alloc_size : 8208 empty_pop_pages : 0 first_bit : 63102 free_bytes : 852 contig_bytes : 12 sum_frag : 852 max_frag : 12 cur_min_alloc : 4 cur_med_alloc : 8 cur_max_alloc : 8208 Chunk: nr_alloc : 182 max_alloc_size : 936 empty_pop_pages : 3 first_bit : 21 free_bytes : 256452 contig_bytes : 255120 sum_frag : 1332 max_frag : 368 cur_min_alloc : 8 cur_med_alloc : 20 cur_max_alloc : 320 After: https://www.dropbox.com/s/unyzhx4vgo2x30e/stats_after?dl=3D0 > 2) a full perf output https://www.dropbox.com/s/isfcxca3npn5slx/perf.data?dl=3D0 > 3) a reproducer $ sudo tc -b add.0 Example batch file: https://www.dropbox.com/s/ey7cbl5nwu5p0tg/add.0?dl=3D0 Thanks, Vlad