From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.1 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CAE3C7112F for ; Mon, 21 Jan 2019 11:24:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 29D3A20861 for ; Mon, 21 Jan 2019 11:24:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=Mellanox.com header.i=@Mellanox.com header.b="HpJbSH3f" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727931AbfAULYt (ORCPT ); Mon, 21 Jan 2019 06:24:49 -0500 Received: from mail-eopbgr80071.outbound.protection.outlook.com ([40.107.8.71]:39136 "EHLO EUR04-VI1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727721AbfAULYt (ORCPT ); Mon, 21 Jan 2019 06:24:49 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SSMlBgeQE4yDko4dk8lsWgOtkjz7T07uaaYxpc2Cpfg=; b=HpJbSH3fXz1fCNQSJVN5k5UUmqATvr2yppI/C/2xwzoAYhWKmfPrZfXsW5rf0eeFmcfEcOejwYAsIiB6uavdTV0rg8CL5B1dnD1zKaB18hRpkiir2/zgsf5rNvTxC9An+gAT1/LwfY4lV0NmOPbiSS7RL7saSeljio+b8Kyl/9s= Received: from AM6PR05MB5717.eurprd05.prod.outlook.com (20.178.92.214) by AM6PR05MB5174.eurprd05.prod.outlook.com (20.177.197.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1537.29; Mon, 21 Jan 2019 11:24:44 +0000 Received: from AM6PR05MB5717.eurprd05.prod.outlook.com ([fe80::e0bd:9c86:6b53:f76f]) by AM6PR05MB5717.eurprd05.prod.outlook.com ([fe80::e0bd:9c86:6b53:f76f%3]) with mapi id 15.20.1537.031; Mon, 21 Jan 2019 11:24:44 +0000 From: Vlad Buslov To: Eric Dumazet CC: Linux Kernel Network Developers , Yevgeny Kliteynik , Yossef Efraim , Maor Gottlieb Subject: tc filter insertion rate degradation Thread-Topic: tc filter insertion rate degradation Thread-Index: AQHUsXvoWpbt+qtyHk2zMiwViDqQPA== Date: Mon, 21 Jan 2019 11:24:44 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: CWLP265CA0174.GBRP265.PROD.OUTLOOK.COM (2603:10a6:401:4d::18) To AM6PR05MB5717.eurprd05.prod.outlook.com (2603:10a6:20b:94::22) authentication-results: spf=none (sender IP is ) smtp.mailfrom=vladbu@mellanox.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [37.142.13.130] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;AM6PR05MB5174;6:iLHhvEyRbgAwTHe8VWKgPf+dtKgXtGQdlOsP94IxriNpkOTxJXFpQOapuh1lC7p8zIxQx66OF4kH2CrKqOHB4Ig0UGxKyX/uro9JsiuOo90qopJW5Dc82SBQAY+4yvCMqrj8CVHH2AbKiMxHl1pvOiabt2t5hVppjdPm7SYKMeNmKh70xTXtZXB/ffcObspBUQJagCyM8sFYI8XAf1vxfQYOMwXeZUIzTh/aEZjMXrYU3fLDBkM5UodqcyjvZadDNr00awhszik1duZogLgIzVpKMRZOHl5lEL9m/IeASIItQYEXkgR0lZALJ+0H5u/oPV8BAOPqSLhfEsl6smXoGlX2lmclt4oQnWoabkVSbp3OHhK/Moa+9PUddzXa1IHrg9o4QfDc7jx3z+gW9rZmqLZ6r+tdoJW/B8GtixvccBco1wKWEHd3Jgcs3tjWpMy8Of9qryOY/zwEliej4Fyiqw==;5:Sb45aD6++2NzHnE9vmkRDJteK64mMHkhiQnpscmKoMc2jWeY1DT9Nwm6zF3l62yA7e8Ao40rOS4LjzEKGI69M7fZwh44pMcmaR1ZlXda6BeaQl9BbCmKit9JjG+vPX1jJ5Ev6mU9fpJjJazkviU6dviIWyzt590ozGroOKtx+qwca7Mu/H0Tsj7X0ijhAFQMHuoS8iQp6cQJqwZDMOUDuw==;7:m6ynd9EF9JmYICvkzsvsLHvKf2wve/6qyuoTnb1dXz3862ho0wQGk5E0p17udzr33iu6odrmyHaEJfMPH8lNDjzcLz6ga+L4y3uLQ9gXnK20wZzoBw4EEccYeSbi2JyznEXCcD6CYDddWeIGTfowSA== x-ms-office365-filtering-correlation-id: 0f2cfc57-13d3-4329-e85f-08d67f930b40 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600109)(711020)(4618075)(2017052603328)(7153060)(7193020);SRVR:AM6PR05MB5174; x-ms-traffictypediagnostic: AM6PR05MB5174: x-microsoft-antispam-prvs: x-forefront-prvs: 0924C6A0D5 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(366004)(346002)(136003)(396003)(376002)(39860400002)(189003)(199004)(478600001)(52116002)(256004)(107886003)(476003)(4326008)(25786009)(2616005)(486006)(99286004)(106356001)(105586002)(26005)(81166006)(66066001)(186003)(8676002)(6436002)(6916009)(81156014)(386003)(102836004)(6506007)(6486002)(316002)(68736007)(54906003)(8936002)(6512007)(7736002)(71200400001)(305945005)(97736004)(14454004)(2906002)(53936002)(36756003)(86362001)(3846002)(6116002)(71190400001);DIR:OUT;SFP:1101;SCL:1;SRVR:AM6PR05MB5174;H:AM6PR05MB5717.eurprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: D4n9+/PAK/f8wHQzB1xvdcK9g5/ceXX+WfoZEokt0X2kdNXb37sEHdBcA7l9NpiTHYr7YhX0kv/bpQQrIgOdWNKB7bx9SYAdc1Zk7pJ9v5nz2FbArEQH5+pX73UevBeCMM3lcbjTdZ6/Ky/iXTHRoTALIdKzpkrkr9HGM4ghIk+sGyG+9Vh4j8eNA78YaYNDxzFkDP39O3zzvX5ugT4jMRBa+SJ3TU9DePqhUgq6wM1ClWSZadSkOOKf31heLrQrcD38VceuulRk1HMEfVoYeeqv9nUwfZXkIkEVS48BEScZFKRrPKXL0pVnNyK6FcmCqmA671CL95fyq2A8nE5NVXi/tAYCKOSmjUW9Jff7xqU+vSLI6HVcIMBbXBRh7IvUT2+Y2zM83w0G5NW55cHB82J/mSDYZBQ2JGt6G5I//2Y= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0f2cfc57-13d3-4329-e85f-08d67f930b40 X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jan 2019 11:24:43.9052 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR05MB5174 Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Hi Eric, I've been investigating significant tc filter insertion rate degradation and it seems it is caused by your commit 001c96db0181 ("net: align gnet_stats_basic_cpu struct"). With this commit insertion rate is reduced from ~65k rules/sec to ~43k rules/sec when inserting 1m rules from file in tc batch mode on my machine.=20 Tc perf profile indicates that pcpu allocator now consumes 2x CPU: 1) Before: Samples: 63K of event 'cycles:ppp', Event count (approx.): 48796480071 Children Self Co Shared Object Symbol + 21.19% 3.38% tc [kernel.vmlinux] [k] pcpu_alloc + 3.45% 0.25% tc [kernel.vmlinux] [k] pcpu_alloc_area 2) After: Samples1: 92K of event 'cycles:ppp', Event count (approx.): 71446806550 Children Self Co Shared Object Symbol + 44.67% 3.99% tc [kernel.vmlinux] [k] pcpu_alloc + 19.25% 0.22% tc [kernel.vmlinux] [k] pcpu_alloc_area It seems that it takes much more work for pcpu allocator to perform allocation with new stricter alignment requirements. Not sure if it is expected behavior or not in this case. Regards, Vlad