From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751350AbcGLSvn (ORCPT ); Tue, 12 Jul 2016 14:51:43 -0400 Received: from mail-by2nam03on0111.outbound.protection.outlook.com ([104.47.42.111]:34400 "EHLO NAM03-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751316AbcGLSvk (ORCPT ); Tue, 12 Jul 2016 14:51:40 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=waiman.long@hpe.com; Message-ID: <57853C33.8000705@hpe.com> Date: Tue, 12 Jul 2016 14:51:31 -0400 From: Waiman Long User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:10.0.12) Gecko/20130109 Thunderbird/10.0.12 MIME-Version: 1.0 To: Tejun Heo CC: Alexander Viro , Jan Kara , Jeff Layton , "J. Bruce Fields" , Christoph Lameter , , , Ingo Molnar , Peter Zijlstra , Andi Kleen , Dave Chinner , Boqun Feng , Scott J Norton , Douglas Hatch Subject: Re: [RFC PATCH v2 6/7] lib/persubnode: Introducing a simple per-subnode APIs References: <1468258332-61537-1-git-send-email-Waiman.Long@hpe.com> <1468258332-61537-7-git-send-email-Waiman.Long@hpe.com> <20160712142727.GA3190@htj.duckdns.org> In-Reply-To: <20160712142727.GA3190@htj.duckdns.org> Content-Type: text/plain; charset="ISO-8859-1"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [72.71.243.165] X-ClientProxiedBy: BLUPR05CA0077.namprd05.prod.outlook.com (10.141.20.47) To AT5PR84MB0308.NAMPRD84.PROD.OUTLOOK.COM (10.162.138.30) X-MS-Office365-Filtering-Correlation-Id: b2442739-f415-4677-2f1e-08d3aa858e48 X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0308;2:b1jYUfYdPIykZhq14Y9Gkx1MgZvY/rrXRPToIxoZhfHsyP6KPpcRNxAyWcFJVLSj3td4z4j9yaVHvbmY98MkNaurYz7cr5Gb84Qk7BCDyczUpXyUyT19GDUrKaHAwU+tuDWWEKilg3YBop6hcSJhpdNnXnawdu1FpqIyC5tbESlGAw85FLqEzmyDvul6JQFq;3:RAqQ1RLl5+EFHb2OeMwlIvwvd0RqCqeAEHYf55g285Zty7e1oLCSJId7e3ct82eRSHyAsxx6TTXlr9LXXpEuR6KD9IWQB1zukV37T1kv4ecU50pFcTvAElIBbgnTj/3r X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AT5PR84MB0308; X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0308;25:i0VGzq4NvMxMOnc9VlUfTHeoJDiPOKDyDAsgMzVLK2yB55glavuSj4Q3nlqKflCALWOtxcVzVwnt3Coobgq2VTA0fIRPvI8yA3s2GZWZjAZOyE3p8SrAuvNHUQF6iFONE10FIfkMoJXhypYm6UY8OVNSWbW3wpAvanchkShyjQcZBKmCr5rz4p6tudefkwWihdxYQNf0tLJv/BRVD1hQTwIMh8BCq4W/UupJoREXT1TQtltekK/BFCYSOVGL/NH9XyQgYMyrx8/NQs78IPVlOo3NCI079c1n5k51KaRMI9uNHkI3w7Z42G7hy2AFuUPSPBI8b1FN6jVH2nwXSFhyo5nngVkDkUNdt2rDkHX6pACtdshyd5uKF+IykNPbt90edW0A1g7OHg97YWmTIhSziTwp5Ra8Cf8712XlQe9CP6oP7pSPLolOMNqAHCuau4KL5wm3RYhfSiKmOuA3hM/bZHjZQNJYqPvrni1G/a9471bIEIP3rquf2G3LvYZxlMuUUb098KHH95RoahA52BrIoSbC/1beNFFTx1Gi9SBygkXKLKWjWPTSy0dc2HCcnifzTxAKO3wiMLpOeIUiXP/vGy61Gj/guxIfrL/lejM2y0sL6yAgBcr0j5aw0BIoqv+lGohLjiHhpzaYuPH97KdPlRppweqAY/m+SOLF5bKqw8x+ieeCtd15rUuynxEtPKnpA8hhZ1IWJoYEMgu8P3UDTA==;31:135En0nqYecVNguQDIs80Gqwqzn/8ixzahBPdcxcjdPMFcZ6Mc+b4DB/+stpPItWBV3cKFIHmgcLt0w71zvJ061BjcibcGtc2BBEpYxl+gSONHvp9UEfS9n+faxoD/vsVPLN8aOCg8bnJAkmBhCnxpHrnHZpYdvxiLKsyNdhafN2NAHKRU6xhPaPA5T0+/kDEfQbReERtRhANtXeahdfHg== X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0308;20:b7UFahPvGKEYw0B6xJI6BA5YO+6jN4PsAXXMJNC7UhnxRFe13glYoLo4HYpj7TWa5tbv0RLw+clOdzm5lf1jAA99BzUeXugPpuhpP4ViB6fvjUQvDqv0HR4nNXpfoZTFgly8A1i88qM8TxmSgaebsAXNq3rgpj1rdl5osshdBbn9MU/BNwpNDETxDnERv6bydBm58xRjoTHvUgnwxhiRRwIztOO2t79vgH5Jt9uWZ1wnfUK6AE+vGKzCRM1W+a5er0PTAmWm/TLfBI5ToRCTv6VZBc/U1WCdvpW2pLb0SlBS6c01GL/dq6SqRkDWLRje3stP8ls/U9arzcIWhL7vOVBbm+hsRPJc+ohXmjLYrGMMPqC+kt0pMhDYpIsnsgkNdsHuQ5bFV6fMDVgaxT4iM9vQuCW7h198UPsy/tZd2dF9bC3S5D8tj9Td5+ewCaDKOdUdG7fsdD2E3/yVb75Z5WG9K7cnbMLoLDhUsbuwBg0mBFkiipEuQigD+cjiGfh3;4:d8DobuDj/Qr2TNvWfFvz/Zvi1ektbLIhkXzQ9nLXLoN+Z148ejP0UeNfQnePXN6mgeIT6ohw5lLMtbjBfJHaxZsz8NUPvFBfc2ThJQRdYEaP3LPfPIMdIFd43/DsSzV23Kd36fuhYBpip0I3Mu8oorWu+xCPovZ8muoRpBb9706lJCtl7Nx/uXu813EBCxxN+WG9IsF3ehSo26bZelEKKMPew/kuJ7Vfu657fLgh09cH/I96EKX0WloQZYpgn3rcVYP5rT3Z+W+sAD9o9I7NpyKE4gQSzTjOlFaiNW6Gd5hq0x+a8V9Tez7gk/2kLqiU10z1eboyVSvqhU5XezMYo24aEVjZ3b6CncWPa2fkKedIbfOhX2spXD1HMQh54Ag7 X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(10201501046)(3002001);SRVR:AT5PR84MB0308;BCL:0;PCL:0;RULEID:;SRVR:AT5PR84MB0308; X-Forefront-PRVS: 0001227049 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(4630300001)(979002)(6009001)(6049001)(7916002)(24454002)(189002)(377454003)(199003)(68736007)(106356001)(8676002)(99136001)(59896002)(36756003)(4326007)(105586002)(101416001)(42186005)(117156001)(81156014)(81166006)(189998001)(76176999)(87266999)(65816999)(50986999)(54356999)(3846002)(586003)(110136002)(6116002)(230700001)(80316001)(23756003)(97736004)(77096005)(47776003)(65956001)(65806001)(7736002)(305945005)(4001350100001)(50466002)(86362001)(92566002)(33656002)(2906002)(66066001)(64126003)(7846002)(83506001)(2950100001)(217873001)(969003)(989001)(999001)(1009001)(1019001);DIR:OUT;SFP:1102;SCL:1;SRVR:AT5PR84MB0308;H:[192.168.142.166];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?iso-8859-1?Q?1;AT5PR84MB0308;23:Hoo/4T1/inu/d4T8kkUprJIBY8wRnU5bjsTRdUT?= =?iso-8859-1?Q?0iNDiLxQXWuh7i319lGTAsFKGnumS9HSXP60quWYPP4+PMMyUDVvoiRaBZ?= =?iso-8859-1?Q?leVeosGq2l0/fS8xxpLCwGc76sijk04n3vZF1c+DSbXTmPEY8hWTw3fUSC?= =?iso-8859-1?Q?ZkNL8aX3cgLpTibH7NUTf7ODhTU0i9Dl+kAD80FmZM5kSds6Fsfa3MpFez?= =?iso-8859-1?Q?EpZOXiiTajr57R35AFPuythJTXh8qbo8usUlh3q0fSEudAj6PuXN9F42+W?= =?iso-8859-1?Q?AvCgnznByke3RAy4Ltwz/S538q7l1sqe3MRv3nkly8/CzmvYv0AqXs20Yk?= =?iso-8859-1?Q?S5VjkKWEV2pZH91fO8nXdTqdUjE03XneDNeG/GQcQOJnWfxWVQiLkwanzy?= =?iso-8859-1?Q?aJy6X8F+U2DU/Aap+AsKzw0uPrkE8jdfZSWk0XHdVlFzVBiEm5+0WDPL74?= =?iso-8859-1?Q?6M+Xl/vDhUlJFC68nRLx8wSKbb4lht9BDJgW1au6xMcKUicTuekAN1T+Vo?= =?iso-8859-1?Q?W3dXW1MrxDH6+5bN1KPcvmH9IUpqg302OoPb3Xf7E9ds72NlQU+ATMaPNR?= =?iso-8859-1?Q?zQGokdpQocmQH+EE6QGfY4/j1JpiGealtiviqV2Td/kv9VjdJGHQTSBgUs?= =?iso-8859-1?Q?T/sF6/WQLh2mAiu9eAZHMuSrZHgypCeXtZHUB5BKgqRifOl5RMGBBHEOLE?= =?iso-8859-1?Q?96AUfHELDQwMpiIghBP4mXyL36WLCxy1SS0wzv5DM5xScBYJ3fAVWZ6aXi?= =?iso-8859-1?Q?xcci7jzRfTORNROyPLAJKsO2i3aILw+bpmj3q/Ylo38ybDFZ6XDSghZ7UF?= =?iso-8859-1?Q?mi0nRNdKHtyGdkfWjleHeqA4ad6VcQEYyal2c0YjdXvJOLhFd+RVziNpIQ?= =?iso-8859-1?Q?hHUDxRU4kNKtiK8qWh0bc7JWc4Mns6aEzdXpH2E0kFpQcFqpTkT4DhH848?= =?iso-8859-1?Q?4sP/ZmqNW37iBelj13fhXk+MuKWLHr/0XsxDWhSKru7+zi1uGgbUZOngSc?= =?iso-8859-1?Q?Y/GckLsths++aFIv+w1PMeSUxlydPGq48dyBAtI197v0d7FKJo1NSwBDhn?= =?iso-8859-1?Q?zNkzdFC0guCP3ozBD6B3N9bdMU8t05EuAdI9D53j6vl9DSXia+jbpuzzZO?= =?iso-8859-1?Q?9dkT4nEF9CXAM0aGCq+RfrbbdZnYuJ8sQLdvaQJ3ZsPc1dD/P/xiC0MUOg?= =?iso-8859-1?Q?rFKvH7+oBezVdLE4neQuvFFw5VzPSR1sGpC/6BNz4kxv43kk88MP8nmsHv?= =?iso-8859-1?Q?ilE9fl16SYDvhNc+l0qQb3poGQlJSWkZDsx5j6U2DesoweKB8T4DbTZcTN?= =?iso-8859-1?Q?7LnBL/YAcpqgnyWccnlW++z4n8tIjtxh9wC70PKfug/bzy3RNuCRiwDtTa?= =?iso-8859-1?Q?yAQ3XCSV2fSUcYS9g/84R0iWrxcAF7OBxImrkM28OVqSqVs23vvigdJMtr?= =?iso-8859-1?Q?f2YKC3yxprg9miO1GGMxsH0AmpcIvQX7jWa?= X-Microsoft-Exchange-Diagnostics: 1;AT5PR84MB0308;6:5EM1Ych6P6ZstA9mwxfF5H80wqVU8qXzs6L9JIaCt38ru7RB+ynaAuBvIYvmxOfOjfKNrJ7cSN3gteHEaTNfjwK89r+GxnC9lkDwijb77Q5ZYu7Vo7ABdmT+xN8UwaRjvz0QrGHL3PlgV4AItyK5ChGfbi1kn2A5qy/okOSMXgHYxxc/8nRkaP190rdo4lQoCZnQSIiJVnMxx+glW/M7c+jfMG4RN8LkRBhtoCgE1fRPb2K855q5nfa82d8efmDGrynsbSp0PlkzwxvhZEaXbOYzdpiHyQPzs3uL0Z1Hsk4=;5:VijXuNN/xqk5h6VcRw+4Kgu+8rulvUJAn0PqP9LCXYcQbjvu8sJExJkRXqVVdj6HhAHstBCR5K2elPslV9UYp5ddKIxFzSK0eOOQQKflh6pEyE8FjdRz/Z5S3QUlve+2bOY9ZF1R8Ex6YB7F83wcyw==;24:ooZOYwumsv2WDgPk/2Z0YRif31A4rCL9SsB+l8DNejwHIyvIRGDRc59ld76I+zFW2Ipotjjoe9N22qeKX8B/65+3VnKxFZdNv/9jBTcTjBY=;7:U+2V3v6TJYmH0uaU336olJ+qYPbiaznEorwakhO0GkK7iq6Uu61aHuW8Nsfkt0VYwjLLm78k9L2ez60f7ijPrx1+SAd+0bHM9D8gygIHmDxqje/G9z2UwyZTllA2Qk6Gr44Vku0wI06kZJymbf+Ss//AP8AjM2vIJrU4o3ZLMQC1MMDFUlKQh3JxU9dB4sB1hB81qtsqXDaw5rjbcNQFwpxWSyBEBFrMxuQV9scCb1VrDkkQbGEtA11Ksbwd8vPZ SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: hpe.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Jul 2016 18:51:37.4251 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AT5PR84MB0308 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/12/2016 10:27 AM, Tejun Heo wrote: > Hello, > > On Mon, Jul 11, 2016 at 01:32:11PM -0400, Waiman Long wrote: >> The percpu APIs are extensively used in the Linux kernel to reduce >> cacheline contention and improve performance. For some use cases, the >> percpu APIs may be too fine-grain for distributed resources whereas >> a per-node based allocation may be too coarse as we can have dozens >> of CPUs in a NUMA node in some high-end systems. >> >> This patch introduces a simple per-subnode APIs where each of the >> distributed resources will be shared by only a handful of CPUs within >> a NUMA node. The per-subnode APIs are built on top of the percpu APIs >> and hence requires the same amount of memory as if the percpu APIs >> are used. However, it helps to reduce the total number of separate >> resources that needed to be managed. As a result, it can speed up code >> that need to iterate all the resources compared with using the percpu >> APIs. Cacheline contention, however, will increases slightly as each >> resource is shared by more than one CPU. As long as the number of CPUs >> in each subnode is small, the performance impact won't be significant. >> >> In this patch, at most 2 sibling groups can be put into a subnode. For >> an x86-64 CPU, at most 4 CPUs will be in a subnode when HT is enabled >> and 2 when it is not. > I understand that there's a trade-off between local access and global > traversing and you're trying to find a sweet spot between the two, but > this seems pretty arbitrary. What's the use case? What are the > numbers? Why are global traversals often enough to matter so much? The last 2 RFC patches were created in response to Andi's comment to have coarser granularity than per-cpu. In this particular use case, I don't think global list traversals are frequent enough to really have any noticeable performance impact. So I don't have any benchmark number to support this change. However, it may not be true for other future use cases. These 2 patches were created to gauge if using a per-subnode API for this use case is a good idea or not. I am perfectly happy to keep it as per-cpu and scrap the last 2 RFC patches. My main goal is to make this patchset more acceptable to be moved forward instead of staying in limbo. Cheers, Longman