From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1030180AbcFOThO (ORCPT ); Wed, 15 Jun 2016 15:37:14 -0400 Received: from mail-bn1on0072.outbound.protection.outlook.com ([157.56.110.72]:1296 "EHLO na01-bn1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751907AbcFOThJ (ORCPT ); Wed, 15 Jun 2016 15:37:09 -0400 X-Greylist: delayed 39698 seconds by postgrey-1.27 at vger.kernel.org; Wed, 15 Jun 2016 15:37:09 EDT Authentication-Results: spf=pass (sender IP is 63.163.107.21) smtp.mailfrom=sandisk.com; fb.com; dkim=none (message not signed) header.d=none;fb.com; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac160c69-353ff70000000bcf-2c-5761ad39d033 Subject: Re: [PATCH 02/13] irq: Introduce IRQD_AFFINITY_MANAGED flag To: Keith Busch References: <1465934346-20648-1-git-send-email-hch@lst.de> <1465934346-20648-3-git-send-email-hch@lst.de> <0412b942-ea0d-d4eb-c724-8243d12ff6f3@sandisk.com> <20160615102311.GA16619@lst.de> <67ef7a1c-56e1-db2c-b038-f9784fc1f52f@sandisk.com> <20160615151415.GA1919@localhost.localdomain> <7f0b16bd-b39f-99e6-c1c1-6a508bf9bbbf@sandisk.com> <20160615160316.GB1919@localhost.localdomain> CC: Christoph Hellwig , "tglx@linutronix.de" , "axboe@fb.com" , "linux-block@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-kernel@vger.kernel.org" From: Bart Van Assche Message-ID: <86aa652b-48d0-a7bb-683e-bf43939aa811@sandisk.com> Date: Wed, 15 Jun 2016 21:36:54 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <20160615160316.GB1919@localhost.localdomain> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFupkkeLIzCtJLcpLzFFi42JZI8azSNdmbWK4wea/3Bb/9xxjs1i5+iiT xb2jX5gs9t7Stri8aw6bxfxlT9ktzs47zmaxedNUZgcOj4nN79g9Fu95yeTx7tw5do/NS+o9 dt9sYPP4vEkugC2KyyYlNSezLLVI3y6BK+P9puNsBQsFK+Y82cLSwHiCt4uRk0NCwERi29TD LCC2kMBcJolPm5W7GLmA7B2MEmu+r2aHKTrR1ssEkVjKKHHq62Q2kISwgItE47WvzCC2iIC6 xL9fr9gginqYJfa9+MwK4jALvGOSuH3jB1gVm4CRxLf3M4H2cXDwCthJtEw0AgmzCKhKHG6d CzZUVCBCovfXErCTeAUEJU7OfAJmcwpYSVy9MoEdpJVZwF7iwdYykDCzgLzE9rdzmEFWSQjc ZZX4f+MBK8Q76hInl8xnmsAoPAvJqFkI7bOQtC9gZF7FKJabmVOcm55aYGiqV5yYl5JZnK2X nJ+7iRESQZk7GO8+8T7EKMDBqMTDu2NNYrgQa2JZcWXuIUYJDmYlEd6+lUAh3pTEyqrUovz4 otKc1OJDjNIcLErivG73voQJCaQnlqRmp6YWpBbBZJk4OKWAkdEfIbnSML7af9cndcGIbz/f u96paVt7Wm59lE4dZ03HL63MDYF7XLgUZ6z9JCladLmWi6uz1uOOfaYbh07n5i9ewU+1Vu5O /GLUz7tNU27urULOnbqP8hxesn3U8X4YN03K3tys3k3Dle3h/h3XF76/eH8dg7jlPvG33oKJ hYuFGxbPrDylxFKckWioxVxUnAgAqMxWDZwCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpgluLIzCtJLcpLzFFi42Lh2siRohuzLjHcYOU5MYv/e46xWaxcfZTJ 4t7RL0wWe29pW1zeNYfNYv6yp+wWZ+cdZ7PYvGkqswOHx8Tmd+wei/e8ZPJ4d+4cu8fmJfUe u282sHl83iQXwBbFZZOSmpNZllqkb5fAlfF+03G2goWCFXOebGFpYDzB28XIySEhYCJxoq2X CcQWEljMKPHqdjmILSzgItF47SsziC0ioC7x79crti5GLqCaHmaJfS8+s4I4zALvmCTuHP0K 1s0mYCTx7f1Mli5GDg5eATuJlolGIGEWAVWJw61z2UBsUYEIid5fS1hAbF4BQYmTM5+A2ZwC VhJXr0xgB7GZBWwl7szdzQxhy0tsfzuHeQIj3ywkLbOQlM1CUraAkXkVo1huZk5xbnpmgaGR XnFiXkpmcbZecn7uJkZwGHNG7WC8PtH8ECMTB6dUA+OuGXOnu5862sI+u7ryutV+Lm9Fro+O r6TrXJZ5/eCOOXS7gnGpQySjXZ9RscuHeLZdP0qX9038vOWQQ9OOvp0botV+V2hUikbuPL/L 5wvDqku/y/fE9wrYf9zzb0fAhodV5bs8br2cZXtgt6awS5hxgkFX5iqu/SsC/vv/051llz1N bcKy2yuUWIozEg21mIuKEwG1+u/oEwIAAA== X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:63.163.107.21;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(377454003)(189002)(199003)(24454002)(2950100001)(50466002)(53416004)(106466001)(31696002)(2906002)(8936002)(8676002)(81166006)(356003)(4326007)(77096005)(5008740100001)(69596002)(87936001)(68736007)(81156014)(97736004)(4001350100001)(5003600100002)(230700001)(47776003)(561944003)(33646002)(54356999)(76176999)(50986999)(64126003)(83506001)(189998001)(65806001)(86362001)(31686004)(6806005)(586003)(93886004)(92566002)(23746002)(65956001)(2270400002)(36756003)(11100500001)(110136002)(65826006);DIR:OUT;SFP:1101;SCL:1;SRVR:CY1PR0201MB1770;H:milsmgep15.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BL2FFO11FD007;1:i7AuCxXN2MGMnbaiVI18WCQZ8ma+RE7FjeKsxETT3KxDTSpPr5++af8QowhtahwbWSOjTKV4p5qLr1GAw4QOPXTb8QbnfIbEEbfTnifNfDEwVbajVI9K0PS7f9C2zoaDLdjepuE1Hvu5SsPt1mD6S6jMjeSTpD9gVGaximHMP50OMaxs19ixGKO3MXrc07kzNVPFIRtlQAvanYNE62gMHAHJTtuGiF5milwU0eHdDdZ84IM7XnL0Ag98qtT7c9RrVXVjQLGIDljwWTh02UroRshg7F1Nbnb5hj8afhK6lkQOOEKR0mCXkWOwRtbXUrq0f1oxFv4vgGF8CXWGg5s6Bt9nV99RfPeJe+g0Ibm+XQixTjO+wCPEfuzIZYOLYBE3HXDIkOHOzbuZZWocDK63iwRDF5YcxDp2cDlYzvwrUXQ6oom6gd9UcNPzIhJkinuRuPMdYrgOBgsBMR1jYYiL0a2NiF4gLUsH8n6TkGrkteS1RIUi4cozvCMHTwfnyWr6 X-MS-Office365-Filtering-Correlation-Id: 8835452e-e35a-449e-14aa-08d395546e83 X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1770;2:papf6PYHjc8MZsBGtOXTQpSUG4Y9ySO/nRp0ns/k+k5rOcqtJuNO0XcYpZhs9Ta9xVL8e5iTKBvSI/6BSoxQPgbNDvDjHRjsytXtF6xfvPZn4UCiPkD0THMMqN0RBxqUDp8UNbBTu6PF2H3rpIkYZKSqUl20Us0FZudTZt8oqY+cmId7ls6u0EzwzDJ1ehU/;3:vcFcme3rQAGB3u43678myuxDP+8Lx5YecOjPY52PEHGAQAM9BAcXnBbaErmEpWnD2WEI1nAw3H/s+WjI6n87FXMrlpCn1RibLXSp/QIeCEJTg8h2vY9ub4PbSsAWodpwRQZ6tyqM0gdsw9l2CC34XXwnRBZqP3hCwRNpzDrx5p6IDbjh4OjSqBC0W3y73lRYgfE2zVCo0tTFeQK+CzYy9j9tj6puDCvdob6H2XXgqGEHFhm5ywzv71Ltb7T6QunV2oOIJBJQ6/NN7qlEGSb0Zw== X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:CY1PR0201MB1770; X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1770;25:TGifoRIV8MAmuK2pEV59itGTAcb7VhkuM0KrzNoZi5urKzi4W6Lww3dN8QFppnmKT4dne6b4sEVYsOiqneFnTfjrmO22D5K2S0IpTp71OD83HVh+OA5sYURr/PAAQY5Ou2/HIOCR3VzrhkDk5Z2xfdtH2xUoS5cVorfT/s27uaQqWG+iOsxqKnHLcW1Uy5cHjIiRjbbHG5ZWhmUmcov8FooPUVQPKNPS+nKUy9HdO9HxJWWmxqUB9NcUWj1g84eGDIgOFdioW505WHMhGFR4Vbg3AXgRqcfvQRUfGgQO3n1TzzWbpJ61XG2WfvxpXWdKXgg7Rhw58WT9B4KezMf5kU9FNsPqoHm9uf704mg13i4gkeZF4knGp11xdGXw33aGIjk+gdA2aiuT8nYX411YmeaVxTDYwUrZlMndpVzesdwjRj5GGaMxbDN+26uDoXDB1AxrKgQnKy9zHxYstr/T2GoZJxLeiopV/DitjaxLl2VVCDzeXAdHx0iBMD4huY69IFyFTe2nUoVPC2u7y9BhN7QfGJDzqNmzwfETOJIwVgoMFpyTzFt839eYtVm0nbBcC192gL0mUkPmoX/774w9BB7exoAqt8hPHlEFDrGRdnrE7Iun7wYYHWYTvrQ792nUNsgJoJOcrkqNBxIL3i7c29DOa88/FVYaP4vzap+mk/u31fBainkj2UoV0ewGZzyDyCWAtNl8gR3W6QB9HYjy5IiQptBlW4AIxdCh9+CSWgA6UTkCDWD+GSYgmrpqhGFaBytIej85092O+FJYEzYO9w== X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1770;20:OAawPLGRo83crZ+nfZzeCJpms6HVkEZ5G3KPVEGHenW5N8hvdd6AYVNzhsmMoC1ptugIFhEwmhgZKYrU2O6jvq7feUrP4zQW2fWV+8MT6Y01vDvNo+wyQ9pBdE5nQ0ANwPhMOXv+2huUv8rCRQJMtqlUKTBVw+dRW0AmvlnPa1hIPWVhM2dYdw5H6LuIcYB+6VaZlUjpJUrdzY+d6WzH7F4LCi3U5rUR2qNuPZJ9d2gaRSAmE40ZMG/XIhqFL22w1NuxjYXBlN6eqVQU1/JLybdHB8031bg37CYPlQhAF5P7scsicHIVTsyxmkf+z8zkY5MIRla9oXI1KJZEsgCSdzrWU9vCZw8HepMVIAW8K5vKu5deaSN6G3xvDaIw0++LDJ1PmijsuImv4JU9+aCI+1dzhZceo1boJapCRDovu1VRZ1ZJqq2tLHHqYd/HX+7iXsFdfL/NzKc+s5KjSN7/iUZ0cknqlJb4nzV3NGy/Vo9v2m+BvEyickIF40BTnqV/;4:ZPuuQ9ZNBedql1T8VwwyRO2PycoGlzfaO5gq+u9zMpmFn75/bEJFbcfS8jJyyIXu1G0ldAc3VriOGJSPQVJMEz85byFxDVmXTFSN7ZRCb53I6FTOc0PSwuJiw2sFYg86w7R6OIAPG1qpohv0YWtgg/nXpaWhy2P4d3JoaBqMX1njXGAKN/zYTo3So5vVExeoLdxLDReL6P8OcOLNlmDY5sNbwYXSbaWC9e0VsQC8yPMqgxMkVwBpUOnY9W02FMhLaAUd/Icwszof26EqBWNQXTa7NYZhnYHAKeXIm2od487svybyrA1pgkVucj2xuhU9WAXjArMjOGOuYnZTSzyJqmxE8eh6R8djtL5grGy7SOTErh5r5kaExLjBTjlexOCVrtFanaiLTSP6ZlQzdZp8RdfF1+EK/Obt7d8/m7kbo9w= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(13016025)(13018025)(8121501046)(5005006)(10201501046)(3002001)(6055026);SRVR:CY1PR0201MB1770;BCL:0;PCL:0;RULEID:;SRVR:CY1PR0201MB1770; X-Forefront-PRVS: 09749A275C X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;CY1PR0201MB1770;23:UDHJ/N3GWgd+jJcxRTPHXxeNCkVNCKLmWBy?= =?Windows-1252?Q?tPwlaE4O5LoPz38Io5TrwNVekZ1bJUmJyqz7Ag5hq2zYrVbLcqX6tw8a?= =?Windows-1252?Q?ZJCQrCEpJH1BuxdmvfqrENFhkI0v63/GJ/MvpThS/6W+KDSSOJiwoLJg?= =?Windows-1252?Q?h35Xql9RDRMoLAa/l8BN4nkQF/Mv7WNOz7PuJxyEW8sxxtGgLQeXzbv4?= =?Windows-1252?Q?z0cIFbiE96BDcuSQe3p0vSodsZtMC1cxoUa9vov4AXxdbAx6+qwrE6Yk?= =?Windows-1252?Q?kTGiYyqUu8t+0YigqpDeCY8cBVnmX5yTck4DE1wcv+5ANB3wwopGz99V?= =?Windows-1252?Q?QsMcLHH9+vpmKd1FPKpzTfawQUNz/CpOpeaiGJiRIxCkGj4gA68v8/YO?= =?Windows-1252?Q?3O1D4VKPLLQ2XXYAguaickvuc8zqc5iK/eDAwSKOAO+Gr5Vt7e+stTNG?= =?Windows-1252?Q?EtIO+Diu47W3ZgZ2MRoxuh2Qgv65GNxdDHs8t/vhyaYm0oN39SdT8UTP?= =?Windows-1252?Q?yKfS6cFI57cIJ8DLAgIqKJ6k7u6oQ3FisEOz2RHUCIcyUUuABnKuJZDg?= =?Windows-1252?Q?sjCZMDCkEvWARCnoErc9GpAi0syqEPJDE9ww3dqjP56dSjD5hXByovXx?= =?Windows-1252?Q?+N+RhBm479o5b7K4f5veTY0ZQOwBdNaxuYa0rIrthKQMUHmWMCU2fd6r?= =?Windows-1252?Q?yLBoIX5vhhnD6FrdjdwIeWl6IUGZ1SVQY2xH/WXPWbDfAg0q+ATwx18l?= =?Windows-1252?Q?YkVqjgc4UNs1CNIo4/zdJ5AlDEMw/1i8+t1NXhlHdQYZTDIYdt2MxTA+?= =?Windows-1252?Q?4JyOJO7Avn3LVVaJRKSZtKLOL/9Hiu3MxNFOa6oNjB+cQlcBEGDEueX1?= =?Windows-1252?Q?yQT58bdXKT/V4ebZjqSQ6tecL21vK76CW62tQ0NFp+JvKn6tQOz4s2Q7?= =?Windows-1252?Q?JYVU62yaLg8CWxF69xdSYgldj0ouxyWdlweKDdzyUxqsMfC49EIwWrDk?= =?Windows-1252?Q?01dRPeIlUjikJ+IsKLyQi/sM/n9SR1UVq9TTPostd8ZYWB4eIp9zWooZ?= =?Windows-1252?Q?6IF/Ryq5Fnh8QSgvTgmxZaYdOiaAVQf1mE47c5yaMzgvIM92eldWUaNr?= =?Windows-1252?Q?RGcPVaQ+SI6DGonXH4NN6JzKjzE7S3F1Mh9bRVQi6CYYF23okSVYYEHz?= =?Windows-1252?Q?DpOWrQvGFGN3CIICWFRyL+DMe9G6OSF8x9p1zq4uNit6FbEA83YVD0eA?= =?Windows-1252?Q?wVROp2+q7+WJksw1ekCyVpkmyA7abAOjGzNMwIM25bgeiMfQUb8fiyEg?= =?Windows-1252?Q?DMKNSe8bK0vKySxuc2iWhoZHr3UFyinrb8bFZodU4tQlopUlgOQOw1pE?= =?Windows-1252?Q?QwzWY79n23kAnVTZBTbdnvL7UwqSdAx7LDg=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;CY1PR0201MB1770;5:GnzIpq3ya2LcfcM6rcZdiKcP7P+dydrzbIXibnMJVqk639RzOlf+ey74FmIOomehjuIrVvzoT0cVFZKX2NkuUqEELE38AOSxjTf93F2km+8e+gvLddI6EGJ61sptXBLKhD2MZ8SpfyAgapM3ks8nAg==;24:+xmQO4vKklmhxpwFBbT1phUoivXIa0M5vH77wtwDwztD2t9iF3Tvchgu6FzQYGwzx60R4yM9qrQIEIh1WAMcD3oJRge21+WjyN7PVuocmb8=;7:Wrl1+mfJhsgUtPOOYSSbZ9vT2692G/5rsOWYvhwXpGzY+JM1SoS4wp0YkquYhmiaqTaKYhFjOdjiCO9v8Whnu5PTE3cfxGmNQYqGcylvM38kbGzvWGsHQOC5Fsws0LaXyLVnitLyzvJroFtt+0Tm3VupMvSRQNZALUTfHGgSmUtxIhCYzkWBzvWHvCjzhmwlcudlHbFSnUDHjrBYRrqr5A==;20:vvJCLfES6AtEscjza9CxXXoIL5BvfnTBqo1C+hwR+Q1VwSZa6pz/oISiFnmbngGnAenG2oULq1uIpJmtmNrwpUfCedNcRu0iUKuNGwNWyga01a5f9wBAdVK9crrG5BmLGDUUqxVYPFUKFZc7BH81DG2OeibHRjvY7PAzItdrchIsmBnusNHn4IDSotqq59CqCC8pb992i5lJHmDBnodEa/c+h7mGKJwpDaQ+RaDiCxcG/9d/O35zIQMsZFCAN8fl SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jun 2016 19:37:04.5603 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.21];Helo=[milsmgep15.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR0201MB1770 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 06/15/2016 06:03 PM, Keith Busch wrote: > On Wed, Jun 15, 2016 at 05:28:54PM +0200, Bart Van Assche wrote: >> On 06/15/2016 05:14 PM, Keith Busch wrote: >>> I think the idea is have the irq_affinity mask match the CPU mapping on >>> the submission side context associated with that particular vector. If >>> two identical adapters generate the same submission CPU mapping, I don't >>> think we can do better than matching irq_affinity masks. >> >> Has this been verified by measurements? Sorry but I'm not convinced that >> using the same mapping for multiple identical adapters instead of spreading >> interrupts will result in better performance. > > The interrupts automatically spread based on which CPU submitted the > work. If you want to spread interrupts across more CPUs, then you can > spread submissions to the CPUs you want to service the interrupts. > > Completing work on the same CPU that submitted it is quickest with > its cache hot access. I have equipment available to demo this. What > affinty_mask policy would you like to see compared with the proposal? Hello Keith, Sorry that I had not yet this made this clear but my concern is about a system equipped with two or more adapters and with more CPU cores than the number of MSI-X interrupts per adapter. Consider e.g. a system with two adapters (A and B), 8 interrupts per adapter (A0..A7 and B0..B7), 32 CPU cores and two NUMA nodes. Assuming that hyperthreading is disabled, will the patches from this patch series generate the following interrupt assignment? 0: A0 B0 1: A1 B1 2: A2 B2 3: A3 B3 4: A4 B4 5: A5 B5 6: A6 B6 7: A7 B7 8: (none) ... 31: (none) The mapping I would like to see is as follows (assuming CPU cores 0..15 correspond to NUMA node 0 and CPU cores 16..31 correspond to NUMA node 1): 0: A0 1: B0 2: (none) 3: (none) 4: A1 5: B1 6: (none) 7: (none) 8: A2 9: B2 10: (none) 11: (none) 12: A3 13: B3 14: (none) 15: (none) ... 31: (none) Do you agree that - ignoring other interrupt assignments - that the latter interrupt assignment scheme would result in higher throughput and lower interrupt processing latency? Thanks, Bart.