From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753186AbcFOKnM (ORCPT ); Wed, 15 Jun 2016 06:43:12 -0400 Received: from mail-by2on0086.outbound.protection.outlook.com ([207.46.100.86]:43461 "EHLO na01-by2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1750976AbcFOKnG (ORCPT ); Wed, 15 Jun 2016 06:43:06 -0400 Authentication-Results: spf=pass (sender IP is 63.163.107.21) smtp.mailfrom=sandisk.com; vger.kernel.org; dkim=none (message not signed) header.d=none;vger.kernel.org; dmarc=bestguesspass action=none header.from=sandisk.com; X-AuditID: ac160c69-33fff70000000bcf-37-57613010889d Subject: Re: [PATCH 02/13] irq: Introduce IRQD_AFFINITY_MANAGED flag To: Christoph Hellwig References: <1465934346-20648-1-git-send-email-hch@lst.de> <1465934346-20648-3-git-send-email-hch@lst.de> <0412b942-ea0d-d4eb-c724-8243d12ff6f3@sandisk.com> <20160615102311.GA16619@lst.de> CC: "tglx@linutronix.de" , "axboe@fb.com" , "linux-block@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-nvme@lists.infradead.org" , "linux-kernel@vger.kernel.org" From: Bart Van Assche Message-ID: <67ef7a1c-56e1-db2c-b038-f9784fc1f52f@sandisk.com> Date: Wed, 15 Jun 2016 12:42:53 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <20160615102311.GA16619@lst.de> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFmpikeLIzCtJLcpLzFFi42JZI8azSFfYIDHc4PZjdYv/e46xWaxcfZTJ Yu8tbYvLu+awWcxf9pTd4uy842wWmzdNZXZg95jY/I7d4925c+wem5fUe+y+2cDm8XmTXABr FJdNSmpOZllqkb5dAlfGn//7mQq+C1Ws7v3A3MDYyd/FyMkhIWAiceLJBLYuRi4OIYG5TBKb DvWzQzg7GCVOTJ7J2MXIAVb1cYsYRHwpo8TK7WeZQLqFBVwkGq99ZQaxRQSUJJ6+OssIUXSI UaLrwVYmEIdZ4AyTROOkI2BVbAJGEt/ez2QBsXkF7CQ6f/8Ds1kEVCVefroINlVUIEKi99cS qBpBiZMzn4DZnAI6En9PbmQDsZkFLCRmzj/PCGHLS2x/O4cZZJmEwElWibX3joINEhJQlzi5 ZD7TBEbhWUhmzULSPwtJ/wJG5lWMYrmZOcW56akFhqZ6xYl5KZnF2XrJ+bmbGCExk7mD8e4T 70OMAhyMSjy8EvYJ4UKsiWXFlblAD3MwK4nwxusnhgvxpiRWVqUW5ccXleakFh9ilOZgURLn dbv3JUxIID2xJDU7NbUgtQgmy8TBKdXAONtpWaRrXuWjSv/vZz4bTH4jGTQ1QUftbEtv9puk LY9UH529VGDpx9kkv+ioxZ32h7m3yi+vMvcwKeWasexr7aT/GXGxJ0olX7EKnO4Iafp1e4Jc xbQzs97fCnku7/hG5b/A8gM5xXdu/+Ze3bQ9UOGqRWfTXh7PsylT3bYW/ra/FPn0p+CcSUos xRmJhlrMRcWJAG8L2suVAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrGJMWRmVeSWpSXmKPExsXCtZEjRdfIMDHc4Mt5dYv/e46xWaxcfZTJ Yu8tbYvLu+awWcxf9pTd4uy842wWmzdNZXZg95jY/I7d4925c+wem5fUe+y+2cDm8XmTXABr FJdNSmpOZllqkb5dAlfGn//7mQq+C1Ws7v3A3MDYyd/FyMEhIWAi8XGLWBcjJ4eQwGJGib1P E0BsYQEXicZrX5lBbBEBJYmnr84ydjFyAdUcYpToerCVCcRhFjjDJPFj9mEWkCo2ASOJb+9n gtm8AnYSnb//gdksAqoSLz9dZAKxRQUiJHp/LYGqEZQ4OfMJmM0poCPx9+RGNhCbWcBMYt7m h8wQtrzE9rdzmCcw8s1C0jILSdksJGULGJlXMYrlZuYU56ZnFhga6RUn5qVkFmfrJefnbmIE hyxn1A7G6xPNDzEycXBKNTAa2fnac7vI/6g0ufurfV7K9KWtzku4thTtjlv829PthEdCu8sR xUT3nCPaCxlO7fOobmm60Byusfa7yrfVDV19608dY42O2Cz771+izIV4oRqThhNHZ+0uyL67 oEnlwuJ3zh9tvs5b8uug6ymLK5NOXs1b6Rd4cfrPQ6rNykW/74l3FXjGxTxXYinOSDTUYi4q TgQAB9RppQkCAAA= X-EOPAttributedMessage: 0 X-Forefront-Antispam-Report: CIP:63.163.107.21;IPV:NLI;CTRY:US;EFV:NLI;SFV:NSPM;SFS:(10009020)(6009001)(7916002)(2980300002)(438002)(24454002)(377454003)(51854002)(199003)(189002)(47776003)(53416004)(86362001)(33646002)(31686004)(4326007)(50466002)(106466001)(2906002)(586003)(356003)(11100500001)(83506001)(8936002)(64126003)(2270400002)(36756003)(77096005)(6806005)(76176999)(54356999)(2950100001)(189998001)(69596002)(87936001)(93886004)(92566002)(5003600100002)(4001350100001)(31696002)(5008740100001)(65806001)(81166006)(50986999)(230700001)(97736004)(23676002)(81156014)(8676002)(110136002)(65956001)(68736007)(65826006);DIR:OUT;SFP:1101;SCL:1;SRVR:BLUPR02MB519;H:milsmgep15.sandisk.com;FPR:;SPF:Pass;PTR:InfoDomainNonexistent;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: 1;BN1BFFO11FD042;1:eZN8zzEPq2kascncfe91o5N819aRKqvJNfbJm4Gx4laQlJmJkpOQg7J730q21msNA79nNRPEiw0akWbV0YdosxBWxj5V6dDwZGirJ4cNN8gVdVDKB7ZrSKHyFqAdD3GIqQZ3RUDJpXEZU+bZBBgMArpsyFV35amwjA0hOluxi+U16GEXaRIu1rNCoznuk26ShfWeA5DBWMYgr3TKoYWwNOmGWD+GTBuhO7skg1+dFz3SyTZzs0uxQXnsfEZVo9DJpHQHyRhj2cqLRGvt9aYM50a3I5EWcbq1Bj1WvdK3zERaLNfqUCzLKfH1y+xbYvE0t3dKXkEeIjN6uRDstZ6W8MJ4AiqkBJaVYhOAoOBb2d0WPusjIIW9CKqfBzByEeGghUKklNZMxG/h/HjetFw0Bkj/VQhmd43Q+fJfkWRMsr/qHHUd2DZ3WSGMhd92+ybfN09k0JnT7E1mLnear8g/4g== X-MS-Office365-Filtering-Correlation-Id: e5615c83-4d81-4568-6cf9-08d39509d375 X-Microsoft-Exchange-Diagnostics: 1;BLUPR02MB519;2:oM+pFXLxYwfk3zEwXplLAXNfFhdsAAZAyuYaLyUBPSPuVMbDYjVL+78gZRI5+YBjcfv/t6H2kd65IwSTI2SNjhrb3Tnqm1FfZCNnDJKzbfE4q4bGilGaQTbKXFdC7g1Ximkc8IhtJjRgme48A+A8RS/fa8bq1wpoWcstYTmPj7rPp03hD2VOv1y6GR+Zuzzm;3:s2iAFjQ7u1Z7MmLxZ6SIaQtfo6fyq5JsagUfpuqMtIEwug4slUBjaK7T9WXgsCpuaMfSFlYhReEO/A1rG8R8E3H5W6UzoJUpj/NDUM0VtQG+QNLR4Z31FNn7ad1aMpoo9yLqvcSkJD9QhjLdaia0+KnlThZ/lBNE085qrJ3WoWRo1yfaYbiVCVf17lQ95Sa0KkJe1kWKLdcmXiMqNMCQgeSosWZ2E8LM7TRzjTqAPfI4Bk/CiEaUoEL4rO+WR/e67TBOBigGBIdfcpMM+8H2Qw==;25:3gj6BotmZ7bH+tqjNr/bNmIcJd8sqoJPHnHZC+3P4qyV4Grt3dlbdYofZA/NPlI8TDfTUBqbUqmBELLvZKITFBSmIEhGWq3q47bHH7pT0aewCVRKSRQODlZLMJ9asr52PHFba8E7bKEFffgJy6TnsismZzud/2oRq13XVzFiVEPceG5lgn3wLjTzvrFZmV2iXCtgZwrKgeEmg+kpfrnXXm7AyQ3Wmk2BUyqJRyWqqZModcxW3T72X9td8SFSM+1G3gMVgMQZAWzHd2482vqvKGRuE+NYp8a621Bh5flThGsfKs9QBqDEXjT5FFLj8vGDNPuPQGHMwJL8hawsixaPWO3e8+iV7v+AsGjvAQGB5JUxYWSMTd80PNihLkDCCVvhRjlob7nabIPTv94GFkVKa7uEed+zcCyg8skJnXSTXqk= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(8251501002);SRVR:BLUPR02MB519; X-Microsoft-Exchange-Diagnostics: 1;BLUPR02MB519;20:ZZbfEkNxI10J7dQSk4asWnxHcJaSQG8Vh2PiA+EcjWZHFgbAhaufvR5ofQqtH4359bXvhUGKVX9f1M5/oA5dsy9Sv1adEW10Bbudc1NIngbz+hHIqehNi6ySCDTkl8Zg1HKwHScYfuAWCeB8pYMyfDPk8/1rbZRmaqmdcIVHRNeNGs2EW8XBpr9yevOJr6VbLXfXT0IHeJcW8p9A20hoCa5qQkon12dTUMoJmJCo62p5aZWbun2fEasAFVsuHxeVHVNaxgrQ7DTvwWrBKmONhDb6bGEAOsZCx0fjPYXoU42tGHi/bd87+ojckBuPqbPZoL0Dz1Rg20rWC3Y7GOsyzoyOUaUCMjrGiX38pljGm9X0GMfyt6/fCSIGEbV5XG3ZANDnRoRkMzcCQePpy66kdBqu+GA/z8OdyIE7JidiqOFbL5H1IKTbyNGhOnMGSXu6L5O8CVbOmW9DSCXFvFgGibVN/l+zLopKXHZ/GL68yMjxPJNH52TwRqoToZxUdpOL;4:Z4B7KBRDcEgVrZCSSyPda4jZPWgrGwTzlxNLlm8jEN5+uTVDO//79WSFhCs9iNgARpX+5hh6q++Xni1z6FDCHmp0lq03Nqj2aj4RX3euDoDIwW6uo9nSCBC7mXkON3LN5s+lXyCHwMvFPJh2/qRg8cNysoBfOuH2A/47In5PHGOuWKPaGhfArD2AeutCWOk4/eCTN7mrsnrJVY3GKjAjsVeCRFwSQklAYgUPTesUxF9qBu5tB5ylwOWe4Id30/8mULCpaon+muMpJbGeRiJm701PIdgIY8buy3m88XxSNL5Wpw1Ce+SFRk+Zprhz4MPqfoR0Nivp3kjpyznPkajbhGP3K27QAeZeR34zla781HZ4gHPqZLsEaNPyMXkSHj1cO5+/PSjqzPvg2VV3JRE4eRsCZ/0Yi6I0sXz3LUQKnTI= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(13018025)(13016025)(5005006)(10201501046)(3002001)(6055026);SRVR:BLUPR02MB519;BCL:0;PCL:0;RULEID:;SRVR:BLUPR02MB519; X-Forefront-PRVS: 09749A275C X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtCTFVQUjAyTUI1MTk7MjM6SDNsQ1FlY0RKQkhZcVFjS1J3azl1ZmYzVEF0?= =?utf-8?B?SkQxNkFqcXlTa2ZYc2MxeXNyWmRERWM3eWk5S1VDVzEvV3lQelhLNkQzWW1Z?= =?utf-8?B?VU1kSitPbmNrMjN3cXRXOE4wbEo3N1lYblR2UXNXcHVyNjFrZG50ZHhnSVk2?= =?utf-8?B?VzU5ZU9wTnVRTU85RVlTUmJ4YjZ4R3JnS1k3R3lSL21VN3M2OERvTU1PV3Rx?= =?utf-8?B?NmNwd00xYnllQi9NVlVwa0IvTThtRmRGZjRLWXFoZGVxTjB2STRhVXc1Zmk5?= =?utf-8?B?SXhqMlRmRDZEQWx2bDhOWk5tL2tHTTJxZmFKOUNUV3JGS0dFZHZpcXpxTVk1?= =?utf-8?B?QjNxc0JtYTlJNmV0ZmNUaVVaTm82NDZPdDZscm5DWFZESFV4ZXFGUkkxKzNH?= =?utf-8?B?RmM1Wk91c0kvRkZVSzJKemltZ3p2TTREV2JiTGpIbHlLRmdiN0RQdllSWnlF?= =?utf-8?B?MTdjRVVJZkdMQmkraWprZ2pXZ3BhdTI0M1hJem11UjlYK0IyV3BvTHVrMDAv?= =?utf-8?B?MlN3N2laZVRpSFN0bHNJSy84MkpKSUR4eEdxSFAzN0ZvKzhiam9BYXVJVU9n?= =?utf-8?B?SUJweEZxZTJ4L1pGa0JnbjBvY3FnUWw0aHdWcThITnZOaENqaFB5aDlxY1Vy?= =?utf-8?B?TEt3eksrM2ZoZThIakZ1TWk0c013YUtSajErNkxwNFJxMGxqdllwOXJhbzMw?= =?utf-8?B?amZENUpGWVBWQzd1dWhxV1JURXl0eXFyZUJGczRlU3RHR2tJVGsraVNTbFpi?= =?utf-8?B?TUdpdzVKcTJNQkpsL0Yzd1M3VDd0WlRybTFPNXBZcDI2OHpTYjZ1RUJPZXVa?= =?utf-8?B?K2NteU01QXZidXVMWWhoSlZ0NU1yYXNZWnJFaCtwRzF5Um5WMnNNZHp1aWVR?= =?utf-8?B?bjFIaFlVaytXNVVxeTVPL1VUTXRDa1JGdWhlNDRIdlA2Q1Q2OEZZNHBOSVl2?= =?utf-8?B?K3JoNXVOZnQ0Uk9aOEFmNXV0YVh2SjUwRmp5NnJ2cFA2UWZKM2xjUGxWRjhG?= =?utf-8?B?TG9VVm1UMm43SU9JclYxSVlRdHlJMy9EOGxLZUV1NzhzUlAvalBRVnM0TG5y?= =?utf-8?B?MkhiUFBxREt1Q0N4V3dxY1h6RWJrdDNtQTMwS0pTWTNVeUpyb2tSM25rb2J6?= =?utf-8?B?Y09RMXlNZGR3dmFPN2JyMlZLZ1g3ZXlhbXk4eTh4REJJc2tpdEhFdFh4UXdC?= =?utf-8?B?VmdwWlBYNDZiWTZiVEVuNGZNbnZQLzVvM1pNZmtOMTYxb1VuRUdrMGFaa0lx?= =?utf-8?B?dTNya1FscmpSc0lMSzhFdXZlenMzVUN6b0J3ZmM2a0dGUmdIUUx3VHFKNWNK?= =?utf-8?B?SzdZMDZ6Z0g4VnhDTHJoWWsyNERVSlZ5QUh0TEhKcHFCQ0FZOFA2UWptdDhr?= =?utf-8?B?T2R4ZEN6VE1NcUxXMVBSaXgwZmV3ODF2V3FESjBGMHJNMVNTMldtNUQrWXdN?= =?utf-8?B?cHoyUk82NkVuT1RvWkZ5SDFEbGljZUtkZmRmajk4QjNFUTRBWnZPOW4vV29S?= =?utf-8?B?NWRzNWthTjBIR01aeVE2UDBIMk9jUm5PWUtOVWJkWDlqcGdRRStwcEJUdDh1?= =?utf-8?B?Z2x1Y0wwQTJreldMNy8ybmtVRFlhYXlpaEJKSHE4NEN5WkxOaHRObExmUkJl?= =?utf-8?B?ejVXVFNrRG9rL1RWR1BXOFVjR2VxY0lxVnBUTlEwRDY2R3o0RWI0LzEwTXY3?= =?utf-8?B?eUk1ZFVQTXJULzBOc1lGSkVQL2pWTHFSWWVueVQveEh3RDdwZ3BGaS8yd2dw?= =?utf-8?B?ZElOTk5YL2FzODRET3FnPT0=?= X-Microsoft-Exchange-Diagnostics: 1;BLUPR02MB519;5:wq4n1+4CWKwkK3Ua8lFzXZxpESNpRIUKmVw4/1hjCXVGh3QNcSr2J257YZt36pxF01L04YuA0QZDjVvLym1wWukZ8HHN9u7lJAVxh4ZO0L5Svhi5Sx+tSIZNqVX4phuOTAHhUkrpfLaH/nr+eYB62Q==;24:ZmNc+7jOf18vv8TAtUWUpJU/tkihLSY117q3lnMcYhwfOCQMpQ8ZdUebbJYJX0THY085rQd9xUzsPW89T58D4HLC8DaVZA7AbjGHUw2GZf4=;7:mfst2VeX1im9xoPcwZFLSYtyRSo1HKmNuHr2E766yz1tNLba7ePEHBKdJCJWr5D67sKCmg91yMM+L/syRVbaz+iKIvnvZ4RysE4VwjTJjrmg21FfhsdPngPl4IrvNVg1rkK4yIwqqdQt7SPs0aoOWpLDgdTlQTu4KiSt5umDCnKM4XafiVQKvDid5/xNw2FnIRw9Mspd3ZRa3a5p85scdA==;20:+49pWWliKoEkRPO376c/ieJZfOHXy4iz2aGD4ijMfrDwApLERsQzHPsrR9z8GwQoebKAo6t/aCwoh9A67yYtVq68kCDzkFXxkAgd8j9a62/UoAzjrw0qr+UDYDbES2NjvLrE/s8WN7eEfW0cCjQ945r91//Y6iK4XIivivgb7+6fWO1Q5Py6Qv50AKs4F83QOjX3za7y3J1+INaDjQlMYhxjjQXVA8K+muBJuFzztAyD4okfhim8v3J54v5ZSBVK SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: sandisk.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Jun 2016 10:43:01.7284 (UTC) X-MS-Exchange-CrossTenant-Id: fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=fcd9ea9c-ae8c-460c-ab3c-3db42d7ac64d;Ip=[63.163.107.21];Helo=[milsmgep15.sandisk.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BLUPR02MB519 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 06/15/2016 12:23 PM, Christoph Hellwig wrote: > Hi Bart, > > On Wed, Jun 15, 2016 at 10:44:37AM +0200, Bart Van Assche wrote: >> However, is excluding these interrupts from irqbalanced really the >> way to go? > > What positive effect will irqbalanced have on explcititly spread > interrupts? > >> Suppose e.g. that a system is equipped with two RDMA adapters, >> that these adapters are used by a blk-mq enabled block initiator driver and >> that each adapter supports eight MSI-X vectors. Should the interrupts of >> the two RDMA adapters be assigned to different CPU cores? If so, which >> software layer should realize this? The kernel or user space? > > RDMA should eventually use the interrupt spreading implemented in this > series, as should networking (RDMA actually is on my near term todo list). > > RDMA block protocols will then pick up the queue information from the > HCA driver. I've not actually implemented this yet, but my current idea > is: > > - the HCA drivers are switch to use pci_alloc_irq_vectors to spread > their interrupt vectors around the system > - the HCA drivers will expose the irq_affinity affinity array > in struct ib_device (we'll need to consider what do about the > odd completion vectors instead of irq terminology in the RDMA stack, > but that's not a show stopper) > - multiqueue aware block drivers will then feed the irq_affinity > cpumask from the hca driver to blk-mq. We'll also need to ensure > the number of protocol queues aligns nicely to the number of hardware > queues. My current thinking is that they should be the same or > a fraction of the hardware completion queues, but this might need > some careful benchmarking. Hello Christoph, Today irqbalanced is responsible for deciding how to assign interrupts from different adapters to CPU cores. Does the above mean that for adapters that support multiple MSI-X interrupts the kernel will have full responsibility for assigning interrupt vectors to CPU cores? If two identical adapters are present in a system, will these generate the same irq_affinity mask? Do you agree that interrupt vectors from different adapters should be assigned to different CPU cores if enough CPU cores are available? If so, which software layer will assign interrupt vectors from different adapters to different CPU cores? Thanks, Bart.