From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp1.linuxfoundation.org (smtp1.linux-foundation.org [172.17.192.35]) by mail.linuxfoundation.org (Postfix) with ESMTPS id AD13D9F2 for ; Wed, 15 Jul 2015 17:25:58 +0000 (UTC) Received: from mail-pd0-f170.google.com (mail-pd0-f170.google.com [209.85.192.170]) by smtp1.linuxfoundation.org (Postfix) with ESMTPS id 607BD214 for ; Wed, 15 Jul 2015 17:25:58 +0000 (UTC) Received: by pdbep18 with SMTP id ep18so29394635pdb.1 for ; Wed, 15 Jul 2015 10:25:58 -0700 (PDT) Message-ID: <55A697A3.3090305@kernel.dk> Date: Wed, 15 Jul 2015 11:25:55 -0600 From: Jens Axboe MIME-Version: 1.0 To: Keith Busch , Bart Van Assche References: <20150715120708.GA24534@infradead.org> <55A67F11.1030709@sandisk.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: ksummit-discuss@lists.linuxfoundation.org, linux-rdma@vger.kernel.org, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, Christoph Hellwig Subject: Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On 07/15/2015 11:19 AM, Keith Busch wrote: > On Wed, 15 Jul 2015, Bart Van Assche wrote: >> * With blk-mq and scsi-mq optimal performance can only be achieved if >> the relationship between MSI-X vector and NUMA node does not change >> over time. This is necessary to allow a blk-mq/scsi-mq driver to >> ensure that interrupts are processed on the same NUMA node as the >> node on which the data structures for a communication channel have >> been allocated. However, today there is no API that allows >> blk-mq/scsi-mq drivers and irqbalanced to exchange information >> about the relationship between MSI-X vector ranges and NUMA nodes. > > We could have low-level drivers provide blk-mq the controller's irq > associated with a particular h/w context, and the block layer can provide > the context's cpumask to irqbalance with the smp affinity hint. > > The nvme driver already uses the hwctx cpumask to set hints, but this > doesn't seems like it should be a driver responsibility. It currently > doesn't work correctly anyway with hot-cpu since blk-mq could rebalance > the h/w contexts without syncing with the low-level driver. > > If we can add this to blk-mq, one additional case to consider is if the > same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu > assignment needs to be aware of this to prevent sharing a vector across > NUMA nodes. Exactly. I may have promised to do just that at the last LSF/MM conference, just haven't done it yet. The point is to share the mask, I'd ideally like to take it all the way where the driver just asks for a number of vecs through a nice API that takes care of all this. Lots of duplicated code in drivers for this these days, and it's a mess. -- Jens Axboe From mboxrd@z Thu Jan 1 00:00:00 1970 From: axboe@kernel.dk (Jens Axboe) Date: Wed, 15 Jul 2015 11:25:55 -0600 Subject: [Ksummit-discuss] [TECH TOPIC] IRQ affinity In-Reply-To: References: <20150715120708.GA24534@infradead.org> <55A67F11.1030709@sandisk.com> Message-ID: <55A697A3.3090305@kernel.dk> On 07/15/2015 11:19 AM, Keith Busch wrote: > On Wed, 15 Jul 2015, Bart Van Assche wrote: >> * With blk-mq and scsi-mq optimal performance can only be achieved if >> the relationship between MSI-X vector and NUMA node does not change >> over time. This is necessary to allow a blk-mq/scsi-mq driver to >> ensure that interrupts are processed on the same NUMA node as the >> node on which the data structures for a communication channel have >> been allocated. However, today there is no API that allows >> blk-mq/scsi-mq drivers and irqbalanced to exchange information >> about the relationship between MSI-X vector ranges and NUMA nodes. > > We could have low-level drivers provide blk-mq the controller's irq > associated with a particular h/w context, and the block layer can provide > the context's cpumask to irqbalance with the smp affinity hint. > > The nvme driver already uses the hwctx cpumask to set hints, but this > doesn't seems like it should be a driver responsibility. It currently > doesn't work correctly anyway with hot-cpu since blk-mq could rebalance > the h/w contexts without syncing with the low-level driver. > > If we can add this to blk-mq, one additional case to consider is if the > same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu > assignment needs to be aware of this to prevent sharing a vector across > NUMA nodes. Exactly. I may have promised to do just that at the last LSF/MM conference, just haven't done it yet. The point is to share the mask, I'd ideally like to take it all the way where the driver just asks for a number of vecs through a nice API that takes care of all this. Lots of duplicated code in drivers for this these days, and it's a mess. -- Jens Axboe From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jens Axboe Subject: Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity Date: Wed, 15 Jul 2015 11:25:55 -0600 Message-ID: <55A697A3.3090305@kernel.dk> References: <20150715120708.GA24534@infradead.org> <55A67F11.1030709@sandisk.com> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-rdma-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org To: Keith Busch , Bart Van Assche Cc: Thomas Gleixner , Christoph Hellwig , linux-rdma-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-nvme-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r@public.gmane.org, linux-kernel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, ksummit-discuss-cunTk1MwBs98uUxBSJOaYoYkZiVZrdSR2LY78lusg7I@public.gmane.org List-Id: linux-rdma@vger.kernel.org On 07/15/2015 11:19 AM, Keith Busch wrote: > On Wed, 15 Jul 2015, Bart Van Assche wrote: >> * With blk-mq and scsi-mq optimal performance can only be achieved if >> the relationship between MSI-X vector and NUMA node does not change >> over time. This is necessary to allow a blk-mq/scsi-mq driver to >> ensure that interrupts are processed on the same NUMA node as the >> node on which the data structures for a communication channel have >> been allocated. However, today there is no API that allows >> blk-mq/scsi-mq drivers and irqbalanced to exchange information >> about the relationship between MSI-X vector ranges and NUMA nodes. > > We could have low-level drivers provide blk-mq the controller's irq > associated with a particular h/w context, and the block layer can provide > the context's cpumask to irqbalance with the smp affinity hint. > > The nvme driver already uses the hwctx cpumask to set hints, but this > doesn't seems like it should be a driver responsibility. It currently > doesn't work correctly anyway with hot-cpu since blk-mq could rebalance > the h/w contexts without syncing with the low-level driver. > > If we can add this to blk-mq, one additional case to consider is if the > same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu > assignment needs to be aware of this to prevent sharing a vector across > NUMA nodes. Exactly. I may have promised to do just that at the last LSF/MM conference, just haven't done it yet. The point is to share the mask, I'd ideally like to take it all the way where the driver just asks for a number of vecs through a nice API that takes care of all this. Lots of duplicated code in drivers for this these days, and it's a mess. -- Jens Axboe -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752331AbbGOR0A (ORCPT ); Wed, 15 Jul 2015 13:26:00 -0400 Received: from mail-pd0-f180.google.com ([209.85.192.180]:33332 "EHLO mail-pd0-f180.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751764AbbGORZ6 (ORCPT ); Wed, 15 Jul 2015 13:25:58 -0400 Message-ID: <55A697A3.3090305@kernel.dk> Date: Wed, 15 Jul 2015 11:25:55 -0600 From: Jens Axboe User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Keith Busch , Bart Van Assche CC: Thomas Gleixner , Christoph Hellwig , linux-rdma@vger.kernel.org, linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, ksummit-discuss@lists.linuxfoundation.org Subject: Re: [Ksummit-discuss] [TECH TOPIC] IRQ affinity References: <20150715120708.GA24534@infradead.org> <55A67F11.1030709@sandisk.com> In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 07/15/2015 11:19 AM, Keith Busch wrote: > On Wed, 15 Jul 2015, Bart Van Assche wrote: >> * With blk-mq and scsi-mq optimal performance can only be achieved if >> the relationship between MSI-X vector and NUMA node does not change >> over time. This is necessary to allow a blk-mq/scsi-mq driver to >> ensure that interrupts are processed on the same NUMA node as the >> node on which the data structures for a communication channel have >> been allocated. However, today there is no API that allows >> blk-mq/scsi-mq drivers and irqbalanced to exchange information >> about the relationship between MSI-X vector ranges and NUMA nodes. > > We could have low-level drivers provide blk-mq the controller's irq > associated with a particular h/w context, and the block layer can provide > the context's cpumask to irqbalance with the smp affinity hint. > > The nvme driver already uses the hwctx cpumask to set hints, but this > doesn't seems like it should be a driver responsibility. It currently > doesn't work correctly anyway with hot-cpu since blk-mq could rebalance > the h/w contexts without syncing with the low-level driver. > > If we can add this to blk-mq, one additional case to consider is if the > same interrupt vector is used with multiple h/w contexts. Blk-mq's cpu > assignment needs to be aware of this to prevent sharing a vector across > NUMA nodes. Exactly. I may have promised to do just that at the last LSF/MM conference, just haven't done it yet. The point is to share the mask, I'd ideally like to take it all the way where the driver just asks for a number of vecs through a nice API that takes care of all this. Lots of duplicated code in drivers for this these days, and it's a mess. -- Jens Axboe