From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1D9BF22D4DA for ; Fri, 9 May 2025 02:04:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746756286; cv=none; b=m2M5Uer1XyyulTkcC70yYMW9POuko9wUpeg/P4gM5fkCUL1sJQ7QZdzkDlR8oGEIgqX1lOvZ+TpWOVrdUetV/PhacL5la8vCfCK1g4EcXmXGBX91Pi/xR5+LNjKPzDQpmwQ+iXzS1WjzpoT8moRnhzxIBqI43cPgU9KhIxbsFn8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1746756286; c=relaxed/simple; bh=+qfZDALLUhrJhLp9Y8tdMpOzOUBGedANZ5WjS6Kpy/A=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=YSNqUUsoNDJdXE7sMl+wI0cDOhkOJZ+QtqC8dg6p40v5F9yGHg3trG5GZ+ItUwkZx20us9kYi8/YLX6C5l+f9h3AFUe0XVBDpzkUavJy6OjSr076PaQl6CYFT2T162qGIxTN7xQMagvwLIMUKH5bt/h6Z+DhCYqw6EmyqIHa22s= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=A2nhecG5; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="A2nhecG5" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1746756284; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=xmiJGCWH2hP12TenpYZdFu2FrjT+vq7C54lLFOzY/dY=; b=A2nhecG53IxTMARtdqGYE/0p4nEGG+qrE0wkwSaDoxOKhezUZY9xQClcEYxiDQOJc6pN8J q/4qOloLhQZJC/98zG2htCsoqP7l6Z2g+KMTDiSlJ9G3q3Vn+9ABda4srDJGFKlkodNZdu GPtoihclY53nUozzjyTFp6OjUEWFXVw= Received: from mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-47-Zis-y8ssOV-_Ywxk8Qv7Cw-1; Thu, 08 May 2025 22:04:40 -0400 X-MC-Unique: Zis-y8ssOV-_Ywxk8Qv7Cw-1 X-Mimecast-MFC-AGG-ID: Zis-y8ssOV-_Ywxk8Qv7Cw_1746756278 Received: from mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.93]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id D32241956080; Fri, 9 May 2025 02:04:37 +0000 (UTC) Received: from fedora (unknown [10.72.116.120]) by mx-prod-int-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 24C4A180049D; Fri, 9 May 2025 02:04:23 +0000 (UTC) Date: Fri, 9 May 2025 10:04:18 +0800 From: Ming Lei To: Daniel Wagner Cc: Hannes Reinecke , Daniel Wagner , Jens Axboe , Keith Busch , Christoph Hellwig , Sagi Grimberg , "Michael S. Tsirkin" , "Martin K. Petersen" , Thomas Gleixner , Costa Shulyupin , Juri Lelli , Valentin Schneider , Waiman Long , Frederic Weisbecker , Mel Gorman , Mathieu Desnoyers , linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-nvme@lists.infradead.org, megaraidlinux.pdl@broadcom.com, linux-scsi@vger.kernel.org, storagedev@microchip.com, virtualization@lists.linux.dev, GR-QLogic-Storage-Upstream@marvell.com Subject: Re: [PATCH v6 6/9] isolation: introduce io_queue isolcpus type Message-ID: References: <20250424-isolcpus-io-queues-v6-0-9a53a870ca1f@kernel.org> <20250424-isolcpus-io-queues-v6-6-9a53a870ca1f@kernel.org> <2db989db-4849-46a9-9bad-0b67d85d1650@suse.de> Precedence: bulk X-Mailing-List: linux-block@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Scanned-By: MIMEDefang 3.4.1 on 10.30.177.93 On Fri, Apr 25, 2025 at 09:32:16AM +0200, Daniel Wagner wrote: > On Fri, Apr 25, 2025 at 08:26:22AM +0200, Hannes Reinecke wrote: > > On 4/24/25 20:19, Daniel Wagner wrote: > > > Multiqueue drivers spreading IO queues on all CPUs for optimal > > > performance. The drivers are not aware of the CPU isolated requirement > > > and will spread all queues ignoring the isolcpus configuration. > > > > > > Introduce a new isolcpus mask which allows the user to define on which > > > CPUs IO queues should be placed. This is similar to the managed_irq but > > > for drivers which do not use the managed IRQ infrastructure. > > > > > > Signed-off-by: Daniel Wagner > > > --- > > > include/linux/sched/isolation.h | 1 + > > > kernel/sched/isolation.c | 7 +++++++ > > > 2 files changed, 8 insertions(+) > > > > > Reviewed-by: Hannes Reinecke > > Just realized I forgot to also add some document on this new argument: > > io_queue > Isolate from IO queue work caused by multiqueue > device drivers. Restrict the placement of > queues to housekeeping CPUs only, ensuring that > all IO work is processed by a housekeeping CPU. > > Note: When an isolated CPU issues an IO, it is > forwarded to a housekeeping CPU. This will > trigger a software interrupt on the completion > path. > > Note: It is not possible to offline housekeeping > CPUs that serve isolated CPUs. This patch adds kernel parameter only, but not apply it at all, the above words just confuses everyone, so I'd suggest to not expose the kernel command line & document until the whole mechanism is supported. Especially 'irqaffinity=0 isolcpus=io_queue' requires the application to offline CPU in order, which has to be documented: https://lore.kernel.org/all/cc5e44dd-e1dc-4f24-88d9-ce45a8b0794f@flourine.local/ Thanks, Ming