From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 28F86C4167D for ; Mon, 30 Oct 2023 16:38:20 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.625320.974520 (Exim 4.92) (envelope-from ) id 1qxVHJ-0003Hr-6f; Mon, 30 Oct 2023 16:38:13 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 625320.974520; Mon, 30 Oct 2023 16:38:13 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qxVHJ-0003Hk-3j; Mon, 30 Oct 2023 16:38:13 +0000 Received: by outflank-mailman (input) for mailman id 625320; Mon, 30 Oct 2023 16:38:11 +0000 Received: from se1-gles-flk1-in.inumbo.com ([94.247.172.50] helo=se1-gles-flk1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1qxVHH-0002l9-LL for xen-devel@lists.xenproject.org; Mon, 30 Oct 2023 16:38:11 +0000 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [2a00:1450:4864:20::529]) by se1-gles-flk1.inumbo.com (Halon) with ESMTPS id b5de12e3-7742-11ee-9b0e-b553b5be7939; Mon, 30 Oct 2023 17:38:09 +0100 (CET) Received: by mail-ed1-x529.google.com with SMTP id 4fb4d7f45d1cf-53de8fc1ad8so7593693a12.0 for ; Mon, 30 Oct 2023 09:38:09 -0700 (PDT) Received: from localhost ([213.195.118.109]) by smtp.gmail.com with ESMTPSA id f26-20020aa7d85a000000b0053e3d8f1d9fsm6263960eds.67.2023.10.30.09.38.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 30 Oct 2023 09:38:09 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: b5de12e3-7742-11ee-9b0e-b553b5be7939 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=citrix.com; s=google; t=1698683889; x=1699288689; darn=lists.xenproject.org; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=GAtWOQuGuBciqx4zP+O4k/vnaYzXMJngCsGOR6ayFhE=; b=EGxrAQAETufQd+V37qbzJvfCCbNTfpqcRDbLjM+wV3mRBbCSaxafwGy4kprekn1Sp2 a/z6o7hYlLScv3VCXx+2wpi3Af3cGUR0K5l85bwgRjvawAFsegALKRvfEipImrTLmROj 2Sclw0j/Oq3q/nPUHKO6I8dSfqIFcoXwl+PEw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698683889; x=1699288689; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GAtWOQuGuBciqx4zP+O4k/vnaYzXMJngCsGOR6ayFhE=; b=wC1/X+byrcIUY2Qre990T94gVDVK75MzitpriwkSI0BotG68seyIrp/9NpX760ySoM SHVCC2tBk+zj3defxLl3DlS+Za5qLsbfWu46upNkDCLVHzqyA5Gy8U/aAi21LllP1BbT R0uPCCR66Z6VFD94j4pD8UDRnVNCvCjcy6mZrX7f1X/hl2PhWoJbmI6hA8IEXxvV70hw 1yBLjPbofz+hcIqRFE70M57NQWm/wv7mnF7n+DDpqv3ONSFdIr47K/sgH7+xMij5NRfH HGZ3/juO2TXbWkNwaQnl63JhuXamOTDEpWVOpt0XjVNLULvFV8t3kUIQ9IR8yJRMidiQ TfUA== X-Gm-Message-State: AOJu0Yxx4zegUYkFzc9T3D40KxnasLOIVr7hcEybrs0q0nK0lzHdFA4H sxPeSSdUMA+5KfJez2UabOFhEA== X-Google-Smtp-Source: AGHT+IFtVR07qq7+4xAQ2nAHMyO8TrEGfDIGvlruH4f0frpQpGmz64WTa6FrdQICYILM47+uu56DOg== X-Received: by 2002:aa7:d351:0:b0:53e:467c:33f1 with SMTP id m17-20020aa7d351000000b0053e467c33f1mr9334254edr.8.1698683889559; Mon, 30 Oct 2023 09:38:09 -0700 (PDT) Date: Mon, 30 Oct 2023 17:38:08 +0100 From: Roger Pau =?utf-8?B?TW9ubsOp?= To: Jan Beulich Cc: Andrew Cooper , George Dunlap , Julien Grall , Stefano Stabellini , Wei Liu , xen-devel@lists.xenproject.org Subject: Re: [PATCH] x86/x2apic: introduce a mixed physical/cluster mode Message-ID: References: <20231024135150.49232-1-roger.pau@citrix.com> <6734e477-0aa5-c74c-4f64-02ca0415ae9e@suse.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: On Mon, Oct 30, 2023 at 05:34:48PM +0100, Jan Beulich wrote: > On 30.10.2023 17:10, Roger Pau Monné wrote: > > On Mon, Oct 30, 2023 at 03:32:56PM +0100, Jan Beulich wrote: > >> On 24.10.2023 15:51, Roger Pau Monne wrote: > >>> The current implementation of x2APIC requires to either use Cluster Logical or > >>> Physical mode for all interrupts. However the selection of Physical vs Logical > >>> is not done at APIC setup, an APIC can be addressed both in Physical or Logical > >>> destination modes concurrently. > >>> > >>> Introduce a new x2APIC mode called mixed, which uses Logical Cluster mode for > >>> IPIs, and Physical mode for external interrupts, thus attempting to use the > >>> best method for each interrupt type. > >>> > >>> Using Physical mode for external interrupts allows more vectors to be used, and > >>> interrupt balancing to be more accurate. > >>> > >>> Using Logical Cluster mode for IPIs allows less accesses to the ICR register > >>> when sending those, as multiple CPUs can be targeted with a single ICR register > >>> write. > >>> > >>> A simple test calling flush_tlb_all() 10000 times in a tight loop on a 96 CPU > >>> box gives the following average figures: > >>> > >>> Physical mode: 26617931ns > >>> Mixed mode: 23865337ns > >>> > >>> So ~10% improvement versus plain Physical mode. > >> > >> Nice. > >> > >>> Note that Xen uses Cluster > >>> mode by default, and hence is already using the fastest way for IPI delivery at > >>> the cost of reducing the amount of vectors available system-wide. > >>> > >>> Make the newly introduced mode the default one. > >>> > >>> Suggested-by: Andrew Cooper > >>> Signed-off-by: Roger Pau Monné > >>> --- > >>> Do we want to keep a full Logical Cluster mode available? I don't see a reason > >>> to target external interrupts in Logical Cluster mode, but maybe there's > >>> something I'm missing. > >>> > >>> It's not clear to me whether the ACPI FADT flags are meant to apply only to > >>> external interrupt delivery mode, or also to IPI delivery. If > >>> ACPI_FADT_APIC_PHYSICAL is only meant to apply to external interrupts and not > >>> IPIs then we could possibly get rid of physical mode IPI delivery. > >>> > >>> Still need to put this under XenServer extensive testing, but wanted to get > >>> some feedback before in case approach greatly changes. > >> > >> Looks quite okay, just a couple of minor remarks: > > > > Thanks. > > > > Do we still want to keep the pure cluster mode? > > I think we should keep it for a full release cycle or two. OK, will make the requested changes and send v2 then. Thanks, Roger.