From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3EFE1C54EBD for ; Mon, 9 Jan 2023 21:10:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237735AbjAIVJc (ORCPT ); Mon, 9 Jan 2023 16:09:32 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S238113AbjAIVIz (ORCPT ); Mon, 9 Jan 2023 16:08:55 -0500 Received: from out-97.mta0.migadu.com (out-97.mta0.migadu.com [91.218.175.97]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 883A2A6FC0 for ; Mon, 9 Jan 2023 13:00:37 -0800 (PST) Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1673298035; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=mpz5QV62fd9Z/8dQCjZswuJCHVa8jPKuhq5d3gUONBY=; b=kjgDRTMPn7t5yVbfabYw3U4Bt86OfUO9sPG5HT3E7m0rSa22kXiU1LjNxmbaXzxeG5wCwC CwIoYRtI441fXya0QIcD1Dy5FSM5f3KcRh6qO/eaggEgJc1TrRwP4osowhvZwtMfH3jIO/ 1ui4TstB3NRV8yKFr3NDiPhKTs4WT58= Date: Mon, 9 Jan 2023 14:00:33 -0700 MIME-Version: 1.0 Subject: Re: [PATCH] PCI: vmd: Do not disable MSI-X remapping in VMD 28C0 controller Content-Language: en-US To: Xinghui Li Cc: Keith Busch , nirmal.patel@linux.intel.com, lpieralisi@kernel.org, linux-pci@vger.kernel.org, linux-kernel@vger.kernel.org, Xinghui Li References: <20221222072603.1175248-1-korantwork@gmail.com> <3d1834d9-7905-1225-741a-f298dd5b8a8e@linux.dev> X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Jonathan Derrick In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org As the bypass mode seems to affect performance greatly depending on the specific configuration, it may make sense to use a moduleparam to control it I'd vote for it being in VMD mode (non-bypass) by default. On 12/27/2022 7:19 PM, Xinghui Li wrote: > Jonathan Derrick 于2022年12月28日周三 06:32写道： >> >> The bypass mode should help in the cases where drives irqs (eg nproc) exceed >> VMD I/O irqs. VMD I/O irqs for 28c0 should be min(63, nproc). You have >> very few cpus for a Skylake system with that many drives, unless you mean you >> are explicitly restricting the 12 drives to only 6 cpus. Either way, bypass mode >> is effectively VMD-disabled, which points to other issues. Though I have also seen >> much smaller interrupt aggregation benefits. > > Firstly，I am sorry for my words misleading you. We totally tested 12 drives. > And each drive run in 6 CPU cores with 8 jobs. > > Secondly, I try to test the drives with VMD disabled,I found the results to > be largely consistent with bypass mode. I suppose the bypass mode just > "bypass" the VMD controller. > > The last one，we found in bypass mode the CPU idle is 91%. But in remapping mode > the CPU idle is 78%. And the bypass's context-switchs is much fewer > than the remapping > mode's. It seems the system is watiing for something in bypass mode.