From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8E4D914C5B3 for ; Thu, 24 Oct 2024 08:00:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729756859; cv=none; b=SwE9ch08KYrEAnGMhJH0LeeaRL+DOxvX3sm3dcB17HnPZkbJS/EacM8k96i6MLtGn0EGBWTDH1FfCF2xLyeJxOwAI9bqPqtH208B9BWHPO+SPMGpHXOvi+jxmkee/aYglEhN/0KAjWZSGqWbLam8FzrNsv+8Vy/2/qV0G+hTRN0= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1729756859; c=relaxed/simple; bh=HAP0L2CUBvGLHAnYq2VFTU0BIBBgFUpMKfUzSDe20kk=; h=Date:Message-ID:From:To:Cc:Subject:In-Reply-To:References: MIME-Version:Content-Type; b=nX57o63ZJIwSnG5oj+PzeonTdmxG+ol7nWFP6KI/pW44lH9SXdYKMu0LSZk5+n4yljeS1D++UgWj8RZoY9/pkN0KqXnQHs+Y7S4Bk3/2SQEQlBCbQEe7PALmXpJANWiwEB0n7RrSHQiGqdSDkCP+V0U/WsFhk7OtrFIoaqGJMdw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LWy2MlXX; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LWy2MlXX" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1C4B3C4CEC7; Thu, 24 Oct 2024 08:00:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1729756859; bh=HAP0L2CUBvGLHAnYq2VFTU0BIBBgFUpMKfUzSDe20kk=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=LWy2MlXXsV37H/QfYuBbQz/a7SCirBcAtsJN6O8PuAT09J+Vdr/5/ZfG9PtXWh3G5 sq8QXT7tUwn8T55YNnzY6ENlELnDbPrPt4/3Z+7LtRHkCgFSanUkk2Lm3V/fNdaT3C dJ4PHfc9toYuF1LCZMdDRA3LgQw+LU+IGxC8M3v38tXwAlNwncLDFIGAkWc9LkIzja Y9fYBD1WDxlkZQ8Fp9o1qIviNw8b+kgNQAxFpkLCDAC358SjWTZvPWb8IYJuKDRudt YhXrBG/x6fu+SZp5jfBXK7OcVtnyKlPQmdgRUWbpXFf662HzUIMfEr+XbyGPqrnu84 6wuH0ubFjncSg== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1t3sm8-006MXf-O8; Thu, 24 Oct 2024 09:00:57 +0100 Date: Thu, 24 Oct 2024 09:00:56 +0100 Message-ID: <86y12e2amf.wl-maz@kernel.org> From: Marc Zyngier To: Zhiqiang Ni Cc: Oliver Upton , , , , , , , , , Subject: Re: [bug report] KVM: arm64: vgic-its: Performance degradation on GICv3 LPI injection In-Reply-To: References: User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.4 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: nizhiqiang1@huawei.com, oliver.upton@linux.dev, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, wanghaibin.wang@huawei.com, tangnianyao@huawei.com, wangzhou1@hisilicon.com, yuzenghui@huawei.com, wangyanan55@huawei.com, jiangkunkun@huawei.com, jiaqingtong@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false On Thu, 24 Oct 2024 06:06:58 +0100, Zhiqiang Ni wrote: > > Hi all, > > I found a performance degradation on GICv3 LPI injection after this > commit ad362fe07fecf0aba839ff2cc59a3617bd42c33f(KVM: arm64: vgic-its: > Avoid potential UAF in LPI translation cache). > > In my testcase, the vm's configuration is 60 VCPU 120G RAM with a > 32-queue NIC and the kernel version is 5.10. The number of new TCP > connections per second changed from 150,000 to 50,000 after this > patch, with the %sys of cpu changed from 15% to 85%. What is the VM running? How is the traffic generated? Without a reproducer, I struggle to see how we are going to analyse this issue. We can't go back to the previous situation anyway, as it has been shown that what we had before was simply unsafe (the commit message explains why). > From the ftrace, I found that the duration of vgic_put_irq() is > 13.320 us, which may be the reason for the performance degradation. > > The call stack looks like below: > kvm_arch_set_irq_inatomic() > vgic_has_its(); > vgic_its_inject_cached_translation() > vgic_its_check_cache() > vgic_queue_irq_unlock() > vgic_put_irq() Are you suggesting that it is the combination of vgic_get_irq_kref() + vgic_irq_put() that leads to excessive latency? Both are essentially atomic operations, which should be pretty cheap on a modern CPU (anything with FEAT_LSE). The patch below indicates that you are looking at a rather old kernel (6.8). What is the result on a more recent kernel (from 6.10)? Thanks, M. -- Without deviation from the norm, progress is not possible.