From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 23AB7C3DA4A for ; Wed, 14 Aug 2024 22:01:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:Cc:To:From: Subject:Message-ID:References:Mime-Version:In-Reply-To:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=tTaieRrwiRFgVqREdWRHopdwoY rfCJWNAvBe3JbfgG0z6TcWcq+b9+MKI91Jdvm2jj8Db5F5qdb2KZaYBC269BDBtV8TDxEJylSdgYa pY4I0Zn4VGPnncesZn/L+R4MkacxEcbKDZShXyMxr94sl80Rg8GKmQCZGnLjQdTpKwPnRkm1jg7b3 ZUVmB7Q8PY9mRD8/16ClC3AHsw4NMyWlAVBtvsTv3JD+h0YbPbdbndToXNhmQGfrOFbFevknZBtUs a17C0OEGDAR7jwV+YQCX0FLeb4YMeFPOsJtL+EuuwVO87UJzfFfk43G+gaZ3hBgi4pB0e/mcLdjr2 bGCBrWSg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1seM3b-00000008O9t-1TX7; Wed, 14 Aug 2024 22:01:27 +0000 Received: from mail-pl1-x649.google.com ([2607:f8b0:4864:20::649]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1seM2v-00000008Nxv-1AId for linux-arm-kernel@lists.infradead.org; Wed, 14 Aug 2024 22:00:48 +0000 Received: by mail-pl1-x649.google.com with SMTP id d9443c01a7336-201e24bd4d9so4701865ad.0 for ; Wed, 14 Aug 2024 15:00:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1723672844; x=1724277644; darn=lists.infradead.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=euqwT0W81BDIhGMP2QFKuPI/nRkqD1mUGBYz/0Hjm+nRwcKnGmTwmmE4//w7KAO3Vt ub9rQVnK9/zijKak/zw96IZb0FoUtEYwHRJPu2hkOeFelRvAYp6H4o6/qyTPuxlNr5MK /+ACkjzanwYULRlWFCODwlIDwvQxhNO1dG2AHS6mwrsArZoItwuMf4bIJXJKevuuiZRW nb1FKkfuxS5Koi7eDA6K9edEpG8L6bk1AZOmuKeTAHPEG4C5Ubg1bLy39F9JU/VDwv39 19bYdi3begS9IgYE5hBC0iT/hMo4Ak7oEAlmqSFa5HITKe74S+7LvakrCOno/xMwToZR 77fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723672844; x=1724277644; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5oH6P3nqpQ94wKrIC/Mozd39AchaUcxSzRhAmiF3FGo=; b=rx5vAsi3l7KTwbzTxDVWPiEjaHOtd72uiTlSkaKtfuxsei0hILFeLglhCZXjUgsuHP tEDKweTvXFcWfdfyZU6q86aV29jx7vJqtOMVpSgMtqHripAgNfHGdiBqq+6d7mOA165N 8M9FgeYP6X1fdMekUD2Ozb2PWWKOneFbqee/dy2jCtW2/sGwU+H61nMS4mZXRDfezdKe TYkFF86m+jjQQo3CX1Fnuq+hlKizdGBJwSOvWX969iZBhGhCocsUFGa/zqCAAriqb/ME DRnjzCCovJBry01ssfiZRIDouMMDZ4cipk4gGnS3Ansl7pdSd80wjbU84xV2I14ERcfj mTOQ== X-Forwarded-Encrypted: i=1; AJvYcCW7ocGIDUXAGVQuyAAskXV7d3eCq3J95M9H4795zrrFglRb4S2xEMjtlTv0bj7PSiBZIz48FPKCt/sg41NdRNwpk7QDDASeNqL++uY8hqm367npI/c= X-Gm-Message-State: AOJu0YwQew6FMD3Cf9dh8Vv3A7HfL226GDzRx1v6J1ZjVNC9hVK+n3fz 7OJNRKng4Hk+l44ei1ApY+PVwFAigbwn1VNP6f3KWDEArX/QcKSyUO8AkEVYiMwYK3G/g1gy1OS M6A== X-Google-Smtp-Source: AGHT+IELSmNcIfcWqD1y6wfROgLi5On6yYa9ztLE8WB0+uhpire+X0WHbKhlLbGeYLBF6mQ5SB34xpmbxzM= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:902:da8b:b0:1fd:6529:7443 with SMTP id d9443c01a7336-201d64b4f7bmr3318245ad.11.1723672843639; Wed, 14 Aug 2024 15:00:43 -0700 (PDT) Date: Wed, 14 Aug 2024 15:00:42 -0700 In-Reply-To: Mime-Version: 1.0 References: <20240809160909.1023470-1-peterx@redhat.com> <20240814123715.GB2032816@nvidia.com> <20240814144307.GP2032816@nvidia.com> Message-ID: Subject: Re: [PATCH 00/19] mm: Support huge pfnmaps From: Sean Christopherson To: Jason Gunthorpe Cc: Peter Xu , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador , Axel Rasmussen , linux-arm-kernel@lists.infradead.org, x86@kernel.org, Will Deacon , Gavin Shan , Paolo Bonzini , Zi Yan , Andrew Morton , Catalin Marinas , Ingo Molnar , Alistair Popple , Borislav Petkov , David Hildenbrand , Thomas Gleixner , kvm@vger.kernel.org, Dave Hansen , Alex Williamson , Yan Zhao , Oliver Upton , Marc Zyngier Content-Type: text/plain; charset="us-ascii" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240814_150045_371651_6683EDFB X-CRM114-Status: GOOD ( 19.59 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Aug 14, 2024, Sean Christopherson wrote: > TL;DR: it's probably worth looking at mmu_stress_test (was: max_guest_memory_test) > on arm64, specifically the mprotect() testcase[1], as performance is significantly > worse compared to x86, and there might be bugs lurking the mmu_notifier flows. > > When running mmu_stress_test the mprotect() phase that makes guest memory read-only > takes more than three times as long on arm64 versus x86. The time to initially > popuplate memory (run1) is also notably higher on arm64, as is the time to > mprotect() back to RW protections. > > The test doesn't go super far out of its way to control the environment, but it > should be a fairly reasonable apples-to-apples comparison. > > Ouch. I take that back, it's not apples-to-apples, because the test does more > work for x86. On x86, during mprotect(PROT_READ), the userspace side skips the > faulting instruction on -EFAULT and so vCPUs keep writing for the entire duration. > Other architectures stop running the vCPU after the first write -EFAULT and wait > for the mproptect() to complete. If I comment out the x86-only logic and have > vCPUs stop on the first -EFAULT, the mprotect() goes way down. > > /me fiddles with arm64 > > And if I have arm64 vCPUs keep faulting, the time goes up, as exptected. > > With 128GiB of guest memory (aliased to a single 2GiB chunk of physical memory), > and 48 vCPUs (on systems with 64+ CPUs), stopping on the first fault: > > x86: > run1 = 6.873408794s, reset = 0.000165898s, run2 = 0.035537803s, ro = 6.149083106s, rw = 7.713627355s > > arm64: > run1 = 13.960144969s, reset = 0.000178596s, run2 = 0.018020005s, ro = 50.924434051s, rw = 14.712983786 > > and skipping on -EFAULT and thus writing throughout mprotect(): > > x86: > run1 = 6.923218747s, reset = 0.000167050s, run2 = 0.034676225s, ro = 14.599445790s, rw = 7.763152792s > > arm64: > run1 = 13.543469513s, reset = 0.000018763s, run2 = 0.020533896s, ro = 81.063504438s, rw = 14.967504024s Oliver pointed out off-list that the hardware I was using doesn't have forced write-back, and so the overhead on arm64 is likely due to cache maintenance.