From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 021BEA46 for ; Fri, 9 Jun 2023 00:59:43 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-ba87bd29e9dso1724481276.3 for ; Thu, 08 Jun 2023 17:59:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=FMN3dhDuy89W3YL34iGrOcxlPjg1bzMkLLXo1TdyO66RGSvbHlps7gdvHr44ss3LGk MR0VFD1DRCFTNC9tyqK5ReMEOiPx2JNiCbTUTyUv9FMGeK9nCNoAU6WnXWYbBC+Dw27V niwud/7XN7pCGjqFT4B3VS+EBw+uLus4pO0EQ6k6SszjrU1zFcD9yQh/qzHLrmZknfvX XuK4cXbAzi1gsEYKdbrLfMfylLB6JTl2vYXAwWR55yMmpFjT1Lojk1XODC7sstmSmHo4 uXWebCC18fjzj/uapeQiZlzXqVxfdSytOTcShko5dqCAeC6bbLhPv2RLA/d/FJo+pzzy govw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=KX+nWapWDo2QjnHCy9UzdQ2O3VwLz635wdVVGKU6C4a0pXpzvwj4qndpRXJBjU9RuD SGAWzqEk73zwq320shgtceQ5eAaz21uDgTTAuZF8wMvJsu84HCvaSQ0toaLDuyicAyP9 wjjcq3kvwJPn3NTE4+FX12QiXueBr3FG9brkzZn7nVARDC5r7n0rId6Z9hVr2wi3a7Fh 4nRY53/gHQyRbXjXwKe80xxJVAgae7BvY+00cOIdM/1u36ELyBZpJuxd3JSYXahfFpza tUudSaivJpwWe1p2lz5u5SMmKt4sRdUT48NWLtmNJlA6cITBQh1VXN1O/vnXv2AQM0h2 3K5A== X-Gm-Message-State: AC+VfDx1DyYCbOqwcY+a7ByZOhy3MGhKqFVk/OuDWmxCzssStCBm80mw ikUu0bPcIU2PynZ+8KyEdP/yi28v48o= X-Google-Smtp-Source: ACHHUZ6QOfrw0l3bdpvvlmXUNnamszuXXqEgiHjhlXZ1bthx0t+lBEyPW/rr7RnEuOk5SiS3J1yRldpoYHg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a5b:784:0:b0:bad:99d:f087 with SMTP id b4-20020a5b0784000000b00bad099df087mr697256ybq.6.1686272382836; Thu, 08 Jun 2023 17:59:42 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:39 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005940.42722-1-yuzhao@google.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/powerpc: memcached benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com Content-Type: text/plain; charset="UTF-8" TLDR ==== Memcached achieved 10% more operations per second (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 184 On-line CPU(s) list: 0-183 Model name: POWER9 (raw), altivec supported Model: 2.2 (pvr 004e 1202) Thread(s) per core: 4 Core(s) per socket: 23 Socket(s): 2 CPU max MHz: 3000.0000 CPU min MHz: 2300.0000 Caches (sum of all): L1d: 1.4 MiB (46 instances) L1i: 1.4 MiB (46 instances) L2: 12 MiB (24 instances) L3: 240 MiB (24 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-91 NUMA node1 CPU(s): 92-183 Vulnerabilities: Itlb multihit: Not affected L1tf: Mitigation; RFI Flush, L1D private per thread Mds: Not affected Meltdown: Mitigation; RFI Flush, L1D private per thread Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio) Spectre v1: Mitigation; __user pointer sanitization, ori31 speculation barrier enabled Spectre v2: Mitigation; Indirect branch serialisation (kernel only), Indirect branch cache disabled, Software link stack flush Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-91 node 0 size: 261659 MB node 0 free: 259152 MB node 1 cpus: 92-183 node 1 size: 261713 MB node 1 free: 261076 MB node distances: node 0 1 0: 10 40 1: 40 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" HOST $ uname -a Linux ppc 6.3.0 #1 SMP Sun Jun 4 18:26:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838272 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x0009 HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-ppc64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" GUEST $ cat /etc/memcached.conf ... -t 92 -m 262144 -B binary -s /var/run/memcached/memcached.sock -a 0766 GUEST $ memtier_benchmark -v memtier_benchmark 1.4.0 Copyright (C) 2011-2022 Redis Ltd. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-ppc64 \ -M pseries,accel=kvm,kvm-type=HV -cpu host -smp 92 -m 270g -nographic -nic user \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 1:0 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=P:P \ -n allkeys -d 2000 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 0:1 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=R:R \ -n allkeys --randomize --distinct-client-seed Results ======= Before [1] After Change ------------------------------------------------- Ops/sec 721586.10 800210.12 +10% Avg. Latency 0.12546 0.11260 -10% p50 Latency 0.08700 0.08700 N/C p99 Latency 0.28700 0.24700 -13% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 80829C7EE29 for ; Fri, 9 Jun 2023 01:00:41 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4QcjS74gcnz3f6B for ; Fri, 9 Jun 2023 11:00:39 +1000 (AEST) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20221208 header.b=FMN3dhDu; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=flex--yuzhao.bounces.google.com (client-ip=2607:f8b0:4864:20::b49; helo=mail-yb1-xb49.google.com; envelope-from=3fnmczaykdkudzemftlttlqj.htrqnszcuuh-ijaqnxyx.teqfgx.twl@flex--yuzhao.bounces.google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20221208 header.b=FMN3dhDu; dkim-atps=neutral Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4QcjR56RZdz3c71 for ; Fri, 9 Jun 2023 10:59:45 +1000 (AEST) Received: by mail-yb1-xb49.google.com with SMTP id 3f1490d57ef6-ba87bd29e9dso1724483276.3 for ; Thu, 08 Jun 2023 17:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=FMN3dhDuy89W3YL34iGrOcxlPjg1bzMkLLXo1TdyO66RGSvbHlps7gdvHr44ss3LGk MR0VFD1DRCFTNC9tyqK5ReMEOiPx2JNiCbTUTyUv9FMGeK9nCNoAU6WnXWYbBC+Dw27V niwud/7XN7pCGjqFT4B3VS+EBw+uLus4pO0EQ6k6SszjrU1zFcD9yQh/qzHLrmZknfvX XuK4cXbAzi1gsEYKdbrLfMfylLB6JTl2vYXAwWR55yMmpFjT1Lojk1XODC7sstmSmHo4 uXWebCC18fjzj/uapeQiZlzXqVxfdSytOTcShko5dqCAeC6bbLhPv2RLA/d/FJo+pzzy govw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=VLg1lmMcwKgsmWKTUA9t4CfBcMLywIv3wR91E0yA4exRX/PuzJf17tBE4n0ls6HgZf excs28RgvOeOmNRC8wd59tzcBpQmTitlZsBDmXGI8kovGIlZxi46K5VM4M/+XAWOthhs hMnIy6VyqWoXUos3PNGSPS0kcRkmZXZ1IZ3Ab86QrEcDwPrpM1ZoXwkOkWsQCClYXZOc lFcON8YsjBg8rL9jf3msDd5jzEYEi1QIIU0mpJ7xu9dBMXCGoEjseeo+eQbsWwAqyWb2 sQP4iVvlYYFq9Ug9MOCKyJ2DgB5Y5gYWtNYnuk1llyBqe26/yppQLDkrF6lH7s/RV5P5 Zx+Q== X-Gm-Message-State: AC+VfDxPVdv235qEebW4YYNpbVTgUygzyhdGLtY8FbS0xqjWAsCRaSKy uOG5Qs1IpMwuGtMki1ulewgadzrlJBA= X-Google-Smtp-Source: ACHHUZ6QOfrw0l3bdpvvlmXUNnamszuXXqEgiHjhlXZ1bthx0t+lBEyPW/rr7RnEuOk5SiS3J1yRldpoYHg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a5b:784:0:b0:bad:99d:f087 with SMTP id b4-20020a5b0784000000b00bad099df087mr697256ybq.6.1686272382836; Thu, 08 Jun 2023 17:59:42 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:39 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005940.42722-1-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/powerpc: memcached benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Content-Type: text/plain; charset="UTF-8" X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "Jason A. Donenfeld" , x86@kernel.org, Gavin Shan , kvm@vger.kernel.org, linux-doc@vger.kernel.org, Catalin Marinas , Dave Hansen , Peter Xu , linux-mm@kvack.org, Ben Gardon , Chao Peng , Will Deacon , Gaosheng Cui , Marc Zyngier , "H. Peter Anvin" , Jonathan Corbet , Alistair Popple , Jason Gunthorpe , Ingo Molnar , Zenghui Yu , linux-trace-kernel@vger.kernel.org, linux-mm@google.com, Thomas Huth , Suzuki K Poulose , Nicholas Piggin , Borislav Petkov , Steven Rostedt , kvmarm@lists.linux.dev, Thomas Gleixner , linux-arm-kernel@lists.infradead.org, Fab iano Rosas , Michael Larabel , Sean Christopherson , linux-kernel@vger.kernel.org, Oliver Upton , James Morse , Masami Hiramatsu , Anup Patel , linuxppc-dev@lists.ozlabs.org, Mike Rapoport Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" TLDR ==== Memcached achieved 10% more operations per second (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 184 On-line CPU(s) list: 0-183 Model name: POWER9 (raw), altivec supported Model: 2.2 (pvr 004e 1202) Thread(s) per core: 4 Core(s) per socket: 23 Socket(s): 2 CPU max MHz: 3000.0000 CPU min MHz: 2300.0000 Caches (sum of all): L1d: 1.4 MiB (46 instances) L1i: 1.4 MiB (46 instances) L2: 12 MiB (24 instances) L3: 240 MiB (24 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-91 NUMA node1 CPU(s): 92-183 Vulnerabilities: Itlb multihit: Not affected L1tf: Mitigation; RFI Flush, L1D private per thread Mds: Not affected Meltdown: Mitigation; RFI Flush, L1D private per thread Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio) Spectre v1: Mitigation; __user pointer sanitization, ori31 speculation barrier enabled Spectre v2: Mitigation; Indirect branch serialisation (kernel only), Indirect branch cache disabled, Software link stack flush Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-91 node 0 size: 261659 MB node 0 free: 259152 MB node 1 cpus: 92-183 node 1 size: 261713 MB node 1 free: 261076 MB node distances: node 0 1 0: 10 40 1: 40 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" HOST $ uname -a Linux ppc 6.3.0 #1 SMP Sun Jun 4 18:26:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838272 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x0009 HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-ppc64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" GUEST $ cat /etc/memcached.conf ... -t 92 -m 262144 -B binary -s /var/run/memcached/memcached.sock -a 0766 GUEST $ memtier_benchmark -v memtier_benchmark 1.4.0 Copyright (C) 2011-2022 Redis Ltd. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-ppc64 \ -M pseries,accel=kvm,kvm-type=HV -cpu host -smp 92 -m 270g -nographic -nic user \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 1:0 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=P:P \ -n allkeys -d 2000 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 0:1 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=R:R \ -n allkeys --randomize --distinct-client-seed Results ======= Before [1] After Change ------------------------------------------------- Ops/sec 721586.10 800210.12 +10% Avg. Latency 0.12546 0.11260 -10% p50 Latency 0.08700 0.08700 N/C p99 Latency 0.28700 0.24700 -13% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/ From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A13ADC7EE29 for ; Fri, 9 Jun 2023 01:00:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1cS7wTCLFdTMeeKt3ChfQ1edAlwU3O65jl4u2E661vc=; b=cmLm6ACyzJ86RlwRol862IFF7i MP3xytDl1gTaySp0gRVcDayKDcsjWienoBRA3GLjBcG+vd6a67WAXN2/jFTy+2p3OdLqJPvJPb3ve GTBFI9LbqJNhRurNhT9yEA9aLzs4l2dAIWTt7iuTAAgkI8hk3Ea/s0K/YSf1Ds25N8JqyY5/tF8XW voVLxPbNojPVP/BXQR3STERPoyP/z4qubGPHg7cddJisxG8Q2Iqz60O0ZRLusvw/H6tXxvVKz55e0 rsq1Lk31WPxgff49t1jQt+NzYdT7ZRMZvm1fJT8TuoPHJv+xG6VhFjvkn1SwYENbZvMRudKg5lh/y 4mrbiybQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q7QTj-00B7ay-2j; Fri, 09 Jun 2023 00:59:47 +0000 Received: from mail-yb1-xb4a.google.com ([2607:f8b0:4864:20::b4a]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q7QTg-00B7Y0-1b for linux-arm-kernel@lists.infradead.org; Fri, 09 Jun 2023 00:59:46 +0000 Received: by mail-yb1-xb4a.google.com with SMTP id 3f1490d57ef6-babb79a17b8so1749601276.0 for ; Thu, 08 Jun 2023 17:59:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=FMN3dhDuy89W3YL34iGrOcxlPjg1bzMkLLXo1TdyO66RGSvbHlps7gdvHr44ss3LGk MR0VFD1DRCFTNC9tyqK5ReMEOiPx2JNiCbTUTyUv9FMGeK9nCNoAU6WnXWYbBC+Dw27V niwud/7XN7pCGjqFT4B3VS+EBw+uLus4pO0EQ6k6SszjrU1zFcD9yQh/qzHLrmZknfvX XuK4cXbAzi1gsEYKdbrLfMfylLB6JTl2vYXAwWR55yMmpFjT1Lojk1XODC7sstmSmHo4 uXWebCC18fjzj/uapeQiZlzXqVxfdSytOTcShko5dqCAeC6bbLhPv2RLA/d/FJo+pzzy govw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686272383; x=1688864383; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kfRq4neZrPjb9Lck7talXrCmxVTayuyWN8AeTYnUB40=; b=dMLS6zHsptY4lC3jyAXLeQx4KuroilTE2PWK7gnqExJg/vZr/OsVqSor6EcUBr+ltb VlyqvsWNaNnOy5MIys2PGH4yLGwg2jfGIEOoWJXrFgvSTNZAYScicQbBcuc1AWvuT811 UYVXEmL3mqZjMl4eqHr6fzH8BP0r+H8Z3IP224Wfd8tXz7zdGgTjZCPkzsoN+0tECQjx KXVTZhoswpHCR996kCQzYYvITdhxs4A6nWSzl0EtyN6wpMyNTj+3YDihLNnZ50Kfd56+ pkclzGJTm4g8xKBcucAwwNAL3cAor6xuoFeGPj1wceQqjM6wEl9nfmeww7/nmt7kUEYs 5LkQ== X-Gm-Message-State: AC+VfDw+bf9dKGRu8FEQ9m/v6Tc19iJCVFFeEKGaEJF78z+7xXKQgcEn /IIn9RxBqk4Z+k6HFqyfCiE31jMhjWs= X-Google-Smtp-Source: ACHHUZ6QOfrw0l3bdpvvlmXUNnamszuXXqEgiHjhlXZ1bthx0t+lBEyPW/rr7RnEuOk5SiS3J1yRldpoYHg= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:f582:c9e5:6c95:4461]) (user=yuzhao job=sendgmr) by 2002:a5b:784:0:b0:bad:99d:f087 with SMTP id b4-20020a5b0784000000b00bad099df087mr697256ybq.6.1686272382836; Thu, 08 Jun 2023 17:59:42 -0700 (PDT) Date: Thu, 8 Jun 2023 18:59:39 -0600 In-Reply-To: <20230526234435.662652-1-yuzhao@google.com> Message-Id: <20230609005940.42722-1-yuzhao@google.com> Mime-Version: 1.0 References: <20230526234435.662652-1-yuzhao@google.com> X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Subject: kvm/powerpc: memcached benchmark From: Yu Zhao To: Andrew Morton , Paolo Bonzini Cc: Alistair Popple , Anup Patel , Ben Gardon , Borislav Petkov , Catalin Marinas , Chao Peng , Christophe Leroy , Dave Hansen , Fabiano Rosas , Gaosheng Cui , Gavin Shan , "H. Peter Anvin" , Ingo Molnar , James Morse , "Jason A. Donenfeld" , Jason Gunthorpe , Jonathan Corbet , Marc Zyngier , Masami Hiramatsu , Michael Ellerman , Michael Larabel , Mike Rapoport , Nicholas Piggin , Oliver Upton , Paul Mackerras , Peter Xu , Sean Christopherson , Steven Rostedt , Suzuki K Poulose , Thomas Gleixner , Thomas Huth , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linuxppc-dev@lists.ozlabs.org, linux-trace-kernel@vger.kernel.org, x86@kernel.org, linux-mm@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230608_175944_534089_5F7F651D X-CRM114-Status: UNSURE ( 7.40 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org TLDR ==== Memcached achieved 10% more operations per second (in ~4 hours) after this patchset [1]. Hardware ======== HOST $ lscpu Architecture: ppc64le Byte Order: Little Endian CPU(s): 184 On-line CPU(s) list: 0-183 Model name: POWER9 (raw), altivec supported Model: 2.2 (pvr 004e 1202) Thread(s) per core: 4 Core(s) per socket: 23 Socket(s): 2 CPU max MHz: 3000.0000 CPU min MHz: 2300.0000 Caches (sum of all): L1d: 1.4 MiB (46 instances) L1i: 1.4 MiB (46 instances) L2: 12 MiB (24 instances) L3: 240 MiB (24 instances) NUMA: NUMA node(s): 2 NUMA node0 CPU(s): 0-91 NUMA node1 CPU(s): 92-183 Vulnerabilities: Itlb multihit: Not affected L1tf: Mitigation; RFI Flush, L1D private per thread Mds: Not affected Meltdown: Mitigation; RFI Flush, L1D private per thread Mmio stale data: Not affected Retbleed: Not affected Spec store bypass: Mitigation; Kernel entry/exit barrier (eieio) Spectre v1: Mitigation; __user pointer sanitization, ori31 speculation barrier enabled Spectre v2: Mitigation; Indirect branch serialisation (kernel only), Indirect branch cache disabled, Software link stack flush Srbds: Not affected Tsx async abort: Not affected HOST $ numactl -H available: 2 nodes (0-1) node 0 cpus: 0-91 node 0 size: 261659 MB node 0 free: 259152 MB node 1 cpus: 92-183 node 1 size: 261713 MB node 1 free: 261076 MB node distances: node 0 1 0: 10 40 1: 40 10 HOST $ cat /sys/class/nvme/nvme0/model INTEL SSDPF21Q800GB HOST $ cat /sys/class/nvme/nvme0/numa_node 0 Software ======== HOST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04 LTS" HOST $ uname -a Linux ppc 6.3.0 #1 SMP Sun Jun 4 18:26:37 UTC 2023 ppc64le ppc64le ppc64le GNU/Linux HOST $ cat /proc/swaps Filename Type Size Used Priority /dev/nvme0n1p2 partition 466838272 0 -2 HOST $ cat /sys/kernel/mm/lru_gen/enabled 0x0009 HOST $ cat /sys/kernel/mm/transparent_hugepage/enabled always madvise [never] HOST $ cat /sys/kernel/mm/transparent_hugepage/defrag always defer defer+madvise madvise [never] HOST $ qemu-system-ppc64 --version QEMU emulator version 6.2.0 (Debian 1:6.2+dfsg-2ubuntu6.6) Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers GUEST $ cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=22.04 DISTRIB_CODENAME=jammy DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS" GUEST $ cat /etc/memcached.conf ... -t 92 -m 262144 -B binary -s /var/run/memcached/memcached.sock -a 0766 GUEST $ memtier_benchmark -v memtier_benchmark 1.4.0 Copyright (C) 2011-2022 Redis Ltd. This is free software. You may redistribute copies of it under the terms of the GNU General Public License . There is NO WARRANTY, to the extent permitted by law. Procedure ========= HOST $ sudo numactl -N 0 -m 0 qemu-system-ppc64 \ -M pseries,accel=kvm,kvm-type=HV -cpu host -smp 92 -m 270g -nographic -nic user \ -drive if=virtio,format=raw,file=/dev/nvme0n1p1 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 1:0 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=P:P \ -n allkeys -d 2000 GUEST $ memtier_benchmark -S /var/run/memcached/memcached.sock \ -P memcache_binary -c 1 -t 92 --pipeline 1 --ratio 0:1 \ --key-minimum=1 --key-maximum=120000000 --key-pattern=R:R \ -n allkeys --randomize --distinct-client-seed Results ======= Before [1] After Change ------------------------------------------------- Ops/sec 721586.10 800210.12 +10% Avg. Latency 0.12546 0.11260 -10% p50 Latency 0.08700 0.08700 N/C p99 Latency 0.28700 0.24700 -13% Notes ===== [1] "mm: rmap: Don't flush TLB after checking PTE young for page reference" was included so that the comparison is apples to Apples. https://lore.kernel.org/r/20220706112041.3831-1-21cnbao@gmail.com/ _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel