From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BC2C9E7716D for ; Wed, 4 Dec 2024 08:53:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Type:MIME-Version: References:In-Reply-To:Subject:Cc:To:From:Message-ID:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=1XWjXY0FYz76bkGczTM1ejKTo2cEoKRp+bVoknS0rKU=; b=2P1ojwPA8Eu05kNXGec/tfauS0 QxrnbgTiWZD8jlHvDfuAKgM7EGs1JXaFrqjtzFZOusb4UrMjbuN6R80paSY9oLN54WgoqK3nXPyKF ZNptYYPs93Yp6kuols8DtQ6QEHMEkayEphwdMz3/hX08M8RSsV6vs1dbNsWEnVW95onYjhub0uGB1 MBoY6VOsYMwfbNCLllQFW+fZjuz4F3prk85u76vbbTQLyRVhRQSTjeuX+dU3363PoyDkDmXx7DPUi ocF/jW0vvg0b4NE/ODNy240YyndWIqt2+QYaCtvA9etn4vmGe/VreIMvoQCk3bIScLb7MbLXw1F0V s/FwwJEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tIl8V-0000000BxDk-1FU8; Wed, 04 Dec 2024 08:53:31 +0000 Received: from nyc.source.kernel.org ([147.75.193.91]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tIl7U-0000000Bx6k-2Ncp for linux-arm-kernel@lists.infradead.org; Wed, 04 Dec 2024 08:52:29 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by nyc.source.kernel.org (Postfix) with ESMTP id 718E3A41CE3; Wed, 4 Dec 2024 08:50:35 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2A23CC4CED1; Wed, 4 Dec 2024 08:52:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1733302347; bh=H+6ZPXSfRxIrWcaR3h/acU4Q4bj6nn/C2IsiJDBTtOM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=SHEjzf1v3SsYhBL6mDo2WpswRpfzOXuUmd3vRfNS9yA/Ou1vLb7d7iHUFDJ2RCxPx yE0spsz9YU58RsMIvywQ0BuqEfn//w5bPn3OiQXzwFBfPRod7hpb5rfjozLl3pnv6i LgujaVC+RYTvEYM3UhjLjxrIS/KxaElXRwFCQ9+Fuk5wa3R2hporNBTi4F7Qns3zvW LEEX7lqTUV5WcOSvqzaQQIGpkFHoViDjZqkMulxPHi8V9QfnspPdNn+HJq+QackTt5 1rbZhNl+u4R05tYvgF/cpanYDj4fmco/2yjOuuTFtiR/pXC5wfN/eXiQNwcBjOl2al CfdJZup2LyxfQ== Received: from 82-132-237-135.dab.02.net ([82.132.237.135] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1tIl6r-000NUO-Jr; Wed, 04 Dec 2024 08:52:24 +0000 Date: Wed, 04 Dec 2024 08:51:26 +0000 Message-ID: <87jzcfsuep.wl-maz@kernel.org> From: Marc Zyngier To: Vitaly Chikunov Cc: Shameerali Kolothum Thodi , Will Deacon , "james.morse@arm.com" , "linux-arm-kernel@lists.infradead.org" , Catalin Marinas , "linux-kernel@vger.kernel.org" , "oliver.upton@linux.dev" , "mark.rutland@arm.com" , "Wangzhou (B)" , "Dmitry V. Levin" Subject: Re: v6.13-rc1: Internal error: Oops - Undefined instruction: 0000000002000000 [#1] SMP In-Reply-To: <20241203221453.mwh6sozyczi4ec2k@altlinux.org> References: <20241202045830.e4yy3nkvxtzaybxk@altlinux.org> <20241202153618.GA6834@willie-the-truck> <86ttbmt71k.wl-maz@kernel.org> <20241202155940.p267a3tz5ypj4sog@altlinux.org> <86ser6t6fs.wl-maz@kernel.org> <20241202223119.k3uod4ksnlf7gqh2@altlinux.org> <20241203092721.j473dthkbq6wzez7@altlinux.org> <1847e34fa7724d28aeb22d93752f64f2@huawei.com> <20241203221453.mwh6sozyczi4ec2k@altlinux.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/29.4 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 82.132.237.135 X-SA-Exim-Rcpt-To: vt@altlinux.org, shameerali.kolothum.thodi@huawei.com, will@kernel.org, james.morse@arm.com, linux-arm-kernel@lists.infradead.org, catalin.marinas@arm.com, linux-kernel@vger.kernel.org, oliver.upton@linux.dev, mark.rutland@arm.com, wangzhou1@hisilicon.com, ldv@altlinux.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241204_005228_746130_E8F8B2D5 X-CRM114-Status: GOOD ( 55.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 03 Dec 2024 22:14:53 +0000, Vitaly Chikunov wrote: > > Shameer, Marc, Oliver, Will, > > On Tue, Dec 03, 2024 at 10:03:11AM +0000, Shameerali Kolothum Thodi wrote: > > > -----Original Message----- > > > From: linux-arm-kernel On > > > Behalf Of Vitaly Chikunov > > > Sent: Tuesday, December 3, 2024 9:27 AM > > > To: Marc Zyngier > > > Cc: Will Deacon ; james.morse@arm.com; linux-arm- > > > kernel@lists.infradead.org; Catalin Marinas ; > > > linux-kernel@vger.kernel.org; oliver.upton@linux.dev; > > > mark.rutland@arm.com > > > Subject: Re: v6.13-rc1: Internal error: Oops - Undefined instruction: > > > 0000000002000000 [#1] SMP > > > > > > Marc, > > > > > > On Tue, Dec 03, 2024 at 01:31:19AM +0300, Vitaly Chikunov wrote: > > > > On Mon, Dec 02, 2024 at 04:07:03PM +0000, Marc Zyngier wrote: > > > > > On Mon, 02 Dec 2024 15:59:40 +0000, > > > > > Vitaly Chikunov wrote: > > > > > > > > > > > > Marc, > > > > > > > > > > > > On Mon, Dec 02, 2024 at 03:53:59PM +0000, Marc Zyngier wrote: > > > > > > > > > > > > > > What the log doesn't say is what the host is. Is it 6.13-rc1 as well? > > > > > > > > > > > > No, host is 6.6.60. > > > > > > > > > > Right. I wouldn't be surprised if: > > > > > > > > > > - this v6.6 kernel doesn't hide the MPAM feature as it should (and > > > > > that's proably something we should backport) > > > > > > > > How to confirm this? Currently I cannot find any (case-insensitive) > > > > "MPAM" files in /sys, nor mpam string in /proc/cpuinfo, nor MPAM > > > > strings in `strace -v` (as it decodes some KVM ioctls) of qemu process. > > > > > > > > > > > > > > - you get a nastygram in the host log telling you that the guest has > > > > > executed something it shouldn't (you'll get the encoding of the > > > > > instruction) > > > > > > > > I requested admins of the box for dmesg output since I don't have root > > > > access myself and nowadays dmesg is not accessible for a user. > > > > > > This is what they reported: > > > > > > kvm [2502822]: Unsupported guest sys_reg access at: ffff80008003e9f0 > > > [000000c5] > > > { Op0( 3), Op1( 0), CRn(10), CRm( 4), Op2( 4), func_read }, > > > > > > > As Will pointed out I think this is access to MPAMIDR_EL1 and is from this > > code here, > > > > +++ b/arch/arm64/kernel/cpuinfo.c > > @@ -478,6 +478,9 @@ static void __cpuinfo_store_cpu(struct cpuinfo_arm64 *info) > > if (id_aa64pfr0_32bit_el0(info->reg_id_aa64pfr0)) > > __cpuinfo_store_cpu_32bit(&info->aarch32); > > > > + if (id_aa64pfr0_mpam(info->reg_id_aa64pfr0)) > > + info->reg_mpamidr = read_cpuid(MPAMIDR_EL1); > > + > > cpuinfo_detect_icache_policy(info); > > } > > > > I did manage to boot my setup in 6.6 and this is what happens, > > > > Host kernel 6.6 > > Guest Kernel 6.13-rc1 > > > > [ 0.195392] smp: Brought up 1 node, 8 CPUs > > [ 0.219000] SMP: Total of 8 processors activated. > > [ 0.219629] CPU: All CPU(s) started at EL1 > > ... > > [ 0.223212] CPU features: detected: RAS Extension Support > > [ 0.223927] CPU features: detected: Memory Partitioning And Monitoring > > [ 0.224796] CPU features: detected: Memory Partitioning And Monitoring Virtualisation > > [ 0.225961] alternatives: applying system-wide alternatives > > ... > > > > Guest detects MPAM and boots fine. > > > > Host kernel 6.13-rc1 > > Guest Kernel 6.13-rc1 > > > > [ 0.196625] smp: Brought up 1 node, 8 CPUs > > [ 0.222093] SMP: Total of 8 processors activated. > > [ 0.222769] CPU: All CPU(s) started at EL1 > > ... > > [ 0.226620] CPU features: detected: RAS Extension Support > > [ 0.227453] alternatives: applying system-wide alternatives > > > > MPAM is not visible to Guest in this case. > > > > So as I pointed out earlier could it be a case where the ID register reports MPAM support > > but the firmware has not enabled MPAM? > > > > James seems to be mentioning that case here, > > > > " (If you have a boot failure that bisects here its likely your CPUs > > advertise MPAM in the id registers, but firmware failed to either enable > > or MPAM, or emulate the trap as if it were disabled)" > > I tried to verify that MPAM is advertised with qemu+gdb method, as > suggested by Oliver, but ID_AA64PFR0_EL1 register is not there. > > (gdb) i r ID_AA64PFR0_EL1 > Invalid register `ID_AA64PFR0_EL1' Then there is a bug in either QEMU or the GDB stubs. This register exists, or you wouldn't be here. > > Are there other suggestions? Mark has described what the problem is likely to be. 6.6-stable needs to have 6685f5d572c22e10 backported, and it probably should have been Cc: to stable. Can you please apply the following patch to your *host* machine and retest? diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 370a1a7bd369..258a39bcd3c7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1330,6 +1330,7 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu, val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_MTE); val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_SME); + val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_MPAM_frac); break; case SYS_ID_AA64ISAR1_EL1: if (!vcpu_has_ptrauth(vcpu)) @@ -1472,6 +1473,13 @@ static u64 read_sanitised_id_aa64pfr0_el1(struct kvm_vcpu *vcpu, val &= ~ID_AA64PFR0_EL1_AMU_MASK; + /* + * MPAM is disabled by default as KVM also needs a set of PARTID to + * program the MPAMVPMx_EL2 PARTID remapping registers with. But some + * older kernels let the guest see the ID bit. + */ + val &= ~ID_AA64PFR0_EL1_MPAM_MASK; + return val; } @@ -1560,6 +1568,29 @@ static int set_id_dfr0_el1(struct kvm_vcpu *vcpu, return set_id_reg(vcpu, rd, val); } +static int set_id_aa64pfr0_el1(struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd, u64 user_val) +{ + u64 hw_val = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1); + u64 mpam_mask = ID_AA64PFR0_EL1_MPAM_MASK; + + /* + * Commit 011e5f5bf529f ("arm64/cpufeature: Add remaining feature bits + * in ID_AA64PFR0 register") exposed the MPAM field of AA64PFR0_EL1 to + * guests, but didn't add trap handling. KVM doesn't support MPAM and + * always returns an UNDEF for these registers. The guest must see 0 + * for this field. + * + * But KVM must also accept values from user-space that were provided + * by KVM. On CPUs that support MPAM, permit user-space to write + * the sanitizied value to ID_AA64PFR0_EL1.MPAM, but ignore this field. + */ + if ((hw_val & mpam_mask) == (user_val & mpam_mask)) + user_val &= ~ID_AA64PFR0_EL1_MPAM_MASK; + + return set_id_reg(vcpu, rd, user_val); +} + /* * cpufeature ID register user accessors * @@ -2018,7 +2049,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ID_AA64PFR0_EL1), .access = access_id_reg, .get_user = get_id_reg, - .set_user = set_id_reg, + .set_user = set_id_aa64pfr0_el1, .reset = read_sanitised_id_aa64pfr0_el1, .val = ID_AA64PFR0_EL1_CSV2_MASK | ID_AA64PFR0_EL1_CSV3_MASK, }, ID_SANITISED(ID_AA64PFR1_EL1), > > https://lore.kernel.org/all/20241030160317.2528209-4-joey.gouly@arm.com/ > > > > Is there a way you can find out the BIOS version on that board? > > Unfortunately, admins of the server do not provide me with this > info. This doesn't really help, I'm afraid. > For such cases, when MPAM is incorrectly advertised, can we have kernel > command line parameter like mpam=0 to override it's detection? We could, but only when we can confirm what the problem is. > I think with "If you have a boot failure that bisects here" it's > acknowledged possibility and it's confirmed by our server. Not really. This talks about firmware. We are debugging the hypervisor here. This might be closely related, but these are not the same things. Thanks, M. -- Without deviation from the norm, progress is not possible.