From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from fhigh2-smtp.messagingengine.com (fhigh2-smtp.messagingengine.com [103.168.172.153]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7883418A952 for ; Thu, 8 Aug 2024 10:31:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=103.168.172.153 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723113095; cv=none; b=nCAGNDSRjLJRnzIRrz+4BMYdcqbLEtFiJIab1eagp08rj7wtb8hQKfy3vN9QjPY0RYK+bJSiPdqAP1I9gRxJcnH4M+BjDM4Y1ZLPCCI898wOwKIg2JxJ3T0oVNnRL4u8fziy0onSWEoAlkWSU78cB5J02+abtKwxGczOcSxoijE= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1723113095; c=relaxed/simple; bh=bT9AI8B5Rt/jTvoUWHcjOzWMOy/YdVV4pdOpyGVjc5o=; h=Date:From:To:Cc:Subject:Message-ID:MIME-Version:Content-Type: Content-Disposition; b=YY4tF4w/Q6qtnBNH6hsuvCIC0gzJajjJR0aR2EqkI7xQGeBcURoI/bB9bnxpYIK/Yp2HPUpLV0ILlFlHBtQKA0swrDxH37qYtuPos54Hnz6SNk+o85/WXnmHlmt/L05m0gZIW8XRssajzWdtTibs7wLK3GT2r8W6DbfwYzT6u6c= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com; spf=none smtp.mailfrom=invisiblethingslab.com; dkim=pass (2048-bit key) header.d=invisiblethingslab.com header.i=@invisiblethingslab.com header.b=Sfzg6hgX; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b=k+p26n/Y; arc=none smtp.client-ip=103.168.172.153 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=invisiblethingslab.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=invisiblethingslab.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=invisiblethingslab.com header.i=@invisiblethingslab.com header.b="Sfzg6hgX"; dkim=pass (2048-bit key) header.d=messagingengine.com header.i=@messagingengine.com header.b="k+p26n/Y" Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.nyi.internal (Postfix) with ESMTP id A269C1146C2A; Thu, 8 Aug 2024 06:31:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute6.internal (MEProxy); Thu, 08 Aug 2024 06:31:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= invisiblethingslab.com; h=cc:cc:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :subject:subject:to:to; s=fm3; t=1723113092; x=1723199492; bh=MX 44/EWhOUxV2E2PRqPcKlXwkRrpynBXksdVpdffh/o=; b=Sfzg6hgXJ1wKHB9P/L +Fel/YzCpIfhYgYm/2ItWvfwPJRzbHti4ZIUPmVBCvaZYLlnjPW78aHFUpWsZHbT AQFOU/RWjxM6F6StxaKYPoC1+4vrR0qLmazfcnUbwHfjwN3RwJEM5bEgs5ZnNm0H xTJFKOoSAFqgkGVH5ovD73lTbj5iD9eL4Ic0rzxqBzRK5VdKcoSn6rliFrmiELtr AKBpksKgFzOvPgG+q6TpguvENTi0KzaZN4dGXsTVxAgduGWkBUy2GiftfpIUAaRZ jhfAxdGXdFukphepXXQegqdObXN2T2/bIaDGSLw8tyRGwPzO3SC32ECMMAO9yqBk QzMg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:message-id :mime-version:reply-to:subject:subject:to:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t= 1723113092; x=1723199492; bh=MX44/EWhOUxV2E2PRqPcKlXwkRrpynBXksd Vpdffh/o=; b=k+p26n/Y3/RUqrhpgNcikgQf5f/0GmeD7IPqTmvi+EofKyVm734 TjADUpCWWvFZ4nnMnivCcpEeT0oo2gaG4HZjhUrU1v1aXSh8LJAV6Sw/OPzUd/08 2D9isZCEol7pojvle//jXE8IiuHe2WfwSbfpnpgDepAzvyuWlegADrCXODsK6nsy pELW6o4IxH/nlkux0dirrP9xlfY7N+3Nul0CCHZCfObUEALteAfMORVf4vSn9ZLN dgjLWzzsy4RZIiklUQcsw+fU4oUtAqDLcB1/Pot6UEXYJN4HPYjqC04bLsWQ4Xeo CIkzNZm5MyjDY+Oao8IOZ+dTG6LZgtiAlQg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeeftddrledvgddvlecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpggftfghnshhusghstghrihgsvgdpuffr tefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnth hsucdlqddutddtmdenucfjughrpeffhffvvefukfggtggusehgtderredttdejnecuhfhr ohhmpeforghrvghkucforghrtgiihihkohifshhkihdqifpkrhgvtghkihcuoehmrghrmh grrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmqeenucggtffrrght thgvrhhnpeehgfeuvdejkeeikeeludelteevfeegffegteefffetheetffduudejuedugf dvteenucffohhmrghinhepqhhusggvshdqohhsrdhorhhgpdhgihhthhhusgdrtghomhen ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrghrmh grrhgvkhesihhnvhhishhisghlvghthhhinhhgshhlrggsrdgtohhmpdhnsggprhgtphht thhopeehpdhmohguvgepshhmthhpohhuthdprhgtphhtthhopeigvghnqdguvghvvghlse hlihhsthhsrdigvghnphhrohhjvggtthdrohhrghdprhgtphhtthhopehjghhrohhsshes shhushgvrdgtohhmpdhrtghpthhtohepshhsthgrsggvlhhlihhniheskhgvrhhnvghlrd horhhgpdhrtghpthhtohepohhlvghkshgrnhgurhgpthihshhhtghhvghnkhhosegvphgr mhdrtghomhdprhgtphhtthhopehrvghgrhgvshhsihhonhhssehlihhsthhsrdhlihhnuh igrdguvghv X-ME-Proxy: Feedback-ID: i1568416f:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Thu, 8 Aug 2024 06:31:31 -0400 (EDT) Date: Thu, 8 Aug 2024 12:31:13 +0200 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: xen-devel Cc: Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , regressions@lists.linux.dev Subject: [REGRESSION] kernel NULL pointer dereference in xen-balloon with mem hotplug Message-ID: Precedence: bulk X-Mailing-List: regressions@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="kt7u1tiUe8A/rt3t" Content-Disposition: inline --kt7u1tiUe8A/rt3t Content-Type: text/plain; protected-headers=v1; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Date: Thu, 8 Aug 2024 12:31:13 +0200 From: Marek =?utf-8?Q?Marczykowski-G=C3=B3recki?= To: xen-devel Cc: Juergen Gross , Stefano Stabellini , Oleksandr Tyshchenko , regressions@lists.linux.dev Subject: [REGRESSION] kernel NULL pointer dereference in xen-balloon with mem hotplug Hi, When testing Linux 6.11-rc2, I've got the crash like below. It's a PVH guest started with 400MB memory, and then extended via mem hotplug (I don't know to what exact size it was at this time, but up to 4GB), it was quite early in the domU boot process, I suspect it could be the first mem hotplug even happening there. Unfortunately I don't have reliable reproducer, it crashed only once over several test runs. I don't remember seeing such crash before, so it looks like a regression in 6.11. I'm not sure if that matters, but it's on ADL, very similar to the qubes-hw2 gitlab runner. The crash message: [ 3.606538] BUG: kernel NULL pointer dereference, address: 000000000= 0000000 [ 3.606556] #PF: supervisor read access in kernel mode [ 3.606568] #PF: error_code(0x0000) - not-present page [ 3.606580] PGD 0 P4D 0=20 [ 3.606590] Oops: Oops: 0000 [#1] PREEMPT SMP NOPTI [ 3.606603] CPU: 1 UID: 0 PID: 45 Comm: xen-balloon Not tainted 6.11= =2E0-0.rc2.1.qubes.1.fc37.x86_64 #1 [ 3.606623] RIP: 0010:phys_pmd_init+0x96/0x500 [ 3.606636] Code: 89 ed 48 c1 e8 12 48 81 e7 00 00 e0 ff 25 f8 0f 00= 00 4c 8d af 00 00 20 00 4c 8d 24 03 48 8b 1c 24 4c 39 fd 0f 83 89 02 00 00= <49> 8b 0c 24 48 f7 c1 9f ff ff ff 0f 84 b6 01 00 00 48 8b 05 d2 99 [ 3.606680] RSP: 0018:ffffc90000987b90 EFLAGS: 00010287 [ 3.606695] RAX: 0000000000000000 RBX: 8000000000000163 RCX: 0000000= 000000004 [ 3.606713] RDX: 0000000090000000 RSI: 0000000080000000 RDI: 0000000= 080000000 [ 3.606729] RBP: 0000000080000000 R08: 8000000000000163 R09: 0000000= 000000001 [ 3.606748] R10: 0000000000000000 R11: 0000000000ffff0a R12: 0000000= 000000000 [ 3.606766] R13: 0000000080200000 R14: 0000000000000000 R15: 0000000= 090000000 [ 3.606784] FS: 0000000000000000(0000) GS:ffff888018500000(0000) kn= lGS:0000000000000000 [ 3.606802] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.606819] CR2: 0000000000000000 CR3: 00000000107bc000 CR4: 0000000= 000750ef0 [ 3.606840] PKRU: 55555554 [ 3.606847] Call Trace: [ 3.606854] [ 3.606862] ? __die+0x23/0x70 [ 3.606876] ? page_fault_oops+0x95/0x190 [ 3.606887] ? exc_page_fault+0x76/0x190 [ 3.606900] ? asm_exc_page_fault+0x26/0x30 [ 3.606917] ? phys_pmd_init+0x96/0x500 [ 3.606927] phys_pud_init+0xe8/0x4f0 [ 3.606940] __kernel_physical_mapping_init+0x1d5/0x380 [ 3.606955] ? synchronize_rcu_normal.part.0+0x45/0x70 [ 3.606971] init_memory_mapping+0xb0/0x1f0 [ 3.606983] arch_add_memory+0x2f/0x50 [ 3.606997] add_memory_resource+0xff/0x2c0 [ 3.607008] reserve_additional_memory+0x162/0x1d0 [ 3.607026] balloon_thread+0xe4/0x490 [ 3.607041] ? __pfx_autoremove_wake_function+0x10/0x10 [ 3.607060] ? __pfx_balloon_thread+0x10/0x10 [ 3.607076] kthread+0xcf/0x100 [ 3.607090] ? __pfx_kthread+0x10/0x10 [ 3.607101] ret_from_fork+0x31/0x50 [ 3.607112] ? __pfx_kthread+0x10/0x10 [ 3.607123] ret_from_fork_asm+0x1a/0x30 [ 3.607135] [ 3.607141] Modules linked in: xenfs binfmt_misc nft_reject_inet nf_= reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_conntrack nf_defrag_ipv6 nf= _defrag_ipv4 nf_tables nfnetlink intel_rapl_msr intel_rapl_common intel_unc= ore_frequency_common crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clm= ulni polyval_generic ghash_clmulni_intel sha512_ssse3 sha256_ssse3 sha1_sss= e3 xen_netfront xen_privcmd xen_gntdev xen_gntalloc xen_blkback xen_evtchn = loop fuse ip_tables overlay xen_blkfront [ 3.607266] CR2: 0000000000000000 [ 3.607277] ---[ end trace 0000000000000000 ]--- [ 3.607291] RIP: 0010:phys_pmd_init+0x96/0x500 [ 3.607307] Code: 89 ed 48 c1 e8 12 48 81 e7 00 00 e0 ff 25 f8 0f 00= 00 4c 8d af 00 00 20 00 4c 8d 24 03 48 8b 1c 24 4c 39 fd 0f 83 89 02 00 00= <49> 8b 0c 24 48 f7 c1 9f ff ff ff 0f 84 b6 01 00 00 48 8b 05 d2 99 [ 3.607356] RSP: 0018:ffffc90000987b90 EFLAGS: 00010287 [ 3.607371] RAX: 0000000000000000 RBX: 8000000000000163 RCX: 0000000= 000000004 [ 3.607389] RDX: 0000000090000000 RSI: 0000000080000000 RDI: 0000000= 080000000 [ 3.607406] RBP: 0000000080000000 R08: 8000000000000163 R09: 0000000= 000000001 [ 3.607428] R10: 0000000000000000 R11: 0000000000ffff0a R12: 0000000= 000000000 [ 3.607449] R13: 0000000080200000 R14: 0000000000000000 R15: 0000000= 090000000 [ 3.607469] FS: 0000000000000000(0000) GS:ffff888018500000(0000) kn= lGS:0000000000000000 [ 3.607488] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3.607504] CR2: 0000000000000000 CR3: 00000000107bc000 CR4: 0000000= 000750ef0 [ 3.607525] PKRU: 55555554 [ 3.607533] Kernel panic - not syncing: Fatal exception [ 3.607599] Kernel Offset: disabled Full domU log: https://openqa.qubes-os.org/tests/108883/file/system_tests-qubes.tests.inte= g.vm_qrexec_gui.TC_20_NonAudio_whonix-workstation-17.test_105.guest-test-in= st-vm2.log Other logs, including dom0 and Xen messages: https://openqa.qubes-os.org/tests/108883#downloads Kernel config is build from merging https://github.com/QubesOS/qubes-linux-kernel/blob/005ae1ac3819d957379e48fb= 2cfd33f511a47275/config-base with https://github.com/QubesOS/qubes-linux-kernel/blob/005ae1ac3819d957379e48fb= 2cfd33f511a47275/config-qubes (options set in the latter takes precedence) Especially, it has: CONFIG_XEN_BALLOON_MEMORY_HOTPLUG=3Dy CONFIG_XEN_UNPOPULATED_ALLOC=3Dy #regzbot introduced: v6.10..v6.11-rc2 --=20 Best Regards, Marek Marczykowski-G=C3=B3recki Invisible Things Lab --kt7u1tiUe8A/rt3t Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEhrpukzGPukRmQqkK24/THMrX1ywFAma0nnIACgkQ24/THMrX 1yyw4wf/VHGBECVm870HmY0iwiXw0Wjf7qkIVuUbdK4O/z3R5o8tcv1MnkMM8wQG otXtOP6C3QN5GP0iILNlObzmsrtFpNXFDoDi8fWZkHJ2nOHo3tr9iGaOzlB+L2KM XpAT5n9JFSHAz8xY/DbU81m2CGGMvvRXTq9o6d/B5wO9lYbqOSN5yjPZHCifsHb9 Vsnf3c0hzk2qBCzb5wq806kOit37+8RHjYy0vPQVjPwaZmTwT6NRxvunDmTjCXz2 H2F93zXnRaC/6KGICTFJCjOYOYyLGhXUbYO+EOuVys6XDHQFO1YXa5Ob/1ylFVFg CI79rS8pTHkv9Il8KfiwnNc8ovwEAQ== =ZqgP -----END PGP SIGNATURE----- --kt7u1tiUe8A/rt3t--