From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1FA6C49361 for ; Thu, 17 Jun 2021 01:08:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5B98C613DB for ; Thu, 17 Jun 2021 01:08:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B98C613DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D28D96B0070; Wed, 16 Jun 2021 21:08:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CD8C66B0071; Wed, 16 Jun 2021 21:08:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADD256B0072; Wed, 16 Jun 2021 21:08:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 73EAD6B0070 for ; Wed, 16 Jun 2021 21:08:17 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 05DC2FB7A for ; Thu, 17 Jun 2021 01:08:17 +0000 (UTC) X-FDA: 78261429834.11.06740B6 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf24.hostedemail.com (Postfix) with ESMTP id EB104A0001A9 for ; Thu, 17 Jun 2021 01:08:05 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id EA738613CA; Thu, 17 Jun 2021 01:08:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623892095; bh=RQPUupKWZ39Ts6YqMKJZ4NPWboTMlQ8j5iTcoZkvN1k=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=Uzj+Y5/0oeum/sHwVpH3FrC9rXrZG+fQBJaFLaQe2AOqs6Y5Y5jw9mxqwhIBR1lom kD13ydZHMuEsO2EF2BxS4VXNnf2huKbhzP6lqozAuO3h9jJdnYeNFmosBrKo65cbSx Z+YptmJn/u0tIv1ZH2QguJXNmRZ7N9zOc8vxPzj+kqr8JtNtjH520Dbb/eXcwPR7qf rsgbQwnj08bFdvEfk1/zj2dD2X+1I8aEh9pYhd1WNrnV3JK+5bLaPxJe73jxsaVxCi oGEs2CnGYClYHt+Ozd09LnTcX9hgu3QeVjl56DRNhr4+bVvKl8YZ/fzF6zRSYvtmzT F6/eLuSZZx/RA== Subject: Re: vmemmap alloc failure in hot_add_req() To: David Hildenbrand Cc: "K. Y. Srinivasan" , Haiyang Zhang , Stephen Hemminger , Wei Liu , Dexuan Cui , linux-hyperv@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Oscar Salvador , Hillf Danton References: <20210612021115.2136-1-hdanton@sina.com> <951ddbaf-3d74-7043-4866-3809ff991cfd@redhat.com> From: Nathan Chancellor Message-ID: Date: Wed, 16 Jun 2021 18:08:15 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 MIME-Version: 1.0 In-Reply-To: <951ddbaf-3d74-7043-4866-3809ff991cfd@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="Uzj+Y5/0"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf24.hostedemail.com: domain of nathan@kernel.org designates 198.145.29.99 as permitted sender) smtp.mailfrom=nathan@kernel.org X-Rspamd-Server: rspam02 X-Stat-Signature: 6rz7kff7ny6q5ixq7dijunwcn1y5rkob X-Rspamd-Queue-Id: EB104A0001A9 X-HE-Tag: 1623892085-925464 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi David, On 6/14/2021 12:38 AM, David Hildenbrand wrote: > On 12.06.21 04:11, Hillf Danton wrote: >> On Fri, 11 Jun 2021 12:48:26 -0700 Nathan Chancellor wrote: >>> Hi all, >>> >>> I am occasionally seeing a kernel warning when running virtual machin= es >>> in Hyper-V, which usually happens a minute or so after boot. It does = not >>> happen on every boot and it is reproducible on at least v5.10. I thin= k >>> it might have something to do with constant reboots, which I do when >>> testing various kernels. >>> >>> The stack trace is as follows: >>> >>> [=C2=A0=C2=A0 49.215291] kworker/0:1: vmemmap alloc failure: order:9,= =20 >>> mode:0x4cc0(GFP_KERNEL|__GFP_RETRY_MAYFAIL),=20 >>> nodemask=3D(null),cpuset=3D/,mems_allowed=3D0 >>> [=C2=A0=C2=A0 49.215299] CPU: 0 PID: 18 Comm: kworker/0:1 Not tainted= =20 >>> 5.13.0-rc5 #1 >>> [=C2=A0=C2=A0 49.215301] Hardware name: Microsoft Corporation Virtual= =20 >>> Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.0 11/01/2019 >>> [=C2=A0=C2=A0 49.215302] Workqueue: events hot_add_req [hv_balloon] >> >> Apart from order:9 (mm Cced), events_unbound is the right workqueue=20 >> instead >> because the report shows the risk that hot_add_req could block other=20 >> pending >> events longer than thought. Any special reason for the events wq? >> >>> [=C2=A0=C2=A0 49.215307] Call Trace: >>> [=C2=A0=C2=A0 49.215310]=C2=A0 dump_stack+0x76/0x94 >>> [=C2=A0=C2=A0 49.215314]=C2=A0 warn_alloc.cold+0x78/0xdc >>> [=C2=A0=C2=A0 49.215316]=C2=A0 ? __alloc_pages+0x200/0x230 >>> [=C2=A0=C2=A0 49.215319]=C2=A0 vmemmap_alloc_block+0x86/0xdc >>> [=C2=A0=C2=A0 49.215323]=C2=A0 vmemmap_populate+0x10e/0x31c >>> [=C2=A0=C2=A0 49.215324]=C2=A0 __populate_section_memmap+0x38/0x4e >>> [=C2=A0=C2=A0 49.215326]=C2=A0 sparse_add_section+0x12c/0x1cf >>> [=C2=A0=C2=A0 49.215329]=C2=A0 __add_pages+0xa9/0x130 >>> [=C2=A0=C2=A0 49.215330]=C2=A0 add_pages+0x12/0x60 >>> [=C2=A0=C2=A0 49.215333]=C2=A0 add_memory_resource+0x180/0x300 >>> [=C2=A0=C2=A0 49.215335]=C2=A0 __add_memory+0x3b/0x80 >>> [=C2=A0=C2=A0 49.215336]=C2=A0 add_memory+0x2e/0x50 >>> [=C2=A0=C2=A0 49.215337]=C2=A0 hot_add_req+0x3fc/0x5a0 [hv_balloon] >>> [=C2=A0=C2=A0 49.215340]=C2=A0 process_one_work+0x214/0x3e0 >>> [=C2=A0=C2=A0 49.215342]=C2=A0 worker_thread+0x4d/0x3d0 >>> [=C2=A0=C2=A0 49.215344]=C2=A0 ? process_one_work+0x3e0/0x3e0 >>> [=C2=A0=C2=A0 49.215345]=C2=A0 kthread+0x133/0x150 >>> [=C2=A0=C2=A0 49.215347]=C2=A0 ? kthread_associate_blkcg+0xc0/0xc0 >>> [=C2=A0=C2=A0 49.215348]=C2=A0 ret_from_fork+0x22/0x30 >>> [=C2=A0=C2=A0 49.215351] Mem-Info: >>> [=C2=A0=C2=A0 49.215352] active_anon:251 inactive_anon:140868 isolate= d_anon:0 >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 active_file:47497 inactive_file:88505 isolate= d_file:0 >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 unevictable:8 dirty:14 writeback:0 >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 slab_reclaimable:12013 slab_unreclaimable:114= 03 >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 mapped:131701 shmem:12671 pagetables:3140 bou= nce:0 >>> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 free:41388 free_pcp:37 free_cma:0 >>> [=C2=A0=C2=A0 49.215355] Node 0 active_anon:1004kB inactive_anon:5634= 72kB=20 >>> active_file:189988kB inactive_file:354020kB unevictable:32kB=20 >>> isolated(anon):0kB isolated(file):0kB mapped:526804kB dirty:56kB=20 >>> writeback:0kB shmem:50684kB shmem_thp: 0kB shmem_pmdmapped: 0kB=20 >>> anon_thp: 0kB writeback_tmp:0kB kernel_stack:5904kB=20 >>> pagetables:12560kB all_unreclaimable? no >>> [=C2=A0=C2=A0 49.215358] Node 0 DMA free:6496kB min:480kB low:600kB h= igh:720kB=20 >>> reserved_highatomic:0KB active_anon:0kB inactive_anon:3120kB=20 >>> active_file:2584kB inactive_file:2792kB unevictable:0kB=20 >>> writepending:0kB present:15996kB managed:15360kB mlocked:0kB=20 >>> bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB >>> [=C2=A0=C2=A0 49.215361] lowmem_reserve[]: 0 1384 1384 1384 1384 >>> [=C2=A0=C2=A0 49.215364] Node 0 DMA32 free:159056kB min:44572kB low:5= 5712kB=20 >>> high:66852kB reserved_highatomic:0KB active_anon:1004kB=20 >>> inactive_anon:560352kB active_file:187004kB inactive_file:350864kB=20 >>> unevictable:32kB writepending:56kB present:1555760kB=20 >>> managed:1432388kB mlocked:32kB bounce:0kB free_pcp:172kB=20 >>> local_pcp:0kB free_cma:0kB >>> [=C2=A0=C2=A0 49.215367] lowmem_reserve[]: 0 0 0 0 0 >>> [=C2=A0=C2=A0 49.215369] Node 0 DMA: 17*4kB (UM) 13*8kB (M) 10*16kB (= M) 3*32kB=20 >>> (ME) 3*64kB (UME) 4*128kB (UME) 1*256kB (E) 2*512kB (UE) 2*1024kB=20 >>> (ME) 1*2048kB (E) 0*4096kB =3D 6508kB >>> [=C2=A0=C2=A0 49.215377] Node 0 DMA32: 8061*4kB (UME) 5892*8kB (UME) = 2449*16kB=20 >>> (UME) 604*32kB (UME) 207*64kB (UME) 49*128kB (UM) 7*256kB (M) 1*512kB= =20 >>> (M) 0*1024kB 0*2048kB 0*4096kB =3D 159716kB >>> [=C2=A0=C2=A0 49.215388] 148696 total pagecache pages >>> [=C2=A0=C2=A0 49.215388] 0 pages in swap cache >>> [=C2=A0=C2=A0 49.215389] Swap cache stats: add 0, delete 0, find 0/0 >>> [=C2=A0=C2=A0 49.215390] Free swap=C2=A0 =3D 0kB >>> [=C2=A0=C2=A0 49.215390] Total swap =3D 0kB >>> [=C2=A0=C2=A0 49.215391] 392939 pages RAM >>> [=C2=A0=C2=A0 49.215391] 0 pages HighMem/MovableOnly >>> [=C2=A0=C2=A0 49.215391] 31002 pages reserved >>> [=C2=A0=C2=A0 49.215392] 0 pages cma reserved >>> [=C2=A0=C2=A0 49.215393] 0 pages hwpoisoned >>> >>> Is this a known issue and/or am I doing something wrong? I only notic= ed >>> this because there are times when I am compiling something intensive = in >>> the VM such as LLVM and the VM runs out of memory even though I have >>> plenty of free memory on the host but I am not sure if this warning i= s >>> related to that issue. >=20 > Hi, >=20 > Is hotplugged memory getting onlined automatically (either from user=20 > space via a udev script or via the kernel, for example, with=20 > CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE)? It does look like this kernel configuration has=20 CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=3Dy. > If it's not getting onlined, you easily sport after hotplug e.g., via=20 > "lsmem" that there are quite some offline memory blocks. >=20 > Note that x86_64 code will fallback from populating huge pages to=20 > populating base pages for the vmemmap; this can happen easily when unde= r=20 > memory pressure. Not sure if it is relevant or not but this warning can show up within a=20 minute of startup without me doing anything in particular. > If adding memory would fail completely, you'd see another "hot_add=20 > memory failed error is ..." error message from hyper-v in the kernel=20 > log. If that doesn't show up, it's simply suboptimal, but hotplugging=20 > memory still succeeded. I did notice that from the code in hv_balloon.c but I do not think I=20 have ever seen that message in my logs. > Note: we could support "memmap_on_memory" in some cases (e.g., no memor= y=20 > holes in hotadded range) when hotplugging memory blocks via hyper-v,=20 > which would result in this warning less trigger less frequently. Cheers, Nathan