From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 55CF83563EB for ; Wed, 8 Jul 2026 03:11:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.214.202 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480298; cv=none; b=OT+bndCDcVRY1XAk/+rIhnhE88a/g/scFuN6cbLgIj/WP1mrkk48F6QPZeHrwumBlmp3MptUZtAtELzRz7YRs4egGi/1QSipTNB2dPSgU82+9kUtFYf5r+lGqprdT2siejfO6eklYGCLIVCjKaf+anlPxBkTzxCfuTRs0DvB7RU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1783480298; c=relaxed/simple; bh=E/qfkyaqMo6dpCLXdpPfatjv9XTJb2Q9fC6Bd5QP5jE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=B6kzaB/dCYeSFlaBAYGQudqT6bmF54OV2wC19/H0xplvJ9xtCumYCPs9eQuHnDtlm87kvy38N65lotdFXqhqOW6G2GcLGotxhx3t2p84APcuOtUaapHOIqAkXGCDzjmQHoWBTv9IrdIx7PTb59QqMzF2mMJ1qhVZWn5elgmqwRs= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=GQxwbKHL; arc=none smtp.client-ip=209.85.214.202 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--jthoughton.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="GQxwbKHL" Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2ca0481106cso3703585ad.0 for ; Tue, 07 Jul 2026 20:11:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20251104; t=1783480297; x=1784085097; darn=vger.kernel.org; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:from:to:cc:subject:date:message-id:reply-to :content-type; bh=ziyOxc8kPU1zkWqQd/xo7T152fH4QK1CRdbLsKeEh/w=; b=GQxwbKHL5TWXIvaoO0eYNDe+8ioslQ+bY9iapcWZWYzwNMQmPMY6az6jlLgQVjoR/P TzyOycrbulO4eSHyl8T8h6D7O1Xs4QnuaEhxkS7Rwj3VgimKo9cjnssctcRD0qOPE9Fq l1mCRSpZtbHYTm3dqB56yxCMc9LWhhXot5e0IQi/Tgg7jRwPQhRuJ5urTrXLZ1j57AnP kKwbgGp08UixOnOhIJaSjT+fm1IvzWUyCGzxiUjapOlywyHCuozOmKLcR/NQ23Q1un+/ Ve4ck+xqr042eoYQBZd0TeH3mhM+lxLNdukNuY96oHrX48moOeaIskpU6DrY9qX0JgHg iCSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1783480297; x=1784085097; h=content-type:cc:to:from:subject:message-id:references:mime-version :in-reply-to:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to:content-type; bh=ziyOxc8kPU1zkWqQd/xo7T152fH4QK1CRdbLsKeEh/w=; b=RmsG3Ci7B+ohn5eapqmTjTIqP+K7KTVUdub+qkyBhiOipg+886T96K7Wym2jJ2lmSD WXBbEqZ4SuGNFRX1kzf9mBJVulyPc+2Qkr/9PMSwaXTRsKCoiaqBwDkJjG4b7xsKs+Af mgb6ZzHPRzHqf4tzESA8KuZhjbXL8tTiPzLzZN8yHHOisDK3jwwiVLxZ9Uzp5pgts4H+ CGgbsmpo2ZtMjRrd4iJiblui+jjN4lfxA/O5dO31YCnEjpEP4Fz7cowbK2coYPuNJA/M pncwrWz9fzCAT+95gigOSP+CBUwwWELPRMmYRUksg6/G1Q2KVdKd8MX5vVlN0Y+mhDz5 DLEw== X-Forwarded-Encrypted: i=1; AHgh+Ro/0KZIo5Nmf6IgqhwfcI9VCtNhtJWTO4mDzVFClpyFbnTI/sNjwg0a9oZVpmBOeHNWSrVpgJnIvn/3POM=@vger.kernel.org X-Gm-Message-State: AOJu0YwiWqnczfngBhEKZoIvf4CLImCUK3ckR9UpXQglhOcd+moPqi3g txd/Kyc+78RxrL5t2CNEmGV6GrXdM34FU/tnb2/WsLIxvQnos3erJZDV0Xae+z2xdhFqDwgHMiC dNru/VJ65B7fZWzKUj1evyg== X-Received: from plbkv5.prod.google.com ([2002:a17:903:28c5:b0:2ca:d66c:97b2]) (user=jthoughton job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1c9:b0:2cc:86a9:ebfb with SMTP id d9443c01a7336-2ccea47c361mr7188645ad.33.1783480296362; Tue, 07 Jul 2026 20:11:36 -0700 (PDT) Date: Wed, 8 Jul 2026 03:11:13 +0000 In-Reply-To: <20260708031129.3503195-1-jthoughton@google.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20260708031129.3503195-1-jthoughton@google.com> X-Mailer: git-send-email 2.55.0.795.g602f6c329a-goog Message-ID: <20260708031129.3503195-4-jthoughton@google.com> Subject: [PATCH 03/18] hugetlb_vmemmap: Leave pages partially HVOed upon restore failure From: James Houghton To: Will Deacon , Catalin Marinas , Muchun Song , Oscar Salvador Cc: Nikos Nikoleris , Linu Cherian , Mark Rutland , David Hildenbrand , Andrew Morton , Ryan Roberts , Nanyong Sun , Yu Zhao , Frank van der Linden , David Rientjes , James Houghton , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Right now it is assumed that the restore routine in vmemmap_remap_free() will not fail; that is, if we fail to fully optimize a folio, it is assumed that restoring the folio to its pre-HVO state is guaranteed to succeed. Although vmemmap restoration may never fail in practice, properly handle such failure cases now. This will leave folios in a partially HVOed state. In a follow-up patch, failure certainly becomes possible. In the event we end up with partially-HVOed folios, make sure to: 1. Leave the HVO page flag in place. 2. Free any unused vmemmap pages. Always pass the vmemmap_tail page to the restore routine to avoid leaking the non-optimized vmemmap pages for a partially HVOed page. Signed-off-by: James Houghton --- mm/hugetlb_vmemmap.c | 49 ++++++++++++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 11 deletions(-) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 8d4e8ee51fdd..5fee01bf5b34 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -236,12 +236,15 @@ static void vmemmap_restore_pte(pte_t *pte, unsigned long addr, { struct page *src = pte_page(ptep_get(pte)), *dst; + if (WARN_ON_ONCE(!walk->vmemmap_tail)) + return; + /* - * When rolling back vmemmap_remap_free(), keep the copied head page + * When restoring a partially-HVOed page, keep the copied head page * mapping and restore only PTEs currently pointing at the shared tail * page. */ - if (walk->vmemmap_tail && walk->vmemmap_tail != src) + if (walk->vmemmap_tail != src) return; VM_WARN_ON_ONCE(PageHead((const struct page *)addr)); @@ -277,6 +280,7 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end) return vmemmap_remap_range(start, end, &walk); } +static const int VMEMMAP_REMAP_INCOMPLETE = 1; /** * vmemmap_remap_free - remap the vmemmap virtual address range [@start, @end) * to use @vmemmap_head/tail, then free vmemmap which @@ -291,7 +295,8 @@ static int vmemmap_remap_split(unsigned long start, unsigned long end) * responsibility to free pages. * @flags: modifications to vmemmap_remap_walk flags * - * Return: %0 on success, negative error code otherwise. + * Return: %0 on success, VMEMMAP_REMAP_INCOMPLETE if the page is incompletely + * optimized, negative error code otherwise. */ static int vmemmap_remap_free(unsigned long start, unsigned long end, struct page *vmemmap_head, @@ -326,7 +331,8 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, .flags = 0, }; - vmemmap_remap_range(start, end, &walk); + if (vmemmap_remap_range(start, end, &walk)) + return VMEMMAP_REMAP_INCOMPLETE; return ret; } @@ -385,6 +391,8 @@ static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) * vmemmap_remap_alloc - remap the vmemmap virtual address range [@start, end) * to the page which is from the @vmemmap_pages * respectively. + * @h: the hstate for the folio whose vmemmap is getting remapped + * @folio: the folio whose vmemmap is getting remapped * @start: start address of the vmemmap virtual address range that we want * to remap. * @end: end address of the vmemmap virtual address range that we want to @@ -393,20 +401,35 @@ static struct page *vmemmap_get_tail(unsigned int order, struct zone *zone) * * Return: %0 on success, negative error code otherwise. */ -static int vmemmap_remap_alloc(unsigned long start, unsigned long end, +static int vmemmap_remap_alloc(const struct hstate *h, struct folio *folio, + unsigned long start, unsigned long end, unsigned long flags) { LIST_HEAD(vmemmap_pages); - struct vmemmap_remap_walk walk = { + struct vmemmap_remap_walk walk; + struct page *vmemmap_tail; + int ret; + + vmemmap_tail = vmemmap_get_tail(h->order, folio_zone(folio)); + if (WARN_ON_ONCE(!vmemmap_tail)) + return -ENOMEM; + + if (alloc_vmemmap_page_list(start, end, &vmemmap_pages)) + return -ENOMEM; + + walk = (struct vmemmap_remap_walk) { .remap_pte = vmemmap_restore_pte, + .vmemmap_tail = vmemmap_tail, .vmemmap_pages = &vmemmap_pages, .flags = flags, }; - if (alloc_vmemmap_page_list(start, end, &vmemmap_pages)) - return -ENOMEM; + ret = vmemmap_remap_range(start, end, &walk); - return vmemmap_remap_range(start, end, &walk); + /* Not all pages may have been consumed */ + free_vmemmap_page_list(&vmemmap_pages); + + return ret; } static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); @@ -439,7 +462,7 @@ static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, * When a HugeTLB page is freed to the buddy allocator, previously * discarded vmemmap pages must be allocated and remapping. */ - ret = vmemmap_remap_alloc(vmemmap_start, vmemmap_end, flags); + ret = vmemmap_remap_alloc(h, folio, vmemmap_start, vmemmap_end, flags); if (!ret) folio_clear_hugetlb_vmemmap_optimized(folio); @@ -572,7 +595,11 @@ static int __hugetlb_vmemmap_optimize_folio(const struct hstate *h, vmemmap_head, vmemmap_tail, vmemmap_pages, flags); out: - if (ret) + /* + * If ret == VMEMMAP_REMAP_INCOMPLETE, the folio might be partially + * HVOed. Leave the HVO page folio flag in place. + */ + if (ret < 0) folio_clear_hugetlb_vmemmap_optimized(folio); return ret; -- 2.55.0.795.g602f6c329a-goog