From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3BF935966 for ; Tue, 4 Aug 2026 04:27:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817665; cv=none; b=uNSgFHr/8x75USqCNhyFvrCmnNdhZ55jjdPkjfwYrY3tRJ9iFHcRB4xy0vblqTU9E5lEs/RQ0wds94U/XOP3iSl+oJaTb4zqU7E1JswVTbGufitJhl062NBtrjqgWC/A41UOBdBdFdmb0P0w7Y/a1sdom/GziiYgZCp7MwqlTp8= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1785817665; c=relaxed/simple; bh=cXoQkyyXfXDi6EeAvKIQ+4F4h1tKWVLYSbGn0nVkvXo=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=QK+/racbGgYNxgaURG4lx/svqNn8oGuFvvMI2F8R4tqmI8mcIStzPfVS2XAHCUjsGC3gmBQxJleNtdK/+YN3y0jNPlWT4JP0Zx63H3P7kOPIpmzMgAtPIxhTPd3LsFL0anx943IilSgXDNfC17u9wtu9TU6vg/yH5MgK6S3Ijac= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=VTqUA1aD; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b=aIjCGpop; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="VTqUA1aD"; dkim=pass (2048-bit key) header.d=redhat.com header.i=@redhat.com header.b="aIjCGpop" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1785817662; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=VTqUA1aDn+yI3lr5PlkXEBO9vUb6QudUNU1MLlwz8lQe1oQnPq3i8s8NMDS1Iqh3QgGocO 2KcSITw7sOqlWNuOzVrcSO3jXVmzZ/O4vzTzwewIcDXJ5m4O1mKaKkeMxOti/P8an0oyGj eG8sFoEl6v7+3qgKbn942eMeQixalvQ= Received: from mail-lf1-f70.google.com (mail-lf1-f70.google.com [209.85.167.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-459-HNmyUF5VMiyEtkkJQ8EZsg-1; Tue, 04 Aug 2026 00:27:34 -0400 X-MC-Unique: HNmyUF5VMiyEtkkJQ8EZsg-1 X-Mimecast-MFC-AGG-ID: HNmyUF5VMiyEtkkJQ8EZsg_1785817653 Received: by mail-lf1-f70.google.com with SMTP id 2adb3069b0e04-5b15484404cso2283758e87.0 for ; Mon, 03 Aug 2026 21:27:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=google; t=1785817653; x=1786422453; darn=vger.kernel.org; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to:content-type; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=aIjCGpop5XjXRBsijO0mwA2P0BI0gv8s5R14MnBN4DrANgKCCrJWhePjAmTXI8jZyu jbLRpUMnY5R6jZw/BtHNK+mPR40UBsaUIX4nYzni0klPu6Avv7lyT8v2kC4zgiU/rcwo zWETNX/RYgEvx3RrdGupNEpPeVafcJ5r5iuyg+2LEOtAPBn4uEK+AtvXF6DmvDopi7VA dz/90tmS0SNlUAtzP7sc6wn49JvydnxK2iJrbV9Ogm0YQuFaKma6vlU5Z8kcYVjrRWvQ eFB4IvrLQYknMEBViOEnUJnif3PeSwQMhrOANoYLm7DVOiN48zuho1r7+Kb2RB+112+M t3Ng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1785817653; x=1786422453; h=content-transfer-encoding:content-type:mime-version:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=csSbLn2HNgFXzmHTSRr7vcVb4Wk3QmqGPDut2K4sshw=; b=ZfLJ257OaKmutrTdsj5uIkUWQ6s+cDyKtPUlWT7V23cntDb1UXXtsqC3SvjAcLRAuC pYeLsqkNNRUNOnctA5gT4e4RCMugbA3v1yrQAk9kZ8RGiKoq5wblak1bRKF+U0UHh1CB sC8a2lVi0jtHG8XfzCF2wyJeV3BovUfQgZZxT/a7FF/p+HKWkoATFnD1ss4ALfRW8G98 x98kW9bdwTTNmBtW8+hBDhLPejO/FHxlg5STUsMmiOMAvp4dlAVFJ3WQ7rCVDdOcCRE4 wWsDiB5H4CDNPQQSI1K3KL4rj19kBR72jzKlxJg0BhJwmLwHT5Z4XOT8ub4HUlQTeGiS /ctw== X-Forwarded-Encrypted: i=1; AHgh+Ropxk2E6vmNHCAfd9yDmamH9YaWV0P+jki5fLLag5iunRjmmng9jaEXx1FfLBZbwVU+gJdtpRVbfCd0XDY=@vger.kernel.org X-Gm-Message-State: AOJu0YwaIZo1ArtEAE4omG60+L9tQBhG8M1JB6CVb5YPOADtm0vUtySc 8j63WPB8aYIIN93pyLn9QSLTELcNFF3BBEh3q+7yiOwYjrMxQBu+3GsazokyJYv3tNFdf4Bs70n ZrH3l0rNBHj75noys7BQwlENpHmdiLRE945JShbE64+5h0Mr/R1Z1vatyl9ZmWUuvgtdB1wZZNl M= X-Gm-Gg: AR+sD10+cJmmnL0wTVTuTQVXMnwvp8tj2EN8za3GHCVQ/woKmxP+0fJTuDfffthyrb6 GeC7y/tvA63KFJ1icyqWfDTIQksVdU0oTbs87Rj5h9Zv5pj1Nw16X9f693Hvbwz+kuF6w0hPbGG uHE9bAzkqhop6DLp+Hk8dwz28+hOrZDtU0NG/Bloy2DLKWm2d99MYWFY5vywLKGjTRf4AdO6/8N Jgs265wpdydIMmPxYvHga/SkuQZGtk+ncwvkN3p2M8KhQD28aWPQQHATXSMLnt3BOvlhPRE2yX7 ncJku747/diy1ndVFmAg7IS366VKcnbjIzukc5aJbpuxp7Qn1yEocow00Hqn19Kxd5duc7ILGWw ThCAlMqcKtbw3u4uB X-Received: by 2002:a05:6512:1386:b0:5b1:4e5e:559c with SMTP id 2adb3069b0e04-5b2f2391564mr119394e87.30.1785817652763; Mon, 03 Aug 2026 21:27:32 -0700 (PDT) X-Received: by 2002:a05:6512:1386:b0:5b1:4e5e:559c with SMTP id 2adb3069b0e04-5b2f2391564mr119387e87.30.1785817652298; Mon, 03 Aug 2026 21:27:32 -0700 (PDT) Received: from fedora (85-23-51-1.bb.dnainternet.fi. [85.23.51.1]) by smtp.gmail.com with ESMTPSA id 2adb3069b0e04-5b2e23c46c4sm2311351e87.17.2026.08.03.21.27.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 Aug 2026 21:27:30 -0700 (PDT) From: mpenttil@redhat.com To: linux-mm@kvack.org Cc: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org, linux-kernel@vger.kernel.org, =?UTF-8?q?Mika=20Penttil=C3=A4?= , David Hildenbrand , Jason Gunthorpe , Leon Romanovsky , Alistair Popple , Balbir Singh , Zi Yan , Matthew Brost , Andrew Morton , Lorenzo Stoakes , "Liam R. Howlett" , Vlastimil Babka , Mike Rapoport , Suren Baghdasaryan , Michal Hocko Subject: [PATCH v13 09/11] mm/hmm: implement rollback for device page migration in HMM pagewalk Date: Tue, 4 Aug 2026 07:26:29 +0300 Message-ID: <20260804042631.2175585-10-mpenttil@redhat.com> X-Mailer: git-send-email 2.55.0 In-Reply-To: <20260804042631.2175585-1-mpenttil@redhat.com> References: <20260804042631.2175585-1-mpenttil@redhat.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit From: Mika Penttilä During the migration pagewalk, the PTE table could be cleared and/or changed into PMD leaf or even another PTE table while dropped locks. In these cases the possibly inserted migration ptes are gone. We have to however undo the collecting done so far, so unlock the folios and drop reference taken. During the pagewalk we notice such scenarios if going to recollect a pfn but have already committed to migrate the entry with HMM_PFN_MIGRATE, in which case rollback. If we encounter migration ptes they are just skipped to allow for restart own walks. Cc: David Hildenbrand Cc: Jason Gunthorpe Cc: Leon Romanovsky Cc: Alistair Popple Cc: Balbir Singh Cc: Zi Yan Cc: Matthew Brost Suggested-by: Alistair Popple Signed-off-by: Mika Penttilä --- include/linux/hmm.h | 22 +++++++++++++ mm/hmm.c | 76 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 98 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index bfedafc1c143..fbc9bb74d160 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -111,6 +111,28 @@ static inline unsigned int hmm_pfn_to_map_order(unsigned long hmm_pfn) return (hmm_pfn >> HMM_PFN_ORDER_SHIFT) & 0x1F; } +/* + * hmm_pfn_collected() - is this pfn entry prepared for migration ? + * If collected the folio's refcount is increased and the folio + * is locked. + */ +static inline bool hmm_pfn_collected(unsigned long hmm_pfn) +{ + return (hmm_pfn & (HMM_PFN_VALID | HMM_PFN_MIGRATE)) == + (HMM_PFN_VALID | HMM_PFN_MIGRATE); +} + +/* + * hmm_pfn_rollback() - undoes the collecction of hmm_pfn + * + * Note for total rollback the folio's refcount has to be put + * and folio has to be unlocked. + */ +static inline unsigned long hmm_pfn_rollback_collected(unsigned long hmm_pfn) +{ + return hmm_pfn & ~(HMM_PFN_VALID | HMM_PFN_MIGRATE | HMM_PFN_COMPOUND); +} + /* * struct hmm_range - track invalidation lock on virtual address range * diff --git a/mm/hmm.c b/mm/hmm.c index 631f6b4d52fc..03622455302a 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -78,6 +78,11 @@ enum { HMM_PFN_P2PDMA_BUS, }; +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn); + static int hmm_pfns_fill(unsigned long addr, unsigned long end, struct hmm_vma_walk *hmm_vma_walk, unsigned long cpu_flags) { @@ -94,6 +99,8 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, } } + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, end, &range->hmm_pfns[i]); + if (migrate && thp_migration_supported() && (minfo & MIGRATE_VMA_SELECT_COMPOUND) && IS_ALIGNED(addr, HPAGE_PMD_SIZE) && @@ -277,6 +284,8 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, required_fault, walk); } + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + end, hmm_pfns); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { hmm_pfns[i] &= HMM_PFN_INOUT_FLAGS; @@ -407,6 +416,9 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, new_pfn_flags = pte_pfn(pte) | cpu_flags; out: + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn = (*hmm_pfn & HMM_PFN_INOUT_FLAGS) | new_pfn_flags; return 0; @@ -445,6 +457,9 @@ static int hmm_vma_handle_absent_pmd(struct mm_walk *walk, unsigned long start, if (softleaf_is_device_private_write(entry)) cpu_flags |= HMM_PFN_WRITE; + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + start, end, + hmm_pfns); /* * Fully populate the PFN list though subsequent PFNs could be * inferred, because drivers which are not yet aware of large @@ -547,6 +562,48 @@ static int migrate_vma_split_folio(struct folio *folio, return 0; } +/* + * Due to dropping ptl locks for splitting for instance, would we + * overwrite already collected pfns? This could happen when pmd + * pointing to a page table has vanished and been replaced + * with a leaf pmd, or another page table. + * In that case unref and unlock the folios, + * the pfns of which were collected from the disappeared + * page tables. + */ +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ + struct hmm_range *range = hmm_vma_walk->range; + struct migrate_vma *migrate = range->migrate; + struct folio *fault_folio = NULL; + enum migrate_vma_info minfo; + struct folio *folio; + unsigned long i; + + minfo = hmm_select_migrate(range); + if (!minfo) + return; + + WARN_ON_ONCE(!migrate); + + fault_folio = migrate->fault_page ? + page_folio(migrate->fault_page) : NULL; + + for (i = 0; start < end; start += PAGE_SIZE, i++) { + if (hmm_pfn_collected(hmm_pfn[i])) { + folio = page_folio(hmm_pfn_to_page(hmm_pfn[i])); + if (folio != fault_folio) + folio_unlock(folio); + folio_put(folio); + hmm_pfn[i] = hmm_pfn_rollback_collected(hmm_pfn[i]); + + } + } +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -690,6 +747,11 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, pte = ptep_get(ptep); if (pte_none(pte)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); + if (vma_is_anonymous(walk->vma)) { *hmm_pfn &= HMM_PFN_INOUT_FLAGS; *hmm_pfn |= HMM_PFN_MIGRATE; @@ -737,6 +799,10 @@ static int hmm_vma_handle_migrate_prepare(const struct mm_walk *walk, pfn = pte_pfn(pte); if (is_zero_pfn(pfn) && (minfo & MIGRATE_VMA_SELECT_SYSTEM)) { + hmm_vma_handle_migrate_prepare_rollback(hmm_vma_walk, + addr, + addr + PAGE_SIZE, + hmm_pfn); *hmm_pfn = HMM_PFN_MIGRATE; goto out; } @@ -913,6 +979,13 @@ static int hmm_vma_walk_split(pmd_t *pmdp, return ret; } #else +static void hmm_vma_handle_migrate_prepare_rollback(const struct hmm_vma_walk *hmm_vma_walk, + unsigned long start, + unsigned long end, + unsigned long *hmm_pfn) +{ +} + static int hmm_vma_handle_migrate_prepare_pmd(const struct mm_walk *walk, pmd_t *pmdp, unsigned long start, @@ -1139,6 +1212,9 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, if (ptep) { lazy_mmu_mode_enable(); hmm_vma_walk->ptelocked = true; + } else { + /* The pte table is gone */ + hmm_vma_handle_migrate_prepare_rollback(walk->private, addr, end, hmm_pfns); } } else { ptep = pte_offset_map(pmdp, addr); -- 2.55.0