Re: [v1 resend 04/12] mm/migrate_device: THP migration of zone device pages

[Matthew Brost <matthew.brost@intel.com>]

On Fri, Jul 18, 2025 at 12:21:36AM -0700, Matthew Brost wrote:
> On Fri, Jul 18, 2025 at 05:04:39PM +1000, Balbir Singh wrote:
> > On 7/18/25 16:59, Matthew Brost wrote:
> > > On Fri, Jul 04, 2025 at 09:35:03AM +1000, Balbir Singh wrote:
> > >> +	if (thp_migration_supported() &&
> > >> +		(migrate->flags & MIGRATE_VMA_SELECT_COMPOUND) &&
> > >> +		(IS_ALIGNED(start, HPAGE_PMD_SIZE) &&
> > >> +		 IS_ALIGNED(end, HPAGE_PMD_SIZE))) {
> > >> +		migrate->src[migrate->npages] = MIGRATE_PFN_MIGRATE |
> > >> +						MIGRATE_PFN_COMPOUND;
> > >> +		migrate->dst[migrate->npages] = 0;
> > >> +		migrate->npages++;
> > >> +		migrate->cpages++;
> > > 
> > > It's a bit unclear what cpages and npages actually represent when
> > > collecting a THP. In my opinion, they should reflect the total number of
> > > minimum sized pages collected—i.e., we should increment by the shifted
> > > order (512) here. I'm fairly certain the logic in migrate_device.c would
> > > break if a 4MB range was requested and a THP was found first, followed by a
> > > non-THP.
> > > 
> > 
> > cpages and npages represent entries in the array and when or'ed with MIGRATE_PFN_COMPOUND
> > represent the right number of entries populated. If you have a test that shows
> > the breakage, I'd be keen to see it. We do populate other entries in 4k size(s) when
> > collecting to allow for a split of the folio.
> > 
> 
> I don’t have a test case, but let me quickly point out a logic bug.
> 
> Look at migrate_device_unmap. The variable i is incremented by
> folio_nr_pages, which seems correct. However, in the earlier code, we
> populate migrate->src using migrate->npages as the index, then increment
> it by 1. So, if two THPs are found back to back, they’ll occupy entries
> 0 and 1, while migrate_device_unmap will access entries 0 and 512.
> 
> Given that we have no idea what mix of THP vs non-THP we’ll encounter,
> the only sane approach is to populate the input array at minimum
> page-entry alignment. Similarly, npages and cpages should reflect the
> number of minimum-sized pages found, with the caller (and
> migrate_device) understanding that src and dst will be sparsely
> populated based on each entry’s folio order.
> 

I looked into this further and found another case where the logic breaks.

In __migrate_device_pages, the call to migrate_vma_split_pages assumes
that based on folio's order it can populate subsequent entries upon
split. This requires the source array to reflect the folio order upon
finding it.

Here’s a summary of how I believe the migrate_vma_setup interface should
behave, assuming 4K pages and 2M THPs:

Example A: 4MB requested, 2 THPs found and unmapped
src[0]:     folio, order 9, migrate flag set
src[1–511]: not present
src[512]:   folio, order 9, migrate flag set
src[513–1023]: not present
npages = 1024, cpages = 1024

Example B: 4MB requested, 2 THPs found, first THP unmap fails
src[0]:     folio, order 9, migrate flag clear
src[1–511]: not present
src[512]:   folio, order 9, migrate flag set
src[513–1023]: not present
npages = 1024, cpages = 512

Example C: 4MB requested, 512 small pages + 1 THP found, some small pages fail to unmap
src[0–7]:   folio, order 0, migrate flag clear  
src[8–511]: folio, order 0, migrate flag set  
src[512]:   folio, order 9, migrate flag set  
src[513–1023]: not present  
npages = 1024, cpages = 1016  

As I suggested in my previous reply to patch #2, this should be
documented—preferably in kernel-doc—so the final behavior is clear to
both migrate_device.c (and the structs in migrate.h) and the layers
above. I can help take a pass at writing kernel-doc for both, as its
behavior is fairly before you changes.

Matt

> Matt
> 
> > Thanks for the review,
> > Balbir Singh