Re: [PATCH] tests/qtest/migrate-test: Add a postcopy memfile test

["Nicholas Piggin" <npiggin@gmail.com>]

On Thu May 30, 2024 at 3:35 AM AEST, Fabiano Rosas wrote:
> Peter Xu <peterx@redhat.com> writes:
>
> > On Wed, May 29, 2024 at 09:54:30AM -0300, Fabiano Rosas wrote:
> >> Nicholas Piggin <npiggin@gmail.com> writes:
> >> 
> >> > Postcopy requires userfaultfd support, which requires tmpfs if a memory
> >> > file is used.
> >> >
> >> > This adds back support for /dev/shm memory files, but adds preallocation
> >> > to skip environments where that mount is limited in size.
> >> >
> >> > Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
> >> > ---
> >> >
> >> > How about this? This goes on top of the reset of the patches
> >> > (I'll re-send them all as a series if we can get to some agreement).
> >> >
> >> > This adds back the /dev/shm option with preallocation and adds a test
> >> > case that requires tmpfs.
> >> 
> >> Peter has stronger opinions on this than I do. I'll leave it to him to
> >> decide.
> >
> > Sorry if I gave that feeling; it's more of a stronger willingness to at
> > some point enable shmem for QEMU migration, rather than wanting to push
> > back what Nicholas was trying to do.
>
> Of course, I didn't mean to imply that. I just saying that using /tmp
> would have been fine with me and I don't want to get in the way.
>
> > Enabling more arch for migration
> > tests is definitely worthwhile on its own.
> >
> > Shmem is just some blank spot that IMHO we should start to think about
> > better coverarge. E.g. it is the only sane way to boot the VM that is able
> > to do fast qemu upgrades using ignore-shared, that was true even before
> > Steve's cpr-exec work, which would be much easier than anonymous. And it's
> > also possible shmem can be (in the next 3-5 years) the 1G page provider to
> > replace hugetlb for postcopy's sake - this one is far beyond our current
> > discussion so I won't extend..
>
> Interesting, good to know.
>
> >
> > IMHO shmem should just be a major backend just like anonymous, and the only
> > possible file backend we can test in CI - as hugetlb is harder to manage
> > there.
> >
> >> 
> >> Just note that now we're making the CI less deterministic in relation to
> >> the migration tests. When a test that uses shmem fails, we'll not be
> >> able to consistently reproduce because the test might not even run
> >> depending on what has consumed the shmem first.
> >> 
> >> Let's also take care that the other consumers of shmem (I think just
> >> ivshmem-test) are able to cope with the migration-test taking all the
> >> space, otherwise the CI will still break.
> >
> > Looks like ivshmem-test only uses 1MB shmem constantly so probably that
> > will succeed if the migration test will, but true they face the same
> > challenge and they interfere with each other..  that test sidently pass
> > (instead of skip) if mktempshm() fails.  I guess we don't have a way to
> > solidly test shmem as shmem simply may not be around.
>
> Here we have each of the 5 migration archs taking up some amount of
> memory + each of the 3 supported arches for ivshmem. They all could be
> running in parallel through make check. In practice there's maybe less
> overlap due to timing and not all CI jobs building all archs, but still.

I could just add back the GITLAB_CI gate for shm tests for now then.
>
> >
> > For this patch alone personally I'd avoid using "use_uffd_memfile" as the
> > name, as that's definitely confusing, since shmem can be tested in other
> > setups too without uffd.  Nicolas, please feel free to move ahead with your

I was just thinking uffd could be used with another memfile (hugetlbfs)
but on second thoughts that's a bit silly. So use_shm_memfile would be
better.

> > arch enablement series with /tmp if you want to separate the shmem issue.
>
> Or just leave ignore_shared untouched for the ppc series.

Good idea, I think everybody is happy enough with ppc series so I
will send that first.

Thanks,
Nick