RE: [PATCH] migration: Move bitmap_mutex out of migration_bitmap_clear_dirty()

["Wang, Wei W" <wei.w.wang@intel.com>]

On Wednesday, July 7, 2021 2:00 AM, Peter Xu wrote:
> On Fri, Jul 02, 2021 at 02:29:41AM +0000, Wang, Wei W wrote:
> > With that, if free page opt is off, the mutex is skipped, isn't it?
> 
> Yes, but when free page is on, it'll check once per page.  As I mentioned I still
> don't think it's the right thing to do.

With free page opt on, if the migration thread waits for lock acquire on a page, it actually means that it is trying to skip the transfer of a page.
For example, waiting for the lock takes 100ns, then the skip of sending a page saves back 1000ns, then overall we saved 900ns per page (i.e. pay less and earn more).

> 
> We encountered this problem when migrating a 3tb vm and the mutex spins and
> eats tons of cpu resources.  It shouldn't happen with/without balloon, imho.

I think we should compare the overall migration time.

> 
> Not to mention the hard migration issues are mostly with non-idle guest, in that
> case having the balloon in the guest will be disastrous from this pov since it'll start
> to take mutex for each page, while balloon would hardly report anything valid
> since most guest pages are being used.

If no pages are reported, migration thread wouldn't wait on the lock then.

To conclude: to decide whether the per page lock hurts the performance considering that the lock in some sense actually prevents the migration thread from sending free pages which it shouldn't, we need to compare the overall migration time.
(previous data could be found here:https://patchwork.kernel.org/project/kvm/cover/1535333539-32420-1-git-send-email-wei.w.wang@intel.com/, I think the situation should be the same for either 8GB or 3TB guest, in terms of the overall migration time comparison) 

Best,
Wei