Re: [PATCH v6 00/23] migration: File based migration with multifd and mapped-ram

["Daniel P. Berrangé" <berrange@redhat.com>]

On Mon, Mar 04, 2024 at 08:35:36PM +0800, Peter Xu wrote:
> Fabiano,
> 
> On Thu, Feb 29, 2024 at 12:29:54PM -0300, Fabiano Rosas wrote:
> > => guest: 128 GB RAM - 120 GB dirty - 1 vcpu in tight loop dirtying memory
> 
> I'm curious normally how much time does it take to do the final fdatasync()
> for you when you did this test.
> 
> I finally got a relatively large system today and gave it a quick shot over
> 128G (100G busy dirty) mapped-ram snapshot with 8 multifd channels.  The
> migration save/load does all fine, so I don't think there's anything wrong
> with the patchset, however when save completes (I'll need to stop the
> workload as my disk isn't fast enough I guess..) I'll always hit a super
> long hang of QEMU on fdatasync() on XFS during which the main thread is in
> UNINTERRUPTIBLE state.

That isn't very surprising. If you don't have O_DIRECT enabled, then
all that disk I/O from the migrate is going to be in RAM, and thus the
fdatasync() is likely to trigger writing out alot of data.

Blocking the main QEMU thread though is pretty unhelpful. That suggests
the data sync needs to be moved to a non-main thread.

With O_DIRECT meanwhile there should be essentially no hit from fdatasync.

> 
> [<0>] rq_qos_wait+0xbb/0x130
> [<0>] wbt_wait+0x9c/0x100
> [<0>] __rq_qos_throttle+0x23/0x40
> [<0>] blk_mq_submit_bio+0x183/0x580
> [<0>] __submit_bio_noacct+0x7e/0x1e0
> [<0>] iomap_submit_ioend+0x4e/0x80
> [<0>] iomap_writepage_map+0x22a/0x400
> [<0>] write_cache_pages+0x17c/0x4c0
> [<0>] iomap_writepages+0x1c/0x40
> [<0>] xfs_vm_writepages+0x7a/0xb0 [xfs]
> [<0>] do_writepages+0xcf/0x1d0
> [<0>] filemap_fdatawrite_wbc+0x66/0x90
> [<0>] __filemap_fdatawrite_range+0x54/0x80
> [<0>] file_write_and_wait_range+0x48/0xb0
> [<0>] xfs_file_fsync+0x5a/0x240 [xfs]
> [<0>] __x64_sys_fdatasync+0x46/0x80
> [<0>] do_syscall_64+0x5c/0x90
> [<0>] entry_SYSCALL_64_after_hwframe+0x72/0xdc
> 
> Do you also have it, or it's just my host kernel / other config that is
> different?
> 
> -- 
> Peter Xu
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|