RE: [PATCH 0/2] Migration time prediction using calc-dirty-rate

[Gudkov Andrei via <qemu-devel@nongnu.org>]

Thank you for the review. I submitted new version of the patch:
https://patchew.org/QEMU/cover.1682598010.git.gudkov.andrei@huawei.com/

> -----Original Message-----
> From: Daniel P. Berrangé [mailto:berrange@redhat.com]
> Sent: Tuesday, April 18, 2023 20:18
> To: Gudkov Andrei <gudkov.andrei@huawei.com>
> Cc: qemu-devel@nongnu.org; quintela@redhat.com; dgilbert@redhat.com
> Subject: Re: [PATCH 0/2] Migration time prediction using calc-dirty-rate
> 
> On Tue, Feb 28, 2023 at 04:16:01PM +0300, Andrei Gudkov via wrote:
> > Summary of calc-dirty-rate changes:
> >
> > 1. The most important change is that now calc-dirty-rate produces
> >    a *vector* of dirty page measurements for progressively increasing time
> >    periods: 125ms, 250, 500, 750, 1000, 1500, .., up to specified calc-time.
> >    The motivation behind such change is that number of dirtied pages as
> >    a function of time starting from "clean state" (new migration iteration)
> >    is far from linear. Shape of this function depends on the workload type
> >    and intensity. Measuring number of dirty pages at progressively
> >    increasing periods allows to reconstruct this function using piece-wise
> >    interpolation.
> >
> > 2. New metric added -- number of all-zero pages.
> >    Predictor needs to distinguish between number of zero and non-zero pages
> >    because during migration only 8 byte header is placed on the wire for
> >    all-zero page.
> >
> > 3. Hashing function was changed from CRC32 to xxHash.
> >    This reduces overhead of sampling by ~10 times, which is important since
> >    now some of the measurement periods are sub-second.
> 
> Very good !
> 
> >
> > 4. Other trivial metrics were added for convenience: total number
> >    of VM pages, number of sampled pages, page size.
> >
> >
> > After these changes output from calc-dirty-rate looks like this:
> >
> > {
> >   "page-size": 4096,
> >   "periods": [125, 250, 375, 500, 750, 1000, 1500,
> >               2000, 3000, 4001, 6000, 8000, 10000,
> >               15000, 20000, 25000, 30000, 35000,
> >               40000, 45000, 50000, 60000],
> >   "status": "measured",
> >   "sample-pages": 512,
> >   "dirty-rate": 98,
> >   "mode": "page-sampling",
> >   "n-dirty-pages": [33, 78, 119, 151, 217, 236, 293, 336,
> >                     425, 505, 620, 756, 898, 1204, 1457,
> >                     1723, 1934, 2141, 2328, 2522, 2675, 2958],
> >   "n-sampled-pages": 16392,
> >   "n-zero-pages": 10060,
> >   "n-total-pages": 8392704,
> >   "start-time": 2916750,
> >   "calc-time": 60
> > }
> 
> Ok, so "periods" and "n-dirty-pages" pages arrays correlate with
> each other.
> 
> >
> > Passing this data into prediction script, we get the following estimations:
> >
> > Downtime> |    125ms |    250ms |    500ms |   1000ms |   5000ms |    unlim
> > ---------------------------------------------------------------------------
> >  100 Mbps |        - |        - |        - |        - |        - |   16m59s
> >    1 Gbps |        - |        - |        - |        - |        - |    1m40s
> >    2 Gbps |        - |        - |        - |        - |    1m41s |      50s
> >  2.5 Gbps |        - |        - |        - |        - |    1m07s |      40s
> >    5 Gbps |      48s |      46s |      31s |      28s |      25s |      20s
> >   10 Gbps |      13s |      12s |      12s |      12s |      12s |      10s
> >   25 Gbps |       5s |       5s |       5s |       5s |       4s |       4s
> >   40 Gbps |       3s |       3s |       3s |       3s |       3s |       3s
> 
> This is fascinating and really helpful as an idea. It so nicely
> shows the when it is not even worth bothering to try to start the
> migrate unless you're willing to put up with large (5 sec) downtime.
> or use autoconverge/post-copy.
> 
> I wonder if the calc-dirty-rate measurements also give enough info
> to predict the likely number/duration of async page fetches needed
> during post-copy phase ? Or does this give enough info to predict
> how far down auto-converge should throttle the guest to enable
> convergance.

I also was thinking about supporting more migration features.
Currently my understanding is the following:

1. It *should* be possible to support throttling directly inside the
   prediction script without any changes to calc-dirty-rate. Maybe we can
   suggest the level of throttling required to achieve target downtime.

2. Support for compression would be harder because we would have to know
   average compression ratio and compression speed. This would require
   more changes to calc-dirty-rate.

3. To support post-copy, we would need to know network characteristics, namely
   latency and jitter. Both can be quite unstable unless source and target
   hosts are located very close in network topology.

> 
> > Quality of prediction was tested with YCSB benchmark. Memcached instance
> > was installed into 32GiB VM, and a client generated a stream of requests.
> > Between experiments we varied request size distribution, number of threads,
> > and location of the client (inside or outside the VM).
> > After short preheat phase, we measured calc-dirty-rate:
> > 1. {"execute": "calc-dirty-rate", "arguments":{"calc-time":60}}
> > 2. Wait 60 seconds
> > 3. Collect results with {"execute": "query-dirty-rate"}
> >
> > Afterwards we tried to migrate VM after randomly selecting max downtime
> > and bandwidth limit. Typical prediction error is 6-7%, with only 180 out
> > of 5779 experiments failing badly: prediction error >=25% or incorrectly
> > predicting migration success when in fact it didn't converge.
> 
> Nice results
> 
> 
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|