Re: [PATCH 0/2] Migration time prediction using calc-dirty-rate

["Daniel P. Berrangé" <berrange@redhat.com>]

On Tue, Feb 28, 2023 at 04:16:01PM +0300, Andrei Gudkov via wrote:
> Summary of calc-dirty-rate changes:
> 
> 1. The most important change is that now calc-dirty-rate produces
>    a *vector* of dirty page measurements for progressively increasing time
>    periods: 125ms, 250, 500, 750, 1000, 1500, .., up to specified calc-time.
>    The motivation behind such change is that number of dirtied pages as
>    a function of time starting from "clean state" (new migration iteration)
>    is far from linear. Shape of this function depends on the workload type
>    and intensity. Measuring number of dirty pages at progressively
>    increasing periods allows to reconstruct this function using piece-wise
>    interpolation.
> 
> 2. New metric added -- number of all-zero pages.
>    Predictor needs to distinguish between number of zero and non-zero pages
>    because during migration only 8 byte header is placed on the wire for
>    all-zero page.
> 
> 3. Hashing function was changed from CRC32 to xxHash.
>    This reduces overhead of sampling by ~10 times, which is important since
>    now some of the measurement periods are sub-second.

Very good !

> 
> 4. Other trivial metrics were added for convenience: total number
>    of VM pages, number of sampled pages, page size.
> 
> 
> After these changes output from calc-dirty-rate looks like this:
> 
> {
>   "page-size": 4096,
>   "periods": [125, 250, 375, 500, 750, 1000, 1500,
>               2000, 3000, 4001, 6000, 8000, 10000,
>               15000, 20000, 25000, 30000, 35000,
>               40000, 45000, 50000, 60000],
>   "status": "measured",
>   "sample-pages": 512,
>   "dirty-rate": 98,
>   "mode": "page-sampling",
>   "n-dirty-pages": [33, 78, 119, 151, 217, 236, 293, 336,
>                     425, 505, 620, 756, 898, 1204, 1457,
>                     1723, 1934, 2141, 2328, 2522, 2675, 2958],
>   "n-sampled-pages": 16392,
>   "n-zero-pages": 10060,
>   "n-total-pages": 8392704,
>   "start-time": 2916750,
>   "calc-time": 60
> }

Ok, so "periods" and "n-dirty-pages" pages arrays correlate with
each other.

> 
> Passing this data into prediction script, we get the following estimations:
> 
> Downtime> |    125ms |    250ms |    500ms |   1000ms |   5000ms |    unlim
> ---------------------------------------------------------------------------
>  100 Mbps |        - |        - |        - |        - |        - |   16m59s  
>    1 Gbps |        - |        - |        - |        - |        - |    1m40s
>    2 Gbps |        - |        - |        - |        - |    1m41s |      50s  
>  2.5 Gbps |        - |        - |        - |        - |    1m07s |      40s
>    5 Gbps |      48s |      46s |      31s |      28s |      25s |      20s
>   10 Gbps |      13s |      12s |      12s |      12s |      12s |      10s
>   25 Gbps |       5s |       5s |       5s |       5s |       4s |       4s
>   40 Gbps |       3s |       3s |       3s |       3s |       3s |       3s

This is fascinating and really helpful as an idea. It so nicely
shows the when it is not even worth bothering to try to start the
migrate unless you're willing to put up with large (5 sec) downtime.
or use autoconverge/post-copy.

I wonder if the calc-dirty-rate measurements also give enough info
to predict the likely number/duration of async page fetches needed
during post-copy phase ? Or does this give enough info to predict
how far down auto-converge should throttle the guest to enable
convergance.

> Quality of prediction was tested with YCSB benchmark. Memcached instance
> was installed into 32GiB VM, and a client generated a stream of requests.
> Between experiments we varied request size distribution, number of threads,
> and location of the client (inside or outside the VM).
> After short preheat phase, we measured calc-dirty-rate:
> 1. {"execute": "calc-dirty-rate", "arguments":{"calc-time":60}}
> 2. Wait 60 seconds
> 3. Collect results with {"execute": "query-dirty-rate"}
> 
> Afterwards we tried to migrate VM after randomly selecting max downtime
> and bandwidth limit. Typical prediction error is 6-7%, with only 180 out
> of 5779 experiments failing badly: prediction error >=25% or incorrectly
> predicting migration success when in fact it didn't converge.

Nice results


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|