From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, HTML_MESSAGE,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A359C433DF for ; Fri, 15 May 2020 03:17:29 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 30FB52054F for ; Fri, 15 May 2020 03:17:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 30FB52054F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59878 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jZQql-00057H-Dq for qemu-devel@archiver.kernel.org; Thu, 14 May 2020 23:17:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45386) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jZQpp-0003pw-Fj for qemu-devel@nongnu.org; Thu, 14 May 2020 23:16:29 -0400 Received: from mga09.intel.com ([134.134.136.24]:8787) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1jZQpm-0006cL-TY for qemu-devel@nongnu.org; Thu, 14 May 2020 23:16:28 -0400 IronPort-SDR: /fRF7LmriesGQTPbWe9P1bU456Han1lA1qoN04c73Uh1iqnsWj3r8rt5UPQGENUoA+QkW15Lbb nAbpkHLTOmjA== X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga004.jf.intel.com ([10.7.209.38]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 14 May 2020 20:16:21 -0700 IronPort-SDR: naaBlX9Iz0pBghTl3Fa3l/jxFwJhAtZDwq6xbrBMXyXXYBF7TTy+mLaNHgb7qPKtJOrdZbj1EH Fp6mBHW++tQQ== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.73,393,1583222400"; d="scan'208,217";a="410311654" Received: from fmsmsx108.amr.corp.intel.com ([10.18.124.206]) by orsmga004.jf.intel.com with ESMTP; 14 May 2020 20:16:20 -0700 Received: from shsmsx601.ccr.corp.intel.com (10.109.6.141) by FMSMSX108.amr.corp.intel.com (10.18.124.206) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 14 May 2020 20:16:20 -0700 Received: from shsmsx605.ccr.corp.intel.com (10.109.6.215) by SHSMSX601.ccr.corp.intel.com (10.109.6.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Fri, 15 May 2020 11:16:18 +0800 Received: from shsmsx605.ccr.corp.intel.com ([10.109.6.215]) by SHSMSX605.ccr.corp.intel.com ([10.109.6.215]) with mapi id 15.01.1713.004; Fri, 15 May 2020 11:16:18 +0800 From: "Zhang, Chen" To: zhanghailiang , "Dr . David Alan Gilbert" , qemu-devel , Li Zhijian Subject: About migration/colo issue Thread-Topic: About migration/colo issue Thread-Index: AdYqZjudkmlVb8B9TvKOADpJ4VTuWA== Date: Fri, 15 May 2020 03:16:18 +0000 Message-ID: <7a26ed7efed94d2dbff591521d31076a@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-product: dlpe-windows dlp-version: 11.2.0.6 dlp-reaction: no-action x-originating-ip: [10.239.127.36] Content-Type: multipart/alternative; boundary="_000_7a26ed7efed94d2dbff591521d31076aintelcom_" MIME-Version: 1.0 Received-SPF: pass client-ip=134.134.136.24; envelope-from=chen.zhang@intel.com; helo=mga09.intel.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/05/14 23:16:21 X-ACL-Warn: Detected OS = FreeBSD 9.x or newer [fuzzy] X-Spam_score_int: -68 X-Spam_score: -6.9 X-Spam_bar: ------ X-Spam_report: (-6.9 / 5.0 requ) BAYES_00=-1.9, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001 autolearn=_AUTOLEARN X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Wang , Lukas Straub Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --_000_7a26ed7efed94d2dbff591521d31076aintelcom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Hailiang/Dave. I found a urgent problem in current upstream code, COLO will stuck on secon= dary checkpoint and later. The guest will stuck by this issue. I have bisect upstream code, this issue caused by Hailiang's optimize patch= : >From 0393031a16735835a441b6d6e0495a1bd14adb90 Mon Sep 17 00:00:00 2001 From: zhanghailiang Date: Mon, 24 Feb 2020 14:54:10 +0800 Subject: [PATCH] COLO: Optimize memory back-up process This patch will reduce the downtime of VM for the initial process, Previously, we copied all these memory in preparing stage of COLO while we need to stop VM, which is a time-consuming process. Here we optimize it by a trick, back-up every page while in migration process while COLO is enabled, though it affects the speed of the migration, but it obviously reduce the downtime of back-up all SVM'S memory in COLO preparing stage. Signed-off-by: zhanghailiang Message-Id: <20200224065414.36524-5-zhang.zhanghailiang@huawei.com> Signed-off-by: Dr. David Alan Gilbert minor typo fixes Hailiang, do you have time to look into it? The detail log: Primary node: 13322@1589511271.917346:colo_receive_message Receive 'checkpoint-ready' mes= sage {"timestamp": {"seconds": 1589511271, "microseconds": 917406}, "event": "RE= SUME"} 13322@1589511271.917842:colo_vm_state_change Change 'stop' =3D> 'run' 13322@1589511291.243346:colo_send_message Send 'checkpoint-request' message 13322@1589511291.243978:colo_receive_message Receive 'checkpoint-reply' mes= sage {"timestamp": {"seconds": 1589511291, "microseconds": 244096}, "event": "ST= OP"} 13322@1589511291.244444:colo_vm_state_change Change 'run' =3D> 'stop' 13322@1589511291.244561:colo_send_message Send 'vmstate-send' message 13322@1589511291.258594:colo_send_message Send 'vmstate-size' message 13322@1589511305.412479:colo_receive_message Receive 'vmstate-received' mes= sage 13322@1589511309.031826:colo_receive_message Receive 'vmstate-loaded' messa= ge {"timestamp": {"seconds": 1589511309, "microseconds": 31862}, "event": "RES= UME"} 13322@1589511309.033075:colo_vm_state_change Change 'stop' =3D> 'run' {"timestamp": {"seconds": 1589511311, "microseconds": 111617}, "event": "VN= C_CONNECTED", "data": {"server": {"auth": "none", "family": "ipv4", "servic= e": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": "i= pv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} {"timestamp": {"seconds": 1589511311, "microseconds": 116197}, "event": "VN= C_INITIALIZED", "data": {"server": {"auth": "none", "family": "ipv4", "serv= ice": "5907", "host": "0.0.0.0", "websocket": false}, "client": {"family": = "ipv4", "service": "51564", "host": "10.239.13.19", "websocket": false}}} 13322@1589511311.243271:colo_send_message Send 'checkpoint-request' message 13322@1589511311.351361:colo_receive_message Receive 'checkpoint-reply' mes= sage {"timestamp": {"seconds": 1589511311, "microseconds": 351439}, "event": "ST= OP"} 13322@1589511311.415779:colo_vm_state_change Change 'run' =3D> 'stop' 13322@1589511311.416001:colo_send_message Send 'vmstate-send' message 13322@1589511311.418620:colo_send_message Send 'vmstate-size' message Secondary node: {"timestamp": {"seconds": 1589510920, "microseconds": 778207}, "event": "RE= SUME"} 23619@1589510920.778835:colo_vm_state_change Change 'stop' =3D> 'run' 23619@1589510920.778891:colo_send_message Send 'checkpoint-ready' message 23619@1589510940.105539:colo_receive_message Receive 'checkpoint-request' m= essage {"timestamp": {"seconds": 1589510940, "microseconds": 105712}, "event": "ST= OP"} 23619@1589510940.105917:colo_vm_state_change Change 'run' =3D> 'stop' 23619@1589510940.105971:colo_send_message Send 'checkpoint-reply' message 23619@1589510940.106767:colo_receive_message Receive 'vmstate-send' message 23619@1589510940.122808:colo_flush_ram_cache_begin dirty_pages 2456 23619@1589510953.618672:colo_flush_ram_cache_end 23619@1589510953.945083:colo_receive_message Receive 'vmstate-size' message 23619@1589510954.274816:colo_send_message Send 'vmstate-received' message qemu-system-x86_64: warning: TSC frequency mismatch between VM (2792980 kHz= ) and host (2925999 kHz), and TSC scaling unavailable {"timestamp": {"seconds": 1589510957, "microseconds": 754184}, "event": "RE= SUME"} 23619@1589510957.894113:colo_vm_state_change Change 'stop' =3D> 'run' 23619@1589510957.894162:colo_send_message Send 'vmstate-loaded' message 23619@1589510960.105977:colo_receive_message Receive 'checkpoint-request' m= essage {"timestamp": {"seconds": 1589510960, "microseconds": 106148}, "event": "ST= OP"} 23619@1589510960.213773:colo_vm_state_change Change 'run' =3D> 'stop' 23619@1589510960.213797:colo_send_message Send 'checkpoint-reply' message 23619@1589510960.278771:colo_receive_message Receive 'vmstate-send' message 23619@1589510960.423268:colo_flush_ram_cache_begin dirty_pages 25 --_000_7a26ed7efed94d2dbff591521d31076aintelcom_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Hailiang/Dave.

 

I found a urgent problem in current upstream code, C= OLO will stuck on secondary checkpoint and later.

The guest will stuck by this issue.

I have bisect upstream code, this issue caused by Ha= iliang’s optimize patch:

 

From 0393031a16735835a441b6d6e0495a1bd14adb90 Mon Se= p 17 00:00:00 2001
From: zhanghailiang <zhang.zhanghailiang@huawei.com>
Date: Mon, 24 Feb 2020 14:54:10 +0800
Subject: [PATCH] COLO: Optimize memory back-up process

This patch will reduce the downtime of VM for the initial process,
Previously, we copied all these memory in preparing stage of COLO
while we need to stop VM, which is a time-consuming process.
Here we optimize it by a trick, back-up every page while in migration
process while COLO is enabled, though it affects the speed of the
migration, but it obviously reduce the downtime of back-up all SVM'S
memory in COLO preparing stage.

Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Message-Id: <20200224065414.36524-5-zhang.zhanghailiang@huawei.com> Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
  minor typo fixes

 

Hailiang, do you have time to look into it?

 

 

The detail log:

Primary node:

13322@1589511271.917346:colo_receive_message Receive= 'checkpoint-ready' message
{"timestamp": {"seconds": 1589511271, "microsecond= s": 917406}, "event": "RESUME"}
13322@1589511271.917842:colo_vm_state_change Change 'stop' =3D> 'run' 13322@1589511291.243346:colo_send_message Send 'checkpoint-request' message=
13322@1589511291.243978:colo_receive_message Receive 'checkpoint-reply' mes= sage
{"timestamp": {"seconds": 1589511291, "microsecond= s": 244096}, "event": "STOP"}
13322@1589511291.244444:colo_vm_state_change Change 'run' =3D> 'stop' 13322@1589511291.244561:colo_send_message Send 'vmstate-send' message
13322@1589511291.258594:colo_send_message Send 'vmstate-size' message
13322@1589511305.412479:colo_receive_message Receive 'vmstate-received' mes= sage
13322@1589511309.031826:colo_receive_message Receive 'vmstate-loaded' messa= ge
{"timestamp": {"seconds": 1589511309, "microsecond= s": 31862}, "event": "RESUME"}
13322@1589511309.033075:colo_vm_state_change Change 'stop' =3D> 'run' {"timestamp": {"seconds": 1589511311, "microsecond= s": 111617}, "event": "VNC_CONNECTED", "data&= quot;: {"server": {"auth": "none", "fami= ly": "ipv4", "service": "5907", "ho= st": "0.0.0.0", "websocket": false}, "client&= quot;: {"family": "ipv4", "service": "51= 564", "host": "10.239.13.19", "websocket": false}}}
{"timestamp": {"seconds": 1589511311, "microsecond= s": 116197}, "event": "VNC_INITIALIZED", "dat= a": {"server": {"auth": "none", "fa= mily": "ipv4", "service": "5907", "= host": "0.0.0.0", "websocket": false}, "clien= t": {"family": "ipv4", "service": "= 51564", "host": "10.239.13.19", "websocket": false}}}
13322@1589511311.243271:colo_send_message Send 'checkpoint-request' message=
13322@1589511311.351361:colo_receive_message Receive 'checkpoint-reply' mes= sage
{"timestamp": {"seconds": 1589511311, "microsecond= s": 351439}, "event": "STOP"}
13322@1589511311.415779:colo_vm_state_change Change 'run' =3D> 'stop' 13322@1589511311.416001:colo_send_message Send 'vmstate-send' message
13322@1589511311.418620:colo_send_message Send 'vmstate-size' message<= /o:p>

 

Secondary node:

{"timestamp": {"seconds": 158951= 0920, "microseconds": 778207}, "event": "RESUME&qu= ot;}
23619@1589510920.778835:colo_vm_state_change Change 'stop' =3D> 'run' 23619@1589510920.778891:colo_send_message Send 'checkpoint-ready' message 23619@1589510940.105539:colo_receive_message Receive 'checkpoint-request' m= essage
{"timestamp": {"seconds": 1589510940, "microsecond= s": 105712}, "event": "STOP"}
23619@1589510940.105917:colo_vm_state_change Change 'run' =3D> 'stop' 23619@1589510940.105971:colo_send_message Send 'checkpoint-reply' message 23619@1589510940.106767:colo_receive_message Receive 'vmstate-send' message=
23619@1589510940.122808:colo_flush_ram_cache_begin dirty_pages 2456
23619@1589510953.618672:colo_flush_ram_cache_end
23619@1589510953.945083:colo_receive_message Receive 'vmstate-size' message=
23619@1589510954.274816:colo_send_message Send 'vmstate-received' message qemu-system-x86_64: warning: TSC frequency mismatch between VM (2792980 kHz= ) and host (2925999 kHz), and TSC scaling unavailable
{"timestamp": {"seconds": 1589510957, "microsecond= s": 754184}, "event": "RESUME"}
23619@1589510957.894113:colo_vm_state_change Change 'stop' =3D> 'run' 23619@1589510957.894162:colo_send_message Send 'vmstate-loaded' message
23619@1589510960.105977:colo_receive_message Receive 'checkpoint-request' m= essage
{"timestamp": {"seconds": 1589510960, "microsecond= s": 106148}, "event": "STOP"}
23619@1589510960.213773:colo_vm_state_change Change 'run' =3D> 'stop' 23619@1589510960.213797:colo_send_message Send 'checkpoint-reply' message 23619@1589510960.278771:colo_receive_message Receive 'vmstate-send' message=
23619@1589510960.423268:colo_flush_ram_cache_begin dirty_pages 25

 

 

 

 

 

 

 

--_000_7a26ed7efed94d2dbff591521d31076aintelcom_--