From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFD96C2D0E4 for ; Mon, 23 Nov 2020 09:38:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9F2182078E for ; Mon, 23 Nov 2020 09:38:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=ovhcloud.com header.i=@ovhcloud.com header.b="CQeHFOpg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F2182078E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=ovhcloud.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:59566 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kh8Iy-0004cx-CV for qemu-devel@archiver.kernel.org; Mon, 23 Nov 2020 04:38:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:55782) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kh8HV-0003zE-3j for qemu-devel@nongnu.org; Mon, 23 Nov 2020 04:37:09 -0500 Received: from 1.mo301.mail-out.ovh.net ([137.74.110.64]:53723) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kh8HR-0005xQ-JT for qemu-devel@nongnu.org; Mon, 23 Nov 2020 04:37:08 -0500 Received: from EX2.OVH.local (gw2.corp.ovh.com [51.255.55.227]) by mo301.mail-out.ovh.net (Postfix) with ESMTPS id 4C0247FDEE for ; Mon, 23 Nov 2020 10:36:56 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ovhcloud.com; s=mailout; t=1606124216; bh=DyM1vlwzyEOPY6R/PFEufKgDfXd17opKzh6vXoAnPxc=; h=From:To:Subject:Date:From; b=CQeHFOpgdapUUxyR7mmeQyIFPMNBer+mxuBZLii3poZBg5X8bAxtfblKDWcdkhC7J 1BKLc4NX+LFnsdbrHIRIuRAwy0GRdddQ4ZJr0O0Zx2r44naaAT8uU36BSpIl/uzolI k6tsobrVSElhTxtGvgwmDJvuFjMzKeSE0/9ZUatg= Received: from DAGFR7EX3.OVH.local (172.16.2.22) by EX2.OVH.local (172.16.2.2) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2106.2; Mon, 23 Nov 2020 10:36:55 +0100 Received: from DAGFR7EX2.OVH.local (172.16.2.21) by DAGFR7EX3.OVH.local (172.16.2.22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.721.2; Mon, 23 Nov 2020 10:36:55 +0100 Received: from DAGFR7EX2.OVH.local ([fe80::81c3:c718:13b4:fd11]) by DAGFR7EX2.OVH.local ([fe80::81c3:c718:13b4:fd11%3]) with mapi id 15.02.0721.003; Mon, 23 Nov 2020 10:36:55 +0100 From: Quentin Grolleau To: "qemu-devel@nongnu.org" Subject: [raw] Guest stuck during live live-migration Thread-Topic: [raw] Guest stuck during live live-migration Thread-Index: AQHWwXs6AGfiafuLa06oUseJIin6Sw== Date: Mon, 23 Nov 2020 09:36:55 +0000 Message-ID: Accept-Language: fr-FR, en-US Content-Language: fr-FR X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [109.190.254.58] x-ovh-corplimit-skip: true Content-Type: multipart/alternative; boundary="_000_e6f25c7e67ce4cfea5e01e4e46f0a3d8ovhcloudcom_" MIME-Version: 1.0 X-Ovh-Tracer-Id: 9419278622906236480 X-VR-SPAMSTATE: OK X-VR-SPAMSCORE: 0 X-VR-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgedujedrudegiedgtdejucetufdoteggodetrfdotffvucfrrhhofhhilhgvmecuqfggjfdpvefjgfevmfevgfenuceurghilhhouhhtmecuhedttdenucenucfjughrpefhvffuthffkfhitgggsegrtdhjredttddunecuhfhrohhmpefsuhgvnhhtihhnucfirhholhhlvggruhcuoehquhgvnhhtihhnrdhgrhholhhlvggruhesohhvhhgtlhhouhgurdgtohhmqeenucggtffrrghtthgvrhhnpeekheehhffhveetheetudefffetgeehgfeggeeugfevveevtdffueduffdvvdevjeenucffohhmrghinhepghhnuhdrohhrghdpghhithhhuhgsrdgtohhmnecukfhppedtrddtrddtrddtpddutdelrdduledtrddvheegrdehkeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhhouggvpehsmhhtphdqohhuthdphhgvlhhopefgigdvrdfqggfjrdhlohgtrghlpdhinhgvtheptddrtddrtddrtddpmhgrihhlfhhrohhmpehquhgvnhhtihhnrdhgrhholhhlvggruhesohhvhhgtlhhouhgurdgtohhmpdhrtghpthhtohepqhgvmhhuqdguvghvvghlsehnohhnghhnuhdrohhrgh Received-SPF: pass client-ip=137.74.110.64; envelope-from=quentin.grolleau@ovhcloud.com; helo=1.mo301.mail-out.ovh.net X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --_000_e6f25c7e67ce4cfea5e01e4e46f0a3d8ovhcloudcom_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello, In our company, we are hosting a large number of Vm, hosted behind Openstac= k (so libvirt/qemu). A large majority of our Vms are runnign with local data only, stored on NVM= E, and most of them are RAW disks. With Qemu 4.0 (can be even with older version) we see strange live-migrati= on comportement: - some Vms live migrate at very high speed without issue (> 6 Gbps) - some Vms are running correctly, but migrating at a strange low speed = (3Gbps) - some Vms are migrating at a very low speed (1Gbps, sometime less) and= during the migration the guest is completely I/O stuck When this issue happen the VM is completly block, iostat in the Vm show us = a latency of 30 secs First we thought it was related to an hardware issue we check it, we compar= ing different hardware, but no issue where found there So one of my colleague had the idea to limit with "tc" the bandwidth on the= interface the migration was done, and it worked the VM didn't lose any pin= g nor being I/O stuck Important point : Once the Vm have been migrate (with the limitation ) one = time, if we migrate it again right after, the migration will be done at ful= l speed (8-9Gb/s) without freezing the Vm It only happen on existing VM, we tried to replicate with a fresh instance = with exactly the same spec and nothing was happening We tried to replicate the workload inside the VM but there was no way to re= plicate the case. So it was not related to the workload nor to the server t= hat hosts the Vm So we thought about the disk of the instance : the raw file. We also tried to strace -c the process during the live-migration and it was= doing a lot of "lseek" and we found this : https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00462.html So i rebuilt Qemu with this patch and the live-migration went well, at high= speed and with no VM freeze ( https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ) Do you have a way to avoid the "lseek" mechanism as it consumes more resour= ces to find the holes in the disk and don't let any for the VM ? Server hosting the VM : - Bi-Xeon hosts With NVME storage and 10 Go Network card - Qemu 4.0 And Libvirt 5.4 - Kernel 4.18.0.25 Guest having the issue : - raw image with Debian 8 Here the qemu img on the disk : > qemu-img info disk image: disk file format: raw virtual size: 400G (429496729600 bytes) disk size: 400G Quentin GROLLEAU --_000_e6f25c7e67ce4cfea5e01e4e46f0a3d8ovhcloudcom_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

Hello,

In our company, we are hosting a large number of Vm, hosted behind Opensta= ck (so libvirt/qemu). 
A large majority of our Vms are runnign with local data= only, stored on NVME, and most of them are RAW disks.

With Qemu 4.0 (can be even with older version) we see = strange  live-migration comportement:
    - some Vms live migrate at very high= speed without issue (> 6 Gbps)
    - some Vms are running correctly, bu= t migrating at a strange low speed (3Gbps)
    - some Vms are migrating at a very = low speed (1Gbps, sometime less) and during the migration the guest is comp= letely I/O stuck
    
When this iss= ue happen th= e VM is completly bl= ock, iostat in the Vm show us a latency of 30 secs

First we thought it was related to an hardw= are issue we check it, we comparing different hardware, but no issue where found there

So one of my colleague had the idea to limit wit= h "tc" the bandwidth on the interface the migration was done, and= it worked the VM didn't lose any ping nor being  I/O  stuck
Important point : On= ce the Vm have been migrate (= with the = limitation ) one time= , if we migrate it again right after, the migration will be done at full sp= eed (8-9Gb/s) wi= thout freezing the Vm

It only happen on existing VM, we tried to repli= cate with a fresh instance with exactly the same spec and nothing was happe= ning

We tried to replicate the workload inside the VM= but there was no way to replicate the case. So it was not related to the w= orkload nor to the server that hosts the Vm

So we thought about the disk of the instance : t= he raw file.

We also tried to strace -c the process during th= e live-migration and it was doing a lot of "lseek"

and we found this : 


So i rebuilt Qemu with this patch and the live-m= igration went well, at high speed and with no VM freeze
( https://github.com/qemu/qem= u/blob/master/block/file-posix.c#L2601 )

Do you have a way to= avoid the "lseek" mechanism as it consumes more resources to fin= d the holes in the disk and don't let any for the VM ?


Server hosting the VM : 
    - Bi-Xeon = hosts With NV= ME storage and 10 Go Network card
    - Qemu 4.0 And Libvirt 5.4
    - Kernel 4.18.0.25

Guest having the issue : 
    - raw image with Debian 8

Here the qemu img on the disk : 
> qemu-img info disk
image: disk
file format: raw
virtual size: 400G (429496729600 bytes)
disk size: 400G


Quentin GROLLEAU

--_000_e6f25c7e67ce4cfea5e01e4e46f0a3d8ovhcloudcom_--