From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756777AbdJKGB0 (ORCPT ); Wed, 11 Oct 2017 02:01:26 -0400 Received: from mga01.intel.com ([192.55.52.88]:32640 "EHLO mga01.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756206AbdJKGBZ (ORCPT ); Wed, 11 Oct 2017 02:01:25 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.43,360,1503385200"; d="scan'208";a="1180792462" Message-ID: <59DDB428.4020208@intel.com> Date: Wed, 11 Oct 2017 14:03:20 +0800 From: Wei Wang User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "Michael S. Tsirkin" CC: "virtio-dev@lists.oasis-open.org" , "linux-kernel@vger.kernel.org" , "qemu-devel@nongnu.org" , "virtualization@lists.linux-foundation.org" , "kvm@vger.kernel.org" , "linux-mm@kvack.org" , "mhocko@kernel.org" , "akpm@linux-foundation.org" , "mawilcox@microsoft.com" , "david@redhat.com" , "cornelia.huck@de.ibm.com" , "mgorman@techsingularity.net" , "aarcange@redhat.com" , "amit.shah@redhat.com" , "pbonzini@redhat.com" , "willy@infradead.org" , "liliang.opensource@gmail.com" , "yang.zhang.wz@gmail.com" , "quan.xu@aliyun.com" Subject: Re: [PATCH v16 5/5] virtio-balloon: VIRTIO_BALLOON_F_CTRL_VQ References: <1506744354-20979-1-git-send-email-wei.w.wang@intel.com> <1506744354-20979-6-git-send-email-wei.w.wang@intel.com> <20171001060305-mutt-send-email-mst@kernel.org> <286AC319A985734F985F78AFA26841F73932025A@shsmsx102.ccr.corp.intel.com> <20171010180636-mutt-send-email-mst@kernel.org> In-Reply-To: <20171010180636-mutt-send-email-mst@kernel.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 10/10/2017 11:15 PM, Michael S. Tsirkin wrote: > On Mon, Oct 02, 2017 at 04:38:01PM +0000, Wang, Wei W wrote: >> On Sunday, October 1, 2017 11:19 AM, Michael S. Tsirkin wrote: >>> On Sat, Sep 30, 2017 at 12:05:54PM +0800, Wei Wang wrote: >>>> +static void ctrlq_send_cmd(struct virtio_balloon *vb, >>>> + struct virtio_balloon_ctrlq_cmd *cmd, >>>> + bool inbuf) >>>> +{ >>>> + struct virtqueue *vq = vb->ctrl_vq; >>>> + >>>> + ctrlq_add_cmd(vq, cmd, inbuf); >>>> + if (!inbuf) { >>>> + /* >>>> + * All the input cmd buffers are replenished here. >>>> + * This is necessary because the input cmd buffers are lost >>>> + * after live migration. The device needs to rewind all of >>>> + * them from the ctrl_vq. >>> Confused. Live migration somehow loses state? Why is that and why is it a good >>> idea? And how do you know this is migration even? >>> Looks like all you know is you got free page end. Could be any reason for this. >> >> I think this would be something that the current live migration lacks - what the >> device read from the vq is not transferred during live migration, an example is the >> stat_vq_elem: >> Line 476 at https://github.com/qemu/qemu/blob/master/hw/virtio/virtio-balloon.c > This does not touch guest memory though it just manipulates > internal state to make it easier to migrate. > It's transparent to guest as migration should be. > >> For all the things that are added to the vq and need to be held by the device >> to use later need to consider the situation that live migration might happen at any >> time and they need to be re-taken from the vq by the device on the destination >> machine. >> >> So, even without this live migration optimization feature, I think all the things that are >> added to the vq for the device to hold, need a way for the device to rewind back from >> the vq - re-adding all the elements to the vq is a trick to keep a record of all of them >> on the vq so that the device side rewinding can work. >> >> Please let me know if anything is missed or if you have other suggestions. > IMO migration should pass enough data source to destination for > destination to continue where source left off without guest help. > I'm afraid it would be difficult to pass the entire VirtQueueElement to the destination. I think that would also be the reason that stats_vq_elem chose to rewind from the guest vq, which re-do the virtqueue_pop() --> virtqueue_map_desc() steps (the QEMU virtual address to the guest physical address relationship may be changed on the destination). How about another direction which would be easier - using two 32-bit device specific configuration registers, Host2Guest and Guest2Host command registers, to replace the ctrlq for command exchange: The flow can be as follows: 1) Before Host sending a StartCMD, it flushes the free_page_vq in case any old free page hint is left there; 2) Host writes StartCMD to the Host2Guest register, and notifies the guest; 3) Upon receiving a configuration notification, Guest reads the Host2Guest register, and detaches all the used buffers from free_page_vq; (then for each StartCMD, the free_page_vq will always have no obsolete free page hints, right? ) 4) Guest start report free pages: 4.1) Host may actively write StopCMD to the Host2Guest register before the guest finishes; or 4.2) Guest finishes reporting, write StopCMD the Guest2HOST register, which traps to QEMU, to stop. Best, Wei