From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 530D8C47080 for ; Tue, 1 Jun 2021 21:54:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 31C576124B for ; Tue, 1 Jun 2021 21:54:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234833AbhFAVzu (ORCPT ); Tue, 1 Jun 2021 17:55:50 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:21050 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234698AbhFAVzu (ORCPT ); Tue, 1 Jun 2021 17:55:50 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622584447; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TLmTFs79Kk/HdqD2U3Jsivs3GJU2uRn31j0/SZFo+5Y=; b=CZz+aERLjfH5M645R6+VPZh+PxvBh/Fpg2G7nFz0Nodw0KtVSox8uvTNnJAlcLKAbS+gq+ la2xgZ83+TN9fNNev+og522oiX9mvbvJDd9pGA/rGR8lPimbFNMBI3w9Cicw135LMGGTt4 q/B/dXfJR0JRU1lFDkk8klWQzksv4eo= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-563-_jN1rs2aMiqh_DH2_CK8Lw-1; Tue, 01 Jun 2021 17:54:06 -0400 X-MC-Unique: _jN1rs2aMiqh_DH2_CK8Lw-1 Received: by mail-qk1-f198.google.com with SMTP id u9-20020a05620a4549b02902e956c2a3c8so184015qkp.20 for ; Tue, 01 Jun 2021 14:54:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=TLmTFs79Kk/HdqD2U3Jsivs3GJU2uRn31j0/SZFo+5Y=; b=suXMXML5r9I+1vL9nPcAe9C2cHYLfomXVwFu1boZJhj7aSBWDb6faREgPXbvdN/WKn YX6m4mEkUAtYWdrHNp+BM5yPolauVibZ/ikyYjo+dst++6sgMAsSViw+b/T3/b+zLPCG vI6cvo1dSnsmtLvjR+g4PZ5mZzc/KW0dAqLoG7f3hG+lkCZgM219pcjp6DmSp4SPr+dn Et6WD+9JdxxfekdEm81t/6M1SvjzlJ+l3/ZzUAbDCkWfSnR52xl5sBd9Md59JKo+86so ChXWwEJzJmp5gL4MijQUQ2vYuzIQ6YblwM69LgIIf3vyZH7WorUDPOFulrF6yVsuB2tH 0aeA== X-Gm-Message-State: AOAM533mgk5Ds3QVMjePo4WYaHg94a28LkxykE8ibrJl8yufmA5vZJcV XRSlos0dWnlFMImQXXzhWayRlu2LumRom0ztadRW9R2v+srHQ+jYPOZwjdwRfRSgrHiCWZ/mz+g aM0RnYGAzmC7a X-Received: by 2002:ac8:698b:: with SMTP id o11mr12834843qtq.148.1622584445794; Tue, 01 Jun 2021 14:54:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzw3PnREwzRNFGSUnTvVgUPY8IkQ4jACsTTBjVGul0hqJfwLurKsrUBHidzwlCZJ8f3TECqFA== X-Received: by 2002:ac8:698b:: with SMTP id o11mr12834820qtq.148.1622584445481; Tue, 01 Jun 2021 14:54:05 -0700 (PDT) Received: from t490s (bras-base-toroon474qw-grc-61-184-147-118-108.dsl.bell.ca. [184.147.118.108]) by smtp.gmail.com with ESMTPSA id i1sm11121297qtg.81.2021.06.01.14.54.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Jun 2021 14:54:04 -0700 (PDT) Date: Tue, 1 Jun 2021 17:54:03 -0400 From: Peter Xu To: huangy81@chinatelecom.cn Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Juan Quintela , "Dr. David Alan Gilbert" , Paolo Bonzini , Chuan Zheng Subject: Re: [PATCH v1 0/6] support dirtyrate at the granualrity of vcpu Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Tue, Jun 01, 2021 at 01:02:45AM +0800, huangy81@chinatelecom.cn wrote: > From: Hyman Huang(黄勇) > > Since the Dirty Ring on QEMU part has been merged recently, how to use > this feature is under consideration. > > In the scene of migration, it is valuable to provide a more accurante > interface to track dirty memory than existing one, so that the upper > layer application can make a wise decision, or whatever. More importantly, > dirtyrate info at the granualrity of vcpu could provide a possibility to > make migration convergent by imposing restriction on vcpu. With Dirty > Ring, we can calculate dirtyrate efficiently and cheaply. > > The old interface implemented by sampling pages, it consumes cpu > resource, and the larger guest memory size become, the more cpu resource > it consumes, namely, hard to scale. New interface has no such drawback. Yong, Thanks for working on this! Some high-level comments: - The layout of the patch looks a bit odd. E.g., you introduced the new "vcpu" qmp parameter in patch 3, however it's not yet implemented, meanwhile I feel like you squashed mostly all the rest into patch 6. It's okay to use a single big patch, but IMHO better to not declare that flag in QMP before it's working, so ideally that should be the last patch to do that. From that POV: patch 1/2/4 look ok to be separated; perhaps squash patch 3/5/6 into one single patch to enable the new method as the last one? - You used "vcpu" across the patchset to show the per-vcpu new method. Shall we rename it globally to "per_vcpu" or "vcpu_based"? A raw "vcpu" looks more like a struct pointer not a boolean. - Using memory_global_dirty_log_start|stop() may not be wise too IMHO, at least we need to make sure it's not during migration, otherwise we could call the stop() before migration ends then that'll be a problem.. Maybe we can start to make global_dirty_log a bitmask? Then we define: GLOBAL_DIRTY_MIGRATION GLOBAL_DIRTY_DIRTY_RATE All references to global_dirty_log should mostly be untouched because any bit set there should justify that global dirty logging is enabled (either for migration or for dirty rate measurement). Migration starting half-way of dirty rate measurement seems okay too even taking things like init-all-set into account, afaict.. as long as dirty rate code never touches the qemu dirty bitmap, but only do the accounting when collecting the pages... Feel free to think more about it on any other potential conflict with migration, but in general seems working to me. - Would you consider picking up my HMP patch and let HMP work from the 1st day? - Please Cc the author of dirty rate too (Chuan Zheng ), while I already started to do so in this email. Thanks, -- Peter Xu From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9017C4708F for ; Tue, 1 Jun 2021 21:54:55 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 684436124B for ; Tue, 1 Jun 2021 21:54:55 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 684436124B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:33480 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1loCLe-0005nq-JL for qemu-devel@archiver.kernel.org; Tue, 01 Jun 2021 17:54:54 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35254) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1loCKy-00055L-Hf for qemu-devel@nongnu.org; Tue, 01 Jun 2021 17:54:12 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:23357) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1loCKv-0000UU-FE for qemu-devel@nongnu.org; Tue, 01 Jun 2021 17:54:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1622584448; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=TLmTFs79Kk/HdqD2U3Jsivs3GJU2uRn31j0/SZFo+5Y=; b=KO7rqGLQZNNAvA/v+Ug3OdaUuWZqLU1+3XCHSoxejWz6A+NamUL0cgvJZ+qOFWSEceAwTX SG/RUAi/f4JE8FsRWuG9djvc0iUbSjPKYSubpmf7tA87T4UMgqbzxH+IiINDn5EmUfEJ3P Rj2ewYA47VasnOrTIewUJ/O9uczCNqE= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-104-ibhYnZ3XPs6bh91Izr6nIA-1; Tue, 01 Jun 2021 17:54:06 -0400 X-MC-Unique: ibhYnZ3XPs6bh91Izr6nIA-1 Received: by mail-qt1-f198.google.com with SMTP id q3-20020a05622a0303b02902390ac8c906so234007qtw.11 for ; Tue, 01 Jun 2021 14:54:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=TLmTFs79Kk/HdqD2U3Jsivs3GJU2uRn31j0/SZFo+5Y=; b=HDIM5cb4YChnVCe8jjofF6rTOnR/QNnQ+gJHNKKb1AQSHpamSgzHK3ntVKPThJnLiY 0EGaN2UxlnOfG4Rt76ERG5zNoS4X8t7U0owTkZf9pPhzvcqNJfqvCqD9+AIl8chENCet 4ug1EI5us0LLDzbZ8qFeVqTdvCg3xsntdXHbbJsuxVP/axMlUO/2O/FIHUn5otjPCwmo TPHb0RRfOgVpMsQAAxTykstbzU13EbfZJrAg2Emi4YRdJBq/PKStGFhp4lzn3XKIAUls S1EJh4M2E0DQhlpCIUFb+/naOHBAN6LN+FH+pL6EvLbG5JZNExVCFcwctjGjJlv1UjeG Yn0Q== X-Gm-Message-State: AOAM533FZT2r9OR0R8iSlV2bdRijpxepTdSNjI2v6mO5v6+9GWy+4O0A wGpS5mcLSFrP551PTkTUdDPjnicHOTq3mnN2F+OQg/QMVz0uAPmIyVeo2gLb32qoD0GtfL50dxs xxCoP0cajyEP7V0c= X-Received: by 2002:ac8:698b:: with SMTP id o11mr12834847qtq.148.1622584445795; Tue, 01 Jun 2021 14:54:05 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzw3PnREwzRNFGSUnTvVgUPY8IkQ4jACsTTBjVGul0hqJfwLurKsrUBHidzwlCZJ8f3TECqFA== X-Received: by 2002:ac8:698b:: with SMTP id o11mr12834820qtq.148.1622584445481; Tue, 01 Jun 2021 14:54:05 -0700 (PDT) Received: from t490s (bras-base-toroon474qw-grc-61-184-147-118-108.dsl.bell.ca. [184.147.118.108]) by smtp.gmail.com with ESMTPSA id i1sm11121297qtg.81.2021.06.01.14.54.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Jun 2021 14:54:04 -0700 (PDT) Date: Tue, 1 Jun 2021 17:54:03 -0400 From: Peter Xu To: huangy81@chinatelecom.cn Subject: Re: [PATCH v1 0/6] support dirtyrate at the granualrity of vcpu Message-ID: References: MIME-Version: 1.0 In-Reply-To: Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit Received-SPF: pass client-ip=170.10.133.124; envelope-from=peterx@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.371, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kvm@vger.kernel.org, Juan Quintela , "Dr. David Alan Gilbert" , qemu-devel@nongnu.org, Chuan Zheng , Paolo Bonzini Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" On Tue, Jun 01, 2021 at 01:02:45AM +0800, huangy81@chinatelecom.cn wrote: > From: Hyman Huang(黄勇) > > Since the Dirty Ring on QEMU part has been merged recently, how to use > this feature is under consideration. > > In the scene of migration, it is valuable to provide a more accurante > interface to track dirty memory than existing one, so that the upper > layer application can make a wise decision, or whatever. More importantly, > dirtyrate info at the granualrity of vcpu could provide a possibility to > make migration convergent by imposing restriction on vcpu. With Dirty > Ring, we can calculate dirtyrate efficiently and cheaply. > > The old interface implemented by sampling pages, it consumes cpu > resource, and the larger guest memory size become, the more cpu resource > it consumes, namely, hard to scale. New interface has no such drawback. Yong, Thanks for working on this! Some high-level comments: - The layout of the patch looks a bit odd. E.g., you introduced the new "vcpu" qmp parameter in patch 3, however it's not yet implemented, meanwhile I feel like you squashed mostly all the rest into patch 6. It's okay to use a single big patch, but IMHO better to not declare that flag in QMP before it's working, so ideally that should be the last patch to do that. From that POV: patch 1/2/4 look ok to be separated; perhaps squash patch 3/5/6 into one single patch to enable the new method as the last one? - You used "vcpu" across the patchset to show the per-vcpu new method. Shall we rename it globally to "per_vcpu" or "vcpu_based"? A raw "vcpu" looks more like a struct pointer not a boolean. - Using memory_global_dirty_log_start|stop() may not be wise too IMHO, at least we need to make sure it's not during migration, otherwise we could call the stop() before migration ends then that'll be a problem.. Maybe we can start to make global_dirty_log a bitmask? Then we define: GLOBAL_DIRTY_MIGRATION GLOBAL_DIRTY_DIRTY_RATE All references to global_dirty_log should mostly be untouched because any bit set there should justify that global dirty logging is enabled (either for migration or for dirty rate measurement). Migration starting half-way of dirty rate measurement seems okay too even taking things like init-all-set into account, afaict.. as long as dirty rate code never touches the qemu dirty bitmap, but only do the accounting when collecting the pages... Feel free to think more about it on any other potential conflict with migration, but in general seems working to me. - Would you consider picking up my HMP patch and let HMP work from the 1st day? - Please Cc the author of dirty rate too (Chuan Zheng ), while I already started to do so in this email. Thanks, -- Peter Xu