From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([209.51.188.92]:53270) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJNZk-0006R5-6X for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:29:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJNN2-0001MD-RB for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:15:54 -0400 Received: from mail-yb1-xb44.google.com ([2607:f8b0:4864:20::b44]:44357) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hJNN2-0001If-JN for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:15:52 -0400 Received: by mail-yb1-xb44.google.com with SMTP id u187so7532705ybg.11 for ; Wed, 24 Apr 2019 12:15:51 -0700 (PDT) MIME-Version: 1.0 References: <20190424160942.13567-1-brijesh.singh@amd.com> In-Reply-To: <20190424160942.13567-1-brijesh.singh@amd.com> From: Steve Rutherford Date: Wed, 24 Apr 2019 12:15:12 -0700 Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Subject: Re: [Qemu-devel] [RFC PATCH v1 00/10] Add AMD SEV guest live migration support List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Singh, Brijesh" Cc: "kvm@vger.kernel.org" , "qemu-devel@nongnu.org" , Thomas Gleixner , Ingo Molnar , "H. Peter Anvin" , Paolo Bonzini , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Joerg Roedel , Borislav Petkov , "Lendacky, Thomas" , "x86@kernel.org" , "linux-kernel@vger.kernel.org" On Wed, Apr 24, 2019 at 9:10 AM Singh, Brijesh wrot= e: > > The series add support for AMD SEV guest live migration commands. To prot= ect the > confidentiality of an SEV protected guest memory while in transit we need= to > use the SEV commands defined in SEV API spec [1]. > > SEV guest VMs have the concept of private and shared memory. Private memo= ry > is encrypted with the guest-specific key, while shared memory may be encr= ypted > with hypervisor key. The commands provided by the SEV FW are meant to be = used > for the private memory only. The patch series introduces a new hypercall. > The guest OS can use this hypercall to notify the page encryption status. > If the page is encrypted with guest specific-key then we use SEV command = during > the migration. If page is not encrypted then fallback to default. > > The patch adds a new ioctl KVM_GET_PAGE_ENC_BITMAP. The ioctl can be used > by the qemu to get the page encrypted bitmap. Qemu can consult this bitma= p > during the migration to know whether the page is encrypted. > > [1] https://developer.amd.com/wp-content/resources/55766.PDF > > The series is tested with the Qemu, I am in process of cleaning > up the Qemu code and will submit soon. > > While implementing the migration I stumbled on the follow question: > > - Since there is a guest OS changes required to support the migration, > so how do we know whether guest OS is updated? Should we extend KVM > capabilities/feature bits to check this? > > TODO: > - add an ioctl to build encryption bitmap. The encryption bitmap is buil= t during > the guest bootup/execution. We should provide an ioctl so that destina= tion > can build the bitmap as it receives the pages. > - reset the bitmap on guest reboot. > > The complete tree with patch is available at: > https://github.com/codomania/kvm/tree/sev-migration-rfc-v1 > > Cc: Thomas Gleixner > Cc: Ingo Molnar > Cc: "H. Peter Anvin" > Cc: Paolo Bonzini > Cc: "Radim Kr=C4=8Dm=C3=A1=C5=99" > Cc: Joerg Roedel > Cc: Borislav Petkov > Cc: Tom Lendacky > Cc: x86@kernel.org > Cc: kvm@vger.kernel.org > Cc: linux-kernel@vger.kernel.org > > Brijesh Singh (10): > KVM: SVM: Add KVM_SEV SEND_START command > KVM: SVM: Add KVM_SEND_UPDATE_DATA command > KVM: SVM: Add KVM_SEV_SEND_FINISH command > KVM: SVM: Add support for KVM_SEV_RECEIVE_START command > KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command > KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command > KVM: x86: Add AMD SEV specific Hypercall3 > KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall > KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl > mm: x86: Invoke hypercall when page encryption status is changed > > .../virtual/kvm/amd-memory-encryption.rst | 116 ++++ > Documentation/virtual/kvm/hypercalls.txt | 14 + > arch/x86/include/asm/kvm_host.h | 3 + > arch/x86/include/asm/kvm_para.h | 12 + > arch/x86/include/asm/mem_encrypt.h | 3 + > arch/x86/kvm/svm.c | 560 +++++++++++++++++- > arch/x86/kvm/vmx/vmx.c | 1 + > arch/x86/kvm/x86.c | 17 + > arch/x86/mm/mem_encrypt.c | 45 +- > arch/x86/mm/pageattr.c | 15 + > include/uapi/linux/kvm.h | 51 ++ > include/uapi/linux/kvm_para.h | 1 + > 12 files changed, 834 insertions(+), 4 deletions(-) > > -- > 2.17.1 > What's the back-of-the-envelope marginal cost of transferring a 16kB region from one host to another? I'm interested in what the end to end migration perf changes look like for this. If you have measured migration perf, I'm interested in that also. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E688C10F11 for ; Wed, 24 Apr 2019 20:13:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 439B52175B for ; Wed, 24 Apr 2019 20:13:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="raMrLPW2" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 439B52175B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=nongnu.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([127.0.0.1]:46744 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJOH3-0008S8-Fj for qemu-devel@archiver.kernel.org; Wed, 24 Apr 2019 16:13:45 -0400 Received: from eggs.gnu.org ([209.51.188.92]:53270) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hJNZk-0006R5-6X for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:29:01 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hJNN2-0001MD-RB for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:15:54 -0400 Received: from mail-yb1-xb44.google.com ([2607:f8b0:4864:20::b44]:44357) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1hJNN2-0001If-JN for qemu-devel@nongnu.org; Wed, 24 Apr 2019 15:15:52 -0400 Received: by mail-yb1-xb44.google.com with SMTP id u187so7532705ybg.11 for ; Wed, 24 Apr 2019 12:15:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=KXoQ7BFg8MCQ5w2hqas+YRo88Lj083QKeczBFQqo+mk=; b=raMrLPW2n1ySXsJ6Qmx9cvhAR1LPiM58H94Ej4RtkOTT/cuhxC+TgVkMETA3AMnIZW utg/Xp0jYhyNQ2H0AVcXTLZVJvOz9TLtcgtL+sReblkJjBdvC4PBznLzpAABE4F+EV9g LpzKI5aigxtjVsAe8KIdWlwvTIMSy7GJ/t8h7uXcvxZOZX+s8SBXcv3i9F0kl83jP46R rRrAr1i7EC8YDeAHhdSFtG1mSuZv4GATkUIjYr11v4HNq4rO825DI6hK0MfAOPwbhvU4 8On6Xcpq3txtwSHoXqX6ZLDu5m6sTdXXCdgTROj7iE0SzYl/X6bmPveBwIOoGBYZUJHz uL7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=KXoQ7BFg8MCQ5w2hqas+YRo88Lj083QKeczBFQqo+mk=; b=jkUAjDiyYtRJYeeSRyY0qb5Z7A0ZggB+iGy7lXVe97fCsqI/8NYD3+2cNexA2xIhv+ 1mpyZi5o+d5Sr6mSR3o8ZjYhvd8wmvPocqdQIG7b0MVr9tqrwt7wHn+ceg+cJboDiTgK hqQ7LSET6BrcfreIVEmLFWhjmqac9d266zi3hev3t0ooFmXQcSkhbWWILxE8jgxuy3/z VSezV03IAZ2DpSD0+tXdDa1GYnkOxGXSaYDm1FU0H9GvRAmtKyUl/MV7YJ4LC1K7LfPl jUHrBoLOaA8HmOQoBl8bFLeBVBqTn7PZwreWGTQ+mgrSQvpHJnAksUQFFgpzUE+vo0O7 RBhg== X-Gm-Message-State: APjAAAV3jFYmNtQpZQkBz3fo1O8/pGKfSAqgfZtHt2RdUJQDq8pvsvgz 1wK9b7N7PGNwGHc1X/g8kVQT+iAvWHMQCi1tmR6tOQ== X-Google-Smtp-Source: APXvYqwuvnaNjb+LHJwQCOoaewr1dMCcdZFhFpVWce8CYO3hVcV9gxLrABhtjfqSjY46v+1HYZXRfcPfBZu3clAo3O0= X-Received: by 2002:a25:e081:: with SMTP id x123mr27613524ybg.152.1556133349594; Wed, 24 Apr 2019 12:15:49 -0700 (PDT) MIME-Version: 1.0 References: <20190424160942.13567-1-brijesh.singh@amd.com> In-Reply-To: <20190424160942.13567-1-brijesh.singh@amd.com> Date: Wed, 24 Apr 2019 12:15:12 -0700 Message-ID: To: "Singh, Brijesh" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-Received-From: 2607:f8b0:4864:20::b44 X-Mailman-Approved-At: Wed, 24 Apr 2019 16:12:52 -0400 Subject: Re: [Qemu-devel] [RFC PATCH v1 00/10] Add AMD SEV guest live migration support X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , From: Steve Rutherford via Qemu-devel Reply-To: Steve Rutherford Cc: "Lendacky, Thomas" , "kvm@vger.kernel.org" , =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= , Joerg Roedel , "x86@kernel.org" , "qemu-devel@nongnu.org" , "linux-kernel@vger.kernel.org" , Ingo Molnar , "H. Peter Anvin" , Paolo Bonzini , Thomas Gleixner , Borislav Petkov Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message-ID: <20190424191512.5fJmg-PEBjurgOUaMSKpdpGBEQ8cskOXVlkS8hCU3F0@z> On Wed, Apr 24, 2019 at 9:10 AM Singh, Brijesh wrot= e: > > The series add support for AMD SEV guest live migration commands. To prot= ect the > confidentiality of an SEV protected guest memory while in transit we need= to > use the SEV commands defined in SEV API spec [1]. > > SEV guest VMs have the concept of private and shared memory. Private memo= ry > is encrypted with the guest-specific key, while shared memory may be encr= ypted > with hypervisor key. The commands provided by the SEV FW are meant to be = used > for the private memory only. The patch series introduces a new hypercall. > The guest OS can use this hypercall to notify the page encryption status. > If the page is encrypted with guest specific-key then we use SEV command = during > the migration. If page is not encrypted then fallback to default. > > The patch adds a new ioctl KVM_GET_PAGE_ENC_BITMAP. The ioctl can be used > by the qemu to get the page encrypted bitmap. Qemu can consult this bitma= p > during the migration to know whether the page is encrypted. > > [1] https://developer.amd.com/wp-content/resources/55766.PDF > > The series is tested with the Qemu, I am in process of cleaning > up the Qemu code and will submit soon. > > While implementing the migration I stumbled on the follow question: > > - Since there is a guest OS changes required to support the migration, > so how do we know whether guest OS is updated? Should we extend KVM > capabilities/feature bits to check this? > > TODO: > - add an ioctl to build encryption bitmap. The encryption bitmap is buil= t during > the guest bootup/execution. We should provide an ioctl so that destina= tion > can build the bitmap as it receives the pages. > - reset the bitmap on guest reboot. > > The complete tree with patch is available at: > https://github.com/codomania/kvm/tree/sev-migration-rfc-v1 > > Cc: Thomas Gleixner > Cc: Ingo Molnar > Cc: "H. Peter Anvin" > Cc: Paolo Bonzini > Cc: "Radim Kr=C4=8Dm=C3=A1=C5=99" > Cc: Joerg Roedel > Cc: Borislav Petkov > Cc: Tom Lendacky > Cc: x86@kernel.org > Cc: kvm@vger.kernel.org > Cc: linux-kernel@vger.kernel.org > > Brijesh Singh (10): > KVM: SVM: Add KVM_SEV SEND_START command > KVM: SVM: Add KVM_SEND_UPDATE_DATA command > KVM: SVM: Add KVM_SEV_SEND_FINISH command > KVM: SVM: Add support for KVM_SEV_RECEIVE_START command > KVM: SVM: Add KVM_SEV_RECEIVE_UPDATE_DATA command > KVM: SVM: Add KVM_SEV_RECEIVE_FINISH command > KVM: x86: Add AMD SEV specific Hypercall3 > KVM: X86: Introduce KVM_HC_PAGE_ENC_STATUS hypercall > KVM: x86: Introduce KVM_GET_PAGE_ENC_BITMAP ioctl > mm: x86: Invoke hypercall when page encryption status is changed > > .../virtual/kvm/amd-memory-encryption.rst | 116 ++++ > Documentation/virtual/kvm/hypercalls.txt | 14 + > arch/x86/include/asm/kvm_host.h | 3 + > arch/x86/include/asm/kvm_para.h | 12 + > arch/x86/include/asm/mem_encrypt.h | 3 + > arch/x86/kvm/svm.c | 560 +++++++++++++++++- > arch/x86/kvm/vmx/vmx.c | 1 + > arch/x86/kvm/x86.c | 17 + > arch/x86/mm/mem_encrypt.c | 45 +- > arch/x86/mm/pageattr.c | 15 + > include/uapi/linux/kvm.h | 51 ++ > include/uapi/linux/kvm_para.h | 1 + > 12 files changed, 834 insertions(+), 4 deletions(-) > > -- > 2.17.1 > What's the back-of-the-envelope marginal cost of transferring a 16kB region from one host to another? I'm interested in what the end to end migration perf changes look like for this. If you have measured migration perf, I'm interested in that also.