From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67B0DC433B4 for ; Fri, 21 May 2021 08:49:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 49BCB613AC for ; Fri, 21 May 2021 08:49:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233264AbhEUIuX (ORCPT ); Fri, 21 May 2021 04:50:23 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:35738 "EHLO us-smtp-delivery-124.mimecast.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230288AbhEUIuX (ORCPT ); Fri, 21 May 2021 04:50:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1621586940; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=C+RrPrGVxZisWP0sBdZiifQusgNF1wAFIFuVSD1WN80=; b=Xgc1+y5Xkt84wbhy+bMkKjQCsXdL4gPBIKazOWg66tsJJZk3vYkN4cS0oIKF3mpksz4kMH gllWeEPhr14Us6at72VWBRSwOaijOTPPnzdBa1xUHK58tsHfw/1cQMcAfbxq8jxNTK8W23 qIkd+lNpzesR0pYl/Oe01T1usBahLeY= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-457-pCFmskScNqWyAFelhh7pfg-1; Fri, 21 May 2021 04:48:58 -0400 X-MC-Unique: pCFmskScNqWyAFelhh7pfg-1 Received: by mail-wm1-f72.google.com with SMTP id 12-20020a1c010c0000b0290176491efde9so3556441wmb.4 for ; Fri, 21 May 2021 01:48:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=C+RrPrGVxZisWP0sBdZiifQusgNF1wAFIFuVSD1WN80=; b=uapcLhtcAQRbM0pxX4NAc95m/daLUH8yXI7BP9DPCtWkQqPhtfvK/EDWS44Z8nNZFt zegGR/Ssk9YSZ1EwpkhN1IZcgtvuMh+/5157+SPmwiXBmwUmUzAzvf1t08NkuuxQpheI cgZVZS13gZG8yOq5fnIuN4ebKMScAA7A3raOwNRVSOoDddTakYQNcNPrUNOLm0ksbrYR GtxFgonAEKSv69c8IjLXsTMg5i7BYUjkDjDkKVaO0q2nds5s4hnnbQzuu2JdMbGQnJnG CgE59HtxOMlRHSlogof4N0Kvi2/+zVYTpvJq7b6//l1M1Q8/h84OD++IOSkd+i8eAnz1 BFJg== X-Gm-Message-State: AOAM533aefiTh56Bokje6pUowRqVN+606/a3HUBj3SnYu+XeYcquoPO7 Fum80IJ9dFxwXKG5fbRuxzElLOqK3jfvmyKKxdluPFIP0yeWqPSVYOwBzf7R2pSvFIDgb9eRZ6g 0HsyMqVmtCo/n X-Received: by 2002:a5d:4a81:: with SMTP id o1mr7955095wrq.177.1621586937343; Fri, 21 May 2021 01:48:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwx9FKQyrvOPP7boT1WEW/qQzOKlleOxVcg2ZXiXvfEsG7bp5bbEt8GrCAOXSmuZgRMrpaDzQ== X-Received: by 2002:a5d:4a81:: with SMTP id o1mr7955083wrq.177.1621586937185; Fri, 21 May 2021 01:48:57 -0700 (PDT) Received: from vitty.brq.redhat.com (g-server-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id m11sm1226436wri.44.2021.05.21.01.48.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 01:48:56 -0700 (PDT) From: Vitaly Kuznetsov To: Liang Li Cc: qemu-devel@nongnu.org, kvm@vger.kernel.org, Tianyu.Lan@microsoft.com Subject: Re: About the performance of hyper-v In-Reply-To: References: <87cztmkdlp.fsf@vitty.brq.redhat.com> Date: Fri, 21 May 2021 10:48:55 +0200 Message-ID: <87y2c8iia0.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Liang Li writes: >> > Hi Vitaly, >> > >> > I found a case that the virtualization overhead was almost doubled >> > when turning on Hper-v related features compared to that without any >> > no hyper-v feature. It happens when running a 3D game in windows >> > guest in qemu kvm environment. >> > >> > By investigation, I found there are a lot of IPIs triggered by guest, >> > when turning on the hyer-v related features including stimer, for the >> > apicv is turned off, at least two vm exits are needed for processing a >> > single IPI. >> > >> > >> > perf stat will show something like below [recorded for 5 seconds] >> > >> > --------- >> > >> > Analyze events for all VMs, all VCPUs: >> > VM-EXIT Samples Samples% Time% Min Time Max >> > Time Avg time >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us >> > 65.42us 2.34us ( +- 0.11% ) >> > MSR_WRITE 238932 30.33% 23.07% 0.48us >> > 41.05us 1.56us ( +- 0.14% ) >> > >> > Total Samples:787803, Total events handled time:1611193.84us. >> > >> > I tried turning off hyper-v for the same workload and repeat the test, >> > the overall virtualization overhead reduced by about of 50%: >> > >> > ------- >> > >> > Analyze events for all VMs, all VCPUs: >> > >> > VM-EXIT Samples Samples% Time% Min Time Max >> > Time Avg time >> > APIC_WRITE 255152 74.43% 50.72% 0.49us >> > 50.01us 1.42us ( +- 0.14% ) >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us >> > 686.05us 7.27us ( +- 0.43% ) >> > DR_ACCESS 35003 10.21% 4.64% 0.32us >> > 40.03us 0.95us ( +- 0.32% ) >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us >> > 57.38us 2.25us ( +- 1.42% ) >> > >> > Total Samples:342788, Total events handled time:715695.62us. >> > >> > For this scenario, hyper-v works really bad. stimer works better >> > than hpet, but on the other hand, it relies on SynIC which has >> > negative effects for IPI intensive workloads. >> > Do you have any plans for improvement? >> > >> >> Hey, >> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM >> automatically disables APICv and this can explain the overhead and the >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only >> inhibit APICv if the recommendation was ignored. This is implemented in >> the following KVM patch series: >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/ >> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it >> can be tested with 'hv-passthrough'. >> >> It would be great if you could give it a spin! >> >> -- >> Vitaly > > It's great to know that you already have a solution for this. :) > > By the way, is there any requirement for the version of windows or > windows updates for the new feature to work? AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd expect WS2008 to ignore it completely (and thus SynIC will always be disabling APICv for it). -- Vitaly From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE31C433B4 for ; Fri, 21 May 2021 08:51:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F03716109F for ; Fri, 21 May 2021 08:50:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F03716109F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37192 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lk0ry-00042x-Tm for qemu-devel@archiver.kernel.org; Fri, 21 May 2021 04:50:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:58204) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lk0qE-0002tV-0p for qemu-devel@nongnu.org; Fri, 21 May 2021 04:49:11 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]:31655) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lk0q5-0000om-GV for qemu-devel@nongnu.org; Fri, 21 May 2021 04:49:07 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1621586940; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=C+RrPrGVxZisWP0sBdZiifQusgNF1wAFIFuVSD1WN80=; b=Xgc1+y5Xkt84wbhy+bMkKjQCsXdL4gPBIKazOWg66tsJJZk3vYkN4cS0oIKF3mpksz4kMH gllWeEPhr14Us6at72VWBRSwOaijOTPPnzdBa1xUHK58tsHfw/1cQMcAfbxq8jxNTK8W23 qIkd+lNpzesR0pYl/Oe01T1usBahLeY= Received: from mail-wr1-f72.google.com (mail-wr1-f72.google.com [209.85.221.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-40-E6lgFdUsMbmyrH1j-iEMFw-1; Fri, 21 May 2021 04:48:58 -0400 X-MC-Unique: E6lgFdUsMbmyrH1j-iEMFw-1 Received: by mail-wr1-f72.google.com with SMTP id u5-20020adf9e050000b029010df603f280so9140692wre.18 for ; Fri, 21 May 2021 01:48:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:in-reply-to:references:date :message-id:mime-version; bh=C+RrPrGVxZisWP0sBdZiifQusgNF1wAFIFuVSD1WN80=; b=JoLo7GFQ8Yv53B2dQLvsAwsqnPQt/AkkTgR+lckNAmBMFqK+HBi9xk+nbqGLkGzKnO z5L5wqbhtQFO0FbAgS7L8WgZ0ivxAGQbc7oXbZBcq8bqcDcX79pJlDpZFZhmIaCXbqEf GHhRscyKgn2UJK5t/EmV4Bwo13IbAV6LiEEeonte3RMhZRFZX9SIzWGn3VYLHfCBUrT2 IuxEOXbKQfqjvYl+Byf8N63d6Ce7Wv2Cl6VzyY1sNmzrKPRFgwNKuTABMnzVCddb6ZYt apzS1Q6lmpUHeiIwmYM73eQBjzVLTgHcQQvzc+BFU0W31vIm7SV5C4ZKM5u/HZWHubFD LVIg== X-Gm-Message-State: AOAM531iUZJ4NMVA5CFqTsg8QOM+diZTAO1ZjvEerEFUuApoH0Ninf4j /Osy+Bg/qmb7kzDcKhLckI4dWeZwHCOjAjcpePU+J8rlHwb1i6h7eI0u8SKqCdPwhW0itMbR/8w YRz7Mwt3q4WY8RO8= X-Received: by 2002:a5d:4a81:: with SMTP id o1mr7955099wrq.177.1621586937372; Fri, 21 May 2021 01:48:57 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwx9FKQyrvOPP7boT1WEW/qQzOKlleOxVcg2ZXiXvfEsG7bp5bbEt8GrCAOXSmuZgRMrpaDzQ== X-Received: by 2002:a5d:4a81:: with SMTP id o1mr7955083wrq.177.1621586937185; Fri, 21 May 2021 01:48:57 -0700 (PDT) Received: from vitty.brq.redhat.com (g-server-2.ign.cz. [91.219.240.2]) by smtp.gmail.com with ESMTPSA id m11sm1226436wri.44.2021.05.21.01.48.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 May 2021 01:48:56 -0700 (PDT) From: Vitaly Kuznetsov To: Liang Li Subject: Re: About the performance of hyper-v In-Reply-To: References: <87cztmkdlp.fsf@vitty.brq.redhat.com> Date: Fri, 21 May 2021 10:48:55 +0200 Message-ID: <87y2c8iia0.fsf@vitty.brq.redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=vkuznets@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain Received-SPF: pass client-ip=170.10.133.124; envelope-from=vkuznets@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -31 X-Spam_score: -3.2 X-Spam_bar: --- X-Spam_report: (-3.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.39, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tianyu.Lan@microsoft.com, qemu-devel@nongnu.org, kvm@vger.kernel.org Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Liang Li writes: >> > Hi Vitaly, >> > >> > I found a case that the virtualization overhead was almost doubled >> > when turning on Hper-v related features compared to that without any >> > no hyper-v feature. It happens when running a 3D game in windows >> > guest in qemu kvm environment. >> > >> > By investigation, I found there are a lot of IPIs triggered by guest, >> > when turning on the hyer-v related features including stimer, for the >> > apicv is turned off, at least two vm exits are needed for processing a >> > single IPI. >> > >> > >> > perf stat will show something like below [recorded for 5 seconds] >> > >> > --------- >> > >> > Analyze events for all VMs, all VCPUs: >> > VM-EXIT Samples Samples% Time% Min Time Max >> > Time Avg time >> > EXTERNAL_INTERRUPT 471831 59.89% 68.58% 0.64us >> > 65.42us 2.34us ( +- 0.11% ) >> > MSR_WRITE 238932 30.33% 23.07% 0.48us >> > 41.05us 1.56us ( +- 0.14% ) >> > >> > Total Samples:787803, Total events handled time:1611193.84us. >> > >> > I tried turning off hyper-v for the same workload and repeat the test, >> > the overall virtualization overhead reduced by about of 50%: >> > >> > ------- >> > >> > Analyze events for all VMs, all VCPUs: >> > >> > VM-EXIT Samples Samples% Time% Min Time Max >> > Time Avg time >> > APIC_WRITE 255152 74.43% 50.72% 0.49us >> > 50.01us 1.42us ( +- 0.14% ) >> > EPT_MISCONFIG 39967 11.66% 40.58% 1.55us >> > 686.05us 7.27us ( +- 0.43% ) >> > DR_ACCESS 35003 10.21% 4.64% 0.32us >> > 40.03us 0.95us ( +- 0.32% ) >> > EXTERNAL_INTERRUPT 6622 1.93% 2.08% 0.70us >> > 57.38us 2.25us ( +- 1.42% ) >> > >> > Total Samples:342788, Total events handled time:715695.62us. >> > >> > For this scenario, hyper-v works really bad. stimer works better >> > than hpet, but on the other hand, it relies on SynIC which has >> > negative effects for IPI intensive workloads. >> > Do you have any plans for improvement? >> > >> >> Hey, >> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM >> automatically disables APICv and this can explain the overhead and the >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only >> inhibit APICv if the recommendation was ignored. This is implemented in >> the following KVM patch series: >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/ >> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it >> can be tested with 'hv-passthrough'. >> >> It would be great if you could give it a spin! >> >> -- >> Vitaly > > It's great to know that you already have a solution for this. :) > > By the way, is there any requirement for the version of windows or > windows updates for the new feature to work? AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd expect WS2008 to ignore it completely (and thus SynIC will always be disabling APICv for it). -- Vitaly