From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 043EEECAAD2 for ; Fri, 2 Sep 2022 00:21:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:Content-Type: Content-Transfer-Encoding:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:In-Reply-To:Subject:From:References:Cc:To: MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XtJFmBTnUxHrEgXwA1bHcLne3sdPjp9UmUCMZ1DTRL0=; b=KgD/T7Qlj+Etzx jeKUwX5i/R/a5EtSMlhQd1d9SuzantU0xAo10hcsnEXJTbnxFsdcSub4shDWsEjz/tEv82VE7lCFf B/msbUCg3Oqb36P7KvN/8WS8/0PPgcCyfBs8qQ8wtDKA7oPSEw5tj5s6gHcz+MTXAMX/6djJ20zQy vxYP13WGJx7wGY3mjZwNeGkrkNObYpB6r/c8fVMmMEVLYz6VCrgFYDrrnjKzrv7rZzwPyi8lg1wgt mRacO+3TJ381stiKfCEJRlTVtRO/c1HJ1wz1CrEmRm9EFqknzyFB12KBeoVUS7di3+4oiapxvY/s1 u4bfXouUWVnvKqsF4niA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTuPe-00G7aH-Mz; Fri, 02 Sep 2022 00:19:59 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oTuPX-00G7YP-Ma for linux-arm-kernel@lists.infradead.org; Fri, 02 Sep 2022 00:19:54 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1662077990; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dO2Mn5n5WZ/MoIF+K7eeDvmnGSBHo08xkcQ+P3b4bKQ=; b=BdKTBxvozNixSzd2DnYAHMSbF/kQq49nkYRu66226UvaHF7tGUKyclZmQhk96X+PfkarJu CJSJkO3VhTVZAWIRjtzJAa+9VbtoIFjIjHSBFJzv8MyWo6qpGym0XOz2wE9oNh5LQ9EuKN ls06qT2NQakTUd4/8/B8AqrABdCRseo= Received: from mail-ej1-f72.google.com (mail-ej1-f72.google.com [209.85.218.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-386-aXZBhwaOPTi-RnQuboGfIA-1; Thu, 01 Sep 2022 20:19:49 -0400 X-MC-Unique: aXZBhwaOPTi-RnQuboGfIA-1 Received: by mail-ej1-f72.google.com with SMTP id jg40-20020a170907972800b0074148cdc7baso167976ejc.12 for ; Thu, 01 Sep 2022 17:19:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:subject:from:references:cc:to :content-language:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date; bh=dO2Mn5n5WZ/MoIF+K7eeDvmnGSBHo08xkcQ+P3b4bKQ=; b=xIWex4B9J4LYauP1RhbRh/vQ4g8603cFOk4vekSJ4PEwSArORWcUpgQ7STGAGdvt8x 3KedtFw/KD8oTsSOeOGiLImkPuiL1gmRMnsKETXbytQmjUZAdBOeq6hwbmyJAMAoByCO uOzPTIGE9TQMrYriJlcdfJn2A3UV+ACp/EgMHp82BSM1xeALfNff1mIj/05A9+T5QRQv r0FilGPKAiMasgXNWJoBdWyHrCW03HszU/WsbhJnVuNregeuG+2cbvstHaa5EqnZ6yCm UveSF2gaPy4bHuvqwPvQxI78wMZFZ96aUA58SNmebI3T+2Ewu2RFLkkB2EotAs4mpfJf wUJA== X-Gm-Message-State: ACgBeo2ytTWJiYROY8Gy0SqBOUioQSHDiAbOeacNfVo2knxDBMA8lGGY 5NPC/aE384zq4zaeUrD8BtmduZKI6GoFWd77EnCgZQL9SZEDEXIXG6MytLV5a1cCuhxTBoC2MtE pfE7GG1RUVq8fMmLyvpImMu8/GmpKXuRB95o= X-Received: by 2002:a17:906:4795:b0:73d:d6e8:52a7 with SMTP id cw21-20020a170906479500b0073dd6e852a7mr23729574ejc.59.1662077988845; Thu, 01 Sep 2022 17:19:48 -0700 (PDT) X-Google-Smtp-Source: AA6agR71etqoXM4HGXVz/MwtlqW3T8BbNkFPT5a/e9qOoDZWnrG77GI6lD81PpRdFOR9EqVhGDa/oA== X-Received: by 2002:a17:906:4795:b0:73d:d6e8:52a7 with SMTP id cw21-20020a170906479500b0073dd6e852a7mr23729550ejc.59.1662077988617; Thu, 01 Sep 2022 17:19:48 -0700 (PDT) Received: from ?IPV6:2001:b07:6468:f312:9af8:e5f5:7516:fa89? ([2001:b07:6468:f312:9af8:e5f5:7516:fa89]) by smtp.googlemail.com with ESMTPSA id lb14-20020a170907784e00b00741a0c3f4cdsm352041ejc.189.2022.09.01.17.19.47 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 01 Sep 2022 17:19:48 -0700 (PDT) Message-ID: <44a42d03-4dd1-3f1c-3a60-7c2a6a7d417a@redhat.com> Date: Fri, 2 Sep 2022 02:19:46 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.2.0 To: Peter Xu , Marc Zyngier Cc: Oliver Upton , Gavin Shan , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, corbet@lwn.net, james.morse@arm.com, alexandru.elisei@arm.com, suzuki.poulose@arm.com, catalin.marinas@arm.com, will@kernel.org, shuah@kernel.org, seanjc@google.com, drjones@redhat.com, dmatlack@google.com, bgardon@google.com, ricarkol@google.com, zhenyzha@redhat.com, shan.gavin@gmail.com References: <20220819005601.198436-1-gshan@redhat.com> <20220819005601.198436-2-gshan@redhat.com> <87lerkwtm5.wl-maz@kernel.org> <41fb5a1f-29a9-e6bb-9fab-4c83a2a8fce5@redhat.com> <87fshovtu0.wl-maz@kernel.org> <87a67uwve8.wl-maz@kernel.org> <99364855-b4e9-8a69-e1ca-ed09d103e4c8@redhat.com> <874jxzvxak.wl-maz@kernel.org> From: Paolo Bonzini Subject: Re: [PATCH v1 1/5] KVM: arm64: Enable ring-based dirty memory tracking In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220901_171951_876349_5764D676 X-CRM114-Status: GOOD ( 19.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii"; Format="flowed" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 8/30/22 16:42, Peter Xu wrote: > Marc, > > I thought we won't hit this as long as we properly take care of other > orderings of (a) gfn push, and (b) gfn collect, but after a second thought > I think it's indeed logically possible that with a reversed ordering here > we can be reading some garbage gfn before (a) happens butt also read the > valid flag after (b). > > It seems we must have all the barriers correctly applied always. If that's > correct, do you perhaps mean something like this to just add the last piece > of barrier? Okay, so I thought about it some more and it's quite tricky. Strictly speaking, the synchronization is just between userspace and kernel. The fact that the actual producer of dirty pages is in another CPU is a red herring, because reset only cares about harvested pages. In other words, the dirty page ring is essentially two ring buffers in one and we only care about the "harvested ring", not the "produced ring". On the other hand, it may happen that userspace has set more RESET flags while the ioctl is ongoing: CPU0 CPU1 CPU2 fill gfn0 store-rel flags for gfn0 fill gfn1 store-rel flags for gfn1 load-acq flags for gfn0 set RESET for gfn0 load-acq flags for gfn1 set RESET for gfn1 do ioctl! -----------> ioctl(RESET_RINGS) fill gfn2 store-rel flags for gfn2 load-acq flags for gfn2 set RESET for gfn2 process gfn0 process gfn1 process gfn2 do ioctl! etc. The three load-acquire in CPU0 synchronize with the three store-release in CPU2, but CPU0 and CPU1 are only synchronized up to gfn1 and CPU1 may miss gfn2's fields other than flags. The kernel must be able to cope with invalid values of the fields, and userspace will invoke the ioctl once more. However, once the RESET flag is cleared on gfn2, it is lost forever, therefore in the above scenario CPU1 must read the correct value of gfn2's fields. Therefore RESET must be set with a store-release, that will synchronize with a load-acquire in CPU1 as you suggested. Paolo > diff --git a/virt/kvm/dirty_ring.c b/virt/kvm/dirty_ring.c > index f4c2a6eb1666..ea620bfb012d 100644 > --- a/virt/kvm/dirty_ring.c > +++ b/virt/kvm/dirty_ring.c > @@ -84,7 +84,7 @@ static inline void kvm_dirty_gfn_set_dirtied(struct kvm_dirty_gfn *gfn) > > static inline bool kvm_dirty_gfn_harvested(struct kvm_dirty_gfn *gfn) > { > - return gfn->flags & KVM_DIRTY_GFN_F_RESET; > + return smp_load_acquire(&gfn->flags) & KVM_DIRTY_GFN_F_RESET; > } > > int kvm_dirty_ring_reset(struct kvm *kvm, struct kvm_dirty_ring *ring) > ===8<=== > > Thanks, > > -- > Peter Xu > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel