From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71178C433FE for ; Fri, 21 Oct 2022 08:06:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id DB7184B693; Fri, 21 Oct 2022 04:06:33 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 1AkLW8WsGOx8; Fri, 21 Oct 2022 04:06:32 -0400 (EDT) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 94D8F4B637; Fri, 21 Oct 2022 04:06:32 -0400 (EDT) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 984AB4B644 for ; Fri, 21 Oct 2022 04:06:31 -0400 (EDT) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id vdnMpUnhgQ5r for ; Fri, 21 Oct 2022 04:06:30 -0400 (EDT) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 26EBA4B636 for ; Fri, 21 Oct 2022 04:06:30 -0400 (EDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 261FF61DF1; Fri, 21 Oct 2022 08:06:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C73F1C433D7; Fri, 21 Oct 2022 08:06:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666339589; bh=qSm2i5Br2iJiLK3bp/4+e+tp9/qjJ9+Yqz6PbZqjpH4=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=to5fgacIEtDrjqddpAOqD6wCTwohpux3Y8ef0jeIT/Xpd6vOsslJpxZwuKqnQ7qsX riZiMyEqeJ/khrhoxjmgFvJx8yvhemzJMBbrN39xU8CwWY2oZjhMZN1oi43oun64cW A11seabrX+R9Kti1Qxqd4ksaCzi3BE73oniBe0SRN5YH+jlN2M8wZrn0iNyTgXMdn/ V9roR9jRRR32R7T8S7OzGyvrliXkvZdnPV2wIIqKz3iM/LN6k2EYSaU5a5whBA8f23 G+HDLQnIQ9RTCUQLVdhA0AL+xJooAXGAKI/rfIP54GhEt8ssF9NsHulMFZ3WfwrMc5 Bq/L0gU3037Ng== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oln2w-000Rsz-Kp; Fri, 21 Oct 2022 09:06:26 +0100 Date: Fri, 21 Oct 2022 09:06:26 +0100 Message-ID: <8635bhfvnh.wl-maz@kernel.org> From: Marc Zyngier To: Sean Christopherson Subject: Re: [PATCH v6 3/8] KVM: Add support for using dirty ring in conjunction with bitmap In-Reply-To: References: <20221011061447.131531-1-gshan@redhat.com> <20221011061447.131531-4-gshan@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: seanjc@google.com, gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, peterx@redhat.com, will@kernel.org, catalin.marinas@arm.com, bgardon@google.com, shuah@kernel.org, andrew.jones@linux.dev, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, oliver.upton@linux.dev, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: shuah@kernel.org, kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, dmatlack@google.com, shan.gavin@gmail.com, bgardon@google.com, kvmarm@lists.linux.dev, pbonzini@redhat.com, zhenyzha@redhat.com, will@kernel.org, kvmarm@lists.cs.columbia.edu X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Fri, 21 Oct 2022 00:44:51 +0100, Sean Christopherson wrote: > > On Tue, Oct 11, 2022, Gavin Shan wrote: > > Some architectures (such as arm64) need to dirty memory outside of the > > context of a vCPU. Of course, this simply doesn't fit with the UAPI of > > KVM's per-vCPU dirty ring. > > What is the point of using the dirty ring in this case? KVM still > burns a pile of memory for the bitmap. Is the benefit that > userspace can get away with scanning the bitmap fewer times, > e.g. scan it once just before blackout under the assumption that > very few pages will dirty the bitmap? Apparently, the throttling effect of the ring makes it easier to converge. Someone who actually uses the feature should be able to tell you. But that's a policy decision, and I don't see why we should be prescriptive. > Why not add a global ring to @kvm? I assume thread safety is a > problem, but the memory overhead of the dirty_bitmap also seems like > a fairly big problem. Because we already have a stupidly bloated API surface, and that we could do without yet another one based on a sample of *one*? Because dirtying memory outside of a vcpu context makes it incredibly awkward to handle a "ring full" condition? > > > Introduce a new flavor of dirty ring that requires the use of both vCPU > > dirty rings and a dirty bitmap. The expectation is that for non-vCPU > > sources of dirty memory (such as the GIC ITS on arm64), KVM writes to > > the dirty bitmap. Userspace should scan the dirty bitmap before > > migrating the VM to the target. > > > > Use an additional capability to advertize this behavior and require > > explicit opt-in to avoid breaking the existing dirty ring ABI. And yes, > > you can use this with your preferred flavor of DIRTY_RING[_ACQ_REL]. Do > > not allow userspace to enable dirty ring if it hasn't also enabled the > > ring && bitmap capability, as a VM is likely DOA without the pages > > marked in the bitmap. This is wrong. The *only* case this is useful is when there is an in-kernel producer of data outside of the context of a vcpu, which is so far only the ITS save mechanism. No ITS? No need for this. Userspace knows what it has created the first place, and should be in charge of it (i.e. I want to be able to migrate my GICv2 and GICv3-without-ITS VMs with the rings only). > > > > Suggested-by: Marc Zyngier > > Suggested-by: Peter Xu > > Co-developed-by: Oliver Upton > > Co-developed-by needs Oliver's SoB. > > > #ifndef CONFIG_HAVE_KVM_DIRTY_RING > > +static inline bool kvm_dirty_ring_exclusive(struct kvm *kvm) > > What about inverting the naming to better capture that this is about the dirty > bitmap, and less so about the dirty ring? It's not obvious what "exclusive" > means, e.g. I saw this stub before reading the changelog and assumed it was > making a dirty ring exclusive to something. > > Something like this? > > bool kvm_use_dirty_bitmap(struct kvm *kvm) > { > return !kvm->dirty_ring_size || kvm->dirty_ring_with_bitmap; > } > > > @@ -3305,15 +3305,20 @@ void mark_page_dirty_in_slot(struct kvm *kvm, > > struct kvm_vcpu *vcpu = kvm_get_running_vcpu(); > > > > #ifdef CONFIG_HAVE_KVM_DIRTY_RING > > - if (WARN_ON_ONCE(!vcpu) || WARN_ON_ONCE(vcpu->kvm != kvm)) > > + if (WARN_ON_ONCE(vcpu && vcpu->kvm != kvm)) > > return; > > + > > +#ifndef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + if (WARN_ON_ONCE(!vcpu)) > > To cut down on the #ifdefs, this can be: > > if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP) && !vcpu) > > though that's arguably even harder to read. Blech. > > > + return; > > +#endif > > #endif > > > > if (memslot && kvm_slot_dirty_track_enabled(memslot)) { > > unsigned long rel_gfn = gfn - memslot->base_gfn; > > u32 slot = (memslot->as_id << 16) | memslot->id; > > > > - if (kvm->dirty_ring_size) > > + if (vcpu && kvm->dirty_ring_size) > > kvm_dirty_ring_push(&vcpu->dirty_ring, > > slot, rel_gfn); > > else > > @@ -4485,6 +4490,9 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) > > return KVM_DIRTY_RING_MAX_ENTRIES * sizeof(struct kvm_dirty_gfn); > > #else > > return 0; > > +#endif > > +#ifdef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + case KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP: > > #endif > > case KVM_CAP_BINARY_STATS_FD: > > case KVM_CAP_SYSTEM_EVENT_DATA: > > @@ -4499,6 +4507,11 @@ static int kvm_vm_ioctl_enable_dirty_log_ring(struct kvm *kvm, u32 size) > > { > > int r; > > > > +#ifdef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + if (!kvm->dirty_ring_with_bitmap) > > + return -EINVAL; > > +#endif > > This one at least is prettier with IS_ENABLED > > if (IS_ENABLED(CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP) && > !kvm->dirty_ring_with_bitmap) > return -EINVAL; > > But dirty_ring_with_bitmap really shouldn't need to exist. It's > mandatory for architectures that have > HAVE_KVM_DIRTY_RING_WITH_BITMAP, and unsupported for architectures > that don't. In other words, the API for enabling the dirty ring is > a bit ugly. > > Rather than add KVM_CAP_DIRTY_LOG_RING_ACQ_REL, which hasn't been > officially released yet, and then KVM_CAP_DIRTY_LOG_ING_WITH_BITMAP > on top, what about usurping bits 63:32 of cap->args[0] for flags? > E.g. > > Ideally we'd use cap->flags directly, but we screwed up with > KVM_CAP_DIRTY_LOG_RING and didn't require flags to be zero :-( > > Actually, what's the point of allowing > KVM_CAP_DIRTY_LOG_RING_ACQ_REL to be enabled? I get why KVM would > enumerate this info, i.e. allowing checking, but I don't seen any > value in supporting a second method for enabling the dirty ring. > > The acquire-release thing is irrelevant for x86, and no other > architecture supports the dirty ring until this series, i.e. there's > no need for KVM to detect that userspace has been updated to gain > acquire-release semantics, because the fact that userspace is > enabling the dirty ring on arm64 means userspace has been updated. Do we really need to make the API more awkward? There is an established pattern of "enable what is advertised". Some level of uniformity wouldn't hurt, really. M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 38EC68F59 for ; Fri, 21 Oct 2022 08:06:29 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C73F1C433D7; Fri, 21 Oct 2022 08:06:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1666339589; bh=qSm2i5Br2iJiLK3bp/4+e+tp9/qjJ9+Yqz6PbZqjpH4=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=to5fgacIEtDrjqddpAOqD6wCTwohpux3Y8ef0jeIT/Xpd6vOsslJpxZwuKqnQ7qsX riZiMyEqeJ/khrhoxjmgFvJx8yvhemzJMBbrN39xU8CwWY2oZjhMZN1oi43oun64cW A11seabrX+R9Kti1Qxqd4ksaCzi3BE73oniBe0SRN5YH+jlN2M8wZrn0iNyTgXMdn/ V9roR9jRRR32R7T8S7OzGyvrliXkvZdnPV2wIIqKz3iM/LN6k2EYSaU5a5whBA8f23 G+HDLQnIQ9RTCUQLVdhA0AL+xJooAXGAKI/rfIP54GhEt8ssF9NsHulMFZ3WfwrMc5 Bq/L0gU3037Ng== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oln2w-000Rsz-Kp; Fri, 21 Oct 2022 09:06:26 +0100 Date: Fri, 21 Oct 2022 09:06:26 +0100 Message-ID: <8635bhfvnh.wl-maz@kernel.org> From: Marc Zyngier To: Sean Christopherson Cc: Gavin Shan , kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, peterx@redhat.com, will@kernel.org, catalin.marinas@arm.com, bgardon@google.com, shuah@kernel.org, andrew.jones@linux.dev, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, oliver.upton@linux.dev, shan.gavin@gmail.com Subject: Re: [PATCH v6 3/8] KVM: Add support for using dirty ring in conjunction with bitmap In-Reply-To: References: <20221011061447.131531-1-gshan@redhat.com> <20221011061447.131531-4-gshan@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: seanjc@google.com, gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, peterx@redhat.com, will@kernel.org, catalin.marinas@arm.com, bgardon@google.com, shuah@kernel.org, andrew.jones@linux.dev, dmatlack@google.com, pbonzini@redhat.com, zhenyzha@redhat.com, james.morse@arm.com, suzuki.poulose@arm.com, alexandru.elisei@arm.com, oliver.upton@linux.dev, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Message-ID: <20221021080626.BCCT6cClhz76Jf7QBEjqL1QWHhBttAaDfhW30PVWgy0@z> On Fri, 21 Oct 2022 00:44:51 +0100, Sean Christopherson wrote: > > On Tue, Oct 11, 2022, Gavin Shan wrote: > > Some architectures (such as arm64) need to dirty memory outside of the > > context of a vCPU. Of course, this simply doesn't fit with the UAPI of > > KVM's per-vCPU dirty ring. > > What is the point of using the dirty ring in this case? KVM still > burns a pile of memory for the bitmap. Is the benefit that > userspace can get away with scanning the bitmap fewer times, > e.g. scan it once just before blackout under the assumption that > very few pages will dirty the bitmap? Apparently, the throttling effect of the ring makes it easier to converge. Someone who actually uses the feature should be able to tell you. But that's a policy decision, and I don't see why we should be prescriptive. > Why not add a global ring to @kvm? I assume thread safety is a > problem, but the memory overhead of the dirty_bitmap also seems like > a fairly big problem. Because we already have a stupidly bloated API surface, and that we could do without yet another one based on a sample of *one*? Because dirtying memory outside of a vcpu context makes it incredibly awkward to handle a "ring full" condition? > > > Introduce a new flavor of dirty ring that requires the use of both vCPU > > dirty rings and a dirty bitmap. The expectation is that for non-vCPU > > sources of dirty memory (such as the GIC ITS on arm64), KVM writes to > > the dirty bitmap. Userspace should scan the dirty bitmap before > > migrating the VM to the target. > > > > Use an additional capability to advertize this behavior and require > > explicit opt-in to avoid breaking the existing dirty ring ABI. And yes, > > you can use this with your preferred flavor of DIRTY_RING[_ACQ_REL]. Do > > not allow userspace to enable dirty ring if it hasn't also enabled the > > ring && bitmap capability, as a VM is likely DOA without the pages > > marked in the bitmap. This is wrong. The *only* case this is useful is when there is an in-kernel producer of data outside of the context of a vcpu, which is so far only the ITS save mechanism. No ITS? No need for this. Userspace knows what it has created the first place, and should be in charge of it (i.e. I want to be able to migrate my GICv2 and GICv3-without-ITS VMs with the rings only). > > > > Suggested-by: Marc Zyngier > > Suggested-by: Peter Xu > > Co-developed-by: Oliver Upton > > Co-developed-by needs Oliver's SoB. > > > #ifndef CONFIG_HAVE_KVM_DIRTY_RING > > +static inline bool kvm_dirty_ring_exclusive(struct kvm *kvm) > > What about inverting the naming to better capture that this is about the dirty > bitmap, and less so about the dirty ring? It's not obvious what "exclusive" > means, e.g. I saw this stub before reading the changelog and assumed it was > making a dirty ring exclusive to something. > > Something like this? > > bool kvm_use_dirty_bitmap(struct kvm *kvm) > { > return !kvm->dirty_ring_size || kvm->dirty_ring_with_bitmap; > } > > > @@ -3305,15 +3305,20 @@ void mark_page_dirty_in_slot(struct kvm *kvm, > > struct kvm_vcpu *vcpu = kvm_get_running_vcpu(); > > > > #ifdef CONFIG_HAVE_KVM_DIRTY_RING > > - if (WARN_ON_ONCE(!vcpu) || WARN_ON_ONCE(vcpu->kvm != kvm)) > > + if (WARN_ON_ONCE(vcpu && vcpu->kvm != kvm)) > > return; > > + > > +#ifndef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + if (WARN_ON_ONCE(!vcpu)) > > To cut down on the #ifdefs, this can be: > > if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP) && !vcpu) > > though that's arguably even harder to read. Blech. > > > + return; > > +#endif > > #endif > > > > if (memslot && kvm_slot_dirty_track_enabled(memslot)) { > > unsigned long rel_gfn = gfn - memslot->base_gfn; > > u32 slot = (memslot->as_id << 16) | memslot->id; > > > > - if (kvm->dirty_ring_size) > > + if (vcpu && kvm->dirty_ring_size) > > kvm_dirty_ring_push(&vcpu->dirty_ring, > > slot, rel_gfn); > > else > > @@ -4485,6 +4490,9 @@ static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg) > > return KVM_DIRTY_RING_MAX_ENTRIES * sizeof(struct kvm_dirty_gfn); > > #else > > return 0; > > +#endif > > +#ifdef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + case KVM_CAP_DIRTY_LOG_RING_WITH_BITMAP: > > #endif > > case KVM_CAP_BINARY_STATS_FD: > > case KVM_CAP_SYSTEM_EVENT_DATA: > > @@ -4499,6 +4507,11 @@ static int kvm_vm_ioctl_enable_dirty_log_ring(struct kvm *kvm, u32 size) > > { > > int r; > > > > +#ifdef CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP > > + if (!kvm->dirty_ring_with_bitmap) > > + return -EINVAL; > > +#endif > > This one at least is prettier with IS_ENABLED > > if (IS_ENABLED(CONFIG_HAVE_KVM_DIRTY_RING_WITH_BITMAP) && > !kvm->dirty_ring_with_bitmap) > return -EINVAL; > > But dirty_ring_with_bitmap really shouldn't need to exist. It's > mandatory for architectures that have > HAVE_KVM_DIRTY_RING_WITH_BITMAP, and unsupported for architectures > that don't. In other words, the API for enabling the dirty ring is > a bit ugly. > > Rather than add KVM_CAP_DIRTY_LOG_RING_ACQ_REL, which hasn't been > officially released yet, and then KVM_CAP_DIRTY_LOG_ING_WITH_BITMAP > on top, what about usurping bits 63:32 of cap->args[0] for flags? > E.g. > > Ideally we'd use cap->flags directly, but we screwed up with > KVM_CAP_DIRTY_LOG_RING and didn't require flags to be zero :-( > > Actually, what's the point of allowing > KVM_CAP_DIRTY_LOG_RING_ACQ_REL to be enabled? I get why KVM would > enumerate this info, i.e. allowing checking, but I don't seen any > value in supporting a second method for enabling the dirty ring. > > The acquire-release thing is irrelevant for x86, and no other > architecture supports the dirty ring until this series, i.e. there's > no need for KVM to detect that userspace has been updated to gain > acquire-release semantics, because the fact that userspace is > enabling the dirty ring on arm64 means userspace has been updated. Do we really need to make the API more awkward? There is an established pattern of "enable what is advertised". Some level of uniformity wouldn't hurt, really. M. -- Without deviation from the norm, progress is not possible.