From mboxrd@z Thu Jan  1 00:00:00 1970
From: Marcelo Tosatti <mtosatti@redhat.com>
Subject: Re: [PATCH v2 3/5] KVM: Flush TLB in mmu notifier without holding
 mmu_lock
Date: Mon, 21 May 2012 17:58:50 -0300
Message-ID: <20120521205850.GA27076@amt.cnet>
References: <1337250284-18607-1-git-send-email-avi@redhat.com>
 <1337250284-18607-3-git-send-email-avi@redhat.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: Xiao Guangrong <xiaoguangrong@linux.vnet.ibm.com>,
	Takuya Yoshikawa <yoshikawa.takuya@oss.ntt.co.jp>,
	kvm@vger.kernel.org
To: Avi Kivity <avi@redhat.com>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:17276 "EHLO mx1.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1752634Ab2EUVDu (ORCPT <rfc822;kvm@vger.kernel.org>);
	Mon, 21 May 2012 17:03:50 -0400
Content-Disposition: inline
In-Reply-To: <1337250284-18607-3-git-send-email-avi@redhat.com>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>

On Thu, May 17, 2012 at 01:24:42PM +0300, Avi Kivity wrote:
> Signed-off-by: Avi Kivity <avi@redhat.com>
> ---
>  virt/kvm/kvm_main.c |   16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 585ab45..9f6d15d 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -302,11 +302,11 @@ static void kvm_mmu_notifier_invalidate_page(struct mmu_notifier *mn,
>  	kvm->mmu_notifier_seq++;
>  	if (kvm_unmap_hva(kvm, address))
>  		kvm_mark_tlb_dirty(kvm);
> -	/* we've to flush the tlb before the pages can be freed */
> -	kvm_cond_flush_remote_tlbs(kvm);
> -
>  	spin_unlock(&kvm->mmu_lock);
>  	srcu_read_unlock(&kvm->srcu, idx);
> +
> +	/* we've to flush the tlb before the pages can be freed */
> +	kvm_cond_flush_remote_tlbs(kvm);
>  }

There are still sites that assumed implicitly that acquiring mmu_lock
guarantees that sptes and remote TLBs are in sync. Example:

void kvm_mmu_zap_all(struct kvm *kvm)
{
        struct kvm_mmu_page *sp, *node;
        LIST_HEAD(invalid_list);

        spin_lock(&kvm->mmu_lock);
restart:
        list_for_each_entry_safe(sp, node, &kvm->arch.active_mmu_pages,
link)
                if (kvm_mmu_prepare_zap_page(kvm, sp, &invalid_list))
                        goto restart;

        kvm_mmu_commit_zap_page(kvm, &invalid_list);
        spin_unlock(&kvm->mmu_lock);
}

kvm_mmu_commit_zap_page only flushes if the TLB was dirtied by this
context, not before. kvm_mmu_unprotect_page is similar.

In general:

context 1                       context 2

lock(mmu_lock)
modify spte
mark_tlb_dirty
unlock(mmu_lock)
                                lock(mmu_lock)
                                read spte
                                make a decision based on spte value
                                unlock(mmu_lock)
flush remote TLBs


Is scary.

Perhaps have a rule that says:

1) Conditionally flush remote TLB after acquiring mmu_lock, 
before anything (even perhaps inside the lock macro).
2) Except special cases where it is clear that this is not 
necessary.