From mboxrd@z Thu Jan  1 00:00:00 1970
From: Marcelo Tosatti <mtosatti@redhat.com>
Subject: Re: cr3 OOS optimisation breaks 32-bit GNU/kFreeBSD guest
Date: Mon, 23 Feb 2009 11:52:08 -0300
Message-ID: <20090223145208.GA5782@amt.cnet>
References: <20090223003305.GW12976@hall.aurel32.net> <20090223014713.GA11438@amt.cnet> <20090223140115.GB5946@hall.aurel32.net>
Mime-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: QUOTED-PRINTABLE
Cc: kvm@vger.kernel.org
To: Aurelien Jarno <aurelien@aurel32.net>
Return-path: <kvm-owner@vger.kernel.org>
Received: from mx2.redhat.com ([66.187.237.31]:45819 "EHLO mx2.redhat.com"
	rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
	id S1753759AbZBWOwt (ORCPT <rfc822;kvm@vger.kernel.org>);
	Mon, 23 Feb 2009 09:52:49 -0500
Content-Disposition: inline
In-Reply-To: <20090223140115.GB5946@hall.aurel32.net>
Sender: kvm-owner@vger.kernel.org
List-ID: <kvm.vger.kernel.org>


On Mon, Feb 23, 2009 at 03:01:15PM +0100, Aurelien Jarno wrote:
> > Maybe there is a bug in the syncing code (eg: not all global pages =
are
> > sync'ed when the OS requests a global sync), or FreeBSD is "relying=
" on
> > invlpg/cr3 write to sync global pages (remember TLB entries can be
> > invalidated internally by CPU).
>=20
> As far as I understand the Intel IA32 manual, using invlpg to sync
> global pages is correct. OTOH, cr3 is not. I think that if FreeBSD is
> using cr3 to sync global pages that should also cause a problem on re=
al
> hardware sooner or later, right?

=46rom my understanding of the documentation, yes. Note:

5.1 Invalidation Instructions

The processor is always free to invalidate additional entries in the
TLBs and paging-structure caches. The following are some examples:

=E2=80=A2 INVLPG may invalidate TLB entries for pages other than the on=
e
corresponding to its linear-address operand.

=E2=80=A2 MOV to CR3 may invalidate TLB entries for global pages.
	     ^^^

=E2=80=A2 On a processor supporting Hyper-Threading Technology, invalid=
ations
performed on one logical processor may invalidate entries in the TLBs
and paging-structure caches used by other logical processors.

> I'll try to look at the kernel code.
>=20
> > If you want to debug it, would suggest looping over all MMU pages i=
n
> > mmu_sync_global, after the kvm_sync_page loop, and
> >=20
> >       WARN_ON(sp->unsync && sp->global);
> >=20
> > If that fails, check if the unsync and global flags mean what they =
are
> > supposed to.
>=20
> This doesn't detect any problem, which means that all pages marked as
> global are synced correctly.
>=20
> I have also tried to call kvm_mmu_sync_global() in kvm_set_cr3(), and
> as expected the guest works correctly in that case.
>=20
> If I understand correctly, and if the problem is not on the FreeBSD
> side, the only remaining possible problem is if normal pages are wron=
gly
> marked as global, and thus should be synced with cr3, while they are=20
> not. Am I right here?

Yes, this is the most likely problematic scenario.

> > Sorry for the trouble and thanks for the detailed report, will take=
 a
> > close look at it this week.
> >=20
>=20
> Thanks for your fast answer and for your help for debugging.

If you confirm that FreeBSD is indeed relying on cr3 to sync global
pages, it might be better to disable the optimization. Lets hope that i=
s
not the case.

Thanks