From mboxrd@z Thu Jan  1 00:00:00 1970
From: Jeremy Fitzhardinge <jeremy@goop.org>
Subject: Re: RE: rdtsc: correctness vs performance on Xen  (and
	KVM?)
Date: Tue, 01 Sep 2009 15:21:36 -0700
Message-ID: <4A9D9E70.8000905@goop.org>
References: <b5efa656-50f2-452d-89b3-2632afabadfc@default>
Mime-Version: 1.0
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 7bit
Return-path: <xen-devel-bounces@lists.xensource.com>
In-Reply-To: <b5efa656-50f2-452d-89b3-2632afabadfc@default>
List-Unsubscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=unsubscribe>
List-Post: <mailto:xen-devel@lists.xensource.com>
List-Help: <mailto:xen-devel-request@lists.xensource.com?subject=help>
List-Subscribe: <http://lists.xensource.com/mailman/listinfo/xen-devel>,
	<mailto:xen-devel-request@lists.xensource.com?subject=subscribe>
Sender: xen-devel-bounces@lists.xensource.com
Errors-To: xen-devel-bounces@lists.xensource.com
To: Dan Magenheimer <dan.magenheimer@oracle.com>
Cc: Alan Cox <alan@lxorguk.ukuu.org.uk>, "Xen-Devel (E-mail)" <xen-devel@lists.xensource.com>, Keir Fraser <keir.fraser@eu.citrix.com>, Jan Beulich <JBeulich@novell.com>
List-Id: xen-devel@lists.xenproject.org

On 09/01/09 15:08, Dan Magenheimer wrote:
>>> Just like what is being used to allow apps to get the CPU 
>>>       
>> number on native
>>     
>>> kernels (or the vCPU one on Xen-ified ones): Have a GDT 
>>>       
>> entry the limit of
>>     
>>> which is the number you want, and have the app use the lsl 
>>>       
>> instruction to
>>     
>>> get at it.
>>>       
>> Yes, that's true. Xen could provide such a segment descriptor 
>> in its private
>> area of the GDT. The issue then would be that, in a compound pvclock
>> operation spanning multiple machine instructions, the pCPU 
>> number revealed
>> by the LSL instruction can be stale by the time it is used 
>> later in the
>> compound operation.
>>     
> The algorithm could check the pCPU number before and after
> reading the pvclock data and doing the rdtsc, and if they
> don't match, start again.  (Doesn't the pvclock algorithm
> already do that with some versioning number in the pvclock
> data itself to ensure that the rest of the data didn't
> change while it was being read?)
>   
There's still a race there, if the thread switched PCPU twice during the
operation:

    <running on PCPU A>
    get CPU #
    <switch to PCPU B>
    read tsc
    apply corrections from (from PCPU A)
    <switch to PCPU A>
    check CPU # is the same as we started with: all OK!

note that the <switch to PCPU B> could either be a result of the Xen
scheduler moving the VCPU *or* the Linux scheduler moving the thread to
a different VCPU.  In the former case, Xen could update a version
counter to help detect the discontinuity, but it doesn't really know
about guest scheduling decisions.  I guess the guest kernel could update
the pvclock version counter itself.

> I'm clueless about GDTs and the LSL instrution so would
> need some help prototyping this.
>   

It's what vsyscall already uses.  Your scheme is precisely analogous to
what's already there.

    J