Re: 2.6.33-rc2: Xen/Guest switching to user mode with no user page tables

[Cyrill Gorcunov <gorcunov@gmail.com>]

On Sun, Jan 10, 2010 at 12:59:03PM +0000, Ian Campbell wrote:
> On Sun, 2010-01-10 at 11:09 +0300, Cyrill Gorcunov wrote:
> > On Sat, Jan 09, 2010 at 08:50:04PM -0500, Brian Gerst wrote:
> > ...
> > > > ---
> > > > x86: kernel_thread -- initialize SS to a known state
> > > >
> > > > Before the kernel_thread was converted into "C" we had
> > > > pt_regs::ss set to __KERNEL_DS (by SAVE_ALL asm macro).
> > > >
> > > > Though I must admit I didn't find any *explicit* load of
> > > > %ss from this structure the better to be on a safe side
> > > > and set it to a known value.
> > > 
> > > It shouldn't make any difference, but maybe Xen is doing something
> > > subtle.  In 64-bit mode the %ss segment register is supposed to be
> > > ignored, which is why it is left set to zero.  It works properly on
> > > real hardware.  It can't hurt anything to put __KERNEL_DS back in, but
> > > I'd just like to know why Xen requires it if this does fix it.
> > 
> > Yeah, I didn't found any explicit %ss reloading for this _particular_
> > case (as I marked in patch changelog). So the only suspicious is Xen
> > itself. So as only Christian get ability to test -- we will see the
> > results.
> 
> The difference with Xen is that it must squash the RPL of SS (to 3 for
> 64 bit and 1 for 32 bit, 32 bit doesn't matter here though). Perhaps a
> NULL selector can only have RPL==0? (I'm away from my architecture docs
> so I can't check). In any case specifying a non-NULL SS selector allows
> the squashing to occur correctly.
> 
> However this is not the cause of the original "Guest switching to user
> mode with no user page tables" error. This is down to 
>         commit f443ff4201dd25cd4dec183f9919ecba90c8edc2
>         Author: Brian Gerst <brgerst@gmail.com>
>         Date:   Wed Dec 9 12:34:43 2009 -0500
>         
>             x86: Sync 32/64-bit kernel_thread
>             
>             Signed-off-by: Brian Gerst <brgerst@gmail.com>
>             LKML-Reference: <1260380084-3707-5-git-send-email-brgerst@gmail.com>
>             Signed-off-by: H. Peter Anvin <hpa@zytor.com>
> which on 64 bit resulted in changing regs.cs from "__KERNEL_CS" to
> "__KERNEL_CS | get_kernel_rpl()". The later seems more logical (and is
> correct for 32 bit) but on 64 bit we frequently use a pattern like "cmpl
> $3, CS(%rsp); je foo" quite a bit to detect return to user vs kernel and
> an RPL of 1 will unfortunately incorrectly trigger the return to
> userspace paths.
> 
> The correct fix is for the Xen backend to declare kernel RPL == 0 for 64
> bit guests -- the hyervisor already takes care of all the necessary
> squashing to ring 3 transparently (because making the guest worry about
> it would break the very common assumption that you can distinguish user
> from kernel CS by RPL).
> 
> With just the CS RPL fix below I see a GPF at kernel_thread_helper with
> SS=3 (hence my hypothesis about NULL selectors and non-zero RPL above).
> With both the SS and CS fixes things work fine.

any of CS,SS loaded with NULL descriptor should lead to #GP

> 
> Ian.
> 
> --- 
> Subject: xen: 64 bit kernel RPL should be 0.
> 
...

Good catch Ian! I've noted that Xen use it's own get_kernel_rpl
while discussing this problem in a chat. But I must admit *I simply don't know*
what Xen does, or how it works internally (neither I have will to learn it at
moment :)

That said -- I'm happy if yor patch fixes problem (and it looks that
get_kernel_rpl is guilty here indeed).

	-- Cyrill