[PATCH 1/3] xen: setup percpu data pointers

[PATCH 1/3] xen: setup percpu data pointers

[Jeremy Fitzhardinge]

We need to access percpu data fairly early, so set up the percpu
registers as soon as possible.  We only need to load the appropriate
segment register.  We already have a GDT, but its hard to change it
early because we need to manipulate the pagetable to do so, and that
hasn't been set up yet.

Also, set the kernel stack when bringing up secondary CPUs.  If we
don't they all end up sharing the same stack...

[This depends on the two other percpu patches that Tejun has in his tree. ]

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 arch/x86/xen/enlighten.c |   16 ++++++++++++++++
 arch/x86/xen/smp.c       |    4 +++-
 arch/x86/xen/xen-ops.h   |    2 ++
 3 files changed, 21 insertions(+), 1 deletion(-)

diff -r 9f052779cc5d arch/x86/xen/enlighten.c
--- a/arch/x86/xen/enlighten.c	Fri Jan 30 17:20:43 2009 -0800
+++ b/arch/x86/xen/enlighten.c	Fri Jan 30 17:37:05 2009 -0800
@@ -61,6 +61,8 @@
 enum xen_domain_type xen_domain_type = XEN_NATIVE;
 EXPORT_SYMBOL_GPL(xen_domain_type);
 
+void *xen_initial_gdt;
+
 /*
  * Identity map, in addition to plain kernel map.  This needs to be
  * large enough to allocate page table pages to allocate the rest.
@@ -1646,6 +1648,20 @@
 	have_vcpu_info_placement = 0;
 #endif
 
+#ifdef CONFIG_X86_64
+	/*
+	 * Setup percpu state.  We only need to do this for 64-bit
+         * because 32-bit already has %fs set properly.
+	 */
+	load_percpu_segment(0);
+#endif
+	/*
+	 * The only reliable way to retain the initial address of the
+	 * percpu gdt_page is to remember it here, so we can go and
+	 * mark it RW later, when the initial percpu area is freed.
+	 */
+	xen_initial_gdt = &per_cpu(gdt_page, 0);
+
 	xen_smp_init();
 
 	/* Get mfn list */
diff -r 9f052779cc5d arch/x86/xen/smp.c
--- a/arch/x86/xen/smp.c	Fri Jan 30 17:20:43 2009 -0800
+++ b/arch/x86/xen/smp.c	Fri Jan 30 17:37:05 2009 -0800
@@ -170,7 +170,7 @@
 
 	/* We've switched to the "real" per-cpu gdt, so make sure the
 	   old memory can be recycled */
-	make_lowmem_page_readwrite(&per_cpu_var(gdt_page));
+	make_lowmem_page_readwrite(xen_initial_gdt);
 
 	xen_setup_vcpu_info_placement();
 }
@@ -235,6 +235,8 @@
 	ctxt->user_regs.ss = __KERNEL_DS;
 #ifdef CONFIG_X86_32
 	ctxt->user_regs.fs = __KERNEL_PERCPU;
+#else
+	ctxt->gs_base_kernel = per_cpu_offset(cpu);
 #endif
 	ctxt->user_regs.eip = (unsigned long)cpu_bringup_and_idle;
 	ctxt->user_regs.eflags = 0x1000; /* IOPL_RING1 */
@@ -284,6 +286,9 @@
 	irq_ctx_init(cpu);
 #else
 	clear_tsk_thread_flag(idle, TIF_FORK);
+	per_cpu(kernel_stack, cpu) =
+		(unsigned long)task_stack_page(idle) -
+		KERNEL_STACK_OFFSET + THREAD_SIZE;
 #endif
 	xen_setup_timer(cpu);
 	xen_init_lock_cpu(cpu);
diff -r 9f052779cc5d arch/x86/xen/xen-ops.h
--- a/arch/x86/xen/xen-ops.h	Fri Jan 30 17:20:43 2009 -0800
+++ b/arch/x86/xen/xen-ops.h	Fri Jan 30 17:37:05 2009 -0800
@@ -10,6 +10,8 @@
 extern const char xen_hypervisor_callback[];
 extern const char xen_failsafe_callback[];
 
+extern void *xen_initial_gdt;
+
 struct trap_info;
 void xen_copy_trap_info(struct trap_info *traps);
 

[PATCH 1/3] xen: setup percpu data pointers

[Jeremy Fitzhardinge]

We need to access percpu data fairly early, so set up the percpu
registers as soon as possible.  We only need to load the appropriate
segment register.  We already have a GDT, but its hard to change it
early because we need to manipulate the pagetable to do so, and that
hasn't been set up yet.

Also, set the kernel stack when bringing up secondary CPUs.  If we
don't they all end up sharing the same stack...

Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
---
 arch/x86/xen/enlighten.c |   16 ++++++++++++++++
 arch/x86/xen/smp.c       |    7 ++++++-
 arch/x86/xen/xen-ops.h   |    2 ++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff -r 42a1fb2574f0 arch/x86/xen/enlighten.c
--- a/arch/x86/xen/enlighten.c	Mon Feb 02 12:44:32 2009 -0800
+++ b/arch/x86/xen/enlighten.c	Mon Feb 02 13:24:32 2009 -0800
@@ -66,6 +66,8 @@
 
 struct shared_info xen_dummy_shared_info;
 
+void *xen_initial_gdt;
+
 /*
  * Point at some empty memory to start with. We map the real shared_info
  * page as soon as fixmap is up and running.
@@ -917,8 +919,19 @@
 	have_vcpu_info_placement = 0;
 #endif
 
-	/* setup percpu state */
+#ifdef CONFIG_X86_64
+	/*
+	 * Setup percpu state.  We only need to do this for 64-bit
+	 * because 32-bit already has %fs set properly.
+	 */
 	load_percpu_segment(0);
+#endif
+	/*
+	 * The only reliable way to retain the initial address of the
+	 * percpu gdt_page is to remember it here, so we can go and
+	 * mark it RW later, when the initial percpu area is freed.
+	 */
+	xen_initial_gdt = &per_cpu(gdt_page, 0);
 
 	xen_smp_init();
 
diff -r 42a1fb2574f0 arch/x86/xen/smp.c
--- a/arch/x86/xen/smp.c	Mon Feb 02 12:44:32 2009 -0800
+++ b/arch/x86/xen/smp.c	Mon Feb 02 13:24:32 2009 -0800
@@ -170,8 +170,7 @@
 
 	/* We've switched to the "real" per-cpu gdt, so make sure the
 	   old memory can be recycled */
-	make_lowmem_page_readwrite(__per_cpu_load +
-				   (unsigned long)&per_cpu_var(gdt_page));
+	make_lowmem_page_readwrite(xen_initial_gdt);
 
 	xen_setup_vcpu_info_placement();
 }
@@ -287,6 +286,9 @@
 	irq_ctx_init(cpu);
 #else
 	clear_tsk_thread_flag(idle, TIF_FORK);
+	per_cpu(kernel_stack, cpu) =
+		(unsigned long)task_stack_page(idle) -
+		KERNEL_STACK_OFFSET + THREAD_SIZE;
 #endif
 	xen_setup_timer(cpu);
 	xen_init_lock_cpu(cpu);
diff -r 42a1fb2574f0 arch/x86/xen/xen-ops.h
--- a/arch/x86/xen/xen-ops.h	Mon Feb 02 12:44:32 2009 -0800
+++ b/arch/x86/xen/xen-ops.h	Mon Feb 02 13:24:32 2009 -0800
@@ -10,6 +10,8 @@
 extern const char xen_hypervisor_callback[];
 extern const char xen_failsafe_callback[];
 
+extern void *xen_initial_gdt;
+
 struct trap_info;
 void xen_copy_trap_info(struct trap_info *traps);
 

Re: [PATCH 1/3] xen: setup percpu data pointers

[Tejun Heo]

Jeremy Fitzhardinge wrote:
> We need to access percpu data fairly early, so set up the percpu
> registers as soon as possible.  We only need to load the appropriate
> segment register.  We already have a GDT, but its hard to change it
> early because we need to manipulate the pagetable to do so, and that
> hasn't been set up yet.
> 
> Also, set the kernel stack when bringing up secondary CPUs.  If we
> don't they all end up sharing the same stack...
> 
> Signed-off-by: Jeremy Fitzhardinge <jeremy.fitzhardinge@citrix.com>
> ---
> arch/x86/xen/enlighten.c |   16 ++++++++++++++++
> arch/x86/xen/smp.c       |    7 ++++++-
> arch/x86/xen/xen-ops.h   |    2 ++
> 3 files changed, 24 insertions(+), 1 deletion(-)

Applied 1-3 to #tj-percpu with trivial modification on #2 (will reply
to the message).

Thanks.

-- 
tejun

Re: [PATCH 1/3] xen: setup percpu data pointers

[Tejun Heo]

Tejun Heo wrote:
> Applied 1-3 to #tj-percpu with trivial modification on #2 (will reply
> to the message).

Oh... hpa already merged them into x86/paravirt.  I'll just pull
x86/paravirt into core/percpu because the first patch needs to be in
core/percpu to keep xen working.

hpa, can you please take the other xen patch[1] into x86/paravirt too?
So that I can receive it via you?

Thanks.

-- 
tejun

[1] [PATCH] xen: fix 32-bit build resulting from mmu move

Re: [PATCH 1/3] xen: setup percpu data pointers

[Jeremy Fitzhardinge]

Tejun Heo wrote:
> Tejun Heo wrote:
>   
>> Applied 1-3 to #tj-percpu with trivial modification on #2 (will reply
>> to the message).
>>     
>
> Oh... hpa already merged them into x86/paravirt.  I'll just pull
> x86/paravirt into core/percpu because the first patch needs to be in
> core/percpu to keep xen working.
>
> hpa, can you please take the other xen patch[1] into x86/paravirt too?
> So that I can receive it via you?

Looks like its already there.

    J

Re: [PATCH 1/3] xen: setup percpu data pointers

[Tejun Heo]

Jeremy Fitzhardinge wrote:
> Tejun Heo wrote:
>> Tejun Heo wrote:
>>  
>>> Applied 1-3 to #tj-percpu with trivial modification on #2 (will reply
>>> to the message).
>>>     
>>
>> Oh... hpa already merged them into x86/paravirt.  I'll just pull
>> x86/paravirt into core/percpu because the first patch needs to be in
>> core/percpu to keep xen working.
>>
>> hpa, can you please take the other xen patch[1] into x86/paravirt too?
>> So that I can receive it via you?
> 
> Looks like its already there.

Yes, right, which means tj-percpu == x86/paravirt for the time being.
Ingo, please pull from the following to fast forward core/percpu to
x86/paravirt (or you can just do it yourself :-)

  git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

Thanks.

-- 
tejun

Re: [PATCH 1/3] xen: setup percpu data pointers

[Ingo Molnar]

* Tejun Heo <tj@kernel.org> wrote:

> Jeremy Fitzhardinge wrote:
> > Tejun Heo wrote:
> >> Tejun Heo wrote:
> >>  
> >>> Applied 1-3 to #tj-percpu with trivial modification on #2 (will reply
> >>> to the message).
> >>>     
> >>
> >> Oh... hpa already merged them into x86/paravirt.  I'll just pull
> >> x86/paravirt into core/percpu because the first patch needs to be in
> >> core/percpu to keep xen working.
> >>
> >> hpa, can you please take the other xen patch[1] into x86/paravirt too?
> >> So that I can receive it via you?
> > 
> > Looks like its already there.
> 
> Yes, right, which means tj-percpu == x86/paravirt for the time being.
> Ingo, please pull from the following to fast forward core/percpu to
> x86/paravirt (or you can just do it yourself :-)
> 
>   git://git.kernel.org/pub/scm/linux/kernel/git/tj/misc.git tj-percpu

done, thanks Tejun. Testing results look good so far.

	Ingo