[PATCH][2.5] VMWare doesn't like sysenter

[PATCH][2.5] VMWare doesn't like sysenter

[Zwane Mwaikambo]

I get a monitor error in VMWare4 with a sysenter syscall enabled kernel, 
this patch simply disables sysenter based syscalls but doesn't clear the 
SEP bit in the capabilities.

Index: linux-2.5.69-mm5/arch/i386/kernel/sysenter.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.69/arch/i386/kernel/sysenter.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 sysenter.c
--- linux-2.5.69-mm5/arch/i386/kernel/sysenter.c	6 May 2003 12:20:51 -0000	1.1.1.1
+++ linux-2.5.69-mm5/arch/i386/kernel/sysenter.c	15 May 2003 07:46:05 -0000
@@ -20,6 +20,7 @@
 #include <asm/unistd.h>
 
 extern asmlinkage void sysenter_entry(void);
+static int nosysenter __initdata;
 
 /*
  * Create a per-cpu fake "SEP thread" stack, so that we can
@@ -51,6 +52,13 @@ void enable_sep_cpu(void *info)
 	put_cpu();	
 }
 
+static int __init do_nosysenter(char *s)
+{
+	nosysenter = 1;
+	return 1;
+}
+__setup("nosysenter", do_nosysenter);
+
 /*
  * These symbols are defined by vsyscall.o to mark the bounds
  * of the ELF DSO images included therein.
@@ -64,7 +72,7 @@ static int __init sysenter_setup(void)
 
 	__set_fixmap(FIX_VSYSCALL, __pa(page), PAGE_READONLY);
 
-	if (!boot_cpu_has(X86_FEATURE_SEP)) {
+	if (nosysenter || !boot_cpu_has(X86_FEATURE_SEP)) {
 		memcpy((void *) page,
 		       &vsyscall_int80_start,
 		       &vsyscall_int80_end - &vsyscall_int80_start);

-- 
function.linuxpower.ca

Re: [PATCH][2.5] VMWare doesn't like sysenter

[Randy.Dunlap]

On Thu, 15 May 2003 04:02:31 -0400 (EDT) Zwane Mwaikambo <zwane@linuxpower.ca> wrote:

| I get a monitor error in VMWare4 with a sysenter syscall enabled kernel, 
| this patch simply disables sysenter based syscalls but doesn't clear the 
| SEP bit in the capabilities.

| +static int __init do_nosysenter(char *s)
| +{
| +	nosysenter = 1;
| +	return 1;
| +}
| +__setup("nosysenter", do_nosysenter);

Needs entry in Documentation/kernel-parameters.txt also
if/when accepted.

--
~Randy

Re: [PATCH][2.5] VMWare doesn't like sysenter

[Zwane Mwaikambo]

On Thu, 15 May 2003, Randy.Dunlap wrote:

> On Thu, 15 May 2003 04:02:31 -0400 (EDT) Zwane Mwaikambo <zwane@linuxpower.ca> wrote:
> 
> | I get a monitor error in VMWare4 with a sysenter syscall enabled kernel, 
> | this patch simply disables sysenter based syscalls but doesn't clear the 
> | SEP bit in the capabilities.
> 
> | +static int __init do_nosysenter(char *s)
> | +{
> | +	nosysenter = 1;
> | +	return 1;
> | +}
> | +__setup("nosysenter", do_nosysenter);
> 
> Needs entry in Documentation/kernel-parameters.txt also
> if/when accepted.

Thanks for the heads up.

Index: linux-2.5.69-mm5/Documentation/kernel-parameters.txt
===================================================================
RCS file: /build/cvsroot/linux-2.5.69/Documentation/kernel-parameters.txt,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 kernel-parameters.txt
--- linux-2.5.69-mm5/Documentation/kernel-parameters.txt	6 May 2003 12:21:18 -0000	1.1.1.1
+++ linux-2.5.69-mm5/Documentation/kernel-parameters.txt	15 May 2003 15:14:23 -0000
@@ -1063,6 +1063,10 @@ running once the system is up.
 
 	sym53c8xx=	[HW,SCSI]
 			See Documentation/scsi/ncr53c8xx.txt.
+	
+	nosysenter	[IA-32]
+			Disable SYSENTER for syscalls, does not clear the SEP
+			capabilities bit.
 
 	t128=		[HW,SCSI]
 			See header of drivers/scsi/t128.c.
Index: linux-2.5.69-mm5/arch/i386/kernel/sysenter.c
===================================================================
RCS file: /build/cvsroot/linux-2.5.69/arch/i386/kernel/sysenter.c,v
retrieving revision 1.1.1.1
diff -u -p -B -r1.1.1.1 sysenter.c
--- linux-2.5.69-mm5/arch/i386/kernel/sysenter.c	6 May 2003 12:20:51 -0000	1.1.1.1
+++ linux-2.5.69-mm5/arch/i386/kernel/sysenter.c	15 May 2003 07:46:05 -0000
@@ -20,6 +20,7 @@
 #include <asm/unistd.h>
 
 extern asmlinkage void sysenter_entry(void);
+static int nosysenter __initdata;
 
 /*
  * Create a per-cpu fake "SEP thread" stack, so that we can
@@ -51,6 +52,13 @@ void enable_sep_cpu(void *info)
 	put_cpu();	
 }
 
+static int __init do_nosysenter(char *s)
+{
+	nosysenter = 1;
+	return 1;
+}
+__setup("nosysenter", do_nosysenter);
+
 /*
  * These symbols are defined by vsyscall.o to mark the bounds
  * of the ELF DSO images included therein.
@@ -64,7 +72,7 @@ static int __init sysenter_setup(void)
 
 	__set_fixmap(FIX_VSYSCALL, __pa(page), PAGE_READONLY);
 
-	if (!boot_cpu_has(X86_FEATURE_SEP)) {
+	if (nosysenter || !boot_cpu_has(X86_FEATURE_SEP)) {
 		memcpy((void *) page,
 		       &vsyscall_int80_start,
 		       &vsyscall_int80_end - &vsyscall_int80_start);
-- 
function.linuxpower.ca

Re: [PATCH][2.5] VMWare doesn't like sysenter

[Petr Vandrovec]

On 15 May 03 at 11:15, Zwane Mwaikambo wrote:
> On Thu, 15 May 2003, Randy.Dunlap wrote:
> 
> > On Thu, 15 May 2003 04:02:31 -0400 (EDT) Zwane Mwaikambo <zwane@linuxpower.ca> wrote:
> > 
> > | I get a monitor error in VMWare4 with a sysenter syscall enabled kernel, 
> > | this patch simply disables sysenter based syscalls but doesn't clear the 
> > | SEP bit in the capabilities.
> > 
> > | +static int __init do_nosysenter(char *s)
> > | +{
> > | +   nosysenter = 1;
> > | +   return 1;
> > | +}
> > | +__setup("nosysenter", do_nosysenter);
> > 
> > Needs entry in Documentation/kernel-parameters.txt also
> > if/when accepted.
> 
> Thanks for the heads up.
> 
> Index: linux-2.5.69-mm5/Documentation/kernel-parameters.txt
> ===================================================================
> RCS file: /build/cvsroot/linux-2.5.69/Documentation/kernel-parameters.txt,v
> retrieving revision 1.1.1.1
> diff -u -p -B -r1.1.1.1 kernel-parameters.txt
> --- linux-2.5.69-mm5/Documentation/kernel-parameters.txt    6 May 2003 12:21:18 -0000   1.1.1.1
> +++ linux-2.5.69-mm5/Documentation/kernel-parameters.txt    15 May 2003 15:14:23 -0000
> @@ -1063,6 +1063,10 @@ running once the system is up.
>  
>     sym53c8xx=  [HW,SCSI]
>             See Documentation/scsi/ncr53c8xx.txt.
> +   
> +   nosysenter  [IA-32]
> +           Disable SYSENTER for syscalls, does not clear the SEP
> +           capabilities bit.

RedHat's 9 backport of vsyscalls uses 'nosysinfo' name for 
option which does simillar task (it just stops kernel from reporting
relevant AT_SYSINFO completely).

BTW, what's reason for this backport? They just always put int 0x80
into this page, so even on real CPU kernel booted with nosysinfo
is faster than one which uses this vsyscall entry point.

VMware's panic should be fixed in next update, but I'm afraid that
penalty from exectuing code in upper couple of MBs of address space 
will not disappear. Moving SYSINFO page from FFFFE000 to FF7FF000
(PKMAP_BASE - PAGE_SIZE == VMALLOC_END + PAGE_SIZE) would fix both
problems.
                                                    Petr Vandrovec
                                                    vandrove@vc.cvut.cz