2.6.8.1-mm2 breaks vmware

2.6.8.1-mm2 breaks vmware

[Ville Herva]

Andrew, I know you are not interested in closed source vmware, I'm just
blatantly Cc'ing you in case you would have some suggestion of the top of
your head. As stuff slowly trickles from -mm to mainline, this could
eventually end up biting more people.

In short, there are two (afaict) separate problem:

(1) vmmon.ko gives this:

	vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
	vmmon: I'll try accessing page tables directly, but you should know that your
	vmmon: kernel is br0ken and you should uninstall all additional patches you
	vmmon: have installed!
	vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)

(2) vmware fails to start any guest os, telling it cannot allocate memory:

	VMX|[msg.msg.noMem] Cannot allocate memory.


(1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
(2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite the
warning.

So I backed out these patches from 2.6.8.1-mm2:

	flexible-mmap-2.6.7-mm3-A8.patch
	flex-mmap-for-ppc64.patch
	flex-mmap-for-s390x.patch
	sysctl-tunable-for-flexmmap.patch
	get_user_pages-latency-fix.patch
	increase-mlock-limit-to-32k.patch
	mlock-as-user-for-268-rc2-mm2.patch
	(All conveniently available for reference at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm2/broken-out/
         if anyone is interested)

I had a vague hunch that flex-mmap stuff might affect (2) and
get_user_pages patch perhaps (1).

After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.

Then I tried 2.6.8.1 vanilla. It does NOT suffer from either (1) or (2).

All experiments are done with Petr Vandrovec's newest
vmware-any-any-update81 (apart from 2.6.6-mm4 that had some older any-to-any
patch) and VMwareWorkstation-3.2.0-2230.

Before I continue backing stuff out: does anyone have ideas or suggestions
what -mm patches might be suspectible problem (2)? 2.6.8.1 -> 2.6.8.1-mm2 is
rather large patch and so is 2.6.6-mm4 -> 2.6.8.1-mm2, and the patches
listed above were everything I thought might be suspectible.

Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is
not sufficient to tell.


-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Andrew Morton]

Ville Herva <vherva@viasys.com> wrote:
>
> Andrew, I know you are not interested in closed source vmware, I'm just
> blatantly Cc'ing you in case you would have some suggestion of the top of
> your head. As stuff slowly trickles from -mm to mainline, this could
> eventually end up biting more people.
> 
> In short, there are two (afaict) separate problem:
> 
> (1) vmmon.ko gives this:
> 
> 	vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> 	vmmon: I'll try accessing page tables directly, but you should know that your
> 	vmmon: kernel is br0ken and you should uninstall all additional patches you
> 	vmmon: have installed!
> 	vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> 
> (2) vmware fails to start any guest os, telling it cannot allocate memory:
> 
> 	VMX|[msg.msg.noMem] Cannot allocate memory.
> 
> 
> (1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
> (2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite the
> warning.

Try -mm3, please.  It'll have the same problem.

> So I backed out these patches from 2.6.8.1-mm2:
> 
> 	flexible-mmap-2.6.7-mm3-A8.patch
> 	flex-mmap-for-ppc64.patch
> 	flex-mmap-for-s390x.patch
> 	sysctl-tunable-for-flexmmap.patch

These have all been lumped together in mm3.

Try setting /proc/sys/vm/legacy_va_layout to 1

> 	get_user_pages-latency-fix.patch

It won't be this.

> 	increase-mlock-limit-to-32k.patch
> 	mlock-as-user-for-268-rc2-mm2.patch

Unlikely to be these.

> 	(All conveniently available for reference at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm2/broken-out/
>          if anyone is interested)
> 
> I had a vague hunch that flex-mmap stuff might affect (2) and
> get_user_pages patch perhaps (1).
> 
> After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.

Try setting /proc/sys/vm/overcommit_memory to 1

> Then I tried 2.6.8.1 vanilla. It does NOT suffer from either (1) or (2).
> 
> All experiments are done with Petr Vandrovec's newest
> vmware-any-any-update81 (apart from 2.6.6-mm4 that had some older any-to-any
> patch) and VMwareWorkstation-3.2.0-2230.

Maybe Peter could take a look sometime?

> Before I continue backing stuff out: does anyone have ideas or suggestions
> what -mm patches might be suspectible problem (2)? 2.6.8.1 -> 2.6.8.1-mm2 is
> rather large patch and so is 2.6.6-mm4 -> 2.6.8.1-mm2, and the patches
> listed above were everything I thought might be suspectible.
> 
> Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is
> not sufficient to tell.

Doubtful.

Re: 2.6.8.1-mm2 breaks vmware

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 347 bytes --]


> So I backed out these patches from 2.6.8.1-mm2:
> 
> 	flexible-mmap-2.6.7-mm3-A8.patch
> 	flex-mmap-for-ppc64.patch
> 	flex-mmap-for-s390x.patch
> 	sysctl-tunable-for-flexmmap.patch
unlikely candidates; these are in the fedora kernel too and no problems
reported with vmware like this... 
(and we do get reports when vmware breaks ;)

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 03:51:42AM -0700, you [Andrew Morton] wrote:
> Ville Herva <vherva@viasys.com> wrote:
> >
> > Andrew, I know you are not interested in closed source vmware, I'm just
> > blatantly Cc'ing you in case you would have some suggestion of the top of
> > your head. As stuff slowly trickles from -mm to mainline, this could
> > eventually end up biting more people.
> > 
> > In short, there are two (afaict) separate problem:
> > 
> > (1) vmmon.ko gives this:
> > 
> > 	vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> > 	vmmon: I'll try accessing page tables directly, but you should know that your
> > 	vmmon: kernel is br0ken and you should uninstall all additional patches you
> > 	vmmon: have installed!
> > 	vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> > 
> > (2) vmware fails to start any guest os, telling it cannot allocate memory:
> > 
> > 	VMX|[msg.msg.noMem] Cannot allocate memory.
> > 
> > 
> > (1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
> > (2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite
> > the warning.)
> 
> Try -mm3, please.  It'll have the same problem.

I can try. I'm a bit confused what I should learn from trying -mm3 -- it
doesn't have any big changes on this area, right?
 
> > So I backed out these patches from 2.6.8.1-mm2:
> > 
> > 	flexible-mmap-2.6.7-mm3-A8.patch
> > 	flex-mmap-for-ppc64.patch
> > 	flex-mmap-for-s390x.patch
> > 	sysctl-tunable-for-flexmmap.patch
> 
> These have all been lumped together in mm3.
> 
> Try setting /proc/sys/vm/legacy_va_layout to 1

As I said, backing these out did not cure (2) (vmware failing to allocate
memory). And as Arjan said, flex-mmap is in Fedora kernels, and they have no
problems. 

Also, flex-mmap has been introduced after 2.6.6-mm4, right? That would mean
it can't explain (1) ("get_user_pages() returns -EFAULT even though
copy_from_user() return 0")

That would mean (1) is cured by reversing get_user_pages-latency-fix.patch,
increase-mlock-limit-to-32k.patch or mlock-as-user-for-268-rc2-mm2.patch.

> > 	get_user_pages-latency-fix.patch
> 
> It won't be this.
> 
> > 	increase-mlock-limit-to-32k.patch
> > 	mlock-as-user-for-268-rc2-mm2.patch
> 
> Unlikely to be these.

But what can explain (1), then? flex-mmap is not in 2.6.6-mm4, still it
gives that warning.

I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try
backing that out first. I'll report back if I find anything interesting 
with different patch mixtures.

> > After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.
> 
> Try setting /proc/sys/vm/overcommit_memory to 1

Forgot to tell: problem (2) (vmware failing to allocate memory) is
consistent with 2.6.8.1-mm2 even when I try to configure the guest os to use
8MB memory. 128MB or 256MB is no problem with 2.6.8.1 mainline and it as
/proc/sys/vm/overcommit_memory set to 0. (I have 512MB of RAM and 512MB of
swap.)
 
> Maybe Peter could take a look sometime?

I already harrashed him, but I couldn't tell off the top of his head.
 
> > Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is
> > not sufficient to tell.
> 
> Doubtful.

Ok.



-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 04:18:25PM +0300, you [Ville Herva] wrote:
> 
> I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try
> backing that out first. I'll report back if I find anything interesting 
> with different patch mixtures.

Well, I just tried 2.6.8.1-mm2 minus get_user_pages-handle-VM_IO.patch but
that didn't help with the "cannot allocate memory" problem. Curiously, I
didn't get the "get_user_pages() returns -EFAULT" warning with this kernel.

Also, neither "echo 1 > /proc/sys/vm/overcommit_memory" nor "echo 1 >
/proc/sys/vm/legacy_va_layout" helped.

It seems this is "cannot allocate memory" might have something to do with
/dev/mem mmap() permissions - here's a strace:

--8<-----------------------------------------------------------------------
5022  geteuid32()                       = 1414
5022  setresuid32(-1, 0, -1)            = 0
5022  open("/dev/mem", O_RDWR)          = 8
5022  getuid32()                        = 1414
5022  setresuid32(-1, 1414, -1)         = 0
5022  old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_SHARED, 8, 0x3d6000) = -1 EPERM (Operation not permitted)
5022  close(8)                          = 0
5022  ioctl(4, 0xf0, 0x3d6000)          = 0
5022  time(NULL)                        = 1093011756
5022  open("/etc/localtime", O_RDONLY)  = 8
5022  fstat64(8, {st_mode=S_IFREG|0644, st_size=682, ...}) = 0
5022  close(8)                          = 0
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"Msg_Post: Error\n", 16}], 3) = 37
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"[msg.msg.noMem] Cannot allocate "..., 40}], 3) = 61
5022  writev(3, [{"Aug 20 17:22:36: ", 17}, {"VMX|", 4}, {"--------------------------------"..., 41}], 3) = 62
--8<-----------------------------------------------------------------------

I just put get_user_pages-handle-VM_IO.patch back and reverted
dev-mem-restriction-patch.patch - I'll report back when it has compiled. 


-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 05:43:04PM +0300, you [Ville Herva] wrote:
> On Fri, Aug 20, 2004 at 04:18:25PM +0300, you [Ville Herva] wrote:
> > 
> > I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try
> > backing that out first. I'll report back if I find anything interesting 
> > with different patch mixtures.
> 
> Well, I just tried 2.6.8.1-mm2 minus get_user_pages-handle-VM_IO.patch but
> that didn't help with the "cannot allocate memory" problem. Curiously, I
> didn't get the "get_user_pages() returns -EFAULT" warning with this kernel.
> 
> I just put get_user_pages-handle-VM_IO.patch back and reverted
> dev-mem-restriction-patch.patch - I'll report back when it has compiled. 

Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot
allocate memory" problem. 

With this kernel I still get the 

--8<-----------------------------------------------------------------------
vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
vmmon: I'll try accessing page tables directly, but you should know that your
vmmon: kernel is br0ken and you should uninstall all additional patches you vmmon: have installed!
vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
--8<-----------------------------------------------------------------------

warning, but vmware appears to work now (well apart from altgr not working,
but that has been broken since 2.4 -> 2.6 transition.)

I'm still not 100% which of the patches causes that get_user_pages()
warning.


-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Andrew Morton]

Ville Herva <vherva@viasys.com> wrote:
>
>  On Fri, Aug 20, 2004 at 05:43:04PM +0300, you [Ville Herva] wrote:
>  > On Fri, Aug 20, 2004 at 04:18:25PM +0300, you [Ville Herva] wrote:
>  > > 
>  > > I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try
>  > > backing that out first. I'll report back if I find anything interesting 
>  > > with different patch mixtures.
>  > 
>  > Well, I just tried 2.6.8.1-mm2 minus get_user_pages-handle-VM_IO.patch but
>  > that didn't help with the "cannot allocate memory" problem. Curiously, I
>  > didn't get the "get_user_pages() returns -EFAULT" warning with this kernel.
>  > 
>  > I just put get_user_pages-handle-VM_IO.patch back and reverted
>  > dev-mem-restriction-patch.patch - I'll report back when it has compiled. 
> 
>  Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot
>  allocate memory" problem. 

Thanks for working that out.

Strange.  I'd have assumed that the Fedora kernels include that patch.

>  With this kernel I still get the 
> 
>  --8<-----------------------------------------------------------------------
>  vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
>  vmmon: I'll try accessing page tables directly, but you should know that your
>  vmmon: kernel is br0ken and you should uninstall all additional patches you vmmon: have installed!
>  vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
>  --8<-----------------------------------------------------------------------
> 
>  warning, but vmware appears to work now (well apart from altgr not working,
>  but that has been broken since 2.4 -> 2.6 transition.)
> 
>  I'm still not 100% which of the patches causes that get_user_pages()
>  warning.

If you could work that out sometime, it would help.

Re: 2.6.8.1-mm2 breaks vmware

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 598 bytes --]

On Fri, Aug 20, 2004 at 11:45:18AM -0700, Andrew Morton wrote:
> >  > I just put get_user_pages-handle-VM_IO.patch back and reverted
> >  > dev-mem-restriction-patch.patch - I'll report back when it has compiled. 
> > 
> >  Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot
> >  allocate memory" problem. 
> 
> Thanks for working that out.
> 
> Strange.  I'd have assumed that the Fedora kernels include that patch.

we do. Maybe you have an older (rare) vmware version ???

question is wtf vmware is doing with that memory.. it's outside the bios
area after all.... petr?




[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 11:45:18AM -0700, you [Andrew Morton] wrote:
> > 
> >  Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot
> >  allocate memory" problem. 
> 
> Thanks for working that out.

No problem. (If only compile-reboot-startx-launch vmware was faster on my
computer...)
 
> >   vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> >   vmmon: I'll try accessing page tables directly, but you should know that your
> >   vmmon: kernel is br0ken and you should uninstall all additional patches you vmmon: have installed!
> >   vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> > 
> >  I'm still not 100% which of the patches causes that get_user_pages()
> >  warning.
> 
> If you could work that out sometime, it would help.

I'll try to do that as soon as I find time.



-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 08:49:33PM +0200, you [Arjan van de Ven] wrote:
> > > 
> > >  Ok, 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch fixes the "cannot
> > >  allocate memory" problem. 
> > 
> > Thanks for working that out.
> > 
> > Strange.  I'd have assumed that the Fedora kernels include that patch.
> 
> we do. Maybe you have an older (rare) vmware version ???

Yes, VMwareWorkstation-3.2.0-2230. 

Looks like I'll need to break my piggy bank and consider upgrading.
Commercial software... ;@)


-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Fri, Aug 20, 2004 at 11:45:18AM -0700, you [Andrew Morton] wrote:
> > 
> >  --8<-----------------------------------------------------------------------
> >  vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> >  vmmon: I'll try accessing page tables directly, but you should know that your
> >  vmmon: kernel is br0ken and you should uninstall all additional patches you vmmon: have installed!
> >  vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> >  --8<-----------------------------------------------------------------------
> > 
> >  warning, but vmware appears to work now (well apart from altgr not working,
> >  but that has been broken since 2.4 -> 2.6 transition.)
> > 
> >  I'm still not 100% which of the patches causes that get_user_pages()
> >  warning.
> 
> If you could work that out sometime, it would help.

* 2.6.8.1-mm2 minus just dev-mem-restriction-patch.patch fixes the "cannot
  allocate memory" problem.

* 2.6.8.1-mm2 minus dev-mem-restriction-patch.patch and
  get_user_pages-handle-VM_IO.patch fixes both the "cannot allocate memory"
  and "get_user_pages() returns -EFAULT" problems.

"Cannot allocate memory" may be specific to older vmware 3.2.0 since it
hasn't been reported by anyone else (even though the patch is present in
Fedora). 



-- v -- 

v@iki.fi

Re: 2.6.8.1-mm2 breaks vmware

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 276 bytes --]

Salut,

On Fri, Aug 20, 2004 at 10:30:24PM +0300, Ville Herva wrote:
> Yes, VMwareWorkstation-3.2.0-2230. 
> 
> Looks like I'll need to break my piggy bank and consider upgrading.
> Commercial software... ;@)

Does VMware > 4.0 still require cmov?

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.8.1-mm2 breaks vmware

[Petr Vandrovec]

On Sun, Aug 22, 2004 at 01:42:14PM +0200, Tonnerre wrote:
> Salut,
> 
> On Fri, Aug 20, 2004 at 10:30:24PM +0300, Ville Herva wrote:
> > Yes, VMwareWorkstation-3.2.0-2230. 
> > 
> > Looks like I'll need to break my piggy bank and consider upgrading.
> > Commercial software... ;@)
> 
> Does VMware > 4.0 still require cmov?

Yes, this requirement is not going to go away: VMware >= 4.0 / GSX >= 3.0 require
processors which support CMOV instruction.
							Petr

Re: 2.6.8.1-mm2 breaks vmware

[Alan Cox]

On Sul, 2004-08-22 at 14:59, Petr Vandrovec wrote:
> > Does VMware > 4.0 still require cmov?
> 
> Yes, this requirement is not going to go away: VMware >= 4.0 / GSX >= 3.0 require
> processors which support CMOV instruction.

qemu isnt as fast and isnt as featured but doesn't need cmov. The
current one runs stuff like Win98 rather well including PCI video card
emulation.

Alan

Re: 2.6.8.1-mm2 breaks vmware

[Petr Vandrovec]

On Sat, Aug 21, 2004 at 09:07:30PM +0200, Arjan van de Ven wrote:
> On Sat, Aug 21, 2004 at 10:00:27PM +0300, Ville Herva wrote:
> 
> > > can you try the one below as a replacement ?
> > >
> > > diff -purN linux-2.6.8/arch/i386/mm/init.c linux/arch/i386/mm/init.c
> > > --- linux-2.6.8/arch/i386/mm/init.c	2004-08-21 15:38:10.323915313 +0200
> > > +++ linux/arch/i386/mm/init.c	2004-08-21 15:43:15.082088389 +0200
> > > @@ -93,6 +93,26 @@ static inline int page_is_ram(unsigned l
> > >  	return 0;
> > >  }
> > 
> > (Sorry for the delay.)
> > 
> > Yes, this works. 
> 
> Andrew,
> 
> could you replace the devmem patch in your tree with this one?
> Ingo and I verified that the behavior as to blocking access to physical
> memory is the same while Ville verified it unbreaks this old vmware version
> (which seems to do something nasty just not harmful)
> Thanks,

Sorry that it took so long, I was without connectivity during weekend, trying to
find what's going on.

VMware Workstation 3.2.x (builds 2230 to 2242) uses very strange way to allocate
memory below 4GB range (older prods than 3.2.0 crash on systems with page
tables above 4GB, and VMware 4.x use better technology).

Whole situation (with originally released vmmon) looked like that vmware binary
issued ioctl() to allocate memory, marked that page PG_RESERVED, and returned
physical page number to userspace. Userspace then opened /dev/mem, and mapped
that page to the process.  On cleanup /dev/mem was unmapped, PG_RESERVED bit
was cleared, and page released (in my updates PG_RESERVED setting/clearing is
removed, as it badly intereferes with page's refcounting).

This is broken by two patches in MM: 
* get_user_pages_handle-VM_IO, which marks all /dev/mem memory mmaps as VM_IO 
  (thus breaking get_user_pages() on these pages, which is later needed) and 
* dev-mem-restricton-patch which prevents main memory to be mapped through
  /dev/mem (if page obtained from kernel is above 1MB).

During weekend I was able to create binary patch for VMware Workstation 3.2.1 
(patch available at http://platan.vc.cvut.cz/ftp/pub/vmware/vmware-any-any-update82.tar.gz)
which turns that messy ioctl & /dev/mem mmap to simpler, safer and better /dev/vmmon mmap
(and unmap + ioctl to simple unmap) (as used by WS4+/GSX3+).  After that WS 3.2.1 works on 
your 2.6.8.1-mm3 without problems (after applying two patches I'm sending you
separately to get -mm3 to work at all on my notebook).

Andrew, because I have a binary patch for affected products, I think that you should 
leave your tree as is.  Although I do not agree with dev-mem-restriction-patch,
maybe some other people will feel safer with this restriction in place.  

You can redirect VMware Workstation 3.2.x and VMware GSX Server 2.x customers to 
me, and I'll create binary patches for these other products on demand (I do not 
want to waste my time, and GSX customers should either use supported OSes, or 
latest released GSX version anyway).

Ville, please try applying vmware-any-any-update82.tar.gz.  During application it
must say that it found 'VMware Workstation 3.2.1-build 2242'.  If it will say that it found
build-2230 to 2242, new binary pattern was not recognized and I'll need your vmware binary,
as I did not find build 2230 in my archive. Or you can upgrade to the build 2242.

					Best regards,
						Petr Vandrovec
						vandrove@vc.cvut.cz

Re: 2.6.8.1-mm2 breaks vmware

[Arjan van de Ven]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

On Sun, Aug 22, 2004 at 04:31:12PM +0200, Petr Vandrovec wrote:
> Whole situation (with originally released vmmon) looked like that vmware binary
> issued ioctl() to allocate memory, marked that page PG_RESERVED, and returned
> physical page number to userspace. Userspace then opened /dev/mem, and mapped
> that page to the process.  On cleanup /dev/mem was unmapped, PG_RESERVED bit
> was cleared, and page released (in my updates PG_RESERVED setting/clearing is
> removed, as it badly intereferes with page's refcounting).

I've sent andrew a patch that allows such mmaps again for PG_RESERVED pages.
The approach vmware did is rather questionable (as you say) and thankfully
fixed later, but breaking it is not entirely required.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva]

On Sun, Aug 22, 2004 at 04:31:12PM +0200, you [Petr Vandrovec] wrote:
> 
> During weekend I was able to create binary patch for VMware Workstation
> 3.2.1 (patch available at
> http://platan.vc.cvut.cz/ftp/pub/vmware/vmware-any-any-update82.tar.gz)
> which turns that messy ioctl & /dev/mem mmap to simpler, safer and better
> /dev/vmmon mmap (and unmap + ioctl to simple unmap) (as used by
> WS4+/GSX3+).  After that WS 3.2.1 works on your 2.6.8.1-mm3 without
> problems (after applying two patches I'm sending you separately to get
> -mm3 to work at all on my notebook).

Incredible. 

I was sure I was out of look with the aged vmware 3.2.0, and now suddenly
there is patch for both the kernel and the application :).
 
> Ville, please try applying vmware-any-any-update82.tar.gz.  During
> application it must say that it found 'VMware Workstation 3.2.1-build
> 2242'.  If it will say that it found build-2230 to 2242, new binary
> pattern was not recognized and I'll need your vmware binary, as I did not
> find build 2230 in my archive. Or you can upgrade to the build 2242.

I didn't even realize 2242 had been released ;). (As I said, I've lately
been using vmware less and less.)

I just upgraded to 2242 + any2any82 and it works fine (*). I'm still using
the 2.6.8.1-mm2 with the updated dev-mem restriction patch. I'll retest this
as soon as I get to upgrade my kernel again to something that doesn't
contain Arjan's new dev-mem patch. I'll let you know.

*) Apart from that even 3.2.1 2242 does not fix the altgr key that does not
   seem to work with 2.6 new input system. I presume the new input layer is
   the reason; I do know altgr used to work with 2.4 host kernel and winxp
   guest.



-- v -- 

v@iki.fi

[OT] vmware, 2.6 kernel and altgr key (Re: 2.6.8.1-mm2 breaks vmware)

[Ville Herva]

On Mon, Aug 23, 2004 at 12:19:03AM +0300, you [Ville Herva] wrote:
> 
> *) Apart from that even 3.2.1 2242 does not fix the altgr key that does not
>    seem to work with 2.6 new input system. 

For the record, Petr found a solution for this one, too.

For whatever reason, xev gives the following for altgr (right alt) these
days:

    state 0x0, keycode 113 (keysym 0xfe03, ISO_Level3_Shift), same_screen YES,
                       ^^^

Adding 
 
  xkeymap.keycode.113 = 312

into /etc/vmware/config or into the per guest .cfg lets altgr work again
with vmware 3.2.1.


-- v -- 

v@iki.fi