Re: 2.6.8.1-mm2 breaks vmware

[Andrew Morton <akpm@osdl.org>]

Ville Herva <vherva@viasys.com> wrote:
>
> Andrew, I know you are not interested in closed source vmware, I'm just
> blatantly Cc'ing you in case you would have some suggestion of the top of
> your head. As stuff slowly trickles from -mm to mainline, this could
> eventually end up biting more people.
> 
> In short, there are two (afaict) separate problem:
> 
> (1) vmmon.ko gives this:
> 
> 	vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> 	vmmon: I'll try accessing page tables directly, but you should know that your
> 	vmmon: kernel is br0ken and you should uninstall all additional patches you
> 	vmmon: have installed!
> 	vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> 
> (2) vmware fails to start any guest os, telling it cannot allocate memory:
> 
> 	VMX|[msg.msg.noMem] Cannot allocate memory.
> 
> 
> (1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
> (2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite the
> warning.

Try -mm3, please.  It'll have the same problem.

> So I backed out these patches from 2.6.8.1-mm2:
> 
> 	flexible-mmap-2.6.7-mm3-A8.patch
> 	flex-mmap-for-ppc64.patch
> 	flex-mmap-for-s390x.patch
> 	sysctl-tunable-for-flexmmap.patch

These have all been lumped together in mm3.

Try setting /proc/sys/vm/legacy_va_layout to 1

> 	get_user_pages-latency-fix.patch

It won't be this.

> 	increase-mlock-limit-to-32k.patch
> 	mlock-as-user-for-268-rc2-mm2.patch

Unlikely to be these.

> 	(All conveniently available for reference at ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.8.1/2.6.8.1-mm2/broken-out/
>          if anyone is interested)
> 
> I had a vague hunch that flex-mmap stuff might affect (2) and
> get_user_pages patch perhaps (1).
> 
> After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.

Try setting /proc/sys/vm/overcommit_memory to 1

> Then I tried 2.6.8.1 vanilla. It does NOT suffer from either (1) or (2).
> 
> All experiments are done with Petr Vandrovec's newest
> vmware-any-any-update81 (apart from 2.6.6-mm4 that had some older any-to-any
> patch) and VMwareWorkstation-3.2.0-2230.

Maybe Peter could take a look sometime?

> Before I continue backing stuff out: does anyone have ideas or suggestions
> what -mm patches might be suspectible problem (2)? 2.6.8.1 -> 2.6.8.1-mm2 is
> rather large patch and so is 2.6.6-mm4 -> 2.6.8.1-mm2, and the patches
> listed above were everything I thought might be suspectible.
> 
> Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is
> not sufficient to tell.

Doubtful.