Re: 2.6.8.1-mm2 breaks vmware

[Ville Herva <vherva@viasys.com>]

On Fri, Aug 20, 2004 at 03:51:42AM -0700, you [Andrew Morton] wrote:
> Ville Herva <vherva@viasys.com> wrote:
> >
> > Andrew, I know you are not interested in closed source vmware, I'm just
> > blatantly Cc'ing you in case you would have some suggestion of the top of
> > your head. As stuff slowly trickles from -mm to mainline, this could
> > eventually end up biting more people.
> > 
> > In short, there are two (afaict) separate problem:
> > 
> > (1) vmmon.ko gives this:
> > 
> > 	vmmon: Your kernel is br0ken. get_user_pages(current, current->mm, b7dd1000, 1, 1, 0, &page, NULL) returned -14.
> > 	vmmon: I'll try accessing page tables directly, but you should know that your
> > 	vmmon: kernel is br0ken and you should uninstall all additional patches you
> > 	vmmon: have installed!
> > 	vmmon: FYI, copy_from_user(b7dd1000) returns 0 (if not 0 maybe your kernel is not br0ken)
> > 
> > (2) vmware fails to start any guest os, telling it cannot allocate memory:
> > 
> > 	VMX|[msg.msg.noMem] Cannot allocate memory.
> > 
> > 
> > (1) happened with 2.6.6-mm4 and with 2.6.8.1-mm2.
> > (2) only happened with 2.6.8.1-mm2 (with 2.6.6-mm4 vmware worked despite
> > the warning.)
> 
> Try -mm3, please.  It'll have the same problem.

I can try. I'm a bit confused what I should learn from trying -mm3 -- it
doesn't have any big changes on this area, right?
 
> > So I backed out these patches from 2.6.8.1-mm2:
> > 
> > 	flexible-mmap-2.6.7-mm3-A8.patch
> > 	flex-mmap-for-ppc64.patch
> > 	flex-mmap-for-s390x.patch
> > 	sysctl-tunable-for-flexmmap.patch
> 
> These have all been lumped together in mm3.
> 
> Try setting /proc/sys/vm/legacy_va_layout to 1

As I said, backing these out did not cure (2) (vmware failing to allocate
memory). And as Arjan said, flex-mmap is in Fedora kernels, and they have no
problems. 

Also, flex-mmap has been introduced after 2.6.6-mm4, right? That would mean
it can't explain (1) ("get_user_pages() returns -EFAULT even though
copy_from_user() return 0")

That would mean (1) is cured by reversing get_user_pages-latency-fix.patch,
increase-mlock-limit-to-32k.patch or mlock-as-user-for-268-rc2-mm2.patch.

> > 	get_user_pages-latency-fix.patch
> 
> It won't be this.
> 
> > 	increase-mlock-limit-to-32k.patch
> > 	mlock-as-user-for-268-rc2-mm2.patch
> 
> Unlikely to be these.

But what can explain (1), then? flex-mmap is not in 2.6.6-mm4, still it
gives that warning.

I just noticed I had missed get_user_pages-handle-VM_IO.patch - I'll try
backing that out first. I'll report back if I find anything interesting 
with different patch mixtures.

> > After this, problem (1) went away for 2.6.8.1-mm2, but problem (2) remained.
> 
> Try setting /proc/sys/vm/overcommit_memory to 1

Forgot to tell: problem (2) (vmware failing to allocate memory) is
consistent with 2.6.8.1-mm2 even when I try to configure the guest os to use
8MB memory. 128MB or 256MB is no problem with 2.6.8.1 mainline and it as
/proc/sys/vm/overcommit_memory set to 0. (I have 512MB of RAM and 512MB of
swap.)
 
> Maybe Peter could take a look sometime?

I already harrashed him, but I couldn't tell off the top of his head.
 
> > Could get_user_pages-latency-fix.patch explain (1)? My kernel expertise is
> > not sufficient to tell.
> 
> Doubtful.

Ok.



-- v -- 

v@iki.fi