AMD64: 64 bit kernel 32 bit userland - some pending questions

AMD64: 64 bit kernel 32 bit userland - some pending questions

[Thomas Glanzmann]

Hello everyone,
I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
but a 32 bit userland (Debian Sarge). I have a few questions about this:

        - Is it possible to give the userland 3Gbyte virtual address
          space (default for 2.4 and 2.6). But give the Kernel a 64 bit
          virtual address space so that I get more than 1 Gbyte physical
          Memory into LOWMEM - say I want 8 Gbyte - without using HIGHMEM
          at all? If this scenario is possible I would get cheap memory
          access at the benefit of a well tested userland. I don't have
          applications that need more than 2 Gbyte virtual address
          space.

        - What is the easiest way to build a 64 bit kernel on a 32 bit
          Debian sarge. Are there crosscompiler packages available? Are
          there any guides on this?

        - If the above scenario works out like I imagine it, does this
          add some additional overhead I am not aware of when I switch
          for example from 32 bit userland to 64 bit kernel space which
          would override the performance gain I get from the huge LOWMEM
          virtuall address space?

        Thomas

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Andi Kleen]

Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:

> Hello everyone,
> I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> but a 32 bit userland (Debian Sarge). I have a few questions about this:

The main caveat is that iptables and ipsec need 64bit executables
to be set up. The rest should work.

> 
>         - Is it possible to give the userland 3Gbyte virtual address
>           space (default for 2.4 and 2.6).

The default is 4GB, but you can get 3GB by running it under linux32 --3gb

> But give the Kernel a 64 bit
>           virtual address space so that I get more than 1 Gbyte physical
>           Memory into LOWMEM - say I want 8 Gbyte - without using HIGHMEM

The 64bit kernel never uses highmem.

>         - What is the easiest way to build a 64 bit kernel on a 32 bit
>           Debian sarge. Are there crosscompiler packages available? Are
>           there any guides on this?

If all fails you can get a cross compiler from crosstool.
Then normal kernel compilation command with 

make ... ARCH=x86_64 CROSS_COMPILE=x86_64-linux-
 

-Andi

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Thomas Glanzmann]

Hello Andi,

> The main caveat is that iptables and ipsec need 64bit executables to
> be set up. The rest should work.

I see. But that isn't a showstopper for me because I can compile
iptables myself and the machines are protected through another firewall
anyway.

> The default is 4GB, but you can get 3GB by running it under linux32
> --3gb

4 Gbyte is fine for me.

> The 64bit kernel never uses highmem.

I see, it wouldn't make any sense.

> If all fails you can get a cross compiler from crosstool.
> Then normal kernel compilation command with 

> make ... ARCH=x86_64 CROSS_COMPILE=x86_64-linux-

I see.

Thanks a lot for your feedback,
                                Thomas

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Rafael J. Wysocki]

On Tuesday 06 June 2006 12:42, Andi Kleen wrote:
> Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> 
> > Hello everyone,
> > I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> > but a 32 bit userland (Debian Sarge). I have a few questions about this:
> 
> The main caveat is that iptables and ipsec need 64bit executables
> to be set up. The rest should work.

Recently I've had a problem running wine with a 16-bit windows application
on a 64-bit kernel.  I guess it's a wine's problem, then?

Greetings,
Rafael

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Andi Kleen]

On Tuesday 06 June 2006 13:51, Rafael J. Wysocki wrote:
> On Tuesday 06 June 2006 12:42, Andi Kleen wrote:
> > Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> > 
> > > Hello everyone,
> > > I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> > > but a 32 bit userland (Debian Sarge). I have a few questions about this:
> > 
> > The main caveat is that iptables and ipsec need 64bit executables
> > to be set up. The rest should work.
> 
> Recently I've had a problem running wine with a 16-bit windows application
> on a 64-bit kernel.  I guess it's a wine's problem, then?

At some point it worked - i ran 16bit solitaire and some other programs,
but it's not regularly tested.  When it works on a 32bit kernel with
the same wine version it should work on the 64bit kernel too. If not
it's likely a kernel bug.

-Andi

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Marcus Meissner]

On Tue, Jun 06, 2006 at 01:55:04PM +0200, Andi Kleen wrote:
> On Tuesday 06 June 2006 13:51, Rafael J. Wysocki wrote:
> > On Tuesday 06 June 2006 12:42, Andi Kleen wrote:
> > > Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> > > 
> > > > Hello everyone,
> > > > I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> > > > but a 32 bit userland (Debian Sarge). I have a few questions about this:
> > > 
> > > The main caveat is that iptables and ipsec need 64bit executables
> > > to be set up. The rest should work.
> > 
> > Recently I've had a problem running wine with a 16-bit windows application
> > on a 64-bit kernel.  I guess it's a wine's problem, then?
> 
> At some point it worked - i ran 16bit solitaire and some other programs,
> but it's not regularly tested.  When it works on a 32bit kernel with
> the same wine version it should work on the 64bit kernel too. If not
> it's likely a kernel bug.

It should work fine.

If not, what is the actual bug/error message? :)

Ciao, Marcus

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Rafael J. Wysocki]

On Tuesday 06 June 2006 13:56, Marcus Meissner wrote:
> On Tue, Jun 06, 2006 at 01:55:04PM +0200, Andi Kleen wrote:
> > On Tuesday 06 June 2006 13:51, Rafael J. Wysocki wrote:
> > > On Tuesday 06 June 2006 12:42, Andi Kleen wrote:
> > > > Thomas Glanzmann <sithglan@stud.uni-erlangen.de> writes:
> > > > 
> > > > > Hello everyone,
> > > > > I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> > > > > but a 32 bit userland (Debian Sarge). I have a few questions about this:
> > > > 
> > > > The main caveat is that iptables and ipsec need 64bit executables
> > > > to be set up. The rest should work.
> > > 
> > > Recently I've had a problem running wine with a 16-bit windows application
> > > on a 64-bit kernel.  I guess it's a wine's problem, then?
> > 
> > At some point it worked - i ran 16bit solitaire and some other programs,
> > but it's not regularly tested.  When it works on a 32bit kernel with
> > the same wine version it should work on the 64bit kernel too. If not
> > it's likely a kernel bug.
> 
> It should work fine.
> 
> If not, what is the actual bug/error message? :)

Segmentation fault. :-)

I'll try to reproduce it and get some details.

Greetings,
Rafael

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[Albert Cahalan]

Thomas Glanzmann writes:

> I would like to use an AMD64 Opteron System with a 64 bit Linux Kernel,
> but a 32 bit userland (Debian Sarge). I have a few questions about this:
>
> - Is it possible to give the userland 3Gbyte virtual address
>   space (default for 2.4 and 2.6). But give the Kernel a 64 bit
>   virtual address space so that I get more than 1 Gbyte physical
>   Memory into LOWMEM - say I want 8 Gbyte - without using HIGHMEM
>   at all? If this scenario is possible I would get cheap memory
>   access at the benefit of a well tested userland. I don't have
>   applications that need more than 2 Gbyte virtual address
>   space.

Why do you want cheap memory access? Normally people want this
for performance, but 32-bit apps are noticably slower.

The "well tested userland" is only well tested on a 32-bit kernel.
There are many ways in which a 64-bit kernel fails to correctly
run 32-bit apps. I just found one the other day, in the way signal
handler info gets translated from 64-bit to 32-bit. Some of your
userspace would be something I wrote, which I happen to know will
not run 100% correctly and I just don't care to "fix" it for your
wildly abnormal configuration.

I suspect you have a Windows mindset. In the Windows world, almost
nothing is 64-bit clean. Nearly every Linux app is well-tested in
64-bit form. This is because Linux was ported to the Alpha CPU
back around 1994, give or take a year. That's over a decade of
64-bit experience. It's mainly the proprietary apps that don't
run in 64-bit mode. The one big exception is OpenOffice, which
was also proprietary until Sun opened it up.

Just forget the old 32-bit crud if at all possible. Hopefully
you won't have even one 32-bit library or executable. Make it
your goal to eliminate this buggy cruft. You can use chroot()
or VMWare as needed, but cleaning house is so much nicer.

> - If the above scenario works out like I imagine it, does this
>   add some additional overhead I am not aware of when I switch
>   for example from 32 bit userland to 64 bit kernel space which
>   would override the performance gain I get from the huge LOWMEM
>   virtuall address space?

Of course there is overhead. There are also bugs, some of which
won't ever get fixed.

The bigger problem is that 32-bit code has fewer CPU registers.
(nobody has done an ILP32 ABI for long mode) This is slow.

Re: AMD64: 64 bit kernel 32 bit userland - some pending questions

[H. Peter Anvin]

Followup to:  <787b0d920606062011j21083e80v659228a7565ecfab@mail.gmail.com>
By author:    "Albert Cahalan" <acahalan@gmail.com>
In newsgroup: linux.dev.kernel
> 
> The bigger problem is that 32-bit code has fewer CPU registers.
> (nobody has done an ILP32 ABI for long mode) This is slow.
> 

The issue isn't the ABI, the issue is that the processor doesn't
support it, since some of the opcodes mean different things in 16-,
32- and 64-bit mode.  The opcodes which access the high half of the
register sets (REX prefixes) in 64-bit mode are INC and DEC
instructions in 16- and 32-bit mode.

AMD was apparently considering adding a "REX32" mode at some point,
but rather predictably noone was interested enough to make it
worthwhile.

	-hpa