vm: shmat() address for 32 vs 64 bit kernel

vm: shmat() address for 32 vs 64 bit kernel

[Michael Tokarev]

I were needed to run a legacy application which uses
Oracle 8 database.  Tried to run it on a 64bit kernel,
which is running on my machine for a long time, together
with 32bits userland.  Everything worked but not oracle,
which segfaults at startup on 64bits kernel, but works
just fine when running the same but 32bits kernel
(userspace is the same).

After much debugging I found this difference which may
be relevant.  It's the address returned by shmat() routine.

Here are the relevant stuff for 32bits:

shmget(0x4d21f3bc, 579067904, IPC_CREAT|IPC_EXCL|0640) = 174358532
shmat(174358532, 0, 0)            = 0x9569f000

Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.

And here's what happens with 64bits kernel:

shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640) = 294912
shmat(294912, 0, 0)               = 0xd5548000

Here, the segment ends at 3965Mb.

And right after that shmat, oracle process terminates with SIGSEGV
(on 64bits kernel, but continues just fine on 32bits).

I understand full well many bad points here: proprietary software,
old legacy code, not supported, buggy.

But the question really is: why the difference in the first place,
for 32bits userland, between 32 and 64bits kernels?  Can this diff
be "fixed" somehow?  For example, I remember kernel/user memory
split kconfig option - is there similar thing for 32bits userspace
on 64bits kernel?

(And for the record, in case others will encounter the same issue:
the process started it all sees this:

SQL> startup
ORA-24323: value not allowed
ORA-03113: end-of-file on communication channel

and in the alert file we see this:

Errors in file /usr/oracle/8.1/rdbms/log/ora_3365.trc:
ORA-00445: background process "PMON" did not start after 120 seconds
)

Thanks!

/mjt

Re: vm: shmat() address for 32 vs 64 bit kernel

[Nicholas Miell]

On Thu, 2009-01-29 at 22:08 +0300, Michael Tokarev wrote:
> I were needed to run a legacy application which uses
> Oracle 8 database.  Tried to run it on a 64bit kernel,
> which is running on my machine for a long time, together
> with 32bits userland.  Everything worked but not oracle,
> which segfaults at startup on 64bits kernel, but works
> just fine when running the same but 32bits kernel
> (userspace is the same).
> 
> After much debugging I found this difference which may
> be relevant.  It's the address returned by shmat() routine.
> 
> Here are the relevant stuff for 32bits:
> 
> shmget(0x4d21f3bc, 579067904, IPC_CREAT|IPC_EXCL|0640) = 174358532
> shmat(174358532, 0, 0)            = 0x9569f000
> 
> Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.
> 
> And here's what happens with 64bits kernel:
> 
> shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640) = 294912
> shmat(294912, 0, 0)               = 0xd5548000
> 
> Here, the segment ends at 3965Mb.
> 
> And right after that shmat, oracle process terminates with SIGSEGV
> (on 64bits kernel, but continues just fine on 32bits).
> 
> I understand full well many bad points here: proprietary software,
> old legacy code, not supported, buggy.

Fortunately, compatibility is compatibility, and none of these are
relevant. :)

> But the question really is: why the difference in the first place,
> for 32bits userland, between 32 and 64bits kernels?  Can this diff
> be "fixed" somehow?  For example, I remember kernel/user memory
> split kconfig option - is there similar thing for 32bits userspace
> on 64bits kernel?
> 

On 32-bit systems, the kernel is mapped into the last gigabyte of the 4
GB address space for every process. On 64-bit systems, the kernel is
mapped in the negative half of the 64-bit address space for all
processes, which leaves the top gigabyte of the 4 GB address space free
for userspace use.

Most of the time this just means more memory for the processes that need
it, but (as you have discovered) sometimes it can cause problems, which
is why the setarch command has the --3gb flag.

> (And for the record, in case others will encounter the same issue:
> the process started it all sees this:
> 
> SQL> startup
> ORA-24323: value not allowed
> ORA-03113: end-of-file on communication channel
> 
> and in the alert file we see this:
> 
> Errors in file /usr/oracle/8.1/rdbms/log/ora_3365.trc:
> ORA-00445: background process "PMON" did not start after 120 seconds
> )
> 
> Thanks!
> 
> /mjt

-- 
Nicholas Miell <nmiell@comcast.net>

Re: vm: shmat() address for 32 vs 64 bit kernel

[Michael Tokarev]

Nicholas Miell wrote:
> On Thu, 2009-01-29 at 22:08 +0300, Michael Tokarev wrote:
>>
>> shmat(174358532, 0, 0)            = 0x9569f000
>> Here, the segment ends at (0x9569f000+579067904)/1024/1024 = 2942Mb.
>>
>> And here's what happens with 64bits kernel:
>>
>> shmget(0xc9a7c840, 579067904, IPC_CREAT|IPC_EXCL|0640) = 294912
>> Here, the segment ends at 3965Mb.
>>
[]
>> I understand full well many bad points here: proprietary software,
>> old legacy code, not supported, buggy.
> 
> Fortunately, compatibility is compatibility, and none of these are
> relevant. :)

Well, compatibility for 64bits kernel and 32bits userland it not
the first priority, so to say.

[]
> On 32-bit systems, the kernel is mapped into the last gigabyte of the 4
> GB address space for every process. On 64-bit systems, the kernel is
> mapped in the negative half of the 64-bit address space for all
> processes, which leaves the top gigabyte of the 4 GB address space free
> for userspace use.
> 
> Most of the time this just means more memory for the processes that need
> it, but (as you have discovered) sometimes it can cause problems, which
> is why the setarch command has the --3gb flag.

Oh.  Nice thing, I just discovered personality(2) stuff ;)
Thank you very much for the pointer!

Except of some.. interesting stuff here.  It does not quite work as
*I*'d expect (not to say it's wrong but just unexpected).  Namely,
when setarch is also 32bits, it can't execute anything after setting
ADDR_LIMIT_3GB (or PER_LINUX32_3GB).  execve(2) returns immediately
with EFAULT error.  But if doing the same from a 64bit executable,
it all works as expected.  Or when doing the same from 32bit exe
running on a 32bit kernel.  I.e. the only problematic case for
personality(PER_LINUX32_3GB) call is 32bit userspace on 64bit kernel.

personality(PER_LINUX32_3GB)            = 0
execve("/bin/echo", ["/bin/echo", "foo"], [/* 13 vars */]) = -1 EFAULT (Bad address)

Should it work? ;)

For now, it is running correctly when using a 64bits setarch executable
in-between.  Sure it will not work anymore if I'll reboot into 32bits
kernel ;)  (I run 64bits now because I'm trying to move data between
two databases, one old 32bits and another is 64 bits).

Thank you!

/mjt