2.6.9, 64bit, 4GB memory => panics ...

2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

Dear Kernel Gurus ! :*)

I just installed Fedora Core 3, x86_64. It boots fine, but
crashes. The "crash" part occurs as soon as I start any
app that apparently uses some "larger" amount of memory.
At first I thought there was a problem with X, and my video
card. So I tried to login on one of the virtual screens (ALT-F1).
Here I was able to see a kernel panic screen when I started
several random apps. I am working in duplicating that and
capturing the full panic message.

My hardware is:
Tyan Tiger K8W (S2875) with two Opterons, 4 SATA drives
(with softraid) and a low end Nvidia MX-400 video card.
My BIOS version is 2.03.

I can use the system 100% with only 2GB of memory installed.
I have run the memtest86+ for over a day and did not get any
errors (with the full 4gb installed). Previously I was running
Fedora Core 2 but 32 bit, also with the 2.6.9 kernel and did
not have any problems with 4 GB of memory. My guess is the combination
of 64 bit and 4GB is causing the problem.

I have tried to compile my own kernel, and also got the 667
and 681 releases (for 2.6.9) from Fedora.org. Problem seems
to be persistent.

Here is the ver_linux output
-----------------------------------------------------------
Linux cpu10 2.6.9-1.681_FC3smp #1 SMP Thu Nov 18 15:30:11 EST 2004
x86_64 x86_64 x86_64 GNU
/Linux

Gnu C                  3.4.2
Gnu make               3.80
binutils               2.15.92.0.2
util-linux             2.12a
mount                  2.12a
module-init-tools      3.1-pre5
e2fsprogs              1.35
reiserfsprogs          line
reiser4progs           line
pcmcia-cs              3.2.7
quota-tools            3.12.
PPP                    2.4.2
isdn4k-utils           3.3
nfs-utils              1.0.6
Linux C Library        2.3.3
Dynamic linker (ldd)   2.3.3
Procps                 3.2.3
Net-tools              1.60
Kbd                    1.12
Sh-utils               5.2.1
Modules Loaded         autofs4 i2c_dev i2c_core nfs lockd sunrpc ds
yenta_socket pcmcia_cor
e dm_mod button battery ac nvidia md5 ipv6 ohci1394 ieee1394 ohci_hcd
uhci_hcd ehci_hcd hw_
random snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm
snd_timer snd_page_all
oc gameport snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore
e1000 floppy ext3 jbd
raid5 xor sata_sil libata sd_mod scsi_mod
-----------------------------------------------------------

Any suggestions what may be causing the crashes ?

I do not subscribe to this list, so would sincerely appreciate
a cc of any replies. I'm sure I forgot to mention something,
please let me know what other information can be help full to
nail this down.

Thanks a lot in advance !

Kind Regards,
rudi

Re: 2.6.9, 64bit, 4GB memory => panics ...

[William Lee Irwin III]

On Fri, Dec 03, 2004 at 06:20:34PM +0700, Rudolf Usselmann wrote:
> I just installed Fedora Core 3, x86_64. It boots fine, but
> crashes. The "crash" part occurs as soon as I start any
> app that apparently uses some "larger" amount of memory.
> At first I thought there was a problem with X, and my video
> card. So I tried to login on one of the virtual screens (ALT-F1).
> Here I was able to see a kernel panic screen when I started
> several random apps. I am working in duplicating that and
> capturing the full panic message.

The full panic message would help a lot. We can't do much without it.


-- wli

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

On Fri, 2004-12-03 at 18:37, William Lee Irwin III wrote:
> The full panic message would help a lot. We can't do much without it.
> 
> -- wli

I was hoping to be able to get the entire panic screen, but
am only getting the bottom of it ...

Does this help ?


FS: 0000002a95cea6e0(0000) GS: ffffffff80516b00(0000) knlGS: 0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000002 CR3: 0000000000011000 CR4: 00000000000006ed
Process as (pid: 4312, threadinfo 00000101735e0000, task 00000101746c9070)
Stack: ffffffff80141160 0000000000000012 0000010176b87fa0 0000000000000000
       000001000709a200 0000010175b37730 000001017aed32c0 000000000000005d
       ffffffff801c6f1d 000001016bfd1310
Call Trace: <ffffffff80141160>{__mod_timer+160}    <ffffffff801c6f1d>{start_this_handle+605}
            <ffffffff80159bcd>{filemap_nopage+413} <ffffffff8019335c>{__d_lookup+300}
            <ffffffff80187c47>{do_lookup+55}       <ffffffff801c724f>{journal_start+239}
            <ffffffff801c20bd>{ext3_create+45}     <ffffffff8018957d>{vfs_create+237}
            <ffffffff80189a00>{open_namei+432}     <ffffffff801796a7>{filp_open+39}
            <ffffffff801797be>{get_unused_fd+238}  <ffffffff8017990c>{sys_open+76}
            <ffffffff801105f6>{system_call+126}

Code: f0 fe 0f 0f 88 8e 02 00 00 c3 66 66 90 66 66 90 c6 07 01 c3
RIP <ffffffff8038443e0>{spin_lock+0} RSP <00000101735e1cc0>
CR2: 0000000000000002



Kind Regards,
rudi

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

FYI, I tried 2.9.10-rc3 kernel, same problem.

Can anyone guide me or give me some pointers how I can enable
login on serial ports - that would hopefully allow me to capture
more/longer kernal panic screens. Doing an rlogin doesn't help,
the kernel panic is still going to the root console (24 lines
visible).

Any other tips/pointers as to how to capture kernel panics or
what else I can do to help you guys to help me would be highly
appreciated. :*)

Kind Regards,
rudi

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Zwane Mwaikambo]

Hello Rudolf,

On Sun, 5 Dec 2004, Rudolf Usselmann wrote:

> FYI, I tried 2.9.10-rc3 kernel, same problem.
> 
> Can anyone guide me or give me some pointers how I can enable
> login on serial ports - that would hopefully allow me to capture
> more/longer kernal panic screens. Doing an rlogin doesn't help,
> the kernel panic is still going to the root console (24 lines
> visible).
> 
> Any other tips/pointers as to how to capture kernel panics or
> what else I can do to help you guys to help me would be highly
> appreciated. :*)

Make sure that you have the following option in your kernel;

CONFIG_SERIAL_8250_CONSOLE=y

Then boot the kernel with this kernel parameter;

console=ttyS0,38400

Make necessary adjustments for baud rate and serial port. Then you may 
additionally add a getty on the console with something akin to the 
following /etc/inittab entry;

7:35:respawn:/sbin/agetty 38400 ttyS0

Finally all you need is a program on the monitoring system, minicom 
should suffice if you have nothing else.

	Zwane

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

On Sun, 2004-12-05 at 13:26, Zwane Mwaikambo wrote:
> Make sure that you have the following option in your kernel;
> 
> CONFIG_SERIAL_8250_CONSOLE=y
> 
...
> 
> 7:35:respawn:/sbin/agetty 38400 ttyS0
> 
> Finally all you need is a program on the monitoring system, minicom 
> should suffice if you have nothing else.
> 
> 	Zwane

Thanks a lot Zwane, appreciate these tips !

I am now trying to be better prepared for my next 4gb test, and
have been working on a small program that should "eat memory".
I'm not sure I am doing the right thing though ...

The idea was to create an app that would consume a lot of memory
without touching any other system resources.

At first I used ksysguard to watch the memory consumption.
Interestingly enough, it was "wired": The VmSize column seems
to jump randomly, once I get 1,051 than I will see 3,148,615
than I will get 2,789 than I might see 1,073,976 than it will
give me 3,148 .... all during the same run of my mem eater app.

... just mentioning it here in case there is a connection between
my problem and this wired display ...

Now this testing was done with 2GB at first ... no kernel panics
with 2GB.

Than I decided to use /proc/meminfo, but that suggests that I
am not using all memory at all ! With 2GB malloc fails after
510 iterations (510 MB ???) ... (I have a 10GB swap partition)
And /proc/meminfo confirms that:
MemTotal:      2056528 kB
MemFree:       1526424 kB

About 1/2 gig is used, I checked limits, and that said unlimited ...
What else can block memory usage ? May be I just need to start my
app several times ?!

Hmm, after forking mem eater 6 times I get:
MemTotal:      2056528 kB
MemFree:        610000 kB
Which is almost the same as after 4 and 5 spins ...

Any ideas/suggestion ?

Thanks a lot guys & gals !

Kind Regards,
rudi


// -------------------------- eat mem ---------------------------------

#include "stdio.h"
#include "stdlib.h"

int main() {

int mem[10000];
int i, n;

for(i=0;i<2000;i++) {
	printf("Doing alloc %0d ...\n",i);
	mem[i] = (int)malloc(1024*1024*1024);
	if(mem[i] == NULL)
		printf("Malloc failed ...\n");
	else
		for(n=0;n<(1024*1024*1024);n=n+640)	mem[i] = n;
   }

while(1);

return(0);
}

// --------------------------------------------------------------------

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Adam Heath]

On Sun, 5 Dec 2004, Rudolf Usselmann wrote:

> int mem[10000];
> int i, n;
>
> for(i=0;i<2000;i++) {
> 	printf("Doing alloc %0d ...\n",i);
> 	mem[i] = (int)malloc(1024*1024*1024);

allocate a 1 gig block, but store it in an int array?  That's wrong.

> 	if(mem[i] == NULL)
> 		printf("Malloc failed ...\n");
> 	else
> 		for(n=0;n<(1024*1024*1024);n=n+640)	mem[i] = n;

You alter n, but then always only set mem[i], without varying i.

Your program is buggy, and memleaks.

Plus, the kernel uses overcommit by default, and since you aren't actually
touching the memory you are allocating, you are only limited by the address
space size allowed per process(depends on how you configured the kernel).

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

On Sun, 2004-12-05 at 14:20, Adam Heath wrote:
> On Sun, 5 Dec 2004, Rudolf Usselmann wrote:
> 
> > int mem[10000];
> > int i, n;
> >
> > for(i=0;i<2000;i++) {
> > 	printf("Doing alloc %0d ...\n",i);
> > 	mem[i] = (int)malloc(1024*1024*1024);
> 
> allocate a 1 gig block, but store it in an int array?  That's wrong.
> 
> > 	if(mem[i] == NULL)
> > 		printf("Malloc failed ...\n");
> > 	else
> > 		for(n=0;n<(1024*1024*1024);n=n+640)	mem[i] = n;
> 
> You alter n, but then always only set mem[i], without varying i.
> 
> Your program is buggy, and memleaks.
> 
> Plus, the kernel uses overcommit by default, and since you aren't actually
> touching the memory you are allocating, you are only limited by the address
> space size allowed per process(depends on how you configured the kernel).



Thanks for pointing that out Adam ! Will this work better:



#include "stdio.h"
#include "stdlib.h"

int main() {

char *mem[10000];
int i, n;

for(i=0;i<2000;i++) {
	printf("Doing alloc %0d ...\n",i);
	mem[i] = (char *)malloc(1024*1024*1024);
	if(mem[i] == NULL)
		printf("Malloc failed ...\n");
	else
		for(n=0;n<(1024*1024*1024);n=n+640)	mem[i][n] = n;
   }

while(1);

return(0);
}


Thanks !
rudi

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Zwane Mwaikambo]

On Sun, 5 Dec 2004, Rudolf Usselmann wrote:

> // -------------------------- eat mem ---------------------------------
> 
> #include "stdio.h"
> #include "stdlib.h"
> 
> int main() {
> 
> int mem[10000];

Array of integers, you probably want int *mem[10000] if anything.

> int i, n;
> 
> for(i=0;i<2000;i++) {
> 	printf("Doing alloc %0d ...\n",i);
> 	mem[i] = (int)malloc(1024*1024*1024);

Wrongly assigning pointer to signed integer, casting can sometimes be 
evil.

> 	if(mem[i] == NULL)
> 		printf("Malloc failed ...\n");
> 	else
> 		for(n=0;n<(1024*1024*1024);n=n+640)	mem[i] = n;

You lose the pointer value here and do not touch the allocated memory, 
which means that the VM isn't forced to commit the memory you allocated. 
You also want to be looking at the Committed_AS and Mapped fields of 
/proc/meminfo

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]


I think I got it !!!

After fixing my mem eater program (per Adam and Zwane's
suggestions), I get the attached "end-less" panic ...
This happens immediately when I start the a.out ...

So does this make any sense to you guys ? What is wrong with
my kernel/system ?

Thanks a lot guys !!!


Kind Regards,
rudi


[-- Attachment #2: boot2.log.gz --]
[-- Type: application/x-gzip, Size: 11447 bytes --]

Re: 2.6.9, 64bit, 4GB memory => panics ...

[William Lee Irwin III]

On Sun, Dec 05, 2004 at 02:44:45PM +0700, Rudolf Usselmann wrote:
> I think I got it !!!
> After fixing my mem eater program (per Adam and Zwane's
> suggestions), I get the attached "end-less" panic ...
> This happens immediately when I start the a.out ...
> So does this make any sense to you guys ? What is wrong with
> my kernel/system ?
> Thanks a lot guys !!!

Much better. Looks like someone broke the VM from that.


-- wli

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

On Sun, 2004-12-05 at 14:53, William Lee Irwin III wrote:
> 
> Much better. Looks like someone broke the VM from that.
> 
> 
> -- wli

:*)


Do I need to notify the maintainer of that specific code ?
Or whats the standard procedure for bug reports ?

I could not identify a maintainer in the MAINTAINERS file,
and memory.c/mm has a last modified in 1999 entry !

What should I do next ?

Thanks !

Kind Regards,
rudi

Re: 2.6.9, 64bit, 4GB memory => panics ...

[Rudolf Usselmann]

Hmm, ok my eat mem was useless. Could not get the system to
crash ...

Thank I was thinking and it occured to me that each time I was
experiencing the crashes thgere was heavy file system activity.

So I tried two things:

tar cf /dev/null <large directory 500+MB>

That went ok, than I did:

tar cf archive.tar <large directory 500+MB>

And that gave me:

Unable to handle kernel paging request at 0000000000001170 RIP:
<ffffffff8015df54>{kmem_getpages+132}
PML4 17102c067 PGD 171058067 PMD 0
Oops: 0000 [1] SMP
CPU 1
Modules linked in: nvidia ipv6 autofs4 sunrpc binfmt_misc dm_mod button
battery ac ohci_hcy
Pid: 3454, comm: tar Tainted: P   2.6.9RU1.1
RIP: 0010:[<ffffffff8015df54>] <ffffffff8015df54>{kmem_getpages+132}
RSP: 0018:000001017100da38  EFLAGS: 00010213
RAX: ffffffff7fffffff RBX: 000001017ffc9680 RCX: 0000000000000000
RDX: 000001000000f500 RSI: 000001000000f580 RDI: 000001000000f840
RBP: 0000010037ffe000 R08: 000001016ac59000 R09: 000001017ffc96c8
R10: 0000000000001000 R11: 0000000000000000 R12: 000001017ffc9680
R13: 000001017ffc96a8 R14: 0000000000000000 R15: 0000000000000000
FS:  0000002a95572b00(0000) GS:ffffffff80512980(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000001170 CR3: 0000000037f08000 CR4: 00000000000006e0
Process tar (pid: 3454, threadinfo 000001017100c000, task
000001017fc5a7d0)
Stack: 000001016b3ca088 0000000000001f7d 0000000000000001
ffffffff8015e720
       000001017ffc9708 0000000000000001 0000005000000050
0000000000000000
       0000000000000000 0000000000001000
Call Trace:<ffffffff8015e720>{cache_alloc_refill+656}
<ffffffff8015e416>{kmem_cache_alloc+
       <ffffffff8017ac91>{alloc_buffer_head+17}
<ffffffff8017b307>{create_buffers+39}
       <ffffffff8017b3a4>{create_empty_buffers+20}
<ffffffff8017baa5>{__block_prepare_writ
       <ffffffff801bb8f0>{ext3_get_block+0}
<ffffffff8017be0a>{block_prepare_write+26}
       <ffffffff801b9a25>{ext3_prepare_write+101}
<ffffffff80157e12>{generic_file_buffered
       <ffffffff80191f4e>{inode_update_time+158}
<ffffffff80158686>{generic_file_aio_write
       <ffffffff8015875f>{generic_file_aio_write+127}
<ffffffff801b7f33>{ext3_file_write+3
       <ffffffff80177a7d>{do_sync_write+173}
<ffffffff80186dc9>{may_open+105}
       <ffffffff80136360>{autoremove_wake_function+0}
<ffffffff803803e9>{thread_return+41}
       <ffffffff8019418e>{dnotify_parent+46}
<ffffffff80177b94>{vfs_write+228}
       <ffffffff80177cc3>{sys_write+83}
<ffffffff801105d6>{system_call+126}


Code: 48 8b 91 70 11 00 00 76 07 b8 00 00 00 80 eb 0a 48 b8 00 00
RIP <ffffffff8015df54>{kmem_getpages+132} RSP <000001017100da38>
CR2: 0000000000001170
 Killed
[rudi@cpu10 /home]$ <1>Unable to handle kernel paging request at
0000000000001170 RIP:
<ffffffff8015ab8a>{free_pages+154}
PML4 179d73067 PGD 0
Oops: 0000 [2] SMP
CPU 1
Modules linked in: nvidia ipv6 autofs4 sunrpc binfmt_misc dm_mod button
battery ac ohci_hcy
Pid: 2078, comm: syslogd Tainted: P   2.6.9RU1.1
RIP: 0010:[<ffffffff8015ab8a>] <ffffffff8015ab8a>{free_pages+154}
RSP: 0018:000001017cdf9dd0  EFLAGS: 00010213
RAX: ffffffff7fffffff RBX: 000001016ac5b010 RCX: 0000000000000000
RDX: 000001007fc3b838 RSI: 0000000000000000 RDI: 000001016ac5b000
RBP: 000001016ac5b010 R08: 000001000000f000 R09: 000001017fd01f48
R10: 000001017fd01f40 R11: ffffffff8031b400 R12: 0000000000000000
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000001
FS:  0000002a957a3b00(0000) GS:ffffffff80512980(0000)
knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000001170 CR3: 0000000037f08000 CR4: 00000000000006e0
Process syslogd (pid: 2078, threadinfo 000001017cdf8000, task
000001017e8757d0)
Stack: ffffffff8018b543 000001017e8cac80 0000000000000002
0000000000000001
       ffffffff8018ba43 0000000000000000 0000000000000000
0000000000000001
       0000000000000000 0000000000000000
Call Trace:<ffffffff8018b543>{poll_freewait+67}
<ffffffff8018ba43>{do_select+1027}
       <ffffffff8018b550>{__pollwait+0}
<ffffffff80199868>{__writeback_single_inode+728}
       <ffffffff8018bde5>{sys_select+885}
<ffffffff801105d6>{system_call+126}


Code: 48 8b 91 70 11 00 00 76 0d b8 00 00 00 80 eb 10 66 66 90 66
RIP <ffffffff8015ab8a>{free_pages+154} RSP <000001017cdf9dd0>
CR2: 0000000000001170



-- 
rudi               
=============================================================
Rudolf Usselmann,  ASICS World Services,  http://www.asics.ws
Your Partner for IP Cores, Design, Verification and Synthesis