possible problem in memory management

possible problem in memory management

[Matthias Fuchs]

[-- Attachment #1: Type: text/plain, Size: 1383 bytes --]

Hi,

in the last days we noticed some strange memory problems with Monta
Vista's kernel port for the IBM405GP/CR (kernel 2.4.2 and 2.4.0). Some
userspace programs crash when calling free() and the bash shell cannot
startup because it does not get memory (see attached bootlog).

Here aresome examples:

1) While unloading a custom kernel module, we get:
Trying to vfree() nonexistent vm area (da03c000). We had no problems on
x86 architecture and 2.4.0-test2.

2) see attached bootlog.

3) When using busybox 0.50, the busybox shell crashes, when the command
buffer is free()ed. This is for example done, when you press return at
the console. I put a debug message right before the free and behind it
and only the message before the free is printed !

We are shure that these problem have something todo the 2.4.x kernel for
the 405 !

Are problems like this known and solved issues ?

Matthias
--
-------------------------------------------------
\ Matthias Fuchs                                 \
 \ esd electronic system design Gmbh              \
  \ Vahrenwalder Straße 205                        \
   \ D-30165 Hannover                               \
    \ email: matthias.fuchs@esd-electronics.com      \
     \ phone: +49-511-37298-0                         \
      \ fax:   +49-511-37298-68                        \
       --------------------------------------------------

[-- Attachment #2: bootlog1.txt --]
[-- Type: text/plain, Size: 5261 bytes --]



ppcboot 0.7.1 (Jan  8 2001 - 16:20:48)

Initializing...
  CPU:   IBM PowerPC 405GP Rev. C at 198 MHz (PLB=99, OPB=49, EBC=33 MHz)
           PCI sync clock at 33 MHz, internal PCI arbiter enabled
           16 kB I-Cache 8 kB D-Cache
  Board: ### No HW ID - assuming CPCI405
  FPGA:  cpci4052.ncd s05xlvq100 2000/11/29 14:57:32
  DRAM:  16 MB
  FLASH:  4 MB
  IDE:   Bus 0: OK
    Device 0: Model: SanDisk SDCFB-64  Serial #: 213006F040  Capacity: 61.3 MB = 0.1 GB
  Input:  serial
  Output: serial

Hit any key to stop autoboot:  3 \b\b\b 2 \b\b\b 0
=> bootp
ENET Speed is 10 Mbs...
HALF duplex connection
BOOTP broadcast 1
ARP broadcast 1
TFTP from server 10.0.5.190; our IP address is 10.0.3.176
Filename '/tftpboot/pImage'. Size is 580 kB => 91000 Bytes
Load address: 0x100000
Loading: *\b####################################################################################################################
done

=> bootm
## Booting Linux kernel at 00100000 ...
   Image Name:   Test-Image
   Image Type:   PowerPC Linux Kernel Image (gzip compressed)
   Data Size:    593782 Bytes = 579 kB = 0 MB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
----------  progress:  0x200 id mach(): done
Linux version 2.4.2-mvista_010303 (frank@pc-linux-dev) (gcc version 2.95.2 19991030 (2.95.3 prerelease/franzo)) #7 Tue Mar 20 20:43:35 CET 2001
----------  progress:  0x3eab setup_arch: enter
----------  progress:  0x3eab setup_arch: bootmem
----------  progress:  0x3eab arch: exit
On node 0 totalpages: 4096
zone(0): 4096 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: bootargs1=root=/dev/nfs ip=10.0.3.176:10.0.5.190:10.0.0.79:255.255.0.0 nfsroot=10.0.5.190:/opt/hardhat/devkit/ppc/4xx/target
Warning: real time clock seems stuck!
Calibrating delay loop... 197.83 BogoMIPS
Memory: 14500k available (1064k kernel code, 436k data, 64k init, 0k highmem)
Dentry-cache hash table entries: 2048 (order: 2, 16384 bytes)
Buffer-cache hash table entries: 1024 (order: 0, 4096 bytes)
Page-cache hash table entries: 4096 (order: 2, 16384 bytes)
Inode-cache hash table entries: 1024 (order: 1, 8192 bytes)
POSIX conformance testing by UNIFIX
PCI: Probing PCI hardware
PCI: Cannot allocate resource region 0 of device 00:00.0
PCI: Cannot allocate resource region 2 of device 00:00.0
PCI: Cannot allocate resource region 3 of device 00:00.0
PCI: Cannot allocate resource region 4 of device 00:00.0
PCI: Cannot allocate resource region 5 of device 00:00.0
----------  progress:  0xffff
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Starting kswapd v1.8
i2c-core.o: i2c core module
i2c-dev.o: i2c /dev entries driver module
i2c-core.o: driver i2c-dev dummy driver registered.
pty: 256 Unix98 ptys configured
block: queued sectors max/low 9554kB/3184kB, 64 slots per queue
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
loop: enabling 8 loop devices
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
cpci405ide: physical address=f0100000
cpci405ide: mapped address=da000000
ide0: CPCI405 IDE interface
hda: SanDisk SDCFB-64, ATA DISK drive
ide0 at 0xda000000-0xda000007,0xda00000e on irq 31
hda: 125440 sectors (64 MB) w/1KiB Cache, CHS=490/8/32
Partition check:
 hda: hda1
phy speed read failed
phy duplex read failed
<5>eth0: PPC405 EMAC 10 Mbs Half duplex MAC 00:02:27:00:00:6b
Serial driver version 5.02 (2000-08-09) with MANY_PORTS SHARE_IRQ SERIAL_PCI enabled
ttyS00 at 0x0000 (irq = 0) is a 16550A
ttyS01 at 0x0000 (irq = 1) is a 16550A
Real Time Clock Driver v1.10d
PPC 405 gpio driver version 00.08.02.d
PPC 405 watchdog driver v0.3
PPP generic driver version 2.4.1
PPP Deflate Compression module registered
PPC 405 iic (i2c) algorithm module 2001.03.01
iic_ppc405_init: PPC 405 iic adapter module
enable_irq(2) unbalanced
i2c-dev.o: Registered 'PPC405 IIC adapter' as minor 0
i2c-core.o: adapter PPC405 IIC adapter registered as adapter 0.
iic_ppc405_init: found device at 0xef600500.
NET4: Linux TCP/IP 1.0 for NET4.0
IP Protocols: ICMP, UDP, TCP
IP: routing cache hash table of 512 buckets, 4Kbytes
TCP: Hash tables configured (established 1024 bind 1024)
Reset ethernet interfaces
eth0: PPC405 Enet open completed
NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
Looking up port of RPC 100003/2 on 10.0.5.190
Looking up port of RPC 100005/2 on 10.0.5.190
VFS: Mounted root (nfs filesystem) readonly.
Freeing unused kernel memory: 64k init 4k openfirmware
INIT: version 2.77 booting
sh: xmalloc: cannot INIT: Entering runlevel: 3
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
sh: xmalloc: cannot allocate 8 bytes (0 bytes allocated)
INIT: Id "sh" respawning too fast: disabled for 5 minutes
INIT: no more processes left in this runlevel

Re: possible problem in memory management

[Dan Malek]

Matthias Fuchs wrote:

>   CPU:   IBM PowerPC 405GP Rev. C at 198 MHz (PLB=99, OPB=49, EBC=33 MHz)

Get yourself an up to date Rev. D CPU.  The Rev. C workarounds in
the kernel were reasonable attempts, but probably aren't complete.
Since the Rev. D fixes all of these problems, we stopped debugging
all of the little Rev. C problems because they were just too time
consuming.

You also need specially patched user libraries for anything older
than Rev. D, and you can't use these patches libraries on Rev. D,
so we stopped supporting that as well and simply suggest (require)
the upgrade to the working Rev. D silicon.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: possible problem in memory management

[Frank Rowand]

Dan Malek wrote:
>
> Matthias Fuchs wrote:
>
> >   CPU:   IBM PowerPC 405GP Rev. C at 198 MHz (PLB=99, OPB=49, EBC=33 MHz)
>
> Get yourself an up to date Rev. D CPU.  The Rev. C workarounds in
> the kernel were reasonable attempts, but probably aren't complete.

Dan is correct.  We did not do an exhaustive scan of the kernel to add
workarounds in all places they might be necesary, just enough to make the
kernel fairly stable and usable.  Even in the places where the workarounds
are in place, they are not guaranteed to be 100% effective.


> Since the Rev. D fixes all of these problems, we stopped debugging
> all of the little Rev. C problems because they were just too time
> consuming.
>
> You also need specially patched user libraries for anything older
> than Rev. D, and you can't use these patches libraries on Rev. D,
> so we stopped supporting that as well and simply suggest (require)
> the upgrade to the working Rev. D silicon.
>
>         -- Dan

-Frank
--
Frank Rowand <frank_rowand@mvista.com>
MontaVista Software, Inc

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: possible problem in memory management

[Kenneth Johansson]

Frank Rowand wrote:
>
> Dan Malek wrote:
> >
> > Matthias Fuchs wrote:
> >
> > >   CPU:   IBM PowerPC 405GP Rev. C at 198 MHz (PLB=99, OPB=49, EBC=33 MHz)
> >
> > Get yourself an up to date Rev. D CPU.  The Rev. C workarounds in
> > the kernel were reasonable attempts, but probably aren't complete.
>
> Dan is correct.  We did not do an exhaustive scan of the kernel to add
> workarounds in all places they might be necesary, just enough to make the
> kernel fairly stable and usable.  Even in the places where the workarounds
> are in place, they are not guaranteed to be 100% effective.

But I do think there is a memory bug inte the version of the kernel he is using.
I can produce a error state at will running debian (potato) on that kernel.
Running emacs or simply login over telnet puts it into that state. I currently
is not working to resolv the problem so I don't have more infomation than that
it fails, sorry.




--
Kenneth Johansson
Ericsson Business Innovation AB   Tel: +46 8 404 71 83
Viderögatan 3                     Fax: +46 8 404 72 72
164 80 Stockholm                  kenneth.johansson@inn.ericsson.se

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: possible problem in memory management

[Dan Malek]

Kenneth Johansson wrote:

> But I do think there is a memory bug inte the version of the kernel he is using.

There certainly could be, but the Rev.C errata can show up as this
kind of symptom.  We haven't found anything on the Rev. D with our
QA testing.

> Running emacs or simply login over telnet puts it into that state.

I know that login over telnet works for me.  That isn't much of test,
as it is nice to be able to debug memory problems with carefully
controlled tests so you know if you have really corrected problems.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: possible problem in memory management

[Kenneth Johansson]

Dan Malek wrote:
>
> Kenneth Johansson wrote:
> > Running emacs or simply login over telnet puts it into that state.
>
> I know that login over telnet works for me.  That isn't much of test,
> as it is nice to be able to debug memory problems with carefully
> controlled tests so you know if you have really corrected problems.

You'r right I did some more testing and emacs was a false alarm. It turns out
that the offender is the tcpd wrapper that debian has in inetd.conf.
In the default configuration doing a telnet,finger or talk puts the system in
this "funny" state. Removing the wrapper and everything is fine.

I also found out that the man command get segmentation faults after tcpd has run.

So to sum it up "man ls" and "dpkg --configure" both works OK before the first run of
/usr/sbin/tcpd from debian (potato). After that they alwas get segmentation fault.

I think this is a good enough test however I don't even have a theory on why it's happening.


--
Kenneth Johansson
Ericsson Business Innovation AB   Tel: +46 8 404 71 83
Viderögatan 3                     Fax: +46 8 404 72 72
164 80 Stockholm                  kenneth.johansson@inn.ericsson.se

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Inquiry : ECC on embedded system

[David Chiu]

Just curious if there are any efforts in ECC memory support (handling of
correctable and uncorrectable memory errors) on PPC embedded systems. I
vaguely recall finding something through google search, although that effort
appears to be largely directed toward x86 desktop.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/