2.4.20-aa and LARGE Squid process -> SIGSEGV

2.4.20-aa and LARGE Squid process -> SIGSEGV

[Ralf Hildebrandt]

Hi!

Right now we're trying to implement a large scale Squid proxy on
Debian/testing. We're using 2.4.20-aa and Squid-2.4.7-1.

We're encountering sporadic crashes of the squid children (SIGSEGV,
signal 11). We were investigating in several directions:

* the Kernel has highmem support enabled (we have 2GB RAM and 4 GB swap)

* we tried different kernels (with or without highmem support)

* we tried another squid version (Squid-2.4.6 from the "stable"
  distribution of Debian)

* We recompiled Squid using gcc-3.2, since -- according to the FAQ --
  squid may crash with signal 11 with optimization enabled when using
  gcc-2.95.4 (Debian uses gcc-2.95.4, but still build squid using -O!)

* we closely observed dmesg, messages and syslog. No oddities were
  found. Squid simply crashes.

* we tried both ufs and aufs as cache filesystems, since the FAQ tells
  us the async I/O may have bugs.

* We use two identical machines -- the crash happens on both machines.
  Same CPU, disks, RAM, manufacturer, heck -- even the same series!
  
To no avail -- Squid simply crashes from time to time. It's
impossible to predict when.

Then we wrote a program which allocates large amounts of memory:

--- snip ---
#include <stdlib.h>
#include <stdio.h>

main(){
  char *buf;
  long c;
  FILE *fp;

  fp = fopen("/dev/null","a");
  while(1){
    buf = (char *)malloc(100000000);
    c = random();
    if (c > 100000000)
      continue;
    fprintf(fp,"%c",buf[c]);
    printf("hier\n");
  }
}
--- snip ---

And we found that this program will be killed with a SIGSEGV as well.

So, what are we doing wrong? Since we have 2GB RAM and 4GB swap,
shouldn't the machine start to swap and execute the program above
successfully?

# cat /proc/meminfo 
        total:    used:    free:  shared: buffers:  cached:
Mem:  2119065600 2111709184  7356416        0   299008 1438240768
Swap: 4293509120        0 4293509120
MemTotal:      2069400 kB
MemFree:          7184 kB
MemShared:           0 kB
Buffers:           292 kB
Cached:        1404532 kB
SwapCached:          0 kB
Active:         387796 kB
Inactive:      1317064 kB
HighTotal:     1703872 kB
HighFree:         2960 kB
LowTotal:       365528 kB
LowFree:          4224 kB
SwapTotal:     4192880 kB
SwapFree:      4192880 kB
BigFree:             0 kB

# cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 15
model           : 2
model name      : Intel(R) Xeon(TM) CPU 2.80GHz
stepping        : 7
cpu MHz         : 2785.629
cache size      : 512 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm
bogomips        : 5557.45

# cat /proc/swaps   
Filename                        Type            Size    Used    Priority
/dev/sda6                       partition       2096440 0       -1
/dev/sda7                       partition       2096440 0       -2

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
I've got the perfect system.  I never need to do maintenance on it, or
software upgrades, patches, or anything.  It's great.  It never wakes me up,
or gets hacked into.  It's completely perfect.  That was the first step in
my plan to build the perfect Postfix system. 
The second step is to plug it in. 

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[J.A. Magallon]

On 2002.12.20 Ralf Hildebrandt wrote:
>Hi!
[real problem snipped]
>
>Then we wrote a program which allocates large amounts of memory:
>
>--- snip ---
>#include <stdlib.h>
>#include <stdio.h>
>
>main(){
>  char *buf;
>  long c;
>  FILE *fp;
>
>  fp = fopen("/dev/null","a");
>  while(1){
>    buf = (char *)malloc(100000000);
>    c = random();
>    if (c > 100000000)
>      continue;
>    fprintf(fp,"%c",buf[c]);
>    printf("hier\n");
>  }
>}
>--- snip ---
>
>And we found that this program will be killed with a SIGSEGV as well.
>

Normal. You are running OOM. Look at what you do:

    while (1)
    {
         malloc(much mem)
         // do not free the mem !!!!
    }

So in a couple steps you are OOM.

I suppose what you want to do is

    buf = malloc(...)
    while (1)
        touch random page

But...you 'touch' is read-only (the printf), so the page will never
really be allocated. Try with this:

#include <stdlib.h>

// 4Gb
#define SZ 4*1024*1024*1024

main(){
    char *buf;

    buf = malloc(SZ);
    if (!buf)
    {
        perror("bad try");
        exit(1);
    }
    while(1){
        buf[random()%SZ] = 0;
    }
}

Ah, with 2Gb of ram you will need to compile with 3Gb userspace,
to let a one only process allocate a chunk of mem that does not fit
into core memory. Or, easier, run several instances...

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.1 (Cooker) for i586
Linux 2.4.20-jam2 (gcc 3.2 (Mandrake Linux 9.1 3.2-4mdk))

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[Ralf Hildebrandt]

* J.A. Magallon <jamagallon@able.es>:

> Normal. You are running OOM. Look at what you do:

But that manifests itself in a segfault? The kernel never logs an OOM
in the logs.

> Ah, with 2Gb of ram you will need to compile with 3Gb userspace,
> to let a one only process allocate a chunk of mem that does not fit
> into core memory. Or, easier, run several instances...

Right now we're running a Xeon with 2GB - from the menuconfig:

      (4GB) High Memory Support

This was based on the help in the kernel config:

"If the machine has between 1 and 4 Gigabytes physical RAM, then
answer "4GB" here."

      (3.5GB) User address space size
      [*] HIGHMEM I/O support

dmesg says:

Linux version 2.4.20aa1 (root@spidergirl) (gcc version 2.95.4 20011002
(Debian prerelease)) #1 Thu Dec 19 14:35:35 CET 2002
BIOS-provided physical RAM map:
 BIOS-e820: 0000000000000000 - 00000000000a0000 (usable)
 BIOS-e820: 0000000000100000 - 000000007fff0000 (usable)
 BIOS-e820: 000000007fff0000 - 000000007fffec00 (ACPI data)
 BIOS-e820: 000000007fffec00 - 000000007ffff000 (reserved)
 BIOS-e820: 00000000fec00000 - 00000000fec10000 (reserved)
 BIOS-e820: 00000000fee00000 - 00000000fee10000 (reserved)
 BIOS-e820: 00000000fff80000 - 0000000100000000 (reserved)
1663MB HIGHMEM available.
384MB LOWMEM available.
...
Calibrating delay loop... 5557.45 BogoMIPS
Memory: 2069264k/2097088k available (1659k kernel code, 27440k
reserved, 651k data, 136k init, 1703872k highmem)
Dentry cache hash table entries: 262144 (order: 9, 2097152 bytes)
Inode cache hash table entries: 131072 (order: 8, 1048576 bytes)
Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
Buffer-cache hash table entries: 131072 (order: 7, 524288 bytes)
Page-cache hash table entries: 524288 (order: 9, 2097152 bytes)
CPU: L1 I cache: 0K, L1 D cache: 8K
CPU: L2 cache: 512K
Intel machine check architecture supported.
Intel machine check reporting enabled on CPU#0.
CPU:     After generic, caps: bfebfbff 00000000 00000000 00000000
CPU:             Common caps: bfebfbff 00000000 00000000 00000000
CPU: Intel(R) Xeon(TM) CPU 2.80GHz stepping 07
...
-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
"All programmers are playwrights and all computers are lousy actors."  -Anon.

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[J.A. Magallon]

On 2002.12.20 Ralf Hildebrandt wrote:
>* J.A. Magallon <jamagallon@able.es>:
>
>> Normal. You are running OOM. Look at what you do:
>
>But that manifests itself in a segfault? The kernel never logs an OOM
>in the logs.
>

Well, I used the OOM term, but I suspect it 'technically' only applies
to the situation when a kernel thread needs memory to do its work, and
it can not allocate it. The system can not continue without that memory,
you are OOM, and the OOM killer will begin to kill other things.

For user space memory, there is no real OOM state. The system (glibc) just
does not give you the memory, returns NULL in the malloc, and it is your
responsibility to check malloc's return value. If you do not check it,
you try to access a null pointer and _bang_. So in your case, after enough
iterations on malloc() without free(), it returns NULL and you fall into
a null pointer dereference.

AFAIK.

-- 
J.A. Magallon <jamagallon@able.es>      \                 Software is like sex:
werewolf.able.es                         \           It's better when it's free
Mandrake Linux release 9.1 (Cooker) for i586
Linux 2.4.20-jam2 (gcc 3.2.1 (Mandrake Linux 9.1 3.2.1-2mdk))

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[Ralf Hildebrandt]

* J.A. Magallon <jamagallon@able.es>:

> For user space memory, there is no real OOM state. The system (glibc) just
> does not give you the memory, returns NULL in the malloc, and it is your
> responsibility to check malloc's return value. If you do not check it,
> you try to access a null pointer and _bang_. So in your case, after enough
> iterations on malloc() without free(), it returns NULL and you fall into
> a null pointer dereference.

Ergo: Squid is br0ken.

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
Windows is the answer, but only if the question was 'what is the
intellectual equivalent of being a galley slave?' 

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[Reuben Farrelly]

No, squid is not br0ken in this fashion.  If squid cannot be allocated 
enough memory by the system, it logs a message and _dies_.  Relevant files 
to look at in your squid source are squid/lib/util.c for xcalloc() and 
xmalloc().

Aside from this, if squid ever does get to the point of swapping, it is 
misconfigured and your performance has just gone to hell anyway...  (see 
the FAQ at www.squid-cache.org)

Questions relating to squid performance and stability should really be 
discussed on the squid users mailing list, you can find details on this 
list at http://www.squid-cache.org/mailing-lists.html

Reuben


At 08:52 AM 21/12/2002 +0100, Ralf Hildebrandt wrote:
>* J.A. Magallon <jamagallon@able.es>:
>
> > For user space memory, there is no real OOM state. The system (glibc) just
> > does not give you the memory, returns NULL in the malloc, and it is your
> > responsibility to check malloc's return value. If you do not check it,
> > you try to access a null pointer and _bang_. So in your case, after enough
> > iterations on malloc() without free(), it returns NULL and you fall into
> > a null pointer dereference.
>
>Ergo: Squid is br0ken.
>
>--
>Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
>Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
>Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
>Windows is the answer, but only if the question was 'what is the
>intellectual equivalent of being a galley slave?'
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[Ralf Hildebrandt]

* Reuben Farrelly <reuben-linux@reub.net>:

> No, squid is not br0ken in this fashion.  If squid cannot be allocated 
> enough memory by the system, it logs a message and _dies_.  Relevant files 
> to look at in your squid source are squid/lib/util.c for xcalloc() and 
> xmalloc().

Why can't squid allocate more than 1GB on a system with 2GB RAM?
 
> Aside from this, if squid ever does get to the point of swapping, it is 
> misconfigured and your performance has just gone to hell anyway...  (see 
> the FAQ at www.squid-cache.org)

It's not swapping. That's the whole point. We have 2GB and can use at
most 1GB for Squid.

-- 
Ralf Hildebrandt (Im Auftrag des Referat V a)   Ralf.Hildebrandt@charite.de
Charite Campus Mitte                            Tel.  +49 (0)30-450 570-155
Referat V a - Kommunikationsnetze -             Fax.  +49 (0)30-450 570-916
Microsoft: "Where do you want to go today?"
Linux:     "Where do you want to be tomorrow?"
BSD:       "Are you guys coming, or what?"

Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV

[Reuben Farrelly]

[Forwarded for the purposes of continuity, Robert is not on lkml]


>Subject: Re: 2.4.20-aa and LARGE Squid process -> SIGSEGV
>From: Robert Collins <robertc@squid-cache.org>
>To: Reuben Farrelly <reuben-linux@reub.net>
>Cc: Ralf Hildebrandt <Ralf.Hildebrandt@charite.de>
>X-Mailer: Ximian Evolution 1.0.8
>Date: 21 Dec 2002 20:12:21 +1100
>
>On Sat, 2002-12-21 at 20:06, Ralf Hildebrandt wrote:
> > * Reuben Farrelly <reuben-linux@reub.net>:
> >
> > > No, squid is not br0ken in this fashion.  If squid cannot be allocated
> > > enough memory by the system, it logs a message and _dies_.  Relevant 
> files
> > > to look at in your squid source are squid/lib/util.c for xcalloc() and
> > > xmalloc().
> >
> > Why can't squid allocate more than 1GB on a system with 2GB RAM?
>
>In general, it can.
>Folk run it with up to 2Gb with no special options on linux, all the
>time.
>
>Have you tried asking on the *right* list?
>Or running it under gdb?
>Or getting a stacktrace?
>
>SEGSIGV does *not* mean out of memory, you *may* have hit a genuine bug,
>but it *not* due to squid handling a failed malloc wrongly.
>
>Rob