glibc: realloc(): invalid next size

glibc: realloc(): invalid next size

[James Colannino]

Hey everyone.  I'm having great difficulty debugging a function of mine 
that adds more memory as needed when reading variable length lines.

When I run the code, it always fails after readline() calls addmemory() 
for the second time (no matter how I change linebuf->buflen, or don't 
change it...) If readline only has to call addmemory() once, no matter 
what the size of memory being passed is, it works.  The following is the 
error I get:

*** glibc detected *** realloc(): invalid next size: 0x0804a170 ***

Frankly, this has me absolutely stumped.  I scoured the code looking for 
things like buffer overruns, etc. and couldn't find a thing.  I'm sure 
there must be something I'm missing, but for the life of me, I can't 
figure it out.  Other than this problem, readline() works great.  Below 
is the code for the two functions (along with the main function that 
initializes the structure used by readline), along with the output of a 
backtrace from gdb.

**********
Backtrace:
**********

(gdb) run
Starting program: 
/home/james/svn/projects/exercises/linux-prog-ptr/03-memory/src/a.out

*** glibc detected *** realloc(): invalid next size: 0x0804a170 ***

Program received signal SIGABRT, Aborted.
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0  0xffffe410 in __kernel_vsyscall ()
#1  0xb7df9391 in raise () from /lib/tls/libc.so.6
#2  0xb7dfab27 in abort () from /lib/tls/libc.so.6
#3  0xb7e2c98e in __fsetlocking () from /lib/tls/libc.so.6
#4  0xb7e36666 in mallopt () from /lib/tls/libc.so.6
#5  0xb7e35a89 in mallopt () from /lib/tls/libc.so.6
#6  0xb7e3421c in realloc () from /lib/tls/libc.so.6
#7  0x080486cb in addmemory (linebuf=0xbfa28a30) at 03-01-readline-getc.c:79
#8  0x080485d4 in readline (linebuf=0xbfa28a30) at 03-01-readline-getc.c:30
#9  0x080487f3 in main (argc=2, argv=0xbfa28ad4) at 03-01-test.c:28

**************************
main(), from 03-01-test.c:
**************************

struct line {
    size_t buflen;
    char *buf;
    FILE *fp;
};

int main(int argc, char **argv) {

    struct line linebuf;

    if (argc != 2) {
       fprintf(stderr, "correct usage: %s filename\n", argv[1]);
       exit(EXIT_FAILURE);
    }

    if ((linebuf.fp = fopen(argv[1], "r")) == NULL) {
       fprintf(stderr, "error: couldn't open %s\n", argv[1]);
       exit(EXIT_FAILURE);
    }

    if ((linebuf.buf = (char *)calloc(81, sizeof(char))) == NULL) {
       fprintf(stderr, "error: not enough memory :(\n");
       exit(EXIT_FAILURE);
    }

    linebuf.buflen = sizeof(char) * 81;

    switch(readline(&linebuf)) {
       case  0:
          break;
       case -1:
          printf("EOF!\n");
          break;
       case -2:
          fprintf(stderr, "error: linebuf->fp not opened\n");
          break;
       case -3:
          break;
    }

    printf("string: %s", linebuf.buf);
    return 0;

***************************************************
readline() and addmemory(), from 03-01-readline.c:
***************************************************

/* readline(): reads one single line from linebuf->fp into linebuf->buf.

    The programmer should make sure that buf is malloc'd with an initial
    amount of memory, that the size of that allocation (in bytes) is
    stored in linebuf->buflen and that linebuf->fp points to an open
    file.  If any of these conditions are not met, readline() will fail.

    Return Codes:
       0 - success
      -1 - EOF
      -2 - linebuf->fp not an opened file
      -3 - other error
*/
int readline(struct line *linebuf) {

    char *start;                     /* points to beginning of buffer */
    char *bufpos;                    /* our current position in 
linebuf->buf */

    size_t    buflen;                /* size of buffer in bytes */
    ptrdiff_t bufcount;              /* how many characters we've read 
so far */

    int c;                           /* stores the result of getc(fp) */

    if (linebuf->fp == NULL)         /* linebuf->fp is not an opened file */
       return -2;

    bufpos = start = linebuf->buf;
    buflen = linebuf->buflen;

    while ((c = getc(linebuf->fp)) != EOF) {

       if (buflen <= 2) {            /* need more memory */
          bufcount = bufpos - start;
          printf("hello!\n");
          start = addmemory(linebuf);
          buflen = linebuf->buflen;
          bufpos = start + bufcount;
       }

       *(bufpos++) = c;
       buflen -= 1;

       if (c == '\n') {              /* we're done with one line */
          *bufpos = '\0';
          break;
       }
    }

    if (c == EOF) {

       if (strlen(start) == 0)   /* we got nothing else before EOF */
          return -1;

       else if ( start[strlen(start) - 1] != '\n') {

                if (buflen <= 2) {       /* need more memory */
                   bufcount = bufpos - start;
                   start = addmemory(linebuf);
                   buflen = linebuf->buflen;
                   bufpos = start + bufcount;
                }

                *(bufpos++) = '\n';
                *bufpos = '\0';
       }

       else
          return -1;     /* the line was complete */
    }

    return 0;
}


/* addmemory(): doubles the size of linebuf->buf.  Returns a pointer
    to the newly allocated memory or exits if it fails */
char *addmemory(struct line *linebuf) {

    char *bufcpy = linebuf->buf;

    linebuf->buflen *= 2;
    printf("%ld\n", linebuf->buflen);

    if ((linebuf->buf = (char *)realloc(linebuf->buf, linebuf->buflen)) \
== NULL) {
       free(bufcpy);
       fprintf(stderr, "error: out of memory! :( \n");
       exit(EXIT_FAILURE);
    }

    return linebuf->buf;
}

James
-- 
My blog: http://www.crazydrclaw.com/
My homepage: http://james.colannino.org/

"Black holes are where God divided by zero." --Steven Wright

Re: glibc: realloc(): invalid next size

[James Colannino]

James Colannino wrote:

> Hey everyone.  I'm having great difficulty debugging a function of mine 
> that adds more memory as needed when reading variable length lines.
> 
> When I run the code, it always fails after readline() calls addmemory() 
> for the second time (no matter how I change linebuf->buflen, or don't 
> change it...) If readline only has to call addmemory() once, no matter 
> what the size of memory being passed is, it works.[...]

I found the bug (for what it's worth.)  It was very subtle, but it 
caused catastrophic results at runtime.  Notice line 6 below (I numbered 
them.)  This is the while loop in readline().  It says that it should 
start counting down again from linebuf->buflen elements after the 
reallocation.  The problem is, half of those spaces are already filled, 
so when I start counting down again from such a high number, forgetting 
that half is already filled, I think I must blow away bookeeping 
information stored by the memory allocator, thus, while the first call 
to realloc() is successful, the second one isn't.

I changed that line of code to:
6: buflen = linebuf->buflen - bufcount;

That way, it makes sure to take into account those spaces which have 
already been used.  This code when compiled runs perfectly.  Sorry for 
the noise to the list.  I honestly have been banging my head over this 
one and really couldn't find a good solution (I was starting to suspect 
something was up with glibc, which probably wasn't very bright :-P), 
otherwise I wouldn'tve asked the question.


>1:    while ((c = getc(linebuf->fp)) != EOF) {
>2: 
>3:       if (buflen <= 2) {            /* need more memory */
>4:          bufcount = bufpos - start;
>5:          start = addmemory(linebuf);
>6:          buflen = linebuf->buflen;
>7:          bufpos = start + bufcount;
>8:       }
>9: 
>10:      *(bufpos++) = c;
>11:      buflen -= 1;
>12: 
>13:      if (c == '\n') {              /* we're done with one line */
>14:        *bufpos = '\0';
>15:         break;
>16:      }
>17:   }

James
-- 
My blog: http://www.crazydrclaw.com/
My homepage: http://james.colannino.org/

"Black holes are where God divided by zero." --Steven Wright