Problems handling large files??

Problems handling large files??

[Michel Lanners]

Hi all,

I've come across another problem in the 2.3.99 kernels (I've not had a
chance to try 2.4.0-test). This is with 2.3.99-pre9 from Paul's rsync.

Whenever a file larger than a few megabytes is created, the system will
report 'out of memory' errors at random file syizes, while writing the
file.

I'v seen it while unpacking the XFree4 sources (three files over 10 MB
each), and now while running bonnie (should be run with 100-200 MB
depending on RAM).

I got by the XFree problem by running rpm -u multiple times in a row on
the source rpm, until it succeeded.

However, with bonnie, I've not ben able to create files larger than 60
MB (there's plenty of room on the device...). Here is the result from
bonnie's attempts to create large files, all got errors:

-rwxrwx---   1 mlan     users    60620800 Jun 19 00:44 Bonnie.883
-rwxrwx---   1 mlan     users     6524928 Jun 19 00:44 Bonnie.885
-rwxrwx---   1 mlan     users     4177920 Jun 19 00:45 Bonnie.887
-rwxrwx---   1 mlan     users     8466432 Jun 19 00:45 Bonnie.889
-rwxrwx---   1 mlan     users    14770176 Jun 19 00:45 Bonnie.891
-rwxrwx---   1 mlan     users    30285824 Jun 19 00:45 Bonnie.893
-rwxrwx---   1 mlan     users    15667200 Jun 19 00:51 Bonnie.901
-rwxrwx---   1 mlan     users     3592192 Jun 19 00:51 Bonnie.905

Here's the exact error message:

File '/mnt/testdrive/Bonnie.905', size: 104857600
Writing with putc()...Bonnie: drastic I/O error (putc): Cannot allocate memory

Here's the relevant code leading to the error:

  /* Fill up a file, writing it a char at a time with the stdio putc() call */
  fprintf(stderr, "Writing with putc()...");
  newfile(name, &fd, &stream, 1);
  timestamp();
  for (words = 0; words < size; words++)
    if (putc(words & 0x7f, stream) == EOF)
      io_error("putc");

static void
io_error(char * message)
{
  char buf[Chunk];

  sprintf(buf, "Bonnie: drastic I/O error (%s)", message);
  perror(buf);
  exit(1);
}

So the 'Cannot allocate memory' comes from the system (libc? system
call?).

Anybody know what's going on? Can you verify this on a 2.4.0-test
kernel? FWIW, bonnie _was_ working OK on 2.2.13 (later unverified) and
2.3.48 (latest benchamrk I have).

Clues?

Thanks
		Michel

-------------------------------------------------------------------------
Michel Lanners                 |  " Read Philosophy.  Study Art.
23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
L-1710 Luxembourg              |
email   mlan@cpu.lu            |
http://www.cpu.lu/~mlan        |                     Learn Always. "


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Problems handling large files??

[Paul Schinder]

At 12:57 AM +0200 6/19/00, Michel Lanners wrote:
>}
>
>So the 'Cannot allocate memory' comes from the system (libc? system
>call?).
>
>Anybody know what's going on? Can you verify this on a 2.4.0-test
>kernel? FWIW, bonnie _was_ working OK on 2.2.13 (later unverified) and
>2.3.48 (latest benchamrk I have).


All I can tell you is my own experience with 2.3.99 (pre 9, I think,
but it's long gone) and 2.4.0-test1-ac18.  With 2.3.99, I tried doing
a backup with cddump (a perl script that runs mkisofs and cdrecord
for backup up onto CD's), and it failed with both read and write
errors ("fwrite: cannot write" and "cannot open", as I recall).  This
morning, under 2.4.0-test1-ac18, I made a 400+ Mb cd image and burned
it with no trouble.  So if I were you, I'd rsync from Paul's tree
(ac18 as of Saturday morning EDT) and try a 2.4.0.

Of course, the fact that the clock drifted everywhere (xntpd kept on
making ~ 10 s corrections) and that I couldn't get my new UMAX Astra
2200 scanner working under 2.4.0 motivated me to switch to 2.2.16.
(The scanner still won't work, but the clock doesn't drift.)

>
>Clues?
>
>Thanks
>		Michel
>
>-------------------------------------------------------------------------
>Michel Lanners                 |  " Read Philosophy.  Study Art.
>23, Rue Paul Henkes            |    Ask Questions.  Make Mistakes.
>L-1710 Luxembourg              |
>email   mlan@cpu.lu            |
>http://www.cpu.lu/~mlan        |                     Learn Always. "
>

--
--
Paul Schinder
schinder@pobox.com

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Problems handling large files??

[Gabriel Paubert]

On Sun, 18 Jun 2000, Paul Schinder wrote:

>
> At 12:57 AM +0200 6/19/00, Michel Lanners wrote:
> >}
> >
> >So the 'Cannot allocate memory' comes from the system (libc? system
> >call?).
> >
> >Anybody know what's going on? Can you verify this on a 2.4.0-test
> >kernel? FWIW, bonnie _was_ working OK on 2.2.13 (later unverified) and
> >2.3.48 (latest benchamrk I have).
>
>
> All I can tell you is my own experience with 2.3.99 (pre 9, I think,
> but it's long gone) and 2.4.0-test1-ac18.  With 2.3.99, I tried doing
> a backup with cddump (a perl script that runs mkisofs and cdrecord
> for backup up onto CD's), and it failed with both read and write
> errors ("fwrite: cannot write" and "cannot open", as I recall).  This
> morning, under 2.4.0-test1-ac18, I made a 400+ Mb cd image and burned
> it with no trouble.  So if I were you, I'd rsync from Paul's tree
> (ac18 as of Saturday morning EDT) and try a 2.4.0.
>
> Of course, the fact that the clock drifted everywhere (xntpd kept on
> making ~ 10 s corrections) and that I couldn't get my new UMAX Astra
> 2200 scanner working under 2.4.0 motivated me to switch to 2.2.16.
> (The scanner still won't work, but the clock doesn't drift.)

Well, it seems nobody noticed my recent (delayed because of linuxppc.org
problems) message about clock problems and the patch to solve it. Actually
the patch was for 2.2.12, I plan to produce a patch for more recent
kernels next week (I'm leaving for a short holiday tomorrow), but the
upgrade for my machines (MVME boards) is painful.

The clock of all PPC kernels drifts a lot, although the 2.4 code is even
more bogus. 2.2 can easily give random drifts of a few seconds per day, I
brought it down by a factor of 30 on my machines (which have a very bad
oscillators), by using a _rigorous_ algorithm which eliminates all kinds
of hazards.

Highest differences I've see since then on NTP are 350 microseconds (the
machine is used as an NTP reference clock, directly on a combination of
GPS and atomic (rubidium and hydrogen maser) clocks, for me 300
microseconds is still large but it seems that I can't do anything about it
since this damned main CPU oscillator frequency varies by about 0.6 ppm
depending on the load on the machine (bus activity, not thermal).

	Gabriel.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Problems handling large files??

[Andreas Tobler]

Michel Lanners wrote:
>
> Hi all,
>
> I've come across another problem in the 2.3.99 kernels (I've not had a
> chance to try 2.4.0-test). This is with 2.3.99-pre9 from Paul's rsync.
>
> Whenever a file larger than a few megabytes is created, the system will
> report 'out of memory' errors at random file syizes, while writing the
> file.
[snip]
> So the 'Cannot allocate memory' comes from the system (libc? system
> call?).
>
> Anybody know what's going on? Can you verify this on a 2.4.0-test
> kernel? FWIW, bonnie _was_ working OK on 2.2.13 (later unverified) and
> 2.3.48 (latest benchamrk I have).
>
> Clues?

As reported before, an update to 2.4.0-test1-ac19 wouldn't be a bad idea.
I tried bonnie a few minutes ago on a fsmlabs tree 2.4.0-test-ac19 which
still reports as ac18. And I had no problems so far. Bonnie ended with
giving me some ? results. I did it two times with a 100MB file on a
wallstreet. One try without hdparm tuning the other one with.

Andreas

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/