mmapping large files hits swap in 2.4?

mmapping large files hits swap in 2.4?

[Daniel Egger]

Hija,

I'm currently looking into optimizing GIMPs own swapping algorithm
by replacing naive file operations by mmap-based ones. Unfortunately
my test machine (PPC, 256MB) gets hit really hard by mmapping files over
100MB into memory: The swap utilization grows up to the file size
and the machine is completely unresponsive for several seconds up to
a few minutes. Seemingly the writes to the mmaped area first hit the
swap and then are read from there again to fit the designated file.

I'm doing something along the lines of:
area = mmap (0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

I also tried MAP_PRIVATE and MAP_LOCKED both with a private and a
shared mapping, but to no avail.

This is kernel version 2.4.19-rc3 (in the benh-variant).

Is there anything I can do to improve the situation or is it just 
the kernel or the architecture?

-- 
Servus,
       Daniel

Re: mmapping large files hits swap in 2.4?

[Richard Zidlicky]

On Fri, Aug 09, 2002 at 07:26:14PM +0200, Daniel Egger wrote:
> Hija,
> 
> I'm currently looking into optimizing GIMPs own swapping algorithm
> by replacing naive file operations by mmap-based ones. Unfortunately
> my test machine (PPC, 256MB) gets hit really hard by mmapping files over
> 100MB into memory: The swap utilization grows up to the file size
> and the machine is completely unresponsive for several seconds up to
> a few minutes. Seemingly the writes to the mmaped area first hit the
> swap and then are read from there again to fit the designated file.
> 
> I'm doing something along the lines of:
> area = mmap (0, size, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

seems like you are doing something else, like hitting all
of the file.

# uname -a
Linux sirizidl.dialin.rrze.uni-erlangen.de 2.4.18 #27 Wed Jul 24 17:25:39 CEST 2002 m68k unknown

main()
{
  char *area;
  int fd=open("/msrc/linux/distr/cd.image", O_RDWR);
  area = mmap (0, 168088*4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
  if (area == -1) perror("mmap");
}

# time ./mmap
real    0m0.035s
user    0m0.020s
sys     0m0.020s

Richard

Re: mmapping large files hits swap in 2.4?

[Daniel Egger]

Am Sam, 2002-08-10 um 14.12 schrieb Richard Zidlicky:

> seems like you are doing something else, like hitting all
> of the file.
 
> # uname -a
> Linux sirizidl.dialin.rrze.uni-erlangen.de 2.4.18 #27 Wed Jul 24 17:25:39 CEST 2002 m68k unknown
 
> main()
> {
>   char *area;
>   int fd=open("/msrc/linux/distr/cd.image", O_RDWR);
>   area = mmap (0, 168088*4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
>   if (area == -1) perror("mmap");
> }

Heh. Well of course I'm also trying to use the memory I mmaped. :)
It is just that it seems the mmaped region is not really bakked by
the underlying file but by swap space which was exactly what I 
was trying to avoid by using a file.
 
-- 
Servus,
       Daniel

Re: mmapping large files hits swap in 2.4?

[Richard Zidlicky]

On Sat, Aug 10, 2002 at 06:17:25PM +0200, Daniel Egger wrote:
> Am Sam, 2002-08-10 um 14.12 schrieb Richard Zidlicky:
> 
> > seems like you are doing something else, like hitting all
> > of the file.
>  
> > # uname -a
> > Linux sirizidl.dialin.rrze.uni-erlangen.de 2.4.18 #27 Wed Jul 24 17:25:39 CEST 2002 m68k unknown
>  
> > main()
> > {
> >   char *area;
> >   int fd=open("/msrc/linux/distr/cd.image", O_RDWR);
> >   area = mmap (0, 168088*4096, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
> >   if (area == -1) perror("mmap");
> > }
> 
> Heh. Well of course I'm also trying to use the memory I mmaped. :)
> It is just that it seems the mmaped region is not really bakked by
> the underlying file but by swap space which was exactly what I 
> was trying to avoid by using a file.

still works as expected for me, accessing a 650 MB file with only 
about 350 MB swap is no problem. It takes some time on that old
m68k box but clearly dominated by HD read time.. a compilation 
runing in the background doesn't seem to make any difference
at all.

Richard

Re: mmapping large files hits swap in 2.4?

[Daniel Egger]

Am Sam, 2002-08-10 um 20.49 schrieb Mark Hahn:

> > It is just that it seems the mmaped region is not really bakked by
> > the underlying file but by swap space which was exactly what I 
> > was trying to avoid by using a file.

> why do you think that?

Because the amount of free swap shrinks continously with mmaped memory
being touched. If I understood the concept of mmap correctly the system
should buffer read/writes to the mapped memory location with real RAM
and page out to the file.

My problem actually is that although I have enough memory to buffer the
whole area the kernel decides to hit hard on the disc which makes the
performance suck.

-- 
Servus,
       Daniel

Re: mmapping large files hits swap in 2.4?

[Helge Hafting]

Daniel Egger wrote:
> 
> Am Sam, 2002-08-10 um 20.49 schrieb Mark Hahn:
> 
> > > It is just that it seems the mmaped region is not really bakked by
> > > the underlying file but by swap space which was exactly what I
> > > was trying to avoid by using a file.
> 
> > why do you think that?
> 
> Because the amount of free swap shrinks continously with mmaped memory
> being touched. If I understood the concept of mmap correctly the system
> should buffer read/writes to the mapped memory location with real RAM
> and page out to the file.
                  ^^^^^^^^^
The system buffer reads/writes by bringing parts of the file into
the page cache.  Writing dirties those pages, and they
will be written back to the file during memory pressure,
or at umount time.

Your mmapped file should NOT get swapped to a swap device.
But other memory certainly may!  When you touch a lot of your
big mmapped file you create memory pressure.  Some of that
pressure writes pages back to your file.  Some of that
pressure swaps other programs / other memory out to the
swap device.  

In short - memory used to cache your big mmapped file don't merely 
compete with memory used for caching other parts of that file.
It competes with all other swappable (or discardable) memory
in the system, and some of that might go to the swap device.

Maybe you only need a little of that big file at a time - but
the VM system cannot know that.  It simply looks at _all_
memory, considers "what is recently used, and what is _not_"
and goes on to swap/writeback the latter parts.

 
> My problem actually is that although I have enough memory to buffer the
> whole area the kernel decides to hit hard on the disc which makes the
> performance suck.

You have enough RAM, but was all of it _free_ according to free?
Lots of it will usually be in use as cache, so something must be
evicted.  Cache are freed sometimes, swapping happens at
other times.

Helge Hafting

Re: mmapping large files hits swap in 2.4?

[Daniel Egger]

Am Mon, 2002-08-12 um 11.11 schrieb Helge Hafting:
 
> In short - memory used to cache your big mmapped file don't merely 
> compete with memory used for caching other parts of that file.
> It competes with all other swappable (or discardable) memory
> in the system, and some of that might go to the swap device.

That's what I though.

> Maybe you only need a little of that big file at a time - but
> the VM system cannot know that.  It simply looks at _all_
> memory, considers "what is recently used, and what is _not_"
> and goes on to swap/writeback the latter parts.

Actually I need 95% and the file will grow on demand so it's pretty
hefty in use; I really need to invent some hack to avoid touching
the memory as much as we do now without any need.

> You have enough RAM, but was all of it _free_ according to free?
> Lots of it will usually be in use as cache, so something must be
> evicted.  Cache are freed sometimes, swapping happens at
> other times.

Interestingly the behaviour seems to be quite different on PPC vs.
i386. On my PPC machine with 256 MB RAM I have constant use swap and
all of the "free" memory is used as cache. On a i386 maschine with
512MB RAM the kernel never touched a single byte of swap and around
150MB are always free. Both machines always have a load > 1 and are
used for big compile jobs like gcc, OpenOffice and alike.
 
-- 
Servus,
       Daniel