madvise failure

madvise failure

[Andries E. Brouwer]

In an application something like
	p = mmap(0, sz, PROT_READ, MAP_PRIVATE, fd, 0);
	madvise(p, sz, MADV_SEQUENTIAL);
is done for a small number of files, each with a size of a few GB.
A single sequential pass is done over these files - essentially a merge.

On an old machine the madvise is useful, and decreases total time.
But on a more recent machine, with more memory, the madvise makes
things worse. There, it seems better not to reveal to the kernel
that the data will be read sequentially.

Timing (six files of 4GB each, quadcore Intel Q9550, 16GB memory,
kernel 2.6.27 [Ubuntu], two other processes active):
with madvise, 7 runs: real time varying 9m10s - 37m29s,
without madvise, 6 runs: real time fairly constant 5m45s - 5m54s.


Andries

Re: madvise failure

[KAMEZAWA Hiroyuki]

On Thu, 7 May 2009 14:10:17 +0200
"Andries E. Brouwer" <Andries.Brouwer@cwi.nl> wrote:

> In an application something like
> 	p = mmap(0, sz, PROT_READ, MAP_PRIVATE, fd, 0);
> 	madvise(p, sz, MADV_SEQUENTIAL);
> is done for a small number of files, each with a size of a few GB.
> A single sequential pass is done over these files - essentially a merge.
> 
> On an old machine the madvise is useful, and decreases total time.
With the same kernel (Ubuntu/2.6.27) ? old one ?

> But on a more recent machine, with more memory, the madvise makes
> things worse. There, it seems better not to reveal to the kernel
> that the data will be read sequentially.
> 
> Timing (six files of 4GB each, quadcore Intel Q9550, 16GB memory,
> kernel 2.6.27 [Ubuntu], two other processes active):
> with madvise, 7 runs: real time varying 9m10s - 37m29s,
> without madvise, 6 runs: real time fairly constant 5m45s - 5m54s.
> 
> 
Accessing each page one by one sequentially ? or 
Sequential but (may) skip some pages in each access ?

Thanks,
-Kame

Re: madvise failure

[Andries E. Brouwer]

On Fri, May 08, 2009 at 10:07:45AM +0900, KAMEZAWA Hiroyuki wrote:
> "Andries E. Brouwer" <Andries.Brouwer@cwi.nl> wrote:
> 
>> In an application something like
>> 	p = mmap(0, sz, PROT_READ, MAP_PRIVATE, fd, 0);
>> 	madvise(p, sz, MADV_SEQUENTIAL);
>> is done for a small number of files, each with a size of a few GB.
>> A single sequential pass is done over these files - essentially a merge.
>> 
>> On an old machine the madvise is useful, and decreases total time.

> With the same kernel (Ubuntu/2.6.27) ? old one ?

Both Ubuntu/2.6.27: 2.6.27-11-generic (new) vs 2.6.27-11-server (old)

>> But on a more recent machine, with more memory, the madvise makes
>> things worse. There, it seems better not to reveal to the kernel
>> that the data will be read sequentially.
>> 
>> Timing (six files of 4GB each, quadcore Intel Q9550, 16GB memory,
>> kernel 2.6.27 [Ubuntu], two other processes active):
>> with madvise, 7 runs: real time varying 9m10s - 37m29s,
>> without madvise, 6 runs: real time fairly constant 5m45s - 5m54s.

> Accessing each page one by one sequentially ?

Yes. Essentially a merge.

> or Sequential but (may) skip some pages in each access ?

No.

Andries