Followup: [PATCH -mm] make swapin readahead skip over holes

Followup: [PATCH -mm] make swapin readahead skip over holes

[Dan Magenheimer]

Hi Rik --

I saw this patch in 3.4-rc1 (because it caused a minor merge
conflict with frontswap) and wondered about its impact.
Since I had a server still set up from running benchmarks
before LSFMM, I ran my kernel compile -jN workload (with
N varying from 4 to 40) on 1GB of RAM, on 3.4-rc2 both with
and without this patch.

For values of N=24 and N=28, your patch made the workload
run 4-9% percent faster.  For N=16 and N=20, it was 5-10%
slower.  And for N=36 and N=40, it was 30%-40% slower!

Is this expected?  Since the swap "disk" is a partition
on the one active drive, maybe the advantage is lost due
to contention?

Thanks,
Dan

commit removed 67f96aa252e606cdf6c3cf1032952ec207ec0cf0

Workload:
	kernel compile "make -jN" with varying N
	measurements in elapsed seconds
	boot kernel: 3.4-rc2
	Oracle Linux 6 distro with ext4
	fresh reboot for each test run
	all tests run as root in multi-user mode

Hardware:
	Dell Optiplex 790 = ~$500
	Intel Core i5-2400 @ 3.10 GHz, 4coreX2thread, 6M cache
	1GB RAM DDR3 1333Mhz (to force swapping)
	One 7200rpm SATA 6.0Gb/s drive with 8MB cache
	10GB swap partition

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Followup: [PATCH -mm] make swapin readahead skip over holes

[Rik van Riel]

On 04/16/2012 02:34 PM, Dan Magenheimer wrote:
> Hi Rik --
>
> I saw this patch in 3.4-rc1 (because it caused a minor merge
> conflict with frontswap) and wondered about its impact.
> Since I had a server still set up from running benchmarks
> before LSFMM, I ran my kernel compile -jN workload (with
> N varying from 4 to 40) on 1GB of RAM, on 3.4-rc2 both with
> and without this patch.
>
> For values of N=24 and N=28, your patch made the workload
> run 4-9% percent faster.  For N=16 and N=20, it was 5-10%
> slower.  And for N=36 and N=40, it was 30%-40% slower!
>
> Is this expected?  Since the swap "disk" is a partition
> on the one active drive, maybe the advantage is lost due
> to contention?

There are several things going on here:

1) you are running a workload that thrashes

2) the speed at which data is swapped in is increased
    with this patch

3) with only 1GB memory, the inactive anon list is
    the same size as the active anon list

4) the above points combined mean that less of the
    working set could be in memory at once

One solution may be to decrease the swap cluster for
small systems, when they are thrashing.

On the other hand, for most systems swap is very much
a special circumstance, and you want to focus on quickly
moving excess stuff into swap, and moving it back into
memory when needed.

Workloads that thrash are very much an exception, and
probably not what we should optimize for.


-- 
All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

RE: Followup: [PATCH -mm] make swapin readahead skip over holes

[Dan Magenheimer]

> From: Rik van Riel [mailto:riel@redhat.com]
> Subject: Re: Followup: [PATCH -mm] make swapin readahead skip over holes
> 
> On 04/16/2012 02:34 PM, Dan Magenheimer wrote:
> > Hi Rik --
> >
> > For values of N=24 and N=28, your patch made the workload
> > run 4-9% percent faster.  For N=16 and N=20, it was 5-10%
> > slower.  And for N=36 and N=40, it was 30%-40% slower!
> >
> > Is this expected?  Since the swap "disk" is a partition
> > on the one active drive, maybe the advantage is lost due
> > to contention?
> 
> There are several things going on here:
> 
> 1) you are running a workload that thrashes
> 
> 2) the speed at which data is swapped in is increased
>     with this patch
> 
> 3) with only 1GB memory, the inactive anon list is
>     the same size as the active anon list
> 
> 4) the above points combined mean that less of the
>     working set could be in memory at once
> 
> One solution may be to decrease the swap cluster for
> small systems, when they are thrashing.
> 
> On the other hand, for most systems swap is very much
> a special circumstance, and you want to focus on quickly
> moving excess stuff into swap, and moving it back into
> memory when needed.

Hmmm... as I look at this patch more, I think I get a
picture of what's going on and I'm still concerned.
Please correct me if I am misunderstanding:

What the patch does is increase the average size of
a "cluster" of sequential pages brought in per "read"
from the swap device.  As a result there are more pages
brought back into memory "speculatively" because it is
presumably cheaper to bring in more pages per disk seek,
even if it results in a lower "swapcache hit rate".
In effect, you've done the equivalent of increasing the
default swap cluster size (on average).

If the above is wrong, please cut here and ignore
the following. :-)  But in case it is right (or
close enough), let me continue...

In other words, you are both presuming a "swap workload"
that is more sequential than random for which this patch
improves performance, and assuming a "swap device" 
for which the cost of a seek is high enough to overcome
the costs of filling the swap cache with pages that won't
be used.

While it is easy to write a simple test/benchmark that
swaps a lot (and we probably all have similar test code
that writes data into a huge bigger-than-RAM array and then
reads it back), such a test/benchmark is usually sequential,
so one would assume most swap testing is done with a
sequential-favoring workload.  The kernbench workload
apparently exercises swap quite a bit more randomly and
your patch makes it run slower for low and high levels
of swapping, while faster for moderate swapping.

I also suspect (without proof) that the patch will
result in lower performance on non-rotating devices, such
as SSDs.

(Sure one can change the swap cluster size to 1, but how
many users or even sysadmins know such a thing even
exists... so the default is important.)

I'm no I/O expert, but I suspect if one of the Linux
I/O developers proposed a patch that unilaterally made
all sequential I/O faster and all random I/O slower,
it would get torn to pieces.

I'm certainly not trying to tear your patch to pieces,
just trying to evaluate it.  Hope that's OK.

Thanks,
Dan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Followup: [PATCH -mm] make swapin readahead skip over holes

[Rik van Riel]

On 04/17/2012 11:20 AM, Dan Magenheimer wrote:

> In other words, you are both presuming a "swap workload"
> that is more sequential than random for which this patch
> improves performance, and assuming a "swap device"
> for which the cost of a seek is high enough to overcome
> the costs of filling the swap cache with pages that won't
> be used.

Indeed, on spinning media the cost of seeking to
a cluster and reading one page is essentially the
same as the cost of seeking to a cluster and
reading the whole thing.


> While it is easy to write a simple test/benchmark that
> swaps a lot (and we probably all have similar test code
> that writes data into a huge bigger-than-RAM array and then
> reads it back), such a test/benchmark is usually sequential,
> so one would assume most swap testing is done with a
> sequential-favoring workload.

Lots of programs allocate fairly large memory
objects, and access them again in the same
large chunks.

Take a look at a desktop application like a
web browser, for example.

> The kernbench workload
> apparently exercises swap quite a bit more randomly and
> your patch makes it run slower for low and high levels
> of swapping, while faster for moderate swapping.

The kernbench workload consists of a large number
of fairly small, short lived processes. I suspect
this is a very non-typical workload to run into
swap, on today's systems.

A more typical workload consists of multiple large
processes, with the working set moving from one
part of memory (now inactive) to somewhere else.

We need to maximize swap IO throughput in order to
allow the system to quickly move to the new working
set.

> I also suspect (without proof) that the patch will
> result in lower performance on non-rotating devices, such
> as SSDs.
>
> (Sure one can change the swap cluster size to 1, but how
> many users or even sysadmins know such a thing even
> exists... so the default is important.)

If the default should be changed for some systems,
that is worth doing.

How does your test run with smaller swap cluster
sizes?

Would a swap cluster of 4 or 5 be closer to optimal
for a 1GB system?

-- 
All rights reversed

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>