Swap defragging

Swap defragging

[Raymond Jennings]

Just a two cent question, but is there any merit to having the kernel
defragment swap space?

Making a process's swapped out stuff together?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Johannes Weiner]

On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
> Just a two cent question, but is there any merit to having the kernel
> defragment swap space?

That is a good question.

Swap does fragment quite a bit, and there are several reasons for
that.

We swap pages in our LRU list order, but this list is sorted by first
access, not by access frequency (not quite that cookie cutter, but the
ordering is certainly fairly coarse).  This means that the pages may
already be in suboptimal order for swap in at the time of swap out.

Once written to disk, the layout tends to stick.  One reason is that
we actually try to not free swap slots unless there is a shortage of
swap space to save future swap out IO (grep for vm_swap_full()).  The
other reason is that if a page shared among multiple threads is
swapped out, it can not be removed from swap until all threads have
faulted the page back in because of page table entries still referring
to the swap slot on disk.  In a multi-threaded application, this is
rather unlikely.

So even though the referencing order of the application might change,
the disk layout won't.  But adjusting the disk layout speculatively
increases disk IO, so it could be hard to prove that you came up with
a net improvement.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Raymond Jennings]

Not to mention that swapped pages get freed when modified in RAM IIRC.

On Thu, Mar 7, 2013 at 6:35 PM, Johannes Weiner <hannes@cmpxchg.org> wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
>
> That is a good question.
>
> Swap does fragment quite a bit, and there are several reasons for
> that.
>
> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of
> swap space to save future swap out IO (grep for vm_swap_full()).  The
> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Will Huck]

Hi Johannes,
On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
> That is a good question.
>
> Swap does fragment quite a bit, and there are several reasons for
> that.

Are there any tools to test and monitor swap subsystem and page reclaim 
subsystem?

> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of
> swap space to save future swap out IO (grep for vm_swap_full()).  The
> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Johannes Weiner]

On Sat, Mar 09, 2013 at 10:00:37AM +0800, Will Huck wrote:
> Hi Johannes,
> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> >On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
> >>Just a two cent question, but is there any merit to having the kernel
> >>defragment swap space?
> >That is a good question.
> >
> >Swap does fragment quite a bit, and there are several reasons for
> >that.
> 
> Are there any tools to test and monitor swap subsystem and page
> reclaim subsystem?

seekwatcher is great to see the IO patterns.  Anything that uses
anonymous memory can test swap: a java job, multiplying matrixes,
kernel builds etc.  I mostly log /proc/vmstat by taking snapshots at a
regular interval during the workload, then plot and visually correlate
the swapin/swapout counters with the individual LRU sizes, page fault
rate, what have you, to get a feeling for what it's doing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Will Huck]

Hi Johannes,
On 03/13/2013 12:52 AM, Johannes Weiner wrote:
> On Sat, Mar 09, 2013 at 10:00:37AM +0800, Will Huck wrote:
>> Hi Johannes,
>> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
>>> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>>>> Just a two cent question, but is there any merit to having the kernel
>>>> defragment swap space?
>>> That is a good question.
>>>
>>> Swap does fragment quite a bit, and there are several reasons for
>>> that.
>> Are there any tools to test and monitor swap subsystem and page
>> reclaim subsystem?

One offline question,

active:inactive => 1:1 for file page and active:inactive => 
inactive_ratio for anonymous page, why has this different?

> seekwatcher is great to see the IO patterns.  Anything that uses
> anonymous memory can test swap: a java job, multiplying matrixes,
> kernel builds etc.  I mostly log /proc/vmstat by taking snapshots at a
> regular interval during the workload, then plot and visually correlate
> the swapin/swapout counters with the individual LRU sizes, page fault
> rate, what have you, to get a feeling for what it's doing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Will Huck]

Hi Johannes,
On 03/13/2013 12:52 AM, Johannes Weiner wrote:
> On Sat, Mar 09, 2013 at 10:00:37AM +0800, Will Huck wrote:
>> Hi Johannes,
>> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
>>> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>>>> Just a two cent question, but is there any merit to having the kernel
>>>> defragment swap space?
>>> That is a good question.
>>>
>>> Swap does fragment quite a bit, and there are several reasons for
>>> that.
>> Are there any tools to test and monitor swap subsystem and page
>> reclaim subsystem?

pgscan_kswapd_dma 0
pgscan_kswapd_dma32 0
pgscan_kswapd_normal 0
pgscan_kswapd_movable 0
pgscan_direct_dma 0
pgscan_direct_dma32 0
pgscan_direct_normal 0
pgscan_direct_movable 0
pgscan_direct_throttle 0
zone_reclaim_failed 0
pginodesteal 0
slabs_scanned 328704

slab cache is scaned but file-backed/swap-backed pages are not scanned, why?

> seekwatcher is great to see the IO patterns.  Anything that uses
> anonymous memory can test swap: a java job, multiplying matrixes,
> kernel builds etc.  I mostly log /proc/vmstat by taking snapshots at a
> regular interval during the workload, then plot and visually correlate
> the swapin/swapout counters with the individual LRU sizes, page fault
> rate, what have you, to get a feeling for what it's doing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Ric Mason]

Hi Johannes,
On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
> That is a good question.
>
> Swap does fragment quite a bit, and there are several reasons for
> that.
>
> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of

If all the swap slots will be freed when swapoff?

> swap space to save future swap out IO (grep for vm_swap_full()).  The
> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Johannes Weiner]

On Mon, Mar 11, 2013 at 10:54:32AM +0800, Ric Mason wrote:
> Hi Johannes,
> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> >On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
> >>Just a two cent question, but is there any merit to having the kernel
> >>defragment swap space?
> >That is a good question.
> >
> >Swap does fragment quite a bit, and there are several reasons for
> >that.
> >
> >We swap pages in our LRU list order, but this list is sorted by first
> >access, not by access frequency (not quite that cookie cutter, but the
> >ordering is certainly fairly coarse).  This means that the pages may
> >already be in suboptimal order for swap in at the time of swap out.
> >
> >Once written to disk, the layout tends to stick.  One reason is that
> >we actually try to not free swap slots unless there is a shortage of
> 
> If all the swap slots will be freed when swapoff?

Well, yeah, but that's hardly an interesting case, is it?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Simon Jeons]

Hi Johannes,
On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
> That is a good question.

One question here:

The comments of setup_swap_extents:
An ordered list of swap extents is built at swapon time and is then used 
at swap_writepage/swap_readpage tiem for locating where on disk a page 
belongs.
But I didn't see any handle of swap extents in 
swap_writepage/swap_readpage, why?

>
> Swap does fragment quite a bit, and there are several reasons for
> that.
>
> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of
> swap space to save future swap out IO (grep for vm_swap_full()).  The
> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Johannes Weiner]

On Mon, Mar 11, 2013 at 11:11:26AM +0800, Simon Jeons wrote:
> Hi Johannes,
> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> >On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
> >>Just a two cent question, but is there any merit to having the kernel
> >>defragment swap space?
> >That is a good question.
> 
> One question here:
> 
> The comments of setup_swap_extents:
> An ordered list of swap extents is built at swapon time and is then
> used at swap_writepage/swap_readpage tiem for locating where on disk
> a page belongs.
> But I didn't see any handle of swap extents in
> swap_writepage/swap_readpage, why?

This is not the right place for such questions.  If you are interested
in how the kernel works, buy a book, read the code, consult the kernel
newbies project if you get stuck.

Also, the answer to your question is within a few lines of each other
in that same code file.  Make an effort.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Jaegeuk Hanse]

Hi Johannes,
On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
> That is a good question.
>
> Swap does fragment quite a bit, and there are several reasons for
> that.
>
> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of
> swap space to save future swap out IO (grep for vm_swap_full()).  The

Since anonymous page will be swap out if it's dirty and the contents of 
the page and data store in swap area is not equal now, why can avoid 
future swap out IO?

> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Johannes Weiner]

On Mon, Mar 11, 2013 at 11:16:29AM +0800, Jaegeuk Hanse wrote:
> Hi Johannes,
> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> >On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
> >>Just a two cent question, but is there any merit to having the kernel
> >>defragment swap space?
> >That is a good question.
> >
> >Swap does fragment quite a bit, and there are several reasons for
> >that.
> >
> >We swap pages in our LRU list order, but this list is sorted by first
> >access, not by access frequency (not quite that cookie cutter, but the
> >ordering is certainly fairly coarse).  This means that the pages may
> >already be in suboptimal order for swap in at the time of swap out.
> >
> >Once written to disk, the layout tends to stick.  One reason is that
> >we actually try to not free swap slots unless there is a shortage of
> >swap space to save future swap out IO (grep for vm_swap_full()).  The
> 
> Since anonymous page will be swap out if it's dirty and the contents
> of the page and data store in swap area is not equal now, why can
> avoid future swap out IO?

Modified pages get written out freshly, but in a multi-threaded
application, the original page stays put until all threads have
modified it or faulted it back in.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Jaegeuk Hanse]

Hi Johannes,
On 03/13/2013 01:08 AM, Johannes Weiner wrote:
> On Mon, Mar 11, 2013 at 11:16:29AM +0800, Jaegeuk Hanse wrote:
>> Hi Johannes,
>> On 03/08/2013 10:35 AM, Johannes Weiner wrote:
>>> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>>>> Just a two cent question, but is there any merit to having the kernel
>>>> defragment swap space?
>>> That is a good question.
>>>
>>> Swap does fragment quite a bit, and there are several reasons for
>>> that.
>>>
>>> We swap pages in our LRU list order, but this list is sorted by first
>>> access, not by access frequency (not quite that cookie cutter, but the
>>> ordering is certainly fairly coarse).  This means that the pages may
>>> already be in suboptimal order for swap in at the time of swap out.
>>>
>>> Once written to disk, the layout tends to stick.  One reason is that
>>> we actually try to not free swap slots unless there is a shortage of
>>> swap space to save future swap out IO (grep for vm_swap_full()).  The
>> Since anonymous page will be swap out if it's dirty and the contents
>> of the page and data store in swap area is not equal now, why can
>> avoid future swap out IO?
> Modified pages get written out freshly, but in a multi-threaded
> application, the original page stays put until all threads have
> modified it or faulted it back in.

Sorry, you didn't resolve my confuse! It seems that this is your second 
reason for why disk layout tends to stick. However, what I confuse is 
your first reason. You said that we actually try to not free swap slots 
unless there is a shortage of swap space to save future swap out IO, 
why? Anonymous pages are swapped out since they are dirty, how can don't 
swap out and swap IO?


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Swap defragging

[Will Huck]

Hi Johannes,
On 03/08/2013 10:35 AM, Johannes Weiner wrote:
> On Thu, Mar 07, 2013 at 06:07:23PM -0800, Raymond Jennings wrote:
>> Just a two cent question, but is there any merit to having the kernel
>> defragment swap space?
> That is a good question.
>
> Swap does fragment quite a bit, and there are several reasons for
> that.
>
> We swap pages in our LRU list order, but this list is sorted by first
> access, not by access frequency (not quite that cookie cutter, but the
> ordering is certainly fairly coarse).  This means that the pages may
> already be in suboptimal order for swap in at the time of swap out.
>
> Once written to disk, the layout tends to stick.  One reason is that
> we actually try to not free swap slots unless there is a shortage of
> swap space to save future swap out IO (grep for vm_swap_full()).  The

What I concern is scan_swap_map won't free swap slot which is lower 
than  p->lowest_bit and larger than p->highest_bit.

> other reason is that if a page shared among multiple threads is
> swapped out, it can not be removed from swap until all threads have
> faulted the page back in because of page table entries still referring
> to the swap slot on disk.  In a multi-threaded application, this is
> rather unlikely.
>
> So even though the referencing order of the application might change,
> the disk layout won't.  But adjusting the disk layout speculatively
> increases disk IO, so it could be hard to prove that you came up with
> a net improvement.
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>