From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail202.messagelabs.com (mail202.messagelabs.com [216.82.254.227]) by kanga.kvack.org (Postfix) with SMTP id 0FF715F0019 for ; Wed, 3 Jun 2009 13:01:43 -0400 (EDT) Received: by pxi37 with SMTP id 37so147451pxi.12 for ; Wed, 03 Jun 2009 10:01:42 -0700 (PDT) Message-ID: <4A26AC73.6040804@gmail.com> Date: Wed, 03 Jun 2009 10:01:39 -0700 From: Joel Krauska MIME-Version: 1.0 Subject: swapoff throttling and speedup? Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: linux-mm@kvack.org List-ID: On occasion we need to unswap a system that's gotten unruly. Scenario: Some leaky app eats up way more RAM than it should, and pushes a few gigs of the running system in to swap. The leaky app is killed, but there's still lots of good stuff sitting in swap that we need to tidy up to get the system back to normal performance levels. The normal recourse is to run swapoff -a ; swapon -a I have two related questions about the swap tools and how they work. 1. Has anyone tried making a nicer swapoff? Right now swapoff can be pretty aggressive if the system is otherwise heavily loaded. On systems that I need to leave running other jobs, swapoff compounds the slowness of the system overall by burning up a single CPU and lots of IO I wrote a perl wrapper that briefly runs swapoff and then kills it, but it would seem more reasonable to have a knob to make swapoff less aggressive. (max kb/s, etc) It looked to me like the swapoff code was immediately hitting kernel internals instead of doing more lifting itself (and making it obvious where I could insert some sleeps) Has anyone found better options here? 2. A faster(multithreaded?) swapoff? >>From what I can tell, swapoff is single threaded, which seems to make unswapping a CPU bound activity. In the opposite use case of my first question, on systems that I /can/ halt all the running code (assuming if they've gone off the deep end and have several gigs in SWAP) it can take quite a long time for unswap to tidy up the mess. Has anyone considered improvements to swapoff to speed it up? (multiple threads?) I'm hoping others have been down this road before. As a rule, we try to avoid swapping when possible, but using: vm.swappiness = 1 But it does still happen on occasion and that lead to this mail. Cheers, Joel -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail144.messagelabs.com (mail144.messagelabs.com [216.82.254.51]) by kanga.kvack.org (Postfix) with SMTP id A75346B005C for ; Wed, 3 Jun 2009 22:06:29 -0400 (EDT) Received: from m2.gw.fujitsu.co.jp ([10.0.50.72]) by fgwmail7.fujitsu.co.jp (Fujitsu Gateway) with ESMTP id n5426RvI027788 for (envelope-from kamezawa.hiroyu@jp.fujitsu.com); Thu, 4 Jun 2009 11:06:27 +0900 Received: from smail (m2 [127.0.0.1]) by outgoing.m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 582B945DE55 for ; Thu, 4 Jun 2009 11:06:27 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (s2.gw.fujitsu.co.jp [10.0.50.92]) by m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 3371945DE51 for ; Thu, 4 Jun 2009 11:06:27 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id 1D6911DB803B for ; Thu, 4 Jun 2009 11:06:27 +0900 (JST) Received: from m108.s.css.fujitsu.com (m108.s.css.fujitsu.com [10.249.87.108]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id C4C941DB803A for ; Thu, 4 Jun 2009 11:06:26 +0900 (JST) Date: Thu, 4 Jun 2009 11:04:56 +0900 From: KAMEZAWA Hiroyuki Subject: Re: swapoff throttling and speedup? Message-Id: <20090604110456.90b0ebcb.kamezawa.hiroyu@jp.fujitsu.com> In-Reply-To: <4A26AC73.6040804@gmail.com> References: <4A26AC73.6040804@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Joel Krauska Cc: linux-mm@kvack.org List-ID: On Wed, 03 Jun 2009 10:01:39 -0700 Joel Krauska wrote: > On occasion we need to unswap a system that's gotten unruly. > > Scenario: Some leaky app eats up way more RAM than it should, and pushes > a few gigs of the running system in to swap. The leaky app is killed, > but there's still lots of good stuff sitting in swap that we need to tidy > up to get the system back to normal performance levels. > > > The normal recourse is to run > swapoff -a ; swapon -a > > > I have two related questions about the swap tools and how they work. > > > 1. Has anyone tried making a nicer swapoff? > Right now swapoff can be pretty aggressive if the system is otherwise > heavily loaded. On systems that I need to leave running other jobs, > swapoff compounds the slowness of the system overall by burning up > a single CPU and lots of IO > > I wrote a perl wrapper that briefly runs swapoff > and then kills it, but it would seem more reasonable to have a knob > to make swapoff less aggressive. (max kb/s, etc) > > It looked to me like the swapoff code was immediately hitting kernel > internals instead of doing more lifting itself (and making it > obvious where I could insert some sleeps) > I haven't heard this swapoff issue for years. Hmm, swapoff -a is proper operation ? (I don't think so..) I think most of people just want "fast" swapoff. > Has anyone found better options here? > > If you know what are the leaky apps, memory cgroup may be a help for avoiding unnecessary swap-out. How about throttling swapoff's cpu usage by cpu scheduler cgroup ? No help ? > > 2. A faster(multithreaded?) swapoff? > From what I can tell, swapoff is single threaded, which seems to make > unswapping a CPU bound activity. > > In the opposite use case of my first question, on systems that I /can/ > halt all the running code (assuming if they've gone off the deep end and have > several gigs in SWAP) it can take quite a long time for unswap to > tidy up the mess. > > Has anyone considered improvements to swapoff to speed it up? > (multiple threads?) > not heared of in this mailing list. But I think swapoff() is a system-call and making it as multithreaded is not easy. (And we have to take care of complex racy cases...) > > I'm hoping others have been down this road before. > > As a rule, we try to avoid swapping when possible, but using: > vm.swappiness = 1 > > But it does still happen on occasion and that lead to this mail. > Thanks, -Kame -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail144.messagelabs.com (mail144.messagelabs.com [216.82.254.51]) by kanga.kvack.org (Postfix) with SMTP id F334D6B004D for ; Wed, 3 Jun 2009 22:43:08 -0400 (EDT) Received: by rv-out-0506.google.com with SMTP id l9so175086rvb.41 for ; Wed, 03 Jun 2009 19:43:07 -0700 (PDT) Message-ID: <4A2734BA.7080004@gmail.com> Date: Wed, 03 Jun 2009 19:43:06 -0700 From: Joel Krauska MIME-Version: 1.0 Subject: Re: swapoff throttling and speedup? References: <4A26AC73.6040804@gmail.com> <20090604110456.90b0ebcb.kamezawa.hiroyu@jp.fujitsu.com> In-Reply-To: <20090604110456.90b0ebcb.kamezawa.hiroyu@jp.fujitsu.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: KAMEZAWA Hiroyuki Cc: linux-mm@kvack.org List-ID: KAMEZAWA Hiroyuki wrote: >> 1. Has anyone tried making a nicer swapoff? >> Right now swapoff can be pretty aggressive if the system is otherwise >> heavily loaded. On systems that I need to leave running other jobs, >> swapoff compounds the slowness of the system overall by burning up >> a single CPU and lots of IO >> >> I wrote a perl wrapper that briefly runs swapoff >> and then kills it, but it would seem more reasonable to have a knob >> to make swapoff less aggressive. (max kb/s, etc) >> >> It looked to me like the swapoff code was immediately hitting kernel >> internals instead of doing more lifting itself (and making it >> obvious where I could insert some sleeps) >> I find I need a slower swapoff when a system that's already running very hot needs to be recovered from lots of swapping without overly impacting the other running processes. The bulk of the work is still being done in normal RAM, and the overhead of consuming an entire CPU just for swapoff degrades my other running processes. > How about throttling swapoff's cpu usage by cpu scheduler cgroup ? > No help ? I think swapoff is all done as systemcalls, not in userspace, so I'm not sure that cgroups would apply here. (granted I had never heard of control groups until just now...) My initial analogy and insight for this was the MD RAID rebuild throttle toggles. /proc/sys/dev/raid/speed_limit_max Which I've had to tune down on occasion to reduce impact to other running processes. (aside: MD RAID rebuilds do seem to be multi-threaded?) Thanks Kame, Joel -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail191.messagelabs.com (mail191.messagelabs.com [216.82.242.19]) by kanga.kvack.org (Postfix) with SMTP id 06CFB6B004D for ; Wed, 3 Jun 2009 22:56:26 -0400 (EDT) Received: from m2.gw.fujitsu.co.jp ([10.0.50.72]) by fgwmail5.fujitsu.co.jp (Fujitsu Gateway) with ESMTP id n542uJVG015614 for (envelope-from kamezawa.hiroyu@jp.fujitsu.com); Thu, 4 Jun 2009 11:56:19 +0900 Received: from smail (m2 [127.0.0.1]) by outgoing.m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 89F7E45DE51 for ; Thu, 4 Jun 2009 11:56:19 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (s2.gw.fujitsu.co.jp [10.0.50.92]) by m2.gw.fujitsu.co.jp (Postfix) with ESMTP id 549BB45DD79 for ; Thu, 4 Jun 2009 11:56:19 +0900 (JST) Received: from s2.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id 3DC5D1DB803F for ; Thu, 4 Jun 2009 11:56:19 +0900 (JST) Received: from m107.s.css.fujitsu.com (m107.s.css.fujitsu.com [10.249.87.107]) by s2.gw.fujitsu.co.jp (Postfix) with ESMTP id E56501DB803E for ; Thu, 4 Jun 2009 11:56:18 +0900 (JST) Date: Thu, 4 Jun 2009 11:54:48 +0900 From: KAMEZAWA Hiroyuki Subject: Re: swapoff throttling and speedup? Message-Id: <20090604115448.c1b434ed.kamezawa.hiroyu@jp.fujitsu.com> In-Reply-To: <4A2734BA.7080004@gmail.com> References: <4A26AC73.6040804@gmail.com> <20090604110456.90b0ebcb.kamezawa.hiroyu@jp.fujitsu.com> <4A2734BA.7080004@gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org To: Joel Krauska Cc: linux-mm@kvack.org List-ID: On Wed, 03 Jun 2009 19:43:06 -0700 Joel Krauska wrote: > KAMEZAWA Hiroyuki wrote: > >> 1. Has anyone tried making a nicer swapoff? > >> Right now swapoff can be pretty aggressive if the system is otherwise > >> heavily loaded. On systems that I need to leave running other jobs, > >> swapoff compounds the slowness of the system overall by burning up > >> a single CPU and lots of IO > >> > >> I wrote a perl wrapper that briefly runs swapoff > >> and then kills it, but it would seem more reasonable to have a knob > >> to make swapoff less aggressive. (max kb/s, etc) > >> > >> It looked to me like the swapoff code was immediately hitting kernel > >> internals instead of doing more lifting itself (and making it > >> obvious where I could insert some sleeps) > >> > > I find I need a slower swapoff when a system that's already running very hot > needs to be recovered from lots of swapping without overly impacting the other > running processes. > > The bulk of the work is still being done in normal RAM, and the overhead > of consuming an entire CPU just for swapoff degrades my other running processes. > > > How about throttling swapoff's cpu usage by cpu scheduler cgroup ? > > No help ? > > I think swapoff is all done as systemcalls, not in userspace, so I'm not > sure that cgroups would apply here. (granted I had never heard of control > groups until just now...) > IIUC, some "cond_resched()" , means "reschedule if necessary", are inserted to swapoff's main loop. Then, limiting usage of cpu may have effects, I think. Thanks, -Kame -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail144.messagelabs.com (mail144.messagelabs.com [216.82.254.51]) by kanga.kvack.org (Postfix) with SMTP id B22ED6B004D for ; Thu, 4 Jun 2009 11:50:50 -0400 (EDT) Date: Thu, 4 Jun 2009 16:50:24 +0100 (BST) From: Hugh Dickins Subject: Re: swapoff throttling and speedup? In-Reply-To: <4A26AC73.6040804@gmail.com> Message-ID: References: <4A26AC73.6040804@gmail.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-linux-mm@kvack.org To: Joel Krauska Cc: KAMEZAWA Hiroyuki , linux-mm@kvack.org List-ID: On Wed, 3 Jun 2009, Joel Krauska wrote: > On occasion we need to unswap a system that's gotten unruly. > > Scenario: Some leaky app eats up way more RAM than it should, and pushes > a few gigs of the running system in to swap. The leaky app is killed, but > there's still lots of good stuff sitting in swap that we need to tidy > up to get the system back to normal performance levels. > > > The normal recourse is to run > swapoff -a ; swapon -a > > > I have two related questions about the swap tools and how they work. > > > 1. Has anyone tried making a nicer swapoff? > Right now swapoff can be pretty aggressive if the system is otherwise > heavily loaded. On systems that I need to leave running other jobs, > swapoff compounds the slowness of the system overall by burning up > a single CPU and lots of IO > > I wrote a perl wrapper that briefly runs swapoff and then kills it, but it > would seem more reasonable to have a knob > to make swapoff less aggressive. (max kb/s, etc) Though what you're doing is inelegant, it does seem a good solution to me: why add fancy kernel tunables, for what is (I think) a rare use case? > It looked to me like the swapoff code was immediately hitting kernel internals > instead of doing more lifting itself (and making it obvious where I could > insert some sleeps) Yes, all the hard work is in the one swapoff(2) system call. You could add an option to swapoff(8), to alarm and sleep and retry; but why bother when you've already got your perl script? > > Has anyone found better options here? Could the answer be in your question: "a nicer swapoff?" Does running "nice swapoff -a" behave more as you want? Won't help on the IO side, I suppose. > > 2. A faster(multithreaded?) swapoff? > From what I can tell, swapoff is single threaded, which seems to make > unswapping a CPU bound activity. > In the opposite use case of my first question, on systems that I /can/ > halt all the running code (assuming if they've gone off the deep end and have > several gigs in SWAP) it can take quite a long time for unswap to tidy up the > mess. > Has anyone considered improvements to swapoff to speed it up? > (multiple threads?) Until there's some unforeseen revolution in swap usage, swapoff will always be nasty. It's hugely inefficient, but you're one of the very few people to be hurt by that: for most people, it's something that only gets run at shutdown time, when there's very little work left for it to do, and nothing left to mind anyway. Certainly there are ways in which swapoff could be made much nicer; but at the cost of maintaining additional pointers which use more memory and slow down codepaths critical to performance, when most critical processes will never go to swap in the first place. So, in view of the tradeoffs, swapoff has remained a nasty backwater: it goes about its business in that hugely inefficient way, to save the rest of mm from having to support it throughout normal usage. Multiple threads? Hmm, never pondered on that. It would certainly make your first case worse, occupying all CPUs instead of just one. If you can muster a chorus of other users upset by swapoff's behaviour, then it would be fun to improve it somewhat; but I'd much rather cut down its CPU usage, than spread the same across CPUs. I've often thought that the 16 bits of the swap_map are poorly used (rarely can more than 2 be set): we could make better use of them with a hash to reduce the amount of blind scanning by an order or two of magnitude. But that's always seemed more a bloat of kernel text than a good use of a developer's time. And it's a very long time since I tried swapin_readahead() instead of the read_swap_cache_async() in try_to_unuse(): more likely to help with your first use case (when you've competing IO) than your second (when the disk is likely to be caching the readahead anyway). > > I'm hoping others have been down this road before. > > As a rule, we try to avoid swapping when possible, but using: > vm.swappiness = 1 > > But it does still happen on occasion and that lead to this mail. Thanks for taking the trouble to write: opinions, anyone? Hugh -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail138.messagelabs.com (mail138.messagelabs.com [216.82.249.35]) by kanga.kvack.org (Postfix) with SMTP id 9F8DF6B004D for ; Thu, 4 Jun 2009 12:25:37 -0400 (EDT) Received: from m4.gw.fujitsu.co.jp ([10.0.50.74]) by fgwmail5.fujitsu.co.jp (Fujitsu Gateway) with ESMTP id n54GPXOh029035 for (envelope-from kamezawa.hiroyu@jp.fujitsu.com); Fri, 5 Jun 2009 01:25:33 +0900 Received: from smail (m4 [127.0.0.1]) by outgoing.m4.gw.fujitsu.co.jp (Postfix) with ESMTP id CAD0545DE7A for ; Fri, 5 Jun 2009 01:25:32 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (s4.gw.fujitsu.co.jp [10.0.50.94]) by m4.gw.fujitsu.co.jp (Postfix) with ESMTP id 615F445DE70 for ; Fri, 5 Jun 2009 01:25:32 +0900 (JST) Received: from s4.gw.fujitsu.co.jp (localhost.localdomain [127.0.0.1]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id CA80D1DB803E for ; Fri, 5 Jun 2009 01:25:31 +0900 (JST) Received: from ml12.s.css.fujitsu.com (ml12.s.css.fujitsu.com [10.249.87.102]) by s4.gw.fujitsu.co.jp (Postfix) with ESMTP id 615601DB803B for ; Fri, 5 Jun 2009 01:25:31 +0900 (JST) Message-ID: <990133947abefb130319d1a7339b718d.squirrel@webmail-b.css.fujitsu.com> In-Reply-To: References: <4A26AC73.6040804@gmail.com> Date: Fri, 5 Jun 2009 01:25:30 +0900 (JST) Subject: Re: swapoff throttling and speedup? From: "KAMEZAWA Hiroyuki" MIME-Version: 1.0 Content-Type: text/plain;charset=iso-2022-jp Content-Transfer-Encoding: 8bit Sender: owner-linux-mm@kvack.org To: Hugh Dickins Cc: Joel Krauska , KAMEZAWA Hiroyuki , linux-mm@kvack.org List-ID: Hugh Dickins wrote: > On Wed, 3 Jun 2009, Joel Krauska wrote: >> I'm hoping others have been down this road before. >> >> As a rule, we try to avoid swapping when possible, but using: >> vm.swappiness = 1 >> >> But it does still happen on occasion and that lead to this mail. > > Thanks for taking the trouble to write: opinions, anyone? > Is there anyone who wants a system call like this ? int mem_swapin(int pid, start-addr, size) - try to swap in pages from range [addr, addr+size) of pid. we can do this force-pagein against file caches and shmem now. this is for swap. I doubts there are no one who can make use of this in sane way. But I'm sometimes surprised to find that there are people make use of swap intentionally... Thanks, -Kame -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org