From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx191.postini.com [74.125.245.191]) by kanga.kvack.org (Postfix) with SMTP id 2488E6B004D for ; Fri, 20 Jan 2012 17:45:15 -0500 (EST) Date: Fri, 20 Jan 2012 14:45:13 -0800 From: Andrew Morton Subject: Re: [Bug 12309] Large I/O operations result in poor interactive performance and high iowait times Message-Id: <20120120144513.f457a58d.akpm@linux-foundation.org> In-Reply-To: <201201201611.q0KGBPf6029256@bugzilla.kernel.org> References: <201201201611.q0KGBPf6029256@bugzilla.kernel.org> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Wu Fengguang On Fri, 20 Jan 2012 16:11:25 GMT bugzilla-daemon@bugzilla.kernel.org wrote: > https://bugzilla.kernel.org/show_bug.cgi?id=12309 We've had some recent updates to the world's largest bug report. Apparently our large-writer-paralyses-the-machine problems have worsened in recent kernels. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx117.postini.com [74.125.245.117]) by kanga.kvack.org (Postfix) with SMTP id 429656B005D for ; Sat, 28 Jan 2012 09:44:31 -0500 (EST) From: "Wu, Fengguang" Subject: RE: [Bug 12309] Large I/O operations result in poor interactive performance and high iowait times Date: Sat, 28 Jan 2012 14:44:24 +0000 Message-ID: References: <201201201611.q0KGBPf6029256@bugzilla.kernel.org>,<20120120144513.f457a58d.akpm@linux-foundation.org> In-Reply-To: <20120120144513.f457a58d.akpm@linux-foundation.org> Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , "linux-mm@kvack.org" Cc: "fengguang.wu@gmail.com" [replying from webmail and CC my gmail account]=0A= =0A= >> https://bugzilla.kernel.org/show_bug.cgi?id=3D12309=0A= =0A= > We've had some recent updates to the world's largest bug report.=0A= > Apparently our large-writer-paralyses-the-machine problems have=0A= > worsened in recent kernels.=0A= =0A= Yeah I can reproduce the interactive problem on 3.3-rc1, and the main stall= s seem to happen when loading the task and allocating memory.=0A= =0A= My test case is to start a small 1MB dd when there are 10 long running dd t= asks=0A= each writing to one JBOD disk.=0A= =0A= [long running dd tasks]=0A= root 5232 28.6 0.0 8316 632 ? D 21:19 0:36 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdc1/zero-1=0A= root 5235 27.4 0.0 8316 632 ? D 21:19 0:34 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdd1/zero-1=0A= root 5238 30.5 0.0 8316 632 ? R 21:19 0:38 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sde1/zero-1=0A= root 5241 29.7 0.0 8316 628 ? R 21:19 0:37 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdf1/zero-1=0A= root 5244 25.8 0.0 8316 628 ? D 21:19 0:32 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdg1/zero-1=0A= root 5247 23.4 0.0 8316 632 ? D 21:19 0:29 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdh1/zero-1=0A= root 5250 23.4 0.0 8316 632 ? R 21:19 0:29 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdi1/zero-1=0A= root 5253 23.7 0.0 8316 632 ? D 21:19 0:30 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdj1/zero-1=0A= root 5256 22.3 0.0 8316 632 ? D 21:19 0:28 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdk1/zero-1=0A= root 5259 23.7 0.0 8316 628 ? D 21:19 0:30 dd bs=3D4k= if=3D/dev/zero of=3D/fs/sdl1/zero-1=0A= =0A= [short 1MB dd command which is repeated for many times]=0A= wfg@lkp-nex04 /fs/sdj1% sudo dd if=3D/dev/zero of=3Dzero2 bs=3D1M count=3D= 1=0A= =0A= The long running dd is progressing at 60-70MB/s, however the small 1MB dd i= s=0A= rather slow in all but one invocations:=0A= =0A= 1048576 bytes (1.0 MB) copied, 0.2296 s, 4.6 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.366903 s, 2.9 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.47013 s, 2.2 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.257523 s, 4.1 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.36692 s, 2.9 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.473319 s, 2.2 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.50198 s, 2.1 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.018758 s, 55.9 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.341166 s, 3.1 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.348311 s, 3.0 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.418185 s, 2.5 MB/s=0A= 1048576 bytes (1.0 MB) copied, 0.444071 s, 2.4 MB/s=0A= =0A= And the 10MB dd can achieve much better throughput:=0A= =0A= wfg@lkp-nex04 /fs/sdj1% sudo dd if=3D/dev/zero of=3Dzero2 bs=3D1M count=3D= 10=0A= =0A= 10485760 bytes (10 MB) copied, 0.417411 s, 25.1 MB/s=0A= 10485760 bytes (10 MB) copied, 0.468597 s, 22.4 MB/s=0A= =0A= Below is the balance_dirty_pages trace events for the small 1MB/10MB dd tas= ks=0A= (dd-5253 is the long running dd task on bdi 8:144).=0A= =0A= dd-5563 [025] .... 165.504338: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950856 bdi_setpoint=3D83005 = bdi_dirty=3D81032 dirty_ratelimit=3D63408 task_ratelimit=3D63592 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5563 [025] .... 165.504703: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950942 bdi_setpoint=3D83005 = bdi_dirty=3D81088 dirty_ratelimit=3D63408 task_ratelimit=3D63528 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294824915 period=3D2 think=3D429= 4824917=0A= dd-5563 [025] .... 165.505039: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951051 bdi_setpoint=3D83005 = bdi_dirty=3D81088 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5563 [025] .... 165.505469: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950754 bdi_setpoint=3D82981 = bdi_dirty=3D81200 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5563 [025] .... 165.505971: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950845 bdi_setpoint=3D82973 = bdi_dirty=3D81200 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5563 [025] .... 165.506398: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950999 bdi_setpoint=3D82973 = bdi_dirty=3D81312 dirty_ratelimit=3D63408 task_ratelimit=3D63408 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= =0A= The traces show that balance_dirty_pages is trying to throttle the short li= ved=0A= dd tasks at around task_ratelimit=3D63592 KB/s, which is the right thing to= do.=0A= =0A= Judging from the timestamps and the pause=3DX fields, we can tell that=0A= =0A= - the short lived dd tasks spend very little time in balance_dirty_pages()= =0A= =0A= For dd-5563, pause=3D0,-4294824915,0,0,0,7 which adds up to 7ms.=0A= =0A= - 99% block time should occur in loading the dd process and allocating the = memory=0A= =0A= Because the dd-5563 balance_dirty_pages events start at 165.504338 and ends= at=0A= 165.506398 which only takes 2ms in between.=0A= =0A= Thanks,=0A= Fengguang=0A= ---=0A= wfg@bee /export/writeback/lkp-nex04/JBOD-10HDD-thresh=3D4G/ext4-1dd-1-3.3.0= -rc1% bzgrep 8:144: trace.bz2 |g -v dd-5253|g dd=0A= dd-5563 [025] .... 165.504338: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950856 bdi_setpoint=3D83005 = bdi_dirty=3D81032 dirty_ratelimit=3D63408 task_ratelimit=3D63592 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5563 [025] .... 165.504703: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950942 bdi_setpoint=3D83005 = bdi_dirty=3D81088 dirty_ratelimit=3D63408 task_ratelimit=3D63528 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294824915 period=3D2 think=3D429= 4824917=0A= dd-5563 [025] .... 165.505039: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951051 bdi_setpoint=3D83005 = bdi_dirty=3D81088 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5563 [025] .... 165.505469: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950754 bdi_setpoint=3D82981 = bdi_dirty=3D81200 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5563 [025] .... 165.505971: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950845 bdi_setpoint=3D82973 = bdi_dirty=3D81200 dirty_ratelimit=3D63408 task_ratelimit=3D63468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5563 [025] .... 165.506398: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950999 bdi_setpoint=3D82973 = bdi_dirty=3D81312 dirty_ratelimit=3D63408 task_ratelimit=3D63408 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5572 [041] .... 167.463729: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950826 bdi_setpoint=3D82916 = bdi_dirty=3D83720 dirty_ratelimit=3D63400 task_ratelimit=3D61788 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5572 [041] .... 167.464038: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950760 bdi_setpoint=3D82910 = bdi_dirty=3D83720 dirty_ratelimit=3D63400 task_ratelimit=3D61788 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294826878 period=3D2 think=3D429= 4826880=0A= dd-5572 [041] .... 167.464323: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950737 bdi_setpoint=3D82910 = bdi_dirty=3D83720 dirty_ratelimit=3D63400 task_ratelimit=3D61788 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5572 [041] .... 167.464645: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950667 bdi_setpoint=3D82898 = bdi_dirty=3D83776 dirty_ratelimit=3D63400 task_ratelimit=3D61728 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5572 [041] .... 167.464964: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950804 bdi_setpoint=3D82898 = bdi_dirty=3D83832 dirty_ratelimit=3D63400 task_ratelimit=3D61728 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5572 [041] .... 167.465286: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950903 bdi_setpoint=3D82891 = bdi_dirty=3D83832 dirty_ratelimit=3D63400 task_ratelimit=3D61728 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5581 [051] .... 169.594955: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955229 bdi_setpoint=3D83567 = bdi_dirty=3D84000 dirty_ratelimit=3D63400 task_ratelimit=3D61540 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5581 [051] .... 169.595226: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955141 bdi_setpoint=3D83563 = bdi_dirty=3D84000 dirty_ratelimit=3D63400 task_ratelimit=3D61540 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294829013 period=3D2 think=3D429= 4829015=0A= dd-5581 [051] .... 169.595486: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955174 bdi_setpoint=3D83563 = bdi_dirty=3D84056 dirty_ratelimit=3D63400 task_ratelimit=3D61480 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5581 [051] .... 169.595752: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955196 bdi_setpoint=3D83563 = bdi_dirty=3D84112 dirty_ratelimit=3D63400 task_ratelimit=3D61480 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5581 [051] .... 169.596042: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955262 bdi_setpoint=3D83563 = bdi_dirty=3D84112 dirty_ratelimit=3D63400 task_ratelimit=3D61480 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5581 [051] .... 169.596334: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955196 bdi_setpoint=3D83558 = bdi_dirty=3D84168 dirty_ratelimit=3D63400 task_ratelimit=3D61416 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5592 [049] .... 171.845210: bdi_dirty_ratelimit: bdi 8:= 144: write_bw=3D64504 awrite_bw=3D63908 dirty_rate=3D22072 dirty_ratelimit= =3D63400 task_ratelimit=3D63960 balanced_dirty_ratelimit=3D63908=0A= dd-5592 [049] .... 171.845211: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953514 bdi_setpoint=3D84104 = bdi_dirty=3D81032 dirty_ratelimit=3D63400 task_ratelimit=3D63956 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5592 [049] .... 171.845603: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953504 bdi_setpoint=3D84096 = bdi_dirty=3D81088 dirty_ratelimit=3D63400 task_ratelimit=3D63892 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294831268 period=3D2 think=3D429= 4831270=0A= dd-5592 [049] .... 171.846095: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953608 bdi_setpoint=3D84096 = bdi_dirty=3D81144 dirty_ratelimit=3D63400 task_ratelimit=3D63832 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5592 [049] .... 171.846484: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953723 bdi_setpoint=3D84096 = bdi_dirty=3D81256 dirty_ratelimit=3D63400 task_ratelimit=3D63768 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5592 [049] .... 171.846899: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953871 bdi_setpoint=3D84096 = bdi_dirty=3D81312 dirty_ratelimit=3D63400 task_ratelimit=3D63768 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5592 [049] .... 171.847315: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953890 bdi_setpoint=3D84088 = bdi_dirty=3D81368 dirty_ratelimit=3D63400 task_ratelimit=3D63708 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5601 [049] .... 173.794360: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955596 bdi_setpoint=3D84205 = bdi_dirty=3D84504 dirty_ratelimit=3D63400 task_ratelimit=3D61604 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5601 [049] .... 173.794888: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955651 bdi_setpoint=3D84205 = bdi_dirty=3D84560 dirty_ratelimit=3D63400 task_ratelimit=3D61540 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294833221 period=3D2 think=3D429= 4833223=0A= dd-5601 [049] .... 173.795259: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955410 bdi_setpoint=3D84187 = bdi_dirty=3D84560 dirty_ratelimit=3D63400 task_ratelimit=3D61604 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5601 [049] .... 173.795634: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955375 bdi_setpoint=3D84187 = bdi_dirty=3D84616 dirty_ratelimit=3D63400 task_ratelimit=3D61540 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5601 [049] .... 173.796186: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955480 bdi_setpoint=3D84187 = bdi_dirty=3D84616 dirty_ratelimit=3D63400 task_ratelimit=3D61540 dirtied=3D= 32 dirtied_pause=3D32 paused=3D1 pause=3D0 period=3D2 think=3D-2=0A= dd-5601 [049] .... 173.796807: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955461 bdi_setpoint=3D84248 = bdi_dirty=3D84560 dirty_ratelimit=3D63400 task_ratelimit=3D61604 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5610 [041] .... 176.710294: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D958022 bdi_setpoint=3D82789 = bdi_dirty=3D85736 dirty_ratelimit=3D63400 task_ratelimit=3D59620 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5610 [041] .... 176.710729: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957956 bdi_setpoint=3D82781 = bdi_dirty=3D85736 dirty_ratelimit=3D63400 task_ratelimit=3D59620 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294836143 period=3D2 think=3D429= 4836145=0A= dd-5610 [041] .... 176.711224: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D958077 bdi_setpoint=3D82781 = bdi_dirty=3D85848 dirty_ratelimit=3D63400 task_ratelimit=3D59560 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5610 [041] .... 176.711747: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D958187 bdi_setpoint=3D82772 = bdi_dirty=3D85904 dirty_ratelimit=3D63400 task_ratelimit=3D59496 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5610 [041] .... 176.712235: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D958152 bdi_setpoint=3D82762 = bdi_dirty=3D85960 dirty_ratelimit=3D63400 task_ratelimit=3D59436 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5610 [041] .... 176.712732: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D958343 bdi_setpoint=3D82762 = bdi_dirty=3D86072 dirty_ratelimit=3D63400 task_ratelimit=3D59372 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5641 [049] .... 190.728161: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955683 bdi_setpoint=3D80839 = bdi_dirty=3D86408 dirty_ratelimit=3D63168 task_ratelimit=3D57924 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5641 [049] .... 190.728460: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955767 bdi_setpoint=3D80839 = bdi_dirty=3D86408 dirty_ratelimit=3D63168 task_ratelimit=3D57924 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294850188 period=3D2 think=3D429= 4850190=0A= dd-5641 [049] .... 190.728773: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955894 bdi_setpoint=3D80839 = bdi_dirty=3D86408 dirty_ratelimit=3D63168 task_ratelimit=3D57924 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5641 [049] .... 190.729086: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955889 bdi_setpoint=3D80833 = bdi_dirty=3D86464 dirty_ratelimit=3D63168 task_ratelimit=3D57860 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5641 [049] .... 190.729380: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955997 bdi_setpoint=3D80833 = bdi_dirty=3D86464 dirty_ratelimit=3D63168 task_ratelimit=3D57860 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5641 [049] .... 190.729687: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956072 bdi_setpoint=3D80833 = bdi_dirty=3D86520 dirty_ratelimit=3D63168 task_ratelimit=3D57860 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5668 [057] .... 206.594016: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957370 bdi_setpoint=3D84074 = bdi_dirty=3D83944 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 1 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D0 think=3D0=0A= dd-5668 [057] .... 206.594029: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957379 bdi_setpoint=3D84074 = bdi_dirty=3D83944 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5668 [057] .... 206.594041: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957379 bdi_setpoint=3D84074 = bdi_dirty=3D83944 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294866085 period=3D2 think=3D429= 4866087=0A= dd-5668 [057] .... 206.594051: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957379 bdi_setpoint=3D84074 = bdi_dirty=3D83944 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5668 [057] .... 206.594315: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957456 bdi_setpoint=3D84074 = bdi_dirty=3D83944 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5668 [057] .... 206.594651: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957475 bdi_setpoint=3D84166 = bdi_dirty=3D83832 dirty_ratelimit=3D63072 task_ratelimit=3D61468 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5668 [057] .... 206.595011: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957579 bdi_setpoint=3D84166 = bdi_dirty=3D83888 dirty_ratelimit=3D63072 task_ratelimit=3D61344 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5668 [057] .... 206.602945: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D957726 bdi_setpoint=3D84063 = bdi_dirty=3D84112 dirty_ratelimit=3D63072 task_ratelimit=3D61160 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5677 [057] .... 212.243213: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941736 bdi_setpoint=3D84089 = bdi_dirty=3D85232 dirty_ratelimit=3D63072 task_ratelimit=3D61900 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5677 [057] .... 212.243532: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941813 bdi_setpoint=3D84089 = bdi_dirty=3D85288 dirty_ratelimit=3D63072 task_ratelimit=3D61840 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294871746 period=3D2 think=3D429= 4871748=0A= dd-5677 [057] .... 212.243853: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941615 bdi_setpoint=3D84069 = bdi_dirty=3D85288 dirty_ratelimit=3D63072 task_ratelimit=3D61840 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5677 [057] .... 212.244175: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941648 bdi_setpoint=3D84069 = bdi_dirty=3D85344 dirty_ratelimit=3D63072 task_ratelimit=3D61776 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5677 [057] .... 212.244500: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941755 bdi_setpoint=3D84069 = bdi_dirty=3D85344 dirty_ratelimit=3D63072 task_ratelimit=3D61776 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5677 [057] .... 212.244824: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D941840 bdi_setpoint=3D84069 = bdi_dirty=3D85400 dirty_ratelimit=3D63072 task_ratelimit=3D61776 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5686 [058] .... 218.734557: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952833 bdi_setpoint=3D83354 = bdi_dirty=3D85792 dirty_ratelimit=3D63072 task_ratelimit=3D60236 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5686 [058] .... 218.734893: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952973 bdi_setpoint=3D83354 = bdi_dirty=3D85792 dirty_ratelimit=3D63072 task_ratelimit=3D60176 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294878250 period=3D2 think=3D429= 4878252=0A= dd-5686 [058] .... 218.735281: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953120 bdi_setpoint=3D83354 = bdi_dirty=3D85848 dirty_ratelimit=3D63072 task_ratelimit=3D60176 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5686 [058] .... 218.735630: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953293 bdi_setpoint=3D83354 = bdi_dirty=3D85848 dirty_ratelimit=3D63072 task_ratelimit=3D60176 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5686 [058] .... 218.735978: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953455 bdi_setpoint=3D83354 = bdi_dirty=3D85904 dirty_ratelimit=3D63072 task_ratelimit=3D60112 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5686 [058] .... 218.736332: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953595 bdi_setpoint=3D83354 = bdi_dirty=3D85904 dirty_ratelimit=3D63072 task_ratelimit=3D60052 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5695 [041] .... 227.307415: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955590 bdi_setpoint=3D83450 = bdi_dirty=3D84336 dirty_ratelimit=3D62952 task_ratelimit=3D60800 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5695 [041] .... 227.307810: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955642 bdi_setpoint=3D83443 = bdi_dirty=3D84336 dirty_ratelimit=3D62952 task_ratelimit=3D60800 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294886840 period=3D2 think=3D429= 4886842=0A= dd-5695 [041] .... 227.308235: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955772 bdi_setpoint=3D83443 = bdi_dirty=3D84448 dirty_ratelimit=3D62952 task_ratelimit=3D60676 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5695 [041] .... 227.308845: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955756 bdi_setpoint=3D83437 = bdi_dirty=3D84504 dirty_ratelimit=3D62952 task_ratelimit=3D60676 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5695 [041] .... 227.309538: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955708 bdi_setpoint=3D83430 = bdi_dirty=3D84560 dirty_ratelimit=3D62952 task_ratelimit=3D60612 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5695 [041] .... 227.310120: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955741 bdi_setpoint=3D83430 = bdi_dirty=3D84616 dirty_ratelimit=3D62952 task_ratelimit=3D60552 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5695 [041] .... 227.310631: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955622 bdi_setpoint=3D83417 = bdi_dirty=3D84616 dirty_ratelimit=3D62952 task_ratelimit=3D60612 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5717 [057] .... 277.024811: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956666 bdi_setpoint=3D83572 = bdi_dirty=3D83496 dirty_ratelimit=3D62744 task_ratelimit=3D61088 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5717 [057] .... 277.025137: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956776 bdi_setpoint=3D83572 = bdi_dirty=3D83496 dirty_ratelimit=3D62744 task_ratelimit=3D61028 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294936656 period=3D2 think=3D429= 4936658=0A= dd-5717 [057] .... 277.025605: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956609 bdi_setpoint=3D83622 = bdi_dirty=3D83440 dirty_ratelimit=3D62744 task_ratelimit=3D61148 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5717 [057] .... 277.026232: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956541 bdi_setpoint=3D83616 = bdi_dirty=3D83440 dirty_ratelimit=3D62744 task_ratelimit=3D61148 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5717 [057] .... 277.026938: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956522 bdi_setpoint=3D83611 = bdi_dirty=3D83496 dirty_ratelimit=3D62744 task_ratelimit=3D61148 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-2=0A= dd-5717 [057] .... 277.027526: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956653 bdi_setpoint=3D83611 = bdi_dirty=3D83496 dirty_ratelimit=3D62744 task_ratelimit=3D61148 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5717 [057] .... 277.028148: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D956770 bdi_setpoint=3D83605 = bdi_dirty=3D83552 dirty_ratelimit=3D62744 task_ratelimit=3D61028 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D7 period=3D2 think=3D-5=0A= dd-5726 [041] .... 285.898859: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954043 bdi_setpoint=3D81632 = bdi_dirty=3D86856 dirty_ratelimit=3D62704 task_ratelimit=3D57988 dirtied=3D= 1 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D0 think=3D0=0A= dd-5726 [041] .... 285.899181: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954098 bdi_setpoint=3D81632 = bdi_dirty=3D86856 dirty_ratelimit=3D62704 task_ratelimit=3D57988 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5726 [041] .... 285.899502: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954153 bdi_setpoint=3D81632 = bdi_dirty=3D86856 dirty_ratelimit=3D62704 task_ratelimit=3D57924 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294945548 period=3D2 think=3D429= 4945550=0A= dd-5726 [009] .... 285.899890: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954072 bdi_setpoint=3D81624 = bdi_dirty=3D86912 dirty_ratelimit=3D62704 task_ratelimit=3D57924 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5726 [009] .... 285.900243: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954066 bdi_setpoint=3D81617 = bdi_dirty=3D87024 dirty_ratelimit=3D62704 task_ratelimit=3D57864 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5726 [009] .... 285.900599: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953930 bdi_setpoint=3D81601 = bdi_dirty=3D87024 dirty_ratelimit=3D62704 task_ratelimit=3D57864 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5726 [009] .... 285.901007: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D953867 bdi_setpoint=3D81593 = bdi_dirty=3D87136 dirty_ratelimit=3D62704 task_ratelimit=3D57804 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D6 period=3D2 think=3D-4=0A= dd-5726 [018] .... 285.908755: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954143 bdi_setpoint=3D81603 = bdi_dirty=3D87192 dirty_ratelimit=3D62704 task_ratelimit=3D57680 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5726 [019] .... 285.918580: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954457 bdi_setpoint=3D81692 = bdi_dirty=3D87192 dirty_ratelimit=3D62704 task_ratelimit=3D57744 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [019] .... 285.928517: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954717 bdi_setpoint=3D81678 = bdi_dirty=3D87416 dirty_ratelimit=3D62704 task_ratelimit=3D57560 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [020] .... 285.939941: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D954777 bdi_setpoint=3D81641 = bdi_dirty=3D87584 dirty_ratelimit=3D62704 task_ratelimit=3D57436 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D7 period=3D10 think=3D3=0A= dd-5726 [021] .... 285.948496: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955090 bdi_setpoint=3D81643 = bdi_dirty=3D87808 dirty_ratelimit=3D62704 task_ratelimit=3D57252 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [021] .... 285.958924: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955120 bdi_setpoint=3D81614 = bdi_dirty=3D87976 dirty_ratelimit=3D62704 task_ratelimit=3D57128 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5726 [022] .... 285.968287: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955009 bdi_setpoint=3D81672 = bdi_dirty=3D88088 dirty_ratelimit=3D62704 task_ratelimit=3D57128 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [022] .... 285.978468: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955194 bdi_setpoint=3D81658 = bdi_dirty=3D88200 dirty_ratelimit=3D62704 task_ratelimit=3D56944 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [023] .... 285.988475: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955302 bdi_setpoint=3D81629 = bdi_dirty=3D88424 dirty_ratelimit=3D62704 task_ratelimit=3D56824 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [023] .... 285.998281: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955190 bdi_setpoint=3D81608 = bdi_dirty=3D88648 dirty_ratelimit=3D62704 task_ratelimit=3D56640 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [016] .... 286.008352: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955498 bdi_setpoint=3D81602 = bdi_dirty=3D88760 dirty_ratelimit=3D62704 task_ratelimit=3D56580 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [016] .... 286.018027: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955419 bdi_setpoint=3D81660 = bdi_dirty=3D88816 dirty_ratelimit=3D62704 task_ratelimit=3D56580 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [017] .... 286.028230: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955365 bdi_setpoint=3D81631 = bdi_dirty=3D89040 dirty_ratelimit=3D62704 task_ratelimit=3D56396 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5726 [049] .... 286.038490: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955468 bdi_setpoint=3D81610 = bdi_dirty=3D89096 dirty_ratelimit=3D62704 task_ratelimit=3D56396 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5726 [049] .... 286.048435: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D955739 bdi_setpoint=3D81682 = bdi_dirty=3D89208 dirty_ratelimit=3D62704 task_ratelimit=3D56272 dirtied=3D= 153 dirtied_pause=3D153 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [041] .... 294.267435: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950852 bdi_setpoint=3D82067 = bdi_dirty=3D84616 dirty_ratelimit=3D62664 task_ratelimit=3D59968 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D0=0A= dd-5735 [041] .... 294.267765: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950970 bdi_setpoint=3D82067 = bdi_dirty=3D84616 dirty_ratelimit=3D62664 task_ratelimit=3D59968 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D-4294953933 period=3D2 think=3D429= 4953935=0A= dd-5735 [041] .... 294.268098: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951017 bdi_setpoint=3D82060 = bdi_dirty=3D84672 dirty_ratelimit=3D62664 task_ratelimit=3D59908 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D1=0A= dd-5735 [041] .... 294.268424: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951029 bdi_setpoint=3D82060 = bdi_dirty=3D84672 dirty_ratelimit=3D62664 task_ratelimit=3D59908 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-1=0A= dd-5735 [041] .... 294.268742: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951095 bdi_setpoint=3D82060 = bdi_dirty=3D84728 dirty_ratelimit=3D62664 task_ratelimit=3D59848 dirtied=3D= 32 dirtied_pause=3D32 paused=3D0 pause=3D0 period=3D2 think=3D-3=0A= dd-5735 [041] .... 294.269061: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950833 bdi_setpoint=3D82042 = bdi_dirty=3D84728 dirty_ratelimit=3D62664 task_ratelimit=3D59848 dirtied=3D= 32 dirtied_pause=3D32 paused=3D1 pause=3D7 period=3D2 think=3D-4=0A= dd-5735 [041] .... 294.277895: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951014 bdi_setpoint=3D82044 = bdi_dirty=3D84896 dirty_ratelimit=3D62664 task_ratelimit=3D59784 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [010] .... 294.287018: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D950953 bdi_setpoint=3D82027 = bdi_dirty=3D85120 dirty_ratelimit=3D62664 task_ratelimit=3D59604 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5735 [010] .... 294.297275: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951081 bdi_setpoint=3D82070 = bdi_dirty=3D85120 dirty_ratelimit=3D62664 task_ratelimit=3D59604 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [011] .... 294.307161: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951125 bdi_setpoint=3D81988 = bdi_dirty=3D85456 dirty_ratelimit=3D62664 task_ratelimit=3D59296 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [012] .... 294.316854: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951369 bdi_setpoint=3D82043 = bdi_dirty=3D85568 dirty_ratelimit=3D62664 task_ratelimit=3D59296 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5735 [013] .... 294.327260: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951431 bdi_setpoint=3D82026 = bdi_dirty=3D85736 dirty_ratelimit=3D62664 task_ratelimit=3D59172 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [014] .... 294.337097: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D951718 bdi_setpoint=3D82022 = bdi_dirty=3D85904 dirty_ratelimit=3D62664 task_ratelimit=3D59052 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [015] .... 294.346927: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952048 bdi_setpoint=3D82017 = bdi_dirty=3D86128 dirty_ratelimit=3D62664 task_ratelimit=3D58868 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5735 [008] .... 294.357367: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952092 bdi_setpoint=3D82060 = bdi_dirty=3D86184 dirty_ratelimit=3D62664 task_ratelimit=3D58868 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [009] .... 294.366817: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952000 bdi_setpoint=3D82044 = bdi_dirty=3D86408 dirty_ratelimit=3D62664 task_ratelimit=3D58684 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5735 [010] .... 294.376936: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952338 bdi_setpoint=3D82098 = bdi_dirty=3D86464 dirty_ratelimit=3D62664 task_ratelimit=3D58624 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [011] .... 294.386956: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952088 bdi_setpoint=3D82070 = bdi_dirty=3D86632 dirty_ratelimit=3D62664 task_ratelimit=3D58560 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [012] .... 294.396998: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952441 bdi_setpoint=3D82066 = bdi_dirty=3D86744 dirty_ratelimit=3D62664 task_ratelimit=3D58440 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D8 period=3D10 think=3D2=0A= dd-5735 [013] .... 294.406820: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952484 bdi_setpoint=3D82115 = bdi_dirty=3D86800 dirty_ratelimit=3D62664 task_ratelimit=3D58440 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= dd-5735 [014] .... 294.416669: balance_dirty_pages: bdi 8:= 144: limit=3D1048576 setpoint=3D917504 dirty=3D952273 bdi_setpoint=3D82092 = bdi_dirty=3D86912 dirty_ratelimit=3D62664 task_ratelimit=3D58380 dirtied=3D= 152 dirtied_pause=3D152 paused=3D0 pause=3D9 period=3D10 think=3D1=0A= -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from psmtp.com (na3sys010amx152.postini.com [74.125.245.152]) by kanga.kvack.org (Postfix) with SMTP id 1F0796B13F0 for ; Wed, 1 Feb 2012 09:02:28 -0500 (EST) Received: by pbaa12 with SMTP id a12so1373302pba.14 for ; Wed, 01 Feb 2012 06:02:27 -0800 (PST) Date: Wed, 1 Feb 2012 22:02:17 +0800 From: Wu Fengguang Subject: Re: [Bug 12309] Large I/O operations result in poor interactive performance and high iowait times Message-ID: <20120201140217.GA11896@localhost> References: <201201201611.q0KGBPf6029256@bugzilla.kernel.org> <20120120144513.f457a58d.akpm@linux-foundation.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton , "linux-mm@kvack.org" Cc: Mel Gorman On Sat, Jan 28, 2012 at 02:44:24PM +0000, Wu, Fengguang wrote: > [replying from webmail and CC my gmail account] > > >> https://bugzilla.kernel.org/show_bug.cgi?id=12309 > > > We've had some recent updates to the world's largest bug report. > > Apparently our large-writer-paralyses-the-machine problems have > > worsened in recent kernels. > > Yeah I can reproduce the interactive problem on 3.3-rc1, and the main stalls seem to happen when loading the task and allocating memory. However I find no allocstall/nr_vmscan_write in this test box: wfg@bee /export/writeback/lkp-nex04% g vmscan JBOD*/*/vmstat-end JBOD-10HDD-thresh=1000M/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_write 0 JBOD-10HDD-thresh=1000M/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_immediate_reclaim 39380 JBOD-10HDD-thresh=100M/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_write 0 JBOD-10HDD-thresh=100M/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_immediate_reclaim 12795 JBOD-10HDD-thresh=100M/xfs-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_write 0 JBOD-10HDD-thresh=100M/xfs-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_immediate_reclaim 2328 JBOD-10HDD-thresh=4G/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_write 0 JBOD-10HDD-thresh=4G/ext4-1dd-1-3.3.0-rc1/vmstat-end:nr_vmscan_immediate_reclaim 42120 wfg@bee /export/writeback/lkp-nex04% g stall JBOD*/*/vmstat-end JBOD-10HDD-thresh=1000M/ext4-1dd-1-3.3.0-rc1/vmstat-end:allocstall 0 JBOD-10HDD-thresh=100M/ext4-1dd-1-3.3.0-rc1/vmstat-end:allocstall 0 JBOD-10HDD-thresh=100M/xfs-1dd-1-3.3.0-rc1/vmstat-end:allocstall 8 JBOD-10HDD-thresh=4G/ext4-1dd-1-3.3.0-rc1/vmstat-end:allocstall 0 On the other side, I'd like to share some facts that indicate improved responsiveness since Linux 3.2: - the vmstat "allocstall" and "nr_vmscan_write" fields mostly drop to 0 thanks to Mel's IO-less page reclaim patches (data listed below) - I personally feel a lot better responsiveness when ssh into the test boxes checking things out when they are doing heavy writeback tests, thanks to the IO-less balance_dirty_pages() patches. There is a user report on impressive improvements on desktop responsiveness (see below email). Thanks, Fengguang --- wfg@bee /export/writeback/snb% g allocstall JBOD*/*3.1.0+/vmstat-end JBOD-4HDD-thresh=100M/btrfs-100dd-1-3.1.0+/vmstat-end:allocstall 38 JBOD-4HDD-thresh=100M/btrfs-100dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-10dd-1-3.1.0+/vmstat-end:allocstall 20 JBOD-4HDD-thresh=100M/btrfs-10dd-2-3.1.0+/vmstat-end:allocstall 36 JBOD-4HDD-thresh=100M/btrfs-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-100dd-1-3.1.0+/vmstat-end:allocstall 39 JBOD-4HDD-thresh=100M/ext3-100dd-2-3.1.0+/vmstat-end:allocstall 59 JBOD-4HDD-thresh=100M/ext3-10dd-1-3.1.0+/vmstat-end:allocstall 16 JBOD-4HDD-thresh=100M/ext3-10dd-2-3.1.0+/vmstat-end:allocstall 14 JBOD-4HDD-thresh=100M/ext3-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-100dd-1-3.1.0+/vmstat-end:allocstall 1502 JBOD-4HDD-thresh=100M/ext4-100dd-2-3.1.0+/vmstat-end:allocstall 953 JBOD-4HDD-thresh=100M/ext4-10dd-1-3.1.0+/vmstat-end:allocstall 158 JBOD-4HDD-thresh=100M/ext4-10dd-2-3.1.0+/vmstat-end:allocstall 107 JBOD-4HDD-thresh=100M/ext4-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-100dd-1-3.1.0+/vmstat-end:allocstall 498 JBOD-4HDD-thresh=100M/xfs-100dd-2-3.1.0+/vmstat-end:allocstall 486 JBOD-4HDD-thresh=100M/xfs-10dd-1-3.1.0+/vmstat-end:allocstall 159 JBOD-4HDD-thresh=100M/xfs-10dd-2-3.1.0+/vmstat-end:allocstall 621 JBOD-4HDD-thresh=100M/xfs-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-100dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-100dd-2-3.1.0+/vmstat-end:allocstall 29 JBOD-4HDD-thresh=1G/btrfs-10dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-10dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-100dd-1-3.1.0+/vmstat-end:allocstall 307 JBOD-4HDD-thresh=1G/ext3-100dd-2-3.1.0+/vmstat-end:allocstall 128 JBOD-4HDD-thresh=1G/ext3-10dd-1-3.1.0+/vmstat-end:allocstall 59 JBOD-4HDD-thresh=1G/ext3-10dd-2-3.1.0+/vmstat-end:allocstall 40 JBOD-4HDD-thresh=1G/ext3-1dd-1-3.1.0+/vmstat-end:allocstall 45 JBOD-4HDD-thresh=1G/ext3-1dd-2-3.1.0+/vmstat-end:allocstall 31 JBOD-4HDD-thresh=1G/ext4-100dd-1-3.1.0+/vmstat-end:allocstall 2022 JBOD-4HDD-thresh=1G/ext4-100dd-2-3.1.0+/vmstat-end:allocstall 1923 JBOD-4HDD-thresh=1G/ext4-10dd-1-3.1.0+/vmstat-end:allocstall 39 JBOD-4HDD-thresh=1G/ext4-10dd-2-3.1.0+/vmstat-end:allocstall 26 JBOD-4HDD-thresh=1G/ext4-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-100dd-1-3.1.0+/vmstat-end:allocstall 1838 JBOD-4HDD-thresh=1G/xfs-100dd-2-3.1.0+/vmstat-end:allocstall 1650 JBOD-4HDD-thresh=1G/xfs-10dd-1-3.1.0+/vmstat-end:allocstall 2294 JBOD-4HDD-thresh=1G/xfs-10dd-2-3.1.0+/vmstat-end:allocstall 2144 JBOD-4HDD-thresh=1G/xfs-1dd-1-3.1.0+/vmstat-end:allocstall 61 JBOD-4HDD-thresh=1G/xfs-1dd-2-3.1.0+/vmstat-end:allocstall 103 JBOD-4HDD-thresh=8G/btrfs-100dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-100dd-2-3.1.0+/vmstat-end:allocstall 49 JBOD-4HDD-thresh=8G/btrfs-10dd-1-3.1.0+/vmstat-end:allocstall 186 JBOD-4HDD-thresh=8G/btrfs-10dd-2-3.1.0+/vmstat-end:allocstall 139 JBOD-4HDD-thresh=8G/btrfs-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-100dd-1-3.1.0+/vmstat-end:allocstall 670 JBOD-4HDD-thresh=8G/ext3-100dd-2-3.1.0+/vmstat-end:allocstall 875 JBOD-4HDD-thresh=8G/ext3-10dd-1-3.1.0+/vmstat-end:allocstall 5 JBOD-4HDD-thresh=8G/ext3-10dd-2-3.1.0+/vmstat-end:allocstall 49 JBOD-4HDD-thresh=8G/ext3-1dd-1-3.1.0+/vmstat-end:allocstall 13 JBOD-4HDD-thresh=8G/ext3-1dd-2-3.1.0+/vmstat-end:allocstall 19 JBOD-4HDD-thresh=8G/ext4-100dd-1-3.1.0+/vmstat-end:allocstall 2784 JBOD-4HDD-thresh=8G/ext4-100dd-2-3.1.0+/vmstat-end:allocstall 1997 JBOD-4HDD-thresh=8G/ext4-10dd-1-3.1.0+/vmstat-end:allocstall 13 JBOD-4HDD-thresh=8G/ext4-10dd-2-3.1.0+/vmstat-end:allocstall 4 JBOD-4HDD-thresh=8G/ext4-1dd-1-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext4-1dd-2-3.1.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-100dd-1-3.1.0+/vmstat-end:allocstall 11282 JBOD-4HDD-thresh=8G/xfs-100dd-2-3.1.0+/vmstat-end:allocstall 11677 JBOD-4HDD-thresh=8G/xfs-10dd-1-3.1.0+/vmstat-end:allocstall 3848 JBOD-4HDD-thresh=8G/xfs-10dd-2-3.1.0+/vmstat-end:allocstall 3183 JBOD-4HDD-thresh=8G/xfs-1dd-1-3.1.0+/vmstat-end:allocstall 206 JBOD-4HDD-thresh=8G/xfs-1dd-2-3.1.0+/vmstat-end:allocstall 189 wfg@bee /export/writeback/snb% g allocstall JBOD*/*3.2.0+/vmstat-end JBOD-4HDD-thresh=100M/btrfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/btrfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext3-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/ext4-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=100M/xfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/btrfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext3-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/ext4-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=1G/xfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/btrfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-100dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext3-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext4-100dd-1-3.2.0+/vmstat-end:allocstall 45 JBOD-4HDD-thresh=8G/ext4-100dd-2-3.2.0+/vmstat-end:allocstall 33 JBOD-4HDD-thresh=8G/ext4-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext4-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext4-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/ext4-1dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-100dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-100dd-2-3.2.0+/vmstat-end:allocstall 12 JBOD-4HDD-thresh=8G/xfs-10dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-10dd-2-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-1dd-1-3.2.0+/vmstat-end:allocstall 0 JBOD-4HDD-thresh=8G/xfs-1dd-2-3.2.0+/vmstat-end:allocstall 0 wfg@bee /export/writeback/snb% g nr_vmscan_write JBOD*/*3.1.0+/vmstat-end JBOD-4HDD-thresh=100M/btrfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 61 JBOD-4HDD-thresh=100M/btrfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 97 JBOD-4HDD-thresh=100M/btrfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 38 JBOD-4HDD-thresh=100M/btrfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 33 JBOD-4HDD-thresh=100M/btrfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 13 JBOD-4HDD-thresh=100M/btrfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 3 JBOD-4HDD-thresh=100M/ext3-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 11 JBOD-4HDD-thresh=100M/ext3-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 7 JBOD-4HDD-thresh=100M/ext3-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 12 JBOD-4HDD-thresh=100M/ext3-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 26 JBOD-4HDD-thresh=100M/ext3-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 4 JBOD-4HDD-thresh=100M/ext3-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 16 JBOD-4HDD-thresh=100M/ext4-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 19 JBOD-4HDD-thresh=100M/ext4-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 13 JBOD-4HDD-thresh=100M/ext4-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 25 JBOD-4HDD-thresh=100M/ext4-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 36 JBOD-4HDD-thresh=100M/ext4-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 23 JBOD-4HDD-thresh=100M/ext4-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 11 JBOD-4HDD-thresh=100M/xfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 1 JBOD-4HDD-thresh=100M/xfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 3 JBOD-4HDD-thresh=100M/xfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 7 JBOD-4HDD-thresh=100M/xfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 1 JBOD-4HDD-thresh=100M/xfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 234 JBOD-4HDD-thresh=1G/btrfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 166 JBOD-4HDD-thresh=1G/btrfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 65 JBOD-4HDD-thresh=1G/btrfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 63 JBOD-4HDD-thresh=1G/btrfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 8 JBOD-4HDD-thresh=1G/btrfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 15 JBOD-4HDD-thresh=1G/ext3-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 46 JBOD-4HDD-thresh=1G/ext3-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 55 JBOD-4HDD-thresh=1G/ext3-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 102 JBOD-4HDD-thresh=1G/ext3-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 49 JBOD-4HDD-thresh=1G/ext3-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 91 JBOD-4HDD-thresh=1G/ext3-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 77 JBOD-4HDD-thresh=1G/ext4-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 1 JBOD-4HDD-thresh=1G/ext4-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 3 JBOD-4HDD-thresh=1G/ext4-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 14 JBOD-4HDD-thresh=1G/ext4-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 3 JBOD-4HDD-thresh=1G/ext4-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 6 JBOD-4HDD-thresh=1G/ext4-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 10 JBOD-4HDD-thresh=1G/xfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 5 JBOD-4HDD-thresh=1G/xfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 5 JBOD-4HDD-thresh=1G/xfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 8 JBOD-4HDD-thresh=1G/xfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 2 JBOD-4HDD-thresh=1G/xfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 159 JBOD-4HDD-thresh=1G/xfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 178 JBOD-4HDD-thresh=8G/btrfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 282 JBOD-4HDD-thresh=8G/btrfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 203 JBOD-4HDD-thresh=8G/btrfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 107 JBOD-4HDD-thresh=8G/btrfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 82 JBOD-4HDD-thresh=8G/btrfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 19 JBOD-4HDD-thresh=8G/btrfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 46 JBOD-4HDD-thresh=8G/ext3-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 288 JBOD-4HDD-thresh=8G/ext3-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 330 JBOD-4HDD-thresh=8G/ext3-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 229 JBOD-4HDD-thresh=8G/ext3-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 227 JBOD-4HDD-thresh=8G/ext3-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 171 JBOD-4HDD-thresh=8G/ext3-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 129 JBOD-4HDD-thresh=8G/ext4-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 13 JBOD-4HDD-thresh=8G/ext4-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 453 JBOD-4HDD-thresh=8G/ext4-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 224 JBOD-4HDD-thresh=8G/ext4-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 14 JBOD-4HDD-thresh=8G/ext4-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 11 JBOD-4HDD-thresh=8G/xfs-100dd-1-3.1.0+/vmstat-end:nr_vmscan_write 7 JBOD-4HDD-thresh=8G/xfs-100dd-2-3.1.0+/vmstat-end:nr_vmscan_write 5 JBOD-4HDD-thresh=8G/xfs-10dd-1-3.1.0+/vmstat-end:nr_vmscan_write 1126 JBOD-4HDD-thresh=8G/xfs-10dd-2-3.1.0+/vmstat-end:nr_vmscan_write 98 JBOD-4HDD-thresh=8G/xfs-1dd-1-3.1.0+/vmstat-end:nr_vmscan_write 252 JBOD-4HDD-thresh=8G/xfs-1dd-2-3.1.0+/vmstat-end:nr_vmscan_write 278 wfg@bee /export/writeback/snb% g nr_vmscan_write JBOD*/*3.2.0+/vmstat-end JBOD-4HDD-thresh=100M/btrfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/btrfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/btrfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/btrfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/btrfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/btrfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext3-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/ext4-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=100M/xfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/btrfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext3-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/ext4-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=1G/xfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/btrfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext3-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/ext4-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-100dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-100dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-10dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-10dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-1dd-1-3.2.0+/vmstat-end:nr_vmscan_write 0 JBOD-4HDD-thresh=8G/xfs-1dd-2-3.2.0+/vmstat-end:nr_vmscan_write 0 From: Manuel Krause To: "Wu, Fengguang" Cc: Bcc: Subject: Re: Encouraged by Con Kolivas: Your IO-less-dirty-throttling patches v12 Reply-To: In-Reply-To: <20111217022028.GA19178@localhost> user-agent: Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20110929 Thunderbird/7.0.1 Hi Fengguang, On 2011-12-17 03:20, Wu Fengguang wrote: > Hi Manuel, > >> Hi, >> >> I'm running an openSUSE 11.4 with a 3.1.4 (non-vanilla) kernel >> source that is somehow patched by SUSE at first. Upon that I am >> used to patch the most recent CK and BFQ patches since years. >> >> I'm following Con Kolivas' Blog, currently >> http://ck-hack.blogspot.com/2011/11/310-ck2-bfs-0415.html?showComment=1324079830206#c3565024166985189501 >> (There you'll find many things I'd write in the following. So to >> speak that's a kind of history.) >> >> Some kernels ago my usual workflow broke, as there were too many >> stalls and hickups when watching a video via vlc at the same time. >> >> My "testcase" (as Con named it) is the following: >> Old system >> PIII-1400 (Tualatin) >> SDRAM 2GB (Kingston, highest timings) >> ASUS-TUV4X, VIA Chipset, 133/33MHz, AGP 4x >> normally 2 IDE disks @UDMA-100 (now, recently, one failed) >> reiserfs 3.6 as standard FS >> swap as partiton on second disk with 4GB (originally, now on >> first disk). >> shmfs mounted with 3GB >> >> So I have 2GB RAM, 4GB swap and 3GB shmfs. Where shmfs and swap >> and RAM may interfere. > > Yeah, the shmfs looks large. The original idea behind this setup was: The normal file sizes to be decoded are about 400 to 800MB and would fit into the RAM part of the overlapping shmfs resulting in a fast processing, what is true. Files bigger than that would simply swap out to disk. I originally thought that to be as simple as if I decoded directly from disk to disk (and so giving the swap partition an extra use than to only be a suspend-to-disk area). But that obviously isn't the case (any more), read below. From some kernel version on (that I don't remember) it became worse with stuttering, stalling or even hours to wait for the system to recover. > >> >> I then decode an encrypted file to /dev/shm (Simple cp makes the >> same issues without extra CPU utilization). Open it in Avidemux >> and cut it and copy it back to the original partition (with >> previously deleting the original file on the later=same target >> partition). > > And it's a bit complex operations. Did you do that _while_ running > vlc? No wonder vlc is impacted.. Yes, of course! I've found that to be a good test of system responsiveness / low latency / advantage of the CK patchset over vanilla. Also it's a usual workflow for me. Scrolling a website in firefox does show the same stutterings like vlc video playback, btw. Watching a video while decoding is just more convenient ;-) >> Whenever the file's size exceeded about 900MB/1GB, whether to be >> decoded to shm, later to be opened by Avidemux or to be processed >> in Avidemux at the end, my system began to stall often and for >> long times. Sometimes for minutes, sometimes for hours. Because >> of weird swapping behavior. Setting back some vm things like >> swappiness or dirty_ratio to kernel defaults (against ck2) did >> not help out. >> > > Yeah, that stress use of your memory can easily lead to a slow > responding system. Last night I retested the decoding, now, from and to the same partition and there was no loss of desktop responsiveness or stuttering in vlc video playback at all (same 1.2GB file as in the shmfs test before). > >> Yesterday I applied your 11 patches from v12 and they >> applied+compiled fine. And they seem to work like a charm. I'd >> consider that as a great effort! > > Thanks for trying it out! It's actually a big surprising for me. What makes you be surprised? > > FYI, most of the patches are now in 3.2-rcX and the others are pending > for 3.3. So you won't need to carry the patches for long time :-) Nice to hear that, but patching is no problem. Can you tell what patches remain pending for 3.3? > >> Most of the hickups in vlc video playing (at the same time) are >> gone now, they are much shorter, when decoding like explained >> above. Also the recovery times and the overall processing time at >> all ist much shorter due to much fewer stalls. >> >> I know this is no "benchmark" as usual, but it definitely shows >> that your patches WILL have benefit for normal users. > > Thanks for the feedback, I appreciate it a lot! :-) Feel free to use it as argument in discussions about the usefulness of your patches. If you have any ideas on how to improve my vm experience in my setup or experimental patches or hints for tunable variables, I would be glad to read from you, best regards, and cheers, too, Manuel Krause -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org