From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756503Ab1HXJcL (ORCPT ); Wed, 24 Aug 2011 05:32:11 -0400 Received: from mga14.intel.com ([143.182.124.37]:50200 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751557Ab1HXJcJ (ORCPT ); Wed, 24 Aug 2011 05:32:09 -0400 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.68,274,1312182000"; d="scan'208";a="41839535" Date: Wed, 24 Aug 2011 17:32:05 +0800 From: Wu Fengguang To: Pekka Enberg Cc: Stefan Priebe - Profihost AG , LKML , "linux-mm@kvack.org" , Andrew Morton , Mel Gorman , Jens Axboe , Linux Netdev List Subject: Re: slow performance on disk/network i/o full speed after drop_caches Message-ID: <20110824093205.GA5214@localhost> References: <4E5494D4.1050605@profihost.ag> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.20 (2009-06-14) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wed, Aug 24, 2011 at 02:20:07PM +0800, Pekka Enberg wrote: > On Wed, Aug 24, 2011 at 9:06 AM, Stefan Priebe - Profihost AG > wrote: > > i hope this is the correct list to write to if it would be nice to give me a > > hint where i can ask. > > > > Kernel: 2.6.38 > > > > I'm seeing some strange problems on some of our servers after upgrading to > > 2.6.38. > > > > I'm copying a 1GB file via scp from Machine A to Machine B. When B is > > freshly booted the file transfer is done with about 80 to 85 Mb/s. I can > > repeat that various times to performance degrease. > > > > Then after some days copying is only done with about 900kb/s up to 3Mb/s > > going up and down while transfering the file. > > > > When i then do drop_caches it works again on 80Mb/s. > > > > sync && echo 3 >/proc/sys/vm/drop_caches && sleep 2 && echo 0 > >>/proc/sys/vm/drop_caches > > > > Attached is also an output of meminfo before and after drop_caches. > > > > What's going on here? MemFree is pretty high. > > > > Please CC me i'm not on list. > > Interesting. I can imagine one or more of the following to be > involved: networking, vmscan, block, and writeback. Lets CC all of > them! > > > # before drop_caches > > > > # cat /proc/meminfo > > MemTotal:        8185544 kB > > MemFree:         6670292 kB > > Buffers:          105164 kB > > Cached:           166672 kB > > SwapCached:            0 kB > > Active:           728308 kB > > Inactive:         567428 kB > > Active(anon):     639204 kB > > Inactive(anon):   394932 kB > > Active(file):      89104 kB > > Inactive(file):   172496 kB > > Unevictable:        2976 kB > > Mlocked:            2992 kB > > SwapTotal:       1464316 kB > > SwapFree:        1464316 kB > > Dirty:                52 kB > > Writeback:             0 kB Since dirty/writeback pages are low, it seems not being throttled by balance_dirty_pages(). Stefan, would you please run this several times on the server? ps -eo user,pid,tid,class,rtprio,ni,pri,psr,pcpu,vsz,rss,pmem,stat,wchan:28,cmd | grep scp It will show where the scp task is blocked (the wchan field). Hope it helps. Thanks, Fengguang From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail137.messagelabs.com (mail137.messagelabs.com [216.82.249.19]) by kanga.kvack.org (Postfix) with SMTP id 37FC86B0169 for ; Wed, 24 Aug 2011 05:32:10 -0400 (EDT) Date: Wed, 24 Aug 2011 17:32:05 +0800 From: Wu Fengguang Subject: Re: slow performance on disk/network i/o full speed after drop_caches Message-ID: <20110824093205.GA5214@localhost> References: <4E5494D4.1050605@profihost.ag> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Pekka Enberg Cc: Stefan Priebe - Profihost AG , LKML , "linux-mm@kvack.org" , Andrew Morton , Mel Gorman , Jens Axboe , Linux Netdev List On Wed, Aug 24, 2011 at 02:20:07PM +0800, Pekka Enberg wrote: > On Wed, Aug 24, 2011 at 9:06 AM, Stefan Priebe - Profihost AG > wrote: > > i hope this is the correct list to write to if it would be nice to give me a > > hint where i can ask. > > > > Kernel: 2.6.38 > > > > I'm seeing some strange problems on some of our servers after upgrading to > > 2.6.38. > > > > I'm copying a 1GB file via scp from Machine A to Machine B. When B is > > freshly booted the file transfer is done with about 80 to 85 Mb/s. I can > > repeat that various times to performance degrease. > > > > Then after some days copying is only done with about 900kb/s up to 3Mb/s > > going up and down while transfering the file. > > > > When i then do drop_caches it works again on 80Mb/s. > > > > sync && echo 3 >/proc/sys/vm/drop_caches && sleep 2 && echo 0 > >>/proc/sys/vm/drop_caches > > > > Attached is also an output of meminfo before and after drop_caches. > > > > What's going on here? MemFree is pretty high. > > > > Please CC me i'm not on list. > > Interesting. I can imagine one or more of the following to be > involved: networking, vmscan, block, and writeback. Lets CC all of > them! > > > # before drop_caches > > > > # cat /proc/meminfo > > MemTotal: A A A A 8185544 kB > > MemFree: A A A A 6670292 kB > > Buffers: A A A A A 105164 kB > > Cached: A A A A A 166672 kB > > SwapCached: A A A A A A 0 kB > > Active: A A A A A 728308 kB > > Inactive: A A A A 567428 kB > > Active(anon): A A 639204 kB > > Inactive(anon): A 394932 kB > > Active(file): A A A 89104 kB > > Inactive(file): A 172496 kB > > Unevictable: A A A A 2976 kB > > Mlocked: A A A A A A 2992 kB > > SwapTotal: A A A 1464316 kB > > SwapFree: A A A A 1464316 kB > > Dirty: A A A A A A A A 52 kB > > Writeback: A A A A A A 0 kB Since dirty/writeback pages are low, it seems not being throttled by balance_dirty_pages(). Stefan, would you please run this several times on the server? ps -eo user,pid,tid,class,rtprio,ni,pri,psr,pcpu,vsz,rss,pmem,stat,wchan:28,cmd | grep scp It will show where the scp task is blocked (the wchan field). Hope it helps. Thanks, Fengguang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/ Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wu Fengguang Subject: Re: slow performance on disk/network i/o full speed after drop_caches Date: Wed, 24 Aug 2011 17:32:05 +0800 Message-ID: <20110824093205.GA5214@localhost> References: <4E5494D4.1050605@profihost.ag> Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Cc: Stefan Priebe - Profihost AG , LKML , "linux-mm@kvack.org" , Andrew Morton , Mel Gorman , Jens Axboe , Linux Netdev List To: Pekka Enberg Return-path: Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-Id: netdev.vger.kernel.org On Wed, Aug 24, 2011 at 02:20:07PM +0800, Pekka Enberg wrote: > On Wed, Aug 24, 2011 at 9:06 AM, Stefan Priebe - Profihost AG > wrote: > > i hope this is the correct list to write to if it would be nice to gi= ve me a > > hint where i can ask. > > > > Kernel: 2.6.38 > > > > I'm seeing some strange problems on some of our servers after upgradi= ng to > > 2.6.38. > > > > I'm copying a 1GB file via scp from Machine A to Machine B. When B is > > freshly booted the file transfer is done with about 80 to 85 Mb/s. I = can > > repeat that various times to performance degrease. > > > > Then after some days copying is only done with about 900kb/s up to 3M= b/s > > going up and down while transfering the file. > > > > When i then do drop_caches it works again on 80Mb/s. > > > > sync && echo 3 >/proc/sys/vm/drop_caches && sleep 2 && echo 0 > >>/proc/sys/vm/drop_caches > > > > Attached is also an output of meminfo before and after drop_caches. > > > > What's going on here? MemFree is pretty high. > > > > Please CC me i'm not on list. >=20 > Interesting. I can imagine one or more of the following to be > involved: networking, vmscan, block, and writeback. Lets CC all of > them! >=20 > > # before drop_caches > > > > # cat /proc/meminfo > > MemTotal: =C2=A0 =C2=A0 =C2=A0 =C2=A08185544 kB > > MemFree: =C2=A0 =C2=A0 =C2=A0 =C2=A0 6670292 kB > > Buffers: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0105164 kB > > Cached: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 166672 kB > > SwapCached: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00 kB > > Active: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 728308 kB > > Inactive: =C2=A0 =C2=A0 =C2=A0 =C2=A0 567428 kB > > Active(anon): =C2=A0 =C2=A0 639204 kB > > Inactive(anon): =C2=A0 394932 kB > > Active(file): =C2=A0 =C2=A0 =C2=A089104 kB > > Inactive(file): =C2=A0 172496 kB > > Unevictable: =C2=A0 =C2=A0 =C2=A0 =C2=A02976 kB > > Mlocked: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A02992 kB > > SwapTotal: =C2=A0 =C2=A0 =C2=A0 1464316 kB > > SwapFree: =C2=A0 =C2=A0 =C2=A0 =C2=A01464316 kB > > Dirty: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A052 kB > > Writeback: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0 kB Since dirty/writeback pages are low, it seems not being throttled by balance_dirty_pages(). Stefan, would you please run this several times on the server? ps -eo user,pid,tid,class,rtprio,ni,pri,psr,pcpu,vsz,rss,pmem,stat,wchan:= 28,cmd | grep scp It will show where the scp task is blocked (the wchan field). Hope it hel= ps. Thanks, Fengguang -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Fight unfair telecom internet charges in Canada: sign http://stopthemeter= .ca/ Don't email: email@kvack.org