* Re: Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31,... [not found] ` <20091102135554.b10ece3e.akpm@linux-foundation.org> @ 2009-11-03 10:06 ` Christoph Hellwig 2009-11-03 10:42 ` NeilBrown 0 siblings, 1 reply; 5+ messages in thread From: Christoph Hellwig @ 2009-11-03 10:06 UTC (permalink / raw) To: device-mapper development Cc: linux-fsdevel, linux-raid, Laurent CORBES, linux-kernel On Mon, Nov 02, 2009 at 01:55:54PM -0800, Andrew Morton wrote: > On Tue, 13 Oct 2009 12:09:55 +0200 > Laurent CORBES <laurent.corbes@smartjog.com> wrote: > > > Hi all, > > > > While benchmarking some systems I discover a big sequential read performance > > drop using ext3 on ~ big files. The drop seems to be introduced in 2.6.30. I'm > > testing with 2.6.28.6 -> 2.6.29.6 -> 2.6.30.4 -> 2.6.31.3. > > Seems that large performance regressions aren't of interest to this > list :( No sure which list you mean, but dm-devel is for dm, not md. We're also seeing similarly massive performance drops with md and ext3/xfs as already reported on the list. Someone tracked it down to writeback changes as usual, but there it got stuck. ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31,... 2009-11-03 10:06 ` Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31, Christoph Hellwig @ 2009-11-03 10:42 ` NeilBrown 2009-11-03 10:55 ` [dm-devel] " Laurent CORBES 2009-11-03 16:50 ` Andrew Morton 0 siblings, 2 replies; 5+ messages in thread From: NeilBrown @ 2009-11-03 10:42 UTC (permalink / raw) To: device-mapper development, akpm Cc: linux-fsdevel, linux-raid, Laurent CORBES, linux-kernel On Tue, November 3, 2009 9:06 pm, Christoph Hellwig wrote: > On Mon, Nov 02, 2009 at 01:55:54PM -0800, Andrew Morton wrote: >> On Tue, 13 Oct 2009 12:09:55 +0200 >> Laurent CORBES <laurent.corbes@smartjog.com> wrote: >> >> > Hi all, >> > >> > While benchmarking some systems I discover a big sequential read >> performance >> > drop using ext3 on ~ big files. The drop seems to be introduced in >> 2.6.30. I'm >> > testing with 2.6.28.6 -> 2.6.29.6 -> 2.6.30.4 -> 2.6.31.3. >> >> Seems that large performance regressions aren't of interest to this >> list :( > > No sure which list you mean, but dm-devel is for dm, not md. We're also > seeing similarly massive performance drops with md and ext3/xfs as > already reported on the list. Someone tracked it down to writeback > changes as usual, but there it got stuck. I'm still looking - running some basic tests on 4 filesystems over half a dozen recent kernels to see what has been happening. I have a suspicion that there a multiple problems. In particular, XFS has a strange degradation which was papered over by commit c8a4051c3731b. I'm beginning to wonder if it was caused by commit 17bc6c30cf6bf but I haven't actually tested that yet. NeilBrown ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dm-devel] Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31,... 2009-11-03 10:42 ` NeilBrown @ 2009-11-03 10:55 ` Laurent CORBES 2009-11-04 7:16 ` Neil Brown 2009-11-03 16:50 ` Andrew Morton 1 sibling, 1 reply; 5+ messages in thread From: Laurent CORBES @ 2009-11-03 10:55 UTC (permalink / raw) To: NeilBrown Cc: device-mapper development, akpm, linux-fsdevel, linux-raid, linux-kernel Hi all, > >> > Hi all, > >> > > >> > While benchmarking some systems I discover a big sequential read > >> performance > >> > drop using ext3 on ~ big files. The drop seems to be introduced in > >> 2.6.30. I'm > >> > testing with 2.6.28.6 -> 2.6.29.6 -> 2.6.30.4 -> 2.6.31.3. > >> > >> Seems that large performance regressions aren't of interest to this > >> list :( Or +200MB/s is enough for a lot of people :) > > No sure which list you mean, but dm-devel is for dm, not md. We're also > > seeing similarly massive performance drops with md and ext3/xfs as > > already reported on the list. Someone tracked it down to writeback > > changes as usual, but there it got stuck. > > I'm still looking - running some basic tests on 4 filesystems over > half a dozen recent kernels to see what has been happening. > > I have a suspicion that there a multiple problems. > In particular, XFS has a strange degradation which was papered over > by commit c8a4051c3731b. > I'm beginning to wonder if it was caused by commit 17bc6c30cf6bf > but I haven't actually tested that yet. What is really strange is that from all the tests I did the raw md perfs never dropped. only a few MB of diff between kernel (~2%). This is maybe related to the way upper FS write datas on the md layer. I'll make the tests on raw disks to see if there is some troubles here also. I can also test with other raid layers. Is there any tuning/debug I can make for you ? I can also setup a remote access to this system if needed. Thanks. -- Laurent Corbes - laurent.corbes@smartjog.com SmartJog SAS | Phone: +33 1 5868 6225 | Fax: +33 1 5868 6255 | www.smartjog.com 27 Blvd Hippolyte Marquès, 94200 Ivry-sur-Seine, France A TDF Group company -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dm-devel] Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31,... 2009-11-03 10:55 ` [dm-devel] " Laurent CORBES @ 2009-11-04 7:16 ` Neil Brown 0 siblings, 0 replies; 5+ messages in thread From: Neil Brown @ 2009-11-04 7:16 UTC (permalink / raw) To: Laurent CORBES Cc: device-mapper development, akpm, linux-fsdevel, linux-raid, linux-kernel [-- Attachment #1: message body text --] [-- Type: text/plain, Size: 984 bytes --] On Tuesday November 3, laurent.corbes@smartjog.com wrote: > > What is really strange is that from all the tests I did the raw md perfs never > dropped. only a few MB of diff between kernel (~2%). This is maybe related to > the way upper FS write datas on the md layer. That isn't all that strange. It just says that the problem isn't with MD, but is in some other part of Linux closer to the filesystem. I did some tests with a range of kernels (all 'mainline', not the 'stable' versions that you used) and while I do see a noticeable dip at 2.6.30 (except with ext3) is see improved performance in 2.6.31 and even greater improvements with 2.6.32-rc5. So while I confirm that 2.6.30 is worse than earlier kernels, and that there was a general decline leading to that point, things have become dramatically better. So I don't think it is worth exploring very deeply. All the numbers in the graph come from 'bonnie' over the various file-systems on a 5-drive RAID6. NeilBrown [-- Attachment #2: read.png --] [-- Type: image/png, Size: 7396 bytes --] ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31,... 2009-11-03 10:42 ` NeilBrown 2009-11-03 10:55 ` [dm-devel] " Laurent CORBES @ 2009-11-03 16:50 ` Andrew Morton 1 sibling, 0 replies; 5+ messages in thread From: Andrew Morton @ 2009-11-03 16:50 UTC (permalink / raw) To: NeilBrown Cc: linux-fsdevel, linux-raid, device-mapper development, Laurent CORBES, linux-kernel On Tue, 3 Nov 2009 21:42:30 +1100 "NeilBrown" <neilb@suse.de> wrote: > On Tue, November 3, 2009 9:06 pm, Christoph Hellwig wrote: > > On Mon, Nov 02, 2009 at 01:55:54PM -0800, Andrew Morton wrote: > >> On Tue, 13 Oct 2009 12:09:55 +0200 > >> Laurent CORBES <laurent.corbes@smartjog.com> wrote: > >> > >> > Hi all, > >> > > >> > While benchmarking some systems I discover a big sequential read > >> performance > >> > drop using ext3 on ~ big files. The drop seems to be introduced in > >> 2.6.30. I'm > >> > testing with 2.6.28.6 -> 2.6.29.6 -> 2.6.30.4 -> 2.6.31.3. > >> > >> Seems that large performance regressions aren't of interest to this > >> list :( > > > > No sure which list you mean, but dm-devel is for dm, not md. bah. > We're also > > seeing similarly massive performance drops with md and ext3/xfs as > > already reported on the list. Someone tracked it down to writeback > > changes as usual, but there it got stuck. > > I'm still looking - running some basic tests on 4 filesystems over > half a dozen recent kernels to see what has been happening. > > I have a suspicion that there a multiple problems. > In particular, XFS has a strange degradation which was papered over > by commit c8a4051c3731b. > I'm beginning to wonder if it was caused by commit 17bc6c30cf6bf > but I haven't actually tested that yet. I think Laurent's workload involves only reads, with no writes. ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2009-11-04 7:16 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <20091013120955.6bd5844b@smartjog.com>
[not found] ` <20091102135554.b10ece3e.akpm@linux-foundation.org>
2009-11-03 10:06 ` Re: Ext3 sequential read performance drop 2.6.29 -> 2.6.30,2.6.31, Christoph Hellwig
2009-11-03 10:42 ` NeilBrown
2009-11-03 10:55 ` [dm-devel] " Laurent CORBES
2009-11-04 7:16 ` Neil Brown
2009-11-03 16:50 ` Andrew Morton
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).