From mboxrd@z Thu Jan 1 00:00:00 1970 From: mark delfman Subject: Re: MD write performance issue - found Catalyst patches Date: Sun, 18 Oct 2009 11:00:22 +0100 Message-ID: <66781b10910180300j2006a4b7q21444bb27dd9434e@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary=0016364272288d92dd047632b259 Return-path: Sender: linux-raid-owner@vger.kernel.org To: =?ISO-8859-1?Q?Mattias_Hellstr=F6m?= , Linux RAID Mailing List , NeilBrown List-Id: linux-raid.ids --0016364272288d92dd047632b259 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable We have tracked the performance drop to the attached two commits in 2.6.28.6. The performance never fully recovers in later kernels so I presuming that the change in the write cache is still affecting MD today. The problem for us is that although we have slowly tracked it down, we have no understanding of linux at this level and simply wouldn=92t know where go from this point. Considering this seems to only effect MD and not hardware based RAID (in our tests) I thought that this would be an appropriate place to post these patches and findings. There are 2 patches which impact MD performance via a filesystem: a) commit 66c85494570396661479ba51e17964b2c82b6f39 - write-back: fix nr_to_write counter b) commit fa76ac6cbeb58256cf7de97a75d5d7f838a80b32 - Fix page writeback thinko, causing Berkeley DB slowdown 1) no patches applied into 2.6.28.5 kernel: write speed is 1.1 GB/s via xfs 2) both patches are applied into 2.6.28.5 kernel: xfs drops to circa: 680 MB/s (like in kernel 2.6.28.6 and later) 3) put only one patch: 66c85494570396661479ba51e17964b2c82b6f39 (write-back: fix nr_to_write counter) - performance goes down to circa 780 MB/s 4) put only one patch: fa76ac6cbeb58256cf7de97a75d5d7f838a80b32 (Fix page writeback thinko) - the performance is good: 1.1 GB/s (on XFS) change log for 28.6 ftp://ftp.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.28.6 Hopefully this helps to resolve this.... Mark 2009/10/17 Mattias Hellstr=F6m : > (If I were you) I would further test the revisions between the > following and then look at the changelog for the culprit. Looks like > versions after this are just trying to regain the missing speed. > > Linux linux-tlfp 2.6.27.14-vanilla #1 SMP Fri Oct 16 00:56:25 BST 2009 > x86_64 x86_64 x86_64 GNU/Linux > > RAW: 1.1 > XFS 1.1 > > Linux linux-tlfp 2.6.27.20-vanilla #1 SMP Thu Oct 15 23:59:32 BST 2009 > x86_64 x86_64 x86_64 GNU/Linux > > RAw 1.1 GB/s > XFS: 487 MB/s > --0016364272288d92dd047632b259 Content-Type: application/octet-stream; name="fa76ac6cbeb58256cf7de97a75d5d7f838a80b32.patch" Content-Disposition: attachment; filename="fa76ac6cbeb58256cf7de97a75d5d7f838a80b32.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_g0xmjzks0 Y29tbWl0IGZhNzZhYzZjYmViNTgyNTZjZjdkZTk3YTc1ZDVkN2Y4MzhhODBiMzIKQXV0aG9yOiBO aWNrIFBpZ2dpbiA8bnBpZ2dpbkBzdXNlLmRlPgpEYXRlOiAgIFRodSBGZWIgMTIgMDQ6MzQ6MjMg MjAwOSArMDEwMAoKICAgIEZpeCBwYWdlIHdyaXRlYmFjayB0aGlua28sIGNhdXNpbmcgQmVya2Vs ZXkgREIgc2xvd2Rvd24KICAgIAogICAgY29tbWl0IDNhNGM2ODAwZjMxZWE4Mzk1NjI4YWY1ZTdl NDkwMjcwZWU1ZDA1ODUgdXBzdHJlYW0uCiAgICAKICAgIEEgYnVnIHdhcyBpbnRyb2R1Y2VkIGlu dG8gd3JpdGVfY2FjaGVfcGFnZXMgY3ljbGljIHdyaXRlb3V0IGJ5IGNvbW1pdAogICAgMzFhMTI2 NjZkOGYwYzIyMjM1Mjk3ZTFjMTU3NWY4MjA2MTQ4MDAyOSAoIm1tOiB3cml0ZV9jYWNoZV9wYWdl cyBjeWNsaWMKICAgIGZpeCIpLiAgVGhlIGludGVudGlvbiAoYW5kIGNvbW1lbnRzKSBpcyB0aGF0 IHdlIHNob3VsZCBjeWNsZSBiYWNrIGFuZAogICAgbG9vayBmb3IgbW9yZSBkaXJ0eSBwYWdlcyBh dCB0aGUgYmVnaW5uaW5nIG9mIHRoZSBmaWxlIGlmIHRoZXJlIGlzIG5vCiAgICBtb3JlIHdvcmsg dG8gYmUgZG9uZS4KICAgIAogICAgQnV0IHRoZSAhZG9uZSBjb25kaXRpb24gd2FzIGRyb3BwZWQg ZnJvbSB0aGUgdGVzdC4gIFRoaXMgbWVhbnMgdGhhdCBhbnkKICAgIHRpbWUgdGhlIHBhZ2Ugd3Jp dGVvdXQgbG9vcCBicmVha3MgKGVnLiAgZHVlIHRvIG5yX3RvX3dyaXRlID09IDApLCB3ZQogICAg d2lsbCBzZXQgaW5kZXggdG8gMCwgdGhlbiBnb3RvIGFnYWluLiAgVGhpcyB3aWxsIHNldCBkb25l X2luZGV4IHRvCiAgICBpbmRleCwgdGhlbiBmaW5kIGRvbmUgaXMgc2V0LCBzbyB3aWxsIHByb2Nl ZWQgdG8gdGhlIGVuZCBvZiB0aGUKICAgIGZ1bmN0aW9uLiAgV2hlbiB1cGRhdGluZyBtYXBwaW5n LT53cml0ZWJhY2tfaW5kZXggZm9yIGN5Y2xpYyB3cml0ZW91dCwKICAgIHdlIG5vdyB1c2UgZG9u ZV9pbmRleCA9PSAwLCBzbyB3ZSdyZSBhbHdheXMgY3ljbGluZyBiYWNrIHRvIDAuCiAgICAKICAg IFRoaXMgc2VlbWVkIHRvIGJlIGNhdXNpbmcgcmFuZG9tIG1tYXAgd3JpdGVzIChzbGFwYWRkIGFu ZCBpb3pvbmUpIHRvCiAgICBzdGFydCB3cml0aW5nIG1vcmUgcGFnZXMgZnJvbSB0aGUgTFJVIGFu ZCB3cml0ZW91dCB3b3VsZCBzbG93ZG93biwgYW5kCiAgICBjYXVzZWQgYnVnemlsbGEgZW50cnkK ICAgIAogICAgCWh0dHA6Ly9idWd6aWxsYS5rZXJuZWwub3JnL3Nob3dfYnVnLmNnaT9pZD0xMjYw NAogICAgCiAgICBhYm91dCBCZXJrZWxleSBEQiBzbG93aW5nIGRvd24gZHJhbWF0aWNhbGx5Lgog ICAgCiAgICBXaXRoIHRoaXMgcGF0Y2gsIGlvem9uZSByYW5kb20gd3JpdGUgcGVyZm9ybWFuY2Ug aXMgaW5jcmVhc2VkIG5lYXJseQogICAgNXggb24gbXkgc3lzdGVtIChpb3pvbmUgLUIgLXIgNGsg LXMgNjRrIC1zIDUxMm0gLXMgMTIwMG0gb24gZXh0MikuCiAgICAKICAgIFNpZ25lZC1vZmYtYnk6 IE5pY2sgUGlnZ2luIDxucGlnZ2luQHN1c2UuZGU+CiAgICBSZXBvcnRlZC1hbmQtdGVzdGVkLWJ5 OiBKYW4gS2FyYSA8amFja0BzdXNlLmN6PgogICAgU2lnbmVkLW9mZi1ieTogTGludXMgVG9ydmFs ZHMgPHRvcnZhbGRzQGxpbnV4LWZvdW5kYXRpb24ub3JnPgogICAgU2lnbmVkLW9mZi1ieTogR3Jl ZyBLcm9haC1IYXJ0bWFuIDxncmVna2hAc3VzZS5kZT4KCmRpZmYgLS1naXQgYS9tbS9wYWdlLXdy aXRlYmFjay5jIGIvbW0vcGFnZS13cml0ZWJhY2suYwppbmRleCAwOGQyYjk2Li4xMTQwMGVkIDEw MDY0NAotLS0gYS9tbS9wYWdlLXdyaXRlYmFjay5jCisrKyBiL21tL3BhZ2Utd3JpdGViYWNrLmMK QEAgLTk5Nyw3ICs5OTcsNyBAQCBjb250aW51ZV91bmxvY2s6CiAJCXBhZ2V2ZWNfcmVsZWFzZSgm cHZlYyk7CiAJCWNvbmRfcmVzY2hlZCgpOwogCX0KLQlpZiAoIWN5Y2xlZCkgeworCWlmICghY3lj bGVkICYmICFkb25lKSB7CiAJCS8qCiAJCSAqIHJhbmdlX2N5Y2xpYzoKIAkJICogV2UgaGl0IHRo ZSBsYXN0IHBhZ2UgYW5kIHRoZXJlIGlzIG1vcmUgd29yayB0byBiZSBkb25lOiB3cmFwCg== --0016364272288d92dd047632b259 Content-Type: application/octet-stream; name="66c85494570396661479ba51e17964b2c82b6f39.patch" Content-Disposition: attachment; filename="66c85494570396661479ba51e17964b2c82b6f39.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_g0xmjzlo1 Y29tbWl0IDY2Yzg1NDk0NTcwMzk2NjYxNDc5YmE1MWUxNzk2NGIyYzgyYjZmMzkKQXV0aG9yOiBB cnRlbSBCaXR5dXRza2l5IDxBcnRlbS5CaXR5dXRza2l5QG5va2lhLmNvbT4KRGF0ZTogICBNb24g RmViIDIgMTg6MzM6NDkgMjAwOSArMDIwMAoKICAgIHdyaXRlLWJhY2s6IGZpeCBucl90b193cml0 ZSBjb3VudGVyCiAgICAKICAgIGNvbW1pdCBkY2Y2YTc5ZGRhNWNjMmEyYmVjMTgzZTUwZDgyOTAz MGMwOTcyYWFhIHVwc3RyZWFtLgogICAgCiAgICBDb21taXQgMDVmZTQ3OGRkMDRlMDJmYTIzMGMz MDVhYjliNTYxNjY2OTgyMWRkMyBpbnRyb2R1Y2VkIHNvbWUKICAgIEB3YmMtPm5yX3RvX3dyaXRl IGJyZWFrYWdlLgogICAgCiAgICBJdCBtYWRlIHRoZSBmb2xsb3dpbmcgY2hhbmdlczoKICAgICAx LiBEZWNyZW1lbnQgd2JjLT5ucl90b193cml0ZSBpbnN0ZWFkIG9mIG5yX3RvX3dyaXRlCiAgICAg Mi4gRGVjcmVtZW50IHdiYy0+bnJfdG9fd3JpdGUgX29ubHlfIGlmIHdiYy0+c3luY19tb2RlID09 IFdCX1NZTkNfTk9ORQogICAgIDMuIElmIHN5bmNlZCBucl90b193cml0ZSBwYWdlcywgc3RvcCBv bmx5IGlmIGlmIHdiYy0+c3luY19tb2RlID09CiAgICAgICAgV0JfU1lOQ19OT05FLCBvdGhlcndp c2Uga2VlcCBnb2luZy4KICAgIAogICAgSG93ZXZlciwgYWNjb3JkaW5nIHRvIHRoZSBjb21taXQg bWVzc2FnZSwgdGhlIGludGVudGlvbiB3YXMgdG8gb25seSBtYWtlCiAgICBjaGFuZ2UgMy4gIENo YW5nZSAxIGlzIGEgYnVnLiAgQ2hhbmdlIDIgZG9lcyBub3Qgc2VlbSB0byBiZSBuZWNlc3Nhcnks CiAgICBhbmQgaXQgYnJlYWtzIFVCSUZTIGV4cGVjdGF0aW9ucywgc28gaWYgbmVlZGVkLCBpdCBz aG91bGQgYmUgZG9uZQogICAgc2VwYXJhdGVseSBsYXRlci4gIEFuZCBjaGFuZ2UgMiBkb2VzIG5v dCBzZWVtIHRvIGJlIGRvY3VtZW50ZWQgaW4gdGhlCiAgICBjb21taXQgbWVzc2FnZS4KICAgIAog ICAgVGhpcyBwYXRjaCBkb2VzIHRoZSBmb2xsb3dpbmc6CiAgICAgMS4gVW5kbyBjaGFuZ2VzIDEg YW5kIDIKICAgICAyLiBBZGQgYSBjb21tZW50IGV4cGxhaW5pbmcgY2hhbmdlIDMgKGl0IHZlcnkg dXNlZnVsIHRvIGhhdmUgY29tbWVudHMKICAgICAgICBpbiBfY29kZV8sIG5vdCBvbmx5IGluIHRo ZSBjb21taXQpLgogICAgCiAgICBTaWduZWQtb2ZmLWJ5OiBBcnRlbSBCaXR5dXRza2l5IDxBcnRl bS5CaXR5dXRza2l5QG5va2lhLmNvbT4KICAgIEFja2VkLWJ5OiBOaWNrIFBpZ2dpbiA8bnBpZ2dp bkBzdXNlLmRlPgogICAgQ2M6IEFuZHJldyBNb3J0b24gPGFrcG1AbGludXgtZm91bmRhdGlvbi5v cmc+CiAgICBTaWduZWQtb2ZmLWJ5OiBMaW51cyBUb3J2YWxkcyA8dG9ydmFsZHNAbGludXgtZm91 bmRhdGlvbi5vcmc+CiAgICBTaWduZWQtb2ZmLWJ5OiBHcmVnIEtyb2FoLUhhcnRtYW4gPGdyZWdr aEBzdXNlLmRlPgoKZGlmZiAtLWdpdCBhL21tL3BhZ2Utd3JpdGViYWNrLmMgYi9tbS9wYWdlLXdy aXRlYmFjay5jCmluZGV4IDExNDAwZWQuLjBjNDEwMGUgMTAwNjQ0Ci0tLSBhL21tL3BhZ2Utd3Jp dGViYWNrLmMKKysrIGIvbW0vcGFnZS13cml0ZWJhY2suYwpAQCAtOTgxLDEzICs5ODEsMjIgQEAg Y29udGludWVfdW5sb2NrOgogCQkJCX0KICAJCQl9CiAKLQkJCWlmICh3YmMtPnN5bmNfbW9kZSA9 PSBXQl9TWU5DX05PTkUpIHsKLQkJCQl3YmMtPm5yX3RvX3dyaXRlLS07Ci0JCQkJaWYgKHdiYy0+ bnJfdG9fd3JpdGUgPD0gMCkgewotCQkJCQlkb25lID0gMTsKLQkJCQkJYnJlYWs7Ci0JCQkJfQor CQkJaWYgKG5yX3RvX3dyaXRlID4gMCkKKwkJCQlucl90b193cml0ZS0tOworCQkJZWxzZSBpZiAo d2JjLT5zeW5jX21vZGUgPT0gV0JfU1lOQ19OT05FKSB7CisJCQkJLyoKKwkJCQkgKiBXZSBzdG9w IHdyaXRpbmcgYmFjayBvbmx5IGlmIHdlIGFyZSBub3QKKwkJCQkgKiBkb2luZyBpbnRlZ3JpdHkg c3luYy4gSW4gY2FzZSBvZiBpbnRlZ3JpdHkKKwkJCQkgKiBzeW5jIHdlIGhhdmUgdG8ga2VlcCBn b2luZyBiZWNhdXNlIHNvbWVvbmUKKwkJCQkgKiBtYXkgYmUgY29uY3VycmVudGx5IGRpcnR5aW5n IHBhZ2VzLCBhbmQgd2UKKwkJCQkgKiBtaWdodCBoYXZlIHN5bmNlZCBhIGxvdCBvZiBuZXdseSBh cHBlYXJlZAorCQkJCSAqIGRpcnR5IHBhZ2VzLCBidXQgaGF2ZSBub3Qgc3luY2VkIGFsbCBvZiB0 aGUKKwkJCQkgKiBvbGQgZGlydHkgcGFnZXMuCisJCQkJICovCisJCQkJZG9uZSA9IDE7CisJCQkJ YnJlYWs7CiAJCQl9CisKIAkJCWlmICh3YmMtPm5vbmJsb2NraW5nICYmIGJkaV93cml0ZV9jb25n ZXN0ZWQoYmRpKSkgewogCQkJCXdiYy0+ZW5jb3VudGVyZWRfY29uZ2VzdGlvbiA9IDE7CiAJCQkJ ZG9uZSA9IDE7Cg== --0016364272288d92dd047632b259--