From mboxrd@z Thu Jan 1 00:00:00 1970 From: Manish Awasthi Subject: Re: md raid performance with 3-18-rc3 Date: Tue, 9 Dec 2014 13:56:00 +0530 Message-ID: <5486B218.8070204@caviumnetworks.com> References: <5472E7DE.5070702@caviumnetworks.com> <20141125133742.0154a4d4@notabene.brown> <54758B3B.5080907@caviumnetworks.com> <20141203172101.17859aee@notabene.brown> <5486B15C.8060109@caviumnetworks.com> <5486B1C2.1000508@caviumnetworks.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="------------040007090406030909010904" Return-path: In-Reply-To: <5486B1C2.1000508@caviumnetworks.com> Sender: linux-raid-owner@vger.kernel.org To: NeilBrown Cc: linux-raid@vger.kernel.org List-Id: linux-raid.ids --------------040007090406030909010904 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 7bit this time with attachment: manish On 12/09/2014 01:54 PM, Manish Awasthi wrote: > resending: > > dirty_ratio same for both the kernels. >> >> vm.dirty_background_bytes = 0 >> vm.dirty_background_ratio = 10 >> vm.dirty_bytes = 0 >> vm.dirty_expire_centisecs = 3000 >> vm.dirty_ratio = 20 >> vm.dirty_writeback_centisecs = 500 >> >> >> I re-ran the tests with the same set of kernel without enabling >> multithread support on 3.18 and measured a few things with perf. >> >> perf-stat-.txt: test ran for some time and measured various >> parameters. >> >> Meanwhile I'm also running complete test under perf record. I'll >> share the results soon. >> >> Manish >> >> On 12/03/2014 11:51 AM, NeilBrown wrote: >>> On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi >>> wrote: >>> >>>> Whatever data I have on comparison is attached, I have consolidated >>>> this >>>> from log files to excel. See if this helps. >>> raid_3_18_performance.xls shows read throughput to be consistently >>> 20% down >>> on 3.18 compared to 3.6.11. >>> >>> Writes are a few percent better for 4G/8G files, 20% better for >>> 16G/32G files. >>> unchanged above that. >>> Given that you have 8G of RAM, that seems like it could be some >>> change in >>> caching behaviour, and not necessarily a change in RAID behaviour. >>> >>> The CPU utilization roughly follows the throughput: 40% higher when >>> write >>> throughput is 20% better. >>> Could you check if the value of /proc/sys/vm/dirty_ratio is the same >>> for both >>> tests. That number has changed occasionally and could affect these >>> tests. >>> >>> >>> The second file, 3SSDs-perf-2-Cores-3.18-rc1 has the "change" numbers >>> negative where I expected positive.. i.e. negative mean an increase. >>> >>> Writes consistently have higher CPU utilisation. >>> Reads consistently have much lower CPU utilization. >>> >>> I don't know what that means ... it might not mean anything. >>> >>> Could you please run the tests between the two kernels *with* RAID. >>> i.e. >>> directly on an SSD. That will give us a baseline for what changes >>> are caused >>> by other parts of the kernel (filesystem, block layer, MM, etc). >>> Then we can >>> see how much change RAID5 is contributing. >>> >>> The third file, 3SSDs-perf-4Core.xls seems to show significantly >>> reduced >>> throughput across the board. >>> CPU utilization is less (better) for writes, but worse for reads. >>> That is >>> the reverse of what the second file shows. >>> >>> I might try running some tests across a set of kernel versions and >>> see what I >>> can come up with. >>> >>> NeilBrown >> > --------------040007090406030909010904 Content-Type: text/plain; charset="UTF-8"; name="perf-stat-3.6.11.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="perf-stat-3.6.11.txt" cGVyZiBzdGF0IG9uIG1kMTI1X3JhaWQ1IC0tIGtlcm5lbCAzLjYuMTEKCiMgcGVyZiBzdGF0 IC1wIDI2MTMgLWUgY3ljbGVzLGluc3RydWN0aW9ucyxjYWNoZS1yZWZlcmVuY2VzLGNhY2hl LW1pc3NlcyxicmFuY2hlcyxicmFuY2gtbWlzc2VzLGJ1cy1jeWNsZXMsc3RhbGxlZC1jeWNs ZXMtZnJvbnRlbmQscmVmLWN5Y2xlcyxjcHUtY2xvY2ssdGFzay1jbG9jayxmYXVsdHMsY29u dGV4dC1zd2l0Y2hlcyxjcHUtbWlncmF0aW9ucyxtaW5vci1mYXVsdHMsbWFqb3ItZmF1bHRz LGFsaWdubWVudC1mYXVsdHMsZW11bGF0aW9uLWZhdWx0cyxMMS1kY2FjaGUtbG9hZC1taXNz ZXMsTDEtZGNhY2hlLXN0b3JlLW1pc3NlcyxMMS1kY2FjaGUtcHJlZmV0Y2gtbWlzc2VzLEwx LWljYWNoZS1sb2FkLW1pc3NlcyxMTEMtbG9hZHMsTExDLXN0b3JlcyxMTEMtcHJlZmV0Y2hl cyxkVExCLWxvYWQtbWlzc2VzLGRUTEItc3RvcmUtbWlzc2VzLGlUTEItbG9hZHMsaVRMQi1s b2FkLW1pc3NlcyxicmFuY2gtbG9hZHMsYnJhbmNoLWxvYWQtbWlzc2VzCl5DIAogUGVyZm9y bWFuY2UgY291bnRlciBzdGF0cyBmb3IgcHJvY2VzcyBpZCAnMjYxMyc6CgogICAxMDMsMjAw LDY3Nyw3MjEgICAgICBjeWNsZXMgICAgICAgICAgICAgICAgICAgICMgICAgMi44NDggR0h6 ICAgICAgICAgICAgICAgICAgICAgWzIyLjcyJV0KICAgIDY5LDY2OSw4MTMsOTgzICAgICAg aW5zdHJ1Y3Rpb25zICAgICAgICAgICAgICAjICAgIDAuNjggIGluc25zIHBlciBjeWNsZSAg ICAgICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgIyAgICAxLjA3ICBzdGFsbGVkIGN5Y2xlcyBwZXIgaW5zbiBbMjcuMjYlXQogICAgIDIs NjY4LDQ2NSw3NjkgICAgICBjYWNoZS1yZWZlcmVuY2VzICAgICAgICAgICMgICA3My42NDgg TS9zZWMgICAgICAgICAgICAgICAgICAgWzI3LjM1JV0KICAgICAxLDQwOCw0OTMsNjgwICAg ICAgY2FjaGUtbWlzc2VzICAgICAgICAgICAgICAjICAgNTIuNzgzICUgb2YgYWxsIGNhY2hl IHJlZnMgICAgIFsyNy4xNyVdCiAgICAxMyw2MDksMjExLDMyMSAgICAgIGJyYW5jaGVzICAg ICAgICAgICAgICAgICAgIyAgMzc1LjYwNyBNL3NlYyAgICAgICAgICAgICAgICAgICBbMjcu MTklXQogICAgICAgMTIxLDU5Myw1OTggICAgICBicmFuY2gtbWlzc2VzICAgICAgICAgICAg ICMgICAgMC44OSUgb2YgYWxsIGJyYW5jaGVzICAgICAgICAgWzI3LjMyJV0KICAgICAzLDQy MCw3MjUsMzU5ICAgICAgYnVzLWN5Y2xlcyAgICAgICAgICAgICAgICAjICAgOTQuNDEwIE0v c2VjICAgICAgICAgICAgICAgICAgIFsxOC4wNyVdCiAgICA3NCwzNjIsMzY4LDI1MiAgICAg IHN0YWxsZWQtY3ljbGVzLWZyb250ZW5kICAgIyAgIDcyLjA2JSBmcm9udGVuZCBjeWNsZXMg aWRsZSAgICBbMTguMTYlXQogICAxMTIsNTUzLDk0NSw2NTAgICAgICByZWYtY3ljbGVzICAg ICAgICAgICAgICAgICMgMzEwNi40MjcgTS9zZWMgICAgICAgICAgICAgICAgICAgWzIyLjc2 JV0KICAgICAgMzYyMzMuNzY2NDExICAgICAgY3B1LWNsb2NrIChtc2VjKSAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgIDM2MjMyLjYwNTQ5OSAg ICAgIHRhc2stY2xvY2sgKG1zZWMpICAgICAgICAgIyAgICAwLjE4MSBDUFVzIHV0aWxpemVk ICAgICAgICAgIAogICAgICAgICAgICAgICAgIDAgICAgICBmYXVsdHMgICAgICAgICAgICAg ICAgICAgICMgICAgMC4wMDAgSy9zZWMgICAgICAgICAgICAgICAgICAKICAgICAgICAgICA0 NDIsODg1ICAgICAgY29udGV4dC1zd2l0Y2hlcyAgICAgICAgICAjICAgIDAuMDEyIE0vc2Vj ICAgICAgICAgICAgICAgICAgCiAgICAgICAgICAgICA5LDY0NiAgICAgIGNwdS1taWdyYXRp b25zICAgICAgICAgICAgIyAgICAwLjI2NiBLL3NlYyAgICAgICAgICAgICAgICAgIAogICAg ICAgICAgICAgICAgIDAgICAgICBtaW5vci1mYXVsdHMgICAgICAgICAgICAgICMgICAgMC4w MDAgSy9zZWMgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAwICAgICAgbWFq b3ItZmF1bHRzICAgICAgICAgICAgICAjICAgIDAuMDAwIEsvc2VjICAgICAgICAgICAgICAg ICAgCiAgICAgICAgICAgICAgICAgMCAgICAgIGFsaWdubWVudC1mYXVsdHMgICAgICAgICAg IyAgICAwLjAwMCBLL3NlYyAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgIDAg ICAgICBlbXVsYXRpb24tZmF1bHRzICAgICAgICAgICMgICAgMC4wMDAgSy9zZWMgICAgICAg ICAgICAgICAgICAKICAgICAzLDE4OCw4NjUsOTM2ICAgICAgTDEtZGNhY2hlLWxvYWQtbWlz c2VzICAgICAjICAgODguMDExIE0vc2VjICAgICAgICAgICAgICAgICAgIFsyMi45NiVdCiAg ICAgMSw2NTgsODMxLDk1NyAgICAgIEwxLWRjYWNoZS1zdG9yZS1taXNzZXMgICAgIyAgIDQ1 Ljc4MyBNL3NlYyAgICAgICAgICAgICAgICAgICBbMjIuODklXQogICAgICAgMzM4LDc0NCww MjkgICAgICBMMS1kY2FjaGUtcHJlZmV0Y2gtbWlzc2VzICMgICAgOS4zNDkgTS9zZWMgICAg ICAgICAgICAgICAgICAgWzIzLjA0JV0KICAgICAgIDQ0NSwwNjYsOTk1ICAgICAgTDEtaWNh Y2hlLWxvYWQtbWlzc2VzICAgICAjICAgMTIuMjg0IE0vc2VjICAgICAgICAgICAgICAgICAg IFsyMi45OSVdCiAgICAgMSw1NzgsMDY3LDIyNSAgICAgIExMQy1sb2FkcyAgICAgICAgICAg ICAgICAgIyAgIDQzLjU1NCBNL3NlYyAgICAgICAgICAgICAgICAgICBbMTguMTklXQogICAg IDEsMzE3LDgyMiw5OTkgICAgICBMTEMtc3RvcmVzICAgICAgICAgICAgICAgICMgICAzNi4z NzEgTS9zZWMgICAgICAgICAgICAgICAgICAgWzE4LjIzJV0KICAgICAgIDc5OCwwMDQsNjEw ICAgICAgTExDLXByZWZldGNoZXMgICAgICAgICAgICAjICAgMjIuMDI0IE0vc2VjICAgICAg ICAgICAgICAgICAgIFsgOS4wOSVdCiAgICAgICAgICAgICAgICAgMCAgICAgIGRUTEItbG9h ZC1taXNzZXMgICAgICAgICAgIyAgICAwLjAwMCBLL3NlYyAgICAgICAgICAgICAgICAgICBb MTMuNTIlXQogICAgICAgICA3LDYzMywyMzYgICAgICBkVExCLXN0b3JlLW1pc3NlcyAgICAg ICAgICMgICAgMC4yMTEgTS9zZWMgICAgICAgICAgICAgICAgICAgWzE4LjAzJV0KICAgICAg ICAxMCwwMjQsNDY0ICAgICAgaVRMQi1sb2FkcyAgICAgICAgICAgICAgICAjICAgIDAuMjc3 IE0vc2VjICAgICAgICAgICAgICAgICAgIFsxNy45MiVdCiAgICAgICAgIDMsMTU3LDE0MSAg ICAgIGlUTEItbG9hZC1taXNzZXMgICAgICAgICAgIyAgIDMxLjQ5JSBvZiBhbGwgaVRMQiBj YWNoZSBoaXRzICBbMTguMTIlXQogICAgMTMsNjE2LDg1Nyw2NDUgICAgICBicmFuY2gtbG9h ZHMgICAgICAgICAgICAgICMgIDM3NS44MTggTS9zZWMgICAgICAgICAgICAgICAgICAgWzE4 LjE2JV0KICAgICAgIDExOSwyNTAsNDUwICAgICAgYnJhbmNoLWxvYWQtbWlzc2VzICAgICAg ICAjICAgIDMuMjkxIE0vc2VjICAgICAgICAgICAgICAgICAgIFsxOC4xNCVdCgogICAgIDIw MC4xOTAxODE2MjMgc2Vjb25kcyB0aW1lIGVsYXBzZWQKCgoK --------------040007090406030909010904 Content-Type: text/plain; charset="UTF-8"; name="perf-stat-3.18.txt" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="perf-stat-3.18.txt" cGVyZiBzdGF0IG9uIG1kMTI1X3JhaWQ1IC0tIGtlcm5lbCAzLjE4CgojIHBlcmYgc3RhdCAt cCAyNzc4IC1lIGN5Y2xlcyxpbnN0cnVjdGlvbnMsY2FjaGUtcmVmZXJlbmNlcyxjYWNoZS1t aXNzZXMsYnJhbmNoZXMsYnJhbmNoLW1pc3NlcyxidXMtY3ljbGVzLHN0YWxsZWQtY3ljbGVz LWZyb250ZW5kLHJlZi1jeWNsZXMsY3B1LWNsb2NrLHRhc2stY2xvY2ssZmF1bHRzLGNvbnRl eHQtc3dpdGNoZXMsY3B1LW1pZ3JhdGlvbnMsbWlub3ItZmF1bHRzLG1ham9yLWZhdWx0cyxh bGlnbm1lbnQtZmF1bHRzLGVtdWxhdGlvbi1mYXVsdHMsTDEtZGNhY2hlLWxvYWQtbWlzc2Vz LEwxLWRjYWNoZS1zdG9yZS1taXNzZXMsTDEtZGNhY2hlLXByZWZldGNoLW1pc3NlcyxMMS1p Y2FjaGUtbG9hZC1taXNzZXMsTExDLWxvYWRzLExMQy1zdG9yZXMsTExDLXByZWZldGNoZXMs ZFRMQi1sb2FkLW1pc3NlcyxkVExCLXN0b3JlLW1pc3NlcyxpVExCLWxvYWRzLGlUTEItbG9h ZC1taXNzZXMsYnJhbmNoLWxvYWRzLGJyYW5jaC1sb2FkLW1pc3NlcwpeQwogUGVyZm9ybWFu Y2UgY291bnRlciBzdGF0cyBmb3IgcHJvY2VzcyBpZCAnMjc3OCc6CgogICAxOTEsMjEyLDc3 OCw5ODEgICAgICBjeWNsZXMgICAgICAgICAgICAgICAgICAgICMgICAgMi45NDIgR0h6ICAg ICAgICAgICAgICAgICAgICAgWzIyLjk5JV0KICAgMTYwLDMxOCw2MjgsMzY3ICAgICAgaW5z dHJ1Y3Rpb25zICAgICAgICAgICAgICAjICAgIDAuODQgIGluc25zIHBlciBjeWNsZSAgICAg ICAgCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg IyAgICAwLjc3ICBzdGFsbGVkIGN5Y2xlcyBwZXIgaW5zbiBbMjcuNDklXQogICAgIDMsODAw LDY4OCw2OTUgICAgICBjYWNoZS1yZWZlcmVuY2VzICAgICAgICAgICMgICA1OC40ODUgTS9z ZWMgICAgICAgICAgICAgICAgICAgWzI3LjQwJV0KICAgICAxLDQxOCw0MzEsNjkzICAgICAg Y2FjaGUtbWlzc2VzICAgICAgICAgICAgICAjICAgMzcuMzIwICUgb2YgYWxsIGNhY2hlIHJl ZnMgICAgIFsyNy4yNyVdCiAgICAzMyw2MzUsNTUyLDk1MSAgICAgIGJyYW5jaGVzICAgICAg ICAgICAgICAgICAgIyAgNTE3LjU4NiBNL3NlYyAgICAgICAgICAgICAgICAgICBbMjcuMTIl XQogICAgICAgMzUyLDI2NCw1MTYgICAgICBicmFuY2gtbWlzc2VzICAgICAgICAgICAgICMg ICAgMS4wNSUgb2YgYWxsIGJyYW5jaGVzICAgICAgICAgWzI3LjE5JV0KICAgICA2LDAzNSw4 MDYsODY3ICAgICAgYnVzLWN5Y2xlcyAgICAgICAgICAgICAgICAjICAgOTIuODc5IE0vc2Vj ICAgICAgICAgICAgICAgICAgIFsxOC4yMSVdCiAgIDEyMiw5ODAsNDAxLDI4NSAgICAgIHN0 YWxsZWQtY3ljbGVzLWZyb250ZW5kICAgIyAgIDY0LjMyJSBmcm9udGVuZCBjeWNsZXMgaWRs ZSAgICBbMTguMTYlXQogICAxOTcsODI5LDYxOCwzMTIgICAgICByZWYtY3ljbGVzICAgICAg ICAgICAgICAgICMgMzA0NC4yMTYgTS9zZWMgICAgICAgICAgICAgICAgICAgWzIyLjcyJV0K ICAgICAgNjUwMzkuNzM4MjY3ICAgICAgY3B1LWNsb2NrIChtc2VjKSAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgCiAgICAgIDY0OTg1LjQxNTU2OCAgICAg IHRhc2stY2xvY2sgKG1zZWMpICAgICAgICAgIyAgICAwLjE4NiBDUFVzIHV0aWxpemVkICAg ICAgICAgIAogICAgICAgICAgICAgICAgIDAgICAgICBmYXVsdHMgICAgICAgICAgICAgICAg ICAgICMgICAgMC4wMDAgSy9zZWMgICAgICAgICAgICAgICAgICAKICAgICAgICAgMyw0Mzcs OTQ1ICAgICAgY29udGV4dC1zd2l0Y2hlcyAgICAgICAgICAjICAgIDAuMDUzIE0vc2VjICAg ICAgICAgICAgICAgICAgCiAgICAgICAgICAgICAgIDIzNyAgICAgIGNwdS1taWdyYXRpb25z ICAgICAgICAgICAgIyAgICAwLjAwNCBLL3NlYyAgICAgICAgICAgICAgICAgIAogICAgICAg ICAgICAgICAgIDAgICAgICBtaW5vci1mYXVsdHMgICAgICAgICAgICAgICMgICAgMC4wMDAg Sy9zZWMgICAgICAgICAgICAgICAgICAKICAgICAgICAgICAgICAgICAwICAgICAgbWFqb3It ZmF1bHRzICAgICAgICAgICAgICAjICAgIDAuMDAwIEsvc2VjICAgICAgICAgICAgICAgICAg CiAgICAgICAgICAgICAgICAgMCAgICAgIGFsaWdubWVudC1mYXVsdHMgICAgICAgICAgIyAg ICAwLjAwMCBLL3NlYyAgICAgICAgICAgICAgICAgIAogICAgICAgICAgICAgICAgIDAgICAg ICBlbXVsYXRpb24tZmF1bHRzICAgICAgICAgICMgICAgMC4wMDAgSy9zZWMgICAgICAgICAg ICAgICAgICAKICAgICA1LDMyOSw3MTEsOTM5ICAgICAgTDEtZGNhY2hlLWxvYWQtbWlzc2Vz ICAgICAjICAgODIuMDE0IE0vc2VjICAgICAgICAgICAgICAgICAgIFsyMi44MyVdCiAgICAg MiwxMzgsNDAwLDEwNyAgICAgIEwxLWRjYWNoZS1zdG9yZS1taXNzZXMgICAgIyAgIDMyLjkw NiBNL3NlYyAgICAgICAgICAgICAgICAgICBbMjIuNTIlXQogICAgICAgNjY3LDY0Niw5Njgg ICAgICBMMS1kY2FjaGUtcHJlZmV0Y2gtbWlzc2VzICMgICAxMC4yNzQgTS9zZWMgICAgICAg ICAgICAgICAgICAgWzIyLjQ4JV0KICAgICAyLDI1OSw0MjUsODMwICAgICAgTDEtaWNhY2hl LWxvYWQtbWlzc2VzICAgICAjICAgMzQuNzY4IE0vc2VjICAgICAgICAgICAgICAgICAgIFsy Mi40NSVdCiAgICAgMiwwOTAsNTk2LDc3NyAgICAgIExMQy1sb2FkcyAgICAgICAgICAgICAg ICAgIyAgIDMyLjE3MCBNL3NlYyAgICAgICAgICAgICAgICAgICBbMTcuOTMlXQogICAgIDEs Njc5LDI4NywyNzEgICAgICBMTEMtc3RvcmVzICAgICAgICAgICAgICAgICMgICAyNS44NDEg TS9zZWMgICAgICAgICAgICAgICAgICAgWzE4LjA0JV0KICAgICAxLDEyMCwwODYsMTQ3ICAg ICAgTExDLXByZWZldGNoZXMgICAgICAgICAgICAjICAgMTcuMjM2IE0vc2VjICAgICAgICAg ICAgICAgICAgIFsgOS4wOSVdCiAgICAgICA0NjUsMTQyLDYyMiAgICAgIGRUTEItbG9hZC1t aXNzZXMgICAgICAgICAgIyAgICA3LjE1OCBNL3NlYyAgICAgICAgICAgICAgICAgICBbMTMu NjklXQogICAgICAgIDI2LDY3MiwyOTggICAgICBkVExCLXN0b3JlLW1pc3NlcyAgICAgICAg ICMgICAgMC40MTAgTS9zZWMgICAgICAgICAgICAgICAgICAgWzE4LjI2JV0KICAgICAgICA2 Niw3MjMsNDc1ICAgICAgaVRMQi1sb2FkcyAgICAgICAgICAgICAgICAjICAgIDEuMDI3IE0v c2VjICAgICAgICAgICAgICAgICAgIFsxOC4zNyVdCiAgICAgICAgIDksNzM2LDcyOSAgICAg IGlUTEItbG9hZC1taXNzZXMgICAgICAgICAgIyAgIDE0LjU5JSBvZiBhbGwgaVRMQiBjYWNo ZSBoaXRzICBbMTguNDMlXQogICAgMzMsMjM4LDA4Miw2NjQgICAgICBicmFuY2gtbG9hZHMg ICAgICAgICAgICAgICMgIDUxMS40NzAgTS9zZWMgICAgICAgICAgICAgICAgICAgWzE4LjQ0 JV0KICAgICAgIDM0NiwwMjUsOTkzICAgICAgYnJhbmNoLWxvYWQtbWlzc2VzICAgICAgICAj ICAgIDUuMzI1IE0vc2VjICAgICAgICAgICAgICAgICAgIFsxOC40NiVdCgogICAgIDM0OC45 NDY4NTM5NTggc2Vjb25kcyB0aW1lIGVsYXBzZWQKCgoK --------------040007090406030909010904--