Re: md raid performance with 3-18-rc3

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 2737 bytes --]

On Mon, 24 Nov 2014 13:40:06 +0530 Manish Awasthi
<manish.awasthi@caviumnetworks.com> wrote:

> Hi,
> 
> We benchmarked the md raid driver performance on 3-18-rc3 kernel and 
> compared the results with that of 3.6.11. The reason for this exercise 
> is to understand if multithreaded raid driver has any performance 
> benefits over 3.6.11 which is single threaded. Here are some details 
> about the setup

Thanks for doing this!!!! I love it when people report test results.


> 
> System: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz 4 cores (8threads), 
> 8GB RAM.
> Setup: 3 SSDs create a raid5 array
> test tool: iozone (only read/re-read, write/re-write tested), blocksize: 
> 4k-64k, filesize: 1Gig to 200Gig
> 
> Comparison was done for speed of data transfer in kBytes/sec and also 
> the CPU utilization as reported by iozone.
> 
> raid on 3.18.0-rc3 performed much worse than raid on 3.6.11.
> 
> Read/Write: raid on 3.18.0-rc3 operated at almost half the speed of raid 
> on 3.6.11

That really isn't very good.... Can you try some of the kernels in between
and see if there was a single point where performance dropped, or if there
were several steps?


> 
> CPU Utilization: With md raid on 3.18.0-rc3, the CPU utilization was 
> less than half of md raid on 3.6.11 on WRITE operations. However, for 
> READ operations, 3.18-0.rc3 had more CPU utilization than 3.6.11.

Can you use "perf" to determine where the extra time is going?

  perf record
  run test
  stop perf
  perf report

or something like that.

> 
> Also, I noticed that scaling up the CPU cores of the system scales down 
> the raid througput with 3.18.0-rc3.

This is by writing numbers to "group_thread_cnt" ??? Can you provide a simple
table comparing thread count to throughput?  Or maybe a graph.  I love
graphs :-)


> 
> I do have detailed logs of the comparison but I'm not sure I should send 
> those on this mailing list.

A few megabytes?  Yes.  100Meg?  No.

If you could put them on a website somewhere that I can browse or download
I'll try to have a look.

> 
> If my observation aligns with someone else's, then what is really the 
> gain with multithreaded raid.

Some testing shows real improvements.  Obviously we cannot test everything
and I'm very glad to have extra testing from other people.
If we can quantify the regressions and confirm exactly when they occurred, we
can start looking for a solution.

Thanks a lot!

NeilBrown


> 
> Manish
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]