Re: md raid performance with 3-18-rc3

[NeilBrown <neilb@suse.de>]

[-- Attachment #1: Type: text/plain, Size: 4767 bytes --]

On Wed, 26 Nov 2014 13:41:39 +0530 Manish Awasthi
<manish.awasthi@caviumnetworks.com> wrote:

> 
> On 11/25/2014 08:07 AM, NeilBrown wrote:
> > On Mon, 24 Nov 2014 13:40:06 +0530 Manish Awasthi
> > <manish.awasthi@caviumnetworks.com> wrote:
> >
> >> Hi,
> >>
> >> We benchmarked the md raid driver performance on 3-18-rc3 kernel and
> >> compared the results with that of 3.6.11. The reason for this exercise
> >> is to understand if multithreaded raid driver has any performance
> >> benefits over 3.6.11 which is single threaded. Here are some details
> >> about the setup
> > Thanks for doing this!!!! I love it when people report test results.
> >
> >
> >> System: Intel(R) Xeon(R) CPU E3-1230 V2 @ 3.30GHz 4 cores (8threads),
> >> 8GB RAM.
> >> Setup: 3 SSDs create a raid5 array
> >> test tool: iozone (only read/re-read, write/re-write tested), blocksize:
> >> 4k-64k, filesize: 1Gig to 200Gig
> >>
> >> Comparison was done for speed of data transfer in kBytes/sec and also
> >> the CPU utilization as reported by iozone.
> >>
> >> raid on 3.18.0-rc3 performed much worse than raid on 3.6.11.
> >>
> >> Read/Write: raid on 3.18.0-rc3 operated at almost half the speed of raid
> >> on 3.6.11
> > That really isn't very good.... Can you try some of the kernels in between
> > and see if there was a single point where performance dropped, or if there
> > were several steps?
> Can you give me some starting point when multithread support for raid 
> was added. It might be a good starting point. For instance, I have a 
> benchmark on 3.6.11, now I'd like to go to the first kernel that has 
> support for multithread raid and then take it from there.

Multi-thread support appeared in 3.12.


> >
> >
> >> CPU Utilization: With md raid on 3.18.0-rc3, the CPU utilization was
> >> less than half of md raid on 3.6.11 on WRITE operations. However, for
> >> READ operations, 3.18-0.rc3 had more CPU utilization than 3.6.11.
> > Can you use "perf" to determine where the extra time is going?
> >
> >    perf record
> >    run test
> >    stop perf
> >    perf report
> >
> > or something like that.
> I can do this but as I mentioned below, Its better if I can get to 
> understand all the possible tweaks that can be done to get the optimal 
> results unless ofcourse you expect 3.18.0 to perform better than that of 
> 3.6.11 even if default case without any tweaks.

I have no particular expectations.  I like to see concrete measurements and
then try to interpret them.

I prefer to compare default setting (no tweaks) in the first instance.
Because that is what most people will be using.


> >
> >> Also, I noticed that scaling up the CPU cores of the system scales down
> >> the raid througput with 3.18.0-rc3.
> > This is by writing numbers to "group_thread_cnt" ??? Can you provide a simple
> > table comparing thread count to throughput?  Or maybe a graph.  I love
> > graphs :-)
> I did not tweak anything on the 3.18.0 kernel. I assumed all the 
> required support is built-in and did not bother to go into the depth of 
> the code as we're still in nascent stages where we are comparing the 
> data on specific kernel versions. Can you point me to some text that can 
> describe tweaks like "group_thread_cnt" etc?

Multi-threading is disabled by default, so if you haven't explicitly enabled
it, then it cannot be affecting your performance.

If you
  echo 8 > /sys/block/mdXXX/md/group_thread_cnt

it will use 8 thread to perform 'xor' calculations and submit IO requests.


> >
> >> I do have detailed logs of the comparison but I'm not sure I should send
> >> those on this mailing list.
> > A few megabytes?  Yes.  100Meg?  No.
> 
> Whatever data I have on comparison is attached, I have consolidated this 
> from log files to excel. See if this helps.

Thanks.  I'll have a look at the tables and see if anything looks interesting.

Thanks,
NeilBrown



> >
> > If you could put them on a website somewhere that I can browse or download
> > I'll try to have a look.
> >
> >> If my observation aligns with someone else's, then what is really the
> >> gain with multithreaded raid.
> > Some testing shows real improvements.  Obviously we cannot test everything
> > and I'm very glad to have extra testing from other people.
> > If we can quantify the regressions and confirm exactly when they occurred, we
> > can start looking for a solution.
> >
> > Thanks a lot!
> >
> > NeilBrown
> >
> >
> >> Manish
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 811 bytes --]