Ceph Write Path Improvement

Ceph Write Path Improvement

[Somnath Roy]

Hi,
Here is the link of the document I presented in today's performance meeting.
https://docs.google.com/presentation/d/1lCoLpFRjD8t_YCeHyWDV7ddv7ZkwfETgyjUzXw0-ttU/edit?usp=sharing

It has the benchmark result of the filestore changes I proposed earlier for the ceph write path optimization.

Thanks & Regards
Somnath

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

Re: Ceph Write Path Improvement

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Somnath,

I'm having a hard time with your slide deck. Am I understanding
correctly that the default Hammer install was performed on SSDs with
co-located journals, but the optimized code was performed on the same
SSDs but the journal was in NVRAM? If so I'm having a hard time
understanding how these tests can be comparable. I really like the
performance gains you are seeing, but I'm trying to understand how
much the optimized code alone helps performance.

Thanks,
-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.0.2
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJV6HPoCRDmVDuy+mK58QAAJXQQAKz4ncncNQOFxRhKaAS4
gifdxT8PvzielfQ+f3mcgMpzDW2wdrThAe+ZOVP2wztv+6HvpJ0e2Vh/0y8r
zCp1wSZS52F2OeWNMXqzde2prBotpQqZGrZwd6q/TTBpT9Mg/qf6wEQvUqBb
w8k5IfeEQpvg4jxDiw4MzpETMIfnSPyyhoULV/vc9MxaXYgoch04xo6uD+VC
hZDuOs+JPT971X0U+67iyidGIv0pjz5tED93D2eyLe/PedZhh34+5A/b/KZ8
SFCEhmL0/5FXjKdOZk4glYKyVlWYNy1IXTUCbQ9+F1v1YAHtOLMw+IWMljXw
mAGudzt8sGGzX9XF0n7UMPpNJb5LKCutDNq3Jinkin5lFe32HILi9T2s8DQL
aagAyJFP9RVdnDlnPAanq7p3F5MBc8TcyKLISRp0cNJVRGnVjv6RJbo6RaZ6
lWeLhPkXCVJXbIo9BJ67QdgZUDz1FZxNkARL4cDGI7oZwzD5e99LJcKoz6yZ
d/0BcgJyzQPTfeOft7AwFTrP1at1mwK/aFK+jClEPU3GdDSrMluFCOH4Qu+F
pvIy0PxdGlGDS4P4Xs+NiWIeCu12NjtO8dRg++BO0rPbladBlsF3ZD+G7R/q
pG/QTKhf41s+pbDal+HJoNKiK7+WZgHhy8IBXdR4a8SQeMHRa/MYJk6MFLKr
vK/6
=wcJB
-----END PGP SIGNATURE-----
----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, Sep 2, 2015 at 5:06 PM, Somnath Roy <Somnath.Roy@sandisk.com> wrote:
> Hi,
> Here is the link of the document I presented in today's performance meeting.
> https://docs.google.com/presentation/d/1lCoLpFRjD8t_YCeHyWDV7ddv7ZkwfETgyjUzXw0-ttU/edit?usp=sharing
>
> It has the benchmark result of the filestore changes I proposed earlier for the ceph write path optimization.
>
> Thanks & Regards
> Somnath
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph Write Path Improvement

[Mark Nelson]

On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Somnath,
>
> I'm having a hard time with your slide deck. Am I understanding
> correctly that the default Hammer install was performed on SSDs with
> co-located journals, but the optimized code was performed on the same
> SSDs but the journal was in NVRAM? If so I'm having a hard time
> understanding how these tests can be comparable. I really like the
> performance gains you are seeing, but I'm trying to understand how
> much the optimized code alone helps performance.

Hi Robert,

We talked about this a bit at the weekly performance meeting.  I think 
Somnath just hasn't gotten a chance to do those tests yet and is 
planning on doing them in the coming weeks.  I believe he started out 
with hammer on the SSDs and then tried to figure out how to tweak things 
to make the NVRAM configuration perform better.  Now he has to go back 
and retest the original configuration but with the new code.

Mark

Re: Ceph Write Path Improvement

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Mark,

Thanks for the update. The performance meetings are at a bad time for
me and it usually takes a while for them to become available to review
later so I'm usually behind on the freshest news. I'm looking forward
to the results.
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Thu, Sep 3, 2015 at 10:42 AM, Mark Nelson  wrote:
> On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Somnath,
>>
>> I'm having a hard time with your slide deck. Am I understanding
>> correctly that the default Hammer install was performed on SSDs with
>> co-located journals, but the optimized code was performed on the same
>> SSDs but the journal was in NVRAM? If so I'm having a hard time
>> understanding how these tests can be comparable. I really like the
>> performance gains you are seeing, but I'm trying to understand how
>> much the optimized code alone helps performance.
>
>
> Hi Robert,
>
> We talked about this a bit at the weekly performance meeting.  I think
> Somnath just hasn't gotten a chance to do those tests yet and is planning on
> doing them in the coming weeks.  I believe he started out with hammer on the
> SSDs and then tried to figure out how to tweak things to make the NVRAM
> configuration perform better.  Now he has to go back and retest the original
> configuration but with the new code.
>
> Mark

-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.0.2
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJV6IWbCRDmVDuy+mK58QAAC4MP/RmUaee7sDJcUvMDDySc
BexzhwpCQ6Ur/MUrjdWvf2YhPr4d0+gSrFDFpU8bwG056nz8F0QIRufmGY9p
b76xhcnTLvwIbIlInAfBXVWYp7uheAz+4znEuVYbmYiCau3FlWe4jnd+ohCg
hOuAy/fkcnjXcjv6TToFUr1KIe07xNZkvt4ALWcSeafTwvl3JgddUvpKgBYq
KdM2g4RL7KvYDIAmgf4/KKBtX2bcbLXkL48Tc6Ev4NJCcAZk5VlX3eBvQYgY
ZWnlRHDNeBlDRvwHzS5aaWdar19qXpSkMjqW4u5YiT+kL0k0jbcTVpu2W+qC
1dOGwMInFnAzT+vgrw8tFy8XEcZ2PWoNmY1wiYTLp+mgvDDeSoqmldmF7R9N
lM1woA/NRhpYY9RqHUC8JkUdcW5mZ3yjCMP7fj/5HxPZAe8QAsZrwg91xqew
5WrCnh0kLUGQltE/jSpqBnn1tLGpGkoE5Uowy5nODEWWQbOz9rECOiMdOrAY
3lZXz9QzetYPAa+kAJLQ6AG5fDmq0sIvc30kyVBkOa83aQPrdsSXpwGj1B7W
V1iRCb132zzzGFGGlQZZDKkErBKz5Txwki66ExKpiNI7lJ0YYIw1+hMrdLoy
SrgYIdZB/7jOnAaDkK68MTrvQKmDUtxLKkBqd8zFhpqMNAj7vlShWfic7fxl
Pj/3
=zJ9M
-----END PGP SIGNATURE-----

RE: Ceph Write Path Improvement

[Somnath Roy]

Yes, As Mark said I will collect all the data and hopefully I can present in the next performance meeting.
BTW, I have tested with Hammer code base + NvRAM journal initially, but that performance is very spiky with ~10% performance gain (at max). I thought there is no point of collecting more data with that config.
That's why I have introduced a new throttling scheme that should benefit in all the scenarios.

Thanks & Regards
Somnath

-----Original Message-----
From: Mark Nelson [mailto:mnelson@redhat.com]
Sent: Thursday, September 03, 2015 9:42 AM
To: Robert LeBlanc; Somnath Roy
Cc: ceph-devel
Subject: Re: Ceph Write Path Improvement

On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Somnath,
>
> I'm having a hard time with your slide deck. Am I understanding
> correctly that the default Hammer install was performed on SSDs with
> co-located journals, but the optimized code was performed on the same
> SSDs but the journal was in NVRAM? If so I'm having a hard time
> understanding how these tests can be comparable. I really like the
> performance gains you are seeing, but I'm trying to understand how
> much the optimized code alone helps performance.

Hi Robert,

We talked about this a bit at the weekly performance meeting.  I think Somnath just hasn't gotten a chance to do those tests yet and is planning on doing them in the coming weeks.  I believe he started out with hammer on the SSDs and then tried to figure out how to tweak things to make the NVRAM configuration perform better.  Now he has to go back and retest the original configuration but with the new code.

Mark

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

RE: Ceph Write Path Improvement

[Blinick, Stephen L]

Somnath -- thanks for publishing all the data, will be great to look at it offline.  I didn't find this info:  How many RBD volumes, and what size, did you use for your mixed tests?  Was it just one RBD w/ num_jobs=1 & 10?  Also how many client systems were necessary to drive the workload on the 4 storage nodes?      

I saw the same behavior quite a while back when playing with ramdisk journal... Not a lot of improvement.

Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 3, 2015 10:42 AM
To: Mark Nelson; Robert LeBlanc
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Yes, As Mark said I will collect all the data and hopefully I can present in the next performance meeting.
BTW, I have tested with Hammer code base + NvRAM journal initially, but that performance is very spiky with ~10% performance gain (at max). I thought there is no point of collecting more data with that config.
That's why I have introduced a new throttling scheme that should benefit in all the scenarios.

Thanks & Regards
Somnath

-----Original Message-----
From: Mark Nelson [mailto:mnelson@redhat.com]
Sent: Thursday, September 03, 2015 9:42 AM
To: Robert LeBlanc; Somnath Roy
Cc: ceph-devel
Subject: Re: Ceph Write Path Improvement

On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Somnath,
>
> I'm having a hard time with your slide deck. Am I understanding 
> correctly that the default Hammer install was performed on SSDs with 
> co-located journals, but the optimized code was performed on the same 
> SSDs but the journal was in NVRAM? If so I'm having a hard time 
> understanding how these tests can be comparable. I really like the 
> performance gains you are seeing, but I'm trying to understand how 
> much the optimized code alone helps performance.

Hi Robert,

We talked about this a bit at the weekly performance meeting.  I think Somnath just hasn't gotten a chance to do those tests yet and is planning on doing them in the coming weeks.  I believe he started out with hammer on the SSDs and then tried to figure out how to tweak things to make the NVRAM configuration perform better.  Now he has to go back and retest the original configuration but with the new code.

Mark

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

\x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m \x1e \x17^  b  ^n r   z \x1a  h    &  \x1e G   h \x03( 階 ݢj"  \x1a ^[m     z ޖ   f   h   ~ m 

RE: Ceph Write Path Improvement

[Somnath Roy]

Stephen,
It's a 1 RBD volume (preconditioned) of 2 TB size from one physical client box.
fio-rbd script I am running with 10 jobs and each with 64 QD.
For mixed workload it is with QD = 8 and num_job= 1 and 10.


Thanks & Regards
Somnath

-----Original Message-----
From: Blinick, Stephen L [mailto:stephen.l.blinick@intel.com] 
Sent: Thursday, September 03, 2015 1:02 PM
To: Somnath Roy
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Somnath -- thanks for publishing all the data, will be great to look at it offline.  I didn't find this info:  How many RBD volumes, and what size, did you use for your mixed tests?  Was it just one RBD w/ num_jobs=1 & 10?  Also how many client systems were necessary to drive the workload on the 4 storage nodes?      

I saw the same behavior quite a while back when playing with ramdisk journal... Not a lot of improvement.

Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 3, 2015 10:42 AM
To: Mark Nelson; Robert LeBlanc
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Yes, As Mark said I will collect all the data and hopefully I can present in the next performance meeting.
BTW, I have tested with Hammer code base + NvRAM journal initially, but that performance is very spiky with ~10% performance gain (at max). I thought there is no point of collecting more data with that config.
That's why I have introduced a new throttling scheme that should benefit in all the scenarios.

Thanks & Regards
Somnath

-----Original Message-----
From: Mark Nelson [mailto:mnelson@redhat.com]
Sent: Thursday, September 03, 2015 9:42 AM
To: Robert LeBlanc; Somnath Roy
Cc: ceph-devel
Subject: Re: Ceph Write Path Improvement

On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Somnath,
>
> I'm having a hard time with your slide deck. Am I understanding 
> correctly that the default Hammer install was performed on SSDs with 
> co-located journals, but the optimized code was performed on the same 
> SSDs but the journal was in NVRAM? If so I'm having a hard time 
> understanding how these tests can be comparable. I really like the 
> performance gains you are seeing, but I'm trying to understand how 
> much the optimized code alone helps performance.

Hi Robert,

We talked about this a bit at the weekly performance meeting.  I think Somnath just hasn't gotten a chance to do those tests yet and is planning on doing them in the coming weeks.  I believe he started out with hammer on the SSDs and then tried to figure out how to tweak things to make the NVRAM configuration perform better.  Now he has to go back and retest the original configuration but with the new code.

Mark

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

\x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m \x1e \x17^  b  ^n r   z \x1a  h    &  \x1e G   h \x03( 階 ݢj"  \x1a ^[m     z ޖ   f   h   ~ m 

RE: Ceph Write Path Improvement

[Somnath Roy]

Hi,
Here is the updated presentation we discussed in the performance meeting today with performance data incorporated for the scenario where both journal/data on the same SSD.

https://docs.google.com/presentation/d/15-Uqk0b4s1fVV1cG1G6Kba9xafcnIoLvfq8LUY7KBL0/edit#slide=id.p4


Here is the summary:
--------------------------

For 4K :
–	~2.5X IOPs improvement, ~2.68X latency improvement with NVRAM as journal
–	~2.2X IOPS improvement, ~2.3X latency improvement  without NVRAM as journal
–	More stable performance with NVRAM as journal
For 16K:
–	~2.3X IOPS improvement, ~2.47X latency improvement with NVRAM as journal
–	~1.9X IOPS improvement, ~2X latency improvement  without NVRAM as journal
–	More stable performance with NVRAM as journal
For 64K:
–	~2X BW improvement, ~2X latency improvement with NVRAM as journal
–	~1.5X BW improvement, ~1.5X latency improvement  without NVRAM as journal
–	More stable performance with NVRAM as journal
For 4M:
–	~1.5X BW improvement, ~1.6X latency improvement with NVRAM as journal
–	~1.1X BW improvement, ~1.2X latency improvement  without NVRAM as journal

For Mixed workload:
------------------------
QD = 8 :
	~1.6X IOPs improvement, ~1.6X latency improvement with NVRAM as journal
	~1.5X IOPs improvement, ~1.5X latency improvement without NVRAM as journal
	More stable performance with NVRAM as journal

QD = 80 :
	~2.7X IOPs improvement, ~2.8X latency improvement with NVRAM as journal
	~2.4X IOPs improvement, ~2.5X latency improvement without NVRAM as journal
	More stable performance with NVRAM as journal

Will try to send out a pull request by next week.

Thanks & Regards
Somnath

-----Original Message-----
From: Somnath Roy 
Sent: Thursday, September 03, 2015 1:06 PM
To: 'Blinick, Stephen L'
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Stephen,
It's a 1 RBD volume (preconditioned) of 2 TB size from one physical client box.
fio-rbd script I am running with 10 jobs and each with 64 QD.
For mixed workload it is with QD = 8 and num_job= 1 and 10.


Thanks & Regards
Somnath

-----Original Message-----
From: Blinick, Stephen L [mailto:stephen.l.blinick@intel.com]
Sent: Thursday, September 03, 2015 1:02 PM
To: Somnath Roy
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Somnath -- thanks for publishing all the data, will be great to look at it offline.  I didn't find this info:  How many RBD volumes, and what size, did you use for your mixed tests?  Was it just one RBD w/ num_jobs=1 & 10?  Also how many client systems were necessary to drive the workload on the 4 storage nodes?      

I saw the same behavior quite a while back when playing with ramdisk journal... Not a lot of improvement.

Thanks,

Stephen

-----Original Message-----
From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
Sent: Thursday, September 3, 2015 10:42 AM
To: Mark Nelson; Robert LeBlanc
Cc: ceph-devel
Subject: RE: Ceph Write Path Improvement

Yes, As Mark said I will collect all the data and hopefully I can present in the next performance meeting.
BTW, I have tested with Hammer code base + NvRAM journal initially, but that performance is very spiky with ~10% performance gain (at max). I thought there is no point of collecting more data with that config.
That's why I have introduced a new throttling scheme that should benefit in all the scenarios.

Thanks & Regards
Somnath

-----Original Message-----
From: Mark Nelson [mailto:mnelson@redhat.com]
Sent: Thursday, September 03, 2015 9:42 AM
To: Robert LeBlanc; Somnath Roy
Cc: ceph-devel
Subject: Re: Ceph Write Path Improvement

On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Somnath,
>
> I'm having a hard time with your slide deck. Am I understanding 
> correctly that the default Hammer install was performed on SSDs with 
> co-located journals, but the optimized code was performed on the same 
> SSDs but the journal was in NVRAM? If so I'm having a hard time 
> understanding how these tests can be comparable. I really like the 
> performance gains you are seeing, but I'm trying to understand how 
> much the optimized code alone helps performance.

Hi Robert,

We talked about this a bit at the weekly performance meeting.  I think Somnath just hasn't gotten a chance to do those tests yet and is planning on doing them in the coming weeks.  I believe he started out with hammer on the SSDs and then tried to figure out how to tweak things to make the NVRAM configuration perform better.  Now he has to go back and retest the original configuration but with the new code.

Mark

________________________________

PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).

\x13  칻\x1c & ~ & \x18  +-  ݶ\x17  w  ˛   m \x1e \x17^  b  ^n r   z \x1a  h    &  \x1e G   h \x03( 階 ݢj"  \x1a ^[m     z ޖ   f   h   ~ m 

Re: Ceph Write Path Improvement

[Robert LeBlanc]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

This is great work. Thanks for sharing. I'm looking forward to the
resolution of the scale out issues!
- ----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1


On Wed, Sep 9, 2015 at 6:33 PM, Somnath Roy  wrote:
> Hi,
> Here is the updated presentation we discussed in the performance meeting today with performance data incorporated for the scenario where both journal/data on the same SSD.
>
> https://docs.google.com/presentation/d/15-Uqk0b4s1fVV1cG1G6Kba9xafcnIoLvfq8LUY7KBL0/edit#slide=id.p4
>
>
> Here is the summary:
> --------------------------
>
> For 4K :
> –       ~2.5X IOPs improvement, ~2.68X latency improvement with NVRAM as journal
> –       ~2.2X IOPS improvement, ~2.3X latency improvement  without NVRAM as journal
> –       More stable performance with NVRAM as journal
> For 16K:
> –       ~2.3X IOPS improvement, ~2.47X latency improvement with NVRAM as journal
> –       ~1.9X IOPS improvement, ~2X latency improvement  without NVRAM as journal
> –       More stable performance with NVRAM as journal
> For 64K:
> –       ~2X BW improvement, ~2X latency improvement with NVRAM as journal
> –       ~1.5X BW improvement, ~1.5X latency improvement  without NVRAM as journal
> –       More stable performance with NVRAM as journal
> For 4M:
> –       ~1.5X BW improvement, ~1.6X latency improvement with NVRAM as journal
> –       ~1.1X BW improvement, ~1.2X latency improvement  without NVRAM as journal
>
> For Mixed workload:
> ------------------------
> QD = 8 :
>         ~1.6X IOPs improvement, ~1.6X latency improvement with NVRAM as journal
>         ~1.5X IOPs improvement, ~1.5X latency improvement without NVRAM as journal
>         More stable performance with NVRAM as journal
>
> QD = 80 :
>         ~2.7X IOPs improvement, ~2.8X latency improvement with NVRAM as journal
>         ~2.4X IOPs improvement, ~2.5X latency improvement without NVRAM as journal
>         More stable performance with NVRAM as journal
>
> Will try to send out a pull request by next week.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Somnath Roy
> Sent: Thursday, September 03, 2015 1:06 PM
> To: 'Blinick, Stephen L'
> Cc: ceph-devel
> Subject: RE: Ceph Write Path Improvement
>
> Stephen,
> It's a 1 RBD volume (preconditioned) of 2 TB size from one physical client box.
> fio-rbd script I am running with 10 jobs and each with 64 QD.
> For mixed workload it is with QD = 8 and num_job= 1 and 10.
>
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Blinick, Stephen L [mailto:stephen.l.blinick@intel.com]
> Sent: Thursday, September 03, 2015 1:02 PM
> To: Somnath Roy
> Cc: ceph-devel
> Subject: RE: Ceph Write Path Improvement
>
> Somnath -- thanks for publishing all the data, will be great to look at it offline.  I didn't find this info:  How many RBD volumes, and what size, did you use for your mixed tests?  Was it just one RBD w/ num_jobs=1 & 10?  Also how many client systems were necessary to drive the workload on the 4 storage nodes?
>
> I saw the same behavior quite a while back when playing with ramdisk journal... Not a lot of improvement.
>
> Thanks,
>
> Stephen
>
> -----Original Message-----
> From: ceph-devel-owner@vger.kernel.org [mailto:ceph-devel-owner@vger.kernel.org] On Behalf Of Somnath Roy
> Sent: Thursday, September 3, 2015 10:42 AM
> To: Mark Nelson; Robert LeBlanc
> Cc: ceph-devel
> Subject: RE: Ceph Write Path Improvement
>
> Yes, As Mark said I will collect all the data and hopefully I can present in the next performance meeting.
> BTW, I have tested with Hammer code base + NvRAM journal initially, but that performance is very spiky with ~10% performance gain (at max). I thought there is no point of collecting more data with that config.
> That's why I have introduced a new throttling scheme that should benefit in all the scenarios.
>
> Thanks & Regards
> Somnath
>
> -----Original Message-----
> From: Mark Nelson [mailto:mnelson@redhat.com]
> Sent: Thursday, September 03, 2015 9:42 AM
> To: Robert LeBlanc; Somnath Roy
> Cc: ceph-devel
> Subject: Re: Ceph Write Path Improvement
>
> On 09/03/2015 11:23 AM, Robert LeBlanc wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> Somnath,
>>
>> I'm having a hard time with your slide deck. Am I understanding
>> correctly that the default Hammer install was performed on SSDs with
>> co-located journals, but the optimized code was performed on the same
>> SSDs but the journal was in NVRAM? If so I'm having a hard time
>> understanding how these tests can be comparable. I really like the
>> performance gains you are seeing, but I'm trying to understand how
>> much the optimized code alone helps performance.
>
> Hi Robert,
>
> We talked about this a bit at the weekly performance meeting.  I think Somnath just hasn't gotten a chance to do those tests yet and is planning on doing them in the coming weeks.  I believe he started out with hammer on the SSDs and then tried to figure out how to tweak things to make the NVRAM configuration perform better.  Now he has to go back and retest the original configuration but with the new code.
>
> Mark
>
> ________________________________
>
> PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies).
>
>    칻  & ~ &    +-  ݶ   w  ˛   m    ^  b  ^n r   z    h    &    G   h  ( 階 ݢj"     m     z ޖ   f   h   ~ m

-----BEGIN PGP SIGNATURE-----
Version: Mailvelope v1.0.2
Comment: https://www.mailvelope.com

wsFcBAEBCAAQBQJV8bvwCRDmVDuy+mK58QAAii0QAJRExHq2QlmLwI7rOD8K
JeXJ+rWJlP6CluRmuS3nN2UwT+hwxzzokyA4vzqhe0lvQkiiVPKXnuSp3hlo
U6uPjSigR0XwXdXq8nlIr51lXByB7/dbWUu62HORgi350nY1I2NzLDLUo8Hx
qxBrC4ZkH/h2+pfU2ann5nOBqe/0EM0NlbI0VQUvCo3M5il5GbqVWq/ZHDKj
LnrQtiz/QEWhq0xnJWdYlhFzkpGGNszdoHCzmSL/oGDbEpzDp6RxjF4YB1e8
Cf1xTDwmdZf3SJKKVsPbpmWG8/F3O6PKf4sbbWdo20DBYmrBbnpFWtJEr6q0
TKr2bXycSza7TNhoBNR+TGVyDuHDxnpMGneIN/NtVffwHn5dASPvveC2Y2OE
fRWccfDeaEXESfrwzMPWFk1n+ty6hpLvClyQaXenb2POmSSJ3VN9kxye8x0U
/7DiwJNWj1YaIgWGlpbOmQSWX2/PrI++MNRIB7ck0O1LG7ERhPUwdgYRahZ7
ImN8JfvOdUb7b160hdRAKpepNElfAeIkJKRlC6+6mZsOqw53Oa0MZtYMSzwU
3VbwBeJWqUS9itGCkqITNhtikdfKdWzIcwAGRPDFPqLqXWBs1twstN/ScPBa
fWDSbSjoi09LTAIsXrnLta44QZwuQDhEWUEhUcidtIbaD1t4kbCeGC+TKuA8
VluX
=/0NL
-----END PGP SIGNATURE-----
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html