Re: Infernalis on NVMe cephx authentication impact

[Mark Nelson <mnelson@redhat.com>]

Hi Orlando,

Hrm, looks like https://github.com/ceph/ceph/pull/3896 didn't make it 
into Hammer at release. :(  That might also account for a large part of 
the performance disparity.  Coincidentally Josh Durgin made a branch 
with the PR backported to hammer a couple of weeks ago here:

https://github.com/ceph/ceph/commits/wip-auth-hammer

Mark

On 02/02/2016 06:18 PM, Moreno, Orlando wrote:
> Following up on the first pass of all-flash performance numbers for Infernalis, we ran the same tests with cephx authentication on. Previously, for both Hammer and Infernalis, the performance reported was with authentication turned off. Since the authentication interface changed to libnss between Hammer and Infernalis, we wanted to verify the impact of authentication in a high-performance Ceph cluster.
>
> With authentication on, random 4K read performance drops by about 11% and clearly hits a wall in terms of max IOPS. Random writes see a greater impact with max performance reaching 176K IOPS compared to the 200K+ IOPS with authentication off. The mixed workload seems to be affected by this as well and maxes out at 408K IOPS. Considering the high throughput of the cluster, these numbers seem reasonable and the overhead of adding authentication did not degrade the performance as much as Hammer's authentication implementation where we saw at least a 30% hit on reads/writes.
>
> Below is a comparison table of authentication on vs off. More detailed data is available for anyone that is interested.
>
> 				Infernalis			Infernalis w/ cephx	
> 		IODepth	IOPS		Avg Lat (ms)	IOPS		Avg Lat (ms)
> ========================================================================
> 100% Rand Read	4	383747		0.619167	347850		0.683139
> 			8	645551		0.7345		581384		0.815726
> 			16	955990		0.994833	820765		1.153785
> 			32	1072001	1.774667	937832		2.023074
> 			64	1028112	3.578667	942742		4.036471
> 			96	1070847	5.402833	941746		6.06505
> 			128	1088625	7.085		N/A		N/A
>
> 100% Rand Write	4	131447		1.820833	111689		2.135115
> 			8	175180		2.8385		138931		3.43494
> 			16	198219		5.129333	163417		5.844878
> 			32	191775		10.0895		174522		10.956354
> 			64	185602		21.089167	176733		21.669579
> 			96	204202		30.601833	163963		35.046025
> 			128	233095		37.532667	150943		50.75762
>
> 70% Rand Read		4	234445		1.015333	210798		1.129461
> 			8	337808		1.417667	309584		1.538977
> 			16	394676		2.425333	360150		2.649063
> 			32	445295		4.3465		391638		4.879287
> 			64	478867		8.297833	408463		9.364712
> 			96	513590		11.9885		407493		14.084794
> 			128	532439		15.970333	406757		18.814887
>
>
> Thanks,
>
> Orlando
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>