Ceph & Cinder & HA

Ceph & Cinder & HA

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

Hi Florian,

I attended the Ceph & Cinder & HA session today at the summit. I've never really thought about strategies to implement HA and how Ceph can help. The extent of my knowledge on this topic is to make sure I do not confuse something that is "highly reliable" with something "highly available" ;-)

Guilio came to me after the session and I had to confess my ignorance on the topic of HA. Hence this mail and the hope that you can shed some light on this topic.

Cheers

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Ceph & Cinder & HA

[David Moreau Simard]

Loic,

Was there an etherpad ? I'm curious as I didn't attend that one.

HA with Cinder mostly revolves around setting up multiple cinder-api and cinder-volume servers. Was there something else involved ?
I can probably contribute my knowledge on that..

--
David Moreau Simard

> On Nov 6, 2014, at 4:57 PM, Loic Dachary <loic@dachary.org> wrote:
> 
> Hi Florian,
> 
> I attended the Ceph & Cinder & HA session today at the summit. I've never really thought about strategies to implement HA and how Ceph can help. The extent of my knowledge on this topic is to make sure I do not confuse something that is "highly reliable" with something "highly available" ;-)
> 
> Guilio came to me after the session and I had to confess my ignorance on the topic of HA. Hence this mail and the hope that you can shed some light on this topic.
> 
> Cheers
> 
> -- 
> Loïc Dachary, Artisan Logiciel Libre
> 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph & Cinder & HA

[Giulio Fidente]

On 11/07/2014 09:46 AM, David Moreau Simard wrote:
> Loic,
>
> Was there an etherpad ? I'm curious as I didn't attend that one.
>
> HA with Cinder mostly revolves around setting up multiple cinder-api and cinder-volume servers. Was there something else involved ?
> I can probably contribute my knowledge on that..

hi David,

indeed it would be great if we could get in touch! I'm still at the 
summit, in the TripleO room, in case you are around too.

So yes there is an etherpad and also a TripleO spec which goes a bit 
more into the details (linked from the etherpad):

https://etherpad.openstack.org/p/tripleo-kilo-l3-and-cinder-ha

The architecture we're considering is as follows:

   keepalived
     haproxy
       ceph-mon

where haproxy and ceph-mon are always up on all openstack 'controller' 
nodes (also running cinder) and keepalived is taking care of managing a 
virtual ip hosted on one of those
-- 
Giulio Fidente
GPG KEY: 08D733BA | IRC: giulivo

Re: Ceph & Cinder & HA

[David Moreau Simard]

Giulio,

Right. The volume part is tricky because the volume is ultimately tied to one cinder-volume instance.
I don't have a magic solution for that yet.

I'm not going to the contributor summit unfortunately.

I'm not personally a fan of keepalived since I've been spoiled by corosync. Let's see if we can chat about this later.

--
David Moreau Simard

> On Nov 7, 2014, at 10:35 AM, Giulio Fidente <gfidente@redhat.com> wrote:
> 
> On 11/07/2014 09:46 AM, David Moreau Simard wrote:
>> Loic,
>> 
>> Was there an etherpad ? I'm curious as I didn't attend that one.
>> 
>> HA with Cinder mostly revolves around setting up multiple cinder-api and cinder-volume servers. Was there something else involved ?
>> I can probably contribute my knowledge on that..
> 
> hi David,
> 
> indeed it would be great if we could get in touch! I'm still at the summit, in the TripleO room, in case you are around too.
> 
> So yes there is an etherpad and also a TripleO spec which goes a bit more into the details (linked from the etherpad):
> 
> https://etherpad.openstack.org/p/tripleo-kilo-l3-and-cinder-ha
> 
> The architecture we're considering is as follows:
> 
>  keepalived
>    haproxy
>      ceph-mon
> 
> where haproxy and ceph-mon are always up on all openstack 'controller' nodes (also running cinder) and keepalived is taking care of managing a virtual ip hosted on one of those
> -- 
> Giulio Fidente
> GPG KEY: 08D733BA | IRC: giulivo

Re: Ceph & Cinder & HA

[Giulio Fidente]

On 11/07/2014 10:50 AM, David Moreau Simard wrote:
> Giulio,
>
> Right. The volume part is tricky because the volume is ultimately tied to one cinder-volume instance.
> I don't have a magic solution for that yet.

yep but this can be worked around by customizing the host= setting in 
Cinder so all instances of cinder-volume look like a single 'host'

this was discussed with Cinder guy here, which I am in touch with too, 
and is known to expose some issues due to multiple cinder-volume 
instances potentially trying to get access to the same volume when 
multiple tasks are queued ... yet this is 'known stuff' and they are 
already working on a 'state machine' for Cinder which will fix the a/a 
scenario

until then, the idea is to deploy cinder-volume in a a/p scenario, yet 
customizing the host= setting
-- 
Giulio Fidente
GPG KEY: 08D733BA | IRC: giulivo

Re: Ceph & Cinder & HA

[Sebastien Han]

How do you envision the HAProxy setup?
I’m in the dev room, we can discuss that.

HAProxy is a good idea but you need to be careful on how you monitor the ceph-mon.

> On 07 Nov 2014, at 10:56, Giulio Fidente <gfidente@redhat.com> wrote:
> 
> On 11/07/2014 10:50 AM, David Moreau Simard wrote:
>> Giulio,
>> 
>> Right. The volume part is tricky because the volume is ultimately tied to one cinder-volume instance.
>> I don't have a magic solution for that yet.
> 
> yep but this can be worked around by customizing the host= setting in Cinder so all instances of cinder-volume look like a single 'host'
> 
> this was discussed with Cinder guy here, which I am in touch with too, and is known to expose some issues due to multiple cinder-volume instances potentially trying to get access to the same volume when multiple tasks are queued ... yet this is 'known stuff' and they are already working on a 'state machine' for Cinder which will fix the a/a scenario
> 
> until then, the idea is to deploy cinder-volume in a a/p scenario, yet customizing the host= setting
> -- 
> Giulio Fidente
> GPG KEY: 08D733BA | IRC: giulivo
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers.
–––– 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien.han@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph & Cinder & HA

[Justin Erenkrantz]

On Fri, Nov 7, 2014 at 4:35 AM, Giulio Fidente <gfidente@redhat.com> wrote:
> The architecture we're considering is as follows:
>
>   keepalived
>     haproxy
>       ceph-mon
>
> where haproxy and ceph-mon are always up on all openstack 'controller' nodes
> (also running cinder) and keepalived is taking care of managing a virtual ip
> hosted on one of those

For Ceph, you almost certainly don't need to use keepalived or haproxy
as all Ceph clients will fail over to the full set of monitor IPs (you
need a way of distributing those IPs to the cluster -
chef/puppet/ansible/etc/etc can handle that).  Having
keepalived/haproxy in the loop for ceph-mon is one more thing that
could break.

We do run cinder services behind the controller/headnodes nodes via
keepalived/haproxy - the "trick" such as it is for Ceph is to set all
the hostnames in cinder to being the same and it works okay.  See:

https://github.com/bloomberg/chef-bcpc/blob/master/cookbooks/bcpc/templates/default/cinder.conf.erb

We haven't seen any issues in an active/active configuration; though
there are some MySQL schema deadlocks that aren't necessarily related
to Ceph sitting in Cinder.

Cheers.  -- justin

Re: Ceph & Cinder & HA

[Sebastien Han]

Indeed, that’s what we discussed with Giulio, using HAProxy for the Monitors is a bit overkill since the client is natively able to switch between different monitor source.

Loic, maybe you can elaborate a bit more on the usage of HAProxy for the monitors?

> On 09 Nov 2014, at 22:54, Justin Erenkrantz <justin@erenkrantz.com> wrote:
> 
> On Fri, Nov 7, 2014 at 4:35 AM, Giulio Fidente <gfidente@redhat.com> wrote:
>> The architecture we're considering is as follows:
>> 
>>  keepalived
>>    haproxy
>>      ceph-mon
>> 
>> where haproxy and ceph-mon are always up on all openstack 'controller' nodes
>> (also running cinder) and keepalived is taking care of managing a virtual ip
>> hosted on one of those
> 
> For Ceph, you almost certainly don't need to use keepalived or haproxy
> as all Ceph clients will fail over to the full set of monitor IPs (you
> need a way of distributing those IPs to the cluster -
> chef/puppet/ansible/etc/etc can handle that).  Having
> keepalived/haproxy in the loop for ceph-mon is one more thing that
> could break.
> 
> We do run cinder services behind the controller/headnodes nodes via
> keepalived/haproxy - the "trick" such as it is for Ceph is to set all
> the hostnames in cinder to being the same and it works okay.  See:
> 
> https://github.com/bloomberg/chef-bcpc/blob/master/cookbooks/bcpc/templates/default/cinder.conf.erb
> 
> We haven't seen any issues in an active/active configuration; though
> there are some MySQL schema deadlocks that aren't necessarily related
> to Ceph sitting in Cinder.
> 
> Cheers.  -- justin
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


Cheers.
–––– 
Sébastien Han 
Cloud Architect 

"Always give 100%. Unless you're giving blood."

Phone: +33 (0)1 49 70 99 72 
Mail: sebastien.han@enovance.com 
Address : 11 bis, rue Roquépine - 75008 Paris
Web : www.enovance.com - Twitter : @enovance 

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph & Cinder & HA

[Loic Dachary]

[-- Attachment #1: Type: text/plain, Size: 2515 bytes --]



On 10/11/2014 11:34, Sebastien Han wrote:
> Indeed, that’s what we discussed with Giulio, using HAProxy for the Monitors is a bit overkill since the client is natively able to switch between different monitor source.
> 
> Loic, maybe you can elaborate a bit more on the usage of HAProxy for the monitors?

Hi Sebastien,

I can't elaborate because I know nothing about HAProxy (well, close to nothing ;-). I'd like to learn more about how HA (via HAProxy or something else) can leverage Ceph in a meaningfull way thouhg, hence my interest for the topic.

Cheers

> 
>> On 09 Nov 2014, at 22:54, Justin Erenkrantz <justin@erenkrantz.com> wrote:
>>
>> On Fri, Nov 7, 2014 at 4:35 AM, Giulio Fidente <gfidente@redhat.com> wrote:
>>> The architecture we're considering is as follows:
>>>
>>>  keepalived
>>>    haproxy
>>>      ceph-mon
>>>
>>> where haproxy and ceph-mon are always up on all openstack 'controller' nodes
>>> (also running cinder) and keepalived is taking care of managing a virtual ip
>>> hosted on one of those
>>
>> For Ceph, you almost certainly don't need to use keepalived or haproxy
>> as all Ceph clients will fail over to the full set of monitor IPs (you
>> need a way of distributing those IPs to the cluster -
>> chef/puppet/ansible/etc/etc can handle that).  Having
>> keepalived/haproxy in the loop for ceph-mon is one more thing that
>> could break.
>>
>> We do run cinder services behind the controller/headnodes nodes via
>> keepalived/haproxy - the "trick" such as it is for Ceph is to set all
>> the hostnames in cinder to being the same and it works okay.  See:
>>
>> https://github.com/bloomberg/chef-bcpc/blob/master/cookbooks/bcpc/templates/default/cinder.conf.erb
>>
>> We haven't seen any issues in an active/active configuration; though
>> there are some MySQL schema deadlocks that aren't necessarily related
>> to Ceph sitting in Cinder.
>>
>> Cheers.  -- justin
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> Cheers.
> –––– 
> Sébastien Han 
> Cloud Architect 
> 
> "Always give 100%. Unless you're giving blood."
> 
> Phone: +33 (0)1 49 70 99 72 
> Mail: sebastien.han@enovance.com 
> Address : 11 bis, rue Roquépine - 75008 Paris
> Web : www.enovance.com - Twitter : @enovance 
> 

-- 
Loïc Dachary, Artisan Logiciel Libre


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 263 bytes --]

Re: Ceph & Cinder & HA

[Giulio Fidente]

On 11/10/2014 11:51 AM, Loic Dachary wrote:
>
>
> On 10/11/2014 11:34, Sebastien Han wrote:
>> Indeed, that’s what we discussed with Giulio, using HAProxy for the Monitors is a bit overkill since the client is natively able to switch between different monitor source.
>>
>> Loic, maybe you can elaborate a bit more on the usage of HAProxy for the monitors?
>
> Hi Sebastien,
>
> I can't elaborate because I know nothing about HAProxy (well, close to nothing ;-). I'd like to learn more about how HA (via HAProxy or something else) can leverage Ceph in a meaningfull way thouhg, hence my interest for the topic.

Thanks all for helping.

Loic explained requests for the non-leader monitors go to the monitor 
anyway in which case I understand it makes more sense to avoid the 
complication of balancing the monitors.

-- 
Giulio Fidente
GPG KEY: 08D733BA
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph & Cinder & HA

[Gregory Farnum]

On Mon, Nov 10, 2014 at 3:50 AM, Giulio Fidente <gfidente@redhat.com> wrote:
> On 11/10/2014 11:51 AM, Loic Dachary wrote:
>>
>>
>>
>> On 10/11/2014 11:34, Sebastien Han wrote:
>>>
>>> Indeed, that’s what we discussed with Giulio, using HAProxy for the
>>> Monitors is a bit overkill since the client is natively able to switch
>>> between different monitor source.
>>>
>>> Loic, maybe you can elaborate a bit more on the usage of HAProxy for the
>>> monitors?
>>
>>
>> Hi Sebastien,
>>
>> I can't elaborate because I know nothing about HAProxy (well, close to
>> nothing ;-). I'd like to learn more about how HA (via HAProxy or something
>> else) can leverage Ceph in a meaningfull way thouhg, hence my interest for
>> the topic.
>
>
> Thanks all for helping.
>
> Loic explained requests for the non-leader monitors go to the monitor anyway
> in which case I understand it makes more sense to avoid the complication of
> balancing the monitors.

Well, that's not quite right — *modification* requests get routed
through the leader; read-only operations are handled by the non-leader
monitors.
But I can't imagine how HAProxy would or could fit into that
communications path, and the clients do switch independently between
monitors as required (and there are a few different methods to
approximate load-balancing in the system, although nothing too
serious).
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html