OSD network failure

OSD network failure

[Gandalf Corvotempesta]

Hi,
what happens in case of OSD network failure? Is ceph smart enough to
isolate OSDs not synced?
Should I use LACP in ODS network or a single 10GBe per server should be ok?

LACP will need stackable switches and much more hardware investment.

Re: OSD network failure

[Josh Durgin]

On 11/13/2012 06:15 AM, Gandalf Corvotempesta wrote:
> Hi,
> what happens in case of OSD network failure? Is ceph smart enough to
> isolate OSDs not synced?
> Should I use LACP in ODS network or a single 10GBe per server should be ok?
>
> LACP will need stackable switches and much more hardware investment.

OSDs send heartbeats to each other and report failure to receive
a heartbeat in a certain interval to the monitor cluster.
When the monitor cluster receives enough of these reports,
it marks the OSD 'down' in the OSD map, and after a grace period
to allow for flapping or daemon restarts, marks the osd 'out'
as well. This makes the cluster rebalance any data that was on the
failed OSD, and places no new data there.

A lot of this is configurable, but that's the basic model.

In this model, a network failure is equivalent to extreme slowness or a
crashed OSD - everything results in an updated map of the cluster
eventually, and the OSDs maintain strong consistency of the data
through the peering and recovery processes.

So basically you'd only need a single nic per storage node. Multiple
can be useful to separate frontend and backend traffic, but ceph
is designed to maintain strong consistency when failures occur.

Josh

Re: OSD network failure

[Gandalf Corvotempesta]

2012/11/15 Josh Durgin <josh.durgin@inktank.com>:
> So basically you'd only need a single nic per storage node. Multiple
> can be useful to separate frontend and backend traffic, but ceph
> is designed to maintain strong consistency when failures occur.

Probably i've not exaplained well.
I'll have multiple nics, one for frontend, one for backend used as ODS
sync network.
What happens in case of backend network failure? The frontend network
is still ok, OSD is
still reachable but is not able to sync datas.

Re: OSD network failure

[Josh Durgin]

On 11/15/2012 01:51 AM, Gandalf Corvotempesta wrote:
> 2012/11/15 Josh Durgin <josh.durgin@inktank.com>:
>> So basically you'd only need a single nic per storage node. Multiple
>> can be useful to separate frontend and backend traffic, but ceph
>> is designed to maintain strong consistency when failures occur.
>
> Probably i've not exaplained well.
> I'll have multiple nics, one for frontend, one for backend used as ODS
> sync network.
> What happens in case of backend network failure? The frontend network
> is still ok, OSD is
> still reachable but is not able to sync datas.

Ah, ok. By default, the OSDs use the backend network for heartbeats,
so if it fails, they will notice and report peers they can't reach as
failed to the monitors, and the normal failure handling takes care
of things.

If you're worried about consistency, remember that a write won't
complete until it's on disk on all replicas. If you're interested
in the gory details of maintaining consistency, check out the peering
process [1].

Josh

[1] http://ceph.com/docs/master/dev/peering/

Re: OSD network failure

[Gandalf Corvotempesta]

2012/11/17 Josh Durgin <josh.durgin@inktank.com>:
> Ah, ok. By default, the OSDs use the backend network for heartbeats,
> so if it fails, they will notice and report peers they can't reach as
> failed to the monitors, and the normal failure handling takes care
> of things.
>
> If you're worried about consistency, remember that a write won't
> complete until it's on disk on all replicas. If you're interested
> in the gory details of maintaining consistency, check out the peering
> process [1].

Ok. now it's clear.
So it will be safe to use a single NIC for backend network.

Re: OSD network failure

[Gregory Farnum]

On Fri, Nov 16, 2012 at 5:56 PM, Josh Durgin <josh.durgin@inktank.com> wrote:
> On 11/15/2012 01:51 AM, Gandalf Corvotempesta wrote:
>>
>> 2012/11/15 Josh Durgin <josh.durgin@inktank.com>:
>>>
>>> So basically you'd only need a single nic per storage node. Multiple
>>> can be useful to separate frontend and backend traffic, but ceph
>>> is designed to maintain strong consistency when failures occur.
>>
>>
>> Probably i've not exaplained well.
>> I'll have multiple nics, one for frontend, one for backend used as ODS
>> sync network.
>> What happens in case of backend network failure? The frontend network
>> is still ok, OSD is
>> still reachable but is not able to sync datas.
>
>
> Ah, ok. By default, the OSDs use the backend network for heartbeats,
> so if it fails, they will notice and report peers they can't reach as
> failed to the monitors, and the normal failure handling takes care
> of things.
>
> If you're worried about consistency, remember that a write won't
> complete until it's on disk on all replicas. If you're interested
> in the gory details of maintaining consistency, check out the peering
> process [1].
>
> Josh
>
> [1] http://ceph.com/docs/master/dev/peering/

Actually, right now a failed cluster and an up public network is
something the OSDs do not handle well — they will mark each other down
on the monitor and then tell the monitor "hey, I'm not dead!" and
start flapping pretty horrendously. We first ran across it a couple
weeks ago and have started to think about it, but I'm not sure a fix
for this is going to make it into the initial Bobtail release. :(
-Greg
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html