How will Ceph cope with a failed Journal device?

How will Ceph cope with a failed Journal device?

[Guido Winkelmann]

Hi,

We have been having a lot of discussions at my workplace about whether to 
employ a Ceph cluster in production or not, and if yes, how to set up the 
hardware for it. During that discussion, I mentioned that, according to the 
documentation, we should see significant speedups from using dedicated SSDs 
for the OSD's journals. Unfortunately, my colleagues did not like this idea at 
all - many of them had bad experiences with SSDs failing or at least read a 
lot about that on the Internet, and there's a general consensus that SSDs are 
just not quite reliable enough yet for production servers.

This leads me to the question: What exactly can happen if an OSD's journal 
device suddenly fails during operations? Can that lead to data loss or 
corruption, or disruptions of the service?

In my experience with the small three-machine test cluster I have here, a 
single failed node usually would lead to a pretty severe outage of the entire 
cluster on the order of ten minutes or more (probably much more when it's a 
really big node that fails), though so far no data loss or corruption...

Regards,

	Guido

Re: How will Ceph cope with a failed Journal device?

[Calvin Morrow]

I posted the same question to the list last week and never got a
reply.  In addition, I'd also like to know if there's a difference in
failure behavior between XFS backed Ceph (writeahead journaling) and
BTRFS backed Ceph (parallel journaling).

Calvin

On Fri, May 18, 2012 at 12:30 PM, Guido Winkelmann
<guido-ceph@thisisnotatest.de> wrote:
> Hi,
>
> We have been having a lot of discussions at my workplace about whether to
> employ a Ceph cluster in production or not, and if yes, how to set up the
> hardware for it. During that discussion, I mentioned that, according to the
> documentation, we should see significant speedups from using dedicated SSDs
> for the OSD's journals. Unfortunately, my colleagues did not like this idea at
> all - many of them had bad experiences with SSDs failing or at least read a
> lot about that on the Internet, and there's a general consensus that SSDs are
> just not quite reliable enough yet for production servers.
>
> This leads me to the question: What exactly can happen if an OSD's journal
> device suddenly fails during operations? Can that lead to data loss or
> corruption, or disruptions of the service?
>
> In my experience with the small three-machine test cluster I have here, a
> single failed node usually would lead to a pretty severe outage of the entire
> cluster on the order of ten minutes or more (probably much more when it's a
> really big node that fails), though so far no data loss or corruption...
>
> Regards,
>
>        Guido
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How will Ceph cope with a failed Journal device?

[Tommi Virtanen]

On Fri, May 18, 2012 at 11:36 AM, Calvin Morrow <calvin.morrow@gmail.com> wrote:
> I posted the same question to the list last week and never got a
> reply.  In addition, I'd also like to know if there's a difference in
> failure behavior between XFS backed Ceph (writeahead journaling) and
> BTRFS backed Ceph (parallel journaling).

Actually, Sam replied to your email a week ago:

http://thread.gmane.org/gmane.comp.file-systems.ceph.devel/6258/focus=6358

Summarizing/rephrasing:

On both xfs & btrfs, a journal device giving I/O errors should make
the osd stop operating.

Losing a journal with xfs: you should destroy & recreate the osd, and
your data is safe because of Ceph replication across osds.

Losing a journal with btrfs: creating a new journal should let the osd
recover the missing parts from replicas (and your data is safe mostly
because of Ceph replication, recovery is just faster).
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How will Ceph cope with a failed Journal device?

[Tommi Virtanen]

On Fri, May 18, 2012 at 1:01 PM, Tommi Virtanen <tv@inktank.com> wrote:
> Actually, Sam replied to your email a week ago:

Your email *from* a week ago. He replied yesterday.

Re: How will Ceph cope with a failed Journal device?

[Jerker Nyberg]

On Fri, 18 May 2012, Tommi Virtanen wrote:

> Losing a journal with btrfs: creating a new journal should let the osd
> recover the missing parts from replicas (and your data is safe mostly
> because of Ceph replication, recovery is just faster).

Cool! No more SSDs (that might fail over being written to continuously 
after a couple of months depending in size, prize, write cycles etc) just 
add a lot of RAM, keep the journals on tmpfs and make sure to run Ceph on 
Btrfs? While keeping the replicas separated so not all fail at once.

The contents of the storage node will not be corrupt or something (just a 
bit old) when losing the journal?

--jerker

Re: How will Ceph cope with a failed Journal device?

[Tommi Virtanen]

[Whoops, resending as plain text to make vger happy.]

On Fri, Jun 1, 2012 at 4:35 AM, Jerker Nyberg <jerker@update.uu.se> wrote:
> Cool! No more SSDs (that might fail over being written to continuously
> after a couple of months depending in size, prize, write cycles etc) just
> add a lot of RAM, keep the journals on tmpfs and make sure to run Ceph on
> Btrfs? While keeping the replicas separated so not all fail at once.
>
> The contents of the storage node will not be corrupt or something (just a
> bit old) when losing the journal?

The problem with storing things in RAM is, what if your rack/row/data
center loses power, all at once. It's really hard to guard against
those kinds of massive failures. If you don't have a persistent
journal, you might as well not have a journal at all.

The memory-only systems you see out there are typically only used to
the kinds of applications where rolling back to last (on-disk)
snapshot is acceptable -- stuff like shopping carts. Ceph is built on
significantly stronger promises, so it's not the ideal match for an
architecture like that.

It's also unclear to me on what would happen to Ceph if all the
replicas lost their journals at the same time. That might cause bigger
problems, since there's no up to date replica to pull the lost data
from.

Re: How will Ceph cope with a failed Journal device?

[Jerker Nyberg]

On Fri, 1 Jun 2012, Tommi Virtanen wrote:

> The problem with storing things in RAM is, what if your rack/row/data
> center loses power, all at once. It's really hard to guard against
> those kinds of massive failures. If you don't have a persistent
> journal, you might as well not have a journal at all.

The storage nodes can be put in different data centers with different 
kinds of uninterruptable power, from emergency power (diesel backed up 
batteries) to a small UPS (batteries) for individual nodes. This would 
hopefully give Ceph enough time to write down the journal to disk.

I am not sure that using for example battery backed up hardware raid cards 
would give better performance and reliability for the same money. 
Especially if existing infrastructure may be used for uninterruptable 
power. That would be interesting to figure out.

> The memory-only systems you see out there are typically only used to
> the kinds of applications where rolling back to last (on-disk)
> snapshot is acceptable -- stuff like shopping carts. Ceph is built on
> significantly stronger promises, so it's not the ideal match for an
> architecture like that.

I will begin with rsyncing disk backups there and maybe use it for 
temporary but fast storage. Rolling back to a consistent state some 
(shorter) time ago would be fine. If Ceph itself is stable it could later 
be used for data that cannot so easily be recreated.

It would be great to be able for individual users and existing 
applications to use the data directly from Ceph and old snapshots there 
instead of having to read back the data from Ceph into a normal file 
server by a system administrator.

> It's also unclear to me on what would happen to Ceph if all the
> replicas lost their journals at the same time. That might cause bigger
> problems, since there's no up to date replica to pull the lost data
> from.

It would be very interesting to know what would happen.

--jerker