Ceph version 0.56.1, data loss on power failure

Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

Hi,

I hava a ubuntu 12.10 host with image mapped with rbd module. The
problem is that when this host
face a power failure all data on the mapped image is lost. If I unmap
the image via the rbd unmap command, the data is preserved. I've
noticed this problem also in version 0.56

Best regards,
Marcin

Re: Ceph version 0.56.1, data loss on power failure

[Wido den Hollander]

On 01/10/2013 02:54 PM, Marcin Szukala wrote:
> Hi,
>
> I hava a ubuntu 12.10 host with image mapped with rbd module. The
> problem is that when this host
> face a power failure all data on the mapped image is lost. If I unmap
> the image via the rbd unmap command, the data is preserved. I've
> noticed this problem also in version 0.56
>

What kind of filesystem are you using on the mapped RBD image?

Are you sure the filesystem isn't corrupted?

Which kernel are you using on the host, the stock kernel from Ubuntu 12.10?

Wido

> Best regards,
> Marcin
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ceph version 0.56.1, data loss on power failure

[Sylvain Munaut]

Hi,

> I hava a ubuntu 12.10 host with image mapped with rbd module. The
> problem is that when this host
> face a power failure all data on the mapped image is lost. If I unmap
> the image via the rbd unmap command, the data is preserved. I've
> noticed this problem also in version 0.56

Make sure you use a recent kernel (like latest 3.6.x or more recent).

I remember having RBD corruption and panics when I was using 3.2
series, but since I moved to recent RBD code, it never happenned
again.

Cheers,

    Sylvain

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

Hi,

Scenario is correct but the last line. I can mount the image, but the
data that was written to the image before power failure is lost.

Currently the ceph cluster is not healthy, but i dont think its
related because I had this issue before the cluster itsef had issues
(about that I will write in different post not to mix topics).

Regards,
Marcin


2013/1/10 Sylvain Munaut <s.munaut@whatever-company.com>:
> Hi,
>
>> I hava a ubuntu 12.10 host with image mapped with rbd module. The
>> problem is that when this host
>> face a power failure all data on the mapped image is lost. If I unmap
>> the image via the rbd unmap command, the data is preserved. I've
>> noticed this problem also in version 0.56
>
> Make sure you use a recent kernel (like latest 3.6.x or more recent).
>
> I remember having RBD corruption and panics when I was using 3.2
> series, but since I moved to recent RBD code, it never happenned
> again.
>
> Cheers,
>
>     Sylvain

Re: Ceph version 0.56.1, data loss on power failure

[Gregory Farnum]

On Thu, Jan 10, 2013 at 8:56 AM, Marcin Szukala
<szukala.marcin@gmail.com> wrote:
> Hi,
>
> Scenario is correct but the last line. I can mount the image, but the
> data that was written to the image before power failure is lost.
>
> Currently the ceph cluster is not healthy, but i dont think its
> related because I had this issue before the cluster itsef had issues
> (about that I will write in different post not to mix topics).

This sounds like one of two possibilities:
1) You aren't actually committing data to RADOS very often and so when
the power fails you lose several minutes of writes. How much data are
you losing, how's it generated, and is whatever you're doing running
any kind of fsync or sync? And what filesystem are you using?
2) Your cluster is actually not accepting writes and so RBD never
manages to do a write but you aren't doing much and so you don't
notice. What's the output of ceph -s?
-Greg

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

2013/1/10 Gregory Farnum <greg@inktank.com>:
> On Thu, Jan 10, 2013 at 8:56 AM, Marcin Szukala
> <szukala.marcin@gmail.com> wrote:
>> Hi,
>>
>> Scenario is correct but the last line. I can mount the image, but the
>> data that was written to the image before power failure is lost.
>>
>> Currently the ceph cluster is not healthy, but i dont think its
>> related because I had this issue before the cluster itsef had issues
>> (about that I will write in different post not to mix topics).
>
> This sounds like one of two possibilities:
> 1) You aren't actually committing data to RADOS very often and so when
> the power fails you lose several minutes of writes. How much data are
> you losing, how's it generated, and is whatever you're doing running
> any kind of fsync or sync? And what filesystem are you using?
> 2) Your cluster is actually not accepting writes and so RBD never
> manages to do a write but you aren't doing much and so you don't
> notice. What's the output of ceph -s?
> -Greg

Hi,

Today I have created new ceph cluster from scratch.
root@ceph-1:~# ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at
{a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
election epoch 4, quorum 0,1,2 a,b,d
   osdmap e65: 56 osds: 56 up, 56 in
    pgmap v3892: 13744 pgs: 13744 active+clean; 73060 MB data, 147 GB
used, 51983 GB / 52131 GB avail
   mdsmap e1: 0/0/1 up

The issue persisst.
I`am losing all of data on the image. On the mounted image I have 5
logical volumes.

root@compute-9:~# mount
(snip)
/dev/mapper/compute--9-nova on /var/lib/nova type xfs (rw)
/dev/mapper/compute--9-tmp on /tmp type xfs (rw)
/dev/mapper/compute--9-libvirt on /etc/libvirt type xfs (rw)
/dev/mapper/compute--9-log on /var/log type xfs (rw)
/dev/mapper/compute--9-openvswitch on /var/lib/openvswitch type xfs (rw)

So I have directories with little to none data writes and with a lot
of writes (logs). No fsync or sync. Filesystem is xfs.

Regards,
Marcin

Re: Ceph version 0.56.1, data loss on power failure

[Gregory Farnum]

On Fri, Jan 11, 2013 at 3:07 AM, Marcin Szukala
<szukala.marcin@gmail.com> wrote:
> 2013/1/10 Gregory Farnum <greg@inktank.com>:
>> On Thu, Jan 10, 2013 at 8:56 AM, Marcin Szukala
>> <szukala.marcin@gmail.com> wrote:
>>> Hi,
>>>
>>> Scenario is correct but the last line. I can mount the image, but the
>>> data that was written to the image before power failure is lost.
>>>
>>> Currently the ceph cluster is not healthy, but i dont think its
>>> related because I had this issue before the cluster itsef had issues
>>> (about that I will write in different post not to mix topics).
>>
>> This sounds like one of two possibilities:
>> 1) You aren't actually committing data to RADOS very often and so when
>> the power fails you lose several minutes of writes. How much data are
>> you losing, how's it generated, and is whatever you're doing running
>> any kind of fsync or sync? And what filesystem are you using?
>> 2) Your cluster is actually not accepting writes and so RBD never
>> manages to do a write but you aren't doing much and so you don't
>> notice. What's the output of ceph -s?
>> -Greg
>
> Hi,
>
> Today I have created new ceph cluster from scratch.
> root@ceph-1:~# ceph -s
>    health HEALTH_OK
>    monmap e1: 3 mons at
> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
> election epoch 4, quorum 0,1,2 a,b,d
>    osdmap e65: 56 osds: 56 up, 56 in
>     pgmap v3892: 13744 pgs: 13744 active+clean; 73060 MB data, 147 GB
> used, 51983 GB / 52131 GB avail
>    mdsmap e1: 0/0/1 up
>
> The issue persisst.
> I`am losing all of data on the image.

So you mean you mount the image, format it with 5 XFS filesystems as
below, run it for a while, and then the power on the system fails.
Then you turn the system back on, attach the image, and it has no
filesystems on it at all? Or the filesystems remain and can be mounted
but they have no data?
-Greg

> On the mounted image I have 5 logical volumes.
>
> root@compute-9:~# mount
> (snip)
> /dev/mapper/compute--9-nova on /var/lib/nova type xfs (rw)
> /dev/mapper/compute--9-tmp on /tmp type xfs (rw)
> /dev/mapper/compute--9-libvirt on /etc/libvirt type xfs (rw)
> /dev/mapper/compute--9-log on /var/log type xfs (rw)
> /dev/mapper/compute--9-openvswitch on /var/lib/openvswitch type xfs (rw)
>
> So I have directories with little to none data writes and with a lot
> of writes (logs). No fsync or sync. Filesystem is xfs.

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

Hi,

The filesystem remains, but all of the data is lost.

Regards,
Marcin

2013/1/11 Gregory Farnum <greg@inktank.com>:
> On Fri, Jan 11, 2013 at 3:07 AM, Marcin Szukala
> <szukala.marcin@gmail.com> wrote:
>> 2013/1/10 Gregory Farnum <greg@inktank.com>:
>>> On Thu, Jan 10, 2013 at 8:56 AM, Marcin Szukala
>>> <szukala.marcin@gmail.com> wrote:
>>>> Hi,
>>>>
>>>> Scenario is correct but the last line. I can mount the image, but the
>>>> data that was written to the image before power failure is lost.
>>>>
>>>> Currently the ceph cluster is not healthy, but i dont think its
>>>> related because I had this issue before the cluster itsef had issues
>>>> (about that I will write in different post not to mix topics).
>>>
>>> This sounds like one of two possibilities:
>>> 1) You aren't actually committing data to RADOS very often and so when
>>> the power fails you lose several minutes of writes. How much data are
>>> you losing, how's it generated, and is whatever you're doing running
>>> any kind of fsync or sync? And what filesystem are you using?
>>> 2) Your cluster is actually not accepting writes and so RBD never
>>> manages to do a write but you aren't doing much and so you don't
>>> notice. What's the output of ceph -s?
>>> -Greg
>>
>> Hi,
>>
>> Today I have created new ceph cluster from scratch.
>> root@ceph-1:~# ceph -s
>>    health HEALTH_OK
>>    monmap e1: 3 mons at
>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
>> election epoch 4, quorum 0,1,2 a,b,d
>>    osdmap e65: 56 osds: 56 up, 56 in
>>     pgmap v3892: 13744 pgs: 13744 active+clean; 73060 MB data, 147 GB
>> used, 51983 GB / 52131 GB avail
>>    mdsmap e1: 0/0/1 up
>>
>> The issue persisst.
>> I`am losing all of data on the image.
>
> So you mean you mount the image, format it with 5 XFS filesystems as
> below, run it for a while, and then the power on the system fails.
> Then you turn the system back on, attach the image, and it has no
> filesystems on it at all? Or the filesystems remain and can be mounted
> but they have no data?
> -Greg
>
>> On the mounted image I have 5 logical volumes.
>>
>> root@compute-9:~# mount
>> (snip)
>> /dev/mapper/compute--9-nova on /var/lib/nova type xfs (rw)
>> /dev/mapper/compute--9-tmp on /tmp type xfs (rw)
>> /dev/mapper/compute--9-libvirt on /etc/libvirt type xfs (rw)
>> /dev/mapper/compute--9-log on /var/log type xfs (rw)
>> /dev/mapper/compute--9-openvswitch on /var/lib/openvswitch type xfs (rw)
>>
>> So I have directories with little to none data writes and with a lot
>> of writes (logs). No fsync or sync. Filesystem is xfs.

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

Hi all,

Any ideas how can I resolve my issue? Or where the problem is?

Let me describe the issue.
Host boots up and maps RBD image with XFS filesystems
Host mounts the filesystems from the RBD image
Host starts to write data to the mounted filesystems
Host experiences power failure
Host comes up and map the RBD image
Host mounts the filesystems from the RBD image
All data from all filesystems is lost
Host is able to use the filesystems with no problems.

Filesystem is XFS, no errors on filesystem,

Kernel 3.5.0-19-generic

root@openstack-1:/etc/init# ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at
{a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
election epoch 10, quorum 0,1,2 a,b,d
   osdmap e132: 56 osds: 56 up, 56 in
    pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
used, 52028 GB / 52131 GB avail
   mdsmap e1: 0/0/1 up

Regards,
Marcin

Re: Ceph version 0.56.1, data loss on power failure

[Wido den Hollander]

On 01/16/2013 11:50 AM, Marcin Szukala wrote:
> Hi all,
>
> Any ideas how can I resolve my issue? Or where the problem is?
>
> Let me describe the issue.
> Host boots up and maps RBD image with XFS filesystems
> Host mounts the filesystems from the RBD image
> Host starts to write data to the mounted filesystems
> Host experiences power failure
> Host comes up and map the RBD image
> Host mounts the filesystems from the RBD image
> All data from all filesystems is lost
> Host is able to use the filesystems with no problems.
>
> Filesystem is XFS, no errors on filesystem,
>

That simply does not make sense to me. How can all data be gone and the 
FS just mount cleanly.

Can you try to format the RBD with EXT4 and see if that makes any 
difference.

Could you also try to run a "sync" prior to pulling the power from the 
host to see if that makes any difference.

Wido

> Kernel 3.5.0-19-generic
>
> root@openstack-1:/etc/init# ceph -s
>     health HEALTH_OK
>     monmap e1: 3 mons at
> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
> election epoch 10, quorum 0,1,2 a,b,d
>     osdmap e132: 56 osds: 56 up, 56 in
>      pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
> used, 52028 GB / 52131 GB avail
>     mdsmap e1: 0/0/1 up
>
> Regards,
> Marcin
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Ceph version 0.56.1, data loss on power failure

[Yann Dupont]

Le 16/01/2013 11:53, Wido den Hollander a écrit :
>
>
> On 01/16/2013 11:50 AM, Marcin Szukala wrote:

>> Hi all,
>>
>> Any ideas how can I resolve my issue? Or where the problem is?
>>
>> Let me describe the issue.
>> Host boots up and maps RBD image with XFS filesystems
>> Host mounts the filesystems from the RBD image
>> Host starts to write data to the mounted filesystems
>> Host experiences power failure
you are not doing sync there, right ?

>> Host comes up and map the RBD image
>> Host mounts the filesystems from the RBD image
>> All data from all filesystems is lost
>> Host is able to use the filesystems with no problems.
>>
>> Filesystem is XFS, no errors on filesystem,

you MAY have hit an XFS issue.

Please follow XFS list, in particular this thread :
http://oss.sgi.com/pipermail/xfs/2012-December/023021.html

If i Remember well, this one is after 3.4 kernel, and I think the fix 
isn't in the current ubuntu kernel.


cheers,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph version 0.56.1, data loss on power failure

[Dino Yancey]

Hi Marcin,

Not sure if anyone asked, but are your OSD journals on actual disk or
are you using tmpfs?

Dino

On Wed, Jan 16, 2013 at 4:53 AM, Wido den Hollander <wido@widodh.nl> wrote:
>
>
> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>>
>> Hi all,
>>
>> Any ideas how can I resolve my issue? Or where the problem is?
>>
>> Let me describe the issue.
>> Host boots up and maps RBD image with XFS filesystems
>> Host mounts the filesystems from the RBD image
>> Host starts to write data to the mounted filesystems
>> Host experiences power failure
>> Host comes up and map the RBD image
>> Host mounts the filesystems from the RBD image
>> All data from all filesystems is lost
>> Host is able to use the filesystems with no problems.
>>
>> Filesystem is XFS, no errors on filesystem,
>>
>
> That simply does not make sense to me. How can all data be gone and the FS
> just mount cleanly.
>
> Can you try to format the RBD with EXT4 and see if that makes any
> difference.
>
> Could you also try to run a "sync" prior to pulling the power from the host
> to see if that makes any difference.
>
> Wido
>
>
>> Kernel 3.5.0-19-generic
>>
>> root@openstack-1:/etc/init# ceph -s
>>     health HEALTH_OK
>>     monmap e1: 3 mons at
>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
>> election epoch 10, quorum 0,1,2 a,b,d
>>     osdmap e132: 56 osds: 56 up, 56 in
>>      pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
>> used, 52028 GB / 52131 GB avail
>>     mdsmap e1: 0/0/1 up
>>
>> Regards,
>> Marcin
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
______________________________
Dino Yancey
2GNT.com Admin

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

Hi Dino,

journals are on dedicated SSD

Regard,
Marcin

2013/1/16 Dino Yancey <dino2gnt@gmail.com>:
> Hi Marcin,
>
> Not sure if anyone asked, but are your OSD journals on actual disk or
> are you using tmpfs?
>
> Dino
>
> On Wed, Jan 16, 2013 at 4:53 AM, Wido den Hollander <wido@widodh.nl> wrote:
>>
>>
>> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>>>
>>> Hi all,
>>>
>>> Any ideas how can I resolve my issue? Or where the problem is?
>>>
>>> Let me describe the issue.
>>> Host boots up and maps RBD image with XFS filesystems
>>> Host mounts the filesystems from the RBD image
>>> Host starts to write data to the mounted filesystems
>>> Host experiences power failure
>>> Host comes up and map the RBD image
>>> Host mounts the filesystems from the RBD image
>>> All data from all filesystems is lost
>>> Host is able to use the filesystems with no problems.
>>>
>>> Filesystem is XFS, no errors on filesystem,
>>>
>>
>> That simply does not make sense to me. How can all data be gone and the FS
>> just mount cleanly.
>>
>> Can you try to format the RBD with EXT4 and see if that makes any
>> difference.
>>
>> Could you also try to run a "sync" prior to pulling the power from the host
>> to see if that makes any difference.
>>
>> Wido
>>
>>
>>> Kernel 3.5.0-19-generic
>>>
>>> root@openstack-1:/etc/init# ceph -s
>>>     health HEALTH_OK
>>>     monmap e1: 3 mons at
>>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
>>> election epoch 10, quorum 0,1,2 a,b,d
>>>     osdmap e132: 56 osds: 56 up, 56 in
>>>      pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
>>> used, 52028 GB / 52131 GB avail
>>>     mdsmap e1: 0/0/1 up
>>>
>>> Regards,
>>> Marcin
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>
> --
> ______________________________
> Dino Yancey
> 2GNT.com Admin

Re: Ceph version 0.56.1, data loss on power failure

[Marcin Szukala]

2013/1/16 Yann Dupont <Yann.Dupont@univ-nantes.fr>:
> Le 16/01/2013 11:53, Wido den Hollander a écrit :
>
>>
>>
>> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>
>
>>> Hi all,
>>>
>>> Any ideas how can I resolve my issue? Or where the problem is?
>>>
>>> Let me describe the issue.
>>> Host boots up and maps RBD image with XFS filesystems
>>> Host mounts the filesystems from the RBD image
>>> Host starts to write data to the mounted filesystems
>>> Host experiences power failure
>
> you are not doing sync there, right ?

Nope, no sync.
>
>
>>> Host comes up and map the RBD image
>>> Host mounts the filesystems from the RBD image
>>> All data from all filesystems is lost
>>> Host is able to use the filesystems with no problems.
>>>
>>> Filesystem is XFS, no errors on filesystem,
>
>
> you MAY have hit an XFS issue.
>
> Please follow XFS list, in particular this thread :
> http://oss.sgi.com/pipermail/xfs/2012-December/023021.html
>
> If i Remember well, this one is after 3.4 kernel, and I think the fix isn't
> in the current ubuntu kernel.

It looks like it, with ext4 I have no issue. Also if i do sync, the
data is not lost.

Thank You All for help.

Regards,
Marcin
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph version 0.56.1, data loss on power failure

[Jeff Mitchell]

FWIW, my ceph data dirs (for e.g. mons) are all on XFS. I've
experienced a lot of corruption on these on power loss to the node --
and in some cases even when power wasn't lost, and the box was simply
rebooted. This is on Ubuntu 12.04 with the ceph-provied 3.6.3 kernel
(as I'm using RBD on these).

It's pretty much to the point where I'm thinking of changing them all
over to ext4 for these data dirs, as the hassle of rebuilding mons
constantly is just not worth the trouble.

--Jeff

On Wed, Jan 16, 2013 at 9:32 AM, Marcin Szukala
<szukala.marcin@gmail.com> wrote:
> 2013/1/16 Yann Dupont <Yann.Dupont@univ-nantes.fr>:
>> Le 16/01/2013 11:53, Wido den Hollander a écrit :
>>
>>>
>>>
>>> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>>
>>
>>>> Hi all,
>>>>
>>>> Any ideas how can I resolve my issue? Or where the problem is?
>>>>
>>>> Let me describe the issue.
>>>> Host boots up and maps RBD image with XFS filesystems
>>>> Host mounts the filesystems from the RBD image
>>>> Host starts to write data to the mounted filesystems
>>>> Host experiences power failure
>>
>> you are not doing sync there, right ?
>
> Nope, no sync.
>>
>>
>>>> Host comes up and map the RBD image
>>>> Host mounts the filesystems from the RBD image
>>>> All data from all filesystems is lost
>>>> Host is able to use the filesystems with no problems.
>>>>
>>>> Filesystem is XFS, no errors on filesystem,
>>
>>
>> you MAY have hit an XFS issue.
>>
>> Please follow XFS list, in particular this thread :
>> http://oss.sgi.com/pipermail/xfs/2012-December/023021.html
>>
>> If i Remember well, this one is after 3.4 kernel, and I think the fix isn't
>> in the current ubuntu kernel.
>
> It looks like it, with ext4 I have no issue. Also if i do sync, the
> data is not lost.
>
> Thank You All for help.
>
> Regards,
> Marcin
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph version 0.56.1, data loss on power failure

[Sage Weil]

On Wed, 16 Jan 2013, Wido den Hollander wrote:
> 
> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
> > Hi all,
> > 
> > Any ideas how can I resolve my issue? Or where the problem is?
> > 
> > Let me describe the issue.
> > Host boots up and maps RBD image with XFS filesystems
> > Host mounts the filesystems from the RBD image
> > Host starts to write data to the mounted filesystems
> > Host experiences power failure
> > Host comes up and map the RBD image
> > Host mounts the filesystems from the RBD image
> > All data from all filesystems is lost
> > Host is able to use the filesystems with no problems.
> > 
> > Filesystem is XFS, no errors on filesystem,
> > 
> 
> That simply does not make sense to me. How can all data be gone and the FS
> just mount cleanly.
> 
> Can you try to format the RBD with EXT4 and see if that makes any difference.
> 
> Could you also try to run a "sync" prior to pulling the power from the host to
> see if that makes any difference.

A few other quick questions:

What version of qemu and librbd are you using?  What is the command line 
that is used to start the VM?  This could be a problem with the qemu 
and librbd caching configuration.

Thanks!
sage


> 
> Wido
> 
> > Kernel 3.5.0-19-generic
> > 
> > root@openstack-1:/etc/init# ceph -s
> >     health HEALTH_OK
> >     monmap e1: 3 mons at
> > {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
> > election epoch 10, quorum 0,1,2 a,b,d
> >     osdmap e132: 56 osds: 56 up, 56 in
> >      pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
> > used, 52028 GB / 52131 GB avail
> >     mdsmap e1: 0/0/1 up
> > 
> > Regards,
> > Marcin
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Ceph version 0.56.1, data loss on power failure

[Wido den Hollander]

Op 16 jan. 2013 om 18:00 heeft Sage Weil <sage@inktank.com> het volgende geschreven:

> On Wed, 16 Jan 2013, Wido den Hollander wrote:
>> 
>> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>>> Hi all,
>>> 
>>> Any ideas how can I resolve my issue? Or where the problem is?
>>> 
>>> Let me describe the issue.
>>> Host boots up and maps RBD image with XFS filesystems
>>> Host mounts the filesystems from the RBD image
>>> Host starts to write data to the mounted filesystems
>>> Host experiences power failure
>>> Host comes up and map the RBD image
>>> Host mounts the filesystems from the RBD image
>>> All data from all filesystems is lost
>>> Host is able to use the filesystems with no problems.
>>> 
>>> Filesystem is XFS, no errors on filesystem,
>> 
>> That simply does not make sense to me. How can all data be gone and the FS
>> just mount cleanly.
>> 
>> Can you try to format the RBD with EXT4 and see if that makes any difference.
>> 
>> Could you also try to run a "sync" prior to pulling the power from the host to
>> see if that makes any difference.
> 
> A few other quick questions:
> 
> What version of qemu and librbd are you using?  What is the command line 
> that is used to start the VM?  This could be a problem with the qemu 
> and librbd caching configuration.
> 

I don't think he uses Qemu. From what I understand he uses kernel RBD since he uses the words 'map' and 'unmap'

Wido

> Thanks!
> sage
> 
> 
>> 
>> Wido
>> 
>>> Kernel 3.5.0-19-generic
>>> 
>>> root@openstack-1:/etc/init# ceph -s
>>>    health HEALTH_OK
>>>    monmap e1: 3 mons at
>>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
>>> election epoch 10, quorum 0,1,2 a,b,d
>>>    osdmap e132: 56 osds: 56 up, 56 in
>>>     pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
>>> used, 52028 GB / 52131 GB avail
>>>    mdsmap e1: 0/0/1 up
>>> 
>>> Regards,
>>> Marcin
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Ceph version 0.56.1, data loss on power failure

[Sage Weil]

On Wed, 16 Jan 2013, Wido den Hollander wrote:
> 
> Op 16 jan. 2013 om 18:00 heeft Sage Weil <sage@inktank.com> het volgende geschreven:
> 
> > On Wed, 16 Jan 2013, Wido den Hollander wrote:
> >> 
> >> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
> >>> Hi all,
> >>> 
> >>> Any ideas how can I resolve my issue? Or where the problem is?
> >>> 
> >>> Let me describe the issue.
> >>> Host boots up and maps RBD image with XFS filesystems
> >>> Host mounts the filesystems from the RBD image
> >>> Host starts to write data to the mounted filesystems
> >>> Host experiences power failure
> >>> Host comes up and map the RBD image
> >>> Host mounts the filesystems from the RBD image
> >>> All data from all filesystems is lost
> >>> Host is able to use the filesystems with no problems.
> >>> 
> >>> Filesystem is XFS, no errors on filesystem,
> >> 
> >> That simply does not make sense to me. How can all data be gone and the FS
> >> just mount cleanly.
> >> 
> >> Can you try to format the RBD with EXT4 and see if that makes any difference.
> >> 
> >> Could you also try to run a "sync" prior to pulling the power from the host to
> >> see if that makes any difference.
> > 
> > A few other quick questions:
> > 
> > What version of qemu and librbd are you using?  What is the command line 
> > that is used to start the VM?  This could be a problem with the qemu 
> > and librbd caching configuration.
> > 
> 
> I don't think he uses Qemu. From what I understand he uses kernel RBD 
> since he uses the words 'map' and 'unmap'

That's what I originally thought too, and then I saw

> >>> root@openstack-1:/etc/init# ceph -s

and wasn't sure...

Marcin?

sage





> >>>    health HEALTH_OK
> >>>    monmap e1: 3 mons at
> >>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
> >>> election epoch 10, quorum 0,1,2 a,b,d
> >>>    osdmap e132: 56 osds: 56 up, 56 in
> >>>     pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
> >>> used, 52028 GB / 52131 GB avail
> >>>    mdsmap e1: 0/0/1 up
> >>> 
> >>> Regards,
> >>> Marcin
> >>> --
> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >>> the body of a message to majordomo@vger.kernel.org
> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Ceph version 0.56.1, data loss on power failure

[Tomasz Paszkowski]

Marcin is using kernel RBD.


On Wed, Jan 16, 2013 at 7:10 PM, Sage Weil <sage@inktank.com> wrote:
> On Wed, 16 Jan 2013, Wido den Hollander wrote:
>>
>> Op 16 jan. 2013 om 18:00 heeft Sage Weil <sage@inktank.com> het volgende geschreven:
>>
>> > On Wed, 16 Jan 2013, Wido den Hollander wrote:
>> >>
>> >> On 01/16/2013 11:50 AM, Marcin Szukala wrote:
>> >>> Hi all,
>> >>>
>> >>> Any ideas how can I resolve my issue? Or where the problem is?
>> >>>
>> >>> Let me describe the issue.
>> >>> Host boots up and maps RBD image with XFS filesystems
>> >>> Host mounts the filesystems from the RBD image
>> >>> Host starts to write data to the mounted filesystems
>> >>> Host experiences power failure
>> >>> Host comes up and map the RBD image
>> >>> Host mounts the filesystems from the RBD image
>> >>> All data from all filesystems is lost
>> >>> Host is able to use the filesystems with no problems.
>> >>>
>> >>> Filesystem is XFS, no errors on filesystem,
>> >>
>> >> That simply does not make sense to me. How can all data be gone and the FS
>> >> just mount cleanly.
>> >>
>> >> Can you try to format the RBD with EXT4 and see if that makes any difference.
>> >>
>> >> Could you also try to run a "sync" prior to pulling the power from the host to
>> >> see if that makes any difference.
>> >
>> > A few other quick questions:
>> >
>> > What version of qemu and librbd are you using?  What is the command line
>> > that is used to start the VM?  This could be a problem with the qemu
>> > and librbd caching configuration.
>> >
>>
>> I don't think he uses Qemu. From what I understand he uses kernel RBD
>> since he uses the words 'map' and 'unmap'
>
> That's what I originally thought too, and then I saw
>
>> >>> root@openstack-1:/etc/init# ceph -s
>
> and wasn't sure...
>
> Marcin?
>
> sage
>
>
>
>
>
>> >>>    health HEALTH_OK
>> >>>    monmap e1: 3 mons at
>> >>> {a=10.3.82.102:6789/0,b=10.3.82.103:6789/0,d=10.3.82.105:6789/0},
>> >>> election epoch 10, quorum 0,1,2 a,b,d
>> >>>    osdmap e132: 56 osds: 56 up, 56 in
>> >>>     pgmap v87165: 13744 pgs: 13744 active+clean; 52727 MB data, 102 GB
>> >>> used, 52028 GB / 52131 GB avail
>> >>>    mdsmap e1: 0/0/1 up
>> >>>
>> >>> Regards,
>> >>> Marcin
>> >>> --
>> >>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >>> the body of a message to majordomo@vger.kernel.org
>> >>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> --
>> >> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> >> the body of a message to majordomo@vger.kernel.org
>> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> > the body of a message to majordomo@vger.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Tomasz Paszkowski
SS7, Asterisk, SAN, Datacenter, Cloud Computing
+48500166299

Re: Ceph version 0.56.1, data loss on power failure

[Yann Dupont]

Le 16/01/2013 17:56, Jeff Mitchell a écrit :
> FWIW, my ceph data dirs (for e.g. mons) are all on XFS. I've
> experienced a lot of corruption on these on power loss to the node --
> and in some cases even when power wasn't lost, and the box was simply
> rebooted. This is on Ubuntu 12.04 with the ceph-provied 3.6.3 kernel
> (as I'm using RBD on these).
>
> It's pretty much to the point where I'm thinking of changing them all
> over to ext4 for these data dirs, as the hassle of rebuilding mons
> constantly is just not worth the trouble.
In october, I've lost a complete ceph cluster, because of a combination of
a memory management bug in kernel 3.6 + a bug in XFS (another BUG) (I 
Had 12 Nodes, replication was at 2, 5/6 machines were crashed in a row, 
because of mm bug, and 2 ended with unrecoverable corruption)

so, 150 TB of data on the cluster were unrecoverable. Hopefully it was 
only test data.

if you want the gory details see here :

http://oss.sgi.com/archives/xfs/2012-10/msg00420.html

This XFS bug was corrected in 3.0.52, 3.2.34,3.4.19,3.6.7. Dave chinner 
was very quick to fix the problem.

Add the last bug, (journal not flushed properly), not yet fixed on 
latest kernels.... I can understand your reaction...

But, believe it or not, I'm still confident with XFS. I've been using it 
for more than 10 years on TB and TB of data, and apart those recents 
problems , XFS have been extremely good (stability, performance, crash 
tolerance) all this time.

Not saying ext4 isn't good, but if you follow kernel developpement, 
you'll see that it's not bug-free either...

And not speaking of btrfs which was totally unstable with ceph on my 
last tries (6 month ago)

In fact, ceph is hammering hardware strongly, so it's very good to find 
bugs in linux kernel :)


So, for the moment, i'm sticking with 3.4.25 kernel. Longterm kernel, 
proven, stable : no mm problems, no xfs problems.


Cheers,

-- 
Yann Dupont - Service IRTS, DSI Université de Nantes
Tel : 02.53.48.49.20 - Mail/Jabber : Yann.Dupont@univ-nantes.fr

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html