Forever growing data in ceph using RBD image

Forever growing data in ceph using RBD image

[Alphe Salas]

Hello,
I would like to know if there is something planned to correct the 
"forever growing" effet when using rbd image.
My experience shows that the replicas of a rbd images are never 
discarded and never overwriten. Lets say my physical share is about 30 
TB I make an image of 13TB (half the real space - 25% of disfunction osd 
support). My experience shows that the rbd image is overwriten so if I 
top the 13TB once i get a 26TB of real space used (replicas set to 2) if 
I delete 8TB from those 13TB I see the real space used unchanged.
If I write back 4TB then ceph collapse it is nearfull and I have to go 
buy another 30TB integrate it to my cluster to hold the problem. But 
still soon I have in my ceph more useless replicas of "delete" datas 
than usefull data with they replicas.

Usually when I talk to dev team about this problem they tell me that the 
real problem is the lack of trim in XFS, but my own analysis shows that 
the real problem is ceph internal way to handle data. It is ceph that 
never discard any replicas and never "clean" itself to only keep records 
of the data in use.

If ceph was behaving properly then for a replicas set to 2 I would have 
my rbd image of 13 TB the 13TB replicas corresponding, and a fix 26TB
of overall used data. When I would "free" data in the rbd image the 
corresponding replicas would be considered as discarded by ceph and when 
the real data in the rbd image is overwriten their corresponding 
replicas would be overwriten too with the new data. That would show the 
overall data space used as fixed.

In case of the failure of 2 osd then the ceph system would have just 
enough space to clone the replicas of the missing data which actually is
not the case in a environement that reach near full state.



So my ask is what is planned to correct this problem?

Best regards,

-- 
Alphe Salas
I.T ingeneer

Re: Forever growing data in ceph using RBD image

[Sage Weil]

On Thu, 17 Jul 2014, Alphe Salas wrote:
> Hello,
> I would like to know if there is something planned to correct the "forever
> growing" effet when using rbd image.
> My experience shows that the replicas of a rbd images are never discarded and
> never overwriten. Lets say my physical share is about 30 TB I make an image of
> 13TB (half the real space - 25% of disfunction osd support). My experience
> shows that the rbd image is overwriten so if I top the 13TB once i get a 26TB
> of real space used (replicas set to 2) if I delete 8TB from those 13TB I see
> the real space used unchanged.
> If I write back 4TB then ceph collapse it is nearfull and I have to go buy
> another 30TB integrate it to my cluster to hold the problem. But still soon I
> have in my ceph more useless replicas of "delete" datas than usefull data with
> they replicas.
> 
> Usually when I talk to dev team about this problem they tell me that the 
> real problem is the lack of trim in XFS, but my own analysis shows that 
> the real problem is ceph internal way to handle data. It is ceph that 
> never discard any replicas and never "clean" itself to only keep records 
> of the data in use.

You are correct that if XFS (or whatever FS you are using) does not issue 
discard/trim, then deleting data inside the fs on top of RBD won't free 
any space.  Note that you usually have to explicitly enable this via a 
mount option; most (all?) kernels still leave this off by default.

Are you taking RBD snapshots?  If not, then there will never be more than 
the rbd image size * num_replicas space used (ignoring the few % of file 
system overhead for the moment).

If you are taking snapshots, then yes.. you will see more space used until 
the snapshot is deleted because we will keep old copies of objects around.

> If ceph was behaving properly then for a replicas set to 2 I would have 
> my rbd image of 13 TB the 13TB replicas corresponding, and a fix 26TB of 
> overall used data. When I would "free" data in the rbd image the 
> corresponding replicas would be considered as discarded by ceph and when 
> the real data in the rbd image is overwriten their corresponding 
> replicas would be overwriten too with the new data. That would show the 
> overall data space used as fixed.

Both ceph *and* the file system on top of RBD have to be "behaving 
properly".  RBD can't free space until it is told to do so by the file 
system, and by default, most/all do not...

sage

Re: Forever growing data in ceph using RBD image

[Christoph Hellwig]

On Thu, Jul 17, 2014 at 10:11:09AM -0400, Alphe Salas wrote:
> Usually when I talk to dev team about this problem they tell me that the
> real problem is the lack of trim in XFS, but my own analysis shows that the
> real problem is ceph internal way to handle data. It is ceph that never
> discard any replicas and never "clean" itself to only keep records of the
> data in use.

XFS supports TRIM just fine - with -o discard it will issue TRIMs on
unlink, and with fstrim you can do it explicitly.  Note that the online
discard might be very slow for all Linux filesystems, although I've not
actually tested it on top of RBD.

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

Alphe Salas
I.T ingeneer

On 07/17/2014 12:35 PM, Sage Weil wrote:
> On Thu, 17 Jul 2014, Alphe Salas wrote:
>> Hello,
>> I would like to know if there is something planned to correct the "forever
>> growing" effet when using rbd image.
>> My experience shows that the replicas of a rbd images are never discarded and
>> never overwriten. Lets say my physical share is about 30 TB I make an image of
>> 13TB (half the real space - 25% of disfunction osd support). My experience
>> shows that the rbd image is overwriten so if I top the 13TB once i get a 26TB
>> of real space used (replicas set to 2) if I delete 8TB from those 13TB I see
>> the real space used unchanged.
>> If I write back 4TB then ceph collapse it is nearfull and I have to go buy
>> another 30TB integrate it to my cluster to hold the problem. But still soon I
>> have in my ceph more useless replicas of "delete" datas than usefull data with
>> they replicas.
>>
>> Usually when I talk to dev team about this problem they tell me that the
>> real problem is the lack of trim in XFS, but my own analysis shows that
>> the real problem is ceph internal way to handle data. It is ceph that
>> never discard any replicas and never "clean" itself to only keep records
>> of the data in use.

>
> You are correct that if XFS (or whatever FS you are using) does not issue
> discard/trim, then deleting data inside the fs on top of RBD won't free
> any space.  Note that you usually have to explicitly enable this via a
> mount option; most (all?) kernels still leave this off by default.
>
> Are you taking RBD snapshots?  If not, then there will never be more than
> the rbd image size * num_replicas space used (ignoring the few % of file
> system overhead for the moment).
>
> If you are taking snapshots, then yes.. you will see more space used until
> the snapshot is deleted because we will keep old copies of objects around.

I am not using snapshot. I dont have enought space to write to the disk 
after some round of write / delete /write / delete so I can t affort to 
use fancy features like snapshots. I use regular image rbd type 1  not 
even able to be snapshoot.

I tryed to activate XFS trim system but that shown no change at all. 
(discard mount option just have no real effect try in ubuntu 14.04)

Like I said what seems to grow in fact are the replica side of the data.
There is no overwriting of the replicas when real data are overwriten so 
slowly I see the real disk weight of my datas in the ceph cluster grow, 
grow, grow and never come to a stable size.



>
>> If ceph was behaving properly then for a replicas set to 2 I would have
>> my rbd image of 13 TB the 13TB replicas corresponding, and a fix 26TB of
>> overall used data. When I would "free" data in the rbd image the
>> corresponding replicas would be considered as discarded by ceph and when
>> the real data in the rbd image is overwriten their corresponding
>> replicas would be overwriten too with the new data. That would show the
>> overall data space used as fixed.
>
> Both ceph *and* the file system on top of RBD have to be "behaving
> properly".  RBD can't free space until it is told to do so by the file
> system, and by default, most/all do not...
>
> sage
>
There is the trick which layer of XFS are we talking about the layer 
inside the rbd image ? or the one below the RBD image ?
I already see a bug ticket from 2009 in ceph bug track that state that
XFS trim is not taken in consideration by ceph. That ticket doesn t seem 
to have got a solution.

and if I have XFS as format on the low end Ceph cluster and ext4 in the 
rbd image how will trim works?

Low level XFS (of the osd disks ) have mount options that are not 
managed by the user it is auto process of mount when the osd is 
activated in that consideration how do I activate the trim ? Do I have 
to put the hands on udev level scripts ?

Thank you for your reply I really want to find a solution, maybe it is 
some level of wrong understanding of how ceph works and should be set 
and I am open to test any suggestions on that topic.

Best regards

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

Alphe Salas
I.T ingeneer

On 07/17/2014 12:47 PM, Christoph Hellwig wrote:
> On Thu, Jul 17, 2014 at 10:11:09AM -0400, Alphe Salas wrote:
>> Usually when I talk to dev team about this problem they tell me that the
>> real problem is the lack of trim in XFS, but my own analysis shows that the
>> real problem is ceph internal way to handle data. It is ceph that never
>> discard any replicas and never "clean" itself to only keep records of the
>> data in use.
>
> XFS supports TRIM just fine - with -o discard it will issue TRIMs on
> unlink, and with fstrim you can do it explicitly.  Note that the online
> discard might be very slow for all Linux filesystems, although I've not
> actually tested it on top of RBD.
>
I will test it again probably the point I missed it that you have to 
unmount / remount the XFS RBD image in order to see the data disapear 
from the real disk space old in ceph. I will try it today and comeback 
with copy /paste of ceph -s etc..


> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Forever growing data in ceph using RBD image

[Sage Weil]

On Thu, 17 Jul 2014, Alphe Salas wrote:
> On 07/17/2014 12:35 PM, Sage Weil wrote:
> > On Thu, 17 Jul 2014, Alphe Salas wrote:
> > > Hello,
> > > I would like to know if there is something planned to correct the "forever
> > > growing" effet when using rbd image.
> > > My experience shows that the replicas of a rbd images are never discarded
> > > and
> > > never overwriten. Lets say my physical share is about 30 TB I make an
> > > image of
> > > 13TB (half the real space - 25% of disfunction osd support). My experience
> > > shows that the rbd image is overwriten so if I top the 13TB once i get a
> > > 26TB
> > > of real space used (replicas set to 2) if I delete 8TB from those 13TB I
> > > see
> > > the real space used unchanged.
> > > If I write back 4TB then ceph collapse it is nearfull and I have to go buy
> > > another 30TB integrate it to my cluster to hold the problem. But still
> > > soon I
> > > have in my ceph more useless replicas of "delete" datas than usefull data
> > > with
> > > they replicas.
> > > 
> > > Usually when I talk to dev team about this problem they tell me that the
> > > real problem is the lack of trim in XFS, but my own analysis shows that
> > > the real problem is ceph internal way to handle data. It is ceph that
> > > never discard any replicas and never "clean" itself to only keep records
> > > of the data in use.
> 
> > 
> > You are correct that if XFS (or whatever FS you are using) does not issue
> > discard/trim, then deleting data inside the fs on top of RBD won't free
> > any space.  Note that you usually have to explicitly enable this via a
> > mount option; most (all?) kernels still leave this off by default.
> > 
> > Are you taking RBD snapshots?  If not, then there will never be more than
> > the rbd image size * num_replicas space used (ignoring the few % of file
> > system overhead for the moment).
> > 
> > If you are taking snapshots, then yes.. you will see more space used until
> > the snapshot is deleted because we will keep old copies of objects around.
> 
> I am not using snapshot. I dont have enought space to write to the disk after
> some round of write / delete /write / delete so I can t affort to use fancy
> features like snapshots. I use regular image rbd type 1  not even able to be
> snapshoot.
> 
> I tryed to activate XFS trim system but that shown no change at all. (discard
> mount option just have no real effect try in ubuntu 14.04)

I believe you have to have mounted with -o discard at the time the data is 
deleted; simply enabling the option later won't help.  This is what 
the fstrim utility is for; see

	http://man7.org/linux/man-pages/man8/fstrim.8.html

> Like I said what seems to grow in fact are the replica side of the data.
> There is no overwriting of the replicas when real data are overwriten so
> slowly I see the real disk weight of my datas in the ceph cluster grow, grow,
> grow and never come to a stable size.

This is simply not true.  RADOS object are overwritten in place.  If you 
create a 10 TB image and write it 100x with dd, you will still only 
consume 10 TB * num_replicas.  If you are seeing something other 
than this, ignore everything else in this email and go figure out what 
else is writing files to the underlying volumes.

> There is the trick which layer of XFS are we talking about the layer inside
> the rbd image ? or the one below the RBD image ?
>
> I already see a bug ticket from 2009 in ceph bug track that state that
> XFS trim is not taken in consideration by ceph. That ticket doesn t seem to
> have got a solution.
> 
> and if I have XFS as format on the low end Ceph cluster and ext4 in the rbd
> image how will trim works?

I assume you are using kvm/qemu?  It may be that older versions aren't 
passing through trims; Josh would know more.  Or maybe the trim sizes are 
too small to let rados effectively deallocate entire objects.  Logs might 
help there.

But, as I said, if you see more data written than the size of your image 
then stop worrying about trim and sort that out first...

> Low level XFS (of the osd disks ) have mount options that are not managed by
> the user it is auto process of mount when the osd is activated in that
> consideration how do I activate the trim ? Do I have to put the hands on udev
> level scripts ?

Trim on the underlying XFS volumes isn't necessary or important.  When RBD 
gets a discard, it will either delete, truncate, or punch holes in the 
underlying XFS object files the image maps too.

sage

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

On 07/17/2014 02:27 PM, Sage Weil wrote:
> On Thu, 17 Jul 2014, Alphe Salas wrote:
>> On 07/17/2014 12:35 PM, Sage Weil wrote:
>>> On Thu, 17 Jul 2014, Alphe Salas wrote:
>>>> Hello,
>>>> I would like to know if there is something planned to correct the "forever
>>>> growing" effet when using rbd image.
>>>> My experience shows that the replicas of a rbd images are never discarded
>>>> and
>>>> never overwriten. Lets say my physical share is about 30 TB I make an
>>>> image of
>>>> 13TB (half the real space - 25% of disfunction osd support). My experience
>>>> shows that the rbd image is overwriten so if I top the 13TB once i get a
>>>> 26TB
>>>> of real space used (replicas set to 2) if I delete 8TB from those 13TB I
>>>> see
>>>> the real space used unchanged.
>>>> If I write back 4TB then ceph collapse it is nearfull and I have to go buy
>>>> another 30TB integrate it to my cluster to hold the problem. But still
>>>> soon I
>>>> have in my ceph more useless replicas of "delete" datas than usefull data
>>>> with
>>>> they replicas.
>>>>
>>>> Usually when I talk to dev team about this problem they tell me that the
>>>> real problem is the lack of trim in XFS, but my own analysis shows that
>>>> the real problem is ceph internal way to handle data. It is ceph that
>>>> never discard any replicas and never "clean" itself to only keep records
>>>> of the data in use.
>>
>>>
>>> You are correct that if XFS (or whatever FS you are using) does not issue
>>> discard/trim, then deleting data inside the fs on top of RBD won't free
>>> any space.  Note that you usually have to explicitly enable this via a
>>> mount option; most (all?) kernels still leave this off by default.
>>>
>>> Are you taking RBD snapshots?  If not, then there will never be more than
>>> the rbd image size * num_replicas space used (ignoring the few % of file
>>> system overhead for the moment).
>>>
>>> If you are taking snapshots, then yes.. you will see more space used until
>>> the snapshot is deleted because we will keep old copies of objects around.
>>
>> I am not using snapshot. I dont have enought space to write to the disk after
>> some round of write / delete /write / delete so I can t affort to use fancy
>> features like snapshots. I use regular image rbd type 1  not even able to be
>> snapshoot.
>>
>> I tryed to activate XFS trim system but that shown no change at all. (discard
>> mount option just have no real effect try in ubuntu 14.04)
>
> I believe you have to have mounted with -o discard at the time the data is
> deleted; simply enabling the option later won't help.  This is what
> the fstrim utility is for; see
>
> 	http://man7.org/linux/man-pages/man8/fstrim.8.html
>
>> Like I said what seems to grow in fact are the replica side of the data.
>> There is no overwriting of the replicas when real data are overwriten so
>> slowly I see the real disk weight of my datas in the ceph cluster grow, grow,
>> grow and never come to a stable size.
>
> This is simply not true.  RADOS object are overwritten in place.  If you
> create a 10 TB image and write it 100x with dd, you will still only
> consume 10 TB * num_replicas.  If you are seeing something other
> than this, ignore everything else in this email and go figure out what
> else is writing files to the underlying volumes.
>
Well I know it is dificult to beleive that the data are forever growing
in ceph I was thinking like you that data will overwrite on themselves 
for ever and ever after and that was not the case the rbd image part 
with or without triming was overwriten properly.
For example in my rbd image of 13TB i write 13TB then have the 
corresponding 13TB of replicas I delete 3TB of data normally I would see 
not data groth since rbd image is overwriten and replicas too. By in 
fact ceph -s show me then an overall use of 29TB which means 3TB of data 
have been added to the pull at this point ceph state is on warning too 
full and some osd just stop to receive anymore data.

  I have a mini ceph cluster where i will reproduce that behavior and 
bring you with the full log of it (step by step commands list and results).

>> There is the trick which layer of XFS are we talking about the layer inside
>> the rbd image ? or the one below the RBD image ?
>>
>> I already see a bug ticket from 2009 in ceph bug track that state that
>> XFS trim is not taken in consideration by ceph. That ticket doesn t seem to
>> have got a solution.
>>
>> and if I have XFS as format on the low end Ceph cluster and ext4 in the rbd
>> image how will trim works?
>
> I assume you are using kvm/qemu?  It may be that older versions aren't
> passing through trims; Josh would know more.  Or maybe the trim sizes are
> too small to let rados effectively deallocate entire objects.  Logs might
> help there.
>
> But, as I said, if you see more data written than the size of your image
> then stop worrying about trim and sort that out first...
>
>> Low level XFS (of the osd disks ) have mount options that are not managed by
>> the user it is auto process of mount when the osd is activated in that
>> consideration how do I activate the trim ? Do I have to put the hands on udev
>> level scripts ?
>
> Trim on the underlying XFS volumes isn't necessary or important.  When RBD
> gets a discard, it will either delete, truncate, or punch holes in the
> underlying XFS object files the image maps too.
>
> sage
>
>

As usual thank you for dedicating time to interact with me I know you 
have a billion things doing but this is bothering me and I need to sort 
it out.


Alphe Salas
I.T ingeneer

Re: Forever growing data in ceph using RBD image

[Christoph Hellwig]

On Thu, Jul 17, 2014 at 11:27:31AM -0700, Sage Weil wrote:
> I assume you are using kvm/qemu?  It may be that older versions aren't 
> passing through trims; Josh would know more.  Or maybe the trim sizes are 
> too small to let rados effectively deallocate entire objects.  Logs might 
> help there.

At least for the qemu version from a few month ago that I'm using in my
testing I explicitly have to enable passthrough of UNMAP/TRIM on the
qemu command line.

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

I don t use QEMU /KVM just a plain regular rbd image format 1 map and 
mounted to a real machine that is used as proxy for data droping to ceph.


---

Alphe Salas
I.T ingeneer

On 07/17/2014 02:57 PM, Christoph Hellwig wrote:
> On Thu, Jul 17, 2014 at 11:27:31AM -0700, Sage Weil wrote:
>> I assume you are using kvm/qemu?  It may be that older versions aren't
>> passing through trims; Josh would know more.  Or maybe the trim sizes are
>> too small to let rados effectively deallocate entire objects.  Logs might
>> help there.
>
> At least for the qemu version from a few month ago that I'm using in my
> testing I explicitly have to enable passthrough of UNMAP/TRIM on the
> qemu command line.
>

Re: Forever growing data in ceph using RBD image

[Sage Weil]

On Thu, 17 Jul 2014, Alphe Salas wrote:
> I don t use QEMU /KVM just a plain regular rbd image format 1 map and mounted
> to a real machine that is used as proxy for data droping to ceph.

Oh, that explains part of it at least. The discard support isn't yet 
upstream for the kernel RBD driver.  The patches are mostly ready but 
there was a QA failure they're chasing down first.  Hopefully it will make 
it into the next kernel cycle.

sage

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

Thank you I though I was seeing things :)
So it will come soon can I help chasing the things that give errors ?



Alphe Salas
I.T ingeneer

On 07/17/2014 03:28 PM, Sage Weil wrote:
> On Thu, 17 Jul 2014, Alphe Salas wrote:
>> I don t use QEMU /KVM just a plain regular rbd image format 1 map and mounted
>> to a real machine that is used as proxy for data droping to ceph.
>
> Oh, that explains part of it at least. The discard support isn't yet
> upstream for the kernel RBD driver.  The patches are mostly ready but
> there was a QA failure they're chasing down first.  Hopefully it will make
> it into the next kernel cycle.
>
> sage
>

Re: Forever growing data in ceph using RBD image

[Alphe Salas]

How can I follow the state of advancement of the resolution of this 
issue ? I really need it and I really need to know when it will be 
release in order to immediatly install the new ceph version which patch
the forever growing data in ceph using RBD bold image (no QEMU etc..)




Alphe Salas
I.T ingeneer

On 07/17/2014 05:24 PM, Alphe Salas wrote:
> Thank you I though I was seeing things :)
> So it will come soon can I help chasing the things that give errors ?
>
>
>
> Alphe Salas
> I.T ingeneer
>
> On 07/17/2014 03:28 PM, Sage Weil wrote:
>> On Thu, 17 Jul 2014, Alphe Salas wrote:
>>> I don t use QEMU /KVM just a plain regular rbd image format 1 map and
>>> mounted
>>> to a real machine that is used as proxy for data droping to ceph.
>>
>> Oh, that explains part of it at least. The discard support isn't yet
>> upstream for the kernel RBD driver.  The patches are mostly ready but
>> there was a QA failure they're chasing down first.  Hopefully it will
>> make
>> it into the next kernel cycle.
>>
>> sage
>>