About sparse images

About sparse images

[Sébastien Han]

Hi all!

I know that ceph images (rbd) are sparse. It's good but I have some
interrogations about the ceph -s (-w) output. Let's say your create a
10G image, map it, format it and mount it. Finally you filled up the
entire space available. You notice that the 'data' and 'used' space
had grown. Now delete the content of your image but **not** this
image, not now. You notice that nothing changed from the ceph output,
it's not surprising because of the sparse capability. At this point of
time  the output from the command ceph is neither right nor wrong. It
only shows the space used by the sparse image, but it doesn't mean
that you have 'real' data in it, just a simple object allocation. It's
not so relevant, the only solution is to remove the image (obviously).
My feeling is that we should be careful about the ceph's output, it's
not a good indicator for 'real' data. Ok Ceph won't replace a
monitoring system but I'll be happy to trust the ceph output at 100%
:)

I would appreciate some reactions about that :)

Thank you in advance.

Cheers!

Re: About sparse images

[Smart Weblications GmbH]

Am 26.08.2012 20:11, schrieb Sébastien Han:
> Hi all!
> 
> I know that ceph images (rbd) are sparse. It's good but I have some
> interrogations about the ceph -s (-w) output. Let's say your create a
> 10G image, map it, format it and mount it. Finally you filled up the
> entire space available. You notice that the 'data' and 'used' space
> had grown. Now delete the content of your image but **not** this
> image, not now. You notice that nothing changed from the ceph output,
> it's not surprising because of the sparse capability. At this point of
> time  the output from the command ceph is neither right nor wrong. It
> only shows the space used by the sparse image, but it doesn't mean
> that you have 'real' data in it, just a simple object allocation. It's
> not so relevant, the only solution is to remove the image (obviously).
> My feeling is that we should be careful about the ceph's output, it's
> not a good indicator for 'real' data. Ok Ceph won't replace a
> monitoring system but I'll be happy to trust the ceph output at 100%
> :)
> 
> I would appreciate some reactions about that :)
> 


why not gather real image size with rbd info <imagename>, sum up used space,
then look at ceph -s to get total avail space?



-- 

Mit freundlichen Grüßen,


Smart Weblications GmbH
Martinsberger Str. 1
D-95119 Naila

fon.: +49 9282 9638 200
fax.: +49 9282 9638 205
24/7: +49 900 144 000 00 - 0,99 EUR/Min*
http://www.smart-weblications.de

--
Sitz der Gesellschaft: Naila
Geschäftsführer: Florian Wiessner
HRB-Nr.: HRB 3840 Amtsgericht Hof
*aus dem dt. Festnetz, ggf. abweichende Preise aus dem Mobilfunknetz
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: About sparse images

[Gregory Farnum]

On Sunday, August 26, 2012 at 11:11 AM, Sébastien Han wrote:
> Hi all!
>  
> I know that ceph images (rbd) are sparse. It's good but I have some
> interrogations about the ceph -s (-w) output. Let's say your create a
> 10G image, map it, format it and mount it. Finally you filled up the
> entire space available. You notice that the 'data' and 'used' space
> had grown. Now delete the content of your image but **not** this
> image, not now. You notice that nothing changed from the ceph output,
> it's not surprising because of the sparse capability. At this point of
> time the output from the command ceph is neither right nor wrong. It
> only shows the space used by the sparse image, but it doesn't mean
> that you have 'real' data in it, just a simple object allocation. It's
> not so relevant, the only solution is to remove the image (obviously).
> My feeling is that we should be careful about the ceph's output, it's
> not a good indicator for 'real' data. Ok Ceph won't replace a
> monitoring system but I'll be happy to trust the ceph output at 100%
> :)

The output of ceph -s is intended to describe the amount of space allocated by RADOS across the OSDs. As far as any part of the Ceph project knows, the objects in those RBD images are real data and can't be removed.
If you're interested in removing them, subsets of the RBD clients do support TRIM operations, which will remove the objects themselves. Josh or somebody will have to speak about which clients, though.
-Greg

--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: About sparse images

[Wido den Hollander]

On 08/26/2012 09:42 PM, Gregory Farnum wrote:
> On Sunday, August 26, 2012 at 11:11 AM, Sébastien Han wrote:
>> Hi all!
>>
>> I know that ceph images (rbd) are sparse. It's good but I have some
>> interrogations about the ceph -s (-w) output. Let's say your create a
>> 10G image, map it, format it and mount it. Finally you filled up the
>> entire space available. You notice that the 'data' and 'used' space
>> had grown. Now delete the content of your image but **not** this
>> image, not now. You notice that nothing changed from the ceph output,
>> it's not surprising because of the sparse capability. At this point of
>> time the output from the command ceph is neither right nor wrong. It
>> only shows the space used by the sparse image, but it doesn't mean
>> that you have 'real' data in it, just a simple object allocation. It's
>> not so relevant, the only solution is to remove the image (obviously).
>> My feeling is that we should be careful about the ceph's output, it's
>> not a good indicator for 'real' data. Ok Ceph won't replace a
>> monitoring system but I'll be happy to trust the ceph output at 100%
>> :)
>
> The output of ceph -s is intended to describe the amount of space allocated by RADOS across the OSDs. As far as any part of the Ceph project knows, the objects in those RBD images are real data and can't be removed.
> If you're interested in removing them, subsets of the RBD clients do support TRIM operations, which will remove the objects themselves. Josh or somebody will have to speak about which clients, though.
> -Greg
>

As far as I know TRIM / Discard only works with KVM and SCSI disks, 
VirtIO still seems to lack this functionality, not sure about kernel RBD.

On the filesystem side, ext4 with the "discard" mount option does this.

But it's important (like Greg already said) to note that RBD/RADOS can't 
know which object is still important and which isn't. When you unlink a 
file the filesystem doesn't zero that particular block, it just clears 
all references to it so the freed up space can be used again.

Wido

> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: About sparse images

[Josh Durgin]

On 08/26/2012 12:53 PM, Wido den Hollander wrote:
> As far as I know TRIM / Discard only works with KVM and SCSI disks,
> VirtIO still seems to lack this functionality, not sure about kernel RBD.

Kernel rbd doesn't support TRIM/discard yet 
(http://www.tracker.newdream.net/issues/190).