Determining RBD storage utilization

Determining RBD storage utilization

[Travis Rhoden]

Hey folks,

I have two questions about determining how much storage has been used
*inside* of an RBD.

First, I'm confused by the output of df.  I've created, mapped, and
mounted a 500GB RBD, and see the following:

# df -h /srv/test
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd44      500G  7.5G  467G   2% /srv/test

# cd /srv/test
# du -sh .
20K	.


Any ideas way a brand-new, no files added mount shows 7.5GB of used
space?  Does this happen from the file system formatting (ext4 in this
case)?
Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
the huge discrepancy?  I don't expect the numbers to add up exact due
to rounding from kB, MB, GB, etc, but they should be darn close, a la

df -h /dev/sda1
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        15G  1.7G   13G  12% /


Second question, is it possible to know how much storage has been used
in the RBD without mounting it and running df or du?  For the same RBD
as above, I see:

# rbd info test
rbd image 'test'':
	size 500 GB in 128000 objects
	order 22 (4096 KB objects)
	block_name_prefix: rb.0.18f9.2d9c66c6
	parent:  (pool -1)

Is there perhaps a way to know the number of objects that have been
'used'?  Then I could take that and multiply by the object size (4MB).

I'm running 0.48.1argonaut on Ubuntu 12.04.
RBD maps are also on Ubuntu 12.04, with the stock 3.2.0-29-generic kernel.

Thanks,

 - Travis

Re: Determining RBD storage utilization

[Damien Churchill]

On 10 October 2012 18:10, Travis Rhoden <trhoden@gmail.com> wrote:
> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
> the huge discrepancy?  I don't expect the numbers to add up exact due
> to rounding from kB, MB, GB, etc, but they should be darn close, a la

ext4 keeps some reserved space, 5% by default, for when the disk is
full so you are still able to use the filesystem and clean it up.

500G * 0.05 = 25G
500G - (25G + 7.5G) = 467G

Can't tell you where the 7.5G comes from though!

Re: Determining RBD storage utilization

[Travis Rhoden]

Damien,

Thanks for solving that part of the mystery.  I can't believe I forgot
about that.  Thanks for the reminder and the clear explanation.

 - Travis

On Wed, Oct 10, 2012 at 1:28 PM, Damien Churchill <damoxc@gmail.com> wrote:
> On 10 October 2012 18:10, Travis Rhoden <trhoden@gmail.com> wrote:
>> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
>> the huge discrepancy?  I don't expect the numbers to add up exact due
>> to rounding from kB, MB, GB, etc, but they should be darn close, a la
>
> ext4 keeps some reserved space, 5% by default, for when the disk is
> full so you are still able to use the filesystem and clean it up.
>
> 500G * 0.05 = 25G
> 500G - (25G + 7.5G) = 467G
>
> Can't tell you where the 7.5G comes from though!

Re: Determining RBD storage utilization

[Josh Durgin]

On 10/10/2012 10:10 AM, Travis Rhoden wrote:
> Hey folks,
>
> I have two questions about determining how much storage has been used
> *inside* of an RBD.
>
> First, I'm confused by the output of df.  I've created, mapped, and
> mounted a 500GB RBD, and see the following:
>
> # df -h /srv/test
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/rbd44      500G  7.5G  467G   2% /srv/test
>
> # cd /srv/test
> # du -sh .
> 20K	.
>
>
> Any ideas way a brand-new, no files added mount shows 7.5GB of used
> space?  Does this happen from the file system formatting (ext4 in this
> case)?
> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
> the huge discrepancy?  I don't expect the numbers to add up exact due
> to rounding from kB, MB, GB, etc, but they should be darn close, a la
>
> df -h /dev/sda1
> Filesystem      Size  Used Avail Use% Mounted on
> /dev/sda1        15G  1.7G   13G  12% /
>
>
> Second question, is it possible to know how much storage has been used
> in the RBD without mounting it and running df or du?  For the same RBD
> as above, I see:
>
> # rbd info test
> rbd image 'test'':
> 	size 500 GB in 128000 objects
> 	order 22 (4096 KB objects)
> 	block_name_prefix: rb.0.18f9.2d9c66c6
> 	parent:  (pool -1)
>
> Is there perhaps a way to know the number of objects that have been
> 'used'?  Then I could take that and multiply by the object size (4MB).

You can get an upper bound by looking at the number of objects in the
image:

rados --pool rbd ls | grep -c '^rb\.0\.18f9\.2d9c66c6'

Each object represents a section of the block device, but they may not
be entirely filled (objects are sparse), so this will probably still be
a higher estimate than df. Also note that listing all the objects in a
pool is an expensive operation, so it shouldn't be done very often.

Josh

> I'm running 0.48.1argonaut on Ubuntu 12.04.
> RBD maps are also on Ubuntu 12.04, with the stock 3.2.0-29-generic kernel.
>
> Thanks,
>
>   - Travis

Re: Determining RBD storage utilization

[Gregory Farnum]

I don't know the ext4 internals at all, but filesystems tend to
require allocation tables of various sorts (for managing extents,
etc). 7.5GB out of 500GB seems a little large for that metadata, but
isn't ridiculously so...

On Wed, Oct 10, 2012 at 10:28 AM, Damien Churchill <damoxc@gmail.com> wrote:
> On 10 October 2012 18:10, Travis Rhoden <trhoden@gmail.com> wrote:
>> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
>> the huge discrepancy?  I don't expect the numbers to add up exact due
>> to rounding from kB, MB, GB, etc, but they should be darn close, a la
>
> ext4 keeps some reserved space, 5% by default, for when the disk is
> full so you are still able to use the filesystem and clean it up.
>
> 500G * 0.05 = 25G
> 500G - (25G + 7.5G) = 467G
>
> Can't tell you where the 7.5G comes from though!
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Determining RBD storage utilization

[Travis Rhoden]

Thanks for the input, Gregory and Josh.

What I am hearing is that this has everything to do with the
filesystem, and nothing to do with the block device on Ceph.

Thanks again,

 - Travis

On Wed, Oct 10, 2012 at 1:55 PM, Gregory Farnum <greg@inktank.com> wrote:
> I don't know the ext4 internals at all, but filesystems tend to
> require allocation tables of various sorts (for managing extents,
> etc). 7.5GB out of 500GB seems a little large for that metadata, but
> isn't ridiculously so...
>
> On Wed, Oct 10, 2012 at 10:28 AM, Damien Churchill <damoxc@gmail.com> wrote:
>> On 10 October 2012 18:10, Travis Rhoden <trhoden@gmail.com> wrote:
>>> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
>>> the huge discrepancy?  I don't expect the numbers to add up exact due
>>> to rounding from kB, MB, GB, etc, but they should be darn close, a la
>>
>> ext4 keeps some reserved space, 5% by default, for when the disk is
>> full so you are still able to use the filesystem and clean it up.
>>
>> 500G * 0.05 = 25G
>> 500G - (25G + 7.5G) = 467G
>>
>> Can't tell you where the 7.5G comes from though!
>> --
>> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Determining RBD storage utilization

[Sage Weil]

On Wed, 10 Oct 2012, Gregory Farnum wrote:
> I don't know the ext4 internals at all, but filesystems tend to
> require allocation tables of various sorts (for managing extents,
> etc). 7.5GB out of 500GB seems a little large for that metadata, but
> isn't ridiculously so...

ext3/4 are particularly bad about this, with lots of space statically set 
aside for inodes and allocation metadata.

s


> 
> On Wed, Oct 10, 2012 at 10:28 AM, Damien Churchill <damoxc@gmail.com> wrote:
> > On 10 October 2012 18:10, Travis Rhoden <trhoden@gmail.com> wrote:
> >> Additionally, 500G - 7.5G != 467G (the number shown as Avail).  Why
> >> the huge discrepancy?  I don't expect the numbers to add up exact due
> >> to rounding from kB, MB, GB, etc, but they should be darn close, a la
> >
> > ext4 keeps some reserved space, 5% by default, for when the disk is
> > full so you are still able to use the filesystem and clean it up.
> >
> > 500G * 0.05 = 25G
> > 500G - (25G + 7.5G) = 467G
> >
> > Can't tell you where the 7.5G comes from though!
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Determining RBD storage utilization

[Damien Churchill]

On 10 October 2012 18:55, Gregory Farnum <greg@inktank.com> wrote:
> I don't know the ext4 internals at all, but filesystems tend to
> require allocation tables of various sorts (for managing extents,
> etc). 7.5GB out of 500GB seems a little large for that metadata, but
> isn't ridiculously so...
>

Just for comparisons sake, Just formatted a spare 500G disk (-8GB swap
partition at the start) as ext4:

$ df -h /dev/sdc2
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc2       451G  198M  428G   1% /mnt