btrfs snapshot sizes

btrfs snapshot sizes

[Marc MERLIN]

So have others found a good way to have an idea about how much space is
taken by each snapshot?

I've tried quota trees, but I'm not sure how to read the output, or if it's
correct (including the negative numbers some have mentioned). Are there
other options?

I think the main problem is that the shared data field is not working,
making it harder to know which blocks are only used in a given snapshot.


subvol                                          group         total    unshared  
------------------------------------------------------------------------------- 
backup/debian32                                 0/262       403.84G      -5.46G  
backup/debian32_daily_20140504_00:03:01         0/3660      446.45G       0.00G  
backup/debian32_daily_20140505_00:03:01         0/3687      431.11G       0.00G  
backup/debian32_daily_20140506_00:03:00         0/3705      420.83G       0.00G  
backup/debian32_daily_20140507_00:03:01         0/3724      411.87G       0.00G  
backup/debian32_weekly_20140504_00:04:01        0/3675      446.45G       0.00G  

backup/debian64                                 0/263       855.97G      -1.50G  
backup/debian64_daily_20140504_00:03:01         0/3662      860.19G       0.00G  
backup/debian64_daily_20140505_00:03:01         0/3690      859.32G       0.00G  
backup/debian64_daily_20140506_00:03:00         0/3707      858.15G       0.00G  
backup/debian64_daily_20140507_00:03:01         0/3726      857.47G       0.00G  
backup/debian64_weekly_20140504_00:04:01        0/3676      860.19G       0.00G  

backup/ubuntu                                   0/264       360.28G       0.00G  
backup/ubuntu_daily_20140504_00:03:01           0/3664      364.53G       0.00G  
backup/ubuntu_daily_20140505_00:03:01           0/3692      362.44G       0.00G  
backup/ubuntu_daily_20140506_00:03:00           0/3709      360.91G       0.00G  
backup/ubuntu_daily_20140507_00:03:01           0/3727      360.33G       0.00G  
backup/ubuntu_weekly_20140504_00:04:01          0/3677      364.53G       0.00G  


Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs snapshot sizes

[David Pottage]

On 07/05/14 12:19, Marc MERLIN wrote:
> So have others found a good way to have an idea about how much space is
> taken by each snapshot?
>
> I've tried quota trees, but I'm not sure how to read the output, or if it's
> correct (including the negative numbers some have mentioned). Are there
> other options?
>
> I think the main problem is that the shared data field is not working,
> making it harder to know which blocks are only used in a given snapshot.

In my understanding (devs please correct me if I am wrong), a snapshot 
is just a subvolume that happens to share a lot of data with another 
subvolume. The idea of taking regular snapshots to preserve the state of 
the filing system at a point in time is a userland concept. From the 
kernel's point of view the user has asked for a clone of a subvolume, 
and both copies are equal. What the user does with one or other clone 
after that is their affair.

This means that suppose you have a subvolume representing your home 
directory that contains around 1Gb of data, and then take daily 
snapshots, asking the kernel how big each snapshot is will not give the 
answer you expect. They all contain roughly 1Gb.

The question you should be asking, is to compare two subvolumes. (eg the 
current /home and a snapshot taken of it last week), and ask how much 
data is different between the two. Depending on how you count the "size" 
of the snapshot will be the total amount of data that is not shared, or 
just the data that is in the snapshot but not the base.

The thing is, I don't think there is an easy way to get a report of the 
amount of non shared data without walking the file-systems in both 
subvolumes and building a large data structure of inodes or suchlike.

Measuring the size of snapshots will get even more thorny when you take 
many snapshots. For example suppose you take one every hour, and you 
have just deleted a large file. All your old hourly snapshots will 
contain a reference to that large file, but the data will only be on 
disc once, so you don't want to count it's size more than once when 
considering how much of you disc is taken up by snapshots.

NB: I am not a btrfs developer, just an interested user, and lurker on 
this list.

-- 
David Pottage

Re: btrfs snapshot sizes

[Marc MERLIN]

On Fri, May 09, 2014 at 08:42:22AM +0100, David Pottage wrote:
> On 07/05/14 12:19, Marc MERLIN wrote:
> >So have others found a good way to have an idea about how much space is
> >taken by each snapshot?
> >
> >I've tried quota trees, but I'm not sure how to read the output, or if it's
> >correct (including the negative numbers some have mentioned). Are there
> >other options?
> >
> >I think the main problem is that the shared data field is not working,
> >making it harder to know which blocks are only used in a given snapshot.
> 
> In my understanding (devs please correct me if I am wrong), a
> snapshot is just a subvolume that happens to share a lot of data
> with another subvolume. The idea of taking regular snapshots to

Yes.

> This means that suppose you have a subvolume representing your home
> directory that contains around 1Gb of data, and then take daily
> snapshots, asking the kernel how big each snapshot is will not give
> the answer you expect. They all contain roughly 1Gb.
 
Actually want I'm looking for is the size diff compared to the reference
volume they're snapshotted against.

> The question you should be asking, is to compare two subvolumes. (eg
> the current /home and a snapshot taken of it last week), and ask how

Both comparing snapshots against one another in size difference (unique
blocks they don't share) would be useful too.

> The thing is, I don't think there is an easy way to get a report of
> the amount of non shared data without walking the file-systems in
> both subvolumes and building a large data structure of inodes or
> suchlike.

http://bj0z.wordpress.com/2011/04/27/determining-snapshot-size-in-btrfs/
was supposed to do it, but it's not compatible with recent btrfs-tools
anymore.
 
> Measuring the size of snapshots will get even more thorny when you
> take many snapshots. For example suppose you take one every hour,
> and you have just deleted a large file. All your old hourly
> snapshots will contain a reference to that large file, but the data

Yes, I'm very aware of that :)
But showing the size of each compared to the base volume would quickly
show that they are all more than 1GB different and clue me in.

If we could get that code fixed (hint, it needs
http://bj0z.wordpress.com/2011/04/22/fiemap-ioctl-from-python/ ), I
think it'd be in good shape.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: btrfs snapshot sizes

[Josef Bacik]

On 05/07/2014 07:19 AM, Marc MERLIN wrote:
> So have others found a good way to have an idea about how much space is
> taken by each snapshot?
>
> I've tried quota trees, but I'm not sure how to read the output, or if it's
> correct (including the negative numbers some have mentioned). Are there
> other options?
>
> I think the main problem is that the shared data field is not working,
> making it harder to know which blocks are only used in a given snapshot.
>
>
> subvol                                          group         total    unshared
> -------------------------------------------------------------------------------
> backup/debian32                                 0/262       403.84G      -5.46G
> backup/debian32_daily_20140504_00:03:01         0/3660      446.45G       0.00G
> backup/debian32_daily_20140505_00:03:01         0/3687      431.11G       0.00G
> backup/debian32_daily_20140506_00:03:00         0/3705      420.83G       0.00G
> backup/debian32_daily_20140507_00:03:01         0/3724      411.87G       0.00G
> backup/debian32_weekly_20140504_00:04:01        0/3675      446.45G       0.00G
>
> backup/debian64                                 0/263       855.97G      -1.50G
> backup/debian64_daily_20140504_00:03:01         0/3662      860.19G       0.00G
> backup/debian64_daily_20140505_00:03:01         0/3690      859.32G       0.00G
> backup/debian64_daily_20140506_00:03:00         0/3707      858.15G       0.00G
> backup/debian64_daily_20140507_00:03:01         0/3726      857.47G       0.00G
> backup/debian64_weekly_20140504_00:04:01        0/3676      860.19G       0.00G
>
> backup/ubuntu                                   0/264       360.28G       0.00G
> backup/ubuntu_daily_20140504_00:03:01           0/3664      364.53G       0.00G
> backup/ubuntu_daily_20140505_00:03:01           0/3692      362.44G       0.00G
> backup/ubuntu_daily_20140506_00:03:00           0/3709      360.91G       0.00G
> backup/ubuntu_daily_20140507_00:03:01           0/3727      360.33G       0.00G
> backup/ubuntu_weekly_20140504_00:04:01          0/3677      364.53G       0.00G
>
>

Quota's wont work quite right unless you have my most recent patch, give 
that a whirl and you should be good to go.  Thanks,

Josef