How can I get blockdev offsets of btrfs chunks for a file?

How can I get blockdev offsets of btrfs chunks for a file?

[Eric Wheeler]

Hello all,

We do btrfs subvolume snapshots over time for backups.  I would like to 
traverse the files in the subvolumes and find the total unique chunk count 
to calculate total space for a set of subvolumes.

This sounds kind of like the beginning of what a deduplicator would do, 
but I just want to count the blocks, so no submission for deduplication.  
I started looking at bedup and other deduplicator code, but the answer to 
this question wasn't obvious (to me, anyway).

Questions:

Is there an ioctl (or some other way) to get the block device offset for a 
file (or file offset) so I can count the unique occurances?

What API documentation should I review?

Can you point me at the ioctl(s) that would handle this?


Thank you for your help!


--
Eric Wheeler

Re: How can I get blockdev offsets of btrfs chunks for a file?

[Tomasz Kusmierz]

No answer here, but mate if you are involved in anything that will provide some more automated backup tool for btrfs you got a lot of silent people rooting for you.

> On 16 Jul 2016, at 00:21, Eric Wheeler <btrfs@lists.ewheeler.net> wrote:
> 
> Hello all,
> 
> We do btrfs subvolume snapshots over time for backups.  I would like to 
> traverse the files in the subvolumes and find the total unique chunk count 
> to calculate total space for a set of subvolumes.
> 
> This sounds kind of like the beginning of what a deduplicator would do, 
> but I just want to count the blocks, so no submission for deduplication.  
> I started looking at bedup and other deduplicator code, but the answer to 
> this question wasn't obvious (to me, anyway).
> 
> Questions:
> 
> Is there an ioctl (or some other way) to get the block device offset for a 
> file (or file offset) so I can count the unique occurances?
> 
> What API documentation should I review?
> 
> Can you point me at the ioctl(s) that would handle this?
> 
> 
> Thank you for your help!
> 
> 
> --
> Eric Wheeler
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: How can I get blockdev offsets of btrfs chunks for a file?

[Adam Borowski]

On Fri, Jul 15, 2016 at 04:21:31PM -0700, Eric Wheeler wrote:
> We do btrfs subvolume snapshots over time for backups.  I would like to 
> traverse the files in the subvolumes and find the total unique chunk count 
> to calculate total space for a set of subvolumes.
> 
> This sounds kind of like the beginning of what a deduplicator would do, 
> but I just want to count the blocks, so no submission for deduplication.  
> I started looking at bedup and other deduplicator code, but the answer to 
> this question wasn't obvious (to me, anyway).
> 
> Questions:
> 
> Is there an ioctl (or some other way) to get the block device offset for a 
> file (or file offset) so I can count the unique occurances?

Yes, FIEMAP.

You can play with it via "/usr/sbin/filefrag -v".  That /usr/sbin is
misleading -- FIEMAP doesn't require root, although its predecessor did need
that, https://bugs.debian.org/819923

> What API documentation should I review?

In kernel sources, Documentation/filesystems/fiemap.txt


Meow!
-- 
An imaginary friend squared is a real enemy.

Re: How can I get blockdev offsets of btrfs chunks for a file?

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

On Fri, Jul 15, 2016 at 04:21:31PM -0700, Eric Wheeler wrote:
> Hello all,
> 
> We do btrfs subvolume snapshots over time for backups.  I would like to 
> traverse the files in the subvolumes and find the total unique chunk count 
> to calculate total space for a set of subvolumes.

   btrfs fi du may help here. Alternatively, qgroups should be able to
tell you for groups of subvols, if it's set up correctly. You
shouldn't need to implement this at a low level yourself...

> This sounds kind of like the beginning of what a deduplicator would do, 
> but I just want to count the blocks, so no submission for deduplication.  
> I started looking at bedup and other deduplicator code, but the answer to 
> this question wasn't obvious (to me, anyway).
> 
> Questions:
> 
> Is there an ioctl (or some other way) to get the block device offset for a 
> file (or file offset) so I can count the unique occurances?

   This is very much an X/Y question. There already exist a couple of
things that are at least close to the thing you actually want to
do. :)

   Hugo.

> What API documentation should I review?
> 
> Can you point me at the ioctl(s) that would handle this?
> 
> 
> Thank you for your help!
> 
> 

-- 
Hugo Mills             | Reintarnation: Coming back from the dead as a
hugo@... carfax.org.uk | hillbilly
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]