Re: Retrieving CSUM-Tree

Hans van Kranenburg
2019-01-25T20:49:32Z

On 1/25/19 9:45 PM, Tobias Reinhard wrote:
> Am 25.01.2019 um 19:05 schrieb Hans van Kranenburg:
>> On 1/25/19 5:59 PM, Tobias Reinhard wrote:
>>> Am 13.01.2019 um 12:02 schrieb Qu Wenruo:
>>>> On 2019/1/13 下午6:19, Tobias Reinhard wrote:
>>>>> Hi,
>>>>>
>>>>> I want to read the complete CSUM-Tree from userspace. I tried it
>>>>> via the
>>>>> ioctl. This is what the code looks like:
>>>>>
>>>>> struct btrfs_sv2_args sv2_args;
>>>>> int fd = open(filename, O_RDONLY);
>>>>> sv2_args.key.tree_id = BTRFS_CSUM_TREE_OBJECTID;
>>>>> sv2_args.key.min_objectid = 0;
>>>>> sv2_args.key.max_objectid = -1;
>>>>> sv2_args.key.min_offset = 0;
>>>>> sv2_args.key.max_offset = -1;
>>>>> sv2_args.key.min_transid = 0;
>>>>> sv2_args.key.max_transid = -1;
>>>>> sv2_args.key.min_type = BTRFS_CSUM_ITEM_KEY;
>>>>> sv2_args.key.max_type = BTRFS_CSUM_ITEM_KEY;
>>>>> sv2_args.key.nr_items = -1;
>>>>> sv2_args.buf_size = sizeof(sv2_args.buf);
>>>>> ioctl(fd, BTRFS_IOC_TREE_SEARCH_V2, &sv2_args);
>>>>>
>>>>> But the device is not small and I hit the limit of the
>>>>> btrfs_sv2_args.buf which seems to be 16 MB.
>>>>>
>>>>> How can I get the *complete* CSUM-Tree?
>>>>>
>>>>> Limiting to offset does not work (My first idea was to do it this way
>>>>> and get it in chunks).
>>>> That's strange.
>>>>
>>>> Are you still using 0~-1 objectid and 0~-1 type, just last_offset~-1?
>>>>
>>>> Have tried searching using the following parameters?
>>>> min_objectid = max_objectid = BTRFS_EXTENT_CSUM_OBJECTID
>>>> min_type = max_type = BTRFS_CSUM_ITEM_KEY;
>>>> min_offset = last_found_csum_offset
>>>> max_offset = -1
>>> Sorry for my late response.
>>>
>>> If I set
>>>
>>> min_objectid = max_objectid = BTRFS_EXTENT_CSUM_OBJECTID
>>>
>>> I don't get anything. I have to set it to max=-1 (min doesn't matter).
>>>
>>> And in that I case, min_offset and max_offset doesn't matter - I always
>>> get the same result. I can even use "wrong" filters like min=1000
>>> max=500.
>> First, it's important to understand how all these min/max values play
>> together:
>>
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/include/uapi/linux/btrfs.h#n441
>>
>>
>> So, you define a single start key, and a single end key, and then you
>> get everything that's in between (including the end value).
>>
>> So, this...
>>
>> sv2_args.key.min_objectid = 0;
>> sv2_args.key.max_objectid = -1;
>> sv2_args.key.min_offset = 0;
>> sv2_args.key.max_offset = -1;
>> sv2_args.key.min_transid = 0;
>> sv2_args.key.max_transid = -1;
>> sv2_args.key.min_type = BTRFS_CSUM_ITEM_KEY;
>> sv2_args.key.max_type = BTRFS_CSUM_ITEM_KEY;
>>
>> ...translates to:
>>
>> min key: (0, CSUM_ITEM_KEY, 0)
>> max key: (18446744073709551615, CSUM_ITEM_KEY, 18446744073709551615)
>>
>> Since the keys end up being just a single 136 bit number, it makes no
>> sense to do anything with the middle field, if the first field, objectid
>> is not the same in both start and end key. The search space is linear,
>> not 3 dimensional. The invidual min/max values for objectid, type and
>> offset cannot be used to filter the result, they only define the
>> endpoints of an interval.
>>
>> Since all csum items have the same objectid number anyway, the second
>> suggestion is fine, and gives you this start and end:
>>
>> min key: (EXTENT_CSUM_OBJECTID, CSUM_ITEM_KEY, 0)
>> max key: (EXTENT_CSUM_OBJECTID, CSUM_ITEM_KEY, 18446744073709551615)
>>
>> Works for me (here in python, but using same ioctl):
>>
>> -$ cat show_csum_keys.py
>> #!/usr/bin/python3
>>
>> import btrfs
>> from btrfs.ctree import Key, CSUM_TREE_OBJECTID, \
>>          EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY
>>
>> with btrfs.FileSystem('/mnt/tutorial') as fs:
>>      min_key = Key(EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY, 0)
>>      max_key = Key(EXTENT_CSUM_OBJECTID, EXTENT_CSUM_KEY + 1, 0) - 1
>>      print("Searching from {} to {}".format(min_key, max_key))
>>      for header, data in btrfs.ioctl.search_v2(fs.fd, CSUM_TREE_OBJECTID,
>>                                                min_key, max_key):
>>          print(Key(header.objectid, header.type, header.offset))
>>
>> -# ./show_csum_keys.py
>> Searching from (EXTENT_CSUM EXTENT_CSUM 0) to (EXTENT_CSUM EXTENT_CSUM
>> -1)
>> (EXTENT_CSUM EXTENT_CSUM 5700059136)
>> (EXTENT_CSUM EXTENT_CSUM 5700321280)
>> (EXTENT_CSUM EXTENT_CSUM 5700583424)
>> (EXTENT_CSUM EXTENT_CSUM 5700845568)
>> (EXTENT_CSUM EXTENT_CSUM 5701107712)
>> (EXTENT_CSUM EXTENT_CSUM 5704646656)
>> (EXTENT_CSUM EXTENT_CSUM 5705039872)
>> (EXTENT_CSUM EXTENT_CSUM 5706350592)
>> [...]
>>
> Reading your example, I noticed my mistake.
>
> I took  BTRFS_CSUM_ITEM_KEY  instead of BTRFS_EXTENT_CSUM_KEY for the
> type, doh.
>
> Now, it seems to works.

Hah! Well done. I also didn't spot that. :)

Probably too much CSUM dancing before our eyes... Wait until you start
unpacking the search results, will only get worse...

https://reactiongifs.me/wp-content/uploads/2014/06/reading-ikea-intructions-big-lebowski-confused.gif

--
Hans van Kranenburg