RE: FIEMAP ioctl gets "wrong" address for the extent

["Rebraca Dejan (BSOT/PJ-ES1-Bg)" <Dejan.Rebraca@rs.bosch.com>]

Thanks a lot Qu for this clarification, I was not aware of it.

Cheers,
Dejan


-----Original Message-----
From: Qu Wenruo <quwenruo.btrfs@gmx.com> 
Sent: četvrtak, 02. jul 2020. 13:22
To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; linux-btrfs@vger.kernel.org
Subject: Re: FIEMAP ioctl gets "wrong" address for the extent



On 2020/7/2 下午7:08, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi Qu,
> 
> I'm using this structure to get the address of file extent:
> 
> struct fiemap_extent {
> 	__u64	fe_logical;  /* logical offset in bytes for the start of
> 			      * the extent */
> 	__u64	fe_physical; /* physical offset in bytes for the start
> 			      * of the extent */

fe_physical in btrfs is btrfs logical address.

> 	__u64	fe_length;   /* length in bytes for the extent */
> 	__u64	fe_reserved64[2];
> 	__u32	fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
> 	__u32	fe_reserved[3];
> };
> 
> And using fe_physical field I verified that it really reflects the offset in filesystem image - I can see that file content begins at this offset.
> The problem is that I run into some specific case where file content doesn't begin at fe_physical, I rather have something else at this offset.

As said, there is no guarantee that btrfs logical address is mapped 1:1 on disk.
It's possible, but never guaranteed.

You need to pass that fe_physical number to btrfs-map-logical to find the real on-disk offset.

Thanks,
Qu


> 
> Thanks,
> Dejan
> 
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: četvrtak, 02. jul 2020. 12:19
> To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; 
> linux-btrfs@vger.kernel.org
> Subject: Re: FIEMAP ioctl gets "wrong" address for the extent
> 
> 
> 
> On 2020/7/2 下午5:11, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
>> Hi all,
>>
>> I'm collecting file extents for our application from BtrFs filesystem image.
>> I've noticed that for some files a get the "wrong" physical offset for start of the extent.
> 
> First thing first, btrfs fiemap ioctl only returns btrfs logical address.
> That's an address space in [0, U64_MAX), thus it's not a physical offset you can find in the device.
> 
> For example, for a btrfs on a 10G disk, btrfs fiemap can return address at 128G, which you can never find on that disk.
> 
> This is not that strange, as btrfs can be a multi-device fs, thus we must have an internal address space, and then map part of the logical address into physical disk space.
> 
>> I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.
> 
> If you're using the logical address returned from disk directly, then you won't get the correct data obviously.
> 
> What you need is to map the btrfs logical address to physical device offset, that is done by referring to chunk tree.
> And even after the conversion, it's not always the case for all profiles.
> For SINGLE/DUP/RAID0/RAID1/RAID10/RAID1C*, you can find the data directly, but for RAID5/6, you need to bother the P/Q stripe.
> 
> And furthermore, there are compressed data extents, which on-disk data is compressed, which also diffs from the uncompressed data.
> 
> 
> For the chunk mapping, you can verify the mapping of <logical address> to <physical address> using btrfs inspect dump-tree -t chunk <device>.
> 
> The details of the btrfs_chunk on-disk format can be found here:
> https://btrfs.wiki.kernel.org/index.php/On-disk_Format#CHUNK_ITEM_.28e
> 4.29
> 
> Thanks,
> Qu
> 
>>
>> Environment:
>> - 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 
>> 2020
>> x86_64 x86_64 x86_64 GNU/Linux
>> - btrfs-progs v4.4
>>
>> Does anyone has any idea? I would appreciate your help on this one.
>> Tnx.
>>
>> Best regards,
>> Dejan
>>
>