FIEMAP ioctl gets "wrong" address for the extent

FIEMAP ioctl gets "wrong" address for the extent

[Rebraca Dejan (BSOT/PJ-ES1-Bg)]

Hi all,

I'm collecting file extents for our application from BtrFs filesystem image.
I've noticed that for some files a get the "wrong" physical offset for start of the extent. I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.

Environment:
- 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- btrfs-progs v4.4

Does anyone has any idea? I would appreciate your help on this one.
Tnx.

Best regards,
Dejan

Re: FIEMAP ioctl gets "wrong" address for the extent

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 2174 bytes --]



On 2020/7/2 下午5:11, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi all,
> 
> I'm collecting file extents for our application from BtrFs filesystem image.
> I've noticed that for some files a get the "wrong" physical offset for start of the extent.

First thing first, btrfs fiemap ioctl only returns btrfs logical address.
That's an address space in [0, U64_MAX), thus it's not a physical offset
you can find in the device.

For example, for a btrfs on a 10G disk, btrfs fiemap can return address
at 128G, which you can never find on that disk.

This is not that strange, as btrfs can be a multi-device fs, thus we
must have an internal address space, and then map part of the logical
address into physical disk space.

> I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.

If you're using the logical address returned from disk directly, then
you won't get the correct data obviously.

What you need is to map the btrfs logical address to physical device
offset, that is done by referring to chunk tree.
And even after the conversion, it's not always the case for all profiles.
For SINGLE/DUP/RAID0/RAID1/RAID10/RAID1C*, you can find the data
directly, but for RAID5/6, you need to bother the P/Q stripe.

And furthermore, there are compressed data extents, which on-disk data
is compressed, which also diffs from the uncompressed data.


For the chunk mapping, you can verify the mapping of <logical address>
to <physical address> using btrfs inspect dump-tree -t chunk <device>.

The details of the btrfs_chunk on-disk format can be found here:
https://btrfs.wiki.kernel.org/index.php/On-disk_Format#CHUNK_ITEM_.28e4.29

Thanks,
Qu

> 
> Environment:
> - 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
> - btrfs-progs v4.4
> 
> Does anyone has any idea? I would appreciate your help on this one.
> Tnx.
> 
> Best regards,
> Dejan
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

RE: FIEMAP ioctl gets "wrong" address for the extent

[Rebraca Dejan (BSOT/PJ-ES1-Bg)]

Hi Qu,

I'm using this structure to get the address of file extent:

struct fiemap_extent {
	__u64	fe_logical;  /* logical offset in bytes for the start of
			      * the extent */
	__u64	fe_physical; /* physical offset in bytes for the start
			      * of the extent */
	__u64	fe_length;   /* length in bytes for the extent */
	__u64	fe_reserved64[2];
	__u32	fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
	__u32	fe_reserved[3];
};

And using fe_physical field I verified that it really reflects the offset in filesystem image - I can see that file content begins at this offset.
The problem is that I run into some specific case where file content doesn't begin at fe_physical, I rather have something else at this offset.

Thanks,
Dejan

-----Original Message-----
From: Qu Wenruo <quwenruo.btrfs@gmx.com> 
Sent: četvrtak, 02. jul 2020. 12:19
To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; linux-btrfs@vger.kernel.org
Subject: Re: FIEMAP ioctl gets "wrong" address for the extent



On 2020/7/2 下午5:11, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi all,
> 
> I'm collecting file extents for our application from BtrFs filesystem image.
> I've noticed that for some files a get the "wrong" physical offset for start of the extent.

First thing first, btrfs fiemap ioctl only returns btrfs logical address.
That's an address space in [0, U64_MAX), thus it's not a physical offset you can find in the device.

For example, for a btrfs on a 10G disk, btrfs fiemap can return address at 128G, which you can never find on that disk.

This is not that strange, as btrfs can be a multi-device fs, thus we must have an internal address space, and then map part of the logical address into physical disk space.

> I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.

If you're using the logical address returned from disk directly, then you won't get the correct data obviously.

What you need is to map the btrfs logical address to physical device offset, that is done by referring to chunk tree.
And even after the conversion, it's not always the case for all profiles.
For SINGLE/DUP/RAID0/RAID1/RAID10/RAID1C*, you can find the data directly, but for RAID5/6, you need to bother the P/Q stripe.

And furthermore, there are compressed data extents, which on-disk data is compressed, which also diffs from the uncompressed data.


For the chunk mapping, you can verify the mapping of <logical address> to <physical address> using btrfs inspect dump-tree -t chunk <device>.

The details of the btrfs_chunk on-disk format can be found here:
https://btrfs.wiki.kernel.org/index.php/On-disk_Format#CHUNK_ITEM_.28e4.29

Thanks,
Qu

> 
> Environment:
> - 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 
> x86_64 x86_64 x86_64 GNU/Linux
> - btrfs-progs v4.4
> 
> Does anyone has any idea? I would appreciate your help on this one.
> Tnx.
> 
> Best regards,
> Dejan
> 

Re: FIEMAP ioctl gets "wrong" address for the extent

[Qu Wenruo]

[-- Attachment #1.1: Type: text/plain, Size: 3697 bytes --]



On 2020/7/2 下午7:08, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi Qu,
> 
> I'm using this structure to get the address of file extent:
> 
> struct fiemap_extent {
> 	__u64	fe_logical;  /* logical offset in bytes for the start of
> 			      * the extent */
> 	__u64	fe_physical; /* physical offset in bytes for the start
> 			      * of the extent */

fe_physical in btrfs is btrfs logical address.

> 	__u64	fe_length;   /* length in bytes for the extent */
> 	__u64	fe_reserved64[2];
> 	__u32	fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
> 	__u32	fe_reserved[3];
> };
> 
> And using fe_physical field I verified that it really reflects the offset in filesystem image - I can see that file content begins at this offset.
> The problem is that I run into some specific case where file content doesn't begin at fe_physical, I rather have something else at this offset.

As said, there is no guarantee that btrfs logical address is mapped 1:1
on disk.
It's possible, but never guaranteed.

You need to pass that fe_physical number to btrfs-map-logical to find
the real on-disk offset.

Thanks,
Qu


> 
> Thanks,
> Dejan
> 
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com> 
> Sent: četvrtak, 02. jul 2020. 12:19
> To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; linux-btrfs@vger.kernel.org
> Subject: Re: FIEMAP ioctl gets "wrong" address for the extent
> 
> 
> 
> On 2020/7/2 下午5:11, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
>> Hi all,
>>
>> I'm collecting file extents for our application from BtrFs filesystem image.
>> I've noticed that for some files a get the "wrong" physical offset for start of the extent.
> 
> First thing first, btrfs fiemap ioctl only returns btrfs logical address.
> That's an address space in [0, U64_MAX), thus it's not a physical offset you can find in the device.
> 
> For example, for a btrfs on a 10G disk, btrfs fiemap can return address at 128G, which you can never find on that disk.
> 
> This is not that strange, as btrfs can be a multi-device fs, thus we must have an internal address space, and then map part of the logical address into physical disk space.
> 
>> I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.
> 
> If you're using the logical address returned from disk directly, then you won't get the correct data obviously.
> 
> What you need is to map the btrfs logical address to physical device offset, that is done by referring to chunk tree.
> And even after the conversion, it's not always the case for all profiles.
> For SINGLE/DUP/RAID0/RAID1/RAID10/RAID1C*, you can find the data directly, but for RAID5/6, you need to bother the P/Q stripe.
> 
> And furthermore, there are compressed data extents, which on-disk data is compressed, which also diffs from the uncompressed data.
> 
> 
> For the chunk mapping, you can verify the mapping of <logical address> to <physical address> using btrfs inspect dump-tree -t chunk <device>.
> 
> The details of the btrfs_chunk on-disk format can be found here:
> https://btrfs.wiki.kernel.org/index.php/On-disk_Format#CHUNK_ITEM_.28e4.29
> 
> Thanks,
> Qu
> 
>>
>> Environment:
>> - 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 2020 
>> x86_64 x86_64 x86_64 GNU/Linux
>> - btrfs-progs v4.4
>>
>> Does anyone has any idea? I would appreciate your help on this one.
>> Tnx.
>>
>> Best regards,
>> Dejan
>>
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: FIEMAP ioctl gets "wrong" address for the extent

[David Sterba]

On Thu, Jul 02, 2020 at 09:11:20AM +0000, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi all,
> 
> I'm collecting file extents for our application from BtrFs filesystem image.
> I've noticed that for some files a get the "wrong" physical offset for
> start of the extent. I verified it using hexdump of the filesystem
> image: when dump the content starting from the address returned from
> FIEMAP ioctl, I see that the content is absolutely different from the
> content of the file itself. Also, the FIEMAP ioctl reports regular
> extent, it is not inline.

There are 3 address spaces:

- device physical offsets
- filesystem physical offsets
- filesystem logical offsets

What you seem to expect is that device physical and filesystem physical
and the same. This is not true in general in btrfs and fiemap will
return only the filesystem offsets. To get to the device offsets you'd
need to do the reverse mapping.

RE: FIEMAP ioctl gets "wrong" address for the extent

[Rebraca Dejan (BSOT/PJ-ES1-Bg)]

Thanks a lot Qu for this clarification, I was not aware of it.

Cheers,
Dejan


-----Original Message-----
From: Qu Wenruo <quwenruo.btrfs@gmx.com> 
Sent: četvrtak, 02. jul 2020. 13:22
To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; linux-btrfs@vger.kernel.org
Subject: Re: FIEMAP ioctl gets "wrong" address for the extent



On 2020/7/2 下午7:08, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> Hi Qu,
> 
> I'm using this structure to get the address of file extent:
> 
> struct fiemap_extent {
> 	__u64	fe_logical;  /* logical offset in bytes for the start of
> 			      * the extent */
> 	__u64	fe_physical; /* physical offset in bytes for the start
> 			      * of the extent */

fe_physical in btrfs is btrfs logical address.

> 	__u64	fe_length;   /* length in bytes for the extent */
> 	__u64	fe_reserved64[2];
> 	__u32	fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
> 	__u32	fe_reserved[3];
> };
> 
> And using fe_physical field I verified that it really reflects the offset in filesystem image - I can see that file content begins at this offset.
> The problem is that I run into some specific case where file content doesn't begin at fe_physical, I rather have something else at this offset.

As said, there is no guarantee that btrfs logical address is mapped 1:1 on disk.
It's possible, but never guaranteed.

You need to pass that fe_physical number to btrfs-map-logical to find the real on-disk offset.

Thanks,
Qu


> 
> Thanks,
> Dejan
> 
> -----Original Message-----
> From: Qu Wenruo <quwenruo.btrfs@gmx.com>
> Sent: četvrtak, 02. jul 2020. 12:19
> To: Rebraca Dejan (BSOT/PJ-ES1-Bg) <Dejan.Rebraca@rs.bosch.com>; 
> linux-btrfs@vger.kernel.org
> Subject: Re: FIEMAP ioctl gets "wrong" address for the extent
> 
> 
> 
> On 2020/7/2 下午5:11, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
>> Hi all,
>>
>> I'm collecting file extents for our application from BtrFs filesystem image.
>> I've noticed that for some files a get the "wrong" physical offset for start of the extent.
> 
> First thing first, btrfs fiemap ioctl only returns btrfs logical address.
> That's an address space in [0, U64_MAX), thus it's not a physical offset you can find in the device.
> 
> For example, for a btrfs on a 10G disk, btrfs fiemap can return address at 128G, which you can never find on that disk.
> 
> This is not that strange, as btrfs can be a multi-device fs, thus we must have an internal address space, and then map part of the logical address into physical disk space.
> 
>> I verified it using hexdump of the filesystem image: when dump the content starting from the address returned from FIEMAP ioctl, I see that the content is absolutely different from the content of the file itself. Also, the FIEMAP ioctl reports regular extent, it is not inline.
> 
> If you're using the logical address returned from disk directly, then you won't get the correct data obviously.
> 
> What you need is to map the btrfs logical address to physical device offset, that is done by referring to chunk tree.
> And even after the conversion, it's not always the case for all profiles.
> For SINGLE/DUP/RAID0/RAID1/RAID10/RAID1C*, you can find the data directly, but for RAID5/6, you need to bother the P/Q stripe.
> 
> And furthermore, there are compressed data extents, which on-disk data is compressed, which also diffs from the uncompressed data.
> 
> 
> For the chunk mapping, you can verify the mapping of <logical address> to <physical address> using btrfs inspect dump-tree -t chunk <device>.
> 
> The details of the btrfs_chunk on-disk format can be found here:
> https://btrfs.wiki.kernel.org/index.php/On-disk_Format#CHUNK_ITEM_.28e
> 4.29
> 
> Thanks,
> Qu
> 
>>
>> Environment:
>> - 4.15.0-96-generic #97~16.04.1-Ubuntu SMP Wed Apr 1 03:03:31 UTC 
>> 2020
>> x86_64 x86_64 x86_64 GNU/Linux
>> - btrfs-progs v4.4
>>
>> Does anyone has any idea? I would appreciate your help on this one.
>> Tnx.
>>
>> Best regards,
>> Dejan
>>
> 

Re: FIEMAP ioctl gets "wrong" address for the extent

[Omar Sandoval]

On Thu, Jul 02, 2020 at 07:21:42PM +0800, Qu Wenruo wrote:
> 
> 
> On 2020/7/2 下午7:08, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> > Hi Qu,
> > 
> > I'm using this structure to get the address of file extent:
> > 
> > struct fiemap_extent {
> > 	__u64	fe_logical;  /* logical offset in bytes for the start of
> > 			      * the extent */
> > 	__u64	fe_physical; /* physical offset in bytes for the start
> > 			      * of the extent */
> 
> fe_physical in btrfs is btrfs logical address.
> 
> > 	__u64	fe_length;   /* length in bytes for the extent */
> > 	__u64	fe_reserved64[2];
> > 	__u32	fe_flags;    /* FIEMAP_EXTENT_* flags for this extent */
> > 	__u32	fe_reserved[3];
> > };
> > 
> > And using fe_physical field I verified that it really reflects the offset in filesystem image - I can see that file content begins at this offset.
> > The problem is that I run into some specific case where file content doesn't begin at fe_physical, I rather have something else at this offset.
> 
> As said, there is no guarantee that btrfs logical address is mapped 1:1
> on disk.
> It's possible, but never guaranteed.
> 
> You need to pass that fe_physical number to btrfs-map-logical to find
> the real on-disk offset.
> 
> Thanks,
> Qu

FYI, I have a utility that does this mapping for all extents in a file:
https://github.com/osandov/osandov-linux/blob/master/scripts/btrfs_map_physical.c

$ sudo ./btrfs_map_physical archlinux-2020.07.01-x86_64.iso | column -ts $'\t'        
FILE OFFSET  FILE SIZE  EXTENT OFFSET  EXTENT TYPE  LOGICAL SIZE  LOGICAL OFFSET  PHYSICAL SIZE  DEVID  PHYSICAL OFFSET
0            6811648    0              regular      6815744       304680529920    6815744        1      89898610688
6811648      4096       0              regular      4096          304594616320    4096           1      89812697088
6815744      70250496   0              regular      70254592      419517255680    70254592       1      168228114432
77066240     4096       0              regular      4096          419587510272    4096           1      168298369024
77070336     127647744  0              regular      127647744     443648155648    127647744      1      209538883584
204718080    552960     0              regular      557056        304605401088    557056         1      89823481856
205271040    134217728  0              regular      134217728     491017764864    134217728      1      238654881792
339488768    43814912   0              regular      43814912      419587514368    43814912       1      168298373120
383303680    241664     0              regular      245760        304605958144    245760         1      89824038912
383545344    4096       0              regular      4096          304594620416    4096           1      89812701184
383549440    134217728  0              regular      134217728     517290700800    134217728      1      255264141312
517767168    78376960   0              regular      78381056      451287601152    78381056       1      213957103616
596144128    82284544   0              regular      82284544      517424918528    82284544       1      255398359040

Re: FIEMAP ioctl gets "wrong" address for the extent

[Anand Jain]

On 2/7/20 7:43 pm, David Sterba wrote:
> On Thu, Jul 02, 2020 at 09:11:20AM +0000, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
>> Hi all,
>>
>> I'm collecting file extents for our application from BtrFs filesystem image.
>> I've noticed that for some files a get the "wrong" physical offset for
>> start of the extent. I verified it using hexdump of the filesystem
>> image: when dump the content starting from the address returned from
>> FIEMAP ioctl, I see that the content is absolutely different from the
>> content of the file itself. Also, the FIEMAP ioctl reports regular
>> extent, it is not inline.
> 
> There are 3 address spaces:
> 
> - device physical offsets
> - filesystem physical offsets
> - filesystem logical offsets
> 
> What you seem to expect is that device physical and filesystem physical
> and the same. This is not true in general in btrfs and fiemap will
> return only the filesystem offsets. To get to the device offsets you'd
> need to do the reverse mapping.

Do you think is it a good idea to rather update vfs? A quick check 
indicates struct fiemap_extent has reserved space to hold the devid, and 
should handle the backward compatibility issues.

struct fiemap_extent {
	__u64	fe_logical; /* logical offset in bytes for the start of
  * the extent */
	__u64	fe_physical; /* physical offset in bytes for the start
  * of the extent */
	__u64	fe_length; /* length in bytes for the extent */
	__u64	fe_reserved64[2];
	__u32	fe_flags; /* FIEMAP_EXTENT_* flags for this extent */
	__u32	fe_reserved[3];
};

Re: FIEMAP ioctl gets "wrong" address for the extent

[David Sterba]

On Tue, Jul 07, 2020 at 05:43:09PM +0800, Anand Jain wrote:
> On 2/7/20 7:43 pm, David Sterba wrote:
> > On Thu, Jul 02, 2020 at 09:11:20AM +0000, Rebraca Dejan (BSOT/PJ-ES1-Bg) wrote:
> >> Hi all,
> >>
> >> I'm collecting file extents for our application from BtrFs filesystem image.
> >> I've noticed that for some files a get the "wrong" physical offset for
> >> start of the extent. I verified it using hexdump of the filesystem
> >> image: when dump the content starting from the address returned from
> >> FIEMAP ioctl, I see that the content is absolutely different from the
> >> content of the file itself. Also, the FIEMAP ioctl reports regular
> >> extent, it is not inline.
> > 
> > There are 3 address spaces:
> > 
> > - device physical offsets
> > - filesystem physical offsets
> > - filesystem logical offsets
> > 
> > What you seem to expect is that device physical and filesystem physical
> > and the same. This is not true in general in btrfs and fiemap will
> > return only the filesystem offsets. To get to the device offsets you'd
> > need to do the reverse mapping.
> 
> Do you think is it a good idea to rather update vfs? A quick check 
> indicates struct fiemap_extent has reserved space to hold the devid, and 
> should handle the backward compatibility issues.

This was proposed a few years back on LSF/MM, whether to extend fiemap
with the device related information or to add a completely new ioctl
that would not have to extend the existing interface in a way that could
become unwieldy.