* [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 12:40 ` Mark Fasheh
@ 2008-09-10 12:49 ` Mark Fasheh
2008-09-10 13:47 ` Christoph Hellwig
2008-09-10 18:46 ` Andrew Morton
2008-09-10 12:49 ` [PATCH 1/3] vfs: vfs-level fiemap interface Mark Fasheh
` (2 subsequent siblings)
3 siblings, 2 replies; 16+ messages in thread
From: Mark Fasheh @ 2008-09-10 12:49 UTC (permalink / raw)
To: Andrew Morton; +Cc: Andreas Dilger, Eric Sandeen, linux-ext4
Hello,
The following patches are the latest attempt at implementing a
fiemap ioctl, which can be used by userspace software to get extent
information for an inode in an efficient manner.
These patches are against 2.6.27-rc6. I hope we've incorporated
enough feedback by now that they're ready for inclusion in -mm.
There is an ioctl wrapper program available for testing at:
http://www.kernel.org/pub/linux/kernel/people/mfasheh/fiemap/tests/
Changes from last posting:
* s/FIEMAP_FLAG_NO_DIRECT/FIEMAP_FLAG_NO_BYPASS/
* Updated wording for FIEMAP_EXTENT_UNWRITTEN
* Ext4 patch temporarily dropped because it no longer applies. I'm sure we
can rectify this quickly.
Below this I will include the contents of fiemap.txt to make it convenient
for folks to get details on the API.
--Mark
============
Fiemap Ioctl
============
The fiemap ioctl is an efficient method for userspace to get file
extent mappings. Instead of block-by-block mapping (such as bmap), fiemap
returns a list of extents.
Request Basics
--------------
A fiemap request is encoded within struct fiemap:
struct fiemap {
__u64 fm_start; /* logical offset (inclusive) at
* which to start mapping (in) */
__u64 fm_length; /* logical length of mapping which
* userspace cares about (in) */
__u32 fm_flags; /* FIEMAP_FLAG_* flags for request (in/out) */
__u32 fm_mapped_extents; /* number of extents that were
* mapped (out) */
__u32 fm_extent_count; /* size of fm_extents array (in) */
__u32 fm_reserved;
struct fiemap_extent fm_extents[0]; /* array of mapped extents (out) */
};
fm_start, and fm_length specify the logical range within the file
which the process would like mappings for. Extents returned mirror
those on disk - that is, the logical offset of the 1st returned extent
may start before fm_start, and the range covered by the last returned
extent may end after fm_length. All offsets and lengths are in bytes.
Certain flags to modify the way in which mappings are looked up can be
set in fm_flags. If the kernel doesn't understand some particular
flags, it will return EBADR and the contents of fm_flags will contain
the set of flags which caused the error. If the kernel is compatible
with all flags passed, the contents of fm_flags will be unmodified.
It is up to userspace to determine whether rejection of a particular
flag is fatal to it's operation. This scheme is intended to allow the
fiemap interface to grow in the future but without losing
compatibility with old software.
fm_extent_count specifies the number of elements in the fm_extents[] array
that can be used to return extents. If fm_extent_count is zero, then the
fm_extents[] array is ignored (no extents will be returned), and the
fm_mapped_extents count will hold the number of extents needed in
fm_extents[] to hold the file's current mapping. Note that there is
nothing to prevent the file from changing between calls to FIEMAP.
Currently, there are three flags which can be set in fm_flags:
* FIEMAP_FLAG_SYNC
If this flag is set, the kernel will sync the file before mapping extents.
* FIEMAP_FLAG_XATTR
If this flag is set, the extents returned will describe the inodes
extended attribute lookup tree, instead of it's data tree.
Extent Mapping
--------------
Extent information is returned within the embedded fm_extents array
which userspace must allocate along with the fiemap structure. The
number of elements in the fiemap_extents[] array should be passed via
fm_extent_count. The number of extents mapped by kernel will be
returned via fm_mapped_extents. If the number of fiemap_extents
allocated is less than would be required to map the requested range,
the maximum number of extents that can be mapped in the fm_extent[]
array will be returned and fm_mapped_extents will be equal to
fm_extent_count. In that case, the last extent in the array will not
complete the requested range and will not have the FIEMAP_EXTENT_LAST
flag set (see the next section on extent flags).
Each extent is described by a single fiemap_extent structure as
returned in fm_extents.
struct fiemap_extent {
__u64 fe_logical; /* logical offset in bytes for the start of
* the extent */
__u64 fe_physical; /* physical offset in bytes for the start
* of the extent */
__u64 fe_length; /* length in bytes for the extent */
__u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */
__u32 fe_device; /* device number for extent */
};
All offsets and lengths are in bytes and mirror those on disk. It is valid
for an extents logical offset to start before the request or it's logical
length to extend past the request. Unless FIEMAP_EXTENT_NOT_ALIGNED is
returned, fe_logical, fe_physical, and fe_length will be aligned to the
block size of the file system. With the exception of extents flagged as
FIEMAP_EXTENT_MERGED, adjacent extents will not be merged.
The fe_flags field contains flags which describe the extent returned.
A special flag, FIEMAP_EXTENT_LAST is always set on the last extent in
the file so that the process making fiemap calls can determine when no
more extents are available, without having to call the ioctl again.
Some flags are intentionally vague and will always be set in the
presence of other more specific flags. This way a program looking for
a general property does not have to know all existing and future flags
which imply that property.
For example, if FIEMAP_EXTENT_DATA_INLINE or FIEMAP_EXTENT_DATA_TAIL
are set, FIEMAP_EXTENT_NOT_ALIGNED will also be set. A program looking
for inline or tail-packed data can key on the specific flag. Software
which simply cares not to try operating on non-aligned extents
however, can just key on FIEMAP_EXTENT_NOT_ALIGNED, and not have to
worry about all present and future flags which might imply unaligned
data. Note that the opposite is not true - it would be valid for
FIEMAP_EXTENT_NOT_ALIGNED to appear alone.
* FIEMAP_EXTENT_LAST
This is the last extent in the file. A mapping attempt past this
extent will return nothing.
* FIEMAP_EXTENT_UNKNOWN
The location of this extent is currently unknown. This may indicate
the data is stored on an inaccessible volume or that no storage has
been allocated for the file yet.
* FIEMAP_EXTENT_DELALLOC
- This will also set FIEMAP_EXTENT_UNKNOWN.
Delayed allocation - while there is data for this extent, it's
physical location has not been allocated yet.
* FIEMAP_EXTENT_NO_BYPASS
Direct access to the data in this extent is illegal or will have
undefined results.
* FIEMAP_EXTENT_SECONDARY
The data for this extent is in secondary storage (e.g. HSM). If the
data is not also in the filesystem, then FIEMAP_EXTENT_NO_BYPASS
should also be set.
* FIEMAP_EXTENT_NET
- This will also set FIEMAP_EXTENT_NO_BYPASS
The data for this extent is not stored in a locally-accessible device.
* FIEMAP_EXTENT_DATA_COMPRESSED
- This will also set FIEMAP_EXTENT_NO_BYPASS
The data in this extent has been compressed by the file system.
* FIEMAP_EXTENT_DATA_ENCRYPTED
- This will also set FIEMAP_EXTENT_NO_BYPASS
The data in this extent has been encrypted by the file system.
* FIEMAP_EXTENT_NOT_ALIGNED
Extent offsets and length are not guaranteed to be block aligned.
* FIEMAP_EXTENT_DATA_INLINE
This will also set FIEMAP_EXTENT_NOT_ALIGNED
Data is located within a meta data block.
* FIEMAP_EXTENT_DATA_TAIL
This will also set FIEMAP_EXTENT_NOT_ALIGNED
Data is packed into a block with data from other files.
* FIEMAP_EXTENT_UNWRITTEN
Unwritten extent - the extent is allocated but it's data has not been
initialized. This indicates the extent's data will be all zero if read
through the filesystem but the contents are undefined if read directly from
the device.
* FIEMAP_EXTENT_MERGED
This will be set when a file does not support extents, i.e., it uses a block
based addressing scheme. Since returning an extent for each block back to
userspace would be highly inefficient, the kernel will try to merge most
adjacent blocks into 'extents'.
VFS -> File System Implementation
---------------------------------
File systems wishing to support fiemap must implement a ->fiemap callback on
their inode_operations structure. The fs ->fiemap call is responsible for
defining it's set of supported fiemap flags, and calling a helper function on
each discovered extent:
struct inode_operations {
...
int (*fiemap)(struct inode *, struct fiemap_extent_info *, u64 start,
u64 len);
->fiemap is passed struct fiemap_extent_info which describes the
fiemap request:
struct fiemap_extent_info {
unsigned int fi_flags; /* Flags as passed from user */
unsigned int fi_extents_mapped; /* Number of mapped extents */
unsigned int fi_extents_max; /* Size of fiemap_extent array */
struct fiemap_extent *fi_extents_start; /* Start of fiemap_extent array */
};
It is intended that the file system should not need to access any of this
structure directly.
Flag checking should be done at the beginning of the ->fiemap callback via the
fiemap_check_flags() helper:
int fiemap_check_flags(struct fiemap_extent_info *fieinfo, u32 fs_flags);
The struct fieinfo should be passed in as recieved from ioctl_fiemap(). The
set of fiemap flags which the fs understands should be passed via fs_flags. If
fiemap_check_flags finds invalid user flags, it will place the bad values in
fieinfo->fi_flags and return -EBADR. If the file system gets -EBADR, from
fiemap_check_flags(), it should immediately exit, returning that error back to
ioctl_fiemap().
For each extent in the request range, the file system should call
the helper function, fiemap_fill_next_extent():
int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical,
u64 phys, u64 len, u32 flags, u32 dev);
fiemap_fill_next_extent() will use the passed values to populate the
next free extent in the fm_extents array. 'General' extent flags will
automatically be set from specific flags on behalf of the calling file
system so that the userspace API is not broken.
fiemap_fill_next_extent() returns 0 on success, and 1 when the
user-supplied fm_extents array is full. If an error is encountered
while copying the extent to user memory, -EFAULT will be returned.
--
Mark Fasheh
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 12:49 ` [PATCH 0/3] Fiemap, an extent mapping ioctl Mark Fasheh
@ 2008-09-10 13:47 ` Christoph Hellwig
2008-09-10 14:07 ` Theodore Tso
2008-09-10 16:10 ` Mark Fasheh
2008-09-10 18:46 ` Andrew Morton
1 sibling, 2 replies; 16+ messages in thread
From: Christoph Hellwig @ 2008-09-10 13:47 UTC (permalink / raw)
To: Mark Fasheh; +Cc: Andrew Morton, Andreas Dilger, Eric Sandeen, linux-ext4
On Wed, Sep 10, 2008 at 05:49:34AM -0700, Mark Fasheh wrote:
> * FIEMAP_FLAG_XATTR
> If this flag is set, the extents returned will describe the inodes
> extended attribute lookup tree, instead of it's data tree.
So does this actually make sense for any filesystem but XFS? Still
seems like a not too useful option for a highlevel generic interface.
> __u32 fe_device; /* device number for extent */
As sayd before please kill thise one. It doesn't make any sense at all
for any merged or near mainline FS. It'd be an utterly stupid
lustre-specific hack that still doesn't make much sense.
> * FIEMAP_EXTENT_NO_BYPASS
> Direct access to the data in this extent is illegal or will have
> undefined results.
Huh? What is direct access? Direct access as in bypassing the
filesystem and writing to the blockdev directly always has undefined
results.
> * FIEMAP_EXTENT_SECONDARY
> The data for this extent is in secondary storage (e.g. HSM). If the
> data is not also in the filesystem, then FIEMAP_EXTENT_NO_BYPASS
> should also be set.
No HSM in mainline, so please drop it. We can add it once we get HSM
support.
> * FIEMAP_EXTENT_NET
> - This will also set FIEMAP_EXTENT_NO_BYPASS
> The data for this extent is not stored in a locally-accessible device.
Doesn't make any sense currently, please drop.
> * FIEMAP_EXTENT_DATA_COMPRESSED
> - This will also set FIEMAP_EXTENT_NO_BYPASS
> The data in this extent has been compressed by the file system.
Add once we have users for it.
> * FIEMAP_EXTENT_DATA_ENCRYPTED
> - This will also set FIEMAP_EXTENT_NO_BYPASS
> The data in this extent has been encrypted by the file system.
>
> * FIEMAP_EXTENT_NOT_ALIGNED
> Extent offsets and length are not guaranteed to be block aligned.
>
> * FIEMAP_EXTENT_DATA_INLINE
> This will also set FIEMAP_EXTENT_NOT_ALIGNED
> Data is located within a meta data block.
>
> * FIEMAP_EXTENT_DATA_TAIL
> This will also set FIEMAP_EXTENT_NOT_ALIGNED
> Data is packed into a block with data from other files.
>
> * FIEMAP_EXTENT_UNWRITTEN
> Unwritten extent - the extent is allocated but it's data has not been
> initialized. This indicates the extent's data will be all zero if read
> through the filesystem but the contents are undefined if read directly from
> the device.
>
> * FIEMAP_EXTENT_MERGED
> This will be set when a file does not support extents, i.e., it uses a block
> based addressing scheme. Since returning an extent for each block back to
> userspace would be highly inefficient, the kernel will try to merge most
> adjacent blocks into 'extents'.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 13:47 ` Christoph Hellwig
@ 2008-09-10 14:07 ` Theodore Tso
2008-09-10 14:11 ` Christoph Hellwig
2008-09-10 16:10 ` Mark Fasheh
1 sibling, 1 reply; 16+ messages in thread
From: Theodore Tso @ 2008-09-10 14:07 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Mark Fasheh, Andrew Morton, Andreas Dilger, Eric Sandeen,
linux-ext4
On Wed, Sep 10, 2008 at 09:47:27AM -0400, Christoph Hellwig wrote:
> On Wed, Sep 10, 2008 at 05:49:34AM -0700, Mark Fasheh wrote:
> > * FIEMAP_FLAG_XATTR
> > If this flag is set, the extents returned will describe the inodes
> > extended attribute lookup tree, instead of it's data tree.
>
> So does this actually make sense for any filesystem but XFS? Still
> seems like a not too useful option for a highlevel generic interface.
Yes, Andreas has submitted a proposed extension which would make this
make sense for ext4.
> > __u32 fe_device; /* device number for extent */
>
> As sayd before please kill thise one. It doesn't make any sense at all
> for any merged or near mainline FS. It'd be an utterly stupid
> lustre-specific hack that still doesn't make much sense.
As a suggestion, how about adding some __u32 and __u64 reserved
fields, some of which are reserved for use by the filesystem (i.e.,
you have to check f_type as returned by statfs to know what can be
used), and some for future generic expansions to the generic FIEMAP
interface. When designing a userspace interfaces, leaving room in
data structures for future expansion is a good thing, since it can
avoid needing to have multiple versions of an interface when we need
to expand the data structure.
While I can understand not wanting to have any new kernel functions
without any mainline users for that interface, leaving a few reserved
fields in data structures is just a smart thing to do, and has little
or no downsides.
Whether or not we explicitly define bitfields like
FIEMAP_EXTENT_SECONDARY is much less of an issue, since we can always
grab a bit position later without changing the size of the interface.
But if it turns out we want to add something like some way of
identifying which physical device is used by for a particular extent
--- something which btrfs will need, by the way --- not having room in
the data structure will just bite us later. So why not reserve some
room now?
Interface minimalism to allow for flexibility later is one thing; but
taken to extremes, it it's just stupid. For example, if we didn't
have any filesystems that needed 64-bit fields in mainline, would that
be a justification for making data structures to use 32-bit fields and
forcing a flag data interface change in the future. Sometimes we can
look ahead just a little bit and design interfaces which are foward
compatible. And it's pretty clear that btrfs will need something like
fe_device, although whether it should be a 32-bit index or something
else like a 128-bit UUID admittedly might not be clear at the moment.
But leaving room in the data structure for future growth is just a
intelligent thing to do.
Regards,
- Ted
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 14:07 ` Theodore Tso
@ 2008-09-10 14:11 ` Christoph Hellwig
2008-09-10 20:33 ` Andreas Dilger
0 siblings, 1 reply; 16+ messages in thread
From: Christoph Hellwig @ 2008-09-10 14:11 UTC (permalink / raw)
To: Theodore Tso
Cc: Christoph Hellwig, Mark Fasheh, Andrew Morton, Andreas Dilger,
Eric Sandeen, linux-ext4
On Wed, Sep 10, 2008 at 10:07:19AM -0400, Theodore Tso wrote:
> As a suggestion, how about adding some __u32 and __u64 reserved
> fields, some of which are reserved for use by the filesystem (i.e.,
> you have to check f_type as returned by statfs to know what can be
> used), and some for future generic expansions to the generic FIEMAP
> interface. When designing a userspace interfaces, leaving room in
> data structures for future expansion is a good thing, since it can
> avoid needing to have multiple versions of an interface when we need
> to expand the data structure.
>
> While I can understand not wanting to have any new kernel functions
> without any mainline users for that interface, leaving a few reserved
> fields in data structures is just a smart thing to do, and has little
> or no downsides.
Leaving some spare space in the structure is fine with me, although I
doubt we'll ever use it for a scheme like that, more like other things.
> Interface minimalism to allow for flexibility later is one thing; but
> taken to extremes, it it's just stupid. For example, if we didn't
> have any filesystems that needed 64-bit fields in mainline, would that
> be a justification for making data structures to use 32-bit fields and
> forcing a flag data interface change in the future. Sometimes we can
> look ahead just a little bit and design interfaces which are foward
> compatible. And it's pretty clear that btrfs will need something like
> fe_device, although whether it should be a 32-bit index or something
> else like a 128-bit UUID admittedly might not be clear at the moment.
> But leaving room in the data structure for future growth is just a
> intelligent thing to do.
No, if you look at btrfs or the not public bit for mirrored allocations
groups in XFS you'll notice that an extent can be happily on more than
once device. So to support this we'd need a list of device plus a field
for the algorithm how to balance it over the devices.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 14:11 ` Christoph Hellwig
@ 2008-09-10 20:33 ` Andreas Dilger
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Dilger @ 2008-09-10 20:33 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Theodore Tso, Mark Fasheh, Andrew Morton, Eric Sandeen,
linux-ext4
On Sep 10, 2008 10:11 -0400, Christoph Hellwig wrote:
> On Wed, Sep 10, 2008 at 05:49:34AM -0700, Mark Fasheh wrote:
> > * FIEMAP_FLAG_XATTR
> > If this flag is set, the extents returned will describe the inodes
> > extended attribute lookup tree, instead of it's data tree.
>
> So does this actually make sense for any filesystem but XFS? Still
> seems like a not too useful option for a highlevel generic interface.
You are being mislead by the use of "tree" in the description here.
Both ext3 and ext4 can already have 2 places to store xattr data (in
the inode and in a separate block). It is likely that ext4 will add
the ability to store more xattr data in the future. Would it be better
to just remove the use of "tree" and write:
* FIEMAP_FLAG_XATTR
If this flag is set, the extents returned will describe the inode's
extended attribute data, instead of the regular file data.
> > __u32 fe_device; /* device number for extent */
>
> As sayd before please kill thise one. It doesn't make any sense at all
> for any merged or near mainline FS. It'd be an utterly stupid
> lustre-specific hack that still doesn't make much sense.
I think this is short sighted. Lustre is only one of several filesystems
that aggregate multiple underlying devices into a single filesystem.
Others include XFS, NFSv4, ZFS, chunkfs. It would even be possible to
have FIEMAP return the underlying mapping for "normal" filesystems on
MD/LVM devices, but there are already endless roadblocks to the current
basic functionality that I didn't want to get that mixed in as well.
Even for "normal" filesystems that don't map multiple block devices,
having the fe_device field returned with FIEMAP makes sense for
applications like LILO that need it.
Just because we don't have (m)any filesystems using 64-bit inodes does
that mean we shouldn't have a stat() interface that supports it?
It's true that we started with FIEMAP for Lustre, and thought it would
be helpful to Linux as a whole (better than Solaris SEEK_HOLE/SEEK_DATA,
IMHO) and made sure we got input from many of the filesystem developers
during the design, and now it seems you want to rip out everything
that makes it useful to Lustre and any "future" filesystems that aren't
plain-old block-based filesystems.
> > * FIEMAP_EXTENT_NO_BYPASS
> > Direct access to the data in this extent is illegal or will have
> > undefined results.
>
> Huh? What is direct access? Direct access as in bypassing the
> filesystem and writing to the blockdev directly always has undefined
> results.
Sure, but LILO still does it, as does dump(8). There were a bunch of
people commenting on previous FIEMAP discussions that want to do this,
because it is "safe" according to their environment/usage/definition
of safe. Just because you don't need to do it (nor do I, for that matter)
it doesn't mean that other people do not.
> > * FIEMAP_EXTENT_SECONDARY
> > The data for this extent is in secondary storage (e.g. HSM). If the
> > data is not also in the filesystem, then FIEMAP_EXTENT_NO_BYPASS
> > should also be set.
>
> No HSM in mainline, so please drop it. We can add it once we get HSM
> support.
This was added because XFS_BMAP had it, and XFS is using HSM in some
environments as you well know. #defining a flag that isn't used by the
core Linux code isn't going to kill anyone, and makes life a tiny bit
easier for Dave at zero cost to anyone else.
> > * FIEMAP_EXTENT_NET
> > - This will also set FIEMAP_EXTENT_NO_BYPASS
> > The data for this extent is not stored in a locally-accessible device.
>
> Doesn't make any sense currently, please drop.
Again, it doesn't hurt anyone to include this flag. Filesystems like
CRFS will likely want to use this, in addition to Lustre of course.
It will likel also be useful for NFSv4.
You seem to think that the lack of any current in-kernel users means that
the API should be dumbed down until there is no room to grow, while I'd
prefer to design it in a way that ensures it won't be obsolete as soon
as someone thinks outside the "single local block device" box.
> > * FIEMAP_EXTENT_DATA_COMPRESSED
> > - This will also set FIEMAP_EXTENT_NO_BYPASS
> > The data in this extent has been compressed by the file system.
>
> Add once we have users for it.
A bunch of the cramfs/logfs/etc filesystems have compression support.
There were ext2/3 patches for compressed filesystems, and may be in
the future for btrfs as well. It doesn't seem at all unreasonable to
define this.
> On Wed, Sep 10, 2008 at 10:07:19AM -0400, Theodore Tso wrote:
> > Interface minimalism to allow for flexibility later is one thing; but
> > taken to extremes, it it's just stupid. For example, if we didn't
> > have any filesystems that needed 64-bit fields in mainline, would that
> > be a justification for making data structures to use 32-bit fields and
> > forcing a flag data interface change in the future. Sometimes we can
> > look ahead just a little bit and design interfaces which are foward
> > compatible. And it's pretty clear that btrfs will need something like
> > fe_device, although whether it should be a 32-bit index or something
> > else like a 128-bit UUID admittedly might not be clear at the moment.
> > But leaving room in the data structure for future growth is just a
> > intelligent thing to do.
>
> No, if you look at btrfs or the not public bit for mirrored allocations
> groups in XFS you'll notice that an extent can be happily on more than
> once device. So to support this we'd need a list of device plus a field
> for the algorithm how to balance it over the devices.
Actually, you misunderstand how the API is designed to be used.
Instead of having to return a list of devices for a single extent,
it would return an extent per device. That is especially important
because the logical->physical mapping may be different for every extent.
Having simple flags for "RAID1", "RAID5", "RAID6" would cover most of
the cases, and in all cases setting NO_BYPASS for complex layouts
would avoid non-broken applications trying to access the data directly.
The majority of FIEMAP users will only be filefrag and other tools that
want to know relative locations of file data and some information about
the block layout (e.g. filesystem developers, performance tuning, etc),
they won't be trying to read from the raw block device.
To be honest, I don't know why there is such a huge objection to this
patch. First it was ext4/Lustre specific, then people said "this is
generic stuff that should go into the VFS" and now people say "this
is too complex to be in the VFS as-is, let's make it dumb". People
didn't present such an uproar when blktrace was added to the kernel,
people didn't complain about the EXT2_GETFLAGS ioctl that migrated
over to the VFS, even though it has flags not applicable to all
filesystems (or even applicable to ext* like NOTAIL, which was added
to e2fsprogs just for Reiserfs).
It's just an ioctl, folks. We've spent 1000's of collective hours arguing
about minor changes in semantics, as if we were rewriting the whole VFS.
Christoph, if you really hate FIEMAP that much, it can just lose the
VFS changes entirely, and be implemented in ext4 as we like it and other
filesystem maintainers that want e2fsprogs filefrag to provide the improved
information can follow the implementation there if they want.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 13:47 ` Christoph Hellwig
2008-09-10 14:07 ` Theodore Tso
@ 2008-09-10 16:10 ` Mark Fasheh
2008-09-10 16:20 ` Christoph Hellwig
1 sibling, 1 reply; 16+ messages in thread
From: Mark Fasheh @ 2008-09-10 16:10 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: Andrew Morton, Andreas Dilger, Eric Sandeen, linux-ext4
On Wed, Sep 10, 2008 at 09:47:27AM -0400, Christoph Hellwig wrote:
> On Wed, Sep 10, 2008 at 05:49:34AM -0700, Mark Fasheh wrote:
> > * FIEMAP_FLAG_XATTR
> > If this flag is set, the extents returned will describe the inodes
> > extended attribute lookup tree, instead of it's data tree.
>
> So does this actually make sense for any filesystem but XFS? Still
> seems like a not too useful option for a highlevel generic interface.
>
> > __u32 fe_device; /* device number for extent */
>
> As sayd before please kill thise one. It doesn't make any sense at all
> for any merged or near mainline FS. It'd be an utterly stupid
> lustre-specific hack that still doesn't make much sense.
>
> > * FIEMAP_EXTENT_NO_BYPASS
> > Direct access to the data in this extent is illegal or will have
> > undefined results.
>
> Huh? What is direct access? Direct access as in bypassing the
> filesystem and writing to the blockdev directly always has undefined
> results.
>
> > * FIEMAP_EXTENT_SECONDARY
> > The data for this extent is in secondary storage (e.g. HSM). If the
> > data is not also in the filesystem, then FIEMAP_EXTENT_NO_BYPASS
> > should also be set.
>
> No HSM in mainline, so please drop it. We can add it once we get HSM
> support.
>
> > * FIEMAP_EXTENT_NET
> > - This will also set FIEMAP_EXTENT_NO_BYPASS
> > The data for this extent is not stored in a locally-accessible device.
>
> Doesn't make any sense currently, please drop.
>
> > * FIEMAP_EXTENT_DATA_COMPRESSED
> > - This will also set FIEMAP_EXTENT_NO_BYPASS
> > The data in this extent has been compressed by the file system.
>
> Add once we have users for it.
Ok, aside from NO_BYPASS all your proposed changes have been made. Not sure
about NO_BYPASS. Maybe we just update the description? In the meantime, can
we please just put this in -mm? I'll happily do a patch on top of it all to
rename EXTENT_NO_BYPASS once we agree on a name.
Please pull
git://git.kernel.org/pub/scm/linux/kernel/git/mfasheh/ocfs2.git fiemap
--Mark
--
Mark Fasheh
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 16:10 ` Mark Fasheh
@ 2008-09-10 16:20 ` Christoph Hellwig
0 siblings, 0 replies; 16+ messages in thread
From: Christoph Hellwig @ 2008-09-10 16:20 UTC (permalink / raw)
To: Mark Fasheh
Cc: Christoph Hellwig, Andrew Morton, Andreas Dilger, Eric Sandeen,
linux-ext4
On Wed, Sep 10, 2008 at 09:10:37AM -0700, Mark Fasheh wrote:
> Ok, aside from NO_BYPASS all your proposed changes have been made. Not sure
> about NO_BYPASS. Maybe we just update the description? In the meantime, can
> we please just put this in -mm? I'll happily do a patch on top of it all to
> rename EXTENT_NO_BYPASS once we agree on a name.
AFAICS it's meant to say "not a filesystem block but something else",
so we should say that in the flag. Otherwise this one looks good to me.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 12:49 ` [PATCH 0/3] Fiemap, an extent mapping ioctl Mark Fasheh
2008-09-10 13:47 ` Christoph Hellwig
@ 2008-09-10 18:46 ` Andrew Morton
2008-09-10 19:46 ` Andreas Dilger
1 sibling, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2008-09-10 18:46 UTC (permalink / raw)
To: Mark Fasheh; +Cc: Andreas Dilger, Eric Sandeen, linux-ext4
On Wed, 10 Sep 2008 05:49:34 -0700 Mark Fasheh <mfasheh@suse.com> wrote:
> Hello,
>
> The following patches are the latest attempt at implementing a
> fiemap ioctl, which can be used by userspace software to get extent
> information for an inode in an efficient manner.
As these are applicable to all filesystems, Cc:ing only linux-ext4 is
not sufficient. All filesystem developers (at least) need an
opportunity to review and understand these changes.
>
> ...
>
> The fiemap ioctl is an efficient method for userspace to get file
> extent mappings. Instead of block-by-block mapping (such as bmap), fiemap
> returns a list of extents.
>
The above is, afacit, the only offered rationale for the addition of
these new feature. I don't recall seeing anyone complain about bmap()
inefficiency. In fact I rarely hear of anyone using bmap() at all.
This rationale needs expanding, please. A lot.
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH 0/3] Fiemap, an extent mapping ioctl
2008-09-10 18:46 ` Andrew Morton
@ 2008-09-10 19:46 ` Andreas Dilger
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Dilger @ 2008-09-10 19:46 UTC (permalink / raw)
To: Andrew Morton; +Cc: Mark Fasheh, Eric Sandeen, linux-ext4
On Sep 10, 2008 11:46 -0700, Andrew Morton wrote:
> On Wed, 10 Sep 2008 05:49:34 -0700 Mark Fasheh <mfasheh@suse.com> wrote:
> > The following patches are the latest attempt at implementing a
> > fiemap ioctl, which can be used by userspace software to get extent
> > information for an inode in an efficient manner.
>
> As these are applicable to all filesystems, Cc:ing only linux-ext4 is
> not sufficient. All filesystem developers (at least) need an
> opportunity to review and understand these changes.
That was an oversight, partly caused because I launched the thread for
this on linux-ext4 after an ext4 concall. All previous patches have
gone to linux-fsdevel and been through numerous discussions.
> > The fiemap ioctl is an efficient method for userspace to get file
> > extent mappings. Instead of block-by-block mapping (such as bmap), fiemap
> > returns a list of extents.
>
> The above is, afacit, the only offered rationale for the addition of
> these new feature. I don't recall seeing anyone complain about bmap()
> inefficiency. In fact I rarely hear of anyone using bmap() at all.
>
> This rationale needs expanding, please. A lot.
There are several reasons for this new API:
- it avoids tools like "filefrag" (which currently use FIBMAP) having to
do an ioctl for every block in a file, have the kernel map that block
from an on-disk extent (in most newer filesystems), then re-assemble the
extents in userspace.
- it works with filesystems that are not block based (e.g. NTFS, btrfs, etc)
that may align file data on boundaries other than $blocksize boundaries
- it provides a much more rich API for finding out about on-disk allocation,
such as whether allocated blocks are unwritten (e.g. fallocate), if they
are packed along with other data, if the data is in the inode, etc.
- it can share existing XFS-specific functionality (which FIEMAP was designed
to provide a superset of functionality for) with other filesystems.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH 1/3] vfs: vfs-level fiemap interface
2008-09-10 12:40 ` Mark Fasheh
2008-09-10 12:49 ` [PATCH 0/3] Fiemap, an extent mapping ioctl Mark Fasheh
@ 2008-09-10 12:49 ` Mark Fasheh
2008-09-10 12:50 ` [PATCH 2/3] generic block based fiemap implementation Mark Fasheh
2008-09-10 12:50 ` [PATCH 3/3] ocfs2: fiemap support Mark Fasheh
3 siblings, 0 replies; 16+ messages in thread
From: Mark Fasheh @ 2008-09-10 12:49 UTC (permalink / raw)
To: Andrew Morton; +Cc: Andreas Dilger, Eric Sandeen, linux-ext4
Basic vfs-level fiemap infrastructure, which sets up a new ->fiemap
inode operation.
Userspace can get extent information on a file via fiemap ioctl. As input,
the fiemap ioctl takes a struct fiemap which includes an array of struct
fiemap_extent (fm_extents). Size of the extent array is passed as
fm_extent_count and number of extents returned will be written into
fm_mapped_extents. Offset and length fields on the fiemap structure
(fm_start, fm_length) describe a logical range which will be searched for
extents. All extents returned will at least partially contain this range.
The actual extent offsets and ranges returned will be unmodified from their
offset and range on-disk.
The fiemap ioctl returns '0' on success. On error, -1 is returned and errno
is set. If errno is equal to EBADR, then fm_flags will contain those flags
which were passed in which the kernel did not understand. On all other
errors, the contents of fm_extents is undefined.
As fiemap evolved, there have been many authors of the vfs patch. As far as
I can tell, the list includes:
Kalpak Shah <kalpak.shah@sun.com>
Andreas Dilger <adilger@sun.com>
Eric Sandeen <sandeen@redhat.com>
Mark Fasheh <mfasheh@suse.com>
Signed-off-by: Mark Fasheh <mfasheh@suse.com>
---
Documentation/filesystems/fiemap.txt | 229 ++++++++++++++++++++++++++++++++++
fs/ioctl.c | 158 +++++++++++++++++++++++
include/linux/fiemap.h | 68 ++++++++++
include/linux/fs.h | 18 +++
4 files changed, 473 insertions(+), 0 deletions(-)
create mode 100644 Documentation/filesystems/fiemap.txt
create mode 100644 include/linux/fiemap.h
diff --git a/Documentation/filesystems/fiemap.txt b/Documentation/filesystems/fiemap.txt
new file mode 100644
index 0000000..743c092
--- /dev/null
+++ b/Documentation/filesystems/fiemap.txt
@@ -0,0 +1,229 @@
+============
+Fiemap Ioctl
+============
+
+The fiemap ioctl is an efficient method for userspace to get file
+extent mappings. Instead of block-by-block mapping (such as bmap), fiemap
+returns a list of extents.
+
+
+Request Basics
+--------------
+
+A fiemap request is encoded within struct fiemap:
+
+struct fiemap {
+ __u64 fm_start; /* logical offset (inclusive) at
+ * which to start mapping (in) */
+ __u64 fm_length; /* logical length of mapping which
+ * userspace cares about (in) */
+ __u32 fm_flags; /* FIEMAP_FLAG_* flags for request (in/out) */
+ __u32 fm_mapped_extents; /* number of extents that were
+ * mapped (out) */
+ __u32 fm_extent_count; /* size of fm_extents array (in) */
+ __u32 fm_reserved;
+ struct fiemap_extent fm_extents[0]; /* array of mapped extents (out) */
+};
+
+
+fm_start, and fm_length specify the logical range within the file
+which the process would like mappings for. Extents returned mirror
+those on disk - that is, the logical offset of the 1st returned extent
+may start before fm_start, and the range covered by the last returned
+extent may end after fm_length. All offsets and lengths are in bytes.
+
+Certain flags to modify the way in which mappings are looked up can be
+set in fm_flags. If the kernel doesn't understand some particular
+flags, it will return EBADR and the contents of fm_flags will contain
+the set of flags which caused the error. If the kernel is compatible
+with all flags passed, the contents of fm_flags will be unmodified.
+It is up to userspace to determine whether rejection of a particular
+flag is fatal to it's operation. This scheme is intended to allow the
+fiemap interface to grow in the future but without losing
+compatibility with old software.
+
+fm_extent_count specifies the number of elements in the fm_extents[] array
+that can be used to return extents. If fm_extent_count is zero, then the
+fm_extents[] array is ignored (no extents will be returned), and the
+fm_mapped_extents count will hold the number of extents needed in
+fm_extents[] to hold the file's current mapping. Note that there is
+nothing to prevent the file from changing between calls to FIEMAP.
+
+Currently, there are three flags which can be set in fm_flags:
+
+* FIEMAP_FLAG_SYNC
+If this flag is set, the kernel will sync the file before mapping extents.
+
+* FIEMAP_FLAG_XATTR
+If this flag is set, the extents returned will describe the inodes
+extended attribute lookup tree, instead of it's data tree.
+
+
+Extent Mapping
+--------------
+
+Extent information is returned within the embedded fm_extents array
+which userspace must allocate along with the fiemap structure. The
+number of elements in the fiemap_extents[] array should be passed via
+fm_extent_count. The number of extents mapped by kernel will be
+returned via fm_mapped_extents. If the number of fiemap_extents
+allocated is less than would be required to map the requested range,
+the maximum number of extents that can be mapped in the fm_extent[]
+array will be returned and fm_mapped_extents will be equal to
+fm_extent_count. In that case, the last extent in the array will not
+complete the requested range and will not have the FIEMAP_EXTENT_LAST
+flag set (see the next section on extent flags).
+
+Each extent is described by a single fiemap_extent structure as
+returned in fm_extents.
+
+struct fiemap_extent {
+ __u64 fe_logical; /* logical offset in bytes for the start of
+ * the extent */
+ __u64 fe_physical; /* physical offset in bytes for the start
+ * of the extent */
+ __u64 fe_length; /* length in bytes for the extent */
+ __u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */
+ __u32 fe_device; /* device number for extent */
+};
+
+All offsets and lengths are in bytes and mirror those on disk. It is valid
+for an extents logical offset to start before the request or it's logical
+length to extend past the request. Unless FIEMAP_EXTENT_NOT_ALIGNED is
+returned, fe_logical, fe_physical, and fe_length will be aligned to the
+block size of the file system. With the exception of extents flagged as
+FIEMAP_EXTENT_MERGED, adjacent extents will not be merged.
+
+The fe_flags field contains flags which describe the extent returned.
+A special flag, FIEMAP_EXTENT_LAST is always set on the last extent in
+the file so that the process making fiemap calls can determine when no
+more extents are available, without having to call the ioctl again.
+
+Some flags are intentionally vague and will always be set in the
+presence of other more specific flags. This way a program looking for
+a general property does not have to know all existing and future flags
+which imply that property.
+
+For example, if FIEMAP_EXTENT_DATA_INLINE or FIEMAP_EXTENT_DATA_TAIL
+are set, FIEMAP_EXTENT_NOT_ALIGNED will also be set. A program looking
+for inline or tail-packed data can key on the specific flag. Software
+which simply cares not to try operating on non-aligned extents
+however, can just key on FIEMAP_EXTENT_NOT_ALIGNED, and not have to
+worry about all present and future flags which might imply unaligned
+data. Note that the opposite is not true - it would be valid for
+FIEMAP_EXTENT_NOT_ALIGNED to appear alone.
+
+* FIEMAP_EXTENT_LAST
+This is the last extent in the file. A mapping attempt past this
+extent will return nothing.
+
+* FIEMAP_EXTENT_UNKNOWN
+The location of this extent is currently unknown. This may indicate
+the data is stored on an inaccessible volume or that no storage has
+been allocated for the file yet.
+
+* FIEMAP_EXTENT_DELALLOC
+ - This will also set FIEMAP_EXTENT_UNKNOWN.
+Delayed allocation - while there is data for this extent, it's
+physical location has not been allocated yet.
+
+* FIEMAP_EXTENT_NO_BYPASS
+Direct access to the data in this extent is illegal or will have
+undefined results.
+
+* FIEMAP_EXTENT_SECONDARY
+The data for this extent is in secondary storage (e.g. HSM). If the
+data is not also in the filesystem, then FIEMAP_EXTENT_NO_BYPASS
+should also be set.
+
+* FIEMAP_EXTENT_NET
+ - This will also set FIEMAP_EXTENT_NO_BYPASS
+The data for this extent is not stored in a locally-accessible device.
+
+* FIEMAP_EXTENT_DATA_COMPRESSED
+ - This will also set FIEMAP_EXTENT_NO_BYPASS
+The data in this extent has been compressed by the file system.
+
+* FIEMAP_EXTENT_DATA_ENCRYPTED
+ - This will also set FIEMAP_EXTENT_NO_BYPASS
+The data in this extent has been encrypted by the file system.
+
+* FIEMAP_EXTENT_NOT_ALIGNED
+Extent offsets and length are not guaranteed to be block aligned.
+
+* FIEMAP_EXTENT_DATA_INLINE
+ This will also set FIEMAP_EXTENT_NOT_ALIGNED
+Data is located within a meta data block.
+
+* FIEMAP_EXTENT_DATA_TAIL
+ This will also set FIEMAP_EXTENT_NOT_ALIGNED
+Data is packed into a block with data from other files.
+
+* FIEMAP_EXTENT_UNWRITTEN
+Unwritten extent - the extent is allocated but it's data has not been
+initialized. This indicates the extent's data will be all zero if read
+through the filesystem but the contents are undefined if read directly from
+the device.
+
+* FIEMAP_EXTENT_MERGED
+This will be set when a file does not support extents, i.e., it uses a block
+based addressing scheme. Since returning an extent for each block back to
+userspace would be highly inefficient, the kernel will try to merge most
+adjacent blocks into 'extents'.
+
+
+VFS -> File System Implementation
+---------------------------------
+
+File systems wishing to support fiemap must implement a ->fiemap callback on
+their inode_operations structure. The fs ->fiemap call is responsible for
+defining it's set of supported fiemap flags, and calling a helper function on
+each discovered extent:
+
+struct inode_operations {
+ ...
+
+ int (*fiemap)(struct inode *, struct fiemap_extent_info *, u64 start,
+ u64 len);
+
+->fiemap is passed struct fiemap_extent_info which describes the
+fiemap request:
+
+struct fiemap_extent_info {
+ unsigned int fi_flags; /* Flags as passed from user */
+ unsigned int fi_extents_mapped; /* Number of mapped extents */
+ unsigned int fi_extents_max; /* Size of fiemap_extent array */
+ struct fiemap_extent *fi_extents_start; /* Start of fiemap_extent array */
+};
+
+It is intended that the file system should not need to access any of this
+structure directly.
+
+
+Flag checking should be done at the beginning of the ->fiemap callback via the
+fiemap_check_flags() helper:
+
+int fiemap_check_flags(struct fiemap_extent_info *fieinfo, u32 fs_flags);
+
+The struct fieinfo should be passed in as recieved from ioctl_fiemap(). The
+set of fiemap flags which the fs understands should be passed via fs_flags. If
+fiemap_check_flags finds invalid user flags, it will place the bad values in
+fieinfo->fi_flags and return -EBADR. If the file system gets -EBADR, from
+fiemap_check_flags(), it should immediately exit, returning that error back to
+ioctl_fiemap().
+
+
+For each extent in the request range, the file system should call
+the helper function, fiemap_fill_next_extent():
+
+int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical,
+ u64 phys, u64 len, u32 flags, u32 dev);
+
+fiemap_fill_next_extent() will use the passed values to populate the
+next free extent in the fm_extents array. 'General' extent flags will
+automatically be set from specific flags on behalf of the calling file
+system so that the userspace API is not broken.
+
+fiemap_fill_next_extent() returns 0 on success, and 1 when the
+user-supplied fm_extents array is full. If an error is encountered
+while copying the extent to user memory, -EFAULT will be returned.
diff --git a/fs/ioctl.c b/fs/ioctl.c
index 7db32b3..8300862 100644
--- a/fs/ioctl.c
+++ b/fs/ioctl.c
@@ -16,6 +16,9 @@
#include <asm/ioctls.h>
+/* So that the fiemap access checks can't overflow on 32 bit machines. */
+#define FIEMAP_MAX_EXTENTS (UINT_MAX / sizeof(struct fiemap_extent))
+
/**
* vfs_ioctl - call filesystem specific ioctl methods
* @filp: open file to invoke ioctl method on
@@ -71,6 +74,159 @@ static int ioctl_fibmap(struct file *filp, int __user *p)
return put_user(res, p);
}
+/**
+ * fiemap_fill_next_extent - Fiemap helper function
+ * @fieinfo: Fiemap context passed into ->fiemap
+ * @logical: Extent logical start offset, in bytes
+ * @phys: Extent physical start offset, in bytes
+ * @len: Extent length, in bytes
+ * @flags: FIEMAP_EXTENT flags that describe this extent
+ * @dev: Device on which this extent resides
+ *
+ * Called from file system ->fiemap callback. Will populate extent
+ * info as passed in via arguments and copy to user memory. On
+ * success, extent count on fieinfo is incremented.
+ *
+ * Returns 0 on success, -errno on error, 1 if this was the last
+ * extent that will fit in user array.
+ */
+#define SET_UNKNOWN_FLAGS (FIEMAP_EXTENT_DELALLOC)
+#define SET_NO_BYPASS_FLAGS (FIEMAP_EXTENT_DATA_COMPRESSED \
+ |FIEMAP_EXTENT_DATA_ENCRYPTED \
+ |FIEMAP_EXTENT_NET)
+#define SET_NOT_ALIGNED_FLAGS (FIEMAP_EXTENT_DATA_TAIL|FIEMAP_EXTENT_DATA_INLINE)
+int fiemap_fill_next_extent(struct fiemap_extent_info *fieinfo, u64 logical,
+ u64 phys, u64 len, u32 flags, dev_t dev)
+{
+ struct fiemap_extent extent;
+ struct fiemap_extent *dest = fieinfo->fi_extents_start;
+
+ /* only count the extents */
+ if (fieinfo->fi_extents_max == 0) {
+ fieinfo->fi_extents_mapped++;
+ return (flags & FIEMAP_EXTENT_LAST) ? 1 : 0;
+ }
+
+ if (fieinfo->fi_extents_mapped >= fieinfo->fi_extents_max)
+ return 1;
+
+ if (flags & SET_UNKNOWN_FLAGS)
+ flags |= FIEMAP_EXTENT_UNKNOWN;
+ if (flags & SET_NO_BYPASS_FLAGS)
+ flags |= FIEMAP_EXTENT_NO_BYPASS;
+ if (flags & SET_NOT_ALIGNED_FLAGS)
+ flags |= FIEMAP_EXTENT_NOT_ALIGNED;
+
+ extent.fe_logical = logical;
+ extent.fe_physical = phys;
+ extent.fe_length = len;
+ extent.fe_flags = flags;
+ extent.fe_device = new_encode_dev(dev);
+
+ dest += fieinfo->fi_extents_mapped;
+ if (copy_to_user(dest, &extent, sizeof(extent)))
+ return -EFAULT;
+
+ fieinfo->fi_extents_mapped++;
+ if (fieinfo->fi_extents_mapped == fieinfo->fi_extents_max)
+ return 1;
+ return (flags & FIEMAP_EXTENT_LAST) ? 1 : 0;
+}
+EXPORT_SYMBOL(fiemap_fill_next_extent);
+
+/**
+ * fiemap_check_flags - check validity of requested flags for fiemap
+ * @fieinfo: Fiemap context passed into ->fiemap
+ * @fs_flags: Set of fiemap flags that the file system understands
+ *
+ * Called from file system ->fiemap callback. This will compute the
+ * intersection of valid fiemap flags and those that the fs supports. That
+ * value is then compared against the user supplied flags. In case of bad user
+ * flags, the invalid values will be written into the fieinfo structure, and
+ * -EBADR is returned, which tells ioctl_fiemap() to return those values to
+ * userspace. For this reason, a return code of -EBADR should be preserved.
+ *
+ * Returns 0 on success, -EBADR on bad flags.
+ */
+int fiemap_check_flags(struct fiemap_extent_info *fieinfo, u32 fs_flags)
+{
+ u32 incompat_flags;
+
+ incompat_flags = fieinfo->fi_flags & ~(FIEMAP_FLAGS_COMPAT & fs_flags);
+ if (incompat_flags) {
+ fieinfo->fi_flags = incompat_flags;
+ return -EBADR;
+ }
+ return 0;
+}
+EXPORT_SYMBOL(fiemap_check_flags);
+
+static int fiemap_check_ranges(struct super_block *sb,
+ u64 start, u64 len, u64 *new_len)
+{
+ *new_len = len;
+
+ if (len == 0)
+ return -EINVAL;
+
+ if (start > sb->s_maxbytes)
+ return -EFBIG;
+
+ /*
+ * Shrink request scope to what the fs can actually handle.
+ */
+ if ((len > sb->s_maxbytes) ||
+ (sb->s_maxbytes - len) < start)
+ *new_len = sb->s_maxbytes - start;
+
+ return 0;
+}
+
+static int ioctl_fiemap(struct file *filp, unsigned long arg)
+{
+ struct fiemap fiemap;
+ struct fiemap_extent_info fieinfo = { 0, };
+ struct inode *inode = filp->f_path.dentry->d_inode;
+ struct super_block *sb = inode->i_sb;
+ u64 len;
+ int error;
+
+ if (!inode->i_op->fiemap)
+ return -EOPNOTSUPP;
+
+ if (copy_from_user(&fiemap, (struct fiemap __user *)arg,
+ sizeof(struct fiemap)))
+ return -EFAULT;
+
+ if (fiemap.fm_extent_count > FIEMAP_MAX_EXTENTS)
+ return -EINVAL;
+
+ error = fiemap_check_ranges(sb, fiemap.fm_start, fiemap.fm_length,
+ &len);
+ if (error)
+ return error;
+
+ fieinfo.fi_flags = fiemap.fm_flags;
+ fieinfo.fi_extents_max = fiemap.fm_extent_count;
+ fieinfo.fi_extents_start = (struct fiemap_extent *)(arg + sizeof(fiemap));
+
+ if (fiemap.fm_extent_count != 0 &&
+ !access_ok(VERIFY_WRITE, fieinfo.fi_extents_start,
+ fieinfo.fi_extents_max * sizeof(struct fiemap_extent)))
+ return -EFAULT;
+
+ if (fieinfo.fi_flags & FIEMAP_FLAG_SYNC)
+ filemap_write_and_wait(inode->i_mapping);
+
+ error = inode->i_op->fiemap(inode, &fieinfo, fiemap.fm_start, len);
+ fiemap.fm_flags = fieinfo.fi_flags;
+ fiemap.fm_mapped_extents = fieinfo.fi_extents_mapped;
+ if (copy_to_user((char *)arg, &fiemap, sizeof(fiemap)))
+ error = -EFAULT;
+
+ return error;
+}
+
static int file_ioctl(struct file *filp, unsigned int cmd,
unsigned long arg)
{
@@ -80,6 +236,8 @@ static int file_ioctl(struct file *filp, unsigned int cmd,
switch (cmd) {
case FIBMAP:
return ioctl_fibmap(filp, p);
+ case FS_IOC_FIEMAP:
+ return ioctl_fiemap(filp, arg);
case FIGETBSZ:
return put_user(inode->i_sb->s_blocksize, p);
case FIONREAD:
diff --git a/include/linux/fiemap.h b/include/linux/fiemap.h
new file mode 100644
index 0000000..a5661ba
--- /dev/null
+++ b/include/linux/fiemap.h
@@ -0,0 +1,68 @@
+/*
+ * FS_IOC_FIEMAP ioctl infrastructure.
+ *
+ * Some portions copyright (C) 2007 Cluster File Systems, Inc
+ *
+ * Authors: Mark Fasheh <mfasheh@suse.com>
+ * Kalpak Shah <kalpak.shah@sun.com>
+ * Andreas Dilger <adilger@sun.com>
+ */
+
+#ifndef _LINUX_FIEMAP_H
+#define _LINUX_FIEMAP_H
+
+struct fiemap_extent {
+ __u64 fe_logical; /* logical offset in bytes for the start of
+ * the extent from the beginning of the file */
+ __u64 fe_physical; /* physical offset in bytes for the start
+ * of the extent from the beginning of the disk */
+ __u64 fe_length; /* length in bytes for this extent */
+ __u32 fe_flags; /* FIEMAP_EXTENT_* flags for this extent */
+ __u32 fe_device; /* device number for this extent */
+};
+
+struct fiemap {
+ __u64 fm_start; /* logical offset (inclusive) at
+ * which to start mapping (in) */
+ __u64 fm_length; /* logical length of mapping which
+ * userspace wants (in) */
+ __u32 fm_flags; /* FIEMAP_FLAG_* flags for request (in/out) */
+ __u32 fm_mapped_extents;/* number of extents that were mapped (out) */
+ __u32 fm_extent_count; /* size of fm_extents array (in) */
+ __u32 fm_reserved;
+ struct fiemap_extent fm_extents[0]; /* array of mapped extents (out) */
+};
+
+#define FIEMAP_MAX_OFFSET (~0ULL)
+
+#define FIEMAP_FLAG_SYNC 0x00000001 /* sync file data before map */
+#define FIEMAP_FLAG_XATTR 0x00000002 /* map extended attribute tree */
+
+#define FIEMAP_FLAGS_COMPAT (FIEMAP_FLAG_SYNC | FIEMAP_FLAG_XATTR)
+
+#define FIEMAP_EXTENT_LAST 0x00000001 /* Last extent in file. */
+#define FIEMAP_EXTENT_UNKNOWN 0x00000002 /* Data location unknown. */
+#define FIEMAP_EXTENT_DELALLOC 0x00000004 /* Location still pending.
+ * Sets EXTENT_UNKNOWN. */
+#define FIEMAP_EXTENT_NO_BYPASS 0x00000008 /* Data mapping undefined */
+#define FIEMAP_EXTENT_SECONDARY 0x00000010 /* Data copied offline. May
+ * set EXTENT_NO_BYPASS. */
+#define FIEMAP_EXTENT_NET 0x00000020 /* Data stored remotely.
+ * Sets EXTENT_NO_BYPASS. */
+#define FIEMAP_EXTENT_DATA_COMPRESSED 0x00000040 /* Data is compressed by fs.
+ * Sets EXTENT_NO_BYPASS. */
+#define FIEMAP_EXTENT_DATA_ENCRYPTED 0x00000080 /* Data is encrypted by fs.
+ * Sets EXTENT_NO_BYPASS. */
+#define FIEMAP_EXTENT_NOT_ALIGNED 0x00000100 /* Extent offsets may not be
+ * block aligned. */
+#define FIEMAP_EXTENT_DATA_INLINE 0x00000200 /* Data mixed with metadata.
+ * Sets EXTENT_NOT_ALIGNED.*/
+#define FIEMAP_EXTENT_DATA_TAIL 0x00000400 /* Multiple files in block.
+ * Sets EXTENT_NOT_ALIGNED.*/
+#define FIEMAP_EXTENT_UNWRITTEN 0x00000800 /* Space allocated, but
+ * no data (i.e. zero). */
+#define FIEMAP_EXTENT_MERGED 0x00001000 /* File does not natively
+ * support extents. Result
+ * merged for efficiency. */
+
+#endif /* _LINUX_FIEMAP_H */
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 580b513..5964b67 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -231,6 +231,7 @@ extern int dir_notify_enable;
#define FS_IOC_SETFLAGS _IOW('f', 2, long)
#define FS_IOC_GETVERSION _IOR('v', 1, long)
#define FS_IOC_SETVERSION _IOW('v', 2, long)
+#define FS_IOC_FIEMAP _IOWR('f', 11, struct fiemap)
#define FS_IOC32_GETFLAGS _IOR('f', 1, int)
#define FS_IOC32_SETFLAGS _IOW('f', 2, int)
#define FS_IOC32_GETVERSION _IOR('v', 1, int)
@@ -291,6 +292,7 @@ extern int dir_notify_enable;
#include <linux/mutex.h>
#include <linux/capability.h>
#include <linux/semaphore.h>
+#include <linux/fiemap.h>
#include <asm/atomic.h>
#include <asm/byteorder.h>
@@ -1179,6 +1181,20 @@ extern void dentry_unhash(struct dentry *dentry);
extern int file_permission(struct file *, int);
/*
+ * VFS FS_IOC_FIEMAP helper definitions.
+ */
+struct fiemap_extent_info {
+ unsigned int fi_flags; /* Flags as passed from user */
+ unsigned int fi_extents_mapped; /* Number of mapped extents */
+ unsigned int fi_extents_max; /* Size of fiemap_extent array */
+ struct fiemap_extent *fi_extents_start; /* Start of fiemap_extent
+ * array */
+};
+int fiemap_fill_next_extent(struct fiemap_extent_info *info, u64 logical,
+ u64 phys, u64 len, u32 flags, dev_t dev);
+int fiemap_check_flags(struct fiemap_extent_info *fieinfo, u32 fs_flags);
+
+/*
* File types
*
* NOTE! These match bits 12..15 of stat.st_mode
@@ -1287,6 +1303,8 @@ struct inode_operations {
void (*truncate_range)(struct inode *, loff_t, loff_t);
long (*fallocate)(struct inode *inode, int mode, loff_t offset,
loff_t len);
+ int (*fiemap)(struct inode *, struct fiemap_extent_info *, u64 start,
+ u64 len);
};
struct seq_file;
--
1.5.4.5
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH 2/3] generic block based fiemap implementation
2008-09-10 12:40 ` Mark Fasheh
2008-09-10 12:49 ` [PATCH 0/3] Fiemap, an extent mapping ioctl Mark Fasheh
2008-09-10 12:49 ` [PATCH 1/3] vfs: vfs-level fiemap interface Mark Fasheh
@ 2008-09-10 12:50 ` Mark Fasheh
2008-09-10 12:50 ` [PATCH 3/3] ocfs2: fiemap support Mark Fasheh
3 siblings, 0 replies; 16+ messages in thread
From: Mark Fasheh @ 2008-09-10 12:50 UTC (permalink / raw)
To: Andrew Morton; +Cc: Josef Bacik, linux-ext4
From: Josef Bacik <jbacik@redhat.com>
Any block based fs (this patch includes ext3) just has to declare its own
fiemap() function and then call this generic function with its own
get_block_t. This works well for block based filesystems that will map
multiple contiguous blocks at one time, but will work for filesystems that
only map one block at a time, you will just end up with an "extent" for each
block. One gotcha is this will not play nicely where there is hole+data
after the EOF. This function will assume its hit the end of the data as soon
as it hits a hole after the EOF, so if there is any data past that it will
not pick that up. AFAIK no block based fs does this anyway, but its in the
comments of the function anyway just in case.
Signed-off-by: Josef Bacik <jbacik@redhat.com>
Signed-off-by: Mark Fasheh <mfasheh@suse.com>
---
fs/ext2/ext2.h | 2 +
fs/ext2/file.c | 1 +
fs/ext2/inode.c | 8 +++
fs/ext3/file.c | 1 +
fs/ext3/inode.c | 8 +++
fs/ioctl.c | 119 +++++++++++++++++++++++++++++++++++++++++++++++
include/linux/ext3_fs.h | 2 +
include/linux/fs.h | 3 +
8 files changed, 144 insertions(+), 0 deletions(-)
diff --git a/fs/ext2/ext2.h b/fs/ext2/ext2.h
index 47d88da..bae998c 100644
--- a/fs/ext2/ext2.h
+++ b/fs/ext2/ext2.h
@@ -133,6 +133,8 @@ extern void ext2_truncate (struct inode *);
extern int ext2_setattr (struct dentry *, struct iattr *);
extern void ext2_set_inode_flags(struct inode *inode);
extern void ext2_get_inode_flags(struct ext2_inode_info *);
+extern int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 start, u64 len);
int __ext2_write_begin(struct file *file, struct address_space *mapping,
loff_t pos, unsigned len, unsigned flags,
struct page **pagep, void **fsdata);
diff --git a/fs/ext2/file.c b/fs/ext2/file.c
index 5f2fa9c..45ed071 100644
--- a/fs/ext2/file.c
+++ b/fs/ext2/file.c
@@ -86,4 +86,5 @@ const struct inode_operations ext2_file_inode_operations = {
#endif
.setattr = ext2_setattr,
.permission = ext2_permission,
+ .fiemap = ext2_fiemap,
};
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index 991d6df..7658b33 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -31,6 +31,7 @@
#include <linux/writeback.h>
#include <linux/buffer_head.h>
#include <linux/mpage.h>
+#include <linux/fiemap.h>
#include "ext2.h"
#include "acl.h"
#include "xip.h"
@@ -704,6 +705,13 @@ int ext2_get_block(struct inode *inode, sector_t iblock, struct buffer_head *bh_
}
+int ext2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 start, u64 len)
+{
+ return generic_block_fiemap(inode, fieinfo, start, len,
+ ext2_get_block);
+}
+
static int ext2_writepage(struct page *page, struct writeback_control *wbc)
{
return block_write_full_page(page, ext2_get_block, wbc);
diff --git a/fs/ext3/file.c b/fs/ext3/file.c
index acc4913..3be1e06 100644
--- a/fs/ext3/file.c
+++ b/fs/ext3/file.c
@@ -134,5 +134,6 @@ const struct inode_operations ext3_file_inode_operations = {
.removexattr = generic_removexattr,
#endif
.permission = ext3_permission,
+ .fiemap = ext3_fiemap,
};
diff --git a/fs/ext3/inode.c b/fs/ext3/inode.c
index 507d868..ebfec4d 100644
--- a/fs/ext3/inode.c
+++ b/fs/ext3/inode.c
@@ -36,6 +36,7 @@
#include <linux/mpage.h>
#include <linux/uio.h>
#include <linux/bio.h>
+#include <linux/fiemap.h>
#include "xattr.h"
#include "acl.h"
@@ -981,6 +982,13 @@ out:
return ret;
}
+int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 start, u64 len)
+{
+ return generic_block_fiemap(inode, fieinfo, start, len,
+ ext3_get_block);
+}
+
/*
* `handle' can be NULL if create is zero
*/
diff --git a/fs/ioctl.c b/fs/ioctl.c
index 8300862..bedd447 100644
--- a/fs/ioctl.c
+++ b/fs/ioctl.c
@@ -13,6 +13,8 @@
#include <linux/security.h>
#include <linux/module.h>
#include <linux/uaccess.h>
+#include <linux/writeback.h>
+#include <linux/buffer_head.h>
#include <asm/ioctls.h>
@@ -227,6 +229,123 @@ static int ioctl_fiemap(struct file *filp, unsigned long arg)
return error;
}
+#define blk_to_logical(inode, blk) (blk << (inode)->i_blkbits)
+#define logical_to_blk(inode, offset) (offset >> (inode)->i_blkbits);
+
+/*
+ * @inode - the inode to map
+ * @arg - the pointer to userspace where we copy everything to
+ * @get_block - the fs's get_block function
+ *
+ * This does FIEMAP for block based inodes. Basically it will just loop
+ * through get_block until we hit the number of extents we want to map, or we
+ * go past the end of the file and hit a hole.
+ *
+ * If it is possible to have data blocks beyond a hole past @inode->i_size, then
+ * please do not use this function, it will stop at the first unmapped block
+ * beyond i_size
+ */
+int generic_block_fiemap(struct inode *inode,
+ struct fiemap_extent_info *fieinfo, u64 start,
+ u64 len, get_block_t *get_block)
+{
+ struct buffer_head tmp;
+ unsigned int start_blk;
+ long long length = 0, map_len = 0;
+ u64 logical = 0, phys = 0, size = 0;
+ u32 flags = FIEMAP_EXTENT_MERGED;
+ int ret = 0;
+ dev_t dev = inode->i_sb->s_dev;
+
+ if ((ret = fiemap_check_flags(fieinfo, FIEMAP_FLAG_SYNC)))
+ return ret;
+
+ start_blk = logical_to_blk(inode, start);
+
+ /* guard against change */
+ mutex_lock(&inode->i_mutex);
+
+ length = (long long)min_t(u64, len, i_size_read(inode));
+ map_len = length;
+
+ do {
+ /*
+ * we set b_size to the total size we want so it will map as
+ * many contiguous blocks as possible at once
+ */
+ memset(&tmp, 0, sizeof(struct buffer_head));
+ tmp.b_size = map_len;
+
+ ret = get_block(inode, start_blk, &tmp, 0);
+ if (ret)
+ break;
+
+ /* HOLE */
+ if (!buffer_mapped(&tmp)) {
+ /*
+ * first hole after going past the EOF, this is our
+ * last extent
+ */
+ if (length <= 0) {
+ flags = FIEMAP_EXTENT_MERGED|FIEMAP_EXTENT_LAST;
+ ret = fiemap_fill_next_extent(fieinfo, logical,
+ phys, size,
+ flags, dev);
+ break;
+ }
+
+ length -= blk_to_logical(inode, 1);
+
+ /* if we have holes up to/past EOF then we're done */
+ if (length <= 0)
+ break;
+
+ start_blk++;
+ } else {
+ if (length <= 0 && size) {
+ ret = fiemap_fill_next_extent(fieinfo, logical,
+ phys, size,
+ flags, dev);
+ if (ret)
+ break;
+ }
+
+ logical = blk_to_logical(inode, start_blk);
+ phys = blk_to_logical(inode, tmp.b_blocknr);
+ size = tmp.b_size;
+ flags = FIEMAP_EXTENT_MERGED;
+
+ length -= tmp.b_size;
+ start_blk += logical_to_blk(inode, size);
+
+ /*
+ * if we are past the EOF we need to loop again to see
+ * if there is a hole so we can mark this extent as the
+ * last one, and if not keep mapping things until we
+ * find a hole, or we run out of slots in the extent
+ * array
+ */
+ if (length <= 0)
+ continue;
+
+ ret = fiemap_fill_next_extent(fieinfo, logical, phys,
+ size, flags, dev);
+ if (ret)
+ break;
+ }
+ cond_resched();
+ } while (1);
+
+ mutex_unlock(&inode->i_mutex);
+
+ /* if ret is 1 then we just hit the end of the extent array */
+ if (ret == 1)
+ ret = 0;
+
+ return ret;
+}
+EXPORT_SYMBOL(generic_block_fiemap);
+
static int file_ioctl(struct file *filp, unsigned int cmd,
unsigned long arg)
{
diff --git a/include/linux/ext3_fs.h b/include/linux/ext3_fs.h
index 80171ee..8120fa1 100644
--- a/include/linux/ext3_fs.h
+++ b/include/linux/ext3_fs.h
@@ -837,6 +837,8 @@ extern void ext3_truncate (struct inode *);
extern void ext3_set_inode_flags(struct inode *);
extern void ext3_get_inode_flags(struct ext3_inode_info *);
extern void ext3_set_aops(struct inode *inode);
+extern int ext3_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 start, u64 len);
/* ioctl.c */
extern int ext3_ioctl (struct inode *, struct file *, unsigned int,
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 5964b67..d2dc493 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -1998,6 +1998,9 @@ extern int vfs_fstat(unsigned int, struct kstat *);
extern int do_vfs_ioctl(struct file *filp, unsigned int fd, unsigned int cmd,
unsigned long arg);
+extern int generic_block_fiemap(struct inode *inode,
+ struct fiemap_extent_info *fieinfo, u64 start,
+ u64 len, get_block_t *get_block);
extern void get_filesystem(struct file_system_type *fs);
extern void put_filesystem(struct file_system_type *fs);
--
1.5.4.5
^ permalink raw reply related [flat|nested] 16+ messages in thread* [PATCH 3/3] ocfs2: fiemap support
2008-09-10 12:40 ` Mark Fasheh
` (2 preceding siblings ...)
2008-09-10 12:50 ` [PATCH 2/3] generic block based fiemap implementation Mark Fasheh
@ 2008-09-10 12:50 ` Mark Fasheh
3 siblings, 0 replies; 16+ messages in thread
From: Mark Fasheh @ 2008-09-10 12:50 UTC (permalink / raw)
To: Andrew Morton; +Cc: ocfs2-devel, linux-ext4
Plug ocfs2 into ->fiemap. Some portions of ocfs2_get_clusters() had to be
refactored so that the extent cache can be skipped in favor of going
directly to the on-disk records. This makes it easier for us to determine
which extent is the last one in the btree. Also, I'm not sure we want to be
caching fiemap lookups anyway as they're not directly related to data
read/write.
Signed-off-by: Mark Fasheh <mfasheh@suse.com>
---
fs/ocfs2/alloc.c | 9 --
fs/ocfs2/alloc.h | 9 ++
fs/ocfs2/extent_map.c | 347 +++++++++++++++++++++++++++++++++++++++++--------
fs/ocfs2/extent_map.h | 3 +
fs/ocfs2/file.c | 1 +
5 files changed, 307 insertions(+), 62 deletions(-)
diff --git a/fs/ocfs2/alloc.c b/fs/ocfs2/alloc.c
index 10bfb46..29ff57e 100644
--- a/fs/ocfs2/alloc.c
+++ b/fs/ocfs2/alloc.c
@@ -990,15 +990,6 @@ out:
}
/*
- * This is only valid for leaf nodes, which are the only ones that can
- * have empty extents anyway.
- */
-static inline int ocfs2_is_empty_extent(struct ocfs2_extent_rec *rec)
-{
- return !rec->e_leaf_clusters;
-}
-
-/*
* This function will discard the rightmost extent record.
*/
static void ocfs2_shift_records_right(struct ocfs2_extent_list *el)
diff --git a/fs/ocfs2/alloc.h b/fs/ocfs2/alloc.h
index 42ff94b..60cd3d5 100644
--- a/fs/ocfs2/alloc.h
+++ b/fs/ocfs2/alloc.h
@@ -146,4 +146,13 @@ static inline unsigned int ocfs2_rec_clusters(struct ocfs2_extent_list *el,
return le16_to_cpu(rec->e_leaf_clusters);
}
+/*
+ * This is only valid for leaf nodes, which are the only ones that can
+ * have empty extents anyway.
+ */
+static inline int ocfs2_is_empty_extent(struct ocfs2_extent_rec *rec)
+{
+ return !rec->e_leaf_clusters;
+}
+
#endif /* OCFS2_ALLOC_H */
diff --git a/fs/ocfs2/extent_map.c b/fs/ocfs2/extent_map.c
index c58668a..a4d5fc8 100644
--- a/fs/ocfs2/extent_map.c
+++ b/fs/ocfs2/extent_map.c
@@ -25,6 +25,7 @@
#include <linux/fs.h>
#include <linux/init.h>
#include <linux/types.h>
+#include <linux/fiemap.h>
#define MLOG_MASK_PREFIX ML_EXTENT_MAP
#include <cluster/masklog.h>
@@ -32,6 +33,7 @@
#include "ocfs2.h"
#include "alloc.h"
+#include "dlmglue.h"
#include "extent_map.h"
#include "inode.h"
#include "super.h"
@@ -282,6 +284,51 @@ out:
kfree(new_emi);
}
+static int ocfs2_last_eb_is_empty(struct inode *inode,
+ struct ocfs2_dinode *di)
+{
+ int ret, next_free;
+ u64 last_eb_blk = le64_to_cpu(di->i_last_eb_blk);
+ struct buffer_head *eb_bh = NULL;
+ struct ocfs2_extent_block *eb;
+ struct ocfs2_extent_list *el;
+
+ ret = ocfs2_read_block(OCFS2_SB(inode->i_sb), last_eb_blk,
+ &eb_bh, OCFS2_BH_CACHED, inode);
+ if (ret) {
+ mlog_errno(ret);
+ goto out;
+ }
+
+ eb = (struct ocfs2_extent_block *) eb_bh->b_data;
+ el = &eb->h_list;
+
+ if (!OCFS2_IS_VALID_EXTENT_BLOCK(eb)) {
+ ret = -EROFS;
+ OCFS2_RO_ON_INVALID_EXTENT_BLOCK(inode->i_sb, eb);
+ goto out;
+ }
+
+ if (el->l_tree_depth) {
+ ocfs2_error(inode->i_sb,
+ "Inode %lu has non zero tree depth in "
+ "leaf block %llu\n", inode->i_ino,
+ (unsigned long long)eb_bh->b_blocknr);
+ ret = -EROFS;
+ goto out;
+ }
+
+ next_free = le16_to_cpu(el->l_next_free_rec);
+
+ if (next_free == 0 ||
+ (next_free == 1 && ocfs2_is_empty_extent(&el->l_recs[0])))
+ ret = 1;
+
+out:
+ brelse(eb_bh);
+ return ret;
+}
+
/*
* Return the 1st index within el which contains an extent start
* larger than v_cluster.
@@ -373,42 +420,28 @@ out:
return ret;
}
-int ocfs2_get_clusters(struct inode *inode, u32 v_cluster,
- u32 *p_cluster, u32 *num_clusters,
- unsigned int *extent_flags)
+static int ocfs2_get_clusters_nocache(struct inode *inode,
+ struct buffer_head *di_bh,
+ u32 v_cluster, unsigned int *hole_len,
+ struct ocfs2_extent_rec *ret_rec,
+ unsigned int *is_last)
{
- int ret, i;
- unsigned int flags = 0;
- struct buffer_head *di_bh = NULL;
- struct buffer_head *eb_bh = NULL;
+ int i, ret, tree_height, len;
struct ocfs2_dinode *di;
- struct ocfs2_extent_block *eb;
+ struct ocfs2_extent_block *uninitialized_var(eb);
struct ocfs2_extent_list *el;
struct ocfs2_extent_rec *rec;
- u32 coff;
-
- if (OCFS2_I(inode)->ip_dyn_features & OCFS2_INLINE_DATA_FL) {
- ret = -ERANGE;
- mlog_errno(ret);
- goto out;
- }
-
- ret = ocfs2_extent_map_lookup(inode, v_cluster, p_cluster,
- num_clusters, extent_flags);
- if (ret == 0)
- goto out;
+ struct buffer_head *eb_bh = NULL;
- ret = ocfs2_read_block(OCFS2_SB(inode->i_sb), OCFS2_I(inode)->ip_blkno,
- &di_bh, OCFS2_BH_CACHED, inode);
- if (ret) {
- mlog_errno(ret);
- goto out;
- }
+ memset(ret_rec, 0, sizeof(*ret_rec));
+ if (is_last)
+ *is_last = 0;
di = (struct ocfs2_dinode *) di_bh->b_data;
el = &di->id2.i_list;
+ tree_height = le16_to_cpu(el->l_tree_depth);
- if (el->l_tree_depth) {
+ if (tree_height > 0) {
ret = ocfs2_find_leaf(inode, el, v_cluster, &eb_bh);
if (ret) {
mlog_errno(ret);
@@ -431,46 +464,143 @@ int ocfs2_get_clusters(struct inode *inode, u32 v_cluster,
i = ocfs2_search_extent_list(el, v_cluster);
if (i == -1) {
/*
- * A hole was found. Return some canned values that
- * callers can key on. If asked for, num_clusters will
- * be populated with the size of the hole.
+ * Holes can be larger than the maximum size of an
+ * extent, so we return their lengths in a seperate
+ * field.
*/
- *p_cluster = 0;
- if (num_clusters) {
+ if (hole_len) {
ret = ocfs2_figure_hole_clusters(inode, el, eb_bh,
- v_cluster,
- num_clusters);
+ v_cluster, &len);
if (ret) {
mlog_errno(ret);
goto out;
}
+
+ *hole_len = len;
}
- } else {
- rec = &el->l_recs[i];
+ goto out_hole;
+ }
- BUG_ON(v_cluster < le32_to_cpu(rec->e_cpos));
+ rec = &el->l_recs[i];
- if (!rec->e_blkno) {
- ocfs2_error(inode->i_sb, "Inode %lu has bad extent "
- "record (%u, %u, 0)", inode->i_ino,
- le32_to_cpu(rec->e_cpos),
- ocfs2_rec_clusters(el, rec));
- ret = -EROFS;
- goto out;
+ BUG_ON(v_cluster < le32_to_cpu(rec->e_cpos));
+
+ if (!rec->e_blkno) {
+ ocfs2_error(inode->i_sb, "Inode %lu has bad extent "
+ "record (%u, %u, 0)", inode->i_ino,
+ le32_to_cpu(rec->e_cpos),
+ ocfs2_rec_clusters(el, rec));
+ ret = -EROFS;
+ goto out;
+ }
+
+ *ret_rec = *rec;
+
+ /*
+ * Checking for last extent is potentially expensive - we
+ * might have to look at the next leaf over to see if it's
+ * empty.
+ *
+ * The first two checks are to see whether the caller even
+ * cares for this information, and if the extent is at least
+ * the last in it's list.
+ *
+ * If those hold true, then the extent is last if any of the
+ * additional conditions hold true:
+ * - Extent list is in-inode
+ * - Extent list is right-most
+ * - Extent list is 2nd to rightmost, with empty right-most
+ */
+ if (is_last) {
+ if (i == (le16_to_cpu(el->l_next_free_rec) - 1)) {
+ if (tree_height == 0)
+ *is_last = 1;
+ else if (eb->h_blkno == di->i_last_eb_blk)
+ *is_last = 1;
+ else if (eb->h_next_leaf_blk == di->i_last_eb_blk) {
+ ret = ocfs2_last_eb_is_empty(inode, di);
+ if (ret < 0) {
+ mlog_errno(ret);
+ goto out;
+ }
+ if (ret == 1)
+ *is_last = 1;
+ }
}
+ }
+
+out_hole:
+ ret = 0;
+out:
+ brelse(eb_bh);
+ return ret;
+}
+
+static void ocfs2_relative_extent_offsets(struct super_block *sb,
+ u32 v_cluster,
+ struct ocfs2_extent_rec *rec,
+ u32 *p_cluster, u32 *num_clusters)
+
+{
+ u32 coff = v_cluster - le32_to_cpu(rec->e_cpos);
+
+ *p_cluster = ocfs2_blocks_to_clusters(sb, le64_to_cpu(rec->e_blkno));
+ *p_cluster = *p_cluster + coff;
+
+ if (num_clusters)
+ *num_clusters = le16_to_cpu(rec->e_leaf_clusters) - coff;
+}
+
+int ocfs2_get_clusters(struct inode *inode, u32 v_cluster,
+ u32 *p_cluster, u32 *num_clusters,
+ unsigned int *extent_flags)
+{
+ int ret;
+ unsigned int uninitialized_var(hole_len), flags = 0;
+ struct buffer_head *di_bh = NULL;
+ struct ocfs2_extent_rec rec;
- coff = v_cluster - le32_to_cpu(rec->e_cpos);
+ if (OCFS2_I(inode)->ip_dyn_features & OCFS2_INLINE_DATA_FL) {
+ ret = -ERANGE;
+ mlog_errno(ret);
+ goto out;
+ }
- *p_cluster = ocfs2_blocks_to_clusters(inode->i_sb,
- le64_to_cpu(rec->e_blkno));
- *p_cluster = *p_cluster + coff;
+ ret = ocfs2_extent_map_lookup(inode, v_cluster, p_cluster,
+ num_clusters, extent_flags);
+ if (ret == 0)
+ goto out;
- if (num_clusters)
- *num_clusters = ocfs2_rec_clusters(el, rec) - coff;
+ ret = ocfs2_read_block(OCFS2_SB(inode->i_sb), OCFS2_I(inode)->ip_blkno,
+ &di_bh, OCFS2_BH_CACHED, inode);
+ if (ret) {
+ mlog_errno(ret);
+ goto out;
+ }
- flags = rec->e_flags;
+ ret = ocfs2_get_clusters_nocache(inode, di_bh, v_cluster, &hole_len,
+ &rec, NULL);
+ if (ret) {
+ mlog_errno(ret);
+ goto out;
+ }
- ocfs2_extent_map_insert_rec(inode, rec);
+ if (rec.e_blkno == 0ULL) {
+ /*
+ * A hole was found. Return some canned values that
+ * callers can key on. If asked for, num_clusters will
+ * be populated with the size of the hole.
+ */
+ *p_cluster = 0;
+ if (num_clusters) {
+ *num_clusters = hole_len;
+ }
+ } else {
+ ocfs2_relative_extent_offsets(inode->i_sb, v_cluster, &rec,
+ p_cluster, num_clusters);
+ flags = rec.e_flags;
+
+ ocfs2_extent_map_insert_rec(inode, &rec);
}
if (extent_flags)
@@ -478,7 +608,6 @@ int ocfs2_get_clusters(struct inode *inode, u32 v_cluster,
out:
brelse(di_bh);
- brelse(eb_bh);
return ret;
}
@@ -521,3 +650,115 @@ int ocfs2_extent_map_get_blocks(struct inode *inode, u64 v_blkno, u64 *p_blkno,
out:
return ret;
}
+
+static int ocfs2_fiemap_inline(struct inode *inode, struct buffer_head *di_bh,
+ struct fiemap_extent_info *fieinfo,
+ u64 map_start)
+{
+ int ret;
+ unsigned int id_count;
+ struct ocfs2_dinode *di;
+ u64 phys;
+ u32 flags = FIEMAP_EXTENT_DATA_INLINE|FIEMAP_EXTENT_LAST;
+ struct ocfs2_inode_info *oi = OCFS2_I(inode);
+
+ di = (struct ocfs2_dinode *)di_bh->b_data;
+ id_count = le16_to_cpu(di->id2.i_data.id_count);
+
+ if (map_start < id_count) {
+ phys = oi->ip_blkno << inode->i_sb->s_blocksize_bits;
+ phys += offsetof(struct ocfs2_dinode, id2.i_data.id_data);
+
+ ret = fiemap_fill_next_extent(fieinfo, 0, phys, id_count,
+ flags, inode->i_sb->s_dev);
+ if (ret < 0)
+ return ret;
+ }
+
+ return 0;
+}
+
+#define OCFS2_FIEMAP_FLAGS (FIEMAP_FLAG_SYNC)
+
+int ocfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 map_start, u64 map_len)
+{
+ int ret, is_last;
+ u32 mapping_end, cpos;
+ unsigned int hole_size;
+ struct ocfs2_super *osb = OCFS2_SB(inode->i_sb);
+ u64 len_bytes, phys_bytes, virt_bytes;
+ struct buffer_head *di_bh = NULL;
+ struct ocfs2_extent_rec rec;
+
+ ret = fiemap_check_flags(fieinfo, OCFS2_FIEMAP_FLAGS);
+ if (ret)
+ return ret;
+
+ ret = ocfs2_inode_lock(inode, &di_bh, 0);
+ if (ret) {
+ mlog_errno(ret);
+ goto out;
+ }
+
+ down_read(&OCFS2_I(inode)->ip_alloc_sem);
+
+ /*
+ * Handle inline-data separately.
+ */
+ if (OCFS2_I(inode)->ip_dyn_features & OCFS2_INLINE_DATA_FL) {
+ ret = ocfs2_fiemap_inline(inode, di_bh, fieinfo, map_start);
+ goto out_unlock;
+ }
+
+ cpos = map_start >> osb->s_clustersize_bits;
+ mapping_end = ocfs2_clusters_for_bytes(inode->i_sb,
+ map_start + map_len);
+ mapping_end -= cpos;
+ is_last = 0;
+ while (cpos < mapping_end && !is_last) {
+ u32 fe_flags;
+
+ ret = ocfs2_get_clusters_nocache(inode, di_bh, cpos,
+ &hole_size, &rec, &is_last);
+ if (ret) {
+ mlog_errno(ret);
+ goto out;
+ }
+
+ if (rec.e_blkno == 0ULL) {
+ cpos += hole_size;
+ continue;
+ }
+
+ fe_flags = 0;
+ if (rec.e_flags & OCFS2_EXT_UNWRITTEN)
+ fe_flags |= FIEMAP_EXTENT_UNWRITTEN;
+ if (is_last)
+ fe_flags |= FIEMAP_EXTENT_LAST;
+ len_bytes = (u64)le16_to_cpu(rec.e_leaf_clusters) << osb->s_clustersize_bits;
+ phys_bytes = le64_to_cpu(rec.e_blkno) << osb->sb->s_blocksize_bits;
+ virt_bytes = (u64)le32_to_cpu(rec.e_cpos) << osb->s_clustersize_bits;
+
+ ret = fiemap_fill_next_extent(fieinfo, virt_bytes, phys_bytes,
+ len_bytes, fe_flags,
+ inode->i_sb->s_dev);
+ if (ret)
+ break;
+
+ cpos = le32_to_cpu(rec.e_cpos)+ le16_to_cpu(rec.e_leaf_clusters);
+ }
+
+ if (ret > 0)
+ ret = 0;
+
+out_unlock:
+ brelse(di_bh);
+
+ up_read(&OCFS2_I(inode)->ip_alloc_sem);
+
+ ocfs2_inode_unlock(inode, 0);
+out:
+
+ return ret;
+}
diff --git a/fs/ocfs2/extent_map.h b/fs/ocfs2/extent_map.h
index de91e3e..1b97490 100644
--- a/fs/ocfs2/extent_map.h
+++ b/fs/ocfs2/extent_map.h
@@ -50,4 +50,7 @@ int ocfs2_get_clusters(struct inode *inode, u32 v_cluster, u32 *p_cluster,
int ocfs2_extent_map_get_blocks(struct inode *inode, u64 v_blkno, u64 *p_blkno,
u64 *ret_count, unsigned int *extent_flags);
+int ocfs2_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,
+ u64 map_start, u64 map_len);
+
#endif /* _EXTENT_MAP_H */
diff --git a/fs/ocfs2/file.c b/fs/ocfs2/file.c
index ec2ed15..ed38796 100644
--- a/fs/ocfs2/file.c
+++ b/fs/ocfs2/file.c
@@ -2228,6 +2228,7 @@ const struct inode_operations ocfs2_file_iops = {
.getattr = ocfs2_getattr,
.permission = ocfs2_permission,
.fallocate = ocfs2_fallocate,
+ .fiemap = ocfs2_fiemap,
};
const struct inode_operations ocfs2_special_file_iops = {
--
1.5.4.5
^ permalink raw reply related [flat|nested] 16+ messages in thread