[RFD] : Allowing xfs_fsr to defragment portions of a file

[RFD] : Allowing xfs_fsr to defragment portions of a file

[Dhruvesh Rathore]

Hi,

The xfs_fsr utility in its current state does not support defragmentation of
parts of file. Either the entire file is defragmented, or if free space is not
available then the operation is not performed.
We would like to enhance the utility allowing it to defragment parts
of the file, ins such a condition.
---------------------------------------------------------------------------------------------------------------

Our initial task was to understand how xfs_fsr currently defragments the file:

1) In /fsr/xfs_fsr.c from main() the control is passed to fsrfile().
In fsrfile(), we
open the original file and collect bstat information. A call is then made to
fsrfile_common()

2) In fsrfile_common() we check if sufficient free space is available to
defragment the file, and if not then we exit. If free space is available, then a
call is made to packfile()

3) In packfile(), read_fd_bmap() function provides us with the current number of
extents used by the file (cur_nextents) and the ideal number of extents needed
to save the file (nextents). In function read_fd_bmap() the the
outmap[ ] structure
is also populated.

4) packfile() will initially allocate new extents (nextents) to the
temporary file and
then copy(swap) extents form the existing file. We then fchown() the new temp
file and space occupied by the prev fragmented copy is then freed.

---------------------------------------------------------------------------------------------------------------

Our understanding to the approach of defragmenting portions of a file
is as follows:

1) In fsrfile_common(), a new function will be called when the code
determines that
there is insufficient free space to defragment the entire file.

2) The new function performs the following tasks:

---> Call read_fd_bmap() to get cur_nextents and nextents(ideal). Using
nextents we will
then have to determine how many extents can be defragmented against
the free space
available.

---> Creating a temp file will not be feasible as it will create a
problem when we want to
change owners.

---> The function will have to hence focus on swapping the existing
extents from the
fragmented file. For this a swap_extent() function will be needed,
which will take arguments
as the current extent and the location(in free space) where it needs
to be swapped to.

For the above functionality Dave suggested that we have a look at
EXT4_IOC_MOVE_EXT
and its usage.

We are however unsure of how to compress the extents which have been chosen for
defragmentation and keep the other extents untouched. We are also in
doubt about how to
proceed without creating a temp file for defragmentation.


Any thoughts, comments, opinions etc, are welcome
Thank you.

Regards
A-DRS.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFD] : Allowing xfs_fsr to defragment portions of a file

[Dave Chinner]

On Sun, Feb 15, 2015 at 08:08:46PM +0530, Dhruvesh Rathore wrote:
> Hi,
> 
> The xfs_fsr utility in its current state does not support defragmentation of
> parts of file. Either the entire file is defragmented, or if free space is not
> available then the operation is not performed.
> We would like to enhance the utility allowing it to defragment parts
> of the file, ins such a condition.
> ---------------------------------------------------------------------------------------------------------------
> 
> Our initial task was to understand how xfs_fsr currently defragments the file:
> 
> 1) In /fsr/xfs_fsr.c from main() the control is passed to fsrfile().
> In fsrfile(), we
> open the original file and collect bstat information. A call is then made to
> fsrfile_common()
> 
> 2) In fsrfile_common() we check if sufficient free space is available to
> defragment the file, and if not then we exit. If free space is available, then a
> call is made to packfile()
> 
> 3) In packfile(), read_fd_bmap() function provides us with the current number of
> extents used by the file (cur_nextents) and the ideal number of extents needed
> to save the file (nextents). In function read_fd_bmap() the the
> outmap[ ] structure
> is also populated.
> 
> 4) packfile() will initially allocate new extents (nextents) to the
> temporary file and
> then copy(swap) extents form the existing file. We then fchown() the new temp
> file and space occupied by the prev fragmented copy is then freed.
> 
> ---------------------------------------------------------------------------------------------------------------
> 
> Our understanding to the approach of defragmenting portions of a file
> is as follows:
> 
> 1) In fsrfile_common(), a new function will be called when the code
> determines that
> there is insufficient free space to defragment the entire file.
> 
> 2) The new function performs the following tasks:
> 
> ---> Call read_fd_bmap() to get cur_nextents and nextents(ideal). Using
> nextents we will
> then have to determine how many extents can be defragmented against
> the free space
> available.
> 
> ---> Creating a temp file will not be feasible as it will create a
> problem when we want to
> change owners.

fsr runs as root. There is no issue with changing owners, though
that should not be necessary with extent swap operations between
two files. The tempfile creation is the same as fsr is currently
already doing for full file defragmentation, so you shouldn't need
to be doing anything different here.

> ---> The function will have to hence focus on swapping the existing
> extents from the
> fragmented file. For this a swap_extent() function will be needed,
> which will take arguments
> as the current extent and the location(in free space) where it needs
> to be swapped to.

You can't swap an in use extent with a free space location. You need
to preallocate space in a temporary file, copy the data across to
it, and then swap the extent ranges between the two files.

> For the above functionality Dave suggested that we have a look at
> EXT4_IOC_MOVE_EXT
> and its usage.

The extent swap between two files is exactly what that ioctl does.
Look at the API, especially the struct move_extent that defines the
information passed to the kernel to perform the extent swap
operation. The XFS APi will be similar.

> We are however unsure of how to compress the extents which have been chosen for
> defragmentation and keep the other extents untouched. We are also in
> doubt about how to
> proceed without creating a temp file for defragmentation.

The process in fsr is this:

	1. bulkstat original file inode [bstat] for change
	   detection whilst doing copying operation.
	2. get block map of range [off, len] of orignal file
	   that needs to be swapped
	3. create temp file
	4. preallocate extent [toff, tlen] in temp file
	5. check the number of extents in [toff, tlen] is less than
	   the number of extents in [off, len] in the original file.
	   If it's not an improvement, abort.
	6. copy data from [orig, off, len] to [temp, toff, tlen]
	7. ioctl(orig_fd, XFS_IOC_SWAPEXT_RANGE, [bstat, off, len, temp, toff])

The kernel side of the ioctl is somewhat more complex, and we'll get
to that once the fsr infrastructure is ready to use it.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFD] : Allowing xfs_fsr to defragment portions of a file

[Dhruvesh Rathore]

On Tue, Feb 17, 2015 at 5:18 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Sun, Feb 15, 2015 at 08:08:46PM +0530, Dhruvesh Rathore wrote:

>> A new function will have to hence focus on swapping the existing
>> extents from the
>> fragmented file. For this a new_swap_extent() function will be needed,
>> which will take arguments
>> as the current extent and the location(in free space) where it needs
>> to be swapped to.
>
> You can't swap an in use extent with a free space location. You need
> to preallocate space in a temporary file, copy the data across to
> it, and then swap the extent ranges between the two files.
>
>> For the above functionality Dave suggested that we have a look at
>> EXT4_IOC_MOVE_EXT
>> and its usage.
>
> The extent swap between two files is exactly what that ioctl does.
> Look at the API, especially the struct move_extent that defines the
> information passed to the kernel to perform the extent swap
> operation. The XFS APi will be similar.
>

Over the past few days we have been going over the XFS API for
EXT4_IOC_MOVE_EXT available in misc/e4defrag.c, to lay down the
groundwork of the required XFS API.

-----

These are a few things that we felt do not be changed in the new API:

1) The original file will be opened in fsrfile() in xfs_fsr.c, similar to the
original file being opened in file_defrag() in e4defrag.c

2) The check for sufficient freespace done in fsrfile_common() in
xfs_fsr.c achieves the same objective as the check_freesize() in
e4defrag.c

-----

On the same lines, we have identified the following additions
that will be needed in the XFS API for fsr/xfs_fsr.c

1) The read_fd_bmap() function in xfs_fsr.c which gets the block map
of range of original file (map) and the defragmented file (outmap), will
need to be added to when we are performing partial defragmentation.

A function get_file_extents() is called in e4defrag.c twice, which populates
a struct fiemap, struct fiemap_extent and struct fiemap_extent_list
for both original file and donor file (temp file).

The operation of this function will need to be added in xfs_fsr.c as the
EXT4_IOC_MOVE_EXT ioctl primarily uses these fiemap structures.
We will need both the fiemap structures and the existing part of read_fd_bmap( )
in this new XFS API.


2) The temp file getting created in xfs_fsr.c, will be made similar to
the donor file
of e4defrag.c.
We were a bit unsure about the function of join_extents() in
e4defrag.c, which finds a
continuous region (extents group), and how it will fit in the XFS API.
Is it similar to the process of preallocating extents[toff, tlen] in temp file ?


3) Now instead of copying data from [orig, off, len] to [temp, toff,
tlen] and then calling
XFS_IOC_SWPEXT, we will need to simulate the ioctl EXT4_IOC_MOVE_EXT.

In call_defrag( ) in e4defrag.c, the struct fiemap_extent_list
structure, which is basically
a link list of extent information is used.
In our XFS API we will similarly use this structure in a loop and call
the IOCTL for
each of the extents (off donor file) which need to be filled by
swapping data blocks.

In this loop the fields orig_start, donor_start, len and moved_len of
struct move_extent
will be populated from the fiemap_extent_list structure before each ioctl call.

-----

Before all of the above we will need to select the extents that can be
defragmented
i.e the best possible choice, and then mark them.
For this we decided to create another data structure - an array
(user_array[ ]), which
will store extent size information (blocks).
For instance say a file has 5 extents then
user_array[0] : save size of extent 1
user_array[1] : save size of extents 1, 2
user_array[2] : save size of extents 1, 2, 3
user_array[3] : save size of extents 1, 2, 3, 4
user_array[4] : save size of extents 1, 2 3, 4, 5
This way we could perform the selection efficiently, by necessary
comparison with
available space.

----

We would love to hear your comments or directions on the above points. After
confirmation from you, we will finish creating the XFS API on the above lines.

> The process in fsr is this:
>
>         1. bulkstat original file inode [bstat] for change
>            detection whilst doing copying operation.
>         2. get block map of range [off, len] of orignal file
>            that needs to be swapped
>         3. create temp file
>         4. preallocate extent [toff, tlen] in temp file
>         5. check the number of extents in [toff, tlen] is less than
>            the number of extents in [off, len] in the original file.
>            If it's not an improvement, abort.
>         6. copy data from [orig, off, len] to [temp, toff, tlen]
>         7. ioctl(orig_fd, XFS_IOC_SWAPEXT_RANGE, [bstat, off, len, temp, toff])
>
> The kernel side of the ioctl is somewhat more complex, and we'll get
> to that once the fsr infrastructure is ready to use it.
>
> Cheers,
>
> Dave.


Regards,
A-DRS

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFD] : Allowing xfs_fsr to defragment portions of a file

[Dhruvesh Rathore]

On Tue, Feb 17, 2015 at 5:18 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Sun, Feb 15, 2015 at 08:08:46PM +0530, Dhruvesh Rathore wrote:

>> A new function will have to hence focus on swapping the existing
>> extents from the
>> fragmented file. For this a new_swap_extent() function will be needed,
>> which will take arguments
>> as the current extent and the location(in free space) where it needs
>> to be swapped to.
>
> You can't swap an in use extent with a free space location. You need
> to preallocate space in a temporary file, copy the data across to
> it, and then swap the extent ranges between the two files.
>
>> For the above functionality Dave suggested that we have a look at
>> EXT4_IOC_MOVE_EXT
>> and its usage.
>
> The extent swap between two files is exactly what that ioctl does.
> Look at the API, especially the struct move_extent that defines the
> information passed to the kernel to perform the extent swap
> operation. The XFS APi will be similar.
>

Over the past few days we have been going over the XFS API for
EXT4_IOC_MOVE_EXT available in misc/e4defrag.c, to lay down the
groundwork of the required XFS API.

-----

These are a few things that we felt do not be changed in the new API:

1) The original file will be opened in fsrfile() in xfs_fsr.c, similar to the
original file being opened in file_defrag() in e4defrag.c

2) The check for sufficient freespace done in fsrfile_common() in
xfs_fsr.c achieves the same objective as the check_freesize() in
e4defrag.c

-----

On the same lines, we have identified the following additions
that will be needed in the XFS API for fsr/xfs_fsr.c

1) The read_fd_bmap() function in xfs_fsr.c which gets the block map
of range of original file (map) and the defragmented file (outmap), will
need to be added to when we are performing partial defragmentation.

A function get_file_extents() is called in e4defrag.c twice, which populates
a struct fiemap, struct fiemap_extent and struct fiemap_extent_list
for both original file and donor file (temp file).

The operation of this function will need to be added in xfs_fsr.c as the
EXT4_IOC_MOVE_EXT ioctl primarily uses these fiemap structures.
We will need both the fiemap structures and the existing part of read_fd_bmap( )
in this new XFS API.
(P.S We have already achieved the above using XFS_IOC_FIEMAPFS, and have
got extent information in fiemap structures)


2) The temp file getting created in xfs_fsr.c, will be made similar to
the donor file of e4defrag.c.
We were a bit unsure about the function of join_extents() in
e4defrag.c, which finds a continuous region (extents group), and how
it will fit in the XFS API.
Is it similar to the process of preallocating extents[toff, tlen] in temp file ?


3) Now instead of copying data from [orig, off, len] to [temp, toff,
tlen] and then calling
XFS_IOC_SWPEXT, we will need to simulate the ioctl EXT4_IOC_MOVE_EXT.

In call_defrag( ) in e4defrag.c, the struct fiemap_extent_list
structure, which is basically
a link list of extent information is used.
In our XFS API we will similarly use this structure in a loop and call
the IOCTL for
each of the extents (off donor file) which needs to be filled by
swapping data blocks.

In this loop the fields orig_start, donor_start, len and moved_len of
struct move_extent
will be populated from the fiemap_extent_list structure before each ioctl call.

-----

Before all of the above we will need to select the extents that can be
defragmented
i.e the best possible choice, and then mark them.
For this we decided to create another data structure - an array
(user_array[ ]), which
will store extent size information (blocks).
For instance say a file has 5 extents then
user_array[0] : save size of extent 1
user_array[1] : save size of extents 1, 2
user_array[2] : save size of extents 1, 2, 3
user_array[3] : save size of extents 1, 2, 3, 4
user_array[4] : save size of extents 1, 2 3, 4, 5
This way we could perform the selection efficiently, by necessary
comparison with
available space.

----

As said, we have already added to the XFS API, and have got extent info
in fiemap structures.
We would love to hear your comments or directions on the above points. After
confirmation from you, we will finish creating the XFS API on the above lines.

> The process in fsr is this:
>
>         1. bulkstat original file inode [bstat] for change
>            detection whilst doing copying operation.
>         2. get block map of range [off, len] of orignal file
>            that needs to be swapped
>         3. create temp file
>         4. preallocate extent [toff, tlen] in temp file
>         5. check the number of extents in [toff, tlen] is less than
>            the number of extents in [off, len] in the original file.
>            If it's not an improvement, abort.
>         6. copy data from [orig, off, len] to [temp, toff, tlen]
>         7. ioctl(orig_fd, XFS_IOC_SWAPEXT_RANGE, [bstat, off, len, temp, toff])
>
> The kernel side of the ioctl is somewhat more complex, and we'll get
> to that once the fsr infrastructure is ready to use it.
>
> Cheers,
>
> Dave.


Regards,
A-DRS

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

Re: [RFD] : Allowing xfs_fsr to defragment portions of a file

[Dhruvesh Rathore]

On Tue, Feb 17, 2015 at 5:18 AM, Dave Chinner <david@fromorbit.com> wrote:
> On Sun, Feb 15, 2015 at 08:08:46PM +0530, Dhruvesh Rathore wrote:

>> A new function will have to hence focus on swapping the existing
>> extents from the
>> fragmented file. For this a new_swap_extent() function will be needed,
>> which will take arguments
>> as the current extent and the location(in free space) where it needs
>> to be swapped to.
>
> You can't swap an in use extent with a free space location. You need
> to preallocate space in a temporary file, copy the data across to
> it, and then swap the extent ranges between the two files.
>
>> For the above functionality Dave suggested that we have a look at
>> EXT4_IOC_MOVE_EXT
>> and its usage.
>
> The extent swap between two files is exactly what that ioctl does.
> Look at the API, especially the struct move_extent that defines the
> information passed to the kernel to perform the extent swap
> operation. The XFS APi will be similar.
>

Over the past few days we have been going over the XFS API for
EXT4_IOC_MOVE_EXT available in misc/e4defrag.c, to lay down the
groundwork of the required XFS API.

-----

These are a few things that we felt do not be changed in the new API:

1) The original file will be opened in fsrfile() in xfs_fsr.c, similar to the
original file being opened in file_defrag() in e4defrag.c

2) The check for sufficient freespace done in fsrfile_common() in
xfs_fsr.c achieves the same objective as the check_freesize() in
e4defrag.c

-----

On the same lines, we have identified the following additions
that will be needed in the XFS API for fsr/xfs_fsr.c

1) The read_fd_bmap() function in xfs_fsr.c which gets the block map
of range of original file (map) and the defragmented file (outmap), will
need to be added to when we are performing partial defragmentation.

A function get_file_extents() is called in e4defrag.c twice, which populates
a struct fiemap, struct fiemap_extent and struct fiemap_extent_list
for both original file and donor file (temp file).

The operation of this function will need to be added in xfs_fsr.c as the
EXT4_IOC_MOVE_EXT ioctl primarily uses these fiemap structures.
We will need both the fiemap structures and the existing part of read_fd_bmap( )
in this new XFS API.
(P.S We have already achieved the above using XFS_IOC_FIEMAPFS, and have
got extent information in fiemap structures)


2) The temp file getting created in xfs_fsr.c, will be made similar to
the donor file of e4defrag.c.
We were a bit unsure about the function of join_extents() in
e4defrag.c, which finds a continuous region (extents group), and how
it will fit in the XFS API.
Is it similar to the process of preallocating extents[toff, tlen] in temp file ?


3) Now instead of copying data from [orig, off, len] to [temp, toff,
tlen] and then calling
XFS_IOC_SWPEXT, we will need to simulate the ioctl EXT4_IOC_MOVE_EXT.

In call_defrag( ) in e4defrag.c, the struct fiemap_extent_list
structure, which is basically
a link list of extent information is used.
In our XFS API we will similarly use this structure in a loop and call
the IOCTL for
each of the extents (off donor file) which needs to be filled by
swapping data blocks.

In this loop the fields orig_start, donor_start, len and moved_len of
struct move_extent
will be populated from the fiemap_extent_list structure before each ioctl call.

-----

Before all of the above we will need to select the extents that can be
defragmented
i.e the best possible choice, and then mark them.
For this we decided to create another data structure - an array
(user_array[ ]), which
will store extent size information (blocks).
For instance say a file has 5 extents then
user_array[0] : save size of extent 1
user_array[1] : save size of extents 1, 2
user_array[2] : save size of extents 1, 2, 3
user_array[3] : save size of extents 1, 2, 3, 4
user_array[4] : save size of extents 1, 2 3, 4, 5
This way we could perform the selection efficiently, by necessary
comparison with
available space.

----

As said, we have already added to the XFS API, and have got extent info
in fiemap structures.


We would love to hear your comments or directions on the above points. After
confirmation from you, we will finish creating the XFS API on the above lines.

> The process in fsr is this:
>
>         1. bulkstat original file inode [bstat] for change
>            detection whilst doing copying operation.
>         2. get block map of range [off, len] of orignal file
>            that needs to be swapped
>         3. create temp file
>         4. preallocate extent [toff, tlen] in temp file
>         5. check the number of extents in [toff, tlen] is less than
>            the number of extents in [off, len] in the original file.
>            If it's not an improvement, abort.
>         6. copy data from [orig, off, len] to [temp, toff, tlen]
>         7. ioctl(orig_fd, XFS_IOC_SWAPEXT_RANGE, [bstat, off, len, temp, toff])
>
> The kernel side of the ioctl is somewhat more complex, and we'll get
> to that once the fsr infrastructure is ready to use it.
>
> Cheers,
>
> Dave.

Regards,
A-DRS

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs

[RFD] Allowing xfs_fsr to defragment portions of a file

[Dhruvesh Rathore]

Hi,

The xfs_fsr utility in its current state does not support defragmentation of
parts of file. Either the entire file is defragmented, or if free space is not
available then the operation is not performed.
We would like to enhance the utility allowing it to defragment parts
of the file,
when the free space is not available to defragment the entire file.

---------------------------------------------------------------------------------------------------------------

Our initial task was to understand how xfs_fsr currently defragments the file:

1) In /fsr/xfs_fsr.c from main() the control is passed to fsrfile().
In fsrfile(), we
open the original file and collect bstat information. A call is then made to
fsrfile_common()

2) In fsrfile_common() we check if sufficient free space is available to
defragment the file, and if not then we exit. If free space is available, then a
call is made to packfile()

3) In packfile(), read_fd_bmap() function provides us with the current number of
extents used by the file (cur_nextents) and the ideal number of extents needed
to save the file (nextents). In function read_fd_bmap() the the
outmap[ ] structure
is also populated.

4) packfile() will initially allocate new extents (nextents) to the
temporary file and
then copy(swap) extents form the existing file. We then fchown() the new temp
file and space occupied by the prev fragmented copy is then freed.

---------------------------------------------------------------------------------------------------------------

Our understanding to the approach of defragmenting portions of a file
is as follows:

1) In fsrfile_common(), a new function will be called when the code
determines that
there is insufficient free space to defragment the entire file.

2) The new function performs the following tasks:
-> Call read_fd_bmap() to get cur_nextents and nextents(ideal). Using
nextents we will
then have to determine how many extents can be defragmented against
the free space
available.
-> Creating a temp file will not be feasible as it will create a
problem when we want to
change owners.
-> The function will have to hence focus on swapping the existing
extents from the
fragmented file. For this a swap_extent() function will be needed,
which will take arguments
as the current extent and the location(in free space) where it needs
to be swapped to.

For the above functionality Dave suggested that we have a look at
EXT4_IOC_MOVE_EXT
and its usage.

We are however unsure of how to compress the extents which have been chosen for
defragmentation and keep the other extents untouched. We are also in
doubt about how to
proceed without creating a temp file for defragmentation.


Any thoughts, comments, opinions etc, are welcome
Thank you.

Regards
A-DRS.

_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs