Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Sai Chaitanya Mitta]

Hi Team,
           Is there any method/tool available to explicitly mark XFS
file extents as written? One approach I
am aware is explicitly zeroing the entire file (this file may be even
in hundreds of GB in size) through
synchronous/asynchronous(aio/io_uring) mechanism but it is time taking
process for large files,
is there any optimization/approach we can do to explicitly zeroing
file/mark extents as written?


Synchronous Approach:
                    while offset < size {
                        let bytes_written = img_file
                            .write_at(&buf, offset)
                            .map_err(|e| {
                                error!("Failed to zero out file: {}
error: {:?}", vol_name, e);
                            })?;
                        if offset == size {
                            break;
                        }
                        offset = offset + bytes_written as u64;
                    }
                    img_file.sync_all();

Asynchronous approach:
                   Currently used fio with libaio as ioengine but
results are almost same.

-- 
Thanks& Regards,
M.Sai Chaithanya.

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Darrick J. Wong]

On Mon, Dec 23, 2024 at 10:12:32PM +0530, Sai Chaitanya Mitta wrote:
> Hi Team,
>            Is there any method/tool available to explicitly mark XFS
> file extents as written? One approach I
> am aware is explicitly zeroing the entire file (this file may be even
> in hundreds of GB in size) through
> synchronous/asynchronous(aio/io_uring) mechanism but it is time taking
> process for large files,
> is there any optimization/approach we can do to explicitly zeroing
> file/mark extents as written?

Why do you need to mark them written?

--D

> 
> Synchronous Approach:
>                     while offset < size {
>                         let bytes_written = img_file
>                             .write_at(&buf, offset)
>                             .map_err(|e| {
>                                 error!("Failed to zero out file: {}
> error: {:?}", vol_name, e);
>                             })?;
>                         if offset == size {
>                             break;
>                         }
>                         offset = offset + bytes_written as u64;
>                     }
>                     img_file.sync_all();
> 
> Asynchronous approach:
>                    Currently used fio with libaio as ioengine but
> results are almost same.
> 
> -- 
> Thanks& Regards,
> M.Sai Chaithanya.
> 

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Dave Chinner]

On Mon, Dec 23, 2024 at 10:12:32PM +0530, Sai Chaitanya Mitta wrote:
> Hi Team,
>            Is there any method/tool available to explicitly mark XFS
> file extents as written? One approach I

Writing data to the unwritten extent is the only way to do this.
Allowing uninitialised data extents to be converted to a written
state opens a massive hole in system security.

Go search for the discussions around FALLOC_FL_NO_HIDE_STALE from
well over a decade ago.....

-Dave
-- 
Dave Chinner
david@fromorbit.com

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Sai Chaitanya Mitta]

Hi Darrick,
            Thanks for the quick response, we are exposing XFS file (created
through fallocate -l <size> <path>) as block device through
SPDK bdev (https://github.com/spdk/spdk) over NVMe-oF, Now initiator will
connect to the target and provide a block device to database applications.
What I have observed is databases' applications are issuing flush IO post
each/couple of writes, this flush at backend at backend translates to
fsync (through aio/io_uring) operation on FD (which is time taking process),
if we are doing no-op for flush IO then performance is 5x better compared to
serving flush operation. Doing no-op for flush and if system shutdown abruptly
then we are observing data loss (since metadata for new extents are not yet
persistent) to overcome this data loss issue and having better performance
below are the steps used:
1. Created file through fallocate using FALLOC_FL_ZERO_RANGE option
2. Explicitly zeroed file as mentioned in code (this marks all extents as
   written and there are no metadata changes related to data [what I observed],
   but there are atime and mtime updates of file).
3. Expose zeroed file to user as block device (as mentioned above).

Using above approach if system shutdown abruptly then I am not able
to reproduce data loss issue. So, planning to use above method to ensure
both data integrity and better performance

On Tue, Dec 24, 2024 at 3:23 AM Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Mon, Dec 23, 2024 at 10:12:32PM +0530, Sai Chaitanya Mitta wrote:
> > Hi Team,
> >            Is there any method/tool available to explicitly mark XFS
> > file extents as written? One approach I
> > am aware is explicitly zeroing the entire file (this file may be even
> > in hundreds of GB in size) through
> > synchronous/asynchronous(aio/io_uring) mechanism but it is time taking
> > process for large files,
> > is there any optimization/approach we can do to explicitly zeroing
> > file/mark extents as written?
>
> Why do you need to mark them written?
>
> --D
>
> >
> > Synchronous Approach:
> >                     while offset < size {
> >                         let bytes_written = img_file
> >                             .write_at(&buf, offset)
> >                             .map_err(|e| {
> >                                 error!("Failed to zero out file: {}
> > error: {:?}", vol_name, e);
> >                             })?;
> >                         if offset == size {
> >                             break;
> >                         }
> >                         offset = offset + bytes_written as u64;
> >                     }
> >                     img_file.sync_all();
> >
> > Asynchronous approach:
> >                    Currently used fio with libaio as ioengine but
> > results are almost same.
> >
> > --
> > Thanks& Regards,
> > M.Sai Chaithanya.
> >



-- 
Thanks& Regards,
M.Sai Chaithanya.

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Christoph Hellwig]

On Tue, Dec 24, 2024 at 02:42:08PM +1100, Dave Chinner wrote:
> On Mon, Dec 23, 2024 at 10:12:32PM +0530, Sai Chaitanya Mitta wrote:
> > Hi Team,
> >            Is there any method/tool available to explicitly mark XFS
> > file extents as written? One approach I
> 
> Writing data to the unwritten extent is the only way to do this.
> Allowing uninitialised data extents to be converted to a written
> state opens a massive hole in system security.

Yes.

> Go search for the discussions around FALLOC_FL_NO_HIDE_STALE from
> well over a decade ago.....

Or look for the old XFS_IOC_ALLOCSP ioctl which did allocation and
freeing in one syscall, which we removed quite a while ago.

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Darrick J. Wong]

On Tue, Dec 24, 2024 at 11:17:08AM +0530, Sai Chaitanya Mitta wrote:
> Hi Darrick,
>             Thanks for the quick response, we are exposing XFS file (created
> through fallocate -l <size> <path>) as block device through
> SPDK bdev (https://github.com/spdk/spdk) over NVMe-oF, Now initiator will
> connect to the target and provide a block device to database applications.
> What I have observed is databases' applications are issuing flush IO post
> each/couple of writes, this flush at backend at backend translates to
> fsync (through aio/io_uring) operation on FD (which is time taking process),
> if we are doing no-op for flush IO then performance is 5x better compared to
> serving flush operation. Doing no-op for flush and if system shutdown abruptly
> then we are observing data loss (since metadata for new extents are not yet
> persistent) to overcome this data loss issue and having better performance
> below are the steps used:
> 1. Created file through fallocate using FALLOC_FL_ZERO_RANGE option
> 2. Explicitly zeroed file as mentioned in code (this marks all extents as
>    written and there are no metadata changes related to data [what I observed],
>    but there are atime and mtime updates of file).
> 3. Expose zeroed file to user as block device (as mentioned above).
> 
> Using above approach if system shutdown abruptly then I am not able
> to reproduce data loss issue. So, planning to use above method to ensure
> both data integrity and better performance

That sounds brittle -- even if someday a FALLOC_FL_WRITE_ZEROES gets
merged into the kernel, if anything perturbs the file mapping (e.g.
background backup process reflinks the file) then you immediately become
vulnerable to these crash integrity problems without notice.

(Unless you're actually getting leases on the file ranges and reacting
appropriately when the leases break...)

--D

> On Tue, Dec 24, 2024 at 3:23 AM Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Mon, Dec 23, 2024 at 10:12:32PM +0530, Sai Chaitanya Mitta wrote:
> > > Hi Team,
> > >            Is there any method/tool available to explicitly mark XFS
> > > file extents as written? One approach I
> > > am aware is explicitly zeroing the entire file (this file may be even
> > > in hundreds of GB in size) through
> > > synchronous/asynchronous(aio/io_uring) mechanism but it is time taking
> > > process for large files,
> > > is there any optimization/approach we can do to explicitly zeroing
> > > file/mark extents as written?
> >
> > Why do you need to mark them written?
> >
> > --D
> >
> > >
> > > Synchronous Approach:
> > >                     while offset < size {
> > >                         let bytes_written = img_file
> > >                             .write_at(&buf, offset)
> > >                             .map_err(|e| {
> > >                                 error!("Failed to zero out file: {}
> > > error: {:?}", vol_name, e);
> > >                             })?;
> > >                         if offset == size {
> > >                             break;
> > >                         }
> > >                         offset = offset + bytes_written as u64;
> > >                     }
> > >                     img_file.sync_all();
> > >
> > > Asynchronous approach:
> > >                    Currently used fio with libaio as ioengine but
> > > results are almost same.
> > >
> > > --
> > > Thanks& Regards,
> > > M.Sai Chaithanya.
> > >
> 
> 
> 
> -- 
> Thanks& Regards,
> M.Sai Chaithanya.
> 

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Christoph Hellwig]

On Mon, Jan 06, 2025 at 11:46:39AM -0800, Darrick J. Wong wrote:
> That sounds brittle -- even if someday a FALLOC_FL_WRITE_ZEROES gets
> merged into the kernel, if anything perturbs the file mapping (e.g.
> background backup process reflinks the file) then you immediately become
> vulnerable to these crash integrity problems without notice.
> 
> (Unless you're actually getting leases on the file ranges and reacting
> appropriately when the leases break...)

They way I understood the description they have a user space program
exposing the XFS file over the network.  So if a change to the mapping
happens (e.g. due to defragmentation) they would in the worst case pay
the cost of an allocation transaction.

That is if they are really going through the normal kernel file
abstraction and don't try to bypass it by say abusing FIEMAP
information, in which case all hope is lost and the scheme has no chance
of reliably working, unless we add ioctls to expose the pNFS layouts
to userspace and they use that instead of FIEMAP.

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Darrick J. Wong]

On Mon, Jan 06, 2025 at 10:14:35PM -0800, Christoph Hellwig wrote:
> On Mon, Jan 06, 2025 at 11:46:39AM -0800, Darrick J. Wong wrote:
> > That sounds brittle -- even if someday a FALLOC_FL_WRITE_ZEROES gets
> > merged into the kernel, if anything perturbs the file mapping (e.g.
> > background backup process reflinks the file) then you immediately become
> > vulnerable to these crash integrity problems without notice.
> > 
> > (Unless you're actually getting leases on the file ranges and reacting
> > appropriately when the leases break...)
> 
> They way I understood the description they have a user space program
> exposing the XFS file over the network.  So if a change to the mapping
> happens (e.g. due to defragmentation) they would in the worst case pay
> the cost of an allocation transaction.
> 
> That is if they are really going through the normal kernel file
> abstraction and don't try to bypass it by say abusing FIEMAP
> information, in which case all hope is lost and the scheme has no chance
> of reliably working, unless we add ioctls to expose the pNFS layouts
> to userspace and they use that instead of FIEMAP.

I get this funny feeling that a lot of programs might like to lease
space and get told by the kernel when someone wants/took it back.
Swapfiles and lilo ftw.

--D

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Christoph Hellwig]

On Mon, Jan 06, 2025 at 11:04:59PM -0800, Darrick J. Wong wrote:
> I get this funny feeling that a lot of programs might like to lease
> space and get told by the kernel when someone wants/took it back.
> Swapfiles and lilo ftw.

Well, for swapfiles we can't really take them back.  Similarly the lilo
model is just broken as any chance of the mapping would actually require
re-installing the boot load in the boot block pointing to the blocks.

Re: Approach to quickly zeroing large XFS file (or) tool to mark XFS file extents as written

[Darrick J. Wong]

On Tue, Jan 07, 2025 at 12:37:20AM -0800, Christoph Hellwig wrote:
> On Mon, Jan 06, 2025 at 11:04:59PM -0800, Darrick J. Wong wrote:
> > I get this funny feeling that a lot of programs might like to lease
> > space and get told by the kernel when someone wants/took it back.
> > Swapfiles and lilo ftw.
> 
> Well, for swapfiles we can't really take them back.  Similarly the lilo
> model is just broken as any chance of the mapping would actually require
> re-installing the boot load in the boot block pointing to the blocks.

I bet the rdma users might like a lease that can't be taken back.  We
/did/ talk a hojillion years ago at LSFMM about having a type of lease
that expires when someone tries to change the file; and a different type
of lease that can't expire but causes file operations to error out.

OTOH it's not like I've ever tried to solve this problem. :P

--D