Significantly longer fallocate times with Flag Unwritten Extents disabled

Significantly longer fallocate times with Flag Unwritten Extents disabled

[Bill McDuck]

Hello.

I have been testing the performance of XFS with "Flag Unwritten
Extents" enabled and disabled.

For security reasons I know XFS flags all unwritten extents by
default, so that uninitialized disk space cannot be read by the user.
In my application I actually want to have access to this uninitialized
disk space. I have been modifying the superblocks using xfs_repair to
enable this functionality.

Interestingly, I have found that creating a 10 GB file using fallocate
is significantly slower with unwritten flag extents disabled.  For
instance, creating a 10GB file takes 32 seconds on my (slow) machine,
whereas with the flag enabled, the process is almost instantaneous.

Does anyone know why creating an uninitialized file is so much slower
when the unwritten extents flag functionality is disabled?  I know
that the disk is not being initialized to zero, as I can read back non
zero content.

For my application I'm interested in creating large uninitialised
files quickly, so I'm very interested to know what additional
operations are being performed with the flagging disabled.

Thanks for the help.

Bill.

Re: Significantly longer fallocate times with Flag Unwritten Extents disabled

[Brian Foster]

On Mon, Mar 06, 2017 at 12:02:01PM +0000, Bill McDuck wrote:
> Hello.
> 
> I have been testing the performance of XFS with "Flag Unwritten
> Extents" enabled and disabled.
> 
> For security reasons I know XFS flags all unwritten extents by
> default, so that uninitialized disk space cannot be read by the user.
> In my application I actually want to have access to this uninitialized
> disk space. I have been modifying the superblocks using xfs_repair to
> enable this functionality.
> 
> Interestingly, I have found that creating a 10 GB file using fallocate
> is significantly slower with unwritten flag extents disabled.  For
> instance, creating a 10GB file takes 32 seconds on my (slow) machine,
> whereas with the flag enabled, the process is almost instantaneous.
> 
> Does anyone know why creating an uninitialized file is so much slower
> when the unwritten extents flag functionality is disabled?  I know
> that the disk is not being initialized to zero, as I can read back non
> zero content.
> 

On a quick fallocate test, it is actually zeroing out the blocks in the
post-fallocate setattr (to set the inode size). If the fallocate changes
the size of the file, we call xfs_vn_setattr_size(). That eventually
calls an xfs_zero_range() on the blocks between the old eof and new eof.
I suspect this occurs regardless of whether you've disabled the
unwritten flag bit, but the iomap_zero_range() codepath can shortcut
through the extents when they are marked as unwritten. You can get
around this by truncating the file before the fallocate.

BTW, if you are hitting this behavior and can read non-zero data
afterwards that you are sure you have not written or is not already
inside eof at the time of the falloc, you may be hitting a bug. Please
verify and follow up with details of your test case if you can indeed
reproduce that behavior. I don't reproduce it in my test env (without a
truncate first)..

Brian

> For my application I'm interested in creating large uninitialised
> files quickly, so I'm very interested to know what additional
> operations are being performed with the flagging disabled.
> 
> Thanks for the help.
> 
> Bill.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Significantly longer fallocate times with Flag Unwritten Extents disabled

[Christoph Hellwig]

On Mon, Mar 06, 2017 at 12:02:01PM +0000, Bill McDuck wrote:
> Hello.
> 
> I have been testing the performance of XFS with "Flag Unwritten
> Extents" enabled and disabled.

Why do you disable it?  It's been part of the defauly on feature
set for a decade, and manually disabling it will make you run into
completely untested code.  There is a reason it's mandatory for v5
file systems.

> 
> For security reasons I know XFS flags all unwritten extents by
> default, so that uninitialized disk space cannot be read by the user.
> In my application I actually want to have access to this uninitialized
> disk space. I have been modifying the superblocks using xfs_repair to
> enable this functionality.

Don't do that, it's not supported for a reason.

Re: Significantly longer fallocate times with Flag Unwritten Extents disabled

[Bill]

Woops, I forgot to send this response to the mailing list:

Thanks for the information and the explanation. That all makes sense,
as I cannot reproduce the non-zero data behaviour and suspect I called
fallocate() on an existing file.  When I read back freshly fallocated
files I only ever get zeros back.  Apologies for the confusion.

The reason I am trying to create large uninitialized files quickly is
that I have a hardware (FPGA) based system which is capable of writing
to an SSD at a very fast rate. With software in the loop (managing the
OS, file system and IO etc) I am limited to quite low rates (about 1/5
of the hardware performance) as my processor is not very capable.  As
I typically only write large sequential files, I was pondering whether
I could create a large uninitialized file, figure out where it was
physically located on the disk, unmount the drive and then let my
hardware fill the relevant blocks with data, before handing control
back to the software and remounting the drive. This would give me the
advantages of a filesystem, without having to implement one in
hardware, which would be challenging!

This was probably impossible from the get go, as there is no doubt
some glaring hole in my understanding of how file systems work!

Thanks for the help.

On Wed, Mar 8, 2017 at 3:56 PM, Christoph Hellwig <hch@infradead.org> wrote:
> On Mon, Mar 06, 2017 at 12:02:01PM +0000, Bill McDuck wrote:
>> Hello.
>>
>> I have been testing the performance of XFS with "Flag Unwritten
>> Extents" enabled and disabled.
>
> Why do you disable it?  It's been part of the defauly on feature
> set for a decade, and manually disabling it will make you run into
> completely untested code.  There is a reason it's mandatory for v5
> file systems.
>
>>
>> For security reasons I know XFS flags all unwritten extents by
>> default, so that uninitialized disk space cannot be read by the user.
>> In my application I actually want to have access to this uninitialized
>> disk space. I have been modifying the superblocks using xfs_repair to
>> enable this functionality.
>
> Don't do that, it's not supported for a reason.

Re: Significantly longer fallocate times with Flag Unwritten Extents disabled

[Dave Chinner]

On Wed, Mar 08, 2017 at 04:48:09PM +0000, Bill wrote:
> Woops, I forgot to send this response to the mailing list:
> 
> Thanks for the information and the explanation. That all makes sense,
> as I cannot reproduce the non-zero data behaviour and suspect I called
> fallocate() on an existing file.  When I read back freshly fallocated
> files I only ever get zeros back.  Apologies for the confusion.
> 
> The reason I am trying to create large uninitialized files quickly is
> that I have a hardware (FPGA) based system which is capable of writing
> to an SSD at a very fast rate. With software in the loop (managing the
> OS, file system and IO etc) I am limited to quite low rates (about 1/5
> of the hardware performance) as my processor is not very capable.  As
> I typically only write large sequential files, I was pondering whether
> I could create a large uninitialized file, figure out where it was
> physically located on the disk, unmount the drive and then let my
> hardware fill the relevant blocks with data, before handing control
> back to the software and remounting the drive. This would give me the
> advantages of a filesystem, without having to implement one in
> hardware, which would be challenging!

And, realistically, quite naive.

Go look at how the PNFS file layout code works - file leases for
ensuring outstanding client side IO operations are completed
correctly before the filesystem does something that will conflict
with the client side IO (e.g. truncate), struct
export_operations.map_blocks to allocate blocks and layouts that are
handed to the client, client does the IO directly to/from the
device, and then calls export_operations.commit_blocks for the
client to signal the write IO is done.

i.e. you FPGA is the client, write a kernel module that takes a fd
and uses the low level exportfs interfaces into the filesystem to do
block allocation, pass the mapped blocks to the FPGA and then signal
IO completion to the filesystem when each FPGA write completes.

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com