Re: Using O_DIRECT in ext4

[Curt Wohlgemuth <curtw@google.com>]

On Mon, Jul 20, 2009 at 8:41 PM, Eric Sandeen<sandeen@redhat.com> wrote:
> Xiang Wang wrote:
>> Hi,
>>
>> Recently I've been experimenting with O_DIRECT in ext4 to get a
>> feeling of how much file fragmentation will be generated.
>>
>> On a newly formatted ext4 partition(no-journal), I created a top-level
>> directory and under this top-level directory I ran a test program to
>> generate some files.
>>
>> The test program does the following:
>> -- create multiple threads(in my test case: 16 threads)
>> -- each thread creates a file with the O_DIRECT flag and keeps
>> extending the file to 1MB
>> Since these threads run concurrently, they compete in block allocation.
>>
>> After the program ran to a completion, I ran filefrag on each file and
>> measure how many extents there are in the file.
>> And here is a sample result:
>> file0: 6 extents found
>> file1: 20 extents found
>> file2: 7 extents found
>> file3: 6 extents found
>> file4: 6 extents found
>> file5: 5 extents found
>> file6: 6 extents found
>> file7: 20 extents found
>> file8: 20 extents found
>> file9: 20 extents found
>> file10: 20 extents found
>> file11: 20 extents found
>> file12: 20 extents found
>> file13: 19 extents found
>> file14: 19 extents found
>> file15: 19 extents found
>>
>> Looks like these files are quite heavily fragmented.
>
> Multiple parallel extending DIOs in a single dir is a tough case for a
> filesystem - it has no hints about what to do, and can't use delalloc to
> wait to see what's happening; it just has to allocate things as they
> come, more or less.
>
>> For comparison, I did the same experiment on an ext2 partition,
>> resulting in each file having only 1 extent.
>
> Interestinng, not sure I would have expected that.

Same with us; we're looking into more variables to understand it.

>> I also did the experiments of using buffered writes(by removing the
>> O_DIRECT flag) on ext2 and ext4, both resulting in each file having
>> only 1 extent.
>
> delayed allocation at work I suppose.
>
>> I am wondering whether this kind of file fragmentation is already a
>> known issue in ext4 when O_DIRECT is used? Is it something by design?
>> Since it seems like ext2 does not have this issue under my test case,
>> is it necessary that we make the behavior of ext4 similar to ext2
>> under situations like this?
>
> Is this representative of a real workload?

Not exactly perhaps, but we do have apps that are showing
significantly more fragmentation in their files on ext4 than with
ext2, while using O_DIRECT (e.g., 8 extents on ext4 vs 1 on ext2, as
reported by filefrag).  The experiment above is synthetic, but fairly
representative.

(Hence the related questions about fallocate, since this is one
possible, though ugly, workaround.)

Curt