Re: Using O_DIRECT in ext4

[Xiang Wang <xiangw@google.com>]

On Tue, Jul 21, 2009 at 9:38 AM, Eric Sandeen<sandeen@redhat.com> wrote:
> Curt Wohlgemuth wrote:
>> On Mon, Jul 20, 2009 at 8:41 PM, Eric Sandeen<sandeen@redhat.com> wrote:
>>> Xiang Wang wrote:
>
>>>> For comparison, I did the same experiment on an ext2 partition,
>>>> resulting in each file having only 1 extent.
>>> Interestinng, not sure I would have expected that.
>>
>> Same with us; we're looking into more variables to understand it.
>
> To be more clear, I would not have expected ext2 to deal well with it
> either, is more what I meant ;)  I'm not terribly surprised that ext4
> gets fragmented.
>
> For the numbers posted, how big were the files (how many 1m chunks were
> written?)
>
> Just FWIW; I did something like:
>
> # for I in `seq 1 16`; do dd if=/dev/zero of=testfile$I bs=1M count=16
> oflag=direct & done
>
> on a rhel5.4 beta kernel and got:
>
> ~5 extents per file on ext4 (per filefrag output)
> between 41 and 234 extents on ext2.
> ~6 extents per file on ext3.
> ~16 extents per file on xfs
>

I repeated this test(bs=1M count=16) by tuning some parameters in my
test program. And I got the following results(per filefrag output):
ext4:
5 extents per file

ext2:
file0: 5 extents found, perfection would be 1 extent
file1: 5 extents found, perfection would be 1 extent
file2: 6 extents found, perfection would be 1 extent
file3: 4 extents found, perfection would be 1 extent
file4: 4 extents found, perfection would be 1 extent
file5: 6 extents found, perfection would be 1 extent
file6: 4 extents found, perfection would be 1 extent
file7: 5 extents found, perfection would be 1 extent
file8: 6 extents found, perfection would be 1 extent
file9: 4 extents found, perfection would be 1 extent
file10: 5 extents found, perfection would be 1 extent
file11: 6 extents found, perfection would be 1 extent
file12: 6 extents found, perfection would be 1 extent
file13: 8 extents found, perfection would be 1 extent
file14: 4 extents found, perfection would be 1 extent
file15: 7 extents found, perfection would be 1 extent

The results on ext4 look comparable to yours while the results on ext2
look very different.
I am attaching the test program I use in case you want to try it. It
is at the end of the message.
I invoked it like: ./mt_writes 16 1 to have 16 threads writing using O_DIRECT.

> if I created a subdir for each file:
>
> # for I in `seq 1 16`; do mkdir dir$I; dd if=/dev/zero
> of=dir$I/testfile$I bs=1M count=16 oflag=direct & done
>
> ~5 extents per file on ext4
> 1 or 2 extents per file on ext2
> 1 or 2 extents per file on ext3
> ~16 extents per file on xfs.
>
> -Eric
>

======
/*
 * mt_write.c -- multiple threads extending files concurrently.
 */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/types.h>
#include <pthread.h>
#include <sys/stat.h>
#include <fcntl.h>

#define _XOPEN_SOURCE 600
#define O_DIRECT        00040000        /* direct disk access hint */

#define MAX_THREAD 1000
#define BUFSIZE 1048576
#define COUNT 16

typedef struct {
        int id;
        int odirect;
} parm;

void *expand(void *arg)
{
        char *buf;
        char fname[16];
        int fd;
        int i, count;
        parm *p = (parm *)arg;

        // O_DIRECT needs to work with aligned memory
        if (posix_memalign((void *) &buf, 512, BUFSIZE) != 0) {
                fprintf(stderr, "cannot allocate aligned mem!\n");
                return NULL;
        }

        sprintf(fname, "file%d", p->id);
        if (p->odirect)
                fd = open(fname, O_RDWR|O_CREAT|O_APPEND|O_DIRECT);
        else
                fd = open(fname, O_RDWR|O_CREAT|O_APPEND);

        if (fd == -1) {
                fprintf(stderr, "Open %s failed!\n", fname);
                return NULL;
        }

        for(i = 0; i < COUNT; i++) {
                count = write(fd, buf, BUFSIZE);
                if (count == -1) {
                        fprintf(stderr, "Only able to finish %d blocks
of data\n", i);
                        return NULL;
                }
        }

        if (!p->odirect) {
                fsync(fd);
        }
        printf("Done with writing %d blocks of data\n", COUNT);
        close(fd);
        free(buf);
        return NULL;
}

int main(int argc, char* argv[]) {
        int n,i, odirect;
        pthread_t *threads;
        pthread_attr_t pthread_custom_attr;
        parm *p;

        if (argc != 3)
        {
                printf ("Usage: %s <# of threads> <O_DIRECT? 1:0>\n",argv[0]);
                exit(1);
        }

        n=atoi(argv[1]);
        odirect = atoi(argv[2]);

        if ((n < 1) || (n > MAX_THREAD))
        {
                printf ("The # of thread should between 1 and
%d.\n",MAX_THREAD);
                exit(1);
        }

        threads=(pthread_t *)malloc(n*sizeof(*threads));
        pthread_attr_init(&pthread_custom_attr);

        p=(parm *)malloc(sizeof(parm)*n);
        /* Start up thread */
        for (i = 0; i < n; i++)
        {
                p[i].id = i;
                p[i].odirect = odirect;
                pthread_create(&threads[i], &pthread_custom_attr,
expand, (void *)(p+i));
        }

        /* Synchronize the completion of each thread. */

        for (i=0; i<n; i++)
        {
                pthread_join(threads[i],NULL);
        }
        free(p);
        return 0;
}
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html