fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages())

fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages())

[Vivek Goyal]

On Tue, Jun 28, 2011 at 09:53:36PM -0400, Vivek Goyal wrote:

[..]
> > FYI, filesystem development cycles are slow and engineers are
> > conservative because of the absolute requirement for data integrity.
> > Hence we tend to focus development on problems that users are
> > reporting (i.e. known pain points) or functionality they have
> > requested.
> > 
> > In this case, block throttling works OK on most filesystems out of
> > the box, but it has some known problems. If there are people out
> > there hitting these known problems then they'll report them, we'll
> > hear about them and they'll eventually get fixed.
> > 
> > However, if no-one is reporting problems related to block throttling
> > then it either works well enough for the existing user base or
> > nobody is using the functionality. Either way we don't need to spend
> > time on optimising the filesystem for such functionality.
> > 
> > So while you may be skeptical about whether filesystems will be
> > changed, it really comes down to behaviour in real-world
> > deployments. If what we already have is good enough, then we don't
> > need to spend resources on fixing problems no-one is seeing...
>

[CC linux-ext4 list]

Dave,

Just another example where serialization is taking place with ext4.

I created a group with 1MB/s write limit and ran tedso's fsync tester
program with little modification. I used write() system call instead
of pwrite() so that file size grows. This program basically writes
1MB of data and then fsync's it and then measures the fsync time.

I ran two instances of prgram in two groups on two separate files. One
instances is throttled to 1MB/s and other is in root group unthrottled.

Unthrottled program gets serialized behind throttled one. Following
are fsync times.

Throttled instance	Unthrottled Instance
------------------ 	--------------------
fsync time: 1.0051	fsync time: 1.0067
fsync time: 1.0049	fsync time: 1.0075
fsync time: 1.0048	fsync time: 1.0063
fsync time: 1.0073	fsync time: 1.0062
fsync time: 1.0070	fsync time: 1.0078
fsync time: 1.0032	fsync time: 1.0049
fsync time: 0.0154	fsync time: 1.0068
fsync time: 0.0137	fsync time: 1.0048

Without any throttling both the instances do fine
-------------------------------------------------
Throttled instance	Unthrottled Instance
------------------ 	--------------------
fsync time: 0.0139	fsync time: 0.0162
fsync time: 0.0132	fsync time: 0.0156
fsync time: 0.0149	fsync time: 0.0169
fsync time: 0.0165	fsync time: 0.0152
fsync time: 0.0188	fsync time: 0.0135
fsync time: 0.0137	fsync time: 0.0142
fsync time: 0.0148	fsync time: 0.0149
fsync time: 0.0168	fsync time: 0.0163
fsync time: 0.0153	fsync time: 0.0143

So when we are inreasing the size of file and fsyncing it, other
unthrottled instances of similar activities will get throttled
behind it.

IMHO, this is a problem and should be fixed. If filesystem can fix it great.
But if not, then we should consider the option of throttling buffered writes 
in balance_dirty_pages().

Following is the test program.

/*
 *  * fsync-tester.c
 *
 * Written by Theodore Ts'o, 3/21/09.
 *
 * This file may be redistributed under the terms of the GNU Public
 * License, version 2.
 */

#include <unistd.h>
#include <stdlib.h>
#include <stdio.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <time.h>
#include <fcntl.h>
#include <string.h>

#define SIZE (1024*1024)

static float timeval_subtract(struct timeval *tv1, struct timeval *tv2)
{
        return ((tv1->tv_sec - tv2->tv_sec) +
                ((float) (tv1->tv_usec - tv2->tv_usec)) / 1000000);
}

int main(int argc, char **argv)
{
        int     fd;
        struct timeval tv, tv2;
        char buf[SIZE];

        fd = open("fsync-tester.tst-file", O_WRONLY|O_CREAT);
        if (fd < 0) {
                perror("open");
                exit(1);
        }
        memset(buf, 'a', SIZE);
        while (1) {
                write(fd, buf, SIZE);
                gettimeofday(&tv, NULL);
                fsync(fd);
                gettimeofday(&tv2, NULL);
                printf("fsync time: %5.4f\n", timeval_subtract(&tv2,
&tv));
                sleep(1);
        }
}

Thanks
Vivek

Re: fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages())

[Vivek Goyal]

On Thu, Jun 30, 2011 at 04:04:59PM -0400, Vivek Goyal wrote:

[..]
> Dave,
> 
> Just another example where serialization is taking place with ext4.
> 
> I created a group with 1MB/s write limit and ran tedso's fsync tester
> program with little modification. I used write() system call instead
> of pwrite() so that file size grows. This program basically writes
> 1MB of data and then fsync's it and then measures the fsync time.
> 
> I ran two instances of prgram in two groups on two separate files. One
> instances is throttled to 1MB/s and other is in root group unthrottled.
> 
> Unthrottled program gets serialized behind throttled one. Following
> are fsync times.
> 
> Throttled instance	Unthrottled Instance
> ------------------ 	--------------------
> fsync time: 1.0051	fsync time: 1.0067
> fsync time: 1.0049	fsync time: 1.0075
> fsync time: 1.0048	fsync time: 1.0063
> fsync time: 1.0073	fsync time: 1.0062
> fsync time: 1.0070	fsync time: 1.0078
> fsync time: 1.0032	fsync time: 1.0049
> fsync time: 0.0154	fsync time: 1.0068
> fsync time: 0.0137	fsync time: 1.0048
> 
> Without any throttling both the instances do fine
> -------------------------------------------------
> Throttled instance	Unthrottled Instance
> ------------------ 	--------------------
> fsync time: 0.0139	fsync time: 0.0162
> fsync time: 0.0132	fsync time: 0.0156
> fsync time: 0.0149	fsync time: 0.0169
> fsync time: 0.0165	fsync time: 0.0152
> fsync time: 0.0188	fsync time: 0.0135
> fsync time: 0.0137	fsync time: 0.0142
> fsync time: 0.0148	fsync time: 0.0149
> fsync time: 0.0168	fsync time: 0.0163
> fsync time: 0.0153	fsync time: 0.0143
> 
> So when we are inreasing the size of file and fsyncing it, other
> unthrottled instances of similar activities will get throttled
> behind it.
> 
> IMHO, this is a problem and should be fixed. If filesystem can fix it great.
> But if not, then we should consider the option of throttling buffered writes 
> in balance_dirty_pages().

XFS seems to be doing well for this particular test. Unthrottled
fsyncer does not get serialized behind throttled one.

Throttled instance	Unthrottled Instance
------------------ 	--------------------
fsync time: 1.0511	fsync time: 0.0204
fsync time: 1.0486	fsync time: 0.0260
fsync time: 1.0445	fsync time: 0.0260
fsync time: 1.0485	fsync time: 0.0260
fsync time: 1.0446	fsync time: 0.0260
fsync time: 1.2157	fsync time: 0.0260
fsync time: 1.0446	fsync time: 0.0300
fsync time: 1.0484	fsync time: 0.0340
fsync time: 1.0446	fsync time: 0.0221
fsync time: 1.0486	fsync time: 0.0340
fsync time: 1.0406	fsync time: 0.0340

Thanks
Vivek

Re: fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages())

[Dave Chinner]

On Thu, Jun 30, 2011 at 04:44:32PM -0400, Vivek Goyal wrote:
> On Thu, Jun 30, 2011 at 04:04:59PM -0400, Vivek Goyal wrote:
> 
> [..]
> > Dave,
> > 
> > Just another example where serialization is taking place with ext4.
> > 
> > I created a group with 1MB/s write limit and ran tedso's fsync tester
> > program with little modification. I used write() system call instead
> > of pwrite() so that file size grows. This program basically writes
> > 1MB of data and then fsync's it and then measures the fsync time.
> > 
> > I ran two instances of prgram in two groups on two separate files. One
> > instances is throttled to 1MB/s and other is in root group unthrottled.
> > 
> > Unthrottled program gets serialized behind throttled one. Following
> > are fsync times.
> > 
> > Throttled instance	Unthrottled Instance
> > ------------------ 	--------------------
> > fsync time: 1.0051	fsync time: 1.0067
> > fsync time: 1.0049	fsync time: 1.0075
> > fsync time: 1.0048	fsync time: 1.0063
> > fsync time: 1.0073	fsync time: 1.0062
> > fsync time: 1.0070	fsync time: 1.0078
> > fsync time: 1.0032	fsync time: 1.0049
> > fsync time: 0.0154	fsync time: 1.0068
> > fsync time: 0.0137	fsync time: 1.0048
> > 
> > Without any throttling both the instances do fine
> > -------------------------------------------------
> > Throttled instance	Unthrottled Instance
> > ------------------ 	--------------------
> > fsync time: 0.0139	fsync time: 0.0162
> > fsync time: 0.0132	fsync time: 0.0156
> > fsync time: 0.0149	fsync time: 0.0169
> > fsync time: 0.0165	fsync time: 0.0152
> > fsync time: 0.0188	fsync time: 0.0135
> > fsync time: 0.0137	fsync time: 0.0142
> > fsync time: 0.0148	fsync time: 0.0149
> > fsync time: 0.0168	fsync time: 0.0163
> > fsync time: 0.0153	fsync time: 0.0143
> > 
> > So when we are inreasing the size of file and fsyncing it, other
> > unthrottled instances of similar activities will get throttled
> > behind it.
> > 
> > IMHO, this is a problem and should be fixed. If filesystem can fix it great.
> > But if not, then we should consider the option of throttling buffered writes 
> > in balance_dirty_pages().
> 
> XFS seems to be doing well for this particular test. Unthrottled
> fsyncer does not get serialized behind throttled one.
> 
> Throttled instance	Unthrottled Instance
> ------------------ 	--------------------
> fsync time: 1.0511	fsync time: 0.0204
> fsync time: 1.0486	fsync time: 0.0260
> fsync time: 1.0445	fsync time: 0.0260
> fsync time: 1.0485	fsync time: 0.0260
> fsync time: 1.0446	fsync time: 0.0260
> fsync time: 1.2157	fsync time: 0.0260
> fsync time: 1.0446	fsync time: 0.0300
> fsync time: 1.0484	fsync time: 0.0340
> fsync time: 1.0446	fsync time: 0.0221
> fsync time: 1.0486	fsync time: 0.0340
> fsync time: 1.0406	fsync time: 0.0340

And you've just illustrated my point better than I did - that
different filesytsems will suffer from different problems, but some
filesytsems will work better than others out of the box. :)

Of course, there's no guarantee XFS will remain this way - if you
want us to care about regressions of this sort at all, you need to
encapsulate all this behaviour in a set of automated tests.
Preferrably within the xfstests infrastructure because that now has
fairly wide usage within the fs dev community....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com