EACCES race when opening just-created file

EACCES race when opening just-created file

[Noah Misch]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

With the Linux kernel as both NFS client and server, I'm seeing a race
condition where open() can return EACCES for a just-created file.  Is this
expected behavior?  POSIX implicitly allows it, but I failed to locate any
discussion of it.  I have attached a test program; here, it witnesses the race
for roughly a dozen of the 4096 files it creates (different files each time).
One router separates client from server, with overall ping times as follows:

100 packets transmitted, 100 received, 0% packet loss, time 99039ms
rtt min/avg/max/mdev = 0.173/0.259/4.447/0.422 ms

This server has three similar clients, each of which witnesses the race.  I
did not reproduce this in a different NFS setup where client and server were
virtual machines on the same box.  Those are the only two environments I have
tested so far.

-- Server Details
[nm@gcc45 0:0 00:36:42 ~]$ uname -a
Linux gcc45 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux
[nm@gcc45 0:0 00:36:37 ~]$ cat /proc/fs/nfs/exports
# Version 1.1
# Path Client(Flags) # IPs
/home   gcc24.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc23.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
/home   gcc22.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)

-- Client Details
[nm@erpro8-fsf3 0:1 00:37:28 nfs]$ uname -a
Linux erpro8-fsf3 4.1.4 #1 SMP PREEMPT Mon Aug 3 14:22:54 PDT 2015 mips64 GNU/Linux
[nm@erpro8-fsf3 0:1 00:37:49 nfs]$ mount|grep nfs
gcc45.tetaneutral.net:/home on /home type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2a03:7220:8083:9d00::1,local_lock=none,addr=2a03:7220:8081:8100::1)

Thanks,
nm

[-- Attachment #2: nfs-creat-eacces.c --]
[-- Type: text/plain, Size: 1905 bytes --]

/*
 * Demonstrates open() returning EACCES for a recently-created file on NFS.  One
 * thread creates files nfs0 .. nfs4095 in the CWD, and another thread
 * continually attempts to open(O_RDWR) the latest file.  Prints a message
 * whenever the second thread sees open() fail in a way other than ENOENT.
 * Affected NFS configurations will print such messages for a selection of the
 * files, and immune configurations will print nothing.
 *
 * I abstracted this test case from a malfunction in the PostgreSQL database,
 * where one process opened a file just created in another process.  (See
 * src/interfaces/ecpg cases thread/thread and thread/prep.)
 */

#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <pthread.h>
#include <stdio.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

/* no reader/writer synchronization; occasional disagreement is okay */
static int pos;

static void *
creator_task(void *arg)
{
    for (pos = 0; pos < (1 << 12); ++pos)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR | O_CREAT | O_EXCL, 0600);
	if (fd < 0)
	    printf("could not create %s: %s\n", buf, strerror(errno));
	else if (close(fd) != 0)
	    printf("could not close %s after creation: %s\n",
		   buf, strerror(errno));
    }

    pos = -1;

    return NULL;
}

static void *
tester_task(void *arg)
{
    while (pos >= 0)
    {
	char buf[128];
	int fd;

	sprintf(buf, "nfs%d", pos);
	fd = open(buf, O_RDWR);
	if (fd < 0)
	{
	    if (errno != ENOENT)
		printf("could not open %s: %s\n", buf, strerror(errno));
	}
	else if (close(fd) != 0)
	    printf("could not close %s: %s\n", buf, strerror(errno));
    }

    return NULL;
}

int
main(int argc, char **argv)
{
    pthread_t creator;

    pthread_create(&creator, NULL, creator_task, 0);
    tester_task(NULL);
    pthread_join(creator, NULL);

    return 0;
}

Re: EACCES race when opening just-created file

[Jeff Layton]

On Sat, 2016-07-23 at 16:12 -0400, Noah Misch wrote:
> With the Linux kernel as both NFS client and server, I'm seeing a race
> condition where open() can return EACCES for a just-created file.  Is this
> expected behavior?  POSIX implicitly allows it, but I failed to locate any
> discussion of it.  I have attached a test program; here, it witnesses the race
> for roughly a dozen of the 4096 files it creates (different files each time).
> One router separates client from server, with overall ping times as follows:
> 
> 100 packets transmitted, 100 received, 0% packet loss, time 99039ms
> rtt min/avg/max/mdev = 0.173/0.259/4.447/0.422 ms
> 
> This server has three similar clients, each of which witnesses the race.  I
> did not reproduce this in a different NFS setup where client and server were
> virtual machines on the same box.  Those are the only two environments I have
> tested so far.
> 
> -- Server Details
> > [nm@gcc45 0:0 00:36:42 ~]$ uname -a
> Linux gcc45 3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt11-1+deb8u3 (2015-08-04) i686 GNU/Linux
> > [nm@gcc45 0:0 00:36:37 ~]$ cat /proc/fs/nfs/exports
> # Version 1.1
> # Path Client(Flags) # IPs
> /home   gcc24.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
> /home   gcc23.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
> /home   gcc22.tetaneutral.net(rw,no_root_squash,async,wdelay,no_subtree_check,uuid=6797c43c:b0f64b64:bfca4840:f15789eb,sec=1)
> 
> -- Client Details
> > [nm@erpro8-fsf3 0:1 00:37:28 nfs]$ uname -a
> Linux erpro8-fsf3 4.1.4 #1 SMP PREEMPT Mon Aug 3 14:22:54 PDT 2015 mips64 GNU/Linux
> > [nm@erpro8-fsf3 0:1 00:37:49 nfs]$ mount|grep nfs
> gcc45.tetaneutral.net:/home on /home type nfs4 (rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp6,port=0,timeo=600,retrans=2,sec=sys,clientaddr=2a03:7220:8083:9d00::1,local_lock=none,addr=2a03:7220:8081:8100::1)
> 
> Thanks,
> nm

This is due to a limitation of NFSv3 and NFSv4.0. When you do an
exclusive create, the atime and mtime get set to a particular set of
values (the verifier), and the client is expected to override those
values with a follow-on SETATTR call once it gets those values back.
This is to prevent problems during certain server reboot scenarios. 

The NFS server also sets the mode to 0000 during this time (since the
client can't even set the mode during the operation).

NFSv4.1 is less subject to this problem. You may want to try using that
and see if it's any better.
 
-- 
Jeff Layton <jlayton@redhat.com>

Re: EACCES race when opening just-created file

[Noah Misch]

On Sat, Jul 23, 2016 at 04:50:57PM -0400, Jeff Layton wrote:
> On Sat, 2016-07-23 at 16:12 -0400, Noah Misch wrote:
> > With the Linux kernel as both NFS client and server, I'm seeing a race
> > condition where open() can return EACCES for a just-created file.  Is this
> > expected behavior?  POSIX implicitly allows it, but I failed to locate any
> > discussion of it.  I have attached a test program; here, it witnesses the race
> > for roughly a dozen of the 4096 files it creates (different files each time).

> This is due to a limitation of NFSv3 and NFSv4.0. When you do an
> exclusive create, the atime and mtime get set to a particular set of
> values (the verifier), and the client is expected to override those
> values with a follow-on SETATTR call once it gets those values back.
> This is to prevent problems during certain server reboot scenarios. 
> 
> The NFS server also sets the mode to 0000 during this time (since the
> client can't even set the mode during the operation).

Interesting; that fully explains it.  Thanks.

> NFSv4.1 is less subject to this problem. You may want to try using that
> and see if it's any better.

I will look into that.

Re: EACCES race when opening just-created file

[Noah Misch]

On Mon, Jul 25, 2016 at 11:04:32PM -0400, Noah Misch wrote:
> On Sat, Jul 23, 2016 at 04:50:57PM -0400, Jeff Layton wrote:
> > On Sat, 2016-07-23 at 16:12 -0400, Noah Misch wrote:
> > > With the Linux kernel as both NFS client and server, I'm seeing a race
> > > condition where open() can return EACCES for a just-created file.  Is this
> > > expected behavior?  POSIX implicitly allows it, but I failed to locate any
> > > discussion of it.  I have attached a test program; here, it witnesses the race
> > > for roughly a dozen of the 4096 files it creates (different files each time).
> 
> > This is due to a limitation of NFSv3 and NFSv4.0. When you do an
> > exclusive create, the atime and mtime get set to a particular set of
> > values (the verifier), and the client is expected to override those
> > values with a follow-on SETATTR call once it gets those values back.
> > This is to prevent problems during certain server reboot scenarios. 
> > 
> > The NFS server also sets the mode to 0000 during this time (since the
> > client can't even set the mode during the operation).
> 
> Interesting; that fully explains it.  Thanks.
> 
> > NFSv4.1 is less subject to this problem. You may want to try using that
> > and see if it's any better.
> 
> I will look into that.

Mounting with vers=4.1 eliminated the unwanted behavior.  Thanks for the tip.