readdir cookies

readdir cookies

[J. Bruce Fields]

How does the nfs client decide what directory cookies to return?  (As
the d_off field to readdir, for example).

I'd assumed it would return the server's cookie, but testing shows it
returns a simple integer sequence.

--b.

RE: readdir cookies

[Myklebust, Trond]

> -----Original Message-----
> From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> Sent: Monday, September 17, 2012 11:54 AM
> To: Myklebust, Trond; Schumaker, Bryan
> Cc: linux-nfs@vger.kernel.org
> Subject: readdir cookies
> 
> How does the nfs client decide what directory cookies to return?  (As the
> d_off field to readdir, for example).
> 
> I'd assumed it would return the server's cookie, but testing shows it returns a
> simple integer sequence.

We cache the cookies and use them to figure out where the readdir cursor is after a directory update, but we use positive integers for telldir()/seekdir(). The reason is that too many servers return cookies that use > 32 bits (NFSv2-incompatible) or are unsigned.

Cheers
  Trond

Re: readdir cookies

[J. Bruce Fields]

On Mon, Sep 17, 2012 at 04:05:10PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > Sent: Monday, September 17, 2012 11:54 AM
> > To: Myklebust, Trond; Schumaker, Bryan
> > Cc: linux-nfs@vger.kernel.org
> > Subject: readdir cookies
> > 
> > How does the nfs client decide what directory cookies to return?  (As the
> > d_off field to readdir, for example).
> > 
> > I'd assumed it would return the server's cookie, but testing shows it returns a
> > simple integer sequence.
> 
> We cache the cookies and use them to figure out where the readdir cursor is after a directory update, but we use positive integers for telldir()/seekdir(). The reason is that too many servers return cookies that use > 32 bits (NFSv2-incompatible) or are unsigned.

Ok--so the tradeoff is that telldir/seekdir cookies may not be good
indefinitely?

--b.

RE: readdir cookies

[Myklebust, Trond]

> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org]
> Sent: Monday, September 17, 2012 12:51 PM
> To: Myklebust, Trond
> Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> Subject: Re: readdir cookies
> 
> On Mon, Sep 17, 2012 at 04:05:10PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > Sent: Monday, September 17, 2012 11:54 AM
> > > To: Myklebust, Trond; Schumaker, Bryan
> > > Cc: linux-nfs@vger.kernel.org
> > > Subject: readdir cookies
> > >
> > > How does the nfs client decide what directory cookies to return?
> > > (As the d_off field to readdir, for example).
> > >
> > > I'd assumed it would return the server's cookie, but testing shows
> > > it returns a simple integer sequence.
> >
> > We cache the cookies and use them to figure out where the readdir cursor
> is after a directory update, but we use positive integers for telldir()/seekdir().
> The reason is that too many servers return cookies that use > 32 bits (NFSv2-
> incompatible) or are unsigned.
> 
> Ok--so the tradeoff is that telldir/seekdir cookies may not be good
> indefinitely?

Yes. Most apps don't care about telldir()/seekdir(), so it isn't really much of an issue. However, glibc has a bunch of assumptions about the 'd_offs' entries returned by getdents()/getdents64() and will do "sanity checks" on them that often end up failing when we just shove in the raw NFS cookies.

Cheers
  Trond

Re: readdir cookies

[J. Bruce Fields]

On Mon, Sep 17, 2012 at 04:56:10PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > Sent: Monday, September 17, 2012 12:51 PM
> > To: Myklebust, Trond
> > Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> > Subject: Re: readdir cookies
> > 
> > On Mon, Sep 17, 2012 at 04:05:10PM +0000, Myklebust, Trond wrote:
> > > > -----Original Message-----
> > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > Sent: Monday, September 17, 2012 11:54 AM
> > > > To: Myklebust, Trond; Schumaker, Bryan
> > > > Cc: linux-nfs@vger.kernel.org
> > > > Subject: readdir cookies
> > > >
> > > > How does the nfs client decide what directory cookies to return?
> > > > (As the d_off field to readdir, for example).
> > > >
> > > > I'd assumed it would return the server's cookie, but testing shows
> > > > it returns a simple integer sequence.
> > >
> > > We cache the cookies and use them to figure out where the readdir cursor
> > is after a directory update, but we use positive integers for telldir()/seekdir().
> > The reason is that too many servers return cookies that use > 32 bits (NFSv2-
> > incompatible) or are unsigned.
> > 
> > Ok--so the tradeoff is that telldir/seekdir cookies may not be good
> > indefinitely?
> 
> Yes. Most apps don't care about telldir()/seekdir(), so it isn't really much of an issue. However, glibc has a bunch of assumptions about the 'd_offs' entries returned by getdents()/getdents64() and will do "sanity checks" on them that often end up failing when we just shove in the raw NFS cookies.

OK, thanks.  When did the client start doing that?

(And do you remember any details about the libc problems?  ext4 is
returning 64-bit cookies now in some cases, and we've seen a few
problems elsewhere, but none with glibc.)

--b.

RE: readdir cookies

[Myklebust, Trond]

> -----Original Message-----
> From: J. Bruce Fields [mailto:bfields@fieldses.org]
> Sent: Monday, September 17, 2012 1:50 PM
> To: Myklebust, Trond
> Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> Subject: Re: readdir cookies
> 
> On Mon, Sep 17, 2012 at 04:56:10PM +0000, Myklebust, Trond wrote:
> > > -----Original Message-----
> > > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > > Sent: Monday, September 17, 2012 12:51 PM
> > > To: Myklebust, Trond
> > > Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> > > Subject: Re: readdir cookies
> > >
> > > On Mon, Sep 17, 2012 at 04:05:10PM +0000, Myklebust, Trond wrote:
> > > > > -----Original Message-----
> > > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > > Sent: Monday, September 17, 2012 11:54 AM
> > > > > To: Myklebust, Trond; Schumaker, Bryan
> > > > > Cc: linux-nfs@vger.kernel.org
> > > > > Subject: readdir cookies
> > > > >
> > > > > How does the nfs client decide what directory cookies to return?
> > > > > (As the d_off field to readdir, for example).
> > > > >
> > > > > I'd assumed it would return the server's cookie, but testing
> > > > > shows it returns a simple integer sequence.
> > > >
> > > > We cache the cookies and use them to figure out where the readdir
> > > > cursor
> > > is after a directory update, but we use positive integers for
> telldir()/seekdir().
> > > The reason is that too many servers return cookies that use > 32
> > > bits (NFSv2-
> > > incompatible) or are unsigned.
> > >
> > > Ok--so the tradeoff is that telldir/seekdir cookies may not be good
> > > indefinitely?
> >
> > Yes. Most apps don't care about telldir()/seekdir(), so it isn't really much of
> an issue. However, glibc has a bunch of assumptions about the 'd_offs'
> entries returned by getdents()/getdents64() and will do "sanity checks" on
> them that often end up failing when we just shove in the raw NFS cookies.
> 
> OK, thanks.  When did the client start doing that?

Years ago. You want to look at commit 00a2642 (NFS: Hide NFS server-generated readdir cookies from userland) which dates back to Linux-2.6.13.
 
> (And do you remember any details about the libc problems?  ext4 is
> returning 64-bit cookies now in some cases, and we've seen a few problems
> elsewhere, but none with glibc.)

IIRC: At one point, in glibc-2(.1?), they started using getdents64() instead of getdents() in order to avoid the problem of mismatched 'struct dirent' sizes on 32-bit ix86 platforms. Whenever the getdents64() interface returned a 64-bit cookie (or a 32-bit negative cookie), glibc would generate an EOVERFLOW error.

Cheers
  Trond

Re: readdir cookies

[J. Bruce Fields]

On Mon, Sep 17, 2012 at 06:03:26PM +0000, Myklebust, Trond wrote:
> > -----Original Message-----
> > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > Sent: Monday, September 17, 2012 1:50 PM
> > To: Myklebust, Trond
> > Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> > Subject: Re: readdir cookies
> > 
> > On Mon, Sep 17, 2012 at 04:56:10PM +0000, Myklebust, Trond wrote:
> > > > -----Original Message-----
> > > > From: J. Bruce Fields [mailto:bfields@fieldses.org]
> > > > Sent: Monday, September 17, 2012 12:51 PM
> > > > To: Myklebust, Trond
> > > > Cc: Schumaker, Bryan; linux-nfs@vger.kernel.org
> > > > Subject: Re: readdir cookies
> > > >
> > > > On Mon, Sep 17, 2012 at 04:05:10PM +0000, Myklebust, Trond wrote:
> > > > > > -----Original Message-----
> > > > > > From: linux-nfs-owner@vger.kernel.org [mailto:linux-nfs-
> > > > > > owner@vger.kernel.org] On Behalf Of J. Bruce Fields
> > > > > > Sent: Monday, September 17, 2012 11:54 AM
> > > > > > To: Myklebust, Trond; Schumaker, Bryan
> > > > > > Cc: linux-nfs@vger.kernel.org
> > > > > > Subject: readdir cookies
> > > > > >
> > > > > > How does the nfs client decide what directory cookies to return?
> > > > > > (As the d_off field to readdir, for example).
> > > > > >
> > > > > > I'd assumed it would return the server's cookie, but testing
> > > > > > shows it returns a simple integer sequence.
> > > > >
> > > > > We cache the cookies and use them to figure out where the readdir
> > > > > cursor
> > > > is after a directory update, but we use positive integers for
> > telldir()/seekdir().
> > > > The reason is that too many servers return cookies that use > 32
> > > > bits (NFSv2-
> > > > incompatible) or are unsigned.
> > > >
> > > > Ok--so the tradeoff is that telldir/seekdir cookies may not be good
> > > > indefinitely?
> > >
> > > Yes. Most apps don't care about telldir()/seekdir(), so it isn't really much of
> > an issue. However, glibc has a bunch of assumptions about the 'd_offs'
> > entries returned by getdents()/getdents64() and will do "sanity checks" on
> > them that often end up failing when we just shove in the raw NFS cookies.
> > 
> > OK, thanks.  When did the client start doing that?
> 
> Years ago. You want to look at commit 00a2642 (NFS: Hide NFS server-generated readdir cookies from userland) which dates back to Linux-2.6.13.
>  
> > (And do you remember any details about the libc problems?  ext4 is
> > returning 64-bit cookies now in some cases, and we've seen a few problems
> > elsewhere, but none with glibc.)
> 
> IIRC: At one point, in glibc-2(.1?), they started using getdents64() instead of getdents() in order to avoid the problem of mismatched 'struct dirent' sizes on 32-bit ix86 platforms. Whenever the getdents64() interface returned a 64-bit cookie (or a 32-bit negative cookie), glibc would generate an EOVERFLOW error.

Got it, thanks.

--b.