[Question] Behaviour of directory listing in XFS with concurrent deletes

[Question] Behaviour of directory listing in XFS with concurrent deletes

[Vairavanathan, Emalayan]

Hi There,


We have an application that  iterates through a large directory (that has around 1.2 million files)  and lists the names of all the files in the directory. 

Files stored in the directory could be concurrently deleted by another thread in the application while the application lists the files. 

Our application can tolerate false positives when listing a directory. However false negatives in directory listing *cannot* be tolerated.

To this end could someone tell  me how directory listing in XFS behaves with concurrent deletes? Is it  possible to see false negatives ?

Thank you
 Emalayan

Re: [Question] Behaviour of directory listing in XFS with concurrent deletes

[Dave Chinner]

On Thu, Apr 05, 2018 at 11:53:18PM +0000, Vairavanathan, Emalayan wrote:
> Hi There,
> 
> 
> We have an application that  iterates through a large directory
> (that has around 1.2 million files)  and lists the names of all
> the files in the directory. 
> 
> Files stored in the directory could be concurrently deleted by
> another thread in the application while the application lists the
> files. 

XFS complies with Posix requirements for stable directory offsets.
i.e. that dirents don't change offset when other dirents are added
or removed. Hence sequential readdir calls will always start off at
the dirent offset the last call finished at.

However, this guarantees nothing about individual dirent state
when concurrent modifications occur between readdir calls.

> Our application can tolerate false positives when listing a
> directory. However false negatives in directory listing *cannot*
> be tolerated.
> 
> To this end could someone tell  me how directory listing in XFS
> behaves with concurrent deletes? Is it  possible to see false
> negatives ?

What's does "false-negative" mean? I don't know what your
application is trying to do, so I don't know what your failure
criteria is....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [Question] Behaviour of directory listing in XFS with concurrent deletes

[Vairavanathan, Emalayan]

Hi Dave,

Thank you for getting back to me. Please see my response inline.

>> What's does "false-negative" mean?  I don't know what your application is trying to do, so I don't know what your failure criteria is...

Assume the following :

1) No failures (application crash OR disk / sector errors OR any form of disk or memory corruption)
2) xfs file-system is mounted with dirsync option.
3) A directory in XFS with 1.2 million files. fsync() was called on each file after creation.
4) An application with two threads (Thread-A and Thread-B) that uses this directory. Thread-A iterates through the directory and lists the names of all the files in the directory using readdir calls. The Thread-B  concurrently deletes some files from the directory.
5) Files can be only deleted from the directory. No new files will be created once the directory listing is started and we have a coordination layer to ensure this. 

Given this context:
- False-positive means a file that was deleted by Thread-B is seen by Thread-A. 
- False-negative means a file that is already in the directory is not seen by Thread-A (due to concurrent deletes of some other files in the directory).

Does this make sense ? I know that false-positives are possible in this context. Is it possible to see false negatives in this context? 

Thank you. I really appreciate your answers.
Emalayan



From: Dave Chinner <david@fromorbit.com>
Sent: April 5, 2018 6:18:08 PM
To: Vairavanathan, Emalayan
Cc: linux-xfs@vger.kernel.org
Subject: Re: [Question] Behaviour of directory listing in XFS with concurrent deletes
  

On Thu, Apr 05, 2018 at 11:53:18PM +0000, Vairavanathan, Emalayan wrote:
> Hi There,
> 
> 
> We have an application that  iterates through a large directory
> (that has around 1.2 million files)  and lists the names of all
> the files in the directory. 
> 
> Files stored in the directory could be concurrently deleted by
> another thread in the application while the application lists the
> files. 

XFS complies with Posix requirements for stable directory offsets.
i.e. that dirents don't change offset when other dirents are added
or removed. Hence sequential readdir calls will always start off at
the dirent offset the last call finished at.

However, this guarantees nothing about individual dirent state
when concurrent modifications occur between readdir calls.

> Our application can tolerate false positives when listing a
> directory. However false negatives in directory listing *cannot*
> be tolerated.
> 
> To this end could someone tell  me how directory listing in XFS
> behaves with concurrent deletes? Is it  possible to see false
> negatives ?

What's does "false-negative" mean? I don't know what your
application is trying to do, so I don't know what your failure
criteria is....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com
     

Re: [Question] Behaviour of directory listing in XFS with concurrent deletes

[Dave Chinner]

[ Please word wrap your emails at 72 columns! ]

On Tue, Apr 10, 2018 at 12:24:55AM +0000, Vairavanathan, Emalayan
wrote:
> Hi Dave,
> 
> Thank you for getting back to me. Please see my response inline.
> 
> >> What's does "false-negative" mean?  I don't know what your
> >> application is trying to do, so I don't know what your failure
> >> criteria is...
> 
> Assume the following :
> 
> 1) No failures (application crash OR disk / sector errors OR any
> form of disk or memory corruption)
> 2) xfs file-system is mounted with dirsync option.
> 3) A directory in XFS with 1.2 million files. fsync() was called
> on each file after creation.

Directory Size, dirsync, fsync, and failures are irrelevant to
readdir vs concurrent unlink behaviour.

> 4) An application with two threads (Thread-A and Thread-B) that
> uses this directory. Thread-A iterates through the directory and
> lists the names of all the files in the directory using readdir
> calls. The Thread-B  concurrently deletes some files from the
> directory.
> 5) Files can be only deleted from the directory. No new files will
> be created once the directory listing is started and we have a
> coordination layer to ensure this. 

> Given this context:
> - False-positive means a file that was deleted by Thread-B is seen
> by Thread-A. 
> - False-negative means a file that is already in the directory is
> not seen by Thread-A (due to concurrent deletes of some other
> files in the directory).
> 
> Does this make sense ?

Yes.

> I know that false-positives are possible in this context.

Right, POSIX explicitly states that behaviour of readdir() w.r.t.
files added/deleted after opendir()/rewinddir() is unspecified.

> Is it possible to see false negatives in this
> context? 

No.

-Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [Question] Behaviour of directory listing in XFS with concurrent deletes

[Vairavanathan, Emalayan]

Thank you Dave. This helps a lot.

>> [ Please word wrap your emails at 72 columns! ]

I will remember to do this in future.

- Emalayan

From: Dave Chinner <david@fromorbit.com>
Sent: April 10, 2018 2:20 AM
To: Vairavanathan, Emalayan
Cc: linux-xfs@vger.kernel.org
Subject: Re: [Question] Behaviour of directory listing in XFS with concurrent deletes
  

[ Please word wrap your emails at 72 columns! ]

On Tue, Apr 10, 2018 at 12:24:55AM +0000, Vairavanathan, Emalayan
wrote:
> Hi Dave,
> 
> Thank you for getting back to me. Please see my response inline.
> 
> >> What's does "false-negative" mean?  I don't know what your
> >> application is trying to do, so I don't know what your failure
> >> criteria is...
> 
> Assume the following :
> 
> 1) No failures (application crash OR disk / sector errors OR any
> form of disk or memory corruption)
> 2) xfs file-system is mounted with dirsync option.
> 3) A directory in XFS with 1.2 million files. fsync() was called
> on each file after creation.

Directory Size, dirsync, fsync, and failures are irrelevant to
readdir vs concurrent unlink behaviour.

> 4) An application with two threads (Thread-A and Thread-B) that
> uses this directory. Thread-A iterates through the directory and
> lists the names of all the files in the directory using readdir
> calls. The Thread-B  concurrently deletes some files from the
> directory.
> 5) Files can be only deleted from the directory. No new files will
> be created once the directory listing is started and we have a
> coordination layer to ensure this. 

> Given this context:
> - False-positive means a file that was deleted by Thread-B is seen
> by Thread-A. 
> - False-negative means a file that is already in the directory is
> not seen by Thread-A (due to concurrent deletes of some other
> files in the directory).
> 
> Does this make sense ?

Yes.

> I know that false-positives are possible in this context.

Right, POSIX explicitly states that behaviour of readdir() w.r.t.
files added/deleted after opendir()/rewinddir() is unspecified.

> Is it possible to see false negatives in this
> context? 

No.

-Dave.
-- 
Dave Chinner
david@fromorbit.com