Re: fsync() errors is unsafe and risks data loss

[bfields@fieldses.org (J. Bruce Fields)]

On Wed, Apr 11, 2018 at 07:17:52PM -0700, Andres Freund wrote:
> Hi,
> 
> On 2018-04-11 15:52:44 -0600, Andreas Dilger wrote:
> > On Apr 10, 2018, at 4:07 PM, Andres Freund <andres@anarazel.de> wrote:
> > > 2018-04-10 18:43:56 Ted wrote:
> > >> So for better or for worse, there has not been as much investment in
> > >> buffered I/O and data robustness in the face of exception handling of
> > >> storage devices.
> > > 
> > > That's a bit of a cop out. It's not just databases that care. Even more
> > > basic tools like SCM, package managers and editors care whether they can
> > > proper responses back from fsync that imply things actually were synced.
> > 
> > Sure, but it is mostly PG that is doing (IMHO) crazy things like writing
> > to thousands(?) of files, closing the file descriptors, then expecting
> > fsync() on a newly-opened fd to return a historical error.
> 
> It's not just postgres. dpkg (underlying apt, on debian derived distros)
> to take an example I just randomly guessed, does too:
>   /* We want to guarantee the extracted files are on the disk, so that the
>    * subsequent renames to the info database do not end up with old or zero
>    * length files in case of a system crash. As neither dpkg-deb nor tar do
>    * explicit fsync()s, we have to do them here.
>    * XXX: This could be avoided by switching to an internal tar extractor. */
>   dir_sync_contents(cidir);
> 
> (a bunch of other places too)
> 
> Especially on ext3 but also on newer filesystems it's performancewise
> entirely infeasible to fsync() every single file individually - the
> performance becomes entirely attrocious if you do that.

Is that still true if you're able to use some kind of parallelism?
(async io, or fsync from multiple processes?)

--b.