broken repo after power cut

broken repo after power cut

[Richard Weinberger]

Hi!

Yesterday our git server faced a power cut and a git repository broke.
The server is running a ext4 filesystem on top of Linux 3.16 (stable from openSUSE) and git 2.1.4.
We had a backup, so no data was lost but I really would like to figure out
what happened.

This is the output of git fsck:
Checking object directories: 100% (256/256), done.
error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt

To me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it.
As the object was referenced before syncing the data to disk the repo broke.
Could this have happened?
Also, is git designed to survive power cuts? Then referencing an object before synching it do disk would make no sense.

Thanks,
//richard

Re: broken repo after power cut

[Johannes Schindelin]

Hi Richard,

On 2015-06-20 21:40, Richard Weinberger wrote:

> Yesterday our git server faced a power cut and a git repository broke.
> The server is running a ext4 filesystem on top of Linux 3.16 (stable
> from openSUSE) and git 2.1.4.
> We had a backup, so no data was lost but I really would like to figure out
> what happened.
> 
> This is the output of git fsck:
> Checking object directories: 100% (256/256), done.
> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
> fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored
> in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt
> 
> To me it seems like git was creating a new object and got interrupted
> before fsync/fdatasync'ing it.
> As the object was referenced before syncing the data to disk the repo broke.
> Could this have happened?
> Also, is git designed to survive power cuts? Then referencing an
> object before synching it do disk would make no sense.

I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.

I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.

Maybe this leads you into the direction of a work-around in your setup?

Ciao,
Johannes

Re: broken repo after power cut

[Richard Weinberger]

Hi Johannes,

[CC'ing linux-fsdevel and tytso]

Am 21.06.2015 um 14:28 schrieb Johannes Schindelin:
> Hi Richard,
> 
> On 2015-06-20 21:40, Richard Weinberger wrote:
> 
>> Yesterday our git server faced a power cut and a git repository broke.
>> The server is running a ext4 filesystem on top of Linux 3.16 (stable
>> from openSUSE) and git 2.1.4.
>> We had a backup, so no data was lost but I really would like to figure out
>> what happened.
>>
>> This is the output of git fsck:
>> Checking object directories: 100% (256/256), done.
>> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
>> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
>> fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored
>> in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt
>>
>> To me it seems like git was creating a new object and got interrupted
>> before fsync/fdatasync'ing it.
>> As the object was referenced before syncing the data to disk the repo broke.
>> Could this have happened?
>> Also, is git designed to survive power cuts? Then referencing an
>> object before synching it do disk would make no sense.
> 
> I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.
> 
> I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.

You mean the ext4 delayed block allocation feature/issue?
IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE).
But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems.
Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered.

> Maybe this leads you into the direction of a work-around in your setup?

I'm still not sure who to blame. ;-)

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in

Re: broken repo after power cut

[Christoph Hellwig]

On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:
> >> To me it seems like git was creating a new object and got interrupted
> >> before fsync/fdatasync'ing it.
> >> As the object was referenced before syncing the data to disk the repo broke.

Git doesn't fsync by default, and because of that I've seen similar
data losses on ext4/xfs/btrfs.

You can set the core.fsyncobjectfiles to mitigate it, but even with
that I've seen corrupted index files.

Note that I've been mostly on old git versions from various distros,
so in case this was fixed recently I'll take everything I said back.

Re: broken repo after power cut

[Richard Weinberger]

Am 21.06.2015 um 15:59 schrieb Christoph Hellwig:
> On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:
>>>> To me it seems like git was creating a new object and got interrupted
>>>> before fsync/fdatasync'ing it.
>>>> As the object was referenced before syncing the data to disk the repo broke.
> 
> Git doesn't fsync by default, and because of that I've seen similar
> data losses on ext4/xfs/btrfs.
> 
> You can set the core.fsyncobjectfiles to mitigate it, but even with
> that I've seen corrupted index files.

Yeah, after inspecting git's source I've found that config option too.
Now it's also crystal clear that git is not power cut safe at all by default. ;-\

So, anyone that cares about his repos has to enable core.fsyncobjectfiles,
which is IMHO kind of sad.

Thanks,
//richard
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in

Re: broken repo after power cut

[Theodore Ts'o]

On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:

> > I was then shocked to learn that ext4 apparently has a default
> > setting that allows it to truncate files upon power failure
> > (something about a full journal vs a fast journal or some such)

s/ext4/all modern file systems/

POSIX makes **no guarantees** about what happens after a power failure
unless you use fsync() --- which git does not do by default (see below).

> You mean the ext4 delayed block allocation feature/issue?
> IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE).
> But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems.
> Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered.

The hacks (which were agreed upon by all of the major file system
developers --- ext4, btfs, xfs --- at the Linux File Systems and
Storage summit a couple of years ago --- protects against the default
text editors of GNOME and KDE which were saving file without using
fsync(), and in one particularly egregious example (although I don't
remember which program was doing this), updated files by opening the
file with O_TRUNC and then rewritng the new contents of the file.  So
if you crashed just after the open(2), and before the file data was
written, you were guaranteed to lose data.

The hack protects against data loss when programs updated a file
incompetently.  What we agreed to do was that upon renaming a fileA on
top of another fileB, there is an implicit writeback initiated of
fileA.  If the program properly called fsync(2) before closing the
file descriptor for fileA and doing the rename, this implicit
writeback would be no-op.  Simiarly, if a file descriptor was opened
with O_TRUNC, when the file descriptor is closed, we start an implicit
writeback at that point.  Note that this is not the same as a full
fsync; it merely closes the race window from 30 seconds down to a
second or so (depending on how busy the disk is).

But this hack does not protect against freshly written files, which is
the case of git object files or git pack files.  The basic idea here
is that you could have just as easily crashed before the commit as
after the commit, and doing an implicit writeback after all file
closes would have destroyed performance and penalized progams that
didn't really care so much about the file hitting disk.  (For example,
if you do a compile, and you crash, it's not such a big deal.)

The bottome lins is that if you care about files being written, you
need to use fsync().  Should git use fsync() by default?  Well, if you
are willing to accept that if your system crashes within a second or
so of your last git operation, you might need to run "git fsck" and
potentially recover from a busted repo, maybe speed is more important
for you (and git is known for its speed/performance, after all. :-)

The actual state of the source tree would have been written using a
text editor which tends to be paranoid about using fsync (at least, if
you use a real editor like Emacs or Vi, as opposed to the toy notepad
editors shipped with GNOME or KDE :-).  So as long as you know what
you're doing, it's unlikely that you will actually lose any work.

Personally, I have core.fsyncobjectfiles set to yes in my .gitconfig.
Part of this is because I have an SSD, so the speed hit really doesn't
bother me, and needing to recover a corrupted git repository is a pain
(although I have certainly done it in the past).

						- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in

Re: broken repo after power cut

[Richard Weinberger]

Am 22.06.2015 um 02:35 schrieb Theodore Ts'o:
> On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:
> 
>>> I was then shocked to learn that ext4 apparently has a default
>>> setting that allows it to truncate files upon power failure
>>> (something about a full journal vs a fast journal or some such)
> 
> s/ext4/all modern file systems/
> 
> POSIX makes **no guarantees** about what happens after a power failure
> unless you use fsync() --- which git does not do by default (see below).

Thanks for pointing this out.

> The bottome lins is that if you care about files being written, you
> need to use fsync().  Should git use fsync() by default?  Well, if you
> are willing to accept that if your system crashes within a second or
> so of your last git operation, you might need to run "git fsck" and
> potentially recover from a busted repo, maybe speed is more important
> for you (and git is known for its speed/performance, after all. :-)
> 
> The actual state of the source tree would have been written using a
> text editor which tends to be paranoid about using fsync (at least, if
> you use a real editor like Emacs or Vi, as opposed to the toy notepad
> editors shipped with GNOME or KDE :-).  So as long as you know what
> you're doing, it's unlikely that you will actually lose any work.
> 
> Personally, I have core.fsyncobjectfiles set to yes in my .gitconfig.
> Part of this is because I have an SSD, so the speed hit really doesn't
> bother me, and needing to recover a corrupted git repository is a pain
> (although I have certainly done it in the past).

I think core.fsyncObjectFiles documentation really needs an update.
What about this one?

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 43bb53c..b08fa11 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -693,10 +693,16 @@ core.whitespace::
 core.fsyncObjectFiles::
 	This boolean will enable 'fsync()' when writing object files.
 +
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
+For performance reasons git does not call 'fsync()' after writing object
+files. This means that after a power cut your git repository can get
+corrupted as not all data hit the storage media. Especially on modern
+filesystems like ext4, xfs or btrfs this can happen very easily.
+If you have to face power cuts and care about your data it is strongly
+recommended to enable this setting.
+Please note that git's behavior used to be safe on ext3 with data=ordered,
+for any other filesystems or mount settings this is not the case as
+POSIX clearly states that you have to call 'fsync()' to make sure that
+all data is written.

 core.preloadIndex::
 	Enable parallel index preload for operations like 'git diff'

--
Thanks,
//richard

Re: broken repo after power cut

[Theodore Ts'o]

On Mon, Jun 22, 2015 at 01:19:59PM +0200, Richard Weinberger wrote:
> 
> > The bottome lins is that if you care about files being written, you
> > need to use fsync().  Should git use fsync() by default?  Well, if you
> > are willing to accept that if your system crashes within a second or
> > so of your last git operation, you might need to run "git fsck" and
> > potentially recover from a busted repo, maybe speed is more important
> > for you (and git is known for its speed/performance, after all. :-)

I made a typo in the above.  s/second/minute/.  (Linux's writeback
timer is 30 seconds, but if the disk is busy it might take a bit
longer to get all of the data blocks written out to disk and
committed.)

> I think core.fsyncObjectFiles documentation really needs an update.
> What about this one?
> 
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 43bb53c..b08fa11 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -693,10 +693,16 @@ core.whitespace::
>  core.fsyncObjectFiles::
>  	This boolean will enable 'fsync()' when writing object files.
>  +
> -This is a total waste of time and effort on a filesystem that orders
> -data writes properly, but can be useful for filesystems that do not use
> -journalling (traditional UNIX filesystems) or that only journal metadata
> -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
> +For performance reasons git does not call 'fsync()' after writing object
> +files. This means that after a power cut your git repository can get
> +corrupted as not all data hit the storage media. Especially on modern
> +filesystems like ext4, xfs or btrfs this can happen very easily.
> +If you have to face power cuts and care about your data it is strongly
> +recommended to enable this setting.
> +Please note that git's behavior used to be safe on ext3 with data=ordered,
> +for any other filesystems or mount settings this is not the case as
> +POSIX clearly states that you have to call 'fsync()' to make sure that
> +all data is written.


My main complaint about this is that it's a bit Linux-centric.  For
example, the fact that fsync(2) is needed to push data out of the
cache is also true for MacOS (and indeed all other Unix systems going
back three decades) as well as Windows.  In fact, it's not a matter of
"POSIX says", but "POSIX documented", but since standards are held in
high esteem, it's sometimes a bit more convenient to use them as an
appeal to authority.  :-)

(Ext3's data=ordered behaviour is an outlier, and in fact, the reason
why it mostly safe to skip fsync(2) calls when using ext3 data=ordered
was an accidental side effect of another problem which was trying to
solve based on the relatively primitive way it handled block
allocation.)

Cheers,

						- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in