Auto-repo-repair

Auto-repo-repair

[Enrico Weigelt]

Hi folks,

suppose the following scenario:

I've broken some repo (missing objects), eg by messing something up
w/ alternates, broken filesystem, or whatever. And I've got a bunch
of remotes which (together) contain all of the lost objects.

Now I'd like to run some $magic_command which automatically fetches
all the missing objects and so repair my local repo.

Is this already possible right now ?


thx
-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de 

Re: Auto-repo-repair

[Jeff King]

On Fri, Nov 16, 2012 at 06:51:45PM +0100, Enrico Weigelt wrote:

> I've broken some repo (missing objects), eg by messing something up
> w/ alternates, broken filesystem, or whatever. And I've got a bunch
> of remotes which (together) contain all of the lost objects.
> 
> Now I'd like to run some $magic_command which automatically fetches
> all the missing objects and so repair my local repo.
> 
> Is this already possible right now ?

You can't reliably just grab the broken objects, because most transports
don't support grabbing arbitrary objects (you can do it if you have
shell access to a known-good repository, but it's not automated).

The simplest thing is usually to re-clone the known-good remotes, then
copy the resulting packfiles into your original repository. You'll have
duplicated objects until your next "gc", but the resulting repack should
skip any corrupted objects and use the known-good ones.

-Peff

Re: Auto-repo-repair

[Enrico Weigelt]

Hi,

> You can't reliably just grab the broken objects, because most
> transports
> don't support grabbing arbitrary objects (you can do it if you have
> shell access to a known-good repository, but it's not automated).

can we introduce a new or extend existing transports to support that ?


cu
-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de 

Re: Auto-repo-repair

[Drew Northup]

On Sat, Nov 17, 2012 at 4:21 AM, Enrico Weigelt <enrico.weigelt@vnc.biz> wrote:
> Hi,
>
>> You can't reliably just grab the broken objects, because most
>> transports
>> don't support grabbing arbitrary objects (you can do it if you have
>> shell access to a known-good repository, but it's not automated).
>
> can we introduce a new or extend existing transports to support that ?

How would the broken repository be sure of what it is missing to
request it from the other side?

-- 
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

Re: Auto-repo-repair

[Enrico Weigelt]

> How would the broken repository be sure of what it is missing to
> request it from the other side?

fsck will find missing objects.


> 
> --
> -Drew Northup
> --------------------------------------------------------------
> "As opposed to vegetable or mineral error?"
> -John Pescatore, SANS NewsBites Vol. 12 Num. 59
> 

-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de 

Re: Auto-repo-repair

[Drew Northup]

On Sun, Nov 18, 2012 at 11:55 AM, Enrico Weigelt <enrico.weigelt@vnc.biz> wrote:
>
>> How would the broken repository be sure of what it is missing to
>> request it from the other side?
>
> fsck will find missing objects.

And what about the objects referred to by objects that are missing?
Jeff's solution doesn't suffer from this recursivity problem.

-- 
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

Re: Auto-repo-repair

[Enrico Weigelt]

> >> How would the broken repository be sure of what it is missing to
> >> request it from the other side?
> >
> > fsck will find missing objects.
> 
> And what about the objects referred to by objects that are missing?

Will be fetched after multiple iterations.
We could even introduce some 'fsck --autorepair' mode, which triggers
it to fetch any missing object from its remotes.

Maybe even introduce a concept of peer object stores, which (a bit like
alternates) are asked for objects that arent locally availabe - that
could be even a plain venti store.


cu
-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de 

Re: Auto-repo-repair

[Drew Northup]

On Mon, Nov 19, 2012 at 5:35 PM, Enrico Weigelt <enrico.weigelt@vnc.biz> wrote:
>
>> >> How would the broken repository be sure of what it is missing to
>> >> request it from the other side?
>> >
>> > fsck will find missing objects.
>>
>> And what about the objects referred to by objects that are missing?
>
> Will be fetched after multiple iterations.
> We could even introduce some 'fsck --autorepair' mode, which triggers
> it to fetch any missing object from its remotes.
>
> Maybe even introduce a concept of peer object stores, which (a bit like
> alternates) are asked for objects that arent locally availabe - that
> could be even a plain venti store.

I still think that it would make the most sense to do the following
(if you insist on some sort of automated repair):
(1) Fetch a "good" clone (or clones) into a temporary directory;
(2) Cannibalize the objects from it (them);
(3) Re-run git fsck and check for still-missing / unreachable items;
(4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the
mess, discard / "merge" duplicate objects, and fix up the packfiles.

It is step (4) that requires the most user interaction. I could see
building up a shell script that does all but (4) nearly automatically.
None of this requires modifying Git itself.

-- 
-Drew Northup
--------------------------------------------------------------
"As opposed to vegetable or mineral error?"
-John Pescatore, SANS NewsBites Vol. 12 Num. 59

Re: Auto-repo-repair

[Enrico Weigelt]

Hi,

> I still think that it would make the most sense to do the following
> (if you insist on some sort of automated repair):
> (1) Fetch a "good" clone (or clones) into a temporary directory;
> (2) Cannibalize the objects from it (them);
> (3) Re-run git fsck and check for still-missing / unreachable items;
> (4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the
> mess, discard / "merge" duplicate objects, and fix up the packfiles.
> 
> It is step (4) that requires the most user interaction. I could see
> building up a shell script that does all but (4) nearly
> automatically.
> None of this requires modifying Git itself.

Well, I'd like to have some really automatic mode, which does
everything ondemand.

Once we've got this not just for repair, but also to support
quick partial clones that fetch more objects when required.

In fact, finally, I'd like to have some storage cloud where
data automatically gets replicated to nodes which need the data,
not just for VCS, but other purposes (backup, filestore, etc) too.
But before inventing someting completely new (reinventing much
of the wheel), I'd like to investigate whether git can be
extended into this direction step by step.


cu
-- 
Mit freundlichen Grüßen / Kind regards 

Enrico Weigelt 
VNC - Virtual Network Consult GmbH 
Head Of Development 

Pariser Platz 4a, D-10117 Berlin
Tel.: +49 (30) 3464615-20
Fax: +49 (30) 3464615-59

enrico.weigelt@vnc.biz; www.vnc.de