FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[Jason Pyeron]

I remember hitting this a while ago, but just gave up.

It seems to be a problem for others too.

Any ideas on how to debug this so it can be patched?

-----Original Message-----
From: Dave Lindbergh
Sent: Wednesday, December 03, 2014 18:07
To: cygwin
Subject: Re: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

On Wed, Dec 3, 2014 at 3:42 PM, Jason Pyeron <jpyeron@pdinc.us> wrote:
> [copy off list, because the sourceware system admins throws temper tantrums]
>> -----Original Message-----
>> From: Dave L
>> Sent: Wednesday, December 03, 2014 15:30
>>
>> ...but the git from github.com works fine.
>>
>> I installed Cygwin's version of git, and get this:
>>
>>     $ git clone https://github.com/nerdfever/pic32mx-bmf
>>     Cloning into 'pic32mx-bmf'...
>>     remote: Counting objects: 12, done.
>>     remote: Total 12 (delta 0), reused 0 (delta 0)
>>     error: failed to read delta-pack base object
>>     300bdeb2fd209d24afb865584da10b78aa8fefc4
>>     fatal: unpack-objects failed
>
> What file system are you on? Local NTFS or remote?

Aha - you're right.

It works fine on a local NTFS volume.

I get the error when I do it on Z:, which is mapped to a network drive
(on another Windows box).

Is there a workaround? Why does this happen?

--Dave

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple

Re: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[brian m. carlson]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> I remember hitting this a while ago, but just gave up.
> 
> It seems to be a problem for others too.
> 
> Any ideas on how to debug this so it can be patched?
> 
> -----Original Message-----
> From: Dave Lindbergh
> Sent: Wednesday, December 03, 2014 18:07
> To: cygwin
> 
> Aha - you're right.
> 
> It works fine on a local NTFS volume.
> 
> I get the error when I do it on Z:, which is mapped to a network drive
> (on another Windows box).
> 
> Is there a workaround? Why does this happen?

I don't think anyone is sure.  My wild guess is that there's something
about the way that Cygwin wraps Windows calls that makes it fail in this
case.  It might be interesting to run the Windows and Cygwin versions
under an strace equivalent and see where things differ.

It's an interesting problem, but I don't have any Windows systems, so I
can't look into it.
-- 
brian m. carlson / brian with sandals: Houston, Texas, US
+1 832 623 2791 | http://www.crustytoothpaste.net/~bmc | My opinion only
OpenPGP: RSA v4 4096b: 88AC E9B2 9196 305B A994 7552 F1BA 225C 0223 B187

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[Jason Pyeron]

> -----Original Message-----
> From: brian m. carlson
> Sent: Wednesday, December 03, 2014 19:55
> 
> On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > I remember hitting this a while ago, but just gave up.
> > 
> > It seems to be a problem for others too.
> > 
> > Any ideas on how to debug this so it can be patched?
> > 
> > -----Original Message-----
> > From: Dave Lindbergh
> > Sent: Wednesday, December 03, 2014 18:07
> > To: cygwin
> > 
> > Aha - you're right.
> > 
> > It works fine on a local NTFS volume.
> > 
> > I get the error when I do it on Z:, which is mapped to a 
> network drive
> > (on another Windows box).
> > 
> > Is there a workaround? Why does this happen?
> 
> I don't think anyone is sure.  My wild guess is that there's something
> about the way that Cygwin wraps Windows calls that makes it 
> fail in this
> case.  It might be interesting to run the Windows and Cygwin versions
> under an strace equivalent and see where things differ.

[this was posted to the cygwin list]

http://nerdfever.com/files/strace.txt

> 
> It's an interesting problem, but I don't have any Windows 
> systems, so I
> can't look into it.

If it becomes a little less magic, I will chomp on the problem, but I cannot make a predictable test case.

-Jason

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00. 

RE: SPAM: RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[Jason Pyeron]

> -----Original Message-----
> From: git-owner@vger.kernel.org 
> [mailto:git-owner@vger.kernel.org] On Behalf Of Jason Pyeron
> Sent: Thursday, December 04, 2014 16:34
> To: git@vger.kernel.org
> Cc: 'brian m. carlson'
> Subject: SPAM: RE: FW: [cygwin] Cygwin's git says "error: 
> failed to read delta-pack base object"
> 
> > -----Original Message-----
> > From: brian m. carlson
> > Sent: Wednesday, December 03, 2014 19:55
> > 
> > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > I remember hitting this a while ago, but just gave up.
> > > 
> > > It seems to be a problem for others too.
> > > 
> > > Any ideas on how to debug this so it can be patched?
> > > 
> > > -----Original Message-----
> > > From: Dave Lindbergh
> > > Sent: Wednesday, December 03, 2014 18:07
> > > To: cygwin
> > > 
> > > Aha - you're right.
> > > 
> > > It works fine on a local NTFS volume.
> > > 
> > > I get the error when I do it on Z:, which is mapped to a 
> > network drive
> > > (on another Windows box).
> > > 
> > > Is there a workaround? Why does this happen?
> > 
> > I don't think anyone is sure.  My wild guess is that 
> there's something
> > about the way that Cygwin wraps Windows calls that makes it 
> > fail in this
> > case.  It might be interesting to run the Windows and 
> Cygwin versions
> > under an strace equivalent and see where things differ.
> 
> [this was posted to the cygwin list]
> 
> http://nerdfever.com/files/strace.txt
> 
> > 
> > It's an interesting problem, but I don't have any Windows 
> > systems, so I
> > can't look into it.
> 
> If it becomes a little less magic, I will chomp on the 
> problem, but I cannot make a predictable test case.

Corrina at Cygwin devined the strace (see below) and I am working on a test cases and hacks.

Pseudo code and observations
./sha1_file.c:write_loose_object(sha1)
{
 filename=sha1_file_name(sha1)
 (fd,tmp_file)=create_tmpfile(filename)
 write_buffer(fd)
 close_sha1_file(fd)
 return move_temp_to_file(tmp_file, filename)
}

move_temp_to_file(tmpfile, filename)
{
 // I am thinking about forcing renames to see if the problem exists then as well
 // if that "works" then a per repo config option allowing for forced renames
 if (OBJECT_CREATION_USES_RENAMES) goto try_rename
 else if link(tmpfile,filename)

 if (failed except exist failures)
 {
  try_rename:
  rename(tmpfile, filename)
  if (ok) goto out
 }
 unlink_or_warn(tmpfile)
 if (failed except exist failures) return error
 out:
}

write_sha1_file(sha1)
{
 return write_loose_object(sha1)
}


> -----Original Message-----
> From: Corinna Vinschen
> Sent: Friday, December 05, 2014 6:35
> To: cygwin@cygwin.com
<snip/>
> What I found in the strace is this:
> 
> - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
> 
> - open file, write something, close file.
> 
> - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
> 	
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
>   succeeds.
> 
> - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
> 
> - Trying to open
>   
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
>   but the file doesn't exist and NtCreateFile fails with status
>   0xC0000034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
> 
> - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
>   STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
> 
> - git seems to be prepared for such a case, the parent process calls
>   opendir/readdir on the directory.  Enumerating the files in
>   Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
>   "0bdeb2fd209d24afb865584da10b78aa8fefc4".
> 
> - Then git calls lstat on the file, which results in NtOpenFile
>   returning status STATUS_OBJECT_NAME_NOT_FOUND again.
> 
> - From a POSIX POV this means "somebody else" deleted the file,
>   so the dir is empty now.  Git tries to delete the directory again,
>   which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
>   and, internally, a sharing violation which disallows to move the
>   directory out of the way.
> 
> This looks suspiciously like a bug in the remote filesystem.  Link
> succeeded, so there are two links to the same file in the directory.
> Unlinking link 1 succeeds, so there's still one link to the 
> file in the
> directory, but link 2 is inaccessible as if the file has been deleted
> completely.  Thus, a full POSIX git on this drive is broken.
> 
> Can you please run
> 
>   /usr/lib/csih/getVolInfo /cygdrive/z
> 
> and paste the output here?  Maybe I can workaround this in the next
> Cygwin version.
> 
> 
> Corinna
> 

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00. 

RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[Jason Pyeron]

> -----Original Message-----
> From: Jason Pyeron
> Sent: Thursday, December 04, 2014 16:34
> To: git@vger.kernel.org
> Cc: 'brian m. carlson'
> Subject: RE: FW: [cygwin] Cygwin's git says "error: 
> failed to read delta-pack base object"
> 
> > -----Original Message-----
> > From: brian m. carlson
> > Sent: Wednesday, December 03, 2014 19:55
> > 
> > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > I remember hitting this a while ago, but just gave up.
> > > 
> > > It seems to be a problem for others too.
> > > 
> > > Any ideas on how to debug this so it can be patched?
> > > 
> > > -----Original Message-----
> > > From: Dave Lindbergh
> > > Sent: Wednesday, December 03, 2014 18:07
> > > To: cygwin
> > > 
> > > Aha - you're right.
> > > 
> > > It works fine on a local NTFS volume.
> > > 
> > > I get the error when I do it on Z:, which is mapped to a 
> > network drive
> > > (on another Windows box).
> > > 
> > > Is there a workaround? Why does this happen?
> > 
> > I don't think anyone is sure.  My wild guess is that 
> there's something
> > about the way that Cygwin wraps Windows calls that makes it 
> > fail in this
> > case.  It might be interesting to run the Windows and 
> Cygwin versions
> > under an strace equivalent and see where things differ.
> 
> [this was posted to the cygwin list]
> 
> http://nerdfever.com/files/strace.txt
> 
> > 
> > It's an interesting problem, but I don't have any Windows 
> > systems, so I
> > can't look into it.
> 
> If it becomes a little less magic, I will chomp on the 
> problem, but I cannot make a predictable test case.

Corrina at Cygwin devined the strace (see below) and I am working on a test cases and hacks.

Pseudo code and observations
./sha1_file.c:write_loose_object(sha1)
{
 filename=sha1_file_name(sha1)
 (fd,tmp_file)=create_tmpfile(filename)
 write_buffer(fd)
 close_sha1_file(fd)
 return move_temp_to_file(tmp_file, filename)
}

move_temp_to_file(tmpfile, filename)
{
 // I am thinking about forcing renames to see if the problem exists then as well
 // if that "works" then a per repo config option allowing for forced renames
 if (OBJECT_CREATION_USES_RENAMES) goto try_rename
 else if link(tmpfile,filename)

 if (failed except exist failures)
 {
  try_rename:
  rename(tmpfile, filename)
  if (ok) goto out
 }
 unlink_or_warn(tmpfile)
 if (failed except exist failures) return error
 out:
}

write_sha1_file(sha1)
{
 return write_loose_object(sha1)
}


> -----Original Message-----
> From: Corinna Vinschen
> Sent: Friday, December 05, 2014 6:35
> To: cygwin@cygwin.com
<snip/>
> What I found in the strace is this:
> 
> - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
> 
> - open file, write something, close file.
> 
> - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
> 	
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
>   succeeds.
> 
> - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
> 
> - Trying to open
>   
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
>   but the file doesn't exist and NtCreateFile fails with status
>   0xC0000034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
> 
> - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
>   STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
> 
> - git seems to be prepared for such a case, the parent process calls
>   opendir/readdir on the directory.  Enumerating the files in
>   Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
>   "0bdeb2fd209d24afb865584da10b78aa8fefc4".
> 
> - Then git calls lstat on the file, which results in NtOpenFile
>   returning status STATUS_OBJECT_NAME_NOT_FOUND again.
> 
> - From a POSIX POV this means "somebody else" deleted the file,
>   so the dir is empty now.  Git tries to delete the directory again,
>   which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
>   and, internally, a sharing violation which disallows to move the
>   directory out of the way.
> 
> This looks suspiciously like a bug in the remote filesystem.  Link
> succeeded, so there are two links to the same file in the directory.
> Unlinking link 1 succeeds, so there's still one link to the 
> file in the
> directory, but link 2 is inaccessible as if the file has been deleted
> completely.  Thus, a full POSIX git on this drive is broken.
> 
> Can you please run
> 
>   /usr/lib/csih/getVolInfo /cygdrive/z
> 
> and paste the output here?  Maybe I can workaround this in the next
> Cygwin version.
> 
> 
> Corinna
> 

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00. 

RE: FW: [cygwin] Cygwin's git says "error: failed to read delta-pack base object"

[Jason Pyeron]

TLDR = Cygwin remote filesystem sometimes has strange failures -> workaround is to use rename, not link/unlink; 
see https://github.com/pdinc-oss/git/commit/5a36824ed01d4335148ca3846e75cc99c11650e2
> -----Original Message-----
> From: Jason Pyeron
> Sent: Friday, December 05, 2014 10:30
> 
> > -----Original Message-----
> > From: Jason Pyeron
> > Sent: Thursday, December 04, 2014 16:34
> > 
> > > -----Original Message-----
> > > From: brian m. carlson
> > > Sent: Wednesday, December 03, 2014 19:55
> > > 
> > > On Wed, Dec 03, 2014 at 06:31:18PM -0500, Jason Pyeron wrote:
> > > > I remember hitting this a while ago, but just gave up.
> > > > 
> > > > It seems to be a problem for others too.
> > > > 
> > > > Any ideas on how to debug this so it can be patched?
> > > > 
> > > > -----Original Message-----
> > > > From: Dave Lindbergh
> > > > Sent: Wednesday, December 03, 2014 18:07
> > > > To: cygwin
> > > > 
> > > > Aha - you're right.
> > > > 
> > > > It works fine on a local NTFS volume.
> > > > 
> > > > I get the error when I do it on Z:, which is mapped to a 
> > > network drive
> > > > (on another Windows box).
> > > > 
> > > > Is there a workaround? Why does this happen?

I have a really hacky workaround, commit 5a36824ed01d4335148ca3846e75cc99c11650e2 comments out the logic and forces a rename instead of link and unlink.

https://github.com/pdinc-oss/git/tree/cygwin-issue-remoteCIFS-rename

<snip/>

> Pseudo code and observations
> ./sha1_file.c:write_loose_object(sha1)
> {
>  filename=sha1_file_name(sha1)
>  (fd,tmp_file)=create_tmpfile(filename)
>  write_buffer(fd)
>  close_sha1_file(fd)
>  return move_temp_to_file(tmp_file, filename)
> }
> 
> move_temp_to_file(tmpfile, filename)
> {
>  // I am thinking about forcing renames to see if the problem 
> exists then as well
>  // if that "works" then a per repo config option allowing 
> for forced renames
>  if (OBJECT_CREATION_USES_RENAMES) goto try_rename
>  else if link(tmpfile,filename)
> 

Dave has tested a build I made for him on 64 bit Cygwin and it works. I no longer have access to the environment I was having this problem in last February. I will try to investigate this further, but I am not hopeful, maybe Corinna will have luck on the issue. But I was in a secure corporate environment and I thought the host based security system (AV), coupled with the remote file system was causing the problem, namely files created are not available instantly.

I do think that we should have a config option for this, as most users who could encounter such a problem are not likely to be able (or allowed) to rebuild the git executable themselves.

<snip/>

> > -----Original Message-----
> > From: Corinna Vinschen
> > Sent: Friday, December 05, 2014 6:35
> > To: cygwin@cygwin.com
> <snip/>
> > What I found in the strace is this:
> > 
> > - Create file Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ
> > 
> > - open file, write something, close file.
> > 
> > - link (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ,
> > 	
> > 
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4)
> >   succeeds.
> > 
> > - unlink (Z:\pic32mx-bmf\.git\objects\30\tmp_obj_YljwNZ) succeeds
> > 
> > - Trying to open
> >   
> > 
> Z:\pic32mx-bmf\.git\objects\30\0bdeb2fd209d24afb865584da10b78aa8fefc4
> >   but the file doesn't exist and NtCreateFile fails with status
> >   0xC0000034, STATUS_OBJECT_NAME_NOT_FOUND --> ENOENT.
> > 
> > - Subsequent unlink (Z:\pic32mx-bmf\.git\objects\30) fails with a
> >   STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY.
> > 
> > - git seems to be prepared for such a case, the parent process calls
> >   opendir/readdir on the directory.  Enumerating the files in
> >   Z:\pic32mx-bmf\.git\objects\30 shows the entries ".", ".." and
> >   "0bdeb2fd209d24afb865584da10b78aa8fefc4".
> > 
> > - Then git calls lstat on the file, which results in NtOpenFile
> >   returning status STATUS_OBJECT_NAME_NOT_FOUND again.
> > 
> > - From a POSIX POV this means "somebody else" deleted the file,
> >   so the dir is empty now.  Git tries to delete the directory again,
> >   which again results in STATUS_DIRECTORY_NOT_EMPTY --> ENOTEMPTY
> >   and, internally, a sharing violation which disallows to move the
> >   directory out of the way.
> > 
> > This looks suspiciously like a bug in the remote filesystem.  Link
> > succeeded, so there are two links to the same file in the directory.
> > Unlinking link 1 succeeds, so there's still one link to the 
> > file in the
> > directory, but link 2 is inaccessible as if the file has 
> been deleted
> > completely.  Thus, a full POSIX git on this drive is broken.
> > 
> > Can you please run
> > 
> >   /usr/lib/csih/getVolInfo /cygdrive/z
> > 
> > and paste the output here?  Maybe I can workaround this in the next
> > Cygwin version.

Dave's response: https://www.cygwin.com/ml/cygwin/2014-12/msg00066.html

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.