Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

[Dr.-Ing. Christoph Cullmann]

Hi,

on Windows 64-bit, for a repository having a .pack file > 4GB I get during cloning:

$ git clone file:///repositories/test.git test
Cloning into 'test'...
remote: Counting objects: 210294, done.
remote: error: bad object header
remote: error: bad object header
remote: fatal: packed object cae50116ebe36de8bded4811bd262d90670bde2f (stored in ./objects/pack/pack-30ba8b19b029cc6554853ae07729aa807995ebb8.pack) is corrupt
error: git upload-pack: git-pack-objects died with error.
fatal: git upload-pack: aborting due to possible repository remote: aborting due to possible repository corruption on the remote side.
corruption on the remote side.
fatal: early EOF
fatal: index-pack failed

We use git 2.14.1, the 64-bit version.

On systems which have (long int) == 64-bit datatype, all works well, e.g. Linux 64-bit.
(same goes for fscks/...., they work on this git on Linux 64-bit, not on Windows 64-bit)

It seems the Windows port did take care of using the right 64-bit offsets and all
in the OS specific parts, but things like pack-objects.c in the generic parts use "unsigned long"
like it behaves like size_t (at least on the first glance).

As I think the issue is in the generic part, I report this here, not on the Windows list,
thought it seems to be an IL32P64 related issue.

Greetings
Christoph

-- 
----------------------------- Dr.-Ing. Christoph Cullmann ---------
AbsInt Angewandte Informatik GmbH      Email: cullmann@AbsInt.com
Science Park 1                         Tel:   +49-681-38360-22
66123 Saarbrücken                      Fax:   +49-681-38360-20
GERMANY                                WWW:   http://www.AbsInt.com
--------------------------------------------------------------------
Geschäftsführung: Dr.-Ing. Christian Ferdinand
Eingetragen im Handelsregister des Amtsgerichts Saarbrücken, HRB 11234

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

[Johannes Schindelin]

Hi Christoph,

On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:

> on Windows 64-bit, for a repository having a .pack file > 4GB I get
> during cloning:

The reason is Git's source code that over-uses the `unsigned long`
datatype.

In a nearby-thread, an underappreciated effort by Martin Koegler is
underway to get the ball rolling in getting it fixed. Maybe you can help
making Martin a lot more welcome on the Git mailing list, and maybe even
help getting his patches reviewed and integrated?

Ciao,
Johannes

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

[Stefan Beller]

On Mon, Aug 14, 2017 at 9:05 AM, Johannes Schindelin
<Johannes.Schindelin@gmx.de> wrote:
> Hi Christoph,
>
> On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:
>
>> on Windows 64-bit, for a repository having a .pack file > 4GB I get
>> during cloning:
>
> The reason is Git's source code that over-uses the `unsigned long`
> datatype.
>
> In a nearby-thread, an underappreciated effort by Martin Koegler is
> underway to get the ball rolling in getting it fixed. Maybe you can help
> making Martin a lot more welcome on the Git mailing list, and maybe even
> help getting his patches reviewed and integrated?

'nearby' as in [1] ;-)

[1] https://public-inbox.org/git/1502527643-21944-1-git-send-email-martin@mail.zuhause/

I had the impression the review is going well there?

>
> Ciao,
> Johannes

Re: Bug with corruption on clone/fsck/... with large packs + 64-bit Windows, problem with usage of "long" datatype for sizes/offsets?

[Junio C Hamano]

Stefan Beller <sbeller@google.com> writes:

> On Mon, Aug 14, 2017 at 9:05 AM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>> Hi Christoph,
>>
>> On Fri, 11 Aug 2017, Dr.-Ing. Christoph Cullmann wrote:
>>
>>> on Windows 64-bit, for a repository having a .pack file > 4GB I get
>>> during cloning:
>>
>> The reason is Git's source code that over-uses the `unsigned long`
>> datatype.
>>
>> In a nearby-thread, an underappreciated effort by Martin Koegler is
>> underway to get the ball rolling in getting it fixed. Maybe you can help
>> making Martin a lot more welcome on the Git mailing list, and maybe even
>> help getting his patches reviewed and integrated?
>
> 'nearby' as in [1] ;-)
>
> [1] https://public-inbox.org/git/1502527643-21944-1-git-send-email-martin@mail.zuhause/
>
> I had the impression the review is going well there?

I do not know if it is "going well", but I do not agree with the
"underappreciated" bit at all.  I find such a blanket statement
toxic and detrimental to the community.

It is true that many topics in flight are broken with a tree-wide
change that is presented as a single ball of wax.  Unlike the
ongoing "struct object_id" effort, which also is tree-wide but tries
to find a step-wise refinement to reduce its impact on other topics,
it is harder to integrate such a change.  But that does not have any
relation to how much the effort is appreciated.