fast-import problem importing dos format files under cygwin

fast-import problem importing dos format files under cygwin

[Jan Hudec]

[-- Attachment #1: Type: text/plain, Size: 1304 bytes --]

Hello folks,

I have been playing with fast-import in cygwin and I have problems
importing files with CR/LF line-endings. The size in data command is
calculated including the CRs and than the file is copied binary to the
fast-import input stream. However fast-import skips the CRs when reading,
overreads by that number of bytes and fails when it tries to read the next
command from the middle.

Attached is a test input stream and crash report generated by fast-import
when reading it. In case mail system damages it along the way despite
being attached to prevent that, the file should be in unix format -- that
is what my cygwin perl outputs by default -- and has CRs only on lines 15
and 16. The unix.txt and dos.txt should only differ that the '.'s in
former are replaced by '^M's in the later (so the data commands are
otherwise same).

Note, that when I convert the file to dos format, it is read as intended.
However, that is inconsistent with rest of the cygwin environment which
generated and expects files in unix format. I use binary mounts (not
converting) and CYGWIN environment variable is empty. My git version is
1.6.0.4 from official Cygwin package.

Is this behaviour intentional workaround for something or a bug?

-- 
                                        - Jan Hudec <bulb@ucw.cz>

[-- Attachment #2: test1.gfi --]
[-- Type: /, Size: 495 bytes --]

commit refs/heads/master
committer Jan Hudec <bulb@ucw.cz> 1228287890 +0100
data 19
unix-formatted-file
M 100644 inline unix.txt
data 13
some.
lines.
commit refs/heads/master
committer Jan Hudec <bulb@ucw.cz> 1228287892 +0100
data 18
dos-formatted-file
M 100644 inline dos.txt
data 13
some
lines
commit refs/heads/master
committer Jan Hudec <bulb@ucw.cz> 1228287894 +0100
data 23
one more commit for fun
M 100644 inline other.txt
data 22
we need more commands
checkpoint

[-- Attachment #3: fast_import_crash_5212 --]
[-- Type: /, Size: 1034 bytes --]

fast-import crash report:
    fast-import process: 5212
    parent process     : 1696
    at Wed Dec 3 08:29:42 2008

fatal: Unsupported command: mmit refs/heads/master

Most Recent Commands Before Crash
---------------------------------
  commit refs/heads/master
  committer Jan Hudec <bulb@ucw.cz> 1228287890 +0100
  data 19
  M 100644 inline unix.txt
  data 13
  commit refs/heads/master
  committer Jan Hudec <bulb@ucw.cz> 1228287892 +0100
  data 18
  M 100644 inline dos.txt
  data 13
* mmit refs/heads/master

Active Branch LRU
-----------------
    active_branches = 1 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   1)      2 refs/heads/master

Inactive Branches
-----------------
refs/heads/master:
  status      : active loaded
  tip commit  : 2a11b753d155443462523e0f3cef72e4f02817b2
  old tree    : 5bcdcdcafba5303797865311a7fa06e2ce3ddc65
  cur tree    : 5bcdcdcafba5303797865311a7fa06e2ce3ddc65
  commit clock: 2
  last pack   : 0


Marks
-----

-------------------
END OF CRASH REPORT

Ad: fast-import problem importing dos format files under cygwin

[Jan Hudec]

On 3 December 2008, 08:51, Jan Hudec wrote:
> Hello folks,
>
> I have been playing with fast-import in cygwin and I have problems
> importing files with CR/LF line-endings. The size in data command is
> calculated including the CRs and than the file is copied binary to the
> fast-import input stream. However fast-import skips the CRs when reading,
> overreads by that number of bytes and fails when it tries to read the
> next command from the middle.

One addition:

I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the
test, as it was, except it didn't like 'refs/heads/master' as branchname
(and accepted bare 'master', but that created '.git/master').

-- 
                                        - Jan Hudec <bulb@ucw.cz>

Re: Ad: fast-import problem importing dos format files under cygwin

[Johannes Sixt]

Jan Hudec schrieb:
> On 3 December 2008, 08:51, Jan Hudec wrote:
>> Hello folks,
>>
>> I have been playing with fast-import in cygwin and I have problems
>> importing files with CR/LF line-endings. The size in data command is
>> calculated including the CRs and than the file is copied binary to the
>> fast-import input stream. However fast-import skips the CRs when reading,
>> overreads by that number of bytes and fails when it tries to read the
>> next command from the middle.
> 
> One addition:
> 
> I have tried with MSYS version 1.5.6.1.1071.g76fb and it imported the
> test, as it was, except it didn't like 'refs/heads/master' as branchname
> (and accepted bare 'master', but that created '.git/master').

With my current version of MinGW git the import is successful after I
edited test1.gfi to match your description (it had CR on all lines; I
removed all except on lines 15 and 16). The repository content is as one
would it expect given the input. master is
b8ad21c3dc271d43a6e43c261909d6be725fa5b8.

Do you happen to have core.autocrlf set in some way and could it make a
difference for fast-import? I have it unset.

-- Hannes

Re: Ad: fast-import problem importing dos format files under cygwin

[Jan Hudec]

Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> Jan Hudec schrieb:
>> On 3 December 2008, 08:51, Jan Hudec wrote:
>>> Hello folks,
>>>
>>> I have been playing with fast-import in cygwin and I have problems
>>> importing files with CR/LF line-endings. The size in data command is
>>> calculated including the CRs and than the file is copied binary to the
>>> fast-import input stream. However fast-import skips the CRs when
>>> reading,
>>> overreads by that number of bytes and fails when it tries to read the
>>> next command from the middle.
>
> Do you happen to have core.autocrlf set in some way and could it make a
> difference for fast-import? I have it unset.

I have it set to false explicitly in global config. Tried with not having
it set at all and gives the same problem. Since the previous version of
MSys Git worked for me, I suspect it's somehow cygwin-related.

-- 
                                        - Jan Hudec <bulb@ucw.cz>

Re: Ad: fast-import problem importing dos format files under cygwin

[Shawn O. Pearce]

Jan Hudec <bulb@ucw.cz> wrote:
> Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > Jan Hudec schrieb:
> >> On 3 December 2008, 08:51, Jan Hudec wrote:
> >>> Hello folks,
> >>>
> >>> I have been playing with fast-import in cygwin and I have problems
> >>> importing files with CR/LF line-endings. The size in data command is
> >>> calculated including the CRs and than the file is copied binary to the
> >>> fast-import input stream. However fast-import skips the CRs when
> >>> reading,
> >>> overreads by that number of bytes and fails when it tries to read the
> >>> next command from the middle.
> >
> > Do you happen to have core.autocrlf set in some way and could it make a
> > difference for fast-import? I have it unset.
> 
> I have it set to false explicitly in global config. Tried with not having
> it set at all and gives the same problem. Since the previous version of
> MSys Git worked for me, I suspect it's somehow cygwin-related.

Huh.  So fast-import *never* does auto-CRLF conversion, even if the
property is set.  It just doesn't make those calls internally.
It blindly copies data from the input stream into the pack.
No exceptions.

fast-import under-reading near CRs and getting misaligned on its
input indicates that the stdio library has given us a FILE* for stdin
which is converting CRLF pairs into LFs, even within an fread() call.

My guess here is fast-import's stdin is set in text mode, but it
really needs to be in binary mode.  fast-import.c never attempts
to correct that when it starts, so on DOS based systems we are
probably totally screwed from the beginning...

-- 
Shawn.

Re: Ad: fast-import problem importing dos format files under cygwin

[Johannes Schindelin]

Hi,

On Wed, 3 Dec 2008, Shawn O. Pearce wrote:

> Jan Hudec <bulb@ucw.cz> wrote:
> > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > > Jan Hudec schrieb:
> > >> On 3 December 2008, 08:51, Jan Hudec wrote:
> > >>>
> > >>> I have been playing with fast-import in cygwin and I have problems 
> > >>> importing files with CR/LF line-endings. The size in data command 
> > >>> is calculated including the CRs and than the file is copied binary 
> > >>> to the fast-import input stream. However fast-import skips the CRs 
> > >>> when reading, overreads by that number of bytes and fails when it 
> > >>> tries to read the next command from the middle.
> > >
> > > Do you happen to have core.autocrlf set in some way and could it 
> > > make a difference for fast-import? I have it unset.
> > 
> > I have it set to false explicitly in global config. Tried with not 
> > having it set at all and gives the same problem. Since the previous 
> > version of MSys Git worked for me, I suspect it's somehow 
> > cygwin-related.
> 
> Huh.  So fast-import *never* does auto-CRLF conversion, even if the 
> property is set.  It just doesn't make those calls internally. It 
> blindly copies data from the input stream into the pack. No exceptions.
> 
> fast-import under-reading near CRs and getting misaligned on its input 
> indicates that the stdio library has given us a FILE* for stdin which is 
> converting CRLF pairs into LFs, even within an fread() call.
> 
> My guess here is fast-import's stdin is set in text mode, but it really 
> needs to be in binary mode.  fast-import.c never attempts to correct 
> that when it starts, so on DOS based systems we are probably totally 
> screwed from the beginning...

I think you need to set the environment variable

	CYGWIN=binmode

Hth,
Dscho

Re: Ad: fast-import problem importing dos format files under cygwin

[Jan Hudec]

Hi,

On Wed, Dec 03, 2008 at 17:20:54 +0100, Johannes Schindelin wrote:
> Hi,
> On Wed, 3 Dec 2008, Shawn O. Pearce wrote:
> > Jan Hudec <bulb@ucw.cz> wrote:
> > > Dne 3 Prosinec 2008, 13:18, Johannes Sixt napsal(a):
> > > > Jan Hudec schrieb:
> > > >> On 3 December 2008, 08:51, Jan Hudec wrote:
> > > >>>
> > > >>> I have been playing with fast-import in cygwin and I have problems 
> > > >>> importing files with CR/LF line-endings. The size in data command 
> > > >>> is calculated including the CRs and than the file is copied binary 
> > > >>> to the fast-import input stream. However fast-import skips the CRs 
> > > >>> when reading, overreads by that number of bytes and fails when it 
> > > >>> tries to read the next command from the middle.
> >
> > [...] 
> > fast-import under-reading near CRs and getting misaligned on its input 
> > indicates that the stdio library has given us a FILE* for stdin which is 
> > converting CRLF pairs into LFs, even within an fread() call.
> > 
> > My guess here is fast-import's stdin is set in text mode, but it really 
> > needs to be in binary mode.  fast-import.c never attempts to correct 
> > that when it starts, so on DOS based systems we are probably totally 
> > screwed from the beginning...

Yes, it does indeed sound so. Strange thing is why it would be that way, when
it does not seem to be the case for any other process (eg. the shell will
complain loudly if I feed it a DOS formatted script). The standard input is
simple shell redirect from a file on a binary-mounted filesystem. I'll do
some more cross-checks tomorrow.

> I think you need to set the environment variable
> 
> 	CYGWIN=binmode

Will try. Thanks.

-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>