Can git be tweaked to work cross-platform, on FAT32?

Can git be tweaked to work cross-platform, on FAT32?

[Florian v. Savigny]

Hi there,

I would like to use git to develop a somewhat larger Perl project
which I would need to test and develop under both Linux and Windows,
i.e. I would need a repository on a FAT32 partition (to my knowledge,
still the only filesystem that can be fully accessed from both OSes -
correct me if I'm wrong) which could be accessed by two compatible git
executables, such that I could switch back and forth.

Before describing how I (unsuccessfully, which is perhaps not
surprising, as I'm C-illiterate) hacked the Makefile to compile two
git siblings, I would like to generally ask whether this is possible
at all (or should be possible in theory, in which case I'm willing to
test extensively), whether with Cygwin or not. The preliminary
impression was that the Linux version would not accept a repository on
FAT32 (it did work with more Unixish filesystems). - I do not need the
networking capabilities, BTW (standalone development), if that is an
issue in any way.

Thanks for any hint!

Florian

-- 
                                               Florian von Savigny
__________________________________________________________________
lorian@fsavigny.de

Re: Can git be tweaked to work cross-platform, on FAT32?

[Johannes Schindelin]

Hi,

On Wed, 13 Dec 2006, Florian v. Savigny wrote:

> I would like to use git to develop a somewhat larger Perl project
> which I would need to test and develop under both Linux and Windows,
> i.e. I would need a repository on a FAT32 partition (to my knowledge,
> still the only filesystem that can be fully accessed from both OSes -
> correct me if I'm wrong) which could be accessed by two compatible git
> executables, such that I could switch back and forth.

I would not use the _same_ repository from both Windows and Linux, but 
rather fetch/push from/to the Windows partition.

I am using Cygwin to compile git, and I have a USB Stick with FAT32 on 
which I push regularly. Important: since Linux is more intelligent about 
permissions, I had to initialize the repos with Windows.

To get started compiling, you might want to use the autoconf script.

Ciao,
Dscho

Re: Can git be tweaked to work cross-platform, on FAT32?

[Martin Langhoff]

Florian,

a bit of a long shot, but you might be able to access it via Samba,
and just use git on linux. I have to confess to having setup a few git
checkouts for Win32 users this way... in some cases with cygwin+x11
and a shortcut that will do something like

    ssh linuxhost gitk --all

or

    ssh linuxhost qgit

but then again, I'm rumoured to be insane...

cheers,

Re: Can git be tweaked to work cross-platform, on FAT32?

[Florian v. Savigny]

Thank you Martin and Johannes -

before proceeding by following either of your two suggestions, I
would like to make sure that there is no other way, so I would like to
ask:

1. I know, there is absolutely nothing in either of your answers that
   says that the route I conceived is not possible, but anyway: DO
   your answers imply that it is NOT possible (by "possible", I mean
   of course: possible without re-programming the source code ;-)) to
   compile git in a way that would enable it to work DIRECTLY with a
   local repository on FAT32 from both Windows and Linux (or, perhaps
   more simple: to work directly with a repository on FAT32 under
   Linux)?

   I'm repeating my question because it is impossible for me to tell
   whether this would involve a rather simple change (such as: don't
   use symlinks, and/or: don't care about user/group permissions), or
   whether the critical differences are rather an inherent
   characteristic of git, or perhaps even the libraries it uses.

   And: if this were possible, I would consider it a rather attractive
   idea, not only for me, hence I'd like to rule it out before
   dropping it altogether.


2. ad Martin: Ouf! I have no idea of how Samba works (well, yes: I
   thought it was some kind of server or client program permitting to
   connect to Windows servers or clients), hence I don't get a very
   clear idea of what you have in mind. I hope I have made it clear
   that I am using only one computer (double-boot, simply), and that
   everything is local. I am aware that Linux permits to do "virtual"
   networking with localhost, but how would your scheme work on a
   single machine? Can samba "pretend" there is a Windows network
   behind it, while in truth, it accesses a local directory (on
   FAT32)? And can git work directly in a repository it considers to
   be accessible over the network only?

   The interesting thing in this could be that one could use indeed
   the same repository under both OSes. If I got vaguely the right
   idea, that is.

3. ad Johannes: This does sound quite simple and straightforward. If I
   got it right, it would involve having one repository on a, say,
   ext2 partition to work with under Linux, and one on a FAT32
   partition to work with under Windows, and syncing the two after
   booting (fetching from FAT32) and before shutting down (pushing to
   FAT32) Linux.

   It is quite interesting, BTW, that git can /sync/ with a repository
   on FAT32 under Linux, but not work with it.


Thanks very much for both answers! I would still be curious to know
whether there is an answer to my first question, but yours should get
me working one way or other.

Regards, Florian

Re: Can git be tweaked to work cross-platform, on FAT32?

[Johannes Schindelin]

Hi,

On Sat, 16 Dec 2006, Florian v. Savigny wrote:

> DO your answers imply that it is NOT possible (by "possible", I mean of 
> course: possible without re-programming the source code ;-)) to compile 
> git in a way that would enable it to work DIRECTLY with a local 
> repository on FAT32 from both Windows and Linux (or, perhaps more 
> simple: to work directly with a repository on FAT32 under Linux)?

My answer was not so clear...

Question: Is it possible to use git on a FAT32 filesystem?

Answer: Yes.

Question: Is it possible to access such a repository from both Linux and 
Windows?

Answer: Yes.

Question: Is there anything that could cause problems?

Answer: Yes. The working directory is a problem.

	- Symlinks for example. Don't use them.
	- Umlauts: don't use them.
	- The index has to be updated, since the stat information is 
	  unlikely to match. A simple "git status" should suffice, though.
	- Permissions: the permission handling of Linux and Windows are
	  quite different. If you initialize the repository with Cygwin,
	  it automatically sets core.filemode=false, if you initalize(d) 
	  with Linux, you have to adjust manually.

Question: Are there other shortcomings to this approach?

Answer: Definitely. Most are only annoying (like the performance of FAT32, 
which literally sucks), but some are different: for example, if you 
compile things in the working directory, you will be unhappy.

> 3. ad Johannes: This does sound quite simple and straightforward. If I
>    got it right, it would involve having one repository on a, say,
>    ext2 partition to work with under Linux, and one on a FAT32
>    partition to work with under Windows, and syncing the two after
>    booting (fetching from FAT32) and before shutting down (pushing to
>    FAT32) Linux.

This is how I'd do it.

>    It is quite interesting, BTW, that git can /sync/ with a repository
>    on FAT32 under Linux, but not work with it.

You can. But I wouldn't. I like to keep separations clean and obvious.

Ciao,

Re: Can git be tweaked to work cross-platform, on FAT32?

[Martin Langhoff]

On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > 3. ad Johannes: This does sound quite simple and straightforward. If I
> >    got it right, it would involve having one repository on a, say,
> >    ext2 partition to work with under Linux, and one on a FAT32
> >    partition to work with under Windows, and syncing the two after
> >    booting (fetching from FAT32) and before shutting down (pushing to
> >    FAT32) Linux.
>
> This is how I'd do it.

And I concur - I had only introduced Samba to the conversation because
I thought you were talking about several computers. If you are
dual-booting on one machine, I'd do as above.

Note that under windows you can use ext2 -- haven't used it, and don't
know how cygwin behaves with it, but it may be *just* what you need to
avoid case sensitivity problems and have symlink support.

    http://sourceforge.net/projects/ext2fsd

    http://www.fs-driver.org/

cheers,

Re: Can git be tweaked to work cross-platform, on FAT32?

[Stefano Spinucci]

I just tried to use git writing to my FAT32 formatted usb stick.

On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT,
but after the
following actions I always got the error "fatal: Unable to write new
index file" or
"fatal: unable to create '.git/index': File exists":

mkdir git-test-repo
cd git-test-repo
git-init-db
echo iii>test
git add test
git commit -m "test msg"
echo ooo>test1
git add test1 ***error***

converting the stick to NTFS I have no problems.

any hint ???

---

Re: Can git be tweaked to work cross-platform, on FAT32?

[Johannes Schindelin]

Hi,

On Sun, 17 Dec 2006, Stefano Spinucci wrote:

> I just tried to use git writing to my FAT32 formatted usb stick.
> 
> On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT, but 
> after the following actions I always got the error "fatal: Unable to 
> write new index file" or "fatal: unable to create '.git/index': File 
> exists":

Come to think of it, I probably never tried to actually _commit_ on 
FAT32... Sorry.

I will not be able to test this scenario until Tuesday, though. Sorry 
again!

Ciao,

Re: Can git be tweaked to work cross-platform, on FAT32?

[Juergen Ruehle]

Stefano Spinucci writes:
 > I just tried to use git writing to my FAT32 formatted usb stick.
 > 
 > On windows XP, I compiled git with and without NO_D_TYPE_IN_DIRENT,
 > but after the
 > following actions I always got the error "fatal: Unable to write new
 > index file" or
 > "fatal: unable to create '.git/index': File exists":

Compile git with NO_MMAP. The problem is that the old index file is
mmaped while it is replaced. Cygwin supports this operation only on
NTFS. I don't know whether this is a bug in cygwin or a windows
limitation (I suspect the latter).

Unfortunately NO_MMAP makes git pretty slow if the pack files get
larger than about 10-20 MB.

Johannes claims that there are additional problems with mmap on
cygwin, but it passes the complete test suite on NTFS, so it should be
ok for most operations

IIRC it has been suggested on the list that mmaping the index isn't
that important for git performance and it could be replaced by simply
reading the index file into memory on access.

It would be nice to have mmap work on non-NTFS partitions since my
tests suggest that FAT32 is up to twice as fast (in the hot cache
case). (My test version simply uses an unlink before the rename in
lockfile.c to hack around the cygwin rename problem.)

Re: Can git be tweaked to work cross-platform, on FAT32?

[Johannes Schindelin]

Hi,

On Sun, 17 Dec 2006, Juergen Ruehle wrote:

> Johannes claims that there are additional problems with mmap on cygwin, 
> but it passes the complete test suite on NTFS, so it should be ok for 
> most operations

I encountered a problem with git-log, where the fork() tried to 
reestablish a mmap()ed file, which unfortunately was renamed (since it was 
a .lock file[*1*]). This triggered the implementation of NO_MMAP.

I am sure that there are more problems with it. BTW the problem stems from 
Windows having _no_ equivalent to fork().

Notefoot 1: Isn't it funny that Windows cannot delete files which are 
still opened, but does not care when it is renamed?

Ciao,

Re: Can git be tweaked to work cross-platform, on FAT32?

[Stefano Spinucci]

On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> ...
> I am sure that there are more problems with it. BTW the problem stems from
> Windows having _no_ equivalent to fork().
> ...

I recompiled git with NO_MMAP and used it successfully, then I added a
note in the wiki
page http://git.or.cz/gitwiki/WindowsInstall on how to compile git to
work on a FAT32
filesystem.

Now, two questions...

if you are sure that there are more problems with using MMAP, shouldn't NO_MMAP
be commented better in the Makefile and in INSTALL, pointing out to
use it always on
cygwin ???

finally on cygwin there is a precompiled git package (v 1.4.4.1);
however, I found if it was
compiled *with* MMAP support... should I write to the cygwin-apps list
asking for a
different compilation (with NO_MMAP), or for the majority of the users
the standard
compilation is a better choice ???

bye

---

Re: Can git be tweaked to work cross-platform, on FAT32?

[Johannes Schindelin]

Hi,

On Mon, 18 Dec 2006, Stefano Spinucci wrote:

> On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
> > ...
> > I am sure that there are more problems with it. BTW the problem stems from
> > Windows having _no_ equivalent to fork().
> > ...
> 
> if you are sure that there are more problems with using MMAP, shouldn't 
> NO_MMAP be commented better in the Makefile and in INSTALL, pointing out 
> to use it always on cygwin ???

I argued for that a long time ago, but there were a couple "works-for-me"s 
and I just shut up.

It could also be the case that for most operations, git became safe over 
time. For example, my biggest problem was the fork()ed diff in git-log, 
which was substituted by a non-forking inbuilt diff.

> should I write to the cygwin-apps list asking for a different 
> compilation (with NO_MMAP), or for the majority of the users the 
> standard compilation is a better choice ???

Given that most Windows setups nowadays run on NTFS, I doubt that this is 
necessary. I use NO_MMAP myself on Windows, just in case, though.

Ciao,
Dscho

Re: Can git be tweaked to work cross-platform, on FAT32?

[Juergen Ruehle]

Johannes Schindelin writes:
 > On Mon, 18 Dec 2006, Stefano Spinucci wrote:
 > > On 12/17/06, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:
 > > > I am sure that there are more problems with it. BTW the problem stems from
 > > > Windows having _no_ equivalent to fork().
 > > 
 > > if you are sure that there are more problems with using MMAP, shouldn't 
 > > NO_MMAP be commented better in the Makefile and in INSTALL, pointing out 
 > > to use it always on cygwin ???
 > 
 > I argued for that a long time ago, but there were a couple "works-for-me"s 
 > and I just shut up.

It would be nice to have test cases documenting known failures. Then
people could decide whether these cases are important for their work,
since I don't think it's actually possible to loose data this way.

Personally I've been unsuccessful in trying to recreate failures, but
that is probably just an indication of my inability to use git
correctly:-)

Re: Can git be tweaked to work cross-platform, on FAT32?

[Dan Nicholson]

On 12/17/06, Martin Langhoff <martin.langhoff@gmail.com> wrote:
>
> Note that under windows you can use ext2 -- haven't used it, and don't
> know how cygwin behaves with it, but it may be *just* what you need to
> avoid case sensitivity problems and have symlink support.
>
>     http://sourceforge.net/projects/ext2fsd
>
>     http://www.fs-driver.org/

I've used both of these guys, but not the most recent versions. Both
were previously read-only, but now it appears both have write support
from looking at the release notes. Both were a bit heavy handed to get
setup, but a decently computer literate person could probably handle
it. Ext2IFS in particular looks dramatically improved from when I last
used it.

Both worked fine through Cygwin for simple usage. I never did anything
with serious disk usage on either of them, so I can't say how it
stacks up to FAT32 on Linux in terms of really hammering the disk.

A little more reading says ext2ifs can't handle permissions or symbolic links.

http://www.fs-driver.org/faq.html

I don't recall the details of ext2fsd, but I think symbolic links
worked. I'd like to look at this again, but I spend all my time in
Linux at home.

--

Re: Can git be tweaked to work cross-platform, on FAT32?

[David Lang]

On Wed, 13 Dec 2006, Florian v. Savigny wrote:

> I would like to use git to develop a somewhat larger Perl project
> which I would need to test and develop under both Linux and Windows,
> i.e. I would need a repository on a FAT32 partition (to my knowledge,
> still the only filesystem that can be fully accessed from both OSes -
> correct me if I'm wrong)

there are drivers available for windows to let it access ext2/3 partitions. I 
don't know if that would help any in this situation.

David Lang