OSX, ZFS, UTF8, git - somebody hates me in this list

OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

repeat by:
    0) download the OSX ZFS drivers
    1) Make a ZFS partition on OSX.
    2) Clone git.git to the ZFS partition
    3) immediately type git-status:

    # On branch master
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #       "gitweb/test/Ma\314\210rchen"
    nothing added to commit but untracked files present (use "git add" to track)

    4) try "git-clean -f"
    Removing "gitweb/test/Ma\314\210rchen"

    5) try git-status now:
    # On branch master
    # Changed but not updated:
    #   (use "git add/rm <file>..." to update what will be committed)
    #
    #       deleted:    "gitweb/test/M\303\244rchen"
    #
    no changes added to commit (use "git add" and/or "git commit -a")

    6) still broken.  try git-reset --hard HEAD
    HEAD is now at c697ad1 Cleanup xread() loops to use read_in_full()
    7) and now git-status again:

    # On branch master
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #       "gitweb/test/Ma\314\210rchen"
    nothing added to commit but untracked files present (use "git add" to track)

That's broke, and a show-stopper for me to use ZFS on OSX.  Ugh.

Can someone with ZFS on solaris try this, to try to figure out if this
is a problem with git, ZFS, OSX, or some combination?

Thanks.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Shawn O. Pearce]

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:
> 
> repeat by:
>     0) download the OSX ZFS drivers
>     1) Make a ZFS partition on OSX.
>     2) Clone git.git to the ZFS partition
>     3) immediately type git-status:
> 
>     # On branch master
>     # Untracked files:
>     #   (use "git add <file>..." to include in what will be committed)
>     #
>     #       "gitweb/test/Ma\314\210rchen"
>     nothing added to commit but untracked files present (use "git add" to track)

Isn't this just the normal HFS+ name mangling?

Apparently the ZFS port to Mac OS X includes the same braindamaged
UTF normalization that is applied to HFS+ volumes, which means
they have reused the code or pushed it higher in their VFS layer.
Linus has started to add support for teaching Git about this sort
of brain damaged filesystem, but nobody has implemented the Mac OS
X specific name hashing function required to make Git see that the
name it got from the readdir(3) call is the same name Git passed
to the OS, just encoded differently.

-- 
Shawn.

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

>>>>> "Shawn" == Shawn O Pearce <spearce@spearce.org> writes:

Shawn> "Randal L. Schwartz" <merlyn@stonehenge.com> wrote:
>> 
>> repeat by:
>> 0) download the OSX ZFS drivers
>> 1) Make a ZFS partition on OSX.
>> 2) Clone git.git to the ZFS partition
>> 3) immediately type git-status:
>> 
>> # On branch master
>> # Untracked files:
>> #   (use "git add <file>..." to include in what will be committed)
>> #
>> #       "gitweb/test/Ma\314\210rchen"
>> nothing added to commit but untracked files present (use "git add" to track)

Shawn> Isn't this just the normal HFS+ name mangling?

I don't recall having this problem on HFS+.

But, oh, it's because my git.git is on a UFS partition.  So UFS works
fine.  Why shouldn't ZFS (which will be closer to UFS than HFS+)
be the same?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

>>>>> "Randal" == Randal L Schwartz <merlyn@stonehenge.com> writes:

>>>>> "Shawn" == Shawn O Pearce <spearce@spearce.org> writes:

Shawn> Isn't this just the normal HFS+ name mangling?

Randal> I don't recall having this problem on HFS+.

Ahh, oddly enough, it does exactly the same on HFS+, but *not* UFS, which is
why I hadn't noticed it yet, since all of my things I mirror are on my UFS
partition, just in case they have case-colliding names.

I'm trying to get rid of my UFS partition because it's a pain in the rear,
so I was hoping to upgrade to ZFS.  Hence, the query.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Shawn O. Pearce]

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:
> >> 
> >> # On branch master
> >> # Untracked files:
> >> #   (use "git add <file>..." to include in what will be committed)
> >> #
> >> #       "gitweb/test/Ma\314\210rchen"
> >> nothing added to commit but untracked files present (use "git add" to track)
> 
> Shawn> Isn't this just the normal HFS+ name mangling?
> 
> I don't recall having this problem on HFS+.

Everyone has this problem on HFS+.  :-|
 
> But, oh, it's because my git.git is on a UFS partition.  So UFS works
> fine.  Why shouldn't ZFS (which will be closer to UFS than HFS+)
> be the same?

Because Apple is Apple and things must be done The Apple Way(tm),
or no black turtleneck for you?

-- 
Shawn.

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

>>>>> "Shawn" == Shawn O Pearce <spearce@spearce.org> writes:

Shawn> Because Apple is Apple and things must be done The Apple Way(tm),
Shawn> or no black turtleneck for you?

So apparently the problem is known.
Is it also understood?
Is the solution understood, and just waiting for implementor tuits?

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Shawn O. Pearce]

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:
> >>>>> "Shawn" == Shawn O Pearce <spearce@spearce.org> writes:
> 
> Shawn> Because Apple is Apple and things must be done The Apple Way(tm),
> Shawn> or no black turtleneck for you?
> 
> So apparently the problem is known.
> Is it also understood?
> Is the solution understood, and just waiting for implementor tuits?

Yes, I think the issue has been beat to death here on the mailing
list by many parties.  Its more just an issue of someone writing
the code and finding the little implementation details that weren't
discussed yet.

-- 
Shawn.

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[David Brown]

On Sun, May 04, 2008 at 03:00:11PM -0700, Randal L. Schwartz wrote:

>Ahh, oddly enough, it does exactly the same on HFS+, but *not* UFS, which is
>why I hadn't noticed it yet, since all of my things I mirror are on my UFS
>partition, just in case they have case-colliding names.

Anyone know if 'case sensitive' HFS+ does name mangling?  Perhaps ZFS can
also be formed as case sensitive.

David

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Wincent Colaiuta]

El 5/5/2008, a las 8:52, David Brown escribió:
> On Sun, May 04, 2008 at 03:00:11PM -0700, Randal L. Schwartz wrote:
>
>> Ahh, oddly enough, it does exactly the same on HFS+, but *not* UFS,  
>> which is
>> why I hadn't noticed it yet, since all of my things I mirror are on  
>> my UFS
>> partition, just in case they have case-colliding names.
>
> Anyone know if 'case sensitive' HFS+ does name mangling?  Perhaps  
> ZFS can
> also be formed as case sensitive.

I believe case-sensitive HFS+ also mangles names.

I for one am pretty disappointed that Apple has chosen to bring across  
this undesirable behaviour from their legacy file system to a  
completely new one, when this would have been a perfect opportunity  
for them to break with the bad design decisions of the past. It's  
still pre-release, of course, so I guess there's still hope that their  
final version might not preserve this flaw.

Wincent

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

>>>>> "Randal" == Randal L Schwartz <merlyn@stonehenge.com> writes:

Shawn> Isn't this just the normal HFS+ name mangling?

Randal> I don't recall having this problem on HFS+.

Also, ZFS is case-sensitive, which leads me to believe that out of
the box, it's more like UFS.

Still don't know why ZFS breaks but UFS works.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Linus Torvalds]

On Mon, 5 May 2008, Randal L. Schwartz wrote:
> 
> Also, ZFS is case-sensitive, which leads me to believe that out of
> the box, it's more like UFS.
> 
> Still don't know why ZFS breaks but UFS works.

UFS is a traditional Unix filesystem, and will not mangle your filenames.

ZFS apparently acts like a case-sensitive HFS+: it still tries to 
normalize to UTF-8 (and does it badly, at that - picking an Apple-specific 
almost-NFD form of normalization rather than the more sensible NFC form).

So ZFS may not corrupt cases, but it still corrupts UTF-8 filenames.

			Linus

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Tarmigan]

On Mon, May 5, 2008 at 10:53 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>  On Mon, 5 May 2008, Randal L. Schwartz wrote:
>  >
>  > Also, ZFS is case-sensitive, which leads me to believe that out of
>  > the box, it's more like UFS.
>  >
>  > Still don't know why ZFS breaks but UFS works.
>
>  UFS is a traditional Unix filesystem, and will not mangle your filenames.
>
>  ZFS apparently acts like a case-sensitive HFS+: it still tries to
>  normalize to UTF-8 (and does it badly, at that - picking an Apple-specific
>  almost-NFD form of normalization rather than the more sensible NFC form).
>
>  So ZFS may not corrupt cases, but it still corrupts UTF-8 filenames.

Not sure if that is always the case.  This thread suggests otherwise:
http://opensolaris.org/jive/thread.jspa?threadID=53303&tstart=0

Case sensitivity and mangling look like they are configurable.  See
the "casesensitivity" and "normalization" options at
http://docs.sun.com/app/docs/doc/819-2240/zfs-1m

ZFS on OSX may just default to "insensitive" and some form of
normalization especially if they were/are thinking about making it the
default fs and wanted backwards compatibility for broken apps.
Randal, can you make a new FS with casesensitivity=sensitive and
normalization=none and see how it behaves?

-Tarmigan

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Randal L. Schwartz]

>>>>> "Tarmigan" == Tarmigan  <tarmigan+git@gmail.com> writes:

Tarmigan> ZFS on OSX may just default to "insensitive" and some form of
Tarmigan> normalization especially if they were/are thinking about making it the
Tarmigan> default fs and wanted backwards compatibility for broken apps.
Tarmigan> Randal, can you make a new FS with casesensitivity=sensitive and
Tarmigan> normalization=none and see how it behaves?

As I posted in a followup, without twiddling anything, ZFS on OSX
is case *sensitive*, but apparently still "normalizes", which is causing
the trip-up.  I'll look to see if there are any parameters I can tweak
to keep it from doing that, but I doubt it.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Smalltalk/Perl/Unix consulting, Technical writing, Comedy, etc. etc.
See http://methodsandmessages.vox.com/ for Smalltalk and Seaside discussion

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Mitch Tishmack]

On May 5, 2008, at 02:00 PM, Tarmigan wrote:
>
> Not sure if that is always the case.  This thread suggests otherwise:
> http://opensolaris.org/jive/thread.jspa?threadID=53303&tstart=0
>
> Case sensitivity and mangling look like they are configurable.  See
> the "casesensitivity" and "normalization" options at
> http://docs.sun.com/app/docs/doc/819-2240/zfs-1m
>
> ZFS on OSX may just default to "insensitive" and some form of
> normalization especially if they were/are thinking about making it the
> default fs and wanted backwards compatibility for broken apps.
> Randal, can you make a new FS with casesensitivity=sensitive and
> normalization=none and see how it behaves?

Those are only valid in OpenSolaris at the moment, they do nothing on  
the Leopard zfs beta:
macbookprov2# zfs set casesensitivity=sensitive zfs
cannot set property for 'zfs': invalid property 'casesensitivity'
macbookprov2# zfs set normalization=none zfs
cannot set property for 'zfs': invalid property 'normalization'
macbookprov2# uname -a
Darwin macbookprov2.local 9.2.2 Darwin Kernel Version 9.2.2: Tue Mar   
4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386 i386
macbookprov2# zfs get all zfs
NAME  PROPERTY       VALUE                  SOURCE
zfs   type           filesystem             -
zfs   creation       Sun May  4 22:03 2008  -
zfs   used           38.3M                  -
zfs   available      922M                   -
zfs   referenced     38.2M                  -
zfs   compressratio  1.00x                  -
zfs   mounted        yes                    -
zfs   quota          none                   default
zfs   reservation    none                   default
zfs   recordsize     128K                   default
zfs   mountpoint     /Volumes/zfs           default
zfs   sharenfs       off                    default
zfs   checksum       on                     default
zfs   compression    off                    default
zfs   atime          on                     default
zfs   devices        on                     default
zfs   exec           on                     default
zfs   setuid         on                     default
zfs   readonly       off                    default
zfs   zoned          off                    default
zfs   snapdir        hidden                 default
zfs   aclmode        groupmask              default
zfs   aclinherit     secure                 default
zfs   canmount       on                     default
zfs   shareiscsi     off                    default
zfs   xattr          on                     default
zfs   copies         1                      default
zfs   version        1                      -

mitch

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Linus Torvalds]

On Mon, 5 May 2008, Randal L. Schwartz wrote:
> 
> As I posted in a followup, without twiddling anything, ZFS on OSX
> is case *sensitive*, but apparently still "normalizes", which is causing
> the trip-up.  I'll look to see if there are any parameters I can tweak
> to keep it from doing that, but I doubt it.

Well, the good news is that while the utf-8 normalization is crazy, you're 
pretty unlikely to actually hit it in practice. The git archive has the 
Märchen file on purpose to find these kinds of issues, but it's harmless 
appart from the annoyance.

Yes, some projects will have i18n support that may have files with special 
characters on purpose, and they can cause problems. Not least exactly 
because OS X does all the normalization exactly the wrong way (ie it chose 
the _unusual_ way to do things, expanding strings to their longest format 
rather than their shortest format), but I would expect it to be very rare 
in general, partly because it doesn't really matter what SCM you use to 
track things, and you'll have similar issues if you ever develop anything 
across OS's.

We can fix it. Junio hasn't merged my stupid infrastructure into core git 
yet, and I still don't really care enough or have a OS X box to test on 
(I've got a couple of mac minis, but they run that other OS ;), but we'll 
get the insane OS X crap handled *some* day.

			Linus

Re: OSX, ZFS, UTF8, git - somebody hates me in this list

[Johannes Schindelin]

Hi,

On Mon, 5 May 2008, Linus Torvalds wrote:

> We can fix it. Junio hasn't merged my stupid infrastructure into core 
> git yet, and I still don't really care enough or have a OS X box to test 
> on (I've got a couple of mac minis, but they run that other OS ;), but 
> we'll get the insane OS X crap handled *some* day.

FWIW I think you can reproduce on Linux.  AFAIR there is some code in the 
hfs or hfsplus modules to perform the same UTF-8 normalization.

Ciao,
Dscho