Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Thu, 20 Aug 2009 02:17:21 GMT
bugzilla-daemon@bugzilla.kernel.org wrote:

> http://bugzilla.kernel.org/show_bug.cgi?id=14021
> 
>            Summary: hfsplus caused data loss
>            Product: File System
>            Version: 2.5
>     Kernel Version: 2.6.31-rc5
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: HFS/HFSPLUS
>         AssignedTo: zippel@linux-m68k.org
>         ReportedBy: rbrito@ime.usp.br
>         Regression: Yes
> 
> 
> Hi.
> 
> I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> filesystems so that they can be used in Linux as a way to transfer data among
> computers, but I had quite a surprise when I was using the hfsplus module.
> 
> I just created a 100MB file with nulls (dd if=/dev/null ...) and I created an
> HFS+ filesystem on it.
> 
> Then, I loop mounted it and tried to use it a little bit (in particular,
> applying patches with quilt on a source tree). The commands spit some errors
> about not being able to create links (I never had that problem before) and the
> directory where I was became empty!
> 
> Furthermore, here is a quite, quite strange directory listing:
> 
> ,----
> | rbrito@chagas:/media/usb7$ ls -lAF
> | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> | ls: hfsprogs_332.14-7.dsc: No such file or directory
> | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> | ls: hfsprogs_332.18-1.dsc: No such file or directory
> | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> | total 1636
> | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13 hfsprogs_332.14.orig.tar.gz
> | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> hfsprogs_332.18-1_amd64.changes
> | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26 hfsprogs_332.18-1_amd64.deb
> | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35 hfsprogs_332.18.orig.tar.gz
> | rbrito@chagas:/media/usb7$
> `----
> 
> I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to one
> or two days before rc6).
> 
> There are no messages in the dmesg, besides these:
> 
> ,----
> | [30991.501804] loop: module loaded
> | [30991.513337] hfs: create hidden dir...
> | [39897.867830] hfs: create hidden dir...
> | [39960.061622] hfs: splitting index node...
> `----
> 
> No stack traces, no nothing. Oh, I still have the disk image that I created, if
> it is of any interest.
> 
> Please let me know if I can provide any further information.
> 

Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
I cc'ed the ppc guys as I expect that most HFS users are over there.

It seems like a pretty gross failure - others should be hitting it.

I wonder if it's a weird interaction with the loop driver.

Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

[Rogério Brito]

Hi, Andrew.

On Aug 20 2009, Andrew Morton wrote:
> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I
> > created, if it is of any interest.
>
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.

Right.

I think that I could run some stress tests (say, like the exhaustive
regression tests that the ntfs-3g developers use), just to get some
basic functionality on HFS+ working.

> I cc'ed the ppc guys as I expect that most HFS users are over there.

Thanks once again, Andrew.

> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.

Just to make it clear: the reason why I loop-mounted the filesystem was
just to work with a low-risk situation and submit the image to the
recently "ported" fsck, just to get a little bit more confidence that I
didn't mess with the userspace side of things.

But, then, this surprise.

I had some minor inconsistencies with the HFS+ driver in the past, but
they were fixed by running the fsck.hfsplus that I packaged and I didn't
bother reporting it before. My mistake. :-(


Thanks, Rogério Brito.

-- 
Rogério Brito : rbrito@{mackenzie,ime.usp}.br : GPG key 1024D/7C2CAEB8
http://www.ime.usp.br/~rbrito : http://meusite.mackenzie.com.br/rbrito
Projects: algorithms.berlios.de : lame.sf.net : vrms.alioth.debian.org
_______________________________________________
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

[Brad Boyer]

On Thu, Aug 20, 2009 at 03:02:42PM -0700, Andrew Morton wrote:
> 
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
> 
> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=14021
> > 
> >            Summary: hfsplus caused data loss
> >            Product: File System
> >            Version: 2.5
> >     Kernel Version: 2.6.31-rc5
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: HFS/HFSPLUS
> >         AssignedTo: zippel@linux-m68k.org
> >         ReportedBy: rbrito@ime.usp.br
> >         Regression: Yes
> > 
> > 
> > Hi.
> > 
> > I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> > filesystems so that they can be used in Linux as a way to transfer data among
> > computers, but I had quite a surprise when I was using the hfsplus module.
> > 
> > I just created a 100MB file with nulls (dd if=/dev/null ...) and I created an
> > HFS+ filesystem on it.
> > 
> > Then, I loop mounted it and tried to use it a little bit (in particular,
> > applying patches with quilt on a source tree). The commands spit some errors
> > about not being able to create links (I never had that problem before) and the
> > directory where I was became empty!
> > 
> > Furthermore, here is a quite, quite strange directory listing:
> > 
> > ,----
> > | rbrito@chagas:/media/usb7$ ls -lAF
> > | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> > | ls: hfsprogs_332.14-7.dsc: No such file or directory
> > | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.dsc: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> > | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> > | total 1636
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> > | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> > | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13 hfsprogs_332.14.orig.tar.gz
> > | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> > | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> > hfsprogs_332.18-1_amd64.changes
> > | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26 hfsprogs_332.18-1_amd64.deb
> > | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35 hfsprogs_332.18.orig.tar.gz
> > | rbrito@chagas:/media/usb7$
> > `----
> > 
> > I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to one
> > or two days before rc6).
> > 
> > There are no messages in the dmesg, besides these:
> > 
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I created, if
> > it is of any interest.
> > 
> > Please let me know if I can provide any further information.
> > 
> 
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
> I cc'ed the ppc guys as I expect that most HFS users are over there.
> 
> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.

I can explain the ls output in a very generic sense. This is what you
get if the lookup fails on a name returned by a readdir. I suspect
that creating hard links is failing in some way. In particular, the
"hidden dir" mentioned in the log is used to save the real files for
hard links.

As a quick explanation, HFS+ doesn't have a real concept of hard
links. Apple hacked it in after the fact. The way it works is that
there is a special hidden directory, and the real catalog entries
that match the real names are just references to the hidden file.
Both HFS and HFS+ have no separate notion of directory entries and
inode data. The metadata is stored in records keyed by file name.

Hard links were added after I stopped working on the HFS+ code, so
I don't know it in detail. I know it works for reading hard links,
but I never actually tried creating more links.

	Brad Boyer
	flar@allandria.com

Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

[Benjamin Herrenschmidt]

On Thu, 2009-08-20 at 15:02 -0700, Andrew Morton wrote:
> (switched to email.  Please respond via emailed reply-to-all, not via the
> bugzilla web interface).

Is Roman Zippel still around ? (I added him to the CC list). AFAIK, He
maintains HFS and HFS+ (and was the last one to do any major work on
them).

Cheers,
Ben.

> On Thu, 20 Aug 2009 02:17:21 GMT
> bugzilla-daemon@bugzilla.kernel.org wrote:
> 
> > http://bugzilla.kernel.org/show_bug.cgi?id=14021
> > 
> >            Summary: hfsplus caused data loss
> >            Product: File System
> >            Version: 2.5
> >     Kernel Version: 2.6.31-rc5
> >           Platform: All
> >         OS/Version: Linux
> >               Tree: Mainline
> >             Status: NEW
> >           Severity: normal
> >           Priority: P1
> >          Component: HFS/HFSPLUS
> >         AssignedTo: zippel@linux-m68k.org
> >         ReportedBy: rbrito@ime.usp.br
> >         Regression: Yes
> > 
> > 
> > Hi.
> > 
> > I am trying to package a new version of Apple's own fsck/mkfs for HFS+
> > filesystems so that they can be used in Linux as a way to transfer data among
> > computers, but I had quite a surprise when I was using the hfsplus module.
> > 
> > I just created a 100MB file with nulls (dd if=/dev/null ...) and I created an
> > HFS+ filesystem on it.
> > 
> > Then, I loop mounted it and tried to use it a little bit (in particular,
> > applying patches with quilt on a source tree). The commands spit some errors
> > about not being able to create links (I never had that problem before) and the
> > directory where I was became empty!
> > 
> > Furthermore, here is a quite, quite strange directory listing:
> > 
> > ,----
> > | rbrito@chagas:/media/usb7$ ls -lAF
> > | ls: hfsprogs_332.14-7.diff.gz: No such file or directory
> > | ls: hfsprogs_332.14-7.dsc: No such file or directory
> > | ls: hfsprogs_332.14.orig.tar.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.diff.gz: No such file or directory
> > | ls: hfsprogs_332.18-1.dsc: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.changes: No such file or directory
> > | ls: hfsprogs_332.18-1_amd64.deb: No such file or directory
> > | ls: hfsprogs_332.18.orig.tar.gz: No such file or directory
> > | total 1636
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     39 Aug 17 08:13 hfsprogs-332.14/
> > | drwxr-xr-x 1 rbrito rbrito     45 Aug 17 15:30 hfsprogs-332.18/
> > | -rw-r--r-- 1 rbrito rbrito  35609 Aug 17 08:13 hfsprogs_332.14-7.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito   1193 Aug 17 08:13 hfsprogs_332.14-7.dsc
> > | -rw-r--r-- 1 rbrito rbrito 714035 Aug 17 08:13 hfsprogs_332.14.orig.tar.gz
> > | -rw-r--r-- 1 rbrito rbrito  35342 Aug 17 15:26 hfsprogs_332.18-1.diff.gz
> > | -rw-r--r-- 1 rbrito rbrito    954 Aug 17 15:26 hfsprogs_332.18-1.dsc
> > | -rw-r--r-- 1 rbrito rbrito   2148 Aug 17 15:26
> > hfsprogs_332.18-1_amd64.changes
> > | -rw-r--r-- 1 rbrito rbrito 135398 Aug 17 15:26 hfsprogs_332.18-1_amd64.deb
> > | -rw-r--r-- 1 rbrito rbrito 732449 Aug 17 08:35 hfsprogs_332.18.orig.tar.gz
> > | rbrito@chagas:/media/usb7$
> > `----
> > 
> > I am using kernel 2.6.31-rc5-1rb.pre6 (that is, rc5 with git updates up to one
> > or two days before rc6).
> > 
> > There are no messages in the dmesg, besides these:
> > 
> > ,----
> > | [30991.501804] loop: module loaded
> > | [30991.513337] hfs: create hidden dir...
> > | [39897.867830] hfs: create hidden dir...
> > | [39960.061622] hfs: splitting index node...
> > `----
> > 
> > No stack traces, no nothing. Oh, I still have the disk image that I created, if
> > it is of any interest.
> > 
> > Please let me know if I can provide any further information.
> > 
> 
> Gee.  Nobody really does much maintenance work on hfs/hfsplus any more.
> I cc'ed the ppc guys as I expect that most HFS users are over there.
> 
> It seems like a pretty gross failure - others should be hitting it.
> 
> I wonder if it's a weird interaction with the loop driver.
> 
> 
> _______________________________________________
> Linuxppc-dev mailing list
> Linuxppc-dev@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/linuxppc-dev

Re: [Bugme-new] [Bug 14021] New: hfsplus caused data loss

[Andrew Morton]

On Fri, 21 Aug 2009 17:29:01 +1000 Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:

> On Thu, 2009-08-20 at 15:02 -0700, Andrew Morton wrote:
> > (switched to email.  Please respond via emailed reply-to-all, not via the
> > bugzilla web interface).
> 
> Is Roman Zippel still around ? (I added him to the CC list). AFAIK, He
> maintains HFS and HFS+ (and was the last one to do any major work on
> them).

I don't think I've heard from Roman since January of this year.