PROBLEM: hfsplus filesystem allows opendir() on plain files

PROBLEM: hfsplus filesystem allows opendir() on plain files

[Damian Stewart]

1. hfsplus filesystem allows opendir() on plain files.

2. When opendir() is called on a plain file stored in an hfsplus 
filesystem, opendir() fails to return 0 and set errno to ENOTDIR.

3. filesystem, hfsplus

4. Linux version 2.6.27-7-generic (buildd@palmer) (gcc version 4.3.2 
(Ubuntu 4.3.2-1ubuntu11) ) #1 SMP Tue Nov 4 19:33:20 UTC 2008

5. unknown

6. n/a

7. Try the following program on files and directories first under a 
mountpoint for an ext3 partition then under a mountpoint for an hfsplus 
partition.

///////////////////////////////////
#include <dirent.h>
#include <stdio.h>
#include <errno.h>

// usage: opendir_test <file>|<directory>

int main( int argc, char** argv )
{
     DIR* result = opendir( argv[1] );
     printf("ENOTDIR is 0x%x\n", ENOTDIR );
     printf("opendir '%s' returned 0x%x, errno 0x%x\n", argv[1], result, 
errno );
     return 0;
}
//////////////////////////////////


I get the following output:

damian@damian-laptop:~/code/opendir$ mkdir test_directory
damian@damian-laptop:~/code/opendir$ echo hello > test_file.txt
damian@damian-laptop:~/code/opendir$ ./opendir_test test_directory
ENOTDIR is 0x14
opendir 'test_directory' returned 0x88f6008, errno 0x0
damian@damian-laptop:~/code/opendir$ ./opendir_test test_file.txt
ENOTDIR is 0x14
opendir 'test_file.txt' returned 0x0, errno 0x14
# /osx is an HFSPLUS mountpoint
damian@damian-laptop:~/code/opendir$ cd /media/osx
# mkdir test_directory && echo hello > test_file.txt performed in OSX
damian@damian-laptop:/media/osx$ ./opendir_test test_directory
ENOTDIR is 0x14
opendir 'test_directory' returned 0x919d008, errno 0x0
# erroneous result follows
damian@damian-laptop:/media/osx$ ./opendir_test test_file.txt
ENOTDIR is 0x14
opendir 'test_file.txt' returned 0x9a23008, errno 0x0
damian@damian-laptop:/media/osx$

8. If some fields are empty or look unusual you may have an old version.
Compare to the current minimal requirements in Documentation/Changes.

Linux damian-laptop 2.6.27-7-generic #1 SMP Tue Nov 4 19:33:20 UTC 2008 
i686 GNU/Linux

Gnu C                  4.3.2-1ubuntu11)
Gnu make               3.81
binutils               Binutils
util-linux             2.14
mount                  support
module-init-tools      3.3-pre11
e2fsprogs              1.41.3
reiserfsprogs          3.6.19
pcmciautils            014
Linux C Library        2.8.90
Dynamic linker (ldd)   2.8.90
Procps                 3.2.7
Net-tools              1.60
Kbd                    1.14.1
Sh-utils               6.10
udev                   124
wireless-tools         29
Modules Loaded         ipv6 aes_i586 aes_generic nls_utf8 hfsplus i915 drm 
af_packet binfmt_misc btusb sco bridge stp bnep rfcomm l2cap bluetooth 
ppdev acpi_cpufreq cpufreq_stats cpufreq_powersave cpufreq_conservative 
cpufreq_userspace cpufreq_ondemand freq_table container wmi pci_slot sbs 
sbshc iptable_filter ip_tables x_tables sbp2 parport_pc lp parport joydev 
snd_hda_intel arc4 snd_pcm_oss snd_mixer_oss iTCO_wdt ecb crypto_blkcipher 
iTCO_vendor_support snd_pcm ath9k evdev appleir pcspkr appletouch 
snd_seq_dummy mac80211 snd_seq_oss cfg80211 snd_seq_midi snd_rawmidi 
snd_seq_midi_event snd_seq video snd_timer snd_seq_device output battery 
snd ac intel_agp button soundcore agpgart shpchp pci_hotplug snd_page_alloc 
ext3 jbd mbcache sd_mod crc_t10dif sr_mod cdrom sg ata_generic ata_piix 
usbhid hid pata_acpi ohci1394 libata ieee1394 scsi_mod ehci_hcd uhci_hcd 
dock usbcore sky2 thermal processor fan fbcon tileblit font bitblit 
softcursor fuse

thanks
d

-- 
damian stewart | skype: damiansnz | damian@frey.co.nz
frey | live art with machines | http://www.frey.co.nz

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Brad Boyer]

On Tue, Dec 02, 2008 at 04:26:56PM +0100, Damian Stewart wrote:
> 1. hfsplus filesystem allows opendir() on plain files.
> 
> 2. When opendir() is called on a plain file stored in an hfsplus 
> filesystem, opendir() fails to return 0 and set errno to ENOTDIR.

Yes, this is a side effect of an intentional behavior. In osx, you
can open "file/rsrc" to get the resource fork of "file". This
behavior is emulated inside hfsplus on Linux, which means that to
some degree every file acts like a directory. There isn't a readdir
operation so it's not possible to list these, but it is possible to
do a lookup on this specific name. The code path taken by opendir
inside the kernel checks inode->i_op->lookup and returns ENOTDIR if
this function pointer is not set.

We can't fix this without either eliminating this way to open the
resource fork or making real changes to the generic Linux code. I'm
not sure what would be the appropriate thing to do in this case.
Someone on the mailing list is likely to have a strong opinion, so
there are likely to be suggestions of one sort or another.

Maybe the check in the generic code to decide what is a "directory"
should be changed? We could possibly check inode->i_fop->readdir.

	Brad Boyer
	flar@allandria.com

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Damian Stewart]

Brad Boyer wrote:
> Maybe the check in the generic code to decide what is a "directory"
> should be changed? We could possibly check inode->i_fop->readdir.

Hi Brad,

Thanks for your reply.

This issue came up because at least one major project (the VLC video 
player) is using 'opendir()' to determine whether or not a path is a file 
or directory. (They probably ought to be using 'stat()', but they're not, 
so this issue shows up.)

I've reported it on the VLC forums but only got very dismissive replies. 
The few developers who posted bluntly said it was a 'kernel bug' so they 
weren't willing to do anything about it; which is a shame, because it 
leaves those VLC users with videos on HFS+ drives unable to play their 
videos with the latest version of VLC.

Anyways, perhaps your suggestion is the right one, in this case.

Cheers
d
-- 
damian stewart | skype: damiansnz | damian@frey.co.nz
frey | live art with machines | http://www.frey.co.nz

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Brad Boyer]

On Fri, Dec 05, 2008 at 09:53:51AM +0100, Damian Stewart wrote:
> This issue came up because at least one major project (the VLC video 
> player) is using 'opendir()' to determine whether or not a path is a file 
> or directory. (They probably ought to be using 'stat()', but they're not, 
> so this issue shows up.)

Unless they're already needing to open any directories they check, doing
an opendir call to check a file type seems less than ideal. Hopefully
they had a reason, but I know how that sort of thing can happen.

> I've reported it on the VLC forums but only got very dismissive replies. 
> The few developers who posted bluntly said it was a 'kernel bug' so they 
> weren't willing to do anything about it; which is a shame, because it 
> leaves those VLC users with videos on HFS+ drives unable to play their 
> videos with the latest version of VLC.

Well, I actually tend to agree that it is less than ideal behavior
on the part of the kernel in spite of thinking that they are doing
something stupid. It really ought to work.

> Anyways, perhaps your suggestion is the right one, in this case.

We'll see what comes out of this.

	Brad Boyer
	flar@allandria.com

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Christoph Hellwig]

On Tue, Dec 02, 2008 at 02:21:11PM -0800, Brad Boyer wrote:
> We can't fix this without either eliminating this way to open the
> resource fork

Which sould have happened long ago.  See the reiser4 discussion.  You're
breaking userspace semantic without a good reason.

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Brad Boyer]

On Fri, Dec 05, 2008 at 04:39:49AM -0500, Christoph Hellwig wrote:
> On Tue, Dec 02, 2008 at 02:21:11PM -0800, Brad Boyer wrote:
> > We can't fix this without either eliminating this way to open the
> > resource fork
> 
> Which sould have happened long ago.  See the reiser4 discussion.  You're
> breaking userspace semantic without a good reason.

I personally don't care about the feature myself. I'm not sure what
good it does anyway, and I don't know of anything on Linux that
uses it. However, I also wasn't the one that added it. My original
version of hfsplus did not have any way to access them. Maybe it's
best to just remove it and see who complains about it. I do want to
point out that I don't maintain that code anymore and don't have
time to do that work myself. As far as I know, hfsplus is officially
unmaintained these days.

Just as a side note, hfs has the same feature. The two modules are
basically the same code these days.

	Brad Boyer
	flar@allandria.com

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Al Viro]

On Fri, Dec 05, 2008 at 10:32:26PM -0800, Brad Boyer wrote:
> I personally don't care about the feature myself. I'm not sure what
> good it does anyway, and I don't know of anything on Linux that
> uses it. However, I also wasn't the one that added it. My original
> version of hfsplus did not have any way to access them. Maybe it's
> best to just remove it and see who complains about it. I do want to
> point out that I don't maintain that code anymore and don't have
> time to do that work myself. As far as I know, hfsplus is officially
> unmaintained these days.
> 
> Just as a side note, hfs has the same feature. The two modules are
> basically the same code these days.

You know what...  I thought that HFS+ did not support links; looking at
the source, it appears that hardlinks *are* supported and that brings
all the deadlocks into the game, same as we had in reiser4 case.

Which is to say, this stuff *must* go.  In case of HFS it's obnoxious,
but not immediately letal.  In case of HFS+ it's an instant DoS.

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Brad Boyer]

On Sat, Dec 06, 2008 at 06:47:39AM +0000, Al Viro wrote:
> You know what...  I thought that HFS+ did not support links; looking at
> the source, it appears that hardlinks *are* supported and that brings
> all the deadlocks into the game, same as we had in reiser4 case.

Apple added hard links with osx. Back when I wrote hfsplus originally,
there wasn't any such thing on HFS+ either. The support for the hack
that added the appearance of hard links was added after I stopped
maintaining the code. It's really just a matter of hiding the real
file in a special place and putting stubs with special settings where
all the links appear in the namespace. In all truth both HFS and HFS+
are inherently incompatible with the very concept of a hard link or
having a file still exist after removal. It's just a horrible hack
that makes it look like it works.

> Which is to say, this stuff *must* go.  In case of HFS it's obnoxious,
> but not immediately letal.  In case of HFS+ it's an instant DoS.

Feel free. I disavowed any responsibility for this code years ago. And
before anyone thinks of complaining, I'll point out that I wasn't the
one that submitted hfsplus for inclusion in the kernel either.

	Brad Boyer
	flar@allandria.com

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Al Viro]

On Fri, Dec 05, 2008 at 11:21:22PM -0800, Brad Boyer wrote:

> Apple added hard links with osx. Back when I wrote hfsplus originally,
> there wasn't any such thing on HFS+ either. The support for the hack
> that added the appearance of hard links was added after I stopped
> maintaining the code. It's really just a matter of hiding the real
> file in a special place and putting stubs with special settings where
> all the links appear in the namespace. In all truth both HFS and HFS+
> are inherently incompatible with the very concept of a hard link or
> having a file still exist after removal. It's just a horrible hack
> that makes it look like it works.

Well...  That's not too unreasonably way to do them, actually - just
treat your hidden directory as real inode table.  The tricky part is
keeping i_ino stable through all that, but if it's done...

FWIW, fs/qnx4 uses a somewhat similar scheme, IIRC.

To make things workable you need
	* hidden directory that would never be reachable by syscalls
	* recognizable files that would contain references to names
in that directory
	* lookup that would normally lead to such file getting inode
of object in hidden directory
	* link(2) with target being *not* one of those would start with
moving target to hidden directory and creating such "link" file in its
place.  Then it would create a link file in source.  That, BTW, would have
interesting implications wrt locking - we'd need to modify target's parent,
which can get sucky.
	* unlink(2) with victim dentry not being a "link" file *and* being
busy would move the victim to hidden.  Otherwise it'd kill the victim
outright.
	* final iput() of file that'd been taken to hidden does deletion
of directory entry in hidden if i_nlink is 0.

The interesting part is locking - you need to deal with lookup vs. eviction
of link(2) target to hidden and with rename vs. the same thing.

> > Which is to say, this stuff *must* go.  In case of HFS it's obnoxious,
> > but not immediately letal.  In case of HFS+ it's an instant DoS.
> 
> Feel free. I disavowed any responsibility for this code years ago. And
> before anyone thinks of complaining, I'll point out that I wasn't the
> one that submitted hfsplus for inclusion in the kernel either.

Not blaming you...

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Al Viro]

On Sat, Dec 06, 2008 at 10:31:33PM +0000, Al Viro wrote:
> On Fri, Dec 05, 2008 at 11:21:22PM -0800, Brad Boyer wrote:
> 
> > Apple added hard links with osx. Back when I wrote hfsplus originally,
> > there wasn't any such thing on HFS+ either. The support for the hack
> > that added the appearance of hard links was added after I stopped
> > maintaining the code. It's really just a matter of hiding the real
> > file in a special place and putting stubs with special settings where
> > all the links appear in the namespace. In all truth both HFS and HFS+
> > are inherently incompatible with the very concept of a hard link or
> > having a file still exist after removal. It's just a horrible hack
> > that makes it look like it works.
> 
> Well...  That's not too unreasonably way to do them, actually - just
> treat your hidden directory as real inode table.  The tricky part is
> keeping i_ino stable through all that, but if it's done...
> 
> FWIW, fs/qnx4 uses a somewhat similar scheme, IIRC.
> 
> To make things workable you need
> 	* hidden directory that would never be reachable by syscalls
> 	* recognizable files that would contain references to names
> in that directory
> 	* lookup that would normally lead to such file getting inode
> of object in hidden directory
> 	* link(2) with target being *not* one of those would start with
> moving target to hidden directory and creating such "link" file in its
> place.  Then it would create a link file in source.  That, BTW, would have
> interesting implications wrt locking - we'd need to modify target's parent,
> which can get sucky.
> 	* unlink(2) with victim dentry not being a "link" file *and* being
> busy would move the victim to hidden.  Otherwise it'd kill the victim
> outright.
> 	* final iput() of file that'd been taken to hidden does deletion
> of directory entry in hidden if i_nlink is 0.
> 
> The interesting part is locking - you need to deal with lookup vs. eviction
> of link(2) target to hidden and with rename vs. the same thing.

PS: that's for handling link(2) on non-directories with that kind of layout,
*NOT* for file-cross-directory hybrids or links to directories.

Re: PROBLEM: hfsplus filesystem allows opendir() on plain files

[Anton Altaparmakov]

Hi,

On 6 Dec 2008, at 06:47, Al Viro wrote:
> You know what...  I thought that HFS+ did not support links; looking  
> at
> the source, it appears that hardlinks *are* supported and that brings
> all the deadlocks into the game, same as we had in reiser4 case.
>
> Which is to say, this stuff *must* go.  In case of HFS it's obnoxious,
> but not immediately letal.  In case of HFS+ it's an instant DoS.

At least on OS X 10.5 (Leopard) HFS+ not only supports hard links but  
also supports hard links on directories (though they are read-only)!

btw. That is how time machine works (the built in backup tool that  
backs up to a HFS+ file system and then each hour it creates a new  
directory tree by doing hard links to the previous one including the  
directories but the backups are read-only).

Supporting that on Linux as it is today is likely to be impossible  
without major work which no-one is prepared to do / accept...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/