Filesystem for mobile hard drive

Filesystem for mobile hard drive

[Nicolas George]

[-- Attachment #1: Type: text/plain, Size: 1761 bytes --]

Hi.

I am about to buy a mobile hard drive (actually, a FireWire/USB box and a
normal hard drive), and it raises the question of which filesystem to put on
it. I am not wondering which of ext3, reiserfs, XFS or JFS is best, but more
basically whether I should use a Linux/Unix-style filesystem or the horrible
FAT.

The drawbacks of FAT are numerous and well-known: poor efficiency with big
files, fragmentation, bad handling of file names, lack of robustness, and
worst of all, the 4 Go limit. On the other hand, FAT gives the possibility
to easyly read the drive on non-Unix systems (I know there are ext2 and
reiserfs readers for windows, I do not know for XFS or JFS, but at the worst
it should be possible to do something with colinux).

All these elements are rather feeble, but the Unix-style filesystems have a
big drawback as mobile filesystems: they store UIDs. UIDs make sense inside
a given system, but not across systems. On the most annoying case, I can
have my disk automatically mounted on a system where I am not root, and all
my files unreadable because they belong to another user.

Since big mobile mass-storage devices which require efficient filesystems
will become more and more common, I think this problem should be addressed.
Someone suggested to me to use some sort of network filesystem (NFS or SMB),
and its UID mapping facility. That should work, but that is rather an ugly
solution, and that is not something that can be done in five minutes while
visiting a friend.

I believe that we lack an option at the VFS to completely override file
ownership of a filesystem. But maybe there are other solutions.

Did someone already think in depths about this issue?


Regards,

-- 
  Nicolas George

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 185 bytes --]

Re: Filesystem for mobile hard drive

[Phillip Susi]

Nicolas George wrote:
> Hi.
> 
> I am about to buy a mobile hard drive (actually, a FireWire/USB box and a
> normal hard drive), and it raises the question of which filesystem to put on
> it. I am not wondering which of ext3, reiserfs, XFS or JFS is best, but more
> basically whether I should use a Linux/Unix-style filesystem or the horrible
> FAT.
> 
> The drawbacks of FAT are numerous and well-known: poor efficiency with big
> files, fragmentation, bad handling of file names, lack of robustness, and
> worst of all, the 4 Go limit. On the other hand, FAT gives the possibility
> to easyly read the drive on non-Unix systems (I know there are ext2 and
> reiserfs readers for windows, I do not know for XFS or JFS, but at the worst
> it should be possible to do something with colinux).
> 

If by FAT you mean FAT16, then yes, you have an 8 GB limit for the 
entire filesystem.  Fat32 on the other hand, can handle much more and so 
should be suitable in this aspect.  Fragmentation is also a property not 
of the filesystem, but of Microsoft's filesystem drivers.  I'm fairly 
sure that the linux fat implementations do not use absurdly stupid 
allocation algorithms that lead to lots of fragmentation.

> All these elements are rather feeble, but the Unix-style filesystems have a
> big drawback as mobile filesystems: they store UIDs. UIDs make sense inside
> a given system, but not across systems. On the most annoying case, I can
> have my disk automatically mounted on a system where I am not root, and all
> my files unreadable because they belong to another user.

This can be overcome with the UDF filesystem by using the uid and gid 
mount options, allowing the files to appear to be owned by the correct 
local user.  It would be nice if the other filesystems were patched to 
allow such options as well.


> 
> Since big mobile mass-storage devices which require efficient filesystems
> will become more and more common, I think this problem should be addressed.
> Someone suggested to me to use some sort of network filesystem (NFS or SMB),
> and its UID mapping facility. That should work, but that is rather an ugly
> solution, and that is not something that can be done in five minutes while
> visiting a friend.
> 

Network filesystems are not on disk filesystems, so they have nothing to 
do with this discussion; you can't format a disk as "nfs" or "smb".

> I believe that we lack an option at the VFS to completely override file
> ownership of a filesystem. But maybe there are other solutions.

I agree.  I recently submitted a patch fixing a bug in the udf 
filesystem so it correctly uses the uid and gid mount options to 
translate the ownership of files owned on disc by id -1 to the values 
you specify.  It would be nice if you could also specify that should be 
done for all files, not just those owned by nobody, and if all 
filesystems allowed you to do this.

> 
> Did someone already think in depths about this issue?
> 
> 
> Regards,
> 

Re: Filesystem for mobile hard drive

[Nicolas George]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

Le quartidi 24 pluviôse, an CCXIV, Phillip Susi a écrit :
> If by FAT you mean FAT16, then yes, you have an 8 GB limit for the 
> entire filesystem.  Fat32 on the other hand, can handle much more and so 
> should be suitable in this aspect.

According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To
for the whole filesystem. But the limit I was talking of is the file size
limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a
DVD-R does not fit, for example.

>				      Fragmentation is also a property not 
> of the filesystem, but of Microsoft's filesystem drivers.  I'm fairly 
> sure that the linux fat implementations do not use absurdly stupid 
> allocation algorithms that lead to lots of fragmentation.

I am not sure about that: you can not do really good algorthms on bad data
structures, and the data structures of FAT do not provide any support to do
smart allocation.

> This can be overcome with the UDF filesystem by using the uid and gid 
> mount options, allowing the files to appear to be owned by the correct 
> local user.

That is interesting. Do you know how efficient UDF is compared to others
filesystems on normal hard drives? It is optimized for CDs and DVDs, I would
not be surprised if the performances were poor on different supports.

>	       It would be nice if the other filesystems were patched to 
> allow such options as well.

I believe that such options should not be done on a per-filesystem basis.
Something in the common code of the VFS would be more logical. 

> Network filesystems are not on disk filesystems, so they have nothing to 
> do with this discussion; you can't format a disk as "nfs" or "smb".

The idea was to mount the disk with its haphazard UIDs, and then export it
and mount it as a network filesystem over the loopback. By itself, it is
absolutely useless, but networked filesystems have UIDs mapping facilities.


Regards,

-- 
  Nicolas George

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 185 bytes --]

Re: Filesystem for mobile hard drive

[Phillip Susi]

Nicolas George wrote:
> According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To
> for the whole filesystem. But the limit I was talking of is the file size
> limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a
> DVD-R does not fit, for example.
> 

Ahh yes, the per file limit.  BTW, why are you saying "To" and "Go" when 
you apparently mean "TB" and "GB"?

> I am not sure about that: you can not do really good algorthms on bad data
> structures, and the data structures of FAT do not provide any support to do
> smart allocation.
> 

The fat data structures do not encourage fragmentation any more or less 
than ext2/3.  NTFS is slightly better, more comparable to reiserfs than 
ext2/3, but the difference is small.  What causes massive fragmentation 
is how the driver chooses to allocate new blocks as you write to files. 
  Microsoft has always used about the worst possible algorithm for doing 
this you can imagine, which is why fragmentation has always been a big 
problem on their OSes.  Linux is smarter and allocates blocks such that 
fragmentation is kept to a minimum.

> That is interesting. Do you know how efficient UDF is compared to others
> filesystems on normal hard drives? It is optimized for CDs and DVDs, I would
> not be surprised if the performances were poor on different supports.
> 

I have not done any testing, but I know no reason why it would be worse 
than fat.  It does not do transaction logging, and there currently is no 
fsck for it, so for safety reasons, it may not be such a good choice. 
If you want reliability, efficiency, and posix semantics, I'd go with 
reiserfs or ext3.

> I believe that such options should not be done on a per-filesystem basis.
> Something in the common code of the VFS would be more logical. 
> 

I agree.  I think the VFS layer should process the uid/gid options.  By 
default it should replace nobody with the specified id, and fat and ntfs 
  should just report all files as owned by nobody.  Then a new option 
should be added to force the translation for all ids, not just nobody.


> The idea was to mount the disk with its haphazard UIDs, and then export it
> and mount it as a network filesystem over the loopback. By itself, it is
> absolutely useless, but networked filesystems have UIDs mapping facilities.
> 

Eww, that is quite a kludge.

Re: Filesystem for mobile hard drive

[Jan Engelhardt]

>> According to Wikipedia, and what I knew besides, FAT32 has a limit of 2 To
>> for the whole filesystem. But the limit I was talking of is the file size
>> limit: 4 Go perfile. Which is, nowadays, a bit short: an ISO image for a
>> DVD-R does not fit, for example.
>
> Ahh yes, the per file limit.  BTW, why are you saying "To" and "Go" when you
> apparently mean "TB" and "GB"?
>
French wording...



Jan Engelhardt
-- 

Re: Filesystem for mobile hard drive

[Kalin KOZHUHAROV]

Phillip Susi wrote:
> Nicolas George wrote:

[snip]

>> I believe that such options should not be done on a per-filesystem basis.
>> Something in the common code of the VFS would be more logical.
> 
> I agree.  I think the VFS layer should process the uid/gid options.  By
> default it should replace nobody with the specified id, and fat and ntfs
>  should just report all files as owned by nobody.  Then a new option
> should be added to force the translation for all ids, not just nobody.

I might be wrong, but I always thought that NTFS has user/group and a bunch
of other attributes, so it might not be a good idea to replace them hard
under linux. Or am I wrong? I never used NTFS much, the few windoze machines
around me use FAT32 for compatibility.

Kalin.

-- 
|[ ~~~~~~~~~~~~~~~~~~~~~~ ]|
+-> http://ThinRope.net/ <-+
|[ ______________________ ]|

Re: Filesystem for mobile hard drive

[Phillip Susi]

Kalin KOZHUHAROV wrote:
> I might be wrong, but I always thought that NTFS has user/group and a bunch
> of other attributes, so it might not be a good idea to replace them hard
> under linux. Or am I wrong? I never used NTFS much, the few windoze machines
> around me use FAT32 for compatibility.
>
> Kalin.

IIRC, NTFS has the capability to store a posix uid and gid, but it is 
never actually used.  There is no good way to get NT to understand 
mappings to linux uid/gids, so the information it puts there is 
useless.  As a result, the Linux NTFS driver just makes all files owned 
by a fixed id you can specify at mount time, or defaults to root. 

Re: Filesystem for mobile hard drive

[Nicolas George]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

Le quartidi 24 pluviôse, an CCXIV, Phillip Susi a écrit :
> Ahh yes, the per file limit.  BTW, why are you saying "To" and "Go" when 
> you apparently mean "TB" and "GB"?

I use the french word octet instead of byte, because it is less error prone
(when you read "mb", does-it really mean megabit, or does it mean that the
author is lazy about capitalization?) and a little bit more precise. Tough I
actually am French, I did not start using a French word in English by
myself. I copy a practice of the IETF: the RFCs use octet more than byte.

> The fat data structures do not encourage fragmentation any more or less 
> than ext2/3.  NTFS is slightly better, more comparable to reiserfs than 
> ext2/3, but the difference is small.  What causes massive fragmentation 
> is how the driver chooses to allocate new blocks as you write to files. 
>  Microsoft has always used about the worst possible algorithm for doing 
> this you can imagine, which is why fragmentation has always been a big 
> problem on their OSes.  Linux is smarter and allocates blocks such that 
> fragmentation is kept to a minimum.

I believe you about that.

> I have not done any testing, but I know no reason why it would be worse 
> than fat.

That is a very good point. If windows can read UDF on hard drives and not
only DVD, UDF could probably supersede FAT completely.

Thank you for pointing me that direction.

>	    It does not do transaction logging, and there currently is no 
> fsck for it, so for safety reasons, it may not be such a good choice. 

I have a Solaris 9 near at hand, and I see a /lib/fs/udfs/fsck, and in the
source tarball of OpenSolaris, I find a directory
usr/src/cmd/fs.d/udfs/fsck/. It does not compile out of the box, but it may
be possible to port it with limited effort.

> I agree.  I think the VFS layer should process the uid/gid options.  By 
> default it should replace nobody with the specified id, and fat and ntfs 
> should just report all files as owned by nobody.  Then a new option 
> should be added to force the translation for all ids, not just nobody.

I agree with that (except maybe for the NTFS part, which I do not know; let
us just say "UID-less filesystems"). Maybe a full UID translation system
similar th the one in NFS could be useful, or a generic hook for modules,
but having basic UID overriding would be great.

Unfortunately, the VFS subsystem is something too complex for me at this
time.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 185 bytes --]

Re: Filesystem for mobile hard drive

[Phillip Susi]

Nicolas George wrote:
> I use the french word octet instead of byte, because it is less error prone
> (when you read "mb", does-it really mean megabit, or does it mean that the
> author is lazy about capitalization?) and a little bit more precise. Tough I
> actually am French, I did not start using a French word in English by
> myself. I copy a practice of the IETF: the RFCs use octet more than byte.
>   

Ahh, I see.  I've never seen anyone use it in conjunction with an si 
prefix.  I also think that they use it in RFCs because at the time they 
started writing them, bytes were not always 8 bits on all machines.  
Today it is a pretty safe assumption that a byte is 8 bits, so most 
people use the two terms interchangeably ;)

As for 1000 vs 1024 powers, I personally use the general rule that B = 
bytes and is in powers of 1024, and b = bits, and is in powers of 1000.  
For some reason the telecom industry likes to talk in terms of 1000 bits 
per second, but the rest of the sane world uses bytes in even powers of 
two.  Generally it's people trying to sell you something that use powers 
of 1000 so they can make it sound bigger.  I really hate that. 

> That is a very good point. If windows can read UDF on hard drives and not
> only DVD, UDF could probably supersede FAT completely.
>
> Thank you for pointing me that direction.
>
>   

I had that same thought a few weeks ago so I gave it a try.  I formatted 
a partition with UDF, put some files on it, then booted windows to see 
if it would take it.  It didn't :(

> I have a Solaris 9 near at hand, and I see a /lib/fs/udfs/fsck, and in the
> source tarball of OpenSolaris, I find a directory
> usr/src/cmd/fs.d/udfs/fsck/. It does not compile out of the box, but it may
> be possible to port it with limited effort.
>
>   

Hrm... interesting, I wonder how complete it is and what it's license 
is?  The fsck in the udftools package was never actually implemented 
unfortunately.

>> I agree.  I think the VFS layer should process the uid/gid options.  By 
>> default it should replace nobody with the specified id, and fat and ntfs 
>> should just report all files as owned by nobody.  Then a new option 
>> should be added to force the translation for all ids, not just nobody.
>>     
>
> I agree with that (except maybe for the NTFS part, which I do not know; let
> us just say "UID-less filesystems"). Maybe a full UID translation system
> similar th the one in NFS could be useful, or a generic hook for modules,
> but having basic UID overriding would be great.
>
> Unfortunately, the VFS subsystem is something too complex for me at this
> time.
>   

Re: Filesystem for mobile hard drive

[Nicolas George]

[-- Attachment #1: Type: text/plain, Size: 2062 bytes --]

Le quintidi 25 pluviôse, an CCXIV, Phillip Susi a écrit :
> Ahh, I see.  I've never seen anyone use it in conjunction with an si 
> prefix.  I also think that they use it in RFCs because at the time they 
> started writing them, bytes were not always 8 bits on all machines.  
> Today it is a pretty safe assumption that a byte is 8 bits, so most 
> people use the two terms interchangeably ;)

They continue using more octet than bytes even in recent RFCs. I have read I
do not remember where that the goal was to avoid byte/bit confusion.

I am sorry, I did not intend to start an off-topic subthread. I think I
should stick with kB/MB/GB unless I already used the full word earlier.

> I had that same thought a few weeks ago so I gave it a try.  I formatted 
> a partition with UDF, put some files on it, then booted windows to see 
> if it would take it.  It didn't :(

So bad... Perhaps it was asking too much...

> Hrm... interesting, I wonder how complete it is and what it's license 
> is?

The man page (<URL:
http://docs.sun.com/app/docs/doc/816-5166/6mbb1kq22?a=view > for Solaris 10,
I believe OpenSolaris is based on it) tells briefly that the checked
inconsistencies are (I quote):

- Blocks claimed by more than one file or the free list
- Blocks claimed by a file or the free list outside the range of the file system
- Incorrect link counts in file entries
- Incorrect directory sizes
- Bad file entry format
- Blocks not accounted for anywhere
- Directory checks, file pointing to unallocated file entry and absence of a
  parent directory entry
- Descriptor checks, more blocks for files than there are in the file system
- Bad free block list format
- Total free block count incorrect

I do not know UDF at all, so I can not tell if this is enough or not.

As for the licence, it is the one of OpenSolaris <URL:
http://www.opensolaris.org/os/licensing/opensolaris_license/ >, which is
free enough for the FSF to make efforts to have GPL3 compatible with it.


Regards,

-- 
  Nicolas George

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 185 bytes --]