PATCH to access old-style FAT fs

PATCH to access old-style FAT fs

[Frodo Looijaard]

[-- Attachment #1: Type: text/plain, Size: 1878 bytes --]

Hi folks,

I have created and attached a new version of my old-style FAT filesystem
patch, this time for the 2.6.0 kernel. It can also be found on
http://debian.frodo.looijaard.name/. 

Some old implementation of the FAT standard mark the end of the
directory file index by inserting a filename beginning with a byte 00.
All entries after it should be ignored, even though they are not marked
as deleted. At least some EPOC releases (an OS used on Psion PDAs, for
example) still use this policy.

The included patch adds the `oldfat' mount option for FAT-based
filesystems. Without it, doing an 'ls' on old-style FATs will show a lot
of garbage, both old files that have been deleted, and bogus entries
that have never been filled.

If you do not use the `oldfat' mount option, FAT filesystems should work
exactly as before. I have been very careful not to change the logic of
the FAT driver, in order not to break anything. In fact, the patch could
probably be optimized a bit, but I wanted to keep it as risk-free as
possible.

It should be safe to mount a normal FAT filesystem with the `oldfat'
mount option. I have been doing that on and off with some local Win98
filesystems without any trouble. Actually, the only reason I have
introduced the mount option is to keep the new logic strictly seperated
from the old driver.

All feedback would be very welcome. This patch is released under the
GPL. Feel free to include it in stock kernels.

Debian users can install the .deb on http://debian.frodo.looijaard.name
and have the patch applied automatically when compiling 2.6.x kernels.

Thanks,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

[-- Attachment #2: linux-fat-2.6.0.patch.gz --]
[-- Type: application/octet-stream, Size: 1458 bytes --]

Re: PATCH to access old-style FAT fs

[H. Peter Anvin]

Followup to:  <20040126173949.GA788@frodo.local>
By author:    Frodo Looijaard <frodol@dds.nl>
In newsgroup: linux.dev.kernel
> 
> Hi folks,
> 
> I have created and attached a new version of my old-style FAT filesystem
> patch, this time for the 2.6.0 kernel. It can also be found on
> http://debian.frodo.looijaard.name/. 
> 
> Some old implementation of the FAT standard mark the end of the
> directory file index by inserting a filename beginning with a byte 00.
> All entries after it should be ignored, even though they are not marked
> as deleted. At least some EPOC releases (an OS used on Psion PDAs, for
> example) still use this policy.
> 

It's not just "old implementations" -- it's the spec.

After reaching a filename beginning with 00, no further data should be
assumed to be in that filesystem.  MS-DOS itself would only do that
when formatting the filesystem, so *all* the subsequent entries would
be assumed to start with 00, but that doesn't really seem to be to
spec.

	-hpa

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

"H. Peter Anvin" <hpa@zytor.com> writes:

> Followup to:  <20040126173949.GA788@frodo.local>
> By author:    Frodo Looijaard <frodol@dds.nl>
> In newsgroup: linux.dev.kernel
> > 
> > Hi folks,
> > 
> > I have created and attached a new version of my old-style FAT filesystem
> > patch, this time for the 2.6.0 kernel. It can also be found on
> > http://debian.frodo.looijaard.name/. 
> > 
> > Some old implementation of the FAT standard mark the end of the
> > directory file index by inserting a filename beginning with a byte 00.
> > All entries after it should be ignored, even though they are not marked
> > as deleted. At least some EPOC releases (an OS used on Psion PDAs, for
> > example) still use this policy.
> > 
> 
> It's not just "old implementations" -- it's the spec.
> 
> After reaching a filename beginning with 00, no further data should be
> assumed to be in that filesystem.  MS-DOS itself would only do that
> when formatting the filesystem, so *all* the subsequent entries would
> be assumed to start with 00, but that doesn't really seem to be to
> spec.

The new cluster for directory entries must be initialized by 0x00.
And, when the directory entry is deleted, the name[0] is updated by
0xe5 not 0x00.

So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.

The fat driver can stop at name[0] == 0x00, but this is just optimization.
The behavior shouldn't change by this.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[H. Peter Anvin]

OGAWA Hirofumi wrote:
> 
> The new cluster for directory entries must be initialized by 0x00.
> And, when the directory entry is deleted, the name[0] is updated by
> 0xe5 not 0x00.
> 
> So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.
> 
> The fat driver can stop at name[0] == 0x00, but this is just optimization.
> The behavior shouldn't change by this.

I looked at the spec, and yes, that is how the spec reads:

If DIR_Name[0] == 0x00, then the directory entry is free (same as for
0xE5), and there are no allocated directory entries after this one (all
of the DIR_Name[0] bytes in all of the entries after this one are also
set to 0). The special 0 value, rather than the 0xE5 value, indicates to
FAT file system driver code that the rest of the entries in this
directory do not need to be examined because they are all free.

I guess the original poster has found filesystems which have a 0
followed by garbage.  In cases like that, the cardinal rule for FAT is
WWDD (What Would DOS Do)... since I'm pretty sure DOS stops examining at
that point, we should do the same.

It's the same thing as with using 0xF8 for ending clusters; it's correct
according to spec, but WWDD says 0xFF is the right thing.

	-hpa

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

"H. Peter Anvin" <hpa@zytor.com> writes:

> OGAWA Hirofumi wrote:
> > 
> > The new cluster for directory entries must be initialized by 0x00.
> > And, when the directory entry is deleted, the name[0] is updated by
> > 0xe5 not 0x00.
> > 
> > So, if the name[0] is 0x00, it after, all bytes in cluster is 0x00.
> > 
> > The fat driver can stop at name[0] == 0x00, but this is just optimization.
> > The behavior shouldn't change by this.
> 
> I looked at the spec, and yes, that is how the spec reads:
> 
> If DIR_Name[0] == 0x00, then the directory entry is free (same as for
> 0xE5), and there are no allocated directory entries after this one (all
> of the DIR_Name[0] bytes in all of the entries after this one are also
> set to 0). The special 0 value, rather than the 0xE5 value, indicates to
> FAT file system driver code that the rest of the entries in this
> directory do not need to be examined because they are all free.
> 
> I guess the original poster has found filesystems which have a 0
> followed by garbage.  In cases like that, the cardinal rule for FAT is
> WWDD (What Would DOS Do)... since I'm pretty sure DOS stops examining at
> that point, we should do the same.
> 
> It's the same thing as with using 0xF8 for ending clusters; it's correct
> according to spec, but WWDD says 0xFF is the right thing.

The new cluster for directory entries must be initialized by 0x00.
This is required by spec.

If cluster has garbage, the fat driver needs to do such the following
part. Stop at DIR_Name[0] == 0 is not enough, and I don't think DOS
does this.

@@ -709,7 +731,20 @@
 		}
 
-		if (IS_FREE((*de)->name)) {
-			if (++row == slots)
+		if ((oldfat && ((*de)->name[0] == EOD_FLAG)))
+			last_entry = 1;
+		if (last_entry || IS_FREE((*de)->name)) {
+			if (++row == slots) {
+				if (last_entry) {
+					/* If we encounter a last_entry, we
+					 * need to mark the entry after the
+					 * one to be inserted as last_entry
+					 * now! */
+					if (fat_get_entry(dir,&curr,bh,de,i_pos) > -1) {
+						(*de)->name[0] = 0;
+						mark_inode_dirty(dir);
+					}
+				}
 				return offset;
+			}
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[Frodo Looijaard]

On Wed, Jan 28, 2004 at 05:17:45AM +0900, OGAWA Hirofumi wrote:
> "H. Peter Anvin" <hpa@zytor.com> writes:
> 
> > I guess the original poster has found filesystems which have a 0
> > followed by garbage.  

Exactly. At least on those EPOC filesystems, and I seem to have had a
old diskette around (perhaps out of the DR-DOS age?) which had the same
problem. Regrettably, the diskette got hosed.


> The new cluster for directory entries must be initialized by 0x00.
> This is required by spec.

Might be. In that case the EPOC filesystems do not conform to spec. Now
that would be a first time.
 

> If cluster has garbage, the fat driver needs to do such the following
> part. Stop at DIR_Name[0] == 0 is not enough, and I don't think DOS
> does this.

[Patch fragment to set DIR_Name[0] to 0 when new dir entries are added]

I don't know. I do not have a working MS-DOS partition anymore to test
this on.


As I said, I *think* it is safe to have my patch always applied (that
is, stop when DIR_Name[0] == 0, and be careful to add a new DIR_Name[0] = 0
entry when new entries are added at the back). It would conform to the
standard.  But I would not really be surprised if there was yet another
FAT implementation somewhere out there that breaks the standard in some
other subtle way, which works now but exhibits problems with my patch.
That is why I made it a mount option.

Thanks,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

Frodo Looijaard <frodol@dds.nl> writes:

> As I said, I *think* it is safe to have my patch always applied (that
> is, stop when DIR_Name[0] == 0, and be careful to add a new DIR_Name[0] = 0
> entry when new entries are added at the back). It would conform to the
> standard.  But I would not really be surprised if there was yet another
> FAT implementation somewhere out there that breaks the standard in some
> other subtle way, which works now but exhibits problems with my patch.
> That is why I made it a mount option.

"stop when DIR_Name[0] == 0" should be added after cleanup. The option
is not needed.

Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part.
The option is added easy, but it is not removed easy. And we must
maintain it.  (BTW, looks like that patch is buggy)

Those stuff makes a change of the normal path difficult really. At the
same reason, I removed the fat_cvf stuff.

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[Frodo Looijaard]

On Thu, Jan 29, 2004 at 05:08:32AM +0900, OGAWA Hirofumi wrote:
> 
> "stop when DIR_Name[0] == 0" should be added after cleanup. The option
> is not needed.

OK. That would be nice. Like I said, I just hope it does not break other
FAT implementations, but that is not very likely. At least, that would
allow read-only mounted EPOC FAT partitions to be handled correctly.
 
> Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part.
> The option is added easy, but it is not removed easy. And we must
> maintain it. 

I understand. I could always maintain that patch separately for those
who need it (for read-write mounted EPOC FAT partitions).

> (BTW, looks like that patch is buggy)

Could well be. Any suggestions what to look out for?

Thanks,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

Frodo Looijaard <frodol@dds.nl> writes:

> On Thu, Jan 29, 2004 at 05:08:32AM +0900, OGAWA Hirofumi wrote:
> > 
> > "stop when DIR_Name[0] == 0" should be added after cleanup. The option
> > is not needed.
> 
> OK. That would be nice. Like I said, I just hope it does not break other
> FAT implementations, but that is not very likely. At least, that would
> allow read-only mounted EPOC FAT partitions to be handled correctly.

This is spec as hpa saied. So this is right things, and other fat
driver also does, AFAIK.

> > Honestly, I wouldn't like to add the "add a new DIR_Name[0] = 0" part.
> > The option is added easy, but it is not removed easy. And we must
> > maintain it. 
> 
> I understand. I could always maintain that patch separately for those
> who need it (for read-write mounted EPOC FAT partitions).

Thanks.

> > (BTW, looks like that patch is buggy)
> 
> Could well be. Any suggestions what to look out for?

@@ -200,4 +202,8 @@
 parse_record:
 		long_slots = 0;
+		if (oldfat && (de->name[0] == EOD_FLAG)) 
+			last_entry = 1;
+		if (last_entry)
+			continue;
                        ^^^^^^^^
 		if (de->name[0] == DELETED_FLAG)
 			continue;

The above should "goto EODir;". Likewise, another "contiure" of
fat_search_long().


@@ -709,7 +731,20 @@
 		}
 
-		if (IS_FREE((*de)->name)) {
-			if (++row == slots)
+		if ((oldfat && ((*de)->name[0] == EOD_FLAG)))
+			last_entry = 1;
+		if (last_entry || IS_FREE((*de)->name)) {
+			if (++row == slots) {
+				if (last_entry) {
+					/* If we encounter a last_entry, we
+					 * need to mark the entry after the
+					 * one to be inserted as last_entry
+					 * now! */
+					if (fat_get_entry(dir,&curr,bh,de,i_pos) > -1) {
+						(*de)->name[0] = 0;
+						mark_inode_dirty(dir);

mark_inode_dirty(dir) is not needed, instead of it we should do
mark_buffer_dirty(bh).

And this fat_get_entry() updates bh and de, but it should be point to
allocated bh and de, not free entry. It's needed by msdos_add_entry().

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[Frodo Looijaard]

Hi,

I have run a small test in MS-DOS 5.00:

  1) Create a new directory
  2) Create five files in it
  3) Change the first character of the second filename to 0x00 with an editor
  4) Do a DIR listing: only one file is seen
     Linux shows four files!
  5) Create a new file
  6) Do a DIR listing: there are five files

So MS-DOS 5.00 at least does stop when a 0x00 marker is found, but does
not write new 0x00 markers when a 0x00 marker is overwritten. 

Thanks,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

Re: PATCH to access old-style FAT fs

[H. Peter Anvin]

Followup to:  <20040129105624.GA1072@frodo.local>
By author:    Frodo Looijaard <frodol@dds.nl>
In newsgroup: linux.dev.kernel
> 
> I have run a small test in MS-DOS 5.00:
> 
>   1) Create a new directory
>   2) Create five files in it
>   3) Change the first character of the second filename to 0x00 with an editor
>   4) Do a DIR listing: only one file is seen
>      Linux shows four files!
>   5) Create a new file
>   6) Do a DIR listing: there are five files
> 
> So MS-DOS 5.00 at least does stop when a 0x00 marker is found, but does
> not write new 0x00 markers when a 0x00 marker is overwritten. 
> 

This is of course dangerous, since it could create conflicts.

Thus, I suggest we do write new 0x00 markers; even though it's not
required (it only matters if the filesystem is out of spec and
therefore by definition corrupt), it would help the Psion, and
wouldn't cause any problems for in-spec filesystems.

	-hpa

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

"H. Peter Anvin" <hpa@zytor.com> writes:

> This is of course dangerous, since it could create conflicts.
> 
> Thus, I suggest we do write new 0x00 markers; even though it's not
> required (it only matters if the filesystem is out of spec and
> therefore by definition corrupt), it would help the Psion, and
> wouldn't cause any problems for in-spec filesystems.

Yes, that cluster is dangerous. However, repair should be fsck's
job. IMO, for repair we shouldn't add unneeded I/O to normal path.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[Frodo Looijaard]

[-- Attachment #1: Type: text/plain, Size: 1499 bytes --]

Hi,

I have attached a newer, better behaving version of my patch:
  * Implements new mount option oldfat for FAT-derived filesystems.
  * Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
  * Writes a 0 to the next entry DIR_Name[0] when overwriting an entry
    which has DIR_Name[0] = 0 when oldfat is set

It has been tested with both msdos and vfat filesystems and seems to
work well (unlike the patch of a few days ago, which had some issues).

Once you get around to cleaning up the code and making the
stop-scanning-dirs-on-zero the default way, the patch can be shrunken to
include only the third item.

> The above should "goto EODir;". Likewise, another "contiure" of
> fat_search_long().

I am not convinced that the goto is always safe, and I am pretty sure
that it works now (though not as efficient as possible, perhaps), so I
have left that in place for now.

> mark_inode_dirty(dir) is not needed, instead of it we should do
> mark_buffer_dirty(bh).
> 
> And this fat_get_entry() updates bh and de, but it should be point to
> allocated bh and de, not free entry. It's needed by msdos_add_entry().

Both are now implemented using the new fat_write_zero_entry function.

Thanks for your help,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

[-- Attachment #2: linux-fat-2.6.0.patch-20040129.gz --]
[-- Type: application/octet-stream, Size: 1704 bytes --]

Re: PATCH to access old-style FAT fs

[H. Peter Anvin]

Frodo Looijaard wrote:
> Hi,
> 
> I have attached a newer, better behaving version of my patch:
>   * Implements new mount option oldfat for FAT-derived filesystems.
>   * Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
>   * Writes a 0 to the next entry DIR_Name[0] when overwriting an entry
>     which has DIR_Name[0] = 0 when oldfat is set
> 

Please don't call this "oldfat".  There is nothing about this which is
"old-style"; it's a workaround for a bug in a specific OS.

	-hpa

Re: PATCH to access old-style FAT fs

[Andries Brouwer]

On Thu, Jan 29, 2004 at 11:39:44PM +0100, Frodo Looijaard wrote:

> I have attached a newer, better behaving version of my patch:
>   * Implements new mount option oldfat for FAT-derived filesystems.
>   * Stops scanning dirs when DIR_Name[0] = 0 when oldfat is set
>   * Writes a 0 to the next entry DIR_Name[0] when overwriting an entry
>     which has DIR_Name[0] = 0 when oldfat is set

Just amused myself checking all data I can find about such stuff.
And read the replies by Hirofumi and hpa.
I completely agree with them.

The DOS situation is not so much that 0 is an end-of-dir marker.
It marks that this entry, and all following entries, has never
been used.

For the time being there is no justification for "oldfat".
The only place where this behaviour has been reported (not only by you,
I saw several references) is on Psion PDAs. So "psion" would be a more
obvious name, if a mount option is needed.

And maybe no mount option is needed.
Your 2nd and 3rd points can be done always, I think, unless our FAT code
should mimic the DOS FAT behaviour (on non-DOS filesystems).

Andries

Re: PATCH to access old-style FAT fs

[OGAWA Hirofumi]

Frodo Looijaard <frodol@dds.nl> writes:

> > The above should "goto EODir;". Likewise, another "contiure" of
> > fat_search_long().
> 
> I am not convinced that the goto is always safe, and I am pretty sure
> that it works now (though not as efficient as possible, perhaps), so I
> have left that in place for now.

But, if there is directory entries after 0, you must not allow access
to those entries. Because those invalid entries may have reference to
the valid data block.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: PATCH to access old-style FAT fs

[Frodo Looijaard]

[-- Attachment #1: Type: text/plain, Size: 892 bytes --]

Hi folks,

another update of the patch, which addresses some of the reactions and 
concerns which I got in reply:

  * The mount option is now called `epoc' instead of `oldfat'
  * fat_search_long now jumps directly to EODir when it finds a
    zero-marked entry (it took some hard looking, but I belief this
    is safe).
  * fat_readdirx logic is slightly simplified (but can't jump to EODir
    on a zero-marked entry, regrettably).

Thanks for your feedback!

I'll try to keep this patch up-to-date with new 2.6 kernels. The latest
version can always be found on http://debian.frodo.looijaard.name/

Thanks,
  Frodo

-- 
Frodo Looijaard <frodol@dds.nl>  PGP key and more: http://huizen.dds.nl/~frodol
Defenestration n. (formal or joc.):
  The act of removing Windows from your computer in disgust, usually followed
  by the installation of Linux or some other Unix-like operating system.

[-- Attachment #2: linux-fat-epoc-2.6.0.patch-20040201.gz --]
[-- Type: application/octet-stream, Size: 1706 bytes --]