FAT-filesystem EOF marker problem

FAT-filesystem EOF marker problem

[Randy.Dunlap]

Hi,

On 2002-Sept-15 a Linux-USB user reported a problem when using
a file written by Linux to a CompactFlash (CF) card in an MP3
player.  Linux and Windows can still read the CF card correctly,
but the MP3 player cannot.  I have successfully reproduced this
problem.

Initial problem report and thread begin at:
http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2

I found several Boot Record/BIOS Parameter Block differences,
one FAT cluster link difference (EOF marker), and several
directory entry differences.  I believe that I have proved
that the problem lies in the FAT EOF marker difference:

Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
mtools writes 0xfff for a FAT12 EOF marker.
Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
(Yes, I know that 0xff8 is a valid FAT12 EOF marker.)

This causes some cheap/crappy MP3 player(s) firmware to get lost...
yeah, it's a problem with the MP3 player(s).
And Linux users can get around the problem by using mtools
instead of a Linux filesystem, but sometimes using a filesystem
is just preferred.

Anyone have any suggestions for other workarounds or patches?

I would prefer to see something like (a) Linux uses 0xfff for
FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
(b) Linux uses a mount option to indicate the EOF marker value,
or (c) Linux dynamically checks what EOF marker value is in
the FAT already, and continues to use that value [and continues
to use 0xff8 as its default value].

Thanks,
~Randy

Re: FAT-filesystem EOF marker problem

[OGAWA Hirofumi]

"Randy.Dunlap" <randy.dunlap@verizon.net> writes:

> Hi,
> 
> On 2002-Sept-15 a Linux-USB user reported a problem when using
> a file written by Linux to a CompactFlash (CF) card in an MP3
> player.  Linux and Windows can still read the CF card correctly,
> but the MP3 player cannot.  I have successfully reproduced this
> problem.
> 
> Initial problem report and thread begin at:
> http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2
> 
> I found several Boot Record/BIOS Parameter Block differences,
> one FAT cluster link difference (EOF marker), and several
> directory entry differences.  I believe that I have proved
> that the problem lies in the FAT EOF marker difference:
> 
> Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
> mtools writes 0xfff for a FAT12 EOF marker.
> Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
> (Yes, I know that 0xff8 is a valid FAT12 EOF marker.)
> 
> This causes some cheap/crappy MP3 player(s) firmware to get lost...
> yeah, it's a problem with the MP3 player(s).
> And Linux users can get around the problem by using mtools
> instead of a Linux filesystem, but sometimes using a filesystem
> is just preferred.
> 
> Anyone have any suggestions for other workarounds or patches?
> 
> I would prefer to see something like (a) Linux uses 0xfff for
> FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
> (b) Linux uses a mount option to indicate the EOF marker value,
> or (c) Linux dynamically checks what EOF marker value is in
> the FAT already, and continues to use that value [and continues
> to use 0xff8 as its default value].

I vote to (a). It should be very easy for 2.5.x. And I think that you
explained why we should use standard EOF mark.

So, what do you think of trying (a), until the reason we need values
other than standard EOF(fff/ffff/fffffff) is found?

Thanks.
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: FAT-filesystem EOF marker problem

[Randy.Dunlap]

On Mon, 23 Sep 2002, OGAWA Hirofumi wrote:

| "Randy.Dunlap" <randy.dunlap@verizon.net> writes:
|
| > Hi,
| >
| > On 2002-Sept-15 a Linux-USB user reported a problem when using
| > a file written by Linux to a CompactFlash (CF) card in an MP3
| > player.  Linux and Windows can still read the CF card correctly,
| > but the MP3 player cannot.  I have successfully reproduced this
| > problem.
| >
| > Initial problem report and thread begin at:
| > http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2
| >
| > I found several Boot Record/BIOS Parameter Block differences,
| > one FAT cluster link difference (EOF marker), and several
| > directory entry differences.  I believe that I have proved
| > that the problem lies in the FAT EOF marker difference:
| >
| > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
| > mtools writes 0xfff for a FAT12 EOF marker.
| > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
| > (Yes, I know that 0xff8 is a valid FAT12 EOF marker.)
| >
| > This causes some cheap/crappy MP3 player(s) firmware to get lost...
| > yeah, it's a problem with the MP3 player(s).
| > And Linux users can get around the problem by using mtools
| > instead of a Linux filesystem, but sometimes using a filesystem
| > is just preferred.
| >
| > Anyone have any suggestions for other workarounds or patches?
| >
| > I would prefer to see something like (a) Linux uses 0xfff for
| > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
| > (b) Linux uses a mount option to indicate the EOF marker value,
| > or (c) Linux dynamically checks what EOF marker value is in
| > the FAT already, and continues to use that value [and continues
| > to use 0xff8 as its default value].
|
| I vote to (a). It should be very easy for 2.5.x. And I think that you
| explained why we should use standard EOF mark.
|
| So, what do you think of trying (a), until the reason we need values
| other than standard EOF(fff/ffff/fffffff) is found?

Sure, that sounds fine to me.  I expect it to be fairly safe.
And if we did (c), we should use 0xfff instead of 0xff8 as the
default now that I've had more time to think about it.

-- 
~Randy

Re: FAT-filesystem EOF marker problem

[Andries Brouwer]

On Sat, Sep 21, 2002 at 10:30:19PM -0700, Randy.Dunlap wrote:

> On 2002-Sept-15 a Linux-USB user reported a problem when using
> a file written by Linux to a CompactFlash (CF) card in an MP3
> player.  Linux and Windows can still read the CF card correctly,
> but the MP3 player cannot.  I have successfully reproduced this
> problem.
> 
> Initial problem report and thread begin at:
> http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2

There seem to be many more people with this problem.
One web page describing it is
http://home.earthlink.net/~ebrombaugh/fattools.html

> Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
> mtools writes 0xfff for a FAT12 EOF marker.
> Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.

> I would prefer to see something like (a) Linux uses 0xfff for
> FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
> (b) Linux uses a mount option to indicate the EOF marker value,
> or (c) Linux dynamically checks what EOF marker value is in
> the FAT already, and continues to use that value [and continues
> to use 0xff8 as its default value].

My docs say: FAT entry 0 contains the media descriptor byte.
FAT entry 1 contains the EOF marker, except that the high order two
bits can be used for other purposes.

So, I would like to reject (b) - if a device does not work, very few
people will get the idea that the end-of-file marker used by the
FAT filesystem is to blame. And (a) is reasonable, that is what
Microsoft does. And (c) might be replaced by (c'): read the desired
EOF marker from FAT entry 1. If that yields an invalid value, use
what Microsoft does: 0xfff.

Andries

Re: FAT-filesystem EOF marker problem

[Randy.Dunlap]

On Mon, 23 Sep 2002, Andries Brouwer wrote:

| On Sat, Sep 21, 2002 at 10:30:19PM -0700, Randy.Dunlap wrote:
|
| > On 2002-Sept-15 a Linux-USB user reported a problem when using
| > a file written by Linux to a CompactFlash (CF) card in an MP3
| > player.  Linux and Windows can still read the CF card correctly,
| > but the MP3 player cannot.  I have successfully reproduced this
| > problem.
| >
| > Initial problem report and thread begin at:
| > http://marc.theaimsgroup.com/?l=linux-usb-users&m=103210507227709&w=2
|
| There seem to be many more people with this problem.
| One web page describing it is
| http://home.earthlink.net/~ebrombaugh/fattools.html

Hm, OK, thanks for the web search that I didn't do...

| > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
| > mtools writes 0xfff for a FAT12 EOF marker.
| > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
|
| > I would prefer to see something like (a) Linux uses 0xfff for
| > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
| > (b) Linux uses a mount option to indicate the EOF marker value,
| > or (c) Linux dynamically checks what EOF marker value is in
| > the FAT already, and continues to use that value [and continues
| > to use 0xff8 as its default value].
|
| My docs say: FAT entry 0 contains the media descriptor byte.
| FAT entry 1 contains the EOF marker, except that the high order two
| bits can be used for other purposes.

And the value that I see there is 0xfff.

| So, I would like to reject (b) - if a device does not work, very few
| people will get the idea that the end-of-file marker used by the
| FAT filesystem is to blame. And (a) is reasonable, that is what
| Microsoft does. And (c) might be replaced by (c'): read the desired
| EOF marker from FAT entry 1. If that yields an invalid value, use
| what Microsoft does: 0xfff.

Thanks agagin.  Most of my FAT specs are long gone and I
didn't find this particular nugget on the web.
I vote for option (c') in this case.

-- 
~Randy

Re: FAT-filesystem EOF marker problem

[OGAWA Hirofumi]

"Randy.Dunlap" <rddunlap@osdl.org> writes:

> | > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
> | > mtools writes 0xfff for a FAT12 EOF marker.
> | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
> |
> | > I would prefer to see something like (a) Linux uses 0xfff for
> | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
> | > (b) Linux uses a mount option to indicate the EOF marker value,
> | > or (c) Linux dynamically checks what EOF marker value is in
> | > the FAT already, and continues to use that value [and continues
> | > to use 0xff8 as its default value].
> |
> | My docs say: FAT entry 0 contains the media descriptor byte.
> | FAT entry 1 contains the EOF marker, except that the high order two
> | bits can be used for other purposes.
> 
> And the value that I see there is 0xfff.
> 
> | So, I would like to reject (b) - if a device does not work, very few
> | people will get the idea that the end-of-file marker used by the
> | FAT filesystem is to blame. And (a) is reasonable, that is what
> | Microsoft does. And (c) might be replaced by (c'): read the desired
> | EOF marker from FAT entry 1. If that yields an invalid value, use
> | what Microsoft does: 0xfff.
> 
> Thanks agagin.  Most of my FAT specs are long gone and I
> didn't find this particular nugget on the web.
> I vote for option (c') in this case.

Umm, Microsoft use 0xfff always, right? And, since entry 1 is set by
format command, I worry that this problem may happen again when format
command setted value other than 0xfff.

What do you think of above case?
-- 
OGAWA Hirofumi <hirofumi@mail.parknet.co.jp>

Re: FAT-filesystem EOF marker problem

[Randy.Dunlap]

On Mon, 23 Sep 2002, OGAWA Hirofumi wrote:

| "Randy.Dunlap" <rddunlap@osdl.org> writes:
|
| > | > Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
| > | > mtools writes 0xfff for a FAT12 EOF marker.
| > | > Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
| > |
| > | > I would prefer to see something like (a) Linux uses 0xfff for
| > | > FAT(12) EOF [and sign-extended for FAT16 and FAT32], or
| > | > (b) Linux uses a mount option to indicate the EOF marker value,
| > | > or (c) Linux dynamically checks what EOF marker value is in
| > | > the FAT already, and continues to use that value [and continues
| > | > to use 0xff8 as its default value].
| > |
| > | My docs say: FAT entry 0 contains the media descriptor byte.
| > | FAT entry 1 contains the EOF marker, except that the high order two
| > | bits can be used for other purposes.
| >
| > And the value that I see there is 0xfff.
| >
| > | So, I would like to reject (b) - if a device does not work, very few
| > | people will get the idea that the end-of-file marker used by the
| > | FAT filesystem is to blame. And (a) is reasonable, that is what
| > | Microsoft does. And (c) might be replaced by (c'): read the desired
| > | EOF marker from FAT entry 1. If that yields an invalid value, use
| > | what Microsoft does: 0xfff.
| >
| > Thanks agagin.  Most of my FAT specs are long gone and I
| > didn't find this particular nugget on the web.
| > I vote for option (c') in this case.
|
| Umm, Microsoft use 0xfff always, right? And, since entry 1 is set by
| format command, I worry that this problem may happen again when format
| command setted value other than 0xfff.
|
| What do you think of above case?

Um, I guess we can say that Windows ME uses 0xfff always, at least
one version of it.  Don't know about Windows 98 or 98 SE or 2000
or XP etc.  But since FAT entry 1 tells the FAT filesystem what
the EOF value is, it shouldn't matter if it changes from one
filesystem to another, should it?  I don't see a problem with
using the value in fat_entry[1].  Just make sure that it's valid
(0xff8 - 0xfff for FAT12).

Oh, I see.  I'm assuming that this can be done on a per-filesystem
basis instead of one global EOF value for all mounted fat filesystems.
Does that make more sense?  Is it feasible?
I don't see why it couldn't be done this way.

-- 
~Randy

Re: FAT-filesystem EOF marker problem

[H. Peter Anvin]

Followup to:  <3D8D556B.4D3BF8B7@verizon.net>
By author:    "Randy.Dunlap" <randy.dunlap@verizon.net>
In newsgroup: linux.dev.fs.devel
> 
> Windows (ME version at least) writes 0xfff for a FAT12 EOF marker.
> mtools writes 0xfff for a FAT12 EOF marker.
> Linux (v)fat filesystem writes 0xff8 for a FAT12 EOF marker.
> (Yes, I know that 0xff8 is a valid FAT12 EOF marker.)
> 

Valid, perhaps (the official specification is fuzzy on that subject to
say the least), but 0xfff is the standard one and what DOS uses.

The official spec states:

Note that the cluster number whose cluster entry in the FAT contains
the EOC mark is allocated to the file and is also the last cluster
allocated to the file. Microsoft operating system FAT drivers use the
EOC value 0x0FFF for FAT12, 0xFFFF for FAT16, and 0x0FFFFFFF for FAT32
when they set the contents of a cluster to the EOC mark. There are
various disk utilities for Microsoft operating systems that use a
different value, however.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: FAT-filesystem EOF marker problem

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.33L2.0209222319590.20873-100000@dragon.pdx.osdl.net>
By author:    "Randy.Dunlap" <rddunlap@osdl.org>
In newsgroup: linux.dev.fs.devel
> |
> | My docs say: FAT entry 0 contains the media descriptor byte.
> | FAT entry 1 contains the EOF marker, except that the high order two
> | bits can be used for other purposes.
> 
> And the value that I see there is 0xfff.
> 

Fat entry 0 contains the media descriptor byte, and fat entry 1 *used*
to contain the EOC mark, but has been retconned as the volume dirty
bit (Microsoft OSs don't recognize a volume dirty flag for FAT12,
however; this is presumably because of the MS assumption that FAT12 ==
floppy.):

For FAT16: ClnShutBitMask = 0x8000; HrdErrBitMask = 0x4000;
For FAT32: ClnShutBitMask = 0x08000000; HrdErrBitMask = 0x04000000;

Bit ClnShutBitMask   If bit is 1, volume is  clean . If bit is 0,
volume is  dirty . This indicates that the file system driver did not
Dismount the volume properly the last time it had the volume
mounted. It would be a good idea to run a Chkdsk/Scandisk disk repair
utility on it, because it may be damaged.

Bit HrdErrBitMask   If this bit is 1, no disk read/write errors were
encountered. If this bit is 0, the file system driver encountered a
disk I/O error on the Volume the last time it was mounted, which is an
indicator that some sectors may have gone bad on the volume. It would
be a good idea to run a Chkdsk/Scandisk disk repair utility that does
surface analysis on it to look for new bad sectors.

Either way, I vote we should always use all ones of the EOC mark.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: FAT-filesystem EOF marker problem

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.33L2.0209231103021.22676-100000@dragon.pdx.osdl.net>
By author:    "Randy.Dunlap" <rddunlap@osdl.org>
In newsgroup: linux.dev.fs.devel
> 
> Um, I guess we can say that Windows ME uses 0xfff always, at least
> one version of it.  Don't know about Windows 98 or 98 SE or 2000
> or XP etc.  But since FAT entry 1 tells the FAT filesystem what
> the EOF value is, it shouldn't matter if it changes from one
> filesystem to another, should it?  I don't see a problem with
> using the value in fat_entry[1].  Just make sure that it's valid
> (0xff8 - 0xfff for FAT12).
> 

ALL versions of DOS since 1.0 uses all ones for the EOC mark.

Under the principle "be liberal in what you receive, conservative in
what you send" we should recognize anything >= -8 as EOC, but write
-1.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: FAT-filesystem EOF marker problem

[H. Peter Anvin]

By the way, I have posted a copy of the offical FAT spec PDF at:

	http://www.zytor.com/~hpa/FatFormat.pdf

... since I can't find the original URL anymore.

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: FAT-filesystem EOF marker problem

[Andries Brouwer]

On Tue, Sep 24, 2002 at 12:34:22PM -0700, H. Peter Anvin wrote:

> > | My docs say: FAT entry 0 contains the media descriptor byte.
> > | FAT entry 1 contains the EOF marker, except that the high order two
> > | bits can be used for other purposes.
> > 
> > And the value that I see there is 0xfff.
> 
> Fat entry 0 contains the media descriptor byte, and fat entry 1 *used*
> to contain the EOC mark, but has been retconned as the volume dirty bit

Yes, I know.
(As I see it, it still holds the EOC mark, but only the low order bits
are of interest, so the top two bits have a different meaning now.)

Let me mention
	http://www.win.tue.nl/~aeb/linux/fs/fat/fat.html
(my notes on the FAT filesystem).

If anyone has corrections or additions I'd like to hear.

Andries


P.S. I checked some old stuff but did not find anything with a EOC
other than 0x*fff.  I would be interested in hearing about which
non-Linux programs used other values.

Re: FAT-filesystem EOF marker problem

[Andries Brouwer]

On Tue, Sep 24, 2002 at 12:40:17PM -0700, H. Peter Anvin wrote:
> By the way, I have posted a copy of the offical FAT spec PDF at:
> 
> 	http://www.zytor.com/~hpa/FatFormat.pdf
> 
> ... since I can't find the original URL anymore.

ERROR 403: Forbidden.

But maybe it is identical to

	http://www.win.tue.nl/~aeb/linux/fs/fat/fatgen102.pdf

?

Re: FAT-filesystem EOF marker problem

[Randy.Dunlap]

On 24 Sep 2002, H. Peter Anvin wrote:

| Followup to:  <Pine.LNX.4.33L2.0209231103021.22676-100000@dragon.pdx.osdl.net>
| By author:    "Randy.Dunlap" <rddunlap@osdl.org>
| In newsgroup: linux.dev.fs.devel
| >
| > Um, I guess we can say that Windows ME uses 0xfff always, at least
| > one version of it.  Don't know about Windows 98 or 98 SE or 2000
| > or XP etc.  But since FAT entry 1 tells the FAT filesystem what
| > the EOF value is, it shouldn't matter if it changes from one
| > filesystem to another, should it?  I don't see a problem with
| > using the value in fat_entry[1].  Just make sure that it's valid
| > (0xff8 - 0xfff for FAT12).
| >
|
| ALL versions of DOS since 1.0 uses all ones for the EOC mark.
|
| Under the principle "be liberal in what you receive, conservative in
| what you send" we should recognize anything >= -8 as EOC, but write
| -1.

My view exactly.

-- 
~Randy

Re: FAT-filesystem EOF marker problem

[H. Peter Anvin]

Andries Brouwer wrote:
> On Tue, Sep 24, 2002 at 12:40:17PM -0700, H. Peter Anvin wrote:
> 
>>By the way, I have posted a copy of the offical FAT spec PDF at:
>>
>>	http://www.zytor.com/~hpa/FatFormat.pdf
>>
>>... since I can't find the original URL anymore.
> 
> 
> ERROR 403: Forbidden.

Fixed.

> But maybe it is identical to
> 
> 	http://www.win.tue.nl/~aeb/linux/fs/fat/fatgen102.pdf
> 

Indeed it is.

	-hpa