Incorrect handling of . and .. files

Incorrect handling of . and .. files

[Sean Young]

Hi,

Windows allows files and directories called "." and ".." to be created
using UNC paths, i.e. "\\?\D:\..". Now this is totally insane behaviour,
but when an exfat filesytem with such a file is mounted on Linux, those
files show up as another directory and its contents is inaccessible.

I can replicate this using exfat filesystems, but not ntfs.

This is what I did on Windows using rust:

	use std::fs::File;
	use std::io::Write;

	fn main() {
	    let mut file =
		File::create(r"\\?\D:\..").expect("create dot file");
	    file.write_all(b"Hello, world!").expect("write dot file");
	}

Now on Linux (I also created a file called ".").

[root@xywoleh tmp]# mount -t exfat /dev/loop0p1 /mnt
[root@xywoleh tmp]# cd /mnt
[root@xywoleh mnt]# ls -la
total 20
drwxr-xr-x. 5 root root 4096 Sep 27 11:47  .
drwxr-xr-x. 5 root root 4096 Sep 27 11:47  .
dr-xr-xr-x. 1 root root  176 Sep 21 11:05  ..
dr-xr-xr-x. 1 root root  176 Sep 21 11:05  ..
drwxr-xr-x. 2 root root 4096 Sep 27  2021 '$RECYCLE.BIN'
drwxr-xr-x. 2 root root 4096 Sep 27  2021 'System Volume Information'

Microsoft says this:

https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#win32-file-namespaces

	Because it turns off automatic expansion of the path string, the
	"\\?\" prefix also allows the use of ".." and "." in the path names,
	which can be useful if you are attempting to perform operations on a
	file with these otherwise reserved relative path specifiers as part of
	the fully qualified path.

So, in Linux cannot read "." or ".." (i.e., I can't see "Hello, World!"). I
don't know what the correct handling should be, but having two "." and two
".." files does not seem right at all.

Thanks,
Sean

Re: Incorrect handling of . and .. files

[Pali Rohár]

Hello!

I tried to find some information what is allowed and what not.

On Monday 27 September 2021 12:19:48 Sean Young wrote:
> Hi,
> 
> Windows allows files and directories called "." and ".." to be created
> using UNC paths, i.e. "\\?\D:\..". Now this is totally insane behaviour,
> but when an exfat filesytem with such a file is mounted on Linux, those
> files show up as another directory and its contents is inaccessible.
> 
> I can replicate this using exfat filesystems, but not ntfs.

Microsoft exFAT specification explicitly disallow "." and "..", see:

https://docs.microsoft.com/en-us/windows/win32/fileio/exfat-specification#773-filename-field

  The file names "." and ".." have the special meaning of "this
  directory" and "containing directory", respectively. Implementations
  shall not record either of these reserved file names in the FileName
  field.

On the other hand Microsoft FAT32 specification can be understood that
file may have long name (vfat) set to "." or ".." but not short name.

https://download.microsoft.com/download/0/8/4/084c452b-b772-4fe5-89bb-a0cbf082286a/fatgen103.doc (section Long Directory Entries)

  The characters may be any combination of those defined for short names
  with the addition of the period (.) character used multiple times
  within the long name.

Microsoft NTFS specification is not available.

OSTA UDF 2.60 specification does not disallow "." and ".." entries, but
specifies what UNIX operating system should do with invalid file names:

http://osta.org/specs/pdf/udf260.pdf (section 4.2.2.1.5 UNIX)

  A FileIdentifier that contains characters considered invalid within a
  UNIX file name for the current system environment, or not displayable
  in the current environment shall have them translated into “_” (#005E).

By default non-multi-volume Volume Set on UDF use Interchange Level 2
which do not have defined any restriction for File Identifier in UDF
reference ECMA 167 specification. Interchange Level 1 has following
restrictions:

https://www.ecma-international.org/publications-and-standards/standards/ecma-167/ (section 4/15.1)

  A sequence of fileid-characters shall be a sequence of d-characters
  (1/7.2) excluding SPACE, COMMA, FULL STOP and REVERSE SOLIDUS
  characters except as part of a code extension character (see 1/7.2.9.1).

So it means that "." and ".." entries could be stored on disk as valid
file names.

> This is what I did on Windows using rust:
> 
> 	use std::fs::File;
> 	use std::io::Write;
> 
> 	fn main() {
> 	    let mut file =
> 		File::create(r"\\?\D:\..").expect("create dot file");
> 	    file.write_all(b"Hello, world!").expect("write dot file");
> 	}
> 
> Now on Linux (I also created a file called ".").
> 
> [root@xywoleh tmp]# mount -t exfat /dev/loop0p1 /mnt
> [root@xywoleh tmp]# cd /mnt
> [root@xywoleh mnt]# ls -la
> total 20
> drwxr-xr-x. 5 root root 4096 Sep 27 11:47  .
> drwxr-xr-x. 5 root root 4096 Sep 27 11:47  .
> dr-xr-xr-x. 1 root root  176 Sep 21 11:05  ..
> dr-xr-xr-x. 1 root root  176 Sep 21 11:05  ..
> drwxr-xr-x. 2 root root 4096 Sep 27  2021 '$RECYCLE.BIN'
> drwxr-xr-x. 2 root root 4096 Sep 27  2021 'System Volume Information'
> 
> Microsoft says this:
> 
> https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#win32-file-namespaces
> 
> 	Because it turns off automatic expansion of the path string, the
> 	"\\?\" prefix also allows the use of ".." and "." in the path names,
> 	which can be useful if you are attempting to perform operations on a
> 	file with these otherwise reserved relative path specifiers as part of
> 	the fully qualified path.
> 
> So, in Linux cannot read "." or ".." (i.e., I can't see "Hello, World!"). I
> don't know what the correct handling should be, but having two "." and two
> ".." files does not seem right at all.

This is really a bug in Linux kernel. It should not export "." and ".."
into VFS even when filesystem disk format supports such insane file
names.

For FAT32 we could show short file name instead of long name as short
name alias has explicitly disallowed "." and ".." names. I think that
NTFS has optional support for DOS short name alias. UDF specification
defines translation to "_" but it just opens a new question how to deal
with conflicts with real file name "_".

So either Linux needs to completely hide these insane file names from
VFS or translate them to something which do not conflict with other
files in correct directory.

I guess that hiding them for exfat is valid thing as Microsoft
specification explicitly disallow them. Probably fsck.exfat can be teach
to rename these files and/or put them to lost+found directory.

I'm really surprised that some filesystems specifications really allow
such insane file names and also the fact that there is an implementation
(Windows kernel) which allows creating them and it is even documented.

> Thanks,
> Sean

Re: Incorrect handling of . and .. files

[Adam Borowski]

On Sat, Dec 11, 2021 at 03:04:53AM +0100, Pali Rohár wrote:
> I tried to find some information what is allowed and what not.
> 
> On Monday 27 September 2021 12:19:48 Sean Young wrote:
> > Windows allows files and directories called "." and ".." to be created
> > using UNC paths, i.e. "\\?\D:\..". Now this is totally insane behaviour,
> > but when an exfat filesytem with such a file is mounted on Linux, those
> > files show up as another directory and its contents is inaccessible.
> > 
> > I can replicate this using exfat filesystems, but not ntfs.
> 
> Microsoft exFAT specification explicitly disallow "." and "..", see:
[...]
> On the other hand Microsoft FAT32 specification can be understood that
> file may have long name (vfat) set to "." or ".." but not short name.
[...]
> OSTA UDF 2.60 specification does not disallow "." and ".." entries, but
[...]
> So it means that "." and ".." entries could be stored on disk as valid
> file names.

It doesn't matter one whit what the specification says.  Anyone with a disk
editor can craft a filesystem containing filenames such as "." or "..", "/"
"foo/bar" or anything else we would like to ban.

> > So, in Linux cannot read "." or ".." (i.e., I can't see "Hello, World!"). I
> > don't know what the correct handling should be, but having two "." and two
> > ".." files does not seem right at all.
> 
> This is really a bug in Linux kernel. It should not export "." and ".."
> into VFS even when filesystem disk format supports such insane file
> names.

This.

Otherwise, every filesystem driver would need to contain redundant code for
checking for such bad names.

> So either Linux needs to completely hide these insane file names from
> VFS or translate them to something which do not conflict with other
> files in correct directory.

Escaping bad names has the problem of the escaped name also possibly
existing -- perhaps even recursively.  Plus, the filesystem might be using
hashed or tree indices which could go wrong if a name is altered.

But then, I once proposed (and I'm pondering reviving) a ban for characters
\x01..\x1f and possibly others, and if banned, they can still legitimately
occur in old filesystems.

> I guess that hiding them for exfat is valid thing as Microsoft
> specification explicitly disallow them. Probably fsck.exfat can be teach
> to rename these files and/or put them to lost+found directory.

fsck fixing those is a good thing but we still need to handle them at
runtime.


Meow!
-- 
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
⠈⠳⣄⠀⠀⠀⠀

Re: Incorrect handling of . and .. files

[Pali Rohár]

On Monday 13 December 2021 07:47:44 Adam Borowski wrote:
> On Sat, Dec 11, 2021 at 03:04:53AM +0100, Pali Rohár wrote:
> > I tried to find some information what is allowed and what not.
> > 
> > On Monday 27 September 2021 12:19:48 Sean Young wrote:
> > > Windows allows files and directories called "." and ".." to be created
> > > using UNC paths, i.e. "\\?\D:\..". Now this is totally insane behaviour,
> > > but when an exfat filesytem with such a file is mounted on Linux, those
> > > files show up as another directory and its contents is inaccessible.
> > > 
> > > I can replicate this using exfat filesystems, but not ntfs.
> > 
> > Microsoft exFAT specification explicitly disallow "." and "..", see:
> [...]
> > On the other hand Microsoft FAT32 specification can be understood that
> > file may have long name (vfat) set to "." or ".." but not short name.
> [...]
> > OSTA UDF 2.60 specification does not disallow "." and ".." entries, but
> [...]
> > So it means that "." and ".." entries could be stored on disk as valid
> > file names.
> 
> It doesn't matter one whit what the specification says.  Anyone with a disk
> editor can craft a filesystem containing filenames such as "." or "..", "/"
> "foo/bar" or anything else we would like to ban.

That is truth. But question is what should do fsck tools with such file
names on filesystems where "." and ".." are permitted? Fully valid
argument is "do not touch them" because there is nothing bad with these
names.

> > > So, in Linux cannot read "." or ".." (i.e., I can't see "Hello, World!"). I
> > > don't know what the correct handling should be, but having two "." and two
> > > ".." files does not seem right at all.
> > 
> > This is really a bug in Linux kernel. It should not export "." and ".."
> > into VFS even when filesystem disk format supports such insane file
> > names.
> 
> This.
> 
> Otherwise, every filesystem driver would need to contain redundant code for
> checking for such bad names.
> 
> > So either Linux needs to completely hide these insane file names from
> > VFS or translate them to something which do not conflict with other
> > files in correct directory.
> 
> Escaping bad names has the problem of the escaped name also possibly
> existing -- perhaps even recursively.  Plus, the filesystem might be using
> hashed or tree indices which could go wrong if a name is altered.

vfat has already own escaping scheme and it is documented in mount(8)
manpage. Invalid characters are translated either to fixed char '?' or
to ':'... esc sequence if uni_xlate mount option is used. But it looks
like that that kernel vfat driver do not have these two entries "." and
".." in its blacklist.

And, another important thing about vfat is that it has two file names
for each file. One short 8.3 and one long vfat. Short 8.3 do not allow
"." or "..", so another possibility how to handle this issue for vfat is
to show short 8.3 name in VFS when long is invalid.

For UDF case, specification already says how to handle problematic
file names, so I think that udf.ko could implement it according to
specification.

But for all other filesystems it is needed to do something ideally on
VFS layer.

What about generating some deterministic / predicable file names which
will not conflict with other file names in current directory for these
problematic files?

> But then, I once proposed (and I'm pondering reviving) a ban for characters
> \x01..\x1f and possibly others, and if banned, they can still legitimately
> occur in old filesystems.
> 
> > I guess that hiding them for exfat is valid thing as Microsoft
> > specification explicitly disallow them. Probably fsck.exfat can be teach
> > to rename these files and/or put them to lost+found directory.
> 
> fsck fixing those is a good thing but we still need to handle them at
> runtime.

Namjae Jeon, would you be able to implement fixing of such filenames in
fsck.exfat tool?

> 
> Meow!
> -- 
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
> ⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
> ⠈⠳⣄⠀⠀⠀⠀

Re: Incorrect handling of . and .. files

[Namjae Jeon]

2021-12-13 20:39 GMT+09:00, Pali Rohár <pali@kernel.org>:
> On Monday 13 December 2021 07:47:44 Adam Borowski wrote:
>> On Sat, Dec 11, 2021 at 03:04:53AM +0100, Pali Rohár wrote:
>> > I tried to find some information what is allowed and what not.
>> >
>> > On Monday 27 September 2021 12:19:48 Sean Young wrote:
>> > > Windows allows files and directories called "." and ".." to be
>> > > created
>> > > using UNC paths, i.e. "\\?\D:\..". Now this is totally insane
>> > > behaviour,
>> > > but when an exfat filesytem with such a file is mounted on Linux,
>> > > those
>> > > files show up as another directory and its contents is inaccessible.
>> > >
>> > > I can replicate this using exfat filesystems, but not ntfs.
>> >
>> > Microsoft exFAT specification explicitly disallow "." and "..", see:
>> [...]
>> > On the other hand Microsoft FAT32 specification can be understood that
>> > file may have long name (vfat) set to "." or ".." but not short name.
>> [...]
>> > OSTA UDF 2.60 specification does not disallow "." and ".." entries, but
>> [...]
>> > So it means that "." and ".." entries could be stored on disk as valid
>> > file names.
>>
>> It doesn't matter one whit what the specification says.  Anyone with a
>> disk
>> editor can craft a filesystem containing filenames such as "." or "..",
>> "/"
>> "foo/bar" or anything else we would like to ban.
>
> That is truth. But question is what should do fsck tools with such file
> names on filesystems where "." and ".." are permitted? Fully valid
> argument is "do not touch them" because there is nothing bad with these
> names.
>
>> > > So, in Linux cannot read "." or ".." (i.e., I can't see "Hello,
>> > > World!"). I
>> > > don't know what the correct handling should be, but having two "." and
>> > > two
>> > > ".." files does not seem right at all.
>> >
>> > This is really a bug in Linux kernel. It should not export "." and ".."
>> > into VFS even when filesystem disk format supports such insane file
>> > names.
>>
>> This.
>>
>> Otherwise, every filesystem driver would need to contain redundant code
>> for
>> checking for such bad names.
>>
>> > So either Linux needs to completely hide these insane file names from
>> > VFS or translate them to something which do not conflict with other
>> > files in correct directory.
>>
>> Escaping bad names has the problem of the escaped name also possibly
>> existing -- perhaps even recursively.  Plus, the filesystem might be
>> using
>> hashed or tree indices which could go wrong if a name is altered.
>
> vfat has already own escaping scheme and it is documented in mount(8)
> manpage. Invalid characters are translated either to fixed char '?' or
> to ':'... esc sequence if uni_xlate mount option is used. But it looks
> like that that kernel vfat driver do not have these two entries "." and
> ".." in its blacklist.
>
> And, another important thing about vfat is that it has two file names
> for each file. One short 8.3 and one long vfat. Short 8.3 do not allow
> "." or "..", so another possibility how to handle this issue for vfat is
> to show short 8.3 name in VFS when long is invalid.
>
> For UDF case, specification already says how to handle problematic
> file names, so I think that udf.ko could implement it according to
> specification.
>
> But for all other filesystems it is needed to do something ideally on
> VFS layer.
>
> What about generating some deterministic / predicable file names which
> will not conflict with other file names in current directory for these
> problematic files?
>
>> But then, I once proposed (and I'm pondering reviving) a ban for
>> characters
>> \x01..\x1f and possibly others, and if banned, they can still
>> legitimately
>> occur in old filesystems.
>>
>> > I guess that hiding them for exfat is valid thing as Microsoft
>> > specification explicitly disallow them. Probably fsck.exfat can be
>> > teach
>> > to rename these files and/or put them to lost+found directory.
>>
>> fsck fixing those is a good thing but we still need to handle them at
>> runtime.
>
> Namjae Jeon, would you be able to implement fixing of such filenames in
> fsck.exfat tool?
We've recently been finalizing the repair function in fsck.exfat. We
will check it as soon as it is finished.

Thanks for your suggestion!
>
>>
>> Meow!
>> --
>> ⢀⣴⠾⠻⢶⣦⠀
>> ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were
>> good.
>> ⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on
>> #linux-sunxi
>> ⠈⠳⣄⠀⠀⠀⠀
>

Re: Incorrect handling of . and .. files

[Pali Rohár]

On Monday 13 December 2021 12:39:03 Pali Rohár wrote:
> On Monday 13 December 2021 07:47:44 Adam Borowski wrote:
> > On Sat, Dec 11, 2021 at 03:04:53AM +0100, Pali Rohár wrote:
> > > I tried to find some information what is allowed and what not.
> > > 
> > > On Monday 27 September 2021 12:19:48 Sean Young wrote:
> > > > Windows allows files and directories called "." and ".." to be created
> > > > using UNC paths, i.e. "\\?\D:\..". Now this is totally insane behaviour,
> > > > but when an exfat filesytem with such a file is mounted on Linux, those
> > > > files show up as another directory and its contents is inaccessible.
> > > > 
> > > > I can replicate this using exfat filesystems, but not ntfs.
> > > 
> > > Microsoft exFAT specification explicitly disallow "." and "..", see:
> > [...]
> > > On the other hand Microsoft FAT32 specification can be understood that
> > > file may have long name (vfat) set to "." or ".." but not short name.
> > [...]
> > > OSTA UDF 2.60 specification does not disallow "." and ".." entries, but
> > [...]
> > > So it means that "." and ".." entries could be stored on disk as valid
> > > file names.
> > 
> > It doesn't matter one whit what the specification says.  Anyone with a disk
> > editor can craft a filesystem containing filenames such as "." or "..", "/"
> > "foo/bar" or anything else we would like to ban.
> 
> That is truth. But question is what should do fsck tools with such file
> names on filesystems where "." and ".." are permitted? Fully valid
> argument is "do not touch them" because there is nothing bad with these
> names.
> 
> > > > So, in Linux cannot read "." or ".." (i.e., I can't see "Hello, World!"). I
> > > > don't know what the correct handling should be, but having two "." and two
> > > > ".." files does not seem right at all.
> > > 
> > > This is really a bug in Linux kernel. It should not export "." and ".."
> > > into VFS even when filesystem disk format supports such insane file
> > > names.
> > 
> > This.
> > 
> > Otherwise, every filesystem driver would need to contain redundant code for
> > checking for such bad names.
> > 
> > > So either Linux needs to completely hide these insane file names from
> > > VFS or translate them to something which do not conflict with other
> > > files in correct directory.
> > 
> > Escaping bad names has the problem of the escaped name also possibly
> > existing -- perhaps even recursively.  Plus, the filesystem might be using
> > hashed or tree indices which could go wrong if a name is altered.
> 
> vfat has already own escaping scheme and it is documented in mount(8)
> manpage. Invalid characters are translated either to fixed char '?' or
> to ':'... esc sequence if uni_xlate mount option is used. But it looks
> like that that kernel vfat driver do not have these two entries "." and
> ".." in its blacklist.
> 
> And, another important thing about vfat is that it has two file names
> for each file. One short 8.3 and one long vfat. Short 8.3 do not allow
> "." or "..", so another possibility how to handle this issue for vfat is
> to show short 8.3 name in VFS when long is invalid.
> 
> For UDF case, specification already says how to handle problematic
> file names, so I think that udf.ko could implement it according to
> specification.
> 
> But for all other filesystems it is needed to do something ideally on
> VFS layer.
> 
> What about generating some deterministic / predicable file names which
> will not conflict with other file names in current directory for these
> problematic files?

PING? Any opinion how to handle this issue?

For VFAT this is still open question.

> > But then, I once proposed (and I'm pondering reviving) a ban for characters
> > \x01..\x1f and possibly others, and if banned, they can still legitimately
> > occur in old filesystems.
> > 
> > > I guess that hiding them for exfat is valid thing as Microsoft
> > > specification explicitly disallow them. Probably fsck.exfat can be teach
> > > to rename these files and/or put them to lost+found directory.
> > 
> > fsck fixing those is a good thing but we still need to handle them at
> > runtime.
> 
> Namjae Jeon, would you be able to implement fixing of such filenames in
> fsck.exfat tool?
> 
> > 
> > Meow!
> > -- 
> > ⢀⣴⠾⠻⢶⣦⠀
> > ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
> > ⢿⡄⠘⠷⠚⠋⠀                                       -- <willmore> on #linux-sunxi
> > ⠈⠳⣄⠀⠀⠀⠀