mkfs.xfs protofile and paths with spaces

mkfs.xfs protofile and paths with spaces

[Daan De Meyer]

Hi,

We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
population of XFS filesystems. However, the man page does not specify
how to encode filenames with spaces in them. Spaces are used as the
token delimiter so I was wondering if there's some way to escape
filenames with spaces in them?

Cheers,

Daan De Meyer

Re: mkfs.xfs protofile and paths with spaces

[Darrick J. Wong]

On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> Hi,
> 
> We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> population of XFS filesystems. However, the man page does not specify
> how to encode filenames with spaces in them. Spaces are used as the
> token delimiter so I was wondering if there's some way to escape
> filenames with spaces in them?

Spaces in filenames apparently weren't common when protofiles were
introduced in the Fourth Edition Unix in November 1973[1], so that
wasn't part of the specification for them:

    "The prototype file contains tokens separated by spaces or new
     lines."

The file format seems to have spread to other filesystems (minix, xenix,
afs, jfs, aix, etc.) without anybody adding support for spaces in
filenames.

One could make the argument that the protofile parsing code should
implicitly 's/\// /g' in the filename token since no Unix supports
slashes in directory entries, but that's not what people have been
doing for the past several decades.

At this point, 50 years later, it probably would make more sense to
clone the mke2fs -d functionality ("slurp up this directory tree") if
there's interest?  Admittedly, at this point it's so old that we ought
to rev the entire format.

[1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
or https://man.cat-v.org/unix-6th/8/mkfs 

--D

> Cheers,
> 
> Daan De Meyer

Re: mkfs.xfs protofile and paths with spaces

[Daan De Meyer]

> ...

While a "-d" switch would be great to have, it'd also be great if we
could make the protofile format work with escaped spaces. That way we
can just add escaping for spaces in our tooling that calls mkfs.xfs
and we don't have to do ugly version checks on the mkfs binary version
to figure out which option to use.

On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > Hi,
> >
> > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > population of XFS filesystems. However, the man page does not specify
> > how to encode filenames with spaces in them. Spaces are used as the
> > token delimiter so I was wondering if there's some way to escape
> > filenames with spaces in them?
>
> Spaces in filenames apparently weren't common when protofiles were
> introduced in the Fourth Edition Unix in November 1973[1], so that
> wasn't part of the specification for them:
>
>     "The prototype file contains tokens separated by spaces or new
>      lines."
>
> The file format seems to have spread to other filesystems (minix, xenix,
> afs, jfs, aix, etc.) without anybody adding support for spaces in
> filenames.
>
> One could make the argument that the protofile parsing code should
> implicitly 's/\// /g' in the filename token since no Unix supports
> slashes in directory entries, but that's not what people have been
> doing for the past several decades.
>
> At this point, 50 years later, it probably would make more sense to
> clone the mke2fs -d functionality ("slurp up this directory tree") if
> there's interest?  Admittedly, at this point it's so old that we ought
> to rev the entire format.
>
> [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> or https://man.cat-v.org/unix-6th/8/mkfs
>
> --D
>
> > Cheers,
> >
> > Daan De Meyer

Re: mkfs.xfs protofile and paths with spaces

[Darrick J. Wong]

On Wed, Jan 25, 2023 at 12:09:28PM +0100, Daan De Meyer wrote:
> > ...
> 
> While a "-d" switch would be great to have, it'd also be great if we
> could make the protofile format work with escaped spaces. That way we
> can just add escaping for spaces in our tooling that calls mkfs.xfs
> and we don't have to do ugly version checks on the mkfs binary version
> to figure out which option to use.

...which comes at a cost of making *us* figure out some gross hack to
retrofit that into the protofile format.

The easiest hack I can think of is to amend the protofile parser to
change any slash in the name to a space before creating the directory
entry.  Slashes aren't allowed (and right now produce a corrupt
filesystem) so I guess that's the easy way out:

# cat /tmp/protofile
/
0 0
d--775 1000 1000
: Descending path /code/t/fstests
 get/isk.sh   ---775 1000 1000 /code/t/fstests/getdisk.sh
 ext4.ftrace  ---664 1000 1000 /code/t/fstests/ext4.ftrace
 ocfs2.ftrace ---664 1000 1000 /code/t/fstests/ocfs2.ftrace
 xfs.ftrace   ---644 1000 1000 /code/t/fstests/xfs.ftrace
$
# mkfs.xfs -p /tmp/protofile /dev/sda -f
meta-data=/dev/sda               isize=512    agcount=4, agsize=1298176
# xfs_db -c 'ls /' /dev/sda
/:
8          128                directory      0x0000002e   1 . (good)
10         128                directory      0x0000172e   2 .. (good)
12         131                regular        0xd8830694  10 get/isk.sh
(corrupt)
15         132                regular        0x3a30d3ae  11 ext4.ftrace
(good)
18         133                regular        0x3d313221  12 ocfs2.ftrace
(good)
21         134                regular        0x28becaee  10 xfs.ftrace
(good)
# xfs_repair -n /dev/sda
Phase 1 - find and verify superblock...
Phase 2 - using internal log
        - zero log...
        - scan filesystem freespace and inode maps...
        - found root inode chunk
Phase 3 - for each AG...
        - scan (but don't clear) agi unlinked lists...
        - process known inodes and perform inode discovery...
        - agno = 0
entry contains illegal character in shortform dir 128
would have junked entry "get/isk.sh" in directory inode 128
        - agno = 1
        - agno = 2
        - agno = 3
        - process newly discovered inodes...
Phase 4 - check for duplicate blocks...
        - setting up duplicate extent list...
        - check for inodes claiming duplicate blocks...
        - agno = 0
entry contains illegal character in shortform dir 128
would have junked entry "get/isk.sh" in directory inode 128
        - agno = 2
        - agno = 1
        - agno = 3
No modify flag set, skipping phase 5
Phase 6 - check inode connectivity...
        - traversing filesystem ...
        - traversal finished ...
        - moving disconnected inodes to lost+found ...
Phase 7 - verify link counts...
No modify flag set, skipping filesystem flush and exiting.

Would that (replace slash with space) help?

--D

> On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
> >
> > On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > > Hi,
> > >
> > > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > > population of XFS filesystems. However, the man page does not specify
> > > how to encode filenames with spaces in them. Spaces are used as the
> > > token delimiter so I was wondering if there's some way to escape
> > > filenames with spaces in them?
> >
> > Spaces in filenames apparently weren't common when protofiles were
> > introduced in the Fourth Edition Unix in November 1973[1], so that
> > wasn't part of the specification for them:
> >
> >     "The prototype file contains tokens separated by spaces or new
> >      lines."
> >
> > The file format seems to have spread to other filesystems (minix, xenix,
> > afs, jfs, aix, etc.) without anybody adding support for spaces in
> > filenames.
> >
> > One could make the argument that the protofile parsing code should
> > implicitly 's/\// /g' in the filename token since no Unix supports
> > slashes in directory entries, but that's not what people have been
> > doing for the past several decades.
> >
> > At this point, 50 years later, it probably would make more sense to
> > clone the mke2fs -d functionality ("slurp up this directory tree") if
> > there's interest?  Admittedly, at this point it's so old that we ought
> > to rev the entire format.
> >
> > [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> > or https://man.cat-v.org/unix-6th/8/mkfs
> >
> > --D
> >
> > > Cheers,
> > >
> > > Daan De Meyer

Re: mkfs.xfs protofile and paths with spaces

[Daan De Meyer]

> ...

> Would that (replace slash with space) help?

That sounds perfect. I can change our tooling to replace spaces with
slashes in the protofile. Theoretically you could also have newlines
and tabs and such in filenames but I don't think we need to support
that since it's pretty insane to do in the first place.

On Thu, 2 Feb 2023 at 01:56, Darrick J. Wong <djwong@kernel.org> wrote:
>
> On Wed, Jan 25, 2023 at 12:09:28PM +0100, Daan De Meyer wrote:
> > > ...
> >
> > While a "-d" switch would be great to have, it'd also be great if we
> > could make the protofile format work with escaped spaces. That way we
> > can just add escaping for spaces in our tooling that calls mkfs.xfs
> > and we don't have to do ugly version checks on the mkfs binary version
> > to figure out which option to use.
>
> ...which comes at a cost of making *us* figure out some gross hack to
> retrofit that into the protofile format.
>
> The easiest hack I can think of is to amend the protofile parser to
> change any slash in the name to a space before creating the directory
> entry.  Slashes aren't allowed (and right now produce a corrupt
> filesystem) so I guess that's the easy way out:
>
> # cat /tmp/protofile
> /
> 0 0
> d--775 1000 1000
> : Descending path /code/t/fstests
>  get/isk.sh   ---775 1000 1000 /code/t/fstests/getdisk.sh
>  ext4.ftrace  ---664 1000 1000 /code/t/fstests/ext4.ftrace
>  ocfs2.ftrace ---664 1000 1000 /code/t/fstests/ocfs2.ftrace
>  xfs.ftrace   ---644 1000 1000 /code/t/fstests/xfs.ftrace
> $
> # mkfs.xfs -p /tmp/protofile /dev/sda -f
> meta-data=/dev/sda               isize=512    agcount=4, agsize=1298176
> # xfs_db -c 'ls /' /dev/sda
> /:
> 8          128                directory      0x0000002e   1 . (good)
> 10         128                directory      0x0000172e   2 .. (good)
> 12         131                regular        0xd8830694  10 get/isk.sh
> (corrupt)
> 15         132                regular        0x3a30d3ae  11 ext4.ftrace
> (good)
> 18         133                regular        0x3d313221  12 ocfs2.ftrace
> (good)
> 21         134                regular        0x28becaee  10 xfs.ftrace
> (good)
> # xfs_repair -n /dev/sda
> Phase 1 - find and verify superblock...
> Phase 2 - using internal log
>         - zero log...
>         - scan filesystem freespace and inode maps...
>         - found root inode chunk
> Phase 3 - for each AG...
>         - scan (but don't clear) agi unlinked lists...
>         - process known inodes and perform inode discovery...
>         - agno = 0
> entry contains illegal character in shortform dir 128
> would have junked entry "get/isk.sh" in directory inode 128
>         - agno = 1
>         - agno = 2
>         - agno = 3
>         - process newly discovered inodes...
> Phase 4 - check for duplicate blocks...
>         - setting up duplicate extent list...
>         - check for inodes claiming duplicate blocks...
>         - agno = 0
> entry contains illegal character in shortform dir 128
> would have junked entry "get/isk.sh" in directory inode 128
>         - agno = 2
>         - agno = 1
>         - agno = 3
> No modify flag set, skipping phase 5
> Phase 6 - check inode connectivity...
>         - traversing filesystem ...
>         - traversal finished ...
>         - moving disconnected inodes to lost+found ...
> Phase 7 - verify link counts...
> No modify flag set, skipping filesystem flush and exiting.
>
> Would that (replace slash with space) help?
>
> --D
>
> > On Wed, 25 Jan 2023 at 02:53, Darrick J. Wong <djwong@kernel.org> wrote:
> > >
> > > On Mon, Jan 23, 2023 at 10:13:12PM +0100, Daan De Meyer wrote:
> > > > Hi,
> > > >
> > > > We're trying to use mkfs.xfs's "-p" protofile option for unprivileged
> > > > population of XFS filesystems. However, the man page does not specify
> > > > how to encode filenames with spaces in them. Spaces are used as the
> > > > token delimiter so I was wondering if there's some way to escape
> > > > filenames with spaces in them?
> > >
> > > Spaces in filenames apparently weren't common when protofiles were
> > > introduced in the Fourth Edition Unix in November 1973[1], so that
> > > wasn't part of the specification for them:
> > >
> > >     "The prototype file contains tokens separated by spaces or new
> > >      lines."
> > >
> > > The file format seems to have spread to other filesystems (minix, xenix,
> > > afs, jfs, aix, etc.) without anybody adding support for spaces in
> > > filenames.
> > >
> > > One could make the argument that the protofile parsing code should
> > > implicitly 's/\// /g' in the filename token since no Unix supports
> > > slashes in directory entries, but that's not what people have been
> > > doing for the past several decades.
> > >
> > > At this point, 50 years later, it probably would make more sense to
> > > clone the mke2fs -d functionality ("slurp up this directory tree") if
> > > there's interest?  Admittedly, at this point it's so old that we ought
> > > to rev the entire format.
> > >
> > > [1] https://dspinellis.github.io/unix-v4man/v4man.pdf (page 274)
> > > or https://man.cat-v.org/unix-6th/8/mkfs
> > >
> > > --D
> > >
> > > > Cheers,
> > > >
> > > > Daan De Meyer