Squashfs without ./..

Squashfs without ./..

[Jan Engelhardt]

Hello,


I have observed that squashfs, when mounted, does not return any "." or ".." 
pseudo-directories upon readdir.
Could this be added? Would there be any objections?


Jan Engelhardt
-- 

Re: Squashfs without ./..

[Jesper Juhl]

On Tue, 22 Mar 2005, Jan Engelhardt wrote:

> Hello,
> 
> 
> I have observed that squashfs, when mounted, does not return any "." or ".." 
> pseudo-directories upon readdir.
> Could this be added? Would there be any objections?
> 
I can't say if there will be any objections or not, but if that's 
something that people want, then I'd like to take a stab at implementing 
it - could be fun and I'd love to learn a little more about that are of 
the kernel, so I'll have a go at it if noone screams.

/Jesper Juhl

Re: Squashfs without ./..

[Pietro Zuco]

On Tuesday 22 March 2005 16:59, Jesper Juhl wrote:
> On Tue, 22 Mar 2005, Jan Engelhardt wrote:
> > I have observed that squashfs, when mounted, does not return any "." or
> > ".." pseudo-directories upon readdir.
> > Could this be added? Would there be any objections?
>
> I can't say if there will be any objections or not, but if that's
> something that people want

I agree with Jan.
The . / .. will be useful for some scripts that use it.

-Pietro-

-- 
-------------------------------------------
- Pietro Zuco
- Email: pietro@zuco.org
- Email: pietro@member.fsf.org
- Web: http://www.zuco.org
- Join FSF as an Associate Member at:
- http://member.fsf.org/join?referrer=3082>
-
- Linux User Number: #252836
- Get counted! http://counter.li.org
------------------------------------------- 

Re: Squashfs without ./..

[Jan Engelhardt]

>I agree with Jan.
>The . / .. will be useful for some scripts that use it.

Which scripts use that? As stated, these two directory entries exist when you 
stat() them, they just do not show up in readdir(), and I bet few programs 
care for "." and ".." when doing their readdir.


Jan Engelhardt
-- 

Re: Squashfs without ./..

[Tom Vier]

On Wed, Mar 23, 2005 at 06:31:24PM +0100, Jan Engelhardt wrote:
> Which scripts use that? As stated, these two directory entries exist when you 
> stat() them, they just do not show up in readdir(), and I bet few programs 
> care for "." and ".." when doing their readdir.

There's probably a number of apps that skip the first two dirents, instead
of checking for the dot dirs.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: Squashfs without ./..

[Jan Engelhardt]

>> Which scripts use that? As stated, these two directory entries exist when you 
>> stat() them, they just do not show up in readdir(), and I bet few programs 
>> care for "." and ".." when doing their readdir.
>
>There's probably a number of apps that skip the first two dirents, instead
>of checking for the dot dirs.

Does POSIX or some standard say that . and .. need to be the first two 
entries?



Jan Engelhardt
-- 

Re: Squashfs without ./..

[Geert Uytterhoeven]

On Wed, 23 Mar 2005, Tom Vier wrote:
> On Wed, Mar 23, 2005 at 06:31:24PM +0100, Jan Engelhardt wrote:
> > Which scripts use that? As stated, these two directory entries exist when you 
> > stat() them, they just do not show up in readdir(), and I bet few programs 
> > care for "." and ".." when doing their readdir.
> 
> There's probably a number of apps that skip the first two dirents, instead
> of checking for the dot dirs.

Yep, check `-noleaf' in find(1).

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

Re: Squashfs without ./..

[Tommy Reynolds]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

Uttered Geert Uytterhoeven <geert@linux-m68k.org>, spake thus:

> > There's probably a number of apps that skip the first two dirents, instead
> > of checking for the dot dirs.
> 
> Yep, check `-noleaf' in find(1)

Then it is broken in several ways.  

First, file systems are not required to implement ".." (only "." is
magical, ".." is a courtesy).  

Second, skipping the first two entries carries an implied assumtion
about the file name sorting order that is not portable in a
non-US-ASCII world.

Cheers

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Squashfs without ./..

[Jan Engelhardt]

>> > There's probably a number of apps that skip the first two dirents, instead
>> > of checking for the dot dirs.
>> Yep, check `-noleaf' in find(1)
>Then it is broken in several ways.  
>First, file systems are not required to implement ".." (only "." is
>magical, ".." is a courtesy).  

Heh, what would happen if .. disappeared? Would `cd ..` become impossible 
(even if it is a shell builtin, it probably stat()s for ..)?


Jan Engelhardt
-- 

Re: Squashfs without ./..

[Andreas Schwab]

Jan Engelhardt <jengelh@linux01.gwdg.de> writes:

>>> > There's probably a number of apps that skip the first two dirents, instead
>>> > of checking for the dot dirs.
>>> Yep, check `-noleaf' in find(1)
>>Then it is broken in several ways.  
>>First, file systems are not required to implement ".." (only "." is
>>magical, ".." is a courtesy).  
>
> Heh, what would happen if .. disappeared?

"." and ".." are handled in the VFS.  No filesystem code ever sees them
during lookup.

Andreas.

-- 
Andreas Schwab, SuSE Labs, schwab@suse.de
SuSE Linux Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany
Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: Squashfs without ./..

[H. Peter Anvin]

Followup to:  <20050324133628.196a4c41.Tommy.Reynolds@MegaCoder.com>
By author:    Tommy Reynolds <Tommy.Reynolds@MegaCoder.com>
In newsgroup: linux.dev.kernel
> 
> Then it is broken in several ways.  
> 
> First, file systems are not required to implement ".." (only "." is
> magical, ".." is a courtesy).  
> 

Sez who?  Realistically, most programs that display a list of
filenames and has "up one level" as part of the list (not all of them
do it that way) probably expect to get ".." in there to display this.

> Second, skipping the first two entries carries an implied assumtion
> about the file name sorting order that is not portable in a
> non-US-ASCII world.

Doesn't have anything to do with sorting order or US-ASCII, it has to
do with readdir order.  If nothing else, it would be highly surprising
if "." and ".." weren't first; it's certainly a de facto standard, if
not de jure.

	-hpa

Re: Squashfs without ./..

[Kyle Moffett]

Tommy Reynolds wrote:
> Then it is broken in several ways.
>
> First, file systems are not required to implement ".." (only "." is
> magical, ".." is a courtesy).

On Mar 24, 2005, at 14:59, H. Peter Anvin wrote:
> Doesn't have anything to do with sorting order or US-ASCII, it has to
> do with readdir order.  If nothing else, it would be highly surprising
> if "." and ".." weren't first; it's certainly a de facto standard, if
> not de jure.

IMHO, this is one of those cases where "Be liberal in what you accept
and strict in what you emit" applies strongly.  New filesystems should
probably always emit "." and ".." in that order with sane behavior,
and new programs should probably be able to handle it if they don't. I
would add ".." and "." to squashfs, just so that it acts like the rest
of the filesystems on the planet, even if it has to emulate them
internally.  OTOH, I think that the default behavior of find is broken
and should probably be fixed, maybe by making the default use the full
readdir and optionally allowing a -fast option that optimizes the
search using such tricks.

Cheers,
Kyle Moffett

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$
L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+
PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r  
!y?(-)
------END GEEK CODE BLOCK------

Re: Squashfs without ./..

[H. Peter Anvin]

Kyle Moffett wrote:
> 
> IMHO, this is one of those cases where "Be liberal in what you accept
> and strict in what you emit" applies strongly.  New filesystems should
> probably always emit "." and ".." in that order with sane behavior,
> and new programs should probably be able to handle it if they don't. I
> would add ".." and "." to squashfs, just so that it acts like the rest
> of the filesystems on the planet, even if it has to emulate them
> internally.  OTOH, I think that the default behavior of find is broken
> and should probably be fixed, maybe by making the default use the full
> readdir and optionally allowing a -fast option that optimizes the
> search using such tricks.
> 

Note that Linux always accepts . and .. so it's just a matter of making 
them appear in readdir.

	-hpa

Re: Squashfs without ./..

[Jesper Juhl]

On Thu, 24 Mar 2005, H. Peter Anvin wrote:

> Kyle Moffett wrote:
> > 
> > IMHO, this is one of those cases where "Be liberal in what you accept
> > and strict in what you emit" applies strongly.  New filesystems should
> > probably always emit "." and ".." in that order with sane behavior,
> > and new programs should probably be able to handle it if they don't. I
> > would add ".." and "." to squashfs, just so that it acts like the rest
> > of the filesystems on the planet, even if it has to emulate them
> > internally.  OTOH, I think that the default behavior of find is broken
> > and should probably be fixed, maybe by making the default use the full
> > readdir and optionally allowing a -fast option that optimizes the
> > search using such tricks.
> > 
> 
> Note that Linux always accepts . and .. so it's just a matter of making them
> appear in readdir.
> 
I'm working on that, but it's a learning experience for me, so it's going 
a bit slow - but I'll get there.

-- 
Jesper Juhl

Re: Squashfs without ./..

[Jan Harkes]

On Fri, Mar 25, 2005 at 02:59:14PM +0100, Jesper Juhl wrote:
> On Thu, 24 Mar 2005, H. Peter Anvin wrote:
> > Note that Linux always accepts . and .. so it's just a matter of making them
> > appear in readdir.
> > 
> I'm working on that, but it's a learning experience for me, so it's going 
> a bit slow - but I'll get there.

Check the top of coda_venus_readdir in fs/coda/dir.c.

Coda's directories internally don't have the '.' and '..' as the first
two entries. In general it works just fine, I think there was one
application where it was causing a problem so now we use 'f_pos == 0'
and 'f_pos == 1' to spit out those entries based on dcache data.
f_pos >= 2 goes through the normal directory, but we skip the
out-of-order '.' and '..' entries.

Btw. the '.' and '..' entries are used by applications that are linked
against libc5 for the getpwd() implementation. I guess it was the only
way to get path information before the introduction of the dcache.

Oh, and the find -noleaf thing, a workaround for filesystems that don't
count subdirectories is to set the directory linkcount to 1 instead of
2 + number of subdirs. The find optimization then subtracts 2, and as a
result thinks there are (unsigned)-1 aka. UINT_MAX subdirectories and
find will end up calling stat() on every directory entry.

Jan

Re: Squashfs without ./..

[Jesper Juhl]

On Sat, 26 Mar 2005, Jan Harkes wrote:

> On Fri, Mar 25, 2005 at 02:59:14PM +0100, Jesper Juhl wrote:
> > On Thu, 24 Mar 2005, H. Peter Anvin wrote:
> > > Note that Linux always accepts . and .. so it's just a matter of making them
> > > appear in readdir.
> > > 
> > I'm working on that, but it's a learning experience for me, so it's going 
> > a bit slow - but I'll get there.
> 
> Check the top of coda_venus_readdir in fs/coda/dir.c.
> 
Very useful info you provide. Thank you.

-- 
Jesper

Re: Squashfs without ./..

[Eric W. Biederman]

Kyle Moffett <mrmacman_g4@mac.com> writes:

> Tommy Reynolds wrote:
> > Then it is broken in several ways.
> >
> > First, file systems are not required to implement ".." (only "." is
> > magical, ".." is a courtesy).
> 
> On Mar 24, 2005, at 14:59, H. Peter Anvin wrote:
> > Doesn't have anything to do with sorting order or US-ASCII, it has to
> > do with readdir order.  If nothing else, it would be highly surprising
> > if "." and ".." weren't first; it's certainly a de facto standard, if
> > not de jure.
> 
> IMHO, this is one of those cases where "Be liberal in what you accept
> and strict in what you emit" applies strongly.  New filesystems should
> probably always emit "." and ".." in that order with sane behavior,
> and new programs should probably be able to handle it if they don't. I
> would add ".." and "." to squashfs, just so that it acts like the rest
> of the filesystems on the planet, even if it has to emulate them
> internally.  OTOH, I think that the default behavior of find is broken
> and should probably be fixed, maybe by making the default use the full
> readdir and optionally allowing a -fast option that optimizes the
> search using such tricks.

Find is doing a full readdir.  It just looks at the link count
of the directory it is doing the readdir on and if it is the
minimal unix link count of 2 it knows it does not have to stat
directory entries to see if they are directories.

As I recall there is also special handling in find for link count
of 1 to automatically handle filesystems that don't follow the normal
unix conventions so every directory entry must be stated.

Eric

Re: Squashfs without ./..

[Phil Lougher]

On Thu, 24 Mar 2005 15:13:08 -0500, Kyle Moffett <mrmacman_g4@mac.com> wrote:
> I would add ".." and "." to squashfs, just so that it acts like the rest
> of the filesystems on the planet,

Cramfs also doesn't store '.' and '..', which is where I got the idea
from in the first place when originally implementing Squashfs.

Filesystems don't need to store '.' or ''..' in the filesystem, as
they're never looked up by the VFS - as mentioned elsewhere in this
thread, the VFS handles '.' and '..' internally.

Not storing the redundant '.' and '..' entries within the filesystem
achieves a small but nonetheless useful space saving.

> even if it has to emulate them
> internally.

Making readdir return '.' and '..' is trivially easy, as all the
required information to fake '.' and '..' entries are present.

The lack of '.' and '..' entries hasn't caused any problems despite
cramfs/squashfs being used for a large number of years.  I'm inclined
to believe any application that _relies_ on seeing '.' and '..'
returned by readdir is broken.  This situation is easily fixed within
the application rather than forcing the filesystem to unnecessarily
fake '.' and '..' entries which are never used.

> OTOH, I think that the default behavior of find is broken
> and should probably be fixed, maybe by making the default use the full
> readdir and optionally allowing a -fast option that optimizes the
> search using such tricks.
> 

Cramfs/Squashfs and other filesystems set the link count on files and
directories to 1, find correctly interprets this to mean it can't do
any of its 'tricks' and doesn't use any optimisations.

Phillip

Re: Squashfs without ./..

[H. Peter Anvin]

Phil Lougher wrote:
> 
> Making readdir return '.' and '..' is trivially easy, as all the
> required information to fake '.' and '..' entries are present.
> 
> The lack of '.' and '..' entries hasn't caused any problems despite
> cramfs/squashfs being used for a large number of years.  I'm inclined
> to believe any application that _relies_ on seeing '.' and '..'
> returned by readdir is broken.  This situation is easily fixed within
> the application rather than forcing the filesystem to unnecessarily
> fake '.' and '..' entries which are never used.
> 

<sarcasm>

Yeah, let's fix every broken application on the planet instead of fixing 
it in one place...

</sarcasm>

	-hpa

Re: Squashfs without ./..

[Phillip Lougher]

H. Peter Anvin wrote:
> Phil Lougher wrote:
> 
>>
>> Making readdir return '.' and '..' is trivially easy, as all the
>> required information to fake '.' and '..' entries are present.
>>
>> The lack of '.' and '..' entries hasn't caused any problems despite
>> cramfs/squashfs being used for a large number of years.  I'm inclined
>> to believe any application that _relies_ on seeing '.' and '..'
>> returned by readdir is broken.  This situation is easily fixed within
>> the application rather than forcing the filesystem to unnecessarily
>> fake '.' and '..' entries which are never used.
>>
> 
> <sarcasm>
> 
> Yeah, let's fix every broken application on the planet instead of fixing 
> it in one place...
> 
> </sarcasm>
> 

Fixing it in Squashfs implies Squashfs is broken.  It isn't.  If it has 
to be "fixed" in the kenel, fix it in the VFS, it is after all the VFS 
which makes '.' and '..' handling redundant in the filesystem.

>     -hpa
> 

Re: Squashfs without ./..

[Jörn Engel]

On Sat, 26 March 2005 02:14:18 +0000, Phillip Lougher wrote:
> 
> Fixing it in Squashfs implies Squashfs is broken.  It isn't.  If it has 
> to be "fixed" in the kenel, fix it in the VFS, it is after all the VFS 
> which makes '.' and '..' handling redundant in the filesystem.

There are some islands on this planet where behaviour of virtually all
the population wrt. driving on the proper side of the road is broken.
Those people are silly, sure.

Still, does that make you drive on the proper side while everyone else
tries to evade your car with furious maneuvers?

So, what do we learn from these silly islands?  "Wrong" and "Right"
are relative, sometimes is makes much more sense to say "oh well, let
them have their way".


Anyway, I took a look at squashfs and will send you patches shortly.
Hope you don't mind.

Jörn

-- 
Anything that can go wrong, will.
-- Finagle's Law

Re: Squashfs without ./..

[Jan Engelhardt]

>This situation is easily fixed within
>the application rather than forcing the filesystem to unnecessarily
>fake '.' and '..' entries which are never used.

You are right. . and .. do not need to show up (even they have been the 
"leaders" of ls -l ;-), Midnight Commander (`mc`) for example synthesizes ".." 
nevertheless.

So - what about removing . and .. in readdir for all "standard harddisk 
filesystems" (ext*,reiser*, [jx]fs)? I mean, one party always has to loose...


Jan Engelhardt
-- 
No TOFU for me, please.

Re: Squashfs without ./..

[H. Peter Anvin]

Followup to:  <Pine.LNX.4.61.0503261059430.28431@yvahk01.tjqt.qr>
By author:    Jan Engelhardt <jengelh@linux01.gwdg.de>
In newsgroup: linux.dev.kernel
> 
> You are right. . and .. do not need to show up (even they have been the 
> "leaders" of ls -l ;-), Midnight Commander (`mc`) for example synthesizes ".." 
> nevertheless.
> 
> So - what about removing . and .. in readdir for all "standard harddisk 
> filesystems" (ext*,reiser*, [jx]fs)? I mean, one party always has to loose...
> 

Are you seriously suggesting changing our behaviour of all the
conventional filesystems to a non-Unix behaviour, to match cramfs and
squashfs?

	-hpa

Re: Squashfs without ./..

[Paul Jackson]

> Yep, check `-noleaf' in find(1).

No.

At least the copy of find that I just looked at now, in
findutils-4.1.20, makes no such assumption that "." and ".." are the
first two directory entries.

Rather what it does is allow you to suppress an optimization, on file
systems that don't manage their directory link counts so that the link
count on a directory is exactly two more than the number of child
directories, which optimization avoids stat'ing every entry if you are
using some set of find options that are only looking at names, not other
stat data, and if by the link count on the directory, you've already
stat'd all the child directories.  The documentation for find -noleaf
spells this out.

The find command is enabling you to adapt to differing file system
directory link counts with this option.  It is not brokenly forcing a
wrong assumption on you, and in any case, it is an issue of directory
link counts, not of the opendir-readdir order of "." and ".." (if
present).

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@engr.sgi.com> 1.650.933.1373, 1.925.600.0401

Re: Squashfs without ./..

[Adam J. Richter]

Jan Engelhardt wrote:
>[...] . and .. do not need to show up (even they have been the 
>"leaders" of ls -l ;-), Midnight Commander (`mc`) for example synthesizes ".." 
>nevertheless.
>
>So - what about removing . and .. in readdir for all "standard harddisk 
>filesystems" (ext*,reiser*, [jx]fs)? I mean, one party always has to loose...~v

	Eliminating the "." and ".." emulation in many individual
file systems would probably eliminate a moderate amount of code
from libfs/fs.c, a number of other virtual file systems and probably
every physical file system that does not actually store "." and "..".
It is very appealing to me.

	Unfortunately, the description of readdir() in the Single Unix
Specification version 3 says:

| [...] If entries for dot or dot-dot exist, one entry shall be returned
| for dot and one entry shall be returned for dot-dot; otherwise, they
| shall not be returned.

	Unless attempts to access "." and ".." would really return -ENOENT,
then at least the C library's readdir() function has to return them.  
At least that's how I read it.

	Although I do not believe that absolute compliance to SUSPv3
is a requirement demanded by those who make the "official" kernel
releases, I think that complying closely to SUSPv3 and many other
standards is considered to be worth a lot (in terms of technical
trade-offs) so that software that complies to these standards
is more likely to run properly on systems running the Linux kernel.
So, I would expect that patches changing squashfs and other file systems
whose readdir functions currently fail to return "." and ".." would 
be likely to be integrated (if they meet all the other usual quality
standards), at least for now.

	That said, I can think of at least two approaches by which
we could eliminate the "." and ".." emulation littering most Linux
file system drivers.

	The first way would be to change the kernel so that the
underlying readdir system call does not return "." or "..", but
have the C library do the emulation.  The C library can maintain
the state information for this purpose easily because opendir()
returns a pointer to an opaque structure that the C library
allocates.

	Alternatively, we could preserve the opendir system call's
behavior, but pick apart a few of the routines in fs/libfs.c to come
up with some more general utiity routines to implement the common case
where the first readdir returns ".", the second returns "..", a seek
pointer of 0 means before the ".", a seek pointer of 1 means before
the "..", and a seek pointer of 1 means immediately after the "..".
The actual implementation would be pretty short, but having an
interface that the client file systems could easily accomodate might
take some care (for example, accomodating their locking schemes while
keeping the interface simple enough so that the client file system
drivers are still made smaller by using it).

                    __     ______________ 
Adam J. Richter        \ /
adam@yggdrasil.com      | g g d r a s i l

Re: Squashfs without ./..

[Jan Engelhardt]

>>[...] . and .. do not need to show up (even they have been the 
>>"leaders" of ls -l ;-), Midnight Commander (`mc`) for example synthesizes ".." 
>>nevertheless.
>>
>>So - what about removing . and .. in readdir for all "standard harddisk 
>>filesystems" (ext*,reiser*, [jx]fs)? I mean, one party always has to loose...~v

H. Peter Anvin wrote:
>Are you seriously suggesting changing our behaviour of all the
>conventional filesystems to a non-Unix behaviour, to match cramfs and
>squashfs?

Only one can be right - either with ./.. or without it. And the one[s] who
is/are wrong should be fixed. Take it as a cosmetical issue, though.


Adam J. Richter wrote:
>
>	Eliminating the "." and ".." emulation in many individual
>file systems would probably eliminate a moderate amount of code
>from libfs/fs.c, a number of other virtual file systems and probably
>every physical file system that does not actually store "." and "..".
>It is very appealing to me.

As a side note, I am only discussing about ./.. for readdir; removing it from
the entire VFS would probably break things like /etc/mail/../../lib/libc.so.6
_in_ the kernel.

>	The first way would be to change the kernel so that the
>underlying readdir system call does not return "." or "..", but
>have the C library do the emulation.  The C library can maintain
>the state information for this purpose easily because opendir()
>returns a pointer to an opaque structure that the C library
>allocates.

Sounds "good", because ./.. could always be made the first two entries, and
people could optimize <shrug>. That brings up the point if - despite all sus
specs - we really need . and ...

The explorer.exe in the Neighbor OS also does not show . and ..;
and I doubt any app is using it in FindFile{First,} (open-/readdir),
maybe only for dentry lookup.

>but having an
>interface that the client file systems could easily accomodate might
>take some care (for example, accomodating their locking schemes while
>keeping the interface simple enough so that the client file system
>drivers are still made smaller by using it).

Also a nice idea.



Jan Engelhardt
-- 
No TOFU for me, please.