documentation issues

documentation issues

[Xen]

Hi,

a few things.

A user attempting to use autofs for the first time will assume that 
autofs will mount mount poins that already exist. Nowhere is it stated 
that mount points will only get created when accessed (through a "direct 
path").

Such a user (such as me) trying to use this thing (even with a tutorial 
on Ubuntu site) will be flabbergasted that nothing happens; he/she 
expects mount entries to appear, but nothing happens.

This renders the "nobrowse" default dysfunctional for a desktop user, 
who expects to see something before going there. It only works for 
automated systems, really.

Further, you will only become aware of "browse" after having been able 
to use the system, because otherwise you won't go reading entire manuals 
just to see what is wrong.

So in my case someone explained to me that the mount points would only 
get created when accessed, at which point I finally was able to have 
some result.


So I say two things:

- browse only works for static maps, and what person with a static map 
is going to run into performance issues when mount points are created in 
advance? Can we please remove this from the default config? (turning it 
off).

- browse being turned off by default combined with a complete lack of 
mention that you need to access these "nonexistent" directory paths / 
components before you will be able to see them, turns autofs into a 
complete mystery for someone who is not privy to the details yet.


Further I would say that on a desktop OS, the most likely use case is 
going to be this:

- have several shares in your home network
- want to access them through a GUI
- prefer automounting since mounting at boot can be problematic
- need to see them before you can access them in a GUI
- only alternative is "bookmarks" or "places" that work regardless of 
the things not being visible/listed in the directory.

Then I will also say,

that there is no documentation on the format of the map files other than 
auto.master. The documentation for map files is this:

cat /etc/auto.misc.

That is the documentation.

After you've figured out that e.g. auto.smb wants the servername as the 
key ($1) the documentation becomes this:

/etc/auto.smb <servername>

>> watch results pour out.

But auto.smb doesn't mention that it wants the servername as the key.

Of course, it does mention that:

# You'll be able to access Windows and Samba shares in your network
# under /cifs/host.domain/share

But at this point you still don't know that the "host.domain" part will 
be invisible until you access it.

I'm sorry if I sound abrasive but I was doing freaking straces and even 
kernel traces before realizing how this was supposed to work. There is 
just no documentation on it. The most important thing is missing. Is not 
mentioned.

I wasted so much time trying to figure out how this was supposed to 
work... in the game of Zelda 1 I believe in some dungeon there is a wall 
somewhere that you can walk through. You need to go through that wall to 
get ahead in the game, but there is no visual indication that you can go 
through. It is a secret.

Well. So is this :P.

How is anyone supposed to know that automounting a "mount point" does 
not require a mount point to exist in advance?????

I had even (previously, for something else) been reading kernel source 
code in namei.c or whatever and came across autofs in e.g. lookup_slow 
or something. Even then it never dawned to me that the typical use case 
would be for some mount point to not exist yet.

So you start this thing and you cannot write to the mount point any 
longer so you know it is getting managed.

But nothing ever shows. I even .... well let's not say that here :P.

;-) I had done "touch a" and the file was not created; however I had put 
some command in auto.smb that would output $1 to some file. So the file 
said "a" but it never occured to me that this was the result of my 
"touch a" command ;-). Stupid!.

So I would plead for just two things for other people:

a) get rid of the browse_mode = no, in the configuration file, for 
static mounts
b) say something about mounts not being created automatically in the man 
page for auto.master, if nothing else. If you also do (a), this only 
applies now to programmatic mounts.

=============================================================

You could also do nothing but change the default auto.master:
-------------------------------------------------------------

# This is the sample auto.master file installed on your system.
# This map contains entries for mount points that are to be managed
# by autofs.
#
# Each of these mount points will be mounted with the required map
# file as the parameter to the autofs filesystem module that will
# manage this mount point for you in one of its threads.

# Each entry mentions a mount point that must already exist on your
# system, as well as a map file to be loaded onto that mount point.
# Its format is as follows:


# <mount-point>  [map-type[,format]:]<map> [options]

# where <mount-point> indicates a directory on your filesystem and
#       <map> indicates the map file to be used for it.

# For further details see the man page for auto.master(5).

# Example:
#
# /misc  /etc/auto.misc  browse
#
# This will create entries from /etc/auto.misc under /misc. The browse
# option guarantees that its entries will be created straight away as
# sub-directories to /misc.

# Without the browse option, they would only be created after being
# accessed, e.g. by direct reference. The browse option is only
# applicable to static map files.
#
#
# /net   -hosts
#
# This mounts all accessible NFS shares from hosts in your network
# under /net/<hostname>/ when you first access that path.


# NOTE: mounts done from a hosts map will be mounted with the
#       "nosuid" and "nodev" options unless the "suid" and "dev"
#       options are explicitly given.
#
#
# Include *.autofs from /etc/auto.master.d/ and the included files
# must conform to the format of this file.

+dir:/etc/auto.master.d

# Include another central master map if it can be found using
# nsswitch sources. Note that duplicate entries will be discarded
# as keys must be unique and the keys from this file will take
# precedence. That means you can only specify each mount point once.

+auto.master

# Note that map files can be scripts that take the accessed path as a
# parameter. E.g. the following would mount all Samba shares from a
# certain host under /cifs/<hostname>/* when /cifs/<hostname> was
# accessed:

# /cifs   program:/etc/auto.smb
#
# See /etc/auto.smb for more documentation.

-------------------------------------------------------------

I mean you could rewrite more documentation but this is the simplest I 
could do right here right now.

I think that would clarify a lot for everyone but of course perhaps you 
would want to rewrite more so the central documentation is also up to 
date, but please see this as a more modest attempt for making it more 
accessible to a new user.

Regards,

Bart Schouten.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Ian Kent]

On Tue, 2016-07-05 at 23:17 +0200, Xen wrote:
> Hi,
> 
> a few things.
> 
> A user attempting to use autofs for the first time will assume that 
> autofs will mount mount poins that already exist. Nowhere is it stated 
> that mount points will only get created when accessed (through a "direct 
> path").

Umm, "direct path" is an unfortunate choice here as you are talking about autofs
indirect mount maps.

Direct mount maps create a mount trigger for each map entry and so show up as
mount points in the file system.

There are reasons for it being this way.

> 
> Such a user (such as me) trying to use this thing (even with a tutorial 
> on Ubuntu site) will be flabbergasted that nothing happens; he/she 
> expects mount entries to appear, but nothing happens.
> 
> This renders the "nobrowse" default dysfunctional for a desktop user, 
> who expects to see something before going there. It only works for 
> automated systems, really.

Your assuming that autofs primary usage is targeted at the desktop.
It's often not the best use case for that.

There are other more suitable automount alternatives for GUI desktops.

> 
> Further, you will only become aware of "browse" after having been able 
> to use the system, because otherwise you won't go reading entire manuals 
> just to see what is wrong.
> 
> So in my case someone explained to me that the mount points would only 
> get created when accessed, at which point I finally was able to have 
> some result.
> 
> 
> So I say two things:
> 
> - browse only works for static maps, and what person with a static map 
> is going to run into performance issues when mount points are created in 
> advance? Can we please remove this from the default config? (turning it 
> off).

Well, quite a lot of current users actually.

Consider what can happen with a few hundred or a few thousand entries in an
indirect map.

Even though the browse option was added it has a specific set of problems.

First the kernel does not know if you are just listing a directory or actually
need to mount something.

So at the moment, certain types of system call (the module can't know the exact
system call either) don't cause a call back to perform a mount but often user
space will later make a system call that will cause a mount to occur anyway.

That leads to sometimes seeing directories that are those of the autofs file
system not those of the mounted file system, so they have completely different
attributes to those after an automount has been done, a problem in itself.

And, it can often lead to a mount storm, mounting everything in the directory.

OK for a few dozen mounts in a map but really bad for anything with more than a
few hundred entries.

Another reason for the default is that, because of this, historically there was
no "browse" option and this maintains what is (should be) expected.

> 
> - browse being turned off by default combined with a complete lack of 
> mention that you need to access these "nonexistent" directory paths / 
> components before you will be able to see them, turns autofs into a 
> complete mystery for someone who is not privy to the details yet.

It's been a long time for me so I probably have forgotten what it's like for
someone starting out.

And so, how would I know where in the man pages to put that so it gets noticed
by new users, perhaps from your perspective, you could point out a few key
places where it should be added, or better still send a patch.

> 
> 
> Further I would say that on a desktop OS, the most likely use case is 
> going to be this:
> 
> - have several shares in your home network
> - want to access them through a GUI
> - prefer automounting since mounting at boot can be problematic
> - need to see them before you can access them in a GUI
> - only alternative is "bookmarks" or "places" that work regardless of 
> the things not being visible/listed in the directory.

Either use a different automounting mechanism or use a direct mount map.

That would satisfy all these needs, all be it with the map entries always being
mounted on access.

The limitation of these is you can't have a direct map path that is within
another automounted path, if you need to do that you need to use a more
complicated setup. 

> 
> Then I will also say,
> 
> that there is no documentation on the format of the map files other than 
> auto.master. The documentation for map files is this:

That's not quite right.

Man page autofs(5) has a bit of information on both the Sun and amd format maps.

> 
> cat /etc/auto.misc.
> 
> That is the documentation.
> 
> After you've figured out that e.g. auto.smb wants the servername as the 
> key ($1) the documentation becomes this:
> 
> /etc/auto.smb <servername>
> 
> > > watch results pour out.
> 
> But auto.smb doesn't mention that it wants the servername as the key.
> 
> Of course, it does mention that:
> 
> # You'll be able to access Windows and Samba shares in your network
> # under /cifs/host.domain/share
> 
> But at this point you still don't know that the "host.domain" part will 
> be invisible until you access it.

By definition a program map can't know what keys may be passed to it.
And, yes, even autofs(5) doesn't say explicitly that a program map doesn't know
what keys may be passed to it, perhaps that should be ammended.

> 
> I'm sorry if I sound abrasive but I was doing freaking straces and even 
> kernel traces before realizing how this was supposed to work. There is 
> just no documentation on it. The most important thing is missing. Is not 
> mentioned.
> 
> I wasted so much time trying to figure out how this was supposed to 
> work... in the game of Zelda 1 I believe in some dungeon there is a wall 
> somewhere that you can walk through. You need to go through that wall to 
> get ahead in the game, but there is no visual indication that you can go 
> through. It is a secret.
> 
> Well. So is this :P.
> 
> How is anyone supposed to know that automounting a "mount point" does 
> not require a mount point to exist in advance?????
> 
> I had even (previously, for something else) been reading kernel source 
> code in namei.c or whatever and came across autofs in e.g. lookup_slow 
> or something. Even then it never dawned to me that the typical use case 
> would be for some mount point to not exist yet.
> 
> So you start this thing and you cannot write to the mount point any 
> longer so you know it is getting managed.
> 
> But nothing ever shows. I even .... well let's not say that here :P.
> 
> ;-) I had done "touch a" and the file was not created; however I had put 
> some command in auto.smb that would output $1 to some file. So the file 
> said "a" but it never occured to me that this was the result of my 
> "touch a" command ;-). Stupid!.
> 
> So I would plead for just two things for other people:
> 
> a) get rid of the browse_mode = no, in the configuration file, for 
> static mounts

No, that has it's own set of problems which people need to be aware of before
choosing to use the "browse" option.

> b) say something about mounts not being created automatically in the man 
> page for auto.master, if nothing else. If you also do (a), this only 
> applies now to programmatic mounts.

auto.master(5) isn't the place for that, autofs(5) is better and, as I say, make
some suggestions of provide a patch.

> 
> =============================================================
> 
> You could also do nothing but change the default auto.master:
> -------------------------------------------------------------
> 
> # This is the sample auto.master file installed on your system.
> # This map contains entries for mount points that are to be managed
> # by autofs.
> #
> # Each of these mount points will be mounted with the required map
> # file as the parameter to the autofs filesystem module that will
> # manage this mount point for you in one of its threads.
> 
> # Each entry mentions a mount point that must already exist on your
> # system, as well as a map file to be loaded onto that mount point.
> # Its format is as follows:
> 
> 
> # <mount-point>  [map-type[,format]:]<map> [options]
> 
> # where <mount-point> indicates a directory on your filesystem and
> #       <map> indicates the map file to be used for it.
> 
> # For further details see the man page for auto.master(5).
> 
> # Example:
> #
> # /misc  /etc/auto.misc  browse
> #
> # This will create entries from /etc/auto.misc under /misc. The browse
> # option guarantees that its entries will be created straight away as
> # sub-directories to /misc.
> 
> # Without the browse option, they would only be created after being
> # accessed, e.g. by direct reference. The browse option is only
> # applicable to static map files.
> #
> #
> # /net   -hosts
> #
> # This mounts all accessible NFS shares from hosts in your network
> # under /net/<hostname>/ when you first access that path.
> 
> 
> # NOTE: mounts done from a hosts map will be mounted with the
> #       "nosuid" and "nodev" options unless the "suid" and "dev"
> #       options are explicitly given.
> #
> #
> # Include *.autofs from /etc/auto.master.d/ and the included files
> # must conform to the format of this file.
> 
> +dir:/etc/auto.master.d
> 
> # Include another central master map if it can be found using
> # nsswitch sources. Note that duplicate entries will be discarded
> # as keys must be unique and the keys from this file will take
> # precedence. That means you can only specify each mount point once.
> 
> +auto.master
> 
> # Note that map files can be scripts that take the accessed path as a
> # parameter. E.g. the following would mount all Samba shares from a
> # certain host under /cifs/<hostname>/* when /cifs/<hostname> was
> # accessed:
> 
> # /cifs   program:/etc/auto.smb
> #
> # See /etc/auto.smb for more documentation.
> 
> -------------------------------------------------------------

I'll have a look at doing this but beware, often distributions use there own
default master map which I have no control over.

> 
> I mean you could rewrite more documentation but this is the simplest I 
> could do right here right now.
> 
> I think that would clarify a lot for everyone but of course perhaps you 
> would want to rewrite more so the central documentation is also up to 
> date, but please see this as a more modest attempt for making it more 
> accessible to a new user.

Yes, adding the fact that mount point directories aren't pre created for
indirect maps somewhere in auto.master(5) and in a couple of places in autofs(5)
is a good idea. You are right it's something that can catch the unaware.

Ian
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Xen]

Ian Kent schreef op 06-07-2016 2:57:
> On Tue, 2016-07-05 at 23:17 +0200, Xen wrote:
>> Hi,
>> 
>> a few things.
>> 
>> A user attempting to use autofs for the first time will assume that
>> autofs will mount mount poins that already exist. Nowhere is it stated
>> that mount points will only get created when accessed (through a 
>> "direct
>> path").
> 
> Umm, "direct path" is an unfortunate choice here as you are talking 
> about autofs
> indirect mount maps.
> 
> Direct mount maps create a mount trigger for each map entry and so show 
> up as
> mount points in the file system.
> 
> There are reasons for it being this way.

So you are berating me for choice of words ;-).

The only difference between direct and indirect seems to be that with 
direct the share is mounted "directly" on top of the specified mount 
point (actually, none is specified, it is mounted on the key returned 
from the map) and as a result it is not possible to have lazy 
initialization (unless you merged e.g. the root filesystem with an 
overlay filesystem) and so you are required to create this mountpoint, 
whereas with "indirect" you control the mount point itself and can 
choose whether to create the subdirectories or not (if you can acquire a 
list). "Directly" seems to mean "directly on the filesystem" whereas 
"indirectly" seems to mean: inside our own little enclave.

And always as a subdirectory of that. Not even technically necessary, 
you could even have a "indirect map" that triggered mounting I guess, 
don't know. I guess not.



> 
>> 
>> Such a user (such as me) trying to use this thing (even with a 
>> tutorial
>> on Ubuntu site) will be flabbergasted that nothing happens; he/she
>> expects mount entries to appear, but nothing happens.
>> 
>> This renders the "nobrowse" default dysfunctional for a desktop user,
>> who expects to see something before going there. It only works for
>> automated systems, really.
> 
> Your assuming that autofs primary usage is targeted at the desktop.
> It's often not the best use case for that.
> 
> There are other more suitable automount alternatives for GUI desktops.

Linus Torvalds says he developed Linux as a desktop OS. And how 
chagrined he is that it didn't succeed in just that. Well, now we know 
the reason for that ;-).

I know there are probably alternatives; but they are mostly useless for 
someone who wants to do it on his own. Desktop environments often do not 
even have mount operations on directories (even) when they are described 
in fstab. There is no other way that I know of to automatically mount 
e.g. samba shares, unless you navigate directly to it (and then they end 
up in some weird location in the filesystem, if at all). There is 
usually also no "user based" fstab; where you can describe mounts 
specifically for your user, that are then mounted through e.g. fuse, or 
something else that can do suid. Perhaps you mean udisks or udisksd or 
udisksctl but I'm not even sure that supports network-filesystems, they 
are certainly not shown in udisksctl status.

Dolphin can not actually mount a share to some fixed location, the Nemo 
I am using doesn't even succeed in browsing the network. For me there 
are only two solutions that I see, either fstab or autofs. Once you know 
how to do it, running /etc/auto.smb <host> quickly returns a list of 
shares (if you've set the credentials file) that you can then edit to 
become a static map with the browse option.

I do not see what is better suited currently to auto-mounting 
samba-shares. Anything that would work well, would work from both the 
shell and the GUI, in that the systems have to be linked, and ideally 
what the GUI does is stored in state files that work equally well from 
the shell. Any persistent mount would have to be stored somewhere on 
disk. Then the link would need to be automounted, OR mounted through 
e.g. libpam-mount. There is nothing wrong with mounting upon logon. And 
that allows for specific user-dependent stuff. (An automount depending 
on what user first accesses it, is problematic). (As for Samba). So 
perhaps logon-mounts are better suited (just restore the mount from the 
previous session, in that sense) because the $UID problem is hard to 
solve for autofs, you must either duplicate all entries, and hardcore 
the UIDs you need, for every user, or I don't know what, prevent other 
users (including root) from accessing the share you need. You'd 
basically need map files in auto.master.d for every user, including a 
map file for each person to specify the shares, if you want browsable, 
or a modified version of auto.smb that mounts individual shares when 
accessed. (In the latter case you'd have no browse map).

But this script doesn't know what it is being called on, so now you need 
a variable I guess for each of those .d/ files, so that their entries 
would be something like:

/media/<user>/nas  /etc/auto-short.smb -Duser=<user> which would then be 
used for the samba $UID.

If there is a better way to do this, I don't know. I see no alternatives 
other than libpam-mount and of course some GUI could do it, but then 
you'd really just have a form of duplicated functionality between autofs 
and it.


>> - browse only works for static maps, and what person with a static map
>> is going to run into performance issues when mount points are created 
>> in
>> advance? Can we please remove this from the default config? (turning 
>> it
>> off).
> 
> Well, quite a lot of current users actually.
> 
> Consider what can happen with a few hundred or a few thousand entries 
> in an
> indirect map.

Alright, but what indirect maps can cause browse tables to be shown? 
Only static maps, right? A programmatic map can only return a single 
entry. I have not looked into the other ones, but I suppose LDAP is more 
like a programmatic map than anything else.

So then consider that we only use a static map here. An actual file with 
thousands of entries? Of course it can happen. But what would be the 
point of that still (as opposed to LDAP or programmatic).


> Even though the browse option was added it has a specific set of 
> problems.
> 
> First the kernel does not know if you are just listing a directory or 
> actually
> need to mount something.

Do you mean the directory containing the (indirect) entries (e.g. the 
managed directory) or one of those entries themselves?

> So at the moment, certain types of system call (the module can't know 
> the exact
> system call either) don't cause a call back to perform a mount but 
> often user
> space will later make a system call that will cause a mount to occur 
> anyway.
> 
> That leads to sometimes seeing directories that are those of the autofs 
> file
> system not those of the mounted file system, so they have completely 
> different
> attributes to those after an automount has been done, a problem in 
> itself.

Oh... such as ownership etc.

The typical "I mount this, and now suddenly it is not root.root but 
xen.staff".

> And, it can often lead to a mount storm, mounting everything in the 
> directory.

But how can that happen? If I do; "for f in *; ls $f; done" sure.


> 
> OK for a few dozen mounts in a map but really bad for anything with 
> more than a
> few hundred entries.

After thinking a little I came up with what could perhaps be some 
default use case. You have terminals mounting home directories. On the 
"terminal machine" you may have just a single user. Through LDAP you 
find the home directory. There is obviously no sense for all users in 
the LDAP system to now have precreated mount points. The only thing that 
could make sense is to have an "invisible" user directory that now gets 
mounted.

I can see no reason to have hundreds of mount points in a directory 
unless they cannot conform to a specific pattern. If they follow a 
pattern, it is likely that only a few of them need to be mounted at any 
time. In that case being able to browse it seems nonsensical. An 
aggregate of hundreds of different mounts at the same time seems very 
weird. Weird to have a diverse set of unique names to begin with. As if 
all of them were manually added in some way. If they aren't manually 
added, but automatically, then why have a static list? Would you not be 
better off putting that in a database? And if you'd have so many, we'd 
assume they could be different users, or different websites, etc. We 
also assume not all of them are going to be needed, just a subset. But 
so now you have a system on which this list is statically created but 
dynamically updated. This list sits on the disk of the local system. But 
you're not querying a database (could be just as easy) that could also 
store this list and return the entries one by one when needed. The 
static file is the database.

Seems an improper use case but yeah. I don't know how you can have a 
static list that is huge while not choosing to use a database to manage 
it. Because a static map; now you need routines to update it, delete 
entries from it.

Maybe I'm missing something here.


> Another reason for the default is that, because of this, historically 
> there was
> no "browse" option and this maintains what is (should be) expected.

Yeah I thought so, because of the comments near the timeout value. 
However that is a reason to never change anything ever. Someone 
mistakenly choosing "browse" will very quickly find out.

If this person really has hundreds of items in his list, he will notice 
within a second that they are all visible. If this person really has 
hundreds to thousands of items, he or she may also be expected to know 
the ins and outs of the system, and to be aware of the browse option, or 
be able to find out. This is not going to be a novice user.

On the other hand, anyone new to the system will have modest goals. 
However those modest goals are frustrated and refuted because the system 
is geared towards that professional that already knows everything, and 
for whom adding nobrowse, or changing the config file, would probably be 
a nobrainer. The new user however may be flabbergasted that it either 
does not work, or not the way she feels it should work (making a list, 
and then not seeing it in action).



>> - browse being turned off by default combined with a complete lack of
>> mention that you need to access these "nonexistent" directory paths /
>> components before you will be able to see them, turns autofs into a
>> complete mystery for someone who is not privy to the details yet.
> 
> It's been a long time for me so I probably have forgotten what it's 
> like for
> someone starting out.
> 
> And so, how would I know where in the man pages to put that so it gets 
> noticed
> by new users, perhaps from your perspective, you could point out a few 
> key
> places where it should be added, or better still send a patch.

This happens a lot, it is no ones fault. I guess people who knows the 
ins and outs of something usually jump smack in the middle, and forget 
to mention what the system is really like from the outside. A fish 
doesn't notice the water, we don't notice the air.

We are too much immersed in it.

I think there are 3-4 things I would explain.

First the auto.master file on my system mentions man autofs as a place 
for seeing the format of the (sub)map files. However there is nothing in 
there. So that part is missing, but in a sense the contents of auto.misc 
is already somewhat sufficient for that, but still annoying that there 
is no man page.

The second part is that it is often hard to understand what is meant by 
"key". In the master file, key means the mount point that supposedly 
already exists in the filesystem. In the sub-maps, key means the 
subdirectory but also the component of the path getting accessed by the 
user. Both types of files are called map files, and both kind of keys 
are called keys. This is aggravated by the fact that there is no man 
page for the sub-maps, that I know of . Auto.smb then speaks of a key, 
but it doesn't say what it is. Of course after a while you realize it is 
to be the host, but then you don't know how it knows about this host. 
Since at that point you don't realize that this is input from the 
filesystem (e.g. a stat operation) on "hidden directories" you are 
really clueless as to where this information is coming from ;-).

You don't imagine that every freaking access of a directory or path 
component in the managed directory is going to result in network access.

You could do a "for f in `seq 1 10000000`; do vdir $f; done" and have 
fun.

(Don't know if that would still be the case for "browse" maps).

So this hidden directory, secret door thing is really the most confusing 
of all.

It's somewhat similar to something I might like to do, but still you 
don't really expect it ;-).

So the 3-4 things are:

create a man page or section for the format of the sub-maps
indicate more clearly that programmatic maps in principle return a 
single entry
indicate that programmatic maps currently require access of a path 
component and that this path component (user action) is the key to the 
script.
indicate that this is true for "no browse" on "static maps" as well, and 
that directories are only created when they are accessed.
And that no-browse is the configured default.

Currently the documentation for the master map and for (indirect) 
sub-maps is really in auto.master and auto.misc.

Reading config files is usually a lot more pleasant than man pages ;-) 
so that's not such a huge problem, but you can't learn anything from a 
very limited set of examples either ;-).

For instance: direct map won't work if not prefixed with /, but will 
also not give error if not. Indirect map (I think) won't work if 
prefixed with /, but will also not give error if not.

I could say more but my thumb is hurting (infected) so I gotta quit 
this.

I don't mind writing documentation. But I'm not familiar yet with man 
pages and their technical structure and all.

I'll have to see. Thank you for the interest in any case :).


>> 
>> 
>> Further I would say that on a desktop OS, the most likely use case is
>> going to be this:
>> 
>> - have several shares in your home network
>> - want to access them through a GUI
>> - prefer automounting since mounting at boot can be problematic
>> - need to see them before you can access them in a GUI
>> - only alternative is "bookmarks" or "places" that work regardless of
>> the things not being visible/listed in the directory.
> 
> Either use a different automounting mechanism or use a direct mount 
> map.
> 
> That would satisfy all these needs, all be it with the map entries 
> always being
> mounted on access.
> 
> The limitation of these is you can't have a direct map path that is 
> within
> another automounted path, if you need to do that you need to use a more
> complicated setup.

I think that limitation indicates why using relative paths is so much 
more pleasant. A map file using relative files can be "moved around". A 
map file using absolute paths cannot.

Could you not make a direct map out of the key of the master file, and 
the keys of the sub-maps?

If there is really a need for huge static map files (I haven't studied 
LDAP) in principle all you need is a syntax to differentiate between the 
two, right.

/-         ( /this/is/my/abs/path )

or

/this/is/my/abs      ( -/path )

could be identical

normally ( /bla/bla ) indicates an absolute path and can only be used in 
a direct map, from what I've seen. You *could* cause it to turn this 
"indirect map" into a direct one.

ie. ( bla/bla ) == indirect
     ( /bla/bla ) == direct.

A direct map is always a static file right? But then it doesn't matter 
what part of the path is specified in what file.

Or

     ( bla/bla ) == indirect
     ( -/bla/bla ) == direct

So you get:

/start/of/the  (-/path )

or

/start/of/the/-  ( path )

Then any path in the master file ending with /- would be a direct path.

And would cause the (static) map file to be treated as direct.



> 
>> 
>> Then I will also say,
>> 
>> that there is no documentation on the format of the map files other 
>> than
>> auto.master. The documentation for map files is this:
> 
> That's not quite right.
> 
> Man page autofs(5) has a bit of information on both the Sun and amd 
> format maps.

Oh I am sorry. I think many users are not versed in the numbers for the 
man pages. I for one do not notice the difference and I almost never use 
one of those numbers. Sorry, I seriously missed that.

I only noticed just now because autofs(8) references autofs(5), and I'm 
like ....huh?

Okay, allright. Missed that.

Seriously bad system if you ask me, but okay.

So I guess you can have direct maps with LDAP etc. too.

I guess the wildcard also solves the many-user problem.

Well, but only for the sub-map. You couldn't get autofs to manage 
/media, so it needs to be /media/user/<aggregate> in which user can't 
suddenly be a wildcard unless you use a direct map. You could expand 
that to include the master file though:

/media/*/nas    /etc/nas-shares.map

And then in /etc/nas-shares.map:

share1   -fstype=xxx,uid=&   ://nas/share1
share2   -fstype=xxx,uid=&   ://nas/share2

I can't see whether LDAP supports browse, I assume it must, since it 
supports direct.




> By definition a program map can't know what keys may be passed to it.
> And, yes, even autofs(5) doesn't say explicitly that a program map 
> doesn't know
> what keys may be passed to it, perhaps that should be ammended.

In principle even a program map could read all of the available keys 
from whatever database it consults, and return all of them; or return 
something only (as it does now, perhaps) when a given key matches (is 
found).

If you wanted that you would need the program script to e.g. support a 
wildcard as parameter. Meaning, all scripts must then check for "*" to 
be passed to it, in order to return the full list that they can acquire 
(if even possible) or at least return an error of some kind to say that 
they don't support it.

You'd need a parameter for it, and any script would need to be aware of 
that parameter.

That's either gonna be a wildcard or something given on $2, in that case 
every existing script would already be compatible with it, since it 
simply disregards $2.

So, for instance, if $2 is "browse" the script may choose to return a 
browse map, or list of shares, with whatever options are available (it 
could even return the UID/USER of the share given options, so that the 
created directory would have the same UID as the to be mounted share!) 
-- meaning it might simply return ALL entries it could possibly return 
in the default format, which is the meaning of the wildcard, and the 
program (autofs daemon or module(?)) would be able to generate the 
appropriate directory entries based on that. So in that case you're 
probably going to be rid of the problem of the non-matching state of 
pre- and post-mount of directories.


>> a) get rid of the browse_mode = no, in the configuration file, for
>> static mounts
> 
> No, that has it's own set of problems which people need to be aware of 
> before
> choosing to use the "browse" option.

Not everyone.

I have like 5 mounts in this share. 5 shares in this mount. I have not 
noticed issues to begin with. Maybe there are issues like checking 
permissions on the directory prior to mounting it, and then being wrong 
about that once it is mounted. But still, that doesn't seem to happen. 
Actually, something funny is going on. 2 of the 8 shares that are 
'automatically' loaded are not accessible. "No such file or directory".

No, that's nothing but a permission error. Still I am allowed to see 
them but not use them. For a static file no issue; just delete them.

I don't really see what I need to be aware of prior to using browse?

If it is the most sensible "novice" use case, why not allow it by 
default? It things go mayhem people can turn it off, or you could even 
turn it off when the number of mounts in a map exceeds 20 or 30 or 50. 
It's an arbitrary number, but what if I am wrong about LDAP.

What about you mean it is turned off by default for LDAP, but you do not 
consider static files with that?

What about it does function for ldap. What about we just need a separate 
default for LDAP and its like, than we do for static?

Are you seriously considering static files, or do you mean LDAP, yp, 
hisplus, and hesoid?

For anyone using an actual database, I concur.

For static, I just disagree. That is the start case for anyone.





> 
>> b) say something about mounts not being created automatically in the 
>> man
>> page for auto.master, if nothing else. If you also do (a), this only
>> applies now to programmatic mounts.
> 
> auto.master(5) isn't the place for that, autofs(5) is better and, as I 
> say, make
> some suggestions of provide a patch.

autofs(8) is loaded before autofs(5). It's a good document, autofs(5), 
but most people may not even notice it. Personally I would probably 
choose to call them:

autofs.master(5) and autofs.map(5). But then, I also consider 
auto.master a very nondescript name.

In this way no one can ever get confused, so you might just rename 
autofs(5) to autofs.map(5).

Or even auto.map(5) if that is necessary, but I wouldn't do that.


> I'll have a look at doing this but beware, often distributions use 
> there own
> default master map which I have no control over.

I have no clue what mine does, this is Debian, and then Ubuntu, and then 
Kubuntu ;-) but it is just the Debian package still.

You don't have to follow my example or suggestion, maybe there is a 
better improvement than this, it was just an example, I guess. Or I 
could guess.

That was just a thing I was able to change without editing man files 
;-).

I will look into providing a patch, but I hope you can rename autofs(5) 
to autofs.map(5) before doing anything like that. I don't feel like 
improving a system I know myself haven't had access to, in that sense. I 
don't mean you have to do that straight away, but if that is a possible 
direction to go, or change to make, then I could supply a patch or 
anything based on the system that would result from that, seeing that 
autofs(5) is already pretty good in that sense.

>> I mean you could rewrite more documentation but this is the simplest I
>> could do right here right now.
>> 
>> I think that would clarify a lot for everyone but of course perhaps 
>> you
>> would want to rewrite more so the central documentation is also up to
>> date, but please see this as a more modest attempt for making it more
>> accessible to a new user.
> 
> Yes, adding the fact that mount point directories aren't pre created 
> for
> indirect maps somewhere in auto.master(5) and in a couple of places in 
> autofs(5)
> is a good idea. You are right it's something that can catch the 
> unaware.

That would also make the difference between direct and indirect more 
clear. (I still don't know exactly what is the entire difference).

That's also something I could change: how a direct map is explained in 
auto.master(5). But then, I didn't have access to autofs(5) to explain 
it to me better ;-).

Anyway, I think it is a good project, I didn't know autofs would be so 
good when it comes down to it.

It's usually documentation that puts you off ;-).

I often feel like I have to find the door in some invisible wall.

This time it was extreme because I couldn't for the life of me 
understand what a "key" would be in the sub-map file.

Regards, Bart.
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Ian Kent]

On Wed, 2016-07-06 at 06:47 +0200, Xen wrote:
> Ian Kent schreef op 06-07-2016 2:57:
> > On Tue, 2016-07-05 at 23:17 +0200, Xen wrote:
> > > Hi,
> > > 
> > > a few things.
> > > 
> > > A user attempting to use autofs for the first time will assume that
> > > autofs will mount mount poins that already exist. Nowhere is it stated
> > > that mount points will only get created when accessed (through a 
> > > "direct
> > > path").
> > 
> > Umm, "direct path" is an unfortunate choice here as you are talking 
> > about autofs
> > indirect mount maps.
> > 
> > Direct mount maps create a mount trigger for each map entry and so show 
> > up as
> > mount points in the file system.
> > 
> > There are reasons for it being this way.
> 
> So you are berating me for choice of words ;-).

No, I'm trying to establish a common naming convention so both of us can avoid
possible confusion later.

There are a bunch of conventions used (at least I use) that have specific
meaning to me in relation to autofs. If I don't jump on those early there is
often confusion about what I'm trying to say and there can be misunderstandings
even when I do.  

> 
> The only difference between direct and indirect seems to be that with 
> direct the share is mounted "directly" on top of the specified mount 
> point (actually, none is specified, it is mounted on the key returned 
> from the map) and as a result it is not possible to have lazy 
> initialization (unless you merged e.g. the root filesystem with an 
> overlay filesystem) and so you are required to create this mountpoint, 
> whereas with "indirect" you control the mount point itself and can 
> choose whether to create the subdirectories or not (if you can acquire a 
> list). "Directly" seems to mean "directly on the filesystem" whereas 
> "indirectly" seems to mean: inside our own little enclave.

Yes, that's about it.

But I don't understand your use of "lazy initialization".

Yes, autofs will create a mount point path for each entry in a direct mount map.
And yes, there will be an autofs file system mounted on each path to act as a
trigger to cause the corresponding file system to be mounted during a kernel
path walk (ie. when accessed).

So I don't quite understand what your concern is.
Sure, if you have hundreds or thousands of these there can be a performance
problem but that's not what you were trying to do I thought.

It sounded to me like these were a good fit for your case .....

> 
> And always as a subdirectory of that. Not even technically necessary, 
> you could even have a "indirect map" that triggered mounting I guess, 
> don't know. I guess not.
> 
> 
> 
> > 
> > > 
> > > Such a user (such as me) trying to use this thing (even with a 
> > > tutorial
> > > on Ubuntu site) will be flabbergasted that nothing happens; he/she
> > > expects mount entries to appear, but nothing happens.
> > > 
> > > This renders the "nobrowse" default dysfunctional for a desktop user,
> > > who expects to see something before going there. It only works for
> > > automated systems, really.
> > 
> > Your assuming that autofs primary usage is targeted at the desktop.
> > It's often not the best use case for that.
> > 
> > There are other more suitable automount alternatives for GUI desktops.
> 
> Linus Torvalds says he developed Linux as a desktop OS. And how 
> chagrined he is that it didn't succeed in just that. Well, now we know 
> the reason for that ;-).
> 
> I know there are probably alternatives; but they are mostly useless for 
> someone who wants to do it on his own. Desktop environments often do not 
> even have mount operations on directories (even) when they are described 
> in fstab. There is no other way that I know of to automatically mount 
> e.g. samba shares, unless you navigate directly to it (and then they end 
> up in some weird location in the filesystem, if at all). There is 
> usually also no "user based" fstab; where you can describe mounts 
> specifically for your user, that are then mounted through e.g. fuse, or 
> something else that can do suid. Perhaps you mean udisks or udisksd or 
> udisksctl but I'm not even sure that supports network-filesystems, they 
> are certainly not shown in udisksctl status.
> 
> Dolphin can not actually mount a share to some fixed location, the Nemo 
> I am using doesn't even succeed in browsing the network. For me there 
> are only two solutions that I see, either fstab or autofs. Once you know 
> how to do it, running /etc/auto.smb <host> quickly returns a list of 
> shares (if you've set the credentials file) that you can then edit to 
> become a static map with the browse option.
> 
> I do not see what is better suited currently to auto-mounting 
> samba-shares. Anything that would work well, would work from both the 
> shell and the GUI, in that the systems have to be linked, and ideally 
> what the GUI does is stored in state files that work equally well from 
> the shell. Any persistent mount would have to be stored somewhere on 
> disk. Then the link would need to be automounted, OR mounted through 
> e.g. libpam-mount. There is nothing wrong with mounting upon logon. And 
> that allows for specific user-dependent stuff. (An automount depending 
> on what user first accesses it, is problematic). (As for Samba). So 
> perhaps logon-mounts are better suited (just restore the mount from the 
> previous session, in that sense) because the $UID problem is hard to 
> solve for autofs, you must either duplicate all entries, and hardcore 
> the UIDs you need, for every user, or I don't know what, prevent other 
> users (including root) from accessing the share you need. You'd 
> basically need map files in auto.master.d for every user, including a 
> map file for each person to specify the shares, if you want browsable, 
> or a modified version of auto.smb that mounts individual shares when 
> accessed. (In the latter case you'd have no browse map).
> 
> But this script doesn't know what it is being called on, so now you need 
> a variable I guess for each of those .d/ files, so that their entries 
> would be something like:
> 
> /media/<user>/nas  /etc/auto-short.smb -Duser=<user> which would then be 
> used for the samba $UID.
> 
> If there is a better way to do this, I don't know. I see no alternatives 
> other than libpam-mount and of course some GUI could do it, but then 
> you'd really just have a form of duplicated functionality between autofs 
> and it.
> 
> 
> > > - browse only works for static maps, and what person with a static map
> > > is going to run into performance issues when mount points are created 
> > > in
> > > advance? Can we please remove this from the default config? (turning 
> > > it
> > > off).
> > 
> > Well, quite a lot of current users actually.
> > 
> > Consider what can happen with a few hundred or a few thousand entries 
> > in an
> > indirect map.
> 
> Alright, but what indirect maps can cause browse tables to be shown? 
> Only static maps, right? A programmatic map can only return a single 
> entry. I have not looked into the other ones, but I suppose LDAP is more 
> like a programmatic map than anything else.

Of course, yes, program maps can never know in advance what key may be passed to
them.

The problem is usually seen with user applications, cron jobs that scan
directories, GUI file management utilities etc.

It's has been quite a problem over time.

LDAP is a map source, like a file, or nis etc.

Maps stored in LDAP behave much the same as maps stored in files.

> 
> So then consider that we only use a static map here. An actual file with 
> thousands of entries? Of course it can happen. But what would be the 
> point of that still (as opposed to LDAP or programmatic).

People do seem to like program maps but plenty of people use static or semi
-static maps, ie. a bunch of entries followed by the wildcard map key "*"
usually with some macro or "&" substution.

Similarly, what might match the wildcard map entry can't be known in advance.

> 
> 
> > Even though the browse option was added it has a specific set of 
> > problems.
> > 
> > First the kernel does not know if you are just listing a directory or 
> > actually
> > need to mount something.
> 
> Do you mean the directory containing the (indirect) entries (e.g. the 
> managed directory) or one of those entries themselves?

It's hard enough for me to understand VFS path walking sufficiently without
having to describe it too.

All I can say is there is a problem determining whether a call back to the
deamon is needed or not at the times the kernel module is called. That can
sometimes result in all entries in the autofs managed mount point directory
being mounted.

Another aspect of this problem is that the same system call may need to have the
directory trigger a mount while not so at other times it's called and there's no
way to tell what the usage is within in the kernel.

So sometimes people get caught because they want their stat(2) call to return
the mounted file system results but not to cause a mount at other times when it
might lead to a mount storm.

These people are usually quite passionate as well insisting that it be fixed and
sometimes refuse to accept it is a problem.

Usuall all I can say to them is don't use the "browse" option.....

It is a difficult problem and I'm inevitably caught in the middle between the
"browse" and "nobrowse" camps.

Not all cases of accessing such a directory will see this problem either but
user space seems to have a habit of using system calls that don't trigger a
mount together with others that do.

Causing the so called mount storm.

Having said that though, with a lot of work, I could probably do something about
it.

TBH I've avoided it because it is difficult, perhaps in version 6.

It would require not pre-creating mount point directories and adding a readdir
implementation to the kernel to fill in the names (map keys) that aren't already
mounted. But that means significantly more (in terms of quantity of)
communication between the daemon and the kernel module.

That means re-writing the user space <-> kernel space communications completely
in an incompatible manner to what is used now, not to mention significant
changes to the user space code itself.

A big job really.

> 
> > So at the moment, certain types of system call (the module can't know 
> > the exact
> > system call either) don't cause a call back to perform a mount but 
> > often user
> > space will later make a system call that will cause a mount to occur 
> > anyway.
> > 
> > That leads to sometimes seeing directories that are those of the autofs 
> > file
> > system not those of the mounted file system, so they have completely 
> > different
> > attributes to those after an automount has been done, a problem in 
> > itself.
> 
> Oh... such as ownership etc.
> 
> The typical "I mount this, and now suddenly it is not root.root but 
> xen.staff".
> 
> > And, it can often lead to a mount storm, mounting everything in the 
> > directory.
> 
> But how can that happen? If I do; "for f in *; ls $f; done" sure.

It happens, from what I've seen, quite a bit.
Perhaps you would be surprised.

> 
> 
> > 
> > OK for a few dozen mounts in a map but really bad for anything with 
> > more than a
> > few hundred entries.
> 
> After thinking a little I came up with what could perhaps be some 
> default use case. You have terminals mounting home directories. On the 
> "terminal machine" you may have just a single user. Through LDAP you 
> find the home directory. There is obviously no sense for all users in 
> the LDAP system to now have precreated mount points. The only thing that 
> could make sense is to have an "invisible" user directory that now gets 
> mounted.

Yes, you would need to be using the default "nobrowse", or a limited set of keys
with a wildcard entry.

> 
> I can see no reason to have hundreds of mount points in a directory 
> unless they cannot conform to a specific pattern. If they follow a 
> pattern, it is likely that only a few of them need to be mounted at any 
> time. In that case being able to browse it seems nonsensical. An 
> aggregate of hundreds of different mounts at the same time seems very 
> weird. Weird to have a diverse set of unique names to begin with. As if 
> all of them were manually added in some way. If they aren't manually 
> added, but automatically, then why have a static list? Would you not be 
> better off putting that in a database? And if you'd have so many, we'd 
> assume they could be different users, or different websites, etc. We 
> also assume not all of them are going to be needed, just a subset. But 
> so now you have a system on which this list is statically created but 
> dynamically updated. This list sits on the disk of the local system. But 
> you're not querying a database (could be just as easy) that could also 
> store this list and return the entries one by one when needed. The 
> static file is the database.
> 
> Seems an improper use case but yeah. I don't know how you can have a 
> static list that is huge while not choosing to use a database to manage 
> it. Because a static map; now you need routines to update it, delete 
> entries from it.

Yes, but network sources such as NIS and LDAP are treated much the same as a
file map.

Checking if a file map has been updated since a key was last used is easier with
a file map and straight forward with a NIS map but can't sensibly be done for an
LDAP map.

> 
> Maybe I'm missing something here.
> 
> 
> > Another reason for the default is that, because of this, historically 
> > there was
> > no "browse" option and this maintains what is (should be) expected.
> 
> Yeah I thought so, because of the comments near the timeout value. 
> However that is a reason to never change anything ever. Someone 
> mistakenly choosing "browse" will very quickly find out.

And wouldn't itself be sufficient but for the problems I'm attempting to
describe above.

> 
> If this person really has hundreds of items in his list, he will notice 
> within a second that they are all visible. If this person really has 
> hundreds to thousands of items, he or she may also be expected to know 
> the ins and outs of the system, and to be aware of the browse option, or 
> be able to find out. This is not going to be a novice user.
> 
> On the other hand, anyone new to the system will have modest goals. 
> However those modest goals are frustrated and refuted because the system 
> is geared towards that professional that already knows everything, and 
> for whom adding nobrowse, or changing the config file, would probably be 
> a nobrainer. The new user however may be flabbergasted that it either 
> does not work, or not the way she feels it should work (making a list, 
> and then not seeing it in action).
> 
> 
> 
> > > - browse being turned off by default combined with a complete lack of
> > > mention that you need to access these "nonexistent" directory paths /
> > > components before you will be able to see them, turns autofs into a
> > > complete mystery for someone who is not privy to the details yet.
> > 
> > It's been a long time for me so I probably have forgotten what it's 
> > like for
> > someone starting out.
> > 
> > And so, how would I know where in the man pages to put that so it gets 
> > noticed
> > by new users, perhaps from your perspective, you could point out a few 
> > key
> > places where it should be added, or better still send a patch.
> 
> This happens a lot, it is no ones fault. I guess people who knows the 
> ins and outs of something usually jump smack in the middle, and forget 
> to mention what the system is really like from the outside. A fish 
> doesn't notice the water, we don't notice the air.
> 
> We are too much immersed in it.
> 
> I think there are 3-4 things I would explain.
> 
> First the auto.master file on my system mentions man autofs as a place 
> for seeing the format of the (sub)map files. However there is nothing in 
> there. So that part is missing, but in a sense the contents of auto.misc 
> is already somewhat sufficient for that, but still annoying that there 
> is no man page.

That doesn't sound right, there might be a problem with your distribution
package.

[raven@pluto autofs-dev.git]$ wc -l man/autofs.5
611 man/autofs.5
[raven@pluto autofs-dev.git]$ wc -l man/autofs.8.in 
81 man/autofs.8.in

The first of these describes the Sun and amd map format map entries, not a
particularly good job of it I admit, but that is where the changes you recommend
should go.

The second describes the service start and stop, etc.

> 
> The second part is that it is often hard to understand what is meant by 
> "key". In the master file, key means the mount point that supposedly 
> already exists in the filesystem. In the sub-maps, key means the 
> subdirectory but also the component of the path getting accessed by the 
> user. Both types of files are called map files, and both kind of keys 
> are called keys. This is aggravated by the fact that there is no man 
> page for the sub-maps, that I know of . Auto.smb then speaks of a key, 
> but it doesn't say what it is. Of course after a while you realize it is 
> to be the host, but then you don't know how it knows about this host. 
> Since at that point you don't realize that this is input from the 
> filesystem (e.g. a stat operation) on "hidden directories" you are 
> really clueless as to where this information is coming from ;-).

Sure, and that probably needs to be spelled out in autofs(5).
I know people do get confused by this distinction.

> 
> You don't imagine that every freaking access of a directory or path 
> component in the managed directory is going to result in network access.
> 
> You could do a "for f in `seq 1 10000000`; do vdir $f; done" and have 
> fun.
> 
> (Don't know if that would still be the case for "browse" maps).
> 
> So this hidden directory, secret door thing is really the most confusing 
> of all.
> 
> It's somewhat similar to something I might like to do, but still you 
> don't really expect it ;-).
> 
> So the 3-4 things are:
> 
> create a man page or section for the format of the sub-maps

Already exists, needs work.
Lets call these mount maps (not the best but the convention used) as opposed to
the master map.

> indicate more clearly that programmatic maps in principle return a 
> single entry

Maybe but isn't this in autofs(5) already close:

   Executable Maps
       A map can be marked as executable. A program map will be called
       with the key as an argument. It may return no lines of output
       if there's an error, or one or more lines containing a map entry
       (with \ quoting line breaks). The map entry corresponds to what
       would normally follow a map key.

> indicate that programmatic maps currently require access of a path 
> component and that this path component (user action) is the key to the 
> script.

But that is the case for all maps:

Isn't this near the top of autofs(5) sufficient:

       key [-options] location

       key
              For indirect mounts this is the part of the path
              name between the mount point and the path into
              the filesystem when it is mounted. Usually you
              can think about the key as a sub-directory name
              below the autofs managed mount point.

              For direct mounts this is the full path of each
              mount point. This map is always associated with
              the /- mount point in the master map.


> indicate that this is true for "no browse" on "static maps" as well, and 
> that directories are only created when they are accessed.

That seems to be missing from the man page.

> And that no-browse is the configured default.

Not sure about that either.

This (from auto.master(5)) says that I have made "browse" the internal program
default way back when version 5 was first developed but I install a
configuration that turns it off because of (because of the above problems, which
are not spelled out in the man page):

     [no]browse
            This is an autofs specific option that is a pseudo mount
            option and so is given without a leading dash. Use of the
            browse option pre-creates mount point directories for
            indirect mount maps so the map keys can be seen in a
            directory listing without being mounted. Use of this
            option can cause  performance problem if the indirect
            map is large so it should be used with caution. The
            internal program default is to enable browse mode for
            indirect mounts but the default installed configuration
            overrides this by setting BROWSE_MODE to "no" because
            of the potential performance problem.

Don't forget that distributions can, and often do, install their own version of
auto.master and possibly the autofs configuration.

And looking at the man pages shows that I have missed changing some of the
configuration option names from the recent configuration file changes, something
else that needs to be done.

> 
> Currently the documentation for the master map and for (indirect) 
> sub-maps is really in auto.master and auto.misc.
> 
> Reading config files is usually a lot more pleasant than man pages ;-) 
> so that's not such a huge problem, but you can't learn anything from a 
> very limited set of examples either ;-).
> 
> For instance: direct map won't work if not prefixed with /, but will 
> also not give error if not. Indirect map (I think) won't work if 
> prefixed with /, but will also not give error if not.
> 
> I could say more but my thumb is hurting (infected) so I gotta quit 
> this.
> 
> I don't mind writing documentation. But I'm not familiar yet with man 
> pages and their technical structure and all.
> 
> I'll have to see. Thank you for the interest in any case :).
> 
> 
> > > 
> > > 
> > > Further I would say that on a desktop OS, the most likely use case is
> > > going to be this:
> > > 
> > > - have several shares in your home network
> > > - want to access them through a GUI
> > > - prefer automounting since mounting at boot can be problematic
> > > - need to see them before you can access them in a GUI
> > > - only alternative is "bookmarks" or "places" that work regardless of
> > > the things not being visible/listed in the directory.
> > 
> > Either use a different automounting mechanism or use a direct mount 
> > map.
> > 
> > That would satisfy all these needs, all be it with the map entries 
> > always being
> > mounted on access.
> > 
> > The limitation of these is you can't have a direct map path that is 
> > within
> > another automounted path, if you need to do that you need to use a more
> > complicated setup.
> 
> I think that limitation indicates why using relative paths is so much 
> more pleasant. A map file using relative files can be "moved around". A 
> map file using absolute paths cannot.
> 
> Could you not make a direct map out of the key of the master file, and 
> the keys of the sub-maps?

Can't really do that.

The closest thing to that would be multi-mount map entries, again from
autofs(5):

   Multiple Mounts
       A multi-mount map can be used to name multiple filesystems
       to mount. It takes the form:

       key [ -options ] \
           [[/] location \
          [/relative-mount-point [ -options ] location...]...

       This may extend over multiple lines, quoting the line-breaks with
       `\´.  If present, the per-mountpoint mount-options are appended
       to the default mount-options. This behaviour may be overridden
       by the append_options configuration setting.


> If there is really a need for huge static map files (I haven't studied 
> LDAP) in principle all you need is a syntax to differentiate between the 
> two, right.
> 
> /-         ( /this/is/my/abs/path )
> 
> or
> 
> /this/is/my/abs      ( -/path )
> 
> could be identical

Not really, the master map key "/-" is required to specify the direct mount maps
that are to be used, anything else is an indirect mount, and "/-" is only valid
within the master map (although I can't remember if I explicitly check for it
within mount maps).

> 
> normally ( /bla/bla ) indicates an absolute path and can only be used in 
> a direct map, from what I've seen. You *could* cause it to turn this 
> "indirect map" into a direct one.
> 
> ie. ( bla/bla ) == indirect
>      ( /bla/bla ) == direct.

In the Sun map format maps indirect mount map keys are a single path component
only.

The situation is different if using amd format maps but I could work out how
direct mount maps could be usefully used so that mount type has not been
implemented. Even on Solaris where they are supposed to be fully supported I
could make sense of them.

> 
> A direct map is always a static file right? But then it doesn't matter 
> what part of the path is specified in what file.
> 
> Or
> 
>      ( bla/bla ) == indirect
>      ( -/bla/bla ) == direct
> 
> So you get:
> 
> /start/of/the  (-/path )
> 
> or
> 
> /start/of/the/-  ( path )
> 
> Then any path in the master file ending with /- would be a direct path.
> 
> And would cause the (static) map file to be treated as direct.

Nope, that is unlikely to work and would lead to a lot of problems.

One example might be with:

/start/of/the/-/path

people will ask where have the directories below /start/of/the/ gone and the
answer will be they have been covered by the autofs file system mount.

Next it would be but I should be able to see these, surely that's easy to fix.

At hat point I would probably leave the conversation.

In any case a similar thing, but more explicit and I think easier to follow and
maintain are the autofs multi-mount map entries, what I think you called
relative path above.

It isn't clear but that syntax should function within both direct and indirect
mount maps.

The problem with all this is that substitution cannot be used in the key.

The reason for that is same as why the amd regex key matching has not been
implemented.

A linear map scan would be needed for every lookup in order to match what has
been sent to us from the kernel. And as much as you might feel that wouldn't be
a problem because maps are generally small I can assure you it would make like
really unpleasant for me!

> 
> 
> 
> > 
> > > 
> > > Then I will also say,
> > > 
> > > that there is no documentation on the format of the map files other 
> > > than
> > > auto.master. The documentation for map files is this:
> > 
> > That's not quite right.
> > 
> > Man page autofs(5) has a bit of information on both the Sun and amd 
> > format maps.
> 
> Oh I am sorry. I think many users are not versed in the numbers for the 
> man pages. I for one do not notice the difference and I almost never use 
> one of those numbers. Sorry, I seriously missed that.
> 
> I only noticed just now because autofs(8) references autofs(5), and I'm 
> like ....huh?

Yeah, that's what we have ....

> 
> Okay, allright. Missed that.
> 
> Seriously bad system if you ask me, but okay.
> 
> So I guess you can have direct maps with LDAP etc. too.
> 
> I guess the wildcard also solves the many-user problem.
> 
> Well, but only for the sub-map. You couldn't get autofs to manage 
> /media, so it needs to be /media/user/<aggregate> in which user can't 
> suddenly be a wildcard unless you use a direct map. You could expand 
> that to include the master file though:
> 
> /media/*/nas    /etc/nas-shares.map

Can't do that, each /media/*/nas needs to be mounted at program start so that
automounts can be done?

Chicken before the egg problem there.

You might be able to achieve something sensible with a wildcard and offset
mounts (the relative paths above) and macro and & substution.

It is a bit difficult though.

> 
> And then in /etc/nas-shares.map:
> 
> share1   -fstype=xxx,uid=&   ://nas/share1
> share2   -fstype=xxx,uid=&   ://nas/share2
> 
> I can't see whether LDAP supports browse, I assume it must, since it 
> supports direct.

Yep, same as file maps except remote maps cannot use plus map inclusion, like
the
+auto.master
you might see in default installed master maps.

> 
> 
> 
> 
> > By definition a program map can't know what keys may be passed to it.
> > And, yes, even autofs(5) doesn't say explicitly that a program map 
> > doesn't know
> > what keys may be passed to it, perhaps that should be ammended.
> 
> In principle even a program map could read all of the available keys 
> from whatever database it consults, and return all of them; or return 
> something only (as it does now, perhaps) when a given key matches (is 
> found).

I've thought of that too.

But I've resisted because I'm concerned about confusing existing program maps
people have in unpleasant ways.

But that suggestion has been made before and .....

> 
> If you wanted that you would need the program script to e.g. support a 
> wildcard as parameter. Meaning, all scripts must then check for "*" to 
> be passed to it, in order to return the full list that they can acquire 
> (if even possible) or at least return an error of some kind to say that 
> they don't support it.
> 
> You'd need a parameter for it, and any script would need to be aware of 
> that parameter.
> 
> That's either gonna be a wildcard or something given on $2, in that case 
> every existing script would already be compatible with it, since it 
> simply disregards $2.

That's not a bad thought, but what would $1 be so that older scripts wouldn't
choke on it.

> 
> So, for instance, if $2 is "browse" the script may choose to return a 
> browse map, or list of shares, with whatever options are available (it 
> could even return the UID/USER of the share given options, so that the 
> created directory would have the same UID as the to be mounted share!) 
> -- meaning it might simply return ALL entries it could possibly return 
> in the default format, which is the meaning of the wildcard, and the 
> program (autofs daemon or module(?)) would be able to generate the 
> appropriate directory entries based on that. So in that case you're 
> probably going to be rid of the problem of the non-matching state of 
> pre- and post-mount of directories.

which pretty much defeats the only possible solution I can think of for "browse"
option mounts ...

> 
> 
> > > a) get rid of the browse_mode = no, in the configuration file, for
> > > static mounts
> > 
> > No, that has it's own set of problems which people need to be aware of 
> > before
> > choosing to use the "browse" option.
> 
> Not everyone.
> 
> I have like 5 mounts in this share. 5 shares in this mount. I have not 
> noticed issues to begin with. Maybe there are issues like checking 
> permissions on the directory prior to mounting it, and then being wrong 
> about that once it is mounted. But still, that doesn't seem to happen. 
> Actually, something funny is going on. 2 of the 8 shares that are 
> 'automatically' loaded are not accessible. "No such file or directory".
> 
> No, that's nothing but a permission error. Still I am allowed to see 
> them but not use them. For a static file no issue; just delete them.
> 
> I don't really see what I need to be aware of prior to using browse?
> 
> If it is the most sensible "novice" use case, why not allow it by 
> default? It things go mayhem people can turn it off, or you could even 
> turn it off when the number of mounts in a map exceeds 20 or 30 or 50. 
> It's an arbitrary number, but what if I am wrong about LDAP.
> 
> What about you mean it is turned off by default for LDAP, but you do not 
> consider static files with that?
> 
> What about it does function for ldap. What about we just need a separate 
> default for LDAP and its like, than we do for static?
> 
> Are you seriously considering static files, or do you mean LDAP, yp, 
> hisplus, and hesoid?
> 
> For anyone using an actual database, I concur.
> 
> For static, I just disagree. That is the start case for anyone.

Like I said, I wanted to have the default as "browse" but got hit with too many
complaints and was only willing to change the package installed default
configuration to satisfy them.

I'm not going to try that again.

> 
> 
> 
> > 
> > > b) say something about mounts not being created automatically in the 
> > > man
> > > page for auto.master, if nothing else. If you also do (a), this only
> > > applies now to programmatic mounts.
> > 
> > auto.master(5) isn't the place for that, autofs(5) is better and, as I 
> > say, make
> > some suggestions of provide a patch.
> 
> autofs(8) is loaded before autofs(5). It's a good document, autofs(5), 
> but most people may not even notice it. Personally I would probably 
> choose to call them:
> 
> autofs.master(5) and autofs.map(5). But then, I also consider 
> auto.master a very nondescript name.
> 
> In this way no one can ever get confused, so you might just rename 
> autofs(5) to autofs.map(5).

Yeah maybe ...

> 
> Or even auto.map(5) if that is necessary, but I wouldn't do that.
> 
> 
> > I'll have a look at doing this but beware, often distributions use 
> > there own
> > default master map which I have no control over.
> 
> I have no clue what mine does, this is Debian, and then Ubuntu, and then 
> Kubuntu ;-) but it is just the Debian package still.
> 
> You don't have to follow my example or suggestion, maybe there is a 
> better improvement than this, it was just an example, I guess. Or I 
> could guess.
> 
> That was just a thing I was able to change without editing man files 
> ;-).
> 
> I will look into providing a patch, but I hope you can rename autofs(5) 
> to autofs.map(5) before doing anything like that. I don't feel like 
> improving a system I know myself haven't had access to, in that sense. I 
> don't mean you have to do that straight away, but if that is a possible 
> direction to go, or change to make, then I could supply a patch or 
> anything based on the system that would result from that, seeing that 
> autofs(5) is already pretty good in that sense.

At some point I'll need to return to this and make a list.
Unfortunately that time isn't now.

> 
> > > I mean you could rewrite more documentation but this is the simplest I
> > > could do right here right now.
> > > 
> > > I think that would clarify a lot for everyone but of course perhaps 
> > > you
> > > would want to rewrite more so the central documentation is also up to
> > > date, but please see this as a more modest attempt for making it more
> > > accessible to a new user.
> > 
> > Yes, adding the fact that mount point directories aren't pre created 
> > for
> > indirect maps somewhere in auto.master(5) and in a couple of places in 
> > autofs(5)
> > is a good idea. You are right it's something that can catch the 
> > unaware.
> 
> That would also make the difference between direct and indirect more 
> clear. (I still don't know exactly what is the entire difference).
> 
> That's also something I could change: how a direct map is explained in 
> auto.master(5). But then, I didn't have access to autofs(5) to explain 
> it to me better ;-).
> 
> Anyway, I think it is a good project, I didn't know autofs would be so 
> good when it comes down to it.
> 
> It's usually documentation that puts you off ;-).
> 
> I often feel like I have to find the door in some invisible wall.
> 
> This time it was extreme because I couldn't for the life of me 
> understand what a "key" would be in the sub-map file.
> 
> Regards, Bart.
> --
> To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Erwan Loaec]

unsubscribe autof


Le 06/07/2016 à 14:48, Ian Kent a écrit :
> On Wed, 2016-07-06 at 06:47 +0200, Xen wrote:
>> Ian Kent schreef op 06-07-2016 2:57:
>>> On Tue, 2016-07-05 at 23:17 +0200, Xen wrote:
>>>> Hi,
>>>>
>>>> a few things.
>>>>
>>>> A user attempting to use autofs for the first time will assume that
>>>> autofs will mount mount poins that already exist. Nowhere is it stated
>>>> that mount points will only get created when accessed (through a
>>>> "direct
>>>> path").
>>> Umm, "direct path" is an unfortunate choice here as you are talking
>>> about autofs
>>> indirect mount maps.
>>>
>>> Direct mount maps create a mount trigger for each map entry and so show
>>> up as
>>> mount points in the file system.
>>>
>>> There are reasons for it being this way.
>> So you are berating me for choice of words ;-).
> No, I'm trying to establish a common naming convention so both of us can avoid
> possible confusion later.
>
> There are a bunch of conventions used (at least I use) that have specific
> meaning to me in relation to autofs. If I don't jump on those early there is
> often confusion about what I'm trying to say and there can be misunderstandings
> even when I do.
>
>> The only difference between direct and indirect seems to be that with
>> direct the share is mounted "directly" on top of the specified mount
>> point (actually, none is specified, it is mounted on the key returned
>> from the map) and as a result it is not possible to have lazy
>> initialization (unless you merged e.g. the root filesystem with an
>> overlay filesystem) and so you are required to create this mountpoint,
>> whereas with "indirect" you control the mount point itself and can
>> choose whether to create the subdirectories or not (if you can acquire a
>> list). "Directly" seems to mean "directly on the filesystem" whereas
>> "indirectly" seems to mean: inside our own little enclave.
> Yes, that's about it.
>
> But I don't understand your use of "lazy initialization".
>
> Yes, autofs will create a mount point path for each entry in a direct mount map.
> And yes, there will be an autofs file system mounted on each path to act as a
> trigger to cause the corresponding file system to be mounted during a kernel
> path walk (ie. when accessed).
>
> So I don't quite understand what your concern is.
> Sure, if you have hundreds or thousands of these there can be a performance
> problem but that's not what you were trying to do I thought.
>
> It sounded to me like these were a good fit for your case .....
>
>> And always as a subdirectory of that. Not even technically necessary,
>> you could even have a "indirect map" that triggered mounting I guess,
>> don't know. I guess not.
>>
>>
>>
>>>> Such a user (such as me) trying to use this thing (even with a
>>>> tutorial
>>>> on Ubuntu site) will be flabbergasted that nothing happens; he/she
>>>> expects mount entries to appear, but nothing happens.
>>>>
>>>> This renders the "nobrowse" default dysfunctional for a desktop user,
>>>> who expects to see something before going there. It only works for
>>>> automated systems, really.
>>> Your assuming that autofs primary usage is targeted at the desktop.
>>> It's often not the best use case for that.
>>>
>>> There are other more suitable automount alternatives for GUI desktops.
>> Linus Torvalds says he developed Linux as a desktop OS. And how
>> chagrined he is that it didn't succeed in just that. Well, now we know
>> the reason for that ;-).
>>
>> I know there are probably alternatives; but they are mostly useless for
>> someone who wants to do it on his own. Desktop environments often do not
>> even have mount operations on directories (even) when they are described
>> in fstab. There is no other way that I know of to automatically mount
>> e.g. samba shares, unless you navigate directly to it (and then they end
>> up in some weird location in the filesystem, if at all). There is
>> usually also no "user based" fstab; where you can describe mounts
>> specifically for your user, that are then mounted through e.g. fuse, or
>> something else that can do suid. Perhaps you mean udisks or udisksd or
>> udisksctl but I'm not even sure that supports network-filesystems, they
>> are certainly not shown in udisksctl status.
>>
>> Dolphin can not actually mount a share to some fixed location, the Nemo
>> I am using doesn't even succeed in browsing the network. For me there
>> are only two solutions that I see, either fstab or autofs. Once you know
>> how to do it, running /etc/auto.smb <host> quickly returns a list of
>> shares (if you've set the credentials file) that you can then edit to
>> become a static map with the browse option.
>>
>> I do not see what is better suited currently to auto-mounting
>> samba-shares. Anything that would work well, would work from both the
>> shell and the GUI, in that the systems have to be linked, and ideally
>> what the GUI does is stored in state files that work equally well from
>> the shell. Any persistent mount would have to be stored somewhere on
>> disk. Then the link would need to be automounted, OR mounted through
>> e.g. libpam-mount. There is nothing wrong with mounting upon logon. And
>> that allows for specific user-dependent stuff. (An automount depending
>> on what user first accesses it, is problematic). (As for Samba). So
>> perhaps logon-mounts are better suited (just restore the mount from the
>> previous session, in that sense) because the $UID problem is hard to
>> solve for autofs, you must either duplicate all entries, and hardcore
>> the UIDs you need, for every user, or I don't know what, prevent other
>> users (including root) from accessing the share you need. You'd
>> basically need map files in auto.master.d for every user, including a
>> map file for each person to specify the shares, if you want browsable,
>> or a modified version of auto.smb that mounts individual shares when
>> accessed. (In the latter case you'd have no browse map).
>>
>> But this script doesn't know what it is being called on, so now you need
>> a variable I guess for each of those .d/ files, so that their entries
>> would be something like:
>>
>> /media/<user>/nas  /etc/auto-short.smb -Duser=<user> which would then be
>> used for the samba $UID.
>>
>> If there is a better way to do this, I don't know. I see no alternatives
>> other than libpam-mount and of course some GUI could do it, but then
>> you'd really just have a form of duplicated functionality between autofs
>> and it.
>>
>>
>>>> - browse only works for static maps, and what person with a static map
>>>> is going to run into performance issues when mount points are created
>>>> in
>>>> advance? Can we please remove this from the default config? (turning
>>>> it
>>>> off).
>>> Well, quite a lot of current users actually.
>>>
>>> Consider what can happen with a few hundred or a few thousand entries
>>> in an
>>> indirect map.
>> Alright, but what indirect maps can cause browse tables to be shown?
>> Only static maps, right? A programmatic map can only return a single
>> entry. I have not looked into the other ones, but I suppose LDAP is more
>> like a programmatic map than anything else.
> Of course, yes, program maps can never know in advance what key may be passed to
> them.
>
> The problem is usually seen with user applications, cron jobs that scan
> directories, GUI file management utilities etc.
>
> It's has been quite a problem over time.
>
> LDAP is a map source, like a file, or nis etc.
>
> Maps stored in LDAP behave much the same as maps stored in files.
>
>> So then consider that we only use a static map here. An actual file with
>> thousands of entries? Of course it can happen. But what would be the
>> point of that still (as opposed to LDAP or programmatic).
> People do seem to like program maps but plenty of people use static or semi
> -static maps, ie. a bunch of entries followed by the wildcard map key "*"
> usually with some macro or "&" substution.
>
> Similarly, what might match the wildcard map entry can't be known in advance.
>
>>
>>> Even though the browse option was added it has a specific set of
>>> problems.
>>>
>>> First the kernel does not know if you are just listing a directory or
>>> actually
>>> need to mount something.
>> Do you mean the directory containing the (indirect) entries (e.g. the
>> managed directory) or one of those entries themselves?
> It's hard enough for me to understand VFS path walking sufficiently without
> having to describe it too.
>
> All I can say is there is a problem determining whether a call back to the
> deamon is needed or not at the times the kernel module is called. That can
> sometimes result in all entries in the autofs managed mount point directory
> being mounted.
>
> Another aspect of this problem is that the same system call may need to have the
> directory trigger a mount while not so at other times it's called and there's no
> way to tell what the usage is within in the kernel.
>
> So sometimes people get caught because they want their stat(2) call to return
> the mounted file system results but not to cause a mount at other times when it
> might lead to a mount storm.
>
> These people are usually quite passionate as well insisting that it be fixed and
> sometimes refuse to accept it is a problem.
>
> Usuall all I can say to them is don't use the "browse" option.....
>
> It is a difficult problem and I'm inevitably caught in the middle between the
> "browse" and "nobrowse" camps.
>
> Not all cases of accessing such a directory will see this problem either but
> user space seems to have a habit of using system calls that don't trigger a
> mount together with others that do.
>
> Causing the so called mount storm.
>
> Having said that though, with a lot of work, I could probably do something about
> it.
>
> TBH I've avoided it because it is difficult, perhaps in version 6.
>
> It would require not pre-creating mount point directories and adding a readdir
> implementation to the kernel to fill in the names (map keys) that aren't already
> mounted. But that means significantly more (in terms of quantity of)
> communication between the daemon and the kernel module.
>
> That means re-writing the user space <-> kernel space communications completely
> in an incompatible manner to what is used now, not to mention significant
> changes to the user space code itself.
>
> A big job really.
>
>>> So at the moment, certain types of system call (the module can't know
>>> the exact
>>> system call either) don't cause a call back to perform a mount but
>>> often user
>>> space will later make a system call that will cause a mount to occur
>>> anyway.
>>>
>>> That leads to sometimes seeing directories that are those of the autofs
>>> file
>>> system not those of the mounted file system, so they have completely
>>> different
>>> attributes to those after an automount has been done, a problem in
>>> itself.
>> Oh... such as ownership etc.
>>
>> The typical "I mount this, and now suddenly it is not root.root but
>> xen.staff".
>>
>>> And, it can often lead to a mount storm, mounting everything in the
>>> directory.
>> But how can that happen? If I do; "for f in *; ls $f; done" sure.
> It happens, from what I've seen, quite a bit.
> Perhaps you would be surprised.
>
>>
>>> OK for a few dozen mounts in a map but really bad for anything with
>>> more than a
>>> few hundred entries.
>> After thinking a little I came up with what could perhaps be some
>> default use case. You have terminals mounting home directories. On the
>> "terminal machine" you may have just a single user. Through LDAP you
>> find the home directory. There is obviously no sense for all users in
>> the LDAP system to now have precreated mount points. The only thing that
>> could make sense is to have an "invisible" user directory that now gets
>> mounted.
> Yes, you would need to be using the default "nobrowse", or a limited set of keys
> with a wildcard entry.
>
>> I can see no reason to have hundreds of mount points in a directory
>> unless they cannot conform to a specific pattern. If they follow a
>> pattern, it is likely that only a few of them need to be mounted at any
>> time. In that case being able to browse it seems nonsensical. An
>> aggregate of hundreds of different mounts at the same time seems very
>> weird. Weird to have a diverse set of unique names to begin with. As if
>> all of them were manually added in some way. If they aren't manually
>> added, but automatically, then why have a static list? Would you not be
>> better off putting that in a database? And if you'd have so many, we'd
>> assume they could be different users, or different websites, etc. We
>> also assume not all of them are going to be needed, just a subset. But
>> so now you have a system on which this list is statically created but
>> dynamically updated. This list sits on the disk of the local system. But
>> you're not querying a database (could be just as easy) that could also
>> store this list and return the entries one by one when needed. The
>> static file is the database.
>>
>> Seems an improper use case but yeah. I don't know how you can have a
>> static list that is huge while not choosing to use a database to manage
>> it. Because a static map; now you need routines to update it, delete
>> entries from it.
> Yes, but network sources such as NIS and LDAP are treated much the same as a
> file map.
>
> Checking if a file map has been updated since a key was last used is easier with
> a file map and straight forward with a NIS map but can't sensibly be done for an
> LDAP map.
>
>> Maybe I'm missing something here.
>>
>>
>>> Another reason for the default is that, because of this, historically
>>> there was
>>> no "browse" option and this maintains what is (should be) expected.
>> Yeah I thought so, because of the comments near the timeout value.
>> However that is a reason to never change anything ever. Someone
>> mistakenly choosing "browse" will very quickly find out.
> And wouldn't itself be sufficient but for the problems I'm attempting to
> describe above.
>
>> If this person really has hundreds of items in his list, he will notice
>> within a second that they are all visible. If this person really has
>> hundreds to thousands of items, he or she may also be expected to know
>> the ins and outs of the system, and to be aware of the browse option, or
>> be able to find out. This is not going to be a novice user.
>>
>> On the other hand, anyone new to the system will have modest goals.
>> However those modest goals are frustrated and refuted because the system
>> is geared towards that professional that already knows everything, and
>> for whom adding nobrowse, or changing the config file, would probably be
>> a nobrainer. The new user however may be flabbergasted that it either
>> does not work, or not the way she feels it should work (making a list,
>> and then not seeing it in action).
>>
>>
>>
>>>> - browse being turned off by default combined with a complete lack of
>>>> mention that you need to access these "nonexistent" directory paths /
>>>> components before you will be able to see them, turns autofs into a
>>>> complete mystery for someone who is not privy to the details yet.
>>> It's been a long time for me so I probably have forgotten what it's
>>> like for
>>> someone starting out.
>>>
>>> And so, how would I know where in the man pages to put that so it gets
>>> noticed
>>> by new users, perhaps from your perspective, you could point out a few
>>> key
>>> places where it should be added, or better still send a patch.
>> This happens a lot, it is no ones fault. I guess people who knows the
>> ins and outs of something usually jump smack in the middle, and forget
>> to mention what the system is really like from the outside. A fish
>> doesn't notice the water, we don't notice the air.
>>
>> We are too much immersed in it.
>>
>> I think there are 3-4 things I would explain.
>>
>> First the auto.master file on my system mentions man autofs as a place
>> for seeing the format of the (sub)map files. However there is nothing in
>> there. So that part is missing, but in a sense the contents of auto.misc
>> is already somewhat sufficient for that, but still annoying that there
>> is no man page.
> That doesn't sound right, there might be a problem with your distribution
> package.
>
> [raven@pluto autofs-dev.git]$ wc -l man/autofs.5
> 611 man/autofs.5
> [raven@pluto autofs-dev.git]$ wc -l man/autofs.8.in
> 81 man/autofs.8.in
>
> The first of these describes the Sun and amd map format map entries, not a
> particularly good job of it I admit, but that is where the changes you recommend
> should go.
>
> The second describes the service start and stop, etc.
>
>> The second part is that it is often hard to understand what is meant by
>> "key". In the master file, key means the mount point that supposedly
>> already exists in the filesystem. In the sub-maps, key means the
>> subdirectory but also the component of the path getting accessed by the
>> user. Both types of files are called map files, and both kind of keys
>> are called keys. This is aggravated by the fact that there is no man
>> page for the sub-maps, that I know of . Auto.smb then speaks of a key,
>> but it doesn't say what it is. Of course after a while you realize it is
>> to be the host, but then you don't know how it knows about this host.
>> Since at that point you don't realize that this is input from the
>> filesystem (e.g. a stat operation) on "hidden directories" you are
>> really clueless as to where this information is coming from ;-).
> Sure, and that probably needs to be spelled out in autofs(5).
> I know people do get confused by this distinction.
>
>> You don't imagine that every freaking access of a directory or path
>> component in the managed directory is going to result in network access.
>>
>> You could do a "for f in `seq 1 10000000`; do vdir $f; done" and have
>> fun.
>>
>> (Don't know if that would still be the case for "browse" maps).
>>
>> So this hidden directory, secret door thing is really the most confusing
>> of all.
>>
>> It's somewhat similar to something I might like to do, but still you
>> don't really expect it ;-).
>>
>> So the 3-4 things are:
>>
>> create a man page or section for the format of the sub-maps
> Already exists, needs work.
> Lets call these mount maps (not the best but the convention used) as opposed to
> the master map.
>
>> indicate more clearly that programmatic maps in principle return a
>> single entry
> Maybe but isn't this in autofs(5) already close:
>
>     Executable Maps
>         A map can be marked as executable. A program map will be called
>         with the key as an argument. It may return no lines of output
>         if there's an error, or one or more lines containing a map entry
>         (with \ quoting line breaks). The map entry corresponds to what
>         would normally follow a map key.
>
>> indicate that programmatic maps currently require access of a path
>> component and that this path component (user action) is the key to the
>> script.
> But that is the case for all maps:
>
> Isn't this near the top of autofs(5) sufficient:
>
>         key [-options] location
>
>         key
>                For indirect mounts this is the part of the path
>                name between the mount point and the path into
>                the filesystem when it is mounted. Usually you
>                can think about the key as a sub-directory name
>                below the autofs managed mount point.
>
>                For direct mounts this is the full path of each
>                mount point. This map is always associated with
>                the /- mount point in the master map.
>
>
>> indicate that this is true for "no browse" on "static maps" as well, and
>> that directories are only created when they are accessed.
> That seems to be missing from the man page.
>
>> And that no-browse is the configured default.
> Not sure about that either.
>
> This (from auto.master(5)) says that I have made "browse" the internal program
> default way back when version 5 was first developed but I install a
> configuration that turns it off because of (because of the above problems, which
> are not spelled out in the man page):
>
>       [no]browse
>              This is an autofs specific option that is a pseudo mount
>              option and so is given without a leading dash. Use of the
>              browse option pre-creates mount point directories for
>              indirect mount maps so the map keys can be seen in a
>              directory listing without being mounted. Use of this
>              option can cause  performance problem if the indirect
>              map is large so it should be used with caution. The
>              internal program default is to enable browse mode for
>              indirect mounts but the default installed configuration
>              overrides this by setting BROWSE_MODE to "no" because
>              of the potential performance problem.
>
> Don't forget that distributions can, and often do, install their own version of
> auto.master and possibly the autofs configuration.
>
> And looking at the man pages shows that I have missed changing some of the
> configuration option names from the recent configuration file changes, something
> else that needs to be done.
>
>> Currently the documentation for the master map and for (indirect)
>> sub-maps is really in auto.master and auto.misc.
>>
>> Reading config files is usually a lot more pleasant than man pages ;-)
>> so that's not such a huge problem, but you can't learn anything from a
>> very limited set of examples either ;-).
>>
>> For instance: direct map won't work if not prefixed with /, but will
>> also not give error if not. Indirect map (I think) won't work if
>> prefixed with /, but will also not give error if not.
>>
>> I could say more but my thumb is hurting (infected) so I gotta quit
>> this.
>>
>> I don't mind writing documentation. But I'm not familiar yet with man
>> pages and their technical structure and all.
>>
>> I'll have to see. Thank you for the interest in any case :).
>>
>>
>>>>
>>>> Further I would say that on a desktop OS, the most likely use case is
>>>> going to be this:
>>>>
>>>> - have several shares in your home network
>>>> - want to access them through a GUI
>>>> - prefer automounting since mounting at boot can be problematic
>>>> - need to see them before you can access them in a GUI
>>>> - only alternative is "bookmarks" or "places" that work regardless of
>>>> the things not being visible/listed in the directory.
>>> Either use a different automounting mechanism or use a direct mount
>>> map.
>>>
>>> That would satisfy all these needs, all be it with the map entries
>>> always being
>>> mounted on access.
>>>
>>> The limitation of these is you can't have a direct map path that is
>>> within
>>> another automounted path, if you need to do that you need to use a more
>>> complicated setup.
>> I think that limitation indicates why using relative paths is so much
>> more pleasant. A map file using relative files can be "moved around". A
>> map file using absolute paths cannot.
>>
>> Could you not make a direct map out of the key of the master file, and
>> the keys of the sub-maps?
> Can't really do that.
>
> The closest thing to that would be multi-mount map entries, again from
> autofs(5):
>
>     Multiple Mounts
>         A multi-mount map can be used to name multiple filesystems
>         to mount. It takes the form:
>
>         key [ -options ] \
>             [[/] location \
>            [/relative-mount-point [ -options ] location...]...
>
>         This may extend over multiple lines, quoting the line-breaks with
>         `\´.  If present, the per-mountpoint mount-options are appended
>         to the default mount-options. This behaviour may be overridden
>         by the append_options configuration setting.
>
>
>> If there is really a need for huge static map files (I haven't studied
>> LDAP) in principle all you need is a syntax to differentiate between the
>> two, right.
>>
>> /-         ( /this/is/my/abs/path )
>>
>> or
>>
>> /this/is/my/abs      ( -/path )
>>
>> could be identical
> Not really, the master map key "/-" is required to specify the direct mount maps
> that are to be used, anything else is an indirect mount, and "/-" is only valid
> within the master map (although I can't remember if I explicitly check for it
> within mount maps).
>
>> normally ( /bla/bla ) indicates an absolute path and can only be used in
>> a direct map, from what I've seen. You *could* cause it to turn this
>> "indirect map" into a direct one.
>>
>> ie. ( bla/bla ) == indirect
>>       ( /bla/bla ) == direct.
> In the Sun map format maps indirect mount map keys are a single path component
> only.
>
> The situation is different if using amd format maps but I could work out how
> direct mount maps could be usefully used so that mount type has not been
> implemented. Even on Solaris where they are supposed to be fully supported I
> could make sense of them.
>
>> A direct map is always a static file right? But then it doesn't matter
>> what part of the path is specified in what file.
>>
>> Or
>>
>>       ( bla/bla ) == indirect
>>       ( -/bla/bla ) == direct
>>
>> So you get:
>>
>> /start/of/the  (-/path )
>>
>> or
>>
>> /start/of/the/-  ( path )
>>
>> Then any path in the master file ending with /- would be a direct path.
>>
>> And would cause the (static) map file to be treated as direct.
> Nope, that is unlikely to work and would lead to a lot of problems.
>
> One example might be with:
>
> /start/of/the/-/path
>
> people will ask where have the directories below /start/of/the/ gone and the
> answer will be they have been covered by the autofs file system mount.
>
> Next it would be but I should be able to see these, surely that's easy to fix.
>
> At hat point I would probably leave the conversation.
>
> In any case a similar thing, but more explicit and I think easier to follow and
> maintain are the autofs multi-mount map entries, what I think you called
> relative path above.
>
> It isn't clear but that syntax should function within both direct and indirect
> mount maps.
>
> The problem with all this is that substitution cannot be used in the key.
>
> The reason for that is same as why the amd regex key matching has not been
> implemented.
>
> A linear map scan would be needed for every lookup in order to match what has
> been sent to us from the kernel. And as much as you might feel that wouldn't be
> a problem because maps are generally small I can assure you it would make like
> really unpleasant for me!
>
>>
>>
>>>> Then I will also say,
>>>>
>>>> that there is no documentation on the format of the map files other
>>>> than
>>>> auto.master. The documentation for map files is this:
>>> That's not quite right.
>>>
>>> Man page autofs(5) has a bit of information on both the Sun and amd
>>> format maps.
>> Oh I am sorry. I think many users are not versed in the numbers for the
>> man pages. I for one do not notice the difference and I almost never use
>> one of those numbers. Sorry, I seriously missed that.
>>
>> I only noticed just now because autofs(8) references autofs(5), and I'm
>> like ....huh?
> Yeah, that's what we have ....
>
>> Okay, allright. Missed that.
>>
>> Seriously bad system if you ask me, but okay.
>>
>> So I guess you can have direct maps with LDAP etc. too.
>>
>> I guess the wildcard also solves the many-user problem.
>>
>> Well, but only for the sub-map. You couldn't get autofs to manage
>> /media, so it needs to be /media/user/<aggregate> in which user can't
>> suddenly be a wildcard unless you use a direct map. You could expand
>> that to include the master file though:
>>
>> /media/*/nas    /etc/nas-shares.map
> Can't do that, each /media/*/nas needs to be mounted at program start so that
> automounts can be done?
>
> Chicken before the egg problem there.
>
> You might be able to achieve something sensible with a wildcard and offset
> mounts (the relative paths above) and macro and & substution.
>
> It is a bit difficult though.
>
>> And then in /etc/nas-shares.map:
>>
>> share1   -fstype=xxx,uid=&   ://nas/share1
>> share2   -fstype=xxx,uid=&   ://nas/share2
>>
>> I can't see whether LDAP supports browse, I assume it must, since it
>> supports direct.
> Yep, same as file maps except remote maps cannot use plus map inclusion, like
> the
> +auto.master
> you might see in default installed master maps.
>
>>
>>
>>
>>> By definition a program map can't know what keys may be passed to it.
>>> And, yes, even autofs(5) doesn't say explicitly that a program map
>>> doesn't know
>>> what keys may be passed to it, perhaps that should be ammended.
>> In principle even a program map could read all of the available keys
>> from whatever database it consults, and return all of them; or return
>> something only (as it does now, perhaps) when a given key matches (is
>> found).
> I've thought of that too.
>
> But I've resisted because I'm concerned about confusing existing program maps
> people have in unpleasant ways.
>
> But that suggestion has been made before and .....
>
>> If you wanted that you would need the program script to e.g. support a
>> wildcard as parameter. Meaning, all scripts must then check for "*" to
>> be passed to it, in order to return the full list that they can acquire
>> (if even possible) or at least return an error of some kind to say that
>> they don't support it.
>>
>> You'd need a parameter for it, and any script would need to be aware of
>> that parameter.
>>
>> That's either gonna be a wildcard or something given on $2, in that case
>> every existing script would already be compatible with it, since it
>> simply disregards $2.
> That's not a bad thought, but what would $1 be so that older scripts wouldn't
> choke on it.
>
>> So, for instance, if $2 is "browse" the script may choose to return a
>> browse map, or list of shares, with whatever options are available (it
>> could even return the UID/USER of the share given options, so that the
>> created directory would have the same UID as the to be mounted share!)
>> -- meaning it might simply return ALL entries it could possibly return
>> in the default format, which is the meaning of the wildcard, and the
>> program (autofs daemon or module(?)) would be able to generate the
>> appropriate directory entries based on that. So in that case you're
>> probably going to be rid of the problem of the non-matching state of
>> pre- and post-mount of directories.
> which pretty much defeats the only possible solution I can think of for "browse"
> option mounts ...
>
>>
>>>> a) get rid of the browse_mode = no, in the configuration file, for
>>>> static mounts
>>> No, that has it's own set of problems which people need to be aware of
>>> before
>>> choosing to use the "browse" option.
>> Not everyone.
>>
>> I have like 5 mounts in this share. 5 shares in this mount. I have not
>> noticed issues to begin with. Maybe there are issues like checking
>> permissions on the directory prior to mounting it, and then being wrong
>> about that once it is mounted. But still, that doesn't seem to happen.
>> Actually, something funny is going on. 2 of the 8 shares that are
>> 'automatically' loaded are not accessible. "No such file or directory".
>>
>> No, that's nothing but a permission error. Still I am allowed to see
>> them but not use them. For a static file no issue; just delete them.
>>
>> I don't really see what I need to be aware of prior to using browse?
>>
>> If it is the most sensible "novice" use case, why not allow it by
>> default? It things go mayhem people can turn it off, or you could even
>> turn it off when the number of mounts in a map exceeds 20 or 30 or 50.
>> It's an arbitrary number, but what if I am wrong about LDAP.
>>
>> What about you mean it is turned off by default for LDAP, but you do not
>> consider static files with that?
>>
>> What about it does function for ldap. What about we just need a separate
>> default for LDAP and its like, than we do for static?
>>
>> Are you seriously considering static files, or do you mean LDAP, yp,
>> hisplus, and hesoid?
>>
>> For anyone using an actual database, I concur.
>>
>> For static, I just disagree. That is the start case for anyone.
> Like I said, I wanted to have the default as "browse" but got hit with too many
> complaints and was only willing to change the package installed default
> configuration to satisfy them.
>
> I'm not going to try that again.
>
>>
>>
>>>> b) say something about mounts not being created automatically in the
>>>> man
>>>> page for auto.master, if nothing else. If you also do (a), this only
>>>> applies now to programmatic mounts.
>>> auto.master(5) isn't the place for that, autofs(5) is better and, as I
>>> say, make
>>> some suggestions of provide a patch.
>> autofs(8) is loaded before autofs(5). It's a good document, autofs(5),
>> but most people may not even notice it. Personally I would probably
>> choose to call them:
>>
>> autofs.master(5) and autofs.map(5). But then, I also consider
>> auto.master a very nondescript name.
>>
>> In this way no one can ever get confused, so you might just rename
>> autofs(5) to autofs.map(5).
> Yeah maybe ...
>
>> Or even auto.map(5) if that is necessary, but I wouldn't do that.
>>
>>
>>> I'll have a look at doing this but beware, often distributions use
>>> there own
>>> default master map which I have no control over.
>> I have no clue what mine does, this is Debian, and then Ubuntu, and then
>> Kubuntu ;-) but it is just the Debian package still.
>>
>> You don't have to follow my example or suggestion, maybe there is a
>> better improvement than this, it was just an example, I guess. Or I
>> could guess.
>>
>> That was just a thing I was able to change without editing man files
>> ;-).
>>
>> I will look into providing a patch, but I hope you can rename autofs(5)
>> to autofs.map(5) before doing anything like that. I don't feel like
>> improving a system I know myself haven't had access to, in that sense. I
>> don't mean you have to do that straight away, but if that is a possible
>> direction to go, or change to make, then I could supply a patch or
>> anything based on the system that would result from that, seeing that
>> autofs(5) is already pretty good in that sense.
> At some point I'll need to return to this and make a list.
> Unfortunately that time isn't now.
>
>>>> I mean you could rewrite more documentation but this is the simplest I
>>>> could do right here right now.
>>>>
>>>> I think that would clarify a lot for everyone but of course perhaps
>>>> you
>>>> would want to rewrite more so the central documentation is also up to
>>>> date, but please see this as a more modest attempt for making it more
>>>> accessible to a new user.
>>> Yes, adding the fact that mount point directories aren't pre created
>>> for
>>> indirect maps somewhere in auto.master(5) and in a couple of places in
>>> autofs(5)
>>> is a good idea. You are right it's something that can catch the
>>> unaware.
>> That would also make the difference between direct and indirect more
>> clear. (I still don't know exactly what is the entire difference).
>>
>> That's also something I could change: how a direct map is explained in
>> auto.master(5). But then, I didn't have access to autofs(5) to explain
>> it to me better ;-).
>>
>> Anyway, I think it is a good project, I didn't know autofs would be so
>> good when it comes down to it.
>>
>> It's usually documentation that puts you off ;-).
>>
>> I often feel like I have to find the door in some invisible wall.
>>
>> This time it was extreme because I couldn't for the life of me
>> understand what a "key" would be in the sub-map file.
>>
>> Regards, Bart.
>> --
>> To unsubscribe from this list: send the line "unsubscribe autofs" in
> --
> To unsubscribe from this list: send the line "unsubscribe autofs" in

--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Xen]

Ian Kent schreef op 06-07-2016 14:48:

> But I don't understand your use of "lazy initialization".

I meant creating the mount point only when it was accessed.

> So I don't quite understand what your concern is.
> Sure, if you have hundreds or thousands of these there can be a 
> performance
> problem but that's not what you were trying to do I thought.

I'm not sure either at this point but I'm too hungry to read back on 
what I had written before :p.

> Of course, yes, program maps can never know in advance what key may be 
> passed to
> them.
> 
> The problem is usually seen with user applications, cron jobs that scan
> directories, GUI file management utilities etc.
> 
> It's has been quite a problem over time.

Yes that is my concern. I do not like the idea of invisible access 
points at all, unless I was doing it for security, but in that sense you 
can't really hide anything in a Linux system. Invisible mount points 
only make sense when you have LOTS of them and you only need few at a 
time.


> LDAP is a map source, like a file, or nis etc.
> 
> Maps stored in LDAP behave much the same as maps stored in files.

Right so you can browse them if you want.


> 
>> 
>> So then consider that we only use a static map here. An actual file 
>> with
>> thousands of entries? Of course it can happen. But what would be the
>> point of that still (as opposed to LDAP or programmatic).
> 
> People do seem to like program maps but plenty of people use static or 
> semi
> -static maps, ie. a bunch of entries followed by the wildcard map key 
> "*"
> usually with some macro or "&" substution.
> 
> Similarly, what might match the wildcard map entry can't be known in 
> advance.

Right. That makes it programmatic; the (server) response will determine 
if anything was actually there. On the other hand a wildcard (in a 
direct map at least) may determine existing (or match existing) 
filesystem components that you could read in advance. And you need to, 
in a direct map. But direct maps are mounted anyway (or the mountpoint 
is created, I mean). So it currently only applies to indirect maps where 
the wildcard is the final component, and it then behaves like a 
programmatic thing where it only actually gets mounted (and created) if 
e.g. the server reponds with a share.


> It's hard enough for me to understand VFS path walking sufficiently 
> without
> having to describe it too.

I don't know. I studied it in some detail, I just don't understand the 
RCU algorithm. At least, I have studied path_lookupat and the entire 
procedure leading up to lookup_fast and lookup_slow. I'd have to check 
my printouts to say more. But let's just say I'm aware of any 
(recursive) operation involving symlinks, where intermediate symlinks 
get expanded before the original path is being continued, etc. I am also 
aware of the presence of autofs code, particularly in the function that 
specifies one or two other mount things.

I believe that is called by lookup_slow....



> All I can say is there is a problem determining whether a call back to 
> the
> deamon is needed or not at the times the kernel module is called. That 
> can
> sometimes result in all entries in the autofs managed mount point 
> directory
> being mounted.

So the mount point is accessed but the module doesn't know if anything 
really needs to happen at that point. And stat operations or anything 
else all use the same path lookup. Of course I don't know what happens 
pre-mount and post-mount to the directory itself. If post-mount the 
directory is really on another filesystem, the "mnt" value will change. 
And it will return another dentry for that other mount. I remember 
"path" being mnt+dentry.

> 
> Another aspect of this problem is that the same system call may need to 
> have the
> directory trigger a mount while not so at other times it's called and 
> there's no
> way to tell what the usage is within in the kernel.

So you are getting called by lookup_slow or something of the kind near 
it, but you're not sure why. But these lookups are being done for each 
individual path componenent. However a "cd" to the directory should 
mount it, but a mere "stat" may not. Is that the case? Seems odd that it 
would be difficult, but I believe you on your word.

Maybe "cd" shouldn't mount it until the contents is accessed, but I 
don't know.


> 
> So sometimes people get caught because they want their stat(2) call to 
> return
> the mounted file system results but not to cause a mount at other times 
> when it
> might lead to a mount storm.

Hmm. It seems like the first desire is a problem. At least, I don't know 
how directory lookup is done. I don't know if this information is the 
result of a stat call or something else. I do remember seeing code for 
dentry-moving but it was confusing.

For sure you want "ls /directory/" to return the mounted contents. But 
I'm not sure if its possible for your module to differentiate. Ls -d 
"/directory/" should perhaps not cause mounting. I can see quickly 
though that "ls" causes an "open" call but "ls -d" does not.

But then, "open" will not access any path components beyond the 
directory itself. So if your module is only called by path_lookup it is 
going to be impossible to know what for.

Particularly if an "ls" does not actually access path componenets 
(beyond the directory itself) there would be no way to know what is 
being asked.


> Usuall all I can say to them is don't use the "browse" option.....

:).


> It is a difficult problem and I'm inevitably caught in the middle 
> between the
> "browse" and "nobrowse" camps.

You appear to be very conscientious about it.


> Not all cases of accessing such a directory will see this problem 
> either but
> user space seems to have a habit of using system calls that don't 
> trigger a
> mount together with others that do.

such as that "ls" vs "ls -d"?


> Causing the so called mount storm.

Still seems odd but I believe you, but. I guess what it could mean is 
"ls" not only trying to list every share (browse) but also to perform 
calls on those shares (path componenents retrieved) that might require 
additional information. For example, if I am not content just knowing 
the shares, but also want to know the number of files in them.... which 
definitely some programs or user agents would do. Then I'm in trouble 
because this requires mounting. Then, if that is the case, a simple

"list /nas/" might result in the subfolders (/nas/media, /nas/backup, 
etc.) to also be queried and you have what you call a mount-storm.

But this is the result of user space convention and behaviour. Surely 
people cannot request browse while at the same time wanting to deny 
actions on the returned results when these actions stem completely from 
userspace?

If some user agent or GUI is trying to list the number of files in each 
directory (which at this point means a share) then that's not something 
autofs could ever deal with. And I don't think it is a problem autofs 
needs to solve.

If I am at all correct about this and I see what you mean right now, 
this "GUI" behaviour would then simply be incompatible with unmounted 
folders *that-might-need-to-get-mounted-but-not-always* and this GUI 
would need to be aware to not do that thing.




> 
> Having said that though, with a lot of work, I could probably do 
> something about
> it.
> 
> TBH I've avoided it because it is difficult, perhaps in version 6.
> 
> It would require not pre-creating mount point directories and adding a 
> readdir
> implementation to the kernel to fill in the names (map keys) that 
> aren't already
> mounted. But that means significantly more (in terms of quantity of)
> communication between the daemon and the kernel module.
> 
> That means re-writing the user space <-> kernel space communications 
> completely
> in an incompatible manner to what is used now, not to mention 
> significant
> changes to the user space code itself.
> 
> A big job really.

And not a big joy perhaps.

So basically you are going to not precreate, but still hijack kernel 
calls to return the listings for the directories that do not yet exist 
(the listing of those directories).

Which means those directories could be shown, but not opened, or 
something of the kind. Then, the user agent .. well I don't know what 
difference it makes actually if the mount storm is the result of 
userspace calls that directly cause it. So I guess I'm mistaken here and 
the issue is even more complex.... :(.

I don't know how a readdir on the "top" directory could /directly/ 
causes a mountstorm on the listed entries. But then, I don't know much 
;-).


> It happens, from what I've seen, quite a bit.
> Perhaps you would be surprised.

Illuminate me :P. What is the most common cause of it?

> Yes, but network sources such as NIS and LDAP are treated much the same 
> as a
> file map.

Yes, I didn't realize at the start. I have always found LDAP to be a 
rather incomprehensible thing to work with. Without hands-on experience 
it is hard to get a good grasp of how it works. Or even how to organise 
it (if you do create one) as there seem to be standards for everything.


> Checking if a file map has been updated since a key was last used is 
> easier with
> a file map and straight forward with a NIS map but can't sensibly be 
> done for an
> LDAP map.

Right.

> 
>> 
>> Maybe I'm missing something here.
>> 
>> 
>> > Another reason for the default is that, because of this, historically
>> > there was
>> > no "browse" option and this maintains what is (should be) expected.
>> 
>> Yeah I thought so, because of the comments near the timeout value.
>> However that is a reason to never change anything ever. Someone
>> mistakenly choosing "browse" will very quickly find out.
> 
> And wouldn't itself be sufficient but for the problems I'm attempting 
> to
> describe above.

You mean the mountstorm problems are reasons to keep the default at 
no-browse?

It still seems at this point that this is only relevant for LDAP/NIS 
entries.



>> First the auto.master file on my system mentions man autofs as a place
>> for seeing the format of the (sub)map files. However there is nothing 
>> in
>> there. So that part is missing, but in a sense the contents of 
>> auto.misc
>> is already somewhat sufficient for that, but still annoying that there
>> is no man page.
> 
> That doesn't sound right, there might be a problem with your 
> distribution
> package.

No, as I say later this is incomprehension on my part ;-). Or merely the 
fact that needing to use "number" for accessing "sections" of the 
"manual" is so rare (if you are not a developer accessing library 
functions and/or kernel functions) that a regular user will almost NEVER 
use a section number in daily operation.

Most configuration files are called "openvpn.conf" or something of the 
kind, and hence their manual page is also called that, and you don't 
need to do "man 5 openvpn.conf", because "man openvpn.conf" is 
sufficient.

Using numbers would be superfluous in 99% of cases, or even more than 
that. As opposed to perhaps requesting help on function calls, such as 
mount() as opposed to /bin/mount.

So as a developer you may be used to it. But a regular user is not. 
Anyone who does not do C programming, is not.

So, my mistake, I guess.

>> 
>> You don't imagine that every freaking access of a directory or path
>> component in the managed directory is going to result in network 
>> access.
>> 
>> You could do a "for f in `seq 1 10000000`; do vdir $f; done" and have
>> fun.
>> 
>> (Don't know if that would still be the case for "browse" maps).
>> 
>> So this hidden directory, secret door thing is really the most 
>> confusing
>> of all.
>> 
>> It's somewhat similar to something I might like to do, but still you
>> don't really expect it ;-).
>> 
>> So the 3-4 things are:
>> 
>> create a man page or section for the format of the sub-maps
> 
> Already exists, needs work.
> Lets call these mount maps (not the best but the convention used) as 
> opposed to
> the master map.

Alright. Still confusing though.

> 
>> indicate more clearly that programmatic maps in principle return a
>> single entry
> 
> Maybe but isn't this in autofs(5) already close:
> 
>    Executable Maps
>        A map can be marked as executable. A program map will be called
>        with the key as an argument. It may return no lines of output
>        if there's an error, or one or more lines containing a map entry
>        (with \ quoting line breaks). The map entry corresponds to what
>        would normally follow a map key.

I guess.. it's just that you read over "a map entry" as meaning "only a 
single map entry" since just before it says "one or more lines". So it 
is something you only notice if you reread it in detail. After finding 
out that it doesn't work ;-).



> Isn't this near the top of autofs(5) sufficient:
> 
>        key [-options] location
> 
>        key
>               For indirect mounts this is the part of the path
>               name between the mount point and the path into
>               the filesystem when it is mounted. Usually you
>               can think about the key as a sub-directory name
>               below the autofs managed mount point.
> 
>               For direct mounts this is the full path of each
>               mount point. This map is always associated with
>               the /- mount point in the master map.

Yes it is, particularly because it mentions that it is a sub-directory 
name.

Then, I begin to wonder what "usually" means. See, from the looks of it 
it is completely identical to sub-directory name, because:

- they key is going to be directly used as the name of the subdirectory
- programmatically, the script does not even have any say about the key, 
it cannot even return a key, does not determine the key. The only thing 
that can ever happen is that the key as given is turned into a 
subdirectory (when valid).

I mean the logical abstraction between "key" and "value" may make sense 
from a purely abstract logical point of view ;-). But it has no bearing 
on actual domain-specific termininology, it is just a computer science 
term so to speak.

What it actually comes down to is:

- mount point for the actual share, or
- subdirectory created as mountpoint for the actual share(s).

Where "value" would mean:

- path to the mount (share), or
- network path to the share that gets mounted on the subdirectory (the 
key).

the "mount" utility calls these source and target, respectively (source 
for the path, target for the mount point). Otherwise it calls them 
"device" and "mountpoint".

So although "key" also has meaning with regards to "database lookup" I 
don't see how it is ever not directly used as the mountpoint itself. And 
this is what creates the confusion because with the "browse" option it 
is not just going to be a "lookup key".

Further there is no difference between "subdirectory mount point" and 
"lookup key", where these are fundamentally separate concepts, and could 
even imply, in principle, that the actual directory to use is *also* 
retrieved by the key.

So conceptually the idea that "key" instantly means "subdirectory 
mount-point" is confusing.

In fact it is odd to begin with, from the perspective of a user, that 
"auto.smb" mounts the entire share list of the requested "host" or "key" 
in one go under the location that was accessed.

This is not the most common scenario (well, maybe if you are in some big 
network). This is not most common not only because it mounts /all/ the 
shares at once(!) but also because generally people want to have some 
level of control over the directory it ends up in, and what is the host 
for some server (hostname) might not be where you actually want to mount 
it.

So I suggest that even this auto.smb is a bit of a weird entity. I have 
used it to generate a map list for me that I then adjusted and used as a 
static map file.

I know it does the same as that -net or -host thing. It's just that for 
a desktop user (again, you say that is not the target audience, but 
still) what you really want is a fixed location the user chooses (in 
auto.master) and then a browseable list of shares. Apparently this list 
needs to be static. What I envisioned before would be a parameter to a 
script that causes it to output a list of shares rather than the share 
entries themselves, or just the complete list of it.

This would give it "browse" capability. If you had that, you would need 
to be able to pass a parameter to the script as the key, but this is 
rather obscure.

What it comes down to is that this is what I would envision for myself:

Auto.master:

/cifs   /etc/auto.smb   diskstation                <-- diskstation is 
the key

Now the program/daemon/whatever passes diskstation as key to the script. 
It also passes a wildcard parameter such as "*". The script now returns 
a list of shares for "diskstation". Subsequently when the daemon needs 
to mount, it calls the script with diskstation and the requested mount 
point/share. Alternatively, the returned output from the first call must 
be complete and is saved by the daemon. It is then used in place of a 
static map (the script is now nothing but a static map generator).

Personally I prefer the former: the script has two modes: list and 
return entry, and it therefore has two parameters: one to identify what 
to list, and the second to determine what entry to return from what was 
listed.

Otherwise, without the parameter, you'd need to hardcode it in the 
script and duplicate the script for every host, etc.

In fact as a user you may not even want all shares to be present on your 
local system. So you might equally well decide to use a static map. So 
perhaps you will use "smb.auto list" or "smb.auto list diskstation" or 
"smb.auto diskstation *" manually. And perhaps it generates all of the 
mappings for you that you can then prune and save.

Perhaps, in absense of the key (the entry, or wildcard) a call such as 
"smb.auto diskstation" should simply generate the share list, whereas 
"smb.auto diskstation media" generates the key for only that entry.

So you see me use a term like "entry" to determine what you today call a 
"key" and the word "key" to determine what you today call a "entry" ;-).

"entry" means "element of pre-existing list", "key" means "value to use 
for a lookup operation in a database".

"key" can also mean "that what you need to mount or access something".



so if you combine the two, you have like three things:

- subdirectory (mount point)
- key to use for database lookup
- entry in the list of shares

but you can only solve it if you differentiate between "lookup 
operation" and "element of a list".

a lookup operation may /return/ a list, which is why I said:

"diskstation" (hostname) is a key
list of shares is the result of that lookup operation based on that key
entry is an element of that list you can mount.

This applies to a form of browse scenario. If you have invisible 
mounpoint, you get:

"diskstation" (hostname) is a key that is going to be a direct 
subdirectory
the list of shares it returns is going to be a subdirectory list to that 
first subdirectory
this will contain the actual shares, so you get top/key/entries.

And you see we basically have the same thing except that this is one 
level deeper in our hierarchy.

In my desired use case, you get:

top/entry

But in the current implementation of e.g. smb.auto, you get:

top/key/entry

So it is clear diskstation (hostname) IS a key, except that in the 
current case or scenario this key is ALSO used as the direct descendent 
of the toplevel directory used for the configuration in auto.master.

Which gives rise to the idea that maybe "key" and "subdirectory mount 
point" should not be equivalent or identical. Not necessarily identical.


In fact, smb.auto DOES produce a browsable list. It is just /beneath/ 
the subdirectory for its "key" (the hostname) and mounts ALL of the 
shares at once.

So you now already have that mountstorm (it just gets mounted by 
default) it's just a level deeper.

You now don't have a mountstorm on the level of hosts, but you have a 
mountstorm on the level of shares (per host) but it is still the same 
thing.

So the design specifies that keys must become mountpoints in the 
"sub-top-level" directory (the directory managed by autofs for indirect 
mounts).

Whereas perhaps conceptually you would need to differentiate between 
key, and an entry used for mounting, and the difference is that this 
entry used for mounting can be an element of the list returned by the 
key.

So you get key -> list -> entries. In the case of current auto.smb, what 
happens is:

key -> (subdirectory) -> list (itself an entry) -> entries.

Or in other words:

key -> list -> entries, but mounted beneath top/key, as key is used as 
an extra directory in between.

Current auto.smb:

  <managed mount point>   /    <key>   /   entries  / contents of shares

My preference:

  <managed mount point>   /    entries     /   contents of shares

And why? Only because I want to specify mount points for specific hosts 
manually.



So the question becomes how you can combine such things.


The requirement becomes for the script to return the mount point. A 
static list does return mount points. A script does not, the mount point 
is specified as the "key" given to it. So if you want the script to 
determine the mount point the way a static list does, you need it to 
return, in essence, a static list.

Even if your static list is not browseable, so too can a returned 
"dynamic list" also be not browseable. But at this point your script can 
return a mount point of its own. That is again problematic if the key is 
derived from filesystem access.

So you see how hard this is to reconcile.

If we insist in accessing hidden shares and deriving keys from user 
action or filesystem action, or filesystem access, then keys are 
predetermined the moment the system is actually called. The script or 
database can only return whether something is gonna be there or not.

Then the only way to reconcile it is (in my case) for the key to be not 
the only argument to the script, but I give the script my argument I 
define in auto.master:

/cifs   /etc/auto.smb   diskstation

Now my script knows that e.g. $1 is gonna be the filesystem requested 
path, and $2 is gonna be the custom parameter I determined in advance.

I don't need to use the key by itself ($1) to do the lookup. My real key 
is $2. But I will use the second key ($1) to return a share for the host 
determined by the first key ($2).

In essence what happens with auto.smb is that the key ($1) is used as 
the host (first key) and smb.auto will itself derive and load the list 
of shares, and then instantly mount that (it retrieves all of the second 
keys and instantly uses that and mounts everything).

autofs allows for this second list of keys to be instantly used and 
utilized.

But it's not the common use case because you are now fixed to that 
location the user accesses and of course you could use that location 
(the key) ($1) to look up a host in a different way, that is perfectly 
possible. If your script marries "diskstation" to "ds" or "ds" to 
"diskstation" then you could automount /cifs/ds and it would request 
shares from host "diskstation".

But you're still stuck with that intermediate level, that extra level of 
subdirectories.

It's not possible currently to use the "first key" to determine the host 
and then get the "second keys" ($1) to determine the entries. Well, 
actually it is, but not in a browseable fashion:

/cifs   /etc/auto.smb   -Dhost=diskstation


I believe this could work. You pass your real "primary" key as that 
variable, and your script returns shares based on that key, but with the 
"filesystem key" ($1) as the name of the share to return.

Still, rather convoluted at this point, and not possible to browse this 
list.

Because it doesn't return a list. Using $2 to pass that variable 
wouldn't really be different.

But the script is only called when an entry is accessed (through the 
filesystem) and as such is never called without a secondary key.

The current auto.smb doesn't have secondary keys, it loads and mounts 
all of them instantly and automatically, and there is no "automount" for 
the secondary level.

Well, there is, it just mounts all of it in one go.

So how do you skip the intermediate level?

- call the script on purpose to generate a 'static list' instead of just 
for entries alone.

- use the static list to precreate directories (mount points), or

- use the static list to return readdir information, or to filter 
requests for directories, or don't use the static list at all and just 
keep it the same.


So we also see here that the static list *could* be used for the thing 
you described above, that alternative readdir implementation.

Although it's a bit of a hack, I'm sure.

You could specify some interval for invalidating the list. At which 
point it is renewed by another "get list" call to the script.


So in essence I am really advocating "return list" functionality for 
scripts here.

And there is no easy way to implement that while staying compatible with 
current scripts because you would need to: call the script with an empty 
parameter (that will never return an entry) or with a wildcard (that 
will probably never return an entry) and in essence, that might just as 
well mean, that if you want to take this approach:

- use the wildcard as the "list" operation, scripts that do not know 
about the wildcard will simply return zero entries.
- use the extra parameter as $2, in other words: allow for additional 
parameters in auto.master in that form of:

/cifs   /etc/auto.smb   diskstation random

But there are already recognised options. So what we have today (-D) I 
guess already works, it is just a bit obscure, and it seems to be the 
equivalent of what you would do.

Then an alternative auto.smb (I called it auto-short.smb) would:

- be called as:

/cifs   /etc/auto.smb   -Dhost=diskstation

- return a list of shares when called with "host=diskstation 
/etc/auto.smb *"
- alternatively you could have it be called with: "/etc/auto.smb * 
diskstation"

- make sure the shell doesn't expand that * ;-).

- autofs calls this list operation each time auto.master is reread.

- autofs calls this list operation each time some invalidation timeout 
occurs

- not really any other way to trigger it.

- the static list being returned (really a dynamic list now) can be used 
for readdir, or even access filtering (it contains all the valid keys).


Now when you have something like this, you can:

- call the filesystem component you use to access the actual shares, the 
"entry" or simply "share" or "directory". But "key" becomes more 
comprehensible now as well. "key" now corresponds to "entries in the 
returned list". Each "entry in the returned list" is now a "mapping 
between key (directory) and value (device-path). I realize "mount point" 
might confuse with the mountpoint defined in the master file.

"submount" would be ugly. "subdir" is most convenient. "subkey" is 
illustrative. "entry" describes the entire thing, not just the key. But 
in essence we are just talking about a mount-point here.

Nevertheless in principle the "first key" given should not even need to 
be the mount point in fact. It is a key to a lookup-table.

If you wanted to do it right I think you would need to do away with 
hidden (or principally unknown) mount points, and basically always work 
with static lists (or dynamically returned lists) and/or give every 
script a first parameter $1 that is a key that you actually use for it.

Then the second paramter $2 would be the key that the filesystem has 
actually given you for accessing a path.

In this way "access path" becomes secondary.

Perhaps a script in this way still cannot control the mount point (name) 
but at least now it provides it in advance. Of course there are 
situations in which you cannot know it in advance or in which it is 
irrelevant to know it in advance.

You can imagine a list to be much too big to actually retrieve it. In 
itself a sign of a bad system, I think. If it was that big, you would 
need hierarchy to make it smaller.

So I don't know if there is ever any real need for obscurity, and for 
hosts to be accessed while not being able to list them in advance.

I conjecture that this use case in itself would be odd, that you create 
auto-mounts for an unspecified number of hosts you don't really know 
about in advance.

It makes for zero-configuration, true. You don't have to keep a list 
anywhere or even arrange for it to be generated.

On the other hand if a list could not be had, the script would just 
return nothing and the daemon will assume no-browse 
no-information-in-advance lookups the way it does today.

Basically if you keep the current $1 and use it to pass a wildcard, no 
script is likely to ever break on that (it will just return nothing). 
Then $2 can be your primary key. Not every script needs it.

But complete zero-configuration is to me an odd thing: a magical 
directory that will mount every host given to it including all of its 
shares, does not seem sensible to me in the sense that you would at 
/least/ want some configuration for every system right?

One size fits all: the same system for every computer, just place it 
somewhere in your filesystem and you have access to everything and 
everything.

No, normally you would want to tailor it because you do not have access 
to an infinite amount of hosts and the /cifs/host/share structure 
therefore makes not much sense.

/device/share may make more sense, perhaps added with
/hosts/hosts-meeting-a-certain-pattern/share

You already have a pretty big system if you use the latter. Anyway.

I think I should conclude this :p.



What seems sensible to me at least is to add something that doesn't 
break compability by using a wildcard * for scripts that will return a 
certain format of list (or just all shares it can find in the regular 
format) with a $2 parameter designating something such as a "host" or 
perhaps even "department".

In this way you solve the issue that the current auto.smb requires the 
"hostname" to be first key given to it because there is no other way to 
pass a key. Then the actual keys can be passed as "second parameter" in 
this sense that they still become $1, but at least now you can specify 
your host to it in advance instead of guessing it on the wind ;-).

So if I have to say anything I would say that I recommend this 
structure:

- $1 remains the filesystem key
- $2 becomes the additional key a user may specify in auto.master
- $1 as * will designate the request for a list lookup
- the list can in the future be used for readdir implementation, and 
currently for precreated mountpoints in case "browse" is used.





>> indicate that this is true for "no browse" on "static maps" as well, 
>> and
>> that directories are only created when they are accessed.
> 
> That seems to be missing from the man page.
> 
>> And that no-browse is the configured default.
> 
> Not sure about that either.
> 
> This (from auto.master(5)) says that I have made "browse" the internal 
> program
> default way back when version 5 was first developed but I install a
> configuration that turns it off because of (because of the above 
> problems, which
> are not spelled out in the man page):
> 
>      [no]browse
>             This is an autofs specific option that is a pseudo mount
>             option and so is given without a leading dash. Use of the
>             browse option pre-creates mount point directories for
>             indirect mount maps so the map keys can be seen in a
>             directory listing without being mounted. Use of this
>             option can cause  performance problem if the indirect
>             map is large so it should be used with caution. The
>             internal program default is to enable browse mode for
>             indirect mounts but the default installed configuration
>             overrides this by setting BROWSE_MODE to "no" because
>             of the potential performance problem.

I know, but if you, as a user, do not understand what the hell it is 
doing in the first place, and you have no time for extensive manual 
reading in advance (like me) you will never get to that option because 
you still don't understand the basics, and the more advanced stuff (like 
that) that will be too hard to understand.

That information is too difficult to comprehend (or digest) when you are 
someone who still doesn't understand how it works.



>> I think that limitation indicates why using relative paths is so much
>> more pleasant. A map file using relative files can be "moved around". 
>> A
>> map file using absolute paths cannot.
>> 
>> Could you not make a direct map out of the key of the master file, and
>> the keys of the sub-maps?
> 
> Can't really do that.

It's not that I saw it as necessary, I just still think the difference 
and distinction between "direct" and "indirect" is rather I dunno, weird 
I guess ;-).



> The closest thing to that would be multi-mount map entries, again from
> autofs(5):
> 
>    Multiple Mounts
>        A multi-mount map can be used to name multiple filesystems
>        to mount. It takes the form:
> 
>        key [ -options ] \
>            [[/] location \
>           [/relative-mount-point [ -options ] location...]...
> 
>        This may extend over multiple lines, quoting the line-breaks 
> with
>        `\´.  If present, the per-mountpoint mount-options are appended
>        to the default mount-options. This behaviour may be overridden
>        by the append_options configuration setting.

Something I am already using (almost) and I believe this is what 
auto.smb uses by default too. It's just that I use it to mount parts of 
the same share. (Samba smbd often has issues with file-system boundaries 
so I create a share out of each filesystem and share them individually, 
then mount them as above).



>> 
>> normally ( /bla/bla ) indicates an absolute path and can only be used 
>> in
>> a direct map, from what I've seen. You *could* cause it to turn this
>> "indirect map" into a direct one.
>> 
>> ie. ( bla/bla ) == indirect
>>      ( /bla/bla ) == direct.
> 
> In the Sun map format maps indirect mount map keys are a single path 
> component
> only.
> 
> The situation is different if using amd format maps but I could work 
> out how
> direct mount maps could be usefully used so that mount type has not 
> been
> implemented. Even on Solaris where they are supposed to be fully 
> supported I
> could make sense of them.
> 



>> /start/of/the/-  ( path )
>> 
>> Then any path in the master file ending with /- would be a direct 
>> path.
>> 
>> And would cause the (static) map file to be treated as direct.
> 
> Nope, that is unlikely to work and would lead to a lot of problems.
> 
> One example might be with:
> 
> /start/of/the/-/path
> 
> people will ask where have the directories below /start/of/the/ gone 
> and the
> answer will be they have been covered by the autofs file system mount.

But that's no different from a regular mount on that location.


> Next it would be but I should be able to see these, surely that's easy 
> to fix.
> 
> At hat point I would probably leave the conversation.

:-). Haha. But come on, you can't take that seriously, the same thing 
happens with a regular mount, even a bind mount does that. It's 
absolutely not possible to reveal the contents of some subtree when 
another filesystem had mounted on top of it, except by remounting the 
entire subtree elsewhere.

(I am trying out some gimmick that requires me to mount the root device 
under something like /run/mount/rootfsclone, because I am bind mounting 
on top of something that I still want to be accessible. The only way to 
make it accessible is to mount the entire rootfs somewhere else. I 
basically need a persistent location for individual devices without 
anything mounted on top of them; ie. a pure one-file-system mount tree.

Then you have those individual mount-trees (filesystem trees) and THEN 
you have the complete FHS that you normally use.

That seems like the only sensible way to approach a lot of issues. 
Having a /mounts/ with every (custom) mounted device present there.

Recently the number of incomprensible mounts reported by "mount" has 
grown so big that the entire tool has become unusable, mostly due to 
cgroups. And tmpfs's, and debugfs's, and what not.

I now find I need to create a wrapper for mount so that "mount" stays at 
least usable to the user. Such that every subdirectory of /sys, and 
every subdirectory of /run, and every subdirectory of /dev/, is actually 
hidden from display. And from /proc as well. And the same applies to DF.

(I really need to find a way to better save and redistribute these 
modifications).






> 
> In any case a similar thing, but more explicit and I think easier to 
> follow and
> maintain are the autofs multi-mount map entries, what I think you 
> called
> relative path above.
> 
> It isn't clear but that syntax should function within both direct and 
> indirect
> mount maps.
> 
> The problem with all this is that substitution cannot be used in the 
> key.
> 
> The reason for that is same as why the amd regex key matching has not 
> been
> implemented.
> 
> A linear map scan would be needed for every lookup in order to match 
> what has
> been sent to us from the kernel. And as much as you might feel that 
> wouldn't be
> a problem because maps are generally small I can assure you it would 
> make life
> really unpleasant for me!

:). You have lost of enemies; if you don't do things right :P. LOL.

I am not thinking about this now, but I guess you mean that my proposed 
solution of saving those static lists (or dynamic lists) would actually 
help this :p.

Actually what you are saying is that even doing lookups on those lists I 
proposed would be quite annoying.

However currently this lookup is already done on the server (if it is 
some database) or in the static map (if they are not browseable). Right?

Databases are generally better at lookup than small-scale applications. 
Now you need hash-tables etc. Anyway, I can't think about it right now.



>> I only noticed just now because autofs(8) references autofs(5), and 
>> I'm
>> like ....huh?
> 
> Yeah, that's what we have ....




>> /media/*/nas    /etc/nas-shares.map
> 
> Can't do that, each /media/*/nas needs to be mounted at program start 
> so that
> automounts can be done?

Actually the shell easily expands such a pattern to all matching trees?

e.g. if you create /usr/local/doc, and then do

"ls /usr/*/doc/" you will get the output of:

/usr/local/doc  /usr/share/doc

So with that output, you mount all of them, all the while knowing that 
the middle component (here) is a wildcard. Then the daemon knows about 
this wildcard and can use it while processing /etc/nas-shares.map for 
each of them; but they are all individual mounts at that point.

e.g.  /media/me/nas and /media/pamela/nas are now individual autofs 
mounts.

but their "/etc/nas-shares.map" is called with a different wildcard.


I mean that either /media/me/nas will already exist (precreated by the 
user) or your program creates it (I think it already does that).

Actually it cannot create it DUH, so the user will need to precreate 
them.


I know /media is usually seen as a managed directory but this is 
annoying in the first place. Such an important name (/media) and I 
cannot do anything with it because it is already taken. At the same time 
accessing it from your home directory is usually quite annoying. So in 
order to make sense of it I must precreate /media/xen and 
/media/dannielle and prevent the daemon (that /media thing) from wiping 
them.... buh.

The funny thing is that /media/xen is not even owned by xen.

The subfolders are, but the thing itself isn't (how stupid?).

So in this case perhaps not as convenient, but in the general case if I 
precreate:

/mytree/me/nas
/mytree/quinty/nas

etc.

Then the daemon could read

/mytree/*/nas

on program start and see all the shares that match. All the directories 
that match.

Then it knows that "me" and "quinty" are wildcard entries for their 
respective mounts.




>> 
>> 
>> 
>> 
>> > By definition a program map can't know what keys may be passed to it.
>> > And, yes, even autofs(5) doesn't say explicitly that a program map
>> > doesn't know
>> > what keys may be passed to it, perhaps that should be ammended.
>> 
>> In principle even a program map could read all of the available keys
>> from whatever database it consults, and return all of them; or return
>> something only (as it does now, perhaps) when a given key matches (is
>> found).
> 
> I've thought of that too.
> 
> But I've resisted because I'm concerned about confusing existing 
> program maps
> people have in unpleasant ways.

At some point you need to be willing to break previous compability.

I understand that Linux is usually a running-upgrade system and even if 
people do upgrade entire distributions they want the system to keep 
functioning as normal without changing anything.

But e.g. the Debian update program can tell you stuff like "This package 
does something different now, and you need to ensure it can either 
handle or ignore the wildcard".

on every package installation.



> That's not a bad thought, but what would $1 be so that older scripts 
> wouldn't
> choke on it.

I had suggested * because there is no hostname called *.

But scripts may very well mount local filesystems as well. And they 
might very well use a shell to do stuff like "[ -d $key ]" or "[ -d 
"$key" ]" and....

And as always, stuff breaks a lot when people don't use "$key". If 
people's scripts are badly written then [ -d $key ] will malfunction.

In order to cover that you would have to use something like "_", no one 
is going to use _ as an actual filename.





> 
>> 
>> So, for instance, if $2 is "browse" the script may choose to return a
>> browse map, or list of shares, with whatever options are available (it
>> could even return the UID/USER of the share given options, so that the
>> created directory would have the same UID as the to be mounted share!)
>> -- meaning it might simply return ALL entries it could possibly return
>> in the default format, which is the meaning of the wildcard, and the
>> program (autofs daemon or module(?)) would be able to generate the
>> appropriate directory entries based on that. So in that case you're
>> probably going to be rid of the problem of the non-matching state of
>> pre- and post-mount of directories.
> 
> which pretty much defeats the only possible solution I can think of for 
> "browse"
> option mounts ...

You mean the one suggested above. But if you make an alternative readdir 
having a list available won't bug you right?

I mean, having mount information available in advance cannot be a 
problem, only a help.




> Like I said, I wanted to have the default as "browse" but got hit with 
> too many
> complaints and was only willing to change the package installed default
> configuration to satisfy them.
> 
> I'm not going to try that again.

"Design by committee" they call it ;-).

Someone told me that OpenSUSE has had "browse" as the default for at 
least 5 years.

You did think about making a distinction between static and LDAP/NIS 
maps? I'm not sure I am not crossing my bounds here now ;-). Seeing that 
I already suggested it.





>> In this way no one can ever get confused, so you might just rename
>> autofs(5) to autofs.map(5).
> 
> Yeah maybe ...
> 

>> I will look into providing a patch, but I hope you can rename 
>> autofs(5)
>> to autofs.map(5) before doing anything like that. I don't feel like
>> improving a system I know myself haven't had access to, in that sense. 
>> I
>> don't mean you have to do that straight away, but if that is a 
>> possible
>> direction to go, or change to make, then I could supply a patch or
>> anything based on the system that would result from that, seeing that
>> autofs(5) is already pretty good in that sense.
> 
> At some point I'll need to return to this and make a list.
> Unfortunately that time isn't now.


Here is just my short summarize:


- suggested compromise between browse and nobrowse by only making browse 
for static lists the default to please new people unaware of what will 
happen

- suggested perhaps using _ as the wildcard key, not using that "browse" 
flag as $2, but allowing a different "primary" key as $2 if the script 
wants it.

- suggested changing autofs(5) to autofs.map(5) such that users will be 
able to find it.

And the rest is up to you, I think you don't need anyone rewriting it 
for you now, but I could still try my hand if that was necessary. I just 
think using better terminology is dependent on the first two suggestions 
here, particularly perhaps the second, and otherwise you'll have to 
explain current terminology better and create some way to separate "key" 
in auto.master from "key" in "autofs.map" ;-).

However if scripts could accept an additional "key" and the default 
operation or capability would for them to be able to return a "list" 
then it would become more clear that the additional "key" would be the 
index to retrieve that list, and that the entries resulting from that 
may have their own "keys" but these "keys" are then used for the 
individual "mount points".

So then you have reason to call the entry keys "entry keys" and the 
other ones "lookup keys" for instance. Then, static maps use "entry 
keys" but scripts have "lookup keys" as their 2nd parameter. The script 
then has "entry key" as the first, and "lookup key" as the second 
parameter. Then, entry key is equivalent to subdirectory, and lookup key 
is often going to be equivalent to e.g. hostname.

A script without list functionality only accepts "entry keys" but may 
very well use them as lookup keys as well.

A database also uses them as equivalent. But at least now you have 
something of a model to conceptually separate them. Since browse already 
functions for e.g. LDAP, in many cases you will not use a lookup key for 
that, simply because (a) /etc/autofs_ldap_auth.conf already contains 
such paramters and (b) the other parameters are specified in the 
mapname-format for LDAP.

So what we call "lookup key" would already be encoded in the LDAP URL 
but such a thing is not possible for scripts.

So you could call scripts the exception to the rule in not having this. 
Of course it exists,: the "-D" parameters.

(I think? Haven't tested it yet).

So I don't know.




For a database the entry key is always going to be a lookup key to the 
database.
Entry keys can also result from a list fetch.

So they can result from filesystem access or from some list retrieval.

But if "browse" becomes more normal and standard, you would see that it 
becomes easier to talk about this "entries from the list you get when 
quering the database/script for it".

Then the gap between "entries from the list" and "subdirectories 
created" is smaller than the gap between "filesystem access" and 
"subdirectories created".

Meaning: key (database designator) -> list -> entries (database entries) 
is conceptually easier to grasp than key (filesystem access) -> entry 
(database entry).

Or key (filesystem access) -> database lookup -> entry.

Simply because it is hard to understand that user action (filesystem 
lookup) is the input to the entire system and is actually the keys being 
used.

Even if it's not always to be used, having a level in between that ties 
"key" to "pre-existing shares" or "pre-existing entries" becomes easier. 
When a user thinks of "key" they think about either existing shares or 
existing hosts.

Then conceptually it is logical to assume that the script or database 
can give back this list. And that either this list is precreated on the 
harddrive, or at least that it is used to match the "keys" against, for 
instance.

(By harddrive I mean filesystem (hierarchy)).

This is why I think standard operation for at least smaller lists should 
be a form of browse or at least a form of list retrieval, if you are 
going to have dynamically generated lists in any case.

Then, it conforms more to a sense of what a general computer user is 
going to have of the idea of auto-mounting mount-points. It (he/she) 
thinks those mount-points are already going to exist, prior to being 
automounted.

This in turn makes it easier to explain that the key is really a 
subdirectory, and that there can be another key if needed to specify the 
host, or something of the kind.

Then, the most common use case for a new or general computer user 
wanting to try this is going to be met, because that use case is going 
to involve auto-mounting network-shares of which there is a limited 
number.

And it will involve something like

/nas/media
/nas/backup
/nas/home

etc.

Now I'm not sure if it covers the entire use. But at least you'd also 
have a script that can generate a static map for you that you can then 
use in place of the scripts, if you want to prune the mounts that are 
actually visible to you.

Apart from the distributions themselves there is not really any software 
in the Linux world that is tailored to either the desktop user, or the 
server user, but not both, is there?

Apart from the desktop environments.

Anyway this was a long read and write, I guess, for both of us :P.

Time to get some food again or at least some water :p.

See ya.











--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Ian Kent]

On Sun, 2016-07-10 at 19:07 +0200, Xen wrote:
> Ian Kent schreef op 06-07-2016 14:48:
> 
> > But I don't understand your use of "lazy initialization".
> 
> I meant creating the mount point only when it was accessed.
> 
> > So I don't quite understand what your concern is.
> > Sure, if you have hundreds or thousands of these there can be a 
> > performance
> > problem but that's not what you were trying to do I thought.
> 
> I'm not sure either at this point but I'm too hungry to read back on 
> what I had written before :p.
> 
> > Of course, yes, program maps can never know in advance what key may be 
> > passed to
> > them.
> > 
> > The problem is usually seen with user applications, cron jobs that scan
> > directories, GUI file management utilities etc.
> > 
> > It's has been quite a problem over time.
> 
> Yes that is my concern. I do not like the idea of invisible access 
> points at all, unless I was doing it for security, but in that sense you 
> can't really hide anything in a Linux system. Invisible mount points 
> only make sense when you have LOTS of them and you only need few at a 
> time.

And that is a fairly common case.
Don't forget the implications of scanning these directories during expire.
It doesn't take much to increase the overhead of that.

Like I said before, the amount of work to do this is huge.

What is needed is a re-write of the user space <-> kernel space communication
essentially to add readdir functionality.

So that listing a directory would show the possible map mount points without the
need to create the mount point directories.

That is still open to excessive unnecessary mounting but not nearly as much as
pre-created directories.

> 
> 
> > LDAP is a map source, like a file, or nis etc.
> > 
> > Maps stored in LDAP behave much the same as maps stored in files.
> 
> Right so you can browse them if you want.

Yep.

> 
> 
> > 
> > > 
> > > So then consider that we only use a static map here. An actual file 
> > > with
> > > thousands of entries? Of course it can happen. But what would be the
> > > point of that still (as opposed to LDAP or programmatic).
> > 
> > People do seem to like program maps but plenty of people use static or 
> > semi
> > -static maps, ie. a bunch of entries followed by the wildcard map key 
> > "*"
> > usually with some macro or "&" substution.
> > 
> > Similarly, what might match the wildcard map entry can't be known in 
> > advance.
> 
> Right. That makes it programmatic; the (server) response will determine 
> if anything was actually there. On the other hand a wildcard (in a 
> direct map at least) may determine existing (or match existing) 
> filesystem components that you could read in advance. And you need to, 
> in a direct map. But direct maps are mounted anyway (or the mountpoint 
> is created, I mean). So it currently only applies to indirect maps where 
> the wildcard is the final component, and it then behaves like a 
> programmatic thing where it only actually gets mounted (and created) if 
> e.g. the server reponds with a share.

A matter of terminology really, I don't see them as programmatic myself.

But also direct mount maps cannot contain a wildcard entry.

They must be distinct and known at the time the map is read because each entry
is used to mount an autofs file system to act as a trigger for the actual mount.

That's why they can't be nested in any way since that could cover other direct
mounts below the nesting point when a mount is triggered.

> 
> 
> > It's hard enough for me to understand VFS path walking sufficiently 
> > without
> > having to describe it too.
> 
> I don't know. I studied it in some detail, I just don't understand the 
> RCU algorithm. At least, I have studied path_lookupat and the entire 
> procedure leading up to lookup_fast and lookup_slow. I'd have to check 
> my printouts to say more. But let's just say I'm aware of any 
> (recursive) operation involving symlinks, where intermediate symlinks 
> get expanded before the original path is being continued, etc. I am also 
> aware of the presence of autofs code, particularly in the function that 
> specifies one or two other mount things.
> 
> I believe that is called by lookup_slow....

Yeah, while there are no locks held when following a mount point (where
automounting might be done) during the path walk you can't do most of the
remainder of the path walking code lockless or rather without taking references
to the struct path, you might see the terms ref-walk and I think rcu-walk in the
code, that's the main difference between lookup_slow() and lookup_fast() IIRC.

The name rcu-walk I think is possibly not the best naming.

It implies a lockless and reference-less path walk which is much faster for
certain work loads. But if the underlying structures are in danger of changing
in ways that can't be properly handled by the read-copy-update (RCU) mechanism
the path walk needs to revert to a ref-count(ed) walk which also allows locking.

That really just means taking a reference to the struct path (and the ability to
take a lock on certain other structures when needed) so it can't go away from
under you while you are using it, starting over when you hit such a case.

> 
> 
> 
> > All I can say is there is a problem determining whether a call back to 
> > the
> > deamon is needed or not at the times the kernel module is called. That 
> > can
> > sometimes result in all entries in the autofs managed mount point 
> > directory
> > being mounted.
> 
> So the mount point is accessed but the module doesn't know if anything 
> really needs to happen at that point. And stat operations or anything 
> else all use the same path lookup. Of course I don't know what happens 
> pre-mount and post-mount to the directory itself. If post-mount the 
> directory is really on another filesystem, the "mnt" value will change. 
> And it will return another dentry for that other mount. I remember 
> "path" being mnt+dentry.

Actually the main automounting function of the module will be called back with
essentially the path component only.

Most of the logic to decide if a mount should be done has now been taken into
the VFS mount following logic.

And all it has are the lookup flags, like LOOKUP_DIRECTORY, LOOKUP_CREATE, etc.

Essentially all that's known is the class of lookup, directory, create, continue
or other, which is a loose division of the related system calls, where continue
means an intermediate path component.

The directory, create and continue classes trigger automounts, other classes
usually don't (which are generally stat type system calls et. al.).

> 
> > 
> > Another aspect of this problem is that the same system call may need to 
> > have the
> > directory trigger a mount while not so at other times it's called and 
> > there's no
> > way to tell what the usage is within in the kernel.
> 
> So you are getting called by lookup_slow or something of the kind near 
> it, but you're not sure why. But these lookups are being done for each 
> individual path componenent. However a "cd" to the directory should 
> mount it, but a mere "stat" may not. Is that the case? Seems odd that it 
> would be difficult, but I believe you on your word.
> 
> Maybe "cd" shouldn't mount it until the contents is accessed, but I 
> don't know.

Yeah, but "cd" isn't a system call it's a bunch of code that makes system calls
which must result in a mount because this directory is to be set in the VFS as a
process working directory, it must be fully resolved at that point.

> 
> 
> > 
> > So sometimes people get caught because they want their stat(2) call to 
> > return
> > the mounted file system results but not to cause a mount at other times 
> > when it
> > might lead to a mount storm.
> 
> Hmm. It seems like the first desire is a problem. At least, I don't know 
> how directory lookup is done. I don't know if this information is the 
> result of a stat call or something else. I do remember seeing code for 
> dentry-moving but it was confusing.
> 
> For sure you want "ls /directory/" to return the mounted contents. But 
> I'm not sure if its possible for your module to differentiate. Ls -d 
> "/directory/" should perhaps not cause mounting. I can see quickly 
> though that "ls" causes an "open" call but "ls -d" does not.
> 
> But then, "open" will not access any path components beyond the 
> directory itself. So if your module is only called by path_lookup it is 
> going to be impossible to know what for.
> 
> Particularly if an "ls" does not actually access path componenets 
> (beyond the directory itself) there would be no way to know what is 
> being asked.

There's one other thing to be aware of in all this.
Adding a "/" to the end of a path tells the kernel path walk this must resolve
to a directory so the LOOKUP_DIRECTORY flag will be set and automounting will
occur in that case.

That is about all the control available in user space and TBH not a big change
to code if you really need stat to return the actual mounted file system
information, given that non-directories can't be mount points and so aren't a
problem.

> 
> 
> > Usuall all I can say to them is don't use the "browse" option.....
> 
> :).
> 
> 
> > It is a difficult problem and I'm inevitably caught in the middle 
> > between the
> > "browse" and "nobrowse" camps.
> 
> You appear to be very conscientious about it.
> 
> 
> > Not all cases of accessing such a directory will see this problem 
> > either but
> > user space seems to have a habit of using system calls that don't 
> > trigger a
> > mount together with others that do.
> 
> such as that "ls" vs "ls -d"?

Can't remember the difference between these two codewise but, for example, doing
a stat(2) call then an opendir(3) and not being able to handle the fact that
this changes from a directory that doesn't have a mount on it to one that does
between the two system calls.

Which has caused find(1) to get very confused (in the somewhat distant past now
though), ;)

> 
> 
> > Causing the so called mount storm.
> 
> Still seems odd but I believe you, but. I guess what it could mean is 
> "ls" not only trying to list every share (browse) but also to perform 
> calls on those shares (path componenents retrieved) that might require 
> additional information. For example, if I am not content just knowing 
> the shares, but also want to know the number of files in them.... which 
> definitely some programs or user agents would do. Then I'm in trouble 
> because this requires mounting. Then, if that is the case, a simple
> 
> "list /nas/" might result in the subfolders (/nas/media, /nas/backup, 
> etc.) to also be queried and you have what you call a mount-storm.
> 
> But this is the result of user space convention and behaviour. Surely 
> people cannot request browse while at the same time wanting to deny 
> actions on the returned results when these actions stem completely from 
> userspace?
> 
> If some user agent or GUI is trying to list the number of files in each 
> directory (which at this point means a share) then that's not something 
> autofs could ever deal with. And I don't think it is a problem autofs 
> needs to solve.
> 
> If I am at all correct about this and I see what you mean right now, 
> this "GUI" behaviour would then simply be incompatible with unmounted 
> folders *that-might-need-to-get-mounted-but-not-always* and this GUI 
> would need to be aware to not do that thing.

That's about right, and isn't a problem until someone with a largish map set for
browse decides to access it with a GUI, then the complains start, ;)

But it isn't an unreasonable expectation it's just problematic to make it work
the way we would like.

> 
> 
> 
> 
> > 
> > Having said that though, with a lot of work, I could probably do 
> > something about
> > it.
> > 
> > TBH I've avoided it because it is difficult, perhaps in version 6.
> > 
> > It would require not pre-creating mount point directories and adding a 
> > readdir
> > implementation to the kernel to fill in the names (map keys) that 
> > aren't already
> > mounted. But that means significantly more (in terms of quantity of)
> > communication between the daemon and the kernel module.
> > 
> > That means re-writing the user space <-> kernel space communications 
> > completely
> > in an incompatible manner to what is used now, not to mention 
> > significant
> > changes to the user space code itself.
> > 
> > A big job really.
> 
> And not a big joy perhaps.
> 
> So basically you are going to not precreate, but still hijack kernel 
> calls to return the listings for the directories that do not yet exist 
> (the listing of those directories).

Yes, similar to what the NFS READDIR or READDIR_PLUS RPC methods do.
It's not a new idea by any stretch.

> 
> Which means those directories could be shown, but not opened, or 
> something of the kind. Then, the user agent .. well I don't know what 
> difference it makes actually if the mount storm is the result of 
> userspace calls that directly cause it. So I guess I'm mistaken here and 
> the issue is even more complex.... :(.
> 
> I don't know how a readdir on the "top" directory could /directly/ 
> causes a mountstorm on the listed entries. But then, I don't know much 
> ;-).

It's usually not actually the readdir that causes the problems it's associated
calls done by code surrounding the readdir that generally it.

I mean if stat(2) was to cause a mount to occur then there would be a lot of
mount storms reading directories.

Implementing an in kernel readdir won't get rid of the possibility of everything
in a directory being mounted on access but it would go a fair way to avoiding
cases where you didn't want or need everything mounted upon listing a directory.

Just how effect it would be has been something I've thought about many times,
still not sure of the actual answer.

> 
> 
> > It happens, from what I've seen, quite a bit.
> > Perhaps you would be surprised.
> 
> Illuminate me :P. What is the most common cause of it?

GUI applications.
Cron jobs scanning directories and causing so many mounts they cripple the
system.

The point of the statement was more about number of people that happen to have
large maps than anything.

> > Yes, but network sources such as NIS and LDAP are treated much the same 
> > as a
> > file map.
> 
> Yes, I didn't realize at the start. I have always found LDAP to be a 
> rather incomprehensible thing to work with. Without hands-on experience 
> it is hard to get a good grasp of how it works. Or even how to organise 
> it (if you do create one) as there seem to be standards for everything.

You think just checking it out looks incomprehensible, the autofs LDAP lookup
module code is very verbose (incomprehensible really) and quite hard to work
with.

I'd be much happier if LDAP never existed.

> 
> 
> > Checking if a file map has been updated since a key was last used is 
> > easier with
> > a file map and straight forward with a NIS map but can't sensibly be 
> > done for an
> > LDAP map.
> 
> Right.
> 
> > 
> > > 
> > > Maybe I'm missing something here.
> > > 
> > > 
> > > > Another reason for the default is that, because of this, historically
> > > > there was
> > > > no "browse" option and this maintains what is (should be) expected.
> > > 
> > > Yeah I thought so, because of the comments near the timeout value.
> > > However that is a reason to never change anything ever. Someone
> > > mistakenly choosing "browse" will very quickly find out.
> > 
> > And wouldn't itself be sufficient but for the problems I'm attempting 
> > to
> > describe above.
> 
> You mean the mountstorm problems are reasons to keep the default at 
> no-browse?
> 
> It still seems at this point that this is only relevant for LDAP/NIS 
> entries.
> 
> 
> 
> > > First the auto.master file on my system mentions man autofs as a place
> > > for seeing the format of the (sub)map files. However there is nothing 
> > > in
> > > there. So that part is missing, but in a sense the contents of 
> > > auto.misc
> > > is already somewhat sufficient for that, but still annoying that there
> > > is no man page.
> > 
> > That doesn't sound right, there might be a problem with your 
> > distribution
> > package.
> 
> No, as I say later this is incomprehension on my part ;-). Or merely the 
> fact that needing to use "number" for accessing "sections" of the 
> "manual" is so rare (if you are not a developer accessing library 
> functions and/or kernel functions) that a regular user will almost NEVER 
> use a section number in daily operation.

LOL, I got that later, yes.

> 
> Most configuration files are called "openvpn.conf" or something of the 
> kind, and hence their manual page is also called that, and you don't 
> need to do "man 5 openvpn.conf", because "man openvpn.conf" is 
> sufficient.
> 
> Using numbers would be superfluous in 99% of cases, or even more than 
> that. As opposed to perhaps requesting help on function calls, such as 
> mount() as opposed to /bin/mount.
> 
> So as a developer you may be used to it. But a regular user is not. 
> Anyone who does not do C programming, is not.

Yeah, maybe, but you still need to account for the case when there are two
things of the same name, one configuration oriented and one system orientated,
one belonging in section 5 and one belonging section 8.

In the autofs case autofs.conf(5) has it's own man page for configuration and
autofs(8) for startup information and autofs(5) for mount map information.

Actually seems mostly sensible to me, since while mount maps are configuration
in a sense they don't quite fit with the actual application configuration of
autofs.conf(5).

> 
> So, my mistake, I guess.
> 
> > > 
> > > You don't imagine that every freaking access of a directory or path
> > > component in the managed directory is going to result in network 
> > > access.
> > > 
> > > You could do a "for f in `seq 1 10000000`; do vdir $f; done" and have
> > > fun.
> > > 
> > > (Don't know if that would still be the case for "browse" maps).
> > > 
> > > So this hidden directory, secret door thing is really the most 
> > > confusing
> > > of all.
> > > 
> > > It's somewhat similar to something I might like to do, but still you
> > > don't really expect it ;-).
> > > 
> > > So the 3-4 things are:
> > > 
> > > create a man page or section for the format of the sub-maps
> > 
> > Already exists, needs work.
> > Lets call these mount maps (not the best but the convention used) as 
> > opposed to
> > the master map.
> 
> Alright. Still confusing though.
> 
> > 
> > > indicate more clearly that programmatic maps in principle return a
> > > single entry
> > 
> > Maybe but isn't this in autofs(5) already close:
> > 
> >    Executable Maps
> >        A map can be marked as executable. A program map will be called
> >        with the key as an argument. It may return no lines of output
> >        if there's an error, or one or more lines containing a map entry
> >        (with \ quoting line breaks). The map entry corresponds to what
> >        would normally follow a map key.
> 
> I guess.. it's just that you read over "a map entry" as meaning "only a 
> single map entry" since just before it says "one or more lines". So it 
> is something you only notice if you reread it in detail. After finding 
> out that it doesn't work ;-).
> 
> 
> 
> > Isn't this near the top of autofs(5) sufficient:
> > 
> >        key [-options] location
> > 
> >        key
> >               For indirect mounts this is the part of the path
> >               name between the mount point and the path into
> >               the filesystem when it is mounted. Usually you
> >               can think about the key as a sub-directory name
> >               below the autofs managed mount point.
> > 
> >               For direct mounts this is the full path of each
> >               mount point. This map is always associated with
> >               the /- mount point in the master map.
> 
> Yes it is, particularly because it mentions that it is a sub-directory 
> name.
> 
> Then, I begin to wonder what "usually" means. See, from the looks of it 
> it is completely identical to sub-directory name, because:
> 
> - they key is going to be directly used as the name of the subdirectory
> - programmatically, the script does not even have any say about the key, 
> it cannot even return a key, does not determine the key. The only thing 
> that can ever happen is that the key as given is turned into a 
> subdirectory (when valid).

Yeah, I get what your saying and it makes sense.
But it could be a wildcard .... 

> 
> I mean the logical abstraction between "key" and "value" may make sense 
> from a purely abstract logical point of view ;-). But it has no bearing 
> on actual domain-specific termininology, it is just a computer science 
> term so to speak.
> 
> What it actually comes down to is:
> 
> - mount point for the actual share, or
> - subdirectory created as mountpoint for the actual share(s).
> 
> Where "value" would mean:
> 
> - path to the mount (share), or
> - network path to the share that gets mounted on the subdirectory (the 
> key).
> 
> the "mount" utility calls these source and target, respectively (source 
> for the path, target for the mount point). Otherwise it calls them 
> "device" and "mountpoint".
> 
> So although "key" also has meaning with regards to "database lookup" I 
> don't see how it is ever not directly used as the mountpoint itself. And 
> this is what creates the confusion because with the "browse" option it 
> is not just going to be a "lookup key".

From automount(8)s POV it is always a lookup key that it wants to match to a
mount location.

> 
> Further there is no difference between "subdirectory mount point" and 
> "lookup key", where these are fundamentally separate concepts, and could 
> even imply, in principle, that the actual directory to use is *also* 
> retrieved by the key.
> 
> So conceptually the idea that "key" instantly means "subdirectory 
> mount-point" is confusing.
> 
> In fact it is odd to begin with, from the perspective of a user, that 
> "auto.smb" mounts the entire share list of the requested "host" or "key" 
> in one go under the location that was accessed.

Yeah but I'm not going to get to excited about an example program map that gets
installed by the package.

> 
> This is not the most common scenario (well, maybe if you are in some big 
> network). This is not most common not only because it mounts /all/ the 
> shares at once(!) but also because generally people want to have some 
> level of control over the directory it ends up in, and what is the host 
> for some server (hostname) might not be where you actually want to mount 
> it.

Sure it is a bit of a hurdle but the autofs internal hosts map works that way
and has been an expected automount feature for a long time.

I agree that doesn't cater for new users and I don't find creating and
maintaining web presences at all pleasant, quite the opposite, so a wiki for
this sort of information is probably not going to happen any time soon.

> 
> So I suggest that even this auto.smb is a bit of a weird entity. I have 
> used it to generate a map list for me that I then adjusted and used as a 
> static map file.
> 
> I know it does the same as that -net or -host thing. It's just that for 
> a desktop user (again, you say that is not the target audience, but 
> still) what you really want is a fixed location the user chooses (in 
> auto.master) and then a browseable list of shares. Apparently this list 
> needs to be static. What I envisioned before would be a parameter to a 
> script that causes it to output a list of shares rather than the share 
> entries themselves, or just the complete list of it.
> 
> This would give it "browse" capability. If you had that, you would need 
> to be able to pass a parameter to the script as the key, but this is 
> rather obscure.
> 
> What it comes down to is that this is what I would envision for myself:
> 
> Auto.master:
> 
> /cifs   /etc/auto.smb   diskstation                <-- diskstation is 
> the key
> 
> Now the program/daemon/whatever passes diskstation as key to the script. 
> It also passes a wildcard parameter such as "*". The script now returns 
> a list of shares for "diskstation". Subsequently when the daemon needs 
> to mount, it calls the script with diskstation and the requested mount 
> point/share. Alternatively, the returned output from the first call must 
> be complete and is saved by the daemon. It is then used in place of a 
> static map (the script is now nothing but a static map generator).
> 
> Personally I prefer the former: the script has two modes: list and 
> return entry, and it therefore has two parameters: one to identify what 
> to list, and the second to determine what entry to return from what was 
> listed.

I have thought about it a number of times.

It is necessary to remember that whatever is done must be backward compatible
for existing users program maps.

One though was similar to your suggestion above, where diskstation would be
introduced as a mount option, understood to be a program map parameter by
automount(8) for certain lookup operations like the lookup module readmap entry
point.

That's about the closest I've come to something that should work.
Still I haven't actually implemented it.

There's still a lot of detail about this approach that is unclear though, such
as what should the parameter be, how many would be needed, where to store it
internally, etc.

> 
> Otherwise, without the parameter, you'd need to hardcode it in the 
> script and duplicate the script for every host, etc.
> 
> In fact as a user you may not even want all shares to be present on your 
> local system. So you might equally well decide to use a static map. So 
> perhaps you will use "smb.auto list" or "smb.auto list diskstation" or 
> "smb.auto diskstation *" manually. And perhaps it generates all of the 
> mappings for you that you can then prune and save.
> 
> Perhaps, in absense of the key (the entry, or wildcard) a call such as 
> "smb.auto diskstation" should simply generate the share list, whereas 
> "smb.auto diskstation media" generates the key for only that entry.
> 
> So you see me use a term like "entry" to determine what you today call a 
> "key" and the word "key" to determine what you today call a "entry" ;-).
> 
> "entry" means "element of pre-existing list", "key" means "value to use 
> for a lookup operation in a database".
> 
> "key" can also mean "that what you need to mount or access something".
> 
> 
> 
> so if you combine the two, you have like three things:
> 
> - subdirectory (mount point)
> - key to use for database lookup
> - entry in the list of shares
> 
> but you can only solve it if you differentiate between "lookup 
> operation" and "element of a list".
> 
> a lookup operation may /return/ a list, which is why I said:
> 
> "diskstation" (hostname) is a key
> list of shares is the result of that lookup operation based on that key
> entry is an element of that list you can mount.
> 
> This applies to a form of browse scenario. If you have invisible 
> mounpoint, you get:
> 
> "diskstation" (hostname) is a key that is going to be a direct 
> subdirectory
> the list of shares it returns is going to be a subdirectory list to that 
> first subdirectory
> this will contain the actual shares, so you get top/key/entries.
> 
> And you see we basically have the same thing except that this is one 
> level deeper in our hierarchy.
> 
> In my desired use case, you get:
> 
> top/entry
> 
> But in the current implementation of e.g. smb.auto, you get:
> 
> top/key/entry
> 
> So it is clear diskstation (hostname) IS a key, except that in the 
> current case or scenario this key is ALSO used as the direct descendent 
> of the toplevel directory used for the configuration in auto.master.
> 
> Which gives rise to the idea that maybe "key" and "subdirectory mount 
> point" should not be equivalent or identical. Not necessarily identical.
> 
> 
> In fact, smb.auto DOES produce a browsable list. It is just /beneath/ 
> the subdirectory for its "key" (the hostname) and mounts ALL of the 
> shares at once.

Actually it doesn't actually mount the shares.

It mounts autofs trigger mounts that will cause the actual share to be mounted
on access.

> 
> So you now already have that mountstorm (it just gets mounted by 
> default) it's just a level deeper.
> 
> You now don't have a mountstorm on the level of hosts, but you have a 
> mountstorm on the level of shares (per host) but it is still the same 
> thing.
> 
> So the design specifies that keys must become mountpoints in the 
> "sub-top-level" directory (the directory managed by autofs for indirect 
> mounts).
> 
> Whereas perhaps conceptually you would need to differentiate between 
> key, and an entry used for mounting, and the difference is that this 
> entry used for mounting can be an element of the list returned by the 
> key.
> 
> So you get key -> list -> entries. In the case of current auto.smb, what 
> happens is:
> 
> key -> (subdirectory) -> list (itself an entry) -> entries.
> 
> Or in other words:
> 
> key -> list -> entries, but mounted beneath top/key, as key is used as 
> an extra directory in between.
> 
> Current auto.smb:
> 
>   <managed mount point>   /    <key>   /   entries  / contents of shares
> 
> My preference:
> 
>   <managed mount point>   /    entries     /   contents of shares
> 
> And why? Only because I want to specify mount points for specific hosts 
> manually.

auto.smb is an example program map.

Your free to implement your own program map any way you wish.

The thing that gets passed to the program map is a string that represents the
trailing path component of a path the kernel believes needs something done to
it, be it symlink it somewhere, bind mount it somewhere or lookup a location and
mount that on it etc.

You must adhere to that restriction because that's how automounts are triggered.

> 
> 
> 
> So the question becomes how you can combine such things.
> 
> 
> The requirement becomes for the script to return the mount point. A 
> static list does return mount points. A script does not, the mount point 
> is specified as the "key" given to it. So if you want the script to 
> determine the mount point the way a static list does, you need it to 
> return, in essence, a static list.
> 
> Even if your static list is not browseable, so too can a returned 
> "dynamic list" also be not browseable. But at this point your script can 
> return a mount point of its own. That is again problematic if the key is 
> derived from filesystem access.
> 
> So you see how hard this is to reconcile.
> 
> If we insist in accessing hidden shares and deriving keys from user 
> action or filesystem action, or filesystem access, then keys are 
> predetermined the moment the system is actually called. The script or 
> database can only return whether something is gonna be there or not.
> 
> Then the only way to reconcile it is (in my case) for the key to be not 
> the only argument to the script, but I give the script my argument I 
> define in auto.master:
> 
> /cifs   /etc/auto.smb   diskstation
> 
> Now my script knows that e.g. $1 is gonna be the filesystem requested 
> path, and $2 is gonna be the custom parameter I determined in advance.
> 
> I don't need to use the key by itself ($1) to do the lookup. My real key 
> is $2. But I will use the second key ($1) to return a share for the host 
> determined by the first key ($2).
> 
> In essence what happens with auto.smb is that the key ($1) is used as 
> the host (first key) and smb.auto will itself derive and load the list 
> of shares, and then instantly mount that (it retrieves all of the second 
> keys and instantly uses that and mounts everything).
> 
> autofs allows for this second list of keys to be instantly used and 
> utilized.

One other consideration is the lazy mounting (and umounting) of these multi
-level mount trees that autofs manages.

It's very easy to get that wrong and end up with an inoperable mount point that
cannot be recovered.

At the moment automount(8) will do that for you and occasionally gets confused
itself.

To do what is wanted here it would be necessary have a way to tell automount(8)
leave the mountpoint alone.

> 
> But it's not the common use case because you are now fixed to that 
> location the user accesses and of course you could use that location 
> (the key) ($1) to look up a host in a different way, that is perfectly 
> possible. If your script marries "diskstation" to "ds" or "ds" to 
> "diskstation" then you could automount /cifs/ds and it would request 
> shares from host "diskstation".
> 
> But you're still stuck with that intermediate level, that extra level of 
> subdirectories.
> 
> It's not possible currently to use the "first key" to determine the host 
> and then get the "second keys" ($1) to determine the entries. Well, 
> actually it is, but not in a browseable fashion:
> 
> /cifs   /etc/auto.smb   -Dhost=diskstation
> 
> 
> I believe this could work. You pass your real "primary" key as that 
> variable, and your script returns shares based on that key, but with the 
> "filesystem key" ($1) as the name of the share to return.
> 
> Still, rather convoluted at this point, and not possible to browse this 
> list.
> 
> Because it doesn't return a list. Using $2 to pass that variable 
> wouldn't really be different.
> 
> But the script is only called when an entry is accessed (through the 
> filesystem) and as such is never called without a secondary key.
> 
> The current auto.smb doesn't have secondary keys, it loads and mounts 
> all of them instantly and automatically, and there is no "automount" for 
> the secondary level.
> 
> Well, there is, it just mounts all of it in one go.
> 
> So how do you skip the intermediate level?
> 
> - call the script on purpose to generate a 'static list' instead of just 
> for entries alone.
> 
> - use the static list to precreate directories (mount points), or
> 
> - use the static list to return readdir information, or to filter 
> requests for directories, or don't use the static list at all and just 
> keep it the same.
> 
> 
> So we also see here that the static list *could* be used for the thing 
> you described above, that alternative readdir implementation.
> 
> Although it's a bit of a hack, I'm sure.
> 
> You could specify some interval for invalidating the list. At which 
> point it is renewed by another "get list" call to the script.
> 
> 
> So in essence I am really advocating "return list" functionality for 
> scripts here.
> 
> And there is no easy way to implement that while staying compatible with 
> current scripts because you would need to: call the script with an empty 
> parameter (that will never return an entry) or with a wildcard (that 
> will probably never return an entry) and in essence, that might just as 
> well mean, that if you want to take this approach:

Umm .. I think the other possibility is that if you pass an empty string to an
old program map you might cause it to crash.

> - use the wildcard as the "list" operation, scripts that do not know 
> about the wildcard will simply return zero entries.
> - use the extra parameter as $2, in other words: allow for additional 
> parameters in auto.master in that form of:
> 
> /cifs   /etc/auto.smb   diskstation random
> 
> But there are already recognised options. So what we have today (-D) I 
> guess already works, it is just a bit obscure, and it seems to be the 
> equivalent of what you would do.
> 
> Then an alternative auto.smb (I called it auto-short.smb) would:
> 
> - be called as:
> 
> /cifs   /etc/auto.smb   -Dhost=diskstation
> 
> - return a list of shares when called with "host=diskstation 
> /etc/auto.smb *"
> - alternatively you could have it be called with: "/etc/auto.smb * 
> diskstation"
> 
> - make sure the shell doesn't expand that * ;-).
> 
> - autofs calls this list operation each time auto.master is reread.
> 
> - autofs calls this list operation each time some invalidation timeout 
> occurs
> 
> - not really any other way to trigger it.
> 
> - the static list being returned (really a dynamic list now) can be used 
> for readdir, or even access filtering (it contains all the valid keys).
> 
> 
> Now when you have something like this, you can:
> 
> - call the filesystem component you use to access the actual shares, the 
> "entry" or simply "share" or "directory". But "key" becomes more 
> comprehensible now as well. "key" now corresponds to "entries in the 
> returned list". Each "entry in the returned list" is now a "mapping 
> between key (directory) and value (device-path). I realize "mount point" 
> might confuse with the mountpoint defined in the master file.
> 
> "submount" would be ugly. "subdir" is most convenient. "subkey" is 
> illustrative. "entry" describes the entire thing, not just the key. But 
> in essence we are just talking about a mount-point here.
> 
> Nevertheless in principle the "first key" given should not even need to 
> be the mount point in fact. It is a key to a lookup-table.
> 
> If you wanted to do it right I think you would need to do away with 
> hidden (or principally unknown) mount points, and basically always work 
> with static lists (or dynamically returned lists) and/or give every 
> script a first parameter $1 that is a key that you actually use for it.
> 
> Then the second paramter $2 would be the key that the filesystem has 
> actually given you for accessing a path.
> 
> In this way "access path" becomes secondary.
> 
> Perhaps a script in this way still cannot control the mount point (name) 
> but at least now it provides it in advance. Of course there are 
> situations in which you cannot know it in advance or in which it is 
> irrelevant to know it in advance.
> 
> You can imagine a list to be much too big to actually retrieve it. In 
> itself a sign of a bad system, I think. If it was that big, you would 
> need hierarchy to make it smaller.
> 
> So I don't know if there is ever any real need for obscurity, and for 
> hosts to be accessed while not being able to list them in advance.
> 
> I conjecture that this use case in itself would be odd, that you create 
> auto-mounts for an unspecified number of hosts you don't really know 
> about in advance.
> 
> It makes for zero-configuration, true. You don't have to keep a list 
> anywhere or even arrange for it to be generated.
> 
> On the other hand if a list could not be had, the script would just 
> return nothing and the daemon will assume no-browse 
> no-information-in-advance lookups the way it does today.
> 
> Basically if you keep the current $1 and use it to pass a wildcard, no 
> script is likely to ever break on that (it will just return nothing). 
> Then $2 can be your primary key. Not every script needs it.
> 
> But complete zero-configuration is to me an odd thing: a magical 
> directory that will mount every host given to it including all of its 
> shares, does not seem sensible to me in the sense that you would at 
> /least/ want some configuration for every system right?
> 
> One size fits all: the same system for every computer, just place it 
> somewhere in your filesystem and you have access to everything and 
> everything.
> 
> No, normally you would want to tailor it because you do not have access 
> to an infinite amount of hosts and the /cifs/host/share structure 
> therefore makes not much sense.
> 
> /device/share may make more sense, perhaps added with
> /hosts/hosts-meeting-a-certain-pattern/share
> 
> You already have a pretty big system if you use the latter. Anyway.
> 
> I think I should conclude this :p.
> 
> 
> 
> What seems sensible to me at least is to add something that doesn't 
> break compability by using a wildcard * for scripts that will return a 
> certain format of list (or just all shares it can find in the regular 
> format) with a $2 parameter designating something such as a "host" or 
> perhaps even "department".
> 
> In this way you solve the issue that the current auto.smb requires the 
> "hostname" to be first key given to it because there is no other way to 
> pass a key. Then the actual keys can be passed as "second parameter" in 
> this sense that they still become $1, but at least now you can specify 
> your host to it in advance instead of guessing it on the wind ;-).
> 
> So if I have to say anything I would say that I recommend this 
> structure:
> 
> - $1 remains the filesystem key
> - $2 becomes the additional key a user may specify in auto.master
> - $1 as * will designate the request for a list lookup
> - the list can in the future be used for readdir implementation, and 
> currently for precreated mountpoints in case "browse" is used.
> 
> 
> 
> 
> 
> > > indicate that this is true for "no browse" on "static maps" as well, 
> > > and
> > > that directories are only created when they are accessed.
> > 
> > That seems to be missing from the man page.
> > 
> > > And that no-browse is the configured default.
> > 
> > Not sure about that either.
> > 
> > This (from auto.master(5)) says that I have made "browse" the internal 
> > program
> > default way back when version 5 was first developed but I install a
> > configuration that turns it off because of (because of the above 
> > problems, which
> > are not spelled out in the man page):
> > 
> >      [no]browse
> >             This is an autofs specific option that is a pseudo mount
> >             option and so is given without a leading dash. Use of the
> >             browse option pre-creates mount point directories for
> >             indirect mount maps so the map keys can be seen in a
> >             directory listing without being mounted. Use of this
> >             option can cause  performance problem if the indirect
> >             map is large so it should be used with caution. The
> >             internal program default is to enable browse mode for
> >             indirect mounts but the default installed configuration
> >             overrides this by setting BROWSE_MODE to "no" because
> >             of the potential performance problem.
> 
> I know, but if you, as a user, do not understand what the hell it is 
> doing in the first place, and you have no time for extensive manual 
> reading in advance (like me) you will never get to that option because 
> you still don't understand the basics, and the more advanced stuff (like 
> that) that will be too hard to understand.
> 
> That information is too difficult to comprehend (or digest) when you are 
> someone who still doesn't understand how it works.
> 
> 
> 
> > > I think that limitation indicates why using relative paths is so much
> > > more pleasant. A map file using relative files can be "moved around". 
> > > A
> > > map file using absolute paths cannot.
> > > 
> > > Could you not make a direct map out of the key of the master file, and
> > > the keys of the sub-maps?
> > 
> > Can't really do that.
> 
> It's not that I saw it as necessary, I just still think the difference 
> and distinction between "direct" and "indirect" is rather I dunno, weird 
> I guess ;-).
> 
> 
> 
> > The closest thing to that would be multi-mount map entries, again from
> > autofs(5):
> > 
> >    Multiple Mounts
> >        A multi-mount map can be used to name multiple filesystems
> >        to mount. It takes the form:
> > 
> >        key [ -options ] \
> >            [[/] location \
> >           [/relative-mount-point [ -options ] location...]...
> > 
> >        This may extend over multiple lines, quoting the line-breaks 
> > with
> >        `\´.  If present, the per-mountpoint mount-options are appended
> >        to the default mount-options. This behaviour may be overridden
> >        by the append_options configuration setting.
> 
> Something I am already using (almost) and I believe this is what 
> auto.smb uses by default too. It's just that I use it to mount parts of 
> the same share. (Samba smbd often has issues with file-system boundaries 
> so I create a share out of each filesystem and share them individually, 
> then mount them as above).

It does, yes.

> 
> 
> 
> > > 
> > > normally ( /bla/bla ) indicates an absolute path and can only be used 
> > > in
> > > a direct map, from what I've seen. You *could* cause it to turn this
> > > "indirect map" into a direct one.
> > > 
> > > ie. ( bla/bla ) == indirect
> > >      ( /bla/bla ) == direct.
> > 
> > In the Sun map format maps indirect mount map keys are a single path 
> > component
> > only.
> > 
> > The situation is different if using amd format maps but I could work 
> > out how
> > direct mount maps could be usefully used so that mount type has not 
> > been
> > implemented. Even on Solaris where they are supposed to be fully 
> > supported I
> > could make sense of them.
> > 
> 
> 
> 
> > > /start/of/the/-  ( path )
> > > 
> > > Then any path in the master file ending with /- would be a direct 
> > > path.
> > > 
> > > And would cause the (static) map file to be treated as direct.
> > 
> > Nope, that is unlikely to work and would lead to a lot of problems.
> > 
> > One example might be with:
> > 
> > /start/of/the/-/path
> > 
> > people will ask where have the directories below /start/of/the/ gone 
> > and the
> > answer will be they have been covered by the autofs file system mount.
> 
> But that's no different from a regular mount on that location.
> 
> 
> > Next it would be but I should be able to see these, surely that's easy 
> > to fix.
> > 
> > At hat point I would probably leave the conversation.
> 
> :-). Haha. But come on, you can't take that seriously, the same thing 
> happens with a regular mount, even a bind mount does that. It's 
> absolutely not possible to reveal the contents of some subtree when 
> another filesystem had mounted on top of it, except by remounting the 
> entire subtree elsewhere.
> 
> (I am trying out some gimmick that requires me to mount the root device 
> under something like /run/mount/rootfsclone, because I am bind mounting 
> on top of something that I still want to be accessible. The only way to 
> make it accessible is to mount the entire rootfs somewhere else. I 
> basically need a persistent location for individual devices without 
> anything mounted on top of them; ie. a pure one-file-system mount tree.
> 
> Then you have those individual mount-trees (filesystem trees) and THEN 
> you have the complete FHS that you normally use.
> 
> That seems like the only sensible way to approach a lot of issues. 
> Having a /mounts/ with every (custom) mounted device present there.
> 
> Recently the number of incomprensible mounts reported by "mount" has 
> grown so big that the entire tool has become unusable, mostly due to 
> cgroups. And tmpfs's, and debugfs's, and what not.
> 
> I now find I need to create a wrapper for mount so that "mount" stays at 
> least usable to the user. Such that every subdirectory of /sys, and 
> every subdirectory of /run, and every subdirectory of /dev/, is actually 
> hidden from display. And from /proc as well. And the same applies to DF.
> 
> (I really need to find a way to better save and redistribute these 
> modifications).

Oddly enough, a pseudo mount option like "hidden", visible in the /proc files
but left out by utilities like mount, df etc. is what could be done, and should
be done, IMHO, but no-one wants to do it!

> 
> 
> 
> 
> 
> 
> > 
> > In any case a similar thing, but more explicit and I think easier to 
> > follow and
> > maintain are the autofs multi-mount map entries, what I think you 
> > called
> > relative path above.
> > 
> > It isn't clear but that syntax should function within both direct and 
> > indirect
> > mount maps.
> > 
> > The problem with all this is that substitution cannot be used in the 
> > key.
> > 
> > The reason for that is same as why the amd regex key matching has not 
> > been
> > implemented.
> > 
> > A linear map scan would be needed for every lookup in order to match 
> > what has
> > been sent to us from the kernel. And as much as you might feel that 
> > wouldn't be
> > a problem because maps are generally small I can assure you it would 
> > make life
> > really unpleasant for me!
> 
> :). You have lost of enemies; if you don't do things right :P. LOL.
> 
> I am not thinking about this now, but I guess you mean that my proposed 
> solution of saving those static lists (or dynamic lists) would actually 
> help this :p.
> 
> Actually what you are saying is that even doing lookups on those lists I 
> proposed would be quite annoying.
> 
> However currently this lookup is already done on the server (if it is 
> some database) or in the static map (if they are not browseable). Right?

And in the locally cached map and quite frequently queried as part of the expire
processing.

Direct maps and maps that have the browse option must be read completely at
startup and are always present in local storage.

> 
> Databases are generally better at lookup than small-scale applications. 
> Now you need hash-tables etc. Anyway, I can't think about it right now.
> 
> 
> 
> > > I only noticed just now because autofs(8) references autofs(5), and 
> > > I'm
> > > like ....huh?
> > 
> > Yeah, that's what we have ....
> 
> 
> 
> 
> > > /media/*/nas    /etc/nas-shares.map
> > 
> > Can't do that, each /media/*/nas needs to be mounted at program start 
> > so that
> > automounts can be done?
> 
> Actually the shell easily expands such a pattern to all matching trees?
> 
> e.g. if you create /usr/local/doc, and then do
> 
> "ls /usr/*/doc/" you will get the output of:
> 
> /usr/local/doc  /usr/share/doc

automount is not a shell and doesn't have code to expand such things and an
indirect mount may have a single map associated with the given mount point path.

Way back it was possible to associate multiple maps with a given mount point
path but that functionality was removed because it was not a feature of the Sun
automount system which autofs seeks to implement.

> 
> So with that output, you mount all of them, all the while knowing that 
> the middle component (here) is a wildcard. Then the daemon knows about 
> this wildcard and can use it while processing /etc/nas-shares.map for 
> each of them; but they are all individual mounts at that point.
> 
> e.g.  /media/me/nas and /media/pamela/nas are now individual autofs 
> mounts.
> 
> but their "/etc/nas-shares.map" is called with a different wildcard.
> 
> 
> I mean that either /media/me/nas will already exist (precreated by the 
> user) or your program creates it (I think it already does that).

This is so far from what autofs is meant to implement I can't see it not
interfering with what it is supposed to be doing.

> 
> Actually it cannot create it DUH, so the user will need to precreate 
> them.
> 
> 
> I know /media is usually seen as a managed directory but this is 
> annoying in the first place. Such an important name (/media) and I 
> cannot do anything with it because it is already taken. At the same time 
> accessing it from your home directory is usually quite annoying. So in 
> order to make sense of it I must precreate /media/xen and 
> /media/dannielle and prevent the daemon (that /media thing) from wiping 
> them.... buh.
> 
> The funny thing is that /media/xen is not even owned by xen.
> 
> The subfolders are, but the thing itself isn't (how stupid?).
> 
> So in this case perhaps not as convenient, but in the general case if I 
> precreate:
> 
> /mytree/me/nas
> /mytree/quinty/nas
> 
> etc.
> 
> Then the daemon could read
> 
> /mytree/*/nas
> 
> on program start and see all the shares that match. All the directories 
> that match.
> 
> Then it knows that "me" and "quinty" are wildcard entries for their 
> respective mounts.
> 
> 
> 
> 
> > > 
> > > 
> > > 
> > > 
> > > > By definition a program map can't know what keys may be passed to it.
> > > > And, yes, even autofs(5) doesn't say explicitly that a program map
> > > > doesn't know
> > > > what keys may be passed to it, perhaps that should be ammended.
> > > 
> > > In principle even a program map could read all of the available keys
> > > from whatever database it consults, and return all of them; or return
> > > something only (as it does now, perhaps) when a given key matches (is
> > > found).
> > 
> > I've thought of that too.
> > 
> > But I've resisted because I'm concerned about confusing existing 
> > program maps
> > people have in unpleasant ways.
> 
> At some point you need to be willing to break previous compability.
> 
> I understand that Linux is usually a running-upgrade system and even if 
> people do upgrade entire distributions they want the system to keep 
> functioning as normal without changing anything.
> 
> But e.g. the Debian update program can tell you stuff like "This package 
> does something different now, and you need to ensure it can either 
> handle or ignore the wildcard".
> 
> on every package installation.
> 
> 
> 
> > That's not a bad thought, but what would $1 be so that older scripts 
> > wouldn't
> > choke on it.
> 
> I had suggested * because there is no hostname called *.
> 
> But scripts may very well mount local filesystems as well. And they 
> might very well use a shell to do stuff like "[ -d $key ]" or "[ -d 
> "$key" ]" and....
> 
> And as always, stuff breaks a lot when people don't use "$key". If 
> people's scripts are badly written then [ -d $key ] will malfunction.
> 
> In order to cover that you would have to use something like "_", no one 
> is going to use _ as an actual filename.
> 
> 
> 
> 
> 
> > 
> > > 
> > > So, for instance, if $2 is "browse" the script may choose to return a
> > > browse map, or list of shares, with whatever options are available (it
> > > could even return the UID/USER of the share given options, so that the
> > > created directory would have the same UID as the to be mounted share!)
> > > -- meaning it might simply return ALL entries it could possibly return
> > > in the default format, which is the meaning of the wildcard, and the
> > > program (autofs daemon or module(?)) would be able to generate the
> > > appropriate directory entries based on that. So in that case you're
> > > probably going to be rid of the problem of the non-matching state of
> > > pre- and post-mount of directories.
> > 
> > which pretty much defeats the only possible solution I can think of for 
> > "browse"
> > option mounts ...
> 
> You mean the one suggested above. But if you make an alternative readdir 
> having a list available won't bug you right?
> 
> I mean, having mount information available in advance cannot be a 
> problem, only a help.
> 
> 
> 
> 
> > Like I said, I wanted to have the default as "browse" but got hit with 
> > too many
> > complaints and was only willing to change the package installed default
> > configuration to satisfy them.
> > 
> > I'm not going to try that again.
> 
> "Design by committee" they call it ;-).
> 
> Someone told me that OpenSUSE has had "browse" as the default for at 
> least 5 years.
> 
> You did think about making a distinction between static and LDAP/NIS 
> maps? I'm not sure I am not crossing my bounds here now ;-). Seeing that 
> I already suggested it.
> 
> 
> 
> 
> 
> > > In this way no one can ever get confused, so you might just rename
> > > autofs(5) to autofs.map(5).
> > 
> > Yeah maybe ...
> > 
> 
> > > I will look into providing a patch, but I hope you can rename 
> > > autofs(5)
> > > to autofs.map(5) before doing anything like that. I don't feel like
> > > improving a system I know myself haven't had access to, in that sense. 
> > > I
> > > don't mean you have to do that straight away, but if that is a 
> > > possible
> > > direction to go, or change to make, then I could supply a patch or
> > > anything based on the system that would result from that, seeing that
> > > autofs(5) is already pretty good in that sense.
> > 
> > At some point I'll need to return to this and make a list.
> > Unfortunately that time isn't now.
> 
> 
> Here is just my short summarize:
> 
> 
> - suggested compromise between browse and nobrowse by only making browse 
> for static lists the default to please new people unaware of what will 
> happen
> 
> - suggested perhaps using _ as the wildcard key, not using that "browse" 
> flag as $2, but allowing a different "primary" key as $2 if the script 
> wants it.
> 
> - suggested changing autofs(5) to autofs.map(5) such that users will be 
> able to find it.
> 
> And the rest is up to you, I think you don't need anyone rewriting it 
> for you now, but I could still try my hand if that was necessary. I just 
> think using better terminology is dependent on the first two suggestions 
> here, particularly perhaps the second, and otherwise you'll have to 
> explain current terminology better and create some way to separate "key" 
> in auto.master from "key" in "autofs.map" ;-).
> 
> However if scripts could accept an additional "key" and the default 
> operation or capability would for them to be able to return a "list" 
> then it would become more clear that the additional "key" would be the 
> index to retrieve that list, and that the entries resulting from that 
> may have their own "keys" but these "keys" are then used for the 
> individual "mount points".
> 
> So then you have reason to call the entry keys "entry keys" and the 
> other ones "lookup keys" for instance. Then, static maps use "entry 
> keys" but scripts have "lookup keys" as their 2nd parameter. The script 
> then has "entry key" as the first, and "lookup key" as the second 
> parameter. Then, entry key is equivalent to subdirectory, and lookup key 
> is often going to be equivalent to e.g. hostname.
> 
> A script without list functionality only accepts "entry keys" but may 
> very well use them as lookup keys as well.
> 
> A database also uses them as equivalent. But at least now you have 
> something of a model to conceptually separate them. Since browse already 
> functions for e.g. LDAP, in many cases you will not use a lookup key for 
> that, simply because (a) /etc/autofs_ldap_auth.conf already contains 
> such paramters and (b) the other parameters are specified in the 
> mapname-format for LDAP.
> 
> So what we call "lookup key" would already be encoded in the LDAP URL 
> but such a thing is not possible for scripts.
> 
> So you could call scripts the exception to the rule in not having this. 
> Of course it exists,: the "-D" parameters.
> 
> (I think? Haven't tested it yet).
> 
> So I don't know.
> 
> 
> 
> 
> For a database the entry key is always going to be a lookup key to the 
> database.
> Entry keys can also result from a list fetch.
> 
> So they can result from filesystem access or from some list retrieval.
> 
> But if "browse" becomes more normal and standard, you would see that it 
> becomes easier to talk about this "entries from the list you get when 
> quering the database/script for it".
> 
> Then the gap between "entries from the list" and "subdirectories 
> created" is smaller than the gap between "filesystem access" and 
> "subdirectories created".
> 
> Meaning: key (database designator) -> list -> entries (database entries) 
> is conceptually easier to grasp than key (filesystem access) -> entry 
> (database entry).
> 
> Or key (filesystem access) -> database lookup -> entry.
> 
> Simply because it is hard to understand that user action (filesystem 
> lookup) is the input to the entire system and is actually the keys being 
> used.
> 
> Even if it's not always to be used, having a level in between that ties 
> "key" to "pre-existing shares" or "pre-existing entries" becomes easier. 
> When a user thinks of "key" they think about either existing shares or 
> existing hosts.
> 
> Then conceptually it is logical to assume that the script or database 
> can give back this list. And that either this list is precreated on the 
> harddrive, or at least that it is used to match the "keys" against, for 
> instance.
> 
> (By harddrive I mean filesystem (hierarchy)).
> 
> This is why I think standard operation for at least smaller lists should 
> be a form of browse or at least a form of list retrieval, if you are 
> going to have dynamically generated lists in any case.
> 
> Then, it conforms more to a sense of what a general computer user is 
> going to have of the idea of auto-mounting mount-points. It (he/she) 
> thinks those mount-points are already going to exist, prior to being 
> automounted.
> 
> This in turn makes it easier to explain that the key is really a 
> subdirectory, and that there can be another key if needed to specify the 
> host, or something of the kind.
> 
> Then, the most common use case for a new or general computer user 
> wanting to try this is going to be met, because that use case is going 
> to involve auto-mounting network-shares of which there is a limited 
> number.
> 
> And it will involve something like
> 
> /nas/media
> /nas/backup
> /nas/home
> 
> etc.
> 
> Now I'm not sure if it covers the entire use. But at least you'd also 
> have a script that can generate a static map for you that you can then 
> use in place of the scripts, if you want to prune the mounts that are 
> actually visible to you.
> 
> Apart from the distributions themselves there is not really any software 
> in the Linux world that is tailored to either the desktop user, or the 
> server user, but not both, is there?
> 
> Apart from the desktop environments.
> 
> Anyway this was a long read and write, I guess, for both of us :P.
> 
> Time to get some food again or at least some water :p.
> 
> See ya.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe autofs" in
--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: documentation issues

[Marion, Mike]

On Mon, Jul 11, 2016 at 10:09:50PM +0800, Ian Kent wrote:

> What is needed is a re-write of the user space <-> kernel space communication
> essentially to add readdir functionality.
> 
> So that listing a directory would show the possible map mount points without the
> need to create the mount point directories.

Oh my god.. we need this!  

We have huge direct maps here and with the changing of /etc/mtab to a
link to /proc/self/mounts things have gotten horrible.  We've worked
with suse to help speed up the start of autofs (it was taking 6+ hours
to simply start the first time due to how automount reads the mtab file,
stats paths, does the mkdir, etc).  But things like mkinitrd running
with our huge maps loaded can take 20-30 min due to all the parsing
different steps do for various filesystem checks. I've learned to stop
automount and do a full umount -t autofs -a before running something
like a zypper update with a kernel upgrade.

-- 
Mike Marion-Unix SysAdmin/Sr. Staff IT Engineer-http://www.qualcomm.com
Brian: "So enlighten us, what's your idea of the perfect man?"
Helen: "Oh, I don't know...  Solid, sensitive and dependable, but with a sense of 
whimsy, a love of music and children.  A sense of commitment but a respect for my 
independence; adventurous yet a homebody... I really hadn't thought about it that 
much."  Joe: "Sounds great Helen, but Kermit is a frog."  => Wings--
To unsubscribe from this list: send the line "unsubscribe autofs" in

Re: Documentation Issues

[Tom King]

on 11/26/11:

Bernhard Guillon Bernhard.Guillon at opensimpad.org

>Hi,
>today I wanted to create a Template:We-move-to-oe-core (something like
>this artikle is a stub) to mark all pages which needs a change and inform
>new users about the move to oe-core. I created an account but I am unable
>to edit anything. I even cannot edit my user page.>>
>
>Best regards,
>Bernhard

You now have access

Tom

Re: Documentation Issues

[bernhard.guillon]

----- Original Message -----
From: ka6sox@gmail.com
To: openembedded-devel@lists.openembedded.org
Date: 28.11.2011 23:34:56
Subject: Re: [oe] [OE] Documentation Issues


> on 11/26/11:
> 
> Bernhard Guillon Bernhard.Guillon at opensimpad.org
> 
> >Hi,
> >today I wanted to create a Template:We-move-to-oe-core (something like
> >this artikle is a stub) to mark all pages which needs a change and inform
> >new users about the move to oe-core. I created an account but I am unable
> >to edit anything. I even cannot edit my user page.>>
> >
> >Best regards,
> >Bernhard
> 
> You now have access


Hi,  
thanks but I think you approved the wrong Bernhard :)
My username at the wiki is "Mr-nice" because Bernhard was allready taken.

Best regards,
Bernhard

documentation issues

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 393 bytes --]

i'd fix these myself, but i don't know what the right answers are.

in the man page for git-diff-index (posted here):

   http://www.kernel.org/pub/software/scm/git/docs/git-diff-tree.html

there are several references to "git-diff-tree" where it seems like 
"git-diff-index" is really meant.

also, git-diff-index supports a "-r" option, but that doesn't seem to be 
documented on this page.

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 445 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Open Source NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard

Re: documentation issues

[Chris Shoemaker]

On Thu, Dec 01, 2005 at 03:28:53PM -0500, Chuck Lever wrote:
> also, git-diff-index supports a "-r" option, but that doesn't seem to be 
> documented on this page.

I think it is "supported" in the sense that git-diff-index won't
complain about "-r" argument, but it doesn't actually do anything, so
it shouldn't be documented.

-chris

Re: documentation issues

[Junio C Hamano]

Chuck Lever <cel@citi.umich.edu> writes:

> in the man page for git-diff-index (posted here):
>
>   http://www.kernel.org/pub/software/scm/git/docs/git-diff-tree.html

You mean git-diff-index.html, not diff-tree.html, I suppose.

> there are several references to "git-diff-tree" where it seems like 
> "git-diff-index" is really meant.

    Non-cached Mode
    ---------------
    The "non-cached" mode takes a different approach, and is potentially
    the more useful of the two in that what it does can't be emulated with
    a "git-write-tree" + "git-diff-tree". Thus that's the default mode.
    ...
    The non-cached version asks the question:

      show me the differences between HEAD and the currently checked out
      tree - index contents _and_ files that aren't up-to-date

    which is obviously a very useful question too, since that tells you what
    you *could* commit. Again, the output matches the "git-diff-tree -r"
    output to a tee, but with a twist.

Linus does mean diff-tree here.

We had diff-files and diff-tree back then, but diff-index came
much later under the name diff-cache.  He is explaining what
git-diff-index does in terms of git-diff-tree in this paragraph.

He could have said:

    You *could* emulate it with first copying the index file
    into a temporary file, git-update-index all the modified
    paths in the working tree into that temporary index file and
    git-write-tree that, and then run git-diff-tree.

but that is not practical.

> also, git-diff-index supports a "-r" option, but that doesn't seem to be 
> documented on this page.

What Chris Shoemaker said is right.

But we could do this instead if you want.  I am neutral myself.

-- >8 --
[PATCH] Documentation: mention -r for all commands from the diff family.

We did not mention 'git-diff-index' and friends take '-r' but
silently ignore it, because they always operate on the whole tree.

Signed-off-by: Junio C Hamano <junkio@cox.net>

---
diff --git a/Documentation/diff-options.txt b/Documentation/diff-options.txt
index 6b496ed..7299903 100644
--- a/Documentation/diff-options.txt
+++ b/Documentation/diff-options.txt
@@ -58,5 +58,11 @@
 	Swap two inputs; that is, show differences from index or
 	on-disk file to tree contents.
 
+-r::
+        Recurse into sub-trees.  Note that this makes difference
+        only with `git-diff-tree`.  Other commands in the `git-diff-\*`
+        family always operate on the whole project tree and this
+        option is a no-op to them.
+
 For more detailed explanation on these common options, see also
 link:diffcore.html[diffcore documentation].
diff --git a/Documentation/git-diff-tree.txt b/Documentation/git-diff-tree.txt
index 9a2947e..8f66119 100644
--- a/Documentation/git-diff-tree.txt
+++ b/Documentation/git-diff-tree.txt
@@ -33,9 +33,6 @@ include::diff-options.txt[]
 	Note that this parameter does not provide any wildcard or regexp
 	features.
 
--r::
-        recurse into sub-trees
-
 -t::
 	show tree entry itself as well as subtrees.  Implies -r.
 

Re: documentation issues

[Chris Shoemaker]

On Thu, Dec 01, 2005 at 01:34:55PM -0800, Junio C Hamano wrote:
> > also, git-diff-index supports a "-r" option, but that doesn't seem to be 
> > documented on this page.
> 
> What Chris Shoemaker said is right.
> 
> But we could do this instead if you want.  I am neutral myself.

I think it is more clear without this, since we imply elsewhere that
diff-* accepts the options in diff-options.txt.  Documenting "-r" in
the only place it applies (diff-tree) means we don't need exceptions
anywhere else.

Incidentally, I think it would make more sense if diff-tree defaulted
to recursive behavior and required a flag to produce non-recursive
behavior, but I realize that's impractical now, and only of minor
benefit.

-chris

Re: documentation issues

[Chuck Lever]

[-- Attachment #1: Type: text/plain, Size: 358 bytes --]

Junio C Hamano wrote:
>>also, git-diff-index supports a "-r" option, but that doesn't seem to be 
>>documented on this page.
> 
> 
> What Chris Shoemaker said is right.
> 
> But we could do this instead if you want.  I am neutral myself.

thanks for clarifying.

StGIT does use "git-diff-index -r" in git.__tree_status().  i think it 
can be safely removed.

[-- Attachment #2: cel.vcf --]
[-- Type: text/x-vcard, Size: 445 bytes --]

begin:vcard
fn:Chuck Lever
n:Lever;Charles
org:Network Appliance, Incorporated;Open Source NFS Client Development
adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA
email;internet:cel@citi.umich.edu
title:Member of Technical Staff
tel;work:+1 734 763 4415
tel;fax:+1 734 763 4434
tel;home:+1 734 668 1089
x-mozilla-html:FALSE
url:http://www.monkey.org/~cel/
version:2.1
end:vcard